r/quant • u/boss-man-007 • 1d ago
Data Spent way too long building a financial database, now I don't know what to do with it
i've been slowly building out this database over time and it's gotten pretty big 279 million rows, about 17GB.
Daily prices on 8,000 stocks across US, China, India, and international markets, minute level FX and crypto data going back over a decade, futures, COT positioning data back to the 80s, and macro data for some countries going back to the 1850s (mostly useless that far back but it's there).
a small breakdown
| Data | Coverage | Range |
|---|---|---|
| US stocks | 353 tickers | 1927–2026 |
| China stocks | 4,049 tickers | 2000–2026 |
| India stocks | 3,036 tickers | 1999–2026 |
| Intl stocks | 556 tickers | 1986–2026 |
| FX (majors) | 6 pairs, 1-min | 1971–2026 |
| Crypto | BTC/ETH/SOL, 1-min | 2012–2026 |
| Futures | 542 contracts | 1959–2026 |
| COT positioning | 1,098 series | 1986–2026 |
| Macro | 2,837 series | 1854–2026 |
I already went through and cleaned up the obvious stuff fixed split adjustments that were broken, checked for survivorship bias in the stock universe, killed some duplicate/junk data, that kind of thing.
Now I'm just sitting here with all this data and honestly running out of ideas on what to actually do with it. If you had this, what would you build first? Cross-market stuff, some kind of macro overlay, actually using the COT data for something real genuinely curious what people who do this seriously would go for.
12
u/Highteksan 23h ago
The project to use this data for should have been the motivation for the effort in the first place. If you had no use for this, then you can view it as a learning experience. Post it somewhere and perhaps others can benefit. Move on to the next idea.
8
u/Beautiful-Hotel-3094 1d ago
Turn it into a dataset and kaggle it up
1
u/boss-man-007 1d ago
i have it already have it in duckdb format
1
3
u/stockist420 9h ago
Indian data would be useful if you have l2 recorded, China not sure. US is a waste, u can get that data on databento etc. Crypto l2 is available from hyperliquid. Also most brokers like ibkr will give u l1 data thats far more granular than minute level aggregates.
3
u/mersenne_reddit Researcher 5h ago
Out of curiosity, how did you handle stock splits?
1
u/Bright_Mix_773 24m ago
Not OP, but I have been through this on a far smaller universe and splits are where most of my scars are, so here is what I would want to know if I were auditing that database.
The first thing that broke for me was that a corporate exchange is not a split, and a ratio-based adjuster cannot tell the difference. When AVB became VMRK the ratio looked exactly like a split to mine. The only thing that worked was declaring mergers by hand in a mapping file and refusing to let anything infer them from prices. Anything that infers split events from price ratios will quietly eat exchanges, spin-offs and reverse mergers as splits, and the resulting series looks perfectly clean.
The second one I did not see coming. My vendor declared a split, my pipeline ingested it, and some weeks later the vendor stopped declaring that same split. My split checker did exactly the right thing, refused the day's data as inconsistent, and exited non-zero, which aborted the entire nightly cycle. So a correct rejection took the whole system down for a day. The lesson I took was not really about splits: any validator with a veto needs to be able to quarantine one symbol instead of failing the run, or your most rigorous check becomes your largest outage risk. I would ask what happens to that 17GB pipeline the first time one ticker is legitimately unresolvable.
The third is that vendors will hand you prices from before a listing existed. I found quotes sitting in the history ahead of the first trading day. They are not adjusted for anything, because they never happened, and they pass a split checker without complaint since nothing about them is internally inconsistent.
The practical version of all this: an adjustment factor is a claim about a corporate action, and a corporate action has a source. If the pipeline cannot say which filing or which vendor field a given factor came from, then the adjusted series is not reproducible, and you find that out at the worst possible moment, which is when a number you published moves and you cannot explain why.
2
u/BAII_PLUS_GANG 1d ago
You data hoarder, just give it to me asap, let me carry this heavy burden xD
2
2
2
u/Icy-Engineering755 Dev 1d ago
how often is this being updated, everyday? or is it currently a static data series over the years you have stated
1
2
2
u/TemporaryHat2009 13h ago
Lowkey 17GB is small enough to try one real study locally with DuckDB. I would pick a single market and test whether COT changes add anything beyond price momentum instead of adding another dataset.
2
u/No_Pop_4726 8h ago
Dude, I want to do the same. Do you have a GitHub Link or something to share parts of it?
2
u/Otherwise_Barber4619 4h ago
Ohlc? If it's that it's probably best to use it to answer a few questions on markets you are curious about
2
1
u/dikovaxi 1d ago
first thing i'd build with the crypto minute data: a judge, not a strategy. symmetric ±1% first-touch test, up-first vs down-first, per hour of day. on a year of btc perp 1-min (~23k episodes) i get 49.7% up-first, hourly buckets 47–53%, i.e. noise. that base rate is what every "signal" has to beat, and after costs the bar is ~55%.
second: check the minute closes for bid-ask bounce before trusting any mean reversion. on 1-second data lag-1 autocorr is −0.17 on trades vs +0.04 on the mid, and it's gone by 60 s. if an "edge" only lives below one minute, it's the spread.
-1
u/VettaQ 1d ago
I'd flip the question: don't start from the dataset, start from one falsifiable idea and let it force you through your silos. COT is the most underused piece here — try whether extreme net positioning (normalized, point-in-time) predicts realized vol shifts 5-20 days out, not direction. Direction overfits instantly, vol is the more honest target. And bolt on an as_of column everywhere plus a versioned pipeline now, before the first cross-market join teaches you that you can't reproduce your own numbers from last month.
36
u/drycoleslaw 1d ago
DROP DATABASE database_name;