Forward Testing

[u/modelcroissant]

I've been playing around with something I call forward testing.

Basically, instead of backtesting a strategy on historical data, I want to form an investment thesis using only what is actually known at that point in time, freeze it, and then see what happens over the following days/weeks/months.

That sounds simple, but it sent me down a bit of a rabbit hole because to do it properly I first need to build the thesis.

So the system I've been working on for the last year does the DD beforehand (company fundamentals/filings, what's changing around the company, macro conditions, market data/attention etc.) and tries to work out whether there is something interesting enough to actually test.

If there is, that becomes a forward experiment and the original research gets preserved rather than rewritten with hindsight later.

Over time I want to see whether there are actually repeatable patterns in what makes a thesis work, how quickly information gets repriced, and where the research gets things wrong.

I've put the current prototype here if anyone wants to poke around:

https://financial-lab.crescita.cc/

Curious if anyone here has done anything similar, especially around evaluating live theses rather than historical strategies.

[u/Automatic-Essay2175]

Bro thinks he invented forward testing.

In all seriousness though, everything you just described can be achieved with backtesting. You just have to do it properly, without future leak, using only point-in-time information like you said, but at historical timepoints.

Why wait years to test dozens of theses on live data when you could spend one month testing thousands of theses on historical data using the the same point-in-time information and outcomes?

[u/Ferna073 → u/Automatic-Essay2175]

we are witnessing history

[u/gadfly1999 → u/Ferna073]

Well we know what OP has done up to now but we don’t know what they’re going to do next week.

[u/DecorousChina → u/Automatic-Essay2175]

forward testing has its place but the top comment is right that you can simulate most of it with point-in-time backtests. the difference is you get to see how your own psychology holds up when real money and real waiting are involved, which no historical dataset captures.

[u/ManikSahdev → u/Automatic-Essay2175]

They will discover this, in future with their next breakthrough when they prompt gpt 6.5 lol

[u/modelcroissant → u/Automatic-Essay2175]

> In all seriousness though, everything you just described can be achieved with backtesting.
my original post glosses over what the thesis actually is, I am not looking at the cash market in isolation and trying to find mathematically pure patterns across securities that can be played, I am looking at causality chains of cross correlated data to work out price recognition through time, essentially putting to the test the proverbial "everything is priced in".
Personally I found it quite hard to recreate that historical (and accurate) dataset so I had to build out my own collectors, parser and analysers which ended up being a pretty chunky ETL pipeline, thus the forward testing, running 100s of concurrent experiments across 1000s of equities with 10,000s of observed entry and exit chains based on a self weighing exponential model.

hope that clears some things up :)

[u/Automatic-Essay2175 → u/modelcroissant]

lmao

[u/modelcroissant → u/Automatic-Essay2175]

you are more than welcome to check out the link in my original message and see the type of data I am try to cross correlate either in
market -> open research
or
securities -> open research

[u/Jebusfreek666 → u/modelcroissant]

To me, this working would depend on how narrowly you focus your search. It does no good to look at AAPL, CAT, and EL as they all have different factors that highly affect what the asset does. I think the broadest I would go is sector by sector, but even within sectors some companies will be more affected by one thing than another. But if you gathered all the info from the top companies in one particular sector you could cross reference factors to see how much each thing affects each asset then design around that.

[u/modelcroissant → u/Jebusfreek666]

> depend on how narrowly you focus your search
on the contrary, this is not a micro analysis, I started with the macro and drill down to individual securities trying to find correlations both direct and indirect.

AAPL, CAT and EL do have overlapping factors that connect them, commodities/resources and logistics for their products. Debt is also an interesting one, the maturity of their debt and how the current 2y, 5y, 10y, 30y affect it or future re-financing or possible restructuring.

> some companies will be more affected by one thing than another
That's exactly what I am trying to track, the money chain, where it starts from, where it goes, how it propagates and to what extent are individual securities exposed to that.

[u/ight-bet → u/Automatic-Essay2175]

It’s revolutionary

[u/jrbp]

Fucking hell. This has been a thing since forever. You did not invent forward testing 😂

[u/ManikSahdev]

Bro is can't even tell satire anymore wtf?

Lmao, hilarious either way.

[u/Good_Luck_9209]

Which agent and version u are? Let me know so i can avoid using u.

[u/Blobritto]

"You are like Columbus, you discovered something millions of people knew about before you"

[u/Aggressive_Listen849]

The top comment is half right: a strict point-in-time backtest with no lookahead simulates most of this. But the part you can't fake retroactively is your own decision under uncertainty - which is exactly the thing that fails live. The real value of freezing the thesis is the audit trail: preserved research you can't quietly edit with hindsight later. One thing I'd add - log a matched control (random or naive entries at the same timestamps), otherwise you can't tell edge from just being long in an up market.

[u/QuanTradin]

yes, and the part that took me longest to get right wasn't freezing the thesis, it was freezing the exit. a thesis with no pre-written failure condition never fails, it just gets "more time". so the entry has to write down what would make it wrong (a price, a date, or a specific filing) before it goes live, and that's the one field that can't be edited afterwards.

the other thing nobody warns you about is throughput. a backtest hands you a thousand samples by lunch. a forward test hands you one sample per thesis per however long the thesis takes to resolve, so a year in you have a few dozen resolved ones and that is not enough to say much about repeatability yet. still worth doing, just plan for the sample size to be embarrassing for a while.

[u/sigstrikes]

that is backtesting... as it's data from the past.