Case study · finLM · point-in-time equity research, NSE 2010-2026
Five times this project was wrong. Each one was caught a different way.
finLM is two things: a research stack for Indian equities built on 16 years of point-in-time NSE data, and a governance layer that stops a language model from placing an order it should not place. The findings are the smaller half of the story. What follows is the trail of what broke, how each failure was found, and what was built afterwards so it could not happen the same way twice.
The arc, before the detail
Every row is something the project asserted and then had to withdraw. The third column is the part worth reading, because a result you retract yourself is only as good as the mechanism that caught it.
| # | What turned out to be wrong | How it was caught |
|---|---|---|
| 01 | The headline per-trade result: 935 trades, 52.6% win, +1.2% alpha | AUDIT · would not rerun |
| 02 | Market breadth explains why fade alpha varies year to year | PRE-REGISTRATION · falsified |
| 03 | The risk gate cannot be talked past | ADVERSARIAL SUITE · 4 bypasses |
| 04 | A momentum t-statistic of 2.79 that should have read 2.28 | PROVENANCE · DB fingerprint |
| 05 | The sign test is the strongest evidence in the project | MORE DATA · p improved as edge died |
Incident 04 was caught by infrastructure that only exists because of incident 01. That is the single thing this project is actually about.
First, what has and has not been run
Before any number appears, the boundary. This is research infrastructure and a governance layer, not a live trading system, and the distinction is the whole point:
- REALBacktests over 2010-2026 NSE data, point-in-time and survivorship-verified
- REAL13 decisions journaled with their reasoning, scored against actual NSE closes
- BUILTBroker integration implemented and unit-tested against a mock
- ZEROOrders ever placed. Not one.
- NEVERBroker account authenticated. No session token has ever existed.
- NONECapital at risk, at any point
No order was placed because the file authorising one was deliberately never written, so the gate refused every time. That is the design working rather than a missing feature. Where a return figure appears, it is a paper decision scored against real closing prices: real data, real forward scoring, no money.
Incident 01 · caught by audit
The number that would not reproduce
The project's headline was a per-trade figure for the pullback-entry fade strategy. It had been written into the documentation months earlier, quoted in a deck, and treated as settled. Rerunning it as a routine check was supposed to take ten minutes.
- Believed
- 935 trades, 52.6% win rate, +1.2% alpha per trade, profit factor 1.18, improving out of sample
- Broke
- The rerun gave 754 trades, 50.9% win, +0.98% alpha, PF 1.14. Eighteen parameter combinations were tested. None reproduced all four figures together. Counts near 935 existed but always with worse quality, never better.
- Ruled out
- Not a code regression: the entry logic and exit defaults were behaviourally identical between the commit that produced the numbers and HEAD. Not corporate actions, which raise the count. Not a forked code path, since the regime study imports the same simulator.
- Cause
- The original run was never captured. No config dump, no output file, just numbers transcribed by hand into a document. A process defect, not an analytical one, which makes it worse rather than better.
- Changed
- Every result-producing run now writes an artifact stamping the commit hash, whether the working tree was dirty and which files, a database fingerprint, library versions and the full config. There is no opt-out flag, and a test asserts there never will be. Documentation cites the artifact instead of retyping the number.
What survived: the shape of the finding. Across all eighteen configurations fade alpha lands between +0.47% and +1.11% with profit factor 1.00 to 1.21. The direction is robust. The decimal places were never real.
Incident 02 · caught in advance, on purpose
The hypothesis that was killed before it could spread
Fade alpha measured four times larger in the historical sample than in the live window. Survivorship was already excluded, so the candidates were genuine edge decay or a regime effect. There was an obvious regime candidate sitting in the data: market breadth correlated at r = −0.64, meaning narrow years faded better. Directionally sensible, and easy to believe.
Rather than adopt it, the test was written down first: a fitted model, a prediction interval, and an explicit region of outcomes that would kill the hypothesis, committed before live breadth was computed.
- Believed
- Market breadth explains the year-to-year variation in fade alpha
- Rule set first
-
Fit on 11 historical years:
alpha = 12.481 − 0.2002 × breadth. The live result is consistent with the model only if breadth is at least 53.30%. Anything between 47.62 and 53.30, about 94% of the historical range, contradicts it. - Observed
- Breadth came in at 48.35%, inside the historical range and so not an extrapolation. The model predicted +2.80% with a 95% interval of [+1.57%, +4.04%]. Actual alpha was +0.58%.
- Verdict
- Contradicted, and directionally so. A breadth of 48.35% sits near the low end, and the slope is negative, so the model said this should have been one of the best fade regimes on record. It was the worst measured.
- Epilogue
- When the 2021-2024 data gap was later filled, the in-sample correlation itself collapsed from −0.635 to −0.029. The effect did not just fail out of sample. It stopped existing in sample too.
A hypothesis you can only confirm is not a hypothesis. Writing the falsification region down first is what made this a test instead of a story.
Incident 03 · caught by attacking it
The gate broke four ways
An LLM handed an order tool and told to "always ask first" is not governed. The
reference LLM trading server in India gates real orders behind a
confirmed=true parameter that the model sets on itself. That is theatre.
finLM instead puts authorisation in a file the human writes, which no code path in the package can create, plus a filesystem kill switch and no bypass parameter anywhere in the tool surface. Then it was attacked on the working assumption that the model is adversarial. Fifty-eight tests. It lost four times.
| Bypass | How it worked |
|---|---|
| Price spoofing | A market order valued from a caller-supplied price of 1.00 cleared a 50,000 cap while executing at 500,000 |
| Negative quantity | -100 passes the quantity cap and makes value negative, clearing both caps at once |
| Unicode evasion | A zero-width space or full-width characters slid past the symbol blocklist |
| NaN price | Every comparison with NaN is false, so the value cap silently passed |
- Believed
- Authorisation lives outside the model, therefore the gate holds
- Root cause
- All four are one bug wearing four hats: the gate trusted caller-supplied values as ground truth. That is the classic confused deputy problem, a program holding authority the caller lacks, tricked into using it because it believed the caller's description of the request.
- Changed
- Price now carries a source label and only a genuinely binding limit price is trusted; quantities are type-checked before comparison; symbols are Unicode-normalised and stripped of format characters before matching; non-finite values are rejected outright. An aggregate daily value cap was added. The suite runs in CI.
The claim is not that the gate is unbypassable. It is that it was attacked, it broke, it was fixed, and there is now a suite whose only job is to keep it shut.
Incident 04 · caught by instrumentation
The database moved underneath the analysis
With the multiple-testing correction running and a long historical ingest still writing, two things happened at once that should not have. The first was loud: the ingest died on a locked database and left an orphan staging table behind. The second was silent and much worse.
- Symptom
- The momentum t-statistic came back at 2.79. The previous run of the same method on the same code had given 2.28. Same method, same parameters, different answer.
- Cause
- The analysis had read a moving dataset. The database grew past 2020 partway through the run, so the sample at the end of the analysis was not the sample at the beginning.
- Why it mattered
- On its own this looks exactly like a methodology bug, and the natural response is to go and rewrite correct statistics code. Only the database fingerprint written by the provenance layer made it diagnosable as a data problem in minutes.
- Changed
- Freeze the data, then analyse. Written into the project's standing notes with the failure mode described, because the loud version of this failure is harmless and the quiet version silently corrupts a published number.
The provenance layer built after incident 01 caught incident 04. That is the argument for boring infrastructure, made by the project on itself.
Incident 05 · caught by extending the sample
The p-value that improved while the strategy died
The strongest-sounding statistic in the project was a sign test. Fade alpha had been positive in all eleven years of the historical sample, which under a coin-flip null gives p = 1/2048 = 0.0005. It was quotable, it was clean, and it led every summary.
Then the 2021-2024 data gap was filled and the series ran continuously from 2010 to 2026, 7,061,494 rows with zero gaps.
- Believed
- Positive in all 11 years, p = 0.0005. The headline result.
- What happened
- All 16 years came back positive. The sign test went to p = 1/65,536, thirty-two times more significant. Over the same period the effect size fell about 78%, from 3.52% in 2011 to 0.79% in 2025.
- The lesson
- A sign test measures whether the sign is real. It says nothing about magnitude. Sixteen years of +0.1% would produce the identical p-value. The statistic was getting more impressive precisely as the strategy became worthless.
- Changed
- Stopped leading with it. The honest headline is the decay: positive alpha in all 16 years, eroding monotonically at about 0.12 percentage points a year. Spearman r = −0.653, p = 0.0013, every leave-one-out slope negative.
This is a better finding than a stable edge would have been. An anomaly eroding as a market matures is what theory predicts, and it lines up with T+1 settlement completing in January 2023, an event named in the pre-registration before the data was examined. The break test fires at p = 0.0375 but drops to 0.0503 without 2020, so it is reported as suggestive and not promoted.
What survived · net-of-cost benchmark-adjusted alpha per trade, by year
Fade strategy, 2010-2025. Positive in all 16 years, and falling.
06Chasing loses. Fading worked, and is fading.
The scanner finds stocks that spiked on unusual volume. You can chase the move or fade it. Across 16 complete years, chasing lost in every configuration tested (-1.55% per trade net, n = 1,033). That is the cleanest result in the project and it needs no caveats: the system must never recommend buying a stock because it just spiked.
Fading produced positive benchmark-adjusted alpha in every single year, decaying from roughly 2.3% early in the sample to 0.79% in 2025.
07Most published anomalies do not show up here
Eleven documented equity anomalies, tested on NSE 2010-2024 with standard cross-sectional quintile sorts: 290,651 symbol-months, 161 usable months. Motivated by Hou, Xue and Zhang's Replicating Anomalies, which found most US results fail under uniform methodology. In the liquid tradeable universe here, essentially nothing survives. Widen the universe and two effects appear and hold.
| Anomaly | Wide (~445) | Mid (~234) | Liquid (~134) |
|---|---|---|---|
| momentum (12-1) | t=3.37 / +15.3% | 2.19 / +9.8% | 1.72 / +6.5% |
| 52-week high | t=3.05 / +15.3% | 1.87 / +8.1% | 1.29 / +5.1% |
| skewness | -2.43 / -8.6% | -3.15 / -11.6% | -1.43 / -8.7% |
| low volatility | 1.80 / +6.1% | 0.50 / -0.4% | 0.33 / -1.0% |
| short-term reversal | -0.92 / -7.9% | -0.13 / -4.5% | -0.22 / -5.1% |
Benjamini-Hochberg at q = 0.10 leaves three survivors. A bootstrap of the maximum |t| under the null (returns permuted within month, 1,000 iterations) gives observed 3.374 against a null 95th percentile of 2.828, p = 0.012. The strict test passes.
Two things need reading carefully. Skewness is statistically significant but not tradeable: its significance is significance of losing money, and flipped to the profitable direction it is worth about +0.8% a year, which is nil. And short-term reversal is exactly null at monthly horizon, which does not contradict the fade result so much as locate it: reversal in India lives at the daily horizon, not the monthly one.
A conditional double sort, ranking each characteristic within liquidity terciles, confirms these are not liquidity in disguise. Magnitudes shrink by about a third; significance survives.
Where the effects live
Gross t-statistic by universe width. Both survivors decay as the universe narrows toward the names you could actually trade at size.
The cost hurdle, drawn
A strategy with no edge at all, rebalancing monthly at 80% turnover, loses 6.6% a year to costs. Every finding on this page has to clear that line before it means anything.
08Why the surviving numbers are trustworthy
The methodology is the actual product. Entries are at the next day's open, because end-of-day data means you cannot act on today's close, and the scanner filters the panel to prior rows structurally rather than by discipline.
Survivorship is excluded and verified rather than asserted. The universe is built from per-day exchange archives, so each row is what actually traded that day. Of 2,543 symbols across 2010-2020, 771 stop appearing before mid-2020, and the dead end at their real demise: Satyam in July 2013, Educomp in 2017, Gitanjali in July 2018, Jet Airways in September 2019, Unitech in March 2020. A symbol list backfilled from today cannot produce that pattern.
Every result is benchmark-adjusted against an equal-weight liquid-universe index, costs are modelled rather than mentally discounted, and corporate actions are back-adjusted with the full audit trail queryable.
The July-2024 file-format seam between the two data sources was validated before the merged series was used: 100% symbol carry-over, ordinary daily returns across the join, no volatility jump. Without that check the decay finding could have been a schema artifact.
09Limits worth knowing before you trust any of it
- SHORTAbout 97% of the measured edge is short-side. An Indian retail cash account cannot hold a multi-day short, so this is a de-risking filter and not a return strategy.
- DECAYThe edge reached 0.79% in 2025. Extrapolate the slope and it hits zero within a few years.
- SIZENo market capitalisation data exists in the source, so there is no value-weighting and no true size factor.
- EODEnd-of-day data only. No spreads, no order book, no execution modelling.
- OPENFactor attribution has not been run. Until it is, momentum and 52-week-high are candidate findings and not established alpha.
10What comes next
Factor attribution. Regress the two surviving anomalies on size, value and volatility. This is the single control standing between "candidate finding" and "result", and the pre-registered stopping rule points here rather than at more resampling.
Capacity. Both survivors weaken as the universe narrows toward tradeable names. Quantifying how much capital the effect absorbs before it disappears turns a statistical result into an economic one.
Generalise the harness. Everything used to test these findings, the point-in-time universe, the cost model, the Newey-West standard errors, the FDR correction and the bootstrap null, is strategy-agnostic. Pointed at someone else's signals instead of my own, it becomes a way to measure how much of any backtest survives contact with those corrections. That is the next build.