Evaluation

TL;DR Before I merge anything where being wrong would be silent, I paste it into a live Gemini session and ask that model to attack it. Not “review this,” which gets you a rubber stamp with bullet points. Attack it. Over four days in April, that gate killed a performance optimization 88 minutes after it shipped, found two engine bugs that meant every historical number in my trading leaderboard was inflated, and identified five separate ways my “out-of-sample” evaluator was reading the future. Claude wrote all of that code. Claude also reviewed all of that code. Claude caught none of it. The useful property is not that Gemini is smarter, it is that Gemini did not write the thing and has no investment in the design being correct. ...

TL;DR I run an LLM-driven trading hypothesis engine. For a while, every result that came back looked too good — Sharpe ratios above 5, win rates above 70%, all on out-of-sample windows. They were lies. The model was reading dates, headlines, and tickers in the prompt and pattern-matching against its training data, which extends well past my “out-of-sample” cutoff. The fix was a masking layer I now call Blind Oracle: strip every leak before evaluation, run the trigger before the eval, gate promotion on out-of-sample Sharpe with the masking enforced. After it shipped, the inflated numbers collapsed back to honest reality. Some hypotheses survived; most didn’t. That’s exactly what I needed to know. ...