AI summary · arXiv: 2611.03985v1 · cs.AI
Thinking Fast, Trading Late: System One Decision Models vs the Market
The essentials
Under a blinded, point-in-time evaluation, AI news signals weakened out of sample, and commercial-model signals mostly reflected price moves already realized by the market open.
Based on the abstract only: check the details in the original paper.
Question
The paper asks how much stock-return predictability attributed to language models persists when evaluation addresses look-ahead, memorization, and selection bias. It also tests whether news-based signals remain actionable after the overnight market move.
Method
The authors propose OLIVIA, using point-in-time inputs with company identities masked, validation-only model selection, a cryptographically frozen analysis plan, and a single logged unblinding of the test period. They evaluate an open-weights AI decision model fine-tuned on 2024–2025 data, three commercial models, and standard baselines on S&P 500 company news; the abstract reports two validation quarters and a blinded January–July 2026 test.
Findings
-
Result
The fine-tuned model’s daily information coefficient declined from 0.055 and 0.065 on the two validation quarters to 0.020 on the blinded test, with t = 1.7.
See the passage in the paper
The fine-tuned model's signal replicated on two validation quarters (daily information coefficients 0.055 and 0.065) but fell to 0.020 (t = 1.7) on the blinded January-July 2026 test.
-
Authors’ interpretation
Commercial-model readings tracked the overnight move to the open but not subsequent price movement, suggesting the signal was no longer actionable by the open.
See the passage in the paper
The commercial models' readings line up with the overnight move to the open (information coefficient 0.074 to 0.077) but not with the move after it, so the news is priced by the time a trader can act on it at the open.
-
Result
The models did not distinguish overreaction from under-reaction, and no preregistered strategy remained viable after costs and the deflated Sharpe ratio.
See the passage in the paper
No model can tell over- from under-reaction, and no pre-registered strategy survives costs and a deflated Sharpe ratio.
-
Result
Masking company names did not reliably conceal company identities from commercial models.
See the passage in the paper
Commercial models identify masked companies in 62% to 74% of cases against a 28% baseline: masking hides names, not companies.
Limitations
- The reported scope is limited to news about S&P 500 stocks and the specified models and test period.
- The abstract reports a single blinded test period, January through July 2026, so the findings do not establish performance across other periods.
See the passage in the paper
the blinded January-July 2026 test
Why it matters
The authors argue that rigorous, blinded evaluation substantially changes how apparent LLM-based return predictability should be interpreted: signals may weaken out of sample or be priced before traders can act. Their results also emphasize that masking names alone may not prevent company identification.
Written by AI from the paper. Passages are checked word for word against the text we analysed; the rest may contain errors.
Ask questions about this paper to our AI assistant
How to chat with this paper? Illustrated guide
You can also chat with multiple papers at once here.
Keep this conversation — a free account saves your questions and answers, your collections of papers and your alerts.
Continue with Google Create a free accountTools for this paper
Advanced tools Premium
For researchers who follow a field closely: citation alerts, journal-club slides, version comparison and Zotero export. Discover Premium
Papers around this one
See them on a mapRanked by how close their title and abstract are to this paper’s. Select some to read them together.
Closest papers
-
Trader-Company Method: A Metaheuristic for Interpretable Stock Price PredictionKatsuya Ito, Kentaro Minami, Kentaro Imajo et al. · 2020 · Trading and Market Microstructure · cited 16 times · AI summary available
Trader-Company Method uses an interpretable metaheuristic to predict stock prices, while this paper tests language-model news signals under stricter evaluation controls.
Abstract
Investors try to predict returns of financial assets to make successful investment. Many quantitative analysts have used machine learning-based methods to find unknown profitable market rules from large amounts of market data. However, there are several challenges in financial markets hindering practical applications of machine learning-based models. First, in financial markets, there is no single model that can consistently make accurate prediction because traders in markets quickly adapt to newly available information. Instead, there are a number of ephemeral and partially correct models called "alpha factors". Second, since financial markets are highly uncertain, ensuring interpretability…
-
From Knowing to Doing: A Memory-Controlled Benchmark for LLM Trading Agents on Stock MarketsTao-Jie Zhu, Wen-Tao Zhao, Rui Sun et al. · 2026 · arXiv.org · preprint, not peer reviewed · cited 14 times · Not summarized yet
KTD-Fin controls for memorized market knowledge and separates stock-selection skill from raw returns, addressing evaluation risks in LLM trading.
Abstract
Evaluating whether large language model (LLM) agents can profit in capital markets is increasingly framed as end-to-end trading: place an agent in a historical market, let it trade, and measure portfolio returns. This setup is vulnerable to two evaluation failures. First, long backtests often overlap with the knowledge cutoffs of frontier LLMs, allowing memorized tickers, dates, prices, and market narratives to substitute for investment reasoning. Second, raw returns are a noisy proxy for stock-selection ability, since positive performance may come from market beta, style exposure, or favorable regimes rather than genuine alpha. We introduce KTD-Fin (Knowing-To-Doing Financial Benchmark), an…
-
Learning to Trade Like an Expert: Cognitive Fine-Tuning for Stable Financial Reasoning in Language ModelsYuchen Pan, S. Liew · 2026 · arXiv.org · preprint, not peer reviewed · Not summarized yet
Cognitive Fine-Tuning trains LLMs with textbook and historical-market questions, whereas this paper tests news-based return signals with point-in-time controls.
Abstract
Recent deployments of large language models (LLMs) as autonomous trading agents raise questions about whether financial decision-making competence generalizes beyond specific market patterns and how it should be trained and evaluated in noisy markets lacking ground truth. We propose a structured framework for training and evaluating such models. Central to our approach is a curated, multiple-choice question (MCQ) dataset derived from classic textbooks and historical markets, verified by an AI committee, enriched with structured reasoning traces, and augmented to reduce shortcut learning. To evaluate whether performance on isolated MCQs generalizes to real-world trading, we introduce a two-st…
-
Going All-In on LLM Accuracy: Fake Prediction Markets, Real Confidence SignalsMichael Todasco · 2025 · arXiv.org · preprint, not peer reviewed · cited 2 times · Not summarized yet
Fake Prediction Markets tests whether betting-game framing improves LLM forecasts of other models’ answers, rather than stock returns.
Abstract
Large language models are increasingly used to evaluate other models, yet these judgments typically lack any representation of confidence. This pilot study tests whether framing an evaluation task as a betting game (a fictional prediction market with its own LLM currency) improves forecasting accuracy and surfaces calibrated confidence signals. We generated 100 math and logic questions with verifiable answers. Six Baseline models (three current-generation, three prior-generation) answered all items. Three Predictor models then forecasted, for each question-baseline pair, if the baseline would answer correctly. Each predictor completed matched runs in two conditions: Control (simple correct/i…
-
StockBench: Can LLM Agents Trade Stocks Profitably In Real-world Markets?Yanxu Chen, Zijun Yao, Yantao Liu et al. · 2025 · Machine Learning · preprint, not peer reviewed · cited 49 times · AI summary available
StockBench evaluates LLM agents in realistic iterative trading, while this paper tests whether news-based signals predict returns and remain actionable at the open.
Abstract
Large language models (LLMs) have recently demonstrated strong capabilities as autonomous agents, showing promise in reasoning, tool use, and sequential decision-making. While prior benchmarks have evaluated LLM agents in domains such as software engineering and scientific discovery, the finance domain remains underexplored, despite its direct relevance to economic value and high-stakes decision-making. Existing financial benchmarks primarily test static knowledge through question answering, but they fall short of capturing the dynamic and iterative nature of trading. To address this gap, we introduce StockBench, a contamination-free benchmark designed to evaluate LLM agents in realistic, mu…
-
Can LLM-based Financial Investing Strategies Outperform the Market in Long Run?Weixian Waylon Li, Hyeonjun Kim, Mihai Cucuringu et al. · 2025 · Knowledge Discovery and Data Mining · cited 32 times · Not summarized yet
FINSABER tests whether LLM investing strategies outperform across longer periods and broader stock universes, examining robustness to biases in backtests.
Abstract
Large Language Models (LLMs) have recently been leveraged for asset pricing and stock trading applications, enabling AI agents to generate investment decisions from unstructured financial data. However, most evaluations of LLM timing-based investing strategies are conducted on narrow timeframes and limited stock universes, overstating effectiveness due to survivorship and data-snooping biases. We critically assess their generalizability and robustness by proposing FINSABER, a backtesting framework evaluating timing-based strategies across longer periods and a larger universe of symbols. Systematic backtests over two decades and 100+ symbols reveal that previously reported LLM advantages dete…
-
AlphaForgeBench: Benchmarking End-to-End Trading Strategy Design with Large Language ModelsWen-Tao Zhang, Ming-Xuan Zhao, Jin-Cheng Gao et al. · 2026 · Knowledge Discovery and Data Mining · cited 6 times · Not summarized yet
AlphaForgeBench evaluates end-to-end strategy design and trading-agent stability, rather than point-in-time prediction from masked company news.
Abstract
The rapid advancement of Large Language Models (LLMs) has catalyzed the proliferation of diverse financial benchmarks, progressively evolving from static knowledge evaluation to increasingly sophisticated interactive trading simulations. Nevertheless, existing frameworks that assess real-time trading performance largely overlook a fundamental failure mode: the severe behavioral instability exhibited by LLMs in sequential decision-making under financial uncertainty. Through extensive empirical investigation, we demonstrate that when deployed as direct trading agents, LLMs manifest extreme run-to-run variance, produce inconsistent action sequences even under strictly deterministic decoding con…
-
Reflexivity as Prompt: Does Awareness of Self-Reinforcing Market Dynamics Improve LLMs as Financial Market Forecasters?Eugene W Park · 2026 · Not summarized yet
Reflexivity as Prompt studies whether explaining self-reinforcing market dynamics improves LLM forecasts across historical boom-bust episodes.
Abstract
We study how frontier large language models (LLMs) behave as financial forecasters during boom-bust market cycles when made progressively aware of Soros's theory of reflexivity. Standard AI-assisted forecasting treats the market as an exogenous system. Reflexivity theory holds otherwise: prices shape fundamentals, and every forecaster is a participative agent in the loop it analyzes. We evaluate three frontier models - GPT5, Claude Sonnet 4.6, and Gemini 3 Pro - under four accumulating zero-shot conditions across two historically distinct episodes: the dot-com bubble (1996-2001) and the global financial crisis (2004-2009). The primary metric is directional forecasting accuracy; we also repor…
Prior work it builds on
-
Pervasive underreaction: Evidence from high-frequency dataHao Jiang, Sophia Zhengzi Li, Hao Wang · 2020 · Journal of Financial Economics · cited 42 times · Not summarized yet
Pervasive underreaction tests whether stock prices drift after firm news, while this paper finds news priced by the open and no reliable evidence of underreaction.
Abstract
We propose a novel high-frequency decomposition of daily stock returns into news- and non-news-driven components. With it, we uncover evidence of pervasive stock market underreaction to firm news, in the post-millennium period when stocks appear to be priced with increased efficiency. We show that stock prices tend to drift in the same direction as the initial market response, for several days after the news arrival without reversals. A trading strategy exploiting the return drift generates a monthly return of 2.10% after trading costs. To understand the economic mechanism, we find that the return drift is stronger when investors are distracted. Using analyst earnings forecasts as a proxy, w…
-
Approaching Human-Level Forecasting with Language ModelsDanny Halawi, Fred Zhang, Chen Yueh-Han et al. · 2024 · Machine Learning · cited 165 times · AI summary available
Approaching Human-Level Forecasting uses retrieval-augmented language models to generate forecasts, applying language-model prediction to future events rather than stock news.
Abstract
Forecasting future events is important for policy and decision making. In this work, we study whether language models (LMs) can forecast at the level of competitive human forecasters. Towards this goal, we develop a retrieval-augmented LM system designed to automatically search for relevant information, generate forecasts, and aggregate predictions. To facilitate our study, we collect a large dataset of questions from competitive forecasting platforms. Under a test set published after the knowledge cut-offs of our LMs, we evaluate the end-to-end performance of our system against the aggregates of human forecasts. On average, the system nears the crowd aggregate of competitive forecasters, an…
-
Wisdom of the Silicon Crowd: LLM Ensemble Prediction Capabilities Match Human Crowd AccuracyPhilipp Schoenegger, Indre Tuminauskaite, Peter S. Park et al. · 2024 · Computers and Society · cited 131 times · AI summary available
Wisdom of the Silicon Crowd aggregates LLM predictions to forecast events, rather than testing individual model signals on stock returns.
Abstract
Human forecasting accuracy in practice relies on the 'wisdom of the crowd' effect, in which predictions about future events are significantly improved by aggregating across a crowd of individual forecasters. Past work on the forecasting ability of large language models (LLMs) suggests that frontier LLMs, as individual forecasters, underperform compared to the gold standard of a human crowd forecasting tournament aggregate. In Study 1, we expand this research by using an LLM ensemble approach consisting of a crowd of twelve LLMs. We compare the aggregated LLM predictions on 31 binary questions to that of a crowd of 925 human forecasters from a three-month forecasting tournament. Our main anal…
-
The Memorization Problem: Can We Trust LLMs'Economic Forecasts?Alejandro Lopez-Lira, Yue-Hua Tang, Ming Zhu · 2025 · cited 50 times · Not summarized yet
The Memorization Problem shows that LLMs recall financial data and reconstruct masked entities, motivating this paper’s masking and look-ahead controls.
Abstract
Large language models (LLMs) cannot be trusted for economic forecasts during periods covered by their training data. Counterfactual forecasting ability is non-identified when the model has seen the realized values: any observed output is consistent with both genuine skill and memorization. Any evidence of memorization represents only a lower bound on encoded knowledge. We demonstrate LLMs have memorized economic and financial data, recalling exact values before their knowledge cutoff. Instructions to respect historical boundaries fail to prevent recall-level accuracy, and masking fails as LLMs reconstruct entities and dates from minimal context. Post-cutoff, we observe no recall. Memorizatio…
-
Information, Trading, and Volatility: Evidence from Firm-Specific NewsJ. Boudoukh, Ronen Feldman, Shimon Kogan et al. · 2018 · The Review of financial studies · cited 152 times · Not summarized yet
Information, Trading, and Volatility measures firm-specific news information in stock returns, providing a basis for this paper’s study of news-based signals.
Abstract
What moves stock prices? Prior literature concludes that the revelation of private information through trading, and not public news, is the primary driver. We revisit the question by using textual analysis to identify fundamental information in news. We find that this information accounts for 49.6% of overnight idiosyncratic volatility (vs. 12.4% during trading hours), with a considerable fraction due to days with multiple news types. We use our measure of public information arrival to reinvestigate two important contributions in the literature related to individual R2s of stock returns on aggregate factors. (JEL G14)
Go further
How are these papers found?
We compare the title and abstract of this paper with those of every paper summarized on this site, and with the papers Semantic Scholar recommends, those it cites and those citing it (for a much-cited paper, its most recent citations and, from OpenAlex, the most cited ones). Among the prior work and the follow-ups, the papers that use each other in depth, a method it uses and the much-cited follow-ups come a little higher. The levels (very close, close, related) come from the similarity of the texts, not from a judgment of quality; * marks a paper without an abstract, compared by its title. The sentence under a paper is written by an AI from the two abstracts: check it in the papers. A paper marked not relevant by three readers leaves the list.
Updated on 9 November 2026.
Disclaimer: The AI-based summarization tool and virtual assistant provided on this website may not always provide accurate and complete summaries or responses. We encourage you to carefully review and evaluate the generated content to ensure its quality and relevance to your needs.