Test-Time Scaling and Reinforcement-Learned Reasoning in Large Language Models: Verifiers, Budget Forcing, and Emergent Deliberation

Abstract

The rapid development of reasoning-oriented large language models during 2024 and 2025 shifted attention from scaling only pretraining parameters toward allocating additional computation during inference. Test-time scaling allows a model to generate, evaluate, revise, or search among multiple reasoning trajectories before committing to a final answer. This review examines the foundations and principal developments in inference-time reasoning through 2025. Chain-of-thought prompting established that intermediate natural-language reasoning could substantially improve performance on complex tasks, while self-consistency demonstrated the value of sampling multiple reasoning trajectories and aggregating their answers. Self-Taught Reasoner and Quiet-STaR explored mechanisms through which models could learn to generate useful internal rationales. Process-supervision and verifier research further demonstrated the importance of evaluating intermediate reasoning rather than relying solely on final-answer correctness. Tree-of-Thoughts and iterative self-refinement extended reasoning toward explicit search and revision. Research on compute-optimal test-time scaling subsequently formalized the relationship among task difficulty, sampling budget, verifier quality, and inference computation. DeepSeek-R1 demonstrated that reinforcement learning could incentivize sophisticated reasoning behavior, including reflection and verification, while s1 showed that carefully selected training examples combined with budget forcing could produce strong reasoning and controllable test-time computation. This review examines outcome and process reward models, reinforcement learning, search, self-correction, budget forcing, inference latency, reasoning-token allocation, and distillation. Remaining challenges include excessive reasoning, unreliable self-verification, reward hacking, evaluation contamination, hidden computational cost, safety degradation, and the difficulty of establishing whether extended reasoning traces correspond to faithful internal computation.