Reasoning Strategies › Limitations and Practical Implications

These reasoning strategies represent real progress, but they carry important limitations that practitioners must understand before deploying them.

The computational cost of advanced reasoning is substantial and often underappreciated. Tree of Thought and self-consistency require many more language model calls than standard prompting. At inference time, this translates to higher latency and cost. For applications where users wait for responses, generating 40 paths for self-consistency or exploring a ToT tree with depth 4 may be too slow. A user waiting 80 seconds for a self-consistent answer to a math question will not wait; they need a fast response that is occasionally wrong rather than a slow one that is usually right. The strategies that work best in research benchmarks are often impractical for real-time products without significant optimization.

Several approaches address the cost problem. Speculative consistency uses a small model to generate reasoning paths cheaply and a large model only for the final evaluation. Adaptive sampling reduces the number of samples dynamically: stop sampling early if a majority winner has emerged clearly, and sample more only when the vote is close. These heuristics can recover much of self-consistency's accuracy gain at a fraction of the compute cost.

Prompt sensitivity is a deeper challenge. The effectiveness of all these strategies depends heavily on the quality of the prompts: the few-shot decomposition examples for least-to-most, the evaluation prompts for tree of thought, the question format for self-consistency. Small changes to prompt wording can cause large swings in accuracy. This means the strategies require task-specific engineering effort and do not transfer reliably across domains without tuning.

This prompt sensitivity creates a paradox for researchers claiming benchmark improvements. A method that achieves 74% on Game of 24 with carefully engineered prompts may achieve only 50% with prompts from a different researcher who made slightly different choices about decomposition granularity or evaluation categories. The reported numbers reflect the method and the prompt engineering around it. Practitioners replicating these results should expect to invest comparable effort in prompt design.

Evaluating answer consistency is non-trivial for open-ended tasks. Self-consistency assumes answers can be normalized and compared, but for open-ended tasks, two correct answers might be phrased differently and count as disagreeing votes, while two wrong answers might be phrased similarly and incorrectly dominate. The aggregation step that looks trivial for math problems becomes ill-defined for free-form generation.

The evaluation quality problem for Tree of Thought is perhaps the most fundamental limitation. A ToT system is only as good as its value function. Designing an evaluation prompt that reliably distinguishes promising from dead-end states requires deep task understanding and careful prompt engineering. For tasks without clear intermediate checkpoints, this is nearly impossible, which is why ToT has been demonstrated primarily on puzzle-like tasks with objectively correct answers.

Despite these challenges, the practical impact of these strategies is clear. They have pushed language model reasoning to levels that were unachievable with direct prompting, closing much of the gap between model performance and human performance on challenging benchmarks. They have also informed how reasoning is built into newer models: techniques like chain of thought and decomposition are now trained into models directly, not just elicited at inference time. The next generation of reasoning-capable models trains on chain of thought traces, decomposed solutions, and program-of-thought examples, internalizing these strategies so that they emerge without elaborate prompting. We will explore this reasoning-focused training in upcoming chapters.