Tool Selection for LLM Agents: Routing Strategies and Implementation › Limitations and Practical Challenges

Tool selection systems face several inherent limitations that impact reliability and scalability. Understanding these constraints is needed for designing reliable production systems and setting appropriate expectations for performance.

Tool selection remains surprisingly sensitive to description phrasing. Minor edits, such as changing "Retrieve current weather" to "Get current weather," can shift selection probabilities enough to alter routing decisions. This brittleness stems from the fact that LLMs lack true understanding of tool capabilities; they rely on pattern matching between query semantics and description semantics, making them vulnerable to superficial variations in wording.

In production systems, this necessitates extensive A/B testing of description variants and continuous monitoring of selection distributions. A sudden shift in the frequency of calculator invocations might indicate that a description change inadvertently altered the decision boundary between mathematical and factual queries. Maintaining stable tool ecosystems requires treating descriptions as necessary code, with version control and regression testing.

The brittleness problem is compounded by the fact that LLMs themselves change over time. A model update that improves performance on one task might subtly shift the token probability distributions used for tool selection, causing previously stable routing decisions to become unreliable. This means that tool descriptions need to be re-validated whenever the underlying model is updated, not just when the tool ecosystem changes. Organizations that run large tool ecosystems often maintain automated regression test suites specifically for routing behavior.

Another dimension of brittleness is language variation. Users phrase requests differently based on their background, expertise level, and communication style. A tool description tuned to match formal, precise queries might fail on casual or colloquial phrasings. Diverse test sets should include informal and abbreviated queries as well as multilingual ones to assess reliability across real-world language variation.

As tool ecosystems grow, selection accuracy degrades due to the curse of dimensionality in semantic space. With two tools, discrimination is easy. With two hundred, the problem becomes significantly harder, suffering from the "long tail" of rare tools that receive insufficient training signal and are often overshadowed by more frequently mentioned tools.

Research suggests that LLM-based routing struggles beyond approximately twenty tools without specialized training. Solutions include hierarchical routing (tools organized into categories, with a two-stage selection process first choosing the category then the specific tool) and dynamic tool retrieval (only presenting relevant tools based on conversation context or user preferences).

The scaling curve reveals an important design implication: if your tool ecosystem will eventually grow to hundreds of tools, plan for hierarchical routing from the beginning. Retrofitting a flat routing architecture with hierarchical organization requires changes throughout the system, including tool descriptions, routing logic, and evaluation pipelines. Starting with a two-level hierarchy (categories, then tools) even when you have only twenty tools makes it much easier to add tools later without disrupting existing routing behavior.

Just as LLMs hallucinate facts, they can hallucinate tools: generating invocations for non-existent capabilities or parameters outside defined schemas. This is particularly problematic when the model invents tools that "sound right" for a query but do not exist in the registry, or hallucinates parameters that seem plausible but violate the schema.

Mitigation strategies include:

• Constrained decoding: Using grammars or logits processors to force valid tool names and parameter structures, as discussed in Constrained Decoding
• Validation layers: Checking tool names against the registry before execution, rejecting any unknown tools immediately
• Fallback prompts: Explicitly listing available tools and asking the model to choose from that closed set, reducing the space of possible hallucinations

Constrained decoding is the most reliable defense against hallucination because it operates at the token level: the model physically cannot generate invalid tool names because those token sequences are assigned zero probability. The trade-off is that constrained decoding requires integrating with the inference infrastructure at a low level, which is not always possible with third-party API providers. For hosted model APIs, validation layers and fallback prompts provide the next best defense.

It is worth distinguishing between two types of tool hallucination. The first type invents entirely non-existent tools ("use the financial_forecast_pro tool" when no such tool exists). This is relatively easy to catch with a registry validation check. The second type invokes real tools with hallucinated parameters (calling weather_lookup with {"location": "the moon"}). This is harder to catch because the parameter value passes schema validation but would fail at execution. Downstream validation, where tool outputs are checked for reasonableness before being used, is needed to catch this type of error.

Every tool selection strategy involves trade-offs between accuracy and speed. LLM-based routing provides the best accuracy but adds hundreds of milliseconds to response time. Embedding-based routing is faster but less accurate. Hybrid approaches add system complexity and potential failure points at each stage.

For high-frequency applications, the cost of inference becomes significant. Routing a query through GPT-4 to select between local tools may cost more than simply attempting multiple tool calls in parallel and filtering results. The optimal strategy depends on the relative costs of latency and compute along with error rates, requiring careful economic modeling of the trade-off space.

In practice, the right routing architecture is often determined by the failure cost of incorrect selections. For a system that can only send one email per request and cannot undo sends, the cost of a wrong tool selection is high, justifying expensive LLM-based routing. For a system that runs read-only database queries, the cost of a wrong selection is low (just a wasted API call), making fast embedding routing the better choice. Align your routing investment with the cost structure of your tool ecosystem.