Tool Selection for LLM Agents: Routing Strategies and Implementation › Tool Routing Strategies

Routing determines how a system maps incoming requests to tool invocations. The strategy you choose depends on latency requirements, accuracy needs, and the complexity of your tool ecosystem. Simple systems might delegate all decisions to the LLM, while high-throughput applications require sophisticated multi-stage pipelines. Understanding these trade-offs enables architects to match routing mechanisms to operational constraints.

The routing problem can be framed as a classification task: given an input query $q$ and a set of available tools $\mathcal{T} = \{t_1, t_2, \ldots, t_N\}$, we want to find the tool $t^*$ that maximizes some measure of appropriateness:

$$t^* = \underset{t \in \mathcal{T} \cup \{\text{none}\}}{\arg\max} \ P(t \mid q, \mathcal{T})$$

where the "none" option allows the system to decline tool use when no tool is appropriate, preventing hallucinated invocations on unrelated queries. Different routing strategies represent different approaches to estimating this probability, trading off computational cost against estimation quality.

The most straightforward approach uses the LLM itself as the router. The model receives the user query along with all available tool descriptions, and generates a structured output showing which tool (if any) to invoke. This is the standard function calling mechanism we covered previously, using the model's inherent reasoning capabilities to perform semantic matching.

The advantage of LLM-based routing is flexibility. The model can handle fine-grained distinctions between tools, resolve ambiguities through implicit world knowledge, and chain multiple selections in a single reasoning trace. It can recognize that "find my keys" requires a different tool than "find information about keys" based on subtle linguistic cues. However, this flexibility comes at a cost: it incurs the full latency and computational expense of LLM inference for every routing decision, which makes it unsuitable for high-frequency applications or environments with strict latency budgets.

Another important benefit of LLM-based routing is its ability to use conversational context. When routing a query in isolation, the model might ambiguously select between a database lookup and a web search. But given the preceding conversation, the model can infer that the user is working within a company's internal system, ruling out public web search as the appropriate choice. This context-sensitivity is difficult to replicate in simpler routing approaches that treat each query independently.

This JSON representation shows how tool schemas are structured for the LLM context, with each tool's description and parameters clearly delineated to guide the model's selection process.

In this setup, the LLM evaluates the semantic similarity between the user intent and each tool description, effectively performing classification over the tool set. The model's attention mechanisms compute relevance scores implicitly during the forward pass, weighing the semantic overlap between query tokens and description tokens to determine the most appropriate tool.

The practical limitation of pure LLM routing becomes apparent at scale. With 50 tools, each with a 200-word description, you are pushing roughly 10,000 tokens of tool context into every request. At GPT-4 pricing, that adds real cost. Beyond cost, there is a cognitive load problem: models perform worse at discriminating between the correct tool and distractors when the context window contains many irrelevant options. This is a manifestation of the attention dilution problem, where relevant signals get buried in noise.

For applications requiring lower latency or operating at scale, we can decouple routing from the LLM entirely. As mentioned earlier, this approach uses dense retrieval to pre-filter tools. Building on our understanding of Dense Retrieval from Part XLIV, we encode tool descriptions and user queries into the same vector space using bi-encoder architectures.

The routing decision becomes a nearest neighbor search in this embedding space:

$$\text{tool}_{\text{selected}} = \underset{t \in \text{Tools}}{\arg\max} \ \text{sim}(q, \text{desc}(t))$$

where:

• $\text{tool}_{\text{selected}}$: the tool chosen for execution
• $t$: a candidate tool from the available set $\text{Tools}$
• $q$: the embedding vector representing the user's query
• $\text{desc}(t)$: the embedding vector of tool $t$'s description
• $\text{sim}(\cdot, \cdot)$: a similarity function (typically cosine similarity or dot product) measuring alignment between two embeddings

This approach works because embeddings capture semantic meaning through the distributional hypothesis: words and phrases with similar meanings appear in similar contexts during training, causing them to map to nearby points in the vector space. Thus, geometric distance approximates semantic relevance, letting us to find tools whose descriptions match the query intent without expensive neural inference at decision time.

The key question is which embedding model to use. General-purpose models like all-MiniLM-L6-v2 or text-embedding-3-small work well for everyday language but can struggle with domain-specific terminology or unusual tool names. A query about "reconciling journal entries" will embed close to "accounting" and "finance" concepts in general-purpose space, but only a domain-aware model will correctly distance it from "journalism" and "diary writing." For specialized applications, fine-tuning a bi-encoder on domain-specific query-tool pairs consistently outperforms zero-shot retrieval.

Similarity-based routing is computationally efficient but lacks the fine-grained reasoning of LLM-based approaches. It cannot easily handle cases where tool selection depends on subtle constraints, requires multi-hop reasoning, or depends on conversational context not captured in the current query. It also struggles with out-of-distribution queries that are semantically distant from any tool description in the embedding space.

The threshold parameter is one of the most important design decisions in embedding-based routing. Set it too high, and many valid queries get dropped; set it too low, and the system invokes tools for irrelevant queries. Rather than using a single global threshold, calibrated per-tool thresholds tend to work better: a highly specific tool like convert_currency can tolerate a lower threshold because false positives are less costly, while a destructive tool like delete_records should require a higher threshold to prevent accidental invocations.

Production systems often combine both approaches in a cascade that uses the strengths of each while mitigating their weaknesses:

1. Retrieval stage: Use fast embedding similarity to filter from $N$ tools to a candidate set of $k \ll N$ tools. This stage acts as a coarse filter, eliminating obviously irrelevant options at minimal computational cost.
2. Ranking stage: Use the LLM to select from the $k$ candidates, or determine that none are appropriate. This stage provides fine-grained discrimination, handling nuances and dependencies that embeddings cannot capture.

This hybrid approach balances efficiency with accuracy. The embedding step acts as a coarse filter, reducing the cognitive load on the LLM by eliminating distractors, while the LLM provides the final fine-grained discrimination necessary for high-stakes decisions. The approach is analogous to human decision-making: we first retrieve relevant options from memory, then deliberate carefully among the candidates.

The choice of $k$ governs the basic trade-off. A small $k$ (e.g., 3) minimizes LLM context length and latency but risks missing the correct tool if the retriever makes an error. A large $k$ (e.g., 10) provides more safety margin but adds more noise for the LLM to reason through. Empirically, values between 3 and 7 tend to work well for ecosystems with up to a few hundred tools, but the right value depends on how semantically similar your tools are to each other. Ecosystems with many closely related tools benefit from larger $k$; ecosystems with distinct, non-overlapping tools can use smaller $k$.

The diagram illustrates an important pattern: the marginal accuracy gain from moving from Hybrid (k=5) to LLM-Only is small (about 1%), but the latency cost is 2.5 times higher. This diminishing return is typical: the last few percentage points of accuracy in routing systems require disproportionately more compute.

For systems with hundreds or thousands of tools, even presenting $k$ candidates to the LLM becomes expensive or intractable. In these scenarios, we can train a dedicated routing model, typically a smaller classifier or a mixture-of-experts system as discussed in Part XXXIII: Mixture of Experts.

The router is trained on historical query-tool pairs, learning to map directly from query text to tool indices. This can be formulated as a multi-class classification problem or, more scalably, as a retrieval task using contrastive learning where similar query-tool pairs are pulled together in embedding space while dissimilar pairs are pushed apart.

The routing function becomes:

$$P(t \mid q) = \text{softmax}(W_r \cdot \text{Encoder}(q) + b_r)$$

where:

• $P(t \mid q)$: the probability of selecting tool $t$ given query $q$
• $W_r$: the learned weight matrix of the router layer
• $b_r$: the learned bias vector of the router layer
• $\text{Encoder}(\cdot)$: a neural network (typically a transformer or sentence encoder) that converts text into dense vector representations
• $\text{softmax}(\cdot)$: a function that converts raw scores into a probability distribution over tools

The softmax ensures that all selection probabilities sum to 1.0, letting the model to express uncertainty when multiple tools might be appropriate. The learned parameters $W_r$ and $b_r$ adapt the generic encoder outputs to the specific tool selection task through training data. This learned projection can capture domain-specific patterns that generic embeddings miss, such as industry jargon or company-specific terminology.

The training objective for a learned router is typically cross-entropy loss over tool selections:

$$\mathcal{L}_{\text{router}} = -\sum_{(q, t^*) \in \mathcal{D}} \log P(t^* \mid q)$$

where $\mathcal{D}$ is a dataset of query-tool pairs and $t^*$ is the correct tool for each query. This objective pushes the model to assign high probability to correct tools and low probability to incorrect ones, directly optimizing the decision boundary in the embedding space. With enough training data, the learned router can significantly outperform even the embedding-based retrieval approach, because it is optimized end-to-end for the specific tool set rather than relying on a general-purpose encoder.

The trade-off with learned routers is maintenance cost. Adding a new tool requires retraining the classifier, unlike embedding-based approaches where you simply add a new entry to the vector index. This makes learned routers more appropriate for stable, mature tool ecosystems than for rapidly evolving ones. A common hybrid is to use a learned router for the core set of frequently used tools and fall back to embedding retrieval for the long tail of specialized tools that are added less frequently.

All routing approaches should support abstention: the ability to decline routing when confidence is insufficient. Without this capability, the system will always select some tool, even for queries that have nothing to do with available capabilities. This creates a frustrating user experience where the system attempts irrelevant tool calls rather than simply acknowledging it cannot help.

For embedding-based routing, abstention is implemented via the similarity threshold we discussed earlier. For LLM-based routing, it requires explicitly including a "none" option in the tool list and instructing the model to choose it when appropriate:

For learned routers, abstention can be modeled as a special class or implemented by thresholding on the maximum softmax probability. The key insight is that a well-calibrated router should be able to express uncertainty: when the maximum $P(t \mid q)$ across all tools is low, the router should abstain rather than commit to a low-confidence selection.