Speeding Up the Agent › Glossary

Caching: Storing the results of expensive operations so they can be reused without recomputation. For agents, this typically means saving model responses for repeated queries.

Latency: The time delay between when a user makes a request and when they receive a response. Lower latency means a faster, more responsive agent.

Max Tokens: A parameter that limits how many tokens (words or word pieces) a language model can generate in a single response. Lower values produce shorter, faster responses.

Parallel Execution: Running multiple operations simultaneously rather than one after another. This can significantly reduce total execution time when operations don't depend on each other.

Streaming: Sending response data to the user incrementally as it's generated, rather than waiting for the complete response. This improves perceived speed even if total generation time is unchanged.

Token: The basic unit of text that language models process. A token is roughly equivalent to a word or word piece. Both input and output are measured in tokens.