Managing and Reducing Costs › Glossary

API Call: A request made to a language model service. Each call typically incurs a cost based on the number of tokens processed.

Batching: Combining multiple similar requests into a single API call to reduce overhead and costs. More efficient than processing each request individually.

Budget Limit: A maximum spending threshold set to prevent unexpected or runaway costs. Can be daily, monthly, or per-user.

Cache Hit: When a requested response is found in the cache and can be served instantly without making an API call. Saves both time and money.

Cache Miss: When a requested response is not in the cache, requiring a new API call to generate it.

Context Window: The maximum amount of text (measured in tokens) that a model can process in a single request, including both input and output.

Input Tokens: The tokens in your prompt, including system messages, conversation history, and the user's query. Generally cheaper than output tokens.

Output Tokens: The tokens generated by the model in its response. Typically cost more than input tokens because generation requires more computation.

Token: The basic unit of text that language models process, roughly equivalent to a word or word piece. Both costs and context limits are measured in tokens.