Cost Management › Cost Monitoring
Language AI HandbookAll chapters Cost Management › Cost MonitoringOne section · Sections in this chapter5 / 8 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Cost monitoring creates continuous visibility into spending, enabling early detection of anomalies, enforcement of budgets, and tracking of optimization progress. Without monitoring, cost problems are often discovered only at month-end when the bill arrives.
Key Cost Metrics
An effective cost monitoring system tracks metrics at multiple granularities:
Token-level metrics (collected per request):
- Prompt token count and completion token count
- Request cost in dollars (computed from token counts and current rates)
- Cost efficiency ratio: output quality score divided by cost (requires quality evaluation)
Request-level metrics (aggregated over time windows):
- Requests per minute and tokens per minute, broken down by model tier
- Cost per request and cost per token, with trend tracking
- Cache hit rates for prefix caching and semantic caching
- Token rejection rates from per-user budget enforcement
Infrastructure metrics (from the hosting platform):
- GPU utilization (compute and memory), both instantaneous and averaged
- Throughput (tokens/sec) vs. theoretical maximum
- Cost per GPU-hour and effective cost per token
- Reserved instance utilization and on-demand overflow costs
Business metrics (connecting cost to value):
- Cost per active user per day
- Cost per successfully completed task (e.g., per support ticket resolved, per code review)
- Cost as a percentage of revenue or allocated budget
# Cost monitoring pipeline with anomaly detection
from collections import deque
import numpy as np
class CostMonitor:
"""
Rolling cost monitor with anomaly detection using
exponential weighted moving average (EWMA).
"""
def __init__(self, window_minutes: int = 60, alert_multiplier: float = 2.5):
self.window_minutes = window_minutes
self.alert_multiplier = alert_multiplier
# Store (timestamp, cost) tuples
self.cost_history = deque()
self.ewma_cost = None
self.ewma_alpha = 0.1 # Smoothing factor
def record(self, timestamp: float, cost: float):
"""Record a cost observation and update EWMA."""
self.cost_history.append((timestamp, cost))
# Remove observations outside the window
cutoff = timestamp - self.window_minutes * 60
while self.cost_history and self.cost_history[0][0] < cutoff:
self.cost_history.popleft()
# Update EWMA
if self.ewma_cost is None:
self.ewma_cost = cost
else:
self.ewma_cost = (
self.ewma_alpha * cost + (1 - self.ewma_alpha) * self.ewma_cost
)
def is_anomaly(self, cost: float) -> bool:
"""Compare a new observation with the EWMA of prior observations."""
if self.ewma_cost is None or self.ewma_cost == 0:
return False
return cost > self.alert_multiplier * self.ewma_cost
def window_total(self) -> float:
"""Total cost in the current window."""
return sum(cost for _, cost in self.cost_history)
def window_rate(self) -> float:
"""Estimated cost rate in USD per hour."""
if not self.cost_history:
return 0.0
span_seconds = self.cost_history[-1][0] - self.cost_history[0][0]
if span_seconds < 1:
return 0.0
return self.window_total() / span_seconds * 3600
# Simulate monitoring with an injected cost spike
monitor = CostMonitor(window_minutes=60, alert_multiplier=2.5)
alerts = []
timestamps = []
costs = []
ewma_values = []
alert_thresholds = []
rng = np.random.default_rng(seed=42)
base_time = time.time() - 7200 # Start 2 hours ago
for minute in range(120):
t = base_time + minute * 60
# Normal traffic with some noise
normal_cost = 0.05 + rng.normal(0, 0.01)
# Inject a cost spike at minute 75 (runaway batch job scenario)
if 75 <= minute <= 85:
request_cost = normal_cost * 8
else:
request_cost = max(0.001, normal_cost)
threshold_before_update = (
monitor.alert_multiplier * monitor.ewma_cost
if monitor.ewma_cost is not None
else np.nan
)
is_alert = monitor.is_anomaly(request_cost)
if is_alert:
alerts.append(minute)
monitor.record(t, request_cost)
timestamps.append(minute)
costs.append(request_cost)
ewma_values.append(monitor.ewma_cost)
alert_thresholds.append(threshold_before_update)Cost Monitoring Summary (120-minute window): Current window total: $6.73 Estimated rate: $6.73/hr Current EWMA: $0.0555/min Anomaly alerts fired at minutes: [75, 76, 77, 78] Total alerts: 4
Budget Enforcement
Cost monitoring without enforcement is incomplete. Budget enforcement closes the loop by taking action when spending approaches or exceeds thresholds.
A layered enforcement approach works well in practice:
Soft limits trigger alerts and notifications without blocking requests. When spending reaches 70% of a daily budget, an automated notification goes to the responsible team. At 90%, escalation occurs to engineering leadership. This approach preserves service availability while creating urgency for investigation.
Hard limits block new requests when a budget is exhausted. This is appropriate for non-critical, optional features but must be applied carefully to avoid breaking user-facing functionality. Hard limits are more commonly applied to per-user quotas than to organizational budgets.
Rate limiting caps the rate of spending rather than total spend. Rather than blocking all requests after a budget is hit, rate limiting ensures the system spends at most a certain amount per hour. This is more graceful for users, as it slows down expensive operations rather than blocking them entirely.
Graceful degradation automatically routes requests to cheaper model tiers when spending is high. A user who would normally receive a response from a 70B model might receive one from a 7B model when the budget is constrained. This requires that alternative tiers are available and that quality degradation is acceptable for the use case.
# Budget enforcement with graceful degradation
class BudgetEnforcer:
"""
Enforces spending budgets with graceful degradation.
Routes to cheaper tiers as budget utilization increases.
"""
def __init__(self, daily_budget_usd: float):
self.daily_budget_usd = daily_budget_usd
self.spent_today = 0.0
# Tier selection thresholds
self.tier_policy = [
# (max_utilization, allowed_tier, reason)
(0.70, "large", "Normal operation"),
(0.85, "medium", "Approaching budget, degrading to medium"),
(0.95, "small", "Near budget limit, degrading to small"),
(1.00, None, "Budget exhausted, requests blocked"),
]
def get_allowed_tier(self, requested_tier: str) -> tuple:
"""
Return the tier to use given current budget utilization.
Returns (actual_tier, reason) where actual_tier may be None (block).
"""
utilization = self.spent_today / self.daily_budget_usd
for max_util, allowed_tier, reason in self.tier_policy:
if utilization <= max_util:
if allowed_tier is None:
return None, reason
# Return cheapest of requested and allowed
tier_order = ["small", "medium", "large"]
effective_tier = tier_order[
min(
tier_order.index(requested_tier),
tier_order.index(allowed_tier),
)
]
return effective_tier, reason
return None, "Budget exhausted"
def record_spend(self, cost: float):
self.spent_today += cost
# Simulate requests over a day as budget fills
enforcer = BudgetEnforcer(daily_budget_usd=50.0)
tier_decisions = {"small": 0, "medium": 0, "large": 0, "blocked": 0}
rng2 = np.random.default_rng(seed=42)
for _ in range(2000):
requested = rng2.choice(["small", "medium", "large"], p=[0.5, 0.3, 0.2])
actual_tier, reason = enforcer.get_allowed_tier(requested)
if actual_tier is None:
tier_decisions["blocked"] += 1
else:
tier_decisions[actual_tier] += 1
# Simulate actual cost for this request
rates = TIER_COSTS[actual_tier]
prompt_t = int(rng2.lognormal(6, 0.8))
completion_t = int(rng2.lognormal(5, 0.7))
cost = (prompt_t / 1000) * rates["input"] + (
completion_t / 1000
) * rates["output"]
enforcer.record_spend(cost)Budget Enforcement Simulation (2,000 requests, $50 daily budget):
Budget spent: $1.45 (2.9% utilized)
Request routing decisions:
small 1002 requests (50.1%)
medium 605 requests (30.2%)
large 393 requests (19.7%)
blocked 0 requests (0.0%)Cost Dashboards and Reporting
Effective cost dashboards present costs at the right granularity for each audience. Engineers need per-request and per-feature breakdowns to debug cost regressions. Finance teams need daily and monthly totals with trend projections. Product managers need cost-per-outcome metrics that connect spending to business value.
A useful dashboard structure organizes metrics into three layers:
Real-time layer: Current spend rate (USD/hour), active request count, GPU utilization, cache hit rates. Refreshes every minute. Surfaces anomalies and operational issues immediately.
Daily summary layer: Total spend, cost breakdown by feature and team, per-user percentiles (median, P95, P99), cost per token by model tier, comparison to prior day and prior week. Surfaces trends and allocation changes.
Monthly planning layer: Monthly spend totals, forecast vs. actuals, per-feature cost trends, optimization opportunity analysis, reserved instance utilization and coverage recommendations. Supports budget planning and infrastructure decisions.
# Simulate a multi-day cost time series for dashboard illustration
days = 30
daily_costs = []
rng3 = np.random.default_rng(seed=42)
for d in range(days):
# Organic growth trend of ~2% per day
growth_factor = 1.02**d
# Weekend dip (days 5-6, 12-13, etc.)
day_of_week = d % 7
weekday_factor = 0.65 if day_of_week >= 5 else 1.0
# Random variation
noise = rng3.normal(1.0, 0.05)
base_daily = 180.0 # Base cost in USD per day
daily_costs.append(base_daily * growth_factor * weekday_factor * noise)
# Compute rolling 7-day average
rolling_7d = []
for i in range(days):
start = max(0, i - 6)
rolling_7d.append(np.mean(daily_costs[start : i + 1]))
# Compute projected monthly total using last 7-day average rate
last_week_avg = np.mean(daily_costs[-7:])
monthly_projection = last_week_avg * 30
total_mtd = sum(daily_costs)30-Day Cost Summary: Total (30 days): $6,612.60 Average daily: $220.42 Peak daily: $326.54 (day 30) Minimum daily: $120.77 (day 6) Last 7-day avg: $272.26/day Monthly projection (at current rate): $8,167.81
Cost Regression Detection
Cost regressions occur when code changes or configuration updates inadvertently increase cost per request. Without automated detection, they often go unnoticed until they accumulate into significant spending increases.
A cost regression can take several forms: a new prompt template that adds 200 tokens to every request, a code path that requests a larger model than necessary, a bug that disables prefix caching, or a new feature that generates much longer responses than expected.
Detecting regressions requires tracking per-feature cost metrics over time and alerting when they deviate significantly from baseline. The key metrics to track are average tokens per request (both input and output) and cost per request, broken down by feature. A step change in these metrics after a deployment is a strong signal of a cost regression.
