AI API cost review path for prompt caching and tool calls

An AI model cost log is much more useful when it separates input tokens, cached input tokens, and output tokens instead of storing one total number.

Input tokens measure how much text, tool schema, image description, or other prompt material the model had to read. Cached input tokens show how much of that prompt reused a provider cache. Output tokens measure what the model generated. These buckets can have different prices, latency effects, and optimization strategies.

If a team stores only total tokens, the wrong conclusion is easy. A long support prompt may look expensive even when most of the static prefix is cached. A short but verbose answer may look cheap until output cost is separated. A model change may appear to improve cost when the real difference is shorter responses.

A useful log row includes date, feature, provider, model, request type, prompt version, input tokens, cached input tokens, output tokens, latency, error status, and whether tools were used. For repeated jobs, add item count and average cost per item. For chat products, add conversation turn number because long conversations behave differently from single-shot tasks.

The log should also preserve a human-readable reason for the request. "Summarize ticket" and "classify ticket" can use similar token counts but different quality checks. Cost without task type is hard to interpret.

The decision rule is simple: optimize the bucket that is actually large. If uncached input dominates, restructure the prompt or shorten context. If output dominates, tighten the requested format. If tool calls dominate latency, inspect the external service. A single total token number hides all three decisions.

AI model cost logs should separate input, cached input, and output tokens

// COMMENTS

ON THIS PAGE