The True Cost of LLM API Tokens: Input, Output, and Caching

Large language model (LLM) APIs are typically priced based on token usage, but the details vary significantly between providers. Understanding the distinction between input and output tokens, how tokens are counted, and the options for caching response outputs is central to managing LLM costs effectively.

Input vs Output Token Pricing across Providers

OpenAI’s GPT-4 API, as of the March 2024 pricing update, charges $0.03 per 1,000 input tokens and $0.06 per 1,000 output tokens for the 8k context window model (GPT-4-8k). Anthropic's Claude 2 API followed the same input/output split: Claude 2.0 was published at $8 per million input tokens and $24 per million output tokens, and Claude 2.1 at $11.02 input and $32.68 output (per Anthropic's 2023 rate card). Google's PaLM 2 API charges roughly $0.018 per 1,000 input tokens and $0.02 per 1,000 output tokens for text generation, per March 2024 data from Google Cloud’s AI Pricing page. Open-source hosting platforms using models such as Llama 2 or Falcon generally pass through compute costs, pricing either by duration or per token processed, but without standardized input/output separation.

The difference in pricing between input and output tokens can be as wide as 2x or more, with output tokens usually costing more. This reflects the higher compute cost in generating output rather than simply processing input text.

Token Counting Methods and Their Impact on Billing

Tokenization methods vary and affect how many tokens your requests consume. OpenAI uses a Byte Pair Encoding (BPE) tokenizer with custom vocabularies that split words into subword units. Longer or more complex input text generates more tokens, increasing the cost. Similarly, Anthropic uses a tokenization scheme aligned with GPT-2 style BPE, but some anecdotal analyses report slight differences in token counts for equivalent text.

Google’s PaLM 2 also employs SentencePiece tokenization, which differs subtly from BPE, potentially causing discrepancies when porting token counts between providers. Such variation complicates direct cost comparisons across platforms for identical text prompts.

Practitioners should use provider-supplied token counters or standard open-source libraries (e.g., tiktoken for OpenAI models) for accurate estimation. Misestimating tokens leads to budgeting errors and unexpected charges.

Caching Outputs to Control Token Cost

Caching response outputs is a direct method to reduce token consumption and thus cost. When an identical prompt or input can be served from cache, the application avoids sending repeated input tokens to the API and incurring output token charges.

OpenAI’s pricing model treats each API call independently, so caching responses on the client side or via a middleware layer prevents redundant API calls and cuts costs proportionally to cache hit rates. For stateful conversation use cases, session caching reduces the need for resending full conversation history, which can be costly given the input token prices on GPT-4.

Anthropic and Google APIs also benefit from caching, but their distinct pricing models require tailored measurement. For example, Anthropic’s tiered and sometimes flat pricing means high caching efficiency might have marginal cost reduction once minimum commitment levels are met.

Open-source LLM hosting can implement caching at the model-serving layer, with cost savings tied to compute resource consumption rather than direct token invoicing. However, since these platforms charge by GPU runtime or API call count, caching can reduce both token count and total compute.

Key Takeaways for Enterprise AI Buyers

Enterprises should parse billing models closely before committing to an LLM provider, focusing on how input and output tokens are counted and priced. OpenAI’s bifurcated input/output pricing can make verbose prompts significantly more expensive, particularly for GPT-4-32k where rates differ further.

Token counting techniques across providers are not interchangeable; enterprises aiming for cross-platform deployments should standardize token analysis using provider-specific tools to avoid budget overruns.

Caching presents a universal lever for cost control but requires investment in system design—caching granularity, cache invalidation policy, and conversation state management all influence realized savings.