Prompt Caching - MyTokenGate

1. Use Cases

When multiple requests share the same long prefix (a system prompt, a long document, tool definitions, multi-turn history), prompt caching reuses the already-processed prefix to cut cost and reduce time-to-first-token. Cached input tokens are billed at a lower cache-read rate.

Typical scenarios:

• Repeated Q&A over the same long document
• A fixed long system prompt plus many tool definitions
• Multi-turn conversations whose history keeps growing but whose prefix stays stable

2. Two Ways to Cache

The MyTokenGate gateway supports both automatic and explicit caching, depending on which endpoint you use.

2.1 Automatic Caching (OpenAI-Compatible Endpoint)

On /v1/chat/completions you need no extra parameters. The upstream automatically detects a repeated long prefix and serves it from cache; the hit is reported in the usage fields:

"usage": {
  "prompt_tokens": 4200,
  "prompt_tokens_details": {
    "cached_tokens": 4096
  }
}

cached_tokens is the number of input tokens served from cache this request, billed at the cache-read rate. Whether and how much is cached is decided by the upstream, which generally requires the prefix to reach a minimum length and match exactly.

2.2 Explicit Caching (Anthropic-Compatible Endpoint)

On /v1/messages you can add cache_control to system, messages content blocks, or tool definitions to mark a cache breakpoint:

{
  "system": [
    {
      "type": "text",
      "text": "You are an assistant with access to these documents: ...",
      "cache_control": { "type": "ephemeral" }
    }
  ]
}

Hits are reported in the usage fields:

The cache lives 5 minutes by default and each hit refreshes the timer; some models also support a 1-hour lifetime (at a higher write rate).

3. Notes

• Caching works by prefix: only content that matches exactly from the start can hit, and any change invalidates everything after it.
• Put stable content (system prompt, documents, tool definitions) first and the changing user input last for the best hit rate.
• Automatic caching on the OpenAI-compatible endpoint does not accept cache_control; that field is dropped during protocol conversion.
• Exact prices follow the cache-read / cache-write rates shown in the Model List .

For the full usage-field reference, see Chat Completions and Messages.