TL;DR
Butter updated its HTTP proxy cache with automatic template induction so stored responses can generalize via variable placeholders. The system matches incoming queries to templates using deterministic, syntactic checks (no extra LLM calls) and organizes cached interactions as a tree aligned with append-only chat context.
What happened
Butter announced an automatic template-induction feature for its response cache, which sits as an HTTP proxy between clients and LLM inference endpoints. Instead of storing raw messages, the cache now stores templated messages that separate structural content (the template) from dynamic content (bound variables). At request time Butter uses deterministic, syntactic pattern matching—for example regex-style comparisons—to find a matching template, deduce bindings, and hydrate the cached response without invoking additional language models. The cache is modeled as a tree that mirrors append-only, turn-based chat context: each message becomes a node and divergent continuations create branches that represent different workflows. Butter also highlights the problem of "noise" in agent contexts (e.g., timestamps or HTML artifacts) and notes that while some syntactic filters are possible, it currently avoids broad syntactic de-noising and defers semantic noise handling to a later variable-induction step.
Why it matters
- Template induction lets cached responses generalize across similar queries, raising potential cache hit rates without re-running inference.
- Deterministic, syntactic matching avoids extra LLM calls at request time, enabling predictable, LLM-free hotpaths.
- Organizing cache as a tree aligned to append-only context supports multi-turn interactions and workflow branching.
- Handling noise thoughtfully reduces false negatives (missed hits) and avoids incorrectly discarding context that matters to workflows.
Key facts
- Butter is positioned as an HTTP proxy between clients and LLM inference endpoints.
- The new feature performs automatic template induction to split messages into structural content (template) and dynamic content (variables).
- Template matching at request time is syntactic and deterministic, avoiding additional LLM calls.
- Butter models its cache as a tree where each node is a message and branches represent distinct continuations from the same context.
- The system assumes append-only context management, matching templates message-by-message from the start of the context.
- When a template matches, Butter deduces variable bindings and hydrates the cached response template to produce the final reply.
- Butter identifies noisy artifacts (e.g., HTML, timestamps) as a threat to cache matching and discusses de-noising approaches.
- Butter currently chooses not to apply broad syntactic de-noising and defers semantic de-noising to a later variable-induction phase.
What to watch next
- Whether Butter publishes performance benchmarks showing changes in cache hit rate and latency (not confirmed in the source).
- Details on how Variable Induction will operate to address context-dependent (semantic) noise (not confirmed in the source).
- Any announced integrations, supported inference endpoints, or rollout timeline for the feature (not confirmed in the source).
Quick glossary
- Template: A message structure with placeholders for variable pieces of content, used to generalize stored responses.
- Cache hit: When an incoming query matches an existing cache entry such that a stored response can be reused.
- Append-only context: A context-management style where prior messages are passed along in chronological order and appended to each new request.
- De-noising: The process of removing extraneous or irrelevant tokens from input so that matching and logic focus on meaningful content.
- Binding (variable binding): A concrete value substituted into a template placeholder to hydrate a full message or response.
Reader FAQ
Does Butter call additional LLMs during request-time matching?
No. The source says matching is deterministic and syntactic and avoids invoking extra language models at request time.
Is Butter an HTTP proxy?
Yes. The source describes Butter as a cache that sits as an HTTP proxy between clients and LLM inference endpoints.
Does Butter automatically strip timestamps and other noisy tokens syntactically?
No. The source states Butter currently chooses not to apply broad syntactic de-noising and defers semantic de-noising to Variable Induction.
Is the product availability, pricing, or benchmark data provided?
not confirmed in the source
On Automatic Template Induction for Response Caching Updated January 7, 2026 • 14 min read Raymond Tana TAGS #templates #caching #agents #proxy #llm As of last week, Butter’s proxy now…
Sources
- Show HN: An LLM response cache that's aware of dynamic data
- Template-Aware Caching
- Show HN: Butter – A Behavior Cache for LLMs
- Butter – A Cache For Your LLM
Related posts
- X’s Grok deepfake tool sparks global regulatory ire over AI imagery
- OpenAI launches ChatGPT Health as 230 million weekly users ask about health
- Claude Code Emergent Behavior — What Happens When Skills Combine