Token Budgets, Prompt Caching, and the Three Ways to Call an LLM from TypeScript
Frontend developers building agent UIs need to understand token economics and streaming mechanics at the wire level, not just the chat widget. Misjudging context-window pressure or skipping prompt caching can silently multiply API costs by 5–10× as conversations grow, while choosing invoke over stream locks the UI for seconds at a time.
LLM calls are billed by tokens, not characters, and output tokens cost roughly five times more than input. A single agent turn can burn through a context window fast—system prompts, conversation history, tool results, and reasoning all compete for space. When usage hits 75–90% of the window, context compression or summarization becomes mandatory to prevent the model from forgetting early instructions or erroring out. Anthropic's prompt caching cuts costs by 90% for stable prefixes longer than 2,000 tokens, but the cache expires after five minutes and only matches on exact prefix alignment.
LangChain's ChatPromptTemplate turns raw message lists into parameterized, composable templates. A well-structured system prompt covers five elements: role, capability boundaries, output format, style, and counter-examples. Few-shot examples embedded in the template steer the model toward specific JSON structures or tones without post-processing. The MessagesPlaceholder pattern lets you inject dynamic conversation history or RAG results into a fixed template skeleton.
Three calling modes serve different needs. Invoke blocks until the full response arrives—fine for batch extraction, terrible for user-facing UIs. Stream delivers tokens incrementally, enabling typewriter-style rendering and mid-generation cancellation. Batch reuses HTTP connections and enforces concurrency limits, making it 5–10× faster than looping invoke for bulk translation or classification. Production code wraps all three with timeout controls, AbortController signals, and exponential-backoff retry logic that only retries on transient failures like rate limits or 5xx errors.
Token budgeting is a frontend concern now. As agent loops grow longer, the UI layer must track context pressure and trigger summarization, or the model silently degrades mid-conversation.
Prompt caching's 5-minute expiry and prefix-match requirement make it a session-level optimization, not a global one—it rewards architectures that keep system prompts and tool definitions identical across turns.
The 5× output-token premium flips conventional API design instincts: verbose model responses are the real cost driver, not large input contexts, which pushes agent developers toward terse, machine-parseable output formats.
LangChain's batch method is underused. Most tutorials show sequential invoke calls, but any bulk classification or translation task benefits from connection reuse and built-in concurrency gating.