A fully offline interactive-fiction engine that runs a local language
model, coaxes story, game state, and player choices out of a single generation per turn,
and stays coherent across long sessions with a self-maintaining memory summary, with no cloud
dependency and no data leaving the machine.
Internal R&D bench build, not a shipped product. It is documented as an
AI-integration showcase. The focus is the runtime engineering; the scenario content is
interchangeable data the engine never has baked in.
Interactive fiction needs three things from every turn: prose to read, a sense of progression,
and a set of next actions to choose from. The naive approach generates each with separate logic:
narration from the model, choices hardcoded or generated by a second pass, progress tracked
in code. That’s brittle and expensive in calls.
The goal was a self-contained, offline engine where one local model produces all
three at once, the application is a thin parser-and-prompt-builder around it, and arbitrary
scenarios can be dropped in as data without touching the engine. No API keys, no network, no
per-token bill. It runs entirely on local hardware.
Section 02
Architecture
The engine is a prompt-builder and parser wrapped around a locally served model. One prompt goes
out each turn; one reply comes back and is split into everything the interface needs.
A local model is served by KoboldCpp (a llama.cpp-based inference server); the engine talks to
its HTTP API at localhost:5001. On connect it auto-detects which model is running
and selects the matching prompt format (e.g. Mistral vs. ChatML), so swapping the underlying
model needs no code change.
Scenario as data
Each scenario is a JSON file defining the premise: a system prompt and the hand-authored opening
choices. The engine is fully content-agnostic: the same engine runs a training simulation,
an educational branching scenario, an onboarding walkthrough, or any other premise, because all
content lives in user-authored JSON and nothing about the domain is baked into the code.
One call, three outputs
Each turn the engine assembles a single prompt (scenario system prompt + rolling memory summary +
the last ~14 turns + the player’s action + current meter values) and instructs the model to
emit three labelled blocks in one reply: a few paragraphs of narration, a
[STATS] block of meter values (0–100), and an [ACTIONS]
block of five numbered next actions.
Parse, not generate
The streamed reply is split on the [STATS] / [ACTIONS] markers:
text before the markers becomes the story, the stats block becomes the progress bars, the action
lines become the choice buttons. During streaming the machinery blocks are hidden, so the player
sees only prose until parsing completes.
State drives control flow
The meters parsed from the model’s own output feed the UI progress bars, and when a meter
reaches 100 an “ending” path unlocks, sending a special director instruction back
to the model to write the finale. The LLM’s output is the game state.
Section 03
The hard parts
Reliability 01
Structure from an unreliable surface
Local models don’t guarantee clean formatting, and the whole UI depends on parsing
their output. The parser is deliberately defensive: it strips list markers, discards lines too
long, too short, or that look like leftover tags, and caps the set at six. If the model emits
nothing usable, the engine falls back to fixed controls and free typing rather than breaking.
Extracting structured state from free-form generation without trusting the model is the core
reliability work.
Trade-off 02
Single-call structured output
Producing narration, meters, and actions in one call is cheap and keeps them coherent with
each other, but it couples three concerns into one parse surface. The defensive parser is the
price of that choice; a function-calling or constrained-decoding approach would be more robust
but wasn’t practical for the local models in use.
Subsystem 03
Long-session coherence
A fixed context window can’t hold a whole story. A second, separate LLM call runs every
five turns to rewrite a ~180-word running summary of facts and events, fed back into every
subsequent prompt. This is context compression as an explicit subsystem: the engine
spends a call on remembering so the main loop stays consistent over long play.
Portability 04
Model heterogeneity
Different local models expect different prompt templates. Auto-detecting the served model and
switching format keeps the engine model-agnostic, so the underlying model can be swapped for
quality/size trade-offs without touching the loop.
Boundary 05
Human vs. model authorship
Only the first turn’s options are human-authored, per scenario; every set after is the
model’s. The fixed director controls (steering buttons that inject canned
instructions as the player’s action) are hardcoded UI, deliberately not
AI-generated: a clean separation between scripted control and generated content.
Section 04
Configuration & numbers
Architectural constants are listed below as configuration. Runtime figures
(tokens/sec, per-turn latency, VRAM footprint) weren’t instrumented and depend entirely on
the host hardware and model size, so they’re left out rather than estimated.
Four open models, one per size tier, were tested on a single RTX 5070 Ti
(16 GB GDDR7, 896 GB/s, Blackwell with FP4/FP8 support):
Size tier
Base family
Result on 16 GB
12B
Mistral Nemo
Fits easily; weaker format adherence
22B
Mistral Small
Fits
24B (chosen)
Mistral Small
Fits in VRAM with workable context, fast
70B
Llama-70B class
Spills to system RAM → much slower
The deciding constraint is VRAM. At 4-bit, a 24B model’s weights (~14 GB) plus KV cache
fit inside 16 GB with usable context, making ~24B the practical ceiling for this
card. A 70B needs ~35 GB at 4-bit, over double the available memory, so
it runs only by offloading layers to system RAM, which collapses throughput. That matches what was
observed: the 24B ran fast and well; the 70B was much slower with no quality gain worth the wait.
The 24B sweet spot wasn’t a preference; it’s where the hardware budget lands.
Limitations that shaped the engine
Format adherence scales with size. Smaller models honour the three-block
[STATS]/[ACTIONS] contract less reliably, the direct reason the parser is defensive rather
than trusting. Quantization trade-off: fitting 24B in 16 GB means ~4-bit
weights, a quality compromise full-precision cloud hosting wouldn’t require. Context
limits drive the memory subsystem: a fixed window can’t hold a long story, hence the
rolling summary. Quality plateau (observed, not benchmarked): 24B and 70B produced
comparable output for this task, consistent with diminishing returns at this scale for
structured narrative generation.
Section 06
Cloud deployment & cost
How the picture changes off the local card: two paths, at current (2026) rates.
A · Self-host on a rented GPU
Same KoboldCpp stack, bigger hardware: this removes the VRAM ceiling. A 24B runs in full
precision on an L40S (48 GB, ~$0.79/hr) or A100 80 GB (~$0.89–1.79/hr), and the 70B
that crawled locally runs fully in VRAM and fast on an A100 80 GB. Cost is hourly regardless of
use (a 2-hour session ≈ $1.60–3.60), so you pay for idle time unless you
auto-scale to zero (Modal, RunPod serverless). It keeps control and privacy; you own the ops.
B · Per-token managed API Session est.
Pay only for tokens (Together, Fireworks, DeepInfra, OpenRouter). Hosted rates at each size tier:
Model (hosted equivalent)
~Rate / 1M tokens
Mistral Nemo (12B class)
~$0.02
Mistral Small 24B
~$0.05–0.075
Llama 3.3 70B (70B class)
~$0.88
A session is small in token terms: with full context each turn runs a few thousand input + a few
hundred output tokens, so a ~50-turn session is on the order of ~300k tokens. That puts a full
session at roughly $0.02 on a 24B-class model or ~$0.25 on a 70B-class
model: cents either way, with nothing billed when idle. (Session-token figure is
estimated from the engine’s context design, not instrumented.)
The honest conclusion: for a single intermittent user, cloud
doesn’t clearly win. Per-token managed is the cheapest cloud path (cents/session,
zero idle cost) but ships prompts to a third party, losing the offline/privacy property that
motivated the local build. Rented GPU preserves control but wastes money on idle time at single-user
scale. Local (the current setup) is already optimal on both axes for one user: zero
marginal cost and full privacy. And since 24B ≈ 70B for this task, there’s no quality
reason to reach for the larger models cloud would unlock.
Where cloud flips: concurrency
The local engine serves one session on one machine. The moment it’s multi-user, per-token
serverless scales cleanly and stays cheap; dedicated GPUs pay off only at sustained high throughput
(rule of thumb: per-token until hundreds of millions of tokens/day, then dedicated). The deployment
choice is really a function of user count, not the engine.
Section 07
What I'd do differently
Honest forward judgement, drawn from the trade-offs above: the next iteration, not self-criticism.
Replace single-call block-parsing with constrained decoding / a JSON grammar
(e.g. GBNF in KoboldCpp) so structure is guaranteed at generation time and the defensive parser
becomes a safety net rather than the primary mechanism.
Treat the memory summariser as a place for a small eval harness (a
handful of fixed cases checking the summary retains key facts) since silent memory drift is
the most likely long-session failure.
Make the 5-turn memory cadence adaptive by summarising on event density
rather than a fixed counter, to avoid losing detail in busy stretches.
The offline property matters beyond hobby use: because no data leaves the
machine, this architecture maps directly onto privacy-sensitive deployments
such as defence and healthcare, where cloud inference is a non-starter.