Training a tiny model to act like a terminal

A 360 million-parameter language model can run in a browser and behave like a Unix terminal. The demo works best in Chrome or Edge with WebGPU, accepts raw key bytes, and returns exactly the characters that should appear on the screen.

There is no real shell or filesystem behind it; the model itself remembers what you typed a few commands ago, can launch vim, run tmux, and even play snake. The author built the system to test whether a model could keep a consistent world state instead of just spitting out plausible terminal output.

In the sections that follow you’ll see how the terminal is turned into a tokenised grid, why the model predicts screen diffs rather than a linear text stream, and which tricks make scrolling and clearing cheap enough to run interactively in a browser. By the end you’ll have a concrete recipe for turning a modest LLM into a self-contained terminal emulator.

Turning the terminal into a tokenised grid

The terminal is treated as a grid of characters, one cell per column-row pair. The model never sees the grid as an image; it receives a sequence of tokens that describe the grid’s size, its contents, and the cursor position. Every session starts with a snapshot token block such as:

BOS SCREEN_START 32 96 ROW_0 0 'simedw@laptop:~$ ' CURSOR 0 17 1 SCREEN_END

BOS marks the beginning of the stream, SCREEN_START is followed by the number of rows and columns, ROW_0 … ROW_N give the text of each row, and CURSOR supplies the row, column, and a visibility flag (1 = visible). Because the underlying Qwen2.5-Coder-0.5B already contains tokens for all 256 byte values, numbers and characters each occupy a single token.

Row identifiers (ROW_0, ROW_1, …) are also single tokens, and a handful of special operations round the vocabulary out to 333 tokens total. This tiny vocabulary lets the author generate unlimited labelled data by running real shells inside Docker containers, then fine-tune the model on those sessions.

Predicting screen diffs instead of a text stream

Standard language models extend a prompt token-by-token, which works for prose but not for a terminal. A single keystroke can move the cursor, clear the screen, or overwrite part of a line. To match this behavior the model predicts updates to the grid rather than a continuous stream of characters.

After each raw input byte the model outputs a sequence that tells which rows to rewrite and where the cursor ends up. A simple “l” keystroke looks like this:

INPUT_START 'l' INPUT_END
OUTPUT_START ROW_0 17 'l' CURSOR 0 18 1 OUTPUT_END

In plain language: on row 0, column 17 write the character l, then move the cursor to column 18 on the same row, keeping it visible. No explicit clear operation is required; new characters simply overwrite what was there before.

Time is also part of the input. When no key is pressed the model receives a WAIT token together with a duration, which lets it continue moving the snake in the background or keep a tmux pane alive. This explicit time token is essential because many terminal programs rely on timers rather than user input.

Making scrolling and clearing cheap

When the cursor reaches the bottom of the window and the user presses Enter, the terminal must scroll all lines up by one. Emitting every new row would cost roughly 96 tokens per line, quickly exhausting the model’s context window. The author introduced a compact copy operation:

ROW_0 COPY 1 ROW_1 COPY 2 ROW_2 COPY 3 …

Each COPY token tells the decoder to shift an existing row to a new position. Scrolling therefore costs three tokens per row (the row identifier, the COPY keyword, and the target index) instead of a full row’s worth of characters. The same pattern handles clear, line insertion, deletion, and even partial overwrites: copy the target row, then write the changed segment.

Because the vocabulary is tiny, 256 byte tokens, 64 row tokens, and a handful of control tokens, the decoder can enforce a grammar that guarantees every prediction is a valid screen update. This constraint keeps inference fast enough to run in a browser with a 208 MB model file.

Running the demo and what it reveals

The demo is hosted at the author’s site and loads a 208 MB model into the browser. It expects a physical keyboard; a virtual on-screen keyboard adds latency that makes the experience feel sluggish. Chrome or Edge on a desktop with WebGPU gives the smoothest interaction. On a modern phone the model still works but the frame rate drops dramatically.

Typing a misspelled command such as pythoon produces the familiar “command not found” message, showing that the model has learned typical shell responses from its pre-training data. More impressively, the model can launch vim, enter insert mode, and edit a buffer, all without any underlying editor process. It also runs tmux sessions, switches panes, and even plays a simple snake game where the WAIT token drives the snake’s autonomous movement.

The biggest surprise for the author was the model’s ability to remember state across unrelated commands. After creating a file with touch foo.txt, later cat foo.txt correctly displays the file’s contents, even though no real filesystem exists. This memory is purely the model’s internal representation of the screen and the sequence of inputs it has seen.

Trade-offs and limitations

The approach comes with several practical constraints. The model’s context window holds only a few thousand tokens, so long-running sessions eventually push older updates out of memory, causing the model to forget earlier files or commands. Because the model is stochastic, the same keystroke can sometimes produce a different screen update, especially in ambiguous situations like auto-completion; the author mitigates this by fine-tuning on deterministic terminal logs.

Performance on a mid-range GPU averages around 10 fps; larger models would improve accuracy but would no longer fit in the browser’s memory budget. Security is a non-issue, the demo cannot execute arbitrary code on the host machine, but that also means it cannot affect the real system.

The takeaway

Represent the terminal as a tokenised grid and let the model emit screen diffs rather than a linear text stream. Use row-copy operations to compress scrolling, clearing, and line-shifting down to a few tokens each.

With a tiny 333-token vocabulary and grammar-constrained decoding, a 360 M model can run interactively in a browser and maintain a consistent world state without any external shell or filesystem. This demonstrates that modest LLMs can act as self-contained, stateful interfaces when the output format is carefully engineered.