Local models are actually good now - playing with Qwen3.8-27B

The first rounds of local LLM experiments we ran were interesting, but the finished things were not especially good. You could see the shape of what might eventually become useful, usually buried under unfinished interfaces, dead controls, generic dashboards, or code that looked much better than it worked.

Then a 4-bit Qwen3.8-27B running on my laptop chose Conway’s Game of Life and built Lattice, a cellular automata workbench.

I am ridiculously impressed by it.

Not “impressed for a local model” in the slightly patronising way people sometimes talk about small or constrained systems. I mean I opened it, started drawing cells, changed the symmetry, tried its patterns, watched the census and history move, and thought: damn, yeah. This is an actual thing. The model chose the idea itself from one broad Billie-domain prompt, designed the interface, wrote it, tested it, and repaired parts of it through Pi. All of that happened locally on my MacBook Pro.

Local models are actually good now. I do not mean every local model is brilliant at everything. I mean this one stopped feeling like a demo I had to excuse and started feeling like software I wanted to keep using.

Four-bit was surprisingly good

Lattice is not a token Game of Life demo with a start button. It is a workbench.

There are draw and erase tools, play and single-step controls, randomisation, undo, grid and speed settings, symmetry modes, a library of named patterns, population and generation figures, and a small history view. The interface is coherent. The visual decisions feel deliberate. It gives you several ways to interfere with the simulation and then makes the result legible.

This matters because nobody handed the model “build a polished cellular automata workbench” as the task. The prompt gave it the same broad Billie territory as the earlier Local LLM Choice experiments: practical software, independent publishing, analytics, vehicles, travel, making, and curious technical projects. The model selected the idea and decided what the finished object should contain.

It did this at 4-bit.

The formal manual review gave it 4/5 for interest, 3/5 for execution, and 3/5 for taste. Six-bit scored slightly better overall. Four-bit is still the one that changed my mind.

It is not secretly perfect. Several library patterns fail to behave as described, and the model’s own test output showed that some died or remained static before it declared them correct anyway. Four-way symmetry can lose a stamp near an edge because rotations on the rectangular board are clipped. Passing a browser smoke check proves that the controls run without throwing errors. It does not prove the cellular automata are all correct.

Those faults do not make me less surprised by what is there. They make it possible to judge it as software rather than as a magic trick.

We nearly ran the wrong experiment

“I feel like we’ve not done this right” was the useful moment in this project.

The first comparison asked 4-bit, 6-bit, and 8-bit to emit an entire web artifact through one long completion. That measured raw generation, but it did not reproduce the original Local LLM Choice experiment. Those models had worked in agentic rounds. They could create a file, inspect it, run something, notice a problem, and change their approach.

I had removed the bit that makes a coding agent a coding agent, then started drawing conclusions from the result.

So we scrapped that comparison and its premature article. The runner was extended to start Pi in a genuinely empty directory for each quantisation. Every model received the same prompt and sampling settings, with 25 Pi iterations and no retries or post-run repairs.

That correction is why Lattice exists. The model wrote the page, read it, exercised it, and changed it over several rounds. A one-shot completion could show attractive source code. The Pi experiment showed autonomous work.

What happened at 6-bit and 8-bit

The quantisation comparison still matters. It just is not the reason I care about the experiment now.

• 4-bit: 15.1 GB of model files, 12.47 tok/s, 30m 20s wall time, 33.98 GB peak MLX allocation. Completed Lattice.
• 6-bit: 21.9 GB of model files, 9.73 tok/s, 25m 32s wall time, 36.10 GB peak MLX allocation. Completed The Long Now.
• 8-bit: 28.6 GB of model files, 6.88 tok/s before failure, 32m 10s wall time. No comparable final MLX allocation; the partial run ended with Metal OOM.

These are whole-session measures, not simple spec-sheet numbers. The completion rate includes prompt processing as Pi’s working history grows, so it is useful inside this matched experiment but is not raw decode speed. Model-file size is not memory use either. The 8-bit server aborted before producing a comparable final MLX receipt, so I have left that cell empty.

The 6-bit model chose The Long Now, a 5,000-year interactive timeline with logarithmic, hybrid, and linear views. It is polished and unusually thoughtful. During the run, it measured its first signed-log mapping, discovered that the maths did not support its explanatory copy, and implemented a hybrid scale. That inspect-and-repair loop is exactly what the corrected experiment was supposed to expose.

It won the structured scorecard with 4/5 for interest, 3/5 for execution, and 4/5 for taste. It also has overlapping labels and a handful of simplified or wrong historical claims. It is arguably the most composed design. It is not the one I keep opening.

The 8-bit model attempted another Long Now timeline. It had already produced a substantial page and repaired its playhead and speed behaviour when the fourteenth provider call failed. MLX exited with a Metal out-of-memory error and Pi lost the connection.

The preserved page renders, but it is partial. Its layout is sparse, labels crowd together, and the date calculations contain central bugs. More bits gave this run the slowest generation and enough sustained memory pressure to stop it after 32 minutes. “The model loads” and “the agent can finish” are not the same claim.

The local model line moved

My original question was whether this supposedly four-year-old, quite beefy M4 Pro could run Qwen3.8-27B at different quantisations. My response when the arithmetic said yes was basically: all right, hell yeah. Go and orchestrate that.

There are two corrections to make.

First, Apple announced M4 Pro on 30 October 2024. In September 2026 it is not even two years old. I had simply aged it with unusual aggression.

Second, “can it run?” was not ambitious enough. The 4-bit, 6-bit, and 8-bit versions of Qwen3.8-27B all loaded and generated. The more useful question is whether a local model can choose something worth making and carry it far enough that I care about the result.

Four-bit did.

I would use 4-bit as the default on this machine. It was about 28% faster than 6-bit by this experiment’s aggregate measure, leaves more room for everything else, and made my favourite artifact. Six-bit is completely viable when I want to trade some speed for the more considered behaviour it showed in The Long Now. I would not start another long 8-bit Pi run on 48 GB without shortening the history or narrowing the task.

One run per quantisation is not a leaderboard. A different seed or project could reverse the quality order. I did not measure battery use, temperature, or repeated-run variance. “Local models are actually good now” is not a claim that every local model has suddenly become brilliant.

It is the reaction I had after the smallest quantisation in this comparison chose a cellular automata workbench, built it on my laptop, and gave me something I wanted to play with.

The public comparison contains all three exact artifacts, the run receipts, and the defects we found. The model caches are staying where they are. I want to see what 4-bit decides to make next.