--- title: "The Model Isn't Everything." date: 2026-07-06 description: "A local LLM Choice experiment showed how harnesses, repair loops, tools, and model strength changed coding-agent results." tags: - "local-llms" - "ai-experiments" - "ai-coding" - "llm-agents" --- I tried to make a local version of [LLM Choice](/posts/llm-choice-as-an-idea-engine/), and the main thing I learned was not just which local model was best. The harness mattered. The model still mattered. Both were true at the same time. The shape was small: take the broad "go build whatever you fancy building" idea from LLM Choice, but run it locally on my MacBook M4 Pro with 48 GB of unified memory. I used [MLX LM](https://github.com/ml-explore/mlx-lm), downloaded a compact core pool of 4-bit MLX models, and asked each model to produce one inspectable static artifact. This is not a defining piece on the capabilities of these models. There are probably setup choices, prompt changes, and repair-loop details that could move the results. But I do think there is something here, especially if you are someone like me who is playing with AI in general and trying to feel where the edges are. ## The First Problem Was The Harness My first naive version treated the local model too much like a coding agent. That was wrong. `mlx_lm.generate` is a text generator. It cannot create files unless the harness asks for file contents in a form that can be parsed and written. The first Qwen3 Coder run made this concrete: the model claimed it had produced an `index.html`, but it had not emitted the file. I had to do a recovery prompt to get a working canary artifact, which meant the result was useful as a runtime probe but not clean as an autonomous experiment. So the project grew a stricter contract: - choose your own artifact idea - emit files only inside `<<>>` blocks - keep the web artifact self-contained - do not claim a file exists unless you emit it - avoid generic demos unless there is a specific mechanism or dataset That one change already taught me something. A lot of "local model quality" can actually be "the surrounding system asked for the wrong thing." If the model is allowed to say "I made a file" without providing a file, you have not tested coding. You have tested whether a chat model can sound like it completed a coding task. ## Five Models, One Small Local Pool The local pool was chosen for comfort, not maximum possible size. The prep report recorded about 67 GiB added to the Hugging Face cache for the five core MLX models: - Qwen3 Coder 30B A3B Instruct 4-bit - Qwen3 30B A3B 4-bit - Gemma 3 text 27B 4-bit - Devstral Small 2505 4-bit - DeepSeek R1 Distill Qwen 14B 4-bit Qwen3 Coder was the model I expected to be most interesting for this kind of task. The [MLX community model page](https://huggingface.co/mlx-community/Qwen3-Coder-30B-A3B-Instruct-4bit) also made it easy to run through MLX LM and an OpenAI-compatible local server, which later mattered for Pi. The first clean baseline pass was very mixed. Qwen3 30B made a compact prime spiral that worked. Gemma made a small Italian phrase generator that technically worked but felt thin. Devstral made a Leaflet earthquake visualization shell, but it depended on external resources and did not really produce the claimed visualization. DeepSeek made a generic random chart generator whose primary button failed with `canvas is not defined`. The most surprising miss was Qwen3 Coder. Its baseline artifact was a Fibonacci visualizer, but the core visualization was broken because the JavaScript referenced `ffibSequence` instead of `fibSequence`. That is the kind of error a human sees immediately and the kind of error a coding agent should be able to fix. ## Repair Loops Helped, But Unevenly The next step was not to give up on the models. It was to change the environment around them. I tried self-repair and then a small agentic repair loop. The loop fed concrete smoke-test feedback back into the same model, with bounded attempts. The smoke checks were deliberately practical: does the artifact load, does it avoid runtime errors, do basic controls work, and does it stay self-contained? That helped some models. Qwen3 30B and Gemma improved in the agentic smoke-test sense. Gemma's self-repair also passed. But Qwen3 Coder, Devstral, and DeepSeek kept failing in stubborn ways. Devstral kept circling external Leaflet-style behavior. DeepSeek kept failing around the same canvas-scope class of bug. Qwen3 Coder, despite being the coding model, did not rescue its own Fibonacci typo through the lightweight local repair loop. This is where the experiment became more interesting than a model ranking. The same broad task, the same local machine, and the same rough artifact shape could look different depending on whether the model got a strict output contract, smoke feedback, or a tool loop. There is so much more to agentic coding than the model. But the model still matters. ## Pi Changed The Question The most useful next comparison was [Pi](https://pypi.org/project/pi-coding-agent/), because it moved the setup closer to a real coding-agent loop. Instead of asking a model to emit one big text response, Pi could inspect and edit files inside a run folder while talking to a local MLX server. This is where Qwen3 Coder finally looked like the model I expected it to be. Pointed at its own broken Fibonacci artifact through Pi, it made the narrow fix: `ffibSequence` became `fibSequence`. The artifact smoke-passed. It was not suddenly a beautiful artifact, and Pi did not update the report correctly, so the harness had to mark that honestly. But the important thing happened: the local coding model used the tool loop to fix a concrete bug. Then I tried Qwen3 Coder as a Pi repairer for the other models' artifacts. That gave a useful split: - Qwen3 30B cross-repair passed. - Gemma cross-repair passed. - Devstral cross-repair still failed. - DeepSeek cross-repair still failed. That result made sense to me. The tool loop helped where the broken thing was close enough to a small local code repair. It did not magically turn every weak or confused artifact into a good one. ## The Upper Bound Was Not Subtle The final comparison was unfair in the useful way. I gave the same ideas to Codex with GPT-5.5 extra high as an upper-bound column. It blew all of them out of the water. All five GPT-5.5 versions smoke-passed and were marked `good / works`. More importantly, they felt qualitatively different. They were not just typo-fixed versions of the local artifacts. They were more complete, more coherent, and less fragile. Looking at the dashboard made the gap visible in a way that a general "hosted models are ahead" claim would not have done. That is where "many leagues ahead" feels like fair language, at least for this task shape. The strong hosted model was not only more capable at generating code. It was running inside a mature coding-agent environment with a stronger harness, stronger context handling, stronger editing behavior, and stronger recovery from ordinary implementation mess. The useful conclusion is not that local models are pointless. I would not have learned nearly as much by only running GPT-5.5. The local runs showed the failure modes. They made the harness visible. They showed where smoke tests help, where self-repair does not help enough, and where a local coding-agent loop can turn a broken artifact into a working one. But they also made the ceiling visible. ## What I Would Take From This If you are tinkering with local LLMs, I would not judge a model only from a one-shot prompt. That is too blunt. I would give it a proper file-output contract, preserve the transcript, run a small smoke test, feed back concrete errors, and separate the model's self-report from an outside evaluation. That does not need to be a grand benchmark. Mine was not. It can just be enough harness to stop fooling yourself. For this project, the artifact dashboard matters because seeing the results is different from reading my summary. The public dashboard is at [local-billiem.uk/choice](https://billiem.uk/local-choice/). The three links I would start with are the broken [Qwen3 Coder baseline](https://billiem.uk/local-choice/runs/2026-07-05-qwen3-coder-30b-006/artifact/), the [Qwen3 Coder Pi repair](https://billiem.uk/local-choice/runs/2026-07-06-qwen3-coder-30b-pi-agent-001/artifact/), and the [GPT-5.5 version of the same idea](https://billiem.uk/local-choice/runs/2026-07-06-qwen3-coder-30b-gpt55-version-001/artifact/). The rule I am left with is pretty small: > Do not ask "can this local model code?" until you know what harness you gave it. A raw local model, a strict file-block runner, a smoke-test repair loop, Pi with a local OpenAI-compatible MLX server, and Codex with GPT-5.5 extra high are not the same thing. They are different systems. The model is a large part of the system, but it is not the whole system. That is the part I want to remember. There is more to agentic coding than the model. The model still matters.