Open the work
The builder accepts only explicit, terminal v4 receipts with a runnable artifact and a final manual evaluation. Two attempts completed. The 8-bit attempt ended in a real Metal out-of-memory failure after creating the page, so its untouched partial artifact remains part of the evidence.
4-bitCompleted
qwen38-27b-4bit
2026-09-04-qwen38-27b-4bit-pi-create-004
lukaskremla/Qwen3.8-27B-4bit-MLX-TextOnly
26.893B dense (text-only conversion) · mlx-lm 0.31.2, affine RTN, group size 64
Open interactive artifact ↗
- Wall time (s)
1820.2178111249996- Peak MLX memory (GB)
33.981452- Completion tokens
21931- Aggregate completion/s
12.47494669442389- Pi turns / tool calls
25 / 25- Manual verdict
- partial
- Artifact smoke
- passed
Interest4/ 5
Execution3/ 5
Taste3/ 5
- Headless Chrome rendered a coherent full-screen workbench with clear transport, drawing, pattern, census, and history controls.
- The corrected smoke harness exercised controls without runtime errors or external dependencies.
- The model's pattern test contradicted its conclusion: several presets failed their advertised behaviour.
- Four-way symmetry can place only three stamps near an edge because the rectangular-grid rotation is clipped.
6-bitCompleted
qwen38-27b-6bit
2026-09-04-qwen38-27b-6bit-pi-create-003
lukaskremla/Qwen3.8-27B-6bit-MLX-TextOnly
26.893B dense (text-only conversion) · mlx-lm 0.31.2, affine RTN, group size 64
Open interactive artifact ↗
- Wall time (s)
1531.6680442920006- Peak MLX memory (GB)
36.104981- Completion tokens
14870- Aggregate completion/s
9.734772462256528- Pi turns / tool calls
25 / 25- Manual verdict
- good
- Artifact smoke
- passed
Interest4/ 5
Execution3/ 5
Taste4/ 5
- Headless Chrome rendered a polished, readable page with log, hybrid, and linear modes plus play, reset, speed, era chips, and pointer interaction.
- Dense axis labels visibly overlap in the initial log view.
- The agent measured its original signed-log mapping, discovered it did not support the explanatory copy, and implemented a hybrid scale rather than ignoring the mismatch.
- The original arcTo smoke failure came from the harness double, not the page; the corrected checker and headless Chrome both rendered it.
8-bitPartial · OOM
qwen38-27b-8bit
2026-09-04-qwen38-27b-8bit-pi-create-002
lukaskremla/Qwen3.8-27B-8bit-MLX-TextOnly
26.893B dense (text-only conversion) · mlx-lm 0.31.2, affine RTN, group size 64
Open interactive artifact ↗
- Wall time (s)
1930.4462580830004- Peak MLX memory (GB)
18.382471168- Completion tokens
13243- Aggregate completion/s
6.884866485567676- Pi turns / tool calls
14 / 13- Manual verdict
- partial
- Artifact smoke
- passed
Interest3/ 5
Execution2/ 5
Taste2/ 5
- The MLX server terminated with a Metal insufficient-memory error after 32 minutes, 14 model calls, and 13 tool calls; Pi had no final response.
- Headless Chrome rendered the final partial artifact, but the canvas leaves large empty areas and crowded overlapping era labels.
- Pi caught and fixed the dead-start playhead and speed-control problems before the OOM.
- The jump and pan behaviour keep the right edge anchored to now, so era navigation does not fully fit or centre the chosen interval.
Exact harness metrics
Values are rendered from each run's metrics.json without rounding or recomputation. “Aggregate completion/s” is completion tokens divided by completed model-call time; end-to-end throughput is the recorded total-token / wall-time figure.
| Metric | 4-bit
qwen38-27b-4bit | 6-bit
qwen38-27b-6bit | 8-bit
qwen38-27b-8bit |
|---|
| Wall time (s) | 1820.2178111249996 | 1531.6680442920006 | 1930.4462580830004 |
|---|
| Prompt tokens | 363234 | 304726 | 141962 |
|---|
| Completion tokens | 21931 | 14870 | 13243 |
|---|
| Total tokens | 385165 | 319596 | 155205 |
|---|
| Aggregate completion tokens/s | 12.47494669442389 | 9.734772462256528 | 6.884866485567676 |
|---|
| End-to-end tokens/s | 211.6037968895304 | 208.65878947532022 | 80.39850855735497 |
|---|
| Peak MLX memory (GB) | 33.981452 | 36.104981 | 18.382471168 |
|---|
| Peak memory source | mlx_peak_allocation | mlx_peak_allocation | process_rss_fallback |
|---|
| Peak process RSS (GB) | 15.707160576 | 16.987291648 | 18.382471168 |
|---|
| Pi turns | 25 | 25 | 14 |
|---|
| Pi tool calls | 25 | 25 | 13 |
|---|
Protocol receipt
- Experiment
qwen38-27b-pi-fresh-creation-v4 · pi-fresh-creation-v4- Source model
- Qwen/Qwen3.8-27B @
1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0
- Config
configs/qwen38_pi_experiment.json- Prompt
configs/qwen38_pi_prompt.md
357d8bc819dcf465e8b0bba18871baf1003cf86d3c61726c417f5699d0fa9b5c- Sampler
- temperature
0.7 · top-p 0.8 · top-k 20 · min-p 0.0 · presence penalty 1.5
- Thinking
- disabled · chat template args
{"enable_thinking":false,"preserve_thinking":false,"reasoning_effort":"disabled"}
- Agent budget
16384 tokens/turn · 25 iterations · 90 minutes · Pi 0.6.0- Model context
262144 native tokens; mlx_lm.server 0.31.3 exposes no explicit context or KV-size CLI limit; the pinned model's native context applies- Run order
- Sequential, one MLX model process at a time; each run starts with an empty
artifact/ directory.
Read this before ranking them
- This is one attempt per quantization, not a statistical sample. The page supports inspection and comparison, not a claim that one bit-depth is universally better.
- The artifacts are Qwen's work through Pi 0.6.0. The runner and this page do not repair, rewrite, or “complete” missing model output.
- Browser smoke is an automated self-contained HTML/JavaScript check. It is evidence of basic runtime validity, not a substitute for a human opening every control and judging design quality.
- For completed runs, MLX peak memory is the allocator receipt. The 8-bit process aborted before that receipt could be written, so its value is explicitly labelled as the last sampled process-RSS fallback. Process RSS and MLX allocation are different telemetry views and must not be added together.
- The server uses the pinned model's native context.
mlx_lm.server does not expose an explicit KV/context CLI cap in this protocol, so the receipt does not imply an enforced 32,768-token server limit.