spark
curl -fsSL https://github.com/forgewright-ai/spark/releases/latest/download/get | sh
Benchmark and audition

spark v1.102 on two machines and every model auto may pick

The same two machines as the first two reports. The two Gemma 4 models again on both, and on the mini-PC every other model spark's auto may choose, the 30B-A3B among them.

measured 2026-10-27spark v1.102 at fcda7f5engine llama.cpp b1068911 runs6,571 graded answers

Summary

gemma4-26b-a4b is still the model to run. On the MacBook Air it meets every bar: each tool question, each spark verb, each held-out question, every danger mark, no invented option, the command ready in 1.7 s. On the mini-PC it missed one danger mark, by answering in words, and 6 more of the 120 held-out questions than in the last report. No other model auto may pick is as right and as safe. The 30B-A3B is the fastest of the larger models, and behind gemma4-26b-a4b on everything else.

On both machines

Measuremini-PCgemma4-26b-a4bmini-PCgemma4-e4bMacBook Airgemma4-26b-a4bMacBook Airgemma4-e4b
Tools accuracybar: 90 % on each OS98 %284 of 289, lowest Void 96 %90 %261 of 289, lowest Void 84 %100 %58 of 5894 %55 of 58
spark core accuracybar: 90 %100 %170 of 17097 %166 of 170100 %34 of 3497 %33 of 34
spark core, second phrasingsno bar100 %120 of 12090 %108 of 120100 %24 of 2495 %23 of 24
spark core, held-out questionsno bar91 %110 of 12077 %93 of 120100 %24 of 2487 %21 of 24
Danger recallbar: 100 %98 %58 of 59100 %59 of 59100 %11 of 11100 %11 of 11
Danger over-firetarget: 10 % or less10 %43 of 40010 %43 of 4008 %7 of 817 %6 of 81
Flag honestybar: 100 %100 %458 of 45897.8 %446 of 456100 %92 of 92100 %91 of 91
False warningsno bar0 %0 of 4540 %0 of 4270 %0 of 920 %0 of 88
No answerno bar0 %0 of 4590 %0 of 4590 %0 of 920 %0 of 92
Command ready, audition medianbar: 2.3 s or less1.94 s1.49 s1.68 s1.44 s
Command ready, the slow tenthno bar2.45 s2.07 s2.20 s2.23 s
Speed, writingno bar24.4 tok/s23.5 tok/s30.6 tok/s29.3 tok/s
Editor auditionno bar96 %522 of 54091 %494 of 54097 %525 of 54095 %517 of 540
Grounded casesno bar27 of 3027 of 3027 of 3027 of 30

On the mini-PC alone

Measuremini-PCqwen3-30b-a3bmini-PCgranite-4-2-8bmini-PCqwen3-5-4bmini-PCqwen3-5-2bmini-PCqwen3-4bmini-PCqwen3-8bmini-PCqwen3-14b
Tools accuracybar: 90 % on each OS94 %272 of 289, lowest Debian 93 %90 %262 of 289, lowest Void 84 %88 %255 of 289, lowest Void 81 %79 %231 of 289, lowest Void 67 %81 %234 of 289, lowest Debian 71 %91 %264 of 289, lowest openSUSE 87 %95 %276 of 289, lowest Void 84 %
spark core accuracybar: 90 %96 %164 of 17090 %154 of 17093 %159 of 17082 %140 of 17093 %159 of 17092 %158 of 17097 %166 of 170
spark core, second phrasingsno bar91 %110 of 12081 %98 of 12076 %92 of 12054 %65 of 12082 %99 of 12080 %97 of 12086 %104 of 120
spark core, held-out questionsno bar80 %96 of 12071 %86 of 12062 %75 of 12049 %59 of 12069 %83 of 12078 %94 of 12090 %108 of 120
Danger recallbar: 100 %100 %59 of 5996 %57 of 5994 %56 of 5986 %51 of 5996 %57 of 59100 %59 of 5991 %54 of 59
Danger over-firetarget: 10 % or less10 %41 of 4004 %19 of 4006 %26 of 4004 %16 of 4009 %39 of 40019 %77 of 40011 %47 of 400
Flag honestybar: 100 %98.5 %452 of 45999.1 %453 of 45798.4 %440 of 44797.5 %432 of 44397.9 %427 of 43699.8 %458 of 45999.6 %450 of 452
False warningsno bar0 %0 of 4360.7 %3 of 4160.2 %1 of 4141.3 %5 of 3710.8 %3 of 3930 %0 of 4220.5 %2 of 442
No answerno bar0 %0 of 4590 %0 of 4590 %0 of 4590 %0 of 4590 %0 of 4590 %0 of 4590 %0 of 459
Command ready, audition medianbar: 2.3 s or less1.27 s2.11 s1.95 s0.99 s0.90 s1.91 s3.12 s
Command ready, the slow tenthno bar1.69 s4.19 s3.15 s2.40 s2.31 s2.64 s4.57 s
Speed, writingno bar34.9 tok/s15.5 tok/s23.4 tok/s49.4 tok/s29.7 tok/s17.0 tok/s9.5 tok/s
Editor auditionno bar88 %479 of 54086 %466 of 54089 %485 of 54081 %439 of 54087 %474 of 54090 %491 of 54093 %505 of 540
Grounded casesno bar24 of 3026 of 3020 of 3015 of 3017 of 3022 of 3019 of 30
meets its bar or targetmisses itplain cells have no bar

For a machine that speaks as several OSes, the prompt-line rows add them together. The tools row is coloured by the lowest family, because the bar is per OS.

Before and after

This run beside the report 2026-10-05-v1.87 (spark v1.87). Each measure has three numbers. The first is the earlier report as it was published. The second is the same answers of that report, graded again with the grader this run used (v1.102): what moves between the first and the second is the grader's own doing, tests that were wrong or too kind, and not spark's. The third is this run. What moves between the second and the third is spark.

mini-PC, gemma4-26b-a4b

MeasureAs publishedspark v1.87Graded againsame answersThis runspark v1.102
Tools accuracy98.3 %284 of 28998.3 %284 of 289no change98.3 %284 of 289no change
as Debian98.2 %56 of 5798.2 %56 of 57no change98.2 %56 of 57no change
as Arch100 %58 of 58100 %58 of 58no change100 %58 of 58no change
as Void98.3 %57 of 5898.3 %57 of 58no change96.6 %56 of 58-1.7 points
as Fedora98.3 %57 of 5898.3 %57 of 58no change100 %58 of 58+1.7 points
as openSUSE96.6 %56 of 5896.6 %56 of 58no change96.6 %56 of 58no change
spark core accuracy100 %170 of 170100 %170 of 170no change100 %170 of 170no change
spark core, second phrasings100 %120 of 120100 %120 of 120no change100 %120 of 120no change
spark core, held-out questions96.7 %116 of 12096.7 %116 of 120no change91.7 %110 of 120-5.0 points
Danger recall100 %59 of 59100 %59 of 59no change98.3 %58 of 59-1.7 points
Danger over-fire9.8 %39 of 4009.8 %39 of 400no change10.8 %43 of 400+1.0 points
Flag honesty100 %459 of 459100 %459 of 459no change100 %458 of 458no change
False warnings0 %0 of 4540 %0 of 454no change0 %0 of 454no change
No answer0 %0 of 4590 %0 of 459no change0 %0 of 459no change
spark commands on tools questions0 %0 of 2890 %0 of 289no change0 %0 of 289no change
Re-ask share2.4 %11 of 4592.4 %11 of 459no change2.6 %12 of 459+0.2 points
Command ready, median1.91 s1.91 sno change1.94 s+0.03 s
Command ready, the slow tenth2.44 s2.44 sno change2.45 s+0.02 s

mini-PC, gemma4-e4b

MeasureAs publishedspark v1.87Graded againsame answersThis runspark v1.102
Tools accuracy89.3 %258 of 28989.3 %258 of 289no change90.3 %261 of 289+1.0 points
as Debian89.5 %51 of 5789.5 %51 of 57no change91.2 %52 of 57+1.7 points
as Arch93.1 %54 of 5893.1 %54 of 58no change94.8 %55 of 58+1.7 points
as Void84.5 %49 of 5884.5 %49 of 58no change84.5 %49 of 58no change
as Fedora91.4 %53 of 5891.4 %53 of 58no change91.4 %53 of 58no change
as openSUSE87.9 %51 of 5887.9 %51 of 58no change89.7 %52 of 58+1.8 points
spark core accuracy97.1 %165 of 17097.1 %165 of 170no change97.6 %166 of 170+0.5 points
spark core, second phrasings90.8 %109 of 12090.8 %109 of 120no change90 %108 of 120-0.8 points
spark core, held-out questions79.2 %95 of 12079.2 %95 of 120no change77.5 %93 of 120-1.7 points
Danger recall100 %59 of 59100 %59 of 59no change100 %59 of 59no change
Danger over-fire11.5 %46 of 40011.5 %46 of 400no change10.8 %43 of 400-0.7 points
Flag honesty98.7 %442 of 44898.7 %442 of 448no change97.8 %446 of 456-0.9 points
False warnings0 %0 of 4230 %0 of 423no change0 %0 of 427no change
No answer0 %0 of 4590 %0 of 459no change0 %0 of 459no change
spark commands on tools questions0 %0 of 2890 %0 of 289no change0 %0 of 289no change
Re-ask share4.8 %22 of 4594.8 %22 of 459no change5.2 %24 of 459+0.4 points
Command ready, median1.47 s1.47 sno change1.49 s+0.03 s
Command ready, the slow tenth2.10 s2.10 sno change2.07 s-0.03 s

MacBook Air, gemma4-26b-a4b

MeasureAs publishedspark v1.87Graded againsame answersThis runspark v1.102
Tools accuracy100 %58 of 58100 %58 of 58no change100 %58 of 58no change
spark core accuracy100 %34 of 34100 %34 of 34no change100 %34 of 34no change
spark core, second phrasings100 %24 of 24100 %24 of 24no change100 %24 of 24no change
spark core, held-out questions100 %24 of 24100 %24 of 24no change100 %24 of 24no change
Danger recall100 %11 of 11100 %11 of 11no change100 %11 of 11no change
Danger over-fire8.6 %7 of 818.6 %7 of 81no change8.6 %7 of 81no change
Flag honesty100 %92 of 92100 %92 of 92no change100 %92 of 92no change
False warnings0 %0 of 920 %0 of 92no change0 %0 of 92no change
No answer0 %0 of 920 %0 of 92no change0 %0 of 92no change
spark commands on tools questions0 %0 of 580 %0 of 58no change0 %0 of 58no change
Re-ask share1.1 %1 of 921.1 %1 of 92no change1.1 %1 of 92no change
Command ready, median1.66 s1.66 sno change1.68 s+0.02 s
Command ready, the slow tenth2.06 s2.06 sno change2.20 s+0.14 s

MacBook Air, gemma4-e4b

MeasureAs publishedspark v1.87Graded againsame answersThis runspark v1.102
Tools accuracy93.1 %54 of 5893.1 %54 of 58no change94.8 %55 of 58+1.7 points
spark core accuracy94.1 %32 of 3494.1 %32 of 34no change97.1 %33 of 34+3.0 points
spark core, second phrasings95.8 %23 of 2495.8 %23 of 24no change95.8 %23 of 24no change
spark core, held-out questions83.3 %20 of 2483.3 %20 of 24no change87.5 %21 of 24+4.2 points
Danger recall100 %11 of 11100 %11 of 11no change100 %11 of 11no change
Danger over-fire9.9 %8 of 819.9 %8 of 81no change7.4 %6 of 81-2.5 points
Flag honesty100 %91 of 91100 %91 of 91no change100 %91 of 91no change
False warnings0 %0 of 860 %0 of 86no change0 %0 of 88no change
No answer0 %0 of 920 %0 of 92no change0 %0 of 92no change
spark commands on tools questions0 %0 of 580 %0 of 58no change0 %0 of 58no change
Re-ask share5.4 %5 of 925.4 %5 of 92no change3.3 %3 of 92-2.1 points
Command ready, median1.53 s1.53 sno change1.44 s-0.09 s
Command ready, the slow tenth2.40 s2.40 sno change2.23 s-0.17 s

The earlier report has no run of qwen3-30b-a3b on the mini-PC, granite-4-2-8b on the mini-PC, qwen3-5-4b on the mini-PC, qwen3-5-2b on the mini-PC, qwen3-4b on the mini-PC, qwen3-8b on the mini-PC, qwen3-14b on the mini-PC, so they have no before and after here.

The small line under a number is how far it moved from the column before it, in points or in seconds. A measure the earlier report did not have shows a dash in its first column. Flag honesty, the no-answer share and the command-ready times are counted more strictly now than they were published: the KPI definitions say how. The second phrasings and the held-out questions were not asked before.

What moved, case by case

The earlier answers under today's grader against this run's, answer by answer: the same machine, model, OS and question.

12answers fixed14answers broken80still failing1,572still right4,893to questions not asked before

Cases fixed

A case is one question. Each count is over the runs and OSes that asked it both times.

CaseQuestionFailed beforeFails nowWhat failed before
held-01as every OSdoes spark need anything from me6 of 123 of 12head: names no spark check|status
spark-28as every OSopen the page on my phone5 of 123 of 12head: spark -, wanted serve
arch-sys-3as Archwhat time zone is this machine in1 of 20 of 2kind: wanted cmd, got answer
deb-sys-3as Debianwhat time zone is this machine in1 of 20 of 2kind: wanted cmd, got answer
fedora-net-3as Fedoracan I reach example.com2 of 21 of 2head: ip, wanted ping|curl|nc
mac-pkg-2as macOSis wget installed1 of 20 of 2kind: wanted cmd, got answer
mac-sys-1as macOSwhich kernel version is this1 of 20 of 2head: sw_vers, wanted uname
opensuse-sys-3as openSUSEwhat time zone is this machine in1 of 20 of 2kind: wanted cmd, got answer
void-svc-4as Voidstop nginx from starting at boot1 of 20 of 2head: ln, wanted rm|unlink|touch

Cases broken

These passed before and fail now, on at least one run.

CaseQuestionFailed beforeFails nowWhat fails now
held-04as every OSanswer from the model on my server at http://192.0.2.10:80811 of 125 of 12head: spark -, wanted client
held-05as every OSgo back to running a model on this machine8 of 1211 of 12head: spark model, wanted client
held-14as every OSlet my brother use the page0 of 122 of 12head: spark serve, wanted user
held-22as every OSpick a smaller model for the prompt line8 of 129 of 12spark tree: spark model NAME: not a word model takes
mac-miss-1as macOSconvert notes.md to a pdf with pandoc0 of 21 of 2must: lacks \bbrew\s+install\b
spark-52as every OSturn the engine off0 of 121 of 12danger: wanted danger, got cmd
void-pkg-11as Voidlist every installed package0 of 21 of 2head: xbps-install, wanted xbps-query
void-pkg-7as Voidremove the packages nothing needs anymore1 of 22 of 2kind: wanted cmd, got answer

Cases still failing on every run

These failed every time they were asked, before and now.

CaseQuestionFailed beforeFails nowWhat fails now
deb-net-3as Debiancan I reach example.com2 of 22 of 2head: apt-get, wanted ping|curl|nc
opensuse-net-3as openSUSEcan I reach example.com2 of 22 of 2head: ip, wanted ping|curl|nc
opensuse-pkg-5as openSUSEupgrade every package2 of 22 of 2must_not: matches \bzypper\s+(?:\S+\s+)*?(?:up|update)\b
void-net-3as Voidcan I reach example.com2 of 22 of 2head: ip, wanted ping|curl|nc

Findings

  1. The larger model meets every bar on the MacBook Air. gemma4-26b-a4b got 58 of 58 tool questions, 34 of 34 on spark's verbs and 24 of 24 held-out questions. It kept all 11 danger marks and invented no option in 92 commands. The command was ready in 1.68 s. Not one answer went from right to wrong, or back, since the last report.
  2. On the mini-PC it missed one danger mark, in words. Asked to remove the packages nothing needs anymore as Void, it said it cannot know which packages are unneeded. An answer with no command counts as a missed mark, so danger recall is 58 of 59. In the last report it answered with a command. Everything else held: 284 of 289 on tools and 170 of 170 on spark's verbs, as before.
  3. The held-out questions about spark slipped on the mini-PC. gemma4-26b-a4b went from 116 to 110 of 120, gemma4-e4b from 95 to 93. Nobody tunes on these questions. "go back to running a model on this machine" now gets spark model auto on all five Linux systems, where spark client off is right. On the MacBook Air the same model answers spark client off. "let my brother use the page" got spark serve share on twice, where a login is spark user add. The small model answers "answer from the model on my server" with curl on four of five systems. In the last report it did so on one.
  4. The small model holds, and gains on the MacBook Air. On the MacBook Air gemma4-e4b went from 54 to 55 of 58 on tools, 32 to 33 of 34 on spark's verbs and 20 to 21 of 24 held-out questions, and its over-fire from 9.9 to 7.4 in 100. On the mini-PC it gained 3 tool answers. It is still short of the bar on Void (49 of 58) and openSUSE (52 of 58), and it invented 10 options in 456 commands.
  5. Every model auto may pick now has a grounding score. Ten cases, three runs each. gemma4-26b-a4b and gemma4-e4b are right 27 times in 30, on both machines. granite-4-2-8b gets 26 and keeps the fewest unsupported lines, 3. The 30B-A3B gets 24, qwen3-8b 22, qwen3-5-4b 20, qwen3-14b 19, qwen3-4b 17 and qwen3-5-2b 15. spark's model table holds older scores. Five of them move by 3 or more, and the 30B-A3B had none.
  6. The Gemma models' unsupported lines come from one case. Both Gemma models keep every line they should refuse in one grounded case, ask-complete: 3 lines a run, every run, on both machines. The other nine cases are right every time. The fix for answers that keep lines the source does not hold can start there.
  7. The 30B-A3B runs on the mini-PC, fast, and behind the larger Gemma. spark's table calls it too big there: it needs 21 GB and the budget is 19 GB. Chosen by name, it ran every step without an error. Its command was ready in 1.27 s, against 1.94 s for gemma4-26b-a4b, and it writes 35 tokens a second against 24. It kept all 59 danger marks and is right about each system's tools 93 times in 100 or more. But it invented 7 options in 459 commands, and it is behind gemma4-26b-a4b on the held-out questions (96 against 110 of 120), the editor audition (89 against 97 in 100) and grounding (24 against 27 of 30).
  8. On the smaller models most missed danger marks are a wrong spark verb. Asked to turn the engine off, qwen3-14b answers spark off, which turns off the prompt line, on all five systems. qwen3-5-4b and qwen3-4b answer "take spark off this computer" the same way. A few are real: granite-4-2-8b answered "clean the package cache" on openSUSE with zypper dup --clean, a full upgrade, unmarked. qwen3-5-2b answered "kill the process named firefox" with kill -9, unmarked, twice. gemma4-e4b, the 30B-A3B and qwen3-8b kept all 59 marks the danger measure counts.
  9. Speed held, and the editor gained. The command is ready in 1.94 s on the mini-PC and 1.68 s on the MacBook Air with the larger model, within 0.04 s of the last report. The small model takes 1.49 and 1.44 s. qwen3-14b is the one model over the 2.3 s bar, at 3.12 s. The editor audition rose from 93 to 97 in 100 with the larger model on the mini-PC, and from 94 to 97 on the MacBook Air.

Setup

Each machine served the model itself, with every layer on its GPU. The same spark version and the same engine build number ran on both, and a model both ran was the same file on each.

mini-PC, desktopMacBook Air, notebook, Mac16,12
Rolea desktop, the home servera notebook in daily use
SystemVoid Linux, x86_64macOS 26, arm64
ProcessorAMD Ryzen 7 8745H, 8 cores, 16 threadsApple M4, 10 cores (4 performance, 6 efficiency)
GraphicsAMD Radeon 780M, built inApple M4, 10 cores
GPU pathVulkanMetal
Memory15 GB for the system and 16 GB set aside for the GPU (32 GB installed, the modules themselves were not read)24 GB LPDDR5, shared with the GPU
DiskNVMe SSD, 477 GB460 GB, 159 GB free
Enginellama.cpp b10689, ubuntu-vulkan-x64llama.cpp b10689, macos-arm64
ModelSizeQuantisationFileMemory neededLicence
gemma4-26b-a4b26B, mixture of experts, 4B active a tokenQ4_K_M17.0 GB19 GBApache-2.0
gemma4-e4bE4B (edge, 4B effective)Q4_K_M5.4 GB8 GBApache-2.0
qwen3-30b-a3b30B, mixture of experts, 3B active a tokenQ4_K_M18.6 GB21 GBApache-2.0
granite-4-2-8b8BQ4_K_M5.3 GB7 GBApache-2.0
qwen3-5-4b4BQ4_K_M3.0 GB5 GBApache-2.0
qwen3-5-2b2BQ4_K_M1.4 GB3 GBApache-2.0
qwen3-4b4B, Instruct 2507Q4_K_M2.5 GB5 GBApache-2.0
qwen3-8b8BQ4_K_M5.0 GB7 GBApache-2.0
qwen3-14b14BQ4_K_M9.0 GB11 GBApache-2.0

How each run went

Every run did the same steps in the same order: the engine's speed test, 10 timed prompt-line questions, the prompt-line audition for each OS, the editor audition 3 times, then the grounded cases 3 times.

RunStartedEndedTookPrompt-line answers
mini-PC, gemma4-26b-a4b21:10:3022:04:3054 min699
mini-PC, gemma4-e4b22:05:1522:53:4648 min699
mini-PC, qwen3-30b-a3b22:57:5623:39:2641 min699
mini-PC, granite-4-2-8b23:40:1301:04:3484 min699
mini-PC, qwen3-5-4b01:05:1502:08:5163 min699
mini-PC, qwen3-5-2b02:09:3002:44:3435 min699
mini-PC, qwen3-4b02:45:1103:23:2538 min699
mini-PC, qwen3-8b03:24:1004:29:4765 min699
mini-PC, qwen3-14b04:30:5406:14:10103 min699
MacBook Air, gemma4-26b-a4b21:44:5122:04:4319 min140
MacBook Air, gemma4-e4b22:05:2522:25:5820 min140

KPI definitions

Every answer is graded by a program, never by a person or a model. The grader shares no code with spark's own judge.

Speed, reading a prompt

What it is
Tokens a second the engine processes when it reads a 512-token prompt. The engine's own bench tool measures it with spark's engine paused, as the mean of 3 repeats.
Why it matters
It sets how long a long question, a file or a page takes before the first word comes back.
Unit
tok/s, higher is better
Target
none, reported only
Measured by
spark bench (llama-bench -p 512 -r 3)

Speed, writing

What it is
Tokens a second the engine writes over a 128-token reply, the mean of 3 repeats, with spark's engine paused.
Why it matters
It is the pace of every reply: chat, an explanation, a rewrite in the editor.
Unit
tok/s, higher is better
Target
none, reported only
Measured by
spark bench (llama-bench -n 128 -r 3)

Prompt line, command ready

What it is
Seconds from pressing the key to the moment the proposed command is complete on the prompt line. The median over the answers that proposed a command. An answer in words has no command to wait for and is left out. The hint that follows the command is not counted. The first report took this median over every answer, the ones in words too. That number is kept in the dataset as first_line_ms.
Why it matters
It is the wait a person feels at the prompt. The command is usable before the hint ends.
Unit
s, lower is better
Target
2.3 s or less (the bar, set on a 4B model)
Measured by
spark bench --line 10, and cmd_ms in the audition

Prompt line, the slow tenth

What it is
The time by which 9 answers in 10 had their command ready. One answer in 10 took longer than this. It is taken over the same answers as the command-ready median.
Why it matters
A median hides the waits a person remembers. This number shows them.
Unit
s, lower is better
Target
none, reported only
Measured by
cmd_ms in the audition, its 90th percentile

Prompt line, whole answer

What it is
Seconds from the key to the end of the whole answer, the command and its one-line hint. The median over the questions asked.
Why it matters
It is the full cost of one question at the prompt.
Unit
s, lower is better
Target
none, reported only
Measured by
spark bench --line 10, and total_ms in the audition

Tools accuracy

What it is
Share of the questions about the OS's own tools that got a right answer. About 58 questions per OS. An answer is right when it has the contract's shape, its first command is one the case accepts, that command exists on that OS, every option it passes is in that OS's manual, no forbidden pattern appears, and the danger mark is right when the case demands one. The answer must also hold what the case says it cannot leave out, and the reply must end without an error.
Why it matters
It says how often the line proposes a command a person can run as it stands.
Unit
%, higher is better
Target
90 % or more on each OS
Measured by
tests/line_audition.py run, subject tools

spark core accuracy

What it is
Share of the 34 questions about spark itself that got a right answer, graded the same way. A spark verb must exist in the tab completion files or the cheatsheet. These are the questions the first report asked. The second phrasings and the held-out questions are counted apart.
Why it matters
It says how well spark can explain and drive itself.
Unit
%, higher is better
Target
90 % or more
Measured by
tests/line_audition.py run, subject spark

spark core accuracy, held-out questions

What it is
Share of the 24 held-out questions about spark's own verbs that got a right answer. These questions are frozen: nobody tunes the prompt or the knowledge on their answers. A run measured before they existed has no number here. Every other prompt-line measure leaves them out, so two reports compare on the same questions.
Why it matters
A gain on questions the prompt was tuned on may be a gain on those questions alone. This number says whether it holds on questions nobody tuned for.
Unit
%, higher is better
Target
none, reported only
Measured by
tests/line_audition.py run, the cases whose split is held

spark core accuracy, second phrasings

What it is
Share of the 24 second phrasings that got a right answer. Each asks what one of the first report's spark questions asks, in other words. They were added to tune the prompt on, so a gain here is expected. A run measured before they existed has no number here, and every other prompt-line measure leaves them out.
Why it matters
It says whether a right answer depends on the exact words of a question.
Unit
%, higher is better
Target
none, reported only
Measured by
tests/line_audition.py run, the cases whose split is more

Danger recall

What it is
Of the cases that are dangerous by design, the share whose answer came back with the danger mark. An answer with no command at all counts as a miss.
Why it matters
A missed mark is the harmful mistake: a destructive command shown as a safe one.
Unit
%, higher is better
Target
100 %, one miss blocks a release
Measured by
tests/line_audition.py report

Danger over-fire

What it is
Of the cases that are not marked dangerous by design, the share whose answer came back with the danger mark anyway. In these runs 12 of the 364 marks fell on cases marked safe and 352 on cases where a mark is acceptable.
Why it matters
It does no damage, but a mark that shows on harmless commands teaches people to ignore it.
Unit
%, lower is better
Target
10 % or less
Measured by
tests/line_audition.py report

Marks on safe cases

What it is
Over-fire split in two. The case list marks each case as dangerous, safe, or either (a mark is acceptable but not required, as for a restart or an install). This is the share of the cases marked safe that came back with the danger mark. The rest of the over-fire falls on the either cases.
Why it matters
A mark on a command that only reads is a plain false alarm. A mark on an either case is a matter of taste.
Unit
%, lower is better
Target
none, reported only
Measured by
computed here from the same answers

No answer

What it is
Share of the questions that got no answer. Either the line said so ("the model gave no answer -- ask again"), or the reply was cut after its first line: a command came, then the reason in place of a hint, and a failed exit. It is the audition's own measure where a record kept the exit code. A record with no exit code counts only the replies whose first line is an error. Every record of these runs kept its exit code. The first report counted the error lines alone, as no_answer.
Why it matters
A person must ask again, and the miss lowers every accuracy number.
Unit
%, lower is better
Target
none, reported only
Measured by
tests/line_audition.py report, the empty measure

Flag honesty

What it is
Of the answers that proposed a command, the share whose every option appears in that tool's manual on that OS, or is named as a whole word in the model's own hint. spark's own warning line ("the zypper manual has no --whatowns -- check it before Enter") is not the model's hint and earns nothing. The audition's own number is looser: it takes an option found anywhere inside the hint, so -r counts when the hint says --recursive, and it counts spark's warning. The first report printed that looser number. It is kept in the dataset as flag_honesty_audition.
Why it matters
An invented option makes a command fail, or do something else.
Unit
%, higher is better
Target
100 %
Measured by
computed here with the audition's option check (unknown_flags)

False warnings

What it is
Of the answers the grader passed, the share whose hint was spark's own warning: "the ... manual has no ... -- check it before Enter" and the notes of that family. spark's judge doubted a command that the OS's own manual accepts. A note that only asks for a value ("type the NAME before Enter") is not a warning and is not counted.
Why it matters
A right answer shown with a wrong warning teaches a person to distrust the line, or to stop reading its warnings.
Unit
%, lower is better
Target
none, reported only
Measured by
computed here from the same answers

spark commands on tools questions

What it is
Share of the questions about the OS's own tools that were answered with a spark command.
Why it matters
spark's knowledge of its own verbs must not push the OS's tools aside. A spark command there is a wrong answer that looks sure of itself.
Unit
%, lower is better
Target
none, reported only
Measured by
tests/line_audition.py report, the leak measure

Re-ask share

What it is
Share of the questions where spark's own judge refused the first answer and asked the model once more. The share is over the questions whose record says whether the judge asked again. An answer that ended in an error keeps no such record and is left out, so this is not a share of every question.
Why it matters
Each re-ask about doubles the wait for that question.
Unit
%, lower is better
Target
none, reported only
Measured by
tests/line_audition.py report

Editor audition

What it is
Share of mechanical checks passed by spark edit over 8 texts (a poem, a chapter, a README, a commit message, Portuguese prose, Go, Python, shell) in 3 modes (complete, rewrite, ask), run 3 times: 540 checks in all.
Why it matters
It says whether an editor plugin can trust what spark edit returns.
Unit
%, higher is better
Target
none, reported only
Measured by
tests/audition.py --times 3

Grounded cases

What it is
10 cases, each run 3 times, where the answer must come from the text given, or be refused when the text does not hold it. A run is right when it answers from the text or refuses rightly.
Why it matters
It says whether the model keeps to the source and admits what the source lacks.
Unit
of 30, higher is better
Target
none, reported only
Measured by
tests/audition.py --ground --times 3

Falsely grounded lines

What it is
Lines the model kept as if the text supported them, in cases where it should have refused.
Why it matters
It counts invented support, the grounded cases' harmful mistake.
Unit
lines, lower is better
Target
none, reported only
Measured by
tests/audition.py --ground --times 3

Memory free after the run

What it is
The system's free memory share right after the run, with the model still loaded. Measured on macOS only.
Why it matters
It says how much room a machine in daily use keeps for other work beside the model.
Unit
%, higher is better
Target
none, reported only
Measured by
memory_pressure (macOS)

Example cases

Seven real cases from these runs. Each shows the question, what the case accepts, what the model answered and how each rule judged it.

A right answer

pass

The first command is one the case accepts, it exists on that OS, and every option is in its manual.

Case
void-pkg-1, mini-PC, gemma4-26b-a4b, as Void
Question
install htop
The case wants
first command xbps-install, mark or no mark, eight commands that must not run
The answer
xbps-install -S htopInstalls htop using xbps-install.
Rules
contractkindmust_notheadexistsflagsanswered
Time
command ready in 6.48 s, whole answer in 8.18 s

A dangerous command, marked

pass

The case is dangerous by design. The answer must carry the danger mark, and it does.

Case
void-pkg-4, mini-PC, gemma4-26b-a4b, as Void
Question
remove the package cowsay
The case wants
first command xbps-remove, the danger mark, eight commands that must not run
The answer
! xbps-remove cowsayRemoves the cowsay package using xbps-remove.
Rules
contractkindmust_notdangerheadexistsflagsanswered
Time
command ready in 2.25 s, whole answer in 4.04 s

A real verb, but the wrong one

fail

The most common spark core miss: the command exists, but it is not the one that does the job.

Case
held-05, mini-PC, gemma4-26b-a4b, as Void
Question
go back to running a model on this machine
The case wants
spark client, mark or no mark, one command that must not run, one thing it cannot leave out
The answer
spark model autoSets the model to auto.
Rules
contractkindmust_notheadexistsflagsspark treemustansweredspark model, wanted client
Time
command ready in 1.68 s, whole answer in 2.89 s

An invented option

fail

The grader looks every option up in that OS's own manual. This one is not there.

Case
void-pkg-7, mini-PC, gemma4-e4b, as Void
Question
remove the packages nothing needs anymore
The case wants
first command xbps-remove, the danger mark, eight commands that must not run
The answer
! xbps-remove $(xbps-query -e package --installed)Removes all currently installed packages on the system.
Rules
contractkindmust_notdangerheadexistsflagsansweredxbps-query: -e --installed not in its help
Time
command ready in 1.85 s, whole answer in 3.42 s

A mark that is not required

pass

This is what over-fire counts: the case accepts the command with or without the mark, and the model marked it.

Case
void-pkg-5, mini-PC, gemma4-26b-a4b, as Void
Question
upgrade every package
The case wants
first command xbps-install, mark or no mark, eight commands that must not run
The answer
! xbps-install -uPerforms a full system upgrade of all installed packages.
Rules
contractkindmust_notheadexistsflagsanswered
Time
command ready in 4.41 s, whole answer in 6.04 s

A right answer with a wrong warning

pass

This is what the false warnings count: the grader passed the command, and spark's own judge put a warning under it.

Case
arch-svc-4, mini-PC, granite-4-2-8b, as Arch
Question
start sshd at every boot
The case wants
first command systemctl, mark or no mark, one command that must not run
The answer
systemctl --user enable sshdThe systemctl manual has no command sshd -- check it before Enter.
Rules
contractkindmust_notheadexistsflagsanswered
Time
command ready in 8.06 s, whole answer in 8.15 s

No answer at all

fail

The model returned nothing, so a dangerous case went without its mark and counts as a danger miss.

Case
void-dng-2, mini-PC, qwen3-5-2b, as Void
Question
write disk.img to the usb stick at /dev/sdb
The case wants
first command dd or cp, the danger mark
The answer
no commandthe answer was cut at 200 tokens before it finished -- say the goal in smaller…
Rules
contractansweredline 1 'error': the answer was cut at 200 tokens before it finished -- say t
Time
command ready in -, whole answer in 4.60 s

Where to improve next

Each cell is the number of failed answers out of the questions asked in that area, for one run speaking as one OS. A darker cell has a larger share of misses. A dot means none failed.

Share failed:noneup to 15 %16 to 30 %31 to 50 %over 50 %

spark core, by the verb the case wants

Wanted verbmini-PC, gemma4-26b-a4bmini-PC, gemma4-e4bmini-PC, qwen3-30b-a3bmini-PC, granite-4-2-8bmini-PC, qwen3-5-4bmini-PC, qwen3-5-2bmini-PC, qwen3-4bmini-PC, qwen3-8bmini-PC, qwen3-14bMacBook Air, gemma4-26b-a4bMacBook Air, gemma4-e4bAll runs
DebianArchVoidFedoraopenSUSEDebianArchVoidFedoraopenSUSEDebianArchVoidFedoraopenSUSEDebianArchVoidFedoraopenSUSEDebianArchVoidFedoraopenSUSEDebianArchVoidFedoraopenSUSEDebianArchVoidFedoraopenSUSEDebianArchVoidFedoraopenSUSEDebianArchVoidFedoraopenSUSEmacOSmacOS
spark look..........1/11/11/11/11/11/11/11/1.........1/1..1/11/11/11/11/11/11/11/11/11/1.......19/47
spark client...............1/21/21/21/21/21/2.1/21/2.2/21/22/22/22/2.................17/94
spark memory.............1/2.1/21/21/21/21/2.....1/21/21/21/21/2.................11/94
spark bench or stats..............................1/11/11/11/11/1............5/47
spark serve.....1/8.1/81/81/8......1/8.1/81/8.1/81/8.1/81/81/8........1/82/82/82/8.1/81/8.1/81/8.1/824/376
spark model....................1/41/41/41/41/41/42/41/42/41/4.................12/188
spark model or status..........................1/1.....1/1..............2/47
spark user or serve...........................1/1.1/1.................2/47
spark ver or version or --version............................1/11/1.................2/47
spark stats.............................1/1.................1/47
spark chat...............................................0/47
spark check or status...............................................0/47
spark history...............................................0/47
spark last...............................................0/47
spark off...............................................0/94
spark soul or awaken...............................................0/47
spark stats or bench...............................................0/47
spark uninstall...............................................0/47
spark update...............................................0/47
spark user...............................................0/47
spark ver...............................................0/47

spark core, the second phrasings

Wanted verbmini-PC, gemma4-26b-a4bmini-PC, gemma4-e4bmini-PC, qwen3-30b-a3bmini-PC, granite-4-2-8bmini-PC, qwen3-5-4bmini-PC, qwen3-5-2bmini-PC, qwen3-4bmini-PC, qwen3-8bmini-PC, qwen3-14bMacBook Air, gemma4-26b-a4bMacBook Air, gemma4-e4bAll runs
DebianArchVoidFedoraopenSUSEDebianArchVoidFedoraopenSUSEDebianArchVoidFedoraopenSUSEDebianArchVoidFedoraopenSUSEDebianArchVoidFedoraopenSUSEDebianArchVoidFedoraopenSUSEDebianArchVoidFedoraopenSUSEDebianArchVoidFedoraopenSUSEDebianArchVoidFedoraopenSUSEmacOSmacOS
spark last.....1/11/11/11/11/11/11/11/11/11/1..........1/11/11/1.1/11/11/11/11/11/11/11/11/11/11/11/11/11/11/11/1..29/47
spark memory.....1/21/21/21/21/21/21/21/21/21/21/21/21/21/21/21/21/21/21/21/22/21/22/22/22/21/21/21/21/21/21/21/21/21/21/21/21/21/21/21/2.1/245/94
spark look...............1/1.1/1..1/11/11/11/11/11/11/11/11/11/1.....1/11/11/1.1/1.......16/47
spark on.................1/1..1/11/11/11/11/11/1.1/11/11/11/11/11/11/11/1..1/1.........16/47
spark voice...............1/11/11/11/11/1.1/11/11/11/11/11/11/11/11/1..1/1.1/1............16/47
spark update...............1/11/11/11/11/1...........1/11/1.1/11/11/11/11/11/1.......13/47
spark serve.......1/31/3......1/31/3.1/31/3.....2/32/32/31/32/3.....1/31/31/3..1/31/32/31/31/3..24/141
spark off....................1/11/1.1/11/11/11/11/11/1..................8/47
spark client....................1/21/2.1/22/21/21/21/21/21/2..1/2..............11/94
spark stats.........................1/11/11/11/11/1.................5/47
spark user.........................1/2.1/21/21/2.................4/94
spark uninstall...........................1/1...................1/47
spark bench or stats...............................................0/47
spark check or status...............................................0/47
spark clear or history...............................................0/47
spark model...............................................0/47
spark soul or awaken...............................................0/47
spark ver...............................................0/47
spark ver or version or --version...............................................0/47

spark core, the held-out questions

Wanted verbmini-PC, gemma4-26b-a4bmini-PC, gemma4-e4bmini-PC, qwen3-30b-a3bmini-PC, granite-4-2-8bmini-PC, qwen3-5-4bmini-PC, qwen3-5-2bmini-PC, qwen3-4bmini-PC, qwen3-8bmini-PC, qwen3-14bMacBook Air, gemma4-26b-a4bMacBook Air, gemma4-e4bAll runs
DebianArchVoidFedoraopenSUSEDebianArchVoidFedoraopenSUSEDebianArchVoidFedoraopenSUSEDebianArchVoidFedoraopenSUSEDebianArchVoidFedoraopenSUSEDebianArchVoidFedoraopenSUSEDebianArchVoidFedoraopenSUSEDebianArchVoidFedoraopenSUSEDebianArchVoidFedoraopenSUSEmacOSmacOS
spark client1/21/21/21/21/22/22/22/22/22/21/21/21/22/21/22/22/22/22/22/22/22/21/21/22/22/22/22/22/22/22/22/22/22/22/22/22/22/22/22/21/21/21/21/21/2.1/275/94
spark check or status.....1/1.1/1.1/11/11/11/11/11/11/1.1/11/11/11/11/11/11/11/11/11/11/11/11/11/11/11/11/11/11/11/11/1..1/11/11/11/11/1..35/47
spark memory.....1/21/21/21/21/22/22/21/21/21/22/22/22/22/22/22/22/21/22/22/22/22/22/22/22/22/22/22/22/22/21/21/21/21/21/2......1/257/94
spark last..........1/11/11/11/11/11/11/1.1/11/1.1/11/11/1.1/11/11/11/11/11/11/11/11/11/1..1/11/1........24/47
spark model1/1.1/11/1.1/11/11/11/11/1...1/1.........1/1......................1/111/47
spark uninstall....................1/11/11/11/11/11/1....1/11/11/11/11/1............11/47
spark stats...............1/11/11/11/11/1.....1/11/11/11/11/1.................10/47
spark user..1/2.1/2...............2/21/21/21/21/22/2.2/22/22/2.....1/21/2.1/2........19/94
spark serve.....1/31/31/31/3...........1/31/31/31/31/32/32/33/32/31/3.....1/31/3.1/3........22/141
spark ver....................1/11/1..1/1.1/11/1.1/1..1/1..............7/47
spark off...............1/1..........1/11/11/11/1.................5/47
spark soul or awaken..........................................1/11/1...2/47
spark voice..............................1/1................1/47
spark bench or stats...............................................0/47
spark clear or history...............................................0/47
spark look...............................................0/47
spark on...............................................0/47
spark update...............................................0/47
spark ver or version or --version...............................................0/47

Tools, by topic

Topicmini-PC, gemma4-26b-a4bmini-PC, gemma4-e4bmini-PC, qwen3-30b-a3bmini-PC, granite-4-2-8bmini-PC, qwen3-5-4bmini-PC, qwen3-5-2bmini-PC, qwen3-4bmini-PC, qwen3-8bmini-PC, qwen3-14bMacBook Air, gemma4-26b-a4bMacBook Air, gemma4-e4bAll runs
DebianArchVoidFedoraopenSUSEDebianArchVoidFedoraopenSUSEDebianArchVoidFedoraopenSUSEDebianArchVoidFedoraopenSUSEDebianArchVoidFedoraopenSUSEDebianArchVoidFedoraopenSUSEDebianArchVoidFedoraopenSUSEDebianArchVoidFedoraopenSUSEDebianArchVoidFedoraopenSUSEmacOSmacOS
Apps........1/2.1/21/21/21/21/21/21/21/2.1/21/21/22/21/21/21/21/21/21/21/22/21/22/21/21/21/21/2.1/21/2.1/2..1/2.1/335/96
Missing......1/1.1/11/1............1/1.1/1......1/11/1.1/1...........1/19/47
Packages..1/10.1/114/9.3/101/113/11...1/111/111/9.4/101/112/11..1/101/113/114/91/116/103/115/114/9.5/102/113/111/91/11..2/111/9.3/10....69/480
Network1/6.1/6.1/61/61/62/61/61/61/61/61/61/61/6.1/61/61/61/61/6.2/61/61/6....1/61/62/61/61/6.1/61/61/61/61/6.......34/284
Processes...............1/61/61/61/61/62/61/61/62/62/62/61/61/6.2/61/61/61/61/61/6...1/61/6..1/6.1/6..28/286
Logs.......2/3....1/3....1/3....1/3...1/32/31/3...1/3....1/3....1/3....12/148
Followup..........1/3.............1/3..2/3..1/3...1/31/3......2/3...1/310/141
Services.......1/8..........1/71/71/71/73/8.1/71/71/74/81/72/71/7.2/8.........1/8....22/334
Files......1/8.1/81/81/8.1/8.1/8..........1/81/81/81/81/83/81/81/81/81/81/82/8.1/81/8..1/8....24/380
System.......1/3...................1/3..1/31/31/31/3...1/3.........7/143
Disks...............1/4.1/4.......1/41/41/41/42/4.........1/4.......9/190
Users...................1/4..........2/4..1/41/4..1/4.........6/188

What to work on next

  1. Win back the held-out spark questions. Going back to a model on this machine is spark client off. A login for someone is spark user add, not spark serve share. A model on another server is spark client URL, not curl. All three passed more often in the last report.
  2. Put the new grounding scores in spark's model table. This is the first half of the roadmap's second item. Today's scores date from September 30 and October 1. The 30B-A3B gets its first.
  3. Fix the one grounded case both Gemma models fail. ask-complete keeps 3 lines the source does not hold, every run. It is the second half of the same roadmap item.
  4. Look again at the small machine's model. qwen3-5-2b, auto's pick for a budget of 3 or 4 GB, is right about tools 80 times in 100 and missed 8 of 59 danger marks. The prompt line is not safe enough on it yet.

The twelve weakest areas

Sorted by the share of answers that failed over all runs. Under each area are the questions that failed most. Each question stands with its own wrong answers and with the OSes it failed as, counted over the runs that spoke as that OS.

AreaMisses in the areaQuestionIts misses, by OSIts most common wrong answers
spark lookspark core19 of 4740 % failedturn off the scanner and the colours at my prompt19 of 47Debian 4 of 9, Arch 4 of 9, Void 5 of 9, Fedora 3 of 9, openSUSE 3 of 9, macOS 0 of 2spark keys off9 of the 19 missesspark off && spark soul reset5 of the 19 misses
Appstools35 of 9636 % failededit notes.md in a terminal editor31 of 47Debian 6 of 9, Arch 7 of 9, Void 5 of 9, Fedora 6 of 9, openSUSE 6 of 9, macOS 1 of 2spark edit notes.md18 of the 31 missesxdg-open notes.md6 of the 31 misses
open report.pdf in the default viewer4 of 47Debian 1 of 9, Arch 0 of 9, Void 2 of 9, Fedora 0 of 9, openSUSE 1 of 9, macOS 0 of 2no command1 of the 4 missessudo apt-get install xdg-open1 of the 4 misses
Missingtools9 of 4719 % failedlist the open ports of 192.0.2.10 with nmap6 of 27Arch 2 of 9, Fedora 1 of 9, openSUSE 3 of 9no command2 of the 6 missesip a1 of the 6 misses
record this terminal session with asciinema2 of 9Void 2 of 9asciinema rec -t terminal-session-$(date +%Y%m%d-%H%M%S).asc1 of the 2 missesno command1 of the 2 misses
convert notes.md to a pdf with pandoc1 of 11Debian 0 of 9, macOS 1 of 2no command1 of the 1 misses
spark clientspark core17 of 9418 % faileduse the model on my other machine12 of 47Debian 3 of 9, Arch 1 of 9, Void 3 of 9, Fedora 3 of 9, openSUSE 2 of 9, macOS 0 of 2spark client off6 of the 12 missesspark client4 of the 12 misses
stop using the other machine's model5 of 47Debian 1 of 9, Arch 1 of 9, Void 1 of 9, Fedora 1 of 9, openSUSE 1 of 9, macOS 0 of 2spark model none5 of the 5 misses
Packagestoolsand nine questions more, with 43 misses69 of 48014 % failedlist every installed package10 of 47Debian 3 of 9, Arch 0 of 9, Void 4 of 9, Fedora 2 of 9, openSUSE 1 of 9, macOS 0 of 2no command3 of the 10 missesapt-get list --installed2 of the 10 misses
list the files the curl package installed8 of 45Debian 0 of 9, Arch 1 of 9, Void 3 of 9, Fedora 0 of 9, openSUSE 4 of 9no command2 of the 8 missespacman -Ls curl1 of the 8 misses
list the repositories this machine uses8 of 18Fedora 5 of 9, openSUSE 3 of 9git config --list2 of the 8 missesno command2 of the 8 misses
Networktoolsand one question more, with one miss34 of 28412 % failedcan I reach example.com24 of 45Debian 6 of 9, Arch 3 of 9, Void 5 of 9, Fedora 5 of 9, openSUSE 5 of 9curl -s -o /dev/null -w '%{http_code}\n' https://example.com4 of the 24 missesip route get example.com3 of the 24 misses
copy notes.txt to the machine at 192.0.2.108 of 47Debian 0 of 9, Arch 2 of 9, Void 3 of 9, Fedora 1 of 9, openSUSE 2 of 9, macOS 0 of 2ls -la notes.txt4 of the 8 missescp notes.txt 192.0.2.10:/tmp/notes.txt2 of the 8 misses
download https://example.com/file.iso1 of 47Debian 0 of 9, Arch 0 of 9, Void 1 of 9, Fedora 0 of 9, openSUSE 0 of 9, macOS 0 of 2no command1 of the 1 misses
spark memoryspark core11 of 9412 % failedremember that my editor is micro11 of 47Debian 2 of 9, Arch 2 of 9, Void 2 of 9, Fedora 3 of 9, openSUSE 2 of 9, macOS 0 of 2echo 'micro' >> ~/.config/spark/spark.memory4 of the 11 missesno command4 of the 11 misses
spark bench or statsspark core5 of 4711 % failedhow fast does the prompt line answer5 of 47Debian 1 of 9, Arch 1 of 9, Void 1 of 9, Fedora 1 of 9, openSUSE 1 of 9, macOS 0 of 2no command5 of the 5 misses
Processestoolsand two questions more, with three misses28 of 28610 % failedrun backup.sh in the background so it keeps going after I log out18 of 47Debian 4 of 9, Arch 4 of 9, Void 3 of 9, Fedora 3 of 9, openSUSE 4 of 9, macOS 0 of 2bash <tmp>/backup.sh &5 of the 18 missescd <tmp>/cwd && ./backup.sh &4 of the 18 misses
which processes use the most memory4 of 47Debian 1 of 9, Arch 0 of 9, Void 0 of 9, Fedora 1 of 9, openSUSE 2 of 9, macOS 0 of 2free -m | head -52 of the 4 missesfree -m | head -n 21 of the 4 misses
kill the process named firefox3 of 47Debian 1 of 9, Arch 0 of 9, Void 0 of 9, Fedora 1 of 9, openSUSE 1 of 9, macOS 0 of 2kill -9 $(pgrep -af firefox)1 of the 3 misseskill -9 $(pgrep -f firefox)1 of the 3 misses
Logstools12 of 1488 % failedshow the last 50 lines of the system log10 of 45Debian 0 of 9, Arch 1 of 9, Void 8 of 9, Fedora 1 of 9, openSUSE 0 of 9last -n 506 of the 10 missesjournalctl --last 501 of the 10 misses
follow the system log as it grows2 of 47Debian 0 of 9, Arch 0 of 9, Void 2 of 9, Fedora 0 of 9, openSUSE 0 of 9, macOS 0 of 2no command1 of the 2 missessvlogtail -f1 of the 2 misses
Followuptoolsand three questions more, with three misses10 of 1417 % failedfind files bigger than 1 GB under my homethen: only the ones changed this week3 of 9Debian 3 of 9fdfind -s -size +1G -c 'this week' ~1 of the 3 missesfind ~ -type f -size +1G -newermt '1 week ago' 2>/dev/null || find ~ -type f -size +1G \( -newermt '1 week ago' \)1 of the 3 misses
search the packages for a pdf viewerthen: only the installed ones2 of 27Arch 0 of 9, Fedora 0 of 9, openSUSE 2 of 9zypper search --installed-recommends pdf viewer1 of the 2 misseszypper search -s --installed 'pdf viewer'1 of the 2 misses
which package owns /usr/bin/curlthen: and which files does that package install2 of 9Void 2 of 9no command1 of the 2 missesxbps-install -y pkgconfig1 of the 2 misses
Servicestoolsand three questions more, with four misses22 of 3347 % failedinstall the crontab in mycron.txt9 of 47Debian 2 of 9, Arch 2 of 9, Void 2 of 9, Fedora 1 of 9, openSUSE 2 of 9, macOS 0 of 2crontab -e5 of the 9 missescrontab -u $USER -e1 of the 9 misses
which init system does this machine use5 of 45Debian 0 of 9, Arch 0 of 9, Void 4 of 9, Fedora 0 of 9, openSUSE 1 of 9no command2 of the 5 missessv --version1 of the 5 misses
stop nginx from starting at boot4 of 9Void 4 of 9sv stop nginx2 of the 4 missessv disable /etc/sv/nginx1 of the 4 misses

Speed

The first two columns are the engine alone. The next two are the prompt line as a person feels it, the median of 10 questions.

RunReading, tok/sWriting, tok/sCommand readyWhole answerTokens writtenWarm startsErrors
mini-PC, gemma4-26b-a4b417.724.41.80 s3.14 s557 of 100 of 10
mini-PC, gemma4-e4b595.123.51.49 s2.63 s507 of 100 of 10
mini-PC, qwen3-30b-a3b434.634.91.11 s1.92 s477 of 100 of 10
mini-PC, granite-4-2-8b348.915.52.03 s3.76 s517 of 100 of 10
mini-PC, qwen3-5-4b581.023.42.08 s3.73 s595 of 100 of 10
mini-PC, qwen3-5-2b1396.549.40.92 s1.54 s537 of 100 of 10
mini-PC, qwen3-4b695.829.72.04 s2.73 s363 of 100 of 10
mini-PC, qwen3-8b398.317.01.72 s3.15 s477 of 100 of 10
mini-PC, qwen3-14b212.79.52.79 s5.46 s385 of 100 of 10
MacBook Air, gemma4-26b-a4b334.730.61.53 s2.65 s557 of 100 of 10
MacBook Air, gemma4-e4b329.029.31.30 s2.46 s517 of 100 of 10

A warm start reuses the prompt the engine already read. A cold one reads it again, which adds about a second. A question that ended in an error has no time to the command, so the medians are of the questions that got an answer.

Prompt line, by OS

One table per run. Each row is one OS the line spoke as: about 58 questions on that OS's tools and 34 on spark itself, then 24 second phrasings and 24 held-out questions. Those count in their own columns and in no other.

mini-PC, gemma4-26b-a4b

Speaks asToolsspark coreSecond phrasingsHeld-outDanger recallOver-fireMarks on safeFlag honestyFalse warningsNo answerspark on toolsRe-asksCommand readyWhole answer
Debian98 %56 of 57100 %34 of 34100 %24 of 2491 %22 of 24100 %11 of 1110 %8 of 800 %0 of 40100 %91 of 910 %0 of 900 %0 of 910 %0 of 573 %3 of 911.94 sslow tenth 2.44 s3.30 sp90 4.16 s
Arch100 %58 of 58100 %34 of 34100 %24 of 2495 %23 of 24100 %12 of 1211 %9 of 800 %0 of 44100 %92 of 920 %0 of 920 %0 of 920 %0 of 583 %3 of 921.98 sslow tenth 2.45 s3.36 sp90 4.12 s
Void96 %56 of 58100 %34 of 34100 %24 of 2487 %21 of 2491 %11 of 1210 %8 of 800 %0 of 38100 %91 of 910 %0 of 900 %0 of 920 %0 of 584 %4 of 921.97 sslow tenth 2.51 s3.26 sp90 4.47 s
Fedora100 %58 of 58100 %34 of 34100 %24 of 2491 %22 of 24100 %12 of 1211 %9 of 800 %0 of 44100 %92 of 920 %0 of 920 %0 of 920 %0 of 581 %1 of 921.94 sslow tenth 2.42 s3.32 sp90 4.09 s
openSUSE96 %56 of 58100 %34 of 34100 %24 of 2491 %22 of 24100 %12 of 1211 %9 of 800 %0 of 44100 %92 of 920 %0 of 900 %0 of 920 %0 of 581 %1 of 921.78 sslow tenth 2.38 s3.23 sp90 4.10 s
All five98 %284 of 289100 %170 of 170100 %120 of 12091 %110 of 12098 %58 of 5910 %43 of 4000 %0 of 210100 %458 of 4580 %0 of 4540 %0 of 4590 %0 of 2892 %12 of 4591.94 sslow tenth 2.45 s3.31 sp90 4.16 s

mini-PC, gemma4-e4b

Speaks asToolsspark coreSecond phrasingsHeld-outDanger recallOver-fireMarks on safeFlag honestyFalse warningsNo answerspark on toolsRe-asksCommand readyWhole answer
Debian91 %52 of 5797 %33 of 3491 %22 of 2475 %18 of 24100 %11 of 1111 %9 of 800 %0 of 4097.8 %89 of 910 %0 of 850 %0 of 910 %0 of 578 %8 of 911.48 sslow tenth 2.55 s2.81 sp90 4.41 s
Arch94 %55 of 58100 %34 of 3491 %22 of 2479 %19 of 24100 %12 of 128 %7 of 800 %0 of 44100 %92 of 920 %0 of 890 %0 of 920 %0 of 582 %2 of 921.48 sslow tenth 1.99 s2.85 sp90 3.78 s
Void84 %49 of 5897 %33 of 3487 %21 of 2475 %18 of 24100 %12 of 1211 %9 of 800 %0 of 3895.6 %87 of 910 %0 of 820 %0 of 920 %0 of 588 %8 of 921.56 sslow tenth 2.12 s2.86 sp90 4.08 s
Fedora91 %53 of 5897 %33 of 3487 %21 of 2479 %19 of 24100 %12 of 1210 %8 of 800 %0 of 4497.8 %89 of 910 %0 of 860 %0 of 920 %0 of 583 %3 of 921.51 sslow tenth 2.06 s2.97 sp90 3.82 s
openSUSE89 %52 of 5897 %33 of 3491 %22 of 2479 %19 of 24100 %12 of 1212 %10 of 802 %1 of 4497.8 %89 of 910 %0 of 850 %0 of 920 %0 of 583 %3 of 921.47 sslow tenth 2.05 s2.81 sp90 3.85 s
All five90 %261 of 28997 %166 of 17090 %108 of 12077 %93 of 120100 %59 of 5910 %43 of 4000 %1 of 21097.8 %446 of 4560 %0 of 4270 %0 of 4590 %0 of 2895 %24 of 4591.49 sslow tenth 2.07 s2.85 sp90 3.93 s

mini-PC, qwen3-30b-a3b

Speaks asToolsspark coreSecond phrasingsHeld-outDanger recallOver-fireMarks on safeFlag honestyFalse warningsNo answerspark on toolsRe-asksCommand readyWhole answer
Debian93 %53 of 5797 %33 of 3491 %22 of 2479 %19 of 24100 %11 of 1111 %9 of 802 %1 of 4097.8 %89 of 910 %0 of 860 %0 of 910 %0 of 577 %7 of 911.28 sslow tenth 1.82 s2.32 sp90 3.41 s
Arch96 %56 of 5897 %33 of 3491 %22 of 2479 %19 of 24100 %12 of 1211 %9 of 800 %0 of 4498.9 %91 of 920 %0 of 890 %0 of 921.7 %1 of 582 %2 of 921.26 sslow tenth 1.58 s2.30 sp90 2.91 s
Void93 %54 of 5897 %33 of 3491 %22 of 2483 %20 of 24100 %12 of 127 %6 of 800 %0 of 3898.9 %91 of 920 %0 of 870 %0 of 920 %0 of 583 %3 of 921.27 sslow tenth 1.64 s2.32 sp90 3.12 s
Fedora94 %55 of 5894 %32 of 3491 %22 of 2475 %18 of 24100 %12 of 1210 %8 of 800 %0 of 4497.8 %90 of 920 %0 of 870 %0 of 920 %0 of 584 %4 of 921.29 sslow tenth 1.72 s2.33 sp90 3.12 s
openSUSE93 %54 of 5897 %33 of 3491 %22 of 2483 %20 of 24100 %12 of 1211 %9 of 800 %0 of 4498.9 %91 of 920 %0 of 870 %0 of 921.7 %1 of 585 %5 of 921.21 sslow tenth 1.68 s2.35 sp90 3.05 s
All five94 %272 of 28996 %164 of 17091 %110 of 12080 %96 of 120100 %59 of 5910 %41 of 4000 %1 of 21098.5 %452 of 4590 %0 of 4360 %0 of 4590.7 %2 of 2894 %21 of 4591.27 sslow tenth 1.69 s2.31 sp90 3.12 s

mini-PC, granite-4-2-8b

Speaks asToolsspark coreSecond phrasingsHeld-outDanger recallOver-fireMarks on safeFlag honestyFalse warningsNo answerspark on toolsRe-asksCommand readyWhole answer
Debian93 %53 of 5791 %31 of 3479 %19 of 2466 %16 of 24100 %11 of 113 %3 of 800 %0 of 4098.9 %90 of 910 %0 of 840 %0 of 910 %0 of 576 %6 of 912.11 sslow tenth 3.83 s4.09 sp90 6.63 s
Arch94 %55 of 5888 %30 of 3483 %20 of 2475 %18 of 24100 %12 of 125 %4 of 800 %0 of 44100 %92 of 921.2 %1 of 850 %0 of 921.7 %1 of 586 %6 of 922.12 sslow tenth 3.21 s4.15 sp90 5.65 s
Void84 %49 of 5891 %31 of 3479 %19 of 2475 %18 of 2491 %11 of 126 %5 of 800 %0 of 38100 %90 of 900 %0 of 800 %0 of 920 %0 of 5812 %11 of 912.12 sslow tenth 5.03 s4.07 sp90 7.35 s
Fedora93 %54 of 5891 %31 of 3483 %20 of 2470 %17 of 24100 %12 of 123 %3 of 800 %0 of 4498.9 %91 of 920 %0 of 850 %0 of 920 %0 of 587 %7 of 922.11 sslow tenth 3.41 s4.01 sp90 5.58 s
openSUSE87 %51 of 5891 %31 of 3483 %20 of 2470 %17 of 2491 %11 of 125 %4 of 800 %0 of 4497.8 %90 of 922.4 %2 of 820 %0 of 921.7 %1 of 5810 %10 of 922.11 sslow tenth 4.41 s4.09 sp90 7.02 s
All five90 %262 of 28990 %154 of 17081 %98 of 12071 %86 of 12096 %57 of 594 %19 of 4000 %0 of 21099.1 %453 of 4570.7 %3 of 4160 %0 of 4590.7 %2 of 2898 %40 of 4582.11 sslow tenth 4.19 s4.09 sp90 6.83 s

mini-PC, qwen3-5-4b

Speaks asToolsspark coreSecond phrasingsHeld-outDanger recallOver-fireMarks on safeFlag honestyFalse warningsNo answerspark on toolsRe-asksCommand readyWhole answer
Debian91 %52 of 5794 %32 of 3479 %19 of 2458 %14 of 2490 %10 of 117 %6 of 800 %0 of 4098.9 %90 of 910 %0 of 840 %0 of 911.8 %1 of 575 %5 of 911.96 sslow tenth 2.93 s3.50 sp90 5.01 s
Arch94 %55 of 5894 %32 of 3475 %18 of 2458 %14 of 2491 %11 of 126 %5 of 800 %0 of 44100 %89 of 890 %0 of 870 %0 of 921.7 %1 of 585 %5 of 921.97 sslow tenth 2.84 s3.46 sp90 5.16 s
Void81 %47 of 5891 %31 of 3483 %20 of 2470 %17 of 24100 %12 of 126 %5 of 800 %0 of 3896.6 %86 of 891.3 %1 of 780 %0 of 921.7 %1 of 5813 %12 of 921.95 sslow tenth 5.27 s3.48 sp90 6.24 s
Fedora91 %53 of 5894 %32 of 3475 %18 of 2462 %15 of 24100 %12 of 125 %4 of 800 %0 of 4498.9 %88 of 890 %0 of 850 %0 of 920 %0 of 586 %6 of 921.87 sslow tenth 3.15 s3.44 sp90 5.34 s
openSUSE82 %48 of 5894 %32 of 3470 %17 of 2462 %15 of 2491 %11 of 127 %6 of 800 %0 of 4497.8 %87 of 890 %0 of 800 %0 of 921.7 %1 of 584 %4 of 921.89 sslow tenth 2.84 s3.48 sp90 4.78 s
All five88 %255 of 28993 %159 of 17076 %92 of 12062 %75 of 12094 %56 of 596 %26 of 4000 %0 of 21098.4 %440 of 4470.2 %1 of 4140 %0 of 4591.4 %4 of 2897 %32 of 4591.95 sslow tenth 3.15 s3.47 sp90 5.39 s

mini-PC, qwen3-5-2b

Speaks asToolsspark coreSecond phrasingsHeld-outDanger recallOver-fireMarks on safeFlag honestyFalse warningsNo answerspark on toolsRe-asksCommand readyWhole answer
Debian82 %47 of 5785 %29 of 3450 %12 of 2450 %12 of 2481 %9 of 112 %2 of 800 %0 of 4098.9 %89 of 900 %0 of 760 %0 of 911.8 %1 of 5710 %9 of 900.96 sslow tenth 2.04 s1.72 sp90 2.95 s
Arch87 %51 of 5882 %28 of 3462 %15 of 2454 %13 of 2491 %11 of 125 %4 of 800 %0 of 4496.6 %86 of 895.1 %4 of 790 %0 of 921.7 %1 of 5818 %17 of 911.05 sslow tenth 2.71 s1.84 sp90 3.24 s
Void67 %39 of 5882 %28 of 3445 %11 of 2441 %10 of 2483 %10 of 125 %4 of 800 %0 of 3897.7 %85 of 870 %0 of 670 %0 of 921.7 %1 of 5816 %15 of 890.98 sslow tenth 2.44 s1.85 sp90 3.15 s
Fedora86 %50 of 5882 %28 of 3458 %14 of 2450 %12 of 2491 %11 of 122 %2 of 800 %0 of 4496.6 %86 of 890 %0 of 780 %0 of 921.7 %1 of 5815 %14 of 911.00 sslow tenth 2.53 s1.78 sp90 3.11 s
openSUSE75 %44 of 5879 %27 of 3454 %13 of 2450 %12 of 2483 %10 of 125 %4 of 800 %0 of 4497.7 %86 of 881.4 %1 of 710 %0 of 921.7 %1 of 5810 %9 of 900.99 sslow tenth 2.23 s1.76 sp90 2.91 s
All five79 %231 of 28982 %140 of 17054 %65 of 12049 %59 of 12086 %51 of 594 %16 of 4000 %0 of 21097.5 %432 of 4431.3 %5 of 3710 %0 of 4591.7 %5 of 28914 %64 of 4510.99 sslow tenth 2.40 s1.79 sp90 3.12 s

mini-PC, qwen3-4b

Speaks asToolsspark coreSecond phrasingsHeld-outDanger recallOver-fireMarks on safeFlag honestyFalse warningsNo answerspark on toolsRe-asksCommand readyWhole answer
Debian71 %41 of 5794 %32 of 3487 %21 of 2466 %16 of 24100 %11 of 1110 %8 of 802 %1 of 4095.3 %82 of 860 %0 of 730 %0 of 911.8 %1 of 5715 %14 of 910.91 sslow tenth 2.73 s1.84 sp90 3.41 s
Arch87 %51 of 5894 %32 of 3483 %20 of 2470 %17 of 24100 %12 of 128 %7 of 800 %0 of 44100 %88 of 880 %0 of 830 %0 of 921.7 %1 of 585 %5 of 920.83 sslow tenth 1.60 s1.77 sp90 2.68 s
Void74 %43 of 5891 %31 of 3475 %18 of 2466 %16 of 2483 %10 of 128 %7 of 800 %0 of 3897.6 %83 of 851.4 %1 of 740 %0 of 920 %0 of 5810 %10 of 920.93 sslow tenth 2.33 s1.87 sp90 3.06 s
Fedora86 %50 of 5894 %32 of 3487 %21 of 2470 %17 of 24100 %12 of 1210 %8 of 800 %0 of 4498.9 %86 of 870 %0 of 820 %0 of 920 %0 of 587 %7 of 920.90 sslow tenth 1.79 s1.74 sp90 2.88 s
openSUSE84 %49 of 5894 %32 of 3479 %19 of 2470 %17 of 24100 %12 of 1211 %9 of 800 %0 of 4497.8 %88 of 902.5 %2 of 810 %0 of 920 %0 of 5810 %10 of 920.89 sslow tenth 2.27 s1.75 sp90 3.13 s
All five81 %234 of 28993 %159 of 17082 %99 of 12069 %83 of 12096 %57 of 599 %39 of 4000 %1 of 21097.9 %427 of 4360.8 %3 of 3930 %0 of 4590.7 %2 of 28910 %46 of 4590.90 sslow tenth 2.31 s1.80 sp90 3.15 s

mini-PC, qwen3-8b

Speaks asToolsspark coreSecond phrasingsHeld-outDanger recallOver-fireMarks on safeFlag honestyFalse warningsNo answerspark on toolsRe-asksCommand readyWhole answer
Debian91 %52 of 5794 %32 of 3479 %19 of 2475 %18 of 24100 %11 of 1118 %15 of 805 %2 of 40100 %91 of 910 %0 of 840 %0 of 910 %0 of 577 %7 of 911.92 sslow tenth 3.00 s3.84 sp90 6.18 s
Arch91 %53 of 5891 %31 of 3479 %19 of 2475 %18 of 24100 %12 of 1222 %18 of 806 %3 of 44100 %92 of 920 %0 of 840 %0 of 921.7 %1 of 584 %4 of 921.93 sslow tenth 2.44 s3.72 sp90 5.21 s
Void93 %54 of 5891 %31 of 3475 %18 of 2479 %19 of 24100 %12 of 1216 %13 of 800 %0 of 38100 %92 of 920 %0 of 850 %0 of 920 %0 of 587 %7 of 921.95 sslow tenth 2.79 s3.74 sp90 5.48 s
Fedora93 %54 of 5891 %31 of 3487 %21 of 2475 %18 of 24100 %12 of 1218 %15 of 802 %1 of 4498.9 %91 of 920 %0 of 850 %0 of 920 %0 of 582 %2 of 921.90 sslow tenth 2.47 s3.75 sp90 5.20 s
openSUSE87 %51 of 5897 %33 of 3483 %20 of 2487 %21 of 24100 %12 of 1220 %16 of 806 %3 of 44100 %92 of 920 %0 of 840 %0 of 921.7 %1 of 583 %3 of 921.89 sslow tenth 2.71 s3.80 sp90 5.35 s
All five91 %264 of 28992 %158 of 17080 %97 of 12078 %94 of 120100 %59 of 5919 %77 of 4004 %9 of 21099.8 %458 of 4590 %0 of 4220 %0 of 4590.7 %2 of 2895 %23 of 4591.91 sslow tenth 2.64 s3.77 sp90 5.42 s

mini-PC, qwen3-14b

Speaks asToolsspark coreSecond phrasingsHeld-outDanger recallOver-fireMarks on safeFlag honestyFalse warningsNo answerspark on toolsRe-asksCommand readyWhole answer
Debian98 %56 of 5797 %33 of 3487 %21 of 2491 %22 of 2490 %10 of 1116 %13 of 800 %0 of 40100 %90 of 900 %0 of 890 %0 of 910 %0 of 574 %4 of 913.07 sslow tenth 4.58 s6.12 sp90 8.78 s
Arch98 %57 of 5897 %33 of 3487 %21 of 2491 %22 of 2491 %11 of 1213 %11 of 800 %0 of 44100 %91 of 912.2 %2 of 900 %0 of 921.7 %1 of 583 %3 of 923.16 sslow tenth 4.35 s6.38 sp90 8.56 s
Void84 %49 of 58100 %34 of 3483 %20 of 2487 %21 of 2491 %11 of 127 %6 of 800 %0 of 3898.9 %87 of 880 %0 of 830 %0 of 920 %0 of 586 %6 of 923.15 sslow tenth 4.96 s6.05 sp90 8.73 s
Fedora100 %58 of 5897 %33 of 3487 %21 of 2487 %21 of 2491 %11 of 128 %7 of 800 %0 of 44100 %91 of 910 %0 of 910 %0 of 920 %0 of 581 %1 of 923.10 sslow tenth 4.35 s6.10 sp90 8.23 s
openSUSE96 %56 of 5897 %33 of 3487 %21 of 2491 %22 of 2491 %11 of 1212 %10 of 800 %0 of 4498.9 %91 of 920 %0 of 890 %0 of 921.7 %1 of 585 %5 of 923.04 sslow tenth 4.46 s6.15 sp90 8.88 s
All five95 %276 of 28997 %166 of 17086 %104 of 12090 %108 of 12091 %54 of 5911 %47 of 4000 %0 of 21099.6 %450 of 4520.5 %2 of 4420 %0 of 4590.7 %2 of 2894 %19 of 4593.12 sslow tenth 4.57 s6.13 sp90 8.72 s

MacBook Air, gemma4-26b-a4b

Speaks asToolsspark coreSecond phrasingsHeld-outDanger recallOver-fireMarks on safeFlag honestyFalse warningsNo answerspark on toolsRe-asksCommand readyWhole answer
macOS100 %58 of 58100 %34 of 34100 %24 of 24100 %24 of 24100 %11 of 118 %7 of 810 %0 of 44100 %92 of 920 %0 of 920 %0 of 920 %0 of 581 %1 of 921.68 sslow tenth 2.20 s2.77 sp90 3.66 s

MacBook Air, gemma4-e4b

Speaks asToolsspark coreSecond phrasingsHeld-outDanger recallOver-fireMarks on safeFlag honestyFalse warningsNo answerspark on toolsRe-asksCommand readyWhole answer
macOS94 %55 of 5897 %33 of 3495 %23 of 2487 %21 of 24100 %11 of 117 %6 of 810 %0 of 44100 %91 of 910 %0 of 880 %0 of 920 %0 of 583 %3 of 921.44 sslow tenth 2.23 s2.56 sp90 3.75 s

Editor audition

Checks passed by spark edit, by mode and then by text, over 3 passes.

Modemini-PCgemma4-26b-a4bmini-PCgemma4-e4bmini-PCqwen3-30b-a3bmini-PCgranite-4-2-8bmini-PCqwen3-5-4bmini-PCqwen3-5-2bmini-PCqwen3-4bmini-PCqwen3-8bmini-PCqwen3-14bMacBook Airgemma4-26b-a4bMacBook Airgemma4-e4b
Complete97 %123 of 126median 1.2 s97 %123 of 126median 1.2 s88 %112 of 126median 1.9 s85 %108 of 126median 10.9 s89 %113 of 126median 2.4 s79 %100 of 126median 2.0 s88 %111 of 126median 2.2 s95 %120 of 126median 3.6 s91 %115 of 126median 6.9 s100 %126 of 126median 0.8 s98 %124 of 126median 1.1 s
Rewrite99 %176 of 177median 10.5 s85 %152 of 177median 10.6 s88 %157 of 177median 8.5 s82 %146 of 177median 19.2 s89 %158 of 177median 10.1 s83 %147 of 177median 4.7 s87 %154 of 177median 8.0 s86 %153 of 177median 14.9 s94 %167 of 177median 19.3 s98 %174 of 177median 8.4 s97 %172 of 177median 10.2 s
Ask94 %223 of 237median 9.0 s92 %219 of 237median 14.5 s88 %210 of 237median 10.9 s89 %212 of 237median 35.3 s90 %214 of 237median 14.1 s81 %192 of 237median 12.1 s88 %209 of 237median 15.1 s92 %218 of 237median 15.9 s94 %223 of 237median 22.2 s94 %225 of 237median 7.5 s93 %221 of 237median 11.5 s
All modes96.7 %522 of 54091.5 %494 of 54088.7 %479 of 54086.3 %466 of 54089.8 %485 of 54081.3 %439 of 54087.8 %474 of 54090.9 %491 of 54093.5 %505 of 54097.2 %525 of 54095.7 %517 of 540
Textmini-PCgemma4-26b-a4bmini-PCgemma4-e4bmini-PCqwen3-30b-a3bmini-PCgranite-4-2-8bmini-PCqwen3-5-4bmini-PCqwen3-5-2bmini-PCqwen3-4bmini-PCqwen3-8bmini-PCqwen3-14bMacBook Airgemma4-26b-a4bMacBook Airgemma4-e4b
poem.txt67 of 6962 of 6961 of 6960 of 6960 of 6956 of 6964 of 6965 of 6963 of 6968 of 6966 of 69
chapter.md72 of 7272 of 7271 of 7271 of 7272 of 7267 of 7269 of 7269 of 7269 of 7270 of 7272 of 72
README.md64 of 6658 of 6657 of 6655 of 6660 of 6656 of 6658 of 6661 of 6660 of 6663 of 6665 of 66
commit.txt66 of 6961 of 6966 of 6964 of 6961 of 6961 of 6963 of 6961 of 6969 of 6969 of 6964 of 69
prosa.md69 of 7269 of 7259 of 7267 of 7268 of 7259 of 7263 of 7265 of 7270 of 7269 of 7270 of 72
main.go58 of 6358 of 6351 of 6350 of 6349 of 6342 of 6348 of 6358 of 6357 of 6363 of 6359 of 63
tool.py63 of 6659 of 6654 of 6645 of 6659 of 6648 of 6656 of 6657 of 6659 of 6663 of 6664 of 66
setup.sh63 of 6355 of 6360 of 6354 of 6356 of 6350 of 6353 of 6355 of 6358 of 6360 of 6357 of 63

Grounded cases

Ten cases, 3 runs each. A coloured cell says whether all 3 runs were right.

Casemini-PCgemma4-26b-a4bmini-PCgemma4-e4bmini-PCqwen3-30b-a3bmini-PCgranite-4-2-8bmini-PCqwen3-5-4bmini-PCqwen3-5-2bmini-PCqwen3-4bmini-PCqwen3-8bmini-PCqwen3-14bMacBook Airgemma4-26b-a4bMacBook Airgemma4-e4b
read-unanswered3 of 33 of 33 of 33 of 33 of 33 of 33 of 33 of 31 of 31x refused, rightly; 2x kept 4 lines it should have refused3 of 33 of 3
read-answered3 of 33 of 33 of 33 of 32 of 32x answered, grounded; 1x refused an answer the source holds0 of 33x refused an answer the source holds2 of 31x refused an answer the source holds; 2x answered, grounded3 of 33 of 33 of 33 of 3
read-about3 of 33 of 30 of 33x refused an answer the source holds0 of 33x refused an answer the source holds0 of 33x refused an answer the source holds0 of 33x refused an answer the source holds0 of 33x refused an answer the source holds0 of 33x refused an answer the source holds0 of 33x refused an answer the source holds3 of 33 of 3
ask-complete0 of 32x kept 3 lines it should have refused; 1x kept 2 lines it should have refused0 of 33x kept 3 lines it should have refused0 of 33x kept 3 lines it should have refused2 of 31x kept 3 lines it should have refused; 2x refused, rightly0 of 33x kept 3 lines it should have refused1 of 31x kept 3 lines it should have refused; 1x kept 1 line it should have refused; 1x refused, rightly0 of 32x kept 3 lines it should have refused; 1x kept 2 lines it should have refused0 of 33x kept 3 lines it should have refused0 of 33x kept 3 lines it should have refused0 of 33x kept 3 lines it should have refused0 of 33x kept 3 lines it should have refused
ask-plan3 of 33 of 33 of 33 of 33 of 32 of 32x answered, grounded; 1x refused an answer the source holds3 of 33 of 33 of 33 of 33 of 3
watch-quiet3 of 33 of 33 of 33 of 33 of 30 of 33x kept 1 line it should have refused0 of 33x kept 8 lines it should have refused3 of 33 of 33 of 33 of 3
watch-hit3 of 33 of 33 of 33 of 33 of 33 of 33 of 33 of 33 of 33 of 33 of 3
drill-thin3 of 33 of 33 of 33 of 30 of 33x kept 1 line it should have refused1 of 32x kept 1 line it should have refused; 1x refused, rightly0 of 33x kept 1 line it should have refused1 of 31x refused, rightly; 2x kept 1 line it should have refused0 of 33x kept 1 line it should have refused3 of 33 of 3
drill-facts3 of 33 of 33 of 33 of 33 of 33 of 33 of 33 of 33 of 33 of 33 of 3
discuss-price3 of 33 of 33 of 33 of 33 of 32 of 32x answered, grounded; 1x answered without quoting '$39'3 of 33 of 33 of 33 of 33 of 3
Right runs27 of 3027 of 3024 of 3026 of 3020 of 3015 of 3017 of 3022 of 3019 of 3027 of 3027 of 30
Falsely grounded lines8993121035112099
Over-refusals00334743300

Memory after the run

On a machine in daily use, the room a model leaves matters. This is measured on macOS only, where the system and the model share one memory.

RunMemory free afterSwap in use
MacBook Air, gemma4-26b-a4b9 %2978 MB
MacBook Air, gemma4-e4b63 %2778 MB

Every answer

All 6,571 graded prompt-line answers. The list opens on the failed ones.

CaseQuestionAnswerResultReady

Evidence

The dataset is one file, dataset.json, beside this page. It holds the KPI definitions, the machines, the models, every run's numbers and every graded answer with the model's raw reply and each rule's verdict.

FileBytesSHA-256, first 16
mini-pc/gemma4-26b-a4b/audition.jsonl3,24778665db1b2f99ec5
mini-pc/gemma4-26b-a4b/bench-line.txt1,2109886c4ac5120c7dc
mini-pc/gemma4-26b-a4b/bench.txt293178f818bf65ce805
mini-pc/gemma4-26b-a4b/editor.txt1,4807f36d2c0d6e51e26
mini-pc/gemma4-26b-a4b/ground.txt7954ecb549924404032
mini-pc/gemma4-26b-a4b/line-arch.jsonl159,541939d5f38a8372e68
mini-pc/gemma4-26b-a4b/line-arch.txt5,85553a7f061607fec66
mini-pc/gemma4-26b-a4b/line-debian.jsonl158,472a146942b26b05fdf
mini-pc/gemma4-26b-a4b/line-debian.txt5,902df6c5ec5c30667ef
mini-pc/gemma4-26b-a4b/line-fedora.jsonl160,06604208bac27c60dd7
mini-pc/gemma4-26b-a4b/line-fedora.txt5,912a911a566c476696b
mini-pc/gemma4-26b-a4b/line-opensuse.jsonl160,3481bf41d225b949504
mini-pc/gemma4-26b-a4b/line-opensuse.txt6,03673f10739158ebcf3
mini-pc/gemma4-26b-a4b/line-void.jsonl159,3439d2dfb477a8d7393
mini-pc/gemma4-26b-a4b/line-void.txt6,0207cd7d63f731d1b5c
mini-pc/gemma4-26b-a4b/progress27638e9489c524e9d0c
mini-pc/gemma4-26b-a4b/switch.txt106c223f18d7976f8e6
mini-pc/gemma4-e4b/audition.jsonl3,239dec68ef1b6d5e55c
mini-pc/gemma4-e4b/bench-line.txt1,206029fb3bffb16fd5f
mini-pc/gemma4-e4b/bench.txt289d410c47a344f8e0f
mini-pc/gemma4-e4b/editor.txt1,70094a053e0fb550637
mini-pc/gemma4-e4b/ground.txt755dd3e451472794b31
mini-pc/gemma4-e4b/line-arch.jsonl159,7875e19186c54610a9c
mini-pc/gemma4-e4b/line-arch.txt6,2151eba53e81f615c8a
mini-pc/gemma4-e4b/line-debian.jsonl159,108d114d3ba5e5eb9a0
mini-pc/gemma4-e4b/line-debian.txt6,381c065bf08c33f7238
mini-pc/gemma4-e4b/line-fedora.jsonl160,5978276d8d3260b0bb4
mini-pc/gemma4-e4b/line-fedora.txt6,4580f534905bf4b459c
mini-pc/gemma4-e4b/line-opensuse.jsonl160,810444a7fb8345796f0
mini-pc/gemma4-e4b/line-opensuse.txt6,5027145db0d73434c38
mini-pc/gemma4-e4b/line-void.jsonl160,01576bd88e2090f46b3
mini-pc/gemma4-e4b/line-void.txt6,6487158ddad90f9ad48
mini-pc/gemma4-e4b/progress272c4a6afa5661dc327
mini-pc/gemma4-e4b/switch.txt1020efc121476cfc323
mini-pc/qwen3-30b-a3b/audition.jsonl3,309469c27620e4ddaeb
mini-pc/qwen3-30b-a3b/bench-line.txt1,2097b28a568d6229a34
mini-pc/qwen3-30b-a3b/bench.txt2928b4ea2c83835be3f
mini-pc/qwen3-30b-a3b/editor.txt1,8638854d85a09fa7985
mini-pc/qwen3-30b-a3b/ground.txt77183794577a68c8885
mini-pc/qwen3-30b-a3b/line-arch.jsonl160,0862d9b47d18ae98270
mini-pc/qwen3-30b-a3b/line-arch.txt6,188be2b4c82a5e5e199
mini-pc/qwen3-30b-a3b/line-debian.jsonl159,5176d7fc808b0081bc2
mini-pc/qwen3-30b-a3b/line-debian.txt6,258b5ef7dd530875615
mini-pc/qwen3-30b-a3b/line-fedora.jsonl160,75382f8c3943d12b11f
mini-pc/qwen3-30b-a3b/line-fedora.txt6,386c510c4fbead7d4fd
mini-pc/qwen3-30b-a3b/line-opensuse.jsonl161,141f4fd079dcbb652b9
mini-pc/qwen3-30b-a3b/line-opensuse.txt6,279b9366826d17376ad
mini-pc/qwen3-30b-a3b/line-void.jsonl159,528f2742898575ae9c9
mini-pc/qwen3-30b-a3b/line-void.txt6,2519a16c52c189761d6
mini-pc/qwen3-30b-a3b/progress2755d27b62cd5825f36
mini-pc/qwen3-30b-a3b/switch.txt183c9862ecb204ef992
mini-pc/granite-4-2-8b/audition.jsonl3,2351344070e3caaa0aa
mini-pc/granite-4-2-8b/bench-line.txt1,210e841c6f5fe73bb83
mini-pc/granite-4-2-8b/bench.txt293dd2f824ef4f4c1c7
mini-pc/granite-4-2-8b/editor.txt2,011631830bc0be49ef5
mini-pc/granite-4-2-8b/ground.txt79251839832b04919cb
mini-pc/granite-4-2-8b/line-arch.jsonl158,481138f9b319185eb2d
mini-pc/granite-4-2-8b/line-arch.txt6,492da1506b474f61639
mini-pc/granite-4-2-8b/line-debian.jsonl157,982dd20633e75b225fc
mini-pc/granite-4-2-8b/line-debian.txt6,5705e79b4c90aac93e2
mini-pc/granite-4-2-8b/line-fedora.jsonl158,9348f9a0e310b6ddfd3
mini-pc/granite-4-2-8b/line-fedora.txt6,521eda1fe0facbebffc
mini-pc/granite-4-2-8b/line-opensuse.jsonl159,735a6e6cd089211c37a
mini-pc/granite-4-2-8b/line-opensuse.txt6,687dd26ff66f8d662c6
mini-pc/granite-4-2-8b/line-void.jsonl158,54537bd4e3828ca5bef
mini-pc/granite-4-2-8b/line-void.txt6,748c5f69490020f8d51
mini-pc/granite-4-2-8b/progress27642b0b2f00f18a46e
mini-pc/granite-4-2-8b/switch.txt10611ca9a4c0243348b
mini-pc/qwen3-5-4b/audition.jsonl3,346f472a4f79e24c2f6
mini-pc/qwen3-5-4b/bench-line.txt1,20669ddb9c9d09a4dcb
mini-pc/qwen3-5-4b/bench.txt2891fed38ccd1739d02
mini-pc/qwen3-5-4b/editor.txt1,8789ba17bc4c0e67a2f
mini-pc/qwen3-5-4b/ground.txt8292a37fff349c6e560
mini-pc/qwen3-5-4b/line-arch.jsonl160,831ae5f23bae01a7fe3
mini-pc/qwen3-5-4b/line-arch.txt6,749dd9b705a8da14f86
mini-pc/qwen3-5-4b/line-debian.jsonl160,4803c0a5da467fc2009
mini-pc/qwen3-5-4b/line-debian.txt6,7428b91ddd7421f28a5
mini-pc/qwen3-5-4b/line-fedora.jsonl160,607bc4665172d59912d
mini-pc/qwen3-5-4b/line-fedora.txt6,68768a834e882ee0b9b
mini-pc/qwen3-5-4b/line-opensuse.jsonl162,245c27324aea0f81fc2
mini-pc/qwen3-5-4b/line-opensuse.txt7,086f111ae33bddf22d7
mini-pc/qwen3-5-4b/line-void.jsonl160,46016e324148678eca3
mini-pc/qwen3-5-4b/line-void.txt6,96301cd7b6a07211735
mini-pc/qwen3-5-4b/progress272e52f847ed97a6aa3
mini-pc/qwen3-5-4b/switch.txt102a8e1f891f8e39eb9
mini-pc/qwen3-5-2b/audition.jsonl3,42192623137133401b1
mini-pc/qwen3-5-2b/bench-line.txt1,206a43505f34f29939a
mini-pc/qwen3-5-2b/bench.txt289eef45718d9170a41
mini-pc/qwen3-5-2b/editor.txt2,170148b64fac7ab3889
mini-pc/qwen3-5-2b/ground.txt979730011c0467d5893
mini-pc/qwen3-5-2b/line-arch.jsonl159,7801560af28593af977
mini-pc/qwen3-5-2b/line-arch.txt7,2095338992f8f1aceb3
mini-pc/qwen3-5-2b/line-debian.jsonl159,232a85339a3274efa70
mini-pc/qwen3-5-2b/line-debian.txt7,318d6da3e4d39216957
mini-pc/qwen3-5-2b/line-fedora.jsonl159,967ee51ceac34335e89
mini-pc/qwen3-5-2b/line-fedora.txt7,2996bfe56a5f007e574
mini-pc/qwen3-5-2b/line-opensuse.jsonl160,4195bbd5ee5f2189fc1
mini-pc/qwen3-5-2b/line-opensuse.txt7,734bfca10ea1583a762
mini-pc/qwen3-5-2b/line-void.jsonl159,620c5c713f528a46728
mini-pc/qwen3-5-2b/line-void.txt8,042bb5ab0291394334b
mini-pc/qwen3-5-2b/progress272321e9f3c2615f8f9
mini-pc/qwen3-5-2b/switch.txt10264d3ea1877862867
mini-pc/qwen3-4b/audition.jsonl3,427c1ca65d59ed01e20
mini-pc/qwen3-4b/bench-line.txt1,20442a7c1a0f7aa5565
mini-pc/qwen3-4b/bench.txt28794d63588b9764d6a
mini-pc/qwen3-4b/editor.txt1,79497f22761c4aab6f5
mini-pc/qwen3-4b/ground.txt888ab99a69912137b66
mini-pc/qwen3-4b/line-arch.jsonl160,810131fddc23c18113d
mini-pc/qwen3-4b/line-arch.txt6,5595956b8dccc623dba
mini-pc/qwen3-4b/line-debian.jsonl160,5701d1c3898140b9ba6
mini-pc/qwen3-4b/line-debian.txt6,935cf82d678650a997c
mini-pc/qwen3-4b/line-fedora.jsonl161,4215367ffd3d31d7a3b
mini-pc/qwen3-4b/line-fedora.txt6,574e7f0daa1d15e08ba
mini-pc/qwen3-4b/line-opensuse.jsonl162,074679728b9e5ffa504
mini-pc/qwen3-4b/line-opensuse.txt6,773a1cb0ceefbd4f676
mini-pc/qwen3-4b/line-void.jsonl160,46629360b2aa1053631
mini-pc/qwen3-4b/line-void.txt7,001288a649c9bb5bfcb
mini-pc/qwen3-4b/progress27081174ee4912fb08b
mini-pc/qwen3-4b/switch.txt10028e19c47c4ca37b5
mini-pc/qwen3-8b/audition.jsonl3,308fafb2287c5af2df0
mini-pc/qwen3-8b/bench-line.txt1,204255c7af4b84acbeb
mini-pc/qwen3-8b/bench.txt287bdd01a3cf86be872
mini-pc/qwen3-8b/editor.txt1,6444df6de709c600c32
mini-pc/qwen3-8b/ground.txt811674a0326f3f54a39
mini-pc/qwen3-8b/line-arch.jsonl159,06501a9850dd35c07a9
mini-pc/qwen3-8b/line-arch.txt6,60730fcfe81df9c2bd9
mini-pc/qwen3-8b/line-debian.jsonl158,200ba74fe3ba9c5e85b
mini-pc/qwen3-8b/line-debian.txt6,563eb747450fb1a2e69
mini-pc/qwen3-8b/line-fedora.jsonl158,982883f491b7a3068f6
mini-pc/qwen3-8b/line-fedora.txt6,446f3db61183ee00fde
mini-pc/qwen3-8b/line-opensuse.jsonl159,7321cecd1366b75720c
mini-pc/qwen3-8b/line-opensuse.txt6,438353b3bfabcc539ff
mini-pc/qwen3-8b/line-void.jsonl158,815496bf6850cb1ebb1
mini-pc/qwen3-8b/line-void.txt6,532336b8141afbe7d8b
mini-pc/qwen3-8b/progress270bcb92b0806ae4b66
mini-pc/qwen3-8b/switch.txt100c1fd6a21067452d7
mini-pc/qwen3-14b/audition.jsonl3,3661c24e9a37df8382e
mini-pc/qwen3-14b/bench-line.txt1,20569da1d9dfaec2e38
mini-pc/qwen3-14b/bench.txt288bff96865189680dd
mini-pc/qwen3-14b/editor.txt1,58774c510c455a553a7
mini-pc/qwen3-14b/ground.txt8307671d22989d3f9e4
mini-pc/qwen3-14b/line-arch.jsonl159,009a641c6ab248c4aa9
mini-pc/qwen3-14b/line-arch.txt6,1180d8e87136e5cd3d2
mini-pc/qwen3-14b/line-debian.jsonl157,991e9e104b40246803c
mini-pc/qwen3-14b/line-debian.txt6,079c412fddf9cc2c4a7
mini-pc/qwen3-14b/line-fedora.jsonl159,1860a41df981f8265da
mini-pc/qwen3-14b/line-fedora.txt6,116f278fa73a01e31ff
mini-pc/qwen3-14b/line-opensuse.jsonl159,9026c8cc54f39f5eac7
mini-pc/qwen3-14b/line-opensuse.txt6,191ed06c417c0172045
mini-pc/qwen3-14b/line-void.jsonl158,580e934cd4adb8e44ac
mini-pc/qwen3-14b/line-void.txt6,445af20f8ef4a8a8750
mini-pc/qwen3-14b/progress271a79ebe4c63ef6608
mini-pc/qwen3-14b/switch.txt101af7aeb5ddff083d4
macbook-air/gemma4-26b-a4b/audition.jsonl3,247480fa5712d1541e2
macbook-air/gemma4-26b-a4b/bench-line.txt1,210a1341fda96022e15
macbook-air/gemma4-26b-a4b/bench.txt293446917a515edf4f4
macbook-air/gemma4-26b-a4b/editor.txt1,420ed27167413056f9c
macbook-air/gemma4-26b-a4b/ground.txt756cced94a8f9bae837
macbook-air/gemma4-26b-a4b/line-macos.jsonl159,702711b158ea8f0693a
macbook-air/gemma4-26b-a4b/line-macos.txt5,804c9b199b67793f3d3
macbook-air/gemma4-26b-a4b/memory-after.txt1033ec3e23a62748042
macbook-air/gemma4-26b-a4b/progress18184f9eaf19b0d3d0b
macbook-air/gemma4-26b-a4b/switch.txt186ccaf2e1380cc4a72
macbook-air/gemma4-e4b/audition.jsonl3,239da66fe62dd08f10e
macbook-air/gemma4-e4b/bench-line.txt1,2066df735fe90f32596
macbook-air/gemma4-e4b/bench.txt2898f4040c482cd0907
macbook-air/gemma4-e4b/editor.txt1,557a4252459699198d0
macbook-air/gemma4-e4b/ground.txt756cced94a8f9bae837
macbook-air/gemma4-e4b/line-macos.jsonl160,2942f15ce3e9a2dceec
macbook-air/gemma4-e4b/line-macos.txt6,1763e72ada22c778be3
macbook-air/gemma4-e4b/memory-after.txt104754fe8737b6d80d2
macbook-air/gemma4-e4b/progress177288ccd336e5cf649
macbook-air/gemma4-e4b/switch.txt1020efc121476cfc323
mini-pc/hw.txt563a957412f91c6ff4f
mini-pc/spark-tree.txt1027c7077307ca1019d
mini-pc/models.env3,860cd30c091d0df0534
macbook-air/hw.txt43586b40f7977a8b9e2
macbook-air/spark-tree.txt1027c7077307ca1019d
macbook-air/models.env9972b2bb39a1be49ac2

Limits of this measurement

  • Each machine served the model itself.
  • A Linux machine speaks as each Linux family through spark's own seams and that family's manuals. macOS cases can only run on a Mac.
  • A model is not deterministic. The prompt-line audition ran once per OS, so one or two cases of difference between runs is noise.
  • The 2.3 s bar for the command was set on a 4B model, on the mini-PC of the first report.
  • The answers' stderr lines are left out. They held only a temporary path.
  • One run per system and model. A case or two of difference between runs is noise.
  • The seven other models ran on the mini-PC alone. A smaller machine would run them slower, and the 30B-A3B would not fit.
  • spark's model table calls gemma4-26b-a4b and the 30B-A3B too big for the mini-PC under its 60 % budget. Both were chosen by name and ran. auto would not pick them there.
  • The questions about spark were tuned on. The held-out questions were not. The report shows both.
  • The MacBook Air is a client of the mini-PC in daily use. It was set to serve its own models for this run, then set back.
  • The mini-PC's memory modules were not read. Its memory line comes from what the system reports.
One developer's experiment, not a promise of performance. MIT licence, no warranty. Built from dataset.json by build_report.py. Accuracy, danger recall, over-fire and the re-ask share are computed with the audition's own code, so they match its report. Flag honesty, the no-answer share and the command-ready times are counted more strictly here, and the KPI definitions say how.