Benchmark and audition

spark v1.87 on two machines and two models

The same two machines and the same two Gemma 4 models as the first report, measured again after a release that went after its results.

measured 2026-10-05spark v1.87 at cc103ccengine llama.cpp b106894 runs1,678 graded answers

Summary

gemma4-26b-a4b meets the bar on both machines: each system's tools right 96 times in 100 or more, spark's own verbs 204 times in 204, every danger mark, no invented option, the command ready in under 2 s. gemma4-e4b answers every question and keeps every danger mark. It is still short of the bar on the tools of three systems.

Measure	mini-PCgemma4-26b-a4b	mini-PCgemma4-e4b	MacBook Airgemma4-26b-a4b	MacBook Airgemma4-e4b
Tools accuracybar: 90 % on each OS	98 %284 of 289, lowest openSUSE 96 %	89 %258 of 289, lowest Void 84 %	100 %58 of 58	93 %54 of 58
spark core accuracybar: 90 %	100 %170 of 170	97 %165 of 170	100 %34 of 34	94 %32 of 34
spark core, second phrasingsno bar	100 %120 of 120	90 %109 of 120	100 %24 of 24	95 %23 of 24
spark core, held-out questionsno bar	96 %116 of 120	79 %95 of 120	100 %24 of 24	83 %20 of 24
Danger recallbar: 100 %	100 %59 of 59	100 %59 of 59	100 %11 of 11	100 %11 of 11
Danger over-firetarget: 10 % or less	9 %39 of 400	11 %46 of 400	8 %7 of 81	9 %8 of 81
Flag honestybar: 100 %	100 %459 of 459	98.7 %442 of 448	100 %92 of 92	100 %91 of 91
False warningsno bar	0 %0 of 454	0 %0 of 423	0 %0 of 92	0 %0 of 86
No answerno bar	0 %0 of 459	0 %0 of 459	0 %0 of 92	0 %0 of 92
Command ready, audition medianbar: 2.3 s or less	1.91 s	1.47 s	1.66 s	1.53 s
Command ready, the slow tenthno bar	2.44 s	2.10 s	2.06 s	2.40 s
Speed, writingno bar	24.2 tok/s	23.6 tok/s	29.7 tok/s	26.1 tok/s
Editor auditionno bar	93 %504 of 540	91 %496 of 540	94 %510 of 540	95 %513 of 540
Grounded casesno bar	27 of 30	27 of 30	27 of 30	27 of 30

meets its bar or targetmisses itplain cells have no bar

For a machine that speaks as several OSes, the prompt-line rows add them together. The tools row is coloured by the lowest family, because the bar is per OS.

Before and after

This run beside the report 2026-10-04-v1.85 (spark v1.85). Each measure has three numbers. The first is the earlier report as it was published. The second is the same answers of that report, graded again with the grader this run used (v1.87): what moves between the first and the second is the grader's own doing, tests that were wrong or too kind, and not spark's. The third is this run. What moves between the second and the third is spark.

mini-PC, gemma4-26b-a4b

Measure	As publishedspark v1.85	Graded againsame answers	This runspark v1.87
Tools accuracy	92 %266 of 289	92.4 %267 of 289+0.4 points	98.3 %284 of 289+5.9 points
as Debian	93 %53 of 57	93 %53 of 57no change	98.2 %56 of 57+5.2 points
as Arch	93.1 %54 of 58	93.1 %54 of 58no change	100 %58 of 58+6.9 points
as Void	89.7 %52 of 58	91.4 %53 of 58+1.7 points	98.3 %57 of 58+6.9 points
as Fedora	93.1 %54 of 58	93.1 %54 of 58no change	98.3 %57 of 58+5.2 points
as openSUSE	91.4 %53 of 58	91.4 %53 of 58no change	96.6 %56 of 58+5.2 points
spark core accuracy	67.6 %115 of 170	65.3 %111 of 170-2.3 points	100 %170 of 170+34.7 points
spark core, second phrasings	-	-	100 %120 of 120
spark core, held-out questions	-	-	96.7 %116 of 120
Danger recall	100 %54 of 54	100 %59 of 59no change	100 %59 of 59no change
Danger over-fire	18.5 %75 of 405	17.5 %70 of 400-1.0 points	9.8 %39 of 400-7.7 points
Flag honesty	99.6 %451 of 453	99.6 %451 of 453no change	100 %459 of 459+0.4 points
False warnings	-	5.3 %20 of 378	0 %0 of 454-5.3 points
No answer	0 %0 of 459	0 %0 of 459no change	0 %0 of 459no change
spark commands on tools questions	-	0 %0 of 289	0 %0 of 289no change
Re-ask share	6.5 %30 of 459	6.5 %30 of 459no change	2.4 %11 of 459-4.1 points
Command ready, median	2.15 s	2.15 s+0.01 s	1.91 s-0.25 s
Command ready, the slow tenth	2.97 s	2.97 sno change	2.44 s-0.54 s

mini-PC, gemma4-e4b

Measure	As publishedspark v1.85	Graded againsame answers	This runspark v1.87
Tools accuracy	85.1 %246 of 289	84.1 %243 of 289-1.0 points	89.3 %258 of 289+5.2 points
as Debian	89.5 %51 of 57	87.7 %50 of 57-1.8 points	89.5 %51 of 57+1.8 points
as Arch	93.1 %54 of 58	91.4 %53 of 58-1.7 points	93.1 %54 of 58+1.7 points
as Void	72.4 %42 of 58	74.1 %43 of 58+1.7 points	84.5 %49 of 58+10.4 points
as Fedora	91.4 %53 of 58	89.7 %52 of 58-1.7 points	91.4 %53 of 58+1.7 points
as openSUSE	79.3 %46 of 58	77.6 %45 of 58-1.7 points	87.9 %51 of 58+10.3 points
spark core accuracy	61.2 %104 of 170	57.6 %98 of 170-3.6 points	97.1 %165 of 170+39.5 points
spark core, second phrasings	-	-	90.8 %109 of 120
spark core, held-out questions	-	-	79.2 %95 of 120
Danger recall	94.4 %51 of 54	89.8 %53 of 59-4.6 points	100 %59 of 59+10.2 points
Danger over-fire	11.6 %47 of 405	11.2 %45 of 400-0.4 points	11.5 %46 of 400+0.3 points
Flag honesty	99.1 %427 of 431	97.9 %422 of 431-1.2 points	98.7 %442 of 448+0.8 points
False warnings	-	4.7 %16 of 341	0 %0 of 423-4.7 points
No answer	1.1 %5 of 459	1.5 %7 of 459+0.4 points	0 %0 of 459-1.5 points
spark commands on tools questions	-	0 %0 of 289	0 %0 of 289no change
Re-ask share	8.1 %37 of 454	8.1 %37 of 454no change	4.8 %22 of 459-3.3 points
Command ready, median	1.83 s	1.83 sno change	1.47 s-0.37 s
Command ready, the slow tenth	5.71 s	5.77 s+0.06 s	2.10 s-3.67 s

MacBook Air, gemma4-26b-a4b

Measure	As publishedspark v1.85	Graded againsame answers	This runspark v1.87
Tools accuracy	87.9 %51 of 58	89.7 %52 of 58+1.8 points	100 %58 of 58+10.3 points
spark core accuracy	64.7 %22 of 34	61.8 %21 of 34-2.9 points	100 %34 of 34+38.2 points
spark core, second phrasings	-	-	100 %24 of 24
spark core, held-out questions	-	-	100 %24 of 24
Danger recall	100 %10 of 10	100 %11 of 11no change	100 %11 of 11no change
Danger over-fire	14.6 %12 of 82	13.6 %11 of 81-1.0 points	8.6 %7 of 81-5.0 points
Flag honesty	100 %90 of 90	100 %90 of 90no change	100 %92 of 92no change
False warnings	-	0 %0 of 73	0 %0 of 92no change
No answer	0 %0 of 92	0 %0 of 92no change	0 %0 of 92no change
spark commands on tools questions	-	0 %0 of 58	0 %0 of 58no change
Re-ask share	1.1 %1 of 92	1.1 %1 of 92no change	1.1 %1 of 92no change
Command ready, median	1.89 s	1.89 sno change	1.66 s-0.23 s
Command ready, the slow tenth	2.30 s	2.30 sno change	2.06 s-0.24 s

MacBook Air, gemma4-e4b

Measure	As publishedspark v1.85	Graded againsame answers	This runspark v1.87
Tools accuracy	81 %47 of 58	81 %47 of 58no change	93.1 %54 of 58+12.1 points
spark core accuracy	58.8 %20 of 34	58.8 %20 of 34no change	94.1 %32 of 34+35.3 points
spark core, second phrasings	-	-	95.8 %23 of 24
spark core, held-out questions	-	-	83.3 %20 of 24
Danger recall	90 %9 of 10	90.9 %10 of 11+0.9 points	100 %11 of 11+9.1 points
Danger over-fire	9.8 %8 of 82	8.6 %7 of 81-1.2 points	9.9 %8 of 81+1.3 points
Flag honesty	100 %82 of 82	100 %82 of 82no change	100 %91 of 91no change
False warnings	-	0 %0 of 67	0 %0 of 86no change
No answer	7.6 %7 of 92	8.7 %8 of 92+1.1 points	0 %0 of 92-8.7 points
spark commands on tools questions	-	0 %0 of 58	0 %0 of 58no change
Re-ask share	3.5 %3 of 85	3.5 %3 of 85no change	5.4 %5 of 92+1.9 points
Command ready, median	1.96 s	1.97 s+0.01 s	1.53 s-0.44 s
Command ready, the slow tenth	5.58 s	5.58 sno change	2.40 s-3.18 s

The small line under a number is how far it moved from the column before it, in points or in seconds. A measure the earlier report did not have shows a dash in its first column. Flag honesty, the no-answer share and the command-ready times are counted more strictly now than they were published: the KPI definitions say how. The second phrasings and the held-out questions were not asked before.

What a re-grade cannot settle

The first report's runs did not keep the prompt's "Preferred when installed" line, so a re-grade cannot see whether a tool of the measuring machine was named in the prompt. The answers suggest it was: 20 answers use micro, btop, fd or rg, which the OS they spoke as does not have, and they fail (micro 11, btop 7, fd 1, rg 1). A new run checks the prompt before it starts.
The first report's runs did not run the line's not-installed path. All 12 answers to a question about a tool that is not installed fail under today's grader, which wants the command that installs the tool: 9 run the missing tool itself and 3 say in words that it is missing.
Together these are 32 of the 243 failures of the first report under today's grader. A re-grade cannot take them away. Only a new run can.

What moved, case by case

The earlier answers under today's grader against this run's, answer by answer: the same machine, model, OS and question.

211answers fixed15answers broken32still failing844still right576to questions not asked before

Cases fixed

A case is one question. Each count is over the runs and OSes that asked it both times. The table shows the first 15 of 71.

Case	Question	Failed before	Fails now	What failed before
`spark-04`as every OS	use a bigger model for chat	12 of 12	0 of 12	must: lacks --chat
`spark-19`as every OS	stop using the other machine's model	12 of 12	0 of 12	head: spark model, wanted client
`spark-30`as every OS	turn the prompt line off	12 of 12	0 of 12	head: spark keys, wanted off
`spark-35`as every OS	show what spark depends on	12 of 12	0 of 12	head: spark help, wanted ver
`spark-37`as every OS	what has spark sent over the network	12 of 12	0 of 12	head: spark -, wanted stats
`spark-39`as every OS	show the last answer again	12 of 12	0 of 12	head: spark history, wanted last
`spark-22`as every OS	remember that my editor is micro	11 of 12	0 of 12	head: names no spark memory
`spark-08`as every OS	run this machine as a headless box that is always on	12 of 12	2 of 12	head: names no spark serve
`spark-18`as every OS	use the model on my other machine	10 of 12	0 of 12	head: spark model, wanted client
`spark-15`as every OS	stop the engine	7 of 12	0 of 12	head: spark off, wanted serve
`spark-28`as every OS	open the page on my phone	12 of 12	5 of 12	head: spark -, wanted serve
`spark-12`as every OS	who has an account on the page	6 of 12	0 of 12	head: spark -, wanted user\|serve
`spark-23`as every OS	change who spark is	5 of 12	0 of 12	head: spark user, wanted soul\|awaken
`spark-33`as every OS	make a new admin token for the page	5 of 12	0 of 12	must: lacks --new
`spark-03`as every OS	turn off the scanner and the colours at my prompt	4 of 12	0 of 12	head: spark keys, wanted look

Cases broken

These passed before and fail now, on at least one run.

Case	Question	Failed before	Fails now	What fails now
`arch-file-14`as Arch	search every file here for the word TODO	0 of 2	1 of 2	head: fd, wanted grep\|rg
`arch-sys-3`as Arch	what time zone is this machine in	0 of 2	1 of 2	kind: wanted cmd, got answer
`deb-pkg-2`as Debian	is nginx installed	0 of 2	1 of 2	head: apt-get grep, wanted dpkg\|dpkg-query\|apt\|apt-cache
`deb-pkg-3`as Debian	search the packages for a pdf viewer	0 of 2	1 of 2	head: apt-get grep, wanted apt\|apt-cache
`deb-sys-3`as Debian	what time zone is this machine in	0 of 2	1 of 2	kind: wanted cmd, got answer
`fedora-file-14`as Fedora	search every file here for the word TODO	0 of 2	1 of 2	head: fd, wanted grep\|rg
`mac-pkg-2`as macOS	is wget installed	0 of 2	1 of 2	kind: wanted cmd, got answer
`mac-sys-1`as macOS	which kernel version is this	0 of 2	1 of 2	head: sw_vers, wanted uname
`opensuse-file-14`as openSUSE	search every file here for the word TODO	0 of 2	1 of 2	head: fd, wanted grep\|rg
`opensuse-pkg-5`as openSUSE	upgrade every package	1 of 2	2 of 2	must_not: matches \bzypper\s+(?:\S+\s+)*?(?:up\|update)\b
`opensuse-pkg-7`as openSUSE	list the repositories this machine uses	0 of 2	1 of 2	kind: wanted cmd, got answer
`opensuse-sys-3`as openSUSE	what time zone is this machine in	0 of 2	1 of 2	kind: wanted cmd, got answer
`void-net-13`as Void	copy notes.txt to the machine at 192.0.2.10	0 of 2	1 of 2	head: nc, wanted scp\|rsync
`void-net-3`as Void	can I reach example.com	1 of 2	2 of 2	head: ip, wanted ping\|curl\|nc
`void-pkg-7`as Void	remove the packages nothing needs anymore	0 of 2	1 of 2	flags: xbps-query: -e --installed not in its help

Cases still failing on every run

These failed every time they were asked, before and now.

Case	Question	Failed before	Fails now	What fails now
`deb-net-3`as Debian	can I reach example.com	2 of 2	2 of 2	head: apt-get, wanted ping\|curl\|nc
`fedora-net-3`as Fedora	can I reach example.com	2 of 2	2 of 2	head: dnf, wanted ping\|curl\|nc
`opensuse-net-3`as openSUSE	can I reach example.com	2 of 2	2 of 2	head: ip, wanted ping\|curl\|nc

Findings

The larger model meets the bar on both machines. gemma4-26b-a4b is right about each system's own tools 96 to 100 times in 100, and about spark's own verbs 204 times in 204. It kept all 70 danger marks and invented no option in 551 commands. The command was ready in 1.9 s on the mini-PC and 1.7 s on the MacBook Air. In the first report no run met the bar.
spark's own verbs were the weak spot. They are the strong one now. On the 34 questions of the first report gemma4-26b-a4b went from about 2 right in 3 to all of them. gemma4-e4b went from about 6 in 10 to 97 in 100 on the mini-PC and 94 on the MacBook Air. The prompt line now reads spark's own help: each command with what it is for. Before, it saw the names alone.
On questions never tuned on, the larger model holds and the small one drops. 24 new questions about spark were written before the work began and were never used to tune it. gemma4-26b-a4b answers 97 in 100 of them on the mini-PC and all of them on the MacBook Air. gemma4-e4b answers 79 and 83. So its score on the 34 known questions flatters it by about 15 points.
No question went unanswered. In the first report gemma4-e4b returned nothing on 15 questions, counting 3 replies cut after the command. It answered all 551 this time. The model had been spending its whole reply in a hidden thought. spark now holds the reply to its shape from the first character, so a thought cannot start.
Every danger mark held, with both models. All 140 dangerous cases came back marked, 70 with each model. In the first report gemma4-e4b missed 4 of 64. Cleaning a package cache is on spark's own list now, and so is spark serve off. The mark on spark's own commands is spark's decision, not the model's.
A right command is no longer doubted. In the first report 36 right answers carried spark's own warning that the manual had no such command, as for systemctl restart and zypper search. spark had read the command lists of those manuals cut short. No right answer carries the warning now. Second asks fell from 6.5 to 2.4 in 100 with gemma4-26b-a4b on the mini-PC, and from 8.1 to 4.8 with gemma4-e4b.
The slow tenth of the answers is gone. With gemma4-e4b the slow tenth of the answers took over 5.5 s in the first report. It takes 2.1 s on the mini-PC and 2.4 s on the MacBook Air now. With gemma4-26b-a4b it went from 3.0 to 2.4 s on the mini-PC. The median is 1.5 to 1.9 s on every run.
Part of the tools gain is the test becoming fair. About a third of the first report's tool misses were the test's fault. The prompt named the measuring machine's own tools as installed, and the grader then said they did not exist on the system spoken as. And a question about a tool that is not installed could not take the path it takes on a real machine. With both fixed, spark v1.86 itself scored 96 in 100 on tools with gemma4-26b-a4b, the night before this run. On tools the release moved the larger model from 96 to 98, not from 92.
The small model is still short on the tools of three systems. gemma4-e4b scores 84 in 100 as Void, 88 as openSUSE and 89 as Debian. It invents options there (svlogtail -n 50), installs a tool where a test of the network was asked, and answers some questions in words. Its over-fire is 11.5 in 100 on the mini-PC, against a bar of 10. Every one of those marks fell on a case where a mark is acceptable.
Speed is what it was. The engine and the models are the same as in the first report. The mini-PC reads a prompt at 419 and 594 tokens a second and writes at 24. The MacBook Air reads at 334 and 311 and writes at 30 and 26.
The grounded cases went from 24 to 27 of 30. The editor stays near 93. One grounded case of the first report was the test's fault: its source answered the question the case said it could not. With that case fixed, all four runs score 27 of 30. The editor audition is at 92 to 95 in 100.

Setup

Each machine served the model itself, with every layer on its GPU. The same spark version, the same engine build number and the same model files ran on both.

	mini-PC, desktop	MacBook Air, notebook, Mac16,12
Role	a desktop, the home server	a notebook in daily use
System	Void Linux, x86_64	macOS 26, arm64
Processor	AMD Ryzen 7 8745H, 8 cores, 16 threads	Apple M4, 10 cores (4 performance, 6 efficiency)
Graphics	AMD Radeon 780M, built in	Apple M4, 10 cores
GPU path	Vulkan	Metal
Memory	15 GB for the system and 16 GB set aside for the GPU (32 GB installed, the modules themselves were not read)	24 GB LPDDR5, shared with the GPU
Disk	NVMe SSD, 477 GB	460 GB, 294 GB free
Engine	llama.cpp b10689, ubuntu-vulkan-x64	llama.cpp b10689, macos-arm64

Model	Size	Quantisation	File	Memory needed	Licence
`gemma4-26b-a4b`	26B, mixture of experts, 4B active a token	Q4_K_M	17.0 GB	19 GB	Apache-2.0
`gemma4-e4b`	E4B (edge, 4B effective)	Q4_K_M	5.4 GB	8 GB	Apache-2.0

How each run went

Every run did the same steps in the same order: the engine's speed test, 10 timed prompt-line questions, the prompt-line audition for each OS, the editor audition 3 times, then the grounded cases 3 times.

Run	Started	Ended	Took	Prompt-line answers
mini-PC, gemma4-26b-a4b	07:49:18	08:43:21	54 min	699
mini-PC, gemma4-e4b	08:44:04	09:32:12	48 min	699
MacBook Air, gemma4-26b-a4b	07:49:05	08:08:20	19 min	140
MacBook Air, gemma4-e4b	08:09:03	08:29:40	20 min	140

KPI definitions

Every answer is graded by a program, never by a person or a model. The grader shares no code with spark's own judge.

Speed, reading a prompt

What it is: Tokens a second the engine processes when it reads a 512-token prompt. The engine's own bench tool measures it with spark's engine paused, as the mean of 3 repeats.
Why it matters: It sets how long a long question, a file or a page takes before the first word comes back.
Unit: tok/s, higher is better
Target: none, reported only
Measured by: spark bench (llama-bench -p 512 -r 3)

Speed, writing

What it is: Tokens a second the engine writes over a 128-token reply, the mean of 3 repeats, with spark's engine paused.
Why it matters: It is the pace of every reply: chat, an explanation, a rewrite in the editor.
Unit: tok/s, higher is better
Target: none, reported only
Measured by: spark bench (llama-bench -n 128 -r 3)

Prompt line, command ready

What it is: Seconds from pressing the key to the moment the proposed command is complete on the prompt line. The median over the answers that proposed a command. An answer in words has no command to wait for and is left out. The hint that follows the command is not counted. The first report took this median over every answer, the ones in words too. That number is kept in the dataset as first_line_ms.
Why it matters: It is the wait a person feels at the prompt. The command is usable before the hint ends.
Unit: s, lower is better
Target: 2.3 s or less (the bar, set on a 4B model)
Measured by: spark bench --line 10, and cmd_ms in the audition

Prompt line, the slow tenth

What it is: The time by which 9 answers in 10 had their command ready. One answer in 10 took longer than this. It is taken over the same answers as the command-ready median.
Why it matters: A median hides the waits a person remembers. This number shows them.
Unit: s, lower is better
Target: none, reported only
Measured by: cmd_ms in the audition, its 90th percentile

Prompt line, whole answer

What it is: Seconds from the key to the end of the whole answer, the command and its one-line hint. The median over the questions asked.
Why it matters: It is the full cost of one question at the prompt.
Unit: s, lower is better
Target: none, reported only
Measured by: spark bench --line 10, and total_ms in the audition

Tools accuracy

What it is: Share of the questions about the OS's own tools that got a right answer. About 58 questions per OS. An answer is right when it has the contract's shape, its first command is one the case accepts, that command exists on that OS, every option it passes is in that OS's manual, no forbidden pattern appears, and the danger mark is right when the case demands one. The answer must also hold what the case says it cannot leave out, and the reply must end without an error.
Why it matters: It says how often the line proposes a command a person can run as it stands.
Unit: %, higher is better
Target: 90 % or more on each OS
Measured by: tests/line_audition.py run, subject tools

spark core accuracy

What it is: Share of the 34 questions about spark itself that got a right answer, graded the same way. A spark verb must exist in the tab completion files or the cheatsheet. These are the questions the first report asked. The second phrasings and the held-out questions are counted apart.
Why it matters: It says how well spark can explain and drive itself.
Unit: %, higher is better
Target: 90 % or more
Measured by: tests/line_audition.py run, subject spark

spark core accuracy, held-out questions

What it is: Share of the 24 held-out questions about spark's own verbs that got a right answer. These questions are frozen: nobody tunes the prompt or the knowledge on their answers. A run measured before they existed has no number here. Every other prompt-line measure leaves them out, so two reports compare on the same questions.
Why it matters: A gain on questions the prompt was tuned on may be a gain on those questions alone. This number says whether it holds on questions nobody tuned for.
Unit: %, higher is better
Target: none, reported only
Measured by: tests/line_audition.py run, the cases whose split is held

spark core accuracy, second phrasings

What it is: Share of the 24 second phrasings that got a right answer. Each asks what one of the first report's spark questions asks, in other words. They were added to tune the prompt on, so a gain here is expected. A run measured before they existed has no number here, and every other prompt-line measure leaves them out.
Why it matters: It says whether a right answer depends on the exact words of a question.
Unit: %, higher is better
Target: none, reported only
Measured by: tests/line_audition.py run, the cases whose split is more

Danger recall

What it is: Of the cases that are dangerous by design, the share whose answer came back with the danger mark. An answer with no command at all counts as a miss.
Why it matters: A missed mark is the harmful mistake: a destructive command shown as a safe one.
Unit: %, higher is better
Target: 100 %, one miss blocks a release
Measured by: tests/line_audition.py report

Danger over-fire

What it is: Of the cases that are not marked dangerous by design, the share whose answer came back with the danger mark anyway. In these runs 0 of the 100 marks fell on cases marked safe and 100 on cases where a mark is acceptable.
Why it matters: It does no damage, but a mark that shows on harmless commands teaches people to ignore it.
Unit: %, lower is better
Target: 10 % or less
Measured by: tests/line_audition.py report

Marks on safe cases

What it is: Over-fire split in two. The case list marks each case as dangerous, safe, or either (a mark is acceptable but not required, as for a restart or an install). This is the share of the cases marked safe that came back with the danger mark. The rest of the over-fire falls on the either cases.
Why it matters: A mark on a command that only reads is a plain false alarm. A mark on an either case is a matter of taste.
Unit: %, lower is better
Target: none, reported only
Measured by: computed here from the same answers

No answer

What it is: Share of the questions that got no answer. Either the line said so ("the model gave no answer -- ask again"), or the reply was cut after its first line: a command came, then the reason in place of a hint, and a failed exit. It is the audition's own measure where a record kept the exit code. A record with no exit code counts only the replies whose first line is an error. Every record of these runs kept its exit code. The first report counted the error lines alone, as no_answer.
Why it matters: A person must ask again, and the miss lowers every accuracy number.
Unit: %, lower is better
Target: none, reported only
Measured by: tests/line_audition.py report, the empty measure

Flag honesty

What it is: Of the answers that proposed a command, the share whose every option appears in that tool's manual on that OS, or is named as a whole word in the model's own hint. spark's own warning line ("the zypper manual has no --whatowns -- check it before Enter") is not the model's hint and earns nothing. The audition's own number is looser: it takes an option found anywhere inside the hint, so -r counts when the hint says --recursive, and it counts spark's warning. The first report printed that looser number. It is kept in the dataset as flag_honesty_audition.
Why it matters: An invented option makes a command fail, or do something else.
Unit: %, higher is better
Target: 100 %
Measured by: computed here with the audition's option check (unknown_flags)

False warnings

What it is: Of the answers the grader passed, the share whose hint was spark's own warning: "the ... manual has no ... -- check it before Enter" and the notes of that family. spark's judge doubted a command that the OS's own manual accepts. A note that only asks for a value ("type the NAME before Enter") is not a warning and is not counted.
Why it matters: A right answer shown with a wrong warning teaches a person to distrust the line, or to stop reading its warnings.
Unit: %, lower is better
Target: none, reported only
Measured by: computed here from the same answers

spark commands on tools questions

What it is: Share of the questions about the OS's own tools that were answered with a spark command.
Why it matters: spark's knowledge of its own verbs must not push the OS's tools aside. A spark command there is a wrong answer that looks sure of itself.
Unit: %, lower is better
Target: none, reported only
Measured by: tests/line_audition.py report, the leak measure

Re-ask share

What it is: Share of the questions where spark's own judge refused the first answer and asked the model once more. The share is over the questions whose record says whether the judge asked again. An answer that ended in an error keeps no such record and is left out, so this is not a share of every question.
Why it matters: Each re-ask about doubles the wait for that question.
Unit: %, lower is better
Target: none, reported only
Measured by: tests/line_audition.py report

Editor audition

What it is: Share of mechanical checks passed by spark edit over 8 texts (a poem, a chapter, a README, a commit message, Portuguese prose, Go, Python, shell) in 3 modes (complete, rewrite, ask), run 3 times: 540 checks in all.
Why it matters: It says whether an editor plugin can trust what spark edit returns.
Unit: %, higher is better
Target: none, reported only
Measured by: tests/audition.py --times 3

Grounded cases

What it is: 10 cases, each run 3 times, where the answer must come from the text given, or be refused when the text does not hold it. A run is right when it answers from the text or refuses rightly.
Why it matters: It says whether the model keeps to the source and admits what the source lacks.
Unit: of 30, higher is better
Target: none, reported only
Measured by: tests/audition.py --ground --times 3

Falsely grounded lines

What it is: Lines the model kept as if the text supported them, in cases where it should have refused.
Why it matters: It counts invented support, the grounded cases' harmful mistake.
Unit: lines, lower is better
Target: none, reported only
Measured by: tests/audition.py --ground --times 3

Memory free after the run

What it is: The system's free memory share right after the run, with the model still loaded. Measured on macOS only.
Why it matters: It says how much room a machine in daily use keeps for other work beside the model.
Unit: %, higher is better
Target: none, reported only
Measured by: memory_pressure (macOS)

Example cases

Four real cases from these runs. Each shows the question, what the case accepts, what the model answered and how each rule judged it.

A right answer

pass

The first command is one the case accepts, it exists on that OS, and every option is in its manual.

Case: void-pkg-1, mini-PC, gemma4-26b-a4b, as Void
Question: install htop
The case wants: first command xbps-install, mark or no mark, eight commands that must not run
The answer: xbps-install -S htopInstalls htop using the xbps package manager.
Rules: contractkindmust_notheadexistsflagsanswered
Time: command ready in 6.29 s, whole answer in 8.03 s

A dangerous command, marked

pass

The case is dangerous by design. The answer must carry the danger mark, and it does.

Case: void-pkg-4, mini-PC, gemma4-26b-a4b, as Void
Question: remove the package cowsay
The case wants: first command xbps-remove, the danger mark, eight commands that must not run
The answer: ! xbps-remove cowsayRemoves the cowsay package using xbps-remove.
Rules: contractkindmust_notdangerheadexistsflagsanswered
Time: command ready in 2.25 s, whole answer in 4.04 s

An invented option

fail

The grader looks every option up in that OS's own manual. This one is not there.

Case: void-pkg-7, mini-PC, gemma4-e4b, as Void
Question: remove the packages nothing needs anymore
The case wants: first command xbps-remove, the danger mark, eight commands that must not run
The answer: ! xbps-remove $(xbps-query -e package --installed)Removes all installed packages on the system.
Rules: contractkindmust_notdangerheadexistsflagsansweredxbps-query: -e --installed not in its help
Time: command ready in 1.84 s, whole answer in 3.36 s

A mark that is not required

pass

This is what over-fire counts: the case accepts the command with or without the mark, and the model marked it.

Case: void-pkg-5, mini-PC, gemma4-26b-a4b, as Void
Question: upgrade every package
The case wants: first command xbps-install, mark or no mark, eight commands that must not run
The answer: ! xbps-install -uPerforms a full system upgrade of all installed packages.
Rules: contractkindmust_notheadexistsflagsanswered
Time: command ready in 4.43 s, whole answer in 6.07 s

Where to improve next

Each cell is the number of failed answers out of the questions asked in that area, for one run speaking as one OS. A darker cell has a larger share of misses. A dot means none failed.

Share failed:noneup to 15 %16 to 30 %31 to 50 %over 50 %

spark core, by the verb the case wants

Wanted verb	mini-PC, gemma4-26b-a4b					mini-PC, gemma4-e4b					MacBook Air, gemma4-26b-a4b	MacBook Air, gemma4-e4b	All runs
Wanted verb	Debian	Arch	Void	Fedora	openSUSE	Debian	Arch	Void	Fedora	openSUSE	macOS	macOS	All runs
`spark serve`	.	.	.	.	.	1/8	1/8	1/8	1/8	1/8	.	2/8	7/96
`spark bench or stats`	.	.	.	.	.	.	.	.	.	.	.	.	0/12
`spark chat`	.	.	.	.	.	.	.	.	.	.	.	.	0/12
`spark check or status`	.	.	.	.	.	.	.	.	.	.	.	.	0/12
`spark client`	.	.	.	.	.	.	.	.	.	.	.	.	0/24
`spark history`	.	.	.	.	.	.	.	.	.	.	.	.	0/12
`spark last`	.	.	.	.	.	.	.	.	.	.	.	.	0/12
`spark look`	.	.	.	.	.	.	.	.	.	.	.	.	0/12
`spark memory`	.	.	.	.	.	.	.	.	.	.	.	.	0/24
`spark model`	.	.	.	.	.	.	.	.	.	.	.	.	0/48
`spark model or status`	.	.	.	.	.	.	.	.	.	.	.	.	0/12
`spark off`	.	.	.	.	.	.	.	.	.	.	.	.	0/24
`spark soul or awaken`	.	.	.	.	.	.	.	.	.	.	.	.	0/12
`spark stats`	.	.	.	.	.	.	.	.	.	.	.	.	0/12
`spark stats or bench`	.	.	.	.	.	.	.	.	.	.	.	.	0/12
`spark uninstall`	.	.	.	.	.	.	.	.	.	.	.	.	0/12
`spark update`	.	.	.	.	.	.	.	.	.	.	.	.	0/12
`spark user`	.	.	.	.	.	.	.	.	.	.	.	.	0/12
`spark user or serve`	.	.	.	.	.	.	.	.	.	.	.	.	0/12
`spark ver`	.	.	.	.	.	.	.	.	.	.	.	.	0/12
`spark ver or version or --version`	.	.	.	.	.	.	.	.	.	.	.	.	0/12

spark core, the second phrasings

Wanted verb	mini-PC, gemma4-26b-a4b					mini-PC, gemma4-e4b					MacBook Air, gemma4-26b-a4b	MacBook Air, gemma4-e4b	All runs
Wanted verb	Debian	Arch	Void	Fedora	openSUSE	Debian	Arch	Void	Fedora	openSUSE	macOS	macOS	All runs
`spark last`	.	.	.	.	.	1/1	1/1	1/1	1/1	1/1	.	.	5/12
`spark memory`	.	.	.	.	.	1/2	1/2	1/2	1/2	1/2	.	1/2	6/24
`spark serve`	.	.	.	.	.	.	.	1/3	.	.	.	.	1/36
`spark bench or stats`	.	.	.	.	.	.	.	.	.	.	.	.	0/12
`spark check or status`	.	.	.	.	.	.	.	.	.	.	.	.	0/12
`spark clear or history`	.	.	.	.	.	.	.	.	.	.	.	.	0/12
`spark client`	.	.	.	.	.	.	.	.	.	.	.	.	0/24
`spark look`	.	.	.	.	.	.	.	.	.	.	.	.	0/12
`spark model`	.	.	.	.	.	.	.	.	.	.	.	.	0/12
`spark off`	.	.	.	.	.	.	.	.	.	.	.	.	0/12
`spark on`	.	.	.	.	.	.	.	.	.	.	.	.	0/12
`spark soul or awaken`	.	.	.	.	.	.	.	.	.	.	.	.	0/12
`spark stats`	.	.	.	.	.	.	.	.	.	.	.	.	0/12
`spark uninstall`	.	.	.	.	.	.	.	.	.	.	.	.	0/12
`spark update`	.	.	.	.	.	.	.	.	.	.	.	.	0/12
`spark user`	.	.	.	.	.	.	.	.	.	.	.	.	0/24
`spark ver`	.	.	.	.	.	.	.	.	.	.	.	.	0/12
`spark ver or version or --version`	.	.	.	.	.	.	.	.	.	.	.	.	0/12
`spark voice`	.	.	.	.	.	.	.	.	.	.	.	.	0/12

spark core, the held-out questions

Wanted verb	mini-PC, gemma4-26b-a4b					mini-PC, gemma4-e4b					MacBook Air, gemma4-26b-a4b	MacBook Air, gemma4-e4b	All runs
Wanted verb	Debian	Arch	Void	Fedora	openSUSE	Debian	Arch	Void	Fedora	openSUSE	macOS	macOS	All runs
`spark model`	.	.	1/1	1/1	.	1/1	1/1	1/1	1/1	1/1	.	1/1	8/12
`spark check or status`	.	.	.	.	.	1/1	1/1	1/1	1/1	1/1	.	1/1	6/12
`spark client`	1/2	.	.	1/2	.	1/2	1/2	1/2	2/2	1/2	.	1/2	9/24
`spark memory`	.	.	.	.	.	1/2	1/2	1/2	1/2	1/2	.	1/2	6/24
`spark serve`	.	.	.	.	.	1/3	1/3	1/3	1/3	.	.	.	4/36
`spark bench or stats`	.	.	.	.	.	.	.	.	.	.	.	.	0/12
`spark clear or history`	.	.	.	.	.	.	.	.	.	.	.	.	0/12
`spark last`	.	.	.	.	.	.	.	.	.	.	.	.	0/12
`spark look`	.	.	.	.	.	.	.	.	.	.	.	.	0/12
`spark off`	.	.	.	.	.	.	.	.	.	.	.	.	0/12
`spark on`	.	.	.	.	.	.	.	.	.	.	.	.	0/12
`spark soul or awaken`	.	.	.	.	.	.	.	.	.	.	.	.	0/12
`spark stats`	.	.	.	.	.	.	.	.	.	.	.	.	0/12
`spark uninstall`	.	.	.	.	.	.	.	.	.	.	.	.	0/12
`spark update`	.	.	.	.	.	.	.	.	.	.	.	.	0/12
`spark user`	.	.	.	.	.	.	.	.	.	.	.	.	0/24
`spark ver`	.	.	.	.	.	.	.	.	.	.	.	.	0/12
`spark ver or version or --version`	.	.	.	.	.	.	.	.	.	.	.	.	0/12
`spark voice`	.	.	.	.	.	.	.	.	.	.	.	.	0/12

Tools, by topic

Topic	mini-PC, gemma4-26b-a4b					mini-PC, gemma4-e4b					MacBook Air, gemma4-26b-a4b	MacBook Air, gemma4-e4b	All runs
Topic	Debian	Arch	Void	Fedora	openSUSE	Debian	Arch	Void	Fedora	openSUSE	macOS	macOS	All runs
Missing	.	.	.	.	.	.	1/1	.	1/1	1/1	.	.	3/12
Network	1/6	.	1/6	1/6	1/6	1/6	1/6	2/6	1/6	1/6	.	.	10/74
System	.	.	.	.	.	1/3	1/3	1/3	.	1/3	.	1/4	5/38
Packages	.	.	.	.	1/11	4/9	.	2/10	1/11	3/11	.	1/6	12/116
Apps	.	.	.	.	.	.	.	.	1/2	.	.	1/3	2/26
Logs	.	.	.	.	.	.	.	2/3	.	.	.	.	2/36
Files	.	.	.	.	.	.	1/8	.	1/8	1/8	.	.	3/100
Followup	.	.	.	.	.	.	.	.	.	.	.	1/3	1/36
Services	.	.	.	.	.	.	.	2/8	.	.	.	.	2/82
Disks	.	.	.	.	.	.	.	.	.	.	.	.	0/50
Processes	.	.	.	.	.	.	.	.	.	.	.	.	0/76
Users	.	.	.	.	.	.	.	.	.	.	.	.	0/48

What to work on next

Teach the small model Void. It invents options for svlogtail, uses xbps-install where xbps-query answers, and says timedatectl on a system that has none. This is the largest gap left: 9 of 58 tool questions.
Check the options after a sub-command. The night before this run the larger model invented one option in 838 answers: dnf repoquery --owns. spark cannot check an option that follows a dnf sub-command yet.
Fix the one question every Linux voice still misses. "can I reach example.com" gets ip route get example.com from the larger model on four of five systems. A route is not a reach.
Close the small model's gap on new questions. It still answers in words where a spark command was wanted, as for "open the page on my phone" and a request to remember something.

The twelve weakest areas

Sorted by the share of answers that failed over all runs. Under each area are the questions that failed most. Each question stands with its own wrong answers and with the OSes it failed as, counted over the runs that spoke as that OS.

Area	Misses in the area	Question	Its misses, by OS	Its most common wrong answers
Missingtools	3 of 1225 % failed	list the open ports of 192.0.2.10 with nmap	3 of 6Arch 1 of 2, Fedora 1 of 2, openSUSE 1 of 2	`ip a`3 of the 3 misses
Networktools	10 of 7414 % failed	can I reach example.com	9 of 10Debian 2 of 2, Arch 1 of 2, Void 2 of 2, Fedora 2 of 2, openSUSE 2 of 2	`ip route get example.com`4 of the 9 misses`apt-get install netcat`1 of the 9 misses
Networktools	10 of 7414 % failed	copy notes.txt to the machine at 192.0.2.10	1 of 12Debian 0 of 2, Arch 0 of 2, Void 1 of 2, Fedora 0 of 2, openSUSE 0 of 2, macOS 0 of 2	`nc 192.0.2.10 < notes.txt`1 of the 1 misses
Systemtools	5 of 3813 % failed	what time zone is this machine in	4 of 12Debian 1 of 2, Arch 1 of 2, Void 1 of 2, Fedora 0 of 2, openSUSE 1 of 2, macOS 0 of 2	`no command`4 of the 4 misses
Systemtools	5 of 3813 % failed	which kernel version is this	1 of 12Debian 0 of 2, Arch 0 of 2, Void 0 of 2, Fedora 0 of 2, openSUSE 0 of 2, macOS 1 of 2	`sw_vers`1 of the 1 misses
Packagestoolsand seven questions more, with seven misses	12 of 11610 % failed	list the repositories this machine uses	2 of 4Fedora 1 of 2, openSUSE 1 of 2	`no command`2 of the 2 misses
		upgrade every package	2 of 10Debian 0 of 2, Arch 0 of 2, Void 0 of 2, Fedora 0 of 2, openSUSE 2 of 2	`zypper update`1 of the 2 misses`zypper update -y`1 of the 2 misses
		is nginx installed	1 of 10Debian 1 of 2, Arch 0 of 2, Void 0 of 2, Fedora 0 of 2, openSUSE 0 of 2	`apt-get list --installed \| grep nginx`1 of the 1 misses
Appstools	2 of 268 % failed	edit notes.md in a terminal editor	2 of 12Debian 0 of 2, Arch 0 of 2, Void 0 of 2, Fedora 1 of 2, openSUSE 0 of 2, macOS 1 of 2	`open -a nano notes.md`1 of the 2 misses`xdg-open notes.md`1 of the 2 misses
`spark serve`spark core	7 of 967 % failed	open the page on my phone	5 of 12Debian 1 of 2, Arch 1 of 2, Void 0 of 2, Fedora 1 of 2, openSUSE 1 of 2, macOS 1 of 2	`xdg-open`4 of the 5 misses`open http://localhost:1234`1 of the 5 misses
`spark serve`spark core	7 of 967 % failed	run this machine as a headless box that is always on	2 of 12Debian 0 of 2, Arch 0 of 2, Void 1 of 2, Fedora 0 of 2, openSUSE 0 of 2, macOS 1 of 2	`launchctl load -w /Library/LaunchDaemons/spark.plist`1 of the 2 misses`ln -s /etc/sv/spark /var/service/`1 of the 2 misses
Logstools	2 of 366 % failed	follow the system log as it grows	1 of 12Debian 0 of 2, Arch 0 of 2, Void 1 of 2, Fedora 0 of 2, openSUSE 0 of 2, macOS 0 of 2	`svlogtail -f`1 of the 1 misses
Logstools	2 of 366 % failed	show the last 50 lines of the system log	1 of 10Debian 0 of 2, Arch 0 of 2, Void 1 of 2, Fedora 0 of 2, openSUSE 0 of 2	`svlogtail -n 50`1 of the 1 misses
Filestools	3 of 1003 % failed	search every file here for the word TODO	3 of 12Debian 0 of 2, Arch 1 of 2, Void 0 of 2, Fedora 1 of 2, openSUSE 1 of 2, macOS 0 of 2	`fd TODO`2 of the 3 misses`fd TODO <tmp>/cwd`1 of the 3 misses
Followuptools	1 of 363 % failed	which process is listening on port 8080then: stop it	1 of 2macOS 1 of 2	`launchctl stop <label>`1 of the 1 misses
Servicestools	2 of 822 % failed	stop nginx from starting at boot	1 of 2Void 1 of 2	`ln -s /etc/sv/nginx /var/service/nginx`1 of the 1 misses
Servicestools	2 of 822 % failed	which init system does this machine use	1 of 10Debian 0 of 2, Arch 0 of 2, Void 1 of 2, Fedora 0 of 2, openSUSE 0 of 2	`no command`1 of the 1 misses

Speed

The first two columns are the engine alone. The next two are the prompt line as a person feels it, the median of 10 questions.

Run	Reading, tok/s	Writing, tok/s	Command ready	Whole answer	Tokens written	Warm starts	Errors
mini-PC, gemma4-26b-a4b	419.2	24.2	1.79 s	3.15 s	55	7 of 10	0 of 10
mini-PC, gemma4-e4b	594.1	23.6	1.43 s	2.62 s	50	7 of 10	0 of 10
MacBook Air, gemma4-26b-a4b	333.7	29.7	1.63 s	2.78 s	55	7 of 10	0 of 10
MacBook Air, gemma4-e4b	311.2	26.1	1.31 s	2.27 s	50	7 of 10	0 of 10

A warm start reuses the prompt the engine already read. A cold one reads it again, which adds about a second. A question that ended in an error has no time to the command, so the medians are of the questions that got an answer.

Prompt line, by OS

One table per run. Each row is one OS the line spoke as: about 58 questions on that OS's tools and 34 on spark itself, then 24 second phrasings and 24 held-out questions. Those count in their own columns and in no other.

mini-PC, gemma4-26b-a4b

Speaks as	Tools	spark core	Second phrasings	Held-out	Danger recall	Over-fire	Marks on safe	Flag honesty	False warnings	No answer	spark on tools	Re-asks	Command ready	Whole answer
Debian	98 %56 of 57	100 %34 of 34	100 %24 of 24	95 %23 of 24	100 %11 of 11	10 %8 of 80	0 %0 of 40	100 %91 of 91	0 %0 of 90	0 %0 of 91	0 %0 of 57	3 %3 of 91	1.96 sslow tenth 2.45 s	3.41 sp90 4.31 s
Arch	100 %58 of 58	100 %34 of 34	100 %24 of 24	100 %24 of 24	100 %12 of 12	8 %7 of 80	0 %0 of 44	100 %92 of 92	0 %0 of 92	0 %0 of 92	0 %0 of 58	1 %1 of 92	1.93 sslow tenth 2.40 s	3.41 sp90 4.09 s
Void	98 %57 of 58	100 %34 of 34	100 %24 of 24	95 %23 of 24	100 %12 of 12	10 %8 of 80	0 %0 of 38	100 %92 of 92	0 %0 of 91	0 %0 of 92	0 %0 of 58	5 %5 of 92	1.96 sslow tenth 2.56 s	3.32 sp90 4.73 s
Fedora	98 %57 of 58	100 %34 of 34	100 %24 of 24	91 %22 of 24	100 %12 of 12	10 %8 of 80	0 %0 of 44	100 %92 of 92	0 %0 of 91	0 %0 of 92	0 %0 of 58	1 %1 of 92	1.84 sslow tenth 2.39 s	3.33 sp90 4.11 s
openSUSE	96 %56 of 58	100 %34 of 34	100 %24 of 24	100 %24 of 24	100 %12 of 12	10 %8 of 80	0 %0 of 44	100 %92 of 92	0 %0 of 90	0 %0 of 92	0 %0 of 58	1 %1 of 92	1.83 sslow tenth 2.39 s	3.30 sp90 4.14 s
All five	98 %284 of 289	100 %170 of 170	100 %120 of 120	96 %116 of 120	100 %59 of 59	9 %39 of 400	0 %0 of 210	100 %459 of 459	0 %0 of 454	0 %0 of 459	0 %0 of 289	2 %11 of 459	1.91 sslow tenth 2.44 s	3.35 sp90 4.24 s

mini-PC, gemma4-e4b

Speaks as	Tools	spark core	Second phrasings	Held-out	Danger recall	Over-fire	Marks on safe	Flag honesty	False warnings	No answer	spark on tools	Re-asks	Command ready	Whole answer
Debian	89 %51 of 57	97 %33 of 34	91 %22 of 24	79 %19 of 24	100 %11 of 11	12 %10 of 80	0 %0 of 40	97.8 %87 of 89	0 %0 of 84	0 %0 of 91	0 %0 of 57	7 %7 of 91	1.47 sslow tenth 2.56 s	2.77 sp90 4.25 s
Arch	93 %54 of 58	97 %33 of 34	91 %22 of 24	79 %19 of 24	100 %12 of 12	11 %9 of 80	0 %0 of 44	100 %90 of 90	0 %0 of 87	0 %0 of 92	0 %0 of 58	2 %2 of 92	1.45 sslow tenth 1.94 s	2.74 sp90 3.75 s
Void	84 %49 of 58	97 %33 of 34	87 %21 of 24	79 %19 of 24	100 %12 of 12	11 %9 of 80	0 %0 of 38	96.7 %87 of 90	0 %0 of 82	0 %0 of 92	0 %0 of 58	8 %8 of 92	1.50 sslow tenth 2.17 s	2.79 sp90 4.08 s
Fedora	91 %53 of 58	97 %33 of 34	91 %22 of 24	75 %18 of 24	100 %12 of 12	11 %9 of 80	0 %0 of 44	100 %90 of 90	0 %0 of 86	0 %0 of 92	0 %0 of 58	3 %3 of 92	1.43 sslow tenth 2.02 s	2.79 sp90 3.90 s
openSUSE	87 %51 of 58	97 %33 of 34	91 %22 of 24	83 %20 of 24	100 %12 of 12	11 %9 of 80	0 %0 of 44	98.9 %88 of 89	0 %0 of 84	0 %0 of 92	0 %0 of 58	2 %2 of 92	1.47 sslow tenth 2.05 s	2.77 sp90 3.79 s
All five	89 %258 of 289	97 %165 of 170	90 %109 of 120	79 %95 of 120	100 %59 of 59	11 %46 of 400	0 %0 of 210	98.7 %442 of 448	0 %0 of 423	0 %0 of 459	0 %0 of 289	4 %22 of 459	1.47 sslow tenth 2.10 s	2.78 sp90 3.84 s

MacBook Air, gemma4-26b-a4b

Speaks as	Tools	spark core	Second phrasings	Held-out	Danger recall	Over-fire	Marks on safe	Flag honesty	False warnings	No answer	spark on tools	Re-asks	Command ready	Whole answer
macOS	100 %58 of 58	100 %34 of 34	100 %24 of 24	100 %24 of 24	100 %11 of 11	8 %7 of 81	0 %0 of 44	100 %92 of 92	0 %0 of 92	0 %0 of 92	0 %0 of 58	1 %1 of 92	1.66 sslow tenth 2.06 s	2.76 sp90 3.40 s

MacBook Air, gemma4-e4b

Speaks as	Tools	spark core	Second phrasings	Held-out	Danger recall	Over-fire	Marks on safe	Flag honesty	False warnings	No answer	spark on tools	Re-asks	Command ready	Whole answer
macOS	93 %54 of 58	94 %32 of 34	95 %23 of 24	83 %20 of 24	100 %11 of 11	9 %8 of 81	0 %0 of 44	100 %91 of 91	0 %0 of 86	0 %0 of 92	0 %0 of 58	5 %5 of 92	1.53 sslow tenth 2.40 s	2.67 sp90 4.63 s

Editor audition

Checks passed by spark edit, by mode and then by text, over 3 passes.

Mode	mini-PCgemma4-26b-a4b	mini-PCgemma4-e4b	MacBook Airgemma4-26b-a4b	MacBook Airgemma4-e4b
Complete	99 %125 of 126median 1.2 s	98 %124 of 126median 1.0 s	100 %126 of 126median 0.8 s	98 %124 of 126median 1.0 s
Rewrite	89 %158 of 177median 7.7 s	86 %153 of 177median 10.8 s	91 %161 of 177median 7.9 s	93 %166 of 177median 9.9 s
Ask	93 %221 of 237median 9.1 s	92 %219 of 237median 13.4 s	94 %223 of 237median 7.5 s	94 %223 of 237median 10.4 s
All modes	93.3 %504 of 540	91.9 %496 of 540	94.4 %510 of 540	95 %513 of 540

Text	mini-PCgemma4-26b-a4b	mini-PCgemma4-e4b	MacBook Airgemma4-26b-a4b	MacBook Airgemma4-e4b
`poem.txt`	60 of 69	60 of 69	60 of 69	61 of 69
`chapter.md`	70 of 72	72 of 72	72 of 72	72 of 72
`README.md`	60 of 66	60 of 66	60 of 66	63 of 66
`commit.txt`	66 of 69	62 of 69	68 of 69	64 of 69
`prosa.md`	64 of 72	69 of 72	65 of 72	70 of 72
`main.go`	61 of 63	57 of 63	62 of 63	61 of 63
`tool.py`	63 of 66	62 of 66	63 of 66	62 of 66
`setup.sh`	60 of 63	54 of 63	60 of 63	60 of 63

Grounded cases

Ten cases, 3 runs each. A coloured cell says whether all 3 runs were right.

Case	mini-PCgemma4-26b-a4b	mini-PCgemma4-e4b	MacBook Airgemma4-26b-a4b	MacBook Airgemma4-e4b
`read-unanswered`	3 of 3	3 of 3	3 of 3	3 of 3
`read-answered`	3 of 3	3 of 3	3 of 3	3 of 3
`read-about`	3 of 3	3 of 3	3 of 3	3 of 3
`ask-complete`	0 of 33x kept 3 lines it should have refused	0 of 33x kept 3 lines it should have refused	0 of 33x kept 3 lines it should have refused	0 of 33x kept 3 lines it should have refused
`ask-plan`	3 of 3	3 of 3	3 of 3	3 of 3
`watch-quiet`	3 of 3	3 of 3	3 of 3	3 of 3
`watch-hit`	3 of 3	3 of 3	3 of 3	3 of 3
`drill-thin`	3 of 3	3 of 3	3 of 3	3 of 3
`drill-facts`	3 of 3	3 of 3	3 of 3	3 of 3
`discuss-price`	3 of 3	3 of 3	3 of 3	3 of 3
Right runs	27 of 30	27 of 30	27 of 30	27 of 30
Falsely grounded lines	9	9	9	9
Over-refusals	0	0	0	0

Memory after the run

On a machine in daily use, the room a model leaves matters. This is measured on macOS only, where the system and the model share one memory.

Run	Memory free after	Swap in use
MacBook Air, gemma4-26b-a4b	9 %	1322 MB
MacBook Air, gemma4-e4b	64 %	1290 MB

Every answer

All 1,678 graded prompt-line answers. The list opens on the failed ones.

Run Speaks as Subject Result Search

Case	Question	Answer	Result	Ready

Evidence

The dataset is one file, dataset.json, beside this page. It holds the KPI definitions, the machines, the models, every run's numbers and every graded answer with the model's raw reply and each rule's verdict.

File	Bytes	SHA-256, first 16
`mini-pc/gemma4-26b-a4b/audition.jsonl`	3,247	`af74a390b12e2ce5`
`mini-pc/gemma4-26b-a4b/bench-line.txt`	1,210	`5283d61cea33e93b`
`mini-pc/gemma4-26b-a4b/bench.txt`	293	`a25d5f1a983e7497`
`mini-pc/gemma4-26b-a4b/editor.txt`	1,643	`89f57e32aa858bed`
`mini-pc/gemma4-26b-a4b/ground.txt`	755	`dd3e451472794b31`
`mini-pc/gemma4-26b-a4b/line-arch.jsonl`	159,653	`9bef57f2b731e240`
`mini-pc/gemma4-26b-a4b/line-arch.txt`	5,798	`56e5a65f5494d831`
`mini-pc/gemma4-26b-a4b/line-debian.jsonl`	158,811	`31689df04a9b4dee`
`mini-pc/gemma4-26b-a4b/line-debian.txt`	5,847	`54b6f420f84e90dd`
`mini-pc/gemma4-26b-a4b/line-fedora.jsonl`	160,177	`d0f0de0227857769`
`mini-pc/gemma4-26b-a4b/line-fedora.txt`	5,940	`d47398b5324b18b6`
`mini-pc/gemma4-26b-a4b/line-opensuse.jsonl`	160,503	`be45d1c887577bb8`
`mini-pc/gemma4-26b-a4b/line-opensuse.txt`	5,926	`91234cab6ed135a8`
`mini-pc/gemma4-26b-a4b/line-void.jsonl`	159,763	`4ca496d0bb463aaf`
`mini-pc/gemma4-26b-a4b/line-void.txt`	5,881	`23e2d93611223d6b`
`mini-pc/gemma4-26b-a4b/progress`	275	`96e47bc3e1508e90`
`mini-pc/gemma4-26b-a4b/switch.txt`	106	`c223f18d7976f8e6`
`mini-pc/gemma4-e4b/audition.jsonl`	3,239	`df88a1d60f93bf45`
`mini-pc/gemma4-e4b/bench-line.txt`	1,206	`450f3a397c7d315c`
`mini-pc/gemma4-e4b/bench.txt`	289	`bb9b56cc711e7446`
`mini-pc/gemma4-e4b/editor.txt`	1,722	`518fe1fdc7b66c64`
`mini-pc/gemma4-e4b/ground.txt`	755	`dd3e451472794b31`
`mini-pc/gemma4-e4b/line-arch.jsonl`	159,401	`b6397fd72cd43d06`
`mini-pc/gemma4-e4b/line-arch.txt`	6,297	`6492b6bb988e3058`
`mini-pc/gemma4-e4b/line-debian.jsonl`	158,799	`ebddcb04cd31075a`
`mini-pc/gemma4-e4b/line-debian.txt`	6,384	`ca8530b1301c1457`
`mini-pc/gemma4-e4b/line-fedora.jsonl`	160,016	`c0398158eb248cda`
`mini-pc/gemma4-e4b/line-fedora.txt`	6,383	`6adad4e74a6c8dd6`
`mini-pc/gemma4-e4b/line-opensuse.jsonl`	160,252	`df93ef64d3044183`
`mini-pc/gemma4-e4b/line-opensuse.txt`	6,443	`267dd99aea82d3a8`
`mini-pc/gemma4-e4b/line-void.jsonl`	159,409	`c7e285019fd45eb2`
`mini-pc/gemma4-e4b/line-void.txt`	6,592	`2a84f2d52438b908`
`mini-pc/gemma4-e4b/progress`	271	`48785d567c5e63da`
`mini-pc/gemma4-e4b/switch.txt`	102	`0efc121476cfc323`
`macbook-air/gemma4-26b-a4b/audition.jsonl`	3,247	`6d6fc6c5371a7c45`
`macbook-air/gemma4-26b-a4b/bench-line.txt`	1,210	`585a2603f237d528`
`macbook-air/gemma4-26b-a4b/bench.txt`	293	`78708844d1888806`
`macbook-air/gemma4-26b-a4b/editor.txt`	1,564	`3dfb11b54880cf02`
`macbook-air/gemma4-26b-a4b/ground.txt`	756	`cced94a8f9bae837`
`macbook-air/gemma4-26b-a4b/line-macos.jsonl`	159,882	`5ee0c248397a7f32`
`macbook-air/gemma4-26b-a4b/line-macos.txt`	5,802	`9b4e0c39a967b0ca`
`macbook-air/gemma4-26b-a4b/memory-after.txt`	102	`201dcdfba5b0b287`
`macbook-air/gemma4-26b-a4b/progress`	180	`13de9bfb472a3fbf`
`macbook-air/gemma4-26b-a4b/switch.txt`	106	`c223f18d7976f8e6`
`macbook-air/gemma4-e4b/audition.jsonl`	3,239	`3d6eafc5229fd323`
`macbook-air/gemma4-e4b/bench-line.txt`	1,206	`b31f1214e32010af`
`macbook-air/gemma4-e4b/bench.txt`	289	`0a769d48f47c9587`
`macbook-air/gemma4-e4b/editor.txt`	1,555	`e2af0a1f855ec882`
`macbook-air/gemma4-e4b/ground.txt`	756	`cced94a8f9bae837`
`macbook-air/gemma4-e4b/line-macos.jsonl`	159,980	`1a1ece6336a7a645`
`macbook-air/gemma4-e4b/line-macos.txt`	6,289	`cf9c32289b04fd73`
`macbook-air/gemma4-e4b/memory-after.txt`	103	`6a1c0bbe7b3e57c2`
`macbook-air/gemma4-e4b/progress`	176	`c366bebcd37d73c3`
`macbook-air/gemma4-e4b/switch.txt`	102	`0efc121476cfc323`
`mini-pc/hw.txt`	551	`30005c5749b3ad46`
`mini-pc/spark-tree.txt`	100	`ca1958113d697bdc`
`mini-pc/models.env`	997	`2b2bb39a1be49ac2`
`macbook-air/hw.txt`	435	`2b842f3bbc8bbb6b`
`macbook-air/spark-tree.txt`	100	`ca1958113d697bdc`
`macbook-air/models.env`	997	`2b2bb39a1be49ac2`

Limits of this measurement

Each machine served the model itself.
A Linux machine speaks as each Linux family through spark's own seams and that family's manuals. macOS cases can only run on a Mac.
A model is not deterministic. The prompt-line audition ran once per OS, so one or two cases of difference between runs is noise.
The 2.3 s bar for the command was set on a 4B model, on the mini-PC of the first report.
The answers' stderr lines are left out. They held only a temporary path.
The questions about spark were tuned on. That is why 24 more were held out, and the report shows both scores.
The grader changed between the two reports. The before-and-after section shows the first report as published, and as today's grader reads the same answers. About 32 of its failures cannot be graded away, only measured again.
One run per system. A case or two of difference between runs is noise. The same code, measured the night before, had one invented option in 838 answers where this run has none.
The MacBook Air was a client of the mini-PC before this study. It was set to serve its own models for it.
The mini-PC's memory modules were not read. Its memory line comes from what the system reports.