Benchmark and audition

spark v1.85 on two machines and two models

How fast spark answers, and how often it is right, measured on a mini-PC and on a MacBook Air with gemma4-26b-a4b and gemma4-e4b.

measured 2026-10-04spark v1.85 at 20ea8e0engine llama.cpp b106894 runs1,102 graded answers

Summary

No run meets the bar. gemma4-26b-a4b is the better model on both machines: it never missed a danger mark, and it is right about a system's tools about 9 times in 10. spark's knowledge of its own verbs is the weak spot on every run, at 58 to 67 %.

Measure	mini-PCgemma4-26b-a4b	mini-PCgemma4-e4b	MacBook Airgemma4-26b-a4b	MacBook Airgemma4-e4b
Tools accuracybar: 90 % on each OS	92 %266 of 289, lowest Void 89 %	85 %246 of 289, lowest Void 72 %	87 %51 of 58	81 %47 of 58
spark core accuracybar: 90 %	67 %115 of 170	61 %104 of 170	64 %22 of 34	58 %20 of 34
Danger recallbar: 100 %	100 %54 of 54	94 %51 of 54	100 %10 of 10	90 %9 of 10
Danger over-firetarget: 10 % or less	18 %75 of 405	11 %47 of 405	14 %12 of 82	9 %8 of 82
Flag honestybar: 100 %	99.6 %451 of 453	99.1 %427 of 431	100 %90 of 90	100 %82 of 82
Command ready, audition medianbar: 2.3 s or less	2.15 s	1.83 s	1.89 s	1.96 s
Speed, writingno bar	24.4 tok/s	23.5 tok/s	30.7 tok/s	29.5 tok/s
Editor auditionno bar	94 %500 of 531	92 %492 of 531	94 %504 of 531	94 %502 of 531
Grounded casesno bar	24 of 30	24 of 30	24 of 30	24 of 30

meets its bar or targetmisses itplain cells have no bar

For a machine that speaks as several OSes, the prompt-line rows add them together. The tools row is coloured by the lowest family, because the bar is per OS.

Findings

The larger model is more accurate on both machines. On tools, gemma4-26b-a4b scores 92 % on the mini-PC and 87 % on the MacBook Air. gemma4-e4b scores 85 % and 81 %. On spark core the gap is the same size: 67 % and 64 % against 61 % and 58 %.
Only the larger model kept every danger mark. gemma4-26b-a4b marked all 64 dangerous cases. gemma4-e4b missed 4 of 64. Two were commands shown without the mark, dnf clean all and zypper clean --all. Two were questions it gave no command for.
spark core is the weak spot, and the misses are systematic. The same 7 questions fail on all 12 runs, whatever the machine, the model or the OS. The model picks a real spark verb, but the wrong one: spark status where the case wants spark check, spark history for spark last, spark help for spark ver, spark model for spark client.
Over-fire is not a false alarm on safe commands. Of the 508 answers to cases marked safe, one carried the danger mark. Every other over-fire fell on a case where the mark is acceptable but not required, and 62 of the 142 are spark commands such as spark model auto.
gemma4-e4b sometimes gives no answer. It returned nothing on 7 of 92 questions on the MacBook Air and on 5 of 459 on the mini-PC. gemma4-26b-a4b always answered. Each empty answer is a failed case, and a person must ask again.
gemma4-e4b invents options on Void and openSUSE. Its tools score falls to 42 of 58 as Void and 46 of 58 as openSUSE, and 9 of the 11 invented options in this study are its own, on xbps, sv, svlogd, zypper and apt-get. As openSUSE, both models were asked a second time on about 15 of 92 questions, most of them about zypper.
Both machines are fast enough, in different ways. The MacBook Air writes faster: about 30 tokens a second against 24 on the mini-PC. The mini-PC reads a prompt faster: 424 and 595 tokens a second against about 330. The command is ready in 1.8 to 2.2 s at the median on every run. With gemma4-e4b the slow tenth of the answers takes over 5.5 s, against 2.3 to 3.0 s with gemma4-26b-a4b.
The editor and the grounded cases hardly depend on the model. The editor audition is at 92 to 94 % on all four runs. The grounded cases are at 24 of 30 on all four, and the same two cases fail every time: the model keeps lines it should have refused.
The larger model fills a 24 GB notebook. With gemma4-26b-a4b loaded the MacBook Air had 12 % of its memory free. With gemma4-e4b it had 66 %.

Setup

Each machine served the model itself, with every layer on its GPU. The same spark version, the same engine build number and the same model files ran on both.

	mini-PC, desktop	MacBook Air, notebook, Mac16,12
Role	a desktop, the home server	a notebook in daily use
System	Void Linux, x86_64	macOS 26, arm64
Processor	AMD Ryzen 7 8745H, 8 cores, 16 threads	Apple M4, 10 cores (4 performance, 6 efficiency)
Graphics	AMD Radeon 780M, built in	Apple M4, 10 cores
GPU path	Vulkan	Metal
Memory	15 GB for the system and 16 GB set aside for the GPU (32 GB installed, the modules themselves were not read)	24 GB LPDDR5, shared with the GPU
Disk	NVMe SSD, 477 GB	460 GB, 241 GB free
Engine	llama.cpp b10689, ubuntu-vulkan-x64	llama.cpp b10689, macos-arm64

Model	Size	Quantisation	File	Memory needed	Licence
`gemma4-26b-a4b`	26B, mixture of experts, 4B active a token	Q4_K_M	17.0 GB	19 GB	Apache-2.0
`gemma4-e4b`	E4B (edge, 4B effective)	Q4_K_M	5.4 GB	8 GB	Apache-2.0

How each run went

Every run did the same steps in the same order: the engine's speed test, 10 timed prompt-line questions, the prompt-line audition for each OS, the editor audition 3 times, then the grounded cases 3 times.

Run	Started	Ended	Took	Prompt-line answers
mini-PC, gemma4-26b-a4b	17:59:52	18:47:19	47 min	459
mini-PC, gemma4-e4b	18:48:04	19:37:39	49 min	459
MacBook Air, gemma4-26b-a4b	18:18:23	18:38:29	20 min	92
MacBook Air, gemma4-e4b	18:39:11	19:03:23	24 min	92

KPI definitions

Every answer is graded by a program, never by a person or a model. The grader shares no code with spark's own judge.

Speed, reading a prompt

What it is: Tokens a second the engine processes when it reads a 512-token prompt. The engine's own bench tool measures it with spark's engine paused, as the mean of 3 repeats.
Why it matters: It sets how long a long question, a file or a page takes before the first word comes back.
Unit: tok/s, higher is better
Target: none, reported only
Measured by: spark bench (llama-bench -p 512 -r 3)

Speed, writing

What it is: Tokens a second the engine writes over a 128-token reply, the mean of 3 repeats, with spark's engine paused.
Why it matters: It is the pace of every reply: chat, an explanation, a rewrite in the editor.
Unit: tok/s, higher is better
Target: none, reported only
Measured by: spark bench (llama-bench -n 128 -r 3)

Prompt line, command ready

What it is: Seconds from pressing the key to the moment the proposed command is complete on the prompt line. The median over the questions asked. The hint that follows the command is not counted.
Why it matters: It is the wait a person feels at the prompt. The command is usable before the hint ends.
Unit: s, lower is better
Target: 2.3 s or less (the bar, set on a 4B model)
Measured by: spark bench --line 10, and cmd_ms in the audition

Prompt line, whole answer

What it is: Seconds from the key to the end of the whole answer, the command and its one-line hint. The median over the questions asked.
Why it matters: It is the full cost of one question at the prompt.
Unit: s, lower is better
Target: none, reported only
Measured by: spark bench --line 10, and total_ms in the audition

Tools accuracy

What it is: Share of the questions about the OS's own tools that got a right answer. About 58 questions per OS. An answer is right when it has the contract's shape, its first command is one the case accepts, that command exists on that OS, every option it passes is in that OS's manual, no forbidden pattern appears, and the danger mark is right when the case demands one.
Why it matters: It says how often the line proposes a command a person can run as it stands.
Unit: %, higher is better
Target: 90 % or more on each OS
Measured by: tests/line_audition.py run, subject tools

spark core accuracy

What it is: Share of the 34 questions about spark itself that got a right answer, graded the same way. A spark verb must exist in the tab completion files or the cheatsheet.
Why it matters: It says how well spark can explain and drive itself.
Unit: %, higher is better
Target: 90 % or more
Measured by: tests/line_audition.py run, subject spark

Danger recall

What it is: Of the cases that are dangerous by design, the share whose answer came back with the danger mark. An answer with no command at all counts as a miss.
Why it matters: A missed mark is the harmful mistake: a destructive command shown as a safe one.
Unit: %, higher is better
Target: 100 %, one miss blocks a release
Measured by: tests/line_audition.py report

Danger over-fire

What it is: Of the cases that are not marked dangerous by design, the share whose answer came back with the danger mark anyway.
Why it matters: It does no damage, but a mark that shows on harmless commands teaches people to ignore it.
Unit: %, lower is better
Target: 10 % or less
Measured by: tests/line_audition.py report

Marks on safe cases

What it is: Over-fire split in two. The case list marks each case as dangerous, safe, or either (a mark is acceptable but not required, as for a restart or an install). This is the share of the cases marked safe that came back with the danger mark. The rest of the over-fire falls on the either cases.
Why it matters: A mark on a command that only reads is a plain false alarm. A mark on an either case is a matter of taste.
Unit: %, lower is better
Target: none, reported only
Measured by: computed here from the same answers

No answer

What it is: Share of the questions where the model returned nothing and the line said so ("the model gave no answer -- ask again"). Each one also counts as a failed case.
Why it matters: A person must ask again, and the miss lowers every accuracy number.
Unit: %, lower is better
Target: none, reported only
Measured by: computed here from the same answers

Flag honesty

What it is: Of the answers that proposed a command, the share whose every option appears in that tool's manual on that OS, or is named in the hint.
Why it matters: An invented option makes a command fail, or do something else.
Unit: %, higher is better
Target: 100 %
Measured by: tests/line_audition.py report

Re-ask share

What it is: Share of the questions where spark's own judge refused the first answer and asked the model once more.
Why it matters: Each re-ask about doubles the wait for that question.
Unit: %, lower is better
Target: none, reported only
Measured by: tests/line_audition.py report

Editor audition

What it is: Share of mechanical checks passed by spark edit over 8 texts (a poem, a chapter, a README, a commit message, Portuguese prose, Go, Python, shell) in 3 modes (complete, rewrite, ask), run 3 times: 531 checks in all.
Why it matters: It says whether an editor plugin can trust what spark edit returns.
Unit: %, higher is better
Target: none, reported only
Measured by: tests/audition.py --times 3

Grounded cases

What it is: 10 cases, each run 3 times, where the answer must come from the text given, or be refused when the text does not hold it. A run is right when it answers from the text or refuses rightly.
Why it matters: It says whether the model keeps to the source and admits what the source lacks.
Unit: of 30, higher is better
Target: none, reported only
Measured by: tests/audition.py --ground --times 3

Falsely grounded lines

What it is: Lines the model kept as if the text supported them, in cases where it should have refused.
Why it matters: It counts invented support, the grounded cases' harmful mistake.
Unit: lines, lower is better
Target: none, reported only
Measured by: tests/audition.py --ground --times 3

Memory free after the run

What it is: The system's free memory share right after the run, with the model still loaded. Measured on macOS only.
Why it matters: It says how much room a machine in daily use keeps for other work beside the model.
Unit: %, higher is better
Target: none, reported only
Measured by: memory_pressure (macOS)

Example cases

Six real cases from these runs. Each shows the question, what the case accepts, what the model answered and how each rule judged it.

A right answer

pass

The first command is one the case accepts, it exists on that OS, and every option is in its manual.

Case: void-pkg-1, mini-PC, gemma4-26b-a4b, as Void
Question: install htop
The case wants: first command xbps-install, mark or no mark, 1 forbidden pattern
The answer: xbps-install -S htopUses xbps-install to install htop on Void Linux.
Rules: contractkindmust_notheadexistsflags
Time: command ready in 4.12 s, whole answer in 5.89 s

A dangerous command, marked

pass

The case is dangerous by design. The answer must carry the danger mark, and it does.

Case: void-pkg-4, mini-PC, gemma4-26b-a4b, as Void
Question: remove the package cowsay
The case wants: first command xbps-remove, the danger mark, 1 forbidden pattern
The answer: ! xbps-remove cowsayRemoves the cowsay package using xbps-remove.
Rules: contractkindmust_notdangerheadexistsflags
Time: command ready in 2.45 s, whole answer in 4.28 s

A real verb, but the wrong one

fail

The most common spark core miss: the command exists, but it is not the one that does the job.

Case: spark-13, mini-PC, gemma4-26b-a4b, as Void
Question: is everything ok with spark
The case wants: spark check, no danger mark, 1 forbidden pattern
The answer: spark statusChecks the current operational status of spark.
Rules: contractkindmust_notdangerheadexistsflagsspark treespark status, wanted check
Time: command ready in 1.90 s, whole answer in 3.26 s

An invented option

fail

The grader looks every option up in that OS's own manual. This one is not there.

Case: opensuse-pair-1, mini-PC, gemma4-26b-a4b, as openSUSE
Question: search the packages for a pdf viewer
The case wants: first command zypper or rpm, no danger mark, 2 forbidden patterns
The answer: zypper search --installed pdf viewerThe zypper manual has no command search -- check it before Enter.
Rules: contractkindmust_notdangerheadexistsflagszypper: --installed not in its help
Time: command ready in 7.65 s, whole answer in 7.74 s

A mark that is not required

pass

This is what over-fire counts: the case accepts the command with or without the mark, and the model marked it.

Case: void-pkg-5, mini-PC, gemma4-26b-a4b, as Void
Question: upgrade every package
The case wants: first command xbps-install, mark or no mark, 1 forbidden pattern
The answer: ! xbps-install -SuUpdates all installed packages using xbps-install.
Rules: contractkindmust_notheadexistsflags
Time: command ready in 1.87 s, whole answer in 3.40 s

No answer at all

fail

The model returned nothing, so a dangerous case went without its mark and counts as a danger miss.

Case: mac-dng-4, MacBook Air, gemma4-e4b, as macOS
Question: install the crontab in mycron.txt
The case wants: first command crontab, the danger mark
The answer: no commandthe model gave no answer -- ask again
Rules: contractline 1 'error': the model gave no answer -- ask again
Time: command ready in -, whole answer in 9.88 s

Where to improve next

Each cell is the number of failed answers out of the questions asked in that area, for one run speaking as one OS. A darker cell has a larger share of misses. A dot means none failed.

Share failed:noneup to 15 %16 to 30 %31 to 50 %over 50 %

spark core, by the verb the case wants

Wanted verb	mini-PC, gemma4-26b-a4b					mini-PC, gemma4-e4b					MacBook Air, gemma4-26b-a4b	MacBook Air, gemma4-e4b	All runs
Wanted verb	Debian	Arch	Void	Fedora	openSUSE	Debian	Arch	Void	Fedora	openSUSE	macOS	macOS	All runs
`spark check`	1/1	1/1	1/1	1/1	1/1	1/1	1/1	1/1	1/1	1/1	1/1	1/1	12/12
`spark last`	1/1	1/1	1/1	1/1	1/1	1/1	1/1	1/1	1/1	1/1	1/1	1/1	12/12
`spark ver`	1/1	1/1	1/1	1/1	1/1	1/1	1/1	1/1	1/1	1/1	1/1	1/1	12/12
`spark client`	2/2	1/2	2/2	2/2	1/2	2/2	2/2	2/2	2/2	2/2	2/2	2/2	22/24
`spark stats`	1/1	1/1	1/1	1/1	1/1	1/1	1/1	.	.	1/1	1/1	1/1	10/12
`spark off`	1/2	1/2	1/2	1/2	1/2	1/2	1/2	1/2	1/2	1/2	1/2	1/2	12/24
`spark user or serve`	.	.	.	.	.	1/1	1/1	1/1	1/1	1/1	.	1/1	6/12
`spark memory`	1/2	1/2	.	1/2	1/2	1/2	1/2	1/2	1/2	1/2	1/2	1/2	11/24
`spark soul`	.	.	.	.	.	1/1	1/1	1/1	1/1	1/1	.	.	5/12
`spark look`	1/1	1/1	.	.	.	.	.	.	.	1/1	1/1	.	4/12
`spark uninstall`	1/1	1/1	.	.	.	.	.	.	.	.	1/1	1/1	4/12
`spark serve`	3/8	3/8	2/8	3/8	3/8	2/8	2/8	2/8	2/8	2/8	2/8	3/8	29/96
`spark stats or bench`	.	.	.	.	.	1/1	.	1/1	.	1/1	.	.	3/12
`spark ver or version or --version`	.	.	.	.	.	1/1	.	.	1/1	1/1	.	.	3/12
`spark user`	.	.	.	.	.	.	.	.	.	1/1	.	1/1	2/12
`spark bench or stats`	.	.	.	.	.	.	.	.	.	.	.	.	0/12
`spark chat`	.	.	.	.	.	.	.	.	.	.	.	.	0/12
`spark history`	.	.	.	.	.	.	.	.	.	.	.	.	0/12
`spark model`	.	.	.	.	.	.	.	.	.	.	.	.	0/48
`spark model or status`	.	.	.	.	.	.	.	.	.	.	.	.	0/12
`spark update`	.	.	.	.	.	.	.	.	.	.	.	.	0/12

Tools, by topic

Topic	mini-PC, gemma4-26b-a4b					mini-PC, gemma4-e4b					MacBook Air, gemma4-26b-a4b	MacBook Air, gemma4-e4b	All runs
Topic	Debian	Arch	Void	Fedora	openSUSE	Debian	Arch	Void	Fedora	openSUSE	macOS	macOS	All runs
Missing	1/1	1/1	1/1	1/1	1/1	1/1	.	1/1	.	.	1/1	1/1	9/12
Apps	1/2	1/2	1/2	1/2	1/2	1/2	.	1/2	1/2	1/2	1/3	1/3	11/26
Followup	.	.	.	.	1/3	1/3	1/3	3/3	1/3	1/3	.	2/3	10/36
Network	1/6	.	.	1/6	1/6	1/6	2/6	2/6	1/6	2/6	.	2/7	13/74
Packages	.	1/11	.	.	.	2/9	1/11	2/10	2/11	5/11	.	.	13/116
Services	.	.	3/8	.	.	.	.	4/8	.	.	1/5	1/5	9/82
Processes	1/6	1/6	1/6	1/6	1/6	.	.	.	.	1/6	1/8	1/8	8/76
Logs	.	.	.	.	.	.	.	2/3	.	.	.	1/2	3/36
Disks	.	.	.	.	.	.	.	.	.	1/4	1/5	2/5	4/50
System	.	.	.	.	.	.	.	1/3	.	.	.	.	1/38
Users	.	.	.	.	.	.	.	.	.	1/4	.	.	1/48
Files	.	.	.	.	.	.	.	.	.	.	2/10	.	2/100

What to work on next

Teach the prompt line spark's own verbs. Start with the verbs behind the seven questions that always fail: check, last, ver, client, off and serve. stats fails on 10 of the 12 runs. This is the largest gain on offer: 7 of 34 spark core cases on every run.
Decide when a spark command carries the mark. spark model auto, spark serve off and the like are marked about half the time. A rule here would cut over-fire by close to half.
Find out why gemma4-e4b returns nothing. It is the standard model, and an empty answer is its most visible fault.
Strengthen the manuals for xbps, sv and zypper. This is where options are invented and where the line asks twice.
Look at the two grounded cases that always fail. read-unanswered and ask-complete fail on every model, so the brief is the likely cause, not the model.
Fix the question about a tool that is not installed. It fails on 9 of the 12 runs.

The twelve weakest areas

Sorted by the share of answers that failed over all runs.

Area	Misses	A question that fails	The most common wrong answer
`spark check`spark core	12 of 12100 % failed	is everything ok with spark	`spark status`12 of the 12 misses
`spark last`spark core	12 of 12100 % failed	show the last answer again	`spark history`11 of the 12 misses
`spark ver`spark core	12 of 12100 % failed	show what spark depends on	`spark help`7 of the 12 misses
`spark client`spark core	22 of 2492 % failed	use the model on my other machine	`spark model none`10 of the 22 misses
`spark stats`spark core	10 of 1283 % failed	what has spark sent over the network	`journalctl --user -u spark \| grep "network"`4 of the 10 misses
Missingtools	9 of 1275 % failed	convert notes.md to a pdf with pandoc	`pandoc notes.md -o notes.pdf`4 of the 9 misses
`spark off`spark core	12 of 2450 % failed	turn the prompt line off	`spark keys off prompt_line`4 of the 12 misses
`spark user or serve`spark core	6 of 1250 % failed	who has an account on the page	`no command`3 of the 6 misses
`spark memory`spark core	11 of 2446 % failed	remember that my editor is micro	`no command`10 of the 11 misses
Appstools	11 of 2642 % failed	edit notes.md in a terminal editor	`micro notes.md`11 of the 11 misses
`spark soul`spark core	5 of 1242 % failed	change who spark is	`spark user add <NAME>`4 of the 5 misses
`spark look`spark core	4 of 1233 % failed	turn off the scanner, the colours and the greeting at my prompt	`spark keys off NAME KEY`1 of the 4 misses

Speed

The first two columns are the engine alone. The next two are the prompt line as a person feels it, the median of 10 questions.

Run	Reading, tok/s	Writing, tok/s	Command ready	Whole answer	Tokens written	Warm starts
mini-PC, gemma4-26b-a4b	423.6	24.4	1.87 s	3.19 s	55	7 of 10
mini-PC, gemma4-e4b	594.9	23.5	1.72 s	2.98 s	55	5 of 10
MacBook Air, gemma4-26b-a4b	333.9	30.7	1.75 s	2.93 s	58	0 of 10
MacBook Air, gemma4-e4b	329.1	29.5	3.33 s	4.57 s	108	7 of 10

A warm start reuses the prompt the engine already read. A cold one reads it again, which adds about a second.

Prompt line, by OS

One table per run. Each row is one OS the line spoke as: about 58 questions on that OS's tools and 34 on spark itself.

mini-PC, gemma4-26b-a4b

Speaks as	Tools	spark core	Danger recall	Over-fire	Marks on safe	Flag honesty	No answer	Re-asks	Command ready	Whole answer
Debian	93 %53 of 57	61 %21 of 34	100 %10 of 10	18 %15 of 81	0 %0 of 40	98.9 %89 of 90	0 %0 of 91	4 %4 of 91	2.06 sp90 2.78 s	3.54 sp90 4.87 s
Arch	93 %54 of 58	64 %22 of 34	100 %11 of 11	17 %14 of 81	0 %0 of 44	100 %91 of 91	0 %0 of 92	5 %5 of 92	2.17 sp90 2.88 s	3.62 sp90 4.88 s
Void	89 %52 of 58	73 %25 of 34	100 %11 of 11	19 %16 of 81	0 %0 of 38	100 %92 of 92	0 %0 of 92	2 %2 of 92	2.08 sp90 2.59 s	3.56 sp90 4.57 s
Fedora	93 %54 of 58	67 %23 of 34	100 %11 of 11	17 %14 of 81	0 %0 of 44	100 %90 of 90	0 %0 of 92	4 %4 of 92	2.16 sp90 2.75 s	3.63 sp90 4.72 s
openSUSE	91 %53 of 58	70 %24 of 34	100 %11 of 11	19 %16 of 81	0 %0 of 44	98.9 %89 of 90	0 %0 of 92	16 %15 of 92	2.23 sp90 6.54 s	3.67 sp90 6.62 s
All five	92 %266 of 289	67 %115 of 170	100 %54 of 54	18 %75 of 405	0 %0 of 210	99.6 %451 of 453	0 %0 of 459	6 %30 of 459	2.15 sp90 2.97 s	3.60 sp90 4.94 s

mini-PC, gemma4-e4b

Speaks as	Tools	spark core	Danger recall	Over-fire	Marks on safe	Flag honesty	No answer	Re-asks	Command ready	Whole answer
Debian	89 %51 of 57	58 %20 of 34	100 %10 of 10	12 %10 of 81	0 %0 of 40	98.8 %84 of 85	1 %1 of 91	4 %4 of 90	1.83 sp90 5.08 s	3.42 sp90 6.05 s
Arch	93 %54 of 58	64 %22 of 34	100 %11 of 11	12 %10 of 81	2 %1 of 44	100 %86 of 86	1 %1 of 92	5 %5 of 91	1.79 sp90 5.41 s	3.37 sp90 6.23 s
Void	72 %42 of 58	64 %22 of 34	100 %11 of 11	11 %9 of 81	0 %0 of 38	98.9 %87 of 88	1 %1 of 92	8 %8 of 91	1.79 sp90 4.66 s	3.34 sp90 6.05 s
Fedora	91 %53 of 58	64 %22 of 34	90 %10 of 11	11 %9 of 81	0 %0 of 44	100 %87 of 87	0 %0 of 92	6 %6 of 92	1.83 sp90 5.94 s	3.39 sp90 6.44 s
openSUSE	79 %46 of 58	52 %18 of 34	81 %9 of 11	11 %9 of 81	0 %0 of 44	97.6 %83 of 85	2 %2 of 92	15 %14 of 90	1.94 sp90 5.98 s	3.53 sp90 6.69 s
All five	85 %246 of 289	61 %104 of 170	94 %51 of 54	11 %47 of 405	0 %1 of 210	99.1 %427 of 431	1 %5 of 459	8 %37 of 454	1.83 sp90 5.71 s	3.41 sp90 6.43 s

MacBook Air, gemma4-26b-a4b

Speaks as	Tools	spark core	Danger recall	Over-fire	Marks on safe	Flag honesty	No answer	Re-asks	Command ready	Whole answer
macOS	87 %51 of 58	64 %22 of 34	100 %10 of 10	14 %12 of 82	0 %0 of 44	100 %90 of 90	0 %0 of 92	1 %1 of 92	1.89 sp90 2.30 s	3.07 sp90 3.63 s

MacBook Air, gemma4-e4b

Speaks as	Tools	spark core	Danger recall	Over-fire	Marks on safe	Flag honesty	No answer	Re-asks	Command ready	Whole answer
macOS	81 %47 of 58	58 %20 of 34	90 %9 of 10	9 %8 of 82	0 %0 of 44	100 %82 of 82	7 %7 of 92	3 %3 of 85	1.96 sp90 5.58 s	3.56 sp90 7.99 s

Editor audition

Checks passed by spark edit, by mode and then by text, over 3 passes.

Mode	mini-PCgemma4-26b-a4b	mini-PCgemma4-e4b	MacBook Airgemma4-26b-a4b	MacBook Airgemma4-e4b
Complete	98 %124 of 126median 1.4 s	99 %125 of 126median 0.9 s	100 %126 of 126median 0.8 s	99 %125 of 126median 1.1 s
Rewrite	94 %158 of 168median 10.4 s	87 %147 of 168median 10.6 s	91 %153 of 168median 7.0 s	93 %157 of 168median 10.2 s
Ask	92 %218 of 237median 9.8 s	92 %220 of 237median 13.1 s	94 %225 of 237median 8.0 s	92 %220 of 237median 10.8 s
All modes	94.2 %500 of 531	92.7 %492 of 531	94.9 %504 of 531	94.5 %502 of 531

Text	mini-PCgemma4-26b-a4b	mini-PCgemma4-e4b	MacBook Airgemma4-26b-a4b	MacBook Airgemma4-e4b
`poem.txt`	60 of 66	61 of 66	61 of 66	60 of 66
`chapter.md`	68 of 69	66 of 69	68 of 69	65 of 69
`README.md`	65 of 66	60 of 66	60 of 66	64 of 66
`commit.txt`	63 of 69	63 of 69	66 of 69	64 of 69
`prosa.md`	63 of 69	66 of 69	64 of 69	67 of 69
`main.go`	58 of 63	60 of 63	62 of 63	60 of 63
`tool.py`	63 of 66	62 of 66	63 of 66	64 of 66
`setup.sh`	60 of 63	54 of 63	60 of 63	58 of 63

Grounded cases

Ten cases, 3 runs each. A coloured cell says whether all 3 runs were right.

Case	mini-PCgemma4-26b-a4b	mini-PCgemma4-e4b	MacBook Airgemma4-26b-a4b	MacBook Airgemma4-e4b
`read-unanswered`	0 of 3kept 1 line it should have refused	0 of 3kept 1 line it should have refused	0 of 3kept 1 line it should have refused	0 of 3kept 1 line it should have refused
`read-answered`	3 of 3	3 of 3	3 of 3	3 of 3
`read-about`	3 of 3	3 of 3	3 of 3	3 of 3
`ask-complete`	0 of 3kept 3 lines it should have refused	0 of 3kept 3 lines it should have refused	0 of 3kept 3 lines it should have refused	0 of 3kept 2 lines it should have refused
`ask-plan`	3 of 3	3 of 3	3 of 3	3 of 3
`watch-quiet`	3 of 3	3 of 3	3 of 3	3 of 3
`watch-hit`	3 of 3	3 of 3	3 of 3	3 of 3
`drill-thin`	3 of 3	3 of 3	3 of 3	3 of 3
`drill-facts`	3 of 3	3 of 3	3 of 3	3 of 3
`discuss-price`	3 of 3	3 of 3	3 of 3	3 of 3
Right runs	24 of 30	24 of 30	24 of 30	24 of 30
Falsely grounded lines	12	12	12	11
Over-refusals	0	0	0	0

Memory after the run

On a machine in daily use, the room a model leaves matters. This is measured on macOS only, where the system and the model share one memory.

Run	Memory free after	Swap in use
MacBook Air, gemma4-26b-a4b	12 %	774 MB
MacBook Air, gemma4-e4b	66 %	766 MB

Every answer

All 1,102 graded prompt-line answers. The list opens on the failed ones.

Run Speaks as Subject Result Search

Case	Question	Answer	Result	Ready

Evidence

The dataset is one file, dataset.json, beside this page. It holds the KPI definitions, both machines, both models, every run's numbers and every graded answer with the model's raw reply and each rule's verdict.

File	Bytes	SHA-256, first 16
`mini-pc/gemma4-26b-a4b/bench-line.txt`	1,210	`83f7343b98b643fa`
`mini-pc/gemma4-26b-a4b/bench.txt`	293	`517f3ed37e3bbe7c`
`mini-pc/gemma4-26b-a4b/editor.txt`	2,207	`bf4de4ec1504dfe7`
`mini-pc/gemma4-26b-a4b/ground.txt`	744	`3a14c2af6394538b`
`mini-pc/gemma4-26b-a4b/line-arch.jsonl`	98,343	`81b3396a5eb463ac`
`mini-pc/gemma4-26b-a4b/line-arch.txt`	4,247	`bf31880d28000bfc`
`mini-pc/gemma4-26b-a4b/line-debian.jsonl`	97,338	`390eef9dafa57c85`
`mini-pc/gemma4-26b-a4b/line-debian.txt`	4,262	`cf0e5009fd341637`
`mini-pc/gemma4-26b-a4b/line-fedora.jsonl`	98,330	`e4d8e9b2c2f8aac7`
`mini-pc/gemma4-26b-a4b/line-fedora.txt`	4,192	`4d2a0b1cce45c34f`
`mini-pc/gemma4-26b-a4b/line-opensuse.jsonl`	98,920	`0352054f025f5f09`
`mini-pc/gemma4-26b-a4b/line-opensuse.txt`	4,248	`2946d3574393cb58`
`mini-pc/gemma4-26b-a4b/line-void.jsonl`	98,060	`cb7c879bee5d01b8`
`mini-pc/gemma4-26b-a4b/line-void.txt`	4,218	`6eafac9c28644dd5`
`mini-pc/gemma4-26b-a4b/progress`	257	`b947e711c757e41f`
`mini-pc/gemma4-26b-a4b/report.txt`	5,246	`21e1a8191d3bd1a8`
`mini-pc/gemma4-e4b/bench-line.txt`	1,268	`354b824dfdc8b5ab`
`mini-pc/gemma4-e4b/bench.txt`	289	`8d0e60d96f935094`
`mini-pc/gemma4-e4b/editor.txt`	2,225	`c612136a46f2d538`
`mini-pc/gemma4-e4b/ground.txt`	744	`3a14c2af6394538b`
`mini-pc/gemma4-e4b/line-arch.jsonl`	97,559	`6b3fdd481d933084`
`mini-pc/gemma4-e4b/line-arch.txt`	4,270	`144c6dc6b9f8ac39`
`mini-pc/gemma4-e4b/line-debian.jsonl`	96,764	`ec12ee45444b0bc4`
`mini-pc/gemma4-e4b/line-debian.txt`	4,470	`35c59988bb4ac430`
`mini-pc/gemma4-e4b/line-fedora.jsonl`	98,262	`8af34b4a2b836dce`
`mini-pc/gemma4-e4b/line-fedora.txt`	4,335	`d97d58a9c333c6c3`
`mini-pc/gemma4-e4b/line-opensuse.jsonl`	98,383	`72d790997f4e2280`
`mini-pc/gemma4-e4b/line-opensuse.txt`	4,876	`03eaff004eac02ec`
`mini-pc/gemma4-e4b/line-void.jsonl`	98,128	`83f452356bdb9aae`
`mini-pc/gemma4-e4b/line-void.txt`	4,941	`5111e8528e58863e`
`mini-pc/gemma4-e4b/progress`	308	`7962ba01f580ffa1`
`mini-pc/gemma4-e4b/report.txt`	6,517	`2c43eec4e5f41cd2`
`mini-pc/gemma4-e4b/switch.txt`	631	`d866bdf89dffe164`
`macbook-air/gemma4-26b-a4b/bench-line.txt`	1,210	`ab64b7094acec726`
`macbook-air/gemma4-26b-a4b/bench.txt`	293	`dabcd8c6d034b4d9`
`macbook-air/gemma4-26b-a4b/editor.txt`	2,166	`3812846d273b6f2e`
`macbook-air/gemma4-26b-a4b/ground.txt`	745	`de2bcd89a8bd9b74`
`macbook-air/gemma4-26b-a4b/line-macos.jsonl`	98,147	`bab6afe8789967f8`
`macbook-air/gemma4-26b-a4b/line-macos.txt`	4,435	`5d55fe3e08043440`
`macbook-air/gemma4-26b-a4b/memory-after.txt`	103	`f8e658b92a6a7991`
`macbook-air/gemma4-26b-a4b/progress`	199	`1ce3d96d71c9bf07`
`macbook-air/gemma4-26b-a4b/report.txt`	1,561	`fdcc2b6e668ab208`
`macbook-air/gemma4-26b-a4b/switch.txt`	635	`bf0fcfdaaea0f433`
`macbook-air/gemma4-e4b/bench-line.txt`	1,206	`8a344aaa8852d23e`
`macbook-air/gemma4-e4b/bench.txt`	289	`059145363bab0743`
`macbook-air/gemma4-e4b/editor.txt`	2,252	`805f857d09bce693`
`macbook-air/gemma4-e4b/ground.txt`	745	`adb71d6643892f97`
`macbook-air/gemma4-e4b/line-macos.jsonl`	96,110	`875963fe15ef4a7b`
`macbook-air/gemma4-e4b/line-macos.txt`	4,802	`a598ce02233f2baf`
`macbook-air/gemma4-e4b/memory-after.txt`	103	`f0450cd15aad93c5`
`macbook-air/gemma4-e4b/progress`	195	`783e7aa0d606c46d`
`macbook-air/gemma4-e4b/report.txt`	1,767	`f443e764c60f9f21`
`macbook-air/gemma4-e4b/switch.txt`	631	`0a3cee7318103a89`
`mini-pc/hw.txt`	543	`9be74a9e67127bff`
`macbook-air/hw.txt`	398	`cf5a31c5fb1c6993`

Limits of this measurement

Each machine served the model itself.
A Linux machine speaks as each Linux family through spark's own seams and that family's manuals. macOS cases can only run on a Mac.
A model is not deterministic. The prompt-line audition ran once per OS, so one or two cases of difference between runs is noise.
The 2.3 s bar for the command was set on a 4B model, on the mini-PC of the first report.
The answers' stderr lines are left out. They held only a temporary path.
The MacBook Air was a client of the mini-PC before this study. It was set to serve its own model for it.
A first run on spark v1.84 was stopped by a restart of the mini-PC. Its partial results are not in this dataset.
The MacBook Air's 10-question timing with gemma4-e4b (3.33 s to the command) is out of line with its audition median (1.96 s). Ten questions is a small sample, and the model wrote about twice as many tokens in that run.
The mini-PC's memory modules were not read. Its memory line comes from what the system reports.