Tokens per second, by tested combination.
Our output-speed results only. Different workloads are separate rows; missing measurements are not zero. Published reference scores remain in the detailed comparison below.
| Machine | Model / runtime | Route | Workload | Decode tok/s | Aggregate tok/s |
|---|
The model is part of the result.
24B-A2B, 27B, 35B-A3B and 120B-A12B are different models. GGUF, EXL3 and NVFP4 are different weight formats. PAIR is a router, not a model; Hermes and Codex are clients.
Earlier j9 tuning configurations—kept separate from the selected model settings
Both PAIR and direct endpoints—not a route A/B test.
Host, client location, request route and lifecycle owner are separate facts. “Bypasses PAIR” does not mean the benchmark client ran on the GPU machine, or that no gateway was involved.
All scores, with their conditions.
Local runs, 0xSero’s registry recipes and other attributed references. Read the method under each number: bounded decode, streamed text and aggregate output are different measurements.
| Model / setup | Evidence | Request route | Decode tokens/s | Input tokens/s | Context tokens | Checks | Agent seconds | Aggregate tokens/s |
|---|
What the comparison supports
Results with their model attached.
Short, bounded samples. Different models and routes are not a controlled hardware comparison. Decode speed excludes first-token latency; workflow time includes tool work.
What PAIR did—and did not—prove.
The Spark GGUF agent run, the j9 PAIR checks, and the direct Spark NVFP4 suite are separate evidence.
Every run, including the failures.
Each row identifies its exact model and configuration. Historical failures are retained, not silently replaced by later passes.
| Experiment | Model / route | Outcome | Evidence / boundary |
|---|
Pinned serving configurations.
Explicit model settings and immutable runtime pins. This is a recorded snapshot, not a live service monitor.
Separate suites. Explicit targets.
Independent workflow checks
Hardware snapshots—not model names.
Resource figures describe the captured state, not current utilization.
Published elsewhere is not measured here.
Sources & methodology
- Single-stream decode and aggregate output are not interchangeable.
- Eight clients queue behind four active vLLM slots; this does not prove eight simultaneous full-context requests.
- Repeated-word retrieval and a synthetic receipt do not establish general reasoning or document quality.
- Prefix caches were not flushed. No confidence intervals or sustained-load results are claimed.
- Service restart was tested; a host reboot was not performed.