Testing platform for ZShot
| Template | Total Runs | Pass | Fail | Pass Rate | Avg Tokens In | Avg Tokens Out | Tokens In per Pass |
|---|---|---|---|---|---|---|---|
| benefits_hardship_claim | 32 | 16 | 16 | 50.0% | 898,848 | 19,789 | 1,235,917 |
| doom_destroy_a_barrel | 15 | 7 | 8 | 46.7% | 208,211 | 5,319 | 446,166 |
| parcel_redelivery | 281 | 116 | 165 | 41.3% | 181,211 | 15,199 | 412,846 |
| recipe_site_favorite_a_recipe | 16 | 8 | 8 | 50.0% | 15,315 | 609 | 15,315 |
Token figures count runs recorded with --llm-usage. Tokens in per pass is every metered run's input over the metered runs that passed.
| ID | Template | Label | Result | Furthest | Tokens | Est. Cost | Took | Created |
|---|---|---|---|---|---|---|---|---|
| 50 | parcel_redelivery | scout-deepseek | fail | done | 120,121 in / 29,604 out | $0.071 | 2m 56s | 2026-09-28 10:56 |
| 49 | parcel_redelivery | deepseek-noreasoning | fail | done | 79,364 in / 1,108 out | $0.018 | 1m 1s | 2026-09-28 10:55 |
| 48 | parcel_redelivery | opus | fail | slot | 373,988 in / 23,506 out | $1.97 | 7m 18s | 2026-09-28 10:52 |
| 47 | parcel_redelivery | deepseek-noreasoning | fail | done | 264,069 in / 3,700 out | $0.068 | 2m 36s | 2026-09-28 10:52 |
| 46 | parcel_redelivery | qwen-noreasoning | fail | preferences | 384,036 in / 4,240 out | $0.000 | 9m 6s | 2026-09-28 10:52 |
| 45 | parcel_redelivery | gemini | fail | preferences | 275,572 in / 145,978 out | $0.754 | 12m 26s | 2026-09-28 10:52 |
| 44 | parcel_redelivery | deepseek | fail | preferences | 235,861 in / 179,575 out | $0.263 | 15m 5s | 2026-09-28 03:51 |
| 43 | parcel_redelivery | scout-sonnet-opus | fail | done | 59,126 in / 3,177 out | $0.033 | 31s | 2026-09-28 03:40 |
| 42 | parcel_redelivery | scout-sonnet-opus | fail | done | 83,382 in / 4,192 out | $0.043 | 37s | 2026-09-28 03:39 |
| 41 | parcel_redelivery | scout-sonnet | fail | done | 60,473 in / 2,835 out | $0.058 | 38s | 2026-09-28 03:38 |
| 40 | parcel_redelivery | scout-sonnet | fail | done | 80,560 in / 3,646 out | $0.042 | 38s | 2026-09-28 03:38 |
| 39 | parcel_redelivery | scout-opus | pass | done | 148,140 in / 4,430 out | $0.424 | 1m 33s | 2026-09-28 03:36 |
| 38 | parcel_redelivery | deepseek | fail | preferences | 264,375 in / 155,574 out | $0.249 | 15m 6s | 2026-09-28 03:35 |
| 37 | parcel_redelivery | scout-opus | pass | done | 164,501 in / 4,978 out | $0.437 | 1m 36s | 2026-09-28 03:35 |
| 36 | parcel_redelivery | opus-thinking | fail | confirm | 173,136 in / 5,492 out | $0.802 | 2m 35s | 2026-09-28 03:32 |
| 35 | parcel_redelivery | scout-qwen | fail | done | 58,573 in / 2,837 out | $0.022 | 36s | 2026-09-28 03:26 |
| 34 | parcel_redelivery | scout-qwen | fail | done | 64,855 in / 2,807 out | $0.021 | 50s | 2026-09-28 03:25 |
| 33 | parcel_redelivery | opus-thinking | fail | confirm | 422,288 in / 24,969 out | $2.19 | 7m 37s | 2026-09-28 03:24 |
| 32 | parcel_redelivery | deepseek-low | fail | preferences | 324,309 in / 129,276 out | $0.229 | 12m 31s | 2026-09-28 03:23 |
| 31 | parcel_redelivery | sonnet | fail | preferences | 152,833 in / 15,462 out | $0.460 | 3m 57s | 2026-09-28 03:21 |
| 30 | parcel_redelivery | scout-gemini | fail | done | 89,950 in / 4,461 out | $0.044 | 44s | 2026-09-28 03:20 |
| 29 | parcel_redelivery | qwen-noreasoning | fail | preferences | 362,284 in / 3,285 out | $0.000 | 8m 44s | 2026-09-28 03:16 |
| 28 | parcel_redelivery | scout-gemini | fail | done | 85,205 in / 56,580 out | $0.249 | 4m 30s | 2026-09-28 03:16 |
| 27 | parcel_redelivery | deepseek-low | fail | preferences | 242,838 in / 89,943 out | $0.162 | 9m 13s | 2026-09-28 03:14 |
| 26 | parcel_redelivery | scout-gpt | fail | done | 53,959 in / 2,725 out | $0.033 | 30s | 2026-09-28 03:13 |
| 25 | parcel_redelivery | scout-deepseek | fail | done | 105,988 in / 4,647 out | $0.044 | 43s | 2026-09-28 03:13 |
| 24 | parcel_redelivery | scout-gpt | fail | done | 53,959 in / 2,707 out | $0.032 | 29s | 2026-09-28 03:13 |
| 23 | parcel_redelivery | scout-deepseek | fail | done | 52,528 in / 3,052 out | $0.022 | 29s | 2026-09-28 03:13 |
| 22 | parcel_redelivery | sonnet | fail | preferences | 521,147 in / 33,588 out | $1.38 | 8m 48s | 2026-09-28 03:12 |
| 21 | parcel_redelivery | gemini | fail | preferences | 62,678 in / 54,076 out | $0.250 | 4m 26s | 2026-09-28 03:12 |
| 20 | parcel_redelivery | gpt | pass | done | 108,979 in / 844 out | $1.07 | 1m 30s | 2026-09-28 03:11 |
| 19 | parcel_redelivery | deepseek-noreasoning | fail | preferences | 109,336 in / 3,268 out | $0.026 | 1m 42s | 2026-09-28 03:11 |
| 18 | parcel_redelivery | opus | fail | confirm | 148,744 in / 3,077 out | $0.657 | 2m 10s | 2026-09-28 03:10 |
| 17 | parcel_redelivery | scout-local | fail | lookup | 8,066 in / 747 out | $0.003 | 8s | 2026-09-28 03:07 |
| 16 | parcel_redelivery | scout-local | fail | lookup | 5,479 in / 528 out | $0.002 | 6s | 2026-09-28 03:07 |
| 15 | parcel_redelivery | local | fail | consent | 0 in / 0 out | $0.000 | 2s | 2026-09-28 03:07 |
| 14 | parcel_redelivery | opus | pass | done | 156,820 in / 3,417 out | $0.696 | 2m 16s | 2026-09-28 03:07 |
| 13 | parcel_redelivery | deepseek-noreasoning | fail | done | 366,601 in / 6,776 out | $0.072 | 3m 33s | 2026-09-28 03:07 |
| 12 | parcel_redelivery | gpt | pass | done | 309,250 in / 2,660 out | $3.23 | 4m 1s | 2026-09-28 03:07 |
| 11 | parcel_redelivery | qwen-noreasoning | fail | preferences | 363,407 in / 4,373 out | $0.000 | 8m 58s | 2026-09-28 03:07 |
| 10 | parcel_redelivery | gemini | fail | preferences | 46,875 in / 47,233 out | $0.212 | 4m 17s | 2026-09-28 03:07 |
| 9 | parcel_redelivery | local | fail | consent | 0 in / 0 out | $0.000 | 3s | 2026-09-28 03:07 |
| 8 | parcel_redelivery | deepseek-noreasoning | active | — | — | — | — | 2026-09-28 03:05 |
| 7 | parcel_redelivery | deepseek-noreasoning | fail | done | 104,133 in / 3,439 out | $0.024 | 1m 33s | 2026-09-28 03:03 |
| 6 | parcel_redelivery | deepseek-noreasoning | fail | done | 153,518 in / 3,644 out | $0.038 | 2m 2s | 2026-09-28 03:01 |
| 5 | parcel_redelivery | scout+writer-turns-v3-no-captcha | fail | confirm | 279,904 in / 35,978 out | $1.19 | 9m 20s | 2026-09-27 17:35 |
| 4 | parcel_redelivery | scout+writer-turns-v2-no-captcha | fail | confirm | 197,821 in / 13,116 out | $0.598 | 2m 51s | 2026-09-27 16:25 |
| 3 | parcel_redelivery | scout+writer-turns-no-captcha | fail | confirm | 252,803 in / 32,430 out | $1.10 | 8m 13s | 2026-09-27 16:03 |
| 2 | parcel_redelivery | scout+writer-turns | fail | human | 465,984 in / 11,258 out | $0.413 | 2m 10s | 2026-09-27 16:00 |
| 1 | parcel_redelivery | scout+writer | fail | human | 56,197 in / 3,067 out | $0.037 | 29s | 2026-09-27 15:28 |