Testing platform for ZShot
| Template | Total Runs | Pass | Fail | Pass Rate | Avg Tokens In | Avg Tokens Out | Tokens In per Pass |
|---|---|---|---|---|---|---|---|
| benefits_hardship_claim | 32 | 16 | 16 | 50.0% | 898,848 | 19,789 | 1,235,917 |
| doom_destroy_a_barrel | 15 | 7 | 8 | 46.7% | 208,211 | 5,319 | 446,166 |
| parcel_redelivery | 281 | 116 | 165 | 41.3% | 181,211 | 15,199 | 412,846 |
| recipe_site_favorite_a_recipe | 16 | 8 | 8 | 50.0% | 15,315 | 609 | 15,315 |
Token figures count runs recorded with --llm-usage. Tokens in per pass is every metered run's input over the metered runs that passed.
| ID | Template | Label | Result | Furthest | Tokens | Est. Cost | Took | Created |
|---|---|---|---|---|---|---|---|---|
| 100 | parcel_redelivery | scout-deepseek | fail | done | 101,221 in / 16,193 out | $0.050 | 1m 51s | 2026-09-28 15:36 |
| 99 | parcel_redelivery | scout-gemini | fail | done | 100,274 in / 9,914 out | $0.079 | 1m 26s | 2026-09-28 15:36 |
| 98 | parcel_redelivery | scout-deepseek | fail | consent | 0 in / 0 out | $0.000 | 39s | 2026-09-28 13:37 |
| 97 | parcel_redelivery | scout-gemini | fail | consent | 1,887 in / 218 out | $0.001 | 52s | 2026-09-28 13:37 |
| 96 | parcel_redelivery | scout-deepseek | fail | consent | 5,477 in / 539 out | $0.002 | 1m 53s | 2026-09-28 13:35 |
| 95 | parcel_redelivery | scout-gemini | fail | consent | 9,366 in / 872 out | $0.004 | 1m 47s | 2026-09-28 13:35 |
| 94 | parcel_redelivery | scout-deepseek | fail | consent | 4,090 in / 438 out | $0.002 | 2m 14s | 2026-09-28 13:33 |
| 93 | parcel_redelivery | scout-gemini | fail | consent | 3,171 in / 320 out | $0.001 | 1m 36s | 2026-09-28 13:33 |
| 92 | parcel_redelivery | scout-deepseek | fail | consent | 1,884 in / 218 out | $0.001 | 1m 20s | 2026-09-28 13:32 |
| 91 | parcel_redelivery | scout-gemini | fail | consent | 3,171 in / 320 out | $0.001 | 1m 59s | 2026-09-28 13:31 |
| 90 | parcel_redelivery | scout-deepseek | fail | consent | 3,165 in / 320 out | $0.001 | 1m 32s | 2026-09-28 13:30 |
| 89 | parcel_redelivery | scout-gemini | fail | consent | 3,171 in / 320 out | $0.001 | 53s | 2026-09-28 13:30 |
| 88 | parcel_redelivery | scout-deepseek | pass | done | 129,108 in / 18,506 out | $0.062 | 3m 5s | 2026-09-28 13:21 |
| 87 | parcel_redelivery | scout-qwen | fail | confirm | 227,624 in / 5,297 out | $0.048 | 4m 53s | 2026-09-28 13:21 |
| 86 | parcel_redelivery | scout-gemini | fail | done | 91,100 in / 11,012 out | $0.077 | 2m 19s | 2026-09-28 13:20 |
| 85 | parcel_redelivery | scout-gpt | pass | done | 171,995 in / 4,856 out | $0.754 | 2m 28s | 2026-09-28 13:20 |
| 84 | parcel_redelivery | scout-qwen | fail | confirm | 144,694 in / 4,482 out | $0.028 | 4m 18s | 2026-09-28 13:17 |
| 83 | parcel_redelivery | scout-gpt | fail | done | 205,635 in / 4,558 out | $1.18 | 3m 20s | 2026-09-28 13:17 |
| 82 | parcel_redelivery | scout-gemini | fail | done | 115,151 in / 28,198 out | $0.159 | 3m 56s | 2026-09-28 13:16 |
| 81 | parcel_redelivery | scout-deepseek | fail | done | 137,890 in / 44,903 out | $0.095 | 5m 15s | 2026-09-28 13:16 |
| 80 | parcel_redelivery | scout-qwen | fail | preferences | 347,696 in / 6,159 out | $0.026 | 7m 10s | 2026-09-28 11:52 |
| 79 | parcel_redelivery | scout-deepseek | fail | confirm | 201,365 in / 87,506 out | $0.162 | 7m 33s | 2026-09-28 11:49 |
| 78 | parcel_redelivery | scout-deepseek | fail | done | 119,996 in / 18,440 out | $0.058 | 2m 4s | 2026-09-28 11:47 |
| 77 | parcel_redelivery | scout-qwen | fail | preferences | 346,850 in / 6,319 out | $0.023 | 7m 22s | 2026-09-28 11:45 |
| 76 | parcel_redelivery | scout-sonnet | fail | done | 248,899 in / 10,790 out | $0.433 | 2m 55s | 2026-09-28 11:44 |
| 75 | parcel_redelivery | scout-gemini | fail | done | 85,981 in / 10,077 out | $0.070 | 1m 13s | 2026-09-28 11:44 |
| 74 | parcel_redelivery | scout-gemini | fail | done | 81,444 in / 15,939 out | $0.089 | 1m 36s | 2026-09-28 11:42 |
| 73 | parcel_redelivery | scout-sonnet | fail | done | 177,088 in / 7,698 out | $0.302 | 2m 5s | 2026-09-28 11:42 |
| 72 | parcel_redelivery | scout-sonnet-opus | pass | done | 171,985 in / 5,384 out | $0.476 | 1m 49s | 2026-09-28 11:39 |
| 71 | parcel_redelivery | scout-sonnet-opus | pass | done | 241,606 in / 8,203 out | $0.689 | 2m 21s | 2026-09-28 11:37 |
| 70 | parcel_redelivery | scout-sonnet | fail | done | 149,784 in / 5,209 out | $0.225 | 1m 23s | 2026-09-28 11:35 |
| 69 | parcel_redelivery | scout-sonnet | fail | done | 265,073 in / 10,550 out | $0.485 | 2m 57s | 2026-09-28 11:32 |
| 68 | parcel_redelivery | scout-opus | pass | done | 192,639 in / 6,065 out | $0.600 | 2m 13s | 2026-09-28 11:30 |
| 67 | parcel_redelivery | scout-opus | pass | done | 232,613 in / 8,199 out | $0.792 | 2m 55s | 2026-09-28 11:27 |
| 66 | parcel_redelivery | opus-thinking | fail | confirm | 212,288 in / 7,537 out | $1.000 | 4m 10s | 2026-09-28 11:23 |
| 65 | parcel_redelivery | scout-gemini | fail | confirm | 62,117 in / 8,157 out | $0.050 | 59s | 2026-09-28 11:21 |
| 64 | parcel_redelivery | opus-thinking | pass | done | 179,970 in / 4,873 out | $0.817 | 2m 52s | 2026-09-28 11:20 |
| 63 | parcel_redelivery | deepseek | fail | done | 190,480 in / 95,029 out | $0.154 | 8m 32s | 2026-09-28 11:20 |
| 62 | parcel_redelivery | scout-gemini | fail | done | 166,289 in / 32,128 out | $0.209 | 3m 36s | 2026-09-28 11:18 |
| 61 | parcel_redelivery | deepseek | fail | preferences | 93,764 in / 31,350 out | $0.059 | 3m 18s | 2026-09-28 11:16 |
| 60 | parcel_redelivery | sonnet | fail | preferences | 467,846 in / 22,981 out | $1.17 | 7m 2s | 2026-09-28 11:13 |
| 59 | parcel_redelivery | scout-qwen | fail | confirm | 110,097 in / 3,369 out | $0.023 | 1m 46s | 2026-09-28 11:11 |
| 58 | parcel_redelivery | deepseek-low | fail | done | 233,042 in / 68,628 out | $0.133 | 7m 9s | 2026-09-28 11:09 |
| 57 | parcel_redelivery | scout-qwen | fail | done | 207,231 in / 4,764 out | $0.039 | 3m 18s | 2026-09-28 11:07 |
| 56 | parcel_redelivery | gemini | fail | preferences | 300,334 in / 143,388 out | $0.763 | 13m 1s | 2026-09-28 11:05 |
| 55 | parcel_redelivery | sonnet | fail | preferences | 537,973 in / 26,686 out | $1.34 | 10m 13s | 2026-09-28 11:03 |
| 54 | parcel_redelivery | deepseek-low | fail | preferences | 339,288 in / 53,147 out | $0.143 | 6m 32s | 2026-09-28 11:03 |
| 53 | parcel_redelivery | qwen-noreasoning | fail | preferences | 260,715 in / 2,361 out | $0.000 | 6m 15s | 2026-09-28 11:01 |
| 52 | parcel_redelivery | opus | pass | done | 202,780 in / 5,898 out | $0.929 | 3m 27s | 2026-09-28 10:59 |
| 51 | parcel_redelivery | scout-deepseek | fail | confirm | 203,700 in / 38,398 out | $0.112 | 3m 56s | 2026-09-28 10:59 |