Testing platform for ZShot
| Template | Total Runs | Pass | Fail | Pass Rate | Avg Tokens In | Avg Tokens Out | Tokens In per Pass |
|---|---|---|---|---|---|---|---|
| benefits_hardship_claim | 32 | 16 | 16 | 50.0% | 898,848 | 19,789 | 1,235,917 |
| doom_destroy_a_barrel | 15 | 7 | 8 | 46.7% | 208,211 | 5,319 | 446,166 |
| parcel_redelivery | 281 | 116 | 165 | 41.3% | 181,211 | 15,199 | 412,846 |
| recipe_site_favorite_a_recipe | 16 | 8 | 8 | 50.0% | 15,315 | 609 | 15,315 |
Token figures count runs recorded with --llm-usage. Tokens in per pass is every metered run's input over the metered runs that passed.
| ID | Template | Label | Result | Furthest | Tokens | Est. Cost | Took | Created |
|---|---|---|---|---|---|---|---|---|
| 150 | parcel_redelivery | gpt@local-1002 | pass | done | 147,902 in / 776 out | $1.45 | 2m 22s | 2026-10-02 17:13 |
| 149 | parcel_redelivery | opus@local-1002 | pass | done | 194,170 in / 4,544 out | $0.868 | 2m 48s | 2026-10-02 17:13 |
| 148 | parcel_redelivery | sonnet@local-1002 | pass | done | 165,865 in / 4,969 out | $0.381 | 2m 3s | 2026-10-02 17:11 |
| 147 | parcel_redelivery | deepseek@local-1002 | fail | done | 163,489 in / 121,802 out | $0.181 | 10m 14s | 2026-10-02 17:11 |
| 146 | parcel_redelivery | scout-sonnet-opus@local-1002 | fail | done | 69,687 in / 4,072 out | $0.041 | 34s | 2026-10-02 17:11 |
| 145 | parcel_redelivery | gpt@local-1002 | pass | done | 212,360 in / 1,033 out | $2.18 | 2m 38s | 2026-10-02 17:11 |
| 144 | parcel_redelivery | opus@local-1002 | pass | done | 174,237 in / 4,098 out | $0.779 | 3m 0s | 2026-10-02 17:10 |
| 143 | parcel_redelivery | sonnet@local-1002 | pass | done | 166,157 in / 4,910 out | $0.381 | 2m 2s | 2026-10-02 17:09 |
| 142 | parcel_redelivery | scout-sonnet-opus@local-1002 | pass | done | 224,486 in / 7,659 out | $0.484 | 1m 46s | 2026-10-02 17:09 |
| 141 | parcel_redelivery | deepseek@local-1002 | pass | done | 132,842 in / 35,360 out | $0.077 | 3m 54s | 2026-10-02 17:07 |
| 140 | parcel_redelivery | sonnet@local-1002 | pass | done | 166,093 in / 5,631 out | $0.388 | 2m 3s | 2026-10-02 17:07 |
| 139 | parcel_redelivery | scout-sonnet-opus@local-1002 | pass | done | 221,739 in / 7,660 out | $0.607 | 2m 2s | 2026-10-02 17:07 |
| 138 | parcel_redelivery | opus@local-1002 | pass | done | 174,963 in / 4,614 out | $0.792 | 2m 41s | 2026-10-02 17:07 |
| 137 | parcel_redelivery | gpt@local-1002 | pass | done | 257,832 in / 1,548 out | $2.66 | 3m 32s | 2026-10-02 17:07 |
| 136 | parcel_redelivery | scout-sonnet | fail | done | 169,470 in / 7,165 out | $0.216 | 1m 25s | 2026-09-28 19:32 |
| 135 | parcel_redelivery | scout-sonnet-opus | pass | done | 195,820 in / 6,700 out | $0.493 | 2m 3s | 2026-09-28 19:32 |
| 134 | parcel_redelivery | sonnet | pass | done | 164,863 in / 5,464 out | $0.384 | 1m 57s | 2026-09-28 19:32 |
| 133 | parcel_redelivery | scout-sonnet-opus | pass | done | 193,058 in / 6,269 out | $0.572 | 2m 7s | 2026-09-28 19:30 |
| 132 | parcel_redelivery | scout-sonnet | fail | done | 257,770 in / 11,822 out | $0.453 | 2m 33s | 2026-09-28 19:30 |
| 131 | parcel_redelivery | sonnet | pass | done | 191,474 in / 6,656 out | $0.450 | 2m 15s | 2026-09-28 19:30 |
| 130 | parcel_redelivery | scout-sonnet | pass | done | 209,867 in / 8,304 out | $0.338 | 1m 54s | 2026-09-28 19:28 |
| 129 | parcel_redelivery | scout-sonnet-opus | pass | done | 197,212 in / 6,683 out | $0.565 | 2m 1s | 2026-09-28 19:28 |
| 128 | parcel_redelivery | sonnet | fail | confirm | 156,472 in / 4,712 out | $0.360 | 1m 46s | 2026-09-28 19:28 |
| 127 | parcel_redelivery | scout-mercury | fail | lookup | 11,315 in / 2,870 out | $0.005 | 12s | 2026-09-28 16:50 |
| 126 | parcel_redelivery | scout-mercury | fail | lookup | 11,309 in / 3,512 out | $0.005 | 16s | 2026-09-28 16:50 |
| 125 | parcel_redelivery | scout-mercury | fail | lookup | 25,855 in / 3,951 out | $0.011 | 17s | 2026-09-28 16:50 |
| 124 | parcel_redelivery | scout-mercury | fail | preferences | 146,857 in / 11,552 out | $0.346 | 1m 47s | 2026-09-28 16:48 |
| 123 | parcel_redelivery | scout-mercury-deepseek | fail | confirm | 157,722 in / 33,136 out | $0.081 | 3m 4s | 2026-09-28 16:45 |
| 122 | parcel_redelivery | scout-mercury-deepseek | fail | done | 162,143 in / 74,191 out | $0.130 | 6m 13s | 2026-09-28 16:38 |
| 121 | parcel_redelivery | scout-mercury-deepseek | fail | done | 153,551 in / 54,695 out | $0.108 | 4m 47s | 2026-09-28 16:34 |
| 120 | parcel_redelivery | scout-mercury | fail | consent | — | — | 0s | 2026-09-28 16:30 |
| 119 | parcel_redelivery | scout-mercury | fail | consent | — | — | 0s | 2026-09-28 16:30 |
| 118 | parcel_redelivery | scout-mercury | fail | consent | — | — | 0s | 2026-09-28 16:30 |
| 117 | parcel_redelivery | scout-mercury | fail | consent | — | — | 0s | 2026-09-28 16:30 |
| 116 | parcel_redelivery | scout-mercury-deepseek | fail | confirm | 203,110 in / 63,951 out | $0.123 | 5m 13s | 2026-09-28 16:28 |
| 115 | parcel_redelivery | scout-mercury | pass | done | 234,010 in / 15,321 out | $0.588 | 2m 18s | 2026-09-28 16:27 |
| 114 | parcel_redelivery | scout-mercury-deepseek | fail | done | 105,884 in / 10,519 out | $0.044 | 55s | 2026-09-28 16:27 |
| 113 | parcel_redelivery | scout-mercury | pass | done | 322,346 in / 13,213 out | $0.787 | 4m 52s | 2026-09-28 16:15 |
| 112 | parcel_redelivery | scout-mercury | pass | done | 360,317 in / 14,388 out | $0.908 | 7m 35s | 2026-09-28 16:07 |
| 111 | parcel_redelivery | scout-mercury | fail | done | 187,622 in / 9,534 out | $0.173 | 10m 1s | 2026-09-28 15:57 |
| 110 | parcel_redelivery | scout-mercury | pass | done | 254,573 in / 11,731 out | $0.700 | 2m 23s | 2026-09-28 15:55 |
| 109 | parcel_redelivery | scout-deepseek | pass | done | 143,479 in / 25,057 out | $0.072 | 3m 37s | 2026-09-28 15:53 |
| 108 | parcel_redelivery | scout-deepseek | fail | done | 197,810 in / 41,376 out | $0.112 | 5m 40s | 2026-09-28 15:48 |
| 107 | parcel_redelivery | scout-mercury | pass | done | 279,529 in / 12,381 out | $0.715 | 7m 38s | 2026-09-28 15:47 |
| 106 | parcel_redelivery | scout-gemini | fail | done | 121,543 in / 18,973 out | $0.123 | 2m 58s | 2026-09-28 15:45 |
| 105 | parcel_redelivery | scout-deepseek | fail | done | 148,998 in / 24,220 out | $0.075 | 4m 16s | 2026-09-28 15:43 |
| 104 | parcel_redelivery | scout-gemini | fail | done | 119,717 in / 14,807 out | $0.111 | 4m 0s | 2026-09-28 15:41 |
| 103 | parcel_redelivery | scout-gemini | fail | done | 150,411 in / 17,651 out | $0.144 | 2m 27s | 2026-09-28 15:38 |
| 102 | parcel_redelivery | scout-deepseek | pass | done | 179,844 in / 59,884 out | $0.121 | 5m 52s | 2026-09-28 15:38 |
| 101 | parcel_redelivery | scout-gemini | fail | done | 72,235 in / 8,534 out | $0.055 | 1m 2s | 2026-09-28 15:37 |