Paved Gate · Benchmark

Fast evaluator gate vs. zero-shot frontier classification

Latency per request

Milliseconds, p50 and p95. Lower is faster.

p50 p95

Cost per million requests

USD, from token usage × list prices. Lower is cheaper.

Results

Raw numbers behind both charts.

Arm	Mode	Requests	Errors	p50 ms	p95 ms	Mean ms	$ / request	$ / 1M req.	Actual cost this run

Cost basis: live arms use provider-reported tokens × list price. Mock and simulated arms (marked est.) use estimated tokens × list price and made no billed calls. Actual cost this run counts successful requests only.

Caveats, prices, and raw timing samples

Classification accuracy

Arm	Mode	Accuracy	Correct / n	Macro F1	Weighted F1	Invalid outputs	API errors

Accuracy and F1 are computed only from real label comparisons on live arms. API errors are excluded from n; invalid model outputs (refusals, labels outside the list) count as wrong.

Confusion matrix

Rows: true intent. Columns: predicted intent. Shade = share of that row's tickets; numbers are counts.

Most frequent confusions