We ran a bunch (2,725) of Tessl verifier tests which are part of our test suite for our internal code base and switched the judge model to Jev and GPT Luna 6
The results:
- Jev is 13.6x faster
- Jev is 2.7x cheaper
- The models had a 85.9% agreement on verdicts
Jev finished the suite in 32 seconds. GPT Luna 6 took 436.5 seconds.
The interesting part is where the models disagreed, particularly on rules about comment structure and content. We published the approach, what we tested, and a workflow you can reproduce on your own codebase.
- If it is offline processing (e.g. classifying text snippets stored in a DB (from call transcript, from logs)), you can batch to Luna
- If you batch to Luna and then divide both cost and time by the batch size, you will find that Luna beats Jev.
- Now tune your batch size for your test dataset and see where the batch size causes accuracy falloff.
Tested this approach with GPT-6 Luna. It was 1.6x cheaper, 1.2x slower, within margin of error performance vs Jev with batch size 20 using a CFPB complaint dataset. Tuning batch size up to even double would likely yield similar accuracy while reducing both the per-record run time as well as per-record token cost.
The methodology of comparing single record is only valid for the on-line, real-time use case. For every other case, Luna can match or beat Jev by simply batching. If you find no dropoff at larger batch sizes, you will be significantly cheaper than Jev.
* Batching here does not mean the native batch API but actually placing 20 records (batch size) into one prompt and getting 20 results back in one response.
We ran a bunch (2,725) of Tessl verifier tests which are part of our test suite for our internal code base and switched the judge model to Jev and GPT Luna 6
The results:
- Jev is 13.6x faster - Jev is 2.7x cheaper - The models had a 85.9% agreement on verdicts
Jev finished the suite in 32 seconds. GPT Luna 6 took 436.5 seconds.
The interesting part is where the models disagreed, particularly on rules about comment structure and content. We published the approach, what we tested, and a workflow you can reproduce on your own codebase.
- This is only true if your constraint is online
- If it is offline processing (e.g. classifying text snippets stored in a DB (from call transcript, from logs)), you can batch to Luna
- If you batch to Luna and then divide both cost and time by the batch size, you will find that Luna beats Jev.
- Now tune your batch size for your test dataset and see where the batch size causes accuracy falloff.
Tested this approach with GPT-6 Luna. It was 1.6x cheaper, 1.2x slower, within margin of error performance vs Jev with batch size 20 using a CFPB complaint dataset. Tuning batch size up to even double would likely yield similar accuracy while reducing both the per-record run time as well as per-record token cost.
The methodology of comparing single record is only valid for the on-line, real-time use case. For every other case, Luna can match or beat Jev by simply batching. If you find no dropoff at larger batch sizes, you will be significantly cheaper than Jev.
* Batching here does not mean the native batch API but actually placing 20 records (batch size) into one prompt and getting 20 results back in one response.