We developed an early vector embedding model at Lawrence Berkeley National Lab and extended it called the Tuatara Vector Model. It's a blend and scored against Jev's 3,080 BANKING77 test messages resulting in 91.79% versus 92.40%, a statistical tie with some good cost savings.
As most may know, BANKING77 is a public dataset from PolyAI with 13,083 messages to a bank, each labelled with intents. The split was 10,003 messages for training and 3,080 for testing.
The run used the pinned model jev-1.13.0, all 77 intents as options in a single question, and up to 24 training examples retrieved for each message. Jev got 2,846 of 3,080 right.
Recomputing Jev’s accuracy from the file gives 92.40%, the same figure the experiment reports.
As most may know, BANKING77 is a public dataset from PolyAI with 13,083 messages to a bank, each labelled with intents. The split was 10,003 messages for training and 3,080 for testing.
The run used the pinned model jev-1.13.0, all 77 intents as options in a single question, and up to 24 training examples retrieved for each message. Jev got 2,846 of 3,080 right.
Recomputing Jev’s accuracy from the file gives 92.40%, the same figure the experiment reports.
More stats: https://cymetica.com/blog/matching-jev-on-banking77-at-a-tho...