"Online la oru item order panni romba naal aachu, aana innum varave illa. Enakku podhum! En money-a thirupi kudunga, indha aalungaluku naan pay panna maaten."
(I ordered something online ages ago and it still hasn't come. I've had enough! Give me my money back, I'm not paying these people.)
That's a Banking77 refund request, rewritten by a Claude agent in Tanglish: Tamil and English in one message, the way many of us type to a bank. Cosine similarity filed it under supported_cards_and_currencies. A classifier trained on 35 English examples per intent got it wrong in 2 of 3 runs. TypeSafe's Jev, a model that picks one option from a list you give it, chose request_refund with confidence 0.98 (how sure it is, from 0 to 1).
I expected Jev to win everywhere. It didn't. On clean English, five labelled examples per category let a free classifier match Jev. On Hinglish, Tanglish and trick questions, 35 English examples didn't close the gap, even with a multilingual embedder.
I couldn't find a write-up testing Jev on how Indian users actually type, so here's one, with a router you can copy. Every answer is in the repo.
The contest in 30 seconds
The job: read a bank customer's message and pick which of 77 intents it is, from Banking77 (card_arrival, declined_card_payment, request_refund, …). Each intent gets a one-line description, like "The customer ordered a card and it has not arrived yet". A labelled example is a past message where a person already wrote down the right intent.
| Method | How it decides, in plain words | Labelled examples |
|---|---|---|
| Cosine | Turns the message and each description into a list of numbers (an embedding) that captures meaning, and picks the closest description | none |
| Jev | Reads the message and all 77 descriptions together, like a multiple-choice question | none |
| Cross-encoder | A small model that reads the message beside each description and scores how well they fit | none |
| Cosine voting | Finds the 5 past messages closest in meaning; they vote | 1 to 35 per intent |
| Classifier | Learns from those same past messages (logistic regression) and predicts | 1 to 35 per intent |
Think of two clerks: a fast, free one (the last two run on a laptop) and a careful one, Jev, who reads every option. When is the fast one enough?
Jev returns one option and a probability for every option, not text, so there's nothing to parse. The whole call:
from typesafe_sdk import Choice, RetryPolicy, TypeSafeClient
client = TypeSafeClient(api_key="public", base_url="https://opencode.ai/zen", # free route
retry=RetryPolicy(max_retries=0)) # retries count against its daily cap
question = Choice(
instructions="Which banking support intent does the customer_message express? "
"Pick the single best match.",
criteria=descriptions, # {"card_arrival": "The customer ordered a card and it has not arrived yet, ...", ...} x 77
)
response = client.system_one({"customer_message": text}, {"intent": question}, model="jev-1.13-free")
answer = response.choices["intent"]
answer.choice, answer.confidence # ("request_refund", 0.98)
Every call sends all 77 descriptions, about 2,400 input tokens.
Round 1 · Clean English: five labelled examples match Jev
With zero examples, Jev beats cosine by 10 points across 1,001 test messages.
The classifier matches Jev at 5 examples per category, 385 labelled messages in total. Its 0.9-point edge is within the margin of error, so call it a tie. Cosine voting needs 10.
One example is worse than one sentence. A single labelled message per category scores 63%. A one-line description scores 73%.
The cross-encoder did no better than cosine. Pasting the same 5 examples into Jev's descriptions reached 90.4% on 249 messages, a borderline lead, at 3.7 times the tokens.
On clean English, the fast clerk catches up with five examples. Then I changed the language.
Round 2 · Hinglish and Tanglish: the embeddings fall apart
Your users won't type like your test set. Hinglish mixes Hindi and English, Tanglish mixes Tamil and English, both in Latin letters. Claude agents rewrote 144 test messages (all 77 intents) in each, keeping words like card and PIN in English; other agents checked every rewrite, but no native speaker has yet.
| Message | Cosine | Classifier, 35 examples (3 random draws) | Jev | |
|---|---|---|---|---|
| English | "I ordered something ages ago online but it's simply not arriving… Give me back my money…" | ✗ card_arrival
|
✓ | ✓ 0.99 |
| Hinglish | "Maine kaafi time pehle online kuch order kiya tha… Mere paise wapas do…" | ✗ getting_virtual_card
|
✗ in all 3 draws | ✓ 0.99 |
| Tanglish | "Online la oru item order panni romba naal aachu… En money-a thirupi kudunga…" | ✗ supported_cards_and_currencies
|
✗ in 2 of 3 draws | ✓ 0.98 |
One message, picked to show the pattern. All 144:
"Isn't that just your English-only embedding model?" Fair question. Re-running both with the multilingual bge-m3, about 17 times bigger (the dashed line), helps by 3 to 6 points. Jev stays clearly ahead on Hinglish; on Tanglish, 144 messages make the gap borderline. Training on Hinglish or Tanglish examples is the next experiment.
Round 3 · Trick questions: more examples don't help
Next, 105 new messages built to fool keyword matching. Claude agents wrote them and three more labelled each one blind; all 105 passed, so the check may have been lenient.
| Message (and the right answer) | Cosine | Classifier, 35 examples | Jev |
|---|---|---|---|
Negation: "It's not contactless that's the problem, tapping my card works fine. It's Google Pay that refuses to add my card." → apple_pay_or_google_pay
|
✗ contactless_not_working
|
✗ same, 86–91% sure | ✓ |
Decoy: "An ATM in Lisbon swallowed my card last week, but that's been dealt with and a replacement is ordered. How long does delivery usually take?" → card_delivery_estimate
|
✗ card_swallowed
|
✗ same, all 3 draws | ✓ |
Indirect: "The four numbers I tap in at shops are my old house number and I'd like something different. How do I swap them?" → change_pin
|
✗ edit_personal_details
|
✗ same, all 3 draws | ✓ |
Jev got all 105 right: read that as "Jev doesn't fall for these traps", not "Jev is perfect". The classifier stayed at 64% whether it had 5 examples per intent or 35 (76.5% with bge-m3), and cosine took the planted bait 23 times.
My favourite detail: the bug was in my description
Jev's worst category on English was get_physical_card: 0 out of 13. Cosine's too.
As a fairness rule, I wrote every description from the label name alone: "The customer asks how to get their physical card." Then I read the messages:
"Where is my PIN number located?" · "Is PIN delivered separately?" · "I just got my new card but am not sure how to check its PIN."
In Banking77, get_physical_card really means "where's my PIN?" (105 of its 106 training messages mention one). Jev sent all 13 to change_pin, 9 of them at confidence 0.95 or higher. I haven't re-run it fixed, so the cosine, cross-encoder and Jev numbers include this bug; voting and the classifier never read descriptions.
Your descriptions are your prompt. Read five training messages per category before you write one.
When to trust an answer, and when to ask Jev
Both Jev and the classifier say how sure they are. On English, Jev's answers at confidence 1.0 (half of them) are 98% right, and the classifier's confidence works just as well. On Hinglish and Tanglish, only about 1 in 10 classifier answers clear 0.9: it knows it's lost. So let the fast clerk answer and pass the odd letter to the careful one:
Let the classifier answer. When its confidence is below 0.7, ask Jev.
from sentence_transformers import SentenceTransformer
from sklearn.linear_model import LogisticRegression
model = SentenceTransformer("BAAI/bge-small-en-v1.5")
embed = lambda texts: model.encode(texts, normalize_embeddings=True)
clf = LogisticRegression(C=100, max_iter=5000).fit(embed(examples), labels) # 35 per intent; C=1 underfits
def route(text: str) -> str:
probs = clf.predict_proba(embed([text]))[0]
if probs.max() >= 0.7:
return clf.classes_[probs.argmax()]
answer = client.system_one({"customer_message": text}, {"intent": question},
model="jev-1.13-free").choices["intent"]
return answer.choice # client and question: from the Jev call above
With 35 examples per intent, on Round 2's 144 messages and Round 3's 105 (the English column is the 144 originals, so Jev shows 86.1%, not 83.3%):
| English (144) | Hinglish (144) | Tanglish (144) | Trick (105) | |
|---|---|---|---|---|
| Jev alone | 86.1% | 85.4% | 77.1% | 100% |
| Classifier alone | 93.1% | 56.2% | 52.8% | 64.1% |
| Classifier first, Jev below 0.7 | 92.6% | 84.5% | 78.7% | 91.7% |
| Messages sent to Jev | 14% | 73% | 75% | 54% |
On English and code-mixed text it stays within about a point of the better single method (above it on Tanglish), while skipping 86% of Jev calls on English but only about a quarter on Hinglish and Tanglish. Trick questions leak: the classifier was 86–91% sure of its wrong Google Pay answer.
I didn't tune 0.7 (it's the middle of a fixed 0.9 / 0.7 / 0.5 grid), so sweep 0.5 to 0.9 on a few hundred of your own held-out messages, as TRIAGE.md does. It also has a simpler rule, "trust it when voting and the classifier agree", that wins on clean English and breaks on Hinglish.
So which one should you use?
flowchart TD
Q1{"Labelled messages<br/>per intent?"} -- "fewer than 5" --> J["Jev<br/>(cosine on descriptions if it must be free)"]
Q1 -- "5 or more" --> Q2{"Only clean English?"}
Q2 -- yes --> E["Classifier on embeddings"]
Q2 -- "no, or not sure" --> H["Classifier first;<br/>ask Jev below 0.7<br/>(measured with 35 per intent)"]
J --> P["Jev's answer below 0.95?<br/>Send it to a person<br/>(Jev alone: 3 in 10 English, 6 in 10 Tanglish)"]
H --> P
For scale: the classifier answers in 7 ms on a laptop CPU (43 ms with bge-m3), for $0. Jev takes 0.76 s per message from India and costs about $0.10 per 1,000 messages at TypeSafe's list price ($0.042 per million input tokens).
What's missing, honestly
- The classifier only saw English examples; training on code-mixed examples is untested.
- The extra test sets are small: 105 to 144 messages, roughly ±6 to ±9 points.
- Banking77 is public, so Jev may have seen it in training.
- One cross-encoder, trained for search; larger rerankers may do much better.
- Jev ran through a free third-party route: checked against TypeSafe's published answers, not proven identical, capped near 250 calls a day, and down four days in late October. Latency is from one laptop in India.
Steal this, then run it
Needs Python 3.12 or newer; the install pulls in PyTorch.
git clone https://github.com/Madheshvivekanandan/jev-vs-cosine && cd jev-vs-cosine
python3.12 -m venv .venv && .venv/bin/pip install -e .
.venv/bin/python -m app.main triage && cat results/TRIAGE.md # no key, no API calls
Every Jev answer is in the repo, so the reports rebuild for free. To call Jev yourself, .env.example uses OpenCode Zen's free route, and verify-route first checks it against TypeSafe's published answers. Install only the official typesafe-sdk; lookalikes exist.
If you take three things from this post:
- Write descriptions from real messages, not label names. One wrong sentence zeroed a whole category.
- Test on how your users actually type. English said five examples were enough. Tanglish disagreed.
- Route on confidence, and pick the threshold on your own data. Let the cheap model answer what it's sure of; escalate the rest to Jev or to a person.
Run it on your own users' messages and tell me where Jev breaks. If you speak Hindi or Tamil and a rewrite reads wrong, I genuinely want the issue.
Previously: The Coding Agent That Can't Grade Its Own Homework, on why the agent that writes the code shouldn't be the one that grades it.
All illustrations in this post were generated with ChatGPT; the charts were drawn from the repo's result files.
Banking77 is by PolyAI (CC BY 4.0). Not affiliated with TypeSafe AI.






Top comments (0)