DEV Community

Cover image for Five Labelled Examples Match Jev. Then I Tried Tanglish.
Madhesh Vivekanandan
Madhesh Vivekanandan

Posted on Edited on

Five Labelled Examples Match Jev. Then I Tried Tanglish.

"Online la oru item order panni romba naal aachu, aana innum varave illa. Enakku podhum! En money-a thirupi kudunga, indha aalungaluku naan pay panna maaten."

(I ordered something online ages ago and it still hasn't come. I've had enough! Give me my money back, I'm not paying these people.)

That's a Banking77 refund request, rewritten by a Claude agent in Tanglish: Tamil and English in one message, the way many of us type to a bank. Cosine similarity filed it under supported_cards_and_currencies. A classifier trained on 35 English examples per intent got it wrong in 2 of 3 runs. TypeSafe's Jev, a model that picks one option from a list you give it, chose request_refund with confidence 0.98 (how sure it is, from 0 to 1).

I expected Jev to win everywhere. It didn't. On clean English, five labelled examples per category let a free classifier match Jev. On Hinglish, Tanglish and trick questions, 35 English examples didn't close the gap, even with a multilingual embedder.

Four bar charts. Clean English: the better of cosine voting and the classifier with 35 labelled examples 91.4%, Jev 83.3%, cosine 73.2%, cross-encoder 71.1%. Hinglish: Jev 85.4%, best with 35 examples 63.7%. Tanglish: Jev 77.1%, best with 35 examples 64.6%. Trick questions: Jev 100%, cross-encoder 79.0%, best with 35 examples 64.1%.

I couldn't find a write-up testing Jev on how Indian users actually type, so here's one, with a router you can copy. Every answer is in the repo.

The contest in 30 seconds

The job: read a bank customer's message and pick which of 77 intents it is, from Banking77 (card_arrival, declined_card_payment, request_refund, …). Each intent gets a one-line description, like "The customer ordered a card and it has not arrived yet". A labelled example is a past message where a person already wrote down the right intent.

Method How it decides, in plain words Labelled examples
Cosine Turns the message and each description into a list of numbers (an embedding) that captures meaning, and picks the closest description none
Jev Reads the message and all 77 descriptions together, like a multiple-choice question none
Cross-encoder A small model that reads the message beside each description and scores how well they fit none
Cosine voting Finds the 5 past messages closest in meaning; they vote 1 to 35 per intent
Classifier Learns from those same past messages (logistic regression) and predicts 1 to 35 per intent

Think of two clerks: a fast, free one (the last two run on a laptop) and a careful one, Jev, who reads every option. When is the fast one enough?

Jev returns one option and a probability for every option, not text, so there's nothing to parse. The whole call:

from typesafe_sdk import Choice, RetryPolicy, TypeSafeClient

client = TypeSafeClient(api_key="public", base_url="https://opencode.ai/zen",  # free route
                        retry=RetryPolicy(max_retries=0))  # retries count against its daily cap
question = Choice(
    instructions="Which banking support intent does the customer_message express? "
                 "Pick the single best match.",
    criteria=descriptions,  # {"card_arrival": "The customer ordered a card and it has not arrived yet, ...", ...} x 77
)
response = client.system_one({"customer_message": text}, {"intent": question}, model="jev-1.13-free")
answer = response.choices["intent"]
answer.choice, answer.confidence  # ("request_refund", 0.98)
Enter fullscreen mode Exit fullscreen mode

Every call sends all 77 descriptions, about 2,400 input tokens.

Round 1 · Clean English: five labelled examples match Jev

With zero examples, Jev beats cosine by 10 points across 1,001 test messages.

Line chart of accuracy against labelled examples per category. The classifier reaches 84.2% at 5 examples, inside the blue band of Jev's 95% interval around 83.3%. Cosine voting passes Jev at 10. Both reach about 91% at 35. At 1 example both score about 63%, below cosine's 73.2% with one-line descriptions.

The classifier matches Jev at 5 examples per category, 385 labelled messages in total. Its 0.9-point edge is within the margin of error, so call it a tie. Cosine voting needs 10.

One example is worse than one sentence. A single labelled message per category scores 63%. A one-line description scores 73%.

The cross-encoder did no better than cosine. Pasting the same 5 examples into Jev's descriptions reached 90.4% on 249 messages, a borderline lead, at 3.7 times the tokens.

On clean English, the fast clerk catches up with five examples. Then I changed the language.

Round 2 · Hinglish and Tanglish: the embeddings fall apart

Your users won't type like your test set. Hinglish mixes Hindi and English, Tanglish mixes Tamil and English, both in Latin letters. Claude agents rewrote 144 test messages (all 77 intents) in each, keeping words like card and PIN in English; other agents checked every rewrite, but no native speaker has yet.

Message Cosine Classifier, 35 examples (3 random draws) Jev
English "I ordered something ages ago online but it's simply not arriving… Give me back my money…" ✗ card_arrival ✓ ✓ 0.99
Hinglish "Maine kaafi time pehle online kuch order kiya tha… Mere paise wapas do…" ✗ getting_virtual_card ✗ in all 3 draws ✓ 0.99
Tanglish "Online la oru item order panni romba naal aachu… En money-a thirupi kudunga…" ✗ supported_cards_and_currencies ✗ in 2 of 3 draws ✓ 0.98

One message, picked to show the pattern. All 144:

Slope chart across English, Hinglish and Tanglish. Jev goes 86.1% to 85.4% to 77.1%. The classifier with 35 English examples falls from 93.1% to 56.2% to 52.8%. Cosine voting falls to 63.7% and 64.6%. The multilingual bge-m3 embedder, the better of voting and classifier, reaches 69.9% and 67.6%. Cosine with descriptions falls from 72.2% to 41.7%.

"Isn't that just your English-only embedding model?" Fair question. Re-running both with the multilingual bge-m3, about 17 times bigger (the dashed line), helps by 3 to 6 points. Jev stays clearly ahead on Hinglish; on Tanglish, 144 messages make the gap borderline. Training on Hinglish or Tanglish examples is the next experiment.

Illustration (blue robot = Jev, orange = embeddings): three versions of the same message ride a conveyor. The blue robot drops all three into the same refund slot; the orange robot's scatter plot puts them in three different places.

Round 3 · Trick questions: more examples don't help

Next, 105 new messages built to fool keyword matching. Claude agents wrote them and three more labelled each one blind; all 105 passed, so the check may have been lenient.

Message (and the right answer) Cosine Classifier, 35 examples Jev
Negation: "It's not contactless that's the problem, tapping my card works fine. It's Google Pay that refuses to add my card." → apple_pay_or_google_pay ✗ contactless_not_working ✗ same, 86–91% sure ✓
Decoy: "An ATM in Lisbon swallowed my card last week, but that's been dealt with and a replacement is ordered. How long does delivery usually take?" → card_delivery_estimate ✗ card_swallowed ✗ same, all 3 draws ✓
Indirect: "The four numbers I tap in at shops are my old house number and I'd like something different. How do I swap them?" → change_pin ✗ edit_personal_details ✗ same, all 3 draws ✓

Jev got all 105 right: read that as "Jev doesn't fall for these traps", not "Jev is perfect". The classifier stayed at 64% whether it had 5 examples per intent or 35 (76.5% with bge-m3), and cosine took the planted bait 23 times.

Illustration: left, an orange robot lunges at a keyword lure; right, a blue robot reads the whole note, where the word NOT is highlighted.

My favourite detail: the bug was in my description

Jev's worst category on English was get_physical_card: 0 out of 13. Cosine's too.

As a fairness rule, I wrote every description from the label name alone: "The customer asks how to get their physical card." Then I read the messages:

"Where is my PIN number located?" · "Is PIN delivered separately?" · "I just got my new card but am not sure how to check its PIN."

In Banking77, get_physical_card really means "where's my PIN?" (105 of its 106 training messages mention one). Jev sent all 13 to change_pin, 9 of them at confidence 0.95 or higher. I haven't re-run it fixed, so the cosine, cross-encoder and Jev numbers include this bug; voting and the classifier never read descriptions.

Your descriptions are your prompt. Read five training messages per category before you write one.

When to trust an answer, and when to ask Jev

Both Jev and the classifier say how sure they are. On English, Jev's answers at confidence 1.0 (half of them) are 98% right, and the classifier's confidence works just as well. On Hinglish and Tanglish, only about 1 in 10 classifier answers clear 0.9: it knows it's lost. So let the fast clerk answer and pass the odd letter to the careful one:

Let the classifier answer. When its confidence is below 0.7, ask Jev.

from sentence_transformers import SentenceTransformer
from sklearn.linear_model import LogisticRegression

model = SentenceTransformer("BAAI/bge-small-en-v1.5")
embed = lambda texts: model.encode(texts, normalize_embeddings=True)
clf = LogisticRegression(C=100, max_iter=5000).fit(embed(examples), labels)  # 35 per intent; C=1 underfits

def route(text: str) -> str:
    probs = clf.predict_proba(embed([text]))[0]
    if probs.max() >= 0.7:
        return clf.classes_[probs.argmax()]
    answer = client.system_one({"customer_message": text}, {"intent": question},
                               model="jev-1.13-free").choices["intent"]
    return answer.choice  # client and question: from the Jev call above
Enter fullscreen mode Exit fullscreen mode

With 35 examples per intent, on Round 2's 144 messages and Round 3's 105 (the English column is the 144 originals, so Jev shows 86.1%, not 83.3%):

English (144) Hinglish (144) Tanglish (144) Trick (105)
Jev alone 86.1% 85.4% 77.1% 100%
Classifier alone 93.1% 56.2% 52.8% 64.1%
Classifier first, Jev below 0.7 92.6% 84.5% 78.7% 91.7%
Messages sent to Jev 14% 73% 75% 54%

On English and code-mixed text it stays within about a point of the better single method (above it on Tanglish), while skipping 86% of Jev calls on English but only about a quarter on Hinglish and Tanglish. Trick questions leak: the classifier was 86–91% sure of its wrong Google Pay answer.

I didn't tune 0.7 (it's the middle of a fixed 0.9 / 0.7 / 0.5 grid), so sweep 0.5 to 0.9 on a few hundred of your own held-out messages, as TRIAGE.md does. It also has a simpler rule, "trust it when voting and the classifier agree", that wins on clean English and breaks on Hinglish.

Illustration: an orange robot sorts the plain envelopes itself and passes the mixed-language and tricky ones through a hatch to a blue robot, who reads them carefully.

So which one should you use?

flowchart TD
  Q1{"Labelled messages<br/>per intent?"} -- "fewer than 5" --> J["Jev<br/>(cosine on descriptions if it must be free)"]
  Q1 -- "5 or more" --> Q2{"Only clean English?"}
  Q2 -- yes --> E["Classifier on embeddings"]
  Q2 -- "no, or not sure" --> H["Classifier first;<br/>ask Jev below 0.7<br/>(measured with 35 per intent)"]
  J --> P["Jev's answer below 0.95?<br/>Send it to a person<br/>(Jev alone: 3 in 10 English, 6 in 10 Tanglish)"]
  H --> P

For scale: the classifier answers in 7 ms on a laptop CPU (43 ms with bge-m3), for $0. Jev takes 0.76 s per message from India and costs about $0.10 per 1,000 messages at TypeSafe's list price ($0.042 per million input tokens).

What's missing, honestly

  • The classifier only saw English examples; training on code-mixed examples is untested.
  • The extra test sets are small: 105 to 144 messages, roughly ±6 to ±9 points.
  • Banking77 is public, so Jev may have seen it in training.
  • One cross-encoder, trained for search; larger rerankers may do much better.
  • Jev ran through a free third-party route: checked against TypeSafe's published answers, not proven identical, capped near 250 calls a day, and down four days in late October. Latency is from one laptop in India.

Steal this, then run it

Needs Python 3.12 or newer; the install pulls in PyTorch.

git clone https://github.com/Madheshvivekanandan/jev-vs-cosine && cd jev-vs-cosine
python3.12 -m venv .venv && .venv/bin/pip install -e .
.venv/bin/python -m app.main triage && cat results/TRIAGE.md   # no key, no API calls
Enter fullscreen mode Exit fullscreen mode

Every Jev answer is in the repo, so the reports rebuild for free. To call Jev yourself, .env.example uses OpenCode Zen's free route, and verify-route first checks it against TypeSafe's published answers. Install only the official typesafe-sdk; lookalikes exist.

If you take three things from this post:

  1. Write descriptions from real messages, not label names. One wrong sentence zeroed a whole category.
  2. Test on how your users actually type. English said five examples were enough. Tanglish disagreed.
  3. Route on confidence, and pick the threshold on your own data. Let the cheap model answer what it's sure of; escalate the rest to Jev or to a person.

Run it on your own users' messages and tell me where Jev breaks. If you speak Hindi or Tamil and a rewrite reads wrong, I genuinely want the issue.


Previously: The Coding Agent That Can't Grade Its Own Homework, on why the agent that writes the code shouldn't be the one that grades it.

All illustrations in this post were generated with ChatGPT; the charts were drawn from the repo's result files.

Banking77 is by PolyAI (CC BY 4.0). Not affiliated with TypeSafe AI.

Top comments (0)