Interim results: 517 of 1,000 reviews answered by both models so far. The run is still in progress.

Blind benchmark · IMDB movie reviews · Vercel AI Gateway

517 movie reviews.
Two models. No answer key.

We asked Jev and Qwen3.7 Flash to call each IMDB review positive or negative, without ever showing them the human rating. Then we scored every answer.

Jevtypesafe-ai/jev
96.7%
500 of 517 correct · 95% CI 94.8%97.9%
Median response
262 ms
Cost / 1k reviews
$0.0267
Purpose-built evaluation model
Qwen3.7 Flashalibaba/qwen3.7-flash
96.7%
500 of 517 correct · 95% CI 94.8%97.9%
Median response
4.6 s
Cost / 1k reviews
$0.0670
General-purpose flash LLM

Gap 0.0 ptsPaired test p = 1.000Class balance 50 / 50Chance 50%

  1. 01Sample

    1,000 reviews drawn at random from 49,581 unique IMDB reviews, half positive and half negative (seed 42).

  2. 02Blind

    The human label is set aside. Each model sees only the review and the task.

  3. 03Ask

    Both models are called through Vercel AI Gateway with the same instructions and label definitions.

  4. 04Score

    Every answer is checked against the IMDB star-rating label and filed, once.

Technical report · September 2026

Blind binary sentiment classification on IMDB: an evaluation model versus a general-purpose flash LLM

Abstract

We compare typesafe-ai/jev, an evaluation model that returns a structured choice with probabilities, against alibaba/qwen3.7-flash, a general-purpose generative model, on 517 class-balanced movie reviews from the IMDB dataset of Maas et al. [1]. Neither model saw the reference label. Jev labelled 96.7% of reviews correctly (95% CI 94.8%97.9%) and Qwen3.7 Flash 96.7% (94.8%97.9%). The two models tied. Both models missed the same 12 reviews; we publish all 517 answers for inspection.

1Headline accuracy

Accuracy is the share of all 517 reviews labelled correctly; an answer that could not be parsed counts as wrong. Because the sample is exactly half positive and half negative, a coin flip scores 50%, and so does a model that always answers the same way.

Jev
96.7%
Qwen3.7 Flash
96.7%
90%95%100%
Figure 1. Accuracy with 95% Wilson score intervals [2]. Intervals are for each model on its own; the paired comparison is in Figure 4.
ModelAccuracy95% CIMacro F1Recall +Recall −No answerMedian latencyTokens in / outExpected cost / 1kBilled / 1k
Jev96.7%94.8%97.9%0.96796.9%96.6%0262 ms637 / 32$0.0267$0.0000
Qwen3.7 Flash96.7%94.8%97.9%0.96797.3%96.2%04.6 s384 / 426$0.0670$0.0670
Table 1. Recall + / − is accuracy on positive and on negative reviews. Median latency comes from a separate speed test of 50 reviews per model: one request at a time from one client through AI Gateway, each timed as a single attempt, with any request the gateway throttled discarded rather than retried (95th percentile: Jev 2.6 s, Qwen3.7 Flash 11.2 s). Tokens are the mean per review. Expected cost is those tokens at list price (Jev $0.042 in / $0 out; Qwen3.7 Flash $0.03 in / $0.13 out per million tokens), whatever free tier or credits apply. Billed is what AI Gateway charged this run; Jev was billed $0 on this account (free tier).

2Every verdict, one square each

Each square below is one review, in the same position in both grids, so you can compare the two models review by review. Grey squares are correct and red squares are mistakes. Hover a square to see what each model said, or switch the highlight to the reviews where the models disagreed.

  • Correct
  • Wrong
  • No usable answer
Jev17 missed
Positive reviews255
Negative reviews262
Qwen3.7 Flash17 missed
Positive reviews255
Negative reviews262
Figure 2. All 517 verdicts per model, grouped by the human label. Full per-review answers are in the CSV download.
Jev
actual ↓  predicted →positivenegative
positive2478
negative9253
Qwen3.7 Flash
actual ↓  predicted →positivenegative
positive2487
negative10252
Figure 3. Confusion matrices. Rows are the human label and columns are the model’s answer; shading is the share of the row. Errors off the diagonal show whether a model leans positive or negative.

3Where they disagree

Both models saw the same reviews, so the fair comparison is paired. The models agreed on 507 reviews (98.1%) and disagreed on 10. Only those disagreements tell the models apart, and McNemar’s test [3] asks whether they split more unevenly than chance would. Jev was right on 5 and Qwen3.7 Flash on 5 (p = 1.000).

Qwen3.7 Flash right Qwen3.7 Flash wrong
Jev right49595.7%51.0%
Jev wrong51.0%122.3%
Figure 4. Paired outcomes. The off-diagonal cells are the discordant pairs used by the exact McNemar test.

Does length matter? Short reviews carry fewer clues, while long ones may spend paragraphs on plot before giving a verdict. Figure 5 splits the sample into five equal groups by word count.

  • Jev
  • Qwen3.7 Flash
90%
95%
100%
99.0%97.1%
42116116150150201205327327986
Review length, words (five equal-size groups of 103)
View as table
WordsReviewsJevQwen3.7 Flash
4211610396.1%96.1%
11615010496.2%96.2%
15020110396.1%97.1%
20532710496.2%97.1%
32798610399.0%97.1%
Figure 5. Accuracy by review length. Each group holds 103 reviews, so each point has a 95% interval of roughly ±3 points. Read the trend, not single points.

4Does Jev know when it’s wrong?

As an evaluation model, Jev returns a probability for each label, not just an answer. A well-calibrated model that says “90% sure” should be right about 90% of the time. Its average confidence was 99.3% on answers it got right and 87.8% on answers it got wrong. The expected calibration error [4] is 2.2 points, which is well calibrated. Qwen3.7 Flash returns only text, so it has no equivalent chart.

0%
25%
50%
75%
100%
perfectly calibrated
50%60%70%80%90%100%
Jev’s probability for the label it chose · bubble area = number of reviews
View as table
Confidence binReviewsMean confidenceAccuracy
50%55%152.0%0.0%
60%65%261.0%50.0%
65%70%369.0%66.7%
70%75%272.5%50.0%
80%85%783.7%85.7%
85%90%688.5%66.7%
90%95%794.3%71.4%
95%100%48999.8%98.4%
Figure 6. Reliability diagram for Jev, in 5-point confidence bins. Points below the diagonal are over-confident and points above it are under-confident.

5Error analysis

Accuracy doesn’t show what the mistakes look like. These are the actual reviews the models got wrong, with the human label and each model’s answer.

6Method

Data. The IMDB Large Movie Review dataset [1] labels a review negative if its star rating is 4/10 or lower and positive if it is 7/10 or higher; mid-range reviews are excluded. From the 50,000-review distribution we dropped 419 exact duplicates, converted <br /> tags to line breaks, and drew 500 positive and 500 negative reviews with a seeded shuffle (seed 42). The sample is committed in data/sample.json.

Blinding. A request contains only the review text, the task instructions and the two label definitions below. The reference label stays on disk and is joined back only when scoring.

Models. Both models were called once per review through Vercel AI Gateway with the AI SDK (ai@7.0.113). Jev is an evaluation model, called with experimental_evaluate and a single choice question; it returns the chosen label with a probability for each option. Qwen3.7 Flash is a chat model, called with generateText at temperature 0. Its reply was matched against the words “positive” and “negative”, and a reply containing neither or both counted as no answer (there were none).

Jev · typesafe-ai/jev
{
  "state": "<review text>",
  "questions": {
    "sentiment": {
      "type": "choice",
      "instructions": "Classify the overall sentiment the author expresses toward the film in this IMDB movie review.",
      "criteria": {
        "positive": "The reviewer's overall opinion of the movie is favorable.",
        "negative": "The reviewer's overall opinion of the movie is unfavorable."
      }
    }
  }
}
Qwen3.7 Flash · alibaba/qwen3.7-flash
system
You are a sentiment classifier.
Classify the overall sentiment the author expresses toward the film in this IMDB movie review.
positive: The reviewer's overall opinion of the movie is favorable.
negative: The reviewer's overall opinion of the movie is unfavorable.
Answer with exactly one lowercase word, positive or negative, and nothing else.

user
<review>
<review text>
</review>

Listing 1. The exact requests. Both models get the same instructions and label definitions.

Statistics. Intervals are 95% Wilson score intervals [2]. The two models are compared with a two-sided exact McNemar test [3] (a binomial test on the discordant pairs). Calibration uses expected calibration error [4] over 5-point bins of the evaluation model’s probability for its chosen label.

7Limitations

  • An easier task than real use. Excluding 5–6/10 reviews removes the genuinely ambivalent cases, so real product reviews will score lower.
  • Labels come from star ratings, not the text. A reviewer can write something that reads like a pan and still give it 7 stars. Some of each model’s “errors” may be label noise; the gallery above is the place to check.
  • Possible contamination. IMDB reviews have been public since 2011 and may be in either model’s training data.
  • One run, one prompt. Results are from a single pass with one wording. Different instructions, few-shot examples or a different answer parser could move the numbers by a point or two.
  • Different interfaces. The evaluation model answers in a fixed format; the chat model has to be parsed. That is part of what is being compared, not a flaw to correct for.

References

  1. A. L. Maas, R. E. Daly, P. T. Pham, D. Huang, A. Y. Ng, C. Potts. Learning Word Vectors for Sentiment Analysis. ACL, 2011. ai.stanford.edu/~amaas/data/sentiment
  2. E. B. Wilson. Probable inference, the law of succession, and statistical inference. JASA 22(158), 1927.
  3. Q. McNemar. Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika 12(2), 1947.
  4. C. Guo, G. Pleiss, Y. Sun, K. Q. Weinberger. On Calibration of Modern Neural Networks. ICML, 2017.

Data & reproduction

Every answer is filed in data/results.json and in results.csv, one row per review. The site is static and makes no model calls. To re-run:

npm install
cp .env.example .env.local        # add AI_GATEWAY_API_KEY
npm run eval                      # 1,000 reviews × 2 models, resumable
npm run build