Technical report · September 2026
Blind binary sentiment classification on IMDB: an evaluation model versus a general-purpose flash LLM
Abstract
We compare typesafe-ai/jev, an evaluation model that returns a structured choice with probabilities, against alibaba/qwen3.7-flash, a general-purpose generative model, on 517 class-balanced movie reviews from the IMDB dataset of Maas et al. [1]. Neither model saw the reference label. Jev labelled 96.7% of reviews correctly (95% CI 94.8%–97.9%) and Qwen3.7 Flash 96.7% (94.8%–97.9%). The two models tied. Both models missed the same 12 reviews; we publish all 517 answers for inspection.
1Headline accuracy
Accuracy is the share of all 517 reviews labelled correctly; an answer that could not be parsed counts as wrong. Because the sample is exactly half positive and half negative, a coin flip scores 50%, and so does a model that always answers the same way.
| Model | Accuracy | 95% CI | Macro F1 | Recall + | Recall − | No answer | Median latency | Tokens in / out | Expected cost / 1k | Billed / 1k |
|---|---|---|---|---|---|---|---|---|---|---|
| Jev | 96.7% | 94.8%–97.9% | 0.967 | 96.9% | 96.6% | 0 | 262 ms | 637 / 32 | $0.0267 | $0.0000 |
| Qwen3.7 Flash | 96.7% | 94.8%–97.9% | 0.967 | 97.3% | 96.2% | 0 | 4.6 s | 384 / 426 | $0.0670 | $0.0670 |
2Every verdict, one square each
Each square below is one review, in the same position in both grids, so you can compare the two models review by review. Grey squares are correct and red squares are mistakes. Hover a square to see what each model said, or switch the highlight to the reviews where the models disagreed.
- Correct
- ✕ Wrong
- No usable answer
| actual ↓ predicted → | positive | negative |
|---|---|---|
| positive | 24797% | 83% |
| negative | 93% | 25397% |
| actual ↓ predicted → | positive | negative |
|---|---|---|
| positive | 24897% | 73% |
| negative | 104% | 25296% |
3Where they disagree
Both models saw the same reviews, so the fair comparison is paired. The models agreed on 507 reviews (98.1%) and disagreed on 10. Only those disagreements tell the models apart, and McNemar’s test [3] asks whether they split more unevenly than chance would. Jev was right on 5 and Qwen3.7 Flash on 5 (p = 1.000).
| Qwen3.7 Flash right | Qwen3.7 Flash wrong | |
|---|---|---|
| Jev right | 49595.7% | 51.0% |
| Jev wrong | 51.0% | 122.3% |
Does length matter? Short reviews carry fewer clues, while long ones may spend paragraphs on plot before giving a verdict. Figure 5 splits the sample into five equal groups by word count.
- Jev
- Qwen3.7 Flash
View as table
| Words | Reviews | Jev | Qwen3.7 Flash |
|---|---|---|---|
| 42–116 | 103 | 96.1% | 96.1% |
| 116–150 | 104 | 96.2% | 96.2% |
| 150–201 | 103 | 96.1% | 97.1% |
| 205–327 | 104 | 96.2% | 97.1% |
| 327–986 | 103 | 99.0% | 97.1% |
4Does Jev know when it’s wrong?
As an evaluation model, Jev returns a probability for each label, not just an answer. A well-calibrated model that says “90% sure” should be right about 90% of the time. Its average confidence was 99.3% on answers it got right and 87.8% on answers it got wrong. The expected calibration error [4] is 2.2 points, which is well calibrated. Qwen3.7 Flash returns only text, so it has no equivalent chart.
View as table
| Confidence bin | Reviews | Mean confidence | Accuracy |
|---|---|---|---|
| 50%–55% | 1 | 52.0% | 0.0% |
| 60%–65% | 2 | 61.0% | 50.0% |
| 65%–70% | 3 | 69.0% | 66.7% |
| 70%–75% | 2 | 72.5% | 50.0% |
| 80%–85% | 7 | 83.7% | 85.7% |
| 85%–90% | 6 | 88.5% | 66.7% |
| 90%–95% | 7 | 94.3% | 71.4% |
| 95%–100% | 489 | 99.8% | 98.4% |
5Error analysis
Accuracy doesn’t show what the mistakes look like. These are the actual reviews the models got wrong, with the human label and each model’s answer.
Reviews where exactly one model was right. Jev won 5 of these 10. Shortest first.
We went to the cinema expecting a biggish budget release and got an art-house movie. The movie was projected digitally onto about two thirds of the screen real estate with sloping edges classic of digital projection, and had a limited stereo soundtrack which was wasted on the cinema experience. The content of the film was the same old historical content we have all seen before, but heavily sanitized to prevent the audience being sick. Live action scenes what little of them there were, were re-used constantly in classic documentary style, which became annoying after a while. I was somewhat amazed that only 4 people turned up to watch it, guess the rest knew something we didn't. I suspect the producers made the film to recognize the ninetieth anniversary of Gallipoli. I have to question whether they should have bothered. Seven out of Ten for trying, and out of respect for the ANZAC's.
This movie isn't terrible, really. Somebody commented that Mo is the type of American Europeans snicker at. But there are those, and not necessarily Anglo-Saxon yahoos, who do not care for Frenchmen; and the Xavier character isn't going to sway them. Let's consider his stereotypical Frenchman attributes: 1). Cynical - very cynical. Check. 2). Reedy, underfed appearance, check, despite: 3). A great appreciation of cuisine. Check. 4). Lukewarm work ethic. Check. (Forget the fact he is supposedly a rich stockbroker, from watching him in the film he seems to put in ten hour workweeks.) 5). Beautiful wife, check. Despite that: 6). Loose interpretation of the marriage vows. Check. 7). Big sexual ego, which says an American girl owes you sex if you buy her dinner. Check. Whether Mo is a hick or not, there's no reason for her to fall for this smug European twit other than the script dictates so. On the other hand, as other male reviewers have, I did enjoy seeing Karen Allen's cute, petite body. I'll give the movie four stars; two of them are for that.
Jack Black and Kyle Gass play fantasy versions of themselves in this comic showcase for their side-band Tenacious D, an art-rock outfit with satirical, barbed lyrics. An ex-runaway obsessed with heavy metal and a beachfront-living, pot-smoking slacker who pretends he's a rock god meet and form a band (the birthmarks on both their butt-cheeks form the group's moniker). Opening with a funny prologue which apes a Twisted Sister video from the '80s, "The Pick of Destiny" is a fairly well-produced movie aimed at older kids; it occasionally resembles nothing more than a middle-aged variation of "Wayne's World", with jokey-stoner interludes and a climactic bout with Beelzebub himself, yet Black and Gass have an enormously comfortable rapport (they also acted as producers, co-wrote the script and all the music). The target audience will obviously go for it, though inspiration is a bit low, particularly in the second-half (just about the time our heroes impulsively outrun the cops in a student-driver car). The music sequences are far more successful than the attempts at movie satire and, for the first thirty minutes or so, Jack Black's manic enthusiasm is infectious. *1/2 from ****
This film captures the short moments between a mother and son in rural Russia, as she lays dying. I am so torn between being nice to the film or declaring it a test of patience. On one hand, the film is beautiful, with the sparse dialog capturing the essence of their feelings. There is really nothing to say, because everything that needs to be said is conveyed beyond words. The son shows so much care, love and patience towards his mother, that I think it is a celebration of unconditional love towards one's family. It also cruelly reminds me that I could be in a situation like this, stuck in a joyless place, having to take care of a very ill person. "Mat I Syn" is cruel reality. On the other hand, "Mat I Syn" moves really too slowly. Do I really need to watch a train passing by the horizon for over 1 minute? With my previous experience of "Telets" and "Aleksandra", I am so tempted to put "Mat I Syn" among them as a total bore. I guess one has to be in the right state of mind to appreciate this film. I surely see the beauty of it, but maybe I am not in the right state of mind.
Capt. Gallagher (Lemmon) and flight attendant Eve Clayton (Vaccaro) are a supposedly hot item in this death trip; a luxury 747 airliner decked out to look like a nightclub-slash-hotel there's even a blind piano player who falls in love. Karen Wallace (Grant) is the hysterical b!$3& who'll do anything to get attention from henpecked husband Martin (Christopher Lee) and, later, the rest of the people on board. Memorable Moments: Boeing 747 doing a belly flop in the Atlantic Ocean, Karen getting her chops busted when she goes too far, and furniture (and screaming people) who become 'ball bearings' in a sinking 'pinball machine.' The action and rescue sequences here are relatively phenomenal, but not much goes on in between. Hitchcock was supposed to have directed this sequel, but I forget the reason why not He would've done wonders for the 1970 original, on which this sequel is partly inspired ('77 also got inspiration from `The Flight of the Phoenix'). Actors Cotten and de Havilland reunite from their days on `Hush, Hush, Sweet Charlotte' (apparently here they are not playing heavies, just reunited Autumn Years' lovers). And isn't the actress playing Emily's companion the same one who played the hammered-to-death maid on `Whatever Happened to Baby Jane?' TV actors include the girlfriend from `Mayberry RFD' (her character's daughter wins a drawing contest, or something lame like that), `Buck Rogers' Gil Gerard and `Dynasty's' Pamela Bellwood.
'Bloody Birthday' is an odd and, at times, humorous low-budget horror flick along the lines of 'Mikey' or a less intelligent version of 'The Good Son'. Set in a small Californian town, three babies are born at the height of an eclipse, where planetary alignment means they are somehow born without emotions. Ten years later, our three little psychopaths take themselves on a killing spree, doing away with parents, siblings, teachers and anyone else who irritates them. Only one teenage girl knows the truth to be able to stop them. There is no explanation for why babies across the world born at the same time aren't equally as twisted but there you go! For a slasher film, it's very tame in terms of violence and gore, which I suppose highlights the problem of casting child characters as the killers as there is only so much you can expose the young actors to. Instead, it's amusing and a little disturbing seeing three ten-year-olds plotting murders and carrying out their plans using guns, knives and crossbows. The main reason why it doesn't descend into being totally ridiculous is because the child actors are very convincing in their roles with the way they slyly play the little innocents in front of undiscerning adults while showing their dark side to the girl who knows the truth. 'Bloody Birthday' is rather mediocre as a horror flick, with few scares and little blood, but because it has the shock factor of having kids as the killers, it is a bit unique in that way. One to watch if there's nothing else on.
A friend once asked me to read a screenplay of his that had been optioned by a movie studio. To say it was one of the most inept and insipid scripts I'd ever read would be a bold understatement. Yet I never told him this. Why? Because in a world where films like "While She Was Out" can be green-lighted and attract an Oscar- winning star like Kim Basinger, a screenplay lacking in character, content and common sense is no guarantee that it won't sell. As so many other reviewers have pointed out, "While She Was Out" is a dreadfully under-written Woman-in-Peril film that has abused housewife Basinger hunted by four unlikely hoods on Christmas Eve. Every gripe is legitimate, from the weak dialog and bad acting to the jaw-dropping lapses of logic, but Basinger is such an interesting actress and the premise is not without promise. Here are a couple of things that struck me: 1) I don't care how much we are supposed to think her husband is a jerk, the house IS a mess with toys. Since when did it become child abuse to make kids pick up after themselves? 2) Racially diverse gangs are rare everywhere except Hollywood, where they are usually the only racially balanced groups on screen. 3) Sure the film is stupid. But so are the countless "thrillers" I've sat through where the women are portrayed as wailing, helpless victims of male sadism. Stupid or not, I found it refreshing to see a woman getting the best of her tormentors. 4) I LOVED the ending! 5) Though an earlier reviewer coined this phrase, I really DO think this film should be retitled "The Red Toolbox of Doom."
This one's a doozy. Dating from 1949, Scene of the Crime often plays more like a Coen Bros. movie set in the 1940's and filmed in black and white, except that the writer's ear for pastiche here isn't quite so well-tuned -- maybe this can be seen instead as the forerunner to Oscar-baiting schlock like Road to Perdition. Frankly, it's a wonder that this film isn't considered a classic by film professors and critics everywhere, considering how much it offers in term of overly articulated mannerist thrills cloaked in false significance ( much like the grandaddy of all such "fake art" films, Citizen Kane, or anything by Murnau. ) MGM is usually a studio that can do no wrong in my eyes, and I think any story, any atmosphere, even "gritty realism," can only benefit from grotesque overaestheticization. You could say I'm a disciple of the Minnelli school. But it takes a certain light touch to write mannered tough-guy dialogue of the Dashiell Hammett stripe, a willingness, perhaps, to let maybe one or two scenes pass without a line like "Careful, Mr. Wiggly, or you'll have thirteen fish to fry and no little wormies to catch them with." I made most of that up -- "Mr. Wiggly," unfortunately, made the cut -- but believe me, the dialogue is just that loonily inflated and riddled with non sequiturs. Even the lead cop's wife played by Arlene Dahl speaks like she has a moon-shaped scar under one eye and the Christian name Rocco. By the time Van Johnson turns in his badge with the line, "I'm sick to death of death and homicide," you'll wonder how the writer's fixation with ornate literary devices -- in this case, zeugma -- could ever have been misconstrued as "street." For those who have outgrown The Naked Gun series, this is the funniest cops-n-robbers film going.
I would probably want to give this movie a zero if not for the climax, which involves not really Snakes on a Train, but rather Train IN a Snake. The premise was cooked up far more than likely over the course of a night of beers after hearing about Snakes on a Plane in production (this, in fact, was released to coincide with that film's release). The joke is probably not lost on those who will seek this out; I don't think there would be a soul out there who would consider this anything as a serious action-thriller effort (unless on an ironic level beyond the capacity for rational thought). It's about a Mayan curse placed on a woman who's damned by her family for leaving with another man, and is soon seen sickened and coughing up green slime laced with, of course, snakes. She and her beau go on a train headed for Los Angeles, and very soon after the more-than-cliché characters are privy to snakes overtaking the train- with the originator woman becoming a snake herself. If it would be worth listing more about the movie I would, but there isn't enough time during the day. All that can be said for the quality factor is that it's almost on-existent; there are student short films with larger budgets. Maybe that was a wise calculation on the filmmakers' end, that there would be so many copies sold, just for the joke factor alone, that they would re-coup their budget in the first weekend. Because by looking at the sets (the trains themselves change randomly in the middle of a scene!), the actors (if you can call them that, with only one other actor- the one with the very thin hair who hits on the one woman throughout the movie- who benefited from the flick being produced), the FX (also next to non-existent, making the effects in Snakes on a Plane seem like Star Wars), and the actual CGI snakes themselves, with the final huge behemoth snake something to behold in sci-fi movie channel terms. This all means, basically, that it is a laugh riot every step of the way (especially, as cruel as it sounds, when a little girl becomes involved in a snake's "attention"), with the very disregard for good taste working well in its favor. This being said, it is also 100% disposable, like a B-movie sour-flavor lollipop.
"The Duke" is a film based in the heart of the British aristocracy where an old Duke (John Neville) dies and to avoid his large property and the vast riches to be taken from him after death by 2 devious aristocrats (Oliver Muirhead and Sophie Heyman); he passes his Dukeship down to his dog. The dog's "best friend", an American girl named Charlotte (played by Courtnee Draper) whose parents die becomes orphaned by the former Duke due to him being her great uncle. A young chef named Florian arrives, with him and Charlotte instantly connecting creating a romantic sub-plot which in its own way, acts as the under tone for the main plot line being the activities of Hubert, The 'Duke' and his many activities and love interests with other dogs. All this is watched over by James Doohan who plays a Butler who is determined to try and serve his old Duke by doing his best to serve his new master, Hubert. Doohan acts as the older character there to comfort and advise the younger ones whilst he over looks and performs various ridiculous tasks under his new master. A fine comedic performance mixed with elements of drama to end the career of a fine and influential actor. Though from the technical viewpoint, I dislike this film as I only watched upon discovering James Doohan's role. Though the plot is good, elements of the writing and directing have to be obscured. Ignoring the absurdity of a "Doggie Duke", I personally dislike the over use of comedic and outrageous jokes upon the 'bad' characters due to their ludicrousness. The dependence on hygiene related humour as you like is much too apparent and general silliness of many characters doesn't appeal to me. The director, Phillip Spink uses mid-long shots too often with either an overdubbed voice or affecting the overall sound quality. Plus, the acting of Muirhead and Heyman, whilst good at the dramatic and romantic sequences, fall drastically short fulfilling the wholesale requirements regarding the role. But, as a mature movie fan, I find it easy to be over critical of a simple family film designed to make you laugh. I can admit I found certain bits funny and other bits touching along with a plot that may be far-fetched, but has definite connections. I do not recommend this film to mature movie fans but I do highly recommend it to Dog lovers, families who wish to enjoy a funny film for their child and James Doohan fans who wish to see 'Scotty' in his final role.
6Method
Data. The IMDB Large Movie Review dataset [1] labels a review negative if its star rating is 4/10 or lower and positive if it is 7/10 or higher; mid-range reviews are excluded. From the 50,000-review distribution we dropped 419 exact duplicates, converted <br /> tags to line breaks, and drew 500 positive and 500 negative reviews with a seeded shuffle (seed 42). The sample is committed in data/sample.json.
Blinding. A request contains only the review text, the task instructions and the two label definitions below. The reference label stays on disk and is joined back only when scoring.
Models. Both models were called once per review through Vercel AI Gateway with the AI SDK (ai@7.0.113). Jev is an evaluation model, called with experimental_evaluate and a single choice question; it returns the chosen label with a probability for each option. Qwen3.7 Flash is a chat model, called with generateText at temperature 0. Its reply was matched against the words “positive” and “negative”, and a reply containing neither or both counted as no answer (there were none).
typesafe-ai/jev{
"state": "<review text>",
"questions": {
"sentiment": {
"type": "choice",
"instructions": "Classify the overall sentiment the author expresses toward the film in this IMDB movie review.",
"criteria": {
"positive": "The reviewer's overall opinion of the movie is favorable.",
"negative": "The reviewer's overall opinion of the movie is unfavorable."
}
}
}
}alibaba/qwen3.7-flashsystem You are a sentiment classifier. Classify the overall sentiment the author expresses toward the film in this IMDB movie review. positive: The reviewer's overall opinion of the movie is favorable. negative: The reviewer's overall opinion of the movie is unfavorable. Answer with exactly one lowercase word, positive or negative, and nothing else. user <review> <review text> </review>
Listing 1. The exact requests. Both models get the same instructions and label definitions.
Statistics. Intervals are 95% Wilson score intervals [2]. The two models are compared with a two-sided exact McNemar test [3] (a binomial test on the discordant pairs). Calibration uses expected calibration error [4] over 5-point bins of the evaluation model’s probability for its chosen label.
7Limitations
- An easier task than real use. Excluding 5–6/10 reviews removes the genuinely ambivalent cases, so real product reviews will score lower.
- Labels come from star ratings, not the text. A reviewer can write something that reads like a pan and still give it 7 stars. Some of each model’s “errors” may be label noise; the gallery above is the place to check.
- Possible contamination. IMDB reviews have been public since 2011 and may be in either model’s training data.
- One run, one prompt. Results are from a single pass with one wording. Different instructions, few-shot examples or a different answer parser could move the numbers by a point or two.
- Different interfaces. The evaluation model answers in a fixed format; the chat model has to be parsed. That is part of what is being compared, not a flaw to correct for.
References
- A. L. Maas, R. E. Daly, P. T. Pham, D. Huang, A. Y. Ng, C. Potts. Learning Word Vectors for Sentiment Analysis. ACL, 2011. ai.stanford.edu/~amaas/data/sentiment
- E. B. Wilson. Probable inference, the law of succession, and statistical inference. JASA 22(158), 1927.
- Q. McNemar. Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika 12(2), 1947.
- C. Guo, G. Pleiss, Y. Sun, K. Q. Weinberger. On Calibration of Modern Neural Networks. ICML, 2017.
Data & reproduction
Every answer is filed in data/results.json and in results.csv, one row per review. The site is static and makes no model calls. To re-run:
npm install cp .env.example .env.local # add AI_GATEWAY_API_KEY npm run eval # 1,000 reviews × 2 models, resumable npm run build