Judge duel
Which small model can catch a three-armed dinosaur?
We make illustrated books and multi-scene videos with open-weight image models. Some of the pictures come out with three arms. We tested which judge should catch them: a frontier API, three open "typed decision" models, and four other open vision models, on public data and on our own live book service.
TL;DR
- Winner: Jev-Omni, an open 12B typed-decision model, in 4-bit on one consumer GPU.
- Against the frontier API: as accurate or better, a third of the false alarms on our book art, and about 0.12 s of model time per image instead of seconds.
- Small models: none of the 2B to 9B models we tried matched it on images.
- Surprise: for images, the prompt format mattered more than the fine-tune.
- One 24 GB card: one Jev-Omni in 4-bit plus one small SemIf text model, about 21 GB. Two copies of the same model add isolation, not speed.
- Lesson: the model was fast, the plumbing was not. Measure the whole path.
Why we needed a judge
Brains.biz turns a book idea into a finished Amazon KDP book: research, text, and a picture on every page. Uncen, the engine underneath it, makes multi-scene videos the same way. Both use open-weight image models on our own GPUs, and open models sometimes draw a dinosaur with three arms, a cat with two tails, or two raptors fused into one body.
So every image now goes through a judge. When the judge sees a defect, the system draws the picture again, keeps up to three versions, and uses the best one. The judge has to be accurate, but it also has to avoid false alarms, because every false alarm is a wasted redraw. And it has to be fast, because it runs on every single image.
Our first judge was DeepSeek 4.1 Flash, a strong and cheap vision API. It worked, but it took about 8 seconds per image in production, and it kept flagging pictures that were fine.
What a "typed decision" model is
Most software decisions are small: is this image defective, yes or no? A chat model answers by writing text that the program then parses back into an if. A typed-decision classifier skips the writing. It reads the probability of each answer option straight from one forward pass. That makes it fast, and the probability is directly usable as a threshold.
Jev is TypeSafe's closed service for this, and it is text only. Three independent open projects copy the interface:
- Jev-Omni (
akhilaaa3/Jev-Omni): a 12B model fine-tuned from Gemma 4, for text, images, audio and video. Apache-2.0. - openjev (
AlexWortega/openjev): a 4B entailment model. It judges whether an image supports a written claim. MIT. - SemIf: a method that reads option probabilities from a normal open model. We run it with Qwen3.5-4B as our production text classifier.
None of these projects is affiliated with TypeSafe or with us.
How we tested
- Public set: 200 images from Distorted-5K (Apache-2.0), AI images of people with distorted regions marked by three human annotators. 100 defective (two or more annotators agree), 100 clean (no annotator marked anything).
- Our set: 133 real pictures from eight Brains.biz books: dinosaurs, watercolor animals, coloring-book line art, sci-fi scenes. One person labeled them. Four have a clear defect.
- Text: 272 questions from DecisionBench Medium and SemIf's own 144 short decisions.
- Hardware: the accuracy runs used an RTX 3060 12 GB. Production speed comes from a live RTX 3090 in our fleet.
We kept the rigor at "good enough to decide". The public intervals are about plus or minus 0.07 AUROC, and four defects in our set is a small number. We say where that matters.
Result 1: the frontier API was not the best judge
| Judge | Public AUROC | Our books AUROC | Clean book pictures flagged | Time per image |
|---|---|---|---|---|
| Jev-Omni 12B, 4-bit | 0.73 | 0.89 | 10 of 129 | 0.3 s (RTX 3090) |
| DeepSeek 4.1 Flash (API) | 0.70 | 0.74 | 31 of 128 | 2.9 s (8 s in production) |
| openjev 4B | 0.55 | 0.63 | n/a | 0.9 s (RTX 3060) |
The public column uses the same 60 images for all three, because we did not want to spend API calls on thousands of images. Both caught three of our four real defects. The difference is the false alarms: DeepSeek flagged one clean book picture in four.
Reading its reasons explained why. It mixes "does not match the prompt" with "is deformed". It marked a correct four-legged hadrosaur as having extra legs, and scored a correct two-character scene 0 out of 100 because the prompt described one character. Those are real observations, but they are not reasons to redraw a picture.
openjev was close to chance. Asking "does the image support the claim that a creature has an extra limb?" does not work for anatomy.
Result 2: no small model matched Jev-Omni on images
| Model | Public AUROC | Our books AUROC | Clean book pictures flagged | ms per image (3060) | VRAM |
|---|---|---|---|---|---|
| Jev-Omni 12B, 4-bit | 0.70 | 0.89 | 9 of 129 | 561 | 7.3 GB |
| Gemma-4-12B base, 4-bit, Jev prompt | 0.68 | 0.87 | 15 of 129 | 560 | 7.3 GB |
| Qwen3.5-9B, 4-bit | 0.68 | 0.80 | 18 to 42 of 129 | 1,556 | 7.7 GB |
| Qwen3.5-4B, BF16 | 0.61 | 0.77 | 24 of 129 | 1,014 | 8.8 GB |
| Qwen3.5-2B, BF16 | 0.62 | 0.74 | 28 of 129 | 570 | 4.3 GB |
| MiniCPM-V 4.6, BF16 | 0.48 | 0.79 | 48 of 129 | 498 | 2.7 GB |
Every model was read the same way: the probability of "Yes" to the question "Does this image contain a body or anatomy error, such as an extra or missing limb, extra or missing fingers, extra heads, merged bodies, or a deformed face or body?". The threshold for each model catches 60 of the 100 public defects.
- Qwen3.5-9B ties on the public set (within noise), but it is three times slower and raises two to five times the false alarms on illustrated book art.
- The 2B and 4B models are usable but clearly weaker. If you only have 6 GB free, Qwen3.5-2B is the least bad option.
- MiniCPM-V answered "No" to almost every public image, so this method does not suit it.
Result 3: the prompt format mattered more than the fine-tune
The surprise: plain Gemma-4-12B, with no Jev training, nearly matched Jev-Omni on images when we used Jev-Omni's numbered-option prompt (0.68 public, 0.87 ours). With a plainer prompt it dropped to 0.64 and 0.77. For images, Jev-Omni's advantage comes mostly from Gemma's vision plus a good decision format. Its fine-tune shows up much more on text.
Result 4: on text, Jev-Omni is far more accurate, SemIf is faster
| Judge | DecisionBench (272 q) | Calibration error | SemIf's 144 short decisions | Short decision (3090) |
|---|---|---|---|---|
| Jev-Omni 12B, 4-bit | 0.83 | 0.040 | 0.90 | 0.12 s |
| SemIf with Qwen3.5-4B | 0.65 | 0.078 | 0.81 | about 0.05 s |
| Qwen3.5-9B, 4-bit | 0.66 | 0.155 | 0.92 | about 0.1 s |
DecisionBench states are long, about 2,300 tokens each. There Jev-Omni is 18 points ahead, far beyond the 3-point sampling noise. On short, simple decisions everyone does well, and SemIf's small model answers twice as fast. Our SemIf run scored 0.809 on its own set against its published 0.813, which confirms the test harness. One caution: DecisionBench comes from the Jev-Omni author, so it may suit Jev-Omni. Jev-Omni still beat SemIf on SemIf's own set.
What fits on one 24 GB card
Our fleet is three consumer boxes with 24 GB cards, so this was the question we cared about most. Two measurements decide it.
Memory versus input length. Jev-Omni in 4-bit is almost flat: 7.2 GB of weights, 7.4 GB at 1,000 tokens, 8.4 GB at 9,000. The Qwen3.5 models grow fast, from 8 GB to 19.6 GB at 9,000 tokens. Part of that is our Windows test machine, which lacks the fast attention kernels. Measure your Linux host before you plan long inputs.
Copies versus throughput. An image or a long prompt already keeps the GPU busy with one request. Batching inside one process gained 1.4x for images, 1.6 to 2.3x for short text, and nothing for long text. A second copy of the same model on the same card adds isolation, not speed.
| Configuration on 24 GB | VRAM | Verdict |
|---|---|---|
| 1 Jev-Omni 4-bit + 1 SemIf 4B, SemIf input capped near 4.5k tokens | about 21 GB | Recommended |
| 1 Jev-Omni 4-bit alone | about 9 GB | Simplest; the best on every quality test |
| 2 Jev-Omni 4-bit | about 18 GB | Fits; isolation only, no speed gain |
| 1 Jev-Omni + 2 SemIf | about 28 GB | Does not fit |
| 2 small vision models doing text and images | about 20 GB, short inputs only | Weaker on both tasks; not worth it |
| 1 Jev-Omni 8-bit | about 14 GB | Fits; avoids 4-bit drift; test its speed first |
| On a 32 GB RTX 5090: 1 Jev-Omni BF16 | about 24 GB | Full precision, no drift |
On our live RTX 3090, Jev-Omni in 4-bit uses 7.9 GB, answers a short text question in about 115 ms, and judges an image in 285 to 346 ms. It loads in 8.4 seconds. On the RTX 4090 where it now lives, one image takes 119 to 150 ms.
One honest caveat about speed. The model is fast, but our first production judge call took 5 to 7 seconds. Almost all of it was plumbing. The GPU host downloaded each 2 MB PNG from storage (2.2 to 2.5 s), and the request crossed a job queue both ways. We fixed the download first: the web server now fetches the image in 0.6 s, shrinks it to a 768 px JPEG of about 100 KB, and puts it inside the job. The model only sees about 750 px anyway, so nothing is lost. That removed the download from the GPU host completely. Then we went further: pictures drawn on the same machine as the judge are now checked there, the moment they are drawn, and the answer is stored with the picture. For those pictures the judge step dropped from about 4 seconds to 18 to 56 milliseconds, and the image job pays about one second for it. Pictures from our other GPU boxes still use the small-JPEG path. Measure the full path, not only the model.
A note on 4-bit
The full Jev-Omni needs about 50 GB. The 4-bit build fits a consumer card, and on confident yes/no cases it matched the author's reference answers within 0.03. On an uncertain four-way question it drifted by 0.39. For a yes/no defect screen that is fine. For fine-grained probabilities, use BF16 on a bigger card.
How we use it
Every picture in a Brains.biz book now goes through Jev-Omni first. In our test it flagged about one clean picture in thirteen, against one in four for the API. Only the flagged pictures go to DeepSeek, which writes the list of problems the author sees next to each picture. A defect starts a redraw, up to three versions per page, and the best score wins. The author can still pick any version.
The same judge covers the keyframes of Uncen's multi-scene videos, where a bad frame becomes a bad scene. The judge stays warm on an RTX 4090 that our image and video models do not use, next to the SemIf text classifier, at about 16 GB of 24.
What we would tell you
- Do not assume a frontier API is the best judge. Measure false alarms on your own images, not only accuracy.
- Keep deformity separate from prompt adherence. Mixing them causes most false alarms.
- For images, a 12B model with a numbered-option prompt beats every small model we tried.
- On one 24 GB card, batch requests inside one process instead of running copies.
- Label a hundred of your own images. It took us an hour and it changed the decision.
About us
I am Linh, and I build a family of AI products under XP.COM LLC in Colorado. They all run on Uncen, one engine with a job queue, a small GPU fleet, billing, a character bible that keeps characters consistent from scene to scene, and now this image judge. The engine is why a new product takes weeks instead of months.
The products include Brains.biz (illustrated books for Amazon KDP), Uncen (multi-scene AI video), ChatFU (chat that can turn a conversation into a video, with you and your AI companion in the frame), Asset Yard and its alpha Creator app (assets for game developers), DogPoint (dog story videos), Voiceless (free text to speech and an AI phone-call lab), SF Gear (science fiction audio series), and games: Rope Ninja, Humanity: First Year, and Alarm Jerk. The full list is at ai-hacker.com.
AI Duel hosts blind comparisons of complete AI productions, and reports like this one. Write to us and we will share the test scripts and our labels.