How we test

One job at a time

We don't publish general rankings. A model that is best for realistic people can be poor at Chinese text on a poster, so every result belongs to one job, such as “posters with Chinese text” or “PPT decks”. We pick jobs from what people actually search for and struggle with, and we start with the ones where a wrong choice costs the most time or money.

How a test round works

1. Fixed briefs. Each job has a set of briefs (10 to 20 for video, 10 to 15 for images, 3 to 10 for writing and office work). Every model gets the same brief, the same input files and the closest equivalent settings.
2. Blind grading. Outputs are shuffled and labelled A, B, C. A panel of four graders scores them against the job's rubric without knowing which model made which. Each grader sees the outputs in a different order.
3. Hard checks where possible. Chinese text is checked character by character. Citations are opened and verified one by one. Résumés go through an applicant-tracking-style parse check. Websites are deployed and edited.
4. Cost per usable result. We record every attempt, including retries, so we can report what a usable result actually costs, not just the list price.

The Reference Kit

Image-to-video and editing tests start from the same pixels for every model. Without a shared reference, a test mostly measures the prompt, not the model. The kit has:

• 6 synthetic characters (no real person's likeness), each with front, side and back views and 4 expressions, across ages and ethnicities, including East Asian faces.
• 4 products: a bottle, a phone, a shoe and packaged food, each on a clean white background.
• 3 locations: a café interior, a city street at night and a living room from an architectural rendering.
• 2 art styles: anime and 3D-cartoon versions of two characters, for comic-drama tests.
• Start frames that place kit characters in kit locations, for image-to-video tests. First and last frame pairs come next.

Every image is synthetic and free to download, so anyone can reproduce a result. Made on 31 Oct 2026.

The BiBi Score

Each job page gives every model a BiBi Score from 0 to 100, built from four parts that are always shown separately too:

• Looks (quality), 50%: the blind panel's average on this job's rubric, adjusted for each grader's habits (below), times 10.
• Reliability, 20%: our own checks against the brief: every word right, product unchanged, actions done in order. A pass counts 1, a small slip ½, a failure 0.
• Value, 20%: cost per usable result. The cheapest model gets 100 and every doubling of cost loses 30 points, so a model four times the price scores 40.
• Ease, 10%: the steps, settings and limits a non-expert runs into. While every model in a round runs on the same platform there is nothing to separate them here, so the other three parts are rescaled to 100.

The weights are published and versioned (this is BiBi Score v1.0). They never change for a sponsor. A model that can't be used in a region is labelled, not scored down.

Keeping personal taste out of the score

Whether a poster or a video looks good is partly a matter of taste, and four people will never fully agree. We don't pretend otherwise. Instead we limit how much taste can move a result, and we tell you how much it did:

• Taste is at most half the score. The other half is things anyone can check: wrong characters, a changed product, a missed action, the price.
• Strict and generous graders count the same. In round 1 our most generous grader averaged 7.2 out of 10 and our strictest 5.0. Before averaging, each grader's scores are converted to a shared scale, so what counts is how they ranked the outputs, not how high they scored.
• We measure agreement. For every job we check whether the graders put the models in the same order (rank correlation, plus within-brief agreement and ICC). Each results page says whether agreement was high, medium or low. When it is low, as it was for product photos in round 1, we call looks a tie and let the checkable parts decide.
• We show a range, not just a number. We resample graders and briefs 2,000 times. If a model's range overlaps the leader's, we label it tied with the best rather than claim a winner.
• Machines check what they can. Chinese text is checked character by character, and identity in video is checked with a face-recognition model that compares every clip with the start frame.

What this can't fix: our panel is small and works in film, design and software, so it reflects people who make content for a living. If you have a strong visual preference, look at the outputs; every one is on the page, unedited.

Prompt packs and test logs

With each graded round we publish the full prompt pack and a test log with every output, so you can check our work. Test logs are snapshots and keep their date; job pages are updated in place when we re-test.

Health, study and money topics

• Health pages are reviewed by a named, licensed doctor before they are published, use only de-identified sample reports, and are not medical advice.
• Study and research tests are about finding real papers, checking citations and summarising. We report each tool's fake-citation rate and don't publish “write my paper” or “avoid AI detection” content.
• Sponsors never see scores before publication and can't change them. Losing results are published too.

Corrections

When we get something wrong we fix it and note the change and the date on the page. Spotted a mistake? Tell us.