Evaluate AI image-model outputs without cherry-picking

Define the comparison before looking at outputs

Evaluate AI image-model outputs against a named production task, not a vague idea of which image looks best. Write the intended placement, required subject, composition, prompt facts, forbidden defects, and editing tolerance before generation. NIST notes that evaluation changes with operating context and that different characteristics need different measurements. A model can therefore be useful for one brief without being a universal winner.

Turn that brief into separate criteria: prompt adherence, composition, anatomy or object integrity, exact-text handling, visual artifacts, placement fit, and the amount of acceptable post-production. Give each criterion a pass condition or an anchored scale before reviewers see model names. Do not collapse the result into one beauty score; a visually striking candidate can still fail the required product count, crop, or wording.

Unit of comparison

One task, one prompt record, one ratio, and the same number of runs per modelRecord any setting that cannot be held equal and treat it as a limitation rather than silently normalizing it.

Decision rule

Per-criterion pass, fail, or tie before an overall choiceA pairwise preference identifies the preferred candidate; it does not prove that either candidate clears a release threshold.

Build matched prompts that expose the real job

Use a small prompt suite rather than one showcase prompt. Include an ordinary brief, a composition constraint, a counting or spatial-relation case, a difficult material or lighting case, and a known failure risk such as small exact text. The Gecko evaluation research found that conclusions from one capability or annotation slice were not stable enough to generalize; its suite instead spans curated prompts, multiple annotation templates, and pairwise and pointwise tasks.

Submit the same authored prompt, aspect ratio, and run count to each model. Also capture the effective prompt when a service rewrites it. Google's current Imagen documentation says prompt enhancement can be enabled by default and that an enhanced prompt can be returned with each prediction. If one model receives rewritten text and another receives the literal prompt, disclose that difference; the outputs are not strictly matched even when the visible input box was identical.

Randomness makes a single output weak evidence. Diffusers documents that diffusion starts from random noise, that repeated use changes generator state, and that identical seeds still do not guarantee identical results across platforms. Run each prompt more than once, retain every completed candidate in generation order, and never replace an inconvenient result with an unreported retry.

Prompt record

Authored text, effective text, ratio, model identifier, run index, and timestampAdd seed or sampler only when the actual interface exposes it; otherwise write “not exposed” instead of inventing parity.

Balanced batch

Equal attempts for every model and promptKeep failed or blocked calls in the log, but compare visual candidates separately from service reliability observations.

Specify a complete contact sheet before any comparison

Lay out every retained run in a fixed grid: prompts as rows, anonymized models as columns, and run numbers in the same order within each cell. Preserve the original files and inspect both the full frame and consistent detail crops. Randomize the model-column order for each reviewer, hide filenames and provider labels, and collect criterion scores before revealing identity. OpenAI's evaluation guidance recommends clear, detailed rubrics and favors pairwise comparison or pass/fail judgments over open-ended impressions; it also says automated judges should be calibrated against human labels.

Treat this block as a prospective specification, not an output exhibit. A publishable comparison would need at least two current image models, the same prompt suite and ratio, equal run counts, unedited outputs, anonymized review records, and a disclosed selection rule. Without that complete evidence set, the methodology cannot support a Studio model ranking, win rate, representative-output claim, or comparative quality conclusion.

Prospective evidence set

Complete anonymized grid plus original generation recordsRequire matched candidates and review records before drawing an empirical conclusion.

Publication gate

Reveal model identity only after criterion scores are lockedPublish ties, rejects, failed calls, and the denominator alongside any preferred model.

Record what Studio exposes and what it does not

The published OfflineCreator MCP 0.1.2 `generate` tool accepts a current model ID, creative prompt, optional `16:9`, `9:16`, or `1:1` aspect ratio, and a wait flag. The same public surface appears in 0.1.1. It does not expose a seed, sampler, inference-step count, guidance value, or prompt-rewrite switch. `list_models` supplies the current launch catalog, while completed jobs can be retrieved through generation-status and download tools. These facts describe the public interface, not the hidden provider configuration.

For a Studio comparison, record the returned model identifier, generation ID, submitted prompt, ratio, start and completion times, output file, and any model metadata returned by the live catalog. Mark unavailable controls explicitly. Do not claim seed-matched testing when the interface does not expose a seed, and do not infer that two differently routed models share a sampler or rewriting policy. Refresh the public schema and live catalog before the test because this page follows a monthly freshness cadence.

Currently controllable in public MCP

Model ID, prompt, aspect ratio, and waiting behaviorThe comparison record should preserve exactly what was submitted.

Not exposed in public MCP 0.1.2

Seed and low-level diffusion controlsAbsence from this public schema is not proof about an underlying provider's capabilities.

Choose the next action from the evaluation gap

Use the model directory when the candidate set is still undefined. Open the budgeting guide before selecting run count so every model receives the same approved number of attempts. Move to the FLUX Pro guide only after the blind record identifies a task-specific reason to inspect that model more closely. These routes preserve the distinction between catalog facts, spend planning, and observed output evidence.

Consolidate the methodology until required evidence exists

This researched draft provides methodology only: matched task prompts, repeated runs, blind criteria, complete candidate disclosure, and uncertainty reporting. It makes no empirical ranking, output, quality, speed, reliability, editing-effort, creator-preference, pricing-value, or customer-outcome claim.

Real Studio outputs and a completed review record are required evidence for this standalone page but are unavailable. Consolidate this methodology into the parent model hub at `/models/mcp-model-guides` and keep this route unpublished. Reconsider a standalone evaluation page only when attributable, matched Studio outputs and the documented blind-review method can be maintained together.