How SI-IQ is calculated

Last updated Oct 18, 2026

SI-IQ is a statistical index. It puts AI models on one ability scale using results that independent evaluators have published on public benchmarks, and it expresses that scale like an IQ score: 100 is the average of a fixed reference group of AI models released in 2025, and 15 points is one standard deviation of that group (the group is listed in section 4). It is not an IQ test, it does not test models with IQ questions, and it cannot be compared with human IQ scores.

This page describes the method completely. Everything needed to recompute every published number is in /data/index.json; the worked example in section 5 shows the arithmetic for one model.

1. Which results count

A benchmark is used only if all of the following hold:

1. Independent evaluation. The scores come from an evaluator that is not the model's developer (for example Epoch AI's own evaluation runs). Numbers that developers report about their own models are never used.
2. Republishable data. The evaluator's licence lets us republish the numbers, so that anyone can download all inputs and recompute the index. Results whose terms forbid republishing, or that carry a non-commercial licence, are not used.
3. Enough overlap. At calibration time at least 10 models with at least 4 results each have a result, at least 3 of them from the reference group, and the benchmark is connected to the rest of the data through shared models.
4. It measures ability. Scores must rise with overall ability; a benchmark whose fitted loading is not positive is dropped.

Which score of a model. When an evaluator ran a model in several settings (reasoning effort, thinking budget), we take the best score it published for that model on that benchmark. Higher product tiers sold as separate offerings (for example a "Pro" mode) are not treated as a setting of the base model. For OpenRouter, which runs each model through several hosting providers, we take the providers whose runs covered on average at least 95% of the questions (total questions attempted divided by the number of runs) — only the developer's own hosting if it is among them — and use the median one; with an even number we take the lower of the two middle values, so the number we use is always one provider's result exactly as published.

Which model. Models are identified by Epoch AI's model groups. A model version is used only if it matches its group in product type (a "VL", "Mini" or "Pro" version filed under another group is left out). Names from other sources are matched exactly — after removing explicit run-setting suffixes such as "(high)" — and only when version numbers and product words agree; names we cannot match with certainty wait until they are confirmed and are never ranked meanwhile. If the results we use for a model come from versions released more than 60 days apart, the model is shown but not ranked.

The benchmarks currently in the index, with their parameters:

n: number of questions. ≈ = effective number derived from the evaluator's standard errors; * = assumed (no count published). “Models in fit” = models whose results set the parameters at the last calibration.

Not in the index at the moment: Furniture Assembly (fewer than 3 reference-group models)

GPQA Diamond appears twice because Epoch AI and OpenRouter ran the same 198 questions independently. Results that share a question set form a family and together count as one result (section 5).

2. From a raw score to a common scale

For a score s (fraction correct) on a benchmark with chance level g, effective ceiling c and n questions:

p   = clip((s − g) / (c − g), 0, 1)
p*  = (p·n + 0.5) / (n + 1)
y   = ln(p* / (1 − p*))
s*  = g + (c − g)·p*
v   = s*(1 − s*)·n / ((n + 1)²·(c − g)²·p*²·(1 − p*)²)

p is the chance-corrected score, p* the same with a continuity correction (so it is never exactly 0 or 1), y its logit and v the sampling variance of y. v is the delta-method variance of y for a score that is a binomial proportion over n independent questions: Var(s) ≈ s*(1 − s*)/n, multiplied by the squared derivative of the whole transform, (1/(c − g))·(n/(n + 1))·1/(p*(1 − p*)). When p is clipped (a score below chance or above the ceiling) the derivative inside the range is used.

The logit stretches the top and bottom of the scale, so going from 95% to 99% counts as much more progress than going from 50% to 54%, which matches how hard those gains are. Because v grows near 0% and 100%, nearly saturated benchmarks automatically carry less weight.

What n is. n counts distinct test questions; running the same questions several times does not increase it. The table above says where each n comes from: the benchmark's definition (the number of questions), an effective number derived from the evaluator's published standard errors (n = s(1 − s)/SE², the number of independent questions that would give that standard error), or a stated assumption of 100 where neither exists. An assumption is not a safe bound: if the real number of independent questions is smaller, v and the interval are too small and the benchmark gets too much weight. Scores that average several sub-tests are treated as one binomial score, which is an approximation.

3. The model and its calibration

Each transformed score is modelled as

y = α_b · θ_m + β_b + e,      Var(e) = σ_b² + v

where e is the error, θ_m is the ability of model m, α_b is how strongly benchmark b separates stronger from weaker models (its loading), β_b sets its difficulty, and σ_b² is the variation that one ability does not explain. This is a linearised version of the two-parameter item-response model that Epoch AI's Capabilities Index also uses.

Calibration fits α, β and σ² once a month (first run on or after the 1st, 06:00 UTC) — and also, on any day, when a frozen benchmark is switched to revised values (section 8) — on models with at least four eligible results:

1. Keep only benchmarks and models that meet section 1 and the largest group of models and benchmarks connected through shared results. Being connected is not enough: the shared models must tie every benchmark to one common scale. After fitting, a rank check of the fitted values' derivatives with respect to all θ, α and β must leave exactly the two free directions that the reference group fixes (shifting and stretching all θ together). If a group of benchmarks could be stretched against the rest — for example two groups that share only one model — the benchmark whose removal fixes this (fewest models first) is dropped and the calibration restarts; if no single benchmark fixes it, the calibration is rejected.
2. Start: for each benchmark, z-score its y values (population standard deviation); each model's start θ is the mean of its z-scores; standardise θ on the reference group (mean 0, population standard deviation 1). Start values for every benchmark: α = 1, β = 0, σ² = 0.25.
3. Repeat until no θ changes by more than 10⁻¹⁰ (at most 500 rounds):
   • for each benchmark, a weighted least-squares line of y on θ with weights 1/(σ² + v) gives α and β (α is not allowed below 0.05); σ² = max(Σ residual² / (number of models − 2) − mean(v), 0.10);
   • for each model, θ from the formula in section 5 (without the final map);
   • re-standardise θ on the reference group and fold the shift and scale into α and β.
4. Refit α, β, σ² once more on the final θ. A benchmark whose α sits at 0.05 is dropped and the calibration restarts.
5. Affine map. The section-5 formula includes a small pull λ towards zero, which slightly shrinks the scale. We therefore compute it for every reference model and publish shift (their mean) and scale (their population standard deviation); subtracting shift and dividing by scale puts the reference group at exactly mean 0 and standard deviation 1 at calibration time. Between calibrations the reference models' scores move when they get new results, so the group's mean drifts slightly from 100 until the next calibration.
6. κ (an empirical widening factor for the intervals): models are split into 5 folds: sort the fitted models' slugs by Unicode code point, number them from 0, and fold = number mod 5. For each fold, steps 1–5 are re-run without that fold's models, with the same thresholds; the fold's reference models are the reference-group members it contains (no minimum number), and a fold whose calibration fails is skipped. For each held-out model with at least 4 results on the fold's calibrated benchmarks, its result families are split alternately (code-point order of the family names) into two halves A and B — each half needs at least 2 results — and z = (θ_A − θ_B) / √(SE_A² + SE_B²) is computed with κ = 1 in both standard errors. κ = (the 90th-percentile |z|, nearest rank) / 1.644854, and never below 1. With fewer than 3 successful folds or fewer than 30 pairs κ is undetermined and the calibration is rejected.

The calibration uses the accepted results at the moment it runs. It is rejected — and the previous one stays in force — if it does not converge, produces non-finite values, fails the rank check, or if any model of the frozen reference group (section 4) is missing from the fitted set: the reference group is never silently replaced by the members that happen to be available. A recalibration (monthly, or when a frozen benchmark switches) is also rejected if a benchmark would leave the index, the number of ranked models would change by more than 10%, or scores would move more than expected (rank correlation below 0.95 or median change above 3 points). A benchmark may join in such a recalibration only if it is already configured and was left out before solely because too few models had results. Other structural changes need a new major method version; the last index of each earlier major version stays downloadable under /data/archive/.

Between calibrations the parameters are frozen: a new model or a new score is scored with the frozen parameters, so adding a model never changes anyone else's SI-IQ. Every result is identical when the same inputs are processed again: there are no random numbers anywhere.

4. The reference group (what 100 means)

The reference group is a fixed list of models: those released in 2025 that had enough independent data at the first calibration. It is not every model released in 2025 — it leans towards well-known models that evaluators chose to test. The list only changes with a major method version (at most once a year, with a published conversion table). Current reference group:

Claude 3.7 Sonnet, Claude Haiku 4.5, Claude Opus 4, Claude Opus 4.1, Claude Opus 4.5, Claude Sonnet 4, Claude Sonnet 4.5, DeepSeek-R1, DeepSeek-R1 (May 2025), DeepSeek-V3.2-Exp, DeepSeek-V3 (Mar 2025), Gemini 2.5 Pro (Jun 2025), Gemini 3 Flash, Gemini 3 Pro, Gemma 3 12B, Gemma 3 27B, Gemma 3 4B, GLM-4.7, GPT-4.1, GPT-4.1 mini, GPT-4.1 nano, GPT-5, GPT-5.1, GPT-5.2, GPT-5 mini, GPT-5 nano, gpt-oss-120b, gpt-oss-20b, Grok 3, Grok-3 mini, Grok 4, Llama 4 Maverick, Llama 4 Scout, Mistral Medium 3, o3, o3-mini, o4-mini, Qwen2.5-Max, Qwen3-14B, Qwen3-235B-A22B-Thinking (Jul 2025), Qwen3-30B-A3B, Qwen3-30B-A3B-Instruct (Jul 2025), Qwen3-32B, Qwen3-Max

Human baselines exist for some benchmarks, but they use different populations and cover too few benchmarks to define a common human scale, so SI-IQ makes no claim about human IQ.

5. A model's SI-IQ — the formula you can check

With the published parameters, each model's ability is a closed-form weighted average of its results:

f_b   = 1 / F_b
w_b   = ω_b · f_b / (σ_b² + v_b)
D     = λ + Σ w_b·α_b²
θ₀    = Σ w_b·α_b·(y_b − β_b) / D
θ     = (θ₀ − shift) / scale
SE    = κ · √(Σ w_b²·α_b²·(σ_b² + v_b)) / D / scale
SI-IQ      = 100 + 15·θ
SI-IQ_low  = 100 + 15·(θ − 1.644854·SE)
SI-IQ_high = 100 + 15·(θ + 1.644854·SE)

• F_b is the number of this model's results in the same family as b; the 90% interval runs from SI-IQ_low to SI-IQ_high.
• ω_b = 1/√K, where K is the number of index benchmarks in the same area (knowledge, math, reasoning, coding, agents), so that an area with many benchmarks does not dominate.
• λ = 0.1 gives a tiny pull towards the middle for models with very little data.
• SE is a sandwich standard error: it stays valid when the weights are not plain inverse variances. It treats a model's results as independent apart from the family rule, and it does not include the uncertainty of the calibrated parameters (see section 9).
• Rounding: all comparisons use full precision. Displayed values are rounded to the nearest integer with Python's round() (exact halves go to the even number); interval ends are rounded the same way, separately.

Worked example (all numbers from the current data):

Here is the full calculation for GPT-6 Astra, the highest-ranked model right now.

D     = 0.10 + 24.3255 = 24.4255
θ₀    = 80.5180 / 24.4255 = 3.2965
θ     = (3.2965 − 0.004024) / 0.994832 = 3.3096
SE    = 1.0000 × √14.3103 / 24.4255 / 0.994832 = 0.1557
SI-IQ = 100 + 15 × 3.3096 = 149.64 → 150
90%   = 145.8 … 153.5

6. Ability by area

Area scores (Knowledge, Math, Reasoning, Coding, Agents) use the same formula restricted to that area's benchmarks, with one unit of weight pulling them towards the model's overall unmapped ability θ₀:

θ_d,0   = (Σ_d w·α·(y − β) + 1·θ₀) / (Σ_d w·α² + 1)
θ_d     = (θ_d,0 − shift) / scale
SI-IQ_d = 100 + 15·θ_d

(d = the area; Σ_d runs over the area's results.)

Its standard error and 90% interval use the same sandwich form over the area's results, treating θ₀ as known:

SE_d = κ · √(Σ_d w²·α²·(σ² + v)) / (Σ_d w·α² + 1) / scale

An area score is shown only when the model has results from at least two different families in that area.

7. Who gets a rank

A model is ranked when its results come from at least 5 benchmark families, cover at least 3 areas, come from at least 2 different evaluators, and its versions are not more than 60 days apart. Other models are listed under "too little data" with a provisional score and no rank. Ranked models are ordered by θ (ties by slug). Each model page also shows a rank range: the best rank is 1 + the number of ranked models whose interval lies entirely above this model's interval; the worst is the number of ranked models whose upper end is at least this model's lower end. It describes which intervals overlap; it is not a probability and not a significance test — two models whose rank ranges overlap may still have intervals that do not overlap.

8. Updates and checks

• Every source is checked every hour with conditional requests. OpenRouter is read through its manifest and the immutable dated file it names, verified by SHA-256; files hosted on GitHub are linked at the exact commit. Every distinct benchmark-results file we parsed is archived.
• Each new or changed number is re-read from its source row (with the column headers) by an AI model from a different family than the one that wrote this site's text; it must confirm the model, the benchmark, the number and the model identity. If the checking service is unavailable the whole update stops and the previous version stays online.
• A score that changes by more than 10 points, or a new result far from what a model's other results predict, is held until a later, different source snapshot still shows it (at least six hours later), or the other GPQA Diamond run confirms it; waiting alone never releases it. A model that was not ranked before (new, or listed with too little data) and would now score more than 5 points above the current leader keeps its earlier published state (nothing, for a new model) and its new or changed results are published only when every one of them is confirmed — at least six hours after it first appeared, its source has published a newer snapshot that still lists it with the same value — and the model's score is still the same within 3 points. Another source's refresh does not count.
• If a source changes format or suddenly loses more than 30% of its rows — or of one benchmark's rows — its last good data is kept, the failure is shown on /status.json, and nothing from it is updated. If it changes the definition of one benchmark (score column, scale, chance level, ceiling, number of questions), only that benchmark is frozen (see below) until the change is checked by a person; the source's other benchmarks keep updating. A reduced file is accepted as the new normal only if our parser still keeps the same share of the file's rows as before (so the rows were removed upstream, not lost by us) and every file that parsed during the following 24 hours was just as small; a fall in that share is never accepted automatically. A result that ages out of OpenRouter's 90-day window, or of an archived source, keeps its last value; a result removed by Epoch AI is removed here after six hours. When a source name is re-matched to a different model, the old match is removed at once — including from archived sources and from history curves.
• Benchmark revisions. Re-scoring many existing results at once usually means the benchmark itself changed (new questions, a new scorer). When at least 5 existing results of one benchmark, and at least a quarter of them, change by more than half a point in one source snapshot, the benchmark is frozen: its published values stay as they were and no new or changed value is used. The benchmark is switched to the new values in one step, together with a recalibration, as soon as every changed value has passed the cross-family check and a later source snapshot, at least six hours after it first appeared, still shows it — and only if that recalibration passes all its checks; otherwise it stays frozen (a failed attempt is retried after a day). An existing result that the source re-scores in a later snapshot (by more than half a point) becomes part of the same revision and must be confirmed as well, so a re-scoring that arrives in several snapshots is switched in one step; smaller later changes keep their published value and are checked as usual after the switch. Models the source added since the freeze do not hold the switch up; their results are checked as usual afterwards. A changed value the check finally rejects is left out (that model goes without this benchmark). Unchanged values stay; values that aged out of a source that keeps last values stay too. For an archived source, which never publishes a newer snapshot, a person must approve the switch. Frozen benchmarks are listed in /status.json.
• Held values that are never confirmed. A held value that no newer snapshot confirms within 7 days (archived sources never publish one) stays held until a later snapshot confirms it or a person releases it after checking it at the source, and it is reported to the operator.
• Before publishing, the site is checked automatically: every page parses, internal links and language alternates resolve, and every SI-IQ in /data/index.json is recomputed from its inputs. If ranked models disappear or many scores move without a recalibration, publishing stops. It resumes automatically only if every affected model's inputs changed (accepted, checked source changes) and the state persists for six hourly runs; a change in a model whose inputs did not change is never published automatically.
• Every input row is published as it appeared in the source, with the source file's address and SHA-256, in /data/evidence.jsonl, so readers can check each number even after the source has changed.

9. Known limitations

• One number hides differences. Models increasingly specialise; the area scores and the per-benchmark table on each model page show where a model is stronger or weaker.
• Coverage is uneven. Few republishable, independent benchmarks exist for coding and agent tasks, and some are no longer updated, so those area scores are missing for most recent models. There is no multimodal area yet.
• Benchmarks age and leak. Older benchmarks saturate or may have appeared in training data. The logit transform reduces their influence near the ceiling but cannot remove contamination; private question sets (such as FrontierMath) are less exposed.
• Different evaluators, different setups. The same questions run with different prompts, tools or effort settings can give different scores; we keep each evaluator's runs as a separate benchmark.
• "Best published setting" favours models tested in many settings, because the best of several runs tends to be higher.
• Developer claims are excluded, even though they often appear first; developer-reported numbers tend to be favourable and are not comparable across companies.
• The scale is relative. SI-IQ compares AI models with each other. A score of 140 does not mean a model is as capable as a person with an IQ of 140.
• The intervals are conditional. They assume the model in section 3 is right. They leave out the uncertainty of the calibrated parameters, correlations between a model's results beyond the family rule, and the effect of which models evaluators chose to test; κ widens them by how much independent halves of the data actually disagree, but there is no guarantee that 90% intervals cover the true value 90% of the time.
• Assumed question counts. Where n is an assumption or an effective number (section 2), the weight and the interval of that benchmark are approximate.

10. Data sources and licences

Required attributions:

• Aider polyglot leaderboard data (Apache-2.0) — https://aider.chat/docs/leaderboards/
• Arena leaderboard dataset on Hugging Face (CC BY 4.0) — https://huggingface.co/datasets/lmarena-ai/leaderboard-dataset
• Berkeley Function Calling Leaderboard, Gorilla project (Apache-2.0) — https://gorilla.cs.berkeley.edu/leaderboard.html
• Epoch AI, ‘Capabilities & benchmarking’ (CC BY 4.0) — https://epoch.ai/benchmarks
• API list prices via LiteLLM's price table (MIT); each price links to the developer's pricing page — https://github.com/BerriAI/litellm/blob/main/model_prices_and_context_window.json
• Source: OpenRouter (openrouter.ai), licensed under CC BY 4.0 — https://openrouter.ai/api/v1/datasets/exports/README.md

11. Versions

The method version is shown on every model page and in every data file. Patch versions fix data or parsing errors without changing parameters; minor versions are recalibrations (monthly, or when a frozen benchmark switches to revised values); major versions change the transform, the model, the areas, the benchmark set or the reference group, and come with recomputed history. Model-page history curves always re-score each day's archived inputs with the current parameters, so they never mix methods; those inputs are published in /data/history.json, so every point can be recomputed with the formula of section 5.

• 1.0.0 (October 18, 2026): first version.