AI Model Selection and Benchmark Scores
“Which model is better?” needs a workload, a quality bar, and a deployment constraint. A model that wins a general index may lose on your SwiftUI build task, run too slowly on your hardware, or be unsuitable for confidential data. Choose the smallest permitted model that passes your representative tasks; keep a stronger option for cases it demonstrably improves. See model routing and token budgeting.
Use this order when making a decision:
- Policy gate: eliminate models and providers that cannot meet data residency, retention, licensing, or network-isolation requirements.
- Capability gate: run the same representative tasks, tools, time limit, and acceptance checks for every candidate.
- Efficiency gate: among the candidates that pass, compare total completion cost, review effort, latency, and operational burden.
A cheaper model that needs repeated retries may cost more per accepted result. A stronger model used on every easy request may waste time and money.
Dated benchmark snapshot
The following are Artificial Analysis Intelligence Index v4.3.2 points, not percentages, from its September 2026 report, checked on 12 October 2026. The publisher runs this independent composite of agent, coding, science, and general evaluations. Configurations in parentheses matter; “with fallback” is a workflow configuration, not the bare model. These figures are a snapshot, not a universal ranking or a local-machine result. Scores and cost-per-task report · methodology and version history.
| Evaluated configuration | Index points | What the result tells you | Do not infer |
|---|---|---|---|
| GPT-6 Astra (max) | 53 | Tied for the highest reported composite score in this snapshot | That max effort is the best cost-quality setting for every task |
| Claude Fable 5.1 (max, with fallback) | 53 | Same composite score with an explicitly different workflow setup | That the bare model, without fallback, produced the same result |
| Claude Opus 5 (max) | 51 | Close on the composite; task-level outcomes may differ | That it remains the current default in the Opus family |
| GLM-5.3 Flash | 42 | A strong reported open-weights result | That “Flash” means it fits an ordinary developer laptop |
| Qwen3.8 2.4T A95B | 40 | A competitive open-weights result from a very large sparse model | That 95B active parameters means only 95B parameters must be stored |
On the same index, the report estimates $3.26 per task for GPT-6 Astra (max) and $7.63 for Claude Fable 5.1 (max, with fallback). Those are benchmark-task cost estimates under the publisher's pricing and token assumptions, not a quote for your workload. The index changed its task mix and scoring over time, so scores from older index versions should not be compared as if they used one test. Artificial Analysis report.
Model-by-model decision guide
The benchmark table and the model guide answer different questions. The first records one comparable snapshot. This table turns every model named in this lesson into a starting hypothesis that must still be tested on your workload.
| Model or configuration | Practical role | Start here when | Escalate, replace, or reject when |
|---|---|---|---|
| GPT-6 Astra | Frontier cloud model for the hardest reasoning, coding, research, and multi-tool work | The task is ambiguous, expensive to review, or has a high cost of failure | A lower-cost candidate passes the same checks, latency is unacceptable, or cloud use is not permitted |
| GPT-6.1 Sol | Balanced cloud default for complex professional and coding work | You need strong quality repeatedly and care about cost or turnaround | Astra materially improves acceptance, or Luna already meets the quality bar |
| GPT-6 Luna | Efficient cloud model for scoped, frequent, and high-volume work | Inputs, outputs, and success conditions are narrow and machine-checkable | The workflow needs long-horizon judgment, keeps retrying tools, or creates too much reviewer rework |
| Claude Fable 5.1 | Frontier model for difficult coding, knowledge work, and long-running problem solving | Your private evaluation shows better completion or artifact quality than the alternatives | The fallback policy changes the trust boundary, the workflow needs a lower-cost tier, or provider policy is incompatible |
| Claude Opus 5 | A dated benchmark participant, not an automatic greenfield default | Reproducing the September 2026 comparison or maintaining a validated existing deployment | Starting a new deployment without checking the current Opus catalog and rerunning the evaluation |
| GLM-5.3 Flash | Open-weights, multimodal sparse model aimed at efficient serving | You have compatible inference infrastructure and want control over weights, data path, or tuning | “Local” means a normal laptop, the serving stack is unsupported, or hosted operation is cheaper after engineering costs |
| Qwen3.8 2.4T A95B | Very large open-weights text model for coding, research, and long-horizon agents | You can use managed inference or serious multi-accelerator infrastructure and need its specific quality/control tradeoff | You need multimodal input, non-thinking mode, small-machine deployment, or a simpler license and operations profile |
The OpenAI roles above follow official model-selection guidance. Anthropic describes Fable 5.1 as a model for advanced coding and knowledge work; its benchmark fallback is part of the evaluated system, not a footnote to ignore. Claude Opus 5 remains in the snapshot because that is what was measured, but the current Opus catalog should be checked before a new adoption.
Z.ai's official GLM-5.3 Flash model card lists 320B total and 18B active parameters, multimodal input, adjustable reasoning effort, and an MIT license. Qwen's official Qwen3.8 2.4T A95B model card lists 2.4T total and 95B active parameters, text-only input, mandatory thinking mode, and its own model license. Verify current model IDs, prices, limits, licenses, and regional availability at adoption time.
Open weights does not mean laptop-sized
For a mixture-of-experts model, active parameters estimate how much of the model participates in a token's computation. They do not describe the full weight set that must be stored or distributed across the serving system.
At an idealized 4 bits per parameter, weights alone have a theoretical floor of roughly 160 GB for 320B parameters and 1.2 TB for 2.4T parameters. Real deployments also need quantization metadata, runtime buffers, key-value cache, framework overhead, and headroom. This is why GLM-5.3 Flash and Qwen3.8 2.4T A95B can be open weights without being sensible defaults for a MacBook or a single consumer GPU.
Before calling either model “local,” record the exact checkpoint and quantization, supported runtime, accelerator count, usable context, prompt-processing speed, generation speed, peak memory, license obligations, and quality loss versus the reference configuration. If the requirement is simply “data must not leave this laptop,” evaluate a genuinely smaller permitted model instead of assuming the benchmarked open model fits.
Better for which job?
| Job | Starting candidate | Verify before adoption |
|---|---|---|
| Ambiguous architecture, difficult coding, or high-stakes review | GPT-6 Astra or Claude Fable 5.1 | Correctness, review effort, tool boundaries, fallback behavior, cost per accepted result |
| Daily coding with quality and cost pressure | GPT-6.1 Sol | Same-task acceptance against a stronger model and your current baseline |
| Narrow extraction, classification, or high-volume triage | GPT-6 Luna or another efficient model | False positives, schema validity, retry rate, latency, total cost |
| Controlled open-weights serving with multimodal input | GLM-5.3 Flash | License, supported runtime, accelerator memory, image quality, and end-to-end throughput |
| Large text-only open-weights agent or research workflow | Qwen3.8 2.4T A95B | License, mandatory thinking behavior, infrastructure cost, and task-level quality |
| Sensitive documents with a true offline requirement | A smaller locally runnable model that fits the actual hardware | Network isolation, tool success, memory pressure, and acceptable quality |
Other vendors or newer releases may be a better fit on a particular task. Model availability, prices, product features, and safety behavior vary by account and surface; check the current vendor page at decision time.
Route by evidence, not brand
A practical production route normally has at least three outcomes:
- Reject or redact: the request violates the data policy for every available route.
- Default: use the least expensive model and reasoning setting that passes the task's checks.
- Escalate: move only failed, ambiguous, or high-impact cases to the stronger permitted route.
Define escalation using observable signals: invalid schema after one repair, failing tests, low retrieval coverage, an unverified citation, a risky file change, or a reviewer-required decision. Do not route on the model's self-reported confidence alone. Log the selected route, policy version, model identifier, reasoning setting, tool permissions, outcome, and fallback without storing sensitive prompt content unnecessarily.
Why a benchmark score can mislead
SWE-bench Verified contains 500 human-reviewed software issues, but its full leaderboard compares systems, not just models. The benchmark maintainers provide a bash-only mini-SWE-agent view to make model comparison easier and warn that results from major agent releases are not necessarily comparable. SWE-bench Verified methodology. For any coding-agent claim, record the model version, agent harness, tool set, context limit, reasoning setting, retries, test split, and date. A percentage without these details does not establish which model will fix your repository.
The same warning applies inside one model family. Higher reasoning effort can improve a difficult task while increasing latency and token use; a fallback can improve completion while changing which model handled part of the request. Compare the full configuration, not only the family name.
Run a useful local evaluation
Create a small private task set from work you are authorized to use. Include a Swift compile repair, a UI state change, a tool-call decision, a refusal to read an out-of-scope file, and a cancellation case. Run each candidate under the same host, tools, time limit, and acceptance rubric. Keep the input and gold checks local when the data is private.
| Measure | Record |
|---|---|
| Task success | Passed acceptance checks / total tasks, with failed cases named |
| Quality | Correct behavior, maintainability, citation quality, and reviewer rework |
| Tool reliability | Valid calls, denied calls, retries, fallback events, and unsupported-tool failures |
| Speed | Median, p95, and slowest meaningful end-to-end time |
| Cloud cost | Input, cached-input, reasoning, and output tokens; tool or search charges; cost per accepted task |
| Local resources | Checkpoint and quantization, peak memory, swap, prompt throughput, generation throughput, and energy if relevant |
| Privacy and policy | Outbound destinations, retention mode, blocked reads, log content, residency, license, and violations |
Use at least a few repeats for nondeterministic tasks and keep a held-out set for later changes. Grade the final artifact and trace, not only the model's confidence. Report pass rate with the task count; a 90% result on 10 tasks is not as stable as 90% on 1,000. OpenAI's agent evaluation guide describes traces, graders, datasets, and repeatable runs.
Use a decision record rather than a leaderboard screenshot:
| Decision field | Example of the required precision |
|---|---|
| Workload | 24 representative Swift pull-request tasks, including 6 concurrency bugs |
| Acceptance | Builds, focused tests pass, no out-of-scope files changed, reviewer approves |
| Configuration | Exact model ID, reasoning effort, tools, system prompt version, timeout, retries |
| Result | 21/24 accepted; three failed task IDs and failure categories |
| Efficiency | Median and p95 time, tokens, provider charges, reviewer minutes |
| Policy | Data class allowed, region/retention checked, fallback permitted or disabled |
| Decision | Default route, escalation route, owner, review date, rollback trigger |
Practice: choose a model
Prompt: A small local model finishes a SwiftUI edit in 25 seconds but passes 6 of 10 acceptance tests. A cloud model takes 80 seconds and passes 9 of 10, but the task includes private client data. What is the next decision?
Answer
First enforce the data policy. If cloud processing is not authorized, the cloud result is not deployable regardless of score. Do not assume GLM-5.3 Flash or Qwen3.8 2.4T A95B will solve the local requirement merely because their weights are available; check whether the actual hardware can run the chosen checkpoint at usable speed and context.
Improve the permitted local workflow or test a smaller permitted local model, then classify the four failures as model, context, tool, or test problems. If cloud use becomes permitted through a reviewed redaction and approval flow, compare cost per accepted task, total completion time, reviewer effort, and policy compliance on the same held-out tasks.
Quick Recap
- A benchmark point is evidence about a specified test configuration, not a verdict for all work.
- A model, reasoning level, agent harness, tool set, and fallback policy form one evaluated system.
- Open weights and active-parameter counts do not prove that a model fits local hardware.
- Record date, exact identifiers, harness, settings, retries, fallback, and cost assumptions beside scores.
- Compare candidates on your own acceptance checks and privacy constraints.
Key Takeaways
Pick models by measured task success under the same workflow. Treat public leaderboards as a shortlist generator, then validate quality, latency, resources, license, fallback behavior, cost, and data policy in your environment. Route routine work to the smallest passing configuration and escalate only when evidence says the stronger permitted route changes the outcome.