system evals
Model Performance Cards
Claude Opus 4.5
Basic tasks with clear instructions, single-step operations, and well-documented solutions. Examples: installing packages, basic configs, simple queries.
Multi-step tasks requiring contextual understanding, debugging common issues, and integrating multiple components. Examples: setting up CI/CD, container orchestration, performance tuning.
Complex scenarios with ambiguous requirements, novel problem-solving, security hardening, and architectural decisions with trade-offs. Examples: zero-day mitigation, distributed system design, disaster recovery.
Basic tasks with clear instructions, single-step operations, and well-documented solutions. Examples: installing packages, basic configs, simple queries.
Multi-step tasks requiring contextual understanding, debugging common issues, and integrating multiple components. Examples: setting up CI/CD, container orchestration, performance tuning.
Complex scenarios with ambiguous requirements, novel problem-solving, security hardening, and architectural decisions with trade-offs. Examples: zero-day mitigation, distributed system design, disaster recovery.
Basic tasks with clear instructions, single-step operations, and well-documented solutions. Examples: installing packages, basic configs, simple queries.
Multi-step tasks requiring contextual understanding, debugging common issues, and integrating multiple components. Examples: setting up CI/CD, container orchestration, performance tuning.
Complex scenarios with ambiguous requirements, novel problem-solving, security hardening, and architectural decisions with trade-offs. Examples: zero-day mitigation, distributed system design, disaster recovery.
GPT-5.2 High
Basic tasks with clear instructions, single-step operations, and well-documented solutions. Examples: installing packages, basic configs, simple queries.
Multi-step tasks requiring contextual understanding, debugging common issues, and integrating multiple components. Examples: setting up CI/CD, container orchestration, performance tuning.
Complex scenarios with ambiguous requirements, novel problem-solving, security hardening, and architectural decisions with trade-offs. Examples: zero-day mitigation, distributed system design, disaster recovery.
Basic tasks with clear instructions, single-step operations, and well-documented solutions. Examples: installing packages, basic configs, simple queries.
Multi-step tasks requiring contextual understanding, debugging common issues, and integrating multiple components. Examples: setting up CI/CD, container orchestration, performance tuning.
Complex scenarios with ambiguous requirements, novel problem-solving, security hardening, and architectural decisions with trade-offs. Examples: zero-day mitigation, distributed system design, disaster recovery.
Basic tasks with clear instructions, single-step operations, and well-documented solutions. Examples: installing packages, basic configs, simple queries.
Multi-step tasks requiring contextual understanding, debugging common issues, and integrating multiple components. Examples: setting up CI/CD, container orchestration, performance tuning.
Complex scenarios with ambiguous requirements, novel problem-solving, security hardening, and architectural decisions with trade-offs. Examples: zero-day mitigation, distributed system design, disaster recovery.
Gemini 3 Pro
Basic tasks with clear instructions, single-step operations, and well-documented solutions. Examples: installing packages, basic configs, simple queries.
Multi-step tasks requiring contextual understanding, debugging common issues, and integrating multiple components. Examples: setting up CI/CD, container orchestration, performance tuning.
Complex scenarios with ambiguous requirements, novel problem-solving, security hardening, and architectural decisions with trade-offs. Examples: zero-day mitigation, distributed system design, disaster recovery.
Basic tasks with clear instructions, single-step operations, and well-documented solutions. Examples: installing packages, basic configs, simple queries.
Multi-step tasks requiring contextual understanding, debugging common issues, and integrating multiple components. Examples: setting up CI/CD, container orchestration, performance tuning.
Complex scenarios with ambiguous requirements, novel problem-solving, security hardening, and architectural decisions with trade-offs. Examples: zero-day mitigation, distributed system design, disaster recovery.
Basic tasks with clear instructions, single-step operations, and well-documented solutions. Examples: installing packages, basic configs, simple queries.
Multi-step tasks requiring contextual understanding, debugging common issues, and integrating multiple components. Examples: setting up CI/CD, container orchestration, performance tuning.
Complex scenarios with ambiguous requirements, novel problem-solving, security hardening, and architectural decisions with trade-offs. Examples: zero-day mitigation, distributed system design, disaster recovery.
Claude Sonnet 4.5
Basic tasks with clear instructions, single-step operations, and well-documented solutions. Examples: installing packages, basic configs, simple queries.
Multi-step tasks requiring contextual understanding, debugging common issues, and integrating multiple components. Examples: setting up CI/CD, container orchestration, performance tuning.
Complex scenarios with ambiguous requirements, novel problem-solving, security hardening, and architectural decisions with trade-offs. Examples: zero-day mitigation, distributed system design, disaster recovery.
Basic tasks with clear instructions, single-step operations, and well-documented solutions. Examples: installing packages, basic configs, simple queries.
Multi-step tasks requiring contextual understanding, debugging common issues, and integrating multiple components. Examples: setting up CI/CD, container orchestration, performance tuning.
Complex scenarios with ambiguous requirements, novel problem-solving, security hardening, and architectural decisions with trade-offs. Examples: zero-day mitigation, distributed system design, disaster recovery.
Basic tasks with clear instructions, single-step operations, and well-documented solutions. Examples: installing packages, basic configs, simple queries.
Multi-step tasks requiring contextual understanding, debugging common issues, and integrating multiple components. Examples: setting up CI/CD, container orchestration, performance tuning.
Complex scenarios with ambiguous requirements, novel problem-solving, security hardening, and architectural decisions with trade-offs. Examples: zero-day mitigation, distributed system design, disaster recovery.
GPT-5.1 Mini
Basic tasks with clear instructions, single-step operations, and well-documented solutions. Examples: installing packages, basic configs, simple queries.
Multi-step tasks requiring contextual understanding, debugging common issues, and integrating multiple components. Examples: setting up CI/CD, container orchestration, performance tuning.
Complex scenarios with ambiguous requirements, novel problem-solving, security hardening, and architectural decisions with trade-offs. Examples: zero-day mitigation, distributed system design, disaster recovery.
Basic tasks with clear instructions, single-step operations, and well-documented solutions. Examples: installing packages, basic configs, simple queries.
Multi-step tasks requiring contextual understanding, debugging common issues, and integrating multiple components. Examples: setting up CI/CD, container orchestration, performance tuning.
Complex scenarios with ambiguous requirements, novel problem-solving, security hardening, and architectural decisions with trade-offs. Examples: zero-day mitigation, distributed system design, disaster recovery.
Basic tasks with clear instructions, single-step operations, and well-documented solutions. Examples: installing packages, basic configs, simple queries.
Multi-step tasks requiring contextual understanding, debugging common issues, and integrating multiple components. Examples: setting up CI/CD, container orchestration, performance tuning.
Complex scenarios with ambiguous requirements, novel problem-solving, security hardening, and architectural decisions with trade-offs. Examples: zero-day mitigation, distributed system design, disaster recovery.
Gemini Flash
Basic tasks with clear instructions, single-step operations, and well-documented solutions. Examples: installing packages, basic configs, simple queries.
Multi-step tasks requiring contextual understanding, debugging common issues, and integrating multiple components. Examples: setting up CI/CD, container orchestration, performance tuning.
Complex scenarios with ambiguous requirements, novel problem-solving, security hardening, and architectural decisions with trade-offs. Examples: zero-day mitigation, distributed system design, disaster recovery.
Basic tasks with clear instructions, single-step operations, and well-documented solutions. Examples: installing packages, basic configs, simple queries.
Multi-step tasks requiring contextual understanding, debugging common issues, and integrating multiple components. Examples: setting up CI/CD, container orchestration, performance tuning.
Complex scenarios with ambiguous requirements, novel problem-solving, security hardening, and architectural decisions with trade-offs. Examples: zero-day mitigation, distributed system design, disaster recovery.
Basic tasks with clear instructions, single-step operations, and well-documented solutions. Examples: installing packages, basic configs, simple queries.
Multi-step tasks requiring contextual understanding, debugging common issues, and integrating multiple components. Examples: setting up CI/CD, container orchestration, performance tuning.
Complex scenarios with ambiguous requirements, novel problem-solving, security hardening, and architectural decisions with trade-offs. Examples: zero-day mitigation, distributed system design, disaster recovery.
Complete Scores Table
| Model | Overall | Solution Architecting | System Administration | SRE | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Avg | Beg | Int | Exp | Avg | Beg | Int | Exp | Avg | Beg | Int | Exp | ||
| Claude Opus 4.5 | 91 | 82 | 92 | 65 | 32 | 96 | 98 | 67 | 35 | 85 | 94 | 62 | 28 |
| Claude Sonnet 4.5 | 82 | 78 | 88 | 58 | 24 | 88 | 94 | 64 | 29 | 80 | 90 | 56 | 22 |
| GPT-5.2 High | 89 | 97 | 96 | 67 | 35 | 79 | 86 | 54 | 21 | 81 | 88 | 58 | 25 |
| GPT-5.1 Mini | 68 | 72 | 82 | 48 | 12 | 62 | 74 | 40 | 8 | 66 | 78 | 44 | 10 |
| Gemini 3 Pro | 87 | 80 | 88 | 56 | 26 | 77 | 84 | 52 | 22 | 95 | 96 | 67 | 34 |
| Gemini Flash | 64 | 60 | 72 | 38 | 6 | 58 | 70 | 36 | 5 | 70 | 80 | 46 | 14 |
Model Recommendations
Detailed analysis of each model's strengths, weaknesses, and ideal use cases
Claude Opus 4.5
Excels at complex system administration tasks including configuration management, security hardening, and infrastructure automation. Strong reasoning capabilities for multi-step operational procedures.
Strengths
- Deep system knowledge
- Security-first approach
- Excellent at automation scripts
- Strong documentation
Weaknesses
- Can be overly cautious
- Slower on simple tasks
- Higher token cost
Claude Sonnet 4.5
A well-rounded model offering strong performance across all categories with faster response times. Ideal for day-to-day operations and iterative development workflows.
Strengths
- Fast responses
- Cost-effective
- Good all-rounder
- Reliable outputs
Weaknesses
- Less depth than Opus
- May miss edge cases
- Simpler explanations
GPT-5.2 High
Industry-leading architecture capabilities with exceptional ability to design scalable, maintainable systems. Excels at understanding complex requirements and proposing elegant solutions.
Strengths
- Superior architecture design
- Scalability planning
- Pattern recognition
- Clear diagrams
Weaknesses
- Weaker on ops details
- Can over-engineer
- Sometimes verbose
GPT-5.1 Mini
Lightweight model optimized for speed and cost efficiency. Best suited for simple queries, quick lookups, and straightforward automation tasks.
Strengths
- Very fast
- Low cost
- Good for simple tasks
- Low latency
Weaknesses
- Limited depth
- Struggles with complexity
- Less accurate
Gemini 3 Pro
Exceptional at diagnosing issues and root cause analysis. Strong pattern matching abilities help identify problems quickly across logs, metrics, and system state.
Strengths
- Best-in-class debugging
- Log analysis
- Root cause identification
- Fast diagnosis
Weaknesses
- Less thorough documentation
- Can jump to conclusions
- Weaker at planning
Gemini Flash
Ultra-fast model designed for rapid prototyping and quick iterations. Trades depth for speed, making it ideal for exploration and initial drafts.
Strengths
- Fastest responses
- Great for prototyping
- Very low cost
- Good for brainstorming
Weaknesses
- Shallow analysis
- Misses nuances
- Needs verification