ifixai-ai/iFixAi

Most agent evaluation tools ask whether a model is fast, cheap, or resistant to prompt injection. iFixAi asks a different question: is the agent actually doing the job it was hired to do, according to the business rules and organizational structure it operates within? That reframing drives the entire design.

The tool runs 49 inspections grouped into 18 categories, five of which form the graded core. Those five pillars — fabrication, manipulation, deception, unpredictability, and opacity — are weighted unevenly, with manipulation carrying the heaviest load at 0.35. The resulting A–F letter grade is a weighted average of those five only, so scores stay comparable across agents regardless of what capabilities each one exposes. Mandatory minimums add a hard floor: miss the thresholds on B01, B08, or P01 and the overall score is capped at 60%, regardless of how well the agent performs elsewhere.

The remaining 13 categories — covering things like sandbagging, oversight atrophy, identity attestation, and stakeholder conflict — are scored and reported but deliberately excluded from the grade. The README is explicit about why: keeping them off the grade ensures comparability between agents that expose different capabilities. One of those premium categories, P01, is the exception; it can still trigger the 60% cap. Everything here is Apache 2.0, so "premium" describes a capability tier, not a pricing wall.

Testing your own agent rather than a bare model API is the point. A bare model scores lower by design, because it lacks the system prompt, tools, retrieval, and guardrails that a real deployed agent would have. iFixAi treats the agent as a black box: point it at an OpenAI-compatible HTTP endpoint and it measures the governance already in place. Expose more of the agent's internals through optional adapter hooks and more inspections can be scored rather than marked insufficient_evidence.

The independent-judge architecture is worth noting. A grade is described as "citable" only when a second provider from a different vendor grades the agent — the SUT's own vendor is explicitly excluded from the judge role. The README suggests two practical setups: a single Sonnet judge at roughly $12–18 for a full suite, or a Gemini 2.5 Pro and GPT-5.4-mini pair that comes in under that cost while adding cross-vendor robustness.

The case studies are an honest touch. Two fixtures reconstructed from public incident reports — one involving a pizza chain complaint, one involving social media account takeovers — both land at F. The README is careful to note these are reconstructions, not tests of any production system. They exist to show what failure looks like on the scorecard, not to indict anyone.

The mock provider that ships seeded defects and scores 15 out of 49 on purpose is a small but smart decision: it means the first thing a new user sees is a failing scorecard, which is far more instructive than a passing one.