Answer
What is an AI quality scorecard?
An AI quality scorecard is a structured definition of quality that an evaluation system can apply consistently. In Kynesis a scorecard has weighted sections, criteria with point scales, pass thresholds, and hard or soft critical-failure rules — and it applies equally to human interactions, AI outputs, content and workflows.
Anatomy of a scorecard
Sections
Groups of related criteria, each carrying a weight in the overall score.
Criteria
The individual checks, with a point scale and a clear pass condition.
Thresholds
The score at which an interaction passes, needs review or fails.
Hard critical failures
Breaches that fail the evaluation outright — regulatory, safety or policy.
Soft critical failures
Risk signals that are flagged for attention without failing the evaluation.
Knowledge grounding
Reference material the evaluation can check answers against.
Designing one that works
- 01
Start from decisions
Only include criteria that would change coaching, a fix or a product decision.
- 02
Make each criterion observable
Write it so two reviewers reading the same transcript would agree.
- 03
Separate risk from quality
Keep compliance breaches as critical failures rather than points lost.
- 04
Weight deliberately
Weights are a statement about what your team actually values.
- 05
Version it
Change the scorecard as policy changes, and keep older versions for comparability.
What a scored result contains
Per-criterion score
Where quality was gained or lost, not just a total.
Quoted evidence
The exact part of the interaction or output the score is based on.
Reasoning
Why that evidence produced that score.
Recommendation
The concrete next action for the agent, team or product.
Detected issues
Compliance flags and risk signals surfaced alongside the score.