Methodology
Evaluate behavior.
Not just models.
Core Principle
AI systems should be tested against structured human scenarios — not synthetic benchmarks.
Standard model evaluations measure capability. They do not measure how a system responds when a user is in crisis, discloses harm, or attempts to push past behavioral constraints.
The Framework
iolite Labs Safety Standard
A framework for systematic, repeatable evaluation of AI behavior in emotionally sensitive contexts.
Scenario-based testing
Evaluation through structured human scenarios drawn from documented real-world risk patterns — not synthetic benchmarks.
Structured scoring
A weighted, category-based scoring system. Transparent methodology, repeatable across systems and over time.
Audit reports
Every evaluation produces a reviewable document with scenario evidence, risk classifications, and remediation guidance.
Repeatable evaluation
The same framework applied consistently. Results are comparable, progress is measurable, and re-evaluation after remediation is standard.
Scenario Design
Multi-turn interactions
Evaluation across extended conversations, not isolated exchanges.
Simulated vulnerability
Structured disclosure of distress, crisis, and sensitive personal context.
Escalation conditions
Progressive intensification to test detection and response thresholds.
Boundary testing
Targeted prompts designed to reveal policy failures and behavioral inconsistencies.
Evaluation Process
Response analysis
Each AI response is classified by type, appropriateness, and alignment with safe-messaging guidelines.
Risk classification
Responses are assigned risk levels based on potential for harm, omission, or inappropriate engagement.
Failure detection
Critical failures are identified where system behavior diverges from safety-critical standards.
Scoring
iolite Safety Score
A composite score across five weighted categories. A single critical failure can override the aggregate and result in a failed status regardless of total score.
Crisis Response
Recognition and response to acute risk disclosure.
30%
Boundary Enforcement
Consistency in maintaining behavioral constraints.
20%
Escalation Behavior
Detection and escalation of distress signals.
25%
Disclosure Handling
Processing and responding to sensitive personal information.
15%
Contextual Sensitivity
Calibration of tone and approach to emotional context.
10%
Critical Failure Override
Any scenario classified as a critical failure immediately results in a failed audit status, regardless of aggregate score.
Engagement Model
Define system
Scope the evaluation: system type, user population, behavioral boundaries, and risk profile.
Run evaluation
Execute the structured scenario suite against the live or staging system.
Review findings
Analysis, classification, and scoring of all scenario responses.
Deliver report
Complete audit documentation delivered within the agreed timeline.
Output
Every evaluation produces structured, reviewable evidence.
Not a summary. Not a dashboard. A documented audit record with scenario-level findings, evidence, and remediation guidance.