architecture
Pattern: Quality Assurance & Safety
Patterns for ensuring agent output quality and safety. Covers validation, human-in-the-loop, monitoring, and pre-deployment safety checklists.
Overview
AI agents require quality gates and safety measures before and during production deployment. This pattern covers validation, human oversight, monitoring, and safety checklists.
1. Output Validation
Validation Framework
class OutputValidator:
def __init__(self, validators: list):
self.validators = validators
def validate(self, output: str, context: dict) -> ValidationResult:
results = []
for validator in self.validators:
result = validator.check(output, context)
results.append(result)
return ValidationResult(
passed=all(r.passed for r in results),
issues=[r.issue for r in results if not r.passed],
warnings=[r.warning for r in results if r.warning]
)
Common Validators
Hallucination Detector
class HallucinationDetector:
"""Check if output makes claims not supported by source."""
def check(self, output: str, context: dict) -> CheckResult:
source_text = context.get("source_documents", "")
# Extract claims from output
claims = self.extract_claims(output)
# Verify each claim against source
unsupported = []
for claim in claims:
if not self.is_supported(claim, source_text):
unsupported.append(claim)
return CheckResult(
passed=len(unsupported) == 0,
issue=f"Unsupported claims: {unsupported}" if unsupported else None
)
Format Validator
class FormatValidator:
"""Ensure output matches expected schema."""
def __init__(self, schema: dict):
self.schema = schema
def check(self, output: str, context: dict) -> CheckResult:
try:
parsed = json.loads(output)
jsonschema.validate(parsed, self.schema)
return CheckResult(passed=True)
except (json.JSONDecodeError, jsonschema.ValidationError) as e:
return CheckResult(passed=False, issue=str(e))
PII Scanner
class PIIScanner:
"""Detect personally identifiable information."""
PATTERNS = {
"ssn": r"\b\d{3}-\d{2}-\d{4}\b",
"email": r"\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b",
"phone": r"\b\d{3}[-.]?\d{3}[-.]?\d{4}\b",
"credit_card": r"\b\d{4}[-\s]?\d{4}[-\s]?\d{4}[-\s]?\d{4}\b"
}
def check(self, output: str, context: dict) -> CheckResult:
found_pii = []
for pii_type, pattern in self.PATTERNS.items():
if re.search(pattern, output):
found_pii.append(pii_type)
return CheckResult(
passed=len(found_pii) == 0,
issue=f"PII detected: {found_pii}" if found_pii else None
)
Tone Checker
class ToneChecker:
"""Ensure output matches required tone."""
def __init__(self, llm, target_tone: str):
self.llm = llm
self.target_tone = target_tone
def check(self, output: str, context: dict) -> CheckResult:
prompt = f"""
Analyze the tone of this text:
{output}
Expected tone: {self.target_tone}
Does the text match the expected tone? Answer YES or NO with brief reason.
"""
result = self.llm.invoke(prompt)
passed = "YES" in result.upper()
return CheckResult(passed=passed, warning=result if not passed else None)
2. Human-in-the-Loop Patterns
Pattern 1: Approval Gate
βββββββββ βββββββββ βββββββββ βββββββββ
β Agent ββββΊβ Draft ββββΊβ Human ββββΊβ Send β
β β β β β Reviewβ β β
βββββββββ βββββββββ βββββββββ βββββββββ
Use for:
- External communications
- Legal/compliance documents
- Executive reports
- High-stakes decisions
Implementation:
class ApprovalGate:
def __init__(self, queue_service):
self.queue = queue_service
async def submit_for_approval(self, output: dict, metadata: dict):
approval_request = {
"id": uuid4(),
"output": output,
"metadata": metadata,
"status": "pending",
"created_at": datetime.now()
}
await self.queue.add(approval_request)
return approval_request["id"]
async def check_approval(self, request_id: str) -> dict:
request = await self.queue.get(request_id)
return {
"status": request["status"],
"approved_output": request.get("approved_output"),
"feedback": request.get("feedback")
}
Pattern 2: Confidence-Based Routing
βββββββββ βββββββββ βββββββββ
β Agent ββββΊβ Check ββββΊβ Auto β
β β β Conf β β Send β
βββββββββ βββββ¬ββββ βββββββββ
β Low confidence
βΌ
βββββββββ
β Human β
β Queue β
βββββββββ
Use for:
- High-volume tasks
- Variable quality scenarios
- Training data collection
Implementation:
class ConfidenceRouter:
def __init__(self, threshold: float = 0.8):
self.threshold = threshold
def route(self, output: str, confidence: float) -> str:
if confidence >= self.threshold:
return "auto_approve"
elif confidence >= self.threshold * 0.7:
return "quick_review" # Spot check
else:
return "full_review" # Human required
Pattern 3: Spot Check Sampling
βββββββββ βββββββββ βββββββββ
β Agent ββββΊβ Auto ββββΊβ Log ββββΊ 5% Sample β Human Review
β β β Send β β β
βββββββββ βββββββββ βββββββββ
Use for:
- Mature, stable agents
- Quality monitoring
- Training data collection
Implementation:
class SpotChecker:
def __init__(self, sample_rate: float = 0.05):
self.sample_rate = sample_rate
def should_sample(self) -> bool:
return random.random() < self.sample_rate
def log_for_review(self, output: dict, outcome: dict):
if self.should_sample():
self.review_queue.add({
"output": output,
"outcome": outcome,
"sampled_at": datetime.now()
})
3. Monitoring & Observability
Key Metrics
| Metric | Description | Alert Threshold |
|---|---|---|
| Latency | Response time | p95 > 5s |
| Error rate | Failed generations | > 1% |
| Validation failure | Failed checks | > 5% |
| Human escalation | Routed to human | > 20% |
| User satisfaction | Thumbs up ratio | < 80% |
| Cost per query | Token cost | > budget |
Logging Schema
@dataclass
class AgentLog:
request_id: str
timestamp: datetime
agent_name: str
input_tokens: int
output_tokens: int
latency_ms: int
model: str
validation_passed: bool
validation_issues: list
human_routed: bool
user_feedback: Optional[str]
cost_usd: float
Dashboard Components
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β AGENT HEALTH DASHBOARD β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
β β
β Requests/hr: ββββββββββββ 1,234 Latency p50: 1.2s β
β Error rate: ββββββββββββ 0.8% Latency p95: 3.4s β
β Human route: ββββββββββββ 12% Latency p99: 5.1s β
β β
β Validation Issues (24h): β
β ββ Format errors: 23 β
β ββ PII detected: 5 β
β ββ Tone mismatch: 12 β
β ββ Hallucination: 3 β
β β
β User Feedback: β
β π 87% π 8% No response 5% β
β β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
4. Pre-Deployment Safety Checklist
Data Safety
- [ ] PII handling
- [ ] No PII in prompts (or properly anonymized)
- [ ] No PII in logs (or encrypted)
- [ ] PII scanner enabled on outputs
- [ ] Credentials
- [ ] No secrets in prompts
- [ ] API keys in secure vault
- [ ] No credential logging
- [ ] Data retention
- [ ] Retention policy defined
- [ ] Auto-deletion configured
- [ ] User data deletion process
Output Safety
- [ ] Hallucination mitigation
- [ ] Source citation required
- [ ] Fact-checking for critical claims
- [ ] "I don't know" fallback
- [ ] Content filtering
- [ ] Harmful content detection
- [ ] Brand safety checks
- [ ] Regulatory compliance
- [ ] Output validation
- [ ] Format validation enabled
- [ ] Business rule validation
- [ ] Human review for edge cases
Operational Safety
- [ ] Rate limiting
- [ ] Per-user limits configured
- [ ] Global limits configured
- [ ] Burst protection
- [ ] Cost controls
- [ ] Budget alerts configured
- [ ] Auto-shutoff at threshold
- [ ] Cost monitoring dashboard
- [ ] Reliability
- [ ] Fallback for API failures
- [ ] Graceful degradation
- [ ] Timeout handling
- [ ] Monitoring
- [ ] Logging enabled
- [ ] Alerts configured
- [ ] Dashboard available
Compliance
- [ ] Legal review
- [ ] Terms of service updated
- [ ] Disclosure of AI use
- [ ] Liability considerations
- [ ] Bias testing (if applicable)
- [ ] Demographic analysis
- [ ] Fairness metrics
- [ ] Mitigation plan
- [ ] Audit trail
- [ ] All decisions logged
- [ ] Human override available
- [ ] Audit export capability
5. Incident Response
Severity Levels
| Level | Criteria | Response Time |
|---|---|---|
| P0 | Customer-facing errors, data breach | 15 min |
| P1 | High error rate, quality degradation | 1 hour |
| P2 | Performance issues, minor bugs | 4 hours |
| P3 | Cosmetic issues, improvements | Next sprint |
Response Playbook
## P0/P1 Incident Response
1. **Detect** (automated)
- Alert fires from monitoring
- User reports
2. **Triage** (5 min)
- Confirm incident
- Assess scope and impact
- Assign owner
3. **Mitigate** (15 min)
- Roll back if possible
- Disable affected agent
- Route to human fallback
4. **Investigate** (ongoing)
- Review logs
- Identify root cause
- Document findings
5. **Resolve**
- Fix root cause
- Test fix
- Deploy with monitoring
6. **Review**
- Post-mortem
- Update runbooks
- Improve monitoring
Related Documents
- Pattern: Learning Strategies (Prompting to RLHF) - Feedback collection for improvement
- Implementation Guide: Agent Evaluation & Testing - Testing strategies
- AI Agents Knowledge Base - Index - Agent catalog index