architecture

Pattern: Quality Assurance & Safety

Patterns for ensuring agent output quality and safety. Covers validation, human-in-the-loop, monitoring, and pre-deployment safety checklists.

Overview

AI agents require quality gates and safety measures before and during production deployment. This pattern covers validation, human oversight, monitoring, and safety checklists.


1. Output Validation

Validation Framework

class OutputValidator:
    def __init__(self, validators: list):
        self.validators = validators

    def validate(self, output: str, context: dict) -> ValidationResult:
        results = []
        for validator in self.validators:
            result = validator.check(output, context)
            results.append(result)

        return ValidationResult(
            passed=all(r.passed for r in results),
            issues=[r.issue for r in results if not r.passed],
            warnings=[r.warning for r in results if r.warning]
        )

Common Validators

Hallucination Detector

class HallucinationDetector:
    """Check if output makes claims not supported by source."""

    def check(self, output: str, context: dict) -> CheckResult:
        source_text = context.get("source_documents", "")

        # Extract claims from output
        claims = self.extract_claims(output)

        # Verify each claim against source
        unsupported = []
        for claim in claims:
            if not self.is_supported(claim, source_text):
                unsupported.append(claim)

        return CheckResult(
            passed=len(unsupported) == 0,
            issue=f"Unsupported claims: {unsupported}" if unsupported else None
        )

Format Validator

class FormatValidator:
    """Ensure output matches expected schema."""

    def __init__(self, schema: dict):
        self.schema = schema

    def check(self, output: str, context: dict) -> CheckResult:
        try:
            parsed = json.loads(output)
            jsonschema.validate(parsed, self.schema)
            return CheckResult(passed=True)
        except (json.JSONDecodeError, jsonschema.ValidationError) as e:
            return CheckResult(passed=False, issue=str(e))

PII Scanner

class PIIScanner:
    """Detect personally identifiable information."""

    PATTERNS = {
        "ssn": r"\b\d{3}-\d{2}-\d{4}\b",
        "email": r"\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b",
        "phone": r"\b\d{3}[-.]?\d{3}[-.]?\d{4}\b",
        "credit_card": r"\b\d{4}[-\s]?\d{4}[-\s]?\d{4}[-\s]?\d{4}\b"
    }

    def check(self, output: str, context: dict) -> CheckResult:
        found_pii = []
        for pii_type, pattern in self.PATTERNS.items():
            if re.search(pattern, output):
                found_pii.append(pii_type)

        return CheckResult(
            passed=len(found_pii) == 0,
            issue=f"PII detected: {found_pii}" if found_pii else None
        )

Tone Checker

class ToneChecker:
    """Ensure output matches required tone."""

    def __init__(self, llm, target_tone: str):
        self.llm = llm
        self.target_tone = target_tone

    def check(self, output: str, context: dict) -> CheckResult:
        prompt = f"""
        Analyze the tone of this text:
        {output}

        Expected tone: {self.target_tone}

        Does the text match the expected tone? Answer YES or NO with brief reason.
        """
        result = self.llm.invoke(prompt)
        passed = "YES" in result.upper()
        return CheckResult(passed=passed, warning=result if not passed else None)

2. Human-in-the-Loop Patterns

Pattern 1: Approval Gate

β”Œβ”€β”€β”€β”€β”€β”€β”€β”   β”Œβ”€β”€β”€β”€β”€β”€β”€β”   β”Œβ”€β”€β”€β”€β”€β”€β”€β”   β”Œβ”€β”€β”€β”€β”€β”€β”€β”
β”‚ Agent │──►│ Draft │──►│ Human │──►│ Send  β”‚
β”‚       β”‚   β”‚       β”‚   β”‚ Reviewβ”‚   β”‚       β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”˜   β””β”€β”€β”€β”€β”€β”€β”€β”˜   β””β”€β”€β”€β”€β”€β”€β”€β”˜   β””β”€β”€β”€β”€β”€β”€β”€β”˜

Use for:

  • External communications
  • Legal/compliance documents
  • Executive reports
  • High-stakes decisions

Implementation:

class ApprovalGate:
    def __init__(self, queue_service):
        self.queue = queue_service

    async def submit_for_approval(self, output: dict, metadata: dict):
        approval_request = {
            "id": uuid4(),
            "output": output,
            "metadata": metadata,
            "status": "pending",
            "created_at": datetime.now()
        }
        await self.queue.add(approval_request)
        return approval_request["id"]

    async def check_approval(self, request_id: str) -> dict:
        request = await self.queue.get(request_id)
        return {
            "status": request["status"],
            "approved_output": request.get("approved_output"),
            "feedback": request.get("feedback")
        }

Pattern 2: Confidence-Based Routing

β”Œβ”€β”€β”€β”€β”€β”€β”€β”   β”Œβ”€β”€β”€β”€β”€β”€β”€β”   β”Œβ”€β”€β”€β”€β”€β”€β”€β”
β”‚ Agent │──►│ Check │──►│ Auto  β”‚
β”‚       β”‚   β”‚ Conf  β”‚   β”‚ Send  β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”˜   β””β”€β”€β”€β”¬β”€β”€β”€β”˜   β””β”€β”€β”€β”€β”€β”€β”€β”˜
                β”‚ Low confidence
                β–Ό
           β”Œβ”€β”€β”€β”€β”€β”€β”€β”
           β”‚ Human β”‚
           β”‚ Queue β”‚
           β””β”€β”€β”€β”€β”€β”€β”€β”˜

Use for:

  • High-volume tasks
  • Variable quality scenarios
  • Training data collection

Implementation:

class ConfidenceRouter:
    def __init__(self, threshold: float = 0.8):
        self.threshold = threshold

    def route(self, output: str, confidence: float) -> str:
        if confidence >= self.threshold:
            return "auto_approve"
        elif confidence >= self.threshold * 0.7:
            return "quick_review"  # Spot check
        else:
            return "full_review"   # Human required

Pattern 3: Spot Check Sampling

β”Œβ”€β”€β”€β”€β”€β”€β”€β”   β”Œβ”€β”€β”€β”€β”€β”€β”€β”   β”Œβ”€β”€β”€β”€β”€β”€β”€β”
β”‚ Agent │──►│ Auto  │──►│ Log   │──► 5% Sample β†’ Human Review
β”‚       β”‚   β”‚ Send  β”‚   β”‚       β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”˜   β””β”€β”€β”€β”€β”€β”€β”€β”˜   β””β”€β”€β”€β”€β”€β”€β”€β”˜

Use for:

  • Mature, stable agents
  • Quality monitoring
  • Training data collection

Implementation:

class SpotChecker:
    def __init__(self, sample_rate: float = 0.05):
        self.sample_rate = sample_rate

    def should_sample(self) -> bool:
        return random.random() < self.sample_rate

    def log_for_review(self, output: dict, outcome: dict):
        if self.should_sample():
            self.review_queue.add({
                "output": output,
                "outcome": outcome,
                "sampled_at": datetime.now()
            })

3. Monitoring & Observability

Key Metrics

Metric Description Alert Threshold
Latency Response time p95 > 5s
Error rate Failed generations > 1%
Validation failure Failed checks > 5%
Human escalation Routed to human > 20%
User satisfaction Thumbs up ratio < 80%
Cost per query Token cost > budget

Logging Schema

@dataclass
class AgentLog:
    request_id: str
    timestamp: datetime
    agent_name: str
    input_tokens: int
    output_tokens: int
    latency_ms: int
    model: str
    validation_passed: bool
    validation_issues: list
    human_routed: bool
    user_feedback: Optional[str]
    cost_usd: float

Dashboard Components

β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚                    AGENT HEALTH DASHBOARD                    β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚                                                             β”‚
β”‚  Requests/hr: β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ 1,234    Latency p50: 1.2s      β”‚
β”‚  Error rate:  β–ˆβ–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘ 0.8%     Latency p95: 3.4s      β”‚
β”‚  Human route: β–ˆβ–ˆβ–ˆβ–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘ 12%      Latency p99: 5.1s      β”‚
β”‚                                                             β”‚
β”‚  Validation Issues (24h):                                   β”‚
β”‚  β”œβ”€ Format errors:     23                                   β”‚
β”‚  β”œβ”€ PII detected:      5                                    β”‚
β”‚  β”œβ”€ Tone mismatch:     12                                   β”‚
β”‚  └─ Hallucination:     3                                    β”‚
β”‚                                                             β”‚
β”‚  User Feedback:                                              β”‚
β”‚  πŸ‘ 87%  πŸ‘Ž 8%  No response 5%                               β”‚
β”‚                                                             β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

4. Pre-Deployment Safety Checklist

Data Safety

- [ ] PII handling
  - [ ] No PII in prompts (or properly anonymized)
  - [ ] No PII in logs (or encrypted)
  - [ ] PII scanner enabled on outputs

- [ ] Credentials
  - [ ] No secrets in prompts
  - [ ] API keys in secure vault
  - [ ] No credential logging

- [ ] Data retention
  - [ ] Retention policy defined
  - [ ] Auto-deletion configured
  - [ ] User data deletion process

Output Safety

- [ ] Hallucination mitigation
  - [ ] Source citation required
  - [ ] Fact-checking for critical claims
  - [ ] "I don't know" fallback

- [ ] Content filtering
  - [ ] Harmful content detection
  - [ ] Brand safety checks
  - [ ] Regulatory compliance

- [ ] Output validation
  - [ ] Format validation enabled
  - [ ] Business rule validation
  - [ ] Human review for edge cases

Operational Safety

- [ ] Rate limiting
  - [ ] Per-user limits configured
  - [ ] Global limits configured
  - [ ] Burst protection

- [ ] Cost controls
  - [ ] Budget alerts configured
  - [ ] Auto-shutoff at threshold
  - [ ] Cost monitoring dashboard

- [ ] Reliability
  - [ ] Fallback for API failures
  - [ ] Graceful degradation
  - [ ] Timeout handling

- [ ] Monitoring
  - [ ] Logging enabled
  - [ ] Alerts configured
  - [ ] Dashboard available

Compliance

- [ ] Legal review
  - [ ] Terms of service updated
  - [ ] Disclosure of AI use
  - [ ] Liability considerations

- [ ] Bias testing (if applicable)
  - [ ] Demographic analysis
  - [ ] Fairness metrics
  - [ ] Mitigation plan

- [ ] Audit trail
  - [ ] All decisions logged
  - [ ] Human override available
  - [ ] Audit export capability

5. Incident Response

Severity Levels

Level Criteria Response Time
P0 Customer-facing errors, data breach 15 min
P1 High error rate, quality degradation 1 hour
P2 Performance issues, minor bugs 4 hours
P3 Cosmetic issues, improvements Next sprint

Response Playbook

## P0/P1 Incident Response

1. **Detect** (automated)
   - Alert fires from monitoring
   - User reports

2. **Triage** (5 min)
   - Confirm incident
   - Assess scope and impact
   - Assign owner

3. **Mitigate** (15 min)
   - Roll back if possible
   - Disable affected agent
   - Route to human fallback

4. **Investigate** (ongoing)
   - Review logs
   - Identify root cause
   - Document findings

5. **Resolve**
   - Fix root cause
   - Test fix
   - Deploy with monitoring

6. **Review**
   - Post-mortem
   - Update runbooks
   - Improve monitoring