architecture
Pattern: Learning Strategies (Prompting to RLHF)
Decision framework for agent learning strategies: prompting, few-shot, fine-tuning, and RLHF. Covers when to use each approach and implementation guidance.
Overview
How should an agent "learn" its task? The spectrum ranges from simple prompting to full RLHF.
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β LEARNING SPECTRUM β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
β β
β PROMPTING FEW-SHOT FINE-TUNING DPO RLHF β
β β
β βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββΊ β
β β
β No examples 5-20 500+ 1,000+ 10,000+ β
β Minutes Hours Days Days Weeks β
β Most flexible βββββββββββββββββββββββΊ Most consistent β
β Highest cost/query βββββββββββββββββββββββΊ Lowest cost/query β
β β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
Decision Framework
START: How should this agent learn?
β
βββΊ Requirements changing frequently?
β βββΊ Yes β PROMPTING
β
βββΊ Need consistent output format?
β βββΊ Have 500+ examples?
β β βββΊ Yes β FINE-TUNING
β βββΊ Have 5-20 examples?
β βββΊ FEW-SHOT PROMPTING
β
βββΊ Quality is subjective (preference-based)?
β βββΊ Have 1,000+ preference pairs + ML team?
β β βββΊ Yes β DPO or RLHF
β βββΊ No β FEEDBACK COLLECTION β periodic fine-tuning
β
βββΊ Default: PROMPTING with iteration
1. Prompting (Zero-Shot)
When to Use
- Rapid iteration needed
- Requirements still evolving
- General-purpose tasks
- Prototyping
Implementation
SYSTEM_PROMPT = """
You are a sales email writer. Follow these rules:
STYLE:
- Open with something specific to the prospect
- Never use "I hope this finds you well"
- Keep under 150 words
- Single, clear CTA
TONE:
- Professional but human
- Confident, not pushy
- No exclamation points
"""
Pros/Cons
| Pros | Cons |
|---|---|
| Instant iteration | Variable output quality |
| No training data needed | Higher token cost |
| Maximum flexibility | Less consistency |
| Easy to debug | Prompt engineering skill required |
2. Few-Shot Prompting
When to Use
- Have 5-20 high-quality examples
- Need more consistency than zero-shot
- Not enough data for fine-tuning
- Quick improvement needed
Implementation
FEW_SHOT_PROMPT = """
You are a sales email writer.
Example 1:
Input: VP Engineering at Series B startup, interested in API integrations
Output:
Subject: Quick question about your API roadmap
Hi [Name],
Saw TechCorp's recent product launchβcongrats on the traction.
I noticed you're expanding your integration ecosystem. We've helped similar
teams cut integration development time by 60%.
Would a 15-minute call to share how make sense?
Best,
[Rep]
Example 2:
[Another example...]
Example 3:
[Another example...]
Now write for this input:
{input}
"""
Best Practices
| Practice | Why |
|---|---|
| Diverse examples | Cover different scenarios |
| High-quality outputs | Model learns from examples |
| Consistent format | Easier pattern matching |
| Order matters | Put best examples first |
| 3-5 examples usually enough | Diminishing returns after |
3. Fine-Tuning
See ENT-FINE-TUNING for detailed concepts.
When to Use
- 500+ high-quality examples available
- Consistent output format required
- Latency critical (<500ms needed)
- High volume (cost matters at scale)
- Task is narrow and stable
Decision Checklist
## Fine-Tuning Readiness
### Data Requirements
- [ ] 500+ examples minimum
- [ ] Examples are human-verified
- [ ] Examples cover edge cases
- [ ] Consistent format across examples
- [ ] No PII or properly anonymized
### Task Requirements
- [ ] Task is well-defined and narrow
- [ ] Output format is rigid/structured
- [ ] Requirements unlikely to change
- [ ] Latency requirement: ___ms
### Resources
- [ ] Budget for training: $_____
- [ ] Time for data preparation
- [ ] Ongoing maintenance capacity
### Verdict: [ ] Ready [ ] Not Ready
Training Data Format
// OpenAI format
{
"messages": [
{"role": "system", "content": "Extract CRM fields from email."},
{"role": "user", "content": "Email: Thanks for the call..."},
{"role": "assistant", "content": "{\"next_step\": \"Follow-up\", ...}"}
]
}
Catalog Applications
| Agent | Why Fine-Tune |
|---|---|
| CRM Extractor | Consistent JSON schema |
| Expense Categorizer | Fixed category mapping |
| Resume Screener | Structured assessment |
4. DPO (Direct Preference Optimization)
When to Use
- Quality is subjective
- Have 1,000+ preference pairs
- Simpler than full RLHF
- No separate reward model needed
Implementation
from trl import DPOTrainer
# Preference data format
preferences = [
{
"prompt": "Write email to VP Engineering...",
"chosen": "Hi Sarah, Noticed your team's API work...",
"rejected": "Dear Ms. Chen, I hope this email finds you well..."
},
# ... 1000+ pairs
]
trainer = DPOTrainer(
model=base_model,
ref_model=reference_model,
train_dataset=preferences,
beta=0.1
)
trainer.train()
Collecting Preference Data
class PreferenceCollector:
def present_options(self, prompt: str, outputs: list):
"""Show user multiple outputs to rank."""
# UI presents options A and B
# User selects preferred
return {
"prompt": prompt,
"chosen": selected_output,
"rejected": rejected_output
}
5. RLHF (Full Pipeline)
See ENT-RLHF for detailed concepts.
When to Use
- Very high volume (>10K queries/month)
- Maximum quality alignment needed
- ML engineering team available
- Significant budget ($10K+)
Pipeline
SFT Model β Reward Model β PPO Optimization β Final Model
(from preferences)
Typically Overkill For
- Most enterprise agents
- Initial deployment
- Low-volume use cases
6. Continuous Learning Loop
For most agents, the best approach is iterative improvement:
βββββββββββ βββββββββββ βββββββββββ βββββββββββ
β Deploy βββββΊβ Collect βββββΊβ Analyze βββββΊβ Improve βββββ
β v1 β βFeedback β β Patternsβ β Prompts β β
βββββββββββ βββββββββββ βββββββββββ βββββββββββ β
β² β
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
Feedback Collection
class FeedbackCollector:
def log_output(self, prompt, output, metadata=None):
self.storage.insert({
"prompt": prompt,
"output": output,
"feedback": None,
"timestamp": datetime.now()
})
def record_feedback(self, output_id, feedback: dict):
"""
feedback types:
- thumbs_up/thumbs_down
- edit (with edited version)
- rating (1-5)
- accepted/rejected
"""
self.storage.update(output_id, {"feedback": feedback})
def export_for_training(self):
"""Periodically export for fine-tuning."""
positive = self.storage.query({"feedback.rating": {"$gte": 4}})
return create_training_data(positive)
Improvement Cycle
| Week | Action | Data |
|---|---|---|
| 1-2 | Deploy with prompting | - |
| 3-4 | Collect feedback | 100+ samples |
| 5-6 | Analyze patterns | Find failure modes |
| 7-8 | Improve prompts | Address failures |
| 9-12 | Continue collection | 500+ samples |
| 13+ | Consider fine-tuning | If patterns stable |
Comparison Matrix
| Factor | Prompting | Few-Shot | Fine-Tuning | DPO/RLHF |
|---|---|---|---|---|
| Setup time | Minutes | Hours | Days | Weeks |
| Data needed | 0 | 5-20 | 500+ | 1,000+ |
| Iteration speed | Instant | Instant | Hours | Days |
| Output consistency | Low | Medium | High | Highest |
| Inference cost | High | Higher | Low | Lowest |
| Flexibility | High | Medium | Low | Lowest |
| Best for | Prototyping | Quick wins | Production | Scale |
Related Documents
- ENT-FINE-TUNING - Fine-tuning concepts
- ENT-RLHF - RLHF concepts
- Pattern: Quality Assurance & Safety - Feedback collection
- Implementation Guide: Agent Evaluation & Testing - Testing strategies