architecture

Pattern: Learning Strategies (Prompting to RLHF)

Decision framework for agent learning strategies: prompting, few-shot, fine-tuning, and RLHF. Covers when to use each approach and implementation guidance.

Overview

How should an agent "learn" its task? The spectrum ranges from simple prompting to full RLHF.

β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚                        LEARNING SPECTRUM                                 β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚                                                                         β”‚
β”‚  PROMPTING    FEW-SHOT    FINE-TUNING    DPO         RLHF              β”‚
β”‚                                                                         β”‚
β”‚  ◄─────────────────────────────────────────────────────────────────►   β”‚
β”‚                                                                         β”‚
β”‚  No examples   5-20        500+          1,000+      10,000+            β”‚
β”‚  Minutes       Hours       Days          Days        Weeks              β”‚
β”‚  Most flexible             ──────────────────────► Most consistent     β”‚
β”‚  Highest cost/query        ──────────────────────► Lowest cost/query   β”‚
β”‚                                                                         β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

Decision Framework

START: How should this agent learn?
β”‚
β”œβ”€β–Ί Requirements changing frequently?
β”‚   └─► Yes β†’ PROMPTING
β”‚
β”œβ”€β–Ί Need consistent output format?
β”‚   β”œβ”€β–Ί Have 500+ examples?
β”‚   β”‚   └─► Yes β†’ FINE-TUNING
β”‚   └─► Have 5-20 examples?
β”‚       └─► FEW-SHOT PROMPTING
β”‚
β”œβ”€β–Ί Quality is subjective (preference-based)?
β”‚   β”œβ”€β–Ί Have 1,000+ preference pairs + ML team?
β”‚   β”‚   └─► Yes β†’ DPO or RLHF
β”‚   └─► No β†’ FEEDBACK COLLECTION β†’ periodic fine-tuning
β”‚
└─► Default: PROMPTING with iteration

1. Prompting (Zero-Shot)

When to Use

  • Rapid iteration needed
  • Requirements still evolving
  • General-purpose tasks
  • Prototyping

Implementation

SYSTEM_PROMPT = """
You are a sales email writer. Follow these rules:

STYLE:
- Open with something specific to the prospect
- Never use "I hope this finds you well"
- Keep under 150 words
- Single, clear CTA

TONE:
- Professional but human
- Confident, not pushy
- No exclamation points
"""

Pros/Cons

Pros Cons
Instant iteration Variable output quality
No training data needed Higher token cost
Maximum flexibility Less consistency
Easy to debug Prompt engineering skill required

2. Few-Shot Prompting

When to Use

  • Have 5-20 high-quality examples
  • Need more consistency than zero-shot
  • Not enough data for fine-tuning
  • Quick improvement needed

Implementation

FEW_SHOT_PROMPT = """
You are a sales email writer.

Example 1:
Input: VP Engineering at Series B startup, interested in API integrations
Output:
Subject: Quick question about your API roadmap

Hi [Name],

Saw TechCorp's recent product launchβ€”congrats on the traction.

I noticed you're expanding your integration ecosystem. We've helped similar
teams cut integration development time by 60%.

Would a 15-minute call to share how make sense?

Best,
[Rep]

Example 2:
[Another example...]

Example 3:
[Another example...]

Now write for this input:
{input}
"""

Best Practices

Practice Why
Diverse examples Cover different scenarios
High-quality outputs Model learns from examples
Consistent format Easier pattern matching
Order matters Put best examples first
3-5 examples usually enough Diminishing returns after

3. Fine-Tuning

See ENT-FINE-TUNING for detailed concepts.

When to Use

  • 500+ high-quality examples available
  • Consistent output format required
  • Latency critical (<500ms needed)
  • High volume (cost matters at scale)
  • Task is narrow and stable

Decision Checklist

## Fine-Tuning Readiness

### Data Requirements
- [ ] 500+ examples minimum
- [ ] Examples are human-verified
- [ ] Examples cover edge cases
- [ ] Consistent format across examples
- [ ] No PII or properly anonymized

### Task Requirements
- [ ] Task is well-defined and narrow
- [ ] Output format is rigid/structured
- [ ] Requirements unlikely to change
- [ ] Latency requirement: ___ms

### Resources
- [ ] Budget for training: $_____
- [ ] Time for data preparation
- [ ] Ongoing maintenance capacity

### Verdict: [ ] Ready [ ] Not Ready

Training Data Format

// OpenAI format
{
  "messages": [
    {"role": "system", "content": "Extract CRM fields from email."},
    {"role": "user", "content": "Email: Thanks for the call..."},
    {"role": "assistant", "content": "{\"next_step\": \"Follow-up\", ...}"}
  ]
}

Catalog Applications

Agent Why Fine-Tune
CRM Extractor Consistent JSON schema
Expense Categorizer Fixed category mapping
Resume Screener Structured assessment

4. DPO (Direct Preference Optimization)

When to Use

  • Quality is subjective
  • Have 1,000+ preference pairs
  • Simpler than full RLHF
  • No separate reward model needed

Implementation

from trl import DPOTrainer

# Preference data format
preferences = [
    {
        "prompt": "Write email to VP Engineering...",
        "chosen": "Hi Sarah, Noticed your team's API work...",
        "rejected": "Dear Ms. Chen, I hope this email finds you well..."
    },
    # ... 1000+ pairs
]

trainer = DPOTrainer(
    model=base_model,
    ref_model=reference_model,
    train_dataset=preferences,
    beta=0.1
)
trainer.train()

Collecting Preference Data

class PreferenceCollector:
    def present_options(self, prompt: str, outputs: list):
        """Show user multiple outputs to rank."""
        # UI presents options A and B
        # User selects preferred
        return {
            "prompt": prompt,
            "chosen": selected_output,
            "rejected": rejected_output
        }

5. RLHF (Full Pipeline)

See ENT-RLHF for detailed concepts.

When to Use

  • Very high volume (>10K queries/month)
  • Maximum quality alignment needed
  • ML engineering team available
  • Significant budget ($10K+)

Pipeline

SFT Model β†’ Reward Model β†’ PPO Optimization β†’ Final Model
              (from preferences)

Typically Overkill For

  • Most enterprise agents
  • Initial deployment
  • Low-volume use cases

6. Continuous Learning Loop

For most agents, the best approach is iterative improvement:

β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”    β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”    β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”    β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ Deploy  │───►│ Collect │───►│ Analyze │───►│ Improve │───┐
β”‚ v1      β”‚    β”‚Feedback β”‚    β”‚ Patternsβ”‚    β”‚ Prompts β”‚   β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜    β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜    β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜    β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜   β”‚
     β–²                                                      β”‚
     β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

Feedback Collection

class FeedbackCollector:
    def log_output(self, prompt, output, metadata=None):
        self.storage.insert({
            "prompt": prompt,
            "output": output,
            "feedback": None,
            "timestamp": datetime.now()
        })

    def record_feedback(self, output_id, feedback: dict):
        """
        feedback types:
        - thumbs_up/thumbs_down
        - edit (with edited version)
        - rating (1-5)
        - accepted/rejected
        """
        self.storage.update(output_id, {"feedback": feedback})

    def export_for_training(self):
        """Periodically export for fine-tuning."""
        positive = self.storage.query({"feedback.rating": {"$gte": 4}})
        return create_training_data(positive)

Improvement Cycle

Week Action Data
1-2 Deploy with prompting -
3-4 Collect feedback 100+ samples
5-6 Analyze patterns Find failure modes
7-8 Improve prompts Address failures
9-12 Continue collection 500+ samples
13+ Consider fine-tuning If patterns stable

Comparison Matrix

Factor Prompting Few-Shot Fine-Tuning DPO/RLHF
Setup time Minutes Hours Days Weeks
Data needed 0 5-20 500+ 1,000+
Iteration speed Instant Instant Hours Days
Output consistency Low Medium High Highest
Inference cost High Higher Low Lowest
Flexibility High Medium Low Lowest
Best for Prototyping Quick wins Production Scale