LLMOps Intermediate — 109: simulate empty retrieval on `Synthetic eval data ethics` — memo `728409 [109]`

Lesson 109: Synthetic eval data ethics

Focus

Anchor this drill to one production LLM workflow—even hypothetical. Token Synthetic eval data ethics:109 keeps neighbouring lessons differentiable.

Key ideas

• Thread: Synthetic eval data ethics · drill v9 · spin 391734.
• Habit: attach a trace_id to every completion you would paste into an ops dashboard.
• Guardrail: add one RACI bullet for prompt or index changes before tomorrow's standup.

Deep dive notebook

Synthetic drill artefacts

Prompt scaffold

ROLE: LLMOps analyst cohort 12
INPUTS:
- excerpts tagged [chunk_id ...]
- policy_bundle_0
TASK:
  1) Summarize deltas with citations
  2) Confidence LOW|MED|HIGH + evidence
  3) If facts missing emit MISSING_FACTS
USER_SEED >>> What changed between rollout 10 and 51?

Practice

Practice List five adversarial prompts using your org's domain nouns. — 109 Bump 23.