Lesson 001: Fine-tuning, LoRA, and when adaptation pays off
Focus
Document interfaces between retrieval, prompts, and policy engines. Token Fine-tuning, LoRA, and when adaptation pays off|1 keeps neighbouring lessons differentiable.
Key ideas
- Thread: Fine-tuning, LoRA, and when adaptation pays off · drill v1 · spin
8747. - Habit: attach a trace_id to every completion you would paste into an ops dashboard.
- Guardrail: add one RACI bullet for prompt or index changes before tomorrow's standup.
Deep dive notebook
Canonical LLMOps lesson — long-form reference material (expanded for production readers; syllabus used only as structural inspiration).
Overview
Why this matters now
Adapt only when prompts and RAG plateau on your private eval set. Teams rarely fail because nobody read a paper—they fail because interfaces between data, models, and humans are underspecified. Use this page as a working document: paste links to your runbooks, ticket templates, and evaluation dashboards (as plain text descriptions if URLs are internal).
Stakeholder translation: If you must explain the same idea to leadership and engineers, prepare two paragraphs: one with outcomes and risk, one with system components and dependencies.
Learning outcomes (detailed)
- Prefer RAG when facts change frequently; prefer adapters for stable style or format.
- LoRA reduces trainable parameters; full fine-tunes cost more and complicate rollback.
- Always compare adapted models against a strong prompt+RAG baseline on the same harness.
Deep dive: applying this in production systems
Start by separating three clocks: model release cadence, data/index refresh cadence, and policy review cadence. When those drift, users experience “correct yesterday, wrong today” behavior even if accuracy metrics look flat. At Northwind Analytics, a common pattern is to snapshot evaluation sets per release and run them automatically against staging before any traffic shift. That sounds bureaucratic until the first time a tokenizer or retrieval change silently shifts answer style in a regulated workflow—then the audit trail pays for itself.
Second, write down interfaces between teams the way you would between services: who owns prompt text, who approves new tools/plugins, who gets paged when refusal rates spike, and where customer complaints land. At Cedarpoint Logistics, the breakthrough was not a better base model—it was a weekly 30-minute review where support brought verbatim failure cases and engineering classified them into “data gap,” “policy gap,” “model limitation,” and “user expectation mismatch.” That taxonomy turned random anecdotes into a prioritized backlog.
Adapt only when prompts and RAG plateau on your private eval set. Use this section as scratch space: paste identifiers (not secrets) from your systems so future readers know which deployment you meant.
Real-world scenario
Setting: You are an applied ML engineer at Harborline Finance. Adapt only when prompts and RAG plateau on your private eval set.
Tension: Leadership wants a public demo in six weeks. Meanwhile, audit questions about data lineage and model updates, and support teams handling escalations need a clear story—not only a model accuracy number.
What good looks like: Decisions are documented (what shipped, what was excluded), failures have owners, and the team can replay an incident with logs and prompts redacted appropriately. This lesson’s ideas apply even if your stack differs; translate nouns (vector DB, gateway, policy engine) to your internal services.
What would you measure first?
Pick one primary metric this week—not ten. Examples: P95 latency for first token, fraction of answers with a cited retrieval span, human escalation rate, or quantum job success rate vs queue depth. At Harborline Finance, the team posted that metric in a shared dashboard with a threshold and a rollback plan when crossed. If you cannot graph it, you are not ready to argue you improved it.
Worked example (adapt freely)
Below is a template you can copy into your notes. Replace placeholders with your environment’s names so the example stays concrete.
# Tiny pattern: separate "what we trust" from "what we generate"
trusted_facts = retrieve(query, top_k=5)
draft = model.generate(query, context=trusted_facts)
return verify(draft, sources=trusted_facts)
Visual reference
Conceptual flow from sources to answers—your stack may add more boxes.
Pitfalls teams actually hit
| Pitfall | Safer habit |
|---|---|
| Assuming one metric tells the whole story | Report slices: region, language, risky intents. |
| Skipping failure drills | Run tabletop exercises for model + infra failures. |
| Unbounded prompts in logs | Redact and set retention; classify sensitive fields. |
Tradeoff lens
| Dimension | Favor left when… | Favor right when… |
|---|---|---|
| Prototype speed | Optimize for learning | Harden for repeatability |
| Model choice | Largest available | Right-sized + eval suite |
| Governance | Ad hoc review | Named owner + calendar |
Mini case study (fictional, composite)
Cedarpoint Logistics ran a six-week pilot. Week 1–2 focused on instrumentation (latency, errors, human escalations). Week 3–4 tightened prompts and retrieval settings. Week 5–6 measured delta against the Week 1 baseline on the same tasks—avoiding “improvement” claims from a cherry-picked demo set. Their postmortem explicitly listed three refused or unsafe requests that surfaced, and how routing changed afterward. Copy that discipline: celebrate wins, but file the near-misses.
FAQ (short)
Q: Where should we start if we have only two weeks?
A: Pick one workflow, one metric, and one rollback story. Expand after you can demonstrate improvement on that slice.
Q: How do we avoid “slide-ware”?
A: Tie every recommendation to an observable: latency, cost, defect rate, or human review load—not generic “best practices.”
These answers are generic on purpose; replace them in your internal wiki with org-specific links.
Practice (from your catalog)
Pick one use case and argue for RAG-only vs LoRA vs full fine-tune in five bullets.
Try the exercise twice: once quickly, once after sleeping on it—often the second pass surfaces edge cases.
Before you close this lesson
| Check | Done |
|---|---|
| Named the single workflow or concept this page helps | ☐ |
| Listed one metric you will watch for two weeks | ☐ |
| Identified who approves changes to prompts/policies | ☐ |
| Captured one “bad outcome” and how you’d detect it early | ☐ |
Closing
Keep this lesson inside Quanta GenAI: add screenshots (as new static assets if your admins allow), links to internal tickets, and names of partners. The goal is not perfection on first read—it is repeatable improvement with evidence.
Bundled reference content for Quanta GenAI Learn. Extend with your organization’s specifics.
Practice
Practice Pair with security on one prompt-injection tabletop scenario. — 1 Bump 18.