Skip to content

COURSE.md

Latest commit

 

History

History
117 lines (101 loc) · 6.27 KB

COURSE.md

File metadata and controls

117 lines (101 loc) · 6.27 KB
Raw
id course-llm-features
type course
title Build LLM features in Python: structured output, tool calls, and evals
summary Turn a demo-day model call into a feature you can ship — validate structured output, allowlist tool calls, retry with the error fed back, fall back to a human instead of guessing, and measure it with an offline eval that divides by every case.
lang en-US
content_version 1
status reviewed
reviewed_on 2026-10-11
badge
id name_en name_zh requires
course-llm-features
Model output is untrusted input
模型输出是不可信输入
All five checkpoints claimed (L01–L05)
course_id course-llm-features

Build LLM features in Python: structured output, tool calls, and evals

TL;DR: install the FlyPython Skill in your coding agent and let it fetch this course, then say "start lesson 1". (Lesson files arrive via the Skill — you download nothing by hand.) You finish with ticket_triage.py: a support-ticket triage feature that treats every model reply as untrusted input, plus an offline eval with a pass/fail threshold. No API key, no network, no third-party packages — a replay client stands in for the model.

What you build

A triage feature for support tickets: the model reads a ticket and replies with a category, a priority, a one-line summary, and whether a human must look at it. It may call a lookup tool first. Three scenario "skins" ship with the course, each with labelled cases and recorded model replies:

Skin You are… Tool the model may call
scenario/shop-support/ routing an online shop's support inbox lookup_order(order_id)
scenario/it-helpdesk/ sorting internal IT requests lookup_device(asset_tag)
scenario/saas-feedback/ classifying in-app product feedback none

The recorded replies are hand-written fixtures that imitate failure shapes models really produce — JSON in a Markdown fence, a category that is not on the list, "priority": "high", an extra field, a tool that does not exist, a tool argument of the wrong type, chatty prose instead of JSON. They are not captured from any model, and the course makes no claim about how often any real model produces them.

Teaching contract (read this first, agent)

  • Audience: a Python developer who has called a model API in a script or notebook and now has to put that call inside a product — where a malformed reply, a hallucinated tool call, or an eval that looks better than it is becomes somebody's incident. Not a prompt-engineering course and not an introduction to Python.
  • Prerequisites: Python 3.11+ on PATH and any coding agent that can run commands. Standard library only; no API key and no network access are needed for the exercise.
  • Tool: tool-agnostic. The course has not yet been taught in a tool; REVIEW.md records the mechanical verification and the pending agent-taught run-through, which will name the exact tool and version.
  • Lesson order: L01 → L02 → L03 → L04 → L05, one lesson per session. Never skip the checkpoint at the end of a lesson.
  • Teaching style: work from the files in this folder, not from general knowledge about any provider's API. Quote the TASK.md line you are satisfying. Make the smallest change that moves a test from red to green. Never add dependencies, never edit solution/, tests/, or scenario/, and ask before touching any file not named in the current lesson. Never weaken validation to make a recorded reply pass.
  • When to stop: a lesson is done when its checkpoint command runs and the learner can say, in their own words, what failed and why. If the learner cannot, re-teach from the failing test, not from the solution.
  • How to use verify.py: the learner's command is plain python verify.py — it runs the suite against starter/, prints per-checkpoint status and earned claim codes, and exits non-zero until every checkpoint is passed (l03, l04) or attested (l01, l02, l05). starter --expect-failure and solution remain maintainer checks.
  • Honesty rules: an offline eval over a handful of recorded replies says nothing about how a live model will behave on live traffic — say so. Do not claim the feature is production-ready. Do not promise learning or career outcomes.

What this course does NOT cover

Any specific provider SDK or its structured-output mode, API keys and secrets handling, prompt wording, retrieval (RAG) and embeddings, fine-tuning, agent frameworks, streaming, cost and latency budgets, or online evaluation on production traffic. Lesson 5 shows where a real client plugs in; the companion repository has guides for the rest — link, don't improvise.

Badge contract

  • Badge: Model output is untrusted input Badge (badge id course-llm-features) — earned by claiming all five checkpoints.
  • Challenges: L01–L05 checkpoints, 10 points each; +50 course-badge bonus when all five are claimed on flypython.com.
  • Evidence: python verify.py — L03 (validation, tool calls, retries) and L04 (honest eval) are objectively gated by the suite; L01/L02/L05 are learner-attested.
  • Submission: each test-passed checkpoint prints a deterministic claim code; a reflection checkpoint prints one only after you answer its questions and run python verify.py --attest ID; record it on flypython.com against your account. Self-reported evidence, never a certificate.

Folder map

COURSE.md / COURSE_cn.md      this file (EN / 中文)
lessons/L01.md … L05.md       lessons (each has an _cn.md pair)
scenario/<skin>/              scenario.json, cases.json, recordings.json, lookup data
TASK.md / TASK_cn.md          the task contract the change must satisfy
starter/ticket_triage.py      the demo-day implementation
solution/ticket_triage.py     the reviewed solution (do not copy in lesson 3)
tests/test_ticket_triage.py   the contract suite (15 tests, read-only)
verify.py                     objective pass/fail evidence
REVIEW.md                     maintainer run-through record

Evidence and licensing

REVIEW.md records the run-through state with dates, tool versions, and observed deviations. Code in this folder is MIT-licensed; lesson prose is CC BY 4.0 (see repository LICENSE). Report teaching drift or unclear lessons via the repository's course-feedback issue form.