Module 5 Lab: The Verification Capstone

Download the starter pack (zip) Module 5 slides (PDF) Module 5 notes

This lab has no new dataset and no new pipeline to build. You take a pipeline you already have, either Module 3’s EDGAR text panel or Module 4’s WRDS (or public-fallback) funda-style pipeline, and harden it against the rubric below. The goal is a repo that a coauthor, a referee or you a year from now could pick up cold and trust. Everything you need is in lab5_starter.zip: the rubric reproduced below (you grade yourself against it), an audit-command card, DECISIONS.md/LOG.md templates, a container quick-start for the environment stretch, a small worked example of the results.tex / \input{} pattern, and a one-page git handout with a .gitignore starter and the rollback commands. The starter pack contains no data. You point it at a pipeline you built earlier in this course.

What you’re building in this module

By the end of the lab you will have: one existing pipeline (Module 3 or Module 4) rerun clean from a README alone, with a git history of meaningful, checkpointed commits; a DECISIONS.md, LOG.md and attrition table that you have cross-checked yourself against the commit log, with one specific claim checked against the diff (it holds up or it doesn’t); one number traced end-to-end from raw data through a results.tex file to a rendered output, with nothing typed by hand along the way; a clean credential audit with the output of both checks pasted into your notes; one banal writing artifact you delegated and one meaningful artifact you wrote yourself; and, if you have time, a rerun of the same pipeline inside a container, with pasted output showing that the sandbox blocked access outside the project folder.

Pick your pipeline

Choose Module 3’s EDGAR text pipeline (CIK lookup → filing extraction → DuckDB store → keyword or section-level result) or Module 4’s WRDS funda pipeline (or its public-FSDS fallback, if you were on that track) — whichever is in better shape right now. You are not building anything new. You are making an existing pipeline defensible. If neither pipeline runs, spend the first ten minutes getting one back to the point where it has run once, then start hardening it. The rubric assumes a pipeline that already works; it does not cover rebuilding a broken one.

The capstone rubric

You grade yourself against this rubric at the end of the lab. When I run the course with a group, I also spot-check a few projects live in the debrief. It has four sections. The fourth is a stretch, not a requirement.

Reproducibility

The pipeline reruns start-to-finish from your README’s instructions alone, in a clean clone of the repository (a fresh folder, not your working directory). No step may depend on something you remember that the README doesn’t say.
Git history shows meaningful commits at every real checkpoint (download, clean, measure, figure/table), each with a message that says why, not just what. One giant commit for the whole pipeline does not pass.

Verification

DECISIONS.md, LOG.md, and a sample-attrition table are all present, and you have checked them against the commit history instead of assuming they are accurate. Pick one specific claim from DECISIONS.md, find the commit (or commits) it should correspond to, and write down in your notes whether the diff supports the claim.
Two sanity checks are recorded with sources: one benchmark comparison (a number checked against something Claude didn’t touch) and one plausibility check (does the pattern make sense given what you know about the setting).
One number is traced end-to-end: from the raw data, through the script that computes it, into a results.tex file, and into a rendered output, with nothing typed by hand anywhere along that chain. Write down which number you traced and the file path at each stage.

Hygiene

Both credential-audit checks from audit_commands.md are run against this repository: the search for committed credential files (git log --all --full-history --oneline -- '*.env' '*.env.*' '.mcp.json') and the search for your actual password in every commit and every uncommitted file. Both come back empty, or the credential has been changed and removed from history. Paste both outputs into your lab notes; describing them as “clean” is not enough.
.gitignore is correct: credentials, raw licensed data, and any WRDS-derived files are excluded, and git log --all --full-history --oneline -- 'data/raw/*' '*.parquet' '*.dta' '*.csv' (adjusted to your own data paths) returns nothing. Paste that output too.
If your pipeline used WRDS, you have confirmed the Duo/SSH tunnel is closed.
One banal writing artifact was delegated to Claude (for example, a variable-definition table generated directly from your metadata table) and one meaningful artifact was written by hand: a three-sentence note stating your pipeline’s contribution or a specific limitation, in your own words, that you could defend without the document in front of you.

Environment (stretch)

Rerun the pipeline, or a meaningful part of it, inside a Docker dev container, agent-safehouse or Claude Code’s built-in /sandbox, and paste the output showing blocked access: the command you ran that should fail outside the project folder, and the output confirming it did (“Operation not permitted”, or “No such file or directory” in a dev container; see container_quickstart.md).

Verify this: grade yourself honestly, then spot-check yourself

Go through the rubric above and mark each line pass or fail based on what is in your repository, not what you intended to do. Then run the spot-check on yourself. Open DECISIONS.md, take the third entry (not the one you already know holds), and check it against its commit in under five minutes, with no preparation. If a colleague can look at your repository, let them pick the entry instead. Mark a line “pass” only if it would hold up under that check.

Environment stretch: quick-start

If you’re attempting the container stretch, lab_starter/container_quickstart.md gives the exact commands for three options: a Docker dev container, agent-safehouse (lighter, Mac-native) and Claude Code’s built-in /sandbox (macOS and Linux). Speed doesn’t matter here. The goal is one concrete piece of evidence that the sandbox blocks access, the same kind shown on the module page (cat ~/.ssh/id_ed25519 failing with “Operation not permitted” from inside the sandbox). The full deliverable for this section is a pipeline that runs the same way inside the container, plus one blocked-access command that fails as it should. Don’t try to move all your course work into the container.

Optional drills: referee and response letter

If you finish the rubric with time left, try one of these on your own pipeline. Each one applies a section of the module page to your own work.

Clean-room referee. Write a two-paragraph sample-construction description for your pipeline, as it would appear in a paper. Copy that text and your attrition table into an empty folder, add the deny rule from the module page’s clean-room section, and start a fresh session there. Ask it to write a referee report to review/ checking whether the stated filters can produce your final N. Note one thing it caught that you had stopped seeing.
Response-letter audit. Write a short mock response letter (three or four claims) describing changes you actually made to your pipeline today, plus one change you did not make. Ask Claude to build the three-column audit table against your repository and leave cells empty rather than guess. Did it find the change you never made? Use only your own pipeline and a mock letter here, never a real referee report or a confidential review.

Self-check and exit ticket

Do the DECISIONS.md check from the callout above on one claim, if you haven’t already, and notice what is easy to verify and what isn’t. The difference is the verification-debt 2×2 from the module page, applied to your own work instead of a hypothetical one. When the course runs with a group, this is a live debrief: I pull up three or four repositories and we audit one claim together in front of the room.

Then write down one banal task you will always delegate and one meaningful task you will never let an agent start. This is the delegation rule from the module’s writing framework. It also answers the coauthor-handoff question: what would a real coauthor need to see in this repository before trusting it without redoing your work?

Complete the standard three-question exit ticket (one concept that clicked, one thing that broke, one thing you’d want to verify before trusting a similar result), plus this module’s specific item: your one-sentence delegation rule, in writing.

Module 6 readings: read them before you start Module 6

Module 6 rests on six short papers from 2025–2026: Bastani et al. on learning, Budzyń et al. on expert deskilling, Brodeur et al. on AI-assisted reproduction, Lu et al. on an AI system that writes papers, and Kusumegi et al. and Hao et al. on what AI does to scientific output. There are also accounting-specific swap options. The full citations, and which sections of each to read, are on the Module 6 page. Plan on about 90 minutes in total. Read them before you start Module 6, because its three questions assume you have.

When the course runs with a group, debate positions are assigned at this point. Working alone, pick the side of each Module 6 question you find least convincing, and write a paragraph making its strongest case before you start the module.