Verification
Every module in this course asks you to check something before you trust it, and the checks get more demanding as the course goes on — not because Module 1’s habits stop mattering, but because a full pipeline has more places for an error to hide than a single script does. This page collects the whole protocol in one place: the three mantras that sit underneath everything, the course’s escalating verification load module by module, and the verification-debt framework that explains why the load has to escalate at all. Print it; it’s meant to work as a protocol card, not just a reading.
The three mantras
These three sentences, introduced in Module 1, carry the entire course.
- Files persist; context doesn’t. Anything you need to survive past this session has to be written down — a decision, a row count, a variable definition. A conversation evaporates at the end of the session; a file on disk doesn’t.
- Trust, but verify — same as an RA. Delegate freely. Sign off only after you’ve checked something specific — one number, one benchmark, one hand-read filing — against a source the agent didn’t touch.
- Be specific: file, operation, aggregation, output. Name all four, every time. A vague request doesn’t fail loudly; it produces something plausible-looking that isn’t quite what you meant, and the gap compounds the longer it goes unchecked.
Verification escalates, module by module
Each module adds a verification habit on top of the ones before it. None of them retire — Module 5’s protocol includes Module 1’s one-number check, it just isn’t the only thing you’re doing anymore.
Module 1 — one number, one independent source. Before accepting any computed output — a row count, a ratio, a figure — pick one value you can check against a source the agent did not touch, and check it by hand. It’s the one step in the lab the agent cannot do for you.
Module 2 — two sanity checks, named in advance. A benchmark comparison (does the result match a pattern you already have a strong prior about — pharma and software should show higher R&D intensity than utilities or retail) and a plausibility check tied to a known event (does a series show some signature around a real shock, or does a suspiciously smooth line suggest over-aggregation). Name both checks before the figure renders, not after, so you aren’t reasoning backward from whatever it happens to show.
Module 3 — a quality report, plus the audit-sample protocol. After any extraction pass, read the quality report on the failures rather than treating a headline success rate as the finding — a 99%-extracted claim hides a selection problem if the missing 1% is systematically different from the rest. Before a text-based measure goes into a paper’s main table, run a stratified human audit: fifty filings spanning high/middle/low values of the measure, across firm size and industry, coded against a rubric decided in advance, with an inter-rater agreement statistic reported if more than one person codes it — or a plain statement that only one person did, if that’s the truth.
Module 4 — four inspection checks, plus the iron laws. After every WRDS pull, before telling anyone the extract is good: a row count checked against an expectation you wrote down beforehand, a .head()/.sample() eyeball of actual values, a null check on the columns you plan to use, and a date-range check against what the query should have returned. Alongside that: the four Compustat iron-law filters (indfmt = 'INDL', datafmt = 'STD', popsrc = 'D', consol = 'C') applied before you ever look at a row, and the reminder that CRSP’s share-type and exchange filters are never applied automatically — that’s a manual step, every time, no exceptions for “a quick test.”
Module 5 — the full paper trail, cross-checked against commits. A DECISIONS.md, a LOG.md, and a sample-attrition table, all requested up front rather than reconstructed afterward — and then treated as a hypothesis to check, not a fact to file away. An LLM will write a documentation entry claiming work was done that never happened; the fix is picking one specific claim, finding the commit it should correspond to, and confirming the diff actually does what the prose says. Layered on top: the two-command credential audit (git log --all --full-history -- .env and a grep for any credential string, both of which should return nothing) and the no-hand-typed-numbers rule — every number in a manuscript comes from \input{}-ing a file a script generated, never from a person copying a terminal output into a draft.
Verification debt
Module 5 names the underlying reason the load has to escalate. Every task has two costs: the cost of making it, and the cost of verifying it’s correct. Before agentic tools, those costs were often correlated — something expensive to build had usually been checked along the way, as a side effect of how slowly it was built. Agentic coding breaks that correlation: a keyword-based text measure or a regression table can now be made in minutes, while the cost of verifying it hasn’t fallen at anywhere near the same rate. The gap between those two costs is verification debt.
The safe zone is expensive-to-make, expensive-to-verify — the slowness of production forces verification to happen along the way. The danger zone is the opposite corner: cheap to make, costly to verify, which is where an AI-drafted regression table or a decade of keyword counts over 10-K text tends to land. Autonomy makes this worse, not better: the more freedom an agent has to work unsupervised, the faster debt accumulates, because the verification burden that would have been distributed across many small checkpoints instead arrives all at once, as a finished-looking repo. Approving each step manually doesn’t fix this by itself — approving a step without understanding what it did is a rubber stamp with extra clicks. The real lever is how much you actually internalize and check as the work happens.
This page is meant to be printed and kept next to your desk during a build module — the three mantras and the module-by-module checklist above are the protocol card referenced from every module in this course.