Module 3 Lab: EDGAR Pipeline, Validity Protocol, First Skill

Download the starter pack (zip) Module 3 slides (PDF) Module 3 notes

This lab has three parts, and each one builds on the last. First you build a real 10-K text pipeline end to end. Then you stop and write (prose, not code) a validity protocol for the measure you just built. Finally, you convert the whole pipeline into your first SKILL.md. The starter pack (lab3_starter.zip) contains a pinned list of twenty CIKs across four industry groups, a front-loaded prompt template, a SKILL.md skeleton with anatomy comments already in place, and this lab’s verify checklist. No filing data ships with the starter pack. EDGAR is public, so you fetch everything live, the same way you would for a real project.

NoteWhat you’re building in this module

By the end of the lab you will have three things. The first is a DuckDB file (filings, sections, keywords tables) built from real 10-K Risk Factors sections across roughly twenty firms and three fiscal years, with a cache directory, a quality report, and an extraction rate of at least 95 percent with every failure diagnosed in writing. The second is a short, written audit-sample protocol for the keyword measure you computed. It does not claim the measure is valid; it is the concrete plan you would run to check it. The third is a working edgar-panel/SKILL.md that fires correctly in a fresh session and reproduces the pipeline on two firms it has never seen.

Part A: Pipeline (45 min)

Start from an empty project folder, as in every lab in this course. Open the front-loaded prompt template from the starter pack and fill in the pieces specific to your run: the topic you’ll measure (pick one from the menu below), the fiscal years (2021–2023 is a reasonable default), and the output paths. The template already states the operational facts from this morning’s lecture: the CIK list, the rate limit, the User-Agent requirement, and the cache path. You should not need to work any of them out again. If you find yourself typing a fact the template already states, stop and fill in the template instead of writing a fresh prompt from scratch.

Pick one topic from the menu for your keyword measure: tariffs (trade policy exposure), cyber (cybersecurity and data-breach risk), climate (climate and physical-risk disclosure), or supply chain (supply-chain disruption risk). Any of the four works for this module’s pipeline. Pick the one you would most plausibly use in your own research, since you will read the actual sentences in Part B.

Build the pipeline in the order the lecture described: CIK resolution and filing-index retrieval for the pinned firm list, cached raw-HTML download, Item 1A section extraction with a fallback chain of regex patterns, a load into DuckDB, and then one result table with your keyword measure by firm-fiscal-year. Three requirements are mandatory. First, the cache must prevent re-downloading a filing that already succeeded. Test this by re-running the pipeline once and confirming zero new network requests for filings already in the cache. Second, the pipeline must produce a quality report that states the extraction rate and gives a one-line diagnosis of what went wrong for every failure (a real edge case, a regex gap, a malformed filing). Third, the extraction rate must reach at least 95 percent before you move to Part B. If the rate falls short, you are seeing the morning’s central lesson in your own data. Read the quality report, ask Claude why the remaining filings are missing, and fix what you can before you accept the rest as real edge cases.

WarningCommon failure: accepting a low extraction rate without reading why

The 95 percent floor in Part A still applies. On the way there, an 80 percent rate is not automatically a problem, and a 99 percent rate is not automatically fine. What matters is whether you can say, for each failure, what went wrong. The mistake to avoid is treating the percentage as the finding and moving on. If your quality report just says “16 of 20 succeeded” with no diagnosis attached, you do not have a quality report yet. You have a number. Ask Claude to open the cached HTML for each failure and report how its structure differs before you decide which failures are fixable and which are real edge cases.

Part B: Validity check (15 min)

This part is short, and you write it rather than code it. The judgment here is the one piece of this lab you cannot delegate, and the goal is to practice stating that judgment explicitly, not to produce more code. Pick one firm from your panel and compute the year-over-year textual similarity (Jaccard or cosine, whichever your pipeline already supports) between its Risk Factors section this year and the same section last year. Read both sections side by side. Classify what you see. Is the year-over-year change in your keyword count driven by boilerplate (largely unchanged language, a high similarity score, a keyword count that moved because a paragraph got slightly longer)? Or is it driven by a meaningful rewrite, where the similarity score is lower and the keyword signal comes from the new language?

Then write, in three sentences, the audit-sample protocol you would run before this measure went into a real paper’s main table: which fifty filings you would pull and how you’d stratify the sample (by measure value, firm size, and industry, not a convenience sample of whichever firms are easiest to read), who would code them and against what rubric decided in advance, and what you’d report if more than one person coded any part of the sample. The deliverable is the protocol, not a claim that your measure is valid. Fifty filings are not enough to prove validity at any real scale. Writing “this measure is valid” without having run the audit is the validity failure the lecture warned about.

Part C: The skill (20 min)

Convert the pipeline you just built into edgar-panel/SKILL.md, installed at project scope (.claude/skills/) rather than globally. The pipeline is tied to this project’s data, and at project scope a coauthor who clones your repository gets the skill automatically. Use the five-step workflow from lecture: notice the repetition (you just wrote this procedure out once already, in Part A’s prompt), name the inputs (CIK list, form type, fiscal-year range), design the output structure (DuckDB file, metadata.csv, quality report) before writing the skill body, let Claude draft the SKILL.md from a short spec covering those three things, then test-fire it.

Do not skip the test-fire. Open a fresh session, not the one you built the skill in, and confirm the skill triggers on a natural request (something like “build an EDGAR panel for these two CIKs”) without you naming the skill file. Then run it on two CIKs not in your pinned list and confirm it reproduces the same output structure. If it does not fire, check the likely cause from lecture: does your description field read as a trigger condition or as a title? If it reads as a title, rewrite it.

Once your skill fires correctly, install one or two skills from the course’s Barrios Skills catalog. latex-tables and stata-regression are good choices whatever your primary language. Then read one mature SKILL.md from the catalog to see what a skill looks like after several rounds of revision. Look at how its description is phrased and how its steps are structured. That is the standard your own skill is working toward, even if this module’s version does not reach it yet.

Stretch, if you finish early: build a second skill, paper-reader, from paper_reader_SKILL_skeleton.md in the starter pack. Test it in a fresh session on one paper from your own reading list. Then check every number in its notes (each N and each coefficient) against the PDF’s tables, and write down how many were wrong.

TipVerify this: your full checklist

Work through every line below before you consider the lab complete. This is a self-graded, pass/fail checklist. The instructor will spot-check a sample rather than grade every box.

Debrief and exit ticket (2:45–3:05)

Come back together for a short debrief. Be ready to share, informally, where your extraction rate landed and whether your failures were real edge cases or regex gaps a better fallback chain could have caught earlier. The instructor will read two or three student SKILL.md descriptions aloud, without naming whose they are, and ask the room to vote on whether each one would trigger on a natural request. Seeing the title-versus-trigger difference in someone else’s description makes it easier to catch in your own.

Complete the standard three-question exit ticket (one concept that clicked, one thing that broke, one thing you’d want to verify before trusting a similar result), plus this module’s item: what would your second skill be? Name one other recurring procedure in your own research workflow that you would convert next, and describe its inputs and output structure.