Module 4: WRDS MCP, Scale, and the Stata Bridge
Module 3 ended with a CIK and a question hanging in the air: the filing text is parsed, the entity is identified, and the obvious next move is to attach fundamentals — leverage, profitability, size — to every firm-year in the panel. Those fundamentals live in Compustat and CRSP, and they are not sitting in a public bulk file the way EDGAR text is. They sit behind three barriers at once: authentication (a licensed WRDS account, MFA-gated), licensing (a subscription your institution pays for, not yours to redistribute), and scale (Compustat’s full annual file runs to hundreds of variables and decades of history — not something you read into a chat window). A plain chat tool cannot clear any of the three. This module’s stack clears all three, in order, and this module is built around watching that happen on a real account with real data.
The shape of the module mirrors the shape of the problem. An MCP server solves access: the protocol handles WRDS authentication so your credentials never enter the conversation with Claude, no matter how many queries you run. Iron-law filters and mandatory inspection solve correctness: a query that runs without an error is not the same thing as a query that returns the right rows, and Compustat in particular will hand you plausible-looking garbage if you skip a filter. Parquet, DuckDB, and a metadata table solve scale and memory: the data-layer version of Module 1’s central fact that files persist and context doesn’t. The payoff is the moment a plain-English request turns into a correctly filtered funda extract landing on your own account. The warning that goes with it: this exact moment — the query worked, it feels solved — is precisely when trust is most dangerous.
What an MCP actually is
Model Context Protocol is a fancy name for a simple idea: instead of you writing and maintaining the plumbing that connects an AI agent to an external service — the connection handling, the authentication, the query submission, the error translation — a small dedicated server does that job, and the agent talks to the server using a standard protocol instead of talking to the service directly. wrds-mcp is exactly this for WRDS. It is a Python package that runs as a local process, exposes a fixed set of named tools (wrds_list_libraries, wrds_describe_table, wrds_run_sql, wrds_download_data, and a handful of pre-built helpers for CRSP, Compustat, and the CRSP–Compustat merge, nine tools in total), and reads your WRDS credentials from the environment — never from a line of code, never from the conversation.
The comparison worth holding in your head is a hand-rolled psycopg2 script that opens a direct Postgres connection to WRDS. That script is a completely reasonable thing to write, and plenty of researchers do exactly that. The difference is what it costs you every time you want Claude to use it: you’d have to paste a password into a script, or into an environment variable Claude reads and could in principle echo back into a response, and you’d be responsible for writing the SSH/Duo handling yourself, from scratch, correctly, under time pressure. The protocol framing inverts that. The MCP server is the only thing that ever touches your WRDS password. Claude issues a plain-language request; the MCP server translates it into a parameterized SQL call, or invokes one of its pre-built helpers, and returns rows. Your credentials never enter the prompt or the model’s context window, because nothing in the interaction requires them to.
That sentence — the protocol carries authentication for you, so your credentials never appear in the conversation — is the single most important thing to remember about MCP going into the rest of this module.
This also finishes a thread from Module 1’s tool ladder. Claude Code without any MCP servers sits at the fourth rung: filesystem access and local execution. Wiring it to an external, authenticated service through a protocol like this is the fifth rung, and everything in this module happens on that rung.
The architecture, one hop at a time
The full chain has four links, and the lecture spends real time on this figure because getting the mental model right here prevents a specific, common confusion later: thinking that “Claude has my WRDS password” because Claude can pull WRDS data. It does not, at any point.
Link one: you, typing a plain-language request into Claude Code. You never type a password here. You type something like “pull annual Compustat fundamentals for industrials, 2000–2023, gvkey/fyear/sale/at/dltt/ni.”
Link two: the wrds-mcp server, a local process running from ~/.wrds-mcp-env/bin/wrds-mcp. It receives the request over the MCP protocol (stdio, JSON-RPC — mechanics you don’t need to memorize, just know it’s a defined local channel, not the open internet). It reads WRDS_USERNAME, WRDS_PASSWORD, and WRDS_TUNNEL_PORT from environment variables that live in a file on your disk (~/.wrds-mcp.env) — never hardcoded, never printed, never part of the conversation Claude sees. Internally it uses SQLAlchemy and psycopg2 to build and issue the actual SQL.
Link three: the persistent Duo tunnel, a background daemon (tunnel_daemon.py) that uses paramiko — a pure-Python SSH library — to open one keyboard-interactive SSH session to the WRDS-cloud bastion, approve a single Duo push, and hold that session open indefinitely. This is the detail worth over-emphasizing: WRDS’s bastion only accepts keyboard-interactive authentication, which is how Duo gets delivered, and sshpass cannot do keyboard-interactive auth — it simply doesn’t support it. If you see an old doc telling you to brew install sshpass, ignore it; paramiko is already a dependency of the package and handles this correctly. The daemon listens on local port 49600 and forwards everything on 127.0.0.1:49600 through the SSH tunnel to WRDS’s Postgres endpoint. You approve Duo exactly once, at tunnel start, and every query for the rest of the session — dozens, if Claude is working through a multi-step pull — reuses that one approved connection.
Link four: WRDS PostgreSQL itself, a database cluster with schemas like comp (Compustat), crsp, tr_insiders, trace_enhanced, and more, gated by your institution’s subscription.
The consequence worth stating plainly, because it’s the thing students most often get backwards: sandboxing the agent’s execution environment (a Docker container, a restricted shell) controls what Claude can do. It does nothing to control what Claude can see once data is in front of it — a mounted file is a mounted file regardless of what container it sits in. What actually keeps your WRDS password out of Claude’s hands is that it never enters the chain at the point where Claude operates: it lives in an environment file the MCP server reads directly, one link below where Claude’s language-level reasoning happens. That distinction — access control by architecture, not by sandbox — is the thing to take out of this segment, and it comes back with more teeth in the governance section below.
Query results themselves land on your disk in ~/wrds_data by default (the WRDS_DOWNLOAD_DIR environment variable, if you ever need to point it elsewhere) — not in Claude’s context, not printed into the transcript in full. Claude sees a summary — a shape, a .head(), a row count — the same discipline Module 1 taught you to apply to any large file.
Ritual and hygiene: the boring parts that save you
Everything above only works if a short list of unglamorous habits happens in order, every session, without exception. None of this is intellectually hard. All of it gets skipped under time pressure, which is why it’s worth over-drilling in this module rather than assuming it’ll stick from a single read of the setup page.
Install wrds-mcp into its own virtual environment — never your system Python, never a shared environment used for something else — with python3 -m venv ~/.wrds-mcp-env followed by pip install -e . from inside the package directory. Configure credentials through a private env file, never hardcoded in a script and never typed directly into a prompt: copy .env.example to ~/.wrds-mcp.env, fill in your own username and password, and set that file’s permissions so only you can read it (chmod 600). If you’re working inside a git-tracked project directory at all — and by now you probably are — write your .gitignore entry for any credential file before you create the file it’s supposed to exclude, not after. A .gitignore written after the fact only protects you if you remember to write it before your next git add .; a .gitignore written first protects you by default.
Then the tunnel ritual, which has a strict order: up, work, teardown. Source your env file, run tunnel_up.sh, approve the single Duo push within about thirty seconds, confirm the daemon reports it’s listening on port 49600, and only then open Claude Code or issue your first query. At the end of the session — every session, not just the ones where you remember — run tunnel_down.sh. This last step is the one everyone skips, because nothing visibly breaks if you don’t do it; the tunnel just sits open until you close your laptop or it eventually drops on its own. Skipping teardown isn’t catastrophic the way skipping an iron-law filter is, but it’s sloppy in a way that compounds: an open tunnel is an open authenticated session to a licensed institutional resource, sitting there for no reason. This module’s verify checklist puts tunnel teardown as its literal last item, on purpose, so that it’s the last thing you check, not something you remember three days later you forgot.
Governance is not sandboxing
It’s worth stating the distinction from the architecture section as its own idea, because it generalizes well past WRDS and this course returns to it, in more depth, in Module 5. Running an agent inside a restricted execution environment — a container, a locked-down shell, a permissions-limited account — constrains what that agent can do: which commands it can run, which processes it can spawn, which parts of the filesystem it can write to. It does essentially nothing to constrain what the agent can see once data has been placed in front of it. A CSV mounted into a sandboxed container is just as readable by an agent inside that sandbox as one sitting on your open desktop; the sandbox boundary is about action, not visibility.
That distinction is exactly why the course’s data-handling rule for WRDS is architectural rather than a matter of Claude’s good behavior: nothing about Claude’s judgment is what keeps your password safe, the fact that the password never enters the conversational context in the first place is what keeps it safe. The same logic governs what you’re allowed to do with WRDS data, as opposed to WRDS credentials, and here the constraint is legal, not architectural: your WRDS subscription is a license held by your institution, and it grants you, personally, the right to query and use the data for your own research — it does not grant you the right to redistribute extracts, post them, commit them to a shared or public repository, or hand them to someone without their own valid WRDS access. This course’s hard rule follows directly: zero WRDS data distribution, anywhere, ever — not in a lab starter zip, not in this website, not in a shared Slack channel. Every pull in this module’s live track happens on your own account, lands in your own ~/wrds_data, and stays there. The fallback track you’ll meet in the lab is not a workaround for this rule; it’s a completely separate, 100% public data source, built precisely so that nothing WRDS-derived ever needs to circulate for the class to function.
The iron laws: four filters, mandatory every time
In Module 3 you learned the anatomy of a skill — a packaged, reusable procedure with a checklist a session can load automatically. In this module you see one used for real enforcement rather than as an abstraction: the WRDS filter skill in this course’s skills catalog encodes the four filters below as a machine-checkable list that ships with every WRDS session, so that “I forgot the filter” stops being a plausible excuse. This is that skill anatomy doing actual work, not a new concept to learn on top of it.
Pulling comp.funda or comp.fundq — Compustat’s annual and quarterly fundamentals tables — without all four of the following filters does not produce an error. It produces a query that runs cleanly and returns rows that look entirely reasonable, which is exactly what makes the omission dangerous.
WHERE indfmt = 'INDL' -- industrial format only
AND datafmt = 'STD' -- standardized data
AND popsrc = 'D' -- domestic population source
AND consol = 'C' -- consolidated statements onlyDrop indfmt = 'INDL' and your “industrial firm” sample silently absorbs financial companies reported under the financial-services format — banks and insurers with balance sheet structures that will distort any leverage or asset-turnover measure you compute across the pooled sample. Drop datafmt = 'STD' and you mix Compustat’s standardized figures with as-reported figures that use different accounting treatments for the same line item, so the same variable name no longer means the same thing across rows. Drop popsrc = 'D' and non-domestic population sources creep into a sample you meant to restrict to U.S. filers. Drop consol = 'C' and unconsolidated subsidiary statements appear as separate rows alongside their parent’s consolidated statement — an instant, silent duplication that will not show up as an error but will show up as wrong regression coefficients three steps downstream, at which point it’s much harder to trace back to its source.
CRSP works differently and it’s worth knowing this before you hit it: CRSP’s filters are not applied automatically by any tool in this stack, including the wrds_get_crsp_returns helper — the helper joins the returns file to the names file on permno and a name-date range and stops there. Share-type and exchange restrictions (the modern CRSP v2 equivalent of the old shrcd IN (10, 11) convention) are something you apply yourself, as a filter step after the pull, every time. There is no iron-law shortcut here — just the discipline of knowing this is a manual step and not assuming a pre-built helper has done it for you.
The broader habit both cases teach is the same one: identify which filters apply, validate that your query includes them before you execute, execute, then inspect — in that order, every time, no exceptions for “a quick test.” A quick test that skips a filter and happens to look fine is not a validated shortcut; it’s a coin flip that came up heads once.
Inspect before you trust: the Rationalization Table
A query that executes without throwing an error is not the same claim as a query that returned the correct data, and the gap between those two claims is where this module’s discipline lives. Four checks are mandatory after every WRDS pull, before you tell anyone — including yourself — that the extract is good: a row count, checked against what you expected rather than just glanced at; a .head() or .sample() eyeball of the actual values, not just the column names; a null check on the columns you plan to use; and a date-range check (min/max of fyear or datadate) against what the query should have returned. None of these is difficult. All four together take under a minute. Skipping them anyway is the single most common way a bad extract becomes a published number.
The instructor will project a small prop during lecture worth carrying with you afterward — call it the Rationalization Table, a short list of the sentences people say to themselves right before they ship a number they didn’t actually check:
| What you tell yourself | Why it’s a rationalization, not a check |
|---|---|
| “It ran without an error.” | Absence of an error means the SQL was syntactically valid — nothing about correctness, filters, or joins. |
| “The row count looks about right.” | “Looks about right” is not a number you compared to an expectation you wrote down before running the query. |
| “I’ve pulled this table before, it’s probably fine.” | Prior experience with a table doesn’t verify this query, on this date range, with these filters. |
| “The query worked, so it’s correct.” | The most dangerous one on this list — it collapses “executed successfully” and “returned correct data” into a single claim, when they are two entirely separate claims that happen to feel the same in the moment. |
That last row is the one to actually remember. Claiming a result is correct because the query didn’t error is not a smaller version of verification — it’s a different activity entirely that happens to look like verification from the outside. The standard this course holds you to, in this module and every module after, is the same one Module 1 introduced for any agent-produced output: trust the process enough to delegate the pull, but verify the result — actual numbers, written down — before you build the next step on top of it.
Scale discipline: filter first, never SELECT *
Compustat’s full annual file runs to several hundred variables across decades of history for tens of thousands of firms; the full CRSP daily file is bigger still. Neither is a file you should ever try to load whole into a pandas DataFrame, and neither is a file you should ever ask Claude to read directly into the conversation the way Module 1 warned against for any large CSV — the same context-window mechanics apply here, just with a data source that makes the mistake far more expensive when you make it.
Two rules, and they work together. First: filter at the database, not in pandas — a WHERE clause that restricts by fyear, by the iron-law filters, and by an explicit list of gvkeys or a SIC-code range does the work at the source, before a single row crosses the wire. Second: never SELECT *, ever, in a script that touches a WRDS table — name the columns you actually need. SELECT * on a table with hundreds of columns is not a convenience; it’s an unforced, unbounded cost paid on every single query, and it’s also a signal, when you see it in someone else’s script, that they didn’t stop to think about what they actually needed.
Once a filtered, column-limited extract lands on disk, convert it to Parquet rather than leaving it as CSV. Parquet is column-oriented and compressed, which matters because a wide, mostly-numeric fundamentals table is the ideal case for it: a CSV export of a filtered Compustat extract that ran to roughly 1.7 GB compressed down to about 114 MB as Parquet in this course’s own test conversion — numbers you can reproduce yourself on the public fallback sample, not a vendor claim to take on faith. Query the Parquet file with DuckDB, which evaluates lazily: it reads only the columns and row groups a given query actually touches, rather than materializing the whole file in memory first. The practical effect for you is that a SELECT gvkey, fyear, sale, at FROM funda WHERE fyear >= 2000 against a Parquet file runs fast and cheap, in a way the same query against a raw CSV loaded fully into pandas simply does not.
Metadata as context engineering
This is the data-layer version of Module 1’s first mantra — files persist, context doesn’t — applied specifically to the problem of Compustat’s famously opaque variable names. dltt, che, csho, prcc_f: none of these are self-explanatory, and a fresh Claude session, or a coauthor opening your project for the first time, has no way to know that dltt means long-term debt in millions of dollars unless something on disk tells it so.
The fix is a metadata table, built once and queried forever after: a small DuckDB (or Parquet) table with one row per variable you use, documenting its name, a plain-language label, its units, and a screening note — the kind of thing that would otherwise live only in your head or in a comment you forgot to write. SELECT * FROM compustat_metadata WHERE varname = 'dltt' should return something like “long-term debt, total, in millions of dollars, screen: > 0 only where you’re computing a leverage ratio that requires positive debt” — a sentence a fresh session can read and act on immediately, the same way it would read a CLAUDE.md at the start of a project.
The demo makes this concrete in a way that’s worth watching for, not just reading about: after building the metadata table, the instructor closes the session entirely and opens a brand-new one, then orients that fresh session with nothing but SELECT * FROM metadata. No re-explanation, no re-pasted variable list — the table on disk does the entire job of bringing a session with zero memory of the prior conversation up to speed. That’s the whole point of writing state to files instead of relying on context: the metadata table survives the two things a conversation does not — compaction and a new session — and it survives handing the project to a coauthor who never saw your original prompts at all.
Feasibility-assessment prompting: ask before you build
The last habit of the module is a prompt pattern, and it’s worth learning it by its exact wording rather than a paraphrase, because the specific phrasing is what reliably gets Claude to surface blockers before committing to a build rather than discovering them mid-pull:
“I want a firm-year panel with leverage and ROA for US industrials 2000–2023 from funda. Tell me what’s involved before you try anything complicated. What might be missing?”
Notice what this prompt does not ask for: it does not ask Claude to write the query yet. It asks Claude to reason about the task first — which filters apply, which variables are needed, where the CRSP–Compustat link might introduce duplicate rows, whether the date range crosses a known Compustat data-format transition — and report back on what it finds before any SQL gets executed. In the live demo, this exact prompt surfaces the filter question and the linking-table caveat before a single row is pulled, turning what would otherwise be a mid-pull discovery into an upfront one. The pattern generalizes past this module: any time a task is nontrivial enough that surprises are plausible — a classification task, a linkage across datasets, anything with more than one moving part — asking “tell me what’s involved before you try anything complicated” is cheaper than finding out the hard way three steps into a build that has to be unwound.
Where this module leaves you
By the end of the live demo you will have watched a plain-English request become a correctly filtered funda extract, inspected with real numbers, converted to Parquet, queried lazily in DuckDB, and documented in a metadata table that a fresh session can read cold — on the instructor’s own account, with the tunnel torn down on camera at the end. The lab gives you the same pipeline on your own account if your WRDS/Duo runway is clear, or on an equivalent public dataset if it isn’t; either way, the skills you leave with are identical, because the discipline — not the data source — is the actual deliverable.
From here, go to the WRDS MCP Setup page to install and verify your own connection before lab, and to this module’s lab for the two-track exercise itself. The Verification page has this module’s inspection checks added to the protocol card you started in Module 1.