ai / ai engineering / llm / mapreduce / ocr / document processing / agentic engineering / congressional trades
Pelosi Trades Are Public PDFs. MapReduce on the Ones OCR Can't Read.
Congressional periodic transaction reports are public record. The catch is the files: tens of thousands of PDFs, and a lot of them are photographs of paper that do not parse as text.
This is the follow-up to the MapReduce video. Instead of sending one LLM request and hoping, the pipeline sends OCR text to one model and page images to another. If those two reads agree, the row is saved. If they disagree, the raw pages plus both answers go into a third reducer model for the final call.
I implemented that in a brand-new open-source repo, capitol-gains. The CLAUDE.md and AGENTS.md at the root are the implementation. I am also publishing the extracted dataset so you do not have to pay a politician-trades vendor to use the filings in your own research. This page does not claim the interview question in the opening was about MapReduce, and it does not name a dataset URL until one is live.
Transcript
0:00I'm a senior AI engineer, and today I had an interview that went so well that the interview was scheduled for one hour.
0:08I finished it in 30 minutes. And this includes the last question, when he outright said I wasn't expected to get it.
0:15He didn't get it when he applied for the job.
0:21Yesterday I made a video on an algorithm called MapReduce, and today I'm going to explain practical applications of MapReduce that I've implemented in my own agentic engineering.
0:31So if you're a software engineer and you want to know why do I need to know about these advanced AI techniques, why? Stop and follow me, because I make videos like this literally all the time.
0:47So to summarize MapReduce: instead of sending an LLM request and getting a response, what we do is we want to send it to multiple different LLMs.
1:01We might send OCR text to one LLM, we might send batches of images to the other LLM, and then we send it all to one reducer LLM that gives us our final answer.
1:21I implemented MapReduce in this brand-new open-source repo that I call capitol-gains, and I want to show the little gif that I created.
1:33So these are politicians filing. You know those ads, like, oh you can copy Nancy Pelosi's trades, you can copy this politician's trades.
1:43Well, basically, their trades are public information. They're available online publicly, and most PDFs are like regular PDFs, they're like electronic filings. But not all of them.
1:56Some PDFs are literally like scanned photos, like these are some scanned photos that aren't really parsing correctly.
2:07So what we do is we send them all to get OCR'd, and we also send it to an LLM. That gif was moving around too much. We send it to an OCR, we also send it to an LLM.
2:22And then I basically say: do these two sources agree with each other? And if they both agree, they say the same thing, we save it to our data store.
2:32If they don't agree, then we take the raw inputs and both of these outputs into a third LLM, and then we take the final reducer's answer.
2:45So you can go to the CLAUDE.md or the AGENTS.md and you can read the implementation yourself. You can use this to parse your own politician's trades.
2:56And I'm also publishing the actual dataset online so you don't have to pay $50 a month to QuiverQuant or whatever. You can use it for your own trading for free.
Join the conversation
Loading conversation…