The archive · Developer & Business Tools · Product decision · 2026
Firecrawl open-sources pdf-inspector to fix PDF parsing
Firecrawl's classify-don't-OCR bet: an open-source Rust PDF reader hit 15k stars by mid-August and 18.4k by early September 2026.
Firecrawl
What the business is
pdf-inspector is an open-source Rust library from Firecrawl, the web-scraping-for-LLMs company behind the Firecrawl crawler. It reads a PDF's internal structure — fonts, text operators, xref tables, image coverage — and in single-digit-to-low-double-digit milliseconds decides per page whether the content is text-based or needs OCR, then extracts clean Markdown from text-based pages with reading order, tables and headings reconstructed. It exposes a pages_needing_ocr list and confidence score so callers route only the expensive pages to OCR or vision models, runs fully offline with bindings for Rust, Python, Node.js and browsers via WebAssembly, and powers Firecrawl's paid /parse and /scrape endpoints.
How it started
Firecrawl was founded around the thesis that AI agents need a clean pipe to the web, building crawlers and scraping endpoints that turn pages into LLM-ready data. Co-founder Eric Ciarla previously co-founded Mendable, an AI search tool used by companies including Snapchat, Coinbase and MongoDB. By 2026 the team concluded that PDFs were the least reliable part of that pipe: no single library handled every common document format cleanly, and most pipelines treated 'PDF' as one category and OCR'd everything.
What happened
Firecrawl began open-sourcing its parsing stack in early 2026: pdf-inspector appeared in February, and the company's April launch of its Fire-PDF parsing engine was built around the library's per-page classification. On 2026-08-06 Firecrawl announced AnyDoc, a companion open-source Rust library covering 14 office formats, framing the two as one 'document parsing stack' with pdf-inspector reporting 13k stars. Independent reviews followed in August — one measured it at 36x faster than PyMuPDF4LLM on a 200-document benchmark by refusing OCR entirely — and by early September the repo had grown to 18.4k stars and sat at #4 on GitHub's daily trending list.
How it ended up
As of 2026-09-05 pdf-inspector is live and still growing fast, at 18.4k stars with roughly 10.5k added in the previous month, and it now sits inside a broader strategy: AnyDoc extends the same classify-and-extract approach to 14 non-PDF formats, and both libraries feed Firecrawl's paid /parse and /scrape endpoints. The deliberate gap is scanned documents — about 46% of real corpora per Firecrawl's README — which still need an OCR or vision-model fallback. Whether giving away the fast layer converts into paid parsing volume, or simply trains teams to self-host the whole stack, is the open question.
Background
pdf-inspector is an open-source Rust library from Firecrawl that reads a PDF's internal structure — fonts, text operators, xref tables — and classifies each page as text-based or needing OCR in milliseconds, without rendering or running a model. Text-based pages are extracted natively into structured Markdown; scanned pages are flagged with a reason so a vision pipeline handles only them.
The bet embedded in the design is that the AI industry's reflex — route every unstructured input through OCR or a vision model — is wrong for most PDFs. Firecrawl's README cites that roughly 54% of PDFs in real pipelines already contain extractable text, so a rule-based classifier built on a single dependency (lopdf) is dramatically faster and cheaper for the majority of pages.
Firecrawl, the web-scraping-for-LLMs company co-founded by Eric Ciarla (ex-Mendable), released pdf-inspector in February 2026 and built its Fire-PDF engine and paid /parse endpoints around it. By early August the repo had 13k stars; an independent DEV Community review in mid-August measured 15.1k stars after a single week gaining over 4,000, and OpenGithubs counted 18.4k stars by September 2, when it ranked #4 on GitHub's daily trending list.
In August 2026 Firecrawl added AnyDoc, a companion library covering 14 office document formats, framing the pair as one open-source parsing stack for AI pipelines. The libraries run fully offline — including in the browser via WebAssembly — and the deliberate gap is genuinely scanned documents, which still require OCR or vision-model fallback, the paid part of Firecrawl's stack.
What has to be true
- Per-page classification with a pages_needing_ocr list beats the industry default of one binary OCR decision per file, cutting cost and latency for mixed documents.
- Reading PDF internals instead of rendering means single-digit-millisecond decisions with no GPU, no model weights and no network round trip — a different operational surface from ML parsers.
- Open-sourcing the classifier makes Firecrawl the default front door of document pipelines, the same wedge its open-source crawler used for web data.
- The ~46% of genuinely scanned pages still need the expensive fallback — which is exactly the paid service Firecrawl keeps.
What can be applied
Give away the cheap layer that decides what needs the expensive one — the classifier builds trust and adoption, while the fallback it routes to stays the business.
Aftermath
pdf-inspector went from a February 2026 repository to one of GitHub's fastest-growing developer tools of late summer: 13k stars by early August, 15.1k by mid-August, 18.4k by September 2, ranking #4 daily. Firecrawl extended the bet with AnyDoc, an open-source converter for 14 office formats, and both libraries power its paid /parse and /scrape endpoints. The open question is whether the free classify-and-extract layer funnels document-heavy workloads into Firecrawl's hosted parsing business, or whether teams self-host the whole stack and leave it with goodwill instead of revenue.
Sources
- Firecrawl's pdf-inspector Parses PDFs 30x Faster By Refusing to Use OCR or ML at All
- GitHub Daily Rank 2026-09-02 — firecrawl/pdf-inspector entry
spotted an error? The archive wants to know.
Your turn
You just read one. Describe what you are building, and see who is betting on the same thing.
Free account · 3 free questions · no card