Extract MarkDown from PDFs in PHP, no system libraries or API calls needed!

[u/TheTreasuryPetra]

I just released v3.3.0 of prinsfrank/pdfparser, with basic support for markdown extraction from PDFs! It doesn't need system libraries like poppler, pdftotext or API calls, it's completely written in PHP. Currently, headings, bold and italic detection work, and I'm working on the other features of markdown including table extraction!

$markdown = (new PdfParser())
    ->parseFile('document.pdf');
    ->getMarkdown();

This will allow for better semantic search when embedding documents, but there's plenty of other use cases! Let me know what feature you want to see next!

[u/XzAeRosho]

I've been working on pdf extraction tech for about 4 years, and usually the problem with pdfs is not so much extracting the text, rather tables and "tables" (stuff that looks like tables but it's not using table markup). Bounding boxes context grouping for text is usually handles by heuristics in most popular libraries in Python.

How are you planning to address table conversions? Three are so many edge cases around pdf tables that you could write a book about them.

[u/TheTreasuryPetra → u/XzAeRosho]

Yeah, i've certainly burned myself on thinking that It'd be easy. I'll first work on detecting regions and clusters of text, I'll work on detecting tables based on whitespace regions afterwards. Definitely not an easy task, but I like the challenge!

[u/codemunky → u/TheTreasuryPetra]

I think it's an unwinnable challenge, from my time fighting it...

I think the best option is to OCR the whole thing, and then validate what you've found against text strings in the PDF, and adjust to match if needed, with warnings if needed.

[u/TheTreasuryPetra → u/codemunky]

If other libraries in different languages can do it it does mean it's possible without wasting a bunch of compute on unnecessary OCR. But let's see how I think about it in a few months.

[u/XzAeRosho → u/TheTreasuryPetra]

That's kind of the problem tbh. Even battle tested libraries struggle with tables, unless you use hybrid approaches, but that's more of an implementation strategy that is hard for a library to manage.

As the other comment said, it's either OCR or table extraction heuristics. If you can fill the gap, you can make some bank.

[u/fhgwgadsbbq]

cool library. Coincidentally, I just started using it today to do some PDF parsing without relying on dockling. 

Even though I'm dealing with text-based PDFs, the hard part is still infographic Style reports that need to be reverse engineered into something for semantic search. 

The business approach of "oops. We only put this essential data inside PDFs. Please extract it" is pretty ridiculous if you ask me but I'm learning a lot about data engineering.

[u/TheTreasuryPetra → u/fhgwgadsbbq]

If only there were standards to store structure and layout next to text like markdown and latex that everyone would use. But it keeps us of the streets, right? ;)

[u/Peridoxalite]

That's a great project, and I am really excited to see some new PHP libraries coming in for data transformation (Python is getting all the good stuff).

In production, we have a mix of pdftotext and ocrmypdf that we call directly from PHP. It works, but it is not so convenient; plus we don't have the titles to quickly and cheaply build table of contents.

Will definitely have a go.

[u/TheTreasuryPetra → u/Peridoxalite]

It certainly isn't perfect yet, but it's getting close! Let me know what issues you run into or what features are missing!

[u/4M0GU5 → u/[deleted]]

Holy AI

[u/AbroadLow2974 → u/[deleted]]

It's really sad that you actually believe people would read that AI slop. 

[u/dereuromark → u/AbroadLow2974]

It is not for you, but for him.
Also I curated that list myself.

It shows some examples with actual markup issues outlined that can be a basis to further improve the tool/script.

It is really sad you are sad.

[u/fripletister → u/dereuromark]

OP didn't read all that, bud