Incomplete text values of ligatures

[pulsar314]

Description

Text extracted from PDF files may contain incomplete values of ligatures (and probably other glyphs) if they have composite names. More precisely, only the first Unicode symbol is present. In this file skakvattenbad_bs_milmedtek.pdf there are problems with fi in "Specifications & Ordering Information" ("Specifcations"), fl in "the bath fluid" ("fuid") and tt in "Multiple LED displays for setting various values"/"various ways of glassware settings" ("seting"/"setings").

Apparently, the issue is with decoding non-standard glyph names (not from Adobe Glyph List). Examples above are /f_i, /f_l and /t_t glyphs correspondingly. Accordingly the procedure described here https://github.com/adobe-type-tools/agl-specification#2-the-mapping, they should be mapped as sequences /f/i, /f/l and /t/t

This behavior seems not to depend on presence or absence of explicit /ToUnicode mappings.

How to reproduce

Read text from PDF using any method get_text, get_texttrace etc.

Expected behavior

Text extracted from PDF contains all Unicode symbols.

Configuration

• Linux MINT
• Python 3.8
• PyMuPDF 1.23.5, installed via pip

[JorjMcKie]

This example file contains non-standard encodings, which makes it impossible for any text extraction tool to extract text satisfactorily.
I tried Adobe, PDF XChange, Nitro, XPDF, Foxit: none of them is able to correctly extract "glassware settings" or "bath fluid".

So this is a problem of this file, not an extraction tool issue.

[pulsar314]

Hi @JorjMcKie. I used poppler tools for comparison and it prints text properly. And actually Firefox allows you to copy the correct text (not MS Edge though).

[JorjMcKie]

Whatever - the fact remains that the PDF uses non-standard encoding for some fonts. So there is no basis for claiming a specific text extraction behavior.

Some tools may be successful by applying guesswork - potentially based on the name of the font or whatever.

Which browser appears to be doing things right or "wrong" depends on which PDF plugin it uses - not on the browser itself.

[pulsar314]

I based my assumptions for the behavior on PDF specification (PDF 32000-1:2008 (1.7)). 9.10.2 Mapping Character Codes to Unicode Values option 2 refers to Adobe Glyph List, which I found here https://github.com/adobe-type-tools/agl-specification.

If this is not supposed to be mandatory, would it be possible to add the specific font identifier (name from page's /Fonts or its xref) to the text data? Probably to output of the get_texttrace method. Since given names are not unique and can be omitted completely, it is impossible to recover details outside of the library.

[JorjMcKie]

No font is required to have this mapping - at all or for all the glyphs it contains.
A font may be a sloppily made thing: glyph mapping back to unicodes may be incomplete.
Some creators may even deliberately omit that mapping, to make text extraction impossible.

If ToUnicode is missing or incomplete, nothing on earth can be done - except OCR.
If a PDF creator does not want to use a ToUnicode then he may specify a standard encoding instead (e.g. WinANSIEncoding), often done. But also here some special glyphs often arer used that are not covered by the chosen standard encoding - and you land at this type of problem.

[JorjMcKie]

BTW the font name already is part of page.get_texttrace() and also appears in page.get_text("dict"). By default this name omits the subset font identifier (like "ABCDEF+"). Setting a global option will however include it. Based on this info, the xref of the font can be found out.

BTW I will move this post to "Discussions" - it is no issue in any sense of the word.