Replace `pdftotext` with {pdftools}

[brownag]

download_GDS() and download_KST() use a system call to pdftotext that appears to be generating warnings. Safe to assume that these parts are not being routinely updated (granted the PDFs are unchanged and are sourced from a standard location. Not sure what is causing "error in running command". Could potentially switch to {pdftools} for parsing--but will need to ensure that the same settings (-raw -nodiag) can be used and approximately the same TXT output generated

══ Warnings ════════════════════════════════════════════════════════════════════
── Warning ('test-refresh.R:17:3'): refresh (from source; pdftools required) ───
error in running command
Backtrace:
     ▆
  1. └─SoilKnowledgeBase::refresh(keep_pdf = TRUE) at test-refresh.R:17:2
  2.   ├─base::try(lapply(exportfn_sub, function(x) get(x)(...)))
  3.   │ └─base::tryCatch(...)
  4.   │   └─base (local) tryCatchList(expr, classes, parentenv, handlers)
  5.   │     └─base (local) tryCatchOne(expr, names, parentenv, handlers[[1L]])
  6.   │       └─base (local) doTryCatch(return(expr), name, parentenv, handler)
  7.   └─base::lapply(exportfn_sub, function(x) get(x)(...))
  8.     └─SoilKnowledgeBase (local) FUN(X[[i]], ...)
  9.       └─get(x)(...)
 10.         └─SoilKnowledgeBase:::parse_GDS(...)
 11.           └─SoilKnowledgeBase:::download_GDS(...)
 12.             └─base::system(sprintf("pdftotext -raw -nodiag GDS.pdf"))
── Warning ('test-refresh.R:17:3'): refresh (from source; pdftools required) ───
'./inst/extdata/GDS' already exists
Backtrace:
     ▆
  1. └─SoilKnowledgeBase::refresh(keep_pdf = TRUE) at test-refresh.R:17:2
  2.   ├─base::try(lapply(exportfn_sub, function(x) get(x)(...)))
  3.   │ └─base::tryCatch(...)
  4.   │   └─base (local) tryCatchList(expr, classes, parentenv, handlers)
  5.   │     └─base (local) tryCatchOne(expr, names, parentenv, handlers[[1L]])
  6.   │       └─base (local) doTryCatch(return(expr), name, parentenv, handler)
  7.   └─base::lapply(exportfn_sub, function(x) get(x)(...))
  8.     └─SoilKnowledgeBase (local) FUN(X[[i]], ...)
  9.       └─get(x)(...)
 10.         └─SoilKnowledgeBase:::parse_GDS(...)
 11.           └─SoilKnowledgeBase:::download_GDS(...)
 12.             └─base::dir.create(file.path(outpath, "GDS"), recursive = TRUE)
── Warning ('test-refresh.R:17:3'): refresh (from source; pdftools required) ───
error in running command
Backtrace:
     ▆
  1. └─SoilKnowledgeBase::refresh(keep_pdf = TRUE) at test-refresh.R:17:2
  2.   ├─base::try(lapply(exportfn_sub, function(x) get(x)(...)))
  3.   │ └─base::tryCatch(...)
  4.   │   └─base (local) tryCatchList(expr, classes, parentenv, handlers)
  5.   │     └─base (local) tryCatchOne(expr, names, parentenv, handlers[[1L]])
  6.   │       └─base (local) doTryCatch(return(expr), name, parentenv, handler)
  7.   └─base::lapply(exportfn_sub, function(x) get(x)(...))
  8.     └─SoilKnowledgeBase (local) FUN(X[[i]], ...)
  9.       └─get(x)(...)
 10.         ├─base::try(...)
 11.         │ └─base::tryCatch(...)
 12.         │   └─base (local) tryCatchList(expr, classes, parentenv, handlers)
 13.         │     └─base (local) tryCatchOne(expr, names, parentenv, handlers[[1L]])
 14.         │       └─base (local) doTryCatch(return(expr), name, parentenv, handler)
 15.         └─SoilKnowledgeBase:::download_KST(...)
 16.           └─base::system(sprintf("pdftotext -raw -nodiag %s", fn))
── Warning ('test-refresh.R:17:3'): refresh (from source; pdftools required) ───
error in running command
Backtrace:
     ▆
  1. └─SoilKnowledgeBase::refresh(keep_pdf = TRUE) at test-refresh.R:17:2
  2.   ├─base::try(lapply(exportfn_sub, function(x) get(x)(...)))
  3.   │ └─base::tryCatch(...)
  4.   │   └─base (local) tryCatchList(expr, classes, parentenv, handlers)
  5.   │     └─base (local) tryCatchOne(expr, names, parentenv, handlers[[1L]])
  6.   │       └─base (local) doTryCatch(return(expr), name, parentenv, handler)
  7.   └─base::lapply(exportfn_sub, function(x) get(x)(...))
  8.     └─SoilKnowledgeBase (local) FUN(X[[i]], ...)
  9.       └─get(x)(...)
 10.         ├─base::try(...)
 11.         │ └─base::tryCatch(...)
 12.         │   └─base (local) tryCatchList(expr, classes, parentenv, handlers)
 13.         │     └─base (local) tryCatchOne(expr, names, parentenv, handlers[[1L]])
 14.         │       └─base (local) doTryCatch(return(expr), name, parentenv, handler)
 15.         └─SoilKnowledgeBase:::download_KST(...)
 16.           └─base::system(sprintf("pdftotext -raw -nodiag %s", fn))

[brownag]

Unfortunately {pdftools} does not offer an analog of -raw -nodiag at this time. Replacing with pdf_text() results in different formatting output, notably preserving separate columns per line in multi-column pages instead of using the "reading order"

Short of getting an update into pdftools probably not much that can be done here. Even for pdftotext -raw is no longer recommended (not sure why: "Keep the text in content stream order. This is a hack which often "undoes" column formatting, etc. Use of raw mode is no longer recommended.")

Perhaps this can be revisited when all of our PDFs have a single column, but for the time being I think we need to stick with command-line pdftotext.

Also, attempting to reconstruct the stream order text from pdf_data() is too much work

[brownag]

After closing this issue I studied the libpoppler C++ API a bit more closely and found that it was a pretty trivial change to support the raw page layout format. I made a PR and it was accepted into pdftools v3.6.0 which is now on CRAN. It should be possible to switch from pdftotext over to pdftools internally, so reopening the issue, but it's not really a huge priority.