Feature request: fermentation products

[morganx]

It's very useful to be able to pull glucose fermentation products out of Bergey data. For example, because human microbiome-related publications regularly claim 'X organism is a butyrate producer' and this is only sometimes correct.

Here's a linked example of a Bergey chapter (https://drive.google.com/file/d/1L1sgkZClTp3NjDW3RgAnZvzx4fiobyCG/view?usp=sharing). I've highlighted some of the different ways this data is displayed for various bugs. You need to download it to see highlights (sorry) - Google doc view does not show them. It would be challenging to parse out this data, but really useful to include as a phenotype if possible in future.

[lwaldron]

We have never actually parsed the Bergey's full-text like this, only abstracts. Why did you select Table 3 instead of Table 2 or instead of both? Could we focus just on the tables? There are PDF table parsers that, depending on how the PDF is built, might be able to automate this. Otherwise it will be a Natural Language Processing project.

[morganx]

@lwaldron I don't believe every chapter is formatted consistently to the point that Table 1 or Table 2 would always have fermentation products. I highlighted several different examples because I don't always find the info in the same place.

I haven't thought much about how to solve this problem because I didn't even realize you were working on it until we talked today, but it might be possible to at least curate a medium-to-large chunk just by parsing tables with "fermentation products".

[lwaldron]

Yeah at the level of tables a semi-automated process could look something like (not necessarily 3 people but just to show how it could be broken up - Person 1 probably would be more of a microbiome expert than 2-3 though):

1. Person 1 creates a written TODO list of tables to be ingested
2. Person 2 cuts PDFs to put those tables each in a 1-page PDF of its own.
3. Person 3 runs a PDF table parser on each of those files, does some manual cleanup, and does sanity checks against the original PDFs

If you would start noting tables you'd like to see ingested, I'll put them on a priority list for as soon as I can find a person with bandwidth. It doesn't necessarily have to be just fermentation products, and once we're pulling those from a table like this we might as well pull the other rows too.

[JonathanYe3]

I would like to take a shot at being Person 2 and Person 3! Just need Person 1 to direct me to the tables since I'm not microbiome expert.

[kbeckenrode]

@JonathanYe3 let me know what you need when you get around to this

[JonathanYe3]

Thanks @kbeckenrode! All I need is a list of tables that we want to parse and cleanup. Can @lwaldron confirm if this pdf (https://drive.google.com/file/d/1L1sgkZClTp3NjDW3RgAnZvzx4fiobyCG/view?usp=sharing) is the only one we're working with?

[kbeckenrode]

@JonathanYe3 any table that describes fermentation we'd like to capture, Let me know if you need help.

[JonathanYe3]

Yup, thank you! I think Eric Yu is working hard on this project so best of luck to him.

[sdgamboa]

Link from related issue #204. I think this was the output from the webscrapping.