Not able to download ANY <s>more than 10</s> pdfs using OpenAlex api

[Pritichahal]

System info: MacOs
Python: Python 3.8.20
I have used the given code pygetpapers --api openalex -q "invasive plant" -k 100 -o "invasive_plant_species_test" -c --makehtml -x --pdf --save_query

to download 100 papers from openalex api. the same code is working fine when i am putting number of paper 10.
for more than 10 papers this is giving me error

https://api.openalex.org/works?search=invasive plant&cursor=*&per-page=20
https://api.openalex.org/works?search=invasive plant&cursor=IlsxNTk5LjI1MSwgLTM3ODY5MTIwMDAwMCwgJ2h0dHBzOi8vb3BlbmFsZXgub3JnL1c0MjkxMTc0OTAwJ10i&per-page=20
https://api.openalex.org/works?search=invasive plant&cursor=IlsxMjI3LjY3MDQsIDEzODM2MDk2MDAwMDAsICdodHRwczovL29wZW5hbGV4Lm9yZy9XMjE3MjE1NTc1OSddIg==&per-page=20
https://api.openalex.org/works?search=invasive plant&cursor=IlsxMTI3LjI3MDEsIDE0MjQxMzEyMDAwMDAsICdodHRwczovL29wZW5hbGV4Lm9yZy9XMTk3NTA5ODg4NCddIg==&per-page=20
https://api.openalex.org/works?search=invasive plant&cursor=IlsxMDEzLjI2MDc0LCA5MzA3ODcyMDAwMDAsICdodHRwczovL29wZW5hbGV4Lm9yZy9XMjE0NjQzNzE2NCddIg==&per-page=20
Traceback (most recent call last):
  File "/Users/priti/miniconda3/envs/pygetpapers/bin/pygetpapers", line 8, in <module>
    sys.exit(main())
  File "/Users/priti/miniconda3/envs/pygetpapers/lib/python3.8/site-packages/pygetpapers/pygetpapers.py", line 589, in main
    callpygetpapers.create_argparser()
  File "/Users/priti/miniconda3/envs/pygetpapers/lib/python3.8/site-packages/pygetpapers/pygetpapers.py", line 583, in create_argparser
    self.runs_pygetpapers_for_given_args(self.query_namespace)
  File "/Users/priti/miniconda3/envs/pygetpapers/lib/python3.8/site-packages/pygetpapers/pygetpapers.py", line 366, in runs_pygetpapers_for_given_args
    api_handler.check_query_logic_and_run()
  File "/Users/priti/miniconda3/envs/pygetpapers/lib/python3.8/site-packages/pygetpapers/pygetpapers.py", line 225, in check_query_logic_and_run
    self.api.apipaperdownload(self.query_namespace)
  File "/Users/priti/miniconda3/envs/pygetpapers/lib/python3.8/site-packages/pygetpapers/repository/openalex.py", line 207, in apipaperdownload
    self.download_and_save_results(
  File "/Users/priti/miniconda3/envs/pygetpapers/lib/python3.8/site-packages/pygetpapers/repository/openalex.py", line 187, in download_and_save_results
    result_dict = self.openalex(
  File "/Users/priti/miniconda3/envs/pygetpapers/lib/python3.8/site-packages/pygetpapers/repository/openalex.py", line 85, in openalex
    json_return_dict = self.download_tools._make_dict_from_list(
  File "/Users/priti/miniconda3/envs/pygetpapers/lib/python3.8/site-packages/pygetpapers/download_tools.py", line 697, in _make_dict_from_list
    if key.startswith("https://doi.org/"):
AttributeError: 'NoneType' object has no attribute 'startswith'```

[petermr]

This is a BUG!
My guess is that there is something wrong with one paper. We need to mend pygetpapers to make it skip this .

Suggest you clone pygetpapers and trap None values in self.download_tools._make_dict_from_list()

Happy to talk on Zoom if you need help

[petermr]

I tried to repeat this and got a different error (Unknown API):

(base) pm286macbook-2:2025March pm286$ pygetpapers --api openalex -q "invasive plant" -k 100 -o "invasive_plant_species_test" -c --makehtml -x --pdf --save_query
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.12/bin/pygetpapers", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/pygetpapers/pygetpapers.py", line 537, in main
    callpygetpapers.create_argparser()
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/pygetpapers/pygetpapers.py", line 530, in create_argparser
    self.runs_pygetpapers_for_given_args(self.query_namespace)
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/pygetpapers/pygetpapers.py", line 309, in runs_pygetpapers_for_given_args
    raise PygetpapersError("API not supported yet")
pygetpapers.pgexceptions.PygetpapersErrorWARNING: API not supported yet
: <exception str() failed>
(base) pm286macbook-2:2025March pm286$ pygetpapers -v
INFO: You are running pygetpapers version 1.2.5

What version of pygetpapers are you using? How did you build it?

[petermr]

Have tried to fix pygetpapers and to jump over this error. However it seems to be due to a bad key and I suspect OpenAlex has changed its API:

python3 -m pygetpapers.pygetpapers --api openalex -q "invasive plant" -k 100 -o "invasive_plant_species_test" -c --makehtml -x --pdf --save_query
<frozen runpy>:128: RuntimeWarning: 'pygetpapers.pygetpapers' found in sys.modules after import of package 'pygetpapers', but prior to execution of 'pygetpapers.pygetpapers'; this may result in unpredictable behaviour
INFO: Making Request to OpenAlex
https://api.openalex.org/works?search=invasive plant&cursor=*&per-page=20
https://api.openalex.org/works?search=invasive plant&cursor=IlsxNTg4LjYyMjcsIDEwNzk5MTM2MDAwMDAsICdodHRwczovL29wZW5hbGV4Lm9yZy9XMjE2MDczNzU4NiddIg==&per-page=20
https://api.openalex.org/works?search=invasive plant&cursor=IlsxMjQ2LjI3MjcsIDEzODM2MDk2MDAwMDAsICdodHRwczovL29wZW5hbGV4Lm9yZy9XMjE3MjE1NTc1OSddIg==&per-page=20
https://api.openalex.org/works?search=invasive plant&cursor=IlsxMTE0Ljg1NTgsIDk2MTQ1OTIwMDAwMCwgJ2h0dHBzOi8vb3BlbmFsZXgub3JnL1cyMDE0MDcyNDYzJ10i&per-page=20
https://api.openalex.org/works?search=invasive plant&cursor=IlsxMDExLjYxMDUsIDEzOTM4MDQ4MDAwMDAsICdodHRwczovL29wZW5hbGV4Lm9yZy9XMjAxNTM3MDk4NSddIg==&per-page=20
INFO: looking for key doi
ERROR: key is None/missing for doi
INFO: looking for key doi
ERROR: key is None/missing for doi
INFO: looking for key doi
ERROR: key is None/missing for doi
INFO: looking for key doi
ERROR: key is None/missing for doi
INFO: looking for key doi

[petermr]

@priti I have mended pygetpapers but I don't understand where the code came from. It's not installed on PyPI so is presumably run locally. It doesn't have an updated version.

I can commit the code but I need your help.

Did you talk with Ayush, Shweata or anyone else?

[Pritichahal]

Hi Peter Can we connect on zoom at 1500 IST, if you are available. Thanks and Regards Priti

…

On Fri, 23 May, 2025, 19:46 petermr, ***@***.***> wrote: *petermr* left a comment (petermr/pygetpapers#54) <#54 (comment)> @priti <https://github.com/priti> I have mended pygetpapers but I don't understand where the code came from. It's not installed on PyPI so is presumably run locally. It doesn't have a version. I can commit the code but I need your help. — Reply to this email directly, view it on GitHub <#54 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BKGJ3FAXD3TOLIH3YBKGTIL274USNAVCNFSM6AAAAAB5AHCZQSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDSMBUGU3TIMRYGQ> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[petermr]

1500 IST == 1030 BST == 0930 UTC.
That works well.

[petermr]

This bug caused unpredictable errors. The following should trap most of the predictable ones.
pygetpapers.download_tools.DownloadTools::_make_dict_from_list

The following lines (ca line 700) catch missing ids and DOIs. I have committed to branch pmr_datatables

    def _make_dict_from_list(metadata_list, paper_key):

        paper_by_key = {}

        for paper in metadata_list:
            id = paper.get("id")
            if id is None:
                logging.warning(f"no id for paper ; skipped")
                continue
            # logger.info(f"id: {id}")
            key = paper[paper_key] # normally doi
            if key is None:
                logging.warning(f"no paper_key for id={id}")
                continue

            if key.startswith(HTTPS_DOI_ORG):
                key = key.replace(HTTPS_DOI_ORG, "")
            else:
                logging.warning(f"key {key} does not start with {HTTPS_DOI_ORG}; skipped id={id}")
                continue