Run scripts with python

[alecandido]

Reorganize run scripts in a unique python package

• add install/run script run.py (steal from yadism) ba28f12
• add pre-commit cc4416c
• run_dis.sh & run_dis.py
• move pineappl to madgraph (for a yadism run, python package should be sufficient)
• check annoying message ("do you want to recompute?") when the thing is computed the first time 70247cd
• run.sh (reproduce behavior when dir already exists) 2e8d43a
• run_implement_user_defined_cuts.py
• update cuts_code and cuts_variables with master 07f6f3f
• change the package name ec885be
• update_metadata.sh -> update 03665f8
• merge_grids.sh -> merge
• update README
• documenting

They will all receive a corresponding command to be run with poetry run <cmd> or directly as <cmd> from inside the environment.

[alecandido]

Still missing:

• lhapdf installation for DIS grids generation
  • option 1: install mg5 in any case and point to lhapdf shipped by it
  • option 2: install separately lhapdf and make mg5 use that one
  • note: if install lhapdf on its own in prefix, than naively running mg5 won't work (errors obtained)
• parse results.mg5_aMC and put in a dataframe with the same format of the one generated for dis (yadism_results output)
  • pineappl results have 10 columns, so $13 is the third column of results.mg5_aMC table and so on
• use lz4 with python (in this way it's a dependency)
• reorganize the structure
  • improve with a class with standard interface mg5.py
  • apply same structure to yad.py
  • clean up tool.py

[alecandido]

I actually reproduced your issue, seems like it's a bad interplay between poetry and the differential dependencies of pygit2.
I'm fixing for the time being by manually installing cached_property if python is less than 3.7.

[felixhekhorn]

@cschwan just as a heads up: starting from 286f423 you have to explicitly pass a path to a theory runcard to the script (as we always intended) - this is for the moment used only inside yadism for mg5 we need to face https://github.com/NNPDF/runcards/blob/e3cc517833888dd54d2b142104a338350f1b77e9/runcardsrunner/external/mg5.py#L59

for the moment I provided the theories 200 = default NNLO (actually not working with the pip version of yadism, but (hopefully) with the development branch)) and 213 = LO

[cschwan]

Commit 15f33fc fixes a problem when Python 2 is the default:

Installing the current project: runcardsrunner (0.0.0)                                                                                                                                                                                                                                     
Traceback (most recent call last):                                                                                                                     
  File "./rr", line 115, in <module>                                                                                                                                                                                                                                                       
    subprocess.run(f"{python_exe} -m pip install cached_property".split())                                                                                           
  File "/usr/lib/python3.7/subprocess.py", line 472, in run                                                                                                          
    with Popen(*popenargs, **kwargs) as process:                                                                                                   
  File "/usr/lib/python3.7/subprocess.py", line 775, in __init__                                                                                                                   
    restore_signals, start_new_session)                                                                                                                                                        
  File "/usr/lib/python3.7/subprocess.py", line 1522, in _execute_child                                                                                                                                                                                                                
    raise child_exception_type(errno_num, err_msg, err_filename)                                                                                                                                                                                                                       
FileNotFoundError: [Errno 2] No such file or directory: '/home/cschwan/projects/runcards/bin/python': '/home/cschwan/projects/runcards/bin/python'                                                        

[alecandido]

Now it is working even with python 3.7 (since I'm installing cached_property separately).

But it would not work with 15f33fc @cschwan, because that piece of code was making use of python executable inside the poetry environment. Why did you remove? And what's the problem with python 2?

It looks to me that poetry was failing to resolve the environment, and give you back the current folder instead of environment path. But definitely the python_exe should be not the one that is running rr: rr should run outside the environment, especially when doing rr instal since it's also installing poetry, while pip install cached_property should be inside the environment (managed by poetry).

[cschwan]

@alecandido I see... The problem is that you directly call poetry, which starts Python 2, of course, since its shebang is #!/usr/bin/python. I wonder whether it's worthwhile to fix all the Python 2 problems.

[alecandido]

Yeah, I thought about something like {sys.executable} -m poetry, but unfortunately it's not going to work on my computer in the first place.

The reason is that I'm installing poetry with pipx in order not to pollute my global environment (I did add it to yadism, but I didn't here in order to avoid some complications). Than there will be no poetry module for sys.executable, since pipx is using a dedicated environment. The only way is then to rely on the CLI, that is the default way to use poetry in any case.

We can have a look into poetry to see what they suggest for python 2 (if they care at all).

Nevertheless remember that poetry is used only for development, in production you will install the package and it will provide the rr CLI standalone, without any poetry at all.

[alecandido]

Just to recap the status of the business:

• still need a clear cut decision about interactivity: I would like "not at all" at this point, but the alternative is "only while installing" (that is the previous one) a197100
• python 2 poetry issue (if ever) 896b38c
• install lhapdf before pineappl: even if we depend on the python package, pineappl still depends on the C library, so there is no way out 14a263e

On a separate level there is:

• implement theory in mg5

because this I could do in no time if we change the theory to contain all the information here:
https://github.com/NNPDF/runcards/blob/e3cc517833888dd54d2b142104a338350f1b77e9/runcardsrunner/variables.json#L2-L10
For me reading from a json or a formerly loaded yaml is the exact same, but the more we change the theory the more we'll need a database upgrade.

If there are any other business left merge it here, but the three above are the only ones that imo we need before merging, the ones left in the op, and this last one on theory could be faced later on (they are further upgrades with respect to run.sh).

[cschwan]

Regarding interactivity I'd say none at all (even if there was in run.sh).

[alecandido]

• python 2 poetry issue (if ever)

Possible solutions:

• install python-is-python3
• install poetry with pipx (at that point it will be inside the environment, so python will just be the python of the environment)

This is for poetry, so it's pip fault at some point (and most likely his own fault when packaged, because poetry is the build tool for poetry package as well), but there is a solution to make poetry managed by poetry itself (it starts to be complicated, but it is the suggested way of doing it), essentially doing:

curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/install-poetry.py | python -

as described here (notice that it's the unreleased version of poetry, but it is said to use that from stable, because they are transitioning from a script name to another).

Hope this will solve the issue (in an elegant way), I'll try tomorrow.

---

If instead you're problem had been (and it's not) inside the environment, i.e. you want to use a different version of python, then for that one there are documented ways available.

[cschwan]

The runner, both the installation with ./rr install and the run of ./rr run TEST_RUN_SH theories/theory_200.yaml now work like a charm here on Debian 10, thanks for all the fixes!

[cschwan]

In commit d56bd86 I'm switching to the same branch of Madgraph5 that's also used in master; this might've lead to differences in the past (at some points we did a comparison and noticed small numerical differences, which are probably explained by this).

[alecandido]

The runner, both the installation with ./rr install and the run of ./rr run TEST_RUN_SH theories/theory_200.yaml now work like a charm here on Debian 10, thanks for all the fixes!

You're welcome, I'm glad to here it

In commit d56bd86 I'm switching to the same branch of Madgraph5 that's also used in master; this might've lead to differences in the past (at some points we did a comparison and noticed small numerical differences, which are probably explained by this).

Thank you!

Now or we face immediately the theory in madgraph:

• implement theory in mg5

or we just merge and complete everything else afterwards.

[cschwan]

I'll merge this first thing on Monday, and we can open a new Issue for the remaining tasks.

[cschwan]

I've merged this, and all remaining issues are listed here: #108.