A more obvious structure of the code

[jmarecek]

Hi all,

it would be great to refactor the code to resemble
https://github.com/humancompatible/interconnect
https://github.com/humancompatible/detect
and any other "major" Python package. That is, within the
https://github.com/humancompatible/explain,
there would be
/tests with unit tests
/examples with Jupyter notebooks, ideally names 01_intro, 02_feature1, 03_feature2, etc.
/docs with Sphinx or similar setup to generate documentation, exported to readthedocs.io (cf. https://humancompatible-detect.readthedocs.io/en/latest/)
/humancompatible/explain with code
.gitignore
CONTRIBUTING.md
LICENSE
README.md with very clear instructions as to how to use it and links to the Jupyter notebooks
requirements.txt tested regularly
setup.py

Thank you!
Jakub

[ntheol]

Hello @jmarecek,

Our understanding is that the humancompatible/explain repository serves as a central location for multiple explainability methods developed, rather than being dedicated to a single approach. For that reason, we organized the repo with separate subdirectories for each method. Inside each method's folder (GLANCE or FACTS), we've followed the structure you recommended — including /tests, /docs, /examples, along with requirements.txt, README.md, and other standard components. You can see this structure in the image below.

Given that the repository is intended to host multiple methods, we’d appreciate your thoughts on the best way to structure it overall.
Looking forward to your guidance!

Best regards,
Nikolas.

[jmarecek]

Hi Nikolas, all,

I think that we have made a good push in terms of the research. I very much appreciate the NeurIPS paper by ARC (FACTS). There is also the ICLR paper by CTU (https://arxiv.org/abs/2401.14086), and multiple NeurIPS submissions this year. Having said that, we also need to deliver a toolkit.

What is a toolkit is, to some extent, subjective.

1. Some baseline is provided by Python packaging guidelines (https://packaging.python.org/en/latest/tutorials/packaging-projects/), generic Python style guidelines (e.g., https://peps.python.org/pep-0008/), Google guidelines (https://google.github.io/styleguide/pyguide.html) etc. I would specifically point to the Python packaging guidelines for guidance on the directory structure:

packaging_tutorial/
├── LICENSE
├── pyproject.toml
├── README.md
├── src/
│   └── example_package_YOUR_USERNAME_HERE/
│       ├── __init__.py
│       └── example.py
└── tests/

2. CTU is working on three other toolkits:
   https://github.com/humancompatible/interconnect
   https://github.com/humancompatible/train
   https://github.com/humancompatible/detect
   and I would like
   https://github.com/humancompatible/explain
   to look similar. By similar, I mean:

• that we replace the src with humancompatible and amend the setup.py appropriately.
• that in /examples, there are Jupyter notebooks, ideally names 01_intro, 02_feature1, 03_feature2, and being very short, perhaps 2-3 pages at most.
• that there is /docs with Sphinx or similar setup to generate documentation, exported to readthedocs.io (cf. https://humancompatible-detect.readthedocs.io/en/latest/)
• that there is .gitignore
• that there is CONTRIBUTING.md, perhaps similar to the other humancompatible toolkits?
• that there is LICENSE with Apache License 2.0
• README.md with very clear instructions as to how to use it and links to the Jupyter notebooks
• that there is requirements.txt and setup.py, which get tested reasonably often.

3. Ultimately, a toolkit needs to be easy to use. Following the rules above is just "means to an end" of usability by the broader community. A really useful test of how "usable" a toolkit is would be to ask a professional software engineer whether they could start using the code in 10 minutes? This is a test Christos Varytimidis suggested. I am happy to conduct such a test, once you suggest the toolkit is ready, and explain the outcome.

4. A high bar for a toolkit would be the JMLR MLOSS. There are submission guidelines are at:

https://www.jmlr.org/mloss/mloss-info.html

and you can go through some of the recently accepted submissions to see that most of the accepted submissions live up to a higher standard, with an active community of developers, posting issues with feature requests, bug fixes, discussing them, and then closing them with pull requests. Having hundreds of open issues and hundreds of closed pull requests is quite common in toolkits accepted in MLOSS.

I would very much like to rely on common sense and good judgement of partners in the consortium.

Jakub

[jmarecek]

You may also like https://github.com/cookiecutter/cookiecutter

[ntheol]

Hello @jmarecek.
I’ve refactored the repository following your instructions. Here's the current structure:

• /docs, documentation of the methods
• /examples, demo notebooks for both methods in separate folders
• /humancompatible/explain, implementation code for each method in separate folders
• /tests, corresponding tests for each method in separate folders
• .gitignore file
• Contributing.MD, based on the ones you have on the other repos you mentioned
• LICENESE
• ReadME.md
• pyproject.toml, requirements.txt and setup.py, for dependency installation.

Everything is set up, except for making the documentation public via Read the Docs. I’ve attempted to configure it, but I don’t have the necessary permissions. I’ve attached a screenshot showing the error from Read the Docs.

Let me know if you can grant the required permissions needed.

Thanks!

[Epanemu]

I had the same issue - it can be solved by being an owner of the humancompatible group - I've just added you

[ntheol]

Thank you very much, I successfully generated the docs!