PHEP 4: PyHC Package Tiering #31

jibarnum · 2024-07-09T20:22:31Z

This PR proposes a new process PHEP to the PyHC. PHEP 4 establishes a new tiering structure to PyHC projects, which will automatically affect PyHC packages once it goes into effect. Included herein is information on requirements for each of the new four tiers of PyHC projects (Gold, Silver, Bronze, and Bronze), as well as benefits accrued at each tier.

jibarnum · 2024-07-09T20:43:28Z

@jameswilburlewis @aburrell @rweigel @sandyfreelance @darrendezeeuw can't add you all as reviewers (I think I need to invite you to the PyHC org on GitHub first. But for your awareness and comments.

aburrell

First impressions.

pheps/phep-9999.md

aburrell · 2024-07-09T21:04:29Z

pheps/phep-9999.md

+   - PyHC Software Env Inclusion: indicates whether a package will be included within the PyHC software environment
+   - PyHC-Chat Bot Inclusion: indicates whether a packages will have up-to-date information included within the ChatGPT4-powered PyHC-Chat bot
+   - pyOpenSci Verified Badge: a badge that shows whether a package has completed the pyOpenSci review process
+   - Standards Compliance Assistance: indicates whether a package will receive extra help and/or advice from PyHC leadership in conforming to the PyHC standards


So, this seems counterintuitive to me. Shouldn't the packages that are less up-to-snuff receive more help? Perhaps a better way of splitting time would be having a pool of volunteers that could give time for things like code reviews and then receive code reviews from other people. That would be a nice way of creating more of a community environment in PyHC, but is not directly related to the tiers.

Fair point. There's also an idea that packages who've done the work to be at a higher level get more help if there's extra funding for someone to poke around their code base (e.g. work through bug issues or solve long-standing open PRs).

That's a good point! We should have community support mechanisms for packages to level up, as you suggest.

📌 I'm wondering if something like an annual unstructured Helio Hack Week (like Astro Hack Week) would provide good opportunities for us to support each other in this way.

As a potential use case, having a path forward to help packages be pip/conda-installable would be a good support. A Hack Week could be useful here.

pheps/phep-9999.md

Changing Honorable Mention to Copper Co-authored-by: Angeline Burrell <aburrell@users.noreply.github.com>

jklenzing

Initial thoughts. I think this is a good step forward.

pheps/phep-9999.md

jklenzing · 2024-07-10T20:47:42Z

pheps/phep-9999.md

+   - PyHC Software Env Inclusion: indicates whether a package will be included within the PyHC software environment
+   - PyHC-Chat Bot Inclusion: indicates whether a packages will have up-to-date information included within the ChatGPT4-powered PyHC-Chat bot
+   - pyOpenSci Verified Badge: a badge that shows whether a package has completed the pyOpenSci review process
+   - Standards Compliance Assistance: indicates whether a package will receive extra help and/or advice from PyHC leadership in conforming to the PyHC standards


As a potential use case, having a path forward to help packages be pip/conda-installable would be a good support. A Hack Week could be useful here.

pheps/phep-9999.md

sapols · 2024-07-11T17:45:19Z

Initial thoughts/issues:

Excited to see this draft! This is a great move for PyHC. Don't mean to focus on issues but will anyway to keep my comment short
Is it a typo that the first table ends with "Copper" instead of "Honorable Mention"?
It'd be helpful to add a hyperlink to the PyHC env to clarify which env we mean. Probably even a specific Docker image for extra clarity?
- Although (and maybe this is a bigger question) do we need a new "PyHC env" to facilitate this? The purpose of the current one is to hold all PyHC packages, whereas this PHEP specifies only Gold-tier packages get inclusion in the env. (Which also begs the question how will packages know if they're compatible with the env if they're not included in it?)
Question: how will this affect "core" package status? Will "core" packages still exist, or does Gold-tier become the new "core"?

aburrell · 2024-07-12T14:46:16Z

@sapols "copper" was my suggestion, to keep with the medal terminology. It's the next medal after "bronze".

rebeccaringuette · 2024-08-05T22:53:58Z

I would like to echo Shawn's comments on this PHEP being a great step forward for PyHC. Some comments:

Agree that benefit items like Python env inclusion and chat bot inclusion should only be available to the packages that have put in the effort to make their inclusion simple. Could allow silver packages for those items given justification (e.g. number of users, effort made for those items but gold level not achieved).
We need two versions of the PyHC environment to avoid creating an environment so large that no one wants to wait for it to install/load. Suggest allowing some bronze + all silver + all gold in an 'all-PyHC' env, assuming no installation conflicts and necessary effort completed. For trimmed down version, could restrict to some silver + all gold and include breadth of use in the eligibility requirements (e.g. used by a large number of people -> include in the trimmed down PyHC env). This trimmed down version would be what we start with for the PyHC summer school. Using this approach provides a more flexible method to include packages in the summer school and to include packages in the PyHC environments.
agreed that standards compliance assistance should be available to all upon request. How much assistance is made available will have to depend on the level of PyHC funding available, maybe some simple justification requirement, and other factors
also hesitant about including interoperability status. Would rather see metadata compliance status (thinking of interoperability with the HSSI effort) as a column in the table and a software license status (should exclude the NOSA license on the gold tier since it makes collaboration extremely difficult).
also agree on including the PyHC env installation conflicts in the table, with a modification to include two versions as above
package DOI should be for the software repository, not an associated publication. Could be a preference, not a requirement for now, and there should obviously be some exceptions (e.g. JOSS).
PyHC standard grades should not be determined by self-evaluation. If a self evaluation is an allowed method, there must be a complete review by PyHC leadership (e.g. Shawn) to confirm those compliance ratings.
need to specify the current PyHC env, or a version created in the last year, not just any past version
like the idea of the term 'core packages' being removed. much prefer gold instead.
need to make some funding available for packages to go through the pyOpenSci process (e.g. through ROSES B.20, maybe small amounts available through a PyHC NSF or NASA grant for that purpose).
Need to state a time frame for packages to submit the tier they best align with (and how they fulfill the requirements for that tier) to PyHC leadership after this goes into effect (e.g. 6 months to 1 year?).

jibarnum · 2024-08-26T22:50:46Z

@sapols thanks for your thoughts.

Is it a typo that the first table ends with "Copper" instead of "Honorable Mention"?

No, as @aburrell pointed out, that was changed to keep with the "medal" terminology we used for the other categories.

It'd be helpful to add a hyperlink to the PyHC env to clarify which env we mean. Probably even a specific Docker image for extra clarity? Although (and maybe this is a bigger question) do we need a new "PyHC env" to facilitate this? The purpose of the current one is to hold all PyHC packages, whereas this PHEP specifies only Gold-tier packages get inclusion in the env. (Which also begs the question how will packages know if they're compatible with the env if they're not included in it?)

I think we want to establish some specific environments for this. @rebeccaringuette had the interesting suggestion in her comment (below yours) re creating two environments. I think some kind of split of Gold + Silver and then Gold + Silver + Bronze for PyHC-top-tier and PyHC-all environments, respectively (happy for some help in workshopping that terminology).

Question: how will this affect "core" package status? Will "core" packages still exist, or does Gold-tier become the new "core"?

I think this would make core go away, yes, leaving us the highest level being "Gold". It'd get confusing in my mind to delineate the differences between Gold and core. Further, we've always struggled to say what exactly it meant to be a core package, or how to become core package (apart from a nod of approval from current leadership and core package maintainers).

jibarnum · 2024-08-26T23:34:37Z

@rebeccaringuette thanks for your thoughts above!

Agree that benefit items like Python env inclusion and chat bot inclusion should only be available to...

Indeed, I'm trying to make that a bit more clear in the soon-to-come commit.

We need two versions of the PyHC environment to avoid creating an environment so large that no one wants to wait for it to install/load...

I like this thought. I'll include it. However, I do wonder how we intend to include the bronze categories, which allow some major conflicts to exist with installation into the software environment... thoughts?

agreed that standards compliance assistance should be available to all upon request

I mostly agree. I think if you're already at Gold, you probably will only get assistance if you're in danger of dropping down a level.

also hesitant about including interoperability status...

Yeah, I nixed that one. The metadata suggestion is good, though can you elaborate on how we would evaluate that?

also agree on including the PyHC env installation

Yep.

package DOI should be for the software repository,

Sure, that makes sense.

PyHC standard grades should not be determined by self-evaluation...

Indeed, and thus the point of doing a pyOpenSci review process. But the self-evaluation is just step one to getting there. Shawn does also do a general review to make sure the grades are commensurate with the state of a repository.

need to specify the current PyHC env...

Sure.

like the idea of the term 'core packages'...

Same, I'm nixing that once (if) this PHEP goes into place.

need to make some funding available for packages

For sure. First we need to get a good definition on what we want for PyHC-specific requirements for a pyOpenSci process to show we have the process in place and ready to go for packages.

Need to state a time frame for packages to submit the tier they best align with

For sure, I need to include some wording on this. I don't want to wait too long, so perhaps 6 months is best. I'll find out soon if that's a terrible idea by how many tomatoes are thrown my way with the next commit. :)

…m initial round of reviews on GitHub

jibarnum · 2024-08-27T00:31:18Z

Alright, all. Tried to catch and incorporate as many comments as I could. Please review and let me know what concerns/suggestions I didn't capture or have come up with the changes. Thanks!

jibarnum · 2024-08-27T00:32:59Z

pheps/phep-9999.md

+| PyHC Standard Grades | Mostly green, some yellow | Several yellow, no red | A couple red | N/A |
+| pyOpenSci Review Status | Completed | In progress | Not Started | Not Started |
+| PyHC Env Installation Conflicts | No conflicts | A couple conflicts exist | Major conflicts exist | Major conflicts exist |
+| HSSI Metadata Compliant | Fully Compliant | A couple issues exist | Major issues exist | Major issues exist |


This needs a bit more defining. Hoping to get some input on that from you @rebeccaringuette if you have thoughts from the metadata perspective? Are we looking for packages to have a specific set of metadata included at time of tier submission?

Sure.

Copper: Name of package, link to repository.

Bronze: + DOI, license, description, (publisher, publication year, authors -> needed to create a DOI), mandatory fields for HSSI*

Silver: + most recommended fields for HSSI*

Gold: + all recommended and some optional fields for HSSI*
*to be determined

Ideally, the PyHC package submission form should incorporate HSSI metadata fields.

That is perfect, thank you!

pheps/phep-9999.md

jibarnum · 2024-08-27T00:33:57Z

pheps/phep-9999.md

+
+Descriptions for each heading are as follows:
+   - Summer School Inclusion: indicates whether a package will be included in summer school teaching materials
+   - PyHC-top-tier Env Inclusion: indicates whether a package will be included within the current PyHC software environment used at the summer school (also included within env in Science Platforms Coordination group???)


That question in the parentheses is once again for you @sapols :D

I'd actually say the env being made for the Science Platforms Coordination group is out of scope here. It's not an explicitly PyHC thing; we're not even including all core PyHC packages in that env.

Makes sense. Will remove.

jtniehof

I have suggestions inline, but I don't think they're required. I'll hold back on an "approve" until we're voting.

jtniehof · 2025-06-06T12:07:48Z

pheps/phep-9999.md

@@ -0,0 +1,147 @@
+```
+PHEP: 9999


@sapols can you formally assign the number?

PHEP number assigned and DOI reserved.

jtniehof · 2025-06-06T12:13:02Z

pheps/phep-9999.md

+
+# PyHC Package Tiering Specifications
+<a name="specification"></a>
+There are three tiers proposed in this PHEP: Bronze, Silver, and Gold. These would replace the current structure of "Core Packages", "Other Packages", and "Un-evaluated Packages". Extensive updates have been made to the following tiering specifications, based on discussions held at the [PyHC Fall 2024 meeting](https://doi.org/10.5281/zenodo.15080483). See the table below for requirements associated with each tier:


I'm not sure we need "updates have been made..." in the spec as such, but I like the idea of linking the fall meeting notes.

How about in this section we have:

...and "Un-evaluated Packages". See the table below for the baseline requirements associated with each tier; other standards-track PHEPs may add additional requirements.

And then in "Rejected Items":

Originally this PHEP contained detailed standards for each tier. After discussions held at the PyHC Fall 2024 meeting, it was decided to have this PHEP focus on the tiering process, and have the detailed standards by tier contained in other standards-track PHEPs. This allowed discussion of the tiering structure to be separated from discussion of which standards were appropriate for each tier.

jtniehof · 2025-06-06T12:14:09Z

pheps/phep-9999.md

+
+|       | **Gold** | **Silver** | **Bronze** | 
+| :-: | :----------: | :-----------: | :------------: | 
+| Self-Evaluation | No | No | Yes | 


Persnickety: would this be more clear if it were instead "Formal Evaluation: Yes Yes No"?

pheps/phep-9999.md

jtniehof

This is just trying to make the CI happy. I think I found all the trailing whitespace and one typo.

pheps/phep-0004.md

Co-authored-by: Jon Niehof <jtniehof@gmail.com>

jtniehof · 2025-10-20T14:04:10Z

Looks like that squashed all the whitespace issues. Let's never use the GH suggestions again. Holy smoke.

rebeccaringuette · 2025-11-03T17:32:21Z

The "* = See HSSI metadata schema for details" line no longer applies. The voting to regrade (or not) a given package needs to be explained a bit more, e.g. what quorum of the TSC is needed (e.g., 2/3?), will it be a simple majority vote?

…

On Mon, Oct 20, 2025 at 10:04 AM Jon Niehof ***@***.***> wrote: *jtniehof* left a comment (heliophysicsPy/standards#31) <#31 (comment)> Looks like that squashed all the whitespace issues. Let's never use the GH suggestions again. Holy smoke. — Reply to this email directly, view it on GitHub <#31 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALX7QXSPZ2POSMLVXZZU24T3YTTXFAVCNFSM6AAAAABKTRZAKWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTIMRSGIYTSOBSGQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

jibarnum · 2025-11-20T04:00:20Z

The "* = See HSSI metadata schema for details" line no longer applies. The voting to regrade (or not) a given package needs to be explained a bit more, e.g. what quorum of the TSC is needed (e.g., 2/3?), will it be a simple majority vote?
…
On Mon, Oct 20, 2025 at 10:04 AM Jon Niehof @.> wrote: jtniehof left a comment (heliophysicsPy/standards#31) <#31 (comment)> Looks like that squashed all the whitespace issues. Let's never use the GH suggestions again. Holy smoke. — Reply to this email directly, view it on GitHub <#31 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/ALX7QXSPZ2POSMLVXZZU24T3YTTXFAVCNFSM6AAAAABKTRZAKWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTIMRSGIYTSOBSGQ . You are receiving this because you were mentioned.Message ID: @.>

Removed that line. I'm going to suggest a simple majority. I also added some more detail surrounding the TSC, how they're brought on and for how long, etc.

jtniehof · 2025-11-20T17:13:37Z

pheps/phep-0004.md

+    - 1) The PR is merged if there's a simple majority of TSC member in agreement with the self-evaluated tier, or
+    - 2) The PR is rejected (and the TSC collectively craft and provide written reasoning through the PyHC Lead for a differing tier level) if a simple majority of TSC members do not agree with the self-evaluated tier.
+
+\* The TSC includes the PyHC Lead, PyHC Tech Lead, as well as volunteer members of the PyHC not involved with the submitted package, with a minimum of 4 members up to a maximum of 6 members. Volunteer members of the TSC will serve a one-year tenure, decided upon at the nearest telecon or bi-annual meeting following the written email communication of this PHEP's implementation and at a yearly cadence thereafter. After the year tenture ends, TSC volunteer members can choose to stay or cycle out with a new PyHC member. If a PyHC member is interested in being involved, but cannot attend that telecon or bi-annual meeting where TSC members are chosen, they may email the PyHC Lead with their request for candidacy at least one business day beforehand (to give the PyHC Lead time to see the email).


Is the min/max of 4/6 the total, or just the volunteers? Suggest "minimum of 4 total members up to a maximum of 6 total members". Or 6/8.

Some explicit statement that "future PHEPs may further define the TSC and may supersede this PHEP regarding the TSC without affecting other standards and processes in this PHEP"

jtniehof · 2025-11-20T17:19:14Z

This is the voting comment for the first vote, from the fall 2025 meeting. Thumbs up to approve, thumbs down to disapprove, anything else to abstain.

Vote is for current status with these changes included.

Quote / reply to this if you want to include commentary with your vote.

jibarnum · 2025-11-20T17:21:54Z

This is the voting comment for the first vote, from the fall 2025 meeting. Thumbs up to approve, thumbs down to disapprove, anything else to abstain.

Vote is for current status with these changes included.

Quote / reply to this if you want to include commentary with your vote.

2 abstained votes

jibarnum · 2025-11-20T20:13:51Z

Added in your suggestions for the TSC portion @jtniehof . This should now be up-to-date

adding in draft for the PyHC package tiering PHEP

beb0c9e

jibarnum requested review from jtniehof, namurphy, rstoneback, sapols and wtbarnes July 9, 2024 20:24

jibarnum self-assigned this Jul 9, 2024

jibarnum requested a review from Cadair July 9, 2024 20:40

jibarnum requested a review from jklenzing July 9, 2024 20:45

aburrell reviewed Jul 9, 2024

View reviewed changes

namurphy reviewed Jul 9, 2024

View reviewed changes