Feature: CMR Concept ID <-> Human Readible Name translation tool

[lsterzinger]

Hi python-cmr folks,

I'm a developer at GES DISC, and during a hackweek last week my colleague @eni-awowale created a CLI tool (tentatively named CNR: CMR Name Resolver) that translates concept_id's to native_ids (and vise-versa). It's not publically shareable at this point but if you have access to Earthdata bitbucket you can see our repo here: https://git.earthdata.nasa.gov/projects/GDDS/repos/cnr/browse

After discussion with our manager @briannapagan we were wondering if this would be something you'd be interested in merging into python-cmr. Before opening any PRs I wanted to start a discussion on this to make sure 1) this is something you all are interested in having in this project and 2) we're on the same page for how to implement.

The tool is a python script that gets installed as a CLI tool in the ./bin directory of your python install by using [project.scripts] in pyproject.toml. Example usage:

$ cnr M2T1NXSLV_5.12.4 
prod    C1276812863-GES_DISC
uat    C1245662776-EEDTEST
uat    C1215802944-GES_DISC

$ cnr -e prod M2T1NXSLV_5.12.4 
prod    C1276812863-GES_DISC

$ cnr C1215664070-GES_DISC           
uat	GLDAS_NOAH10_3H_2.0

$ cnr -p MERRA2_100.inst1_2d_asm_Nx.19800101.nc4 -p GES_DISC
prod	G1276974976-GES_DISC
uat	G1256129374-GES_DISC
sit	None

The full list of aceeptable arguments:

$ cnr -h
usage: cnr [-h] [-e {prod,sit,uat}] [-p PROVIDER] [-c] [-g] [-s] id

Get a concept-id or native-id given either

positional arguments:
  id                    Collection ID or Native ID to convert

options:
  -h, --help            show this help message and exit
  -e {prod,sit,uat}, --environment {prod,sit,uat}
                        Environment to use
  -p PROVIDER, --provider PROVIDER
                        Used for a granule and collection query in CMR with a provider
                        id
  -c, --collection      Used to denote a collection query in CMR
  -g, --granule         Used to denote a granule query in CMR
  -s, --service         Used to denote a service query in CMR

[owenlittlejohns]

Comment from Slack thread:

I guess one question I had was what the overlap here is between CNR converting concept IDs into things like short name and version (or other concept specific fields) and the CMR GraphQL implementation.

Just to expand a little bit - when making a request against the CMR GraphQL interface, you have to specify the fields you get back in the response. What I'm trying to think through is whether:

a) Can this work be backed by CMR GraphQL to make more efficient queries?
b) Does the CMR GraphQL API essentially already handle this use-case and are we primarily missing some communication about the super-powers CMR GraphQL already gives us?
c) Do the parameter sets for the concepts supported by CMR GraphQL include all the things you want to query with (concept ID, for sure, but maybe not things like short name and version, yet)?

One really cool thing CMR GraphQL doesn't give, but this tool automatically does, is results from all of production, UAT and SIT.

[eudoroolivares2016]

@owenlittlejohns
a) you could make python-cmr query against cmr-graphql but, that might be a big change since everything else here is hitting cmr directly so that would probably be outside the scope of this issue

b) I'm almost certain we can make a custom field with graphql that could do this use-case (I've done something like this before on a personal project I think)

c) we do support querying by shortname on cmr-graphql but, not short-name + version. I don't know how much trouble it would be to add that to graphql. I would think it wouldn't be crazy difficult since cmr supports it https://cmr.earthdata.nasa.gov/search/site/docs/search/api.html#c-version but, I'm not certain

[abbottry]

Querying by short-name + version isn't desirable because you can only do so via a single record. Some time ago the DAACs got together and discussed this, the result was Entry ID which allows users to search to by the combination of short-name and version via a single, supported and indexed field.

Looking through the docs, and testing a few responses, that value does not come back in either json or umm_json and should be added, that would seem to solve a pretty big need.

With respect to the environments, SIT and UAT should be considered testing environments, I would not recommend features in production that provided the user base with test data. That being said, it has come up recently that the use of environments is useful from a curation perspective and there should be some discussion around that.

So, if this library had a feature that would present these results, using supported params and keys that are indexed, that would be a neat feature of a tool.

[eni-awowale]

Thanks everyone for sharing your thoughts! To clarify we don't use short_name + version to query collections for concept-ids, we use native-ids, which to some is the same as short_name + version, but this is not consistent across DAACs. I like the idea of using entry-id since that is the agreed upon field for short_name + version searches.

[abbottry]

So, it seems like something that would be useful would be filing a ticket with CMR to expose Entry ID (there are multiple teams actually looking to utilize this in the short term)

[eni-awowale]

Hey folks! Restarting this thread now that we are moving into the IP. What do you all think are our next steps?

[frankinspace]

@eni-awowale I'm late to the party but re-reading the thread it sounds like this got redirected into a feature request for CMR directly? Once searching by Entry ID is supported we could add that to this library and that would solve your original use case?

[eni-awowale]

Hi @frankinspace ! Thanks for re-starting this thread. I wasn't aware that this got redirected into a feature request for CMR. Can you share it with me? I do think once searching via Entry ID is supported adding the CL functionality to this library would solve our original use case.

[frankinspace]

@eni-awowale That was meant as a question as I was reviewing the discussion on this issue; it sounds like the suggestion is work required by CMR to support Entry ID. I don't feel I have a strong enough grasp of the issue to make the request but maybe I can poke @abbottry to see if he would be willing to fill out that ticket to CMR and link it here?