Feature: add API to get metadata of images from a registry.

[t0randr]

Description

Seqera platform needs a way to dynamically define the list of available images for studio templates, so that these can be dynamically updated when a new image is released, or when it is no longer supported, without the need of changing configuration and restarting the application.

The proposed solution is to expose an API from wave that allows to retrieve the list of images, given a registry/repository, with their annotations (LABELS).

A set of well known labels will be used to define relevant information about the specific image:tag,

The immediate goal, is to expose the support status of data studio templates, so that they can be tagged with:

• deprecated
• preferred
• experimental

[pditommaso]

Can you draft an ideal endpoint definition from your point of view? How the request and response would be?

[pditommaso]

For avoidance of any doubt, draft not the code, just as endpoint definition

[t0randr]

I think it would be good for wave to be completely agnostic about the specific needs, eg the label names.

Ideally we want to issue a GET request with a specific namespace, and get back the available images with their labels only, something like:

GET /v1alpha1/images/labels/<registry>/<namespace>

{
    "images": [
        {
           "repository": "registry/namespace/image:tag",
           "labels": [
              { "name": "io.seqera.example.label", "value": "example-value"}
             ...
           ]
        },
       ....
    ],
}

[pditommaso]

Your plan is to list all repos and images in a registry?

[t0randr]

For our specific use case the images are all in a single namespace, but I understand that would be to much.

We can instead use:

GET /v1alpha1/images/labels/<registry>/<namespace>/<repository>

And use pagination to further limit the amount of retrieved tags.

This is not ideal, because if we want to add a new studio, we will still need to reconfigure platform.
We could proxy the _catalog endpoint (again with pagination), and filter for the repositories that we are interested in, so that we than can make multiple requests to get the manifests.

I don't think there is any standard search functionality offered by the registry API to find only the repositories we are interested in.

We also intend to cache the results for long periods, but wave cannot make this assumption, but we can rate limit these endpoints.

[t0randr]

Also we can move all the images we are interested in from their original location cr.seqera.io/public to cr.seqera.io/public/studios, so that we don't need to chain requests.

Eg:

GET /v1alpha1/images/labels/cr.seqera.io/public/studios

But again we cannot assume that the number of images tags will be small.

[pditommaso]

Let's agree on definition

• registry: -> public.cr.seqera.io
• namespace: -> public.cr.seqera.io/platform
• repository: -> public.cr.seqera.io/platform/data-studio-jupyter
• container image: -> public.cr.seqera.io/platform/data-studio-jupyter:4.2.5-0.7.5

You need to list images in a namespace of repository?

[t0randr]

Ideally we need to list images in a namespace.

We can list images for specific repository, with the caveat that than we cannot add a new studio type without reconfiguration.

[pditommaso]

I think Registry APIs allows you to list repositories and tags per repo.

Therefore what you are suggesting would require a combination of both of them, and seems a bit overkilling.

Could not be enough listing labels per repository?

[t0randr]

Yes:

We can list images for specific repository, with the caveat that than we cannot add a new studio type without reconfiguration.

[pditommaso]

Ok, we have already an inspect endpoint, I'd suggest considering an extension of that functionality and add

GET /v1alpha1/inspect/labels?image={image}

where image be wither with or without tag.

What do you think?

[t0randr]

If we settle to request specific images (without tags) does this endpoint need to be extended?
Doesn't it return the annotations already if they are defined in the manifest?

[pditommaso]

Inspect currently requires the tag/reference to be specified if i'm not wrong

[pditommaso]

If we settle to request specific images (without tags) does this endpoint need to be extended?

Just realised the current implementation fallback to latest by default.

I'd propose to add a new endpoint similar to list image tags, but returning also all labels. Something like

{
    "name": <name>,
    [
       {
          "tag": <>,
          "digest": "sha256:<>"
          "annotations": <>
       }  
    ]
}

The problem the resulting payload could be large and there's no good way to paginate it on backend side