ci: Bugs in Docker image building workflow

[phoevos]

The GitHub Action responsible for building and pushing Docker images is currently misconfigured; we have identified the following issues:

• building the images by manually executing the workflow doesn't work for branches other than the default
• any run of the workflow originating from any branch overwrites the dev image
• there is no mapping between the built images and the commits that generated them

Manual runs on arbitrary branches

The workflow currently runs when triggered manually, a commit is pushed to the default branch, or a PR is opened against the default branch. Triggering the workflow manually specifically is extremely important for development purposes, but it's currently broken. Even though triggering the workflow manually is allowed, the build-and-push job is skipped due to the following check:

CogStack-ModelServe/.github/workflows/docker.yaml

Lines 28 to 29 in ebda7d2

	github.ref == 'refs/heads/master' &&
	(github.event_name == 'push' || github.event_name == 'workflow_dispatch')

The above should either be removed, allowing the top level Workflow configuration to determine when the workflow as a whole runs, or be edited if there's a need for more fine-grained control of the specific job runs.

Avoid overwriting the dev image from arbitrary branches

The multi-platform build executed by the workflow always pushed to the dev tag, regardless of where it originates from:

CogStack-ModelServe/.github/workflows/docker.yaml

Lines 61 to 62 in ebda7d2

	tags: |
	${{ env.REGISTRY }}/${{ env.DOCKER_IMAGE_NAME }}:dev

That is of course an issue, since pipeline executions from different feature, test, or release branches should not overwrite the dev which is also currently used in production.

Behind the scenes and even though not explicitly configured AFAICT, the workflow also pushes a second tag (e.g. here), probably related to the attestations, which is however not useable on different platforms since it's not generated by the multi-platform build step.

Tag images using git commit SHA

Currently, there is no way of telling which commit each image was generated from or which image each commit generated. Going through the logs of a CI run would a way (even though extremely inconvenient), but GH only retails those for 30 days, after which they are removed. We should of course include convenience tags like the ones referring to specific named branches, dev, or latest, but we should also make sure that we can deploy to production using images from specific commits and also rollback if needed.

Solution Recommendation

There are of course different ways to handle the latter two issues. One of them, already implemented in the Model Gateway involves manually setting tags, explicitly adding the sha tag so that we always have Docker image pointing to the commit that generated it (helps with traceability), including semver tags for release branches, and only setting the latest (or dev for CMS) tag when the event is triggered on the default branch.