Improve container startup time and simplify maintenance by not using custom image #147
Description
Why are we doing this?
We currently build a custom docker image that is referenced in this template's devcontainer.json. The build process for this is moderately complicated and "bakes in" various requirements to support having local R and Python environments that match those of the action images (v1 currently, we would prefer users to use v2 now). This complexity increases container startup time and maintenance overhead.
A major source of that complexity is basing our custom image on the rocker 4.0.5 base. R 4.0.5 is what's used in the opensafely r:v1 image but quite old now and hard to install. Equally, setting up RStudio to work within a codespace/devcontainer is moderately tricky and this base provides that for us. However, since we made this design decision opensafely launch rstudio provides a much more convenient means of launching RStudio within a container that is using the R action image (v1 or v2, defaults to the latter). This new command negates the need for using a rocker base.
Rather than building in the R and Python action image executables and libraries at image build time, we have investigated bind mounting them from a running docker container of the action images. This almost works but not quite enough to be completely workable.
However, if we use a lighter weight standard base image (e.g. the MS devcontainers Ubuntu Jammy image, to match the OS version of the action image base), we can install the R/Python executables (and their dependencies) at container startup and docker cp (or even bind mount them) their libraries into the "host" devcontainer/codespace. Alternatively we could use the OpenSAFELY action image base image and add in the various QoL developer tooling and settings provided in the MS base image (initial investigations suggest this is a bit complex and thus undesirable despite the appeal of having a consistent base for all our images).
This will allow us to avoid the complexity of maintaining a custom image, particularly handling action image version updates. This also massively improves codespace startup time, particularly the time to having a responsive VS Code environment - by moving the expensive operations to codespace postCreate, these can run in the background whilst the user has a responsive UI.
It also opens up opportunities for a bit of dynamism as to which action image version we load in (whilst also switching out which dependencies.txt gets installed). The ideal end goal for this would be some sort of shim binary in front of opensafely exec which would mean nothing would need to be installed in the host codespace/devcontainer - @bloodearnest has been investigating this.
How will we know when it's done?
- Codespace startup times are greatly reduced
- Users have a R:v2 based development environment available (with a simple path for future upgrades)
- We no longer need to maintain a custom base image
What are we doing?
- Migrating the base image for our codespaces to a officially supported third party image
- Moving expensive container setup operations to post-startup background processes
- Sourcing R and Python libraries from action images post-startup
- Providing RStudio via
opensafely launch