Move all build activity to docker build stage

[topiaruss]

Hi everyone,

I return to Plone only periodically, and these days don't contribute, therefore I want to show due respect to those who have made Plone what it now is. Thank You!

I also don't have a picture of the roadmap, so feel free to educate me in responses.

On the other hand, I do have a lot of experience of Docker, k8s, complex container build chains, and so on. I have worked on several microservice projects that deployed up to 30 docker containers in k8s.

That experience makes me strict about limiting off-cluster accesses at runtime. I don't want a container ever to start by doing a buildout. My worldview has all of that activity taking place at docker build time. In my world, the image is a complete, sealed, unit, that is loaded and instantly run, with the bare minimum of changes or updates.

I noticed issue 79 #79

When we get to that point, I think we should target a production image with zero unresolved build dependencies.

Depending how I solve my current needs, I may have something to contribute. Please let me know if any work already in progress in this area. I may be able to collaborate.

Thanks again.
--r.

[topiaruss]

After looking deeper, I see that Alin's work on no-ui branch puts me on a happier path than current master. And I've added my [instance] customisations to buildout.cfg.

[svx]

Please have a look at https://docs.plone.org/manage/docker/docs/index.html or https://github.com/plone/plone.docker/blob/master/docs/index.rst.

The current images are NOT meant to run in production, they NEVER were.

As you stated yourself for running in production people should use multistage (build) images or other solutions and this images should only include the bare minimum.

Since lots of people have different needs and use cases we decided not to do that per default.

Never the less we are thinking about that we maybe provide some examples. Still for running Plone in docker in production serious, you will need to have deep knowledge about docker and of course the 'hosting platform of your choice' may it k8, dcos, openstack, something else, etc, etc.

I am sorry I do not want to offend you, but reading things like "puts me on a happier path than current master" make me really sad.

As I wrote before we have this clearly written our docs.

And yes the no-ui branch is changing some stuff but also here, we (Alin) started this for a complete different reason, so this branch is also not really focusing on 'production' .

You can btw also see that in the branch, this branch still ships with lots of stuff what is fine and nice for testing/playing but what you do not want to have in a serious production grade image.

[topiaruss]

Thanks Sven, all points noted.

I've had Plone 5.0.8 running in Docker in production, and it's worked wonderfully. The issue is that the upgrade path is challenging, since it seems the community has not committed to promoting production plone on docker, and nor had it when I took that decision when 5.0.8 was current.

Don't be sad. I took no_ui as a sign that a zero-buildout-at-runtime transition might be in sight, pending any comment on my issue.

I'll assess what's there, and decide if I need to build my own multistage dockerfile.

Personally, the drive to smaller images is fine, but I won't spend effort on reducing size. Memory and disk cost almost nothing. The main thing for me is to get functional ASAP.

For me, usually Docker is the default. It makes everything, development, deployment, maintenance easier. It would be good to see Plone converge around it for all aspects of development/test, even if there was still a non-docker deployment solution at the end of the development process. I imagine there are some who resist that path.

Thanks for your response, and I hope not to have offended.

--r

[topiaruss]

What is the background and intent around the no-ui work, beyond the obvious expansion of the acronym?

[svx]

no-ui stands for no Unified Installer, the current images basically using the Plone Unified Installer inside the image/container to setup Plone.

We are moving away from that since the Unified Installer was never made with microservices in mind.
The UI is great ! Just not for in microservices.

There is also some work going on on a image which is not based on buildout but on pip only.

Still since there are lots of moving parts, including using currently various 'unofficial' forks/branches out of various repositories this is at the moment alpha state and not suited for a 'official' images.

Different people are using that already which is great because by doing so it will help to find bugs, fix missing things and get closer to a 'stable' build and release.

Like I said since lots of people have different needs for production state images, for example zeodb vs rel storage, different security, different add-on packages for the OS and/or Plone, etc, etc we do not provide them.

People should build them on their own.

For the last point :) I disagree IMHO size,matter. Maybe not if you run 30 containers but from 100 on or so it matters, same as the OS, start time, what is running in your container, etc, etc :)

Also smaller images are meaning faster releases = faster image pushes and pulls.
If you release a couple of times a day this is important !

There is way more to that than RAM and HD size :)

It is always 'easy' to have some kind of image, but designing a good image from 'scratch' is a bit more work :)

We try to follow common best docker practices together with best Plone practices for the Plone images.
This is a compromise we decided on.

I personally also consider things like process management, improved security, start times, locked down kernels, secomp, boot and starting times, only minimal requirements, etc, etc, ....

But yeah this are some things what I do in my personal images, this is not always suited for 'common usage' since some of these are also maybe too 'custom'.

Like I said, everyone has a different use case and different needs :)

Have a nice weekend !

[topiaruss]

Many good points. All noted. Thanks for this summary and the insights. —r.

[djay]

I agree that not running pip or buildout in production is useful, but I would not like to lose the easy of new users being able to add plugins via an env variable. Maybe there should be two base images, one that supports adding addons dynamically, and another that doesn;t?

[svx]

@djay There is already a ticket for that, and I am sorry if you misunderstood or if I explained not good enough, but this is exactly why we decided not release prod images. As in the docs stated
Our images are excellent for test driving Plone, checking add-ons or as a base for development/theming

[topiaruss]

What if there were a core image that does all the buildout/pip at docker-build stage? It would be that image that would trigger full automated testing, be the base perhaps for docker-based experimentation. Possibly become the defacto cross-platform test-ground for core Plone.

Then there would be (an)other image(s) built on top of the first with options for environment specified plugins at runtime, or again docker-build time extensions.

I have been outside the community for a LONG time, just dipping in occasionally, but there are means within the Docker ecosystem for automatically building hierarchies of layered images. And you are beginning to use them. It seems to me that perhaps 80% of the needs of the community could be wrapped into a carefully thought-out set of pre-build, tested, static images, with various options for the other 20% to build at runtime for demo/test. Once configurations work within docker and are proven through the test pipeline, they can still be extracted for use in a non-docker environment - thus also catering for docker sceptics.
Personally, I am only using plone to support a long-term client, and it's becoming harder to do so, when I am not a full-on Plone-shop. I look at the CMS options in Helm, and weep. I can't come to the Plone community and get a solid offering that fits my (sorry) Docker mindset, and Helm won't be there for a while.
(My docker mindset comes from my need to support flexible development and test of more complex microservices, and clusters, as well as my interest in Deep Learning, for which I use nvidia-docker - a docker platform that lets me build and test many varied python-based network types without the install/pip/numpy/scipy/anaconda complexities that would otherwise drive me mad.)
Finally, much respect to everyone involved in keeping Plone alive.