Support `npm start` on host for MFE development

[kdmccormick]

Background

There are ~9 MFEs enabled by default in tutor-mfe now, and that number grows every release. In prod, all MFEs share one container, so it's not problem. In dev, though, we spin up one container per MFE. This consumes an absurd number of resources, and probably isn't necessary since MFEs seem to run fine on everyone's maching with a simple npm clean-install && npm start.

Here's a good illustration of the problem:
https://discuss.openedx.org/t/after-installing-a-tutor-nightly-system-is-working-slow-which-container-can-i-stop-temporary/10681

In addition to resource consumption, making changes to MFE code in Docker containers through bind-mounts is unnecessarily complex. It'd be better if developers could just edit MFE code on their host and see the changes manifest immediately.

Tasks

Prototype an overhaul of tutor-mfe's dev mode that solves this problem, something along the lines of:

• One central Docker container serving all MFEs, similar to prod.
• When a dev wants to work on an MFE, they clone it, run "some command", and run npm start. Tutor should allow them to see the MFE that they npm started rather than the MFE pre-build MFE from the Docker container. The rest of the MFEs should continue to be served from the Docker container.
• We're not sure what "some command" would be, but ideally, it would do something to "switch" Tutor over to the host instance of the MFE instead of the container instance of the MFE.
• There should be an inverse to "some command" which switches Tutor back to using the pre-built container MFE.

Rationale

• On one hand, all MFEs should run bundled together by default, so that the Tutor Dev offering works out-of-the-box without consuming too many resources.
• On the other hand, it should be easy to edit an MFE's code locally, so devs can quickly fix bugs and iterate on features.

Goals:

• Make Tutor great for MFE dev. Currently all MFE devs use devstack.
• Also unblock backend devs who are being overwhelmed by resource usage of MFEs.

Notes

Read @brian-smith-tcril 's comment here: openedx-unsupported/wg-developer-experience#154 (comment)

[brian-smith-tcril]

• move away from container per MFE, use tutor local style MFE hosting instead
  • just change endpoints to point to dev machine instead of containers when doing local dev
  • verify local MFE dev is complete by rebuilding MFE container
• setting endpoints in MFE (as is done by https://github.com/openedx/frontend-template-application/blob/master/.env.development)
  • using dynamic config is probably a good path forward
    • https://openedx.github.io/frontend-platform/module-Config.html
    • https://github.com/overhangio/tutor-mfe/blob/master/tutormfe/patches/openedx-lms-development-settings
    • https://github.com/overhangio/tutor-mfe/blob/b081cf9c7d7a9ecd4045dad4a26a72f00db5e6d7/tutormfe/templates/mfe/build/mfe/Dockerfile#L59-L62
• make sure to communicate as soon as early POC is built

[brian-smith-tcril]

Update on the discovery/prototyping I've done so far:

Restating/clarifying goals

The main goal I have is parity with the devstack development experience.

For example, working on frontend-app-authn in devstack, all one needs to do is:

• run make dev.stop.frontend-app-authn in their devstack venv
• run npm start in their local checkout of the frontend-app-authn repo

My feeling is, "if it is more complicated than that to do MFE dev with tutor we have failed."

MFE side plan

• Update frontend-build to use different .env files for tutor and devstack (example here)
• Update MFEs to be able to run npm start, npm run start-devstack , and npm run start-tutor (example here)
  • I'd like to simplify this to pretty much only use MFE_CONFIG_API_URL, just haven't gotten to that point yet.

In this example, I

• Renamed .env.development to .env.devstack.development
• added start-devstack and start-tutor scripts to the package.json
  • These set the appropriate DEVELOPMENT_ENVIRONMENT environment variable, which is read by frontend-build to determine which .env to use, then run webpack-dev-server
• updated npm start to run npm run start-devstack

Tutor side plan

• Update tutor dev to run all MFEs in one container just like tutor local does (example here)
• Create a command that will serve the same effective purpose that make dev.stop.frontend-app-authn does in devstack
  • Actually stopping serving the MFE in this case feels like overkill, simply updating the config to point to appropriate localhost URLs would be ideal
  • I haven't gotten to the point of making the command yet, in the example I shared I'm simply trying to hardcode tutor into "frontend-app-authn local dev mode"

Current issues

• How to handle CORS/CSRF? I'm not sure how to handle CORS/CSRF between localhost and services in tutor containers.
  • In a slack conversation, @ghassanmas suggested "one just has to set the right proxy, or/and make sure webpack-dev config matches tutor"
    • For the "make sure webpack-dev config matches tutor" part, I think the "MFE side plan" accounts for this nicely. It should be fairly seamless
    • For the "one just has to set the right proxy" part, I feel this fails the complexity test. Unless it is handled automatically, setting up a proxy is decidedly more complicated than the devstack flow.

tl;dr

CORS/CSRF are getting in the way of making connecting to services hosted in Tutor from a localhost MFE as easy as it is to connect to services hosted with devstack. Any suggestions on how handle this nicely would be amazing!

[arbrandes]

While I'm totally for parity with devstack in terms of developer experience and wholly support your efforts here, and while the current .env.development targets the devstack pretty explicitly, I don't think we should be hard-coding references to specific dev environments in the frontend repos.

I'm sure there's a simple way we can improve this (and both Tutor and the devstack) by using something like the module.config.js strategy.

Edit: yes, even if it means issuing PRs to the devstack, too.

[kdmccormick]

How to handle CORS/CSRF? I'm not sure how to handle CORS/CSRF between localhost and services in tutor containers.

@brian-smith-tcril Could you refresh me on what the issue is with CORS/CSRF?

Is it that LMS & CMS will reject requests originating from host-mode MFEs, which are hosted at localhost:1xxx? Would adding the host-served MFE domains to CORS_ORIGIN_WHITELIST and CSRF_TRUSTED_ORIGINS suffice?

[ghassanmas]

@kdmccormick I think yes, because by default tutor uses its own hostname for mfe/lms vs devstack which relies on the default localhost. There might be additional, settings, but this of course one of the needed step.

[brian-smith-tcril]

@kdmccormick I tried that: https://github.com/brian-smith-tcril/tutor-mfe/pull/2/files#diff-9dd8a655b8597bfd6cdb64b96b2a805174561cae462059559fdc4a2b8be0ac06R92-R93, When it didn't work I thought maybe localhost isn't referring to the host machine but instead the container's localhost? I did a little searching but nothing came up

[ghassanmas]

@brian-smith-tcril You are not specifying the port though, hence https://github.com/overhangio/tutor-mfe/blob/94b058ed559684583b89409f01a5a46549f4c389/tutormfe/patches/openedx-lms-development-settings#L25-L26

[kdmccormick]

Actually stopping serving the MFE in this case feels like overkill,

@brian-smith-tcril yeah, I agree, this seems like overkill, I'm not even sure how we'd do it if we wanted to.

simply updating the config to point to appropriate localhost URLs would be ideal

With runtime config, I am optimistic that we will be able to tell MFEs to link to host-served MFEs as appropriate by updating LMS's MFE_CONFIG. Is that what you had in mind as well?

[brian-smith-tcril]

With runtime config, I am optimistic that we will be able to tell MFEs to link to host-served MFEs as appropriate by updating LMS's MFE_CONFIG. Is that what you had in mind as well?

More or less yeah. I think in some (most? all?) cases we won't even need to modify the MFE_CONFIG we have in tutor at all, just tell the local MFE to get the config from tutor.

The main point of updating config within tutor is ensuring all the correct env vars are set (AUTHN_MICROFRONTEND_URL and AUTHN_MICROFRONTEND_DOMAIN for example)

[kdmccormick]

More or less yeah. I think in some (most? all?) cases we won't even need to modify the MFE_CONFIG we have in tutor at all, just tell the local MFE to get the config from tutor

Right, I was thinking not of tutor-mfe's config dictionary, but rather the MFE_CONFIG dict in LMS settings, which feeds the mfe_config API. Turns out it's just ACCOUNT_PROFILE_URL that would need to be updated there: https://github.com/brian-smith-tcril/tutor-mfe/pull/2/files#diff-9dd8a655b8597bfd6cdb64b96b2a805174561cae462059559fdc4a2b8be0ac06R101

The main point of updating config within tutor is ensuring all the correct env vars are set (AUTHN_MICROFRONTEND_URL and AUTHN_MICROFRONTEND_DOMAIN for example)

👍🏻

[brian-smith-tcril]

Is it that LMS & CMS will reject requests originating from host-mode MFEs, which are hosted at localhost:1xxx? Would adding the host-served MFE domains to CORS_ORIGIN_WHITELIST and CSRF_TRUSTED_ORIGINS suffice?

So I realized my mistake from before, I had just put localhost when I needed to have the port too. I've switched to looping through the MFE apps and adding localhost:{port} for each which seems to have resolved it.

[kdmccormick]

Woohoo!