[BUG] MUSE stopping without error/warning message

[martinstringer]

Describe the bug

Muse stops running long before completion without issuing a warning or error message.

To Reproduce

Try running the model at MUSE_UK/MUSE_input_files.

Expected behavior

Model stops during first timestep in imports sector at Computing investment: scipy.

Context

• Operating system (eg. Windows 10): macOS 15.7.2
• MUSE version (eg. 1.0.1): 1.5.3
• Installation method (eg. pipx, pip, development mode): development mode
• Python version (you can get this running python --version): 3.9.18

[martinstringer]

This model has now run successfully on the HPC with 64Gb memory.

To close this issue we would ideally:

• Find a way for muse to generate an error message when this happens (not just exit)
• Make any changes to the inputs (and input file generation process) that will reduce memory needs in future.

[tsmbland]

There's lots of discussion about the memory problem in #389, but I'll give a quick overview/update here:

The biggest memory-consuming object is the constraints matrix for the max_production constraint, which for each subsector is a square matrix with side length (n_assets * n_technologies * n_commodities * n_timeslices). For example, if there are 10 existing assets in the sector, 20 technologies in the search space, 5 end use commodities, and 16 timeslices, this will be a 16000x16000 matrix, which would occupy about 2GB memory stored as a numpy array. Double the number of technologies, you expand this four-fold. Important to note that this is for each subsector and not the model as a whole. Adding another subsector with the same number of technologies/commodities/assets would have minimal impact on peak memory requirements as these subsectors would be solved sequentially.

The reason for this shape is that:

• for each existing asset (n_assets)
• we consider all technologies in the search space as potential replacements (n_technologies)
• and we model how much they produce of each end use commodity (n_commodities)
• in all timeslices (n_timeslices)

Square because the constraints matrix is n_constraints x n_decision_variables and we have one constraint for each decision variable.

This is a horribly inefficient way of doing it, but unfortunately this is very much embedded into the way that MUSE is written and at this point not at all practical to change. Indeed, this is a big reason for the rewrite in MUSE2 where the problem has been reformulated in such a way that the max production constraint is currently at worst ((n_assets + n_technologies) * n_timeslices)

However, while we're stuck with MUSE1, the only way to address this is to modify the model in such a way that it decreases one or more dimensions of the constraints matrix (n_assets*, n_technologies, n_commodities or n_timeslices). n_timeslices is simple - decrease the number of timeslices. Up to you if that's a feasible option. You can also use the timeslice_level parameter if the subsector doesn't need to be balanced at the individual timeslice level - but note from past discussions that the behaviour of this can be a bit counter-intuitive. Not necessarily wrong, just needs to be understood. For the other dimensions, the best approach (other than throwing away technologies/commodities) would be to split subsectors into multiple smaller subsectors, each with a smaller set of technologies in charge of a smaller set of commodities (ideally a single commodity if possible). This is usually quite practical and effective, although there are limitations (e.g. if any technology produces multiple commodities, then these commodities need to be in the same subsector)

*Note about n_assets: decreasing the number of starting assets can help, but by the end of the simulation this will mostly be assets installed by MUSE, which you don't have much control over.