Current state and next steps of NPV

[Aurashk]

The purpose of this discussion is to flesh out issue #716 (NPV is broken) by establishing the current state of things and work out the next steps/ break down into sub-issues. Below is my current understanding, please correct anything I got wrong:

1. Since NPV is currently not working it is hidden behind 'please_give_me_broken_results' input, the model setup will panic if 'npv' is selected unless the user sets it to true.
   issue Gate known-broken model options behind another option #897 pr Scare users away from using broken options #902

2. Currently we aren't 100% certain how we will fix NPV, but irrespective of that we want to support alternative pricing methods to the current shadow pricing strategy. The more sophisticated pricing methods may help us fix NPV or may not. Adam has suggestions for the pricing strategies:

Use petgraph to order commodities to support direct marginal, scarcity and capex/fixed-costs inclusive commodity pricing options. Need to start at the top of the system and work down to service demands. Need to handle circularities (only recursive methods work?) Processes that have more than one SED/SVD output and processes that have more than one SED/SVD output (can be done, but needs rules worked out).

3. We already have a scarcity pricing option from Calculate reduced costs and add scarcity_pricing option #684 but the current implementation is flagged as producing bad results I'm not sure if this uses the same approach suggested in 2 though

4. There is separate problem mentioned about the profitability_index function which we also need to address to get NPV working.

So seems like the prudent way forward is to see if we can get the alternative pricing strategies working first in a reliable way (possibly building off 3), then we have a new set of tools to fix the NPV method

@dalonsoa @ahawkes @tsmbland

[ahawkes]

It appears the the NPV calculation is working as per the docs for simple cases. I have not checked for complex cases where there are more sophisticated availability constraints etc (I think there is an issue open for this anyway, e.g. where availability is specified at multiple time slice levels).

But I did think of one thing that may help (a bit at least):

We are calculating contribution to annual total operating cost per time slice (ACt in the docs), and then using this in the annualised profit optimisation. I've noticed that a lot of the AC values are very close to zero (like epsilon close). In fact, by my calculation that are very slightly negative (i.e. tiny profit per activity in time slices). But if you have an existing asset, you could arguably operate when profit is zero as it makes no difference to you. Suggest changing the code so this happens, which I think might be as follows:

1. Instead of maximise (-AFCcap - SUM(act_tAC_t)), we maximise -SUM((act_t*AC_t) only. i.e. remove the capacity term from the optimisation.
2. In time slices where AC_t is negative and small (I'm seeing -1E-14 ish) the optimisation should max out activity. So the asset will be fully on in these time slices, subject to availability limits, for a given capacity.
3. Then we calculate the profitability index as per normal. We select the asset with the best profitability index. We warn the user if an asset has been selected that shows a negative profitability index.

I realise this doesn't solve all problems here. But should avoid the "no feasible investment options for commodity..." error.

As an aside, I guess we are checking that AFC is not negative?

[tsmbland]

1. Instead of maximise (-AFC_cap - SUM(act_t_AC_t)), we maximise -SUM((act_t*AC_t) only. i.e. remove the capacity term from the optimisation.

We already exclude capital costs for existing assets - are you proposing to exclude the fixed_operating_cost as well?

2. In time slices where AC_t is negative and small (I'm seeing -1E-14 ish) the optimisation should max out activity. So the asset will be fully on in these time slices, subject to availability limits, for a given capacity.

Yes I think this is a good suggestion. I guess we can just add a small epsilon to all activity coefficients (noting that we're doing a maximise optimisation so +ve activity coefficient encourages activity).

3. Then we calculate the profitability index as per normal. We select the asset with the best profitability index. We warn the user if an asset has been selected that shows a negative profitability index.

I'm not sure it's actually possible to have a negative profitability index. If an asset wasn't profitable, the appraisal would give us a capacity of zero, and in this case the profitability index is undefined. Perhaps in this case, if its really the case that all assets are unprofitable, then it is the right approach for the model to break. Or, we run the year again with a huge price penalty on this commodity so no assets will consume it (obviously doesn't work for SVD commodities)

[ahawkes]

We already exclude capital costs for existing assets - are you proposing to exclude the fixed_operating_cost as well?

Yes that's right - exclude all elements of fixed costs (capital and foxed O&M). We would need to choose a capacity (in tranches as per LCOX?) and that choice will be important - not too big (low utilisation) and not too small (too many iterations to solve the whole load curve).

This combined with the "dispatch if zero AC" should go some way to making NPV work. Side note - this is not really "NPV" actually - it's actually "Annualised revenue" - important distinction (NPV is sensitive to project life, annualise revenue less so).

[tsmbland]

@ahawkes I think I understand this. Certainly seems sensible for existing assets since it pays those costs regardless (we constrain capacity anyway so the coefficient doesn't make a difference). For candidate assets we'd be saying, so long as the activity coefficient is positive (/0), let it try to install as much capacity as it needs (within the limits of the tranche), even if it can't offset the capital/fixed costs, then let the profitability index decide which candidate would be most profitable. Hopefully we'd have something with a PI > 1 (i.e. activity surplus can offset fixed/capital costs). If not, not sure what would happen in real life (presumably the price would go up enough for this asset to start breaking even, which I guess it will once we start introducing new pricing strategies), so I guess we go ahead and select it anyway with a warning?

[ahawkes]

@tsmbland yes that sounds good (to warn user when bad investment choices are being made).
Maybe should install even if activity coeff is negative? Still taking the best option even if they are all losing money on operation alone.

[tsmbland]

Perhaps, but with the way we do the optimisation (even with the proposed changes here), that's not possible: negative activity coeff -> zero activity -> zero capacity

[ahawkes]

@Aurashk actually I take it back that NPV is working. I think it's getting the sign of the result wrong. In the missing_commodity model, if I put the A0_RES agent objective to npv, turn please_give_me_wrong_results to true and run it, I get -0.09940125836903009 for the investment appraisal metric in debug_appraisal_results.csv output for the RELCHP existing asset. By my calculation it should be the +ve of that number.

Then, I increase the capacity of RELCHP such that it can meet all demand for RSHEAT (to 3655.82). This results in a very small negative number for the metric.

However, investment for RSHEAT still fails. It shouldn't fail because there is enough capacity of RELCHP and it has a +ve PI.

@alexdewar had said that this is because the algorithm actually chooses the most -ve of the metrics for investment, but I'm not sure that's what is happening. I think RELCHP should have been selected, but that's not what is happening. With the above changes, I think this model should succeed on investment in the first MSY (and fail on the second MSY - 2030).

@tsmbland FYI

[ahawkes]

P.S. Oh I guess RSHEAT investment should still fail, as RELCHP isn't dispatched even though it has sufficient capacity - because only a few time slices have a really profitable result (all others are really close to zero - which hopefully would be resolved by my first post above).

[tsmbland]

I've created a branch that gives more detailed data about the appraisal, and I think it helps explain the problem. #989

I think at least part of the issue is with the way we calculate activity coefficient for the NPV. We're calculating it as

net revenue from flows - operating cost
where
net revenue from flows = revenues from output commodities - costs from input commodities

Crucially, revenue from output commodities includes the primary commodity (conversely when using LCOX we exclude the primary commodity).

In the simple case where there's only a single asset/process producing the commodity, I think that because of the way we calculate commodity prices (which ultimately comes down to the cost coefficients in the dispatch, which are operating cost + costs from input commodities) the above expression is guaranteed to be zero. If the activity coefficient is zero, then there's no incentive for activity to be greater than zero (note we also don't apply VoLL for NPV appraisal, unlike LCOX).

I think the only scenarios where the above expression wouldn't be zero:

• multiple technology options: in this case the activity coefficient would be positive in timeslices where the technology has a competitive advantage compared to the market price, and negative in timeslices where it has a competitive disadvantage (note that we do a maximise optimisation when using NPV, so positive coefficient encourages activity)
• the process has side-products which would contribute to the revenue (perhaps this would be factored into the price though, not sure...)
• operating costs/flow costs/levies have changed compared to the previous year when prices were calculated

If the activity coefficient is zero/negative for all options (or at least not positive enough to offset the capacity cost), that's when it fails

I think it's getting the sign of the result wrong. In the missing_commodity model, if I put the A0_RES agent objective to npv, turn please_give_me_wrong_results to true and run it, I get -0.09940125836903009 for the investment appraisal metric in debug_appraisal_results.csv output for the RELCHP existing asset. By my calculation it should be the +ve of that number.

Correct. Internally this would be the value calculated, however we then turn it negative so that we can always do a minimise operation to choose the best asset regardless of the objective, and this is the value that gets reported in the output file

Then, I increase the capacity of RELCHP such that it can meet all demand for RSHEAT (to 3655.82). This results in a very small negative number for the metric. However, investment for RSHEAT still fails. It shouldn't fail because there is enough capacity of RELCHP and it has a +ve PI.

In this case RELCHP is selected for retention. However, because the activity coefficient is (effectively) zero all timeslices in the appraisal optimisation, even though it can meet the full demand (and would do if a commodity balance constraint was applied), the investment algorithm still thinks there's demand left to fill after selecting RELCHP for retention, and then it fails later on when all the remaining options have an optimal capacity of zero

[ahawkes]

Thanks @tsmbland - yes your analysis of the AC coefficient looks correct. I don't think there's necessarily a problem with it though.
And thanks for clarification on RELCHP being selected - makes sense.

[Aurashk]

My understanding is that this is why we need to support other pricing strategies for NPV. Accounting for

• scarcity/surge pricing
• fixed operating costs
• capital expenditure

all make it more more appealing for a given asset to provide the supply and should result in more realistic activity coefficients. Still getting my head around this though.

[tsmbland]

@Aurashk Yes good point. Agree the problem probably isn't with how we're calculating activity coefficients, but just that the NPV objective is incompatible with the way we're currently calculating commodity prices.

I think adding an epsilon to the activity coefficient will help though. Do you want to open an issue and have a go at this?

[ahawkes]

@tsmbland yes it's a bit counterintuitive, but this is how (some) markets actually work. It's related to competition. If they ask for a higher price, a competitor can undercut them and still not lose money. A perfect market therefore tends towards the marginal price of the most expensive producer. In reality of course there are complications - and producers add margins etc - but such prices can only be sustained if there is tacit agreement between producers - which is many markets is probably illegal.

The market somehow works because plant with lower marginal cost make money when something more expensive is operating.

Scarcity pricing is a bit different. Peak producers do add on a premium, usually quite a lot, to recover CAPEX etc. Some electricity markets even have explicit premiums for peak times (I think the Texas electricity market is an example).

[ahawkes]

@Aurashk yes I agree we need alternative pricing paradigms. I had jotted a few thoughts on this in the priority list for this engagement. For a reason that I no longer remember I had proposed calculating marginal cost directly (instead of using the shadow prices) and adding on the CAPEX and FIXOM elements (we do something similar in MUSE1). We would need to think through how this might work and try a few things.

I agree NPV (annualised revenue) probably won't be robust until prices can cover CAPEX/FIXOM. But as @tsmbland says, I think getting the NPV (annualised revenue) to switch assets on when AC is zero should help a fair bit.

[tsmbland]

@tsmbland yes it's a bit counterintuitive, but this is how (some) markets actually work. It's related to competition. If they ask for a higher price, a competitor can undercut them and still not lose money. A perfect market therefore tends towards the marginal price of the most expensive producer. In reality of course there are complications - and producers add margins etc - but such prices can only be sustained if there is tacit agreement between producers - which is many markets is probably illegal.

The market somehow works because plant with lower marginal cost make money when something more expensive is operating.

Scarcity pricing is a bit different. Peak producers do add on a premium, usually quite a lot, to recover CAPEX etc. Some electricity markets even have explicit premiums for peak times (I think the Texas electricity market is an example).

Ok thanks for explaining! That makes a lot of sense now. I guess I was too focused on the simple case of a single producer, but that makes sense in a system with many producers: most expensive producer breaks even, all other producers able to profit

[Aurashk]

I've added issue #991 for adding the epsilon to activity coefficient. Will leave this discussion open for forming further issues about the pricing methods.

[ahawkes]

Ah OK

…

________________________________ From: Tom Bland ***@***.***> Sent: 06 November 2025 13:28 To: EnergySystemsModellingLab/MUSE2 ***@***.***> Cc: Hawkes, Adam D ***@***.***>; Mention ***@***.***> Subject: Re: [EnergySystemsModellingLab/MUSE2] Current state and next steps of NPV (Discussion #987) Perhaps, but with the way we do the optimisation (even with the proposed changes here), that's not possible: negative activity coeff -> zero activity -> zero capacity — Reply to this email directly, view it on GitHub<#987 (reply in thread)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AC37JLLSLQNUI4WRKOHOGND33NEH5AVCNFSM6AAAAACLAKV64GVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTIOBZGE3TQMQ>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[Aurashk]

I've implemented @ahawkes suggestions and it seems promising. See the results in #998 for the simple model. @tsmbland