Ideas for tidying up the data structures

[tsmbland]

Currently there's a real lack of consistency in the way we're reading in, validating and structuring the data, which is making the code very difficult to work with. I feel we need to come to a consensus and use a consistent approach throughout, particularly for dealing with maps/connections between tables.

I had fun with #474, but leaning against this approach as it feels like overengineering and it only really works when year is the only key, which is never. I think #475 is more the approach I'm leaning towards, although it's still incomplete.

This is very much a scattered collection of thoughts/notes/ideas, but I think we can come up with a solid plan based on this. Also very happy to hear other opinions and ideas

Notes:

• 1:1 connections between tables should not exist (in this case tables should be combined)
• 1:n connections between tables to be represented by map objects
• We can create a custom Map object (can come up with a more creative name) which could just be an alias for HashMap to start with, but would allow us to add features e.g. more helpful error messages, validation methods
• Instead of key objects, we could create types for CommodityID, AgentID etc. so it's clear how each map can be looked up. Conversion of a key from a string to an ID type could require checking the string against the list of valid IDs, so an ID type always implies the key is valid
• For example, AgentSearchSpaceMap would be Map<(CommodityID, YearID), SearchSpace>
• Top-level tables (e.g. Processes) should also be kept in maps (i.e. ProcessMap = Map<ProcessID, Process>)
• I think we should move code for defining data objects and map objects into a data module. This should should just contain the struct definitions, without any functions
• We need to think about whether the maps contain the actual objects or references to the objects (e.g. Map<ProcessID, Rc<Process>>), and be consistent. I guess it's better to store references, but not sure
• For tidiness I think we should remove IDs from objects if these are stored as keys in the map (or a parent map). E.g. AgentObjective should not have a year field because year is a key in the map
• Maps with tuple keys should always be dense after the data is fully read in. E.g. if region and year are the keys, then every combination of region and year should be specified (within a subset of regions and years). We can provide a method within Map to validate this
• We deal with "all" by cloning the data (/reference) over multiple keys. We can provide a method in Map to ensure that there are no clashes/overwriting of data (e.g. if "all" and a specific year are specified)
• Each table should be read into a XRaw struct, which should have a to_x method (which can optionally call a validate method to perform row-level validation). We could modify to_x to return the final object as well as a tuple of keys, for easy insertion into a map

The final data structures would look something like this:

commodity: Map<CommodityID, Commodity>

• commodity_demand: Map<(RegionID, YearID, TimeSliceID), f64>
• commodity_costs: Map<(RegionID, YearID, TimesliceID), CommodityCost>
• commodity_constraints: TODO

process: Map<ProcessID, Process>

• process_parameters: Map<(RegionID, YearID), ProcessParameter>
• process_availabilities: Map<(RegionID, YearID, TimesliceID), RangeInclusive<f64>>
• process_flows: Map<(RegionID, YearID), Vec<ProcessFlow>>
• process_flow_share_constraints: TODO
• process_investment_constraints: TODO

agent: Map<AgentID, Agent>

• agent_commodity_portions: Map<(CommodityID, YearID), f64>
• agent_objectives: Map<(CommodityID, YearID), <Vec<AgentObjective>>>
• agent_search_space: Map<(CommodityID, YearID), SearchSpace>
• agent_cost_limits: TODO

assets: Map<AssetID, Asset>

regions: Map<RegionID, Region>

[tsmbland]

@alexdewar I know you're extremely busy, but if you have any thoughts or time to chat about this then let me know!