Results and comments on the TulipaEnergyModel code review

[datejada]

Input data names

• Double-check the is_seasonal name, since it is used by more assets than the storage (use_timeframe?)
• Align terminology with INES-spec #622

create_input_dataframes

• Remove all csv related reading functions and use TulipaIO instead.

• Change to use strings instead of Symbols in tha create_input_dataframes function (this means Tulipa will now expect strings instead of Symbols)
• Verify if we can use only one df (without using a dict) for assets_profiles and assets_timeframe_profiles (better after having a pipeline with timeframe); have a pipeline example before changing that.
• table_tree stores all the data in dfs

create_internal_structures

• Refactor the inputs to use DuckDB as much as possible. Wherever we have table_tree (df) try to use DuckDB. This should take out TODO in create_internal_structures.
• Use DuckDB directly to calculate the weight as sum of the mapping in the RepresentativePeriod.
• Clean the graph such that we only store the data that are not created before to avoid duplication, e.g., only leave names for flows/edges.
• Explore the option of having an internal table instead of a internal dictionary for the rep_periods_partitions. This allows more DuckDB integration.
  Example:

  graph["Midgard"].rep_periods_partitions = Dict(
      1 => [1:3, 4:5, 6:12],
      2 => [1:1, 2:2, ... 6:6],
  )
  asset | rep_period | k | begin | end
  "Mid" | 1          | 1 |     1 |   3
  "Mid" | 1          | 2 |     4 |   5
  "Mid" | 1          | 3 |     6 |  12
  

• Double-check that where we need rep_periods profiles, maybe just call DuckDB, instead of creating all of them apriori.

compute_constraints_partitions

• Rename constraint partitions to be closer the constraint name itself (e.g., power or balance)

compute_rp_partitions

• This function needs a full refactor when changing to table (DuckDB)
• Big change

solve_model and solve_model!

• Store the solution into tables (a.k.a. DuckDB) instead of the structure solution and the graph.
• Clean the solution structure to (maybe?) don't store any vector/array (only keep objective function and some extra info). Even delete it?
• Create a new table for the investments (assets, and flows)

create_model

• comment -> if we do change all the partitioning to dataframe including the constraints partition computation, then construct_dataframes is not necessary.
• constraints_partitions from Dict to df
• When using tables the filters will be on the tables not in the graph
• Change sets and names to be verbose (e.g., Ai -> assets_to_invest)
• Double check the constraint dataframes AffExpr are correctly created with the aggregation in the model.
• constraints partition: lowest/highest -> to get the correct partition; sum of workspace expressions: sum/unique -> to get the correct expression
• add_expression_terms_intra_rp_contraints will need a refactor, and it will determine changes we need before and after
• Note: profile_aggregation functions are in the table in the documentation

[datejada]

@abelsiqueira and @gnawin, this is for our records to review this work and define the plan of action to continue with the code refactor.

[gnawin]

#415 is super useful to get the complete picture of the relationships of the inner functions

[abelsiqueira]

Just storing some code snippets to use for the partition changes using DuckDB: https://gist.github.com/abelsiqueira/1849dfed7eb25519bc3e5da9a672c268

[datejada]

Closing this discussion since it is now the epic issue #701