Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Store everest results in ERT storage #9161

Open
wants to merge 7 commits into
base: main
Choose a base branch
from

Conversation

yngve-sk
Copy link
Contributor

@yngve-sk yngve-sk commented Nov 6, 2024

Issue
Resolves #8811

Store it as some dataframes and dump it in storage, not entirely sure how much of it should be saved. Will use everest_data_api as reference for what is required.

Objectives, constraints and control names are pivoted so that each distinct value has its own column, which is more concise and better to serve via an API. It makes each row in the dataset represent the entire set of controls, objectives, and constraints for one batch | batch -> realization | batch -> realization -> perturbation.

objectives_for_batch.to_dicts()
Out[5]: 
[{'result_id': 0,
  'batch_id': 0,
  'total_objective_value': 1886.010009765625,
  'npv_function': 1886.010009765625}]
objectives_per_real.to_dicts()[:4]
Out[6]: 
[{'batch_id': 0,
  'realization': 0,
  'npv_function': 1886.010009765625,
  'group_w14': 0.0623,
  'group_w01': 0.0627,
  'group_w12': 0.0621,
  'group_w09': 0.0618,
  'group_w03': 0.0629,
  'group_w13': 0.0622,
  'group_w07': 0.0633,
  'group_w04': 0.063,
  'group_w02': 0.0628,
  'group_w10': 0.0619,
  'group_w08': 0.0617,
  'group_w05': 0.0631,
  'group_w11': 0.062,
  'group_w15': 0.0624,
  'group_w06': 0.0632,
  'group_w00': 0.0626},
 {'batch_id': 0,
  'realization': 1,
  'npv_function': 1886.010009765625,
  'group_w14': 0.0623,
  'group_w01': 0.0627,
  'group_w12': 0.0621,
  'group_w09': 0.0618,
  'group_w03': 0.0629,
  'group_w13': 0.0622,
  'group_w07': 0.0633,
  'group_w04': 0.063,
  'group_w02': 0.0628,
  'group_w10': 0.0619,
  'group_w08': 0.0617,
  'group_w05': 0.0631,
  'group_w11': 0.062,
  'group_w15': 0.0624,
  'group_w06': 0.0632,
  'group_w00': 0.0626}]

Objective results:

objectives_for_batch
Out[1]: 
shape: (1, 4)
┌───────────┬──────────┬───────────────────────┬──────────────┐
│ result_id ┆ batch_id ┆ total_objective_value ┆ npv_function │
│ ---       ┆ ---      ┆ ---                   ┆ ---          │
│ i64       ┆ i64      ┆ f64                   ┆ f64          │
╞═══════════╪══════════╪═══════════════════════╪══════════════╡
│ 0         ┆ 0        ┆ 1886.01001            ┆ 1886.01001   │
└───────────┴──────────┴───────────────────────┴──────────────┘
objectives_per_real
Out[2]: 
shape: (2, 19)
┌──────────┬───────────┬───────────┬───────────┬───┬───────────┬───────────┬───────────┬───────────┐
│ batch_id ┆ realizati ┆ npv_funct ┆ group_w14 ┆ … ┆ group_w11 ┆ group_w15 ┆ group_w06 ┆ group_w00 │
│ ---      ┆ on        ┆ ion       ┆ ---       ┆   ┆ ---       ┆ ---       ┆ ---       ┆ ---       │
│ i64      ┆ ---       ┆ ---       ┆ f64       ┆   ┆ f64       ┆ f64       ┆ f64       ┆ f64       │
│          ┆ i64       ┆ f64       ┆           ┆   ┆           ┆           ┆           ┆           │
╞══════════╪═══════════╪═══════════╪═══════════╪═══╪═══════════╪═══════════╪═══════════╪═══════════╡
│ 0        ┆ 0         ┆ 1886.0100 ┆ 0.0623    ┆ … ┆ 0.062     ┆ 0.0624    ┆ 0.0632    ┆ 0.0626    │
│          ┆           ┆ 1         ┆           ┆   ┆           ┆           ┆           ┆           │
│ 0        ┆ 1         ┆ 1886.0100 ┆ 0.0623    ┆ … ┆ 0.062     ┆ 0.0624    ┆ 0.0632    ┆ 0.0626    │
│          ┆           ┆ 1         ┆           ┆   ┆           ┆           ┆           ┆           │
└──────────┴───────────┴───────────┴───────────┴───┴───────────┴───────────┴───────────┴───────────┘
constraints_for_batch
Out[3]: 
shape: (1, 66)
┌───────────┬──────────┬───────────┬───────────┬───┬───────────┬───────────┬───────────┬───────────┐
│ result_id ┆ batch_id ┆ oil_prod_ ┆ oil_prod_ ┆ … ┆ oil_prod_ ┆ oil_prod_ ┆ oil_prod_ ┆ oil_prod_ │
│ ---       ┆ ---      ┆ rate_000  ┆ rate_001  ┆   ┆ rate_012. ┆ rate_013. ┆ rate_014. ┆ rate_015. │
│ i64       ┆ i64      ┆ ---       ┆ ---       ┆   ┆ scaled_vi ┆ scaled_vi ┆ scaled_vi ┆ scaled_vi │
│           ┆          ┆ f64       ┆ f64       ┆   ┆ ola…      ┆ ola…      ┆ ola…      ┆ ola…      │
│           ┆          ┆           ┆           ┆   ┆ ---       ┆ ---       ┆ ---       ┆ ---       │
│           ┆          ┆           ┆           ┆   ┆ f64       ┆ f64       ┆ f64       ┆ f64       │
╞═══════════╪══════════╪═══════════╪═══════════╪═══╪═══════════╪═══════════╪═══════════╪═══════════╡
│ 0         ┆ 0        ┆ 1000.0    ┆ 1000.0    ┆ … ┆ 0.133333  ┆ 0.133333  ┆ 0.133333  ┆ 0.133333  │
└───────────┴──────────┴───────────┴───────────┴───┴───────────┴───────────┴───────────┴───────────┘
constraints_per_real
Out[4]: 
shape: (2, 50)
┌──────────┬───────────┬───────────┬───────────┬───┬───────────┬───────────┬───────────┬───────────┐
│ batch_id ┆ realizati ┆ oil_prod_ ┆ oil_prod_ ┆ … ┆ group_w12 ┆ group_w13 ┆ group_w14 ┆ group_w15 │
│ ---      ┆ on        ┆ rate_000  ┆ rate_001  ┆   ┆ ---       ┆ ---       ┆ ---       ┆ ---       │
│ i64      ┆ ---       ┆ ---       ┆ ---       ┆   ┆ f64       ┆ f64       ┆ f64       ┆ f64       │
│          ┆ i64       ┆ f64       ┆ f64       ┆   ┆           ┆           ┆           ┆           │
╞══════════╪═══════════╪═══════════╪═══════════╪═══╪═══════════╪═══════════╪═══════════╪═══════════╡
│ 0        ┆ 0         ┆ 6000.0    ┆ 6000.0    ┆ … ┆ 0.0621    ┆ 0.0622    ┆ 0.0623    ┆ 0.0624    │
│ 0        ┆ 1         ┆ 6000.0    ┆ 6000.0    ┆ … ┆ 0.0621    ┆ 0.0622    ┆ 0.0623    ┆ 0.0624    │
└──────────┴───────────┴───────────┴───────────┴───┴───────────┴───────────┴───────────┴───────────┘

Gradient results:

obj_gradient_for_batch
Out[4]: 
shape: (1, 4)
┌───────────┬──────────┬──────────────┬────────────────────┐
│ result_id ┆ batch_id ┆ npv_function ┆ npv_function.total │
│ ---       ┆ ---      ┆ ---          ┆ ---                │
│ i64       ┆ i64      ┆ f64          ┆ f64                │
╞═══════════╪══════════╪══════════════╪════════════════════╡
│ 1         ┆ 1        ┆ -0.0         ┆ -0.0               │
└───────────┴──────────┴──────────────┴────────────────────┘
obj_gradient_perturbations
Out[5]: 
shape: (4, 21)
┌───────────┬──────────┬───────────┬───────────┬───┬───────────┬───────────┬───────────┬───────────┐
│ result_id ┆ batch_id ┆ realizati ┆ perturbat ┆ … ┆ group_w08 ┆ group_w07 ┆ group_w01 ┆ group_w05 │
│ ---       ┆ ---      ┆ on        ┆ ion       ┆   ┆ ---       ┆ ---       ┆ ---       ┆ ---       │
│ i64       ┆ i64      ┆ ---       ┆ ---       ┆   ┆ f64       ┆ f64       ┆ f64       ┆ f64       │
│           ┆          ┆ i64       ┆ i64       ┆   ┆           ┆           ┆           ┆           │
╞═══════════╪══════════╪═══════════╪═══════════╪═══╪═══════════╪═══════════╪═══════════╪═══════════╡
│ 1         ┆ 1        ┆ 1         ┆ 0         ┆ … ┆ 0.059048  ┆ 0.068553  ┆ 0.062976  ┆ 0.062975  │
│ 1         ┆ 1        ┆ 0         ┆ 1         ┆ … ┆ 0.061752  ┆ 0.069882  ┆ 0.065536  ┆ 0.061705  │
│ 1         ┆ 1        ┆ 0         ┆ 0         ┆ … ┆ 0.057099  ┆ 0.070677  ┆ 0.055112  ┆ 0.071082  │
│ 1         ┆ 1        ┆ 1         ┆ 1         ┆ … ┆ 0.064173  ┆ 0.066663  ┆ 0.068711  ┆ 0.062269  │
└───────────┴──────────┴───────────┴───────────┴───┴───────────┴───────────┴───────────┴───────────┘
constraint_gradient_for_batch
Out[6]: 
shape: (1, 33)
┌──────────┬───────────┬───────────┬───────────┬───┬───────────┬───────────┬───────────┬───────────┐
│ batch_id ┆ oil_prod_ ┆ oil_prod_ ┆ oil_prod_ ┆ … ┆ oil_prod_ ┆ oil_prod_ ┆ oil_prod_ ┆ oil_prod_ │
│ ---      ┆ rate_004  ┆ rate_003  ┆ rate_006  ┆   ┆ rate_014. ┆ rate_005. ┆ rate_012. ┆ rate_010. │
│ i64      ┆ ---       ┆ ---       ┆ ---       ┆   ┆ scaled    ┆ scaled    ┆ scaled    ┆ scaled    │
│          ┆ f64       ┆ f64       ┆ f64       ┆   ┆ ---       ┆ ---       ┆ ---       ┆ ---       │
│          ┆           ┆           ┆           ┆   ┆ f64       ┆ f64       ┆ f64       ┆ f64       │
╞══════════╪═══════════╪═══════════╪═══════════╪═══╪═══════════╪═══════════╪═══════════╪═══════════╡
│ 1        ┆ 0.0       ┆ 0.0       ┆ 0.0       ┆ … ┆ 0.0       ┆ 0.0       ┆ 0.0       ┆ 0.0       │
└──────────┴───────────┴───────────┴───────────┴───┴───────────┴───────────┴───────────┴───────────┘
constraint_gradient_perturbations
Out[7]: 
shape: (4, 51)
┌──────────┬───────────┬───────────┬───────────┬───┬───────────┬───────────┬───────────┬───────────┐
│ batch_id ┆ realizati ┆ perturbat ┆ oil_prod_ ┆ … ┆ group_w12 ┆ group_w13 ┆ group_w14 ┆ group_w15 │
│ ---      ┆ on        ┆ ion       ┆ rate_000  ┆   ┆ ---       ┆ ---       ┆ ---       ┆ ---       │
│ i64      ┆ ---       ┆ ---       ┆ ---       ┆   ┆ f64       ┆ f64       ┆ f64       ┆ f64       │
│          ┆ i64       ┆ i64       ┆ f64       ┆   ┆           ┆           ┆           ┆           │
╞══════════╪═══════════╪═══════════╪═══════════╪═══╪═══════════╪═══════════╪═══════════╪═══════════╡
│ 1        ┆ 0         ┆ 0         ┆ 6000.0    ┆ … ┆ 0.066225  ┆ 0.067566  ┆ 0.05967   ┆ 0.062201  │
│ 1        ┆ 0         ┆ 1         ┆ 6000.0    ┆ … ┆ 0.069046  ┆ 0.058626  ┆ 0.061307  ┆ 0.059147  │
│ 1        ┆ 1         ┆ 0         ┆ 6000.0    ┆ … ┆ 0.066712  ┆ 0.066355  ┆ 0.067376  ┆ 0.059584  │
│ 1        ┆ 1         ┆ 1         ┆ 6000.0    ┆ … ┆ 0.065746  ┆ 0.069017  ┆ 0.060187  ┆ 0.06393   │
└──────────┴───────────┴───────────┴───────────┴───┴───────────┴───────────┴───────────┴───────────┘

@codecov-commenter
Copy link

codecov-commenter commented Nov 6, 2024

Codecov Report

Attention: Patch coverage is 40.00000% with 3 lines in your changes missing coverage. Please review.

Project coverage is 90.72%. Comparing base (721f3c6) to head (f9c476a).

Files with missing lines Patch % Lines
src/ert/storage/local_ensemble.py 33.33% 2 Missing ⚠️
src/ert/run_models/everest_run_model.py 50.00% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #9161      +/-   ##
==========================================
+ Coverage   90.70%   90.72%   +0.02%     
==========================================
  Files         351      351              
  Lines       21903    21908       +5     
==========================================
+ Hits        19867    19876       +9     
+ Misses       2036     2032       -4     
Flag Coverage Δ
cli-tests 39.23% <20.00%> (+0.02%) ⬆️
gui-tests 71.73% <20.00%> (-0.04%) ⬇️
performance-tests 49.37% <20.00%> (-0.03%) ⬇️
unit-tests 79.63% <40.00%> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@yngve-sk yngve-sk force-pushed the 24.10.25.store-everest-opt-results-in-ertstorage branch 14 times, most recently from 7081e58 to 32193bd Compare November 13, 2024 11:20
@yngve-sk yngve-sk changed the title (wip) Store everest results in ERT storage Store everest results in ERT storage Nov 13, 2024
@yngve-sk yngve-sk marked this pull request as ready for review November 13, 2024 11:26
@yngve-sk yngve-sk force-pushed the 24.10.25.store-everest-opt-results-in-ertstorage branch 13 times, most recently from 5b54ee7 to dda3db9 Compare November 15, 2024 13:06
@yngve-sk yngve-sk force-pushed the 24.10.25.store-everest-opt-results-in-ertstorage branch 6 times, most recently from fb5948d to a737a84 Compare November 18, 2024 11:21
@@ -437,6 +426,8 @@ def run_experiment(
self._result = OptimalResult.from_seba_optimal_result(
seba_storage.get_optimal_result() # type: ignore
)
optimal_result_from_everstorage = self.ever_storage.get_optimal_result()
assert self._result == optimal_result_from_everstorage
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(This is temporary/sanity check to be removed. If all tests pass with this, it means the storage is producing the same optimal results as the old. )

@yngve-sk yngve-sk force-pushed the 24.10.25.store-everest-opt-results-in-ertstorage branch 7 times, most recently from 3b6d06d to 8583aa9 Compare November 25, 2024 07:48
@yngve-sk yngve-sk force-pushed the 24.10.25.store-everest-opt-results-in-ertstorage branch 3 times, most recently from 98b447a to 84d7248 Compare December 5, 2024 12:30
@yngve-sk yngve-sk force-pushed the 24.10.25.store-everest-opt-results-in-ertstorage branch from 84d7248 to 14b9e4a Compare December 10, 2024 08:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Refactor communication and storage of optimization results
3 participants