Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update JSON generator to work with vt's LBDataHolder #527

Closed
1 of 4 tasks
cwschilly opened this issue Sep 5, 2024 · 4 comments · Fixed by #528
Closed
1 of 4 tasks

Update JSON generator to work with vt's LBDataHolder #527

cwschilly opened this issue Sep 5, 2024 · 4 comments · Fixed by #528
Assignees

Comments

@cwschilly
Copy link
Contributor

cwschilly commented Sep 5, 2024

In order for vt to successfully encode information in an object's ID, we need to:


As a model, vt's LBDataHolder has this function:

void LBDataHolder::outputEntity(nlohmann::json& j, ElementIDStruct const& id) const {
  j["type"] = "object";
  j["id"] = id.id;
  j["home"] = id.getHomeNode();
  j["migratable"] = id.isMigratable();
  if (node_idx_.find(id) != node_idx_.end()) {
    auto const& proxy_id = std::get<0>(node_idx_.find(id)->second);
    auto const& idx_vec = std::get<1>(node_idx_.find(id)->second);
    j["collection_id"] = proxy_id;
    for (std::size_t x = 0; x < idx_vec.size(); x++) {
      j["index"][x] = idx_vec[x];
    }
  } else if (node_objgroup_.find(id) != node_objgroup_.end()) {
    auto const& proxy_id = node_objgroup_.find(id)->second;
    j["objgroup_id"] = proxy_id;
  } else {
    // bare handler
  }
}

which is called any time an object is being written, whether in the communications field or tasks field. For example:

      // Adding entity to tasks
      outputEntity(j["tasks"][i]["entity"], id);
      
      // Adding entity to communications["to"] or communications["from"]
      outputEntity(j["communications"][i]["from"], key.fromObj());
      outputEntity(j["communications"][i]["to"], key.toObj());
@cwschilly cwschilly self-assigned this Sep 6, 2024
@cwschilly cwschilly changed the title Add collection_id to generated JSON files Update JSON generator to work with vt's LBDataHolder Sep 6, 2024
@cwschilly cwschilly assigned tlamonthezie and unassigned cwschilly Sep 6, 2024
@ppebay
Copy link
Contributor

ppebay commented Sep 6, 2024

@cwschilly I am unsure I understand from the description, whether seq_id will replace id, or be in addition to it.

In either case, is it correct to understandseq_id in the same sense, as in examples such as below:
screenshot_2024-08-18_at_18 26 40_720
?

@cwschilly
Copy link
Contributor Author

cwschilly commented Sep 6, 2024

@ppebay seq_id should replace id in LBAF's generated JSON files.

The convention we have landed on in vt is:

  • id denotes a bit-encoded ID (ie it contains information regarding the object's home, migratability, and node)
  • seq_id is a standard integer identifier, which is what the LBAF generator currently creates.

Basically, what LBAF is currently calling an id is actually a seq_id in vt.

(We'll need to update LBAF's reader in order to accommodate this change.)

@ppebay
Copy link
Contributor

ppebay commented Sep 6, 2024

Alright, thanks @cwschilly -- and also following discussion with @lifflander I understand that any object would now be identified via a triplet (seq_id, home_id, migratable) in $\mathbb{N}^2\times\mathbb{Z}_2$.

Will the generator be then responsible for turning this triplet into a packed id; or would that be done by vt itself when reading a JSON using that format?

@cwschilly
Copy link
Contributor Author

cwschilly commented Sep 6, 2024

@ppebay vt will read in the plain seq_id and generate the packed id from there. This work is being done in vt PR 2343

This being said, in the future it might be advantageous for LBAF to use the same identification schema as vt. But for now, LBAF's generator is not responsible for packing the ids.

tlamonthezie added a commit that referenced this issue Sep 9, 2024
tlamonthezie added a commit that referenced this issue Sep 9, 2024
…ases example compressed data files
tlamonthezie added a commit that referenced this issue Sep 9, 2024
tlamonthezie added a commit that referenced this issue Sep 9, 2024
tlamonthezie added a commit that referenced this issue Sep 10, 2024
tlamonthezie added a commit that referenced this issue Sep 13, 2024
tlamonthezie added a commit that referenced this issue Sep 13, 2024
tlamonthezie added a commit that referenced this issue Sep 13, 2024
tlamonthezie added a commit that referenced this issue Sep 13, 2024
tlamonthezie added a commit that referenced this issue Sep 17, 2024
tlamonthezie added a commit that referenced this issue Sep 17, 2024
cwschilly pushed a commit that referenced this issue Sep 20, 2024
* #527: output `home` and `migratable` to the communication node

* #527: replace entity field `id` by `seq_id` in output json entity nodes

* #527: rename entity `id` to `seq_id` in synthetic data files

* #527: rename entity `id` to `seq_id` in ccm example data files

* #527: add error in console for non-migrated data sets about new seq_id entity field missing

* #527: update generator entity keys to add communication from and to keys

* #527: rename remaining entity `id` to `seq_id` in synthetic data files

* #527: add helper script to compress or decompress data files

* #527: rename entity `id` to `seq_id` in nolb-8color-16nodes-11firstphases example compressed data files

* #527: convert id to seq_id field in reader for temp development purpose

* #527: enable and add warning on communications invalid object reference in writer

* #527: fix invalid dict key name

* #527: fix missing logic case causing an error during a test

* #527: update validator loader url to get latest one

* #527: update expected error in test_json_data_files_validator_wrong_file_schema against incoming vt new schema

* #527: fix unit test failing on machines with vttv installed

* #527: fix invalid old field name in unit test

* #527: remove unit test duplicate

* #527: add missing unused field objgroup_id to object class

* #527: write communication node even if from or to object is missing

* #527: fix vt writer unit tests to support entity seq_id field

* #527: remove extra blank line

* #527: fix a unit test due to schema update

* #527: rename a file in unit tests

* #527: set acceptance tests max_objects_per_transfer to 1 for imbalance to be zero

* #527: update datasets to rename `id` to `seq_id` in entities and communications from & to

* #527: rename and finalized dataset copy script for possible future reuse

* #527: clean Data reader class and add some typing use

* #527: fix path when running a unit test locally

* #527: remove temporary base url for vt scripts because of vt 2343 merged

* #527: update comments in copy_dataset script

* #527: copy entity unused fields to the communication from and to nodes

* #527: add option to set the task `collection_id` field in json generator

* #527: fix new task specification normalization for json generator

* #527: update json generator unit tests files

* #527: add python 3.12 test environment

* #527: add missing dependency for python 3.12

* #527: remove python 3.12 tests

* #527: fix test configuration file

* #527: fix remaining errors in tests for json data files maker

* #527: fix python 3.10 error

* #527: remove trailing white space

* #527: fix output bad data in writer

* #527: update nolb-8color-16nodes-every50phases dataset

* #527: update nolb-8color-16nodes-data dataset

* #527: update script comment

* #527: try ignore trialing whitespace error in binary file

* #527: restore datasets

* #527: restore a ci file

* #527: enable both `id` and `seq_id` in object class

* #527: fix call of object id in reader and in phase

* #527: fix invalid names

* #527: fix datawriter error

* #527: fix id getter calls

* #527: fix a unit test error

* #527: use `seq_id` instead of `id` for small ids in 3 data sets

* #527: fix reader and writer to deal with id and seq id

* #527: update writer test to keep entity id and seq_id data

* #527: update test datasets to use `seq_id` instead of `id`

* #527: update lbaf data set in unit tests to use `seq_id`instead of `id`

* #527: update synthetic blocks dataset to use entity `seq_id` instead of `id`

* #527: update synthetic lb datasets to use entity `seq_id` instead of `id`

* #527: update yaml specification file for synthetic blocks

* #527: update test JSON data file validator dataset to use entity `seq_id` instead of `id`

* #527: update testsynthetic lb data set with wrong schema to use entity `seq_id` instead of `id`

* #527: regenerate synthetic blocks files with the lb data files maker

* #527: update demo example to CCM example in lb data files maker

* #527: add ccm example specification yaml file

* #527: fix writer dict key order for consistency with vt

* #527: fix json writer communication target migratable value and make pahse ranks a set

* #527: update ccm example dataset by using the LB data files maker

* #527: regenerate synthetic blocks dataset to fix some falsy migratable value in communication targets

* #527: regenerate synthetic lb data and add spec file for the data files maker

* #527: update a doc string

* #527: update an error message

* #527: set `collection_id` on some dataset entities

* #527: make collection_id required and default to 7 for migratable objects

* #527: update copy_dataset utility script to ignore non json files

* #527: clean code in a test file

* #527: make collection_id an attribute of object class

* #527: fix missing data in writer output

* #527: add comment on top of synthetic blocks specification file

* #527: remove commented lines in copy_dataset script

* #527: update documentation in object class

* #527: clarify some comment in test_vt_writer file

* #527: fix import error in unit tests

* #527: replace `shared_block_id` by `shared_id` everywhere

* #527: fix specification save error in data files maker interactive mode

* #527: fix unit test error

* #527: fix logic error in a unit test

* #527: upgrade upload action to version 4

* #527: fix id displayed by the Object __repr__ method

* #527: fix typo
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants