-
Notifications
You must be signed in to change notification settings - Fork 2
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Update JSON generator to work with vt's LBDataHolder
#527
Comments
collection_id
to generated JSON filesLBDataHolder
@cwschilly I am unsure I understand from the description, whether In either case, is it correct to understand |
@ppebay The convention we have landed on in vt is:
Basically, what LBAF is currently calling an (We'll need to update LBAF's reader in order to accommodate this change.) |
Alright, thanks @cwschilly -- and also following discussion with @lifflander I understand that any object would now be identified via a triplet ( Will the generator be then responsible for turning this triplet into a packed |
@ppebay vt will read in the plain This being said, in the future it might be advantageous for LBAF to use the same identification schema as vt. But for now, LBAF's generator is not responsible for packing the ids. |
…ases example compressed data files
…ile_schema against incoming vt new schema
…e value in communication targets
* #527: output `home` and `migratable` to the communication node * #527: replace entity field `id` by `seq_id` in output json entity nodes * #527: rename entity `id` to `seq_id` in synthetic data files * #527: rename entity `id` to `seq_id` in ccm example data files * #527: add error in console for non-migrated data sets about new seq_id entity field missing * #527: update generator entity keys to add communication from and to keys * #527: rename remaining entity `id` to `seq_id` in synthetic data files * #527: add helper script to compress or decompress data files * #527: rename entity `id` to `seq_id` in nolb-8color-16nodes-11firstphases example compressed data files * #527: convert id to seq_id field in reader for temp development purpose * #527: enable and add warning on communications invalid object reference in writer * #527: fix invalid dict key name * #527: fix missing logic case causing an error during a test * #527: update validator loader url to get latest one * #527: update expected error in test_json_data_files_validator_wrong_file_schema against incoming vt new schema * #527: fix unit test failing on machines with vttv installed * #527: fix invalid old field name in unit test * #527: remove unit test duplicate * #527: add missing unused field objgroup_id to object class * #527: write communication node even if from or to object is missing * #527: fix vt writer unit tests to support entity seq_id field * #527: remove extra blank line * #527: fix a unit test due to schema update * #527: rename a file in unit tests * #527: set acceptance tests max_objects_per_transfer to 1 for imbalance to be zero * #527: update datasets to rename `id` to `seq_id` in entities and communications from & to * #527: rename and finalized dataset copy script for possible future reuse * #527: clean Data reader class and add some typing use * #527: fix path when running a unit test locally * #527: remove temporary base url for vt scripts because of vt 2343 merged * #527: update comments in copy_dataset script * #527: copy entity unused fields to the communication from and to nodes * #527: add option to set the task `collection_id` field in json generator * #527: fix new task specification normalization for json generator * #527: update json generator unit tests files * #527: add python 3.12 test environment * #527: add missing dependency for python 3.12 * #527: remove python 3.12 tests * #527: fix test configuration file * #527: fix remaining errors in tests for json data files maker * #527: fix python 3.10 error * #527: remove trailing white space * #527: fix output bad data in writer * #527: update nolb-8color-16nodes-every50phases dataset * #527: update nolb-8color-16nodes-data dataset * #527: update script comment * #527: try ignore trialing whitespace error in binary file * #527: restore datasets * #527: restore a ci file * #527: enable both `id` and `seq_id` in object class * #527: fix call of object id in reader and in phase * #527: fix invalid names * #527: fix datawriter error * #527: fix id getter calls * #527: fix a unit test error * #527: use `seq_id` instead of `id` for small ids in 3 data sets * #527: fix reader and writer to deal with id and seq id * #527: update writer test to keep entity id and seq_id data * #527: update test datasets to use `seq_id` instead of `id` * #527: update lbaf data set in unit tests to use `seq_id`instead of `id` * #527: update synthetic blocks dataset to use entity `seq_id` instead of `id` * #527: update synthetic lb datasets to use entity `seq_id` instead of `id` * #527: update yaml specification file for synthetic blocks * #527: update test JSON data file validator dataset to use entity `seq_id` instead of `id` * #527: update testsynthetic lb data set with wrong schema to use entity `seq_id` instead of `id` * #527: regenerate synthetic blocks files with the lb data files maker * #527: update demo example to CCM example in lb data files maker * #527: add ccm example specification yaml file * #527: fix writer dict key order for consistency with vt * #527: fix json writer communication target migratable value and make pahse ranks a set * #527: update ccm example dataset by using the LB data files maker * #527: regenerate synthetic blocks dataset to fix some falsy migratable value in communication targets * #527: regenerate synthetic lb data and add spec file for the data files maker * #527: update a doc string * #527: update an error message * #527: set `collection_id` on some dataset entities * #527: make collection_id required and default to 7 for migratable objects * #527: update copy_dataset utility script to ignore non json files * #527: clean code in a test file * #527: make collection_id an attribute of object class * #527: fix missing data in writer output * #527: add comment on top of synthetic blocks specification file * #527: remove commented lines in copy_dataset script * #527: update documentation in object class * #527: clarify some comment in test_vt_writer file * #527: fix import error in unit tests * #527: replace `shared_block_id` by `shared_id` everywhere * #527: fix specification save error in data files maker interactive mode * #527: fix unit test error * #527: fix logic error in a unit test * #527: upgrade upload action to version 4 * #527: fix id displayed by the Object __repr__ method * #527: fix typo
In order for vt to successfully encode information in an object's ID, we need to:
collection_id
to eachentity
fieldentity
in thecommunications
to/from fields (includinghome
, andmigratable
)seq_id
instead ofid
for objects/tasks (sinceid
now means that the value is encoded)shared_id
instead ofshared_block_id
everywhere #531As a model, vt's
LBDataHolder
has this function:which is called any time an object is being written, whether in the communications field or tasks field. For example:
The text was updated successfully, but these errors were encountered: