Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

LightningDataModule for webdataset #100

Merged
merged 70 commits into from
Aug 30, 2024
Merged
Changes from 1 commit
Commits
Show all changes
70 commits
Select commit Hold shift + click to select a range
0d71fa0
BugFix: parser of docker version string
DejunL Jul 23, 2024
1d30c43
Enhancement: migrate diffdock utils modules from v1
DejunL Jul 29, 2024
9a7f82e
Enhancement: explicitly pass tr/rot/tor_sigma's to t_to_sigma
DejunL Jul 30, 2024
756a2e3
Enhancement: add diffdock score model data module and tests
DejunL Aug 1, 2024
0b8411a
BugFix: control wds shardshuffle randomness
DejunL Aug 2, 2024
76ec1c7
Test: refactor ScoreModelWDS fixture to allow comparing different ins…
DejunL Aug 2, 2024
ca0cc65
Test: consistent ligand position after NoiseTransform
DejunL Aug 2, 2024
7d95a67
BugFix: reduce total_size factor to ensure enough number of shards
DejunL Aug 2, 2024
b82ee03
Test: parametrize by split type
DejunL Aug 2, 2024
8420700
Enhancement: move NoiseTransform to GenerateNoise in the diffusion mo…
DejunL Aug 3, 2024
662ce27
BugFix: always GenerateNoise in score model data
DejunL Aug 3, 2024
99d8e6e
Regress: with_length -> with_epoch
DejunL Aug 3, 2024
e604a49
BugFix: define length on SizeAwareBatching dataset
DejunL Aug 5, 2024
8e236d6
Test: setup data loader
DejunL Aug 5, 2024
0e4c7bd
Enhancement: factor out GenerateNoise outside of data module
DejunL Aug 5, 2024
88035d6
Enhancement: use a dict: Split -> List[str] as the input subset compl…
DejunL Aug 5, 2024
377f276
BugFix: check _xform_gen_wds's avail before composing
DejunL Aug 5, 2024
7c5add1
Enhancement: move the batching from wds to webloader
DejunL Aug 6, 2024
69620fb
Test: assert sample and batch type
DejunL Aug 6, 2024
6ee03c0
Enhancement: factor out the batching method as input webloader compos…
DejunL Aug 6, 2024
598a4ae
Enhancement: remove local_batch_size
DejunL Aug 6, 2024
213095b
BugFix: remove unused random.seed() call
DejunL Aug 6, 2024
1d8f162
Enhancement: rename ScoreModelWDS -> PickledDataWDS
DejunL Aug 6, 2024
add7870
Enhancement: internalize the wds archive formatter
DejunL Aug 6, 2024
7bbc651
Test: rename score model test data directory
DejunL Aug 6, 2024
78c1f80
Test: confidence model data module
DejunL Aug 7, 2024
e13b325
Test: parametrize tests with model type
DejunL Aug 7, 2024
c8a128b
Enhancement: factor out the webdataset-specific method into a base da…
DejunL Aug 8, 2024
80149e2
Enhancement: reduce args redundancy in PickledDataWDS by passing *arg…
DejunL Aug 8, 2024
cc911b0
Enhancement: add integration tests in lightning trainer workflows
DejunL Aug 8, 2024
6541bc2
Enhancement: implement predict stage
DejunL Aug 8, 2024
bc00979
BugFix: remove select for sample size > 1
DejunL Aug 10, 2024
975c91e
Enhancement: remove legacy dataloader config
DejunL Aug 10, 2024
a3b07e2
Enhancement: allow input an iterable of generators passed as pipelines
DejunL Aug 10, 2024
2c8a2c8
Enhancement: move the webdataset data module into bionemo-core
DejunL Aug 13, 2024
b622a2c
Enhancement: rename WDSModule -> WebDataModule
DejunL Aug 13, 2024
1ffde95
Enhancement: actually support Iterable[str] as suffix_keys_wds
DejunL Aug 13, 2024
5948230
Test: WebDataModule with mock-up dataset and model
DejunL Aug 14, 2024
78ed189
Enhancement: remove hard-coded shuffle from workflow
DejunL Aug 14, 2024
01bb63e
Enhancement: rename kwargs_dl -> kwargs_wld
DejunL Aug 14, 2024
c34dd0f
Enhancement: replace hard-coded webdatast kwargs with user input
DejunL Aug 14, 2024
40044fc
Test: move data into bionemo2_root/test_data
DejunL Aug 14, 2024
6f0d4b2
Doc: docstring and readme.md
DejunL Aug 14, 2024
19c3dcd
Test: PickledDataWDS init and setup()
DejunL Aug 14, 2024
2e6e8db
Doc: update class description
DejunL Aug 15, 2024
9693aba
Test: re-org test data location
DejunL Aug 15, 2024
71391a8
Test: generate the test data for webdatamodule
DejunL Aug 15, 2024
eff5093
Enhancement: remove unused functions
DejunL Aug 15, 2024
c8c396b
BugFix: remove license
DejunL Aug 15, 2024
458611d
Enhancement: use list comprehension
DejunL Aug 15, 2024
e6f9c3f
License: clean up old licenses
DejunL Aug 16, 2024
c470b21
BugFix: add __init__.py
DejunL Aug 16, 2024
e1e5551
BugFix: my email
DejunL Aug 19, 2024
0f46ed8
Package: move webdatamodule under bionemo-webdatamodule
DejunL Aug 19, 2024
b8f2bf5
Package: remove bionemo-diffdock
DejunL Aug 19, 2024
57b6c3a
BugFix: change license header
DejunL Aug 20, 2024
009fa23
BugFix: ignore E731 and fix format
DejunL Aug 20, 2024
5d01a19
Regress: use PEP604
DejunL Aug 20, 2024
6bcccb6
Enhancement: pickle within fh context
DejunL Aug 20, 2024
168bc9b
Bugfix: default to python 3.10 for consistency
DejunL Aug 20, 2024
4c611ba
BugFix: f-string typo
DejunL Aug 20, 2024
86f97e3
Enhancement: use sample counts instead of size to enforce number of t…
DejunL Aug 23, 2024
553258d
Test: sample overlap tests via md5 hash of pickles
DejunL Aug 23, 2024
d5284e6
Test: assert throw from webdataset when num_workers > num_shards
DejunL Aug 24, 2024
6e51a7e
Enhancement: reuse the webdatamodule dirs_tars_wds arg in PickledDataWDS
DejunL Aug 28, 2024
5b6f309
Enhancement: allow multiple data objects for each sample
DejunL Aug 28, 2024
8522273
Regress: remove default python version
DejunL Aug 28, 2024
26a59d6
Test: rename test to be consistent with module
DejunL Aug 28, 2024
0d5c144
Test: factor pickle generation into a new fixture
DejunL Aug 29, 2024
a35765f
Test: multiple data objects per sample
DejunL Aug 29, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Enhancement: explicitly pass tr/rot/tor_sigma's to t_to_sigma
DejunL committed Aug 29, 2024
commit 9a7f82ea4f341f53cb4f466b7d4d0204875f41e8
Original file line number Diff line number Diff line change
@@ -19,10 +19,11 @@
from bionemo.contrib.model.molecule.diffdock.utils.torsion import modify_conformer_torsion_angles


def t_to_sigma(t_tr, t_rot, t_tor, cfg):
tr_sigma = cfg.diffusion.tr_sigma_min ** (1 - t_tr) * cfg.diffusion.tr_sigma_max**t_tr
rot_sigma = cfg.diffusion.rot_sigma_min ** (1 - t_rot) * cfg.diffusion.rot_sigma_max**t_rot
tor_sigma = cfg.diffusion.tor_sigma_min ** (1 - t_tor) * cfg.diffusion.tor_sigma_max**t_tor
def t_to_sigma(t_tr, t_rot, t_tor, tr_sigma_min, tr_sigma_max, rot_sigma_min,
rot_sigma_max, tor_sigma_min, tor_sigma_max):
tr_sigma = tr_sigma_min ** (1 - t_tr) * tr_sigma_max**t_tr
rot_sigma = rot_sigma_min ** (1 - t_rot) * rot_sigma_max**t_rot
tor_sigma = tor_sigma_min ** (1 - t_tor) * tor_sigma_max**t_tor
return tr_sigma, rot_sigma, tor_sigma