Skip to content

Commit

Permalink
Update data prep notebook (NVIDIA#8532) (NVIDIA#8533)
Browse files Browse the repository at this point in the history
Signed-off-by: Mingyuan Ma <[email protected]>
Co-authored-by: Ming <[email protected]>
  • Loading branch information
github-actions[bot] and Victor49152 authored Feb 27, 2024
1 parent 60fc43f commit cccd80d
Showing 1 changed file with 2 additions and 0 deletions.
2 changes: 2 additions & 0 deletions tutorials/multimodal/Multimodal Data Preparation.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -430,6 +430,7 @@
"source": [
"from omegaconf import OmegaConf\n",
"precache_cfg = OmegaConf.load(os.path.join(CONF_DIR, \"precache_sd.yaml\"))\n",
"del precache_cfg.encodings[1].encoder_config['use_fp16']\n",
"# visualize the config\n",
"print(OmegaConf.to_yaml(precache_cfg))"
]
Expand Down Expand Up @@ -511,6 +512,7 @@
"metadata": {},
"outputs": [],
"source": [
"! pip install --upgrade webdataset\n",
"! SLURM_ARRAY_TASK_ID=0 SLURM_ARRAY_TASK_COUNT=2 python $SCRIPT_DIR/precache_encodings.py \\\n",
" input_dir=$DATA_DIR/tarfiles_reorganized \\\n",
" output_dir=$DATA_DIR/tarfiles_precached \\\n",
Expand Down

0 comments on commit cccd80d

Please sign in to comment.