Skip to content
This repository has been archived by the owner on Jan 15, 2024. It is now read-only.

[Fix][SageMaker] Make sure that the installation works in SageMaker #1348

Merged
merged 2 commits into from
Sep 4, 2020
Merged

[Fix][SageMaker] Make sure that the installation works in SageMaker #1348

merged 2 commits into from
Sep 4, 2020

Conversation

sxjscience
Copy link
Member

Description

Make sure the installation works on SageMaker Notebook with conda_python3 kernel.

cc @dmlc/gluon-nlp-team

@sxjscience sxjscience requested a review from a team as a code owner September 3, 2020 18:48
@sxjscience sxjscience changed the title [WIP][Fix][SageMaker] Make sure that the installation works in SageMaker [Fix][SageMaker] Make sure that the installation works in SageMaker Sep 3, 2020
@sxjscience
Copy link
Member Author

@szha @leezu @barry-jin @szhengac Currently I'm fixing the problem manually but we may consider to add some sagemaker-based unittests to make sure that everything works on SageMaker Notebook.

@codecov
Copy link

codecov bot commented Sep 3, 2020 •
edited
Loading

Codecov Report

Merging #1348 into master will not change coverage.
The diff coverage is n/a.

Impacted file tree graph

@@           Coverage Diff           @@
##           master    #1348   +/-   ##
=======================================
  Coverage   81.75%   81.75%           
=======================================
  Files          52       52           
  Lines        6862     6862           
=======================================
  Hits         5610     5610           
  Misses       1252     1252
Impacted Files Coverage Δ
setup.py 0.00% <ø> (ø)
src/gluonnlp/data/tokenizers/subword_nmt.py 78.50% <0.00%> (-0.94%) ⬇️
src/gluonnlp/data/tokenizers/yttm.py 82.75% <0.00%> (+0.86%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e0b293e...e05e77e. Read the comment docs.

szha
szha reviewed Sep 3, 2020
View reviewed changes
setup.py Outdated
@@ -47,7 +47,7 @@ def find_version(*file_paths):
'tokenizers==0.8.1',
'click>=7.0', # Dependency of youtokentome
'youtokentome>=1.0.6',
'fasttext>=0.9.2'
'fasttext==0.9.1' # Fix to 0.9.1 due to https://github.com/facebookresearch/fastText/issues/1052
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we do != to blacklist some versions instead? my hope is that our CI can catch installation problems as soon as they happen. if you adopt this approach, then we should also specify any minimum version requirement if we have any.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we do fasttext>=0.9.1, !=0.9.2 ? Also, the problem is actually related to the fact that the latest fasttext does not work on SageMaker.

@sxjscience sxjscience merged commit 203d84c into dmlc:master Sep 4, 2020
@sxjscience
Copy link
Member Author

Thanks @preeyank5 for reporting the issue. This should have been fixed. However, we still need to figure out the way to test on SageMaker Notebook instance.

zheyuye added a commit to zheyuye/gluon-nlp that referenced this pull request Oct 20, 2020
@zheyuye @MoisesHer
@sxjscience @barry-jin @hutao965 @leezu @ZiyueHuang
Keep up with others (#2)
23d9d73 
* Fix BERT fp16 bugs, add test (dmlc#1270)

* Fix fp16 bug: not passing dtype to TransformerEncoderLayer

* Re-hybridize after casting & add BERT test

* Skip fp16 test if CPU ctx

* remove debugging messages

Co-authored-by: root <[email protected]>

* [Fix][SageMaker] Make sure that the installation works in SageMaker (dmlc#1348)

* Fasttext to 0.9.1

* Update setup.py

* [CI] Add Codecov and Test Logs (dmlc#1349)

* [Fix] Some minor fixes for AMLC Tutorial (dmlc#1355)

* update

update

update

update

* Update test_utils_misc.py

* update

* update

* Update test_layers.py

* Update misc.py

* Update mobilebert.py

* add in_units and in_channels

* Update __init__.py

* Update mobilebert.py

* Update README.md

* fix test case

* fix

* Update test_utils_misc.py

* fix bug

* [FEATURE] gpt2 generation scripts (dmlc#1354)

* remove prev_len in hybrid_forward parameters

* update

* sample

* update

* add gpt2_1558M

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

Co-authored-by: Hu <[email protected]>

* [Fix] Minor fix for AMLC Tutorial - QA (dmlc#1359)

* update

Update README.md

update

try to use dataclasses

* Update squad_utils.py

* Update preprocessing.py

* Update squad_utils.py

* Update run_squad.py

* [Log Message Improvement] Improve nlp process (dmlc#1362)

* Update learn_subword.py

* Update learn_subword.py

* Update learn_subword.py

* Update apply_subword.py

* Set default ctx in conftest (dmlc#1363)

* Fix the correctness of the Horovod support on squad (dmlc#1353)

* revise squad

* tiny fix

* fix total_norm logging

* shuffle before and after splitting

* make pre_shuffle_seed fixed

* fix flags

* remove do_pre_shuffle

* remove inside_split_shuffle

Co-authored-by: Ubuntu <[email protected]>

* [CI][BUGFIX] Custom Step for Uploading Code Coverage in Pull Request Event (dmlc#1364)

* [FEATURE]Generation script improvement (dmlc#1365)

* update

* update

* update

* update

* update

* udpate

* update

* update

* update

* update

Co-authored-by: Hu <[email protected]>

* [Website][CI] Build Website without Warnings + Add Workflow for Building Website  (dmlc#1327)

* [Website] Documentation warnings Fixed + Create Makefile

[Website] Documentation bug fix

[Website] Bug fix

[Website] Build without model_zoo

[Website] Fix notebook

* [Website][CI] Add workflow for building website

* [CI] Add more dependencies

* [CI] Update buildwebsite.yml

[CI] Update buildwebsite.yml

* [CI] Update buildwebsite.yml

* [CI] Update buildwebsite.yml

* [CI] Update buildwebsite.yml

* [CI] Update buildwebsite.yml

* [CI] Update buildwebsite.yml

* [CI] Update buildwebsite.yml

* [CI] Update buildwebsite.yml

* [CI] Update buildwebsite.yml

[CI] Update buildwebsite.yml

[CI] Update buildwebsite.yml

[CI] Update buildwebsite.yml

[CI] Update buildwebsite.yml

[CI] Update buildwebsite.yml

[CI] Update buildwebsite.yml

[CI] Update buildwebsite.yml

[CI] Update buildwebsite.yml

* [CI] Update buildwebsite.yml

* [CI] Update buildwebsite.yml

* [Website] Add more dependencies

* [Website][CI] Add Compile notebook step + Preview website

* [CI] Add shell script for compiling notebooks

* [CI] Add permission for shell script

* [Website] Update

* [Website] Update

* [CI] Add uploading build artifacts

* [CI] Update

* [CI] Update Indentation

* [CI] Remove some dependencies

* [BUGFIX] Fix URL encoding (dmlc#1370)

* [FEATURE]Update readme of nmt (dmlc#1373)

* update

* update

* update

* update

* update

* update

* update

* update

Co-authored-by: Hu <[email protected]>

* [CI] Improve website building workflow (dmlc#1377)

* BERT pretraining (dmlc#1376)

* bert

* update

* address comments

* update

* [Fix][Docker] Fix the docker image + Fix pretrain_corpus document. (dmlc#1378)

* update

* Update ubuntu18.04-devel-gpu.Dockerfile

* fix the docker image

* Update README.md

* Update ubuntu18.04-devel-gpu.Dockerfile

* Update README.md

* fix readme

* Add CPU DockerFile

* update

* update

* Update ubuntu18.04-devel-gpu.Dockerfile

* update

* prepare to add TVM to docker

* try to update

* Update ubuntu18.04-devel-gpu.Dockerfile

* Update ubuntu18.04-devel-gpu.Dockerfile

* Update install_openmpi.sh

* update

* Create install_llvm.sh

* Update ubuntu18.04-base-gpu.Dockerfile

* Update ubuntu18.04-base-gpu.Dockerfile

* Update run_squad2_albert_base.sh

* Update prepare_squad.py

* Update prepare_squad.py

* Update prepare_squad.py

* fix

* Update README.md

* update

* update

* Update README.md

* Update README.md

* Update ubuntu18.04-devel-gpu.Dockerfile

* update

* Update README.md

* fix

* Update ubuntu18.04-base-cpu.Dockerfile

* update

* add tvm to lazy import

* update

* Update README.md

* update

* Update README.md

* Update run_squad2_albert_base.sh

* update

* update

* update

* update

* update

* Update README.md

* Update install_ubuntu18.04_core.sh

* update

* update

* update

* fix

* Update README.md

* Update run_batch_squad.sh

* update

* Update run_batch_squad.sh

* Update run_batch_squad.sh

* update

* Update README.md

* fix

* Update gluon_nlp_job.sh

* update

* Update README.md

* Update README.md

* Update README.md

* update

* Update README.md

* update

* Update install_python_packages.sh

* Update install_llvm.sh

* Update install_python_packages.sh

* Update install_llvm.sh

* update

* Update install_ubuntu18.04_core.sh

* fix

* Update submit-job.py

* Update submit-job.py

* Update README.md

* Update README.md

* Update prepare_gutenberg.py

* Delete gluon_nlp_cpu_job.sh

* Update prepare_gutenberg.py

* Update prepare_gutenberg.py

* Update prepare_gutenberg.py

* Update conf.py

* update

* Update generate_commands.py

* fix readme

* use os.link for hard link

* Update README.md

* Update README.md

* Update gluon_nlp_job.sh

* Update __init__.py

* Update benchmark_utils.py

* try to use multi-stage build

* Update benchmark_utils.py

* multi-stage build

* Update README.md

* Update README.md

* update

* Update submit-job.py

* fix documentation

* fix

* update

* Update test.sh

* Update test.sh

* Update test.sh

* Update test.sh

* Update README.md

* Update test.sh

* fix

* Update README.md

* Update gluon_nlp_job.sh

* [Website] Add AMLC Tutorial to Website (dmlc#1379)

* [Website] Add AMLC Tutorial

* [Website] Add tsv encoding

* [Website] Add model zoo

* [Website] Update Makefile

* [Website] Update Makefile

* [Website] Update Makefile

* [Website] Update compile_notebooks.sh

* [Website] Update Makefile

* [Website] Add title to generation

* [Website] Update workflow

* update

* [Website] Update model_zoo.rst

* [Website] Update model_zoo.rst

* [BUGFIX] Fix Codecov (dmlc#1391)

* Update coveragerc

* Update coveragerc

* Update coveragerc

* Update workflow

* Update workflow

* update

* update

Co-authored-by: MoisesHer <[email protected]>
Co-authored-by: root <[email protected]>
Co-authored-by: Xingjian Shi <[email protected]>
Co-authored-by: barry-jin <[email protected]>
Co-authored-by: ht <[email protected]>
Co-authored-by: Hu <[email protected]>
Co-authored-by: Leonard Lausen <[email protected]>
Co-authored-by: Ubuntu <[email protected]>
Co-authored-by: Ziyue Huang <[email protected]>
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@szha szha szha left review comments

@zheyuye zheyuye zheyuye approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@sxjscience @szha @zheyuye