…7841)

* c++ dataloader and built-in image/bbox

* update

* fix error

* fix import error

* fix ci build

* fix vs openmp loop type

* fix warning as error with sign/unsign comp

* sign/unsign comp

* update to pytest

* remove nose

* fix tear_down

* address comments

* thread safe dataset

* address comments

* address comments

* fix

* serial pytest for data download

* fix the error message of reshape()

* Fixing issue #16655 reshape() error message

Co-authored-by: Ubuntu <[email protected]>

Signed-off-by: Serge Panev <[email protected]>

* fixed overwrite of args/aux variables

* fixed spacing

)

* Load the user's locale before performing tests

* Change the locale for the CentOS CI jobs to weed out locale-related bugs

* Mark tests that fail due to the decimal point with xfail

* Run localedef when generating the CentOS CI image

* Cancel some Scala tests when C locale uses a non-standard decimal sep.

* Rename xfail helper to xfail_when_nonstandard_decimal_separator

* Fix scalastyle errors

* Disable more Python tests that fail due to locale-related issues

* Move assumeStandardDecimalSeparator into separate object to fix scaladoc

* Disable the "symbol pow" test when running with non-standard decimal sep

* Disable new tests that fail due to locale-related issues

* fix doc

* fix doc

* fix axis

Co-authored-by: Lin <[email protected]>

* Android build containers

* ARM build containers

* ARM test containers

* Fix naming scheme

* Set WORKDIR at correct location

* port nd.multinomial to npx.sample_categorical

* move to npx.random

…at (#18263)

* FFI new feature

* Feature ffi x 5

* Fix pylint error

* Fix pylint error

* Fix around error

* repeat modified

* update nvidiadocker command & remove cuda compat

* replace cu101 with cuda since compat is no longer to be used

* skip flaky tests

* get rid of ubuntu_build_cuda and point ubuntu_cu101 to base gpu instead of cuda compat

* Revert "skip flaky tests"

This reverts commit 1c720fa.

* revert removal of ubuntu_build_cuda

* add linux gpu g4 node to all steps using g3 in unix-gpu pipeline

Co-authored-by: Lin <[email protected]>

* Revert "Fix and optimize handling of vectorized memory accesses (#17767)"

This reverts commit 5542d03.

* add license to reverted file

* Fix input gradient calculation for bidirectional LSTM

For bidiractional LSTM with number of layers > 2 input gradient calculation was incorrect.
Reason of wrong calculations was overwriting y derivative (dy) tensor by
calculated x derivative (dx) tensor before right2left layer could use dy for own
gradient calculations.
Propsed fix uses additional space to avoid overwriting.

* Fix gradient calculation for GRU

For GRU with number of layers > 2 i2h_weight gradient for
layers in the middle (all except last and first) was incorrect.
Wrong caluculations were caused by assigning output pointer to
input instead of calculating new input pointer.

* Enable tests for GRU and LSTM gradients

* Fix comments

* Change loop iteration deduction

* Add more test cases for fused rnn layers

* finish 5 changes

* move metric.py to gluon, replace mx.metric with mx.gluon.metric in python/mxnet/

* fix importError

* replace mx.metric with mx.gluon.metric in tests/python

* remove global support

* remove macro support

* rewrite BinaryAccuracy

* extend F1 to multiclass/multilabel

* add tests for new F1, remove global tests

* use mxnet.numpy instead of numpy

* fix sanity

* rewrite ce and ppl, improve some details

* use mxnet.numpy.float64

* remove sklearn

* remove reset_local() and get_global in other files

* fix test_mlp

* replace mx.metric with mx.gluon.metric in example

* fix context difference

* Disable -DUSE_TVM_OP on GPU builds

* Fix disable tvm op for gpu runs

* use label.ctx in metric.py; remove gluoncv dependency in test_cvnets

* fix sanity

* fix importError

* remove nose

Co-authored-by: Ubuntu <[email protected]>
Co-authored-by: Leonard Lausen <[email protected]>

…18307)

TVMOP feature is now disabled on GPU builds, which caused this test to fail previously

Fix leak of ndarray objects in the frontend due to reference cycle.

…t initialized (#18306)

* avoid race condition in profiler init

* Update storage_profiler.h

Co-authored-by: Ubuntu <[email protected]>

* run operator tests with naive engine

* fix take tests

* update skip mark

* fix cuda error reset

* adjust tests

* disable parallel testing and naive engine for mkl/mkldnn #18244

* add dlpack functions to npx

* improve tests

* further improve test

* fix comment

…ays built (#18308)

* remove Profiler from the runtime feature list, since its always built

* Update libinfo.cc

* Update RunTime.pm

Co-authored-by: Lin <[email protected]>

The cmake staticbuild script doesn't link openblas statically, but rather
provides the so file to be included in the distribution. This is intentional, as
it simplifies the build script, but update of Jenkinsifle was missed when
switching the CD to rely on cmake.

By default cmake accepts any compiler that supports at least some part of C++17.
In practice this leads to hard to understand error messages at later stages
during compilation, as for example gcc5 supports part of C++17 but does not
support C++17 sufficiently to compile MXNet.

* Disable test_sequential_rnn_cells

* Disable test_np_mixed_precision_binary_funcs

* Disable test_rnn_forward_backward

* ffi_min_max

* pylint fix

Co-authored-by: Hao Jin <[email protected]>

* Fix cmake mkldnn install target. Previously mkldnn headers are installed to CMAKE_INSTALL_INCLUDEDIR instead of CMAKE_INSTALL_INCLUDEDIR/mkldnn

* Fix pypi_package.sh pip/setup.py for mkldnn builds

* init global search ui

* add hover effect to icon and refactor js

* add search bar ui styles

* fix search UI's effect on navbar height

* add fade in/out effect to search ui and navbar

* update search trigger to click and add x button for close

* add version select for search

* fix version typo

* update dropdown

* fix hitsperpage reset after change version

* fix nav trigger not show

* update search border css class name

* make dropdown style consistent

* global search mobile&tablet UI

* adjust mobile search result width

* extract global search related styles to a seperate scss

* restore formatting to existing code

* format & coding style

* fix caret height bug

* add mobile compatible UI

* add license header to js files and update dropdown width

* put docsearch css before main to overrides

* update search result panel height

* dynamically generate version dropdown

* use more accurate selector over search result

* use vh for height

* add comments  to scss

* move versions to Jekyll global variable

* remove redundant version key

* make global search default version the same as website version

Co-authored-by: Yang Shi <[email protected]>

* apply #17283

* fix issue #18060

* fix error

* remove redundant code

* fix CI error

* replace Flase to False

* add 'dtype=False' to set_np()

* fix doc

* default 'arange' default np dtype as int64

Co-authored-by: Yang Shi <[email protected]>

* Testing new S3 bucket without publish step

* Adding publish step & restricted nodes for prod

* Drop restricted on primary node for testing'

* Revert to restricted nodes with publishing

* Add tests

* Fix block_scope

Co-authored-by: Haibin Lin <[email protected]>
Co-authored-by: Lin <[email protected]>

* serial tests

* pytest fixture temp_dir

* address comments

* fix activation numpy bug

* Update test_numpy_gluon.py

* fix

* Fix FInferShape for some ops to support partial type inference

Signed-off-by: Serge Panev <[email protected]>

* Add missing ndim check in in matrix_op-inl.h

Signed-off-by: Serge Panev <[email protected]>

* add boolean support for concatenate (#18213)

* fix binary scalar logic dtype (#16964)

* common_expr test remove

* binary scalar op support scalar dtype

Co-Authored-By: Wentao Xu <[email protected]>

* test_error_fix

Co-authored-by: Wentao Xu <[email protected]>

… universally (#18376)

* Fix reference leak in Block._children

* Activate everywhere

* Disable on CentOS

* Fix

…18365)

* Consider BUILD_SHARED_LIBS flag

* Use MXNET_BUILD_SHARED_LIBS

* prod

* mean

* nan

* sanity

* change kernel

* include

Co-authored-by: Ubuntu <[email protected]>

When "PATH" are not in the environment, we will get error when reading os.environ['PATH'].
Change to os.environ.get('PATH', '') to fix it.

* add doc about default dtype

* fix sanity error

* Remove references to MXNet convenience releases on Maven

These convenience binaries redistributed Category-X binaries in violation of ASF
policies.

References:
- https://www.apache.org/legal/resolved.html#category-x
- https://lists.apache.org/thread.html/rad2fbedc0ae3d0ab9e6f9f453d2b4000f6a318939c7b92783d9e5693%40%3Cdev.mxnet.apache.org%3E
- https://lists.apache.org/thread.html/r738c102492649c754f49e2195985bc765df07df3c8a1e06942e8872f%40%3Cdev.mxnet.apache.org%3E

* Update get_started page

* change _Pragma to #pragma._Pragma not work with msvc

* new plan

* fix lint

* add arg

* fix

* fix lint

* fix

#18400

* Update to OpenBlas 0.3.10 pre-release

Includes OpenMathLib/OpenBLAS#2527

* Enable support for older architectures in OpenBLAS dynamic architecture feature

* Remove mx.model.FeedForward API

* remove more tests

* remove tools depending on FeedForward

* Update model.py

Co-authored-by: Ubuntu <[email protected]>
Co-authored-by: Lin <[email protected]>

Cache priming was previously only handled when using the ./ci/build.py or
./ci/docker_cache.py script. The former is used during most CI jobs. The latter
is used for containers not declared in docker-compose.yml during the "Docker
cache build" job. More and more containers are declared in
docker-compose.yml now, but prior to this commit their cache wasn't primed in the
"Docker cache build" job.

* Update ecosystem.html

* Add optuna.png

#18382

…RT 1 (#18431)

* website - redirect Chinese visitors to Apache Chinese CDN provider

* switch to mod_geoip for most accurate redirecting

Co-authored-by: Yang Shi <[email protected]>

Update basic_layers.py

fix

fix

Update basic_layers.py

fix bug

* Improve log_softmax performance by OneDNN library

* Adapt tests for MKLDNN log_softmax

* Fix lint errors

* Fix indent and comments

By moving the NCCL installation before the COPY runtime_functions.sh statement,
users will not suffer from cache invalidation due to the COPY and don't have to
repeat the nccl installation locally as there will be a cache hit.

* More clear description to `transform_first`

After teaching an MLU-CV class, some students feel confused when they are using the `transform_first` function. So I modified it a bit to make it more clear. :)

* remove space

Co-authored-by: JackieWu <[email protected]>

* part cpu

* index_add forward & test

* fix wrong doc

* fix index_add_sanity_error

* index_update_test

* remove index_update & implement index_add backward

* fix sanity error

* reduce code length

* depart into two file

* test CI compiler

* test CI

* test CI

* reduce mshadow & allow more dtype

* fix sanity error

* fix conflict

* reduce fwd macro code

* reduce bwd macro code

* fix compile error

* tensor ind

* remove cudaMalloc/cudaFree

* fix windows compile error

* fix compile error

* use value instead of references

* remove pragma

* fix naive engine error

* try to pass CI

* fix sanity error

* depart gradient into three node

* resolve comment & initialize mshadow::Shape

* fix werror

Co-authored-by: Ubuntu <[email protected]>
Co-authored-by: Ubuntu <[email protected]>
Co-authored-by: Xingjian Shi <[email protected]>

* stop running test_conv.py script as it no longer exists

* revert adding mkldnn test for cu* builds

Co-authored-by: Yang Shi <[email protected]>

* fix negation

* support boolean type

* Update elemwise_unary_op.h

* Update test_numpy_op.py

* Julia has changed the folder of 7z.exe.

* Update julia/deps/build.jl

7z.exe folder issue resolved

Co-authored-by: Iblis Lin <[email protected]>

Co-authored-by: Iblis Lin <[email protected]>

* dynamically generate general version dropdown

* dynamic generate installation guide version dropdown

* fix type

* use var

Co-authored-by: Yang Shi <[email protected]>

* * Fix einsum Bug

* * Fix sanity

* * Fix one dim start bug

* * Fix test case gt

* init dev guide

* move dev guide above FAQ

* update format and images

* hoist git docs and fix styles

* use relative urls

* remove useless code block

* use consistent url and file name

* update heading

* add apache license header

* init dev guide

* move dev guide above FAQ

* update format and images

* hoist git docs and fix styles

* use relative urls

* remove useless code block

* use consistent url and file name

* update heading

* add apache license header

* update doc - git clone recursive

* reviewing the dev guide - proof reading and text edits

Co-authored-by: Yang Shi <[email protected]>
Co-authored-by: Talia Chopra <[email protected]>

* [MXNET-#16795] Byteps-KVStore: Intergrate Byteps into mxnet as new type of kvstore backend (#17555)

* Add Byteps backend for kvstore

* Add a temp launcher for byteps backend

* make the test fit for byteps kvstore.

* final workable test

* Remove trashy print and logs

* correct comment

* add hostfile for ci test

* add ci test for byteps kvstore

* add visibile devices for byteps-kvstore ci test

* add licenses for tools/byteps_launcher.py

* syntax error

* pylint error (remove unused import like logging)

* pylint error

* pylint error

* enable launching without hostfile (local byteps)

* 1. rename byteps_kvstore.py to byteps.py; 2. shorten the launch option  to ; 3. add instruction for -H and -SH options for launch; 4. add documentation for byteps kvstore in kvstore/base.py: create(name='local')

* edit documentation of KVStoreBase::is_capable(capability); reture fasle for BytePS(KVStoreBase):is_capable(any).

* pylint error

* remove an error of arg.byteps

* use --env option to set workers' environment

* error in byteps-launcher.py

* remove the unpurposed editing mistake in runtime_functions.sh

* disable cpu support for byteps kvstore.

* 1. format the document to avoid julia doc build error;
2. little change to nightly test;
3. add byteps copy right declararation in byteps_launcher.py
4. if args.byteps == True ===> if args.byteps

* remove the --scheduler_ip and --scheduler_port options in launch.py

* 1. maintain the origin value of broadcast and pushpull
2. optimize when out = value or [out]=value
3. add some missing documentation to avoid doc building error.

* Add bytePS to CI

* add dependency

* +integrationtest_ubuntu_gpu_byteps

* add byteps pipeline

* disable a few tests

* remove more tests

* fix permission

* remove apt-get

* fix python path

* improve logging

* fix printns

* add back CI

Co-authored-by: Ubuntu <[email protected]>
Co-authored-by: Piyush Ghai <[email protected]>
Co-authored-by: eric-haibin-lin <[email protected]>
Co-authored-by: eric-haibin-lin <--global>
Co-authored-by: Lin <[email protected]>

* fix byteps logging and declare tensor

* check exceptions and return -1

* print logging in CI

* Update byteps.py

* Update runtime_functions.sh

* add numa dependency

* pin dependency

* Update runtime_functions.sh

* Update Dockerfile.build.ubuntu

* Update runtime_functions.sh

* Update runtime_functions.sh

* Update runtime_functions.sh

* Update runtime_functions.sh

* Update Jenkins_steps.groovy

* remove launcher. use bpslauncher instead.

Co-authored-by: Chaokun Chang <[email protected]>
Co-authored-by: Ubuntu <[email protected]>
Co-authored-by: Piyush Ghai <[email protected]>
Co-authored-by: Lin <[email protected]>
Co-authored-by: Ubuntu <[email protected]>
Co-authored-by: EC2 Default User <[email protected]>
Co-authored-by: Ubuntu <[email protected]>

)

* Update ci/README.md

* Update raspberry pi instructions

This broke the version selector.

Co-authored-by: Yang Shi <[email protected]>

* add image classification tutorial for jetson

* update code to use gluon model zoo; update doc

* referencing MXNet official website for Jetson installation guide

* add config Makefile for jetson

* modify jetson install guide

* implement onnx translation helpers

* bump onnx version to 1.5

* add export only test cases for topk and slice_axis