[Relay] ConvertLayout pass. #4335

anijain2305 · 2019-11-14T04:48:00Z

RFC - https://discuss.tvm.ai/t/layout-conversion-pass/4009

Refactored to share code between AlterOpLayout and ConvertLayout pass.

There is considerable refactoring. Following description might help

Older alter_op_layout.h is renames as infer_layout_util.h that contains only the FInferCorrect Related methods.
transform_layout.h has now several methods from older alter_op_layout.cc. The idea is to put the shared code among AlterOpLayout and ConvertLayout pass in transform_layout.h.
Both AlterOpLayout and ConvertLayout use FForwardRewrite. FForwardRewrite accepts a context. This context now has a transform function which is specialized for ConvertOpLayout and AlterOpLayout separately. For example, ConvertOpLayout context-transform function extracts op's "FTVMConvertOpLayout" attribute, and and sends the user-supplied-desired layout to the packed function.

yzhliu · 2019-11-14T19:33:57Z

I'm ok with deprecating plevel. @tqchen what's your thought?

tqchen · 2019-11-15T02:04:50Z

I feel live patching the OpAttr is not a pattern we are looking for in here. It would be great if we can factor out some common logic between AlterOpLayout and ConvertLayout(e.g. a base class), so ConvertLayout can directly inject the necessary functions

anijain2305 · 2019-11-15T21:20:39Z

Ok, let me spend some more time thinking about this :)

anijain2305 · 2019-11-19T23:46:46Z

@tqchen Refactored to share code between AlterOpLayout and ConvertLayout pass.

There is considerable refactoring. Following description might help

Older alter_op_layout.h is broken into 2 files - infer_layout_util.h and transform_layout.h.
infer_layout_util.h contains the FInferCorrect related methods.
transform_layout.h has now several methods from older alter_op_layout.cc. The idea is to put the shared code among AlterOpLayout and ConvertLayout pass in transform_layout.h.
Both AlterOpLayout and ConvertLayout use FForwardRewrite. FForwardRewrite accepts a context. This context now has a transform function which is specialized for ConvertOpLayout and AlterOpLayout separately. For example, ConvertOpLayout context-transform function extracts op's "FTVMConvertOpLayout" attribute, and and sends the user-supplied-desired layout to the packed function.

@yzhliu

anijain2305 · 2019-11-21T18:04:45Z

@tqchen @yzhliu @zhiics Let me know if you have some cycles to review this. It will be good to know if you think the high-level design makes sense.

anijain2305 · 2019-11-25T02:23:48Z

Ping for review :)

yzhliu · 2019-11-25T22:16:25Z

will do shortly

yzhliu

high level looks good to me.

yzhliu · 2019-11-26T01:46:48Z

src/relay/pass/alter_op_layout.cc

+   * \param new_args The traversed/recursed args to the call.
+   * \return The new Call after calling the packed func.
+   */
+  Call GetCallWithNewLayouts(const Call& ref_call, const std::vector<Expr>& new_args) const {


how about name it AlterCall ?

This function name is shared between Alter and Convert. But, I will try to come up with a better name.

yzhliu · 2019-11-26T02:04:23Z

src/relay/pass/transform_layout.h

+ */
+
+/*!
+ *  Copyright (c) 2019 by Contributors


remember to remove all copyright

yzhliu · 2019-11-26T02:11:46Z

src/relay/pass/transform_layout.h

+
+/*!
+ *  Copyright (c) 2019 by Contributors
+ * \file transform_layout.h


would be better to move implement into .cc

The impl is shared between AlterOp and ConvertLayout. So, it has to be in .h file.

well, I feel .h can contain just the data structure and function signature, while large implement of the function can be moved into corresponding transform_layout.cc

I checked again, most of the functions are templated, requiring definition in the header file. What do you think?

anijain2305 · 2019-11-28T14:40:59Z

@yzhliu This is ready for review. I made a couple of more refactoring that still keeps the high level same, and also added docs.

anijain2305 · 2019-12-01T09:21:20Z

Ping @yzhliu

yzhliu

overall LGTM. sorry I missed the reminding last week.

yzhliu · 2019-12-10T01:48:57Z

src/relay/pass/transform_layout.h

+
+/*!
+ *  Copyright (c) 2019 by Contributors
+ * \file transform_layout.h


well, I feel .h can contain just the data structure and function signature, while large implement of the function can be moved into corresponding transform_layout.cc

kice · 2019-12-14T20:19:47Z

I have many transpose when using tensor core due to NCHW. About 20% of the time is doing transpose.

So, I want a NCHW to NHWC converter.

anijain2305 · 2019-12-23T19:54:06Z

I have many transpose when using tensor core due to NCHW. About 20% of the time is doing transpose.

So, I want a NCHW to NHWC converter.

That is doable. I will first work on getting this PR in. This will setup the whole infrastructure. The NCHW to NHWC layout changes then can go as a second PR.

yzhliu · 2019-12-26T19:16:08Z

Thanks @anijain2305 this is merged. Sorry for the delay.

anijain2305 · 2019-12-26T20:45:59Z

Thanks @yzhliu
No worries about the delay :)

@cchung100m

* Change upstream url * Fix bias_add gradient (apache#4516) * Fix bias_add gradient A change caused collapse_sum_like to reject implicit dimension broadcasting for bias_add gradient, so switch to explicit sum reduction on the non-bias axis dimensions. * Lint fix * [Bugfix][Frontend][TFlite] Fix wrong function call in TANH tests (apache#4517) * Replace sigmoid() with tanh() in tests for TANH * Fixed extra reshape parameter bug. (apache#4524) * Use the best tuner possible (apache#4397) * Use the best tuner possible * Add comment denoting availability of better tuners * Fix typos and wording * [ir] use DataType instead of Type for readability because Type has been deprecated (apache#4513) * add bfloat16 typeflag support (apache#4525) * fix empty config caused KeyError (apache#4520) * fix onnx shape dtype (apache#4528) * fix crash issue in tsim backend (apache#4527) * PIL is depreciated and should be replaced with pillow (a fork of PIL) (apache#4533) Change-Id: If2075df5475505f2da87dae7145af5a7ab83d8a4 * [Relay] External codegen (apache#4482) * Update legacy places from nnvm to relay. (apache#4535) * Update legacy places from nnvm to relay. This PR prepares the current mainline to remove nnvm compiler dep. * remove legacy stage * Implement 1d deconvolution (apache#4476) * [relay][op] add expand op (from ONNX) to relay frontend (apache#4483) * Add Expand to onnx.py * add test function for expand * Fix a onnx frontend test * Add tests for the value itself instead of shape only on test_expand * Cleaned up some unnecessary modifications. * [TOPI] Allow batch matmul to be fused into injective ops (apache#4537) * [TOPI] Fixed nms max_output_size loop (apache#4541) One of the loops in hybrid_nms used for performing the max_output_size reordering was incorrectly designated as parallel resulting in incorrect behaviour. This patch changes that loop to a serial loop. Change-Id: I97184f5887f5f028d8ab339fa2808eb7630a4017 * [DOCS] Mention Ninja build system in install/from_source.rst (apache#4554) * [DOCS] Mention Ninja build system in install/from_source.rst * Address comments * [PYTHON][FFI] Cythonize NDArray.copyto (apache#4549) * [PYTHON][FFI] Cythonize NDArray.copyto * Cythonize the shape property * vm external codegen (apache#4544) * [COMMUNITY] @cchung100m -> reviewer (apache#4557) * [VTA] improved virtual memory mapping (apache#4545) * [VTA] improved virtual memory mapping * Update virtual_memory.cc * [IR] fix style in ir_mutator and ir_visitor (apache#4561) * [RUNTIME][VULKAN] Fix compiler warning (apache#4559) * [REFACTOR][DTYPE] Isolate dtype to runtime (apache#4560) dtype.h -> runtime/data_type.h Changes: - Rename all old reference of tvm::Type to DataType - ExprNode.type -> ExprNode.dtype - Expr.type() -> Expr.dtype() - Change Expr related functions to expr_operator. - DataType::min() -> min_value(DataType) - DataType::max() -> max_value(DataType) - Move type constructor Int, UInt, Float, Handle, Bool into DataType. - Int(bits) -> DataType::Int(bits) - UInt(bits) -> DataType::UInt(bits) * Support standardize runtime module (apache#4532) * [Relay][Frontend][ONNX] Support auto_pad in Conv and ConvTranspose (apache#4563) * [TEST] Remove nnvm related code in topi and test script (apache#4562) * [TEST] Remove nnvm related code in topi and test script * Remove docs dep * [Relay] add max_pool3d in relay and TF converter (apache#4551) * [Relay] add max_pool3d in relay and TF converter * fix comments * Remove nnvm (apache#4565) * [VTA][Chisel] End-to-end Inference with Chisel VTA (apache#4574) * [VTA][Chisel] End-to-end Inference with Chisel VTA * Update TensorAlu.scala * remove unnecessary cast to int32 (apache#4573) * Fix llvm-enabled build by adding missing intrinsics headers (apache#4575) * [DEPRECATION] Remove NNVM compiler (apache#4571) * Remove NNVM compiler * [Relay/Topi][Op] Added native DepthToSpace and SpaceToDepth Operators (apache#4566) * Added tvm function stencil for subpixel operations to topi. * Topi subpixel operators added and tested. * Added subpixel attrs. * Added depth_to_space relay attributes. * depth_to_space fully working. * Fixed NHWC shape bug. * SpaceToDepth in and all tests passing. * lint fixes. * Added string include * Fixed topi formatting. * Added DCR/CDR mode to depthtospace operator. * [DOC] fix doc in api.py (apache#4580) * [DEPRECATION] Cleanup legacy verilog support (apache#4576) This PR cleans up the left over code for legacy verilog support which was experimental. The new hardware backend path is now support by VTA via TSIM. * [RUNTIME] Remove Extension VTable in favor of Unified Object system. (apache#4578) Before the unified object protocol, we support pass additional extension objects around by declaring a type as an extension type. The old extension mechanism requires the types to register their constructor and deleter to a VTable and does not enjoy the benefit of the self-contained deletion property of the new Object system. This PR upgrades the extension example to make use of the new object system and removed the old Extension VTable. Note that the register_extension funtion in the python side continues to work when the passed argument does not require explicit container copy/deletion, which covers the current usecases of the extension mechanism. * Some Windows and MSVC fixes (apache#4569) * fix python exception creation in Windows * better string conversion for msvc * fix cpp style issue * [NEWS] add v0.6 release (apache#4558) * [NEWS] add v0.6 release * remove link prefix * fix issue number * [DOCS]fix typos in autotvm tutorial (apache#4585) * [Quantization, Calibrate] Fix context creation when current_target is explicity set (apache#4582) * [Container] Fix NDArray SaveDLTensor declaration and implementation signature different (apache#4586) * [TOPI][AutoTVM] NHWC conv2d templates for ARM (apache#3859) * [AutoTVM][TOPI] NHWC conv2d templates (spatial pack) for ARM As some frontends (tflite for example) are using NHWC as the default layout, we are enabling NHWC schedule templates in TOPI and AutoTVM. * some comments fix * [FIX][TOPI][X86] schedule dense pack (apache#4539) * [Relay] Convert Layout Pass. (apache#4335) * [Relay][AlterLayout] Broadcast with scalar shape (apache#4577) * [TOPI] add 3D upsampling Op. (apache#4584) * [TOPI] add 3D upsampling Op. * fix lint issues * change align_corners to coordinate_transformation_mode * fix resize3d half_pixel * make a simple function and clean up trilinear_resize3d_python * fix doc * [Runtime] add necessary const qualifier for NDArray container of parameters (apache#4590) * [autotvm] fix typos in comment (apache#4591) * fix tf.compat.v1 issue for tf verison <=1.12 (apache#4593) * [FRONTEND][TF] conv2d_transpose 'SAME' support kernel more than 1x1 (apache#4484) * [FRONTEND][TF] conv3d_transpose 'SAME' support kernel more than 1x1 * revised per as review comments * add more fallback wolkaround to make all tests pass * [GraphRuntime] Support parameter out in the graph runtime debug (apache#4598) * [GraphRuntime] Support parameter out in the graph runtime debug * Dummy commit to trigger build * [Perf] Add CublasLt extern support for better Igemm performance (apache#4550) * cublaslt added * fix lint * address comments * address more comments * Trigger CI * Trigger CI * fix codegenc (apache#4597) * [REFACTOR][RUNTIME] Update NDArray use the Unified Object System (apache#4581) * [REFACTOR][RUNTIME] Move NDArray to Object System. Previously NDArray has its own object reference counting mechanism. This PR migrates NDArray to the unified object protocol. The calling convention of NDArray remained intact. That means NDArray still has its own type_code and its handle is still DLTensor compatible. In order to do so, this PR added a few minimum runtime type detection in TVMArgValue and RetValue only when the corresponding type is a base type(ObjectRef) that could also refer to NDArray. This means that even if we return a base reference object ObjectRef which refers to the NDArray. The type_code will still be translated correctly as kNDArrayContainer. If we assign a non-base type(say Expr) that we know is not compatible with NDArray during compile time, no runtime type detection will be performed. This PR also adopts the object protocol for NDArray sub-classing and removed the legacy NDArray subclass protocol. Examples in apps/extension are now updated to reflect that. Making NDArray as an Object brings all the benefits of the object system. For example, we can now use the Array container to store NDArrays. * Address review comments * [Relay][Convert Layout] Handling batch norm layout change. (apache#4600) * [relay][refactor] Cache Op::Get in passes to reduce lookup overhead (apache#4594) * Refactor to use IsOp utility * retrigger CI * Update dmlc_tvm_commit_id.txt * disable one test_batch_norm unit test for now to check CI * enable test_batch_norm Co-authored-by: SWu <[email protected]> Co-authored-by: Ina Dobreva <[email protected]> Co-authored-by: Josh Fromm <[email protected]> Co-authored-by: miheer vaidya <[email protected]> Co-authored-by: Liang ZOU <[email protected]> Co-authored-by: YixinBao <[email protected]> Co-authored-by: Cody Yu <[email protected]> Co-authored-by: masahi <[email protected]> Co-authored-by: Liangfu Chen <[email protected]> Co-authored-by: lhutton1 <[email protected]> Co-authored-by: Tianqi Chen <[email protected]> Co-authored-by: Alex Gladkov <[email protected]> Co-authored-by: Takato Yamada <[email protected]> Co-authored-by: Haichen Shen <[email protected]> Co-authored-by: mbarrett97 <[email protected]> Co-authored-by: Hideto Ueno <[email protected]> Co-authored-by: Siyuan Feng <[email protected]> Co-authored-by: Zhao Wu <[email protected]> Co-authored-by: Neo Chien <[email protected]> Co-authored-by: Yong Wu <[email protected]> Co-authored-by: Dmitri Makarov <[email protected]> Co-authored-by: Bohan Hou <[email protected]> Co-authored-by: kice <[email protected]> Co-authored-by: Yizhi Liu <[email protected]> Co-authored-by: Wang Yucheng <[email protected]> Co-authored-by: 王振华(Zhenhua WANG) <[email protected]> Co-authored-by: deepIgnorance <[email protected]> Co-authored-by: Animesh Jain <[email protected]> Co-authored-by: optima2005 <[email protected]> Co-authored-by: zhuochen <[email protected]> Co-authored-by: Leyuan Wang <[email protected]>

anijain2305 force-pushed the convert_layout branch from 1fb5cc4 to a066107 Compare November 14, 2019 19:31

anijain2305 force-pushed the convert_layout branch 2 times, most recently from 1a764d8 to 76d2092 Compare November 14, 2019 19:47

anijain2305 force-pushed the convert_layout branch from 76d2092 to b356a13 Compare November 19, 2019 23:35

tqchen assigned yzhliu and ZihengJiang Nov 26, 2019

yzhliu reviewed Nov 26, 2019

View reviewed changes

anijain2305 force-pushed the convert_layout branch from b356a13 to ad222fc Compare November 27, 2019 01:58

anijain2305 changed the title ~~[Relay][WIP] ConvertLayout pass.~~ [Relay] ConvertLayout pass. Nov 27, 2019

anijain2305 force-pushed the convert_layout branch 3 times, most recently from 3b6010b to 0bbee0b Compare November 28, 2019 11:28

anijain2305 force-pushed the convert_layout branch from 0bbee0b to b45ae89 Compare November 29, 2019 10:22

yzhliu reviewed Dec 10, 2019

View reviewed changes

kice mentioned this pull request Dec 15, 2019

Optimization for subpixel layer on Tensor core #4523

Closed

[Relay] Convert Layout Pass.

f930b23

anijain2305 force-pushed the convert_layout branch from b45ae89 to f930b23 Compare December 23, 2019 20:57

yzhliu approved these changes Dec 26, 2019

View reviewed changes

yzhliu merged commit 73dda6b into apache:master Dec 26, 2019

anijain2305 mentioned this pull request Dec 26, 2019

[AutoTVM] Tuning fails for an NHWC network on Arm CPU #4542

Closed

zhiics pushed a commit to zhiics/tvm that referenced this pull request Dec 31, 2019

[Relay] Convert Layout Pass. (apache#4335)

628a660

zhiics pushed a commit to neo-ai/tvm that referenced this pull request Jan 11, 2020

[Relay] Convert Layout Pass. (apache#4335)

ebde64f

Beya2019 mentioned this pull request Aug 28, 2020

[Relay] add conv2d_transpose alter layout #6358

Merged

This was referenced Sep 10, 2020

[RELAY][OP] roi_align operator alter layout #6439

Closed

[RELAY][OP] roi_align operator alter layout #6443

Merged

zhiics mentioned this pull request Sep 15, 2020

TVM v0.7 Release Note Candidate #6486

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Relay] ConvertLayout pass. #4335

[Relay] ConvertLayout pass. #4335

anijain2305 commented Nov 14, 2019 •

edited

Loading

yzhliu commented Nov 14, 2019

tqchen commented Nov 15, 2019

anijain2305 commented Nov 15, 2019

anijain2305 commented Nov 19, 2019

anijain2305 commented Nov 21, 2019

anijain2305 commented Nov 25, 2019

yzhliu commented Nov 25, 2019

yzhliu left a comment

yzhliu Nov 26, 2019

anijain2305 Nov 26, 2019

yzhliu Nov 26, 2019

yzhliu Nov 26, 2019

anijain2305 Nov 26, 2019 •

edited

Loading

yzhliu Dec 10, 2019

anijain2305 Dec 23, 2019

anijain2305 commented Nov 28, 2019

anijain2305 commented Dec 1, 2019

yzhliu left a comment

yzhliu Dec 10, 2019

kice commented Dec 14, 2019

anijain2305 commented Dec 23, 2019

yzhliu commented Dec 26, 2019

anijain2305 commented Dec 26, 2019

[Relay] ConvertLayout pass. #4335

[Relay] ConvertLayout pass. #4335

Conversation

anijain2305 commented Nov 14, 2019 • edited Loading

yzhliu commented Nov 14, 2019

tqchen commented Nov 15, 2019

anijain2305 commented Nov 15, 2019

anijain2305 commented Nov 19, 2019

anijain2305 commented Nov 21, 2019

anijain2305 commented Nov 25, 2019

yzhliu commented Nov 25, 2019

yzhliu left a comment

Choose a reason for hiding this comment

yzhliu Nov 26, 2019

Choose a reason for hiding this comment

anijain2305 Nov 26, 2019

Choose a reason for hiding this comment

yzhliu Nov 26, 2019

Choose a reason for hiding this comment

yzhliu Nov 26, 2019

Choose a reason for hiding this comment

anijain2305 Nov 26, 2019 • edited Loading

Choose a reason for hiding this comment

yzhliu Dec 10, 2019

Choose a reason for hiding this comment

anijain2305 Dec 23, 2019

Choose a reason for hiding this comment

anijain2305 commented Nov 28, 2019

anijain2305 commented Dec 1, 2019

yzhliu left a comment

Choose a reason for hiding this comment

yzhliu Dec 10, 2019

Choose a reason for hiding this comment

kice commented Dec 14, 2019

anijain2305 commented Dec 23, 2019

yzhliu commented Dec 26, 2019

anijain2305 commented Dec 26, 2019

anijain2305 commented Nov 14, 2019 •

edited

Loading

anijain2305 Nov 26, 2019 •

edited

Loading