[RELAY][TRANSFORM] Migrate buildmodule to transform

[zhiics]

This PR is the second item of #3202. We focus on the following items

• Register optimization passes.
• Remove RelayBuildConfig in BuildModule and replace them with PassContext.
• Migrate Optimize to Sequential in BuildModule.
• Add unit tests for Sequential using With scoping.

cc @tqchen @jroesch @MarisaKirisame @wweic @icemelon9

[MarisaKirisame]

I dont think type inference should be a pass. It should be something we always call pre/post to doing anything.

[MarisaKirisame]

also can you rebase?

[tqchen]

TypeInference could be a pass that can be required by some other passes

[zhiics]

@MarisaKirisame Yes, I also think TypeInference should be a pass. Otherwise, we have to force to execute it explicitly.

[MarisaKirisame]

I think it should be implicit, and all pass will call it pre/post the pass (this is register in passmanager.cc).
Type Inference do not change the runtime behavior of the program in anyway, so we should just always do them.
Even if some pass run on untyped code, I still dont think it is a good idea: running type inference help catch error early (and indeed multiple error had been caught by just running it).

[zhiics]

@MarisaKirisame Yeah, I see your point, but shouldn't be better to add it to the required list and invoke it only when necessary?

[MarisaKirisame]

@zhiics dont think of it as an analysis; think of it as a safety check.

[zhiics]

I think @MarisaKirisame's point makes sense at some extend. But I still think it might be more appropriate to add TypeInference as a required pass and apply it when necessary. If we want to do safety check before every pass, we should probably encode type inference there.

In the future, we may need to do pass validation after optimization where type inference might be explicitly called any way. @MarisaKirisame Do you think this is okay? @tqchen @jroesch thoughts?

[tqchen]

As a side note, to demonstrate the flexibility of the new transform API. I hope we can revive relay.build in python, which calls into the transform API without calling into C++. This is a slight divergence but given that the code is short enough, I believe a useful thing to do. Eventually, we can also have a DefaultOptPipeline that returns the default sequential pipeline we use for optimization.

[jroesch]

As a side note, to demonstrate the flexibility of the new transform API. I hope we can revive relay.build in python, which calls into the transform API without calling into C++. This is a slight divergence but given that the code is short enough, I believe a useful thing to do. Eventually, we can also have a DefaultOptPipeline that returns the default sequential pipeline we use for optimization.

Doesn't this actively work against the goal to have a pure C++ pipeline for the compiler?

[tqchen]

What I mean is to have a python pipeline alone with the c++ one, given both of them are short enough, we want to make sure it is easy to customize compilation pipeline from python

[zhiics]

yeah, we can probably have a separate discussion about adding the parallel optimization pipeline. Currently, we can finish the migration and clean up ir_pass.py and pass.h. How do you guys think?

[zhiics]

@tqchen Now all passes are moved to Optimize and it is Module to Module.

[wweic]

lgtm.

[tqchen]

I can merge this in as it is clearly an improvement over what we had before. However, I am still not happy with the Optimize. Here is a mock example of an "ideal code" of optimize:

# mostly device invariant optimizations
opt = transform.Sequential([
    transform.FoldConstant(),
    transform.SimplifyInference(),
    transform.EnableIf(lambda ctx: len(ctx.targets) == 1, transform.AlterLayout()),
 ])

# target specific optimizations
lower = transform.Sequential([
       transform.DeviceAnnotation(), # only enables for multiple target
       transform.FuseOps()
    ])

with transform.PassContext(target=target_device_map):
     mod = optimzie(mod)
     mod = lower(mod)

This is only a quick alternative that pops out from my head. @zhiics feel free to the critic. But we should really move to minimize the code of this lowering process.

Also cc @merrymercy @vinx13 @yzhliu @jroesch for more thoughts, perhaps we could use another RFC if necessary :)

[tqchen]

some further nits

[zhiics]

I can merge this in as it is clearly an improvement over what we had before. However, I am still not happy with the Optimize. Here is a mock example of an "ideal code" of optimize:
This is only a quick alternative that pops out from my head. @zhiics feel free to the critic. But we should really move to minimize the code of this lowering process.

Sorry. I totally missed this. I totally agree with you that it makes sense to separate target dependent and independent opts. Actually, I have been thinking of this as well. We can probably have another RFC for it. But let us keep this code change minimum for now.

[tqchen]

Thanks @wweic @zhiics