Fix dist error with lr decay layer #9489
Conversation
python/paddle/fluid/optimizer.py
Outdated
| @@ -24,6 +24,7 @@ | |||
| from regularizer import append_regularization_ops | |||
| from clip import append_gradient_clip_ops, error_clip_callback | |||
| from contextlib import contextmanager | |||
| from distribute_transpiler import UnionFind | |||
There was a problem hiding this comment.
optimizer.py should not depend on distribute_transpiler, either put _get_lr_decay_ops in the transpiler or put UnionFind in a single file. I'd prefer the first method, because we don't have to change current demo files then.
There was a problem hiding this comment.
optimizer.py should not depend on distribute_transpiler
Thanks @typhoonzero ,I think it's a good point, and maybe we can pass Optimizer instance to transpile interface so that we can support regularization for future.
There was a problem hiding this comment.
We can consider that when moving regularizer and clipping to the server side.
There was a problem hiding this comment.
Done, moved to transpiler.
Yancey1989
changed the title
| for _, opt_op in enumerate(opt_op_on_pserver): | ||
| for _, op in enumerate(self.optimize_ops): | ||
| # optimizer is connected to itself | ||
| if ufind.is_connected(op, opt_op) and \ | ||
| op not in global_ops: | ||
| __append_optimize_op__(op, per_opt_block) | ||
| per_opt_block = pserver_program.create_block(0) | ||
| per_opt_block = pserver_program.create_block(append_block.idx) |
| find_ops = [] | ||
| # find ops which output is lr var | ||
| # make a union-find struct by all ops | ||
| block = default_main_program().global_block() |
There was a problem hiding this comment.
Use self.program instead of default_main_program because we may need to transpile a program that is specified by user.
Yancey1989
force-pushed
the
fix_dist_with_lr_decay
branch
from
d9d11a3 to
b7ffd5d
Compare
* commit '33b8b3d22034423455a493712955e419aac7b19b': (251 commits) Remove redundant commands in build.sh and build_doc.sh Add dependencies Move v2/api/fluid to fluid/api and Adjust doc build commands Plain LRN op throws an exception when is_test is set in backward pass fix compiler error of profiler_test in ONLY_CPU mode fix server shutdown Translation for Model Configuration (PaddlePaddle#9513) Fix data transform when inplace (PaddlePaddle#9450) refine parallel add FAQ (PaddlePaddle#9494) Fix dist error with lr decay layer (PaddlePaddle#9489) add prefetch_op (PaddlePaddle#9495) Fix some errors (PaddlePaddle#9403) hookup WITH_FLUID_ONLY in TeamCity build.sh (PaddlePaddle#9509) Fix the order of reads and write from buffered channel (PaddlePaddle#9423) change WITH_FLUID to WITH_FLUID_ONLY (PaddlePaddle#9427) fix block num Revert "make append activation in place by default (PaddlePaddle#9417)" Speed/sequence op1 (PaddlePaddle#9217) fix a compile error ...
Fixed #9429