-
Notifications
You must be signed in to change notification settings - Fork 4.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
JIT: Flow Graph Modernization and Improved Block Layout #93020
Comments
Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch Issue DetailsOverviewThe current block layout algorithm in the JIT is based local permutations of block order. It is complicated and likely far from optimal. Before we can tackle this problem there are several important (and sizeable) prerequisites, which we can lump together as "flow graph modernization." There are a lot of details here, but at a high level:
It is not yet clear how much progress we can make during .NET 9. The list of items below is preliminary and subject to change. Flow Graph Modernization
Block Layout
cc @amanasifkhalid @dotnet/jit-contrib
|
Next step for #93020, per conversation on #93772. Replacing BBJ_NONE with BBJ_ALWAYS to the next block helps limit our use of implicit fall-through (though we still expect BBJ_COND to fall through when its false branch is taken; #93772 should eventually address this). I've added a small peephole optimization to skip emitting unconditional branches to the next block during codegen.
As a prerequisite for some of the block reordering work we'll likely need to change loop alignment to be more centralized. We currently identify the initial candidate blocks to place loop alignment instructions in during loop finding and apply some heuristics when computing loop side effects during VN. We will probably need to defer all of these decisions to happen after block reordering. I am also inclined to say that we should just recompute the loops at that point, instead of trying to maintain loop information -- we have a lot of code that works hard to maintain |
…96265) Part of #93020. Previously, bbFalseTarget was hard-coded to match bbNext in BasicBlock::SetNext. We still require bbFalseTarget to point to the next block for BBJ_COND blocks, but I've removed the logic for updating bbFalseTarget from SetNext, and placed calls to SetFalseTarget wherever bbFalseTarget needs to be updated because the BBJ_COND block has been created or moved relative to its false successor. This helps set us up to start removing logic that enforces the block's false successor is the next block.
…e target (#96431) Next step for #93020. When doing hot/cold splitting, if the first cold block succeeds a BBJ_COND block (meaning the false target is the first cold block), we previously needed to insert a BBJ_ALWAYS block at the end of the hot section to unconditionally jump to the cold section. Since we will need to conditionally generate a jump to the false target depending on its location once bbFalseTarget can diverge from bbNext, this seemed like a nice opportunity to add that logic in, and instead generate a jump to the cold section by checking if a jump is needed to the false target, rather than by appending a BBJ_ALWAYS block to the hot section.
Before finalizing the block layout with optOptimizeLayout, we call fgReorderBlocks in a few optimization passes that modify the flowgraph (though without the intent to actually reorder any blocks, by passing useProfile=false). Removing all of these early calls -- except for the one in optOptimizeFlow, which can probably be replaced by moving fgReorderBlocks's branch optimization logic to fgUpdateFlowGraph -- incurs relatively few diffs, and gets us closer to #93020's goal of deferring block reordering until late in the JIT's optimization phases.
Before finalizing the block layout with optOptimizeLayout, we call fgReorderBlocks in a few optimization passes that modify the flowgraph (though without the intent to actually reorder any blocks, by passing useProfile=false). Removing all of these early calls -- except for the one in optOptimizeFlow, which can probably be replaced by moving fgReorderBlocks's branch optimization logic to fgUpdateFlowGraph -- incurs relatively few diffs, and gets us closer to dotnet#93020's goal of deferring block reordering until late in the JIT's optimization phases.
…imization phase (dotnet#96609) Next step for dotnet#93020. Working backwards through the JIT flowgraph phases, this change allows bbFalseTarget to diverge from bbNext in Compiler::optOptimizeLayout and onwards.
Part of #93020. This change adds back in most of #97191 and #96609, except for any significant changes to the flowgraph optimization passes to reduce churn. With this change, the false target of a BBJ_COND can diverge from the next block until Compiler::optOptimizeLayout, in which we reestablish implicit fall-through with fgConnectFallThrough to preserve the existing block reordering behavior. Note that the deferral of these fall-through fixups causes diffs in the edge weights, which can alter the behavior of fgReorderBlocks, hence some of the size regressions
Advance profile consistency check through inlining. Turns out there are five reasons why inlining may make profile data inconsistent. Account for these and add metrics. Also add separate metrics for consistency before and after inlining, since pre-inline phases are run on inlinees and so don't give us good insight into overall consistency rates. And add some metrics for inlining itself. Contributes to #93020. Co-authored-by: Aman Khalid <[email protected]>
When dynamic PGO is active we would like for all methods to have some profile data, so we don't have to handle a mixture of profiled and unprofiled methods during or after inlining. But to reduce profiling overhead, the JIT will not instrument methods that have straight-line control flow, or flow where all branches lead to throws (aka "minimal profiling"). When the JIT tries to recover profile data for these methods it won't get any data back. SO there is a fairly high volume of these profiled/unprofiled mixtures today and they lead to various poor decisions in the JIT. This change enables the JIT to see if dynamic PGO is active. The JIT does not yet do anything with the information. A subsequent change will have the JIT synthesize data for methods with no profile data in this case. We could also solve this by creating a placeholder PGO schema for theswith no data, but it seems simpler and less resource intensive to have the runtime tell the JIT that dynamic PGO is active. This also changes the JIT GUID for the new API surface. Contributes to dotnet#93020.
Part of dotnet#93020. Compiler::fgDoReversePostOrderLayout reorders blocks based on a RPO of the flowgraph's successor edges. When reordering based on the RPO, we only reorder blocks within the same EH region to avoid breaking up their contiguousness. After establishing an RPO-based layout, we do another pass to move cold blocks to the ends of their regions in fgMoveColdBlocks. The "greedy" part of this layout isn't all that greedy just yet. For now, we use edge likelihoods to make placement decisions only for BBJ_COND blocks' successors. I plan to extend this greediness to other multi-successor block kinds (BBJ_SWITCH, etc) in a follow-up so we can independently evaluate the value in doing so. This new layout is disabled by default for now.
…101739) If we know dynamic PGO is active, and we do not find a PGO schema for a method, synthesize PGO data. The schema may be missing if the method was prejitted but not covered by static PGO, or was considered too simple to need profiling (aka minimal profiling). This synthesis removes the possibility of a mixed PGO/no PGO situation. These are problematic, especially in methods that do a lot of inlining. Now when dynamic PGO is active all methods that get optimized will have some form of PGO data. Only run profile incorporation when optimizing. Reset BBOPT/pgo vars if we switch away from optimization or have a min opts failover. Contributes to dotnet#93020.
Advance profile consistency check through inlining. Turns out there are five reasons why inlining may make profile data inconsistent. Account for these and add metrics. Also add separate metrics for consistency before and after inlining, since pre-inline phases are run on inlinees and so don't give us good insight into overall consistency rates. And add some metrics for inlining itself. Contributes to dotnet#93020. Co-authored-by: Aman Khalid <[email protected]>
…ayout (#102461) Part of #93020. In #102343, we noticed the RPO-based layout sometimes makes suboptimal decisions in terms of placing a block's hottest predecessor before it -- in particular, this affects loops that aren't entered at the top. To address this, after establishing a baseline RPO layout, fgMoveBackwardJumpsToSuccessors will try to move backward unconditional jumps to right behind their targets to create fallthrough, if the predecessor block is sufficiently hot.
Instead of giving hander regions a fraction of the entry weight, give them a small fixed weight. This is intended to combat the lack of profile propagation out of handler regions, where there are currently sometimes weight discontinuities large enough to cause profile check asserts. Contributes to dotnet#93020.
Move the full profile check down past the importer. Attempt local repair for cases where the importer alters BBJ_COND. If that is unable to guarantee consistency, mark the PGO data as inconsistent. If the importer alters BBJ_SWITCH don't attempt repair, just mark the profile as inconsistent. If in an OSR method the original method entry is a loop header, and that is not the loop that triggered OSR, mark the profile as inconsistent. If the importer re-imports a LEAVE, there are still orphaned blocks left from the first importation, these can mess up profiles. In that case, mark the profile as inconsistent. Exempt blocks with EH preds (catches, etc) from inbound checking, as profile data propagation along EH edges is not modelled. Modify the post-phase checks to allow either small relative errors or small absolute errors, so that flow out of EH regions though intermediaries (say step blocks) does not trip the checker. Ensure the initial pass of likelihood adjustments pays attention to throws. And only mark throws as rare in the importer if we have not synthesized profile data (which may in fact tell us the throw is not cold). Contributes to dotnet#93020
Part of dotnet#93020. Removes FlowEdge::m_edgeWeightMin and FlowEdge::m_edgeWeightMax, and relies on block weights and edge likelihoods to determine edge weights via FlowEdge::getLikelyWeight.
…1011) Fixes the following areas with proper profile updates: * GDV chaining * instrumentation-introduces flow * OSR step blocks * fgSplitEdge (used by instrumentation) Adds checking bypasses for: * callfinally pair tails * original method entries in OSR methods Contributes to dotnet#93020
When dynamic PGO is active we would like for all methods to have some profile data, so we don't have to handle a mixture of profiled and unprofiled methods during or after inlining. But to reduce profiling overhead, the JIT will not instrument methods that have straight-line control flow, or flow where all branches lead to throws (aka "minimal profiling"). When the JIT tries to recover profile data for these methods it won't get any data back. SO there is a fairly high volume of these profiled/unprofiled mixtures today and they lead to various poor decisions in the JIT. This change enables the JIT to see if dynamic PGO is active. The JIT does not yet do anything with the information. A subsequent change will have the JIT synthesize data for methods with no profile data in this case. We could also solve this by creating a placeholder PGO schema for theswith no data, but it seems simpler and less resource intensive to have the runtime tell the JIT that dynamic PGO is active. This also changes the JIT GUID for the new API surface. Contributes to dotnet#93020.
Part of dotnet#93020. Compiler::fgDoReversePostOrderLayout reorders blocks based on a RPO of the flowgraph's successor edges. When reordering based on the RPO, we only reorder blocks within the same EH region to avoid breaking up their contiguousness. After establishing an RPO-based layout, we do another pass to move cold blocks to the ends of their regions in fgMoveColdBlocks. The "greedy" part of this layout isn't all that greedy just yet. For now, we use edge likelihoods to make placement decisions only for BBJ_COND blocks' successors. I plan to extend this greediness to other multi-successor block kinds (BBJ_SWITCH, etc) in a follow-up so we can independently evaluate the value in doing so. This new layout is disabled by default for now.
…101739) If we know dynamic PGO is active, and we do not find a PGO schema for a method, synthesize PGO data. The schema may be missing if the method was prejitted but not covered by static PGO, or was considered too simple to need profiling (aka minimal profiling). This synthesis removes the possibility of a mixed PGO/no PGO situation. These are problematic, especially in methods that do a lot of inlining. Now when dynamic PGO is active all methods that get optimized will have some form of PGO data. Only run profile incorporation when optimizing. Reset BBOPT/pgo vars if we switch away from optimization or have a min opts failover. Contributes to dotnet#93020.
Advance profile consistency check through inlining. Turns out there are five reasons why inlining may make profile data inconsistent. Account for these and add metrics. Also add separate metrics for consistency before and after inlining, since pre-inline phases are run on inlinees and so don't give us good insight into overall consistency rates. And add some metrics for inlining itself. Contributes to dotnet#93020. Co-authored-by: Aman Khalid <[email protected]>
…ayout (dotnet#102461) Part of dotnet#93020. In dotnet#102343, we noticed the RPO-based layout sometimes makes suboptimal decisions in terms of placing a block's hottest predecessor before it -- in particular, this affects loops that aren't entered at the top. To address this, after establishing a baseline RPO layout, fgMoveBackwardJumpsToSuccessors will try to move backward unconditional jumps to right behind their targets to create fallthrough, if the predecessor block is sufficiently hot.
Potential .NET 10 items Block layout:
Flowgraph Modernization:
cc @AndyAyersMS, feel free to add to this. |
I think that's a good start. I'm going to move this issue to .NET 10 for now, later we can decide if we want to split off a new issue or just revise this one. |
Let's split off a new issue for future work. |
Overview
The current block layout algorithm in the JIT is based on local permutations of block order. It is complicated and likely far from optimal. We would like to improve the overall block layout algorithm used by the JIT, in particular adopting a global cost-minimizing approach to layout—for instance, one in the style of Young et. al.'s Near-optimal intraprocedural branch alignment. Additional complexities arise in our case because of various EH reporting requirements, so the JIT cannot freely reorder all blocks, but we should be able to apply the global ordering techniques within EH regions.
Before we can tackle this problem there are several important (and sizeable) prerequisites, which we can lump together as "flow graph modernization." There are a lot of details here, but at a high level:
It is not yet clear how much progress we can make during .NET 9. The list of items below is preliminary and subject to change.
Motivation
Past studies have shown that the two phases that benefit most from block-level PGO data are inlining and block layout. In a previous compiler project, the net benefit from PGO was on the order of 15%, with about 12% attributable to inlining, and 2% to improved layout.
The current JIT is likely seeing a much smaller benefit from layout. The goal here is to ensure that we are using the accurate PGO data to make informed decisions about the ordering of blocks, with the hope of realizing perhaps a 1 or 2% net benefit across a wide range of applications (with some benefiting much more, and others, not at all).
Flow Graph Modernization
BasicBlock
behind setters/getters (bbNext
,bbJumpKind
,bbJumpTarget
, ...)bbNext
for blocks with fall-through jump kindsbbNext
. Might be best to do this gradually, working front-to-back through the phases, and then restoring correspondence. Eventually we'll be restoring right at block layout. Main work here is root out places that implicitly rely onbbNext
ordering having some important semantic. Note thebbNext
order can still reflect likely layout order (e.g., it can agree with fall throughs after we renumber/etc.).BasicBlock
references forbbJumpTarget
with the appropriateFlowEdge
. Consider (perhaps) linkingFlowEdges
in a successor list as well as predecessor list.FlowEdge
and the code that sets them, update all customers to use new likelihood based weights.BB_UNITY_WEIGHT
to be 1.0Block Layout
Compiler::fgFindInsertPoint
, and similar logic that attempts to maintain reasonable orderings before block layout is run (see comment).cc @amanasifkhalid @dotnet/jit-contrib
The text was updated successfully, but these errors were encountered: