perf(): reduce setCoords calls #9550

ShaMan123 · 2023-12-17T09:28:19Z

Description

After removing lineCoords we have 2 sets of coords
Corner coords (aCoords) used for object geometry (isOnScreen, intersection etc.) that belong to the parent plane, getCoords returning them in the scene plane.
Control coords (oCoords) used for control rendering/hit detection that belong to the viewport plane.

This distinction is very important and useful.
It means that viewport changes do not affect corner coords.
Also since they belong to the parent plane they need to be recalculated only if the object itself changes its own transform/size. This means that if an ancestor changes transform we do not need to recalculate the corner coords.
Since control coords exist only on the active object it seems important to separate use cases of setCoords to reduce computation.

I have found that fabric used to call setCoords before drawing the controls (at the end of the rendering cycle) instead of after a transform action. This led to a stale coords state between the 2 calls affecting apps using corner controls in the rendering cycle. It was always a frame behind. This is fixed now.

Looking into setCoords in general it seems we should decide when they are first set. It is ambiguous.
Does it happen when an object mounts the canvas? What about an object added to a group? We can check if the group has a canvas set, therefore it is mounted , therefore setCoords should be called.
It also means that object geometry should not be used until an object is mounted.

This PR is part of a performance effort I am looking into.
I have a ground breaking idea about a binary index that will reduce computation cost of isOnScreen exponentially.
I wish to contribute it to the repo but can't wait with it more than a number of days so if it can't merge soon I will depart from fabric (at least with this) or try to expose it as a standalone but that will be a shame.

Another change I have introduced is interactive including subTargetCheck when true.
To be able to select sub targets both need to be flagged as true.
subTargetCheck flags searchPossibleTargets to traverse the children of a group.
interactive flags to return the top most sub target found within searchPossibleTargets.
So essentially interactive = true contains subTargetCheck = true.
It is more straight forward as well.

In Action

github-actions · 2023-12-17T09:29:46Z

Build Stats

file / KB (diff)	bundled	minified
fabric	914.791 (-0.642)	305.343 (-0.295)

ShaMan123

I also made Group#interactive include subTargetCheck logically so now it is enough to

new Group(objects, { interactive: true })

This will not change anything else.

There is a decision to make and then I can complete this PR

ShaMan123 · 2023-12-17T09:40:53Z

src/canvas/Canvas.ts

-        // hoverCursor should come from top-most subTarget,
-        // so we walk the array backwards


What is wrong exactly?

the order of targets is [topMost, ..., bottomMost]?

Sorted in descending z index order, top most object first

ShaMan123 · 2023-12-17T09:41:01Z

src/canvas/SelectableCanvas.ts

@@ -190,6 +190,7 @@ export class SelectableCanvas<EventSpec extends CanvasEvents = CanvasEvents>

  /**
   * Keep track of the subTargets for Mouse Events
+   * Sorted in descending z index order, top most object first


ShaMan123 · 2023-12-17T09:42:51Z

src/shapes/ActiveSelection.ts

@@ -101,6 +106,7 @@ export class ActiveSelection extends Group {
    // the object will be in the objects array of both the ActiveSelection and the Group
    // but referenced in the group's _activeObjects so that it won't be rendered twice.
    this._enterGroup(object, removeParentTransform);
+    object.setCoords();


needs a mounted decision

ShaMan123 · 2023-12-17T09:43:45Z

src/shapes/Object/InteractiveObject.ts

@@ -511,7 +511,6 @@ export class InteractiveFabricObject<
      ctx.strokeStyle = options.cornerStrokeColor;
    }
    this._setLineDash(ctx, options.cornerDashArray);
-    this.setCoords();


bad approach, leading to unnecessary computation and a stale state

.vscode/settings.json

src/brushes/SprayBrush.ts

asturur · 2024-01-07T13:54:47Z

src/canvas/SelectableCanvas.ts

-        if (isCollection(target) && target.subTargetCheck) {
+        if (
+          isCollection(target) &&
+          (target.subTargetCheck || target.interactive)


target interactive requires subtargetcheck, so we shouldn't be checking for both.

I thought I wrote about this in the description, sorry, updated now.

Another change I have introduced is interactive including subTargetCheck.
To be able to select sub targets both need to be flagged as true.
subTargetCheck flags searchPossibleTargets to traverse the children of a group.
interactive flags to return the top most sub target found within searchPossibleTargets.
So essentially interactive = true contains subTargetCheck = true.
It is more straight forward as well.

asturur · 2024-01-07T14:03:53Z

src/canvas/StaticCanvas.ts

-    for (let i = 0; i < len; i++) {
-      const object = this._objects[i];
-      object.group || object.setCoords();
-    }
-    if (backgroundObject) {
-      backgroundObject.setCoords();
-    }
-    if (overlayObject) {
-      overlayObject.setCoords();
-    }


why deleting those? control coords need to be set when we change the vpt.

We need control coords only for the active object

And regarding corner coords, we need them on these objects only if they do not ignore the vpt

Looking at the code we do not even check if these objects are on screen and these objects are not selectable apart from imperatively calling setActiveObject (which I believe should call setCoords if it doesn't) thus making the setCoords call redundant

So this was here because before we were selecting by lineCoords, while now we select by aCoords that doesn't change by the zoom or pan.

Exactly, the active object calls setCoords, the rest should not. Mind that setActiveObject call setCoords so it seals the gap.

asturur · 2024-01-07T14:09:34Z

src/shapes/Text/Text.ts

    if (this._forceClearCache) {
      this.initDimensions();
+      this.setCoords();
    }


this whole thing the text has that can change the size just before rendering is wrong, is probably a patch that was done fore some reason, but this shouldn't exist. Render renders, period.

suggestions?

i don't have a suggestion but is also evident that if there is a dependency between having the correct dimensions and being detected on screen, so for correctness this check and setCoords shoud be called before the onScreen early return, and also Object will do the isOnscreen check again.
This render method should take in account that if _forceClearCache is true, should execute the initDimension and then move to super.render. the precheck is not useful, is a double cost 99.9% of the time

asturur · 2024-01-07T14:16:16Z

src/LayoutManager/LayoutManager.ts

@@ -248,6 +250,8 @@ export class LayoutManager {
      left: object.left + offset.x,
      top: object.top + offset.y,
    });
+    // TODO: should we delete aCoords/oCoords here so they are not stale if subTargetCheck is false?
+    //  or should we apply offset instead of recalculating aCoords?


an object without coords could create more damages than one with stale, different typings, always have to check if the coords are there.
We should document that if you have a group without subtargetCheck and you plan on using the group inner object coordinates, you have to update them when you use them.

ShaMan123

rebased

ShaMan123 · 2024-01-08T06:43:50Z

src/LayoutManager/LayoutManager.ts

@@ -216,6 +216,8 @@ export class LayoutManager {
      target.setPositionByOrigin(nextCenter, CENTER, CENTER);
      // invalidate
      target.setCoords();
+      (target.subTargetCheck || target.interactive) &&
+        target.forEachObject((object) => object.setCoords());


here is a good example of the benefit of separating setCoords to setCornerCoords and setControlCoords because here we need only setCornerCoords

Do you have objections doing that?

Also this line is incorrect because it goes down 1 level but should recurse down

src/controls/wrapWithFixedAnchor.ts

ShaMan123 · 2024-04-08T08:20:33Z

src/shapes/Group.ts

-  setCoords() {
-    super.setCoords();
-    this._shouldSetNestedCoords() &&
-      this.forEachObject((object) => object.setCoords());
-  }


@jiayihu look here, this removes the perf hit you described

src/controls/polyControl.ts

ShaMan123

All in all this looks like an important PR

ShaMan123

rebased and waiting for response before fixing some issues

ShaMan123 · 2024-04-08T08:44:07Z

src/canvas/Canvas.ts

-      this.setCursor(hoverCursor);
+      // hoverCursor should come from top-most subTarget
+      const hoverCursor =
+        (target as Group).subTargetCheck &&


should it be:

Suggested change

(target as Group).subTargetCheck &&

(target as Group).interactive &&

?

ShaMan123

cleaned up a small bit

ShaMan123 · 2024-04-08T08:55:05Z

src/canvas/Canvas.ts

-      transform.target.isMoving = true;
-      this.setCursor(transform.target.moveCursor || this.moveCursor);
+      target.isMoving = true;
+      this.setCursor(target.moveCursor || this.moveCursor);


doesn't belong here

jiayihu · 2024-04-08T08:58:15Z

Chipping in as requested to report that current setCoords performance is terrible for a few reasons:

setCoords always set the control coords even though most of the time you don't need to show the controls. In fabric by default actually only the activeObject renders the controls
obj.drawControls calls obj.setCoords internally, so whenever controls are renderered, their position is recalculated. With groups, this is aggravated by point 3
Group setCoords calls it recursively for the children if subTargetCheck. Due to point 2, calling group.drawControls will call group.setCoords(), which in turn calls the children setCoords. But then when child.drawControls is called, this gets repeated. This results probably in factorial O(n!) time complexity.

As just an example, we recently implemented a Table, which is a Group of cells. Each cell is in turn a group of content. Each cell has 4 controls. Re-rendering a table with only 150 cells changes from 40s to 2.5 once setCoords is removed from drawControls:

asturur · 2024-04-08T12:24:54Z

my biggest issue with this PR and the reason why i left it here is because it says it is for performance but never gave a figure of the performance it was improving.

asturur · 2024-04-08T12:27:01Z

I also want to do some retrospect on how we got here.
SetCords was basically never called and was always left to the developer to don't mess up with it, how we end up that is called at draw control time i do not know.

asturur · 2024-04-08T12:32:00Z

src/canvas/StaticCanvas.ts

-    if (overlayObject) {
-      overlayObject.setCoords();
-    }
+    this.viewportTransform = [...vpt];


this wasn't spread before, it shouldn't be spread now

Are you sure?
When we init the canvas we spread to avoid mutation, why not here?

it is unrelated to what we are doing here. no?

jiayihu · 2024-04-08T15:15:42Z

never gave a figure of the performance it was improving.

I suggested @ShaMan123 we should add a benchmark test that simply renders a Group with 100 Groups, with 2 children per group. Then measure how much times it takes and how much this PR affects it. Also it will avoid future regressions.

SetCords was basically never called and was always left to the developer to don't mess up with it, how we end up that is called at draw control time i do not know.

It was there since v5.3 but not on v3.6 (which we were using at my company before migrating to v6) but it's definitely suspicious to call setCoords during drawing. Maybe it was added to workaround stale coords without minding about performance.

ShaMan123 · 2024-04-09T10:59:05Z

@asturur this discussion can lead to how fabric uses isOnScreen and should one day. It is very wasteful. We are starting to look into it if you want in.

asturur · 2024-04-09T17:18:41Z

@asturur this discussion can lead to how fabric uses isOnScreen and should one day. It is very wasteful. We are starting to look into it if you want in.

never gave a figure of the performance it was improving.

I suggested @ShaMan123 we should add a benchmark test that simply renders a Group with 100 Groups, with 2 children per group. Then measure how much times it takes and how much this PR affects it. Also it will avoid future regressions.

SetCords was basically never called and was always left to the developer to don't mess up with it, how we end up that is called at draw control time i do not know.

It was there since v5.3 but not on v3.6 (which we were using at my company before migrating to v6) but it's definitely suspicious to call setCoords during drawing. Maybe it was added to workaround stale coords without minding about performance.

Was added in 4.0 when the ability to call render controls with styleoverrides was added.
I think because you could then run multiple controls renders and we didn't want the controls to be offplace.
In that moment probably the same oCoords for interactivity where used for renders ( and maybe now too ) and we thought it was nicer if you didn't have to call that manually.
There weren't nested ocoords tho.
So i m sure that is the reason why that was there, but for sure now it can't be there at all.

asturur · 2024-04-09T17:20:36Z

@asturur this discussion can lead to how fabric uses isOnScreen and should one day. It is very wasteful. We are starting to look into it if you want in.

Just keep in mind isOnScreen was used to skip some rendering for first level objects. That was all, there aren't more intention behind it.
I don't see how is wasteful in theory it should check at worst 5 points in a rectangle i don't see how that is so bad. But please open a discussion on the same example as i m doing for the other changes. Keep it in draft until you want other to ask/comment

I m interested to see why is bad, that yes.

jiayihu · 2024-04-10T07:36:29Z

There weren't nested ocoords tho.

Bear in mind that it's not only about nested oCoords. It still does it for all the canvas objects, which is very expensive. Internally we have changed it to be done only for the needed objects, typically the activeObject.
So I kept the method call in obj.drawControls, which is not optimal during rendering of course, for exactly the reason of not having to call it manually. It'd be great if we can find a way to have both, i.e. call it not during rendering but not having to remember it.

asturur · 2024-04-10T08:00:08Z

If we are doing all objects 'setCoords' every render then we introduced a bug. That was never the intention, again setCoords was something that was called on setViewport and at the end of a transform.
At some point it was added during a drag operation but always on a single object.

Anyway this PR won't be breaking, so we can move forward with a release.
My request is to keep it strict to removing setCoords call and removing all the accessory changes ( like overlapping interactive with subtargetcheck and whatever is not a setCoord call )

asturur · 2024-04-10T08:13:32Z

i also see that calcOcoords with the nested cords for groups interactivity became way slower in the case of groups. so that is also probaly a cause of general slowness
We don't have a setCoords for the specific situation and a setCoords generic that updates all.

When doing setCoords on a group that isn't interactive, setting the oCoords of all the nested objects is a waste, and if the group is interactive but selected, is a waste anyway.

But setcoords is imperative, maybe having it going deeply nested was an error because most of the time aCoords do not change and that is all we need for targeting check.

But that is topic for a different discussion/pr

ShaMan123 · 2024-04-10T10:01:06Z

If we are doing all objects 'setCoords' every render then we introduced a bug. That was never the intention, again setCoords was something that was called on setViewport and at the end of a transform.
At some point it was added during a drag operation but always on a single object.

I was surprised to see that all the above are not as expected. Calling setCoords when drawing controls, not calling them otherwise

asturur · 2024-04-10T11:38:07Z

If we are doing all objects 'setCoords' every render then we introduced a bug. That was never the intention, again setCoords was something that was called on setViewport and at the end of a transform.
At some point it was added during a drag operation but always on a single object.

I was surprised to see that all the above are not as expected. Calling setCoords when drawing controls, not calling them otherwise

The gotcha always said, if something is misaligned doesn't work call setCoords.
Exactly because we never called if not when strictly necessary

ShaMan123 · 2024-04-10T19:57:22Z

If we are doing all objects 'setCoords' every render then we introduced a bug. That was never the intention, again setCoords was something that was called on setViewport and at the end of a transform.
At some point it was added during a drag operation but always on a single object.

I was surprised to see that all the above are not as expected. Calling setCoords when drawing controls, not calling them otherwise

The gotcha always said, if something is misaligned doesn't work call setCoords. Exactly because we never called if not when strictly necessary

That is what the gotcha says but the code does something entirely different so the gotcha is unrelated.
Fabric doesn't call setCoords after a transform action, which is the only place it 100% should.

asturur · 2024-04-10T22:23:49Z

@ShaMan123 is not unrelated, things changed pr after pr probably.
on 5.x setCoords was on finalizeCurrentTransform ( correct ), on scale method ( correct ), on adjustPosition ( correct ), on onComplete of animate ( maybe not correct ) , text editing ( correct ) and group layouting ( correct ) and on renderControls ( maybe not correct )

Also today is still on '_finalizeCurrentTransform' so that aspect is still there, not sure why you say it doesn't

asturur · 2024-04-10T22:56:15Z

A red flag i see in this PR is that we remove the nested call of group setCoords done with the subclass and then we comment in 2 places of added code that the call should be recursive, but we just removed the recursiveness from the code.
I don't think that change is a good one.

The only changes that are 100% correct here is the one from viewport and then moving the setCoord from the drawControl call earlier to the performAction could be correct without much doubts too.
But at that point you should remove the one from 'finalizeCurrentTransform'.

Where are the performance improvement coming from?
drawControl is removed but performAction compensate it
The only thing i see is removing the nesting from the group setCoords, but also then realizing is sitll necessary in many cases

ShaMan123 · 2024-04-11T05:15:56Z

drawControl is removed but performAction compensate it

Only during a transform, if for other reasons you need to render the canvas this is better.

But at that point you should remove the one from 'finalizeCurrentTransform'.

agreed

Regarding the recursive setCoords call of group. Do we need it?
For geometry we need it.
For isOnScreen we do not because it doesn't check nested objects.
For controls we do not because setActiveObject will handle that.
Thoughts?

ShaMan123 · 2024-04-11T05:17:36Z

Regarding the recursive setCoords call of group. Do we need it?
For geometry we need it.

This makes my suggestion to delete aCoords/oCoords relevant. If they do not exist they could be recalculated on the fly from getCoords etc.

jiayihu · 2024-04-11T06:57:38Z

If they do not exist they could be recalculated on the fly from getCoords

Then you'd get worse performance having to recompute them everytime, for instance for _checkTarget->_pointIsInObjectSelectionArea

ShaMan123 · 2024-04-11T07:01:52Z

Why? We delete them only if an ancestors invalidates geometry

asturur · 2024-04-11T07:53:47Z

my suggestion is to take the no brainers from this PR and to think better what to do for the rest.

Do we need to have oCoords cached? at this point i do not know.
You need to have them cached only if you are running findTargetControl on non selected objects.
Fabric by default doesn't change the controls anymore if the object is not already selected, and if you select it you can call setCoords once and then keep updated during transforms.

But of course if you know you are using an override where you can click a control also when the object is not selected, then things change.

ShaMan123 · 2024-04-11T11:18:50Z

Do we need to have oCoords cached? at this point i do not know.

Great point!
Ok then, so the remaining question is about the nested setCoords. I think it is an important perf hit we should resolve so I shall reiterate on it after the non brainers.
I have an idea for hashing both the matrices and the coords.

ShaMan123 · 2024-04-11T15:26:23Z

closing in favor of #9793

ShaMan123 closed this Dec 17, 2023

ShaMan123 reopened this Dec 17, 2023

ShaMan123 requested a review from asturur December 17, 2023 09:33

ShaMan123 mentioned this pull request Dec 17, 2023

feat(FabricObject): attached/detached events #9342

Closed

ShaMan123 commented Dec 17, 2023

View reviewed changes

asturur reviewed Jan 7, 2024

View reviewed changes

.vscode/settings.json Outdated Show resolved Hide resolved

asturur reviewed Jan 7, 2024

View reviewed changes

src/brushes/SprayBrush.ts Outdated Show resolved Hide resolved

asturur reviewed Jan 7, 2024

View reviewed changes

ShaMan123 force-pushed the perf/set-coords branch from 58f98fe to 4a2a3a7 Compare January 8, 2024 06:42

ShaMan123 commented Jan 8, 2024

View reviewed changes