-
Notifications
You must be signed in to change notification settings - Fork 2.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Rendering of "bitmaprenderer"-backed canvas #7833
Comments
Thinking back to discussions from several years ago, and reviewing the specification at https://html.spec.whatwg.org/multipage/canvas.html#the-imagebitmap-rendering-context : The intent of Per the Given these assumptions I think that Safari's and Chrome's behavior on this test case is correct. It's what would happen if a canvas with a 2D context were resized to the size of the While |
Just a note that creating a new FWIW, I too think Safari and Chrome's behavior is the least surprising here. This way it behaves like an Footnotes
|
Thank you for providing that context. I think it's extremely weird that the |
It was crucial in the design that transferring of an ImageBitmap into the rendering context not be a layout-inducing event, which is why Apologies for any weirdness in specification we designers of this API induced. I hope that we can preserve the behavior Safari and Chrome implement - I think it's the preferred semantic from the developer's point of view. |
Well, at the moment the weirdness is only in implementations as the specification does require layout changes to occur. Changes to the Rendering section would have to be made to account for this somehow as far as I can tell. Semantically that also leaves us in a pickle as the intent is for these attributes to not be presentational. I suppose we could try to forbid them on I'm curious though what others think as at least to me Firefox's behavior is much more consistent with how |
Aside: I figured out why I didn't discover this difference when I tested the behavior: my ImageBitmap was smaller that my target canvas. In this case, Chrome/Firefox/Safari all have the same behavior (they zoom in, not "letterbox"). The behavior differs when the ImageBitmap is larger than the target canvas (zoom out vs crop). |
FWIW, the object-fit CSS property does work with canvases, but in current implementations, it seems to not intervene in the resizing that maps the bitmap to the size defined by the width and height HTML attributes. The object-fit comes into play when mapping from the intrinsic size to the CSS size. Example: https://jsfiddle.net/651yzwrc/ I actually think the spec'ed behaviour (which no browsers have implemented) is the sanest because:
As Ken pointed out, this has the disadvantage of that calling transferFromImageBitmap may invalidate the page's layout. Is that really a dealbreaker? |
I think it is unless there's an easy application-level workaround to make If the CSS width and height of the destination canvas are explicitly set by the application (for example, to the canvas's width and height), then is |
FWIW I personally find this behavior quite confusing, at least since Also see #7798. OffscreenCanvas actually has a special step where setting the placeholder canvas As an aside it might be worth noting that Safari and Chrome behaviors do differ (as seen in Justin's fiddle). Safari will apparently change the size of the canvas when the ImageBitmap is transferred there before the canvas has been rendered: https://jsfiddle.net/1tuvsk96/ |
I'm happy that canvas's intrinsic width and height become the imagebitmap's width/height, as if we were setting img.src and relying on intrinsic width and height for layout. It's also good that canvas.width/height are still useful to be able to size an ImageBitmapRenderingContext "placeholder" "bitmap" for the purposes of intrinsic width/height. I think our two options are:
|
I agree. Setting an img.src also changes the width and height (if not set) asychronously. And even if you set img.width, img.height those have no effect on the resolution of the image. In other words
Will give you a 10x10 pixel backing store for the image, scaled to 5x7 CSS pixels. After this img.width = 5, img.height = 7, img.naturalWidth = 10, img.naturalHeight = 10 That's how I'd expect bitmaprenderer to work if I never read any docs. And I'd expect if you want the actual size of the bitmap in the bitmap rendered you'd have to get that size some other way (like naturalWidth, naturalHeight on images)
ImageBitmap.width and ImageBitmap.height is where you'd get actual resolution |
I discussed this with @kkinnunen-apple. There's three (four) sizes in play here:
The design for WebGL however does allow for the backing size to be smaller, although this isn't universally implemented so it's not quite enshrined just yet. And it's not defined what happens when the canvas element is a source of input. E.g., if you use Now bitmaprenderer has taken the WebGL approach and has the same problems. In practice it seems that at least some implementations use the backing size when the canvas element is used as a source of input. The big question here is whether we can make changes still or whether there is too much deployed content for that to be reasonable. I see these options, first two inspired by @kdashg:
2/3 have my personal preference. 1 is probably the most compatible, but it will be some work defining things properly and going through the long list of canvas-element consuming endpoints and ensuring they do the right thing. |
I'd pick 1 because seems more consistent with the rest of the platform. Setting an
img width and height are now 200x100
image is now displayed at 400x200 but its "backingStore" is still 200x100
image width and height are still 400x200 but it's backing store has changed to 300x150 image has
The image drawn will be 200x100, not 22x11 Image (and Video) both show
This is an incorrect assumption. The image width and height IDL attributes make this clear it is an incorrect assumption. Implementations should always use the backing size. Implementations not using the backingsize should arguably be fixed to use the backing size. This would make them match the image tag and therefore do the expected thing. Having the canvas behave differently than the video and image tag seems more confusing to me. |
I'm sure you already know this but, to be a little pedantic because many often get mislead
It's not quite true that display size is a multiple of CSS size. An example of when this is not true. |
I don't think comparing |
I'm not sure I understand. Setting the width and height of both an image, and a canvas, change their layout size unless the there is CSS overriding them. To me, an image, a video, and a canvas are the same. They are all rectangles of pixels. Their only difference is the source of the pixels. An image's source is an image file, a videos source is a video file/stream, a canvas's source is JavaScript via one of many context. They should all behave the same. |
I think having a canvas size and a backing size that are independent from each other adds unnecessary complexity to an already complex data model. Are there use cases that absolutely require HTMLCanvasElement.width/height to be settable when using IBRC? What if, when a canvas's context is an IBRC, the width/height attribute setters threw an InvalidStateError (or did nothing), effectively making the attributes read-only? basically the canvas width/height would reflect the backing size. If authors don't want the backing size to affect layout, or if they want the canvas to have a layout size prior to transferring in the first ImageBitmap, then they can just set the CSS size to something other than "auto". The only behavior I can think of that is not covered is: what happens when we read the canvas before it receives its first image bitmap? For example, if we upload it to a WebGL texture or call toDataURL or toBlob. These case should generate blank images. But how does one control the size of that blank image? If we think this use case matters, then perhaps width/height are read-only only when the IBRC has a backing, otherwise they are settable and control the dimensions of the default (blank) image. |
I think we could essentially ignore the However, as @Kaiido points out we already have one context that does something like this, so maybe it's not too bad. (Although we should fix the placeholder context to do something similar. Whereby we ignore the content attributes rather than throw when they are set. Perhaps as we do this we can fix both bitmaprenderer and placeholder together.) If Chromium is willing to champion that approach I think WebKit would be okay following, but there's probably a fair amount of risk and @kkinnunen-apple was worried that it would make certain optimizations harder as you could no longer be as positive that the layout size would not change upon changing the bitmap. |
In transferFromImageBitmap, one could have something like this to only recompute layout when necessary:
Or am I missing something more subtle that could cause layout changes in transferFromImageBitmap? The main counterargument I see for making transferFromImageBitmap never affect layout would be if we are trying to coerce authors into avoiding re-layout by forcing them to explicitly change the canvas size when that is the desired behaviour. |
@junov it gets more complicated with |
I don't think the placeholder has to communicate anything back to the OffscreenCanvas. The OffscreenCanvas that produced the frame does not need to know anything about layout: its width and height attributes are always writeable and control the size of the backing, just like a regular canvas with a regular rendering context. However, it is a bit awkward that the placeholder receives frames asynchronously via regular event loop tasks. Perhaps this could be improved in the event loop processing model. We could have a special task queue just for this that is flushed somewhere predictable, like in the Update the Rendering stage of the event loop. This would not eliminate re-layout, but it would make it saner by synchronizing it with rendering updates, and could potentially eliminate out-of-band layout calculations for some use cases. |
We kept discussing this in the WebGL WG. (last discussion 2023-01-12) dstRc.transferFromImageBitmap#1(src) {
dstRc.canvas.width = src.width;
dstRc.canvas.height = src.height;
dstRc.[drawImage](src);
src.close();
} For canvases today, width and height are presentational (and can incur relayout) unless overridden by CSS. If you want to prevent relayout, just override with CSS like normal for canvases. The main alternative is to bring in webgl's concept of readonly drawingBufferWidth,Height, a la: dstRc.transferFromImageBitmap#2(src) {
dstRc.drawingBufferWidth = src.width;
dstRc.drawingBufferHeight = src.height;
dstRc.[drawImage](src);
src.close();
} |
To be clear, they're not presentational. They establish the size of the rendering context. That's how they were designed. (WebGL does something murky, but that's more of a flaw in WebGL.) Resizing seems like an acceptable solution though. The only risk there is compatibility. Did the WebGL group discuss how to make this change and who would write tests and such? |
(Sure, technically e.g. (I wouldn't call e.g. Yes, we (at least Google and Mozilla) agreed to mutually prototype, and update tests as part of that prototyping. |
…romImageBitmap. This patch makes it so that when a new ImageBitmap is provided to an ImageBitmapRenderingContext, we resize its owning canvas rather than crop or scale the given ImageBitmap to fit inside the canvas dimensions. See discussion in whatwg/html#7833. Differential Revision: https://phabricator.services.mozilla.com/D188126
…romImageBitmap. This patch makes it so that when a new ImageBitmap is provided to an ImageBitmapRenderingContext, we resize its owning canvas rather than crop or scale the given ImageBitmap to fit inside the canvas dimensions. See discussion in whatwg/html#7833. Differential Revision: https://phabricator.services.mozilla.com/D188126
…romImageBitmap. r=gfx-reviewers,lsalzman This patch makes it so that when a new ImageBitmap is provided to an ImageBitmapRenderingContext, we resize its owning canvas rather than crop or scale the given ImageBitmap to fit inside the canvas dimensions. See discussion in whatwg/html#7833. Differential Revision: https://phabricator.services.mozilla.com/D188126
…romImageBitmap. r=gfx-reviewers,lsalzman This patch makes it so that when a new ImageBitmap is provided to an ImageBitmapRenderingContext, we resize its owning canvas rather than crop or scale the given ImageBitmap to fit inside the canvas dimensions. See discussion in whatwg/html#7833. Differential Revision: https://phabricator.services.mozilla.com/D188126
…romImageBitmap. r=gfx-reviewers,lsalzman This patch makes it so that when a new ImageBitmap is provided to an ImageBitmapRenderingContext, we resize its owning canvas rather than crop or scale the given ImageBitmap to fit inside the canvas dimensions. See discussion in whatwg/html#7833. Differential Revision: https://phabricator.services.mozilla.com/D188126 UltraBlame original commit: cbd593a4819d1bc653fae6d21f86b25c9b1bdd58
…romImageBitmap. r=gfx-reviewers,lsalzman This patch makes it so that when a new ImageBitmap is provided to an ImageBitmapRenderingContext, we resize its owning canvas rather than crop or scale the given ImageBitmap to fit inside the canvas dimensions. See discussion in whatwg/html#7833. Differential Revision: https://phabricator.services.mozilla.com/D188126 UltraBlame original commit: cbd593a4819d1bc653fae6d21f86b25c9b1bdd58
…romImageBitmap. r=gfx-reviewers,lsalzman This patch makes it so that when a new ImageBitmap is provided to an ImageBitmapRenderingContext, we resize its owning canvas rather than crop or scale the given ImageBitmap to fit inside the canvas dimensions. See discussion in whatwg/html#7833. Differential Revision: https://phabricator.services.mozilla.com/D188126 UltraBlame original commit: cbd593a4819d1bc653fae6d21f86b25c9b1bdd58
Consider this example: https://software.hixie.ch/utilities/js/live-dom-viewer/saved/10227 (HTML reproduced below).
According to the specification the
canvas
's intrinsic dimensions should end up being those of the image and those should also be used for rendering it as a replaced element. However, thecanvas
is still sized according to itswidth
andheight
attributes (or the replaced element defaults if those are absent).In Chrome/Safari the image appears scaled and in Firefox it's cropped. It's not possible to determine the size of the context I think as that's not exposed. So you cannot reliably tell if the context is actually larger than the canvas or not.
What was the intent here?
(I personally think the Firefox behavior is the most logical behavior here and consistent with how
width
andheight
are supposed to function, but it doesn't really match the current specification. However, neither does the Chrome/Safari behavior.)(This came up in a long discussion over at gpuweb/gpuweb#2416.)
cc @whatwg/canvas
The text was updated successfully, but these errors were encountered: