Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Responsive images causes performance issues on high-resolution devices #4421

Open
mor10 opened this issue Mar 7, 2019 · 53 comments · May be fixed by #5901
Open

Responsive images causes performance issues on high-resolution devices #4421

mor10 opened this issue Mar 7, 2019 · 53 comments · May be fixed by #5901

Comments

@mor10
Copy link

mor10 commented Mar 7, 2019

tl;dr: Correctly marked up responsive images (srcset + sizes) result in high-resolution devices (2x, 3x, 4x, 5x, 6x dpi) downloading and using very large files causing significant performance hits, especially on mobile devices on mobile networks.

Scenario

  • Layout with full-bleed (edge-to-edge) image, served in a responsive layout to all device widths.
  • srcset list includes images of varying src options, from small (~400px wide) to very large (~3000px wide) to account for mobile, tablet, desktop, and large monitors, high-resolution laptops etc.
  • sizes attribute marked up to spec with natural breakpoints.

Under these circumstances, a high-resolution smartphone will download larger image files to meet screen resolution demands, thus negating the original intent (as I understand it) of the RICG responsive images work which was to lower the bandwidth hit on mobile devices when viewing images.

This scenario is further complicated by modern design trends including full-bleed images as described above. Because the srcset list must include very large image sources to account for large high-resolution screens, small devices with high-resolution screens have access to and will use these same sources resulting in wasted bandwidth and significantly degraded performance.

Practical use case

This issue came to my attention while working on the new content editor for WordPress ("Gutenberg"). The content editor allows for images to be aligned wide and full, the latter meaning the image takes up the full available width which often means the width of the viewport.

When updating WordPress to generate appropriate srcset and sizes attributes to account for these layouts, I found significant performance reductions on mobile devices caused by the download of larger-than-required images as per the sizes attribute.

A functional example, with detailed explanation, is available here:
https://gutenberg.mor10.com/responsive-images-demo-corrected-code/

The result of this discovery was a sub-optimal workaround for WordPress core which is live on every WordPress site running version 5.0 or above: To avoid the performance hit on mobile devices, the largest image source in the srcset list is 1024px unless the theme explicitly states otherwise. In other words, browsers are served with smaller-than-necessary images, causing a poor user experience.

A functional example of the current auto-generated output as of WordPress 5.1 is available here:
https://gutenberg.mor10.com/image-test-current/

Summation: Due to the performance issues introduced by the srcset + sizes combination, 33.3% of the web is currently shipping the wrong image sources on purpose as a workaround. That said, this same issue will be experienced by anyone setting up a site with full-bleed images as described above, WordPress or not.

Possible solutions

From my perspective as a front-end developer, the ideal solution would be to amend the spec to allow developers to declare, through attributes or similar on each individual image, the pixel density the sizes attribute should be measured against. Something like this:

<img 
  src="fish-1-1024x684.jpg" alt="" 
  srcset="
    fish-1-1024x684.jpg 1024w, 
    fish-1-300x200.jpg 300w, 
    fish-1-768x513.jpg 768w, 
    fish-1-1568x1047.jpg 1568w" 
  sizes="
    (min-width: 768px) calc(8 * (100vw / 12) - 28px), 
    (min-width: 1168) calc(6 * 100vw/12) - 28px),
    calc(100% - (2 * 1rem))"
  resolution="1x"
/>

Alternatively, the browser could detect available bandwidth and other factors and actively throttle the srcset list accordingly so high-resolution mobile devices on mobile networks would receive the appropriate resolution interpretation based on available data and performance. This of course brings into question how to measure bandwidth and download limits, especially for users on max-megabytes-per-month plans.

I know this type of throttling is technically possible using client hints, but configuring client hints and server side solutions is beyond the capacity of most CMSes and site owners, and puts the onus of having the web work as expected on the individual user.

In lieu of the suggestions above type of bandwidth throttling, a third option could be to put browser limits on how high the resolution can be for images (2x limit on a 4x screen as an example).

cc @yoavweiss @joemcgill @getsource

@Hlsgs
Copy link

Hlsgs commented Mar 7, 2019

Alternatively, the browser could detect available bandwidth and other factors and actively throttle the srcset list accordingly so high-resolution mobile devices on mobile networks would receive the appropriate resolution interpretation based on available data and performance.

^ This. IMHO, the spec is about ar art directionish as it should be and all this should be left to the browsers. But I wholeheartedly agree that this needs to be adressed.

@yoavweiss
Copy link
Contributor

I've discussed this with @mor10 last week. It seems like the ubiquity of high resolution screens has made it so that there's a need for a cap on the DPR levels browsers take into account, either implicitly or explicitly.

From my perspective, the main hurdle towards an on-by-default cap, automatically enforced by the browser, is the lack of data on the cut-off ratio, where higher-resolution doesn't necessarily mean better user-experience. If such data could be provided, that'd be helpful on that front.

Barring that, the fastest route towards a solution here would be an opt-in cap (e.g. a maxresolution attribute).
Those are not mutually exclusive, and we could start out with an opt-in and modify its default value from the current infinity into a more reasonable default once we have data to back it up.

Regarding bandwidth based restrictions, it's possible that @tarunban has looked into that.

/cc @tabatkins @eeeps

@joemcgill
Copy link

If memory serves, most browsers used to cap the selection process at 2x. If browsers are no longer specifically doing this, then I would agree that a markup solution would be nice so that developers could control the resolution caps themselves. I would suggest doing this via a meta value that applied to the whole page, rather than on an individual image basis, but I could see a use case for both options.

@tabatkins
Copy link
Contributor

The entire point of this feature is that browsers should be smart enough to be able to intelligently choose the "best" resolution to download; they have access to more information than the page author does, and whatever heuristics they come up with to intelligently pick between options get automatically applied to every page using this feature, rather than only being for whatever tiny % of people use a particularly good image-choosing JS library.

If browsers aren't making good decisions, that's a bug on browsers. There shouldn't be any need for the page to further intervene here. It's definitely plausible that browsers are being naive here, so please file bugs showing bad results. ^_^

@eeeps
Copy link
Contributor

eeeps commented Mar 18, 2019

First of all: I agree with @tabatkins , the spec gives browsers great power here, and with that power comes all of the responsibility. The bugs belong with them.

While you're all here though... I barfed out some initial research and thoughts re: Hi-DPR’s diminishing quality returns. https://observablehq.com/@eeeps/visual-acuity-and-device-pixel-ratio

TL;DR @joemcgill I don't remember any browser limiting DPR to 2, but by the looks of it, maybe that's not such a terrible rule-of-thumb to start with.

@mor10
Copy link
Author

mor10 commented Mar 18, 2019

I too agree browsers should be doing this, but from what I see in the real world, they are not. Which gives us an impossible dilemma of either serving gigantic images for fancy 5x screens and making people on expensive data plans pay for image data they neither want nor need, or serving too-small images to all devices causing people on larger screens to question their eyesight.

How do we move forward in a constructive way here? Is the next step reaching out to browser manufacturers and asking them where bandwidth / dataplan throttling is in the pipeline?

Just to reiterate, this issue is currently causing 33% of the web to receive incorrectly sized images via WordPress (through no fault of WordPress). Finding a path toward a solution is non-trivial to the overall UX of the web.

@yoavweiss
Copy link
Contributor

yoavweiss commented Mar 19, 2019

While you're all here though... I barfed out some initial research and thoughts re: Hi-DPR’s diminishing quality returns. https://observablehq.com/@eeeps/visual-acuity-and-device-pixel-ratio

@eeeps - That's amazing work. Thank you for that!!
My read is that the mean 25YO can't see much more than 2x, but that probably means that some chunk of that population can. Is there more data/research on the distribution within that group?

@tabatkins - thoughts on the above and the conclusion? Should browsers use that to cut-off DPR calculations at ~2x? Higher?

@yoavweiss
Copy link
Contributor

Is the next step reaching out to browser manufacturers and asking them where bandwidth / dataplan throttling is in the pipeline?

@mor10 - you already have 😺
As a browser implementor, the next step is to gather data on what the ideal cut-off is, which @eeeps has already started above. Once we reach conclusions on that research, I can probably send an intent to modify the current behavior.

Once Chrome ships this and proves that it's useful behavior, I suspect it won't take too long for other browser vendors to follow.

@colinbendell
Copy link

As an aside, I do think that while a default should be spec'ed, that this should be overridden by the UA and/or markup.

I can imagine scenarios where the UA might desire to have a max-DPR that is non-standard and >2x. I think of TVs as an example. While mobile devices (phones, laptops, watches) have a generally accepted interaction distance, TVs have a less standard use. Home viewing is one thing, but dashboards in the office or campus, or even signage in a store or trade-hall have very different viewing and interaction distances. For this reason I could reasonably see a TV manufacture wanting a 5x DPR to compensate for the viewing and interaction distances.

@yoavweiss
Copy link
Contributor

As an aside, I do think that while a default should be spec'ed, that this should be overridden by the UA and/or markup.

I'm not sure we even want to specify a default, although we may add some data-backed recommendation of a good cut-off value, which browsers can then adopt as they see fit.

@eeeps
Copy link
Contributor

eeeps commented Mar 19, 2019

@mor10 do you have any more detailed data on the “costs” side of things?

I ran your test page through Chrome dev tools on a 1024x768 px viewport at DPRs:

1x: 839K
2x: 1.4MB
3x: 1.4MB

...which, 1.4MB for a page with three images on it is not great, but I feel like there's a better story here about the sort of widespread performance impact that WordPress is trying to avoid by limiting images to 1024w. What would happen to a typical WP site if it adopted the new Twenty-Ninteen theme and all limits were removed, so that the largest srcset resources were the full, user-uploaded versions?

@mor10
Copy link
Author

mor10 commented Mar 19, 2019

I'll update the example with larger image sizes. The original was put together to demonstrate the negative effects of small images on wide screens.

@mor10
Copy link
Author

mor10 commented Mar 19, 2019

@eeeps Here's a new post with an extended range of srcset options, most of which are wider than what WordPress now outputs. The new generated image sizes are based on this proposal which also holds the rationale for the size breakdown based on popular viewport widths.

For reference, my Pixel 3 in horizontal mode now pulls down the 2304px images for all three images in the example.

https://gutenberg.mor10.com/responsive-images-demo-extended-srcset-range/

@aFarkas
Copy link

aFarkas commented Mar 20, 2019

I asked for this or a similar feature already in 2014 (I think optimumdensity attribute).

As it was shut down I created a JS plugin for lazySizes.

@eeeps
There is also a demo page which allows you to constrain pixel density and outputs the sizes. If you think this could be useful I can tweak the tool with other images and a wider range of density. Note: You must use a high density device to use this tool currently.

@yoavweiss
It would be also nice to tweak source selection algorithm with this. Because the simple arithmetic middle isn't that good either. My algorithm looked shit because I'm bad in math but having an algorithm that tends to select the smaller image if both images are high dpi is better and of course the algorithm should tend to select a higher dpi image if both are low dpi.

@mor10
Copy link
Author

mor10 commented Mar 21, 2019

For reference, the WordPress core ticket to find a solution has been punted to a future release: https://core.trac.wordpress.org/ticket/45407#comment:33

@lukasz-galka
Copy link

Hi, has something changed in this topic?

pull bot pushed a commit to Alan-love/chromium that referenced this issue Oct 1, 2020
PSA: https://groups.google.com/a/chromium.org/d/msg/blink-dev/sIGNgOAC0oc/TFST-dJxAwAJ

When srcset selects images, it takes the screen density into account,
but beyond a certain density, that can result in excessive downloaded
images.

Currently, when adding `w` descriptors to srcset, developers run a
risk that large `w` descriptor images will be used on small screens
with very-high density values.

For example, when defining
`<img srcset="300.jpg 300w,
              600.jpg 600w,
              1200.jpg 1200w,
              2400.jpg 2400w">`,
developers that define the 2400px wide image typically intend for it
to be used for very large screens (e.g. a 32" 2x screen), but it often
ends up also being used on small screens (e.g. a phone with 4x DPR and
viewport width of 600+px) .

Research[1] shows that density above 2-2.2x is not visible by most
people, and therefore we could avoid wasting those bytes and avoid
the related slowdown.

See whatwg/html#4421 for full discussion

Change-Id: I52afd505b023767883088a939083f9f835e7fa14

BUG: 1125973

[1] https://observablehq.com/@eeeps/visual-acuity-and-device-pixel-ratio

Change-Id: I52afd505b023767883088a939083f9f835e7fa14
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/2395619
Reviewed-by: Christian Biesinger <[email protected]>
Reviewed-by: Mason Freed <[email protected]>
Commit-Queue: Yoav Weiss <[email protected]>
Cr-Commit-Position: refs/heads/master@{#812698}
@verlok
Copy link

verlok commented Mar 1, 2021

I can see "SrcsetMaxDensity" was committed here, but I can't understand whether it shipped or not.

@yoavweiss would you be able to send an update? Thanks

PS: of course using picture is an option, but it's much more verbose and I think it should be used only when the images have different ratios and other corner cases.

@nhoizey
Copy link

nhoizey commented Mar 1, 2021

I've never seen a good solution with <picture> and srcset-w in <source>s.

You could use <picture> with multiple <source>s with srcset-x, but you won't be able to deal with a fluid layout properly. It only works for multi-steps fixed layouts (as The Guardian for example).

What I usually do is reduce the quality of the image (increasing the compression) on high density screens, as with Compressive Images, to limit network data. But it's really really bad for memory usage on the device.

Here is some simplified pseudo-code for a full viewport width image:

<picture>
  <source
    media="(min-resolution: 3dppx)"
    sizes="100vw"
    srcset="
      img.jpg?w=300&q=15 300w,
      img.jpg?w=600&q=15 600w, …">
  <img
    sizes="100vw"
    srcset="
      img.jpg?w=300&q=auto 300w,
      img.jpg?w=600&q=auto 600w, …"
    alt="">
</picture>

I use q=15 as a "low quality" parameter, and q=auto as the "best" computed quality for optimized weight and preserved visual quality. The actual syntax depends on the image CDN/service you use.

media="(min-resolution: 3dppx)" is "enough" here even if Safari doesn't support dppx because I chose to cap the quality for devices with density above 3dppx and there is no Apple device with a higher density (yet?).

If you want to limit quality above 2dppx (as it would be enough as shown in @eeeps 's study), you have to add min-device-pixel-ratio and it starts becoming more verbose.

So yes, a simple way to limit the maximum resolution the browser uses to compute the size of the required image would be awesome.

@verlok
Copy link

verlok commented Mar 2, 2021

I've never seen a good solution with and srcset-w in s.

I wrote an article about it, How to cap image fidelity to 2x and save 45% image weight on high-end mobile phones, which uses some sources with x, some others with w.

That solution works well to cap image fidelity on smaller viewports (smartphones), and let the w descriptor do its job on larger viewports (tablets and computers).

The thing is that solution is a nightmare quite difficult to design even for an experienced developer, it requires a lot of upfront thinking, and it's a hell to maintain. Especially when you have to write server-side code that generates that picture tag.

That's why I think that a max-density attribute would be great to solve that problem with a single img tag.

@yoavweiss
Copy link
Contributor

That's amazing data, thanks!
/cc @KenjiBaheux

@nhoizey
Copy link

nhoizey commented Mar 2, 2021

@verlok I never thought about using both x and w but for different viewports, well done! 👍

Depending on the target of the site, I would still consider feature phones with a 1dppx density, which would really benefit smaller images. I use a Nokia 8110 4G, for example, with a 240 px viewport and 167 ppi density, to test some sites built for some clients.

As you say, it's quite difficult to generate such code, even more if the layout changes a lot on multiple viewport breakpoints.

We need max-density!

@verlok
Copy link

verlok commented Mar 2, 2021

max-density to the rescue! 💪

@eeeps
Copy link
Contributor

eeeps commented Mar 3, 2021

max-density is cool, but I think we need to think about who gets to make max-useful-resolution decisions, and why...

Initial conversations around max-useful-resolutions were predicated on srcset selection being a browser decision – leveraging the srcset spec's flexibility to eliminate wasted bytes and pixels that almost no one (p90?) would ever see. Changes that affect every srcset loaded by the browser must necessarily be conservative and worst-case (boo) because they affect every single srcset load (yay!).

max-density makes this an author decision (what tradeoff do I want to make for this particular image?). This gives authors the ability to make content- and context- informed decisions using information that browsers don't have access to (yay) but requires authors to do work/not make mistakes, doesn't work for existing content, and probably will never work for a majority of content. (boo)

I strongly think there should be some kind of user preference here, too. Maybe that's just some spec language mandating that browsers can always override max-density (and then tie an override value to something like low-data mode). Author's context/content-dependent decisions are one thing -- end-user decisions about their own particular priorities (especially ones that crank the max down) should trump everything.

Long term, I want all of the above, structured according to the priority of constituencies:

  1. Browsers do their best to make very conservative gains here, for everyone, leveraging the flexibility afforded them by the srcset spec.
  2. Authors can override browsers' default values with some kind of attribute (max-density).
  3. Users via some preference or another (e.g. data-saver mode), have ultimate say, and can override author-provided maxes.

@aFarkas
Copy link

aFarkas commented Mar 3, 2021

@eeeps
I thought we shouldn't introduce more attributes with a dash. isn't maxdensity as content attribute and maxDensity as IDL the way to go?

@verlok
Copy link

verlok commented Apr 13, 2021

More generically, I think browsers should do more in automatically selecting smaller images from a srcset attribute when:

  • network conditions are poor
  • the device is high-end
  • user preference is to save data

What do you think?

@SamuelMiller
Copy link

SamuelMiller commented May 29, 2022

I found someone else's post on StackOverflow that explores a strategy for controlling image pixel density for mobile phones: https://stackoverflow.com/questions/69586765/responsive-images-with-srcset-sizes-media-queries-prevent-loading-huge-image

The basic idea is to associate media queries of the targeted pixel densities with an equivalent size. The browser uses the "sizes" element as a guide to select the appropriate image defined by the (w)idths in the srcset element. In this example, devices with 2x (as default), 3x and 4x pixel ratios will at most display 2x the image. To serve only 1x images to 4x, 3x and 2x devices, specify "24vw", "33vw" and "50vw" in the sizes element. The advantage of this approach is that only one srcset is required.

<img src="500.jpg" title="..." alt="..." srcset="500.jpg 500w, 1000.jpg 1000w, 2000.jpg 2000w" sizes="(min-resolution: 4dppx) 50vw, (min-resolution: 3dppx) 66vw, 100vw" />

The disadvantage is that Safari doesn't support min-resolution. Using "-webkit-min-device-pixel-ratio" as well might give broader browser support. For example,
<img src="500.jpg" title="..." alt="..." srcset="500.jpg 500w, 1000.jpg 1000w, 2000.jpg 2000w" sizes="(-webkit-min-device-pixel-ratio: 4dppx) 50vw, (min-resolution: 4dppx) 50vw,(-webkit-min-device-pixel-ratio: 3dppx) 66vw, (min-resolution: 3dppx) 66vw, 100vw" />

What do ya'll think of this approach?

@verlok
Copy link

verlok commented May 29, 2022

That could work, @SamuelMiller, but the sizes attribute should be left to determine the size of the image, instead of using it to hack and cap image pixel density.

And things would get even more complicate when having to manage different image sizes at different media queries, for real.

We already have a way to cap image pixel density on high dpi devices, and it’s to use the picture tag. I wrote about it In this blog post.

I really long for a maxDensity attribute though, so we wouldn’t have to use the picture tag, which is more complex to write and maintain. Any news on this, @yoavweiss?

Even if now it’s less complicated, thanks to my responsive images automator. :)

@gilbertococchi
Copy link

@yoavweiss Do you think a web standard discussion may be possible at this point?

I came to the conclusion that it's hard for developers to not only cap image qualities to the right viewport size, but it's also not easy to use other signals (like Data Saver, Memory, ETC) to customize the IMG size being requested without using JS...

@yoavweiss
Copy link
Contributor

I'm highly supportive of API improvements that would help developers have more control here. Unfortunately, I won't have time to work on those myself in the near future, but more than happy to guide folks through this if anyone here is interested in contributing :)

mjfroman pushed a commit to mjfroman/moz-libwebrtc-third-party that referenced this issue Oct 14, 2022
PSA: https://groups.google.com/a/chromium.org/d/msg/blink-dev/sIGNgOAC0oc/TFST-dJxAwAJ

When srcset selects images, it takes the screen density into account,
but beyond a certain density, that can result in excessive downloaded
images.

Currently, when adding `w` descriptors to srcset, developers run a
risk that large `w` descriptor images will be used on small screens
with very-high density values.

For example, when defining
`<img srcset="300.jpg 300w,
              600.jpg 600w,
              1200.jpg 1200w,
              2400.jpg 2400w">`,
developers that define the 2400px wide image typically intend for it
to be used for very large screens (e.g. a 32" 2x screen), but it often
ends up also being used on small screens (e.g. a phone with 4x DPR and
viewport width of 600+px) .

Research[1] shows that density above 2-2.2x is not visible by most
people, and therefore we could avoid wasting those bytes and avoid
the related slowdown.

See whatwg/html#4421 for full discussion

Change-Id: I52afd505b023767883088a939083f9f835e7fa14

BUG: 1125973

[1] https://observablehq.com/@eeeps/visual-acuity-and-device-pixel-ratio

Change-Id: I52afd505b023767883088a939083f9f835e7fa14
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/2395619
Reviewed-by: Christian Biesinger <[email protected]>
Reviewed-by: Mason Freed <[email protected]>
Commit-Queue: Yoav Weiss <[email protected]>
Cr-Commit-Position: refs/heads/master@{#812698}
GitOrigin-RevId: 06f97a0fcf40024ca83284dff7b6bace790aaa35
@nhoizey
Copy link

nhoizey commented May 16, 2023

I still think a maxdensity (or maxdpr) attribute would be a great and simple solution.

A difficulty @yoavweiss expressed when we met last week at We Love Speed conference in Paris would be to define which default value it should have.

I agree having a default value (not too agressive, maybe 3) would help improve the situation at a large scale, but I know some developers want to keep their really high fidelity images on 4x screens (maybe even more someday).

So I think it should have no default value:

  • it doesn't change current behavior without developer opt-in
  • it's easier to ship

I wish Yoav's code (or adapted for other browsers of course) was available behind a flag and/or origin trial. It still say "not supported", as reported by @verlok.

@yoavweiss
Copy link
Contributor

@nhoizey - FWIW, you can test that experimental attribute in Chrome with the --enable-blink-features=SrcsetMaxDensity command line argument.

@nhoizey
Copy link

nhoizey commented May 16, 2023

@yoavweiss as @verlok said, Chrome says it is not supported (here in French 😅):
image

EDIT: But I did check, and it works! 😲
I remember not being able to test months before, I must have done something wrong.

@yoavweiss
Copy link
Contributor

It doesn't change much if you just want to test the feature. The warning is there so that real users won't use this not-necessarily-tested feature.

@nhoizey
Copy link

nhoizey commented May 16, 2023

Ok, now I understand "unsupported" doesn't mean the "browser" doesn't support this feature, but "Google" doesn't support you if you use it. 🤦‍♂️

Sorry.

@yoavweiss
Copy link
Contributor

^^ @progers - FYI

@AndresInSpace
Copy link

AndresInSpace commented Sep 24, 2024

Responsive images, eg. srcset/sizes, are not meant to reduce your image weight.
They are meant to serve the appropriately sized images based on device capabilities and screen size.
It is doing that.

What has changed?
Mobile devices are now 3 or 4x pixel density and everyone is on a 5G mobile network, compared to 2x max density and 3G connections back in the day.

There is documentation saying the browsers do take network conditions into account when deciding what picture source or srcset size to use, but I am not sure on how this is actually calculated and how it effects what size/source the browser picks.

I believe this is entirely an implementation issue with how browsers are currently selecting the srcset size or picture source to be used and the browsers are not properly taking network condition/speed into account during this selection process.

The hard part is thresholds and knowing 'what is slow' relative to 'what source is going to be too large?' when the only known information to pick a proper size/source is the aspect ratio, device pixel density, and network condition.
Ie. I could have a 3000px image that weighs 60kb or I could have a 1000px image that weighs 180kb. Browser has know way of knowing this relative to connection or defined 'size'.

@verlok
Copy link

verlok commented Sep 25, 2024

Being able to assign a maxdpr attribute (was srcsetmaxdensity, I just renamed it to a shorter form) in the img tag would solve the problem of devices being 3x or 4x nowadays.

So, on those devices, the multiplication browsers would have to do to find the ideal intrinsic width of the image to download would be:

idealWidth = sizes * min(deviceDPR, maxDPR)

@verlok
Copy link

verlok commented Sep 25, 2024

I mean I wouldn’t point fingers to browser vendors, as the quality of the image might depend on the website we are building and on the page type.

I usually advise to cap DPR to 2 on product listing pages, whilst I leave it to 3 on product detail pages, and it should be uncapped size on PDP Zoom images.

@verlok
Copy link

verlok commented Sep 25, 2024

What do we need to do to push for this attribute to become a standard?

@AndresInSpace
Copy link

AndresInSpace commented Sep 25, 2024

Being able to assign a maxdpr attribute (was srcsetmaxdensity, I just renamed it to a shorter form) in the img tag would solve the problem of devices being 3x or 4x nowadays.

So, on those devices, the multiplication browsers would have to do to find the ideal intrinsic width of the image to download would be:

idealWidth = sizes * min(deviceDPR, maxDPR)

I understand. I just don't believe that this effectively resolves the underlying issue - it is a bandaid and I find it redundant. Why code in a 3x source when you are gonna add this maxdpr flag anyways? Just remove the source reference. Makes no sense to me. CDN scripts/app components creating srcset/sizes automatically is not an excuse.

A proper resolution would not remove or hinder functionality from devices that can handle 3x 4x in order to 'fix this' for devices that are technically able, but in practically sense, incapable. The underlying variable is connection speed, not device density. If your connection is fiber optics, then maxdpr wouldn't even be discussed, and no one would bat an eye.

There are good points made here regarding if we actually are able to distinguish 2x+ quality increases, etc, but I believe those conversations are off topic in regards to contributing towards a resolution for this specific issue.

Just look at video. Best quality eg 4k, fullhd, or 720p, is used based on connection speed, unless the user specified a preference. Same deal, really.

I see one path forward here, and it's connection speed assisting determination of device density browser uses, or something of similar effect to ensure assets load in a timely manner relative to connection.

Bad connection? Consider your device now a 1x display.

Edit: consider this Real life Scenario:
I'm in a car as a passenger on a phone with a 3x display. The car is passing through a 3g area, and in 5-10 minutes I will be back on 5g.
Q: How do I avoid long loading images to ensure optimal site loading / ux while I'm on a bad connection with device density over 1?
A: Ensure we don't serve any 1x+ images even if device is capable when we know connection won't handle it.

Possible Browser Flow:
browser connection check uses connection based override for img calc eg 1x, caches img downloaded, connection improvement triggers update to recheck img sizes eg 3x, apply new src

Result:
I can now visit webpage and use it, not having to wait for large images to download.
Additionally, image quality improves automatically when my connection does.
Should not cause recalc as img ratio is same we are simply grabbing the true device density now.

@verlok
Copy link

verlok commented Sep 25, 2024

Why code in a 3x source when you are gonna add this maxdpr flag anyways?

When you declare the srcset with the w descriptor, you might need a wider image to serve the correct size to a wide, desktop, hidpi display.

@verlok
Copy link

verlok commented Sep 25, 2024

I see one path forward here, and it's connection speed assisting determination of device density browser uses, or something of similar effect to ensure assets load in a timely manner relative to connection.

I disagree. Users of a luxury e-commerce website still want to see the full quality image, especially on the product detail page, no matter what the connection speed. They are willing to wait. I have proven this with data.

So it depends on the use case and this choice should be left to developers.

@AndresInSpace
Copy link

I see one path forward here, and it's connection speed assisting determination of device density browser uses, or something of similar effect to ensure assets load in a timely manner relative to connection.

I disagree. Users of a luxury e-commerce website still want to see the full quality image, especially on the product detail page, no matter what the connection speed. They are willing to wait. I have proven this with data.

So it depends on the use case and this choice should be left to developers.

A flag to enforce device density and ignore connection optimization makes more sense to me for what you describe.

Why do you want to kneecap devices with maxdpr in first place? Cause it's slow if they have a 4x used for a thumbnail?.. Isn't that the point of srcset and whoever defined the 4x source - it gets used.
but also you saying they want to wait, to get 4x, but not for thumbnails only for hero or fullsize.

Browsers:
Use defined srcsets when possible, as currently doing.
Use connection based calc over device density, unless flagged to disable network optimization, to select img srcset src

Satisfies maximum devices capability, satisfies ux improvement, satisfies user control / use cases I believe

@AndresInSpace
Copy link

AndresInSpace commented Sep 28, 2024

I read back and looked at this more.
@SamuelMiller posted a workaround that can be implemented currently #4421 (comment)

From my understanding:

  • when devices came out supporting higher resolutions, the css pixel ratio was defined and resolution/pixel ratio calculation was implemented to avoid putting the onus of resolution scale on the web developer.
  • images are set resolutions
  • srcset sizes provided web developers a way to provide images that address the upscale at the same ratio eg resolution scaling for device density
  • srcset sizes now causes increased bandwidth

I guess the viewpoints I am looking at this from is:

  • I don't think the web developer OR the browser should decide what the client is capable of - it's capable of 4x it knows that about itself.
  • Browsers have much more information available to them to make optimization decisions than Web Developers do. Web developers would additionally have to write more code to 'optimize'.
  • On a high-speed connection devices will not suffer the described topic issue related to srcset size selection and resolution scale. srcsetmaxdensity usage would be a moot point to implement as it is not solving an issue anymore and is instead de-optimizing the devices capability.
  • Additionally, I don't think it should be up to web dev or browser to decide if the client can 'perceive the difference', as the issue is not with perception, but bandwidth. I think this is an overstep. This would be like having a 4k TV, buying media to play on it, yet the media forces 1080p despite it also having the 4k source. Now maybe you put that media on a platform to stream to your 4k TV and connection is bad -- then that makes sense for it to downscale to 1080p. Concept is the same. I would also imagine that the device manufactures with literally millions of dollars of R&D can tell us the exact threshold for perception across ages and they wouldn't be making devices past XYZ and essentially wasting company money.
  • I do think the browser should optimize the choice of scale based on connection, in order to avoid it being another issue web developers have to specifically add code to address.
  • I do think web developers should be given a choice to override the browser optimization to enforce quality over connection in order to ensure high-priority imagery is sent based on client capabilities.

In conclusion:
I just don't understand why we are talking about implementing srcsetmaxdensity for web developers when the spec for srcset puts the onus for bandwidth optimization on the browser, when the only reason srcsetmaxdensity would be used is because the scaled source matched is causing bandwidth bottlenecks.

I mean even something like sizes="(min-resolution: 1x) 1x, 100vw" would make more sense in my eyes allowing granular control. Browsers handle connection optimization and if sizes x unit is defined for query it overrides the browser calc to enforce quality defined over the connection optimization.

Additional Info:
Several JS frameworks with image components already implement this connection check and then remove the scale factor appropriately. They console log it out saying instead of a 2x a 1x was served based on connection speeds.

Why can't browsers implement this natively to address the underlying issue, ie. connection?
#4421 (comment)

@verlok
Copy link

verlok commented Sep 29, 2024

@AndresInSpace I think you’re off track with your rational. You’re following the hack path and not what the sizes attribute means. I suggest you to read this very good article about Eric Portis which clarifies what srcset and sizes mean. Hopefully it will clarify the meaning to you as it did to me.

@verlok
Copy link

verlok commented Sep 29, 2024

My point of view on the hack is still this.

And as @gilbertococchi was suggesting, I think this is time to try and make it a standard.

I read again the comments from months ago and the difficulties at choosing a default value for this attribute. I agree the default shkukd be none. Optionally, we could discuss an auto value which takes into consideration network speed, the “save data” setting, device memory, etc. as I think @AndresInSpace is suggesting in his latest comments, but I still think the default value should be none, for backwards compatibility.

@yoavweiss can you point me to the right direction on how to propose it as a standard?

@AndresInSpace
Copy link

AndresInSpace commented Sep 29, 2024

@verlok Yes, I read that before posting actually! Made sure to refresh my memory lol!
Web developers can enforce a size as that is the point of size attribute.
Ie. here is small device resolution, middle device resolution, big device resolution.
I am trying to convey the distinction between effect required - which we agree on essentially - and proposed implementation using maxdpr. (Also, I agree with your statement.)

We need both, working together
Edit: essentially what @eeeps described

  1. Browsers implementing an optimization process to change scale during concrete size calculation / srcset selection based on defined conditions (ie network, device memory, etc) as @yoavweiss discussed implementing here and @aFarkas discussed here and as CSS specification defines here and as Web.dev outlines here

  2. Feature implementation (spec change) to give developers a way to override the browsers selected factor for scale used during optimization of srcset selection / concrete size.
    Ie. what is now the currently proposed 'maxdpr'.


Facts:
The resolution size factor is scaled by the browser according to dpr/dppx to pick srcset match ( concrete/default object size).

The browsers currently do not optimize this process by down-scaling, ie changing dpr/dppx applied in srcset match calculation, to account for the underlying issues of 1 and this whole thread.
The optimization process would account for:

network speed, the “save data” setting, device memory, etc.

The Distinction:

The proposed implementation of maxdpr (2), in my eyes, seems to rest all of this on the web developer and should instead be in tandem with browsers optimization to scale selection process (1).
The proposed 'maxdpr' can not be just an override, without browsers implementing optimization to scale selection process which would be the default/auto value, in order to truly resolve the issue at hand for the future, as well as right now.


Edit - taking picture into account
Additionally the proposed change for 'maxdpr' to be applied on the img tag would create a blanket effect when used with picture and interfere with art direction and the intended usage for maxdpr.

Use case:
on mobile I serve 4:5 ratio and cap at 1.5x as a blanket speed optimization, also allowing browser to further down-scale to 1x if connection is real bad
anything on desktop is 16:9 and I want these devices to support their capabilities to the max provided connection allows. eg 4x
picture w img maxdpr makes them all cap 1.5x

This was why I provided the size example using a 'x' unit size as it would address the maxdpr for img and work with picture, and still effectively is in-line with your statement that size should tell the size - we are - just the scaled size. (I mean, it's an interesting discussion in itself, an 'x' css unit size, but diverges from the topic at hand so I digress).

Ensuring that the maxdpr is applied to <source> for <picture> and overrides the <img> maxdpr, would address this use case.

@verlok
Copy link

verlok commented Sep 29, 2024

I would define that maxdpr aka srcset-max-density is ignored when the picture tag is used.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Development

Successfully merging a pull request may close this issue.