Picture/Image Optimisations - Solving DPR & Scaling bugs by rendering…

… sources for each breakpoint (#3641) * Refactor picture component to resolve HDPI bug, add basic tests * Fix side effects from using sort in getClosestSetForWidth, add tests for getSourcesFromSrcSets * Update some image sizes to be more accurate * Prettier fixes * Add sources optimiser to reduce DOM size * Fix supporting image query * Update comments * Add support for edge case where there are no hdpi sources * Fix MDPI media query, fix stype on mpdi source variable names * comment typo fix * Use react fragment for fallback case whern there is no hdpi sources * Add Pixel as type alias to increase readability The code use 'number' to describe both 'pixel' and 'viewport widths'. This add `Pixel` type alias to make the return type explicit and ease code reading. * Refactor Picture.tsx to make types more clear & defined, tidy up logic & functions for readability * Round breakpont / 2 to avoid .5 in sizes attribute * WIP: Picture Documentation * Update pictures documentation * Update picture documentation with more accurate explanation of Source sets Co-Authored-By: Max Duval <[email protected]> * Remove sizes attribute from sources as they`re redundant * Update documentation to remove sizes as they`re redundant Co-authored-by: Mariot Chauvin <[email protected]> Co-authored-by: Max Duval <[email protected]>
guardian · Nov 29, 2021 · d7921a4 · d7921a4
1 parent b488069
commit d7921a4
Show file tree

Hide file tree

Showing 3 changed files with 549 additions and 67 deletions.
diff --git a/dotcom-rendering/docs/architecture/027-pictures.md b/dotcom-rendering/docs/architecture/027-pictures.md
@@ -0,0 +1,128 @@
+# Pictures in DCR
+
+This document outlines how (and why) pictures in DCR work the way they do. A lot of methodology comes from Frontend, however DCR didn't immediately have parity with Frontend for image/picture rendering, and it has been built on over time to improve it.
+
+The overall goal of the picture rendering to ensure that we serve the correct resolution image to the user in all cases. This can involve looking at things like the page & image layout, as well as client properties like DPR.
+
+## Background Info
+
+This background info section aims to give enough information for anyone who hasn't worked a lot with picture or source elements, or source sets. Feel free to skip it if you like!
+
+DCR uses the html `<picture>` tag to render images. This offers us a advantage over regular `<img>` tags, and that is the use of `<source>` elements to further help the browser understand what size & quality image to download, saving our users bandwidth & us money.
+
+### Fastly Image Optimiser
+
+They key enabler in this is the [fastly image optimiser](https://developer.fastly.com/reference/io/). This allows us to specify image widths when requesting an image URL. for example for a given image, say `https://i.guim.co.uk/img/media/xxxxx/yyy.jpg`, we are able to specify some important image transformation properties: `width`, `quality` & `dpr` (& others).
+
+-   `width` allows us to specify the width of the image in pixels.
+-   `quality` allows us to specify how much to compress the image, 0 being very compressed, and 100 preserving the best quality possible.
+-   `dpr` or Device Pixel Ratio allows us to scale the size of the image by this number, for example using `?width=300&dpr=2` would return a 600px wide image.
+
+### Media queries (`media="query"`)
+
+Within any given `<picture>` tag, you can have multiple `<source>` child elements. Using the `media` html attribute, which uses the same syntax as CSS media queries, you can tell the browser which `<source>` element to look for an image source in.
+
+For example `<source media="min-width: 600px">` would tell the browser to pick this source if the viewport is 600px or wider.
+
+The browser will choose and stick with the first matching source element it finds, so it's important to ensure they're in the DOM in the right order.
+
+Another important media queries we use is (`(orientation: portrait)`) to check if it's a portrait device (e.g a smartphone).
+
+### Sizes (`sizes=`)
+
+The sizes html attribute acts as the translation layer between the size of the viewport and the size of the image source you'd like to pick. A good way to think about this is that, beyond a certain width, main media (inline) images never go beyond `620px` wide, and this html attribute gives us a way to communicate that to the browser.
+
+For example `<source sizes="(min-width: 660px) 620px, 100vw">` tells the browser, "Hey, if your viewport is 660px or wider, always look for an image source which is 620px wide. If not, default to an image which is the same width as 100% of the viewport width". We can provide as many sizes & queries as we want, with the `(query) px, (query) px, ... , px)` syntax, where the last argument without a query is the default if none others match.
+
+### Source Set (`srcset=`)
+
+Source sets work as our final piece of the puzzle. Our browser has already picked a `source` element, and has used `sizes` to figure out what size (in width only) image it is looking for. Our source set allows us to provide a list of URLs & the width of the image for each one, which the browser will then use to look for the best fitting image from.
+
+Source sets are formatted like: `<source srcset="https://url.to/image 300w, https://url.to/larger/image 600w">` Our comma separated list of sources specifies first the URL to a given image source, then the real pixel width for that image. The unit `w` is used, to distinguish pixels inside the image from CCS `px` on the screen–there are many pixels per CSS `px` on high DPI screens. In our case where we use Fastly image optimiser, the only thing changing between these image urls is in the query parameters, e.g `?dpr=2` or `?width=300` and `?width=600`.
+
+### DPR?!
+
+DPR originates from the concept that the pixel widths which we use for CSS media queries is often different from the actual resolution of a devices display.
+For example imagine a phone with a super high resolution 1200px wide (2400px high) screen. Following our breakpoint sizes, we'd try and render a desktop type experience for this user. However, the reality is this screen is only maybe 5 inches across, so the site would be totally unusable.
+
+CSS Pixels & DPR to the rescue! Our browser can use a different width for calculating breakpoints, media queries, etc than the real resolution of the screen. This is CSS pixels. So let's say for the sake of argument the browser uses 300px for our CSS pixel width - Wohoo, we're displaying a mobile experience, all is well. The DPR is the ratio between CSS pixels and actual resolution, so 1200 / 300 gives us a DPR of 4. Why this is important will be discussed later.
+
+#### The DPR Problem
+
+This problem comes from how the browser tries to compensate for high DPR displays when choosing an image source. In previous iterations of Frontend & DCR, we provided only 2 sources - a regular set, and a set of sources for high DPR displays; targeted with a media query to ensure it's picked.
+
+Unfortunately the browser itself would try to compensate for high DPR as well, but in a less efficient way. Once the browser had figured out what size source it wants from the `sizes` attribute on our source element, it's then multiplied by the DPR of the display to get the desired width it will look for in our `srcset`:
+
+```
+# desiredWidth is the width of an image that the browser will look for in `srcset`
+# size is the chosen size based off our queries in the `sizes` attribute
+# DPR is the ration between CSS Pixels & device resolution (e.g 2, 3, 4)
+desiredWidth = size * DPR
+```
+
+This posed a problem for us, because if we tell the browser "Hey choose a 620px image", and the browser has a DPR of say 3, it will actually look for an image source for 1860px, far higher resolution than is needed for the user to have a good experience.
+
+#### The DPR Solution
+
+This problem was first solved in Frontend, then replicated in DCR
+
+Rather than having just 2 source elements, we instead have 2 source elements per breakpoint (one for high DPR, one for regular displays), and provide just one size & source set for that source element.
+
+Lets look at a simplified example (with only 1 source per breakpoint):
+
+```html
+<picture>
+    <source
+        media="min-width: 980px"
+        srcset="https://xxx.png?width=620px 620w"
+    />
+    <source
+        media="min-width: 660px"
+        srcset="https://xxx.png?width=620px 620w"
+    />
+    <source
+        media="min-width: 480px"
+        srcset="https://xxx.png?width=480px 480w"
+    />
+    <source
+        media="min-width: 375px"
+        srcset="https://xxx.png?width=420px 420w"
+    />
+</picture>
+```
+
+In this example, we have the logic that usually would have been in our sizes attribute (`sizes="(min-width: 660px) 620px, 100vw"`) extrapolated into individual source elements. All our breakpoint which are 660px or larger offer only 1 choice, the 620px source. The lower breakpoints have looked for sources which are closest to their own size, as a replacement for `100vw`.
+
+This solves our DPR problem because, the `media` attribute uses CSS pixels, and because we only offer 1 image source for each of these elements, once the browser has picked its source element, we basically strong arm it into which source to use.
+
+## What does DCR do?
+
+DCR Maintains some parity with Frontend's implementation of images, the key difference's being:
+
+1. DCR Relies on Frontend to generate it's image sources
+2. Frontend offers higher resolution source for portait immersives
+3. DCR Removes redundant sources to make the DOM more effecient.
+
+The implementation DCR picked involves creating 2 <source> elements for each breakpoint, one regular, and one for high DPR displays.
+
+For Example:
+
+```html
+<picture>
+    <source srcset="https://i.guim.co.uk/img/media/picture.jpg?width=620&;quality=45&;auto=format&;fit=max&;dpr=2&; 1240w" media="(min-width: 980px) and (-webkit-min-device-pixel-ratio: 1.25), (min-width: 980px) and (min-resolution: 120dpi)">
+    <source srcset="https://i.guim.co.uk/img/media/picture.jpg?width=620&;quality=85&;auto=format&;fit=max&; 620w" media="(min-width: 980px)">
+    <source srcset="https://i.guim.co.uk/img/media/picture.jpg?width=700&;quality=45&;auto=format&;fit=max&;dpr=2&; 1400w" media="(min-width: 740px) and (-webkit-min-device-pixel-ratio: 1.25), (min-width: 740px) and (min-resolution: 120dpi)">
+    <source srcset="https://i.guim.co.uk/img/media/picture.jpg?width=700&;quality=85&;auto=format&;fit=max&; 700w" media="(min-width: 740px)">
+    <source srcset="https://i.guim.co.uk/img/media/picture.jpg?width=620&;quality=45&;auto=format&;fit=max&;dpr=2&; 1240w" media="(min-width: 660px) and (-webkit-min-device-pixel-ratio: 1.25), (min-width: 660px) and (min-resolution: 120dpi)">
+    <source srcset="https://i.guim.co.uk/img/media/picture.jpg?width=620&;quality=85&;auto=format&;fit=max&; 620w" media="(min-width: 660px)">
+    <source srcset="https://i.guim.co.uk/img/media/picture.jpg?width=620&;quality=45&;auto=format&;fit=max&;dpr=2&; 1240w" media="(min-width: 480px) and (-webkit-min-device-pixel-ratio: 1.25), (min-width: 480px) and (min-resolution: 120dpi)">
+    <source srcset="https://i.guim.co.uk/img/media/picture.jpg?width=620&;quality=85&;auto=format&;fit=max&; 620w" media="(min-width: 480px)">
+    <source srcset="https://i.guim.co.uk/img/media/picture.jpg?width=465&;quality=45&;auto=format&;fit=max&;dpr=2&; 930w" media="(min-width: 375px) and (-webkit-min-device-pixel-ratio: 1.25), (min-width: 375px) and (min-resolution: 120dpi)">
+    <source srcset="https://i.guim.co.uk/img/media/picture.jpg?width=465&;quality=85&;auto=format&;fit=max&; 465w" media="(min-width: 375px)">
+    <source srcset="https://i.guim.co.uk/img/media/picture.jpg?width=465&;quality=45&;auto=format&;fit=max&;dpr=2&; 930w" media="(min-width: 320px) and (-webkit-min-device-pixel-ratio: 1.25), (min-width: 320px) and (min-resolution: 120dpi)">
+    <source srcset="https://i.guim.co.uk/img/media/picture.jpg?width=465&;quality=85&;auto=format&;fit=max&; 465w" media="(min-width: 320px)">
+    <source srcset="https://i.guim.co.uk/img/media/picture.jpg?width=465&;quality=45&;auto=format&;fit=max&;dpr=2&; 930w" media="(min-width: 0px) and (-webkit-min-device-pixel-ratio: 1.25), (min-width: 0px) and (min-resolution: 120dpi)">
+    <source srcset="https://i.guim.co.uk/img/media/picture.jpg?width=465&;quality=85&;auto=format&;fit=max&; 465w" media="(min-width: 0px)">
+    <img alt="The Palace Theatre, London, showing Harry Potter and the Cursed Child" src="https://i.guim.co.uk/img/media/picture.jpg?width=465&;quality=45&;auto=format&;fit=max&;dpr=2&;s=0492ab78e73c5167d8b4b841e601fbd4" height="1200" width="2000" class="dcr-b5pnrc-css">
+</picture>
+```
diff --git a/dotcom-rendering/src/web/components/Picture.test.tsx b/dotcom-rendering/src/web/components/Picture.test.tsx
@@ -0,0 +1,223 @@
+import { breakpoints } from '@guardian/src-foundations/mq';
+import type { DesiredWidth } from './Picture';
+import { getBestSourceForDesiredWidth, removeRedundantWidths } from './Picture';
+
+const hdpiSources: SrcSetItem[] = [
+	{
+		src: '1',
+		width: 1400,
+	},
+	{
+		src: '2',
+		width: 1240,
+	},
+	{
+		src: '3',
+		width: 930,
+	},
+	{
+		src: '4',
+		width: 1290,
+	},
+];
+
+const mdpiSources: SrcSetItem[] = [
+	{
+		src: '1',
+		width: 620,
+	},
+	{
+		src: '2',
+		width: 700,
+	},
+	{
+		src: '3',
+		width: 465,
+	},
+	{
+		src: '4',
+		width: 645,
+	},
+];
+
+/**
+ *       mobile: 320
+ *       mobileMedium: 375
+ *       mobileLandscape: 480
+ *       phablet: 660
+ *       tablet: 740
+ *       desktop: 980
+ *       leftCol: 1140
+ *       wide: 1300
+ */
+
+describe(`Picture`, () => {
+	describe('getClosestSetForWidth', () => {
+		it('Gets the closest source for a given width (hdpi)', () => {
+			// Breakpoints
+			expect(
+				getBestSourceForDesiredWidth(
+					breakpoints.mobile * 2,
+					hdpiSources,
+				).width,
+			).toBe(930);
+			expect(
+				getBestSourceForDesiredWidth(
+					breakpoints.mobileMedium * 2,
+					hdpiSources,
+				).width,
+			).toBe(930);
+			expect(
+				getBestSourceForDesiredWidth(
+					breakpoints.mobileLandscape * 2,
+					hdpiSources,
+				).width,
+			).toBe(1240);
+			expect(
+				getBestSourceForDesiredWidth(
+					breakpoints.phablet * 2,
+					hdpiSources,
+				).width,
+			).toBe(1400);
+			expect(
+				getBestSourceForDesiredWidth(
+					breakpoints.tablet * 2,
+					hdpiSources,
+				).width,
+			).toBe(1400);
+			expect(
+				getBestSourceForDesiredWidth(
+					breakpoints.desktop * 2,
+					hdpiSources,
+				).width,
+			).toBe(1400);
+			expect(
+				getBestSourceForDesiredWidth(
+					breakpoints.leftCol * 2,
+					hdpiSources,
+				).width,
+			).toBe(1400);
+			expect(
+				getBestSourceForDesiredWidth(breakpoints.wide * 2, hdpiSources)
+					.width,
+			).toBe(1400);
+
+			// Example widths
+			expect(
+				getBestSourceForDesiredWidth(620 * 2, hdpiSources).width,
+			).toBe(1240);
+		});
+
+		it('Gets the closest source for a given width (mdpi)', () => {
+			// Breakpoints
+			expect(
+				getBestSourceForDesiredWidth(breakpoints.mobile, mdpiSources)
+					.width,
+			).toBe(465);
+			expect(
+				getBestSourceForDesiredWidth(
+					breakpoints.mobileMedium,
+					mdpiSources,
+				).width,
+			).toBe(465);
+			expect(
+				getBestSourceForDesiredWidth(
+					breakpoints.mobileLandscape,
+					mdpiSources,
+				).width,
+			).toBe(620);
+			expect(
+				getBestSourceForDesiredWidth(breakpoints.phablet, mdpiSources)
+					.width,
+			).toBe(700);
+			expect(
+				getBestSourceForDesiredWidth(breakpoints.tablet, mdpiSources)
+					.width,
+			).toBe(700);
+			expect(
+				getBestSourceForDesiredWidth(breakpoints.desktop, mdpiSources)
+					.width,
+			).toBe(700);
+			expect(
+				getBestSourceForDesiredWidth(breakpoints.leftCol, mdpiSources)
+					.width,
+			).toBe(700);
+			expect(
+				getBestSourceForDesiredWidth(breakpoints.wide, mdpiSources)
+					.width,
+			).toBe(700);
+
+			// Example widths
+			expect(getBestSourceForDesiredWidth(620, mdpiSources).width).toBe(
+				620,
+			);
+		});
+	});
+
+	describe('optimiseBreakpointSizes', () => {
+		it('Leaves un-optimisable breakpointSizes as-is', () => {
+			const breakPointSizes: DesiredWidth[] = [
+				{ breakpoint: 1000, width: 500 },
+				{ breakpoint: 800, width: 400 },
+				{ breakpoint: 600, width: 300 },
+				{ breakpoint: 400, width: 200 },
+			];
+			expect(removeRedundantWidths(breakPointSizes)).toEqual(
+				breakPointSizes,
+			);
+		});
+
+		it('Correctly removes optimisable breakpointSizes', () => {
+			expect(
+				removeRedundantWidths([
+					{ breakpoint: 1000, width: 500 },
+					{ breakpoint: 800, width: 400 },
+					{ breakpoint: 600, width: 400 },
+					{ breakpoint: 400, width: 200 },
+				]),
+			).toEqual([
+				{ breakpoint: 1000, width: 500 },
+				{ breakpoint: 600, width: 400 },
+				{ breakpoint: 400, width: 200 },
+			]);
+
+			expect(
+				removeRedundantWidths([
+					{ breakpoint: 1000, width: 500 },
+					{ breakpoint: 800, width: 400 },
+					{ breakpoint: 600, width: 200 },
+					{ breakpoint: 400, width: 200 },
+				]),
+			).toEqual([
+				{ breakpoint: 1000, width: 500 },
+				{ breakpoint: 800, width: 400 },
+				{ breakpoint: 400, width: 200 },
+			]);
+
+			expect(
+				removeRedundantWidths([
+					{ breakpoint: 1000, width: 500 },
+					{ breakpoint: 800, width: 200 },
+					{ breakpoint: 600, width: 200 },
+					{ breakpoint: 400, width: 200 },
+				]),
+			).toEqual([
+				{ breakpoint: 1000, width: 500 },
+				{ breakpoint: 400, width: 200 },
+			]);
+
+			expect(
+				removeRedundantWidths([
+					{ breakpoint: 1000, width: 500 },
+					{ breakpoint: 800, width: 500 },
+					{ breakpoint: 600, width: 300 },
+					{ breakpoint: 400, width: 200 },
+				]),
+			).toEqual([
+				{ breakpoint: 800, width: 500 },
+				{ breakpoint: 600, width: 300 },
+				{ breakpoint: 400, width: 200 },
+			]);
+		});
+	});
+});