core(throttling): add option to slowdown to a device class

[patrickhulce]

Summary
First iteration of what #5871 laid the groundwork for, there are quite a few ways of going about doing this, but the determination from before was that we can't do precise 2.353x throttling decisions but we can tell when a device has WAAAAAY different perf characteristics from another and adjust accordingly.

Nothing will change by default, but this PR introduces a new throttling property cpuTargetDeviceClass (@benschwarz we need your sage naming wisdom here 😆), with 4 main device classes based on the data collected for #5871. 8x 4x 2x and 1x on a Macbook Pro. If the host device is in the 2x class, and the target device class is 4x, the multiplier will be reduced to 2x instead of using 4x, and so on.

Related Issues/PRs
#5871

[benschwarz]

@patrickhulce given the range of devices people are likely to run LH on, do we need more data provided? I can more than likely help there

[benschwarz]

This looks really good so far @patrickhulce - The only thing that I'm 🤔 about is how people will remember what very-slow actually means? Could we class them in some other way? (I thought of years, "-1 year" -- like how people use autoprefixer etc… but that doesn't feel right, I think)

[patrickhulce]

given the range of devices people are likely to run LH on, do we need more data provided? I can more than likely help there

Sure the more the better! The most important thing IMO would be selecting cutoffs that will affect the fewest people. i.e. there's the fewest devices on the border of two categories.

how people will remember what very-slow actually means? Could we class them in some other way?

In the original throttling doc I wrote I had used something more like "Premium Mobile", "Budget Mobile", etc, but given the disparity between premium android and premium iOS it's not exactly accurate or fair. Year is also tricky because new budget android devices (for example Alcatel 1X) are much slower than phones from 5 years ago. Definitely on board with finding something as descriptive/accurate as possible but I'm at a loss for what that actually is haha 🤷‍♀️

[benschwarz]

Definitely on board with finding something as descriptive/accurate as possible but I'm at a loss for what that actually is haha 🤷‍♀️

I've the same feels atm. Let's sit on it for a day or two and see if anything falls out?

[patrickhulce]

Let's sit on it for a day or two and see if anything falls out?

So now that it's 7 days, has anything fallen out of anyone? ;)

[benschwarz]

I think from what we've seen, professional performance advocates will do the research into the devices and choose something that works for the circumstances that that are being tested for, whereas "performance interested" folks are more likely to take our guidance on what the test devices are.

Given that, I think the best naming convention that comes to mind is exactly what you've come up with.

I'd love to see a table of the kinds of devices that each setting represents, something that can be referenced and updated in (likely a year or so?). Maybe with an accompanying benchmarkIndex for completeness:

Setting	Benchmark index range	Devices
very-slow	0—100	Alcatel 1C
slow	100—500	Motorola G3
medium	500—700	iPhone 6
fast	> 1000	iPhone X
none	> 1250	Late model Macbook

[patrickhulce]

@brendankenny @paulirish any opinions here?

[brendankenny]

immediate thoughts:

• ideally we'd only have one way to set the CPU multiplier. Does anyone use cpuSlowdownMultiplier except DevTools? We could get rid of it with v4 (internally we can use whatever form(s) we want to, of course)
• we try not to alter settings after being passed in (I can't say for sure we don't :). Since this should be just for emulation at gathering time and the rest of the time used in audits for lantern stuff (and assuming eliminating cpuSlowdownMultiplier re: above), could we have an emulation property for this in gather-runner and have the rest be based on an artifact?

[brendankenny]

Does anyone use cpuSlowdownMultiplier except DevTools? We could get rid of it with v4

just to be clear, I mean get rid of it and replace it with cpuTargetDeviceClass selection instead :)

[patrickhulce]

I definitely like all the things you're saying. Treating this as a computed artifact for the real multiplier SGTM assuming the other pieces look good to folks to proceed.

I'm not convinced we can eliminate cpuSlowdownMultiplier though :/ I think the strongest use case for keeping it is in LR/DAZL-like situations when you have a fleet of servers you want to run lots of tests with exact same settings and not waste time running the benchmark every time (yet to be implemented :D). I see a future world in which we only run the benchmark when cpuTargetDeviceClass is set.

[patrickhulce]

The other of the stagnant PRs I mentioned today :)

This would be nice to clear up before v4 if our decision involves eliminating cpuSlowdownMultiplier (I'm not sure we want to do that), but otherwise can wait for post-CDS.

[brendankenny]

Yeah, your point above is a good one, and even if do get more confident that we can eliminate cpuSlowdownMultiplier, we'll likely want a decent transition time anyways as we tune the benchmark, interpretation of the results, etc.

Makes sense to start with it as just additive.

[patrickhulce]

bump :)

[paulirish]

This works for me.

Title of the PR is a little misleading though. Feel more like... `core(throttling): add option to slowdown to a device class" or sumpthin.

The plan is to try this out and see how it goes before setting the multiplier dynamically for everyone, right?

[exterkamp]

LGTM
I know this is a preliminary PR and isn't really applied yet, but I was thinking about how we might surface this to the user to let them know that the benchmark index has been adjusted, some ideas:



Or just the final device class in wording:

[patrickhulce]

Feel more like... `core(throttling): add option to slowdown to a device class" or sumpthin.

done!

The plan is to try this out and see how it goes before setting the multiplier dynamically for everyone, right?

yep

I was thinking about how we might surface this to the user to let them know that the benchmark index has been adjusted, some ideas:

Oooh I like those. I was also thinking it might be part of a much larger rework of the environment settings. Something that communicates more clearly how it was run and what we were targeting.

[benschwarz]

I know this is a preliminary PR and isn't really applied yet, but I was thinking about how we might surface this to the user to let them know that the benchmark index has been adjusted, some ideas:

I prefer your first option @exterkamp — It's important that people can clearly see what the machine was rated at, as well as the slowing applied to it 👍

[patrickhulce]

This isn't ready for primetime, and we've decided to revisit this with perhaps a new benchmark in the future.