Support budget for categories

[alekseykulikov]

It's useful to keep Lighthouse categories in check, especially those, that rely on static page analysis, like accessibility, seo, and best-practices. Decrease of the score is often a sign of a quality regression.

This feature will allow analyzing pages with Lighthouse in CI and will help to make projects like https://github.com/treosh/lighthouse-ci-action more useful.

Proposed format:

[
  {
    "path": "/*",
    "categories": [
      {
        "id": "accessibility",
        "minimumScore": 1
      },
      {
        "id": "seo",
        "minimumScore": 0.9
      },
      {
        "id": "best-practices",
        "minimumScore": 0.8
      }
    ]
  }
]

The format is open for debate. The proposed one is inspired the current Lighthouse report on categories with id and minimumScore value between 0 and 1.

[khempenius]

Huge +1 to this. This is a great idea. Thanks for opening this issue.

Happy to bike shed over format :)

This is the format that I'm currently leaning towards. Very interested in feedback on this:

[
  {
    "path": "/*",
    "lighthouse": [
      {
        "category": "accessibility",
        "budget": 1
      },
      {
        "category": "seo",
        "budget": 0.9
      },
      {
        "category": "best-practices",
        "budget": 0.8
      }
    ]
  }
]

• minimumScore --> budget
budget would match the syntax used elsewhere in budget.json to declare budgets. However, the one possibly weird thing about this is that in this context a budget would represent a minimum threshold, but everywhere else it is a maximum threshold. I have very mixed feelings on this and I can also see score or mimimumScore making more sense here. @patrickhulce might also have opinions here since we were just discussing something similar here.

• categories --> lighthouse
  If Lighthouse were to change how it works (e.g. no categories, or a significant feature that isn't categories), this would be more flexible.

• Integer vs. decimal (for budget/score)
  I had mixed feelings at first but I think I'm also leaning towards using decimal format here. I think which format seems more intuitive to a user probably depends on how they use Lighthouse: if they use the HTML report an integer probably makes more sense; if they consume the JSON version then decimal probably makes more sense. However the fact that LighthouseCI also uses the decimal score format probably pushed me over the edge for preferring decimal over integer.

[patrickhulce]

Specific Responses/Thoughts

However, the one possibly weird thing about this is that in this context a budget would represent a minimum threshold, but everywhere else it is a maximum threshold.

The name here to me is highly dependent on how the statisticalApproach shakes out. If it stays as the current proposal (applies with file-scope and is named optimistic/pessimistic) then I think we're fine to continue using budget and put the onus on the budget validator logic to correct for this flip of min/max desirability.

categories --> lighthouse

I like this change a lot 👍 not only for the flexibility of Lighthouse assertions into the future but also the siloing of lighthouse-specific config going into an explicitly named property, just for it (see meta point below).

Meta Point

I'm not sure yet if I love lighthouse-specific assertions in the budgets.json format :)

On the one hand, I see budgets.json as an important generic web tooling format that could encourage every tool to start caring about webperf and be able to directly leverage the existing budgets.json files out there is a low barrier to entry. I'm a little concerned that including lighthouse-specific things in there might muddy that message a little. When we have alternative, lighthouse-specific solutions in lighthouse-ci on the horizon, feels a tad like an unnecessary risk.

On the other hand, I'm a Lighthouse engineer that would love to see these aspects of web quality taken into consideration as widely as possible, so inclusion in a generic budgets format is awesome! :)

[alekseykulikov]

@khempenius I agree with all your suggestions. Moving LH logic to a separate section makes sense. budget is a better name, and min/max logic should go from the context. Scores and sizes have reverse nature, but conceptually the idea of budget is always to cap the worst case.

Meta-level

@patrickhulce initially, I thought about budget.json as a tool to run assertions for Lighthouse and programmatically generate snapshots to capture the state of the URL. But now, I see lighthouse-ci assertions, and also not sure if budget.json should have LH specific logic, or to be purely generic format that follows web standards.

[shortcircuit3]

👍 Would love to see this happen!