Make benchmarks navigable and add fps tests for setData

[mcwhittemore]

@lucaswoj & @ansis here is a benchmark showing fps degrading as the zoom level approaches 22 when setting data on a geojson source.

Would love feedback on how to make this better judge performance. FPS doesn't capture what is rendered on the canvas that well for instance.

[mcwhittemore]

Is this good to merge?

[lucaswoj]

Not yet. I'll give this a review later today.

[lucaswoj]

There is a syntax error here (no , after 'id': 'geojson-polygon-fill').

My apologies, I broke the linter in #2314 and fixed in #2347. Can you rebase on master and fix the lint errors?

[lucaswoj]

Do you think we could generalize our existing fps benchmark to capture this perf regression? (i.e. by measuring FPS over the set of tiles specified in coords.js)

Having fewer & robuster benchmarks will help us find future performance regressions.

[mcwhittemore]

@lucaswoj - Looking into this more, what I had feels a bit past contrived. Limiting this PR to a simple restructuring of the benchmarks as I look into this some more.

[mcwhittemore]

This adds to setData tests to the fps bench as well per chat. One for a large feature collection, for a small feature collection.

@mourner or @lucaswoj for the review

[lucaswoj]

Why did you choose fps to measure setData performance? Can we measure tile creation time directly?

[lucaswoj]

Ah! I just saw where you wrote:

This adds to setData tests to the fps bench as well per chat. One for a large feature collection, for a small feature collection.

So sorry for misspeaking. I didn't mean to imply that we should test setData within the FPS benchmark. These should be two separate benchmarks.

• the fps benchmark tests the fps across a variety of streets-v8 viewports (i.e. those in coords.js)
• the set-data benchmark tests the time from a GeoJSONSource#setData call to the data being rendered on screen for different sized GeoJSONs

[mcwhittemore]

Cool. Breaking these up. I put them together as seeing the normal fps was helpful, but I agree this limits the FPS benchmark from getter better.

[mcwhittemore]

@lucaswoj - updated the PR to break setData and FPS apart. Also changed setData to track the average time it takes to load all the tiles after a setData. This should be a better metric for setData perf than fps, though time to load tiles could go down and fps be negatively impacted.

I don't think we want to break this test into two files per: Having fewer & robuster benchmarks will help us find future performance regressions So I'm using two ends which feels dirty. To be honest, I think we should worry about the best way to do this kind of tracking when we're writing the automation benchmark code. Right now we're trying to find what is best for our own usage. We can confirm automation to that.

[lucaswoj]

Having two scores in this benchmark is a non-starter for me. Breaking this up is very easy. You are measuring two separate things.

[mcwhittemore]

@lucaswoj updated to be two tests.

[lucaswoj]

Looks good 🚢