fix: Selecting top-most hierarchy value by default in filters

[bprusinowski]

Fixes #1017.

The main problem with #1016 was that I only updated the types in client.readQuery in configurator-state.tsx

DataCubeMetadataWithComponentValuesQuery → DataCubeMetadataWithComponentValuesQueryVariables
DataCubeMetadataWithComponentValuesQueryVariables → DataCubeMetadataWithComponentValuesAndHierarchiesQueryVariables

but missed updating the document (DataCubeMetadataWithComponentValuesDocument → DataCubeMetadataWithComponentValuesAndHierarchiesDocument), so we were accessing dimensions without hierarchies in this place.

This PR fixes that (and also adds a unit test to make sure we select top-most hierarchy value by default in filters).

TODO

• Remove DataCubeMetadataWithComponentValues & DataCubeMetadataWithComponentsValuesAndHierarchies in favor of using DataCubeMetadata & DimensionValues (so we can avoid fetching the dimension data twice in the editor mode in some cases)

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
visualization-tool	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Apr 14, 2023 8:58am

[Rdataflow]

good to see the functionality is recovering - OTOH loading times for published charts double by this PR. 👁️
i.e
https://visualization-tool-nsubpeats-ixt1.vercel.app/v/y_66ZxOF-Mox?dataSource=Test ~5s
vs
https://test.visualize.admin.ch/v/G3Plp_XPARAI?dataSource=Test ~2s

why not for published charts rely on the list and ordering in the chartConfig?
i.e. the order stored in chartConfig.filters.$iri.values from the DB?

[Rdataflow]

@bprusinowski preview performance is also negatively impacted
see https://visualization-tool-nsubpeats-ixt1.vercel.app/browse?dataset=https%3A%2F%2Fenvironment.ld.admin.ch%2Ffoen%2Ffab_foreign_trade_test_1%2F1&dataSource=Test&flag__debug=true takes 5s now
where https://test.visualize.admin.ch/browse?dataset=https%3A%2F%2Fenvironment.ld.admin.ch%2Ffoen%2Ffab_foreign_trade_test_1%2F1&dataSource=Test took 2s before

also https://visualization-tool-nsubpeats-ixt1.vercel.app/de/browse?dataset=https%3A%2F%2Fenvironment.ld.admin.ch%2Ffoen%2Fnfi%2FJstat_fullHierarchy_C-94%2Fcube%2F2023-1&dataSource=Test&flag__debug=true takes 6s
where https://test.visualize.admin.ch/de/browse?dataset=https%3A%2F%2Fenvironment.ld.admin.ch%2Ffoen%2Fnfi%2FJstat_fullHierarchy_C-94%2Fcube%2F2023-1&dataSource=Test took 2s

this is very unfortunate 🤔

[bprusinowski]

Hi @Rdataflow, thank you for taking the time to do the testing 🙇‍♂️

I am fairly confident that there were no changes introduced in this pull request that could affect loading times of published charts or dataset preview, as there were only changes related to components used in the editing mode 👀

It's true that the data on TEST seems to loads faster, but – if we look at the preview deployment where the initial performance improvements were introduced (#1016, link to the same chart created using that deployment: https://visualization-tool-git-perf-published-chart-ixt1.vercel.app/en/v/4HVZl08NN58Y?dataSource=Test) it looks like loading time is comparable between the old preview deployment and the chart created by you using preview deployment from this PR.

I am not 100% sure, but my assumption is that it should work faster on the TEST environment, @ptbrowne do you see any potential explanations here?

PS. I also tested the performance manually by refreshing the pages with dataset preview for this PR (link) and for #1016 (link) and it looked quite random; sometimes this deployment was faster and sometimes it was the old deployment.

[Rdataflow]

@ptbrowne in case you feel confident the observed bad performance is not related to the code you may go on and double check again after this PR is merged 👍 we just need to ensure there is no bad impact

[Rdataflow]

performance on test as of now is updated in above comments

[ptbrowne]

LGTM, thanks Bartosz.

[Rdataflow]

@bprusinowski unfortunately https://test.visualize.admin.ch/de/browse?dataset=https%3A%2F%2Fenvironment.ld.admin.ch%2Ffoen%2Fnfi%2FJstat_fullHierarchy_C-94%2Fcube%2F2023-1&dataSource=Test now takes much longer than 2s previously, rather 4s+. can you check and fix this regression?

NB: the other examples (see above) look fine and had no impact 👍

cc @ZellerSabine

[bprusinowski]

Hi @Rdataflow, I used an old deployment preview coming from March 31 (#1005), before the changes in this PR (and any recent changes to data fetching) to check the loading times before / after this PR. It seems that it also used to take about 4s to load the dataset (see link).

Can you share more details on how you tested the regression in this example?

[Rdataflow]

@bprusinowski I used what was deployed on TEST as of Apr 6 at 3PM...

[bprusinowski]

Hi, thanks for the info! Taking a look right now at TEST, it seems to load quick https://test.visualize.admin.ch/de/browse?dataset=https%3A%2F%2Fenvironment.ld.admin.ch%2Ffoen%2Fnfi%2FJstat_fullHierarchy_C-94%2Fcube%2F2023-1&dataSource=Test. Maybe the server was busy last time when testing? Is it also loading fast for you @Rdataflow now?

[adintegra]

Thanks for looking into this @bprusinowski 🙇‍♂️ Agree that there may have been some transient issues upstream unrelated to the code changes in this PR. The shared URLs are responding fine for me as well.

[Rdataflow]

@bprusinowski the cube switched to a new url https://test.visualize.admin.ch/de/browse?dataset=https%3A%2F%2Fenvironment.ld.admin.ch%2Ffoen%2Fnfi%2Fnfi_C-94%2Fcube%2F2023-1&dataSource=Test

the total page loading time is still the same (5s+)
this GraphQL query alone takes 4s typically while before none of the graphql took so long (see above)

 time curl -s 'https://test.visualize.admin.ch/api/graphql' -H 'x-visualize-cache-control: no-cache'   --data-raw $'{"query":"query DataCubePreview($iri: String\u0021, $sourceType: String\u0021, $sourceUrl: String\u0021, $locale: String\u0021, $latest: Boolean, $filters: Filters) {\\n  dataCubeByIri(\\n    iri: $iri\\n    sourceType: $sourceType\\n    sourceUrl: $sourceUrl\\n    locale: $locale\\n    latest: $latest\\n  ) {\\n    iri\\n    title\\n    description\\n    publicationStatus\\n    dimensions(sourceType: $sourceType, sourceUrl: $sourceUrl) {\\n      ...dimensionMetadata\\n      __typename\\n    }\\n    measures(sourceType: $sourceType, sourceUrl: $sourceUrl) {\\n      ...dimensionMetadata\\n      __typename\\n    }\\n    observations(sourceType: $sourceType, sourceUrl: $sourceUrl, limit: 10) {\\n      data\\n      sparql\\n      sparqlEditorUrl\\n      __typename\\n    }\\n    __typename\\n  }\\n}\\n\\nfragment dimensionMetadata on Dimension {\\n  iri\\n  label\\n  description\\n  isNumerical\\n  isKeyDimension\\n  dataType\\n  order\\n  values(sourceType: $sourceType, sourceUrl: $sourceUrl, filters: $filters)\\n  unit\\n  related {\\n    iri\\n    type\\n    __typename\\n  }\\n  ... on TemporalDimension {\\n    timeUnit\\n    timeFormat\\n    __typename\\n  }\\n  ... on NumericalMeasure {\\n    isCurrency\\n    currencyExponent\\n    resolution\\n    isDecimal\\n    __typename\\n  }\\n}\\n","operationName":"DataCubePreview","variables":{"iri":"https://environment.ld.admin.ch/foen/nfi/nfi_C-94/cube/2023-1","sourceType":"sparql","sourceUrl":"https://test.lindas.admin.ch/query","locale":"de"}}'   -o /dev/null

real 0m4.054s
user 0m0.015s
sys 0m0.046s

@adintegra can you repro page loading time of 2s - with serverside cache: disabled and share how to achieve that? I would be interested...