feat(DRAFT): Vendored vega_datasets demo

[dangotbanned]

Related

• `vega_datasets` source #3150
• feat: allow for creating new_series / from_dict without specifying a backend narwhals-dev/narwhals#876 (comment)

Description

Early WIP

Providing a minimal, but up-to-date source for https://github.com/vega/vega-datasets

Notes

• Investigating bundling metadata (22a5039), (1792340)
  • Depending on how well the compression scales, it might be reasonable to include this for some number of versions
  • Deliberately including redundant info early on - can always chip away at it later

Outstanding issues

Currently a low priority item personally, so keeping track of the longer term issues to fix here

npm/vega-datasets does not have every version available at https://github.com/vega/vega-datasets/tags

• Known missing
  • https://github.com/vega/vega-datasets/releases/tag/v1.9.0
  • https://github.com/vega/vega-datasets/releases/tag/v1.6.0
  • https://github.com/vega/vega-datasets/releases/tag/v1.4.0
• feat(DRAFT): Add a source for available npm versions
• Need to add some handling to invalidate these entries returned from list-repository-tags once confirmed they cannot be requested from npm
  • Can technically request from github, but during testing this was much slower
  • Also, these versions would not have been available from https://github.com/altair-viz/vega_datasets, since that only used npm

Plan strategy for user-configurable dataset cache

• Everything so far has been building the tools for a compact bundled index
  • 1, 2, 3, 4, 5
  • Refreshing the index would not be included in altair, each release would simply ship with changes baked in
• Trying to avoid bloating altair package size with datasets
• User-facing
  • Goal of requesting each unique dataset version once
    • The user cache would not need to be updated between altair versions
  • Some kind of opt-in config to say store the datasets in this directory please
    • Basic solution would be defining an env variable like ALTAIR_DATASETS_DIR
    • When not provided, always perform remote requests
      • User motivation would be that it would be faster to enable caching
• There may be opportunities to reduce the cache footprint further
  • e.g. storing the .(csv|tsv|json) files as .parquet
• Need to do more testing on this though to ensure
  • the shape of each dataset is preserved
  • where relevant - intentional errors remain intact