[Maps] Ensure EMS has versioned public APIs

[thomasneirynck]

EMS will need to be backward-compliant using a date-based versioning. Ensure Kibana connects appropriately.

Edits by @jsanz 👇

Changes needed

With the new date-based version labels we need to make some modifications to the different components of the Elastic Maps Service.

New APIs

Apart from the unified Stack SemVer versions, EMS will be exposing two different API endpoints, one for the data boundaries at vector.maps.elastic.co (EMS File Service), and another for the basemaps served at tiles.maps.elastic.co.

File Service

Currently the data boundaries are served from a static hosting. All data is stored in a single folder as GeoJSON and TopoJSON files, and manifests are generated on version folders (v7.1, v7.11, etc.).

So first we need to extend current datasets metadata properties to include support for date-based versions. We need to generate new manifests in folders named as new API date-based versions, very likely keeping the the newest version at a pinned latest folder.

Second, a new vector.maps.elastic.co backend needs to:

• Keep serving manifests and styles from the SemVer requests as before (ex: https://vector.maps.elastic.co/v8.7/manifest)
• Serve the latest manifest from the root folder when no header is passed to https://vector.maps.elastic.co/manifest
• If date-based version header is present, reverse proxy to the folder URL, returning the document or a correct error status if it is not found.

Tile Service

The requests to our manifest and basemaps styles contain the SemVer label in the URL, but just to validate the string without any further processing. There is only one basemaps and one set of styles and manifests.

As per #136642 we are doing a small version negotiation to include new style names to our basemaps. The affected code path will need also to understand the new versioning scheme.

The new header could be only understood by the main /manifest endpoint. The response will contain the date-based version in the styles URLS, so we don't need to modify Maplibre requests to include a header.

If we ever want to version also our styles and basemaps data we can include the logic in the same fashion we will do to our basemap names.

EMS Landing Page

EMS Landing Page is currently also versioned with folders. Since this is not an API endpoint we don't need to change how we serve the website and keep it in the static hosting we use now. We may want to change the logic for the version in the root folder to be always the latest release, so the serverless version of Kibana can always point to https://maps.elastic.co and we can run the deploy of the root with tags with the date naming convention independently if run them from the main branch or the last major release.

Currently EMS Landing Page already generates folders for branches with the date name schema, check the Buildkite pipeline definition

Tasks

Kibana and EMS Client

Preview Give feedback

1. [EMS] Point EMS Landing Page to the root directory #161791
   Feature:Maps Team:Presentation impact:medium loe:small +4
   
2. [Maps] Ensure EMS Client makes valid Serverless requests #161790
   Feature:Maps Project:Serverless Team:Presentation impact:high loe:medium +5
   
3. Update the client to understand serverless versioning ems-client#185
   v8.5.0 +1

EMS File Service

Preview Give feedback

1. Generate EMS File Service manifests for serverless ems-file-service#286
   +0
   
2. Process EMS File Service requests in the current EMS Tile Service backend
3. Ensure EMS buckets accept elastic-api-version
4. [Maps] Create OAS files for EMS File and Tile Services #166286
   Feature:Maps Team:Presentation +2
   
5. EMS File Service: migrate vector.maps.elastic.co and catalogue.maps.elastic.co to point to the same backend for tiles.maps.elastic.co,

EMS Tile Service

Preview Give feedback

1. EMS Tile Service: support the new header for the manifest request, providing style locations with the API version in the URL
2. EMS Tile Service: support the date-based version in the styles requests
3. Deploy to Staging
4. Deploy to Production
5. [Maps] Create OAS files for EMS File and Tile Services #166286
   Feature:Maps Team:Presentation +2
   

EMS Landing Page

Preview Give feedback

1. Ensure date-based versions can be deployed to the root ems-landing-page#630
   +0
   
2. Upgrade ems-client to use latest endpoints ems-landing-page#668
   enhancement +1
   

[elasticmachine]

Pinging @elastic/kibana-presentation (Team:Presentation)

[jsanz]

@thomasneirynck @nickpeihl I'm going to leave the documentation part aside and set this effort as done.

This week finally got merged changes in both Kibana¹ and EMS Landing Page² so both products access the /latest/manifest endpoints. OAS documentation has been published in the ems-client repo.

Not directly related with serverless, but I included in this effort also the update of the basemaps styles and data but I found an issue and I left that for later.

Closing, but please let me know if there is feedback or comments.

Footnotes

1. in main, to be included in next Serverless promotion ↩

2. in staging now, to be published soon as well ↩