Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Heartbeat] Support Upload / Download of suite zips to GCP cloud storage #30295

Closed
Tracked by #432
andrewvc opened this issue Feb 9, 2022 · 2 comments
Closed
Tracked by #432
Labels
enhancement Heartbeat Team:obs-ds-hosted-services Label for the Observability Hosted Services team

Comments

@andrewvc
Copy link
Contributor

andrewvc commented Feb 9, 2022

As part of elastic/uptime#432 we'll need to support sending zips to GCP cloud storage after the discovery phase, and the retrieval of that same asset in the execution phase. This lets the discovery phase happen on one node, and many parallel invocations of execution happen on other nodes without re-downloading the asset from the original source. Additionally, we can upload a zip with node_modules pre-installed speeding up execution.

This would work hand-in-hand with #29917 and is blocked by it.

Sample Discovery Config

# Discovery config
heartbeat.monitors:
 - id: elastic-monitor
  name: Elastic Monitor
  discovery:
    enabled: true
    store:
      gcp:
        bucket: "my-bucket-name"
        base_path: "/path/to/my/zips"
        auth: # GCP auth settings if not using instance creds
  schedule: '@every 1m'
  source:
    zip_url: 
      url: "https://github.com/elastic/synthetics-demo/archive/refs/heads/main.zip"

Additional fields in discovery doc

The bucket and full path would be added to the discovery doc as:

{
  "suite": { }, // omitting for brevity
  "cached_source": {
    "gcp_url": { // note the structure here is identical to the heartbeat yaml in the execution config under `source`
       "bucket": "my-bucket-name",
       "path": "/path/to/my/zips/oeuh89g89h9823.zip"
     }
  }
}

Sample Execution Config

# Execution config
heartbeat.monitors:
 - id: elastic-monitor
  name: Elastic Monitor
  schedule: '@every 1m'
  source:
    gcp_url: 
      bucket: "my-bucket-name"
      path: "/path/to/my/zips/oeuh89g89h9823.zip"
  filter_journeys:
    match: my-exact-journey-name
@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Feb 9, 2022
@andrewvc andrewvc added Heartbeat Team:obs-ds-hosted-services Label for the Observability Hosted Services team and removed needs_team Indicates that the issue/PR needs a Team:* label labels Feb 9, 2022
@elasticmachine
Copy link
Collaborator

Pinging @elastic/uptime (Team:Uptime)

@andrewvc
Copy link
Contributor Author

Closing in favor of elastic/synthetics#470

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement Heartbeat Team:obs-ds-hosted-services Label for the Observability Hosted Services team
Projects
None yet
Development

No branches or pull requests

2 participants