Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Heartbeat] Support google cloud storage for suites #30571

Closed
andrewvc opened this issue Feb 24, 2022 · 4 comments
Closed

[Heartbeat] Support google cloud storage for suites #30571

andrewvc opened this issue Feb 24, 2022 · 4 comments
Labels
enhancement Heartbeat Team:obs-ds-hosted-services Label for the Observability Hosted Services team

Comments

@andrewvc
Copy link
Contributor

andrewvc commented Feb 24, 2022

In #29917 (discovery mode) we'll want to enable the use of google cloud storage for downloaded suites, where after the discovery phase we upload the zip, and during the execution phase we pull a zip from GCP instead of a standard URL.

During the discovery phase we need to support cloud storage for a new discovery.source_cache option as in the example below. Note that we only need to support pre-signed URLs for our purposes.

heartbeat.monitors:
 - id: elastic-monitor
  name: Elastic Monitor
  discovery:
    enabled: true
      # optional, place to re-upload the downloaded zip for subsequent executions.
    source_cache:
      gcp_cloud_storage:
        signed_url: "http://presignedurl"
  schedule: '@every 1m'
  source:
    zip_url: 
      url: "https://github.com/elastic/synthetics-demo/archive/refs/heads/main.zip"

During execution we need to support this option as a source type, where it can entered as such:

heartbeat.monitors:
 - id: elastic-monitor
  name: Elastic Monitor
  source:
    gcp_cloud_storage:
      signed_url: "http://presignedurl"
      deps_preinstalled: true # disable the NPM install because it's already correctly bundled in the zip
  schedule: '@every 1m'
@andrewvc andrewvc added enhancement Heartbeat Team:obs-ds-hosted-services Label for the Observability Hosted Services team labels Feb 24, 2022
@elasticmachine
Copy link
Collaborator

Pinging @elastic/uptime (Team:Uptime)

@vigneshshanmugam
Copy link
Member

vigneshshanmugam commented Feb 25, 2022

I am actually in favor of calling them cloud_storage as opposed to leaking cloud provider information on the fields. We could add provider metadata as a sub field. Thoughts?

Example:

heartbeat.monitors:
  source:
    cloud_storage:
      provider: gcp
      signed_url: "http://presignedurl"
  schedule: '@every 1m'

Also, we haven't explicitly mentioned how Heartbeat would only run a single journey when invoking the Synthetics while running in the discovery mode for suites. Is it going to filter journeys by name or tags? Can we also add here to the proposal so it completes the whole model.

@paulb-elastic
Copy link
Contributor

@andrewvc is this a dupe of 30295?

@andrewvc
Copy link
Contributor Author

andrewvc commented Mar 8, 2022

Yep ,closing as dupe of #30295

@andrewvc andrewvc closed this as completed Mar 8, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement Heartbeat Team:obs-ds-hosted-services Label for the Observability Hosted Services team
Projects
None yet
Development

No branches or pull requests

4 participants