Specify storage resolution

[patrobinson]

If metrics are retrieved at a higher resolution than 1min we should store them at a higher resolution in CloudWatch

[triarius]

It looks like the API has the ability to change the PollDuration dynamically with a response header:

buildkite-agent-metrics/collector/collector.go

Lines 152 to 159 in a0e44e3

	if pollSeconds := res.Header.Get(PollDurationHeader); pollSeconds != "" {
	pollSecondsInt, err := strconv.ParseInt(pollSeconds, 10, 64)
	if err != nil {
	log.Printf("Failed to parse %s header: %v", PollDurationHeader, err)
	} else {
	result.PollDuration = time.Duration(pollSecondsInt) * time.Second
	}
	}

buildkite-agent-metrics/main.go

Lines 122 to 164 in a0e44e3

	f := func() (time.Duration, error) {
	t := time.Now()
	result, err := c.Collect()
	if err != nil {
	return time.Duration(0), err
	}
	if !*dryRun {
	err = bk.Collect(result)
	if err != nil {
	return time.Duration(0), err
	}
	}
	log.Printf("Finished in %s", time.Now().Sub(t))
	return result.PollDuration, nil
	}
	minPollDuration, err := f()
	if err != nil {
	fmt.Println(err)
	}
	if *interval > 0 {
	for {
	waitTime := *interval
	// Respect the min poll duration returned by the API
	if *interval < minPollDuration {
	log.Printf("Increasing poll duration based on rate-limit headers")
	waitTime = minPollDuration
	}
	log.Printf("Waiting for %v (minimum of %v)", waitTime, minPollDuration)
	time.Sleep(waitTime)
	minPollDuration, err = f()
	if err != nil {
	fmt.Println(err)
	}
	}
	}

Though the API's source code seems to have it hard-coded to 10s.

This would mean that everyone using the cloudwatch backend would suddenly be using the high resolution storage if this were merged, and they started using it.

I think we should make the high resolution storage optional for this reason. Perhaps we can make the default to use the high resolution storage but allow customers to turn it off (and poll less frequently) if it costs them more than they expected.

I don't think we should increase the hard-coded 10s value in the API. Presumably it is more relevant to other backends.

[patrobinson]

oh weird, we do send an interval along in the user agent, but the API ignores it?

[patrobinson]

The first time this runs lastPollTime will be 0 so this will be epoch seconds (i.e. 1720764897). But since we only care if it's less than or more than 60 seconds then that's inconsequential. The first metric we send within this lambda context won't be high frequency, but that should be OK

[wolfeidau]

👍🏻 🚀