Skip to content

Commit

Permalink
Refactoring of azure obj store #3957 (#3970)
Browse files Browse the repository at this point in the history
* Rebased feature - initial commit of azure obj store extend

Signed-off-by: Wiard van Rij <[email protected]>

* use custom cortex to fix config change

Signed-off-by: Wiard van Rij <[email protected]>

* modules acting up

Signed-off-by: Wiard van Rij <[email protected]>

* remove sprint

Signed-off-by: Wiard van Rij <[email protected]>

* adds dot

Signed-off-by: Wiard van Rij <[email protected]>

* removed need for changes on cortex side

Signed-off-by: Wiard van Rij <[email protected]>

* adds changelog

Signed-off-by: Wiard van Rij <[email protected]>

* Update docs/storage.md

Co-authored-by: Philip Laine <[email protected]>
Signed-off-by: Wiard van Rij <[email protected]>

* Update pkg/objstore/azure/azure_test.go

Co-authored-by: Philip Laine <[email protected]>
Signed-off-by: Wiard van Rij <[email protected]>

* Update pkg/objstore/azure/azure_test.go

Co-authored-by: Philip Laine <[email protected]>
Signed-off-by: Wiard van Rij <[email protected]>

* update a few cleanups

Signed-off-by: Wiard van Rij <[email protected]>

* fixes go.mod and go.sum?

Signed-off-by: Wiard van Rij <[email protected]>

* fixes whitespace

Signed-off-by: Wiard van Rij <[email protected]>

* fixes space

Signed-off-by: Wiard van Rij <[email protected]>

* updates azure Go-autorest

Signed-off-by: Wiard van Rij <[email protected]>

* fixes readme

Signed-off-by: Wiard van Rij <[email protected]>

* fixes go.sum.. (╯°□°)╯︵ ┻━┻

Signed-off-by: Wiard van Rij <[email protected]>

* e2e retest

Signed-off-by: Wiard van Rij <[email protected]>

Co-authored-by: Philip Laine <[email protected]>
  • Loading branch information
wiardvanrij and phillebaba authored Jul 5, 2021
1 parent 897f35f commit 9355a82
Show file tree
Hide file tree
Showing 7 changed files with 389 additions and 113 deletions.
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ We use *breaking :warning:* to mark changes that are not backward compatible (re
- [#4239](https://github.com/thanos-io/thanos/pull/4239) Add penalty based deduplication mode for compactor.
- [#4292](https://github.com/thanos-io/thanos/pull/4292) Receive: Enable exemplars ingestion and querying.
- [#4392](https://github.com/thanos-io/thanos/pull/4392) Tools: Added `--delete-blocks` to bucket rewrite tool to mark the original blocks for deletion after rewriting is done.
- [#3970](https://github.com/thanos-io/thanos/pull/3970) Azure: Adds more configuration options for Azure blob storage. This allows for pipeline and reader specific configuration. Implements HTTP transport configuration options. These options allows for more fine-grained control on timeouts and retries. Implements MSI authentication as second method of authentication via a service principal token.

### Fixed

Expand Down
20 changes: 20 additions & 0 deletions docs/storage.md
Original file line number Diff line number Diff line change
Expand Up @@ -319,8 +319,28 @@ config:
container: ""
endpoint: ""
max_retries: 0
msi_resource: ""
pipeline_config:
max_tries: 0
try_timeout: 0s
retry_delay: 0s
max_retry_delay: 0s
reader_config:
max_retry_requests: 0
http_config:
idle_conn_timeout: 0s
response_header_timeout: 0s
insecure_skip_verify: false
tls_handshake_timeout: 0s
expect_continue_timeout: 0s
max_idle_conns: 0
max_idle_conns_per_host: 0
max_conns_per_host: 0
disable_compression: false
```

If `msi_resource` is used, authentication is done via ServicePrincipalToken. The value for Azure should be `https://<storage-account-name>.blob.core.windows.net`. The generic `max_retries` will be used as value for the `pipeline_config`'s `max_tries` and `reader_config`'s `max_retry_requests`. For more control, `max_retries` could be ignored (0) and one could set specific retry values.

#### OpenStack Swift

Thanos uses [ncw/swift](https://github.com/ncw/swift) client to upload Prometheus data into [OpenStack Swift](https://docs.openstack.org/swift/latest/).
Expand Down
3 changes: 2 additions & 1 deletion go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ require (
cloud.google.com/go/storage v1.10.0
github.com/Azure/azure-pipeline-go v0.2.2
github.com/Azure/azure-storage-blob-go v0.8.0
github.com/Azure/go-autorest/autorest/azure/auth v0.5.8
github.com/NYTimes/gziphandler v1.1.1
github.com/alecthomas/units v0.0.0-20210208195552-ff826a37aa15
github.com/aliyun/aliyun-oss-go-sdk v2.0.4+incompatible
Expand Down Expand Up @@ -63,7 +64,7 @@ require (
go.uber.org/atomic v1.7.0
go.uber.org/automaxprocs v1.2.0
go.uber.org/goleak v1.1.10
golang.org/x/crypto v0.0.0-20210220033148-5ea612d1eb83
golang.org/x/crypto v0.0.0-20210513164829-c07d793c2f9a
golang.org/x/oauth2 v0.0.0-20210427180440-81ed05c6b58c
golang.org/x/sync v0.0.0-20210220032951-036812b2e83c
golang.org/x/text v0.3.6
Expand Down
12 changes: 11 additions & 1 deletion go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -61,13 +61,19 @@ github.com/Azure/go-autorest v14.2.0+incompatible/go.mod h1:r+4oMnoxhatjLLJ6zxSW
github.com/Azure/go-autorest/autorest v0.9.0/go.mod h1:xyHB1BMZT0cuDHU7I0+g046+BFDTQ8rEZB0s4Yfa6bI=
github.com/Azure/go-autorest/autorest v0.9.3-0.20191028180845-3492b2aff503/go.mod h1:xyHB1BMZT0cuDHU7I0+g046+BFDTQ8rEZB0s4Yfa6bI=
github.com/Azure/go-autorest/autorest v0.11.12/go.mod h1:eipySxLmqSyC5s5k1CLupqet0PSENBEDP93LQ9a8QYw=
github.com/Azure/go-autorest/autorest v0.11.17/go.mod h1:eipySxLmqSyC5s5k1CLupqet0PSENBEDP93LQ9a8QYw=
github.com/Azure/go-autorest/autorest v0.11.18 h1:90Y4srNYrwOtAgVo3ndrQkTYn6kf1Eg/AjTFJ8Is2aM=
github.com/Azure/go-autorest/autorest v0.11.18/go.mod h1:dSiJPy22c3u0OtOKDNttNgqpNFY/GeWa7GH/Pz56QRA=
github.com/Azure/go-autorest/autorest/adal v0.5.0/go.mod h1:8Z9fGy2MpX0PvDjB1pEgQTmVqjGhiHBW7RJJEciWzS0=
github.com/Azure/go-autorest/autorest/adal v0.8.1-0.20191028180845-3492b2aff503/go.mod h1:Z6vX6WXXuyieHAXwMj0S6HY6e6wcHn37qQMBQlvY3lc=
github.com/Azure/go-autorest/autorest/adal v0.9.5/go.mod h1:B7KF7jKIeC9Mct5spmyCB/A8CG/sEz1vwIRGv/bbw7A=
github.com/Azure/go-autorest/autorest/adal v0.9.11/go.mod h1:nBKAnTomx8gDtl+3ZCJv2v0KACFHWTB2drffI1B68Pk=
github.com/Azure/go-autorest/autorest/adal v0.9.13 h1:Mp5hbtOePIzM8pJVRa3YLrWWmZtoxRXqUEzCfJt3+/Q=
github.com/Azure/go-autorest/autorest/adal v0.9.13/go.mod h1:W/MM4U6nLxnIskrw4UwWzlHfGjwUS50aOsc/I3yuU8M=
github.com/Azure/go-autorest/autorest/azure/auth v0.5.8 h1:TzPg6B6fTZ0G1zBf3T54aI7p3cAT6u//TOXGPmFMOXg=
github.com/Azure/go-autorest/autorest/azure/auth v0.5.8/go.mod h1:kxyKZTSfKh8OVFWPAgOgQ/frrJgeYQJPyR5fLFmXko4=
github.com/Azure/go-autorest/autorest/azure/cli v0.4.2 h1:dMOmEJfkLKW/7JsokJqkyoYSgmR08hi9KrhjZb+JALY=
github.com/Azure/go-autorest/autorest/azure/cli v0.4.2/go.mod h1:7qkJkT+j6b+hIpzMOwPChJhTqS8VbsqqgULzMNRugoM=
github.com/Azure/go-autorest/autorest/date v0.1.0/go.mod h1:plvfp3oPSKwf2DNjlBjWF/7vwR+cUD/ELuzDCXwHUVA=
github.com/Azure/go-autorest/autorest/date v0.2.0/go.mod h1:vcORJHLJEh643/Ioh9+vPmf1Ij9AEBM5FuBIXLmIy0g=
github.com/Azure/go-autorest/autorest/date v0.3.0 h1:7gUk1U5M/CQbp9WoqinNzJar+8KY+LPI6wiWrP/myHw=
Expand Down Expand Up @@ -295,6 +301,9 @@ github.com/dgryski/go-sip13 v0.0.0-20200911182023-62edffca9245/go.mod h1:vAd38F8
github.com/dhui/dktest v0.3.0/go.mod h1:cyzIUfGsBEbZ6BT7tnXqAShHSXCZhSNmFl70sZ7c1yc=
github.com/digitalocean/godo v1.60.0 h1:o/vimtn/HKtYSakFAAZ59Zc5ASORd41S4z1X7pAXPn8=
github.com/digitalocean/godo v1.60.0/go.mod h1:p7dOjjtSBqCTUksqtA5Fd3uaKs9kyTq2xcz76ulEJRU=
github.com/dimchansky/utfbom v1.1.0/go.mod h1:rO41eb7gLfo8SF1jd9F8HplJm1Fewwi4mQvIirEdv+8=
github.com/dimchansky/utfbom v1.1.1 h1:vV6w1AhK4VMnhBno/TPVCoK9U/LP0PkLCS9tbxHdi/U=
github.com/dimchansky/utfbom v1.1.1/go.mod h1:SxdoEBH5qIqFocHMyGOXVAybYJdr71b1Q/j0mACtrfE=
github.com/dnaeon/go-vcr v1.0.1 h1:r8L/HqC0Hje5AXMu1ooW8oyQyOFv4GxqpL0nRP7SLLY=
github.com/dnaeon/go-vcr v1.0.1/go.mod h1:aBB1+wY4s93YsC3HHjMBMrwTj2R9FHDzUr9KyGc8n1E=
github.com/docker/distribution v2.7.0+incompatible/go.mod h1:J2gT2udsDAN96Uj4KfcMRqY0/ypR+oyYUYmja8H+y+w=
Expand Down Expand Up @@ -1401,8 +1410,9 @@ golang.org/x/crypto v0.0.0-20200728195943-123391ffb6de/go.mod h1:LzIPMQfyMNhhGPh
golang.org/x/crypto v0.0.0-20201002170205-7f63de1d35b0/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
golang.org/x/crypto v0.0.0-20201208171446-5f87f3452ae9/go.mod h1:jdWPYTVW3xRLrWPugEBEK3UY2ZEsg3UU495nc5E+M+I=
golang.org/x/crypto v0.0.0-20201221181555-eec23a3978ad/go.mod h1:jdWPYTVW3xRLrWPugEBEK3UY2ZEsg3UU495nc5E+M+I=
golang.org/x/crypto v0.0.0-20210220033148-5ea612d1eb83 h1:/ZScEX8SfEmUGRHs0gxpqteO5nfNW6axyZbBdw9A12g=
golang.org/x/crypto v0.0.0-20210220033148-5ea612d1eb83/go.mod h1:jdWPYTVW3xRLrWPugEBEK3UY2ZEsg3UU495nc5E+M+I=
golang.org/x/crypto v0.0.0-20210513164829-c07d793c2f9a h1:kr2P4QFmQr29mSLA43kwrOcgcReGTfbE9N577tCTuBc=
golang.org/x/crypto v0.0.0-20210513164829-c07d793c2f9a/go.mod h1:P+XmwS30IXTQdn5tA2iutPOUgjI07+tq3H3K9MVA1s8=
golang.org/x/exp v0.0.0-20180321215751-8460e604b9de/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA=
golang.org/x/exp v0.0.0-20180807140117-3d87b88a115f/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA=
golang.org/x/exp v0.0.0-20190121172915-509febef88a4/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA=
Expand Down
136 changes: 116 additions & 20 deletions pkg/objstore/azure/azure.go
Original file line number Diff line number Diff line change
Expand Up @@ -11,11 +11,13 @@ import (
"os"
"strings"
"testing"
"time"

blob "github.com/Azure/azure-storage-blob-go/azblob"
"github.com/go-kit/kit/log"
"github.com/go-kit/kit/log/level"
"github.com/pkg/errors"
"github.com/prometheus/common/model"
"github.com/thanos-io/thanos/pkg/objstore"
yaml "gopkg.in/yaml.v2"
)
Expand All @@ -24,13 +26,64 @@ const (
azureDefaultEndpoint = "blob.core.windows.net"
)

// Set default retry values to default Azure values. 0 = use Default Azure.
var DefaultConfig = Config{
PipelineConfig: PipelineConfig{
MaxTries: 0,
TryTimeout: 0,
RetryDelay: 0,
MaxRetryDelay: 0,
},
ReaderConfig: ReaderConfig{
MaxRetryRequests: 0,
},
HTTPConfig: HTTPConfig{
IdleConnTimeout: model.Duration(90 * time.Second),
ResponseHeaderTimeout: model.Duration(2 * time.Minute),
TLSHandshakeTimeout: model.Duration(10 * time.Second),
ExpectContinueTimeout: model.Duration(1 * time.Second),
MaxIdleConns: 100,
MaxIdleConnsPerHost: 100,
MaxConnsPerHost: 0,
DisableCompression: false,
},
}

// Config Azure storage configuration.
type Config struct {
StorageAccountName string `yaml:"storage_account"`
StorageAccountKey string `yaml:"storage_account_key"`
ContainerName string `yaml:"container"`
Endpoint string `yaml:"endpoint"`
MaxRetries int `yaml:"max_retries"`
StorageAccountName string `yaml:"storage_account"`
StorageAccountKey string `yaml:"storage_account_key"`
ContainerName string `yaml:"container"`
Endpoint string `yaml:"endpoint"`
MaxRetries int `yaml:"max_retries"`
MSIResource string `yaml:"msi_resource"`
PipelineConfig PipelineConfig `yaml:"pipeline_config"`
ReaderConfig ReaderConfig `yaml:"reader_config"`
HTTPConfig HTTPConfig `yaml:"http_config"`
}

type ReaderConfig struct {
MaxRetryRequests int `yaml:"max_retry_requests"`
}

type PipelineConfig struct {
MaxTries int32 `yaml:"max_tries"`
TryTimeout model.Duration `yaml:"try_timeout"`
RetryDelay model.Duration `yaml:"retry_delay"`
MaxRetryDelay model.Duration `yaml:"max_retry_delay"`
}

type HTTPConfig struct {
IdleConnTimeout model.Duration `yaml:"idle_conn_timeout"`
ResponseHeaderTimeout model.Duration `yaml:"response_header_timeout"`
InsecureSkipVerify bool `yaml:"insecure_skip_verify"`

TLSHandshakeTimeout model.Duration `yaml:"tls_handshake_timeout"`
ExpectContinueTimeout model.Duration `yaml:"expect_continue_timeout"`
MaxIdleConns int `yaml:"max_idle_conns"`
MaxIdleConnsPerHost int `yaml:"max_idle_conns_per_host"`
MaxConnsPerHost int `yaml:"max_conns_per_host"`
DisableCompression bool `yaml:"disable_compression"`
}

// Bucket implements the store.Bucket interface against Azure APIs.
Expand All @@ -42,34 +95,77 @@ type Bucket struct {

// Validate checks to see if any of the config options are set.
func (conf *Config) validate() error {
if conf.StorageAccountName == "" ||
conf.StorageAccountKey == "" {
return errors.New("invalid Azure storage configuration")
}
if conf.StorageAccountName == "" && conf.StorageAccountKey != "" {
return errors.New("no Azure storage_account specified while storage_account_key is present in config file; both should be present")
}
if conf.StorageAccountName != "" && conf.StorageAccountKey == "" {
return errors.New("no Azure storage_account_key specified while storage_account is present in config file; both should be present")

var errMsg []string
if conf.MSIResource == "" {
if conf.StorageAccountName == "" ||
conf.StorageAccountKey == "" {
errMsg = append(errMsg, "invalid Azure storage configuration")
}
if conf.StorageAccountName == "" && conf.StorageAccountKey != "" {
errMsg = append(errMsg, "no Azure storage_account specified while storage_account_key is present in config file; both should be present")
}
if conf.StorageAccountName != "" && conf.StorageAccountKey == "" {
errMsg = append(errMsg, "no Azure storage_account_key specified while storage_account is present in config file; both should be present")
}
} else {
if conf.StorageAccountName == "" {
errMsg = append(errMsg, "MSI resource is configured but storage account name is missing")
}
if conf.StorageAccountKey != "" {
errMsg = append(errMsg, "MSI resource is configured but storage account key is used")
}
}

if conf.ContainerName == "" {
return errors.New("no Azure container specified")
errMsg = append(errMsg, "no Azure container specified")
}
if conf.Endpoint == "" {
conf.Endpoint = azureDefaultEndpoint
}
if conf.MaxRetries < 0 {
return errors.New("the value of maxretries must be greater than or equal to 0 in the config file")

if conf.PipelineConfig.MaxTries < 0 {
errMsg = append(errMsg, "The value of max_tries must be greater than or equal to 0 in the config file")
}

if conf.ReaderConfig.MaxRetryRequests < 0 {
errMsg = append(errMsg, "The value of max_retry_requests must be greater than or equal to 0 in the config file")
}

if len(errMsg) > 0 {
return errors.New(strings.Join(errMsg, ", "))
}

return nil
}

// parseConfig unmarshals a buffer into a Config with default values.
func parseConfig(conf []byte) (Config, error) {
config := DefaultConfig
if err := yaml.UnmarshalStrict(conf, &config); err != nil {
return Config{}, err
}

// If we don't have config specific retry values but we do have the generic MaxRetries.
// This is for backwards compatibility but also ease of configuration.
if config.MaxRetries > 0 {
if config.PipelineConfig.MaxTries == 0 {
config.PipelineConfig.MaxTries = int32(config.MaxRetries)
}
if config.ReaderConfig.MaxRetryRequests == 0 {
config.ReaderConfig.MaxRetryRequests = config.MaxRetries
}
}

return config, nil
}

// NewBucket returns a new Bucket using the provided Azure config.
func NewBucket(logger log.Logger, azureConfig []byte, component string) (*Bucket, error) {
level.Debug(logger).Log("msg", "creating new Azure bucket connection", "component", component)

var conf Config
if err := yaml.Unmarshal(azureConfig, &conf); err != nil {
conf, err := parseConfig(azureConfig)
if err != nil {
return nil, err
}

Expand Down Expand Up @@ -227,7 +323,7 @@ func (b *Bucket) getBlobReader(ctx context.Context, name string, offset, length
Parallelism: uint16(3),
Progress: nil,
RetryReaderOptionsPerBlock: blob.RetryReaderOptions{
MaxRetryRequests: b.config.MaxRetries,
MaxRetryRequests: b.config.ReaderConfig.MaxRetryRequests,
},
},
); err != nil {
Expand Down
Loading

0 comments on commit 9355a82

Please sign in to comment.