Skip to content

Commit

Permalink
AzDatalake Support (Azure#21387)
Browse files Browse the repository at this point in the history
* [AzDatalake] Generated Layer + mod file (#20725)

* first generation

* removed pagination

* cleanup spacing and add build file

* doc fix

* Merge main into datalake feature branch (#20833)

* Enable gocritic during linting (#20715)

Enabled gocritic's evalOrder to catch dependencies on undefined behavior
on return statements.
Updated to latest version of golangci-lint.
Fixed issue in azblob flagged by latest linter.

* Cosmos DB: Enable merge support (#20716)

* Adding header and value

* Wiring and tests

* format

* Fixing value

* change log

* [azservicebus, azeventhubs] Stress test and logging improvement (#20710)

Logging improvements:

* Updating the logging to print more tracing information (per-link) in prep for the bigger release coming up.
* Trimming out some of the verbose logging, seeing if I can get it a bit more reasonable.

Stress tests:

* Add a timestamp to the log name we generate and also default to append, not overwrite.
* Use 0.5 cores, 0.5GB as our baseline. Some pods use more and I'll tune them more later.

* update proxy version (#20712)

Co-authored-by: Scott Beddall <[email protected]>

* Return an error when you try to send a message that's too large. (#20721)

This now works just like the message batch - you'll get an ErrMessageTooLarge
if you attempt to send a message that's too large for the link's configured
size.

NOTE: there's a patch to `internal/go-amqp/Sender.go` to match what's in go-amqp's
main so it returns a programmatically useful error when the message is too large.

Fixes #20647

* Changes in test that is failing in pipeline (#20693)

* [azservicebus, azeventhubs] Treat 'entity full' as a fatal error (#20722)

When the remote entity is full we get a resource-limit-exceeded condition. This isn't something we should keep retrying on and it's best to just abort and let the user know immediately, rather than hoping it might eventually clear out.

This affected both Event Hubs and Service Bus.

Fixes #20647

* [azservicebus/azeventhubs] Redirect stderr and stdout to tee (#20726)

* Update changelog with latest features (#20730)

* Update changelog with latest features

Prepare for upcoming release.

* bump minor version

* pass along the artifact name so we can override it later (#20732)

Co-authored-by: scbedd <[email protected]>

* [azeventhubs] Fixing checkpoint store race condition (#20727)

The checkpoint store wasn't guarding against multiple owners claiming for the first time - fixing this by using IfNoneMatch

Fixes #20717

* Fix azidentity troubleshooting guide link (#20736)

* [Release] sdk/resourcemanager/paloaltonetworksngfw/armpanngfw/0.1.0 (#20437)

* [Release] sdk/resourcemanager/paloaltonetworksngfw/armpanngfw/0.1.0 generation from spec commit: 85fb4ac6f8bfefd179e6c2632976a154b5c9ff04

* client factory

* fix

* fix

* update

* add sdk/resourcemanager/postgresql/armpostgresql live test (#20685)

* add sdk/resourcemanager/postgresql/armpostgresql live test

* update assets.json

* set subscriptionId default value

* format

* add sdk/resourcemanager/eventhub/armeventhub live test (#20686)

* add sdk/resourcemanager/eventhub/armeventhub live test

* update assets

* add sdk/resourcemanager/compute/armcompute live test (#20048)

* add sdk/resourcemanager/compute/armcompute live test

* skus filter

* fix subscriptionId default value

* fix

* gofmt

* update recording

* sdk/resourcemanager/network/armnetwork live test (#20331)

* sdk/resourcemanager/network/armnetwork live test

* update subscriptionId default value

* update recording

* add sdk/resourcemanager/cosmos/armcosmos live test (#20705)

* add sdk/resourcemanager/cosmos/armcosmos live test

* update assets.json

* update assets.json

* update assets.json

* update assets.json

* Increment package version after release of azcore (#20740)

* [azeventhubs] Improperly resetting etag in the checkpoint store (#20737)

We shouldn't be resetting the etag to nil - it's what we use to enforce a "single winner" when doing ownership claims.

The bug here was two-fold: I had bad logic in my previous claim ownership, which I fixed in a previous PR, but we need to reflect that same constraint properly in our in-memory checkpoint store for these tests.

* Eng workflows sync and branch cleanup additions (#20743)

Co-authored-by: James Suplizio <[email protected]>

* [azeventhubs] Latest start position can also be inclusive (ie, get the latest message) (#20744)

* Update GitHubEventProcessor version and remove pull_request_review procesing (#20751)

Co-authored-by: James Suplizio <[email protected]>

* Rename DisableAuthorityValidationAndInstanceDiscovery (#20746)

* fix (#20707)

* AzFile (#20739)

* azfile: Fixing connection string parsing logic (#20798)

* Fixing connection string parse logic

* Update README

* [azadmin] fix flaky test (#20758)

* fix flaky test

* charles suggestion

* Prepare azidentity v1.3.0 for release (#20756)

* Fix broken podman link (#20801)

Co-authored-by: Wes Haggard <[email protected]>

* [azquery] update doc comments (#20755)

* update doc comments

* update statistics and visualization generation

* prep-for-release

* Fixed contribution section (#20752)

Co-authored-by: Bob Tabor <[email protected]>

* [azeventhubs,azservicebus] Some API cleanup, renames (#20754)

* Adding options to UpdateCheckpoint(), just for future potential expansion
* Make Offset an int64, not a *int64 (it's not optional, it'll always come back with ReceivedEvents)
* Adding more logging into the checkpoint store.
* Point all imports at the production go-amqp

* Add supporting features to enable distributed tracing (#20301) (#20708)

* Add supporting features to enable distributed tracing

This includes new internal pipeline policies and other supporting types.
See the changelog for a full description.
Added some missing doc comments.

* fix linter issue

* add net.peer.name trace attribute

sequence custom HTTP header policy before logging policy.
sequence logging policy after HTTP trace policy.
keep body download policy at the end.

* add span for iterating over pages

* Restore ARM CAE support for azcore beta (#20657)

This reverts commit 902097226ff3fe2fc6c3e7fc50d3478350253614.

* Upgrade to stable azcore (#20808)

* Increment package version after release of data/azcosmos (#20807)

* Updating changelog (#20810)

* Add fake package to azcore (#20711)

* Add fake package to azcore

This is the supporting infrastructure for the generated SDK fakes.

* fix doc comment

* Updating CHANGELOG.md (#20809)

* changelog (#20811)

* Increment package version after release of storage/azfile (#20813)

* Update changelog (azblob) (#20815)

* Updating CHANGELOG.md

* Update the changelog with correct version

* [azquery] migration guide (#20742)

* migration guide

* Charles feedback

* Richard feedback

---------

Co-authored-by: Charles Lowell <[email protected]>

* Increment package version after release of monitor/azquery (#20820)

* [keyvault] prep for release (#20819)

* prep for release

* perf tests

* update date

---------

Co-authored-by: Joel Hendrix <[email protected]>
Co-authored-by: Matias Quaranta <[email protected]>
Co-authored-by: Richard Park <[email protected]>
Co-authored-by: Azure SDK Bot <[email protected]>
Co-authored-by: Scott Beddall <[email protected]>
Co-authored-by: siminsavani-msft <[email protected]>
Co-authored-by: scbedd <[email protected]>
Co-authored-by: Charles Lowell <[email protected]>
Co-authored-by: Peng Jiahui <[email protected]>
Co-authored-by: James Suplizio <[email protected]>
Co-authored-by: Sourav Gupta <[email protected]>
Co-authored-by: gracewilcox <[email protected]>
Co-authored-by: Wes Haggard <[email protected]>
Co-authored-by: Bob Tabor <[email protected]>
Co-authored-by: Bob Tabor <[email protected]>

* AzDatalake APIView (#20757)

* Enable gocritic during linting (#20715)

Enabled gocritic's evalOrder to catch dependencies on undefined behavior
on return statements.
Updated to latest version of golangci-lint.
Fixed issue in azblob flagged by latest linter.

* Cosmos DB: Enable merge support (#20716)

* Adding header and value

* Wiring and tests

* format

* Fixing value

* change log

* [azservicebus, azeventhubs] Stress test and logging improvement (#20710)

Logging improvements:

* Updating the logging to print more tracing information (per-link) in prep for the bigger release coming up.
* Trimming out some of the verbose logging, seeing if I can get it a bit more reasonable.

Stress tests:

* Add a timestamp to the log name we generate and also default to append, not overwrite.
* Use 0.5 cores, 0.5GB as our baseline. Some pods use more and I'll tune them more later.

* update proxy version (#20712)

Co-authored-by: Scott Beddall <[email protected]>

* Return an error when you try to send a message that's too large. (#20721)

This now works just like the message batch - you'll get an ErrMessageTooLarge
if you attempt to send a message that's too large for the link's configured
size.

NOTE: there's a patch to `internal/go-amqp/Sender.go` to match what's in go-amqp's
main so it returns a programmatically useful error when the message is too large.

Fixes #20647

* Changes in test that is failing in pipeline (#20693)

* [azservicebus, azeventhubs] Treat 'entity full' as a fatal error (#20722)

When the remote entity is full we get a resource-limit-exceeded condition. This isn't something we should keep retrying on and it's best to just abort and let the user know immediately, rather than hoping it might eventually clear out.

This affected both Event Hubs and Service Bus.

Fixes #20647

* [azservicebus/azeventhubs] Redirect stderr and stdout to tee (#20726)

* Update changelog with latest features (#20730)

* Update changelog with latest features

Prepare for upcoming release.

* bump minor version

* pass along the artifact name so we can override it later (#20732)

Co-authored-by: scbedd <[email protected]>

* [azeventhubs] Fixing checkpoint store race condition (#20727)

The checkpoint store wasn't guarding against multiple owners claiming for the first time - fixing this by using IfNoneMatch

Fixes #20717

* Fix azidentity troubleshooting guide link (#20736)

* [Release] sdk/resourcemanager/paloaltonetworksngfw/armpanngfw/0.1.0 (#20437)

* [Release] sdk/resourcemanager/paloaltonetworksngfw/armpanngfw/0.1.0 generation from spec commit: 85fb4ac6f8bfefd179e6c2632976a154b5c9ff04

* client factory

* fix

* fix

* update

* add sdk/resourcemanager/postgresql/armpostgresql live test (#20685)

* add sdk/resourcemanager/postgresql/armpostgresql live test

* update assets.json

* set subscriptionId default value

* format

* add sdk/resourcemanager/eventhub/armeventhub live test (#20686)

* add sdk/resourcemanager/eventhub/armeventhub live test

* update assets

* add sdk/resourcemanager/compute/armcompute live test (#20048)

* add sdk/resourcemanager/compute/armcompute live test

* skus filter

* fix subscriptionId default value

* fix

* gofmt

* update recording

* sdk/resourcemanager/network/armnetwork live test (#20331)

* sdk/resourcemanager/network/armnetwork live test

* update subscriptionId default value

* update recording

* add sdk/resourcemanager/cosmos/armcosmos live test (#20705)

* add sdk/resourcemanager/cosmos/armcosmos live test

* update assets.json

* update assets.json

* update assets.json

* update assets.json

* Increment package version after release of azcore (#20740)

* [azeventhubs] Improperly resetting etag in the checkpoint store (#20737)

We shouldn't be resetting the etag to nil - it's what we use to enforce a "single winner" when doing ownership claims.

The bug here was two-fold: I had bad logic in my previous claim ownership, which I fixed in a previous PR, but we need to reflect that same constraint properly in our in-memory checkpoint store for these tests.

* Eng workflows sync and branch cleanup additions (#20743)

Co-authored-by: James Suplizio <[email protected]>

* [azeventhubs] Latest start position can also be inclusive (ie, get the latest message) (#20744)

* Update GitHubEventProcessor version and remove pull_request_review procesing (#20751)

Co-authored-by: James Suplizio <[email protected]>

* Rename DisableAuthorityValidationAndInstanceDiscovery (#20746)

* added basic fs methods

* corrections

* fix (#20707)

* AzFile (#20739)

* azfile: Fixing connection string parsing logic (#20798)

* Fixing connection string parse logic

* Update README

* [azadmin] fix flaky test (#20758)

* fix flaky test

* charles suggestion

* Prepare azidentity v1.3.0 for release (#20756)

* Fix broken podman link (#20801)

Co-authored-by: Wes Haggard <[email protected]>

* [azquery] update doc comments (#20755)

* update doc comments

* update statistics and visualization generation

* prep-for-release

* Fixed contribution section (#20752)

Co-authored-by: Bob Tabor <[email protected]>

* [azeventhubs,azservicebus] Some API cleanup, renames (#20754)

* Adding options to UpdateCheckpoint(), just for future potential expansion
* Make Offset an int64, not a *int64 (it's not optional, it'll always come back with ReceivedEvents)
* Adding more logging into the checkpoint store.
* Point all imports at the production go-amqp

* Add supporting features to enable distributed tracing (#20301) (#20708)

* Add supporting features to enable distributed tracing

This includes new internal pipeline policies and other supporting types.
See the changelog for a full description.
Added some missing doc comments.

* fix linter issue

* add net.peer.name trace attribute

sequence custom HTTP header policy before logging policy.
sequence logging policy after HTTP trace policy.
keep body download policy at the end.

* add span for iterating over pages

* Restore ARM CAE support for azcore beta (#20657)

This reverts commit 902097226ff3fe2fc6c3e7fc50d3478350253614.

* Upgrade to stable azcore (#20808)

* Increment package version after release of data/azcosmos (#20807)

* Updating changelog (#20810)

* Add fake package to azcore (#20711)

* Add fake package to azcore

This is the supporting infrastructure for the generated SDK fakes.

* fix doc comment

* Updating CHANGELOG.md (#20809)

* changelog (#20811)

* Increment package version after release of storage/azfile (#20813)

* Update changelog (azblob) (#20815)

* Updating CHANGELOG.md

* Update the changelog with correct version

* [azquery] migration guide (#20742)

* migration guide

* Charles feedback

* Richard feedback

---------

Co-authored-by: Charles Lowell <[email protected]>

* Increment package version after release of monitor/azquery (#20820)

* [keyvault] prep for release (#20819)

* prep for release

* perf tests

* update date

* more models

* changed generated code and added more models

* added client files

* regenrated and added more models

* file and dir clients setup

* combined into one package

* added listing methods

* separated clients

* added additional clients

* clean up on path client

* more cleanup

* more models for file client

* cleanup

* regenerated using new autorest

* more models for fs

* cleanup imports

* further cleanup

* lease acc conditions fixes

* lease acc conditions fixes

* rename methods return type

* added lease

* cleanup

* cleanup

* cleanup

* added go version

* feedback and cleanup

* feedback and cleanup

* move path to internal and create common

* moved access helpers to internal

* make access conditions common

* cleanup

* handled feedback

* cleanup

* conn string parsing

---------

Co-authored-by: Joel Hendrix <[email protected]>
Co-authored-by: Matias Quaranta <[email protected]>
Co-authored-by: Richard Park <[email protected]>
Co-authored-by: Azure SDK Bot <[email protected]>
Co-authored-by: Scott Beddall <[email protected]>
Co-authored-by: siminsavani-msft <[email protected]>
Co-authored-by: scbedd <[email protected]>
Co-authored-by: Charles Lowell <[email protected]>
Co-authored-by: Peng Jiahui <[email protected]>
Co-authored-by: James Suplizio <[email protected]>
Co-authored-by: Sourav Gupta <[email protected]>
Co-authored-by: gracewilcox <[email protected]>
Co-authored-by: Wes Haggard <[email protected]>
Co-authored-by: Bob Tabor <[email protected]>
Co-authored-by: Bob Tabor <[email protected]>

* [Datalake] SAS Support (#21019)

* Enable gocritic during linting (#20715)

Enabled gocritic's evalOrder to catch dependencies on undefined behavior
on return statements.
Updated to latest version of golangci-lint.
Fixed issue in azblob flagged by latest linter.

* Cosmos DB: Enable merge support (#20716)

* Adding header and value

* Wiring and tests

* format

* Fixing value

* change log

* [azservicebus, azeventhubs] Stress test and logging improvement (#20710)

Logging improvements:

* Updating the logging to print more tracing information (per-link) in prep for the bigger release coming up.
* Trimming out some of the verbose logging, seeing if I can get it a bit more reasonable.

Stress tests:

* Add a timestamp to the log name we generate and also default to append, not overwrite.
* Use 0.5 cores, 0.5GB as our baseline. Some pods use more and I'll tune them more later.

* update proxy version (#20712)

Co-authored-by: Scott Beddall <[email protected]>

* Return an error when you try to send a message that's too large. (#20721)

This now works just like the message batch - you'll get an ErrMessageTooLarge
if you attempt to send a message that's too large for the link's configured
size.

NOTE: there's a patch to `internal/go-amqp/Sender.go` to match what's in go-amqp's
main so it returns a programmatically useful error when the message is too large.

Fixes #20647

* Changes in test that is failing in pipeline (#20693)

* [azservicebus, azeventhubs] Treat 'entity full' as a fatal error (#20722)

When the remote entity is full we get a resource-limit-exceeded condition. This isn't something we should keep retrying on and it's best to just abort and let the user know immediately, rather than hoping it might eventually clear out.

This affected both Event Hubs and Service Bus.

Fixes #20647

* [azservicebus/azeventhubs] Redirect stderr and stdout to tee (#20726)

* Update changelog with latest features (#20730)

* Update changelog with latest features

Prepare for upcoming release.

* bump minor version

* pass along the artifact name so we can override it later (#20732)

Co-authored-by: scbedd <[email protected]>

* [azeventhubs] Fixing checkpoint store race condition (#20727)

The checkpoint store wasn't guarding against multiple owners claiming for the first time - fixing this by using IfNoneMatch

Fixes #20717

* Fix azidentity troubleshooting guide link (#20736)

* [Release] sdk/resourcemanager/paloaltonetworksngfw/armpanngfw/0.1.0 (#20437)

* [Release] sdk/resourcemanager/paloaltonetworksngfw/armpanngfw/0.1.0 generation from spec commit: 85fb4ac6f8bfefd179e6c2632976a154b5c9ff04

* client factory

* fix

* fix

* update

* add sdk/resourcemanager/postgresql/armpostgresql live test (#20685)

* add sdk/resourcemanager/postgresql/armpostgresql live test

* update assets.json

* set subscriptionId default value

* format

* add sdk/resourcemanager/eventhub/armeventhub live test (#20686)

* add sdk/resourcemanager/eventhub/armeventhub live test

* update assets

* add sdk/resourcemanager/compute/armcompute live test (#20048)

* add sdk/resourcemanager/compute/armcompute live test

* skus filter

* fix subscriptionId default value

* fix

* gofmt

* update recording

* sdk/resourcemanager/network/armnetwork live test (#20331)

* sdk/resourcemanager/network/armnetwork live test

* update subscriptionId default value

* update recording

* add sdk/resourcemanager/cosmos/armcosmos live test (#20705)

* add sdk/resourcemanager/cosmos/armcosmos live test

* update assets.json

* update assets.json

* update assets.json

* update assets.json

* Increment package version after release of azcore (#20740)

* [azeventhubs] Improperly resetting etag in the checkpoint store (#20737)

We shouldn't be resetting the etag to nil - it's what we use to enforce a "single winner" when doing ownership claims.

The bug here was two-fold: I had bad logic in my previous claim ownership, which I fixed in a previous PR, but we need to reflect that same constraint properly in our in-memory checkpoint store for these tests.

* Eng workflows sync and branch cleanup additions (#20743)

Co-authored-by: James Suplizio <[email protected]>

* [azeventhubs] Latest start position can also be inclusive (ie, get the latest message) (#20744)

* Update GitHubEventProcessor version and remove pull_request_review procesing (#20751)

Co-authored-by: James Suplizio <[email protected]>

* Rename DisableAuthorityValidationAndInstanceDiscovery (#20746)

* fix (#20707)

* AzFile (#20739)

* azfile: Fixing connection string parsing logic (#20798)

* Fixing connection string parse logic

* Update README

* [azadmin] fix flaky test (#20758)

* fix flaky test

* charles suggestion

* Prepare azidentity v1.3.0 for release (#20756)

* Fix broken podman link (#20801)

Co-authored-by: Wes Haggard <[email protected]>

* [azquery] update doc comments (#20755)

* update doc comments

* update statistics and visualization generation

* prep-for-release

* Fixed contribution section (#20752)

Co-authored-by: Bob Tabor <[email protected]>

* [azeventhubs,azservicebus] Some API cleanup, renames (#20754)

* Adding options to UpdateCheckpoint(), just for future potential expansion
* Make Offset an int64, not a *int64 (it's not optional, it'll always come back with ReceivedEvents)
* Adding more logging into the checkpoint store.
* Point all imports at the production go-amqp

* Add supporting features to enable distributed tracing (#20301) (#20708)

* Add supporting features to enable distributed tracing

This includes new internal pipeline policies and other supporting types.
See the changelog for a full description.
Added some missing doc comments.

* fix linter issue

* add net.peer.name trace attribute

sequence custom HTTP header policy before logging policy.
sequence logging policy after HTTP trace policy.
keep body download policy at the end.

* add span for iterating over pages

* Restore ARM CAE support for azcore beta (#20657)

This reverts commit 902097226ff3fe2fc6c3e7fc50d3478350253614.

* Upgrade to stable azcore (#20808)

* Increment package version after release of data/azcosmos (#20807)

* Updating changelog (#20810)

* Add fake package to azcore (#20711)

* Add fake package to azcore

This is the supporting infrastructure for the generated SDK fakes.

* fix doc comment

* Updating CHANGELOG.md (#20809)

* changelog (#20811)

* Increment package version after release of storage/azfile (#20813)

* Update changelog (azblob) (#20815)

* Updating CHANGELOG.md

* Update the changelog with correct version

* [azquery] migration guide (#20742)

* migration guide

* Charles feedback

* Richard feedback

---------

Co-authored-by: Charles Lowell <[email protected]>

* Increment package version after release of monitor/azquery (#20820)

* [keyvault] prep for release (#20819)

* prep for release

* perf tests

* update date

* added sas support

* small fix

* query params fix

* fix

* added some tests

* added more tests

* resolved some comments

* added encoding

---------

Co-authored-by: Joel Hendrix <[email protected]>
Co-authored-by: Matias Quaranta <[email protected]>
Co-authored-by: Richard Park <[email protected]>
Co-authored-by: Azure SDK Bot <[email protected]>
Co-authored-by: Scott Beddall <[email protected]>
Co-authored-by: siminsavani-msft <[email protected]>
Co-authored-by: scbedd <[email protected]>
Co-authored-by: Charles Lowell <[email protected]>
Co-authored-by: Peng Jiahui <[email protected]>
Co-authored-by: James Suplizio <[email protected]>
Co-authored-by: Sourav Gupta <[email protected]>
Co-authored-by: gracewilcox <[email protected]>
Co-authored-by: Wes Haggard <[email protected]>
Co-authored-by: Bob Tabor <[email protected]>
Co-authored-by: Bob Tabor <[email protected]>

* [Datalake] Client Constructors (#21034)

* Enable gocritic during linting (#20715)

Enabled gocritic's evalOrder to catch dependencies on undefined behavior
on return statements.
Updated to latest version of golangci-lint.
Fixed issue in azblob flagged by latest linter.

* Cosmos DB: Enable merge support (#20716)

* Adding header and value

* Wiring and tests

* format

* Fixing value

* change log

* [azservicebus, azeventhubs] Stress test and logging improvement (#20710)

Logging improvements:

* Updating the logging to print more tracing information (per-link) in prep for the bigger release coming up.
* Trimming out some of the verbose logging, seeing if I can get it a bit more reasonable.

Stress tests:

* Add a timestamp to the log name we generate and also default to append, not overwrite.
* Use 0.5 cores, 0.5GB as our baseline. Some pods use more and I'll tune them more later.

* update proxy version (#20712)

Co-authored-by: Scott Beddall <[email protected]>

* Return an error when you try to send a message that's too large. (#20721)

This now works just like the message batch - you'll get an ErrMessageTooLarge
if you attempt to send a message that's too large for the link's configured
size.

NOTE: there's a patch to `internal/go-amqp/Sender.go` to match what's in go-amqp's
main so it returns a programmatically useful error when the message is too large.

Fixes #20647

* Changes in test that is failing in pipeline (#20693)

* [azservicebus, azeventhubs] Treat 'entity full' as a fatal error (#20722)

When the remote entity is full we get a resource-limit-exceeded condition. This isn't something we should keep retrying on and it's best to just abort and let the user know immediately, rather than hoping it might eventually clear out.

This affected both Event Hubs and Service Bus.

Fixes #20647

* [azservicebus/azeventhubs] Redirect stderr and stdout to tee (#20726)

* Update changelog with latest features (#20730)

* Update changelog with latest features

Prepare for upcoming release.

* bump minor version

* pass along the artifact name so we can override it later (#20732)

Co-authored-by: scbedd <[email protected]>

* [azeventhubs] Fixing checkpoint store race condition (#20727)

The checkpoint store wasn't guarding against multiple owners claiming for the first time - fixing this by using IfNoneMatch

Fixes #20717

* Fix azidentity troubleshooting guide link (#20736)

* [Release] sdk/resourcemanager/paloaltonetworksngfw/armpanngfw/0.1.0 (#20437)

* [Release] sdk/resourcemanager/paloaltonetworksngfw/armpanngfw/0.1.0 generation from spec commit: 85fb4ac6f8bfefd179e6c2632976a154b5c9ff04

* client factory

* fix

* fix

* update

* add sdk/resourcemanager/postgresql/armpostgresql live test (#20685)

* add sdk/resourcemanager/postgresql/armpostgresql live test

* update assets.json

* set subscriptionId default value

* format

* add sdk/resourcemanager/eventhub/armeventhub live test (#20686)

* add sdk/resourcemanager/eventhub/armeventhub live test

* update assets

* add sdk/resourcemanager/compute/armcompute live test (#20048)

* add sdk/resourcemanager/compute/armcompute live test

* skus filter

* fix subscriptionId default value

* fix

* gofmt

* update recording

* sdk/resourcemanager/network/armnetwork live test (#20331)

* sdk/resourcemanager/network/armnetwork live test

* update subscriptionId default value

* update recording

* add sdk/resourcemanager/cosmos/armcosmos live test (#20705)

* add sdk/resourcemanager/cosmos/armcosmos live test

* update assets.json

* update assets.json

* update assets.json

* update assets.json

* Increment package version after release of azcore (#20740)

* [azeventhubs] Improperly resetting etag in the checkpoint store (#20737)

We shouldn't be resetting the etag to nil - it's what we use to enforce a "single winner" when doing ownership claims.

The bug here was two-fold: I had bad logic in my previous claim ownership, which I fixed in a previous PR, but we need to reflect that same constraint properly in our in-memory checkpoint store for these tests.

* Eng workflows sync and branch cleanup additions (#20743)

Co-authored-by: James Suplizio <[email protected]>

* [azeventhubs] Latest start position can also be inclusive (ie, get the latest message) (#20744)

* Update GitHubEventProcessor version and remove pull_request_review procesing (#20751)

Co-authored-by: James Suplizio <[email protected]>

* Rename DisableAuthorityValidationAndInstanceDiscovery (#20746)

* fix (#20707)

* AzFile (#20739)

* azfile: Fixing connection string parsing logic (#20798)

* Fixing connection string parse logic

* Update README

* [azadmin] fix flaky test (#20758)

* fix flaky test

* charles suggestion

* Prepare azidentity v1.3.0 for release (#20756)

* Fix broken podman link (#20801)

Co-authored-by: Wes Haggard <[email protected]>

* [azquery] update doc comments (#20755)

* update doc comments

* update statistics and visualization generation

* prep-for-release

* Fixed contribution section (#20752)

Co-authored-by: Bob Tabor <[email protected]>

* [azeventhubs,azservicebus] Some API cleanup, renames (#20754)

* Adding options to UpdateCheckpoint(), just for future potential expansion
* Make Offset an int64, not a *int64 (it's not optional, it'll always come back with ReceivedEvents)
* Adding more logging into the checkpoint store.
* Point all imports at the production go-amqp

* Add supporting features to enable distributed tracing (#20301) (#20708)

* Add supporting features to enable distributed tracing

This includes new internal pipeline policies and other supporting types.
See the changelog for a full description.
Added some missing doc comments.

* fix linter issue

* add net.peer.name trace attribute

sequence custom HTTP header policy before logging policy.
sequence logging policy after HTTP trace policy.
keep body download policy at the end.

* add span for iterating over pages

* Restore ARM CAE support for azcore beta (#20657)

This reverts commit 902097226ff3fe2fc6c3e7fc50d3478350253614.

* Upgrade to stable azcore (#20808)

* Increment package version after release of data/azcosmos (#20807)

* Updating changelog (#20810)

* Add fake package to azcore (#20711)

* Add fake package to azcore

This is the supporting infrastructure for the generated SDK fakes.

* fix doc comment

* Updating CHANGELOG.md (#20809)

* changelog (#20811)

* Increment package version after release of storage/azfile (#20813)

* Update changelog (azblob) (#20815)

* Updating CHANGELOG.md

* Update the changelog with correct version

* [azquery] migration guide (#20742)

* migration guide

* Charles feedback

* Richard feedback

---------

Co-authored-by: Charles Lowell <[email protected]>

* Increment package version after release of monitor/azquery (#20820)

* [keyvault] prep for release (#20819)

* prep for release

* perf tests

* update date

* added sas support

* small fix

* query params fix

* fix

* added some tests

* added more tests

* resolved some comments

* added encoding

* added constructors

* cleanup

* cleanup

* gmt

* added all necessary builders

* merge branch

* removed path client

---------

Co-authored-by: Joel Hendrix <[email protected]>
Co-authored-by: Matias Quaranta <[email protected]>
Co-authored-by: Richard Park <[email protected]>
Co-authored-by: Azure SDK Bot <[email protected]>
Co-authored-by: Scott Beddall <[email protected]>
Co-authored-by: siminsavani-msft <[email protected]>
Co-authored-by: scbedd <[email protected]>
Co-authored-by: Charles Lowell <[email protected]>
Co-authored-by: Peng Jiahui <[email protected]>
Co-authored-by: James Suplizio <[email protected]>
Co-authored-by: Sourav Gupta <[email protected]>
Co-authored-by: gracewilcox <[email protected]>
Co-authored-by: Wes Haggard <[email protected]>
Co-authored-by: Bob Tabor <[email protected]>
Co-authored-by: Bob Tabor <[email protected]>

* [AzDatalake] Filesystem Client Implementation + Test Framework (#21067)

* Enable gocritic during linting (#20715)

Enabled gocritic's evalOrder to catch dependencies on undefined behavior
on return statements.
Updated to latest version of golangci-lint.
Fixed issue in azblob flagged by latest linter.

* Cosmos DB: Enable merge support (#20716)

* Adding header and value

* Wiring and tests

* format

* Fixing value

* change log

* [azservicebus, azeventhubs] Stress test and logging improvement (#20710)

Logging improvements:

* Updating the logging to print more tracing information (per-link) in prep for the bigger release coming up.
* Trimming out some of the verbose logging, seeing if I can get it a bit more reasonable.

Stress tests:

* Add a timestamp to the log name we generate and also default to append, not overwrite.
* Use 0.5 cores, 0.5GB as our baseline. Some pods use more and I'll tune them more later.

* update proxy version (#20712)

Co-authored-by: Scott Beddall <[email protected]>

* Return an error when you try to send a message that's too large. (#20721)

This now works just like the message batch - you'll get an ErrMessageTooLarge
if you attempt to send a message that's too large for the link's configured
size.

NOTE: there's a patch to `internal/go-amqp/Sender.go` to match what's in go-amqp's
main so it returns a programmatically useful error when the message is too large.

Fixes #20647

* Changes in test that is failing in pipeline (#20693)

* [azservicebus, azeventhubs] Treat 'entity full' as a fatal error (#20722)

When the remote entity is full we get a resource-limit-exceeded condition. This isn't something we should keep retrying on and it's best to just abort and let the user know immediately, rather than hoping it might eventually clear out.

This affected both Event Hubs and Service Bus.

Fixes #20647

* [azservicebus/azeventhubs] Redirect stderr and stdout to tee (#20726)

* Update changelog with latest features (#20730)

* Update changelog with latest features

Prepare for upcoming release.

* bump minor version

* pass along the artifact name so we can override it later (#20732)

Co-authored-by: scbedd <[email protected]>

* [azeventhubs] Fixing checkpoint store race condition (#20727)

The checkpoint store wasn't guarding against multiple owners claiming for the first time - fixing this by using IfNoneMatch

Fixes #20717

* Fix azidentity troubleshooting guide link (#20736)

* [Release] sdk/resourcemanager/paloaltonetworksngfw/armpanngfw/0.1.0 (#20437)

* [Release] sdk/resourcemanager/paloaltonetworksngfw/armpanngfw/0.1.0 generation from spec commit: 85fb4ac6f8bfefd179e6c2632976a154b5c9ff04

* client factory

* fix

* fix

* update

* add sdk/resourcemanager/postgresql/armpostgresql live test (#20685)

* add sdk/resourcemanager/postgresql/armpostgresql live test

* update assets.json

* set subscriptionId default value

* format

* add sdk/resourcemanager/eventhub/armeventhub live test (#20686)

* add sdk/resourcemanager/eventhub/armeventhub live test

* update assets

* add sdk/resourcemanager/compute/armcompute live test (#20048)

* add sdk/resourcemanager/compute/armcompute live test

* skus filter

* fix subscriptionId default value

* fix

* gofmt

* update recording

* sdk/resourcemanager/network/armnetwork live test (#20331)

* sdk/resourcemanager/network/armnetwork live test

* update subscriptionId default value

* update recording

* add sdk/resourcemanager/cosmos/armcosmos live test (#20705)

* add sdk/resourcemanager/cosmos/armcosmos live test

* update assets.json

* update assets.json

* update assets.json

* update assets.json

* Increment package version after release of azcore (#20740)

* [azeventhubs] Improperly resetting etag in the checkpoint store (#20737)

We shouldn't be resetting the etag to nil - it's what we use to enforce a "single winner" when doing ownership claims.

The bug here was two-fold: I had bad logic in my previous claim ownership, which I fixed in a previous PR, but we need to reflect that same constraint properly in our in-memory checkpoint store for these tests.

* Eng workflows sync and branch cleanup additions (#20743)

Co-authored-by: James Suplizio <[email protected]>

* [azeventhubs] Latest start position can also be inclusive (ie, get the latest message) (#20744)

* Update GitHubEventProcessor version and remove pull_request_review procesing (#20751)

Co-authored-by: James Suplizio <[email protected]>

* Rename DisableAuthorityValidationAndInstanceDiscovery (#20746)

* fix (#20707)

* AzFile (#20739)

* azfile: Fixing connection string parsing logic (#20798)

* Fixing connection string parse logic

* Update README

* [azadmin] fix flaky test (#20758)

* fix flaky test

* charles suggestion

* Prepare azidentity v1.3.0 for release (#20756)

* Fix broken podman link (#20801)

Co-authored-by: Wes Haggard <[email protected]>

* [azquery] update doc comments (#20755)

* update doc comments

* update statistics and visualization generation

* prep-for-release

* Fixed contribution section (#20752)

Co-authored-by: Bob Tabor <[email protected]>

* [azeventhubs,azservicebus] Some API cleanup, renames (#20754)

* Adding options to UpdateCheckpoint(), just for future potential expansion
* Make Offset an int64, not a *int64 (it's not optional, it'll always come back with ReceivedEvents)
* Adding more logging into the checkpoint store.
* Point all imports at the production go-amqp

* Add supporting features to enable distributed tracing (#20301) (#20708)

* Add supporting features to enable distributed tracing

This includes new internal pipeline policies and other supporting types.
See the changelog for a full description.
Added some missing doc comments.

* fix linter issue

* add net.peer.name trace attribute

sequence custom HTTP header policy before logging policy.
sequence logging policy after HTTP trace policy.
keep body download policy at the end.

* add span for iterating over pages

* Restore ARM CAE support for azcore beta (#20657)

This reverts commit 902097226ff3fe2fc6c3e7fc50d3478350253614.

* Upgrade to stable azcore (#20808)

* Increment package version after release of data/azcosmos (#20807)

* Updating changelog (#20810)

* Add fake package to azcore (#20711)

* Add fake package to azcore

This is the supporting infrastructure for the generated SDK fakes.

* fix doc comment

* Updating CHANGELOG.md (#20809)

* changelog (#20811)

* Increment package version after release of storage/azfile (#20813)

* Update changelog (azblob) (#20815)

* Updating CHANGELOG.md

* Update the changelog with correct version

* [azquery] migration guide (#20742)

* migration guide

* Charles feedback

* Richard feedback

---------

Co-authored-by: Charles Lowell <[email protected]>

* Increment package version after release of monitor/azquery (#20820)

* [keyvault] prep for release (#20819)

* prep for release

* perf tests

* update date

* added new client methods

* moved access conditions and wrote first test

* added more tests

* added path listing

* list deleted paths

* fixed formatter and added more tests

* added more tests

* marker fix

* log

* fixed constructor validation

* comment

* assets update:

---------

Co-authored-by: Joel Hendrix <[email protected]>
Co-authored-by: Matias Quaranta <[email protected]>
Co-authored-by: Richard Park <[email protected]>
Co-authored-by: Azure SDK Bot <[email protected]>
Co-authored-by: Scott Beddall <[email protected]>
Co-authored-by: siminsavani-msft <[email protected]>
Co-authored-by: scbedd <[email protected]>
Co-authored-by: Charles Lowell <[email protected]>
Co-authored-by: Peng Jiahui <[email protected]>
Co-authored-by: James Suplizio <[email protected]>
Co-authored-by: Sourav Gupta <[email protected]>
Co-authored-by: gracewilcox <[email protected]>
Co-authored-by: Wes Haggard <[email protected]>
Co-authored-by: Bob Tabor <[email protected]>
Co-authored-by: Bob Tabor <[email protected]>

* Feedback + Service Client (#21096)

* Enable gocritic during linting (#20715)

Enabled gocritic's evalOrder to catch dependencies on undefined behavior
on return statements.
Updated to latest version of golangci-lint.
Fixed issue in azblob flagged by latest linter.

* Cosmos DB: Enable merge support (#20716)

* Adding header and value

* Wiring and tests

* format

* Fixing value

* change log

* [azservicebus, azeventhubs] Stress test and logging improvement (#20710)

Logging improvements:

* Updating the logging to print more tracing information (per-link) in prep for the bigger release coming up.
* Trimming out some of the verbose logging, seeing if I can get it a bit more reasonable.

Stress tests:

* Add a timestamp to the log name we generate and also default to append, not overwrite.
* Use 0.5 cores, 0.5GB as our baseline. Some pods use more and I'll tune them more later.

* update proxy version (#20712)

Co-authored-by: Scott Beddall <[email protected]>

* Return an error when you try to send a message that's too large. (#20721)

This now works just like the message batch - you'll get an ErrMessageTooLarge
if you attempt to send a message that's too large for the link's configured
size.

NOTE: there's a patch to `internal/go-amqp/Sender.go` to match what's in go-amqp's
main so it returns a programmatically useful error when the message is too large.

Fixes #20647

* Changes in test that is failing in pipeline (#20693)

* [azservicebus, azeventhubs] Treat 'entity full' as a fatal error (#20722)

When the remote entity is full we get a resource-limit-exceeded condition. This isn't something we should keep retrying on and it's best to just abort and let the user know immediately, rather than hoping it might eventually clear out.

This affected both Event Hubs and Service Bus.

Fixes #20647

* [azservicebus/azeventhubs] Redirect stderr and stdout to tee (#20726)

* Update changelog with latest features (#20730)

* Update changelog with latest features

Prepare for upcoming release.

* bump minor version

* pass along the artifact name so we can override it later (#20732)

Co-authored-by: scbedd <[email protected]>

* [azeventhubs] Fixing checkpoint store race condition (#20727)

The checkpoint store wasn't guarding against multiple owners claiming for the first time - fixing this by using IfNoneMatch

Fixes #20717

* Fix azidentity troubleshooting guide link (#20736)

* [Release] sdk/resourcemanager/paloaltonetworksngfw/armpanngfw/0.1.0 (#20437)

* [Release] sdk/resourcemanager/paloaltonetworksngfw/armpanngfw/0.1.0 generation from spec commit: 85fb4ac6f8bfefd179e6c2632976a154b5c9ff04

* client factory

* fix

* fix

* update

* add sdk/resourcemanager/postgresql/armpostgresql live test (#20685)

* add sdk/resourcemanager/postgresql/armpostgresql live test

* update assets.json

* set subscriptionId default value

* format

* add sdk/resourcemanager/eventhub/armeventhub live test (#20686)

* add sdk/resourcemanager/eventhub/armeventhub live test

* update assets

* add sdk/resourcemanager/compute/armcompute live test (#20048)

* add sdk/resourcemanager/compute/armcompute live test

* skus filter

* fix subscriptionId default value

* fix

* gofmt

* update recording

* sdk/resourcemanager/network/armnetwork live test (#20331)

* sdk/resourcemanager/network/armnetwork live test

* update subscriptionId default value

* update recording

* add sdk/resourcemanager/cosmos/armcosmos live test (#20705)

* add sdk/resourcemanager/cosmos/armcosmos live test

* update assets.json

* update assets.json

* update assets.json

* update assets.json

* Increment package version after release of azcore (#20740)

* [azeventhubs] Improperly resetting etag in the checkpoint store (#20737)

We shouldn't be resetting the etag to nil - it's what we use to enforce a "single winner" when doing ownership claims.

The bug here was two-fold: I had bad logic in my previous claim ownership, which I fixed in a previous PR, but we need to reflect that same constraint properly in our in-memory checkpoint store for these tests.

* Eng workflows sync and branch cleanup additions (#20743)

Co-authored-by: James Suplizio <[email protected]>

* [azeventhubs] Latest start position can also be inclusive (ie, get the latest message) (#20744)

* Update GitHubEventProcessor version and remove pull_request_review procesing (#20751)

Co-authored-by: James Suplizio <[email protected]>

* Rename DisableAuthorityValidationAndInstanceDiscovery (#20746)

* fix (#20707)

* AzFile (#20739)

* azfile: Fixing connection string parsing logic (#20798)

* Fixing connection string parse logic

* Update README

* [azadmin] fix flaky test (#20758)

* fix flaky test

* charles suggestion

* Prepare azidentity v1.3.0 for release (#20756)

* Fix broken podman link (#20801)

Co-authored-by: Wes Haggard <[email protected]>

* [azquery] update doc comments (#20755)

* update doc comments

* update statistics and visualization generation

* prep-for-release

* Fixed contribution section (#20752)

Co-authored-by: Bob Tabor <[email protected]>

* [azeventhubs,azservicebus] Some API cleanup, renames (#20754)

* Adding options to UpdateCheckpoint(), just for future potential expansion
* Make Offset an int64, not a *int64 (it's not optional, it'll always come back with ReceivedEvents)
* Adding more logging into the checkpoint store.
* Point all imports at the production go-amqp

* Add supporting features to enable distributed tracing (#20301) (#20708)

* Add supporting features to enable distributed tracing

This includes new internal pipeline policies and other supporting types.
See the changelog for a full description.
Added some missing doc comments.

* fix linter issue

* add net.peer.name trace attribute

sequence custom HTTP header policy before logging policy.
sequence logging policy after HTTP trace policy.
keep body download policy at the end.

* add span for iterating over pages

* Restore ARM CAE support for azcore beta (#20657)

This reverts commit 902097226ff3fe2fc6c3e7fc50d3478350253614.

* Upgrade to stable azcore (#20808)

* Increment package version after release of data/azcosmos (#20807)

* Updating changelog (#20810)

* Add fake package to azcore (#20711)

* Add fake package to azcore

This is the supporting infrastructure for the generated SDK fakes.

* fix doc comment

* Updating CHANGELOG.md (#20809)

* changelog (#20811)

* Increment package version after release of storage/azfile (#20813)

* Update changelog (azblob) (#20815)

* Updating CHANGELOG.md

* Update the changelog with correct version

* [azquery] migration guide (#20742)

* migration guide

* Charles feedback

* Richard feedback

---------

Co-authored-by: Charles Lowell <[email protected]>

* Increment package version after release of monitor/azquery (#20820)

* [keyvault] prep for release (#20819)

* prep for release

* perf tests

* update date

* added new client methods

* moved access conditions and wrote first test

* added more tests

* added path listing

* list deleted paths

* fixed formatter and added more tests

* added more tests

* marker fix

* log

* fixed constructor validation

* comment

* assets update:

* some cleanup

* missing models and log

* cleanup, resolved apiview feedback

* test

* cleanup

* added service tests

* implemented listing

* cleanup

* cleanup

---------

Co-authored-by: Joel Hendrix <[email protected]>
Co-authored-by: Matias Quaranta <[email protected]>
Co-authored-by: Richard Park <[email protected]>
Co-authored-by: Azure SDK Bot <[email protected]>
Co-authored-by: Scott Beddall <[email protected]>
Co-authored-by: siminsavani-msft <[email protected]>
Co-authored-by: scbedd <[email protected]>
Co-authored-by: Charles Lowell <[email protected]>
Co-authored-by: Peng Jiahui <[email protected]>
Co-authored-by: James Suplizio <[email protected]>
Co-authored-by: Sourav Gupta <[email protected]>
Co-authored-by: gracewilcox <[email protected]>
Co-authored-by: Wes Haggard <[email protected]>
Co-authored-by: Bob Tabor <[email protected]>
Co-authored-by: Bob Tabor <[email protected]>

* [AzDatalake] File Client Support (#21141)

* Enable gocritic during linting (#20715)

Enabled gocritic's evalOrder to catch dependencies on undefined behavior
on return statements.
Updated to latest version of golangci-lint.
Fixed issue in azblob flagged by latest linter.

* Cosmos DB: Enable merge support (#20716)

* Adding header and value

* Wiring and tests

* format

* Fixing value

* change log

* [azservicebus, azeventhubs] Stress test and logging improvement (#20710)

Logging improvements:

* Updating the logging to print more tracing information (per-link) in prep for the bigger release coming up.
* Trimming out some of the verbose logging, seeing if I can get it a bit more reasonable.

Stress tests:

* Add a timestamp to the log name we generate and also default to append, not overwrite.
* Use 0.5 cores, 0.5GB as our baseline. Some pods use more and I'll tune them more later.

* update proxy version (#20712)

Co-authored-by: Scott Beddall <[email protected]>

* Return an error when you try to send a message that's too large. (#20721)

This now works just like the message batch - you'll get an ErrMessageTooLarge
if you attempt to send a message that's too large for the link's configured
size.

NOTE: there's a patch to `internal/go-amqp/Sender.go` to match what's in go-amqp's
main so it returns a programmatically useful error when the message is too large.

Fixes #20647

* Changes in test that is failing in pipeline (#20693)

* [azservicebus, azeventhubs] Treat 'entity full' as a fatal error (#20722)

When the remote entity is full we get a resource-limit-exceeded condition. This isn't something we should keep retrying on and it's best to just abort and let the user know immediately, rather than hoping it might eventually clear out.

This affected both Event Hubs and Service Bus.

Fixes #20647

* [azservicebus/azeventhubs] Redirect stderr and stdout to tee (#20726)

* Update changelog with latest features (#20730)

* Update changelog with latest features

Prepare for upcoming release.

* bump minor version

* pass along the artifact name so we can override it later (#20732)

Co-authored-by: scbedd <[email protected]>

* [azeventhubs] Fixing checkpoint store race condition (#20727)

The checkpoint store wasn't guarding against multiple owners claiming for the first time - fixing this by using IfNoneMatch

Fixes #20717

* Fix azidentity troubleshooting guide link (#20736)

* [Release] sdk/resourcemanager/paloaltonetworksngfw/armpanngfw/0.1.0 (#20437)

* [Release] sdk/resourcemanager/paloaltonetworksngfw/armpanngfw/0.1.0 generation from spec commit: 85fb4ac6f8bfefd179e6c2632976a154b5c9ff04

* client factory

* fix

* fix

* update

* add sdk/resourcemanager/postgresql/armpostgresql live test (#20685)

* add sdk/resourcemanager/postgresql/armpostgresql live test

* update assets.json

* set subscriptionId default value

* format

* add sdk/resourcemanager/eventhub/armeventhub live test (#20686)

* add sdk/resourcemanager/eventhub/armeventhub live test

* update assets

* add sdk/resourcemanager/compute/armcompute live test (#20048)

* add sdk/resourcemanager/compute/armcompute live test

* skus filter

* fix subscriptionId default value

* fix

* gofmt

* update recording

* sdk/resourcemanager/network/armnetwork live test (#20331)

* sdk/resourcemanager/network/armnetwork live test

* update subscriptionId default value

* update recording

* add sdk/resourcemanager/cosmos/armcosmos live test (#20705)

* add sdk/resourcemanager/cosmos/armcosmos live test

* update assets.json

* update assets.json

* update assets.json

* update assets.json

* Increment package version after release of azcore (#20740)

* [azeventhubs] Improperly resetting etag in the checkpoint store (#20737)

We shouldn't be resetting the etag to nil - it's what we use to enforce a "single winner" when doing ownership claims.

The bug here was two-fold: I had bad logic in my previous claim ownership, which I fixed in a previous PR, but we need to reflect that same constraint properly in our in-memory checkpoint store for these tests.

* Eng workflows sync and branch cleanup additions (#20743)

Co-authored-by: James Suplizio <[email protected]>

* [azeventhubs] Latest start position can also be inclusive (ie, get the latest message) (#20744)

* Update GitHubEventProcessor version and remove pull_request_review procesing (#20751)

Co-authored-by: James Suplizio <[email protected]>

* Rename DisableAuthorityValidationAndInstanceDiscovery (#20746)

* fix (#20707)

* AzFile (#20739)

* azfile: Fixing connection string parsing logic (#20798)

* Fixing connection string parse logic

* Update README

* [azadmin] fix flaky test (#20758)

* fix flaky test

* charles suggestion

* Prepare azidentity v1.3.0 for release (#20756)

* Fix broken podman link (#20801)

Co-authored-by: Wes Haggard <[email protected]>

* [azquery] update doc comments (#20755)

* update doc comments

* update statistics and visualization generation

* prep-for-release

* Fixed contribution section (#20752)

Co-authored-by: Bob Tabor <[email protected]>

* [azeventhubs,azservicebus] Some API cleanup, renames (#20754)

* Adding options to UpdateCheckpoint(), just for future potential expansion
* Make Offset an int64, not a *int64 (it's not optional, it'll always come back with ReceivedEvents)
* Adding more logging into the checkpoint store.
* Point all imports at the production go-amqp

* Add supporting features to enable distributed tracing (#20301) (#20708)

* Add supporting features to enable distributed tracing

This includes new internal pipeline policies and other supporting types.
See the changelog for a full description.
Added some missing doc comments.

* fix linter issue

* add net.peer.name trace attribute

sequence custom HTTP header policy before logging policy.
sequence logging policy after HTTP trace policy.
keep body download policy at the end.

* add span for iterating over pages

* Restore ARM CAE support for azcore beta (#20657)

This reverts commit 902097226ff3fe2fc6c3e7fc50d3478350253614.

* Upgrade to stable azcore (#20808)

* Increment package version after release of data/azcosmos (#20807)

* Updating changelog (#20810)

* Add fake package to azcore (#20711)

* Add fake package to azcore

This is the supporting infrastructure for the generated SDK fakes.

* fix doc comment

* Updating CHANGELOG.md (#20809)

* changelog (#20811)

* Increment package version after release of storage/azfile (#20813)

* Update changelog (azblob) (#20815)

* Updating CHANGELOG.md

* Update the changelog with correct version

* [azquery] migration guide (#20742)

* migration guide

* Charles feedback

* Richard feedback

---------

Co-authored-by: Charles Lowell <[email protected]>

* Increment package version after release of monitor/azquery (#20820)

* [keyvault] prep for release (#20819)

* prep for release

* perf tests

* update date

* added apis and models

* cleanup

* cleanup + block blob client

* added tests

* added tests

* added tests

* added more tests

* added more tests

* dir block client

* sas test

* sas test

---------

Co-authored-by: Joel Hendrix <[email protected]>
Co-authored-by: Matias Quaranta <[email protected]>
Co-authored-by: Richard Park <[email protected]>
Co-authored-by: Azure SDK Bot <[email protected]>
Co-authored-by: Scott Beddall <[email protected]>
Co-authored-by: siminsavani-msft <[email protected]>
Co-authored-by: scbedd <[email protected]>
Co-authored-by: Charles Lowell <[email protected]>
Co-authored-by: Peng Jiahui <[email protected]>
Co-authored-by: James Suplizio <[email protected]>
Co-authored-by: Sourav Gupta <[email protected]>
Co-authored-by: gracewilcox <[email protected]>
Co-authored-by: Wes Haggard <[email protected]>
Co-authored-by: Bob Tabor <[email protected]>
Co-authored-by: Bob Tabor <[email protected]>

* [AzDatalake] Cleanup + Improvements (#21222)

* Enable gocritic during linting (#20715)

Enabled gocritic's evalOrder to catch dependencies on undefined behavior
on return statements.
Updated to latest version of golangci-lint.
Fixed issue in azblob flagged by latest linter.

* Cosmos DB: Enable merge support (#20716)

* Adding header and value

* Wiring and tests

* format

* Fixing value

* change log

* [azservicebus, azeventhubs] Stress test and logging improvement (#20710)

Logging improvements:

* Updating the logging to print more tracing information (per-link) in prep for the bigger release coming up.
* Trimming out some of the verbose logging, seeing if I can get it a bit more reasonable.

Stress tests:

* Add a timestamp to the log name we generate and also default to append, not overwrite.
* Use 0.5 cores, 0.5GB as our baseline. Some pods use more and I'll tune them more later.

* update proxy version (#20712)

Co-authored-by: Scott Beddall <[email protected]>

* Return an error when you try to send a message that's too large. (#20721)

This now works just like the message batch - you'll get an ErrMessageTooLarge
if you attempt to send a message that's too large for the link's configured
size.

NOTE: there's a patch to `internal/go-amqp/Sender.go` to match what's in go-amqp's
main so it returns a programmatically useful error when the message is too large.

Fixes #20647

* Changes in test that is failing in pipeline (#20693)

* [azservicebus, azeventhubs] Treat 'entity full' as a fatal error (#20722)

When the remote entity is full we get a resource-limit-exceeded condition. This isn't something we should keep retrying on and it's best to just abort and let the user know immediately, rather than hoping it might eventually clear out.

This affected both Event Hubs and Service Bus.

Fixes #20647

* [azservicebus/azeventhubs] Redirect stderr and stdout to tee (#20726)

* Update changelog with latest features (#20730)

* Update changelog with latest features

Prepare for upcoming release.

* bump minor version

* pass along the artifact name so we can override it later (#20732)

Co-authored-by: scbedd <[email protected]>

* [azeventhubs] Fixing checkpoint store race condition (#20727)

The checkpoint store wasn't guarding against multiple owners claiming for the first time - fixing this by using IfNoneMatch

Fixes #20717

* Fix azidentity troubleshooting guide link (#20736)

* [Release] sdk/resourcemanager/paloaltonetworksngfw/armpanngfw/0.1.0 (#20437)

* [Release] sdk/resourcemanager/paloaltonetworksngfw/armpanngfw/0.1.0 generation from spec commit: 85fb4ac6f8bfefd179e6c2632976a154b5c9ff04

* client factory

* fix

* fix

* update

* add sdk/resourcemanager/postgresql/armpostgresql live test (#20685)

* add sdk/resourcemanager/postgresql/armpostgresql live test

* update assets.json

* set subscriptionId default value

* format

* add sdk/resourcemanager/eventhub/armeventhub live test (#20686)

* add sdk/resourcemanager/eventhub/armeventhub live test

* update assets

* add sdk/resourcemanager/compute/armcompute live test (#20048)

* add sdk/resourcemanager/compute/armcompute live test

* skus filter

* fix subscriptionId default value

* fix

* gofmt

* update recording

* sdk/resourcemanager/network/armnetwork live test (#20331)

* sdk/resourcemanager/network/armnetwork live test

* update subscriptionId default value

* update recording

* add sdk/resourcemanager/cosmos/armcosmos live test (#20705)

* add sdk/resourcemanager/cosmos/armcosmos live test

* update assets.json

* update assets.json

* update assets.json

* update assets.json

* Increment package version after release of azcore (#20740)

* [azeventhubs] Improperly resetting etag in the checkpoint store (#20737)

We shouldn't be resetting the etag to nil - it's what we use to enforce a "single winner" when doing ownership claims.

The bug here was two-fold: I had bad logic in my previous claim ownership, which I fixed in a previous PR, but we need to reflect that same constraint properly in our in-memory checkpoint store for these tests.

* Eng workflows sync and branch cleanup additions (#20743)

Co-authored-by: James Suplizio <[email protected]>

* [azeventhubs] Latest start position can also be inclusive (ie, get the latest message) (#20744)

* Update GitHubEventProcessor version and remove pull_request_review procesing (#20751)

Co-authored-by: James Suplizio <[email protected]>

* Rename DisableAuthorityValidationAndInstanceDiscovery (#20746)

* fix (#20707)

* AzFile (#20739)

* azfile: Fixing connection string parsing logic (#2079…
  • Loading branch information
Show file tree
Hide file tree
Showing 107 changed files with 39,385 additions and 0 deletions.
4 changes: 4 additions & 0 deletions eng/config.json
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,10 @@
"Name": "azqueue",
"CoverageGoal": 0.60
},
{
"Name": "azdatalake",
"CoverageGoal": 0.60
},
{
"Name": "azfile",
"CoverageGoal": 0.75
Expand Down
7 changes: 7 additions & 0 deletions sdk/storage/azdatalake/CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
# Release History

## 0.1.0-beta.1 (2023-08-16)

### Features Added

* This is the initial preview release of the `azdatalake` library
21 changes: 21 additions & 0 deletions sdk/storage/azdatalake/LICENSE.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
MIT License

Copyright (c) Microsoft Corporation. All rights reserved.

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE
280 changes: 280 additions & 0 deletions sdk/storage/azdatalake/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,280 @@
# ADLS Gen2 Storage SDK for Go

> Service Version: 2020-10-02
Azure Data Lake Storage Gen2 (ADLS Gen2) is Microsoft's hierarchical object storage solution for the cloud with converged capabilities with Azure Blob Storage.
For example, Data Lake Storage Gen2 provides file system semantics, file-level security, and scale.
Because these capabilities are built on Blob storage, you also get low-cost, tiered storage, with high availability/disaster recovery capabilities.
ADLS Gen2 makes Azure Storage the foundation for building enterprise data lakes on Azure.
Designed from the start to service multiple petabytes of information while sustaining hundreds of gigabits of throughput, ADLS Gen2 allows you to easily manage massive amounts of data.

[Source code][source] | [API reference documentation][docs] | [REST API documentation][rest_docs]

## Getting started

### Install the package

Install the ADLS Gen2 Storage SDK for Go with [go get][goget]:

```Powershell
go get github.com/Azure/azure-sdk-for-go/sdk/storage/azdatalake
```

If you're going to authenticate with Azure Active Directory (recommended), install the [azidentity][azidentity] module.
```Powershell
go get github.com/Azure/azure-sdk-for-go/sdk/azidentity
```

### Prerequisites

A supported [Go][godevdl] version (the Azure SDK supports the two most recent Go releases).

You need an [Azure subscription][azure_sub] and a
[Storage Account][storage_account_docs] to use this package.

To create a new Storage Account, you can use the [Azure Portal][storage_account_create_portal],
[Azure PowerShell][storage_account_create_ps], or the [Azure CLI][storage_account_create_cli].
Here's an example using the Azure CLI:

```Powershell
az storage account create --name MyStorageAccount --resource-group MyResourceGroup --location westus --sku Standard_LRS
```

### Authenticate the client

In order to interact with the ADLS Gen2 Storage service, you'll need to create an instance of the `Client` type. The [azidentity][azidentity] module makes it easy to add Azure Active Directory support for authenticating Azure SDK clients with their corresponding Azure services.

```go
// create a credential for authenticating with Azure Active Directory
cred, err := azidentity.NewDefaultAzureCredential(nil)
// TODO: handle err

// create a service.Client for the specified storage account that uses the above credential
client, err := service.NewClient("https://MYSTORAGEACCOUNT.dfs.core.windows.net/", cred, nil)
// TODO: handle err
// you can also create filesystem, file and directory clients
```

Learn more about enabling Azure Active Directory for authentication with Azure Storage in [our documentation][storage_ad] and [our samples](#next-steps).

## Key concepts

ADLS Gen2 provides:
- Hadoop-compatible access
- Hierarchical directory structure
- Optimized cost and performance
- Finer grain security model
- Massive scalability

ADLS Gen2 storage is designed for:

- Serving images or documents directly to a browser.
- Storing files for distributed access.
- Streaming video and audio.
- Writing to log files.
- Storing data for backup and restore, disaster recovery, and archiving.
- Storing data for analysis by an on-premises or Azure-hosted service.

ADLS Gen2 storage offers three types of resources:

- The _storage account_
- One or more _filesystems_ in a storage account
- One or more _files_ or _directories_ in a filesystem

Instances of the `Client` type provide methods for manipulating filesystems and paths within a storage account.
The storage account is specified when the `Client` is constructed. The clients available are referenced below.
Use the appropriate client constructor function for the authentication mechanism you wish to use.

### Goroutine safety
We guarantee that all client instance methods are goroutine-safe and independent of each other ([guideline](https://azure.github.io/azure-sdk/golang_introduction.html#thread-safety)). This ensures that the recommendation of reusing client instances is always safe, even across goroutines.

### About metadata
ADLS Gen2 metadata name/value pairs are valid HTTP headers and should adhere to all restrictions governing HTTP headers. Metadata names must be valid HTTP header names, may contain only ASCII characters, and should be treated as case-insensitive. Base64-encode or URL-encode metadata values containing non-ASCII characters.

### Additional concepts
<!-- CLIENT COMMON BAR -->
[Client options](https://pkg.go.dev/github.com/Azure/azure-sdk-for-go/sdk/azcore/policy#ClientOptions) |
[Accessing the response](https://pkg.go.dev/github.com/Azure/azure-sdk-for-go/sdk/azcore/runtime#WithCaptureResponse) |
[Handling failures](https://pkg.go.dev/github.com/Azure/azure-sdk-for-go/sdk/azcore#ResponseError) |
[Logging](https://pkg.go.dev/github.com/Azure/azure-sdk-for-go/sdk/azcore/log)
<!-- CLIENT COMMON BAR -->

## Examples

### Creating and uploading a file (assuming filesystem exists)

```go
const (
path = "https://MYSTORAGEACCOUNT.dfs.core.windows.net/sample-fs/sample-file"
)

// authenticate with Azure Active Directory
cred, err := azidentity.NewDefaultAzureCredential(nil)
// TODO: handle error

// create a client for the specified storage account
client, err := file.NewClient(path, cred, nil)
// TODO: handle error

_, err = client.Create(context.TODO(), nil)
// TODO: handle error

// open the file for reading
fh, err := os.OpenFile(sampleFile, os.O_RDONLY, 0)
// TODO: handle error
defer fh.Close()

// upload the file to the specified filesystem with the specified file name
_, err = client.UploadFile(context.TODO(), fh, nil)
// TODO: handle error
```

### Downloading a file

```go
const (
path = "https://MYSTORAGEACCOUNT.dfs.core.windows.net/sample-fs/cloud.jpg"
)

// authenticate with Azure Active Directory
cred, err := azidentity.NewDefaultAzureCredential(nil)
// TODO: handle error

// create a client for the specified storage account
client, err := file.NewClient(path, cred, nil)
// TODO: handle error

// create or open a local file where we can download the file
file, err := os.Create("cloud.jpg")
// TODO: handle error
defer file.Close()

// download the file
_, err = client.DownloadFile(context.TODO(), file, nil)
// TODO: handle error
```

### Creating and deleting a filesystem

```go
const (
fs = "https://MYSTORAGEACCOUNT.dfs.core.windows.net/sample-fs"
)

// authenticate with Azure Active Directory
cred, err := azidentity.NewDefaultAzureCredential(nil)
// TODO: handle error

// create a client for the specified storage account
client, err := filesystem.NewClient(fs, cred, nil)
// TODO: handle error

_, err = client.Create(context.TODO(), nil)
// TODO: handle error

_, err = client.Delete(context.TODO(), nil)
// TODO: handle error
```

### Enumerating paths (assuming filesystem exists)

```go
const (
fs = "https://MYSTORAGEACCOUNT.dfs.core.windows.net/sample-fs"
)

// authenticate with Azure Active Directory
cred, err := azidentity.NewDefaultAzureCredential(nil)
// TODO: handle error

// create a filesystem client for the specified storage account
client, err := filesystem.NewClient(fs, cred, nil)
// TODO: handle error

// path listings are returned across multiple pages
pager := client.NewListPathsPager(true, nil)

// continue fetching pages until no more remain
for pager.More() {
// advance to the next page
page, err := pager.NextPage(context.TODO())
// TODO: handle error

// print the path names for this page
for _, path := range page.PathList.Paths {
fmt.Println(*path.Name)
fmt.Println(*path.IsDirectory)
}
}
```

## Troubleshooting

All Datalake service operations will return an
[*azcore.ResponseError][azcore_response_error] on failure with a
populated `ErrorCode` field. Many of these errors are recoverable.
The [datalakeerror][datalake_error] package provides the possible Storage error codes
along with various helper facilities for error handling.


### Specialized clients

The ADLS Gen2 Storage SDK for Go provides specialized clients in various subpackages.

The [file][file] package contains APIs related to file path types.

The [directory][directory] package contains APIs related to directory path types.

The [lease][lease] package contains clients for managing leases on paths (paths represent both directory and file paths) and filesystems. Please see the [reference docs](https://docs.microsoft.com/rest/api/storageservices/lease-blob#remarks) for general information on leases.

The [filesystem][filesystem] package contains APIs specific to filesystems. This includes APIs setting access policies or properties, and more.

The [service][service] package contains APIs specific to Datalake service. This includes APIs for manipulating filesystems, retrieving account information, and more.

The [sas][sas] package contains utilities to aid in the creation and manipulation of Shared Access Signature tokens.
See the package's documentation for more information.


You can find additional context and examples in our samples for each subpackage (named examples_test.go).

## Contributing

See the [Storage CONTRIBUTING.md][storage_contrib] for details on building,
testing, and contributing to this library.

This project welcomes contributions and suggestions. Most contributions require
you to agree to a Contributor License Agreement (CLA) declaring that you have
the right to, and actually do, grant us the rights to use your contribution. For
details, visit [cla.microsoft.com][cla].

This project has adopted the [Microsoft Open Source Code of Conduct][coc].
For more information see the [Code of Conduct FAQ][coc_faq]
or contact [[email protected]][coc_contact] with any
additional questions or comments.

<!-- LINKS -->
[source]: https://github.com/Azure/azure-sdk-for-go/tree/main/sdk/storage/azdatalake
[docs]: https://pkg.go.dev/github.com/Azure/azure-sdk-for-go/sdk/storage/azdatalake
[rest_docs]: https://docs.microsoft.com/rest/api/storageservices/data-lake-storage-gen2
[godevdl]: https://go.dev/dl/
[goget]: https://pkg.go.dev/cmd/go#hdr-Add_dependencies_to_current_module_and_install_them
[storage_account_docs]: https://docs.microsoft.com/azure/storage/common/storage-account-overview
[storage_account_create_ps]: https://docs.microsoft.com/azure/storage/common/storage-quickstart-create-account?tabs=azure-powershell
[storage_account_create_cli]: https://docs.microsoft.com/azure/storage/common/storage-quickstart-create-account?tabs=azure-cli
[storage_account_create_portal]: https://docs.microsoft.com/azure/storage/common/storage-quickstart-create-account?tabs=azure-portal
[azure_sub]: https://azure.microsoft.com/free/
[azidentity]: https://pkg.go.dev/github.com/Azure/azure-sdk-for-go/sdk/azidentity
[storage_ad]: https://docs.microsoft.com/azure/storage/common/storage-auth-aad
[azcore_response_error]: https://pkg.go.dev/github.com/Azure/azure-sdk-for-go/sdk/azcore#ResponseError
[datalake_error]: https://github.com/Azure/azure-sdk-for-go/tree/main/sdk/storage/azdatalake/datalakeerror/error_codes.go
[filesystem]: https://github.com/Azure/azure-sdk-for-go/tree/main/sdk/storage/azdatalake/filesystem/client.go
[lease]: https://github.com/Azure/azure-sdk-for-go/tree/main/sdk/storage/azdatalake/lease
[file]: https://github.com/Azure/azure-sdk-for-go/tree/main/sdk/storage/azdatalake/file/client.go
[directory]: https://github.com/Azure/azure-sdk-for-go/tree/main/sdk/storage/azdatalake/directory/client.go
[sas]: https://github.com/Azure/azure-sdk-for-go/tree/main/sdk/storage/azdatalake/sas
[service]: https://github.com/Azure/azure-sdk-for-go/tree/main/sdk/storage/azdatalake/service/client.go
[storage_contrib]: https://github.com/Azure/azure-sdk-for-go/blob/main/CONTRIBUTING.md
[cla]: https://cla.microsoft.com
[coc]: https://opensource.microsoft.com/codeofconduct/
[coc_faq]: https://opensource.microsoft.com/codeofconduct/faq/
[coc_contact]: mailto:[email protected]
6 changes: 6 additions & 0 deletions sdk/storage/azdatalake/assets.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
{
"AssetsRepo": "Azure/azure-sdk-assets",
"AssetsRepoPrefixPath": "go",
"TagPrefix": "go/storage/azdatalake",
"Tag": "go/storage/azdatalake_c3c16cffab"
}
28 changes: 28 additions & 0 deletions sdk/storage/azdatalake/ci.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
trigger:
branches:
include:
- main
- feature/*
- hotfix/*
- release/*
paths:
include:
- sdk/storage/azdatalake

pr:
branches:
include:
- main
- feature/*
- hotfix/*
- release/*
paths:
include:
- sdk/storage/azdatalake


stages:
- template: /eng/pipelines/templates/jobs/archetype-sdk-client.yml
parameters:
ServiceDirectory: 'storage/azdatalake'
RunLiveTests: true
Loading

0 comments on commit beed165

Please sign in to comment.