Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Azure Storage: Azure storage support #3253 #3254

Merged
merged 81 commits into from
Oct 3, 2024
Merged
Changes from 1 commit
Commits
Show all changes
81 commits
Select commit Hold shift + click to select a range
06eac15
Setting up module for Azure storage #3253
sfali Aug 23, 2024
42b8505
Azure storage common settings #3253
sfali Aug 23, 2024
4412dec
Some common functions #3253
sfali Aug 23, 2024
dc26cea
Supporting model classes #3253
sfali Aug 23, 2024
882afcd
Exception representing returned error #3253
sfali Aug 23, 2024
5a8728e
Helper class to sign and populate Authorization header #3253
sfali Aug 23, 2024
3616c5f
Implementation of Azure storage functions #3253
sfali Aug 23, 2024
7626390
Scala and Java facade to implementation #3253
sfali Aug 23, 2024
f1de209
Introducing Akka stream attributes in order to configure http request…
sfali Aug 24, 2024
426b800
Resolve settings from Akka stream attributes #3253
sfali Aug 24, 2024
e41869e
Added support for SharedKeyLite authorization #3253
sfali Aug 24, 2024
5d3b55d
Added support for SAS authorization #3253
sfali Aug 24, 2024
dcc9c21
use option for null String #3253
sfali Aug 24, 2024
582edb3
function to get default settings #3253
sfali Aug 24, 2024
7dd77c6
Support for SAS authorization #3253
sfali Aug 24, 2024
d29f3f9
Splitting service operations based on service type #3253
sfali Aug 24, 2024
997746a
Splitting service operations based on service type #3253
sfali Aug 24, 2024
34832f6
StorageException test #3253
sfali Aug 25, 2024
6cc3254
creating constants for authorization types #3253
sfali Aug 25, 2024
d51c06a
implicit class to handle validations #3253
sfali Aug 25, 2024
7b3ed57
validations of configs and falling back to default values #3253
sfali Aug 25, 2024
07e64a5
StorageSettings tests #3253
sfali Aug 25, 2024
44c2f84
support for create container operation #3253
sfali Aug 25, 2024
93c2aab
support for endpoint URL for local testing #3253
sfali Aug 25, 2024
7ac0aa4
using end point if defined to create URI #3253
sfali Aug 25, 2024
9f013cb
move signer out of retry flow, we don't need to sign every time #3253
sfali Aug 27, 2024
9b81f84
copy right header #3253
sfali Aug 27, 2024
3ad11ea
pass implicit ActorSystem #3253
sfali Aug 27, 2024
790fd5a
parse content MD5 as well #3253
sfali Aug 27, 2024
d140558
populate environment variables to run test #3253
sfali Aug 27, 2024
db681c8
Integration tests #3253
sfali Aug 27, 2024
a46d27c
Added documentation for storage connector #3253
sfali Aug 27, 2024
a9f31ea
setting mimaPreviousArtifacts to make build work #3253
sfali Aug 27, 2024
9c1dd26
Added missing endPointUrl #3253
sfali Aug 28, 2024
5576940
minor return types fix #3253
sfali Aug 28, 2024
b281abb
base class for wire mock #3253
sfali Aug 28, 2024
e0afb81
Scala and Java mock tests for documentation #3253
sfali Aug 28, 2024
6a5f80c
fix java file paths #3253
sfali Aug 28, 2024
b96ed37
fix compilation #3253
sfali Aug 28, 2024
bc45086
fix compilations #3253
sfali Aug 28, 2024
5006117
fix documentation #3253
sfali Aug 28, 2024
9be3bf8
minor typo #3253
sfali Aug 28, 2024
b8600fc
extracted some constants to be used in sub classes #3253
sfali Aug 28, 2024
be29661
FileService test #3253
sfali Aug 28, 2024
49deaae
introducing factory class to create headers #3253
sfali Aug 28, 2024
cf48d55
updated implementation to get headers from dsl #3253
sfali Aug 28, 2024
c10b3d1
rename "putBlob" to "putBlockBlob" #3253
sfali Aug 28, 2024
96601a8
functions to put (create) page and append block #3253
sfali Aug 28, 2024
a4e0687
unit tests for put (create) page and append block #3253
sfali Aug 29, 2024
dc22e08
fix return types #3253
sfali Aug 29, 2024
562950e
rename in order to accommodate page blob header #3253
sfali Aug 29, 2024
e90a717
fix compilation #3253
sfali Aug 29, 2024
70466ce
Headers for page blob update #3253
sfali Aug 29, 2024
f167907
updated rename field #3253
sfali Aug 29, 2024
c750818
extracted common function and delegate all functions to this common f…
sfali Aug 29, 2024
c8afd96
clean up, headers move to separate class #3253
sfali Aug 29, 2024
ba27834
fix compiler warning #3253
sfali Aug 29, 2024
31bdf55
using ScalaIgnore #3253
sfali Aug 29, 2024
d1b5410
fix link #3253
sfali Aug 29, 2024
9882881
introduction of request builder #3253
sfali Aug 31, 2024
9358d5e
test for loading settings from custom path #3253
sfali Aug 31, 2024
2a0220b
remove unused import #3253
sfali Aug 31, 2024
9329318
Signer spec #3253
sfali Sep 2, 2024
e770b53
Fix for multiple content length header #3253
sfali Sep 4, 2024
1cbb77e
Clean up duplicate code #3253
sfali Sep 5, 2024
da8d82f
Add more test #3253
sfali Sep 7, 2024
9fb81d1
remove support for SharedKeyLite #3253
sfali Sep 7, 2024
b18fd50
Update azure-storage/src/test/resources/logback-test.xml
sfali Sep 9, 2024
67c2eaa
Update azure-storage/src/main/resources/reference.conf
sfali Sep 9, 2024
4cfe27a
Update azure-storage/src/main/scala/akka/stream/alpakka/azure/storage…
sfali Sep 9, 2024
f5971a1
Update azure-storage/src/main/scala/akka/stream/alpakka/azure/storage…
sfali Sep 9, 2024
1eaf5df
Update azure-storage/src/main/scala/akka/stream/alpakka/azure/storage…
sfali Sep 9, 2024
18f4336
Update azure-storage/src/main/scala/akka/stream/alpakka/azure/storage…
sfali Sep 10, 2024
fcd4326
Update azure-storage/src/main/scala/akka/stream/alpakka/azure/storage…
sfali Sep 10, 2024
948db31
Fail safe error message #3253
sfali Sep 10, 2024
60f9ae9
Merge remote-tracking branch 'origin/3253_azure_storage' into 3253_az…
sfali Sep 10, 2024
689cd53
Delete Container support #3253
sfali Sep 11, 2024
d6d9ed0
Create & Delete directory support #3253
sfali Sep 11, 2024
1c9c3cf
Merge branch 'master' into 3253_azure_storage
sfali Sep 30, 2024
43f347f
fix import #3253
sfali Sep 30, 2024
1e07711
Merge branch 'master' into 3253_azure_storage
sfali Oct 1, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
support for endpoint URL for local testing #3253
sfali committed Aug 25, 2024
commit 93c2aab79c7f1e1595fd3ea932297fcc65bb4d6f
3 changes: 3 additions & 0 deletions azure-storage/src/main/resources/reference.conf
Original file line number Diff line number Diff line change
@@ -4,6 +4,9 @@ alpakka {
api-version = ${?AZURE_STORAGE_API_VERSION}
signing-algorithm = "HmacSHA256"

# for local testing via emulator
# endpoint-url = ""

credentials {
# valid values are anon (annonymous), SharedKey, SharedKeyLite, sas
authorization-type = anon
Original file line number Diff line number Diff line change
@@ -18,6 +18,7 @@ import scala.util.Try

final class StorageSettings(val apiVersion: String,
val authorizationType: String,
val endPointUrl: Option[String],
val azureNameKeyCredential: AzureNameKeyCredential,
val sasToken: Option[String],
val retrySettings: RetrySettings,
@@ -29,6 +30,9 @@ final class StorageSettings(val apiVersion: String,
/** Java API */
def getAuthorizationType: String = authorizationType

/** Java API */
def getEndPointUrl: Optional[String] = endPointUrl.toJava

/** Java API */
def getAzureNameKeyCredential: AzureNameKeyCredential = azureNameKeyCredential

@@ -47,12 +51,15 @@ final class StorageSettings(val apiVersion: String,
/** Java API */
def withAuthorizationType(authorizationType: String): StorageSettings = copy(authorizationType = authorizationType)

/** Java API */
def withSasToken(sasToken: String): StorageSettings = copy(sasToken = emptyStringToOption(sasToken))

/** Java API */
def withAzureNameKeyCredential(azureNameKeyCredential: AzureNameKeyCredential): StorageSettings =
copy(azureNameKeyCredential = azureNameKeyCredential)

/** Java API */
def withSasToken(sasToken: String): StorageSettings = copy(sasToken = emptyStringToOption(sasToken))
def withEndPointUrl(endPointUrl: String): StorageSettings = copy(endPointUrl = emptyStringToOption(endPointUrl))

/** Java API */
def withRetrySettings(retrySettings: RetrySettings): StorageSettings = copy(retrySettings = retrySettings)
@@ -88,12 +95,19 @@ final class StorageSettings(val apiVersion: String,
private def copy(
apiVersion: String = apiVersion,
authorizationType: String = authorizationType,
endPointUrl: Option[String] = endPointUrl,
azureNameKeyCredential: AzureNameKeyCredential = azureNameKeyCredential,
sasToken: Option[String] = sasToken,
retrySettings: RetrySettings = retrySettings,
algorithm: String = algorithm
) =
StorageSettings(apiVersion, authorizationType, azureNameKeyCredential, sasToken, retrySettings, algorithm)
StorageSettings(apiVersion,
authorizationType,
endPointUrl,
azureNameKeyCredential,
sasToken,
retrySettings,
algorithm)
}

object StorageSettings {
@@ -104,24 +118,33 @@ object StorageSettings {
def apply(
apiVersion: String,
authorizationType: String,
endPointUrl: Option[String],
azureNameKeyCredential: AzureNameKeyCredential,
sasToken: Option[String],
retrySettings: RetrySettings,
algorithm: String
): StorageSettings =
new StorageSettings(apiVersion, authorizationType, azureNameKeyCredential, sasToken, retrySettings, algorithm)
new StorageSettings(apiVersion,
authorizationType,
endPointUrl,
azureNameKeyCredential,
sasToken,
retrySettings,
algorithm)

/** Java API */
def create(
apiVersion: String,
authorizationType: String,
endPointUrl: Optional[String],
azureNameKeyCredential: AzureNameKeyCredential,
sasToken: Optional[String],
retrySettings: RetrySettings,
algorithm: String
): StorageSettings =
StorageSettings(apiVersion,
authorizationType,
Option(endPointUrl.orElse(null)),
azureNameKeyCredential,
Option(sasToken.orElse(null)),
retrySettings,
@@ -145,6 +168,7 @@ object StorageSettings {
StorageSettings(
apiVersion = apiVersion,
authorizationType = authorizationType,
endPointUrl = config.getOptionalString("endpoint-url"),
azureNameKeyCredential = AzureNameKeyCredential(credentials),
sasToken = credentials.getOptionalString("sas-token"),
retrySettings = retrySettings,
Original file line number Diff line number Diff line change
@@ -124,5 +124,11 @@ class StorageSettingsSpec extends AnyWordSpec with Matchers {
val settings = mkSettings("retry-settings.min-backoff=hello")
settings.retrySettings shouldBe RetrySettings.Default
}

"populate endpoint URL if provided" in {
val settings = mkSettings("""endpoint-url="http://localhost:1234" """)
settings.endPointUrl shouldBe defined
settings.endPointUrl.get shouldBe "http://localhost:1234"
}
}
}