Releases: GoogleCloudDataproc/hadoop-connectors
Releases · GoogleCloudDataproc/hadoop-connectors
2.2.19
- Upgrade java-storage to 2.29.0
- Upgrade guava to 32.1.2-jre
3.0.0
- Remove Hadoop 2.x support.
- Update all dependencies to the latest versions.
- Add support for downscoped tokens in
AccessTokenProvider
. - Implement
FileSystem.openFile
to take advantage of theFileStatus
if
passed. - Remove an obsolete
AuthorizationHandler
and related properties:fs.gs.authorization.handler.impl fs.gs.authorization.handler.properties.<AUTHORIZATION_HANDLER_PROPERTY>
- Remove support for Apache HTTP transport and related property:
fs.gs.http.transport.type
- Support GCS fine-grained action in AuthorizationHandlers.
- Decrease log level for
hflush
rate limit log message. - Remove Cooperative Locking support for directory operations and related
properties:fs.gs.cooperative.locking.enable fs.gs.cooperative.locking.expiration.timeout.ms fs.gs.cooperative.locking.max.concurrent.operations
- Migrate authentication to
com.google.auth.Credentials
and remove obsolete
properties:fs.gs.auth.service.account.email fs.gs.auth.service.account.keyfile fs.gs.auth.service.account.private.key fs.gs.auth.service.account.private.key.id
- Refactor authentication configuration to use an explicit
fs.gs.auth.type
enum property, instead of relying on inference of the authentication type
based on the set configuration properties, and remove obsolete properties:fs.gs.auth.null.enable fs.gs.auth.service.account.enable
- Add support for a new
USER_CREDENTIALS
authentication type that retrieves
a refresh token using the authorisation code grant flow configured via the
following properties:fs.gs.auth.client.id fs.gs.auth.client.secret fs.gs.auth.refresh.token
- Merge all output stream types functionality in the default output stream
that behaves similarly to theFLUSHABLE_COMPOSITE
stream, and remove
obsoletefs.gs.outputstream.type
property. - Set default value for
fs.gs.list.max.items.per.call
property to5000
. - Set socket read timeout (
fs.gs.http.read-timeout
) as early as possible on
new sockets returned from the customSSLSocketFactory
. This guarantees the
timeout is enforced during TLS handshakes when using Conscrypt as the
security provider. - The Google Cloud Storage Connector now can be used as a
Hadoop Credential Provider. - Added dependency on the Cloud Storage Client Library
(google-cloud-storage). - Rename
fs.gs.rewrite.max.bytes.per.call
property to
fs.gs.rewrite.max.chunk.size
. - Remove support of the deprecated
fs.gs.io.buffersize.write
property. - Add support for size suffixes (
k
,m
,g
, etc) in values of size-related
properties:fs.gs.inputstream.inplace.seek.limit fs.gs.inputstream.min.range.request.size fs.gs.outputstream.buffer.size fs.gs.outputstream.pipe.buffer.size fs.gs.outputstream.upload.cache.size fs.gs.outputstream.upload.chunk.size fs.gs.rewrite.max.chunk.size
- Remove
.ms
suffix from names and add support for time suffixes (ms
,s
,
m
, etc) in values of time-related properties:fs.gs.http.connect-timeout fs.gs.http.read-timeout fs.gs.max.wait.for.empty.object.creation fs.gs.outputstream.sync.min.interval fs.gs.performance.cache.max.entry.age
- Change default values of properties:
fs.gs.http.connect-timeout (default: 20s -> 5s) fs.gs.http.read-timeout (default: 20s -> 5s) fs.gs.outputstream.upload.chunk.size (default: 64m -> 24m)
- Upgrade Hadoop to 3.3.5.
- Upgrade java-storage to 2.25.0
- Add support for
WORKLOAD_IDENTITY_FEDERATION_CREDENTIAL_CONFIG_FILE
authentication type that retrieves a refresh token using workload identity federation configuraiton defined in:fs.gs.auth.workload.identity.federation.credential.config.file
3.0.0-RC01
- Remove Hadoop 2.x support.
- Update all dependencies to the latest versions.
- Add support for downscoped tokens in
AccessTokenProvider
. - Implement
FileSystem.openFile
to take advantage of theFileStatus
if
passed. - Remove an obsolete
AuthorizationHandler
and related properties:fs.gs.authorization.handler.impl fs.gs.authorization.handler.properties.<AUTHORIZATION_HANDLER_PROPERTY>
- Remove support for Apache HTTP transport and related property:
fs.gs.http.transport.type
- Support GCS fine-grained action in AuthorizationHandlers.
- Decrease log level for
hflush
rate limit log message. - Remove Cooperative Locking support for directory operations and related
properties:fs.gs.cooperative.locking.enable fs.gs.cooperative.locking.expiration.timeout.ms fs.gs.cooperative.locking.max.concurrent.operations
- Migrate authentication to
com.google.auth.Credentials
and remove obsolete
properties:fs.gs.auth.service.account.email fs.gs.auth.service.account.keyfile fs.gs.auth.service.account.private.key fs.gs.auth.service.account.private.key.id
- Refactor authentication configuration to use an explicit
fs.gs.auth.type
enum property, instead of relying on inference of the authentication type
based on the set configuration properties, and remove obsolete properties:fs.gs.auth.null.enable fs.gs.auth.service.account.enable
- Add support for a new
USER_CREDENTIALS
authentication type that retrieves
a refresh token using the authorisation code grant flow configured via the
following properties:fs.gs.auth.client.id fs.gs.auth.client.secret fs.gs.auth.refresh.token
- Merge all output stream types functionality in the default output stream
that behaves similarly to theFLUSHABLE_COMPOSITE
stream, and remove
obsoletefs.gs.outputstream.type
property. - Set default value for
fs.gs.list.max.items.per.call
property to5000
. - Set socket read timeout (
fs.gs.http.read-timeout
) as early as possible on
new sockets returned from the customSSLSocketFactory
. This guarantees the
timeout is enforced during TLS handshakes when using Conscrypt as the
security provider. - The Google Cloud Storage Connector now can be used as a
Hadoop Credential Provider. - Added dependency on the Cloud Storage Client Library
(google-cloud-storage). - Rename
fs.gs.rewrite.max.bytes.per.call
property to
fs.gs.rewrite.max.chunk.size
. - Remove support of the deprecated
fs.gs.io.buffersize.write
property. - Add support for size suffixes (
k
,m
,g
, etc) in values of size-related
properties:fs.gs.inputstream.inplace.seek.limit fs.gs.inputstream.min.range.request.size fs.gs.outputstream.buffer.size fs.gs.outputstream.pipe.buffer.size fs.gs.outputstream.upload.cache.size fs.gs.outputstream.upload.chunk.size fs.gs.rewrite.max.chunk.size
- Remove
.ms
suffix from names and add support for time suffixes (ms
,s
,
m
, etc) in values of time-related properties:fs.gs.http.connect-timeout fs.gs.http.read-timeout fs.gs.max.wait.for.empty.object.creation fs.gs.outputstream.sync.min.interval fs.gs.performance.cache.max.entry.age
- Change default values of properties:
fs.gs.http.connect-timeout (default: 20s -> 5s) fs.gs.http.read-timeout (default: 20s -> 5s) fs.gs.outputstream.upload.chunk.size (default: 64m -> 24m)
- Upgrade Hadoop to 3.3.5.
- Upgrade java-storage to 2.25.0
- Add support for
WORKLOAD_IDENTITY_FEDERATION_CREDENTIAL_CONFIG_FILE
authentication type that retrieves a refresh token using workload identity federation configuraiton defined in:fs.gs.auth.workload.identity.federation.credential.config.file
2.2.18
- Upgrade java-storage to 2.28.0
- Integrate
journaling
,bufferToDIskThenUpload
andParallelCompositeUpload
APIs.
2.2.17
- Upgrade java-storage to 2.25.0
2.2.16
- Upgrade java-storage to 2.23.0
2.2.15
- Upgrade java-storage to 2.22.3.
- Add more instrumentation to GCS connector.
2.2.14
- Upgrade java-storage to 2.22.2
2.2.13
- Upgrade Hadoop to 2.10.2/3.2.4.
2.2.12
- Added implementation of gRPC read/write path using java-storage
- Upgrade Google Cloud Storage dependency to 2.17.2