v4.0.1

This is essentially a rerelease of 4.0.0, so full release notes included here for simplicity:

⛑️ Breaking changes

• Removes a number of static objects, in particular the "default" Atlan client that was accessible through Atlan.getDefaultClient() by @cmgrote in #1072. Every operation that interacts with Atlan (ultimately sending it an API request) must now be explicitly passed an AtlanClient. This change was made for two primary reasons:
  1. It makes explicit which operations will interact with Atlan (those that require an AtlanClient to be passed) vs those that only operate on information locally in-memory (those that do not require an AtlanClient to be passed).
  2. It removes concurrency limitations and risks with shared static resources across the entire JVM (including all threads therein). This applies not only to the default client but also to some of the caches that were being automatically managed.
• Packages developed using this release of the package toolkit will now need to make use of a broader PackageContext object that similarly encapsulates the client and any extra caches specific to packages.

🧪 Experimental

• Adds limited ability to configure an SSO provider by @cmgrote in #1069

🐞 Bug fixes

• Fixes validations when creating a README by @cmgrote in #1059

📦 Packages

• Adds multi-pass handling of cyclical relationships to asset import package by @cmgrote in #1061
• Fixes Python outputs generated by package toolkit by @ErnestoLoma in #1068
• Adds the ability in the package toolkit to set default values into the configuration objects by defining a fallback for any inputs in the package.pkl by @cmgrote in #1072
• Fixes impersonation within packages by @cmgrote in #1084
• Adds further type-checking on input CSVs by @cmgrote in #1086
• Fixes potential NPE in user iteration listing by @cmgrote in #1087

🥗 QOL improvements

• Migrates bulk streaming of search to use search_after by @cmgrote in #1060

Full Changelog: v3.1.2...v4.0.1

AtlanClient changes

As mentioned in the breaking changes, this release introduces a complete revamp of the way the AtlanClient is managed and passed around to operations that actually call Atlan APIs behind-the-scenes.

Resource-isolated

Previously the client itself and some caches were managed statically across the entire JVM. This could cause non-obvious and very difficult reproduce issues when heavy concurrency exists. With this release, all caches are managed within the AtlanClient directly, and there is no longer a JVM-wide static client. Instead, you can create a new client and have it "live" only for as long as you need it:

try (AtlanClient client = new AtlanClient()) {
    // Do anything you need against Atlan within this block
}
// Once you exit the block, the client and all its internal caches will be removed

Must be explicitly passed

Furthermore, any operation that actually interacts with Atlan directly (ultimately via its APIs behind-the-scenes) must now be explicitly passed such a client:

GlossaryTerm term = GlossaryTerm.findByName(client, "A", "B");
term.trimToRequired().userDescription("Hi there!").build().save(client);

This means there is an extensive set of changes to many of the methods across the SDK — if you're planning to adopt this release (and higher), rather than reading a huge list of such operations it'll probably be simplest to update your dependency to this latest release in your codebase and then look for any compilation breakages. The explicit AtlanClient is almost always expected as the new first parameter in any methods that now require it.

Package toolkit

Introduction of PackageContext

In the same way we have resource-isolated the client and its caches for the SDK, we have resource-isolated the context of a running package and its further caches in this release. The full context needed for running a package is now bundled into an object called a PackageContext, which can be created from a package's configuration using Utils.initializeContext() and passing it te configuration. Where you previously may have just used this pattern:

@JvmStatic
fun main(args: Array<String>) {
    val config = Utils.setPackageOps<OpenAPISpecLoaderCfg>()
    // Receive inputs from the config to decide what to do in your logic
}

You should now use this pattern:

@JvmStatic
fun main(args: Array<String>) {
    Utils.initializeContext<OpenAPISpecLoaderCfg>().use { ctx ->
        // Receive inputs from the config to decide what to do in your logic
        val url = ctx.config.specUrl // config items will be available nested under ctx.config
        val client = ctx.client // the AtlanClient will be nested under ctx.client
        val termCache = ctx.termCache // a number of lazy-load-managed caches are also available in the context
    }
}

v4.0.0

⛑️ Breaking changes

• Removes a number of static objects, in particular the "default" Atlan client that was accessible through Atlan.getDefaultClient() by @cmgrote in #1072. Every operation that interacts with Atlan (ultimately sending it an API request) must now be explicitly passed an AtlanClient. This change was made for two primary reasons:
  1. It makes explicit which operations will interact with Atlan (those that require an AtlanClient to be passed) vs those that only operate on information locally in-memory (those that do not require an AtlanClient to be passed).
  2. It removes concurrency limitations and risks with shared static resources across the entire JVM (including all threads therein). This applies not only to the default client but also to some of the caches that were being automatically managed.
• Packages developed using this release of the package toolkit will now need to make use of a broader PackageContext object that similarly encapsulates the client and any extra caches specific to packages.

🧪 Experimental

• Adds limited ability to configure an SSO provider by @cmgrote in #1069

🐞 Bug fixes

• Fixes validations when creating a README by @cmgrote in #1059

📦 Packages

• Adds multi-pass handling of cyclical relationships to asset import package by @cmgrote in #1061
• Fixes Python outputs generated by package toolkit by @ErnestoLoma in #1068
• Adds the ability in the package toolkit to set default values into the configuration objects by defining a fallback for any inputs in the package.pkl by @cmgrote in #1072

🥗 QOL improvements

• Migrates bulk streaming of search to use search_after by @cmgrote in #1060

Full Changelog: v3.1.2...v4.0.0

AtlanClient changes

As mentioned in the breaking changes, this release introduces a complete revamp of the way the AtlanClient is managed and passed around to operations that actually call Atlan APIs behind-the-scenes.

Resource-isolated

Previously the client itself and some caches were managed statically across the entire JVM. This could cause non-obvious and very difficult reproduce issues when heavy concurrency exists. With this release, all caches are managed within the AtlanClient directly, and there is no longer a JVM-wide static client. Instead, you can create a new client and have it "live" only for as long as you need it:

try (AtlanClient client = new AtlanClient()) {
    // Do anything you need against Atlan within this block
}
// Once you exit the block, the client and all its internal caches will be removed

Must be explicitly passed

Furthermore, any operation that actually interacts with Atlan directly (ultimately via its APIs behind-the-scenes) must now be explicitly passed such a client:

GlossaryTerm term = GlossaryTerm.findByName(client, "A", "B");
term.trimToRequired().userDescription("Hi there!").build().save(client);

This means there is an extensive set of changes to many of the methods across the SDK — if you're planning to adopt this release (and higher), rather than reading a huge list of such operations it'll probably be simplest to update your dependency to this latest release in your codebase and then look for any compilation breakages. The explicit AtlanClient is almost always expected as the new first parameter in any methods that now require it.

Package toolkit

Introduction of PackageContext

In the same way we have resource-isolated the client and its caches for the SDK, we have resource-isolated the context of a running package and its further caches in this release. The full context needed for running a package is now bundled into an object called a PackageContext, which can be created from a package's configuration using Utils.initializeContext() and passing it te configuration. Where you previously may have just used this pattern:

@JvmStatic
fun main(args: Array<String>) {
    val config = Utils.setPackageOps<OpenAPISpecLoaderCfg>()
    // Receive inputs from the config to decide what to do in your logic
}

You should now use this pattern:

@JvmStatic
fun main(args: Array<String>) {
    Utils.initializeContext<OpenAPISpecLoaderCfg>().use { ctx ->
        // Receive inputs from the config to decide what to do in your logic
        val url = ctx.config.specUrl // config items will be available nested under ctx.config
        val client = ctx.client // the AtlanClient will be nested under ctx.client
        val termCache = ctx.termCache // a number of lazy-load-managed caches are also available in the context
    }
}

v3.1.2

🐞 Bug fixes

• Fixes bug that caused full delta calculation in relational and cube assets builders to miss net-new assets by @cmgrote in #1055

Full Changelog: v3.1.1...v3.1.2

v3.1.1

🎉 New features

• Adds immediateNeighbors to FluentLineage to simplify lineage retrieval by @pavanmanish-atlan in #1048

🐞 Bug fixes

• Fixes bug with automated token exchange on expired API token retries by @cmgrote in #1043

📦 Packages

• Corrects downloads for GCS and ADLS deltas by @cmgrote in #1041
• Adds full delta calculation logic for relational and cube assets builders by @cmgrote in #1044
• Prevents renaming of connections as part of the enrichment migrator by @ErnestoLoma in #1049

New Contributors

• @pavanmanish-atlan made their first contribution in #1048

Full Changelog: v3.0.1...v3.1.0

v3.0.1

🎉 New features

• Adds username and extra (non-sensitive) fields to credential response by @cmgrote in #1026

🧪 Experimental

• Adds application typedefs and further data model typedef enhancements by @cmgrote in #1035

🐞 Bug fixes

• Corrects typo that caused CategoryCache to be loaded incorrectly by @cmgrote in #1030
• Avoids page overruns in edge cases where total results is very nearly the limit before query rewriting needs to happen by @cmgrote in #1033

📦 Packages

• Adds case insensitivity and table view agnosticism options to enrichment migrator by @ErnestoLoma in #1032
• Fixes bug where a purpose with no tag could cause admin export to fail by @cmgrote in #1036

Full Changelog: v3.0.0...v3.0.1

v3.0.0

🎉 New features

• Adds off-heap caching to reduce memory footprint by @cmgrote in #984
• Parses precision, scale, max length and raw definition from SQL dataType by @cmgrote in #1006
• Adds new connector types by @Aryamanz29 in #1005

🧪 Experimental

• Adds a new Application type with creator and unit tests by @prateekrai-atlan in #1008

⛑️ Breaking changes

• Previously, there were at least 2-3 separate approaches used for caching across the SDK and custom package toolkits, despite all of them relying purely on on-heap memory. In this release caches have been revamped:
  • to be more consistent (same operations across caches)
  • to move them off-heap (with disk spilling)
• We have tried to minimize the impact of this change, but if you are using these caches directly (or related custom package runtime toolkit functionality) you may be impacted. See the end of the release notes for further details.
• The custom package testing toolkit's interface has also been changed slightly:
  • The PackageTest class now requires a "tag" parameter, that is used in all generated names for that test to make them more immediately identifiable.
  • Rather than calling the setup() method with your custom package's configuration and then directly invoking your custom package's code (typically via its main() method), you should now use the runCustomPackage() method — which takes both the configuration and your custom package's execution method as parameters. This ensures your custom package is executed with appropriate isolation from any others that may be tested in parallel.

🐞 Bug fixes

• Fixes an issue in asset batching related to case-insensitivity handling by @ErnestoLoma in #990

📦 Packages

• Fixes a dependency issue on Google Cloud by @ErnestoLoma in #974
• Fixes container name used on Azure by @ErnestoLoma in #1013
• Passes through failOnErrors in relational assets builder, to allow continuation beyond failures by @cmgrote in #994
• Asset import improvements by @cmgrote in #1014
• Extends off-heap caching to all custom package asset caches by @cmgrote in #988
• Fixes generalization-specialization relationship attribute names for data model assets by @Aryamanz29 in #975

🥗 QOL improvements

• Extends off-heap caching to asset batch processing tracking by @cmgrote in #988

Full Changelog: v2.3.2...v3.0.0

Caching changes

As mentioned in the breaking changes, this release introduces a complete revamp of the way caching is done across both the SDK and custom package toolkits.

Moved off-heap

Previously the caching was done entirely in-memory, on-heap. This naturally limited its scale — to the point where we had begun to observe OutOfMemory errors in some cases. With this release, all caches are moved off-heap and managed by an embedded, persistent key-value store that still offers high performance. (Note that this does mean some temporary files will be created locally to manage the caches — these are created using Files.createTempDirectory(), so the specific location will depend on how the JVM manages this on your system.)

Consistency changes

We've consolidated 2-3 different interfaces for caching that had evolved independently during the lifecycle of the SDK and package toolkit to date, resolving that tech debt so that all can directly benefit from the move to off-heap.

• Each cache now manages the following for each object in it:
  • an ID (always a UUID)
  • a human-readable name
  • an optional secondary ID (such as the hashed-string internal representation for tags and custom metadata)
  • the actual object that was cached
• Retrieval now uses these options almost exclusively, for consistency. For example, instead of .getCustomMetadataDef() on the custom metadata cache, you would now just use .getByName().
• Prior operations like lookupAssetByIdentity and lookupAssetByGuid in some cache variations have been made consistent, to lookupByIdentity and lookupById.
• Wherever possible the caches are now implemented using Java generics typing, so should return strong types without needing casting (rather than generic types like Asset).

Necessary interface changes

Various places that used to make use of collections (Collection<Asset>, List<Asset>, etc) now instead either directly make use of the new OffHeapAssetCache or of Stream<Asset> — to no longer require everything in-memory on the heap in order to be able to process it. All caches are now generally Closeable, too, to allow them to self-manage cleaning up any temporary files they use for persistence.

• Utils.updateConnectionCache() in the custom package runtime toolkit (now uses OffHeapAssetCache)
• PersistentConnectionCache.deleteAssets() and .addAssets() in the custom package runtime toolkit (now process Stream<Asset>)
• AssetGenerator.cacheCreated() interface (now processes Stream<Asset>)
• ImportResults.Details internal tracking of created, updated, restored and skipped assets (now use OffHeapAssetCache)
• AssetRemover.deleteAssets() in the custom package runtime toolkit (now uses OffHeapAssetCache)
• DeltaProcessor.run() in the custom package runtime toolkit (now uses OffHeapAssetCache)
• AtlanClient itself is now Closeable, so it can clean up its own managed caches
• AssetBatch and ParallelBatch no longer offer fluent-builder creation, but only overloaded constructors, and both now track their processing of created, updated, restored and skipped assets in OffHeapAssetCaches, and are now Closeable so they can clean up their own managed caches.

v2.3.2

🎉 New features

• Latest models by @cmgrote in #954

🧪 Experimental

• Adds a new attribute ModelAttributeAssociation to track entity relationship for which it provides details @Aryamanz29 in #960
• Adds generalization-specialization relationship to data model assets by @Aryamanz29 in #951

🐞 Bug fixes

• Fixes logic used to populate LinkCache by @ErnestoLoma in #959, #962 and #968
• Fixes an infinite loop in loading data products with the same name by @ErnestoLoma in #965
• Fixes construction of user-agent headers to avoid potential misinterpretation in back-end parsing by @cmgrote in #969

📦 Packages

• Makes LFTagsOnTable and LFTagsOnColumns optional in LF tag sync package by @ErnestoLoma in #952

🥗 QOL improvements

• Dependency bumps by @cmgrote in #963

Full Changelog: v2.3.1...v2.3.2

v2.3.1

🎉 New features

• Adds bulk streaming for non-indexsearch by @cmgrote in #926
• Enhances concurrency for parallel batching by @cmgrote in #927 and #940

🧪 Experimental

• Model asset attribute changes by @Aryamanz29 in #935 and #937

🐞 Bug fixes

• Adds missing deserialization for domain policy actions by @cmgrote in #933
• Fixes a bug where stale connection admins could prevent adding others by @cmgrote in #938

📦 Packages

• Changes default load option for relational assets builder back to incremental by @cmgrote in #944
• Corrects GCS -> ADLS for logging by @cmgrote in #936

🥗 QOL improvements

• Reuses version catalog and enforces versions during build by @cmgrote in #945

Full Changelog: v2.3.0...v2.3.1

v2.3.0

🎉 New features

• Adds INRIVER connector type by @Aryamanz29 in #923
• Adds stronger concurrency controls to parallel batching by @cmgrote in #927
• Adds bulk streaming (paging beyond 10,000 entries) for non-indexsearch searches by @cmgrote in #926

🧪 Experimental

• Adds a new relationship for the implementation of an entity by one or more assets by @Aryamanz29 and @cmgrote in #921 and #925
• Fixes model asset references to use a version-agnostic qualifiedName by @cmgrote in #916 and #918

🐞 Bug fixes

• Fixes attribute validations for assets to avoid potential NPE by @cmgrote in #917
• Fixes an issue where deleted Atlan tags in audit log entries could cause an NPE by @cmgrote in #926

📦 Packages

• Uses requested connection qualifiedName as fallback for the OpenAPI spec loader package by @cmgrote in #922
• Adds support for assuming roles in AWS to the package runtime toolkit by @ErnestoLoma in #924

Full Changelog: v2.2.6...v2.3.0

v2.2.6

🐞 Bug fixes

• Fixes bug with timestamp granularity by @cmgrote in #913

Full Changelog: v2.2.5...v2.2.6