v2.1.0

What's Changed

• VAMS v2.1 Release by @scheurik in #201

[2.1.0] (2024-11-15)

This minor version includes changes to VAMS pipelines, use-case pipeline implementations, and v2.0 bug fixes.

Recommended Upgrade Path: A/B Stack Deployment with data migration using staging bucket configuration and upgrade migration scripts for DynamoDB tables in ./infra/upgradeMigrationScripts

⚠ BREAKING CHANGES

• Due to packaged library version upgrades in the solution, customer must make sure they are using the latest global installs of aws cli/CDK
• Pipelines are now changed to support a new pipelineType meaning, and the old pipelineType was renamed to pipelineExecutionType.
• Execution workflow input parameter names to pipelines have also changed, which can break existing workflows/pipelines.

Due to DynamoDB table structure changes, a A/B Stack deployment with migration script is recommended if there are existing pipelines that need to be automatically brought over.

Features

• Re-worked infrastructure CDK components and project directory structure to split out use-case pipelines (i.e., PotreeViewer/Visualizer Pipelines) from the rest of the lambda backend and stack infrastructures. This will allow for future upgrades that will split these components completely out into their own open-source project.
• PotreeViewerPipeline (previously VisualizerPipeline) is now baselined to the new standard use-case pipeline pattern to support external state machine callbacks (i.e., from VAMS pipeline workflows)
• • PreviewPotreeViewerPipeline (previously VisualizerPipeline) can now be registered and called from VAMS pipeline workflows (suggested to be called from a preview type pipeline) via the 'vamsExecutePreviewPcPotreeViewerPipeline' lambda function.
• Added a new use-case pipeline and configuration option for GenAiMetadata3dLabelingPipeline that can take in OBJ, FBX, GLB, USD, STL, PLY, DAE, and ABC files from an asset and use generative AI to analyze the file through 2D renders what keywords, tags, or other metadata the file should be associated with. Pipeline can be called by registering 'vamsExecuteGenAiMetadata3dLabelingPipeline' lambda function with VAMS pipelines / workflows.
• Added a new use-case pipeline and configuration option for Conversion3dBasic that can convert between STL, OBJ, PLY, GLTF, GLB, 3MF, XAML, 3DXML, DAE, and XYZ file types. VAMS pipeline registration outputType will define for each pipeline registration what the output file extension type will be.
• • This pipeline for non-GovCloud deployments is enabled by default in the infrastructure configuration.
• Web Added pipelineExecutionType to VAMS pipelines (previously pipelineType) and added a new context to pipelineType. Current pipeline types are StandardFile and PreviewFile. These are implemented to support future roadmap implementations of different pipeline types and auto-executions options on asset file uploads.
• Web Added inputParameters to pipelines to allow the optional specification of a JSON object which can be used within a pipeline execution to set pipeline configuration options. This is set at the time of creating a VAMS pipeline.
• Added inputMetadata to pipeline inputs which automatically pulls in asset name, description, tags, and all metadata fields of the asset to a pipeline execution. This can also be used in the future to pull through user-defined inputMetadata at the time of an execution with additional UI/UX.
• Changed inputPath and outputPath of pipeline function execution inputs to inputS3AssetFilePath and outputS3AssetFilesPath
• Added outputS3AssetPreviewPath, outputS3AssetMetadataPath, and inputOutputS3AssetAuxiliaryFilesPath pipeline execution parameter inputs to support different location paths for asset data outputs and writing to asset auxiliary temporary path locations
• Added outputType for user-specified expected file extension output for pipelines based on the VAMS pipeline registration. OutputType is not enforced and is something pipelines need to work into their own business logic as appropriate.
• • All asset write-back locations are now temporary job execution specific to allow for better security, file checks, proper back-versioning into an asset, and to start abstracting pipelines from writing directly to assets. Once the UploadV2 process is completed in a future update, direct access by use-case pipelines to S3 asset buckets will be removed in favor of API uploads / presigned URLs for storage abstraction.
• Updated processWorkflowExecutionOutput lambda function (previously uploadAllAssets) to also account for metadata data object outputs of pipelines to update against assets. Preview image output logic is stubbed out but will not be fully implemented until the new upload / storage process overhaul is completed in a future version.
• Added credTokenTimeoutSeconds authProvider config on the infrastructure side to allow manual specification of access, ID, and pre-signed URL tokenExpiration. Extending this can fix upload timeouts for larger files or slower connections. Auth refresh tokens timeouts are fixed to 24 hours currently.
• • Implements a new approach for s3ScopedAccess for upload that allows tokens up to 12 hours using AssumeRoleWithWebIdentity.
• Web Added PointCloud viewer and pipeline support for .ply file formats, moved from the 3D Mesh 3D Online Viewer
• Web The asset file viewer now says (primary) next to the assets main/primary associated file. The primary file is what get's used right now for pipeline ingestion when launching a workflow.
• Changed access logs S3 bucket lifecycle policy to only remove logs after 90 days
• Added lifecycle polcies on asset and asset auxiliary bucket to remove incomplete upload parts after 14 days

Bug Fixes

• Fixed CreateWorkflow error seen in v2.0 (Mac/Linux builds) with updated library dependencies and setting a standardized docker platforms across the board to linux/amd64
• Re-worked PreviewPotreeViewerPipeline (previously VisualizerPipeline) state machine and associated functions to properly handle errors
• Fixed benign logger errors in OpenSearch indexing lambda function (streams)
• Fixed existing functionality with processWorkflowExecutionOutput (previously uploadAllAssets) not working
• Fixed pipeline execution to properly account for asset file primary key names that contain spaces. Previously, could cause pipelines to error on execution.
• Web The asset file viewer now appropriately shows multiple files that are uploaded to the asset
• Web Hid the View %AssetName% Metadata button for top-level root folder on asset details page file manager that led to a blank page. The metadata for this is already on the asset details page.
• Fixed GovCloud deployments where v2 Lambda PreTokenGen for Cognito are not supported, reverted to v1 lambdas that only support Access Tokens (instead of both ID and Access token use for VAMS authorizers)
• Fixed GovCloud deployments for erronouesly including a GeoServices reference that is not supported in GovCloud partition
• Fixed KMS key IAM policy principals (for non-externally imported key setting) to include OpenSearch when using OpenSearch deployment configurations
• Added logic to look at other claims data if "vams:*" claims are not in the original JWT token. This is in prepartion for external IDP support and some edge case setups customers have.
• Fixed CDK deployment bug not deploying the required VPC endpoints during particular configurations of OpenSearch Provisioned, Not using all Lambda's behind VPCs, and using the option to use VPC endpoints
• Web Fixed bug where adding asset links had swapped the child/parent asset (WebUI only bug, API direct calls were not affected)
• Fixed CDK deployment bug of encrypting the WebAppLogsBucket when deploying with ALB and KMS encryption. The WebAppLogsBucket cannot be KMS encrypted when used for ALB logging output.
• Fixed bug for exceeding PolicyLimitSize of STS temporary role calls in S3ScopedAccess used during asset upload from the Web UI when KMS encryption is enabled.
• Increased CustomResource lambda timeouts for OpenSearch schema deployment that caused issues intermitently during GovCloud deployments
• Fixed bug in constraint service API that was saving constraints on POST/PUT properly but was erroring on generating a 200 response resulting in a 500 error
• Fixed bug in OpenSearch indexing (bad logging method) during certain edge cases that prevented adding new data to the index
• Fixed bug in CDK storageResource helper function where S3 buckets were not getting the proper resource policies applied

Chores

• VisualizerPipeline now re-named to PreviewPotreeViewerPipeline as the previous name was too generic and other "visualizer" or viewer pipelines may exist later
• 'visualizerAssets' S3 bucket renamed to 'assetAuxiliary'. This bucket will now be used for all pipeline or otherwise auto-generated files (previews/thumbnails) associated with assets that should not be versioned
• 'visualizerAssets/{proxy+}' API route and related function re-named to 'auxililaryPreviewAssets/stream/{proxy+}'. This function is used for retrieving auto-generated preview files that should be rapidly streamed such as the PreviewPotreeViewerPipeline files.
• Renamed and moved uploadAllAssets lambda function handler. It is now processWorkflowExecutionOutput and moved to the workflows backend folder
• Updated Workflow ListExecutions to write stopDate, startDate, and executionStatus back to DynamoDB table after an SFN fetch where the execution has stopped. This is done for performance / caching reasons.
• Workflow executions are now limited to only 1 active running...

v2.0.0

Highlights

1. CDK Infrastructure Overhaul: This release represents a major overhaul of the CDK constructs, splitting the core logic into multiple nested stacks to support more scalable deployment configurations.
2. Configuration System: A new CDK configuration system has been introduced using config.json and cdk.json files. Many previously implemented features, such as OpenSearch or Location Services, can now be turned on or off.
3. New Configuration Options: Numerous new configuration options have been added, such as VPC/subnet management, Application Load Balancer (ALB) static web support instead of CloudFront, KMS encryption, OpenSearch configurations (including the ability to turn off OpenSearch), and more. These options can be toggled based on specific deployment requirements.
4. Security Controls: A major aspect of this release focuses on security tightening and controls. Implementers will now be able to deploy across AWS partitions, including GovCloud, and have more control over WAF, FIPS, Lambdas in VPCs, and Docker SSL Proxy configurations.
5. New Access Control System: A new Attribute-Based Access Control (ABAC) and Role-Based Access Control (RBAC) system has been implemented, replacing the previous Cognito group-based access control. This provides fine-grained access control to various VAMS resources.
6. Asset Tagging and Linking: A new mechanism for adding tags and tag types to assets has been introduced, along with the ability to create parent/child and related-to links between assets within the same database.
7. Image and PointCloud Viewers: Support for Image and PointCloud file visualizations has been added, including an infrastructure data pipeline to support viewer conversions for LAS, LAZ, and E57 input formats.
8. Upgraded File Manager: The web assets viewer has a new file manager UI/UX for viewing asset files and provides functionality for uploading multiple asset files within folders.
9. Email Subscription System: A new email subscription system has been implemented which allows VAMS users to subscribe to various data changes. Asset data objects are the first to be implemented as part of this version to allow users to receive notifications when new asset file versions are uploaded.
10. Performance and Bug Fixes: Various performance improvements and bug fixes have been implemented, including API input validations, optimizations for OpenSearch indexing, log group naming, unique resource naming, and workflow execution handling.
11. Deprecations and Removals: SageMaker pipeline types have been removed to focus development efforts on Lambda pipelines.

⚠️ BREAKING CHANGES

• Possible break CDK configuration and feature switch system using ./infra/config/config.json file. Some backwards compatibility with existing CDK deployment commands.
• CDK overhaul to split core logic into 10+ nested stacks means that an in-place upgrade for existing stack deployments is not possible, use A/B deployment.
• Lambdas converted into inline code functions with layers (away from Lambda ECR-backed containers).
• (SEO breakage) Switch Web infrastructure to use React hash router instead of web router to support ALB configuration option, which breaks search engine optimizations (SEO).
• New ABAC/RBAC systems will require new roles and constraints to be set up to allow application access. Existing Cognito groups will no longer be recognized, and user memberships must be transferred to the role and constraint mechanisms.
• SageMaker is no longer a pipeline type available. Existing SageMaker pipelines should be converted to be executed from a lambda pipeline.
• Restrict VAMS workflow pipelines to only have permission to lambdas that contain vams in the function name by default. If you have external pipeline lambdas, please add invoke permissions for them to the appropriate workflow execution role or update your lambda function name to contain vams.
• Pipelines created using the default lambda artifact sample will now need to be re-created and re-inserted into workflows due to using different database fields to store the name of these.
• /assets/all (PUT) API call is deprecated in favor of using the existing /assets (PUT) and the newer /ingestAsset (POST) API.
• Previously created workflows of pipelines that had pipeline nodes that didn't use wait_for_callback need to be re-created/re-saved from the VAMS UI or modified in the AWS Console to remove TaskToken.$ from node tasks parameters if there is no callback on that node.
• API response bodies for data retrieval calls that return several records have been standardized to responseBody: {message: {Items, NextToken}}.

Features

• Implement CDK configuration system using ./infra/config/config.json file.
• Implement local Docker package build file configuration override to support customization in ./infra/config/docker/Dockerfile-customDependencyBuildConfig (such as in cases of HTTPS SSL proxy certificate support).
• Add default template files for various configuration environments (commercial (default- config.json), GovCloud).
• Implement new CDK environment system variables using ./infra/cdk.json file.
• Add global stack resource tagging.
• Add global new role permission boundary support.
• Add global new role name prefix tagging.
• Implement feature switch system and storage for Web feature toggling (new DynamoDB table).
• Web Load/cache enabledFeatures as part of the backend web configuration load to the frontend.
• Implement GovCloud feature switch which toggles other features on/off based on GovCloud service support and certain best practices.
• Implement FIPS support configuration option.
• Implement WAF configuration option (existing WAF functionality, ability to now toggle off).
• Implement Global VPC configuration option used for particular configuration needs.
• Support new VPC/Subnet generation.
• Support an option for external VPC/subnet imports (instead of new VPC generation).
• Added implementation of LoadContext Deployment configuration to support VPC context loading before main deployment.
• Support an option for auto-adding*new VPC endpoints based on other configuration switches (*with some exceptions in particular configurations that will still auto-add regardless of this flag).
• Support putting all deployed lambdas behind VPC (FedRamp best practices for GovCloud).
• Implement ALB configuration option for static WebApp delivery (replaces CloudFront when enabled).
• Requirement Note: ALB tied to a registered domain that must be provided.
• Support WAF (if used) to deploy globally or regionally based on ALB/CloudFront deployments.
• Support for using public private subnets for ALB.
• Support/Requirement for SSL/TLS ACM certificate import for ALB.
• Support for optional externally imported Route53 HostedZone updating for ALB deployment.
• Implement KMS CMK encryption configuration option for all*at-rest storage (*with some S3 bucket exceptions in particular configurations such as ALB use).
• Support new key generation on stack deploy.
• Support option for external CMK key import instead of new key generation.
• Disable all KMS CMK keys use implemented previously when configuration feature disabled (e.g., S3 bucket SNS notification queues). Uses default/AWS-managed encryption when KMS CMK disabled.
• Implement OpenSearch provisioned, serverless, or no (neither serverless nor provisioned enabled) open search configuration options; No open search will disable VAMS asset search functionality.
• Implement location service configuration option and feature switch (existing location service functionality, ability to now toggle off).
• Web Hides Map view from Assets web page when turned off.
• Implement point cloud visualization configuration option (existing pipeline functionality, ability to now toggle off through configuration file).
• Add VAMS upgrade migration scripts to support A/B deployments and data migration between stack deployments in ./infra/deploymentDataMigration.
• (Future Full-Implementation) Implement authentication provider configuration option and feature switch. Note: Currently, only the Cognito useSaml configuration flag is observed (moved from saml-config.ts file), other auth types will cause an unimplemented error.
• Implement new initial ABAC/RBAC access control systems to allow for fine-grained access to various VAMS resources (built on the Casbin open-source library).
• ABAC defines the primary constraints and access controls.
• ABAC currently supports resources of Databases, Assets, and "APIs".
• Note Databases and Assets control primary VAMS storage resources. APIs control access to top-level system functionality (administrative pages, pipelines/workflows, etc.).
• RBAC roles map to ABAC constraints to allow for backward compatibility with role/group-based access systems.
• ABAC constraints can also map directly to users if organizations choose to go solely with the ABAC system.
• Removed the previous Cognito group and constraint system.
• Note Starts to reduce dependency on Cognito functionalities.
• Created default admin role and constraint groups on new VAMS deployment. Stack deployment user will be auto-added to this new role group.
• All lambdas now check access against the new ABAC constraints system.
• Web Allowed Web routes controlled by ABAC constraints.
• Web Administrative UI pages to support roles, role membership, constraints, and constraint membership modifications.
• Implement new tag and tag type mechanism for adding additional information on assets (tags/tag types are currently global across all databases).
• Note Requirement that Tags must have a tag type assigned.
• Web Ability to search tags on assets on the asset search page.
• Web Ability to assign/unassign tags to assets on asset creat...

v1.4.0

What's Changed

• Metadata schemas for VAMS databases by @archieco in #86
• feat: Easily replace terms Asset and Database by @archieco in #88
• feat!: Support uploading folders as assets by @ravij3 in #92
• fix: repair regression on createPipeline by @archieco in #93
• fix: single folder single file upload by @ravij3 in #95
• fix: dependency conflict was causing downloads to fail by @ravij3 in #94
• Added File Level Metadata by @ravij3 in #96
• feat: hiding sign up by @ravij3 in #104
• OpenSearch Integration by @archieco in #103
• chore(deps): bump semver from 5.7.1 to 5.7.2 by @dependabot in #105
• fix: download asset only if they are marked as distributatble by @ravij3 in #106
• fix: repair first deployment with opensearch by @archieco in #107
• Rel 1.4 fixes by @ravij3 in #108
• fix: Release fixes by @ravij3 in #109
• Rel 1.4 fixes by @ravij3 in #113
• fix: simplify auth constraints screen by @archieco in #115
• Fixing download links and cancel button on update asset button by @ravij3 in #117
• feat(web): improvements to metadata component by @archieco in #110
• add multi file archiving capability by @amigitamz in #112
• chore(deps): bump certifi from 2022.12.7 to 2023.7.22 in /backend by @dependabot in #111

New Contributors

• @amigitamz made their first contribution in #112

Full Changelog: v1.3.1...v1.4.0

v1.3.1

What's Changed

• One small improvement to RBAC
• A fix was made to the workflow table on the ViewAsset screen

Full Changelog: v1.3.0...v1.3.1

For a full description of changes in the 1.3.0 release, see https://github.com/awslabs/visual-asset-management-system/releases/tag/v1.3.0

v1.3.0

What's New

• VAMS operators can now leverage SAML to federate users from their primary identity provider such as Microsoft Active Directory or any SAML IdP with Amazon Cognito.
• A new Role Based Access Control system enables VAMS administrators to provision access to users in accordance with their roles and responsibilities.
• A configuration preview of Attribute Based Access Control demonstrates fine grained access control to VAMS assets using metadata.
• New support for long running Pipelines with Step Functions' wait for callback feature. Pipeline execution can now take up to 1 year whereas before they were limited to 15 minutes for Lambda functions or 24 hours for Sagemaker Jobs.
• Additional automated security tests are executed on every change made to VAMS in its continuous integration pipeline leveraging the Automated Security Helper (ASH, https://github.com/aws-samples/automated-security-helper)

Detailed Changes

• chore(release): 1.2.0 by @ravij3 in #45
• feat(web): add new model visualizer supporting .obj, .gltf, .glb, .st… by @stephcurt in #42
• feat: apigw authorizer for amplify config endpoint by @archieco in #47
• Jjbain add security tests to ci script by @jjbainAWS in #52
• chore: prettier configuration and reformatting by @archieco in #51
• chore: made corrections to links in changelog by @archieco in #53
• eslint by @archieco in #54
• feat: enable cloudfront compression by @archieco in #48
• Federated authentication using SAML. by @archieco in #57
• added automated security helper by @jjbainAWS in #56
• Adjusted default permission for S3 buckets to match latest security changes by @AMZN-stankowi in #62
• Update README.md by @archieco in #61
• Issue 63 Fix - Windows Build Error by @copystart in #64
• Authrbac1 by @archieco in #60
• feat: Fine grained authorization rule definition by @archieco in #67
• fix: Hitting Execute Workflow button from the assets page doesn't work by @ravij3 in #70
• fix: automatically navigate to asset page once asset upload completes by @ravij3 in #71
• fix: resolves issue #68, workflow editor added extra pipelines by @archieco in #74
• fix: renaming userpool causes failures in existing stack by @ravij3 in #72
• fix: Updated cdk-nag suppression by @archieco in #77
• fix: congitoUsername --> cognitoUsername, added dependency to userGroupAttachment by @lmarbleAWS in #79
• docs: file formats list for online viewing by @archieco in #81
• fix: updated the workflow editor by @archieco in #80
• fix: cdk nag suppressions for python 3.9 and nodejs14.x by @archieco in #78
• feat: Support long running pipelines with Step Functions' wait for callback feature. by @archieco in #76
• chore(deps): bump requests from 2.30.0 to 2.31.0 in /backend by @dependabot in #82
• fix: resolve to fast-xml-parser 4.2.4 by @archieco in #89
• Resolved a couple of issues related to new authorization functionality. by @archieco in #85
• Jjbain update asset ux fix by @jjbainAWS in #83

New Contributors

• @AMZN-stankowi made their first contribution in #62
• @copystart made their first contribution in #64
• @lmarbleAWS made their first contribution in #79

Full Changelog: v1.2.0...v1.3.0

v1.2.0

Highlighted Changes

• A new asset upload wizard was created that enables users to upload files, create metadata in key/value pairs, and select workflows to execute on the asset upon upload.
• The asset detail screen now includes controls to update metadata for the asset.
• The user interface was updated to use the Cloudscape design system.
• Users with software that runs in containers or AWS Lambda functions can provide the ARN (Amazon Resource Name) referencing ECR image URI or AWS Lambda function to leverage their software to transform assets in VAMS.
• VAMS allow list of file types now includes STEP files.
• Assets are identified by a UUID while the user provided asset name is preserved with each asset rather than using the asset name as a natural key.

What's Changed

• fix: change log s3 bucket encryption type to S3_MANAGED by @stephcurt in #7
• fix: change all buckets to S3_MANAGED encryption by @archieco in #8
• feat(infra) parameterized region code by @jjbainAWS in #3
• chore(deps): bump certifi from 2022.9.24 to 2022.12.7 in /backend by @dependabot in #2
• chore: prettier configuration by @archieco in #12
• infra: adding uploadAssetWorkflow components by @ravij3 in #13
• ci: adding ci tools for backend repository by @ravij3 in #14
• fix: updating ci.yml by @ravij3 in #15
• ci: added ci for web and cdk by @ravij3 in #16
• chore(deps): bump axios from 0.21.1 to 0.26.0 in /web by @dependabot in #9
• fix: fixing loader-utils security vulnerability by @ravij3 in #18
• feat: Added uploadAssetWorkflow lambda function by @ravij3 in #20
• Metadata backend and frontend by @archieco in #19
• feat: updates to UploadAssetWorkflow stepFunction by @ravij3 in #22
• fix: downgrading default notebook platform by @ravij3 in #26
• chore(release): 1.0.1 by @archieco in #21
• feat: uploadAssetWorkflow stepfunction orchestration by @ravij3 in #27
• feat(web): awsui css replaced with cloudscape css by @archieco in #23
• chore: update broken links on DeveloperGuide by @ravij3 in #29
• feat(web): New asset upload screen by @archieco in #28
• chore: Repair copyright headers by @archieco in #30
• chore: update to list_objects_v2 by @archieco in #33
• fix: s3 copy_object calls include owner acct ids by @archieco in #32
• chore: remove unused resources by @archieco in #31
• chore(deps): bump werkzeug from 2.2.2 to 2.2.3 in /backend by @archieco in #34
• chore: adding fbx file formats for pipelines by @ravij3 in #35
• fix: security updates reported through automated security helper by @ravij3 in #36
• fix(web): update create asset buttons by @archieco in #40
• Several minor fixes and updates by @archieco in #37
• docs: Document the schemas of the dynamodb tables by @archieco in #41
• Jjbain bring your own arn by @jjbainAWS in #38
• docs: updates to documentation by @ravij3 in #43
• docs: update developer docs by @ravij3 in #44

New Contributors

• @stephcurt made their first contribution in #7
• @archieco made their first contribution in #8
• @jjbainAWS made their first contribution in #3
• @dependabot made their first contribution in #2

Full Changelog: https://github.com/awslabs/visual-asset-management-system/commits/v1.2.0

v1.0.1

What's Changed

• fix: change log s3 bucket encryption type to S3_MANAGED by @stephcurt in #7
• fix: change all buckets to S3_MANAGED encryption by @archieco in #8

New Contributors

• @stephcurt made their first contribution in #7

Full Changelog: https://github.com/awslabs/visual-asset-management-system/commits/v1.0.1

v1.0.0

Full Changelog: https://github.com/awslabs/visual-asset-management-system/commits/v1.0.0