Release: API v7.10.0 SDK v0.1.8 (no10ds#84)

* Fix issues with links not working within read the docs (no10ds#83) * Feature/sdk additional functions (no10ds#73) * user create/delete * add functions * add tests * fix sdk test functions bug * fix test function url bug * cover user create and dataset delete errors raise * fix errors handling and testing * fix syntax bug * fix error with mocked responses * standarize functions when undefined error code * fix test mock response * Extra Output Blocks (no10ds#80) * output blocks * respond to comments * fix ckan_sg issue (no10ds#79) * Changelog --------- Co-authored-by: Jarek-Rolski <[email protected]> Co-authored-by: MotwaniM <[email protected]>
co-cddo · Mar 21, 2024 · d8a8e90 · d8a8e90
1 parent 864f098
commit d8a8e90
Show file tree

Hide file tree

Showing 15 changed files with 437 additions and 69 deletions.
diff --git a/docs/api/data_access.md b/docs/api/data_access.md
@@ -23,5 +23,5 @@ not the reverse.
 Protected datasets as isolated from the hierarchy and are domain specific. This means that you have to be granted access
 to specific protected domains to gain access.
 
-These domains must first be [created](/api/routes/protected_domain/#create) and then assigned to
-a [client](/api/routes/client/#create) or [user](/api/routes/user/#create) for usage.
+These domains must first be [created](./routes/protected_domain.md/#create) and then assigned to
+a [client](./routes/client.md/#create) or [user](./routes/user.md/#create) for usage.
diff --git a/docs/api/schema.md b/docs/api/schema.md
@@ -35,7 +35,7 @@ A schema is defined with the following structure:
 The sensitivity level of a dataset can be described by one of three values: `PUBLIC`, `PRIVATE` and `PROTECTED`.
 These determine the access level that different clients will have to the data depending on their permissions.
 
-Notes if you wish to use the sensitivity level `PROTECTED` then you must first create a Protected Domain for your Dataset. See the [data access docs](data_access.md)
+Notes if you wish to use the sensitivity level `PROTECTED` then you must first create a Protected Domain for your Dataset. See the [data access docs](./data_access.md)
 
 ### Description
 

diff --git a/docs/api/usage.md b/docs/api/usage.md
@@ -2,14 +2,14 @@ The rAPId API serves to make data storage and retrieval as easy and consistent a
 
 The API functionality includes:
 
-- [Uploading a schema (i.e. creating a new dataset definition)](/api/routes/schema/)
+- [Uploading a schema (i.e. creating a new dataset definition)](./routes/schema.md)
   - Also creating a new version of an existing schema
-- [Uploading data to any version of a dataset](/api/routes/dataset/#upload)
-- [Listing available data](/api/routes/dataset/#list)
-- [Querying data from any version of a dataset](/api/routes/dataset/#query)
-- [Deleting data](/api/routes/dataset/#delete-data-file)
-- Creating [users](/api/routes/user/#create) and [clients](/api/routes/client/#create)
-- [Managing user and client permissions](/api/routes/subject/#modify-subject-permissions)
+- [Uploading data to any version of a dataset](./routes/dataset.md/#upload)
+- [Listing available data](./routes/dataset.md/#list)
+- [Querying data from any version of a dataset](./routes/dataset.md/#query)
+- [Deleting data](./routes/dataset.md/#delete-data-file)
+- Creating [users](./routes/user.md/#create) and [clients](./routes/client.md/#create)
+- [Managing user and client permissions](./routes/subject.md/#modify-subject-permissions)
 
 ## Application Usage Overview
 

diff --git a/docs/changelog/api.md b/docs/changelog/api.md
@@ -1,10 +1,16 @@
 # API Changelog
 
-# Changelog
+## v7.10.0 - _2024-03-21_
 
-## v7.0.9 - _2024-02-06_
+### Fixes
+
+- Issues with the documentation where links were not re-directing successfully.
+
+### Features
+
+- Extra infrastructure output blocks
 
-See [v7.0.9] changes
+## v7.0.9 - _2024-02-06_
 
 ### Features
 
@@ -112,15 +118,3 @@ See [v7.0.9] changes
 ### Migration
 
 - See the [migration doc](migration.md) for details on how to migrate to v7 from v6.
-
-[Unreleased changes]: https://github.com/no10ds/rapid/compare/v7.0.9...HEAD
-[v7.0.9]: https://github.com/no10ds/rapid/compare/v7.0.8...v7.0.9
-[v7.0.8 / v0.1.6 (sdk)]: https://github.com/no10ds/rapid/v7.0.7...v7.0.8
-[v7.0.7 / v0.1.5 (sdk)]: https://github.com/no10ds/rapid/v7.0.6...v7.0.7
-[v7.0.6 / v0.1.4 (sdk)]: https://github.com/no10ds/rapid/v7.0.5...v7.0.6
-[v7.0.5 / v0.1.3 (sdk)]: https://github.com/no10ds/rapid/v7.0.4...v7.0.5
-[v7.0.4 / v0.1.2 (sdk)]: https://github.com/no10ds/rapid/v7.0.3...v7.0.4
-[v7.0.3 / v0.1.2 (sdk)]: https://github.com/no10ds/rapid/v7.0.2...v7.0.3
-[v7.0.2 / v0.1.2 (sdk)]: https://github.com/no10ds/rapid/v7.0.1...v7.0.2
-[v7.0.1 / v0.1.2 (sdk)]: https://github.com/no10ds/rapid/v7.0.0...v7.0.1
-[v7.0.0 / v0.1.1 (sdk)]: https://github.com/no10ds/rapid/v7.0.0
diff --git a/docs/changelog/sdk.md b/docs/changelog/sdk.md
@@ -1,5 +1,11 @@
 # SDK Changelog
 
+## v0.1.8 - _2024-03-21_
+
+### Features
+
+- Ability to now perform the following rAPId functions via the sdk; create user, delete user, list subjects, list layers, list protected domains and delete dataset.
+
 ## v0.1.7 - _2023-02-06_
 
 ### Features
@@ -67,14 +73,3 @@
 ### Migration
 
 - See the [migration doc](migration.md) for details on how to migrate to v7 from v6.
-
-[Unreleased changes]: https://github.com/no10ds/rapid/compare/v7.0.8...HEAD
-[v7.0.8 / v0.1.6 (sdk)]: https://github.com/no10ds/rapid/v7.0.7...v7.0.8
-[v7.0.7 / v0.1.5 (sdk)]: https://github.com/no10ds/rapid/v7.0.6...v7.0.7
-[v7.0.6 / v0.1.4 (sdk)]: https://github.com/no10ds/rapid/v7.0.5...v7.0.6
-[v7.0.5 / v0.1.3 (sdk)]: https://github.com/no10ds/rapid/v7.0.4...v7.0.5
-[v7.0.4 / v0.1.2 (sdk)]: https://github.com/no10ds/rapid/v7.0.3...v7.0.4
-[v7.0.3 / v0.1.2 (sdk)]: https://github.com/no10ds/rapid/v7.0.2...v7.0.3
-[v7.0.2 / v0.1.2 (sdk)]: https://github.com/no10ds/rapid/v7.0.1...v7.0.2
-[v7.0.1 / v0.1.2 (sdk)]: https://github.com/no10ds/rapid/v7.0.0...v7.0.1
-[v7.0.0 / v0.1.1 (sdk)]: https://github.com/no10ds/rapid/v7.0.0
diff --git a/docs/getting_started.md b/docs/getting_started.md
@@ -2,8 +2,8 @@
 
 We provide two options for deploying rAPId within an AWS environment:
 
-1. If you have existing infrastructure (e.g a VPC) that you would like to deploy rAPId within, then you can use the [rAPId module](/infrastructure/deployment/existing/), passing in specific variables relating to your AWS account.
-2. If you do not have any existing infrastructure, you can instead deploy the [entire rAPId stack](/infrastructure/deployment/full_stack/) creating all the relevant infrastructure.
+1. If you have existing infrastructure (e.g a VPC) that you would like to deploy rAPId within, then you can use the [rAPId module](./infrastructure/deployment.md/#rapid-module), passing in specific variables relating to your AWS account.
+2. If you do not have any existing infrastructure, you can instead deploy the [entire rAPId stack](./infrastructure/deployment.md/#full_stack) creating all the relevant infrastructure.
 
 # Usage
 
@@ -15,4 +15,4 @@ Navigate to your AWS account and under secrets manager find the `<rAPId-prefix>_
 
 # Developing
 
-Alternatively you can run rAPId locally for development. For more details, please see the [contributing section](/contributing/).
+Alternatively you can run rAPId locally for development. For more details, please see the [contributing section](./contributing.md).
diff --git a/docs/infrastructure/deployment.md b/docs/infrastructure/deployment.md
@@ -35,7 +35,7 @@ Provide the required inputs as described:
 
 - `aws_account` - AWS account where the application will be hosted
 - `aws_region` - AWS region where the application will be hosted
-- `domain_name` - Application hostname ([can be a domain or a subdomain](/infrastructure/domains_subdomains/))
+- `domain_name` - Application hostname ([can be a domain or a subdomain](./domains_subdomains.md))
 - `ip_whitelist` - A list of IP addresses that are allowed to access the service.
 - `public_subnet_ids_list` - List of public subnets for the load balancer
 - `private_subnet_ids_list` - List of private subnets for the ECS service
@@ -48,7 +48,7 @@ There are also these optional inputs:
 - `application_version` - The service's image version
 - `ui_version` - The static UI version
 - `hosted_zone_id` - If provided, will add an alias for the application load balancer to use the provided domain using that HZ. Otherwise, it will create a HZ and the alias
-- `certificate_validation_arn` - If provided, will link the certificate to the load-balancer https-listener. Otherwise, will create a new certificate and link it. ([managing certificates](/infrastructure/certificates/))
+- `certificate_validation_arn` - If provided, will link the certificate to the load-balancer https-listener. Otherwise, will create a new certificate and link it. ([managing certificates](./certificates.md))
 - `app-replica-count-desired` - if provided, will set the number of desired running instances for a service. Otherwise,
   it will default the count to 1
 - `app-replica-count-max` - if provided, will set the number of maximum running instances for a service. Otherwise, it
@@ -197,14 +197,14 @@ Required:
 - `application_version` - service's docker
   image version
 - `ui_version` - Static UI version
-- `domain_name` - application hostname ([can be a domain or a subdomain](/infrastructure/domains_subdomains/))
+- `domain_name` - application hostname ([can be a domain or a subdomain](./domains_subdomains.md))
 - `aws_account` - aws account id where the application will be hosted
 - `aws_region` - aws region where the application will be hosted
 - `iam_users` - IAM users to be created automatically, with roles to be attached to them
 - `manual_users` - IAM users that has been already created manually, with roles to be attached. (Can be left empty)
 - `set_iam_user_groups` - User groups that need to be present on each user. (i.e. if the value is set to admin, then all
   the users will require the admin role)
-- `support_emails_for_cloudwatch_alerts` - list of engineer emails that should receive alert notifications [more info](/infrastructure/alerting_monitoring/)
+- `support_emails_for_cloudwatch_alerts` - list of engineer emails that should receive alert notifications [more info](./alerting_monitoring.md)
 - `ip_whitelist` - ip range to add to application whitelist. The expected value is a list of strings.
 
 Optional:
@@ -213,7 +213,7 @@ Optional:
 - `hosted_zone_id` - if provided, will add an alias for the application load balancer to use the provided domain using
   that HZ. Otherwise, it will create a HZ and the alias
 - `certificate_validation_arn` - if provided, will link the certificate to the load-balancer https-listener. Otherwise,
-  will create a new certificate and link it. [managing certificates](/infrastructure/certificates/)
+  will create a new certificate and link it. [managing certificates](./certificates.md)
 - `tags` - if provided, it will tag the resources with the defined value. Otherwise, it will default to "Resource = '
   data-f1-rapid'"
 

diff --git a/infrastructure/blocks/app-cluster/output.tf b/infrastructure/blocks/app-cluster/output.tf
@@ -33,6 +33,11 @@ output "ecs_task_execution_role_arn" {
   description = "The ECS task execution role ARN"
 }
 
+output "ecs_task_execution_role_name" {
+  value       = module.app_cluster.ecs_task_execution_role_name
+  description = "The ECS task execution role name"
+}
+
 output "log_error_alarm_notification_arn" {
   value       = module.app_cluster.log_error_alarm_notification_arn
   description = "The arn of the sns topic that receives notifications on log error alerts"
@@ -47,3 +52,8 @@ output "service_table_arn" {
   value       = module.app_cluster.service_table_arn
   description = "The arn of the dynamoDB table that stores the user service"
 }
+
+output "aws_dynamodb_table_service_table_name" {
+  value       = module.app_cluster.aws_dynamodb_table_service_table_name
+  description = "Name of the DynammoDB table that contains schema data"
+}
diff --git a/infrastructure/blocks/auth/output.tf b/infrastructure/blocks/auth/output.tf
@@ -1,6 +1,6 @@
 output "user_pool_endpoint" {
   value       = module.auth.user_pool_endpoint
-  description = "The Cognito rapid user pool arn"
+  description = "The Cognito rapid user pool endpoint"
 }
 
 output "resource_server_scopes" {
@@ -18,6 +18,16 @@ output "cognito_user_pool_id" {
   description = "The Cognito rapid user pool id"
 }
 
+output "cognito_user_pool_arn" {
+  value       = module.auth.cognito_user_pool_arn
+  description = "The Cognito rapid user pool arn"
+}
+
+output "cognito_user_pool_domain" {
+  value       = module.auth.cognito_user_pool_domain
+  description = "The Cognito rapid user pool domain"
+}
+
 output "cognito_client_app_secret_manager_name" {
   value       = module.auth.cognito_client_app_secret_manager_name
   description = "Secret manager name where client app info is stored"

diff --git a/infrastructure/modules/app-cluster/load_balancer.tf b/infrastructure/modules/app-cluster/load_balancer.tf
@@ -68,45 +68,63 @@ resource "aws_security_group" "load_balancer_security_group_http" {
   # checkov:skip=CKV_AWS_260: Limits by prefix list ID's
   vpc_id      = var.vpc_id
   description = "ALB Security Group"
-  ingress {
-    from_port       = 80
-    to_port         = 80
-    protocol        = "tcp"
-    prefix_list_ids = [data.aws_ec2_managed_prefix_list.cloudwatch.id]
-    description     = "Allow HTTP ingress"
-  }
-  egress {
-    from_port        = 0
-    to_port          = 0
-    protocol         = "-1"
-    cidr_blocks      = ["0.0.0.0/0"]
-    ipv6_cidr_blocks = ["::/0"]
-    description      = "Allow all egress"
-  }
-  tags = var.tags
+  tags        = var.tags
 
   lifecycle {
     create_before_destroy = true
   }
 }
+
+resource "aws_security_group_rule" "load_balancer_security_group_rule_ingress_http" {
+  type              = "ingress"
+  from_port         = 80
+  to_port           = 80
+  protocol          = "tcp"
+  prefix_list_ids   = [data.aws_ec2_managed_prefix_list.cloudwatch.id]
+  security_group_id = aws_security_group.load_balancer_security_group_http.id
+}
+
+resource "aws_security_group_rule" "load_balancer_security_group_rule_egress_http" {
+  type              = "egress"
+  from_port         = 0
+  to_port           = 65535
+  protocol          = "tcp"
+  cidr_blocks       = ["0.0.0.0/0"]
+  ipv6_cidr_blocks  = ["::/0"]
+  security_group_id = aws_security_group.load_balancer_security_group_http.id
+}
+
+
 resource "aws_security_group" "load_balancer_security_group_https" {
   # checkov:skip=CKV_AWS_260: Limits by prefix list ID's
   vpc_id      = var.vpc_id
   description = "ALB Security Group"
-  ingress {
-    from_port       = 443
-    to_port         = 443
-    protocol        = "tcp"
-    prefix_list_ids = [data.aws_ec2_managed_prefix_list.cloudwatch.id]
-    description     = "Allow HTTPS ingress"
-  }
-  tags = var.tags
+  tags        = var.tags
 
   lifecycle {
     create_before_destroy = true
   }
 }
 
+resource "aws_security_group_rule" "load_balancer_security_group_rule_ingress_https" {
+  type              = "ingress"
+  from_port         = 443
+  to_port           = 443
+  protocol          = "tcp"
+  prefix_list_ids   = [data.aws_ec2_managed_prefix_list.cloudwatch.id]
+  security_group_id = aws_security_group.load_balancer_security_group_https.id
+}
+
+resource "aws_security_group_rule" "load_balancer_security_group_rule_egress_https" {
+  type              = "egress"
+  from_port         = 0
+  to_port           = 65535
+  protocol          = "tcp"
+  cidr_blocks       = ["0.0.0.0/0"]
+  ipv6_cidr_blocks  = ["::/0"]
+  security_group_id = aws_security_group.load_balancer_security_group_https.id
+}
+
 resource "aws_lb_target_group" "target_group" {
   name        = "${var.resource-name-prefix}-tg"
   port        = 80

diff --git a/infrastructure/modules/app-cluster/outputs.tf b/infrastructure/modules/app-cluster/outputs.tf
@@ -8,6 +8,11 @@ output "ecs_task_execution_role_arn" {
   description = "The ECS task execution role ARN"
 }
 
+output "ecs_task_execution_role_name" {
+  value       = aws_iam_role.ecsTaskExecutionRole.name
+  description = "The ECS task execution role name"
+}
+
 output "load_balancer_dns" {
   value       = aws_alb.application_load_balancer.dns_name
   description = "The DNS name of the load balancer"
@@ -50,3 +55,8 @@ output "service_table_arn" {
 output "application_version" {
   value = var.application_version
 }
+
+output "aws_dynamodb_table_service_table_name" {
+  value       = aws_dynamodb_table.service_table.name
+  description = "Name of the DynammoDB table that contains schema data"
+}
diff --git a/infrastructure/modules/auth/output.tf b/infrastructure/modules/auth/output.tf
@@ -18,6 +18,16 @@ output "cognito_user_pool_id" {
   description = "The Cognito rapid user pool id"
 }
 
+output "cognito_user_pool_arn" {
+  value       = aws_cognito_user_pool.rapid_user_pool.arn
+  description = "The Cognito rapid user pool arn"
+}
+
+output "cognito_user_pool_domain" {
+  value       = aws_cognito_user_pool.rapid_user_pool.domain
+  description = "The Cognito rapid user pool domain"
+}
+
 output "cognito_client_app_secret_manager_name" {
   value       = aws_secretsmanager_secret.client_secrets_cognito.name
   description = "Secret manager name where client app info is stored"

diff --git a/sdk/rapid/exceptions.py b/sdk/rapid/exceptions.py
@@ -92,3 +92,9 @@ class InvalidDomainNameException(Exception):
 
 class DomainConflictException(Exception):
     pass
+
+class ClientDoesNotHaveUserAdminPermissionsException(Exception):
+    pass
+
+class ClientDoesNotHaveDataAdminPermissionsException(Exception):
+    pass