Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merge prod to main #78

Open
wants to merge 97 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
97 commits
Select commit Hold shift + click to select a range
fb607fc
2303132232
indiciumx Mar 14, 2023
8dba48f
This commit adds a cloudfront function that is run
Mar 22, 2023
5e05bb9
Merge pull request #64 from Canadian-Geospatial-Platform/feature/clou…
PVautour Mar 22, 2023
f0ed385
Updates ETL lambdas to conditionally support multiple versions betwee…
bo-lu Mar 25, 2023
ee743b1
Restore collections lambda, renames to lambda with datestamp
bo-lu Mar 27, 2023
670676d
Restores STAC on dev. The lambda now does both harvest and transforma…
bo-lu Mar 27, 2023
f482af0
Minor fix for stac harvest and transform
bo-lu Mar 27, 2023
a87075e
Update geocore-to-parquet MemorySize to 4096MB
bo-lu Mar 27, 2023
2910908
Update environment variables for GeocoretoParquet lambda
bo-lu Mar 27, 2023
90e9135
New popularity lambda and update to geo lambda
bo-lu Mar 28, 2023
94518d8
Versioned lambdas between environments
bo-lu Mar 29, 2023
b286567
Use AWS managed lambda layer instead of including in zip package
bo-lu Mar 29, 2023
85d3795
Minor fixes to popularity lambda
bo-lu Mar 29, 2023
d00770f
Small typo in prod featured lambda
bo-lu Mar 30, 2023
dbe1ebf
Issue with parquet lambda, use AWS Lambda Layer instead
bo-lu Mar 30, 2023
73da4e0
Update parquet lambda and give dynamodb permission
bo-lu Mar 30, 2023
c9aa8cb
Change to API integration request
bo-lu Mar 30, 2023
e45186b
Proposed change to Athena table schema
bo-lu Mar 31, 2023
6a6f2e9
This commit adds a cloudfront function that is run
Mar 22, 2023
630216b
add handler function to redirect www. to non www. url for app.geo.ca
Apr 4, 2023
ee05a45
bug fix: add code to preserve request querystring in redirect.
Apr 4, 2023
d104758
bug fix: code was added to preserve request querystring in redirect.
Apr 4, 2023
37b9790
refactor: remove unused variable.
Apr 4, 2023
11f9eee
Merge pull request #68 from Canadian-Geospatial-Platform/feature/clou…
PVautour Apr 4, 2023
56e61aa
Merge branch 'dev' into feature/cloudfront-function/app-handler
PVautour Apr 4, 2023
4631c60
Merge pull request #67 from Canadian-Geospatial-Platform/feature/clou…
PVautour Apr 4, 2023
6611b6f
Enable Compress objects automatically
Apr 13, 2023
3f09c76
Merge pull request #70 from jaredkinger/dev
PVautour Apr 13, 2023
dcb2dfd
Merge pull request #72 from Canadian-Geospatial-Platform/dev-athena-fix
bo-lu Apr 18, 2023
35a3061
Adds workflow_dispatch
bo-lu Apr 18, 2023
498ce1b
Adds a custom ResponseHeadersPolicy for cache-control
Apr 20, 2023
fed2276
Merge pull request #73 from jaredkinger/dev
bo-lu Apr 27, 2023
98c50f2
Update STAC harvest and transform for deploying in staging/prod
xinli-cai Jun 6, 2023
1604e7d
Changes to CF for June release
bo-lu Jun 12, 2023
e07f7a1
Fixes to geo lambda when searching for themes
bo-lu Jun 12, 2023
966113f
Lambdas and CF fixes
bo-lu Jun 14, 2023
e1f3077
Fixes to geocore stack and addition of new stac lambda zip
bo-lu Jun 14, 2023
e9b250b
2306212159
indiciumx Jun 22, 2023
8cd3125
Merge branch 'dev' of https://github.com/Canadian-Geospatial-Platform…
indiciumx Jun 22, 2023
a283078
3006231121
indiciumx Jun 30, 2023
380147e
Update to harvester lambda to use lambda layers
bo-lu Jul 12, 2023
8570d01
Fixes to geocore-stack and reinstatement of id lambda as v1
bo-lu Aug 18, 2023
d300b0e
Commenting out unfinished work
bo-lu Aug 23, 2023
bac8b08
Changes needed to id lambda for similarity engine
bo-lu Aug 23, 2023
02cc007
Update parquet lambda for similarity engine and enable multiprocessing
xinli-cai Aug 23, 2023
4a5e112
Alter similarity engine CF commit to use AWS::AccountId intrinsic par…
bo-lu Aug 25, 2023
b9b5097
CloudFormation added to deploy TiTiler as a Lambda Layer and cofigura…
bo-lu Sep 20, 2023
34ee669
Merge pull request #77 from Canadian-Geospatial-Platform/cf-titiler-b…
bo-lu Sep 20, 2023
61f970c
Add reference to TiTiler in the common stack
bo-lu Sep 22, 2023
a690f98
Fixes to TiTiler stack, execution role, endpoint and api paths
bo-lu Sep 26, 2023
425bf8e
Patch release for v1.1.2
bo-lu Sep 26, 2023
d1bb905
Add CloudFormation to deploy STAC harvester as lambda functions
xinli-cai Dec 18, 2023
f2a470d
Add pagination in STAC harvester lambda
xinli-cai Dec 18, 2023
9224572
#131 Fix regression on the search lambda related to searching for mul…
bo-lu Dec 19, 2023
c835636
Initial release for GET responses for GCS and RCS configurations
bo-lu Dec 19, 2023
cd91086
Add CF template for EO sentinel1 harvestor stack
xinli-cai Jan 18, 2024
29dcb0d
Add 4 deployment zip for eo sentinel1 harvester
xinli-cai Jan 18, 2024
67f27ac
Merge branch 'dev' of https://github.com/Canadian-Geospatial-Platform…
bo-lu Jan 19, 2024
405afe5
Add EO properties in GeoCore format template
xinli-cai Feb 1, 2024
28bdeb6
update geolocator code.
Feb 7, 2024
2c3fa75
add readme to assist with geolocator deployment.
Feb 7, 2024
8307358
add note about deploying geolocator bucket content along side new code.
Feb 8, 2024
a7c0d5f
Add new eofilters in CloudFormation and rename eo stack harvester
bo-lu Feb 26, 2024
294567b
Alter table for eofilters in Athena
bo-lu Feb 26, 2024
85fce37
updated api gateway for adding eofilters params
Mar 7, 2024
ebf555e
documented new columns for metadata table in athena named query
Mar 8, 2024
f483be5
updated lambda geo zip filename for dev
Mar 15, 2024
0d2243e
update geolocator to commit id 86a61bcfc2be8a628f41c2ead2002d3a7e9fc709.
Mar 21, 2024
2636d93
add s3 write policy to geolocator.
Mar 25, 2024
5c9cf64
Externalize PARQUET_FILENAME as environment variable in hnap-harvest
bo-lu Jun 10, 2024
4b2f25a
New caching strategy for the geocore stack
bo-lu Jun 12, 2024
c8e5a6f
Use GITHUB_TOKEN instead of PAT
bo-lu Jun 12, 2024
0cc7901
Set maximum in collections lambda to prevent exceeding API Gateway si…
bo-lu Jun 13, 2024
864254e
Initial release for harvesting EODMS sentinel-1 collection to geo.ca
xinli-cai Jun 19, 2024
6ed36e5
Add reference to Semantic Search API in the common stack
xinli-cai Jun 26, 2024
5645061
Initial release for OpenSearch and Semantic Search API endpoint
xinli-cai Jun 26, 2024
46eb096
Add image processor lambda and API endpoint to dynamically flip senti…
bo-lu Jun 28, 2024
47332f1
Update Athena schema and update geo lambda
bo-lu Jun 28, 2024
ffd1ad4
Update STAC harvester lambda for including sourceSystemname used in E…
xinli-cai Jun 28, 2024
97fab0c
update geolocator to commit 15b0de5722c2362d49cdfdbdfd2cb9f97c94059e.
PVautour Sep 5, 2024
1da42f3
update geolocator.
PVautour Sep 11, 2024
7f749e9
Update stage-deployment.yml
bo-lu Sep 11, 2024
cac314b
Moves image processor from the geocore stack to the tiler stack
bo-lu Sep 17, 2024
a48aef6
Deploys v1.4.4 which modifies where image-processor resides and add P…
bo-lu Sep 24, 2024
59afa3f
Merge pull request #87 from Canadian-Geospatial-Platform/stage
bo-lu Sep 24, 2024
90678b1
Initial commit for RCM STAC harvester
bo-lu Oct 15, 2024
b428d43
Bug fix so Rule is added to CollectorFunction in PermissionForEventsT…
bo-lu Oct 15, 2024
3a84ba3
Changes to DynamoDB tables, S3 bucket logging and S3 creation logic
bo-lu Oct 22, 2024
2b0af65
Merge pull request #88 from Canadian-Geospatial-Platform/stage
bo-lu Oct 22, 2024
2457ace
Initial commit for RCM-ARD EO Harvester
bo-lu Oct 22, 2024
482275c
Merge pull request #89 from Canadian-Geospatial-Platform/dev
bo-lu Oct 22, 2024
98ea404
Initial upload of rcm-ard lambda functions
bo-lu Oct 23, 2024
d6de467
Typo in condition
bo-lu Oct 24, 2024
4f62cd1
Syntax error in table name of ProcessTable
bo-lu Oct 24, 2024
c2d827c
Merge pull request #90 from Canadian-Geospatial-Platform/dev
bo-lu Oct 24, 2024
2510d85
Fix to replace EOItemLinksS3Bucket name since it may not be created i…
bo-lu Oct 29, 2024
8791e36
Merge pull request #91 from Canadian-Geospatial-Platform/dev
bo-lu Oct 29, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 5 additions & 3 deletions .github/workflows/dev-deployment.yml
Original file line number Diff line number Diff line change
@@ -1,8 +1,10 @@
name: Find and Replace Dev Deployment
name: Push changes from Dev to Stage

on:
push:
branches: [ dev ]
# Allows you to run this workflow manually from the Actions tab
workflow_dispatch:

jobs:
build:
Expand All @@ -13,8 +15,8 @@ jobs:
- name: Initiate - Dev
uses: actions/checkout@v3
with:
token: ${{ secrets.PAT }}
token: ${{ secrets.GITHUB_TOKEN }}
ref: ${{ github.head_ref }}

- name: Push changes - Init Stage
run: git push origin -f dev:stage
run: git push origin -f dev:stage
8 changes: 5 additions & 3 deletions .github/workflows/stage-deployment.yml
Original file line number Diff line number Diff line change
@@ -1,8 +1,10 @@
name: Find and Replace Stage Deployment
name: Push changes from Stage to Prod

on:
push:
branches: [ stage ]
# Allows you to run this workflow manually from the Actions tab
workflow_dispatch:

jobs:
build:
Expand All @@ -13,8 +15,8 @@ jobs:
- name: Initiate - Stage
uses: actions/checkout@v3
with:
token: ${{ secrets.PAT }}
token: ${{ secrets.GITHUB_TOKEN }}
ref: ${{ github.head_ref }}

- name: Push changes - Init Prod
run: git push origin -f stage:prod
run: git push origin -f stage:prod
42 changes: 38 additions & 4 deletions docs/cloudformation/app-geo-ca-stack.yml
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ Parameters:
SslCertArn:
Type: AWS::SSM::Parameter::Value<String>
Default: /webpresence/app-geo-ca/ssl-cert-arn
Description: SSM parameter name for app.geo.ca ACM SSL Cert ARN
Description: SSM parameter name for app.geo.ca ACM SSL Cert ARN
WebAclArn:
Type: String
Description: ARN of the WAF web ACL to use for CloudFront
Expand Down Expand Up @@ -100,7 +100,7 @@ Resources:
- 'logs:CreateLogGroup'
- 'logs:CreateLogStream'
- 'logs:PutLogEvents'
Resource: '*'
Resource: '*'


AppCodeBuildProject:
Expand Down Expand Up @@ -313,7 +313,7 @@ Resources:
- Name: AppBuild
RunOrder: 1
Configuration:
ProjectName: !Ref AppCodeBuildProject
ProjectName: !Ref AppCodeBuildProject
- Name: Deploy
Actions:
- Name: Deploy
Expand Down Expand Up @@ -371,6 +371,9 @@ Resources:
LambdaFunctionAssociations:
- EventType: origin-response
LambdaFunctionARN: !Ref SecHeadersLambdaEdgeArn
FunctionAssociations:
- EventType: viewer-request
FunctionARN: !GetAtt CloudfrontAppHandler.FunctionMetadata.FunctionARN
DefaultRootObject: index.html
HttpVersion: http2
WebACLId: !Ref WebAclArn
Expand Down Expand Up @@ -398,4 +401,35 @@ Resources:
ResponsePagePath: /index.html
ResponseCode: 200


CloudfrontAppHandler:
Type: AWS::CloudFront::Function
Properties:
AutoPublish: true
FunctionCode: >
function handler(event) {
var request = event.request;
// Redirects requests to the www. url to the non www. url.
if (request.headers && request.headers.host && request.headers.host.value.includes('www.')) {
var newurl = request.headers.host.value.replace('www.', '')
var qs = ''
Object.keys(request.querystring).forEach((e) => {
if (qs === '') {
qs += '?';
} else { qs += '&' };
qs += e + '=' + request.querystring[e].value;
})
var response = {
statusCode: 301,
statusDescription: 'Moved Permanently',
headers:
{ "location": { "value": 'https://' + newurl + request.uri + qs } }
}
return response;
}

return request;
}
FunctionConfig:
Comment: A function run on every request to app.geo.ca
Runtime: cloudfront-js-1.0
Name: app-handler
70 changes: 69 additions & 1 deletion docs/cloudformation/common-stack.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,8 +12,12 @@ Metadata:
- GeocoreSslCertArn
- GeocoreMetadataSslCertArn
- GeocoreHarvestSslCertArn
- GeocorePygeoapiSslCertArn
- GeolocatorSslCertArn
- TiTilerSslCertArn
- GeoCaSslCertArn
- AppGeoCaSslCertArn
- SearchSslCertArn
- GithubUsername
- GithubKey

Expand All @@ -26,6 +30,13 @@ Parameters:
- prod
Default: dev
Description: Type of App Environment - dev stage prod
DynamoDBCreationBool:
Type: String
AllowedValues:
- true
- false
Default: false
Description: True or false to recreate all dynamodb tables
DeploymentBucket:
Description: S3 bucket where all the deployment related files are
Type: String
Expand All @@ -38,20 +49,36 @@ Parameters:
Type: String
GeocoreHarvestSslCertArn:
Description: ARN of the SSL cert stored in ACM to be used for Geocore HNAP JSON Harvester
Type: String
Type: String
GeocorePygeoapiSslCertArn:
Description: ARN of the SSL cert stored in ACM to be used for Geocore pygeoapi
Type: String
GeolocatorSslCertArn:
Description: ARN of the SSL cert stored in ACM to be used for Geolocator
Type: String
TiTilerSslCertArn:
Description: ARN of the SSL cert stored in ACM to be used for TiTiler
Type: String
GeoCaSslCertArn:
Description: ARN of the SSL cert stored in ACM to be used for geo.ca
Type: String
AppGeoCaSslCertArn:
Description: ARN of the SSL cert stored in ACM to be used for app.geo.ca
Type: String
SearchSslCertArn:
Description: ARN of the SSL cert stored in ACM to be used for search-recherche.geocore.api.geo.ca
Type: String
GithubUsername:
Description: Github Username for accessing Canadian Geospatial Platform
Type: String
GithubKey:
Description: Github Key for accessing Canadian Geospatial Platform
Type: String

Conditions:
IsProd: !Equals [prod, !Ref Environment]
IsStage: !Equals [stage, !Ref Environment]
IsDev: !Equals [dev, !Ref Environment]

Resources:

Expand Down Expand Up @@ -145,13 +172,46 @@ Resources:
Type: String
Value: !Ref Environment

DynamoDBCreationParam:
Type: AWS::SSM::Parameter
Properties:
Description: True or false to recreate all dynamodb tables
Name: /webpresence/dynamodb_creation
Type: String
Value: !Ref DynamoDBCreationBool

GeocoreHarvestSslCertArnParam:
Type: AWS::SSM::Parameter
Properties:
Description: ARN of the SSL cert stored in ACM to be used for Geocore HNAP JSON harvester
Name: /webpresence/geocore-hnap-harvest/ssl-cert-arn
Type: String
Value: !Ref GeocoreHarvestSslCertArn

GeocorePygeoapiSslCertArnParam:
Condition: IsDev
Type: AWS::SSM::Parameter
Properties:
Description: ARN of the SSL cert stored in ACM to be used for Geocore pygeoapi
Name: /webpresence/geocore-pygeoapi/ssl-cert-arn
Type: String
Value: !Ref GeocorePygeoapiSslCertArn

GeolocatorSslCertArnParam:
Type: AWS::SSM::Parameter
Properties:
Description: ARN of the SSL cert stored in ACM to be used for Geolocator
Name: /webpresence/geolocator/ssl-cert-arn
Type: String
Value: !Ref GeolocatorSslCertArn

TiTilerSslCertArnParam:
Type: AWS::SSM::Parameter
Properties:
Description: ARN of the SSL cert stored in ACM to be used for TiTiler
Name: /webpresence/titiler/ssl-cert-arn
Type: String
Value: !Ref TiTilerSslCertArn

GeocoreSslCertArnParam:
Type: AWS::SSM::Parameter
Expand Down Expand Up @@ -185,6 +245,14 @@ Resources:
Type: String
Value: !Ref AppGeoCaSslCertArn

SearchSslCertArnParam:
Type: AWS::SSM::Parameter
Properties:
Description: ARN of the SSL cert stored in ACM to be used for search-recherche.geocore.api.geo.ca
Name: /webpresence/search-geocore-api-geo-ca/ssl-cert-arn
Type: String
Value: !Ref SearchSslCertArn

GithubUsernameParam:
Type: AWS::SecretsManager::Secret
Properties:
Expand Down
61 changes: 60 additions & 1 deletion docs/cloudformation/geo-ca-stack.yml
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,18 @@ Resources:
Action: s3:GetObject
Resource: !Sub 'arn:aws:s3:::webpresence-geo-ca-${Environment}/*'

GeoCaResponseHeadersPolicy:
Type: AWS::CloudFront::ResponseHeadersPolicy
Properties:
ResponseHeadersPolicyConfig:
Name: geoca-response-headers-policy
Comment: Add custom response header to set cache-control
CustomHeadersConfig:
Items:
- Header: cache-control
Value: public, max-age=31536000
Override: true

AppPipelineRole:
Type: AWS::IAM::Role
Properties:
Expand Down Expand Up @@ -102,7 +114,7 @@ Resources:
Resource:
- Fn::Sub: arn:aws:lambda:*


LambdaExecutionRole:
Type: AWS::IAM::Role
Properties:
Expand Down Expand Up @@ -273,12 +285,17 @@ Resources:
OriginAccessIdentity: !Join ['', ['origin-access-identity/cloudfront/', !Ref GeoCaOai]]
Enabled: 'true'
DefaultCacheBehavior:
Compress: true
ViewerProtocolPolicy: redirect-to-https
TargetOriginId: !Ref GeoCaStoreBucket
CachePolicyId: 658327ea-f89d-4fab-a63d-7e88639e58f6 # https://docs.amazonaws.cn/en_us/AmazonCloudFront/latest/DeveloperGuide/using-managed-cache-policies.html
ResponseHeadersPolicyId: !Ref GeoCaResponseHeadersPolicy
LambdaFunctionAssociations:
- EventType: origin-response
LambdaFunctionARN: !Ref SecHeadersLambdaEdgeArn
FunctionAssociations:
- EventType: viewer-request
FunctionARN: !GetAtt CloudfrontRootHandler.FunctionMetadata.FunctionARN
DefaultRootObject: index.html
HttpVersion: http2
WebACLId: !Ref WebAclArn
Expand All @@ -305,3 +322,45 @@ Resources:
ErrorCode: 404
ResponsePagePath: /404.html
ResponseCode: 404

CloudfrontRootHandler:
Type: AWS::CloudFront::Function
Properties:
AutoPublish: true
FunctionCode: >
function handler(event) {
var request = event.request;
var uri = request.uri;

// Check whether the URI is missing a file name.
if (uri.endsWith('/')) {
request.uri += 'index.html';
}
// Check whether the URI is missing a file extension.
else if (!uri.includes('.')) {
request.uri += '/index.html';
}
// Redirects requests to the www. url to the non www. url.
if (request.headers && request.headers.host && request.headers.host.value.includes('www.')) {
var newurl = request.headers.host.value.replace('www.', '')
var qs = ''
Object.keys(request.querystring).forEach((e) => {
if (qs === '') {
qs += '?';
} else { qs += '&' };
qs += e + '=' + request.querystring[e].value;
})
var response = {
statusCode: 301,
statusDescription: 'Moved Permanently',
headers:
{ "location": { "value": 'https://' + newurl + request.uri + qs } }
}
return response;
}
return request;
}
FunctionConfig:
Comment: A function run on every request to geo.ca
Runtime: cloudfront-js-1.0
Name: root-handler
Loading
Loading