You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Ingest subset of roughly 100-200 WV03_MSI_L2A granules into UAT to allow verification of metadata and metrics.
ACCOUNT: [csda-cumulus-uat-1686]
To find a date range that has roughly 100-200 granules, make use of AWS Athena by doing the following:
In the MCP IMPACT-CSDAP AWS account's console, open the Athena query editor
Select the AwsDataCatalog Data Source and the default Database
Run the following query to update the partitions: MSCK REPAIR TABLE maxar_transfer
Run queries similar to the following (tweaking dt and the path pattern -- see notes below) until a count near the 100-200 range is obtained:
SELECT * FROM "maxar_transfer$partitions" ORDER BY dt DESC
SELECT * FROM "maxar_transfer" WHERE dt='2024-05-17-01-00' AND key like 'css/nga/WV03/2A/2009/%P1BS%' AND key like 'css/nga/WV03/2A/2009/%BROWSE.jpg%' limit 100;
This approach uses the count of *BROWSE.jpg files as a proxy for the number of granules in the date range.
NOTE: the value of dt in the query above must be set to the most recent date of the S3 inventory (likely the current date, but the day prior to the current date should always work since the most recent 3 days are retained).
NOTE: days like 2009/321 and 2009/339 (because the end date is exclusive in the rule definition) must be converted to yyyy/MM/DD for determining ISO 8601 datetime values for the start and end dates for the sample rule. This site provides conversion: https://dqydj.com/day-of-the-year-calculator/
2009/321 is 2009/11/17T00:00:00Z
2009/339 is 2009/12/05T00:00:00Z
Acceptance criteria:
define a rule named WV03_MSI_L2A___1_UAT.json that spans a period of time that includes roughly 100-200 granules (using instructions above for finding appropriate values for startDate and endDate)
load the rule into UAT
DOTENV=.env.uat make bash
cumulus rules add --data app/stacks/cumulus/resources/rules/WV02_Pan_L1B/v1/WV03_MSI_L2A___1_UAT.json
trigger the rule to ingest the granules
DOTENV=.env.uat make bash
cumulus rules enable --name WV03_MSI_L2A___1_UAT
cumulus rules run --name WV03_MSI_L2A___1_UAT
Ingest subset of roughly 100-200 WV03_MSI_L2A granules into UAT to allow verification of metadata and metrics.
ACCOUNT: [csda-cumulus-uat-1686]
To find a date range that has roughly 100-200 granules, make use of AWS Athena by doing the following:
MSCK REPAIR TABLE maxar_transfer
dt
and the path pattern -- see notes below) until a count near the 100-200 range is obtained:This approach uses the count of
*BROWSE.jpg
files as a proxy for the number of granules in the date range.NOTE: the value of
dt
in the query above must be set to the most recent date of the S3 inventory (likely the current date, but the day prior to the current date should always work since the most recent 3 days are retained).NOTE: days like 2009/321 and 2009/339 (because the end date is exclusive in the rule definition) must be converted to
yyyy/MM/DD
for determining ISO 8601 datetime values for the start and end dates for the sample rule. This site provides conversion: https://dqydj.com/day-of-the-year-calculator/2009/11/17T00:00:00Z
2009/12/05T00:00:00Z
Acceptance criteria:
WV03_MSI_L2A___1_UAT.json
that spans a period of time that includes roughly 100-200 granules (using instructions above for finding appropriate values forstartDate
andendDate
)q
should be collection short name)The text was updated successfully, but these errors were encountered: