Fix: S3 server access logs are inherently incomplete (#5043)

[dsotirho-ucsc]

Connected issues: #5043

Checklist

Author

• PR is a draft
• Target branch is develop
• Name of PR branch matches issues/<GitHub handle of author>/<issue#>-<slug>
• PR title references all connected issues
• PR title matches¹ that of a connected issue _{or comment in PR explains why they're different}
• For each connected issue, there is at least one commit whose title references that issue
• PR is connected to all connected issues via ZenHub
• PR description links to connected issues
• Added partial label to PR _{or this PR completely resolves all connected issues}

¹ when the issue title describes a problem, the corresponding PR
title is Fix: followed by the issue title

Author (reindex, API changes)

• Added r tag to commit title _{or this PR does not require reindexing}
• Added reindex label to PR _{or this PR does not require reindexing}
• Added a (compatible changes) or A (incompatible ones) tag to commit title _{or this PR does not modify the Azul service API}
• Added API label to connected issues _{or this PR does not modify the Azul service API}

Author (chains)

• This PR is blocked by previous PR in the chain _{or this PR is not chained to another PR}
• Added base label to the blocking PR _{or this PR is not chained to another PR}
• Added chained label to this PR _{or this PR is not chained to another PR}

Author (upgrading)

• Documented upgrading of deployments in UPGRADING.rst _{or this PR does not require upgrading}
• Added u tag to commit title _{or this PR does not require upgrading}
• Added upgrade label to PR _{or this PR does not require upgrading}

Author (operator tasks)

• Added checklist items for additional operator tasks _{or this PR does not require additional tasks}

Author (hotfixes)

• Added F tag to main commit title _{or this PR does not include permanent fix for a temporary hotfix}
• Reverted the temporary hotfixes for any connected issues _{or the prod branch has no temporary hotfixes for any connected issues}

Author (before every review)

• Rebased PR branch on develop, squashed old fixups
• Ran make requirements_update _{or this PR does not touch requirements*.txt, common.mk, Makefile and Dockerfile}
• Added R tag to commit title _{or this PR does not touch requirements*.txt}
• Added reqs label to PR _{or this PR does not touch requirements*.txt}
• make integration_test passes in personal deployment _{or this PR does not touch functionality that could break the IT}

Peer reviewer (after requesting changes)

Uncheck the Author (before every review) checklists.

Peer reviewer (after approval)

• PR is not a draft
• Ticket is in Review requested column
• Requested review from primary reviewer
• Assigned PR to primary reviewer

Primary reviewer (after requesting changes)

Uncheck the before every review checklists. Update the N reviews label.

Primary reviewer (after approval)

• Actually approved the PR
• Labeled connected issues as demo or no demo
• Commented on connected issues about demo expectations _{or all connected issues are labeled no demo}
• Decided if PR can be labeled no sandbox
• PR title is appropriate as title of merge commit
• N reviews label is accurate
• Moved ticket to Approved column
• Assigned PR to current operator

Operator (before pushing merge the commit)

• Checked reindex label and r commit title tag
• Checked that demo expectations are clear _{or all connected issues are labeled no demo}
• PR has checklist items for upgrading instructions _{or PR is not labeled upgrade}
• Squashed PR branch and rebased onto develop
• Sanity-checked history
• Pushed PR branch to GitHub
• Pushed PR branch to GitLab dev and added sandbox label _{or PR is labeled no sandbox}
• Pushed PR branch to GitLab anvildev _{or PR is labeled no sandbox}
• Build passes in sandbox deployment _{or PR is labeled no sandbox}
• Build passes in anvilbox deployment _{or PR is labeled no sandbox}
• Reviewed build logs for anomalies in sandbox deployment _{or PR is labeled no sandbox}
• Reviewed build logs for anomalies in anvilbox deployment _{or PR is labeled no sandbox}
• Deleted unreferenced indices in sandbox _{or this PR does not remove catalogs or otherwise causes unreferenced indices}
• Deleted unreferenced indices in anvilbox _{or this PR does not remove catalogs or otherwise causes unreferenced indices}
• Started reindex in sandbox _{or this PR does not require reindexing sandbox}
• Started reindex in anvilbox _{or this PR does not require reindexing sandbox}
• Checked for failures in sandbox _{or this PR does not require reindexing sandbox}
• Checked for failures in anvilbox _{or this PR does not require reindexing sandbox}
• Added PR reference to merge commit title
• Collected commit title tags in merge commit title
• Moved connected issues to Merged column in ZenHub
• Pushed merge commit to GitHub

Operator (after pushing the merge commit)

• Shortened the PR chain _{or this PR is not labeled base}
• Manually deployed dev.shared
• Pushed merge commit to GitLab dev _{or PR is labeled no sandbox}
• Manually deployed anvildev.shared
• Pushed merge commit to GitLab anvildev _{or PR is labeled no sandbox}
• Build passes on GitLab dev¹
• Reviewed build logs for anomalies on GitLab dev¹
• Build passes on GitLab anvildev¹
• Reviewed build logs for anomalies on GitLab anvildev¹
• Deleted PR branch from GitHub
• Deleted PR branch from GitLab dev
• Deleted PR branch from GitLab anvildev

¹ When pushing the merge commit is skipped due to the PR being
labelled no sandbox, the next build triggered by a PR whose merge commit is
pushed determines this checklist item.

Operator (reindex)

• Deleted unreferenced indices in dev _{or this PR does not remove catalogs or otherwise causes unreferenced indices}
• Deleted unreferenced indices in anvildev _{or this PR does not remove catalogs or otherwise causes unreferenced indices}
• Started reindex in dev _{or this PR does not require reindexing}
• Started reindex in anvildev _{or this PR does not require reindexing}
• Checked for and triaged indexing failures in dev _{or this PR does not require reindexing}
• Checked for and triaged indexing failures in anvildev _{or this PR does not require reindexing}
• Emptied fail queues in dev deployment _{or this PR does not require reindexing}
• Emptied fail queues in anvildev deployment _{or this PR does not require reindexing}

Operator

• Added checklist item to prod promotion PR:

• Manually deployed prod.shared

• Unassigned PR

Shorthand for review comments

• L line is too long
• W line wrapping is wrong
• Q bad quotes
• F other formatting problem

[coveralls]

Changes Unknown when pulling 4d89bf4 on issues/dsotirho-ucsc/5043-s3-data-event-logging into ** on develop**.

[codecov]

Codecov Report

Merging #5230 (4d89bf4) into develop (b954f7d) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff            @@
##           develop    #5230   +/-   ##
========================================
  Coverage    84.35%   84.35%           
========================================
  Files          148      148           
  Lines        18278    18278           
========================================
  Hits         15418    15418           
  Misses        2860     2860           

[dsotirho-ucsc]

Test deployed PR to dev (shared).
Confirmed configuration of the azul-trail-dev trail for logging of S3 data events:

S3 access log captured in trail:

[
    {
        "@timestamp": "2023-05-23 00:21:14.787",
        "@message": {
            "eventVersion": "1.08",
            "userIdentity": {
                "type": "AssumedRole",
                "principalId": "AROAJVOACJ4NC6UKRSSIS:[email protected]",
                "arn": "arn:aws:sts::122796619775:assumed-role/developer/[email protected]",
                "accountId": "122796619775",
                "accessKeyId": "ASIARZFZ7W77ZU6T5HHJ",
                "sessionContext": {
                    "sessionIssuer": {
                        "type": "Role",
                        "principalId": "AROAJVOACJ4NC6UKRSSIS",
                        "arn": "arn:aws:iam::122796619775:role/developer",
                        "accountId": "122796619775",
                        "userName": "developer"
                    },
                    "attributes": {
                        "creationDate": "2023-05-22T18:31:23Z",
                        "mfaAuthenticated": "true"
                    }
                }
            },
            "eventTime": "2023-05-23T00:19:40Z",
            "eventSource": "s3.amazonaws.com",
            "eventName": "GetObject",
            "awsRegion": "us-east-1",
            "sourceIPAddress": "73.222.193.39",
            "userAgent": "[Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36]",
            "requestParameters": {
                "X-Amz-Date": "20230523T001939Z",
                "bucketName": "data-browser.dev.lungmap.net",
                "X-Amz-Algorithm": "AWS4-HMAC-SHA256",
                "response-content-disposition": "attachment",
                "X-Amz-SignedHeaders": "host",
                "Host": "s3.us-east-1.amazonaws.com",
                "X-Amz-Expires": "300",
                "key": "~partytown/partytown.js"
            },
            "responseElements": null,
            "additionalEventData": {
                "SignatureVersion": "SigV4",
                "CipherSuite": "ECDHE-RSA-AES128-GCM-SHA256",
                "bytesTransferredIn": 0,
                "AuthenticationMethod": "QueryString",
                "x-amz-id-2": "iNZrWAyCxVnsnOG1wjz2ELcObuUaI74kwRAucrA0lZ4xBQuAXoZkdT2yi+wcDB0tx98IRtuy4Cw=",
                "bytesTransferredOut": 1384
            },
            "requestID": "MNW0TMJSP7D4ZY6F",
            "eventID": "2768e477-5f07-485e-b40c-7122daff6d8c",
            "readOnly": true,
            "resources": [
                {
                    "type": "AWS::S3::Object",
                    "ARN": "arn:aws:s3:::data-browser.dev.lungmap.net/~partytown/partytown.js"
                },
                {
                    "accountId": "122796619775",
                    "type": "AWS::S3::Bucket",
                    "ARN": "arn:aws:s3:::data-browser.dev.lungmap.net"
                }
            ],
            "eventType": "AwsApiCall",
            "managementEvent": false,
            "recipientAccountId": "122796619775",
            "eventCategory": "Data",
            "tlsDetails": {
                "tlsVersion": "TLSv1.2",
                "cipherSuite": "ECDHE-RSA-AES128-GCM-SHA256",
                "clientProvidedHostHeader": "s3.us-east-1.amazonaws.com"
            }
        },
        "errorCode": "",
        "errorMessage": "",
        "eventSource": "s3.amazonaws.com",
        "userIdentity.principalId": "AROAJVOACJ4NC6UKRSSIS:[email protected]"
    }
]

[achave11-ucsc]

Changes look good, base branch is outdated.