Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

crawler build fails because azcopy sync returns a 403 #3875

Closed
NotMyFault opened this issue Dec 22, 2023 · 19 comments
Closed

crawler build fails because azcopy sync returns a 403 #3875

NotMyFault opened this issue Dec 22, 2023 · 19 comments
Assignees
@dduportal @smerle33
Labels
trusted.ci.jenkins.io
Milestone
infra-team-sync-2...

Comments

@NotMyFault
Copy link
Member

Service(s)

trusted.ci.jenkins.io

Summary

The crawler job is failing again, this time azcopy sync yields a 403, causing the job to fail.

Reproduction steps

No response
@NotMyFault NotMyFault added the triage Incoming issues that need review label Dec 22, 2023
@dduportal dduportal self-assigned this Dec 23, 2023
@dduportal dduportal removed the triage Incoming issues that need review label Dec 23, 2023
@dduportal dduportal added this to the infra-team-sync-2024-01-02 milestone Dec 23, 2023
@dduportal
Copy link
Contributor

Audit:

Checking...

dduportal added a commit to jenkins-infra/crawler that referenced this issue Dec 23, 2023
@dduportal
hotfix(Jenkinsfile) disable azcopy temporarily as credential is not v…
d0efde1 
…alid

Ref. jenkins-infra/helpdesk#3875
@dduportal
Copy link
Contributor

dduportal commented Dec 23, 2023 •
edited
Loading

As I don't know what @lemeurherve did in #3414 except the linked terraform PRs (see #3875 (comment))

As we're in holidays for the next week, I've commented out the azcopy command jenkins-infra/crawler@d0efde1 and a build is currently running (I'll watch it).~

We'll work on this in January then

@dduportal
Copy link
Contributor

And now

$ aws s3 sync ./updates/ s3://westeurope-updates-jenkins-io/updates/ --no-progress --no-follow-symlinks --size-only --exclude .svn --endpoint-url https://8d1838a43923148c5cee18ccc356a594.r2.cloudflarestorage.com/
09:50:50  fatal error: An error occurred (Unauthorized) when calling the ListObjectsV2 operation: Unauthorized

New hotfix incoming

@dduportal
Copy link
Contributor

jenkins-infra/crawler@e196bb7 (watching build on trusted)

@dduportal
Copy link
Contributor

Finished: SUCCESS we can finish 2023 in this state \o/

@NotMyFault
Copy link
Member Author

Thanks for checking 🎄

@dduportal
Copy link
Contributor

As I don't know what @lemeurherve did in #3414 except the linked terraform PRs, and because we're in holidays for the next week, I've commented out the azcopy command jenkins-infra/crawler@d0efde1 and a build is currently running (I'll watch it).

We'll work on this in January then

Nothing done in #3414 explains the HTTP/403. I've reopened #3818 and I'll check if I did not forget a firewall rule somewhere. That could also explain the aws s3 errors after the first hotfix.

@dduportal dduportal mentioned this issue Dec 23, 2023
Closed
@dduportal
Copy link
Contributor

Resuming analysis on this topic.

@dduportal
Copy link
Contributor

Retried and got the following detailed error message for the azcopy operation:

Time:2024-01-02T13:25:44.6252585Z</Message><AuthenticationErrorDetail>Signature not valid in the specified time frame: Start [Fri, 06 Oct 2023 00:00:00 GMT] - Expiry [Fri, 22 Dec 2023 00:00:00 GMT] - Current [Tue, 02 Jan 2024 13:25:44 GMT]</AuthenticationErrorDetail></Error>

@dduportal
Copy link
Contributor

https://github.com/jenkins-infra/azure/blob/240beb8f6f96fdf3bb114f66091120a971938481/updates.jenkins.io.tf#L44 => gotcha.

@dduportal
Copy link
Contributor

jenkins-infra/azure#565 to rotate expiry

@dduportal
Copy link
Contributor

Update:

WiP:

  • Retrieve the new credential and apply it in trusted.ci
  • Check it work again for azcopy
  • Check error for the aws s3 sync and fix it if still present
  • Test access restriction for the SA

@dduportal dduportal mentioned this issue Jan 2, 2024
Open
@dduportal
Copy link
Contributor

dduportal commented Jan 2, 2024 •
edited
Loading

Update:

WiP:

  • Check error for the aws s3 sync and fix it if still present
  • Add a calendar/updatecli process

@dduportal
Copy link
Contributor

WiP: Check error for the aws s3 sync and fix it if still present

  • The error is fatal error: An error occurred (Unauthorized) when calling the ListObjectsV2 operation: Unauthorized when trying to S3-sync to the cloudflare bucket
  • There is no credential IaC configuration (AFAIR the cloudflare provider for terraform does not support - yet?- credentials generation) in https://github.com/jenkins-infra/cloudflare/blob/main/updates.jenkins.io.tf
    => next step, checking in the Cloudflare UI

@dduportal
Copy link
Contributor

Gotcha: all API tokens have reached their TTL in Cloudflare. It also breaks the jenkins-infra/cloudflare project since last month (at least).

See #2649 (comment)

@smerle33 smerle33 mentioned this issue Jan 4, 2024
Merged
@dduportal
Copy link
Contributor

Update:

@dduportal dduportal mentioned this issue Jan 5, 2024
Merged
dduportal added a commit to jenkins-infra/azure that referenced this issue Jan 5, 2024
@dduportal
cleanup(cert,trusted) migrate NAT gateway management to azure-net (#567)
befe0c5 
While working on jenkins-infra/helpdesk#3875,
the need to manage NAT gateways close to network resources appeared.

This PR removes the definitions of the 2 existing NAT gateway so we can
start managing them in jenkins-infra/azure-net.

Note that the associated resources (marked as "to be deleted" in the
build 1) will be removed manually from the terraform state (see command
in the comment below).
Then a 2nd build will be re-triggered before merging, to ensure no
changes will be done by this PR.

Signed-off-by: Damien Duportal <[email protected]>
This was referenced Jan 5, 2024
Merged
Merged
dduportal added a commit to jenkins-infra/azure that referenced this issue Jan 6, 2024
@smerle33 @dduportal
chore(updatecli): add expiry manifest with date bash scripts (#566)
079d8e8 
as per
jenkins-infra/helpdesk#3875 (comment)

---------

Co-authored-by: Damien Duportal <[email protected]>
@dduportal
Copy link
Contributor

Update:

Todo:

  • Ensure crawler still works with SAS Ip restriction to be added

@dduportal dduportal mentioned this issue Jan 6, 2024
Merged
dduportal added a commit to jenkins-infra/azure that referenced this issue Jan 6, 2024
@dduportal
feat(updates.jenkins.io) restrict SA to only a few administrative IPs (…
29c783c 
…#570)

This PR is related to both
jenkins-infra/helpdesk#2649 (comment)
and jenkins-infra/helpdesk#3875

It restricts the Azure SA used for updates.jenkins.io to only a few
administrative IPs including trusted.ci's 2 networks.

It also allows trusted.ci new network (sponsored) agents to access
publick8s controler (for update center)

---------

Signed-off-by: Damien Duportal <[email protected]>
@dduportal dduportal mentioned this issue Jan 6, 2024
Merged
dduportal added a commit to jenkins-infra/azure that referenced this issue Jan 6, 2024
@dduportal
hotfix(updates.jenkins.io) SA network restriction must have plain IPs…
3190e23 
… and should not disable public access to apply restriction (#571)

Fixup of #570 

Related to jenkins-infra/helpdesk#2649 and
jenkins-infra/helpdesk#3875

Signed-off-by: Damien Duportal <[email protected]>
@dduportal
Copy link
Contributor

Job still failing: #2649 (comment)

This was referenced Jan 6, 2024
Merged
Merged
@dduportal
Copy link
Contributor

Last steps: #2649 (comment)

Last build of crawler built without any problem

@smerle33 smerle33 mentioned this issue Jan 8, 2024
Merged
dduportal pushed a commit to jenkins-infra/azure that referenced this issue Jan 17, 2024
@smerle33
chore(updatecli): change date detection for expiry and new certificat…
8e1f345 
…e lifespan (#573)

as per
jenkins-infra/helpdesk#3875 (comment)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@dduportal dduportal

@smerle33 smerle33

Labels
trusted.ci.jenkins.io
Projects
None yet
Milestone
infra-team-sync-2024-01-09
Development

No branches or pull requests
3 participants
@dduportal @NotMyFault @smerle33