Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Cannot get EC2 user data after enabled IMDSv2 #498

Closed
vincentwenatsa opened this issue Dec 20, 2023 · 12 comments · Fixed by #507
Closed

Cannot get EC2 user data after enabled IMDSv2 #498

vincentwenatsa opened this issue Dec 20, 2023 · 12 comments · Fixed by #507
Assignees
@syuu1228
Labels
P2

Comments

@vincentwenatsa
Copy link

vincentwenatsa commented Dec 20, 2023 •
edited
Loading

Dec 19 15:38:09 ip-172-19-20-73 scylla_post_start.py[3760]: 2023-12-19 15:38:09,968 - [user_data] - WARNING - Error getting user data: HTTP Error 401: Unauthorized. Will use defaults!

Dec 19 15:38:09 ip-172-19-20-73 scylla_post_start.py[3760]: Error getting user data: HTTP Error 401: Unauthorized. Will use defaults!

the scylla_post_start.py script failed to fetch user data after enabled IMDSv2. It needs to fetch a IMDSv2 token https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html
@fruch
Copy link
Collaborator

fruch commented Dec 20, 2023

it was introduced a year ago in:
b1c8990

which version of scylla AMI you are using ?

@vincentwenatsa
Copy link
Author

5.2.10 @fruch

@vincentwenatsa
Copy link
Author

Will it be backport to any 5.2 version?

@yaronkaikov yaronkaikov assigned syuu1228 and unassigned yaronkaikov Dec 20, 2023
@marqueurs404
Copy link

marqueurs404 commented Dec 21, 2023 •
edited
Loading

I am encountering this on 5.4.0 scylla AMI as well, on both the scylla_post_start and scylla-image-setup services

@marqueurs404
Copy link

marqueurs404 commented Dec 21, 2023 •
edited
Loading

I presume it's because the fetched token wasn't used in

scylla-machine-image/lib/scylla_cloud.py

Line 819 in 26b93d5

base_contents = curl(self.META_DATA_BASE_URL).splitlines()
and

scylla-machine-image/lib/scylla_cloud.py

Line 821 in 26b93d5

return curl(self.META_DATA_BASE_URL + 'user-data')

This bug was likely only noticed recently as EC2 defaults to IMDSv2-only since November 2023 https://aws.amazon.com/blogs/aws/amazon-ec2-instance-metadata-service-imdsv2-by-default/

@yaronkaikov yaronkaikov added the P2 label Dec 21, 2023
@fruch
Copy link
Collaborator

fruch commented Dec 21, 2023

@syuu1228

We should consider setting the v2 usage in the AMI once this issue is fixed

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/configuring-IMDS-new-instances.html

@fruch
Copy link
Collaborator

fruch commented Dec 21, 2023

@marqueurs404

Thanks for spotting it out, seems you are correct, and some calls were missed

@mykaul
Copy link

mykaul commented Dec 21, 2023

scylladb/seastar#1051
scylladb/scylla-manager#3244

(and we have some others, that are not completely IMDSv2 friendly)

@fruch
Copy link
Collaborator

fruch commented Dec 21, 2023

scylladb/seastar#1051 scylladb/scylla-manager#3244

(and we have some others, that are not completely IMDSv2 friendly)

exactly why I've suggested enabling in the AMI, to flush all those out.

@vincentwenatsa
Copy link
Author

Will this issue get fixed any time soon?

@yaronkaikov
Copy link
Collaborator

@syuu1228 ping

syuu1228 added a commit to syuu1228/scylla-machine-image that referenced this issue Feb 5, 2024
@syuu1228
aws_instance: fix broken user_data()
96f8c01 
user_data() is mistakenly unsupported IMDSv2, since it's not using
__instance_metadata().

Fixed the code to use __instance_metadata().

Fixes scylladb#498
@syuu1228 syuu1228 mentioned this issue Feb 5, 2024
Closed
syuu1228 added a commit to syuu1228/scylla-machine-image that referenced this issue Feb 6, 2024
@syuu1228
aws_instance: fix broken user_data()
5fd6575 
user_data() is mistakenly unsupported IMDSv2, since it's not using
__instance_metadata().

Fixed the code to use __instance_metadata().

Fixes scylladb#498
syuu1228 added a commit to syuu1228/scylla-machine-image that referenced this issue Feb 6, 2024
@syuu1228
aws_instance: fix broken user_data()
c61b740 
user_data() is mistakenly unsupported IMDSv2, since it's not using
__instance_metadata().

Fixed the code to use __instance_metadata().

Also, while testing this code, scylla_configure.py causes
"binascii.Error: Incorrect padding" on
base64.b64decode(post_configuration_script).
It seems because running b64decode() on already decoded string, so
dropped the line and just use post_configuration_script directly.

Fixes scylladb#498
@syuu1228 syuu1228 mentioned this issue Feb 6, 2024
Merged
syuu1228 added a commit to syuu1228/scylla-machine-image that referenced this issue Feb 6, 2024
@syuu1228
aws_instance: fix broken user_data()
d8cf7f4 
user_data() is mistakenly unsupported IMDSv2, since it's not using
__instance_metadata().

Fixed the code to use __instance_metadata().

Fixes scylladb#498
yaronkaikov pushed a commit that referenced this issue Feb 6, 2024
@syuu1228 @yaronkaikov
aws_instance: fix broken user_data()
6e92623 
user_data() is mistakenly unsupported IMDSv2, since it's not using
__instance_metadata().

Fixed the code to use __instance_metadata().

Fixes #498
@yaronkaikov
Copy link
Collaborator

@vincentwenatsa once this fix is promoted. we will backport to 5.4 and 5.2, it will be part of next release

yaronkaikov pushed a commit that referenced this issue Mar 7, 2024
@syuu1228 @yaronkaikov
aws_instance: fix broken user_data()
b823956 
user_data() is mistakenly unsupported IMDSv2, since it's not using
__instance_metadata().

Fixed the code to use __instance_metadata().

Fixes #498

(cherry picked from commit 6e92623)
yaronkaikov pushed a commit that referenced this issue Mar 7, 2024
@syuu1228 @yaronkaikov
aws_instance: fix broken user_data()
2081850 
user_data() is mistakenly unsupported IMDSv2, since it's not using
__instance_metadata().

Fixed the code to use __instance_metadata().

Fixes #498

(cherry picked from commit 6e92623)
syuu1228 added a commit to syuu1228/scylla-machine-image that referenced this issue Mar 16, 2024
@syuu1228
scripts: exit with 1 when Exception catched
7b241cb 
On scylladb#498, status of scylla-image-post-start.service is NOT "failed" even
the script causing error, because scylla_post_start.py catches all
exceptions and just logging it, so the script finishes without error.

To getting notify users the script failed operation during running, we
should sys.exit(1) when the exception catched.

fixes scylladb#505
@syuu1228 syuu1228 mentioned this issue Mar 18, 2024
Merged
syuu1228 added a commit to syuu1228/scylla-machine-image that referenced this issue May 22, 2024
@syuu1228
scripts: exit with 1 when Exception catched
e7aad4d 
On scylladb#498, status of scylla-image-post-start.service is NOT "failed" even
the script causing error, because scylla_post_start.py catches all
exceptions and just logging it, so the script finishes without error.

To getting notify users the script failed operation during running, we
should sys.exit(1) when the exception catched.

fixes scylladb#505
syuu1228 added a commit to syuu1228/scylla-machine-image that referenced this issue May 27, 2024
@syuu1228
scripts: exit with 1 when Exception catched
4f52e55 
On scylladb#498, status of scylla-image-post-start.service is NOT "failed" even
the script causing error, because scylla_post_start.py catches all
exceptions and just logging it, so the script finishes without error.

To getting notify users the script failed operation during running, we
should sys.exit(1) when the exception catched.

fixes scylladb#505

Signed-off-by: Takuya ASADA <[email protected]>
yaronkaikov pushed a commit that referenced this issue May 29, 2024
@syuu1228 @yaronkaikov
scripts: exit with 1 when Exception catched
2c1786c 
On #498, status of scylla-image-post-start.service is NOT "failed" even
the script causing error, because scylla_post_start.py catches all
exceptions and just logging it, so the script finishes without error.

To getting notify users the script failed operation during running, we
should sys.exit(1) when the exception catched.

fixes #505
syuu1228 added a commit to syuu1228/seastar-1 that referenced this issue Nov 27, 2024
@syuu1228
scripts/perftune.py: Implement AWS IMDSv2 call
c53e0b5 
perftune's __check_host_type method uses IMDSv1 for retrieving instance
metadata.
It turns out that - as described in scylladb/scylladb#10490 - Ec2 instances may
have only IMDSv2 HTTP calls allowed.

When requests to IMDSv1 fail, calling is_aws_i3_non_metal_instance() will
evaluate to False, which means that i3.nonmetal instance types will get tuned
as if they weren't i3 instances.

To fix the problem, import IMDSv2 implementation from
scylla-machine-image.

Fixes scylladb#1051
related scylladb/scylla-machine-image#498
related scylladb/scylladb#10490
@syuu1228 syuu1228 mentioned this issue Nov 27, 2024
Open
syuu1228 added a commit to syuu1228/seastar-1 that referenced this issue Nov 30, 2024
@syuu1228
scripts/perftune.py: Implement AWS IMDSv2 call
46c13e8 
perftune's __check_host_type method uses IMDSv1 for retrieving instance
metadata.
It turns out that - as described in scylladb/scylladb#10490 - Ec2 instances may
have only IMDSv2 HTTP calls allowed.

When requests to IMDSv1 fail, calling is_aws_i3_non_metal_instance() will
evaluate to False, which means that i3.nonmetal instance types will get tuned
as if they weren't i3 instances.

To fix the problem, import IMDSv2 implementation from
scylla-machine-image.

Fixes scylladb#1051
related scylladb/scylla-machine-image#498
related scylladb/scylladb#10490
syuu1228 added a commit to syuu1228/seastar-1 that referenced this issue Nov 30, 2024
@syuu1228
scripts/perftune.py: Implement AWS IMDSv2 call
4cd52f8 
perftune's __check_host_type method uses IMDSv1 for retrieving instance
metadata.
It turns out that - as described in scylladb/scylladb#10490 - Ec2 instances may
have only IMDSv2 HTTP calls allowed.

When requests to IMDSv1 fail, calling is_aws_i3_non_metal_instance() will
evaluate to False, which means that i3.nonmetal instance types will get tuned
as if they weren't i3 instances.

To fix the problem, import IMDSv2 implementation from
scylla-machine-image.

Fixes scylladb#1051
related scylladb/scylla-machine-image#498
related scylladb/scylladb#10490
syuu1228 added a commit to syuu1228/seastar-1 that referenced this issue Nov 30, 2024
@syuu1228
scripts/perftune.py: Implement AWS IMDSv2 call
dbe231c 
perftune's __check_host_type method uses IMDSv1 for retrieving instance
metadata.
It turns out that - as described in scylladb/scylladb#10490 - Ec2 instances may
have only IMDSv2 HTTP calls allowed.

When requests to IMDSv1 fail, calling is_aws_i3_non_metal_instance() will
evaluate to False, which means that i3.nonmetal instance types will get tuned
as if they weren't i3 instances.

To fix the problem, import IMDSv2 implementation from
scylla-machine-image.

Fixes scylladb#1051
related scylladb/scylla-machine-image#498
related scylladb/scylladb#10490
syuu1228 added a commit to syuu1228/seastar-1 that referenced this issue Nov 30, 2024
@syuu1228
scripts/perftune.py: Implement AWS IMDSv2 call
c00533b 
perftune's __check_host_type method uses IMDSv1 for retrieving instance
metadata.
It turns out that - as described in scylladb/scylladb#10490 - Ec2 instances may
have only IMDSv2 HTTP calls allowed.

When requests to IMDSv1 fail, calling is_aws_i3_non_metal_instance() will
evaluate to False, which means that i3.nonmetal instance types will get tuned
as if they weren't i3 instances.

To fix the problem, import IMDSv2 implementation from
scylla-machine-image.

Fixes scylladb#1051
related scylladb/scylla-machine-image#498
related scylladb/scylladb#10490
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@syuu1228 syuu1228

Labels
P2
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

aws_instance: fix broken user_data() syuu1228/scylla-machine-image
6 participants
@syuu1228 @fruch @marqueurs404 @mykaul @yaronkaikov @vincentwenatsa