feat: add Worker pre-install commands, --start, and Job.get_logs #25
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jericht
changed the title
jericht
force-pushed
the
jericht/more_fixes
branch
3 times, most recently
from
95e3dac to
938de80
Compare
jericht
changed the title
jericht
force-pushed
the
jericht/more_fixes
branch
3 times, most recently
from
0029fff to
01e314a
Compare
jericht
commented
Oct 12, 2023
Comment on lines
+421
to
+424
| # TODO: Support multiple job users on Docker | ||
| assert ( | ||
| len(self.configuration.job_users) == 1 | ||
| ), f"Multiple job users not supported on Docker worker: {self.configuration.job_users}" |
There was a problem hiding this comment.
Going to do the Docker stuff later. It's mostly a dev convenience, and we're at a time crunch at the moment.
jericht
force-pushed
the
jericht/more_fixes
branch
2 times, most recently
from
415f508 to
b7f25c4
Compare
Signed-off-by: Jericho Tolentino <[email protected]>
jericht
force-pushed
the
jericht/more_fixes
branch
from
b7f25c4 to
e57b537
Compare
AWS-Samuel
approved these changes
Oct 12, 2023
ttblanchard
approved these changes
Oct 12, 2023
ddneilson
suggested changes
Oct 12, 2023
| session_log_map: dict[str, list[CloudWatchLogEvent]] = {} | ||
| for session in sessions: | ||
| session_id = session["sessionId"] | ||
| get_log_events_response = logs_client.get_log_events( |
There was a problem hiding this comment.
This is a paginated API. As written, this won't be able to fetch the full log stream if there's a lot of data in it.
| Returns: | ||
| JobLogs: The job logs | ||
| """ | ||
| list_sessions_response = deadline_client.list_sessions( |
There was a problem hiding this comment.
This is a paginated API. Jobs that have a lot of tasks/steps may end up with more sessions than can be sent in a single response.
moorec-aws
reviewed
Oct 12, 2023
| Returns: | ||
| JobLogs: The job logs | ||
| """ | ||
| list_sessions_response = deadline_client.list_sessions( |
There was a problem hiding this comment.
Canary permissions will need to be updated
moorec-aws
reviewed
Oct 12, 2023
| session_log_map: dict[str, list[CloudWatchLogEvent]] = {} | ||
| for session in sessions: | ||
| session_id = session["sessionId"] | ||
| get_log_events_response = logs_client.get_log_events( |
There was a problem hiding this comment.
Canary permissions will need updating
Signed-off-by: Jericho Tolentino <[email protected]>
jericht
force-pushed
the
jericht/more_fixes
branch
from
7706c1f to
2e3eb1c
Compare
ddneilson
approved these changes
Oct 12, 2023
jericht
requested review from
moorec-aws,
ttblanchard,
ddneilson and
AWS-Samuel
AWS-Samuel
approved these changes
Oct 12, 2023
ttblanchard
approved these changes
Oct 12, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Related aws-deadline/deadline-cloud-worker-agent#54
What was the problem/requirement? (What/Why)
Need more features and some fixes to add a Worker agent test for the
jobRunAsUserfeatureWhat was the solution? (How)
Major changes:
DeadlineWorkerConfiguration,pre_install_commandsproperty that is a list of commands that run before the Worker agent installer is runworker_configfixture fromfixtures.py(see pytest docs on overriding fixtures), then return a newDeadlineWorkerConfigurationwith the modified configurationDeadlineWorkerConfiguration.start_serviceand pass it to the Worker agent installer so that the systemd service can be started during installationJob.get_logswhich retrieves the logs for a jobMinor changes / fixes:
raw_kwargsoption to all resource methods that wrap an AWS API call (e.g. CreateJob, CreateQueue, etc.). This gives us the ability to not have to upgrade the library version in order to write testsFARM_IDandFLEET_IDto Docker container worker so that users do not need to define those env vars when not usingBYO_DEADLINEUSE_DOCKER_WORKERso that setting it to something other thantruewill not use a Docker workerWhat is the impact of this change?
jobRunAsUsertests for Worker agentHow was this change tested?
Was this change documented?
Is this a breaking change?