Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

buck2_client: download traces via buck2.log_url config key #770

Draft
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

thoughtpolice
Copy link
Contributor

@thoughtpolice thoughtpolice commented Sep 5, 2024

When doing a command like buck2 log replay --trace-id ... it helps to be able to download previously created logs from users, automated CI, etc. This functionality exists within Meta, as logs are available for download through the "Manifest" system, but it isn't usable in OSS, despite being useful for diagnostics and getting help.

The only missing thing to really get things working is a download mechanism for log files, and a way to know where they should come from.

With this patch, if a user configures the buck2.log_url key, which is expected to look something like:

[buck2]
log_url = https://example.com

Then, upon executing a log command with a --trace-id flag, this server will be queried with a:

GET /v1/get/{uuid}

request, which is expected to return the raw zst-encoded protobuf file. So, buck2 will do that and download the trace, and then use it like normal.

The request is done with curl, though in principle it could be done within Buck itself.

This shares as much code as possible with the existing infrastructure and tries to only insert a small key set of #[cfg(fbcode_build)] directives. Notably, the path Meta uses for their "Manifold" client is still fully available in the OSS version; fbcode_build is only used to gate what the default choice is.

How the server gets access to these files and how they are uploaded is another question. But for now, this can be done several ways outside of buck2 core, so this is good enough.

GitHub Issue: #441

@facebook-github-bot facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Sep 5, 2024
@thoughtpolice
Copy link
Contributor Author

This can't do things like pass authentication credentials just yet, but I'd like to know if this is the right shape. As it is, it's still incredibly useful, so I'd like to know if it looks like roughly the right thing before doing much more.

@thoughtpolice thoughtpolice changed the title buck2_client: download traces via buck2.log_download_url config key buck2_client: download traces via buck2.log_url config key Sep 5, 2024
@thoughtpolice thoughtpolice marked this pull request as draft September 5, 2024 16:30
@thoughtpolice
Copy link
Contributor Author

I'd like to make some small API changes to the GET URI so I'm holding this. But I'd still really like something like this to go in :)

@thoughtpolice thoughtpolice force-pushed the aseipp/push-xnmzuqpwomuk branch 4 times, most recently from ba03f55 to 505c4c2 Compare September 8, 2024 22:26
@thoughtpolice thoughtpolice force-pushed the aseipp/push-xnmzuqpwomuk branch 2 times, most recently from 9206ebd to 7e86204 Compare September 20, 2024 22:15
@thoughtpolice thoughtpolice force-pushed the aseipp/push-xnmzuqpwomuk branch 2 times, most recently from 0949481 to 947d903 Compare October 16, 2024 20:35
When doing a command like `buck2 log replay --trace-id ...` it helps to be
able to download previously created logs from users, automated CI, etc. This
functionality exists within Meta, as logs are available for download through
the "Manifest" system, but it isn't usable in OSS, despite being useful for
diagnostics and getting help.

The only missing thing to really get things working is a download mechanism for
log files, and a way to know where they should come from.

With this patch, if a user configures the `buck2.log_url` key, which is
expected to look something like:

    [buck2]
    log_url = https://example.com

Then, upon executing a `log` command with a `--trace-id` flag, this server will
be queried with a:

    GET /v1/get/{uuid}

request, which is expected to return the raw zst-encoded protobuf file. So,
buck2 will do that and download the trace, and then use it like normal.

The request is done with `curl`, though in principle it could be done within
Buck itself.

This shares as much code as possible with the existing infrastructure and tries
to only insert a small key set of `#[cfg(fbcode_build)]` directives. Notably,
the path Meta uses for their "Manifold" client is still fully available in the
OSS version; `fbcode_build` is only used to gate what the default choice is.

How the server gets access to these files and how they are uploaded is another
question. But for now, this can be done several ways outside of buck2 core, so
this is good enough.

GitHub Issue: facebook#441

Signed-off-by: Austin Seipp <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants