bazel: add build targets for HTTP API documentation, start building in CI #65814
Assignees
Labels
A-build-system
C-bug
Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior.
T-dev-inf
Comments
rickystewart
added
C-bug
|
|
rickystewart
changed the title
rickystewart
added a commit
to rickystewart/cockroach
that referenced
this issue
Aug 12, 2021
This flag [instructs Bazel](bazelbuild/rules_proto#56 (comment)) to set a [command-line flag](protocolbuffers/protobuf#7623 (comment)) when invoking `protoc` that causes the generated proto descriptor sets to contain extra info: ``` --include_source_info When using --descriptor_set_out, do not strip SourceCodeInfo from the FileDescriptorProto. This results in vastly larger descriptors that include information about the original location of each decl in the source file as well as surrounding comments. ``` Setting this solves two problems: 1. We need the descriptor sets to have comments for cockroachdb#65814. 2. Without this change, generated `.pb.go` files from the sandbox won't contain comments. This makes the files more difficult to read and dirties the files in your checkout if you copy those `.pb.go` files to your workspace. Also delete an unnecessary `--symlink_prefix=_bazel/` from the `test` configuration (it's inherited from the `build` configuration so it's redundant). Release note: None
craig bot
pushed a commit
that referenced
this issue
Aug 12, 2021
68102: storage/storageccl: Add an option to break export mid key r=pbardea,sumeerbhola a=aliher1911 Previously when exporting ranges with very large number of versions export could hit the situation where single key export would exceed maximum export byte limit. To resolve this, safety threshold has to be raised which could cause nodes going out of memory. To address this, this patch adds an ability to stop and resume on the arbitrary key timestamp so that it could be stopped mid key and resumed later. Release note: None Fixes #68231 68390: ui: rebuild the databases pages r=matthewtodd a=matthewtodd Previously, the databases pages featured an older, outdated design. This change aligns their UX with that of the other modern pages in the DB console, statements, transactions, and sessions. This is a first pass at the job. We will return to layer on more features, including search, and more data, including node & region information. Addresses #65737. Before | After --- | --- <img width="1372" alt="Screen Shot 2021-08-05 at 4 54 31 PM" src="https://user-images.githubusercontent.com/5261/128420405-7a096c2f-32a1-42d9-a8f0-186f699fb612.png"> | <img width="1372" alt="Screen Shot 2021-08-03 at 2 50 12 PM" src="https://user-images.githubusercontent.com/5261/128070069-28d0f461-69dd-47cb-a04b-bc6cd0126ca4.png"> <img width="1372" alt="Screen Shot 2021-08-05 at 4 54 57 PM" src="https://user-images.githubusercontent.com/5261/128420491-71d03570-1c0d-413b-ab40-09b45cf614ee.png"> | <img width="1372" alt="Screen Shot 2021-08-03 at 2 50 21 PM" src="https://user-images.githubusercontent.com/5261/128070090-bd62d125-a1b2-4c83-b0a5-33d0723849eb.png"> <img width="1372" alt="Screen Shot 2021-08-05 at 5 03 37 PM" src="https://user-images.githubusercontent.com/5261/128420959-338ea79b-abf0-4773-91bf-cecac64fec5f.png"> | (This overview of database grants no longer exists.) (This rolled-up view of table grants did not previously exist.) | <img width="1372" alt="Screen Shot 2021-08-03 at 2 50 27 PM" src="https://user-images.githubusercontent.com/5261/128070100-1305c94f-c017-40a5-ba2d-f19fd61bdecf.png"> <img width="1372" alt="Screen Shot 2021-08-05 at 4 55 21 PM" src="https://user-images.githubusercontent.com/5261/128420596-2d9f04ad-718c-4dce-b045-b5647f718339.png"> | <img width="1372" alt="Screen Shot 2021-08-03 at 2 50 35 PM" src="https://user-images.githubusercontent.com/5261/128070103-831fd5c3-38da-4864-9d82-9091f3209606.png"> <img width="1372" alt="Screen Shot 2021-08-05 at 4 55 28 PM" src="https://user-images.githubusercontent.com/5261/128420658-4042c631-b283-4b0c-ac4a-3235bd3bcf39.png"> | <img width="1372" alt="Screen Shot 2021-08-03 at 2 50 41 PM" src="https://user-images.githubusercontent.com/5261/128070113-73528f2c-56bf-4fcf-a8d8-fcc3f100695f.png"> Release note (ui change): The databases pages in the DB console were updated to bring them into alignment with our modern UX. 68619: storage: add file storing min version needed for backwards compatibility r=jbowens a=andyyang890 A new STORAGE_MIN_VERSION file containing the minimum version that the storage engine needs to maintain backwards compatibility with will be added to the directory for each storage engine. This will be used in the encryption-at-rest registry migration. Release note: None 68621: dev: implement `dev generate docs` r=rail a=rickystewart Do some refactoring so that we don't have to repeat code in a bunch of places. The implementation is pretty straightforward and just involves building the docs then copying them over to the workspace. Closes #68563. Release note: None 68663: dev: hoist generated code out of sandbox into workspace r=rail a=rickystewart (Only the most recent commit applies for this review.) One might be expected that this command be implemented as `dev generate protobuf` or `dev generate execgen` or similar, but that's not really practical for a couple reasons: 1. Building all the protobuf in-tree isn't really possible on its own: see #58018 (comment). 2. For non-proto generated code, one can imagine that we do a bunch of `bazel query`s to list all the generated files in tree and to find which files are generated. Given that there are hundreds of generated files in tree, and a single `bazel query` can take ~1s, this is not tractable. The implementation here is ad-hoc and a bit hacky. Basically we search `bazel-bin` for files ending in `.go` and apply some heuristics to figure out whether they should be copied and where those files should go. This is gated behind a feature flag for now because actually copying these files into the workspace right now dirties a ton of files. (The diffs are benign -- mostly comments.) We can flip this option to be on by default when it's safe (probably after the `Makefile` is deleted). Closes #68565. Release note: None 68806: bazel: set `--include_source_info` when generating protobuf code r=rail a=rickystewart This flag [instructs Bazel](bazelbuild/rules_proto#56 (comment)) to set a [command-line flag](protocolbuffers/protobuf#7623 (comment)) when invoking `protoc` that causes the generated proto descriptor sets to contain extra info: ``` --include_source_info When using --descriptor_set_out, do not strip SourceCodeInfo from the FileDescriptorProto. This results in vastly larger descriptors that include information about the original location of each decl in the source file as well as surrounding comments. ``` Setting this solves two problems: 1. We need the descriptor sets to have comments for #65814. 2. Without this change, generated `.pb.go` files from the sandbox won't contain comments. This makes the files more difficult to read and dirties the files in your checkout if you copy those `.pb.go` files to your workspace. Also delete an unnecessary `--symlink_prefix=_bazel/` from the `test` configuration (it's inherited from the `build` configuration so it's redundant). Release note: None Co-authored-by: Oleg Afanasyev <[email protected]> Co-authored-by: Matthew Todd <[email protected]> Co-authored-by: Andy Yang <[email protected]> Co-authored-by: Ricky Stewart <[email protected]>
rickystewart
added a commit
to rickystewart/cockroach
that referenced
this issue
Aug 12, 2021
We generate this in a different way than the `Makefile` does -- we take all the proto descriptor sets (that we need to generate anyway as part of the build) and pass them in to `protoc` rather than passing a bunch of include paths, which wouldn't work terribly well in the sandbox. Also teach CI and `dev` to start building these docs. I deleted a stray comment in `pkg/server/serverpb/admin.proto` that was causing a diff in the generated file. I also had to patch `@com_github_pseudomuto_protoc_gen_doc` to quash a compiler error (I just deleted a dependency that isn't actually necessary). Closes cockroachdb#65814. Release note: None
rickystewart
added a commit
to rickystewart/cockroach
that referenced
this issue
Aug 16, 2021
We generate this in a different way than the `Makefile` does -- we take all the proto descriptor sets (that we need to generate anyway as part of the build) and pass them in to `protoc` rather than passing a bunch of include paths, which wouldn't work terribly well in the sandbox. Also teach CI and `dev` to start building these docs. I deleted a stray comment in `pkg/server/serverpb/admin.proto` that was causing a diff in the generated file. I also had to patch `@com_github_pseudomuto_protoc_gen_doc` to quash a compiler error (I just deleted a dependency that isn't actually necessary). Closes cockroachdb#65814. Release note: None
craig bot
pushed a commit
that referenced
this issue
Aug 17, 2021
68834: bazel: build http docs w bazel r=rail a=rickystewart We generate this in a different way than the `Makefile` does -- we take all the proto descriptor sets (that we need to generate anyway as part of the build) and pass them in to `protoc` rather than passing a bunch of include paths, which wouldn't work terribly well in the sandbox. Also teach CI and `dev` to start building these docs. I deleted a stray comment in `pkg/server/serverpb/admin.proto` that was causing a diff in the generated file. I also had to patch `@com_github_pseudomuto_protoc_gen_doc` to quash a compiler error (I just deleted a dependency that isn't actually necessary). Closes #65814. Release note: None 69019: kv: deflake TestTxnCoordSenderPipelining r=nvanbenschoten a=nvanbenschoten Fixes #68948. This commit disables the transaction heartbeat loop in the test, which removes a dependency on timing. However, this doesn't explain why the test became slower over the past few days. We see in CI that on the two separate occasions where the test flaked, it took 1.08s and 1.41s, both above the 1s necessary for the test to flake. Co-authored-by: Ricky Stewart <[email protected]> Co-authored-by: Nathan VanBenschoten <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Build HTTP API documentation with Bazel.
These are generated with
docgen http.blocks #65070
Epic CRDB-8306
The text was updated successfully, but these errors were encountered: