Merge #65924 #71254 #71359

65924: docs: RFC for db-console Bazel build r=koorosh a=koorosh

Described proposal and changes required for
`pkg/ui` UI projects to establish Bazel build
workflow and integration with other bazel
rules.

Release note: None

71254: cli: identify log source more easily when running merge-logs r=ajwerner,cameronnunez,rauchenstein a=jtsiros

resolves  #55395

Previously, merge-logs output was prefixed by short machine name by default which made it difficult to identify the originating node when looking at the merged results.

This change adds support for ${fpath} in the template when using the --prefix arg to include unique file path contents as part of the prefix. The file path prefix excludes common tokens across all file paths and tokens that contain specific delimiters.

Release note (cli change): adds support for "${fpath}" in --prefix arg


71359: spanconfig: introduce the spanconfig.SQLTranslator r=irfansharif,nvanbenschoten,ajwerner a=arulajmani

This patch introduces the spanconfig.SQLTranslator which translates
zone configurations to span configurations.

The zone config and span config protos look similar, but they differ
from each other in a couple of ways:
- zone configurations correspond to {descriptor, zone} IDs whereas span
configurations correspond to keyspans.
- zone configurations have a notion of inheritance which span
configurations do not. Instead, they're fully hydrated and flattened
out.

When the SQLTranslator is given a {zone,descriptor} ID it generates
span configurations for all objects under the given ID in the zone
configuration hierarchy.

The SQLTranslator fills in any missing fields by following up the
inheritance chain to fully hydrate span configs. It also accounts
for subzones (and subzone spans) when constructing (span, span config)
tuples.

Split out from #69661.

Epic: CRDB-10489

Release note: None

Co-authored-by: Andrii Vorobiov <[email protected]>
Co-authored-by: Jon Tsiros <[email protected]>
Co-authored-by: arulajmani <[email protected]>

.github/CODEOWNERS

@@ -93,6 +93,7 @@
 
 /pkg/ccl/backupccl/          @cockroachdb/bulk-prs
 /pkg/ccl/importccl/          @cockroachdb/bulk-prs
+/pkg/ccl/spanconfigccl/      @cockroachdb/kv-prs
 /pkg/ccl/storageccl/         @cockroachdb/bulk-prs
 /pkg/cloud/                  @cockroachdb/bulk-prs
 /pkg/sql/distsql_plan_csv.go @cockroachdb/bulk-prs
@@ -221,7 +222,7 @@
 /pkg/scheduledjobs/          @cockroachdb/bulk-prs
 /pkg/security/               @cockroachdb/server-prs @cockroachdb/prodsec
 /pkg/settings/               @cockroachdb/server-prs
-/pkg/spanconfig/             @cockroachdb/multiregion
+/pkg/spanconfig/             @cockroachdb/kv-prs
 /pkg/startupmigrations/      @cockroachdb/server-prs @cockroachdb/sql-schema
 /pkg/streaming/              @cockroachdb/bulk-prs
 /pkg/testutils/              @cockroachdb/test-eng-noreview

docs/RFCS/20210423_db-cosole-bazel-build.md

@@ -0,0 +1,92 @@
+- Feature Name:
+- Status: draft
+- Start Date: 2021-04-23
+- Authors: Andrii Vorobiov
+- RFC PR: [#65924](https://github.com/cockroachdb/cockroach/pull/65924)
+- Cockroach Issue:
+
+# Summary
+
+Migration from Make builds to Bazel build tool for the entire project is one of OKRAs which is aimed to
+deliver the product in a timely and sustainable manner. This proposal outlines a subset of changes required to perform 
+migration to Bazel build for DB Console (and UI related packages).
+
+# Motivation
+
+Migration should not affect the current build process with `Make` so it can be gradual (without
+breaking changes).
+The migration process can be broken down into the following parts:
+- establish Bazel build for Db Console (with support for both OSS and CCL licensing);
+- establish Bazel build and publishing artifacts to NPM registry for `cluster-ui` package;
+- build `crdb-protobufjs-client` package for both CCL and OSS licences.
+
+Bazel builds (comparing to Make build) allow defining granular steps which can be run independently for particular
+package instead of the entire project, which in turn reduces the build time and a chain of unnecessary rebuilds for
+unchanged parts of code.
+
+Github `Bazel` project already includes following tasks related to migration to Bazel for Db Console   
+- [build: Bazelize UI](https://github.com/cockroachdb/cockroach/issues/56068)
+- [build: Get built UI to bindata (bazel)](https://github.com/cockroachdb/cockroach/issues/56069)
+- [build,bazel: Implement JS protobuf support for Bazel](https://github.com/cockroachdb/cockroach/issues/59329)
+- [build: Get bazel building both ccl and oss builds](https://github.com/cockroachdb/cockroach/issues/56071)
+
+# Technical design
+
+Current UI project has following dependencies
+
+![DbConsole project dependencies](images/db-console-bazel-build.png?raw=true)
+
+where:
+- `*.proto` files are set of protobuf files that are required to build JS protobuf client code
+- `ccl/oss` specifies build condition, whether the project is built for CCL or OSS licence
+- `ccl protos/oss protos` - autogenerated JS protobuf client code
+- `db-console` is main `ui` project
+- `cluster-ui` is a nested `ui/cluster-ui` JS project
+- `ui/(ccl|oss)/bidndata.go` - autogenerated Go sources that embed JS bundled files
+
+#### Dependency graph of bazel rules:
+
+![Bazel rules dependencies](images/db-console-bazel-build-dependency-graph.svg)
+
+Bazel build pipeline for UI can be broken down in following steps:
+- Build protobuf js client code [db-console: bazel build for protobufjs clients #64065](https://github.com/cockroachdb/cockroach/pull/64065)
+- `cluster-ui` standalone build (react + webpack project) 
+- `db-console` standalone build (react + webpack project)
+- embed JS assets into go libraries (`pkg/ui/distoss/distoss.go` and `pkg/ui/distccl/distccl.go`)
+
+### Generate protobuf JS client code
+Protobuf CCL and OSS packages are considered as an independent bazel builds. The installation of `protobufjs` package
+requires additional steps to install required dependencies for protobufjs-cli tool. This problem persists in Makefile
+as well and will be fixed with major release of protobufjs ver 7. Installation of dependencies and binding protobuf cli
+is done according to example provided here: https://github.com/bazelbuild/rules_nodejs/tree/stable/examples/protobufjs.
+
+### Build `db-console` and `cluster-ui` packages
+`db-console` and `cluster-ui` are typescript packages which have their own webpack configuration to build production
+bundle, run dev server and watch for changes.
+To build these projects, additional project changes required:
+- change relative paths for input and output files to be absolute paths in webpack configs because bazel resolves
+  relative paths from workspace root dir (which is root of entire `cockroach` project)
+- optional dependencies that are required only for web development and was defined as a separate NPM package (with its
+own package.json file in `pkg/ui/opts`) is removed and dependencies are moved in `pkg/ui/workspaces/db-console/package.json`
+under `optionalDependencies` section. Along with this, installation of NPM packages is performed with `--ignore-optional`
+flag to prevent installation of optional during production builds (ie `yarn install --ignore-optional`).
+
+### Refactor project structure to use Yarn workspaces
+Existing project structure had multiple `node_modules` directories for different purposes. With Bazel, every 
+`node_modules` is treated as separate external workspaces and with such configuration it restricts to use dependencies 
+from single external workspace.
+
+To overcome this limitation, project structure is refactored to use Yarn workspaces, it allows to hoist all common
+dependencies to the top level in a single `node_modules` which is considered as a single external workspace for bazel.
+- `rules_nodejs` doesn't support Yarn workspaces (https://github.com/bazelbuild/rules_nodejs/issues/266) but for now the
+only limitation is that project specific dependencies are included in main workspaces rather than in external workspace.
+- all hoisted dependencies can be accessed from single external workspace (`@npm`) and project specific dependencies are
+bounded with `filegroup`'s.
+
+### Embeding JS assets into go libraries
+`go:embed` directive (available starting from ver 1.6) is used instead of `go-bindata` for embedding JS assets into 
+go libraries. It simplifies build process for both Make and Bazel builds because it eliminates running external
+commands to generate go sources with embedded data.
+
+# Unresolved questions
+- TBD

pkg/BUILD.bazel

@@ -32,6 +32,7 @@ ALL_TESTS = [
     "//pkg/ccl/partitionccl:partitionccl_test",
     "//pkg/ccl/serverccl/diagnosticsccl:diagnosticsccl_test",
     "//pkg/ccl/serverccl:serverccl_test",
+    "//pkg/ccl/spanconfigccl:spanconfigccl_test",
     "//pkg/ccl/sqlproxyccl/denylist:denylist_test",
     "//pkg/ccl/sqlproxyccl/idle:idle_test",
     "//pkg/ccl/sqlproxyccl/tenant:tenant_test",

pkg/ccl/changefeedccl/helpers_tenant_shim_test.go

@@ -70,6 +70,7 @@ func (t *testServerShim) Clock() *hlc.Clock                     { panic(unsuppor
 func (t *testServerShim) DistSenderI() interface{}              { panic(unsupportedShimMethod) }
 func (t *testServerShim) MigrationServer() interface{}          { panic(unsupportedShimMethod) }
 func (t *testServerShim) SpanConfigAccessor() interface{}       { panic(unsupportedShimMethod) }
+func (t *testServerShim) SpanConfigSQLTranslator() interface{}  { panic(unsupportedShimMethod) }
 func (t *testServerShim) SQLServer() interface{}                { panic(unsupportedShimMethod) }
 func (t *testServerShim) SQLLivenessProvider() interface{}      { panic(unsupportedShimMethod) }
 func (t *testServerShim) StartupMigrationsManager() interface{} { panic(unsupportedShimMethod) }

pkg/ccl/spanconfigccl/BUILD.bazel

@@ -0,0 +1,36 @@
+load("@io_bazel_rules_go//go:def.bzl", "go_test")
+
+go_test(
+    name = "spanconfigccl_test",
+    srcs = [
+        "datadriven_test.go",
+        "main_test.go",
+    ],
+    data = glob(["testdata/**"]),
+    deps = [
+        "//pkg/base",
+        "//pkg/ccl/partitionccl",
+        "//pkg/ccl/utilccl",
+        "//pkg/config/zonepb",
+        "//pkg/kv",
+        "//pkg/roachpb:with-mocks",
+        "//pkg/security",
+        "//pkg/security/securitytest",
+        "//pkg/server",
+        "//pkg/spanconfig",
+        "//pkg/sql",
+        "//pkg/sql/catalog/catalogkv",
+        "//pkg/sql/catalog/descpb",
+        "//pkg/sql/catalog/descs",
+        "//pkg/sql/catalog/tabledesc",
+        "//pkg/sql/sem/tree",
+        "//pkg/testutils/serverutils",
+        "//pkg/testutils/sqlutils",
+        "//pkg/testutils/testcluster",
+        "//pkg/util/leaktest",
+        "//pkg/util/log",
+        "//pkg/util/randutil",
+        "@com_github_cockroachdb_datadriven//:datadriven",
+        "@com_github_stretchr_testify//require",
+    ],
+)