Merge #65397 #65871 #65938 #65943 #65945 #65946 #65947 #65949 #65950 #…

…65952 #65953 #65958

65397: cloud: add new stanza to specify custom certs for Prometheus r=taroface a=taroface

Add a stanza to the Prometheus manifest that allows the user to specify custom certs. The `cockroach-node` specifies the default node secret used by the K8s Operator, as advised by @chrisseto.

Relates to cockroachdb/cockroach-operator#469.

A doc update to the Prometheus tutorial will refer to this change.

65871: sql: fix a bug which prevented self referencing NOT VALID foreign keys r=fqazi a=ajwerner

This bug is due to both using the same object in descs.Txn (sort of) and the
fact the fact that MakeMutationComplete does not remove the mutation any
longer. I have a feeling, but have not checked, that we lost this back-
reference in older versions.

Release note (bug fix): Fixed a bug which prevented adding self-referencing
FOREIGN KEY constraints in the NOT VALID state.

65938: sql: fix bug in column backfill with virtual NOT NULL columns r=mgartner a=ajwerner

Prior to this change we'd inform the column backfiller that it needed to read
the virtual columns. These virtual columns don't exist and thus won't be read.
If the columns are marked as NOT NULL, then an assertion will fire from inside
the row fetcher. This PR fixes the bug by not requesting the virtual columns.

Fixes #65915.

Release note (bug fix): Fixed a bug which prevented adding columns to tables
which contain data and use NOT NULL virtual columns

65943: release: configure ssh key before using git r=rail a=rail

In #65555 we added check to make sure we don't try to push the same tag
twice. The `git ls-remote` command, which uses `ssh://` was missing a
step that configures the SSH key.

This patch configures SSH earlier in the process.

Release note: None

65945: authors: add JeffSwenson to authors r=JeffSwenson a=JeffSwenson

Release note: None

65946: authors: add adwittumuluri to authors r=adwittumuluri a=adwittumuluri

Release note: None

65947: authors: add todd to authors r=matthewtodd a=matthewtodd

Release note: None

65949: authors: add sarkesian to authors r=AlexTalks a=AlexTalks

Release note: None

65950: Adding Toshi to Authors r=noguchitoshi a=noguchitoshi

[title]

65952: authors: add "Duoc Nguyen" to authors r=duoclikebook a=duoclikebook

Release note: None

65953: add nancy.vargas to authors r=nancy-vargas a=nancy-vargas



65958: authors: add <livlobo> to authors r=livlobo a=livlobo

Release note: None

Co-authored-by: taroface <[email protected]>
Co-authored-by: Andrew Werner <[email protected]>
Co-authored-by: Rail Aliiev <[email protected]>
Co-authored-by: Jeff Swenson <[email protected]>
Co-authored-by: Adwit Tumuluri <[email protected]>
Co-authored-by: Matthew Todd <[email protected]>
Co-authored-by: Alex Sarkesian <[email protected]>
Co-authored-by: Toshi Noguchi <[email protected]>
Co-authored-by: Duoc Nguyen <[email protected]>
Co-authored-by: Nancy Vargas Balderas <[email protected]>
Co-authored-by: Liv Lobo <[email protected]>

AUTHORS

@@ -34,13 +34,15 @@ Adam Spindler <[email protected]>
 Adam Storm <[email protected]>
 Adam Yao <[email protected]>
 Aditya Maru <[email protected]> <@cockroachlabs.com>
+Adwit Tumuluri <[email protected]> <@cockroachlabs.com>
 Aid Idrizović <[email protected]>
 Ajaya Agrawal <[email protected]>
 Alan Acosta <[email protected]>
 Alex Barganier <[email protected]>
 Alex Gaynor <[email protected]>
 Alex Lunev <[email protected]> <[email protected]>
 Alex Robinson <[email protected]> <[email protected]> <@cockroachlabs.com>
+Alex Sarkesian <[email protected]> <[email protected]>
 Alex Yarosh <[email protected]>
 Alfonso Subiotto Marqués <[email protected]> Alfonso Subiotto Marques <[email protected]>
 Alyshan Jahani <[email protected]>
@@ -120,6 +122,7 @@ Dmitry Vorobev <[email protected]>
 Dominique Luna <[email protected]>
 Dong Liang <[email protected]>
 Drew Kimball <[email protected]> <[email protected]>
+Duoc Nguyen <[email protected]>
 Dustin Hiatt <[email protected]>
 Eamon Zhang <[email protected]> EamonZhang <[email protected]>
 Eli Lindsey <[email protected]>
@@ -168,6 +171,7 @@ Jay Kominek <[email protected]>
 Jay Lim <[email protected]> <[email protected]> <[email protected]>
 Jay Rauchenstein <[email protected]>
 Jayant Shrivastava <[email protected]>
+Jeff Swenson <[email protected]>
 Jeffrey Dallatezza <[email protected]>
 Jeffrey Xiao <[email protected]> <@cockroachlabs.com>
 Jennifer Georgevich <[email protected]>
@@ -218,6 +222,7 @@ Lasantha Pambagoda <[email protected]>
 Lauren Hirata <[email protected]> Lauren <[email protected]> lhirata <[email protected]>
 Lee Reilly <[email protected]>
 Levon Lloyd <[email protected]>
+Liv Lobo <[email protected]>
 liyanan <[email protected]>
 Lizhong <[email protected]>
 Louis Hust <[email protected]> louishust <[email protected]>
@@ -243,6 +248,7 @@ Matt Jibson <[email protected]> <@cockroachlabs.com>
 Matt Sherman <[email protected]>
 Matt Tracy <[email protected]> <[email protected]>
 Matthew O'Connor <[email protected]> <[email protected]>
+Matthew Todd <[email protected]> <[email protected]>
 Max Lang <[email protected]> <[email protected]>
 Mayank Oli <[email protected]>
 mbonaci <[email protected]>
@@ -254,6 +260,7 @@ Mo Firouz <[email protected]>
 Mohamed Elqdusy <[email protected]>
 Monica Xu <[email protected]> <[email protected]>
 Namrata Kodali <[email protected]> <[email protected]>
+Nancy Vargas Balderas <[email protected]> <[email protected]>
 Nandu Pokhrel <[email protected]> <[email protected]>
 Nate Stewart <[email protected]> Nate <[email protected]> <[email protected]>
 Nathan Johnson <[email protected]>
@@ -343,6 +350,7 @@ Thomas Hardy <[email protected]>
 Tim O'Brien <[email protected]> tim-o <[email protected]> <@cockroachlabs.com>
 Tommy Reilly <[email protected]>
 Tommy Truongchau <[email protected]> <[email protected]>
+Toshi Noguchi <[email protected]> <[email protected]>
 Timothy Chen <[email protected]>
 Tobias Grieger <[email protected]> <[email protected]> <[email protected]> <[email protected]>
 Tristan Ohlson <[email protected]> <@cockroachlabs.com>

build/release/teamcity-publish-release.sh

@@ -59,6 +59,8 @@ tc_end_block "Variable Setup"
 
 
 tc_start_block "Check remote tag"
+github_ssh_key="${GITHUB_COCKROACH_TEAMCITY_PRIVATE_SSH_KEY}"
+configure_git_ssh_key
 if git_wrapped ls-remote --exit-code --tags "ssh://[email protected]/${git_repo_for_tag}.git" "${build_name}"; then
   echo "Tag ${build_name} already exists"
   exit 1
@@ -110,8 +112,6 @@ tc_end_block "Make and push docker images"
 
 
 tc_start_block "Push release tag to GitHub"
-github_ssh_key="${GITHUB_COCKROACH_TEAMCITY_PRIVATE_SSH_KEY}"
-configure_git_ssh_key
 git_wrapped push "ssh://[email protected]/${git_repo_for_tag}.git" "$build_name"
 tc_end_block "Push release tag to GitHub"
 

cloud/kubernetes/prometheus/README.md

@@ -34,10 +34,13 @@ one another, we'd have two different prometheus jobs that had duplicated
 backends.
 * `kubectl label svc cockroachdb prometheus=cockroachdb`
 
+Check for the latest Prometheus Operator 
+[release version](https://github.com/prometheus-operator/prometheus-operator/blob/master/RELEASE.md). 
+Specify the version number in the below command.
 
 Install Prometheus Operator:
 * `kubectl apply -f
-https://raw.githubusercontent.com/coreos/prometheus-operator/release-0.20/bundle.yaml`
+https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.47.1/bundle.yaml`
 
 Ensure that the instance of prometheus-operator has started before
 continuing.  The `kubectl get` command and its desired output is below:

cloud/kubernetes/prometheus/prometheus.yaml

@@ -60,11 +60,12 @@ spec:
   - port: http
     path: /_status/vars
     tlsConfig:
-      # The HTTPS certs are signed by the kubernetes internal
-      # certificate authority.
-      caFile: "/var/run/secrets/kubernetes.io/serviceaccount/ca.crt"
-      # This overrides the hostname verification check for the admin
-      # UI port to match our quickstart secure-mode cluster setup.
+      ca:
+        secret:
+          key: ca.crt
+          # This is the secret name used by the CockroachDB Kubernetes Operator.
+          # When using a custom CA, replace this with your secret name
+          name: cockroachdb-node
       serverName: "127.0.0.1"
 ---
 # Have prometheus-operator run a replicated Prometheus cluster

pkg/sql/backfill/backfill.go

@@ -108,9 +108,13 @@ func (cb *ColumnBackfiller) init(
 		cb.updateExprs[j+len(cb.added)] = tree.DNull
 	}
 
-	// We need all the columns.
+	// We need all the non-virtual columns.
 	var valNeededForCol util.FastIntSet
-	valNeededForCol.AddRange(0, len(desc.PublicColumns())-1)
+	for i, c := range desc.PublicColumns() {
+		if !c.IsVirtual() {
+			valNeededForCol.Add(i)
+		}
+	}
 
 	tableArgs := row.FetcherTableArgs{
 		Desc:            desc,

pkg/sql/logictest/testdata/logic_test/alter_table

@@ -1812,3 +1812,19 @@ CREATE TABLE duplicate_index_test (k INT PRIMARY KEY, v INT, INDEX idx (v));
 
 statement error pgcode 42P07 duplicate index name: \"idx\"
 ALTER TABLE duplicate_index_test ADD CONSTRAINT idx UNIQUE (v)
+
+# Regression test for a bug which occurred when adding a foreign key
+# constraint which is marked NOT VALID and is self-referencing.
+subtest self_reference_fk_not_valid
+
+statement ok
+DROP TABLE IF EXISTS t;
+CREATE TABLE t (i INT PRIMARY KEY, j INT);
+ALTER TABLE t ADD CONSTRAINT fk FOREIGN KEY (j) REFERENCES t(i) NOT VALID;
+
+# Demonstrate that the constraint is enforced.
+statement error pgcode 23503 insert on table "t" violates foreign key constraint "fk"
+INSERT INTO t VALUES (1, 0)
+
+statement ok
+DROP TABLE t;

pkg/sql/logictest/testdata/logic_test/virtual_columns

@@ -1175,3 +1175,21 @@ CREATE TABLE public.t63167_b (
    CONSTRAINT "primary" PRIMARY KEY (rowid ASC),
    FAMILY "primary" (a, rowid)
 )
+
+# Test that columns backfills to tables with virtual columns work.
+subtest column_backfill
+
+statement ok
+CREATE TABLE t_65915 (i INT PRIMARY KEY, j INT AS (i + 1) VIRTUAL NOT NULL);
+INSERT INTO t_65915 VALUES (1)
+
+statement ok
+ALTER TABLE t_65915 ADD COLUMN k INT DEFAULT 42;
+
+query III
+SELECT * FROM t_65915;
+----
+1  2  42
+
+statement ok
+DROP TABLE t_65915

pkg/sql/schema_changer.go

@@ -1102,8 +1102,10 @@ func (sc *SchemaChanger) done(ctx context.Context) error {
 						return err
 					}
 					backrefTable.InboundFKs = append(backrefTable.InboundFKs, constraint.ForeignKey())
-					if err := descsCol.WriteDescToBatch(ctx, kvTrace, backrefTable, b); err != nil {
-						return err
+					if backrefTable != scTable {
+						if err := descsCol.WriteDescToBatch(ctx, kvTrace, backrefTable, b); err != nil {
+							return err
+						}
 					}
 				}
 			}