Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

hack/update-codegen.sh is broken on macOS 10.14.6 (Mojave) #9792

Closed
jsoref opened this issue Oct 13, 2020 · 5 comments
Closed

hack/update-codegen.sh is broken on macOS 10.14.6 (Mojave) #9792

jsoref opened this issue Oct 13, 2020 · 5 comments
Labels
area/build Build topics specifically related to Knative kind/bug Categorizes issue or PR as related to a bug.

Comments

@jsoref
Copy link
Contributor

jsoref commented Oct 13, 2020

In what area(s)?

/area build

What version of Knative?

master?

$ git describe --dirty
fatal: No names found, cannot describe anything.

Expected Behavior

running:

./hack/update-deps.sh

should work (I don't have a definition for this, as so far, I have not managed to see this)

Actual Behavior

Initially #9778 (comment):
$ hack/update-codegen.sh
hack/../vendor/knative.dev/test-infra/scripts/library.sh: line 25: conditional binary operator expected

$ hack/update-codegen.sh
Generating checksums for configmap _example keys
Generating deepcopy funcs
Generating clientset for serving:v1 serving:v1alpha1 autoscaling:v1alpha1 at knative.dev/serving/pkg/client/clientset
Generating listers for serving:v1 serving:v1alpha1 autoscaling:v1alpha1 at knative.dev/serving/pkg/client/listers
Generating informers for serving:v1 serving:v1alpha1 autoscaling:v1alpha1 at knative.dev/serving/pkg/client/informers
Generating injection for serving:v1 serving:v1alpha1 autoscaling:v1alpha1 at knative.dev/serving/pkg/client/injection
=== Update Deps for Golang
--- Go mod tidy and vendor
warning: ignoring symlink /Users/jsoref/code/knative/knative-serving/cmd/activator/kodata/VENDOR-LICENSE
warning: ignoring symlink /Users/jsoref/code/knative/knative-serving/cmd/activator/kodata/refs
warning: ignoring symlink /Users/jsoref/code/knative/knative-serving/cmd/autoscaler/kodata/VENDOR-LICENSE
warning: ignoring symlink /Users/jsoref/code/knative/knative-serving/cmd/autoscaler/kodata/refs
warning: ignoring symlink /Users/jsoref/code/knative/knative-serving/cmd/autoscaler-hpa/kodata/VENDOR-LICENSE
warning: ignoring symlink /Users/jsoref/code/knative/knative-serving/cmd/autoscaler-hpa/kodata/refs
warning: ignoring symlink /Users/jsoref/code/knative/knative-serving/cmd/controller/kodata/VENDOR-LICENSE
warning: ignoring symlink /Users/jsoref/code/knative/knative-serving/cmd/controller/kodata/refs
warning: ignoring symlink /Users/jsoref/code/knative/knative-serving/cmd/networking/nscert/kodata/VENDOR-LICENSE
warning: ignoring symlink /Users/jsoref/code/knative/knative-serving/cmd/queue/kodata/VENDOR-LICENSE
warning: ignoring symlink /Users/jsoref/code/knative/knative-serving/cmd/queue/kodata/refs
warning: ignoring symlink /Users/jsoref/code/knative/knative-serving/cmd/webhook/kodata/VENDOR-LICENSE
warning: ignoring symlink /Users/jsoref/code/knative/knative-serving/cmd/webhook/kodata/refs
warning: ignoring symlink /Users/jsoref/code/knative/knative-serving/test/performance/benchmarks/dataplane-probe/continuous/kodata/refs
warning: ignoring symlink /Users/jsoref/code/knative/knative-serving/test/performance/benchmarks/deployment-probe/continuous/kodata/refs
warning: ignoring symlink /Users/jsoref/code/knative/knative-serving/test/performance/benchmarks/load-test/continuous/kodata/refs
warning: ignoring symlink /Users/jsoref/code/knative/knative-serving/test/performance/benchmarks/rollout-probe/continuous/kodata/refs
warning: ignoring symlink /Users/jsoref/code/knative/knative-serving/test/performance/benchmarks/scale-from-zero/continuous/kodata/refs
go: downloading knative.dev/networking v0.0.0-20201009180721-8cc9a61966f9
go: downloading knative.dev/pkg v0.0.0-20201009175121-3c4df8c27293
go: downloading golang.org/x/net v0.0.0-20200904194848-62affa334b73
go: downloading k8s.io/code-generator v0.18.8
go: downloading go.opencensus.io v0.22.4
go: downloading github.com/prometheus/client_golang v0.9.2
go: downloading github.com/google/go-cmp v0.5.2
go: downloading knative.dev/caching v0.0.0-20201009023721-0a00448bbad0
go: downloading github.com/google/go-containerregistry v0.1.3
go: downloading github.com/google/mako v0.0.0-20190821191249-122f8dcef9e3
go: downloading knative.dev/test-infra v0.0.0-20201009204121-322fb08edae7
go: downloading contrib.go.opencensus.io/exporter/stackdriver v0.13.2
go: downloading google.golang.org/grpc v1.31.1
go: downloading k8s.io/gengo v0.0.0-20200205140755-e0e292d8aa12
go: downloading golang.org/x/time v0.0.0-20200630173020-3af7569d3a1e
go: downloading contrib.go.opencensus.io/exporter/prometheus v0.2.1-0.20200609204449-6bcf6f8577f0
go: downloading github.com/kelseyhightower/envconfig v1.4.0
go: downloading github.com/googleapis/gnostic v0.4.0
go: downloading go.uber.org/zap v1.15.0
go: downloading google.golang.org/genproto v0.0.0-20200904004341-0bd0a958aa1d
go: downloading golang.org/x/text v0.3.3
go: downloading github.com/openzipkin/zipkin-go v0.2.2
go: downloading github.com/mattbaird/jsonpatch v0.0.0-20171005235357-81af80346b1a
go: downloading github.com/gorilla/websocket v1.4.2
go: downloading gopkg.in/yaml.v3 v3.0.0-20190709130402-674ba3eaed22
go: downloading google.golang.org/protobuf v1.25.0
go: downloading github.com/vdemeester/k8s-pkg-credentialprovider v1.17.4
go: downloading github.com/prometheus/statsd_exporter v0.15.0
go: downloading sigs.k8s.io/structured-merge-diff v1.0.1-0.20191108220359-b1b620dd3f06
go: downloading cloud.google.com/go v0.65.0
go: downloading sigs.k8s.io/structured-merge-diff/v3 v3.0.1-0.20200706213357-43c19bbb7fba
go: downloading github.com/aws/aws-sdk-go v1.31.12
go: downloading github.com/census-instrumentation/opencensus-proto v0.3.0
go: downloading github.com/json-iterator/go v1.1.10
go: downloading gomodules.xyz/jsonpatch/v2 v2.1.0
go: downloading google.golang.org/appengine v1.6.6
go: downloading github.com/tsenart/vegeta v1.2.1-0.20190917092155-ab06ddb56e2f
go: downloading github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b
go: downloading google.golang.org/api v0.31.0
go: downloading github.com/imdario/mergo v0.3.9
go: downloading github.com/go-logr/logr v0.1.0
go: downloading github.com/influxdata/tdigest v0.0.1
go: downloading golang.org/x/tools v0.0.0-20200916195026-c9a70fc28ce3
go: downloading k8s.io/apiextensions-apiserver v0.18.8
go: downloading github.com/gorilla/mux v1.7.4
go: downloading go.uber.org/automaxprocs v1.3.0
go: downloading k8s.io/utils v0.0.0-20200603063816-c1c6865ac451
go: downloading github.com/golang/groupcache v0.0.0-20200121045136-8c9f03a8e57e
go: downloading github.com/googleapis/gax-go v2.0.2+incompatible
go: downloading golang.org/x/sys v0.0.0-20200905004654-be1d3432aa8f
go: downloading honnef.co/go/tools v0.0.1-2020.1.5
go: downloading github.com/mailru/easyjson v0.7.1-0.20191009090205-6c0755d89d1e
go: downloading github.com/prometheus/procfs v0.0.11
go: downloading github.com/jmespath/go-jmespath v0.3.0
go: downloading contrib.go.opencensus.io/exporter/zipkin v0.1.1
go: downloading github.com/Azure/go-autorest v13.4.0+incompatible
go: downloading github.com/Azure/go-autorest/autorest v0.10.2
go: downloading github.com/go-openapi/spec v0.19.6
go: downloading contrib.go.opencensus.io/exporter/ocagent v0.7.1-0.20200907061046-05415f1de66d
go: downloading go.uber.org/atomic v1.6.0
go: downloading golang.org/x/crypto v0.0.0-20200728195943-123391ffb6de
go: downloading github.com/Azure/azure-sdk-for-go v38.2.0+incompatible
go: downloading github.com/dgryski/go-gk v0.0.0-20200319235926-a69029f61654
go: downloading k8s.io/legacy-cloud-providers v0.17.4
go: downloading github.com/grpc-ecosystem/grpc-gateway v1.14.8
go: downloading github.com/streadway/quantile v0.0.0-20150917103942-b0c588724d25
go: downloading github.com/markbates/inflect v1.0.4
go: downloading github.com/bmizerany/perks v0.0.0-20141205001514-d9a9656a3a4b
go: downloading github.com/Azure/go-autorest/autorest/to v0.3.0
go: downloading golang.org/x/exp v0.0.0-20200224162631-6cc2880d07d6
go: downloading github.com/go-openapi/swag v0.19.7
go: downloading github.com/google/go-github/v27 v27.0.6
go: downloading github.com/miekg/dns v1.1.29
go: downloading github.com/docker/cli v0.0.0-20200210162036-a4bedce16568
go: downloading golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1
go: downloading github.com/c2h5oh/datasize v0.0.0-20200112174442-28bbd4740fee
go: downloading github.com/Azure/go-autorest/autorest/mocks v0.3.0
go: downloading gonum.org/v1/gonum v0.0.0-20181121035319-3f7ecaa7e8ca
go: downloading gopkg.in/check.v1 v1.0.0-20200227125254-8fa46927fb4f
go: downloading github.com/Azure/go-autorest/autorest/adal v0.8.3
go: downloading github.com/tsenart/go-tsz v0.0.0-20180814235614-0bd30b3df1c3
go: downloading github.com/onsi/ginkgo v1.12.1
go: downloading github.com/gobuffalo/envy v1.7.1
go: downloading github.com/onsi/gomega v1.10.1
go: downloading go.uber.org/multierr v1.5.0
go: downloading github.com/rogpeppe/go-internal v1.5.2
go: downloading k8s.io/klog/v2 v2.0.0
go: downloading gopkg.in/alecthomas/kingpin.v2 v2.2.6
go: downloading github.com/Azure/go-autorest/autorest/date v0.2.0
go: downloading github.com/nxadm/tail v1.4.4
go: downloading github.com/dgryski/go-lttb v0.0.0-20180810165845-318fcdf10a77
go: downloading github.com/niemeyer/pretty v0.0.0-20200227124842-a10e7caefd8e
go: downloading github.com/alecthomas/units v0.0.0-20190924025748-f65c72e2690d
go: downloading gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7
go: downloading github.com/fsnotify/fsnotify v1.4.9
go: downloading go.uber.org/tools v0.0.0-20190618225709-2cfd321de3ee
go: downloading github.com/alecthomas/template v0.0.0-20190718012654-fb15b899a751
go: downloading github.com/kr/text v0.2.0
go: downloading github.com/docker/docker v1.13.1
go: downloading gonum.org/v1/netlib v0.0.0-20181029234149-ec6d1f5cefe6
go: downloading github.com/Azure/go-autorest/autorest/validation v0.2.0
warning: ignoring symlink /Users/jsoref/code/knative/knative-serving/cmd/activator/kodata/VENDOR-LICENSE
warning: ignoring symlink /Users/jsoref/code/knative/knative-serving/cmd/activator/kodata/refs
warning: ignoring symlink /Users/jsoref/code/knative/knative-serving/cmd/autoscaler/kodata/VENDOR-LICENSE
warning: ignoring symlink /Users/jsoref/code/knative/knative-serving/cmd/autoscaler/kodata/refs
warning: ignoring symlink /Users/jsoref/code/knative/knative-serving/cmd/autoscaler-hpa/kodata/VENDOR-LICENSE
warning: ignoring symlink /Users/jsoref/code/knative/knative-serving/cmd/autoscaler-hpa/kodata/refs
warning: ignoring symlink /Users/jsoref/code/knative/knative-serving/cmd/controller/kodata/VENDOR-LICENSE
warning: ignoring symlink /Users/jsoref/code/knative/knative-serving/cmd/controller/kodata/refs
warning: ignoring symlink /Users/jsoref/code/knative/knative-serving/cmd/networking/nscert/kodata/VENDOR-LICENSE
warning: ignoring symlink /Users/jsoref/code/knative/knative-serving/cmd/queue/kodata/VENDOR-LICENSE
warning: ignoring symlink /Users/jsoref/code/knative/knative-serving/cmd/queue/kodata/refs
warning: ignoring symlink /Users/jsoref/code/knative/knative-serving/cmd/webhook/kodata/VENDOR-LICENSE
warning: ignoring symlink /Users/jsoref/code/knative/knative-serving/cmd/webhook/kodata/refs
warning: ignoring symlink /Users/jsoref/code/knative/knative-serving/test/performance/benchmarks/dataplane-probe/continuous/kodata/refs
warning: ignoring symlink /Users/jsoref/code/knative/knative-serving/test/performance/benchmarks/deployment-probe/continuous/kodata/refs
warning: ignoring symlink /Users/jsoref/code/knative/knative-serving/test/performance/benchmarks/load-test/continuous/kodata/refs
warning: ignoring symlink /Users/jsoref/code/knative/knative-serving/test/performance/benchmarks/rollout-probe/continuous/kodata/refs
warning: ignoring symlink /Users/jsoref/code/knative/knative-serving/test/performance/benchmarks/scale-from-zero/continuous/kodata/refs
--- Removing unwanted vendor files
--- Updating licenses
go: downloading github.com/google/go-licenses v0.0.0-20200602185517-f29a4c695c3d
go: github.com/google/go-licenses upgrade => v0.0.0-20200602185517-f29a4c695c3d
go: downloading github.com/otiai10/copy v1.2.0
go: downloading golang.org/x/tools v0.0.0-20191118222007-07fc4c7f2b98
go: downloading github.com/google/licenseclassifier v0.0.0-20190926221455-842c0d70d702
go: downloading golang.org/x/sys v0.0.0-20191119060738-e882bf8e40c2
go: downloading golang.org/x/net v0.0.0-20191119073136-fc4aabc6c914
go: downloading golang.org/x/crypto v0.0.0-20191117063200-497ca9f6d64f
/Users/jsoref/code/knative/knative-serving/hack/../vendor/knative.dev/test-infra/scripts/library.sh: line 503: go-licenses: command not found
--- FAIL: go-licenses failed to update licenses

Steps to Reproduce the Problem

Run hack/update-codegen.sh on macOS 10.14.6 (using the branch from #9778 / commit: af905a7)

@jsoref jsoref added the kind/bug Categorizes issue or PR as related to a bug. label Oct 13, 2020
@knative-prow-robot knative-prow-robot added the area/build Build topics specifically related to Knative label Oct 13, 2020
@jsoref jsoref mentioned this issue Oct 13, 2020
@jsoref
Copy link
Contributor Author

jsoref commented Oct 13, 2020

For reference, here's the change (partial revert of 250b0f2) that lets this run to the second failure point:

index b11cd33be..ce3b43156 100644
--- a/vendor/knative.dev/test-infra/scripts/library.sh
+++ b/vendor/knative.dev/test-infra/scripts/library.sh
@@ -22,7 +22,7 @@
 readonly KNATIVE_TESTS_PROJECT=knative-tests

 # Conveniently set GOPATH if unset
-if [[ ! -v GOPATH ]]; then
+if [[ -z "${GOPATH:-}" ]]; then
   export GOPATH="$(go env GOPATH)"
   if [[ -z "${GOPATH}" ]]; then
     echo "WARNING: GOPATH not set and go binary unable to provide it"
@@ -30,9 +30,9 @@ if [[ ! -v GOPATH ]]; then
 fi

 # Useful environment variables
-[[ -v PROW_JOB_ID ]] && IS_PROW=1 || IS_PROW=0
+[[ -n "${PROW_JOB_ID:-}" ]] && IS_PROW=1 || IS_PROW=0
 readonly IS_PROW
-[[ ! -v REPO_ROOT_DIR ]] && REPO_ROOT_DIR="$(git rev-parse --show-toplevel)"
+[[ -z "${REPO_ROOT_DIR:-}" ]] && REPO_ROOT_DIR="$(git rev-parse --show-toplevel)"
 readonly REPO_ROOT_DIR
 readonly REPO_NAME="$(basename ${REPO_ROOT_DIR})"

@@ -478,19 +478,24 @@ function start_latest_eventing_sugar_controller() {
 function run_go_tool() {
   local tool=$2
   local install_failed=0
-  if [[ -z "$(which ${tool})" ]]; then
-    local action=get
-    [[ $1 =~ ^[\./].* ]] && action=install
+  local action="get"
+  [[ $1 =~ ^[\./].* ]] && action="install"
+  # Install the tool in the following situations:
+  #   - The tool does not exist.
+  #   - The tool needs to be installed from a local path.
+  #   - Version of the tool is specificied in the given tool path.
+  # TODO(chizhg): derive a better versioning story for the tools being used.
+  if [[ -z "$(which "${tool}")" || "${action}" == "install" || "${tool}" =~ "@" ]]; then
     # Avoid running `go get` from root dir of the repository, as it can change go.sum and go.mod files.
     # See discussions in https://github.com/golang/go/issues/27643.
     if [[ ${action} == "get" && $(pwd) == "${REPO_ROOT_DIR}" ]]; then
       local temp_dir="$(mktemp -d)"
       # Swallow the output as we are returning the stdout in the end.
       pushd "${temp_dir}" > /dev/null 2>&1
-      GOFLAGS="" go ${action} "$1" || install_failed=1
+      GOFLAGS="" go ${action} $1 || install_failed=1
       popd > /dev/null 2>&1
     else
-      GOFLAGS="" go ${action} "$1" || install_failed=1
+      GOFLAGS="" go ${action} $1 || install_failed=1
     fi
   fi
   (( install_failed )) && return ${install_failed}
@@ -587,6 +592,17 @@ function run_kntest() {
   fi
 }

+# Run "kntest" tool
+# Parameters: $1..$n - parameters passed to the tool
+function run_kntest() {
+  if [[ "${REPO_NAME}" == "test-infra" ]]; then
+    go run "${REPO_ROOT_DIR}"/kntest/cmd/kntest "$@"
+  else
+    # Always run the latest version.
+    run_go_tool knative.dev/test-infra/kntest/cmd/kntest@master "$@"
+  fi
+}
+
 # Run go-licenses to update licenses.
 # Parameters: $1 - output file, relative to repo root dir.
 #             $2 - directory to inspect.
@@ -671,7 +687,7 @@ function get_canonical_path() {
 # This is implemented as a function so it can be mocked in unit tests.
 # It will fail if a file name ever contained a newline character (which is bad practice anyway)
 function list_changed_files() {
-  if [[ -v PULL_BASE_SHA ]] && [[ -v PULL_PULL_SHA ]]; then
+  if [[ ! -z "${PULL_BASE_SHA:-}" ]] && [[ ! -z "${PULL_PULL_SHA:-}" ]]; then
     # Avoid warning when there are more than 1085 files renamed:
     # https://stackoverflow.com/questions/7830728/warning-on-diff-renamelimit-variable-when-doing-git-push
     git config diff.renames 0
@@ -724,29 +740,6 @@ function get_latest_knative_yaml_source() {
   echo "https://storage.googleapis.com/knative-nightly/${repo_name}/latest/${yaml_name}.yaml"
 }

-function shellcheck_new_files() {
-  declare -a array_of_files
-  local failed=0
-  readarray -t -d '\n' array_of_files < <(list_changed_files)
-  for filename in "${array_of_files[@]}"; do
-    if echo "${filename}" | grep -q "^vendor/"; then
-      continue
-    fi
-    if file "${filename}" | grep -q "shell script"; then
-      # SC1090 is "Can't follow non-constant source"; we will scan files individually
-      if shellcheck -e SC1090 "${filename}"; then
-        echo "--- PASS: shellcheck on ${filename}"
-      else
-        echo "--- FAIL: shellcheck on ${filename}"
-        failed=1
-      fi
-    fi
-  done
-  if [[ ${failed} -eq 1 ]]; then
-    fail_script "shellcheck failures"
-  fi
-}
-
 # Initializations that depend on previous functions.
 # These MUST come last.

@n3wscott
Copy link
Contributor

Using bash 4 should solve this for you. It is why we use

#!/usr/bin/env bash

Vs

#!/bin/bash

Several other options are bash 4 only that are in use.

@julz
Copy link
Member

julz commented Oct 13, 2020

looks like we don't currently document this in DEVELOPMENT.md, Ill PR something

@jsoref
Copy link
Contributor Author

jsoref commented Oct 13, 2020

Fwiw, the reason git describe --dirty failed is because out of habit, I remove all branches+tags from my forks before cloning. (I clone a lot and generally don't need branches/tags -- I create enough on my own.)

This would have returned something vaguely useful:

$ git describe --dirty --always
9dc2ab410-dirty

@mattmoor
Copy link
Member

I think this has been resolved via @julz documentation update

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/build Build topics specifically related to Knative kind/bug Categorizes issue or PR as related to a bug.
Projects
None yet
Development

No branches or pull requests

5 participants