Skip to content

Commit

Permalink
*: use native embed package from Go 1.16 (#1151)
Browse files Browse the repository at this point in the history
* cluster: use native embed from Go 1.16

* doc: update dev doc

* cluster: update template reading

* ci: remove reprotest 1.13

* ci: fix tests

Co-authored-by: Ti Chi Robot <[email protected]>
  • Loading branch information
AstroProfundis and ti-chi-bot authored Feb 24, 2021
1 parent 97f7579 commit 3d00875
Show file tree
Hide file tree
Showing 65 changed files with 117 additions and 318 deletions.
21 changes: 15 additions & 6 deletions .github/workflows/install.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -18,45 +18,54 @@ jobs:
with:
ref: ${{ github.event.pull_request.head.sha }}
path: go/src/github.com/${{ github.repository }}
- name: Build TiUP
working-directory: ${{ env.working-directory }}
run: make build

- name: Install And Check
working-directory: ${{ env.working-directory }}
run: |
sh install.sh
source ~/.profile
which tiup || (echo "no tiup found" && exit 1)
cp ./bin/tiup $(which tiup)
! tiup update --self | grep -i "WARN: adding root certificate"
tiup --version
local_install:
name: Local Install
runs-on: ubuntu-latest
env:
working-directory: ${{ github.workspace }}/go/src/github.com/${{ github.repository }}
steps:
- name: Set up Go 1.16
uses: actions/setup-go@v2
with:
go-version: 1.16
id: go

- name: Check out code into the Go module directory
uses: actions/checkout@v2
with:
ref: ${{ github.event.pull_request.head.sha }}
path: go/src/github.com/${{ github.repository }}

- name: Build TiUP
working-directory: ${{ env.working-directory }}
run: make build
run: make tiup

- name: Setup TiUP
run: |
mkdir -p ~/.tiup/bin
curl https://tiup-mirrors.pingcap.com/root.json -o ~/.tiup/bin/root.json
- name: Clone Mirror
working-directory: ${{ env.working-directory }}
run: ./bin/tiup mirror clone test-mirror

- name: Local Install And Check
working-directory: ${{ env.working-directory }}/test-mirror
run: |
sh local_install.sh
source ~/.profile
which tiup || (echo "no tiup found" && exit 1)
tiup --version
cp ../bin/tiup $(which tiup)
! tiup update --self | grep -i "WARN: adding root certificate"
! tiup list | grep -i "WARN: adding root certificate"
tiup --version
19 changes: 15 additions & 4 deletions .github/workflows/reprotest.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,6 @@ jobs:
strategy:
matrix:
go:
- 1.13
- 1.16

# Steps represent a sequence of tasks that will be executed as part of the job
Expand All @@ -34,8 +33,20 @@ jobs:
with:
go-version: ${{ matrix.go }}

- name: Install reprotest
run: sudo apt-get update && sudo apt-get -qy install reprotest
- name: Install reprotest and prepare
id: prepare_env
run: |
echo ::set-output name=GOROOT::$GOROOT
echo ::set-output name=GOPATH::$GOPATH
sudo apt-get update && sudo apt-get -qy install reprotest
- name: Check for reproducible build
run: sudo reprotest "make clean && BUILD_FLAG='-trimpath -buildmode=pie' make build" bin
run: |
sudo reprotest \
"make clean && \
GOROOT=${{ steps.prepare_env.GOROOT }} \
GOPATH=${{ steps.prepare_env.GOPATH }} \
PATH=$GOROOT/bin:$PATH \
BUILD_FLAG='-trimpath -buildmode=pie' \
make build" \
bin
3 changes: 0 additions & 3 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -106,9 +106,6 @@ failpoint-disable: tools/bin/failpoint-ctl
tools/bin/failpoint-ctl: go.mod
$(GO) build -o $@ github.com/pingcap/failpoint/failpoint-ctl

pkger:
$(GO) run tools/pkger/main.go -s templates -d pkg/cluster/embed

fmt:
@echo "gofmt (simplify)"
@gofmt -s -l -w $(FILES) 2>&1
Expand Down
34 changes: 18 additions & 16 deletions doc/dev/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,56 +2,58 @@

## Building

You can build TiUp on any platform that supports Go.
You can build TiUP on any platform that supports Go.

Prerequisites:

* Go (minimum version: 1.13; [installation instructions](https://golang.org/doc/install))
* Go (minimum version: 1.16; [installation instructions](https://golang.org/doc/install))
* golint (`go get -u golang.org/x/lint/golint`)
* make

To build TiUp, run `make`.
To build TiUP, run `make`.

## Running locally for development

For development, you don't want to use any global directories. You may also want to supply your own metadata. TiUp can be modified using the following environment variables:
For development, you don't want to use any global directories. You may also want to supply your own metadata. TiUP can be modified using the following environment variables:

* `TIUP_HOME` the profile directory, where TiUp stores its metadata. If not set, `~/.tiup` will be used.
* `TIUP_MIRRORS` set the location of TiUp's registry, can be a directory or URL. If not set, `https://tiup-mirrors.pingcap.com` will be used.
* `TIUP_HOME` the profile directory, where TiUP stores its metadata. If not set, `~/.tiup` will be used.
* `TIUP_MIRRORS` set the location of TiUP's registry, can be a directory or URL. If not set, `https://tiup-mirrors.pingcap.com` will be used.

> **Note**
> TiUP need a certificate file (root.json) installed in `${TIUP_HOME}/bin` directory. If this is your first time getting TiUP, you can run `curl https://tiup-mirrors.pingcap.com/root.json -o ${TIUP_HOME}/bin/root.json` to get it installed.
## Testing

TiUp has unit and integration tests; you can run both by running `make test`.
TiUP has unit and integration tests; you can run unit tests by running `make test`.

Unit tests are alongside the code they test, following the Go convention of using a `_test` suffix for test files. Integration tests are in the [tests](tests) directory.

## Architecture overview

Each TiUp command has its own executable, their source is in the [cmd](cmd) directory. The main TiUp executable is [root.go](cmd/root.go).
Each TiUP command has its own executable, their source is in the [cmd](cmd) directory. The main TiUP executable is [root.go](cmd/root.go).

The core of TiUp is defined in the [pkg](pkg) directory.
The core of TiUP is defined in the [pkg](pkg) directory.

[localdata](pkg/localdata) manages TiUp's metadata held on the user's computer.
[localdata](pkg/localdata) manages TiUP's metadata held on the user's computer.

[meta](pkg/meta) contains high-level functions for managing components.

[repository](pkg/repository) handles remote repositories.

The [set](pkg/set), [tui](pkg/tui), and [utils](pkg/utils) packages contain utility types and functions. The [version](pkg/version) package contains version data for TiUp and utility functions for handling that data.
The [set](pkg/set), [tui](pkg/tui), and [utils](pkg/utils) packages contain utility types and functions. The [version](pkg/version) package contains version data for TiUP and utility functions for handling that data.

The [mock](pkg/mock) package is a utility for testing.

[embed](embed) contains static files used by builtin components (mainly `cluster` as of now), the template files are in [embed/templates](embed/templates) directory.

Some key concepts:

* *Repository* a source of components and metadata concerning those components and TiUp in general.
* *Profile* the state of an installation of TiUp and the components it manages.
* *Component* a piece of software that can be managed by TiUp, e.g., TiDB or the playground.
* *Command* a top-level command run by TiUp, e.g., `update`, `list`.
* *Repository* a source of components and metadata concerning those components and TiUP in general.
* *Profile* the state of an installation of TiUP and the components it manages.
* *Component* a piece of software that can be managed by TiUP, e.g., TiDB or the playground.
* *Command* a top-level command run by TiUP, e.g., `update`, `list`.

### TiUp registry structure
### TiUP registry structure

* tiup-manifest.index: the manifest file in json format.
* Manifests for each component named tiup-component-$name.index, where %name is the name of the component.
Expand Down
13 changes: 13 additions & 0 deletions embed/embed.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
package embed

import (
goembed "embed"
)

//go:embed templates
var embededFiles goembed.FS

// ReadFile read the file embed.
func ReadFile(path string) ([]byte, error) {
return embededFiles.ReadFile(path)
}
12 changes: 3 additions & 9 deletions pkg/cluster/embed/embed_test.go → embed/embed_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,6 @@ import (
"io/ioutil"
"os"
"path/filepath"
"strings"
"testing"

"github.com/pingcap/check"
Expand Down Expand Up @@ -38,23 +37,18 @@ func getAllFilePaths(dir string) (paths []string, err error) {
}

// Test can read all file in /templates
// If files in /templates changed, you may need run `make pkger` to regenerate the autogen_pkger.go
func (s *embedSuite) TestCanReadTemplates(c *check.C) {
root, err := filepath.Abs("../../../")
c.Assert(err, check.IsNil)

paths, err := getAllFilePaths(filepath.Join(root, "templates"))
paths, err := getAllFilePaths("templates")
c.Assert(err, check.IsNil)
c.Assert(len(paths), check.Greater, 0)

for _, path := range paths {
embedPath := strings.TrimPrefix(path, root)
c.Log("check file: ", path, " ", embedPath)
c.Log("check file: ", path)

data, err := ioutil.ReadFile(path)
c.Assert(err, check.IsNil)

embedData, err := ReadFile(embedPath)
embedData, err := ReadFile(path)
c.Assert(err, check.IsNil)

c.Assert(embedData, check.BytesEquals, data)
Expand Down
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
4 changes: 1 addition & 3 deletions go.mod
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
module github.com/pingcap/tiup

go 1.13
go 1.16

require (
github.com/AstroProfundis/sysinfo v0.0.0-20210201035811-eb96b87c86b3
Expand Down Expand Up @@ -28,7 +28,6 @@ require (
github.com/juju/ansiterm v0.0.0-20180109212912-720a0952cc2a
github.com/lunixbochs/vtclean v1.0.0 // indirect
github.com/magiconair/properties v1.8.4
github.com/markbates/pkger v0.17.1
github.com/mattn/go-runewidth v0.0.10
github.com/otiai10/copy v1.4.2
github.com/pingcap/check v0.0.0-20200212061837-5e12011dc712
Expand All @@ -51,7 +50,6 @@ require (
github.com/spf13/pflag v1.0.5
github.com/stretchr/testify v1.7.0
github.com/tikv/pd v1.1.0-beta.0.20210105064748-f4e7924b50b6
github.com/tj/assert v0.0.0-20190920132354-ee03d75cd160
github.com/tj/go-termd v0.0.2-0.20200115111609-7f6aeb166380
github.com/xo/usql v0.7.8
go.etcd.io/etcd v0.5.0-alpha.5.0.20200824191128-ae9734ed278b
Expand Down
10 changes: 0 additions & 10 deletions go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -241,7 +241,6 @@ github.com/cznic/mathutil v0.0.0-20181122101859-297441e03548/go.mod h1:e6NPNENfs
github.com/cznic/parser v0.0.0-20160622100904-31edd927e5b1/go.mod h1:2B43mz36vGZNZEwkWi8ayRSSUXLfjL8OkbzwW4NcPMM=
github.com/cznic/sortutil v0.0.0-20181122101858-f5f958428db8/go.mod h1:q2w6Bg5jeox1B+QkJ6Wp/+Vn0G/bo3f1uY7Fn3vivIQ=
github.com/cznic/strutil v0.0.0-20171016134553-529a34b1c186/go.mod h1:AHHPPPXTw0h6pVabbcbyGRK1DckRn7r/STdZEeIDzZc=
github.com/cznic/strutil v0.0.0-20181122101858-275e90344537 h1:MZRmHqDBd0vxNwenEbKSQqRVT24d3C05ft8kduSwlqM=
github.com/cznic/strutil v0.0.0-20181122101858-275e90344537/go.mod h1:AHHPPPXTw0h6pVabbcbyGRK1DckRn7r/STdZEeIDzZc=
github.com/cznic/y v0.0.0-20170802143616-045f81c6662a/go.mod h1:1rk5VM7oSnA4vjp+hrLQ3HWHa+Y4yPCa3/CsJrcNnvs=
github.com/daaku/go.zipexe v1.0.0/go.mod h1:z8IiR6TsVLEYKwXAoE/I+8ys/sDkgTzSL0CLnGVd57E=
Expand Down Expand Up @@ -290,12 +289,9 @@ github.com/envoyproxy/go-control-plane v0.9.1-0.20191026205805-5f8ba28d4473/go.m
github.com/envoyproxy/protoc-gen-validate v0.1.0/go.mod h1:iSmxcyjqTsJpI2R4NaDN7+kN2VEUnK/pcBlmesArF7c=
github.com/erikstmartin/go-testdb v0.0.0-20160219214506-8d10e4a1bae5/go.mod h1:a2zkGnVExMxdzMo3M0Hi/3sEU+cWnZpSni0O6/Yb/P0=
github.com/facebookgo/ensure v0.0.0-20160127193407-b4ab57deab51/go.mod h1:Yg+htXGokKKdzcwhuNDwVvN+uBxDGXJ7G/VN1d8fa64=
github.com/facebookgo/ensure v0.0.0-20200202191622-63f1cf65ac4c h1:8ISkoahWXwZR41ois5lSJBSVw4D0OV19Ht/JSTzvSv0=
github.com/facebookgo/ensure v0.0.0-20200202191622-63f1cf65ac4c/go.mod h1:Yg+htXGokKKdzcwhuNDwVvN+uBxDGXJ7G/VN1d8fa64=
github.com/facebookgo/stack v0.0.0-20160209184415-751773369052 h1:JWuenKqqX8nojtoVVWjGfOF9635RETekkoH6Cc9SX0A=
github.com/facebookgo/stack v0.0.0-20160209184415-751773369052/go.mod h1:UbMTZqLaRiH3MsBH8va0n7s1pQYcu3uTb8G4tygF4Zg=
github.com/facebookgo/subset v0.0.0-20150612182917-8dac2c3c4870/go.mod h1:5tD+neXqOorC30/tWg0LCSkrqj/AR6gu8yY8/fpw1q0=
github.com/facebookgo/subset v0.0.0-20200203212716-c811ad88dec4 h1:7HZCaLC5+BZpmbhCOZJ293Lz68O7PYrF2EzeiFMwCLk=
github.com/facebookgo/subset v0.0.0-20200203212716-c811ad88dec4/go.mod h1:5tD+neXqOorC30/tWg0LCSkrqj/AR6gu8yY8/fpw1q0=
github.com/fatih/color v1.7.0/go.mod h1:Zm6kSWBoL9eyXnKyktHP6abPY2pDugNf5KwzbycvMj4=
github.com/fatih/color v1.9.0/go.mod h1:eQcE1qtQxscV5RaZvpXrrb8Drkc3/DdQ+uUYCNjL+zU=
Expand Down Expand Up @@ -406,8 +402,6 @@ github.com/gobuffalo/gitgen v0.0.0-20190315122116-cc086187d211/go.mod h1:vEHJk/E
github.com/gobuffalo/gogen v0.0.0-20190315121717-8f38393713f5/go.mod h1:V9QVDIxsgKNZs6L2IYiGR8datgMhB577vzTDqypH360=
github.com/gobuffalo/gogen v0.1.0/go.mod h1:8NTelM5qd8RZ15VjQTFkAW6qOMx5wBbW4dSCS3BY8gg=
github.com/gobuffalo/gogen v0.1.1/go.mod h1:y8iBtmHmGc4qa3urIyo1shvOD8JftTtfcKi+71xfDNE=
github.com/gobuffalo/here v0.6.0 h1:hYrd0a6gDmWxBM4TnrGw8mQg24iSVoIkHEk7FodQcBI=
github.com/gobuffalo/here v0.6.0/go.mod h1:wAG085dHOYqUpf+Ap+WOdrPTp5IYcDAs/x7PLa8Y5fM=
github.com/gobuffalo/logger v0.0.0-20190315122211-86e12af44bc2/go.mod h1:QdxcLw541hSGtBnhUc4gaNIXRjiDppFGaDqzbrBd3v8=
github.com/gobuffalo/mapi v1.0.1/go.mod h1:4VAGh89y6rVOvm5A8fKFxYG+wIW6LO1FMTG9hnKStFc=
github.com/gobuffalo/mapi v1.0.2/go.mod h1:4VAGh89y6rVOvm5A8fKFxYG+wIW6LO1FMTG9hnKStFc=
Expand Down Expand Up @@ -708,8 +702,6 @@ github.com/mailru/easyjson v0.0.0-20190626092158-b2ccc519800e/go.mod h1:C1wdFJiN
github.com/mailru/easyjson v0.7.1/go.mod h1:KAzv3t3aY1NaHWoQz1+4F1ccyAH66Jk7yos7ldAVICs=
github.com/maratori/testpackage v1.0.1/go.mod h1:ddKdw+XG0Phzhx8BFDTKgpWP4i7MpApTE5fXSKAqwDU=
github.com/markbates/oncer v0.0.0-20181203154359-bf2de49a0be2/go.mod h1:Ld9puTsIW75CHf65OeIOkyKbteujpZVXDpWK6YGZbxE=
github.com/markbates/pkger v0.17.1 h1:/MKEtWqtc0mZvu9OinB9UzVN9iYCwLWuyUv4Bw+PCno=
github.com/markbates/pkger v0.17.1/go.mod h1:0JoVlrol20BSywW79rN3kdFFsE5xYM+rSCQDXbLhiuI=
github.com/markbates/safe v1.0.1/go.mod h1:nAqgmRi7cY2nqMc92/bSEeQA+R4OheNU2T1kNSCBdG0=
github.com/matoous/godox v0.0.0-20190911065817-5d6d842e92eb/go.mod h1:1BELzlh859Sh1c6+90blK8lbYy0kwQf1bYlBhBysy1s=
github.com/mattn/go-adodb v0.0.1/go.mod h1:jaSTRde4bohMuQgYQPxW3xRTPtX/cZKyxPrFVseJULo=
Expand Down Expand Up @@ -846,7 +838,6 @@ github.com/openzipkin/zipkin-go v0.2.2/go.mod h1:NaW6tEwdmWMaCDZzg8sh+IBNOxHMPnh
github.com/otiai10/copy v1.4.2 h1:RTiz2sol3eoXPLF4o+YWqEybwfUa/Q2Nkc4ZIUs3fwI=
github.com/otiai10/copy v1.4.2/go.mod h1:XWfuS3CrI0R6IE0FbgHsEazaXO8G0LpMp9o8tos0x4E=
github.com/otiai10/curr v0.0.0-20150429015615-9b4961190c95/go.mod h1:9qAhocn7zKJG+0mI8eUu6xqkFDYS2kb2saOteoSB3cE=
github.com/otiai10/curr v1.0.0 h1:TJIWdbX0B+kpNagQrjgq8bCMrbhiuX73M2XwgtDMoOI=
github.com/otiai10/curr v1.0.0/go.mod h1:LskTG5wDwr8Rs+nNQ+1LlxRjAtTZZjtJW4rMXl6j4vs=
github.com/otiai10/mint v1.3.0/go.mod h1:F5AjcsTsWUqX+Na9fpHb52P8pcRX2CI6A3ctIT91xUo=
github.com/otiai10/mint v1.3.2 h1:VYWnrP5fXmz1MXvjuUvcBrXSjGE6xjON+axB/UrpO3E=
Expand Down Expand Up @@ -924,7 +915,6 @@ github.com/pingcap/go-tpc v1.0.4 h1:17gdQ+hkmbLD88575a8QcVRyKFKj6r93yqc1NQ8QYuo=
github.com/pingcap/go-tpc v1.0.4/go.mod h1:KU/Cs2kFYOr5/hQoRN1w20VccYHThipZiZfbPRPvL2s=
github.com/pingcap/go-ycsb v0.0.0-20210129115622-04d8656123e4 h1:dZgE6OnunTz0+ey2n7dzcKDTRdGFzHJvW6KI5iX7ibc=
github.com/pingcap/go-ycsb v0.0.0-20210129115622-04d8656123e4/go.mod h1:B9UJ3Lbpk4r+qFNDAeS2l6ORGkVaVwMPO1mSqDXiNQc=
github.com/pingcap/gofail v0.0.0-20181217135706-6a951c1e42c3 h1:04yuCf5NMvLU8rB2m4Qs3rynH7EYpMno3lHkewIOdMo=
github.com/pingcap/gofail v0.0.0-20181217135706-6a951c1e42c3/go.mod h1:DazNTg0PTldtpsQiT9I5tVJwV1onHMKBBgXzmJUlMns=
github.com/pingcap/goleveldb v0.0.0-20171020122428-b9ff6c35079e/go.mod h1:O17XtbryoCJhkKGbT62+L2OlrniwqiGLSqrmdHCMzZw=
github.com/pingcap/goleveldb v0.0.0-20191226122134-f82aafb29989 h1:surzm05a8C9dN8dIUmo4Be2+pMRb6f55i+UIYrluu2E=
Expand Down
Loading

0 comments on commit 3d00875

Please sign in to comment.