lighthorizon: Sync with upstream master branch (#4404)

* services/ticker: ingest assets optimizations (#4218) * Add CHANGELOG entry for Horizon 2.14.0 release (#4208) (#4220) * Make sure we test reingestion for all possible operations (#4231) * services/horizon: Allow captive core to run with sqlite database (#4092) * services/horizon: Release DB connection in /paths when no longer needed (#4228) * services/horizon: Exclude trades with >10% rounding slippage from trade aggregations (#4178) * all: staticcheck fixes (#4239) * Migrate Horizon integration tests to GitHub Actions (#4242) * Fix StreamAllLiquidityPools and StreamAllOffers (#4236) * all: run builds and tests with go1.18rc1 (#4143) * all: cache go module downloads and other build and test artifacts (#3727) * services/horizon: Add LedgerHashStore to Captive-Core config (#4251) * all: migrate the rest of the CircleCI jobs to GitHub Actions (#4250) * horizon: Fix GitHub action problem with verify-range push in master (#4253) * all: fix ci ref_protected check for caching (#4254) * Switch over from CircleCI to GitHub A tions (#4256) * all: [GitHub actions] Reset the module and build cache in master/protected (#4266) * Forgot to add sudo in #4266 (#4270) * all: More go-setup github action fixes (#4274) * xdr: add instructions for generating xdr (#4280) * services/ticker: cache tomls during scraping (#4286) * services/ticker: use log fields during asset ingestion (#4288) * services/ticker: reduce size of toml cache in memory (#4289) * historyarchive: add --skip-optional flag (#3906) * all: Add Protocol 19 XDR and update StrKey to support Signed Payloads (#4279) * Replace keybase with publicnode in the stellar core config (#4291) * Fix captive core tests to write to /tmp, instead of polluting the repo (#4296) * all: remove go1.16 add go1.18 (#4284) * Rename methods and functions in submission system (#4298) * PR feedback (#4300) * Support new account fields for protocol-19. (#4294) * xdr, keypair: Add helpers to create CAP-40 decorated signatures (#4302) * services/horizon: Update txsub queue to account for new CAP-21 preconditions (#4301) * Uncomment StateVerifier test that generates account v3 extensions now that they are implemented. (#4304) * txnbuild: Add support for new CAP-21 preconditions. (#4303) * services/horizon: Support new CAP-21 transaction conditions (#4297) * txnbuild: Complete rename, avoid using XDR types in `TransactionParams`. (#4307) * all: Update Protocol 19 XDR to the latest (#4308) * services/horizon: Add a rate limit for path finding requests. (#4310) * clients/horizonclient: fix multi-parameter url for claimable balance query (#4248) * all: Fix Horizon integration tests (#4292) * horizon: Fix integration tests (#4314) * horizon: Set up protocol 19 integration tests infrastructure (#4312) * all: Change outdated CircleCI build badge (#4324) * horizon: Test new protocol 19 account fields (#4322) * all: update staticcheck to 2022.1 (#4326) * all: remove go.list and related docs (#4328) * horizon: Add transaction submission test for Protocol 19 (#4327) * Horizon v2.16.1 CHANGELOG (#4333) * Revert "Pin go versions temporarily" (#4338) * services/horizon: Use `bigint` over `timestamp` to accommodate large years (#4337) * xdr: Update xdrgen (#4341) * services/horizon: Change `min_account_sequence_age` column from `bigint` to string (#4339) * services/horizon: Bump stellar-core to v19.0.0rc1 for Horizon tests (#4345) * services/horizon: expose supported protocol version on root endpoint (#4347) * horizon: Small transaction submission refactoring (#4344) * services/horizon: Pass through nil ExtraSigners to avoid nil pointer deref (#4349) * doc: rename license file (#4350) * all: upgrade dep github.com/valyala/fasthttp (#4351) * services/horizon: Promote Stellar Core to v19.0.0 stable. (#4353) * services/horizon/integration: Precondition edge cases and V18->19 upgrade boundary. (#4354) * xdr: Synchronizes monorepo XDR with Stellar Core (#4355) * services/horizon: Properly allow nullable Protocol 19 account fields (#4357) * services/friendbot: include txhash in logs (#4359) * services/horizon: Improve transaction precondition `omitempty` behavior (#4360) * tools/horizon-cmp: Improve panic error message (#4365) * services/horizon: Merge stable v2.17.0 back into master: (#4363) * Use UNIX timestamps instead of RFC3339 strings for timebounds. (#4361) * xdrgen: remove gemfile and rakefile to just use docker for the xdrgen (#4366) * Conservatively limit the number of DB connections of integration tests (#4368) * internal/integrations: db_test should drop test db instances when finished (#4185) * GHA: Bump Core version to v19.0.1 in Horizon workflows. (#4378) * services/horizon, clients/horizonclient: Allow filtering ingested transactions by account or asset. (#4277) * Push stellar/ledger-state-diff images from Github actions (#4380) * services/horizon: Fixes copy-paste typo in `--help` text (#4383) * tools/alb-replay: Add new features to alb-replay (#4384) * services/horizon: Optimize claimable balances query to limit records earlier (#4385) * support/db, services/horizon/internal: Configure postgres client connection timeouts for read only db (#4390) * Refactor trade aggregation query. (#4389) * services/horizon/internal/db2/history: Implement StreamAllOffers using batches (#4397) * Add flag to disable path finding endpoints (#4399) Co-authored-by: stfung77 <[email protected]> Co-authored-by: Leigh McCulloch <[email protected]> Co-authored-by: Alfonso Acosta <[email protected]> Co-authored-by: Paul Bellamy <[email protected]> Co-authored-by: Bartek Nowotarski <[email protected]> Co-authored-by: tamirms <[email protected]> Co-authored-by: Alfonso Acosta <[email protected]> Co-authored-by: Graydon Hoare <[email protected]> Co-authored-by: Satyam Zode <[email protected]> Co-authored-by: Satyam Zode <[email protected]> Co-authored-by: erika-sdf <[email protected]> Co-authored-by: iateadonut <[email protected]> Co-authored-by: Shawn Reuland <[email protected]> Co-authored-by: shawn <[email protected]> Co-authored-by: Shivendra Mishra <[email protected]> Co-authored-by: Jacek Nykis <[email protected]> Co-authored-by: jacekn <[email protected]>
stellar · May 24, 2022 · 4eeba9d · 4eeba9d
1 parent ac4417a
commit 4eeba9d
Show file tree

Hide file tree

Showing 288 changed files with 12,266 additions and 3,069 deletions.
diff --git a/.circleci/config.yml b/.circleci/config.yml
diff --git a/.github/actions/setup-go/action.yml b/.github/actions/setup-go/action.yml
@@ -0,0 +1,55 @@
+name: 'Setup the Go environment'
+description: 'Installs go and restores/saves the build/module cache'
+inputs:
+  go-version:
+    required: true
+runs:
+  using: "composite"
+  steps:
+  - name: Set up Go
+    uses: actions/setup-go@v2
+    with:
+      go-version: ${{ inputs.go-version }}
+      stable: ${{ !(contains(inputs.go-version, 'beta') || contains(inputs.go-version, 'rc')) }}
+
+  # Restore original modification time of files based on the date of the most
+  # recent commit that modified them as mtimes affect the Go test cache.
+  - name: Restore modification time of checkout files
+    shell: bash
+    run: |
+      # Set a base, fixed modification time of all directories.
+      # git-restore-mtime doesn't set the mtime of all directories.
+      # (see https://github.com/MestreLion/git-tools/issues/47 for details)
+      touch -m -t '201509301646' $(find . -type d -not -path '.git/*')
+      # Restore original modification time from git. git clone sets the
+      # modification time to the current time, but Go tests that access fixtures
+      # get invalidated if their modification times change.
+      sudo apt-get install -y git-restore-mtime
+      git restore-mtime
+
+  # The PREFIX must uniquely identify the specific instance of a job executing.
+  - shell: bash
+    run: echo 'PREFIX=${{ github.workflow }}-${{ github.job }}-${{ runner.os }}-${{ inputs.go-version }}-matrix(${{ join(matrix.*,'|') }})' >> $GITHUB_ENV
+
+  # Cache the Go Modules downloaded during the job.
+  - uses: actions/cache@v2
+    with:
+      path: ~/go/pkg/mod
+      key: ${{ env.PREFIX }}-go-mod-${{ hashFiles('**/go.sum') }}
+      restore-keys: ${{ env.PREFIX }}-go-mod-
+
+  # Cache any build and test artifacts during the job, which will speed up
+  # rebuilds and cause test runs to skip tests that have no reason to rerun.
+  - uses: actions/cache@v2
+    with:
+      path: ~/.cache/go-build
+      key: ${{ env.PREFIX }}-go-build-${{ github.ref }}-${{ hashFiles('**', '!.git') }}
+      restore-keys: |
+        ${{ env.PREFIX }}-go-build-${{ github.ref }}-
+        ${{ env.PREFIX }}-go-build-
+
+  # Reset the cache for master/protected branches, to ensure they build and run the tests from zero
+  # and that the module cache is cleaned (otherwise it accumulates orphan dependencies over time).
+  - if: github.ref_protected
+    shell: bash
+    run: sudo rm -rf ~/.cache/go-build ~/go/pkg/mod
diff --git a/.github/workflows/go.yml b/.github/workflows/go.yml
@@ -11,12 +11,14 @@ jobs:
     strategy:
       matrix:
         os: [ubuntu-latest]
-        go: [1.17]
+        go: [1.18]
     runs-on: ${{ matrix.os }}
     steps:
-    - uses: actions/checkout@v2
-    - name: Set up Go
-      uses: actions/setup-go@v2
+    - uses: actions/checkout@v3
+      with:
+        # For pull requests, build and test the PR head not a merge of the PR with the destination.
+        ref: ${{ github.event.pull_request.head.sha || github.ref }}
+    - uses: ./.github/actions/setup-go
       with:
         go-version: ${{ matrix.go }}
     - run: ./gomod.sh
@@ -29,12 +31,14 @@ jobs:
     strategy:
       matrix:
         os: [ubuntu-latest]
-        go: [1.17, 1.16.7]
+        go: [1.17, 1.18]
     runs-on: ${{ matrix.os }}
     steps:
-    - uses: actions/checkout@v2
-    - name: Set up Go
-      uses: actions/setup-go@v2
+    - uses: actions/checkout@v3
+      with:
+        # For pull requests, build and test the PR head not a merge of the PR with the destination.
+        ref: ${{ github.event.pull_request.head.sha || github.ref }}
+    - uses: ./.github/actions/setup-go
       with:
         go-version: ${{ matrix.go }}
     - run: go build ./...
@@ -43,7 +47,7 @@ jobs:
     strategy:
       matrix:
         os: [ubuntu-latest]
-        go: [1.17, 1.16.7]
+        go: [1.17, 1.18]
         pg: [9.6.5, 10]
     runs-on: ${{ matrix.os }}
     services:
@@ -78,9 +82,14 @@ jobs:
       REDIS_HOST: localhost
       REDIS_PORT: 6379
     steps:
-    - uses: actions/checkout@v2
-    - name: Set up Go
-      uses: actions/setup-go@v2
+    - uses: actions/checkout@v3
+      with:
+        # For pull requests, build and test the PR head not a merge of the PR with the destination.
+        ref: ${{ github.event.pull_request.head.sha || github.ref }}
+        # We need to full history for git-restore-mtime to know what modification dates to use.
+        # Otherwise, the Go test cache will fail (due to the modification time of fixtures changing).
+        fetch-depth: '0'
+    - uses: ./.github/actions/setup-go
       with:
         go-version: ${{ matrix.go }}
     - run: go test -race -cover ./...
diff --git a/.github/workflows/horizon-master.yml b/.github/workflows/horizon-master.yml
@@ -0,0 +1,53 @@
+name: Horizon master
+
+on:
+  push:
+    branches: [master]
+
+jobs:
+
+  push-horizon-image-sha:
+    name: Push stellar/horizon:sha to DockerHub
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v3
+
+      - name: Get image tag (short sha)
+        shell: bash
+        id: get_tag
+        run: echo ::set-output name=TAG::$(git rev-parse --short ${{ github.sha }} )
+
+      - name: Login to DockerHub
+        uses: docker/login-action@bb984efc561711aaa26e433c32c3521176eae55b
+        with:
+          username: ${{ secrets.DOCKERHUB_USERNAME }}
+          password: ${{ secrets.DOCKERHUB_TOKEN }}
+
+      - name: Build and push to DockerHub
+        uses: docker/build-push-action@7f9d37fa544684fb73bfe4835ed7214c255ce02b
+        with:
+          # TODO: Commented out until we disable the CircleCI jobs
+          # push: true
+          tags: stellar/horizon:${{ steps.get_tag.outputs.TAG }}
+          file: services/horizon/docker/Dockerfile.dev
+
+  push-state-diff-image:
+    name: Push stellar/ledger-state-diff:{sha,latest} to DockerHub
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v3
+
+      - name: Login to DockerHub
+        uses: docker/login-action@bb984efc561711aaa26e433c32c3521176eae55b
+        with:
+          username: ${{ secrets.DOCKERHUB_USERNAME }}
+          password: ${{ secrets.DOCKERHUB_TOKEN }}
+
+      - name: Build and push to DockerHub
+        uses: docker/build-push-action@7f9d37fa544684fb73bfe4835ed7214c255ce02b
+        with:
+          push: true
+          tags: stellar/ledger-state-diff:${{ github.sha }},stellar/ledger-state-diff:latest
+          file: exp/tools/dump-ledger-state/Dockerfile
+          build-args: GITCOMMIT=${{ github.sha }}
+          no-cache: true
diff --git a/.github/workflows/horizon-release.yml b/.github/workflows/horizon-release.yml
@@ -0,0 +1,40 @@
+name: Horizon release
+
+on:
+  push:
+    tags: ['horizon-v*']
+
+jobs:
+
+  publish-artifacts:
+    runs-on: ubuntu-latest
+    name: Upload artifacts to GitHub release
+    steps:
+      - name: Run deprecation tests
+        shell: bash
+        run: |
+          export VERSION=${GITHUB_REF_NAME#horizon-v}
+          echo "Searching for \"Action needed in release ${VERSION}\" tags..."
+          # Negate the result so process exits with 1 if anything found
+          ! egrep -irn -A 1 --include=*.go "Action.+needed.+in.+release:.+$VERSION" ./
+
+      - uses: actions/checkout@v2
+
+      - uses: ./.github/actions/setup-go
+        with:
+          go-version: 1.17
+
+      - name: Check dependencies
+        run: ./gomod.sh
+
+      - name: Build binaries
+        run: go run ./support/scripts/build_release_artifacts/main.go
+
+      - name: Upload binaries to release
+        uses: svenstaro/upload-release-action@483c1e56f95e88835747b1c7c60581215016cbf2
+        with:
+          repo_token: ${{ secrets.GITHUB_TOKEN }}
+          tag: ${{ github.ref_name }}
+          file: ./dist/*
+          file_glob: true
+          overwrite: true
diff --git a/.github/workflows/horizon.yml b/.github/workflows/horizon.yml
@@ -0,0 +1,121 @@
+name: Horizon
+
+on:
+  push:
+    branches: [master]
+  pull_request:
+
+jobs:
+
+  integration:
+    name: Integration tests
+    strategy:
+      matrix:
+        os: [ubuntu-latest]
+        go: [1.17, 1.18]
+        pg: [9.6.5]
+        ingestion-backend: [db, captive-core, captive-core-remote-storage]
+        protocol-version: [18, 19]
+    runs-on: ${{ matrix.os }}
+    services:
+      postgres:
+        image: postgres:${{ matrix.pg }}
+        env:
+          POSTGRES_USER: postgres
+          POSTGRES_DB: postgres
+          POSTGRES_PASSWORD: postgres
+        options: >-
+          --health-cmd pg_isready
+          --health-interval 10s
+          --health-timeout 5s
+          --health-retries 5
+        ports:
+        - 5432:5432
+    env:
+      HORIZON_INTEGRATION_TESTS_ENABLED: true
+      HORIZON_INTEGRATION_TESTS_CORE_MAX_SUPPORTED_PROTOCOL: ${{ matrix.protocol-version }}
+      PROTOCOL_19_CORE_DEBIAN_PKG_VERSION: 19.0.1-923.2b38097cb.focal
+      PROTOCOL_19_CORE_DOCKER_IMG: stellar/stellar-core:19.0.1-923.2b38097cb.focal
+      PROTOCOL_18_CORE_DEBIAN_PKG_VERSION: 19.0.1-923.2b38097cb.focal
+      PROTOCOL_18_CORE_DOCKER_IMG: stellar/stellar-core:19.0.1-923.2b38097cb.focal
+      PGHOST: localhost
+      PGPORT: 5432
+      PGUSER: postgres
+      PGPASSWORD: postgres
+      PGDATABASE: postgres
+    steps:
+    - uses: actions/checkout@v3
+      with:
+        # For pull requests, build and test the PR head not a merge of the PR with the destination.
+        ref: ${{ github.event.pull_request.head.sha || github.ref }}
+        # We need to full history for git-restore-mtime to know what modification dates to use.
+        # Otherwise, the Go test cache will fail (due to the modification time of fixtures changing).
+        fetch-depth: '0'
+
+    # In order to debug the integration tests, run 'touch continue' once you connect to the ssh session
+    #
+    # - name: Setup upterm session
+    #  uses: lhotari/action-upterm@d23c2722bdab893785c9fbeae314cbf080645bd7
+    #  with:
+    #    ## limits ssh access and adds the ssh public key for the user which triggered the workflow
+    #    limit-access-to-actor: true
+    #    ## limits ssh access and adds the ssh public keys of the listed GitHub users
+    #    limit-access-to-users: <yourGithubUser>
+
+    - uses: ./.github/actions/setup-go
+      with:
+        go-version: ${{ matrix.go }}
+
+    - name: Pull and set Stellar Core image
+      shell: bash
+      run: |
+        docker pull "$PROTOCOL_${{ matrix.protocol-version }}_CORE_DOCKER_IMG"
+        echo HORIZON_INTEGRATION_TESTS_DOCKER_IMG="$PROTOCOL_${{ matrix.protocol-version }}_CORE_DOCKER_IMG" >> $GITHUB_ENV
+
+    - if: ${{ startsWith(matrix.ingestion-backend, 'captive-core') }}
+      name: Install and enable Captive Core
+      run: |
+        # Workaround for https://github.com/actions/virtual-environments/issues/5245,
+        # libc++1-8 won't be installed if another version is installed (but apt won't give you a helpul
+        # message about why the installation fails)
+        sudo apt-get remove -y libc++1-10 libc++abi1-10 || true
+        
+        sudo wget -qO - https://apt.stellar.org/SDF.asc | APT_KEY_DONT_WARN_ON_DANGEROUS_USAGE=true sudo apt-key add -
+        sudo bash -c 'echo "deb https://apt.stellar.org focal unstable" > /etc/apt/sources.list.d/SDF-unstable.list'
+        sudo apt-get update && sudo apt-get install -y stellar-core="$PROTOCOL_${{ matrix.protocol-version }}_CORE_DEBIAN_PKG_VERSION"
+        echo "Using stellar core version $(stellar-core version)"
+        echo 'HORIZON_INTEGRATION_TESTS_ENABLE_CAPTIVE_CORE=true' >> $GITHUB_ENV
+        echo 'HORIZON_INTEGRATION_TESTS_CAPTIVE_CORE_BIN=/usr/bin/stellar-core' >> $GITHUB_ENV
+
+    - if: ${{ matrix.ingestion-backend == 'captive-core-remote-storage' }}
+      name: Setup Captive Core Remote Storage
+      run: echo 'HORIZON_INTEGRATION_TESTS_CAPTIVE_CORE_USE_DB=true' >> $GITHUB_ENV
+
+    - run: go test -race -timeout 25m -v ./services/horizon/internal/integration/...
+
+  verify-range:
+    name: Test (and push) verify-range image
+    runs-on: ubuntu-latest
+    steps:
+    - uses: actions/checkout@v3
+      with:
+        # For pull requests, build and test the PR head not a merge of the PR with the destination.
+        ref:  ${{ github.event.pull_request.head.sha || github.ref }}
+
+    - name: Build and test the Verify Range Docker image
+      run: |
+        docker build -f services/horizon/docker/verify-range/Dockerfile -t stellar/horizon-verify-range services/horizon/docker/verify-range/
+        # Any range should do for basic testing, this range was chosen pretty early in history so that it only takes a few mins to run
+        docker run -e BRANCH=$(git rev-parse HEAD) -e FROM=10000063 -e TO=10000127 stellar/horizon-verify-range
+
+    # Push image
+    - if: github.ref == 'refs/heads/master'
+      name: Login to DockerHub
+      uses: docker/login-action@bb984efc561711aaa26e433c32c3521176eae55b
+      with:
+        username: ${{ secrets.DOCKERHUB_USERNAME }}
+        password: ${{ secrets.DOCKERHUB_TOKEN }}
+
+    - if: github.ref == 'refs/heads/master'
+      name: Push to DockerHub
+      run: docker push stellar/horizon-verify-range:latest
diff --git a/COPYING b/COPYING
@@ -6,10 +6,10 @@ You may obtain a copy of the License at
 
     http://www.apache.org/licenses/LICENSE-2.0
 
-As well as the file LICENSE-APACHE.txt adjacent to this file.
+As well as the file LICENSE adjacent to this file.
 
 Unless required by applicable law or agreed to in writing, software
 distributed under the License is distributed on an "AS IS" BASIS,
 WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 See the License for the specific language governing permissions and
-limitations under the License.
+limitations under the License.