ARROW-8064: [Dev] Implement Comment bot via Github actions

See it in action: kszucs#16 (comment)

Main drawback that is is much slower than ursabot, but we can optimize it by:
- porting crossbow to only depend on pygithub instead of libgit2 (will consume the rate limit, but should fit in)
- use caching or docker

Theoretically CROSSBOW_GITHUB_TOKEN is set as a github actions secret, see https://issues.apache.org/jira/browse/INFRA-19954
We can trigger a build once this is merged into master.

Closes #6571 from kszucs/master and squashes the following commits:

7a604a8 <Krisztián Szűcs> note that the license is BSD2
8586eb7 <Krisztián Szűcs> add license reference
def8724 <Krisztián Szűcs> RAT
a96e7e2 <Krisztián Szűcs> flake8
6f5da63 <Krisztián Szűcs> add requirements to docker whitelist
6678c2e <Krisztián Szűcs> update archery dependencies
33f65d4 <Krisztián Szűcs> revert removing the rest of the workflows
a82b879 <Krisztián Szűcs> test dep
06a7716 <Krisztián Szűcs> responses test dep
ba25229 <Krisztián Szűcs> fix archery workflow syntax
9352ee0 <Krisztián Szűcs> run archery unittests
deb857f <Krisztián Szűcs> checkout@v2 and fetch tags
215495a <Krisztián Szűcs> fix result path
748832f <Krisztián Szűcs> message formatter
ea1b7c8 <Krisztián Szűcs> no dry run
6c83b0c <Krisztián Szűcs> dry run
4789ac5 <Krisztián Szűcs> response ormatter
1b0b15d <Krisztián Szűcs> cleanup
2270a35 <Krisztián Szűcs> validate
035024f <Krisztián Szűcs> validate callback
e791c62 <Krisztián Szűcs> diag
641227f <Krisztián Szűcs> diab
b22b204 <Krisztián Szűcs> token
d95e86b <Krisztián Szűcs> path to event payload
3e9a279 <Krisztián Szűcs> pygithub
ca1592d <Krisztián Szűcs> typo
3c1358e <Krisztián Szűcs> triger event handler
55e65fa <Krisztián Szűcs> crossbow command
92568eb <Krisztián Szűcs> first draft of bot
99ea0c2 <Krisztián Szűcs> cat
3c0f16d <Krisztián Szűcs> remove all other workflows
1f8f21d <Krisztián Szűcs> diag event handling
2f613dd <Krisztián Szűcs> Check event handling (#15)

Authored-by: Krisztián Szűcs <[email protected]>
Signed-off-by: Krisztián Szűcs <[email protected]>

.dockerignore

@@ -27,6 +27,7 @@
 # include explicitly
 !ci/**
 !c_glib/Gemfile
+!dev/archery/requirements*.txt
 !python/requirements*.txt
 !python/manylinux1/**
 !python/manylinux2010/**

.github/workflows/archery.yml

@@ -0,0 +1,49 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+name: Archery
+
+on:
+  push:
+    paths:
+      - '.github/workflows/archery.yml'
+      - 'dev/archery/**'
+  pull_request:
+    paths:
+      - '.github/workflows/archery.yml'
+      - 'dev/archery/**'
+
+jobs:
+
+  archery:
+    name: Archery Unittests
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout Arrow
+        uses: actions/checkout@v1
+        with:
+          submodules: true
+      - name: Setup Python
+        uses: actions/setup-python@v1
+        with:
+          python-version: '3.7'
+      - name: Install
+        working-directory: dev/archery
+        run: pip install pytest responses ruamel.yaml -e .
+      - name: Test
+        working-directory: dev/archery
+        run: pytest -v archery

.github/workflows/comment_bot.yml

@@ -0,0 +1,62 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+name: Comment Bot
+
+on:
+  # TODO(kszucs): support pull_request_review_comment
+  issue_comment:
+    types:
+      - created
+      - edited
+
+jobs:
+
+  debug:
+    name: Debug
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout Arrow
+        uses: actions/checkout@v2
+        with:
+          path: arrow
+      # in order to generate version number based on git we need the tags
+      # fetched but checkout@v2 doesn't do that by default
+      - name: Featch Arrow tags
+        working-directory: arrow
+        run: git fetch --depth=1 origin +refs/tags/*:refs/tags/*
+      # because libgit2 is a dependency of crossbow so prefer conda
+      - name: Setup Conda
+        uses: s-weigand/setup-conda@v1
+        with:
+          update-conda: true
+          python-version: 3.7
+          conda-channels: conda-forge
+      - name: Instal Archery and Crossbow dependencies
+        run: |
+          conda install -y --file arrow/ci/conda_env_crossbow.txt pygithub
+          pip install -e arrow/dev/archery
+      - name: Handle Github comment event
+        env:
+          ARROW_GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+          CROSSBOW_GITHUB_TOKEN: ${{ secrets.CROSSBOW_GITHUB_TOKEN }}
+        run: |
+          archery trigger-bot \
+            --event-name ${{ github.event_name }} \
+            --event-payload ${{ github.event_path }}
+
+

LICENSE.txt

@@ -967,6 +967,36 @@ OR OTHER DEALINGS IN THE SOFTWARE.
 
 --------------------------------------------------------------------------------
 
+The projects includes code from the Ursabot project under the dev/archery
+directory.
+
+License: BSD 2-Clause
+
+Copyright 2019 RStudio, Inc.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are met:
+
+1. Redistributions of source code must retain the above copyright notice, this
+   list of conditions and the following disclaimer.
+
+2. Redistributions in binary form must reproduce the above copyright notice,
+   this list of conditions and the following disclaimer in the documentation
+   and/or other materials provided with the distribution.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
+ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
+WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
+FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
+SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+--------------------------------------------------------------------------------
+
 This project include code from CMake.
 
 * cpp/cmake_modules/FindGTest.cmake is based on code from CMake.

ci/conda_env_archery.yml

@@ -0,0 +1,20 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+click
+pygithub
+ruamel.yaml

ci/docker/conda-integration.dockerfile

@@ -25,7 +25,9 @@ ARG node=11
 ARG jdk=8
 ARG go=1.12
 
+COPY ci/conda_env_archery.yml /arrow/ci/
 RUN conda install -q \
+        --file arrow/ci/conda_env_archery.yml \
         numpy \
         maven=${maven} \
         nodejs=${node} \

ci/docker/linux-apt-lint.dockerfile

@@ -55,7 +55,13 @@ RUN rustup install ${rust} && \
 # Use python3 by default in scripts
 RUN ln -s /usr/bin/python3 /usr/local/bin/python && \
     ln -s /usr/bin/pip3 /usr/local/bin/pip
-RUN pip install flake8 cmake_format==0.5.2 click
+
+COPY dev/archery/requirements.txt \
+     dev/archery/requirements-lint.txt \
+     /arrow/dev/archery/
+RUN pip install \
+      -r arrow/dev/archery/requirements.txt \
+      -r arrow/dev/archery/requirements-lint.txt
 
 ENV LC_ALL=C.UTF-8 \
     LANG=C.UTF-8