Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reuse of Opensearch core and Opensearch Dashboard setup for CI workflow #1212

Merged
merged 5 commits into from
Dec 6, 2022
Merged

Reuse of Opensearch core and Opensearch Dashboard setup for CI workflow #1212

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
e1ae6b8
Reuse of Opensearch core and Opensearch Dashboard setup for CI workflow
RyanL1997 Nov 29, 2022
2b88141
Reuse of Security repo action
RyanL1997 Dec 1, 2022
30aa018
Apply reusable workflow for both integtation and cyress tests
RyanL1997 Dec 1, 2022
be2ba5b
Reference to security repo
RyanL1997 Dec 2, 2022
1f9d42b
Merge branch 'main' into sec-dashboards-github-action
RyanL1997 Dec 5, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 0 additions & 3 deletions .github/actions/install-dashboards/action.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,6 @@ outputs:
description: "The directory where the plugin has been configured"
value: ${{ steps.determine-plugin-directory.outputs.plugin-directory }}


RyanL1997 marked this conversation as resolved.
Show resolved Hide resolved
runs:
using: "composite"
steps:
Expand All @@ -25,7 +24,6 @@ runs:

- id: determine-plugin-directory
run: echo "::set-output name=plugin-directory::./OpenSearch-Dashboards/plugins/${{ inputs.plugin_name }}"
# run: echo "plugin-directory=./OpenSearch-Dashboards/plugins/${{ inputs.plugin_name }}" >> $GITHUB_OUTPUT
shell: bash

- uses: actions/checkout@v2
Expand All @@ -42,7 +40,6 @@ runs:
- uses: actions/checkout@v2
RyanL1997 marked this conversation as resolved.
Show resolved Hide resolved
with:
path: ${{ steps.determine-plugin-directory.outputs.plugin-directory }}
# ref: ${{ github.ref }}

- id: osd-version
continue-on-error: true
Expand Down
96 changes: 23 additions & 73 deletions .github/workflows/cypress-test.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,95 +9,45 @@ env:
START_CMD: 'node ../scripts/opensearch_dashboards --dev --no-base-path --no-watch --opensearch_security.multitenancy.enable_aggregation_view=true'
OPENSEARCH_SNAPSHOT_CMD: 'node ../scripts/opensearch snapshot'
SPEC: 'cypress/integration/plugins/security-dashboards-plugin/aggregation_view.js,'
OPENSEARCH_VERSION: 3.0.0
PLUGIN_NAME: opensearch-security
PLUGIN_VERSION: 3.0.0.0

jobs:
tests:
name: Run aggregation view cypress test
name: Run Cypress tests
runs-on: ubuntu-latest
RyanL1997 marked this conversation as resolved.
Show resolved Hide resolved
steps:
- name: Download OpenSearch Core
run: |
wget https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/3.0.0/latest/linux/x64/tar/builds/opensearch/dist/opensearch-min-3.0.0-linux-x64.tar.gz
tar -xzf opensearch-*.tar.gz
rm -f opensearch-*.tar.gz

- name: Checkout Branch
uses: actions/checkout@v2

- name: Download OpenSearch Security Plugin
run: wget -O opensearch-security.zip https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/3.0.0/latest/linux/x64/tar/builds/opensearch/plugins/opensearch-security-3.0.0.0.zip

- name: Run OpenSearch with plugin
run: wget --progress=bar:force:noscroll -O opensearch-security.zip https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/${{ env.OPENSEARCH_VERSION }}/latest/linux/x64/tar/builds/opensearch/plugins/${{ env.PLUGIN_NAME }}-${{ env.PLUGIN_VERSION }}.zip
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why --progress=bar:force:noscroll here?

Copy link
Collaborator Author

@RyanL1997 RyanL1997 Dec 3, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @DarshitChanpura thanks for the review! We use this flag because otherwise the log will look like this https://github.com/RyanL1997/security-dashboards-plugin/actions/runs/3576589551/jobs/6014577673#step:7:837. This one is so hard to track.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lets replace this with peternied/download-file@v2 in a fast-following PR, as these actions should support multi-platform which it does not


- name: Create Setup Script
if: ${{ runner.os == 'Linux' }}
run: |
cat > os-ep.sh <<EOF
yes | opensearch-plugin install file:///docker-host/security-plugin.zip
chmod +x plugins/opensearch-security/tools/install_demo_configuration.sh
yes | plugins/opensearch-security/tools/install_demo_configuration.sh
echo "plugins.security.unsupported.restapi.allow_securityconfig_modification: true" >> /opensearch/config/opensearch.yml
chown 1001:1001 -R /opensearch
su -c "/opensearch/bin/opensearch" -s /bin/bash opensearch
cat > setup.sh <<'EOF'
chmod +x ./opensearch-${{ env.OPENSEARCH_VERSION }}-SNAPSHOT/plugins/${{ env.PLUGIN_NAME }}/tools/install_demo_configuration.sh
/bin/bash -c "yes | ./opensearch-${{ env.OPENSEARCH_VERSION }}-SNAPSHOT/plugins/${{ env.PLUGIN_NAME }}/tools/install_demo_configuration.sh"
EOF
docker build -t opensearch-test:latest -f- . <<EOF
FROM ubuntu:latest
COPY --chown=1001:1001 os-ep.sh /docker-host/
COPY --chown=1001:1001 opensearch-security.zip /docker-host/security-plugin.zip
COPY --chown=1001:1001 opensearch* /opensearch/
RUN chmod +x /docker-host/os-ep.sh
RUN useradd -u 1001 -s /sbin/nologin opensearch
ENV PATH="/opensearch/bin:${PATH}"
WORKDIR /opensearch/
ENTRYPOINT /docker-host/os-ep.sh
EOF
docker run -d -p 9200:9200 -p 9600:9600 -i opensearch-test:latest

- name: Checkout OpenSearch Dashboard
uses: actions/checkout@v2
with:
path: OpenSearch-Dashboards
repository: opensearch-project/OpenSearch-Dashboards
ref: 'main'
fetch-depth: 0

- name: Create plugins dir
run: |
cd ./OpenSearch-Dashboards
mkdir -p plugins

- name: Checkout OpenSearch Dashboard Security plugin
uses: actions/checkout@v2
- name: Run Opensearch with A Single Plugin
uses: opensearch-project/security/.github/actions/start-opensearch-with-one-plugin@main
with:
path: OpenSearch-Dashboards/plugins/security-dashboards-plugin
ref: ${{ github.ref }}

- name: Check OpenSearch Running
continue-on-error: true
run: curl -XGET https://localhost:9200 -u 'admin:admin' -k
opensearch-version: ${{ env.OPENSEARCH_VERSION }}
plugin-name: ${{ env.PLUGIN_NAME }}
setup-script-name: setup

- name: Get node and yarn versions
id: versions
run: |
echo "::set-output name=node_version::$(cat ./OpenSearch-Dashboards/.node-version)"
echo "::set-output name=yarn_version::$(jq -r '.engines.yarn' ./OpenSearch-Dashboards/package.json)"

- name: Setup node
uses: actions/setup-node@v1
- name: Run Dashboard with Security Dashboards Plugin
uses: ./.github/actions/install-dashboards
with:
node-version: ${{ steps.versions.outputs.node_version }}
registry-url: 'https://registry.npmjs.org'

- name: Install correct yarn version for OpenSearch Dashboards
run: |
npm uninstall -g yarn
echo "Installing yarn ${{ steps.versions_step.outputs.yarn_version }}"
npm i -g yarn@${{ steps.versions.outputs.yarn_version }}

- name: Check OpenSearch Running
continue-on-error: true
run: curl -XGET https://localhost:9200 -u 'admin:admin' -k
plugin_name: security-dashboards-plugin

- name: Bootstrap OpenSearch Dashboards
- name: Configure and Run OpenSearch Dashboards
continue-on-error: false
run: |
cd ./OpenSearch-Dashboards
yarn osd bootstrap
RyanL1997 marked this conversation as resolved.
Show resolved Hide resolved
echo 'server.host: "0.0.0.0"' >> ./config/opensearch_dashboards.yml
echo 'opensearch.hosts: ["https://localhost:9200"]' >> ./config/opensearch_dashboards.yml
echo 'opensearch.ssl.verificationMode: none' >> ./config/opensearch_dashboards.yml
RyanL1997 marked this conversation as resolved.
Show resolved Hide resolved
Expand Down
52 changes: 21 additions & 31 deletions .github/workflows/integration-test.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,57 +5,47 @@ on: [push, pull_request]
env:
TEST_BROWSER_HEADLESS: 1
CI: 1
OPENSEARCH_VERSION: 3.0.0
PLUGIN_NAME: opensearch-security
PLUGIN_VERSION: 3.0.0.0

jobs:
tests:
name: Run integration tests
runs-on: ubuntu-latest
steps:
- name: Checkout Branch
uses: actions/checkout@v2

- uses: browser-actions/setup-geckodriver@latest
- run: geckodriver --version

- uses: browser-actions/setup-firefox@latest
- run: firefox --version

- name: Download OpenSearch Core
run: |
wget https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/3.0.0/latest/linux/x64/tar/builds/opensearch/dist/opensearch-min-3.0.0-linux-x64.tar.gz
tar -xzf opensearch-*.tar.gz
rm -f opensearch-*.tar.gz

- name: Download OpenSearch Security Plugin
run: wget -O opensearch-security.zip https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/3.0.0/latest/linux/x64/tar/builds/opensearch/plugins/opensearch-security-3.0.0.0.zip

- name: Run OpenSearch with plugin
run: wget --progress=bar:force:noscroll -O opensearch-security.zip https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/${{ env.OPENSEARCH_VERSION }}/latest/linux/x64/tar/builds/opensearch/plugins/${{ env.PLUGIN_NAME }}-${{ env.PLUGIN_VERSION }}.zip

- name: Create Setup Script
if: ${{ runner.os == 'Linux' }}
run: |
cat > os-ep.sh <<EOF
yes | opensearch-plugin install file:///docker-host/security-plugin.zip
chmod +x plugins/opensearch-security/tools/install_demo_configuration.sh
yes | plugins/opensearch-security/tools/install_demo_configuration.sh
echo "plugins.security.unsupported.restapi.allow_securityconfig_modification: true" >> /opensearch/config/opensearch.yml
chown 1001:1001 -R /opensearch
su -c "/opensearch/bin/opensearch" -s /bin/bash opensearch
EOF
docker build -t opensearch-test:latest -f- . <<EOF
FROM ubuntu:latest
COPY --chown=1001:1001 os-ep.sh /docker-host/
COPY --chown=1001:1001 opensearch-security.zip /docker-host/security-plugin.zip
COPY --chown=1001:1001 opensearch* /opensearch/
RUN chmod +x /docker-host/os-ep.sh
RUN useradd -u 1001 -s /sbin/nologin opensearch
ENV PATH="/opensearch/bin:${PATH}"
WORKDIR /opensearch/
ENTRYPOINT /docker-host/os-ep.sh
cat > setup.sh <<'EOF'
chmod +x ./opensearch-${{ env.OPENSEARCH_VERSION }}-SNAPSHOT/plugins/${{ env.PLUGIN_NAME }}/tools/install_demo_configuration.sh
/bin/bash -c "yes | ./opensearch-${{ env.OPENSEARCH_VERSION }}-SNAPSHOT/plugins/${{ env.PLUGIN_NAME }}/tools/install_demo_configuration.sh"
echo "plugins.security.unsupported.restapi.allow_securityconfig_modification: true" >> ./opensearch-${{ env.OPENSEARCH_VERSION }}-SNAPSHOT/config/opensearch.yml
EOF
docker run -d --network=host -i opensearch-test:latest

- uses: actions/checkout@v2
- name: Run Opensearch with A Single Plugin
uses: opensearch-project/security/.github/actions/start-opensearch-with-one-plugin@main
with:
opensearch-version: ${{ env.OPENSEARCH_VERSION }}
plugin-name: ${{ env.PLUGIN_NAME }}
setup-script-name: setup

- id: install-dashboards
uses: ./.github/actions/install-dashboards
with:
plugin_name: security-dashboards-plugin
plugin_name: security-dashboards-plugin

- name: Start Dashboards in background
run: node scripts/build_opensearch_dashboards_platform_plugins.js
Expand Down
3 changes: 2 additions & 1 deletion .github/workflows/unit-test.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,8 @@ jobs:
if: ${{ matrix.os == 'windows-latest' }}
run: git config --system core.longpaths true

- uses: actions/checkout@v2
- name: Checkout Branch
uses: actions/checkout@v2

- id: install-dashboards
uses: ./.github/actions/install-dashboards
Expand Down