buildtest tutorial on perlmutter #1338

shahzebsiddiqui · 2023-01-11T14:59:40Z

In preparation for the buildtest tutorial at ECPAM this PR as a first draft of the buildtest tutorial on Perlmutter. The docs was rendered in the PR see https://buildtest--1338.org.readthedocs.build/en/1338/buildtest_perlmutter.html or look at the CI checks below

@jscook2345: Can you please help peer-review this MR in preparation for the tutorial.

@wspear: I would like to get your thoughts on this as well. I wasn't sure to what extent we can cover 'E4S Testsuite' and 'Spack Test' on Perlmutter. Note that there will be a hands-on tutorial covering the buildtest-spack integration in https://buildtest.readthedocs.io/en/devel/buildspecs/spack.html that is performed in the container.

codecov · 2023-01-11T15:05:47Z

Codecov Report

Base: 71.03% // Head: 71.03% // No change to project coverage 👍

Coverage data is based on head (98ccb59) compared to base (b928070).
Patch has no changes to coverable lines.

❗ Current head 98ccb59 differs from pull request most recent head 9e006cc. Consider uploading reports for the commit 9e006cc to get more accurate results

Additional details and impacted files

@@           Coverage Diff           @@
##            devel    #1338   +/-   ##
=======================================
  Coverage   71.03%   71.03%           
=======================================
  Files          57       57           
  Lines        6130     6130           
  Branches     1090     1090           
=======================================
  Hits         4354     4354           
  Misses       1774     1774           
  Partials        2        2

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

prathmesh4321 · 2023-01-23T20:37:13Z

Hi @shahzebsiddiqui . I see few changes to be made.

The link to the install buildtest mentioned here is broken I guess. It gives 404 error. See https://buildtest--1338.org.readthedocs.build/en/1338/buildtest_perlmutter.html#:~:text=Next%2C%20you%20should%20Install%20buildtest%20by%20cloning%20the%20repository%20in%20your%20%24HOME%20directory.
The path to clone here https://buildtest--1338.org.readthedocs.build/en/1338/buildtest_perlmutter.html#:~:text=HOME%0Agit%20clone-,https%3A//github.com/buildtesters/buildtest,-%24HOME/buildtest%2Dnersc should be for buildtest-nersc repo instead of the buildtest repo.
While running the "buildtest build" command in Exercise 1, the path should be "$BUILDTEST_
ROOT/perlmutter_tutorial/ex1/hostname.yml --pollinterval=10" instead of "perlmutter_tutorial/ex1/hostname.yml --pollinterval=10" ? See https://buildtest--1338.org.readthedocs.build/en/1338/buildtest_perlmutter.html#:~:text=perlmutter_tutorial/ex1/hostname.yml%20%2D%2Dpollinterval%3D10

…lmutter

shahzebsiddiqui · 2023-01-23T22:44:34Z

Hi @shahzebsiddiqui . I see few changes to be made.

The link to the install buildtest mentioned here is broken I guess. It gives 404 error. See https://buildtest--1338.org.readthedocs.build/en/1338/buildtest_perlmutter.html#:~:text=Next%2C%20you%20should%20Install%20buildtest%20by%20cloning%20the%20repository%20in%20your%20%24HOME%20directory.

The path to clone here https://buildtest--1338.org.readthedocs.build/en/1338/buildtest_perlmutter.html#:~:text=HOME%0Agit%20clone-,https%3A//github.com/buildtesters/buildtest,-%24HOME/buildtest%2Dnersc should be for buildtest-nersc repo instead of the buildtest repo.

While running the "buildtest build" command in Exercise 1, the path should be "$BUILDTEST_
ROOT/perlmutter_tutorial/ex1/hostname.yml --pollinterval=10" instead of "perlmutter_tutorial/ex1/hostname.yml --pollinterval=10" ? See https://buildtest--1338.org.readthedocs.build/en/1338/buildtest_perlmutter.html#:~:text=perlmutter_tutorial/ex1/hostname.yml%20%2D%2Dpollinterval%3D10

thanks for catching these mistakes. I made the corrections. Note you can make comments in-line if you click the review button you can comment directly on the line number. You may find this link https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/reviewing-proposed-changes-in-a-pull-request helpful for reviewing PRs on Github

prathmesh4321

@shahzebsiddiqui LGTM !

jscook2345 · 2023-01-26T15:47:16Z

docs/builder.rst

@@ -49,7 +49,7 @@ is processed.
   :scale: 75 %


-For every discovered buildspecs, buildtest will validate the buildspecs in the :ref:`parse stage <_parse_buildspecs>` to
+For every discovered buildspecs, buildtest will validate the buildspecs in the :ref:`parse stage <parse_buildspecs>` to


I believe these should be buildspec instead of buildspecs since you're talking about the singular here.

jscook2345 · 2023-01-26T15:48:33Z

docs/buildtest_perlmutter.rst

+Buildtest Tutorial on Perlmutter
+===================================
+
+This tutorial will be conducted on `Perlmutter <https://docs.nersc.gov/systems/perlmutter/>`_ system. If you need account access please


Suggestion: 'conducted on the Perlmutter system.'

jscook2345 · 2023-01-26T15:49:25Z

docs/buildtest_perlmutter.rst

+Setup
+------
+
+Once you have a NERSC account, you can `connect to NERSC system <https://docs.nersc.gov/connect/>`_. You will need access to a


Suggestion: 'you can connect to any NERSC system.

jscook2345 · 2023-01-26T15:51:54Z

docs/buildtest_perlmutter.rst

+------
+
+Once you have a NERSC account, you can `connect to NERSC system <https://docs.nersc.gov/connect/>`_. You will need access to a
+terminal client and ssh into perlmutter as follows::


Suggestion: 'You will need access to ssh. With that you can connect to perlmutter as follows'

jscook2345 · 2023-01-26T15:52:14Z

docs/buildtest_perlmutter.rst

+Once you have a NERSC account, you can `connect to NERSC system <https://docs.nersc.gov/connect/>`_. You will need access to a
+terminal client and ssh into perlmutter as follows::
+
+    ssh <user>@perlmutter-p1.nersc.gov


Maybe tell them about the using the MFA + password here?

mfa not required for training accounts

jscook2345 · 2023-01-26T15:52:58Z

docs/buildtest_perlmutter.rst

+
+    module load python
+
+Next, you should :ref:`Install buildtest <installing_buildtest>` by cloning the repository in your $HOME directory.


Suggestion: 'cloning the repository into your home directory'

Suggest giving them a git command to do this

jscook2345 · 2023-01-26T15:53:58Z

docs/buildtest_perlmutter.rst

+
+Next, you should :ref:`Install buildtest <installing_buildtest>` by cloning the repository in your $HOME directory.
+
+Once you have buildtest setup, please clone the following repository https://github.com/buildtesters/buildtest-nersc in your $HOME directory as follows::


Missing instructions to setup buildtest

Suggestion: 'please clone the following repository into your home directory:'

Since the git command has the full repo you don't need to list it twice.

yeah so the setup requires them to read the Installing buildtest page, this includes creating a python virtual environment and sourcing the setup script. Instead of redocumenting i just put link to page.

docs/buildtest_perlmutter.rst

jscook2345 · 2023-01-26T15:56:58Z

docs/buildtest_perlmutter.rst

+Exercise 1: Running a Batch Job
+--------------------------------
+
+In this exercise, we will submit a batch job that will run `hostname` in the slurm cluster. Shown below is the example buildspec


Suggestion: 'will run the hostname command'.

Suggestion: 'Here is an example buildspec'

jscook2345 · 2023-01-26T15:58:17Z

docs/buildtest_perlmutter.rst

+.. literalinclude:: ../perlmutter_tutorial/ex1/hostname.yml
+   :language: yaml
+
+Let's run this test and poll interval for 10 secs::


Suggestion: "Let's run this test with a poll interval of ten seconds"

jscook2345 · 2023-01-26T15:58:53Z

docs/buildtest_perlmutter.rst

+
+   buildtest build -b $BUILDTEST_ROOT/perlmutter_tutorial/ex1/hostname.yml --pollinterval=10
+
+Once test is complete, check the output of test by running::


Suggestion: "Once the test is complete, you can check the output of the test by running"

jscook2345 · 2023-01-26T16:16:08Z

docs/buildtest_perlmutter.rst

+
+    buildtest inspect query -o hostname_perlmutter
+
+Next, let's update the test such that it runs on both **regular** and **debug** queue. You will need to update the **executor** property and


Suggestion: 'runs on both the...'

jscook2345 · 2023-01-26T16:17:17Z

docs/buildtest_perlmutter.rst

+specify a regular expression. Please refer to :ref:`Multiple Executors <multiple_executors>` for reference. You can retrieve a list of available executors
+by running ``buildtest config executors``.
+
+Once you have updated the test, please rerun the test, now you should expect to see two runs for same test.


Suggestion: "Once you have updated and re-run the test, you should now see two results"

And additionally maybe show an example result

jscook2345 · 2023-01-26T16:18:37Z

docs/buildtest_perlmutter.rst

+Exercise 2: Performing Status Check
+------------------------------------
+
+In this exercise, we will check version of Lmod via environment **LMOD_VERSION** and specify the


Suggestion: "... check the version of ..."

Suggestion: "... Lmod using the environment variable ..."

Suggestion: "... and specifying the output using a ..."

jscook2345 · 2023-01-26T16:19:45Z

docs/buildtest_perlmutter.rst

+.. literalinclude:: ../perlmutter_tutorial/ex2/module_version.yml
+   :language: yaml
+
+This buildspec is invalid, your first task is to make sure buildspec is valid. Once you have accomplished this task, try building


I don't like the idea of mixing an invalid buildspec with a new type of test. I'd rather repeat with a broken spec, but that's up to you

Suggestion: "This buildspec is invalid. Your first task is to fix it."

jscook2345 · 2023-01-26T16:21:14Z

docs/buildtest_perlmutter.rst

+   :language: yaml
+
+This buildspec is invalid, your first task is to make sure buildspec is valid. Once you have accomplished this task, try building
+the test and check the output of test. If your test passes, try updating the regular expression and see if test fails. Revert the change


Suggestion: "...try building the test and verifying its output."

jscook2345 · 2023-01-26T16:22:37Z

docs/buildtest_perlmutter.rst

+
+    buildtest buildspec find --root $HOME/buildtest-nersc/buildspecs --rebuild -q
+
+In this task you will be required to do the following


You give them instructions but you do not tell them how to do any of it. Is that intended?

yeah it was intended for them to do this exercise by learning how to use buildtest buildspec commands. We will cover the first part of tutorial on command line that is being covered in #1353

jscook2345 · 2023-01-26T16:22:58Z

docs/buildtest_perlmutter.rst

+In this task you will be required to do the following
+
+1. Find all tags
+2. List all filter and format fields


jscook2345 · 2023-01-26T16:23:07Z

docs/buildtest_perlmutter.rst

+
+1. Find all tags
+2. List all filter and format fields
+3. Format table via fields ``name``, ``description``


jscook2345 · 2023-01-26T16:23:19Z

docs/buildtest_perlmutter.rst

+1. Find all tags
+2. List all filter and format fields
+3. Format table via fields ``name``, ``description``
+4. Filter buildspec by tag ``e4s``


jscook2345 · 2023-01-26T16:23:46Z

docs/buildtest_perlmutter.rst

+Exercise 4: Querying Test Reports
+----------------------------------
+
+In this exercise you will be learn how to :ref:`query test report <test_reports>`. This can be done by


jscook2345 · 2023-01-26T16:24:01Z

docs/buildtest_perlmutter.rst

+In this exercise you will be learn how to :ref:`query test report <test_reports>`. This can be done by
+running ``buildtest report``. In this task please do the following
+
+1. List all filter and format fields


jscook2345 · 2023-01-26T16:24:34Z

docs/buildtest_perlmutter.rst

+running ``buildtest report``. In this task please do the following
+
+1. List all filter and format fields
+2. Query all test by returncode 0


jscook2345 · 2023-01-26T16:24:42Z

docs/buildtest_perlmutter.rst

+
+1. List all filter and format fields
+2. Query all test by returncode 0
+3. Query all test by tag ``e4s``


jscook2345 · 2023-01-26T16:24:58Z

docs/buildtest_perlmutter.rst

+1. List all filter and format fields
+2. Query all test by returncode 0
+3. Query all test by tag ``e4s``
+4. Print total count of failed tests


Print the total count of all failed tests

jscook2345 · 2023-01-26T16:25:19Z

docs/buildtest_perlmutter.rst

+3. Query all test by tag ``e4s``
+4. Print total count of failed tests
+
+Let's upload the test to CDASH by running the following::


Will the user be able to upload to cdash without a token of some sort?

yep this should work.

jscook2345 · 2023-01-26T16:25:58Z

docs/buildtest_perlmutter.rst

+
+    buildtest cdash upload $USER-buildtest-tutorial
+
+Take some time to analyze the output in CDASH by opening the link including PASS/FAIL test.


Remove 'including PASS/FAIL test'

jscook2345 · 2023-01-26T16:33:20Z

docs/buildtest_perlmutter.rst

+Exercise 5: Specifying Performance Checks
+--------------------------------------------
+
+In this task, we will using :ref:`performance checks <perf_checks>` to determine state of test.


we will be using

to determine what? 'state of test' does not make sense to me in context of performance tests. What should go here?

jscook2345 · 2023-01-26T16:34:16Z

docs/buildtest_perlmutter.rst

+
+In this task, we will using :ref:`performance checks <perf_checks>` to determine state of test.
+In this exercise, we will be running the STREAM benchmark. Shown below is an example buildspec that you
+will be working with


Move to previous line

jscook2345 · 2023-01-26T16:34:56Z

docs/buildtest_perlmutter.rst

+  buildtest inspect query -o stream_test
+
+Take a close look at the metrics value. In this task, you are requested to use use :ref:`assert_ge` with metric ``copy`` and
+``scale`` with reference value. For reference value please experiment with different metrics and see if test pass/fail.


...with a reference value.

For the reference value...

see if the test passes or fails.

initial prototype of buildtest tutorial on perlmutter

73b5294

pull-request-size bot added the size/M label Jan 11, 2023

add exercise for 3 and 4

20dabcd

pull-request-size bot added size/L and removed size/M labels Jan 11, 2023

shahzebsiddiqui added 4 commits January 11, 2023 14:05

add exercise 5 on performance checks

6171668

add solutions for ex3, ex4, ex5

74fcc7a

Merge branch 'devel' into buildtest_tutorial

fe51db2

fix issue with documentation build

0ea9f70

shahzebsiddiqui self-assigned this Jan 13, 2023

shahzebsiddiqui added the documentation documentation fix label Jan 23, 2023

shahzebsiddiqui added 2 commits January 23, 2023 17:39

Merge branch 'devel' into buildtest_tutorial

48d17cb

fix some typos and incorrect information in buildtest tutorial on Per…

98ccb59

…lmutter

shahzebsiddiqui requested a review from prathmesh4321 January 23, 2023 22:42

prathmesh4321 approved these changes Jan 24, 2023

View reviewed changes

fix undefined labels reported by documentation CI check

9e006cc

shahzebsiddiqui merged commit 663fc85 into devel Jan 24, 2023

shahzebsiddiqui deleted the buildtest_tutorial branch January 24, 2023 02:26

shahzebsiddiqui linked an issue Jan 24, 2023 that may be closed by this pull request

buildtest tutorial on Perlmutter #1337

Closed

jscook2345 reviewed Jan 26, 2023

View reviewed changes

docs/buildtest_perlmutter.rst Show resolved Hide resolved

jscook2345 reviewed Jan 26, 2023

View reviewed changes

shahzebsiddiqui mentioned this pull request Jan 26, 2023

update to perlmutter tutorial page #1357

Merged


		module load python

		Next, you should :ref:`Install buildtest <installing_buildtest>` by cloning the repository in your $HOME directory.


		Next, you should :ref:`Install buildtest <installing_buildtest>` by cloning the repository in your $HOME directory.

		Once you have buildtest setup, please clone the following repository https://github.com/buildtesters/buildtest-nersc in your $HOME directory as follows::


		buildtest build -b $BUILDTEST_ROOT/perlmutter_tutorial/ex1/hostname.yml --pollinterval=10

		Once test is complete, check the output of test by running::


		buildtest inspect query -o hostname_perlmutter

		Next, let's update the test such that it runs on both regular and debug queue. You will need to update the executor property and


		buildtest buildspec find --root $HOME/buildtest-nersc/buildspecs --rebuild -q

		In this task you will be required to do the following


		buildtest cdash upload $USER-buildtest-tutorial

		Take some time to analyze the output in CDASH by opening the link including PASS/FAIL test.

buildtest tutorial on perlmutter #1338

buildtest tutorial on perlmutter #1338

Conversation

shahzebsiddiqui commented Jan 11, 2023 • edited Loading

codecov bot commented Jan 11, 2023 • edited Loading

Codecov Report

prathmesh4321 commented Jan 23, 2023

shahzebsiddiqui commented Jan 23, 2023

prathmesh4321 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jscook2345 Jan 26, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shahzebsiddiqui commented Jan 11, 2023 •

edited

Loading

codecov bot commented Jan 11, 2023 •

edited

Loading

jscook2345 Jan 26, 2023 •

edited

Loading