# Environment

SNAP_DEBUG=1                   - show debug info
SNAP_ENFORCE_RESQUASHFS=0      - don't enforce resquashfs tests
SNAP_ENFORCE_RESQUASHFS_COMP=0 - don't enforce resquashfs compression algorithm
SNAP_FAKEROOT_RESQUASHFS=1     - use fakeroot with resquashfs tests
SNAP_DEBUG_RESQUASHFS=1        - show debug info with failed resquashfs tests
SNAP_DEBUG_RESQUASHFS=2        - drop to a shell with failed resquashfs
SNAP_FORCE_STATE_CHECK=1       - force state checks on disallowed snaps

For snap-updates-available:
RT_SEND_EMAIL=1           - enable sending emails
RT_EMAIL_FROM=<addr>      - override configured From address
RT_EMAIL_TO=<addr>        - override calculated To addresses
RT_EMAIL_SERVER=<server>  - use <server> instead of localhost
RT_EMAIL_NOPROMPT=1       - force sending of email without prompting

For extra tests:
RT_EXTRAS_PATH=/path/to/extras


# Contributing

 1. Clone locally (create origin with master branch)
    $ git clone git+ssh://<lpid>@git.launchpad.net/review-tools
    $ git branch
    * master

 2. Add personal repo/fork remote in LP
    $ git remote add <lpid> git+ssh://<lpid>@git.launchpad.net/~<lpid>/review-tools

 3. Branch and submit
    $ git fetch
    $ git checkout -b mything origin/master
    $ git push <lpid> mything

 4. Propose for merging via:
    https://code.launchpad.net/~<lpid>/review-tools/+git/review-tools/+ref/mything

 5. If a committer, once approved can commit to master with:
    $ git checkout master
    $ git fetch && git pull --ff-only
    $ git merge mything
    $ git push origin master
    $ git branch -D mything
    $ git push --delete <lpid> mything  # optional


# Tests

Runnable snap v2 tests:
- bin/snap-check-declaration: declaration tests
- bin/snap-check-lint: lint tests
- bin/snap-check-security: security tests
- bin/snap-run-checks: all tests

This gives an alternate view on bin/snap-run-checks:
- bin/snap-review

Running checks locally:
$ make check-deps  # install what it tells you
$ PYTHONPATH=$PWD ./bin/snap-review /path/to/package

Importable tests:
- reviewtools/sr_lint.py: lint tests
- reviewtools/sr_security.py: security tests
- ...

In general, add or modify tests and report by using:
 self._add_result(<type>, <name>, <message>)

Where <type> is one of 'info', 'warn', 'error'. <name> is the name of the
test (prefixed by <review_type>_), which is set when creating a SnapReview
object. After all tests are run, if there are any errors, the exit status is
'2', if there are no errors but some warnings, the exit status is '1',
otherwise it is '0.

See snap-check-skeleton and sr_skeleton.py for how to create new tests. In
short:
 * create a snap-check-<something> and a sr_<something>.py script based off of
   the skeleton. IMPORTANT: the new script must be snap-check-<something> so
   other tools that use review-tools (eg, ubuntu-sdk) can find them.
 * modify snap-check-<something> to use sr_<something>.py
 * add tests to sr_<something>.py. If you name the tests 'check_<sometest>'
   SnapReview.do_checks() will enumerate and run them automatically

To run tests, just execute:
$ ./run-tests                       # all tests
$ ./run-tests test_sr_security.py   # only security tests

Or to run a specific test:
$ python3 -m unittest reviewtools.tests.<filename>.<class>.<test>

Invocations for existing tests:

 * available.py
   $ python3 -m unittest reviewtools.tests.test_available.TestAvailable.foo

 * common.py
   $ python3 -m unittest reviewtools.tests.test_common.TestCommon.bar

 * debversion.py
   $ python3 -m unittest reviewtools.tests.test_debversion.TestDebVersion.baz

 * email.py
   $ python3 -m unittest reviewtools.tests.test_email.TestEmail.norf

 * modules.py
   $ python3 -m unittest reviewtools.tests.test_modules.TestModules.corge

 * sr_common.py
   $ python3 -m unittest reviewtools.tests.test_sr_common.TestSnapReviewCommon.qux

 * sr_declaration.py:
   $ python3 -m unittest reviewtools.tests.test_sr_declaration.TestSnapReviewDeclaration.test_all_checks_as_v2

 * sr_functional.py:
   $ python3 -m unittest reviewtools.tests.test_sr_functional.TestSnapReviewFunctional.test_all_checks_as_v2
   $ python3 -m unittest reviewtools.tests.test_sr_functional.TestSnapReviewFunctionalNoMock.quxx

 * sr_lint.py
   $ python3 -m unittest reviewtools.tests.test_sr_lint.TestSnapReviewLint.test_all_checks_as_v2
   $ python3 -m unittest reviewtools.tests.test_sr_lint.TestSnapReviewLintNoMock.blah

 * sr_security.py:
   $ python3 -m unittest reviewtools.tests.test_sr_security.TestSnapReviewSecurity.test_all_checks_as_v2
   $ python3 -m unittest reviewtools.tests.test_sr_security.TestSnapReviewSecurityNoMock.blah2

 * store.py
   $ python3 -m unittest reviewtools.tests.test_store.TestStore.blah3

 * usn.py
   $ python3 -m unittest reviewtools.tests.test_usn.TestUSN.blah4


Additional checks are:
$ ./run-flake8
$ ./run-black
$ ./tests/test.sh
$ ./tests/test.sh system  # requies 'review-tools' snap to be installed

All tests (except './tests/test.sh system') can be run with:
$ make check
$ make coverage && make coverage-report  # run unit tests with coverage

TODO: update for git hook
If you are going to develop the tools regularly, you might want to add a bzr
hook to run the testsuite before committing. Eg, add something like this to
~/.bazaar/plugins/hooks/__init__.py:

  #!/usr/bin/python
  from bzrlib.branch import Branch

  def run_tests_rt(local, master, old_revno, old_revid, new_revno, new_revid,
                   seven, eight):
      #print local, master, old_revno, old_revid, new_revno, new_revid, seven, eight
      if 'click-reviewers-tools' in master.base:
          import subprocess
          print ''
          rc = subprocess.call(['./run-tests'])
          if rc != 0:
              import sys
              sys.exit(1)

  Branch.hooks.install_named_hook('pre_commit',
                                  run_tests_rt,
                                  'click-reviewers-tools tests')

# Releases

The review-tools historically used tags for released versions and modified
debian/changelog to include what changed for that released version. The last
released version and tag that followed this was 0.48.

Today, the review-tools no longer releases versions and instead adds a git tag
to signify something that is usable for others (eg, a stable snap, the snap
store, etc). The debian/changelog file is still used and the version in it
loosely corresponds to a given tag. Eg, if there are tags for 20190930 and
20191018, then the changelog might look like:

review-tools (0.48+YYYYMMDD-1) UNRELEASED; urgency=medium

  * new thing changed since git tag from 20191018...
  ...

review-tools (0.48+20191018-1) RELEASED; urgency=medium

  * thing changed since previous git tag
  * other thing changed since previous git tag
  ...

review-tools (0.48+20190930-1) RELEASED; urgency=medium
  ...

When it is time to issue a new git tag, the UNRELEASED entry should be updated
to have the date of the git tag and then changed to RELEASED. The general
procedure for creating a new tag is:

1. use 'dch -r' and update debian/changelog to use 0.48-YYYYMMDD-# with
   RELEASED (substituting YYYYMMDD for today's date and '#' for '1'
   (incrementing if multiple tags per day))
2. git commit debian/changelog && git push
3. Create a tag with: git tag $(date --utc "+%Y%m%d-%H%MUTC")
4. Push the tag with: git push origin $(git tag | tail -1)

These steps will result in a new version of the review-tools snap available in
the --edge channel. Before promoting to --beta, make sure further validations
are run on the new version:

1. Force the review-tools snap refresh with: 'snap refresh review-tools --edge'
2. Connect to $HOME interface if not connected already: 'sudo snap connect review-tools:home'
3. Run the extensive set of available automated tests with 'make check' and
'make functest-system' (the last one tests the installed snap)
4. Run any other manual test as needed

Whenever confident with validations, promote to --beta channel with:
'snapcraft promote --from-channel edge --to-channel beta review-tools'


# Updating reviewtools/data/snapd-base-declaration.yaml

Keeping reviewtools/data/snapd-base-declaration.yaml up to date is important
for certain tests. At a high level, run 'snap debug get-base-declaration' and
then pull those changes into reviewtools/data/snapd-base-declaration.yaml. The
format of snapd-base-declaration.yaml is generally the same as what 'snap debug
get-base-declaration', except for the following:

* there is a preamble (comments) in snapd-base-declaration.yaml
* the 'plugs' and 'slots' in snapd-base-declaration.yaml should be keys in the
  "16" dictionary (for historical reasons)
* snapd-base-declaration.yaml should not contain the "dummy" test interface
* snapd-base-declaration.yaml should not contain the newline and '$builtin'
  line from the end of 'snap debug get-base-declaration'

In general can do on a system with the current stable release of snapd:
* sudo snap debug base-declaration > /path/bd
* format /path/bd based on the above
* cp /path/bd reviewtools/data/snapd-base-declaration.yaml
* review the change with 'git diff'

(TODO: a tool could mostly automate this)


# Helpful commands

Install the required snaps for the below commands:

    $ sudo snap install review-tools
    $ sudo snap disconnect review-tools:home   # optional, but good for security

To download a snap (and all of its assertions):

    $ snap download emoj
    Fetching snap "emoj"
    Fetching assertions for "emoj"
    Install the snap with:
       snap ack emoj_53.assert
       snap install emoj_53.snap

To perform a review via snap:

    $ snap-review $HOME/snap/review-tools/common/path/to/snap

(need not be in SNAP_USER_COMMON if 'home' is connected)

To unpack the snap to 'squashfs-root' in the cwd (files are owned by invoking
user unless using sudo, which isn't recommended):

    $ unsquashfs ./the.snap

Find the snap name for a given snap-id:

    $ review-tools.store-query --snap-info HjdctDwq9fm9TnLPSvUXgyXeY9zmRsB5 | grep " name:"
      name: emoj

Find the snap-id for a given snap name:

    $ review-tools.store-query --snap-info emoj | grep 'snap-id'
      snap-id: HjdctDwq9fm9TnLPSvUXgyXeY9zmRsB5

Download the snap.yaml for a given snap (and optionally channel and
architecture) from the store (defaults to latest/stable amd64):

    $ review-tools.store-query --snap-yaml emoj
    apps:
      emoj:
        command: command-emoj.wrapper
    ...

    $ review-tools.store-query --snap-yaml --channel=17.03/stable --arch=amd64 docker
    apps:
      compose:
        command: command-compose.wrapper
    ...

Download the snap-declaration for a given snap from the store:

    $ review-tools.store-query --snap-decl ufw
    ...
    plugs:
      firewall-control:
        allow-auto-connection: true
    snap-id: Jb8klqgs5djfejP5egB9Za8KYVK686Pe
    snap-name: ufw
    type: snap-declaration
    ...


# References

* https://api.snapcraft.io/docs
* https://api.snapcraft.io/docs/info.html
* https://docs.ubuntu.com/core/en/reference/assertions
* https://assertions.ubuntu.com
* https://docs.ubuntu.com/core/en/reference/rest