Automated Testing

We strive to practice Test-Driven Development (TDD) -- that means we try to write our tests first, and then write our code. While we don't always live up to this ideal, we do write automated tests for as much of our back-end code as possible, and in most cases, we do not accept pull requests for code changes that aren't covered by automated tests.

We use RSpec unit tests (located in the spec folder) and Cucumber integration tests (in the features folder).

In addition to running tests on our development environments, we also run them using two continuous integration services:

• GitHub Actions run our tests when a pull request is submitted or merged
• Codeship also runs tests when a pull request is merged, and in addition to that, it deploys the new code to our staging environment

Table of Contents Unit Tests Our RSpec Standards Integration Tests Our Cucumber Standards Running Tests Prepare the database Run tests Generate coverage Known Test Failures Pending Tests

Unit Tests

Use RSpec for unit testing Ruby code. Unit tests match to code functionality and not to features, and run quickly. They are helpful to ensure that you're not changing behavior accidentally when refactoring or adding new functionality, and that your code is working the way you'd expect it to.

Writing unit tests makes it easier to catch mistakes before committing them and reduces the burden on our human testing volunteers.

Resources on writing good tests with RSpec:

• Everyday Rails Testing with RSpec
• Better Specs
• RSpec Style Guide
• Structure of RSpec Tests Slides

Our RSpec Standards

• Use double quotes for strings
• Avoid "should" when describing behavior, e.g. it "has a valid subject line" rather than it "should have a valid subject line"
• Use create(:comment) rather than FactoryBot.create(:comment) for factories
• Use the Redirect Expectation Helper when testing redirects with flash messages

Integration Tests

We write specification in Cucumber. These specify behavior and are easy to understand if you're not a developer.

They run much more slowly than RSpec; don't introduce too many unnecessary steps, and be mindful of unnecessary database calls!

Resources on writing good tests with Cucumber:

• Write Great Cucumber Tests
• Writing Better Cucumber Scenarios; or, Why We're Deprecating FactoryGirl's Cucumber Steps

Our Cucumber Standards

• Prefer user-facing field labels to input IDs or names, e.g. When I fill in "Additional Tags" with "Tag Name" rather than When I fill in "work_freeforms" with "Tag Name"
  • Input IDs or names are fine in step definitions, e.g. fill_in("work_freeforms", with: "Tag Name")
• Prefer factories to fixtures
• Prefer Cucumber expressions to regular expressions in step definitions
  • For existing step definitions using regular expressions, prefer "(.*?)" to "([^"]*)"

Running Tests

The instructions below are for a local install.

If you are using Vagrant, please follow the testing instructions on our Vagrant page.

Prepare the database

Before you run the automated tests, you may need to prepare your test database. This is usually only necessary the first time you run tests or after running a migration:

  bundle exec rake db:test:prepare

Failing that, you can recreate the test database altogether:

  mysql -e "drop database vagrant_test;"
  mysql -e "create database vagrant_test DEFAULT CHARACTER SET utf8mb4 DEFAULT COLLATE utf8mb4_unicode_ci;"
  RAILS_ENV=test bundle exec rake db:schema:load
  RAILS_ENV=test bundle exec rake db:migrate

Run tests

• Run all RSpec tests: RAILS_ENV=test bundle exec rspec spec
• Run a specific RSpec test: RAILS_ENV=test bundle exec rspec spec/my_model_spec.rb
• Run all Cucumber tests: RAILS_ENV=test bundle exec cucumber features
• Run a specific Cucumber test: RAILS_ENV=test bundle exec cucumber features/my_archive_process.feature
• Run a specific scenario within a Cucumber test: RAILS_ENV=test bundle exec cucumber features/my_archive_process.feature:10, where :10 is the line number of the scenario
• See verbose output: add --format pretty (more info)

If you are using Docker, you can run the tests inside the container of the main app since it has the code directory mounted already and all the right dependencies installed.

Generate coverage

Whenever you run a test, coverage results will be at coverage/index.html, which can be viewed in a browser.

You should also check out Codecov, if you need to generate coverage for the entire test suite.

Known Test Failures

Tests can fail intermittently, particularly when run on CI services. If any tests fail frequently enough, we'll document them here.

features/other_a/gift.feature:256 # Scenario: A user should be able to re-accept a gift

expected #has_no_content?("by gifter for giftee1") to return true, got false (RSpec::Expectations::ExpectationNotMetError)
./features/step_definitions/web_steps.rb:190:in `block (2 levels) in <top (required)>'
./features/step_definitions/web_steps.rb:6:in `with_scope'
./features/step_definitions/web_steps.rb:188:in `/^(?:|I )should not see "([^"]*)"(?: within "([^"]*)")?$/'
features/other_a/gift.feature:261:in `And I should not see "by gifter for giftee1"'

cucumber features/works/work_edit_multiple.feature:157 # Scenario: User can change the pseud on multiple works at once

expected to find text "My New Pseud" in "author" (RSpec::Expectations::ExpectationNotMetError)
./features/step_definitions/web_steps.rb:131:in `block (2 levels) in <top (required)>'
./features/step_definitions/web_steps.rb:6:in `block in with_scope'
./features/step_definitions/web_steps.rb:6:in `with_scope'
./features/step_definitions/web_steps.rb:130:in `/^(?:|I )should see "([^"]*)"(?: within "([^"]*)")?$/'
features/works/work_edit_multiple.feature:167:in `Then I should see "My New Pseud" within ".byline"'

cucumber features/admins/admin_works.feature:120 # Scenario: Can edit external works

expected to find text "Admin-Added Creator" in "Show External Work | Example Archive Main Content While we've done our best to make the core functionality of this site accessible without javascript, it will work better with it enabled. Please consider turning it on! Example Archive beta Hi, testadmin-policy_and_abuse! Log Out Site Navigation Fandoms All Fandoms Uncategorized Fandoms Browse Works Bookmarks Tags Collections Search Works Bookmarks Tags People About About Us News FAQ Wrangling Guidelines Donate or Volunteer Search Works Work Search: tip: arthur merlin words>1000 sort:hits Admin Navigation Manage Users Find Users Bulk Email Search Invitations Invite New Users Manage Requests Manage Queue Admin Posts AO3 News Archive FAQ Known Issues Wrangling Guidelines Blacklist Spam Settings Banners Skins Approval Queue Approved Skins Rejected Skins Tag Wrangling Locales Activities Manage API Tokens External work was successfully updated. External Work Admin-Added Title by Zooey Glass Fandoms: Admin-Added Fandom This work isn't hosted on the Archive so this blurb might not be complete or accurate. Mature No Archive Warnings Apply M/M External Work 12 Dec 2020 Tags No Archive Warnings ApplyAdmin-Added RelationshipAdmin-Added CharacterAdmin-Added Freeform Summary Admin-added summary Bookmarks: 1 Related Works: 0 Admin Actions Hide External Work Edit External Work Delete External Work Footer About the Archive Site Map Diversity Statement Terms of Service DMCA Policy Contact Us Report Abuse Technical Support and Feedback Development Known Issues GPL by the OTW if (typeof jQuery == 'undefined') { document.write(unescape(\"%3Cscript src='/javascripts/jquery.min.js' type='text/javascript'%3E%3C/script%3E\")); document.write(unescape(\"%3Cscript src='/javascripts/jquery-ui.min.js' type='text/javascript'%3E%3C/script%3E\")); } jQuery.find.selectors.filters.focus = function(elem) { var doc = elem.ownerDocument; return elem === doc.activeElement && !!(elem.type || elem.href); } jQuery.fx.off = true; $j = jQuery.noConflict();" (RSpec::Expectations::ExpectationNotMetError)
./features/step_definitions/web_steps.rb:131:in `block (2 levels) in <top (required)>'
./features/step_definitions/web_steps.rb:6:in `with_scope'
./features/step_definitions/web_steps.rb:130:in `/^(?:|I )should see "([^"]*)"(?: within "([^"]*)")?$/'
features/admins/admin_works.feature:138:in `Then I should see "Admin-Added Creator"'

We also see occasional segmentation faults in the tests. They don't always happen in the same tests, but here is an example:

/home/runner/work/otwarchive/otwarchive/vendor/bundle/ruby/2.7.0/gems/webrick-1.7.0/lib/webrick/log.rb:151: [BUG] Segmentation fault at 0x0000000077359419
ruby 2.7.3p183 (2021-04-05 revision 6847ee089d) [x86_64-linux]

Pending Tests

Mark failing tests pending if a test is written but isn't working for known reasons, or if you need to merge something urgently and don't want to break the build.

In RSpec, do this using xit:

xit "should only get works under that tag" do
  get :index, tag_id: @fandom.name
  expect(assigns(:works).items).to include(@work)
  expect(assigns(:works).items).not_to include(@work2)
end 

In Cucumber, use the When /^"([^\"]*)" is fixed$/ step and comment out any failing steps:

Scenario: Post Without Preview
  Given I am logged in as "whoever" with password "whatever"
    And I add the work "public" to series "be_public"
    And I follow "be_public"
    And "Issue 2169" is fixed
  # Then I should not see the "title" text "Restricted" within "h2"