Automated Testing

We strive to practice Test-Driven Development (TDD) -- that means we try to write our tests first, and then write our code. While we don't always live up to this ideal, we do write automated tests for as much of our back-end code as possible, and in most cases, we do not accept pull requests for code changes that aren't covered by automated tests.

We use RSpec unit tests (located in the spec folder) and Cucumber integration tests (in the features folder).

In addition to running tests on our development environments, we also run them using two continuous integration services:

• GitHub Actions run our tests when a pull request is submitted or merged
• Codeship also runs tests when a pull request is merged, and in addition to that, it deploys the new code to our staging environment

Table of Contents Unit Tests Our RSpec Standards Integration Tests Our Cucumber Standards Running Tests Prepare the database Run tests Generate coverage Known Test Failures Pending Tests

Unit Tests

Use RSpec for unit testing Ruby code. Unit tests match to code functionality and not to features, and run quickly. They are helpful to ensure that you're not changing behavior accidentally when refactoring or adding new functionality, and that your code is working the way you'd expect it to.

Writing unit tests makes it easier to catch mistakes before committing them and reduces the burden on our human testing volunteers.

Resources on writing good tests with RSpec:

• Everyday Rails Testing with RSpec
• Better Specs
• RSpec Style Guide
• Structure of RSpec Tests Slides

Our RSpec Standards

• Use double quotes for strings
• Avoid "should" when describing behavior, e.g. it "has a valid subject line" rather than it "should have a valid subject line"
• Use create(:comment) rather than FactoryBot.create(:comment) for factories
• Use the Redirect Expectation Helper when testing redirects with flash messages

Integration Tests

We write specification in Cucumber. These specify behavior and are easy to understand if you're not a developer.

They run much more slowly than RSpec; don't introduce too many unnecessary steps, and be mindful of unnecessary database calls!

Resources on writing good tests with Cucumber:

• Write Great Cucumber Tests
• Writing Better Cucumber Scenarios; or, Why We're Deprecating FactoryGirl's Cucumber Steps

Our Cucumber Standards

• Prefer user-facing field labels to input IDs or names, e.g. When I fill in "Additional Tags" with "Tag Name" rather than When I fill in "work_freeforms" with "Tag Name"
  • Input IDs or names are fine in step definitions, e.g. fill_in("work_freeforms", with: "Tag Name")
• Prefer factories to fixtures
• Prefer Cucumber expressions to regular expressions in step definitions
  • For existing step definitions using regular expressions, prefer "(.*?)" to "([^"]*)"

Running Tests

The instructions below are for a local install.

If you are using Vagrant, please follow the testing instructions on our Vagrant page.

Prepare the database

Before you run the automated tests, you may need to prepare your test database. This is usually only necessary the first time you run tests or after running a migration:

  bundle exec rake db:test:prepare

Failing that, you can recreate the test database altogether:

  mysql -e "drop database vagrant_test;"
  mysql -e "create database vagrant_test DEFAULT CHARACTER SET utf8mb4 DEFAULT COLLATE utf8mb4_unicode_ci;"
  RAILS_ENV=test bundle exec rake db:schema:load
  RAILS_ENV=test bundle exec rake db:migrate

Run tests

• Run all RSpec tests: RAILS_ENV=test bundle exec rspec spec
• Run a specific RSpec test: RAILS_ENV=test bundle exec rspec spec/my_model_spec.rb
• Run all Cucumber tests: RAILS_ENV=test bundle exec cucumber features
• Run a specific Cucumber test: RAILS_ENV=test bundle exec cucumber features/my_archive_process.feature
• Run a specific scenario within a Cucumber test: RAILS_ENV=test bundle exec cucumber features/my_archive_process.feature:10, where :10 is the line number of the scenario
• See verbose output: add --format pretty (more info)

If you are using Docker, you can run the tests inside the container of the main app since it has the code directory mounted already and all the right dependencies installed.

You can also push your branch and run tests using GitHub Actions.

Generate coverage

Whenever you run a test, coverage results will be at coverage/index.html, which can be viewed in a browser.

You should also check out Codecov, if you need to generate coverage for the entire test suite.

Known Test Failures

Tests can fail intermittently, particularly when run on CI services. If any tests fail frequently enough, we'll document them here.

features/admins/admin_skins.feature:9 # Scenario: Admin can cache and uncache a public skin

expected to find text "The following skins were updated: public skin" in [...] (RSpec::Expectations::ExpectationNotMetError)
./features/step_definitions/web_steps.rb:137:in `block (2 levels) in <top (required)>'
./features/step_definitions/web_steps.rb:6:in `with_scope'
./features/step_definitions/web_steps.rb:136:in `/^(?:|I )should see "([^"]*)"(?: within "([^"]*)")?$/'
features/admins/admin_skins.feature:16:in `And I should see "The following skins were updated: public skin"'

features/collections/collection_anonymity.feature:461 # Scenario: Moving a work with two collections from an anonymous collection to a non-anonymous collection should reveal the creator.

expected to find text "Secret Work" in [...] (RSpec::Expectations::ExpectationNotMetError)
./features/step_definitions/web_steps.rb:137:in `block (2 levels) in <top (required)>'
./features/step_definitions/web_steps.rb:6:in `with_scope'
./features/step_definitions/web_steps.rb:136:in `/^(?:|I )should see "([^"]*)"(?: within "([^"]*)")?$/'
features/collections/collection_anonymity.feature:461:in `Then I should see "Secret Work"'

features/other_a/gift.feature:256 # Scenario: A user should be able to re-accept a gift

expected #has_no_content?("by gifter for giftee1") to return true, got false (RSpec::Expectations::ExpectationNotMetError)
./features/step_definitions/web_steps.rb:190:in `block (2 levels) in <top (required)>'
./features/step_definitions/web_steps.rb:6:in `with_scope'
./features/step_definitions/web_steps.rb:188:in `/^(?:|I )should not see "([^"]*)"(?: within "([^"]*)")?$/'
features/other_a/gift.feature:261:in `And I should not see "by gifter for giftee1"'

features/works/chapter_edit.feature:533 # Scenario: You should be able to invite a co-creator to a chapter if they allow it.

expected to find text "brenda, rusty" in "Main Content While we've done our best to make the core functionality of this site accessible without javascript, it will work better with it enabled. Please consider turning it on! Example Archive beta User Navigation Hi, brenda! My Dashboard My History My Preferences Post New Work Import Work Log Out Site Navigation Fandoms All Fandoms Uncategorized Fandoms Browse Works Bookmarks Tags Collections Search Works Bookmarks Tags People About About Us News FAQ Wrangling Guidelines Donate or Volunteer Search Works Work Search: tip: \"uchiha sasuke/uzumaki naruto\" angst kudos>10 Actions Add Chapter Edit Edit Tags Entire Work ← Previous Chapter Chapter Index Chapter Index 1. Chapter 1 2. Chapter 2 Full-page index Bookmark Cancel Bookmark Comments Share Download AZW3 EPUB MOBI PDF HTML Work Header Rating: Not Rated Archive Warning: No Archive Warnings Apply Category: Other Fandom: Stargate SG-1 Additional Tags: Scary tag Language: English Stats: Published:2022-06-12Completed:2022-06-12Words:17Chapters:2/2Hits:0 Rusty Has Two Moms rusty Chapter Management Edit Chapter Chapter 2 Chapter Text la la la la la la la la la la la Actions ↑ Top ←Previous Chapter Add To Collections Bookmark Kudos Comments Post Comment Comment as brenda (Plain text with limited HTML ?) Comment 10000 characters left Add Rusty Has Two Moms to collections Collection name(s): Submit Cancel Bookmark Bookmark × brenda, save a bookmark! Write Comments Notes The creator's summary is added automatically. Plain text with limited HTML ? 5000 characters left Your tags The creator's tags are added automatically. Comma separated, 100 characters per tag Add to collections Choose Type and Post Private bookmark Rec Footer About the Archive Site Map Diversity Statement Terms of Service DMCA Policy Contact Us Policy Questions & Abuse Reports Technical Support & Feedback Development Known Issues GPL by the OTW". (However, it was found 1 time including non-visible text.) (RSpec::Expectations::ExpectationNotMetError)
./features/step_definitions/web_steps.rb:137:in `block (2 levels) in <top (required)>'
./features/step_definitions/web_steps.rb:6:in `with_scope'
./features/step_definitions/web_steps.rb:136:in `/^(?:|I )should see "([^"]*)"(?: within "([^"]*)")?$/'
features/works/chapter_edit.feature:554:in `Then I should see "brenda, rusty"'

We also see occasional segmentation faults in the tests. They don't always happen in the same tests, but here is an example:

/home/runner/work/otwarchive/otwarchive/vendor/bundle/ruby/2.7.0/gems/webrick-1.7.0/lib/webrick/log.rb:151: [BUG] Segmentation fault at 0x0000000077359419
ruby 2.7.3p183 (2021-04-05 revision 6847ee089d) [x86_64-linux]

This seems to be a bug in Ruby 2.7's Time#strftime.

Pending Tests

Mark failing tests pending if a test is written but isn't working for known reasons, or if you need to merge something urgently and don't want to break the build.

In RSpec, do this using xit:

xit "should only get works under that tag" do
  get :index, tag_id: @fandom.name
  expect(assigns(:works).items).to include(@work)
  expect(assigns(:works).items).not_to include(@work2)
end 

In Cucumber, use the When /^"([^\"]*)" is fixed$/ step and comment out any failing steps:

Scenario: Post Without Preview
  Given I am logged in as "whoever" with password "whatever"
    And I add the work "public" to series "be_public"
    And I follow "be_public"
    And "Issue 2169" is fixed
  # Then I should not see the "title" text "Restricted" within "h2"