Alphabetical ordering inconsistent between test() and check() using testthat::expect_known_output()

[benj919]

Hi All

I have written some tests using the testthat framework. I'm working in Rstudio, currently devtools 2.2.0 is installed. When running them with devtools::test() or ctrl + shit + T they pass, running them with devtools::check() or ctrl + shift + E they fail, citing a changed alphabetical order of expected outputs. The failure occurs with "." and "_" being sorted differently, though I have not tried other combinations of symbols/characters.

I have reproduced the issue in the attached minimal package. In essence the code writes a number of variables to an environment and at some point prints all names in that environment. I'm using testthat::expect_known_output() match this printed list with a previous, verified printout stored in "tests/testthat/ref.txt" (produced by running devtools::test() once before).

I realize that this issue may be caused by a lot of different things along the execution paths up to and including the language settings of the host . But as the devtools are my current entry point I'm posting it here first.

I'm running R3.6.1 on Ubuntu 18.04.3 LTS 64bit.

The files/functions as reference point:
print_env.R:

print_env <- function() {
  test_env <- new.env()
  assign("a_c", NA, envir = test_env)
  assign("a_b", NA, envir = test_env)
  assign("a.c", NA, envir = test_env)
  env_names <- ls(test_env)
  cat(env_names, sep = "\n")
}

tests/testthat/test-print_env.R:

test_that("print_env_comparison" , {
  expect_known_output(print_env(),
                      file = "ref.txt",
                      update = TRUE)
})

Running these commands either from the console or hotkeys:

devtools::test()
# Loading orderingTest
# Testing orderingTest
# ✔ |  OK F W S | Context
# ✔ |   1       | print_env
#
# ══ Results 
# 
# OK:       1
# Failed:   0
# Warnings: 0
# Skipped:  0

devtools::check()
# (omited output)
# > test_check("orderingTest")
# ── 1. Failure: print_env_comparison (@test-print_env.R#3)  
# Results have changed from known value recorded in 'ref.txt'.
# 3/3 mismatches
# x[1]: "a.c"
# y[1]: "a_b"
# 
# x[2]: "a_b"
# y[2]: "a_c"
# 
# x[3]: "a_c"
# y[3]: "a.c"

Obviously I'd expect the test to produce the same result independently on how it is called.
Attached is the minimal package, including the reference output file for the testcase.

Thank you and cheers.

orderingTest.zip

[jimhester]

This is just due to the collation setting, R CMD check always runs with a C locale in the collation, but you are likely using a UTF-8 collation, e.g. here is your example without using devtools at all.

x <- c("a_c", "a_b", "a.c")
Sys.getlocale("LC_COLLATE")
#> [1] "en_US.UTF-8"
sort(x)
#> [1] "a_b" "a_c" "a.c"
withr::with_collate("C",
  sort(x)
)
#> [1] "a.c" "a_b" "a_c"

^{Created on 2019-09-23 by the reprex package (v0.3.0)}

[benj919]

That is indeed the case and your fix solves my issue.
Thank you.

[lock]

This old issue has been automatically locked. If you believe you have found a related problem, please file a new issue (with reprex) and link to this issue. https://reprex.tidyverse.org/