Executing axe in headless chrome yields different results

[mohanraj-r]

Expectation: Running a accessibility check on a page rendered by a browser in headless mode should yield the same results

Actual: Accessibility results from a headless chrome instance differs for the same test page

Motivation: Projects like axe-cli, lighthouse, puppeteer etc use headless mode. Tests in CI are often run in headless mode (inside a docker container). Difference in a11y results for the same browser would lead to inconsistent results and confusion.

axe-core version: 3.5.1
axe-selenium webdriver Java lib: 3.0

Browser and Assistive Technology versions: Chrome 80, ChromeDriver 80.0.3987.106

For Tooling issues:
- Node version: NA
- Platform:  Mac

com.deque.axe.ExampleTest#setUp:

ChromeOptions opts = new ChromeOptions();
opts.addArguments("headless");
driver = new ChromeDriver(opts);

For the same test page noticed that in reported violations from headless mode:

• for several rules, lesser number of elements are reported
• some rules like color-contrast as not reported

Is this expected? Does axe-core detect headless mode and do anything differently? Had assumed that Chrome run with headless option would render the DOM the same - isn't that true?

[straker]

Interesting. Axe-core itself shouldn't be returning different results. What are you running as the base for your results (axe extension, axe-core script on the page, etc.)? I know the axe extension runs a different set of rules than the axe-core script, which can cause a different number of results.

[mohanraj-r]

@straker Running axe-core v3.5.1 via axe-selenium-java lib v3.0

Index: src/test/java/com/deque/axe/ExampleTest.java
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
--- src/test/java/com/deque/axe/ExampleTest.java	(revision 73a5009b22afad5243d60db5f0d751de7165519a)
+++ src/test/java/com/deque/axe/ExampleTest.java	(date 1583873624632)
@@ -24,6 +24,7 @@
 import org.openqa.selenium.By;
 import org.openqa.selenium.WebDriver;
 import org.openqa.selenium.chrome.ChromeDriver;
+import org.openqa.selenium.chrome.ChromeOptions;
 
 import java.net.URL;
 
@@ -40,8 +41,10 @@
 	 */
 	@Before
 	public void setUp() {
+		final ChromeOptions chromeOptions = new ChromeOptions();
+		chromeOptions.addArguments("headless");
 		// ChromeDriver needed to test for Shadow DOM testing support
-		driver = new ChromeDriver();
+		driver = new ChromeDriver(chromeOptions);
 	}
 
 	/**
@@ -52,6 +55,14 @@
 		driver.quit();
 	}
 
+	@Test
+	public void testHeadless() {
+		driver.get("https://dequeuniversity.com/demo/mars/");
+		JSONObject responseJSON = new AXE.Builder(driver, scriptUrl).analyze();
+		JSONArray violations = responseJSON.getJSONArray("violations");
+		AXE.writeResults(testName.getMethodName(), violations);
+	}
+
 	/**
 	 * Basic test
 	 */
@@ -281,4 +292,4 @@
 			assertTrue("No violations found", false);
 		}
 	}
-}
\ No newline at end of file
+}

Uploaded output from running axe on the same page ("https://dequeuniversity.com/demo/mars/") with headless and normal mode in https://gist.github.com/mohanraj-r/bf1d99f1e2d9369ea02adce9ac33562f

[WilcoFiers]

I'm pretty sure there are things that you can't do in headless mode, that you can do in a full fledge browser. Off the top of my head, I think it was elementsFromPoint that doesn't work in headless mode ,I could be wrong about that.

Either way, axe-core doesn't guarantee it'll run exactly the same way regardless of how you run it. What you may see is some things get reported as needs review in headless mode that get tested in a normal run. Axe-core doesn't downgrade to the lowest possible configuration. That would not be beneficial.

If you find specific things of issues that get flagged in one mode that are passed in another, please open separate issues for this. That definitely would need to be addressed, but axe-core doesn't run the same way irrespective of how you are running it.

[mohanraj-r]

Thanks for the clarification @WilcoFiers
But could we document this caveat clearly given the popularity of headless mode (e.g. puppeteer), and integrations such as dequelabs/axe-puppeteer and that even dequelabs/axe-cli itself runs in headless mode (with seemingly no option to opt-out of headless mode) ?