Test runner does not work on windows-latest, but does on ubuntu-latest

[dev-cmurphy]

DESCRIPTION

I've got an error when I try to run the latest (2.10) release of the test runner:

 ##[group***Run game-ci/[email protected]
with:
  artifactsPath: test-artifacts
  githubToken: ***
  checkName: All Modes Test Results
  coverageOptions: generateAdditionalMetrics;generateHtmlReport;generateBadgeReport;assemblyFilters:+Aleo.*-Aleo.Tests.*
  unityVersion: auto
  testMode: all
  useHostNetwork: false
env:
  UNITY_LICENSE: ***
##[endgroup***
D:\a\_actions\game-ci\unity-test-runner\v2.1.0\dist\main.js:1
index.js
^

ReferenceError: index is not defined
    at Object.<anonymous> (D:\a\_actions\game-ci\unity-test-runner\v2.1.0\dist\main.js:1:1)
    at Module._compile (node:internal/modules/cjs/loader:1101:14)
    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1153:10)
    at Module.load (node:internal/modules/cjs/loader:981:32)
    at Function.Module._load (node:internal/modules/cjs/loader:822:12)
    at Function.executeUserEntryPoint [as runMain*** (node:internal/modules/run_main:81:12)
    at node:internal/main/run_main_module:17:47

But it works fine when I run it on ubuntu-latest.
The issue seems to be with the file dist\main.js which seems to be a recent addition to the unity-test-runner repo.
How to reproduce

REPRO STEPS & EXPECTED BEHAVIOUR

Use windows-latest as the value of the job runs-on parameter.

Expected: the action runs as normal.
Actual:

D:\a\_actions\game-ci\unity-test-runner\v2.1.0\dist\main.js:1
index.js
^

ReferenceError: index is not defined
    at Object.<anonymous> (D:\a\_actions\game-ci\unity-test-runner\v2.1.0\dist\main.js:1:1)
    at Module._compile (node:internal/modules/cjs/loader:1101:14)
    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1153:10)
    at Module.load (node:internal/modules/cjs/loader:981:32)
    at Function.Module._load (node:internal/modules/cjs/loader:822:12)
    at Function.executeUserEntryPoint [as runMain*** (node:internal/modules/run_main:81:12)
    at node:internal/main/run_main_module:17:47

CONTEXT

Just tried to run the test-runner on windows-latest with the latest release of the test-runner (2.1.0).

[webbertakken]

Probably related to #198 and #199

[vayan]

Still happening for now, any older version of the windows runner that is working?

[timcassell]

Looks like it's only working on v2.0.3.

[zeroZshadow]

The issue is the symlinks introduced in #198 these are no resolved on windows machines.

I would LOVE if this could be resolved. I need to be able to run on windows machines for code coverage analysis with sonar.

[Cyantoo]

Very interested in this too.

[codegefluester]

Is there any workaround for this? We're very keen to trial GameCI but this is blocking us hard right now

[sokuhatiku]

Hello there.

I am trying to fix this problem and found that it is very complex. I have been able to fix some of the issues but it still does not work completely and I need help to resolve it.

symlink issue (resolved)

As already pointed out, symlink did not work properly on Windows, so I wrote JavaScript in main.js and post.js to call main() and post() in index.js and it worked fine.

Unity execution path issue (resolved)

Next, I noticed that artifacts were not being uploaded. I looked at the logs and found that the cause was an incorrect Unity.exe path passed to Start-Process in run_tests.ps1. This was fixed by pasting the correct path from activate.ps1.

Graphics initialization error

Additionally, an error Failed to initialize unity graphics. was logged. I did not know how to solve the underlying problem, as I have little knowledge of virtual environments. Adding the -nographics option stopped the error, but this has the side effect of disabling some graphics APIs, so it may need a better solution.

GfxDevice: creating device client; threaded=0; jobified=0
d3d11: failed to create device and context (887a0004).
GfxDevice: creating device client; threaded=0; jobified=0
d3d11: failed to create device and context (887a0004).
GfxDevice: creating device client; threaded=0; jobified=0
d3d11: failed to create device and context (887a0004).
Failed to initialize graphics.

Make sure you have DirectX 11 installed, have up to date

drivers for your graphics card and have not disabled

3D acceleration in display settings.
Failed to initialize unity graphics.
Editor exited with error(s): Failed to initialize graphics.
Make sure you have DirectX 11 installed, have up to date
drivers for your graphics card and have not disabled
3D acceleration in display settings.
Fatal Error! Failed to initialize unity graphics.
Editor exited with error(s): Failed to initialize graphics.
Make sure you have DirectX 11 installed, have up to date
drivers for your graphics card and have not disabled
3D acceleration in display settings.
Crash!!!

Standalone testing issues

Finally, at the moment I am facing a problem with the standalone test not completing until the workflow times out. I checked the editor logs and found that the error Test execution timed out. No activity received from the player in 600 seconds. was logged. My guess is that the built client is also not starting properly due to the graphics not being available in the container, but I couldn't find any solution.

Test execution timed out. No activity received from the player in 600 seconds.
UnityEngine.StackTraceUtility:ExtractStackTrace ()
UnityEngine.DebugLogHandler:LogFormat (UnityEngine.LogType,UnityEngine.Object,string,object[])
UnityEngine.Logger:Log (UnityEngine.LogType,object)
UnityEngine.Debug:LogError (object)
UnityEditor.TestTools.TestRunner.Api.CallbacksDelegator:RunFailed (string) (at ./Library/PackageCache/[email protected]/UnityEditor.TestRunner/Api/CallbacksDelegator.cs:64)
UnityEditor.TestRunner.TestLaunchers.RemoteTestRunController:TimeoutCallback () (at ./Library/PackageCache/[email protected]/UnityEditor.TestRunner/TestLaunchers/RemotePlayerTestController.cs:91)
UnityEditor.TestTools.TestRunner.DelayedCallback:Update () (at ./Library/PackageCache/[email protected]/UnityEditor.TestRunner/TestLaunchers/DelayedCallback.cs:32)
UnityEditor.EditorApplication:Internal_CallUpdateFunctions ()

(Filename: ./Library/PackageCache/[email protected]/UnityEditor.TestRunner/Api/CallbacksDelegator.cs Line: 64)

Currentry results of my work so far are in my forked repository.

[timcassell]

@sokuhatiku That standalone issue is strange. It shouldn't be executing the tests from the editor or waiting at all. The player should be built separately with the WaitForPlayerConnection disabled. Maybe the test-standalone-scripts scripts are not getting copied in?

unity-test-runner/dist/test-standalone-scripts/Assets/Editor/UnityTestRunnerAction/PlayerBuildModifier.cs

Lines 19 to 20 in 7c242f7

	// Do not launch the player after the build completes. Disable the PlayerConnection.
	playerOptions.options &= ~(BuildOptions.AutoRunPlayer | BuildOptions.ConnectToHost | BuildOptions.WaitForPlayerConnection);

[sokuhatiku]

@timcassell Thanks for letting me know.

I checked the code and found that $Env:UNITY_PROJECT_PATH was used in the code that copies the assets for the Standalone build to the project in run_test.ps1. This variable is not an environment variable, so $UNITY_PROJECT_PATH is correct. After fixing it, the test seems to pass successfully. ( https://github.com/sokuhatiku/unity-test-runner/actions/runs/7463004561/job/20306702517 )

[timcassell]

I think this issue can be closed. It seems to be working now.