Add standalone testmode support #219

timcassell · 2023-04-14T09:13:53Z

Resolves #208.

Changes

testMode: standalone now builds the player with tests (StandaloneLinux64 or StandaloneWindows64) and runs it directly. This is useful for running tests in the IL2CPP runtime (of course it's not restricted to IL2CPP).

Checklist

Read the contribution guide and accept the code of conduct
Readme (updated or not needed)
Tests (added, updated or not needed)

Additional Info

Requires Unity 2019.2 or newer and Test Framework 1.0.15 or newer.

testmode: all still only runs editmode and playmode (so as to not break existing users).

I got a working test run here. I was unable to test it on a Windows runner due to #200.

github-actions · 2023-04-14T09:21:12Z

Add test for IL2CPP.

timcassell · 2023-04-16T06:36:00Z

I'm not sure why the first test is failing, but I fixed the other tests I added.

timcassell · 2023-04-19T20:40:50Z

@webbertakken @davidmfinol Can you please take a look?

webbertakken

Looks good to me. Though I have little context about the specifics.

Can you confirm both?

This works
This does not break previous flows

dist/steps/run_tests.sh

dist/test-standalone-scripts/Assets/Player/UnityTestRunnerAction/TestRunCallback.cs

webbertakken · 2023-04-19T21:14:39Z

I'm also seeing some errors in the workflows:

-----CompilerOutput:-stdout--exitcode: 1--compilationhadfailure: True--outfile: Temp/UnityEngine.TestRunner.dll
Microsoft (R) Visual C# Compiler version 2.9.1.65535 (9d34608e)
Copyright (C) Microsoft Corporation. All rights reserved.

Library/PackageCache/[email protected]/UnityEngine.TestRunner/NUnitExtensions/Runner/UnityLogCheckDelegatingCommand.cs(115,57): error CS0246: The type or namespace name 'TestMustExpectAllLogsAttribute' could not be found (are you missing a using directive or an assembly reference?)
Library/PackageCache/[email protected]/UnityEngine.TestRunner/NUnitExtensions/Runner/UnityLogCheckDelegatingCommand.cs(124,63): error CS0246: The type or namespace name 'TestMustExpectAllLogsAttribute' could not be found (are you missing a using directive or an assembly reference?)
Library/PackageCache/[email protected]/UnityEngine.TestRunner/NUnitExtensions/Runner/UnityLogCheckDelegatingCommand.cs(136,65): error CS0246: The type or namespace name 'TestMustExpectAllLogsAttribute' could not be found (are you missing a using directive or an assembly reference?)
-----CompilerOutput:-stderr----------
-----EndCompilerOutput---------------
- Finished compile Library/ScriptAssemblies/UnityEngine.TestRunner.dll
Library/PackageCache/[email protected]/UnityEngine.TestRunner/NUnitExtensions/Runner/UnityLogCheckDelegatingCommand.cs(115,57): error CS0246: The type or namespace name 'TestMustExpectAllLogsAttribute' could not be found (are you missing a using directive or an assembly reference?)
Library/PackageCache/[email protected]/UnityEngine.TestRunner/NUnitExtensions/Runner/UnityLogCheckDelegatingCommand.cs(124,63): error CS0246: The type or namespace name 'TestMustExpectAllLogsAttribute' could not be found (are you missing a using directive or an assembly reference?)
Library/PackageCache/[email protected]/UnityEngine.TestRunner/NUnitExtensions/Runner/UnityLogCheckDelegatingCommand.cs(136,65): error CS0246: The type or namespace name 'TestMustExpectAllLogsAttribute' could not be found (are you missing a using directive or an assembly reference?)
[ScriptCompilation] Recompiling all scripts because: Reload of assemblies failed
- Starting compile Library/ScriptAssemblies/UnityEngine.TestRunner.dll

https://github.com/game-ci/unity-test-runner/actions/runs/4711881330/jobs/8433363879?pr=219#step:4:682

Not sure if they were introduced in this PR or not, but we can't merge if we're blind.

…on/TestRunCallback.cs Co-authored-by: Webber Takken <[email protected]>

timcassell · 2023-04-19T21:58:25Z

I'm also seeing some errors in the workflows:

That's weird. The tests passed when I ran them in my fork. https://github.com/timcassell/unity-test-runner/actions/runs/4748246522
Only the first test was failing there (I have no knowledge of ESLINT).

timcassell · 2023-04-19T22:22:39Z

Can you confirm both?

This works

This does not break previous flows

Yes, details under additional info in the PR description.

timcassell · 2023-04-19T22:41:22Z

@webbertakken I think those failures are Unity's fault (the only change in the project in those workflows is the TestFramework version update). I changed the workflows to use 2019.4.40f1 which seems more stable.

About the other failing test, any idea what this is about?

output

Run yarn lint
yarn run v1.22.19
$ prettier --check "src/**/*.{js,ts}" && eslint src --ext .js,.ts --max-warnings=0
Checking formatting...
All matched files use Prettier code style!

Oops! Something went wrong! :(

ESLint: 8.38.0

/home/runner/work/unity-test-runner/unity-test-runner/node_modules/eslint-plugin-github/lib/rules/role-supports-aria-props.js:12
  key.attributes = key.attributes?.filter(attribute => !('constraints' in attribute))
                                  ^

SyntaxError: Unexpected token '.'
    at wrapSafe (internal/modules/cjs/loader.js:915:16)
    at Module._compile (internal/modules/cjs/loader.js:9[63](https://github.com/timcassell/unity-test-runner/actions/runs/4748493372/jobs/8434746160#step:5:64):27)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1027:10)
    at Module.load (internal/modules/cjs/loader.js:863:32)
    at Function.Module._load (internal/modules/cjs/loader.js:708:14)
    at Module.require (internal/modules/cjs/loader.js:887:19)
    at require (internal/modules/cjs/helpers.js:74:18)
    at Object.<anonymous> (/home/runner/work/unity-test-runner/unity-test-runner/node_modules/eslint-plugin-github/lib/index.js:21:33)
    at Module._compile (internal/modules/cjs/loader.js:999:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1027:10)
error Command failed with exit code 2.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
Error: Process completed with exit code 2.

Print player log.

timcassell · 2023-04-20T07:27:47Z

Looking at vercel/next.js#38530 with the same error, it appears that syntax requires node 14 or higher. The test is using node-version: 12.x. Can that be updated, or is 12.x required for some reason?

webbertakken · 2023-04-20T20:59:50Z

Tests seem to be passing. Still looks good to me.

Perhaps Andrew or David could have another look and merge it if they're also on board.

davidmfinol

LGTM

joaoborks · 2023-06-28T01:20:18Z

I'm using the unity-test-runner version 2.1.1 that should include this feature, but I'm getting the error:

Run game-ci/unity-test-runner@v[2](https://github.com/mygamedevtools/scene-loader/actions/runs/5396165109/jobs/9799475943#step:4:2).1.1
  with:
    projectPath: ./
    testMode: standalone
    artifactsPath: standalone-artifacts
    githubToken: ***
    checkName: standalone Test Results
    coverageOptions: generateAdditionalMetrics;assemblyFilters:+MyGameDevTools.SceneLoading
    unityVersion: auto
    useHostNetwork: false
  env:
    UNITY_LICENSE: ***
Error: Invalid testMode standalone

timcassell · 2023-06-28T01:46:47Z

@joaoborks A new version with this feature has not been released yet. You can try it with

game-ci/unity-test-runner@main

or for the specific commit

game-ci/unity-test-runner@31086d985910613d75c32ba965f657df9c298820

Also note that your coverageOptions won't do anything with testMode: standalone.

joaoborks · 2023-06-28T14:10:41Z

Thanks @timcassell, it worked with the commit hash. Is there any ETA on the new release version?

timcassell added 3 commits April 14, 2023 02:03

Add standalone support.

40d638e

Add standalone tests.

3ea1841

UnityStandaloneScripts volume on windows

8318c45

timcassell added 2 commits April 14, 2023 14:19

Update test framework in test project.

8f13885

Revert IL2CPP setting in test project.

2d0b6eb

Add test for IL2CPP.

timcassell force-pushed the standalone branch from 8fab551 to 2d0b6eb Compare April 16, 2023 06:24

webbertakken approved these changes Apr 19, 2023

View reviewed changes

dist/steps/run_tests.sh Outdated Show resolved Hide resolved

dist/test-standalone-scripts/Assets/Player/UnityTestRunnerAction/TestRunCallback.cs Outdated Show resolved Hide resolved

Update dist/test-standalone-scripts/Assets/Player/UnityTestRunnerActi…

b96ed73

…on/TestRunCallback.cs Co-authored-by: Webber Takken <[email protected]>

Use 2019.2.21f1

5ff0649

timcassell force-pushed the standalone branch from 3722353 to 5ff0649 Compare April 19, 2023 22:26

Unity is being dumb, use 2019.4.40f1 for all workflows.

9eb8a55

timcassell added 2 commits April 20, 2023 00:33

Disable PlayerConnection on build.

178a747

Print player log.

Add comment about code coverage support in standalone.

8a62ceb

Update node-version in test

627fe0a

timcassell requested a review from webbertakken April 20, 2023 20:45

webbertakken requested review from davidmfinol and AndrewKahr April 20, 2023 20:59

davidmfinol approved these changes Apr 21, 2023

View reviewed changes

davidmfinol merged commit 31086d9 into game-ci:main Apr 21, 2023

timcassell mentioned this pull request Apr 21, 2023

Add Standalone option. game-ci/documentation#391

Merged

3 tasks

johnfoconnor mentioned this pull request Jul 19, 2023

update from upstream machinezone/unity-test-runner#2

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add standalone testmode support #219

Add standalone testmode support #219

timcassell commented Apr 14, 2023 •

edited

Loading

github-actions bot commented Apr 14, 2023

timcassell commented Apr 16, 2023

timcassell commented Apr 19, 2023

webbertakken left a comment

webbertakken commented Apr 19, 2023

timcassell commented Apr 19, 2023 •

edited

Loading

timcassell commented Apr 19, 2023

timcassell commented Apr 19, 2023

timcassell commented Apr 20, 2023

webbertakken commented Apr 20, 2023

davidmfinol left a comment

joaoborks commented Jun 28, 2023

timcassell commented Jun 28, 2023 •

edited

Loading

joaoborks commented Jun 28, 2023

Add standalone testmode support #219

Add standalone testmode support #219

Conversation

timcassell commented Apr 14, 2023 • edited Loading

Changes

Checklist

Additional Info

github-actions bot commented Apr 14, 2023

timcassell commented Apr 16, 2023

timcassell commented Apr 19, 2023

webbertakken left a comment

Choose a reason for hiding this comment

webbertakken commented Apr 19, 2023

timcassell commented Apr 19, 2023 • edited Loading

timcassell commented Apr 19, 2023

timcassell commented Apr 19, 2023

timcassell commented Apr 20, 2023

webbertakken commented Apr 20, 2023

davidmfinol left a comment

Choose a reason for hiding this comment

joaoborks commented Jun 28, 2023

timcassell commented Jun 28, 2023 • edited Loading

joaoborks commented Jun 28, 2023

timcassell commented Apr 14, 2023 •

edited

Loading

timcassell commented Apr 19, 2023 •

edited

Loading

timcassell commented Jun 28, 2023 •

edited

Loading