Feature: Tooling to run tests for Unity packages

[antja0]

Hi!

First of all, thanks to you and all contributors for this repository ❤️

Background

I'm currently open sourcing some Unity packages (tools etc.) from our company to be used with the new Unity Package Manager.
I'm using this repository's action to run tests for those packages.

• Package I'm currently developing: https://github.com/3d-group/unity-simple-notifications (open source)
• Unity docs on custom packages: https://docs.unity3d.com/Manual/CustomPackages.html

Problem

It seems like Unity requires an Unity project to run tests.

To circumvent this problem I created an empty Unity project at my repository and ignored it with .npmignore file and just ran tests there using the projectPath variable.

But I cannot help thinking that it is a bit silly to include empty Unity project just for running tests 😄

Suggested solution

I think this could be automated and included in this action like so:

1. Check if there is package.json in project root in order to detect if project is an upm (unity package manager) package.
   or perhaps this could be told via env variable?
2. Create new Unity project (either copy it from somewhere or download from Unity website?)
3. Edit manifest.json at UnityProject/Packages/

Add following lines to dependencies and testables sections:

{
  "dependencies": {
    "com.company.package": "https://github.com/company/repository.git"
  },
  "testables": [ "com.company.package" ]
}

Perhaps Unity project could also be cached for future action runs?

[webbertakken]

Hi @antja0 and thank you for your feature request.

We are open to improve test runner to account for cases like this.

I think your suggested solution might be a great first iteration. Unity itself is able to create a project, which might be the appropriate way to use here, because a new project may differ per unity version.

Another alternative would be to run the tests directly in C#. I'm not very experienced in Unity tests (looks like nobody is yet), so it would be helpful if you could determine whether you think this would work in general, and for your projects specifically.

Looking forward to hear your thoughts on this.

[BLaZeKiLL]

you could also look into how open upm recommends setting up upm respositories.

basically they use the default unity project as the repo and create release branches using git sub tree, example this way I can have multiple upm packages in 1 repo also

[trudeaua21]

Hey, it's been quite a while since this was active, but I'm developing an open source Unity Package as well and came across this issue.

I was able to get a flow similar to what @antja0 described working locally for my purposes. Next time I get around to working on it (which should be within the next week), I'll try seeing if the same method works on the remote GitHub workflow server (using this and unity-activate). If I end up getting it working, mind if I take a stab at contributing an implementation of this in the action?

Here's my preliminary thoughts on how we could go about implementing this:

• take an input named something like testPackage to determine whether a Unity package is being tested, and maybe an input of the name/absolute file location of the package
• like @webbertakken suggested, use the Unity command Unity.exe -batchMode -createProject ./DummyProject -quit to create a Unity project (maybe one that could be optionally cached?)
• like @antja0 suggested, use a bash/batch script to modify the manifest.json file of the dummy project to add the dependency to the package being tested
• run tests

This process worked for me (sans the bash script to modify the manifest file - I did that in VSCode manually 😅 ), so I feel confident it would work well here!

Edit: I also agree with @BLaZeKiLL - it would be good to document the option of maintaining the package within an otherwise empty Unity project. Since that prevents the package from being installed with git through the Package Manager, though, I still think it would be good to implement creation of an empty project through this action.

[GabLeRoux]

like @antja0 suggested, use a bash/batch script to modify the manifest.json file of the dummy project to add the dependency to the package being tested

Sounds like a good use case for jq cli. Here's how we can insert additional key value:

echo '{"hello": "world"}' | jq --arg foo bar '. + {foo: $foo}'
# {
#   "hello": "world",
#   "foo": "bar"
# }

And some other examples (in the comments too)

[trudeaua21]

I may have figured it out - skip to the final edit

Hey, I'm working on getting this implemented, but I'm new to Docker and running into an issue where the Docker.build test seems to be failing (here's the failing check on my fork, and here's the diff). I would super appreciate it if someone could help me out with this!

The only things I've changed to make the tests fail are adding the jq cli apt-get installation call in the Dockerfile, and adding a basic log in the main workflow to test whether jq installed properly. Oddly enough, all the Unity test suites pass, and jq seems to have installed properly per the test logs, but the Docker.build test fails with the following error:

FAIL src/model/docker.test.js (9.997 s)
  ● Docker › builds

    The process '/usr/bin/docker' failed with exit code 127

      at ExecState._setResult (node_modules/@actions/exec/src/toolrunner.ts:653:17)
      at ExecState.CheckComplete (node_modules/@actions/exec/src/toolrunner.ts:634:12)
      at ChildProcess.<anonymous> (node_modules/@actions/exec/src/toolrunner.ts:507:15)

I've tried looking into this specific error, but due to my lack of Docker experience I'm having a bit of trouble interpreting what I've found.

EDIT - to potentially save someone a Google search - it seems that 127 signifies an illegal command

EDIT 2 - I believe I figured out the reason, being that the test Docker image is using Alpine, which can't use apt-get. This would make sense with the error codes I was encountering. Would it be alright if I changed this to an Ubuntu image, or some other image compatible with apt-get? I only ask because it doesn't look like jq cli is available for apk.

[trudeaua21]

Hey, sorry to ping anyone on the thread if they don't want it. Just wanted to say in case anyone Googles and finds this thread that I'm still in the process of working on this - I had intended to finish it back in October but had some problems getting older versions of Unity to run tests from a package in my local environment. I'm picking it back up again now and hopefully I'll resolve the issue while plugging away.

Edit - Really close now! I'm planning on submitting my PR for this soon - I have it functionally complete and tested through the main workflow. I'm going to make sure that it actually works on one of my own packages before submitting the PR, which may take me a couple days since the package I'm going to test this on doesn't have tests yet 😅

[trudeaua21]

Sorry again in case this pings anyone who doesn't want it - the feature is pretty much complete on my fork. I've added tests to the workflows so I'm pretty sure it works, but I'd like to test it myself on an actual package first before I open my PR.

That said, I'm dealing with an injury atm, so I don't really have the capacity to test it on a real package for at least a couple weeks. If anyone else reading this feels like it, you can give it a shot and test it on your own package before that, and if you can just let me know here whether it works and that will be enough for me to fill out the PR. If not, then I'll just get to it in a few weeks. Thanks everyone! 🎉

[Renardjojo]

Hi @trudeaua21 ! I tested your feature on my package template, but I ran issues each time. Can you help me ?
Here you can find my test project and here is the action that I use :

name: TestPackage2
on:
  workflow_dispatch:

jobs:
  testAllPackageModesLikeInTheReadme:
    name: Test package mode 📦 in ${{ matrix.testMode }} on version ${{ matrix.unityVersion }}
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        projectPath:
          - ./
        unityVersion:
          - 2019.2.11f1
          #- 2020.3.29f1
        testMode:
          - editmode
          #- playmode
    steps:
      ###########################
      #         Checkout        #
      ###########################
      - uses: actions/checkout@v2
         
      #######################
      #         Test        #
      #######################
      # Configure test runner
      - name: Run tests
        id: packageEditMode
        uses: trudeaua21/unity-test-runner@main
        env:
          # Personal license
          UNITY_LICENSE: ${{ secrets.UNITY_LICENSE }}
        with:
          projectPath: ${{ matrix.projectPath }}
          unityVersion: ${{ matrix.unityVersion }}
          testMode: ${{ matrix.testMode }}
          artifactsPath: ${{ matrix.testMode }}-packageArtifacts
          packageMode: true
         
      #############################
      #   Upload artifacts        #
      #############################
      - name: Upload test results
        uses: actions/upload-artifact@v2
        with:
          name: Package test results (edit mode)
          path: ${{ steps.packageEditMode.outputs.artifactsPath }}
          retention-days: 14

If I use 2019.2.11f1 licence with this code, I have this issue :

###########################
#   Testing in EditMode   #
###########################

Aborted (core dumped)
cat: write error: Resource temporarily unavailable
Initiating legacy licensing module

LICENSE SYSTEM [2022228 20:33:49] Next license update check is after 2022-03-01T20:30:13

And if I try with more recent licence (generate with my hub), I had this log :

Package name found: com.open-source-unity-package.test-action
Creating an empty Unity project to add the package to.
Adding package to the temporary project's dependencies and testables...

Packages/mainfest.json was not created properly. This indicates a problem with the Action, not with your package. Aborting...

Error: The process '/usr/bin/docker' failed with exit code 1

I hope this could help to understand providing of the issue

[trudeaua21]

@Renardjojo Thanks for the info! I'll be doing some refactoring for some suggested changes I received from one of the maintainers, so I'll make sure to keep these errors in mind while I'm modifying things.

Did you run the tests on an open-source package? If not, no worries! If so, I'll probably fork it so I can dig deeper into why the issue is occurring there and not on my own test project. My bad, I missed your link the first time I read your message - I'll try to make sure I can get the tests working on your project before I ask for re-review on my PR!

[trudeaua21]

Quick update @Renardjojo - I think at least one of your issues (the one when using the 2019.2.11f1 license was caused by not having a Tests folder (which will likely need to actually have tests in it) in the package. I'll add a condition to check for that case to make that error visible to the developer!

As far as the other issue goes (the one where you used a more recent license), I can't really figure out why it failed there. My first thought was that it was something to do with the license, but I messed around with the licensing on my own project for a bit and was unable to reproduce a failure on that specific step. I was even able to run my own test workflow with the same version as the one where the workflow failed, and the tests ended up running correctly.

My guess is that it's either a license issue that I just don't know enough about to debug, or that maybe Unity just sort of had a random exception. For now, I've just added a little more logging to the step where that second failure occurred.

Thank you so much for your help in testing this out! 😄 If you end up adding some tests to your example project and the issues persist, I'll be happy to take another look!

[Renardjojo]

I test again but it still doesn't work (same error with Tests folder). I have "GUID [...] conflicts with" error and "cat: /github/workspace/editmode-packageArtifacts/editmode-results.xml: No such file or directory" errors.
I have a question about this tool. Is it possible to just install a package in VM and just return if it was correctly installed in unity (see if error or warning happened during installation ? ) like user could do when they install a package thanks to git url ? I made a package template to speed up package creation process and I simply need to check if a user that clone this template creates a package that can be installed in unity. Is it possible with this tool ?

[trudeaua21]

@Renardjojo

About Errors with the Action

I'm sorry to hear that you're still having those errors - are you able to get any package tests running on your own machine? To do so, follow these steps:

• make sure you have your package located somewhere on your machine
• Follow the steps here to add tests to the package if they aren't present already: https://docs.unity3d.com/Manual/cus-tests.html (note that the last step, Enabling Tests for a Package, will have to be done after the next couple of steps)
• open an existing or new project in the Unity editor
• use the Unity Package Manager to add your package to the project you have open
• Follow the steps from the link above in the section titled Enabling Tests for a Package to make it so that your package's tests are picked up by the project that's open in Unity
• Use the steps here to run your tests: https://docs.unity3d.com/Packages/[email protected]/manual/workflow-run-test.html
• If you've done everything right, the tests from your package should be showing up in the Unity Test Runner window and passing

If you aren't able to successfully get the tests working this way, then I can't guarantee that this action will work as intended. That said, if you can successfully get tests running using the above steps, and the action still doesn't work after that, then that's definitely a problem.

I should probably document this somewhere, to be honest - I'll likely have to contribute to the game-ci documentation repo as part of this work anyhow, so I'll make sure to include a snippet about this requirement!

About your Other Question

About your question:

Is it possible to just install a package in VM and just return if it was correctly installed in unity (see if error or warning happened during installation ? ) like user could do when they install a package thanks to git url ?

The short answer is maybe? Honestly, I'm not sure what it would take to make something like that.

This is because Unity doesn't really have a way to install a package from the package manager through command line commands. Based on this blurb from UPM, a Unity Package Manager/registry, it would seem like just manually editing a project's manifest file is enough to "install" a package to the project.

The problem is that it's pretty hard to define what "correctly installed" means in your scenario. I don't really have experience using Unity from the command line past my bit of work on this action, so I don't really know what I would look for in Unity's logging to see if it has any problems processing a package's contents.

I will say that if you use this action to run tests on a package, and everything works correctly, then it's likely that the package would work properly in an actual Unity project. However, this isn't really an intended or supported use-case of this action, I'd say - the only real point of the functionality I've built in this action is to run tests from a package.

In my opinion, there's a couple ways to achieve your goal:

1. On your own machine, you could try to add a package to a Unity project's project manifest incorrectly (you could do this by having a copy of the package on your machine, adding the package to the project using the Package Manager, then going into the project manifest file and changing the path to the package to something incorrect). Then, from the command line, you could run the project using Unity Editor commands and see if it throws any errors related to the package's path being wrong. If that happens, I think the tool you're looking for could be built by doing this same process on a GitHub Action.
2. You could simply recommend that before users try adding their package to a project using a git URL, they clone the package to their own machine and add it to a project on their own machine using the Unity Package Manger window. If that works, then it's safe to say that the package would work. This sort of pushes the problem off to the user, but I think it would be a faster process than trying to do what you're looking for in a CI/CD environment.

Apologies for the lengthy message! Please let me know if you have any more questions/clarifications on this topic.

[Renardjojo]

Hi @trudeaua21 !
Thank you very much for your complete description to test manually if all work with my package. I created a new repository package with test and new repository with unity version (2020.3.29f1) that I used in my test. Unfortunately it works manually without any error, but it doesn't work with version 2019.2.11f1 (same error that I reported to you above) and the version 2020.3.29f1 time out after 6h of intensive job 😅
Here the result of my test
Here the package that I tested (don't forgot to include secret key UNITY_LICENSE. I also include action that give you license in this project but you need to activate it manually)
Here the unity project that I used (TestProject is already in manifest)

I think like you that if unity package was installed correctly, he will not find Tests/ folder and send error. So It will do the job (but log could be inexplicit... to test 😉)

[trudeaua21]

Quick update to this and #164 - I'm still planning on finishing the work up for this, but I still need to get around to figuring out how to reproduce and resolve the reported bugs, and tweaking the docker images to include the tools this feature needs. I haven't gotten around to it much as of late because of being busy with my day job, but I should have an opportunity to dig into this again in the next week or so!

[Renardjojo]

Hey @trudeaua21 do you found time to track this issue ? Is your tool work with another package than mine ?

[trudeaua21]

@Renardjojo Thanks for asking! I was testing your issue, but moved on to other work on this package for a week or so (like tweaking some of the configuration this feature was doing over to the Docker image, etc.).

Until the PR I made for the game-ci docker image gets merged, there's not much other testing I can do for this feature. Here's a comment on the PR for this feature I made that explains where this work currently stands: #164 (comment)

I will say that when I was testing to try to resolve your issue, I came fairly close to a solution. I'll make sure your issue is resolved before this feature gets merged 😄

[trudeaua21]

Update on this - it actually ended up getting merged shortly after I wrote that comment. Next Monday I'll have the time to try to work through this issue/make the final push to finish this feature! Thanks everyone for your patience! 😄

Edit- Actually, I won't be able to guarantee that this upcoming batch of work will be the last - there's a PR for another pretty sizable feature that will likely get merged before this, so I'll have to make sure this is compatible with that update. I don't really know how long that will take, but I'll try my hardest to work through it ASAP!

[trudeaua21]

@Renardjojo I took another look at why the test runner was failing on your example project. I made a fork, fixed up some details of your test workflow/.meta files, and ran the action. It's still failing, but it's giving script compilation errors, which I think indicates something is misconfigured in the package somehow (see below).

One guess I have as to what could be causing the problem is that in that example package, there's no scripts in the Editor or Runtime folders? The assembly definition files could also be misconfigured somehow maybe.

I have my own package that seems to be leveraging this action successfully. Maybe a step towards getting your package to work successfully with this could be to copy some bits of that package into your template/example package, or maybe fork my package and try to remove files from it to make it more barebones? I'd love to give more in-depth help than that, but I'm a bit short on time recently, so I probably won't be able to do too much more past this message other than answering questions for the next couple of weeks.

[Renardjojo]

Hello @trudeaua21!
Thanks to your example, I found the solution to my errors. It works for you because you add a folder to the root of your repository. After some tests on my project, it works when I also add a folder to the root of the repository. So if I add projectPath: ./, it does not work...
The problem is that I would like to avoid it because it prevents Unity to add the package from a git URL. Is this a bug? Is it a constraint of this extension that I misunderstood?
Thank you very much for your investment. Your work will certainly help the unity community to create quality packages.

[Renardjojo]

It also prevents all the "GUID conflicts"

[trudeaua21]

Hello @trudeaua21! Thanks to your example, I found the solution to my errors. It works for you because you add a folder to the root of your repository. After some tests on my project, it works when I also add a folder to the root of the repository. So if I add projectPath: ./, it does not work... The problem is that I would like to avoid it because it prevents Unity to add the package from a git URL. Is this a bug? Is it a constraint of this extension that I misunderstood? Thank you very much for your investment. Your work will certainly help the unity community to create quality packages.

Just want to double check that I understand this correctly - the reason it works on my package is because the folder structure is like so:

ROOT
  - .github
  - com.trudeaua.easyaccessibility <-- Tell GitHub this is the folder location
  - README.md
  - ...

And the reason it does not work on your project is that the folder structure is like this:

ROOT <-- Tell GitHub this is the folder location
  - .github
  - com.example.somepackage
  - README.md
  - ...

And from your tests, the action is unable to properly get the package from the root of the repository.

Does that all sound correct? Please let me know if I'm misinterpreting.

If I am understanding this correctly, it may be possible that I could get a workaround for this in by copying the package to a different folder on the GitHub hosted runner, and then test the package from there.

Thank you for bringing this up! It absolutely should work on packages with any valid folder structure, so it not working under this circumstance is a problem.

[Renardjojo]

I am not sure we are talking about the same things, and I would like to make sure we understand each other:
Your repository is as you present it in your post above. You add it to a folder inside your repository:

The structure that allow unity to recognize package from git don't allow this folder and the runtime, sample~, editor folders need to be including directly in the root of the repository I wan't add it from git URL.

I made a test and Unity don't recognize your package, but you probably use npm to avoid including it directly from your git URL.
So, in your package you specify the root of your package like that projectPath: ./com.trudeaua.easyaccessibility but in my case, I don't have a folder that contain my package (it's my repository) and it's the problem with the action.

[trudeaua21]

@Renardjojo gotcha. I have some free time tonight, so I'll take a look at resolving this issue then. Thanks for bringing this up! 😄

[Renardjojo]

Ahah you are the best ! ;D

[trudeaua21]

@Renardjojo I was able to come up with a fix for this, which I implemented here.

Here is the workflow file that resulted in the successful test on that branch

Essentially, what I'm doing here is copying the package from the project root into a sub-directory, then running the test runner on that. This seems to get around whatever compilation issue the test runner was having.

I'm still not totally sure why the issue happens in the first place - my only guess is that something about the project root messes with Unity's compilation of the package. At a glance, it looks like when I copied the package from the repository root into a different folder, the file _activate-license didn't get copied over for one reason or another. My best guess is that that file messes with Unity, and copying the package over to a folder without that file avoids the compilation issue.

That's all just speculation though.

I'll work on getting this built into the action itself tomorrow night asap. Thanks again for finding this!

[Renardjojo]

Hi @trudeaua21 ! I test your hack in a new project, and it works perfectly !
I have two questions about your feature :
1: Is there a way to use cache action ? It yes, how can I use it ?
2: I tried to get the codecoverage output like you, but I got warnings that the covergae directory is empty? Do you know why?

[trudeaua21]

Hi @trudeaua21 ! I test your hack in a new project, and it works perfectly ! I have two questions about your feature : 1: Is there a way to use cache action ? It yes, how can I use it ? 2: I tried to get the codecoverage output like you, but I got warnings that the covergae directory is empty? Do you know why?

@Renardjojo Glad that it works!

1. Currently, I don't have caching set up in any way, and I'm not planning on adding it before this gets initially released.
2. I am unsure what's going wrong with the codecoverage - I'll have to look into it. Also, just a heads up, I've been pretty busy the past couple weeks, so I'm not sure when I'll get around to looking at this issue.

[dev-fredericfox]

@trudeaua21 Any updates on this by chance?

[trudeaua21]

@dev-fredericfox My last update last summer was here: #164 (comment)

Essentially, I was working on #164 up until Unity made some decisions as a company that really soured my opinion of them and sort of zapped my motivation. Since then I haven't looked at it.

That said, I had intended to finish up what work I did have done, but never really got around to it.

Truth be told, I feel a bit bad about it. I still don't have a ton of time to dedicate to working on this, but I think I'm going to try to put in a couple hours each weekend until I can get what's done into a releasable state.

I would totally welcome anybody else who'd like to pick up the work to take over from my fork, though it may be a bit out of date by now.

[k-karuna]

@trudeaua21 Is it possible to use your solution RN to test UPM in any way?

[calebscivista]

I am also very interested in seeing this issue resolved. My team is pretty much exclusively using Unity Packages. Would love to use GameCI, but it's much more difficult to do without this fix. It seems like @trudeaua21 was going down a really good direction. I'll check out the PR he'd posted and his fork. From my reading so far, it seems like this was very close to a conclusion, then progress halted. Will be keeping an eye on this! Thanks.

[trudeaua21]

@trudeaua21 Is it possible to use your solution RN to test UPM in any way?

@k-karuna I'm not familiar with how UPM works, so I don't really know one way or the other. Sorry that's not a super helpful answer.

Once the first iteration of package testing is merged, there will be either an issue or several issues discussing limitations of the feature. I'll try to remember to ping you to start a discussion in the relevant issue once that happens if you'd like 😄