AltUnityTester – Testing Unity games and apps using Appium

[Update 2022] AltUnity Tester becomes AltTester Unity SDK starting with v. 1.8.0 and similarly, AltUnity Inspector becomes AltTester Desktop starting with v. 1.5.0 and they are both available here

This comes with a series of major changes:

1. AltTester Unity SDK is now available only through our website and is no longer available on the Unity Asset Store

2. We moved the AltTester Unity SDK project from GitLab to GitHub.

Read more about all the major changes in this article and join our Discord Server, if you have any questions or issues with AltTester!


Some background info: Appium and Mobile Games Automation

Before going into details about AltUnityTester, I want to tell you a bit about my own context and the struggles I’ve had in mobile games automation.

A few years ago, I started working on a few projects with Bitbar developing automated scripts for mobile game companies. Most of these companies were looking to get fast feedback on their games by running basic checks on many different devices using Testdroid Cloud. Most of them were also interested in running checks on both iOS and Android and they mostly wanted to see that the game starts, loads correctly and in a reasonable amount of time on as many devices as possible. Some companies were also looking into adding tests to verify the first steps of the game, finishing the first few levels, navigating through the main menus or trying out the most important features.

Appium seemed like the best choice for this, since it offers great interaction with the device on both iOS and Android, supports many programming languages (so each company could choose their favorite) and is a favorite of many testers that are familiar with UI Automation in general, as it’s similar to Selenium WebDriver.

But, for games developed using game engines like Unity or Unreal, that don’t use the native UI frameworks, Appium couldn’t detect any of the objects on the game. Clicking at specific coordinates was not an option considering the device fragmentation problem that most companies were trying to solve and the diversity of resolutions available out there (Here is a presentation I did at ETC 2016 about our approach). We decided to use image recognition to detect “objects” (pictures) on the screen.  We had a lot of success with that, but, as expected, image recognition is slow and this approach was not suitable for games like endless runners or games where reaction time is essential.

Unity for App Development

A couple of yeas after that, I started working with a different company as a tester/app developer in their New Concepts team. We were trying out a lot of new ideas, developing small apps and prototyping a lot of solutions in a very fast paced environment. Along with new ideas, we also needed to try new graphical designs and we needed to create apps that were fast to develop and could be deployed to a multitude of platforms if needed. We focused mostly on iOS and Android, but we needed to know that the apps could run on Windows, Mac, etc. After trying out a few options, including some web frameworks, we decided to develop all our apps in Unity. It has support for any platform that we could ever need and it’s excellent for fast prototyping, with incredible graphics on every platform and with an abundance of ready-made artworks, scripts, plugins and solution available on their Asset Store.

While Unity allowed us to be very productive on the development side, I was again struggling on the testing side. I had to deal with a lot of apps that still required a lot of testing. Most of these apps also had to work together as part of a system, and we often found ourselves trying to figure out ways to deploy and run some automated checks on our environment automatically. While the apps were relatively simple and would have been the perfect candidate for simple automated checks with Appium (which works on Android, iOS and even Windows nowadays), Unity didn’t have a solution for running these types of system level / UI driven tests on real devices.

Looking for a solution

While searching for a solution for these two problems, Rasmus Selsmark, a fellow tester who works at Unity in Helsinki, told me about this hack-day project that provides one possible option for running Appium tests against Unity games and getting feedback from Unity about the objects on the screen via the Debug.Log() functionality – so essentially by injecting information about specific objects in the device logs (logcat on Android or syslog on iOS), from where they can then be parsed and used inside the Appium tests.

I tried it out and then decided to try to implement something similar using a TCP socket running from the Unity game that can listen to commands sent from the tests. And so AltUnityTester came to be.

AltUnityTester

AltUnityTester – now also available on the Unity Asset Store – will let you find objects in your game and get their coordinates on the screen so you can interact with them from Appium.

Currently, AltUnityTester has bindings implemented for Python, C# and Java allowing you to:

  • create an AltUnityDriver
  • connect to the AltUnityTester component running on the device
  • ask for information on objects from the Unity Game
  • find elements and get their coordinate
  • simulate any kind of device input

Once you get the coordinates for an element (like “Bed” in the example below) you can use Appium to tap() on screen at those coordinates.

AltUnityTester used with Appium

Source code and documentation

We decided to have AltUnityTester open sourced and we have posted it on our GitLab account here:

Getting Started Video

To help you setup your game and get started with AltUnityTester, we’ve created a video tutorial:



Running in the cloud

There’s an older tutorial video we used to show how Unity’s Adventure example could be tested using AltUnityTester. I wanted to give it a try in Testdroid Cloud as well, since you can run Appium tests in their cloud. I used their Python server-side Appium example scripts as a starting point and made the following modifications:

  • I added the python binding installation in the requirements.txt script:altunityrunner
  • I added the adb forward command in “run-scripts.sh”, just before starting Appium:
     
    adb forward tcp:13001 tcp:13000
     

I then used the following Python script to run the tests using AltUnityTester in the cloud (click on the photo for the link to the entire code snippet):

Sample Appium scripts used to run AltUnityTester tests in Testdroid cloud

Here is the result of these tests on a Nexus 5 from Testdroid Cloud:

Screenshot of test results using AltUnityTester in Testdroid cloud

If you’d like to know more about AltUnityTester or about any of the aspects covered in this blog post, leave a comment, we’d love to hear from you!

Subscribe to our newsletter

And get our best articles in your inbox

 Back to top

56 responses to “AltUnityTester – Testing Unity games and apps using Appium

  1. Interesting blog. I have looked at using BDD for automating games in the past. Even thought of coming up with a new development approach GDD Gaming driven development somehow using gherkin syntax.

    1. Thanks! GDD – interesting idea. Did you already have some thoughts on how to make it specific to games?

  2. This is exactly what i needed. I have been working in automating games test developed in Unity through image recognition. I started to do some stuff with the logs to find elements positions, and thinking of other ways to get unity elements. This will help me so much!

    1. That’s great to hear! If you need any help or run into any problems, please let me know, we’ll try to help!

    1. Yes, both are on the roadmap.

      Java will be straightforward to implement with Appium, so we are thinking of doing that next.

      The C# + Appium combination is not the easiest, so for C# we are actually thinking of just implementing something using the Unity Test framework and without Appium. That will mean we could run the tests on any platform, from the Unity Editor.

      We’ll keep you posted!

          1. It will be similar to Unity Test Framework, using NUnit and with Unity Editor support. Hopefully we’ll have it out later today, I’ll send another message if we do.

  3. Hi,
    This is looking really great!

    I’ve been trying out the sample which works great on Android and now have a few questions.
    Should it work or are there plans to get this to work with Windows (ideally both classic and UWP)? I’ve tried using the sample with windows classic and the only issue I find is it dislikes the “tap” method.

    The c# solution you mention will use the Unity Test Framework, I assume that limits it to editor time testing only?

    Thanks!

    1. Thanks for trying it out!

      I haven’t tested this on Windows myself as I haven’t tested the Windows Appium driver at all (I’m primarily a Mac user) but I do think it should work just fine with Windows as well. The tap method now uses Appium to tap at coordinates, so I’m not sure how the Windows Appium driver performs a click, but I think it should be doable. I’ve now added this as an issue in the Gitlab repository, I’ll try to work on it as soon as possible: https://gitlab.com/altom/altunitytester/issues/6

      About the C# solution: yes, the idea would be to get it to work with Unity Test Framework, using the editor time testing. However, this would mean that the Editor tests would be running against the game/app that’s running on a real device somewhere on the network (on a mobile device, or maybe even on the same host as the Unity Editor, but as a standalone app) so we would be able to run end-to-end tests against it. Hopefully! 🙂

  4. Would like to know how to access Game Objects when we have canvas based screens/games.

    The only option using Appium is doing screen comparison and that to has drawbacks with different resolutions.

    We want to access game objects via Appium

    1. Hi! AltUnityTester helps with exactly that! You can access game objects via Appium because we expose those using a socket connection. You need to instrument the game to allow the socket to be opened, but after that you can search for objects via AltUnityTester, get their coordinates and click/interact with them using Appium.

  5. Hi, I’m new to automated testing (and by no means an engineer), but I have got this solution working now on Windows 10 🙂 This looks very promising!

    I have two quick questions.

    Does every game object need to have a unique name or can I select objects through their parents or hierarchy? Buttons usually do have unique names, but the text objects beneath them do not. And Ok or Cancel buttons have generic names, but are unique within a screen or panel.

    Secondly, can I combine altunitydriver with other drivers? My game uses native keyboard to enter player name for instance. So I would like to use the send_keys command for this.

    Thanks!

    1. Hi Anne,

      It should be possible to use
      altdriver.find_element(“Parent/Element”)

      to access elements via their parent’s name. This seems to work fine for me, but we have some reports that this doesn’t work correctly on some environments so I am still investigating why. Here is a link to the issue that was logged in Gitlab: https://gitlab.com/altom/altunitytester/issues/12

      Yes, you should be able to combine AltUnityTester with other drivers. The example Python script from here: https://gitlab.com/altom/altunitytester/blob/master/Assets/AltUnityTester/ExamplesAndTests%20(can%20be%20deleted)/sampletest.py shows that you will basically have 2 different drivers: “altdriver” is the AltUnityDriver for finding elements/game objects from Unity, and “driver” is you usual Appium driver that you can use to interact with the device as you would normally.

      I hope this helps! Let us know if you have more questions.

    1. At the moment, we only have bindings written in Python. We are thinking of also adding Java, we just haven’t had the time to do that yet.. The Python bindings do work on Windows, so maybe that would be an option?

  6. Hi, I’m super new to Appium and just found AltUnity Tester through this blog. I’ve been following the exact steps from the video and have come across this error when running the first simple tests:

    ModuleNotFoundError: No module named ‘altunityrunner’

    https://i.imgur.com/VS0K5So.png

    Maybe I did not set up Appium correctly.

    Any suggestions? Thanks so much!

    1. It sounds like the command to install the python module didn’t work properly. Can you try re-running it?

      This is the command:

      pip install altunityrunner

      and if you are not using python virtual environments and you want to install it globally, you have to run it with sudo, like this:

      sudo pip install altunityrunner

      I hope this helps!

  7. Does it only work on mobile game? I’m trying to use AltUnity for a WebGl game that I want to test on a browser.

    1. For now, AltUnityTester only work on mobile, but we are hoping to expand support to other platforms too in the future!

  8. Thank you for this blog article, that was really interesting!

    We also want to create automated tests for our application, and are currently looking for the best solution to do that. Until now, we only tested Airtest Project that seems pretty awesome! Did you know this solution?

    1. Thanks! We are just about to release a new version that has support for a lot more features, so stay tuned! 🙂 I didn’t know of Airtest, I will check it out soon.

  9. First, thank you for this great solution.

    During my research on automated testing for Unity3D, I found also this solution:

    http://airtest.netease.com/
    https://github.com/AirtestProject/Airtest
    https://github.com/AirtestProject/Poco

    It is very similar to AltomUnityTester, but does not work with Appium and therefore I’m no sure if it would work also on test farms.

    Another great solution:
    https://github.com/gwaredd/unium
    It’s limited to Unity3D input, but has great support of query language (XPath like).

    1. Thanks! Airtest looks interesting, but it uses image recognition, we’ve tried that as well but with problems when you have your tests running on different devices + different resolutions. Unium looks really interesting! Thanks for sharing these, we’ll have a closer look at both!

      1. Yes, it looks like Airtest uses image recognition, but poco works exactly as AltUnityTester and I think, you can mix both of them together (I’ve tried only poco).

    1. I didn’t know about Katalon before, I just googled it now. It seems interesting, we will have a look and see if we can use it! At the moment though, AltUnityTester doesn’t work with it.

      1. Hi Ru,
        I’m also interested in the possibility of integrating Katalon Studio and AltUnityTester! I recently learned about the application and would love to use it for UnityApps.

        Is integrating Katalon and AltUnityTester still a possibility or is there something in the works and/or a release date?? 😉

        Thank you in advance!

        1. Regarding Katalon, we don’t yet have any plans to support it, we’ll try to keep you updated here if that changes and we have a more clear plan in that direction.

  10. So this can no longer be used with Python? Looking in the source history, looks like this was all removed a month ago.

  11. Hi Ru, It is great solution. I wanted to ask if there is a specific version of Unity it works with? For me every time I try it Unity compiler complains about “The type or namespace name ‘current’ does not exist in the namespace ‘EventSystem'”
    Has it ever happened to you?

    1. Hi, AltUnityTester does not require a specific Unity Version. We didn’t have this problem before so can you please write an issue about this error at https://gitlab.com/altom/altunitytester/issues for further disscusion. Please mention what version of Unity and what version of AltUnityTester(0.2.3, 1.2.0 or 1.2.2) are you using. Also please include an screenshot with console where the error is shown.

  12. Hi Ru, thanks for AltUnity Tester.
    I was tried using in my game project, but I have a doubt ’cause in your Python example do you used desired.caps to set up and device configuration. But when I make the same test in C# I don’t know how to configure the capabilities of my project. I tried import Selenium Web Driver but don’t work.

    1. If you use AltUnityTester with our C# binding, you write your tests from UnityEditor/Visual Studio (or whichever IDE you use in Unity) and you don’t use Appium at all. UnityEditor will build and run the app on your device, you select the platform from AltUnityEditor and run the tests from there as well. So when you use C#, you don’t have any web driver/appium capabilities. Check out this link for more info: https://gitlab.com/altom/altunitytester#altunitytester-user-interface

    1. We haven’t actually tried it, but we think you should be able to test VR apps on mobile just like you would any other game. As long as you can interact with the app in mobile mode, then it should work. If you give it a try, please let us know if it worked!

  13. Hi, is possible simulate python script for iOS and Android on running game in Unity editor?

    That allows polish script in quick time and then make real test on devices.

    Thanks

    1. Hi Milan,
      Yes, with the latest version of AltUnityTester, you can run the game in Editor and try out the Python scripts against it.

        1. If you import the latest package (1.3.0) in Unity Editor and open the AltUnityTester window, you have a “Play in Editor” button. When the game in playing in editor, you can run your Python scripts against it on port 13000 locally.

          You can see both the button and a video of me running C# tests against the game playing in Editor in this blog post: https://alttester.com//altunitytester-version-1-3-0/

          1. Hi Ru,
            I tested it in editor and on device. Looks really nice. Good work.

            One note.
            If I run it in editor and my python script crash on some error, I cant connect again to the same already running server and must stop and run game in Unity editor again.

            This waste debugging time.

  14. Hello Ru,
    Nice post Ru.
    How would you approach to test the Unity app which has some native dialogs/buttons from Android or iOS, for example the buy button in In-app-purchases?

Leave a Reply

Your email address will not be published. Required fields are marked *