Trying out XCode 7 Beta – Code Coverage and XCTest UI Testing

After watching the WWDC 2015 presentation of the new testing tools that will be available in XCode 7, which promises UI testing and code coverage right inside XCode, I decided to give it a try myself.  This blog post is a summary of my experience.

My Configuration

First of all, here’s what I’m using at the moment:

• XCode 7 beta 4 version
• OS X 10.10
• iOS Simulator – iPhone 6 running iOS 9
• AUT: I am using my K-Cards app, a very simple app that I’ve created to replace the need for physical color cards for facilitating peer-conferences. More info on the app here: https://itunes.apple.com/fi/app/k-cards/id990993385?mt=8 The main use case for the app: people at a conference can use their phone to show to the facilitator, using color codes, that they want to contribute to the discussion. They can start a new thread (show a green card), have a comment on something that is currently being discussed (a yellow card) etc. The app also allows them to add their name on the screen, and has a separate view with info about the color codes.

The source code for the app itself and these tests is available here: https://github.com/rucindrea/K-Cards.

Code Coverage in XCode 7

Before, I used to keep an eye on code coverage for my iOS projects using all sorts of different tools like Cover Story, so I was quite excited to try the new beta version of XCode and have the code coverage info right inside my code. At first though, I didn’t know how to configure it – my Report Navigator in XCode didn’t show the “Coverage” tab, I only had “Tests” and “Logs”.

I tried to add the Build settings needed in previous XCode version in order to have code coverage data generated: setting “Instrument Program Flow” and “Generate Test Coverage Files” to Yes. That didn’t help at all, running my unit tests didn’t generate any Coverage info in the XCode reports.

After some googling, I found out how to configure XCode Code Coverage reports:

Now when I run my unit tests, I can see the code coverage info when I go to “Report navigator” and choose the “Coverage” tab:

As said before, the unit tests I have written are all about changing the background color and checking that the correct label is displayed, so in my ViewController.m file, I can see that the changeColor method is fully covered by my checks. In XCode, I can now go and check the coverage info straight from my ViewController.m class. By default, XCode will display per-line iteration counts on the right side of the editor, as shown below:

In my case, it looks like when I run my 6 unit tests, the alert view buttons are never clicked, the changeColor method is called 11 times, and inside that, for example, the tests change the color from green to yellow 4 times. If I run only one unit test, testChangeColorToYellow, here how’s my coverage looks for the same method:

The fact that I can see all this straight from XCode is great! Now, I knew already and also saw on my code coverage report that my checks don’t deal at all with the alert view that changes the name displayed in the app, so the next part of this blog post will focus on adding some UI checks that will do exactly that.

XCode 7 UI Tests

First, I had to add another testing target to try out the new UI automation features in XCode 7.

From File->New->Target in XCode, I chose iOS UI Testing Bundle and gave it a name (K_CardsUITests). XCode will automatically create another target and a folder/group in my project, and create the K_CardsUITests.m file under it:

The first UI checks that I want to add are related to adding or changing the name that is displayed by the app.

Issue no. 3 (https://github.com/rucindrea/K-Cards/issues/3) reported for the app was regarding the fact that the Cancel button in the Change Name alert view would actually confirm the change. For example, if I had some name saved, just tapping the Add/Change name button and choosing Cancel would clear the name. I have fixed this bug, but I now want to add a UI check for it and run it on all different simulators.

I can now rename the example test to “testCancelChangeName” and try the new “recording” functionality from XCode 7. When my cursor is inside a test method, the red recording button at the bottom of the editor view is enabled and I can start my recording. I’ve included here a video with my first recorded test:

I now need to add some asserts to check that my text field has the values that I expect it to have. To do that, I often set a breakpoint in the tests and use the debugger to check different properties and values for the elements that I am interested in, so I know how to use them in my checks.

I also need to split the recorded bit of code into two different checks, each with a separate assert:

• testCancelChangeName: the first part of the recording, where I click Cancel and check that the name is not cleared
• testClearName: the second part of the recording, where I click OK and expect to see that the name is cleared

Here’s another short video that I use the debugger to pause execution and explore the elements inside my app, and I then split the tests and use two different asserts for each of them:

So now we have two tests that will clear the name and will check that if I cancel the alert view, the name will not be cleared. I also need to add a test that will actually set a new name. To do that, I need to type a name in the alert dialog. I can start with these basic first few steps:

and set a breakpoint on the last line. When I run the test, after I also execute the last line, I can use the console to explore my elements and find out which one is the textfield that I want to type in:
po collectionViewsQuery.textfields
which will give me:

It looks like we only have one text field, which means the following query will resolve to a single element:
    and I can use:    

Now that I have a way to refer to the textfield, I can simply type the text like this:

and assert that after pressing OK, the name is correctly saved.
Here’s the whole check:

I now have 3 checks for the name label and I can run them all by going to the Test Navigator and selecting the K_CardsUITests test suite.

After I run all 3 checks, I can go to the Report Navigator from XCode and check the results. For each check, I can see all the steps taken and can even see a screenshot with the application state (by clicking on the “eye” icon next to each step) – here’s how this looks for me:

Conclusions and Observations

Overall, I was very happy to see that Apple decided to focus more on the code coverage and test tools in this next XCode release. I find the UI Test tools extremely useful – the checks are fast and easy to implement and fast to run. The same goes for the code coverage tools: I was happy with the easy setup and with having all the information right there in XCode.

With this being a beta release still, here are the main problems I’ve had while trying this out:

• Lack of documentation – this is normal at this beta stage, but it was quite difficult at times to figure out where to look for information. I found this link: https://github.com/joemasilotti/XCTest-Documentation which is an unofficial and auto-generated documentation for the new UI Testing framework – useful to have!
• Code coverage for UI Tests missing? – while code coverage was generated for all the unit tests I created, I couldn’t get the same to happen when running the new UI tests in XCode. My coverage report only contains info about the code exercised by the unit tests, and this is true even if I try to collect code coverage data “the old way”. I’m hoping this will be solved in the following releases of XCode 7.
• Lots and lots of crashes – I tried out first XCode 7 Beta 3, then Beta 4. Both versions crash very often for me, especially when using the recorder or running UI tests. Again, I’m sure this will no longer be the case in following releases.

If I get a chance to try this again when the official release is out, I’ll try to update this post with more results. Stay tuned!