Measure your code coverage to find unit testing holes
One of the first things I do when working on any Xcode project is set up code coverage. If the coverage shows a hole, I know that area is lacking unit tests.
(Be careful, the opposite isn’t true: Just because some code has been touched by unit test execution doesn’t mean it’s actually covered. If altering the behavior of the code causes a test to fail, then you know it’s covered.)
Many people use CoverStory, a a code coverage browser app written by my friend Dave MacLachlan. Others use gcovr to integrate code coverage into their Jenkins continuous integration. Me, I use lcov because it lets me exclude third-party libraries from the measurements before generating an HTML report.
When a project is built from the command-line with xcodebuild, it places build artifacts into a “build” folder, kind of like the old days. But I want to measure coverage of unit tests as I run them from Xcode itself. This complicates things because build artifacts go into some obscure DerivedData subfolder. I solve this by having the build process export some of the project’s environment variables to a file. I’ll show you where to get the shell scripts I use to do all this.
Download the code coverage tools
I have a GitHub repository called XcodeCoverage. You’ll probably want to make some per-project changes. So if you use git for your projects, think about how you want to manage your fork – branch – submodule workflow.
Place the XcodeCoverage folder in the same folder as your Xcode project.
Then download lcov-1.10. Place the lcov-1.10 folder inside the XcodeCoverage folder.
Also (if you haven’t done so already) go to Xcode Preferences, into Downloads, and install Command Line Tools to get Xcode’s coverage instrumentation.
Set up your project
First, I assume you’re using Xcode 4.5. If so, you’re in luck: It’s easy. (Code coverage used to be a little more complicated in earlier versions of Xcode, as Claus Broch describes in Code Coverage with Xcode 4.2 and Code Coverage and fopen$UNIX2003 Problems.)
The first steps are the same, regardless of what coverage tool you use. Select your project, and go into its Build Settings. (I like to make these changes at the project level so that they’ll apply to all targets.) Find the “Generate Test Coverage Files” setting. Spin down the disclosure triangle to reveal your configurations. Enable this settings for your Debug configuration:
Do the same for the “Instrument Program Flow” setting:
So far, so good. Now let’s add the extra step so my scripts can get the project’s environment variables.
Select your primary target, and go into its Build Phases. Click “Add Build Phase” at the bottom, selecting “Add Run Script”. Edit it so it runs XcodeCoverage/exportenv.sh:
This last step has tripped up some people. If you use OCUnit — where the the main target is executed with tests injected — make sure you add the script to your main target (your app or library), not your test target.
On the other hand, if you use a different testing framework that depends on a separate testing app, then add the script to your test app, not the main target.
Run!
Now we’re ready for the good stuff. Run your unit tests.
Open a Terminal window, and cd to your project’s XcodeCoverage folder. Once the tests complete, run my getcov script:
Things will fly by. Eventually, a browser window should open. Now you can see your code coverage!
Change test code, then re-run
We measure code coverage to find the holes in our tests. So now let’s say you’ve added some new tests, and want to measure the coverage. As long as you haven’t changed your production code, the quickest way is to clean out the coverage data and re-measure. To do this, run the cleancov script:
With the previous results zeroed out, now we can re-run the tests, then do getcov again.
Change production code, then re-run
Time has passed, and you’ve made changes to your production code. When you decide to measure code coverage again, you should clear out earlier build artifacts. (Leftovers confuse lcov.)
To do this, hold down the Option key in Xcode’s “Product” menu. Select “Clean Build Folder”.
Now you’re ready to re-run the tests, followed by getcov.
Modify the scripts
There are two places in the XcodeCoverage scripts you may want to modify for your project:
First: In the envcov.sh script, LCOV_INFO determines the name shown in the report. By default, the name is Coverage.info. But if you want to maintain coverage reports for multiple projects, it would be best to change this name to match the project name.
Second: If your project uses third-party libraries, you’ll probably want to exclude them from code coverage measurements. You can do so by editing the getcov script and changing the exclude_data() function.
What I do is put all third-party libraries into a folder named ThirdParty. Then I exclude the pattern "ThirdParty/*" using lcov’s --remove command.
Let me know
Let me know how it goes! Do you have any questions, about XcodeCoverage in particular, or about code coverage in general?
Photo by Salim Virji (license)
Thanks for sharing this Jon, i’ve been banging my head against my desk for some time trying to simplify this task.
No matter how I try and set up the project though I get the following error;
ERROR: no .gcda files found in /Users/ which results in no trace, so no valid records.
I tried setting specifically at target level instead of porject level but it had the same effect. Have you ever seen this?
Thanks!
In order to get it working, I had to set the target specifically, project level settings didn’t work and if you’re running tests in the simulator, it must close before it generates files. Once that was all done it worked a treat. Thanks Jon.
Adam, I’m glad it was helpful! Two questions:
When you set it at the project level, and it didn’t take at the target level, was your target overriding those settings?
What do you mean “it must close before it generates files”?
Hmm, the path above got truncated, it wasn’t looking at users it was a valid path to the build folder in derived data
I tried generating the code coverage files from Xcode 4.5 with no luck.
My first attempt was with a C++ program that used Google Testing Framework and figured that the above instructions may only work with projects using Apple’s built in unit testing, so I created a small Cocoa app and used Apple’s built in Unit-Testing framework, but I still don’t get any output files.
I working on Mac OS 10.7, so I went over to my Mac OS 10.8 machine and created a Cocoa app there and again, I couldn’t find the files.
What may I be missing?
Thanks!
As another test, I made an iOS project with the same settings and still no luck.
Jaime, if you post your sample project somewhere, I’ll take a look.
Where shall I place it? Is there a mechanism on this site to post attachments? Thanks!
You can email me: jon [at] qualitycoding.org
Hi Jon,
Thanks for the great article!
This actually work with my Google C++ tests and it is a great addition to my development workflow!
Thanks again!
-Jaime
Glad to hear it, Jaime!
thanks for this post.
i got this error.
Found 30 data files in /Users/ignazioc/[.....]/Objects-normal/i386
Processing i386/AboutViewController.gcda
geninfo: ERROR: /Users/ignazioc/Desktop/[.....]/Objects-normal/i386/AboutViewController.gcno: could not open file
but “AboutViewController” has no test associated…maybe i need to have at least one test for class to run this tool?
thanks.
Ignazio, I work on legacy projects with many classes not covered by tests. So it must be something else. Try creating a sample project with some toy tests, then see if you can diagnose the difference.
I am trying to get this to work and got everything setup except when I run getcov, I get and error
geninfo: ERROR: need tool gcov!
Do I still need gcov to run this? I am running Moutain Lion with the latest Xcode.
Thanks!
Yes, download gcov-1.10. Then place the lcov-1.10 folder inside the XcodeCoverage folder.
I did download lcov, installed it into the XcodeCoverate folder and ran the script getcov but lcov seems to want gcov installed. Confused by your last replay since you said I did need gcov but you pointed to lcov.
Output from getcov
~/Library/Developer/Xcode/DerivedData/ContactManager-dxjozpbnyibstudkdiqnmmswcw
/Build/Products/Debug ~/Dev/ContactManager/XcodeCoverage
~/Dev/ContactManager/XcodeCoverage
Capturing coverage data from /Users/scottdensmore/Library/Developer/Xcode/DerivedData/
ContactManager-dxjozpbnyibstudkdiqnmmswcw/Build/Intermediates/
ContactManager.build/Debug/ContactManager.build/Objects-normal/i386
geninfo: ERROR: need tool gcov!
Reading tracefile Coverage.info
lcov: ERROR: cannot read file Coverage.info!
Reading tracefile Coverage.info
lcov: ERROR: cannot read file Coverage.info!
Reading data file Coverage.info
genhtml: ERROR: cannot read file Coverage.info!
The file /Users/scottdensmore/Library/Developer/Xcode/DerivedData/ContactManager-dxjozpbnyibstudkdiqnmmswcwyw/Build/Products/Debug/lcov/index.html does not exist.
Thanks,
Scott
Ohhh, sorry! To get gcov, go to Xcode Preferences and into the Downloads tab. Install Command Line Tools. I’ll add this to my instructions, thanks for staying on me.
No problem. I just realized that and was going to post it. I don’t install the tools anymore since most things work without them.
Found one more problem with the script. If you use NATIVE_ARCH you get some problems. I get i386 even though it should be x86_64. You should use CURRENT_ARCH. I found this discussion on the apple list : http://lists.apple.com/archives/xcode-users/2011/Jul/msg00086.html. I will send you a pull request with the change.
Merged. Many thanks!
Hi Jon,
Great post.
I am having issue setting up the Code Coverage for static library project.
I’m getting errors something like below:
Undefined symbols for architecture i386:
“_llvm_gcda_emit_arcs”, referenced from:
___llvm_gcov_writeout in libSomething.a(a.o)
I will not get this error if I disable ‘Instrument Program Flow’.
Any Idea?
Thanks
Hisyam, are you using an older version of Xcode?
I’m using the latest Xcode 4.5.2.
I have revisited my steps. It seems I have misconfigured the “Generate Test Coverage Files” and “Instrument Program Flow” settings.
I have redone the steps and everything works well now.
I am using a library project and getting the same errors
Undefined symbols for architecture i386:
“_llvm_gcda_emit_arcs”
….
Any clue? (on a new application project it works OK, so I assume all is correct)
To reproduce, create a library, setup the project , build is OK, runiing the unit tests generates the problem
I am using xcode 4.5.1
Same answers as elsewhere, Lior. Since it works on a brand-new project — with unit testing enabled — there’s a difference in your Build Settings. If you’re allowed to shared your project with me, I can take a look.
I am trying to implement code coverage with Kiwi Testing. I am not sure if you have ever worked with Kiwi but when I run the code coverage it seems to be returning me code coverage of the test files which is just 100% since the files are getting run all the way through. Is there any way to match the spec’s I’m writing to the files to get the code coverage of the actual file.
Thanks.
Matt, there’s no reason this shouldn’t work with Kiwi. Check the build settings at your target level and see if they were successfully applied.
I have figured out my error. I had your script running in my unit tests not my project it self. It is now working great. Thanks.
Ah! Makes sense, and good to know the symptom. Thanks for reporting back.
Hi, thank you for this great tool. However, do you know how I can get code coverage when the Test target does not compile the actual sources?
I have set up a test target, that holds a dependency on the main target and only has the test classes compiled in the “Compile sources” phase.
Sebastian, it sounds like you’re missing these steps: http://twobitlabs.com/2011/06/adding-ocunit-to-an-existing-ios-project-with-xcode-4/
No, that was not the problem. Actually, I had the same problem as Matt. Adding your script to my main target (and activating the Build settings for the global target) worked. Thank you very much for the script!
OK, that’s too many people stumbling at the same place. I’ve added a note warning people about this. Thanks!
Great post, John! First try with a *new* project in Xcode 4.5 and everything works as you described. However, when perform the same steps on an existing project of mine, no .gcda files end up being generated into the intermediates build folder, and the Coverage.info in the lcov folder stays at 0 bytes.
Would you know what causes this? It’s a library project, but that wouldn’t be the problem, right?
Here’s a link to the gist containing the terminal message and a link to the project on github: https://gist.github.com/c22e52d412d2fb22e457
Cheers, Eric-Paul – Amsterdam, the Netherlands
Hello to Amsterdam! Since it works on a new project but not yours, try to isolate the difference. I’d check a few things:
- When you look at Build Settings for your main app target, did “Generate Test Coverage Files” and “Instrument Program Flow” propagate, or are they overridden to NO?
- Is the exportenv.sh script specified on your main app target?
- Are you running Debug configuration?
I got the same error. Test Coverage Files and Instrument Program Flow are set t be YES. The exportenv.sh script is specified on my main app target.
I have no clue why no .gcda files end up being generated. Any ideas?
Nuo, what activity are you measuring? Is it unit tests? If so, which unit testing framework?
GHUnitIOS
I found those .gcda files in my testapp build. :s i was following your steps.
Good! It’s a little confusing with test apps that are separate from the production app.
Hi Jon – thanks for these tools! Testing with a simple project, using Xcode 4.6, in the LCOV – coverage report, AppDelegate.m, drilled in to the functions page, I get 0 Hit Count for “-[AppDelegate .cxx_destruct]“. What does that mean?
Eric, I’ve never seen cxx_destruct in my coverage results. Are you using Objective-C++?
Hi Jon, It seems a great tool, but I get 0 hit count for any classes I’ve unit tested. It’s not an IOS project. I’ve set up the project as explained here. Does it work the same ? Where should I look to find errors ? Many thanks
Carol, it should work for Mac development, too. Find your DerivedData and navigate down through Build/Intermediates/app.build/Objects-normal. Next to your object files, do you see gcda and gcno files?
Thanks for the tutorial! I have everything up and running, but the code coverage seems wrong. I wrote some test using Specta but the coverage % is always 0%. Only app delegate is 100% and some of my first viewController class got about 3% coverage. I tried Clean project and ./clearcov several times but still no luck. Do you have any idea on this?
BTW, I also tried to read .gcda coverage files with CoverStory. But still got 0% coverage results. So it seems like tools are working correctly but I am missing something. :/
btw, I’m using XCode 4.6
I’m using XCode 4.6
I found the solution. ./getcov is reading coverage files from my main target, not the test target. I have to edit OBJECT_FILE_DIR_normal in env.sh to make it work.
Hello, I succeded to make it work by running the script on the test target instead of the app target. The proble was the same as Hlung. I also had the same issue with CoverStory. Now every thing works just fine. BTW I’m now using your test template, OHHamcrest and OCMockito. Thanks for those great tools !
Yeah, I ended up doing the same as you Carol
Instead of fixing OBJECT_FILE_DIR_normal, moving the script to test target made it generate the correct path for reading the test target coverage files. BTW, I’m using Specta and Expecta to write tests.
Yes, I had written my instructions from an OCUnit perspective, where what gets executed is the actual app (injecting tests) instead of a separate testing target.
Carol, can you confirm that you’re not using OCUnit?
I’ll update the post to note this. Thanks!
Hi Jon,
I following your instructions but I get only .gcno files and no .gcda files. I’m using Xcode 4.6 and GHUnit 0.5.6. The tests run fine in simulator.
Have you any idea how I can fix this problem.
Thank you very much for the script!
GHUnit relies on a separate testing app, rather than injecting tests into the production app. So add the exportenv.sh script to the testing app, not the production app.
Hi Jon,
Thanks for such a great article!
I tried your steps and it works great on the XCode 4.5.1 version. I had a question however, should I set the Generate Test Coverage and Instrument Program Flow options for both Debug and Release versions, or should those settings be configured only for Debug? I guess these settings would probably come into effect only when you run the unit tests, and have no impact in the file size of the release binary. Is that correct?
Thanks,
Asheesh
Asheesh, I’m glad you found it useful.
Instrumentation changes the generated code. Don’t use it for shipping code. That’s why I do it for Debug only.
Hi Jon,
this post is really helpful.
I set up the code coverage for XCode 4.5 based on your instruction.
Thank you very much!!
When I generate output with lcov command, it has warning for all .gcno files ‘.gcno version ’404*’, prefer ’402*”. I searched in website, it’s caused by GCC version inconsistency.
But still I can get coverage data. will the data still accurate?
Regards
Yi
Yi,
Yes, the data is still accurate. The warnings are a nuisance, but harmless. Still, if anyone at Apple is reading… it would be nice to make this quiet…
How can I convert coverage.xml from gcovr to html report?
Sorry, I haven’t used gcovr so I don’t know. But there’s nothing stopping you from using gcovr for automation, then using XcodeCoverage manually on-demand. You certainly don’t need full HTML reports as often as automated reporting.
Thanks, works great. Can I use this in jenkins? I tried to convert output to cobertura xml, but without success
For Cobertura XML, use gcovr instead.
When I am entering ./getcov. I get the folllowng message:
No such file or directory
Also do I need to copy the Icov-1.10.
If so how would the shell script getcov be in the xcodecoverage directory because everything gets copied into the icove-1.10 sub folder.
Also do note I have installed the command line tools using xcode.
Thanks
Sanjeet
Sanjeet,
Yes, you need to download lcov-1.10 and place it within the XcodeCoverage folder. Use this link: http://downloads.sourceforge.net/ltp/lcov-1.10.tar.gz
As I said in mz earlier post
I had put icov1.10 in the XCodeCoverage folder and I get the following message when I tzpe the shell script ./getcov
No such file or directory
If I had that problem, I’d probably start adding echo statements to diagnose the shell script.
I get the following message:
envcov.sh: line 7: env.sh: No such file or directory
Ah! That’s more helpful. It’s not finding env.sh. You need to add a run script to execute XcodeCoverage/exportenv.sh. It’s the last step in the section “Set up your project”.
Currently I am not getting the dcda files generated.
Which testing framework are you using?
unittest++
I am having the same problem with OCUnit.
For OCUnit. I have added the shell script to export the project settings into the app target.
For unittest++ I am added the script to the test target.
Hmm. Strange. At this point, my advice is to set up sample applications using XcodeCoverage. See if you can get it working in the sample, apart from your real project.
Amazing job Jon! It worked like a charm.
Really simple and fast to configure.
Thanks a ton for sharing this!
It worked! Thanks for sharing a success story, Pablo
I have it working using OCUnit. unittest++ which is using c++ instead of objective c is still giving grief. Only the gcno files are being generated. Let me know if you have any ideas?
Hi Jon,
Thanks again for these articles, really useful. I have the code coverage up and running which is really cool to see. As I said in my comment on your UIViewController TDD post, I have been playing around with unit testing with Kiwi a little bit, but didn’t really know what was required for a full TDD implementation. Taking a look at my code coverage shows me that I have a lot of work to do!
I have a quick question, the coverage listing output is showing that I have no coverage on two functions that are not under my control; __copy_helper_block_ and __destroy_helper_block_. I believe this is probably to do with the way Kiwi is implemented, as I normally declare my SUT as a __block variable so that I can modify it within the Kiwi test blocks. Do you know of any way of ignoring the coverage for these two functions? I have taken a look inside your getcov script, and I see the exculde_data() function, I’m just not sure what to write inside that function so that these two items are ignored.
Any advice would be appreciated.
Sincerely,
Ben.