This post describes how to setup test code coverage report for an iOS project on Jenkins. As an example, here I’ll use the ConnectSDK-Lite project. This post assumes your Jenkins setup is based on my post: Jenkins in OSX guest in VirtualBox for iOS jobs – full setup guide, if something doesn’t work (e.g., xctool), please check that post first.
@richardbuckle There’s no more need for the `__gcov_flush()` workaround. Try eliminating the GTM coverage stuff.
And it worked with Xcode 6.1! No need to commit extra files. Let’s get to reports straight away.
Reports
NB: the commands below are tested with a library target, and may or may not work for an application target.
I found two ways of publishing test code coverage reports on Jenkins:
lcov: it generates really nice HTML pages, so you should use the HTML publisher plugin on Jenkins. A disadvantage is that you don’t get a diff between jobs.
gcovr: it converts gcda files from Clang-LLVM into cobertura XML format, so you can use the Cobertura plugin on Jenkins and get diffs between jobs.
I’ll describe both ways.
Gathering required information
The first build step to publish the reports is to generate coverage information. Here is a sample Execute shell build step:
12345678
# cleanup the output directory firstrm -rf "out/build_ccov"# run tests with code coveragexctool -scheme ConnectSDK -sdk iphonesimulator -configuration Debug \ -reporter plain -IDECustomDerivedDataLocation="out/build_ccov"\GCC_GENERATE_TEST_COVERAGE_FILES=YES GCC_INSTRUMENT_PROGRAM_FLOW_ARCS=YES \ clean test
Important bits here are:
-scheme ConnectSDK -sdk iphonesimulator – specify your scheme that has tests; the sdk must be iphonesimulator because these test can run on the iOS simulator only;
-IDECustomDerivedDataLocation="out/build_ccov" – specifies where all the build products should be placed; it simplifies our life in that we don’t have to search for the proper path in ~/Library/Developer/Xcode/DerivedData/;
GCC_GENERATE_TEST_COVERAGE_FILES=YES GCC_INSTRUMENT_PROGRAM_FLOW_ARCS=YES – these flags generate coverage files;
First, this step collects coverage info to a file, then removes info about iOS frameworks (/Applications/*) and 3-rd party frameworks (stored in Frameworks/ directory), because we’re interested in our code only, and generates an HTML report at the end.
In Post-Build Actions, add “Publish HTML reports” step. Add an entry with these values: HTML directory to archive: out/coverage_report/, Index page: index.html, Report title: Test Code Coverage Report (lcov). You’ll get a link in the project’s left side bar to a nice HTML report.
(Since it’s a python script, you can also install it with pip3: pip3 install gcovr. However, in my case of python 3.4 and gcovr 3.4, I needed to patch line 1742 in the /usr/local/bin/gcovr script: in for i in xrange(len(lines)): replace xrange with range (https://github.com/gcovr/gcovr/pull/89))
Only one command here, and an interesting part is the excluding regex: .*#(?:ConnectSDKTests|Frameworks)#.*, here we’re ignoring tests themselves (ConnectSDKTests) and 3-rd party frameworks (Frameworks) directories. I found experimentally that you need to replaces slashes / in paths with hashes #.
Add a Publish Cobertura Coverage Report post-build action with Cobertura xml report pattern set to out/coverage.xml, set Source Encoding to UTF-8. You’ll get a link in the left side bar and also a code coverage trend graph and fully functional report on coverage of packages, files, classes, lines, and conditionals.
Oh, important stuff: this publisher step should be in the post-build actions step (you could use it in build steps using Any Build Step Plugin), otherwise you get a null pointer exception.
Screenshots
To get an idea what the result looks like, here are a couple of screenshots:
Project page with code coverage trend and links to reports:
Cobertura coverage report:
Lcov code coverage report index:
Seeing the real test code coverage report should motivate developers to improve the system, write more tests, and increase the coverage.
Hope it helps. I wish I had found such a detailed instruction before!