Code Coverage with OpenClover for Java code in CI/CD

How to set up OpenClover for Apache Ant and run in a CI/CD server.

Nov 16, 2022

Lately, I have been looking into code coverage solutions in Java. I initially tried Cobertura coverage library, but it was a pain to get all its dependent libraries (many of them) and it was incompatible with Java 17. So, I tried Clover and found it was much easier to set up. Clover was acquired by Atlassian (renamed Atlassian Clover) in 2007 but was turned open source in 2017 (this version is named OpenClover).

This tutorial will show you how to set up OpenClover with Apache Ant.

Table of Contents:
· Demonstration
  ∘ Commands you can run yourself
· Setting up OpenClover
· Run with Ant and Generate Your Clover Report
· Full Build.xml
· Run code coverage in CI/CD or CT server
  ∘ Coverage Check

Note that OpenClover has its own Apache Ant installation guide. However, I had some trouble with the generating the build report when I followed the guide so I have written this article.

Demonstration

Below (animated GIF) is a run of the Clover code coverage for my sample Java project.

Commands you can run yourself

Follow the commands from the above video to try it out yourself (Apache-Ant is required).

git clone https://github.com/testwisely/buildwise-samples.git
cd buildwise-samples/ant-junit
ant -lib lib/clover clean with.clover test.unit.console
ant clover.html
open build/clover-html/index.html

Setting up OpenClover

The ant-junit project in the buildwise-samples repo had the OpenClover set up in the build.xml file. To set up OpenClover in your project, follow these steps:

Add the clover.jar file (downloadable from openclover.org) to the lib folder.

I put it in a separate subfolder lib/clover to separate it from the app’s libraries.

In the build.xml file, follow the Ant QuickStart Guide on the OpenClover website. For convenience, the steps are summarised below:

2. Add the Clover ant task definition at the top of the file before any targets are defined:

<taskdef resource="cloverlib.xml" classpath="lib/clover/clover.jar"/>

3. Create a target to set up Clover for use.

<target name="with.clover">
 <clover-setup/>
</target>

4. Add the desired report target for your report output type/s.
Clover supports HTML, PDF, XML and command-line reporting. I will use the HTML type here but see the guide for the other report types.

<target name="clover.html">
 <clover-html-report outdir="build/clover-html"/>
</target>

You can change the output directory in the outdir argument.

Run with Ant and Generate Your Clover Report

You need to run two Ant commands to get your Clover coverage report. First, run the tests with Clover activated (with.clover):

ant -lib lib/clover clean with.clover test.unit.console

Note: The -lib tells ant where to find the clover.jar file.

This will generate a .clover folder, which will contain several artefact files. If you look inside it, the files will be named similar to these:

ant-junit % ls .cloverclover4_4_1.db clover4_4_1.db1ve9rdjkh1hkc_2_exisgz_l3qlqbrv.s
clover4_4_1.db1ve9rdfmetdmg_0_exisgz_l3qlqbqv.s clover4_4_1.db1ve9rdjkh1hkc_2_njjup9_l3qlqbrv.s
clover4_4_1.db1ve9rdfmetdmg_0_njjup9_l3qlqbqv.s clover4_4_1.dbexisgz_l3qlqbqg
clover4_4_1.db1ve9rdhlfxfl1_1_exisgz_l3qlqbrl.s clover4_4_1.dbnjjup9_l3qlqbr0

Next, run the clover reporting target (in my case, HTML):

ant clover.html

Open the index.html file inside the output directory specified in the clover.html target. It will display the code coverage of your program.

Click the link to find out the coverage for each package and class.

Full Build.xml

For the full build.xml file, see the sample repo’s file here:

<?xml version="1.0"?>
<project name="YourProject" default="go" basedir=".">

    <property name="test.ui.dir" value="rspec"/>
    <property name="test.unit.dir" value="src/test/java"/>
    <property name="build.dir" value="build"/>
    <property name="src.dir" value="src/main/java"/>
    <property name="lib.dir" value="lib"/>
    <property name="instrumented.dir" value="build/instrumented"/>

    <!-- import buildwise-ant to run rwebspecs in BuildWise -->
    <!--
    <import file="buildwise-ant.xml" />
    <import file="buildwise-tomcat-ant.xml" />
    -->

    <!-- Atlassian Clover is now available as an open-source project.
         Effective April 11, 2017 Clover is no longer available for purchase or renewal.  -->
    <!-- reference: https://openclover.org/doc/manual/latest/ant- -installation-guide.html -->
    <taskdef resource="cloverlib.xml" classpath="lib/clover/clover.jar" />

    <target name="go" depends="prepare, compile, test.unit.console, package">

    </target>

    <!-- common build  -->

    <fileset id="classpath.library" dir="${lib.dir}">
        <include name="*.jar"/>
    </fileset>


    <path id="classpath">
        <fileset refid="classpath.library"/>
        <pathelement location="${build.dir}/classes"/>
    </path>


    <target name="clean">
        <delete dir="${build.dir}"/>
        <delete dir=".clover"/>
    </target>

    <target name="prepare">
        <mkdir dir="${build.dir}"/>
    </target>


    <target name="compile">
        <mkdir dir="${build.dir}/classes/main"/>
        <javac fork="true"
               memoryinitialsize="256m"
               memorymaximumsize="512m"
               srcdir="${src.dir}"
               destdir="build/classes/main"
               debug="true"
               includeantruntime="false"
               classpathref="classpath">
            <exclude name="**/SearchTest.java"/>
        </javac>
    </target>

    <path id="test.classpath">
        <fileset refid="classpath.library"/>
        <pathelement location="${build.dir}/classes/main"/>
        <pathelement location="${build.dir}/classes/test"/>
        <!-- needed for code coverage, preferally define another path: test.coverage.classpath -->
        <pathelement path="lib/clover/clover.jar"/>
    </path>

    <!-- compile unit test, run unit test -->
    <target name="test.unit.4" depends="test.compile">
        <junit haltonerror="false" haltonfailure="false" forkmode="">
            <classpath refid="test.classpath"/>
            <batchtest>
                <fileset dir="${test.unit.dir}">
                    <include name="**/*Test*"/>
                </fileset>
            </batchtest>
            <formatter type="brief" usefile="false"/>
        </junit>
    </target>

    <target name="test.compile" depends="compile">
        <mkdir dir="${build.dir}/classes/test"/>
        <javac fork="true"
               memoryinitialsize="256m"
               memorymaximumsize="512m"
               srcdir="${test.unit.dir}"
               destdir="build/classes/test"
               debug="true"
               includeantruntime="false"
               classpathref="test.classpath">
            <exclude name="**/SearchTest.java"/>
        </javac>
    </target>


    <target name="test.unit.console" depends="compile, test.compile">
        <java classpathref="test.classpath" classname="org.junit.platform.console.ConsoleLauncher" fork="true"
              failonerror="true">
            <arg value="--scan-classpath"/>
            <arg line="--reports-dir build/test-report"/>
        </java>
        <!--
        <junitreport todir="build/test-report">
          <fileset dir="build/test-report">
            <include name="TEST-*.xml"/>
          </fileset>
          <report format="frames" todir="build/test-report/html"/>
        </junitreport>
        -->
    </target>

    <!-- Failed to launch in-vm tests -->
    <target name="test.unit.notworking" depends="compile">
        <junitlauncher haltOnFailure="true" printSummary="true">
            <classpath refid="test.classpath"/>
            <testclasses outputdir="build/test-report">
                <fileset dir="build/test">
                    <include name="**/*Tests.class"/>
                </fileset>
                <listener type="legacy-xml" sendSysOut="true" sendSysErr="true"/>
                <listener type="legacy-plain" sendSysOut="true"/>
            </testclasses>
        </junitlauncher>
    </target>

    <!-- generate war file -->
    <target name="package">
    </target>

    <!-- === CODE COVARAGE === -->
    <target name="with.clover">
        <clover-setup/>
    </target>

    <target name="clover.html">
        <clover-html-report outdir="build/clover-html"/>
    </target>

    <!-- Usage: (must run in two steps; also needs to supply clover.jar by -lib flag)
       ant -lib lib/clover clean with.clover test.unit.console
       ant clover.html
    -->
    <target name="clover.all" depends="with.clover, test.unit.console"
            description="Runs a build with Clover and generates a Clover data">
    </target>

    <!--
    <target name="deploy">
      <antcall target="tomcat.deploy"/>
    </target>
    -->
</project>

Run code coverage in CI/CD or CT server

Zhimin: the above section shows how to get code coverage for Java code after unit testing. To make unit testing or code coverage useful, we need to run them in a continuous process.

Unit testing and code coverage are two steps in Continuous Integration. In this article, I will try to add code coverage to a CI/CD process in the BuildWise CT server. While I would use BuildWise mainly for executing automated End-to-End (UI) tests, it can be configured to run unit tests and code coverage as well.

Here is a report of the execution of a Junit suite in BuildWise.

The build step configuration for the CI/CD project:

Add two code coverage build steps, one to create coverage data, and the other to generate the report.

Trigger another build in BuildWise, the output (unit testing report) is the same. The report is not shown in BuildWise.

One way is to publish the coverage report as the build’s artifact. To do that, zip the coverage report in build.xml .

<target name="clover.html">
    <clover-html-report outdir="build/clover-html"/>
    <zip destfile="${build.dir}/coverage.zip" basedir="${build.dir}/clover-html"  />
</target>

Then, configure the build project setting to include the coverage.zip as a build artifact.

Trigger another run, you'll find the coverage.zip under Build artifacts .

Users can download it to view the full coverage report.

Coverage Check

If you want to enable verify code coverage against a specific threshold, add this to build.xml .

<target name="clover.check" depends="with.clover">
    <clover-check target="80%" failureProperty="coverageFailed" haltOnFailure="true"/>
</target>

Add another build step configuration (in BuildWise).

Trigger a run.

The build failed, due to the actual coverage 73.5% is below the target 80% .

Usually, I shall add/modify unit tests to increase the coverage. This time, I will show you a change in the target coverage threshold.

<target name="clover.check" depends="with.clover">
    <clover-check target="72%" failureProperty="coverageFailed" haltOnFailure="true"/>
</target>

Trigger another run in BuildWise.

Now it passed.

I don’t recommend using a CT server such as BuilidWise for unit testing/code coverage, which is better in CI servers such as Jenkins. This way, we can focus on the most challenging CI/CD task: automated end-to-end regression testing in the CT server.

Related reading:

The Agile Way

Discussion about this post