Handle NUnit tests with status 'Warning'

[crossan007]

Currently, NUnit tests with a status of "Warning" are interpreted as "skipped" in the Jenkins UI (https://issues.jenkins-ci.org/browse/JENKINS-56241)

There does not appear to be a direct mapping of "Warning" to the JUnit test result type, which appears to be the "intermediate" step for importing NUnit tests into Jenkins.

This causes the NUnit Jenkins plugin to treat tests with status of "Warning" as a "Fail" rather than a "Skip."

Since "Warning" tests don't include a stack trace, the stack trace field in the JUnit test result is a string indicating: This test case was reported as a "Warning" in NUnit, but converted to "Fail" by Jenkins NUnuit Plugin.

I am opening this PR as a draft, since I want to make this behavior configurable, with the end goal of the following:

• NUnit warnings -> Jenkins build unstable if warningTestsUnstableBuild:true

[slide]

The best thing to do is come up with an example nunit output file and then an expected junit xml file and implement a test.

[crossan007]

Would it be too far a departure from the current plugin state to cause all NUnit warnings to mark the Jeknins build as "UNSTABLE" as proposed below:

 /**
     * <p>Configuration value specifying how to handle NUnit warnings.  
     * Since there is no direct mapping from NUNit's 'Warning' result to a JUnit result, 
     * we need a way to handle the behavior</p>
     * <ul>
     *  <li> unstable - NUnuit warnings are counted as failures; however, 
     *  any count (even exceeding healthScaleFactor) will mark the build as UNSTABLE,
     *  and will not cause the build to fail even if failedTestsFailBuild is set to true</li>
     *  <li> softfail - NUnuit warnings are counted as failures. 
     *  A warning count below the healthScaleFactor will mark the build as UNSTABLE, 
     *  but a A warning above healthScaleFactor will mark the buld FAILURE.
     *  Warnings will not cause the build to fail if failedTestsFailBuild is set to true</li>
     *  <li> normalfail - NUnit warnings are counted as failures, and will be subject to 
     *  the configuration of failedTestsFailBuild and healthScaleFactor to determine whether 
     *  the build is UNSTABLE or FAILURE</li>
     * </ul>
     * <p>Defaults to <code>skip</code>.</p>
     */
    private boolean warningNUnitTestResultHanlingBehavior;

[kalimera8764]

Hello,

Not very familiar with Github, so excuse any mistakes...
Seems I got a similar issue. FYI, I am using NUnit 2.6.4 to execute my tests and v 0.24 for the NUnit plugin in Jenkins.

I had an issue with some tests that were failing with "Wrong number of arguments provided".
In the console it mentioned those as "Invalid".
The result in the nunit xml output was: "executed="False" result="NotRunnable""
Such case in the documentation (here: https://nunit.org/docs/2.6.4/testFixture.html) seems to mention that non runnable tests are handled as errors.
However, in the plugin, those are mentioned as skipped.

To solve the issue, i modified nunit-to-junit.xsl. See the change in the attached below...
nunit-to-junit.zip

I could not find a place for all the possible options for the attribute "result", so we have a more whole solution.

Testing wise, i got a couple executions containing cases -> plugin behaviour
tests pass -> pass
failures, errors -> fail + fail message (stack trace)
skipped -> skipped + skipped message
invalid -> fail + fail message

With kind regards,
Kostis.

[crossan007]

@michaelclinch Good catch.

I'm no longer part of the company where I was using this plugin, so I no longer have a stake in seeing this merged. Feel free to adapt and modify this as you see fit :-)

[jonesbusy]

I've recently adopted the plugin, but my goal is only to keep it updated.

if someone is still interested by this feature feel free to rebase, resolve conflict etc... And I would be more than happy to review and release it.

Regards,

[aw2003]

2026 calling - I have this same problem. I'm using "Warning" as a mechanism to signal a potential error state of the devices under test which requires further investigation (kind of a workaround for something so Specflow BDD can continue to run) - @jonesbusy could this still be merged? I believe we're using this plugin.

[jonesbusy]

See #19 (comment)

Other solution in 2026 to use the JUnit Plugin that consume the JUnit format.

All modern test framework have reporters to generate reports in many formats.