From Batik test suite to EchoSVG tests
One of the most important changes between Apache Batik and EchoSVG was the creation of a new testing infrastructure, and the rationale for that work is worth being described here.
Instead of using a standard testing framework like JUnit or TestNG, Batik developers created their own infrastructure based on XML configuration files. Adding or removing tests was done by editing the right XML configuration file, with its own set of XML configurations. The results of the test were returned through a DOM document and if an exception occurred, it was suppressed and most of the information that it carried was lost, making debugging very difficult.
The old testing framework even came with a small set of self-tests but was annoying to use and essentially unmaintained. The Batik project began to migrate its own tests to JUnit but did not get very far. Therefore, the decision was made to get rid of the old testing framework and replace it with JUnit.
But once this was achieved, it became apparent that it wasn't going to be the only challenge because the test suite had reproducibility issues.
The SVG Generator allows to generate SVG through a Graphics2D object, and is one of the most popular use cases for Apache Batik. In Batik it is the batik-svggen module, in EchoSVG is echosvg-svggen.
The ability to compare the generated SVG document with a reference is essential to the svggen tests, and Batik did that by comparing the text serializations of both documents, line by line. That was good because when a difference was detected, the test would report the expected line versus the generated one, instead of just claiming that "the two documents differ".
The problem was that the Batik DOM implementation does not enforce any specific ordering in the serialization of element attributes, and sometimes it happened that the same DOM document had different serializations, even when produced by the same JDK in the same computer (but under slightly different circumstances). For example, the following line
<g fill="rgb(102,102,153)" text-rendering="optimizeLegibility" font-size="15px" font-weight="bold" stroke="rgb(102,102,153)"is effectively the same as
<g fill="rgb(102,102,153)" text-rendering="optimizeLegibility" stroke="rgb(102,102,153)" font-size="15px" font-weight="bold"but then a bogus test failure would be triggered. This happened from time to time, and was one of the reasons why the Batik tests were not 100% deterministic and reproducible.
I then rewrote the comparison so the attribute ordering did not matter, but sometimes the serialization was still varying by a couple of characters being taken from a line and added to the next.
So as a final check, a full DOM-level comparison was executed to avoid false test failures. Unfortunately, the document/node comparison methods in Batik's DOM were buggy and had to be fixed (see #18). (Incidentally, almost two years after the fork Batik decided to do that DOM-level comparison as well in commit 6174d6c, however they haven't noticed the aforementioned isEqualNode bug!)
A few svggen tests were not being run by Batik because it lacked the necessary features, but those were added to EchoSVG (see #16 and #17) and now all of Batik's Generator tests are run.
See one of the images from the fix to issue #16 (Batik vs EchoSVG):
All of the rendering tests rely on comparing the produced image with a reference. In EchoSVG the comparisons are more accurate and reproducible than in Batik (see IMAGE_COMPARISONS.md for a detailed description). The tests in this project can even compare the metadata of two images.
One of the capabilities of the Batik transcoder module is to transform a Windows Metafile document to SVG. When I executed the tests, I found that the conversion was hardware-dependent because Batik was using an archaic convention to transform inches to pixels, using the actual resolution of the device (which was something found in very old Web specifications). If one uses different length units to specify the sides of an object, this results in distorted shapes.
I checked that Batik was producing images where the drawn objects often had a different shape than those from other available WMF converters. A problem by itself, and yet another factor preventing test reproducibility.
I fixed the problem with commit 6ca74495. The images from that commit speak by themselves (Batik vs EchoSVG):
Nearly all of the Batik tests use fonts that are only available by default on Windows systems, and even others that are non-default and have to be installed manually (for example Courier). This was an important issue for cross-platform execution, not to talk about CI environments where it would be a pain to have to install the fonts before executing the tests.
I replaced most of the proprietary fonts with free alternatives which are provided by the test suite, and in the few cases where this was unsuitable (for example in the WMF sample files), the tests are only executed if the font is installed, otherwise counting as "ignored tests".
The TranscoderInput interface had no way to set a character encoding for input resolution, and as the transcoding tests were designed around a supplied URI (that almost always is a file:), they relied on the default JRE encoding being Windows-1252. Due to that reason a few tests failed on non-Windows platforms, as well as on Java 18 or later (when the default encoding became UTF-8).
With #73, EchoSVG's TranscoderInput gained support for encoding, and the test suite was modified to explicitly use UTF-8 (with the sample files being converted).
Batik's PerformanceTest is an obsolete class intended for performance measurements. Instead of trying to reinvent the wheel by attempting to fix it, this project migrated to JMH, a mature benchmarking framework.
EchoSVG has a largely improved, reliable test suite with more than a thousand tests, which is regularly run by Github CI. Batik now also runs a CI but many tests are being excluded from those runs (and those that aren't excluded are not 100% reproducible and may not run reliably in the future). In contrast, no tests are excluded from EchoSVG's CI.
There is still some pending work to be able to execute some of the tests on non-Windows platforms, but this project has gone a long way in test manageability and reproducibility.