-
Notifications
You must be signed in to change notification settings - Fork 7
/
test-suite-howto.html
265 lines (210 loc) · 20.1 KB
/
test-suite-howto.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
<!DOCTYPE html>
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="content-type" content="text/html; charset=utf-8"/>
<title>Test Suite How-To Guide</title>
<style type="text/css"></style>
</head>
<body>
<h1><span>Test Suite How-To Guide</span></h1>
<p><span>The ITS 2.0 test suite is located at the following address <a href="https://github.com/w3c/its-2.0-testsuite">https://github.com/w3c/its-2.0-testsuite</a>. There are four main parts to the ITS 2.0 test suite which are:</span></p>
<ol>
<li><span><a href="#1">Conformance testing for HTML & XML</a></span></li>
<li><span><a href="#2">Testing for NIF</a></span></li>
<li><span><a href="#3">Input file validation</a></span></li>
<li><span><a href="#4">XLIFF samples</a></span></li>
</ol>
<h2><span><a name="1"></a>1. Conformance testing for HTML & XML</span></h2>
<p><span>This conformance testing is used to determine whether ITS 2.0 implementations meet the
ITS 2.0 specification standard. The test suite has a set of test documents for both XML and
HTML which are then used to validate the different ITS 2.0 constructs available for each
data category. There are a total of 225 test files spread across all 19 data categories for
both HTML and XML.</span></p>
<h3><span>1.1 What is Conformance testing? </span></h3>
<p><span>Conformance testing is a type of testing where a system has to meet some specified standard.
In the case of ITS 2.0 this standard is dictated by the W3C standards document http://www.w3.org/TR/its20/.
To test for compliance a series of tests have been developed externally by TCD. The specification states that
there are 4 different types of conformance which have to be tested for and more details on this can be found
at the following address <a href="http://www.w3.org/TR/its20">http://www.w3.org/TR/its20/#conformance</a>. </span></p>
<h3><span>1.2 Details of the conformance tests </span></h3>
<p><span>The test suite is used to test user implementations conformance to the ITS 2.0 standard. The test suite has a set of test documents for both XML and HTML which are then used to validate the different ITS 2.0 constructs available for each data category. The ITS 2.0 test suite has 225 input test files which have been created for the 19 data categories. There are 136 XML input files and 89 HTML input files. All of these files have been validated successfully against the schemas for ITS 2.0. Section 4 of this document provides more information on how to validate XML files and validate HTML files.
For the conformance testing, each test file in the test suite requires at least two implementations in order to be allowed into the ITS 2.0 standard. This is in line with the conformance clauses which can be found at the following address <a href="http://www.w3.org/TR/qaframe-spec/">http://www.w3.org/TR/qaframe-spec/</a>. The requirement of two implementations per test suite file helps in catching errors in the standard along with making sure implementers are using the standard correctly.
</span></p>
<h3><span>1.3 Outcome of conformance tests </span></h3>
<p><span>
The detailed breakdown of the conformance testing results and of systems which are certified in complying with the ITS 2.0 standard can be found in the test suite implementation report. This report is located at the following address <a href="http://www.w3.org/International/multilingualweb/lt/drafts/its20/its20-implementation-report.html">http://www.w3.org/International/multilingualweb/lt/drafts/its20/its20-implementation-report.html</a>.
</span></p>
<h3><span>1.4 How to use the test suite?</span></h3>
<p><img height="301" src="testsuite-visualization.png" width="483" alt="test suite output format" />
<br/>
<span><b>Figure 1: Testsuite processing 	Input to Output</b></span></p>
<p><span>The validation of the input files is done through processing the input file so that it outputs a file to match a corresponding gold standard output file. The gold standard was developed to be easy to understand and implement for conformance testers. The gold standard output format was developed by the ITS 2.0 working group. Figure 1 above describes the test suite files process. The ITS 2.0 test suite input files are located at the following address <a href="https://github.com/w3c/web-platform-tests/tree/master/conformance-checkers/html-its">https://github.com/w3c/web-platform-tests/tree/master/conformance-checkers/html-its</a>.</span></p>
<h3><span>1.4 Gold Standard output specifications details:</span></h3>
<p><span>	The gold standard output files have the following characteristics:</span></p>
<ol>
<li><span>Every element and attribute path from the XML and HTML file are listed (apart from
the content within script elements in HTML)</span></li>
<li><span>The output has to be tab-delimited
format:								</span>
<p><span class="c3"><i>/html/body[1]/p[1]/span[2]								annotatorsRef="text-analysis|http://enrycher.ijs.si"								taConfidence="0.5"								taIdent="301467919"								taSource="Wordnet3.0"</i></span></p></li>
<li><span>The attributes within elements have
to be in alphabetical order:</span>
<p><span class="c3"><i>/doc/header[1]/its:rules[1]/its:locQualityIssueRule[2]								</i></span></p>
<p><span class="c3"><i>/doc/header[1]/its:rules[1]/its:locQualityIssueRule[2]/@locQualityIssueComment</i></span></p>
<p><span class="c3"><i>/doc/header[1]/its:rules[1]/its:locQualityIssueRule[2]/@locQualityIssueProfileRef</i></span></p>
<p><span class="c3"><i>/doc/header[1]/its:rules[1]/its:locQualityIssueRule[2]/@locQualityIssueSeverity</i></span></p>
<p><span class="c3"><i>/doc/header[1]/its:rules[1]/its:locQualityIssueRule[2]/@locQualityIssueType</i></span></p>
<p><span class="c3"><i>/doc/header[1]/its:rules[1]/its:locQualityIssueRule[2]/@locQualityIssuesRef</i></span></p>
<p><span class="c3"><i>/doc/header[1]/its:rules[1]/its:locQualityIssueRule[2]/@selector</i></span></p></li>
<li><span>The rules output also have to be in alphabetical order:</span>
<p><span class="c3 c12 c8">/html/body[1]/p[1]/span[2]								annotatorsRef="text-analysis|http://enrycher.ijs.si"								taConfidence="0.5"								taIdent="301467919"								taSource="Wordnet3.0"</span></p></li>
<li><span>The rules output does not contain Pointer style attribute values but it has to have its equivalent as results from ITS processing (unless it is in the target pointer data category then it must display targetpointer=”....” and the pointer details): 								</span>
<p><span><b>Incorrect:</b></span></p>
<p><span><i>/doc/para[1]/issue[2]
								locQualityIssueTypePointer="misspelling"								locQualityIssuesRefPointer="#l1234"								</i></span></p>
<p><span><b>Correct:</b></span><span class="c4"
>								</span></p>
<p><span class="c3"><i>/doc/para[1]/issue[2]								locQualityIssueType="misspelling"								locQualityIssuesRef="#l1235"								</i></span></p></li>
<li><span>The rules output for local html rules have to be like their global
counterparts:</span>
<p><span><b>Incorrect:</b></span></p>
<p><span><i>/doc/p[1]
								its-loc-quality-issue-type="misspelling"								locQualityIssuesRef="#l1235"								</i></span></p>
<p><span class="c4 c6"><b>Correct:								</b></span></p>
<p><span><i>/doc/p[1]
								locQualityIssueType="misspelling"
								locQualityIssuesRef="#l1235"								</i></span></p>
</li>
</ol>
<h3><span>1.6 How gold standard output is compared to implementers output?</span></h3>
<p><span>The conformance test output of a proposed ITS2.0 implementation is located in the folder</span>
<span>its-2.0-testsuite/its2.0/</span><span class="c4 c8 c6">outputimplementors</span>.
<span>This</span><span class="c4 c6">	</span><span>can be tested against the gold
standard output files located in the</span>
<span class="c4 c6">its-2.0-testsuite/its2.0/expected</span>
<span>folder. This can be done simply through performing a diff of the implementation’s output files and the corresponding gold standard output files.
To automate this process for implementers a test suite dashboard was created which supports the following tasks :</span></p>
<ol>
<li><span>Help to track the process of implementers in relation to the tests in which they
were committed to complete</span><span class="c4 c22 c6">	(indicated on the dashboard
via</span>
<span class="c4 c12 c22 c8 c6">N/A which means the implementer did not commit to run the
test)</span><span>.</span></li>
<li><span>Help to track if the implementers output file matches the corresponding gold
standard output file</span><span class="c4 c12 c22 c8 c6">	(indicated on the
dashboard via fnf= the output file from the implementer has not been found or OK = the output
file is identical to the reference output file)</span><span>.</span></li>
<li><span>Help to track if the output file that is in the output folder for a particular tests
doesn't match the corresponding gold standard output file</span><span>	</span><span
class="c4 c12 c22 c8 c6">(indicated on the dashboard via error = an error occurred, e.g.
the output file is not available or it is not identical to the reference output file. Move
the mouse over error to see details)</span><span>.</span></li>
<li><span>Help to track whether the implementer has committed an output file or not for a corresponding test </span>
<span class="c4 c12 c8 c6 c22">(indicated on the dashboard via fnf= the output file from the
implementer has not been found)</span><span>.</span></li>
<li><span>The dashboard can also track how many tests a particular implementer has left to
run.</span></li>
</ol>
<h3><span>1.7</span>
<span>Validating Output Test Files</span></h3>
<p><span>To validate the implementers output files the test suite dashboard has to be compiled so
that a diff across all files in the</span>
<span>its-2.0-testsuite/its2.0/outputimplementors</span>
<span class="c4 c8">	folder can be performed </span><span>	</span><span class="c4 c8"
>against the corresponding files in the 	gold standard output</span>
<span>its-2.0-testsuite/its2.0/expected</span><span class="c4 c8">	 folder. The test
suite dashboard can be compiled by doing the following:</span></p>
<ol>
<li><span>Download saxon.jar from here:</span>
<span><a href="http://saxon.sourceforge.net"
>http://saxon.sourceforge.net</a></span></li>
<li><span>Then use this command (Linux/Mac/Windows): java -jar /path/of/file/saxon.jar
testsuiteMaster.xml testsuiteDashboard.xsl -o:testSuiteDashboard.html</span></li>
<li><span>Upload newly compiled testsuiteDashboard.html to the git hub</span></li>
<li><span>Check the state of your files in the related data categories on this web
page:</span>
<span><a class="c15"
href="https://raw.github.com/w3c/its-2.0-testsuite/blob/master/its2.0/testSuiteDashboard.html">
http://htmlpreview.github.io/?https://raw.github.com/w3c/its-2.0-testsuite/blob/master/its2.0/testSuiteDashboard.html</a></span></li>
</ol>
<p><span>The files for the test suite dashboard are as follows:</span></p>
<ol>
<li><span>testsuiteMaster.xml -</span>
<span>has a list of the implementer's and the tests that they are involved in and aids in
the creation of the testSuiteDashboard.html</span></li>
<li><span>testsuiteDashboard.xsl -</span>
<span>does the diffs between the implementer's output and the gold standard
output.</span></li>
<li><span>testSuiteDashboard.xml -</span>
<span>gives information on the errors between the diff's for the implementors output and
the gold standard output</span></li>
<li><span>testSuiteDashboard.html -</span><span>	the html test suite dashboard which is
located live at the following address:</span>
<span><a class="c15"
href="https://raw.github.com/w3c/its-2.0-testsuite/blob/master/its2.0/testSuiteDashboard.html">
http://htmlpreview.github.io/?https://raw.github.com/w3c/its-2.0-testsuite/blob/master/its2.0/testSuiteDashboard.html</a></span><span>	</span></li>
</ol>
<h2><span><a name="2"></a>2. Testing for</span>
<span class="c2">NIF</span><span>	</span><span>	</span></h2>
<p>This part of the test suite is used to determine whether ITS 2.0 implementations of NIF 2.0 meet the ITS 2.0 specification standard for NIF usage. The mapping between an XML and HTML document annotated with ITS to and from NIF is not a normative part of the Its2.0 specification. The NIF test suite has a set of test documents for HTML for the
<span>Localisation Quality Issue</span><span class="c4">	data category which is then used to validate the different aspects of ITS 2.0 and NIF against various constructs available to the Localisation Quality Issue data category. There are a total of 11 files only for HTML in the Localisation Quality Issue data category. The input files are located in the </span><span class="c4">	</span>
<span class="c4 c6">its-2.0-testsuite/its2.0/nif-conversion/input</span>
<span class="c4">folder.</span></p>
<p><span>The validation of the input files is done through processing the input file so that it outputs a file to match a gold standard output file. The gold standard for NIF is RDF output using the ITS 2.0 ontology. The gold standard output for NIF can be reached by following the NIF conversion algorithm discussed in the ITS 2.0 specification located </span>
<span><a class="c15" href="http://www.w3.org/TR/its20">http://www.w3.org/TR/its20/#conversion-to-nif</a></span><span
class="c4">.</span></p>
<h3><span>2.1 How gold standard 	NIF output is compared to implementers output?</span></h3>
<p><span>The NIF output files are compared via the use of SPARQL queries done over the implementers RDF/NIF output files (.ttl). If the SPARQL queries are successful then the NIF output files are correct and meet the gold standard. The implementer’s NIF test output files are located in the its-2.0-testsuite/its2.0/nif-conversion/expected folder.</span></p>
<h3><span>2.2 Validating NIF output files</span></h3>
<p><span>Prerequisites:</span><span>	Java and Unix Shell</span></p>
<ol>
<li><span>create a temporary folder for output files (hence called $datafolder)</span></li>
<li><span>read ITS files from "its2.0/nif-conversion/input/" one by one, convert to NIF and
write output files in turtle to $datafolder</span></li>
<li><span>go to directory cd its2.0/nif-conversion/sparqltest</span></li>
<li><span>run : ./executeTests.sh ../relative/pathTo/$datafolder</span></li>
</ol>
<p><span>Explanations of output:</span></p>
<ol>
<li><span>If no message appears between "Running: test1.sparql" and "Done: test1.sparql" the
test was successful.</span></li>
<li><span>Otherwise the output filename and additional debug output is shown</span></li>
</ol>
<h2><span><a name="3"></a>3. Input file validation</span></h2>
<p><span>This part of the test suite is important to ensure that any test input files, including any used by implementers in addition to the test suite, represent valid use of ITS2.0 annotation in HTML and XML. More information about this validator can be found </span>
<span><a href="http://validator.nu"> http://validator.nu</a></span><span>	and</span>
<span><a href="http://about.validator.nu/">http://about.validator.nu/</a></span><span>.</span></p>
<h3><span>3.1 Validating input test files</span></h3>
<p><span>The following sections detail how to validate the test suite input files both HTML and XML.</span></p>
<h3><span>3.1.1 Validating XML test files</span></h3>
<ol>
<li><span>Download and install Ant from</span>
<span><a href="http://ant.apache.org/">http://ant.apache.org/</a></span></li>
<li><span>Run 'ant validate-xml' command in its2.0 directory</span></li>
</ol>
<h3><span>3.1.2 Validating HTML test files</span></h3>
<ol>
<li><span>Download and install Ant from</span>
<span><a href="http://ant.apache.org/">http://ant.apache.org/</a></span></li>
<li><span>Download html5-its-tools from</span>
<span><a href="https://github.com/kosek/html5-its-tools"
>https://github.com/kosek/html5-its-tools</a></span></li>
<li><span>Modify its2.0/build.properties to point to your local copy of
html5-its-tools</span></li>
<li><span>Run 'ant validate-html' command in its2.0 directory</span></li>
</ol>
<h3><span>3.1.3 Validating all test files</span></h3>
<ol>
<li><span>Make sure that XML and HTML validation described above works for you</span></li>
<li><span>Run 'ant' command in its2.0 directory</span></li>
<li><span>Please note that HTML schema doesn't supports RDFa so RDFa attributes are reported
as errors</span></li>
<li><span>Please note that currently Schematron validation is not performed so some errors are
not detected</span></li>
</ol>
<h2><span><a name="4"></a>4. Xliff samples</span></h2>
<p><span>The test suite also contains some sample XLIFF files. These are not used in conformance testing, but demonstrate the representation of ITS 2.0 metadata in XLIFF. These XLIFF files are located in </span><span>	</span><span class="c6">its-2.0-testsuite</span><span>/</span><span
class="c6">its2.0/</span><span>xliffsamples</span>
<span>folder.</span></p>
</body>
</html>