Update act-rules-format.md for mapping secondary accessibility requirements

[kengdoj]

The addition of the section for secondary accessibility requirements allows rules to include accessibility requirements that may be satisfied, not satisfied, and further testing is needed for either.

Additional naming of these requirements include "situational", "related", "incidental", and "supplemental". Open to other suggestions too!

[WilcoFiers]

Since this impacts implementations, I don't think this can be optional. There could be no secondary requirements for a rule, but I think if there are success criteria that sometimes fail, they have to be listed. Otherwise what should be consistent implementations won't be.

[WilcoFiers]

I think this may need to be flipped. Atomic rules do not have a relationship with composite rules. I.e. you could in theory add a composite rule without updating the atomic rule. I think this requirement should be flipped, in that a composite rule must not include accessibility requirements unless they are part of all its atomic rules. I.e. the compisite rule has the exact same accessibility requirements as its atomic rules. The main difference being that what is secondary and what is not will differ.

[tbostic32]

From how I am reading it, lines 248 and 251 are saying the same thing: an atomic rule cannot map to something that the composite rule does not map to.

In 248 I read it as the atomic rule must map to a (possibly improper) subset of the composite rule's mapping. And in 251, I read it as the composite rule must map to the union of all of the mappings (conformance and secondary) present in the atomic rules.

So I'm not sure it needs to be flipped, but some clarification might be helpful.

[HelenBurge]

I think "Additional" would be better as not meaning lesser but also not difficult to understand the context.

[carlosapaduarte]

I don't understand the 'may' clause. The accessibility requirements that could not be satisfied when the rule outcome is failed should be listed, because they fall under the 'must' clause, correct? So, I don't understand what the 'may' clause is referring to.

[Jym77]

That's about the related one, e.g. Scenario 4: A rule to test 2.4.4 by checking that links have non-empty name could also trigger failure of 1.1.1 on area. We do not want to mark an implementation as non-consistent because it is flagging 1.1.1 in some cases, and it could make sense to do so for some implementation (e.g.RGAA is testing area elements separately). But this is not a conformance requirement since there is no implication "rule fail => 1.1.1 not satisfied".

The "may" part is a bit of a catch-all (e.g., the same rule could also be mapped to 1.4.1 under the same reasoning that some links are only distinguished by color and sometimes would be failing both). This would be for rule writers to decides (and for example, consider unlikely that an implementation would report all the failure of a link together, while it is likely that it would report both 2.4.4 and 1.1.1 on an image link without name; so 1.1.1 should be a secondary mapping while 1.4.1 shouldn't).
That's a bit too loose to my taste, but I'm afraid we can't do much better in this document. This kinda says "rule writter may add secondary requirement as they see fit" 🤷

[WilcoFiers]

Yeah, we spent months trying to come up with a stricter definition of secondary. It doesn't seem possible.

[carlosapaduarte]

Suggested change

When an ACT Rule is designed to test conformance to one or more [=Accessibility requirements documents=], the rule <em class="rfc2119">must</em> list all [=accessibility requirements=] from those documents that are not satisfied when one or more of the [=outcomes=] of the rule is `failed`. The rule <em class="rfc2119">may</em> list accessibility requirements that could be not satisfied when the rule outcome is failed. There are two types of accessibility requirements:

When an ACT Rule is designed to test conformance to one or more [=Accessibility requirements documents=], the rule <em class="rfc2119">must</em> list all [=accessibility requirements=] from those documents that are not satisfied when one or more of the [=outcomes=] of the rule is `failed`. The rule <em class="rfc2119">may</em> list accessibility requirements that might be not satisfied when the rule outcome is failed. There are two types of accessibility requirements:

Changing just this word makes a lot of difference for me. Would it be okay?

[carlosapaduarte]

Given that an outcome of passed can mean further testing is needed, I don't really understand why are we saying that having only passed outcomes equals the accessibility requirement being satisfied.

[Jym77]

I understand this as being the "Passed" condition for "satisfying test".
I.e. "when, additionally, this condition is true, the rule is a satisfying test".

I agree it's not super clear.

[carlosapaduarte]

Why do we need this definition, and why is it mixed with the definition of "Conformance Requirements"?

[WilcoFiers]

This is from the current version of the spec. We can't take it out. The reason it needs to be here is that secondary requirements cannot be satisfying tests.

[carlosapaduarte]

I find this paragraph confusing, and don't think it helps in understand what is a Secondary Requirement.
The following paragraph is much better at defining a secondary requirement (i.e. failed outcomes require further testing).

[Jym77]

That's about the related one, e.g. Scenario 4: A rule to test 2.4.4 by checking that links have non-empty name could also trigger failure of 1.1.1 on area. We do not want to mark an implementation as non-consistent because it is flagging 1.1.1 in some cases, and it could make sense to do so for some implementation (e.g.RGAA is testing area elements separately). But this is not a conformance requirement since there is no implication "rule fail => 1.1.1 not satisfied".

The "may" part is a bit of a catch-all (e.g., the same rule could also be mapped to 1.4.1 under the same reasoning that some links are only distinguished by color and sometimes would be failing both). This would be for rule writers to decides (and for example, consider unlikely that an implementation would report all the failure of a link together, while it is likely that it would report both 2.4.4 and 1.1.1 on an image link without name; so 1.1.1 should be a secondary mapping while 1.4.1 shouldn't).
That's a bit too loose to my taste, but I'm afraid we can't do much better in this document. This kinda says "rule writter may add secondary requirement as they see fit" 🤷

[Jym77]

I understand this as being the "Passed" condition for "satisfying test".
I.e. "when, additionally, this condition is true, the rule is a satisfying test".

I agree it's not super clear.

[Jym77]

With the current definition, 2.4.9 is still a Conformance Requirement for any 2.4.4 rule.
I think we need to add something like:

When requirement B is a stricter version of requirement A, and requirement A is a Conformance requirement; then the rule must list requirement B as a Secondary requirement (not as a Conformance requirement).

With "stricter version" being:

Requirement B is a stricter version of requirement A if every test subject that does not satisfy A also does not satisfy B.

[Jym77]

I think we should move this as Scenario 2.
Both this one and Scenario 1 are about too version of a requirement, a lax and a strict, so it makes sense to keep them close to each other.

[kengdoj]

Suggested change

	Rules that can be used to determine if an accessibility requirement is *satisfied* are called <dfn>satisfying tests</dfn>.
	Rules that can be used to determine if an accessibility requirement is *satisfied* are called <dfn>satisfying tests</dfn>. When all of the outcomes are `passed`, the accessibility requirement is <dfn>satisfied</dfn> for the test subject.

[kengdoj]

Suggested change

- Passed: When all of the outcomes are `passed`, the accessibility requirement is <dfn>satisfied</dfn> for the test subject.

[kengdoj]

Changed this from a bullet to a sentence in the paragraph instead.

[kengdoj]

Suggested change

A secondary accessibility requirement is a requirement that is correlated with the rule, but for which the rule is not designed to test. The outcome of the rule impacts the result of the accessibility requirement, but the rule is not intended to test the conformance of that requirement. This correlation often results in some of the rule's test cases not satisfying the secondary accessibility requirement.

A secondary accessibility requirement is a requirement that is correlated with the rule, where the outcome of the rule may impact the result of the accessibility requirement, but the rule is not intended to test the conformance of that requirement. This correlation often results in some of the rule's test cases not satisfying the secondary accessibility requirement.

[carlosapaduarte]

Much clearer, thanks!

[WilcoFiers]

Suggested change

	- Conformance Requirements
	- [Conformance Requirements](#conformance-requirements)

[WilcoFiers]

Yeah, we spent months trying to come up with a stricter definition of secondary. It doesn't seem possible.

[WilcoFiers]

Suggested change

	### Outcome Mapping ### {#outcome-mapping}
	For each conformance requirement in the mapping, an ACT Rule <em class="rfc2119">must</em> indicate what the [=outcomes=] of the rule mean for satisfying an accessibility requirement for that [=test subject=].
	### Outcome Mapping ### {#outcome-mapping}
	For each conformance requirement in the mapping, an ACT Rule <em class="rfc2119">must</em> indicate what the [=outcomes=] of the rule mean for satisfying an accessibility requirement for that [=test subject=].

[WilcoFiers]

This is from the current version of the spec. We can't take it out. The reason it needs to be here is that secondary requirements cannot be satisfying tests.

[WilcoFiers]

That's not the only difference. A requirement can only be a conformance requirement if when the rule fails, the requirement is not satisfied.

[WilcoFiers]

Suggested change

	5. whether the requirement is a conformance requirement or a secondary requirement.
	5. whether the requirement is a conformance requirement or a secondary requirement.
	<div class="note">
	<p><strong>Editors note:</strong> The ACT Task Force is looking for feedback on the definitions of conformance requirement and secondary requirements. These are currently defined based on author intent, which is not necessarily as strict as many other definitions in this document. One possible solution could be to use "requirement strictness". Requirement B is considered stricter than A if every test subject that does not satisfy A also does not satisfy B, but that A can be satisfied when B is not satisfied.</p>
	<p>The terms <em>conformance requirement</em> and <em>secondary requirements</em> are not yet finalized. The ACT Task Force is looking for alternative terms that could be more descriptive.</p>
	</div>

[tbostic32]

You propose "requirement strictness" as a solution in the editors notes but I don't see how that would solve the problems seen in scenarios 2 and 4. I think it could be part of the solution, but could not be the whole solution.

As I have re-read the scenarios the "rules of thumb" for measuring them seems to be the below for me. I don't think you could wrap 2 and 3 into requirement strictness.

1. "Requirement Strictness" - Handles scenarios 1 and 3
2. "Passed = satisfied outcome, failed $\neq$ failed outcome" - Scenario 2
3. "Applicable to only some, but not all, passed and failed test cases" - Scenario 4

[WilcoFiers]

You may be right. I haven't looked into it yet. It's something to discuss I think.

[tbostic32]

Suggested change

A rule was designed to test a minimum accessibility requirement and there exists a stricter requirement. The rule’s failed outcomes can determine that the stricter accessibility requirement is not satisfied, and the rule’s passed outcomes may not determine that the stricter requirement is satisfied. It is possible that the accessibility requirement may be not satisfied when the rule's outcomes are passed. The stricter requirement is a Secondary requirement.

A rule was designed to test a minimum accessibility requirement and there exists a stricter requirement. The rule’s failed outcomes can determine that the stricter accessibility requirement is not satisfied, but the rule’s passed outcomes may not determine that the stricter requirement is satisfied. Thus, it is possible that the accessibility requirement may be not satisfied when the rule's outcomes are passed. The stricter requirement is a Secondary requirement.

I would prefer this wording slightly, but am okay if you don't want to change it.

[tbostic32]

Suggested change

	A rule was designed to test an accessibility requirement and under certain conditions, other accessibility requirements apply. In this scenario, the other accessibility requirements are sometimes, but not always, satisfied or not satisfied because they are not always applicable in the rule. These other accessibility requirements are a Secondary requirement.
	A rule was designed to test an accessibility requirement and under certain conditions, other accessibility requirements apply. In this scenario, the other accessibility requirements are only sometimes evaluated since they are not always applicable in the rule. These other accessibility requirements are a Secondary requirement.

I found the second sentence here a bit confusing. I think this edit says the same thing but simplifies the langauge a bit? (the word evaluated might need to be changed)

As a second question, does "not always applicable in the rule" == "there exists a passed or failed test case for which the requirement is not applicable" (wording could be better). In my opinion, I think this should be the case, if it isn't then it means we are missing a test case. I think making "not always applicable in the rule" more concrete by referencing test cases makes it more measurable (e.g., applicable to some but not all passed and failed examples == secondary requirement under scenario 4) and makes it more understandable.