Skip to content

Commit

Permalink
Add testResult remapping
Browse files Browse the repository at this point in the history
  • Loading branch information
@alflennik
alflennik committed Aug 26, 2021
1 parent d6645a3 commit 254885f
Show file tree
Hide file tree
Showing 6 changed files with 352 additions and 101 deletions.
2 changes: 1 addition & 1 deletion .eslintrc.json
View file
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@
"ecmaFeatures": {
"jsx": true
},
"ecmaVersion": 2018,
"ecmaVersion": 2020,
"sourceType": "module"
},
"plugins": ["react", "json", "prettier", "jest"],
Expand Down
118 changes: 73 additions & 45 deletions server/graphql-schema.js
View file
Original file line number Diff line number Diff line change
Expand Up @@ -234,13 +234,6 @@ const graphqlSchema = gql`
Link to the HTML page which will be tested.
"""
exampleUrl: String!
# TODO: consider switching to a URL which is more standard
# TODO: consider converting to an array if we support a larger number of
# more-focused and "compositional" startup scripts
"""
Link to JS file which, when run, will prepare the example for testing.
"""
startupScriptContent: String
"""
Loosely structured data which may or may not be consistent or fully
populated across all test plan versions.
Expand All @@ -259,8 +252,8 @@ const graphqlSchema = gql`
"""
type Test {
"""
A base64-encoded unique ID which contains some information used by the
LocationOfData system.
A unique ID which encodes some information used by the LocationOfData
system.
"""
id: ID!
"""
Expand All @@ -277,29 +270,23 @@ const graphqlSchema = gql`
The AT mode the test was written to expect.
"""
atMode: AtMode!
# TODO: determine whether this field should remain following the
# introduction of machine-readable instructions
# TODO: consider switching to a URL which is more standard
# TODO: consider converting to an array if we support a larger number of
# more-focused and "compositional" startup scripts
"""
Human-readable sentences detailing steps that must be completed by
testers before the step which captures the AT output.
Link to JS content which, when run, will prepare the example for testing.
"""
preCommandInstructions: [String]!
# TODO: consider merging instructions and commandInstruction when we
# introduce machine-readable instructions
startupScriptContent: String
# TODO: rethink when introducing machine-readable instructions
"""
The commandInstruction is a human-readable sentence which will be
executed by testers immediately after completing the instructions.
An example is "navigate to an unchecked checkbox". This step produces
the AT output that testers will record (see the ScenarioResult type for
more information). The actual command to use, e.g. "TAB" or "DOWN",
etc., will differ depending on the AT, and are listed one by one in the
array of scenarios.
Human-readable sentences detailing steps that must be completed by
testers before capturing the AT output.
"""
commandInstruction: String!
instructions: [String]!
"""
List of ways the test can be completed, each of which needs to be
executed separately. There might be a different number of Scenarios
for each AT, based on factors like the number of commands that the AT
for each AT, based on factors like the number of Commands that the AT
supports to complete a task.
"""
scenarios(atId: ID): [Scenario]!
Expand All @@ -308,6 +295,13 @@ const graphqlSchema = gql`
info on the Assertion type.
"""
assertions(priority: AssertionPriority): [Assertion]!
# TODO: simplify the test renderer's API and remove this field
"""
An intentionally opaque Base64-encoded JSON string currently containing
data like extraneous key names which are needed to format the test the
way the test renderer expects.
"""
testRendererRemappingData: String!
}
"""
Expand All @@ -319,25 +313,49 @@ const graphqlSchema = gql`
"""
type Scenario {
"""
A base64-encoded unique ID which contains some information used by the
LocationOfData system.
A unique ID which encodes some information used by the LocationOfData
system.
"""
id: ID!
"""
The AT which this scenario is testing.
"""
at: At
# TODO: reconsider when adding machine-readable instuctions
# TODO: rethink when adding machine-readable instuctions
"""
The name of a command, i.e. a key combination or another kind of AT
input, written like "TAB" or "DOWN". The command accomplishes the
human-readable purpose given by the commandInstruction field on the Test
type. There will be one scenario for each command the AT supports, so a
commandInstruction like "navigate to an unchecked checkbox" might have
four scenarios for the keys "X", "F", "TAB" and "DOWN" which all
accomplish that purpose.
The command accomplishes the purpose given by the last instruction on
the Test type. There will be one scenario for each command the AT
supports, so navigating to an unchecked checkbox might have four
scenarios for the keys "X", "F", "TAB" and "DOWN" which all accomplish
that purpose.
"""
command: String!
command: Command!
}
"""
A key combination or another kind of AT input, which has a human-readable ID
like "TAB" or "DOWN" and a textual representation like "Tab" or "Down
Arrow".
"""
type Command {
"""
Human-readable ID which is similar to the text such as "CTRL_OPT_DOWN"
"""
id: ID!
"""
A human-readable version of the command, such as "Control+Alt+Down"
"""
text: String!
}
"""
Minimal plain representation of a Command.
"""
type CommandInput {
"""
See Command type for more information.
"""
id: ID!
}
"""
Expand All @@ -361,6 +379,11 @@ const graphqlSchema = gql`
pass or fail.
"""
type Assertion {
"""
A unique ID which encodes some information used by the LocationOfData
system.
"""
id: ID!
"""
Whether this assertion contributes to the test failing or not.
"""
Expand All @@ -370,7 +393,7 @@ const graphqlSchema = gql`
"""
A human-readable version of the assertion.
"""
manualAssertion: String!
text: String!
}
"""
Expand All @@ -379,8 +402,8 @@ const graphqlSchema = gql`
"""
type TestResult {
"""
A base64-encoded unique ID which contains some information used by the
LocationOfData system.
A unique ID which encodes some information used by the LocationOfData
system.
"""
id: ID!
"""
Expand All @@ -401,6 +424,11 @@ const graphqlSchema = gql`
AT, including the results of all assertions.
"""
scenarioResults: [ScenarioResult]!
"""
An intentionally opaque Base64-encoded JSON string containing non-result
data like which fields should be highlighted due to validation errors.
"""
testRendererInternalState: String!
}
"""
Expand Down Expand Up @@ -432,8 +460,8 @@ const graphqlSchema = gql`
"""
type ScenarioResult {
"""
A base64-encoded unique ID which contains some information used by the
LocationOfData system.
A unique ID which encodes some information used by the LocationOfData
system.
"""
id: ID!
"""
Expand Down Expand Up @@ -490,8 +518,8 @@ const graphqlSchema = gql`
"""
type AssertionResult {
"""
A base64-encoded unique ID which contains some information used by the
LocationOfData system.
A unique ID which encodes some information used by the LocationOfData
system.
"""
id: ID!
"""
Expand Down Expand Up @@ -533,13 +561,13 @@ const graphqlSchema = gql`
"""
type UnexpectedBehavior {
"""
Human-readable ID which is similar to the description.
Human-readable ID which is similar to the text.
"""
id: ID!
"""
Human-readable sentence describing the failure.
"""
description: String!
text: String!
"""
One of the unexpected behaviors is "other", which means the user must
provide text explaining what occurred. For all other unexpected
Expand Down
Loading

0 comments on commit 254885f

Please sign in to comment.