Document compliance differences between different sdks

[justinabrahms]

Building off of #254, I've started to put together a script to pull together the differences the SDKs have. In working on parsing the different repositories, I noticed that some tests are easier to parse than others. I'm reaching out to better understand what folks think the right approach is for addressing this.

The spec_finder utility does a few things:

1. Looks in a source repository to find annotations of some sort about which specs are implemented and the language used.
2. Compare the numbers that SDK covers and the text they use to determine if there is a match.
3. Output a report (both human-friendly and json)

Do we complicate the spec_finder utility to better understand the semantics of the underlying languages? Or do we recommend some sort of easily-parsable convention like comments on each test? Or maybe we scrap this idea and have each language output their coverage to some known place that we can more easily deal with?

These were the ones that were a bit difficult to parse

• ruby
• js
• go
• php

Janky first attempt

Code:

import json
from collections import defaultdict

status_map = {
    'missing': '💔',
    'good': '✅',
    'extra': '❓',
    'different-text': '🎭',
    'unknown': ''
}

def main(files):
    reports = {
        # e.g.
        # 'cs': <json-file-contents>
    }
    output = defaultdict(lambda: {})
    # for example..
    #     '1.1.2': {
    #         'cs': 'extra',
    #         'rs': 'missed',
    #     }
    for filename in files:
        with open(filename) as f:
            report = json.loads(''.join(f.readlines()))

        ext = filename.split('-', 1)[0]
        reports[ext] = report

        for status, nums in report.items():
            for num in nums:
                output[num][ext] = status
    # yes this is gross
    print('''
    <style>
    thead {
      background-color: #333;
      color: white;
    }

    th, td {
      padding: .5em 2em;
    }


    tbody tr:nth-child(odd) {
      background-color: #fff;
    }

    tbody tr:nth-child(even) {
      background-color: #eee;
    }
    </style>
    ''')
    print('<table cellspacing="0" cellpadding="0">')
    print('<thead>')
    print('<tr>')
    print("<th>Number</th>") # account for number
    stable_langs = reports.keys()
    for ext in stable_langs:
        print('<th>%s</th>' % (ext))
    print('</tr>')
    print('</thead>')

    print('<tbody>')
    for num, lang_status in sorted(output.items()):
        print("<tr>")
        print("<td>%s</td>" % num)

        for lang in stable_langs:
            status = lang_status.get(lang, "unknown")
            print("<td>%s</td>" % status_map[status])

        print("</tr>")

    print('</tbody>')
    print('</table>')
if __name__ == '__main__':
    import sys
    main(sys.argv[1:])

Result:

[beeme1mr]

I would love to see something like this. What issues are you seeing in the JS repo? Would we just need to update the test names to closer align with the spec names?

[maxveldink]

I also like this idea of detecting where a requirement is implemented in the SDKs. I'm unsure if there's going to be an easy way to standardize the annotations easily. I wonder if, specifically to communicate spec-compliance, we have something like a spec_compliance.json or similar file that is up to date with the targeted spec version and requirement numbers. Each language could decide how much they want to automate publishing that artifact within the repo, but we can read that in as a standard format for documentation/reporting purposes.

[justinabrahms]

#254 already provides a way to provide per-repo instructions for how to find compliance. We could improve that. I'm thinking, at a minimum, we could have sdk authors do line-based comments like:

// spec:4.1.2:This is the text of the normative section of the spec.

and then parse that out? May run afowl of long-lines, but we can likely handle that too.

[thomaspoignant]

Yes comments are probably the easiest way to start with it.