Publishing results of MDN Short Surveys

[dontcallmedom]

WebDX has already helped with designing and launching 2 short surveys on MDN:

• one on which browsers developer are paying attention when deciding to use a feature
• one on their interoperability need around proposals made for Interop

I expect further MDN short surveys to follow suite. The policy they run under allows for making the results public.

There are 4 levels of publicity we can consider for these results:

1. make the raw data public on a case by case basis (e.g. if we assess they provide easily interpretable data and low risk of misinterpretation)
2. make the raw data public in all cases, but with an editorial interpretation and communication on a case by case basis
3. raw data and interpretation in all cases, but with communication efforts on a case by case basis
4. raw data, interpretation and communication in all cases

My sense from the people involved in the first 2 surveys was that (2) was the minimum we should consider (which I agree with); ideally, I would prefer (3) (i.e. we always provide an interpretation, even if that interpretation is to the effect of "we think the question was wrong" or "we don't understand this"), but this requires a greater commitment than what we've discussed so far.

I don't really see a reason why we should go to (4) - if the interpretation of the results is that the survey was lacking in one way or another, it's a useful learning, but not one that we need to expand outreach capital on IMO.

As a starting point, I suggest we publish the content of the survey itself, the metadata on its running (period, MDN pages it was displayed on, percentage of page views it was displayed on) and the raw results as CSV data of MDN short surveys on this repo in a dedicated folder; I would then seek volunteer to write up an interpretation for review by the group; if there is no volunteer or if we can't get agreement on such an interpretation, this would probably be a sign that (3) is unlikely to be achievable.

Thoughts?

[tidoust]

That looks good to me. I just note that we might want to review the raw data before publication and redact comments, i.e. replace them with something like [redacted xxx] (where xxx is a number to make sure these values cannot get agglomerated by mistake), at least when:

1. the comment could be used to identify someone. For instance, I see one comment in the browsers survey that looks like a cookie string and includes an email address. There is also a SessionID column in the raw data, not sure what session it identifies but probably worth dropping?
2. the comment contains offending content. For instance, one of the comments in the browsers survey contains an anatomical reference. It is arguably benign, but there could be more problematic content in future surveys.

[robnyman]

I generally think that we should provide numbers and percentages, but – like @tidoust said – avoid any comments for the risks that come with that.

[robnyman]

I think we should also explore if there's a simple way of using this raw data to present some nice charts with the results of each survey we run somewhere. Personally I feel that since we run the surveys on MDN, insights.developer.mozilla.org/ seems like a good place to publish.

We need to, of course, create a basic template for presenting that on MDN, and who would do the work of creating these graphs.

[captainbrosset]

Agreed with what's been said so far. We should publish the raw data in all cases (with redacted comments where needed), and publish an interpretation to the best of our abilities. In some cases, interpreting the results might take more resources than we have, and that's ok, the raw data should still be valuable over time.

Publishing data on the github repo sounds good. This should be this group's source of truth, and it could then be used for fancier interpretations and communications in other places such as the insights MDN site.

Over time, I wonder if the WebDX group will run surveys in other places than MDN though. If it does, then having our own website to communicate about the surveys might be something look into.

Also, about the insights website, note that its github repo is now archived. There will probably be some logistics associated with turning it back on and getting the stack running again.