Request for Stats for the WDTK Transparency Report 2021 - 2 #922
Comments
|
Would it also be possible to get: Total number of requests ‘censored’ in 2021 - maybe number of censor rules applied? Sally |
|
Requests which are backpages aren't really removed from the site. There will be other reasons than those listed above, but those are the reasons recorded by the site. So there should be an "other" category. |
Including or excluding those not visible to the public? (Sorry for adding complexity!) |
|
WIP on this; just recording so that I can get back to it. from_date = Time.zone.parse('2020-11-01').at_beginning_of_day
to_date = Time.zone.parse('2021-10-31').at_end_of_day
period = from_date..to_date
hide_events =
InfoRequestEvent.
where(event_type: 'hide').
where(created_at: period)
# Total number
hide_events.count
# => 824
# Events grouped by report reason
hide_events.each_with_object(Hash.new(0)) do |event, memo|
reason = event.params[:reason]
memo[reason] += 1
end
# => { "vexatious" => 29,
# => "not_foi" => 671,
# => nil => 124 }
# Events grouped by new prominence value
hide_events.each_with_object(Hash.new(0)) do |event, memo|
reason = event.params[:prominence]
memo[reason] += 1
end
# => { nil => 700,
# => "hidden" => 8,
# => "backpage" => 11,
# => "requester_only" => 105 } |
|
I've been trying to figure out why we have hide events with no # Events grouped by report reason
hide_events.each_with_object(Hash.new(0)) do |event, memo|
reason = event.params[:reason]
memo[reason] += 1
end
# => { "vexatious" => 29,
# => "not_foi" => 671,
# => nil => 124 }It looks like this happens when we change prominence via "Edit metadata" rather than "Hide the request and notify the user". Ordinarily editing through "Edit metadata" records an …but we have a callback that checks the changed attributes recorded in the event, and if only prominence has changed to something other than I suppose we'll just have to say these were hidden for "other reasons". Why do we only have The |
|
Note that the above query may count multiple hide events against a single request, so it's not quite true to say "we hid 824 requests" – looks like it's 822 requests, with one or more requests being hidden multiple times for whatever reason. hide_events.count
# => 824
hide_events.pluck(:info_request_id).uniq.count
# => 822 |
|
So let's look at why we have lots of prominence_changes = hide_events.each_with_object(Hash.new(0)) do |event, memo|
reason = event.params[:prominence]
memo[reason] += 1
end
# => { nil => 700,
# => "hidden" => 8,
# => "backpage" => 11,
# => "requester_only" => 105 }In this case, these events look like they've been created through the "Hide the request and notify the user" action. In this case we log a We can see that through this action we set the |
|
Finally, censor rules: from_date = Time.zone.parse('2020-11-01').at_beginning_of_day
to_date = Time.zone.parse('2021-10-31').at_end_of_day
period = from_date..to_date
censor_rules = CensorRule.where(created_at: period)
# Total number created within the period
censor_rules.count
# => 881
# Number of individual requests affected by censor rules created within
# the period
censor_rules.pluck(:info_request_id).uniq.compact.size
# => 196
# Number of individual users affected by censor rules created within
# the period
censor_rules.pluck(:user_id).uniq.compact.size
# => 122
# Number of individual public bodies affected by censor rules created
# within the period
censor_rules.pluck(:public_body_id).uniq.compact.size
# => 1
# ---
censor_rules_applied_to_visible_requests =
censor_rules.
joins(:info_request).
references(:info_request).
where(info_requests: { prominence: %w(normal backpage) })
# Total number created within the period that are linked to visible
# requests
censor_rules_applied_to_visible_requests.count
# => 595
# Number of visible individual requests affected by censor rules
# created within the period
censor_rules_applied_to_visible_requests.
pluck(:info_request_id).uniq.compact.size
# => 188 |
|
Wasn't requested but while I was thinking about it, a little info on hiding of individual messages: from_date = Time.zone.parse('2020-11-01').at_beginning_of_day
to_date = Time.zone.parse('2021-10-31').at_end_of_day
period = from_date..to_date
# ---
edit_outgoing_events =
InfoRequestEvent.
where(event_type: 'edit_outgoing').
where(created_at: period)
hide_outgoing_events =
edit_outgoing_events.select { |e| e.params[:prominence] != 'normal' }
# Number of times we've hidden outgoing messages within the period
hide_outgoing_events.each_with_object(Hash.new(0)) do |event, memo|
reason = event.params[:prominence]
memo[reason] += 1
end
# => { "hidden" => 10,
# => "requester_only" => 35 }
# ---
destroy_outgoing_events =
InfoRequestEvent.
where(event_type: 'destroy_outgoing').
where(created_at: period)
# Number of times we've destroyed outgoing messages within the period
destroy_outgoing_events.count
# => 0
# ---
edit_incoming_events =
InfoRequestEvent.
where(event_type: 'edit_incoming').
where(created_at: period)
hide_incoming_events =
edit_incoming_events.select { |e| e.params[:prominence] != 'normal' }
# Number of times we've hidden incoming messages within the period
hide_incoming_events.each_with_object(Hash.new(0)) do |event, memo|
reason = event.params[:prominence]
memo[reason] += 1
end
# => { "hidden" => 43,
# => "requester_only" => 90 }
# ---
destroy_incoming_events =
InfoRequestEvent.
where(event_type: 'destroy_incoming').
where(created_at: period)
# Number of times we've destroyed incoming messages within the period
destroy_incoming_events.count
# => 105 |
|
Thank you for all the data I've now added those into the draft report. Apologies I seem to have forgotten to request one data set, which was total number of requests made. Sally |
No problem! from_date = Time.zone.parse('2020-11-01').at_beginning_of_day
to_date = Time.zone.parse('2021-10-31').at_end_of_day
period = from_date..to_date
# Total number of requests created in the period
InfoRequest.where(created_at: period).count
# => 100092
# Total number of outgoing messages created in the period
OutgoingMessage.where(created_at: period).count
# => 141131
# Total number of incoming messages created in the period
IncomingMessage.where(created_at: period).count
# => 237154 |
|
The published Transparency report can be found https://www.mysociety.org/2021/12/16/whatdotheyknow-transparency-report/ |
Is is possible to get stats from the site for the below; for the time period: 1 November 2020 - 31 October 2021
Total number of requests removed from the site
Broken down into
Visability:
Hidden Requests
Requsestor_Only
BackPage
Reason:
Not a valid FOI request
Personal correspondence
Subject access request
A vexatious request
Deadlines are:
The annual report is scheduled to go out on 16 December
Design is scheduled to be completed by 9 December
Ideally this means that copy should be ready by 2 December
Linked to:
#910
Sally
The text was updated successfully, but these errors were encountered: