Deploying to gh-pages from @ 8b3b6a6 🚀

andylolz · May 30, 2024 · 1b682bd · 1b682bd
commit 1b682bd
Show file tree

Hide file tree

Showing 8 changed files with 385 additions and 0 deletions.
diff --git a/_config.yml b/_config.yml
@@ -0,0 +1,8 @@
+title: Twitter community notes
+
+defaults:
+  -
+    scope:
+      path: ""
+    values:
+      layout: "default"
diff --git a/_data/.gitkeep b/_data/.gitkeep
diff --git a/_data/meta.json b/_data/meta.json
@@ -0,0 +1 @@
+{"scraped_at": "2024-05-30T04:20:28.465192+00:00", "most_recent": "2024-05-28T00:31:35+00:00", "total_notes": 10310, "total_fetched": 7538, "locked_until": "2024-05-30T04:08:11+00:00", "total_tweets": 8766}
diff --git a/_layouts/default.html b/_layouts/default.html
@@ -0,0 +1,51 @@
+<!DOCTYPE html>
+<html>
+<head>
+  <meta charset="utf-8">
+  <meta name="viewport" content="width=device-width, initial-scale=1">
+  <title>{{ page.title }}{% if page.url != '/' %} | {{ site.title }}{% endif %}</title>
+  <link href="//cdn.jsdelivr.net/npm/[email protected]/dist/css/bootstrap.min.css" rel="stylesheet">
+  <link rel="stylesheet" href="//cdn.datatables.net/v/bs5/jq-3.7.0/dt-2.0.7/fh-4.0.1/sp-2.3.1/sl-2.0.1/datatables.min.css">
+  <style type="text/css">
+    td {
+      word-break: break-word;
+    }
+    time {
+      cursor: help;
+      text-decoration: underline;
+      text-decoration-style: dotted;
+    }
+    .dt-type-numeric {
+      text-align: left !important;
+    }
+    .w-inherit {
+      width: inherit;
+    }
+  </style>
+</head>
+<body>
+  <script src="//platform.twitter.com/widgets.js" charset="utf-8"></script>
+  <script src="//cdnjs.cloudflare.com/ajax/libs/luxon/2.3.1/luxon.min.js"></script>
+  <script src="//cdn.jsdelivr.net/npm/[email protected]/dist/js/bootstrap.bundle.min.js"></script>
+  <script src="//cdn.datatables.net/v/bs5/jq-3.7.0/dt-2.0.7/fh-4.0.1/sp-2.3.1/sl-2.0.1/datatables.min.js"></script>
+
+  <div class="container py-3">
+    <header class="d-flex flex-wrap justify-content-end py-3 mb-4 border-bottom">
+      <ul class="nav nav-pills">
+        <li class="nav-item">
+          <a class="nav-link{% if page.url == '/' %} active{% endif %}" href="{{ '/' | relative_url }}">Home</a>
+        </li>
+        <li class="nav-item">
+          <a class="nav-link{% if page.url == '/about/' %} active{% endif %}" href="{{ '/about/' | relative_url }}">About</a>
+        </li>
+      </ul>
+    </header>
+
+    <main>
+      <h1>{{ page.title }}</h1>
+
+      {{ content }}
+    </main>
+  </div>
+</body>
+</html>
diff --git a/about/index.md b/about/index.md
@@ -0,0 +1,62 @@
+---
+title: How it works
+---
+
+Community note data is fetched regularly [from Twitter](https://x.com/i/communitynotes/download-data).
+
+This data is always a couple of days old (**most recent data is from <time class="dt" datetime="{{ site.data.meta.most_recent }}" title="{{ site.data.meta.most_recent | date_to_rfc822 }}">{{ site.data.meta.most_recent }}</time>, scraped <time class="dt" datetime="{{ site.data.meta.scraped_at }}" title="{{ site.data.meta.scraped_at | date_to_rfc822 }}">{{ site.data.meta.scraped_at }}</time>**).
+
+Notes are excluded if they meet any of the following criteria:
+
+* Created more than a week ago
+* Classifying the post as ‘not misleading’ (i.e. in support of the post)
+* Currently rated ‘unhelpful’
+
+We also attempt to filter out notes for deleted tweets and non-English tweets.
+
+### Special twitter language codes
+
+When Twitter can’t determine the language of a tweet, it uses one of several reserved language codes. For the purpose of language filtering, we’ve grouped these all together. But this is the breakdown:
+
+|---------------|---------------------------------------------|
+| Language code | Description                                 |
+|---------------|---------------------------------------------|
+| `art`         | Tweet contains emojis only                  |
+| `qam`         | Tweet contains mentions only                |
+| `qct`         | Tweet contains cashtags only                |
+| `qht`         | Tweet contains hashtags only                |
+| `qme`         | Tweet contains media only                   |
+| `qst`         | Tweet text is very short                    |
+| `und`         | Undefined (couldn’t determine the language) |
+| `zxx`         | Tweet contains media or twitter card only   |
+{: .table .table-striped .w-inherit }
+
+### Tweet indexing status
+
+After fetching new proposed community notes, the text of the tweets that the notes reference is not immediately searchable. In order to make it searchable, we need to fetch these tweets – a process that can take several hours. You can see the current status below.
+
+{% if site.data.meta.total_tweets %}
+  {% assign perc_fetched = site.data.meta.total_fetched | times: 100 | divided_by: site.data.meta.total_tweets %}
+{% else %}
+  {% assign perc_fetched = 0 %}
+{% endif %}
+
+<div class="progress my-2" style="max-width: 500px;" role="progressbar">
+  <div class="progress-bar text-bg-{% if perc_fetched == 100 %}success{% else %}warning{% endif %}" style="width: {{ perc_fetched }}%">{{ perc_fetched }}% ({{ site.data.meta.total_fetched }} / {{ site.data.meta.total_tweets }})</div>
+</div>
+
+{{ perc_fetched }}% of tweets are searchable.
+
+<script>
+  const dts = document.getElementsByClassName('dt');
+  for (var i = 0; i < dts.length; i++) {
+    var dt = dts[i];
+    dt.textContent = luxon.DateTime.fromISO(dt.textContent).toRelative();
+  }
+</script>
+
+### Why is the language unknown for some tweets?
+
+Until we’ve fetched a tweet, we don’t know its language. So ‘unknown language’ may mean we haven’t yet fetched that tweet. Once we’ve fetched it (in the next hour or so) we should know the tweet author, language and text.
+
+‘Unknown language’ may also mean the tweet has been deleted. In this case, we have no way of determining the tweet author, language or text.
diff --git a/data/.gitkeep b/data/.gitkeep
diff --git a/data/notes.json b/data/notes.json
Original file line number	Diff line number	Diff line change
		@@ -0,0 +1 @@
		{"scraped_at": "2024-05-30T04:20:28.465192+00:00", "most_recent": "2024-05-28T00:31:35+00:00", "total_notes": 10310, "total_fetched": 7538, "locked_until": "2024-05-30T04:08:11+00:00", "total_tweets": 8766}