Replace charged language "whitelist/blacklist"

docs/config-carbon.rst

@@ -41,7 +41,7 @@ A given rule is made up of 3 lines:
 * A regex, specified after "pattern="
 * A retention rate line, specified after "retentions="
 
-The retentions line can specify multiple retentions. Each retention of ``frequency:history`` is separated by a comma. 
+The retentions line can specify multiple retentions. Each retention of ``frequency:history`` is separated by a comma.
 
 Frequencies and histories are specified using the following suffixes:
 
@@ -60,7 +60,7 @@ Here's a simple, single retention example:
  pattern = garbageCollections$
  retentions = 10s:14d
 
-The name ``[garbage_collection]`` is mainly for documentation purposes, and will show up in ``creates.log`` when metrics matching this section are created. 
+The name ``[garbage_collection]`` is mainly for documentation purposes, and will show up in ``creates.log`` when metrics matching this section are created.
 
 The regular expression pattern will match any metric that ends with ``garbageCollections``. For example, ``com.acmeCorp.instance01.jvm.memory.garbageCollections`` would match, but ``com.acmeCorp.instance01.jvm.memory.garbageCollections.full`` would not.
 
@@ -80,7 +80,7 @@ Additionally, this example uses multiple retentions. The general rule is to spec
 
 By using multiple retentions, you can store long histories of metrics while saving on disk space and I/O. Because whisper averages (by default) as it downsamples, one is able to determine totals of metrics by reversing the averaging process later on down the road.
 
-Example: You store the number of sales per minute for 1 year, and the sales per hour for 5 years after that.  You need to know the total sales for January 1st of the year before.  You can query whisper for the raw data, and you'll get 24 datapoints, one for each hour.  They will most likely be floating point numbers.  You can take each datapoint, multiply by 60 (the ratio of high-precision to low-precision datapoints) and still get the total sales per hour.  
+Example: You store the number of sales per minute for 1 year, and the sales per hour for 5 years after that.  You need to know the total sales for January 1st of the year before.  You can query whisper for the raw data, and you'll get 24 datapoints, one for each hour.  They will most likely be floating point numbers.  You can take each datapoint, multiply by 60 (the ratio of high-precision to low-precision datapoints) and still get the total sales per hour.
 
 
 Additionally, whisper supports a legacy retention specification for backwards compatibility reasons - ``seconds-per-datapoint:count-of-datapoints``
@@ -140,7 +140,7 @@ You must define at least one section as the default.
 
 aggregation-rules.conf
 ----------------------
-Aggregation rules allow you to add several metrics together as they come in, reducing the need to sum() many metrics in every URL. Note that unlike some other config files, any time this file is modified it will take effect automatically. This requires the carbon-aggregator service to be running. 
+Aggregation rules allow you to add several metrics together as they come in, reducing the need to sum() many metrics in every URL. Note that unlike some other config files, any time this file is modified it will take effect automatically. This requires the carbon-aggregator service to be running.
 
 The form of each line in this file should be as follows:
 
@@ -246,13 +246,28 @@ For example:
 These rules would strip off a suffix of _sum or _avg from any metric names after
 aggregation.
 
+Metric Filters: allowed and blocked Metrics
+-------------------------------------------
+
+The metric filter functionality allows any of the Carbon daemons to only accept
+metrics that are explicitly allowed and/or to reject rejected metrics. The
+functionality can be enabled in carbon.conf with the ``USE_METRIC_FILTERS``
+flag. This can be useful when too many metrics are being sent to a Graphite
+instance or when there are metric senders sending useless or invalid metrics.
+
+``GRAPHITE_CONF_DIR`` is searched for ``allowed_metrics.conf`` and
+``blocked_metrics.conf``.  Each file contains one regular expression per line to
+match against metric values.  If the allowed_metrics configuration is missing or
+empty, all metrics will be passed through by default.
+
 whitelist and blacklist
 -----------------------
-The whitelist functionality allows any of the carbon daemons to only accept metrics that are explicitly
-whitelisted and/or to reject blacklisted metrics. The functionality can be enabled in carbon.conf with
-the ``USE_WHITELIST`` flag. This can be useful when too many metrics are being sent to a Graphite
-instance or when there are metric senders sending useless or invalid metrics.
+The whitelist/blacklist functionality has been renamed to 'allowed' and
+'blocked' so as to use less ambiguous language, and remove possible connotations
+associated with those terms.
+
+The capabilities have been renamed as of this point, but the existing ``whitelist.conf``
+and ``blacklist.conf`` will still be functional for the time being.
 
-``GRAPHITE_CONF_DIR`` is searched for ``whitelist.conf`` and ``blacklist.conf``. Each file contains one regular
-expressions per line to match against metric values. If the whitelist configuration is missing or empty,
-all metrics will be passed through by default.
+Additionally, the ``USE_WHITELIST`` flag in carbon.conf will still be respected,
+but treated as ``USE_METRIC_FILTERS`` until it is deprecated.

docs/config-local-settings.rst

@@ -331,7 +331,7 @@ REMOTE_STORE_FIND_TIMEOUT
 REMOTE_STORE_RETRY_DELAY
   `Default: 60`
 
-  Time in seconds to blacklist a webapp after a timed-out request.
+  Time in seconds to block/filter a webapp after a timed-out request.
 
 REMOTE_FIND_CACHE_DURATION
   `Default: 300`

setup.py

@@ -88,7 +88,7 @@
         'graphite.render',
         'graphite.url_shortener',
         'graphite.version',
-        'graphite.whitelist',
+        'graphite.metric_filters',
       ],
       package_data={'graphite' :
         ['templates/*', 'local_settings.py.example']},

tox.ini

@@ -4,7 +4,7 @@ envlist =
 	py27-django16-pyparsing1, lint, docs
 
 [testenv]
-whitelist_externals =
+whitelist_externals =		# Unrelated to graphite project deprecation of white/black
 	mkdir
 setenv =
 	DJANGO_SETTINGS_MODULE=tests.settings

webapp/content/js/composer_widgets.js

@@ -1029,20 +1029,20 @@ var GraphDataWindow = {
 
   addWlSelected: function (item, e) {
     Ext.Ajax.request({
-      url: document.body.dataset.baseUrl + "whitelist/add",
+      url: document.body.dataset.baseUrl + "metric_filters/add",
       method: "POST",
-      success: function () { Ext.Msg.alert("Result", "Successfully added metrics to whitelist."); },
-      failure: function () { Ext.Msg.alert("Result", "Failed to add metrics to whitelist.");   },
+      success: function () { Ext.Msg.alert("Result", "Successfully added metrics to filter."); },
+      failure: function () { Ext.Msg.alert("Result", "Failed to add metrics to filter.");   },
       params: {metrics: this.getSelectedTargets().join("\n") }
     });
   },
 
   removeWlSelected: function (item, e) {
     Ext.Ajax.request({
-      url: document.body.dataset.baseUrl + "whitelist/remove",
+      url: document.body.dataset.baseUrl + "metric_filters/remove",
       method: "POST",
-      success: function () { Ext.Msg.alert("Result", "Successfully removed metrics from whitelist."); },
-      failure: function () { Ext.Msg.alert("Result", "Failed to remove metrics from whitelist.");   },
+      success: function () { Ext.Msg.alert("Result", "Successfully removed metrics from filter."); },
+      failure: function () { Ext.Msg.alert("Result", "Failed to remove metrics from filter.");   },
       params: {metrics: this.getSelectedTargets().join("\n") }
     });
   },

webapp/graphite/app_settings.py

@@ -59,7 +59,7 @@
   'graphite.composer',
   'graphite.account',
   'graphite.dashboard',
-  'graphite.whitelist',
+  'graphite.metric_filters',
   'graphite.events',
   'graphite.url_shortener',
   'django.contrib.auth',

webapp/graphite/local_settings.py.example

@@ -284,7 +284,7 @@
 #
 #CARBONLINK_HOSTS = ["127.0.0.1:7002:a", "127.0.0.1:7102:b", "127.0.0.1:7202:c"]
 #CARBONLINK_TIMEOUT = 1.0
-#CARBONLINK_RETRY_DELAY = 15 # Seconds to blacklist a failed remote server
+#CARBONLINK_RETRY_DELAY = 15 # Seconds to block a failed remote server
 
 # A "keyfunc" is a user-defined python function that is given a metric name
 # and returns a string that should be used when hashing the metric name.

webapp/graphite/whitelist/urls.py → webapp/graphite/metric_filters/urls.py

@@ -17,7 +17,7 @@
 
 urlpatterns = patterns(
     '',
-    url('add', views.add, name='whitelist_add'),
-    url('remove', views.remove, name='whitelist_remove'),
-    url('', views.show, name='whitelist_show'),
+    url('add', views.add, name='metric_filters_add'),
+    url('remove', views.remove, name='metric_filters_remove'),
+    url('', views.show, name='metric_filters_show'),
 )

webapp/graphite/whitelist/views.py → webapp/graphite/metric_filters/views.py

@@ -23,38 +23,38 @@
 
 def add(request):
   metrics = set( request.POST['metrics'].split() )
-  whitelist = load_whitelist()
-  new_whitelist = whitelist | metrics
-  save_whitelist(new_whitelist)
+  allowed_metrics = load_allowed_metrics()
+  new_allowed_metrics = allowed_metrics | metrics
+  save_allowed_metrics(new_allowed_metrics)
   return HttpResponse(content_type="text/plain", content="OK")
 
 def remove(request):
   metrics = set( request.POST['metrics'].split() )
-  whitelist = load_whitelist()
-  new_whitelist = whitelist - metrics
-  save_whitelist(new_whitelist)
+  allowed_metrics = load_allowed_metrics()
+  new_allowed_metrics = allowed_metrics - metrics
+  save_allowed_metrics(new_allowed_metrics)
   return HttpResponse(content_type="text/plain", content="OK")
 
 def show(request):
-  whitelist = load_whitelist()
-  members = '\n'.join( sorted(whitelist) )
+  allowed_metrics = load_allowed_metrics()
+  members = '\n'.join( sorted(allowed_metrics) )
   return HttpResponse(content_type="text/plain", content=members)
 
-def load_whitelist():
-  buffer = open(settings.WHITELIST_FILE, 'rb').read()
-  whitelist = unpickle.loads(buffer)
-  return whitelist
+def load_allowed_metrics():
+  buffer = open(settings.METRIC_FILTERS_FILE, 'rb').read()
+  allowed_metrics = unpickle.loads(buffer)
+  return allowed_metrics
 
-def save_whitelist(whitelist):
-  serialized = pickle.dumps(whitelist, protocol=-1) #do this instead of dump() to raise potential exceptions before open()
-  tmpfile = '%s-%d' % (settings.WHITELIST_FILE, randint(0, 100000))
+def save_allowed_metrics(allowed_metrics):
+  serialized = pickle.dumps(allowed_metrics, protocol=-1) #do this instead of dump() to raise potential exceptions before open()
+  tmpfile = '%s-%d' % (settings.METRIC_FILTERS_FILE, randint(0, 100000))
   try:
     fh = open(tmpfile, 'wb')
     fh.write(serialized)
     fh.close()
-    if os.path.exists(settings.WHITELIST_FILE):
-      os.unlink(settings.WHITELIST_FILE)
-    os.rename(tmpfile, settings.WHITELIST_FILE)
+    if os.path.exists(settings.METRIC_FILTERS_FILE):
+      os.unlink(settings.METRIC_FILTERS_FILE)
+    os.rename(tmpfile, settings.METRIC_FILTERS_FILE)
   finally:
     if os.path.exists(tmpfile):
       os.unlink(tmpfile)

webapp/graphite/settings.py

@@ -41,7 +41,7 @@
 DASHBOARD_CONF = ''
 GRAPHTEMPLATES_CONF = ''
 STORAGE_DIR = ''
-WHITELIST_FILE = ''
+METRIC_FILTERS_FILE = ''
 INDEX_FILE = ''
 LOG_DIR = ''
 CERES_DIR = ''
@@ -167,8 +167,8 @@
 
 if not STORAGE_DIR:
   STORAGE_DIR = os.environ.get('GRAPHITE_STORAGE_DIR', join(GRAPHITE_ROOT, 'storage'))
-if not WHITELIST_FILE:
-  WHITELIST_FILE = join(STORAGE_DIR, 'lists', 'whitelist')
+if not METRIC_FILTERS_FILE:
+  METRIC_FILTERS_FILE = join(STORAGE_DIR, 'lists', 'allowed_metrics')
 if not INDEX_FILE:
   INDEX_FILE = join(STORAGE_DIR, 'index')
 if not LOG_DIR:

webapp/graphite/urls.py

@@ -35,7 +35,7 @@
     ('^browser/?', include('graphite.browser.urls')),
     ('^account/', include('graphite.account.urls')),
     ('^dashboard/?', include('graphite.dashboard.urls')),
-    ('^whitelist/?', include('graphite.whitelist.urls')),
+    ('^metric_filters/?', include('graphite.metric_filters.urls')),
     ('^version/', include('graphite.version.urls')),
     ('^events/', include('graphite.events.urls')),
     url('^s/(?P<path>.*)',

webapp/tests/test_metric_filters.py

@@ -0,0 +1,115 @@
+import errno
+import mock
+import os
+import pickle
+
+from . import DATA_DIR
+
+from django.conf import settings
+from django.core.urlresolvers import reverse
+from django.test import TestCase
+
+from graphite.metric_filters.views import load_allowed_metrics, save_allowed_metrics
+
+class AllowedMetricsTester(TestCase):
+    settings.METRIC_FILTERS_FILE = os.path.join(DATA_DIR, 'lists/allowed_metrics')
+
+    def wipe_metric_filters(self):
+        try:
+            os.remove(settings.METRIC_FILTERS_FILE)
+        except OSError:
+            pass
+
+    def create_metric_filters(self):
+        try:
+            os.makedirs(settings.METRIC_FILTERS_FILE.replace('allowed_metrics', ''))
+        except OSError:
+            pass
+        fh = open(settings.METRIC_FILTERS_FILE, 'wb')
+        pickle.dump({'a.b.c.d', 'e.f.g.h'}, fh)
+        fh.close()
+
+    def test_metric_filters_show_no_metric_filters(self):
+        url = reverse('metric_filters_show')
+        with self.assertRaises(IOError):
+          response = self.client.get(url)
+
+    def test_metric_filters_show(self):
+        url = reverse('metric_filters_show')
+        self.create_metric_filters()
+        self.addCleanup(self.wipe_metric_filters)
+        response = self.client.get(url)
+        self.assertEqual(response.status_code, 200)
+        self.assertEqual(response.content, "a.b.c.d\ne.f.g.h")
+
+    def test_metric_filters_add(self):
+        self.create_metric_filters()
+        self.addCleanup(self.wipe_metric_filters)
+
+        url = reverse('metric_filters_add')
+        response = self.client.post(url, {'metrics': ['i.j.k.l']})
+        self.assertEqual(response.status_code, 200)
+        self.assertEqual(response.content, "OK")
+
+        url = reverse('metric_filters_show')
+        response = self.client.get(url)
+        self.assertEqual(response.status_code, 200)
+        self.assertEqual(response.content, "a.b.c.d\ne.f.g.h\ni.j.k.l")
+
+    def test_metric_filters_add_existing(self):
+        self.create_metric_filters()
+        self.addCleanup(self.wipe_metric_filters)
+
+        url = reverse('metric_filters_add')
+        response = self.client.post(url, {'metrics': ['a.b.c.d']})
+        self.assertEqual(response.status_code, 200)
+        self.assertEqual(response.content, "OK")
+
+        url = reverse('metric_filters_show')
+        response = self.client.get(url)
+        self.assertEqual(response.status_code, 200)
+        self.assertEqual(response.content, "a.b.c.d\ne.f.g.h")
+
+    def test_metric_filters_remove(self):
+        self.create_metric_filters()
+        self.addCleanup(self.wipe_metric_filters)
+
+        url = reverse('metric_filters_remove')
+        response = self.client.post(url, {'metrics': ['a.b.c.d']})
+        self.assertEqual(response.status_code, 200)
+        self.assertEqual(response.content, "OK")
+
+        url = reverse('metric_filters_show')
+        response = self.client.get(url)
+        self.assertEqual(response.status_code, 200)
+        self.assertEqual(response.content, "e.f.g.h")
+
+    def test_metric_filters_remove_missing(self):
+        self.create_metric_filters()
+        self.addCleanup(self.wipe_metric_filters)
+
+        url = reverse('metric_filters_remove')
+        response = self.client.post(url, {'metrics': ['i.j.k.l']})
+        self.assertEqual(response.status_code, 200)
+        self.assertEqual(response.content, "OK")
+
+        url = reverse('metric_filters_show')
+        response = self.client.get(url)
+        self.assertEqual(response.status_code, 200)
+        self.assertEqual(response.content, "a.b.c.d\ne.f.g.h")
+
+    def test_save_metric_filters(self):
+        try:
+            os.makedirs(settings.METRIC_FILTERS_FILE.replace('allowed_metrics', ''))
+        except OSError:
+            pass
+        self.addCleanup(self.wipe_metric_filters)
+        self.assertEqual(save_allowed_metrics({'a.b.c.d','e.f.g.h'}), None)
+        self.assertEqual(load_allowed_metrics(), {'a.b.c.d','e.f.g.h'})
+
+    @mock.patch('os.rename')
+    def test_save_metric_filters_rename_failure(self, rename):
+        self.addCleanup(self.wipe_metric_filters)
+        rename.side_effect = OSError(errno.EPERM, 'Operation not permitted')
+        with self.assertRaises(OSError):
+            save_allowed_metrics({'a.b.c.d','e.f.g.h'})