Update on Local Storage notes

prometheus-exercises.ipynb

@@ -1289,7 +1289,27 @@
     "- kubeadm init\n",
     "- Copy the admin.conf to ~/.kube/config # Chown the file to user\n",
     "- kubectl taint nodes --all node-role.kubernetes.io/master- (# As this is single node, remove taint)\n",
-    "- Install the cluster network plugin (weavnet or calico)"
+    "- Install the cluster network plugin (weavnet or calico)\n",
+    "- Run an demo service\n",
+    "- See how the Service Discovery works\n",
+    "- use the scrape annotation to scrape a service\n",
+    "- See the default kubernetes component level metrics\n",
+    "- Use kube-state-metrics to see kubernetes object level metrics\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Local Storage\n",
+    "\n",
+    "Some internals on how the samples are compacted and persisted, some pointes\n",
+    "\n",
+    "- Retention of sample in memory - 2h\n",
+    "- Block time\n",
+    "- Compaction into larger blocks, every 2h\n",
+    "- Write-Ahead-Logging ( To persist the in-memory samples to avoid failure)\n",
+    "- Tombstones"
    ]
   },
   {
@@ -1298,7 +1318,7 @@
    "source": [
     "### Remote Storage - LTS\n",
     "\n",
-    "- Influx Db ( slow,)\n",
+    "- Influx Db (slow)\n",
     "- Thanos (Side Car, reuses the tsdb blocks and sends them to Storage(minio/s3..), Thanos Store(reads them and makes the series ready), Thanos Query)\n",
     "- Cortex "
    ]
@@ -1309,8 +1329,9 @@
    "source": [
     "### Monitoring and Debugging Prometheus\n",
     "\n",
-    "- Meta Prometheus\n",
+    "- Run HA Meta Prometheus\n",
     "- Use prometheus'es metrics and run alerting\n",
+    "- See talk on 'dead-mans-switch' for handling complete failure\n",
     "\n",
     "#### Debugging/Profiling\n",
     "API /debug/pprof/profile, /debug/pprof/heap, /debug/pprof/goroutine?debug=2\n",