From 05e8f4229ce64a7680cb597ce9912e31a8849caf Mon Sep 17 00:00:00 2001
From: edolstra <edolstra@users.noreply.github.com>
Date: Wed, 3 Jun 2020 13:02:48 +0000
Subject: [PATCH] Update flake.lock and blogs.xml [ci skip]

---
 blogs.xml | 573 +++++++++++++++++++++++++++++++++++++++++++++++++++---
 1 file changed, 549 insertions(+), 24 deletions(-)

diff --git a/blogs.xml b/blogs.xml
index 9e196c70c0..81f0aefc24 100644
--- a/blogs.xml
+++ b/blogs.xml
@@ -98,6 +98,555 @@ refreshing bevvy while you wait for blocks to sync.&lt;/p&gt;
 	<pubDate>Sat, 18 Apr 2020 00:00:00 +0000</pubDate>
 	<author>support@nixbuild.net (nixbuild.net)</author>
 </item>
+<item>
+	<title>Graham Christensen: Erase your darlings</title>
+	<guid isPermaLink="false">http://grahamc.com//blog/erase-your-darlings</guid>
+	<link>http://grahamc.com/blog/erase-your-darlings</link>
+	<description>&lt;p&gt;I erase my systems at every boot.&lt;/p&gt;
+
+&lt;p&gt;Over time, a system collects state on its root partition. This state
+lives in assorted directories like &lt;code class=&quot;highlighter-rouge&quot;&gt;/etc&lt;/code&gt; and &lt;code class=&quot;highlighter-rouge&quot;&gt;/var&lt;/code&gt;, and represents
+every under-documented or out-of-order step in bringing up the
+services.&lt;/p&gt;
+
+&lt;blockquote&gt;
+  &lt;p&gt;“Right, run &lt;code class=&quot;highlighter-rouge&quot;&gt;myapp-init&lt;/code&gt;.”&lt;/p&gt;
+&lt;/blockquote&gt;
+
+&lt;p&gt;These small, inconsequential “oh, oops” steps are the pieces that get
+lost and don’t appear in your runbooks.&lt;/p&gt;
+
+&lt;blockquote&gt;
+  &lt;p&gt;“Just download ca-certificates to … to fix …”&lt;/p&gt;
+&lt;/blockquote&gt;
+
+&lt;p&gt;Each of these quick fixes leaves you doomed to repeat history in three
+years when you’re finally doing that dreaded RHEL 7 to RHEL 8 upgrade.&lt;/p&gt;
+
+&lt;blockquote&gt;
+  &lt;p&gt;“Oh, &lt;code class=&quot;highlighter-rouge&quot;&gt;touch /etc/ipsec.secrets&lt;/code&gt; or the l2tp tunnel won’t work.”&lt;/p&gt;
+&lt;/blockquote&gt;
+
+&lt;h3 id=&quot;immutable-infrastructure-gets-us-so-close&quot;&gt;Immutable infrastructure gets us &lt;em&gt;so&lt;/em&gt; close&lt;/h3&gt;
+
+&lt;p&gt;Immutable infrastructure is a wonderfully effective method of
+eliminating so many of these forgotten steps. Leaning in to the pain
+by deleting and replacing your servers on a weekly or monthly basis
+means you are constantly testing and exercising your automation and
+runbooks.&lt;/p&gt;
+
+&lt;p&gt;The nugget here is the regular and indiscriminate removal of system
+state. Destroying the whole server doesn’t leave you much room to
+forget the little tweaks you made along the way.&lt;/p&gt;
+
+&lt;p&gt;These techniques work great when you meet two requirements:&lt;/p&gt;
+
+&lt;ul&gt;
+  &lt;li&gt;you can provision and destroy servers with an API call&lt;/li&gt;
+  &lt;li&gt;the servers aren’t inherently stateful&lt;/li&gt;
+&lt;/ul&gt;
+
+&lt;h4 id=&quot;long-running-servers&quot;&gt;Long running servers&lt;/h4&gt;
+
+&lt;p&gt;There are lots of cases in which immutable infrastructure &lt;em&gt;doesn’t&lt;/em&gt;
+work, and the dirty secret is &lt;strong&gt;those servers need good tools the
+most.&lt;/strong&gt;&lt;/p&gt;
+
+&lt;p&gt;Long-running servers cause long outages. Their runbooks are outdated
+and incomplete. They accrete tweaks and turn in to an ossified,
+brittle snowflake — except its arms are load-bearing.&lt;/p&gt;
+
+&lt;p&gt;Let’s bring the ideas of immutable infrastructure to these systems
+too. Whether this system is embedded in a stadium’s jumbotron, in a
+datacenter, or under your desk, we &lt;em&gt;can&lt;/em&gt; keep the state under control.&lt;/p&gt;
+
+&lt;h4 id=&quot;fhs-isnt-enough&quot;&gt;FHS isn’t enough&lt;/h4&gt;
+
+&lt;p&gt;The hard part about applying immutable techniques to long running
+servers is knowing exactly where your application state ends and the
+operating system, software, and configuration begin.&lt;/p&gt;
+
+&lt;p&gt;This is hard because legacy operating systems and the Filesystem
+Hierarchy Standard poorly separate these areas of concern. For
+example, &lt;code class=&quot;highlighter-rouge&quot;&gt;/var/lib&lt;/code&gt; is for state information, but how much of this do
+you actually care about tracking? What did you configure in &lt;code class=&quot;highlighter-rouge&quot;&gt;/etc&lt;/code&gt; on
+purpose?&lt;/p&gt;
+
+&lt;p&gt;The answer is probably not a lot.&lt;/p&gt;
+
+&lt;p&gt;You may not care, but all of this accumulation of junk is a tarpit.
+Everything becomes harder: replicating production, testing changes,
+undoing mistakes.&lt;/p&gt;
+
+&lt;h3 id=&quot;new-computer-smell&quot;&gt;New computer smell&lt;/h3&gt;
+
+&lt;p&gt;Getting a new computer is this moment of cleanliness. The keycaps
+don’t have oils on them, the screen is perfect, and the hard drive
+is fresh and unspoiled — for about an hour or so.&lt;/p&gt;
+
+&lt;p&gt;Let’s get back to that.&lt;/p&gt;
+
+&lt;h2 id=&quot;how-is-this-possible&quot;&gt;How is this possible?&lt;/h2&gt;
+
+&lt;p&gt;NixOS can boot with only two directories: &lt;code class=&quot;highlighter-rouge&quot;&gt;/boot&lt;/code&gt;, and &lt;code class=&quot;highlighter-rouge&quot;&gt;/nix&lt;/code&gt;.&lt;/p&gt;
+
+&lt;p&gt;&lt;code class=&quot;highlighter-rouge&quot;&gt;/nix&lt;/code&gt; contains read-only system configurations, which are specified
+by your &lt;code class=&quot;highlighter-rouge&quot;&gt;configuration.nix&lt;/code&gt; and are built and tracked as system
+generations. These never change. Once the files are created in &lt;code class=&quot;highlighter-rouge&quot;&gt;/nix&lt;/code&gt;,
+the only way to change the config’s contents is to build a new system
+configuration with the contents you want.&lt;/p&gt;
+
+&lt;p&gt;Any configuration or files created on the drive outside of &lt;code class=&quot;highlighter-rouge&quot;&gt;/nix&lt;/code&gt; is
+state and cruft. We can lose everything outside of &lt;code class=&quot;highlighter-rouge&quot;&gt;/nix&lt;/code&gt; and &lt;code class=&quot;highlighter-rouge&quot;&gt;/boot&lt;/code&gt;
+and have a healthy system. My technique is to explicitly opt in and
+&lt;em&gt;choose&lt;/em&gt; which state is important, and only keep that.&lt;/p&gt;
+
+&lt;p&gt;How this is possible comes down to the boot sequence.&lt;/p&gt;
+
+&lt;p&gt;For NixOS, the bootloader follows the same basic steps as a standard
+Linux distribution: the kernel starts with an initial ramdisk, and the
+initial ramdisk mounts the system disks.&lt;/p&gt;
+
+&lt;p&gt;And here is where the similarities end.&lt;/p&gt;
+
+&lt;h3 id=&quot;nixoss-early-startup&quot;&gt;NixOS’s early startup&lt;/h3&gt;
+
+&lt;p&gt;NixOS configures the bootloader to pass some extra information: a
+specific system configuration. This is the secret to NixOS’s
+bootloader rollbacks, and also the key to erasing our disk on each
+boot. The parameter is named &lt;code class=&quot;highlighter-rouge&quot;&gt;systemConfig&lt;/code&gt;.&lt;/p&gt;
+
+&lt;p&gt;On every startup the very early boot stage knows what the system’s
+configuration should be: the entire system configuration is stored in
+the read-only &lt;code class=&quot;highlighter-rouge&quot;&gt;/nix/store&lt;/code&gt;, and the directory passed through
+&lt;code class=&quot;highlighter-rouge&quot;&gt;systemConfig&lt;/code&gt; has a reference to the config. Early boot then
+manipulates &lt;code class=&quot;highlighter-rouge&quot;&gt;/etc&lt;/code&gt; and &lt;code class=&quot;highlighter-rouge&quot;&gt;/run&lt;/code&gt; to match the chosen setup. Usually this
+involves swapping out a few symlinks.&lt;/p&gt;
+
+&lt;p&gt;If &lt;code class=&quot;highlighter-rouge&quot;&gt;/etc&lt;/code&gt; simply doesn’t exist, however, early boot &lt;em&gt;creates&lt;/em&gt; &lt;code class=&quot;highlighter-rouge&quot;&gt;/etc&lt;/code&gt;
+and moves on like it were any other boot. It also &lt;em&gt;creates&lt;/em&gt; &lt;code class=&quot;highlighter-rouge&quot;&gt;/var&lt;/code&gt;,
+&lt;code class=&quot;highlighter-rouge&quot;&gt;/dev&lt;/code&gt;, &lt;code class=&quot;highlighter-rouge&quot;&gt;/home&lt;/code&gt;, and any other core directories that must be present.&lt;/p&gt;
+
+&lt;p&gt;Simply speaking, an empty &lt;code class=&quot;highlighter-rouge&quot;&gt;/&lt;/code&gt; is &lt;em&gt;not surprising&lt;/em&gt; to NixOS. In fact,
+the NixOS netboot, EC2, and installation media all start out this way.&lt;/p&gt;
+
+&lt;h2 id=&quot;opting-out&quot;&gt;Opting out&lt;/h2&gt;
+
+&lt;p&gt;Before we can opt in to saving data, we must opt out of saving data
+&lt;em&gt;by default&lt;/em&gt;. I do this by setting up my filesystem in a way that
+lets me easily and safely erase the unwanted data, while preserving
+the data I do want to keep.&lt;/p&gt;
+
+&lt;p&gt;My preferred method for this is using a ZFS dataset and rolling it
+back to a blank snapshot before it is mounted. A partition of any
+other filesystem would work just as well too, running &lt;code class=&quot;highlighter-rouge&quot;&gt;mkfs&lt;/code&gt; at boot,
+or something similar. If you have a lot of RAM, you could skip the
+erase step and make &lt;code class=&quot;highlighter-rouge&quot;&gt;/&lt;/code&gt; a tmpfs.&lt;/p&gt;
+
+&lt;h3 id=&quot;opting-out-with-zfs&quot;&gt;Opting out with ZFS&lt;/h3&gt;
+&lt;p&gt;When installing NixOS, I partition my disk with two partitions, one
+for the boot partition, and another for a ZFS pool. Then I create and
+mount a few datasets.&lt;/p&gt;
+
+&lt;p&gt;My root dataset:&lt;/p&gt;
+
+&lt;div class=&quot;highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;# zfs create -p -o mountpoint=legacy rpool/local/root
+&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;
+
+&lt;p&gt;Before I even mount it, I &lt;strong&gt;create a snapshot while it is totally
+blank&lt;/strong&gt;:&lt;/p&gt;
+
+&lt;div class=&quot;highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;# zfs snapshot rpool/local/root@blank
+&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;
+
+&lt;p&gt;And then mount it:&lt;/p&gt;
+
+&lt;div class=&quot;highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;# mount -t zfs rpool/local/root /mnt
+&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;
+
+&lt;p&gt;Then I mount the partition I created for the &lt;code class=&quot;highlighter-rouge&quot;&gt;/boot&lt;/code&gt;:&lt;/p&gt;
+
+&lt;div class=&quot;highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;# mkdir /mnt/boot
+# mount /dev/the-boot-partition /mnt/boot
+&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;
+
+&lt;p&gt;Create and mount a dataset for &lt;code class=&quot;highlighter-rouge&quot;&gt;/nix&lt;/code&gt;:&lt;/p&gt;
+
+&lt;div class=&quot;highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;# zfs create -p -o mountpoint=legacy rpool/local/nix
+# mkdir /mnt/nix
+# mount -t zfs rpool/local/nix /mnt/nix
+&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;
+
+&lt;p&gt;And a dataset for &lt;code class=&quot;highlighter-rouge&quot;&gt;/home&lt;/code&gt;:&lt;/p&gt;
+
+&lt;div class=&quot;highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;# zfs create -p -o mountpoint=legacy rpool/safe/home
+# mkdir /mnt/home
+# mount -t zfs rpool/safe/home /mnt/home
+&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;
+
+&lt;p&gt;And finally, a dataset explicitly for state I want to persist between
+boots:&lt;/p&gt;
+
+&lt;div class=&quot;highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;# zfs create -p -o mountpoint=legacy rpool/safe/persist
+# mkdir /mnt/persist
+# mount -t zfs rpool/safe/persist /mnt/persist
+&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;
+
+&lt;blockquote&gt;
+  &lt;p&gt;&lt;em&gt;Note:&lt;/em&gt; in my systems, datasets under &lt;code class=&quot;highlighter-rouge&quot;&gt;rpool/local&lt;/code&gt; are never backed
+up, and datasets under &lt;code class=&quot;highlighter-rouge&quot;&gt;rpool/safe&lt;/code&gt; are.&lt;/p&gt;
+&lt;/blockquote&gt;
+
+&lt;p&gt;And now safely erasing the root dataset on each boot is very easy:
+after devices are made available, roll back to the blank snapshot:&lt;/p&gt;
+
+&lt;div class=&quot;language-nix highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;p&quot;&gt;{&lt;/span&gt;
+  &lt;span class=&quot;nv&quot;&gt;boot&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nv&quot;&gt;initrd&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nv&quot;&gt;postDeviceCommands&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;nv&quot;&gt;lib&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nv&quot;&gt;mkAfter&lt;/span&gt; &lt;span class=&quot;s2&quot;&gt;''&lt;/span&gt;&lt;span class=&quot;err&quot;&gt;
+&lt;/span&gt;&lt;span class=&quot;s2&quot;&gt;    zfs rollback -r rpool/local/root@blank&lt;/span&gt;&lt;span class=&quot;err&quot;&gt;
+&lt;/span&gt;&lt;span class=&quot;s2&quot;&gt;  ''&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;;&lt;/span&gt;
+&lt;span class=&quot;p&quot;&gt;}&lt;/span&gt;
+&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;
+
+&lt;p&gt;I then finish the installation as normal. If all goes well, your
+next boot will start with an empty root partition but otherwise be
+configured exactly as you specified.&lt;/p&gt;
+
+&lt;h2 id=&quot;opting-in&quot;&gt;Opting in&lt;/h2&gt;
+
+&lt;p&gt;Now that I’m keeping no state, it is time to specify what I do want
+to keep. My choices here are different based on the role of the
+system: a laptop has different state than a server.&lt;/p&gt;
+
+&lt;p&gt;Here are some different pieces of state and how I preserve them. These
+examples largely use reconfiguration or symlinks, but using ZFS
+datasets and mount points would work too.&lt;/p&gt;
+
+&lt;h4 id=&quot;wireguard-private-keys&quot;&gt;Wireguard private keys&lt;/h4&gt;
+
+&lt;p&gt;Create a directory under &lt;code class=&quot;highlighter-rouge&quot;&gt;/persist&lt;/code&gt; for the key:&lt;/p&gt;
+
+&lt;div class=&quot;highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;# mkdir -p /persist/etc/wireguard/
+&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;
+
+&lt;p&gt;And use Nix’s wireguard module to generate the key there:&lt;/p&gt;
+
+&lt;div class=&quot;language-nix highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;p&quot;&gt;{&lt;/span&gt;
+  &lt;span class=&quot;nv&quot;&gt;networking&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nv&quot;&gt;wireguard&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nv&quot;&gt;interfaces&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nv&quot;&gt;wg0&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;{&lt;/span&gt;
+    &lt;span class=&quot;nv&quot;&gt;generatePrivateKeyFile&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;kc&quot;&gt;true&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;;&lt;/span&gt;
+    &lt;span class=&quot;nv&quot;&gt;privateKeyFile&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;s2&quot;&gt;&quot;/persist/etc/wireguard/wg0&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;;&lt;/span&gt;
+  &lt;span class=&quot;p&quot;&gt;};&lt;/span&gt;
+&lt;span class=&quot;p&quot;&gt;}&lt;/span&gt;
+&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;
+
+&lt;h4 id=&quot;networkmanager-connections&quot;&gt;NetworkManager connections&lt;/h4&gt;
+
+&lt;p&gt;Create a directory under &lt;code class=&quot;highlighter-rouge&quot;&gt;/persist&lt;/code&gt;, mirroring the &lt;code class=&quot;highlighter-rouge&quot;&gt;/etc&lt;/code&gt; structure:&lt;/p&gt;
+
+&lt;div class=&quot;highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;# mkdir -p /persist/etc/NetworkManager/system-connections
+&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;
+
+&lt;p&gt;And use Nix’s &lt;code class=&quot;highlighter-rouge&quot;&gt;etc&lt;/code&gt; module to set up the symlink:&lt;/p&gt;
+
+&lt;div class=&quot;language-nix highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;p&quot;&gt;{&lt;/span&gt;
+  &lt;span class=&quot;nv&quot;&gt;etc&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;s2&quot;&gt;&quot;NetworkManager/system-connections&quot;&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;{&lt;/span&gt;
+    &lt;span class=&quot;nv&quot;&gt;source&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;s2&quot;&gt;&quot;/persist/etc/NetworkManager/system-connections/&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;;&lt;/span&gt;
+  &lt;span class=&quot;p&quot;&gt;};&lt;/span&gt;
+&lt;span class=&quot;p&quot;&gt;}&lt;/span&gt;
+&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;
+
+&lt;h4 id=&quot;bluetooth-devices&quot;&gt;Bluetooth devices&lt;/h4&gt;
+
+&lt;p&gt;Create a directory under &lt;code class=&quot;highlighter-rouge&quot;&gt;/persist&lt;/code&gt;, mirroring the &lt;code class=&quot;highlighter-rouge&quot;&gt;/var&lt;/code&gt; structure:&lt;/p&gt;
+
+&lt;div class=&quot;highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;# mkdir -p /persist/var/lib/bluetooth
+&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;
+
+&lt;p&gt;And then use systemd’s tmpfiles.d rules to create a symlink from
+&lt;code class=&quot;highlighter-rouge&quot;&gt;/var/lib/bluetooth&lt;/code&gt; to my persisted directory:&lt;/p&gt;
+
+&lt;div class=&quot;language-nix highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;p&quot;&gt;{&lt;/span&gt;
+  &lt;span class=&quot;nv&quot;&gt;systemd&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nv&quot;&gt;tmpfiles&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nv&quot;&gt;rules&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;
+    &lt;span class=&quot;s2&quot;&gt;&quot;L /var/lib/bluetooth - - - - /persist/var/lib/bluetooth&quot;&lt;/span&gt;
+  &lt;span class=&quot;p&quot;&gt;];&lt;/span&gt;
+&lt;span class=&quot;p&quot;&gt;}&lt;/span&gt;
+&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;
+
+&lt;h4 id=&quot;ssh-host-keys&quot;&gt;SSH host keys&lt;/h4&gt;
+
+&lt;p&gt;Create a directory under &lt;code class=&quot;highlighter-rouge&quot;&gt;/persist&lt;/code&gt;, mirroring the &lt;code class=&quot;highlighter-rouge&quot;&gt;/etc&lt;/code&gt; structure:&lt;/p&gt;
+
+&lt;div class=&quot;highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;# mkdir -p /persist/etc/ssh
+&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;
+
+&lt;p&gt;And use Nix’s openssh module to create and use the keys in that
+directory:&lt;/p&gt;
+
+&lt;div class=&quot;language-nix highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;p&quot;&gt;{&lt;/span&gt;
+  &lt;span class=&quot;nv&quot;&gt;services&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nv&quot;&gt;openssh&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;{&lt;/span&gt;
+    &lt;span class=&quot;nv&quot;&gt;enable&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;kc&quot;&gt;true&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;;&lt;/span&gt;
+    &lt;span class=&quot;nv&quot;&gt;hostKeys&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;
+      &lt;span class=&quot;p&quot;&gt;{&lt;/span&gt;
+        &lt;span class=&quot;nv&quot;&gt;path&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;s2&quot;&gt;&quot;/persist/ssh/ssh_host_ed25519_key&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;;&lt;/span&gt;
+        &lt;span class=&quot;nv&quot;&gt;type&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;s2&quot;&gt;&quot;ed25519&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;;&lt;/span&gt;
+      &lt;span class=&quot;p&quot;&gt;}&lt;/span&gt;
+      &lt;span class=&quot;p&quot;&gt;{&lt;/span&gt;
+        &lt;span class=&quot;nv&quot;&gt;path&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;s2&quot;&gt;&quot;/persist/ssh/ssh_host_rsa_key&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;;&lt;/span&gt;
+        &lt;span class=&quot;nv&quot;&gt;type&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;s2&quot;&gt;&quot;rsa&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;;&lt;/span&gt;
+        &lt;span class=&quot;nv&quot;&gt;bits&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;4096&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;;&lt;/span&gt;
+      &lt;span class=&quot;p&quot;&gt;}&lt;/span&gt;
+    &lt;span class=&quot;p&quot;&gt;];&lt;/span&gt;
+  &lt;span class=&quot;p&quot;&gt;};&lt;/span&gt;
+&lt;span class=&quot;p&quot;&gt;}&lt;/span&gt;
+&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;
+
+&lt;h4 id=&quot;acme-certificates&quot;&gt;ACME certificates&lt;/h4&gt;
+
+&lt;p&gt;Create a directory under &lt;code class=&quot;highlighter-rouge&quot;&gt;/persist&lt;/code&gt;, mirroring the &lt;code class=&quot;highlighter-rouge&quot;&gt;/var&lt;/code&gt; structure:&lt;/p&gt;
+
+&lt;div class=&quot;highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;# mkdir -p /persist/var/lib/acme
+&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;
+
+&lt;p&gt;And then use systemd’s tmpfiles.d rules to create a symlink from
+&lt;code class=&quot;highlighter-rouge&quot;&gt;/var/lib/acme&lt;/code&gt; to my persisted directory:&lt;/p&gt;
+
+&lt;div class=&quot;language-nix highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;p&quot;&gt;{&lt;/span&gt;
+  &lt;span class=&quot;nv&quot;&gt;systemd&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nv&quot;&gt;tmpfiles&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nv&quot;&gt;rules&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;
+    &lt;span class=&quot;s2&quot;&gt;&quot;L /var/lib/acme - - - - /persist/var/lib/acme&quot;&lt;/span&gt;
+  &lt;span class=&quot;p&quot;&gt;];&lt;/span&gt;
+&lt;span class=&quot;p&quot;&gt;}&lt;/span&gt;
+&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;
+
+&lt;h3 id=&quot;answering-the-question-what-am-i-about-to-lose&quot;&gt;Answering the question “what am I about to lose?”&lt;/h3&gt;
+
+&lt;p&gt;I found this process a bit scary for the first few weeks: was I losing
+important data each reboot? No, I wasn’t.&lt;/p&gt;
+
+&lt;p&gt;If you’re worried and want to know what state you’ll lose on the next
+boot, you can list the files on your root filesystem and see if you’re
+missing something important:&lt;/p&gt;
+
+&lt;div class=&quot;highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;# tree -x /
+├── bin
+│   └── sh -&amp;gt; /nix/store/97zzcs494vn5k2yw-dash-0.5.10.2/bin/dash
+├── boot
+├── dev
+├── etc
+│   ├── asound.conf -&amp;gt; /etc/static/asound.conf
+... snip ...
+&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;
+
+&lt;p&gt;ZFS can give you a similar answer:&lt;/p&gt;
+
+&lt;div class=&quot;highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;# zfs diff rpool/local/root@blank
+M	/
++	/nix
++	/etc
++	/root
++	/var/lib/is-nix-channel-up-to-date
++	/etc/pki/fwupd
++	/etc/pki/fwupd-metadata
+... snip ...
+&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;
+
+&lt;h2 id=&quot;your-stateless-future&quot;&gt;Your stateless future&lt;/h2&gt;
+
+&lt;p&gt;You may bump in to new state you meant to be preserving. When I’m
+adding new services, I think about the state it is writing and whether
+I care about it or not. If I care, I find a way to redirect its state
+to &lt;code class=&quot;highlighter-rouge&quot;&gt;/persist&lt;/code&gt;.&lt;/p&gt;
+
+&lt;p&gt;Take care to reboot these machines on a somewhat regular basis. It
+will keep things agile, proving your system state is tracked
+correctly.&lt;/p&gt;
+
+&lt;p&gt;This technique has given me the “new computer smell” on every boot
+without the datacenter full of hardware, and even on systems that do
+carry important state. I have deployed this strategy to systems in the
+large and small: build farm servers, database servers, my NAS and home
+server, my raspberry pi garage door opener, and laptops.&lt;/p&gt;
+
+&lt;p&gt;NixOS enables powerful new deployment models in so many ways, allowing
+for systems of all shapes and sizes to be managed properly and
+consistently. I think this model of ephemeral roots is yet
+another example of this flexibility and power. I would like to see
+this partitioning scheme become a reference architecture and take us
+out of this eternal tarpit of legacy.&lt;/p&gt;</description>
+	<pubDate>Mon, 13 Apr 2020 00:00:00 +0000</pubDate>
+</item>
+<item>
+	<title>Graham Christensen: ZFS Datasets for NixOS</title>
+	<guid isPermaLink="false">http://grahamc.com//blog/nixos-on-zfs</guid>
+	<link>http://grahamc.com/blog/nixos-on-zfs</link>
+	<description>&lt;p&gt;The outdated and historical nature of the &lt;a href=&quot;https://grahamc.com/feed/fhs&quot;&gt;Filesystem Hierarchy
+Standard&lt;/a&gt; means traditional Linux distributions have to go to great
+lengths to separate “user data” from “system data.”&lt;/p&gt;
+
+&lt;p&gt;NixOS’s filesystem architecture does cleanly separate user data from
+system data, and has a much easier job to do.&lt;/p&gt;
+
+&lt;h3 id=&quot;traditional-linuxes&quot;&gt;Traditional Linuxes&lt;/h3&gt;
+
+&lt;p&gt;Because FHS mixes these two concerns across the entire hierarchy,
+splitting these concerns requires identifying every point across
+dozens of directories where the data is the system’s or the user’s.
+When adding ZFS to the mix, the installers typically have to create
+over a dozen datasets to accomplish this.&lt;/p&gt;
+
+&lt;p&gt;For example, Ubuntu’s upcoming ZFS support creates 16 datasets:&lt;/p&gt;
+
+&lt;pre&gt;&lt;code class=&quot;language-tree&quot;&gt;rpool/
+├── ROOT
+│   └── ubuntu_lwmk7c
+│       ├── log
+│       ├── mail
+│       ├── snap
+│       ├── spool
+│       ├── srv
+│       ├── usr
+│       │   └── local
+│       ├── var
+│       │   ├── games
+│       │   └── lib
+│       │       ├── AccountServices
+│       │       ├── apt
+│       │       ├── dpkg
+│       │       └── NetworkManager
+│       └── www
+└── USERDATA
+&lt;/code&gt;&lt;/pre&gt;
+
+&lt;p&gt;Going through the great pains of separating this data comes with
+significant advantages: a recursive snapshot at any point in the tree
+will create an atomic, point-in-time snapshot of every dataset below.&lt;/p&gt;
+
+&lt;p&gt;This means in order to create a consistent snapshot of the system
+data, an administrator would only need to take a recursive snapshot
+at &lt;code class=&quot;highlighter-rouge&quot;&gt;ROOT&lt;/code&gt;. The same is true for user data: take a recursive snapshot of
+&lt;code class=&quot;highlighter-rouge&quot;&gt;USERDATA&lt;/code&gt; and all user data is saved.&lt;/p&gt;
+
+&lt;h3 id=&quot;nixos&quot;&gt;NixOS&lt;/h3&gt;
+
+&lt;p&gt;Because Nix stores all of its build products in &lt;code class=&quot;highlighter-rouge&quot;&gt;/nix/store&lt;/code&gt;, NixOS
+doesn’t mingle these two concerns. NixOS’s runtime system, installed
+packages, and rollback targets are all stored in &lt;code class=&quot;highlighter-rouge&quot;&gt;/nix&lt;/code&gt;.&lt;/p&gt;
+
+&lt;p&gt;User data is not.&lt;/p&gt;
+
+&lt;p&gt;This removes the entire complicated tree of datasets to facilitate
+FHS, and leaves us with only a few needed datasets.&lt;/p&gt;
+
+&lt;h2 id=&quot;datasets&quot;&gt;Datasets&lt;/h2&gt;
+
+&lt;p&gt;Design for the atomic, recursive snapshots when laying out the
+datasets.&lt;/p&gt;
+
+&lt;p&gt;In particular, I don’t back up the &lt;code class=&quot;highlighter-rouge&quot;&gt;/nix&lt;/code&gt; directory. This entire
+directory can always be rebuilt later from the system’s
+&lt;code class=&quot;highlighter-rouge&quot;&gt;configuration.nix&lt;/code&gt;, and isn’t worth the space.&lt;/p&gt;
+
+&lt;p&gt;One way to model this might be splitting up the data into three
+top-level datasets:&lt;/p&gt;
+
+&lt;pre&gt;&lt;code class=&quot;language-tree&quot;&gt;tank/
+├── local
+│   └── nix
+├── system
+│   └── root
+└── user
+    └── home
+&lt;/code&gt;&lt;/pre&gt;
+
+&lt;p&gt;In &lt;code class=&quot;highlighter-rouge&quot;&gt;tank/local&lt;/code&gt;, I would store datasets that should almost never be
+snapshotted or backed up. &lt;code class=&quot;highlighter-rouge&quot;&gt;tank/system&lt;/code&gt; would store data that I would
+want periodic snapshots for. Most importantly, &lt;code class=&quot;highlighter-rouge&quot;&gt;tank/user&lt;/code&gt; would
+contain data I want regular snapshots and backups for, with a long
+retention policy.&lt;/p&gt;
+
+&lt;p&gt;From here, you could add a ZFS dataset per user:&lt;/p&gt;
+
+&lt;pre&gt;&lt;code class=&quot;language-tree&quot;&gt;tank/
+├── local
+│   └── nix
+├── system
+│   └── root
+└── user
+    └── home
+        ├── grahamc
+        └── gustav
+&lt;/code&gt;&lt;/pre&gt;
+
+&lt;p&gt;Or a separate dataset for &lt;code class=&quot;highlighter-rouge&quot;&gt;/var&lt;/code&gt;:&lt;/p&gt;
+
+&lt;pre&gt;&lt;code class=&quot;language-tree&quot;&gt;tank/
+├── local
+│   └── nix
+├── system
+│   ├── var
+│   └── root
+└── user
+&lt;/code&gt;&lt;/pre&gt;
+
+&lt;p&gt;Importantly, this gives you three buckets for independent and
+regular snapshots.&lt;/p&gt;
+
+&lt;p&gt;The important part is having &lt;code class=&quot;highlighter-rouge&quot;&gt;/nix&lt;/code&gt; under its own top-level dataset.
+This makes it a “cousin” to the data you &lt;em&gt;do&lt;/em&gt; want backup coverage on,
+making it easier to take deep, recursive snapshots atomically.&lt;/p&gt;
+
+&lt;h2 id=&quot;properties&quot;&gt;Properties&lt;/h2&gt;
+
+&lt;ul&gt;
+  &lt;li&gt;Enable compression with &lt;code class=&quot;highlighter-rouge&quot;&gt;compression=on&lt;/code&gt;. Specifying &lt;code class=&quot;highlighter-rouge&quot;&gt;on&lt;/code&gt; instead of
+&lt;code class=&quot;highlighter-rouge&quot;&gt;lz4&lt;/code&gt; or another specific algorithm will always pick the best
+available compression algorithm.&lt;/li&gt;
+  &lt;li&gt;The dataset containing journald’s logs (where &lt;code class=&quot;highlighter-rouge&quot;&gt;/var&lt;/code&gt; lives) should
+have &lt;code class=&quot;highlighter-rouge&quot;&gt;xattr=sa&lt;/code&gt; and &lt;code class=&quot;highlighter-rouge&quot;&gt;acltype=posixacl&lt;/code&gt; set to allow regular users to
+read their journal.&lt;/li&gt;
+  &lt;li&gt;Nix doesn’t use &lt;code class=&quot;highlighter-rouge&quot;&gt;atime&lt;/code&gt;, so &lt;code class=&quot;highlighter-rouge&quot;&gt;atime=off&lt;/code&gt; on the &lt;code class=&quot;highlighter-rouge&quot;&gt;/nix&lt;/code&gt; dataset is
+fine.&lt;/li&gt;
+  &lt;li&gt;NixOS requires (as of 2020-04-11) &lt;code class=&quot;highlighter-rouge&quot;&gt;mountpoint=legacy&lt;/code&gt; for all
+datasets. NixOS does not yet have tooling to require implicitly
+created ZFS mounts to settle before booting, and &lt;code class=&quot;highlighter-rouge&quot;&gt;mountpoint=legacy&lt;/code&gt;
+plus explicit mount points in &lt;code class=&quot;highlighter-rouge&quot;&gt;hardware-configuration.nix&lt;/code&gt; will
+ensure all your datasets are mounted at the right time.&lt;/li&gt;
+&lt;/ul&gt;
+
+&lt;p&gt;I don’t know how to pick &lt;code class=&quot;highlighter-rouge&quot;&gt;ashift&lt;/code&gt;, and usually just allow ZFS to guess
+on my behalf.&lt;/p&gt;
+
+&lt;h2 id=&quot;partitioning&quot;&gt;Partitioning&lt;/h2&gt;
+
+&lt;p&gt;I only create two partitions:&lt;/p&gt;
+
+&lt;ol&gt;
+  &lt;li&gt;&lt;code class=&quot;highlighter-rouge&quot;&gt;/boot&lt;/code&gt; formatted &lt;code class=&quot;highlighter-rouge&quot;&gt;vfat&lt;/code&gt; for EFI, or &lt;code class=&quot;highlighter-rouge&quot;&gt;ext4&lt;/code&gt; for BIOS&lt;/li&gt;
+  &lt;li&gt;The ZFS dataset partition.&lt;/li&gt;
+&lt;/ol&gt;
+
+&lt;p&gt;There are spooky articles saying only give ZFS entire disks. The
+truth is, you shouldn’t split a disk into two active partitions.
+Splitting the disk this way is just fine, since &lt;code class=&quot;highlighter-rouge&quot;&gt;/boot&lt;/code&gt; is rarely
+read or written.&lt;/p&gt;
+
+&lt;blockquote&gt;
+  &lt;p&gt;&lt;em&gt;Note:&lt;/em&gt; If you do partition the disk, make sure you set the disk’s
+scheduler to &lt;code class=&quot;highlighter-rouge&quot;&gt;none&lt;/code&gt;. ZFS takes this step automatically if it does
+control the entire disk.&lt;/p&gt;
+
+  &lt;p&gt;On NixOS, you an set your scheduler to &lt;code class=&quot;highlighter-rouge&quot;&gt;none&lt;/code&gt; via:&lt;/p&gt;
+
+  &lt;div class=&quot;language-nix highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;p&quot;&gt;{&lt;/span&gt; &lt;span class=&quot;nv&quot;&gt;boot&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nv&quot;&gt;kernelParams&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;[&lt;/span&gt; &lt;span class=&quot;s2&quot;&gt;&quot;elevator=none&quot;&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;];&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;}&lt;/span&gt;
+&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;  &lt;/div&gt;
+&lt;/blockquote&gt;
+
+&lt;h1 id=&quot;clean-isolation&quot;&gt;Clean isolation&lt;/h1&gt;
+
+&lt;p&gt;NixOS’s clean separation of concerns reduces the amount of complexity
+we need to track when considering and planning our datasets. This
+gives us flexibility later, and enables some superpowers like erasing
+my computer on every boot, which I’ll write about on Monday.&lt;/p&gt;</description>
+	<pubDate>Sat, 11 Apr 2020 00:00:00 +0000</pubDate>
+</item>
 <item>
 	<title>nixbuild.net: New nixbuild.net Resources</title>
 	<guid isPermaLink="true">https://blog.nixbuild.net/posts/2020-03-27-nixbuild-net-beta.html</guid>
@@ -607,30 +1156,6 @@ As of today, all binary caches are served by CloudFlare CDN.</description>
 Current State In order to show how it changes the handling of parameters to derivation, the first example will show the current state with __structuredAttrs set to false and the stdenv.mkDerivation wrapper around derivation. All parameters are passed to the builder as environment variables, canonicalised by Nix in imitation of shell script conventions:</description>
 	<pubDate>Mon, 20 Jan 2020 12:00:00 +0000</pubDate>
 </item>
-<item>
-	<title>Hercules Labs: Hercules CI &amp; Cachix split up</title>
-	<guid isPermaLink="true">https://blog.hercules-ci.com/2020/01/14/hercules-ci-cachix-split-up/</guid>
-	<link>https://blog.hercules-ci.com/2020/01/14/hercules-ci-cachix-split-up/</link>
-	<description>&lt;p&gt;After careful consideration of how to balance between the two products, we’ve decided to split up. Each of the two products will be a separate entity:&lt;/p&gt;
-
-&lt;ul&gt;
-  &lt;li&gt;Hercules CI becomes part of Robert Hensing’s Ensius B.V.&lt;/li&gt;
-  &lt;li&gt;Cachix becomes part of Domen Kožar’s Enlambda OÜ&lt;/li&gt;
-&lt;/ul&gt;
-
-&lt;p&gt;For customers there will be no changes, except for the point of contact in support requests.&lt;/p&gt;
-
-&lt;p&gt;Domen &amp;amp; Robert&lt;/p&gt;</description>
-	<pubDate>Tue, 14 Jan 2020 00:00:00 +0000</pubDate>
-</item>
-<item>
-	<title>Mayflower: Windows-on-NixOS, part 1: Migrating bare-metal to a VM</title>
-	<guid isPermaLink="true">https://nixos.mayflower.consulting/blog/2019/11/27/windows-vm-storage/</guid>
-	<link>https://nixos.mayflower.consulting/blog/2019/11/27/windows-vm-storage/</link>
-	<description>This is part 1 of a series of blog posts explaining how we took an existing Windows installation on hardware and moved it into a VM running on top of NixOS.
-Background We have a decently-equipped desktop PC sitting in our office, which is designated for data experiments using TensorFlow and such. During off-hours, it’s also used for games, and for that purpose it has Windows installed on it. We decided to try moving Windows into a VM within NixOS so that we could run both operating systems in parallel.</description>
-	<pubDate>Wed, 27 Nov 2019 06:00:00 +0000</pubDate>
-</item>
 
 </channel>
 </rss>