Imran's Blog
Stuff I feel like blogging about.

Reworking dad's site (again)

Posted on

I decided to rework my dad's site again. But first some history:

# Site version 1

  1. Dad uploads documents via ftp to hosting company.
  2. Files are now public and viewable.

This was his workflow for years. The problems started when he relocated to our home country and had to pay for renewal of service. The hosting company does not accept payment from our home country, and the conversion makes the cost too unreasonable. This is when my dad asked me to pay for him since I am still in North America. Since this is (and most probably will stay) static files, I thought why not take advantage of free hosting provided by forges for static pages. This led to site version 2.

# Site version 2

  1. Dad uploads documents to my local server via sftp.
  2. Something listens for file changes.
  3. That something then makes a git commit and pushes it.
  4. The forge of choice then publishes this to their static hosting site.

For the forge, I ended up going with GitLab. I already had a GitLab account I was using some other stuff. GitLab has a nice feature where you can give another account a developer role. This ensures the following about that account:

  1. Can not perform destructive actions on the repository.
  2. Can not change any GitLab Pages settings.
  3. Can push code.

This let me create a bot account that can whose credentials would come in handy for step 3 and GitLab CI would take care of 4. Also, in some crazy scenario, if someone was able to access that account's ssh keys or account, generating garbage commits is all they would be able to do.

My dad is unable to use (or understand) ssh keys, so his account would need to be able to login via password. I also did not want someone who got access to his account to be able to mess with the users home directory, where the ssh keys and git repository exists. The solution was a sftp chroot and bind mount. With step 2 remaining, I decided to create my own program for it. I covered this in a previous post.

This ended up with its own issues which led to site 2.1.

# Site version 2.1

This is like version 2 except instead of going over the public network step 1 is now done via tailscale (plain wireguard out of the question for similar reasons to why ssh keys aren't viable).

And voilà! I now have my dads work backed up in git while being invisible to him and it's cheaper than paying the hosting company from version 1. All is well except one big issue. I have broken my dads workflow and subsequently reduced his enjoyment (and frequency) of working on the site.

The biggest difference between version 1 and 2, is the feedback after uploading files. My dad works on his site by editing the html and then viewing the results AFTER uploading the files. This works well enough for him and he was happy with it. I ruined it with version 2. In version 2 there is a couple minutes delay as the changes make their way through git and GitLabs CI to for publishing. I am unsure if caching is in play here, but it would also add to the feedback time.

I should be able to do better (and I did! hence this post), which led to site version 3.

# Site version 3 (now with 100% more nix)

My requirements for site version 3 were:

  1. Immediate feedback from version 1
  2. All the good stuff from version 2

This led me to the following solution:

  1. Cheap VPS
  2. Caddy for serving files and SSL certificates
  3. sftp/bind mount/etc from version 2

The reasons I do not want to host this on my local server is that I have a residential connection and that connection's public IP changes from time to time. I didn't want to complicate this by throwing in dynamic DNS as well.

For the cheap VPS I went with Hetzner which offers a 2 core ARM virtual server with 4GB of ram, 40GB of disk space and 20TB of egress for € 3.3 a month.

I composed site version 2 out of a bunch of custom systemd jobs, a custom binary, a custom user account and some custom ssh configuration. The remaining thing that needs to change is to include caddy for hosting the files out of the sftp chroot. I did not want to manage a bunch of bespoke configuration files and settings, which led me to Nix and Guix.

I chose Nix for a couple reasons. Hetzner already provides Nix ISO's in their web UI. Nix builds upon systemd (which also allows it to infect other host distributions). I am a fan that Guix uses guile as opposed to its own bespoke language, but I am already spending enough innovation tokens on Nix to give up systemd too.

I won't bore you with what Nix/Guix is, but the nice thing is that I now have a file that can recreate the site version 3 from scratch on a new system. I am quite suprised by just how easy the process was. I hit one snag with my custom ssh configuration.

services.openssh = {
  enable = true;
  settings.PasswordAuthentication = false;
  settings.KbdInteractiveAuthentication = false;
  settings.PermitRootLogin = "no";
  extraConfig = ''
  Match User <dads account>
      ChrootDirectory <dads sftp jail directory>
      ForceCommand internal-sftp
      PasswordAuthentication yes
      MaxAuthTries 6
  Match all
  '';
};

By setting PasswordAuthentication to false, Nix disables unix authentication for PAM. This means that <dads account> will never be able to login with the password I set. I rectified it by adding this line to my configuration.nix:

# Allows the sftp stuff about to work
security.pam.services.sshd.unixAuth = lib.mkForce true;

# Thoughts on Nix/Guix

I like them.

Some time during setup, I managed to get to a wonky state where all environment variables were unset. This caused every nix command to fail saying something about an undefined nix path environment variable. I started panicking but then some googling said to set variable to the nix store, and the rollback command worked and I was back to a golden state.

The like hit when I wanted to view metrics on the VPS using prometheus/grafana. The diff to add that to my configuration:

diff --git a/configuration.nix b/configuration.nix
index 4dc89e6..7c33b0c 100644
--- a/configuration.nix
+++ b/configuration.nix
@@ -93,6 +93,7 @@
   systemd.services.tailscaled.serviceConfig = {
       Environment = [
          "TS_NO_LOGS_NO_SUPPORT=true"
+         "TS_PERMIT_CERT_UID=caddy" # let caddy get https certs
       ];
       LogNamespace = "shadow-realm";
   };
@@ -123,6 +124,17 @@
     '';
   };

+
+  services.prometheus = {
+    exporters = {
+      node = {
+        enable = true;
+        enabledCollectors = [ "systemd" ];
+        port = 9100;
+      };
+    };
+  };
+
   # Allows the sftp stuff about to work
   security.pam.services.sshd.unixAuth = lib.mkForce true;
   security.doas.enable = true;
@@ -227,6 +239,14 @@
       file_server
       root * <sftp jail>
     '';
+    virtualHosts."<tailscale host name>".extraConfig = ''
+      handle /metrics/node {
+        rewrite * /metrics
+        reverse_proxy localhost:${toString config.services.prometheus.exporters.node.port}
+      }
+    '';
   };

   # I don't want to install home manager for this

This small diff ensures the following:

  1. Node exported up and running
  2. Caddy and tailscale work together to ensure HTTPS on the private tailscale url
  3. Caddy makes node exporter metrics available under /metrics/node

I then had to go to my local server (running arch btw) and install prometheus and grafana. I had two options:

  1. Use containers (which I am starting to hate and deserve their own rant blog post)
  2. Use system packages

I opted for option 2, and made sure to copy the prometheus configuration to my git repository of rag tag files needed to recreate this server. Compared to using Nix it feels so archaic.

It's not all sunshines and rainbows though, there are things I am not a fan of:

  1. (Nix) Most tutorials are out of date
  2. (Nix) NixLang is ¯\_(ツ)_/¯ (could lua have worked?)
  3. (Guix) Harder stance on Free packages (but it's super easy to add your own channel/packages)

Those are incredibly minor compared to the value these projects provide. In the future I am hoping to get rid of containers/docker on my local server and replace it entirely with Guix, and moving my desktop to one of them.