Lessons learnt from exposing my server to the internet part 2
Posted on
Not too long ago, I had written about how I exposed my ssh port onto the internet (for use by my dad). I had also written about how I started experiencing attacks on said port and what I did to harden against it. This is the next (fingers crossed, last) step in my hardening of my server from the outside world.
What had happened was a determined script kiddie wanted in to my server. This person was trying to brute force, from seemingly infinite IP addresses. Using an IP lookup, they were coming from cloud providers such as digital ocean and alibaba cloud. The script kiddie was smart enough to figure out that I banned on every third failure, so they changed their attack to try twice before switching ips. This brute force attempt went for days. As far as I can tell they were not able to gain access to my system, after all I had disabled login passwords so without a trusted ssh key nothing was to be had. My ssshguard blacklist grew from <10 to almost 1000 ips. Since this was taking up precious CPU cycles and disk space (from logging all these attempts) I decided to move to more extreme measures.
I changed sshguard to blacklist after a single failed attempt and to remember blocked ips for 100x longer. This solution worked for a while, but then it ended up banning my own dad. My dad (as mentioned before) is not tech savvy. This means that he will (and did) mess up his password, or switch around his username. From a security POV it's damn near impossible to tell the difference between an unlucky user trying to login versus an attacker trying to gain access to your system. I resorted to whitelisting his IP for his specific user, and removed his IP from the sshguard blacklist (and iptables rules). This would work until a) his IP changed or b) he inevitably messes up his password again and ends up banned again. Neither is a perfect solution.
I had given some thought to setting up wireguard or similar VPN solution and having my dad use that, but remember a ssh key was too much for him. Then I came across tailscale. The beauty about tailscale is that they have a generous free tier (which also allows sharing machines to other free accounts). They also have incredibly simple to setup clients for every os. Within a 10 minute call I was able to get my dad setup and have him access my server. This means that I no longer need to have an ssh port exposed nor have sshguard running. Using tailscale also means that I can ease up on my draconian sshd configurations and allow my dad more login attempts and longer login grace period, all without having to worry about attackers taking advantage of it.
Another killer feature of tailscale is that you can write tests inside your access control configuration to ensure that no changes break any assumptions the tests test for. In my case I use them to ensure that my dad can access port 22 and nothing else on my server (I have a couple other ports that I use that I do not want him to have access to).
All in all, I am kind of sad that I had to end relying on such a solution, but malicious actors are why we can't have nice things. Also major props to tailscale for having an install process and a client so simple that my father can install and use it without constant help.