Lessons learnt from exposing my server to the internet
Posted on
My previous post was about how I setup a SFTP server for my dad. This post is going to be about all the things I learned about exposing your server to the internet.
# Unable to login
About a week after I exposed my server via SFTP to the world, I wanted to log in to my server to do some maintenance. Much to my shock, my server was unresponsive to my regular ssh commands.
This is when the panic started setting in. My server lives in my home close to my desktop computer. The server appeared to be still on, the lights were still on and the fans were still spinning. I resisted the urge to reboot it and try to ssh again, as I wanted to know why I could not ssh in.
I temporarily repurposed my second monitor on my desktop and plugged in an extra keyboard that I had. The standard login prompt greeted me, which meant the server was not dead and I was able to login.
I checked the status of the ssh process with a systemctl status sshd and (un)surprisingly it's dead.
The next step was to figure out why it's dead.
This started with looking at the logs via journalctl -b -t sshd
Loads of similar lines
---
Oct 08 11:04:33 server sshd[245607]: Unable to negotiate with 86.111.187.162 port 36589: no matching key exchange method found. Their offer: diffie-hellman-group14-sha1,diffie>
Oct 08 11:04:33 server sshd[245610]: Unable to negotiate with 86.111.187.162 port 36596: no matching key exchange method found. Their offer: diffie-hellman-group14-sha1,diffie>
Oct 08 11:04:33 server sshd[245618]: Unable to negotiate with 86.111.187.162 port 36605: no matching key exchange method found. Their offer: diffie-hellman-group14-sha1,diffie>
Oct 08 11:04:33 server sshd[245622]: Unable to negotiate with 86.111.187.162 port 36616: no matching key exchange method found. Their offer: diffie-hellman-group14-sha1,diffie>
Oct 08 11:04:34 server sshd[245628]: Unable to negotiate with 86.111.187.162 port 36623: no matching key exchange method found. Their offer: diffie-hellman-group14-sha1,diffie>
Oct 08 11:04:34 server sshd[245632]: Unable to negotiate with 86.111.187.162 port 36634: no matching key exchange method found. Their offer: diffie-hellman-group14-sha1,diffie>
Oct 08 11:04:34 server sshd[245636]: Unable to negotiate with 86.111.187.162 port 36647: no matching key exchange method found. Their offer: diffie-hellman-group14-sha1,diffie>
Oct 08 11:04:34 server sshd[245640]: Unable to negotiate with 86.111.187.162 port 36652: no matching key exchange method found. Their offer: diffie-hellman-group14-sha1,diffie>
Oh.
Looked like folks over the internet were having a blast trying to break into the server.
The first step I needed to take was to free up my monitor and keyboard and go back to the comfort of working from my desktop.
A simple systemctl start sshd was all that was needed.
# Remediation
It's clear that I need to harden my ssh setup. I already the following setup:
- root login disabled
- password logins disabled (with the exception being the SFTP user)
From the logs, looks like no one has even figured out username for the ftp user and the password for it's a long generated one via Bitwarden.
The first thing I did was reduce MaxAuthTries to 1.
The reasoning was pretty simple, either you know how to get in or you don't.
The next steps were to consult the arch wiki!
Turned out I had already taken most of the necessary steps.
I was just missing a rate limiting/blacklisting solution.
I settled on using sshguard combined with ufw.
I verified with my dad that he could still login and do his work.
Fingers crossed, this would be the last time I would have to deal with something like this.
# Results
My server has been up and running ever since with no issue to ssh.
I have also installed netdata and made it accessible solely over the local network.
This is just to give me a birds eye view of what is going on with my server without having to ssh in.
Taking a quick look at logs shows me that sshguard is working as intended
Nov 03 14:27:28 server sshguard[1006]: Attack from "176.111.173.238" on service SSH with danger 10.
Nov 03 14:27:28 server sshguard[1006]: Blocking "176.111.173.238/32" forever (REDACTED)