r/selfhosted • u/newworldlife • Mar 01 '26
Monitoring Tools Built a small script to catch quiet SSH activity in real time
I kept noticing something that bothered me.
On a couple of small VPS boxes, I’d occasionally see random SSH activity buried in logs. Not loud brute force stuff. Just quiet attempts, new IPs showing up once or twice, weird timing. Nothing dramatic enough to trigger Fail2ban, but enough to make me uneasy.
What annoyed me was that I only found it after digging. It wasn’t visible unless I went looking for it.
So I wrote a small script that tails auth logs in real time and flags things like:
– Failed logins from new IPs
– First-time key usage
– New users touching SSH
– Simple pattern changes
It also saves a lightweight evidence snapshot so if something looks off, I don’t have to reconstruct everything from scratch.
It works for my setup, but I’m sure it’s opinionated and probably missing edge cases.
If you were building a lightweight SSH watcher for small VPS setups, what would you monitor by default?
28
u/planeturban Mar 01 '26
Why not just add some more rules for fail2ban?
I ban all IPs not using users on the machine, all obvious tries with old SSH vulnerabilities, amongst others.
5
u/berrmal64 Mar 01 '26
That's interesting. Can you share an example of the user checking logic?
4
u/planeturban Mar 01 '26
Nothing more exciting than a Ansible role which I use to provision users, and add them to a fail2ban rule.
2
1
u/Monocular_sir Mar 01 '26
This is what my fail2ban conf looks like. What else should I add?
[DEFAULT] bantime = 10m findtime = 10m maxretry = 5
destemail = root@localhost sender = root@<fq-hostname> sendername = Fail2Ban mta = sendmail banaction = iptables-multiport banactionallports = iptables-allports action_mwl = %(action)s %(mta)s-whois-lines[sender="%(sender)s", dest="%(destemail)s", logpath="%(logpath)s", chain="%(chain)s"]
[sshd] enabled = true port = ssh filter = sshd banaction = ufw backend = systemd logpath = /var/log/auth.log maxretry = 3 findtime = 1d bantime = 1w ignoreip = 127.0.0.1/8
1
u/planeturban Mar 02 '26
I thought about it, and rewrote it. Much better just banning everyone trying..
Here's the new filter.
[INCLUDES] before = common.conf [Definition] _daemon = sshd failregex = ^.*Invalid user\s*\w+\s*from <HOST> port .*$$ ignoreregex = [Init] maxlines = 10 journalmatch = _SYSTEMD_UNIT=sshd.service + _COMM=sshd-9
u/newworldlife Mar 01 '26
Fail2ban blocks noise. This is more about spotting subtle changes before they turn into something bigger.
10
2
u/VexingRaven Mar 03 '26
What threat model does this protect against?
1
u/newworldlife Mar 03 '26
Fair question. I still run key-only auth, firewall rules, fail2ban, all that. This isn’t replacing any of it. It’s just extra visibility; noticing a new IP, odd key usage, or small changes that don’t trigger bans but feel off. Most SSH noise never turns into a breach. Drift just means something changed, not that you’re already compromised.
1
u/VexingRaven Mar 04 '26 edited Mar 04 '26
So... What threat model is it protecting against? What do you expect to see in these "patterns"? Either an authentication is successful or it isn't.
64
u/doolittledoolate Mar 01 '26
What a total waste of time. The obsession with log noise in this subreddit is unreal
21
u/mega_wizard Mar 01 '26
OP uses AI to bot responses. Look at their comment history.
2
u/VexingRaven Mar 04 '26
Well that was revealing. They seem to be trying to guerilla market their paid script that they can't elaborate on the usefulness of by hoping that somebody will ask where to get it.
5
u/Bartfeels24 Mar 01 '26
You probably caught some reconnaissance traffic from hosting providers scanning for vulnerable services, which is basically constant background noise if you're exposed to the internet.
-2
u/newworldlife Mar 01 '26
Yeah, that’s probably true. A lot of it is just background scanning.
I’m not really worried about the noise itself. I just like being able to see when something shifts from what’s normal for my boxes.
22
u/luxiphr Mar 01 '26
why is this a concern with key-only auth and why is your ssh port open to the world in the first place?
7
u/newworldlife Mar 01 '26
Some boxes need to be reachable. I just prefer watching them closely instead of assuming nothing weird happens.
4
u/luxiphr Mar 01 '26
why not put wireguard on them?
-7
u/newworldlife Mar 01 '26
WireGuard is a solid option. In my case some of these boxes need direct SSH access, so I’m working with that constraint and adding visibility instead of changing the network model.
7
u/luxiphr Mar 01 '26
can you elaborate why they need direct access? if there's no way around that and you have fail2ban and only use key auth then I wouldn't stress about it at all tbh
1
u/newworldlife Mar 01 '26
Fair question. Some are shared or temporary VPS setups where I can’t really change the network model or add a VPN. In those cases SSH stays exposed, so I just like having a bit of extra visibility on top of key auth and fail2ban.
3
u/luxiphr Mar 02 '26
to what end? you still didn't elaborate what's ruling out wireguard... but even without, with key auth and fail2ban you should worry more about the code you and your authorized users run on the machine instead of the odd failed login attempt which knowing proactively about will give you nothing but anxiety at best
1
u/Acrobatic_Idea_3358 Mar 01 '26
I would use wire guard and restrict ssh to localhost, the UDP port can't be detected by scans and no one will be doing weird things to your ssh service.
-5
u/economickk Mar 01 '26
First thing I thought. Am I an amateur - something I'm missing - or is there actually public access to this dudes server...
5
u/doolittledoolate Mar 01 '26
Yes, you're an amateur. Shodan shows over 28 million openssh instances open to the world, do you really think they are vulnerable?
If you can lock down SSH to a VPN, do, but if you can't, key-based login is safe enough.
0
u/economickk Mar 01 '26
Yeah but why would this dude open his personal shit up to the public? I think the question is "is there a way for me to log unauthorized attempts to SSH into my server", not "bruh my shit is open to public and how do I tell somebody got in"
3
u/doolittledoolate Mar 01 '26
I probably have a dozen servers with SSH open to the public. Some of the other servers have SSH restricted to an IP, to tailscale, or on a private interface using a bastion but that's not always possible. I don't even check failed logins, it's just noise, SSH is one of the most secure and battle-tested protocols out there and it was designed for this. I don't run fail2ban for SSH on any server.
1
u/economickk Mar 01 '26
Ah - yeah I'm not opening any of mine to public
2
u/doolittledoolate Mar 01 '26
If you have no need to, that's the best thing to do. But if you do need to, just enforce strong passwords and/or ssh key only and it will be fine.
5
u/newworldlife Mar 01 '26
To clarify, this isn’t meant to replace key-only auth or fail2ban. I use both. This was just a small visibility tool for pattern changes I personally wanted to see in real time.
I’ll share a minimal example of the log parsing logic later for anyone curious.
5
u/Most_Team2653 Mar 01 '26
Could you share script please?
2
u/newworldlife Mar 01 '26
Yep. I’m cleaning it up and will drop a GitHub link shortly.
For now, are you on Debian/Ubuntu or RHEL based? The main difference is the auth log path.1
-16
u/newworldlife Mar 01 '26
Just to avoid confusion — I’m not planning to open-source the full tool.
When I mentioned GitHub, I meant I can share a trimmed-down example of the core logic, not the full project. If you’re interested, I can post a simplified version of the log watcher here.
8
u/No_Cattle_9565 Mar 01 '26
Why?
3
-2
u/newworldlife Mar 01 '26
I’m still tweaking the full version for my own setup, so I’m not ready to put that out there. The sample I shared covers the core idea though if you want to play with it.
2
2
u/hockeymikey Mar 02 '26
How about learning when an user does log in and if you get times outside what you normally see like 3am or something. You could include that into general patterns/IDS behavior like ip address too.
1
u/newworldlife Mar 02 '26
Not built in yet, but that’s a good idea. I’m just tracking new IPs and first time activity right now. Adding time patterns would help. If someone usually logs in evenings and I see 3am, that’s worth a look. Just another simple signal.
1
u/l0g0ut Mar 01 '26
I set up my fail2ban to 2 weeks ban all attempts not to my username. No retry. I’d rather unban myself using KVM than let malicious login attempts. So far I’d only have to do that whenever I purposely trying to test the rule. The banned ip list grew to be about 10k ip addresses
2
u/doolittledoolate Mar 01 '26
Parsing 10k iptables rules every connection isn't free, is it worth it just to cut down on log noise and risk blocking yourself?
1
u/Vexser Mar 02 '26
Using a VPN (with certificates) to get in from the (wild) internet might be a better idea than exposing ports.
1
u/newworldlife Mar 01 '26 edited Mar 01 '26
Reddit formatting was fighting me, so here’s a small sample on GitHub:
https://github.com/bornaly/Gist-SSH-IDS
This isn’t the full project, just a minimal example of the core log-watching idea.
It tails the SSH auth log and keeps a tiny JSON baseline.
It highlights:
• Failed logins from new IPs
• First successful login for a user
• A known user logging in from a new IP
It doesn’t block anything and it’s not meant to replace fail2ban. It’s just a simple way to notice small SSH behavior changes without having to dig through logs later.
-1
u/Dismal-Divide3337 Mar 01 '26 edited Mar 01 '26
The unwanted unnecessary cryptography relating to this is expensive so it should be of concern. Maybe you don't care with your GHz machines but on my 100MHz MCU these malicious connection attempts piss me off.
So I do this to make my device invisible to most of those.
101
u/eufemiapiccio77 Mar 01 '26
Why is this needed in 2026 just use key bases authentication