Today I'd like to tell you, dear reader, about the time I tracked down a ne'er-do-well on a forum I now own and operate. It's a story as old as time, really. You trust someone to do a simple job, but instead they deface your forum, expose your operational communications, blame it on an innocent, and perform the whole dirty deed using an abandoned account that (for some reason) still has administrative permissions.
It all started on a nice, sunny, Monday in early June. Or at least that's what the historical weather for the date I have in my notes says. I was informed by the then-owner that her forum had been hacked. I'd done small programming projects for her previously. She was a nice lady. And a friend of my wife. And also I hate black hats. My own hat may not be a pristine white, but I'm as chaotic good as they come. So, I picked up my proverbial hammer and went off in search of a black hat to smite with it.
The first thing I learned was that the response to any question akin to "what logging do you have in place?" was a blank stare. As it turns out, they had absolutely no intentional logging. Even the logging that was originally in place to track all administrative actions taken in the software was disabled because it caused performance issues. Or so is what I was told. But what kind of of white(ish) hat would I be if I let something like a complete lack of account logging stop me?
After running headlong into the brick wall that was a complete lack of logging anywhere (not a byte of information to be found in the cPanel, even), I finagled some non-root access to the server. I specify non-root because, apparently, some of the previous programmers for the forum had proven to be a danger to, well, everything, and the host at the time refused to provide root access to anyone. At all. Ever. That was fine, though, all I wanted was to poke around and see if I could find some logs.
As "luck" would have it, I found a web server access log. A web server access log. One. One 22 gigabyte access log. For those unfamiliar with access logs, every single time you access a website, a line is generally inserted into a file with basic information about that access. It includes things like your IP, and the URL being requested. Even though this access log was huge, the good news was that the malicious user performed all their malicious actions in a rather short span of time. So, using some sed and awk (for those unfamiliar, those are nerdy computer things to find specific patterns within a text file) I just yanked out the relevant 12 hours with a cute li'l command and ended up with a much more manageable ~50 megabytes of relevant data. Sweet, some usable data! I loaded it into Splunk and went to work.
The first thing I did was look for any changes made to the compromised admin account. The URL structure used for accessing different functionalities of the forum software allowed me to see that the compromised admin account's information was saved, a password reset email was sent for it, and then it was saved again. I wasn't able to see any of the specific data sent in the request (such as the account actually doing the saving), but seeing that they accessed those specific functionalities, one could safely assume that our super-stealthy attacker went in and changed the email on the account, sent a password recovery email to the changed email, and then removed the changed email to cover their tracks.
This was a mistake on their part, and a rather odd one. The pool of accounts able to change that information for another account included those with the higher levels of access. Only a hand full of people. The same group of people who could have just gone in and changed the password directly. Which would have had the same exact outcome, and actually would have placed two fewer hits in the access log. As a lazy person, I was offended by their superfluous efforts.
After the password on the compromised account was changed, the account was used a dozen times over the following couple hours to engage in general petulance. The board that the staff of the forum used to communicate was made public to everyone, some specific pages were defaced (some in an attempt to pin the naughty misconduct on another person), and a board was deleted (later recovered through the software's 'trash' function)
After getting everything loaded up into Splunk, splitting the logs, and doing the previous investigation, it was getting a bit late. Even back then I was an old man in my mid 20's and had to get to bed. Before I did so, I "casually" mentioned to the staff of the forum that I'd be looking into each of their accounts more thoroughly the next day, and might have some questions for them. I (probably) didn't say it with an implication of "and I will nail you to the walls of hell with Satan's hammer until the inferno consumes your immortal soul", but who's to say how they took it?
By the time I got home from work the next day the perpetrator, perhaps concerned about the ownership of their soul, had already confessed. The way they tell it, they came forward of their own volition, and I had absolutely no impact. The way I tell it (which you just read), I brought the pressure of eternal damnation until they cracked like one of those cheap plastic chairs from Walmart.
In case you're wondering, I would have been able to paint their hands the brightest of red. Even though they copped to their shenanigans, I went back to my trusty access log and spent a couple hours loading up additional days of the access log and tracing the naughty nelly through a half dozen IP addresses until I found one that they accidentally used to login to their own account. But, instead of being able to have the satisfaction of applying the aforementioned paint, I had to go and do something ridiculous and lame, like employ "soft skills". Tch.
I hope you enjoyed this fine tale of heroic white(ish) hattery, and I'll see you next time.