I haven’t written many technical posts of late. I’ve been having far more fun writing fiction. My flights of fancy, however, came down to earth with a bump earlier this week when I discovered a couple of my sites had been hacked. Including this blog.
Now, I’ve been around the block a bit and this was not my first rodeo. In my professional career, I’ve had to deal with two or three security incidents a year of various severity. Not to mention, I secured my first job at a college after hacking their mainframe. A story for another day.
Anyway, I thought I’d have a crack at a bloggers guide to being hacked for the layman. I’ll cover some of the why’s and how’s and what now’s that inevitably come to mind when you are staring at your corrupted website.
This isn’t going to be some thinly veiled pitch for a silver bullet “buy now” product, so relax. I’m also not going to be advocating one blogging platform over another. There are entire threads of that nonsense. Usually written by people with either monetary, intellectual or oddly emotional axe’s to grind.
Let’s start with …
Why Me?
The first question anyone asks when faced with a calamity. In this instance, the answer is sadly, “Why not you?”. What have you done to make yourself less of a target? The reality is that anything put on the Internet is instantly going to become a target. In our online world, there are sticks and stones that can be thrown, even if the name-calling on social media doesn’t hurt you.
Say the word “hacking”, and most people’s ideas are going to be fuelled by films, TV and that article they read while waiting at the dentist. Needless to say, the Hollywood vision of hacking is far more focussed on entertainment than any reflection of reality.
You are not Jason Bourne, this is not some government conspiracy. That nefarious shadowy covert department, well they do exist (be in no doubt) but they have far more interesting targets than hacking your latest “Best chilli ever” post. No matter how intriguing that “secret ingredient” sounds. Let’s be honest, it’s probably dark chocolate. It’s always dark chocolate!
No, for your average blogger like me, most hackers will consider us sheep. At best maybe an antelope. Remember that documentary you watched on the magnificent plains of the Serengeti. Huge herds of migrating antelope. Remember the lions and the cheetahs, the leopards and the crocodiles. Well, you get the picture, the Internet is a digital savanna, you and your site the antelope and the hackers, well they are the hungry predators.
There are of course people who will be singled out and hunted. If you are an activist, and I don’t mean of needlepoint, politically! If you are successful and annoy the wrong crowd, you’ll know you are a target long before your site is hacked.
You may also get singled out if you have something of value, i.e. some really ornate antlers. The only saving grace is that poetry and prose aren’t quite as interesting as passwords, credit card details and people’s personal data. If you host any of these on your website, well more fool you, you have a bullseye painted on your rump. There is NO reason to hold passwords or credit card details on your website in 2019, use reliable 3rd parties whose full-time business it is.
The run of the mill blogger, is unlikely to have much attraction so what value do we have to these hackers?
It’s not you, it’s your resources
As a blogger, you are likely to get hacked because you are an easy target. It’s not personal. It might feel like it, especially given the effort you’ve put in. A leopard doesn’t pick out its prey based on its views. A hacker won’t read, let alone be moved by your perfect prose. You are simply lunch. If you are in any doubt, my other site that got hacked last week was a charity site for kids with special needs. It didn’t matter, the ethics of the situation are not a consideration.
What makes your site a tasty treat is its resources. Your website has value because it can be repurposed into something that is useful to the hacker.
This recent hack focussed on injecting so-called malware onto my site. In this instance, the interloper redirected my users off to their own dodgy sites. Lots of popups, popunders anything to get my visitors to engage with their content, ideally thinking that it was mine.
To labour the antelope analogy. It’s like a predator wounding you rather than just killing. Why? Because when the other antelope’s come along, the chances are they might get infected themselves or at least leave themselves vulnerable to being attacked.
It’s a question of scale
You might still be wondering how much value can your one little site have. Personally, I get a handful of visitors a day, mostly from a fairly small community of other bloggers. Surely my site can’t be that valuable to warrant being singled out?
Of course, it’s not. The thing is, this is not some precision kill with a high calibre rifle after days of skilled hunting. This is someone with a shotgun running through the herd unloading it every which way. They are not interested in getting one antelope they want the whole herd, or at least as many as they can. Thankfully nature isn’t so random. It takes a human hand for this level of inane carnage, be it websites, buffalo or ivory hunters.
This shotgun-toting analogy doesn’t quite work, however. A single idiot running around randomly shooting at the herd is not going to make much of an impact, for we are many. But this isn’t Tanzania, it’s the digital world. It’s easy to create a program that will go off and do all the hard work for you. There’s no dark corner, with an army of hackers tapping away, beating down the walls of your site. Rather, a single hacker runs a single command and countless predators are unleashed into the digital grasslands. The owners of such programs, will then go and get a coffee, sit back and wait for the body count to rack up. At the same time, they’ll be selling their spoils on and that’s where the value is. A single antelope is worthless, half a herd, well that has real value.
How did they hack me?
The technical answer would witter on about attack surfaces and vectors, before delving into some rambling dialogue on script injection, buffer overflows or countless other tricks of the trade. Instead, I’m going to ramble on with my dangerously overworked antelope analogy.
Your blog is an open wound, I know you’ve probably heard worse from friends and trolls. But bare with me. Every service you share on the Internet is a weakness, a wound if you like. Each wound is an opportunity for a predator to bring your site down. The first rule of securing any site is to reduce the number of wounds (reduce the number of visible ports/services) you expose to the world, it’s one of the primary functions of firewalls.
I use WordPress for blogging. It’s a hugely popular, easy to use, flexible content management system. It has a huge eco-system of third party plugins that can do almost anything your heart might desire. Unfortunately, all of this flexibility comes at a cost and that is complexity. Complexity defines the depth and nastiness of your wound. WordPress has a very distinctive wound. It’s one that’s easily spottable at a distance. It’s also a very deep wound with plenty for a predator to try and snag a claw or tooth on.
With so many antelopes sporting the same wound, is it any wonder that the predators might make a concerted effort to target antelope with a common weakness.
I could run my blog using some tiny hand-crafted home-rolled code. It would be an unusual and significantly shallower wound. One a predator is unlikely to recognise let alone be deep enough to get a claw into. The downside is I’d lose convenience, functionality and flexibility. As well as committing the cardinal sin of re-inventing the wheel.
In the case of WordPress, its biggest strength is also its Achille’s heel, plugins. There are so many plugins, written by so many developers. Most of these plugins will use code from other developers in the form of libraries. To further compound the problem these libraries will often use code from other libraries written by even more developers. That is a lot of code, a lot of people and a lot of potential for someone to have inadvertently made a mistake.
For the avoidance of doubt, if you think this an exercise in WordPress bashing, you’re missing the point. This isn’t a unique problem to WordPress. Every content management system I’ve ever used (Mambo/Joomla, Drupal, Sharepoint, Confluence et al) all have been exploited multiple times. Every webserver (IIS, Apache, Nginx etc). every non-trivial library, they’ve all had users cursing a developer at one time or another.
To answer the question (finally), you got hacked because someone made a mistake, someone failed to harden their code against the very worst the Internet can throw at it. Most likely because there was a deadline or it was Friday afternoon and the pub was open.
Why are there security holes?
Software engineering, it’s a great term. I have several qualifications in it, I’ve taught it and I’ve spoken at great length to countless teams about the practice of it. Engineering! It conjures up a vision of skilled architects pouring over blueprints meticulously combining the certainty and reliability of science and mathematics. Of industrial construction, iron, concrete and steel skilfully combined to create our solid, dependable modern world.
Unfortunately putting the word “Software” next to “Engineering” is at best aspirational and at worse deliberately misleading. To illustrate the problem, consider the majestic Golden Gate bridge. How many scenarios did the architects and builders envisage in its construction? Probably thousands, it was after-all a groundbreaking feat of engineering.
Amongst those scenarios, how many dealt with malicious intent? Let’s not be naive maybe hundreds, ranging from idiots planting bombs, lorries/planes/boats being driven into supports, arson and the list no doubt goes on.
Now let’s contrast that with my imaginary GoldenGateBridge web service. A fictitious and popular mainstream piece of software used by hundreds of thousands of businesses to deliver their day to day services over the Internet. How many malicious scenarios should I be considering in the “engineering” of that piece of software? Hundreds? Thousands?
In the real world there is a limiting factor, call it practicality. I could come up with some insane scenarios; why not develop a chemical that makes seagulls poop even more corrosive and sprinkle breadcrumbs on all the support cables?; Or stand at the end of the bridge and redirect the patrons straight into the bay, road runner cartoon style. The possibilities are endless and thankfully in the real-world mostly impractical. Generally speaking (because there’s always an exception) no one would have the time, resources or opportunity to pull off such madness.
In the virtual world, the same practicality test does not hold true. What’s time when we have a billion processors each performing millions of instructions a second? What are the limiting resources? Electricity! Hardly a limiting factor. As for opportunity. Not only can I have my own test bridge to dissect and abuse in private. I can then go online and start poking at other peoples bridges with almost total anonymity.
Try taking a sledgehammer to the real bridge and see how quickly you’ll find yourself in a jail cell. Yet a hacker halfway around the world can be slamming a virtual sledgehammer against your site for months and the chances are you are blissfully unaware, and you definitely won’t ever find out who they are.
The sad reality is EVERYTHING is hackable, given enough time and effort. Some of the hacks I’ve seen over the years have been staggering in scope. Thinking outside the box doesn’t even start to describe the ingenuity. In the face of such an onslaught, there are always going to be security holes. The best that can be done is to plug them when they are found, draw up guidelines to minimise their re-occurrence and ultimately reduce how many wounds we expose.
What Now?
Having discovered you’ve been hacked, start by getting it out of your system. Personally, I find shouting some well-chosen words goes some way to expressing my angst. That done, shut your site down. What happens next depends on one word “backups”:
Happily Ever After
If you have regular backups, congratulations. You can pump your fist in the air like a doomsday prepper discovering a yard full of zombies. It’s the day you planned and waited for. You are not quite out of the woods. One of the challenges you will face is determining how and when you got hacked. You’ll need to know how they did it, otherwise, you’re likely to find your clean restored site hacked again within hours, as they simply reuse the same exploit.
In this case, it took me a couple of hours of comparing filesystem/database dumps between the hacked version and what I considered a clean backup. That allowed me to identify which parts of the site had been maliciously modified. In my case, it pointed to a specific plugin, one that could luckily be disabled without impacting the front end experience. That was a bit of luck.
Having identified the point of ingress I could then determine when it happened. It had only been a few hours, again that’s just luck. It could have been days before I had spotted it. They had gone out of their way to hide the re-direct activity from logged-in users.
For my own interest, I spent a few hours analysing what it was they had actually done, a somewhat painful and fruitless task given they had obfuscated their little gremlin. It seems hackers are not big fans of comments in their code, or in fact leaving any little artefacts that might make it easy to analyse.
All in all, with some reliable backups it took the best part of a day to recover and get to the root of the problem across two sites. But at the end of it, I knew with a high degree of confidence my site was secure again, for another day.
End of Days
What happens if you don’t have backups? Well, you are in trouble. A lot of trouble!
Without a backup to compare good vs evil you are unlikely to be able to find what the hackers have done. Even if you isolate the immediate problem down to a specific feature or plugin and disable or patch it. You’ll never be sure that they didn’t leave some other surprises. Most likely a backdoor back into your system. So that even after you’ve stopped the obvious bleed, they’ll just pop back in and reapply it. Plus they’ll probably do a better job of hiding it from you the next time around.
Once your site is compromised, in most instances it’s game over. Just like a bad zombie horror movie, don’t be that fool who won’t kill their bitten loved one, “It’s just a fever, they’ll be alright!”. We all know how it ends, usually with your brains being an entree. Once bitten, your best course of action is to make it quick and painless, DO NOT try and nurse your site better.
You can try and get your content out and then import it back into a known clean installation. This is a nightmare, it’s tedious and soul-destroying but without backups, it might be your only option. Be wary of export/import functions in your blogging platform that might actually move malicious code to .the new clean install. If you exporting javascript or even CSS you can be infecting your new site before you have even run it. If you do find yourself in this unenviable situation double-check the exported file if possible and make sure it’s just the data you expect.
There are recovery services, I’ve never used them, so I’m afraid I can’t vouch for their effectiveness. The likelihood is they are too expensive for most casual bloggers anyway.
It’s not paranoia, they are out to get you
Like a good scout, be prepared. If you are running a blog today on any platform and you don’t have backups. You are living on borrowed time. Please, STOP reading this post and go and do a backup, right now. You’ll thank me in the future.
Given that most bloggers update their content regularly you will want to automate backups with a frequency that you are comfortable with. There are loads of tools for every platform go find the one that most people are using on yours. I’m not going to link them (because I’d have to maintain them), just go google it you’ll find half a dozen articles for every blogging platform.
Don’t skimp on the number of rolling backups you maintain either. You may need to go back days or weeks to find that good clean copy, dependent on how long it is before you (or someone else) spots an issue. There’s nothing worse than having rolling backups for 7 days only to discover it’s been a month since you were hacked.
Lastly, research security tools for your blogging platform. Find the malware/exploit scanners, firewalls and other tools that might just make you a slightly less vulnerable target.
With just a little bit of planning and preparation, you can ensure that when the worst inevitably happens, that you’ll be back bounding across the digital savannah in no time. Now go check your backups, scoot 😉
Cover image courtesy of: Clint Patterson