A Byzantine Victory

OUR RECENT EXPERIENCE WITH A CYBER CRIME SYNDICATE

Over 8 weeks ago we briefly reported that Sharon had detected a malicious breach in our systems & I was investigating the issue. At that time, I felt the issue was a low risk, low impact event - worthy of highlighting to our clients so they could raise their guard/awareness. 

What I got instead was an organised, sustained, prolonged battle with a team of determined international hackers who were far more than a one trick pony operator. This was a team of diligent, professional hackers who had exceptional system knowledge of Linux, Windows, Android & Chromium; could change tactics & strategies in direct response to my actions & had breached the neighbouring network solely to camp outside our infrastructure.

This was a determined attack to achieve financial gain & further new leads from our client base; with the attackers displaying a tenacious capacity for dynamic thinking as they modified tactics in real time. They demonstrated considerable patience, having camped on our network for two months; doing little more than monitoring patterns of user behaviour & searching for weak points.

Finally, they demonstrated complete mastery across all operating systems; utilising old school techniques combined with modern tools. Overall, it has taken over eight weeks to eliminate the intruders entry points & the aftermath has wrought a terrible scar across the entire business. While the attackers achieved very little in practical terms (no financial systems or crypto wallets were compromised); in defending our systems we lost a substantial amount of costly hardware & several online services have been frozen as a direct result of hacking.

Concerning outstanding orders

This is going to be a very long post (in fact I foresee this requiring a number of days to go over everything) so I want to first address clients whom are concerned about orders not honoured: I am acutely aware of a small number of purchases that meet this criteria. We sincerely apologise for this & over the next few days we will be contacting you to rectify this.

Just imagine....

I would ask that you consider for a moment had we not initiated this lock down & imagine what damage could have been inflicted if they had leap frogged from us to you...This team was not taking prisoners & if left unchallenged would surely drain every conceivable resource accessible to them. So what happened? 8 weeks is a long time to fix an issue? First of all, let me clarify that the breach happened thanks to a supplier providing us with compromised data via a USB transfer. In all honesty, this was a slip up by myself &,my god, have I been punished for it.

We will come back to discussing suppliers later as the episode also revealed alarming practises by some suppliers. The reason it took over 8 weeks to fix was due to the hackers persistence - every time we removed them from our systems they managed to regain entry within 30 minutes.

For as long as they could return we could not trade or even communicate as they had demonstrated incredible abilities in activating microphones & cameras (however we cover all webcams).

We manage a disaster recovery plan for things like total hardware failure & incidents with terminal data loss however this event did not fit any traditional criteria. For starters, disaster recovery normally is the recovery from total data loss; whilst in this instance we had no losses (yet).

However a typical OS has minimal 300 000 files so you have no real chance of identifying which files an intruder has modified; thus the best means to fix a hacked system is to format (erase) the hard drive & re-install your operating system. And it was at this stage my personal hell & almost nervous breakdown happened. Normally, formatting a hard drive and reinstalling Windows is a half day work. If you keep backups (which we do) then you may expect this to take 2 hours; meaning a rapid recovery.

However, on reinstalling Windows on our devices; the damnedest thing happened. They immediately became re-infected & within half an hour could see hackers merrily logging onto our systems. On further investigation, it was apparent the hackers had not only compromised a couple of computers, but inventoried our entire infrastructure. And the means to achieving this was our routers.

On reviewing our routers I noted they had been flashed (the operating system replaced) with one more sympathetic to the criminals. This was the first real indicator of just how skilled these attackers were & my first inclination that we were not dealing with some script kiddies but serious professionals. By hacking the routers & replacing the software they achieved two things - created an open door to continually re-visit us & a mechanism for instantly seeing new devices on the network. The hairs on my neck began to rise with this realisation & the investigation onto what was happening instantly kicked up a gear as this no longer a trivial matter.

On reviewing hidden folders on my PC I then discovered random folders which I had no access too. On granting permission my worst fears were then confirmed. Log files showing persistent packet sniffing over the course of weeks had assisted the crooks in establishing passwords for our Wifi. With this knowledge they could move indiscriminately across the network; without raising alarms. I then noticed all our mobile phones which had yet to be security hardened had signs of intrusion.

By this point 1 week has passed & both Sharon & myself were become increasingly worried. Numerous attempts at wiping our computers had failed; a new server I had built to move us to a virtualised infrastructure was also compromised; costing more time & a series of new laptops had almost instantly became infected; even though the routers which had been infecting devices were no longer on the network. We were running a make shift network to minimise entry points however I could still not pin point how they were accessing our machines.

Finally, after running a network scan I found a smart device which I had forgotten about - a smart light bulb with blue tooth & wifi capabilities had a tiny server installed on it & was capable of pinging our machines! With this removed & any notion of a network now a distant memory I proceeded to perform a further wipe of the computers.

This time, as a belt & braces measure I decided we would move over to Linux; assuming we would benefit from an instant security uplift by merit of confusion. The attackers would now need to retool & rethink any strategic attack. Except this time, they managed to get into the machines even quicker! My jaw about hit the floor & I almost burst into tears - 3 weeks in & every conceivable angle had been considered & yet I was utterly powerless to stop them.

This is no longer a company putting forward a blackout for client benefit but a deeply worried individual realising his opponent might be a state player, are certainly well funded & potentially working in shifts. The fact they could pivot on a dime & remotely adapt to my tactics left me reeling.

What the hell did they want?

Now at this stage I should point out; we are four weeks into this attack; I obviously am aware they are in our systems. They are aware I know they are in our systems. What are they trying to achieve?

To enter the centre piece of our defence strategy. The Password Safe. We operate a policy of never re-using a password & ensure all passwords are a minimum of 16 characters long whilst containing numbers, upper/lower case characters plus symbols. By using a good password safe we can easily ensure no two logon pages use the same password whilst minimising difficulties in remembering passwords; since the password safe is accessible on all devices & retrieves passwords automatically it makes password management convenient.

Our combined password safes hold just shy of 1000 logins so whilst rendering its contents safe, it also attracts attention as it can be seen as the keys to the kingdom.

With the password safe being identified as their target, we then realised that a number of strange IT incidents over the past few weeks were not incidents but acts of sabotage designed to force us to drop our guard. The hackers were not only monitoring us but actively trying to influence our behaviour. Examples included hacking the password safe client - since the attackers could not access it or guess the access code, they instead sabotaged the app so it would no longer show up on screen. We concluded this would encourage us to start emailing passwords or shout them out for recovery from the audio recordings. Unfortunately for them; even whilst unaware of their presence we did not modify our security & instead continued to use the password safe through different channels.

During the full attack they managed to sabotage or main VPN (thus monitor web traffic better); attack our ISP's Domain Name Server (thus direct us to clone sites for man in the middle attacks), freezing our bank account (to causes a financial migration) & deactivate a number of security protocols. All of these were designed to modify our computer usage & encourage bad security behaviour to give them an opening to exploit.

The power of 2 factor authentication

It became clear, once we established their target, that 2 Factor Authentication was causing the hackers significant headaches. 2FA (as it is otherwise known) utilises your phone as a secondary means of proving your access rights. In conjunction with the password safe, this mechanism provided us with a huge amount of security. We also noted a refocus of attack onto Sharons phone as they realised the password safe on its own was useless.

And since we are talking about defences....

It was incredible to note how a few simple Windows tricks hampered our guests. Our computers all logout if unused for five minutes or more. On logout a user must press CTRL, ALT & Delete to get the logon screen. Now the later exists as a mechanism to prevent remote attackers running code as a local user. So this single policy prevented them from working unless we were logged in. From studying our log files it becamse apparent they had a terrible time being constantly kicked off whilst I went to make a tea! It pleased me to think on days whereby I would be distracted every ten minutes meant their work would grind to a halt!

Our defences overall caused a great deal of problems for the attackers & certainly delayed them by a significant magnitude. Whilst far from perfect, I was overall very pleased with the performance as it hampered the attack & we detected them.

Bear in mind, a very similar attack strategy was employed against Encrochat & they did not notice until the fat lady sang. We did notice, we did evict & we did survive. It might have taken 8 weeks, but I am under no illusion - these attackers collapse business & ruin lives; to make it out is, in itself, a significant victory.

And we near the end...

The battle continued for another few weeks. We found them in our security cameras; so they had to be removed & we found them using bluetooth to hop between devices. In the end, they resorted to using neighbouring devices as a forward operating base & every time I reset a machine, they were using bluetooth to re-infect the computers. I almost pulled my hair out in despair trying to work out how this was happening until I put logcat (an android log recorder) on all none secured devices. Armed with this information I popped the wifi/bluetooth chips from our comptuers & finally managed to build systems with no intruders.

Where are we now? #MUCH BETTER SECURITY!

With the attackers gone, we were able to take stock. It is not pretty. All our web services are offline. We have no bank account for business. Clients have been neglected. Computers ruined (and hard drivers ruined because of bios attacks). We are rebuilding under arduous conditions but we have a vastly superior network in its infancy & the network setup has inspired a new network product which will be online soon - a full defence, VPN powered, advert blocking, Network intrusion detecting router which can run on a standard sim.

I have implemented 2 factor authentication even to log onto computers, we have a network modelled on Enterprise principles & at the beginning of initiating a full zero trust security model. As a fluid business, with our bank frozen, we are in a difficult place however I feel this event was necessary; as a security provider I feel we have been pressure tested. We have been found lacking in certain areas & a substantial amount of soul-searching has been the outcome.

But where we have failed, we have also shown steely determination & creativity in delivering security solutions. I have personally learned that a secure phone is no longer sufficient to consider yourself safe - protecting your digital estate is a fully integrated endeavour with all points of entry requiring thought & this will fundamentally restructure our product offering.

No more is an elite phone, outside the reaches of most a good solution. What is required is full spectrum security, so devices must be accessible as the client really must think on broader terms than just their phone. I was shocked by attackers moving to neighbouring devices to circumvent our security hardened technology & begrudgingly have to respect the technical wizardy shown by the hackers - not only were attacks a marvel of engineering, but they were strategically linked up in a manner designed to force mistakes. I will be writing about this again in the future.

The episode has taken a toll even on our collective mental health & the magnitude of this attack must be discussed to make others aware. I have not even touched on their use of virtualisation technologies, their hacking of our ISP nor discussed how I have forensically captured every bit of their work for future study. In an unusual move I am leaving the comments open as I believe people will have questions. Obnoxious, offensive posts will be simply deleted - constructive discussion is encouraged.

I am being entirely transparent about the events as I believe your security depends on trust & I am placing the future of Omerta in your hands - you decide if your security can be entrusted with a supplier whom has been through such an ordeal. I feel we have earned that right, I feel our stance as a security provider is proven but its not for me to decide; its the court of public opinion that will be the final jury.