At a recent partner summit, as we discussed network security, I was reminded of a project I worked on in a previous life. The TL;DR version is that some time ago we filtered a user’s access at the control plane level, instead of using traditional routing or firewall options, and this technique can easily scale to an enterprise level. This is what we did:
Back in the early aughts, more than 10 years ago, I was working with a bunch of integrators on designing and building enterprise networks. Our claim to fame was that we could do this stuff quickly, in 3-4 weeks. But it came at the price of long hours and made us kinda punchy.
One of the ways we let off steam was a long historical cultivated culture of pranking, which also helped keep us on our toes (rule number 1: never tell anyone your hotel room number). Since I was the manager of the project, I was a prime target, so I kept a network of spies to inform me of possible threats.
One day I was quietly informed that a senior engineer was “planning something,” and we decided that we needed to make a preemptive strike.
A few ideas were bounced around, but we wanted this to be subtle and indirect, more an environmental or virtual prank. I quickly vetoed many ideas as the target wasn’t an idiot and had his own protections in place. I also didn’t want to badly break the production or management network, so cosmetic changes to ACL or firewall rules were off limits. But he didn’t have any access to his network control plane, so that’s where we made our move.
The first directive was: “identify all his devices.”
A physical trace of his port quickly got us a device identifier and some initial addresses,
We then flipped his DHCP to fixed addressing and now we controlled his identity on the network. Never underestimate the power of a well-managed IP Address Management (IPAM) system.
The second directive was: “find out where he goes on the internet.”
So, we set up a sniffer to capture all of his DNS traffic. We did this for about 48 hours, then crunched the data to not only find a list of his top 90% websites but also used that data to identify his other devices since they matched the same pattern. DNS query logs were not used back then, but the concept was the same, and that level of visibility into an actor’s behavior is amazingly powerful.
Now we were really in control. We had all his devices, a good idea of his behavior, and we never needed physical access to his equipment or browser history. The next step was to make his life a living hell.
The last directive was: “mess with his DNS and let me know when he finally cracks.”
We steered him to a special instance of the DNS server, we did this with some simple DNS view magic and started messing with his answers.
But only some of the time.
We never touched any site that was related to work (Gmail, google drive, bank accounts, etc.) but for the rest of the namespace, it was open season. Technically, we randomized the RR sets for his top domains, so if he queried one domain, there was a 25% chance he would get an answer for another domain from the same top locations.
Try to go to CNN, sometimes he got Google or ESPN. Try to hit Facebook, every few hours, he ended up at myspace. There was this random chance that no matter what he tried he would end up at the wrong website. But because all the destinations were still known endpoints, it really looked like his browser was messed up. Even if he worked all this out and manually tried to fix his DNS, it wouldn’t have helped because we were proxying port 53 on his behalf.
(No, he never worked that out, and before you ask, yes, I was ethically fine with this. If you’re checking sports scores on company time, you get what you deserve.)
Sometimes, just for fun, we cleaned up the answers, and his life seemed normal for a few hours. If he approached the few people who were in on the game about the problem, they just redirected him to the Routing or Firewall teams. And, as they hadn’t done anything, they could honestly say there was nothing wrong with his traffic.
It drove him mad. We owned him.
It took less than 48 hours before one of his minders said we had to kill the project as he was about to re-image his laptop, it was time to come clean. (But we made him reinstall his browser a few times.)
At the All Hands meeting that evening there was a moment of ritual humiliation when we told him what we had done, how he was being punished and re-told this story to the entire team as an education on the powers of the control plane.
There were a lot of ‘ah ha’ moments that night, and no one ever tried to prank me again. Once they realized I owned the Internet, they gave up.
At the time, I didn’t think I was doing anything clever, I was just taking a logical approach to securing my network. But it is interesting to observe that today, in 2018, this concept is still only just getting traction as a valid approach towards security.
What we did for one person can easily scale to an entire enterprise, you just need the right tools and utilities that can manage things at a control plane level. Allow, disallow or steer traffic decisions before the packets are sent.
And it is trivial to implement what we did, it is just an application of a DNS firewall.
Yeah, you can try and manage traffic by filtering the data path. But steering specific users via the control plane is infinitely more effective.