DDoS mitigation strategies

Thought I'd throw this out there. It's a scenario question that I sometimes ask in interviews just to see how the candidate responds (there's no real right answer), but it's an interesting problem regardless that practically everyone potentially may have to deal with eventually.

Let's say you manage a business-critical web presence based out of your headquarters or datacenter and your individual uplink to the provider is 1 Gbps. Let's double-down and say you had two of these, one on the west coast of North America and another on the east coast.

One day Anonymous announces that because they don't like the color of your corporate logo, in 24 hours your organization will be hit with a 100 Gbps constant flood of who-knows-what. Your operating budget doesn't allow for support from the big-boy DDoS mitigation providers like Prolexic (now part of Akamai), Verisign, Neustar, etc.. Your web properties don't allow for easy transition into something like AWS or Cloudflare. And your CEO is just itching for a reason to fire you because your salary qualifies you to be categorized as a cost-center and cheap labor overseas is mighty tempting.

What would you do?

Kobayashi Maru?

Find more posts tagged with

Comments

JaneDoe

You can call your ISP and ask them to block the traffic upstream, or you can set up your own web proxy service using AWS, and only direct legit traffic to your site.

After you have your proxy set up, you change your IP address so they can't use the IP they got earlier to DDoS your real site.

docrice

The problem with this approach though is that you don't know what's legitimate/illegitimate because the source IPs could be spoofed, and if your customers are worldwide, this becomes an impossible cat-and-mouse to perform in real-time. Many ISPs would rather just null-route your assigned block until the headache goes away.

JaneDoe

There are other ways to tell the legit traffic besides the source IP, like packet type, depending on the type of DDoS attack. If you're running your own proxy, you can control what traffic you let through to your real server. You can also move your site to DDoS protected hosting, which may be much cheaper than traditional DDoS mitigation services, but is still expensive.

Beyond that, there really isn't much you can do; there is only so much traffic that can travel though your pipe. The only way to deal with that amount of traffic is to intercept it before it gets to your internet connection. If you don't have the money or the bandwidth, there is nothing you can do (nothing legal at least).

GreenHornet

In our Noc we just simply block the source IP with an ACL, but if the attacker is using a /26 block along with the distributed (DDoS) we simply just turn down your port or black hole the route due to link saturation on the router and the damage it has with other customers traversing over that link. I'm eager to learn more about the mitigation techniques you would use in this scenario.

SecurityThroughObscurity

docrice wrote: »

a 100 Gbps constant flood of who-knows-what. Your operating budget doesn't allow for support from the big-boy DDoS mitigation providers like Prolexic (now part of Akamai), Verisign, Neustar, etc.. Your web properties don't allow for easy transition into something like AWS or Cloudflare.

In that case you can do nothing, only black holing.
your network equipment will die under this pressure of traffic.

Guys, if you interested, cisco has released a guide to defending against ddos.
A Cisco Guide to Defending Against Distributed Denial of Service Attacks - Cisco Systems

NightShade03

I'd also question you (the interviewer) back to see if anything in the current architecture could be leveraged. For instance if you are using F5 for load balancing inline you can just slap a license key on an some *some* DDoS protection. There are other vendors out there as you mentioned, but if you have the beginnings of the equipment why not use it.

You could also review router/firewall configs to ensure things like null packets, and black hole routing are enabled.

JaneDoe

NightShade03 wrote: »

I'd also question you (the interviewer) back to see if anything in the current architecture could be leveraged. For instance if you are using F5 for load balancing inline you can just slap a license key on an some *some* DDoS protection. There are other vendors out there as you mentioned, but if you have the beginnings of the equipment why not use it.

You could also review router/firewall configs to ensure things like null packets, and black hole routing are enabled.

This would only work if you have more total bandwidth than the attack is using, and it's attacking a higher layer of the OSI model like server resources. 100Gbps on a total of the a 2Gbps pipe is effectivly a physical layer problem. The only way to stop it is to stop the traffic before it gets to your pipe.

As I said earlier, you could create your own proxy service using AWS or something, but that may not be much cheaper than using a big name service.

docrice

I brought up this topic as a bit of a mental stretch exercise to see what people would come up with. The scenario with its restrictions is a damned-if-you, damned-if-you-don't, but it's fun to try and hack the DDoS with improvised countermeasures, even though resistance may end up being futile. If anything, it at least highlights the fundamental problem and the fact that no matter how many anti-attack devices you have, the firewalls, load balancers, and IPS sensors may shatter with that kind of a traffic load.

Some uplink providers may provide their own DDoS mitigation service, but they might not be as good as a specialized, purpose-built service. You could also pay for more bandwidth ... but yeah, 100 Gbps (assuming the capacity is even available) is cost-prohibitive enough that any employer will balk at the first sign of the length of numbers on the quote. Maybe haul your datacenter to another facility and set up shop to a new circuit? Oh wait, the attackers will follow you as soon as your DNS records are updated. Crap.

Make the assumption that [insert stereotypically-criminal country flavor-of-the-week] is attacking from their geographic location and hire a botnet to counterattack. Well, there might be some legal and ethical problems with that. Global thermonuclear war.

SecurityThroughObscurity

And what exactly would you counterattack?
IP addresses of zomibe machines?

BGraves

Fascinating thread as I'm just starting a class with WGU that covers DOS and DDOS.
Props to SecuritythroughObscurity for the link to the Cisco guide. Very helpful to pair with my current reading!

JaneDoe

SecurityThroughObscurity wrote: »

And what exactly would you counterattack?
IP addresses of zomibe machines?

You could do something like hack AWS, or some other large scale cloud provider, or a DDoS mitigation service, and use their resources to deal with the attack. It isn't exactly a counter attack but it would work. However, this is illegal and immoral.

apr911

There is no right answer because there is no answer. You can't really do anything to pre-emptively plan for and mitigate the attack because you dont know the attack vector that will be used. There are dozens of attack vectors to consider and new ones be utilized everyday.

You could implement all sorts of little tricks to try and mitigate the attack in the moment but in truth, you wont succeed because the attack is going to overwhelm your infrastructure anyway.

Even moving to a full-on cloud environment wouldnt save you. First of all, you'd have to have a ton of boxes just serving the website (not including any database or application backends) to support the that sort of traffic flow. Then you'd have to tune each box to mitigate the attack. That's assuming the cloud provider still keeps you online. The second you start to impact other customers on the hypervisor or network they're likely to null route you anyway.

There is a reason DDOS mitigation costs as much as it does.

Frankly, if this were a real world situation... Id quit. The CEO isnt willing to give the budget required to mitigate such an attack and is already itching to fire me for the "cost savings" which basically tells me even mitigating this attack on the cheap isnt going to offer any more job security. At the end of the day its not a place Id want to work.

W Stewart

Start limiting the number of simultaneous, total and half open connections, implement reverse path filtering, rate limiting, wait and hope for the best.

EdTheLad

The main issue here is the link from the ISP will get saturated, only thing you could do is find out the src addresses from your more important customers, create an src based acl only allowing these ip's and blocking everything else. Hopefully you have a good relationship with your ISP and they will implement it on their egress port.
Other than that, have a backup network that's only used when under attack, provide priority customers with the secret subnet, letting them know that the subnet will only be active when under attack.

Chivalry1

Unfortunately there is no "win-win" technical solutions here. Many ISP's are utilizing DDOS technology like Prolexic within the cloud to reduce the DDOS threat. But unfortunately there is not much that can be done. You could potentially attempt to use a hybrid on-premise/cloud architecture using Netscaler "Global Load Balancing Service" (GLBS) to have the host service absorb some of the DDOS traffic. As probably mentioned already rate limiting and TCP packet inspection. You could also attempt to locate which ISP the majority of the source traffic is coming from and contact them directly for assistance. I know most ISP's won't care about your phone call. So soon after you hang up the phone, change the DNS A Record of the WWW website back to one of their SRC IP addresses; and see how they like DDOS traffic.

apr911

Chivalry1 wrote: »

So soon after you hang up the phone, change the DNS A Record of the WWW website back to one of their SRC IP addresses; and see how they like DDOS traffic

Just to be clear for those who might think this a valid mitigation technique, you shouldn't do that unless you want to find yourself in hot water with your domain registrar and facing possible legal troubles.

Changing your DNS record in such a manner would likely cause the target to shift to the new device however you are now reflecting a DDOS and you are now responsible for a DDOS on another network, even if it is the network originating a DDOS on you.

Besides, chances are the other end is merely a misconfigured device that doesnt restrict traffic or detect spoofed packets. Even now, most DDOS's run on a similar methodology as SMURF in the 90's

Chivalry1

apr911 wrote: »

Just to be clear for those who might think this a valid mitigation technique, you shouldn't do that unless you want to find yourself in hot water with your domain registrar and facing possible legal troubles.

Changing your DNS record in such a manner would likely cause the target to shift to the new device however you are now reflecting a DDOS and you are now responsible for a DDOS on another network, even if it is the network originating a DDOS on you.

Besides, chances are the other end is merely a misconfigured device that doesnt restrict traffic or detect spoofed packets. Even now, most DDOS's run on a similar methodology as SMURF in the 90's

LOL...yes it was just a joke.