Distributed Denial of Service

Eildor

I'm doing some research into DDoS mitigation and have come across a number of techniques such as ACL filtering and black holing. I'm slightly confused though... none of these mitigation techniques are going to work if implemented on the inside of your network, are they? If you have a 100 megabit per second internet connection and the DDoS attack is equal to 100 megabits per second then no amount of black holing, filtering, DDoS mitigation appliance or whatever is going to alleviate the problem.

The only way you can mitigate a DDoS attack is if your ISP filters the traffic BEFORE it enters into your network. Is this correct?

networker050184

Yep pretty much. Look into Remote Triggered Black Hole RTBH.

docrice

Another option is to use a third-party DDoS mitigation service which can perform a re-route through their scrubbing network via DNS or BGP changes. This is not a cheap option though. Uplink providers aren't usually specialists in this area although some may provide an add-on service (and in some cases they are merely contracted out to a dedicated mitigation provider anyway).

DevilWAH

if you have mutiply links in and run some BGP routing protocol you can redirect set policys to only allow known good traffic down one link and other routing policies to mitigate attacks.

small, medium business normally need a 3rd party to do the filtering, but larger companies will have the infrastructure to carry it out them selves.

Eildor

DevilWAH wrote: »

if you have mutiply links in and run some BGP routing protocol you can redirect set policys to only allow known good traffic down one link and other routing policies to mitigate attacks.

Could you please elaborate on that a little bit further? Not quite sure what you mean.

DevilWAH

Eildor wrote: »

Could you please elaborate on that a little bit further? Not quite sure what you mean.

If all the DDOS traffic is coming in to a single IP address on the network (say my DNS servers). with BGP I can alter the advertised routes so that all traffic to your DNS servers comes on via Link A and all other traffic uses LINK B. These leaves link B to deal with outgoing, and other business critical traffic, and dedicated the enter link A to deal with the DDoS traffic.

Eildor

networker050184 wrote: »

Yep pretty much. Look into Remote Triggered Black Hole RTBH.

If I've understood this correctly then RTBH sounds like the ideal way to mitigate DDoS attacks. Not only does it stop the attack getting into your network, but it allows you to mitigate the DDoS attack without having to rely on your ISP.

Eildor

DevilWAH wrote: »

If all the DDOS traffic is coming in to a single IP address on the network (say my DNS servers). with BGP I can alter the advertised routes so that all traffic to your DNS servers comes on via Link A and all other traffic uses LINK B. These leaves link B to deal with outgoing, and other business critical traffic, and dedicated the enter link A to deal with the DDoS traffic.

It's not going to make a difference if the attack is greater than the bandwidth you have though, is it? The DNS server is going to go down. Might as well just complete the DDoS attack and shut it down yourself?

docrice

Unfortunately DDoS isn't all that simple to deal with even if you're a large entity. DDoS comes in many forms and flooding is only one of them. If you have a 10 Gig pipe to your provider and you get flooded with 50 Gigs per second, you're still not going to be able to handle that. There are also slow-and-under-the-cover type of attacks that work more at the app layer. Once your firewalls, IPS sensors, and other inline devices get pushed to overload, things just don't work so well.

There's a reason why players like Verisign, Prolexic, and Neustar are in business. I looked into this issue a while back and unless you're able to live with noticeable downtime (not really practical for customer-facing online presences with significant branding / marketing concerns), this is something businesses have to be worried about. There are also other players like Akamai but they're more about leveraging their CDN infrastructure. If you don't mind hosting your content outside of your perimeter and in a place like Cloudflare, then that's another option.

I also saw a bunch of other small players scattered around that provide DDoS mitigation services, but they seemed very low-end. You could also buy equipment from the likes of Arbor and others, but those aren't cheap either. The bigger mitigation providers use a combination of these commercial solutions as well as their own proprietary secret sauce and leverage geographically-diverse cleaning centers to handle significant (hundreds of Gigs) bandwidth.

But as I said, they come with a hefty price tag.

Eildor

docrice wrote: »

Unfortunately DDoS isn't all that simple to deal with even if you're a large entity. DDoS comes in many forms and flooding is only one of them. If you have a 10 Gig pipe to your provider and you get flooded with 50 Gigs per second, you're still not going to be able to handle that. There are also slow-and-under-the-cover type of attacks that work more at the app layer. Once your firewalls, IPS sensors, and other inline devices get pushed to overload, things just don't work so well.

There's a reason why players like Verisign, Prolexic, and Neustar are in business. I looked into this issue a while back and unless you're able to live with noticeable downtime (not really practical for customer-facing online presences with significant branding / marketing concerns), this is something businesses have to be worried about. There are also other players like Akamai but they're more about leveraging their CDN infrastructure. If you don't mind hosting your content outside of your perimeter and in a place like Cloudflare, then that's another option.

I also saw a bunch of other small players scattered around that provide DDoS mitigation services, but they seemed very low-end. You could also buy equipment from the likes of Arbor and others, but those aren't cheap either. The bigger mitigation providers use a combination of these commercial solutions as well as their own proprietary secret sauce and leverage geographically-diverse cleaning centers to handle significant (hundreds of Gigs) bandwidth.

But as I said, they come with a hefty price tag.

That's why filtering the offending traffic before it enters into the network is important -- right? Which RTBH seems to be able to achieve. If you could automate that process by monitoring traffic patterns etc then that would be great.

Surely paying another company and trusting them with DDoS mitigation isn't the way forward. I don't want to sound like a conspiracy theorist but I wonder if these organisation perform DDoS attacks themselves to make the problem even bigger.

docrice

The question then becomes how do you identify what's an attack and what's not? What if your customer base comes from just about everywhere? Even if the only three services on your network are DNS, HTTP, and HTTPS, how do you distinguish good traffic from evil? Unless you run a massive infrastructure as a business (like Amazon, and I believe they were able to withstand attacks in the past just because of the enormous resources they control), it's unlikely you'll be able to figure out what's a bot node verses a live customer trying to click on your website. At the border, all you'll see are TCP SYNs, and perhaps a lot of other TCP ACKs that may or may not be part of a real established session.

On the other hand, if your network is only involved in select communication with specific partners that are well-known beforehand, it gets much easier (unless spoofable protocols like UDP and ICMP are involved).

DevilWAH

Eildor wrote: »

It's not going to make a difference if the attack is greater than the bandwidth you have though, is it? The DNS server is going to go down. Might as well just complete the DDoS attack and shut it down yourself?

Nope, even if the attack is greater than the link then say 1 in 500 packets is legit, logic say some will get through to the DNS server, even is the link is dropping 90% of traffic. setting on the DNS server and inbond link can prevent he DNS server crashing, so while you DNS service might be slow and require retries you can prevent a total fail.

It is also possible using BGP to prevent incoming connections from a specific Autonomous system or community. So for example if you saw all the traffic comming from "china" you could stop advertising your networks to the BGP Automounus systems and stop them being able to send traffic

Eildor

docrice wrote: »

The question then becomes how do you identify what's an attack and what's not? What if your customer base comes from just about everywhere?

But DDoS mitigation providers have that same problem, right?

DevilWAH

docrice wrote: »

On the other hand, if your network is only involved in select communication with specific partners that are well-known beforehand, it gets much easier (unless spoofable protocols like UDP and ICMP are involved).

in terms of splitting your traffic between duel links this is what I was thinking, where you use one link for known good traffic and the second to deal with the DoS attack.

Mitigation comes in two forms, block all possible DoS traffic, which you accept contains good traffic to and black hole it. This you can do with BGP and other routing protocols. but as you have mentioned it is not a perfect solution. With some thing like BGP though you can keep the links open to some known good traffic with out the use of a third party.

Or you can filter every bit of traffic coming in to check if it is good or bad. in this case you need an infrastructure that can deal with the DoS attack as though it is normal traffic flow. So like docrice says if your big enough like Google or Amazon with enough bandwidth and horse power you can move services around and leave services unaffected. Or you can pay a enternal company to do this for youm, when in the case of an outbreak you direct the traffic to them and they clean it up for you.

I was not suggesting BGP is the perfect solution, only that can can mitigate some symptoms of a DoS attack, and prevent it using the bandwidth of your links .

DevilWAH

Eildor wrote: »

But DDoS mitigation providers have that same problem, right?

Yes but they have the computing resources and the technical expertise to analyse the traffic quickly and apply rules to mitigate it. They work by assuming that they might be paid to protect 500 companies, but only one or two will be under attack at any one time. So in simple terms they might have data center full of DNS servers that they place in front of your company when you get hit by an attack. So they spread the DoDS attack over say 5000 DNS servers to insure that you services are still avalible while they analyse the attack and write rules to black the indivual machines carrying it out. then apply these rules and move the DNS service back to your own devices.

Eildor

DevilWAH wrote: »

Nope, even if the attack is greater than the link then say 1 in 500 packets is legit, logic say some will get through to the DNS server, even is the link is dropping 90% of traffic. setting on the DNS server and inbond link can prevent he DNS server crashing, so while you DNS service might be slow and require retries you can prevent a total fail.

So what you're saying is you filter the traffic inside the network, before it gets to the DNS server, right? But whether the DNS server has sufficient resources or not doesn't matter, because the link to ISP A is already completely saturated -- no?

Eildor

DevilWAH wrote: »

Yes but they have the computing resources and the technical expertise to analyse the traffic quickly and apply rules to mitigate it. They work by assuming that they might be paid to protect 500 companies, but only one or two will be under attack at any one time. So in simple terms they might have data center full of DNS servers that they place in front of your company when you get hit by an attack. So they spread the DoDS attack over say 5000 DNS servers to insure that you services are still avalible while they analyse the attack and write rules to black the indivual machines carrying it out. then apply these rules and move the DNS service back to your own devices.

I don't know... I recall reading about a $10,000 per month contract for DDoS mitigation... for that price you could hire a group of people to analyse your traffic for you and work with the ISP to mitigate attacks.

Out of curiosity, how much does a 100 Gbps internet connection cost?

docrice

i think in the absence of resources (budget to outsource the problem to a specialist), I think you'd need a runbook with a multitude of approaches. BGP, redirection of services to another net block (assuming you have a standby or can build one up quickly, although DDoSers will chase you), ACLs at the border, and working with your provider are probably the best bets. I guess if you're desperate, you could also throw up your services / content in a place like the Amazon cloud temporarily, assuming that's technically and legally feasible.

But I think preparation is a big key here. The fact that you're thinking about it now is crucial and you can start planning a what-to-do. When disaster strikes, at least you'll have some kind of formal plan in place that's signed off by management.

docrice

Eildor wrote: »

I don't know... I recall reading about a $10,000 per month contract for DDoS mitigation... for that price you could hire a group of people to analyse your traffic for you and work with the ISP to mitigate attacks.

Out of curiosity, how much does a 100 Gbps internet connection cost?

10k / month is the rough ballpark for some DDoS mitigation providers. I've seen higher. Could be more or less depending on the number of netblocks you're protecting and if you're also using their border monitoring service (SNMP / NetFlow sampling from their SOC).

100 Gbps will cost you a LOT. I don't have numbers off the top of my head, but it also depends which datacenter / provider / part of the country / world you're talking about. Some locations are much more expensive than others due to the concentrated hub peering of providers, etc..

DevilWAH

I am saying you devide you traffic in to site in to known good services (VPN to remote branches for example) and unknown (if its a DNS DoS attack all DNS traffic)

you then insure the known good traffic always gets though, and deal with the DoS traffic as a separate issue.

This way while you may lose part of your services, (in this case the DNS). your other business services continue to run as normal.

So yes the DoS traffic in this case is still coming in to your origination over the link, but it is only affecting a single service and not saturating the link and causing other outages. This then gives you some breathing space to analyse the DoS attack, either mitigating it to a 3rd party, or in house. As I said if I see all the traffc coming from "china" I can make my company drop of the internet as far as china is concerned. If they don't have a route across the internet to me then they can send me traffic and can't saturate my links. OK I lose all of china but this is better than the entire world.

Maybe then I would then request a rate limit put on to DNS traffic from China, and advertise my self to them again. Now the wide world has a normal service and China has a reduced service.

mitigating and sorting out a DoS attack is not a one stop quick fix

DevilWAH

Eildor wrote: »

Out of curiosity, how much does a 100 Gbps internet connection cost?

Let me say 100mb links in the UK will cost you upwards of £25K per year for a good buisness grace line with support.
a 1Gbps and you are looking north of £100K
100Gbps... you better be rich

Eildor

Security sounds a lot more interesting than routing and switching.

DevilWAH

Eildor wrote: »

Security sounds a lot more interesting than routing and switching.

Security, its just routing and switching on steroids

is true the same logic is true for all of networking (and computing)

with switching you read the mac address and make a choice what to do with it.
Routing you do the same with the IP address.

Security you just use a lot more thing to class the traffic and have lots more choices of what to do with it.

In myview good security is a progression from switching and routing, it is not a separate field.

Eildor

DevilWAH wrote: »

I am saying you devide you traffic in to site in to known good services (VPN to remote branches for example) and unknown (if its a DNS DoS attack all DNS traffic)

you then insure the known good traffic always gets though, and deal with the DoS traffic as a separate issue.

This way while you may lose part of your services, (in this case the DNS). your other business services continue to run as normal.

So yes the DoS traffic in this case is still coming in to your origination over the link, but it is only affecting a single service and not saturating the link and causing other outages. This then gives you some breathing space to analyse the DoS attack, either mitigating it to a 3rd party, or in house. As I said if I see all the traffc coming from "china" I can make my company drop of the internet as far as china is concerned. If they don't have a route across the internet to me then they can send me traffic and can't saturate my links. OK I lose all of china but this is better than the entire world.

Maybe then I would then request a rate limit put on to DNS traffic from China, and advertise my self to them again. Now the wide world has a normal service and China has a reduced service.

mitigating and sorting out a DoS attack is not a one stop quick fix

I see what you're saying; yes you are being attacked and your DNS server is most likely going to go down, but at least other services are still up and running while you put together a fix. I guess it depends, if you have 2 ISP's then sure you could make use of it that... otherwise just take the server down and analyse the traffic and put in place the fix.

Eildor

DevilWAH wrote: »

Security, its just routing and switching on steroids

is true the same logic is true for all of networking (and computing)

with switching you read the mac address and make a choice what to do with it.
Routing you do the same with the IP address.

Security you just use a lot more thing to class the traffic and have lots more choices of what to do with it.

In myview good security is a progression from switching and routing, it is not a separate field.

I'd love to learn more about security, but I think I'll leave a lot of it until I have some real world experience. I graduate this year, and I doubt I'd be doing much security wherever I get a job until I have some experience under my belt. I wouldn't give a newbie a security job

DevilWAH

Eildor wrote: »

I'd love to learn more about security, but I think I'll leave a lot of it until I have some real world experience. I graduate this year, and I doubt I'd be doing much security wherever I get a job until I have some experience under my belt. I wouldn't give a newbie a security job

Just make sure when you get a job to sit near the security guy and show an interest. It should not take years to get in to the area you want, showing willingness to learn and an interest in a subject, is can be surprising how quickly you can end up doing the things you enjoy.

Eildor

DevilWAH wrote: »

Just make sure when you get a job to sit near the security guy and show an interest. It should not take years to get in to the area you want, showing willingness to learn and an interest in a subject, is can be surprising how quickly you can end up doing the things you enjoy.

That's if I can get a job. I don't see a great deal of positions for graduates without experience in networking related jobs. Even the low paying NOC jobs with 12 hour shifts want experience

headshot

A pretty good DEFCON presentation on the subject matter at hand.

https://www.youtube.com/watch?v=1EAnjZqXK9E

nerdydad

Eildor wrote: »

That's if I can get a job. I don't see a great deal of positions for graduates without experience in networking related jobs. Even the low paying NOC jobs with 12 hour shifts want experience

I found that they ask for experience, but there seems to be a shortage of networking people. Apply for the positions and knock their socks off in the interview.

Eildor

nerdydad wrote: »

I found that they ask for experience, but there seems to be a shortage of networking people. Apply for the positions and knock their socks off in the interview.

I just worry a lot; but I'm hopeful someone will see the potential I have.