Has anyone implemented network segmentation?

MitM

I was wondering if anyone on the site has successfully implemented network segmentation, to prevent lateral movement from client machines? If so, how did you go about it?

To be more specific, I'm referring preventing lateral movement for clients on the same subnet. I thought of using Private VLANs, but that's not an option, since I make use of the voice vlan. An interface on a cisco switch cannot be set a private vlan and use voice vlan.

iBrokeIT

If you can't implement private vlans or have ISE, why not make use of the host based firewall to prevent lateral movement from your users? Also put your admins on their own vlan and block administrative access to the server vlans from the user vlan.

dave330i

NSX micro-Segmentation.

jamthat

Agreed with the above comment (iBroke's - nsx would speak to the major overhaul portion below, for most..dave replied too quickly!). If it was something you had to do today with what you already had as opposed to bringing in a brand new solution(s) and majorly overhauling, local firewall policies will be your best bet (and probably most effective either way). Reason being - there should be a limited number of systems that need to directly interface with your client systems internally (e.g. initiate traffic to). Every environment will be different, but this is probably something you could start testing and rolling out pretty quickly.

dave330i

Local firewall w/out central management doesn't scale.

jamthat

dave330i wrote: »

Local firewall w/out central management doesn't scale.

Agreed, but in my experience most environments these days do have the ability to centrally manage that function so my comment was made under that assumption

iBrokeIT

dave330i wrote: »

Local firewall w/out central management doesn't scale.

Neither does $5k per socket for NSX for most people's budgets.

Also NSX doesn't address lateral movement between client machines.

MitM

iBrokeIT wrote: »

If you can't implement private vlans or have ISE, why not make use of the host based firewall to prevent lateral movement from your users? Also put your admins on their own vlan and block administrative access to the server vlans from the user vlan.

yeah private vlans are out, but that would be easy. I don't have ISE currently, but possibly in the future. I was thinking of using windows firewall, since that's the only host based firewall I have. Easy enough to manage through group policy. If I did have ISE, would it be beneficial to utilize it instead of host based fw?

I do have some cisco jabber users that use the client on windows OS. I'm no voice guy, but I believe I'd have to poke some holes for that communication


NSX is also possibility in the future for our data center, but as iBroke mentioned, it won't help with this.

DZA_

dave330i wrote: »

NSX micro-Segmentation.

We have this running at our organization. I am not sure whether its due to poor implementation or whether it's a great product but damn does it ever cause a lot of problem's getting projects done due to the granularity/complexity involved. When you start to work with NSX Micro-Segmentation + your traditional firewalls it's like dealing with 4 dimensions!

Cheers,

iBrokeIT

MitM wrote: »

I was thinking of using windows firewall, since that's the only host based firewall I have. Easy enough to manage through group policy. If I did have ISE, would it be beneficial to utilize it instead of host based fw?

I do have some cisco jabber users that use the client on windows OS. I'm no voice guy, but I believe I'd have to poke some holes for that communication

There's no reason for you to not enable it. Yes, you can allow apps or ports through the windows firewall. You should use both ISE and windows firewall if you have both. Windows firewall can also protect your users off prem.

gespenstern

I'm architecting a project that is currently being rolled out... For >1 year and is expected to last for ~2 more years. Very, very problematic.

dave330i

iBrokeIT wrote: »

Neither does $5k per socket for NSX for most people's budgets.

Also NSX doesn't address lateral movement between client machines.

A single Distributed Firewall rule can block all layer 2 communications, so unless OP's workload is all physical, it will work. Layer 2 segmentation is 1 of the major use case for NSX.

MitM

gespenstern wrote: »

I'm architecting a project that is currently being rolled out... For >1 year and is expected to last for ~2 more years. Very, very problematic.

What exactly is problematic? Do you mind giving details?

MitM

dave330i wrote: »

A single Distributed Firewall rule can block all layer 2 communications, so unless OP's workload is all physical, it will work. Layer 2 segmentation is 1 of the major use case for NSX.

Yes, all physical. I’m referring to preventing lateral movement between end user client machines.

For servers, it would be nsx in the future

dave330i

MitM wrote: »

Yes, all physical. I’m referring to preventing lateral movement between end user client machines.

For servers, it would be nsx in the future

Can't help you if it's physical to physical. If you were using VDI, then NSX will stop client to client traffic (Implementing for a client atm). NSX is 2 products in 1. It has a clunky networking piece and a slick Firewall piece.

thomas_

The only that comes to mind for messing with intra-vlan traffic are VACLs, DHCP snooping, and dynamic ARP inspection. I don’t think VACLs will do what you are looking to do. If it does do it, then my gut feeling is that it would get very messy and not be scaleable. The only way I see VACLs even possibly working is matching the mac address to the port it’s found on and forwarding traffic out that port and denying all other traffic to that port and doing that for every port. I haven’t messed around with it, so I’m not even certain that’s an option.

DHCP snooping and dynamic arp inspection aren’t really what you’re looking for, but they do provide a means to reduce the type of funny business an attacker could do with a compromised machine.

gespenstern

MitM wrote: »

What exactly is problematic? Do you mind giving details?

The main problem is a lack of power and support among the enterprise. The risk this project is supposed to address is "WannaCry" or "NotPetya" situation (check with, for example, Maersk or Merck each losing from half a billion up to a billion because of this crap), which is, basically, a black swan type of event. Before WannaCry there were no worms of this scale for almost a decade. And everyone knows that and, besides CISO, nobody is willing to assign much resources for this project and this needs a ton of collaboration. I mean, they agree theoretically. CIO, top management, directors. Theoretically. But in practice...

Another problem is "noboby knows anything". Imagine a huge network spanning the whole damn world that was historically flat or close to that. Not a single application owner was able to provide a concise and precise list of communication channels their app utilizes. They don't know it and they don't care (until you break their application, then the CISO has a hard time justifying the need to get this project completed). Plus previous paragraph.

So you eventually have to resort to some automated tools that watch the traffic (Tetration, Tufin, etc) and then suggest the firewall rules. The problem: they cannot distinguish legit traffic that should be there from various noise and employees doing stuff that they aren't supposed to, various communications between long dead (on paper) servers and devices nobody can even tell you where they are physically located, LOL as they aren't supposed to exist by that moment. You have to do it yourself.

Then, there's this problem of real risks. You know that worms historically exploit either SMB or RPC. The problem: they are needed for a) inter-domain controller comms b) exchange comms c) certificate management (that's the big 3, there are others as well). Therefore, you have to allow exceptions. Which defeats the purpose of having the project in the first place, as if a patient zero launches a worm in Ukraine it will infect first their DCs or CAs or Exchange and spread from there to our DCs or CAs or Exchange and then to the rest of the company. Some things can be alleviated, like, certificate management offloaded to SCCM or Airwatch, but the problem is, each such subproject is huge by itself and affects the whole enterprise. Other things you cannot just block, so at this point you start thinking that the effort is futile and maybe you want to design a kill switch so whenever you feel like there's a worm spreading you just push the button and shut down all comms... Nobody is going to like it if it's a false positive, needless to say. Besides, the worm spread time is ~10 mins, you probably won't have a chance to push the button...

Then, no matter how hard you work at identifying the major communication flows, on day X when you cut off a branch or a certain type of traffic, there will be some obscure communication path that everyone forgot about and that happens once a month or so and yet business-critical. So critical that you wake up to calls at night when you least expect it from your bosses bosses boss who in no ambiguous terms commands you to revert the changes that were happily in place by that moment for 3.5 weeks. Eventually you become the most hated person in the whole company and for what?

I tell you, this network segmentation thing is probably the worst project I've been to in my whole life.

PS That all said, it's doable. The only thing I'm not sure about is if it's worth the effort... I.e. if an annualized risk of NotPetya ruining everything is 0.01% (once in 5 years, 1 out of 2000 huge world companies) then it might not meet the cost/benefit analysis

matt333

Probably the simplest is to use a VACL or something along those lines where ever the SVI's are. Maybe a combination of VACL's and Dot1x with different vlans for corp traffic. Patching of the hosts I'd think would be the best way to prevent anything bad to happen to the hosts. Blocking the right user traffic is pretty much impossible as there are so many weird ports being used on a regular basis.

iBrokeIT

gespenstern wrote: »

Then, no matter how hard you work at identifying the major communication flows, on day X when you cut off a branch or a certain type of traffic, there will be some obscure communication path that everyone forgot about and that happens once a month or so and yet business-critical. So critical that you wake up to calls at night when you least expect it from your bosses bosses boss who in no ambiguous terms commands you to revert the changes that were happily in place by that moment for 3.5 weeks. Eventually you become the most hated person in the whole company and for what?

I tell you, this network segmentation thing is probably the worst project I've been to in my whole life.

PS That all said, it's doable. The only thing I'm not sure about is if it's worth the effort... I.e. if an annualized risk of NotPetya ruining everything is 0.01% (once in 5 years, 1 out of 2000 huge world companies) then it might not meet the cost/benefit analysis

Yikes! As with any project of that scale, leadership support is critical and so is setting their expectations about things that will pop up like this. The decision to roll back instead of fail forward with an allow rule for that one system was a poor decision on their part. With management like that, I don't envy you in the least bit.

fitzlopez

gespenstern wrote: »

Another problem is "noboby knows anything". Imagine a huge network spanning the whole damn world that was historically flat or close to that. Not a single application owner was able to provide a concise and precise list of communication channels their app utilizes. They don't know it and they don't care (until you break their application, then the CISO has a hard time justifying the need to get this project completed).

I hear you, in one of the companies I consult, what we've started to do is build up the TO BE in our perfect happy network. Then put a cut off date where any NEW service or server starts with the hardened and segmented politics. That way the new stuff is mapped, documented, segmented and protected, with an assigned responsible who doesn't see his service online unless he complies. This also applies to renewals of hardware and OS reinstalls. The effort and cost are incremental so management didn't mind as much. We plan on doing the same with ITIL changes but haven't worked out the kinks. We're still fighting the good fight with the legacy stuff but it's going to take years and the ROI is a real hard sell. I figure most servers will have been renewed by the time we finish.

Basically we made 2 projects the new stuff project and the fix old stuff project, in that order.


PS. With NotPetya and such we announced that we were going to disable SMBv1 and that we were going to block the needed ports from the extranet to the intranet, then followed up. Patched all servers for the SMB exploits and started other cleaning up measures. It was horrible, getting someone to fix anything is a pain that's why you have to start right.

gespenstern

fitzlopez wrote: »

PS. With NotPetya and such we announced that we were going to disable SMBv1 and that we were going to block the needed ports from the extranet to the intranet, then followed up. Patched all servers for the SMB exploits and started other cleaning up measures. It was horrible, getting someone to fix anything is a pain that's why you have to start right.

Yeah, that's nice, we also did that. Plus, ensured offline backup of the whole enterprise to make sure that backups don't get encrypted along with everything else like what is seems to have happened to Maersk. Plus, introduced some controls preventing memory scraping of lsass.exe using 3rd party privilege management tools for windows endpoints, MS recommended patches/registry changes, etc.

Also hardened our patch management to make sure we are under 30 days for critical patches for EVERYTHING.

The problem though is it's all generals preparing for the past war. It's not likely that the next worm will rely on SMBv1 etc.

And yeah, ROI of all that is a HARD sell.

MitM

@gespenstern, thanks for the details. It surely is an undertaking. I figured I'd start with segmenting end user client machines as they are straight forward (for the most part). There's a lot of clean up that would need to be done, that's for sure.

ccie14023

Here is another option:

https://www.cisco.com/c/en/us/solutions/enterprise-networks/software-defined-access/index.html