Help with "Production" server

I am currently having a network issue with one of our licensing "production" server, we've tried different steps to resolve the issue with no success.

Steps Taken:

- Replaced Dell Server MOBO
- Consulted w/switch Manufacturer Force 10 (No problem w/switch)
- Rolled Back Nic drivers /uninstall drivers
- Teamed and unteamed interfaces (Bridge)
- Changed Static IP address
- Used a different switch

Facts:

- Arp tables on client include all host on network segmnent, except gateway
- Statically assign a mac address to gateway does not help
- ARP tables on switch identify correct server interface
- Server can ping itself (10.10.0.70), but cannot ping other devices on local LAN

Thanks in advance!!!

Find more posts tagged with

Free for TechExams community: Cybersecurity salary guide

Compare cert salaries and plan your next career move

Button

Comments

jbaello

Ahriakin wrote:

Is the Windows Firewall/ICS Service still off? Have you checked the TCP/IP Filter section on the RCP/IP-Advanced-Options properties page?
Have you scanned for rootkits in Safe Mode + Networking?
Check your IPSEC Policies and make sure the server is not set to require it for all connections.

Firewall is off, I gotta check ICS, I gotta check TCP/IP filtering, I believe it was disabled, hmm how come some TCP/IP configuration advanced tab shows Filter and some doesn't? this rings a bell I forgot.

I'm not familiar with rootkits in Safe Mode + Networking, we don't have an IPSEC Policy enabled either on GPO or locally.

I started backing up system state for last resort, and backed up everything off an external HD, I'll keep you guys posted.

marco71

jbaello wrote:

I am currently having a network issue with one of our licensing "production" server, we've tried different steps to resolve the issue with no success.
...
Facts:

- Arp tables on client include all host on network segmnent, except gateway...

That does mean nothing (gateway could filter ping/arp request)

jbaello wrote:

Facts:
...
- ARP tables on switch identify correct server interface

That means layer 2 connectivity is fine

jbaello wrote:

...
- Server can ping itself (10.10.0.70), but cannot ping other devices on local LAN
Thanks in advance!!!

That means TCP/IP stack is also fine

jbaello wrote:

Server has two NIC and both is plugged/enabled to the switch.

If IP Forwarding is somehow activated (by a local service), that could cause a network loop, which is bad

So, the remaining possible causes are software (firewall/filter)

jbaello wrote:

Active Routes:
Network Destination Netmask Gateway Interface Metric
0.0.0.0 0.0.0.0 10.10.0.65 10.10.0.124 10
0.0.0.0 0.0.0.0 10.10.0.65 10.10.0.123 10
10.10.0.64 255.255.255.192 10.10.0.123 10.10.0.123 10
10.10.0.64 255.255.255.192 10.10.0.124 10.10.0.124 10
10.10.0.123 255.255.255.255 127.0.0.1 127.0.0.1 10
10.10.0.124 255.255.255.255 127.0.0.1 127.0.0.1 10
10.255.255.255 255.255.255.255 10.10.0.123 10.10.0.123 10
10.255.255.255 255.255.255.255 10.10.0.124 10.10.0.124 10
127.0.0.0 255.0.0.0 127.0.0.1 127.0.0.1 1
224.0.0.0 240.0.0.0 10.10.0.123 10.10.0.123 10
224.0.0.0 240.0.0.0 10.10.0.124 10.10.0.124 10
255.255.255.255 255.255.255.255 10.10.0.123 10.10.0.123 1
255.255.255.255 255.255.255.255 10.10.0.124 10.10.0.124 1
Default Gateway: 10.10.0.65

I'm wondering how local TCP/IP packets whould traverse the network, from which IP address? 10.10.0.123 or 10.10.0.124? Metrics are equals :P

astorrs

marco71 wrote:

I'm wondering how local TCP/IP packets whould traverse the network, from which IP address? 10.10.0.123 or 10.10.0.124? Metrics are equals :P

Thats why i was suggesting he disable/remove one of the nics, just to simplify things. gotta be something running in normal mode not running in safe mode w/ networking, you'll need to compare between the two.

d4nmf

I may have missed a trick here, but have you tried:

netsh int ip reset c:\resetlog.txt

Where c:\resetlog.txt is the location of the log file?

astorrs

Yeah he did, no go.

I still vote for a comparison of the processes/services/drivers running in safe mode w/ networking vs. normal mode and going through a process of elimination.

blargoe

When you ping the gateway or other hosts, what is the actual message that is returned to you?

Request timed out?
Destination net unreachable?
Destination host unreachable?
Host not found?

snadam

I know I have said it already, but it probably wouldn't hurt to throw a sniffer on that NIC and see where and when packets are dropping exactly. DNS and netBIOS working properly? Im just throwing out all options.

jbaello

blargoe wrote:

When you ping the gateway or other hosts, what is the actual message that is returned to you?

Request timed out?
Destination net unreachable?
Destination host unreachable?
Host not found?

Destination Host Unreachable, when I tried to ping the Gateway and DNS server.

jbaello

astorrs wrote:

marco71 wrote:

I'm wondering how local TCP/IP packets whould traverse the network, from which IP address? 10.10.0.123 or 10.10.0.124? Metrics are equals :P

Thats why i was suggesting he disable/remove one of the nics, just to simplify things. gotta be something running in normal mode not running in safe mode w/ networking, you'll need to compare between the two.

2nd NIC has been disabled.

blargoe

Yeah, that's routing... I believe that means the packet made it to the gateway (which, for your purpose, your default route to the host you're trying to ping, the network interface the server according to your route print) and it didn't know how to get to the destination (the interface of the router).

I agree, it has to be software, i.e., Windows.

When you replaced the motherboard, did it redetect the hardware when you restarted windows and reinstall the devices again?

Are there any devices in device manager that are in a warning state or are disabled that shouldn't be?

Have you tried to delete the Broadcom network adapters from Device Manager and reboot?

Maybe there are old entries for the Broadcom on the old motherboard in Device Manager the need to be deleted (show hidden devices should show this)?

Have you tried a standalone PCI network card, just as a test?

jbaello

When you replaced the motherboard, did it redetect the hardware when you restarted windows and reinstall the devices again? Yes it did, the old mobo was not the issue, the network admin was just stubborn.

Are there any devices in device manager that are in a warning state or are disabled that shouldn't be? Not that I know off...

Have you tried to delete the Broadcom network adapters from Device Manager and reboot? Yes many times...

Maybe there are old entries for the Broadcom on the old motherboard in Device Manager the need to be deleted (show hidden devices should show this)? old mobo has been put back...

Have you tried a standalone PCI network card, just as a test?[/quote] Not yet we are looking into purchasing a dual nic interface...

Ahriakin

Routing doesn't come into it when you are pinging hosts on your configured subnet though, the default-gateway (or static/dynamic routes) only come into play when you are trying to contact logically external hosts ('logically' since you can have a multinet with multiple subnets on one switch fabric but the hosts will still need routing as they believe they are on separate networks from your subnet mask setup). Anyway it has to be some sort of packet filtering going on locally even if it's caused by a software glitch or malware (there are some that will happily hose your connectivity when you try to remove them). One thing I would ask though is are you sure about the configurations, just because some policies/configuration shouldn't be used doesn't mean someone didn't play around with it anyway.

jbaello

I decided to move the server "computer account" back into "computer container" anyway the Director finally decided to give Microsoft a Buzz this is a paid technical support, we'll see the outcome.

astorrs

Please let us know what fixed it in the end (assuming PSS can help you).

snadam

astorrs wrote:

Please let us know what fixed it in the end (assuming PSS can help you).

+1 curious to know the outcome as well.

jbaello

Ahriakin wrote:

Is the Windows Firewall/ICS Service still off? Have you checked the TCP/IP Filter section on the RCP/IP-Advanced-Options properties page?
Have you scanned for rootkits in Safe Mode + Networking?
Check your IPSEC Policies and make sure the server is not set to require it for all connections.

There was no IPsec policy on the GPO or configured on the box, but IPsec was infact bugged, it's working...

astorrs

That outcome is not surprising, things were definitely looking along those lines. I'm glad you were able to get it resolved.

jbaello

We had to run reports on safe mode vs. normal mode using a utility found here, a tool called MSRPT_Network...

http://www.microsoft.com/downloads/details.aspx?familyid=cebf3c7c-7ca5-408f-88b7-f9c79b7306c0&displaylang=en

I feel like an idiot, for that I'll study more...

snadam

astorrs wrote:

That outcome is not surprising, things were definitely looking along those lines. I'm glad you were able to get it resolved.

im also glad you got it resolved, but i find it weird that IPSec was the issue even though it wasnt enabled. But hey, you learn something new every day.

undomiel

Useful tool! Thanks for the link, something else to add to my arsenal.

jbaello

jbaello wrote:

I decided to move the server "computer account" back into "computer container" anyway the Director finally decided to give Microsoft a Buzz this is a paid technical support, we'll see the outcome.

I was starting to suspect an IPsec configuration, it was like it's almost infront of my nose, but just cannot see it...

Ahriakin was right that something along the IPsec/GPO/configuration could have been modified, it was infact changed, but it was not changed cause someone did, it was bugged, Microsoft confirmed a bug with IPsec.

jbaello

So simply disabling the IPsec service ultimately fixed the problem... This is a no brainer solution that I could have figured out, oh well time to move on, and just to make sure to keep an eye with IPsec from now on...

Something new everyday...

blargoe

No need to feel like an idiot, just chalk this up to weirdness. And hopefully, MS didn't charge you for the call since it was a bug.

Now that you mention the solution, it seems like I might have seen this before one time. I'll certainly remember this in the future if it happens to me.

Thanks for sharing.

Ahriakin

It only occurred to me as it was a suggested Firewalling method on the 2K3 security exams. Anyway, fun mystery, glad it's okay now