Help with "Production" server

jbaello

I am currently having a network issue with one of our licensing "production" server, we've tried different steps to resolve the issue with no success.

Steps Taken:

- Replaced Dell Server MOBO
- Consulted w/switch Manufacturer Force 10 (No problem w/switch)
- Rolled Back Nic drivers /uninstall drivers
- Teamed and unteamed interfaces (Bridge)
- Changed Static IP address
- Used a different switch

Facts:

- Arp tables on client include all host on network segmnent, except gateway
- Statically assign a mac address to gateway does not help
- ARP tables on switch identify correct server interface
- Server can ping itself (10.10.0.70), but cannot ping other devices on local LAN

Thanks in advance!!!

snadam

jbaello wrote:

I am currently having a network issue with one of our licensing "production" server, we've tried different steps to resolve the issue with no success.

Steps Taken:

- Replaced Dell Server MOBO
- Consulted w/switch Manufacturer Force 10 (No problem w/switch)
- Rolled Back Nic drivers /uninstall drivers
- Teamed and unteamed interfaces (Bridge)
- Changed Static IP address
- Used a different switch

Facts:

- Arp tables on client include all host on network segmnent, except gateway
- Statically assign a mac address to gateway does not help
- ARP tables on switch identify correct server interface
- Server can ping itself (10.10.0.70), but cannot ping other devices on local LAN

Thanks in advance!!!

Have you ran a sniffer on it yet to see where exactly the packets start dropping? just for fun, have you tried to configure it to have its address assigned via DHCP? Also have you thrown it on another network separate from the current one to see if the problems still arise?
also, have you tried installing another NIC to see if the problem still continues (although its a stretch and highly unlikely).

jbaello

I went ahead and boot a Win PE and it looks like network is working perfectly, this is a no brainer solution I suggested from one of our Network Admin who's been stucked working with this network issue for the last two days, now hardware is finally eliminated as the culprit, by the look of it the OS has some major issue.

Win PE network configurator is configured as a DHCP host, I'll try and set the OS network to DHCP since the Network Admin finally "thrown the towel" I have not used a sniffer yet at all...

snadam

jbaello wrote:

I went ahead and boot a Win PE and it looks like network is working perfectly, this is a no brainer solution I suggested from one of our Network Admin who's been stucked working with this network issue for the last two days, now hardware is finally eliminated as the culprit, by the look of it the OS has some major issue.

ahh okay. omit my last comments then

maybe TCP/IP is jacked up on this particular machine? have you tried uninstall/reinstall of tcp/ip?

jbaello

snadam wrote:

jbaello wrote:

I went ahead and boot a Win PE and it looks like network is working perfectly, this is a no brainer solution I suggested from one of our Network Admin who's been stucked working with this network issue for the last two days, now hardware is finally eliminated as the culprit, by the look of it the OS has some major issue.

ahh okay. omit my last comments then

maybe TCP/IP is jacked up on this particular machine? have you tried uninstall/reinstall of tcp/ip?

Hmmm Yummy I'll get all the credits as soon this works, I'll try this... but isn't this the same, when you uninstall the NIC driver it basically blew out TCP/IP as well?

snadam

jbaello wrote:

snadam wrote:

jbaello wrote:

I went ahead and boot a Win PE and it looks like network is working perfectly, this is a no brainer solution I suggested from one of our Network Admin who's been stucked working with this network issue for the last two days, now hardware is finally eliminated as the culprit, by the look of it the OS has some major issue.

ahh okay. omit my last comments then

maybe TCP/IP is jacked up on this particular machine? have you tried uninstall/reinstall of tcp/ip?

Hmmm Yummy I'll get all the credits as soon this works, I'll try this... but isn't this the same, when you uninstall the NIC driver it basically blew out TCP/IP as well?

I was always under the impression that TCP/IP isntall/uninstall was OS related. Then you just bind the protocols to the NICS themselves

check out this link from MS http://support.microsoft.com/kb/325356

EDIT: the link is related to DC's...hang on and let me find a better one....http://support.microsoft.com/kb/317518

you should be able to use netsh to 'reset' it

jbaello

No go... after running the netsh commands...

jbaello

When I assign the server to be a DHCP host it was getting an APIPA, our DHCP server is working fine, and able to lease out address just normally, I am trying to disable all 3rd party services and see if this works, any networker out there that can give me more procedures let me know...

astorrs

What OS/Service Pack is this? It definitely sounds like something in the IP stack is screwed up since layer 2 is fine, but you say you can ping yourself? I assume the NIC(s) are integrated on the motherboard (hence why you replaced it) but if you could confirm. How many NICs do you have in the server?

blargoe

Is the network connection set to autonegotiate the link duplex and speed?

jbaello

Started OS on "Safe Mode with Networking" network connection is working fine, it looks like the TCP/IP stack is intact, something else might be blocking network connectivity but which one? all services has already been disabled via "msconfig"

snadam

astorrs wrote:

What OS/Service Pack is this? It definitely sounds like something in the IP stack is screwed up since layer 2 is fine, but you say you can ping yourself? I assume the NIC(s) are integrated on the motherboard (hence why you replaced it) but if you could confirm. How many NICs do you have in the server?

my guess form this is that its server 2k3, but im not sure if its a DC or not. If the NICS are integrated, then we can almost rule out bad hardware, because he said the MOBO was replaced. Perhaps it IS the actual NIC drivers causing the issue? Are the NICs showing up in device manager? I have a dell server with integrated nics, and everytime you restart, the drivers are not detected and i have to go in and manually 'update' them and they instantly find them. i know its kind of off target, but I figured Id ask.

jbaello

Server has two NIC and both is plugged/enabled to the switch...

astorrs

Okay if it works in safe mode with networking then its not drivers. It would help to have confirmation on the O/S...

Do you have any software firewalls enabled? I would disable every service not essential to booting the server, reboot and see if networking works.

jbaello

astorrs wrote:

Okay if it works in safe mode with networking then its not drivers. It would help to have confirmation on the O/S...

Do you have any software firewalls enabled? I would disable every service not essential to booting the server, reboot and see if networking works.

all services has already been disabled via "msconfig"

I tried "netsh winsock reset" - about to see the result...

snadam

astorrs wrote:

Okay if it works in safe mode with networking then its not drivers. It would help to have confirmation on the O/S...

Do you have any software firewalls enabled? I would disable every service not essential to booting the server, reboot and see if networking works.

damn! I thought we were getting somewhere! I agree with astorrs, now that we have that bit o' info that its working in safe mode. Yay, trial and error time!!!
EDIT: man i am slow tonight, let us know how that works out.

jbaello

winsock reset is a no go... I will try to modify registry for winsock entry and delete it...

HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\Winsock
HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\Winsock2

astorrs

Are there any other networking protocols installed on the machine?

jbaello

Windows Server R2 Standard Edition Ver. 5.2 R2 (Build 3790 etc...) SP1

jbaello

astorrs wrote:

Are there any other networking protocols installed on the machine?

no it was bridged at the beginning but I blow out bridging and deleted bridge protocol as well on network connection.

astorrs

Can you double-check that the msconfig actually disabled all the additional services? (I've seen it miss some before because the vendor decided to mark them as system/boot).

jbaello

I'm contemplating about this, but would not blow out the server, since it's a high critical production box, I will try to install a standalone NIC and see d outcome, but for now I'm going to call it a day, I've already done a lot of mistake :P well the network admin did... lol

http://support.microsoft.com/kb/325356

jbaello

astorrs wrote:

Can you double-check that the msconfig actually disabled all the additional services? (I've seen it miss some before because the vendor decided to mark them as system/boot).

All 3rd party services is infact disabled and confirmed, on regular mode DHCP is now able to obtain an IP address but still network connectivity is not working.

astorrs

Wait let me understand this... you are getting a DHCP lease now? Even though you can't actually reach any other resources on the subnet.

jbaello

YES

astorrs

Can you paste the output from a "route print" and an "ipconfig /all"

jbaello

it's a very weird error, I think this is the 1st time I've encountered it, we are contemplating on just doing a setup/repair of windows installation, but first we are thinking of backing up everything...

jbaello

Microsoft Windows [Version 5.2.3790]
(C) Copyright 1985-2003 Microsoft Corp.

C:\Documents and Settings\Administrator>route print

IPv4 Route Table
===========================================================================
Interface List
0x1 ........................... MS TCP Loopback interface
0x2 ...00 15 c5 ea 4c b9 ...... Broadcom BCM5708C NetXtreme II GigE (NDIS VBD
ient) #2
0x3 ...00 15 c5 ea 4c bb ...... Broadcom BCM5708C NetXtreme II GigE (NDIS VBD
ient)
===========================================================================
===========================================================================
Active Routes:
Network Destination Netmask Gateway Interface Metric
0.0.0.0 0.0.0.0 10.10.0.65 10.10.0.124 10
0.0.0.0 0.0.0.0 10.10.0.65 10.10.0.123 10
10.10.0.64 255.255.255.192 10.10.0.123 10.10.0.123 10
10.10.0.64 255.255.255.192 10.10.0.124 10.10.0.124 10
10.10.0.123 255.255.255.255 127.0.0.1 127.0.0.1 10
10.10.0.124 255.255.255.255 127.0.0.1 127.0.0.1 10
10.255.255.255 255.255.255.255 10.10.0.123 10.10.0.123 10
10.255.255.255 255.255.255.255 10.10.0.124 10.10.0.124 10
127.0.0.0 255.0.0.0 127.0.0.1 127.0.0.1 1
224.0.0.0 240.0.0.0 10.10.0.123 10.10.0.123 10
224.0.0.0 240.0.0.0 10.10.0.124 10.10.0.124 10
255.255.255.255 255.255.255.255 10.10.0.123 10.10.0.123 1
255.255.255.255 255.255.255.255 10.10.0.124 10.10.0.124 1
Default Gateway: 10.10.0.65
===========================================================================
Persistent Routes:
None

C:\Documents and Settings\Administrator>
C:\Documents and Settings\Administrator>ipconfig /all

Windows IP Configuration

Host Name . . . . . . . . . . . . : apu
Primary Dns Suffix . . . . . . . : corp.rttusa.com
Node Type . . . . . . . . . . . . : Unknown
IP Routing Enabled. . . . . . . . : No
WINS Proxy Enabled. . . . . . . . : No
DNS Suffix Search List. . . . . . : corp.rttusa.com
rttusa.com

Ethernet adapter Local Area Connection 2:

Connection-specific DNS Suffix . : corp.rttusa.com
Description . . . . . . . . . . . : Broadcom BCM5708C NetXtreme II GigE (N
VBD Client) #2
Physical Address. . . . . . . . . : 00-15-C5-EA-4C-B9
DHCP Enabled. . . . . . . . . . . : Yes
Autoconfiguration Enabled . . . . : Yes
IP Address. . . . . . . . . . . . : 10.10.0.123
Subnet Mask . . . . . . . . . . . : 255.255.255.192
Default Gateway . . . . . . . . . : 10.10.0.65
DHCP Server . . . . . . . . . . . : 10.10.0.66
DNS Servers . . . . . . . . . . . : 10.10.0.66
10.10.0.67
Lease Obtained. . . . . . . . . . : Wednesday, May 28, 2008 7:30:44 PM
Lease Expires . . . . . . . . . . : Thursday, June 05, 2008 7:30:44 PM

Ethernet adapter Local Area Connection:

Connection-specific DNS Suffix . : corp.rttusa.com
Description . . . . . . . . . . . : Broadcom BCM5708C NetXtreme II GigE (N
VBD Client)
Physical Address. . . . . . . . . : 00-15-C5-EA-4C-BB
DHCP Enabled. . . . . . . . . . . : Yes
Autoconfiguration Enabled . . . . : Yes
IP Address. . . . . . . . . . . . : 10.10.0.124
Subnet Mask . . . . . . . . . . . : 255.255.255.192
Default Gateway . . . . . . . . . : 10.10.0.65
DHCP Server . . . . . . . . . . . : 10.10.0.66
DNS Servers . . . . . . . . . . . : 10.10.0.66
10.10.0.67
Lease Obtained. . . . . . . . . . : Wednesday, May 28, 2008 7:30:44 PM
Lease Expires . . . . . . . . . . : Thursday, June 05, 2008 7:30:44 PM

C:\Documents and Settings\Administrator> have fun!!!

Ahriakin

Is the Windows Firewall/ICS Service still off? Have you checked the TCP/IP Filter section on the RCP/IP-Advanced-Options properties page?
Have you scanned for rootkits in Safe Mode + Networking?
Check your IPSEC Policies and make sure the server is not set to require it for all connections.

astorrs

Ahriakin wrote:

Is the Windows Firewall/ICS Service still off? Have you checked the TCP/IP Filter section on the RCP/IP-Advanced-Options properties page?
Have you scanned for rootkits in Safe Mode + Networking?
Check your IPSEC Policies and make sure the server is not set to require it for all connections.

And disconnect one of the two NICs to simplify things.

snadam

jbaello wrote:

0x2 ...00 15 c5 ea 4c b9 ...... Broadcom BCM5708C NetXtreme II GigE (NDIS VBD
ient) #2
0x3 ...00 15 c5 ea 4c bb ...... Broadcom BCM5708C NetXtreme II GigE (NDIS VBD
ient)

yep, have the same NICs in one of my power edge's. Unfortunately I dont have the same problem as you, but at least you can feel a little bit better that they are causing anguish for at least another person


keep us posted tomorrow.