Options
Ten Gigabit Ethernet issue...
Monkerz
Member Posts: 842
in CCNA & CCENT
I was called in tonight to resolve an issue at one of the facilities I work at. I couldn't figure out what changed so I decided to implement a work around until I can look into it tomorrow morning. I'd like to explain the setup and issue and get your opinions, if you don't mind.
The issue was called in, "People is the new addition cannot access anything on the network."
This plant is about 475,000 square foot. The west end of the plant houses our MDF1, we are running a 4507 as a distribution core. 1 year ago we added a 155,000 square foot addition to the east end of the plant, we created MDF2 in this location which houses a 4503 as a second distribution core. We have setup a 10Gbps backbone between cores with a 1Gbps redundant link. The 10gig backbone spans 998 feet and connects using MMF and 10gig-LX4 lasers. The 1gig spans the same distance and connects using MMF and 1gig SFPs.
When I arrive at the plant, I immediately when to my desk (in MDF1) and started pinging servers. All servers residing in my office were ok, however I could not ping anything in the east end of the plant. I checked the interfaces on the 4507 and found the 10gig interface down/down and showing 3 input errors and 3 CRC. I immediately went to MDF2 and checked the 4503. I found the 10gig interface error-disabled with no errors showing in the counters. I cleared the err-disable by shutting down the interface and bringing it back up. I issued a continuious ping to the 4507 from a server. All seemed well, no errors showing up. Then it started timing out, I checked the interface on the 4503 and found it in an up/down state. One minute passed and it was up/up again and pings were successful, then same thing happened. I rebooted both switches thinking someone plugged in something they shouldn't have, but when everything came back up it did the same thing. I finally swapped over the the 1gig backbone and all was well.
This link as ran fine for at least a year. No errors, nothing. Then all the sudden this. I am going to leave the link on 1gig until I can get back into the office in the morning, but I'd like anyone's input if possible. Our lone network engineer was unavailable at this time, so I took it upon myself to get everything working again, but I am still new at this.
Any thoughts? Thanks in advance.
The issue was called in, "People is the new addition cannot access anything on the network."
This plant is about 475,000 square foot. The west end of the plant houses our MDF1, we are running a 4507 as a distribution core. 1 year ago we added a 155,000 square foot addition to the east end of the plant, we created MDF2 in this location which houses a 4503 as a second distribution core. We have setup a 10Gbps backbone between cores with a 1Gbps redundant link. The 10gig backbone spans 998 feet and connects using MMF and 10gig-LX4 lasers. The 1gig spans the same distance and connects using MMF and 1gig SFPs.
When I arrive at the plant, I immediately when to my desk (in MDF1) and started pinging servers. All servers residing in my office were ok, however I could not ping anything in the east end of the plant. I checked the interfaces on the 4507 and found the 10gig interface down/down and showing 3 input errors and 3 CRC. I immediately went to MDF2 and checked the 4503. I found the 10gig interface error-disabled with no errors showing in the counters. I cleared the err-disable by shutting down the interface and bringing it back up. I issued a continuious ping to the 4507 from a server. All seemed well, no errors showing up. Then it started timing out, I checked the interface on the 4503 and found it in an up/down state. One minute passed and it was up/up again and pings were successful, then same thing happened. I rebooted both switches thinking someone plugged in something they shouldn't have, but when everything came back up it did the same thing. I finally swapped over the the 1gig backbone and all was well.
This link as ran fine for at least a year. No errors, nothing. Then all the sudden this. I am going to leave the link on 1gig until I can get back into the office in the morning, but I'd like anyone's input if possible. Our lone network engineer was unavailable at this time, so I took it upon myself to get everything working again, but I am still new at this.
Any thoughts? Thanks in advance.
Comments
-
OptionsMonkerz Member Posts: 842It's shows a link-flap error which caused the err-disable. Link-flap is caused by a loop?
-
Optionsfly351 Member Posts: 360We have setup a 10Gbps backbone between cores with a 1Gbps redundant link. The 10gig backbone spans 998 feet and connects using MMF and 10gig-LX4 lasers. The 1gig spans the same distance and connects using MMF and 1gig SFPs.
hmm probably more of a interface issue or the SFP is faulty.
link flap is when the interface keeps going up/down.CCNP :study: -
OptionsColbyG Member Posts: 1,264link flap is when the interface keeps going up/down.
If he's thinking of a loop, he's probably talking about the host/mac address flapping error. -
Optionsfly351 Member Posts: 360If he's thinking of a loop, he's probably talking about the host/mac address flapping error.CCNP :study:
-
Optionsjason_lunde Member Posts: 567Its most likely a faulty GBIC. Happens here every couple of months....unfortunatly its always the cisco branded 10GB gbics and not the 3rd party ones . Luckily we bundled our 10GB interfaces into portchannels so that if it happens the other automagically picks up the slack. If you have the cash and fiber available you might look into doing the same.
-
Optionsnetworker050184 Mod Posts: 11,962 ModI'd have the line and SFPs checked also.An expert is a man who has made all the mistakes which can be made.
-
Optionstiersten Member Posts: 4,505jason_lunde wrote: »Its most likely a faulty GBIC. Happens here every couple of months....unfortunatly its always the cisco branded 10GB gbics and not the 3rd party ones
-
OptionsMonkerz Member Posts: 842Looks like it was one of the LX4s that crapped out. My spare fixed the issue.
Jason, how long have you been running 10G? Whats your setup consist of? We had bad lasers 12 months ago, I didn't think it would be a reoccurring thing. Honestly one goes bad every few months? -
Optionsnetworker050184 Mod Posts: 11,962 ModWe haven't had any go out yet in our DC in over a years time. We are using 10G-SR and 10G-LX4 Cisco brand on 6708 cards and 3750Es.An expert is a man who has made all the mistakes which can be made.
-
OptionsMonkerz Member Posts: 842We were running SRs but we exceeded the range and had to swap to LX4s. If it comes down to it, I'll just run a 4G etherchannel. I would get yelled at if I spent $2,000 for a laser every few months to slap a bandaid on the backbone.
-
Optionstiersten Member Posts: 4,505You're slightly exceeding the distance for 10GBASE-LX4 if you're running it over 998 feet.
I've not had any unusual failures in any of the 10G optics here. They're generally reliable except if you're the fiber is too short and you don't have an attenuator. -
OptionsMonkerz Member Posts: 842You're slightly exceeding the distance for 10GBASE-LX4 if you're running it over 998 feet.
This is true, I believe we are over 15 feet or so, but I've seen where people run 25 to 50 feet over the recommended max.
I think we will look in to the third party idea. Can't hurt, huh?
Anyone use third party 10G transceivers? Can you suggest one you've used with great reliability? Thanks Guys. -
Optionstiersten Member Posts: 4,505This is true, I believe we are over 15 feet or so, but I've seen where people run 25 to 50 feet over the recommended max.I think we will look in to the third party idea. Can't hurt, huh?
Anyone use third party 10G transceivers? Can you suggest one you've used with great reliability? Thanks Guys. -
Optionsjason_lunde Member Posts: 567Looks like it was one of the LX4s that crapped out. My spare fixed the issue.
Jason, how long have you been running 10G? Whats your setup consist of? We had bad lasers 12 months ago, I didn't think it would be a reoccurring thing. Honestly one goes bad every few months?
Well we basically run 2x10GB links from our core to every distribution switch we have. Over the last 5 months we have had 2 go out; but I should add that the ones that have gone out have been in production for over a year or so...but it still hurts at those prices! I always try to get ones with pretty hefty warranty's, but they always seem to outlive those by a couple months. So honestly I am hoping that our recent failures are kind of "batch" issues, because recently they have all been Cisco branded units that we all got at the same time.