Ten Gigabit Ethernet issue...

I was called in tonight to resolve an issue at one of the facilities I work at. I couldn't figure out what changed so I decided to implement a work around until I can look into it tomorrow morning. I'd like to explain the setup and issue and get your opinions, if you don't mind.

The issue was called in, "People is the new addition cannot access anything on the network."

This plant is about 475,000 square foot. The west end of the plant houses our MDF1, we are running a 4507 as a distribution core. 1 year ago we added a 155,000 square foot addition to the east end of the plant, we created MDF2 in this location which houses a 4503 as a second distribution core. We have setup a 10Gbps backbone between cores with a 1Gbps redundant link. The 10gig backbone spans 998 feet and connects using MMF and 10gig-LX4 lasers. The 1gig spans the same distance and connects using MMF and 1gig SFPs.

When I arrive at the plant, I immediately when to my desk (in MDF1) and started pinging servers. All servers residing in my office were ok, however I could not ping anything in the east end of the plant. I checked the interfaces on the 4507 and found the 10gig interface down/down and showing 3 input errors and 3 CRC. I immediately went to MDF2 and checked the 4503. I found the 10gig interface error-disabled with no errors showing in the counters. I cleared the err-disable by shutting down the interface and bringing it back up. I issued a continuious ping to the 4507 from a server. All seemed well, no errors showing up. Then it started timing out, I checked the interface on the 4503 and found it in an up/down state. One minute passed and it was up/up again and pings were successful, then same thing happened. I rebooted both switches thinking someone plugged in something they shouldn't have, but when everything came back up it did the same thing. I finally swapped over the the 1gig backbone and all was well.

This link as ran fine for at least a year. No errors, nothing. Then all the sudden this. I am going to leave the link on 1gig until I can get back into the office in the morning, but I'd like anyone's input if possible. Our lone network engineer was unavailable at this time, so I took it upon myself to get everything working again, but I am still new at this.

Any thoughts? Thanks in advance.

Find more posts tagged with

Free for TechExams community: Cybersecurity salary guide

Compare cert salaries and plan your next career move

Button

Comments

ColbyG

Should be something in the log. No?

Monkerz

It's shows a link-flap error which caused the err-disable. Link-flap is caused by a loop?

fly351

Monkerz wrote: »

We have setup a 10Gbps backbone between cores with a 1Gbps redundant link. The 10gig backbone spans 998 feet and connects using MMF and 10gig-LX4 lasers. The 1gig spans the same distance and connects using MMF and 1gig SFPs.

hmm probably more of a interface issue or the SFP is faulty.

link flap is when the interface keeps going up/down.

ColbyG

fly351 wrote: »

link flap is when the interface keeps going up/down.

If he's thinking of a loop, he's probably talking about the host/mac address flapping error.

fly351

ColbyG wrote: »

If he's thinking of a loop, he's probably talking about the host/mac address flapping error.

In that case, wouldn't Portfast and BPDU Guard prevent that?

SRTMCSE

Sounds like it could be the sfp or possibly dirty fiber.

jason_lunde

Its most likely a faulty GBIC. Happens here every couple of months....unfortunatly its always the cisco branded 10GB gbics and not the 3rd party ones

. Luckily we bundled our 10GB interfaces into portchannels so that if it happens the other automagically picks up the slack. If you have the cash and fiber available you might look into doing the same.

networker050184

I'd have the line and SFPs checked also.

tiersten

jason_lunde wrote: »

Its most likely a faulty GBIC. Happens here every couple of months....unfortunatly its always the cisco branded 10GB gbics and not the 3rd party ones

Have a look at FlexOptix if so. Their optics are generally significantly cheaper. Up to you if you trust a reprogrammed third party optic tho :P

Monkerz

Looks like it was one of the LX4s that crapped out. My spare fixed the issue.

Jason, how long have you been running 10G? Whats your setup consist of? We had bad lasers 12 months ago, I didn't think it would be a reoccurring thing. Honestly one goes bad every few months?

networker050184

We haven't had any go out yet in our DC in over a years time. We are using 10G-SR and 10G-LX4 Cisco brand on 6708 cards and 3750Es.

Monkerz

We were running SRs but we exceeded the range and had to swap to LX4s. If it comes down to it, I'll just run a 4G etherchannel. I would get yelled at if I spent $2,000 for a laser every few months to slap a bandaid on the backbone.

tiersten

You're slightly exceeding the distance for 10GBASE-LX4 if you're running it over 998 feet.

I've not had any unusual failures in any of the 10G optics here. They're generally reliable except if you're the fiber is too short and you don't have an attenuator.

Monkerz

tiersten wrote: »

You're slightly exceeding the distance for 10GBASE-LX4 if you're running it over 998 feet.

This is true, I believe we are over 15 feet or so, but I've seen where people run 25 to 50 feet over the recommended max.

I think we will look in to the third party idea. Can't hurt, huh?

Anyone use third party 10G transceivers? Can you suggest one you've used with great reliability? Thanks Guys.

tiersten

Monkerz wrote: »

This is true, I believe we are over 15 feet or so, but I've seen where people run 25 to 50 feet over the recommended max.

Yeah. The limit is roughly 984 feet as its 300m and you're over by 14-15 feet. If it works and you're not getting errors then I guess you're okay.

Monkerz wrote: »

I think we will look in to the third party idea. Can't hurt, huh?

Anyone use third party 10G transceivers? Can you suggest one you've used with great reliability? Thanks Guys.

I linked to FlexOptix above but they don't appear to have any LX4 optics. Their GigE optics work well although the programming is a little strange because you do it via a web service with the box connected and its done on a credit system.

jason_lunde

Monkerz wrote: »

Looks like it was one of the LX4s that crapped out. My spare fixed the issue.

Jason, how long have you been running 10G? Whats your setup consist of? We had bad lasers 12 months ago, I didn't think it would be a reoccurring thing. Honestly one goes bad every few months?

Well we basically run 2x10GB links from our core to every distribution switch we have. Over the last 5 months we have had 2 go out; but I should add that the ones that have gone out have been in production for over a year or so...but it still hurts at those prices! I always try to get ones with pretty hefty warranty's, but they always seem to outlive those by a couple months. So honestly I am hoping that our recent failures are kind of "batch" issues, because recently they have all been Cisco branded units that we all got at the same time.