How do you convince Verizon they have a problem?

xwesleyxwillisx

I've been having intermittent latency/packet loss issues for a few days. It manifests itself in slow page load times when browsing to specific sites (netflix.com, google.com, verizon.net).

Trace routes reveal some interesting things:

Tracing route to netflix.com [208.75.76.17]
over a maximum of 30 hops:

1 1 ms 1 ms 1 ms x.x.x.x
2 2 ms 1 ms 1 ms x.x.x.x
3 11 ms 5 ms 6 ms L100.RCMDVA-VFTTP-23.verizon-gni.net [72.84.246.1]
4 * 6 ms 5 ms 11-3-223.RCMDVA-LCR-02.verizon-gni.net [130.81.139.238]
5 14 ms 13 ms 12 ms P15-0-0.RCMDVA-LCR-01.verizon-gni.net [130.81.27.64]
6 12 ms 10 ms 10 ms so-15-0-0-0.LCC1-RES-BB-RTR1-RE1.verizon-gni.net [130.81.28.248]
7 14 ms * 15 ms 130.81.17.197
8 23 ms 24 ms 25 ms tge10-4.fr4.iad.llnw.net [69.28.156.205]
9 26 ms 22 ms 24 ms tge4-1.fr4.lga.llnw.net [69.28.171.154]
10 38 ms * 38 ms tge8-4.fr3.ord.llnw.net [69.28.172.198]
11 98 ms 96 ms 100 ms tge1-3.fr4.sjc.llnw.net [69.28.171.66]
12 96 ms 97 ms 97 ms netflix.tge7-4.fr4.sjc.llnw.net [69.28.179.74]
13 * 98 ms 97 ms vlan954.jnrt-edge01.prod1.netflix.com [69.53.225.26]
14 97 ms 98 ms 97 ms te1-8.csrt-agg01.prod1.netflix.com [69.53.225.6]
15 * 97 ms 97 ms Netflix: Rent as many movies as you want for only $8.99 a month! Free Trial [208.75.76.17]

I understand slight packet loss is to be expected on the Internet. However, consistent packet loss to the same host(s) is not...

I believe the problem resides in the connection between:
4 * 6 ms 5 ms 11-3-223.RCMDVA-LCR-02.verizon-gni.net [130.81.139.238]
5 14 ms 13 ms 12 ms P15-0-0.RCMDVA-LCR-01.verizon-gni.net [130.81.27.64]

Here is a traceroute to hulu:
1 1 ms <1 ms 1 ms x.x.x.x
2 2 ms 1 ms 2 ms x.x.x.x
3 7 ms 5 ms 5 ms L100.RCMDVA-VFTTP-23.verizon-gni.net [72.84.246.1]
4 6 ms 5 ms 6 ms 11-3-123.RCMDVA-LCR-01.verizon-gni.net [130.81.137.232]
5 10 ms 10 ms 10 ms so-15-0-0-0.LCC1-RES-BB-RTR1-RE1.verizon-gni.net [130.81.28.248]
6 11 ms 11 ms 10 ms 0.so-6-1-0.XL3.IAD8.ALTER.NET [152.63.36.233]
7 11 ms 10 ms 10 ms 0.xe-9-0-0.BR1.IAD8.ALTER.NET [152.63.41.49]
8 11 ms 11 ms 10 ms dcp-brdr-02.inet.qwest.net [63.146.26.105]
9 11 ms 10 ms 11 ms dcx-core-02.inet.qwest.net [205.171.251.37]
10 11 ms 11 ms 11 ms dcp-edge-01.inet.qwest.net [67.14.28.46]
11 10 ms 10 ms 9 ms 216.207.68.32

Notice the lack of packet loss. Now, if I run continuous pings to 11-3-223.RCMDVA-LCR-02.verizon-gni.net [130.81.139.238] , I have 0 packet loss. The same for 11-3-123.RCMDVA-LCR-01.verizon-gni.net [130.81.137.232] . It is only when a destination requires traversal from one to the other that I get packet loss.

I guess my question is, how do I convince Verizon it is their issue? How do you get beyond the L1 support that only knows how to restart your PC/router?

HeroPsycho

"Can you hear me cocking the gun I have pressed to your head clicking now? .... GOOD!"

Hyper-Me

Had this many times with comcast.

A traceroute reveals a huge latency spike at their main distribution nodes in the area and when i call they tell me it must be my equipment. lame.

Magically everything starts working again a day or so later.

shodown

Call and get it esclated to another Tier when dealing with them. When they had a problem with our network they would call and try to shoot stright to the top. Might as well do that to them

APA

I take it you only have one outbound path available?

If you have several can you pick a network that has on of the residing slow loading webpages etc..... force it to exit your network via another path other than Verizon and see the problems still exist?

Download WinMTR or get a Linux hosts with MTR on it..... if you only havea windows machine at your disposal then winMTR or perhaps pathping will give you an idea of which hops along the path could potentially be dropping packets.

HOWEVER - Keep in mind that routers de-prioritize ICMP packets if they are extremely busy doing other things... like say..... forwarding crictical\standard data etc...

The best way to prove to a transit provider that they have an issue is lodge a fault and then hammer them with legitimate troubleshooting results that you have come up with. Persistence is the key.......

Our NOC usualy deals with the transit providers initially but if I ever have to eventually due to the troubleshooting you've conducted (if it points anything serious out to them) and your persistence can get you through to the cluely engineers within the providers IP teams.... (I'm not saying all companies have engineers that know what they are doing by the way... )

Forsaken_GA

xwesleyxwillisx wrote: »

I understand slight packet loss is to be expected on the Internet. However, consistent packet loss to the same host(s) is not...

I believe the problem resides in the connection between:
4 * 6 ms 5 ms 11-3-223.RCMDVA-LCR-02.verizon-gni.net [130.81.139.238]
5 14 ms 13 ms 12 ms P15-0-0.RCMDVA-LCR-01.verizon-gni.net [130.81.27.64]

Consistent packet loss to the same host is actually pretty normal when it's ICMP traffic. It usually means the provider has deployed CoPP and is rate limiting ICMP traffic. traceroute can be misleading, far too many folks see a dropped packet and assume that the first node that dropped it is malfunctioning.

Use winmtr, or if you have a unix box, use regular mtr, let it run for about 100 or so cycles, it will show loss statistics. If only 1 or 2 hosts show packet loss, it's likely just ICMP being dropped at those routers. If the packet loss extends to every host after that (ie, if one router shows 10% packet loss, and so does every hop after that router) then the packet loss is real, and that node would be the culprit.

You also need to keep in mind that traffic is bi-directional. There's no guarantee the return path will be the same as the entry path (and I'd be surprised if it was). It's entirely possible that Verizon isn't having a problem and there may be a router on the path back to you that's dropping packets.

Service providers don't like hearing that there's a problem. Far too many commit the cardinal sin of letting customers troubleshoot their network.

Best way to get past them is to inundate them with data. I've always found mtr reports with a healthy dose of tcpdump data cuts through the red tape pretty quickly.

A phone call to the customer service department explaining that you'd like to cancel your account because tech support seems to be unable to help you also works pretty well.

xwesleyxwillisx

Forsaken_GA wrote: »

Consistent packet loss to the same host is actually pretty normal when it's ICMP traffic. It usually means the provider has deployed CoPP and is rate limiting ICMP traffic. traceroute can be misleading, far too many folks see a dropped packet and assume that the first node that dropped it is malfunctioning.

Use winmtr, or if you have a unix box, use regular mtr, let it run for about 100 or so cycles, it will show loss statistics. If only 1 or 2 hosts show packet loss, it's likely just ICMP being dropped at those routers. If the packet loss extends to every host after that (ie, if one router shows 10% packet loss, and so does every hop after that router) then the packet loss is real, and that node would be the culprit.

You also need to keep in mind that traffic is bi-directional. There's no guarantee the return path will be the same as the entry path (and I'd be surprised if it was). It's entirely possible that Verizon isn't having a problem and there may be a router on the path back to you that's dropping packets.

Service providers don't like hearing that there's a problem. Far too many commit the cardinal sin of letting customers troubleshoot their network.

Best way to get past them is to inundate them with data. I've always found mtr reports with a healthy dose of tcpdump data cuts through the red tape pretty quickly.

A phone call to the customer service department explaining that you'd like to cancel your account because tech support seems to be unable to help you also works pretty well.

I have attached the MTR trace as you have suggested. Initially, I thought this confirmed my suspicion. However, performing that same trace to P15-0-0.RCMDVA-LCR-01.verizon-gni.net [130.81.27.64] does NOT show any packet loss as it does with the full trace.

This now leads me to believe the problem lies in the return path (which as you said is often a different path). Somewhere along the way, it would seem the return path goes through a bad router or a bad circuit. The fact it only occurs with certain sites tells me those sites have a route back to my network which traverses that bad router/circuit.

Am I interpreting these results correctly? I'd like to find the smoking gun so I can tell Verizon unequivocally where the problem lies (not me!).

/usr

I assume you're referring to residential service here, rather than a business class account?

If this is just your personal internet connection...I believe you're going to have a hard time getting this fixed.

I say this because I've struggled with Verizon in the past with VERY similar issues. Getting them to even entertain the thought of it being their equipment is nearly impossible. I've even went so far as to speak to a manager, with the same result. They wouldn't schedule a tech to come out unless they could personally see from their end that my internet was acting up, while I called them. Since the issue was intermittent, this was quite difficult.

Funny enough, this was around 4 years ago. My parents still have that same connection, with the same issues, because it's all that's available.

Residential accounts just aren't a priority for them, in my experience. I hope you have better luck than I ever did!

Paul Boz

Are you getting most of your packets delivered in a timely manner? If so, that's all you're paying for. If you want higher quality of service escalate your service to business class. Verizon does not owe you a completely clean route 100% of the time. There are millions of subscribers to manage with a comparatively smaller workforce to facilitate them so the ability to respond to incidents has to be prioritized.

What always makes me laugh about people complaining about mild ISP issues is that when they have a brown out or blackout they never call to curse out the power company and tell them how stupid they are and try to armchair diagnose problems. It's such a ridiculous double standard.

xwesleyxwillisx

Paul Boz wrote: »

Are you getting most of your packets delivered in a timely manner? If so, that's all you're paying for. If you want higher quality of service escalate your service to business class. Verizon does not owe you a completely clean route 100% of the time. There are millions of subscribers to manage with a comparatively smaller workforce to facilitate them so the ability to respond to incidents has to be prioritized.

What always makes me laugh about people complaining about mild ISP issues is that when they have a brown out or blackout they never call to curse out the power company and tell them how stupid they are and try to armchair diagnose problems. It's such a ridiculous double standard.

I appreciate the response but not the hostility. I'm not trying to "armchair diagnose" a perceived problem. I am trying to troubleshoot an ongoing issue (the last 5 days). I have been more than willing to let Verizon do their share. I have humored them by running the TCP optimization on my PC, rebooting the router several times, and replacing the router itself.

What is frustrating is waiting on hold for over an hour to talk to someone and that is all the help you can get. I certainly don't expect perfect service 100% of the time (or should I say 99.999%?) for a home broadband connection, but I do expect a level of service that is more often than not reliable. Consistent packet loss (I am talking any time of day a consistent 5-15% packet loss) to the SAME SITES for 5 days is not what I would consider acceptable.

Simple browsing is not effected to any significant degree, but any video streaming/gaming/voice conversation is crippled.

Paul Boz

If you want to get it fixed open as many support tickets as you can with tech support then call customer service, have them reference the account, and observe the volume of support requests. You can then get them to knock some $$$ off of your bill.

As I said though, most major ISPs are structured in such a way that even if you are a CCIE and have the problem isolated to a bad config on a specific ISP router, you won't get anywhere without a business class service contract. If the average home-use subscriber could get the same level of service as a business they wouldn't have a case for business-class service. I don't think it's right, but it is what it is. The ISP that I used to work for dispatched within 24 hours for residential troubles and for most people that still wasn't enough.

Forsaken_GA

xwesleyxwillisx wrote: »

I have attached the MTR trace as you have suggested. Initially, I thought this confirmed my suspicion. However, performing that same trace to P15-0-0.RCMDVA-LCR-01.verizon-gni.net [130.81.27.64] does NOT show any packet loss as it does with the full trace.

This now leads me to believe the problem lies in the return path (which as you said is often a different path). Somewhere along the way, it would seem the return path goes through a bad router or a bad circuit. The fact it only occurs with certain sites tells me those sites have a route back to my network which traverses that bad router/circuit.

Am I interpreting these results correctly? I'd like to find the smoking gun so I can tell Verizon unequivocally where the problem lies (not me!).

Well, there may be a problem on the return path, only way to tell that is to get a trace from the destination back to your IP, but from that WinMTR, I'd say the problem actually is on your outbound path, and likely on either the third or the fourth hop (I'd actually blame the third hop if I had to pick... you can reach it, but when you try to reach beyond it is when the packet loss starts... in my experience, that's usually a problem with the router sending, though it could just as easily be the one receiving). It does look like there's a healthy amount of ICMP dropping going on though, since the loss varies from hop to hop, and I suppose it's possible that those are all just ICMP drops, but I doubt it. The fact that the loss starts at one hop and there's not a single node after that which doesn't experience some loss tells me that something is definitely up.

There's no nice way to get a provider to own up to the problems. If they can't help you, demand they roll a truck or they lose you as a customer, simple as that.

Kaminsky

I noticed you said broadband. Are you powering off your router at any point ?

The way carriers handle broadband is to keep increasing it over the course of a couple of weeks until the packet loss starts and then lower your rate and fix it at that. If you are resetting that modem, it will start all over again. With broadband routers, unless they are locked up, it's best just to leave them be.

I realise this is a bit like teaching grandma to suck eggs with your knowledge level, but I'm just offering the solution that it might be a carrier problem rather than your ISP.


On the matter of threatening to go with another ISP, I bet they will just go "/shrug .. bye bye then."

xwesleyxwillisx

Kaminsky wrote: »

I noticed you said broadband. Are you powering off your router at any point ?

When I said broadband I was referring to broadband Internet access in general. It is actually Verizon's FiOS service (fiber to the premises). It still gets converted to copper before it gets to me anyway though.

If anyone else has FiOS and has been in a similar situation (or had to deal with bad support) please chime in.

veritas_libertas

Also, call earlier in the day when calls are less likely to be headed overseas. Getting someone in country on the phone is especially beneficial.

GT-Rob

or...


You are a consumer provider paying less in monthly fees than what it will cost an engineer for an hour.

If multiple people are you effected, they already know about it. if its just a few, you are SOL.



This is coming from 2 years in Tier 2 Verizon.

dynamik

Tell them you had better service with AT&T.

TheShadow

dynamik wrote: »

Tell them you had better service with AT&T.

Like they care.

They are both phone companies; they don't have to care. Verizon is still GTE at the core anyway and they share the main backbone with AT&T. Wireless and video they care about, brave new world and all that.

xwesleyxwillisx

I realize this is sort of reviving a dead topic but I just wanted to post that I finally received a resolution from Verizon.

After several calls to Verizon back and forth I was finally able to talk to someone who understood what my problem was and how to escalate the issue to get it fixed. After talking to the technician on the phone he quipped that he could tell I knew more than most of his peers (surprise).

I received a call today from the local CO and they had indeed replaced a piece of equipment. As sure as doughnuts have holes my packet loss is now gone. The only silver lining out of all of this is I now have the number to my CO