Dont use 9.@ Route Patterns!

chmorin

I just recently posted this on my blog (see signature). I wanted to share it with you all because this drove me nuts today:

I just spent the most annoying 5 hours of my life with Cisco TAC troubleshooting a call quality and call routing issue we had once I removed one of our 9.@ route patterns with the proper route patterns for the area.

Background info: We have many sites, and all of them had used 9.@ route patterns. Each site with only one Partition, and one gateway. I inherited this madness. Friday, we cut over from one AT&T T1-PRI to a dual T1-PRI from another provider. The cut over went fine, and I wanted to change the route patterns while we had an outage window. So I removed the 9.@ route pattern, and replaced it with the typical 9.1[2-9]xxxxxxxxx and similar route patterns. Tested calls, they were made successfully. We called it a night.

Monday rolled around and we seem to be getting intermittent call quality issues both from the PSTN and internally.We initially thought it was an issue with our WAN link for the internal issue, and an issue with our new PRI provider for the outbound call problem. The issue was jitter with occasionally crazy static issue. While I'm not sure where the static came from, the jitter was pretty easy to diagnose. We made some tests on our gateway and PSTN, listening to some calls, and slowly realized something. Our gateway was no longer making outbound calls, it was only receiving phone calls!


This created a few more questions. Like 1) Where the hell are the calls being routed to. And 2) Why in the world is our dial patterns not routing as intended.

I ran the DNA on the partition in question, and found something rather odd. When dialing externally, the DNA was matching a 9.@ route pattern in the partition that our DN's were in. I go into the partition and check dependencies, but no 9.@ route pattern was listed.

WHERE THE HELL IS IT COMING FROM?

So I beat my skull in for awhile trying to figure out why this route pattern vodoo was happening. Because it is our corporate HQ with the issues and time was not on my side, I called Cisco TAC with an "emergency". We troubleshot database replication issues, database update issues, other service issues in the back of call manager, CSS information, and the other route patterns. I had the guy stumped. We went ahead and did some detailed call traces with the RTM tool that I didn't know I had, and discovered something interesting:

Even though the DNA was showing the 9.@ Route Pattern being in the local Partition, the call traces were showing it being used in a whole other Partition, and forwarding it out the WAN to find the other gateway!

Well, that explains the sudden jitter, but the question is how to fix the problem. All call routing as had been instructed to me in my current CCVP (CCNP:Voice) studies so far indicated that call routing searched for route patterns inside the local partition, and match the closest one. Well apparently when you introduce the 9.@ route pattern anywhere in your Route Pattern structure, that changes it up a bit.

I have not labbed this yet, but this is what we experienced. Basically CUCM will search for ANY 9.@ route pattern in the CSS assigned to the device BEFORE it looks for a local one that may match better. The DNA will show this as being found in the local partition, but will route the data out whatever gateway it happened to find first matching the above criteria. FAN-FREAKING-TASTIC.

The TAC engineer took a poll from his peers, and all of them seemed to say it "Might" do this, or it "May" Do that. None of them seemed to know 100% what to expect from the 9.@ route pattern.

So what did I have to do? I could either a) Remove all 9.@ route patterns during business hours without change control and add proper route patterns for everything. Or b) Add the 9.@ route pattern back in for the local Partition and forward it out the needed route group, until I can get clearance to make the significant changes I want to make to partitions, CSS, and Route Patterns.

Naturally I did B, for my companies sake (and so I can do my clean up in one foul sweep). I wanted to share this with all of you so, should you experience this, you know what to sort of expect. I found NO documentation on the intended function of the 9.@ route pattern use. I suggest avoiding it. It could very well be depricated and unsupported. If I ever meet the individual who did our Route Patterns, they had better hope I have a job worth keeping because what they did as a 'short-cut' is just totally ludicrous all around.

/End Rant - Hope it was helpful

tokhss

that pattern should be eliminated .. sure its easy to use but all i keep hearing is negative talk about it.

Hey man, at least you will get rid of it cheers to that.

shodown

I thought I told you to ditch that 9@ pattern months ago:)

Also look into local route groups. i know you were studying CIPT1/2, but CCM 7 introduced local route groups which will make a lot of your work eaiser as you don't have to have duplicate patterns.

chmorin

shodown wrote: »

I thought I told you to ditch that 9@ pattern months ago:)

Also look into local route groups. i know you were studying CIPT1/2, but CCM 7 introduced local route groups which will make a lot of your work eaiser as you don't have to have duplicate patterns.

You did, and I have about 10 pages of change request I have made and sent up the latter since then! I wanted to do them all at once, and it seems I really have no choice about that. I also want to reorganize our horrid partition and CSS organization as well at the same time. It's only a matter of time.

You're right, and I have looked into them. The configuration is not all to complicated, but without actually being practiced in them in a lab environment (which is currently in pieces) I'm not comfortable suggesting them for production implementation by my hand just yet.

Besides, if I do it all now I won't have anything fun to do later

shodown

If you feel you have the skill start knocking it down tonight. Once you get good with Voice which you are on you way too, you will loose all time.

As an example of my day so far

1. Rebuilt Unity Server for 1 client who failed to make backups
2. Remade scripting for some ass hat who decided he knew call center, and he overwrote the orginal scripts.

3. Updated several UC5XX boxes with new software for bug
4. Fixed several DST errors(this is an ongoing event for the past 10 years with cisco)

5. One way audio issue, cause the phones were pulling Data Network IP's.


As you read above once you get good you will have no time cause you will be getting pulled in every direction.

chmorin

shodown wrote: »

As you read above once you get good you will have no time cause you will be getting pulled in every direction.

I'm so excited!

shodown

chmorin wrote: »

I'm so excited!

Exited to grow gray hair before 30
Sleeping on the couch cause your GF upset with you over work.
Gut growing out your body overnight
Long nights of troubleshooting
Long Days of Design.


Welcome to the club. As much as that sounds like a complaint I love what I do!!!!!!!!!

chmorin

shodown wrote: »

Exited to grow gray hair before 30
Sleeping on the couch cause your GF upset with you over work.
Gut growing out your body overnight
Long nights of troubleshooting
Long Days of Design.


Welcome to the club. As much as that sounds like a complaint I love what I do!!!!!!!!!

This is the life we choose. I will live with no regrets, only with the hope that I want my pursuits to make a difference somehow.

Besides, if those model girl CCIE's you see at conventions can keep their womanly figure, I'll be dammed if I'll lose my womanly figure!

MichaelCook

Thanks for the info as I can see this was obviously tough to TS. On a side-note I went over to check out your blog. It's quite good and I must say that your age surprised me. Based on your posts here I expected you to be quite a bit older. I guess that makes me part of the problem in IT. Quite the eye-opener that I will have to take into account in my future.

Kelkin

Just think of the experience you received by troubleshooting that problem