Dont use 9.@ Route Patterns!
I just recently posted this on my blog (see signature). I wanted to share it with you all because this drove me nuts today:
I just spent the most annoying 5 hours of my life with Cisco TAC troubleshooting a call quality and call routing issue we had once I removed one of our 9.@ route patterns with the proper route patterns for the area.
Background info: We have many sites, and all of them had used 9.@ route patterns. Each site with only one Partition, and one gateway. I inherited this madness. Friday, we cut over from one AT&T T1-PRI to a dual T1-PRI from another provider. The cut over went fine, and I wanted to change the route patterns while we had an outage window. So I removed the 9.@ route pattern, and replaced it with the typical 9.1[2-9]xxxxxxxxx and similar route patterns. Tested calls, they were made successfully. We called it a night.
Monday rolled around and we seem to be getting intermittent call quality issues both from the PSTN and internally.We initially thought it was an issue with our WAN link for the internal issue, and an issue with our new PRI provider for the outbound call problem. The issue was jitter with occasionally crazy static issue. While I'm not sure where the static came from, the jitter was pretty easy to diagnose. We made some tests on our gateway and PSTN, listening to some calls, and slowly realized something. Our gateway was no longer making outbound calls, it was only receiving phone calls!
This created a few more questions. Like 1) Where the hell are the calls being routed to. And 2) Why in the world is our dial patterns not routing as intended.
I ran the DNA on the partition in question, and found something rather odd. When dialing externally, the DNA was matching a 9.@ route pattern in the partition that our DN's were in. I go into the partition and check dependencies, but no 9.@ route pattern was listed.
WHERE THE HELL IS IT COMING FROM?
So I beat my skull in for awhile trying to figure out why this route pattern vodoo was happening. Because it is our corporate HQ with the issues and time was not on my side, I called Cisco TAC with an "emergency". We troubleshot database replication issues, database update issues, other service issues in the back of call manager, CSS information, and the other route patterns. I had the guy stumped. We went ahead and did some detailed call traces with the RTM tool that I didn't know I had, and discovered something interesting:
Even though the DNA was showing the 9.@ Route Pattern being in the local Partition, the call traces were showing it being used in a whole other Partition, and forwarding it out the WAN to find the other gateway!
Well, that explains the sudden jitter, but the question is how to fix the problem. All call routing as had been instructed to me in my current CCVP (CCNP:Voice) studies so far indicated that call routing searched for route patterns inside the local partition, and match the closest one. Well apparently when you introduce the 9.@ route pattern anywhere in your Route Pattern structure, that changes it up a bit.
I have not labbed this yet, but this is what we experienced. Basically CUCM will search for ANY 9.@ route pattern in the CSS assigned to the device BEFORE it looks for a local one that may match better. The DNA will show this as being found in the local partition, but will route the data out whatever gateway it happened to find first matching the above criteria. FAN-FREAKING-TASTIC.
The TAC engineer took a poll from his peers, and all of them seemed to say it "Might" do this, or it "May" Do that. None of them seemed to know 100% what to expect from the 9.@ route pattern.
So what did I have to do? I could either a) Remove all 9.@ route patterns during business hours without change control and add proper route patterns for everything. Or b) Add the 9.@ route pattern back in for the local Partition and forward it out the needed route group, until I can get clearance to make the significant changes I want to make to partitions, CSS, and Route Patterns.
Naturally I did B, for my companies sake (and so I can do my clean up in one foul sweep). I wanted to share this with all of you so, should you experience this, you know what to sort of expect. I found NO documentation on the intended function of the 9.@ route pattern use. I suggest avoiding it. It could very well be depricated and unsupported. If I ever meet the individual who did our Route Patterns, they had better hope I have a job worth keeping because what they did as a 'short-cut' is just totally ludicrous all around.
/End Rant - Hope it was helpful
