QoS is killing me.

Ladies and Gentleman, can you give your attention to me:

Getting ready to go smooth off here. I'm trying to implement and End-to-End QoS solution beginning with classifying voice bearer and signalling traffic from both hard and soft phones. The voice traffic is originating from about 40 Cisco 2950-SI access switches. The Cisco 2950s are uplinked to Cisco 6513s. I'm using OmniPeek to capture packets off a couple of the VLANs to inspect the traffic and make sure it is getting marked and it looks like a lot of it is getting marked, but a lot of it isn't. Inspecting the packets that aren't getting marked reveals nothing useful. They all match the conditions I have specified in my traffic classification and marking ACLs and Policy Maps. I'm at a loss as to why some of this traffic is not getting marked, unless a 6513 just can't keep up with it (yeah right, CPU is barely over 20% on this thing).

Any input you all can provide would be appreciated. Thanks.

Find more posts tagged with

Comments

xwesleyxwillisx

My first question would be what end devices are you using? Are they Cisco phones? In that case, they should automatically be setting their L3/L2 markings to 5 (ef).

Have you noticed if the unmarked traffic is limited to the soft clients? Are these IPCommunicator clients? I don't have much experience with it but I would think it would automatically be marked as well.

What configuration do you have for the voice ports and trunks to your core? They should be set to trust the markings and not overwrite them.

These were my initial thoughts...

cisco_trooper

Might as well throw this inconsistent output in here:

show policy-map interface

GigabitEthernet1/8
Service-policy input: LAN_QOS_INGRESS
class-map: VOICE-RTP (match-all)
Match: access-group name QOS_VOICE_RTP
set dscp 46:
class-map: VOICE-SIGNALLING (match-all)
Match: access-group name QOS_VOICE_SIGNALLING
set dscp 26:
class-map: class-default (match-any)
Match: any
set dscp 0:

Gig1/8 has no Earl, though there is plenty of traffic.

GigabitEthernet1/10
Service-policy input: LAN_QOS_INGRESS
class-map: VOICE-RTP (match-all)
Match: access-group name QOS_VOICE_RTP
set dscp 46:
Earl in slot 7 :
31747164 bytes
5 minute offered rate 201304 bps
aggregate-forwarded 31747164 bytes
class-map: VOICE-SIGNALLING (match-all)
Match: access-group name QOS_VOICE_SIGNALLING
set dscp 26:
Earl in slot 7 :
20476 bytes
5 minute offered rate 112 bps
aggregate-forwarded 20476 bytes
class-map: class-default (match-any)
Match: any
set dscp 0:
Earl in slot 7 :
14302757 bytes
5 minute offered rate 97800 bps
aggregate-forwarded 14302757 bytes

show int gi1/8
GigabitEthernet1/8 is up, line protocol is up (connected)
Hardware is C6k 1000Mb 802.3, address is 0018.19bd.e687 (bia 0018.19bd.e687)
Description: L1N-1X06Z:Gi0/1:172.16.1.12
MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 1000Mb/s
input flow-control is off, output flow-control is off
Clock mode is auto
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:25, output 00:00:47, output hang never
Last clearing of "show interface" counters 01:43:49
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 180000 bits/sec, 108 packets/sec
5 minute output rate 244000 bits/sec, 147 packets/sec
817508 packets input, 172516805 bytes, 0 no buffer
Received 1467 broadcasts (1352 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 0 multicast, 0 pause input
0 input packets with dribble condition detected
1091779 packets output, 231924909 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 PAUSE output
0 output buffer failures, 0 output buffers swapped out

show int gi1/10
GigabitEthernet1/10 is up, line protocol is up (connected)
Hardware is C6k 1000Mb 802.3, address is 0018.19bd.e689 (bia 0018.19bd.e689)
Description: L1N-6X072:Gi0/1:172.16.1.16
MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 1000Mb/s
input flow-control is off, output flow-control is off
Clock mode is auto
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:57, output 00:00:24, output hang never
Last clearing of "show interface" counters 01:44:27
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 288000 bits/sec, 175 packets/sec
5 minute output rate 337000 bits/sec, 209 packets/sec
2267257 packets input, 494454247 bytes, 0 no buffer
Received 1824 broadcasts (1360 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 0 multicast, 0 pause input
0 input packets with dribble condition detected
2460044 packets output, 526336813 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 PAUSE output
0 output buffer failures, 0 output buffers swapped out

show tcam interface gi1/8 qos type1 ip detail
+-+

+---+----+-+---+--+---+---+
|T|Index| Dest Ip Addr | Source Ip Addr| DPort | SPort | TCP-F |Pro|MRFM|X|TOS|TN|COD|F-P|
+-+

+---+----+-+---+--+---+---+
V 36257 0.0.0.0 0.0.0.0 P=0 P=0

0 ---- 0 0 -- --- 0-0 <-
M 36260 0.0.0.0 0.0.0.0 0 0 0 ---- 0 0 <-
R rslt: 3 <-
V 36828 0.0.0.0 0.0.0.0 P=0 P=0

0 ---- 0 0 -- --- 0-0 <-
M 36836 0.0.0.0 0.0.0.0 0 0 0 X--- 0 0 <-
R rslt: 0 <-
V 36829 0.0.0.0 0.0.0.0 P=0 P=0

0 M--- 0 0 -- --- 0-0
M 36836 0.0.0.0 0.0.0.0 0 0 0 X--- 0 0
R rslt: 3

show tcam interface gi1/10 qos type1 ip detail
+-+

+---+----+-+---+--+---+---+
|T|Index| Dest Ip Addr | Source Ip Addr| DPort | SPort | TCP-F |Pro|MRFM|X|TOS|TN|COD|F-P|
+-+

+---+----+-+---+--+---+---+
V 36182 10.100.6.0 10.255.152.0 10000<P<20000 P=0

17 ---- 0 0 -- --- 0-0 <-
M 36188 255.255.254.0 255.255.254.0 0 255 X-X- 0 0 <-
R rslt: B80F00 <-
V 36191 10.255.152.0 10.100.6.0 P=0 10000<P<20000

17 ---- 0 0 -- --- 0-0 <-
M 36197 255.255.254.0 255.255.254.0 0 255 X-X- 0 0 <-
R rslt: B80F00 <-
V 36202 10.100.6.0 10.255.152.0 P=5060 P=0

17 ---- 0 0 -- --- 0-0 <-
M 36206 255.255.254.0 255.255.254.0 65535 0 255 X-X- 0 0 <-
R rslt: 681000 <-
V 36203 10.255.152.0 10.100.6.0 P=5060 P=0

17 ---- 0 0 -- --- 0-0
M 36206 255.255.254.0 255.255.254.0 65535 0 255 X-X- 0 0
R rslt: 681000
V 36227 0.0.0.0 0.0.0.0 P=0 P=0

0 ---- 0 0 -- --- 0-0 <-
M 36233 0.0.0.0 0.0.0.0 0 0 0 X--- 0 0 <-
R rslt: 1100 <-
V 36301 0.0.0.0 0.0.0.0 P=0 P=0

0 ---- 0 0 -- --- 0-0
M 36305 0.0.0.0 0.0.0.0 0 0 0 ---- 0 0
R rslt: 1100
V 36828 0.0.0.0 0.0.0.0 P=0 P=0

0 ---- 0 0 -- --- 0-0 <-
M 36836 0.0.0.0 0.0.0.0 0 0 0 X--- 0 0 <-
R rslt: 0 <-
V 36829 0.0.0.0 0.0.0.0 P=0 P=0

0 M--- 0 0 -- --- 0-0
M 36836 0.0.0.0 0.0.0.0 0 0 0 X--- 0 0
R rslt: 3

THESE INTERFACE ARE CONFIGURED EXACTLY THE SAME, YET TCAM DOES NOT APPEAR TO BUILD FOR GI1/8. BOTH INTERFACES HAVE PLENTY OF TRAFFIC GOING THROUGH THEM. TCAM RESOURCES ARE PLENTIFUL.

show tcam counts
Used Free Percent Used Reserved
---- ----

Labels

in) 26 4070 0
Labels

eg) 4 4092 0
ACL_TCAM

Masks: 21 4075 0 72
Entries: 102 32666 0 576
QOS_TCAM

Masks: 39 4057 0 18
Entries: 261 32507 0 144
LOU: 4 124 3
ANDOR: 0 16 0
ORAND: 0 16 0
ADJ: 4 2044 0

ANYWAY, SLIGHTLY OFF TOPIC, BUT BOTHERSOME NONETHELESS. ALL OTHER INTERFACES APPEAR AS GI1/10 DOES, AS EXPECTED.

cisco_trooper

xwesleyxwillisx wrote: »

My first question would be what end devices are you using? Are they Cisco phones? In that case, they should automatically be setting their L3/L2 markings to 5 (ef).

The phones attached to the 2950s are marking COS 5, DSCP 46.

xwesleyxwillisx wrote: »

Have you noticed if the unmarked traffic is limited to the soft clients? Are these IPCommunicator clients? I don't have much experience with it but I would think it would automatically be marked as well.

I haven't been able to confirm or deny this as a possibility so far. BUT, since the trust on the 6513 is not working as expected, I don't expect to be able to mark the traffic at the softphone and have it succeed. This was what I was going to do originally.

xwesleyxwillisx wrote: »

What configuration do you have for the voice ports and trunks to your core? They should be set to trust the markings and not overwrite them.

The 6513 trunks WERE set to trust DSCP, but sniffing packets revealed that all values were 0. This was excessively bothersome, so I tried trust COS, with the same results. So now, rather than trusting the markings from the uplink, I am classifying on ingress with Policy-Maps using ACLs. This appears to be working other than some of the traffic that matches the ACLs for some reason doesn't get marked, (or perhaps its slipping past classification entirely).

cisco_trooper

OK. Back to tracing back unmarked QoS traffic. I've been sniffing and tracing packets back to their sources most of the day. The packets that aren't being marked are packets that are coming in on interfaces where QoS TCAM is not built (w t f with that?!!?). Has anyone experienced situations like this before. I'm beginning think I'm going to need a TAC call, but I really want to avoid that if possible. These are interfaces that appear the same as Gi1/8 above.

ilcram19-2

do this on your switch for each port that you have the phone hooked up to
interface renge x-x
mls trust cos
switchport voice vlan x
switchport priority extended trust

you shoudl be all set, the packets shoudl be mark all the way to the phone with the right cos value and any other packet (in case you have a pc hooked up to the phone) should have a diferrent value

cisco_trooper

Just an FYI. This blade actually required a reset not too long ago because it was not even forwarding traffic. Since then I have confirmed with Cisco TAC that the blade is bad due to bad TCAM.

networker050184

cisco_trooper wrote: »

Just an FYI. This blade actually required a reset not too long ago because it was not even forwarding traffic. Since then I have confirmed with Cisco TAC that the blade is bad due to bad TCAM.

Glad you got it sorted out. Its always nice to find out it wasn't something you did wrong.