MPLS: CE routers get other CE routes, but they're unreachable.

Agent6376

Good Evening TE,

I'm having an issue that I can't figure out for the life of me. I just started getting into the MPLS track, and this is really the first bump in the road that I've hit. Basically what's happening is that I receive routes across the L3VPN link, but they're pointing to the loopback interface of the PE router, which is unreachable via the VRF. As a result, communication fails across the MPLS cloud.

Here's a screenshot of the topology in GNS3...



I'm using EBGP to peer from CE to PE. Here are the configs for CustA_CE1, PE1, PE2, and CustA_CE2

CustA_CE1

Building configuration...

Current configuration : 938 bytes
!
version 12.4
service timestamps debug datetime msec
service timestamps log datetime msec
no service password-encryption
!
hostname CustA_CE1
!
boot-start-marker
boot-end-marker
!
!
no aaa new-model
memory-size iomem 5
ip cef
!
!
!
!
no ip domain lookup
ip auth-proxy max-nodata-conns 3
ip admission max-nodata-conns 3
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
interface Loopback0
ip address 9.9.9.9 255.255.255.255
!
interface FastEthernet0/0
ip address 192.168.1.2 255.255.255.0
duplex auto
speed auto
!
interface FastEthernet0/1
no ip address
shutdown
duplex auto
speed auto
!
router bgp 100
no synchronization
bgp log-neighbor-changes
redistribute connected
neighbor 192.168.1.1 remote-as 300
no auto-summary
!
ip forward-protocol nd
!
!
no ip http server
no ip http secure-server
!
!
!
!
control-plane
!
!
!
!
!
!
!
!
!
!
line con 0
exec-timeout 0 0
logging synchronous
line aux 0
line vty 0 4
login
!
!
end

PE1

Current configuration : 1951 bytes
!
version 12.4
service timestamps debug datetime msec
service timestamps log datetime msec
no service password-encryption
!
hostname PE1
!
boot-start-marker
boot-end-marker
!
!
no aaa new-model
memory-size iomem 5
ip cef
!
!
!
!
ip vrf A
rd 1:100
route-target export 1:100
route-target import 1:100
!
ip vrf B
rd 1:200
route-target export 1:200
route-target import 1:200
!
no ip domain lookup
ip auth-proxy max-nodata-conns 3
ip admission max-nodata-conns 3
!
no mpls ip propagate-ttl
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
interface Loopback0
ip address 1.1.1.1 255.255.255.255
!
interface FastEthernet0/0
ip address 10.1.2.3 255.255.255.248
speed 100
full-duplex
mpls ip
mpls mtu 1512
!
interface FastEthernet0/1
ip vrf forwarding A
ip address 192.168.1.1 255.255.255.0
duplex auto
speed auto
!
interface FastEthernet1/0
ip vrf forwarding B
ip address 192.168.1.1 255.255.255.0
speed 100
full-duplex
!
router ospf 1
log-adjacency-changes
network 1.1.1.1 0.0.0.0 area 0
network 10.1.2.0 0.0.0.255 area 0
!
router bgp 300
no synchronization
bgp log-neighbor-changes
neighbor 2.2.2.2 remote-as 300
neighbor 2.2.2.2 update-source Loopback0
no auto-summary
!
address-family vpnv4
neighbor 2.2.2.2 activate
neighbor 2.2.2.2 send-community extended
exit-address-family
!
address-family ipv4 vrf B
redistribute connected
neighbor 192.168.1.2 remote-as 200
neighbor 192.168.1.2 activate
neighbor 192.168.1.2 as-override
no synchronization
exit-address-family
!
address-family ipv4 vrf A
redistribute connected
neighbor 192.168.1.2 remote-as 100
neighbor 192.168.1.2 activate
neighbor 192.168.1.2 as-override
no synchronization
exit-address-family
!
ip forward-protocol nd
!
!
no ip http server
no ip http secure-server
!
!
!
!
control-plane
!
!
!
!
!
!
!
!
!
!
line con 0
exec-timeout 0 0
logging synchronous
line aux 0
line vty 0 4
login
!
!
end

PE2

Current configuration : 1915 bytes
!
version 12.4
service timestamps debug datetime msec
service timestamps log datetime msec
no service password-encryption
!
hostname PE2
!
boot-start-marker
boot-end-marker
!
!
no aaa new-model
memory-size iomem 5
ip cef
!
!
!
!
ip vrf A
rd 1:100
route-target export 1:100
route-target import 1:100
!
ip vrf B
rd 1:200
route-target export 1:200
route-target import 1:200
!
no ip domain lookup
ip auth-proxy max-nodata-conns 3
ip admission max-nodata-conns 3
!
no mpls ip propagate-ttl
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
interface Loopback0
ip address 2.2.2.2 255.255.255.255
!
interface FastEthernet0/0
ip address 10.1.3.3 255.255.255.248
speed 100
full-duplex
mpls ip
mpls mtu 1512
!
interface FastEthernet0/1
ip vrf forwarding A
ip address 192.168.2.1 255.255.255.0
duplex auto
speed auto
!
interface FastEthernet1/0
ip vrf forwarding B
ip address 192.168.2.1 255.255.255.0
duplex auto
speed auto
!
router ospf 1
log-adjacency-changes
network 2.2.2.2 0.0.0.0 area 0
network 10.1.3.0 0.0.0.255 area 0
!
router bgp 300
no synchronization
bgp log-neighbor-changes
neighbor 1.1.1.1 remote-as 300
neighbor 1.1.1.1 update-source Loopback0
no auto-summary
!
address-family vpnv4
neighbor 1.1.1.1 activate
neighbor 1.1.1.1 send-community extended
exit-address-family
!
address-family ipv4 vrf B
redistribute connected
neighbor 192.168.2.2 remote-as 200
neighbor 192.168.2.2 activate
neighbor 192.168.2.2 as-override
no synchronization
exit-address-family
!
address-family ipv4 vrf A
redistribute connected
neighbor 192.168.2.2 remote-as 100
neighbor 192.168.2.2 activate
neighbor 192.168.2.2 as-override
no synchronization
exit-address-family
!
ip forward-protocol nd
!
!
no ip http server
no ip http secure-server
!
!
!
!
control-plane
!
!
!
!
!
!
!
!
!
!
line con 0
exec-timeout 0 0
logging synchronous
line aux 0
line vty 0 4
login
!
!
end

CustA_CE2

version 12.4
service timestamps debug datetime msec
service timestamps log datetime msec
no service password-encryption
!
hostname CustA_CE2
!
boot-start-marker
boot-end-marker
!
!
no aaa new-model
memory-size iomem 5
ip cef
!
!
!
!
no ip domain lookup
ip auth-proxy max-nodata-conns 3
ip admission max-nodata-conns 3
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
interface Loopback0
ip address 10.10.10.10 255.255.255.255
!
interface FastEthernet0/0
ip address 192.168.2.2 255.255.255.0
duplex auto
speed auto
!
interface FastEthernet0/1
no ip address
shutdown
duplex auto
speed auto
!
router bgp 100
no synchronization
bgp log-neighbor-changes
redistribute connected
neighbor 192.168.2.1 remote-as 300
no auto-summary
!
ip forward-protocol nd
!
!
no ip http server
no ip http secure-server
!
!
!
!
control-plane
!
!
!
!
!
!
!
!
!
!
line con 0
exec-timeout 0 0
logging synchronous
line aux 0
line vty 0 4
login
!
!
end

Show commands on PE1

PE1(config)#do sh ip bgp vpnv4 vrf A
BGP table version is 19, local router ID is 1.1.1.1
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
r RIB-failure, S Stale
Origin codes: i - IGP, e - EGP, ? - incomplete

Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 1:100 (default for vrf A)
*> 9.9.9.9/32 192.168.1.2 0 0 100 ?
*>i10.10.10.10/32 2.2.2.2 0 100 0 100 ?
*> 192.168.1.0 0.0.0.0 0 32768 ?
* 192.168.1.2 0 0 100 ?
*>i192.168.2.0 2.2.2.2 0 100 0 ?

PE1(config)#do sh ip route vrf A

Routing Table: A
Codes: C - connected, S - static, R - RIP, M - mobile, B - BGP
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2
i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
ia - IS-IS inter area, * - candidate default, U - per-user static route
o - ODR, P - periodic downloaded static route

Gateway of last resort is not set

9.0.0.0/32 is subnetted, 1 subnets
B 9.9.9.9 [20/0] via 192.168.1.2, 00:37:51
10.0.0.0/32 is subnetted, 1 subnets
B 10.10.10.10 [200/0] via 2.2.2.2, 00:37:48
C 192.168.1.0/24 is directly connected, FastEthernet0/1
B 192.168.2.0/24 [200/0] via 2.2.2.2, 00:37:48

PE1(config)#do ping vrf A 10.10.10.10

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.10.10.10, timeout is 2 seconds:
...
Success rate is 0 percent (0/3)

Any help is appreciated!

Scales

PE-1 is trying to Ping the remote CE from its source address in the VRF (configuration Below)
interface FastEthernet0/1
ip vrf forwarding A
ip address 192.168.1.1 255.255.255.0
duplex auto
speed auto

The routing table on CE-2 doesn't know where 192.168.1.1 is located. And it doesn't have a default route pointing to PE-2.

So what you need to do in BGP underneath the address-family ipv4 vrf A is add redistribute connected and it should work.

Also add redistribute connected to PE2.

Agent6376

Scales wrote: »

PE-1 is trying to Ping the remote CE from its source address in the VRF (configuration Below)
interface FastEthernet0/1
ip vrf forwarding A
ip address 192.168.1.1 255.255.255.0
duplex auto
speed auto

The routing table on CE-2 doesn't know where 192.168.1.1 is located. And it doesn't have a default route pointing to PE-2.

So what you need to do in BGP underneath the address-family ipv4 vrf A is add redistribute connected and it should work.

Also add redistribute connected to PE2.

I do have redistribute connected under the address-family configuration. That doesn't seem to be the issue.

Scales

Agent6376 wrote: »

I do have redistribute connected under the address-family configuration. That doesn't seem to be the issue.

woops. Sorry completely glazed over that when i read the configuration.

What does the show mpls forwarding table say on PE1?

Agent6376

Scales wrote: »

woops. Sorry completely glazed over that when i read the configuration.

NP, thanks for checking it out and your suggestion though.

Scales

Agent6376 wrote: »

NP, thanks for checking it out and your suggestion though.

What does the show mpls forwarding table say on PE1?

Agent6376

Million dollar command?

PE1# show mpls ldp neighbor

I reloaded my topology and realized that my MPLS config on my P routers was blown away, but basic connectivity/OSPF was running.

DOH!

Scales

haha nice one.

kryolla

the loopback interfaces on the PE Routers is used to form neighborships for BPG VPNv4 so you dont get PHP happening to soon if you were to form neighborships with the directly connected link. The VRFs shouldn't have reachability to this as its part of the MPLS core and not the VPNs. HTH

Agent6376

kryolla wrote: »

the loopback interfaces on the PE Routers is used to form neighborships for BPG VPNv4 so you dont get PHP happening to soon if you were to form neighborships with the directly connected link. The VRFs shouldn't have reachability to this as its part of the MPLS core and not the VPNs. HTH

Once I got my MPLS P routers back online and I could ping across successfully, I noticed that the loopbacks are in the VRF even when everything is configured correctly.

I have a followup question as well:

Given the topology above, how would you configure this so that customers A and B could access the VOIP Gateway? Both sites would have routes to the gateway, but the VOIP Gateway can only have 1 192.168.1.x/24 and 192.168.2.x/24 in it's routing table.

Thoughts?

kryolla

Agent6376 wrote: »

Once I got my MPLS P routers back online and I could ping across successfully, I noticed that the loopbacks are in the VRF even when everything is configured correctly.

I have a followup question as well:

Given the topology above, how would you configure this so that customers A and B could access the VOIP Gateway? Both sites would have routes to the gateway, but the VOIP Gateway can only have 1 192.168.1.x/24 and 192.168.2.x/24 in it's routing table.

Thoughts?

I dont understand how the VPNs has the routes to PE loopbacks which are in the global routing table..The VOIP gateway you can do VRF aware NAT on the PE routers and how are the PEs getting the routes from the VOIP gateway

kryolla

Agent6376 wrote: »

but they're pointing to the loopback interface of the PE router

are you talking about the BGP next hop if so then that is correct because you're peering with the loopbacks but you cannot ping this from the VRF as this is part of the global routing table. Inside the MPLS core the routers are label swapping according to what label they have allocated to the BGP next hop that is why the P routers doesnt need to know all the routes of the VPN, they only need to know do I have a label to reach the next hop and in mpls there is no routing table lookup all I need is what label comes in, swap it with the outgoing label and what interface to send it out on and all this is in the mpls forwarding table.

Agent6376

kryolla wrote: »

are you talking about the BGP next hop if so then that is correct because you're peering with the loopbacks but you cannot ping this from the VRF as this is part of the global routing table. Inside the MPLS core the routers are label swapping according to what label they have allocated to the BGP next hop that is why the P routers doesnt need to know all the routes of the VPN, they only need to know do I have a label to reach the next hop and in mpls there is no routing table lookup all I need is what label comes in, swap it with the outgoing label and what interface to send it out on and all this is in the mpls forwarding table.

I am referring to the BGP next hop, yes. I'm confused on why the BGP next hop is being shown in the VRF, if there's no way that the VRF will ever be able to reach the address? At what point does the MPLS label forwarding begin?

CustA_CE1 pings 10.10.10.10. Checks routing table and the next hop is 192.168.1.1.

PE1 receives the packet on the interface labeled VRF A, so it appends the RD 1:100 to the packet and creates the VPNv4 prefix. It checks the route-target export and sees that it uses 1:100 as well. It checks it's next hop for 192.168.1.1 and sees that it's at 2.2.2.2 (via iBGP). Checks the LFIB and sees prefix 2.2.2.0/30 has a label of 17 so it uses MPLS to forward the packet.

Packet traverses MPLS core, P4 pops the top label and PE2 receives the VPNv4 packet. PE2 checks the route-target label and looks at the local import table for the configured VRFs. It sees that vrf A has the route-target import 1:100 command entered, so it peels off the route-target information and places the ipv4 prefix in the VRF table (iBGP prefix now)

Normal EBGP peering between CustA_CE2 takes place and the prefix is sent to CE2 where it hits the routing table.

Some areas are a little gray and I need to review (such as the route target import/export process - isn't a label added?) But I'm hoping I have at least the 1000 foot view down.

Your thoughts?

kryolla

Agent6376 wrote: »

I am referring to the BGP next hop, yes. I'm confused on why the BGP next hop is being shown in the VRF, if there's no way that the VRF will ever be able to reach the address? At what point does the MPLS label forwarding begin?

CustA_CE1 pings 10.10.10.10. Checks routing table and the next hop is 192.168.1.1.

PE1 receives the packet on the interface labeled VRF A, so it appends the RD 1:100 to the packet and creates the VPNv4 prefix. It checks the route-target export and sees that it uses 1:100 as well. It checks it's next hop for 10.10.10.10 vpnv4 label and sees that it's at 2.2.2.2 (via iBGP). Checks the LFIB and sees prefix 2.2.2.0/30 has a label of 17 transport label so it uses MPLS to forward the packet.

Packet traverses MPLS core, P4 pops the top label and PE2 receives the VPNv4 packet. PE2 checks the route-target label and looks at the local import table for the configured VRFs. It sees that vrf A has the route-target import 1:100 command entered, so it peels off the route-target information and places the ipv4 prefix in the VRF table (iBGP prefix now)

Normal EBGP peering between CustA_CE2 takes place and the prefix is sent to CE2 where it hits the routing table.

Some areas are a little gray and I need to review (such as the route target import/export process - isn't a label added?) But I'm hoping I have at least the 1000 foot view down.

Your thoughts?

the PE router puts a VPN label for the VPNv4 prefix then it puts another label for the next hop which is the transport label. This label is what gets swapped in the MPLS core and the PHP router will pop the transport label and send the VPNv4 label to the PE router. If the P router pops the transport label to early it will expose the VPNv4 label and the nex P router doesnt know what to do with it that why you peer with the loopback and not the directly connected link between the P and PE routers. Look at how the control plane works and then how the data plane forwards traffic.

Agent6376

I'll have to review the concepts once again to make sure I understand the process from end to end, but I'm feeling pretty good about it at this point. I want to get the core of L3VPNs and MPLS operation down before moving on to TE/L2TPVPN etc.

Thanks a bunch!

Scales

RE VPN Packet Forwarding:

CE-1 Sends a packet to CE-2 (both in the same MPLS VPN)
The packet arrives at PE-1.
PE-1 performs a route lookup and based on the destination performs label imposition (Push Operation).
Bottom Label (S-bit on) is the VPN label assigned by PE-2.
Top Label (S-Bit off) is the Label to reach PE-2.
Label Switching is performed based on the top label until it reaches the P router before PE-2. This router performs PHP and sends the packet to PE-2 with only the VPN Label. PE-2 sends the packet to CE-2 based on the VPN label it assigned and sent to PE-1 using MP-BGP.

The VOIP Gateway:
You want to create a Central Services VPN.

Agent6376

I found a very good resource for the detailed operation from end to end for MPLS VPNs. It documents the MPLS VPN control and data planes thoroughly, and is a great read. If anyone is just getting into MPLS, it's a great read to clear up a good bit of questions.

Multiprotocol Label Switching > MPLS VPNs

kryolla

Scales wrote: »

RE VPN Packet Forwarding:

The VOIP Gateway:
You want to create a Central Services VPN.

what if the Gateway wasnt in a VPN but in the global routing table, he didnt specify but central services usually involves loopbacks advertised with a specific RT and the op says the gatway only has routes to 192.168.1 and .2 networks and since the same networks are used on 2 different VPNs, NAT is a possible solution. Also no routing is performed in the MPLS core thats why its called label switching and its based on the cef entry and mpls forwarding table with the exception of the PE router.

Scales

kryolla wrote: »

what if the Gateway wasnt in a VPN but in the global routing table, he didnt specify but central services usually involves loopbacks advertised with a specific RT and the op says the gatway only has routes to 192.168.1 and .2 networks and since the same networks are used on 2 different VPNs, NAT is a possible solution. Also no routing is performed in the MPLS core thats why its called label switching and its based on the cef entry and mpls forwarding table with the exception of the PE router.

Yep exactly. What if.

and yes there is routing in the MPLS Core, but yes the packet would be label switched not routed.

kryolla

Scales wrote: »

Yep exactly. What if.

and yes there is routing in the MPLS Core, but yes the packet would be label switched not routed.

I was talking about traffic from 1 VPN site to another VPN site. There is no routing lookups done in the MPLS core with the exception of the PE routers

Scales

kryolla wrote: »

I was talking about traffic from 1 VPN site to another VPN site. There is no routing lookups done in the MPLS core with the exception of the PE routers

then why didn't you say that.

kryolla

Scales wrote: »

then why didn't you say that.

because you said this

Scales wrote: »

RE VPN Packet Forwarding:

CE-1 Sends a packet to CE-2 (both in the same MPLS VPN)
The packet arrives at PE-1.
PE-1 performs a route lookup and based on the destination performs label imposition (Push Operation).
Bottom Label (S-bit on) is the VPN label assigned by PE-2.
Top Label (S-Bit off) is the Label to reach PE-2.
Routing is performed based on the top label until it reaches the P router before PE-2. This router performs PHP and sends the packet to PE-2 with only the VPN Label. PE-2 sends the packet to CE-2 based on the VPN label it assigned and sent to PE-1 using MP-BGP..

so it was inferred I was talking about VPN forwarding but I guess you need it spelled out so why dont you go back and study your CCNA material

Scales

kryolla wrote: »

because you said this



so it was inferred I was talking about VPN forwarding but I guess you need it spelled out so why dont you go back and study your CCNA material

What exactly is your problem?

Edit: Changed from Routing to Label Switching. Happy Now?

kryolla

Scales wrote: »

What exactly is your problem?

Edit: Changed from Routing to Label Switching. Happy Now?

lets start here

Scales wrote: »

PE-1 is trying to Ping the remote CE from its source address in the VRF (configuration Below)
interface FastEthernet0/1
ip vrf forwarding A
ip address 192.168.1.1 255.255.255.0
duplex auto
speed auto

The routing table on CE-2 doesn't know where 192.168.1.1 is located. And it doesn't have a default route pointing to PE-2.

So what you need to do in BGP underneath the address-family ipv4 vrf A is add redistribute connected and it should work.

Also add redistribute connected to PE2.

If the BGP next hop was inaccessible it wouldn't have made it to the routing table due to it not being a best route

Scales

From the Original Post:
PE1(config)#do ping vrf A 10.10.10.10

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.10.10.10, timeout is 2 seconds:
...
Success rate is 0 percent (0/3)

What is the source address of the PING from PE-1?