simple linux routing question

ehnde

I have 7 network cards. eth1 brings my network connection in. They each have their own bridge vmbrX. eth1 gets vmbr0 for right now. If I bring up vmbr3 (eth3) via dhclient, I lose my network connection. Is it necessary to put the other 6 network connections on a different subnet from my production network? Not sure how this works exactly.

Here is my routing table before and after bringing up eth3:

labserver:~# route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
192.168.1.0     *               255.255.255.0   U     0      0        0 vmbr0
default         192.168.1.1     0.0.0.0         UG    0      0        0 vmbr0
labserver:~# route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
192.168.1.0     *               255.255.255.0   U     0      0        0 vmbr0
192.168.1.0     *               255.255.255.0   U     0      0        0 vmbr3
default         192.168.1.1     0.0.0.0         UG    0      0        0 vmbr3
default         192.168.1.1     0.0.0.0         UG    0      0        0 vmbr0

and /etc/network/interfaces:

# network interface settings
auto lo
iface lo inet loopback

auto eth0
iface eth0 inet manual

iface eth1 inet manual

iface eth2 inet manual

iface eth3 inet manual

iface eth4 inet manual

iface eth5 inet manual

iface eth6 inet manual

auto vmbr0
iface vmbr0 inet static
    address  192.168.1.10
    netmask  255.255.255.0
    gateway  192.168.1.1
    bridge_ports eth1
    bridge_stp off
    bridge_fd 0

auto vmbr1
iface vmbr1 inet manual
    bridge_ports none
    bridge_stp off
    bridge_fd 0

auto vmbr2
iface vmbr2 inet manual
    bridge_ports eth2
    bridge_stp off
    bridge_fd 0

auto vmbr3
iface vmbr3 inet manual
    bridge_ports eth3
    bridge_stp off
    bridge_fd 0

auto vmbr4
iface vmbr4 inet manual
    bridge_ports eth4
    bridge_stp off
    bridge_fd 0

auto vmbr5
iface vmbr5 inet manual
    bridge_ports eth5
    bridge_stp off
    bridge_fd 0

auto vmbr6
iface vmbr6 inet manual
    bridge_ports eth6
    bridge_stp off
    bridge_fd 0

I'm wanting to use these interfaces for dynamips, each of them having an independent network connection capable of reaching the internet. If someone could point me in the right direction I'd be grateful.

Forsaken_GA

does vmbr3 actually have a link back to 192.168.1.1? If not, that's why you're losing connectivity. When you bring eth3 up, it's creating another default route out vmbr3, which is essentially black holing your traffic. If that's the case, then that's your fault, having a dhcp server set a default gateway that can't be reached through that interface is a no no.

ehnde

Forsaken_GA wrote: »

does vmbr3 actually have a link back to 192.168.1.1? If not, that's why you're losing connectivity. When you bring eth3 up, it's creating another default route out vmbr3, which is essentially black holing your traffic. If that's the case, then that's your fault, having a dhcp server set a default gateway that can't be reached through that interface is a no no.

That's what has me baffled...eth0/vmbr1 and eth3/vmbr3 are both connected to the same switch, which is connected to my router doing the internet routing/nat/dhcp. I'll go back and review how I have these cables running and see if what you have suggested may be the cause.

Forsaken_GA

Hold on, so you have two seperate interfaces connected to the same network segment?

If so, you have an arp problem. By default, Linux responds to arp requests for all of it's interfaces out all of it's interfaces, so if you have two interfaces connected to the same subnet.. that becomes a problem.

You need to do some research into the arp_filter kernel setting

ehnde

OK, will do. Thanks again for pointing me in the right direction.

ehnde

Just added this to /etc/sysctl.conf:

net.ipv4.conf.vmbr0.arp_filter = 1
net.ipv4.conf.vmbr1.arp_filter = 1
net.ipv4.conf.vmbr2.arp_filter = 1
net.ipv4.conf.vmbr3.arp_filter = 1
net.ipv4.conf.vmbr4.arp_filter = 1
net.ipv4.conf.vmbr5.arp_filter = 1
net.ipv4.conf.vmbr6.arp_filter = 1

I restarted networking, didn't work. Restarted the server, still no go. But then I logged into my router (the one doing DHCP/nat for my home network) and saw this:

192.168.1.10 00:25:22:8A:31:94
192.168.1.15 00:25:22:8A:31:94

Of course my home router only sees the switch. Surely it shouldn't have problems cooperating with a switch, though.

ehnde

Now I'm trying this: Ubuntu Linux - multiple NICs, same LAN... ARP responses always go out a single NIC - Server Fault but still no luck.

Right now in sysctl.conf I have:

net.ipv4.conf.all.arp_ignore=1
net.ipv4.conf.all.arp_announce=2
net.ipv4.conf.all.arp_filter=0

ehnde

It's been suggested I manually create route maps using LARTC style ip, but I'm afraid this is a bit over my head