Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 453580 - Xen virtual machines using different libvirt virtual networks on the same host can only access each other in one direction
Summary: Xen virtual machines using different libvirt virtual networks on the same hos...
Keywords:
Status: CLOSED WONTFIX
Alias: None
Product: Red Hat Enterprise Linux 5
Classification: Red Hat
Component: libvirt
Version: 5.2
Hardware: i386
OS: Linux
medium
medium
Target Milestone: rc
: ---
Assignee: Laine Stump
QA Contact: Virtualization Bugs
URL:
Whiteboard:
: 1064963 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2008-07-01 13:21 UTC by David Egts
Modified: 2018-10-27 12:30 UTC (History)
11 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2011-08-15 21:25:08 UTC


Attachments (Terms of Use)
virsh dumpxml test > ~/virsh_dumpxml_test.txt (deleted)
2008-07-07 14:23 UTC, David Egts
no flags Details
virsh dumpxml rhel5 > ~/virsh_dumpxml_rhel5.txt (deleted)
2008-07-07 14:24 UTC, David Egts
no flags Details
iptables -L -n -v > ~/iptables_-L_-n_-v.txt (deleted)
2008-07-07 14:26 UTC, David Egts
no flags Details
iptables -t nat -L -n -v > iptables_-t_nat_-L_-n_-v.txt (deleted)
2008-07-07 14:27 UTC, David Egts
no flags Details
brctl show > ~/brctl_show.txt (deleted)
2008-07-07 14:27 UTC, David Egts
no flags Details
xenstore-ls > ~/xenstore-ls.txt (deleted)
2008-07-07 14:27 UTC, David Egts
no flags Details
/etc/xen/test (deleted)
2008-07-07 14:31 UTC, David Egts
no flags Details
/etc/xen/rhel5 (deleted)
2008-07-07 14:31 UTC, David Egts
no flags Details

Description David Egts 2008-07-01 13:21:55 UTC
From Bugzilla Helper:
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1.14) Gecko/20080404 Firefox/2.0.0.14

Description of problem:
I have two VMs:  A (attached to vnet1) and B (attached to vnet2).

A can ssh to B, but B can't ssh to (or ping) A.  That's the problem.

B can ping the gateway for vnet1, but B can't ping A which is attached to vnet1.  Both A and B can ssh to dom0, and both can access other nodes on the Internet.

If I shut down the VMs and rewire A to vnet2 and B to vnet1, the problem stays with the vnets where now B can ssh to A, but A can't ssh to B.  Both A and B can ssh to dom0, and both can access other nodes on the Internet.

I have the following set in /etc/sysctl.conf:

net.ipv4.ip_forward = 1

This worked in RHEL 5.1, but doesn't appear to work in RHEL 5.2.

Version-Release number of selected component (if applicable):
libvirt-0.3.3-7.el5

How reproducible:
Always


Steps to Reproduce:
1. Create two virtual networks, say vnet1 and vnet2.  Here are mine:

virsh # net-dumpxml 172_30_2_x
<network>
  <name>172_30_2_x</name>
  <uuid>4893f67b-c3f5-42f0-a020-dab6b2815a66</uuid>
  <forward/>
  <bridge name='vnet1' stp='on' forwardDelay='0' />
  <ip address='172.30.2.1' netmask='255.255.255.0'>
    <dhcp>
      <range start='172.30.2.100' end='172.30.2.254' />
    </dhcp>
  </ip>
</network>

virsh # net-dumpxml 172_30_3_x
<network>
  <name>172_30_3_x</name>
  <uuid>d1c55a64-8c15-be77-6546-15233e491912</uuid>
  <forward/>
  <bridge name='vnet2' stp='on' forwardDelay='0' />
  <ip address='172.30.3.1' netmask='255.255.255.0'>
    <dhcp>
      <range start='172.30.3.100' end='172.30.3.254' />
    </dhcp>
  </ip>
</network>

2. Create two virtual machines (say A and B) with one network interface each.  Set one VM's network interface to bridge=vnet1 and the other VM's network interface to bridge=vnet2.

3. Start the two virtual machines A and B.

Actual Results:
A will be able to ssh to B.  B will not be able to ssh to A.  B will be able to ping the gateway for vnet1 but not be able to access any VMs attached to vnet1 such as A.    Both A and B can ssh to dom0, and both can access other nodes on the Internet.

From the VM on vnet1 (can ping VM on vnet2):
~~~
[root@localhost ~]# ping 172.30.3.109
PING 172.30.3.109 (172.30.3.109) 56(84) bytes of data.
64 bytes from 172.30.3.109: icmp_seq=1 ttl=63 time=0.216 ms
64 bytes from 172.30.3.109: icmp_seq=2 ttl=63 time=0.173 ms
~~~

From the VM on vnet2 (can't ping VM on vnet1, but can ping the vnet1 gateway):
~~~
[root@localhost ~]# ping 172.30.2.170
PING 172.30.2.170 (172.30.2.170) 56(84) bytes of data.
From 172.30.3.1 icmp_seq=1 Destination Port Unreachable
From 172.30.3.1 icmp_seq=2 Destination Port Unreachable
...
[root@localhost ~]# ping 172.30.2.1
PING 172.30.2.1 (172.30.2.1) 56(84) bytes of data.
64 bytes from 172.30.2.1: icmp_seq=1 ttl=64 time=0.140 ms
64 bytes from 172.30.2.1: icmp_seq=2 ttl=64 time=0.115 ms
~~~

Expected Results:
A will be able to ssh to B, and B will be able to ssh into A.  Both A and B can access dom0 and other nodes on the Internet.

Additional info:

Comment 2 Daniel Berrange 2008-07-07 13:18:22 UTC
Please provide the config files for the 2 virtual machines from /etc/xen.

Also include the output of  'virsh dumpxml DOMAINNAME' for each virtual machine

Finally, the output of 'iptables -L -n -v' and 'iptables -t nat -L -n -v'


Comment 3 David Egts 2008-07-07 13:35:02 UTC
VM A is the VM called "test" and is currently connected to vnet1.  VM B is the
VM called "rhel5" and is currently connected to vnet2.

root@koloff:/etc/xen:2# cat test
name = "test"
uuid = "0aa0c6a9-d716-6330-e5dd-8bbd7fbf0662"
maxmem = 256
memory = 256
vcpus = 1
bootloader = "/usr/bin/pygrub"
on_poweroff = "destroy"
on_reboot = "restart"
on_crash = "restart"
vfb = [ "type=vnc,vncunused=1,keymap=en-us" ]
disk = [ "tap:aio:/var/lib/xen/images/test.img,xvda,w" ]
vif = [ "mac=00:16:3e:1c:7e:b1,bridge=vnet1" ]
root@koloff:/etc/xen:3# cat rhel5
name = "rhel5"
uuid = "e3b6a84a-8210-5755-dfac-169d7edba959"
maxmem = 512
memory = 512
vcpus = 2
bootloader = "/usr/bin/pygrub"
on_poweroff = "destroy"
on_reboot = "restart"
on_crash = "restart"
vfb = [ "type=vnc,vncunused=1,keymap=en-us" ]
disk = [ "tap:aio:/var/lib/xen/images/rhel5.img,xvda,w" ]
vif = [ "mac=00:16:3e:3c:6d:d2,bridge=vnet2" ]
root@koloff:/etc/xen:4# virsh dumpxml test
<domain type='xen'>
  <name>test</name>
  <uuid>0aa0c6a9-d716-6330-e5dd-8bbd7fbf0662</uuid>
  <bootloader>/usr/bin/pygrub</bootloader>
  <currentMemory>262144</currentMemory>
  <memory>262144</memory>
  <vcpu>1</vcpu>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <disk type='file' device='disk'>
      <driver name='tap' type='aio'/>
      <source file='/var/lib/xen/images/test.img'/>
      <target dev='xvda'/>
    </disk>
    <interface type='bridge'>
      <mac address='00:16:3e:1c:7e:b1'/>
      <source bridge='vnet1'/>
    </interface>
    <input type='mouse' bus='xen'/>
    <graphics type='vnc' port='-1' keymap='en-us'/>
    <console/>
  </devices>
</domain>

root@koloff:/etc/xen:5# virsh dumpxml rhel5
<domain type='xen'>
  <name>rhel5</name>
  <uuid>e3b6a84a-8210-5755-dfac-169d7edba959</uuid>
  <bootloader>/usr/bin/pygrub</bootloader>
  <currentMemory>524288</currentMemory>
  <memory>524288</memory>
  <vcpu>2</vcpu>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <disk type='file' device='disk'>
      <driver name='tap' type='aio'/>
      <source file='/var/lib/xen/images/rhel5.img'/>
      <target dev='xvda'/>
    </disk>
    <interface type='bridge'>
      <mac address='00:16:3e:3c:6d:d2'/>
      <source bridge='vnet2'/>
    </interface>
    <input type='mouse' bus='xen'/>
    <graphics type='vnc' port='-1' keymap='en-us'/>
    <console/>
  </devices>
</domain>

root@koloff:/etc/xen:6# iptables -L -n -v
Chain INPUT (policy ACCEPT 2322 packets, 2509K bytes)
 pkts bytes target     prot opt in     out     source               destination
        
    0     0 ACCEPT     udp  --  vnet1  *       0.0.0.0/0            0.0.0.0/0  
        udp dpt:53 
    0     0 ACCEPT     tcp  --  vnet1  *       0.0.0.0/0            0.0.0.0/0  
        tcp dpt:53 
    0     0 ACCEPT     udp  --  vnet1  *       0.0.0.0/0            0.0.0.0/0  
        udp dpt:67 
    0     0 ACCEPT     tcp  --  vnet1  *       0.0.0.0/0            0.0.0.0/0  
        tcp dpt:67 
    0     0 ACCEPT     udp  --  virbr0 *       0.0.0.0/0            0.0.0.0/0  
        udp dpt:53 
    0     0 ACCEPT     tcp  --  virbr0 *       0.0.0.0/0            0.0.0.0/0  
        tcp dpt:53 
    0     0 ACCEPT     udp  --  virbr0 *       0.0.0.0/0            0.0.0.0/0  
        udp dpt:67 
    0     0 ACCEPT     tcp  --  virbr0 *       0.0.0.0/0            0.0.0.0/0  
        tcp dpt:67 
    0     0 ACCEPT     udp  --  vnet0  *       0.0.0.0/0            0.0.0.0/0  
        udp dpt:53 
    0     0 ACCEPT     tcp  --  vnet0  *       0.0.0.0/0            0.0.0.0/0  
        tcp dpt:53 
    0     0 ACCEPT     udp  --  vnet0  *       0.0.0.0/0            0.0.0.0/0  
        udp dpt:67 
    0     0 ACCEPT     tcp  --  vnet0  *       0.0.0.0/0            0.0.0.0/0  
        tcp dpt:67 
    0     0 ACCEPT     udp  --  vnet2  *       0.0.0.0/0            0.0.0.0/0  
        udp dpt:53 
    0     0 ACCEPT     tcp  --  vnet2  *       0.0.0.0/0            0.0.0.0/0  
        tcp dpt:53 
    0     0 ACCEPT     udp  --  vnet2  *       0.0.0.0/0            0.0.0.0/0  
        udp dpt:67 
    0     0 ACCEPT     tcp  --  vnet2  *       0.0.0.0/0            0.0.0.0/0  
        tcp dpt:67 
    0     0 ACCEPT     udp  --  vnet3  *       0.0.0.0/0            0.0.0.0/0  
        udp dpt:53 
    0     0 ACCEPT     tcp  --  vnet3  *       0.0.0.0/0            0.0.0.0/0  
        tcp dpt:53 
    0     0 ACCEPT     udp  --  vnet3  *       0.0.0.0/0            0.0.0.0/0  
        udp dpt:67 
    0     0 ACCEPT     tcp  --  vnet3  *       0.0.0.0/0            0.0.0.0/0  
        tcp dpt:67 

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
        
    0     0 ACCEPT     all  --  *      vnet1   0.0.0.0/0           
172.30.2.0/24       state RELATED,ESTABLISHED 
    0     0 ACCEPT     all  --  vnet1  *       172.30.2.0/24        0.0.0.0/0  
        
    0     0 ACCEPT     all  --  vnet1  vnet1   0.0.0.0/0            0.0.0.0/0  
        
    0     0 REJECT     all  --  *      vnet1   0.0.0.0/0            0.0.0.0/0  
        reject-with icmp-port-unreachable 
    0     0 REJECT     all  --  vnet1  *       0.0.0.0/0            0.0.0.0/0  
        reject-with icmp-port-unreachable 
    0     0 ACCEPT     all  --  *      virbr0  0.0.0.0/0           
192.168.122.0/24    state RELATED,ESTABLISHED 
    0     0 ACCEPT     all  --  virbr0 *       192.168.122.0/24     0.0.0.0/0  
        
    0     0 ACCEPT     all  --  virbr0 virbr0  0.0.0.0/0            0.0.0.0/0  
        
    0     0 REJECT     all  --  *      virbr0  0.0.0.0/0            0.0.0.0/0  
        reject-with icmp-port-unreachable 
    0     0 REJECT     all  --  virbr0 *       0.0.0.0/0            0.0.0.0/0  
        reject-with icmp-port-unreachable 
    0     0 ACCEPT     all  --  *      vnet0   0.0.0.0/0           
172.30.1.0/24       state RELATED,ESTABLISHED 
    0     0 ACCEPT     all  --  vnet0  *       172.30.1.0/24        0.0.0.0/0  
        
    0     0 ACCEPT     all  --  vnet0  vnet0   0.0.0.0/0            0.0.0.0/0  
        
    0     0 REJECT     all  --  *      vnet0   0.0.0.0/0            0.0.0.0/0  
        reject-with icmp-port-unreachable 
    0     0 REJECT     all  --  vnet0  *       0.0.0.0/0            0.0.0.0/0  
        reject-with icmp-port-unreachable 
    0     0 ACCEPT     all  --  *      vnet2   0.0.0.0/0           
172.30.3.0/24       state RELATED,ESTABLISHED 
    0     0 ACCEPT     all  --  vnet2  *       172.30.3.0/24        0.0.0.0/0  
        
    0     0 ACCEPT     all  --  vnet2  vnet2   0.0.0.0/0            0.0.0.0/0  
        
    0     0 REJECT     all  --  *      vnet2   0.0.0.0/0            0.0.0.0/0  
        reject-with icmp-port-unreachable 
    0     0 REJECT     all  --  vnet2  *       0.0.0.0/0            0.0.0.0/0  
        reject-with icmp-port-unreachable 
    0     0 ACCEPT     all  --  *      vnet3   0.0.0.0/0           
172.30.4.0/24       state RELATED,ESTABLISHED 
    0     0 ACCEPT     all  --  vnet3  *       172.30.4.0/24        0.0.0.0/0  
        
    0     0 ACCEPT     all  --  vnet3  vnet3   0.0.0.0/0            0.0.0.0/0  
        
    0     0 REJECT     all  --  *      vnet3   0.0.0.0/0            0.0.0.0/0  
        reject-with icmp-port-unreachable 
    0     0 REJECT     all  --  vnet3  *       0.0.0.0/0            0.0.0.0/0  
        reject-with icmp-port-unreachable 

Chain OUTPUT (policy ACCEPT 1596 packets, 135K bytes)
 pkts bytes target     prot opt in     out     source               destination
        
root@koloff:/etc/xen:7# iptables -t nat -L -n -v
Chain PREROUTING (policy ACCEPT 36 packets, 8780 bytes)
 pkts bytes target     prot opt in     out     source               destination
        

Chain POSTROUTING (policy ACCEPT 31 packets, 1975 bytes)
 pkts bytes target     prot opt in     out     source               destination
        
    1   141 MASQUERADE  all  --  *      *       172.30.2.0/24        0.0.0.0/0 
         
    1   141 MASQUERADE  all  --  *      *       192.168.122.0/24     0.0.0.0/0 
         
    1   141 MASQUERADE  all  --  *      *       172.30.1.0/24        0.0.0.0/0 
         
    1   141 MASQUERADE  all  --  *      *       172.30.3.0/24        0.0.0.0/0 
         
    1    40 MASQUERADE  all  --  *      *       172.30.4.0/24        0.0.0.0/0 
         

Chain OUTPUT (policy ACCEPT 36 packets, 2579 bytes)
 pkts bytes target     prot opt in     out     source               destination
        



Comment 4 Daniel Berrange 2008-07-07 13:43:50 UTC
That XML dump shows that neither VM is running. Please do it again when the VMs
are actually running.

Please also include the 'brctl show' and 'xenstore-ls' output while both VMs are
running.

Don't paste this into BZ - attach files, otherwise BZ mangles linebreaks on long
lines.


Comment 5 David Egts 2008-07-07 14:23:32 UTC
Created attachment 311156 [details]
virsh dumpxml test > ~/virsh_dumpxml_test.txt

Comment 6 David Egts 2008-07-07 14:24:21 UTC
Created attachment 311157 [details]
virsh dumpxml rhel5 > ~/virsh_dumpxml_rhel5.txt

Comment 7 David Egts 2008-07-07 14:26:50 UTC
Created attachment 311158 [details]
iptables -L -n -v > ~/iptables_-L_-n_-v.txt

Comment 8 David Egts 2008-07-07 14:27:15 UTC
Created attachment 311159 [details]
iptables -t nat -L -n -v > iptables_-t_nat_-L_-n_-v.txt

Comment 9 David Egts 2008-07-07 14:27:41 UTC
Created attachment 311160 [details]
brctl show > ~/brctl_show.txt

Comment 10 David Egts 2008-07-07 14:27:59 UTC
Created attachment 311161 [details]
xenstore-ls > ~/xenstore-ls.txt

Comment 11 David Egts 2008-07-07 14:31:23 UTC
Created attachment 311162 [details]
/etc/xen/test

Comment 12 David Egts 2008-07-07 14:31:44 UTC
Created attachment 311163 [details]
/etc/xen/rhel5

Comment 13 David Egts 2008-07-07 14:39:44 UTC
Sorry about pasting into BZ.  I'll go the attachment route moving forward.

Output of requested commands are now uploaded as attachments.

I made sure both VMs were running at the time when commands were run.

Comment 14 Daniel Berrange 2008-07-07 15:22:05 UTC
Ok, reproduced the problem. It is caused by the ordering of the iptables rules.
I can't think of any easy way to solve this problem either without major
re-organization of the way our iptables rules work


Comment 15 David Egts 2008-07-07 15:36:03 UTC
Thanks for the confirmation!

For what it's worth, I do know that it worked as expected in RHEL 5.1.  I'm not
sure if we can leverage the effort in RHEL 5.1, or if the code differences
between RHEL 5.1 and 5.2 are too different to be worth while.  Just a suggestion.

Comment 16 Daniel Berrange 2008-07-07 16:00:43 UTC
Actually, I wasn't thinking straight when I wrote the previous message.

The semantics of the virtual networking is that machines on the virtual network
only have out-bound connectivity to the LAN. Machines on one virtual network
should not be able to connect to another virtual network.

So there is a bug here - traffic is supposed to be denied in *both* directions. 

The only intended inbound connectivity to guests is from the Dom0

I'm surprised you notice a difference on 5.1, because AFAIK we didn't change the
rules used in 5.2

Comment 17 David Egts 2008-07-07 16:39:36 UTC
If you try this on a RHEL 5.1 host, traffic is allowed in both directions.

This is actually a desired behavior, at least for me.  Here's a potential use
case...

What I'd like to do is have two vnets.  One vnet is my normal, generic vnet to
which all VMs connect.  I'd also like to have a second vnet which is a Satellite
provisioning network.  This second vnet has libvirt's built in DHCP turned off.

My Satellite VM would have two virtual NICs and be connected to both vnets.  The
Satellite VM would run DHCP and tftp servers listening on the second vnet.

I'd create a VM to PXE boot off of the second vnet.  It would get an IP address
for the second vnet via DHCP from the DHCP server running on the Satellite VM. 
It would pull down the kickstart file from the Satellite, and then do the
installation using the installation url of satellite.example.com whose IP is
associated with the Satellite's first virtual NIC connected to the first vnet.

On RHEL 5.1, this worked great.  On RHEL 5.2, the PXE VM on the second vnet was
able to PXE boot and start the installer, but the installer failed when it tried
to access satellite.example.com on the first vnet.

Is there another, possibly better, way to achieve a similar result?

Here are some alternatives that I came up with:

- I could possibly turn off libvirt DHCP on the first vnet and run the dhcp
server on dom0.  That would work but kind of defeat the purpose of leveraging
the power of libvirt however.

- I could possibly add a second virtual NIC to the PXE VM and connect it to the
first vnet.  Maybe this would be the best option?

- I could move satellite.example.com to the second vnet, but that would prevent
the VMs on the first vnet from accessing it.

My intuition says that there may be other use cases to allow VMs on separate
vnets on the same host to speak w/each other.  Taking a step back and thinking
of my host as a "virtual server room", there would be cases where you want the
hosts on separate vlans to not communicate, and other cases where you would.

Could this at least be an optional setting?  For instance, it could be an option
similar to the libvirt virtual network option to allow the vnet to forward
traffic to physical devices.

Thanks for taking the time to help me!

Comment 18 David Egts 2008-07-07 19:33:12 UTC
So I wound up connecting a second virtual NIC to the PXE VM and connected it to
the first vnet.  That essentially addresses my original requirement for
Satellite PXE provisioning as noted in comment #17.

In other words, I'm able to do what I need by using multiple NICs in a VM
connected to different vnets on the same host.

The original way I addressed this requirement is to have separate VMs connected
to separate vnets and expecting them to be able to communicate with each other
across vnets.  This worked great in RHEL 5.1, but only worked in one direction
in RHEL 5.2.  As noted in comment #16, this shouldn't be allowed to work at all
in either direction.

Anyhow, I'll leave it up to you to disable it in both directions, let it go for
now, add an option to allow communication in both directions, or choose
something else.

I'm still intuitively thinking that allowing this capability as an option would
be useful.  I'll defer to your guidance however.

Thank you again for all the help!

Comment 19 Hugh Brock 2009-10-29 16:46:26 UTC
Laine, can you have a look and see why we're not denying traffic in both directions as per comment #16?

Comment 20 Laine Stump 2009-10-29 18:53:57 UTC
Hugh - I assume you're interested in behavior of current code, not the ancient stuff referenced here?

Comment 21 Dave Allan 2010-07-06 18:19:10 UTC
We are rebasing libvirt in 5.6; that may fix this problem as it contains updates for iptables support.

Comment 27 RHEL Product and Program Management 2011-01-11 21:14:27 UTC
This request was evaluated by Red Hat Product Management for
inclusion in the current release of Red Hat Enterprise Linux.
Because the affected component is not scheduled to be updated in the
current release, Red Hat is unfortunately unable to address this
request at this time. Red Hat invites you to ask your support
representative to propose this request, if appropriate and relevant,
in the next release of Red Hat Enterprise Linux.

Comment 28 RHEL Product and Program Management 2011-01-11 22:58:31 UTC
This request was erroneously denied for the current release of
Red Hat Enterprise Linux.  The error has been fixed and this
request has been re-proposed for the current release.

Comment 31 RHEL Product and Program Management 2011-08-15 21:25:08 UTC
Development Management has reviewed and declined this request.  You may appeal
this decision by reopening this request.

Comment 32 Melvin Jones 2014-02-14 13:43:39 UTC
*** Bug 1064963 has been marked as a duplicate of this bug. ***


Note You need to log in before you can comment on or make changes to this bug.