Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1062908 - Nova instance creation fails with libvirtError: Unable to add port tap9cbcd041-02 to OVS bridge br-int: Operation not permitted
Summary: Nova instance creation fails with libvirtError: Unable to add port tap9cbcd04...
Keywords:
Status: CLOSED INSUFFICIENT_DATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-nova
Version: 4.0
Hardware: Unspecified
OS: Unspecified
medium
high
Target Milestone: ---
: 5.0 (RHEL 7)
Assignee: Ryan Hallisey
QA Contact: Ami Jeain
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2014-02-08 13:48 UTC by shilpa
Modified: 2016-04-26 17:08 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2014-07-02 15:30:30 UTC
Target Upstream Version:


Attachments (Terms of Use)
Nova debug logs (deleted)
2014-02-08 13:51 UTC, shilpa
no flags Details

Description shilpa 2014-02-08 13:48:39 UTC
Description of problem:
Nova instance creation fails with libvirtError: Unable to add port tap9cbcd041-02 to OVS bridge br-int: Operation not permitted. It was a previously working setup. No external network has been setup. Only a test setup to test RHOS with glusterfs. Using 6.5 RHEL

Version-Release number of selected component (if applicable):
libvirt-0.10.2-29.el6_5.3.x86_64
libvirt-client-0.10.2-29.el6_5.3.x86_64
openstack-nova-2013.2.1-2.el6ost.noarch


How reproducible: Ever since the first failure, always.

Steps to reproduce:

1. Installed and configure glance, cinder and nova to use glusterfs volumes.
2. Create and delete several instances. After a few instances, creation of vm's started failing.

# nova boot --flavor 2 --image 082e2e6a-1893-48f2-949c-3104a77b42b7 instance-1
+--------------------------------------+--------------------------------------+
| Property                             | Value                                |
+--------------------------------------+--------------------------------------+
| OS-EXT-STS:task_state                | scheduling                           |
| image                                | cirros                               |
| OS-EXT-STS:vm_state                  | building                             |
| OS-EXT-SRV-ATTR:instance_name        | instance-0000001d                    |
| OS-SRV-USG:launched_at               | None                                 |
| flavor                               | m1.small                             |
| id                                   | 1069b9fb-6ffa-4389-9508-727751d7aa87 |
| security_groups                      | [{u'name': u'default'}]              |
| user_id                              | b6e5391e2ec542fb9a25c23a5648950f     |
| OS-DCF:diskConfig                    | MANUAL                               |
| accessIPv4                           |                                      |
| accessIPv6                           |                                      |
| progress                             | 0                                    |
| OS-EXT-STS:power_state               | 0                                    |
| OS-EXT-AZ:availability_zone          | nova                                 |
| config_drive                         |                                      |
| status                               | BUILD                                |
| updated                              | 2014-02-08T13:39:32Z                 |
| hostId                               |                                      |
| OS-EXT-SRV-ATTR:host                 | None                                 |
| OS-SRV-USG:terminated_at             | None                                 |
| key_name                             | None                                 |
| OS-EXT-SRV-ATTR:hypervisor_hostname  | None                                 |
| name                                 | instance-1                           |
| adminPass                            | C9gBF7p2vp42                         |
| tenant_id                            | ae458c0ee9864efca47906f9f3b7278d     |
| created                              | 2014-02-08T13:39:32Z                 |
| os-extended-volumes:volumes_attached | []                                   |
| metadata                             | {}                                   |
+--------------------------------------+--------------------------------------+


--------+---------------------+
# nova list
+--------------------------------------+------------+--------+------------+-------------+---------------------+
| ID                                   | Name       | Status | Task State | Power State | Networks            |
+--------------------------------------+------------+--------+------------+-------------+---------------------+
| 1069b9fb-6ffa-4389-9508-727751d7aa87 | instance-1 | BUILD  | spawning   | NOSTATE     | public=172.24.4.227 |
+--------------------------------------+------------+--------+------------+-------------+---------------------+
# nova list
+--------------------------------------+------------+--------+------------+-------------+---------------------+
| ID                                   | Name       | Status | Task State | Power State | Networks            |
+--------------------------------------+------------+--------+------------+-------------+---------------------+
| 1069b9fb-6ffa-4389-9508-727751d7aa87 | instance-1 | ERROR  | None       | NOSTATE     | public=172.24.4.227 |
+--------------------------------------+------------+--------+------------+-------------+---------------------+


Expected results:

Instances should have booted successfully.

Additional info:

Logs from compute.log:

2014-02-08 19:09:41.800 24378 ERROR nova.compute.manager [req-2e945b46-b9f1-4013-b5bd-b6e751e3715f b6e5391e2ec542fb9a25c23a5648950f ae458c0ee9864efca47906f9f3b7278d] [instance: 1069b9fb-6ffa-4389-9508-727751d7aa87] Error: Unable to add port tap9cbcd041-02 to OVS bridge br-int: Operation not permitted
2014-02-08 19:09:41.800 24378 TRACE nova.compute.manager [instance: 1069b9fb-6ffa-4389-9508-727751d7aa87] Traceback (most recent call last):
2014-02-08 19:09:41.800 24378 TRACE nova.compute.manager [instance: 1069b9fb-6ffa-4389-9508-727751d7aa87]   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1037, in _build_instance
2014-02-08 19:09:41.800 24378 TRACE nova.compute.manager [instance: 1069b9fb-6ffa-4389-9508-727751d7aa87]     set_access_ip=set_access_ip)
2014-02-08 19:09:41.800 24378 TRACE nova.compute.manager [instance: 1069b9fb-6ffa-4389-9508-727751d7aa87]   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1420, in _spawn
2014-02-08 19:09:41.800 24378 TRACE nova.compute.manager [instance: 1069b9fb-6ffa-4389-9508-727751d7aa87]     LOG.exception(_('Instance failed to spawn'), instance=instance)
2014-02-08 19:09:41.800 24378 TRACE nova.compute.manager [instance: 1069b9fb-6ffa-4389-9508-727751d7aa87]   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1417, in _spawn
2014-02-08 19:09:41.800 24378 TRACE nova.compute.manager [instance: 1069b9fb-6ffa-4389-9508-727751d7aa87]     block_device_info)
2014-02-08 19:09:41.800 24378 TRACE nova.compute.manager [instance: 1069b9fb-6ffa-4389-9508-727751d7aa87]   File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 2080, in spawn
2014-02-08 19:09:41.800 24378 TRACE nova.compute.manager [instance: 1069b9fb-6ffa-4389-9508-727751d7aa87]     block_device_info, context=context)
2014-02-08 19:09:41.800 24378 TRACE nova.compute.manager [instance: 1069b9fb-6ffa-4389-9508-727751d7aa87]   File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 3235, in _create_domain_and_network
2014-02-08 19:09:41.800 24378 TRACE nova.compute.manager [instance: 1069b9fb-6ffa-4389-9508-727751d7aa87]     domain = self._create_domain(xml, instance=instance, power_on=power_on)
2014-02-08 19:09:41.800 24378 TRACE nova.compute.manager [instance: 1069b9fb-6ffa-4389-9508-727751d7aa87]   File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 3178, in _create_domain
2014-02-08 19:09:41.800 24378 TRACE nova.compute.manager [instance: 1069b9fb-6ffa-4389-9508-727751d7aa87]     domain.createWithFlags(launch_flags)
2014-02-08 19:09:41.800 24378 TRACE nova.compute.manager [instance: 1069b9fb-6ffa-4389-9508-727751d7aa87]   File "/usr/lib/python2.6/site-packages/eventlet/tpool.py", line 187, in doit
2014-02-08 19:09:41.800 24378 TRACE nova.compute.manager [instance: 1069b9fb-6ffa-4389-9508-727751d7aa87]     result = proxy_call(self._autowrap, f, *args, **kwargs)
2014-02-08 19:09:41.800 24378 TRACE nova.compute.manager [instance: 1069b9fb-6ffa-4389-9508-727751d7aa87]   File "/usr/lib/python2.6/site-packages/eventlet/tpool.py", line 147, in proxy_call
2014-02-08 19:09:41.800 24378 TRACE nova.compute.manager [instance: 1069b9fb-6ffa-4389-9508-727751d7aa87]     rv = execute(f,*args,**kwargs)
2014-02-08 19:09:41.800 24378 TRACE nova.compute.manager [instance: 1069b9fb-6ffa-4389-9508-727751d7aa87]   File "/usr/lib/python2.6/site-packages/eventlet/tpool.py", line 76, in tworker
2014-02-08 19:09:41.800 24378 TRACE nova.compute.manager [instance: 1069b9fb-6ffa-4389-9508-727751d7aa87]     rv = meth(*args,**kwargs)
2014-02-08 19:09:41.800 24378 TRACE nova.compute.manager [instance: 1069b9fb-6ffa-4389-9508-727751d7aa87]   File "/usr/lib64/python2.6/site-packages/libvirt.py", line 708, in createWithFlags
2014-02-08 19:09:41.800 24378 TRACE nova.compute.manager [instance: 1069b9fb-6ffa-4389-9508-727751d7aa87]     if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
2014-02-08 19:09:41.800 24378 TRACE nova.compute.manager [instance: 1069b9fb-6ffa-4389-9508-727751d7aa87] libvirtError: Unable to add port tap9cbcd041-02 to OVS bridge br-int: Operation not permitted

Tried to restart libvirtd and nova services with no success. Have not found a workaround.

Comment 1 shilpa 2014-02-08 13:51:37 UTC
Created attachment 860883 [details]
Nova debug logs

Debug logs attached after recreating the problem with instance-id "b740efec-47e1-4b86-af7a-6475b347fce1"

Comment 4 Russell Bryant 2014-07-02 15:30:30 UTC
I'm sorry that we never got to this.  Given the age of this bug, I don't think we'll be able to reproduce this at this point.  This also looks like something that we should just handle upstream as a bug in launchpad.  If this is still something that you care about, please just file a bug upstream.


Note You need to log in before you can comment on or make changes to this bug.