Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1595403 - OSP-13 unable to access external network from within instance
Summary: OSP-13 unable to access external network from within instance
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-neutron
Version: 13.0 (Queens)
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: ---
: ---
Assignee: Assaf Muller
QA Contact: Toni Freger
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-06-26 20:35 UTC by karan singh
Modified: 2018-07-02 07:38 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-07-02 07:38:00 UTC
Target Upstream Version:


Attachments (Terms of Use)

Description karan singh 2018-06-26 20:35:02 UTC
Description of problem:

I have a running OSP-13 setup and can launch instances , SSH into the instances. The current problem I am having is -

“I am unable to access external address (internet) from within the Instances”
 
Which means I am not able to subscribe the instances to RHT repo, install packages and proceed with benchmarking.
 
FIY everything was working with OSP-12 and I was able to complete my testing. As soon as I moved to (fresh install) OSP-13 with [2] [3] [4] overcloud templates, I lost internet access from Instances. FYI Egress rules in security group is ANY ANY 0.0.0.0/0

I am not sure if there is something fishy going in OSP-13 Neutron container image. 

see https://bugzilla.redhat.com/show_bug.cgi?id=1592528#c19


 
[1] https://pastebin.com/raw/849xdW9R . <--- Detailed Logs

THT:
----
[2] https://github.com/ksingh7/OSP-12_RHCS_Deployment_Guide/blob/master/templates/network-environment.yaml#L36-L37
[3] https://github.com/ksingh7/OSP-12_RHCS_Deployment_Guide/blob/master/templates/network-environment.yaml#L62
[4] https://github.com/ksingh7/OSP-12_RHCS_Deployment_Guide/blob/master/templates/network-environment.yaml#L74


Version-Release number of selected component (if applicable):

OSP-13 , container image and batch date is 

                "batch": "20180507.1",
                "build-date": "2018-05-07T21:01:31.410343",


How reproducible:
Tried it once

Steps to Reproduce:
1. Install OSP-13 with 1 controller and ceph nodes
2. Create openstack external and private network for instnces
3. Create openstack instnace and assign private / public netowkr (in the form of floating IP)
4. SSH into the instance
5. Ping outside world

Actual results:


v = 0.427/1.013/2.180/0.825 ms

## Able to SSH into the instance

(overcloud) [stack@refarch-r220-02 ~]$ ssh -i stack.pem cirros@192.168.120.159
The authenticity of host '192.168.120.159 (192.168.120.159)' can't be established.
ECDSA key fingerprint is SHA256:lJunwpKsEU2vOLEq5B91km3z/8CSM8GucVgoteJ7T+4.
ECDSA key fingerprint is MD5:a7:8a:2c:8c:d0:92:00:f7:37:d3:92:d5:37:64:23:84.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '192.168.120.159' (ECDSA) to the list of known hosts.
$
$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc pfifo_fast qlen 1000
    link/ether fa:16:3e:e1:88:b3 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.8/24 brd 192.168.1.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fee1:88b3/64 scope link
       valid_lft forever preferred_lft forever
$

## Unable to ping external DNS

$ ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
^C
--- 8.8.8.8 ping statistics ---
6 packets transmitted, 0 packets received, 100% packet loss
$
$
$
$

$ netstat -nr
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
0.0.0.0         192.168.1.1     0.0.0.0         UG        0 0          0 eth0
169.254.169.254 192.168.1.1     255.255.255.255 UGH       0 0          0 eth0
192.168.1.0     0.0.0.0         255.255.255.0   U         0 0          0 eth0
$
$ cat /etc/resolv.conf
search openstacklocal
nameserver 8.8.8.8
$

## Able to ping default gateway

$ ping 192.168.1.1
PING 192.168.1.1 (192.168.1.1): 56 data bytes
64 bytes from 192.168.1.1: seq=0 ttl=64 time=0.313 ms
64 bytes from 192.168.1.1: seq=1 ttl=64 time=0.299 ms
64 bytes from 192.168.1.1: seq=2 ttl=64 time=0.306 ms
^C
--- 192.168.1.1 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 0.299/0.306/0.313 ms
$

## Able to ping externa gateway (overcloud IP)
$ ping 192.168.120.1
PING 192.168.120.1 (192.168.120.1): 56 data bytes
64 bytes from 192.168.120.1: seq=0 ttl=63 time=0.413 ms
64 bytes from 192.168.120.1: seq=1 ttl=63 time=0.445 ms
64 bytes from 192.168.120.1: seq=2 ttl=63 time=0.418 ms
^C
--- 192.168.120.1 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 0.413/0.425/0.445 ms
$
$

## Unable to ping external websites

$ ping google.com
^C
$


## Listing router on controller node
[root@controller-0 ~]# ip netns list
qrouter-a9c9e33d-f9d6-4f2f-875d-d6929e7f1cd9 (id: 2)
qdhcp-e8cb20d8-1587-4a69-93db-6eaba437c607 (id: 1)
qdhcp-3a419b28-b707-4dec-a724-9bbbb840eeef (id: 0)
[root@controller-0 ~]#

## Getting route information from router namespace

[root@controller-0 ~]# ip netns exec qrouter-a9c9e33d-f9d6-4f2f-875d-d6929e7f1cd9 route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.120.1   0.0.0.0         UG    0      0        0 qg-d6210ed9-1a
192.168.1.0     0.0.0.0         255.255.255.0   U     0      0        0 qr-193df81c-c0
192.168.120.0   0.0.0.0         255.255.255.0   U     0      0        0 qg-d6210ed9-1a
[root@controller-0 ~]#

## Unable to ping external from router namespace

[root@controller-0 ~]# ip netns exec qrouter-a9c9e33d-f9d6-4f2f-875d-d6929e7f1cd9 ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
^C
--- 8.8.8.8 ping statistics ---
25 packets transmitted, 0 received, 100% packet loss, time 23999ms

[root@controller-0 ~]#


Expected results:

Access to the internet should be through from within the internet.

Additional info:

Comment 1 Assaf Muller 2018-06-27 13:59:17 UTC
Can you please supply sosreports from controllers? There's not much we can do about this bug report as it is.

Comment 2 karan singh 2018-06-28 18:29:09 UTC
John pointed me to a (long) BZ, where one of the comment said “OVS container had some issues, try restart them”, so I did that. Soon after that, the VMs started to talk to the external network (internet) and I was happy for a while, when I realised that I lost access to openstack (CLI/GUI) as keystone is not available.
 
After some troubleshooting, I found that (For some unknown reason) when I assigned floating IP to my VM, the floating ip create mechanism, hard acquired keystone IP and wired that to my VM. As a result I lost access to keystone.
 
To fix that, I had to login onto controller node manually deleted all the network namespaces associated to that VM, removed the floating IP. Restarted OVS, 5 other services and then had to reboot controller. Once controller node is back, I regained access to keystone. Next I changes floating IP pool range so that it should not hard assign used IPs. After that my VMs were able to access internet.


Note You need to log in before you can comment on or make changes to this bug.