Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1510969 - errors during the minor update to 5.8.2 on the ansible packages
Summary: errors during the minor update to 5.8.2 on the ansible packages
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Red Hat CloudForms Management Engine
Classification: Red Hat
Component: Appliance
Version: 5.8.0
Hardware: All
OS: All
medium
medium
Target Milestone: GA
: cfme-future
Assignee: Nick Carboni
QA Contact: luke couzens
URL:
Whiteboard: ansible_embed:black:upgrade
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-11-08 13:37 UTC by Felix Dewaleyne
Modified: 2017-11-29 20:15 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-11-29 20:15:30 UTC
Category: ---
Cloudforms Team: ---


Attachments (Terms of Use)

Description Felix Dewaleyne 2017-11-08 13:37:54 UTC
Created attachment 1349451 [details]
yum_output.txt

Description of problem:
During the minor upgrade to 5.8.2 errors were raised that look tied to ansible.

Version-Release number of selected component (if applicable):
5.8.2

How reproducible:
specific setup required

Steps to Reproduce:
1. deploy a 5.8.1
2. upgrade to 5.8.2
3.

Actual results:
Running transaction
Traceback (most recent call last):
  File "/bin/virtualenv", line 3, in <module>
    virtualenv.main()
  File "/usr/lib/python2.7/site-packages/virtualenv.py", line 825, in main
    symlink=options.symlink)
  File "/usr/lib/python2.7/site-packages/virtualenv.py", line 960, in create_environment
    site_packages=site_packages, clear=clear, symlink=symlink))
  File "/usr/lib/python2.7/site-packages/virtualenv.py", line 1251, in install_python
    shutil.copyfile(executable, py_executable)
  File "/usr/lib64/python2.7/shutil.py", line 83, in copyfile
    with open(dst, 'wb') as fdst:
IOError: [Errno 26] Text file busy: '/var/lib/awx/venv/tower/bin/python'
...
  Cleanup    : ansible-tower-server-3.1.3-1.el7at.x86_64                                                                   10/12
remove failed: No such file or directory
warning: file /var/lib/awx/venv/tower/lib/python2.7/site-packages/ansible_tower-3.1.3-py2.7.egg-info/SOURCES.txt: remove failed: No such file or directory
warning: file /var/lib/awx/venv/tower/lib/python2.7/site-packages/ansible_tower-3.1.3-py2.7.egg-info/PKG-INFO: remove failed: No such file or directory
warning: file /var/lib/awx/venv/tower/lib/python2.7/site-packages/ansible_tower-3.1.3-py2.7.egg-info: remove failed: No such file or directory

Expected results:
the upgrade completes with no error

Additional info:
full yum trace avaialble as attachment

got ansible collab :
>That error there usually comes when some other process had that file open when >the upgrade tried to replace it. 
>
>But, we dont follow this procedure to upgrade Ansible Tower. We have a setup.sh >which comes with the Ansible Tower installer and that need to be run when you >want upgrade your Ansible Tower. If you do a upgrade of Ansible Tower rpm >without running setup.sh, database migration will not happen and tower instance >may end up unusable. 
>
>So when you install Cloudforms do you run setup.sh for installing Ansible Tower >? Do you install a Ansible Tower instance with cloudforms installation  ? Do >you need the Ansible Tower database ?

from that it seems that the problem is in the scripts of the rpm we ship.

Comment 2 Felix Dewaleyne 2017-11-08 16:45:16 UTC
the  evmserverd was stopped on the worker and UI appliances before proceeding with the yum update.

Comment 3 Felix Dewaleyne 2017-11-15 12:14:24 UTC
Another different trace can happen :

[...]
  Updating   : ansible-tower-server-3.1.5-1.el7at.x86_64                                                                    1/12
tower not found
  Updating   : cfme-gemset-5.8.2.3-1.el7cf.x86_64                                                                           2/12
  Updating   : rh-ruby23-rubygem-nokogiri-1.8.1-2.el7cf.x86_64                                                              3/12
  Updating   : ansible-tower-setup-3.1.5-1.el7at.x86_64                                                                     4/12
  Updating   : cfme-5.8.2.3-1.el7cf.x86_64                                                                                  5/12
/var/www/miq/vmdb/certs /
/
  Updating   : cfme-appliance-5.8.2.3-1.el7cf.x86_64                                                                        6/12
/etc/httpd/conf.d /
/
  Cleanup    : cfme-appliance-5.8.1.5-1.el7cf.x86_64                                                                        7/12
  Cleanup    : cfme-gemset-5.8.1.5-13.el7cf.x86_64                                                                          8/12
  Cleanup    : cfme-5.8.1.5-11.el7cf.x86_64                                                                                 9/12
  Cleanup    : ansible-tower-server-3.1.3-1.el7at.x86_64                                                                   10/12
warning: file /var/lib/awx/venv/tower/lib/python2.7/site-packages/ansible_tower-3.1.3-py2.7.egg-info/top_level.txt: remove failed: No such file or directory
warning: file /var/lib/awx/venv/tower/lib/python2.7/site-packages/ansible_tower-3.1.3-py2.7.egg-info/not-zip-safe: remove failed: No such file or directory
warning: file /var/lib/awx/venv/tower/lib/python2.7/site-packages/ansible_tower-3.1.3-py2.7.egg-info/entry_points.txt: remove failed: No such file or directory
warning: file /var/lib/awx/venv/tower/lib/python2.7/site-packages/ansible_tower-3.1.3-py2.7.egg-info/dependency_links.txt: remove failed: No such file or directory
warning: file /var/lib/awx/venv/tower/lib/python2.7/site-packages/ansible_tower-3.1.3-py2.7.egg-info/SOURCES.txt: remove failed: No such file or directory
warning: file /var/lib/awx/venv/tower/lib/python2.7/site-packages/ansible_tower-3.1.3-py2.7.egg-info/PKG-INFO: remove failed: No such file or directory
warning: file /var/lib/awx/venv/tower/lib/python2.7/site-packages/ansible_tower-3.1.3-py2.7.egg-info: remove failed: No such file or directory
  Cleanup    : ansible-tower-setup-3.1.3-1.el7at.x86_64                         

full messages in attachments

Comment 5 Nick Carboni 2017-11-15 21:40:02 UTC
I can't seem to reproduce this issue.

When I ran an upgrade from 5.8.1 to 5.8.2 none of the ansible packages were upgraded.

It's possible that the packages were not properly installed during a previous upgrade. Can you check the current version of the packages using `rpm -qa | grep ansible`?

Also can you make sure that the appliance file systems were resized on upgrade?

In appliances with less space in the /var/ filesystem sometimes will not be able to install the ansible packages.

The instructions on resizing the filesystem can be found in https://access.redhat.com/documentation/en-us/red_hat_cloudforms/4.5/html/migrating_to_red_hat_cloudforms_4.5/index#resize-disks41

Comment 6 Felix Dewaleyne 2017-11-22 10:42:56 UTC
(In reply to Nick Carboni from comment #5)
> I can't seem to reproduce this issue.
> 
> When I ran an upgrade from 5.8.1 to 5.8.2 none of the ansible packages were
> upgraded.
> 
> It's possible that the packages were not properly installed during a
> previous upgrade. Can you check the current version of the packages using
> `rpm -qa | grep ansible`?
> 
> Also can you make sure that the appliance file systems were resized on
> upgrade?
> 
> In appliances with less space in the /var/ filesystem sometimes will not be
> able to install the ansible packages.
> 
> The instructions on resizing the filesystem can be found in
> https://access.redhat.com/documentation/en-us/red_hat_cloudforms/4.5/html/
> migrating_to_red_hat_cloudforms_4.5/index#resize-disks41

my test appliance was deployed from the 4.5.0 image, 5gb were added to the /var partition and its upgrade to 4.5.1 was successful at the time. from the yum output, an older version of the ansible rpm was installed in both situations and the upgrade of it caused the messages to appear. that said, ansible tower was not enabled on those through cloudforms on the appliances used.

Comment 7 Felix Dewaleyne 2017-11-22 10:53:04 UTC
That said the virtualenv messages were not present on my reproducer. I did some search with yum history and I can confirm I did not hit the virtualenv side but I did hit all the removefailed messages : 

Scriptlet output:
   1 Stopping rabbitmq-server (via systemctl):  [  OK  ]
   2 d/conf.d /
   3 /
   4 warning: file /var/lib/awx/venv/tower/lib/python2.7/site-packages/ansible_tower-3.1.3-py2.7.egg-info/top_level.txt: remove failed: No such file or directory
   5 warning: file /var/lib/awx/venv/tower/lib/python2.7/site-packages/ansible_tower-3.1.3-py2.7.egg-info/not-zip-safe: remove failed: No such file or directory
   6 warning: file /var/lib/awx/venv/tower/lib/python2.7/site-packages/ansible_tower-3.1.3-py2.7.egg-info/entry_points.txt: remove failed: No such file or directory
   7 warning: file /var/lib/awx/venv/tower/lib/python2.7/site-packages/ansible_tower-3.1.3-py2.7.egg-info/dependency_links.txt: remove failed: No such file or directory
   8 warning: file /var/lib/awx/venv/tower/lib/python2.7/site-packages/ansible_tower-3.1.3-py2.7.egg-info/SOURCES.txt: remove failed: No such file or directory
   9 warning: file /var/lib/awx/venv/tower/lib/python2.7/site-packages/ansible_tower-3.1.3-py2.7.egg-info/PKG-INFO: remove failed: No such file or directory
  10 warning: file /var/lib/awx/venv/tower/lib/python2.7/site-packages/ansible_tower-3.1.3-py2.7.egg-info: remove failed: No such file or directory

that seem to have showed up on every appliance.

Comment 9 Felix Dewaleyne 2017-11-22 13:48:55 UTC
(In reply to Nick Carboni from comment #5)
> I can't seem to reproduce this issue.
> 
> When I ran an upgrade from 5.8.1 to 5.8.2 none of the ansible packages were
> upgraded.
> 
> It's possible that the packages were not properly installed during a
> previous upgrade. Can you check the current version of the packages using
> `rpm -qa | grep ansible`?
> 
> Also can you make sure that the appliance file systems were resized on
> upgrade?
> 
> In appliances with less space in the /var/ filesystem sometimes will not be
> able to install the ansible packages.
> 
> The instructions on resizing the filesystem can be found in
> https://access.redhat.com/documentation/en-us/red_hat_cloudforms/4.5/html/
> migrating_to_red_hat_cloudforms_4.5/index#resize-disks41

none of the impacted systems were affected by this issue - confirmed

Comment 11 Nick Carboni 2017-11-22 14:27:57 UTC
(In reply to Felix Dewaleyne from comment #7)

> Scriptlet output:
>    1 Stopping rabbitmq-server (via systemctl):  [  OK  ]
>    2 d/conf.d /
>    3 /
>    4 warning: file
> /var/lib/awx/venv/tower/lib/python2.7/site-packages/ansible_tower-3.1.3-py2.
> 7.egg-info/top_level.txt: remove failed: No such file or directory
>    5 warning: file
> /var/lib/awx/venv/tower/lib/python2.7/site-packages/ansible_tower-3.1.3-py2.
> 7.egg-info/not-zip-safe: remove failed: No such file or directory
>    6 warning: file
> /var/lib/awx/venv/tower/lib/python2.7/site-packages/ansible_tower-3.1.3-py2.
> 7.egg-info/entry_points.txt: remove failed: No such file or directory
>    7 warning: file
> /var/lib/awx/venv/tower/lib/python2.7/site-packages/ansible_tower-3.1.3-py2.
> 7.egg-info/dependency_links.txt: remove failed: No such file or directory
>    8 warning: file
> /var/lib/awx/venv/tower/lib/python2.7/site-packages/ansible_tower-3.1.3-py2.
> 7.egg-info/SOURCES.txt: remove failed: No such file or directory
>    9 warning: file
> /var/lib/awx/venv/tower/lib/python2.7/site-packages/ansible_tower-3.1.3-py2.
> 7.egg-info/PKG-INFO: remove failed: No such file or directory
>   10 warning: file
> /var/lib/awx/venv/tower/lib/python2.7/site-packages/ansible_tower-3.1.3-py2.
> 7.egg-info: remove failed: No such file or directory
> 
> that seem to have showed up on every appliance.

I will give this another shot by testing an upgrade from 5.8.0 to 5.8.2 but it seems to me that these warnings wouldn't really be a problem. They are telling you that a file they are trying to remove isn't present, which is the state you're shooting for.

Is the embedded ansible role working as expected after the upgrade?

Comment 12 Nick Carboni 2017-11-22 19:56:02 UTC
I can reproduce the warnings about being unable to remove files which do not exist when upgrading from 5.8.0 to 5.8.2.

After the upgrade, I was able to successfully start the embedded ansible role and add a repository.

I am not able to reproduce the original traceback.
Is that still an ongoing issue? If not, I'm going to close this bug as NOTABUG as the warnings are not actually causing an issue.

Comment 13 Felix Dewaleyne 2017-11-29 10:51:44 UTC
(In reply to Nick Carboni from comment #12)
> I can reproduce the warnings about being unable to remove files which do not
> exist when upgrading from 5.8.0 to 5.8.2.
> 
> After the upgrade, I was able to successfully start the embedded ansible
> role and add a repository.
> 
> I am not able to reproduce the original traceback.
> Is that still an ongoing issue? If not, I'm going to close this bug as
> NOTABUG as the warnings are not actually causing an issue.

fair enough. the impact of this has not been seen by the customer, who does not use ansible. he was concerned if it could have but from your update it doesn't seem to have any

Comment 14 Nick Carboni 2017-11-29 20:15:30 UTC
Thanks Felix, closing this.


Note You need to log in before you can comment on or make changes to this bug.