Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1364980 - upgrade does not remove gutterball tomcat web application
Summary: upgrade does not remove gutterball tomcat web application
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Satellite 6
Classification: Red Hat
Component: Upgrades
Version: 6.2.0
Hardware: All
OS: Linux
medium
medium vote
Target Milestone: 6.2
Assignee: Chris Roberts
QA Contact: Lukas Pramuk
URL: http://projects.theforeman.org/issues...
Whiteboard:
Depends On: 1410783
Blocks: 1405535
TreeView+ depends on / blocked
 
Reported: 2016-08-08 10:06 UTC by Pavel Moravec
Modified: 2018-12-06 20:49 UTC (History)
10 users (show)

Fixed In Version: katello-installer-base-3.0.0.69-1
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1405535 (view as bug list)
Environment:
Last Closed: 2017-01-26 10:41:06 UTC


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Foreman Issue Tracker 16864 None None None 2016-10-11 15:26:08 UTC
Red Hat Bugzilla 1375659 None None None Never
Red Hat Knowledge Base (Solution) 2608341 None None None 2016-10-25 19:23:52 UTC
Red Hat Product Errata RHBA-2017:0197 normal SHIPPED_LIVE Satellite 6.2.7 Async Bug Release 2017-01-26 15:38:38 UTC

Internal Links: 1375659

Description Pavel Moravec 2016-08-08 10:06:58 UTC
Description of problem:
Despite (fresh installation of) Sat6.2 has removed gutterball from its deployment, an upgrade from 6.1/6.0 does not remove it. postgres gets gutterball connections, tomcat deploys it as an application,..


Version-Release number of selected component (if applicable):
satellite-installer-6.2.0.11-1.el7sat.noarch
satellite-6.2.0-21.1.el7sat.noarch
katello-installer-base-3.0.0.56-1.el7sat.noarch
foreman-installer-1.11.0.9-1.el7sat.noarch
foreman-installer-katello-3.0.0.56-1.el7sat.noarch


How reproducible:
100%


Steps to Reproduce:
1. Install 6.1
2. Upgrade to 6.2
3. Check /var/log/tomcat/catalina.* log
4. Check "ps aux | grep gutter"


Actual results:
3. finds:
Aug 02, 2016 8:11:56 AM org.apache.catalina.startup.HostConfig deployDirectory
INFO: Deployment of web application directory /var/lib/tomcat/webapps/gutterball has finished in 6,969 ms

4. finds:
postgres 30890  0.0  0.0 233452  4312 ?        Ss   10:00   0:00 postgres: gutterball gutterball 127.0.0.1(50250) idle


Expected results:
No gutterball after upgrade to 6.2

Additional info:

Comment 3 Chris Roberts 2016-10-11 15:26:09 UTC
upstream redmine issue created:

http://projects.theforeman.org/issues/16864

Comment 4 Bryan Kearney 2016-10-13 16:07:40 UTC
Moving this bug to POST for triage into Satellite 6 since the upstream issue http://projects.theforeman.org/issues/16864 has been resolved.

Comment 6 Pavel Moravec 2016-10-25 14:10:07 UTC
I think the fix lacks one important step:

rm -rf /var/lib/tomcat*/webapps/gutterball

(rhel7 has "tomcat" dir, rhel6 has "tomcat6", hence the "tomcat*")

For upgrades from 6.1, that directory remains undeleted, so tomcat still deploys gutterball app despite no gutterball RPM present, no gutterball DB. And that still subscribes to the $(fqdn):event qpid queue and still causes https://access.redhat.com/solutions/2608341 - confirmed in field already.

Bryan, does it make sense to move to NEW? Since the current fix does not fix e.g. the BZ subject :).

Comment 7 Mike McCune 2016-10-25 14:12:13 UTC
Moving this back to ASSIGNED as we need to remove the webapp as Pavel describes

Comment 8 Pavel Moravec 2016-11-01 06:58:34 UTC
Hi Toledo,
will the fix contain removal of $(fqdn):event queue from qpid broker as well, such that https://bugzilla.redhat.com/show_bug.cgi?id=1375659 is resolved?

(we know that presence of the queue is the culprit of most problems, thats why I am asking)

Comment 13 Chris Roberts 2016-11-04 15:40:14 UTC
Upgrade Step: remove_event_queue...
[ INFO 2016-11-02 15:41:03 verbose] Upgrade Step: remove_event_queue...

Upgrade completed!
[ INFO 2016-11-02 15:41:04 verbose] Upgrade completed!
[ INFO 2016-11-02 15:41:04 verbose] All hooks in group post finished

Gutterball DB removed in pre hook:

                                  List of databases
   Name    |  Owner   | Encoding |   Collate   |    Ctype    |   Access privileges    
-----------+----------+----------+-------------+-------------+------------------------
 candlepin | postgres | UTF8     | en_US.utf8  | en_US.utf8  | =T/postgres           +
           |          |          |             |             | postgres=CTc/postgres +
           |          |          |             |             | candlepin=CTc/postgres
 foreman   | foreman  | UTF8     | en_US.utf8  | en_US.utf8  | =T/foreman            +
           |          |          |             |             | foreman=CTc/foreman
 postgres  | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 | 
 template0 | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 | =c/postgres           +
           |          |          |             |             | postgres=CTc/postgres
 template1 | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 | =c/postgres           +
           |          |          |             |             | postgres=CTc/postgres
(5 rows)


Webapp directory removed

[root@rhel7 ~]# ls /var/lib/tomcat/webapps/
candlepin

event queue removed

[root@rhel7 ~]# qpid-stat --ssl-certificate /etc/pki/katello/certs/java-client.crt --ssl-key /etc/pki/katello/private/java-client.key -b "amqps://localhost:5671" -q
Queues
  queue                                                       dur  autoDel  excl  msg   msgIn  msgOut  bytes  bytesIn  bytesOut  cons  bind
  ===========================================================================================================================================
  19c6bb96-1fb4-4d32-a06c-6f1cd572b8a1:1.0                         Y        Y        0     8      8       0   4.86k    4.86k        1     2
  19c6bb96-1fb4-4d32-a06c-6f1cd572b8a1:2.0                         Y        Y        0     4      4       0   2.50k    2.50k        1     2
  2b791125-ea03-435a-930b-e81bcc242a01:1.0                         Y        Y        0     2      2       0    486      486         1     2
  4b11fbdc-bed7-4424-9e7b-b28bf0e87b6b:1.0                         Y        Y        0     2      2       0    486      486         1     2
  6eb77d4f-4ad9-41c1-9502-b22870e9821a:1.0                         Y        Y        0     5      5       0   2.67k    2.67k        1     2
  9772ea6e-d763-492e-8780-b3e5091ebec0:1.0                         Y        Y        0     2      2       0    486      486         1     2
  ac2e209c-25e2-4654-8ef5-16408831a921:1.0                         Y        Y        0     0      0       0      0        0         1     2
  b3a19f42-9660-442e-b476-11746b679ed0:0.0                         Y        Y        0     0      0       0      0        0         1     2
  celery                                                      Y                      0     1      1       0    827      827         2     2
  celeryev.0db6053f-dad5-4076-9dd3-33fe5ca6949b                    Y                 0    27     27       0   22.0k    22.0k        1     2
  d2ae1e15-0f85-483b-89f5-c7860499e801:1.0                         Y        Y        0     8      8       0   4.88k    4.88k        1     2
  d2ae1e15-0f85-483b-89f5-c7860499e801:2.0                         Y        Y        0     4      4       0   2.48k    2.48k        1     2
  e34154dc-19e7-48e4-bf53-cde1092a0cf4:1.0                         Y        Y        0     8      8       0   4.86k    4.86k        1     2
  e34154dc-19e7-48e4-bf53-cde1092a0cf4:2.0                         Y        Y        0     4      4       0   2.50k    2.50k        1     2
  f7afcfdf-610d-4033-8053-5ae5aaa9d9d9:1.0                         Y        Y        0     4      4       0   2.42k    2.42k        1     2
  katello_event_queue                                         Y                      0     0      0       0      0        0         1     2
  pulp.task                                                   Y                      0     0      0       0      0        0         3     1
  reserved_resource_worker-0@rhel7.example.com.celery.pidbox       Y                 0     0      0       0      0        0         1     2
  reserved_resource_worker-0@rhel7.example.com.dq             Y    Y                 0     0      0       0      0        0         1     2
  reserved_resource_worker-1@rhel7.example.com.celery.pidbox       Y                 0     0      0       0      0        0         1     2
  reserved_resource_worker-1@rhel7.example.com.dq             Y    Y                 0     0      0       0      0        0         1     2
  resource_manager                                            Y                      0     0      0       0      0        0         1     2
  resource_manager@rhel7.example.com.celery.pidbox                 Y                 0     0      0       0      0        0         1     2
  resource_manager@rhel7.example.com.dq                       Y    Y                 0     0      0       0      0        0         1     2

Will fix changes and resubmit PR

Comment 14 Lukas Pramuk 2017-01-16 11:28:27 UTC
FailedQA.

@satellite-6.2.7-1.0.el7sat.noarch
katello-installer-base-3.0.0.69-1.el7sat.noarch

Doing upgrade 6.1.11 > 6.2.7

# satellite-installer -S satellite --upgrade
...

Upgrade Step: remove_gutterball...
dropdb: database removal failed: ERROR:  database "gutterball" is being accessed by other users
DETAIL:  There are 5 other sessions using the database.

Upgrade step remove_gutterball failed. Check logs for more information.


# qpid-stat -q --ssl-certificate=/etc/pki/katello/qpid_client_striped.crt -b amqps://localhost:5671 | grep :event 
  <SAT_FQDN>:event           Y             Y        0     0      0       0      0        0         1     2


It seems that fix from BZ 1382524 (Upgrade step:remove_event_queue) should be (also) run before dropping gutterball db

Comment 15 Lukas Pramuk 2017-01-16 11:34:01 UTC
Nope. Deleting qpid queue didn't helped. (Not related to remove_event_queue)

# qpid-config --ssl-certificate=/etc/pki/katello/qpid_client_striped.crt -b amqps://localhost:5671 del queue $(hostname -f):event --force

# qpid-stat -q --ssl-certificate=/etc/pki/katello/qpid_client_striped.crt -b amqps://localhost:5671 | grep :event 
<empty>

# satellite-installer -S satellite --upgrade
...
Upgrade Step: remove_gutterball...
dropdb: database removal failed: ERROR:  database "gutterball" is being accessed by other users
DETAIL:  There are 5 other sessions using the database.

Upgrade step remove_gutterball failed. Check logs for more information.

Comment 16 Lukas Pramuk 2017-01-16 12:02:35 UTC
There are 5 connections to the database preventing its drop.

# runuser - postgres -c "psql -c 'select * from pg_stat_activity'" |grep gutterball
 17661 | gutterball | 125594 |    16384 | gutterball |                                       | 127.0.0.1   |                 |       38194 | 2017-01-16 06:37:35.375769-05 |                               | 2017-01-16 06:37:35.380252-05 | 2017-01-16 06:37:35.380286-05 | f       | idle   | SHOW TRANSACTION ISOLATION LEVEL
 17661 | gutterball | 125595 |    16384 | gutterball |                                       | 127.0.0.1   |                 |       38197 | 2017-01-16 06:37:35.376306-05 |                               | 2017-01-16 06:37:35.381788-05 | 2017-01-16 06:37:35.381815-05 | f       | idle   | SHOW TRANSACTION ISOLATION LEVEL
 17661 | gutterball | 125596 |    16384 | gutterball |                                       | 127.0.0.1   |                 |       38196 | 2017-01-16 06:37:35.376557-05 |                               | 2017-01-16 06:37:35.381747-05 | 2017-01-16 06:37:35.381797-05 | f       | idle   | SHOW TRANSACTION ISOLATION LEVEL
 17661 | gutterball | 125597 |    16384 | gutterball |                                       | 127.0.0.1   |                 |       38200 | 2017-01-16 06:37:35.382417-05 |                               | 2017-01-16 06:37:35.386034-05 | 2017-01-16 06:37:35.386054-05 | f       | idle   | SHOW TRANSACTION ISOLATION LEVEL
 17661 | gutterball | 125598 |    16384 | gutterball |                                       | 127.0.0.1   |                 |       38202 | 2017-01-16 06:37:35.383537-05 |                               | 2017-01-16 06:37:35.387371-05 | 2017-01-16 06:37:35.387393-05 | f       | idle   | SHOW TRANSACTION ISOLATION LEVEL

Comment 17 Lukas Pramuk 2017-01-16 12:22:36 UTC
These connections are coming out of gutterball webapp, thus:

# rm -rf /var/lib/tomcat/webapps/gutterball

And after a while (5secs or so)

# runuser - postgres -c "psql -c 'select * from pg_stat_activity'" |grep gutterball
<empty>


We better start with removing gutterball webapp and then continue with gutterball DB drop

Comment 19 Pavel Moravec 2017-01-16 12:31:41 UTC
(In reply to Lukas Pramuk from comment #17)
> These connections are coming out of gutterball webapp, thus:
> 
> # rm -rf /var/lib/tomcat/webapps/gutterball
> 
> And after a while (5secs or so)
> 
> # runuser - postgres -c "psql -c 'select * from pg_stat_activity'" |grep
> gutterball
> <empty>
> 
> 
> We better start with removing gutterball webapp and then continue with
> gutterball DB drop

I *think* safer variant would be stopping tomcat before removing that dir. Just to ensure the running instance does not cache to memory the bits you delete.

(maybe that is already done in some previous step, then let ignore my comment)

Comment 23 Lukas Pramuk 2017-01-18 21:23:25 UTC
VERIFIED.

@satellite-6.2.7-1.0.el7sat.noarch
katello-installer-base-3.0.0.70-1.el7sat.noarch

6.1.11 > 6.2.7

# satellite-installer -S satellite --upgrade
...

Upgrade Step: migrate_candlepin...
Migrating candlepin database
Liquibase Update Successful

Upgrade Step: remove_gutterball...

Upgrade Step: start_tomcat...
Redirecting to /bin/systemctl start  tomcat.service

Success!

Upgrade Step: migrate_foreman...
true


false

Upgrade Step: Running installer...
...


Upgrade Step: remove_gutterball...
gutterball-1.0.15.0.2-1.el7.noarch

No Match for argument: tfm-rubygem-foreman_gutterball
No Match for argument: tfm-rubygem-hammer_cli_gutterball

================================================================================
 Package        Arch   Version          Repository                         Size
================================================================================
Removing:
 gutterball     noarch 1.0.15.0.2-1.el7 @rhel-7-server-satellite-6.1-rpms 301 k
 gutterball-certs
                noarch 1.0-1            installed                          11 k

Transaction Summary
================================================================================
Remove  2 Packages

Installed size: 312 k
warning: /etc/gutterball/gutterball.conf saved as /etc/gutterball/gutterball.conf.rpmsave

Removed:
  gutterball.noarch 0:1.0.15.0.2-1.el7      gutterball-certs.noarch 0:1.0-1     


removed directory: ‘/var/lib/tomcat/webapps/gutterball’
...

>>> Upgrades finish successfully


# ls /var/lib/tomcat{6,}/webapps/gutterball
ls: cannot access /var/lib/tomcat6/webapps/gutterball: No such file or directory
ls: cannot access /var/lib/tomcat/webapps/gutterball: No such file or directory

>>> gutterball webapp is undeployed


# runuser - postgres -c "psql -l" |grep gutterball
<empty>

>>> gutterball DB is dropped

Comment 25 errata-xmlrpc 2017-01-26 10:41:06 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:0197


Note You need to log in before you can comment on or make changes to this bug.