Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1518802 - upgrade 4.1.8 to 4.2 with answer file fails on upgrade postgres issue
Summary: upgrade 4.1.8 to 4.2 with answer file fails on upgrade postgres issue
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: ovirt-engine
Classification: oVirt
Component: Setup.Engine
Version: 4.2.0
Hardware: Unspecified
OS: Unspecified
medium
high vote
Target Milestone: ovirt-4.2.2
: ---
Assignee: Simone Tiraboschi
QA Contact: Lucie Leistnerova
URL:
Whiteboard:
Depends On:
Blocks: 1631202
TreeView+ depends on / blocked
 
Reported: 2017-11-29 15:09 UTC by Lucie Leistnerova
Modified: 2018-12-10 09:24 UTC (History)
9 users (show)

Fixed In Version: ovirt-engine-4.2.2.4
Doc Type: No Doc Update
Doc Text:
undefined
Clone Of:
Environment:
Last Closed: 2018-03-29 11:10:25 UTC
oVirt Team: Integration
rule-engine: ovirt-4.2+


Attachments (Terms of Use)
answerfile (deleted)
2017-11-29 15:09 UTC, Lucie Leistnerova
no flags Details
setup log (deleted)
2017-11-29 15:10 UTC, Lucie Leistnerova
no flags Details
upgrade log 4.1.9 -> 4.2.1 (deleted)
2018-01-26 13:40 UTC, Lucie Leistnerova
no flags Details


Links
System ID Priority Status Summary Last Updated
oVirt gerrit 86830 master MERGED dbms: upgrade: compare engine DBMS port and DWH one as string 2018-02-27 13:12:21 UTC
oVirt gerrit 86868 master MERGED core: dump env keys also if only type changed 2018-01-29 13:48:41 UTC
oVirt gerrit 88253 ovirt-engine-dwh-4.2 MERGED dbms: upgrade: compare engine DBMS port and DWH one as string 2018-02-27 17:21:09 UTC
Red Hat Bugzilla 1535935 None CLOSED engine-setup creates bad 10-setup-database.conf if it has to provision ovirt_engine_history_TIMESTAMP 2019-02-05 11:59:07 UTC

Internal Links: 1535935

Description Lucie Leistnerova 2017-11-29 15:09:12 UTC
Created attachment 1360387 [details]
answerfile

Description of problem:
When I run engine-setup with answerfile, it fails with error
[ ERROR ] Failed to execute stage 'Misc configuration': Command '/opt/rh/rh-postgresql95/root/usr/bin/postgresql-setup' failed to execute

Version-Release number of selected component (if applicable):
ovirt-engine-setup-4.1.8.1-0.1.el7.noarch ->
ovirt-engine-setup-4.2.0-0.5.master.el7.noarch

How reproducible: always with the answerfile


Steps to Reproduce:
1. have running 4.1.8 engine
2. run engine-setup --config-append=<answer.file>
3. enter:
This tool can automatically upgrade PostgreSQL. Automatically upgrade? (Yes, No) [Yes]: 
Do you want to perform an in-place upgrade? (Yes, No) [No]: 
Do you want to automatically clean up the old data directory on success to reclaim its space (64 MB)? (Yes, No) [Yes]: no


Actual results: upgrade fails


Expected results: upgrade is successful


Additional info:
without answerfile upgrade is OK
answerfile and setup log attached

Comment 1 Lucie Leistnerova 2017-11-29 15:10:17 UTC
Created attachment 1360388 [details]
setup log

Comment 2 Yaniv Kaul 2017-11-30 08:02:44 UTC
Why not perform in-place upgrade?

Comment 3 Lucie Leistnerova 2017-11-30 10:00:14 UTC
We tried the in-place and it failed with the same error
ERROR: Data directory /var/opt/rh/rh-postgresql95/lib/pgsql/data is not empty!

We used default options too and the engine-setup failed with error, that /var/lib/pgsql/data does not exist. So we tried to not delete it and it failed also.

Comment 4 Simone Tiraboschi 2017-12-13 13:41:37 UTC
The error is here:

2017-11-29 16:04:22,624+0200 DEBUG otopi.plugins.ovirt_engine_setup.ovirt_engine_dwh.db.dbmsupgrade plugin.execute:926 execute-output: ('/opt/rh/rh-postgresql95/root/usr/bin/postgresql-setup', '--upgrade', '--upgrade-from=postgresql') stderr:
 * upgrading from 'postgresql.service' to 'rh-postgresql95-postgresql.service'
 * Upgrading database.
ERROR: Data directory /var/opt/rh/rh-postgresql95/lib/pgsql/data is not empty!
ERROR: Upgrade failed.
 * See /var/lib/pgsql/upgrade_rh-postgresql95-postgresql.log for details.

but on a clean env /var/opt/rh/rh-postgresql95/lib/pgsql/data should be empty

Comment 5 Lucie Leistnerova 2018-01-26 13:39:27 UTC
The problem seems not to be in the environment and clean rh. We run into similar issue again with upgrade engine 4.1.9.1 -> 4.2.1.3.

It ran the upgrade to postgres 9.5 twice, like in the setup log I provided previously.

first upgrade:
2018-01-26 07:47:33,147+0200 DEBUG otopi.plugins.ovirt_engine_setup.ovirt_engine.db.dbmsupgrade plugin.execute:926 execute-output: ('/opt/rh/rh-postgresql95/root/usr/bin/postgresql-setup', '-upgrade', '-upgrade-from=postgresql') stderr:

    upgrading from 'postgresql.service' to 'rh-postgresql95-postgresql.service'
    Upgrading database.
    Upgraded OK.
    WARNING: The configuration files were replaced by default configuration.
    WARNING: The previous configuration and data are stored in folder
    WARNING: /var/lib/pgsql/data.
    See /var/lib/pgsql/upgrade_rh-postgresql95-postgresql.log for details.
    2018-01-26 07:47:33,148+0200 INFO otopi.plugins.ovirt_engine_setup.ovirt_engine.db.dbmsupgrade postgres.prepare:793 PostgreSQL has been successfully upgraded, starting the new instance (rh-postgresql95-postgresql).

delete old db:
2018-01-26 07:47:34,338+0200 INFO otopi.plugins.ovirt_engine_setup.ovirt_engine.db.dbmsupgrade postgres.commit:848 Cleaning the previous PostgreSQL data directory

second upgrade:
2018-01-26 07:47:34,511+0200 DEBUG otopi.plugins.ovirt_engine_setup.ovirt_engine_dwh.db.dbmsupgrade plugin.execute:926 execute-output: ('/opt/rh/rh-postgresql95/root/usr/bin/postgresql-setup', '-upgrade', '-upgrade-from=postgresql') stderr:

    upgrading from 'postgresql.service' to 'rh-postgresql95-postgresql.service'
    ERROR: config file /var/lib/pgsql/data/postgresql.conf is not readable or does not exist
    FATAL: Old cluster in '/var/lib/pgsql/data' does not seem to be initialized

Comment 6 Lucie Leistnerova 2018-01-26 13:40:16 UTC
Created attachment 1386551 [details]
upgrade log 4.1.9 -> 4.2.1

Comment 7 Simone Tiraboschi 2018-01-26 14:48:24 UTC
The issue seams here:

2018-01-26 07:47:09,493+0200 INFO otopi.plugins.ovirt_engine_setup.ovirt_engine_dwh.db.dbmsupgrade dbmsupgrade._customization:87 Engine DB and DWH one are on two distinct PostgreSQL instances and both of them have to be upgraded.

The setup assumes that Engine DB and DWH DB are on two distinct DBMS instances while instead they are on the same DBMS.
This check https://github.com/oVirt/ovirt-dwh/blob/master/packaging/setup/plugins/ovirt-engine-setup/ovirt-engine-dwh/db/dbmsupgrade.py#L69 for some reason fails.

I think it could be just another side effects of https://bugzilla.redhat.com/show_bug.cgi?id=1535935

Didi?

Comment 8 Simone Tiraboschi 2018-01-26 14:55:35 UTC
OK,
I got it!

In your answer-file you have:
OVESETUP_DB/port=str:5432
OVESETUP_DWH_DB/port=int:5432

and so the check on https://github.com/oVirt/ovirt-dwh/blob/master/packaging/setup/plugins/ovirt-engine-setup/ovirt-engine-dwh/db/dbmsupgrade.py#L69 simply failed due to the different data type (str vs int).

Lucie, how did you got that answerfile?

Comment 9 Simone Tiraboschi 2018-01-26 15:03:50 UTC
On a plain answer file from 4.1.8 engine-setup I got 
 OVESETUP_DB/port=int:5432
 OVESETUP_DWH_DB/port=int:5432
so it should work as expected there.

Not sure on how you got
 OVESETUP_DB/port=str:5432

Comment 10 Yedidyah Bar David 2018-01-28 06:50:40 UTC
Nice catch, but I wonder if we should somehow make the test more robust, if possible. Such as mark the pg instance as "upgraded" inside itself (perhaps pg already has a way to do this) and then check again before actually upgrading.

Comment 11 Lucie Leistnerova 2018-01-29 09:45:19 UTC
I'm not sure where I got the first answer file, but now in tests it was generated with ansible from template
https://github.com/oVirt/ovirt-ansible-engine-setup/blob/master/templates/answerfile_4.2_upgrade.txt.j2
but it is deleted in the end of upgrade and I have only the one generated with egine-setup after. And as you say, in the answer file is mixed db port as int and str.

Comment 13 Lucie Leistnerova 2018-03-20 07:10:44 UTC
4.2 answer file generated with port=str:xxx not int. When old answer file is used with mixed type by the port, upgrade did not fail (both engine+dwh and separate dwh).
Also upgrade automation tests have no problem with answer file.

verified in ovirt-engine-4.2.2.4-0.1.el7.noarch

Comment 14 Sandro Bonazzola 2018-03-29 11:10:25 UTC
This bugzilla is included in oVirt 4.2.2 release, published on March 28th 2018.

Since the problem described in this bug report should be
resolved in oVirt 4.2.2 release, it has been closed with a resolution of CURRENT RELEASE.

If the solution does not work for you, please open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.