Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1515113 - sos doesn't support PostgreSQL95 from Software Collections [rhel-7.4.z]
Summary: sos doesn't support PostgreSQL95 from Software Collections [rhel-7.4.z]
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: sos
Version: 7.4
Hardware: Unspecified
OS: Unspecified
high
urgent
Target Milestone: rc
: ---
Assignee: Pavel Moravec
QA Contact: Miroslav Hradílek
URL:
Whiteboard:
: 1513071 1538365 (view as bug list)
Depends On: 1494420
Blocks: 1494419
TreeView+ depends on / blocked
 
Reported: 2017-11-20 07:30 UTC by Oneata Mircea Teodor
Modified: 2018-01-25 11:34 UTC (History)
14 users (show)

Fixed In Version: sos-3.4-12.el7_4
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1494420
Environment:
Last Closed: 2018-01-25 11:34:34 UTC


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2018:0138 normal SHIPPED_LIVE sos bug fix and enhancement update 2018-01-25 16:15:57 UTC
Github sosreport sos issues 1090 None None None 2017-11-20 07:30:20 UTC
Github sosreport sos issues 1169 None None None 2017-12-19 16:43:49 UTC

Description Oneata Mircea Teodor 2017-11-20 07:30:12 UTC
This bug has been copied from bug #1494420 and has been proposed to be backported to 7.4 z-stream (EUS).

Comment 2 Pavel Moravec 2017-11-20 10:54:41 UTC
committed to rhel-7.4 dist-git

Comment 4 Pavel Moravec 2017-11-20 11:17:28 UTC
Steps to verify:

ensure that sosreport collects the same data for postgreSQL installed from SCL as for "regular" / regularly installed postgreSQL. In particular, sosreport should collect:

1)
From pghome(*) dir, collect:
- all *log files under that dir
- all *conf files under that dir
- data/PG_VERSION
- data/postmaster.opts

(*) /var/lib/pgsql for regular postgres, /var/opt/rh/rh-postgresql95/lib/pgsql for SCL postgres


2)
In case env.variable PGPASSWORD is set(*) or sosreport is run with -k postgresql.password=somePassword, sosreport takes postgres dump via command like:

scl enable rh-postgresql95 -- pg_dump -U engine -h localhost -p 5432 -w -f /tmp/tmpXMhHFL/sos_scl_pgdump.tar -F t engine

and stores it in

sos_commands/postgresql/sos_scl_pgdump.tar


(*) ovirt-log-collector should set the password env.variable


Lukas, could you pls. verify this against the errata in previous comment?

Comment 5 Lukas Svaty 2017-11-20 14:56:02 UTC
tested in 
sos-3.4-10.el7_4.noarch
ovirt-log-collector-4.2.0-0.0.master.20171107155516.git3b8cca7.el7.centos.noarch

1) 
- logs - ok
|-- sos_logs
|   |-- sos.log
|   `-- ui.log
|-- var
|   |-- lib
|   |   `-- pgsql
|   |       `-- initdb_rh-postgresql95-postgresql.log
|   `-- opt
|       `-- rh
|           `-- rh-postgresql95
|               `-- lib
|                   `-- pgsql
|                       `-- data
|                           |-- pg_log
|                           |   |-- postgresql-Fri.log
|                           |   |-- postgresql-Mon.log
|                           |   |-- postgresql-Sat.log
|                           |   |-- postgresql-Sun.log
|                           |   |-- postgresql-Thu.log
|                           |   |-- postgresql-Tue.log
|                           |   `-- postgresql-Wed.log


- conf - ok
|-- var
|   `-- opt
|       `-- rh
|           `-- rh-postgresql95
|               `-- lib
|                   `-- pgsql
|                       `-- data
|                           |-- pg_hba.conf
|                           |-- pg_ident.conf
|                           |-- postgresql.auto.conf
|                           |-- postgresql.conf

- data/PG_VERSION - ok
  - 9.5

- data/postmaster.opts - ok
  - /opt/rh/rh-postgresql95/root/usr/bin/postgres "-D" "/var/opt/rh/rh-postgresql95/lib/pgsql/data"



2)
In case env.variable PGPASSWORD is set(*) or sosreport is run with -k postgresql.password=somePassword, sosreport takes postgres dump - FAIL

no dump available
sos.log:
2017-11-20 14:55:42,874 INFO: [sos.sosreport:setup] executing 'sosreport --batch -o postgresql --tmp-dir=/tmp/logcollector-LZsTXh/working/log-collector-data -k postgresql.dbname=engine -k postgresql.dbhost=localhost -k postgresql.dbport=5432 -k postgresql.username=engine'
2017-11-20 14:55:42,882 INFO: [plugin:postgresql] command 'pg_dump' not found in / - re-trying in host root
2017-11-20 14:55:42,888 INFO: [plugin:postgresql] Unable to execute pg_dump. Error()
2017-11-20 14:55:42,963 INFO: [plugin:postgresql] command 'scl' not found in / - re-trying in host root
2017-11-20 14:55:42,977 INFO: [plugin:postgresql] Unable to execute pg_dump. Error()


(*) ovirt-log-collector should set the password env.variable
# env | grep PGPASS
# 

Might be cause by this, but I suspect it is setting the var only for scope of ovirt-log-collector so should be ok


Pavel can you please take a look?

Comment 6 Pavel Moravec 2017-11-20 16:12:55 UTC
That is due to:

[Bug 1513071] Log Analyzer unable to detect sos_pgdump.tar

that is outside sosreport.

When I manually "fix" the path via adding lines:

                print "PATH=%s, pg_dump_command=%s" % (os.environ["PATH"], pg_dump_command)
                import re
                p = re.compile('pg_dump')
                pg_dump_command = p.sub('/opt/rh/rh-postgresql95/root/usr/bin/pg_dump', pg_dump_command)
                print "updated pg_dump_command to: %s" % pg_dump_command

to /usr/lib/python2.7/site-packages/sos/plugins/postgresql.py, then it starts to work well:


# env | grep PGPASSWORD
PGPASSWORD=*******************
# sosreport --batch -o postgresql -k postgresql.dbname=engine -k postgresql.dbhost=localhost -k postgresql.dbport=5432 -k postgresql.username=engine

sosreport (version 3.4)

This command will collect diagnostic and configuration information from
this Red Hat Enterprise Linux system and installed applications.

An archive containing the collected information will be generated in
/var/tmp/sos.tav5tE and may be provided to a Red Hat support
representative.

Any information provided to Red Hat will be treated in accordance with
the published support policies at:

  https://access.redhat.com/support/

The generated archive may contain data considered sensitive and its
content should be reviewed by the originating organization before being
passed to any third party.

No changes will be made to system configuration.


 Setting up archive ...
 Setting up plugins ...
PATH=/usr/sbin:/usr/bin:/root/bin:/usr/local/bin:/usr/local/sbin, pg_dump_command=pg_dump
updated pg_dump_command to: /opt/rh/rh-postgresql95/root/usr/bin/pg_dump
PATH=/usr/sbin:/usr/bin:/root/bin:/usr/local/bin:/usr/local/sbin, pg_dump_command=scl enable rh-postgresql95 -- pg_dump
updated pg_dump_command to: scl enable rh-postgresql95 -- /opt/rh/rh-postgresql95/root/usr/bin/pg_dump
 Running plugins. Please wait ...

  Running 1/1: postgresql...        

Creating compressed archive...

Your sosreport has been generated and saved in:
  /var/tmp/sosreport-system-ge-3.rhev.lab.eng.brq.redhat.com-20171120181030.tar.xz

The checksum is: fbf551e6bcc54ec32d1786ca016dee15

Please send this file to your support representative.

# tar xJvf /var/tmp/sosreport-system-ge-3.rhev.lab.eng.brq.redhat.com-20171120181030.tar.xz | grep dump
sosreport-system-ge-3.rhev.lab.eng.brq.redhat.com-20171120181030/tmp/tmpz1LqXE/sos_scl_pgdump.tar
sosreport-system-ge-3.rhev.lab.eng.brq.redhat.com-20171120181030/tmp/tmpIqB_OA/sos_pgdump.tar
# 


If the PATH would be set properly (or e.g. if I link pg_dump from /usr/bin), it will work without the above manual change. So I suppose this BZ is fixed in sosreport.

Comment 9 Pavel Moravec 2017-12-10 17:31:02 UTC
Newly committed changes due to bz1513071 & re-spinning build in the z-stream errata.

Comment 15 Pavel Moravec 2017-12-19 16:43:50 UTC
Perspective workaround patch for the generic "command output is treated as UTF-8 only" problem:


--- /usr/lib/python2.7/site-packages/sos/plugins/postgresql.py.orig	2017-12-19 17:39:00.076871338 +0100
+++ /usr/lib/python2.7/site-packages/sos/plugins/postgresql.py	2017-12-19 17:38:52.031781055 +0100
@@ -54,22 +54,33 @@ class PostgreSQL(Plugin):
                 if self.get_option("password") is not False:
                     os.environ["PGPASSWORD"] = str(self.get_option("password"))
 
+                self.tmp_dir = tempfile.mkdtemp()
+                dest_file = os.path.join(self.tmp_dir, filename)
                 if self.get_option("dbhost"):
-                    cmd = "pg_dump -U %s -h %s -p %s -w -F t %s" % (
+                    cmd = "pg_dump -U %s -h %s -p %s -w -f %s -F t %s" % (
                         self.get_option("username"),
                         self.get_option("dbhost"),
                         self.get_option("dbport"),
+                        dest_file,
                         self.get_option("dbname")
                     )
                 else:
-                    cmd = "pg_dump -C -U %s -w -F t %s " % (
+                    cmd = "pg_dump -C -U %s -w -f %s -F t %s " % (
                         self.get_option("username"),
+                        dest_file,
                         self.get_option("dbname")
                     )
 
                 if scl is not None:
                     cmd = self.convert_cmd_scl(scl, cmd)
-                self.add_cmd_output(cmd, suggest_filename=filename)
+                result = self.call_ext_prog(cmd)
+                if (result['status'] == 0):
+                    self.add_copy_spec(dest_file)
+                else:
+                    self._log_info(
+                        "Unable to execute pg_dump. Error(%s)" %
+                        (result['output'])
+                    )
             else:  # no password in env or options
                 self.soslog.warning(
                     "password must be supplied to dump a database."

Comment 16 Pavel Moravec 2017-12-19 16:44:48 UTC
*** Bug 1513071 has been marked as a duplicate of this bug. ***

Comment 23 Lukas Svaty 2018-01-08 17:54:15 UTC
Thanks for the ping,



Tested on sos-3.4-13.el7_4.noarch, within
ovirt-log-collector-4.2.0-1.el7ev.noarch


Tree structure of archive:
.
|-- sos_commands
|   `-- postgresql
|       `-- pgdump-scl-rh-postgresql95.tar
|-- sos_logs
|   |-- sos.log
|   `-- ui.log
|-- sos_reports
|   |-- sos.html
|   `-- sos.txt
|-- var
|   |-- lib
|   |   `-- pgsql
|   |       `-- initdb_rh-postgresql95-postgresql.log
|   `-- opt
|       `-- rh
|           `-- rh-postgresql95
|               `-- lib
|                   `-- pgsql
|                       `-- data
|                           |-- pg_hba.conf
|                           |-- pg_ident.conf
|                           |-- pg_log
|                           |   |-- postgresql-01.log
|                           |   `-- postgresql-Wed.log
|                           |-- PG_VERSION
|                           |-- postgresql.auto.conf
|                           |-- postgresql.conf
|                           `-- postmaster.opts
`-- version.txt


Config files collected ok

sos log seems ok


pg_dump in different path 
`── sos_commands
    └── postgresql
        └── pgdump-scl-rh-postgresql95.tar
 
 
Lets see what's inside:
# tar xvf sos_commands/postgresql/pgdump-scl-rh-postgresql95.tar.gz 
success

directory contains all the sql data

lets restore
# pg_restore sos_commands/postgresql/pgdump-scl-rh-postgresql95.tar
...output omitted...
--
-- PostgreSQL database dump complete
--

restart engine using this db

# service ovirt-engine restart
success!

Moving to verified.

Comment 25 Derrick Ornelas 2018-01-24 23:19:43 UTC
*** Bug 1538365 has been marked as a duplicate of this bug. ***

Comment 27 errata-xmlrpc 2018-01-25 11:34:34 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:0138


Note You need to log in before you can comment on or make changes to this bug.