Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1598669 - cannot backup/restore cloudforms db if it is installed using template which comes with ocp 3.9
Summary: cannot backup/restore cloudforms db if it is installed using template which c...
Keywords:
Status: CLOSED DEFERRED
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Installer
Version: 3.9.0
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: ---
: 3.9.z
Assignee: Scott Dodson
QA Contact: Johnny Liu
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-07-06 07:18 UTC by Ievgen Zapolskyi
Modified: 2018-11-19 20:23 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-11-19 20:23:19 UTC
Target Upstream Version:


Attachments (Terms of Use)

Description Ievgen Zapolskyi 2018-07-06 07:18:35 UTC
Description of problem:
postgre pod is deployed with wrong settings and doesn't pick up some config overrides. As a result default settings don't allow to do cloudforms backup/restore.
Error "pg_basebackup: could not connect to server: FATAL:  number of requested standby connections exceeds max_wal_senders (currently 0)"

It seems that patch has to resolve this issue - https://github.com/openshift/openshift-ansible/pull/8297/files

Template from build http://file.cloudforms.lab.eng.rdu2.redhat.com/builds/cfme/5.9/5.9.3.4/openshift-pods/templates/
doesn't have this issue.

Version-Release number of the following components:

rpm -q openshift-ansible
openshift-ansible-3.9.31-1.git.34.154617d.el7.noarc

ansible --version
ansible 2.4.4.0
  config file = /etc/ansible/ansible.cfg
  configured module search path = [u'/root/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python2.7/site-packages/ansible
  executable location = /usr/bin/ansible
  python version = 2.7.5 (default, Feb 20 2018, 09:19:12) [GCC 4.8.5 20150623 (Red Hat 4.8.5-28)]

rpm -q ansible
ansible-2.4.4.0-1.el7ae.noarch

How reproducible:
100%

Steps to Reproduce:
1. deploy cloudforms 5.9.3.4 by ansible-playbook -v -i ./hosts /usr/share/ansible/openshift-ansible/playbooks/openshift-management/config.yml
(hosts file is attached)

2. copy templates from http://file.cloudforms.lab.eng.rdu2.redhat.com/builds/cfme/5.9/5.9.3.4/openshift-pods/templates/ to openshift server (openshift's templates can also be used)

3. create pv + pvc for backup job and kick off backup job (full list of steps can be found here - https://github.com/ManageIQ/manageiq-pods/blob/gaprindashvili/README.md)
oc project ansible-cfme
oc create -f cfme-pv-backup-example.yaml
oc create -f cfme-backup-pvc.yaml
oc create -f cfme-backup-job.yaml

3. 
oc get pods -w 
oc logs cloudforms-backup-*

Actual results:
backup job finished with error

Expected results:
correct backup/restore

Additional info:

cloudforms deployed by ansible
--------------
[root@cmqe-smicro-628-b14 cfme-5934-0702229]# oc create -f cfme-backup-job.yaml
job "cloudforms-backup" created
[root@cmqe-smicro-628-b14 cfme-5934-0702229]# oc get pods
NAME                      READY     STATUS    RESTARTS   AGE
cloudforms-0              1/1       Running   0          9m
cloudforms-backup-mwp9v   0/1       Error     0          7s
httpd-1-7jxsc             1/1       Running   0          9m
memcached-1-7znk7         1/1       Running   0          9m
postgresql-1-d94p4        1/1       Running   0          9m
[root@cmqe-smicro-628-b14 cfme-5934-0702229]# oc logs cloudforms-backup-mwp9v
== Starting MIQ DB backup ==
Current time is : Thu Jul  5 19:48:15 UTC 2018
pg_basebackup: could not connect to server: FATAL:  number of requested standby connections exceeds max_wal_senders (currently 0)
ERROR:  exited with abnormal status, please check backup status
[root@cmqe-smicro-628-b14 cfme-5934-0702229]# oc rsh postgresql-1-d94p4
sh-4.2$ bash
bash-4.2$ psql
psql (9.5.9)
Type "help" for help.

postgres=# show max_wal_senders;
 max_wal_senders
-----------------
 0
(1 row)

[root@cmqe-smicro-628-b14 cfme-5934-0702229]# oc describe pod postgresql-1-d94p4
Name:           postgresql-1-d94p4
Namespace:      ansible-cfme
Node:           cmqe-smicro-628-b14.cmqe.lab.eng.rdu2.redhat.com/10.8.218.14
Start Time:     Thu, 05 Jul 2018 15:39:14 -0400
Labels:         deployment=postgresql-1
                deploymentconfig=postgresql
                name=postgresql
Annotations:    openshift.io/deployment-config.latest-version=1
                openshift.io/deployment-config.name=postgresql
                openshift.io/deployment.name=postgresql-1
                openshift.io/scc=restricted
Status:         Running
IP:             10.128.1.186
Controlled By:  ReplicationController/postgresql-1
Containers:
  postgresql:
    Container ID:   docker://7d34d87ce00006312e41f6d5e841c23ca66ca7997001c008559200e62be95742
    Image:          brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/cloudforms46/cfme-openshift-postgresql:9.5-100
    Image ID:       docker-pullable://brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/cloudforms46/cfme-openshift-postgresql@sha256:b2d762ab3268f64633393eb4ebc6915cddcecd352d42697e5ee8e1c0a4f892d0
------------------------------------------------------------------------------
cloudforms deployed from template
-----------------
[root@cmqe-smicro-628-b14 cfme-5934-0702229]# oc project izapolsk-s-appl-downstream-59z-180702-1awcoren
Now using project "izapolsk-s-appl-downstream-59z-180702-1awcoren" on server "https://cmqe-smicro-628-b14.cmqe.lab.eng.rdu2.redhat.com:8443".
[root@cmqe-smicro-628-b14 cfme-5934-0702229]# oc get pods
NAME                      READY     STATUS      RESTARTS   AGE
cloudforms-0              1/1       Running     0          4h
cloudforms-backend-0      1/1       Running     0          4h
cloudforms-backup-h9xl7   0/1       Completed   0          54m
httpd-1-ff8xw             1/1       Running     0          4h
memcached-1-b5c6l         1/1       Running     0          4h
postgresql-1-4kkz4        1/1       Running     0          4h
[root@cmqe-smicro-628-b14 cfme-5934-0702229]# oc rsh postgresql-1-4kkz4
sh-4.2$ bash
bash-4.2$ psql
psql (9.5.9)
Type "help" for help.

postgres=# show max_wal_senders;
 max_wal_senders
-----------------
 10
(1 row)

[root@cmqe-smicro-628-b14 cfme-5934-0702229]# oc describe pod postgresql-1-4kkz4
Name:           postgresql-1-4kkz4
Namespace:      izapolsk-s-appl-downstream-59z-180702-1awcoren
Node:           cmqe-smicro-628-b14.cmqe.lab.eng.rdu2.redhat.com/10.8.218.14
Start Time:     Thu, 05 Jul 2018 11:03:58 -0400
Labels:         deployment=postgresql-1
                deploymentconfig=postgresql
                name=postgresql
Annotations:    openshift.io/deployment-config.latest-version=1
                openshift.io/deployment-config.name=postgresql
                openshift.io/deployment.name=postgresql-1
                openshift.io/scc=restricted
Status:         Running
IP:             10.128.1.130
Controlled By:  ReplicationController/postgresql-1
Containers:
  postgresql:
    Container ID:   docker://b85ccce4ff8d5c4a034e8004bfeaf06c32d536ccbc0dd03cf7d8da8d72f96748
    Image:          brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/cloudforms46/cfme-openshift-postgresql:9.5-100
    Image ID:       docker-pullable://brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/cloudforms46/cfme-openshift-postgresql@sha256:b2d762ab3268f64633393eb4ebc6915cddcecd352d42697e5ee8e1c0a4f892d0
    Port:           5432/TCP
    State:          Running
      Started:      Thu, 05 Jul 2018 11:04:05 -0400
    Ready:          True

Comment 2 Russell Teague 2018-11-19 20:23:19 UTC
There appear to be no active cases related to this bug. As such we're closing this bug in order to focus on bugs that are still tied to active customer cases. Please re-open this bug if you feel it was closed in error or a new active case is attached.


Note You need to log in before you can comment on or make changes to this bug.