Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.

Bug 1511732

Summary: [3.5] extended route validation does not catch newlines in certificate
Product: OpenShift Container Platform Reporter: Steven Walter <stwalter>
Component: RoutingAssignee: jtanenba
Status: CLOSED ERRATA QA Contact: zhaozhanqi <zzhao>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 3.5.1CC: aos-bugs, bbennett, bmeng, dmoessne, dsafford, fweimer, haowang, jtanenba, nbhatt, rhowe, sgaikwad, weliang
Target Milestone: ---   
Target Release: 3.5.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Cause: Invalid PEM data is left in the route config file during extended validation. Consequence: The router crashes Fix: Sanitize PEM data from route configs Result: Properly catch malformed certificates in extended validation.
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-12-14 21:02:32 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Description Steven Walter 2017-11-10 00:30:51 UTC
Description of problem:

In the route definition we saw empty lines:
...
    certificate: |
      -----BEGIN CERTIFICATE-----
...
    key: |
      -----BEGIN PRIVATE KEY-----
...

It is hardly noticeable in YAML format. On the OpenShift master console GUI it can be seen better.
Normally it should be |+ instead of just a | character.
Then the haproxy concatenates the certificates into a PEM file where the empty lines appear, and we think it causes those error messages.

Version-Release number of selected component (if applicable):
image: ose-haproxy-router:v3.5.5.31
OpenShift version: v3.5.5.31.24

How reproducible:
Unconfirmed

Steps to Reproduce:
1. Created route in web console

Actual results:
2017-11-09T09:20:32.957490000Z [ALERT] 312/092032 (127789) : parsing [/var/lib/haproxy/conf/haproxy.config:142] : 'bind 127.0.0.1:10444' : 'crt-list' : error processing line 150 in file '/var/lib/haproxy/conf/cert_config.map' : unable to load SSL certificate from PEM file '/var/lib/haproxy/router/certs/example.pem'.
2017-11-09T09:20:32.957682000Z [ALERT] 312/092032 (127789) : Error(s) found in configuration file : /var/lib/haproxy/conf/haproxy.config
2017-11-09T09:20:32.957873000Z [ALERT] 312/092032 (127789) : Fatal errors found in configuration.

Expected results:
Non-fatal warning, not accepting the route

Additional info:
Verified oc set env dc/router EXTENDED_VALIDATION=true
Deleting route causes error to go away

Comment 3 Steven Walter 2017-11-10 00:43:09 UTC
From Clayton:

Extended validation in 3.5 had gaps.  Most were fixed in 3.6.

You can run the 3.6 diagnostic command against a 3.5 cluster to check whether any routes would fail EV - if they can recreate in a test cluster please have them rerun.

Most of the things fixed were invalid PEM blocks and incomplete content.

Comment 7 Ryan Howe 2017-11-10 18:43:06 UTC
To reproduce this I removed one "-" from the -----END CERTIFICATE----

Was only able to reproduce in 3.5 removing a dash from -----END CERTIFICATE----  and only END CERTIFICATE section hit this issue. 


|+  |- and |  all work fine with 3.5 routes. Fatal errors only happened after editing -----END CERTIFICATE----


In 3.6 the router corrected the pem file adding the - back to the route when the pem was added to /var/lib/haproxy/router/certs/, the route object itself still showed the missing -. This did not cause 3.6 router to fail.

Comment 13 Weibin Liang 2017-11-16 20:44:16 UTC
PR/926 fixed the problem. Testing passed

Comment 15 zhaozhanqi 2017-11-29 07:54:31 UTC
QE did the testing on 3.5.5.31.47

1.  As comment 7 said: |+  |- and |  all work fine
2. if I removed one "-" from the -----END CERTIFICATE----, the route will be marked as 'ExtendedValidationFailed'

Comment 16 Dana Safford 2017-11-30 16:24:41 UTC
This situation is now very critical.
I just set the Customer Escalation flag and the am requesting a hotfix (so I flipped that flag too).

Hello Jacob/Rashid

I wrote yesterday in ref Bug 1511732  which is still in Status (Modified) and the following question below from the Action Plan:

1. Are we able to meet customers expectations on being able to deliver an errata fixing this by the 15th of December? If not
2. When is the cut off date on knowing if this expectation cant be achieved?
3. How long would the hotifix take? and when or who can deliver this.

For more info, contact Aaron Ship (Critsit Manager)  aship in irc

Comment 18 zhaozhanqi 2017-12-01 02:20:58 UTC
verified this bug according to comment 15

Comment 21 errata-xmlrpc 2017-12-14 21:02:32 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:3438