Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1514266 - Router SSL handshake failures when testing edge termination route.
Summary: Router SSL handshake failures when testing edge termination route.
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Routing
Version: 3.5.0
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
: 3.9.0
Assignee: Weibin Liang
QA Contact: zhaozhanqi
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-11-16 23:08 UTC by Ryan Howe
Modified: 2018-05-30 09:52 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-02-06 14:36:52 UTC


Attachments (Terms of Use)

Description Ryan Howe 2017-11-16 23:08:43 UTC
Description of problem:
Intermittently route fails on SSL handshake when connecting to route.  

Version-Release number of selected component (if applicable):
openshift3/ose-haproxy-router:v3.5.5.31

How reproducible:
100% with Apache bench mark. 

Actual results:
SSL connection errors 

Expected results:
No errors 

Additional info:

Haproxy logs: 

2017-11-16T21:29:20+00:00 localhost haproxy[3630]: 72.163.48.19:34150 [16/Nov/2017:21:29:19.762] fe_no_sni~ be_edge_http_appmgm-dev-int-rtp_kk-dev-int-rtp/81e86b702d9ae38791d78a2c4b1602ea 612/0/1/1/614 200 157 - - --NI 76/11/0/0/0 0/0 "GET /probe.html HTTP/1.0"

2017-11-16T21:29:20+00:00 localhost haproxy[3630]: 72.163.48.19:34346 [16/Nov/2017:21:29:20.444] public_ssl be_no_sni/<NOSRV> -1/-1/95 0 -- 88/29/27/0/3 0/0
2017-11-16T21:29:20+00:00 localhost haproxy[3630]: 72.163.48.19:34308 [16/Nov/2017:21:29:20.379] public_ssl be_no_sni/<NOSRV> -1/-1/160 0 -- 87/28/27/0/3 0/0
2017-11-16T21:29:20+00:00 localhost haproxy[3630]: 72.163.48.19:34222 [16/Nov/2017:21:29:20.355] public_ssl be_no_sni/fe_no_sni 45/1/184 2155 CD 86/27/26/26/0 0/0

2017-11-16T21:29:20+00:00 localhost haproxy[3630]: 72.163.48.19:34236 [16/Nov/2017:21:29:20.402] fe_no_sni/1: Connection error during SSL handshake
2017-11-16T21:29:20+00:00 localhost haproxy[3630]: 72.163.48.19:34206 [16/Nov/2017:21:29:20.401] fe_no_sni/1: Connection error during SSL handshake
2017-11-16T21:29:20+00:00 localhost haproxy[3630]: 72.163.48.19:34280 [16/Nov/2017:21:29:20.453] fe_no_sni/1: Connection closed during SSL handshake
2017-11-16T21:29:20+00:00 localhost haproxy[3630]: 72.163.48.19:34208 [16/Nov/2017:21:29:20.401] fe_no_sni/1: Connection error during SSL handshake

AB results: 

100118 SSL/TLS Handshake [Start] before/connect initialization
99488 SSL/TLS Handshake [Done] SSL negotiation finished successfully

# grep 'fail' data.  | sort -nr | uniq -c
      3 SSL read failed (5) - closing connection
      1 SSL read failed (1) - closing connection
    516 SSL handshake failed (5).

# grep 'SSL/TLS' data  -i | sort -nr | uniq -c | sort -nr
 100118 SSL/TLS State [connect] before/connect initialization
 100118 SSL/TLS Handshake [Start] before/connect initialization
 100112 SSL/TLS State [connect] SSLv2/v3 write client hello A
  99515 SSL/TLS State [connect] SSLv3 write finished A
  99515 SSL/TLS State [connect] SSLv3 write client key exchange A
  99515 SSL/TLS State [connect] SSLv3 write change cipher spec A
  99515 SSL/TLS State [connect] SSLv3 read server key exchange A
  99515 SSL/TLS State [connect] SSLv3 read server hello A
  99515 SSL/TLS State [connect] SSLv3 read server done A
  99515 SSL/TLS State [connect] SSLv3 read server certificate A
  99515 SSL/TLS State [connect] SSLv3 flush data
  99488 SSL/TLS State [connect] SSLv3 read server session ticket A
  99488 SSL/TLS State [connect] SSLv3 read finished A
  99488 SSL/TLS Handshake [Done] SSL negotiation finished successfully
  99483 SSL/TLS Alert [write] warning:close notify
  99483 SSL/TLS Alert [read] warning:close notify
      3 SSL/TLS State [connect] SSLv2/v3 write client hello B

Comment 3 Ben Bennett 2018-01-17 21:31:26 UTC
Weibin, can you attempt to reproduce please?  I have a hunch that ab is hitting haproxy too fast, but we'll need to see it to confirm and then attempt to mitigate.

Comment 4 Weibin Liang 2018-01-18 15:29:29 UTC
We can duplicate the problem when using the high number of multiple requests in ab command such as -c 1000.

No SSL failure when using -c 100.

What is number of multiple requests you used in ab testing?


[root@dhcp-41-193 ~]# ab -n 50000 -c 1000 https://hello-openshift-default.apps.0117-y-r.qe.rhcloud.com/
This is ApacheBench, Version 2.3 <$Revision: 1430300 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking hello-openshift-default.apps.0117-y-r.qe.rhcloud.com (be patient)
SSL handshake failed (5).
SSL handshake failed (5).
SSL handshake failed (5).
SSL handshake failed (5).


[root@host-172-16-120-4 ~]# oc version
oc v3.9.0-0.20.0

Comment 5 Ben Bennett 2018-02-06 14:36:52 UTC
It looks to me like we are just overloading haproxy.  If you can provide the number of connections you are testing with, and it seems low for the hardware, please feel free to re-open the bug.


Note You need to log in before you can comment on or make changes to this bug.