Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1686476 - Login flows directing user to api discover page instead of console
Summary: Login flows directing user to api discover page instead of console
Keywords:
Status: ON_QA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Auth
Version: 4.1
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
: 4.1.0
Assignee: Mo
QA Contact: Chuan Yu
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-03-07 14:23 UTC by Justin Pierce
Modified: 2019-04-15 14:38 UTC (History)
12 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:
Target Upstream Version:


Attachments (Terms of Use)

Description Justin Pierce 2019-03-07 14:23:43 UTC
Description of problem:
Logging into the console results in constant redirection to page showing available apis.

{
  "paths": [
    "/apis",
    "/healthz",
    "/healthz/log",
    "/healthz/ping",
    "/healthz/poststarthook/oauth.openshift.io-startoauthclientsbootstrapping",
    "/metrics"
  ]
}

The preceding page content results from a 404 during the auth flow:
Request URL: https://console-openshift-console.apps.rhcos-cr.rhcos.sandbox.openshift.com/auth/callback?code=jS2....p4qc&state=90db818c
Request Method: GET
Status Code: 404 
Remote Address: 18.205.100.105:443



Version-Release number of selected component (if applicable):
version   4.0.0-0.alpha-2019-03-05-054505

How reproducible:
100% on two recent clusters

Comment 3 Dan Mace 2019-03-18 21:04:06 UTC
I now have reason to believe the underlying problem is https://bugzilla.redhat.com/show_bug.cgi?id=1688390 — need to do a bit more diagnosis to confirm.

Comment 4 Dan Mace 2019-03-18 21:17:56 UTC
Created https://bugzilla.redhat.com/show_bug.cgi?id=1690146 to track it in 4.0

Comment 5 Dan Mace 2019-03-20 14:26:05 UTC
Regarding https://bugzilla.redhat.com/show_bug.cgi?id=1688390 and https://bugzilla.redhat.com/show_bug.cgi?id=1690146 — yesterday I was able to reproduce the console problem in a cluster which was NOT exhibiting the suspicious haproxy process issue, so the problems may not be related after all.

Comment 7 Hongan Li 2019-03-26 06:38:27 UTC
(In reply to Dan Mace from comment #5)
> Regarding https://bugzilla.redhat.com/show_bug.cgi?id=1688390 and
> https://bugzilla.redhat.com/show_bug.cgi?id=1690146 — yesterday I was able
> to reproduce the console problem in a cluster which was NOT exhibiting the
> suspicious haproxy process issue, so the problems may not be related after
> all.

Hi Dan, I tried but didn't reproduce this issue with several nightly builds recently.
Could you please provide the detail steps if you can reproduce. 
Thanks.

Comment 8 Samuel Padgett 2019-03-26 12:07:45 UTC
We're only seeing this on certain clusters. We don't have a specific steps to reproduce unfortunately.

Comment 16 Dan Mace 2019-03-27 18:02:13 UTC
We've identified the root issue as edge case in our passthrough route handling of HTTP2 endpoints. Given the following conditions:

* A wildcard ingress certificate (eg. *.apps.openshift.example.com)
* A DNS wildcard (eg. *.apps.openshift.example.com) with A records resolving to a static set of ingress load balancer IPs
* A passthrough route to an HTTP2 server (eg. auth.apps.openshift.example.com)
* A an edge or reencrypted route to a server in the same subdomain (eg. console.apps.openshift.example.com)
* An HTTP2 client which coalesces connections (eg. Chrome/Firefox)

It's possible for packets destined for a proxy-terminated route (eg. console) to be misdirected to the passthrough/HTTP2 route (eg. auth).

In brief, a connection to the passthrough/HTTP2 server may be reused by the client for packets destined for other servers for which the wildcard certificate is valid. Because both route host names are valid for the certificate and resolve to the same IPs through DNS, an existing HTTP2 server connection is considered reusable for the other servers' packets. However, because the HTTP2 connection at the proxy is coupled to the HTTP2 server through the initial SNI header from a TLS handshake, and the packets coming through are opaque and cannot be disambiguated by the proxy, the packets cannot be identified by the proxy as misdirected and are all forwarded to the HTTP2 server.

Solutions could be:

1. Discontinue use of HTTP2 for these services
2. Implement mutual TLS in the ingress controller to enable terminating TLS at the proxy for the auth server
3. Implement HTTP 421 misdirected request support at the auth server to hint clients to stop reusing the connection for the request authority 

Our current recommendation in the short term is (3), and longer term we would like to implement mTLS (2) for ease of use.

References:

* https://httpwg.org/specs/rfc7540.html#reuse
* https://httpwg.org/specs/rfc7540.html#MisdirectedRequest
* https://daniel.haxx.se/blog/2016/08/18/http2-connection-coalescing

Comment 17 Dan Mace 2019-03-28 15:09:37 UTC
Another viable solution Paul Weil presented is to use a separate ingresscontroller, domain, and cert for auth.

In the meantime, users can probably work around the issue by waiting for connections to expire (eg. ~30s) and reloading the console page (or using a new browser session).

Comment 20 Mo 2019-04-14 03:52:00 UTC
https://github.com/openshift/origin/pull/22529 is merged


Note You need to log in before you can comment on or make changes to this bug.