Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1510172 - master controller panic during reliability long run - TypeAssertionError during project creation/deletion
Summary: master controller panic during reliability long run - TypeAssertionError duri...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Master
Version: 3.7.0
Hardware: x86_64
OS: Linux
unspecified
medium
Target Milestone: ---
: 3.7.0
Assignee: Dan Mace
QA Contact: Mike Fiedler
URL:
Whiteboard: aos-scalability-37
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-11-06 20:43 UTC by Mike Fiedler
Modified: 2017-11-28 22:21 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
undefined
Clone Of:
Environment:
Last Closed: 2017-11-28 22:21:49 UTC


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2017:3188 normal SHIPPED_LIVE Moderate: Red Hat OpenShift Container Platform 3.7 security, bug, and enhancement update 2017-11-29 02:34:54 UTC

Description Mike Fiedler 2017-11-06 20:43:05 UTC
Description of problem:

During a OCP 3.7 reliability test long run, 8 days into the test, the panic below was seen.   


Nov 06 09:04:09 ip-172-31-29-26.us-west-2.compute.internal atomic-openshift-master-controllers[36345]: E1106 09:04:09.512095   36356 runtime.go:66] Observed a panic: &runtime.TypeAssertionError{interfaceString:"interface {}", concreteString:"cache.DeletedFinalStateUnknown", assertedString:"*apps.DeploymentConfig", missingMethod:""} (interface conversion: interface {} is cache.DeletedFinalStateUnknown, not *apps.DeploymentConfig)
Nov 06 09:04:09 ip-172-31-29-26.us-west-2.compute.internal atomic-openshift-master-controllers[36345]: /builddir/build/BUILD/atomic-openshift-git-0.d787960/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:72
Nov 06 09:04:09 ip-172-31-29-26.us-west-2.compute.internal atomic-openshift-master-controllers[36345]: /builddir/build/BUILD/atomic-openshift-git-0.d787960/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:65
Nov 06 09:04:09 ip-172-31-29-26.us-west-2.compute.internal atomic-openshift-master-controllers[36345]: /builddir/build/BUILD/atomic-openshift-git-0.d787960/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51
Nov 06 09:04:09 ip-172-31-29-26.us-west-2.compute.internal atomic-openshift-master-controllers[36345]: /usr/lib/golang/src/runtime/asm_amd64.s:514
Nov 06 09:04:09 ip-172-31-29-26.us-west-2.compute.internal atomic-openshift-master-controllers[36345]: /usr/lib/golang/src/runtime/panic.go:489
Nov 06 09:04:09 ip-172-31-29-26.us-west-2.compute.internal atomic-openshift-master-controllers[36345]: /usr/lib/golang/src/runtime/iface.go:172
Nov 06 09:04:09 ip-172-31-29-26.us-west-2.compute.internal atomic-openshift-master-controllers[36345]: /builddir/build/BUILD/atomic-openshift-git-0.d787960/_output/local/go/src/github.com/openshift/origin/pkg/image/trigger/deploymentconfigs/deploymentconfigs.go:118
Nov 06 09:04:09 ip-172-31-29-26.us-west-2.compute.internal atomic-openshift-master-controllers[36345]: <autogenerated>:1
Nov 06 09:04:09 ip-172-31-29-26.us-west-2.compute.internal atomic-openshift-master-controllers[36345]: /builddir/build/BUILD/atomic-openshift-git-0.d787960/_output/local/go/src/github.com/openshift/origin/pkg/image/controller/trigger/cache.go:81
Nov 06 09:04:09 ip-172-31-29-26.us-west-2.compute.internal atomic-openshift-master-controllers[36345]: /builddir/build/BUILD/atomic-openshift-git-0.d787960/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/client-go/tools/cache/controller.go:206
Nov 06 09:04:09 ip-172-31-29-26.us-west-2.compute.internal atomic-openshift-master-controllers[36345]: <autogenerated>:59
Nov 06 09:04:09 ip-172-31-29-26.us-west-2.compute.internal atomic-openshift-master-controllers[36345]: /builddir/build/BUILD/atomic-openshift-git-0.d787960/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/client-go/tools/cache/shared_informer.go:549
Nov 06 09:04:09 ip-172-31-29-26.us-west-2.compute.internal atomic-openshift-master-controllers[36345]: /usr/lib/golang/src/runtime/asm_amd64.s:2197


The reliability test is a 4 node cluster (1 master, 1 infra, 2 compute) which performs the following actions on scheduled basis at "normal" intervals.  It is not a stress test.   The purpose of the test is to monitor CPU, memory, I/O over long periods of time.

- create/delete projects
- create/delete apps
- run builds
- scale apps up/down
- send traffic to apps
- etc.


From the log (linked in private followup comment), the panic occurred during a period of project deletion.

NOTE:  There are several issues in this log we are opening separate bz for.   This particular issue can be seen by searching for this string:

Observed a panic: &runtime.TypeAssertionError

Version-Release number of selected component (if applicable): 3.7.0-0.178.0

Comment 4 Mike Fiedler 2017-11-10 12:37:58 UTC
Verified on 3.7.5 - left a create projects -> create apps -> build apps -> delete projects hard loop running 24 hrs and no panics.

Comment 7 errata-xmlrpc 2017-11-28 22:21:49 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2017:3188


Note You need to log in before you can comment on or make changes to this bug.