Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1690660 - Volume metrics should create metrics for total number of volumes in A/D Controller conformance test failure
Summary: Volume metrics should create metrics for total number of volumes in A/D Contr...
Keywords:
Status: ASSIGNED
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Storage
Version: 4.1
Hardware: Unspecified
OS: Unspecified
medium
low
Target Milestone: ---
: 4.2.0
Assignee: Hemant Kumar
QA Contact: Liang Xia
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-03-20 00:10 UTC by Chance Zibolski
Modified: 2019-04-10 07:02 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:
Target Upstream Version:


Attachments (Terms of Use)
Occurrences of this error in CI from 2019-03-19T12:28 to 2019-03-21T20:06 UTC (deleted)
2019-03-22 06:02 UTC, W. Trevor King
no flags Details

Description Chance Zibolski 2019-03-20 00:10:13 UTC
Description of problem:

The following test is failed in installer aws-e2e-serial:

[sig-storage] [Serial] Volume metrics should create metrics for total number of volumes in A/D Controller [Suite:openshift/conformance/serial] [Suite:k8s]

The build log indicates it timed out waiting:

clusterversion.config.openshift.io/version condition met
started: (0/1/40) "[sig-storage] [Serial] Volume metrics should create metrics for total number of volumes in A/D Controller [Suite:openshift/conformance/serial] [Suite:k8s]"

Mar 19 22:58:47.338: INFO: >>> kubeConfig: /tmp/admin.kubeconfig
Mar 19 22:58:47.341: INFO: Waiting up to 30m0s for all (but 100) nodes to be schedulable
Mar 19 22:58:47.439: INFO: Waiting up to 10m0s for all pods (need at least 0) in namespace 'kube-system' to be running and ready
Mar 19 22:58:47.511: INFO: 3 / 3 pods in namespace 'kube-system' are running and ready (0 seconds elapsed)
Mar 19 22:58:47.511: INFO: expected 0 pod replicas in namespace 'kube-system', 0 are Running and Ready.
Mar 19 22:58:47.511: INFO: Waiting up to 5m0s for all daemonsets in namespace 'kube-system' to start
Mar 19 22:58:47.531: INFO: e2e test version: v1.12.4+41bf871
Mar 19 22:58:47.546: INFO: kube-apiserver version: v1.12.4+41bf871
[BeforeEach] [Top Level]
  /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/test/extended/util/test.go:67
[BeforeEach] [sig-storage] [Serial] Volume metrics
  /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/test/e2e/framework/framework.go:147
STEP: Creating a kubernetes client
Mar 19 22:58:47.546: INFO: >>> kubeConfig: /tmp/admin.kubeconfig
STEP: Building a namespace api object, basename pv
[AfterEach] [sig-storage] [Serial] Volume metrics
  /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/test/e2e/framework/framework.go:148
STEP: Destroying namespace "e2e-tests-pv-bdvsl" for this suite.
Mar 19 22:59:56.511: INFO: Waiting up to 30s for server preferred namespaced resources to be successfully discovered
Mar 19 22:59:58.409: INFO: namespace e2e-tests-pv-bdvsl deletion completed in 42.196949318s
[AfterEach] [sig-storage] [Serial] Volume metrics
  /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/test/e2e/storage/volume_metrics.go:69
Mar 19 22:59:58.413: INFO: Running AfterSuite actions on all node
Mar 19 22:59:58.414: INFO: Running AfterSuite actions on node 1
fail [k8s.io/kubernetes/test/e2e/framework/framework.go:239]: Expected error:
    <*errors.errorString | 0xc420691b00>: {
        s: "watch closed before UntilWithoutRetry timeout",
    }
    watch closed before UntilWithoutRetry timeout
not to have occurred


Release stream: https://openshift-release.svc.ci.openshift.org/releasestream/4.0.0-0.ci/release/4.0.0-0.ci-2019-03-19-214323

Build: https://openshift-gce-devel.appspot.com/build/origin-ci-test/logs/release-openshift-origin-installer-e2e-aws-serial-4.0/3080?log#log

Comment 1 W. Trevor King 2019-03-22 06:02:44 UTC
Created attachment 1546780 [details]
Occurrences of this error in CI from 2019-03-19T12:28 to 2019-03-21T20:06 UTC

This occurred in 3 of our 861 failures in *-e2e-aws* jobs across the whole CI system over the past 55 hours.  Generated with [1]:

  $ deck-build-log-plot 'failed: .*Volume metrics should create metrics for total number of volumes in A/D Controller'

[1]: https://github.com/wking/openshift-release/tree/debug-scripts/deck-build-log

Comment 2 Tomas Smetana 2019-04-05 13:10:31 UTC
It's another instance of AWS API throttling:
Mar 19 23:19:18.233 W persistentvolume/pvc-18088252-4a9d-11e9-aba2-0af2371767c2 error deleting EBS volume "vol-09d39299a68605958": "RequestLimitExceeded: Request limit exceeded.\n\tstatus code: 503, request id: ef78d140-bca3-4517-af16-974fc5416286"


Note You need to log in before you can comment on or make changes to this bug.