Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1691841 - Volume metrics should create volume metrics with the correct PVC ref test failure
Summary: Volume metrics should create volume metrics with the correct PVC ref test fai...
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Storage
Version: 4.1
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
: ---
Assignee: Fabio Bertinatto
QA Contact: Liang Xia
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-03-22 16:11 UTC by Frederic Branczyk
Modified: 2019-03-27 16:21 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-03-27 16:21:48 UTC
Target Upstream Version:


Attachments (Terms of Use)

Description Frederic Branczyk 2019-03-22 16:11:42 UTC
Description of problem:

The following test is failed in installer aws-e2e-serial:

openshift-tests [sig-storage] [Serial] Volume metrics should create volume metrics with the correct PVC ref [Suite:openshift/conformance/serial] [Suite:k8s] 10m29s

Mar 22 15:08:18.402: INFO: >>> kubeConfig: /tmp/admin.kubeconfig
Mar 22 15:08:18.406: INFO: Waiting up to 30m0s for all (but 100) nodes to be schedulable
Mar 22 15:08:18.535: INFO: Waiting up to 10m0s for all pods (need at least 0) in namespace 'kube-system' to be running and ready
Mar 22 15:08:18.608: INFO: 3 / 3 pods in namespace 'kube-system' are running and ready (0 seconds elapsed)
Mar 22 15:08:18.608: INFO: expected 0 pod replicas in namespace 'kube-system', 0 are Running and Ready.
Mar 22 15:08:18.608: INFO: Waiting up to 5m0s for all daemonsets in namespace 'kube-system' to start
Mar 22 15:08:18.632: INFO: e2e test version: v1.12.4+0c8ecae
Mar 22 15:08:18.647: INFO: kube-apiserver version: v1.12.4+0c8ecae
[BeforeEach] [Top Level]
  /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/test/extended/util/test.go:67
[BeforeEach] [sig-storage] [Serial] Volume metrics
  /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/test/e2e/framework/framework.go:147
STEP: Creating a kubernetes client
Mar 22 15:08:18.648: INFO: >>> kubeConfig: /tmp/admin.kubeconfig
STEP: Building a namespace api object, basename pv
Mar 22 15:08:19.712: INFO: About to run a Kube e2e test, ensuring namespace is privileged
Mar 22 15:08:19.991: INFO: No PodSecurityPolicies found; assuming PodSecurityPolicy is disabled.
STEP: Waiting for a default service account to be provisioned in namespace
[BeforeEach] [sig-storage] [Serial] Volume metrics
  /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/test/e2e/storage/volume_metrics.go:48
Mar 22 15:08:20.024: INFO: Default storage class: "gp2"
W0322 15:08:20.060383     644 metrics_grabber.go:81] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled.
[It] should create volume metrics with the correct PVC ref [Suite:openshift/conformance/serial] [Suite:k8s]
  /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/test/e2e/storage/volume_metrics.go:113
Mar 22 15:18:20.222: INFO: Unexpected error occurred: timed out waiting for the condition
[AfterEach] [sig-storage] [Serial] Volume metrics
  /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/test/e2e/framework/framework.go:148
STEP: Collecting events from namespace "e2e-tests-pv-vq7jd".
STEP: Found 6 events.
Mar 22 15:18:20.242: INFO: At 2019-03-22 15:08:20 +0000 UTC - event for pvc-d8fbm: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding
Mar 22 15:18:20.242: INFO: At 2019-03-22 15:09:44 +0000 UTC - event for pvc-d8fbm: {persistentvolume-controller } ProvisioningFailed: Failed to provision volume with StorageClass "gp2": RequestLimitExceeded: Request limit exceeded.
	status code: 503, request id: 81cd5908-9571-46d7-86de-69886a06bb53
Mar 22 15:18:20.242: INFO: At 2019-03-22 15:09:45 +0000 UTC - event for pvc-tester-l6x62: {default-scheduler } FailedScheduling: selectedNode annotation value "" not set to scheduled node "ip-10-0-131-104.ec2.internal"
Mar 22 15:18:20.242: INFO: At 2019-03-22 15:09:45 +0000 UTC - event for pvc-tester-l6x62: {default-scheduler } FailedScheduling: pod has unbound immediate PersistentVolumeClaims (repeated 3 times)
Mar 22 15:18:20.242: INFO: At 2019-03-22 15:12:26 +0000 UTC - event for pvc-d8fbm: {persistentvolume-controller } ProvisioningFailed: Failed to provision volume with StorageClass "gp2": RequestLimitExceeded: Request limit exceeded.
	status code: 503, request id: 6d0fd003-c41c-40bb-9144-b8ae5dc82b82
Mar 22 15:18:20.242: INFO: At 2019-03-22 15:12:26 +0000 UTC - event for pvc-tester-l6x62: {default-scheduler } FailedScheduling: AssumePod failed: pod 54238808-4cb4-11e9-80d5-0e3d9ddc55a2 is in the cache, so can't be assumed
Mar 22 15:18:20.275: INFO: skipping dumping cluster info - cluster too large
Mar 22 15:18:20.275: INFO: Waiting up to 3m0s for all (but 100) nodes to be ready
STEP: Destroying namespace "e2e-tests-pv-vq7jd" for this suite.
Mar 22 15:18:44.352: INFO: Waiting up to 30s for server preferred namespaced resources to be successfully discovered
Mar 22 15:18:46.268: INFO: namespace e2e-tests-pv-vq7jd deletion completed in 25.974058226s
[AfterEach] [sig-storage] [Serial] Volume metrics
  /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/test/e2e/storage/volume_metrics.go:69
Mar 22 15:18:46.268: INFO: Deleting PersistentVolumeClaim "pvc-d8fbm"
Mar 22 15:18:46.291: INFO: Running AfterSuite actions on all node
Mar 22 15:18:46.291: INFO: Running AfterSuite actions on node 1
fail [k8s.io/kubernetes/test/e2e/storage/volume_metrics.go:125]: Error starting pod %!(EXTRA string=pvc-tester-l6x62)
Expected error:
    <*errors.errorString | 0xc4206715a0>: {
        s: "timed out waiting for the condition",
    }
    timed out waiting for the condition
not to have occurred

Mar 22 15:17:48.182 I ns/openshift-machine-api machine/ci-op-0jt5c3k8-fd334-8wvk7-worker-us-east-1c-s7ml6 Updated machine ci-op-0jt5c3k8-fd334-8wvk7-worker-us-east-1c-s7ml6 (10 times)
Mar 22 15:17:50.577 I ns/openshift-machine-api machine/ci-op-0jt5c3k8-fd334-8wvk7-master-0 Updated machine ci-op-0jt5c3k8-fd334-8wvk7-master-0 (7 times)
Mar 22 15:17:53.042 I ns/openshift-machine-api machine/ci-op-0jt5c3k8-fd334-8wvk7-master-1 Updated machine ci-op-0jt5c3k8-fd334-8wvk7-master-1 (7 times)
Mar 22 15:17:55.199 I ns/openshift-machine-api machine/ci-op-0jt5c3k8-fd334-8wvk7-master-2 Updated machine ci-op-0jt5c3k8-fd334-8wvk7-master-2 (7 times)
Mar 22 15:17:55.417 I ns/openshift-machine-api machine/ci-op-0jt5c3k8-fd334-8wvk7-worker-us-east-1a-bx9g4 Updated machine ci-op-0jt5c3k8-fd334-8wvk7-worker-us-east-1a-bx9g4 (10 times)
Mar 22 15:17:55.619 I ns/openshift-machine-api machine/ci-op-0jt5c3k8-fd334-8wvk7-worker-us-east-1b-bjbsz Updated machine ci-op-0jt5c3k8-fd334-8wvk7-worker-us-east-1b-bjbsz (10 times)

---

Judging by this line:

```
Mar 22 15:18:20.242: INFO: At 2019-03-22 15:12:26 +0000 UTC - event for pvc-d8fbm: {persistentvolume-controller } ProvisioningFailed: Failed to provision volume with StorageClass "gp2": RequestLimitExceeded: Request limit exceeded.
```

It seems this was "just" a matter of quota.

Comment 1 Frederic Branczyk 2019-03-22 16:12:26 UTC
Sorry missed to set version correctly.

Comment 2 Fabio Bertinatto 2019-03-27 16:21:48 UTC
This seems to be a quota issue.

In any case, I left this test running in a loop and it didn't fail.


Note You need to log in before you can comment on or make changes to this bug.