Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1685807 - Sometimes meet "many-to-many matching not allowed: matching labels must be unique on one side" warn info in prometheus-k8s pod
Summary: Sometimes meet "many-to-many matching not allowed: matching labels must be un...
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Monitoring
Version: 3.11.0
Hardware: Unspecified
OS: Unspecified
low
low
Target Milestone: ---
: ---
Assignee: Frederic Branczyk
QA Contact: Junqi Zhao
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-03-06 07:18 UTC by Junqi Zhao
Modified: 2019-03-07 11:14 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-03-07 11:14:33 UTC
Target Upstream Version:


Attachments (Terms of Use)
"many-to-many matching not allowed: matching labels must be unique on one side" warn info in prometheus-k8s pod (deleted)
2019-03-06 07:18 UTC, Junqi Zhao
no flags Details

Description Junqi Zhao 2019-03-06 07:18:46 UTC
Created attachment 1541234 [details]
"many-to-many matching not allowed: matching labels must be unique on one side" warn info in prometheus-k8s pod

Description of problem:
Sometimes meet "many-to-many matching not allowed: matching labels must be unique on one side" warn info in prometheus-k8s pod, eg:
oc -n openshift-monitoring logs prometheus-k8s-1 -c prometheus | grep "Evaluating rule failed"
level=warn ts=2019-03-06T06:20:25.743554993Z caller=manager.go:343 component="rule manager" group=k8s.rules msg="Evaluating rule failed" rule="record: namespace_name:container_cpu_usage_seconds_total:sum_rate\nexpr: sum by(namespace, label_name) (sum by(namespace, pod_name) (rate(container_cpu_usage_seconds_total{container_name!=\"\",image!=\"\",job=\"kubelet\"}[5m]))\n  * on(namespace, pod_name) group_left(label_name) label_replace(kube_pod_labels{job=\"kube-state-metrics\"},\n  \"pod_name\", \"$1\", \"pod\", \"(.*)\"))\n" err="many-to-many matching not allowed: matching labels must be unique on one side"
level=warn ts=2019-03-06T06:20:25.779452969Z caller=manager.go:343 component="rule manager" group=k8s.rules msg="Evaluating rule failed" rule="record: namespace_name:container_memory_usage_bytes:sum\nexpr: sum by(namespace, label_name) (sum by(pod_name, namespace) (container_memory_usage_bytes{container_name!=\"\",image!=\"\",job=\"kubelet\"})\n  * on(namespace, pod_name) group_left(label_name) label_replace(kube_pod_labels{job=\"kube-state-metrics\"},\n  \"pod_name\", \"$1\", \"pod\", \"(.*)\"))\n" err="many-to-many matching not allowed: matching labels must be unique on one side"
level=warn ts=2019-03-06T06:20:25.782227903Z caller=manager.go:343 component="rule manager" group=k8s.rules msg="Evaluating rule failed" rule="record: namespace_name:kube_pod_container_resource_requests_memory_bytes:sum\nexpr: sum by(namespace, label_name) (sum by(namespace, pod) (kube_pod_container_resource_requests_memory_bytes{job=\"kube-state-metrics\"})\n  * on(namespace, pod) group_left(label_name) label_replace(kube_pod_labels{job=\"kube-state-metrics\"},\n  \"pod_name\", \"$1\", \"pod\", \"(.*)\"))\n" err="many-to-many matching not allowed: matching labels must be unique on one side"

Version-Release number of selected component (if applicable):
oc v3.11.88
monitoring images v3.11.88

How reproducible:
Sometimes

Steps to Reproduce:
1. # oc -n openshift-monitoring logs prometheus-k8s-1 -c prometheus | grep "Evaluating rule failed"
2.
3.

Actual results:
"many-to-many matching not allowed: matching labels must be unique on one side" warn info in prometheus-k8s pod

Expected results:


Additional info:
This warn info is not always happen, and it does not affect function now

Comment 4 Junqi Zhao 2019-03-07 11:14:33 UTC
Based on Comment 3, close it as NOTABUG


Note You need to log in before you can comment on or make changes to this bug.