Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1686356 - when delete a kubeletconfig ,return time is too long(nearly 10 minutes)
Summary: when delete a kubeletconfig ,return time is too long(nearly 10 minutes)
Keywords:
Status: VERIFIED
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Pod
Version: 4.1
Hardware: x86_64
OS: Linux
unspecified
medium
Target Milestone: ---
: 4.1.0
Assignee: Ryan Phillips
QA Contact: Sunil Choudhary
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-03-07 10:18 UTC by MinLi
Modified: 2019-04-03 12:56 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:
Target Upstream Version:


Attachments (Terms of Use)

Description MinLi 2019-03-07 10:18:19 UTC
Description of problem:
when delete a kubeletconfig ,return time is too long(nearly 10 minutes)

Version-Release number of selected component (if applicable):
4.0.0-0.nightly-2019-03-04-234414
oc v4.0.0-0.182.0

How reproducible:
always

Steps to Reproduce:
1.edit machineconfigpool worker, add label"custom-kubelet: max-pods-worker"
#oc edit machineconfigpool worker
for example:
metadata:
  creationTimestamp: 2019-03-07T07:10:04Z
  generation: 1
  labels:
    custom-kubelet: max-pods-worker(add this line)
  ...

2.create a kubeletconfig cr
oc create -f worker-kube-config.yaml
yaml file as:
apiVersion: machineconfiguration.openshift.io/v1
kind: KubeletConfig
metadata:
  name: set-max-pods-worker
spec:
  machineConfigPoolSelector:
    matchLabels:
      custom-kubelet: max-pods-worker
  kubeletConfig:
    maxPods: 249

3.oc get machineconfig
generate a new machineconfig named "99-worker-74b4fa0b-4091-11e9-96ec-067bf345463e-kubelet"

4.login all worker node, check /etc/kubernetes/kubelet.conf has the kubelet configuration "maxPods: 249"

5.oc delete kubeletconfig set-max-pods-worker


Actual results:
5.show message "kubeletconfig.machineconfiguration.openshift.io "set-max-pods-worker" deleted", and return time of this command is nearly 10 minutes.

Expected results:
show message "kubeletconfig.machineconfiguration.openshift.io "set-max-pods-worker" deleted", and return immediately


Additional info:
when take step 5, check machineconfig in another terminal,the 99-worker-74b4fa0b-4091-11e9-96ec-067bf345463e-kubelet disappear immediately.


#oc logs machine-config-controller-8cc654f5d-rlckl -n openshift-machine-config-operator | grep set-max-pods-worker
error log:
I0307 09:45:50.893105       1 kubelet_config_controller.go:412] Applied KubeletConfig set-max-pods-worker on MachineConfigPool worker
I0307 09:45:51.048254       1 kubelet_config_controller.go:245] Error syncing kubeletconfig set-max-pods-worker: Operation cannot be fulfilled on kubeletconfigs.machineconfiguration.openshift.io "set-max-pods-worker": the object has been modified; please apply your changes to the latest version and try again
I0307 09:45:51.698293       1 kubelet_config_controller.go:412] Applied KubeletConfig set-max-pods-worker on MachineConfigPool worker
E0307 09:45:51.859637       1 kubelet_config_controller.go:250] Operation cannot be fulfilled on kubeletconfigs.machineconfiguration.openshift.io "set-max-pods-worker": the object has been modified; please apply your changes to the latest version and try again
I0307 09:45:51.859670       1 kubelet_config_controller.go:251] Dropping kubeletconfig "set-max-pods-worker" out of the queue: Operation cannot be fulfilled on kubeletconfigs.machineconfiguration.openshift.io "set-max-pods-worker": the object has been modified; please apply your changes to the latest version and try again
I0307 09:45:52.500112       1 kubelet_config_controller.go:412] Applied KubeletConfig set-max-pods-worker on MachineConfigPool worker
I0307 09:45:52.648116       1 kubelet_config_controller.go:245] Error syncing kubeletconfig set-max-pods-worker: Operation cannot be fulfilled on kubeletconfigs.machineconfiguration.openshift.io "set-max-pods-worker": the object has been modified; please apply your changes to the latest version and try again
I0307 09:45:53.292696       1 kubelet_config_controller.go:412] Applied KubeletConfig set-max-pods-worker on MachineConfigPool worker
I0307 09:45:53.455527       1 kubelet_config_controller.go:245] Error syncing kubeletconfig set-max-pods-worker: Operation cannot be fulfilled on kubeletconfigs.machineconfiguration.openshift.io "set-max-pods-worker": the object has been modified; please apply your changes to the latest version and try again

Comment 2 Ryan Phillips 2019-03-07 22:58:47 UTC
MinLi: Are you apply this configuration to the master nodes? The master nodes run the Machine Config Controller and would likely take some time to come back and reconcile the deletion.

Comment 3 Ryan Phillips 2019-04-01 14:28:31 UTC
PR merged. https://github.com/openshift/machine-config-operator/pull/536


Note You need to log in before you can comment on or make changes to this bug.