Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1596989 - Pods which go in the Evicted state are automatically deleted after a while [NEEDINFO]
Summary: Pods which go in the Evicted state are automatically deleted after a while
Keywords:
Status: NEW
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: RFE
Version: 3.7.0
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: ---
: ---
Assignee: Derek Carr
QA Contact: Xiaoli Tian
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-07-01 08:50 UTC by Madhusudan Upadhyay
Modified: 2019-04-16 11:23 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:
Target Upstream Version:
maupadhy: needinfo? (decarr)


Attachments (Terms of Use)

Description Madhusudan Upadhyay 2018-07-01 08:50:17 UTC
Proposed title of this feature request:
- Pods in Evicted state should be automatically deleted after a while


Version-Release number of selected component (if applicable):
- Openshift Container Platform 3.7


Account name: Australia and New Zealand Banking Group Limited

CSM customer: yes

TAM customer: yes

Strategic Customer: yes

Nature and description of the request:
- The Customer needs a feature wherein the pods that go into the evicted state are automatically deleted rather than manually progressing towards the deletion.

Business and Functional Requirements:
- Pods staying on the node in Evicted state promote misleading outcomes 
- This creates confusion in user space

Is there already an existing RFE upstream or in Red Hat bugzilla: No

Does the customer have any specific timeline dependencies: No

Would the customer be able to assist in testing this functionality if implemented: Yes

Additional Info:
- For the ease of deleting the pods(if there are plethora of pods on a particular node in Evicted state) I composed a script which would aid in deleting the respective pods. 
- Script:
  >> oc delete pods `oc get pods  -o wide| grep "affected node name" | grep Evicted | gawk -F' ' '{ print $1 }'`  --force --grace-period=0


Please let me know if there is more to append to the RFE from the customer's side.

Comment 3 Derek Carr 2018-10-23 14:40:07 UTC
An evicted pod is no different than any other pod that has reached its terminal status.  Is the user creating these pods via a workload controller (deployment, job, etc.) or some other method of producing standalone pods?  Is the request to more aggressively delete terminal pods from the API server?

The pod garbage collector scans every time the --terminated-pod-gc-threshold is exceeded (defaults to 12500) and will delete the terminal pods (including those evicted) from the cluster.  This value can be adjusted in the controller manager arguments to a lower value for more aggressive deletion.

Any specific changes to handling delete of evicted pods via an alternative policy is not going to happen until kubernetes 1.14 at earliest, but a blanket option to just force-delete them will probably not satisfy all users since it results in loss of understanding why the eviction occurred.  It's possible having a more regular sweep of evicted pods in the pod-gc-controller to remove evicted pods every N period of time could suffice.

Comment 7 Mitchell Rollinson 2019-01-09 00:26:45 UTC
Hi folks,

Can we please have an update for this High Profile APAC cu?

Is there any additional information that you require?

Kind Regards

Mitch

Comment 8 Madhusudan Upadhyay 2019-01-28 12:52:59 UTC
Hello Team, 

Can you please let me know if there is any information required to gain traction on the bugzilla.


Note You need to log in before you can comment on or make changes to this bug.