Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1363575 - Build log retention
Summary: Build log retention
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Pod
Version: 3.2.0
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: ---
Assignee: Jan Chaloupka
QA Contact: DeShuai Ma
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-08-03 06:00 UTC by Jaspreet Kaur
Modified: 2016-08-09 17:32 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-08-09 17:32:02 UTC


Attachments (Terms of Use)

Description Jaspreet Kaur 2016-08-03 06:00:42 UTC
Description of problem: Build logs removed even though the build pods are present. oc logs on an older build pod (3 days old) show :

Error from server: API error (404): no such id: fe59c0d8c019430c1b36f07db295953dd38a10523b7ce217f71c8233de8a8553

But the build and the build pod are still there.



Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 1 Cesar Wong 2016-08-03 13:12:44 UTC
This seems like a general Kube issue. It sounds like containers are being garbage collected while the pods remain.

Comment 2 Jan Chaloupka 2016-08-03 14:20:48 UTC
GC containers collecting algorithm is independent of what pod a given container comes from. Thus, all container that are older than minimum-container-ttl-duration are considered candidates for removal.

Currently, there is no way to skip containers that were created by a build pod. You can increase the minimum-container-ttl-duration to keep containers longer on the system. However, it will effect every container and it will take longer to collect/free your resources.

For docker, it is possible to extend the current GC code to skip certain containers based on their names. However, the solution is hacky. Other containerization technologies can have (e.g. rkt) their own GC implementation which is run independently of Kubernetes.

What I can think of is to update builder to keep the container running (or spin it from time to time for a fraction of time so it is considered dead no longer than the minimum-container-ttl-duration) until it reaches expected minimal build time to live.

Comment 3 Andy Goldstein 2016-08-09 17:32:02 UTC
See comment 2 for information about container garbage collection. If you need pod logs to exist beyond the lifespan of the associated container(s), you'll need to set up a persistent logging solution such as openshift-logging.


Note You need to log in before you can comment on or make changes to this bug.