Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1366143 - Failed container scanning job does not timeout
Summary: Failed container scanning job does not timeout
Alias: None
Product: Red Hat CloudForms Management Engine
Classification: Red Hat
Component: SmartState Analysis
Version: 5.6.0
Hardware: Unspecified
OS: Unspecified
Target Milestone: GA
: 5.7.0
Assignee: Mooli Tayer
QA Contact: Einat Pacifici
Whiteboard: container
: 1367473 (view as bug list)
Depends On:
Blocks: 1366598
TreeView+ depends on / blocked
Reported: 2016-08-11 07:56 UTC by Einat Pacifici
Modified: 2017-01-12 04:46 UTC (History)
10 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1366598 (view as bug list)
Last Closed: 2017-01-11 20:17:24 UTC
Category: ---
Cloudforms Team: ---
Target Upstream Version:

Attachments (Terms of Use)
evm.log attached. (deleted)
2016-08-11 07:56 UTC, Einat Pacifici
no flags Details
screenshots: Overview and Schedule (deleted)
2016-08-11 07:58 UTC, Einat Pacifici
no flags Details
Queue stuck (deleted)
2016-08-14 10:34 UTC, Mooli Tayer
no flags Details

Description Einat Pacifici 2016-08-11 07:56:06 UTC
Created attachment 1189937 [details]
evm.log attached.

Description of problem:
Pod deletion and Creation Trends should show created and deleted pods. The expected number of these pods is not showing up in the dashboard's "Pod Creation and deletion Trends" correctly

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:
1.Setup CFME with Provider and assign OpenSCAP policy to provider 
2.Generate hourly schedule to run SSA/OpenSCAP scan.24
3. Ensure no new pods are generated manually. 24
4.Wait 24 hours
5. Look at Compute --> Containers --> Overview: Pod Creation and Deletion Trends. 

Actual results:
146 Pods generated, 136 Pods deleted. 

Expected results:
24 pods generated. 24 pods deleted. 

Additional info:
attached evm.log; dashboard view; schedule view

Comment 2 Einat Pacifici 2016-08-11 07:58:25 UTC
Created attachment 1189942 [details]
screenshots: Overview and Schedule

Comment 3 Federico Simoncelli 2016-08-11 10:09:48 UTC
Yaacov, can you take a look at this? Thanks.

Comment 4 Federico Simoncelli 2016-08-11 10:09:49 UTC
Yaacov, can you take a look at this? Thanks.

Comment 7 Mooli Tayer 2016-08-14 10:34:10 UTC
Created attachment 1190758 [details]
Queue stuck

Comment 8 Mooli Tayer 2016-08-14 10:35:59 UTC
Fix submitted upstream:

Comment 9 Mooli Tayer 2016-08-14 10:58:16 UTC
relevant log(2016-08-08T03) at:

Attachment too large and bugzilla won't accept it.

Comment 11 Prasad Mukhedkar 2016-08-19 08:00:42 UTC
*** Bug 1367473 has been marked as a duplicate of this bug. ***

Comment 12 Mooli Tayer 2016-08-30 10:46:02 UTC
Since there is a default limit of 3 concurrent container scans per ems[1] jobs that are stuck could lead the job queue to be stuck and not allow new scans from that ems.

Workaround using rails console(while no new jobs are running):

cd /var/www/miq/vmdb/
source /etc/default/evm
bin/rails c

# We will do it in two steps to be careful:

2.3.0 :013 > Job.where("state != 'waiting_to_start' AND state != 'finished'").map(&:id)
 => [219, 217, 218, 225, 227, 228, 226, 229] 

2.3.0 :018 > Job.where(:id => [219, 217, 218, 225, 227, 228, 226, 229]).update(:state => 'finished', :dispatch_status => 'finished',:message => 'manually canceled' )

[1] settings.yml:
  :concurrent_per_ems: 3

Note You need to log in before you can comment on or make changes to this bug.