Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1510069 - Service Template Provision Task Failing When Picked Up by Appliance in Wrong Zone
Summary: Service Template Provision Task Failing When Picked Up by Appliance in Wrong ...
Alias: None
Product: Red Hat CloudForms Management Engine
Classification: Red Hat
Component: Automate
Version: 5.8.0
Hardware: Unspecified
OS: Unspecified
Target Milestone: GA
: 5.10.0
Assignee: Greg McCullough
QA Contact: Shveta
Depends On:
Blocks: 1530674 1565248
TreeView+ depends on / blocked
Reported: 2017-11-06 16:26 UTC by myoder
Modified: 2018-06-21 20:36 UTC (History)
10 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1530674 1565248 (view as bug list)
Last Closed: 2018-06-21 20:36:31 UTC
Category: Bug
Cloudforms Team: CFME Core
Target Upstream Version:

Attachments (Terms of Use)
provision test script (deleted)
2017-12-11 21:28 UTC, Lucy Fu
no flags Details
new messages in miq_provision_virt_workflow.rb from (deleted)
2017-12-20 14:43 UTC, Lucy Fu
no flags Details

Comment 12 Lucy Fu 2017-12-11 21:28:42 UTC
Created attachment 1366312 [details]
provision test script

Comment 18 Lucy Fu 2017-12-20 14:43:51 UTC
Created attachment 1370497 [details]
new messages in miq_provision_virt_workflow.rb from

Comment 20 CFME Bot 2017-12-21 15:56:43 UTC
New commit detected on ManageIQ/manageiq/master:

commit 9c94f30b232d09be49b6d0952bf25f0243573bb1
Author:     Lucy Fu <>
AuthorDate: Wed Dec 20 09:50:58 2017 -0500
Commit:     Lucy Fu <>
CommitDate: Thu Dec 21 10:29:51 2017 -0500

    Fix allowed_vlans to call preload correctly.

 app/models/miq_provision_virt_workflow.rb       | 2 +-
 spec/models/miq_provision_virt_workflow_spec.rb | 6 ++++++
 2 files changed, 7 insertions(+), 1 deletion(-)

Comment 29 mkanoor 2018-02-20 15:02:37 UTC
Because of the network latency between the 2 zones we are seeing the service
provisioning process not getting started within the 10 minute window allocated
for a task to complete. Since CFME is highly asynchronous in task management it has 
strict constraints about how long a task can run. And that timeout is 10 minutes.
For longer running tasks we can make them asynchronous by exiting at apropos points
called states in Automate Model. The states can be restarted at different times.

So the suggestion is to force a retry in pre4/pre5 before we start the service

In the following instance


Find an open slot before the provision state
provision        /Service/Provisioning/StateMachines/Methods/Provision

I am guessing it would be pre4 or pre5

add set the value to  

What this does is calls a method that triggers a restart of the state satisfying 
the task controllers from not terminating non responsive tasks.

Add a ruby new method called set_retry_once in the class

the ruby method set_retry_once looks like

# Description: This method sets the retry once to force a break in the 
#              processing of long running tasks.
$evm.log(:info, "Checking if retry needs to be set")
if $evm.state_var_exist?('retry_once')
  ae_result = 'ok'
  $evm.set_state_var('retry_once', '1')
  ae_result = 'retry'
  $evm.log(:info, 'setting a retry once in the beginning')
$evm.root['ae_result'] = ae_result
$evm.root['retry_interval'] = 1.minute

Comment 31 mkanoor 2018-02-27 14:52:54 UTC
Once the change has been made in the Automate Database you dont have to restart the servers, the Automate Model changes are picked up automatically at runtime.

Comment 33 mkanoor 2018-02-27 15:57:35 UTC
I am not sure what the recommended way of doing this is.
Can you check with Josh.

One way I can think of is to turn off the Automate Role in the CIN Zone which will end up routing all the Automate work to CAR. It might overwhelm the CAR zone so it might have to be done during off peak hours.

How did this get tested with Lucys fix, what was done to route the work to CAR zone during that testing

Comment 34 Greg McCullough 2018-03-14 16:20:23 UTC
Since the work starts as a generic service it is not tied to any zone, just the automate role.  As suggested above one possibility would be to disable the Automate Role in the other zone to force the work to the CAR zone.

Maybe this is something the customer could attempt during off-hours to avoid performance issues.

Comment 36 mkanoor 2018-05-07 15:37:14 UTC
Hi Michael,
Any updates on this one from the customer where they able to test the suggestions.

Comment 38 Greg McCullough 2018-06-08 21:11:35 UTC
Moving to POST based on performance changes from Comment #19.

Note You need to log in before you can comment on or make changes to this bug.