Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.

Bug 1333957

Summary: Can't specify all answers in hosted engine setup answers file
Product: Red Hat Enterprise Virtualization Manager Reporter: Fabian von Feilitzsch <fabian>
Component: ovirt-hosted-engine-setupAssignee: Simone Tiraboschi <stirabos>
Status: CLOSED WORKSFORME QA Contact: meital avital <mavital>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 3.6.5CC: dfediuck, dgao, fabian, gklein, jesusr, jmatthew, jmontleo, lsurette, sbonazzo, stirabos, ykaul
Target Milestone: ovirt-3.6.7Keywords: Regression, Reopened
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-05-17 20:56:09 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Integration RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Bug Depends On:    
Bug Blocks: 1099562    
Attachments:
Description Flags
Example answerfile to deploy on NFS none

Description Fabian von Feilitzsch 2016-05-06 19:30:58 UTC
Description of problem:

When running hosted-engine --deploy --config-append=/path/to/answers, some manual intervention is still required. Specifically, at least the following variables in the answer file are ignored:

  OVEHOSTED_ENGINE/appHostName
  OVEHOSTED_NETWORK/bridgeIf
  OVEHOSTED_COREENV_TEMPDIR

This makes it much more difficult to do an automated, non-interactive deploy of RHEV self-hosted (currently, we achieve this by invoking the command with yes '' | hosted-engine --deploy --config-append=/path/to/answers to accept the defaults).

Version-Release number of selected component (if applicable):
This is with RHEV 3.6 on RHEL 7.2

How reproducible:
100%

Steps to Reproduce:
1. Get/create a fully completed answer file
2. run hosted-engine --deploy /path/to/answers

Actual results:
I am prompted for additional input

Expected results:
No further interaction required with the installer

Additional info:
Specifying it in the answers file isn't necessary, additional command line flags/configuration files would be fine too. Just anything that would allow a fully non-interactive install

Comment 1 John Matthews 2016-05-06 21:56:15 UTC
This issue is impacting the ability to deploy self hosted by the Quickstart Cloud Installer (Red Hat Cloud Suite).

Comment 2 Yaniv Kaul 2016-05-07 09:04:44 UTC
I know we've fixed a similar/identical issue - providing defaults for all answers, but don't recall if it's in 3.6.x or 4.0. I hope you are testing with 3.6.5.

Comment 3 Fabian von Feilitzsch 2016-05-07 15:07:16 UTC
I am using 3.6.5, and there are default values for the answers, but I am prompted to either input a new value or accept the default, and I can't override the defaults in a non-interactive installation.

Comment 4 Sandro Bonazzola 2016-05-09 13:45:51 UTC
(In reply to Fabian von Feilitzsch from comment #3)
> I am using 3.6.5, and there are default values for the answers, but I am
> prompted to either input a new value or accept the default, and I can't
> override the defaults in a non-interactive installation.

Simone please check.
As far as I know if you specify mentioned values in the answer file, they'll be used. If this has been broken, it's a regression.

Comment 5 Simone Tiraboschi 2016-05-10 07:33:51 UTC
(In reply to Fabian von Feilitzsch from comment #0)
>   OVEHOSTED_ENGINE/appHostName
>   OVEHOSTED_NETWORK/bridgeIf
>   OVEHOSTED_COREENV_TEMPDIR

The last key should be OVEHOSTED_CORE/tempDir, trying to reproduce for the others.

Comment 6 Simone Tiraboschi 2016-05-10 10:04:19 UTC
Created attachment 1155641 [details]
Example answerfile to deploy on NFS

Comment 7 Simone Tiraboschi 2016-05-10 10:06:20 UTC
Unable to reproduce with ovirt-hosted-engine-setup 1.3.5.0-1.1.el7 and the answerfile I'm attaching here as an example.

Please note that if you want to have multiple test in the same network env it would be necessary to randomize at lest the mac address of the engine VM.

Comment 8 Fabian von Feilitzsch 2016-05-10 17:42:50 UTC
Using your answers I did get that behavior, thank you for that and my mistake. However, I still get the following prompt:

Make a selection from the options below:
          (1) Continue setup - oVirt-Engine installation is ready and ovirt-engine service is up
          (2) Abort setup
          (3) Power off and restart the VM
          (4) Destroy VM and abort setup

          (1, 2, 3, 4)[1]: 

I am doing the cloud init with my own ISO, I'm not sure if that makes a difference.

Comment 9 John Matthews 2016-05-10 18:11:25 UTC
Simone,

I reopened this due to comment #8.

If you'd prefer we file a new BZ instead, please let us know.

Comment 10 Simone Tiraboschi 2016-05-15 07:26:59 UTC
Fabian,
Are you using also:
 OVEHOSTED_VM/automateVMShutdown=bool:True
 OVEHOSTED_VM/cloudinitExecuteEngineSetup=bool:True
?

Comment 11 Fabian von Feilitzsch 2016-05-15 23:29:24 UTC
I am using  OVEHOSTED_VM/automateVMShutdown=bool:True, but not  OVEHOSTED_VM/cloudinitExecuteEngineSetup=bool:True. Here's is the full template for the answer file I am testing with: 

[environment:default]
OVEHOSTED_CORE/confirmSettings=bool:True
OVEHOSTED_CORE/deployProceed=bool:True
OVEHOSTED_CORE/screenProceed=bool:True
OVEHOSTED_ENGINE/adminPassword=str:{{ engine_admin_password }}
OVEHOSTED_ENGINE/appHostName=str:hosted_engine_1
OVEHOSTED_ENGINE/clusterName=str:{{ cluster_name }}
OVEHOSTED_NETWORK/bridgeIf=str:eth0
OVEHOSTED_NETWORK/bridgeName=str:ovirtmgmt
OVEHOSTED_NETWORK/firewallManager=str:iptables
OVEHOSTED_NETWORK/fqdn=str:{{ engine_fqdn }}
OVEHOSTED_NETWORK/gateway=str:{{ gateway }}
OVEHOSTED_NOTIF/destEmail=str:root@localhost
OVEHOSTED_NOTIF/smtpPort=str:25
OVEHOSTED_NOTIF/smtpServer=str:localhost
OVEHOSTED_NOTIF/sourceEmail=str:root@localhost
OVEHOSTED_STORAGE/domainType=str:nfs3
OVEHOSTED_STORAGE/imgSizeGB=int:50
OVEHOSTED_STORAGE/storageDatacenterName=str:{{ dc_name }}
OVEHOSTED_STORAGE/storageDomainConnection=str:{{ hosted_storage_address }}:{{ hosted_storage_path }}
OVEHOSTED_STORAGE/storageDomainName=str:{{ hosted_storage_name }}
OVEHOSTED_VDSM/caSubject=str:/C=EN/L=Test/O=Test/CN=TestCA
OVEHOSTED_VDSM/consoleType=str:vnc
OVEHOSTED_VDSM/cpu=str:{{ cpu_model }}
OVEHOSTED_VDSM/pkiSubject=str:/C=EN/L=Test/O=Test/CN=Test
OVEHOSTED_VDSM/spicePkiSubject=str:C=EN, L=Test, O=Test, CN=Test
OVEHOSTED_VM/automateVMShutdown=bool:True
OVEHOSTED_VM/emulatedMachine=str:rhel6.5.0
OVEHOSTED_VM/ovfArchive=str:{{ rhevm_appliance_path }}
OVEHOSTED_VM/vmBoot=str:disk
OVEHOSTED_VM/vmMACAddr=str:{{ engine_mac_address  }}
OVEHOSTED_VM/vmMemSizeMB=int:4096
OVEHOSTED_VM/vmVCpus=str:2


OVEHOSTED_VM/cloudInitISO=str:existing
OVEHOSTED_VM/vmCDRom=str:/tmp/cloud.iso
OVEHOSTED_VM/cloudinitRootPwd=str:{{ root_user_password }}

# Multiple hypervisor things
OVEHOSTED_NET_FIRST_HOST_FQDN=str:{{ first_host_fqdn }}
OVEHOSTED_CORE_FETCH_ANSWER=str:Yes
HOST_FIRST_HOST_ROOT_PASSWORD=str:{{ first_host_password }

OVEHOSTED_ENGINE/insecureSSL=none:None
OVEHOSTED_STORAGE/glusterBrick=none:None
OVEHOSTED_STORAGE/LunID=none:None
OVEHOSTED_STORAGE/iSCSIPortalIPAddress=none:None
OVEHOSTED_STORAGE/iSCSITargetName=none:None
OVEHOSTED_STORAGE/glusterProvisionedShareName=str:hosted_engine_glusterfs
OVEHOSTED_STORAGE/iSCSIPortalPort=none:None
OVEHOSTED_STORAGE/glusterProvisioningEnabled=bool:False
OVEHOSTED_STORAGE/iSCSIPortal=none:None
OVEHOSTED_STORAGE/storageType=str:{{ storage_type }}

OVEHOSTED_STORAGE/iSCSIPortalUser=none:None

NETWORK/sshEnable=bool:True
NETWORK/sshUser=str:root

Comment 12 Simone Tiraboschi 2016-05-17 15:44:37 UTC
(In reply to Fabian von Feilitzsch from comment #11)
> # Multiple hypervisor things
> OVEHOSTED_NET_FIRST_HOST_FQDN=str:{{ first_host_fqdn }}
> OVEHOSTED_CORE_FETCH_ANSWER=str:Yes
> HOST_FIRST_HOST_ROOT_PASSWORD=str:{{ first_host_password }

This is not needed anymore since now we read the answerfile directly from the shared domain

Comment 13 Simone Tiraboschi 2016-05-17 16:11:11 UTC
Unable to reproduce with this answer file: it correctly deploys in unmanaged way.

Please take care that 
OVEHOSTED_VM/cloudInitISO=str:existing
OVEHOSTED_VM/vmCDRom=str:/tmp/cloud.iso
means that you want to provide a custom cloud-init script and it's up to that to correctly run engine-setup on the engine VM as hosted-engine setup expects from it.
Are you really sure that you need to use a custom cloud-init script?
If so, take what /usr/share/ovirt-hosted-engine-setup/plugins/ovirt-hosted-engine-setup/vm/cloud_init.py does as a reference.
If not, you can simply pass all the values you have under CloudInit section in /usr/lib/python2.7/site-packages/ovirt_hosted_engine_setup/constants.py
with 
OVEHOSTED_VM/cloudInitISO=str:generate and hosted-engine-setup will generate a cloud-init script for you.

[root@c72he20160517h1 ~]# cat answer.conf 
[environment:default]
OVEHOSTED_CORE/confirmSettings=bool:True
OVEHOSTED_CORE/deployProceed=bool:True
OVEHOSTED_CORE/screenProceed=bool:True
OVEHOSTED_ENGINE/adminPassword=str:testpwd
OVEHOSTED_ENGINE/appHostName=str:hosted_engine_1
OVEHOSTED_ENGINE/clusterName=str:Default
OVEHOSTED_NETWORK/bridgeIf=str:eth0
OVEHOSTED_NETWORK/bridgeName=str:ovirtmgmt
OVEHOSTED_NETWORK/firewallManager=str:iptables
OVEHOSTED_NETWORK/fqdn=str:enginevm.localdomain
OVEHOSTED_NETWORK/gateway=str:192.168.1.1
OVEHOSTED_NOTIF/destEmail=str:root@localhost
OVEHOSTED_NOTIF/smtpPort=str:25
OVEHOSTED_NOTIF/smtpServer=str:localhost
OVEHOSTED_NOTIF/sourceEmail=str:root@localhost
OVEHOSTED_STORAGE/domainType=str:nfs4
OVEHOSTED_STORAGE/imgSizeGB=int:50
OVEHOSTED_STORAGE/storageDatacenterName=str:Default
OVEHOSTED_STORAGE/storageDomainConnection=str:192.168.1.115:/Virtual/exthe6
OVEHOSTED_STORAGE/storageDomainName=str:he_storage
OVEHOSTED_VDSM/caSubject=str:/C=EN/L=Test/O=Test/CN=TestCA
OVEHOSTED_VDSM/consoleType=str:vnc
OVEHOSTED_VDSM/cpu=str:model_SandyBridge
OVEHOSTED_VDSM/pkiSubject=str:/C=EN/L=Test/O=Test/CN=Test
OVEHOSTED_VDSM/spicePkiSubject=str:C=EN, L=Test, O=Test, CN=Test
OVEHOSTED_VM/automateVMShutdown=bool:True
OVEHOSTED_VM/emulatedMachine=str:rhel6.5.0
OVEHOSTED_VM/ovfArchive=str:/usr/share/ovirt-engine-appliance/ovirt-engine-appliance-3.6-20160502.1.el7.centos.ova
OVEHOSTED_VM/vmBoot=str:disk
OVEHOSTED_VM/vmMACAddr=str:00:16:3e:71:a0:c7
OVEHOSTED_VM/vmMemSizeMB=int:4096
OVEHOSTED_VM/vmVCpus=str:2
OVEHOSTED_VM/cloudInitISO=str:existing
OVEHOSTED_VM/vmCDRom=str:/tmp/cloud.iso
OVEHOSTED_VM/cloudinitRootPwd=str:testpwd
OVEHOSTED_VM/cloudinitExecuteEngineSetup=bool:True


OVEHOSTED_ENGINE/insecureSSL=none:None
OVEHOSTED_STORAGE/glusterBrick=none:None
OVEHOSTED_STORAGE/LunID=none:None
OVEHOSTED_STORAGE/iSCSIPortalIPAddress=none:None
OVEHOSTED_STORAGE/iSCSITargetName=none:None
OVEHOSTED_STORAGE/glusterProvisionedShareName=str:hosted_engine_glusterfs
OVEHOSTED_STORAGE/iSCSIPortalPort=none:None
OVEHOSTED_STORAGE/glusterProvisioningEnabled=bool:False
OVEHOSTED_STORAGE/iSCSIPortal=none:None

OVEHOSTED_STORAGE/iSCSIPortalUser=none:None

NETWORK/sshEnable=bool:True
NETWORK/sshUser=str:root


Here my output:
 [root@c72he20160517h1 ~]# hosted-engine --deploy --config-append=answer.conf 
 [ INFO  ] Stage: Initializing
 [ INFO  ] Generating a temporary VNC password.
 [ INFO  ] Stage: Environment setup
           Configuration files: ['/root/answer.conf']
           Log file: /var/log/ovirt-hosted-engine-setup/ovirt-hosted-engine-setup-20160517175320-kcc3t2.log
           Version: otopi-1.4.1 (otopi-1.4.1-1.el7.centos)
 [ INFO  ] Hardware supports virtualization
 [ INFO  ] Bridge ovirtmgmt already created
 [ INFO  ] Stage: Environment packages setup
 [ INFO  ] Stage: Programs detection
 [ INFO  ] Stage: Environment setup
 [ INFO  ] Stage: Environment customization
          
           --== STORAGE CONFIGURATION ==--
          
           During customization use CTRL-D to abort.
 [ INFO  ] Installing on first host
          
           --== SYSTEM CONFIGURATION ==--
          
          
           --== NETWORK CONFIGURATION ==--
          
          
           --== VM CONFIGURATION ==--
          
 [ INFO  ] Checking OVF archive content (could take a few minutes depending on archive size)
 [ INFO  ] Checking OVF XML content (could take a few minutes depending on archive size)
 [WARNING] OVF does not contain a valid image description, using default.
           The following CPU types are supported by this host:
           	 - model_SandyBridge: Intel SandyBridge Family
           	 - model_Westmere: Intel Westmere Family
           	 - model_Nehalem: Intel Nehalem Family
           	 - model_Penryn: Intel Penryn Family
           	 - model_Conroe: Intel Conroe Family
 [WARNING] Minimum requirements for disk size not met
          
           --== HOSTED ENGINE CONFIGURATION ==--
          
 [ INFO  ] Stage: Setup validation
          
           --== CONFIGURATION PREVIEW ==--
          
           Bridge interface                   : eth0
           Engine FQDN                        : enginevm.localdomain
           Bridge name                        : ovirtmgmt
           Host address                       : c72he20160517h1.localdomain
           SSH daemon port                    : 22
           Firewall manager                   : iptables
           Gateway address                    : 192.168.1.1
           Host name for web application      : hosted_engine_1
           Host ID                            : 1
           Image size GB                      : 10
           GlusterFS Share Name               : hosted_engine_glusterfs
           GlusterFS Brick Provisioning       : False
           Storage connection                 : 192.168.1.115:/Virtual/exthe6
           Console type                       : vnc
           Memory size MB                     : 4096
           MAC address                        : 00:16:3e:71:a0:c7
           Boot type                          : disk
           Number of CPUs                     : 2
           OVF archive (for disk boot)        : /usr/share/ovirt-engine-appliance/ovirt-engine-appliance-3.6-20160502.1.el7.centos.ova
           ISO image (cdrom boot/cloud-init)  : /tmp/cloud.iso
           Restart engine VM after engine-setup: True
           CPU Type                           : model_SandyBridge
 [ INFO  ] Stage: Transaction setup
 [ INFO  ] Stage: Misc configuration
 [ INFO  ] Stage: Package installation
 [ INFO  ] Stage: Misc configuration
 [ INFO  ] Configuring libvirt
 [ INFO  ] Configuring VDSM
 [ INFO  ] Starting vdsmd
 [ INFO  ] Waiting for VDSM hardware info
 [ INFO  ] Creating Storage Domain
 [ INFO  ] Creating Storage Pool
 [ INFO  ] Connecting Storage Pool
 [ INFO  ] Verifying sanlock lockspace initialization
 [ INFO  ] Creating Image for 'hosted-engine.lockspace' ...
 [ INFO  ] Image for 'hosted-engine.lockspace' created successfully
 [ INFO  ] Creating Image for 'hosted-engine.metadata' ...
 [ INFO  ] Image for 'hosted-engine.metadata' created successfully
 [ INFO  ] Creating VM Image
 [ INFO  ] Extracting disk image from OVF archive (could take a few minutes depending on archive size)
 [ INFO  ] Validating pre-allocated volume size
 [ INFO  ] Uploading volume to data domain (could take a few minutes depending on archive size)
 [ INFO  ] Image successfully imported from OVF
 [ INFO  ] Destroying Storage Pool
 [ INFO  ] Start monitoring domain
 [ INFO  ] Configuring VM
 [ INFO  ] Updating hosted-engine configuration
 [ INFO  ] Stage: Transaction commit
 [ INFO  ] Stage: Closing up
 [ INFO  ] Creating VM
           You can now connect to the VM with the following command:
           	/bin/remote-viewer vnc://localhost:5900
           Use temporary password "7109aryq" to connect to vnc console.
           Please note that in order to use remote-viewer you need to be able to run graphical applications.
           This means that if you are using ssh you have to supply the -Y flag (enables trusted X11 forwarding).
           Otherwise you can run the command from a terminal in your preferred desktop environment.
           If you cannot run graphical applications you can connect to the graphic console from another host or connect to the serial console using the following command:
           socat UNIX-CONNECT:/var/run/ovirt-vmconsole-console/acc651ff-1168-4b31-8838-45f11630411c.sock,user=ovirt-vmconsole STDIO,raw,echo=0,escape=1
           Please ensure that your Guest OS is properly configured to support serial console according to your distro documentation.
           Follow http://www.ovirt.org/Serial_Console_Setup#I_need_to_access_the_console_the_old_way for more info.
           If you need to reboot the VM you will need to start it manually using the command:
           hosted-engine --vm-start
           You can then set a temporary password using the command:
           hosted-engine --add-console-password
 [ INFO  ] Running engine-setup on the appliance
           |- [ INFO  ] Stage: Initializing
           |- [ INFO  ] Stage: Environment setup
           |-           Configuration files: ['/etc/ovirt-engine-setup.conf.d/10-packaging-jboss.conf', '/etc/ovirt-engine-setup.conf.d/10-packaging.conf', '/root/ovirt-engine-answers', '/root/heanswers.conf']
           |-           Log file: /var/log/ovirt-engine/setup/ovirt-engine-setup-20160517155803-3kmkq0.log
           |-           Version: otopi-1.4.1 (otopi-1.4.1-1.el7.centos)
           |- [ INFO  ] Stage: Environment packages setup
           |- [ INFO  ] Stage: Programs detection
           |- [ INFO  ] Stage: Environment setup
           |- [ INFO  ] Stage: Environment customization
           |-          
           |-           --== PRODUCT OPTIONS ==--
           |-          
           |-          
           |-           --== PACKAGES ==--
           |-          
           |-          
           |-           --== ALL IN ONE CONFIGURATION ==--
           |-          
           |-          
           |-           --== NETWORK CONFIGURATION ==--
           |-          
           |- [WARNING] Failed to resolve enginevm.localdomain using DNS, it can be resolved only locally
           |- [ INFO  ] firewalld will be configured as firewall manager.
           |- [WARNING] Failed to resolve enginevm.localdomain using DNS, it can be resolved only locally
           |- [WARNING] Failed to resolve enginevm.localdomain using DNS, it can be resolved only locally
           |-          
           |-           --== DATABASE CONFIGURATION ==--
           |-          
           |-          
           |-           --== OVIRT ENGINE CONFIGURATION ==--
           |-          
           |-          
           |-           --== STORAGE CONFIGURATION ==--
           |-          
           |-          
           |-           --== PKI CONFIGURATION ==--
           |-          
           |-          
           |-           --== APACHE CONFIGURATION ==--
           |-          
           |-          
           |-           --== SYSTEM CONFIGURATION ==--
           |-          
           |-          
           |-           --== MISC CONFIGURATION ==--
           |-          
           |-          
           |-           --== END OF CONFIGURATION ==--
           |-          
           |- [ INFO  ] Stage: Setup validation
           |- [WARNING] Cannot validate host name settings, reason: cannot resolve own name 'enginevm'
           |- [WARNING] Less than 16384MB of memory is available
           |-          
           |-           --== CONFIGURATION PREVIEW ==--
           |-          
           |-           Application mode                        : virt
           |-           Default SAN wipe after delete           : False
           |-           Firewall manager                        : firewalld
           |-           Update Firewall                         : True
           |-           Host FQDN                               : enginevm.localdomain
           |-           Engine database secured connection      : False
           |-           Engine database host                    : localhost
           |-           Engine database user name               : engine
           |-           Engine database name                    : engine
           |-           Engine database port                    : 5432
           |-           Engine database host name validation    : False
           |-           Engine installation                     : True
           |-           PKI organization                        : localdomain
           |-           Configure local Engine database         : True
           |-           Set application as default page         : True
           |-           Configure Apache SSL                    : True
           |-           Configure VMConsole Proxy               : True
           |-           Engine Host FQDN                        : enginevm.localdomain
           |-           Configure WebSocket Proxy               : True
           |- [ INFO  ] Stage: Transaction setup
           |- [ INFO  ] Stopping engine service
           |- [ INFO  ] Stopping ovirt-fence-kdump-listener service
           |- [ INFO  ] Stopping websocket-proxy service
           |- [ INFO  ] Stage: Misc configuration
           |- [ INFO  ] Stage: Package installation
           |- [ INFO  ] Stage: Misc configuration
           |- [ INFO  ] Initializing PostgreSQL
           |- [ INFO  ] Creating PostgreSQL 'engine' database
           |- [ INFO  ] Configuring PostgreSQL
           |- [ INFO  ] Creating/refreshing Engine database schema
           |- [ INFO  ] Creating/refreshing Engine 'internal' domain database schema
           |- [ INFO  ] Upgrading CA
           |- [ INFO  ] Creating CA
           |- [ INFO  ] Setting up ovirt-vmconsole proxy helper PKI artifacts
           |- [ INFO  ] Setting up ovirt-vmconsole SSH PKI artifacts
           |- [ INFO  ] Configuring WebSocket Proxy
           |- [ INFO  ] Generating post install configuration file '/etc/ovirt-engine-setup.conf.d/20-setup-ovirt-post.conf'
           |- [ INFO  ] Stage: Transaction commit
           |- [ INFO  ] Stage: Closing up
           |-          
           |-           --== SUMMARY ==--
           |-          
           |- [WARNING] Less than 16384MB of memory is available
           |-           SSH fingerprint: 42:f1:cc:db:d0:bd:9e:02:b3:95:0d:fc:83:28:c2:f1
           |-           Internal CA 45:D9:54:31:E8:4F:63:F0:BB:90:EC:8E:AD:81:17:03:61:7F:1B:BA
           |-           Note! If you want to gather statistical information you can install Reports and/or DWH:
           |-               http://www.ovirt.org/Ovirt_DWH
           |-               http://www.ovirt.org/Ovirt_Reports
           |-           Web access is enabled at:
           |-               http://enginevm.localdomain:80/ovirt-engine
           |-               https://enginevm.localdomain:443/ovirt-engine
           |-           Please use the user 'admin@internal' and password specified in order to login
           |-          
           |-           --== END OF SUMMARY ==--
           |-          
           |- [ INFO  ] Starting engine service
           |- [ INFO  ] Restarting httpd
           |- [ INFO  ] Restarting ovirt-vmconsole proxy service
           |- [ INFO  ] Stage: Clean up
           |-           Log file is located at /var/log/ovirt-engine/setup/ovirt-engine-setup-20160517155803-3kmkq0.log
           |- [ INFO  ] Generating answer file '/var/lib/ovirt-engine/setup/answers/20160517160436-setup.conf'
           |- [ INFO  ] Stage: Pre-termination
           |- [ INFO  ] Stage: Termination
           |- [ INFO  ] Execution of setup completed successfully
           |- HE_APPLIANCE_ENGINE_SETUP_SUCCESS
 [ INFO  ] Engine-setup successfully completed 
 [ INFO  ] Engine is still unreachable
 [ INFO  ] Engine is still not reachable, waiting...
 [ INFO  ] Engine is still unreachable
 [ INFO  ] Engine is still not reachable, waiting...
 [ INFO  ] Engine replied: DB Up!Welcome to Health Status!
 [ INFO  ] Acquiring internal CA cert from the engine
 [ INFO  ] The following CA certificate is going to be used, please immediately interrupt if not correct:
 [ INFO  ] Issuer: C=US, O=localdomain, CN=enginevm.localdomain.43545, Subject: C=US, O=localdomain, CN=enginevm.localdomain.43545, Fingerprint (SHA-1): 45D95431E84F63F0BB90EC8EAD811703617F1BBA
 [ INFO  ] Connecting to the Engine
 [ INFO  ] Waiting for the host to become operational in the engine. This may take several minutes...
 [ INFO  ] Still waiting for VDSM host to become operational...
 [ INFO  ] The VDSM Host is now operational
 [ INFO  ] Saving hosted-engine configuration on the shared storage domain
 [ INFO  ] Shutting down the engine VM
 [ INFO  ] Enabling and starting HA services
 [ INFO  ] Stage: Clean up
 [ INFO  ] Generating answer file '/var/lib/ovirt-hosted-engine-setup/answers/answers-20160517180918.conf'
 [ INFO  ] Generating answer file '/etc/ovirt-hosted-engine/answers.conf'
 [ INFO  ] Stage: Pre-termination
 [ INFO  ] Stage: Termination
 [ INFO  ] Hosted Engine successfully set up
 [root@c72he20160517h1 ~]#

Comment 14 Fabian von Feilitzsch 2016-05-17 16:36:12 UTC
> Please take care that 
> OVEHOSTED_VM/cloudInitISO=str:existing
> OVEHOSTED_VM/vmCDRom=str:/tmp/cloud.iso
> means that you want to provide a custom cloud-init script and it's up to that to correctly run engine-setup on the engine VM as hosted-engine setup expects from it.
> Are you really sure that you need to use a custom cloud-init script?
> If so, take what /usr/share/ovirt-hosted-engine-setup/plugins/ovirt-hosted-engine-setup/vm/cloud_init.py does as a reference.
> If not, you can simply pass all the values you have under CloudInit section in /usr/lib/python2.7/site-packages/ovirt_hosted_engine_setup/constants.py 
> with 
> OVEHOSTED_VM/cloudInitISO=str:generate and hosted-engine-setup will generate a cloud-init script for you.

Yes, we have a cloud init ISO that successfully executes engine-setup, as well as some additional configuration (rhevm-dwh + reports, satellite registration, additional ssh configuration, etc). Have you tested an installation using a custom cloud init ISO rather than the generated one? When I run the hosted-engine --deploy, I do not get the engine-setup logs in my hosted-engine log, and it does give me the prompt. If I interact with the prompt after engine-setup completes on the engine VM, then everything finishes out as expected.

Comment 15 Fabian von Feilitzsch 2016-05-17 18:23:19 UTC
I can confirm that I do not get the prompt when using a generated ISO though, and the domain monitor comes up and gives me the engine logs. Is there something I need to do/configure in my answers or in the cloud init script to get the domain monitor to try to connect to my engine rather than prompt and wait?

Comment 16 Simone Tiraboschi 2016-05-17 20:56:09 UTC
(In reply to Fabian von Feilitzsch from comment #14)
> Yes, we have a cloud init ISO that successfully executes engine-setup, as
> well as some additional configuration (rhevm-dwh + reports, satellite
> registration, additional ssh configuration, etc). Have you tested an
> installation using a custom cloud init ISO rather than the generated one?

Yes, the one at comment https://bugzilla.redhat.com/show_bug.cgi?id=1333957#c13 has:
 OVEHOSTED_VM/cloudInitISO=str:existing
 OVEHOSTED_VM/vmCDRom=str:/tmp/cloud.iso
 OVEHOSTED_VM/cloudinitExecuteEngineSetup=bool:True
so it wasn't generating it but it was using an already existing cloud-init ISO available at /tmp/cloud.iso (I simply generated it from hosted-engine-setup and on previous run monkey patching the code for that and simply copied it at /tmp/cloud.iso )


> When I run the hosted-engine --deploy, I do not get the engine-setup logs in
> my hosted-engine log, and it does give me the prompt.

Are you using OVEHOSTED_VM/cloudinitExecuteEngineSetup=bool:True ? you have to.

> If I interact with the
> prompt after engine-setup completes on the engine VM, then everything
> finishes out as expected.

hosted-engine setup has to monitor the execution of engine-setup inside the engine VM, to do that on the generated cloud-init image we redirect the output of engine-setup to a virtio port that is being monitored by hosted-engine-setup on the host.
Are you doing that on your custom cloud-init image? You can use /usr/share/ovirt-hosted-engine-setup/plugins/ovirt-hosted-engine-setup/vm/cloud_init.py as a reference.

Take this as an example:
[root@c72he20160517h1 ci]# cat meta-data 
instance-id: 4fbfff12-d793-435b-9784-873058167f90
local-hostname: enginevm.localdomain
network-interfaces: |
  auto eth0
  iface eth0 inet static
    address 192.168.1.232
    network 192.168.1.0
    netmask 255.255.255.0
    broadcast 192.168.1.255
    gateway 192.168.1.1
[root@c72he20160517h1 ci]# cat user-data 
#cloud-config
# vim: syntax=yaml
ssh_pwauth: True
chpasswd:
  list: |
    root:testpwd
  expire: False
bootcmd:
 - echo "192.168.1.132 c72he20160517h1.localdomain" >> /etc/hosts
 - echo "192.168.1.232 enginevm.localdomain" >> /etc/hosts
 - echo "DNS1=8.8.8.8" >> /etc/sysconfig/network-scripts/ifcfg-eth0
 - echo "DOMAIN=localdomain" >> /etc/sysconfig/network-scripts/ifcfg-eth0
 - ifdown eth0
 - ifup eth0
write_files:
 - content: |
     [environment:default]
     OVESETUP_CONFIG/adminPassword=str:testpwd
     OVESETUP_CONFIG/fqdn=str:enginevm.localdomain
     OVESETUP_PKI/organization=str:localdomain
   path: /root/heanswers.conf
   owner: root:root
   permissions: '0640'
runcmd:
 - /usr/bin/engine-setup --offline --config-append=/root/ovirt-engine-answers --config-append=/root/heanswers.conf 1>/dev/virtio-ports/org.ovirt.hosted-engine-setup.0 2>&1
 - if [ $? -eq 0 ]; then echo "HE_APPLIANCE_ENGINE_SETUP_SUCCESS" >/dev/virtio-ports/org.ovirt.hosted-engine-setup.0; else echo "HE_APPLIANCE_ENGINE_SETUP_FAIL" >/dev/virtio-ports/org.ovirt.hosted-engine-setup.0; fi
 - rm /root/heanswers.conf
 - systemctl mask cloud-init-local ||  chkconfig cloud-init-local off
 - systemctl mask cloud-init || ( chkconfig cloud-init off && chkconfig cloud-config off && chkconfig cloud-final off )

Comment 17 Fabian von Feilitzsch 2016-05-18 19:03:54 UTC
Thank you so much for your time, I found the issue (PEBCAC). I had specified  OVEHOSTED_VM/cloudInitExecuteEngineSetup=bool:True , instead of  OVEHOSTED_VM/cloudinitExecuteEngineSetup=bool:True . With the casing correct, everything works perfectly.