Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1355643 - Node status info load fail on oVirt Dashboard after RHVH installed with ks file.
Summary: Node status info load fail on oVirt Dashboard after RHVH installed with ks file.
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: ovirt-node
Classification: oVirt
Component: UI
Version: 4.0
Hardware: Unspecified
OS: Unspecified
unspecified
high vote
Target Milestone: ovirt-4.0.1
: 4.0
Assignee: Ryan Barry
QA Contact: Wei Wang
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-07-12 07:10 UTC by Wei Wang
Modified: 2016-08-04 13:32 UTC (History)
12 users (show)

Fixed In Version: cockpit-ovirt-0.10.6-1.3.1.el7ev
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-08-04 13:32:06 UTC
oVirt Team: Node
dfediuck: ovirt-4.0.z+
rule-engine: planning_ack+
dfediuck: devel_ack+
cshao: testing_ack+


Attachments (Terms of Use)
picture (deleted)
2016-07-12 07:10 UTC, Wei Wang
no flags Details
ks file (deleted)
2016-07-12 07:11 UTC, Wei Wang
no flags Details
log files (deleted)
2016-07-12 07:13 UTC, Wei Wang
no flags Details
ks2 (deleted)
2016-07-14 06:01 UTC, Wei Wang
no flags Details
ks for verify (deleted)
2016-07-21 07:23 UTC, Wei Wang
no flags Details


Links
System ID Priority Status Summary Last Updated
oVirt gerrit 60602 master MERGED dashboard: ensure superuser to check node status 2016-07-12 15:48:46 UTC
oVirt gerrit 60609 ovirt-4.0 MERGED dashboard: ensure superuser to check node status 2016-07-12 15:48:59 UTC

Description Wei Wang 2016-07-12 07:10:23 UTC
Created attachment 1178755 [details]
picture

Description of problem:
Node status info load fail when installing with ks file.

Version-Release number of selected component (if applicable):
redhat-virtualization-host-4.0-20160708.0.x86_64
imgbased-0.7.2-0.1.el7ev.noarch
cockpit-0.108-1.el7.x86_64

How reproducible:
100%

Steps to Reproduce:
1. Install redhat-virtualization-host-4.0-20160708.0.x86_64 with kickstart file in attachment
2. Login cockpit website hostIP:9090 with root account
3. Enter Virtualization panel.
4. Select Dashboard page
5. Check the "Node Status" info

Actual results:
Node status info load fail when installing with ks file.

Expected results:
Node status info should be loaded on Dashboard page.

Additional info:
The bug cannot be reproduced when installing with ISO file

Comment 1 Wei Wang 2016-07-12 07:11:04 UTC
Created attachment 1178757 [details]
ks file

Comment 2 Wei Wang 2016-07-12 07:13:32 UTC
Created attachment 1178760 [details]
log files

Comment 3 Ying Cui 2016-07-12 12:00:59 UTC
Additional info for bug description.
I can reproduce this issue when RHVH installed by ISO using default ks.cfg, and I login the cockpit UI by non-root user.

Comment 4 Ryan Barry 2016-07-12 13:57:56 UTC
This appears to be because "grubby" cannot without superuser permissions, and "nodectl info" calls it (through imgbased)

"nodectl check" also doesn't appear to run reliably if it's not root.

Comment 6 Ryan Barry 2016-07-12 16:08:02 UTC
(In reply to weiwang from comment #0)
> Created attachment 1178755 [details]
> picture
> 
> Description of problem:
> Node status info load fail when installing with ks file.
> 
> Version-Release number of selected component (if applicable):
> redhat-virtualization-host-4.0-20160708.0.x86_64
> imgbased-0.7.2-0.1.el7ev.noarch
> cockpit-0.108-1.el7.x86_64
> 
> How reproducible:
> 100%
> 
> Steps to Reproduce:
> 1. Install redhat-virtualization-host-4.0-20160708.0.x86_64 with kickstart
> file in attachment
> 2. Login cockpit website hostIP:9090 with root account
> 3. Enter Virtualization panel.
> 4. Select Dashboard page
> 5. Check the "Node Status" info
> 
> Actual results:
> Node status info load fail when installing with ks file.
> 
> Expected results:
> Node status info should be loaded on Dashboard page.
> 
> Additional info:
> The bug cannot be reproduced when installing with ISO file

I cannot reproduce this. The patch is for comment#3

Is it intentional that your kickstart file has no:

%post
imgbase layout --init
%end

That is a required stanza.

We could do better about this as part of cockpit, but if that is the reported bug, please consider it part of bz#1346835, where this element will be disabled.

Comment 7 Red Hat Bugzilla Rules Engine 2016-07-13 08:15:36 UTC
Target release should be placed once a package build is known to fix a issue. Since this bug is not modified, the target version has been reset. Please use target milestone to plan a fix for a oVirt release.

Comment 8 Ying Cui 2016-07-13 11:33:56 UTC
Wei, see comment 6, could you update your ks file with %post session to try again? and do you reproduce this issue as 100% or sometimes?

Comment 9 Wei Wang 2016-07-14 06:00:42 UTC
The issue can be reproduced 100%. No matter ks file with
%post
imgbase layout --init
%end
or not.
The other ks file in attachment also lead the issue occur, which has the %post session.

Comment 10 Wei Wang 2016-07-14 06:01:41 UTC
Created attachment 1179599 [details]
ks2

Comment 11 Ryan Barry 2016-07-14 06:09:15 UTC
imgbase --experimental volume ...

Is not longer necessary.

I'll check this with your kickstart in the morning.

Can you please log in over ssh and check the following?

nodectl check --machine-readable
nodectl info --machine-readable

Comment 12 Wei Wang 2016-07-14 07:50:23 UTC
Ryan,
The two commands show below messages:
 
[root@dhcp-11-209 ~]# nodectl check --machine-readable
{"status": "bad", "thin_storage": {"status": "ok", "checking_available_space_in_thinpool": {"status": "ok"}, "checking_thinpool_auto-extend": {"status": "ok"}}, "basic_storage": {"status": "ok", "initialized_thin_pool": {"status": "ok"}, "initialized_lvs": {"status": "ok"}, "initialized_vg": {"status": "ok"}}, "vdsmd": {"status": "bad"}, "mount_points": {"status": "failed - this can happen if the installation was performed incorrectly", "exception_in_'<function_check_discard_at_0x254db18>'": {"status": " '/var'"}, "discard_is_used": {"status": "error"}, "separate_var": {"status": "ok"}}}


[root@dhcp-11-209 ~]# nodectl info --machine-readable
Traceback (most recent call last):
  File "/usr/lib64/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib64/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/usr/lib/python2.7/site-packages/nodectl/__main__.py", line 42, in <module>
    CliApplication()
  File "/usr/lib/python2.7/site-packages/nodectl/__init__.py", line 171, in CliApplication
    return cmdmap.command(args)
  File "/usr/lib/python2.7/site-packages/nodectl/__init__.py", line 105, in command
    return self.commands[command](**kwargs)
  File "/usr/lib/python2.7/site-packages/nodectl/__init__.py", line 71, in info
    Info(self.imgbased, self.machine).write()
  File "/usr/lib/python2.7/site-packages/nodectl/info.py", line 45, in __init__
    self._fetch_information()
  File "/usr/lib/python2.7/site-packages/nodectl/info.py", line 48, in _fetch_information
    self._get_bootloader_info()
  File "/usr/lib/python2.7/site-packages/nodectl/info.py", line 55, in _get_bootloader_info
    bootinfo["default"] = b.get_default()
  File "/usr/lib/python2.7/site-packages/imgbased/bootloader.py", line 280, in get_default
    return self.bootloader.get_default()
  File "/usr/lib/python2.7/site-packages/imgbased/bootloader.py", line 234, in get_default
    entry = [e for e in entries if entries[e].kernel == kernel][0]
IndexError: list index out of range

Comment 13 Ryan Barry 2016-07-14 13:40:04 UTC
(In reply to weiwang from comment #10)
> Created attachment 1179599 [details]
> ks2

This should fail.

The following will cause an exception in %post if the RHV stage2 is used, because the RHV autopart classes will already have created /var

imgbase --experimental volume --create /var 4G

Again we have a RHEL stage2 being used:

17:02:53,903 INFO kernel:[    0.000000] Command line: initrd=/images/rhevh-ngn-rhel72/initrd.img ksdevice=bootif lang=  kssendmac inst.stage2=http://download.eng.pek2.redhat.com/released/RHEL-7/7.2/Server/x86_64/os inst.ks=http://10.66.10.22:8090/rhevh/ngn/latest/ks/wangwei.ks  BOOTIF=01-00-23-ae-8f-f3-ea BOOT_IMAGE=/images/rhevh-ngn-rhel72/vmlinuz

I tested this morning using a variety of kickstarts, and I was not able to reproduce on any of them, including the one from comment#10 (once "imgbase --experimental volume --create /var 4G" was removed, since this will cause a failure in %post with the RHV installer.

This bug (along with some others) seem to be exclusively reproducible when using a RHEL stage2.

Please try to reproduce this test using the RHV ISO and a kickstart. Or, if PXE booting (since this install was also PXE booted), please ensure that inst.stage2 points to rootfs.img extracted from the RHV ISO, *not* a RHEL stage2.

inst.stage2 expects a directory structure which either indicates installsource in .treeinfo. The simplest way to do this is to loopback mount the RHV ISO and rsync it (only /.treeinfo and /LiveOS/rootfs.img need to be copied even in Fedora 25, and EL7 only requires /LiveOS/rootfs.img).

For example (at the root of my webserver:

[root@legato html]# tree stage2/
stage2/
└── LiveOS
    └── squashfs.img


---------------------------------------------------------------------------

I'll file a bug to track these issues, because it's something to solve in 4.1 (hopefully by having the RHV installclass in Anaconda).

However, I'd ask for a test system, because I can't reproduce it, and there's potentially a bug in imgbased.

Comment 14 Ryan Barry 2016-07-14 15:28:06 UTC
An additional problem seems to be:

(from syslog) Linux version 3.10.0-327.el7.x86_64

This is not the kernel version from RHV-H. Anaconda will also fail if the kernel version from stage2 doesn't match what's booted, since modules cannot be loaded.

I grabbed the base RHEL 7.2 kernel, PXE booted from it, used the same stage2 as syslog, and used the kickstart in comment#10.

The installation failed in %post, because there was no /var (which imbased expects).

The test environment should look like:

-------------------------------------------------------
kernel and initrd from the RHV ISO
stage2 from the RHV ISO

kickstart should not include:
imgbase --experimental volume --create /var 4G

-------------------------------------------------------

Comment 15 Wei Wang 2016-07-21 02:00:10 UTC
Hi Rbarry,
Yes, you are right.Our PXE booted test environment need to update, so I will retest this issue after updating.

Comment 16 Wei Wang 2016-07-21 07:22:50 UTC
Test Version
redhat-virtualization-host-4.0-20160714.3.x86_64
imgbased-0.7.2-0.1.el7ev.noarch

Test Steps:
1. Install redhat-virtualization-host-4.0-20160714.3.x86_64 with kickstart file in attachment
2. Login cockpit website hostIP:9090 with root account
3. Enter Virtualization panel.
4. Select Dashboard page
5. Check the "Node Status" info

Result:
Node status info is loaded on Dashboard page successfully.

The bug cannot be reproduced, so change the status to VERIFIED

Comment 17 Wei Wang 2016-07-21 07:23:24 UTC
Created attachment 1182338 [details]
ks for verify


Note You need to log in before you can comment on or make changes to this bug.