Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1693515 - boot failure after F29 to F30 dnf system upgrade, arg.c:362:missing mandatory option for `users`.
Summary: boot failure after F29 to F30 dnf system upgrade, arg.c:362:missing mandatory...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: grub2
Version: 30
Hardware: ppc64le
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Peter Jones
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-03-28 03:43 UTC by Chris Murphy
Modified: 2019-04-01 00:01 UTC (History)
3 users (show)

Fixed In Version: grub2-2.02-75.fc30
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-04-01 00:01:23 UTC


Attachments (Terms of Use)
photo1 (deleted)
2019-03-28 03:45 UTC, Chris Murphy
no flags Details
photo2 (deleted)
2019-03-28 03:52 UTC, Chris Murphy
no flags Details
photo4 (deleted)
2019-03-28 03:58 UTC, Chris Murphy
no flags Details
photo3 (deleted)
2019-03-28 04:03 UTC, Chris Murphy
no flags Details
f30 grub.cfg (deleted)
2019-03-28 04:07 UTC, Chris Murphy
no flags Details

Description Chris Murphy 2019-03-28 03:43:38 UTC
Description of problem:

Following an upgrade from Fedora 29 to Fedora 30 using `dnf system-upgrade`, boot fails.

Version-Release number of selected component (if applicable):
grub2-ppc64le-2.02-72.fc30.ppc64le

How reproducible:
Always


Steps to Reproduce:
1. Fedora 29 minimal install, qemu-ppc/kvm
2. use `dnf system-upgrade` per https://fedoraproject.org/wiki/DNF_system_upgrade
3. system reboots, and upgrades as expected, reboots again and...

Actual results:

Minimal BASH-like line editing is support...

grub> _


(i.e. grub prompt, no boot menu)


Expected results:

Either a grub menu (it is first boot after an upgrade), or it should boot successfully if hidden grub menu feature applies here.

Additional info:


Very briefly, between the prepboot and GRUB menu appears are three lines (photo1) which I'm able to replicate by trying to manually load the grub.cfg using configfile (photo2).

lsmod (photo3)
set (photo4)

It looks like GRUB does find the grub.cfg but then fails to read or parse it for some reason.

Comment 1 Chris Murphy 2019-03-28 03:45:06 UTC
Created attachment 1548812 [details]
photo1

Comment 2 Chris Murphy 2019-03-28 03:52:24 UTC
Created attachment 1548813 [details]
photo2

Enable debug, then use configfile to try to manually load the grub.cfg.

Comment 3 Chris Murphy 2019-03-28 03:58:21 UTC
Created attachment 1548814 [details]
photo4

grub environment variables

Comment 4 Chris Murphy 2019-03-28 04:03:03 UTC
Created attachment 1548815 [details]
photo3

lsmod

Comment 5 Chris Murphy 2019-03-28 04:07:38 UTC
Created attachment 1548816 [details]
f30 grub.cfg

This is the Fedora 30 /boot/grub2/grub.cfg

Comment 6 Chris Murphy 2019-03-28 04:31:31 UTC
- manually booting with `linux` `initrd` and `boot` commands does work, so at least that much is working and seems isolated to the grub.cfg
- running `grub2-mkconfig -o /boot/grub2/grub.cfg` doesn't fix the problem
- running `grub2-install --no-nvram --no-floppy /dev/vda1` does fix the problem, but I don't know why: lsmod shows the same modules loaded, and set shows the same environment variables (although timeout=5 is missing now for some reason, not sure if it's the grub reinstall or recreating the grub.cfg, anyway, unrelated)

Comment 7 Javier Martinez Canillas 2019-03-28 13:38:19 UTC
Hello Chris,

Thanks a lot for the report.

(In reply to Chris Murphy from comment #6)
> - manually booting with `linux` `initrd` and `boot` commands does work, so
> at least that much is working and seems isolated to the grub.cfg

You are correct, configfile /grub2/grub.cfg.rpmsave also works

> - running `grub2-mkconfig -o /boot/grub2/grub.cfg` doesn't fix the problem
> - running `grub2-install --no-nvram --no-floppy /dev/vda1` does fix the
> problem, but I don't know why: lsmod shows the same modules loaded, and set
> shows the same environment variables (although timeout=5 is missing now for
> some reason, not sure if it's the grub reinstall or recreating the grub.cfg,
> anyway, unrelated)

The problem is the --users $grub_users option in the menu entries. The --users option is used to restrict the access to specific menu entries only to a set of users. But the option requires an argument to either be a constant or a variable that has been set. So for example the following:
    
      menuentry "May be run by superusers or users in $users" --users $users {
                linux /vmlinuz
      }

Would fail if $users is not defined and GRUB would discard the menu entry (which happens in this bug and that's why you see the error messages and there are no entries in the menu). Instead GRUB should allow the --users option to have an optional argument and ignore the option if the argument was not set.

That was fixed in GRUB and that's why it works on a fresh F30 install, but it's not fixed in the F29 GRUB. And since we don't update the GRUB core on a package install, it fails on a system upgrade.

But looking at the F29 grub2.cfg, the menu entries don't have the --users option anyways and just the --unrestricted one. So we will just do the same in F30 to prevent this issue and allow it to work with older GRUB installations that don't have the fix for the --users option.


NOTE: You might be wondering why we have the menu entries in the GRUB config file and are not using the blscfg module for ppc64le? The answer is that we only have GRUB support for ppc64le on a VM (PowerVM) which uses the Open Firmware interface. On ppc64le bare metal (PowerNV) the OPAL firmware interface is used, and the bootloader ins't GRUB in that case but a Linux kernel with a built-in initramfs that contains a user-space bootloader (Petitboot).

We added BLS support to Petitboot, but that's fairly recent and found that many machines are still running old versions of the firmware that don't have the BLS support. Since the OPAL firmware is not controlled by us and updates are provided by the hardware vendor, for now what we do is to generate a grub2.cfg section by using the BLS snippets in /boot/loader/entries (similar to what ostree still does). That way it works with any OPAL firmware version.

Comment 8 Javier Martinez Canillas 2019-03-28 16:33:08 UTC
Please test installing the following grub2-2.02-75.fc30 build [0] and then re-generate your grub2.cfg with `grub2-mkconfig -o /boot/grub2/grub.cfg`.

[0]: build https://koji.fedoraproject.org/koji/taskinfo?taskID=33804676

Comment 9 Chris Murphy 2019-03-28 17:03:40 UTC
(In reply to Javier Martinez Canillas from comment #8)
> Please test installing the following grub2-2.02-75.fc30 build [0] and then
> re-generate your grub2.cfg with `grub2-mkconfig -o /boot/grub2/grub.cfg`.

After doing this, the VM boots successfully.

Comment 10 Fedora Update System 2019-03-29 16:45:10 UTC
grub2-2.02-75.fc30 systemd-241-4.gitcbf14c9.fc30 has been submitted as an update to Fedora 30. https://bodhi.fedoraproject.org/updates/FEDORA-2019-616045ca76

Comment 11 Fedora Update System 2019-03-29 20:32:37 UTC
grub2-2.02-75.fc30, systemd-241-4.gitcbf14c9.fc30 has been pushed to the Fedora 30 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-616045ca76

Comment 12 Fedora Update System 2019-04-01 00:01:23 UTC
grub2-2.02-75.fc30, systemd-241-4.gitcbf14c9.fc30 has been pushed to the Fedora 30 stable repository. If problems still persist, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.