CentOS execution of grub2-mkconfig causes disk to be read-only

I recently encountered a failure where a disk partition was found to become read-only during a cluster software upgrade, causing the storage application to recognize the disk as unavailable. Investigation revealed that the cluster upgrade process regenerates the GRUB configuration file for each node, which causes the disk partition to become read-only during the execution of grub2-mkconfig. Document the flow of the grub2-mkconfig command execution until the real disk read-only command is issued.

Investigate

`grub2-mkconfig`

When executing the grub2-mkconfig command, GRUB will call os-prober (/etc/grub.d/30_os-prober) for scanning other operating systems for subsequent configuration if the configuration GRUB_DISABLE_OS_PROBER=true is not specified.

os-prober

os-prober is used to detect the presence of operating systems on other disks. It is usually provided by individual distributions and is available in source at https://salsa.debian.org/installer-team/os-prober.

The os-prober included by default with CentOS is not the same as the source version and contains some additional configuration, which can be viewed via the RPM changelog at.

Name        : os-prober
Version     : 1.58
Release     : 9.el7
Architecture: x86_64
Install Date: Wed 18 Aug 2021 03:44:43 PM CST
Group       : System Environment/Base
Size        : 97946
License     : GPLv2+ and GPL+
Signature   : RSA/SHA256, Mon 21 Nov 2016 03:50:19 AM CST, Key ID 24c6a8a7f4a80eb5
Source RPM  : os-prober-1.58-9.el7.src.rpm
Build Date  : Sun 06 Nov 2016 11:30:02 AM CST
Build Host  : worker1.bsys.centos.org
Relocations : (not relocatable)
Packager    : CentOS BuildSystem <http://bugs.centos.org>
Vendor      : CentOS
URL         : http://kitenet.net/~joey/code/os-prober/
Summary     : Probes disks on the system for installed operating systems
Description :
This package detects other OSes available on a system and outputs the results
in a generic machine-readable format. Support for new OSes and Linux
distributions can be added easily.
* Wed Sep 14 2016 rmarshall@redhat.com - 1.58-9
- Fix regular expression that missed a corner case when detecting
extended dos partitions.
Resolves: rhbz#1322957
        
* Wed Jul 06 2016 rmarshall@redhat.com - 1.58-8
- Resolve some coverity concerns with how the previous patch detected
whether or not a partition was a device mapper device.
Related: rhbz#1300262
        
* Tue Jul 05 2016 rmarshall@redhat.com - 1.58-7
- Do not resolve device mapper links when generating stanzas
for bootloader.
Resolves: rhbz#1300262
        
* Wed May 04 2016 rmarshall@redhat.com - 1.58-6
- Adding handling to skip probing extended dos partitions
Resolves: rhbz#1322957
- Improve Windows detection for dual boot and support Windows 10
Resolves: rhbz#1322956
- Suppress non-blocking dmraid sector size warning message
Resolves: rhbz#1198918
        
* Fri Jan 24 2014 Daniel Mach <dmach@redhat.com> - 1.58-5
- Mass rebuild 2014-01-24
        
* Mon Jan 20 2014 Peter Jones <pjones@redhat.com> - 1.58-4
- Add man pages.
Resolves: rhbz#948848

You can see exactly what has been changed by downloading the source RPM for os-prober and viewing the os-prober.spec file to see a list of additional patches.

Name:           os-prober
Version:        1.58
Release:        9%{?dist}
Summary:        Probes disks on the system for installed operating systems
            
Group:          System Environment/Base
# For more information about licensing, see copyright file.
License:        GPLv2+ and GPL+
URL:            http://kitenet.net/~joey/code/os-prober/
Source0:        http://ftp.de.debian.org/debian/pool/main/o/os-prober/%{name}_%{version}.tar.gz
# move newns binary outside of os-prober subdirectory, so that debuginfo
# can be automatically generated for it
Patch0001: 0001-Change-filepath-to-newns.patch
Patch0002: 0002-Don-t-count-dummy-mach_kernel-as-MacOS-X-811412.patch
Patch0003: 0003-Detect-OS-installed-to-mdraid-partition-752402.patch
Patch0004: 0004-Yaboot-allows-spaces-in-append-825041.patch
Patch0005: 0005-Detect-ld.so-after-usr-move-826754.patch
Patch0006: 0006-Use-shell-processing-instead-of-basename-875356.patch
Patch0007: 0007-Add-option-for-less-logging-893997.patch
Patch0008: 0008-Improve-btrfs-detection-support-888341.patch
Patch0009: 0009-Support-detection-on-btrfs-software-raid-906847.patch
Patch0010: 0010-Name-lvm-boot-partitions-by-fstab-entry-893472.patch
Patch0011: 0011-Set-correct-boot-partition-906886.patch
Patch0012: 0012-Factor-out-unnecessary-logger-calls-875356.patch
Patch0013: 0013-Issue-with-EFI-detection-in-logger-873207.patch
Patch0014: 0014-Man-pages-missing-948848.patch
Patch0015: 0015-Properly-handle-extended-dos-partitions-1322957.patch
Patch0016: 0016-Windows-detection-requires-binary-grep-1322956.patch
Patch0017: 0017-Add-Windows-10-detection-support-1322956.patch
Patch0018: 0018-Suppress-non-blocking-dmraid-error-info-1198918.patch
Patch0019: 0019-Do-not-resolve-device-mapper-symlinks-1300262.patch
Patch0020: 0020-Use-POSIX-shell-syntax-1300262.patch
Patch0021: 0021-Fix-extended-dos-partition-regex-1322957.patch

It’s a pain to look at each patch individually, so look directly at the os-prober-related files already installed on the OS

$rpm -ql os-prober
/usr/bin/linux-boot-prober
/usr/bin/os-prober
/usr/libexec/linux-boot-probes
/usr/libexec/linux-boot-probes/50mounted-tests
/usr/libexec/linux-boot-probes/mounted
/usr/libexec/linux-boot-probes/mounted/40grub
/usr/libexec/linux-boot-probes/mounted/40grub2
/usr/libexec/linux-boot-probes/mounted/50lilo
/usr/libexec/linux-boot-probes/mounted/90fallback
/usr/libexec/newns
/usr/libexec/os-probes
/usr/libexec/os-probes/50mounted-tests
/usr/libexec/os-probes/init
/usr/libexec/os-probes/init/10filesystems
/usr/libexec/os-probes/mounted
/usr/libexec/os-probes/mounted/05efi
/usr/libexec/os-probes/mounted/10freedos
/usr/libexec/os-probes/mounted/10qnx
/usr/libexec/os-probes/mounted/20macosx
/usr/libexec/os-probes/mounted/20microsoft
/usr/libexec/os-probes/mounted/30utility
/usr/libexec/os-probes/mounted/40lsb
/usr/libexec/os-probes/mounted/70hurd
/usr/libexec/os-probes/mounted/80minix
/usr/libexec/os-probes/mounted/83haiku
/usr/libexec/os-probes/mounted/90linux-distro
/usr/libexec/os-probes/mounted/90solaris
/usr/libexec/os-probes/mounted/efi
/usr/libexec/os-probes/mounted/efi/10elilo
/usr/libexec/os-probes/mounted/efi/20microsoft
/usr/share/doc/os-prober-1.58
/usr/share/doc/os-prober-1.58/README
/usr/share/doc/os-prober-1.58/TODO
/usr/share/doc/os-prober-1.58/changelog
/usr/share/doc/os-prober-1.58/copyright
/usr/share/man/man1/linux-boot-prober.1.gz
/usr/share/man/man1/os-prober.1.gz
/usr/share/os-prober
/usr/share/os-prober/common.sh
/var/lib/os-prober

grub2-mkconfig calls os-prober flow

The ultimate goal of grub2-mkconfig is to generate grub.cfg, which is executed sequentially by calling the configuration scripts under the /etc/grub.d/ path, where the /etc/grub.d/30_os-prober configuration script exists in CentOS.

The specific code to execute os-prober is as follows.

OSPROBED="`os-prober | tr ' ' '^' | paste -s -d ' '`"
if [ -z "${OSPROBED}" ] ; then
  # empty os-prober output, nothing doing
  exit 0
fi

CentOS os-prober execution flow

Create a new namespace, it is recommended to comment out this behavior during debugging os-prober
Ensure that all file system types are supported by executing /usr/libexec/os-prober/init/*
Get the information about the mounted partitions, swap partitions, and RAID device partitions on the node.
Get all partition information of the node.
- Skip if partition already exists in swap partition information or RAID device partition information
- Get the file system information as type by blkid -o value -S type $path, for example.
1 2

$blkid -o value -s TYPE /dev/vdb1 xfs_external_log
- If type == btrfs, determine if it is a btrfs volume, and filter if it is; if not, use /usr/libexec/os-probes/50mounted-tests to probe against the partition
- If the partition is not in the mounted partition information, probe using all methods under the /usr/libexec/os-probes/50mounted-tests path, traversing the execution
- If the partition is in the mounted partition information, probe all ways under the /usr/libexec/os-probes/50mounted-tests path and iterate through them
End of detection

CentOS `/usr/libexec/os-probes/50mounted-tests` Probing process

Parameter checksum to get the fs type property of the partition
- If type is undetected, empty, LVM, swap, ctypto, ntfs, then exit directly
If type == btrfs, then probe process for subvolume
If grub-mount and grub2-probe are present, determine if the partition is mounted by GRUB.
If none of the above branches are entered, make the partition read-only
- blockdev -setro $partition
- and set the signal handling function by trap, when EXIT,HUP,INT,QUIT,TERM signal is received, the partition will be reset to read and write.
Mount the partition to a temporary mount point according to the current type, and try probing under the /usr/libexec/os-probes/mounted/ path, and iterate through it.
End of detection

Summary

If there is a situation on your node where it is unmounted and the file system type is not null, then it is recommended to disable os-prober execution by setting the GRUB configuration: GRUB_DISABLE_OS_PROBER=true to prevent the disk partition from becoming read-only.

Table of Contents