Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1691511 - Repeated systemd-run --scope -- mount -t tmpfs tmpfs /<path> under directories bind-mounted to themselves result in E2BIG failures
Summary: Repeated systemd-run --scope -- mount -t tmpfs tmpfs /<path> under directorie...
Status: POST
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: systemd
Version: 7.6
Hardware: Unspecified
OS: Unspecified
Target Milestone: rc
: ---
Assignee: systemd-maint
QA Contact: Frantisek Sumsal
Depends On:
TreeView+ depends on / blocked
Reported: 2019-03-21 18:58 UTC by Kyle Walker
Modified: 2019-04-02 09:10 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Last Closed:
Target Upstream Version:

Attachments (Terms of Use)

Description Kyle Walker 2019-03-21 18:58:52 UTC
Description of problem:
 With openshift workloads, commands such as the following are issued repeatedly:

    systemd-run --description='Kubernetes transient mount for /var/lib/origin/openshift.local.volumes/pods/<val>/volumes/<val>' --scope -- mount -t tmpfs tmpfs /var/lib/origin/openshift.local.volumes/pods/<val>/volumes/<val>

When the mount point is within a bind-mounted directory, the behaviour results in a large number of "loaded inactive dead" mount points. Eventually the commands will fail with the following error message visible in the system logs.

    systemd: Failed to set up mount unit: Argument list too long

This issue was reported upstream in the following:

Which pushed the investigation to an upstream systemd issue documented in:

Version-Release number of selected component (if applicable):


How reproducible:

    Difficult, though the underlying suspected problem is fairly simple

Steps to Reproduce:
1. Create a directory and bind mount it to itself

    # mkdir /mnt/test && mount --bind /mnt/test /mnt/test

2. Mount additional directories under the above

    # mkdir /mnt/test/subdir1 && mount -t tmpfs tmpfs /mnt/test/subdir1

3. Verify the state of the underlying subdir

    # systemctl list-units --type mount --all | grep test

Actual results:

    # systemctl list-units --type mount --all | grep test
    mnt-test-subdir1.mount        loaded inactive dead    /mnt/test/subdir1
    mnt-test.mount                loaded active   mounted /mnt/test

Expected results:

    # systemctl list-units --type mount --all | grep test
    mnt-test-subdir1.mount        loaded active   mounted /mnt/test/subdir1
    mnt-test.mount                loaded active   mounted /mnt/test    

Additional info:

    The cause of the "systemd: Failed to set up mount unit: Argument list too long" error message is suspected to be continued growth of "inactive dead" mount units that cannot be removed and eventual failure in the following codepath:

    int unit_add_name(Unit *u, const char *text) {
            _cleanup_free_ char *s = NULL, *i = NULL;
            UnitType t;
            int r;
            if (hashmap_size(u->manager->units) >= MANAGER_MAX_NAMES)
                    return -E2BIG;

Comment 2 Kyle Walker 2019-03-21 19:25:04 UTC
Opened a downstream PR with a backport of the following commit:

    core: Fix edge case when processing /proc/self/mountinfo

    Currently, if there are two /proc/self/mountinfo entries with the same
    mount point path, the mount setup flags computed for the second of
    these two entries will overwrite the mount setup flags computed for
    the first of these two entries. This is the root cause of issue systemd#7798.
    This patch changes mount_setup_existing_unit to prevent the
    just_mounted mount setup flag from being overwritten if it is set to
    true. This will allow all mount units created from /proc/self/mountinfo
    entries to be initialized properly.


Comment 5 Lukáš Nykrýn 2019-04-02 09:10:30 UTC
fix merged to staging branch -> -> post

Note You need to log in before you can comment on or make changes to this bug.