Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1696449 - /usr/bin/run-parts has a fatal programming error which can cause cron to hang perpetually
Summary: /usr/bin/run-parts has a fatal programming error which can cause cron to hang...
Status: NEW
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: crontabs
Version: 7.4
Hardware: All
OS: Linux
Target Milestone: rc
: ---
Assignee: Marcel Plch
QA Contact: qe-baseos-daemons
Depends On:
TreeView+ depends on / blocked
Reported: 2019-04-04 21:01 UTC by Kevin Buchs
Modified: 2019-04-15 10:35 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Last Closed:
Target Upstream Version:

Attachments (Terms of Use)

Description Kevin Buchs 2019-04-04 21:01:26 UTC
Description of problem:

/usr/bin/run-parts has a fatal programming error which can cause cron to hang perpetually. This run-parts script is part of the crontabs package. It is used to execute the cron.hourly, cron.daily, etc. actions. 

Version-Release number of selected component (if applicable):

I have seen this reproduced on RHEL 6 and 7, CentOS and Amazon Linux too.

How reproducible:

Just make use of cron.hourly, cron.daily, etc. and you might hit this problem. It is not reproducible at will because it is a race condition.

Actual results:

Cron hangs. Completes no further work.

Expected results:

Cron runs normally.

Additional info:

run-parts is a shell script which contains this pipeline at its heart (where $i is the command from, e.g. "/etc/cron.daily/logrotate", to be executed):
$i 2>&1 | awk -v "progname=$i" \
                              'progname {
                                   print progname ":\n"
                               { print; }'

That awk script is horrendous example of programming. It defines a variable *progname* on the command line, e.g. logrotate. The script also defines a function of the same name. In that AWK function, the AWK function is deleted. That is a race condition. 

The purpose of this code is to echo the progname initially once and then from then on just echo its input. To keep it as an awk script, it should be the following:
$i 2>&1 | awk -v "progname=$i" \
                              'BEGIN { print progname ":\n" }
                               { print; }'

However, there is no need to add the additional burden of awk having to echo each line. This would work just fine:
echo -e "$i:\n"

I posted this in regard to Amazon Linux in the appropriate AWS forum, but got no response. I know they should have escalated it upstream.

Here is what the processes look like when the race condition is hit:

ps axwu|grep cron
root 1793 0.0 0.0 116912 1188 ? Ss 2018 3:21 crond
root 12003 0.0 0.0 103328 860 pts/2 S+ 13:33 0:00 grep cron
root 14361 0.0 0.0 19052 948 ? Ss 2018 0:00 /usr/sbin/anacron -s
root 16875 0.0 0.0 106112 1268 ? SN 2018 0:00 /bin/bash /usr/bin/run-parts /etc/cron.daily
root 16887 0.0 0.0 105972 948 ? SN 2018 0:00 awk -v progname=/etc/cron.daily/logrotate progname {????? print progname ":\n"????? progname="";???? }???? { print; }

The awk process never finishes (it is trying to find the function it was trying to run, I guess it gets lost when the function is deleted. Today is April 2, 2019 and this has been hung since Dec 21, 2018. 

The process of running awk seems to have gotten nowhere:

# ps -p $pid H -www -o pid,cputime,state,lstart,time,etime,cmd
16887 00:00:00 S Fri Dec 21 14:13:01 2018 00:00:00 101-22:45:16 awk -v progname=/etc/cron.daily/logrotate progname {????? print progname ":\n"????? progname="";???? }???? { print; }

Previously, I think I joined the awk process with debugger and found out awk was not executing anything.

Comment 2 Petr Viktorin 2019-04-09 11:46:26 UTC
This is the right place to report the bug. Thank you! And thank you again for the detailed report!

Marcel is on vacation now, and the issue doesn't look time-critical, so it will take some time before we get to it. But it's in the queue.
Please let us know if you think it should be prioritized.

Comment 3 Kevin Buchs 2019-04-09 12:35:17 UTC

Thanks for the response. In terms of priority, since this has been a long standing issue, even an immediate fix won't result in a change seen for a while, especially in RHEL derivatives. The issue results in cron hanging, which I believe most folks would not expect to happen. The criticality certainly depends on the user, but I can imagine some serious impacts. Now, I usually factor in the level of effort required in my thinking on prioritization. Something easy to fix, as this is (cause I gave you the fix), may be done before other tasks because it is fast to do. Those are my thoughts. Waiting for Marcel to return is likely to be a good choice.

Note You need to log in before you can comment on or make changes to this bug.