Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 160794 - word splitting after command/variable substitution inside double quotes?
Summary: word splitting after command/variable substitution inside double quotes?
Alias: None
Product: Fedora
Classification: Fedora
Component: bash
Version: 4
Hardware: All
OS: Linux
Target Milestone: ---
Assignee: Tim Waugh
QA Contact: Ben Levenson
Depends On:
TreeView+ depends on / blocked
Reported: 2005-06-17 11:04 UTC by Jeroen Vermeulen
Modified: 2007-11-30 22:11 UTC (History)
0 users

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Last Closed: 2005-06-20 03:03:28 UTC

Attachments (Terms of Use)

Description Jeroen Vermeulen 2005-06-17 11:04:12 UTC
From Bugzilla Helper:
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.8) Gecko/20050513 Galeon/1.3.20 (Debian package 1.3.20-1)

Description of problem:
This report is based entirely on bug reports and email-debugging by libpqxx users; I don't have Fedora handy myself.  I haven't even been able to get confirmation whether the problem goes away with a different shell.  :/

I'm reporting at this point because a testable hypothesis has emerged--namely that the FC4 version of bash 3.00.16 may be applying word splitting where it shouldn't, after command/variable substitution between double quotes ("").  This construct seems to require one more level of quoting than it does on other systems.

The configure script for libpqxx (generated by the autotools) works fine for a wide range of Unix-like systems including GNU/Linux variants on several architectures, and even Windows using MinGW.  A piece of embedded shell script fails on FC4, but not on e.g. FC3 or Debian.  I could not reproduce the problem on a Debian system with the same upstream automake, autoconf, bash, and gcc versions.  The problem was first reported based on an FC4 prerelease, and continues to exist with the stable version.  It does not appear to be influenced by what system the configure script was generated on, though this has not been thoroughly tested.

Two symptoms appear within the same command line.  The command line consists of an outer "grep -F" (search for any instance of newline-separated list of literals) where the first argument, the search string, is generated from a backticked "inner grep" command.  The command is therefore of the form

  grep -F "`grep "$x" "y"`" z

Symptom 1: In cases where the "inner grep" returns no results, so the argument is empty, the command line was reported to hang (one user reported grep running with full CPU utilization, but this may have gone away in the stable FC4).  This does not happen if the search string is merely an empty argument, but may be plausible if the argument is dropped during word splitting by the shell.  According to the bash manpage, the quotes should have prevented this:

Word Splitting
  [...] Explicit null arguments ("" or '') are retained.  Unquoted implicit null arguments, resulting from the expansion of parameters that have no values, are removed.  If a parameter with no value is expanded within double quotes, a null argument results and is retained.

Presumably this applies to null arguments resulting from command substitution as well; the essential part is that the argument is empty but double-quoted.  We also tried some combinations with intermediate variables but mostly ran into the same problems we encountered otherwise.

Symptom 2: Newlines in the inner grep's result are replaced with regular space, so the first argument to the outer grep is a single line of keywords rather than the newline-separated list that grep -F expects.  As a result, the grep -F never finds any matches.

Again, according to the bash manpage the double quotes should have prevented this normalisation of whitespace:

Command Substitution
  [...] Embedded newlines are not deleted, but they may be removed during word splitting. [...]  If the substitution appears within double quotes, word splitting and pathname expansion are not performed on the results.

Adding an additional level of single quotes apparently fixes the problem:

  grep -F "'`grep "'$x'" "y"`'" z

But this breaks when I try it on a non-FC4 system, because the single quotes are included in the arguments themselves.

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:
1. Download and extract any recent libpqxx source tarball from, or checkout a working copy of the CVS repository.
2. cd libpqxx-*.[0-9]
3. ./configure

Actual Results:  In previous libpqxx releases, the script would hang near the end, where files "include/pqxx/config-*-*.h" are generated.  In the most current CVS versions, these files will not contain results and the configure script will suffix the output messages about such files with "no items--deleting" (and the legitimate cases will say "no items--skipping").

Expected Results:  Two of the messages should say "no items", and the rest should say "ok".

Additional info:

Comment 1 Tim Waugh 2005-06-17 12:16:14 UTC
I can't trawl through hairy configure scripts to look for this -- please supply
a proper minimal test case.

If you are in doubt about how quoting and word-splitting works, read the POSIX

I can tell you that your example, grep -F "`grep "$x" "y"`" z, acts like this:

$ x=x
$ touch y
$ set grep -F "`grep "$x" "y"`" z
$ for a in "$@"; do echo ."$a".; done

.. in other words, there are four words, one of which is empty.

As for newlines being replaced by spaces:

$ set grep -F "`echo one; echo two`" z
$ for a in "$@"; do echo ."$a".; done

So I think you'll need to provide a proper test case.

Comment 2 Jeroen Vermeulen 2005-06-20 03:01:46 UTC
Apparently it's not bash after all.  Back to the drawing board.  Sorry to have
bothered you for nothing.  :-|

Note You need to log in before you can comment on or make changes to this bug.