Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 161985 - O_DIRECT on RHEL v4 may not return correct number of bytes when concurrent I/O
Summary: O_DIRECT on RHEL v4 may not return correct number of bytes when concurrent I/O
Status: CLOSED DUPLICATE of bug 166589
Alias: None
Product: Red Hat Enterprise Linux 4
Classification: Red Hat
Component: kernel
Version: 4.0
Hardware: i386
OS: Linux
Target Milestone: ---
: ---
Assignee: Peter Staubach
QA Contact: Brian Brock
Depends On:
TreeView+ depends on / blocked
Reported: 2005-06-28 20:44 UTC by Richard Fields
Modified: 2007-11-30 22:07 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Last Closed: 2006-03-09 14:41:12 UTC
Target Upstream Version:

Attachments (Terms of Use)

Description Richard Fields 2005-06-28 20:44:06 UTC
From Bugzilla Helper:
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7.6) Gecko/20050225 Firefox/1.0.1

Description of problem:
We discovered a major flaw with O_DIRECT on RHEL v4 on EXT3 that did not exist on previous versions of RHEL.

We have reproduced this on several machines, and can resolve the problem by switching back to RHEL v3 from RHEL v4.  The rest of the environment remains the same, and the problem only occurs under RHEL v4.

The problem is that when there is concurrent I/O on a file that was opened with O_DIRECT, then a pread() of n bytes by another process on the same file may only return part of the requested n bytes.

Additionally, a pread() call may also return an error (errno is set to 5).

We have temporarily hacked around this bug, by getting the second process to retry pread calls that do not return the correct number of bytes.  But, this is obviously a temporary solution.

Thank you,
-Richard Fields

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:
1. Open a file with O_DIRECT
2. Open the same file with a second process.
3. Do a pread() of n bytes in the second process while the first process continues to read and write to the file.

Actual Results:  The second process will sometimes read the correct number of bytes, sometimes read an incorrect number of bytes, and sometimes return an error message.

Additional info:

We actually discovered this running the MySQL database server with O_DIRECT, and using their Hot Backup software to copy the files live.  We have tested this extensively with the MySQL support team, and have determined that the problem only happens when the MySQL server starts with O_DIRECT on RHEL v4.

Comment 1 Peter Staubach 2005-07-18 18:11:43 UTC
I don't suppose that anyone has written a program which reproduces the problem?
That would help me to do the analysis plus to verify the changes when they
become available.

Comment 2 Peter Staubach 2005-08-30 14:31:14 UTC
I really do need some sort of testcase which shows the problem.  Without it,
I am not sure what I can do.  Would you let me know whether there is something
that can be used?

It seems a little odd to be accessing something like a database using
O_DIRECT, but backing it up while not using O_DIRECT.  O_DIRECT avoids
the use of the page cache, while regular i/o uses the page cache, ie.
a copy of the data from the disk, but not necessarily the most recent
copy, due to i/o occuring which does not update the page cache.  Shouldn't
the two applications be using the same kind of i/o and perhaps also be
using something to synchronize between the two?

Comment 4 Richard Fields 2005-08-31 15:57:48 UTC

We are running the MySQL database server with the O_DIRECT option to enhance
performance.  The server runs fine without a problem.

We use the InnoDB Hot Backup utility, so that we can make live backups
consistently without having to take the database server down.  It does not use

Mixed O_DIRECT and normal file I/O seems to work ok in earlier Red Hat distros
than RHEL 4, and similar types of I/O work ok in Windows NT/2000/XP.

Every single machine that we upgrade from RHEL 3 (or earlier versions) has the
problem when we upgrade to RHEL 4, so it seems to only be a problem there.

MySQL says that they could optionally make ibbackup (the Hot Backup program) use
O_DIRECT to read the files.  But, they have to test if THAT does work.  Using
synchronization between mysqld and ibbackup is difficult.

Red Hat should document in the manual if a file cannot be accessed with normal
file I/O when it has been opened with O_DIRECT by another process, if this
behavior is intentional.  There does not seem to be any indication of this in
the man pages currently.

And again, this is only a problem on RHEL 4.


Comment 5 Peter Staubach 2005-11-18 14:57:32 UTC
No, there is no reason that a particular file can not be accessed via
buffered and direct i/o at the sametime.  As Stephen pointed out, the
contents returned are undefined when one process is writing using direct
i/o to the same region that another process is reading using buffered
i/o, but that, of course, is the same situation which occurs even if
the two processes are both using direct i/o.

I am looking into this race.

Note You need to log in before you can comment on or make changes to this bug.