Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 83325 - Updating from glibc-2.3.1-5 to glibc-2.3.1-38 broke qmail
Summary: Updating from glibc-2.3.1-5 to glibc-2.3.1-38 broke qmail
Keywords:
Status: CLOSED RAWHIDE
Alias: None
Product: Red Hat Raw Hide
Classification: Retired
Component: glibc
Version: 1.0
Hardware: i686
OS: Linux
medium
medium
Target Milestone: ---
Assignee: Jakub Jelinek
QA Contact: Brian Brock
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2003-02-02 17:43 UTC by Ville Herva
Modified: 2016-11-24 15:20 UTC (History)
1 user (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2003-02-18 10:42:30 UTC


Attachments (Terms of Use)
glibc-2.3.1-5 strace (deleted)
2003-02-02 17:44 UTC, Ville Herva
no flags Details
glibc-2.3.1-38 strace (deleted)
2003-02-02 17:45 UTC, Ville Herva
no flags Details
(Working) ltrace from glibc-2.3.1-5 (deleted)
2003-02-06 09:35 UTC, Ville Herva
no flags Details
(failing) ltrace from glibc-2.3.1-38 (deleted)
2003-02-06 09:35 UTC, Ville Herva
no flags Details
The relevant code path (deleted)
2003-02-06 11:22 UTC, Ville Herva
no flags Details

Description Ville Herva 2003-02-02 17:43:55 UTC
From Bugzilla Helper:User-Agent: Mozilla/4.78 (Linux 2.4.20-rc1-jam1 i686; U) Opera 6.11  [en] Description of problem:qmail no longer works after updating from glibc-2.3.1-5 to glibc-2.3.1-38.

qmail has not been touched at all, and
ldd /var/qmail/bin/qmail-smtpd says
        libc.so.6 => /lib/libc.so.6 (0x4002d000)
        /lib/ld-linux.so.2 => /lib/ld-linux.so.2 (0x40000000)
so I would be pretty ceratin glibc upgrade broke it.

Symptoms:
> /var/qmail/bin/qmail-smtpd
421 unable to read controls (#4.3.0)

it used to do this:
> /var/qmail/bin/qmail-smtpd
220 v.iki.fi ESMTP<waits for input>

Other qmail binaries are confused as well (give the same error).
See the attached traces - qmail gives up after not find an optional control/ file.

Version-Release number of selected component (if applicable):

How reproducible: Always
Steps to Reproduce:
1. execute any qmail binary
2.
3.   

Actual Results:  
> /var/qmail/bin/qmail-smtpd
421 unable to read controls (#4.3.0)

Expected Results:  
> /var/qmail/bin/qmail-smtpd
220 v.iki.fi ESMTP
<waits for input>

Additional info:
(working) strace with glibc-2.3.1-5
-----------------------------------------------------------------------------
execve("/var/qmail/bin/qmail-smtpd", ["/var/qmail/bin/qmail-smtpd"], [/* 48 vars */]) = 0
uname({sys="Linux", node="babbage", ...}) = 0
brk(0)                                  = 0x804f45c
old_mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x40015000
open("/etc/ld.so.preload", O_RDONLY)    = -1 ENOENT (No such file or directory)
open("/lib/libNoVersion.so.1", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/backup-versioned/2003-01-28T052001/lib/tls/i686/mmx/libc.so.6", O_RDONLY) = -1 ENOENT (No such file o
r directory)
stat64("/backup-versioned/2003-01-28T052001/lib/tls/i686/mmx", 0xbfffe610) = -1 ENOENT (No such file or dire
ctory)
open("/backup-versioned/2003-01-28T052001/lib/tls/i686/libc.so.6", O_RDONLY) = -1 ENOENT (No such file or di
rectory)
stat64("/backup-versioned/2003-01-28T052001/lib/tls/i686", 0xbfffe610) = -1 ENOENT (No such file or director
y)
open("/backup-versioned/2003-01-28T052001/lib/tls/mmx/libc.so.6", O_RDONLY) = -1 ENOENT (No such file or dir
ectory)
stat64("/backup-versioned/2003-01-28T052001/lib/tls/mmx", 0xbfffe610) = -1 ENOENT (No such file or directory
)
open("/backup-versioned/2003-01-28T052001/lib/tls/libc.so.6", O_RDONLY) = -1 ENOENT (No such file or directo
ry)
stat64("/backup-versioned/2003-01-28T052001/lib/tls", 0xbfffe610) = -1 ENOENT (No such file or directory)
open("/backup-versioned/2003-01-28T052001/lib/i686/mmx/libc.so.6", O_RDONLY) = -1 ENOENT (No such file or di
rectory)
stat64("/backup-versioned/2003-01-28T052001/lib/i686/mmx", 0xbfffe610) = -1 ENOENT (No such file or director
y)
open("/backup-versioned/2003-01-28T052001/lib/i686/libc.so.6", O_RDONLY) = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\20[\1B"..., 1024) = 1024
close(3)                                = 0
stat64("/backup-versioned/2003-01-28T052001/lib/i686", {st_mode=S_IFDIR|0755, st_size=248, ...}) = 0
open("/backup-versioned/2003-01-28T052001/lib/mmx/libc.so.6", O_RDONLY) = -1 ENOENT (No such file or directo
ry)
stat64("/backup-versioned/2003-01-28T052001/lib/mmx", 0xbfffe610) = -1 ENOENT (No such file or directory)
open("/backup-versioned/2003-01-28T052001/lib/libc.so.6", O_RDONLY) = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0P[\1\000"..., 1024) = 1024
fstat64(3, {st_mode=S_IFREG|0755, st_size=1210920, ...}) = 0
old_mmap(NULL, 1223588, PROT_READ|PROT_EXEC, MAP_PRIVATE, 3, 0) = 0x40016000
mprotect(0x40138000, 35748, PROT_NONE)  = 0
old_mmap(0x40138000, 20480, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 3, 0x122000) = 0x40138000
old_mmap(0x4013d000, 15268, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x4013d000
close(3)                                = 0
rt_sigaction(SIGPIPE, {SIG_IGN}, NULL, 8) = 0
chdir("/var/qmail")                     = 0
open("control/me", O_RDONLY|O_NONBLOCK) = 3
read(3, "v.iki.fi\n", 64)               = 9
close(3)                                = 0
open("control/smtpgreeting", O_RDONLY|O_NONBLOCK) = -1 ENOENT (No such file or directory)
open("control/localiphost", O_RDONLY|O_NONBLOCK) = -1 ENOENT (No such file or directory)
open("control/timeoutsmtpd", O_RDONLY|O_NONBLOCK) = -1 ENOENT (No such file or directory)
open("control/rcpthosts", O_RDONLY|O_NONBLOCK) = 3
read(3, "v.iki.fi\nbabbage.netspan.fi\nbabb"..., 64) = 64
read(3, "\n\n", 64)                     = 2
read(3, "", 64)                         = 0
close(3)                                = 0
open("control/morercpthosts.cdb", O_RDONLY|O_NONBLOCK) = -1 ENOENT (No such file or directory)
open("control/badmailfrom", O_RDONLY|O_NONBLOCK) = -1 ENOENT (No such file or directory)
open("control/databytes", O_RDONLY|O_NONBLOCK) = -1 ENOENT (No such file or directory)
socket(PF_INET, SOCK_STREAM, IPPROTO_IP) = 3
ioctl(3, 0x8912, 0xbffff188)            = 0
ioctl(3, 0x8912, 0xbffff188)            = 0
ioctl(3, 0x8913, 0x804e3b4)             = 0
ioctl(3, 0x8915, 0x804e3b4)             = 0
ioctl(3, 0x8913, 0x804e3d4)             = 0
ioctl(3, 0x8915, 0x804e3d4)             = 0
ioctl(3, 0x8913, 0x804e3f4)             = 0
ioctl(3, 0x8915, 0x804e3f4)             = 0
ioctl(3, 0x8913, 0x804e414)             = 0
ioctl(3, 0x8913, 0x804e434)             = 0
ioctl(3, 0x8915, 0x804e434)             = 0
ioctl(3, 0x8913, 0x804e454)             = 0
ioctl(3, 0x8915, 0x804e454)             = 0
close(3)                                = 0
select(2, NULL, [1], NULL, {1200, 0})   = 1 (out [1], left {1200, 0})
write(1, "220 v.iki.fi ESMTP\r\n", 20)  = 20
select(1, [0], NULL, NULL, {1200, 0} <unfinished ...>

-----------------------------------------------------------------------------

(broken) strace with glibc-2.3.1-38
-----------------------------------------------------------------------------
execve("/var/qmail/bin/qmail-smtpd", ["/var/qmail/bin/qmail-smtpd"], [/* 48 vars */]) = 0
uname({sys="Linux", node="babbage", ...}) = 0
brk(0)                                  = 0x804f45c
old_mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x40015000
open("/etc/ld.so.preload", O_RDONLY)    = -1 ENOENT (No such file or directory)
open("/lib/libNoVersion.so.1", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/local/lib/tls/i686/mmx/libc.so.6", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/usr/local/lib/tls/i686/mmx", 0xbfffe640) = -1 ENOENT (No such file or directory)
open("/usr/local/lib/tls/i686/libc.so.6", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/usr/local/lib/tls/i686", 0xbfffe640) = -1 ENOENT (No such file or directory)
open("/usr/local/lib/tls/mmx/libc.so.6", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/usr/local/lib/tls/mmx", 0xbfffe640) = -1 ENOENT (No such file or directory)
open("/usr/local/lib/tls/libc.so.6", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/usr/local/lib/tls", 0xbfffe640) = -1 ENOENT (No such file or directory)
open("/usr/local/lib/i686/mmx/libc.so.6", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/usr/local/lib/i686/mmx", 0xbfffe640) = -1 ENOENT (No such file or directory)
open("/usr/local/lib/i686/libc.so.6", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/usr/local/lib/i686", 0xbfffe640) = -1 ENOENT (No such file or directory)
open("/usr/local/lib/mmx/libc.so.6", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/usr/local/lib/mmx", 0xbfffe640) = -1 ENOENT (No such file or directory)
open("/usr/local/lib/libc.so.6", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/usr/local/lib", {st_mode=S_IFDIR|0755, st_size=8192, ...}) = 0
open("/etc/ld.so.cache", O_RDONLY)      = 3
fstat64(3, {st_mode=S_IFREG|0644, st_size=93157, ...}) = 0
old_mmap(NULL, 93157, PROT_READ, MAP_PRIVATE, 3, 0) = 0x40016000
close(3)                                = 0
open("/lib/libc.so.6", O_RDONLY)        = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0 Y\1\000"..., 1024) = 1024
fstat64(3, {st_mode=S_IFREG|0755, st_size=1559788, ...}) = 0
old_mmap(NULL, 1231012, PROT_READ|PROT_EXEC, MAP_PRIVATE, 3, 0) = 0x4002d000
mprotect(0x40154000, 22692, PROT_NONE)  = 0
old_mmap(0x40154000, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 3, 0x127000) = 0x40154000
old_mmap(0x40157000, 10404, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x40157000
close(3)                                = 0
munmap(0x40016000, 93157)               = 0
rt_sigaction(SIGPIPE, {SIG_IGN}, NULL, 8) = 0
chdir("/var/qmail")                     = 0
open("control/me", O_RDONLY|O_NONBLOCK) = 3
read(3, "v.iki.fi\n", 64)               = 9
close(3)                                = 0
open("control/smtpgreeting", O_RDONLY|O_NONBLOCK) = -1 ENOENT (No such file or directory)
select(2, NULL, [1], NULL, {1200, 0})   = 1 (out [1], left {1200, 0})
write(1, "421 unable to read controls (#4."..., 38) = 38
semget(1, 134540800, 0x26|046)          = -1 ENOSYS (Function not implemented)
_exit(1)                                = ?

-----------------------------------------------------------------------------

Comment 1 Ville Herva 2003-02-02 17:44:48 UTC
Created attachment 89781 [details]
glibc-2.3.1-5 strace

Comment 2 Ville Herva 2003-02-02 17:45:15 UTC
Created attachment 89782 [details]
glibc-2.3.1-38 strace

Comment 3 Jakub Jelinek 2003-02-03 11:31:55 UTC
The straces don't reveal anything interesting, the only thing I wonder
is why /lib/tls/libc.so.6 is not used. Do you have LD_ASSUME_KERNEL=2.2.5
in the environment, or do you run older kernel?

Comment 4 Ville Herva 2003-02-03 11:40:02 UTC
I didn't find anything conclusive in the straces either :(.




LD_ASSUME_KERNEL is not set, and kernel is 2.4.20-rc1-jam1 (-jam patch set is 
based on -aa).




Comment 5 Ville Herva 2003-02-03 11:50:24 UTC
Is there anything I can/should try at this point? 




I could try to narrow down which glibc change broke qmail, but -5 to -38 is 33 
releases... Do have any suspects? Is there an way to download all the releases 
in between from somewhere?


Comment 6 Jakub Jelinek 2003-02-03 15:03:34 UTC
I'd start with booting a RH kernel if possible (so that you can see if
the problem persists even with NPTL).
And then I'd think best would be to actually look in the debugger where
the code paths start to differ.

Comment 7 Ville Herva 2003-02-06 09:35:02 UTC
Created attachment 89889 [details]
(Working) ltrace from glibc-2.3.1-5

Comment 8 Ville Herva 2003-02-06 09:35:34 UTC
Created attachment 89890 [details]
(failing) ltrace from glibc-2.3.1-38

Comment 9 Jakub Jelinek 2003-02-06 09:46:00 UTC
There are no differences in ltrace until the
open("control/smtpgreeting", 2048, 027777772214) = -1
and then the program magically chooses to either call select (in the bad case),
or open("control/localiphost", 2048, 027777772134)  = -1
I'm afraid gdb or visual inspection of the code between opening smtpgreeting
and localiphost is necessary.

Comment 10 Ville Herva 2003-02-06 10:23:46 UTC
Yes, I've been meaning to take a stab with gdb. (Sorry - I've been too busy for the two past days.) I was wondering about the 027777772214 vs 027777772134 difference, but that's not significant, I gather?

I'll try to compile a debug version tonight, I'll let you know what it turns out.

PS: glibc-2.3.1-38 -> 40 changes are probably not interesting, are they?

Comment 11 Jakub Jelinek 2003-02-06 11:05:13 UTC
The third argument to open of course doesn't matter at all, the second argument
is O_RDONLY|O_NONBLOCK, the third argument is only used with O_CREAT bit set.
Given that there shouldn't be any difference between -5 and -38 either
at least from the ltrace output, I think -38 -> -4{0,1} shouldn't be interesting at all.

Comment 12 Ville Herva 2003-02-06 11:12:21 UTC
Ok, it seems glibc-2.3.1-38 gives fd=-1,errno=0 from open() when the control/smtpgreeting does not exist - qmail expects fd=-1,errno=2(ENOENT), which is what glibc-2.3.1-5 gives.
(Of course, qmail was and is compiled against much older glibc, but this doesn't seem right anyway.)

Hold on, I'll provide more info shortly.

Comment 13 Ville Herva 2003-02-06 11:22:02 UTC
Created attachment 89893 [details]
The relevant code path

The (short) code path qmail-smtpd traverses before failure.

I added some debug to it, and with glibc-2.3.1-38 it gives:
root@babbage:/scratch>echo QUIT|./qmail-smtpd.DEBUG
at control.c:64: control_readline(control/me): fd=3, errno=0
at control.c:66: expecting error_noent=2, ENOENT=2
at control.c:64: control_readline(control/smtpgreeting): fd=-1, errno=2
at control.c:66: expecting error_noent=2, ENOENT=2
at control.c:64: control_readline(control/localiphost): fd=-1, errno=2
at control.c:66: expecting error_noent=2, ENOENT=2
at control.c:64: control_readline(control/timeoutsmtpd): fd=-1, errno=2
at control.c:66: expecting error_noent=2, ENOENT=2
421 unable to read controls (#4.3.0)

The same with glibc-2.3.1-5:
root@babbage:/scratch>echo QUIT | LD_LIBRARY_PATH="/backup-versioned/2003-01-28

./qmail-smtpd.DEBUG
at control.c:64: control_readline(control/me): fd=3, errno=0
at control.c:66: expecting error_noent=2, ENOENT=2
at control.c:64: control_readline(control/smtpgreeting): fd=-1, errno=2
at control.c:66: expecting error_noent=2, ENOENT=2
at control.c:64: control_readline(control/localiphost): fd=-1, errno=2
at control.c:66: expecting error_noent=2, ENOENT=2
at control.c:64: control_readline(control/timeoutsmtpd): fd=-1, errno=2
at control.c:66: expecting error_noent=2, ENOENT=2
at control.c:64: control_readline(control/databytes): fd=-1, errno=2
at control.c:66: expecting error_noent=2, ENOENT=2
220 v.iki.fi ESMTP
221 v.iki.fi

Comment 14 Ville Herva 2003-02-06 11:25:30 UTC
Uhh, I was obviously wrong about the errno, sorry... Hold on.

Comment 15 Jakub Jelinek 2003-02-06 11:28:17 UTC
Oh yes, I think I know what's going on.
qmail is compiled against glibc 2.0.x and a binutils bug causes /lib/libc.so.6's
errno to be updated, not the one in qmail's .dynbss.

Comment 16 Ville Herva 2003-02-06 11:37:24 UTC
Ok, the error I made with the attached debug output was that those fprintf's and compiling with -ggdb (instead of -O2) changed the way the program behaves. (And of course I was so eager about what I found that I forgot the double-check...)

You explanation makes sense. But, I'm not quite sure what is the fix? ;) 

Do you need any further debug info?

Comment 17 Ville Herva 2003-02-07 13:43:17 UTC
Hmm, mkfs.vfat from dosfstools-2.8-1 (haven't upgraded yet) says this:




Incorrectly built binary which accesses errno, h_errno or _res directly. Needs 
to be fixed.




qmail doesn't...




But I'm still a little lost as to what is the correct solution. Am I out of luck 
with my glibc-2.0 linked binaries (there still are quite a bunch of them) from 
now to eternity, or is there something that can be done?


Comment 18 Jakub Jelinek 2003-02-17 14:37:42 UTC
Please try latest rawhide glibc (2.3.1-46 ATM).

Comment 19 Ville Herva 2003-02-17 21:07:20 UTC
With glibc-2.3.1-46, qmail seems to behave 100% right afaict from (really) short testing. If anything turns out in wider testing, I'll let you know.

Thank you very much!

Comment 20 Ville Herva 2003-02-23 21:51:56 UTC
I came across this message from Phil Edwards:

http://news.gmane.org/article.php?id=13960&group=gmane.mail.qmail.general

which explains the breakage, and has a patch for qmail.

Hopefully this is all moot now that glibc is fixed.


Note You need to log in before you can comment on or make changes to this bug.