Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 451057 - [RHEL 5] lockd not aware of statd -n switch
Summary: [RHEL 5] lockd not aware of statd -n switch
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 5
Classification: Red Hat
Component: nfs-utils
Version: 5.2
Hardware: All
OS: Linux
high
high
Target Milestone: rc
: ---
Assignee: Steve Dickson
QA Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2008-06-12 16:30 UTC by Janne Karhunen
Modified: 2013-08-06 00:43 UTC (History)
9 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2009-01-20 21:01:20 UTC
Target Upstream Version:


Attachments (Terms of Use)
lockd callback fix for rhel5 (deleted)
2008-06-12 16:30 UTC, Janne Karhunen
no flags Details | Diff


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2009:0107 normal SHIPPED_LIVE nfs-utils bug fix and enhancement update 2009-01-20 16:04:38 UTC

Description Janne Karhunen 2008-06-12 16:30:00 UTC
+++ This bug was initially created as a clone of Bug #441559 +++

Description of problem:

Lockd seems to reject callbacks from statd started with -n switch: it seems to
expect statd always to be bound to loopback address (RHEL4U6:
fs/lockd/svc4proc.c:429).


Version-Release number of selected component (if applicable):

nfs-utils-1.0.6-84 (probably valid for all..)


How reproducible:

Always

Steps to Reproduce:
1. Create NFS server fail over pair 
2. Migrate NFS server IP address between servers on failover
3. Try to bind server statd to migrated address on server startup using -n


Actual results:

Nov  8 13:27:55 warning CLA-0 kernel: lockd: rejected NSM callback from c0a80011:776
Nov  8 13:27:55 notice CLA-0 rpc.statd[31552]: recv_rply: [127.0.0.1] RPC status 5

Expected results:

No rejected callbacks

-- Additional comment from steved@redhat.com on 2008-04-08 16:09 EST --
-------- Original Message --------
Subject: IT #137407
Date: Tue, 08 Apr 2008 13:49:43 -0400
From: Janne Karhunen <jkarhune@redhat.com>
To: steved@redhat.com

Hi Steve,

I found a system that reproduces IT #137407. After having checked 
checked the address server lockd now whines about (rejected
statd callback from c0a80035), it turns out to be perfectly valid 
local server address (Directory). So, it should not reject it and
the call is valid.

Looking at:

static int
nlm4svc_proc_sm_notify(struct svc_rqst *rqstp, struct nlm_reboot *argp,
                                              void              *resp)
{
        struct sockaddr_in      saddr = rqstp->rq_addr;
        int                     vers = argp->vers;
        int                     prot = argp->proto >> 1;

        struct nlm_host         *host;

        dprintk("lockd: SM_NOTIFY     called\n");
        if (saddr.sin_addr.s_addr != htonl(INADDR_LOOPBACK)
         || ntohs(saddr.sin_port) >= 1024) {
                printk(KERN_WARNING
                        "lockd: rejected NSM callback from %08x:%d\n",
                        ntohl(rqstp->rq_addr.sin_addr.s_addr),
                        ntohs(rqstp->rq_addr.sin_port));
                return rpc_system_err;
        }
..

So lockd expects statd to be always bound to LOOPBACK. Now, 
looking at statd code we have:

int
statd_get_socket(int port)
{
        struct sockaddr_in      sin;

        if (sockfd >= 0)
                return sockfd;

        if ((sockfd = socket(AF_INET, SOCK_DGRAM, IPPROTO_UDP)) < 0) {
                note(N_CRIT, "Can't create socket: %m");
                return -1;
        }

        FD_SET(sockfd, &SVC_FDSET);

        memset(&sin, 0, sizeof(sin));
        sin.sin_family = AF_INET;
        sin.sin_addr.s_addr = INADDR_ANY;
        /*
         * If a local hostname is given (-n option to statd), bind to
the address
         * specified. This is required to support clients that ignore
the mon_name in
         * the statd protocol but use the source address from the
request packet.
         */
        if (MY_NAME) {
                struct hostent *hp = gethostbyname(MY_NAME);
                if (hp)
                        sin.sin_addr = *(struct in_addr *) hp->h_addr;
..


So these two are clearly out of sync. We are, indeed, using
-n Directory as statd manpage states that:

       -n, --name name
              specify  a  name  for  rpc.statd  to use as the local
hostname. By default, rpc.statd will call gethostname(2) to get the
              local hostname. Specifying a local hostname may be useful
for machines with more than one interfaces.
 
IMHO lockd should be aware of such case. Which is what we have
here.


-- 
// Janne


-- Additional comment from jkarhune@redhat.com on 2008-04-10 11:01 EST --
Created an attachment (id=302005)
draft of lockd part


-- Additional comment from jkarhune@redhat.com on 2008-04-10 11:02 EST --
Created an attachment (id=302006)
draft of statd part


-- Additional comment from jkarhune@redhat.com on 2008-04-10 13:34 EST --
Just about to start testing this - first bug: open return value 

-- Additional comment from steved@redhat.com on 2008-04-10 13:52 EST --
Janne,

When generating patch please the '-p' argument to the diff command 
so the functions names are added to the patch. This makes it
much easier to read the patch and find the changed code.

I used a modified version of /usr/bin/gendiff that has
the following change:

--- /usr/bin/gendiff.orig       2008-04-10 13:50:08.000000000 -0400
+++ /usr/bin/gendiff    2008-04-10 13:50:12.000000000 -0400
@@ -8,7 +8,7 @@
 
 find $1 \( -name "*$2" -o -name ".*$2" \) -print |
 while read f; do
-    U=-u
+    U=-up
     [ "`basename $f`" = "ChangeLog$2" ] && U=-U0
 #    diff ${U} $f `echo $f | sed s/$2\$//`
     if [ -r "$f" ]; then


-- Additional comment from jkarhune@redhat.com on 2008-04-10 15:06 EST --
Thanks Steve, will do. And no - patch does not work as such, possibly byte order
was wrong. Will fix now.

-- Additional comment from jkarhune@redhat.com on 2008-04-11 12:36 EST --
Bah. Fixing the fix indeed fixes it.

-- Additional comment from jkarhune@redhat.com on 2008-04-11 14:14 EST --
Created an attachment (id=302152)
tested nsm patch


-- Additional comment from jkarhune@redhat.com on 2008-04-11 14:14 EST --
Created an attachment (id=302153)
tested nlm patch


-- Additional comment from staubach@redhat.com on 2008-04-11 14:46 EST --
These patches seem a bit IPv4 specific.  Is there a way to do this in
more independent fashion?

-- Additional comment from jkarhune@redhat.com on 2008-04-11 15:04 EST --
You're right. We can certainly switch to using strings instead of addresses.

-- Additional comment from jkarhune@redhat.com on 2008-04-18 14:14 EST --
Humm; as it binds to INADDR_ANY without this option, is it actually completely
blind luck that it works at all? Or is this some obscure corner of the NLM protocol?

-- Additional comment from jkarhune@redhat.com on 2008-04-18 14:42 EST --
Argh, sorry. So it will work; it will just expose everything out.

-- Additional comment from jkarhune@redhat.com on 2008-04-29 09:46 EST --
Two different patch variants that would address this issue have been sent to
linux-nfs list.

-- Additional comment from jkarhune@redhat.com on 2008-05-05 09:45 EST --
So the solution is to forget that -n existed and use -H with notify script, right?

-- Additional comment from jkarhune@redhat.com on 2008-05-05 11:30 EST --
Argh, so the solution:
- use -L to prevent statd from notifying anyone
- use sm-notify manually with -v to notify correct segment

-- Additional comment from jkarhune@redhat.com on 2008-05-05 12:23 EST --
Very non-intrusive fix for RHEL family: remove the -n address bind from
statd_get_socket(). It should work straight out, all it doesn't do is the port
visibility limitation. Lockd will see packets from lo and -v is used for
sm-notify correctly.

-- Additional comment from jkarhune@redhat.com on 2008-05-05 12:52 EST --
Created an attachment (id=304538)
Fixes -n lockd callbacks. Lockd will now see packets from lo.


-- Additional comment from jleddy@redhat.com on 2008-05-21 09:39 EST --
What's the status of this?  There are 2 valid patches against rmtcall.c, one of
which appears to ignore -n altogether.  Is our official stance that -n is obsolete?

-- Additional comment from jkarhune@redhat.com on 2008-05-21 09:52 EST --
-n is absolutely mandatory for lock fail-over between server nodes that have
different names. Without -n server statd will send wrong mon_name to clients on
failover and hence clients would be completely confused who actually rebooted
(and would not initiate lock reclaim).

And no, that patch does not ignore -n. It only removes the -n address binding
that breaks lockd callbacks, so it relies on kernel mangling the packet source
address always to right outgoing network (that is, lo for lockd which makes the
callbacks work).


-- Additional comment from jkarhune@redhat.com on 2008-05-21 10:18 EST --
To clarify: -n is mandatory for rhel4 tree only as it does not have sm-notify.

-- Additional comment from jleddy@redhat.com on 2008-05-21 17:46 EST --
Thanks a lot, that definitely helps.  

So, just to be sure, we are using
https://bugzilla.redhat.com/attachment.cgi?id=304538 , right?

-- Additional comment from jkarhune@redhat.com on 2008-05-22 09:36 EST --
Right.

-- Additional comment from pm-rhel@redhat.com on 2008-05-22 12:24 EST --
This request was evaluated by Red Hat Product Management for inclusion in a Red
Hat Enterprise Linux maintenance release.  Product Management has requested
further review of this request by Red Hat Engineering, for potential
inclusion in a Red Hat Enterprise Linux Update release for currently deployed
products.  This request is not yet committed for inclusion in an Update
release.

-- Additional comment from jkarhune@redhat.com on 2008-06-10 13:38 EST --
I was able to reproduce the issue on local NokiaBlade cluster and verify that
patch indeed fixes it.

-- Additional comment from rdoty@redhat.com on 2008-06-10 15:48 EST --
Since a patch has been developed and partner verified and this is an existing
RHEL 4 bug, requesting an exception for 4.7.

-- Additional comment from staubach@redhat.com on 2008-06-10 16:04 EST --
What is the status of this patch with respect to upstream acceptance?

I had the impression that it wasn't well received.  Or am I confused?

-- Additional comment from rdoty@redhat.com on 2008-06-10 16:18 EST --
My understanding is that this problem doesn't exist in RHEL 5 - that the
upstream changes since RHEL 4 eliminate the need for this patch.

-- Additional comment from staubach@redhat.com on 2008-06-10 16:22 EST --
I would think that the RHEL-4 solution should mirror that for upstream
and/or RHEL-5 as closely as possible.  I am not sure that using one
solution in RHEL-4 and after upgrading to RHEL-5, using a different
solution, makes much sense to me.

-- Additional comment from steved@redhat.com on 2008-06-10 19:24 EST --
Well the RHEL-5 nfs-utils appears to have the same problem as the
RHEL-4 on since they basically do the same thing. 

Now quickly looked into back porting the upstream solution which means 
introducing the sm-notify binary. The first patch what would be needed is 
a 742 line patch. Unfortunately this would not be the only patch needed. 

So I am of the option that if Janne's patch (which is much smaller, 5 lines
and not as risky) works we should go with it... 

But do think we should wait for Janne to complete all of his testing.
Then at the point, decide if the patch should be committed 
based on his results.


-- Additional comment from jkarhune@redhat.com on 2008-06-10 19:56 EST --
(In reply to comment #28)
> What is the status of this patch with respect to upstream acceptance?
> 
> I had the impression that it wasn't well received.  Or am I confused?

Yes you are. It's already in, and has been since 1.1.



-- Additional comment from jkarhune@redhat.com on 2008-06-11 10:18 EST --
FYI: reason this randomly worked for us is probably that bind() itself is
somewhat broken. In RHEL4 you can seemingly bind an interface that does not
exist, but in such case you get ANY regardless of what you asked. When you get
ANY it works right.

-- Additional comment from jkarhune@redhat.com on 2008-06-11 11:45 EST --
Just to make sure everyone understands the only downside of this patch. 

Here we have a packet capture when NFS server is failing over from CLA-0 to
CLA-1, with patch. This is seen from client side.

Directory (NFS) address: 192.168.0.64
CLA-1: 192.168.0.5

tshark -V -R stat -i bond0

    Source: 192.168.0.5 (192.168.0.5)
    Destination: 192.168.0.7 (192.168.0.7)

Network Status Monitor Protocol
    [Program Version: 1]
    [V1 Procedure: NOTIFY (6)]
    Status Change
        Monitor ID Name: Directory
            length: 9
            contents: Directory
            fill bytes: opaque data
    State: 683

Protocol mandates you to use 'nonitor name' (-n) to detect who rebooted. That
information is correct in this patched case. 

Client that would use packet source address as an server name would not work
correctly if kernel gets the source wrong, as is the case above. But:
1) client is doing something wrong if it prefers SOURCE to NAME
2) theres zero chance that this has ever worked, except in the case where bind
does the ANY bind anyway.

As to get this absolutely right for RHEL5 (as in not leaving any chance for the
client to misinterpret things) we would need to:
1) make sure bind works reliably for given interface X and does not succeed on
non-existent interfaces
2) backport the whole sm-notify concept if its not already there


-- Additional comment from jkarhune@redhat.com on 2008-06-11 13:09 EST --
Tested following cases:
1) complete cluster restart(s)
2) node restart(s)
3) server failover(s)
4) node restarts after server failover

Also did a protocol traces and only found the inconsistency explained above.
There were no rejected callbacks, all mon_names were set correctly and even
source addresses were correct with the above mentioned exception.

-- Additional comment from jkarhune@redhat.com on 2008-06-11 14:25 EST --
I'm betting mainline needs a patch here as well. sm-notify should have
IP_FREEBIND set to zero (setsockopt) to make sure source inconsistency does not
happen there as well. This would prevent bind from succeeding when in fact, it
did nothing.

-- Additional comment from jkarhune@redhat.com on 2008-06-12 10:12 EST --
Created an attachment (id=309071)
lockd callback fix for rhel5

Same fix for latest el5 nfs-utils.

-- Additional comment from staubach@redhat.com on 2008-06-12 11:26 EST --
You will probably need to clone this bug for RHEL-5 and then attach
the RHEL-5 patch to it.  Usually, 1 bugzilla for fix per release.

Comment 1 Janne Karhunen 2008-06-12 16:30:52 UTC
Created attachment 309099 [details]
lockd callback fix for rhel5

Comment 2 RHEL Product and Program Management 2008-06-12 18:06:00 UTC
This request was evaluated by Red Hat Product Management for inclusion in a Red
Hat Enterprise Linux maintenance release.  Product Management has requested
further review of this request by Red Hat Engineering, for potential
inclusion in a Red Hat Enterprise Linux Update release for currently deployed
products.  This request is not yet committed for inclusion in an Update
release.

Comment 4 Janne Karhunen 2008-06-16 19:00:29 UTC
Patch tested on rhel5. Works fine.

Comment 6 Russell Doty 2008-09-11 21:09:18 UTC
What needs to be done to get this patch posted? We are getting close to the deadline.

Comment 7 Steve Dickson 2008-09-17 14:44:50 UTC
Fixed in nfs-utils-1.0.9-36.el5

Comment 11 Karel Volný 2008-10-30 09:27:36 UTC
ad comment #4, please provide the details about the package set which was tested; or clarify the reproducer setup

Comment 12 Patrick C. F. Ernzer 2008-10-30 19:14:08 UTC
Janne,
as Comment #4 is too vague and we now have RHEL 5.3 Beta available on london, please test again and report back with the versions you tested.

[the below will make perfect sense to Janne but no sense at all to people with no access to the lab in Helsinki]
If you need machines for the testing, I have reserved stitch 3/8 slots 9 through 14 for you. That is 3 CPU blades, each with their own HD, each pair on a separate loop.
You can access them as fargo 236. 237 and 238.
Please do tell if you reconfigure the fibre loop for this test.
[Helsinki part ends]

It is imperative that you test and report back.

PCFE

Comment 13 Janne Karhunen 2008-10-31 12:39:07 UTC
First you need to set up NFS server failover pair (with migrating server IP address, NOTIFY LIST and UNIFORM SERVER_NAME on both nodes) and have two clients connected to the server. Then you need a basic application that grabs a lock over NFS.

Run locktest application on the client once server1 is up. Watch with ethereal that standard notify links are established both ways (tshark -R stat). Fail over server1 to server2. Watch that server notifies client that server 'rebooted' and verify that client retakes the lock. Double check with client2 that client1 is indeed still holding that lock and no duplicate locks are granted.

Provided that patch is not in you will see that server statd notification towards lockd is dropped as request comes in via unknown interface *IF* statd had 'correctly' bound corresponding interface (the one SERVER_NAME refers to). So it may also accidentally work if migrated interface is not up at the time when statd starts (or precisely sm-notify on rhel5) as IP_FREEBIND is enabled by default.

HTH;

Comment 14 Patrick C. F. Ernzer 2008-10-31 13:26:00 UTC
Janne,
as discused earlier, in case you manage to squeeze in testing forr this next week all of stitch8 is now reserved for you. The reservation of the blades in stitch8 has been revoked.
You can change the fibre loop config in stitch8 to suit your needs, no one else is on that chassis.
http://london.fp.nsn-rdnet.net/LondontestNetwork/stitch/stitch_slot_assignment.html shows you the console ports.

Comment 15 Janne Karhunen 2008-10-31 13:42:39 UTC
As we're not using RHEL5 yet, you have RHEL cluster suite with NFS failover available?

Comment 16 Patrick C. F. Ernzer 2008-10-31 14:17:26 UTC
as discussed with Janne before he left.
Yes CS is available on those nodes through yum.
But recommend you do your test just like in June instead of learning an all new product and getting potential side-effects you do not expect.
If after the w-e you reconsidered and want to use CS, just have a peek at http://www.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/5.2/html/Cluster_Administration/index.html and install the packages with yum.
You can see the contents of the channel easiest on london under /var/ftp/pub/RHEL5.3-Beta/Server/i386/DVD/Cluster/

Comment 17 Janne Karhunen 2008-11-04 13:22:01 UTC
As far as comment #4 goes I might have only tested that it doesn't bind anything 'illegal' on RHEL5 (can't remember). Proper testing with scenario #13 is probably completely undone and will probably show up bunch of new bugs :-/

Comment 18 Janne Karhunen 2008-11-04 13:45:21 UTC
[root@stitch-8-1 ~]# cat /etc/hosts
# Do not remove the following line, or various programs
# that require network functionality will fail.
127.0.0.1		localhost.localdomain localhost
::1		localhost6.localdomain6 localhost6
192.10.10.125		stitch-8-1.testing.local stitch-8-1

192.168.1.1		stitch-nfs.testing.local stitch-nfs

[root@stitch-8-1 ~]# ip a |grep inet
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host 
    inet 192.10.10.125/24 brd 192.10.10.255 scope global eth0
    inet 192.168.1.1/24 brd 192.168.1.255 scope global eth0:1
    inet6 fe80::20e:cff:fe52:d71e/64 scope link 

[root@stitch-8-1 ~]# rpc.statd -n stitch-nfs
[root@stitch-8-1 ~]# 
[root@stitch-8-1 ~]# netstat -anp|grep statd
tcp        0      0 0.0.0.0:880                 0.0.0.0:*                   LISTEN      2394/rpc.statd      
udp        0      0 0.0.0.0:874                 0.0.0.0:*                               2394/rpc.statd      
udp        0      0 0.0.0.0:877                 0.0.0.0:*                               2394/rpc.statd      
unix  2      [ ]         DGRAM                    11120  2394/rpc.statd   

[root@stitch-8-1 ~]# ps -ef |grep statd
root      2394     1  0 15:39 ?        00:00:00 rpc.statd -n stitch-nfs

[root@stitch-8-1 ~]# rpm -qa |grep nfs
nfs-utils-lib-1.0.8-7.2.z2
nfs-utils-1.0.9-38.el5

Comment 20 Chris Ward 2008-11-28 07:14:47 UTC
Partners, this bug should be fixed in the latest RHEL 5.3 Snapshot. We believe that you have some interest in its correct functionality, so we're making a friendly request to send us some testing feedback. 

If you have a chance to test it, please share with us your findings. If you have successfully VERIFIED the fix, please add PartnerVerified to the Bugzilla keywords, along with a description of the results. Thanks!

Comment 27 Chris Ward 2008-12-16 16:29:13 UTC
~~~ Attention Partners ~~~ The *last* RHEL 5.3 Snapshot 6 is now available at partners.redhat.com. A fix for this bug should be present. Please test and update this bug with test results as soon as possible.  If the fix present in Snap6 meets all the expected requirements for this bug, please add the keyword PartnerVerified. If any new bugs are discovered, please CLONE this bug and describe the issues encountered there.

Comment 29 Patrick C. F. Ernzer 2008-12-17 13:27:40 UTC
Janne Karhunen to please test and report back as asked in Comment #27

Comment 30 Kari Hautio 2008-12-18 13:08:42 UTC
The fix itself has been tested with beta in Comment #18.

As only minor changes to other functionality than statd has been done to nfs-utils since that we see that this test is still valid.

nfs-utils: nfs-utils-1.0.9-38.el5 -> nfs-utils-1.0.9-40.el5
-----------------------------------------------------------
Wed Nov 12 2008 Steve Dickson <steved@redhat.com> 1.0.9-40
- Fixed arguments to the hosts_ctl() call in the good_client()
  routine used in the tcpwrapper support. (bz 440120)
Tue Nov 11 2008 Steve Dickson <steved@redhat.com> 1.0.9-39
- Fixed typo in nfs initscript that caused rpc.rquotad daemons
  to be started but not stoppped (bz 470483)

Comment 33 errata-xmlrpc 2009-01-20 21:01:20 UTC
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHBA-2009-0107.html


Note You need to log in before you can comment on or make changes to this bug.