Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1359403 - [RGW-NFS]:- objects created from s3 apis are not visible from nfs mount point
Summary: [RGW-NFS]:- objects created from s3 apis are not visible from nfs mount point
Alias: None
Product: Red Hat Ceph Storage
Classification: Red Hat
Component: RGW
Version: 2.0
Hardware: x86_64
OS: Unspecified
Target Milestone: rc
: 2.2
Assignee: Matt Benjamin (redhat)
QA Contact: Ramakrishnan Periyasamy
Depends On:
TreeView+ depends on / blocked
Reported: 2016-07-23 19:01 UTC by shylesh
Modified: 2017-07-30 15:56 UTC (History)
12 users (show)

Fixed In Version: RHEL: ceph-10.2.5-11.el7cp, nfs-ganesha-2.4.2-3.el7cp Ubuntu: ceph_10.2.5-5redhat1xenial, nfs-ganesha_2.4.2-3redhat1xenial
Doc Type: Known Issue
Doc Text:
Clone Of:
Last Closed: 2017-03-14 15:44:27 UTC
Target Upstream Version:

Attachments (Terms of Use)

System ID Priority Status Summary Last Updated
Ceph Project Bug Tracker 18651 None None None 2017-01-24 17:44:36 UTC 346082 None None None 2017-01-31 23:58:31 UTC
Red Hat Product Errata RHBA-2017:0514 normal SHIPPED_LIVE Red Hat Ceph Storage 2.2 bug fix and enhancement update 2017-03-21 07:24:26 UTC

Description shylesh 2016-07-23 19:01:21 UTC
Description of problem:
objects created directly from s3 api are not visible on the mount nfs mount point

Version-Release number of selected component (if applicable):


How reproducible:

Steps to Reproduce:
1. configured a cluster with rgw with ganesha enabled
2. mounted with nfs
  mount -t nfs -o nfsvers=4.1,sync,noauto,soft,proto=tcp /mnt
3. now if you check on the mount point you will see all the buckets and objects available.
4. from s3 apis create few objects  directly on rgw
5. Now on the mount point check ls

Actual results:
Newly created objects will not be visible

Expected results:
Newly created objects should be visible on the mount point

Additional info:

However if send a lookup on the object then it will appear on the mount point .

From RGW object listing looks like
another-missed-hellonew-another-newad-bucket-creation   2016-07-23T13:05:56.517Z
<Key: another-missed-hellonew-another-newad-bucket-creation,4M>
<Key: another-missed-hellonew-another-newad-bucket-creation,NEWBIG>
<Key: another-missed-hellonew-another-newad-bucket-creation,foobar8>
<Key: another-missed-hellonew-another-newad-bucket-creation,smalllllllllll>
<Key: another-missed-hellonew-another-newad-bucket-creation,smalllllllllll0>
<Key: another-missed-hellonew-another-newad-bucket-creation,smalllllllllll1>
<Key: another-missed-hellonew-another-newad-bucket-creation,smalllllllllll2>
<Key: another-missed-hellonew-another-newad-bucket-creation,smalllllllllll3>
<Key: another-missed-hellonew-another-newad-bucket-creation,smalllllllllll4>
<Key: another-missed-hellonew-another-newad-bucket-creation,smalllllllllll5>
<Key: another-missed-hellonew-another-newad-bucket-creation,smalllllllllll6>
<Key: another-missed-hellonew-another-newad-bucket-creation,smalllllllllll7>
<Key: another-missed-hellonew-another-newad-bucket-creation,smalllllllllll8>
<Key: another-missed-hellonew-another-newad-bucket-creation,smalllllllllll9>
<Policy: First User (owner) = FULL_CONTROL>

From nfs mount point

[root@magna110 another-missed-hellonew-another-newad-bucket-creation]# ll
total 518
-rw-r--r--. 1 4294967294 4294967294 262144 Jul 23 15:35 4M
-rw-rw-rw-. 1 root       root         6413 Jul 23 13:05 foobar8
-rw-r--r--. 1 4294967294 4294967294 262144 Jul 23 13:43 NEWBIG

^^ onlye few objects are visible. 

Now send the lookup on one of the object
[root@magna110 another-missed-hellonew-another-newad-bucket-creation]# ll smalllllllllll0
-rw-rw-rw-. 1 root root 6413 Jul 23 17:26 smalllllllllll0

After the lookup ls on the mount point 
[root@magna110 another-missed-hellonew-another-newad-bucket-creation]# pwd
[root@magna110 another-missed-hellonew-another-newad-bucket-creation]# ll
total 524
-rw-r--r--. 1 4294967294 4294967294 262144 Jul 23 15:35 4M
-rw-rw-rw-. 1 root       root         6413 Jul 23 13:05 foobar8
-rw-r--r--. 1 4294967294 4294967294 262144 Jul 23 13:43 NEWBIG
-rw-rw-rw-. 1 root       root         6413 Jul 23 17:26 smalllllllllll0

 ^^^^^ object "smalllllllllll0" is visible on the mount point.

Comment 2 Matt Benjamin (redhat) 2016-07-25 14:15:29 UTC
This behavior is likely not a bug, or may not be a bug, depending on more detailed semantic requirements discussion.  In other words, these are topics for discussion for 2.1.  This feature is, as well, tech preview in 2.0--so moving to 2.1, as well.

Comment 3 Ramakrishnan Periyasamy 2016-11-07 15:14:32 UTC
Observed this problem during 2.1 testing.

buckets created in NFS mount are listed by rgw s3 api but objects created by rgw s3 api is not listed in NFS mount.

Comment 4 Matt Benjamin (redhat) 2017-01-11 22:27:12 UTC
Ok, this is a defect related to lack of a mechanism to efficiently expire cached directory information, on the librgw side, and lack of a dirent expiry mechanism in nfs-ganesha.

nfs-ganesha has an attribute expiration mechanism which I expected to allow eventual aging out of cached dirents, but this is not the case.  

I don't think this behavior will be correctable with nfs-ganesha 2.4.1/2, but will be addressed with new dirent-caching changes currently being developed.  That suggests we won't be able to fix this issue in 2.3.  We might need a release note for 2.2.

Comment 5 Harish NV Rao 2017-01-18 13:34:42 UTC
@Uday, Could you please let us know the decision of PM w.r.t the proposal made in comment 4?

Comment 9 Matt Benjamin (redhat) 2017-01-21 04:29:25 UTC
My proposed implementation of this feature has been pushed as:  (ceph)

and (nfs-ganesha)

The latter must go through upstream review, sequenced after merge of the ceph change, and probably after at least one upstream bugfix related to synchronous invalidate upcall arguments (details on nfs-ganesha-devel mailing list).


Comment 10 Ramakrishnan Periyasamy 2017-01-23 10:22:23 UTC
Not seeing the buckets created by s3 in NFS mount by default but after restarting NFS service & re-mounting nfs mounts on each clients, buckets and objects created by s3 are visible.

Ceph: 10.2.5-7.el7cp (59e9fee4a935fdd2bc8197e07596dc4313c410a3)
NFS: nfs-ganesha-rgw-2.4.1-3.el7cp.x86_64

Comment 11 shylesh 2017-01-23 10:47:34 UTC
I also observed that objects deleted from s3 are not reflected on nfs mounts and hence objects reappears again upon sending lookups on the object and also operations like write,read succeeds .

Comment 12 Matt Benjamin (redhat) 2017-01-23 14:56:30 UTC
Please see my remarks and candidate code in comment 9.

Comment 23 Ramakrishnan Periyasamy 2017-02-09 06:04:50 UTC
Objects created in S3 are visible in NFS mount. sometimes max it took a minute to reflect the objects created using s3.

Moving this bug to verified state.

Comment 27 errata-xmlrpc 2017-03-14 15:44:27 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

Note You need to log in before you can comment on or make changes to this bug.