Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1361760 - [RGW:Active directory]:- "Access denied" error thrown for AD users after being idle for sometime
Summary: [RGW:Active directory]:- "Access denied" error thrown for AD users after bein...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Ceph Storage
Classification: Red Hat
Component: RGW
Version: 2.0
Hardware: x86_64
OS: Unspecified
unspecified
high
Target Milestone: rc
: 2.0
Assignee: Matt Benjamin (redhat)
QA Contact: shylesh
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-07-30 06:46 UTC by shylesh
Modified: 2017-07-30 15:48 UTC (History)
9 users (show)

Fixed In Version: RHEL: ceph-10.2.2-33.el7cp Ubuntu: ceph_10.2.2-25redhat1xenial
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-08-23 19:45:31 UTC


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2016:1755 normal SHIPPED_LIVE Red Hat Ceph Storage 2.0 bug fix and enhancement update 2016-08-23 23:23:52 UTC

Description shylesh 2016-07-30 06:46:45 UTC
Description of problem:
If AD users doesn't do authentication for sometime (30 minutes or so) then trying to authenticate again throws "Access denied". However restarting rgw process fixes the problem.

Not sure this is cache invalidation or something .

Version-Release number of selected component (if applicable):

10.2.2-24redhat1xenial
How reproducible:
Always

Steps to Reproduce:
1. Configure rgw with AD authentication
2. Do some I/O with s3 apis using the AD user
3. keep idle for 30 minutes or so
4. try to do I/O again with s3 apis

Actual results:

ubuntu@magna118:~$ python AD-s3.py
Traceback (most recent call last):
  File "AD-s3.py", line 19, in <module>
    bucket = conn.create_bucket('ad-fresh-bucket')
  File "/usr/lib/python2.7/dist-packages/boto/s3/connection.py", line 621, in create_bucket
    response.status, response.reason, body)
boto.exception.S3ResponseError: S3ResponseError: 403 Forbidden
<?xml version="1.0" encoding="UTF-8"?><Error><Code>AccessDenied</Code><RequestId>tx000000000000000000045-00579c4b57-4d6848-default</RequestId><HostId>4d6848-default-default</HostId></Error>

Expected results:
Connection should be reestablished and user should not see "access denied"

Additional info:

here is the log snippet from rgw when we see "access denied" 

2016-07-30 06:38:15.634904 7f1e4a7fc700 20 RGWEnv::set(): HTTP_HOST: magna118:8080
2016-07-30 06:38:15.634922 7f1e4a7fc700 20 RGWEnv::set(): HTTP_ACCEPT_ENCODING: identity
2016-07-30 06:38:15.634926 7f1e4a7fc700 20 RGWEnv::set(): HTTP_DATE: Sat, 30 Jul 2016 06:38:15 GMT
2016-07-30 06:38:15.634928 7f1e4a7fc700 20 RGWEnv::set(): CONTENT_LENGTH: 0
2016-07-30 06:38:15.634931 7f1e4a7fc700 20 RGWEnv::set(): HTTP_AUTHORIZATION: AWS ewogICAgIlJHV19UT0tFTiI6IHsKICAgICAgICAidmVyc
2lvbiI6IDEsCiAgICAgICAgInR5cGUiOiAiYWQiLAogICAgICAgICJpZCI6ICJjZXBoIiwKICAgICAgICAia2V5IjogIjVoeTEzNWhAMTIzIgogICAgfQp9Cg==:sXd
KPvugn9QTgwPpOVdCU3HPw4s=
2016-07-30 06:38:15.634935 7f1e4a7fc700 20 RGWEnv::set(): HTTP_USER_AGENT: Boto/2.38.0 Python/2.7.12 Linux/4.4.0-31-generic
2016-07-30 06:38:15.634937 7f1e4a7fc700 20 RGWEnv::set(): REQUEST_METHOD: PUT
2016-07-30 06:38:15.634938 7f1e4a7fc700 20 RGWEnv::set(): REQUEST_URI: /ad-fresh-bucket/
2016-07-30 06:38:15.634939 7f1e4a7fc700 20 RGWEnv::set(): QUERY_STRING:
2016-07-30 06:38:15.634941 7f1e4a7fc700 20 RGWEnv::set(): REMOTE_USER:
2016-07-30 06:38:15.634942 7f1e4a7fc700 20 RGWEnv::set(): SCRIPT_URI: /ad-fresh-bucket/
2016-07-30 06:38:15.634944 7f1e4a7fc700 20 RGWEnv::set(): SERVER_PORT: 8080
2016-07-30 06:38:15.634945 7f1e4a7fc700 20 CONTENT_LENGTH=0
2016-07-30 06:38:15.634946 7f1e4a7fc700 20 HTTP_ACCEPT_ENCODING=identity
2016-07-30 06:38:15.634947 7f1e4a7fc700 20 HTTP_AUTHORIZATION=AWS ewogICAgIlJHV19UT0tFTiI6IHsKICAgICAgICAidmVyc2lvbiI6IDEsCiAgICAgICAgInR5cGUiOiAiYWQiLAogICAgICAgICJpZCI6ICJjZXBoIiwKICAgICAgICAia2V5IjogIjVoeTEzNWhAMTIzIgogICAgfQp9Cg==:sXdKPvugn9QTgwPpOVdCU3HPw4s=
2016-07-30 06:38:15.634949 7f1e4a7fc700 20 HTTP_DATE=Sat, 30 Jul 2016 06:38:15 GMT
2016-07-30 06:38:15.634950 7f1e4a7fc700 20 HTTP_HOST=magna118:8080
2016-07-30 06:38:15.634950 7f1e4a7fc700 20 HTTP_USER_AGENT=Boto/2.38.0 Python/2.7.12 Linux/4.4.0-31-generic
2016-07-30 06:38:15.634951 7f1e4a7fc700 20 QUERY_STRING=
2016-07-30 06:38:15.634951 7f1e4a7fc700 20 REMOTE_USER=
2016-07-30 06:38:15.634952 7f1e4a7fc700 20 REQUEST_METHOD=PUT
2016-07-30 06:38:15.634953 7f1e4a7fc700 20 REQUEST_URI=/ad-fresh-bucket/
2016-07-30 06:38:15.634953 7f1e4a7fc700 20 SCRIPT_URI=/ad-fresh-bucket/
2016-07-30 06:38:15.634954 7f1e4a7fc700 20 SERVER_PORT=8080
2016-07-30 06:38:15.634956 7f1e4a7fc700  1 ====== starting new request req=0x7f1e4a7f67e0 =====
2016-07-30 06:38:15.634976 7f1e4a7fc700  2 req 69:0.000020::PUT /ad-fresh-bucket/::initializing for trans_id = tx000000000000000000045-00579c4b57-4d6848-default
2016-07-30 06:38:15.634981 7f1e4a7fc700 10 host=magna118
2016-07-30 06:38:15.634986 7f1e4a7fc700 20 subdomain= domain= in_hosted_domain=0 in_hosted_domain_s3website=0
2016-07-30 06:38:15.635018 7f1e4a7fc700 20 get_handler handler=25RGWHandler_REST_Bucket_S3
2016-07-30 06:38:15.635023 7f1e4a7fc700 10 handler=25RGWHandler_REST_Bucket_S3
2016-07-30 06:38:15.635025 7f1e4a7fc700  2 req 69:0.000070:s3:PUT /ad-fresh-bucket/::getting op 1
2016-07-30 06:38:15.635032 7f1e4a7fc700 10 op=27RGWCreateBucket_ObjStore_S3
2016-07-30 06:38:15.635033 7f1e4a7fc700  2 req 69:0.000077:s3:PUT /ad-fresh-bucket/:create_bucket:authorizing
2016-07-30 06:38:15.635039 7f1e4a7fc700 15 authorize_v2 LDAP auth uri=ldaps://msad-frog.eng.arb.redhat.com:636
2016-07-30 06:38:15.635145 7f1e4a7fc700 10 authorize_v2 try LDAP auth uri=ldaps://msad-frog.eng.arb.redhat.com:636 token.id=ceph
2016-07-30 06:38:15.635192 7f1e4a7fc700 20 get_system_obj_state: rctx=0x7f1e4a7f47b0 obj=default.rgw.users.keys:ewogICAgIlJHV19UT0tFTiI6IHsKICAgICAgICAidmVyc2lvbiI6IDEsCiAgICAgICAgInR5cGUiOiAiYWQiLAogICAgICAgICJpZCI6ICJjZXBoIiwKICAgICAgICAia2V5IjogIjVoeTEzNWhAMTIzIgogICAgfQp9Cg== state=0x7f1f0c00a168 s->prefetch_data=0
2016-07-30 06:38:15.635206 7f1e4a7fc700 10 cache get: name=default.rgw.users.keys+ewogICAgIlJHV19UT0tFTiI6IHsKICAgICAgICAidmVyc2lvbiI6IDEsCiAgICAgICAgInR5cGUiOiAiYWQiLAogICAgICAgICJpZCI6ICJjZXBoIiwKICAgICAgICAia2V5IjogIjVoeTEzNWhAMTIzIgogICAgfQp9Cg== : type miss (requested=6, cached=0)
2016-07-30 06:38:15.635856 7f1e4a7fc700 10 cache put: name=default.rgw.users.keys+ewogICAgIlJHV19UT0tFTiI6IHsKICAgICAgICAidmVyc2lvbiI6IDEsCiAgICAgICAgInR5cGUiOiAiYWQiLAogICAgICAgICJpZCI6ICJjZXBoIiwKICAgICAgICAia2V5IjogIjVoeTEzNWhAMTIzIgogICAgfQp9Cg== info.flags=0
2016-07-30 06:38:15.635864 7f1e4a7fc700 10 moving default.rgw.users.keys+ewogICAgIlJHV19UT0tFTiI6IHsKICAgICAgICAidmVyc2lvbiI6IDEsCiAgICAgICAgInR5cGUiOiAiYWQiLAogICAgICAgICJpZCI6ICJjZXBoIiwKICAgICAgICAia2V5IjogIjVoeTEzNWhAMTIzIgogICAgfQp9Cg== to cache LRU end
2016-07-30 06:38:15.635876 7f1e4a7fc700  5 error reading user info, uid=ewogICAgIlJHV19UT0tFTiI6IHsKICAgICAgICAidmVyc2lvbiI6IDEsCiAgICAgICAgInR5cGUiOiAiYWQiLAogICAgICAgICJpZCI6ICJjZXBoIiwKICAgICAgICAia2V5IjogIjVoeTEzNWhAMTIzIgogICAgfQp9Cg== can't authenticate
2016-07-30 06:38:15.635878 7f1e4a7fc700 10 failed to authorize request
2016-07-30 06:38:15.635879 7f1e4a7fc700 20 handler->ERRORHANDLER: err_no=-13 new_err_no=-13
2016-07-30 06:38:15.635961 7f1e4a7fc700  2 req 69:0.001005:s3:PUT /ad-fresh-bucket/:create_bucket:op status=0
2016-07-30 06:38:15.635967 7f1e4a7fc700  2 req 69:0.001011:s3:PUT /ad-fresh-bucket/:create_bucket:http status=403
2016-07-30 06:38:15.635972 7f1e4a7fc700  1 ====== req done req=0x7f1e4a7f67e0 op status=0 http_status=403 ======
2016-07-30 06:38:15.635978 7f1e4a7fc700 20 process_request() returned -13
2016-07-30 06:38:15.636011 7f1e4a7fc700  1 civetweb: 0x7f1f0c004160: 10.8.128.118 - - [30/Jul/2016:06:38:15 +0000] "PUT /ad-fresh-bucket/ HTTP/1.1" 403 0 - Boto/2.38.0 Python/2.7.12 Linux/4.4.0-31-generic


However please note that restarting rgw solves this problem and authentication works as usual.

Comment 8 shylesh 2016-08-04 15:12:19 UTC
Verified on 10.2.2-25redhat1xenial and ceph-radosgw-10.2.2-33.el7cp.x86_64. Marking it as verified.

Comment 9 shylesh 2016-08-04 15:12:54 UTC
(In reply to shylesh from comment #8)
> Verified on 10.2.2-25redhat1xenial and ceph-radosgw-10.2.2-33.el7cp.x86_64.
> Marking it as verified.

Even with idle time of 45 minutes authentication is working fine.

Comment 11 errata-xmlrpc 2016-08-23 19:45:31 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-1755.html


Note You need to log in before you can comment on or make changes to this bug.