Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.

Bug 1512400

Summary: Repository sync "Actions::Pulp::Repository::RegenerateApplicability" step might fail with "500 Internal Server Error".
Product: Red Hat Satellite 6 Reporter: Hao Chang Yu <hyu>
Component: PulpAssignee: satellite6-bugs <satellite6-bugs>
Status: CLOSED ERRATA QA Contact: Jitendra Yejare <jyejare>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 6.2.12CC: bbuckingham, brubisch, daviddavis, dkliban, egolov, jyejare, mhrivnak, pcreech, rchan, ttereshc
Target Milestone: UnspecifiedKeywords: Triaged
Target Release: Unused   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: pulp-2.13.4.5-1 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-02-21 16:54:17 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Bug Depends On:    
Bug Blocks: 1122832    

Description Hao Chang Yu 2017-11-13 06:27:49 UTC
Description of problem:
Cursor time out while querying the repository profile applicability with large number of consumers.

Cursor not found, cursor id: 138030585475
 Traceback (most recent call last):
   File "/usr/lib/python2.7/site-packages/django/core/handlers/base.py", line 112, in get_response
     response = wrapped_callback(request, *callback_args, **callback_kwargs)
   File "/usr/lib/python2.7/site-packages/django/views/generic/base.py", line 69, in view
     return self.dispatch(request, *args, **kwargs)
   File "/usr/lib/python2.7/site-packages/django/views/generic/base.py", line 87, in dispatch
     return handler(request, *args, **kwargs)
   File "/usr/lib/python2.7/site-packages/pulp/server/webservices/views/decorators.py", line 241, in _auth_decorator
     return _verify_auth(self, operation, super_user_only, method, *args, **kwargs)
   File "/usr/lib/python2.7/site-packages/pulp/server/webservices/views/decorators.py", line 195, in _verify_auth
     value = method(self, *args, **kwargs)
   File "/usr/lib/python2.7/site-packages/pulp/server/webservices/views/util.py", line 130, in wrapper
     return func(*args, **kwargs)
   File "/usr/lib/python2.7/site-packages/pulp/server/webservices/views/repositories.py", line 913, in post
     queue_regenerate_applicability_for_repos(repo_criteria.as_dict())
   File "/usr/lib/python2.7/site-packages/pulp/server/managers/consumer/applicability.py", line 167, in queue_regenerate_applicability_for_repos
     for batch in paginate(profile_hashes, 10):
   File "/usr/lib/python2.7/site-packages/pulp/plugins/util/misc.py", line 33, in paginate
     page = tuple(itertools.islice(generator, 0, page_size))
   File "/usr/lib/python2.7/site-packages/pulp/plugins/util/misc.py", line 31, in <genexpr>
   File "/usr/lib64/python2.7/site-packages/pymongo/cursor.py", line 1097, in next
     if len(self.__data) or self._refresh():
   File "/usr/lib64/python2.7/site-packages/pymongo/cursor.py", line 1039, in _refresh
   File "/usr/lib64/python2.7/site-packages/pymongo/cursor.py", line 903, in __send_message
     codec_options=self.__codec_options)
   File "/usr/lib64/python2.7/site-packages/pymongo/helpers.py", line 114, in _unpack_response
     raise CursorNotFound(msg, 43, errobj)
 CursorNotFound: Cursor not found, cursor id: 138030585475


Long queries in mongo log:

[conn151862] getmore pulp_database.repo_profile_applicability query: { repo_id: "org-Red_Hat_Enterprise_Linux_Server-Red_Hat_Enterprise_Linux_7_Server_-_Extras_RPMs_x86_64" } cursorid:137808660999 ntoreturn:0 keyUpdates:0 numYields:34344 locks(micros) r:919558477 nreturned:28245 reslen:2965745 1031379ms

[conn5947] getmore pulp_database.repo_profile_applicability query: { repo_id: "org-Red_Hat_Enterprise_Linux_Server-Red_Hat_Satellite_Tools_6_2_for_RHEL_7_Server_RPMs_x86_64" } cursorid:137684188271 ntoreturn:0 keyUpdates:0 numYields:37151 locks(micros) r:749309005 nreturned:26975 reslen:2832395 1258027ms


I found that "repo_profile_applicability" doesn't have index for "repo_id". Adding an index for it may be the potential solution.

db.repo_profile_applicability.getIndexes();
[
        {
                "v" : 1,
                "key" : {
                        "_id" : 1
                },
                "name" : "_id_",
                "ns" : "pulp_database.repo_profile_applicability"
        },
        {
                "v" : 1,
                "unique" : true,
                "key" : {
                        "profile_hash" : -1,
                        "repo_id" : -1
                },
                "name" : "profile_hash_-1_repo_id_-1",
                "ns" : "pulp_database.repo_profile_applicability",
                "background" : true
        }
]


Add a 'repo_id' index and compare the "explain()". The result looks much better.

db.repo_profile_applicability.find({ repo_id: "org-Red_Hat_Enterprise_Linux_Server-Red_Hat_Enterprise_Linux_7_Server_RPMs_x86_64_7Server"}).explain();
db.repo_profile_applicability.createIndex({"repo_id": 1}, { background: true })
db.repo_profile_applicability.find({ repo_id: "org-Red_Hat_Enterprise_Linux_Server-Red_Hat_Enterprise_Linux_7_Server_RPMs_x86_64_7Server"}).explain();

Comment 1 pulp-infra@redhat.com 2017-11-13 20:02:18 UTC
The Pulp upstream bug status is at NEW. Updating the external tracker on this bug.

Comment 2 pulp-infra@redhat.com 2017-11-13 20:02:20 UTC
The Pulp upstream bug priority is at Normal. Updating the external tracker on this bug.

Comment 5 pulp-infra@redhat.com 2017-12-06 21:05:54 UTC
The Pulp upstream bug status is at ASSIGNED. Updating the external tracker on this bug.

Comment 6 pulp-infra@redhat.com 2017-12-08 00:32:46 UTC
The Pulp upstream bug status is at POST. Updating the external tracker on this bug.

Comment 7 pulp-infra@redhat.com 2017-12-21 18:02:49 UTC
The Pulp upstream bug status is at MODIFIED. Updating the external tracker on this bug.

Comment 8 pulp-infra@redhat.com 2017-12-21 18:32:17 UTC
All upstream Pulp bugs are at MODIFIED+. Moving this bug to POST.

Comment 9 pulp-infra@redhat.com 2018-01-24 14:31:55 UTC
The Pulp upstream bug status is at ON_QA. Updating the external tracker on this bug.

Comment 10 pulp-infra@redhat.com 2018-01-30 19:31:53 UTC
The Pulp upstream bug status is at CLOSED - CURRENTRELEASE. Updating the external tracker on this bug.

Comment 11 pm-sat@redhat.com 2018-02-21 16:54:17 UTC
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA.
> 
> For information on the advisory, and where to find the updated files, follow the link below.
> 
> If the solution does not work for you, open a new bug report.
> 
> https://access.redhat.com/errata/RHSA-2018:0336