Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1512400 - Repository sync "Actions::Pulp::Repository::RegenerateApplicability" step might fail with "500 Internal Server Error".
Summary: Repository sync "Actions::Pulp::Repository::RegenerateApplicability" step mig...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Satellite 6
Classification: Red Hat
Component: Pulp
Version: 6.2.12
Hardware: Unspecified
OS: Unspecified
unspecified
medium vote
Target Milestone: Unspecified
Assignee: satellite6-bugs
QA Contact: Jitendra Yejare
URL:
Whiteboard:
Depends On:
Blocks: 1122832
TreeView+ depends on / blocked
 
Reported: 2017-11-13 06:27 UTC by Hao Chang Yu
Modified: 2019-04-01 20:27 UTC (History)
10 users (show)

Fixed In Version: pulp-2.13.4.5-1
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-02-21 16:54:17 UTC


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Pulp Redmine 3133 Normal CLOSED - CURRENTRELEASE CursorNotFound while queueing applicability tasks 2018-01-30 19:31:51 UTC

Description Hao Chang Yu 2017-11-13 06:27:49 UTC
Description of problem:
Cursor time out while querying the repository profile applicability with large number of consumers.

Cursor not found, cursor id: 138030585475
 Traceback (most recent call last):
   File "/usr/lib/python2.7/site-packages/django/core/handlers/base.py", line 112, in get_response
     response = wrapped_callback(request, *callback_args, **callback_kwargs)
   File "/usr/lib/python2.7/site-packages/django/views/generic/base.py", line 69, in view
     return self.dispatch(request, *args, **kwargs)
   File "/usr/lib/python2.7/site-packages/django/views/generic/base.py", line 87, in dispatch
     return handler(request, *args, **kwargs)
   File "/usr/lib/python2.7/site-packages/pulp/server/webservices/views/decorators.py", line 241, in _auth_decorator
     return _verify_auth(self, operation, super_user_only, method, *args, **kwargs)
   File "/usr/lib/python2.7/site-packages/pulp/server/webservices/views/decorators.py", line 195, in _verify_auth
     value = method(self, *args, **kwargs)
   File "/usr/lib/python2.7/site-packages/pulp/server/webservices/views/util.py", line 130, in wrapper
     return func(*args, **kwargs)
   File "/usr/lib/python2.7/site-packages/pulp/server/webservices/views/repositories.py", line 913, in post
     queue_regenerate_applicability_for_repos(repo_criteria.as_dict())
   File "/usr/lib/python2.7/site-packages/pulp/server/managers/consumer/applicability.py", line 167, in queue_regenerate_applicability_for_repos
     for batch in paginate(profile_hashes, 10):
   File "/usr/lib/python2.7/site-packages/pulp/plugins/util/misc.py", line 33, in paginate
     page = tuple(itertools.islice(generator, 0, page_size))
   File "/usr/lib/python2.7/site-packages/pulp/plugins/util/misc.py", line 31, in <genexpr>
   File "/usr/lib64/python2.7/site-packages/pymongo/cursor.py", line 1097, in next
     if len(self.__data) or self._refresh():
   File "/usr/lib64/python2.7/site-packages/pymongo/cursor.py", line 1039, in _refresh
   File "/usr/lib64/python2.7/site-packages/pymongo/cursor.py", line 903, in __send_message
     codec_options=self.__codec_options)
   File "/usr/lib64/python2.7/site-packages/pymongo/helpers.py", line 114, in _unpack_response
     raise CursorNotFound(msg, 43, errobj)
 CursorNotFound: Cursor not found, cursor id: 138030585475


Long queries in mongo log:

[conn151862] getmore pulp_database.repo_profile_applicability query: { repo_id: "org-Red_Hat_Enterprise_Linux_Server-Red_Hat_Enterprise_Linux_7_Server_-_Extras_RPMs_x86_64" } cursorid:137808660999 ntoreturn:0 keyUpdates:0 numYields:34344 locks(micros) r:919558477 nreturned:28245 reslen:2965745 1031379ms

[conn5947] getmore pulp_database.repo_profile_applicability query: { repo_id: "org-Red_Hat_Enterprise_Linux_Server-Red_Hat_Satellite_Tools_6_2_for_RHEL_7_Server_RPMs_x86_64" } cursorid:137684188271 ntoreturn:0 keyUpdates:0 numYields:37151 locks(micros) r:749309005 nreturned:26975 reslen:2832395 1258027ms


I found that "repo_profile_applicability" doesn't have index for "repo_id". Adding an index for it may be the potential solution.

db.repo_profile_applicability.getIndexes();
[
        {
                "v" : 1,
                "key" : {
                        "_id" : 1
                },
                "name" : "_id_",
                "ns" : "pulp_database.repo_profile_applicability"
        },
        {
                "v" : 1,
                "unique" : true,
                "key" : {
                        "profile_hash" : -1,
                        "repo_id" : -1
                },
                "name" : "profile_hash_-1_repo_id_-1",
                "ns" : "pulp_database.repo_profile_applicability",
                "background" : true
        }
]


Add a 'repo_id' index and compare the "explain()". The result looks much better.

db.repo_profile_applicability.find({ repo_id: "org-Red_Hat_Enterprise_Linux_Server-Red_Hat_Enterprise_Linux_7_Server_RPMs_x86_64_7Server"}).explain();
db.repo_profile_applicability.createIndex({"repo_id": 1}, { background: true })
db.repo_profile_applicability.find({ repo_id: "org-Red_Hat_Enterprise_Linux_Server-Red_Hat_Enterprise_Linux_7_Server_RPMs_x86_64_7Server"}).explain();

Comment 1 pulp-infra@redhat.com 2017-11-13 20:02:18 UTC
The Pulp upstream bug status is at NEW. Updating the external tracker on this bug.

Comment 2 pulp-infra@redhat.com 2017-11-13 20:02:20 UTC
The Pulp upstream bug priority is at Normal. Updating the external tracker on this bug.

Comment 5 pulp-infra@redhat.com 2017-12-06 21:05:54 UTC
The Pulp upstream bug status is at ASSIGNED. Updating the external tracker on this bug.

Comment 6 pulp-infra@redhat.com 2017-12-08 00:32:46 UTC
The Pulp upstream bug status is at POST. Updating the external tracker on this bug.

Comment 7 pulp-infra@redhat.com 2017-12-21 18:02:49 UTC
The Pulp upstream bug status is at MODIFIED. Updating the external tracker on this bug.

Comment 8 pulp-infra@redhat.com 2017-12-21 18:32:17 UTC
All upstream Pulp bugs are at MODIFIED+. Moving this bug to POST.

Comment 9 pulp-infra@redhat.com 2018-01-24 14:31:55 UTC
The Pulp upstream bug status is at ON_QA. Updating the external tracker on this bug.

Comment 10 pulp-infra@redhat.com 2018-01-30 19:31:53 UTC
The Pulp upstream bug status is at CLOSED - CURRENTRELEASE. Updating the external tracker on this bug.

Comment 11 pm-sat@redhat.com 2018-02-21 16:54:17 UTC
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA.
> 
> For information on the advisory, and where to find the updated files, follow the link below.
> 
> If the solution does not work for you, open a new bug report.
> 
> https://access.redhat.com/errata/RHSA-2018:0336


Note You need to log in before you can comment on or make changes to this bug.