Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 986070 - Pulp causes MongoDB to segfault
Summary: Pulp causes MongoDB to segfault
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Pulp
Classification: Retired
Component: z_other
Version: 2.1.1
Hardware: x86_64
OS: Linux
unspecified
urgent
Target Milestone: ---
: ---
Assignee: Sayli Karmarkar
QA Contact: Preethi Thomas
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2013-07-18 21:58 UTC by Jim
Modified: 2015-03-23 01:11 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2013-07-24 15:52:36 UTC


Attachments (Terms of Use)

Description Jim 2013-07-18 21:58:12 UTC
Description of problem:

Pulp causes mongodb to segfault


Version-Release number of selected component (if applicable):

2.1.3-1


mongodb-2.2.3-4
pymongo-2.1.1-1


How reproducible:


Steps to Reproduce:

# service mongod status
mongod dead but pid file exists
# service mongod restart
Stopping mongod:                                           [FAILED]
Starting mongod:                                           [  OK  ]
# service mongod status
mongod (pid  30930) is running...
# pulp-manage-db 
Beginning database migrations.
Migration package pulp.server.db.migrations is up to date at version 4
Migration package pulp_rpm.migrations is up to date at version 7
Database migrations complete.
Loading content types.
Content types loaded.
# service httpd restart
Stopping httpd:                                            [  OK  ]
Starting httpd:                                            [  OK  ]
# service mongod status
mongod dead but pid file exists

This has only happened since we upgraded to 2.1.3-1 --- it started happening intermittently on our servers, and one of them has gotten into the state wherein it does this every time

Actual results:

from /var/log/mongodb/mongodb.log

Thu Jul 18 16:46:07 [conn1] build index pulp_database.units_yum_repo_metadata_file { data_type: 1 }
Thu Jul 18 16:46:07 [conn1] build index done.  scanned 4 total records. 0 secs
Thu Jul 18 16:46:12 Invalid access at address: 0 from thread: conn3

Thu Jul 18 16:46:12 Got signal: 11 (Segmentation fault).

Thu Jul 18 16:46:12 Backtrace:
0x97d6b6 0x548f8d 0x5492f7 0x3a4140f4a0 
 /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x26) [0x97d6b6]
 /usr/bin/mongod(_ZN5mongo10abruptQuitEi+0x39d) [0x548f8d]
 /usr/bin/mongod(_ZN5mongo24abruptQuitWithAddrSignalEiP7siginfoPv+0x247) [0x5492f7]
 /lib64/libpthread.so.0() [0x3a4140f4a0]



Expected results:

No SEGFAULT

Additional info:

Notice the address that it is trying to access.... 

It looks like a threading problem.

Comment 1 Jim 2013-07-23 18:10:36 UTC
Here are the pulp.log logs from a single instance of this bug:

2013-07-23 13:06:05,070 pulp.server.db.connection:INFO: Attempting Database connection with seeds = localhost:27017
2013-07-23 13:06:05,071 pulp.server.db.connection:INFO: Database connection established with: seeds = localhost:27017, name = pulp_database
2013-07-23 13:06:05,385 pulp.plugins.loader.loading:INFO: Loading config file: /usr/lib/pulp/plugins/distributors/yum_distributor/yum_distributor.conf
2013-07-23 13:06:05,386 pulp.plugins.loader.manager:INFO: Loaded plugin yum_distributor for types: rpm,srpm,drpm,erratum,distribution,package_category,package_group,yum_repo_metadata_file
2013-07-23 13:06:05,388 pulp.plugins.loader.loading:INFO: Loading config file: /usr/lib/pulp/plugins/distributors/iso_distributor/iso_distributor.conf
2013-07-23 13:06:05,388 pulp.plugins.loader.manager:INFO: Loaded plugin export_distributor for types: rpm,srpm,drpm,erratum,distribution,package_category,package_group
2013-07-23 13:06:05,389 pulp.plugins.loader.loading:INFO: Cannot load plugin: Groupdistributor has no module: yum_distributor.groupdistributor
2013-07-23 13:06:05,389 pulp.plugins.loader.loading:INFO: Loading config file: /usr/lib/pulp/plugins/distributors/iso_distributor/iso_distributor.conf
2013-07-23 13:06:05,390 pulp.plugins.loader.manager:INFO: Loaded plugin export_distributor for types: rpm,srpm,drpm,erratum,distribution,package_category,package_group
2013-07-23 13:06:05,391 pulp.plugins.loader.loading:INFO: Cannot load plugin: Groupimporter has no module: yum_importer.groupimporter
2013-07-23 13:06:05,407 pulp.plugins.loader.loading:INFO: Loading config file: /usr/lib/pulp/plugins/importers/yum_importer/yum_importer.conf
2013-07-23 13:06:05,408 pulp.plugins.loader.manager:INFO: Loaded plugin yum_importer for types: distribution,drpm,erratum,package_group,package_category,rpm,srpm,yum_repo_metadata_file
2013-07-23 13:06:05,421 pulp.plugins.loader.manager:INFO: Loaded plugin rpm_pkg_profiler for types: rpm
2013-07-23 13:06:05,423 pulp.plugins.loader.manager:INFO: Loaded plugin rpm_errata_profiler for types: erratum
2013-07-23 13:06:05,428 pulp.plugins.loader.manager:INFO: Loaded plugin nodes_http_distributor for types: node
2013-07-23 13:06:05,431 pulp.plugins.loader.manager:INFO: Loaded plugin iso_distributor for types: iso
2013-07-23 13:06:05,434 pulp.plugins.loader.manager:INFO: Loaded plugin iso_importer for types: iso
2013-07-23 13:06:05,860 pulp.server.db.connection:WARNING: update operation failed on pulp_database.permissions: tries remaining: 2
2013-07-23 13:06:05,860 pulp.server.dispatch.task:ERROR: connection closed
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/pulp/server/dispatch/task.py", line 138, in _run
    result = call(*args, **kwargs)
  File "/usr/lib/python2.6/site-packages/pulp/server/managers/repo/unit_association.py", line 351, in unassociate_by_criteria
    unassociate_units = association_query_manager.get_units(repo_id, criteria=criteria)
  File "/usr/lib/python2.6/site-packages/pulp/server/managers/repo/unit_association_query.py", line 116, in get_units
    return self.get_units_by_type(repo_id, type_id, criteria=criteria)
  File "/usr/lib/python2.6/site-packages/pulp/server/managers/repo/unit_association_query.py", line 331, in get_units_by_type
    units = list(cursor)
  File "/usr/lib64/python2.6/site-packages/pymongo/cursor.py", line 703, in next
    if len(self.__data) or self._refresh():
  File "/usr/lib64/python2.6/site-packages/pymongo/cursor.py", line 666, in _refresh
    self.__uuid_subtype))
  File "/usr/lib64/python2.6/site-packages/pymongo/cursor.py", line 616, in __send_message
    **kwargs)
  File "/usr/lib64/python2.6/site-packages/pymongo/connection.py", line 889, in _send_message_with_response
    raise AutoReconnect(str(e))
AutoReconnect: connection closed
2013-07-23 13:06:05,920 pulp.server.dispatch.task:INFO: FAILURE: Task 69458161-aec6-42bc-94af-60ed3f2ee85a: CallRequest: RepoUnitAssociationManager.unassociate_by_criteria(u'bt_laser', {'unit_sort': None, 
'remove_duplicates': False, 'skip': None, 'association_fields': None, 'association_filters': {}, '_id': ObjectId('51e85dc1ded29455e1000020'), 'id': '51e85dc1ded29455e1000020', 'unit_filters': {'$and': [{'n
ame': {'$regex': 'bt_laser'}}, {'version': {'$regex': '1931'}}, {'release': {'$regex': '2'}}], '_id': {'$in': [u'6f6de4a7-5fd8-49bd-810b-4e8edbc9456a', u'8fccbca1-1f18-4d24-b3f2-1c6786100c24', u'c8233bcf-7c4f-4e71-add0-5bd2d2c9ef99', u'60bcd6a1-3b78-44dc-8b89-d3ac415bdaf8', u'b0b68ffd-c3a2-4ecb-b0d3-71870b24d408', u'2832aa64-64e5-4efd-b0d8-efcba28b025f', u'4030f0ad-449c-4e42-970c-1ead4ea7b6d8', u'442b3953-c
5ae-4721-a1d6-f299dfb49d01', u'bbd21514-8623-430b-b563-6209d14b45d8', u'd350f79f-30cc-4151-8a35-c18b898e7834', u'ca5f87b4-e7fc-4e85-adbb-f26b18e7ade8', u'162f59dd-043c-4127-b041-105bd1735747', u'9532160e-2
c06-41a0-a0e1-fd55fc19f3fd', u'acc7eccc-646e-448d-9fa2-53fca60e1fc7', u'c63ea245-6e9b-4822-99a6-b73c8d537557', u'f03993cd-ad8d-4766-afbf-2e26ba99b400', u'3c6c0481-e43b-482c-b363-ace75d3a58ea', u'8f25b45f-f
5c7-491c-8463-a3e3e4e53872', u'5027dd46-1a94-49be-b14e-f166a6e7e6f7', u'4c262336-45f0-4080-a07e-8154f012a479', u'96872a9b-9a49-48c5-9d5c-1fea6c46fb26', u'edc77973-fff2-4118-bc4f-220d4b6966b2', u'eccb2ac2-1
0bd-4904-95e5-f780777916bb', u'da0de168-089c-4451-99aa-e6108a88428f', u'fb3750d2-a493-4279-9cf7-efb8e763eda8', u'cd9c57c0-9c38-416a-853b-603ffcbb368b', u'd6935cec-a973-48dd-8d68-ac913c2cfcf1', u'2defd4c7-8
5d4-4763-9365-1bf2a857b9c4', u'ace7146e-d003-444b-9ecb-7a71c4577ca9', u'a82d7256-7ff2-4080-9ecf-2360354a9d81', u'65fd97f9-e057-4597-a18f-71ce0abf2bca', u'944672bb-5364-42da-830e-b00a68bdc3c5', u'4b96e374-b003-45df-bd0e-51c07bce2cb5', u'26bcaffc-a911-4df5-bf0f-41f411b06f73', u'651caf6e-c7fb-488f-ac1e-007be35b41bc', u'49a12744-6116-45a2-acff-20d299e9169e', u'5e438169-d913-4b1f-8eb2-e4bd6cf8e17d', u'dc9ce60d-3dbd-413f-9adc-95113767b7bf', u'd0dacb56-776e-4b07-a6c2-f99a32ca874c', u'95a5327d-354d-449d-9342-c1afa35ea5bb', u'c31904e9-6873-4204-8973-a0e7db85de29', u'fb582850-5df5-4f06-bd7c-a61164d8ddae', u'35c4d80c-b1e7-438e-80b2-0f0a0495d733', u'90e3fa09-1744-41b4-9434-fa5d8d63859b', u'51242e5d-12cf-4942-833f-9190e05239c6', u'1aa4135d-0ec2-42c3-8ee0-8f8f9bef723d', u'56119dad-ccd3-4be1-adeb-876ce885530b', u'f92f4feb-da7c-4c8e-b2c7-d32f8dcd5e31', u'679064d5-5ba2-497e-9d1e-31da35d46efe', u'9eb211bf-7a0d-49a7-b34b-28f7e46b3b6f', u'1a579aa7-258c-4073-9b4a-325560f6f7ab', u'0b6c6730-089f-4095-9a4f-4b04ae5530a4', u'c67fc903-d44a-41a4-8b88-c8ddc0ff6f7b', u'ae34f77b-9b37-4422-9b68-7781a3ed59c2', u'56947e90-0ba0-4b87-a28c-f0fec66a1807', u'bd308ade-9226-4e2d-98ba-01610d2f9f20', u'6fb017de-03e3-446b-af43-20fdf507e775', u'18c0ccd1-1c75-4f80-8e11-8efcbed5e580', u'5bb2493d-58e1-4f4d-b5fd-4437309f73d2', u'91dc2350-a5f2-4a94-af40-fef4f8d030d7', u'1beb1214-c640-43d9-8137-879fc011f5f4', u'11512444-f46b-4077-a618-296fe4ed8ab5', u'0e3c8d6f-f6b4-4f6e-8e08-a23aabdbadc8', u'2e282097-9481-4d81-9405-045d1e9bcb78', u'dfb818bf-2181-40b8-ab62-77446f8c7cfc', u'7324b02e-a7cd-446f-8695-2e5fe0f50a38', u'7e067ca5-f528-4ffb-bd40-3ab114f12cff', u'a4b7f358-44b6-4ba3-858c-6a4017783095', u'fd414505-b815-452c-b2fd-b066fc43da90', u'9aa3791e-8be2-4981-b9d9-61d70e3ae7a4', u'76bc8e7a-49ec-4d81-b2ff-82343beb157c', u'831c6956-8af3-4087-b568-352ef8919f72', u'cf662c4a-2eff-423d-acc7-408c2d520878', u'6d61c1f0-f360-4ece-bde7-24aaaa0eabbe', u'092adddf-ac3d-44b7-9751-eb92c9bd06ae', u'9adb998d-d68e-4ad6-8e8a-0c415714c4ca', u'1cc6a494-34cb-4674-ba29-64fe213f0352', u'5e160bc3-852b-4484-afe1-53d89672d11c', u'e7e6294d-2104-4e3b-82bf-0a9625673aba', u'5ca2ae25-db0d-40b5-af71-54e51cf29b12', u'1d95899e-7150-4efe-b5ff-6ff1c17247a1', u'a08b43be-8555-4e4c-8af3-9593b0332142', u'476d1c1c-d568-4a1b-972c-65500237fda6', u'dae10292-e6d4-45fb-8197-5dfbc5a83ddc', u'9f642b97-4586-4a44-95e1-2d949f947bf5', u'20304f3d-a4d5-4149-aeb7-0f9e96a8da92', u'9559fe3b-6de9-4118-ae26-cf3a43181f4a', u'259efef1-68f0-4147-9ce8-88f001863e91', u'4906473e-1920-4709-8c12-72f6166c2ff0', u'27a617de-91ae-4557-880a-26a99e8db401', u'06b6a8fe-adb2-46bd-b638-daa2da1bc294', u'7b99135b-2983-4a57-a2a4-617dc7e2fccc', u'56ca64ba-ae86-4940-905b-c7c716539b1b', u'685f0995-6984-495f-8592-195dfef5c2e7', u'3a31d599-9905-434d-bd08-dda983bd8958', u'5481666c-40ab-40b4-b76b-5e98d3f0f7b6', u'cf08a540-b66a-4956-b499-1b5746687722', u'd43d372a-39b6-4221-9410-a8ba1a45170f', u'7aca9e21-8f4c-425c-9df3-afb9174f13f2', u'9b29aa7b-5368-44bb-a4c1-02a7a006f0f0', u'e303ebe5-38f1-4256-a38d-8bf36e9109e9', u'55c7826f-0c63-4221-aa92-a1985eb6271d', u'df4255b2-9a51-4257-b87a-27b18f288a4f', u'ce7e9a34-067e-4998-9246-4908c859b9e5', u'26739561-12f5-4fdc-83c6-f04730486615', u'99d2fd90-405d-4365-a5be-f592388a4f8b', u'4f189005-94cc-477c-8bdc-0547e0100dfd', u'2054ae7c-b6c8-4865-b3a5-dfff3ac0cc5e', u'eca11179-a9e4-4f4d-afcf-468fbf5839a3', u'fda3b488-eb52-473c-96f3-67b3bf09a2b6', u'87b08f41-1fa0-4e40-9b47-1ec3595e1c4f', u'c1ff516d-3751-4be6-a217-8272e00ab751', u'd98d06c4-420d-4446-9f2a-fb8be1ff4abb', u'02947069-fe93-4715-85a1-052ead880cd7', u'14dfe251-9ee3-4273-953f-c9f1902d618c', u'b946cdfa-a8fd-44b9-9ca0-05149f640b3d', u'cc43fd84-4da4-43eb-a91e-c4c5eb353956', u'ab885465-b745-4197-827b-010c9d2368b9', u'20e30159-d571-428b-af02-50ecb940b9e6', u'ed50c420-058a-4d96-8e5f-3e18c85d389d', u'a0f459bc-08cc-49bf-af86-12333d788714', u'ad1908a2-b3e7-411a-a6f7-e7da3985c196', u'92f2280b-707f-4948-9f02-e80f327e4cf4', u'9bc0bd2f-76e4-4151-ab72-17630fd355aa', u'cfd0ecd4-4fbc-4c4d-9047-7e0c846679be', u'7936f61f-3c80-4cc4-b939-8ff8d8a15508', u'7853cd6d-2374-421e-8589-f5abddece470', u'2a074ee4-8e56-436c-9134-ddf22f966b89', u'6366960d-8925-417e-8956-ff075263d546', u'a800db20-a90d-4c8e-83e7-565a6de7769b', u'768a42db-bfde-4976-85c0-31c2d0f34622', u'7fab452c-b8b7-4324-97ca-a0560b9263cc', u'9da7e836-b000-405f-80fd-f56fa47c994c', u'5e207563-2126-4197-a961-a13aa15230ab', u'5e0c3a1f-de1d-4a92-ad69-cb362f2c8b3d', u'b0476e52-91e4-46e7-b9a2-900a32d616f5', u'fdea5723-cfcd-4224-a1e0-6bd0190b5b5d', u'8bdcea2e-7abf-46b7-bdd6-35495d5682a8', u'e128aeca-fdec-49b2-95a5-b6e0d3973930', u'44152ff6-0849-46d9-b6df-e4542c385b8e', u'2fe04e45-27cb-4357-bda3-b6c38b1c670c', u'89f41bbe-1045-4319-b429-486561f4e490', u'2002eda5-88b9-44b2-8436-54f639b1f03d', u'75c07d98-f9b5-40fb-82d9-98c81740b10a', u'27351522-598b-4055-88f9-af4b05148180', u'd1404bc4-5f46-4c31-bf99-dcd755044bcb', u'5ea4a9bd-fc22-4a93-8a90-597bc07f3183', u'4ef5a4db-380c-42b9-b9f7-8ed2d690caa7', u'ae8a1f3a-b88d-4679-9cd2-4f4a10990494', u'6e5d2480-80b8-4cdd-8b28-5c76b4526ede', u'ed871c4e-b679-4ac2-834f-081a4b29384d', u'a5ab2f5d-68aa-4d9c-b80e-94d49f081933', u'e3355abc-6bad-4add-af0a-2748f4bdcb6b', u'56fab946-5241-41de-a4b2-7277cf031da1', u'dd21e395-37d1-45d7-9ed4-e0ef40b9dd55', u'797afa0a-6334-4299-a96e-d99a567aa8ca', u'38bce226-6ac4-4e7a-b0cc-2faf2d0e1917', u'6d2fc565-c3d7-4e25-bbd6-03249c334404', u'396b507a-7143-4ada-983d-4f165e5bc34b', u'8c4a004f-3162-4fb0-a3ca-2f981d438304', u'0ae93500-b3ba-4466-b774-c29e0074f2fa', u'18f3def9-aee9-403e-8524-0087a6e78bcd', u'ba56facb-4b24-4487-9123-62249bb3f080', u'610bd2e0-6265-40e5-8de3-d0ea301a9d75', u'8721c5e1-1946-46fa-93d1-7004d612ad43', u'94d43b75-24de-41ee-97d2-e5118d996ff7', u'274cd031-4713-4f41-ae70-f65772db800d', u'2eac516a-1b22-4dae-9561-28bde94c5ed0', u'e563ba57-a6e2-4ab2-8a22-7641c1e26219', u'9bb37188-9f86-4f43-8d52-2f9e71f55004', u'70c6d66b-7d9e-457d-aeb9-42fc82fd52f1', u'f2160315-df71-4fe7-93d4-b0ee20b9223c', u'd3d04681-de98-4953-b7f3-86fe15b519b1', u'a42a3713-4911-4e4c-a3ef-7a335a876459', u'dea33e81-511b-4830-9bdc-7f84a2d5130c', u'952bcb1a-66dd-44c1-be8f-b7744d1f06d4', u'a95861c7-3488-48f2-ae86-eac01f56dd6c', u'977709fb-cced-47a9-844e-4750bc63aea9', u'6a3912fb-4e44-408c-a366-ca32c14d669b', u'f1e6cfab-2a13-4956-8db6-07475b36907d', u'ebefd2aa-924a-4feb-ba97-e1ac19d3f0c4', u'4a5ad562-5410-4fd6-942c-e45ff8a72e61', u'4e113d0f-aa9a-4535-acfb-52080565bf54', u'd69f14d0-224c-4d15-a6dc-711a12f15a0e', u'175c5fec-5f5a-4c29-a7eb-ba8af145756c', u'81be605e-8f7d-4a32-bf6f-5b9479418ca9', u'bb8f589b-1c5d-4e3c-94e1-9daa4d2db11f', u'5990a3cb-4002-4743-a0af-4b1dff80852f', u'46bfc2eb-f581-4a0a-beb5-f01ed90deab0', u'5d157bb3-5104-4119-9128-6b85431c02d0', u'b0e9fcf1-f395-4100-91f7-03596c1fd2f0', u'eea31269-50f8-47fa-ad27-2bd09f84ef95', u'6a25475e-d004-4e6c-94f8-d976926a12f6', u'4ab5a0a2-342e-4b2a-879a-5f7419d636d9', u'559ce8cb-bc6d-45c4-8ab2-48798f46f662', u'bff65156-0202-49e0-9e8e-1debb476c06c', u'814f98bd-028c-403f-baba-ada253d647c6', u'6c145392-2a7a-4787-89d2-5362593ee0fc', u'6998116d-9607-4a14-b2ed-0c67f6327a79', u'82c6f3c2-0fac-4a9c-9391-5432ce6c0ff6', u'fb659adf-386a-4124-8825-0f93e0a27ded', u'34c3de92-adb8-4f1f-8429-c5f6a9a78d25', u'a13402be-b05b-4688-8499-a6551c6e75c6', u'a21ec383-1f35-4393-9132-c4024462952e', u'b84a80b1-72c1-4a37-8155-91dd6bfda3f8', u'a8820e07-7579-4acb-8451-c478cd393302', u'48e06204-482e-444d-b74d-101ad83af1f6', u'f965d577-01d9-421c-95ae-ad0d7194aaa4', u'113d7961-af88-4069-a8b4-779af7753d9b', u'1b305b37-fcab-4583-8bac-80b6707b6f0d', u'4e615577-4b78-4cb9-be20-61cb52f558ba', u'2f1da237-44db-4f6c-b854-90f296d94de9', u'2c7e6c0e-a422-4707-9f48-64b5f376d93e', u'1c7f26b2-1347-458a-a6f5-fb861bf27e6a', u'322836e3-3f74-436e-bf3b-c3869760c8de', u'3dd2bf18-c71e-426c-b4a9-1d6562d4e39e', u'b40146d6-41f0-4484-b66a-4a98086da96e', u'066e1fda-b601-4556-b131-2155d841b09f', u'310e1e32-0b23-46b0-99d7-9f05983304d4', u'5f27a8f4-12ed-460d-8b36-763ac9d5955e', u'a0f88873-e7ec-4e55-8c97-943e7619b772', u'38209a1d-6a00-40ec-809c-613d866343cb', u'3e4fa0a1-8dd8-4b6e-bf40-78e4cb0caea5', u'3434a0d0-767c-428d-b9a1-9f62365d04f4', u'8fff0c17-0d2a-4262-aa8c-f829029c2739', u'7cdcb317-5d76-401f-8a96-3d14b2a1a0ee', u'75bb558f-eb92-460d-93c9-90b187774727', u'e1c927f7-32e0-4c3e-9701-06ea7777ebe6', u'82bd04b4-7357-46bd-b50c-76b7c030060b', u'211ffec3-4834-471e-a274-02b964cb8c6d', u'a17d5393-33ca-44d0-a3c0-eb5da33ec05c', u'f85bb8d7-df88-4eb9-af48-76d6e38ac880', u'999fc680-3a5f-4c28-a0b1-8cf768543b2c', u'e0154386-1301-4e80-a865-b6430b13f690', u'db52b6c7-d7ed-48fa-a05c-29a4f50b4397', u'193f5acb-1975-46b1-9d0a-6a9bff3a8fb3', u'628d13b8-f03e-4bcc-926e-36d61a4a36da', u'8cb6ed10-fe29-49a4-9982-6050bbec2733', u'aad0acb4-900f-450b-a16d-6bfa73081920', u'43cf5cce-c2d2-4ef1-883b-b2999c36dad5', u'bf9eaec8-0c02-4134-baa8-279b25d6f03b', u'c746fd59-2df3-4c41-81fc-b477c2db1069', u'ae48f3a0-7bf5-461a-a79c-cfd2fa9c67a6', u'15e859c4-58ba-45b1-a6cb-70841b4e2b20', u'8cea2d89-3ab7-4148-be81-413da53ab677', u'f6a5baa0-aef2-4594-b033-2fb7645db7ce', u'0b8b44e1-541d-49e2-8171-296671e14287', u'dbd16178-1fe7-437e-8ba4-b21caf400fef', u'ceba9fa2-6572-4c60-b676-2111c0579ece', u'e763a476-126d-4393-becb-a6d5d4db769a', u'57b29f57-c362-4f09-ba2f-29d290c04fda', u'5da077e1-4379-4288-bb3e-c9ceafbec853', u'355b7af6-0471-472d-b001-6d9fe0be16f6', u'50d1b1fa-0b1c-4ee9-ae4b-0903f13ea01d', u'6610a878-04dc-44ea-b8a0-4abfa166d177', u'3cb490c5-230c-4b15-88f5-da72318c2209', u'a0905a9e-a6e7-4ecd-9859-db511e10591e', u'3a7068bd-ca9f-4cb3-b16f-cf97e205e829', u'ffd23479-7ddd-47e9-9a42-0700c20df20e', u'513bea1b-2bd0-4b2c-8a08-acfe38d7afd3', u'4737f1db-f95c-4032-89f8-c308564d42e3', u'377c2400-4b49-4631-acd2-793856b54c21', u'36207071-9132-42a0-bcb7-967737c000e8', u'41f5f48e-27a2-4f0d-93e8-4937384ca537', u'476ae611-8bab-45b5-8206-1897b3aa19ae', u'bddb9c46-2d15-484b-a8bc-9c420b6f3910', u'd1092dd9-48ab-4767-aff4-27f24c2a1177', u'942c1642-1941-43c2-b4f4-eca17a9eec29', u'c65cdf6a-f04f-4f07-a717-d1349325a6dc', u'c62fd484-558f-492b-a18d-35025d55c7c4', u'486a0844-4a95-4e77-b720-d6754aadf8c0', u'3195b352-8a38-4abb-b34b-13a710ba17b1', u'963aaa8a-13a2-4d12-93ce-db677b8a0406', u'46a16e74-4155-4d93-a1ec-e61b891e9380', u'12da86da-2082-4048-bb3e-50f619af34db', u'b4371597-8fef-4384-8931-626ddb1c808a', u'f471732f-73b3-48f7-96f2-32f338a60389', u'e257a292-2dc2-475f-96c1-fdea241f1734', u'5365b100-a54f-4b84-bb8e-4a7f81685dd7', u'3cbfedfb-74ea-4e3f-a76b-1cce81346b9c', u'0b57cc35-cfc9-49c3-8fe1-7ce086bfb915', u'6f54491e-f284-407f-9f7e-fb990970b8a5', u'cb8ed5ed-1510-4280-9163-e4208edcdd04', u'485b9ea4-c785-4aa5-874e-75d1712bf8d6', u'b530b320-a25e-465a-b1b0-a6f9a1b097b5', u'fc0a208f-54e2-4d87-b851-2f9aeaca7cb6', u'438cd708-d177-4685-9177-be1e05bb2a8f', u'ba71e5a6-bd6d-4298-8acb-2e590bca3364', u'575239d5-dccf-4e00-bb1a-a6d73c17d5f3', u'eb740544-f06d-4f69-823f-622c7f4438b9', u'f18bcd51-bf83-4f9f-a34f-f06c439fb2b5', u'7a7e8a65-ae58-4f5f-8f1a-09090373a9fb', u'133caaf8-7a04-4d5d-beb2-5241221d5ec6', u'9e3e44a7-45af-4153-b766-dc327bf7e134', u'8dd46a53-3e57-481a-89c9-b87724280f00', u'1701c3ec-adfe-4dc6-af9c-65996225eb60', u'9f497889-cebb-4946-91e0-1405a011ad90']}}, 'association_sort': None, 'unit_fields': None, 'limit': None, 'type_ids': ['rpm']}, 'user', u'admin')
2013-07-23 13:06:06,161 pulp.server.db.connection:WARNING: update operation failed on pulp_database.permissions: tries remaining: 1
2013-07-23 13:06:06,463 pulp.server.db.connection:WARNING: update operation failed on pulp_database.permissions: tries remaining: 0
2013-07-23 13:06:06,463 pulp.server.webservices.application:CRITICAL: *************************************************************
2013-07-23 13:06:06,464 pulp.server.db.connection:WARNING: remove operation failed on pulp_database.queued_calls: tries remaining: 2
2013-07-23 13:06:06,464 pulp.server.webservices.application:ERROR: The Pulp server encountered an unexpected failure during initialization
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/pulp/server/webservices/application.py", line 177, in wsgi_application
    _initialize_pulp()
  File "/usr/lib/python2.6/site-packages/pulp/server/webservices/application.py", line 132, in _initialize_pulp
    dispatch_factory.initialize()
  File "/usr/lib/python2.6/site-packages/pulp/server/dispatch/factory.py", line 58, in initialize
    _initialize_coordinator()
  File "/usr/lib/python2.6/site-packages/pulp/server/dispatch/factory.py", line 31, in _initialize_coordinator
    _COORDINATOR.start()
  File "/usr/lib/python2.6/site-packages/pulp/server/dispatch/coordinator.py", line 80, in start
    self.execute_call_asynchronously(call_request)
  File "/usr/lib/python2.6/site-packages/pulp/server/dispatch/coordinator.py", line 156, in execute_call_asynchronously
    self._process_tasks([task])
  File "/usr/lib/python2.6/site-packages/pulp/server/dispatch/coordinator.py", line 251, in _process_tasks
    task_queue.enqueue(task)
  File "/usr/lib/python2.6/site-packages/pulp/server/dispatch/taskqueue.py", line 209, in enqueue
    task.call_life_cycle_callbacks(dispatch_constants.CALL_ENQUEUE_LIFE_CYCLE_CALLBACK)
  File "/usr/lib/python2.6/site-packages/pulp/server/dispatch/task.py", line 229, in call_life_cycle_callbacks
    hook(self.call_request, self.call_report)
  File "/usr/lib/python2.6/site-packages/pulp/server/auth/authorization.py", line 163, in __call__
    factory.permission_manager().grant(resource, self.user_name, operations)
  File "/usr/lib/python2.6/site-packages/pulp/server/managers/auth/permission/cud.py", line 158, in grant
    Permission.get_collection().save(permission, safe=True)
  File "/usr/lib/python2.6/site-packages/pulp/server/db/connection.py", line 80, in retry
    return method(*args, **kwargs)
  File "/usr/lib64/python2.6/site-packages/pymongo/collection.py", line 237, in save
    manipulate, safe, _check_keys=True, **kwargs)
  File "/usr/lib/python2.6/site-packages/pulp/server/db/connection.py", line 89, in retry
    (method.__name__, self.full_name, (self.retries + 1)))
PulpCollectionFailure: update operation failed on pulp_database.permissions: database connection still down after 3 tries
2013-07-23 13:06:06,757 pulp.server.webservices.application:CRITICAL: *************************************************************
2013-07-23 13:06:06,765 pulp.server.db.connection:WARNING: remove operation failed on pulp_database.queued_calls: tries remaining: 1
2013-07-23 13:06:07,067 pulp.server.db.connection:WARNING: remove operation failed on pulp_database.queued_calls: tries remaining: 0
2013-07-23 13:06:07,067 pulp.server.dispatch.task:ERROR: remove operation failed on pulp_database.queued_calls: database connection still down after 3 tries
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/pulp/server/dispatch/task.py", line 214, in _call_complete_callback
    self.complete_callback(self)
  File "/usr/lib/python2.6/site-packages/pulp/server/dispatch/taskqueue.py", line 292, in _complete
    self.dequeue(task)
  File "/usr/lib/python2.6/site-packages/pulp/server/dispatch/taskqueue.py", line 244, in dequeue
    self.queued_call_collection.remove({'_id': task.queued_call_id}, safe=True)
  File "/usr/lib/python2.6/site-packages/pulp/server/db/connection.py", line 89, in retry
    (method.__name__, self.full_name, (self.retries + 1)))
PulpCollectionFailure: remove operation failed on pulp_database.queued_calls: database connection still down after 3 tries
2013-07-23 13:06:07,070 pulp.server.db.connection:WARNING: create_index operation failed on pulp_database.archived_calls: tries remaining: 2
2013-07-23 13:06:07,372 pulp.server.db.connection:WARNING: create_index operation failed on pulp_database.archived_calls: tries remaining: 1
2013-07-23 13:06:07,673 pulp.server.db.connection:WARNING: create_index operation failed on pulp_database.archived_calls: tries remaining: 0

Comment 2 Sayli Karmarkar 2013-07-24 15:52:36 UTC
Not a pulp bug.

Comment 3 Jim 2013-07-25 15:11:04 UTC
For posterity, it wound up being this bug: https://jira.mongodb.org/browse/SERVER-9014


Note You need to log in before you can comment on or make changes to this bug.