Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1367758 - [Sat6.2 upgrade] pulp-manage-db creates duplicate records in units_ostree
Summary: [Sat6.2 upgrade] pulp-manage-db creates duplicate records in units_ostree
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Red Hat Satellite 6
Classification: Red Hat
Component: Pulp
Version: 6.2.0
Hardware: x86_64
OS: Linux
unspecified
high vote
Target Milestone: Unspecified
Assignee: satellite6-bugs
QA Contact: Katello QA List
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-08-17 12:00 UTC by Pavel Moravec
Modified: 2017-04-25 16:23 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-08-18 20:32:30 UTC


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Bugzilla 1374723 None CLOSED [Improvement] prevent running pulp-manage-db when pulp workers are up 2019-03-19 23:41:53 UTC
Red Hat Knowledge Base (Solution) 2534911 None None None 2016-08-18 20:32:15 UTC

Internal Links: 1374723

Description Pavel Moravec 2016-08-17 12:00:37 UTC
Description of problem:
Upgrade of Satellite to 6.2 fails on step pulp-manage-db with backtrace suggesting units_ostree collection violates unique key/index.

Manually removing duplicates (and references to them) and re-running just pulp-manage-db fails the same way.


Version-Release number of selected component (if applicable):
Sat 6.1.9 -> 6.2 upgrade
pulp-server 2.6.0.21-1 -> 2.8.3.3-1 upgrade


How reproducible:
??? I guess 100%


Steps to Reproduce:
n.a. at the moment, I guess it is enough to sync an ostree unit in 6.1, put to a repo, publish the repo, and upgrade to 6.2


Actual results:
upgrade fails with:

E11000 duplicate key error index: pulp_database.units_ostree.$remote_id_1_branch_1_commit_1  dup key: { : "174b6356ebe246768ac1600709cfefc6589a43503397a9e7589111d4da9188f6", : "rhel-atomic-host/7/x86_64/standard", : "b672bf8a457cb28e003dee20c53749636ef5fce3e4743afe4aaad269d3aaa62a" }
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/pulp/server/db/manage.py", line 193, in main
    return _auto_manage_db(options)
  File "/usr/lib/python2.7/site-packages/pulp/server/db/manage.py", line 256, in _auto_manage_db
    migrate_database(options)
  File "/usr/lib/python2.7/site-packages/pulp/server/db/manage.py", line 142, in migrate_database
    ensure_database_indexes()
  File "/usr/lib/python2.7/site-packages/pulp/server/db/manage.py", line 179, in ensure_database_indexes
    model_class.ensure_indexes()
  File "/usr/lib/python2.7/site-packages/mongoengine/document.py", line 738, in ensure_indexes
    collection = cls._get_collection()
  File "/usr/lib/python2.7/site-packages/mongoengine/document.py", line 210, in _get_collection
    cls.ensure_indexes()
  File "/usr/lib/python2.7/site-packages/mongoengine/document.py", line 766, in ensure_indexes
    collection.create_index(fields, background=background, **opts)
  File "/usr/lib64/python2.7/site-packages/pymongo/collection.py", line 1380, in create_index
    self.__create_index(keys, kwargs)
  File "/usr/lib64/python2.7/site-packages/pymongo/collection.py", line 1290, in __create_index
    sock_info, cmd, read_preference=ReadPreference.PRIMARY)
  File "/usr/lib64/python2.7/site-packages/pymongo/collection.py", line 205, in _command
    read_concern=read_concern)
  File "/usr/lib64/python2.7/site-packages/pymongo/pool.py", line 211, in command
    read_concern)
  File "/usr/lib64/python2.7/site-packages/pymongo/network.py", line 100, in command
    helpers._check_command_response(response_doc, msg, allowable_errors)
  File "/usr/lib64/python2.7/site-packages/pymongo/helpers.py", line 189, in _check_command_response
    raise DuplicateKeyError(errmsg, code, response)
DuplicateKeyError: E11000 duplicate key error index: pulp_database.units_ostree.$remote_id_1_branch_1_commit_1  dup key: { : "174b6356ebe246768ac1600709cfefc6589a43503397a9e7589111d4da9188f6", : "rhel-atomic-host/7/x86_64/standard", : "b672bf8a457cb28e003dee20c53749636ef5fce3e4743afe4aaad269d3aaa62a" }


Expected results:
no such fail


Additional info:
- as written, removing duplicates from units_ostree and references to them from repo_content_units collection, pulp-manage-db fails the same way
- foreman-debug to be attached

Comment 3 Pavel Moravec 2016-08-17 19:45:23 UTC
Update: what I understood, the upgrade itself does not cause the duplicates. Rather they exist before it. But after the upgrade, new unique key-index is added to units_ostree collection.

Steps to resolve/workaround:
- grab mongo collections units_ostree, repo_content_units and lazy_content_catalog (or some other that can refer to units_ostree, if there is)
- identify the duplicates in units_ostree - mark one representant in each set of duplicates
- in other collected tables, update them to replace duplicates by their representants. Such that no reference to the dups exists.
- remove the dups from units_ostree

Comment 5 Pavel Moravec 2016-08-18 20:32:16 UTC
This was most probably caused by running pulp-manage-db with pulp workers running. If some worker adds an OSTree unit that time, when uniqueness constrains may not be enforced, the unique index can get broken.

Closing as not a bug, created KCS 2534911 how to workaround / fix such dups created.


Note You need to log in before you can comment on or make changes to this bug.