Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1366798 - Manifest fails to import.
Summary: Manifest fails to import.
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Satellite 6
Classification: Red Hat
Component: Subscription Management
Version: 6.2.0
Hardware: Unspecified
OS: Unspecified
unspecified
high vote
Target Milestone: 6.2
Assignee: Justin Sherrill
QA Contact: jcallaha
URL:
Whiteboard:
Depends On: 1369184
Blocks: CEE_Sat6_Top_BZs, GSS_Sat6_Top_Bugs 1426388
TreeView+ depends on / blocked
 
Reported: 2016-08-12 20:52 UTC by Dylan Gross
Modified: 2018-12-06 20:45 UTC (History)
13 users (show)

Fixed In Version: candlepin-0.9.54.9-1 tfm-rubygem-katello-3.0.0.115-1
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1369184 1370205 1426388 (view as bug list)
Environment:
Last Closed: 2017-05-01 14:28:02 UTC


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2016:1993 normal SHIPPED_LIVE Satellite 6.2.2 Async Bug Release 2016-10-04 10:43:20 UTC
Github candlepin candlepin pull 1329 None None None 2016-08-22 15:03:18 UTC
Foreman Issue Tracker 16978 None None None 2016-10-17 22:48:10 UTC

Description Dylan Gross 2016-08-12 20:52:09 UTC
Description of problem:

   Importing of manifest via webui or hammer fails

Version-Release number of selected component (if applicable):

   Red Hat Satellite 6.2.0

How reproducible:

   Unknown

Steps to Reproduce:

   Customer encountered when removing manifest (successfully),  and then attempting to re-attach the manifest.

Actual results:

WebUI screen will spin indefinitely.
The /var/log/candlepin/candlepin.log generates a long trace beginning with ...

2016-08-08 15:10:01,128 [thread=http-bio-8443-exec-4] [req=9156e50d-852d-4256-b49e-8493bd9f1978, org=OrgName] INFO  org.candlepin.resource.OwnerResource - Import attempt completed for owner OrgName
2016-08-08 15:10:01,135 [thread=http-bio-8443-exec-4] [req=9156e50d-852d-4256-b49e-8493bd9f1978, org=OrgName] ERROR org.candlepin.common.exceptions.mappers.CandlepinExceptionMapper - Runtime Error query did not return a unique result: 2 at org.hibernate.internal.AbstractQueryImpl.uniqueElement:914
org.hibernate.NonUniqueResultException: query did not return a unique result: 2
        at org.hibernate.internal.AbstractQueryImpl.uniqueElement(AbstractQueryImpl.java:914) ~[hibernate-core-4.2.5.Final.jar:4.2.5.Final]
        at org.hibernate.internal.CriteriaImpl.uniqueResult(CriteriaImpl.java:396) ~[hibernate-core-4.2.5.Final.jar:4.2.5.Final]
        at org.candlepin.model.PoolCurator.findUeberPool(PoolCurator.java:334) ~[PoolCurator.class:na]
        at com.google.inject.persist.jpa.JpaLocalTxnInterceptor.invoke(JpaLocalTxnInterceptor.java:58) ~[guice-persist-3.0.jar:na]
        at org.candlepin.controller.CandlepinPoolManager.findUeberPool(CandlepinPoolManager.java:1640) ~[CandlepinPoolManager.class:na]
        at org.candlepin.controller.CandlepinPoolManager.refreshPoolsWithRegeneration(CandlepinPoolManager.java:173) ~[CandlepinPoolManager.class:na]
        at org.candlepin.controller.Refresher.run(Refresher.java:110) ~[Refresher.class:na]
        at org.candlepin.sync.Importer.importObjects(Importer.java:450) ~[Importer.class:na]
        at com.google.inject.persist.jpa.JpaLocalTxnInterceptor.invoke(JpaLocalTxnInterceptor.java:66) ~[guice-persist-3.0.jar:na]
        at org.candlepin.sync.Importer.loadExport(Importer.java:291) ~[Importer.class:na]

Expected results:

   The manifest imports successfully.

Additional info:

   Note, while the actual error in the log complains about a non-unique result similar to BZ#1337833, the trace actually begins to differ around the 3rd line when it calls "org.candlepin.model.PoolCurator.findUeberPool(PoolCurator.java:334) ~[PoolCurator.class:na]:

Comment 5 Michael Stead 2016-08-17 12:33:16 UTC
The error was due to bad debug certificate data that was caused by an old debug cert download bug. Right now, I'm not sure how the customer got into this state. The data doesn't appear to be in a state that I've seen in other reported bug. More investigation will be required to nail down what happened.


WORK AROUND
===========
Running the following SQL will clean up the bad data and import should start functioning again.

NOTE: Debug certificates, if in use, should be manually regenerated per org.


SQL TO CLEAN OUT ALL CANDLEPIN DEBUG CERT DATA FOR ALL ORGs
===========================================================
delete from cp_subscription where id in (select s.id from cp_subscription s inner join cp_product p on s.product_id = p.id left outer join cp_pool pool on pool.productid = p.id where p.name LIKE '%_ueber_product');

delete from cp_ent_certificate where entitlement_id in (select e.id from cp_entitlement e inner join cp_pool p on e.pool_id = p.id inner join cp_product prod on prod.id = p.productid where prod.name like '%ueber%');

delete from cp_entitlement where pool_id in (select p.id from cp_pool p inner join cp_product prod on prod.id = p.productid where prod.name like '%ueber%');

delete from cp_pool where id in (select p.id from cp_pool p inner join cp_product prod on prod.id = p.productid where prod.name like '%ueber%');

delete from cp_content where id in (select c.id from cp_content c inner join cp_product_content pc on pc.content_id=c.id inner join cp_product p on p.id=pc.product_id left outer join cp_pool pool on pool.productid = p.id where p.name LIKE '%_ueber_product');

delete from cp_product where name like '%_ueber_product';

delete from cp_consumer where name like '%ueber%';

Comment 7 Chris Duryee 2016-08-17 13:09:39 UTC
I confirmed that if the ueber cert is removed or updated, katello should figure things out and automatically fix affected capsule repositories upon next cpasule sync without user input.

The instructions in #5 should do the trick for workaround.

Comment 12 jcallaha 2016-10-03 00:19:55 UTC
Verified in Satellite 6.2.2 Async

Unable to reproduce this issue with the latest version of satellite.
Started with two systems, one using default, the other using custom certs. Upgraded the systems and didn't encounter the issue after 10 iterations of downloading the ueber certificates and deleting/uploading/refreshing the manifests.

Comment 14 errata-xmlrpc 2016-10-04 06:44:45 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2016:1993

Comment 22 Bryan Kearney 2016-10-18 00:14:24 UTC
Upstream bug assigned to jsherril@redhat.com

Comment 23 Bryan Kearney 2016-10-18 00:14:28 UTC
Upstream bug assigned to jsherril@redhat.com

Comment 24 Bryan Kearney 2016-11-01 18:13:01 UTC
Moving this bug to POST for triage into Satellite 6 since the upstream issue http://projects.theforeman.org/issues/16978 has been resolved.

Comment 26 pm-sat@redhat.com 2017-02-23 21:08:59 UTC
Please add verifications steps for this bug to help QE verify

Comment 28 jcallaha 2017-04-06 18:03:01 UTC
Verified in Satellite 6.2.9 Snap 2

Created a new organization called 'testerr' and monitored the candlepin logs from that period. Here are the ueber specific logs

2017-04-06 13:47:15,367 [thread=http-8443-3] [req=b118cd76-6b65-42e3-a8bf-65bcd66bad8a, org=] INFO  org.candlepin.common.filter.LoggingFilter - Request: verb=GET, uri=/candlepin/owners/testerr/uebercert

2017-04-06 13:47:15,456 [thread=http-8443-3] [req=ac806b48-cc82-41a9-8299-9276d9d27703, org=] INFO  org.candlepin.common.filter.LoggingFilter - Request: verb=POST, uri=/candlepin/owners/testerr/uebercert

2017-04-06 13:47:15,523 [thread=http-8443-3] [req=ac806b48-cc82-41a9-8299-9276d9d27703, org=testerr] INFO  org.candlepin.policy.js.pool.PoolRules - Creating new master pool: Pool<type=NORMAL, product=14915008354908818526412, productName=testerr_ueber_product, id=null, quantity=1>


full logs from that period

2017-04-06 13:47:14,208 [thread=http-8443-3] [req=c01a99ad-5e81-447f-805d-522077470494, org=] INFO  org.candlepin.common.filter.LoggingFilter - Request: verb=GET, uri=/candlepin/status
2017-04-06 13:47:14,212 [thread=http-8443-3] [req=c01a99ad-5e81-447f-805d-522077470494, org=] INFO  org.candlepin.common.filter.LoggingFilter - Response: status=200, content-type="application/json", time=4
2017-04-06 13:47:14,262 [thread=http-8443-3] [req=4105d2af-42cd-46a4-b582-dbb66fdbe268, org=] INFO  org.candlepin.common.filter.LoggingFilter - Request: verb=GET, uri=/candlepin/status
2017-04-06 13:47:14,266 [thread=http-8443-3] [req=4105d2af-42cd-46a4-b582-dbb66fdbe268, org=] INFO  org.candlepin.common.filter.LoggingFilter - Response: status=200, content-type="application/json", time=4
2017-04-06 13:47:15,101 [thread=http-8443-3] [req=35b594e3-1ada-4c06-91a9-1e5d0d6c694d, org=] INFO  org.candlepin.common.filter.LoggingFilter - Request: verb=POST, uri=/candlepin/owners/
2017-04-06 13:47:15,133 [thread=http-8443-3] [req=35b594e3-1ada-4c06-91a9-1e5d0d6c694d, org=] INFO  org.candlepin.resource.OwnerResource - Created owner: Owner [id: 8a90848f5b3f84cd015b4460a133049f, key: testerr]
2017-04-06 13:47:15,134 [thread=http-8443-3] [req=35b594e3-1ada-4c06-91a9-1e5d0d6c694d, org=] INFO  org.candlepin.audit.HornetqEventDispatcher - Sending event: Event [id=null, target=OWNER, type=CREATED, time=Thu Apr 06 13:47:15 EDT 2017, entity=8a90848f5b3f84cd015b4460a133049f]
2017-04-06 13:47:15,140 [thread=http-8443-3] [req=35b594e3-1ada-4c06-91a9-1e5d0d6c694d, org=] INFO  org.candlepin.common.filter.LoggingFilter - Response: status=200, content-type="application/json", time=39
2017-04-06 13:47:15,235 [thread=http-8443-3] [req=2dbfc60d-4a07-441b-8f81-ed2cfd00123f, org=] INFO  org.candlepin.common.filter.LoggingFilter - Request: verb=POST, uri=/candlepin/owners/testerr/environments
2017-04-06 13:47:15,284 [thread=http-8443-3] [req=2dbfc60d-4a07-441b-8f81-ed2cfd00123f, org=testerr] INFO  org.candlepin.common.filter.LoggingFilter - Response: status=200, content-type="application/json", time=49
2017-04-06 13:47:15,367 [thread=http-8443-3] [req=b118cd76-6b65-42e3-a8bf-65bcd66bad8a, org=] INFO  org.candlepin.common.filter.LoggingFilter - Request: verb=GET, uri=/candlepin/owners/testerr/uebercert
2017-04-06 13:47:15,406 [thread=http-8443-3] [req=b118cd76-6b65-42e3-a8bf-65bcd66bad8a, org=testerr] INFO  org.candlepin.common.filter.LoggingFilter - Response: status=404, content-type="application/json", time=39
2017-04-06 13:47:15,456 [thread=http-8443-3] [req=ac806b48-cc82-41a9-8299-9276d9d27703, org=] INFO  org.candlepin.common.filter.LoggingFilter - Request: verb=POST, uri=/candlepin/owners/testerr/uebercert
2017-04-06 13:47:15,521 [thread=http-8443-3] [req=ac806b48-cc82-41a9-8299-9276d9d27703, org=testerr] INFO  org.candlepin.policy.js.pool.PoolRules - Checking if pools need to be created for: Subscription [id = 8a90848f5b3f84cd015b4460a2b004a1, product = 14915008354908818526412, owner = testerr]
2017-04-06 13:47:15,523 [thread=http-8443-3] [req=ac806b48-cc82-41a9-8299-9276d9d27703, org=testerr] INFO  org.candlepin.policy.js.pool.PoolRules - Creating new master pool: Pool<type=NORMAL, product=14915008354908818526412, productName=testerr_ueber_product, id=null, quantity=1>
2017-04-06 13:47:15,592 [thread=http-8443-3] [req=ac806b48-cc82-41a9-8299-9276d9d27703, org=testerr] INFO  org.candlepin.controller.CandlepinPoolManager - Locking pool 8a90848f5b3f84cd015b4460a2c304a2
2017-04-06 13:47:15,596 [thread=http-8443-3] [req=ac806b48-cc82-41a9-8299-9276d9d27703, org=testerr] INFO  org.candlepin.controller.CandlepinPoolManager - Running pre-entitlement rules.
2017-04-06 13:47:15,605 [thread=http-8443-3] [req=ac806b48-cc82-41a9-8299-9276d9d27703, org=testerr] INFO  org.candlepin.controller.CandlepinPoolManager - Processing entitlement.
2017-04-06 13:47:15,605 [thread=http-8443-3] [req=ac806b48-cc82-41a9-8299-9276d9d27703, org=testerr] INFO  org.candlepin.controller.CandlepinPoolManager - Persisting entitlement.
2017-04-06 13:47:15,650 [thread=http-8443-3] [req=ac806b48-cc82-41a9-8299-9276d9d27703, org=testerr] INFO  org.candlepin.controller.CandlepinPoolManager - Getting subscription: 8a90848f5b3f84cd015b4460a2b004a1
2017-04-06 13:47:15,662 [thread=http-8443-3] [req=ac806b48-cc82-41a9-8299-9276d9d27703, org=testerr] INFO  org.candlepin.controller.CandlepinPoolManager - Got subscription
2017-04-06 13:47:15,662 [thread=http-8443-3] [req=ac806b48-cc82-41a9-8299-9276d9d27703, org=testerr] INFO  org.candlepin.service.impl.DefaultEntitlementCertServiceAdapter - Generating entitlement cert.
2017-04-06 13:47:16,997 [thread=http-8443-3] [req=ac806b48-cc82-41a9-8299-9276d9d27703, org=testerr] INFO  org.candlepin.service.impl.DefaultEntitlementCertServiceAdapter - Creating X509 cert.
2017-04-06 13:47:17,021 [thread=http-8443-3] [req=ac806b48-cc82-41a9-8299-9276d9d27703, org=testerr] INFO  org.candlepin.service.impl.DefaultEntitlementCertServiceAdapter - Getting PEM encoded cert.
2017-04-06 13:47:17,023 [thread=http-8443-3] [req=ac806b48-cc82-41a9-8299-9276d9d27703, org=testerr] INFO  org.candlepin.service.impl.DefaultEntitlementCertServiceAdapter - Persisting cert.
2017-04-06 13:47:17,092 [thread=http-8443-3] [req=ac806b48-cc82-41a9-8299-9276d9d27703, org=testerr] INFO  org.candlepin.controller.CandlepinPoolManager - Granted entitlement: 8a90848f5b3f84cd015b4460a31604a5 from pool: 8a90848f5b3f84cd015b4460a2c304a2
2017-04-06 13:47:17,101 [thread=http-8443-3] [req=ac806b48-cc82-41a9-8299-9276d9d27703, org=testerr] INFO  org.candlepin.audit.HornetqEventDispatcher - Sending event: Event [id=null, target=POOL, type=CREATED, time=Thu Apr 06 13:47:15 EDT 2017, entity=8a90848f5b3f84cd015b4460a2c304a2]
2017-04-06 13:47:17,110 [thread=http-8443-3] [req=ac806b48-cc82-41a9-8299-9276d9d27703, org=testerr] INFO  org.candlepin.audit.HornetqEventDispatcher - Sending event: Event [id=null, target=COMPLIANCE, type=CREATED, time=Thu Apr 06 13:47:15 EDT 2017, entity=8a90848f5b3f84cd015b4460a2e504a4]
2017-04-06 13:47:17,116 [thread=http-8443-3] [req=ac806b48-cc82-41a9-8299-9276d9d27703, org=testerr] INFO  org.candlepin.common.filter.LoggingFilter - Response: status=200, content-type="application/json", time=1660
2017-04-06 13:47:19,071 [thread=http-8443-3] [req=403d2066-e3b2-4e60-ae5a-9886d75e6bab, org=] INFO  org.candlepin.common.filter.LoggingFilter - Request: verb=GET, uri=/candlepin/pools/8a90848f5b3f84cd015b4460a2c304a2
2017-04-06 13:47:19,114 [thread=http-8443-3] [req=403d2066-e3b2-4e60-ae5a-9886d75e6bab, org=testerr] INFO  org.candlepin.common.filter.LoggingFilter - Response: status=200, content-type="application/json", time=43
2017-04-06 13:47:19,173 [thread=http-8443-3] [req=5cb729e1-a26c-4881-b200-c7ac5399db60, org=] INFO  org.candlepin.common.filter.LoggingFilter - Request: verb=GET, uri=/candlepin/activation_keys/?include=id&include=pools.pool.id
2017-04-06 13:47:19,188 [thread=http-8443-3] [req=5cb729e1-a26c-4881-b200-c7ac5399db60, org=] INFO  org.candlepin.common.filter.LoggingFilter - Response: status=200, content-type="application/json", time=15

Comment 29 Bryan Kearney 2017-05-01 14:28:02 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:1191

Comment 30 Bryan Kearney 2017-05-01 14:28:47 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:1191


Note You need to log in before you can comment on or make changes to this bug.