Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1355889 - spacewalk-clone-by-date throws rhn_erratafile_eid_file_uq
Summary: spacewalk-clone-by-date throws rhn_erratafile_eid_file_uq
Keywords:
Status: CLOSED WONTFIX
Alias: None
Product: Red Hat Satellite 5
Classification: Red Hat
Component: Server
Version: 570
Hardware: Unspecified
OS: Unspecified
high
medium
Target Milestone: ---
Assignee: Tomáš Kašpárek
QA Contact: Red Hat Satellite QA List
URL:
Whiteboard:
Depends On:
Blocks: 465198 sat570-triage
TreeView+ depends on / blocked
 
Reported: 2016-07-12 20:13 UTC by Shannon Hughes
Modified: 2019-02-06 02:25 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-11-03 16:12:41 UTC


Attachments (Terms of Use)

Description Shannon Hughes 2016-07-12 20:13:14 UTC
Description of problem:

When running spacewalk-clone-by-date on both RHEL 6/7 x86_64 bit channels a 500 is thrown: 

[root@demosat ~]# spacewalk-clone-by-date -u rhnadmin -d 2016-05-28 -l rhel-x86_64-server-7 sus_20160528-rhel-x86_64-server-7-pjw
Password: 
Reading repository information.
.
.
.

Cloning Errata into sus_20160528-rhel-x86_64-server-7-pjw (1198):
________________________________________
Traceback (most recent call last):
  File "/usr/bin/spacewalk-clone-by-date", line 253, in <module>
    sys.exit(abs(main() or 0))
  File "/usr/bin/spacewalk-clone-by-date", line 243, in main
    return cloneByDate.main(args)
  File "/usr/share/rhn/utils/cloneByDate.py", line 197, in main
    cloner.clone(options.skip_depsolve)
  File "/usr/share/rhn/utils/cloneByDate.py", line 358, in clone
    cloner.process()
  File "/usr/share/rhn/utils/cloneByDate.py", line 502, in process
    self.clone()
  File "/usr/share/rhn/utils/cloneByDate.py", line 594, in clone
    self.remote_api.clone_errata(self.to_label, errata_set)
  File "/usr/share/rhn/utils/cloneByDate.py", line 723, in clone_errata
    self.client.errata.cloneAsOriginal(self.auth_token, to_label, errata_list)
  File "/usr/lib64/python2.6/xmlrpclib.py", line 1199, in __call__
    return self.__send(self.__name, args)
  File "/usr/lib64/python2.6/xmlrpclib.py", line 1489, in __request
    verbose=self.__verbose
  File "/usr/lib64/python2.6/xmlrpclib.py", line 1243, in request
    headers
xmlrpclib.ProtocolError: <ProtocolError for localhost/rpc/api: 500 Internal Server Error>

Looking deeper into the catalina.out logs we find that this is a unique constraint violation in the database: 


Hibernate: select nextval ('RHN_ERRATAFILE_ID_SEQ')
Hibernate: insert into rhnErrataFile (checksum_id, fileName, type, errata_id, id) values (?, ?, ?, ?, ?)
2016-07-12 11:45:49,838 [TP-Processor5] WARN  org.hibernate.util.JDBCExceptionReporter - SQL Error: 0, SQLState: 23505
2016-07-12 11:45:49,838 [TP-Processor5] ERROR org.hibernate.util.JDBCExceptionReporter - ERROR: duplicate key value violates unique constraint "rhn_erratafile_eid_file_uq"
.
.
.

Caused by:
org.postgresql.util.PSQLException: ERROR: duplicate key value violates unique constraint "rhn_erratafile_eid_file_uq"
        at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2094)
        at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1827)
        at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:255)
        at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:508)
        at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:384)
        at org.postgresql.jdbc2.AbstractJdbc2Statement.executeUpdate(AbstractJdbc2Statement.java:330)
        at net.rkbloom.logdriver.LogPreparedStatement.executeUpdate(LogPreparedStatement.java:92)
        at com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.executeUpdate(NewProxyPreparedStatement.java:79)
        at org.hibernate.jdbc.NonBatchingBatcher.addToBatch(NonBatchingBatcher.java:46)

There is a workaround that seems to clear up the database to allow the spacewalk-clone-by-date to run. The following procedure must be completed for each channel a clone would be cloned from: 

1) in webui clone a new channel using the same original channel that was used in the failed spacewalk-clone-by-date with no errata, original channel state
2) go to the new channel in the webui (manage channels), click on the channel, then errata and clone errata
3) now go back to channel an sync the errata
4) after syncing is complete, rerun the original spacewalk-clone-by-date and it works

I don't know the RCA for this issue but given the WEBUI procedure seems to clean up the DB so clone by date might give us some insight. The packages/files that were hitting UQ were not part of the recent bad errata that was released as I cross-referenced these files. Also, I did not see any git commits that were committed in the past few months that might have caused an issue.

Comment 2 redhat 2016-10-27 22:45:53 UTC
Not sure if this is helpful or will just get me laughed at, but I am seeing the  same thing in Spacewalk 2.2 on Oracle Linux 6 [the version of spacewalk supported by Oracle from their spacewalk repo] and running against an Oracle database.

# spacewalk-clone-by-date -u USER -c itmisc-dev.cbd

... 

Cloning Errata into dev-itmisc-ol6u7_x86_64_parent (2573):
________________________________________
Traceback (most recent call last):
  File "/usr/bin/spacewalk-clone-by-date", line 375, in <module>
    sys.exit(abs(main() or 0))
  File "/usr/bin/spacewalk-clone-by-date", line 365, in main
    return cloneByDate.main(args)
  File "/usr/share/rhn/utils/cloneByDate.py", line 214, in main
    cloner.clone(options.skip_depsolve)
  File "/usr/share/rhn/utils/cloneByDate.py", line 385, in clone
    cloner.process()
  File "/usr/share/rhn/utils/cloneByDate.py", line 525, in process
    self.clone()
  File "/usr/share/rhn/utils/cloneByDate.py", line 591, in clone
    self.remote_api.clone_errata(self.to_label, errata_set)
  File "/usr/share/rhn/utils/cloneByDate.py", line 735, in clone_errata
    errata_list)
  File "/usr/lib64/python2.6/xmlrpclib.py", line 1199, in __call__
    return self.__send(self.__name, args)
  File "/usr/lib64/python2.6/xmlrpclib.py", line 1489, in __request
    verbose=self.__verbose
  File "/usr/lib64/python2.6/xmlrpclib.py", line 1243, in request
    headers
xmlrpclib.ProtocolError: <ProtocolError for localhost/rpc/api: 500 Internal Server Error>




And then in catalina.out:

Oct 27, 2016 9:02:26 PM redstone.xmlrpc.XmlRpcDispatcher writeError
WARNING: redstone.xmlrpc.XmlRpcFault: unhandled internal exception: could not insert: [com.redhat.rhn.domain.errata.impl.PublishedErrataFile]
2016-10-27 21:02:26,550 [TP-Processor15] WARN  org.hibernate.util.JDBCExceptionReporter - SQL Error: 1, SQLState: 23000
2016-10-27 21:02:26,550 [TP-Processor15] ERROR org.hibernate.util.JDBCExceptionReporter - ORA-00001: unique constraint (SPACEWALK.RHN_ERRATAFILE_EID
_FILE_UQ) violated


...

2016-10-27 21:02:26,550 [TP-Processor15] ERROR org.hibernate.event.def.AbstractFlushingEventListener - Could not synchronize database state with ses
sion
org.hibernate.exception.ConstraintViolationException: could not insert: [com.redhat.rhn.domain.errata.impl.PublishedErrataFile]
        at org.hibernate.exception.SQLStateConverter.convert(SQLStateConverter.java:71)
        at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:43)
        at org.hibernate.persister.entity.AbstractEntityPersister.insert(AbstractEntityPersister.java:2267)
        at org.hibernate.persister.entity.AbstractEntityPersister.insert(AbstractEntityPersister.java:2660)
        at org.hibernate.action.EntityInsertAction.execute(EntityInsertAction.java:56)
        at org.hibernate.engine.ActionQueue.execute(ActionQueue.java:250)
        at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:234)
        at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:141)
        at org.hibernate.event.def.AbstractFlushingEventListener.performExecutions(AbstractFlushingEventListener.java:298)
        at org.hibernate.event.def.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:27)
        at org.hibernate.impl.SessionImpl.flush(SessionImpl.java:1000)
        at org.hibernate.impl.SessionImpl.managedFlush(SessionImpl.java:338)
        at org.hibernate.transaction.JDBCTransaction.commit(JDBCTransaction.java:106)
        at com.redhat.rhn.common.hibernate.ConnectionManager.commitTransaction(ConnectionManager.java:233)
        at com.redhat.rhn.common.hibernate.HibernateFactory.commitTransaction(HibernateFactory.java:331)
        at com.redhat.rhn.frontend.servlets.SessionFilter.doFilter(SessionFilter.java:58)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at com.redhat.rhn.frontend.servlets.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:97)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:299)
        at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:190)
        at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:291)
        at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:769)
        at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:698)
        at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:891)
        at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:690)
        at java.lang.Thread.run(Thread.java:701)
Caused by: java.sql.SQLIntegrityConstraintViolationException: ORA-00001: unique constraint (SPACEWALK.RHN_ERRATAFILE_EID_FILE_UQ) violated






Unfortunately I'm not able to try the workaround.


Note You need to log in before you can comment on or make changes to this bug.