Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1366076 - upgrading to 6.2 fails on katello:upgrades:3.0:update_puppet_repository_distributors [NEEDINFO]
Summary: upgrading to 6.2 fails on katello:upgrades:3.0:update_puppet_repository_distr...
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Red Hat Satellite 6
Classification: Red Hat
Component: Upgrades
Version: 6.2.0
Hardware: Unspecified
OS: Unspecified
unspecified
high vote
Target Milestone: Unspecified
Assignee: satellite6-bugs
QA Contact: Katello QA List
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-08-11 00:24 UTC by Kathryn Dixon
Modified: 2018-01-16 11:55 UTC (History)
8 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-08-11 19:25:50 UTC
kdixon: needinfo? (mmccune)


Attachments (Terms of Use)
output of remove.missing.repo script. (deleted)
2016-08-11 00:24 UTC, Kathryn Dixon
no flags Details

Description Kathryn Dixon 2016-08-11 00:24:54 UTC
Created attachment 1189834 [details]
output of remove.missing.repo script.

Description of problem:

running 6.2 upgrade and fails on 
Upgrade step update_puppet_repository_distributors failed

rake aborted!
RestClient::ResourceNotFound: 404 Resource Not Found
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/abstract_response.rb:48:in `return!'
/opt/theforeman/tfm/root/usr/share/gems/gems/runcible-1.7.0/lib/runcible/base.rb:79:in `block in get_response'
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/request.rb:228:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/request.rb:228:in `process_result'
/opt/theforeman/tfm/root/usr/share/gems/gems/rbovirt-0.0.37/lib/restclient_ext/request.rb:50:in `block in transmit'
/opt/theforeman/tfm/root/usr/share/gems/gems/rbovirt-0.0.37/lib/restclient_ext/request.rb:44:in `transmit'
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/request.rb:64:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/request.rb:33:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/resource.rb:51:in `get'
/opt/theforeman/tfm/root/usr/share/gems/gems/runcible-1.7.0/lib/runcible/base.rb:78:in `get_response'
/opt/theforeman/tfm/root/usr/share/gems/gems/runcible-1.7.0/lib/runcible/base.rb:66:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/runcible-1.7.0/lib/runcible/resources/repository.rb:31:in `retrieve'
/opt/theforeman/tfm/root/usr/share/gems/gems/runcible-1.7.0/lib/runcible/extensions/repository.rb:377:in `retrieve_with_details'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.68/app/lib/actions/pulp/repository/refresh.rb:10:in `plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action.rb:461:in `block (3 levels) in execute_plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:26:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:26:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware.rb:17:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.68/app/lib/actions/middleware/backend_services_check.rb:22:in `plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:22:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:26:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware.rb:17:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.68/app/lib/actions/middleware/remote_action.rb:9:in `plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:22:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:26:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware.rb:17:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware.rb:34:in `plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:22:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:26:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware.rb:17:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware.rb:34:in `plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:22:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/world.rb:30:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action.rb:460:in `block (2 levels) in execute_plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/execution_plan.rb:281:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/execution_plan.rb:281:in `switch_flow'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action.rb:364:in `concurrence'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action.rb:459:in `block in execute_plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action.rb:419:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action.rb:419:in `block in with_error_handling'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action.rb:419:in `catch'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action.rb:419:in `with_error_handling'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action.rb:458:in `execute_plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action.rb:260:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/execution_plan/steps/plan_step.rb:54:in `block in execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/execution_plan/steps/abstract.rb:155:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/execution_plan/steps/abstract.rb:155:in `with_meta_calculation'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/execution_plan/steps/plan_step.rb:53:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/execution_plan.rb:191:in `block (2 levels) in plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/execution_plan.rb:281:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/execution_plan.rb:281:in `switch_flow'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/execution_plan.rb:271:in `with_planning_scope'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/execution_plan.rb:190:in `block in plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:26:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:26:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware.rb:17:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware.rb:42:in `plan_phase'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:22:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:26:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware.rb:17:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware.rb:42:in `plan_phase'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:22:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:26:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware.rb:17:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware.rb:42:in `plan_phase'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:22:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:26:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware.rb:17:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/common/transaction.rb:16:in `block in rollback_on_error'
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/connection_adapters/abstract/database_statements.rb:201:in `block in transaction'
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/connection_adapters/abstract/database_statements.rb:209:in `within_new_transaction'
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/connection_adapters/abstract/database_statements.rb:201:in `transaction'
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/transactions.rb:208:in `transaction'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/transaction_adapters/active_record.rb:5:in `transaction'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/common/transaction.rb:15:in `rollback_on_error'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/common/transaction.rb:5:in `plan_phase'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:22:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/world.rb:30:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/execution_plan.rb:189:in `plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/world.rb:168:in `block in plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/world.rb:166:in `tap'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/world.rb:166:in `plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/world.rb:142:in `trigger'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.7.14.6/lib/foreman_tasks.rb:18:in `trigger'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.7.14.6/lib/foreman_tasks.rb:24:in `trigger_task'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.7.14.6/lib/foreman_tasks.rb:39:in `sync_task'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.68/lib/katello/tasks/upgrades/3.0/update_puppet_repository_distributors.rake:14:in `block (5 levels) in <top (required)>'
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/relation/delegation.rb:46:in `each'
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/relation/delegation.rb:46:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.68/lib/katello/tasks/upgrades/3.0/update_puppet_repository_distributors.rake:13:in `block (4 levels) in <top (required)>'
Tasks: TOP => katello:upgrades:3.0:update_puppet_repository_distributors
(See full trace by running task with --trace)
Updating Puppet Repository Distributors
Updating Content View Puppet Environment Distributors

Upgrade step update_puppet_repository_distributors failed. Check logs for more information.
[ERROR 2016-08-10 14:13:28 verbose] Upgrade step update_puppet_repository_distributors failed. Check logs for more information.


Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:

Tasks: TOP => katello:upgrades:3.0:update_puppet_repository_distributors
(See full trace by running task with --trace)
Updating Puppet Repository Distributors
Updating Content View Puppet Environment Distributors

Upgrade step update_puppet_repository_distributors failed. Check logs for more information.
[ERROR 2016-08-10 14:13:28 verbose] Upgrade step update_puppet_repository_distributors failed. Check logs for more information.

hammer ping is clean
httpd status active

Tried running step manually

# foreman-rake katello:upgrades:3.0:update_puppet_repository_distributors --trace


Expected results: clean upgrade to 6.2


Additional info:

have tried this suggestion.

https://gist.github.com/jlsherrill/3804bcf080eeccdb661e9a0bd5de9b28

fails, and will attach error.

Comment 1 Justin Sherrill 2016-08-11 19:25:20 UTC
The root cause of this issue is that at some point in the past, some task was skipped or unlocked, or deleted that caused a repository to be deleted or not created in pulp.  

We can resolve the situation via:


foreman-rake console


User.current = User.first

::Katello::Repository.puppet_type.each do |repo|
  begin
    puts "Checking #{repo.name} - #{repo.id}"
    Katello.pulp_server.extensions.repository.retrieve_with_details(repo.pulp_id)
  rescue RestClient::ResourceNotFound
    puts "Deleting #{repo.name} - #{repo.id}"
    repo.destroy!
  end
end

::Katello::ContentViewPuppetEnvironment.all.each do |repo|
  begin
    puts "Checking #{repo.name} - #{repo.id}"
    Katello.pulp_server.extensions.repository.retrieve_with_details(repo.pulp_id)
  rescue RestClient::ResourceNotFound
    puts "Deleting #{repo.name} - #{repo.id}"
    repo.destroy!
  end
end

Comment 2 Justin Sherrill 2016-08-11 19:25:50 UTC
Closing as not a bug

Comment 5 Sachin Ghai 2018-01-16 11:55:02 UTC
I'm facing exactly same issue as pointed in comment3 while using a db when upgrading to 6.2.13


~]# foreman-rake katello:upgrades:3.0:update_puppet_repository_distributors --trace
** Invoke katello:upgrades:3.0:update_puppet_repository_distributors (first_time)
** Invoke environment (first_time)
** Execute environment
** Execute katello:upgrades:3.0:update_puppet_repository_distributors
Updating Puppet Repository Distributors
Updating Content View Puppet Environment Distributors
rake aborted!
ForemanTasks::TaskError: Task 8615202d-ab03-480c-b162-792611221b85: Katello::Errors::PulpError: PLP0034: The distributor  xxx-xx-CCV_CSL4_DOCKERMVP-4_0_puppet indicated a failed response when publishing repository xxx-xx-CCV_CSL4_DOCKERMVP-4_0.
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.7.14.14/lib/foreman_tasks.rb:40:in `block in sync_task'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.7.14.14/lib/foreman_tasks.rb:39:in `tap'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.7.14.14/lib/foreman_tasks.rb:39:in `sync_task'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.158/lib/katello/tasks/upgrades/3.0/update_puppet_repository_distributors.rake:15:in `block (5 levels) in <top (required)>'
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/relation/delegation.rb:46:in `each'
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/relation/delegation.rb:46:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.158/lib/katello/tasks/upgrades/3.0/update_puppet_repository_distributors.rake:13:in `block (4 levels) in <top (required)>'
/opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/task.rb:240:in `call'
/opt/rh/


Note You need to log in before you can comment on or make changes to this bug.