Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.

Bug 1513143

Summary: Enabling replication after upgrading from 5.5 to 5.9 fails on db schema
Product: Red Hat CloudForms Management Engine Reporter: Satoe Imaishi <simaishi>
Component: ApplianceAssignee: Nick Carboni <ncarboni>
Status: CLOSED CURRENTRELEASE QA Contact: luke couzens <lcouzens>
Severity: high Docs Contact:
Priority: high    
Version: 5.9.0CC: abellott, jhardy, lcouzens, obarenbo, smallamp
Target Milestone: GAKeywords: Regression, TestOnly
Target Release: 5.9.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: black:replication:upgrade:migration
Fixed In Version: 5.9.0.10 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1508537
: 1517386 (view as bug list) Environment:
Last Closed: 2018-03-06 15:08:04 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: CFME Core
Bug Depends On: 1508537    
Bug Blocks: 1517386    

Comment 2 luke couzens 2017-11-17 11:14:55 UTC
Rake task has been added to latest build but its still not working as expected. We can now run the db:check_schema but the rails r tools/fix_column_ordering.rb task still fails. see output below.

[root@vmdb]# rake db:check_schema
rake aborted!
Schema validation failed for host localhost:

Columns for table computer_systems in the current schema do not match the columns listed in /opt/rh/cfme-gemset/bundler/gems/manageiq-schema-828d50c74f4b/db/schema.yml

expected:
["id", "managed_entity_type", "managed_entity_id", "created_at", "updated_at"]

got:
["id", "managed_entity_id", "managed_entity_type", "created_at", "updated_at"]

Schema validation failed for host localhost:

Columns for table event_streams in the current schema do not match the columns listed in /opt/rh/cfme-gemset/bundler/gems/manageiq-schema-828d50c74f4b/db/schema.yml

expected:
["id", "event_type", "message", "timestamp", "host_name", "host_id", "vm_name", "vm_location", "vm_or_template_id", "dest_host_name", "dest_host_id", "dest_vm_name", "dest_vm_location", "dest_vm_or_template_id", "source", "chain_id", "ems_id", "is_task", "full_data", "created_on", "username", "ems_cluster_id", "ems_cluster_name", "ems_cluster_uid", "dest_ems_cluster_id", "dest_ems_cluster_name", "dest_ems_cluster_uid", "availability_zone_id", "container_node_id", "container_node_name", "container_group_id", "container_group_name", "container_namespace", "type", "target_type", "target_id", "container_id", "container_name", "container_replicator_id", "container_replicator_name", "middleware_server_id", "middleware_server_name", "middleware_deployment_id", "middleware_deployment_name", "generating_ems_id", "physical_server_id", "ems_ref", "middleware_domain_id", "middleware_domain_name", "user_id", "group_id", "tenant_id"]

got:
["id", "event_type", "message", "timestamp", "host_name", "host_id", "vm_name", "vm_location", "vm_or_template_id", "dest_host_name", "dest_host_id", "dest_vm_name", "dest_vm_location", "dest_vm_or_template_id", "source", "chain_id", "ems_id", "is_task", "full_data", "created_on", "username", "ems_cluster_id", "ems_cluster_name", "ems_cluster_uid", "dest_ems_cluster_id", "dest_ems_cluster_name", "dest_ems_cluster_uid", "availability_zone_id", "container_node_id", "container_node_name", "container_group_id", "container_group_name", "container_namespace", "type", "target_id", "target_type", "container_id", "container_name", "container_replicator_id", "container_replicator_name", "middleware_server_id", "middleware_server_name", "middleware_deployment_id", "middleware_deployment_name", "generating_ems_id", "physical_server_id", "ems_ref", "middleware_domain_id", "middleware_domain_name", "user_id", "group_id", "tenant_id"]

Schema validation failed for host localhost:

Columns for table security_contexts in the current schema do not match the columns listed in /opt/rh/cfme-gemset/bundler/gems/manageiq-schema-828d50c74f4b/db/schema.yml

expected:
["id", "resource_type", "resource_id", "se_linux_user", "se_linux_role", "se_linux_type", "se_linux_level"]

got:
["id", "resource_id", "resource_type", "se_linux_user", "se_linux_role", "se_linux_type", "se_linux_level"]
/opt/rh/cfme-gemset/bundler/gems/manageiq-schema-828d50c74f4b/lib/tasks/schema_tasks.rake:5:in `block (2 levels) in <top (required)>'
/opt/rh/cfme-gemset/gems/rake-12.3.0/exe/rake:27:in `<top (required)>'
Tasks: TOP => db:check_schema
(See full trace by running task with --trace)


[root@vmdb]# rails r tools/fix_column_ordering.rb computer_systems
Correcting column ordering for table computer_systems
/opt/rh/rh-ruby23/root/usr/share/gems/gems/psych-2.0.17/lib/psych.rb:471:in `initialize': No such file or directory @ rb_sysopen - /var/www/miq/vmdb/db/schema.yml (Errno::ENOENT)
	from /opt/rh/rh-ruby23/root/usr/share/gems/gems/psych-2.0.17/lib/psych.rb:471:in `open'
	from /opt/rh/rh-ruby23/root/usr/share/gems/gems/psych-2.0.17/lib/psych.rb:471:in `load_file'
	from /var/www/miq/vmdb/tools/column_ordering/column_ordering.rb:18:in `expected_columns'
	from /var/www/miq/vmdb/tools/column_ordering/column_ordering.rb:213:in `assert_column_list_sizes_match!'
	from /var/www/miq/vmdb/tools/column_ordering/column_ordering.rb:30:in `fix_column_ordering'
	from tools/fix_column_ordering.rb:22:in `<top (required)>'
	from /opt/rh/cfme-gemset/gems/railties-5.0.6/lib/rails/commands/runner.rb:60:in `load'
	from /opt/rh/cfme-gemset/gems/railties-5.0.6/lib/rails/commands/runner.rb:60:in `<top (required)>'
	from /opt/rh/cfme-gemset/gems/railties-5.0.6/lib/rails/commands/commands_tasks.rb:138:in `require'
	from /opt/rh/cfme-gemset/gems/railties-5.0.6/lib/rails/commands/commands_tasks.rb:138:in `require_command!'
	from /opt/rh/cfme-gemset/gems/railties-5.0.6/lib/rails/commands/commands_tasks.rb:104:in `runner'
	from /opt/rh/cfme-gemset/gems/railties-5.0.6/lib/rails/commands/commands_tasks.rb:49:in `run_command!'
	from /opt/rh/cfme-gemset/gems/railties-5.0.6/lib/rails/commands.rb:18:in `<top (required)>'
	from bin/rails:4:in `require'
	from bin/rails:4:in `<main>'

Comment 3 Nick Carboni 2017-11-17 13:33:03 UTC
I actually just removed this tool because the new version of pglogical we use doesn't have the same issue with column ordering.

https://github.com/ManageIQ/manageiq/pull/16488

When that gets backported, I'll just close this bug.

Comment 4 Nick Carboni 2017-11-17 18:20:36 UTC
Moving this to POST.

Luke, as a test for this can you just skip the column reordering steps and ensure that replication works?

The new version of pglogical will handle out of order columns.

Comment 5 luke couzens 2017-11-17 19:32:12 UTC
Is the new version of pglogical on 5.9.0.9 then? I still see the same ordering issues when trying to setup replication on the global region.

Error during replication configuration save: Failed to save subscription to : Schema validation failed for host : Columns for table computer_systems in the current schema do not match the columns listed in /opt/rh/cfme-gemset/bundler/gems/manageiq-schema-828d50c74f4b/db/schema.yml expected: ["id", "managed_entity_type", "managed_entity_id", "created_at", "updated_at"] got: ["id", "managed_entity_id", "managed_entity_type", "created_at", "updated_at"] Schema validation failed for host : Columns for table event_streams in the current schema do not match the columns listed in /opt/rh/cfme-gemset/bundler/gems/manageiq-schema-828d50c74f4b/db/schema.yml expected: ["id", "event_type", "message", "timestamp", "host_name", "host_id", "vm_name", "vm_location", "vm_or_template_id", "dest_host_name", "dest_host_id", "dest_vm_name", "dest_vm_location", "dest_vm_or_template_id", "source", "chain_id", "ems_id", "is_task", "full_data", "created_on", "username", "ems_cluster_id", "ems_cluster_name", "ems_cluster_uid", "dest_ems_cluster_id", "dest_ems_cluster_name", "dest_ems_cluster_uid", "availability_zone_id", "container_node_id", "container_node_name", "container_group_id", "container_group_name", "container_namespace", "type", "target_type", "target_id", "container_id", "container_name", "container_replicator_id", "container_replicator_name", "middleware_server_id", "middleware_server_name", "middleware_deployment_id", "middleware_deployment_name", "generating_ems_id", "physical_server_id", "ems_ref", "middleware_domain_id", "middleware_domain_name", "user_id", "group_id", "tenant_id"] got: ["id", "event_type", "message", "timestamp", "host_name", "host_id", "vm_name", "vm_location", "vm_or_template_id", "dest_host_name", "dest_host_id", "dest_vm_name", "dest_vm_location", "dest_vm_or_template_id", "source", "chain_id", "ems_id", "is_task", "full_data", "created_on", "username", "ems_cluster_id", "ems_cluster_name", "ems_cluster_uid", "dest_ems_cluster_id", "dest_ems_cluster_name", "dest_ems_cluster_uid", "availability_zone_id", "container_node_id", "container_node_name", "container_group_id", "container_group_name", "container_namespace", "type", "target_id", "target_type", "container_id", "container_name", "container_replicator_id", "container_replicator_name", "middleware_server_id", "middleware_server_name", "middleware_deployment_id", "middleware_deployment_name", "generating_ems_id", "physical_server_id", "ems_ref", "middleware_domain_id", "middleware_domain_name", "user_id", "group_id", "tenant_id"] Schema validation failed for host : Columns for table security_contexts in the current schema do not match the columns listed in /opt/rh/cfme-gemset/bundler/gems/manageiq-schema-828d50c74f4b/db/schema.yml expected: ["id", "resource_type", "resource_id", "se_linux_user", "se_linux_role", "se_linux_type", "se_linux_level"] got: ["id", "resource_id", "resource_type", "se_linux_user", "se_linux_role", "se_linux_type", "se_linux_level"]

Comment 6 Nick Carboni 2017-11-17 20:24:37 UTC
Ah, nope. Let me unset the fixed-in version.

https://github.com/ManageIQ/manageiq/pull/16488 was just backported today, so it should be in the next build.

Comment 7 luke couzens 2017-11-23 11:37:38 UTC
Verified in 5.9.0.10

Comment 8 CFME Bot 2017-11-29 20:47:50 UTC
New commit detected on ManageIQ/manageiq-schema/gaprindashvili:
https://github.com/ManageIQ/manageiq-schema/commit/828d50c74f4b772d7ed57e78d9a94f14244c866a

commit 828d50c74f4b772d7ed57e78d9a94f14244c866a
Author:     Jason Frey <fryguy9@gmail.com>
AuthorDate: Mon Nov 13 15:52:26 2017 -0500
Commit:     Satoe Imaishi <simaishi@redhat.com>
CommitDate: Tue Nov 14 20:15:54 2017 -0500

    Merge pull request #124 from carbonin/make_schema_check_and_write_public
    
    Move the db:check_schema task to the public tasks.rake file
    (cherry picked from commit ad336bd67a970b12d29a74295881fcf7a4b5895f)
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1513143

 lib/tasks/schema_tasks.rake         | 8 ++++++++
 lib/tasks_private/schema_tasks.rake | 7 -------
 2 files changed, 8 insertions(+), 7 deletions(-)
 create mode 100644 lib/tasks/schema_tasks.rake