Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.

Bug 969026

Summary: [engine] Storage Domain that was destroyed from GUI did not get completely removed from DB
Product: Red Hat Enterprise Virtualization Manager Reporter: Gadi Ickowicz <gickowic>
Component: ovirt-engineAssignee: Liron Aravot <laravot>
Status: CLOSED CURRENTRELEASE QA Contact: Ori Gofen <ogofen>
Severity: high Docs Contact:
Priority: unspecified    
Version: 3.2.0CC: acanan, amureini, bsettle, dron, iheim, lpeer, nlevinki, rbalakri, Rhev-m-bugs, scohen, tnisan, yeylon
Target Milestone: ---Flags: amureini: Triaged+
Target Release: 3.5.0   
Hardware: All   
OS: Linux   
Whiteboard: storage
Fixed In Version: ovirt-3.5.0-alpha2 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-02-16 19:10:13 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Storage RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Bug Depends On:    
Bug Blocks: 1142923, 1156165    
Description Flags
vdsm + engine logs none

Description Gadi Ickowicz 2013-05-30 13:58:51 UTC
Created attachment 754868 [details]
vdsm + engine logs

Description of problem:
After creating a new DC with 1 host and 1 storage domain, and 1 additional domain which is not attached, all storages were destroyed (directly on the storage server). To clean the RHEVM the host was moved to maintenance, DC was force-removed and additional domain was destroyed. However the domain remained in the storage_domain_static table. Now it is not possible to add another domain to the RHEVM with the same name as that one.

Queries to the DB:
# psql -d engine -U postgres -c "select * from storage_domain_static;"
                 id                  |                storage                 | storage_name | storage_domain_type | storage_type | storage_domain_format_type |         _create_date          |         _update_dat
 feb6aa0e-4302-42b9-86a7-84d902fea07e | AxHc8t-CRgE-FKft-qrSl-PG8O-Svs5-9m5dL3 | iscsi_2      |                   1 |            3 | 3                          | 2013-05-26 14:39:33.223723+03 |                    
 7d3d2d42-270f-45dc-b6e4-20fc1f389e37 | iDGExr-4Kch-TKl0-GzEV-BxgG-bvsP-rfK6Ly | iscsi_0      |                   0 |            3 | 3                          | 2013-05-30 14:25:00.450573+03 | 2013-05-30 14:25:04
 f1103bf0-65cb-4d7a-a7cd-1656ea1252cd | RYPDXv-AfXt-7Dyz-SA4d-nVwI-xFOe-I0FjU6 | iscsi_1      |                   1 |            3 | 0                          | 2013-05-30 14:26:22.152056+03 | 2013-05-30 14:26:22

# # psql -d engine -U postgres -c "select * from storage_domains;"
                  id                  |                storage                 | storage_name |           storage_pool_id            | available_disk_size | used_disk_size | commited_disk_size | status | storage_p
 7d3d2d42-270f-45dc-b6e4-20fc1f389e37 | iDGExr-4Kch-TKl0-GzEV-BxgG-bvsP-rfK6Ly | iscsi_0      | 7cc34b9a-ea43-49ba-a73d-f667bb93f664 |                     |                |                  0 |      4 | TestDC   
 f1103bf0-65cb-4d7a-a7cd-1656ea1252cd | RYPDXv-AfXt-7Dyz-SA4d-nVwI-xFOe-I0FjU6 | iscsi_1      |                                      |                  10 |              4 |                  0 |        |          
(2 rows)

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:
1. Create datacenter, cluster and add 1 host
2. Create storage domain and attach it to DC
3. Create additonal storage domain (not attached to DC)
4. Destroy storage (directly on the storage server)
5. When host is in non-operational move it to maintenance and force-remove DC
6. Destroy remaining storage domain

Actual results:
After repeating this procedure several times I was suddenly unable to re-add a storage domain with the same name as the additional domain. Checking the DB showed that the storage domain entry remained in the storage_domain_static table, but not in the storage_domains table

Expected results:
destroying a domain should remove it completely from the DB

Additional info:

Comment 1 Alissa 2013-07-09 12:40:03 UTC
What did you mean by storage_domains table? there is no such table in the db... storage_domains is a view. in which specific table did you see the records remained?

Also, you wrote "After repeating this procedure several times" -is this consistent? How many times the procedure needs to be repeated for reproduction?

Comment 3 Alissa 2013-08-19 14:47:22 UTC
Some information and the cause for the problem in the bug:
relevant storage domain id is feb6aa0e-4302-42b9-86a7-84d902fea07e
1. ReconstructMasterDomainCommand starts, compensates storage_domain_static. 
2. ForceRemoveStorageDomainCommand starts.
It removes the storage domain from the db.
3. ReconstructMasterDomainCommand fails.
It triggers the compensation of the storage domain. Since it was removed from db by step 2 (ForceRemoveStorageDomainCommand), the compensation gets storage domain inserted  again to storage_domain_static table.

Comment 5 Allon Mureinik 2014-02-16 09:09:22 UTC
Liron, was this solved as part of your on reconstruct?

Comment 6 Liron Aravot 2014-03-02 13:51:29 UTC
Allon, no.
I provided patches to solve the issue, added to the tracker.

Comment 7 Kevin Alon Goldblatt 2014-07-07 07:19:48 UTC
All instances of the storage domains were removed from the database after performing the above scenario ONCE. Is this sufficient to move this defect to Verified? Gadi?

Comment 8 Gadi Ickowicz 2014-07-08 08:16:41 UTC
(In reply to Kevin Alon Goldblatt from comment #7)
> All instances of the storage domains were removed from the database after
> performing the above scenario ONCE. Is this sufficient to move this defect
> to Verified? Gadi?

If it worked, then I guess so

Comment 9 Ori Gofen 2014-07-29 07:51:16 UTC
verified on oVirt beta.2

Comment 11 Allon Mureinik 2015-02-16 19:10:13 UTC
RHEV-M 3.5.0 has been released, closing this bug.