Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1171491 - [Rhev-Upgrade] NPE when refreshing a host (VdsUpdateRunTimeInfo) after upgrade
Summary: [Rhev-Upgrade] NPE when refreshing a host (VdsUpdateRunTimeInfo) after upgrade
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: ovirt-engine
Version: 3.5.0
Hardware: Unspecified
OS: Unspecified
unspecified
urgent
Target Milestone: ---
: 3.5.0
Assignee: Doron Fediuck
QA Contact: Nikolai Sednev
URL:
Whiteboard: sla
: 1171824 (view as bug list)
Depends On:
Blocks: 1157378 rhev35rcblocker rhev35gablocker
TreeView+ depends on / blocked
 
Reported: 2014-12-07 17:21 UTC by Ori Gofen
Modified: 2016-05-26 01:49 UTC (History)
17 users (show)

Fixed In Version: org.ovirt.engine-root-3.5.0-25
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2015-02-17 17:14:19 UTC
oVirt Team: SLA
Target Upstream Version:


Attachments (Terms of Use)
logs (deleted)
2014-12-07 17:21 UTC, Ori Gofen
no flags Details


Links
System ID Priority Status Summary Last Updated
oVirt gerrit 36016 master MERGED core: fixed NPE when vdsm returns emtpy balloon info Never
oVirt gerrit 36066 ovirt-engine-3.5 MERGED core: fixed NPE when vdsm returns emtpy balloon info Never

Description Ori Gofen 2014-12-07 17:21:40 UTC
Created attachment 965602 [details]
logs

Description of problem:
The flow is Part of Rhev-Upgrade simulation:
A costumer is having 3.4 DC with iSCSi and NFS domains, DC has 2X clusters

3.4 DC1
  --> HC1_1
      |
       --> brown-vdsc (3.4)
       --> black-vdsb (3.4)
  --> HC1_2
      |
       --> alma04 (3.5)

after alma04 upgrade, The costumer upgrade HC1_2 cluster policy to 3.5 as well.
after cluster's upgrade, engine attempts to create and update OvdStore images on DC1, operation fails and throws NPE every $OvfUpdateIntervalInMinutes.

from logs:

2014-12-07 18:58:29,610 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-73) Failure to refresh Vds runtime info: java.lang.NullPointerException

2014-12-07 18:58:29,611 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-47) [37353cb1] Successfully removed unneeded template/vm OVFs in Data Center DC1_2
2014-12-07 18:58:29,614 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-73) Failed to refresh VDS , vds = 7ccb3feb-8c10-4a33-b425-b0126479d487 : alma04.qa.l
ab.tlv.redhat.com, error = java.lang.NullPointerException, continuing.: java.lang.NullPointerException

2014-12-07 18:58:32,872 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-90) Failure to refresh Vds runtime info: java.lang.NullPointerException

2014-12-07 18:58:32,876 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-90) Failed to refresh VDS , vds = 7ccb3feb-8c10-4a33-b425-b0126479d487 : alma04.qa.l
ab.tlv.redhat.com, error = java.lang.NullPointerException, continuing.: java.lang.NullPointerException

2014-12-07 18:58:36,126 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-48) Failure to refresh Vds runtime info: java.lang.NullPointerException

2014-12-07 18:58:36,129 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-48) Failed to refresh VDS , vds = 7ccb3feb-8c10-4a33-b425-b0126479d487 : alma04.qa.l
ab.tlv.redhat.com, error = java.lang.NullPointerException, continuing.: java.lang.NullPointerException

2014-12-07 18:58:39,387 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-70) Failure to refresh Vds runtime info: java.lang.NullPointerException

2014-12-07 18:58:39,392 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-70) Failed to refresh VDS , vds = 7ccb3feb-8c10-4a33-b425-b0126479d487 : alma04.qa.l
ab.tlv.redhat.com, error = java.lang.NullPointerException, continuing.: java.lang.NullPointerException

2014-12-07 18:58:40,309 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-61) Failure to refresh Vds runtime info: java.lang.NullPointerException

2014-12-07 18:58:40,326 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-61) Failed to refresh VDS , vds = 517e6de1-28b7-4f39-8589-4c82a4509235 : brown-vdsc.
qa.lab.tlv.redhat.com, error = java.lang.NullPointerException, continuing.: java.lang.NullPointerException

2014-12-07 18:58:42,243 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-60) Failure to refresh Vds runtime info: java.lang.NullPointerException

2014-12-07 18:58:42,258 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-60) Failed to refresh VDS , vds = 133687ed-7121-4f09-9f28-62f0208af60a : black-vdsb.
qa.lab.tlv.redhat.com_H, error = java.lang.NullPointerException, continuing.: java.lang.NullPointerException

2014-12-07 18:58:42,555 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-91) Failure to refresh Vds runtime info: java.lang.NullPointerException

2014-12-07 18:58:42,574 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-91) Failed to refresh VDS , vds = 7ccb3feb-8c10-4a33-b425-b0126479d487 : alma04.qa.l
ab.tlv.redhat.com, error = java.lang.NullPointerException, continuing.: java.lang.NullPointerException

2014-12-07 18:58:45,849 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-71) Failure to refresh Vds runtime info: java.lang.NullPointerException

2014-12-07 18:58:45,853 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-71) Failed to refresh VDS , vds = 7ccb3feb-8c10-4a33-b425-b0126479d487 : alma04.qa.l
ab.tlv.redhat.com, error = java.lang.NullPointerException, continuing.: java.lang.NullPointerException


Version-Release number of selected component (if applicable):
mixed

How reproducible:
100%

Expected results:
engine should only try to create|update OvfStore images on 3.5 DC

Additional info:

Comment 1 Allon Mureinik 2014-12-08 11:20:43 UTC
Liron, please take a look?

Comment 2 Allon Mureinik 2014-12-08 13:08:40 UTC
Sorry, meant to assign to Liron - wrong window.

Comment 3 Liron Aravot 2014-12-08 14:39:39 UTC
The errors in the log are of VdsManager when trying to refresh the host, that's not part of the ovf update process.

moving to infra.

Comment 4 Oved Ourfali 2014-12-08 14:53:58 UTC
Does it continue like that forever?
Does it eventually gets fixed?

Comment 5 Oved Ourfali 2014-12-08 14:57:06 UTC
Log shows that it is part of the baloon check. Moving to sla for further examination.

2014-12-07 17:42:35,128 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-67) Failed to refresh VDS , vds = b2304cbd-2e0f-487d-9389-617ca306c94d : alma03.qa.lab.tlv.redhat.com, error = java.lang.NullPointerException, continuing.: java.lang.NullPointerException
	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.proceedBalloonCheck(VdsUpdateRunTimeInfo.java:1481) [vdsbroker.jar:]
	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:948) [vdsbroker.jar:]
	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:658) [vdsbroker.jar:]
	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:494) [vdsbroker.jar:]
	at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:]
	at sun.reflect.GeneratedMethodAccessor85.invoke(Unknown Source) [:1.7.0_71]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_71]
	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_71]
	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]

Comment 6 Ori Gofen 2014-12-10 07:30:35 UTC
the state still continues, it seems to me it will stabilize upon upgrade completion, currently still have running tests on mixed environment(3.4-3.5)

Comment 7 Doron Fediuck 2014-12-15 13:59:11 UTC
*** Bug 1171824 has been marked as a duplicate of this bug. ***

Comment 8 Nikolai Sednev 2014-12-16 09:04:43 UTC
Works for me on these components: 
libvirt-0.10.2-46.el6_6.2.x86_64
vdsm-4.16.8.1-3.el6ev.x86_64
ovirt-hosted-engine-setup-1.2.1-8.el6ev.noarch
sanlock-2.8-1.el6.x86_64
ovirt-host-deploy-1.3.0-2.el6ev.noarch
ovirt-hosted-engine-ha-1.2.4-3.el6ev.noarch
mom-0.4.1-4.el6ev.noarch
rhevm-3.5.0-0.25.el6ev.noarch

Comment 10 Eyal Edri 2015-02-17 17:14:19 UTC
rhev 3.5.0 was released. closing.


Note You need to log in before you can comment on or make changes to this bug.