Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.

Bug 236307

Summary: inconsistent system time
Product: [Fedora] Fedora Reporter: Ask Bjørn Hansen <ask>
Component: kernel-xenAssignee: Eduardo Habkost <ehabkost>
Status: CLOSED WONTFIX QA Contact: Virtualization Bugs <virt-bugs>
Severity: high Docs Contact:
Priority: medium    
Version: 6CC: bstein, xen-maint
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2008-02-27 00:04:40 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Description Ask Bjørn Hansen 2007-04-12 23:35:54 UTC
Description of problem:

With the latest FC6 Xen kernel the system time is completely unreliable, jumping back and forth(!) even 
in dom0.

[ask@app2 ~]$ uname -a
Linux app2.la.sol 2.6.20-1.2933.fc6xen #1 SMP Mon Mar 19 11:17:46 EDT 2007 x86_64 x86_64 
x86_64 GNU/Linux
[ask@app2 ~]$ while `/bin/true`; do date; sleep 1; done
Thu Apr 12 16:33:16 PDT 2007
Thu Apr 12 16:33:17 PDT 2007
Thu Apr 12 16:33:18 PDT 2007
Thu Apr 12 16:33:18 PDT 2007
Thu Apr 12 16:33:20 PDT 2007
Thu Apr 12 16:33:21 PDT 2007
Thu Apr 12 16:33:22 PDT 2007
Thu Apr 12 16:33:23 PDT 2007
Thu Apr 12 16:33:24 PDT 2007
Thu Apr 12 16:33:24 PDT 2007
Thu Apr 12 16:33:26 PDT 2007
Thu Apr 12 16:33:27 PDT 2007
Thu Apr 12 16:33:28 PDT 2007
Thu Apr 12 16:33:28 PDT 2007
Thu Apr 12 16:33:30 PDT 2007

(the seconds that appears to be okay are only that because it seems to more or less average out to be 
correct over time - each line is very "jumpy").

On the non-xen kernel I get:

[root@app3 ~]# uname -a
Linux app3.la.sol 2.6.20-1.2933.fc6 #1 SMP Mon Mar 19 11:00:19 EDT 2007 x86_64 x86_64 x86_64 
GNU/Linux
[root@app3 ~]# while `/bin/true`; do date; sleep 1; done
Thu Apr 12 16:20:28 PDT 2007
Thu Apr 12 16:20:29 PDT 2007
Thu Apr 12 16:20:30 PDT 2007
Thu Apr 12 16:20:31 PDT 2007


Version-Release number of selected component (if applicable):


How reproducible:

Always.

Comment 1 Ask Bjørn Hansen 2007-04-12 23:43:34 UTC
The two boxes above are dual quad-core Intel Xeon 2.33Ghz; I see the same problem on a dual dual-core 
Intel Xeon 2.00Ghz.

http://www.siliconmechanics.com/i5917/Xeon-Server.php

dmesg of xen kernel at http://tmp.askask.com/2007/04/app2-xen.dmesg.txt

dmesg of no-xen kernel at http://tmp.askask.com/2007/04/app3-noxen.dmesg.txt

Comment 2 Ask Bjørn Hansen 2007-04-12 23:45:00 UTC
FWIW, it seems to get better if I disable apic ...

Comment 3 Eduardo Habkost 2007-04-13 12:29:06 UTC
A test using 'date --rfc-3339 ns' in the place of the 'date' command may make 
the problem more visible. Could you do this test?

Comment 4 Ask Bjørn Hansen 2007-04-13 19:22:48 UTC
very cool, didn't know about that option.

Here's a non-xen kernel:

[root@app3 ~]# while `/bin/true`; do date --rfc-3339 ns; sleep 1; done
2007-04-13 12:19:42.766020000-07:00
2007-04-13 12:19:43.770184000-07:00
2007-04-13 12:19:44.774319000-07:00
2007-04-13 12:19:45.777746000-07:00
2007-04-13 12:19:46.781122000-07:00
2007-04-13 12:19:47.790459000-07:00
2007-04-13 12:19:48.794074000-07:00
2007-04-13 12:19:49.797861000-07:00
2007-04-13 12:19:50.801986000-07:00
2007-04-13 12:19:51.805114000-07:00

Here's from the xen kernel:

[root@app2 ~]# while /bin/true; do date --rfc-3339 ns; sleep 1; done
2007-04-13 12:20:15.672158000-07:00
2007-04-13 12:20:16.643917000-07:00
2007-04-13 12:20:17.672203000-07:00
2007-04-13 12:20:17.643942000-07:00
2007-04-13 12:20:19.006156000-07:00
2007-04-13 12:20:19.671671000-07:00
2007-04-13 12:20:20.672290000-07:00
2007-04-13 12:20:21.671717000-07:00
2007-04-13 12:20:21.656193000-07:00
2007-04-13 12:20:22.644064000-07:00
2007-04-13 12:20:23.672360000-07:00
2007-04-13 12:20:24.672386000-07:00
2007-04-13 12:20:25.582588000-07:00
2007-04-13 12:20:26.672925000-07:00
2007-04-13 12:20:27.672468000-07:00
2007-04-13 12:20:28.582667000-07:00
2007-04-13 12:20:28.672996000-07:00
2007-04-13 12:20:29.671917000-07:00
2007-04-13 12:20:29.656400000-07:00
2007-04-13 12:20:31.006121000-07:00
2007-04-13 12:20:31.672586000-07:00
2007-04-13 12:20:33.672630000-07:00


I booted the xen kernel with apic=off to see if it made a difference, but if it did it wasn't much:

title Fedora Core (2.6.20-1.2933.fc6xen)
        root (hd0,0)
        kernel /xen.gz-2.6.20-1.2933.fc6 com1=9600,8n1 com2=115200,8n1 apic=off
        module /vmlinuz-2.6.20-1.2933.fc6xen ro root=/dev/vg0/root console=tty0 console=ttyS0
        module /initrd-2.6.20-1.2933.fc6xen.img



Comment 5 Ask Bjørn Hansen 2007-04-17 06:04:43 UTC
Here's with the new kernel (notice how it starts out going backwards in time ...)

[root@app2 ~]# while /bin/true; do date --rfc-3339 ns; sleep 1; done
2007-04-16 23:02:48.981407000-07:00
2007-04-16 23:02:48.930131000-07:00
2007-04-16 23:02:50.820784000-07:00
2007-04-16 23:02:50.930164000-07:00
2007-04-16 23:02:51.791174000-07:00
2007-04-16 23:02:51.953771000-07:00
2007-04-16 23:02:52.819448000-07:00
2007-04-16 23:02:53.019187000-07:00
2007-04-16 23:02:53.797581000-07:00
2007-04-16 23:02:54.981535000-07:00
2007-04-16 23:02:54.953835000-07:00
2007-04-16 23:02:55.930285000-07:00
2007-04-16 23:02:56.791280000-07:00
2007-04-16 23:02:56.797647000-07:00
2007-04-16 23:02:57.981602000-07:00
2007-04-16 23:02:57.953897000-07:00
2007-04-16 23:02:58.930349000-07:00
2007-04-16 23:02:59.791346000-07:00
2007-04-16 23:02:59.904411000-07:00
2007-04-16 23:03:00.930392000-07:00
2007-04-16 23:03:01.791389000-07:00
2007-04-16 23:03:02.797772000-07:00
2007-04-16 23:03:02.821041000-07:00
2007-04-16 23:03:03.954014000-07:00
2007-04-16 23:03:04.904499000-07:00
2007-04-16 23:03:04.930476000-07:00
2007-04-16 23:03:05.791478000-07:00
2007-04-16 23:03:06.797861000-07:00
2007-04-16 23:03:06.821126000-07:00
2007-04-16 23:03:07.954106000-07:00

Comment 6 Ask Bjørn Hansen 2007-04-17 06:06:11 UTC
[root@app2 ~]# uname -a
Linux app2.la.sol 2.6.20-1.2944.fc6xen #1 SMP Tue Apr 10 18:03:37 EDT 2007 x86_64 x86_64 x86_64 
GNU/Linux

If it can help I think we can arrange for something from RedHat to get access to one of these systems.  
(The boxes have an IPMI KVM and a serial console and IPDU power, so they are easy to mess with 
remotely).


 - ask

Comment 7 Eduardo Habkost 2007-06-05 22:08:57 UTC
Hi,

Sorry for taking so long to take a look on this bug.

Could you check the results of the same command (while /bin/true; do 
date --rfc-3339 ns; sleep 1; done), using the kernel command line 
option "independent_wallclock"?

Also, could you attach dmesg output when using the following command line 
options:
- Using both: independent_wallclock permitted_clock_jitter=0
- Using only: permitted_clock_jitter=0

Probably it will flood your console with messages, but the information from 
them will be useful.

Comment 8 Red Hat Bugzilla 2007-07-25 01:40:27 UTC
change QA contact

Comment 9 Ask Bjørn Hansen 2007-08-08 01:33:00 UTC
Hi Eduardo,

Darn - I missed your comment earlier.

FWIW changing independent_wallclock in the dom0 (with sysctl) doesn't seem to make a difference.  I'll 
try with permitted_clock_jitter=0 later.


 - ask

Linux app3.la.sol 2.6.20-2925.13.fc7xen #1 SMP Tue Jul 17 10:38:19 EDT 2007 x86_64 x86_64 
x86_64 GNU/Linux

[ask@app3 ~]$  while /bin/true; do date --rfc-3339 ns; sleep 1; done
2007-08-07 18:31:35.155215000-07:00
2007-08-07 18:31:37.070061000-07:00
2007-08-07 18:31:38.081759000-07:00
2007-08-07 18:31:39.317794000-07:00
2007-08-07 18:31:41.120665000-07:00
2007-08-07 18:31:42.932862000-07:00
2007-08-07 18:31:44.005449000-07:00
2007-08-07 18:31:45.029116000-07:00
2007-08-07 18:31:46.037569000-07:00

Comment 10 Eduardo Habkost 2007-09-27 20:05:43 UTC
The FC-6 updates system haven't added an automatic note on this bug and I 
haven't noticed. A patch was included since 2.6.20-1.3001.fc6 that should help 
fix this issue.

Could you check what are the results when using 2.6.20-1.3001.fc6 or newer?

If you are using Fedora 7, the patch was added on 2.6.20-2930.fc7.

Comment 11 Chris Lalancette 2008-02-27 00:04:40 UTC
This report targets FC6, which is now end-of-life.

Please re-test against Fedora 7 or later, and if the issue persists, open a new bug.

Thanks