[Solved] A NTP oddness on just one server

[MikeLieberman]

I have about fifty devices, of which there are five severs and six workstations. NTP seems to be running fine on all but one server. I am lost as why it is having problems. In the graphic here,


Note all the times on three are pretty much right where they are supposed to be as I jumped from one to the next to issue the command. However note the red box. That one is way off! All sit on this network and of the four, that odd one and two others are VMs on the same host! All are running Debian 12. All ar set to the same Timezone. All but the NTP server and set to "NTP service: active"

All four have the same RTC time: Tue 2024-04-23 18:55

But the odd one has Universal time matching it's RTC time while the others have Universal time eight hours back. Also not local time on the odd one is ahead eight hours.

RTC in local TZ: no is the same for all four.

Any guidance will be appreciated.

[Aki]

Hello,

You may find this earlier post helpful:

• Re: Clock is temporarily off after every wake-from-sleep/boot

[MikeLieberman]

@Aki ,

This is not a sleep thing. It is consistent and consistently off by exactly eight hours (or sixteen when considering it in another way). Plus, what is off/switched is the local and universal time, and even then it is wrong. Compare those two fields with the proper results from NTP on the other two using NTP.

[Aki]

Hi all,

I suggested my previous post because of its content, not the topic.

Anyway, according to your first post, the correct RTC (real time clock) time (3 installations over four) is about:

• Tue 2024-04-23 10:53:xx UTC

The RTC of your "Network Monitoring" installation is out of date (8 hours forward):

• Tue 2024-03-23 18:55:xx UTC

Therefore, I suppose you need to manually set the RTC of your "Network Monitoring" installation to the correct UTC time.

What is the virtualization software you are using ?

Hope this helps.

--
note: please, don't post images: use text for terminal output. Thanks. :wink:

[MikeLieberman]

@Aki

OK, so color me confused. The Realtime clock is typically set as updated by the NTP, if NTP is active, not the other way around. Is that not right?

In any case, the universal time is set by the NTP with the TimeZone offset, not the RTC, assuming NTP is active. Is that wrong?

As far as I know, as RTC can be very wrong, or just 'off'. It is NTP that keeps everything where it should be. How can Univerisal time be so off? Since when is it set by an incorrect RTC [by eight hours] and then adjusted another eight hours up for Local time?

I am using Oracle VirtualBox. I know all the criticisms of VirtualBox, but it works well for me.

[MikeLieberman]

@Aki

Also. as the only true RTC is on the VM host and the host has it right, as do all the VMs, how would I be able to set the RTC as a VM doesn't really have a RTC?

Here is the VM

Code: Select all

root@im:/home/mike# timedatectl
               Local time: Thu 2024-04-25 04:50:36 PST
           Universal time: Wed 2024-04-24 20:50:36 UTC
                 RTC time: Wed 2024-04-24 20:50:36
                Time zone: Asia/Manila (PST, +0800)
System clock synchronized: no
              NTP service: active
          RTC in local TZ: no

Here is the HOST

Code: Select all

mike@vm-server:~$ timedatectl
               Local time: Wed 2024-04-24 20:51:21 PST
           Universal time: Wed 2024-04-24 12:51:21 UTC
                 RTC time: Wed 2024-04-24 12:51:21
                Time zone: Asia/Manila (PST, +0800)
System clock synchronized: yes
              NTP service: active
          RTC in local TZ: no

[CwF]

how would I be able to set the RTC as a VM doesn't really have a RTC?

in the host, to set a vm's time to the host rtc

Code: Select all

virsh -c qemu:///system domtime --sync vmname 

edit: opps, I should learn to read!

[Aki]

Hello,

Desktops and notebooks usually have an internal "hardware" real time clock (RTC) that is powered by an internal battery. This is your computer's reference real-time clock. It ticks even if the computer is off (as long as it is powered by its battery). Virtual machines (VMs) emulates the RTC for the guest operating system. You can check system logs about RTC with the following command:

Code: Select all

sudo journalctl -b --grep rtc

The operating system (Debian) uses a second clock, the so-called "system clock" (software clock maintained by the kernel). It is initialized from the RTC at boot time by the kernel. The operating system can then update the system time (within certain limits) from another networked computer using "network time servers".

The Real time clock is typically set as updated by the NTP, if NTP is active [..] Is that not right?

The Real time clock (RTC) is not usually updated by the network time protocol (NTP) client.

In any case, the universal time is set by the NTP with the TimeZone offset, not the RTC, assuming NTP is active. Is that wrong?

The Universal Coordinated Time (UTC aka GMC) in the output of the timedatectl is the UTC representation of the system time.

[..] How can Univerisal time be so off? Since when is it set by an incorrect RTC [by eight hours] and then adjusted another eight hours up for Local time? [..]

The system time (expressed as UTC) is the same as RTC time, because RTC time is usually is valorized as UTC in *nix systems.

The network time protocol (NTP) client does not sync the "system time" if the offset between current system time and "network time protocol" time is out of sync more then 1000 seconds (as in your case). Therefore, the system time remains set to the RTC time, plus the timezones. Since your RTC is wrong, the system time is wrong, too.

I am using Oracle VirtualBox. [..] Also. [..] how would I be able to set the RTC as a VM doesn't really have a RTC?

It depends on the VM you are using.

According to the Virtual Box User Manual [1]:

[..]
3.5. System Settings
The System category groups various settings that are related to the basic hardware that is presented to the virtual machine.
Note [..] The following tabs are available.

3.5.1. Motherboard Tab
On the Motherboard tab, you can configure virtual hardware that would normally be on the motherboard of a real computer.
[..]

• Hardware Clock in UTC Time: If selected, Oracle VM VirtualBox will report the system time in UTC format to the guest instead of the local (host) time . This affects how the virtual real-time clock (RTC) operates and may be useful for UNIX-like guest OSes, which typically expect the hardware clock to be set to UTC

[..]

Perhaps, you need to configure properly your virtual box guest. Otherwise, you can check guest's system logs.

---
[1] https://www.virtualbox.org/manual/ch03.html

[MikeLieberman]

@Aki

I give up.

• There are four VMs.

• Three are right and agree with the host.

• All four are configured the same way.

• All are using the same Debian code,

• The system board's virtualization option is enabled via BIOS.

Thanks for the information. It was interesting.

This will remain 'unsolved.'

[donald]

I don't work with a lot of VMs but this is very interesting.

The misbehaving instance on the same machine is set to take its time from its own NTP agent?

Do you have restrict default kod nomodify notrap in each of the ntp.conf files?

[MikeLieberman]

@donald

I didn't touch the ntp.conf files if the exist on the VMs. They are as they were from the installer when installing the VMs.

Our DHCP server, which all the other systems use, also tells each client to get its time from the network time server on this network. All devices so, including CCTV devices which are blocked 100% at the firewall from seeing the outside world. Once I forced all devices (including the blocked ones) to use the local service, all devices finally had the same time, So to that extent, I know the system works.

[MikeLieberman]

OK I solved it I guess.
First, I set the time-zone to GMT
The next day I looked via timedatectl and it was at real GTM
I set it to GM+8 and is responed set the clock to GMT -8
So I set to GMT -8 and yeah, now it is a GMT +8 and is giving the correct time.

Code: Select all

mike@im:~$ timedatectl
               Local time: Wed 2024-05-01 18:52:34 +08
           Universal time: Wed 2024-05-01 10:52:34 UTC
                 RTC time: Wed 2024-05-01 10:52:34
                Time zone: Etc/GMT-8 (+08, +0800)
System clock synchronized: no
              NTP service: active
          RTC in local TZ: no

[Aki]

Hello,

Is this a VM guest time ?

I can see that the RTC time has been corrected, so hopefully the local time with its time zone is now correct.

Did you reconfigured your guest VM ?

Can you please report the terminal commands you gave ?

[donald]

Hello,

Is this a VM guest time ?

I can see that the RTC time has been corrected, so hopefully the local time with its time zone is now correct.

Did you reconfigured your guest VM ?

Can you please report the terminal commands you gave ?

Also if not a bother could you also share the configuration files for the NTP server/clients?

This is super curious.