cockroach service failing: clock synchronization error

[leftwo]

On madrid, after an install of R12, the cockroach zone failed to start the cockroach service.

[leftwo]

In the first cockroachdb log, we can see things starting and then fail:

BRM42220046 # cat /pool/ext/141bf2e7-a86c-48b6-a99c-6a9063d1ffe3/crypt/debug/oxz_cockroachdb_6aaaa6ed-0f77-481b-b17d-c964913399f6/oxide-cockroachdb:default.log.1740523225 | looker -o
 long    

Gives us:

[ Feb 25 22:40:03 Enabled. ]                                                                                                                                                          
[ Feb 25 22:40:03 Rereading configuration. ]                                                                                                                                          
[ Feb 25 22:40:03 Rereading configuration. ]                                                                                                                                          
[ Feb 25 22:40:05 Executing start method ("/opt/oxide/lib/svc/manifest/cockroachdb.sh"). ]                                                                                            
+ set -o errexit                                                                                                                                                                      
+ set -o pipefail                                                                                                                                                                     
+ . /lib/svc/share/smf_include.sh                                                                                                                                                     
++ SMF_EXIT_OK=0                                                                                                                                                                      
++ SMF_EXIT_NODAEMON=94                                                                                                                                                               
++ SMF_EXIT_ERR_FATAL=95                                                                                                                                                              
++ SMF_EXIT_ERR_CONFIG=96                                                                                                                                                             
++ SMF_EXIT_MON_DEGRADE=97                                                                                                                                                            
++ SMF_EXIT_MON_OFFLINE=98                                                                                                                                                            
++ SMF_EXIT_ERR_NOSMF=99                                                                                                                                                              
++ SMF_EXIT_ERR_PERM=100                                                                                                                                                              
++ svcprop -c -p config/listen_addr svc:/oxide/cockroachdb:default                                                                                                                    
+ LISTEN_ADDR='[fd00:1122:3344:103::3]:32221'                                                                                                                                         
++ svcprop -c -p config/store svc:/oxide/cockroachdb:default                                                                                                                          
+ DATASTORE=/data                                                                                                                                                                     
++ /opt/oxide/internal-dns-cli/bin/dnswait cockroach                                                                                                                                  
++ head -n 5                                                                                                                                                                          
++ tr '\n' ,                                                                                                                                                                          
note: configured to log to "/dev/stderr"                                                                                                                                              
2025-02-25 22:40:05.104Z INFO dnswait/3558 on oxz_cockroachdb_6aaaa6ed-0f77-481b-b17d-c964913399f6: using system configuration                                                        
+ JOIN_ADDRS=30308679-49dc-4b82-9075-61dae22fdd86.host.control-plane.oxide.internal.:32221,5a52945b-3069-4431-aa34-37c3aefc5c4a.host.control-plane.oxide.internal.:32221,6aaaa6ed-0f77
-481b-b17d-c964913399f6.host.control-plane.oxide.internal.:32221,adb311ce-135b-42e5-b0c4-4a7baa16a14c.host.control-plane.oxide.internal.:32221,da120202-a298-4c16-bcef-8de396abbe7d.ho
st.control-plane.oxide.internal.:32221,                                                                                                                                               
+ [[ -z 30308679-49dc-4b82-9075-61dae22fdd86.host.control-plane.oxide.internal.:32221,5a52945b-3069-4431-aa34-37c3aefc5c4a.host.control-plane.oxide.internal.:32221,6aaaa6ed-0f77-481b
-b17d-c964913399f6.host.control-plane.oxide.internal.:32221,adb311ce-135b-42e5-b0c4-4a7baa16a14c.host.control-plane.oxide.internal.:32221,da120202-a298-4c16-bcef-8de396abbe7d.host.co
ntrol-plane.oxide.internal.:32221, ]]                                                                                                                                                 
+ args=('--insecure' '--listen-addr' "$LISTEN_ADDR" '--http-addr' '127.0.0.1:8080' '--store' "$DATASTORE" '--join' "$JOIN_ADDRS")                                                     
+ exec /opt/oxide/cockroachdb/bin/cockroach start --insecure --listen-addr '[fd00:1122:3344:103::3]:32221' --http-addr 127.0.0.1:8080 --store /data --join 30308679-49dc-4b82-9075-61d
ae22fdd86.host.control-plane.oxide.internal.:32221,5a52945b-3069-4431-aa34-37c3aefc5c4a.host.control-plane.oxide.internal.:32221,6aaaa6ed-0f77-481b-b17d-c964913399f6.host.control-pla
ne.oxide.internal.:32221,adb311ce-135b-42e5-b0c4-4a7baa16a14c.host.control-plane.oxide.internal.:32221,da120202-a298-4c16-bcef-8de396abbe7d.host.control-plane.oxide.internal.:32221, 
[ Feb 25 22:40:05 Method "start" exited with status 0. ]                                                                                                                              
*                                                                                                                                                                                     
* WARNING: ALL SECURITY CONTROLS HAVE BEEN DISABLED!                                                                                                                                  
*                                                                                                                                                                                     
* This mode is intended for non-production testing only.                                                                                                                              
*                                                                                                                                                                                     
* In this mode:                                                                                                                                                                       
* - Your cluster is open to any client that can access fd00:1122:3344:103::3.                                                                                                         
* - Intruders with access to your machine or network can observe client-server traffic.                                                                                               
* - Intruders can log in without password and read or write any data in the cluster.                                                                                                  
* - Intruders can consume all your server's resources and cause unavailability.            
*                           
*                                           
* INFO: To start a secure server without mandating TLS for clients,
* consider --accept-sql-without-tls instead. For other options, see:                      
*               
* - https://go.crdb.dev/issue-v/53404/v22.1
* - https://www.cockroachlabs.com/docs/v22.1/secure-a-cluster.html
*
*
* INFO: initial startup completed.
* Node will now attempt to join a running cluster, or wait for `cockroach init`.
* Client connections will be accepted after this completes successfully.
* Check the log file(s) for progress. 
*
[ Feb 25 22:40:23 Stopping because all processes in service exited. ]

Looking at the full log, we see it start/stop a few times then give up:

BRM42220046 # cat /pool/ext/141bf2e7-a86c-48b6-a99c-6a9063d1ffe3/crypt/debug/oxz_cockroachdb_6aaaa6ed-0f77-481b-b17d-c964913399f6/oxide-cockroachdb:default.log.1740523225 | looker -o long | egrep "Executing|maintenance"
[ Feb 25 22:40:05 Executing start method ("/opt/oxide/lib/svc/manifest/cockroachdb.sh"). ]
[ Feb 25 22:40:23 Executing stop method (:kill). ]
[ Feb 25 22:40:23 Executing start method ("/opt/oxide/lib/svc/manifest/cockroachdb.sh"). ]
[ Feb 25 22:40:24 Executing stop method (:kill). ]
[ Feb 25 22:40:24 Executing start method ("/opt/oxide/lib/svc/manifest/cockroachdb.sh"). ]
[ Feb 25 22:40:24 Executing stop method (:kill). ]
[ Feb 25 22:40:24 Executing start method ("/opt/oxide/lib/svc/manifest/cockroachdb.sh"). ]
[ Feb 25 22:40:25 Executing stop method (:kill). ]
[ Feb 25 22:40:25 Restarting too quickly, changing state to maintenance. ]

From cockroach logs, specifically cockroach.oxzcockroachdb6aaaa6ed-0f77-481b-b17d-c964913399f6.root.2025-02-25T22_40_08Z.003564.log we find that a clocksync problem:

I250225 22:40:23.077044 237 server/init.go:286 <E2><8B><AE> [n?] 35  joined cluster <E2><80><B9>557af483-9c59-429e-83f2-aadb71e9effe<E2><80><BA> through join rpc
I250225 22:40:23.077143 237 server/init.go:287 <E2><8B><AE> [n?] 36  received node ID: 4
I250225 22:40:23.077164 237 server/init.go:288 <E2><8B><AE> [n?] 37  received cluster version: 22.1
W250225 22:40:23.078213 237 2@gossip/gossip.go:1533 <E2><8B><AE> [n4] 38  no incoming or outgoing connections
I250225 22:40:23.078261 747 1@server/server.go:1319 <E2><8B><AE> [n4] 39  connecting to gossip network to verify cluster ID <E2><80><B9>"557af483-9c59-429e-83f2-aadb71e9effe"<E2><80
<BA>
I250225 22:40:23.079135 237 gossip/gossip.go:401 <E2><8B><AE> [n4] 40  NodeDescriptor set to <E2><80><B9>node_id:4 address:<network_field:"tcp" address_field:"[fd00:1122:3344:103::3]:32221" > attrs:<> locality:<> ServerVersion:<major_val:22 minor_val:1 patch:0 internal:0 > build_tag:"v22.1.22-27-g76e176e260" started_at:1740523223078457595 cluster_name:"" sql_address:<network_field:"tcp" address_field:"[fd00:1122:3344:103::3]:32221" > http_address:<network_field:"tcp" address_field:"127.0.0.1:8080" ><E2><80><BA>
I250225 22:40:23.096208 748 1@vendor/github.com/cockroachdb/circuitbreaker/circuitbreaker.go:322 <E2><8B><AE> [n4] 42  circuitbreaker: <E2><80><B9>gossip [fd00:1122:3344:103::3]:32221->30308679-49dc-4b82-9075-61dae22fdd86.host.control-plane.oxide.internal.:32221<E2><80><BA> tripped: stopped
I250225 22:40:23.096391 748 1@vendor/github.com/cockroachdb/circuitbreaker/circuitbreaker.go:447 <E2><8B><AE> [n4] 43  circuitbreaker: <E2><80><B9>gossip [fd00:1122:3344:103::3]:32221->30308679-49dc-4b82-9075-61dae22fdd86.host.control-plane.oxide.internal.:32221<E2><80><BA> event: <E2><80><B9>BreakerTripped<E2><80><BA>
W250225 22:40:23.096443 748 gossip/client.go:118 <E2><8B><AE> [n4] 44  failed to start gossip client to <E2><80><B9>30308679-49dc-4b82-9075-61dae22fdd86.host.control-plane.oxide.internal.:32221<E2><80><BA>: stopped
F250225 22:40:23.084704 751 1@server/server.go:248 <E2><8B><AE> [n4] 41  clock synchronization error: this node is more than 500ms away from at least half of the known nodes (0 of 1 are within the offset)

Later, we see that something has cleared the state and then things restarted and worked.

[ Feb 26 10:20:35 Leaving maintenance because clear requested. ]
[ Feb 26 10:20:35 Executing start method ("/opt/oxide/lib/svc/manifest/cockroachdb.sh"). ]

In the cockroach zone, I do see a svcadm clear cockroachdb in the command history, but I don't have timestamps:

root@oxz_cockroachdb_6aaaa6ed:~# history
    1  ls
    2  df -h
    3  cd /data
    4  ls
    5  cd logs
    6  ls
    7  ls -ltr
    8  tail -100 cockroach.oxzcockroachdb6aaaa6ed-0f77-481b-b17d-c964913399f6.root.2025-02-25T22_40_25Z.003722.log 
    9  cat $_
   10  date
   11  svcs -x
   12  svcadm clear cockroachdb
   13  svcs -x

[leftwo]

(from @jgallagher in chat, we have):
(This is from the ntp zone on the same sled as cockroachdb)

chrony tracking.log entries around that timestamp:

===================================================================================================================================
   Date (UTC) Time     IP Address   St   Freq ppm   Skew ppm     Offset L Co  Offset sd Rem. corr. Root delay Root disp. Max. error
===================================================================================================================================
1986-12-28 00:05:28 0.0.0.0          0     -0.000 1000000.000  0.000e+00 ?  0  0.000e+00  0.000e+00  1.000e+00  1.000e+00  1.500e+00
1986-12-28 00:06:47 fd00:1122:3344:101::11  4      0.000 1000000.000 -1.204e+09 N  1  1.355e-03  0.000e+00  2.658e-02  1.204e+09  1.500e+00
2025-02-25 22:40:05 fd00:1122:3344:101::11  4      0.000 1000000.000 -4.160e-01 N  1  1.902e-05  0.000e+00  2.658e-02  2.522e-01  1.204e+09
2025-02-25 22:41:10 fd00:1122:3344:101::11  4    -67.576     20.171 -4.398e-03 N  1  5.799e-05  2.397e-01  2.659e-02  9.528e-04  6.593e+01
2025-02-25 22:42:14 fd00:1122:3344:101::11  4    -61.383     34.865  1.280e-03 N  1  3.272e-04  7.091e-02  2.662e-02  1.236e-03  2.597e-01
2025-02-25 22:43:19 fd00:1122:3344:101::11  4    -43.763     34.801  2.478e-03 N  1  3.428e-04 -1.437e-04  2.658e-02  1.265e-03  8.697e-02
2025-02-25 22:44:24 fd00:1122:3344:101::11  4    -40.241     16.498  2.914e-04 N  1  2.837e-04  3.378e-09  2.663e-02  1.387e-03  1.970e-02
2025-02-25 22:45:28 fd00:1122:3344:101::11  4    -40.267      9.716 -3.181e-05 N  1  2.398e-04 -9.407e-05  2.663e-02  1.378e-03  1.613e-02
2025-02-25 22:46:33 fd00:1122:3344:101::11  4    -41.029      7.211 -1.967e-04 N  1  2.279e-04 -5.135e-05  2.656e-02  4.429e-04  1.545e-02
2025-02-25 22:47:37 fd00:1122:3344:101::11  4    -41.677      5.040 -1.694e-04 N  1  1.906e-04  8.418e-05  2.655e-02  3.780e-04  1.440e-02
2025-02-25 22:48:41 fd00:1122:3344:101::11  4    -42.973      3.031 -1.655e-04 N  1  5.362e-05  2.884e-05  2.657e-02  2.376e-04  1.430e-02
2025-02-25 22:49:46 fd00:1122:3344:101::11  4    -42.788      1.886  6.642e-05 N  1  4.663e-05 -8.695e-09  2.655e-02  2.315e-04  1.398e-02
2025-02-25 22:50:50 fd00:1122:3344:101::11  4    -42.510      2.734  8.860e-05 N  1  5.585e-05 -3.699e-06  2.655e-02  2.393e-04  1.376e-02
2025-02-25 22:51:55 fd00:1122:3344:101::11  4    -42.084      2.078  5.847e-05 N  1  5.007e-05  3.728e-09  2.656e-02  2.208e-04  1.386e-02

Which posits the question: did we start cockroach before chrony was sync'd?

[elaine-oxide]

I observed the same issue on dublin after installation of R11 (v11/rc1), where there were 2 cockroachdb svc's affected on BRM23230010. Clock skew/offset logged by chrony was too big at the time that the error message clock synchronization error: this node is more than 500ms away from at least half of the known nodes was logged in cockroachdb zone corresponding to each of the 2 cockroachdb svc's.

[leftwo]

I observed this behavior on a newer omicron commit (2bab8c321b56c05741b19577f1ab56575fcaa759)
In this situation, we had two cockroach zones on sled17 and both zones hit the issue:

root@oxz_cockroachdb_405fc494:~# svcs -x
svc:/oxide/cockroachdb:default (CockroachDB)
 State: maintenance since Tue Mar 11 01:37:08 2025
Reason: Restarting too quickly.
   See: http://illumos.org/msg/SMF-8000-L5
   See: /var/svc/log/oxide-cockroachdb:default.log
Impact: This service is not running.

[Connection to zone 'oxz_cockroachdb_405fc494-6337-4109-bb1c-348ea03d8deb' pts/3 closed]

BRM23230010 # zlogin oxz_cockroachdb_152bbdfe-54b0-41e5-92b5-7eaef7cbe37d
[Connected to zone 'oxz_cockroachdb_152bbdfe-54b0-41e5-92b5-7eaef7cbe37d' pts/3]
The illumos Project     helios-2.0.23256        March 2025
root@oxz_cockroachdb_152bbdfe:~# svcs -x
svc:/oxide/cockroachdb:default (CockroachDB)
 State: maintenance since Tue Mar 11 01:37:08 2025
Reason: Restarting too quickly.
   See: http://illumos.org/msg/SMF-8000-L5
   See: /var/svc/log/oxide-cockroachdb:default.log
Impact: This service is not running.
root@oxz_cockroachdb_152bbdfe:~# 

In the logs:

root@oxz_cockroachdb_152bbdfe:/data/logs# grep "clock synchronization error" cockroach.oxzcockroachdb152bbdfe-54b0-41e5-92b5-7eaef7cbe37d.root.2025-03-11T01_37_0*
cockroach.oxzcockroachdb152bbdfe-54b0-41e5-92b5-7eaef7cbe37d.root.2025-03-11T01_37_04Z.003529.log:F250311 01:37:05.965616 634 1@server/server.go:248 ⋮ [n4] 34  clock synchronization error: this node is more than 500ms away from at least half of the known nodes (0 of 1 are within the offset)
cockroach.oxzcockroachdb152bbdfe-54b0-41e5-92b5-7eaef7cbe37d.root.2025-03-11T01_37_06Z.003618.log:F250311 01:37:06.836411 565 1@server/server.go:248 ⋮ [n4] 36  clock synchronization error: this node is more than 500ms away from at least half of the known nodes (0 of 1 are within the offset)
cockroach.oxzcockroachdb152bbdfe-54b0-41e5-92b5-7eaef7cbe37d.root.2025-03-11T01_37_07Z.003638.log:F250311 01:37:07.699678 645 1@server/server.go:248 ⋮ [n4] 36  clock synchronization error: this node is more than 500ms away from at least half of the known nodes (0 of 1 are within the offset)
cockroach.oxzcockroachdb152bbdfe-54b0-41e5-92b5-7eaef7cbe37d.root.2025-03-11T01_37_08Z.003672.log:F250311 01:37:08.570965 574 1@server/server.go:248 ⋮ [n4] 36  clock synchronization error: this node is more than 500ms away from at least half of the known nodes (0 of 1 are within the offset)

Chrony tracking.log from the ntp zone on the same sled:

BRM23230010 # cat /pool/ext/dbb05ba7-a295-43ec-b438-8626d2c92e04/crypt/zone/oxz_ntp_6469bf76-7f44-4000-9e4d-0ac6778f0e26/root/var/log/chrony/tracking.log
===================================================================================================================================       
   Date (UTC) Time     IP Address   St   Freq ppm   Skew ppm     Offset L Co  Offset sd Rem. corr. Root delay Root disp. Max. error       
===================================================================================================================================       
1986-12-28 00:03:33 0.0.0.0          0     -0.000 1000000.000  0.000e+00 ?  0  0.000e+00  0.000e+00  1.000e+00  1.000e+00  1.500e+00      
1986-12-28 00:05:58 fd00:1122:3344:101::10  4      0.000 1000000.000 -1.206e+09 N  1  5.279e-04  0.000e+00  2.655e-02  1.206e+09  1.500e+00
2025-03-11 01:36:43 fd00:1122:3344:101::10  4      0.000 1000000.000  4.800e-01 N  1  3.738e-05  0.000e+00  2.656e-02  9.626e-01  1.206e+09
2025-03-11 01:37:48 fd00:1122:3344:102::10  4    -48.556    472.711 -2.197e-03 N  1  4.111e-05 -3.035e-01  2.656e-02  1.315e-03  6.634e+01
2025-03-11 01:38:53 fd00:1122:3344:102::10  4    -45.184    114.633  2.658e-04 N  1  3.794e-04 -1.254e-01  2.669e-02  6.739e-04  3.467e-01
2025-03-11 01:39:58 fd00:1122:3344:102::10  4    -50.412     34.339 -1.192e-03 N  1  3.150e-04  2.287e-06  2.661e-02  5.837e-04  1.472e-01
2025-03-11 01:41:02 fd00:1122:3344:102::10  4    -50.633     16.153 -3.043e-05 N  1  2.799e-04 -1.142e-07  2.665e-02  5.119e-04  1.736e-02
2025-03-11 01:42:07 fd00:1122:3344:102::10  4    -50.211      8.545  1.205e-04 N  1  2.125e-04  2.810e-05  2.659e-02  4.493e-04  1.499e-02
2025-03-11 01:43:12 fd00:1122:3344:102::10  4    -49.886      5.597  8.878e-05 N  1  1.784e-04 -6.570e-05  2.660e-02  6.380e-04  1.446e-02
2025-03-11 01:44:16 fd00:1122:3344:102::10  4    -49.477      4.069  1.200e-04 N  1  1.541e-04 -6.553e-05  2.662e-02  6.599e-04  1.452e-02
2025-03-11 01:45:22 fd00:1122:3344:102::10  4    -48.984      3.095  1.315e-04 N  1  1.288e-04 -3.560e-05  2.662e-02  6.357e-04  1.449e-02
2025-03-11 01:46:26 fd00:1122:3344:102::10  4    -48.372      2.100  1.311e-04 N  1  8.946e-05 -2.385e-05  2.662e-02  5.951e-04  1.438e-02
2025-03-11 01:47:32 fd00:1122:3344:102::10  4    -47.766      0.259  1.158e-04 N  1  8.578e-06  7.278e-09  2.659e-02  3.083e-04  1.427e-02
2025-03-11 01:48:36 fd00:1122:3344:102::10  4    -47.770      0.176 -2.017e-06 N  1  8.125e-06  5.332e-09  2.659e-02  3.017e-04  1.380e-02
2025-03-11 01:49:41 fd00:1122:3344:102::10  4    -47.774      0.146 -1.811e-06 N  1  8.595e-06  1.690e-06  2.659e-02  3.012e-04  1.368e-02
2025-03-11 01:50:46 fd00:1122:3344:102::10  4    -47.767      0.153 -1.049e-05 N  1  1.100e-05  2.540e-06  2.659e-02  3.189e-04  1.368e-02
2025-03-11 01:51:50 fd00:1122:3344:102::10  4    -47.775      0.138 -5.736e-06 N  1  1.117e-05  2.843e-06  2.659e-02  3.169e-04  1.370e-02
2025-03-11 01:52:55 fd00:1122:3344:102::10  4    -47.793      0.136 -1.064e-05 N  1  1.235e-05  4.131e-06  2.659e-02  3.225e-04  1.370e-02

[FelixMcFelix]

We seem to have bumped into this (or a similar issue) with the CRDB on sled 16 today during the dogfood update.

root@oxz_cockroachdb_e86845b5:~# svcs -xv
svc:/oxide/cockroachdb:default (CockroachDB)
 State: maintenance since Thu Jun 26 20:39:31 2025
Reason: Restarting too quickly.
   See: http://illumos.org/msg/SMF-8000-L5
   See: /var/svc/log/oxide-cockroachdb:default.log
Impact: This service is not running.

Chrony tracking.log from the same sled:

root@oxz_ntp_209b6213:~# head -n 100 /var/log/chrony/tracking.log
===================================================================================================================================
   Date (UTC) Time     IP Address   St   Freq ppm   Skew ppm     Offset L Co  Offset sd Rem. corr. Root delay Root disp. Max. error
===================================================================================================================================
1986-12-28 00:02:28 0.0.0.0          0     -0.000 1000000.000  0.000e+00 ?  0  0.000e+00  0.000e+00  1.000e+00  1.000e+00  1.500e+00
1986-12-28 00:02:36 fd00:1122:3344:104::e 11      0.000 1000000.000 -1.215e+09 N  1  2.329e-01  0.000e+00  2.091e-04  1.339e+09  1.500e+00
2025-06-26 20:36:43 fd00:1122:3344:104::e 11      0.000 1000000.000  2.154e-01 N  1  3.065e-04  0.000e+00  2.804e-04  1.194e+00  1.215e+09
2025-06-26 20:36:47 fd00:1122:3344:102::e  5      0.000 1000000.000  3.579e-03 N  1  4.136e-03 -2.038e-01  6.046e-02  9.615e-01  5.555e+00
2025-06-26 20:37:46 fd00:1122:3344:104::e  5    -15.134     75.050 -4.559e-03 N  1  1.875e-04 -4.510e-02  2.804e-04  5.089e-03  6.132e+01
2025-06-26 20:38:52 fd00:1122:3344:104::e  5    -31.806     50.166 -1.729e-03 N  1  2.753e-04  2.685e-06  2.804e-04  5.742e-03  5.073e-02
2025-06-26 20:39:57 fd00:1122:3344:104::e  5    -44.055     39.588 -1.591e-03 N  1  4.080e-04 -1.781e-07  2.804e-04  6.841e-03  1.104e-02
2025-06-26 20:41:01 fd00:1122:3344:104::e  5    -47.240     25.914 -3.940e-04 N  1  5.158e-04 -1.444e-07  2.804e-04  6.593e-03  1.131e-02
2025-06-26 20:42:06 fd00:1122:3344:104::e  5    -50.103     13.937 -5.777e-04 N  1  3.687e-04  1.940e-04  2.804e-04  4.379e-03  8.884e-03
2025-06-26 20:43:11 fd00:1122:3344:104::e  5    -50.524      8.995 -7.782e-05 N  1  3.267e-04 -4.671e-08  2.804e-04  3.604e-03  6.279e-03
2025-06-26 20:44:15 fd00:1122:3344:104::e  5    -50.821      6.266 -3.352e-05 N  1  2.846e-04  6.556e-05  2.804e-04  2.964e-03  4.470e-03
2025-06-26 20:45:20 fd00:1122:3344:104::e  5    -50.825      4.439  2.955e-05 N  1  2.498e-04  7.603e-05  2.804e-04  2.469e-03  3.675e-03
2025-06-26 20:46:25 fd00:1122:3344:104::e  5    -50.742      3.682  2.182e-05 N  1  2.524e-04  4.068e-05  2.804e-04  2.345e-03  3.010e-03
2025-06-26 20:47:30 fd00:1122:3344:104::e  5    -49.782      0.949  1.544e-04 N  1  2.729e-05  1.792e-05  6.055e-02  1.194e-03  2.809e-03
2025-06-26 20:47:32 fd00:1122:3344:102::e  6    -49.772      1.205 -4.380e-05 N  1  1.230e-03 -1.183e-04  7.013e-04  6.481e-03  3.161e-02
2025-06-26 20:48:37 fd00:1122:3344:102::e  6    -49.755      1.630 -2.304e-05 N  1  1.130e-03  3.990e-10  7.192e-04  6.349e-03  7.093e-03
2025-06-26 20:49:42 fd00:1122:3344:102::e  6    -49.722      2.375 -2.569e-05 N  1  1.058e-03  2.273e-05  7.228e-04  6.200e-03  6.936e-03
2025-06-26 20:50:47 fd00:1122:3344:102::e  6    -50.090      1.176 -2.202e-04 N  1  9.589e-05  4.693e-05  6.538e-04  5.269e-03  6.855e-03
2025-06-26 20:51:51 fd00:1122:3344:102::e  6    -50.144      0.883 -5.717e-06 N  1  8.158e-05 -1.525e-08  6.229e-04  5.072e-03  6.005e-03
2025-06-26 20:52:56 fd00:1122:3344:102::e  6    -50.141      0.733  6.562e-06 N  1  7.650e-05  5.439e-06  6.681e-04  4.914e-03  5.511e-03
2025-06-26 20:54:00 fd00:1122:3344:102::e  6    -50.147      0.596 -9.699e-10 N  1  6.362e-05 -1.112e-06  6.133e-04  4.794e-03  5.361e-03
2025-06-26 20:55:05 fd00:1122:3344:102::e  6    -50.142      0.521  5.215e-06 N  1  6.053e-05 -1.097e-06  6.133e-04  4.824e-03  5.206e-03
2025-06-26 20:56:09 fd00:1122:3344:102::e  5    -50.138      0.450  2.012e-06 N  1  5.504e-05 -5.878e-06  6.133e-04  4.842e-03  5.234e-03
2025-06-26 20:57:14 fd00:1122:3344:102::e  5    -50.139      0.396 -1.499e-06 N  1  5.171e-05 -7.127e-06  6.133e-04  4.857e-03  5.251e-03
2025-06-26 20:58:20 fd00:1122:3344:102::e  5    -50.158      0.358 -1.671e-05 N  1  4.791e-05 -5.217e-06  6.133e-04  4.866e-03  5.260e-03
2025-06-26 20:59:24 fd00:1122:3344:102::e  5    -50.097      0.282  1.980e-05 N  1  3.561e-05  9.686e-06  6.133e-04  4.845e-03  5.272e-03
2025-06-26 21:00:28 fd00:1122:3344:102::e  5    -50.245      0.513 -1.023e-04 N  1  2.608e-05 -8.191e-06  6.133e-04  4.933e-03  5.245e-03
2025-06-26 21:01:32 fd00:1122:3344:102::e  5    -50.746      0.722 -1.095e-04 N  1  8.763e-06 -4.971e-09  6.039e-02  9.258e-04  5.439e-03
2025-06-26 21:02:37 fd00:1122:3344:102::e  5    -50.927      0.435 -1.395e-05 N  1  6.367e-06 -6.963e-09  6.039e-02  9.459e-04  3.135e-02
2025-06-26 21:03:41 fd00:1122:3344:102::e  5    -50.936      0.294  9.802e-07 N  1  8.112e-06 -5.264e-10  6.039e-02  9.590e-04  3.125e-02
2025-06-26 21:04:46 fd00:1122:3344:102::e  5    -50.976      0.483 -1.974e-05 N  1  1.453e-05 -9.004e-07  6.039e-02  1.059e-03  3.124e-02
2025-06-26 21:05:51 fd00:1122:3344:102::e  5    -51.008      0.402  5.027e-06 N  1  1.262e-05  2.608e-06  6.039e-02  1.024e-03  3.137e-02
===================================================================================================================================
   Date (UTC) Time     IP Address   St   Freq ppm   Skew ppm     Offset L Co  Offset sd Rem. corr. Root delay Root disp. Max. error
===================================================================================================================================
2025-06-26 21:06:55 fd00:1122:3344:102::e  5    -50.943      0.248  2.182e-05 N  1  9.026e-06 -2.111e-06  6.039e-02  9.941e-04  3.131e-02
2025-06-26 21:08:00 fd00:1122:3344:102::e  5    -50.896      0.408  3.579e-05 N  1  2.738e-05  1.219e-09  6.039e-02  1.135e-03  3.129e-02
2025-06-26 21:09:05 fd00:1122:3344:102::e  5    -50.825      0.788  1.047e-04 N  1  3.960e-05 -4.824e-06  6.052e-02  1.399e-03  3.146e-02
2025-06-26 21:10:09 fd00:1122:3344:102::e  5    -50.422      1.492  7.093e-05 N  1  3.210e-05  5.655e-09  6.050e-02  1.453e-03  3.192e-02
2025-06-26 21:11:13 fd00:1122:3344:102::e  5    -50.020      1.130  2.496e-05 N  1  2.188e-05  3.513e-09  6.047e-02  1.451e-03  3.196e-02
2025-06-26 21:12:18 fd00:1122:3344:102::e  5    -50.030      0.914 -1.953e-05 N  1  2.276e-05 -5.874e-06  6.050e-02  1.500e-03  3.185e-02
2025-06-26 21:13:22 fd00:1122:3344:102::e  5    -50.364      0.465 -5.050e-05 N  1  6.601e-06  8.276e-06  6.050e-02  1.375e-03  3.189e-02
2025-06-26 21:14:27 fd00:1122:3344:102::e  5    -50.391      0.436 -3.403e-06 N  1  7.870e-06 -2.370e-09  6.052e-02  1.499e-03  3.178e-02
2025-06-26 21:15:31 fd00:1122:3344:102::e  5    -50.429      0.438 -1.069e-05 N  1  1.302e-05  2.418e-06  6.052e-02  1.532e-03  3.186e-02
2025-06-26 21:16:36 fd00:1122:3344:102::e  5    -50.556      0.397 -3.132e-05 N  1  8.595e-06  4.352e-06  6.043e-02  1.610e-03  3.190e-02
2025-06-26 21:17:40 fd00:1122:3344:102::e  5    -50.578      0.258  8.307e-07 N  1  5.059e-06 -1.506e-09  6.051e-02  1.622e-03  3.195e-02
2025-06-26 21:18:45 fd00:1122:3344:102::e  5    -50.570      0.393  6.304e-06 N  1  1.482e-05 -7.388e-07  6.049e-02  9.613e-04  3.196e-02
2025-06-26 21:19:50 fd00:1122:3344:102::e  5    -50.499      0.728  6.845e-05 N  1  2.548e-05 -4.815e-06  6.042e-02  1.034e-03  3.130e-02
2025-06-26 21:20:55 fd00:1122:3344:102::e  5    -50.195      0.992  3.830e-05 N  1  1.746e-05  3.847e-09  6.042e-02  1.078e-03  3.145e-02
2025-06-26 21:21:59 fd00:1122:3344:102::e  5    -50.109      0.809 -2.297e-06 N  1  1.816e-05  2.055e-09  6.042e-02  1.120e-03  3.146e-02
2025-06-26 21:23:03 fd00:1122:3344:102::e  5    -50.183      0.726 -3.086e-05 N  1  1.597e-05  2.105e-06  6.042e-02  1.153e-03  3.145e-02
2025-06-26 21:24:08 fd00:1122:3344:102::e  5    -50.406      0.219 -2.278e-05 N  1  2.573e-06 -1.558e-09  6.049e-02  1.006e-03  3.151e-02
2025-06-26 21:25:12 fd00:1122:3344:102::e  5    -50.418      0.126 -2.176e-06 N  1  2.586e-06 -5.867e-10  6.049e-02  9.930e-04  3.135e-02
2025-06-26 21:26:17 fd00:1122:3344:102::e  5    -50.423      0.122 -1.487e-06 N  1  4.393e-06  9.412e-07  6.049e-02  1.004e-03  3.131e-02
2025-06-26 21:27:21 fd00:1122:3344:102::e  5    -50.434      0.109 -4.184e-06 N  1  5.146e-06  1.547e-06  6.049e-02  1.003e-03  3.132e-02
2025-06-26 21:28:26 fd00:1122:3344:102::e  5    -50.439      0.083 -1.625e-06 N  1  4.998e-06  1.461e-06  6.049e-02  9.962e-04  3.133e-02
2025-06-26 21:29:30 fd00:1122:3344:102::e  5    -50.451      0.143 -8.231e-06 N  1  6.536e-06  1.818e-06  6.047e-02  1.302e-03  3.131e-02
2025-06-26 21:30:35 fd00:1122:3344:102::e  5    -50.458      0.204  9.134e-08 N  1  6.730e-06 -4.496e-12  6.043e-02  1.418e-03  3.162e-02
2025-06-26 21:31:39 fd00:1122:3344:102::e  5    -50.472      0.329 -1.274e-05 N  1  1.192e-05 -9.044e-08  6.047e-02  1.625e-03  3.171e-02
2025-06-26 21:32:44 fd00:1122:3344:102::e  5    -50.506      0.260 -1.206e-07 N  1  5.407e-06  3.643e-06  6.047e-02  1.546e-03  3.196e-02
2025-06-26 21:33:48 fd00:1122:3344:102::e  5    -50.532      0.154 -6.926e-06 N  1  4.042e-06  2.018e-06  6.047e-02  1.522e-03  3.187e-02
2025-06-26 21:34:53 fd00:1122:3344:102::e  5    -50.391      0.076  2.868e-05 N  2  5.036e-05 -2.573e-10  6.049e-02  1.657e-03  3.184e-02
2025-06-26 21:35:58 fd00:1122:3344:102::e  5    -50.384      0.053 -2.665e-05 N  2  8.845e-05 -1.761e-05  6.046e-02  9.283e-04  3.200e-02
2025-06-26 21:37:02 fd00:1122:3344:102::e  5    -50.390      0.051 -2.499e-05 N  2  1.246e-04  8.428e-06  6.046e-02  9.619e-04  3.124e-02
2025-06-26 21:38:07 fd00:1122:3344:102::e  5    -50.409      0.059 -1.886e-05 N  2  1.528e-04  2.734e-05  6.046e-02  9.751e-04  3.130e-02
2025-06-26 21:39:11 fd00:1122:3344:102::e  5    -50.447      0.204 -1.108e-04 N  2  4.392e-05  3.705e-05  6.046e-02  1.008e-03  3.132e-02
2025-06-26 21:40:16 fd00:1122:3344:102::e  5    -50.660      0.335 -2.119e-05 N  2  2.206e-05 -8.321e-09  6.046e-02  1.067e-03  3.149e-02
===================================================================================================================================
   Date (UTC) Time     IP Address   St   Freq ppm   Skew ppm     Offset L Co  Offset sd Rem. corr. Root delay Root disp. Max. error
===================================================================================================================================
2025-06-26 21:41:20 fd00:1122:3344:102::e  5    -50.739      0.189 -4.541e-06 N  2  1.691e-05  7.683e-06  6.046e-02  1.069e-03  3.141e-02
2025-06-26 21:42:25 fd00:1122:3344:102::e  5    -50.730      0.142  1.081e-05 N  2  1.357e-05  6.286e-06  6.046e-02  1.034e-03  3.139e-02
2025-06-26 21:43:29 fd00:1122:3344:102::e  5    -50.697      0.157  2.279e-05 N  2  1.531e-05 -3.521e-06  6.046e-02  1.039e-03  3.134e-02
2025-06-26 21:44:34 fd00:1122:3344:102::e  5    -50.680      0.134  6.235e-06 N  2  1.647e-05  1.389e-09  6.046e-02  1.036e-03  3.137e-02
2025-06-26 21:45:39 fd00:1122:3344:102::e  5    -50.664      0.118  9.074e-06 N  2  1.584e-05 -4.678e-06  6.046e-02  1.036e-03  3.135e-02
2025-06-26 21:46:43 fd00:1122:3344:102::e  5    -50.622      0.140  2.851e-05 N  2  1.875e-05 -5.800e-06  6.046e-02  1.036e-03  3.135e-02
2025-06-26 21:47:48 fd00:1122:3344:102::e  5    -50.512      0.192  3.563e-05 N  2  3.404e-05  1.817e-09  6.048e-02  1.396e-03  3.138e-02
2025-06-26 21:48:52 fd00:1122:3344:102::e  5    -50.436      0.122  5.198e-06 N  2  3.504e-05 -1.101e-05  6.048e-02  1.354e-03  3.175e-02
2025-06-26 21:49:57 fd00:1122:3344:102::e  5    -50.433      0.090 -4.432e-06 N  2  4.325e-05 -1.121e-05  6.048e-02  1.352e-03  3.168e-02
2025-06-26 21:51:02 fd00:1122:3344:102::e  5    -50.418      0.095  2.103e-05 N  2  9.149e-06 -6.061e-06  6.048e-02  1.367e-03  3.167e-02

Logs of other NTP instances on CRDB sleds below:

BRM44220005 / Sled 9

root@oxz_ntp_76b79b96:~# head -n 30 /var/log/chrony/tracking.log
===================================================================================================================================
   Date (UTC) Time     IP Address   St   Freq ppm   Skew ppm     Offset L Co  Offset sd Rem. corr. Root delay Root disp. Max. error
===================================================================================================================================
1986-12-28 00:01:59 0.0.0.0          0     -0.000 1000000.000  0.000e+00 ?  0  0.000e+00  0.000e+00  1.000e+00  1.000e+00  1.500e+00
1986-12-28 00:07:07 fd00:1122:3344:104::e  5      0.000 1000000.000 -1.215e+09 N  1  6.155e-02  0.000e+00  6.064e-02  1.216e+09  1.500e+00
2025-06-26 20:39:52 fd00:1122:3344:104::e  5      0.000 1000000.000 -4.814e-01 N  1  1.009e-04  0.000e+00  6.065e-02  1.241e-01  1.215e+09
2025-06-26 20:40:57 fd00:1122:3344:104::e  5    -48.173     37.301 -3.116e-03 N  1  1.194e-04  3.061e-01  6.052e-02  1.928e-03  6.596e+01
2025-06-26 20:42:02 fd00:1122:3344:104::e  5    -43.285     19.118  5.330e-04 N  1  1.468e-04  1.339e-01  6.055e-02  1.997e-03  3.439e-01
2025-06-26 20:43:07 fd00:1122:3344:102::e  6    -33.819     37.316  2.720e-03 N  2  5.919e-04 -2.248e-04  6.977e-04  6.582e-03  1.669e-01
2025-06-26 20:44:10 fd00:1122:3344:102::e  6    -32.702     55.838  5.411e-04 N  1  6.379e-04  4.013e-09  7.662e-04  6.054e-03  1.302e-02
2025-06-26 20:45:14 fd00:1122:3344:102::e  6    -29.637     59.278 -3.605e-04 N  1  5.282e-04 -2.292e-04  7.121e-04  5.566e-03  1.141e-02
2025-06-26 20:46:19 fd00:1122:3344:102::e  6    -31.775     26.775 -5.675e-04 N  1  3.848e-04  1.095e-04  6.895e-04  5.154e-03  1.002e-02
2025-06-26 20:47:24 fd00:1122:3344:102::e  6    -34.230     14.457 -3.888e-04 N  1  2.852e-04 -2.904e-08  7.146e-04  5.031e-03  7.995e-03
2025-06-26 20:48:29 fd00:1122:3344:102::e  6    -34.223     10.569 -2.238e-04 N  1  3.223e-04  3.795e-05  6.559e-04  5.025e-03  6.790e-03
2025-06-26 20:49:33 fd00:1122:3344:102::e  6    -37.314      2.296 -4.337e-04 N  1  3.334e-05  1.206e-04  6.734e-04  4.675e-03  6.361e-03
2025-06-26 20:50:38 fd00:1122:3344:102::e  6    -37.329      1.020  4.605e-06 N  1  2.232e-05 -6.738e-08  6.640e-04  4.632e-03  5.779e-03
2025-06-26 20:51:43 fd00:1122:3344:102::e  6    -37.295      0.676  4.812e-06 N  1  2.190e-05 -4.011e-06  6.136e-04  4.479e-03  5.101e-03
2025-06-26 20:52:48 fd00:1122:3344:102::e  6    -37.186      0.790  2.661e-05 N  1  1.308e-05 -6.451e-06  6.199e-04  4.379e-03  4.904e-03
2025-06-26 20:53:53 fd00:1122:3344:102::e  6    -37.232      0.627 -1.651e-06 N  1  1.471e-05  1.526e-09  6.313e-04  4.304e-03  4.841e-03
2025-06-26 20:54:57 fd00:1122:3344:102::e  6    -37.200      0.509  1.186e-05 N  1  1.755e-05  1.531e-06  6.313e-04  4.337e-03  4.726e-03
2025-06-26 20:56:01 fd00:1122:3344:102::e  5    -37.181      0.392  5.885e-06 N  1  1.832e-05 -6.272e-06  6.313e-04  4.353e-03  4.760e-03
2025-06-26 20:57:05 fd00:1122:3344:102::e  5    -37.172      0.286  3.033e-06 N  1  1.681e-05 -6.794e-06  6.313e-04  4.356e-03  4.770e-03
2025-06-26 20:58:10 fd00:1122:3344:102::e  5    -37.191      0.251 -1.010e-05 N  1  1.801e-05 -5.994e-06  6.313e-04  4.367e-03  4.764e-03
2025-06-26 20:59:15 fd00:1122:3344:102::e  5    -37.251      0.253 -3.313e-05 N  1  1.928e-05  3.475e-06  6.313e-04  4.377e-03  4.768e-03
2025-06-26 21:00:19 fd00:1122:3344:102::e  5    -37.398      0.609 -8.543e-05 N  1  1.452e-05 -1.251e-09  6.045e-02  1.887e-03  4.812e-03
2025-06-26 21:01:24 fd00:1122:3344:102::e  5    -37.740      0.812 -7.333e-05 N  1  1.572e-05 -2.920e-09  6.046e-02  9.480e-04  3.233e-02
2025-06-26 21:02:29 fd00:1122:3344:102::e  5    -38.030      1.047 -5.178e-05 N  1  1.721e-05 -2.586e-09  6.046e-02  1.006e-03  3.138e-02
2025-06-26 21:03:33 fd00:1122:3344:102::e  5    -38.094      0.842  2.262e-06 N  1  1.423e-05 -1.708e-09  6.046e-02  1.045e-03  3.143e-02
2025-06-26 21:04:38 fd00:1122:3344:102::e  5    -38.181      0.602 -1.914e-05 N  1  1.317e-05 -2.021e-06  6.046e-02  1.049e-03  3.140e-02
2025-06-26 21:05:43 fd00:1122:3344:102::e  5    -38.185      0.425 -5.509e-07 N  1  1.508e-05  2.377e-06  6.046e-02  1.049e-03  3.140e-02

BRM42220009 / Sled 10

root@oxz_ntp_7529be1c:~# head -n 30 /var/log/chrony/tracking.log
===================================================================================================================================
   Date (UTC) Time     IP Address   St   Freq ppm   Skew ppm     Offset L Co  Offset sd Rem. corr. Root delay Root disp. Max. error
===================================================================================================================================
1986-12-28 00:01:54 0.0.0.0          0     -0.000 1000000.000  0.000e+00 ?  0  0.000e+00  0.000e+00  1.000e+00  1.000e+00  1.500e+00
1986-12-28 00:06:52 fd00:1122:3344:104::e  5      0.000 1000000.000 -1.215e+09 N  1  3.125e-04  0.000e+00  6.057e-02  1.215e+09  1.500e+00
2025-06-26 20:40:20 fd00:1122:3344:102::e  6      0.000 1000000.000  3.386e-01 N  1  4.816e-05  0.000e+00  7.187e-04  7.975e-01  1.215e+09
2025-06-26 20:41:24 fd00:1122:3344:102::e  6    -11.130     80.935 -7.189e-04 N  1  9.765e-05 -1.639e-01  7.020e-04  4.111e-03  6.550e+01
2025-06-26 20:42:29 fd00:1122:3344:102::e  6    -30.175    129.899 -3.793e-03 N  1  7.152e-04  2.521e-06  7.136e-04  4.161e-03  1.730e-01
2025-06-26 20:43:33 fd00:1122:3344:102::e  6    -52.546     58.002 -1.781e-03 N  1  5.252e-04 -8.645e-07  7.021e-04  3.566e-03  1.796e-02
2025-06-26 20:44:38 fd00:1122:3344:102::e  6    -52.880     24.710  2.695e-05 N  1  3.979e-04 -2.037e-07  6.622e-04  3.087e-03  9.577e-03
2025-06-26 20:45:42 fd00:1122:3344:102::e  6    -52.754     14.621  3.615e-05 N  1  3.300e-04 -2.592e-05  6.885e-04  2.807e-03  5.102e-03
2025-06-26 20:46:47 fd00:1122:3344:102::e  6    -52.678      9.510  1.801e-05 N  1  2.874e-04 -5.422e-05  7.064e-04  2.737e-03  4.229e-03
2025-06-26 20:47:52 fd00:1122:3344:102::e  6    -52.588      6.459  2.291e-05 N  1  2.536e-04 -6.012e-05  7.017e-04  2.686e-03  3.846e-03
2025-06-26 20:48:57 fd00:1122:3344:102::e  6    -52.503      4.164  2.230e-05 N  1  1.959e-04 -6.485e-05  6.792e-04  2.607e-03  3.607e-03
2025-06-26 20:50:01 fd00:1122:3344:102::e  6    -52.148      0.791  4.731e-05 N  1  2.918e-05 -6.168e-05  6.295e-04  2.425e-03  3.366e-03
2025-06-26 20:51:06 fd00:1122:3344:102::e  6    -52.160      0.510 -5.866e-06 N  1  2.234e-05  5.468e-09  6.075e-04  2.235e-03  2.965e-03
2025-06-26 20:52:11 fd00:1122:3344:102::e  6    -52.144      0.397  7.241e-06 N  1  2.095e-05  4.835e-06  6.399e-04  2.157e-03  2.643e-03
2025-06-26 20:53:16 fd00:1122:3344:102::e  6    -52.140      0.300  1.223e-06 N  1  1.827e-05 -2.222e-06  6.197e-04  2.094e-03  2.571e-03
2025-06-26 20:54:21 fd00:1122:3344:102::e  6    -52.115      0.260  1.215e-05 N  1  1.731e-05 -3.015e-06  6.197e-04  2.109e-03  2.491e-03
2025-06-26 20:55:25 fd00:1122:3344:102::e  6    -52.141      0.269  1.104e-05 N  1  2.172e-05 -6.331e-06  6.197e-04  2.130e-03  2.516e-03
2025-06-26 20:56:30 fd00:1122:3344:102::e  5    -52.145      0.214  3.410e-07 N  1  1.871e-05 -8.095e-06  6.197e-04  2.134e-03  2.539e-03
2025-06-26 20:57:35 fd00:1122:3344:102::e  5    -52.153      0.182 -6.061e-06 N  1  1.726e-05 -5.878e-06  6.197e-04  2.137e-03  2.531e-03
2025-06-26 20:58:40 fd00:1122:3344:102::e  5    -52.171      0.162 -1.278e-05 N  1  1.565e-05  1.831e-07  6.197e-04  2.139e-03  2.524e-03
2025-06-26 20:59:45 fd00:1122:3344:102::e  5    -52.226      0.184 -2.936e-05 N  1  1.510e-05  5.816e-06  6.197e-04  2.150e-03  2.538e-03
2025-06-26 21:00:50 fd00:1122:3344:102::e  5    -52.265      0.303 -3.230e-05 N  1  2.359e-05 -1.878e-09  6.197e-04  2.261e-03  2.572e-03
2025-06-26 21:01:55 fd00:1122:3344:102::e  5    -52.329      0.362 -2.685e-05 N  1  2.529e-05  2.941e-06  6.197e-04  2.272e-03  2.690e-03
2025-06-26 21:02:59 fd00:1122:3344:102::e  5    -52.595      0.761 -9.875e-05 N  1  1.418e-05  6.305e-06  6.044e-02  9.671e-04  2.702e-03
2025-06-26 21:04:04 fd00:1122:3344:102::e  5    -52.876      0.721 -3.461e-05 N  1  1.536e-05 -6.332e-09  6.044e-02  9.964e-04  3.143e-02
2025-06-26 21:06:15 fd00:1122:3344:102::e  5    -52.816      0.574  3.765e-05 N  1  1.877e-05 -1.201e-09  6.044e-02  1.058e-03  3.148e-02
2025-06-26 21:07:19 fd00:1122:3344:102::e  5    -52.660      0.949  8.333e-05 N  1  3.089e-05 -1.289e-05  6.045e-02  1.273e-03  3.142e-02

BRM42220051 / Sled 14

root@oxz_ntp_3ccea933:~# head -n 30 /var/log/chrony/tracking.log
===================================================================================================================================
   Date (UTC) Time     IP Address   St   Freq ppm   Skew ppm     Offset L Co  Offset sd Rem. corr. Root delay Root disp. Max. error
===================================================================================================================================
1986-12-28 01:16:59 0.0.0.0          0     -0.000 1000000.000  0.000e+00 ?  0  0.000e+00  0.000e+00  1.000e+00  1.000e+00  1.500e+00
1986-12-28 01:26:39 fd00:1122:3344:102::e  6    -37.804      8.773 -1.215e+09 N  1  1.491e-06  0.000e+00  7.670e-04  1.187e+04  1.500e+00
2025-06-26 20:42:42 fd00:1122:3344:104::e  5    -37.804      8.773 -3.834e-01 N  1  1.260e-04  1.301e-11  6.089e-02  2.623e-03  1.215e+09
2025-06-26 20:43:45 fd00:1122:3344:102::e  6    -37.804      8.773 -1.369e-03 N  1  5.067e-05  2.135e-01  7.187e-04  4.846e-03  6.410e-01
2025-06-26 20:44:50 fd00:1122:3344:102::e  6    -37.172     14.242  1.627e-04 N  1  7.302e-05  3.877e-02  6.470e-04  3.541e-03  3.869e-01
2025-06-26 20:45:55 fd00:1122:3344:102::e  6    -36.250      9.797  3.931e-05 N  1  6.640e-05 -3.334e-05  6.598e-04  3.320e-03  4.358e-02
2025-06-26 20:46:59 fd00:1122:3344:102::e  6    -37.018      5.392 -1.147e-04 N  1  6.954e-05 -1.957e-05  6.818e-04  3.264e-03  4.427e-03
2025-06-26 20:48:04 fd00:1122:3344:102::e  6    -37.646      3.001 -1.077e-04 N  1  5.612e-05  8.342e-06  6.440e-04  3.200e-03  4.116e-03
2025-06-26 20:49:08 fd00:1122:3344:102::e  6    -37.864      1.739 -3.243e-05 N  1  4.581e-05 -4.182e-09  6.974e-04  3.114e-03  3.899e-03
2025-06-26 20:50:13 fd00:1122:3344:102::e  6    -37.949      1.173 -1.870e-05 N  1  4.262e-05  1.695e-05  6.866e-04  3.140e-03  3.674e-03
2025-06-26 20:51:18 fd00:1122:3344:102::e  6    -37.940      0.820 -2.473e-06 N  1  3.614e-05  1.590e-05  6.126e-04  2.967e-03  3.661e-03
2025-06-26 20:52:22 fd00:1122:3344:102::e  6    -37.918      0.640  7.894e-06 N  1  3.366e-05  1.219e-05  6.479e-04  2.826e-03  3.409e-03
2025-06-26 20:53:27 fd00:1122:3344:102::e  6    -37.876      0.528 -7.355e-06 N  1  3.163e-05  3.928e-06  6.401e-04  2.733e-03  3.261e-03
2025-06-26 20:54:32 fd00:1122:3344:102::e  6    -37.903      0.314  9.680e-07 N  1  1.730e-05  8.580e-06  6.401e-04  2.738e-03  3.164e-03
2025-06-26 20:55:36 fd00:1122:3344:102::e  6    -37.822      0.259  1.926e-05 N  1  1.108e-05  5.403e-06  6.401e-04  2.740e-03  3.150e-03
2025-06-26 20:56:41 fd00:1122:3344:102::e  5    -37.802      0.153  4.421e-06 N  1  8.106e-06 -2.158e-06  6.401e-04  2.736e-03  3.157e-03
2025-06-26 20:57:47 fd00:1122:3344:102::e  5    -37.820      0.143 -9.693e-06 N  1  9.134e-06 -3.020e-06  6.401e-04  2.744e-03  3.138e-03
2025-06-26 20:58:51 fd00:1122:3344:102::e  5    -37.840      0.124 -8.665e-06 N  1  8.910e-06  3.451e-06  6.401e-04  2.747e-03  3.145e-03
2025-06-26 20:59:56 fd00:1122:3344:102::e  5    -37.859      0.127 -1.080e-05 N  1  1.063e-05  1.157e-06  6.401e-04  2.757e-03  3.151e-03
2025-06-26 21:01:00 fd00:1122:3344:102::e  5    -37.889      0.141 -1.771e-05 N  1  1.293e-05  2.972e-06  6.401e-04  2.770e-03  3.162e-03
2025-06-26 21:02:05 fd00:1122:3344:102::e  5    -38.058      0.333 -7.026e-05 N  1  4.643e-06 -7.424e-10  6.047e-02  8.887e-04  3.185e-03
2025-06-26 21:03:10 fd00:1122:3344:102::e  5    -38.206      0.519 -1.479e-05 N  1  9.448e-06 -2.227e-09  6.047e-02  9.380e-04  3.129e-02
2025-06-26 21:04:14 fd00:1122:3344:102::e  5    -38.406      0.483 -3.318e-05 N  1  8.296e-06 -5.374e-10  6.047e-02  9.522e-04  3.129e-02
2025-06-26 21:05:19 fd00:1122:3344:102::e  5    -38.483      0.335 -9.099e-06 N  1  6.749e-06 -1.020e-09  6.047e-02  9.526e-04  3.132e-02
2025-06-26 21:06:24 fd00:1122:3344:102::e  5    -38.421      0.585  5.062e-05 N  1  2.082e-05  8.952e-07  6.041e-02  1.167e-03  3.128e-02
2025-06-26 21:07:29 fd00:1122:3344:102::e  5    -38.304      1.063  8.424e-05 N  1  4.448e-05  1.878e-09  6.048e-02  1.313e-03  3.153e-02
2025-06-26 21:08:33 fd00:1122:3344:102::e  5    -37.738      1.615  1.048e-04 N  1  2.453e-05  3.218e-09  6.048e-02  1.339e-03  3.179e-02

BRM42220017 / Sled 17

root@oxz_ntp_83257100:~# head -n 30 /var/log/chrony/tracking.log
===================================================================================================================================
   Date (UTC) Time     IP Address   St   Freq ppm   Skew ppm     Offset L Co  Offset sd Rem. corr. Root delay Root disp. Max. error
===================================================================================================================================
1986-12-28 00:01:56 0.0.0.0          0     -0.000 1000000.000  0.000e+00 ?  0  0.000e+00  0.000e+00  1.000e+00  1.000e+00  1.500e+00
1986-12-28 00:11:21 fd00:1122:3344:104::e  5    -42.620     43.394 -1.215e+09 N  1  5.915e-06  0.000e+00  6.045e-02  5.393e+04  1.500e+00
2025-06-26 20:42:36 fd00:1122:3344:102::e  6    -42.620     43.394 -4.383e-01 N  1  4.697e-05  1.906e-11  7.100e-04  5.507e-03  1.215e+09
2025-06-26 20:43:38 fd00:1122:3344:104::e  5    -42.620     43.394 -1.453e-03 N  1  6.619e-05  2.709e-01  6.051e-02  3.870e-03  6.975e-01
2025-06-26 20:43:41 fd00:1122:3344:102::e  6    -54.004     75.739 -3.007e-04 N  1  1.239e-04  2.651e-01  7.497e-04  3.698e-03  3.146e-01
2025-06-26 20:44:46 fd00:1122:3344:102::e  6    -53.148     62.521  7.033e-04 N  1  3.002e-04  9.002e-02  7.257e-04  3.535e-03  2.754e-01
2025-06-26 20:45:50 fd00:1122:3344:102::e  6    -37.892     66.584  2.852e-03 N  1  6.201e-04 -1.821e-04  6.706e-04  3.595e-03  9.730e-02
2025-06-26 20:46:55 fd00:1122:3344:102::e  6    -34.011     33.668  2.532e-04 N  1  5.430e-04  4.402e-09  7.046e-04  3.451e-03  1.162e-02
2025-06-26 20:47:59 fd00:1122:3344:102::e  6    -35.219     20.163 -3.016e-04 N  1  4.788e-04 -1.747e-04  6.978e-04  3.353e-03  6.316e-03
2025-06-26 20:49:04 fd00:1122:3344:102::e  6    -36.744     13.814 -3.693e-04 N  1  4.251e-04  1.045e-04  6.920e-04  3.260e-03  5.206e-03
2025-06-26 20:50:09 fd00:1122:3344:102::e  6    -38.358      9.559 -4.097e-04 N  1  3.414e-04  1.224e-04  6.639e-04  3.175e-03  5.049e-03
2025-06-26 20:51:13 fd00:1122:3344:102::e  6    -39.898      6.324 -3.354e-04 N  1  2.328e-04 -2.509e-08  6.612e-04  2.874e-03  4.731e-03
2025-06-26 20:52:18 fd00:1122:3344:102::e  6    -42.010      0.486 -2.974e-04 N  1  1.198e-05  9.477e-06  6.331e-04  2.561e-03  4.028e-03
2025-06-26 20:53:23 fd00:1122:3344:102::e  6    -41.953      0.198 -4.169e-06 N  1  5.972e-06 -3.570e-08  6.341e-04  2.464e-03  3.281e-03
2025-06-26 20:54:28 fd00:1122:3344:102::e  6    -41.931      0.165  7.236e-06 N  1  6.060e-06  2.225e-06  6.341e-04  2.473e-03  2.863e-03
2025-06-26 20:55:33 fd00:1122:3344:102::e  6    -41.946      0.188 -7.861e-06 N  1  1.017e-05 -2.230e-06  6.341e-04  2.491e-03  2.871e-03
2025-06-26 20:56:37 fd00:1122:3344:102::e  5    -41.975      0.200 -1.235e-05 N  1  1.185e-05  3.579e-06  6.341e-04  2.505e-03  2.891e-03
2025-06-26 20:57:42 fd00:1122:3344:102::e  5    -42.019      0.364 -3.591e-05 N  1  1.233e-05  1.572e-06  6.341e-04  2.605e-03  2.916e-03
2025-06-26 20:58:47 fd00:1122:3344:102::e  5    -42.131      0.446 -1.602e-05 N  1  1.292e-05 -1.598e-09  6.341e-04  2.595e-03  3.054e-03
2025-06-26 20:59:52 fd00:1122:3344:102::e  5    -42.310      0.536 -2.766e-05 N  1  9.170e-06  2.732e-06  6.045e-02  1.803e-03  3.025e-03
2025-06-26 21:00:56 fd00:1122:3344:102::e  5    -42.409      0.304 -1.724e-05 N  1  8.050e-06 -1.291e-09  6.048e-02  8.484e-04  3.216e-02
2025-06-26 21:02:01 fd00:1122:3344:102::e  5    -42.509      0.369 -2.587e-05 N  1  6.761e-06 -5.210e-10  6.048e-02  8.672e-04  3.119e-02
2025-06-26 21:03:05 fd00:1122:3344:102::e  5    -42.537      0.233 -1.208e-06 N  1  7.083e-06 -8.315e-10  6.048e-02  8.740e-04  3.122e-02
2025-06-26 21:04:10 fd00:1122:3344:102::e  5    -42.557      0.180 -6.778e-06 N  1  7.512e-06  1.070e-06  6.048e-02  8.794e-04  3.119e-02
2025-06-26 21:05:15 fd00:1122:3344:102::e  5    -42.559      0.140  1.377e-07 N  1  8.025e-06  2.350e-06  6.048e-02  8.823e-04  3.120e-02
2025-06-26 21:06:19 fd00:1122:3344:102::e  5    -42.529      0.126  1.499e-05 N  1  7.739e-06  1.808e-06  6.048e-02  8.820e-04  3.120e-02
2025-06-26 21:07:23 fd00:1122:3344:102::e  5    -42.521      0.208  6.942e-05 N  1  3.268e-05  6.752e-10  6.046e-02  1.321e-03  3.121e-02

[iliana]

Future reports should include all of:

• /var/log/chrony/tracking.log
• /var/log/chrony/measurements.log
• /var/log/chrony/statistics.log
• /var/svc/log/oxide-ntp:default.log (also check for rotated files)

Consider storing full copies of files on /staff.

[askfongjojo]

All the logs listed above, plus oxide-cockroachdb:default logs, right after the mupdate on June 26 have been downloaded to /staff/core/omicron-7668/20250626. Will add the pertinent snippets in a bit to show the timeline of events.

[askfongjojo]

From boundary NTP logs:

• Sled 12 (fd00:1122:3344:104::e)

1986-12-28T00:02:35Z System clock wrong by 1214858039.487527 seconds
2025-06-26T20:36:35Z System clock was stepped by 1214858039.487527 seconds
2025-06-26T20:36:35Z Out of order sample detected, discarding history for 172.20.0.5
2025-06-26T20:36:35Z Can't synchronise: no selectable sources
2025-06-26T20:36:37Z Selected source 172.20.0.5 (ntp.eng.oxide.computer)
2025-06-26T20:36:38Z System clock was stepped by 0.284646 seconds
2025-06-26T20:36:39Z System clock was stepped by -0.440386 seconds

• Sled 21 (fd00:1122:3344:102::e)

1986-12-28T00:06:20Z System clock wrong by 1214857822.013781 seconds
2025-06-26T20:36:43Z System clock was stepped by 1214857822.013781 seconds
2025-06-26T20:36:48Z Selected source fd00:1122:3344:104::e (boundary-ntp.control-plane.oxide.internal)
2025-06-26T20:37:12Z Selected source 172.20.0.5 (ntp.eng.oxide.computer)
2025-06-26T20:37:53Z Selected source fd00:1122:3344:104::e (boundary-ntp.control-plane.oxide.internal)
2025-06-26T20:55:42Z Selected source 172.20.0.5 (ntp.eng.oxide.computer)

The ntp zone on sled 12 appears to be the authoritative one since it was selected as the source (twice) by the sled 21 ntp zone, if I've read #7675 correctly.

[askfongjojo]

Here are the NTP logs from the 5 sleds hosting the cockroachdb zones, interleaved with some key event timestamps from the crdb log:

Sled 9

• ntp

1986-12-28T00:07:07Z System clock wrong by 1214857901.313777 seconds
2025-06-26T20:38:48Z System clock was stepped by 1214857901.313777 seconds
2025-06-26T20:43:07Z Selected source fd00:1122:3344:102::e (boundary-ntp.control-plane.oxide.internal)

cockroachdb

2025-06-26T20:39:10.335364654Z (start)
2025-06-26T20:43:19.285047013Z (up and running)

Sled 10

• ntp

1986-12-28T00:06:52Z System clock wrong by 1214858007.571172 seconds
2025-06-26T20:40:20Z System clock was stepped by 1214858007.571172 seconds
2025-06-26T20:40:20Z Selected source fd00:1122:3344:102::e (boundary-ntp.control-plane.oxide.internal)

• cockroachdb

2025-06-26T20:42:35.212779102Z (start)
2025-06-26T20:43:19.282687584Z (up and running)

Sled 14

• ntp

1986-12-28T01:26:39Z System clock wrong by 1214853361.962528 seconds
2025-06-26T20:42:41Z System clock was stepped by 1214853361.962528 seconds
2025-06-26T20:42:42Z Selected source fd00:1122:3344:104::e (boundary-ntp.control-plane.oxide.internal)
2025-06-26T20:43:45Z Selected source fd00:1122:3344:102::e (boundary-ntp.control-plane.oxide.internal)

• cockroachdb

2025-06-26T20:45:15.520282881Z (start)
2025-06-26T20:45:31.588544221Z (up and running)

Sled 16

• ntp

1986-12-28T00:02:36Z System clock wrong by 1214858044.119830 seconds
2025-06-26T20:36:41Z System clock was stepped by 1214858044.119830 seconds
2025-06-26T20:36:41Z Can't synchronise: no selectable sources
2025-06-26T20:36:43Z Selected source fd00:1122:3344:104::e (boundary-ntp.control-plane.oxide.internal)
2025-06-26T20:36:45Z Can't synchronise: no selectable sources
2025-06-26T20:36:47Z Selected source fd00:1122:3344:102::e (boundary-ntp.control-plane.oxide.internal)
2025-06-26T20:37:47Z Selected source fd00:1122:3344:104::e (boundary-ntp.control-plane.oxide.internal)
2025-06-26T20:47:32Z Selected source fd00:1122:3344:102::e (boundary-ntp.control-plane.oxide.internal)

• cockroachdb

2025-06-26T20:38:14.261730234Z (start)
2025-06-26T20:39:25.421920173Z (2nd start attempt)
2025-06-26T20:39:27.328761568Z (3rd start attempt)
2025-06-26T20:39:29.226612855 (4th start attempt)
Jun 26 20:39:31 SMF service went into maintenance

Sled 17

• ntp

1986-12-28T00:11:21Z System clock wrong by 1214857872.852073 seconds
2025-06-26T20:42:34Z System clock was stepped by 1214857872.852073 seconds
2025-06-26T20:42:36Z Selected source fd00:1122:3344:102::e (boundary-ntp.control-plane.oxide.internal)
2025-06-26T20:43:38Z Selected source fd00:1122:3344:104::e (boundary-ntp.control-plane.oxide.internal)
2025-06-26T20:43:41Z Selected source fd00:1122:3344:102::e (boundary-ntp.control-plane.oxide.internal)

• cockroachdb

2025-06-26T20:42:54.802228893Z (start)
2025-06-26T20:43:18.958874388Z (up and running)

Taking the timestamps at their face values, sled 16's cockroachdb attempted to start a few minutes ahead of the other peers. It was the only sled using fd00:1122:3344:104::e (sled 12 boundary NTP) as its NTP source when cockroachdb service started up; all other peers were using fd00:1122:3344:102::e.
Sled 16 eventually pointed to fd00:1122:3344:102::e as well after almost 10 minutes.

[askfongjojo]

Sled 21 NTP: chrony tracking.log

===================================================================================================================================
   Date (UTC) Time     IP Address   St   Freq ppm   Skew ppm     Offset L Co  Offset sd Rem. corr. Root delay Root disp. Max. error
===================================================================================================================================
1986-12-28 00:06:19 0.0.0.0          0     -0.000 1000000.000  0.000e+00 ?  0  0.000e+00  0.000e+00  1.000e+00  1.000e+00  1.500e+00
1986-12-28 00:06:20 0.0.0.0          0     -0.000 1000000.000  0.000e+00 ?  0  0.000e+00  0.000e+00  1.000e+00  1.000e+00  1.500e+00
1986-12-28 00:06:20 0.0.0.0          0     -0.000 1000000.000  0.000e+00 ?  0  0.000e+00  0.000e+00  1.000e+00  1.000e+00  1.500e+00
1986-12-28 00:06:20 0.0.0.0          0     -0.000 1000000.000  0.000e+00 ?  0  0.000e+00  0.000e+00  1.000e+00  1.000e+00  1.500e+00
1986-12-28 00:06:20 0.0.0.0          0     -0.000 1000000.000  0.000e+00 ?  0  0.000e+00  0.000e+00  1.000e+00  1.000e+00  1.500e+00
1986-12-28 00:06:20 172.20.0.5       4      0.000 1000000.000 -1.215e+09 N  1  1.828e-05  0.000e+00  6.028e-02  1.215e+09  1.500e+00
2025-06-26 20:36:43 172.20.0.5       4      0.000 1000000.000  1.824e-02 N  1  4.430e-05  0.000e+00  6.023e-02  2.484e-01  1.215e+09
2025-06-26 20:36:44 172.20.0.5       4      0.000 1000000.000  1.956e-03 N  1  1.041e-04 -1.480e-02  6.025e-02  1.617e-01  1.584e+00
2025-06-26 20:36:45 172.20.0.5       4      0.000 1000000.000  2.851e-04 N  1  3.381e-04 -1.402e-02  6.026e-02  5.408e-01  1.221e+00
2025-06-26 20:36:46 172.20.0.5       4      0.000 1000000.000  1.792e-04 N  1  4.435e-04 -1.152e-02  6.029e-02  1.371e-01  1.612e+00
2025-06-26 20:36:47 172.20.0.5       4      0.000 1000000.000  5.062e-05 N  1  5.028e-04 -8.949e-03  6.035e-02  1.425e-02  1.196e+00
2025-06-26 20:36:48 fd00:1122:3344:104::e  5      0.000 1000000.000 -7.627e-04 N  1  9.239e-05 -6.270e-03  4.888e-04  3.630e-01  1.062e+00
2025-06-26 20:37:12 172.20.0.5       4    -51.436     19.039 -2.382e-03 N  2  8.499e-04  6.244e-07  6.030e-02  1.563e-03  2.468e+01
2025-06-26 20:37:13 172.20.0.5       4    -53.788     26.266 -2.795e-05 N  2  8.753e-04  2.199e-03  6.036e-02  1.569e-03  3.418e-02
2025-06-26 20:37:14 172.20.0.5       4    -57.315     48.603 -1.174e-04 N  2  8.936e-04 -4.540e-06  6.039e-02  1.675e-03  3.406e-02
2025-06-26 20:37:15 172.20.0.5       4    -62.292     77.018  4.927e-05 N  2  9.232e-04  1.034e-04  6.035e-02  1.627e-03  3.213e-02
2025-06-26 20:37:16 172.20.0.5       4    -61.483    115.652  1.451e-05 N  2  9.534e-04  5.195e-05  6.036e-02  1.584e-03  3.200e-02
2025-06-26 20:37:17 172.20.0.5       4    -52.413    169.462  2.268e-05 N  2  9.864e-04  3.647e-05  6.037e-02  1.653e-03  3.198e-02
2025-06-26 20:37:18 172.20.0.5       4    -39.644    164.582 -8.310e-06 N  2  1.015e-03  1.365e-05  6.030e-02  1.739e-03  3.209e-02
2025-06-26 20:37:19 172.20.0.5       4    -31.197    144.915 -1.305e-05 N  2  1.046e-03  2.164e-05  6.027e-02  1.593e-03  3.214e-02
2025-06-26 20:37:20 172.20.0.5       4    -45.869     30.363 -4.750e-05 N  2  1.071e-03  3.393e-05  6.027e-02  1.587e-03  3.197e-02
2025-06-26 20:37:21 172.20.0.5       4    -48.341     33.671 -2.181e-05 N  2  1.098e-03  7.727e-05  6.038e-02  1.565e-03  3.190e-02
2025-06-26 20:37:22 172.20.0.5       4    -46.240     19.521 -5.900e-06 N  2  1.128e-03  9.322e-05  6.029e-02  1.546e-03  3.195e-02
2025-06-26 20:37:23 172.20.0.5       4    -48.521     19.738 -2.412e-05 N  2  1.154e-03  9.327e-05  6.029e-02  1.533e-03  3.187e-02
2025-06-26 20:37:25 172.20.0.5       4    -47.963     16.253  6.905e-06 N  2  1.188e-03  1.093e-04  6.029e-02  1.587e-03  3.188e-02
2025-06-26 20:37:26 172.20.0.5       4    -46.444     11.530 -1.082e-05 N  2  1.217e-03  9.660e-05  6.029e-02  1.591e-03  3.191e-02
2025-06-26 20:37:27 172.20.0.5       4    -45.219     10.572 -7.761e-06 N  2  1.247e-03  1.011e-04  6.033e-02  1.534e-03  3.191e-02
2025-06-26 20:37:28 172.20.0.5       4    -46.631     11.807 -1.863e-05 N  2  1.273e-03  1.025e-04  6.033e-02  1.582e-03  3.188e-02
2025-06-26 20:37:29 172.20.0.5       4    -48.584     14.174 -2.018e-05 N  2  1.299e-03  1.135e-04  6.033e-02  1.603e-03  3.194e-02
2025-06-26 20:37:30 172.20.0.5       4    -50.594     15.508 -2.555e-05 N  2  1.321e-03  1.245e-04  6.038e-02  1.562e-03  3.197e-02
2025-06-26 20:37:31 172.20.0.5       4    -53.849     26.274 -2.880e-05 N  2  1.341e-03  1.387e-04  6.029e-02  1.572e-03  3.198e-02
2025-06-26 20:37:32 172.20.0.5       4    -59.275     25.924 -1.837e-05 N  2  1.364e-03  1.535e-04  6.029e-02  1.597e-03  3.197e-02
===================================================================================================================================
   Date (UTC) Time     IP Address   St   Freq ppm   Skew ppm     Offset L Co  Offset sd Rem. corr. Root delay Root disp. Max. error
===================================================================================================================================
2025-06-26 20:37:33 172.20.0.5       4    -56.452     34.127  7.542e-06 N  2  1.393e-03  1.573e-04  6.030e-02  1.530e-03  3.200e-02
2025-06-26 20:37:34 172.20.0.5       4    -56.366     21.496 -1.166e-05 N  2  1.416e-03  1.391e-04  6.033e-02  1.573e-03  3.193e-02
2025-06-26 20:37:35 172.20.0.5       4    -54.293     14.426  3.372e-08 N  2  1.443e-03  1.400e-04  6.029e-02  1.519e-03  3.197e-02
2025-06-26 20:37:36 172.20.0.5       4    -53.369     10.498 -1.386e-06 N  2  1.469e-03  1.310e-04  6.027e-02  1.575e-03  3.188e-02
2025-06-26 20:37:37 172.20.0.5       4    -53.375      7.965 -8.772e-06 N  2  1.493e-03  1.244e-04  6.027e-02  1.573e-03  3.191e-02
2025-06-26 20:37:38 172.20.0.5       4    -52.251     10.150 -1.611e-06 N  2  1.520e-03  1.252e-04  6.028e-02  1.545e-03  3.191e-02
2025-06-26 20:37:39 172.20.0.5       4    -49.841     13.179  1.695e-06 N  2  1.548e-03  1.198e-04  6.027e-02  1.588e-03  3.188e-02
2025-06-26 20:37:40 172.20.0.5       4    -50.173     12.300 -1.243e-05 N  2  1.571e-03  1.122e-04  6.027e-02  1.573e-03  3.191e-02
2025-06-26 20:37:41 172.20.0.5       4    -49.739     12.484 -6.966e-06 N  2  1.596e-03  1.180e-04  6.027e-02  1.553e-03  3.190e-02
2025-06-26 20:37:42 172.20.0.5       4    -48.970     13.611  3.969e-04 N  1  9.391e-06  1.184e-04  6.032e-02  1.542e-03  3.188e-02
2025-06-26 20:37:43 172.20.0.5       4    -47.021     27.147  1.963e-04 N  1  3.667e-05  9.092e-08  6.025e-02  1.639e-03  3.206e-02
2025-06-26 20:37:44 172.20.0.5       4    -40.197     56.797  4.107e-05 N  1  4.134e-05  2.047e-07  6.032e-02  1.628e-03  3.209e-02
2025-06-26 20:37:45 172.20.0.5       4    -39.126     85.251 -8.542e-05 N  1  7.949e-05 -1.418e-05  6.028e-02  1.674e-03  3.199e-02
2025-06-26 20:37:46 172.20.0.5       4    -44.082    137.099 -8.585e-05 N  1  9.367e-05  2.889e-05  6.035e-02  1.711e-03  3.204e-02
2025-06-26 20:37:47 172.20.0.5       4    -71.426    205.069 -1.082e-04 N  1  8.124e-05  2.093e-05  6.039e-02  1.609e-03  3.222e-02
2025-06-26 20:37:48 172.20.0.5       4   -116.206    100.707 -9.032e-05 N  1  3.662e-05 -2.428e-08  6.045e-02  1.684e-03  3.222e-02
2025-06-26 20:37:49 172.20.0.5       4    -99.919    123.131  1.002e-04 N  1  4.781e-05 -2.162e-08  6.034e-02  1.726e-03  3.217e-02
2025-06-26 20:37:50 172.20.0.5       4    -66.185    133.458  1.324e-04 N  1  4.935e-05  2.969e-08  6.039e-02  1.704e-03  3.219e-02
2025-06-26 20:37:51 172.20.0.5       4    -36.204    132.895  8.892e-05 N  1  3.643e-05  2.034e-08  6.032e-02  1.684e-03  3.224e-02
2025-06-26 20:37:52 172.20.0.5       4    -14.480     96.179  3.746e-05 N  1  2.167e-05  9.678e-08  6.031e-02  1.651e-03  3.213e-02
2025-06-26 20:37:53 fd00:1122:3344:104::e  5    -66.827     30.016 -8.920e-05 N  1  6.623e-05 -6.799e-06  4.888e-04  1.695e-03  3.197e-02
2025-06-26 20:38:57 fd00:1122:3344:104::e  5    -58.273     36.769  1.200e-03 N  1  2.690e-04 -1.880e-08  4.888e-04  4.694e-03  7.844e-03
2025-06-26 20:40:02 fd00:1122:3344:104::e  5    -55.422     18.520  1.843e-04 N  1  2.816e-04  1.343e-08  4.888e-04  3.658e-03  1.264e-02
2025-06-26 20:41:06 fd00:1122:3344:104::e  5    -54.360      9.941  1.913e-04 N  1  2.378e-04 -1.044e-04  4.888e-04  2.713e-03  9.174e-03
2025-06-26 20:42:11 fd00:1122:3344:104::e  5    -53.257      6.499  2.141e-04 N  1  2.069e-04 -5.194e-05  4.888e-04  2.208e-03  7.754e-03
2025-06-26 20:43:15 fd00:1122:3344:104::e  5    -52.807      4.374  9.604e-05 N  1  1.876e-04 -3.572e-05  4.888e-04  1.871e-03  7.043e-03
2025-06-26 20:44:20 fd00:1122:3344:104::e  5    -52.117      3.199  2.241e-04 N  1  1.613e-04 -7.014e-05  4.888e-04  1.571e-03  6.424e-03
2025-06-26 20:45:26 fd00:1122:3344:104::e  5    -51.987      2.388  9.558e-05 N  1  1.379e-04  4.752e-10  4.888e-04  1.427e-03  6.240e-03
2025-06-26 20:46:30 fd00:1122:3344:104::e  5    -51.862      2.037  1.606e-05 N  1  1.390e-04 -5.205e-05  4.888e-04  1.367e-03  5.782e-03
2025-06-26 20:47:34 fd00:1122:3344:104::e  5    -51.760      1.732  4.381e-05 N  1  1.359e-04 -4.591e-05  4.888e-04  1.306e-03  5.662e-03
2025-06-26 20:48:38 fd00:1122:3344:104::e  5    -51.680      1.459  3.710e-05 N  1  1.294e-04 -5.021e-05  4.888e-04  1.240e-03  5.605e-03
2025-06-26 20:49:42 fd00:1122:3344:104::e  5    -51.621      1.277  3.040e-05 N  1  1.294e-04 -4.803e-05  4.888e-04  1.220e-03  5.518e-03
===================================================================================================================================
   Date (UTC) Time     IP Address   St   Freq ppm   Skew ppm     Offset L Co  Offset sd Rem. corr. Root delay Root disp. Max. error
===================================================================================================================================
2025-06-26 20:50:48 fd00:1122:3344:104::e  5    -51.537      1.075  5.004e-05 N  1  1.161e-04 -4.628e-05  4.888e-04  1.082e-03  5.536e-03
2025-06-26 20:51:52 fd00:1122:3344:104::e  5    -51.424      0.874  6.863e-05 N  1  9.549e-05 -4.350e-05  4.888e-04  9.631e-04  5.369e-03
2025-06-26 20:52:57 fd00:1122:3344:104::e  5    -51.381      0.740  2.157e-05 N  1  9.068e-05 -2.461e-05  4.888e-04  8.681e-04  5.241e-03
2025-06-26 20:54:01 fd00:1122:3344:104::e  5    -51.221      0.611  6.819e-05 N  1  6.523e-05 -3.053e-05  6.040e-02  1.404e-03  5.062e-03
2025-06-26 20:55:06 fd00:1122:3344:104::e  5    -51.185      0.489  1.316e-05 N  1  6.024e-05  3.457e-10  6.040e-02  1.402e-03  3.563e-02
2025-06-26 20:55:42 172.20.0.5       4    -51.178      0.461  2.345e-05 N  2  5.703e-05 -1.209e-05  6.031e-02  1.645e-03  3.379e-02
2025-06-26 20:55:43 172.20.0.5       4    -51.177      0.452 -1.963e-06 N  2  5.583e-05 -3.513e-05  6.031e-02  1.642e-03  3.190e-02
2025-06-26 20:55:44 172.20.0.5       4    -51.177      0.450 -2.640e-06 N  2  5.372e-05 -1.992e-05  6.032e-02  1.636e-03  3.189e-02
2025-06-26 20:55:45 172.20.0.5       4    -51.177      0.449  2.337e-06 N  2  5.513e-05 -1.359e-05  6.025e-02  1.642e-03  3.187e-02

[citrus-it]

@JustinAzoff had the idea of capturing NTP network traffic during a dogfood update to get a better idea of what was going back and forth between the NTP server and the rack. I ran this on catacomb during the 2025-07-03 update:

pfexec snoop -rd ixgbe0 -o dogfood_upgrade_ntp.pcap udp port 123

No NTP issues surfaced during this upgrade and the resulting capture file is at

/staff/core/omicron-7668/dogfood_upgrade_ntp_20250703_good.pcap

[labbott]

Captured more logs in /staff/core/dogfood-mupdate-20250703/omicron_7668_logs

[askfongjojo]

The cockroachdb issue happened again today after rack2 upgrade. The related chrony and ntp logs have been pulled down to /staff/core/omicron-7668/20250703.

FWIW, these are the one-liners for grabbing them:

for n in 9 10 14 16 17; do mkdir -p /staff/core/omicron-7668/20250703/sled${n}-crdb-ntp; cd /staff/core/omicron-7668/20250703/sled${n}-crdb-ntp; scp gc${n}:/pool/ext/*/crypt/debug/oxz_ntp*/oxide-ntp*.log.175155* .; done
for n in 12 21; do mkdir -p /staff/core/omicron-7668/20250703/sled${n}-boundary-ntp; cd /staff/core/omicron-7668/20250703/sled${n}-boundary-ntp; scp gc${n}:/pool/ext/*/crypt/debug/oxz_ntp*/oxide-ntp*.log.175155* .; done

for n in 9 10 14 16 17; do mkdir -p /staff/core/omicron-7668/20250703/sled${n}-crdb-ntp; cd /staff/core/omicron-7668/20250703/sled${n}-crdb-ntp; scp gc${n}:/pool/ext/*/crypt/zone/oxz_ntp_*/root/var/log/chrony/*.log .; done
for n in 12 21; do mkdir -p /staff/core/omicron-7668/20250703/sled${n}-boundary-ntp; cd /staff/core/omicron-7668/20250703/sled${n}-boundary-ntp; scp gc${n}:/pool/ext/*/crypt/zone/oxz_ntp_*/root/var/log/chrony/*.log .; done