| 1 | | [ May 27 20:57:35 Disabled. ] |
| 2 | | [ May 27 20:57:35 Rereading configuration. ] |
| 3 | | [ May 27 20:57:36 Enabled. ] |
| 4 | | [ May 27 20:57:36 Rereading configuration. ] |
| 5 | | [ May 27 20:57:36 Executing start method ("ctrun -l child -o noorphan,regent /opt/oxide/mg-ddm/pkg/ddm_method_script.sh &"). ] |
| 6 | | [ May 27 20:57:36 Method "start" exited with status 0. ] |
| 7 | 2026-05-27T20:57:36.743Z | INFO | slog-rs: signal handler waiting for context
|
| 8 | 2026-05-27T20:57:36.780Z | WARN | slog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
|
| 9 | 2026-05-27T20:57:36.781Z | ERRO | slog-rs: failed to start stats server: underlay address not found
|
| 10 | 2026-05-27T20:57:36.781Z | INFO | slog-rs: signal handler waiting got context
|
| 11 | 2026-05-27T20:57:36.781Z | INFO | slog-rs: admin: listening on [::1]:8000
|
| 12 | 2026-05-27T20:57:36.782Z | INFO | slog-rs: admin: server started
|
| 13 | 2026-05-27T20:57:37.781Z | WARN | slog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
|
| 14 | 2026-05-27T20:57:38.781Z | WARN | slog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
|
| 15 | 2026-05-27T20:57:39.821Z | WARN | slog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
|
| 16 | 2026-05-27T20:57:40.821Z | WARN | slog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
|
| 17 | 2026-05-27T20:57:41.867Z | WARN | slog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
|
| 18 | 2026-05-27T20:57:42.867Z | WARN | slog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
|
| 19 | 2026-05-27T20:57:43.867Z | WARN | slog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
|
| 20 | 2026-05-27T20:57:44.867Z | WARN | slog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
|
| 21 | 2026-05-27T20:57:45.867Z | WARN | slog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
|
| 22 | 2026-05-27T20:57:46.868Z | WARN | slog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
|
| 23 | 2026-05-27T20:57:47.868Z | INFO | slog-rs: [tfportrear0_0] sm initialized with addr fe80::3c4f:8ff:fe11:2346 on if tfportrear0_0 index 3
|
| 24 | 2026-05-27T20:57:47.868Z | DEBG | slog-rs: [tfportrear0_0] starting discovery handler
|
| 25 | 2026-05-27T20:57:47.868Z | INFO | slog-rs: [tfportrear0_0] nbr is fe80::10cb:66ff:fe8b:39fc@sock server
|
| 26 | 2026-05-27T20:57:47.868Z | DEBG | slog-rs: [tfportrear0_0] transition solicit -> exchange
|
| 27 | 2026-05-27T20:57:47.868Z | INFO | slog-rs: [tfportrear0_0] exchange: listening on [fe80::3c4f:8ff:fe11:2346]:56797
|
| 28 | 2026-05-27T20:57:47.869Z | INFO | slog-rs: [tfportrear0_0] waiting for exchange server to start
|
| 29 | | [ May 27 20:57:48 Rereading configuration. ] |
| 30 | | [ May 27 20:57:48 Executing refresh method ("/opt/oxide/mg-ddm/pkg/refresh_ctrun_child.sh -c 213"). ] |
| 31 | | + CONTRACT='' |
| 32 | | + getopts c: o |
| 33 | | + CONTRACT=213 |
| 34 | | + getopts c: o |
| 35 | | + [[ -n 213 ]] |
| 36 | | + pgrep -c 213 |
| 37 | | + PARENT_PID=2286 |
| 38 | | + [[ -n 2286 ]] |
| 39 | | + pgrep -P 2286 |
| 40 | | + CHILD_PID=2287 |
| 41 | | + [[ -n 2287 ]] |
| 42 | | + kill -USR1 2287 |
| 43 | 2026-05-27T20:57:48.267Z | INFO | slog-rs: handling smf refresh
|
| 44 | | [ May 27 20:57:48 Method "refresh" exited with status 0. ] |
| 45 | 2026-05-27T20:57:48.269Z | INFO | slog-rs: starting stats server on smf refresh
|
| 46 | | May 27 20:57:48.269 DEBG registered DTrace probes |
| 47 | | May 27 20:57:48.270 DEBG registered endpoint, versions: all, path: /{producer_id}, method: GET, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot |
| 48 | | May 27 20:57:48.270 INFO listening, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.7/src/server.rs:328 |
| 49 | | May 27 20:57:48.270 DEBG successfully registered DTrace USDT probes, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot |
| 50 | | May 27 20:57:48.270 DEBG Nexus IP not provided, will use DNS to resolve it |
| 51 | | May 27 20:57:48.270 INFO new DNS resolver, addresses: [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53], component: internal-dns-resolver, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/dfcd9ac/internal-dns/resolver/src/resolver.rs:144 |
| 52 | | May 27 20:57:48.270 DEBG starting producer registration task |
| 53 | | May 27 20:57:48.270 INFO starting oximeter metric producer server, interval: 1s, address: [fd00:1122:3344:101::2]:8001, producer_id: 78e45f3a-93dc-4bce-bdaf-6dab436238aa, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/dfcd9ac/oximeter/producer/src/lib.rs:283 |
| 54 | | May 27 20:57:48.271 DEBG registering as oximeter producer with Nexus, registration_count: 0, component: producer-registration-task |
| 55 | | May 27 20:57:48.272 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("44cb9ec0-dc86-4fc8-b845-9f7a3f3236cb.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("816dd9ca-b0e8-4c88-8f90-0565f3cfdf4c.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("9896502c-ee75-44fe-a03d-86a8ef5b60e5.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("9896502c-ee75-44fe-a03d-86a8ef5b60e5.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::b)) }], valid_until: Instant { tv_sec: 1902, tv_nsec: 225230695 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver |
| 56 | | May 27 20:57:48.272 DEBG will register with Nexus at [fd00:1122:3344:101::a]:12221, component: producer-registration-task |
| 57 | | May 27 20:57:48.274 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::a]:12221/metrics/producers, method: POST, component: producer-registration-task |
| 58 | 2026-05-27T20:57:49.482Z | INFO | slog-rs: advertise tunnel: {
TunnelOrigin {
overlay_prefix: V4(
Ipv4Net {
addr: 0.0.0.0,
width: 0,
},
),
boundary_addr: fdad:65eb:fe26:a38e::1,
vni: 99,
metric: 18446744073709551615,
},
}
|
| 59 | 2026-05-27T20:57:49.871Z | WARN | slog-rs: [tfportrear0_0] exchange pull: hyper client error: client error (Connect)
|
| 60 | 2026-05-27T20:57:49.872Z | INFO | slog-rs: [tfportrear0_0] sending 5 routes to dendrite
|
| 61 | 2026-05-27T20:57:49.872Z | DEBG | slog-rs: [tfportrear0_0] sending to dpd host=localhost port=12224
|
| 62 | 2026-05-27T20:57:49.872Z | INFO | slog-rs: [tfportrear0_0] adding route fd00:1122:3344:1:: -> fe80::10cb:66ff:fe8b:39fc on port Rear(Rear("rear0"))/0
|
| 63 | 2026-05-27T20:57:49.873Z | INFO | slog-rs: [tfportrear0_0] adding route fd00:1122:3344:3:: -> fe80::10cb:66ff:fe8b:39fc on port Rear(Rear("rear0"))/0
|
| 64 | 2026-05-27T20:57:49.874Z | INFO | slog-rs: [tfportrear0_0] adding route fd00:1122:3344:2:: -> fe80::10cb:66ff:fe8b:39fc on port Rear(Rear("rear0"))/0
|
| 65 | 2026-05-27T20:57:49.874Z | INFO | slog-rs: [tfportrear0_0] adding route fdb0:18c0:4d0d:a02a:: -> fe80::10cb:66ff:fe8b:39fc on port Rear(Rear("rear0"))/0
|
| 66 | 2026-05-27T20:57:49.874Z | INFO | slog-rs: [tfportrear0_0] adding route fd00:1122:3344:101:: -> fe80::10cb:66ff:fe8b:39fc on port Rear(Rear("rear0"))/0
|
| 67 | 2026-05-27T20:57:49.875Z | INFO | slog-rs: [tfportrear0_0] removing routes 0 from dendrite
|
| 68 | 2026-05-27T20:57:49.875Z | DEBG | slog-rs: [tfportrear0_0] redistributing update to 0 peers
|
| 69 | | May 27 20:58:03.177 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::a]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task |
| 70 | | May 27 20:58:03.177 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers): operation timed out, delay: 15.00480473s, count: 0, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/dfcd9ac/oximeter/producer/src/lib.rs:379 |
| 71 | | May 27 20:58:03.348 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("44cb9ec0-dc86-4fc8-b845-9f7a3f3236cb.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("816dd9ca-b0e8-4c88-8f90-0565f3cfdf4c.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("9896502c-ee75-44fe-a03d-86a8ef5b60e5.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("9896502c-ee75-44fe-a03d-86a8ef5b60e5.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::b)) }], valid_until: Instant { tv_sec: 1917, tv_nsec: 399508096 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver |
| 72 | | May 27 20:58:03.349 DEBG will register with Nexus at [fd00:1122:3344:101::a]:12221, component: producer-registration-task |
| 73 | | May 27 20:58:03.350 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::a]:12221/metrics/producers, method: POST, component: producer-registration-task |
| 74 | 2026-05-27T20:58:06.075Z | WARN | slog-rs: [tfportrear0_0] neighbor sock@fe80::10cb:66ff:fe8b:39fc missed solicit interval
|
| 75 | 2026-05-27T20:58:08.075Z | WARN | slog-rs: [tfportrear0_0] neighbor sock@fe80::10cb:66ff:fe8b:39fc missed solicit interval
|
| 76 | 2026-05-27T20:58:10.075Z | WARN | slog-rs: [tfportrear0_0] neighbor sock@fe80::10cb:66ff:fe8b:39fc missed solicit interval
|
| 77 | 2026-05-27T20:58:12.075Z | WARN | slog-rs: [tfportrear0_0] neighbor sock@fe80::10cb:66ff:fe8b:39fc missed solicit interval
|
| 78 | 2026-05-27T20:58:14.075Z | WARN | slog-rs: [tfportrear0_0] neighbor sock@fe80::10cb:66ff:fe8b:39fc missed solicit interval
|
| 79 | 2026-05-27T20:58:16.075Z | WARN | slog-rs: [tfportrear0_0] neighbor sock@fe80::10cb:66ff:fe8b:39fc missed solicit interval
|
| 80 | 2026-05-27T20:58:18.075Z | WARN | slog-rs: [tfportrear0_0] neighbor sock@fe80::10cb:66ff:fe8b:39fc missed solicit interval
|
| 81 | | May 27 20:58:18.619 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::a]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task |
| 82 | | May 27 20:58:18.619 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers): operation timed out, delay: 30.178842233s, count: 1, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/dfcd9ac/oximeter/producer/src/lib.rs:379 |
| 83 | | May 27 20:58:18.954 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("44cb9ec0-dc86-4fc8-b845-9f7a3f3236cb.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("816dd9ca-b0e8-4c88-8f90-0565f3cfdf4c.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("9896502c-ee75-44fe-a03d-86a8ef5b60e5.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("9896502c-ee75-44fe-a03d-86a8ef5b60e5.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::b)) }], valid_until: Instant { tv_sec: 1932, tv_nsec: 737838166 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver |
| 84 | | May 27 20:58:18.954 DEBG will register with Nexus at [fd00:1122:3344:101::a]:12221, component: producer-registration-task |
| 85 | | May 27 20:58:18.955 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::a]:12221/metrics/producers, method: POST, component: producer-registration-task |
| 86 | 2026-05-27T20:58:20.075Z | WARN | slog-rs: [tfportrear0_0] neighbor sock@fe80::10cb:66ff:fe8b:39fc missed solicit interval
|
| 87 | 2026-05-27T20:58:22.075Z | WARN | slog-rs: [tfportrear0_0] neighbor sock@fe80::10cb:66ff:fe8b:39fc missed solicit interval
|
| 88 | 2026-05-27T20:58:24.076Z | WARN | slog-rs: [tfportrear0_0] neighbor sock@fe80::10cb:66ff:fe8b:39fc missed solicit interval
|
| 89 | 2026-05-27T20:58:26.076Z | WARN | slog-rs: [tfportrear0_0] neighbor sock@fe80::10cb:66ff:fe8b:39fc missed solicit interval
|
| 90 | 2026-05-27T20:58:28.076Z | WARN | slog-rs: [tfportrear0_0] neighbor sock@fe80::10cb:66ff:fe8b:39fc missed solicit interval
|
| 91 | 2026-05-27T20:58:30.076Z | WARN | slog-rs: [tfportrear0_0] neighbor sock@fe80::10cb:66ff:fe8b:39fc missed solicit interval
|
| 92 | 2026-05-27T20:58:32.076Z | WARN | slog-rs: [tfportrear0_0] neighbor sock@fe80::10cb:66ff:fe8b:39fc missed solicit interval
|
| 93 | | May 27 20:58:33.916 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::a]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task |
| 94 | | May 27 20:58:33.916 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers): operation timed out, delay: 45.517102911s, count: 2, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/dfcd9ac/oximeter/producer/src/lib.rs:379 |
| 95 | 2026-05-27T20:58:34.076Z | WARN | slog-rs: [tfportrear0_0] neighbor sock@fe80::10cb:66ff:fe8b:39fc missed solicit interval
|
| 96 | | May 27 20:58:35.250 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("44cb9ec0-dc86-4fc8-b845-9f7a3f3236cb.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("816dd9ca-b0e8-4c88-8f90-0565f3cfdf4c.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("9896502c-ee75-44fe-a03d-86a8ef5b60e5.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("9896502c-ee75-44fe-a03d-86a8ef5b60e5.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::b)) }], valid_until: Instant { tv_sec: 1949, tv_nsec: 79095955 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver |
| 97 | | May 27 20:58:35.251 DEBG will register with Nexus at [fd00:1122:3344:101::a]:12221, component: producer-registration-task |
| 98 | | May 27 20:58:35.252 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::a]:12221/metrics/producers, method: POST, component: producer-registration-task |
| 99 | 2026-05-27T20:58:36.076Z | WARN | slog-rs: [tfportrear0_0] neighbor sock@fe80::10cb:66ff:fe8b:39fc missed solicit interval
|
| 100 | 2026-05-27T20:58:38.076Z | WARN | slog-rs: [tfportrear0_0] neighbor sock@fe80::10cb:66ff:fe8b:39fc missed solicit interval
|
| 101 | 2026-05-27T20:58:40.076Z | WARN | slog-rs: [tfportrear0_0] neighbor sock@fe80::10cb:66ff:fe8b:39fc missed solicit interval
|
| 102 | 2026-05-27T20:58:42.076Z | WARN | slog-rs: [tfportrear0_0] neighbor sock@fe80::10cb:66ff:fe8b:39fc missed solicit interval
|
| 103 | 2026-05-27T20:58:44.076Z | WARN | slog-rs: [tfportrear0_0] neighbor sock@fe80::10cb:66ff:fe8b:39fc missed solicit interval
|
| 104 | 2026-05-27T20:58:46.076Z | WARN | slog-rs: [tfportrear0_0] neighbor sock@fe80::10cb:66ff:fe8b:39fc missed solicit interval
|
| 105 | 2026-05-27T20:58:48.076Z | WARN | slog-rs: [tfportrear0_0] neighbor sock@fe80::10cb:66ff:fe8b:39fc missed solicit interval
|
| 106 | 2026-05-27T20:58:50.076Z | WARN | slog-rs: [tfportrear0_0] neighbor sock@fe80::10cb:66ff:fe8b:39fc missed solicit interval
|
| 107 | | May 27 20:58:50.212 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::a]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task |
| 108 | | May 27 20:58:50.212 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers): operation timed out, delay: 61.857534923s, count: 3, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/dfcd9ac/oximeter/producer/src/lib.rs:379 |
| 109 | 2026-05-27T20:58:52.076Z | WARN | slog-rs: [tfportrear0_0] neighbor sock@fe80::10cb:66ff:fe8b:39fc missed solicit interval
|
| 110 | | May 27 20:58:53.052 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("44cb9ec0-dc86-4fc8-b845-9f7a3f3236cb.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("816dd9ca-b0e8-4c88-8f90-0565f3cfdf4c.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("9896502c-ee75-44fe-a03d-86a8ef5b60e5.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("9896502c-ee75-44fe-a03d-86a8ef5b60e5.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::b)) }], valid_until: Instant { tv_sec: 1966, tv_nsec: 929119170 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver |
| 111 | | May 27 20:58:53.052 DEBG will register with Nexus at [fd00:1122:3344:101::a]:12221, component: producer-registration-task |
| 112 | | May 27 20:58:53.053 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::a]:12221/metrics/producers, method: POST, component: producer-registration-task |
| 113 | 2026-05-27T20:58:54.076Z | WARN | slog-rs: [tfportrear0_0] neighbor sock@fe80::10cb:66ff:fe8b:39fc missed solicit interval
|
| 114 | 2026-05-27T20:58:56.076Z | WARN | slog-rs: [tfportrear0_0] neighbor sock@fe80::10cb:66ff:fe8b:39fc missed solicit interval
|
| 115 | 2026-05-27T20:58:58.076Z | WARN | slog-rs: [tfportrear0_0] neighbor sock@fe80::10cb:66ff:fe8b:39fc missed solicit interval
|
| 116 | 2026-05-27T20:59:00.076Z | WARN | slog-rs: [tfportrear0_0] neighbor sock@fe80::10cb:66ff:fe8b:39fc missed solicit interval
|
| 117 | 2026-05-27T20:59:02.076Z | WARN | slog-rs: [tfportrear0_0] neighbor sock@fe80::10cb:66ff:fe8b:39fc missed solicit interval
|
| 118 | 2026-05-27T20:59:04.076Z | WARN | slog-rs: [tfportrear0_0] neighbor sock@fe80::10cb:66ff:fe8b:39fc missed solicit interval
|
| 119 | 2026-05-27T20:59:06.076Z | WARN | slog-rs: [tfportrear0_0] neighbor sock@fe80::10cb:66ff:fe8b:39fc missed solicit interval
|
| 120 | | May 27 20:59:08.014 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::a]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task |
| 121 | | May 27 20:59:08.014 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers): operation timed out, delay: 79.707965221s, count: 4, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/dfcd9ac/oximeter/producer/src/lib.rs:379 |
| 122 | 2026-05-27T20:59:08.076Z | WARN | slog-rs: [tfportrear0_0] neighbor sock@fe80::10cb:66ff:fe8b:39fc missed solicit interval
|
| 123 | 2026-05-27T20:59:10.076Z | WARN | slog-rs: [tfportrear0_0] neighbor sock@fe80::10cb:66ff:fe8b:39fc missed solicit interval
|
| 124 | | May 27 20:59:12.066 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("44cb9ec0-dc86-4fc8-b845-9f7a3f3236cb.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("816dd9ca-b0e8-4c88-8f90-0565f3cfdf4c.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("9896502c-ee75-44fe-a03d-86a8ef5b60e5.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("9896502c-ee75-44fe-a03d-86a8ef5b60e5.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::b)) }], valid_until: Instant { tv_sec: 1985, tv_nsec: 995564249 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver |
| 125 | | May 27 20:59:12.067 DEBG will register with Nexus at [fd00:1122:3344:101::a]:12221, component: producer-registration-task |
| 126 | | May 27 20:59:12.068 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::a]:12221/metrics/producers, method: POST, component: producer-registration-task |
| 127 | 2026-05-27T20:59:12.077Z | WARN | slog-rs: [tfportrear0_0] neighbor sock@fe80::10cb:66ff:fe8b:39fc missed solicit interval
|
| 128 | 2026-05-27T20:59:14.077Z | WARN | slog-rs: [tfportrear0_0] neighbor sock@fe80::10cb:66ff:fe8b:39fc missed solicit interval
|
| 129 | 2026-05-27T20:59:16.077Z | WARN | slog-rs: [tfportrear0_0] neighbor sock@fe80::10cb:66ff:fe8b:39fc missed solicit interval
|
| 130 | 2026-05-27T20:59:18.077Z | WARN | slog-rs: [tfportrear0_0] neighbor sock@fe80::10cb:66ff:fe8b:39fc missed solicit interval
|
| 131 | 2026-05-27T20:59:20.077Z | WARN | slog-rs: [tfportrear0_0] neighbor sock@fe80::10cb:66ff:fe8b:39fc missed solicit interval
|
| 132 | 2026-05-27T20:59:22.077Z | WARN | slog-rs: [tfportrear0_0] neighbor sock@fe80::10cb:66ff:fe8b:39fc missed solicit interval
|
| 133 | 2026-05-27T20:59:24.077Z | WARN | slog-rs: [tfportrear0_0] neighbor sock@fe80::10cb:66ff:fe8b:39fc missed solicit interval
|
| 134 | 2026-05-27T20:59:26.077Z | WARN | slog-rs: [tfportrear0_0] neighbor sock@fe80::10cb:66ff:fe8b:39fc missed solicit interval
|
| 135 | | May 27 20:59:27.028 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::a]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task |
| 136 | | May 27 20:59:27.028 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers): operation timed out, delay: 98.774461342s, count: 5, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/dfcd9ac/oximeter/producer/src/lib.rs:379 |
| 137 | 2026-05-27T20:59:28.077Z | WARN | slog-rs: [tfportrear0_0] neighbor sock@fe80::10cb:66ff:fe8b:39fc missed solicit interval
|
| 138 | 2026-05-27T20:59:30.077Z | WARN | slog-rs: [tfportrear0_0] neighbor sock@fe80::10cb:66ff:fe8b:39fc missed solicit interval
|
| 139 | 2026-05-27T20:59:32.077Z | WARN | slog-rs: [tfportrear0_0] neighbor sock@fe80::10cb:66ff:fe8b:39fc missed solicit interval
|
| 140 | | May 27 20:59:33.758 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("44cb9ec0-dc86-4fc8-b845-9f7a3f3236cb.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("816dd9ca-b0e8-4c88-8f90-0565f3cfdf4c.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("9896502c-ee75-44fe-a03d-86a8ef5b60e5.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("9896502c-ee75-44fe-a03d-86a8ef5b60e5.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::b)) }], valid_until: Instant { tv_sec: 2007, tv_nsec: 746535165 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver |
| 141 | | May 27 20:59:33.759 DEBG will register with Nexus at [fd00:1122:3344:101::a]:12221, component: producer-registration-task |
| 142 | | May 27 20:59:33.760 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::a]:12221/metrics/producers, method: POST, component: producer-registration-task |
| 143 | 2026-05-27T20:59:34.077Z | WARN | slog-rs: [tfportrear0_0] neighbor sock@fe80::10cb:66ff:fe8b:39fc missed solicit interval
|
| 144 | 2026-05-27T20:59:36.077Z | WARN | slog-rs: [tfportrear0_0] neighbor sock@fe80::10cb:66ff:fe8b:39fc missed solicit interval
|
| 145 | 2026-05-27T20:59:38.077Z | WARN | slog-rs: [tfportrear0_0] neighbor sock@fe80::10cb:66ff:fe8b:39fc missed solicit interval
|
| 146 | 2026-05-27T20:59:40.077Z | WARN | slog-rs: [tfportrear0_0] neighbor sock@fe80::10cb:66ff:fe8b:39fc missed solicit interval
|
| 147 | 2026-05-27T20:59:42.077Z | WARN | slog-rs: [tfportrear0_0] neighbor sock@fe80::10cb:66ff:fe8b:39fc missed solicit interval
|
| 148 | 2026-05-27T20:59:44.077Z | WARN | slog-rs: [tfportrear0_0] neighbor sock@fe80::10cb:66ff:fe8b:39fc missed solicit interval
|
| 149 | 2026-05-27T20:59:46.077Z | WARN | slog-rs: [tfportrear0_0] neighbor sock@fe80::10cb:66ff:fe8b:39fc missed solicit interval
|
| 150 | 2026-05-27T20:59:48.077Z | WARN | slog-rs: [tfportrear0_0] neighbor sock@fe80::10cb:66ff:fe8b:39fc missed solicit interval
|
| 151 | | May 27 20:59:48.721 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::a]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task |
| 152 | | May 27 20:59:48.721 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers): operation timed out, delay: 120.526672639s, count: 6, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/dfcd9ac/oximeter/producer/src/lib.rs:379 |
| 153 | 2026-05-27T20:59:50.077Z | WARN | slog-rs: [tfportrear0_0] neighbor sock@fe80::10cb:66ff:fe8b:39fc missed solicit interval
|
| 154 | 2026-05-27T20:59:52.077Z | WARN | slog-rs: [tfportrear0_0] neighbor sock@fe80::10cb:66ff:fe8b:39fc missed solicit interval
|
| 155 | 2026-05-27T20:59:54.077Z | WARN | slog-rs: [tfportrear0_0] neighbor sock@fe80::10cb:66ff:fe8b:39fc missed solicit interval
|
| 156 | 2026-05-27T20:59:56.077Z | WARN | slog-rs: [tfportrear0_0] neighbor sock@fe80::10cb:66ff:fe8b:39fc missed solicit interval
|
| 157 | 2026-05-27T20:59:58.078Z | WARN | slog-rs: [tfportrear0_0] neighbor sock@fe80::10cb:66ff:fe8b:39fc missed solicit interval
|
| 158 | 2026-05-27T21:00:00.078Z | WARN | slog-rs: [tfportrear0_0] neighbor sock@fe80::10cb:66ff:fe8b:39fc missed solicit interval
|