Max level shown:
1 [ Feb 24 15:42:04 Disabled. ]
2 [ Feb 24 15:42:04 Rereading configuration. ]
3 [ Feb 24 15:42:06 Enabled. ]
4 [ Feb 24 15:42:06 Rereading configuration. ]
5 [ Feb 24 15:42:06 Executing start method ("ctrun -l child -o noorphan,regent /opt/oxide/mg-ddm/pkg/ddm_method_script.sh &"). ]
6 [ Feb 24 15:42:06 Method "start" exited with status 0. ]
72026-02-24T15:42:06.328ZINFOslog-rs: signal handler waiting for context
82026-02-24T15:42:06.348ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
92026-02-24T15:42:06.348ZERROslog-rs: failed to start stats server: underlay address not found
102026-02-24T15:42:06.348ZINFOslog-rs: signal handler waiting got context
112026-02-24T15:42:06.349ZINFOslog-rs: admin: listening on [::1]:8000
122026-02-24T15:42:06.349ZINFOslog-rs: admin: server started
132026-02-24T15:42:07.348ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
142026-02-24T15:42:08.348ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
152026-02-24T15:42:09.348ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
162026-02-24T15:42:10.348ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
172026-02-24T15:42:11.348ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
182026-02-24T15:42:12.348ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
192026-02-24T15:42:13.348ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
202026-02-24T15:42:14.348ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
212026-02-24T15:42:15.349ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
222026-02-24T15:42:16.349ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
232026-02-24T15:42:17.349ZINFOslog-rs: [tfportrear0_0] sm initialized with addr fe80::8c72:21ff:fe51:f53e on if tfportrear0_0 index 3
242026-02-24T15:42:17.349ZDEBGslog-rs: [tfportrear0_0] starting discovery handler
252026-02-24T15:42:17.349ZINFOslog-rs: [tfportrear0_0] nbr is fe80::2cc5:40ff:fea8:3f02@buskin server
262026-02-24T15:42:17.349ZDEBGslog-rs: [tfportrear0_0] transition solicit -> exchange
272026-02-24T15:42:17.350ZINFOslog-rs: [tfportrear0_0] exchange: listening on [fe80::8c72:21ff:fe51:f53e]:56797
282026-02-24T15:42:17.350ZINFOslog-rs: [tfportrear0_0] waiting for exchange server to start
29 [ Feb 24 15:42:17 Rereading configuration. ]
30 [ Feb 24 15:42:17 Executing refresh method ("/opt/oxide/mg-ddm/pkg/refresh_ctrun_child.sh -c 213"). ]
31 + CONTRACT=''
32 + getopts c: o
33 + CONTRACT=213
34 + getopts c: o
35 + [[ -n 213 ]]
36 + pgrep -c 213
37 + PARENT_PID=2285
38 + [[ -n 2285 ]]
39 + pgrep -P 2285
40 + CHILD_PID=2289
41 + [[ -n 2289 ]]
42 + kill -USR1 2289
432026-02-24T15:42:17.745ZINFOslog-rs: handling smf refresh
44 [ Feb 24 15:42:17 Method "refresh" exited with status 0. ]
452026-02-24T15:42:17.748ZINFOslog-rs: starting stats server on smf refresh
46 Feb 24 15:42:17.749 DEBG registered DTrace probes
47 Feb 24 15:42:17.749 DEBG registered endpoint, versions: all, path: /{producer_id}, method: GET, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot
48 Feb 24 15:42:17.749 INFO listening, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.7/src/server.rs:328
49 Feb 24 15:42:17.750 DEBG successfully registered DTrace USDT probes, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot
50 Feb 24 15:42:17.750 DEBG Nexus IP not provided, will use DNS to resolve it
51 Feb 24 15:42:17.750 INFO new DNS resolver, addresses: [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53], component: internal-dns-resolver, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/194594d/internal-dns/resolver/src/resolver.rs:144
52 Feb 24 15:42:17.750 DEBG starting producer registration task
53 Feb 24 15:42:17.750 INFO starting oximeter metric producer server, interval: 1s, address: [fd00:1122:3344:101::2]:8001, producer_id: 621b50bf-a2b7-420f-999d-5ac0f752c864, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/194594d/oximeter/producer/src/lib.rs:279
54 Feb 24 15:42:17.750 DEBG registering as oximeter producer with Nexus, registration_count: 0, component: producer-registration-task
552026-02-24T15:42:19.049ZINFOslog-rs: advertise tunnel: { TunnelOrigin { overlay_prefix: V4( Ipv4Net { addr: 0.0.0.0, width: 0, }, ), boundary_addr: fdc2:2471:e7aa:f619::1, vni: 99, metric: 18446744073709551615, }, }
562026-02-24T15:42:19.352ZWARNslog-rs: [tfportrear0_0] exchange pull: hyper client error: client error (Connect)
572026-02-24T15:42:19.353ZINFOslog-rs: [tfportrear0_0] sending 5 routes to dendrite
582026-02-24T15:42:19.353ZDEBGslog-rs: [tfportrear0_0] sending to dpd host=localhost port=12224
592026-02-24T15:42:19.354ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:3:: -> fe80::2cc5:40ff:fea8:3f02 on port Rear(Rear("rear0"))/0
602026-02-24T15:42:19.355ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:101:: -> fe80::2cc5:40ff:fea8:3f02 on port Rear(Rear("rear0"))/0
612026-02-24T15:42:19.355ZINFOslog-rs: [tfportrear0_0] adding route fdb0:18c0:4d0d:9fb2:: -> fe80::2cc5:40ff:fea8:3f02 on port Rear(Rear("rear0"))/0
622026-02-24T15:42:19.356ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:1:: -> fe80::2cc5:40ff:fea8:3f02 on port Rear(Rear("rear0"))/0
632026-02-24T15:42:19.356ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:2:: -> fe80::2cc5:40ff:fea8:3f02 on port Rear(Rear("rear0"))/0
642026-02-24T15:42:19.356ZINFOslog-rs: [tfportrear0_0] removing routes 0 from dendrite
652026-02-24T15:42:19.356ZDEBGslog-rs: [tfportrear0_0] redistributing update to 0 peers
66 Feb 24 15:42:22.753 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("195695a2-3340-4078-83ac-2b01bd06e64f.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("3e2caf2f-ac6b-4ba7-9a22-2b19e9eb8ad9.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("9d517f13-d3fc-4874-8f9c-627cd5595d8e.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("9d517f13-d3fc-4874-8f9c-627cd5595d8e.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::a)) }], valid_until: Instant { tv_sec: 664, tv_nsec: 520337392 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
67 Feb 24 15:42:22.754 DEBG will register with Nexus at [fd00:1122:3344:101::b]:12221, component: producer-registration-task
68 Feb 24 15:42:22.754 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::b]:12221/metrics/producers, method: POST, component: producer-registration-task
69 Feb 24 15:42:37.755 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::b]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
70 Feb 24 15:42:37.755 WARN failed to register with Nexus, will retry, error: "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers)", delay: 20.004711435s, count: 0, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/194594d/oximeter/producer/src/lib.rs:371
71 Feb 24 15:42:37.894 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("195695a2-3340-4078-83ac-2b01bd06e64f.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("3e2caf2f-ac6b-4ba7-9a22-2b19e9eb8ad9.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("9d517f13-d3fc-4874-8f9c-627cd5595d8e.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("9d517f13-d3fc-4874-8f9c-627cd5595d8e.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::a)) }], valid_until: Instant { tv_sec: 679, tv_nsec: 661117998 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
72 Feb 24 15:42:37.894 DEBG will register with Nexus at [fd00:1122:3344:101::b]:12221, component: producer-registration-task
73 Feb 24 15:42:37.894 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::b]:12221/metrics/producers, method: POST, component: producer-registration-task
742026-02-24T15:42:43.170ZWARNslog-rs: [tfportrear0_0] neighbor buskin@fe80::2cc5:40ff:fea8:3f02 missed solicit interval
752026-02-24T15:42:45.188ZWARNslog-rs: [tfportrear0_0] neighbor buskin@fe80::2cc5:40ff:fea8:3f02 missed solicit interval
762026-02-24T15:42:49.191ZWARNslog-rs: [tfportrear0_0] neighbor buskin@fe80::2cc5:40ff:fea8:3f02 missed solicit interval
772026-02-24T15:42:51.192ZWARNslog-rs: [tfportrear0_0] neighbor buskin@fe80::2cc5:40ff:fea8:3f02 missed solicit interval
78 Feb 24 15:42:52.686 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::b]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
79 Feb 24 15:42:52.686 WARN failed to register with Nexus, will retry, error: "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers)", delay: 35.145974761s, count: 1, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/194594d/oximeter/producer/src/lib.rs:371
80 Feb 24 15:42:52.997 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("195695a2-3340-4078-83ac-2b01bd06e64f.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("3e2caf2f-ac6b-4ba7-9a22-2b19e9eb8ad9.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("9d517f13-d3fc-4874-8f9c-627cd5595d8e.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("9d517f13-d3fc-4874-8f9c-627cd5595d8e.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::a)) }], valid_until: Instant { tv_sec: 694, tv_nsec: 974048149 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
81 Feb 24 15:42:52.998 DEBG will register with Nexus at [fd00:1122:3344:101::b]:12221, component: producer-registration-task
82 Feb 24 15:42:52.999 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::b]:12221/metrics/producers, method: POST, component: producer-registration-task
832026-02-24T15:42:53.192ZWARNslog-rs: [tfportrear0_0] neighbor buskin@fe80::2cc5:40ff:fea8:3f02 missed solicit interval
84 Feb 24 15:43:08.003 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::b]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
85 Feb 24 15:43:08.004 WARN failed to register with Nexus, will retry, error: "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers)", delay: 50.460222841s, count: 2, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/194594d/oximeter/producer/src/lib.rs:371
86 Feb 24 15:43:08.564 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("195695a2-3340-4078-83ac-2b01bd06e64f.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("3e2caf2f-ac6b-4ba7-9a22-2b19e9eb8ad9.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("9d517f13-d3fc-4874-8f9c-627cd5595d8e.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("9d517f13-d3fc-4874-8f9c-627cd5595d8e.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::a)) }], valid_until: Instant { tv_sec: 710, tv_nsec: 537777399 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
87 Feb 24 15:43:08.565 DEBG will register with Nexus at [fd00:1122:3344:101::b]:12221, component: producer-registration-task
88 Feb 24 15:43:08.565 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::b]:12221/metrics/producers, method: POST, component: producer-registration-task
89 Feb 24 15:43:24.763 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::b]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
90 Feb 24 15:43:25.187 WARN failed to register with Nexus, will retry, error: "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers)", delay: 67.643972775s, count: 3, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/194594d/oximeter/producer/src/lib.rs:371
91 Feb 24 15:43:27.611 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("195695a2-3340-4078-83ac-2b01bd06e64f.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("3e2caf2f-ac6b-4ba7-9a22-2b19e9eb8ad9.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("9d517f13-d3fc-4874-8f9c-627cd5595d8e.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("9d517f13-d3fc-4874-8f9c-627cd5595d8e.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::a)) }], valid_until: Instant { tv_sec: 729, tv_nsec: 583524189 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
92 Feb 24 15:43:27.614 DEBG will register with Nexus at [fd00:1122:3344:101::b]:12221, component: producer-registration-task
93 Feb 24 15:43:27.623 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::b]:12221/metrics/producers, method: POST, component: producer-registration-task
94 Feb 24 15:43:42.624 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::b]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
95 Feb 24 15:43:42.626 WARN failed to register with Nexus, will retry, error: "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers)", delay: 85.082183024s, count: 4, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/194594d/oximeter/producer/src/lib.rs:371
96 Feb 24 15:43:45.027 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("195695a2-3340-4078-83ac-2b01bd06e64f.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("3e2caf2f-ac6b-4ba7-9a22-2b19e9eb8ad9.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("9d517f13-d3fc-4874-8f9c-627cd5595d8e.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("9d517f13-d3fc-4874-8f9c-627cd5595d8e.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::a)) }], valid_until: Instant { tv_sec: 747, tv_nsec: 914307 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
97 Feb 24 15:43:45.028 DEBG will register with Nexus at [fd00:1122:3344:101::b]:12221, component: producer-registration-task
98 Feb 24 15:43:45.028 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::b]:12221/metrics/producers, method: POST, component: producer-registration-task
99 Feb 24 15:44:00.029 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::b]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
100 Feb 24 15:44:00.029 WARN failed to register with Nexus, will retry, error: "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers)", delay: 102.485680009s, count: 5, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/194594d/oximeter/producer/src/lib.rs:371
101 Feb 24 15:44:05.502 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("195695a2-3340-4078-83ac-2b01bd06e64f.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("3e2caf2f-ac6b-4ba7-9a22-2b19e9eb8ad9.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("9d517f13-d3fc-4874-8f9c-627cd5595d8e.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("9d517f13-d3fc-4874-8f9c-627cd5595d8e.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::a)) }], valid_until: Instant { tv_sec: 767, tv_nsec: 474958313 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
102 Feb 24 15:44:05.502 DEBG will register with Nexus at [fd00:1122:3344:101::b]:12221, component: producer-registration-task
103 Feb 24 15:44:05.502 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::b]:12221/metrics/producers, method: POST, component: producer-registration-task
104 Feb 24 15:44:05.509 DEBG client response, result: Ok(Response { url: "http://[fd00:1122:3344:101::b]:12221/metrics/producers", status: 503, headers: {"content-type": "application/json", "x-request-id": "9375d3cd-564c-422c-8031-1305680e0565", "content-length": "133", "date": "Tue, 24 Feb 2026 15:44:05 GMT"} }), component: producer-registration-task
105 Feb 24 15:44:05.510 WARN failed to register with Nexus, will retry, error: "Error Response: status: 503 Service Unavailable; headers: {\"content-type\": \"application/json\", \"x-request-id\": \"9375d3cd-564c-422c-8031-1305680e0565\", \"content-length\": \"133\", \"date\": \"Tue, 24 Feb 2026 15:44:05 GMT\"}; value: Error { error_code: Some(\"ServiceNotAvailable\"), message: \"Service Unavailable\", request_id: \"9375d3cd-564c-422c-8031-1305680e0565\" }", delay: 107.965582616s, count: 6, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/194594d/oximeter/producer/src/lib.rs:371
106 Feb 24 15:44:21.194 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("195695a2-3340-4078-83ac-2b01bd06e64f.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("3e2caf2f-ac6b-4ba7-9a22-2b19e9eb8ad9.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("9d517f13-d3fc-4874-8f9c-627cd5595d8e.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("9d517f13-d3fc-4874-8f9c-627cd5595d8e.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::a)) }], valid_until: Instant { tv_sec: 783, tv_nsec: 168075925 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
107 Feb 24 15:44:21.195 DEBG will register with Nexus at [fd00:1122:3344:101::b]:12221, component: producer-registration-task
108 Feb 24 15:44:21.195 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::b]:12221/metrics/producers, method: POST, component: producer-registration-task
109 Feb 24 15:44:21.205 DEBG client response, result: Ok(Response { url: "http://[fd00:1122:3344:101::b]:12221/metrics/producers", status: 201, headers: {"content-type": "application/json", "x-request-id": "1291b43c-a0e8-43fa-ac6e-dd38a72d4c1a", "content-length": "41", "date": "Tue, 24 Feb 2026 15:44:21 GMT"} }), component: producer-registration-task
110 Feb 24 15:44:21.205 DEBG successfully registered with Nexus, pausing until time to renew lease, wait_period: 150s, lease_duration: 600s, component: producer-registration-task
111 Feb 24 15:44:26.033 INFO accepted connection, remote_addr: [fd00:1122:3344:101::d]:54922, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.7/src/server.rs:1025
112 Feb 24 15:44:26.044 INFO request completed, latency_us: 10226, response_code: 200, uri: /621b50bf-a2b7-420f-999d-5ac0f752c864, method: GET, req_id: 3cf9754d-43f6-4e02-bba9-4fe265612d66, remote_addr: [fd00:1122:3344:101::d]:54922, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.7/src/server.rs:867
113 Feb 24 15:44:26.858 INFO request completed, latency_us: 918, response_code: 200, uri: /621b50bf-a2b7-420f-999d-5ac0f752c864, method: GET, req_id: 9569042d-172c-4e37-bf27-453ded77b1ab, remote_addr: [fd00:1122:3344:101::d]:54922, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.7/src/server.rs:867
114 Feb 24 15:44:27.864 INFO request completed, latency_us: 6598, response_code: 200, uri: /621b50bf-a2b7-420f-999d-5ac0f752c864, method: GET, req_id: ea3bf7ee-edc5-4d68-8930-cfe4763b897f, remote_addr: [fd00:1122:3344:101::d]:54922, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.7/src/server.rs:867
115 Feb 24 15:44:28.858 INFO request completed, latency_us: 1085, response_code: 200, uri: /621b50bf-a2b7-420f-999d-5ac0f752c864, method: GET, req_id: b4cd8ad0-d676-4a0c-9f85-d5038a444299, remote_addr: [fd00:1122:3344:101::d]:54922, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.7/src/server.rs:867
116 Feb 24 15:44:29.856 INFO request completed, latency_us: 657, response_code: 200, uri: /621b50bf-a2b7-420f-999d-5ac0f752c864, method: GET, req_id: e57882c3-5329-45b8-9ee8-45da0130f0fd, remote_addr: [fd00:1122:3344:101::d]:54922, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.7/src/server.rs:867
117 Feb 24 15:44:30.856 INFO request completed, latency_us: 1086, response_code: 200, uri: /621b50bf-a2b7-420f-999d-5ac0f752c864, method: GET, req_id: 4d96dde9-7bd0-498e-9fd8-3fa0700e7821, remote_addr: [fd00:1122:3344:101::d]:54922, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.7/src/server.rs:867
118 Feb 24 15:44:31.856 INFO request completed, latency_us: 1149, response_code: 200, uri: /621b50bf-a2b7-420f-999d-5ac0f752c864, method: GET, req_id: 01e2ff54-f574-40f8-bd57-a854053a07ad, remote_addr: [fd00:1122:3344:101::d]:54922, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.7/src/server.rs:867
119 Feb 24 15:44:32.856 INFO request completed, latency_us: 655, response_code: 200, uri: /621b50bf-a2b7-420f-999d-5ac0f752c864, method: GET, req_id: ae305f3e-3b62-4f23-be79-c8aa7429d1ce, remote_addr: [fd00:1122:3344:101::d]:54922, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.7/src/server.rs:867
120 Feb 24 15:44:33.857 INFO request completed, latency_us: 997, response_code: 200, uri: /621b50bf-a2b7-420f-999d-5ac0f752c864, method: GET, req_id: b173c38a-c4ba-4a8f-8677-4c1d58448896, remote_addr: [fd00:1122:3344:101::d]:54922, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.7/src/server.rs:867
121 Feb 24 15:44:34.857 INFO request completed, latency_us: 655, response_code: 200, uri: /621b50bf-a2b7-420f-999d-5ac0f752c864, method: GET, req_id: e7c81b51-0793-46e2-9d23-a3567473f283, remote_addr: [fd00:1122:3344:101::d]:54922, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.7/src/server.rs:867
122 Feb 24 15:44:35.858 INFO request completed, latency_us: 1151, response_code: 200, uri: /621b50bf-a2b7-420f-999d-5ac0f752c864, method: GET, req_id: 082aa0c2-e38c-43b3-80fa-fe454255aaa8, remote_addr: [fd00:1122:3344:101::d]:54922, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.7/src/server.rs:867
123 Feb 24 15:44:36.857 INFO request completed, latency_us: 1216, response_code: 200, uri: /621b50bf-a2b7-420f-999d-5ac0f752c864, method: GET, req_id: 6dcdcfe9-adaf-4792-ab99-1ef77d512c35, remote_addr: [fd00:1122:3344:101::d]:54922, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.7/src/server.rs:867
124 Feb 24 15:44:37.857 INFO request completed, latency_us: 619, response_code: 200, uri: /621b50bf-a2b7-420f-999d-5ac0f752c864, method: GET, req_id: 2b8fb28a-f7be-40e9-be83-68cd740e9ce9, remote_addr: [fd00:1122:3344:101::d]:54922, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.7/src/server.rs:867
125 Feb 24 15:44:38.858 INFO request completed, latency_us: 1140, response_code: 200, uri: /621b50bf-a2b7-420f-999d-5ac0f752c864, method: GET, req_id: 5e2ff09b-acaa-4e1d-94c7-d6cc4d091e23, remote_addr: [fd00:1122:3344:101::d]:54922, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.7/src/server.rs:867
126 Feb 24 15:44:39.857 INFO request completed, latency_us: 647, response_code: 200, uri: /621b50bf-a2b7-420f-999d-5ac0f752c864, method: GET, req_id: f1177a8c-e3eb-4987-9297-8c2f0ff5149a, remote_addr: [fd00:1122:3344:101::d]:54922, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.7/src/server.rs:867
127 Feb 24 15:44:40.857 INFO request completed, latency_us: 846, response_code: 200, uri: /621b50bf-a2b7-420f-999d-5ac0f752c864, method: GET, req_id: 10b94783-5100-42eb-ac65-060bd230c284, remote_addr: [fd00:1122:3344:101::d]:54922, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.7/src/server.rs:867
128 Feb 24 15:44:41.856 INFO request completed, latency_us: 797, response_code: 200, uri: /621b50bf-a2b7-420f-999d-5ac0f752c864, method: GET, req_id: 909ca90f-0e7d-483a-a0ce-51c3b2753bf7, remote_addr: [fd00:1122:3344:101::d]:54922, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.7/src/server.rs:867
129 Feb 24 15:44:42.855 INFO request completed, latency_us: 604, response_code: 200, uri: /621b50bf-a2b7-420f-999d-5ac0f752c864, method: GET, req_id: c43da8dc-5f42-40e0-ab06-fa7f7aca70d3, remote_addr: [fd00:1122:3344:101::d]:54922, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.7/src/server.rs:867
130 Feb 24 15:44:43.857 INFO request completed, latency_us: 1110, response_code: 200, uri: /621b50bf-a2b7-420f-999d-5ac0f752c864, method: GET, req_id: f835e700-86e7-43cb-8aa3-cfc3f8f2b4be, remote_addr: [fd00:1122:3344:101::d]:54922, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.7/src/server.rs:867
131 Feb 24 15:44:44.856 INFO request completed, latency_us: 673, response_code: 200, uri: /621b50bf-a2b7-420f-999d-5ac0f752c864, method: GET, req_id: efec1294-082a-4a29-80d8-28f8fd4b98f2, remote_addr: [fd00:1122:3344:101::d]:54922, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.7/src/server.rs:867
132 Feb 24 15:44:45.856 INFO request completed, latency_us: 629, response_code: 200, uri: /621b50bf-a2b7-420f-999d-5ac0f752c864, method: GET, req_id: fc4251e1-ea84-4e46-b742-d233a4366293, remote_addr: [fd00:1122:3344:101::d]:54922, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.7/src/server.rs:867
133 Feb 24 15:44:46.857 INFO request completed, latency_us: 629, response_code: 200, uri: /621b50bf-a2b7-420f-999d-5ac0f752c864, method: GET, req_id: 69e02434-5903-4beb-9a56-165e265b3fb6, remote_addr: [fd00:1122:3344:101::d]:54922, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.7/src/server.rs:867
134 Feb 24 15:44:47.856 INFO request completed, latency_us: 665, response_code: 200, uri: /621b50bf-a2b7-420f-999d-5ac0f752c864, method: GET, req_id: 8d0c6539-293a-4ca1-9ca9-ee3fa9817b01, remote_addr: [fd00:1122:3344:101::d]:54922, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.7/src/server.rs:867
135 Feb 24 15:44:48.856 INFO request completed, latency_us: 654, response_code: 200, uri: /621b50bf-a2b7-420f-999d-5ac0f752c864, method: GET, req_id: 91bfd533-ebd2-472f-b086-338ff25e5537, remote_addr: [fd00:1122:3344:101::d]:54922, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.7/src/server.rs:867
136 Feb 24 15:44:49.857 INFO request completed, latency_us: 697, response_code: 200, uri: /621b50bf-a2b7-420f-999d-5ac0f752c864, method: GET, req_id: ce19b324-d487-4615-9583-905b7660c0ae, remote_addr: [fd00:1122:3344:101::d]:54922, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.7/src/server.rs:867
137 Feb 24 15:44:50.857 INFO request completed, latency_us: 629, response_code: 200, uri: /621b50bf-a2b7-420f-999d-5ac0f752c864, method: GET, req_id: 9dad8252-d9b0-4626-9004-efed5a2efa13, remote_addr: [fd00:1122:3344:101::d]:54922, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.7/src/server.rs:867
138 Feb 24 15:44:51.857 INFO request completed, latency_us: 790, response_code: 200, uri: /621b50bf-a2b7-420f-999d-5ac0f752c864, method: GET, req_id: d62cb353-8093-47f1-9325-5d6a59cd5770, remote_addr: [fd00:1122:3344:101::d]:54922, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.7/src/server.rs:867
139 Feb 24 15:44:52.858 INFO request completed, latency_us: 725, response_code: 200, uri: /621b50bf-a2b7-420f-999d-5ac0f752c864, method: GET, req_id: c468c7b2-4e02-464c-b0b3-1aa4bc627ecd, remote_addr: [fd00:1122:3344:101::d]:54922, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.7/src/server.rs:867
140 Feb 24 15:44:53.857 INFO request completed, latency_us: 633, response_code: 200, uri: /621b50bf-a2b7-420f-999d-5ac0f752c864, method: GET, req_id: 9ba79508-7d59-4389-a6c7-3e04021b0704, remote_addr: [fd00:1122:3344:101::d]:54922, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.7/src/server.rs:867
141 Feb 24 15:44:54.857 INFO request completed, latency_us: 686, response_code: 200, uri: /621b50bf-a2b7-420f-999d-5ac0f752c864, method: GET, req_id: b8185a89-f8e6-487d-a793-901c0a80700a, remote_addr: [fd00:1122:3344:101::d]:54922, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.7/src/server.rs:867
142 Feb 24 15:44:55.857 INFO request completed, latency_us: 648, response_code: 200, uri: /621b50bf-a2b7-420f-999d-5ac0f752c864, method: GET, req_id: da89e033-4e20-4502-96a9-9e02ae3d62c0, remote_addr: [fd00:1122:3344:101::d]:54922, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.7/src/server.rs:867
143 Feb 24 15:44:56.857 INFO request completed, latency_us: 1085, response_code: 200, uri: /621b50bf-a2b7-420f-999d-5ac0f752c864, method: GET, req_id: affa206c-de2e-450c-9847-e640d9edb524, remote_addr: [fd00:1122:3344:101::d]:54922, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.7/src/server.rs:867
144 Feb 24 15:44:57.857 INFO request completed, latency_us: 704, response_code: 200, uri: /621b50bf-a2b7-420f-999d-5ac0f752c864, method: GET, req_id: eeaf6b6a-ff2e-4735-b7fe-02aef75274ea, remote_addr: [fd00:1122:3344:101::d]:54922, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.7/src/server.rs:867
145 Feb 24 15:44:58.857 INFO request completed, latency_us: 646, response_code: 200, uri: /621b50bf-a2b7-420f-999d-5ac0f752c864, method: GET, req_id: cf51e3c1-6fae-41c1-8d59-ff47f9f780ee, remote_addr: [fd00:1122:3344:101::d]:54922, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.7/src/server.rs:867
146 Feb 24 15:44:59.857 INFO request completed, latency_us: 697, response_code: 200, uri: /621b50bf-a2b7-420f-999d-5ac0f752c864, method: GET, req_id: 7a989765-9cf2-4a0f-9082-2868f6fbc4ce, remote_addr: [fd00:1122:3344:101::d]:54922, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.7/src/server.rs:867
147 Feb 24 15:45:00.857 INFO request completed, latency_us: 802, response_code: 200, uri: /621b50bf-a2b7-420f-999d-5ac0f752c864, method: GET, req_id: ed8d3068-8260-4783-905e-6a7d5b18903b, remote_addr: [fd00:1122:3344:101::d]:54922, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.7/src/server.rs:867