Max level shown:
1 [ Jun 19 11:42:23 Disabled. ]
2 [ Jun 19 11:42:23 Rereading configuration. ]
3 [ Jun 19 11:42:24 Enabled. ]
4 [ Jun 19 11:42:24 Executing start method ("ctrun -l child -o noorphan,regent /opt/oxide/mgd/pkg/mgd_method_script.sh &"). ]
5 [ Jun 19 11:42:24 Method "start" exited with status 0. ]
6 [ Jun 19 11:42:24 Rereading configuration. ]
7 [ Jun 19 11:42:24 Executing refresh method ("/opt/oxide/mgd/pkg/refresh_ctrun_child.sh -c 213"). ]
8 + CONTRACT=''
9 + getopts c: o
10 + CONTRACT=213
11 + getopts c: o
12 + [[ -n 213 ]]
13 + pgrep -c 213
14 + PARENT_PID=2276
15 + [[ -n 2276 ]]
16 + pgrep -P 2276
17 + CHILD_PID=2277
18 + [[ -n 2277 ]]
19 + kill -USR1 2277
20 [ Jun 19 11:42:24 Method "refresh" exited with status 0. ]
21 [ Jun 19 11:42:24 Stopping because all processes in service exited. ]
22 [ Jun 19 11:42:24 Executing stop method (:kill). ]
23 [ Jun 19 11:42:24 Executing start method ("ctrun -l child -o noorphan,regent /opt/oxide/mgd/pkg/mgd_method_script.sh &"). ]
24 [ Jun 19 11:42:24 Method "start" exited with status 0. ]
252025-06-19T11:42:24.240ZINFOslog-rs: signal handler waiting for context
262025-06-19T11:42:24.261ZINFOslog-rs: signal handler waiting got context
272025-06-19T11:42:24.261ZINFOslog-rs: bgp routers: {}
282025-06-19T11:42:24.261ZINFOslog-rs: bfd peers: []
292025-06-19T11:42:24.261ZERROslog-rs: failed to start stats server: underlay address not found
302025-06-19T11:42:24.263ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = GET path = /bfd/peers unit = api-server versions = all
312025-06-19T11:42:24.263ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = PUT path = /bfd/peers unit = api-server versions = all
322025-06-19T11:42:24.263ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = DELETE path = /bfd/peers/{addr} unit = api-server versions = all
332025-06-19T11:42:24.263ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = POST path = /bgp/clear/neighbor unit = api-server versions = all
342025-06-19T11:42:24.263ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = DELETE path = /bgp/config/checker unit = api-server versions = all
352025-06-19T11:42:24.263ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = GET path = /bgp/config/checker unit = api-server versions = all
362025-06-19T11:42:24.263ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = POST path = /bgp/config/checker unit = api-server versions = all
372025-06-19T11:42:24.263ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = PUT path = /bgp/config/checker unit = api-server versions = all
382025-06-19T11:42:24.263ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = DELETE path = /bgp/config/neighbor unit = api-server versions = all
392025-06-19T11:42:24.263ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = GET path = /bgp/config/neighbor unit = api-server versions = all
402025-06-19T11:42:24.263ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = POST path = /bgp/config/neighbor unit = api-server versions = all
412025-06-19T11:42:24.263ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = PUT path = /bgp/config/neighbor unit = api-server versions = all
422025-06-19T11:42:24.263ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = GET path = /bgp/config/neighbors unit = api-server versions = all
432025-06-19T11:42:24.263ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = DELETE path = /bgp/config/origin4 unit = api-server versions = all
442025-06-19T11:42:24.263ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = GET path = /bgp/config/origin4 unit = api-server versions = all
452025-06-19T11:42:24.263ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = POST path = /bgp/config/origin4 unit = api-server versions = all
462025-06-19T11:42:24.263ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = PUT path = /bgp/config/origin4 unit = api-server versions = all
472025-06-19T11:42:24.263ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = DELETE path = /bgp/config/router unit = api-server versions = all
482025-06-19T11:42:24.263ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = GET path = /bgp/config/router unit = api-server versions = all
492025-06-19T11:42:24.263ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = POST path = /bgp/config/router unit = api-server versions = all
502025-06-19T11:42:24.263ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = PUT path = /bgp/config/router unit = api-server versions = all
512025-06-19T11:42:24.263ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = GET path = /bgp/config/routers unit = api-server versions = all
522025-06-19T11:42:24.263ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = DELETE path = /bgp/config/shaper unit = api-server versions = all
532025-06-19T11:42:24.263ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = GET path = /bgp/config/shaper unit = api-server versions = all
542025-06-19T11:42:24.263ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = POST path = /bgp/config/shaper unit = api-server versions = all
552025-06-19T11:42:24.263ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = PUT path = /bgp/config/shaper unit = api-server versions = all
562025-06-19T11:42:24.263ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = GET path = /bgp/message-history unit = api-server versions = all
572025-06-19T11:42:24.263ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = POST path = /bgp/omicron/apply unit = api-server versions = all
582025-06-19T11:42:24.263ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = GET path = /bgp/status/exported unit = api-server versions = all
592025-06-19T11:42:24.263ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = GET path = /bgp/status/imported unit = api-server versions = all
602025-06-19T11:42:24.263ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = GET path = /bgp/status/neighbors unit = api-server versions = all
612025-06-19T11:42:24.263ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = GET path = /bgp/status/selected unit = api-server versions = all
622025-06-19T11:42:24.263ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = DELETE path = /static/route4 unit = api-server versions = all
632025-06-19T11:42:24.263ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = GET path = /static/route4 unit = api-server versions = all
642025-06-19T11:42:24.263ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = PUT path = /static/route4 unit = api-server versions = all
652025-06-19T11:42:24.263ZINFOslog-rs: admin: listening on [::]:4676
662025-06-19T11:42:24.263ZINFOslog-rs: listening local_addr = [::]:4676 unit = api-server
672025-06-19T11:42:24.264ZDEBGslog-rs: successfully registered DTrace USDT probes local_addr = [::]:4676 unit = api-server
682025-06-19T11:42:24.264ZWARNslog-rs: failed to ensure TEP address fd9d:624c:d012:7319::1 on ASIC: Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6)
692025-06-19T11:42:25.266ZWARNslog-rs: failed to ensure TEP address fd9d:624c:d012:7319::1 on ASIC: Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6)
702025-06-19T11:42:26.267ZWARNslog-rs: failed to ensure TEP address fd9d:624c:d012:7319::1 on ASIC: Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6)
712025-06-19T11:42:27.268ZWARNslog-rs: failed to ensure TEP address fd9d:624c:d012:7319::1 on ASIC: Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6)
722025-06-19T11:42:28.268ZWARNslog-rs: failed to ensure TEP address fd9d:624c:d012:7319::1 on ASIC: Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6)
732025-06-19T11:42:29.269ZWARNslog-rs: failed to ensure TEP address fd9d:624c:d012:7319::1 on ASIC: Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6)
742025-06-19T11:42:30.273ZWARNslog-rs: failed to ensure TEP address fd9d:624c:d012:7319::1 on ASIC: Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6)
752025-06-19T11:42:31.275ZWARNslog-rs: failed to ensure TEP address fd9d:624c:d012:7319::1 on ASIC: Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6)
762025-06-19T11:42:32.276ZWARNslog-rs: failed to ensure TEP address fd9d:624c:d012:7319::1 on ASIC: Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6)
772025-06-19T11:42:33.276ZWARNslog-rs: failed to ensure TEP address fd9d:624c:d012:7319::1 on ASIC: Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6)
782025-06-19T11:42:34.277ZWARNslog-rs: failed to ensure TEP address fd9d:624c:d012:7319::1 on ASIC: Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6)
79 [ Jun 19 11:42:35 Rereading configuration. ]
80 [ Jun 19 11:42:35 Executing refresh method ("/opt/oxide/mgd/pkg/refresh_ctrun_child.sh -c 218"). ]
81 + CONTRACT=''
82 + getopts c: o
83 + CONTRACT=218
84 + getopts c: o
85 + [[ -n 218 ]]
86 + pgrep -c 218
87 + PARENT_PID=2303
88 + [[ -n 2303 ]]
89 + pgrep -P 2303
90 + CHILD_PID=2305
91 + [[ -n 2305 ]]
92 + kill -USR1 2305
932025-06-19T11:42:35.515ZINFOslog-rs: handling smf refresh
942025-06-19T11:42:35.516ZINFOslog-rs: starting stats server on smf refresh
95 [ Jun 19 11:42:35 Method "refresh" exited with status 0. ]
96 Jun 19 11:42:35.517 DEBG registered DTrace probes
97 Jun 19 11:42:35.518 DEBG registered endpoint, versions: all, path: /{producer_id}, method: GET, local_addr: [fd00:1122:3344:101::2]:4677, component: dropshot
98 Jun 19 11:42:35.518 INFO listening, local_addr: [fd00:1122:3344:101::2]:4677, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:328
99 Jun 19 11:42:35.518 DEBG successfully registered DTrace USDT probes, local_addr: [fd00:1122:3344:101::2]:4677, component: dropshot
100 Jun 19 11:42:35.518 DEBG Nexus IP not provided, will use DNS to resolve it
101 Jun 19 11:42:35.518 INFO new DNS resolver, addresses: [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53], component: internal-dns-resolver, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/1d3958c/internal-dns/resolver/src/resolver.rs:111
102 Jun 19 11:42:35.518 DEBG starting producer registration task
103 Jun 19 11:42:35.518 INFO starting oximeter metric producer server, interval: 1s, address: [fd00:1122:3344:101::2]:4677, producer_id: c852a13f-da7c-4959-a50b-633490cc25fd, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/1d3958c/oximeter/producer/src/lib.rs:279
104 Jun 19 11:42:35.518 DEBG registering as oximeter producer with Nexus, registration_count: 0, component: producer-registration-task
1052025-06-19T11:42:35.818ZINFOslog-rs: accepted connection local_addr = [::]:4676 remote_addr = [fd00:1122:3344:101::1]:61623 unit = api-server
1062025-06-19T11:42:35.819ZINFOslog-rs: request completed latency_us = 377 local_addr = [::]:4676 method = PUT remote_addr = [fd00:1122:3344:101::1]:61623 req_id = 7e6c010e-d059-4f07-ad75-603ac67dda1e response_code = 204 unit = api-server uri = /static/route4
1072025-06-19T11:42:35.819ZERROslog-rs: handling change failed: libnet error route io error No such process (os error 3)
1082025-06-19T11:42:35.819ZINFOslog-rs: restarting sync loop
109 Jun 19 11:42:40.520 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("08971322-7141-4ae8-bc67-e581f49794b5.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("0e1f5e7a-d83f-4df4-9fba-054336310284.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("18b41a30-b847-4b47-aadd-6df881cca8ad.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("18b41a30-b847-4b47-aadd-6df881cca8ad.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::a))) }], valid_until: Instant { tv_sec: 701, tv_nsec: 891404759 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
110 Jun 19 11:42:40.521 DEBG will register with Nexus at [fd00:1122:3344:101::c]:12221, component: producer-registration-task
111 Jun 19 11:42:40.521 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::c]:12221/metrics/producers, method: POST, component: producer-registration-task
112 Jun 19 11:42:55.917 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::c]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
113 Jun 19 11:42:55.917 WARN failed to register with Nexus, will retry, error: "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers)", delay: 20.003125063s, count: 0, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/1d3958c/oximeter/producer/src/lib.rs:371
114 Jun 19 11:42:56.065 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("08971322-7141-4ae8-bc67-e581f49794b5.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("0e1f5e7a-d83f-4df4-9fba-054336310284.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("18b41a30-b847-4b47-aadd-6df881cca8ad.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("18b41a30-b847-4b47-aadd-6df881cca8ad.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::a))) }], valid_until: Instant { tv_sec: 717, tv_nsec: 40707763 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
115 Jun 19 11:42:56.065 DEBG will register with Nexus at [fd00:1122:3344:101::c]:12221, component: producer-registration-task
116 Jun 19 11:42:56.065 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::c]:12221/metrics/producers, method: POST, component: producer-registration-task
117 Jun 19 11:43:11.045 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::c]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
118 Jun 19 11:43:11.045 WARN failed to register with Nexus, will retry, error: "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers)", delay: 35.152980243s, count: 1, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/1d3958c/oximeter/producer/src/lib.rs:371
119 Jun 19 11:43:11.724 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("08971322-7141-4ae8-bc67-e581f49794b5.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("0e1f5e7a-d83f-4df4-9fba-054336310284.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("18b41a30-b847-4b47-aadd-6df881cca8ad.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("18b41a30-b847-4b47-aadd-6df881cca8ad.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::a))) }], valid_until: Instant { tv_sec: 732, tv_nsec: 722029844 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
120 Jun 19 11:43:11.724 DEBG will register with Nexus at [fd00:1122:3344:101::c]:12221, component: producer-registration-task
121 Jun 19 11:43:11.725 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::c]:12221/metrics/producers, method: POST, component: producer-registration-task
122 Jun 19 11:43:31.140 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::c]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
123 Jun 19 11:43:31.143 WARN failed to register with Nexus, will retry, error: "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers)", delay: 55.25002715s, count: 2, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/1d3958c/oximeter/producer/src/lib.rs:371
124 Jun 19 11:43:32.652 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("08971322-7141-4ae8-bc67-e581f49794b5.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("0e1f5e7a-d83f-4df4-9fba-054336310284.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("18b41a30-b847-4b47-aadd-6df881cca8ad.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("18b41a30-b847-4b47-aadd-6df881cca8ad.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::a))) }], valid_until: Instant { tv_sec: 753, tv_nsec: 651224478 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
125 Jun 19 11:43:32.655 DEBG will register with Nexus at [fd00:1122:3344:101::c]:12221, component: producer-registration-task
126 Jun 19 11:43:32.667 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::c]:12221/metrics/producers, method: POST, component: producer-registration-task
127 Jun 19 11:43:47.744 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::c]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
128 Jun 19 11:43:47.793 WARN failed to register with Nexus, will retry, error: "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers)", delay: 71.902152945s, count: 3, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/1d3958c/oximeter/producer/src/lib.rs:371
129 Jun 19 11:43:49.360 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("08971322-7141-4ae8-bc67-e581f49794b5.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("0e1f5e7a-d83f-4df4-9fba-054336310284.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("18b41a30-b847-4b47-aadd-6df881cca8ad.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("18b41a30-b847-4b47-aadd-6df881cca8ad.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::a))) }], valid_until: Instant { tv_sec: 770, tv_nsec: 357267056 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
130 Jun 19 11:43:49.363 DEBG will register with Nexus at [fd00:1122:3344:101::c]:12221, component: producer-registration-task
131 Jun 19 11:43:49.388 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::c]:12221/metrics/producers, method: POST, component: producer-registration-task
132 Jun 19 11:44:04.395 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::c]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
133 Jun 19 11:44:04.400 WARN failed to register with Nexus, will retry, error: "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers)", delay: 88.508916778s, count: 4, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/1d3958c/oximeter/producer/src/lib.rs:371
134 Jun 19 11:44:07.110 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("08971322-7141-4ae8-bc67-e581f49794b5.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("0e1f5e7a-d83f-4df4-9fba-054336310284.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("18b41a30-b847-4b47-aadd-6df881cca8ad.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("18b41a30-b847-4b47-aadd-6df881cca8ad.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::a))) }], valid_until: Instant { tv_sec: 788, tv_nsec: 108734565 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
135 Jun 19 11:44:07.110 DEBG will register with Nexus at [fd00:1122:3344:101::c]:12221, component: producer-registration-task
136 Jun 19 11:44:07.110 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::c]:12221/metrics/producers, method: POST, component: producer-registration-task
137 Jun 19 11:44:15.409 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::c]:12221/metrics/producers", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) }), component: producer-registration-task
138 Jun 19 11:44:15.415 WARN failed to register with Nexus, will retry, error: "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers)", delay: 99.518596267s, count: 5, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/1d3958c/oximeter/producer/src/lib.rs:371
1392025-06-19T11:44:23.910ZINFOslog-rs: accepted connection local_addr = [::]:4676 remote_addr = [fd00:1122:3344:101::a]:47624 unit = api-server
1402025-06-19T11:44:23.923ZINFOslog-rs: request completed latency_us = 7997 local_addr = [::]:4676 method = GET remote_addr = [fd00:1122:3344:101::a]:47624 req_id = 82f6ef33-85a4-4b40-8729-85cffec01265 response_code = 200 unit = api-server uri = /bfd/peers
1412025-06-19T11:44:24.109ZINFOslog-rs: accepted connection local_addr = [::]:4676 remote_addr = [fd00:1122:3344:101::c]:57476 unit = api-server
1422025-06-19T11:44:24.109ZINFOslog-rs: request completed latency_us = 46 local_addr = [::]:4676 method = GET remote_addr = [fd00:1122:3344:101::c]:57476 req_id = 8ef9e6c9-a96c-4635-91b5-d95e7150d519 response_code = 200 unit = api-server uri = /bfd/peers
1432025-06-19T11:44:24.278ZINFOslog-rs: accepted connection local_addr = [::]:4676 remote_addr = [fd00:1122:3344:101::b]:55509 unit = api-server
1442025-06-19T11:44:24.278ZINFOslog-rs: request completed latency_us = 30 local_addr = [::]:4676 method = GET remote_addr = [fd00:1122:3344:101::b]:55509 req_id = 371f28d0-41b2-42b5-9a00-5aaab0197676 response_code = 200 unit = api-server uri = /bfd/peers
145 Jun 19 11:44:27.167 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("08971322-7141-4ae8-bc67-e581f49794b5.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("0e1f5e7a-d83f-4df4-9fba-054336310284.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("18b41a30-b847-4b47-aadd-6df881cca8ad.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("18b41a30-b847-4b47-aadd-6df881cca8ad.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::a))) }], valid_until: Instant { tv_sec: 808, tv_nsec: 166166721 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
146 Jun 19 11:44:27.168 DEBG will register with Nexus at [fd00:1122:3344:101::c]:12221, component: producer-registration-task
147 Jun 19 11:44:27.168 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::c]:12221/metrics/producers, method: POST, component: producer-registration-task
148 Jun 19 11:44:27.180 DEBG client response, result: Ok(Response { url: "http://[fd00:1122:3344:101::c]:12221/metrics/producers", status: 201, headers: {"content-type": "application/json", "x-request-id": "8c5a250d-091b-49ec-917b-11d61c6d90eb", "content-length": "41", "date": "Thu, 19 Jun 2025 11:44:27 GMT"} }), component: producer-registration-task
149 Jun 19 11:44:27.181 DEBG successfully registered with Nexus, pausing until time to renew lease, wait_period: 150s, lease_duration: 600s, component: producer-registration-task
150 Jun 19 11:44:38.882 INFO accepted connection, remote_addr: [fd00:1122:3344:101::d]:56981, local_addr: [fd00:1122:3344:101::2]:4677, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:1025
151 Jun 19 11:44:38.893 INFO request completed, latency_us: 10148, response_code: 200, uri: /c852a13f-da7c-4959-a50b-633490cc25fd, method: GET, req_id: 1b7e6378-9e15-41ca-b58f-9bd7b53d93f4, remote_addr: [fd00:1122:3344:101::d]:56981, local_addr: [fd00:1122:3344:101::2]:4677, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867
152 Jun 19 11:44:39.882 INFO request completed, latency_us: 331, response_code: 200, uri: /c852a13f-da7c-4959-a50b-633490cc25fd, method: GET, req_id: 9b20c00e-0648-4bb0-9118-73ea567315c0, remote_addr: [fd00:1122:3344:101::d]:56981, local_addr: [fd00:1122:3344:101::2]:4677, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867
153 Jun 19 11:44:40.882 INFO request completed, latency_us: 622, response_code: 200, uri: /c852a13f-da7c-4959-a50b-633490cc25fd, method: GET, req_id: c65b9683-3bc7-4dc4-bb08-bcf33fb828dc, remote_addr: [fd00:1122:3344:101::d]:56981, local_addr: [fd00:1122:3344:101::2]:4677, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867
154 Jun 19 11:44:41.882 INFO request completed, latency_us: 329, response_code: 200, uri: /c852a13f-da7c-4959-a50b-633490cc25fd, method: GET, req_id: 9e2e148a-6a1f-421d-bc52-eea38ec58b55, remote_addr: [fd00:1122:3344:101::d]:56981, local_addr: [fd00:1122:3344:101::2]:4677, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867
155 Jun 19 11:44:42.882 INFO request completed, latency_us: 331, response_code: 200, uri: /c852a13f-da7c-4959-a50b-633490cc25fd, method: GET, req_id: 42f5880d-b28e-4c17-86a0-03e148dda6c6, remote_addr: [fd00:1122:3344:101::d]:56981, local_addr: [fd00:1122:3344:101::2]:4677, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867
156 Jun 19 11:44:43.882 INFO request completed, latency_us: 364, response_code: 200, uri: /c852a13f-da7c-4959-a50b-633490cc25fd, method: GET, req_id: 72851bc2-40d7-4d0e-b2a8-92f7c25f489a, remote_addr: [fd00:1122:3344:101::d]:56981, local_addr: [fd00:1122:3344:101::2]:4677, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867
157 Jun 19 11:44:44.882 INFO request completed, latency_us: 329, response_code: 200, uri: /c852a13f-da7c-4959-a50b-633490cc25fd, method: GET, req_id: 4dc9b5fe-a40b-4a15-91c9-f0cf7eb207d4, remote_addr: [fd00:1122:3344:101::d]:56981, local_addr: [fd00:1122:3344:101::2]:4677, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867
158 Jun 19 11:44:45.882 INFO request completed, latency_us: 332, response_code: 200, uri: /c852a13f-da7c-4959-a50b-633490cc25fd, method: GET, req_id: 4679790b-1813-4d06-b517-e5a44ebba49c, remote_addr: [fd00:1122:3344:101::d]:56981, local_addr: [fd00:1122:3344:101::2]:4677, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867
159 Jun 19 11:44:46.882 INFO request completed, latency_us: 347, response_code: 200, uri: /c852a13f-da7c-4959-a50b-633490cc25fd, method: GET, req_id: 4c3cc2b0-4ced-4f34-bebb-43ca722222d0, remote_addr: [fd00:1122:3344:101::d]:56981, local_addr: [fd00:1122:3344:101::2]:4677, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867
160 Jun 19 11:44:47.882 INFO request completed, latency_us: 447, response_code: 200, uri: /c852a13f-da7c-4959-a50b-633490cc25fd, method: GET, req_id: a5643fb9-865e-4d13-8c02-516d070d8f84, remote_addr: [fd00:1122:3344:101::d]:56981, local_addr: [fd00:1122:3344:101::2]:4677, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867
161 Jun 19 11:44:48.882 INFO request completed, latency_us: 304, response_code: 200, uri: /c852a13f-da7c-4959-a50b-633490cc25fd, method: GET, req_id: 74da6c4b-257f-4eab-acef-382e9896f7da, remote_addr: [fd00:1122:3344:101::d]:56981, local_addr: [fd00:1122:3344:101::2]:4677, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867
162 Jun 19 11:44:49.882 INFO request completed, latency_us: 547, response_code: 200, uri: /c852a13f-da7c-4959-a50b-633490cc25fd, method: GET, req_id: 1061c40a-d874-4d80-923a-bd3bad8650b8, remote_addr: [fd00:1122:3344:101::d]:56981, local_addr: [fd00:1122:3344:101::2]:4677, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867
163 Jun 19 11:44:50.882 INFO request completed, latency_us: 575, response_code: 200, uri: /c852a13f-da7c-4959-a50b-633490cc25fd, method: GET, req_id: 5fd2848d-5472-4f46-b83f-3a3df0c22d96, remote_addr: [fd00:1122:3344:101::d]:56981, local_addr: [fd00:1122:3344:101::2]:4677, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867
164 Jun 19 11:44:51.882 INFO request completed, latency_us: 497, response_code: 200, uri: /c852a13f-da7c-4959-a50b-633490cc25fd, method: GET, req_id: 756b69f8-ce77-4824-b8c0-fd9d76056141, remote_addr: [fd00:1122:3344:101::d]:56981, local_addr: [fd00:1122:3344:101::2]:4677, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867
165 Jun 19 11:44:52.882 INFO request completed, latency_us: 447, response_code: 200, uri: /c852a13f-da7c-4959-a50b-633490cc25fd, method: GET, req_id: b0fdaf7e-143a-4260-8e7a-b8777b40dfd0, remote_addr: [fd00:1122:3344:101::d]:56981, local_addr: [fd00:1122:3344:101::2]:4677, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867
1662025-06-19T11:44:53.867ZINFOslog-rs: accepted connection local_addr = [::]:4676 remote_addr = [fd00:1122:3344:101::a]:54373 unit = api-server
1672025-06-19T11:44:53.867ZINFOslog-rs: request completed latency_us = 28 local_addr = [::]:4676 method = GET remote_addr = [fd00:1122:3344:101::a]:54373 req_id = 64878e63-4480-4eec-a9ba-5de5bf19d8cd response_code = 200 unit = api-server uri = /bfd/peers
168 Jun 19 11:44:53.882 INFO request completed, latency_us: 382, response_code: 200, uri: /c852a13f-da7c-4959-a50b-633490cc25fd, method: GET, req_id: c38738fd-6460-4f7f-8056-02023a31e48c, remote_addr: [fd00:1122:3344:101::d]:56981, local_addr: [fd00:1122:3344:101::2]:4677, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867
1692025-06-19T11:44:53.907ZINFOslog-rs: accepted connection local_addr = [::]:4676 remote_addr = [fd00:1122:3344:101::c]:65029 unit = api-server
1702025-06-19T11:44:53.907ZINFOslog-rs: request completed latency_us = 22 local_addr = [::]:4676 method = GET remote_addr = [fd00:1122:3344:101::c]:65029 req_id = 0294b346-b010-45ca-9cee-499b29e1a0aa response_code = 200 unit = api-server uri = /bfd/peers
1712025-06-19T11:44:54.082ZINFOslog-rs: accepted connection local_addr = [::]:4676 remote_addr = [fd00:1122:3344:101::b]:50157 unit = api-server
1722025-06-19T11:44:54.082ZINFOslog-rs: request completed latency_us = 24 local_addr = [::]:4676 method = GET remote_addr = [fd00:1122:3344:101::b]:50157 req_id = 0bd9f39b-b677-4f85-87eb-c6d4636e8066 response_code = 200 unit = api-server uri = /bfd/peers
173 Jun 19 11:44:54.882 INFO request completed, latency_us: 323, response_code: 200, uri: /c852a13f-da7c-4959-a50b-633490cc25fd, method: GET, req_id: 4a51deca-f54f-409d-8f87-5f919a13bbe0, remote_addr: [fd00:1122:3344:101::d]:56981, local_addr: [fd00:1122:3344:101::2]:4677, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867
174 Jun 19 11:44:55.882 INFO request completed, latency_us: 341, response_code: 200, uri: /c852a13f-da7c-4959-a50b-633490cc25fd, method: GET, req_id: 0cfd5432-bda1-413e-a627-a3f80c026c99, remote_addr: [fd00:1122:3344:101::d]:56981, local_addr: [fd00:1122:3344:101::2]:4677, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867
175 Jun 19 11:44:56.882 INFO request completed, latency_us: 533, response_code: 200, uri: /c852a13f-da7c-4959-a50b-633490cc25fd, method: GET, req_id: 2ca4925d-2c34-4b8b-b9c6-97683a7ebc2c, remote_addr: [fd00:1122:3344:101::d]:56981, local_addr: [fd00:1122:3344:101::2]:4677, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867
176 Jun 19 11:44:57.882 INFO request completed, latency_us: 357, response_code: 200, uri: /c852a13f-da7c-4959-a50b-633490cc25fd, method: GET, req_id: a03674d0-b897-4d5b-a7f8-99c6c2768619, remote_addr: [fd00:1122:3344:101::d]:56981, local_addr: [fd00:1122:3344:101::2]:4677, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867
177 Jun 19 11:44:58.882 INFO request completed, latency_us: 325, response_code: 200, uri: /c852a13f-da7c-4959-a50b-633490cc25fd, method: GET, req_id: 91a67f69-ba96-40e8-98a7-1b6ddf7df6c7, remote_addr: [fd00:1122:3344:101::d]:56981, local_addr: [fd00:1122:3344:101::2]:4677, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867
178 Jun 19 11:44:59.882 INFO request completed, latency_us: 378, response_code: 200, uri: /c852a13f-da7c-4959-a50b-633490cc25fd, method: GET, req_id: 8cbd7ee9-1fe5-40a5-8a0f-97938343b7b1, remote_addr: [fd00:1122:3344:101::d]:56981, local_addr: [fd00:1122:3344:101::2]:4677, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867
179 Jun 19 11:45:00.882 INFO request completed, latency_us: 436, response_code: 200, uri: /c852a13f-da7c-4959-a50b-633490cc25fd, method: GET, req_id: d255cf45-3b79-455e-9b57-995ad4648839, remote_addr: [fd00:1122:3344:101::d]:56981, local_addr: [fd00:1122:3344:101::2]:4677, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867