Max level shown:
1 [ Aug 13 21:26:04 Disabled. ]
2 [ Aug 13 21:26:04 Rereading configuration. ]
3 [ Aug 13 21:26:04 Enabled. ]
4 [ Aug 13 21:26:04 Rereading configuration. ]
5 [ Aug 13 21:26:04 Executing start method ("ctrun -l child -o noorphan,regent /opt/oxide/mg-ddm/pkg/ddm_method_script.sh &"). ]
6 [ Aug 13 21:26:04 Method "start" exited with status 0. ]
72026-08-13T21:26:04.927ZINFOslog-rs: signal handler waiting for context
82026-08-13T21:26:04.956ZERROslog-rs: failed to start stats server: underlay address not found
92026-08-13T21:26:04.956ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
102026-08-13T21:26:04.956ZINFOslog-rs: signal handler waiting got context
112026-08-13T21:26:04.957ZINFOslog-rs: admin: listening on [::1]:8000
122026-08-13T21:26:05.956ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
132026-08-13T21:26:06.956ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
142026-08-13T21:26:07.956ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
152026-08-13T21:26:08.956ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
162026-08-13T21:26:09.956ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
172026-08-13T21:26:10.956ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
182026-08-13T21:26:11.956ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
192026-08-13T21:26:12.956ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
202026-08-13T21:26:13.957ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
212026-08-13T21:26:14.957ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
222026-08-13T21:26:15.957ZINFOslog-rs: [tfportrear0_0] sm initialized with addr fe80::7c06:5aff:fe16:2975 on if tfportrear0_0 index 3
232026-08-13T21:26:15.957ZDEBGslog-rs: [tfportrear0_0] starting discovery handler
24 [ Aug 13 21:26:16 Rereading configuration. ]
25 [ Aug 13 21:26:16 Executing refresh method ("/opt/oxide/mg-ddm/pkg/refresh_ctrun_child.sh -c 213"). ]
26 + CONTRACT=''
27 + getopts c: o
28 + CONTRACT=213
29 + getopts c: o
30 + [[ -n 213 ]]
31 + pgrep -c 213
32 + PARENT_PID=2293
33 + [[ -n 2293 ]]
34 + pgrep -P 2293
35 + CHILD_PID=2297
36 + [[ -n 2297 ]]
37 + kill -USR1 2297
382026-08-13T21:26:16.241ZINFOslog-rs: handling smf refresh
39 [ Aug 13 21:26:16 Method "refresh" exited with status 0. ]
402026-08-13T21:26:16.243ZINFOslog-rs: starting stats server on smf refresh
41 Aug 13 21:26:16.244 DEBG registered DTrace probes
42 Aug 13 21:26:16.244 DEBG registered endpoint, versions: all, path: /{producer_id}, method: GET, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot
43 Aug 13 21:26:16.245 INFO listening, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:335
44 Aug 13 21:26:16.245 DEBG successfully registered DTrace USDT probes, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot
45 Aug 13 21:26:16.245 DEBG Nexus IP not provided, will use DNS to resolve it
46 Aug 13 21:26:16.245 INFO new DNS resolver, addresses: [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53], component: internal-dns-resolver, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/internal-dns/resolver/src/resolver.rs:144
47 Aug 13 21:26:16.245 DEBG starting producer registration task
48 Aug 13 21:26:16.245 INFO starting oximeter metric producer server, interval: 1s, address: [fd00:1122:3344:101::2]:8001, producer_id: 9bfd2a13-f0c0-49cd-a1b0-bb79448e6869, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/oximeter/producer/src/lib.rs:280
49 Aug 13 21:26:16.245 DEBG registering as oximeter producer with Nexus, registration_count: 0, component: producer-registration-task
50 Aug 13 21:26:16.246 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("87bd9e59-ba1d-4a64-a79d-60c17e91b150.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("95247c42-8a9b-428e-be43-565cc0cf9dad.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("cfc80aeb-ea69-4871-90d1-0edc16faef45.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("cfc80aeb-ea69-4871-90d1-0edc16faef45.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::a)) }], valid_until: Instant { tv_sec: 1037, tv_nsec: 71769750 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
51 Aug 13 21:26:16.247 DEBG will register with Nexus at [fd00:1122:3344:101::b]:12221, component: producer-registration-task
52 Aug 13 21:26:16.249 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::b]:12221/metrics/producers, method: POST, component: producer-registration-task
532026-08-13T21:26:25.958ZINFOslog-rs: [tfportrear0_0] nbr is fe80::74bc:2cff:fe86:fb5b@buskin server
542026-08-13T21:26:25.958ZDEBGslog-rs: [tfportrear0_0] transition solicit -> exchange
552026-08-13T21:26:25.958ZINFOslog-rs: [tfportrear0_0] exchange: listening on [fe80::7c06:5aff:fe16:2975]:56797
562026-08-13T21:26:25.958ZINFOslog-rs: [tfportrear0_0] waiting for exchange server to start
572026-08-13T21:26:28.211ZWARNslog-rs: [tfportrear0_0] exchange pull: timeout error: deadline has elapsed
582026-08-13T21:26:28.212ZINFOslog-rs: [tfportrear0_0] sending 5 routes to dendrite
592026-08-13T21:26:28.212ZDEBGslog-rs: [tfportrear0_0] sending to dpd host=localhost port=12224
602026-08-13T21:26:28.213ZINFOslog-rs: [tfportrear0_0] adding route fdb0:18c0:4d0d:9fb2:: -> fe80::74bc:2cff:fe86:fb5b on port Rear(Rear("rear0"))/0
612026-08-13T21:26:28.214ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:101:: -> fe80::74bc:2cff:fe86:fb5b on port Rear(Rear("rear0"))/0
622026-08-13T21:26:28.215ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:2:: -> fe80::74bc:2cff:fe86:fb5b on port Rear(Rear("rear0"))/0
632026-08-13T21:26:28.215ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:1:: -> fe80::74bc:2cff:fe86:fb5b on port Rear(Rear("rear0"))/0
642026-08-13T21:26:28.215ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:3:: -> fe80::74bc:2cff:fe86:fb5b on port Rear(Rear("rear0"))/0
652026-08-13T21:26:28.215ZINFOslog-rs: [tfportrear0_0] removing routes 0 from dendrite
662026-08-13T21:26:28.216ZDEBGslog-rs: [tfportrear0_0] redistributing update to 0 peers
672026-08-13T21:26:30.281ZINFOslog-rs: advertise tunnel: { TunnelOrigin { overlay_prefix: V4( Ipv4Net { addr: 0.0.0.0, width: 0, }, ), boundary_addr: fdc6:dc68:d0b8:9e27::1, vni: 99, metric: 18446744073709551615, }, }
68 Aug 13 21:26:31.250 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::b]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
69 Aug 13 21:26:31.250 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): operation timed out, delay: 15.004892764s, count: 0, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/oximeter/producer/src/lib.rs:376
70 Aug 13 21:26:31.445 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("87bd9e59-ba1d-4a64-a79d-60c17e91b150.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("95247c42-8a9b-428e-be43-565cc0cf9dad.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("cfc80aeb-ea69-4871-90d1-0edc16faef45.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("cfc80aeb-ea69-4871-90d1-0edc16faef45.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::a)) }], valid_until: Instant { tv_sec: 1052, tv_nsec: 270309282 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
71 Aug 13 21:26:31.445 DEBG will register with Nexus at [fd00:1122:3344:101::b]:12221, component: producer-registration-task
72 Aug 13 21:26:31.446 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::b]:12221/metrics/producers, method: POST, component: producer-registration-task
73 Aug 13 21:26:46.897 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::b]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
74 Aug 13 21:26:46.897 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): operation timed out, delay: 30.202652634s, count: 1, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/oximeter/producer/src/lib.rs:376
75 Aug 13 21:26:47.262 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("87bd9e59-ba1d-4a64-a79d-60c17e91b150.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("95247c42-8a9b-428e-be43-565cc0cf9dad.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("cfc80aeb-ea69-4871-90d1-0edc16faef45.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("cfc80aeb-ea69-4871-90d1-0edc16faef45.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::a)) }], valid_until: Instant { tv_sec: 1067, tv_nsec: 635379628 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
76 Aug 13 21:26:47.265 DEBG will register with Nexus at [fd00:1122:3344:101::b]:12221, component: producer-registration-task
77 Aug 13 21:26:47.266 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::b]:12221/metrics/producers, method: POST, component: producer-registration-task
78 Aug 13 21:27:02.286 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::b]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
79 Aug 13 21:27:02.287 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): operation timed out, delay: 45.57222429s, count: 2, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/oximeter/producer/src/lib.rs:376
80 Aug 13 21:27:03.073 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("87bd9e59-ba1d-4a64-a79d-60c17e91b150.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("95247c42-8a9b-428e-be43-565cc0cf9dad.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("cfc80aeb-ea69-4871-90d1-0edc16faef45.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("cfc80aeb-ea69-4871-90d1-0edc16faef45.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::a)) }], valid_until: Instant { tv_sec: 1083, tv_nsec: 429076336 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
81 Aug 13 21:27:03.076 DEBG will register with Nexus at [fd00:1122:3344:101::b]:12221, component: producer-registration-task
82 Aug 13 21:27:03.077 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::b]:12221/metrics/producers, method: POST, component: producer-registration-task
83 Aug 13 21:27:18.081 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::b]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
84 Aug 13 21:27:18.081 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): operation timed out, delay: 61.364665475s, count: 3, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/oximeter/producer/src/lib.rs:376
85 Aug 13 21:27:20.927 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("87bd9e59-ba1d-4a64-a79d-60c17e91b150.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("95247c42-8a9b-428e-be43-565cc0cf9dad.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("cfc80aeb-ea69-4871-90d1-0edc16faef45.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("cfc80aeb-ea69-4871-90d1-0edc16faef45.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::a)) }], valid_until: Instant { tv_sec: 1101, tv_nsec: 281465123 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
86 Aug 13 21:27:20.927 DEBG will register with Nexus at [fd00:1122:3344:101::b]:12221, component: producer-registration-task
87 Aug 13 21:27:20.928 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::b]:12221/metrics/producers, method: POST, component: producer-registration-task
88 Aug 13 21:27:35.929 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::b]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
89 Aug 13 21:27:35.929 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): operation timed out, delay: 79.212797389s, count: 4, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/oximeter/producer/src/lib.rs:376
90 Aug 13 21:27:37.986 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("87bd9e59-ba1d-4a64-a79d-60c17e91b150.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("95247c42-8a9b-428e-be43-565cc0cf9dad.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("cfc80aeb-ea69-4871-90d1-0edc16faef45.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("cfc80aeb-ea69-4871-90d1-0edc16faef45.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::a)) }], valid_until: Instant { tv_sec: 1118, tv_nsec: 341321436 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
91 Aug 13 21:27:37.987 DEBG will register with Nexus at [fd00:1122:3344:101::b]:12221, component: producer-registration-task
92 Aug 13 21:27:37.988 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::b]:12221/metrics/producers, method: POST, component: producer-registration-task
932026-08-13T21:27:57.224ZWARNslog-rs: [tfportrear0_0] neighbor buskin@fe80::74bc:2cff:fe86:fb5b expire
942026-08-13T21:27:57.228ZINFOslog-rs: [tfportrear0_0] nbr is fe80::74bc:2cff:fe86:fb5b@buskin server
952026-08-13T21:27:57.228ZWARNslog-rs: [tfportrear0_0] expiring peer fe80::74bc:2cff:fe86:fb5b due to discovery event
962026-08-13T21:27:57.230ZINFOslog-rs: [tfportrear0_0] removing routes 5 from dendrite
97 Aug 13 21:27:57.250 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::b]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
98 Aug 13 21:27:58.145 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): operation timed out, delay: 101.390228108s, count: 5, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/oximeter/producer/src/lib.rs:376
992026-08-13T21:27:59.208ZERROslog-rs: [tfportrear0_0] failed to remove tunnel routes: {} OPTE driver is not attached
1002026-08-13T21:27:59.208ZDEBGslog-rs: [tfportrear0_0] redistributing expire to 0 peers
1012026-08-13T21:27:59.234ZDEBGslog-rs: [tfportrear0_0] transition solicit -> exchange
1022026-08-13T21:27:59.263ZINFOslog-rs: [tfportrear0_0] exchange: listening on [fe80::7c06:5aff:fe16:2975]:56797
1032026-08-13T21:27:59.666ZINFOslog-rs: [tfportrear0_0] waiting for exchange server to start
1042026-08-13T21:27:59.821ZINFOslog-rs: [tfportrear0_0] sending 5 routes to dendrite
1052026-08-13T21:27:59.822ZDEBGslog-rs: [tfportrear0_0] sending to dpd host=localhost port=12224
1062026-08-13T21:27:59.842ZINFOslog-rs: [tfportrear0_0] adding route fdb0:18c0:4d0d:9fb2:: -> fe80::74bc:2cff:fe86:fb5b on port Rear(Rear("rear0"))/0
1072026-08-13T21:27:59.845ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:1:: -> fe80::74bc:2cff:fe86:fb5b on port Rear(Rear("rear0"))/0
1082026-08-13T21:27:59.846ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:101:: -> fe80::74bc:2cff:fe86:fb5b on port Rear(Rear("rear0"))/0
1092026-08-13T21:27:59.847ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:3:: -> fe80::74bc:2cff:fe86:fb5b on port Rear(Rear("rear0"))/0
1102026-08-13T21:27:59.847ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:2:: -> fe80::74bc:2cff:fe86:fb5b on port Rear(Rear("rear0"))/0
1112026-08-13T21:27:59.849ZINFOslog-rs: [tfportrear0_0] removing routes 0 from dendrite
1122026-08-13T21:27:59.850ZDEBGslog-rs: [tfportrear0_0] redistributing update to 0 peers
113 Aug 13 21:28:08.349 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("87bd9e59-ba1d-4a64-a79d-60c17e91b150.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("95247c42-8a9b-428e-be43-565cc0cf9dad.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("cfc80aeb-ea69-4871-90d1-0edc16faef45.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("cfc80aeb-ea69-4871-90d1-0edc16faef45.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::a)) }], valid_until: Instant { tv_sec: 1148, tv_nsec: 702638500 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
114 Aug 13 21:28:08.350 DEBG will register with Nexus at [fd00:1122:3344:101::b]:12221, component: producer-registration-task
115 Aug 13 21:28:08.352 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::b]:12221/metrics/producers, method: POST, component: producer-registration-task
116 Aug 13 21:28:23.353 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::b]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
117 Aug 13 21:28:23.354 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): operation timed out, delay: 126.637397828s, count: 6, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/oximeter/producer/src/lib.rs:376
118 Aug 13 21:28:34.519 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("87bd9e59-ba1d-4a64-a79d-60c17e91b150.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("95247c42-8a9b-428e-be43-565cc0cf9dad.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("cfc80aeb-ea69-4871-90d1-0edc16faef45.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("cfc80aeb-ea69-4871-90d1-0edc16faef45.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::a)) }], valid_until: Instant { tv_sec: 1174, tv_nsec: 873854837 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
119 Aug 13 21:28:34.520 DEBG will register with Nexus at [fd00:1122:3344:101::b]:12221, component: producer-registration-task
120 Aug 13 21:28:34.521 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::b]:12221/metrics/producers, method: POST, component: producer-registration-task
121 Aug 13 21:28:37.663 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::b]:12221/metrics/producers", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", [fd00:1122:3344:101::b]:12221, Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) }), component: producer-registration-task
122 Aug 13 21:28:37.664 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): client error (Connect): tcp connect error: Connection refused (os error 146), delay: 140.947195469s, count: 7, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/oximeter/producer/src/lib.rs:376
123 Aug 13 21:29:24.251 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("87bd9e59-ba1d-4a64-a79d-60c17e91b150.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("95247c42-8a9b-428e-be43-565cc0cf9dad.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("cfc80aeb-ea69-4871-90d1-0edc16faef45.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("cfc80aeb-ea69-4871-90d1-0edc16faef45.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::a)) }], valid_until: Instant { tv_sec: 1224, tv_nsec: 605626292 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
124 Aug 13 21:29:24.251 DEBG will register with Nexus at [fd00:1122:3344:101::b]:12221, component: producer-registration-task
125 Aug 13 21:29:24.253 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::b]:12221/metrics/producers, method: POST, component: producer-registration-task
126 Aug 13 21:29:24.277 DEBG client response, result: Ok(Response { url: "http://[fd00:1122:3344:101::b]:12221/metrics/producers", status: 201, headers: {"content-type": "application/json", "x-request-id": "8d36e31a-4d70-4028-b9b3-947920d49dae", "content-length": "41", "date": "Thu, 13 Aug 2026 21:29:24 GMT"} }), component: producer-registration-task
127 Aug 13 21:29:24.278 DEBG successfully registered with Nexus, pausing until time to renew lease, wait_period: 150s, lease_duration: 600s, component: producer-registration-task
128 Aug 13 21:29:37.891 INFO accepted connection, remote_addr: [fd00:1122:3344:101::d]:38534, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:1057
129 Aug 13 21:29:37.898 INFO request completed, latency_us: 6434, response_code: 200, uri: /9bfd2a13-f0c0-49cd-a1b0-bb79448e6869, method: GET, req_id: 35ac7101-c1dc-432d-97e9-5b5e81038af3, remote_addr: [fd00:1122:3344:101::d]:38534, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874
130 Aug 13 21:29:38.892 INFO request completed, latency_us: 1328, response_code: 200, uri: /9bfd2a13-f0c0-49cd-a1b0-bb79448e6869, method: GET, req_id: 667c5c78-8a04-45e3-94ce-6f94dd76f2c6, remote_addr: [fd00:1122:3344:101::d]:38534, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874
131 Aug 13 21:29:39.892 INFO request completed, latency_us: 961, response_code: 200, uri: /9bfd2a13-f0c0-49cd-a1b0-bb79448e6869, method: GET, req_id: 8f029033-76b0-4cec-8f53-70a9ef247dc4, remote_addr: [fd00:1122:3344:101::d]:38534, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874
132 Aug 13 21:29:40.892 INFO request completed, latency_us: 705, response_code: 200, uri: /9bfd2a13-f0c0-49cd-a1b0-bb79448e6869, method: GET, req_id: 06256e8c-ea2c-4a0d-9ded-2145ea422125, remote_addr: [fd00:1122:3344:101::d]:38534, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874
133 Aug 13 21:29:41.892 INFO request completed, latency_us: 938, response_code: 200, uri: /9bfd2a13-f0c0-49cd-a1b0-bb79448e6869, method: GET, req_id: b4c843b0-d7e5-4d2c-8693-3d81f1cf159b, remote_addr: [fd00:1122:3344:101::d]:38534, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874
134 Aug 13 21:29:42.892 INFO request completed, latency_us: 1070, response_code: 200, uri: /9bfd2a13-f0c0-49cd-a1b0-bb79448e6869, method: GET, req_id: c64c9140-576e-407f-8169-75046c0bc2af, remote_addr: [fd00:1122:3344:101::d]:38534, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874
135 Aug 13 21:29:43.892 INFO request completed, latency_us: 753, response_code: 200, uri: /9bfd2a13-f0c0-49cd-a1b0-bb79448e6869, method: GET, req_id: a359dc0f-2ab3-4329-995b-2469230728d4, remote_addr: [fd00:1122:3344:101::d]:38534, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874
136 Aug 13 21:29:44.892 INFO request completed, latency_us: 689, response_code: 200, uri: /9bfd2a13-f0c0-49cd-a1b0-bb79448e6869, method: GET, req_id: 2e7b64bd-4821-4006-9756-eca8097f9752, remote_addr: [fd00:1122:3344:101::d]:38534, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874
137 Aug 13 21:29:45.892 INFO request completed, latency_us: 1031, response_code: 200, uri: /9bfd2a13-f0c0-49cd-a1b0-bb79448e6869, method: GET, req_id: 377a81c4-861a-4d2f-b6f9-834fc31efd54, remote_addr: [fd00:1122:3344:101::d]:38534, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874
138 Aug 13 21:29:46.892 INFO request completed, latency_us: 701, response_code: 200, uri: /9bfd2a13-f0c0-49cd-a1b0-bb79448e6869, method: GET, req_id: 5893d1f4-cb31-4d73-85e8-40a4231c1984, remote_addr: [fd00:1122:3344:101::d]:38534, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874
139 Aug 13 21:29:47.891 INFO request completed, latency_us: 637, response_code: 200, uri: /9bfd2a13-f0c0-49cd-a1b0-bb79448e6869, method: GET, req_id: 3c10023e-3cd4-48ed-8be7-d1c2f90df927, remote_addr: [fd00:1122:3344:101::d]:38534, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874
140 Aug 13 21:29:48.892 INFO request completed, latency_us: 794, response_code: 200, uri: /9bfd2a13-f0c0-49cd-a1b0-bb79448e6869, method: GET, req_id: 053ccbce-6156-4805-9658-c97f0f35b64d, remote_addr: [fd00:1122:3344:101::d]:38534, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874
141 Aug 13 21:29:49.891 INFO request completed, latency_us: 664, response_code: 200, uri: /9bfd2a13-f0c0-49cd-a1b0-bb79448e6869, method: GET, req_id: 94e6f017-2eac-4496-9f9c-adce7b2acc10, remote_addr: [fd00:1122:3344:101::d]:38534, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874
142 Aug 13 21:29:50.892 INFO request completed, latency_us: 799, response_code: 200, uri: /9bfd2a13-f0c0-49cd-a1b0-bb79448e6869, method: GET, req_id: e8a43b85-3834-45c1-8766-6444e0a31f94, remote_addr: [fd00:1122:3344:101::d]:38534, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874
143 Aug 13 21:29:51.893 INFO request completed, latency_us: 815, response_code: 200, uri: /9bfd2a13-f0c0-49cd-a1b0-bb79448e6869, method: GET, req_id: d055094b-1237-42c2-84a8-9e25c9314851, remote_addr: [fd00:1122:3344:101::d]:38534, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874
144 Aug 13 21:29:52.892 INFO request completed, latency_us: 709, response_code: 200, uri: /9bfd2a13-f0c0-49cd-a1b0-bb79448e6869, method: GET, req_id: ddb6ccef-1520-4e24-b9e9-e2d969a18488, remote_addr: [fd00:1122:3344:101::d]:38534, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874
145 Aug 13 21:29:53.892 INFO request completed, latency_us: 728, response_code: 200, uri: /9bfd2a13-f0c0-49cd-a1b0-bb79448e6869, method: GET, req_id: d1296667-5af7-4fd8-991a-5f2d16210302, remote_addr: [fd00:1122:3344:101::d]:38534, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874
146 Aug 13 21:29:54.892 INFO request completed, latency_us: 656, response_code: 200, uri: /9bfd2a13-f0c0-49cd-a1b0-bb79448e6869, method: GET, req_id: c36006d7-81f4-46d4-9004-6811a2e87445, remote_addr: [fd00:1122:3344:101::d]:38534, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874
147 Aug 13 21:29:55.891 INFO request completed, latency_us: 826, response_code: 200, uri: /9bfd2a13-f0c0-49cd-a1b0-bb79448e6869, method: GET, req_id: e76608b1-5a09-43ac-9ea4-10f7dd7e4797, remote_addr: [fd00:1122:3344:101::d]:38534, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874
148 Aug 13 21:29:56.893 INFO request completed, latency_us: 912, response_code: 200, uri: /9bfd2a13-f0c0-49cd-a1b0-bb79448e6869, method: GET, req_id: 19d048de-4626-4c5b-8f66-ebcec5ebd6d6, remote_addr: [fd00:1122:3344:101::d]:38534, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874
149 Aug 13 21:29:57.892 INFO request completed, latency_us: 706, response_code: 200, uri: /9bfd2a13-f0c0-49cd-a1b0-bb79448e6869, method: GET, req_id: 7cfe1332-4886-4416-835c-ecad9a1a5fe8, remote_addr: [fd00:1122:3344:101::d]:38534, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874
150 Aug 13 21:29:58.892 INFO request completed, latency_us: 971, response_code: 200, uri: /9bfd2a13-f0c0-49cd-a1b0-bb79448e6869, method: GET, req_id: 3d39c60b-9174-4b13-8995-ee476c927072, remote_addr: [fd00:1122:3344:101::d]:38534, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874
151 Aug 13 21:29:59.892 INFO request completed, latency_us: 1066, response_code: 200, uri: /9bfd2a13-f0c0-49cd-a1b0-bb79448e6869, method: GET, req_id: ee7bda57-6619-47ce-94e2-952d7887668c, remote_addr: [fd00:1122:3344:101::d]:38534, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874
152 Aug 13 21:30:00.892 INFO request completed, latency_us: 854, response_code: 200, uri: /9bfd2a13-f0c0-49cd-a1b0-bb79448e6869, method: GET, req_id: 97c5836f-deda-458d-ade9-b1ba31642e83, remote_addr: [fd00:1122:3344:101::d]:38534, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874