Max level shown:
1 [ Jul 1 22:37:26 Disabled. ]
2 [ Jul 1 22:37:26 Rereading configuration. ]
3 [ Jul 1 22:37:27 Enabled. ]
4 [ Jul 1 22:37:27 Rereading configuration. ]
5 [ Jul 1 22:37:27 Executing start method ("ctrun -l child -o noorphan,regent /opt/oxide/mg-ddm/pkg/ddm_method_script.sh &"). ]
6 [ Jul 1 22:37:27 Method "start" exited with status 0. ]
72026-07-01T22:37:27.549ZINFOslog-rs: signal handler waiting for context
82026-07-01T22:37:27.570ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
92026-07-01T22:37:27.570ZERROslog-rs: failed to start stats server: underlay address not found
102026-07-01T22:37:27.570ZINFOslog-rs: signal handler waiting got context
112026-07-01T22:37:27.571ZINFOslog-rs: admin: listening on [::1]:8000
122026-07-01T22:37:27.571ZINFOslog-rs: admin: server started
132026-07-01T22:37:28.570ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
142026-07-01T22:37:29.570ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
152026-07-01T22:37:30.570ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
162026-07-01T22:37:31.570ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
172026-07-01T22:37:32.572ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
182026-07-01T22:37:33.572ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
192026-07-01T22:37:34.572ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
202026-07-01T22:37:35.572ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
212026-07-01T22:37:36.572ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
222026-07-01T22:37:37.572ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
232026-07-01T22:37:38.572ZINFOslog-rs: [tfportrear0_0] sm initialized with addr fe80::48f9:e5ff:fe5d:952 on if tfportrear0_0 index 3
242026-07-01T22:37:38.572ZDEBGslog-rs: [tfportrear0_0] starting discovery handler
25 [ Jul 1 22:37:39 Rereading configuration. ]
26 [ Jul 1 22:37:39 Executing refresh method ("/opt/oxide/mg-ddm/pkg/refresh_ctrun_child.sh -c 213"). ]
27 + CONTRACT=''
28 + getopts c: o
29 + CONTRACT=213
30 + getopts c: o
31 + [[ -n 213 ]]
32 + pgrep -c 213
33 + PARENT_PID=2288
34 + [[ -n 2288 ]]
35 + pgrep -P 2288
36 + CHILD_PID=2292
37 + [[ -n 2292 ]]
38 + kill -USR1 2292
392026-07-01T22:37:39.043ZINFOslog-rs: handling smf refresh
40 [ Jul 1 22:37:39 Method "refresh" exited with status 0. ]
412026-07-01T22:37:39.045ZINFOslog-rs: starting stats server on smf refresh
42 Jul 01 22:37:39.045 DEBG registered DTrace probes
43 Jul 01 22:37:39.046 DEBG registered endpoint, versions: all, path: /{producer_id}, method: GET, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot
44 Jul 01 22:37:39.046 INFO listening, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:335
45 Jul 01 22:37:39.046 DEBG successfully registered DTrace USDT probes, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot
46 Jul 01 22:37:39.046 DEBG Nexus IP not provided, will use DNS to resolve it
47 Jul 01 22:37:39.046 INFO new DNS resolver, addresses: [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53], component: internal-dns-resolver, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/internal-dns/resolver/src/resolver.rs:144
48 Jul 01 22:37:39.047 DEBG starting producer registration task
49 Jul 01 22:37:39.047 INFO starting oximeter metric producer server, interval: 1s, address: [fd00:1122:3344:101::2]:8001, producer_id: 930eefaf-9627-4963-9d5e-b1935bd9c078, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/oximeter/producer/src/lib.rs:280
50 Jul 01 22:37:39.047 DEBG registering as oximeter producer with Nexus, registration_count: 0, component: producer-registration-task
51 Jul 01 22:37:39.048 WARN failed to register with Nexus, will retry, error: proto error: no records found for Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, delay: 1.424929ms, count: 0, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/oximeter/producer/src/lib.rs:376
52 Jul 01 22:37:39.351 WARN failed to register with Nexus, will retry, error: proto error: no records found for Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, delay: 304.077181ms, count: 1, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/oximeter/producer/src/lib.rs:376
53 Jul 01 22:37:39.706 WARN failed to register with Nexus, will retry, error: proto error: no records found for Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, delay: 658.832064ms, count: 2, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/oximeter/producer/src/lib.rs:376
542026-07-01T22:37:40.244ZWARNslog-rs: [tfportrear0_0] admin event in solicit state: Announce(Underlay({Ipv6Net { addr: fd07:9c5:77a2:b8fb::1, width: 64 }}))
552026-07-01T22:37:40.244ZINFOslog-rs: advertise tunnel: { TunnelOrigin { overlay_prefix: V4( Ipv4Net { addr: 0.0.0.0, width: 0, }, ), boundary_addr: fd07:9c5:77a2:b8fb::1, vni: 99, metric: 18446744073709551615, }, }
562026-07-01T22:37:40.244ZWARNslog-rs: [tfportrear0_0] admin event in solicit state: Announce(Tunnel({TunnelOrigin { overlay_prefix: V4(Ipv4Net { addr: 0.0.0.0, width: 0 }), boundary_addr: fd07:9c5:77a2:b8fb::1, vni: 99, metric: 18446744073709551615 }}))
572026-07-01T22:37:40.573ZINFOslog-rs: [tfportrear0_0] nbr is fe80::4e:64ff:fe42:e0d6@buskin server
582026-07-01T22:37:40.573ZDEBGslog-rs: [tfportrear0_0] transition solicit -> exchange
592026-07-01T22:37:40.573ZINFOslog-rs: [tfportrear0_0] exchange: listening on [fe80::48f9:e5ff:fe5d:952]:56797
602026-07-01T22:37:40.573ZINFOslog-rs: [tfportrear0_0] waiting for exchange server to start
612026-07-01T22:37:40.575ZINFOslog-rs: [tfportrear0_0] sending 5 routes to dendrite
622026-07-01T22:37:40.576ZDEBGslog-rs: [tfportrear0_0] sending to dpd host=localhost port=12224
632026-07-01T22:37:40.577ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:3:: -> fe80::4e:64ff:fe42:e0d6 on port Rear(Rear("rear0"))/0
642026-07-01T22:37:40.579ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:101:: -> fe80::4e:64ff:fe42:e0d6 on port Rear(Rear("rear0"))/0
652026-07-01T22:37:40.579ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:1:: -> fe80::4e:64ff:fe42:e0d6 on port Rear(Rear("rear0"))/0
662026-07-01T22:37:40.580ZINFOslog-rs: [tfportrear0_0] adding route fdb0:18c0:4d0d:9fb2:: -> fe80::4e:64ff:fe42:e0d6 on port Rear(Rear("rear0"))/0
672026-07-01T22:37:40.580ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:2:: -> fe80::4e:64ff:fe42:e0d6 on port Rear(Rear("rear0"))/0
682026-07-01T22:37:40.580ZINFOslog-rs: [tfportrear0_0] removing routes 0 from dendrite
692026-07-01T22:37:40.581ZDEBGslog-rs: [tfportrear0_0] redistributing update to 0 peers
70 Jul 01 22:37:41.120 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("a8ff36ae-1465-45d3-a058-a7b646f1671d.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("b9692f1d-f06c-4f81-b8a4-c1ea201bbc50.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("d3adb5ce-056c-4861-b376-83383a136fd8.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("d3adb5ce-056c-4861-b376-83383a136fd8.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::c)) }], valid_until: Instant { tv_sec: 679, tv_nsec: 362468936 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
71 Jul 01 22:37:41.121 DEBG will register with Nexus at [fd00:1122:3344:101::b]:12221, component: producer-registration-task
72 Jul 01 22:37:41.122 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::b]:12221/metrics/producers, method: POST, component: producer-registration-task
73 Jul 01 22:37:56.044 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::b]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
74 Jul 01 22:37:56.044 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): operation timed out, delay: 17.07760108s, count: 3, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/oximeter/producer/src/lib.rs:376
75 Jul 01 22:37:58.843 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("a8ff36ae-1465-45d3-a058-a7b646f1671d.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("b9692f1d-f06c-4f81-b8a4-c1ea201bbc50.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("d3adb5ce-056c-4861-b376-83383a136fd8.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("d3adb5ce-056c-4861-b376-83383a136fd8.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::c)) }], valid_until: Instant { tv_sec: 697, tv_nsec: 158475778 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
76 Jul 01 22:37:58.844 DEBG will register with Nexus at [fd00:1122:3344:101::b]:12221, component: producer-registration-task
77 Jul 01 22:37:58.845 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::b]:12221/metrics/producers, method: POST, component: producer-registration-task
78 Jul 01 22:38:13.886 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::b]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
79 Jul 01 22:38:13.886 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): operation timed out, delay: 34.872017219s, count: 4, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/oximeter/producer/src/lib.rs:376
80 Jul 01 22:38:18.807 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("a8ff36ae-1465-45d3-a058-a7b646f1671d.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("b9692f1d-f06c-4f81-b8a4-c1ea201bbc50.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("d3adb5ce-056c-4861-b376-83383a136fd8.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("d3adb5ce-056c-4861-b376-83383a136fd8.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::c)) }], valid_until: Instant { tv_sec: 717, tv_nsec: 69142634 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
81 Jul 01 22:38:18.807 DEBG will register with Nexus at [fd00:1122:3344:101::b]:12221, component: producer-registration-task
82 Jul 01 22:38:18.808 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::b]:12221/metrics/producers, method: POST, component: producer-registration-task
83 Jul 01 22:38:33.851 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::b]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
84 Jul 01 22:38:33.851 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): operation timed out, delay: 54.78336425s, count: 5, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/oximeter/producer/src/lib.rs:376
85 Jul 01 22:38:43.507 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("a8ff36ae-1465-45d3-a058-a7b646f1671d.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("b9692f1d-f06c-4f81-b8a4-c1ea201bbc50.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("d3adb5ce-056c-4861-b376-83383a136fd8.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("d3adb5ce-056c-4861-b376-83383a136fd8.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::c)) }], valid_until: Instant { tv_sec: 741, tv_nsec: 702157863 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
86 Jul 01 22:38:43.508 DEBG will register with Nexus at [fd00:1122:3344:101::b]:12221, component: producer-registration-task
87 Jul 01 22:38:43.509 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::b]:12221/metrics/producers, method: POST, component: producer-registration-task
88 Jul 01 22:38:58.550 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::b]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
89 Jul 01 22:38:58.550 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): operation timed out, delay: 79.414844475s, count: 6, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/oximeter/producer/src/lib.rs:376
90 Jul 01 22:39:17.605 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("a8ff36ae-1465-45d3-a058-a7b646f1671d.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("b9692f1d-f06c-4f81-b8a4-c1ea201bbc50.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("d3adb5ce-056c-4861-b376-83383a136fd8.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("d3adb5ce-056c-4861-b376-83383a136fd8.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::c)) }], valid_until: Instant { tv_sec: 775, tv_nsec: 707904698 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
91 Jul 01 22:39:17.606 DEBG will register with Nexus at [fd00:1122:3344:101::b]:12221, component: producer-registration-task
92 Jul 01 22:39:17.606 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::b]:12221/metrics/producers, method: POST, component: producer-registration-task
93 Jul 01 22:39:32.649 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::b]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
94 Jul 01 22:39:32.649 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): operation timed out, delay: 113.421653444s, count: 7, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/oximeter/producer/src/lib.rs:376
95 Jul 01 22:40:01.693 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("a8ff36ae-1465-45d3-a058-a7b646f1671d.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("b9692f1d-f06c-4f81-b8a4-c1ea201bbc50.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("d3adb5ce-056c-4861-b376-83383a136fd8.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("d3adb5ce-056c-4861-b376-83383a136fd8.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::c)) }], valid_until: Instant { tv_sec: 819, tv_nsec: 676217712 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
96 Jul 01 22:40:01.694 DEBG will register with Nexus at [fd00:1122:3344:101::b]:12221, component: producer-registration-task
97 Jul 01 22:40:01.695 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::b]:12221/metrics/producers, method: POST, component: producer-registration-task
98 Jul 01 22:40:16.737 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::b]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
99 Jul 01 22:40:16.737 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): operation timed out, delay: 157.389857615s, count: 8, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/oximeter/producer/src/lib.rs:376
100 Jul 01 22:41:10.106 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("a8ff36ae-1465-45d3-a058-a7b646f1671d.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("b9692f1d-f06c-4f81-b8a4-c1ea201bbc50.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("d3adb5ce-056c-4861-b376-83383a136fd8.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("d3adb5ce-056c-4861-b376-83383a136fd8.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::c)) }], valid_until: Instant { tv_sec: 888, tv_nsec: 32947959 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
101 Jul 01 22:41:10.107 DEBG will register with Nexus at [fd00:1122:3344:101::b]:12221, component: producer-registration-task
102 Jul 01 22:41:10.108 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::b]:12221/metrics/producers, method: POST, component: producer-registration-task
1032026-07-01T22:41:26.557ZWARNslog-rs: [tfportrear0_0] neighbor buskin@fe80::4e:64ff:fe42:e0d6 missed solicit interval
104 Jul 01 22:41:26.953 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::b]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
105 Jul 01 22:41:27.029 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): operation timed out, delay: 227.664242167s, count: 9, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/oximeter/producer/src/lib.rs:376
106 Jul 01 22:44:25.131 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("a8ff36ae-1465-45d3-a058-a7b646f1671d.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("b9692f1d-f06c-4f81-b8a4-c1ea201bbc50.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("d3adb5ce-056c-4861-b376-83383a136fd8.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("d3adb5ce-056c-4861-b376-83383a136fd8.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::c)) }], valid_until: Instant { tv_sec: 1083, tv_nsec: 56064684 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
107 Jul 01 22:44:25.133 DEBG will register with Nexus at [fd00:1122:3344:101::b]:12221, component: producer-registration-task
108 Jul 01 22:44:25.141 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::b]:12221/metrics/producers, method: POST, component: producer-registration-task
109 Jul 01 22:44:25.150 DEBG client response, result: Ok(Response { url: "http://[fd00:1122:3344:101::b]:12221/metrics/producers", status: 201, headers: {"content-type": "application/json", "x-request-id": "339d7178-dadf-4ba9-8d4a-91ee256a8097", "content-length": "41", "date": "Wed, 01 Jul 2026 22:44:25 GMT"} }), component: producer-registration-task
110 Jul 01 22:44:25.151 DEBG successfully registered with Nexus, pausing until time to renew lease, wait_period: 150s, lease_duration: 600s, component: producer-registration-task
111 Jul 01 22:44:27.589 INFO accepted connection, remote_addr: [fd00:1122:3344:101::d]:61228, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:1057
112 Jul 01 22:44:27.596 INFO request completed, latency_us: 7128, response_code: 200, uri: /930eefaf-9627-4963-9d5e-b1935bd9c078, method: GET, req_id: 58f760ac-a368-4b60-b954-483f948715cd, remote_addr: [fd00:1122:3344:101::d]:61228, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874
113 Jul 01 22:44:28.590 INFO request completed, latency_us: 613, response_code: 200, uri: /930eefaf-9627-4963-9d5e-b1935bd9c078, method: GET, req_id: 69f27c5f-8b1e-4e1b-b694-c3219a679815, remote_addr: [fd00:1122:3344:101::d]:61228, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874
114 Jul 01 22:44:29.589 INFO request completed, latency_us: 575, response_code: 200, uri: /930eefaf-9627-4963-9d5e-b1935bd9c078, method: GET, req_id: 44f22322-a510-42d7-9724-e1f62de66299, remote_addr: [fd00:1122:3344:101::d]:61228, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874
115 Jul 01 22:44:30.590 INFO request completed, latency_us: 618, response_code: 200, uri: /930eefaf-9627-4963-9d5e-b1935bd9c078, method: GET, req_id: fa2ac1f1-4fa4-4546-a69d-4c618e187c1a, remote_addr: [fd00:1122:3344:101::d]:61228, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874
116 Jul 01 22:44:31.590 INFO request completed, latency_us: 604, response_code: 200, uri: /930eefaf-9627-4963-9d5e-b1935bd9c078, method: GET, req_id: 051152b2-7eed-469b-9b08-29267b98abd3, remote_addr: [fd00:1122:3344:101::d]:61228, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874
117 Jul 01 22:44:32.589 INFO request completed, latency_us: 814, response_code: 200, uri: /930eefaf-9627-4963-9d5e-b1935bd9c078, method: GET, req_id: fc7e270d-9145-4892-8714-c0fb7e973a10, remote_addr: [fd00:1122:3344:101::d]:61228, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874
118 Jul 01 22:44:33.590 INFO request completed, latency_us: 615, response_code: 200, uri: /930eefaf-9627-4963-9d5e-b1935bd9c078, method: GET, req_id: d049399e-4821-4d8b-9d3b-bc0aa1a288cc, remote_addr: [fd00:1122:3344:101::d]:61228, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874
119 Jul 01 22:44:34.590 INFO request completed, latency_us: 820, response_code: 200, uri: /930eefaf-9627-4963-9d5e-b1935bd9c078, method: GET, req_id: 15b7ac49-7efc-400b-a78e-8216196bb092, remote_addr: [fd00:1122:3344:101::d]:61228, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874
120 Jul 01 22:44:35.590 INFO request completed, latency_us: 864, response_code: 200, uri: /930eefaf-9627-4963-9d5e-b1935bd9c078, method: GET, req_id: 537499a8-93cf-4571-aaef-3bf34a2364f4, remote_addr: [fd00:1122:3344:101::d]:61228, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874
121 Jul 01 22:44:36.589 INFO request completed, latency_us: 589, response_code: 200, uri: /930eefaf-9627-4963-9d5e-b1935bd9c078, method: GET, req_id: 9554a395-0f16-44ac-86b4-f285a4702c51, remote_addr: [fd00:1122:3344:101::d]:61228, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874
122 Jul 01 22:44:37.591 INFO request completed, latency_us: 835, response_code: 200, uri: /930eefaf-9627-4963-9d5e-b1935bd9c078, method: GET, req_id: a94940d9-77d2-4163-b3bc-e1df8a070729, remote_addr: [fd00:1122:3344:101::d]:61228, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874
123 Jul 01 22:44:38.590 INFO request completed, latency_us: 583, response_code: 200, uri: /930eefaf-9627-4963-9d5e-b1935bd9c078, method: GET, req_id: f7a1d5a4-5318-4d42-90cd-2314fd900d2b, remote_addr: [fd00:1122:3344:101::d]:61228, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874
124 Jul 01 22:44:39.590 INFO request completed, latency_us: 870, response_code: 200, uri: /930eefaf-9627-4963-9d5e-b1935bd9c078, method: GET, req_id: a74e2230-afd4-4824-8cec-f04a00fdfbf5, remote_addr: [fd00:1122:3344:101::d]:61228, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874
125 Jul 01 22:44:40.590 INFO request completed, latency_us: 586, response_code: 200, uri: /930eefaf-9627-4963-9d5e-b1935bd9c078, method: GET, req_id: 558f79bf-62b9-46af-a79a-40d0f374f04a, remote_addr: [fd00:1122:3344:101::d]:61228, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874
126 Jul 01 22:44:41.590 INFO request completed, latency_us: 688, response_code: 200, uri: /930eefaf-9627-4963-9d5e-b1935bd9c078, method: GET, req_id: fe68b800-3f12-4db6-a602-a5f38f8bb34c, remote_addr: [fd00:1122:3344:101::d]:61228, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874
127 Jul 01 22:44:42.590 INFO request completed, latency_us: 667, response_code: 200, uri: /930eefaf-9627-4963-9d5e-b1935bd9c078, method: GET, req_id: c8f8cd0d-8006-4f6d-93e0-b3a148007399, remote_addr: [fd00:1122:3344:101::d]:61228, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874
128 Jul 01 22:44:43.590 INFO request completed, latency_us: 717, response_code: 200, uri: /930eefaf-9627-4963-9d5e-b1935bd9c078, method: GET, req_id: 9aa4a715-5fe3-423a-9859-8d68a778e7a6, remote_addr: [fd00:1122:3344:101::d]:61228, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874
129 Jul 01 22:44:44.590 INFO request completed, latency_us: 707, response_code: 200, uri: /930eefaf-9627-4963-9d5e-b1935bd9c078, method: GET, req_id: 31706d4e-fa78-4124-85f0-3ebf9677d41c, remote_addr: [fd00:1122:3344:101::d]:61228, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874
130 Jul 01 22:44:45.590 INFO request completed, latency_us: 608, response_code: 200, uri: /930eefaf-9627-4963-9d5e-b1935bd9c078, method: GET, req_id: cbad1013-a8c6-406f-958a-d147b0d770ac, remote_addr: [fd00:1122:3344:101::d]:61228, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874
131 Jul 01 22:44:46.589 INFO request completed, latency_us: 698, response_code: 200, uri: /930eefaf-9627-4963-9d5e-b1935bd9c078, method: GET, req_id: 3e2ea5a2-9bb5-479c-96f6-c0eed4a7efe8, remote_addr: [fd00:1122:3344:101::d]:61228, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874
132 Jul 01 22:44:47.590 INFO request completed, latency_us: 704, response_code: 200, uri: /930eefaf-9627-4963-9d5e-b1935bd9c078, method: GET, req_id: 44b29af3-ebc4-4184-bab8-f97f6a749af9, remote_addr: [fd00:1122:3344:101::d]:61228, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874
133 Jul 01 22:44:48.590 INFO request completed, latency_us: 714, response_code: 200, uri: /930eefaf-9627-4963-9d5e-b1935bd9c078, method: GET, req_id: 77e957d0-cfe5-452c-8a68-1726b4c5e9f7, remote_addr: [fd00:1122:3344:101::d]:61228, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874
134 Jul 01 22:44:49.590 INFO request completed, latency_us: 852, response_code: 200, uri: /930eefaf-9627-4963-9d5e-b1935bd9c078, method: GET, req_id: 591376fa-650a-4823-bbdc-e35222b37163, remote_addr: [fd00:1122:3344:101::d]:61228, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874
135 Jul 01 22:44:50.590 INFO request completed, latency_us: 709, response_code: 200, uri: /930eefaf-9627-4963-9d5e-b1935bd9c078, method: GET, req_id: e6f7f0c8-b085-4d4b-bfe4-30bd3d61e074, remote_addr: [fd00:1122:3344:101::d]:61228, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874
136 Jul 01 22:44:51.590 INFO request completed, latency_us: 657, response_code: 200, uri: /930eefaf-9627-4963-9d5e-b1935bd9c078, method: GET, req_id: 4c8425d2-ea17-405f-a482-b1bade049627, remote_addr: [fd00:1122:3344:101::d]:61228, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874
137 Jul 01 22:44:52.590 INFO request completed, latency_us: 611, response_code: 200, uri: /930eefaf-9627-4963-9d5e-b1935bd9c078, method: GET, req_id: bf938ede-0ca2-4033-88e0-d503a0bb1ffe, remote_addr: [fd00:1122:3344:101::d]:61228, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874
138 Jul 01 22:44:53.591 INFO request completed, latency_us: 1023, response_code: 200, uri: /930eefaf-9627-4963-9d5e-b1935bd9c078, method: GET, req_id: 4960f1b2-5442-44b4-bcef-17f7720e8a5a, remote_addr: [fd00:1122:3344:101::d]:61228, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874
139 Jul 01 22:44:54.590 INFO request completed, latency_us: 688, response_code: 200, uri: /930eefaf-9627-4963-9d5e-b1935bd9c078, method: GET, req_id: a2c71559-6662-45c7-8db7-4d0f2ae47bb7, remote_addr: [fd00:1122:3344:101::d]:61228, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874
140 Jul 01 22:44:55.590 INFO request completed, latency_us: 702, response_code: 200, uri: /930eefaf-9627-4963-9d5e-b1935bd9c078, method: GET, req_id: 3b893432-5606-4b44-97fb-6465b3b36a1e, remote_addr: [fd00:1122:3344:101::d]:61228, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874
141 Jul 01 22:44:56.590 INFO request completed, latency_us: 602, response_code: 200, uri: /930eefaf-9627-4963-9d5e-b1935bd9c078, method: GET, req_id: 411fdd00-b820-49eb-aa49-7b84213dceec, remote_addr: [fd00:1122:3344:101::d]:61228, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874
142 Jul 01 22:44:57.590 INFO request completed, latency_us: 692, response_code: 200, uri: /930eefaf-9627-4963-9d5e-b1935bd9c078, method: GET, req_id: 1c273591-6460-4749-bbd1-19b65ed162fa, remote_addr: [fd00:1122:3344:101::d]:61228, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874
143 Jul 01 22:44:58.590 INFO request completed, latency_us: 598, response_code: 200, uri: /930eefaf-9627-4963-9d5e-b1935bd9c078, method: GET, req_id: 0c16836f-0cd6-4e10-a9e1-778406dc9fd8, remote_addr: [fd00:1122:3344:101::d]:61228, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874
144 Jul 01 22:44:59.590 INFO request completed, latency_us: 621, response_code: 200, uri: /930eefaf-9627-4963-9d5e-b1935bd9c078, method: GET, req_id: 80ecde0c-d6b4-4842-8a93-37acaa9f69c4, remote_addr: [fd00:1122:3344:101::d]:61228, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874
145 Jul 01 22:45:00.590 INFO request completed, latency_us: 958, response_code: 200, uri: /930eefaf-9627-4963-9d5e-b1935bd9c078, method: GET, req_id: 4b41386f-53b4-459b-b614-5eda0354f4a2, remote_addr: [fd00:1122:3344:101::d]:61228, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874