Max level shown:
1 [ Aug 14 22:28:34 Enabled. ]
2 [ Aug 14 22:28:34 Rereading configuration. ]
3 [ Aug 14 22:28:34 Rereading configuration. ]
4 [ Aug 14 22:28:55 Executing start method ("ctrun -l child -o noorphan,regent /opt/oxide/dendrite/bin/tfportd --sync-only &"). ]
5 [ Aug 14 22:28:55 Method "start" exited with status 0. ]
6 No non-localhost IPv6 addresses found in SMF properties
72026-08-14T22:28:55.837ZINFOtfportd: connecting to dpd host = [::1] port = 12224
82026-08-14T22:28:55.839ZINFOtfportd: connected to dpd running softnpu-5cb3f22d
92026-08-14T22:28:55.839ZINFOtfportd: running in sync-only mode
102026-08-14T22:28:55.840ZINFOtfportd: no non-localhost IPv6 listen address available, waiting for SMF update unit = oximeter
112026-08-14T22:28:55.841ZINFOtfportd: refreshing SMF config prior to waiting on signals unit = signal_handler
12 No non-localhost IPv6 addresses found in SMF properties
132026-08-14T22:28:55.843ZINFOtfportd: refreshed config: Config { log_file: None, log_format: Json, listen_addresses: [ [::1]:12231, ], pkt_source: None, dpd_host: "[::1]", dpd_port: 12224, vlan_link: None, vlan_data: None, techport0_prefix: None, techport1_prefix: None, rack_id: Some( 7fc4d967-072b-4b5d-8816-23ea333aaea5, ), sled_id: Some( cc158c62-8249-45e1-b37f-67a8657b047f, ), sled_model: Some( "i86pc", ), sled_revision: Some( 0, ), sled_serial: Some( "buskin", ), }
142026-08-14T22:28:55.843ZINFOtfportd: no non-localhost IPv6 listen address available, waiting for SMF update unit = oximeter
152026-08-14T22:28:55.848ZINFOtfportd: added new NDP entry entry = tfportrear0_0:fe80::1cdf:a5ff:fe60:2ff->1e:df:a5:60:02:ff interface = tfportrear0_0
162026-08-14T22:28:55.849ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
172026-08-14T22:28:55.854ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::1cdf:a5ff:fe60:2ff]
182026-08-14T22:28:55.860ZWARNtfportd: tfportqsfp0_0 has no addrs
19 [ Aug 14 22:28:56 Rereading configuration. ]
20 [ Aug 14 22:28:56 Executing refresh method (:kill). ]
21 [ Aug 14 22:28:56 Stopping because all processes in service exited. ]
22 [ Aug 14 22:28:56 Executing stop method (:kill). ]
23 [ Aug 14 22:28:56 Executing start method ("ctrun -l child -o noorphan,regent /opt/oxide/dendrite/bin/tfportd --sync-only &"). ]
24 [ Aug 14 22:28:56 Method "start" exited with status 0. ]
252026-08-14T22:28:56.550ZINFOtfportd: connecting to dpd host = [::1] port = 12224
262026-08-14T22:28:56.551ZINFOtfportd: connected to dpd running softnpu-5cb3f22d
272026-08-14T22:28:56.551ZINFOtfportd: running in sync-only mode
282026-08-14T22:28:56.551ZINFOtfportd: found suitable IPv6 address for oximeter producer server address = fd00:1122:3344:101::2 unit = oximeter
292026-08-14T22:28:56.553ZINFOtfportd: refreshing SMF config prior to waiting on signals unit = signal_handler
302026-08-14T22:28:56.553ZDEBGtfportd: extracted oximeter configuration data config = OximeterConfig { listen_address: fd00:1122:3344:101::2, sled_identifiers: SledIdentifiers { rack_id: 7fc4d967-072b-4b5d-8816-23ea333aaea5, sled_id: cc158c62-8249-45e1-b37f-67a8657b047f, model: "i86pc", revision: 0, serial: "buskin" }, switch_identifiers: SwitchIdentifiers { asic_backend: "softnpu", fab: None, fuse: Some(FuseData { chip_rev: ChipRevision { device_id: 272, rev: "B1", rev_num: 2 }, disabled: DisabledFeatures { bsync: false, mau: [0, 0, 0, 0], pgen: false, pipes: 0, ports: 0, resub: false, speeds: 0, tm_mem: 0 }, frequency: FrequencySettings { bps: 3, bps_ext: 0, cpu_speed_dis: 0, disabled: false, pcie_dis: 0, pps: 3, pps_ext: 0 }, manufacturing: ManufacturingData { core_repair: false, die_rotation: false, pmro_and_skew: 0, silent_spin: 0, soft_pipe_dis: 0, tile_repair: false, voltage_scaling: 0, wf_core_repair: false }, part: PartInfo { part_num: 4660, pkg_id: 1, version: 2 } }), lot: None, lotnum: None, model: "FAKE_SIM_SIDECAR", revision: 0, serial: "SimSidecar0", sidecar_id: a9f3a565-efd6-46ed-9909-8a8c48a13355, slot: 0, wafer: None, wafer_loc: None } } unit = oximeter
312026-08-14T22:28:56.553ZDEBGtfportd: created producer ID producer_id = 1a22f060-6ab4-4851-a260-cece1ee0998f unit = oximeter
322026-08-14T22:28:56.556ZINFOtfportd: refreshed config: Config { log_file: None, log_format: Json, listen_addresses: [ [fd00:1122:3344:101::2]:12231, [::1]:12231, ], pkt_source: None, dpd_host: "[::1]", dpd_port: 12224, vlan_link: None, vlan_data: None, techport0_prefix: None, techport1_prefix: None, rack_id: Some( 7fc4d967-072b-4b5d-8816-23ea333aaea5, ), sled_id: Some( cc158c62-8249-45e1-b37f-67a8657b047f, ), sled_model: Some( "i86pc", ), sled_revision: Some( 0, ), sled_serial: Some( "buskin", ), }
332026-08-14T22:28:56.563ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
342026-08-14T22:28:56.566ZINFOtfportd: added new NDP entry entry = tfportrear0_0:fe80::1cdf:a5ff:fe60:2ff->1e:df:a5:60:02:ff interface = tfportrear0_0
352026-08-14T22:28:56.566ZINFOtfportd: added new NDP entry entry = tfportrear0_0:fe80::c89b:39ff:fe91:93f0->ca:9b:39:91:93:f0 interface = tfportrear0_0
362026-08-14T22:28:56.570ZDEBGtfportd: registered DTrace probes unit = producer-server
372026-08-14T22:28:56.571ZDEBGtfportd (dropshot): registered endpoint local_addr = [fd00:1122:3344:101::2]:55197 method = GET path = /{producer_id} unit = producer-server versions = all
382026-08-14T22:28:56.571ZINFOtfportd (dropshot): listening file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.7/src/server.rs:328 local_addr = [fd00:1122:3344:101::2]:55197 unit = producer-server
392026-08-14T22:28:56.571ZDEBGtfportd (dropshot): successfully registered DTrace USDT probes local_addr = [fd00:1122:3344:101::2]:55197 unit = producer-server
402026-08-14T22:28:56.571ZDEBGtfportd: Requested any available port, Dropshot server has been bound to [fd00:1122:3344:101::2]:55197 unit = producer-server
412026-08-14T22:28:56.571ZDEBGtfportd: Nexus IP not provided, will use DNS to resolve it unit = producer-server
422026-08-14T22:28:56.572ZINFOtfportd (internal-dns-resolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53] file = /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/8e30605/internal-dns/resolver/src/resolver.rs:144 unit = producer-server
432026-08-14T22:28:56.572ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::1cdf:a5ff:fe60:2ff]
442026-08-14T22:28:56.572ZDEBGtfportd: starting producer registration task unit = producer-server
452026-08-14T22:28:56.572ZINFOtfportd: starting oximeter metric producer server address = [fd00:1122:3344:101::2]:55197 file = /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/8e30605/oximeter/producer/src/lib.rs:279 interval = 30s producer_id = 1a22f060-6ab4-4851-a260-cece1ee0998f unit = producer-server
462026-08-14T22:28:56.572ZINFOtfportd: created oximeter producer server unit = oximeter
472026-08-14T22:28:56.572ZDEBGtfportd (producer-registration-task): registering as oximeter producer with Nexus registration_count = 0 unit = producer-server
482026-08-14T22:28:56.573ZDEBGtfportd (kstat-sampler): adding new target unit = oximeter
492026-08-14T22:28:56.574ZDEBGtfportd (kstat-sampler-worker): storing new creation time for tracked target creation_time = 2026-08-14T22:28:29.645320102Z path = KstatPath { module: "link", instance: 0, name: "tfportrear0_0" } unit = oximeter
502026-08-14T22:28:56.574ZDEBGtfportd (kstat-sampler-worker): inserted empty per-target sample queue id = 17094535478246951415 unit = oximeter
512026-08-14T22:28:56.574ZDEBGtfportd: added new link to kstat sampler link_kind = switch-port-control link_name = tfportrear0_0 unit = metrics-task-tracker
522026-08-14T22:28:56.576ZWARNtfportd: tfportqsfp0_0 has no addrs
532026-08-14T22:28:57.333ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
542026-08-14T22:28:57.339ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::1cdf:a5ff:fe60:2ff]
552026-08-14T22:28:57.343ZWARNtfportd: tfportqsfp0_0 has no addrs
562026-08-14T22:28:58.213ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
572026-08-14T22:28:58.219ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::1cdf:a5ff:fe60:2ff]
582026-08-14T22:28:58.222ZWARNtfportd: tfportqsfp0_0 has no addrs
592026-08-14T22:28:58.862ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
602026-08-14T22:28:58.868ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::1cdf:a5ff:fe60:2ff]
612026-08-14T22:28:58.871ZWARNtfportd: tfportqsfp0_0 has no addrs
622026-08-14T22:28:59.601ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
632026-08-14T22:28:59.606ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::1cdf:a5ff:fe60:2ff]
642026-08-14T22:28:59.610ZWARNtfportd: tfportqsfp0_0 has no addrs
652026-08-14T22:29:00.686ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
662026-08-14T22:29:00.693ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::1cdf:a5ff:fe60:2ff]
672026-08-14T22:29:00.696ZWARNtfportd: tfportqsfp0_0 has no addrs
682026-08-14T22:29:01.574ZDEBGtfportd (internal-dns-resolver): lookup_socket_v6 srv dns_name = _nexus._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("84ded4f1-c37e-4acd-b277-44c3a301227b.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("92be6118-aeb7-43c8-b50f-1195aaf134f5.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("d263ad7b-e87f-4142-8c0c-19548c9e8f70.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("d263ad7b-e87f-4142-8c0c-19548c9e8f70.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::c)) }], valid_until: Instant { tv_sec: 1250, tv_nsec: 148844289 } }) unit = producer-server
692026-08-14T22:29:01.575ZDEBGtfportd (producer-registration-task): will register with Nexus at [fd00:1122:3344:101::a]:12221 unit = producer-server
702026-08-14T22:29:01.592ZDEBGtfportd (producer-registration-task): client request body = Some(Body) method = POST unit = producer-server uri = http://[fd00:1122:3344:101::a]:12221/metrics/producers
712026-08-14T22:29:01.932ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
722026-08-14T22:29:01.936ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::1cdf:a5ff:fe60:2ff]
732026-08-14T22:29:01.940ZWARNtfportd: tfportqsfp0_0 has no addrs
742026-08-14T22:29:03.355ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
752026-08-14T22:29:03.360ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::1cdf:a5ff:fe60:2ff]
762026-08-14T22:29:03.364ZWARNtfportd: tfportqsfp0_0 has no addrs
772026-08-14T22:29:04.284ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
782026-08-14T22:29:04.289ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::1cdf:a5ff:fe60:2ff]
792026-08-14T22:29:04.293ZWARNtfportd: tfportqsfp0_0 has no addrs
802026-08-14T22:29:05.015ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
812026-08-14T22:29:05.019ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::1cdf:a5ff:fe60:2ff]
822026-08-14T22:29:05.022ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
832026-08-14T22:29:05.022ZDEBGtfportd (kstat-sampler): adding new target unit = oximeter
842026-08-14T22:29:05.023ZDEBGtfportd (kstat-sampler-worker): storing new creation time for tracked target creation_time = 2026-08-14T22:28:29.645188259Z path = KstatPath { module: "link", instance: 0, name: "tfportqsfp0_0" } unit = oximeter
852026-08-14T22:29:05.023ZDEBGtfportd (kstat-sampler-worker): inserted empty per-target sample queue id = 6774394927573844902 unit = oximeter
862026-08-14T22:29:05.023ZDEBGtfportd: added new link to kstat sampler link_kind = switch-port-control link_name = tfportqsfp0_0 unit = metrics-task-tracker
872026-08-14T22:29:05.186ZINFOtfportd: added new ARP entry entry = 10.151.1.174->8e:ae:23:d6:fb:76 interface = tfportqsfp0_0
882026-08-14T22:29:05.717ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
892026-08-14T22:29:05.722ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::1cdf:a5ff:fe60:2ff]
902026-08-14T22:29:05.724ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
912026-08-14T22:29:06.577ZDEBGtfportd (kstat-sampler-worker): pulled samples from target id = 17094535478246951415 n_samples = 6 unit = oximeter
922026-08-14T22:29:07.192ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
932026-08-14T22:29:07.197ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::1cdf:a5ff:fe60:2ff]
942026-08-14T22:29:07.201ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
952026-08-14T22:29:07.997ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
962026-08-14T22:29:08.002ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::1cdf:a5ff:fe60:2ff]
972026-08-14T22:29:08.005ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
982026-08-14T22:29:08.774ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
992026-08-14T22:29:08.779ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::1cdf:a5ff:fe60:2ff]
1002026-08-14T22:29:08.783ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1012026-08-14T22:29:09.953ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1022026-08-14T22:29:09.957ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::1cdf:a5ff:fe60:2ff]
1032026-08-14T22:29:09.961ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1042026-08-14T22:29:10.769ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1052026-08-14T22:29:10.774ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::1cdf:a5ff:fe60:2ff]
1062026-08-14T22:29:10.778ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1072026-08-14T22:29:12.258ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1082026-08-14T22:29:12.263ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::1cdf:a5ff:fe60:2ff]
1092026-08-14T22:29:12.266ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1102026-08-14T22:29:13.577ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1112026-08-14T22:29:13.582ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::1cdf:a5ff:fe60:2ff]
1122026-08-14T22:29:13.585ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1132026-08-14T22:29:14.103ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1142026-08-14T22:29:14.107ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::1cdf:a5ff:fe60:2ff]
1152026-08-14T22:29:14.110ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1162026-08-14T22:29:15.028ZDEBGtfportd (kstat-sampler-worker): pulled samples from target id = 6774394927573844902 n_samples = 6 unit = oximeter
1172026-08-14T22:29:15.419ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1182026-08-14T22:29:15.424ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::1cdf:a5ff:fe60:2ff]
1192026-08-14T22:29:15.427ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1202026-08-14T22:29:16.415ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1212026-08-14T22:29:16.422ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::1cdf:a5ff:fe60:2ff]
1222026-08-14T22:29:16.427ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1232026-08-14T22:29:16.586ZDEBGtfportd (kstat-sampler-worker): pulled samples from target id = 17094535478246951415 n_samples = 6 unit = oximeter
1242026-08-14T22:29:16.600ZDEBGtfportd (producer-registration-task): client response result = Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::a]:12221/metrics/producers", source: TimedOut }) unit = producer-server
1252026-08-14T22:29:16.602ZWARNtfportd (producer-registration-task): failed to register with Nexus, will retry count = 0 delay = 20.021755381s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers)" file = /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/8e30605/oximeter/producer/src/lib.rs:371 unit = producer-server
1262026-08-14T22:29:16.792ZDEBGtfportd (internal-dns-resolver): lookup_socket_v6 srv dns_name = _nexus._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("84ded4f1-c37e-4acd-b277-44c3a301227b.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("92be6118-aeb7-43c8-b50f-1195aaf134f5.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("d263ad7b-e87f-4142-8c0c-19548c9e8f70.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("d263ad7b-e87f-4142-8c0c-19548c9e8f70.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::c)) }], valid_until: Instant { tv_sec: 1265, tv_nsec: 358372843 } }) unit = producer-server
1272026-08-14T22:29:16.793ZDEBGtfportd (producer-registration-task): will register with Nexus at [fd00:1122:3344:101::a]:12221 unit = producer-server
1282026-08-14T22:29:16.804ZDEBGtfportd (producer-registration-task): client request body = Some(Body) method = POST unit = producer-server uri = http://[fd00:1122:3344:101::a]:12221/metrics/producers
1292026-08-14T22:29:17.126ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1302026-08-14T22:29:17.132ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::1cdf:a5ff:fe60:2ff]
1312026-08-14T22:29:17.136ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1322026-08-14T22:29:17.748ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1332026-08-14T22:29:17.759ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::1cdf:a5ff:fe60:2ff]
1342026-08-14T22:29:17.767ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1352026-08-14T22:29:19.042ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1362026-08-14T22:29:19.055ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::1cdf:a5ff:fe60:2ff]
1372026-08-14T22:29:19.058ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1382026-08-14T22:29:20.051ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1392026-08-14T22:29:20.058ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::1cdf:a5ff:fe60:2ff]
1402026-08-14T22:29:20.062ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1412026-08-14T22:29:21.443ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1422026-08-14T22:29:21.446ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::1cdf:a5ff:fe60:2ff]
1432026-08-14T22:29:21.458ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1442026-08-14T22:29:22.713ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1452026-08-14T22:29:22.717ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::1cdf:a5ff:fe60:2ff]
1462026-08-14T22:29:22.720ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1472026-08-14T22:29:23.467ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1482026-08-14T22:29:23.472ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::1cdf:a5ff:fe60:2ff]
1492026-08-14T22:29:23.475ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1502026-08-14T22:29:24.079ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1512026-08-14T22:29:24.085ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::1cdf:a5ff:fe60:2ff]
1522026-08-14T22:29:24.088ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1532026-08-14T22:29:24.624ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1542026-08-14T22:29:24.629ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::1cdf:a5ff:fe60:2ff]
1552026-08-14T22:29:24.633ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1562026-08-14T22:29:25.057ZDEBGtfportd (kstat-sampler-worker): pulled samples from target id = 6774394927573844902 n_samples = 6 unit = oximeter
1572026-08-14T22:29:25.493ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1582026-08-14T22:29:25.498ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::1cdf:a5ff:fe60:2ff]
1592026-08-14T22:29:25.501ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1602026-08-14T22:29:26.065ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1612026-08-14T22:29:26.072ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::1cdf:a5ff:fe60:2ff]
1622026-08-14T22:29:26.075ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1632026-08-14T22:29:26.610ZDEBGtfportd (kstat-sampler-worker): pulled samples from target id = 17094535478246951415 n_samples = 6 unit = oximeter
1642026-08-14T22:29:26.754ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1652026-08-14T22:29:26.759ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::1cdf:a5ff:fe60:2ff]
1662026-08-14T22:29:26.763ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1672026-08-14T22:29:27.478ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1682026-08-14T22:29:27.483ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::1cdf:a5ff:fe60:2ff]
1692026-08-14T22:29:27.486ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1702026-08-14T22:29:28.202ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1712026-08-14T22:29:28.207ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::1cdf:a5ff:fe60:2ff]
1722026-08-14T22:29:28.210ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1732026-08-14T22:29:29.221ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1742026-08-14T22:29:29.226ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::1cdf:a5ff:fe60:2ff]
1752026-08-14T22:29:29.230ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1762026-08-14T22:29:29.889ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1772026-08-14T22:29:29.895ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::1cdf:a5ff:fe60:2ff]
1782026-08-14T22:29:29.899ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1792026-08-14T22:29:30.562ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1802026-08-14T22:29:30.567ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::1cdf:a5ff:fe60:2ff]
1812026-08-14T22:29:30.570ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1822026-08-14T22:29:31.697ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1832026-08-14T22:29:31.703ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::1cdf:a5ff:fe60:2ff]
1842026-08-14T22:29:31.706ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1852026-08-14T22:29:31.826ZDEBGtfportd (producer-registration-task): client response result = Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::a]:12221/metrics/producers", source: TimedOut }) unit = producer-server
1862026-08-14T22:29:31.826ZWARNtfportd (producer-registration-task): failed to register with Nexus, will retry count = 1 delay = 35.225750369s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers)" file = /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/8e30605/oximeter/producer/src/lib.rs:371 unit = producer-server
1872026-08-14T22:29:32.489ZDEBGtfportd (internal-dns-resolver): lookup_socket_v6 srv dns_name = _nexus._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("84ded4f1-c37e-4acd-b277-44c3a301227b.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("92be6118-aeb7-43c8-b50f-1195aaf134f5.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("d263ad7b-e87f-4142-8c0c-19548c9e8f70.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("d263ad7b-e87f-4142-8c0c-19548c9e8f70.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::c)) }], valid_until: Instant { tv_sec: 1281, tv_nsec: 35160115 } }) unit = producer-server
1882026-08-14T22:29:32.490ZDEBGtfportd (producer-registration-task): will register with Nexus at [fd00:1122:3344:101::a]:12221 unit = producer-server
1892026-08-14T22:29:32.499ZDEBGtfportd (producer-registration-task): client request body = Some(Body) method = POST unit = producer-server uri = http://[fd00:1122:3344:101::a]:12221/metrics/producers
1902026-08-14T22:29:33.126ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1912026-08-14T22:29:33.132ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::1cdf:a5ff:fe60:2ff]
1922026-08-14T22:29:33.136ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1932026-08-14T22:29:34.549ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1942026-08-14T22:29:34.554ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::1cdf:a5ff:fe60:2ff]
1952026-08-14T22:29:34.557ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1962026-08-14T22:29:35.498ZDEBGtfportd (kstat-sampler-worker): pulled samples from target id = 6774394927573844902 n_samples = 6 unit = oximeter
1972026-08-14T22:29:35.992ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1982026-08-14T22:29:36.002ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::1cdf:a5ff:fe60:2ff]
1992026-08-14T22:29:36.015ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
2002026-08-14T22:29:36.612ZDEBGtfportd (kstat-sampler-worker): pulled samples from target id = 17094535478246951415 n_samples = 6 unit = oximeter
2012026-08-14T22:29:37.003ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2022026-08-14T22:29:37.009ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::1cdf:a5ff:fe60:2ff]
2032026-08-14T22:29:37.012ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
2042026-08-14T22:29:38.388ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2052026-08-14T22:29:38.392ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::1cdf:a5ff:fe60:2ff]
2062026-08-14T22:29:38.396ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
2072026-08-14T22:29:39.595ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2082026-08-14T22:29:39.600ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::1cdf:a5ff:fe60:2ff]
2092026-08-14T22:29:39.604ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
2102026-08-14T22:29:40.798ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2112026-08-14T22:29:40.803ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::1cdf:a5ff:fe60:2ff]
2122026-08-14T22:29:40.806ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
2132026-08-14T22:29:41.969ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2142026-08-14T22:29:41.974ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::1cdf:a5ff:fe60:2ff]
2152026-08-14T22:29:41.976ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
2162026-08-14T22:29:42.977ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2172026-08-14T22:29:42.985ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::1cdf:a5ff:fe60:2ff]
2182026-08-14T22:29:42.990ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
2192026-08-14T22:29:44.215ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2202026-08-14T22:29:44.220ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::1cdf:a5ff:fe60:2ff]
2212026-08-14T22:29:44.223ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
2222026-08-14T22:29:45.231ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2232026-08-14T22:29:45.237ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::1cdf:a5ff:fe60:2ff]
2242026-08-14T22:29:45.240ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
2252026-08-14T22:29:45.501ZDEBGtfportd (kstat-sampler-worker): pulled samples from target id = 6774394927573844902 n_samples = 6 unit = oximeter
2262026-08-14T22:29:46.614ZDEBGtfportd (kstat-sampler-worker): pulled samples from target id = 17094535478246951415 n_samples = 6 unit = oximeter
2272026-08-14T22:29:46.641ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2282026-08-14T22:29:46.646ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::1cdf:a5ff:fe60:2ff]
2292026-08-14T22:29:46.649ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
2302026-08-14T22:29:47.501ZDEBGtfportd (producer-registration-task): client response result = Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::a]:12221/metrics/producers", source: TimedOut }) unit = producer-server
2312026-08-14T22:29:47.502ZWARNtfportd (producer-registration-task): failed to register with Nexus, will retry count = 2 delay = 50.900248311s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers)" file = /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/8e30605/oximeter/producer/src/lib.rs:371 unit = producer-server
2322026-08-14T22:29:47.886ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2332026-08-14T22:29:47.891ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::1cdf:a5ff:fe60:2ff]
2342026-08-14T22:29:47.895ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
2352026-08-14T22:29:48.202ZDEBGtfportd (internal-dns-resolver): lookup_socket_v6 srv dns_name = _nexus._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("84ded4f1-c37e-4acd-b277-44c3a301227b.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("92be6118-aeb7-43c8-b50f-1195aaf134f5.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("d263ad7b-e87f-4142-8c0c-19548c9e8f70.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("d263ad7b-e87f-4142-8c0c-19548c9e8f70.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::c)) }], valid_until: Instant { tv_sec: 1296, tv_nsec: 747148635 } }) unit = producer-server
2362026-08-14T22:29:48.203ZDEBGtfportd (producer-registration-task): will register with Nexus at [fd00:1122:3344:101::a]:12221 unit = producer-server
2372026-08-14T22:29:48.209ZDEBGtfportd (producer-registration-task): client request body = Some(Body) method = POST unit = producer-server uri = http://[fd00:1122:3344:101::a]:12221/metrics/producers
2382026-08-14T22:29:49.068ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2392026-08-14T22:29:49.074ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::1cdf:a5ff:fe60:2ff]
2402026-08-14T22:29:49.078ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
2412026-08-14T22:29:49.791ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2422026-08-14T22:29:49.797ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::1cdf:a5ff:fe60:2ff]
2432026-08-14T22:29:49.800ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
2442026-08-14T22:29:50.693ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2452026-08-14T22:29:50.698ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::1cdf:a5ff:fe60:2ff]
2462026-08-14T22:29:50.701ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
2472026-08-14T22:29:51.828ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2482026-08-14T22:29:51.835ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::1cdf:a5ff:fe60:2ff]
2492026-08-14T22:29:51.838ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
2502026-08-14T22:29:52.772ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2512026-08-14T22:29:52.778ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::1cdf:a5ff:fe60:2ff]
2522026-08-14T22:29:52.782ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
2532026-08-14T22:29:53.755ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2542026-08-14T22:29:53.761ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::1cdf:a5ff:fe60:2ff]
2552026-08-14T22:29:53.764ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
2562026-08-14T22:29:54.907ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2572026-08-14T22:29:54.913ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::1cdf:a5ff:fe60:2ff]
2582026-08-14T22:29:54.916ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
2592026-08-14T22:29:55.503ZDEBGtfportd (kstat-sampler-worker): pulled samples from target id = 6774394927573844902 n_samples = 6 unit = oximeter
2602026-08-14T22:29:55.663ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2612026-08-14T22:29:55.669ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::1cdf:a5ff:fe60:2ff]
2622026-08-14T22:29:55.672ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
2632026-08-14T22:29:56.616ZDEBGtfportd (kstat-sampler-worker): pulled samples from target id = 17094535478246951415 n_samples = 6 unit = oximeter
2642026-08-14T22:29:56.795ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2652026-08-14T22:29:56.801ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::1cdf:a5ff:fe60:2ff]
2662026-08-14T22:29:56.805ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
2672026-08-14T22:29:57.567ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2682026-08-14T22:29:57.573ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::1cdf:a5ff:fe60:2ff]
2692026-08-14T22:29:57.576ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
2702026-08-14T22:29:58.375ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2712026-08-14T22:29:58.381ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::1cdf:a5ff:fe60:2ff]
2722026-08-14T22:29:58.385ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
2732026-08-14T22:29:59.610ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2742026-08-14T22:29:59.615ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::1cdf:a5ff:fe60:2ff]
2752026-08-14T22:29:59.618ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]