Max level shown:
1 [ May 28 06:13:51 Enabled. ]
2 [ May 28 06:13:51 Rereading configuration. ]
3 [ May 28 06:13:52 Rereading configuration. ]
4 [ May 28 06:14:03 Executing start method ("ctrun -l child -o noorphan,regent /opt/oxide/dendrite/bin/tfportd --sync-only &"). ]
5 [ May 28 06:14:03 Method "start" exited with status 0. ]
6 No non-localhost IPv6 addresses found in SMF properties
72026-05-28T06:14:03.180ZINFOtfportd: connecting to dpd host = [::1] port = 12224
82026-05-28T06:14:03.182ZINFOtfportd: connected to dpd running softnpu-187aee7d
92026-05-28T06:14:03.182ZINFOtfportd: running in sync-only mode
102026-05-28T06:14:03.184ZINFOtfportd: no non-localhost IPv6 listen address available, waiting for SMF update unit = oximeter
112026-05-28T06:14:03.184ZINFOtfportd: refreshing SMF config prior to waiting on signals unit = signal_handler
12 No non-localhost IPv6 addresses found in SMF properties
132026-05-28T06:14:03.190ZINFOtfportd: refreshed config: Config { log_file: None, log_format: Json, listen_addresses: [ [::1]:12231, ], pkt_source: None, dpd_host: "[::1]", dpd_port: 12224, vlan_link: None, vlan_data: None, techport0_prefix: None, techport1_prefix: None, rack_id: Some( e9a82416-1c50-4bf6-8091-5d5d8f77e37c, ), sled_id: Some( 5784b595-a1ec-42e3-b897-6d07461cd828, ), sled_model: Some( "i86pc", ), sled_revision: Some( 0, ), sled_serial: Some( "sock", ), }
142026-05-28T06:14:03.190ZINFOtfportd: no non-localhost IPv6 listen address available, waiting for SMF update unit = oximeter
152026-05-28T06:14:03.194ZINFOtfportd: added new NDP entry entry = tfportrear0_0:fe80::3c24:87ff:fe43:1f29->3e:24:87:43:1f:29 interface = tfportrear0_0
162026-05-28T06:14:03.195ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
172026-05-28T06:14:03.202ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::3c24:87ff:fe43:1f29]
182026-05-28T06:14:03.209ZWARNtfportd: tfportqsfp0_0 has no addrs
192026-05-28T06:14:03.849ZINFOtfportd: added new NDP entry entry = tfportrear0_0:fe80::1ca4:35ff:fe12:cfd6->1e:a4:35:12:cf:d6 interface = tfportrear0_0
20 [ May 28 06:14:03 Rereading configuration. ]
21 [ May 28 06:14:03 Executing refresh method (:kill). ]
22 [ May 28 06:14:03 Stopping because all processes in service exited. ]
23 [ May 28 06:14:03 Executing stop method (:kill). ]
24 [ May 28 06:14:03 Executing start method ("ctrun -l child -o noorphan,regent /opt/oxide/dendrite/bin/tfportd --sync-only &"). ]
25 [ May 28 06:14:03 Method "start" exited with status 0. ]
262026-05-28T06:14:03.924ZINFOtfportd: connecting to dpd host = [::1] port = 12224
272026-05-28T06:14:03.924ZINFOtfportd: connected to dpd running softnpu-187aee7d
282026-05-28T06:14:03.925ZINFOtfportd: running in sync-only mode
292026-05-28T06:14:03.925ZINFOtfportd: found suitable IPv6 address for oximeter producer server address = fd00:1122:3344:101::2 unit = oximeter
302026-05-28T06:14:03.927ZINFOtfportd: refreshing SMF config prior to waiting on signals unit = signal_handler
312026-05-28T06:14:03.927ZDEBGtfportd: extracted oximeter configuration data config = OximeterConfig { listen_address: fd00:1122:3344:101::2, sled_identifiers: SledIdentifiers { rack_id: e9a82416-1c50-4bf6-8091-5d5d8f77e37c, sled_id: 5784b595-a1ec-42e3-b897-6d07461cd828, model: "i86pc", revision: 0, serial: "sock" }, switch_identifiers: SwitchIdentifiers { asic_backend: "softnpu", fab: None, fuse: Some(FuseData { chip_rev: ChipRevision { device_id: 272, rev: "B1", rev_num: 2 }, disabled: DisabledFeatures { bsync: false, mau: [0, 0, 0, 0], pgen: false, pipes: 0, ports: 0, resub: false, speeds: 0, tm_mem: 0 }, frequency: FrequencySettings { bps: 3, bps_ext: 0, cpu_speed_dis: 0, disabled: false, pcie_dis: 0, pps: 3, pps_ext: 0 }, manufacturing: ManufacturingData { core_repair: false, die_rotation: false, pmro_and_skew: 0, silent_spin: 0, soft_pipe_dis: 0, tile_repair: false, voltage_scaling: 0, wf_core_repair: false }, part: PartInfo { part_num: 4660, pkg_id: 1, version: 2 } }), lot: None, lotnum: None, model: "FAKE_SIM_SIDECAR", revision: 0, serial: "SimSidecar0", sidecar_id: e894cd44-20ae-4a57-8471-238597cc1b74, slot: 0, wafer: None, wafer_loc: None } } unit = oximeter
322026-05-28T06:14:03.927ZDEBGtfportd: created producer ID producer_id = 56b85cc7-9d12-4342-bc77-13df5ea72020 unit = oximeter
332026-05-28T06:14:03.928ZINFOtfportd: refreshed config: Config { log_file: None, log_format: Json, listen_addresses: [ [fd00:1122:3344:101::2]:12231, [::1]:12231, ], pkt_source: None, dpd_host: "[::1]", dpd_port: 12224, vlan_link: None, vlan_data: None, techport0_prefix: None, techport1_prefix: None, rack_id: Some( e9a82416-1c50-4bf6-8091-5d5d8f77e37c, ), sled_id: Some( 5784b595-a1ec-42e3-b897-6d07461cd828, ), sled_model: Some( "i86pc", ), sled_revision: Some( 0, ), sled_serial: Some( "sock", ), }
342026-05-28T06:14:03.935ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
352026-05-28T06:14:03.936ZINFOtfportd: added new NDP entry entry = tfportrear0_0:fe80::3c24:87ff:fe43:1f29->3e:24:87:43:1f:29 interface = tfportrear0_0
362026-05-28T06:14:03.936ZINFOtfportd: added new NDP entry entry = tfportrear0_0:fe80::1ca4:35ff:fe12:cfd6->1e:a4:35:12:cf:d6 interface = tfportrear0_0
372026-05-28T06:14:03.938ZDEBGtfportd: registered DTrace probes unit = producer-server
382026-05-28T06:14:03.939ZDEBGtfportd (dropshot): registered endpoint local_addr = [fd00:1122:3344:101::2]:50901 method = GET path = /{producer_id} unit = producer-server versions = all
392026-05-28T06:14:03.939ZINFOtfportd (dropshot): listening file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.7/src/server.rs:328 local_addr = [fd00:1122:3344:101::2]:50901 unit = producer-server
402026-05-28T06:14:03.940ZDEBGtfportd (dropshot): successfully registered DTrace USDT probes local_addr = [fd00:1122:3344:101::2]:50901 unit = producer-server
412026-05-28T06:14:03.940ZDEBGtfportd: Requested any available port, Dropshot server has been bound to [fd00:1122:3344:101::2]:50901 unit = producer-server
422026-05-28T06:14:03.940ZDEBGtfportd: Nexus IP not provided, will use DNS to resolve it unit = producer-server
432026-05-28T06:14:03.940ZINFOtfportd (internal-dns-resolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53] file = /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/8e30605/internal-dns/resolver/src/resolver.rs:144 unit = producer-server
442026-05-28T06:14:03.940ZDEBGtfportd: starting producer registration task unit = producer-server
452026-05-28T06:14:03.940ZINFOtfportd: starting oximeter metric producer server address = [fd00:1122:3344:101::2]:50901 file = /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/8e30605/oximeter/producer/src/lib.rs:279 interval = 30s producer_id = 56b85cc7-9d12-4342-bc77-13df5ea72020 unit = producer-server
462026-05-28T06:14:03.940ZINFOtfportd: created oximeter producer server unit = oximeter
472026-05-28T06:14:03.940ZDEBGtfportd (producer-registration-task): registering as oximeter producer with Nexus registration_count = 0 unit = producer-server
482026-05-28T06:14:03.941ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::3c24:87ff:fe43:1f29]
492026-05-28T06:14:03.941ZDEBGtfportd (kstat-sampler): adding new target unit = oximeter
502026-05-28T06:14:03.943ZDEBGtfportd (kstat-sampler-worker): storing new creation time for tracked target creation_time = 2026-05-28T06:13:46.986431341Z path = KstatPath { module: "link", instance: 0, name: "tfportrear0_0" } unit = oximeter
512026-05-28T06:14:03.943ZDEBGtfportd (kstat-sampler-worker): inserted empty per-target sample queue id = 11718309190563589792 unit = oximeter
522026-05-28T06:14:03.943ZDEBGtfportd: added new link to kstat sampler link_kind = switch-port-control link_name = tfportrear0_0 unit = metrics-task-tracker
532026-05-28T06:14:03.944ZWARNtfportd: tfportqsfp0_0 has no addrs
542026-05-28T06:14:04.885ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
552026-05-28T06:14:04.890ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::3c24:87ff:fe43:1f29]
562026-05-28T06:14:04.894ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.2.174]
572026-05-28T06:14:04.894ZDEBGtfportd (kstat-sampler): adding new target unit = oximeter
582026-05-28T06:14:04.895ZDEBGtfportd (kstat-sampler-worker): storing new creation time for tracked target creation_time = 2026-05-28T06:13:46.986301318Z path = KstatPath { module: "link", instance: 0, name: "tfportqsfp0_0" } unit = oximeter
592026-05-28T06:14:04.895ZDEBGtfportd (kstat-sampler-worker): inserted empty per-target sample queue id = 9263541390947002991 unit = oximeter
602026-05-28T06:14:04.895ZDEBGtfportd: added new link to kstat sampler link_kind = switch-port-control link_name = tfportqsfp0_0 unit = metrics-task-tracker
612026-05-28T06:14:05.124ZINFOtfportd: added new ARP entry entry = 10.151.2.174->3e:cf:2d:a1:42:6a interface = tfportqsfp0_0
622026-05-28T06:14:06.263ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
632026-05-28T06:14:06.268ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::3c24:87ff:fe43:1f29]
642026-05-28T06:14:06.271ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.2.174]
652026-05-28T06:14:07.741ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
662026-05-28T06:14:07.746ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::3c24:87ff:fe43:1f29]
672026-05-28T06:14:07.749ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.2.174]
682026-05-28T06:14:08.488ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
692026-05-28T06:14:08.493ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::3c24:87ff:fe43:1f29]
702026-05-28T06:14:08.496ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.2.174]
712026-05-28T06:14:08.943ZDEBGtfportd (internal-dns-resolver): lookup_socket_v6 srv dns_name = _nexus._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("496a1328-91a1-4617-bc71-f725a21a25ac.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("c400a850-8d2a-44bc-bfa6-fc6b1772e974.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("d7d18418-33eb-4658-8c3b-9ab35f0e3764.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("d7d18418-33eb-4658-8c3b-9ab35f0e3764.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::a)) }], valid_until: Instant { tv_sec: 1981, tv_nsec: 965050236 } }) unit = producer-server
722026-05-28T06:14:08.943ZDEBGtfportd (producer-registration-task): will register with Nexus at [fd00:1122:3344:101::c]:12221 unit = producer-server
732026-05-28T06:14:08.960ZDEBGtfportd (producer-registration-task): client request body = Some(Body) method = POST unit = producer-server uri = http://[fd00:1122:3344:101::c]:12221/metrics/producers
742026-05-28T06:14:09.560ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
752026-05-28T06:14:09.565ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::3c24:87ff:fe43:1f29]
762026-05-28T06:14:09.569ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.2.174]
772026-05-28T06:14:10.679ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
782026-05-28T06:14:10.684ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::3c24:87ff:fe43:1f29]
792026-05-28T06:14:10.687ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.2.174]
802026-05-28T06:14:11.806ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
812026-05-28T06:14:11.810ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::3c24:87ff:fe43:1f29]
822026-05-28T06:14:11.814ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.2.174]
832026-05-28T06:14:12.552ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
842026-05-28T06:14:12.557ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::3c24:87ff:fe43:1f29]
852026-05-28T06:14:12.560ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.2.174]
862026-05-28T06:14:13.227ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
872026-05-28T06:14:13.232ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::3c24:87ff:fe43:1f29]
882026-05-28T06:14:13.235ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.2.174]
892026-05-28T06:14:13.947ZDEBGtfportd (kstat-sampler-worker): pulled samples from target id = 11718309190563589792 n_samples = 6 unit = oximeter
902026-05-28T06:14:14.093ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
912026-05-28T06:14:14.098ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::3c24:87ff:fe43:1f29]
922026-05-28T06:14:14.101ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.2.174]
932026-05-28T06:14:14.897ZDEBGtfportd (kstat-sampler-worker): pulled samples from target id = 9263541390947002991 n_samples = 6 unit = oximeter
942026-05-28T06:14:15.322ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
952026-05-28T06:14:15.327ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::3c24:87ff:fe43:1f29]
962026-05-28T06:14:15.331ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.2.174]
972026-05-28T06:14:16.363ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
982026-05-28T06:14:16.368ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::3c24:87ff:fe43:1f29]
992026-05-28T06:14:16.372ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.2.174]
1002026-05-28T06:14:17.575ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1012026-05-28T06:14:17.579ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::3c24:87ff:fe43:1f29]
1022026-05-28T06:14:17.582ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.2.174]
1032026-05-28T06:14:19.572ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1042026-05-28T06:14:19.577ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::3c24:87ff:fe43:1f29]
1052026-05-28T06:14:19.581ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.2.174]
1062026-05-28T06:14:20.904ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1072026-05-28T06:14:20.908ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::3c24:87ff:fe43:1f29]
1082026-05-28T06:14:20.910ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.2.174]
1092026-05-28T06:14:22.226ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1102026-05-28T06:14:22.231ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::3c24:87ff:fe43:1f29]
1112026-05-28T06:14:22.234ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.2.174]
1122026-05-28T06:14:22.753ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1132026-05-28T06:14:22.757ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::3c24:87ff:fe43:1f29]
1142026-05-28T06:14:22.760ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.2.174]
1152026-05-28T06:14:23.504ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1162026-05-28T06:14:23.508ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::3c24:87ff:fe43:1f29]
1172026-05-28T06:14:23.512ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.2.174]
1182026-05-28T06:14:24.623ZDEBGtfportd (kstat-sampler-worker): pulled samples from target id = 11718309190563589792 n_samples = 6 unit = oximeter
1192026-05-28T06:14:24.635ZDEBGtfportd (producer-registration-task): client response result = Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::c]:12221/metrics/producers", source: TimedOut }) unit = producer-server
1202026-05-28T06:14:24.636ZWARNtfportd (producer-registration-task): failed to register with Nexus, will retry count = 0 delay = 20.022441519s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers)" file = /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/8e30605/oximeter/producer/src/lib.rs:371 unit = producer-server
1212026-05-28T06:14:24.859ZDEBGtfportd (internal-dns-resolver): lookup_socket_v6 srv dns_name = _nexus._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("496a1328-91a1-4617-bc71-f725a21a25ac.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("c400a850-8d2a-44bc-bfa6-fc6b1772e974.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("d7d18418-33eb-4658-8c3b-9ab35f0e3764.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("d7d18418-33eb-4658-8c3b-9ab35f0e3764.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::a)) }], valid_until: Instant { tv_sec: 1997, tv_nsec: 208223046 } }) unit = producer-server
1222026-05-28T06:14:24.859ZDEBGtfportd (producer-registration-task): will register with Nexus at [fd00:1122:3344:101::c]:12221 unit = producer-server
1232026-05-28T06:14:24.867ZDEBGtfportd (producer-registration-task): client request body = Some(Body) method = POST unit = producer-server uri = http://[fd00:1122:3344:101::c]:12221/metrics/producers
1242026-05-28T06:14:24.961ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1252026-05-28T06:14:24.966ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::3c24:87ff:fe43:1f29]
1262026-05-28T06:14:24.969ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.2.174]
1272026-05-28T06:14:25.570ZDEBGtfportd (kstat-sampler-worker): pulled samples from target id = 9263541390947002991 n_samples = 6 unit = oximeter
1282026-05-28T06:14:25.607ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1292026-05-28T06:14:25.611ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::3c24:87ff:fe43:1f29]
1302026-05-28T06:14:25.615ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.2.174]
1312026-05-28T06:14:26.807ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1322026-05-28T06:14:26.811ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::3c24:87ff:fe43:1f29]
1332026-05-28T06:14:26.814ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.2.174]
1342026-05-28T06:14:27.661ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1352026-05-28T06:14:27.666ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::3c24:87ff:fe43:1f29]
1362026-05-28T06:14:27.669ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.2.174]
1372026-05-28T06:14:28.587ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1382026-05-28T06:14:28.591ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::3c24:87ff:fe43:1f29]
1392026-05-28T06:14:28.595ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.2.174]
1402026-05-28T06:14:29.634ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1412026-05-28T06:14:29.640ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::3c24:87ff:fe43:1f29]
1422026-05-28T06:14:29.643ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.2.174]
1432026-05-28T06:14:30.825ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1442026-05-28T06:14:30.831ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::3c24:87ff:fe43:1f29]
1452026-05-28T06:14:30.834ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.2.174]
1462026-05-28T06:14:31.556ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1472026-05-28T06:14:31.562ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::3c24:87ff:fe43:1f29]
1482026-05-28T06:14:31.566ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.2.174]
1492026-05-28T06:14:32.142ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1502026-05-28T06:14:32.148ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::3c24:87ff:fe43:1f29]
1512026-05-28T06:14:32.152ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.2.174]
1522026-05-28T06:14:32.892ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1532026-05-28T06:14:32.898ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::3c24:87ff:fe43:1f29]
1542026-05-28T06:14:32.901ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.2.174]
1552026-05-28T06:14:34.069ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1562026-05-28T06:14:34.074ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::3c24:87ff:fe43:1f29]
1572026-05-28T06:14:34.078ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.2.174]
1582026-05-28T06:14:34.612ZDEBGtfportd (kstat-sampler-worker): pulled samples from target id = 11718309190563589792 n_samples = 6 unit = oximeter
1592026-05-28T06:14:34.727ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1602026-05-28T06:14:34.735ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::3c24:87ff:fe43:1f29]
1612026-05-28T06:14:34.739ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.2.174]
1622026-05-28T06:14:35.561ZDEBGtfportd (kstat-sampler-worker): pulled samples from target id = 9263541390947002991 n_samples = 6 unit = oximeter
1632026-05-28T06:14:35.685ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1642026-05-28T06:14:35.692ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::3c24:87ff:fe43:1f29]
1652026-05-28T06:14:35.697ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.2.174]
1662026-05-28T06:14:36.244ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1672026-05-28T06:14:36.251ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::3c24:87ff:fe43:1f29]
1682026-05-28T06:14:36.255ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.2.174]
1692026-05-28T06:14:36.863ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1702026-05-28T06:14:36.867ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::3c24:87ff:fe43:1f29]
1712026-05-28T06:14:36.870ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.2.174]
1722026-05-28T06:14:37.845ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1732026-05-28T06:14:37.850ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::3c24:87ff:fe43:1f29]
1742026-05-28T06:14:37.853ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.2.174]
1752026-05-28T06:14:39.059ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1762026-05-28T06:14:39.065ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::3c24:87ff:fe43:1f29]
1772026-05-28T06:14:39.069ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.2.174]
1782026-05-28T06:14:39.848ZDEBGtfportd (producer-registration-task): client response result = Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::c]:12221/metrics/producers", source: TimedOut }) unit = producer-server
1792026-05-28T06:14:39.849ZWARNtfportd (producer-registration-task): failed to register with Nexus, will retry count = 1 delay = 35.255455176s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers)" file = /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/8e30605/oximeter/producer/src/lib.rs:371 unit = producer-server
1802026-05-28T06:14:39.865ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1812026-05-28T06:14:39.871ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::3c24:87ff:fe43:1f29]
1822026-05-28T06:14:39.874ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.2.174]
1832026-05-28T06:14:40.410ZDEBGtfportd (internal-dns-resolver): lookup_socket_v6 srv dns_name = _nexus._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("496a1328-91a1-4617-bc71-f725a21a25ac.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("c400a850-8d2a-44bc-bfa6-fc6b1772e974.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("d7d18418-33eb-4658-8c3b-9ab35f0e3764.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("d7d18418-33eb-4658-8c3b-9ab35f0e3764.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::a)) }], valid_until: Instant { tv_sec: 2012, tv_nsec: 778922977 } }) unit = producer-server
1842026-05-28T06:14:40.411ZDEBGtfportd (producer-registration-task): will register with Nexus at [fd00:1122:3344:101::c]:12221 unit = producer-server
1852026-05-28T06:14:40.419ZDEBGtfportd (producer-registration-task): client request body = Some(Body) method = POST unit = producer-server uri = http://[fd00:1122:3344:101::c]:12221/metrics/producers
1862026-05-28T06:14:40.823ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1872026-05-28T06:14:40.829ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::3c24:87ff:fe43:1f29]
1882026-05-28T06:14:40.832ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.2.174]
1892026-05-28T06:14:42.049ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1902026-05-28T06:14:42.056ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::3c24:87ff:fe43:1f29]
1912026-05-28T06:14:42.060ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.2.174]
1922026-05-28T06:14:42.930ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1932026-05-28T06:14:42.936ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::3c24:87ff:fe43:1f29]
1942026-05-28T06:14:42.939ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.2.174]
1952026-05-28T06:14:43.862ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1962026-05-28T06:14:43.864ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::3c24:87ff:fe43:1f29]
1972026-05-28T06:14:43.867ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.2.174]
1982026-05-28T06:14:44.613ZDEBGtfportd (kstat-sampler-worker): pulled samples from target id = 11718309190563589792 n_samples = 6 unit = oximeter
1992026-05-28T06:14:44.670ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2002026-05-28T06:14:44.676ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::3c24:87ff:fe43:1f29]
2012026-05-28T06:14:44.680ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.2.174]
2022026-05-28T06:14:45.561ZDEBGtfportd (kstat-sampler-worker): pulled samples from target id = 9263541390947002991 n_samples = 6 unit = oximeter
2032026-05-28T06:14:45.715ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2042026-05-28T06:14:45.721ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::3c24:87ff:fe43:1f29]
2052026-05-28T06:14:45.724ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.2.174]
2062026-05-28T06:14:46.727ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2072026-05-28T06:14:46.732ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::3c24:87ff:fe43:1f29]
2082026-05-28T06:14:46.735ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.2.174]
2092026-05-28T06:14:47.287ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2102026-05-28T06:14:47.292ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::3c24:87ff:fe43:1f29]
2112026-05-28T06:14:47.295ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.2.174]
2122026-05-28T06:14:48.318ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2132026-05-28T06:14:48.324ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::3c24:87ff:fe43:1f29]
2142026-05-28T06:14:48.327ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.2.174]
2152026-05-28T06:14:49.300ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2162026-05-28T06:14:49.305ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::3c24:87ff:fe43:1f29]
2172026-05-28T06:14:49.308ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.2.174]
2182026-05-28T06:14:50.248ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2192026-05-28T06:14:50.253ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::3c24:87ff:fe43:1f29]
2202026-05-28T06:14:50.256ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.2.174]
2212026-05-28T06:14:51.447ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2222026-05-28T06:14:51.451ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::3c24:87ff:fe43:1f29]
2232026-05-28T06:14:51.455ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.2.174]
2242026-05-28T06:14:52.485ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2252026-05-28T06:14:52.492ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::3c24:87ff:fe43:1f29]
2262026-05-28T06:14:52.568ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.2.174]
2272026-05-28T06:14:53.208ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2282026-05-28T06:14:53.215ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::3c24:87ff:fe43:1f29]
2292026-05-28T06:14:53.220ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.2.174]
2302026-05-28T06:14:54.056ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2312026-05-28T06:14:54.065ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::3c24:87ff:fe43:1f29]
2322026-05-28T06:14:54.069ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.2.174]
2332026-05-28T06:14:54.615ZDEBGtfportd (kstat-sampler-worker): pulled samples from target id = 11718309190563589792 n_samples = 6 unit = oximeter
2342026-05-28T06:14:55.023ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2352026-05-28T06:14:55.027ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::3c24:87ff:fe43:1f29]
2362026-05-28T06:14:55.031ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.2.174]
2372026-05-28T06:14:55.433ZDEBGtfportd (producer-registration-task): client response result = Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::c]:12221/metrics/producers", source: TimedOut }) unit = producer-server
2382026-05-28T06:14:55.433ZWARNtfportd (producer-registration-task): failed to register with Nexus, will retry count = 2 delay = 50.827280734s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers)" file = /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/8e30605/oximeter/producer/src/lib.rs:371 unit = producer-server
2392026-05-28T06:14:55.563ZDEBGtfportd (kstat-sampler-worker): pulled samples from target id = 9263541390947002991 n_samples = 6 unit = oximeter
2402026-05-28T06:14:55.805ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2412026-05-28T06:14:55.813ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::3c24:87ff:fe43:1f29]
2422026-05-28T06:14:55.817ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.2.174]
2432026-05-28T06:14:56.288ZDEBGtfportd (internal-dns-resolver): lookup_socket_v6 srv dns_name = _nexus._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("496a1328-91a1-4617-bc71-f725a21a25ac.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("c400a850-8d2a-44bc-bfa6-fc6b1772e974.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("d7d18418-33eb-4658-8c3b-9ab35f0e3764.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("d7d18418-33eb-4658-8c3b-9ab35f0e3764.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::a)) }], valid_until: Instant { tv_sec: 2028, tv_nsec: 654414336 } }) unit = producer-server
2442026-05-28T06:14:56.288ZDEBGtfportd (producer-registration-task): will register with Nexus at [fd00:1122:3344:101::c]:12221 unit = producer-server
2452026-05-28T06:14:56.296ZDEBGtfportd (producer-registration-task): client request body = Some(Body) method = POST unit = producer-server uri = http://[fd00:1122:3344:101::c]:12221/metrics/producers
2462026-05-28T06:14:57.056ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2472026-05-28T06:14:57.061ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::3c24:87ff:fe43:1f29]
2482026-05-28T06:14:57.064ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.2.174]
2492026-05-28T06:14:57.701ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2502026-05-28T06:14:57.710ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::3c24:87ff:fe43:1f29]
2512026-05-28T06:14:57.713ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.2.174]
2522026-05-28T06:14:58.797ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2532026-05-28T06:14:58.804ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::3c24:87ff:fe43:1f29]
2542026-05-28T06:14:58.808ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.2.174]
2552026-05-28T06:15:00.225ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2562026-05-28T06:15:00.236ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::3c24:87ff:fe43:1f29]
2572026-05-28T06:15:00.243ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.2.174]
2582026-05-28T06:15:01.045ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2592026-05-28T06:15:01.066ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::3c24:87ff:fe43:1f29]
2602026-05-28T06:15:01.082ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.2.174]
2612026-05-28T06:15:02.255ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2622026-05-28T06:15:02.265ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::3c24:87ff:fe43:1f29]
2632026-05-28T06:15:02.271ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.2.174]