Max level shown:
1 [ Mar 17 02:43:46 Enabled. ]
2 [ Mar 17 02:43:46 Rereading configuration. ]
3 [ Mar 17 02:43:47 Rereading configuration. ]
4 [ Mar 17 02:43:58 Executing start method ("ctrun -l child -o noorphan,regent /opt/oxide/dendrite/bin/tfportd --sync-only &"). ]
5 [ Mar 17 02:43:58 Method "start" exited with status 0. ]
6 No non-localhost IPv6 addresses found in SMF properties
72026-03-17T02:43:58.608ZINFOtfportd: connecting to dpd host = [::1] port = 12224
82026-03-17T02:43:58.610ZINFOtfportd: connected to dpd running softnpu-e8a68118
92026-03-17T02:43:58.610ZINFOtfportd: running in sync-only mode
102026-03-17T02:43:58.611ZINFOtfportd: no non-localhost IPv6 listen address available, waiting for SMF update unit = oximeter
112026-03-17T02:43:58.613ZINFOtfportd: refreshing SMF config prior to waiting on signals unit = signal_handler
12 No non-localhost IPv6 addresses found in SMF properties
132026-03-17T02:43:58.620ZINFOtfportd: refreshed config: Config { log_file: None, log_format: Json, listen_addresses: [ [::1]:12231, ], pkt_source: None, dpd_host: "[::1]", dpd_port: 12224, vlan_link: None, vlan_data: None, techport0_prefix: None, techport1_prefix: None, rack_id: Some( 222ad483-1a6b-4e9c-af48-ac654ace7e29, ), sled_id: Some( d621afbb-91ac-4d0e-a0a2-8169d345c16e, ), sled_model: Some( "i86pc", ), sled_revision: Some( 0, ), sled_serial: Some( "buskin", ), }
142026-03-17T02:43:58.620ZINFOtfportd: no non-localhost IPv6 listen address available, waiting for SMF update unit = oximeter
152026-03-17T02:43:58.624ZINFOtfportd: added new NDP entry entry = tfportrear0_0:fe80::8cff:c9ff:fe76:d174->8e:ff:c9:76:d1:74 interface = tfportrear0_0
162026-03-17T02:43:58.628ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
172026-03-17T02:43:58.633ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::8cff:c9ff:fe76:d174]
182026-03-17T02:43:58.638ZWARNtfportd: tfportqsfp0_0 has no addrs
19 [ Mar 17 02:43:59 Rereading configuration. ]
20 [ Mar 17 02:43:59 Executing refresh method (:kill). ]
21 [ Mar 17 02:43:59 Stopping because all processes in service exited. ]
22 [ Mar 17 02:43:59 Executing stop method (:kill). ]
23 [ Mar 17 02:43:59 Executing start method ("ctrun -l child -o noorphan,regent /opt/oxide/dendrite/bin/tfportd --sync-only &"). ]
24 [ Mar 17 02:43:59 Method "start" exited with status 0. ]
252026-03-17T02:43:59.220ZINFOtfportd: connecting to dpd host = [::1] port = 12224
262026-03-17T02:43:59.221ZINFOtfportd: connected to dpd running softnpu-e8a68118
272026-03-17T02:43:59.222ZINFOtfportd: running in sync-only mode
282026-03-17T02:43:59.222ZINFOtfportd: found suitable IPv6 address for oximeter producer server address = fd00:1122:3344:101::2 unit = oximeter
292026-03-17T02:43:59.224ZINFOtfportd: refreshing SMF config prior to waiting on signals unit = signal_handler
302026-03-17T02:43:59.224ZDEBGtfportd: extracted oximeter configuration data config = OximeterConfig { listen_address: fd00:1122:3344:101::2, sled_identifiers: SledIdentifiers { rack_id: 222ad483-1a6b-4e9c-af48-ac654ace7e29, sled_id: d621afbb-91ac-4d0e-a0a2-8169d345c16e, model: "i86pc", revision: 0, serial: "buskin" }, switch_identifiers: SwitchIdentifiers { asic_backend: "softnpu", fab: None, fuse: Some(FuseData { chip_rev: ChipRevision { device_id: 272, rev: "B1", rev_num: 2 }, disabled: DisabledFeatures { bsync: false, mau: [0, 0, 0, 0], pgen: false, pipes: 0, ports: 0, resub: false, speeds: 0, tm_mem: 0 }, frequency: FrequencySettings { bps: 3, bps_ext: 0, cpu_speed_dis: 0, disabled: false, pcie_dis: 0, pps: 3, pps_ext: 0 }, manufacturing: ManufacturingData { core_repair: false, die_rotation: false, pmro_and_skew: 0, silent_spin: 0, soft_pipe_dis: 0, tile_repair: false, voltage_scaling: 0, wf_core_repair: false }, part: PartInfo { part_num: 4660, pkg_id: 1, version: 2 } }), lot: None, lotnum: None, model: "FAKE_SIM_SIDECAR", revision: 0, serial: "SimSidecar0", sidecar_id: 1b5e78b3-e252-438b-bc68-79205c5b9e2d, slot: 0, wafer: None, wafer_loc: None } } unit = oximeter
312026-03-17T02:43:59.224ZDEBGtfportd: created producer ID producer_id = ef55038b-6995-4019-b382-542cc5d0216a unit = oximeter
322026-03-17T02:43:59.225ZINFOtfportd: refreshed config: Config { log_file: None, log_format: Json, listen_addresses: [ [fd00:1122:3344:101::2]:12231, [::1]:12231, ], pkt_source: None, dpd_host: "[::1]", dpd_port: 12224, vlan_link: None, vlan_data: None, techport0_prefix: None, techport1_prefix: None, rack_id: Some( 222ad483-1a6b-4e9c-af48-ac654ace7e29, ), sled_id: Some( d621afbb-91ac-4d0e-a0a2-8169d345c16e, ), sled_model: Some( "i86pc", ), sled_revision: Some( 0, ), sled_serial: Some( "buskin", ), }
332026-03-17T02:43:59.232ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
342026-03-17T02:43:59.234ZINFOtfportd: added new NDP entry entry = tfportrear0_0:fe80::98d5:8cff:fea4:68e8->9a:d5:8c:a4:68:e8 interface = tfportrear0_0
352026-03-17T02:43:59.234ZINFOtfportd: added new NDP entry entry = tfportrear0_0:fe80::8cff:c9ff:fe76:d174->8e:ff:c9:76:d1:74 interface = tfportrear0_0
362026-03-17T02:43:59.235ZDEBGtfportd: registered DTrace probes unit = producer-server
372026-03-17T02:43:59.236ZDEBGtfportd (dropshot): registered endpoint local_addr = [fd00:1122:3344:101::2]:55275 method = GET path = /{producer_id} unit = producer-server versions = all
382026-03-17T02:43:59.236ZINFOtfportd (dropshot): listening file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.7/src/server.rs:328 local_addr = [fd00:1122:3344:101::2]:55275 unit = producer-server
392026-03-17T02:43:59.236ZDEBGtfportd (dropshot): successfully registered DTrace USDT probes local_addr = [fd00:1122:3344:101::2]:55275 unit = producer-server
402026-03-17T02:43:59.237ZDEBGtfportd: Requested any available port, Dropshot server has been bound to [fd00:1122:3344:101::2]:55275 unit = producer-server
412026-03-17T02:43:59.237ZDEBGtfportd: Nexus IP not provided, will use DNS to resolve it unit = producer-server
422026-03-17T02:43:59.237ZINFOtfportd (internal-dns-resolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53] file = /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/8e30605/internal-dns/resolver/src/resolver.rs:144 unit = producer-server
432026-03-17T02:43:59.237ZDEBGtfportd: starting producer registration task unit = producer-server
442026-03-17T02:43:59.237ZINFOtfportd: starting oximeter metric producer server address = [fd00:1122:3344:101::2]:55275 file = /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/8e30605/oximeter/producer/src/lib.rs:279 interval = 30s producer_id = ef55038b-6995-4019-b382-542cc5d0216a unit = producer-server
452026-03-17T02:43:59.237ZINFOtfportd: created oximeter producer server unit = oximeter
462026-03-17T02:43:59.237ZDEBGtfportd (producer-registration-task): registering as oximeter producer with Nexus registration_count = 0 unit = producer-server
472026-03-17T02:43:59.238ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::8cff:c9ff:fe76:d174]
482026-03-17T02:43:59.238ZDEBGtfportd (kstat-sampler): adding new target unit = oximeter
492026-03-17T02:43:59.240ZDEBGtfportd (kstat-sampler-worker): storing new creation time for tracked target creation_time = 2026-03-17T02:43:41.806089306Z path = KstatPath { module: "link", instance: 0, name: "tfportrear0_0" } unit = oximeter
502026-03-17T02:43:59.240ZDEBGtfportd (kstat-sampler-worker): inserted empty per-target sample queue id = 2566617268062380145 unit = oximeter
512026-03-17T02:43:59.240ZDEBGtfportd: added new link to kstat sampler link_kind = switch-port-control link_name = tfportrear0_0 unit = metrics-task-tracker
522026-03-17T02:43:59.241ZWARNtfportd: tfportqsfp0_0 has no addrs
532026-03-17T02:43:59.799ZINFOtfportd: added new ARP entry entry = 10.151.1.174->5a:0e:03:19:2b:8b interface = tfportqsfp0_0
542026-03-17T02:44:00.461ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
552026-03-17T02:44:00.466ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::8cff:c9ff:fe76:d174]
562026-03-17T02:44:00.470ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
572026-03-17T02:44:00.470ZDEBGtfportd (kstat-sampler): adding new target unit = oximeter
582026-03-17T02:44:00.471ZDEBGtfportd (kstat-sampler-worker): storing new creation time for tracked target creation_time = 2026-03-17T02:43:41.805952373Z path = KstatPath { module: "link", instance: 0, name: "tfportqsfp0_0" } unit = oximeter
592026-03-17T02:44:00.471ZDEBGtfportd (kstat-sampler-worker): inserted empty per-target sample queue id = 15719734224641383838 unit = oximeter
602026-03-17T02:44:00.471ZDEBGtfportd: added new link to kstat sampler link_kind = switch-port-control link_name = tfportqsfp0_0 unit = metrics-task-tracker
612026-03-17T02:44:01.659ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
622026-03-17T02:44:01.663ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::8cff:c9ff:fe76:d174]
632026-03-17T02:44:01.666ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
642026-03-17T02:44:02.190ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
652026-03-17T02:44:02.195ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::8cff:c9ff:fe76:d174]
662026-03-17T02:44:02.198ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
672026-03-17T02:44:02.750ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
682026-03-17T02:44:02.756ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::8cff:c9ff:fe76:d174]
692026-03-17T02:44:02.760ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
702026-03-17T02:44:03.601ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
712026-03-17T02:44:03.606ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::8cff:c9ff:fe76:d174]
722026-03-17T02:44:03.608ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
732026-03-17T02:44:04.240ZDEBGtfportd (internal-dns-resolver): lookup_socket_v6 srv dns_name = _nexus._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("04a4929c-ffe0-4c98-aafd-ebc61a0ed6db.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("7978df5c-2638-45eb-b0a8-5cf40fd0fccb.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("992a28b5-4983-458f-b27d-d85decb98fc4.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("992a28b5-4983-458f-b27d-d85decb98fc4.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::b)) }], valid_until: Instant { tv_sec: 6514, tv_nsec: 342917230 } }) unit = producer-server
742026-03-17T02:44:04.241ZDEBGtfportd (producer-registration-task): will register with Nexus at [fd00:1122:3344:101::c]:12221 unit = producer-server
752026-03-17T02:44:04.284ZDEBGtfportd (producer-registration-task): client request body = Some(Body) method = POST unit = producer-server uri = http://[fd00:1122:3344:101::c]:12221/metrics/producers
762026-03-17T02:44:04.811ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
772026-03-17T02:44:04.815ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::8cff:c9ff:fe76:d174]
782026-03-17T02:44:04.818ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
792026-03-17T02:44:06.267ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
802026-03-17T02:44:06.272ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::8cff:c9ff:fe76:d174]
812026-03-17T02:44:06.274ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
822026-03-17T02:44:07.381ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
832026-03-17T02:44:07.385ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::8cff:c9ff:fe76:d174]
842026-03-17T02:44:07.387ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
852026-03-17T02:44:08.290ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
862026-03-17T02:44:08.294ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::8cff:c9ff:fe76:d174]
872026-03-17T02:44:08.297ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
882026-03-17T02:44:09.150ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
892026-03-17T02:44:09.154ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::8cff:c9ff:fe76:d174]
902026-03-17T02:44:09.158ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
912026-03-17T02:44:09.245ZDEBGtfportd (kstat-sampler-worker): pulled samples from target id = 2566617268062380145 n_samples = 6 unit = oximeter
922026-03-17T02:44:10.044ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
932026-03-17T02:44:10.049ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::8cff:c9ff:fe76:d174]
942026-03-17T02:44:10.052ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
952026-03-17T02:44:10.474ZDEBGtfportd (kstat-sampler-worker): pulled samples from target id = 15719734224641383838 n_samples = 6 unit = oximeter
962026-03-17T02:44:11.233ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
972026-03-17T02:44:11.239ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::8cff:c9ff:fe76:d174]
982026-03-17T02:44:11.242ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
992026-03-17T02:44:11.937ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1002026-03-17T02:44:11.941ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::8cff:c9ff:fe76:d174]
1012026-03-17T02:44:11.944ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1022026-03-17T02:44:12.504ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1032026-03-17T02:44:12.509ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::8cff:c9ff:fe76:d174]
1042026-03-17T02:44:12.512ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1052026-03-17T02:44:13.528ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1062026-03-17T02:44:13.532ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::8cff:c9ff:fe76:d174]
1072026-03-17T02:44:13.536ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1082026-03-17T02:44:14.398ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1092026-03-17T02:44:14.402ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::8cff:c9ff:fe76:d174]
1102026-03-17T02:44:14.405ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1112026-03-17T02:44:15.419ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1122026-03-17T02:44:15.424ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::8cff:c9ff:fe76:d174]
1132026-03-17T02:44:15.426ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1142026-03-17T02:44:16.281ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1152026-03-17T02:44:16.286ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::8cff:c9ff:fe76:d174]
1162026-03-17T02:44:16.288ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1172026-03-17T02:44:17.581ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1182026-03-17T02:44:17.585ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::8cff:c9ff:fe76:d174]
1192026-03-17T02:44:17.587ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1202026-03-17T02:44:19.001ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1212026-03-17T02:44:19.005ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::8cff:c9ff:fe76:d174]
1222026-03-17T02:44:19.008ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1232026-03-17T02:44:19.819ZDEBGtfportd (kstat-sampler-worker): pulled samples from target id = 2566617268062380145 n_samples = 6 unit = oximeter
1242026-03-17T02:44:19.857ZDEBGtfportd (producer-registration-task): client response result = Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::c]:12221/metrics/producers", source: TimedOut }) unit = producer-server
1252026-03-17T02:44:19.858ZWARNtfportd (producer-registration-task): failed to register with Nexus, will retry count = 0 delay = 20.049704031s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers)" file = /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/8e30605/oximeter/producer/src/lib.rs:371 unit = producer-server
1262026-03-17T02:44:20.028ZDEBGtfportd (internal-dns-resolver): lookup_socket_v6 srv dns_name = _nexus._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("04a4929c-ffe0-4c98-aafd-ebc61a0ed6db.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("7978df5c-2638-45eb-b0a8-5cf40fd0fccb.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("992a28b5-4983-458f-b27d-d85decb98fc4.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("992a28b5-4983-458f-b27d-d85decb98fc4.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::b)) }], valid_until: Instant { tv_sec: 6529, tv_nsec: 559126683 } }) unit = producer-server
1272026-03-17T02:44:20.029ZDEBGtfportd (producer-registration-task): will register with Nexus at [fd00:1122:3344:101::c]:12221 unit = producer-server
1282026-03-17T02:44:20.062ZDEBGtfportd (producer-registration-task): client request body = Some(Body) method = POST unit = producer-server uri = http://[fd00:1122:3344:101::c]:12221/metrics/producers
1292026-03-17T02:44:20.231ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1302026-03-17T02:44:20.236ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::8cff:c9ff:fe76:d174]
1312026-03-17T02:44:20.239ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1322026-03-17T02:44:21.043ZDEBGtfportd (kstat-sampler-worker): pulled samples from target id = 15719734224641383838 n_samples = 6 unit = oximeter
1332026-03-17T02:44:21.145ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1342026-03-17T02:44:21.149ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::8cff:c9ff:fe76:d174]
1352026-03-17T02:44:21.152ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1362026-03-17T02:44:22.239ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1372026-03-17T02:44:22.243ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::8cff:c9ff:fe76:d174]
1382026-03-17T02:44:22.246ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1392026-03-17T02:44:23.402ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1402026-03-17T02:44:23.406ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::8cff:c9ff:fe76:d174]
1412026-03-17T02:44:23.409ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1422026-03-17T02:44:24.807ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1432026-03-17T02:44:24.812ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::8cff:c9ff:fe76:d174]
1442026-03-17T02:44:24.815ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1452026-03-17T02:44:25.756ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1462026-03-17T02:44:25.760ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::8cff:c9ff:fe76:d174]
1472026-03-17T02:44:25.763ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1482026-03-17T02:44:26.359ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1492026-03-17T02:44:26.363ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::8cff:c9ff:fe76:d174]
1502026-03-17T02:44:26.366ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1512026-03-17T02:44:27.716ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1522026-03-17T02:44:27.721ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::8cff:c9ff:fe76:d174]
1532026-03-17T02:44:27.724ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1542026-03-17T02:44:28.412ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1552026-03-17T02:44:28.416ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::8cff:c9ff:fe76:d174]
1562026-03-17T02:44:28.419ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1572026-03-17T02:44:29.538ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1582026-03-17T02:44:29.542ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::8cff:c9ff:fe76:d174]
1592026-03-17T02:44:29.546ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1602026-03-17T02:44:29.794ZDEBGtfportd (kstat-sampler-worker): pulled samples from target id = 2566617268062380145 n_samples = 6 unit = oximeter
1612026-03-17T02:44:30.224ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1622026-03-17T02:44:30.228ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::8cff:c9ff:fe76:d174]
1632026-03-17T02:44:30.231ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1642026-03-17T02:44:30.778ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1652026-03-17T02:44:30.782ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::8cff:c9ff:fe76:d174]
1662026-03-17T02:44:30.785ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1672026-03-17T02:44:31.017ZDEBGtfportd (kstat-sampler-worker): pulled samples from target id = 15719734224641383838 n_samples = 6 unit = oximeter
1682026-03-17T02:44:32.071ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1692026-03-17T02:44:32.075ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::8cff:c9ff:fe76:d174]
1702026-03-17T02:44:32.078ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1712026-03-17T02:44:33.163ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1722026-03-17T02:44:33.167ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::8cff:c9ff:fe76:d174]
1732026-03-17T02:44:33.170ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1742026-03-17T02:44:34.167ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1752026-03-17T02:44:34.172ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::8cff:c9ff:fe76:d174]
1762026-03-17T02:44:34.175ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1772026-03-17T02:44:34.708ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1782026-03-17T02:44:34.713ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::8cff:c9ff:fe76:d174]
1792026-03-17T02:44:34.716ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1802026-03-17T02:44:35.022ZDEBGtfportd (producer-registration-task): client response result = Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::c]:12221/metrics/producers", source: TimedOut }) unit = producer-server
1812026-03-17T02:44:35.023ZWARNtfportd (producer-registration-task): failed to register with Nexus, will retry count = 1 delay = 35.255956122s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers)" file = /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/8e30605/oximeter/producer/src/lib.rs:371 unit = producer-server
1822026-03-17T02:44:35.238ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1832026-03-17T02:44:35.242ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::8cff:c9ff:fe76:d174]
1842026-03-17T02:44:35.245ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1852026-03-17T02:44:35.319ZDEBGtfportd (internal-dns-resolver): lookup_socket_v6 srv dns_name = _nexus._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("04a4929c-ffe0-4c98-aafd-ebc61a0ed6db.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("7978df5c-2638-45eb-b0a8-5cf40fd0fccb.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("992a28b5-4983-458f-b27d-d85decb98fc4.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("992a28b5-4983-458f-b27d-d85decb98fc4.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::b)) }], valid_until: Instant { tv_sec: 6544, tv_nsec: 891651255 } }) unit = producer-server
1862026-03-17T02:44:35.320ZDEBGtfportd (producer-registration-task): will register with Nexus at [fd00:1122:3344:101::c]:12221 unit = producer-server
1872026-03-17T02:44:35.351ZDEBGtfportd (producer-registration-task): client request body = Some(Body) method = POST unit = producer-server uri = http://[fd00:1122:3344:101::c]:12221/metrics/producers
1882026-03-17T02:44:36.453ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1892026-03-17T02:44:36.458ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::8cff:c9ff:fe76:d174]
1902026-03-17T02:44:36.460ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1912026-03-17T02:44:37.361ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1922026-03-17T02:44:37.366ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::8cff:c9ff:fe76:d174]
1932026-03-17T02:44:37.369ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1942026-03-17T02:44:38.268ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1952026-03-17T02:44:38.273ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::8cff:c9ff:fe76:d174]
1962026-03-17T02:44:38.276ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1972026-03-17T02:44:38.963ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1982026-03-17T02:44:38.969ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::8cff:c9ff:fe76:d174]
1992026-03-17T02:44:38.972ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
2002026-03-17T02:44:39.768ZDEBGtfportd (kstat-sampler-worker): pulled samples from target id = 2566617268062380145 n_samples = 6 unit = oximeter
2012026-03-17T02:44:40.403ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2022026-03-17T02:44:40.407ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::8cff:c9ff:fe76:d174]
2032026-03-17T02:44:40.410ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
2042026-03-17T02:44:40.992ZDEBGtfportd (kstat-sampler-worker): pulled samples from target id = 15719734224641383838 n_samples = 6 unit = oximeter
2052026-03-17T02:44:41.493ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2062026-03-17T02:44:41.498ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::8cff:c9ff:fe76:d174]
2072026-03-17T02:44:41.501ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
2082026-03-17T02:44:42.754ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2092026-03-17T02:44:42.759ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::8cff:c9ff:fe76:d174]
2102026-03-17T02:44:42.762ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
2112026-03-17T02:44:43.736ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2122026-03-17T02:44:43.741ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::8cff:c9ff:fe76:d174]
2132026-03-17T02:44:43.744ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
2142026-03-17T02:44:44.945ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2152026-03-17T02:44:44.950ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::8cff:c9ff:fe76:d174]
2162026-03-17T02:44:44.953ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
2172026-03-17T02:44:45.711ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2182026-03-17T02:44:45.715ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::8cff:c9ff:fe76:d174]
2192026-03-17T02:44:45.717ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
2202026-03-17T02:44:46.892ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2212026-03-17T02:44:46.897ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::8cff:c9ff:fe76:d174]
2222026-03-17T02:44:46.899ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
2232026-03-17T02:44:47.669ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2242026-03-17T02:44:47.675ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::8cff:c9ff:fe76:d174]
2252026-03-17T02:44:47.678ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
2262026-03-17T02:44:48.489ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2272026-03-17T02:44:48.494ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::8cff:c9ff:fe76:d174]
2282026-03-17T02:44:48.497ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
2292026-03-17T02:44:49.743ZDEBGtfportd (kstat-sampler-worker): pulled samples from target id = 2566617268062380145 n_samples = 6 unit = oximeter
2302026-03-17T02:44:49.757ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2312026-03-17T02:44:49.763ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::8cff:c9ff:fe76:d174]
2322026-03-17T02:44:49.766ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
2332026-03-17T02:44:50.313ZDEBGtfportd (producer-registration-task): client response result = Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::c]:12221/metrics/producers", source: TimedOut }) unit = producer-server
2342026-03-17T02:44:50.314ZWARNtfportd (producer-registration-task): failed to register with Nexus, will retry count = 2 delay = 50.587797557s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers)" file = /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/8e30605/oximeter/producer/src/lib.rs:371 unit = producer-server
2352026-03-17T02:44:51.182ZDEBGtfportd (kstat-sampler-worker): pulled samples from target id = 15719734224641383838 n_samples = 6 unit = oximeter
2362026-03-17T02:44:51.291ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2372026-03-17T02:44:51.438ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::8cff:c9ff:fe76:d174]
2382026-03-17T02:44:51.505ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
2392026-03-17T02:44:51.754ZDEBGtfportd (internal-dns-resolver): lookup_socket_v6 srv dns_name = _nexus._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("04a4929c-ffe0-4c98-aafd-ebc61a0ed6db.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("7978df5c-2638-45eb-b0a8-5cf40fd0fccb.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("992a28b5-4983-458f-b27d-d85decb98fc4.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("992a28b5-4983-458f-b27d-d85decb98fc4.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::b)) }], valid_until: Instant { tv_sec: 6561, tv_nsec: 371660622 } }) unit = producer-server
2402026-03-17T02:44:51.756ZDEBGtfportd (producer-registration-task): will register with Nexus at [fd00:1122:3344:101::c]:12221 unit = producer-server
2412026-03-17T02:44:51.838ZDEBGtfportd (producer-registration-task): client request body = Some(Body) method = POST unit = producer-server uri = http://[fd00:1122:3344:101::c]:12221/metrics/producers
2422026-03-17T02:44:52.087ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2432026-03-17T02:44:52.099ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::8cff:c9ff:fe76:d174]
2442026-03-17T02:44:52.106ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
2452026-03-17T02:44:52.523ZINFOtfportd: removed stale NDP entry :fe80::98d5:8cff:fea4:68e8->9a:d5:8c:a4:68:e8
2462026-03-17T02:44:53.460ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2472026-03-17T02:44:53.468ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::8cff:c9ff:fe76:d174]
2482026-03-17T02:44:53.477ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
2492026-03-17T02:44:54.660ZINFOtfportd: added new NDP entry entry = tfportrear0_0:fe80::98d5:8cff:fea4:68e8->9a:d5:8c:a4:68:e8 interface = tfportrear0_0
2502026-03-17T02:44:54.939ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2512026-03-17T02:44:54.944ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::8cff:c9ff:fe76:d174]
2522026-03-17T02:44:54.947ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
2532026-03-17T02:44:56.308ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2542026-03-17T02:44:56.314ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::8cff:c9ff:fe76:d174]
2552026-03-17T02:44:56.317ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
2562026-03-17T02:44:57.739ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2572026-03-17T02:44:57.745ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::8cff:c9ff:fe76:d174]
2582026-03-17T02:44:57.748ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
2592026-03-17T02:44:58.558ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2602026-03-17T02:44:58.564ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::8cff:c9ff:fe76:d174]
2612026-03-17T02:44:58.567ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
2622026-03-17T02:44:59.615ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2632026-03-17T02:44:59.620ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::8cff:c9ff:fe76:d174]
2642026-03-17T02:44:59.623ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
2652026-03-17T02:44:59.725ZDEBGtfportd (kstat-sampler-worker): pulled samples from target id = 2566617268062380145 n_samples = 6 unit = oximeter
2662026-03-17T02:45:00.527ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2672026-03-17T02:45:00.532ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::8cff:c9ff:fe76:d174]
2682026-03-17T02:45:00.535ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
2692026-03-17T02:45:01.163ZDEBGtfportd (kstat-sampler-worker): pulled samples from target id = 15719734224641383838 n_samples = 6 unit = oximeter
2702026-03-17T02:45:01.261ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2712026-03-17T02:45:01.267ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::8cff:c9ff:fe76:d174]
2722026-03-17T02:45:01.271ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]