Max level shown:
1 [ Sep 5 02:28:41 Enabled. ]
2 [ Sep 5 02:28:41 Rereading configuration. ]
3 [ Sep 5 02:28:42 Rereading configuration. ]
4 [ Sep 5 02:28:53 Executing start method ("ctrun -l child -o noorphan,regent /opt/oxide/dendrite/bin/tfportd --sync-only &"). ]
5 [ Sep 5 02:28:53 Method "start" exited with status 0. ]
6 No non-localhost IPv6 addresses found in SMF properties
72026-09-05T02:28:53.821ZINFOtfportd: connecting to dpd host = [::1] port = 12224
82026-09-05T02:28:53.823ZINFOtfportd: connected to dpd running softnpu-5cb3f22d
92026-09-05T02:28:53.823ZINFOtfportd: running in sync-only mode
102026-09-05T02:28:53.825ZINFOtfportd: no non-localhost IPv6 listen address available, waiting for SMF update unit = oximeter
112026-09-05T02:28:53.826ZINFOtfportd: refreshing SMF config prior to waiting on signals unit = signal_handler
12 No non-localhost IPv6 addresses found in SMF properties
132026-09-05T02:28:53.827ZINFOtfportd: refreshed config: Config { log_file: None, log_format: Json, listen_addresses: [ [::1]:12231, ], pkt_source: None, dpd_host: "[::1]", dpd_port: 12224, vlan_link: None, vlan_data: None, techport0_prefix: None, techport1_prefix: None, rack_id: Some( 15906f2a-ac55-40f6-84d6-b04f780057f4, ), sled_id: Some( c2678b38-bce3-46be-8684-10fba74013c6, ), sled_model: Some( "i86pc", ), sled_revision: Some( 0, ), sled_serial: Some( "sock", ), }
142026-09-05T02:28:53.827ZINFOtfportd: no non-localhost IPv6 listen address available, waiting for SMF update unit = oximeter
152026-09-05T02:28:53.833ZINFOtfportd: added new NDP entry entry = tfportrear0_0:fe80::b800:beff:fe9e:c80c->ba:00:be:9e:c8:0c interface = tfportrear0_0
162026-09-05T02:28:53.835ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
172026-09-05T02:28:53.842ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::b800:beff:fe9e:c80c]
182026-09-05T02:28:53.847ZWARNtfportd: tfportqsfp0_0 has no addrs
192026-09-05T02:28:54.374ZINFOtfportd: added new ARP entry entry = 10.151.2.174->66:07:a8:31:e6:fd interface = tfportqsfp0_0
202026-09-05T02:28:54.392ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
212026-09-05T02:28:54.398ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::b800:beff:fe9e:c80c]
222026-09-05T02:28:54.401ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.2.174]
23 [ Sep 5 02:28:54 Rereading configuration. ]
24 [ Sep 5 02:28:54 Executing refresh method (:kill). ]
25 [ Sep 5 02:28:54 Stopping because all processes in service exited. ]
26 [ Sep 5 02:28:54 Executing stop method (:kill). ]
27 [ Sep 5 02:28:54 Executing start method ("ctrun -l child -o noorphan,regent /opt/oxide/dendrite/bin/tfportd --sync-only &"). ]
28 [ Sep 5 02:28:54 Method "start" exited with status 0. ]
292026-09-05T02:28:54.600ZINFOtfportd: connecting to dpd host = [::1] port = 12224
302026-09-05T02:28:54.601ZINFOtfportd: connected to dpd running softnpu-5cb3f22d
312026-09-05T02:28:54.602ZINFOtfportd: running in sync-only mode
322026-09-05T02:28:54.602ZINFOtfportd: found suitable IPv6 address for oximeter producer server address = fd00:1122:3344:101::2 unit = oximeter
332026-09-05T02:28:54.603ZINFOtfportd: refreshing SMF config prior to waiting on signals unit = signal_handler
342026-09-05T02:28:54.604ZDEBGtfportd: extracted oximeter configuration data config = OximeterConfig { listen_address: fd00:1122:3344:101::2, sled_identifiers: SledIdentifiers { rack_id: 15906f2a-ac55-40f6-84d6-b04f780057f4, sled_id: c2678b38-bce3-46be-8684-10fba74013c6, model: "i86pc", revision: 0, serial: "sock" }, switch_identifiers: SwitchIdentifiers { asic_backend: "softnpu", fab: None, fuse: Some(FuseData { chip_rev: ChipRevision { device_id: 272, rev: "B1", rev_num: 2 }, disabled: DisabledFeatures { bsync: false, mau: [0, 0, 0, 0], pgen: false, pipes: 0, ports: 0, resub: false, speeds: 0, tm_mem: 0 }, frequency: FrequencySettings { bps: 3, bps_ext: 0, cpu_speed_dis: 0, disabled: false, pcie_dis: 0, pps: 3, pps_ext: 0 }, manufacturing: ManufacturingData { core_repair: false, die_rotation: false, pmro_and_skew: 0, silent_spin: 0, soft_pipe_dis: 0, tile_repair: false, voltage_scaling: 0, wf_core_repair: false }, part: PartInfo { part_num: 4660, pkg_id: 1, version: 2 } }), lot: None, lotnum: None, model: "FAKE_SIM_SIDECAR", revision: 0, serial: "SimSidecar0", sidecar_id: c9f0c0b2-3fda-41d5-b780-b9dc290c6353, slot: 0, wafer: None, wafer_loc: None } } unit = oximeter
352026-09-05T02:28:54.604ZDEBGtfportd: created producer ID producer_id = 1e0724a0-2f6c-479d-b5d2-1968f64c2761 unit = oximeter
362026-09-05T02:28:54.605ZINFOtfportd: refreshed config: Config { log_file: None, log_format: Json, listen_addresses: [ [fd00:1122:3344:101::2]:12231, [::1]:12231, ], pkt_source: None, dpd_host: "[::1]", dpd_port: 12224, vlan_link: None, vlan_data: None, techport0_prefix: None, techport1_prefix: None, rack_id: Some( 15906f2a-ac55-40f6-84d6-b04f780057f4, ), sled_id: Some( c2678b38-bce3-46be-8684-10fba74013c6, ), sled_model: Some( "i86pc", ), sled_revision: Some( 0, ), sled_serial: Some( "sock", ), }
372026-09-05T02:28:54.612ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
382026-09-05T02:28:54.614ZINFOtfportd: added new ARP entry entry = 10.151.2.174->66:07:a8:31:e6:fd interface = tfportqsfp0_0
392026-09-05T02:28:54.615ZINFOtfportd: added new NDP entry entry = tfportrear0_0:fe80::87e:deff:feae:19de->0a:7e:de:ae:19:de interface = tfportrear0_0
402026-09-05T02:28:54.615ZINFOtfportd: added new NDP entry entry = tfportrear0_0:fe80::b800:beff:fe9e:c80c->ba:00:be:9e:c8:0c interface = tfportrear0_0
412026-09-05T02:28:54.615ZDEBGtfportd: registered DTrace probes unit = producer-server
422026-09-05T02:28:54.617ZDEBGtfportd (dropshot): registered endpoint local_addr = [fd00:1122:3344:101::2]:35013 method = GET path = /{producer_id} unit = producer-server versions = all
432026-09-05T02:28:54.617ZINFOtfportd (dropshot): listening file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.7/src/server.rs:328 local_addr = [fd00:1122:3344:101::2]:35013 unit = producer-server
442026-09-05T02:28:54.617ZDEBGtfportd (dropshot): successfully registered DTrace USDT probes local_addr = [fd00:1122:3344:101::2]:35013 unit = producer-server
452026-09-05T02:28:54.617ZDEBGtfportd: Requested any available port, Dropshot server has been bound to [fd00:1122:3344:101::2]:35013 unit = producer-server
462026-09-05T02:28:54.617ZDEBGtfportd: Nexus IP not provided, will use DNS to resolve it unit = producer-server
472026-09-05T02:28:54.617ZINFOtfportd (internal-dns-resolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53] file = /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/8e30605/internal-dns/resolver/src/resolver.rs:144 unit = producer-server
482026-09-05T02:28:54.617ZDEBGtfportd: starting producer registration task unit = producer-server
492026-09-05T02:28:54.617ZINFOtfportd: starting oximeter metric producer server address = [fd00:1122:3344:101::2]:35013 file = /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/8e30605/oximeter/producer/src/lib.rs:279 interval = 30s producer_id = 1e0724a0-2f6c-479d-b5d2-1968f64c2761 unit = producer-server
502026-09-05T02:28:54.617ZINFOtfportd: created oximeter producer server unit = oximeter
512026-09-05T02:28:54.617ZDEBGtfportd (producer-registration-task): registering as oximeter producer with Nexus registration_count = 0 unit = producer-server
522026-09-05T02:28:54.618ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::b800:beff:fe9e:c80c]
532026-09-05T02:28:54.618ZDEBGtfportd (kstat-sampler): adding new target unit = oximeter
542026-09-05T02:28:54.620ZDEBGtfportd (kstat-sampler-worker): storing new creation time for tracked target creation_time = 2026-09-05T02:28:37.487139901Z path = KstatPath { module: "link", instance: 0, name: "tfportrear0_0" } unit = oximeter
552026-09-05T02:28:54.620ZDEBGtfportd (kstat-sampler-worker): inserted empty per-target sample queue id = 16360451574083144897 unit = oximeter
562026-09-05T02:28:54.620ZDEBGtfportd: added new link to kstat sampler link_kind = switch-port-control link_name = tfportrear0_0 unit = metrics-task-tracker
572026-09-05T02:28:54.621ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.2.174]
582026-09-05T02:28:54.621ZDEBGtfportd (kstat-sampler): adding new target unit = oximeter
592026-09-05T02:28:54.621ZDEBGtfportd (kstat-sampler-worker): storing new creation time for tracked target creation_time = 2026-09-05T02:28:37.486981579Z path = KstatPath { module: "link", instance: 0, name: "tfportqsfp0_0" } unit = oximeter
602026-09-05T02:28:54.621ZDEBGtfportd (kstat-sampler-worker): inserted empty per-target sample queue id = 982915022302792582 unit = oximeter
612026-09-05T02:28:54.621ZDEBGtfportd: added new link to kstat sampler link_kind = switch-port-control link_name = tfportqsfp0_0 unit = metrics-task-tracker
622026-09-05T02:28:55.210ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
632026-09-05T02:28:55.214ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::b800:beff:fe9e:c80c]
642026-09-05T02:28:55.217ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.2.174]
652026-09-05T02:28:56.101ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
662026-09-05T02:28:56.106ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::b800:beff:fe9e:c80c]
672026-09-05T02:28:56.109ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.2.174]
682026-09-05T02:28:57.145ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
692026-09-05T02:28:57.150ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::b800:beff:fe9e:c80c]
702026-09-05T02:28:57.154ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.2.174]
712026-09-05T02:28:58.582ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
722026-09-05T02:28:58.587ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::b800:beff:fe9e:c80c]
732026-09-05T02:28:58.590ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.2.174]
742026-09-05T02:28:59.620ZDEBGtfportd (internal-dns-resolver): lookup_socket_v6 srv dns_name = _nexus._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("02f42754-8a41-40f2-adf9-e8b024cfa493.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("6980ce11-1ac3-4adb-b9df-52283447f259.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("7ac9f184-2199-44a8-ba6f-32afc31f0d60.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("7ac9f184-2199-44a8-ba6f-32afc31f0d60.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::a)) }], valid_until: Instant { tv_sec: 556, tv_nsec: 386280541 } }) unit = producer-server
752026-09-05T02:28:59.621ZDEBGtfportd (producer-registration-task): will register with Nexus at [fd00:1122:3344:101::b]:12221 unit = producer-server
762026-09-05T02:28:59.638ZDEBGtfportd (producer-registration-task): client request body = Some(Body) method = POST unit = producer-server uri = http://[fd00:1122:3344:101::b]:12221/metrics/producers
772026-09-05T02:28:59.969ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
782026-09-05T02:28:59.973ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::b800:beff:fe9e:c80c]
792026-09-05T02:28:59.977ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.2.174]
802026-09-05T02:29:01.045ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
812026-09-05T02:29:01.049ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::b800:beff:fe9e:c80c]
822026-09-05T02:29:01.052ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.2.174]
832026-09-05T02:29:02.439ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
842026-09-05T02:29:02.445ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::b800:beff:fe9e:c80c]
852026-09-05T02:29:02.448ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.2.174]
862026-09-05T02:29:03.823ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
872026-09-05T02:29:03.828ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::b800:beff:fe9e:c80c]
882026-09-05T02:29:03.831ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.2.174]
892026-09-05T02:29:04.623ZDEBGtfportd (kstat-sampler-worker): pulled samples from target id = 16360451574083144897 n_samples = 6 unit = oximeter
902026-09-05T02:29:04.623ZDEBGtfportd (kstat-sampler-worker): pulled samples from target id = 982915022302792582 n_samples = 6 unit = oximeter
912026-09-05T02:29:04.784ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
922026-09-05T02:29:04.789ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::b800:beff:fe9e:c80c]
932026-09-05T02:29:04.793ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.2.174]
942026-09-05T02:29:05.786ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
952026-09-05T02:29:05.791ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::b800:beff:fe9e:c80c]
962026-09-05T02:29:05.794ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.2.174]
972026-09-05T02:29:06.490ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
982026-09-05T02:29:06.494ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::b800:beff:fe9e:c80c]
992026-09-05T02:29:06.497ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.2.174]
1002026-09-05T02:29:07.271ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1012026-09-05T02:29:07.277ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::b800:beff:fe9e:c80c]
1022026-09-05T02:29:07.279ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.2.174]
1032026-09-05T02:29:08.548ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1042026-09-05T02:29:08.552ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::b800:beff:fe9e:c80c]
1052026-09-05T02:29:08.555ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.2.174]
1062026-09-05T02:29:09.154ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1072026-09-05T02:29:09.159ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::b800:beff:fe9e:c80c]
1082026-09-05T02:29:09.162ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.2.174]
1092026-09-05T02:29:10.280ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1102026-09-05T02:29:10.289ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::b800:beff:fe9e:c80c]
1112026-09-05T02:29:10.291ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.2.174]
1122026-09-05T02:29:10.975ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1132026-09-05T02:29:10.980ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::b800:beff:fe9e:c80c]
1142026-09-05T02:29:10.983ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.2.174]
1152026-09-05T02:29:12.295ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1162026-09-05T02:29:12.301ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::b800:beff:fe9e:c80c]
1172026-09-05T02:29:12.304ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.2.174]
1182026-09-05T02:29:13.489ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1192026-09-05T02:29:13.493ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::b800:beff:fe9e:c80c]
1202026-09-05T02:29:13.497ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.2.174]
1212026-09-05T02:29:14.039ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1222026-09-05T02:29:14.046ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::b800:beff:fe9e:c80c]
1232026-09-05T02:29:14.051ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.2.174]
1242026-09-05T02:29:14.615ZDEBGtfportd (kstat-sampler-worker): pulled samples from target id = 16360451574083144897 n_samples = 6 unit = oximeter
1252026-09-05T02:29:14.616ZDEBGtfportd (kstat-sampler-worker): pulled samples from target id = 982915022302792582 n_samples = 6 unit = oximeter
1262026-09-05T02:29:14.629ZDEBGtfportd (producer-registration-task): client response result = Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::b]:12221/metrics/producers", source: TimedOut }) unit = producer-server
1272026-09-05T02:29:14.630ZWARNtfportd (producer-registration-task): failed to register with Nexus, will retry count = 0 delay = 20.022490824s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers)" file = /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/8e30605/oximeter/producer/src/lib.rs:371 unit = producer-server
1282026-09-05T02:29:14.908ZDEBGtfportd (internal-dns-resolver): lookup_socket_v6 srv dns_name = _nexus._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("02f42754-8a41-40f2-adf9-e8b024cfa493.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("6980ce11-1ac3-4adb-b9df-52283447f259.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("7ac9f184-2199-44a8-ba6f-32afc31f0d60.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("7ac9f184-2199-44a8-ba6f-32afc31f0d60.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::a)) }], valid_until: Instant { tv_sec: 571, tv_nsec: 685379161 } }) unit = producer-server
1292026-09-05T02:29:14.909ZDEBGtfportd (producer-registration-task): will register with Nexus at [fd00:1122:3344:101::b]:12221 unit = producer-server
1302026-09-05T02:29:14.916ZDEBGtfportd (producer-registration-task): client request body = Some(Body) method = POST unit = producer-server uri = http://[fd00:1122:3344:101::b]:12221/metrics/producers
1312026-09-05T02:29:15.004ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1322026-09-05T02:29:15.009ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::b800:beff:fe9e:c80c]
1332026-09-05T02:29:15.012ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.2.174]
1342026-09-05T02:29:16.375ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1352026-09-05T02:29:16.381ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::b800:beff:fe9e:c80c]
1362026-09-05T02:29:16.384ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.2.174]
1372026-09-05T02:29:17.687ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1382026-09-05T02:29:17.699ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::b800:beff:fe9e:c80c]
1392026-09-05T02:29:17.711ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.2.174]
1402026-09-05T02:29:18.573ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1412026-09-05T02:29:18.584ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::b800:beff:fe9e:c80c]
1422026-09-05T02:29:18.590ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.2.174]
1432026-09-05T02:29:19.857ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1442026-09-05T02:29:19.867ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::b800:beff:fe9e:c80c]
1452026-09-05T02:29:19.872ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.2.174]
1462026-09-05T02:29:20.536ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1472026-09-05T02:29:20.541ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::b800:beff:fe9e:c80c]
1482026-09-05T02:29:20.544ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.2.174]
1492026-09-05T02:29:21.829ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1502026-09-05T02:29:21.835ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::b800:beff:fe9e:c80c]
1512026-09-05T02:29:21.838ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.2.174]
1522026-09-05T02:29:23.275ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1532026-09-05T02:29:23.279ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::b800:beff:fe9e:c80c]
1542026-09-05T02:29:23.282ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.2.174]
1552026-09-05T02:29:24.393ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1562026-09-05T02:29:24.399ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::b800:beff:fe9e:c80c]
1572026-09-05T02:29:24.402ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.2.174]
1582026-09-05T02:29:24.605ZDEBGtfportd (kstat-sampler-worker): pulled samples from target id = 16360451574083144897 n_samples = 6 unit = oximeter
1592026-09-05T02:29:24.606ZDEBGtfportd (kstat-sampler-worker): pulled samples from target id = 982915022302792582 n_samples = 6 unit = oximeter
1602026-09-05T02:29:25.588ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1612026-09-05T02:29:25.593ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::b800:beff:fe9e:c80c]
1622026-09-05T02:29:25.596ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.2.174]
1632026-09-05T02:29:26.560ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1642026-09-05T02:29:26.565ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::b800:beff:fe9e:c80c]
1652026-09-05T02:29:26.569ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.2.174]
1662026-09-05T02:29:27.891ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1672026-09-05T02:29:27.899ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::b800:beff:fe9e:c80c]
1682026-09-05T02:29:27.902ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.2.174]
1692026-09-05T02:29:28.808ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1702026-09-05T02:29:28.816ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::b800:beff:fe9e:c80c]
1712026-09-05T02:29:28.819ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.2.174]
1722026-09-05T02:29:29.905ZDEBGtfportd (producer-registration-task): client response result = Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::b]:12221/metrics/producers", source: TimedOut }) unit = producer-server
1732026-09-05T02:29:29.906ZWARNtfportd (producer-registration-task): failed to register with Nexus, will retry count = 1 delay = 35.312846689s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers)" file = /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/8e30605/oximeter/producer/src/lib.rs:371 unit = producer-server
1742026-09-05T02:29:29.920ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1752026-09-05T02:29:29.927ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::b800:beff:fe9e:c80c]
1762026-09-05T02:29:29.930ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.2.174]
1772026-09-05T02:29:30.180ZDEBGtfportd (internal-dns-resolver): lookup_socket_v6 srv dns_name = _nexus._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("02f42754-8a41-40f2-adf9-e8b024cfa493.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("6980ce11-1ac3-4adb-b9df-52283447f259.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("7ac9f184-2199-44a8-ba6f-32afc31f0d60.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("7ac9f184-2199-44a8-ba6f-32afc31f0d60.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::a)) }], valid_until: Instant { tv_sec: 586, tv_nsec: 969921538 } }) unit = producer-server
1782026-09-05T02:29:30.180ZDEBGtfportd (producer-registration-task): will register with Nexus at [fd00:1122:3344:101::b]:12221 unit = producer-server
1792026-09-05T02:29:30.188ZDEBGtfportd (producer-registration-task): client request body = Some(Body) method = POST unit = producer-server uri = http://[fd00:1122:3344:101::b]:12221/metrics/producers
1802026-09-05T02:29:30.832ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1812026-09-05T02:29:30.837ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::b800:beff:fe9e:c80c]
1822026-09-05T02:29:30.840ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.2.174]
1832026-09-05T02:29:31.864ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1842026-09-05T02:29:31.869ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::b800:beff:fe9e:c80c]
1852026-09-05T02:29:31.872ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.2.174]
1862026-09-05T02:29:32.794ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1872026-09-05T02:29:32.798ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::b800:beff:fe9e:c80c]
1882026-09-05T02:29:32.802ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.2.174]
1892026-09-05T02:29:33.587ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1902026-09-05T02:29:33.597ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::b800:beff:fe9e:c80c]
1912026-09-05T02:29:33.600ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.2.174]
1922026-09-05T02:29:34.497ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1932026-09-05T02:29:34.502ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::b800:beff:fe9e:c80c]
1942026-09-05T02:29:34.505ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.2.174]
1952026-09-05T02:29:34.608ZDEBGtfportd (kstat-sampler-worker): pulled samples from target id = 16360451574083144897 n_samples = 6 unit = oximeter
1962026-09-05T02:29:34.608ZDEBGtfportd (kstat-sampler-worker): pulled samples from target id = 982915022302792582 n_samples = 6 unit = oximeter
1972026-09-05T02:29:35.739ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1982026-09-05T02:29:35.747ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::b800:beff:fe9e:c80c]
1992026-09-05T02:29:35.753ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.2.174]
2002026-09-05T02:29:36.330ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2012026-09-05T02:29:36.336ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::b800:beff:fe9e:c80c]
2022026-09-05T02:29:36.340ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.2.174]
2032026-09-05T02:29:38.141ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2042026-09-05T02:29:38.148ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::b800:beff:fe9e:c80c]
2052026-09-05T02:29:38.152ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.2.174]
2062026-09-05T02:29:39.323ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2072026-09-05T02:29:39.330ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::b800:beff:fe9e:c80c]
2082026-09-05T02:29:39.335ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.2.174]
2092026-09-05T02:29:40.406ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2102026-09-05T02:29:40.414ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::b800:beff:fe9e:c80c]
2112026-09-05T02:29:40.419ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.2.174]
2122026-09-05T02:29:41.295ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2132026-09-05T02:29:41.300ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::b800:beff:fe9e:c80c]
2142026-09-05T02:29:41.303ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.2.174]
2152026-09-05T02:29:41.913ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2162026-09-05T02:29:41.919ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::b800:beff:fe9e:c80c]
2172026-09-05T02:29:41.922ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.2.174]
2182026-09-05T02:29:42.715ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2192026-09-05T02:29:42.721ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::b800:beff:fe9e:c80c]
2202026-09-05T02:29:42.724ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.2.174]
2212026-09-05T02:29:43.509ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2222026-09-05T02:29:43.516ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::b800:beff:fe9e:c80c]
2232026-09-05T02:29:43.519ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.2.174]
2242026-09-05T02:29:44.590ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2252026-09-05T02:29:44.596ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::b800:beff:fe9e:c80c]
2262026-09-05T02:29:44.600ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.2.174]
2272026-09-05T02:29:44.611ZDEBGtfportd (kstat-sampler-worker): pulled samples from target id = 16360451574083144897 n_samples = 6 unit = oximeter
2282026-09-05T02:29:44.611ZDEBGtfportd (kstat-sampler-worker): pulled samples from target id = 982915022302792582 n_samples = 6 unit = oximeter
2292026-09-05T02:29:45.164ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2302026-09-05T02:29:45.170ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::b800:beff:fe9e:c80c]
2312026-09-05T02:29:45.175ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.2.174]
2322026-09-05T02:29:45.189ZDEBGtfportd (producer-registration-task): client response result = Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::b]:12221/metrics/producers", source: TimedOut }) unit = producer-server
2332026-09-05T02:29:45.190ZWARNtfportd (producer-registration-task): failed to register with Nexus, will retry count = 2 delay = 50.596414216s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers)" file = /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/8e30605/oximeter/producer/src/lib.rs:371 unit = producer-server
2342026-09-05T02:29:45.903ZDEBGtfportd (internal-dns-resolver): lookup_socket_v6 srv dns_name = _nexus._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("02f42754-8a41-40f2-adf9-e8b024cfa493.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("6980ce11-1ac3-4adb-b9df-52283447f259.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("7ac9f184-2199-44a8-ba6f-32afc31f0d60.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("7ac9f184-2199-44a8-ba6f-32afc31f0d60.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::a)) }], valid_until: Instant { tv_sec: 602, tv_nsec: 693492563 } }) unit = producer-server
2352026-09-05T02:29:45.904ZDEBGtfportd (producer-registration-task): will register with Nexus at [fd00:1122:3344:101::b]:12221 unit = producer-server
2362026-09-05T02:29:45.919ZDEBGtfportd (producer-registration-task): client request body = Some(Body) method = POST unit = producer-server uri = http://[fd00:1122:3344:101::b]:12221/metrics/producers
2372026-09-05T02:29:46.558ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2382026-09-05T02:29:46.568ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::b800:beff:fe9e:c80c]
2392026-09-05T02:29:46.574ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.2.174]
2402026-09-05T02:29:47.174ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2412026-09-05T02:29:47.203ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::b800:beff:fe9e:c80c]
2422026-09-05T02:29:48.255ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.2.174]
2432026-09-05T02:29:49.376ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2442026-09-05T02:29:49.384ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::b800:beff:fe9e:c80c]
2452026-09-05T02:29:49.389ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.2.174]
2462026-09-05T02:29:50.639ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2472026-09-05T02:29:50.796ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::b800:beff:fe9e:c80c]
2482026-09-05T02:29:50.808ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.2.174]
2492026-09-05T02:29:51.475ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2502026-09-05T02:29:51.486ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::b800:beff:fe9e:c80c]
2512026-09-05T02:29:51.507ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.2.174]
2522026-09-05T02:29:52.523ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2532026-09-05T02:29:52.548ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::b800:beff:fe9e:c80c]
2542026-09-05T02:29:52.570ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.2.174]
2552026-09-05T02:29:53.679ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2562026-09-05T02:29:53.687ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::b800:beff:fe9e:c80c]
2572026-09-05T02:29:53.693ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.2.174]
2582026-09-05T02:29:54.281ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2592026-09-05T02:29:54.488ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::b800:beff:fe9e:c80c]
2602026-09-05T02:29:54.493ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.2.174]
2612026-09-05T02:29:54.692ZDEBGtfportd (kstat-sampler-worker): pulled samples from target id = 16360451574083144897 n_samples = 6 unit = oximeter
2622026-09-05T02:29:54.698ZDEBGtfportd (kstat-sampler-worker): pulled samples from target id = 982915022302792582 n_samples = 6 unit = oximeter
2632026-09-05T02:29:55.850ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2642026-09-05T02:29:55.858ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::b800:beff:fe9e:c80c]
2652026-09-05T02:29:55.864ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.2.174]
2662026-09-05T02:29:56.480ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2672026-09-05T02:29:56.683ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::b800:beff:fe9e:c80c]
2682026-09-05T02:29:56.707ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.2.174]
2692026-09-05T02:29:57.594ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2702026-09-05T02:29:57.603ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::b800:beff:fe9e:c80c]
2712026-09-05T02:29:57.611ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.2.174]
2722026-09-05T02:29:59.023ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2732026-09-05T02:29:59.125ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::b800:beff:fe9e:c80c]
2742026-09-05T02:29:59.148ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.2.174]
2752026-09-05T02:30:00.166ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2762026-09-05T02:30:00.177ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::b800:beff:fe9e:c80c]
2772026-09-05T02:30:00.182ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.2.174]
2782026-09-05T02:30:00.950ZDEBGtfportd (producer-registration-task): client response result = Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::b]:12221/metrics/producers", source: TimedOut }) unit = producer-server
2792026-09-05T02:30:00.984ZWARNtfportd (producer-registration-task): failed to register with Nexus, will retry count = 3 delay = 66.390405945s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers)" file = /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/8e30605/oximeter/producer/src/lib.rs:371 unit = producer-server