Max level shown:
1 [ Dec 18 00:13:08 Enabled. ]
2 [ Dec 18 00:13:08 Rereading configuration. ]
3 [ Dec 18 00:13:09 Rereading configuration. ]
4 [ Dec 18 00:13:20 Executing start method ("ctrun -l child -o noorphan,regent /opt/oxide/dendrite/bin/tfportd --sync-only &"). ]
5 [ Dec 18 00:13:20 Method "start" exited with status 0. ]
6 No non-localhost IPv6 addresses found in SMF properties
72025-12-18T00:13:20.838ZINFOtfportd: connecting to dpd host = [::1] port = 12224
82025-12-18T00:13:20.880ZINFOtfportd: connected to dpd running softnpu-606c0be8
92025-12-18T00:13:20.880ZINFOtfportd: running in sync-only mode
102025-12-18T00:13:20.881ZINFOtfportd: no non-localhost IPv6 listen address available, waiting for SMF update unit = oximeter
112025-12-18T00:13:20.883ZINFOtfportd: refreshing SMF config prior to waiting on signals unit = signal_handler
12 No non-localhost IPv6 addresses found in SMF properties
132025-12-18T00:13:20.884ZINFOtfportd: refreshed config: Config { log_file: None, log_format: Json, listen_addresses: [ [::1]:12231, ], pkt_source: None, dpd_host: "[::1]", dpd_port: 12224, vlan_link: None, vlan_data: None, techport0_prefix: None, techport1_prefix: None, rack_id: Some( 9612e001-fa71-44b4-a8ac-7584d32eb73e, ), sled_id: Some( 9fc02d14-537b-4588-becb-c871751274dd, ), sled_model: Some( "i86pc", ), sled_revision: Some( 0, ), sled_serial: Some( "buskin", ), }
142025-12-18T00:13:20.884ZINFOtfportd: no non-localhost IPv6 listen address available, waiting for SMF update unit = oximeter
152025-12-18T00:13:20.889ZINFOtfportd: added new NDP entry entry = tfportrear0_0:fe80::60:5eff:fe6e:12b2->02:60:5e:6e:12:b2 interface = tfportrear0_0
162025-12-18T00:13:20.891ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
172025-12-18T00:13:20.897ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::60:5eff:fe6e:12b2]
182025-12-18T00:13:20.903ZWARNtfportd: tfportqsfp0_0 has no addrs
19 [ Dec 18 00:13:21 Rereading configuration. ]
20 [ Dec 18 00:13:21 Executing refresh method (:kill). ]
21 [ Dec 18 00:13:21 Stopping because all processes in service exited. ]
22 [ Dec 18 00:13:21 Executing stop method (:kill). ]
23 [ Dec 18 00:13:21 Executing start method ("ctrun -l child -o noorphan,regent /opt/oxide/dendrite/bin/tfportd --sync-only &"). ]
24 [ Dec 18 00:13:21 Method "start" exited with status 0. ]
252025-12-18T00:13:21.541ZINFOtfportd: connecting to dpd host = [::1] port = 12224
262025-12-18T00:13:21.587ZINFOtfportd: connected to dpd running softnpu-606c0be8
272025-12-18T00:13:21.587ZINFOtfportd: running in sync-only mode
282025-12-18T00:13:21.587ZINFOtfportd: found suitable IPv6 address for oximeter producer server address = fd00:1122:3344:101::2 unit = oximeter
292025-12-18T00:13:21.589ZINFOtfportd: refreshing SMF config prior to waiting on signals unit = signal_handler
302025-12-18T00:13:21.589ZDEBGtfportd: extracted oximeter configuration data config = OximeterConfig { listen_address: fd00:1122:3344:101::2, sled_identifiers: SledIdentifiers { rack_id: 9612e001-fa71-44b4-a8ac-7584d32eb73e, sled_id: 9fc02d14-537b-4588-becb-c871751274dd, model: "i86pc", revision: 0, serial: "buskin" }, switch_identifiers: SwitchIdentifiers { asic_backend: "softnpu", fab: None, lot: None, model: "FAKE_SIM_SIDECAR", revision: 0, serial: "SimSidecar0", sidecar_id: 9393e19b-1ecc-4ba1-a3cf-54ca06854c56, slot: 0, wafer: None, wafer_loc: None } } unit = oximeter
312025-12-18T00:13:21.589ZDEBGtfportd: created producer ID producer_id = 9a412c8e-b7c6-4a9d-b5a8-e93fa0b98fb5 unit = oximeter
322025-12-18T00:13:21.590ZINFOtfportd: refreshed config: Config { log_file: None, log_format: Json, listen_addresses: [ [fd00:1122:3344:101::2]:12231, [::1]:12231, ], pkt_source: None, dpd_host: "[::1]", dpd_port: 12224, vlan_link: None, vlan_data: None, techport0_prefix: None, techport1_prefix: None, rack_id: Some( 9612e001-fa71-44b4-a8ac-7584d32eb73e, ), sled_id: Some( 9fc02d14-537b-4588-becb-c871751274dd, ), sled_model: Some( "i86pc", ), sled_revision: Some( 0, ), sled_serial: Some( "buskin", ), }
332025-12-18T00:13:21.597ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
342025-12-18T00:13:21.598ZINFOtfportd: added new NDP entry entry = tfportrear0_0:fe80::60:5eff:fe6e:12b2->02:60:5e:6e:12:b2 interface = tfportrear0_0
352025-12-18T00:13:21.599ZINFOtfportd: added new NDP entry entry = tfportrear0_0:fe80::ec5b:28ff:fe1d:5753->ee:5b:28:1d:57:53 interface = tfportrear0_0
362025-12-18T00:13:21.599ZDEBGtfportd: registered DTrace probes unit = producer-server
372025-12-18T00:13:21.600ZDEBGtfportd (dropshot): registered endpoint local_addr = [fd00:1122:3344:101::2]:44638 method = GET path = /{producer_id} unit = producer-server versions = all
382025-12-18T00:13:21.600ZINFOtfportd (dropshot): listening file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.4/src/server.rs:328 local_addr = [fd00:1122:3344:101::2]:44638 unit = producer-server
392025-12-18T00:13:21.600ZDEBGtfportd (dropshot): successfully registered DTrace USDT probes local_addr = [fd00:1122:3344:101::2]:44638 unit = producer-server
402025-12-18T00:13:21.600ZDEBGtfportd: Requested any available port, Dropshot server has been bound to [fd00:1122:3344:101::2]:44638 unit = producer-server
412025-12-18T00:13:21.600ZDEBGtfportd: Nexus IP not provided, will use DNS to resolve it unit = producer-server
422025-12-18T00:13:21.600ZINFOtfportd (internal-dns-resolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53] file = /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/e76962a/internal-dns/resolver/src/resolver.rs:144 unit = producer-server
432025-12-18T00:13:21.600ZDEBGtfportd: starting producer registration task unit = producer-server
442025-12-18T00:13:21.601ZINFOtfportd: starting oximeter metric producer server address = [fd00:1122:3344:101::2]:44638 file = /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/e76962a/oximeter/producer/src/lib.rs:279 interval = 30s producer_id = 9a412c8e-b7c6-4a9d-b5a8-e93fa0b98fb5 unit = producer-server
452025-12-18T00:13:21.601ZINFOtfportd: created oximeter producer server unit = oximeter
462025-12-18T00:13:21.601ZDEBGtfportd (producer-registration-task): registering as oximeter producer with Nexus registration_count = 0 unit = producer-server
472025-12-18T00:13:21.603ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::60:5eff:fe6e:12b2]
482025-12-18T00:13:21.603ZDEBGtfportd (kstat-sampler): adding new target unit = oximeter
492025-12-18T00:13:21.603ZDEBGtfportd (kstat-sampler-worker): storing new creation time for tracked target creation_time = 2025-12-18T00:13:04.735443539Z path = KstatPath { module: "link", instance: 0, name: "tfportrear0_0" } unit = oximeter
502025-12-18T00:13:21.603ZDEBGtfportd (kstat-sampler-worker): inserted empty per-target sample queue id = 11573805073432295311 unit = oximeter
512025-12-18T00:13:21.603ZDEBGtfportd: added new link to kstat sampler link_kind = switch-port-control link_name = tfportrear0_0 unit = metrics-task-tracker
522025-12-18T00:13:21.606ZWARNtfportd: tfportqsfp0_0 has no addrs
532025-12-18T00:13:22.180ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
542025-12-18T00:13:22.185ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::60:5eff:fe6e:12b2]
552025-12-18T00:13:22.189ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
562025-12-18T00:13:22.189ZDEBGtfportd (kstat-sampler): adding new target unit = oximeter
572025-12-18T00:13:22.191ZDEBGtfportd (kstat-sampler-worker): storing new creation time for tracked target creation_time = 2025-12-18T00:13:04.735308248Z path = KstatPath { module: "link", instance: 0, name: "tfportqsfp0_0" } unit = oximeter
582025-12-18T00:13:22.191ZDEBGtfportd (kstat-sampler-worker): inserted empty per-target sample queue id = 13948335495314812778 unit = oximeter
592025-12-18T00:13:22.191ZDEBGtfportd: added new link to kstat sampler link_kind = switch-port-control link_name = tfportqsfp0_0 unit = metrics-task-tracker
602025-12-18T00:13:22.621ZINFOtfportd: added new ARP entry entry = 10.151.1.174->e6:d4:10:b1:08:88 interface = tfportqsfp0_0
612025-12-18T00:13:22.893ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
622025-12-18T00:13:22.898ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::60:5eff:fe6e:12b2]
632025-12-18T00:13:22.901ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
642025-12-18T00:13:24.152ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
652025-12-18T00:13:24.156ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::60:5eff:fe6e:12b2]
662025-12-18T00:13:24.159ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
672025-12-18T00:13:24.912ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
682025-12-18T00:13:24.918ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::60:5eff:fe6e:12b2]
692025-12-18T00:13:24.921ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
702025-12-18T00:13:25.500ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
712025-12-18T00:13:25.505ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::60:5eff:fe6e:12b2]
722025-12-18T00:13:25.508ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
732025-12-18T00:13:26.211ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
742025-12-18T00:13:26.216ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::60:5eff:fe6e:12b2]
752025-12-18T00:13:26.219ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
762025-12-18T00:13:26.604ZDEBGtfportd (internal-dns-resolver): lookup_socket_v6 srv dns_name = _nexus._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("23b8d0ff-58c8-4de2-81b5-34ba69619591.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("588ab1ed-2f1b-4d8b-8357-784ece272f22.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("bc6533e2-7da6-49b7-b39e-67c53f327bfd.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("bc6533e2-7da6-49b7-b39e-67c53f327bfd.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::b)) }], valid_until: Instant { tv_sec: 887, tv_nsec: 412370360 } }) unit = producer-server
772025-12-18T00:13:26.605ZDEBGtfportd (producer-registration-task): will register with Nexus at [fd00:1122:3344:101::a]:12221 unit = producer-server
782025-12-18T00:13:26.636ZDEBGtfportd (producer-registration-task): client request body = Some(Body) method = POST unit = producer-server uri = http://[fd00:1122:3344:101::a]:12221/metrics/producers
792025-12-18T00:13:27.697ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
802025-12-18T00:13:27.702ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::60:5eff:fe6e:12b2]
812025-12-18T00:13:27.705ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
822025-12-18T00:13:28.392ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
832025-12-18T00:13:28.397ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::60:5eff:fe6e:12b2]
842025-12-18T00:13:28.400ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
852025-12-18T00:13:29.072ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
862025-12-18T00:13:29.077ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::60:5eff:fe6e:12b2]
872025-12-18T00:13:29.080ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
882025-12-18T00:13:30.366ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
892025-12-18T00:13:30.371ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::60:5eff:fe6e:12b2]
902025-12-18T00:13:30.374ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
912025-12-18T00:13:31.609ZDEBGtfportd (kstat-sampler-worker): pulled samples from target id = 11573805073432295311 n_samples = 6 unit = oximeter
922025-12-18T00:13:31.819ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
932025-12-18T00:13:31.824ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::60:5eff:fe6e:12b2]
942025-12-18T00:13:31.828ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
952025-12-18T00:13:32.193ZDEBGtfportd (kstat-sampler-worker): pulled samples from target id = 13948335495314812778 n_samples = 6 unit = oximeter
962025-12-18T00:13:33.079ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
972025-12-18T00:13:33.084ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::60:5eff:fe6e:12b2]
982025-12-18T00:13:33.087ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
992025-12-18T00:13:34.448ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1002025-12-18T00:13:34.452ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::60:5eff:fe6e:12b2]
1012025-12-18T00:13:34.455ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1022025-12-18T00:13:35.816ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1032025-12-18T00:13:35.821ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::60:5eff:fe6e:12b2]
1042025-12-18T00:13:35.824ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1052025-12-18T00:13:37.126ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1062025-12-18T00:13:37.130ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::60:5eff:fe6e:12b2]
1072025-12-18T00:13:37.133ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1082025-12-18T00:13:38.828ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1092025-12-18T00:13:38.832ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::60:5eff:fe6e:12b2]
1102025-12-18T00:13:38.835ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1112025-12-18T00:13:40.033ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1122025-12-18T00:13:40.039ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::60:5eff:fe6e:12b2]
1132025-12-18T00:13:40.041ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1142025-12-18T00:13:41.012ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1152025-12-18T00:13:41.017ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::60:5eff:fe6e:12b2]
1162025-12-18T00:13:41.020ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1172025-12-18T00:13:41.685ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1182025-12-18T00:13:41.690ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::60:5eff:fe6e:12b2]
1192025-12-18T00:13:41.693ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1202025-12-18T00:13:42.345ZDEBGtfportd (kstat-sampler-worker): pulled samples from target id = 11573805073432295311 n_samples = 6 unit = oximeter
1212025-12-18T00:13:42.370ZDEBGtfportd (producer-registration-task): client response result = Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::a]:12221/metrics/producers", source: TimedOut }) unit = producer-server
1222025-12-18T00:13:42.371ZWARNtfportd (producer-registration-task): failed to register with Nexus, will retry count = 0 delay = 20.036918934s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers)" file = /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/e76962a/oximeter/producer/src/lib.rs:371 unit = producer-server
1232025-12-18T00:13:42.592ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1242025-12-18T00:13:42.596ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::60:5eff:fe6e:12b2]
1252025-12-18T00:13:42.599ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1262025-12-18T00:13:42.727ZDEBGtfportd (internal-dns-resolver): lookup_socket_v6 srv dns_name = _nexus._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("23b8d0ff-58c8-4de2-81b5-34ba69619591.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("588ab1ed-2f1b-4d8b-8357-784ece272f22.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("bc6533e2-7da6-49b7-b39e-67c53f327bfd.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("bc6533e2-7da6-49b7-b39e-67c53f327bfd.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::b)) }], valid_until: Instant { tv_sec: 902, tv_nsec: 801041887 } }) unit = producer-server
1272025-12-18T00:13:42.728ZDEBGtfportd (producer-registration-task): will register with Nexus at [fd00:1122:3344:101::a]:12221 unit = producer-server
1282025-12-18T00:13:42.758ZDEBGtfportd (producer-registration-task): client request body = Some(Body) method = POST unit = producer-server uri = http://[fd00:1122:3344:101::a]:12221/metrics/producers
1292025-12-18T00:13:42.930ZDEBGtfportd (kstat-sampler-worker): pulled samples from target id = 13948335495314812778 n_samples = 6 unit = oximeter
1302025-12-18T00:13:43.944ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1312025-12-18T00:13:43.953ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::60:5eff:fe6e:12b2]
1322025-12-18T00:13:43.958ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1332025-12-18T00:13:45.415ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1342025-12-18T00:13:45.421ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::60:5eff:fe6e:12b2]
1352025-12-18T00:13:45.424ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1362025-12-18T00:13:46.015ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1372025-12-18T00:13:46.021ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::60:5eff:fe6e:12b2]
1382025-12-18T00:13:46.025ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1392025-12-18T00:13:46.910ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1402025-12-18T00:13:46.916ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::60:5eff:fe6e:12b2]
1412025-12-18T00:13:46.920ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1422025-12-18T00:13:47.670ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1432025-12-18T00:13:47.681ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::60:5eff:fe6e:12b2]
1442025-12-18T00:13:47.687ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1452025-12-18T00:13:48.734ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1462025-12-18T00:13:48.740ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::60:5eff:fe6e:12b2]
1472025-12-18T00:13:48.744ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1482025-12-18T00:13:49.362ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1492025-12-18T00:13:49.372ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::60:5eff:fe6e:12b2]
1502025-12-18T00:13:49.378ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1512025-12-18T00:13:50.459ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1522025-12-18T00:13:50.467ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::60:5eff:fe6e:12b2]
1532025-12-18T00:13:50.471ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1542025-12-18T00:13:51.739ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1552025-12-18T00:13:51.745ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::60:5eff:fe6e:12b2]
1562025-12-18T00:13:51.749ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1572025-12-18T00:13:52.360ZDEBGtfportd (kstat-sampler-worker): pulled samples from target id = 11573805073432295311 n_samples = 6 unit = oximeter
1582025-12-18T00:13:52.942ZDEBGtfportd (kstat-sampler-worker): pulled samples from target id = 13948335495314812778 n_samples = 6 unit = oximeter
1592025-12-18T00:13:53.116ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1602025-12-18T00:13:53.121ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::60:5eff:fe6e:12b2]
1612025-12-18T00:13:53.124ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1622025-12-18T00:13:53.731ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1632025-12-18T00:13:53.735ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::60:5eff:fe6e:12b2]
1642025-12-18T00:13:53.738ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1652025-12-18T00:13:54.687ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1662025-12-18T00:13:54.693ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::60:5eff:fe6e:12b2]
1672025-12-18T00:13:54.696ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1682025-12-18T00:13:55.245ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1692025-12-18T00:13:55.251ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::60:5eff:fe6e:12b2]
1702025-12-18T00:13:55.254ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1712025-12-18T00:13:56.122ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1722025-12-18T00:13:56.132ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::60:5eff:fe6e:12b2]
1732025-12-18T00:13:56.135ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1742025-12-18T00:13:56.731ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1752025-12-18T00:13:56.737ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::60:5eff:fe6e:12b2]
1762025-12-18T00:13:56.740ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1772025-12-18T00:13:57.771ZDEBGtfportd (producer-registration-task): client response result = Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::a]:12221/metrics/producers", source: TimedOut }) unit = producer-server
1782025-12-18T00:13:57.773ZWARNtfportd (producer-registration-task): failed to register with Nexus, will retry count = 1 delay = 35.426054847s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers)" file = /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/e76962a/oximeter/producer/src/lib.rs:371 unit = producer-server
1792025-12-18T00:13:57.949ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1802025-12-18T00:13:57.955ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::60:5eff:fe6e:12b2]
1812025-12-18T00:13:57.960ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1822025-12-18T00:13:58.115ZDEBGtfportd (internal-dns-resolver): lookup_socket_v6 srv dns_name = _nexus._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("23b8d0ff-58c8-4de2-81b5-34ba69619591.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("588ab1ed-2f1b-4d8b-8357-784ece272f22.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("bc6533e2-7da6-49b7-b39e-67c53f327bfd.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("bc6533e2-7da6-49b7-b39e-67c53f327bfd.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::b)) }], valid_until: Instant { tv_sec: 918, tv_nsec: 176301223 } }) unit = producer-server
1832025-12-18T00:13:58.119ZDEBGtfportd (producer-registration-task): will register with Nexus at [fd00:1122:3344:101::a]:12221 unit = producer-server
1842025-12-18T00:13:58.305ZDEBGtfportd (producer-registration-task): client request body = Some(Body) method = POST unit = producer-server uri = http://[fd00:1122:3344:101::a]:12221/metrics/producers
1852025-12-18T00:13:58.800ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1862025-12-18T00:13:58.810ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::60:5eff:fe6e:12b2]
1872025-12-18T00:13:58.820ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1882025-12-18T00:13:59.100ZINFOtfportd: removed stale NDP entry :fe80::ec5b:28ff:fe1d:5753->ee:5b:28:1d:57:53
1892025-12-18T00:13:59.434ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1902025-12-18T00:13:59.443ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::60:5eff:fe6e:12b2]
1912025-12-18T00:13:59.448ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1922025-12-18T00:13:59.691ZINFOtfportd: added new NDP entry entry = tfportrear0_0:fe80::ec5b:28ff:fe1d:5753->ee:5b:28:1d:57:53 interface = tfportrear0_0
1932025-12-18T00:14:00.454ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1942025-12-18T00:14:00.461ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::60:5eff:fe6e:12b2]
1952025-12-18T00:14:00.465ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1962025-12-18T00:14:01.629ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1972025-12-18T00:14:01.636ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::60:5eff:fe6e:12b2]
1982025-12-18T00:14:01.639ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1992025-12-18T00:14:02.366ZDEBGtfportd (kstat-sampler-worker): pulled samples from target id = 11573805073432295311 n_samples = 6 unit = oximeter
2002025-12-18T00:14:02.588ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2012025-12-18T00:14:02.594ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::60:5eff:fe6e:12b2]
2022025-12-18T00:14:02.597ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
2032025-12-18T00:14:02.948ZDEBGtfportd (kstat-sampler-worker): pulled samples from target id = 13948335495314812778 n_samples = 6 unit = oximeter
2042025-12-18T00:14:03.254ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2052025-12-18T00:14:03.259ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::60:5eff:fe6e:12b2]
2062025-12-18T00:14:03.262ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
2072025-12-18T00:14:03.785ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2082025-12-18T00:14:03.790ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::60:5eff:fe6e:12b2]
2092025-12-18T00:14:03.793ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
2102025-12-18T00:14:04.626ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2112025-12-18T00:14:04.631ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::60:5eff:fe6e:12b2]
2122025-12-18T00:14:04.634ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
2132025-12-18T00:14:06.021ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2142025-12-18T00:14:06.026ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::60:5eff:fe6e:12b2]
2152025-12-18T00:14:06.029ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
2162025-12-18T00:14:07.027ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2172025-12-18T00:14:07.032ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::60:5eff:fe6e:12b2]
2182025-12-18T00:14:07.035ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
2192025-12-18T00:14:07.725ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2202025-12-18T00:14:07.730ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::60:5eff:fe6e:12b2]
2212025-12-18T00:14:07.734ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
2222025-12-18T00:14:08.679ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2232025-12-18T00:14:08.684ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::60:5eff:fe6e:12b2]
2242025-12-18T00:14:08.688ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
2252025-12-18T00:14:09.361ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2262025-12-18T00:14:09.367ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::60:5eff:fe6e:12b2]
2272025-12-18T00:14:09.371ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
2282025-12-18T00:14:09.954ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2292025-12-18T00:14:09.960ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::60:5eff:fe6e:12b2]
2302025-12-18T00:14:09.963ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
2312025-12-18T00:14:10.924ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2322025-12-18T00:14:10.929ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::60:5eff:fe6e:12b2]
2332025-12-18T00:14:10.932ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
2342025-12-18T00:14:12.367ZDEBGtfportd (kstat-sampler-worker): pulled samples from target id = 11573805073432295311 n_samples = 6 unit = oximeter
2352025-12-18T00:14:12.411ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2362025-12-18T00:14:12.417ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::60:5eff:fe6e:12b2]
2372025-12-18T00:14:12.430ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
2382025-12-18T00:14:12.949ZDEBGtfportd (kstat-sampler-worker): pulled samples from target id = 13948335495314812778 n_samples = 6 unit = oximeter
2392025-12-18T00:14:13.306ZDEBGtfportd (producer-registration-task): client response result = Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::a]:12221/metrics/producers", source: TimedOut }) unit = producer-server
2402025-12-18T00:14:13.307ZWARNtfportd (producer-registration-task): failed to register with Nexus, will retry count = 2 delay = 50.960787332s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers)" file = /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/e76962a/oximeter/producer/src/lib.rs:371 unit = producer-server
2412025-12-18T00:14:13.682ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2422025-12-18T00:14:13.687ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::60:5eff:fe6e:12b2]
2432025-12-18T00:14:13.690ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
2442025-12-18T00:14:14.650ZDEBGtfportd (internal-dns-resolver): lookup_socket_v6 srv dns_name = _nexus._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("23b8d0ff-58c8-4de2-81b5-34ba69619591.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("588ab1ed-2f1b-4d8b-8357-784ece272f22.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("bc6533e2-7da6-49b7-b39e-67c53f327bfd.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("bc6533e2-7da6-49b7-b39e-67c53f327bfd.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::b)) }], valid_until: Instant { tv_sec: 934, tv_nsec: 712168970 } }) unit = producer-server
2452025-12-18T00:14:14.651ZDEBGtfportd (producer-registration-task): will register with Nexus at [fd00:1122:3344:101::a]:12221 unit = producer-server
2462025-12-18T00:14:14.683ZDEBGtfportd (producer-registration-task): client request body = Some(Body) method = POST unit = producer-server uri = http://[fd00:1122:3344:101::a]:12221/metrics/producers
2472025-12-18T00:14:15.182ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2482025-12-18T00:14:15.187ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::60:5eff:fe6e:12b2]
2492025-12-18T00:14:15.190ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
2502025-12-18T00:14:16.575ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2512025-12-18T00:14:16.581ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::60:5eff:fe6e:12b2]
2522025-12-18T00:14:16.584ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
2532025-12-18T00:14:17.755ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2542025-12-18T00:14:17.760ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::60:5eff:fe6e:12b2]
2552025-12-18T00:14:17.763ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
2562025-12-18T00:14:18.407ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2572025-12-18T00:14:18.413ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::60:5eff:fe6e:12b2]
2582025-12-18T00:14:18.416ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
2592025-12-18T00:14:18.936ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2602025-12-18T00:14:18.942ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::60:5eff:fe6e:12b2]
2612025-12-18T00:14:18.945ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
2622025-12-18T00:14:20.091ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2632025-12-18T00:14:20.096ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::60:5eff:fe6e:12b2]
2642025-12-18T00:14:20.099ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
2652025-12-18T00:14:21.377ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2662025-12-18T00:14:21.383ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::60:5eff:fe6e:12b2]
2672025-12-18T00:14:21.386ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
2682025-12-18T00:14:22.369ZDEBGtfportd (kstat-sampler-worker): pulled samples from target id = 11573805073432295311 n_samples = 6 unit = oximeter
2692025-12-18T00:14:22.721ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2702025-12-18T00:14:22.727ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::60:5eff:fe6e:12b2]
2712025-12-18T00:14:22.730ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
2722025-12-18T00:14:22.952ZDEBGtfportd (kstat-sampler-worker): pulled samples from target id = 13948335495314812778 n_samples = 6 unit = oximeter
2732025-12-18T00:14:23.808ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2742025-12-18T00:14:23.813ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::60:5eff:fe6e:12b2]
2752025-12-18T00:14:23.816ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
2762025-12-18T00:14:25.031ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2772025-12-18T00:14:25.036ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::60:5eff:fe6e:12b2]
2782025-12-18T00:14:25.039ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
2792025-12-18T00:14:25.712ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2802025-12-18T00:14:25.717ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::60:5eff:fe6e:12b2]
2812025-12-18T00:14:25.720ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
2822025-12-18T00:14:26.410ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2832025-12-18T00:14:26.415ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::60:5eff:fe6e:12b2]
2842025-12-18T00:14:26.418ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
2852025-12-18T00:14:27.830ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2862025-12-18T00:14:27.836ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::60:5eff:fe6e:12b2]
2872025-12-18T00:14:27.838ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
2882025-12-18T00:14:28.631ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2892025-12-18T00:14:28.637ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::60:5eff:fe6e:12b2]
2902025-12-18T00:14:28.640ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
2912025-12-18T00:14:29.684ZDEBGtfportd (producer-registration-task): client response result = Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::a]:12221/metrics/producers", source: TimedOut }) unit = producer-server
2922025-12-18T00:14:29.685ZWARNtfportd (producer-registration-task): failed to register with Nexus, will retry count = 3 delay = 67.337859164s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers)" file = /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/e76962a/oximeter/producer/src/lib.rs:371 unit = producer-server
2932025-12-18T00:14:30.078ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2942025-12-18T00:14:30.083ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::60:5eff:fe6e:12b2]
2952025-12-18T00:14:30.086ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
2962025-12-18T00:14:31.070ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2972025-12-18T00:14:31.076ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::60:5eff:fe6e:12b2]
2982025-12-18T00:14:31.080ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
2992025-12-18T00:14:31.483ZDEBGtfportd (internal-dns-resolver): lookup_socket_v6 srv dns_name = _nexus._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("23b8d0ff-58c8-4de2-81b5-34ba69619591.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("588ab1ed-2f1b-4d8b-8357-784ece272f22.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("bc6533e2-7da6-49b7-b39e-67c53f327bfd.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("bc6533e2-7da6-49b7-b39e-67c53f327bfd.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::b)) }], valid_until: Instant { tv_sec: 951, tv_nsec: 545408875 } }) unit = producer-server
3002025-12-18T00:14:31.484ZDEBGtfportd (producer-registration-task): will register with Nexus at [fd00:1122:3344:101::a]:12221 unit = producer-server
3012025-12-18T00:14:31.513ZDEBGtfportd (producer-registration-task): client request body = Some(Body) method = POST unit = producer-server uri = http://[fd00:1122:3344:101::a]:12221/metrics/producers
3022025-12-18T00:14:31.763ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
3032025-12-18T00:14:31.768ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::60:5eff:fe6e:12b2]
3042025-12-18T00:14:31.772ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
3052025-12-18T00:14:32.372ZDEBGtfportd (kstat-sampler-worker): pulled samples from target id = 11573805073432295311 n_samples = 6 unit = oximeter
3062025-12-18T00:14:32.955ZDEBGtfportd (kstat-sampler-worker): pulled samples from target id = 13948335495314812778 n_samples = 6 unit = oximeter
3072025-12-18T00:14:32.966ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
3082025-12-18T00:14:32.971ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::60:5eff:fe6e:12b2]
3092025-12-18T00:14:32.974ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
3102025-12-18T00:14:33.697ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
3112025-12-18T00:14:33.703ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::60:5eff:fe6e:12b2]
3122025-12-18T00:14:33.706ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
3132025-12-18T00:14:34.891ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
3142025-12-18T00:14:34.896ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::60:5eff:fe6e:12b2]
3152025-12-18T00:14:34.899ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
3162025-12-18T00:14:36.082ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
3172025-12-18T00:14:36.089ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::60:5eff:fe6e:12b2]
3182025-12-18T00:14:36.091ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
3192025-12-18T00:14:37.151ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
3202025-12-18T00:14:37.155ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::60:5eff:fe6e:12b2]
3212025-12-18T00:14:37.158ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
3222025-12-18T00:14:38.124ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
3232025-12-18T00:14:38.129ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::60:5eff:fe6e:12b2]
3242025-12-18T00:14:38.132ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
3252025-12-18T00:14:39.592ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
3262025-12-18T00:14:39.598ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::60:5eff:fe6e:12b2]
3272025-12-18T00:14:39.602ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
3282025-12-18T00:14:40.897ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
3292025-12-18T00:14:40.903ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::60:5eff:fe6e:12b2]
3302025-12-18T00:14:40.906ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
3312025-12-18T00:14:42.096ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
3322025-12-18T00:14:42.104ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::60:5eff:fe6e:12b2]
3332025-12-18T00:14:42.109ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
3342025-12-18T00:14:42.376ZDEBGtfportd (kstat-sampler-worker): pulled samples from target id = 11573805073432295311 n_samples = 6 unit = oximeter
3352025-12-18T00:14:42.959ZDEBGtfportd (kstat-sampler-worker): pulled samples from target id = 13948335495314812778 n_samples = 6 unit = oximeter
3362025-12-18T00:14:45.747ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
3372025-12-18T00:14:45.757ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::60:5eff:fe6e:12b2]
3382025-12-18T00:14:45.765ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
3392025-12-18T00:14:46.541ZDEBGtfportd (producer-registration-task): client response result = Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::a]:12221/metrics/producers", source: TimedOut }) unit = producer-server
3402025-12-18T00:14:46.551ZWARNtfportd (producer-registration-task): failed to register with Nexus, will retry count = 4 delay = 84.203310954s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers)" file = /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/e76962a/oximeter/producer/src/lib.rs:371 unit = producer-server
3412025-12-18T00:14:46.846ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
3422025-12-18T00:14:46.854ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::60:5eff:fe6e:12b2]
3432025-12-18T00:14:46.866ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
3442025-12-18T00:14:47.773ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
3452025-12-18T00:14:47.776ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::60:5eff:fe6e:12b2]
3462025-12-18T00:14:47.782ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
3472025-12-18T00:14:49.273ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
3482025-12-18T00:14:49.285ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::60:5eff:fe6e:12b2]
3492025-12-18T00:14:49.293ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
3502025-12-18T00:14:50.313ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
3512025-12-18T00:14:50.322ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::60:5eff:fe6e:12b2]
3522025-12-18T00:14:50.329ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
3532025-12-18T00:14:50.773ZDEBGtfportd (internal-dns-resolver): lookup_socket_v6 srv dns_name = _nexus._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("23b8d0ff-58c8-4de2-81b5-34ba69619591.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("588ab1ed-2f1b-4d8b-8357-784ece272f22.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("bc6533e2-7da6-49b7-b39e-67c53f327bfd.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("bc6533e2-7da6-49b7-b39e-67c53f327bfd.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::b)) }], valid_until: Instant { tv_sec: 970, tv_nsec: 830652119 } }) unit = producer-server
3542025-12-18T00:14:50.775ZDEBGtfportd (producer-registration-task): will register with Nexus at [fd00:1122:3344:101::a]:12221 unit = producer-server
3552025-12-18T00:14:50.870ZDEBGtfportd (producer-registration-task): client request body = Some(Body) method = POST unit = producer-server uri = http://[fd00:1122:3344:101::a]:12221/metrics/producers
3562025-12-18T00:14:51.203ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
3572025-12-18T00:14:51.211ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::60:5eff:fe6e:12b2]
3582025-12-18T00:14:51.215ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
3592025-12-18T00:14:52.554ZDEBGtfportd (kstat-sampler-worker): pulled samples from target id = 11573805073432295311 n_samples = 6 unit = oximeter
3602025-12-18T00:14:52.567ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
3612025-12-18T00:14:52.574ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::60:5eff:fe6e:12b2]
3622025-12-18T00:14:52.578ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
3632025-12-18T00:14:52.993ZDEBGtfportd (kstat-sampler-worker): pulled samples from target id = 13948335495314812778 n_samples = 6 unit = oximeter
3642025-12-18T00:14:53.911ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
3652025-12-18T00:14:53.917ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::60:5eff:fe6e:12b2]
3662025-12-18T00:14:53.920ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
3672025-12-18T00:14:54.721ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
3682025-12-18T00:14:54.730ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::60:5eff:fe6e:12b2]
3692025-12-18T00:14:54.735ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
3702025-12-18T00:14:55.948ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
3712025-12-18T00:14:55.955ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::60:5eff:fe6e:12b2]
3722025-12-18T00:14:55.960ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
3732025-12-18T00:14:56.642ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
3742025-12-18T00:14:56.647ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::60:5eff:fe6e:12b2]
3752025-12-18T00:14:56.650ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
3762025-12-18T00:14:57.723ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
3772025-12-18T00:14:57.739ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::60:5eff:fe6e:12b2]
3782025-12-18T00:14:57.747ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
3792025-12-18T00:14:58.820ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
3802025-12-18T00:14:58.824ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::60:5eff:fe6e:12b2]
3812025-12-18T00:14:58.828ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
3822025-12-18T00:14:59.959ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
3832025-12-18T00:14:59.966ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::60:5eff:fe6e:12b2]
3842025-12-18T00:14:59.970ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]