Max level shown:
1 [ Jan 7 21:58:49 Enabled. ]
2 [ Jan 7 21:58:49 Rereading configuration. ]
3 [ Jan 7 21:58:50 Rereading configuration. ]
4 [ Jan 7 21:59:01 Executing start method ("ctrun -l child -o noorphan,regent /opt/oxide/dendrite/bin/tfportd --sync-only &"). ]
5 [ Jan 7 21:59:01 Method "start" exited with status 0. ]
6 No non-localhost IPv6 addresses found in SMF properties
72026-01-07T21:59:01.659ZINFOtfportd: connecting to dpd host = [::1] port = 12224
82026-01-07T21:59:01.700ZINFOtfportd: connected to dpd running softnpu-c9bf538d
92026-01-07T21:59:01.701ZINFOtfportd: running in sync-only mode
102026-01-07T21:59:01.703ZINFOtfportd: no non-localhost IPv6 listen address available, waiting for SMF update unit = oximeter
112026-01-07T21:59:01.703ZINFOtfportd: refreshing SMF config prior to waiting on signals unit = signal_handler
12 No non-localhost IPv6 addresses found in SMF properties
132026-01-07T21:59:01.705ZINFOtfportd: refreshed config: Config { log_file: None, log_format: Json, listen_addresses: [ [::1]:12231, ], pkt_source: None, dpd_host: "[::1]", dpd_port: 12224, vlan_link: None, vlan_data: None, techport0_prefix: None, techport1_prefix: None, rack_id: Some( 41ed28eb-1770-4c2d-a260-ad012f7c8877, ), sled_id: Some( 258ceed3-b958-4061-8e31-5005a7ca97c3, ), sled_model: Some( "i86pc", ), sled_revision: Some( 0, ), sled_serial: Some( "buskin", ), }
142026-01-07T21:59:01.705ZINFOtfportd: no non-localhost IPv6 listen address available, waiting for SMF update unit = oximeter
152026-01-07T21:59:01.709ZINFOtfportd: added new NDP entry entry = tfportrear0_0:fe80::e8f9:16ff:fe60:2b4a->ea:f9:16:60:2b:4a interface = tfportrear0_0
162026-01-07T21:59:01.713ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
172026-01-07T21:59:01.719ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::e8f9:16ff:fe60:2b4a]
182026-01-07T21:59:01.726ZWARNtfportd: tfportqsfp0_0 has no addrs
19 [ Jan 7 21:59:02 Rereading configuration. ]
20 [ Jan 7 21:59:02 Executing refresh method (:kill). ]
21 [ Jan 7 21:59:02 Stopping because all processes in service exited. ]
22 [ Jan 7 21:59:02 Executing stop method (:kill). ]
23 [ Jan 7 21:59:02 Executing start method ("ctrun -l child -o noorphan,regent /opt/oxide/dendrite/bin/tfportd --sync-only &"). ]
24 [ Jan 7 21:59:02 Method "start" exited with status 0. ]
252026-01-07T21:59:02.324ZINFOtfportd: connecting to dpd host = [::1] port = 12224
262026-01-07T21:59:02.363ZINFOtfportd: connected to dpd running softnpu-c9bf538d
272026-01-07T21:59:02.363ZINFOtfportd: running in sync-only mode
282026-01-07T21:59:02.363ZINFOtfportd: found suitable IPv6 address for oximeter producer server address = fd00:1122:3344:101::2 unit = oximeter
292026-01-07T21:59:02.364ZINFOtfportd: refreshing SMF config prior to waiting on signals unit = signal_handler
302026-01-07T21:59:02.365ZDEBGtfportd: extracted oximeter configuration data config = OximeterConfig { listen_address: fd00:1122:3344:101::2, sled_identifiers: SledIdentifiers { rack_id: 41ed28eb-1770-4c2d-a260-ad012f7c8877, sled_id: 258ceed3-b958-4061-8e31-5005a7ca97c3, model: "i86pc", revision: 0, serial: "buskin" }, switch_identifiers: SwitchIdentifiers { asic_backend: "softnpu", fab: None, lot: None, model: "FAKE_SIM_SIDECAR", revision: 0, serial: "SimSidecar0", sidecar_id: eca88c96-53fa-481b-8c3a-e42f1961fd3d, slot: 0, wafer: None, wafer_loc: None } } unit = oximeter
312026-01-07T21:59:02.365ZDEBGtfportd: created producer ID producer_id = 88c94aae-fdd2-4457-a06b-192b91d7e289 unit = oximeter
322026-01-07T21:59:02.366ZINFOtfportd: refreshed config: Config { log_file: None, log_format: Json, listen_addresses: [ [fd00:1122:3344:101::2]:12231, [::1]:12231, ], pkt_source: None, dpd_host: "[::1]", dpd_port: 12224, vlan_link: None, vlan_data: None, techport0_prefix: None, techport1_prefix: None, rack_id: Some( 41ed28eb-1770-4c2d-a260-ad012f7c8877, ), sled_id: Some( 258ceed3-b958-4061-8e31-5005a7ca97c3, ), sled_model: Some( "i86pc", ), sled_revision: Some( 0, ), sled_serial: Some( "buskin", ), }
332026-01-07T21:59:02.374ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
342026-01-07T21:59:02.374ZINFOtfportd: added new NDP entry entry = tfportrear0_0:fe80::508c:37ff:fe7e:fdee->52:8c:37:7e:fd:ee interface = tfportrear0_0
352026-01-07T21:59:02.375ZINFOtfportd: added new NDP entry entry = tfportrear0_0:fe80::e8f9:16ff:fe60:2b4a->ea:f9:16:60:2b:4a interface = tfportrear0_0
362026-01-07T21:59:02.376ZDEBGtfportd: registered DTrace probes unit = producer-server
372026-01-07T21:59:02.376ZDEBGtfportd (dropshot): registered endpoint local_addr = [fd00:1122:3344:101::2]:33844 method = GET path = /{producer_id} unit = producer-server versions = all
382026-01-07T21:59:02.376ZINFOtfportd (dropshot): listening file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.4/src/server.rs:328 local_addr = [fd00:1122:3344:101::2]:33844 unit = producer-server
392026-01-07T21:59:02.377ZDEBGtfportd (dropshot): successfully registered DTrace USDT probes local_addr = [fd00:1122:3344:101::2]:33844 unit = producer-server
402026-01-07T21:59:02.377ZDEBGtfportd: Requested any available port, Dropshot server has been bound to [fd00:1122:3344:101::2]:33844 unit = producer-server
412026-01-07T21:59:02.377ZDEBGtfportd: Nexus IP not provided, will use DNS to resolve it unit = producer-server
422026-01-07T21:59:02.377ZINFOtfportd (internal-dns-resolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53] file = /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/64b40cb/internal-dns/resolver/src/resolver.rs:144 unit = producer-server
432026-01-07T21:59:02.378ZDEBGtfportd: starting producer registration task unit = producer-server
442026-01-07T21:59:02.378ZINFOtfportd: starting oximeter metric producer server address = [fd00:1122:3344:101::2]:33844 file = /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/64b40cb/oximeter/producer/src/lib.rs:279 interval = 30s producer_id = 88c94aae-fdd2-4457-a06b-192b91d7e289 unit = producer-server
452026-01-07T21:59:02.378ZINFOtfportd: created oximeter producer server unit = oximeter
462026-01-07T21:59:02.378ZDEBGtfportd (producer-registration-task): registering as oximeter producer with Nexus registration_count = 0 unit = producer-server
472026-01-07T21:59:02.379ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::e8f9:16ff:fe60:2b4a]
482026-01-07T21:59:02.379ZDEBGtfportd (kstat-sampler): adding new target unit = oximeter
492026-01-07T21:59:02.379ZDEBGtfportd (kstat-sampler-worker): storing new creation time for tracked target creation_time = 2026-01-07T21:58:45.625581035Z path = KstatPath { module: "link", instance: 0, name: "tfportrear0_0" } unit = oximeter
502026-01-07T21:59:02.380ZDEBGtfportd (kstat-sampler-worker): inserted empty per-target sample queue id = 13744581995329210214 unit = oximeter
512026-01-07T21:59:02.380ZDEBGtfportd: added new link to kstat sampler link_kind = switch-port-control link_name = tfportrear0_0 unit = metrics-task-tracker
522026-01-07T21:59:02.382ZWARNtfportd: tfportqsfp0_0 has no addrs
532026-01-07T21:59:02.920ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
542026-01-07T21:59:02.925ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::e8f9:16ff:fe60:2b4a]
552026-01-07T21:59:02.929ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
562026-01-07T21:59:02.929ZDEBGtfportd (kstat-sampler): adding new target unit = oximeter
572026-01-07T21:59:02.930ZDEBGtfportd (kstat-sampler-worker): storing new creation time for tracked target creation_time = 2026-01-07T21:58:45.625444703Z path = KstatPath { module: "link", instance: 0, name: "tfportqsfp0_0" } unit = oximeter
582026-01-07T21:59:02.930ZDEBGtfportd (kstat-sampler-worker): inserted empty per-target sample queue id = 13745677142907762949 unit = oximeter
592026-01-07T21:59:02.930ZDEBGtfportd: added new link to kstat sampler link_kind = switch-port-control link_name = tfportqsfp0_0 unit = metrics-task-tracker
602026-01-07T21:59:03.023ZINFOtfportd: added new ARP entry entry = 10.151.1.174->ca:36:d5:9d:af:f3 interface = tfportqsfp0_0
612026-01-07T21:59:03.504ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
622026-01-07T21:59:03.509ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::e8f9:16ff:fe60:2b4a]
632026-01-07T21:59:03.512ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
642026-01-07T21:59:04.202ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
652026-01-07T21:59:04.207ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::e8f9:16ff:fe60:2b4a]
662026-01-07T21:59:04.210ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
672026-01-07T21:59:04.769ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
682026-01-07T21:59:04.775ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::e8f9:16ff:fe60:2b4a]
692026-01-07T21:59:04.778ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
702026-01-07T21:59:05.776ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
712026-01-07T21:59:05.781ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::e8f9:16ff:fe60:2b4a]
722026-01-07T21:59:05.784ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
732026-01-07T21:59:06.676ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
742026-01-07T21:59:06.680ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::e8f9:16ff:fe60:2b4a]
752026-01-07T21:59:06.683ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
762026-01-07T21:59:07.381ZDEBGtfportd (internal-dns-resolver): lookup_socket_v6 srv dns_name = _nexus._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("12108c89-0861-4204-8076-0f158628b7df.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("72c00207-46dd-4ed5-9961-32d35103ed6b.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("ca0b25e5-526f-4db7-a30f-ebed4edf5319.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("ca0b25e5-526f-4db7-a30f-ebed4edf5319.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::b)) }], valid_until: Instant { tv_sec: 653, tv_nsec: 114873893 } }) unit = producer-server
772026-01-07T21:59:07.381ZDEBGtfportd (producer-registration-task): will register with Nexus at [fd00:1122:3344:101::c]:12221 unit = producer-server
782026-01-07T21:59:07.413ZDEBGtfportd (producer-registration-task): client request body = Some(Body) method = POST unit = producer-server uri = http://[fd00:1122:3344:101::c]:12221/metrics/producers
792026-01-07T21:59:07.691ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
802026-01-07T21:59:07.695ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::e8f9:16ff:fe60:2b4a]
812026-01-07T21:59:07.698ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
822026-01-07T21:59:08.339ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
832026-01-07T21:59:08.344ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::e8f9:16ff:fe60:2b4a]
842026-01-07T21:59:08.347ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
852026-01-07T21:59:08.910ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
862026-01-07T21:59:08.915ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::e8f9:16ff:fe60:2b4a]
872026-01-07T21:59:08.918ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
882026-01-07T21:59:09.687ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
892026-01-07T21:59:09.692ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::e8f9:16ff:fe60:2b4a]
902026-01-07T21:59:09.694ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
912026-01-07T21:59:10.621ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
922026-01-07T21:59:10.626ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::e8f9:16ff:fe60:2b4a]
932026-01-07T21:59:10.628ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
942026-01-07T21:59:11.665ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
952026-01-07T21:59:11.669ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::e8f9:16ff:fe60:2b4a]
962026-01-07T21:59:11.673ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
972026-01-07T21:59:12.384ZDEBGtfportd (kstat-sampler-worker): pulled samples from target id = 13744581995329210214 n_samples = 6 unit = oximeter
982026-01-07T21:59:12.932ZDEBGtfportd (kstat-sampler-worker): pulled samples from target id = 13745677142907762949 n_samples = 6 unit = oximeter
992026-01-07T21:59:13.040ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1002026-01-07T21:59:13.045ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::e8f9:16ff:fe60:2b4a]
1012026-01-07T21:59:13.048ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1022026-01-07T21:59:13.983ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1032026-01-07T21:59:13.988ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::e8f9:16ff:fe60:2b4a]
1042026-01-07T21:59:13.991ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1052026-01-07T21:59:15.094ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1062026-01-07T21:59:15.099ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::e8f9:16ff:fe60:2b4a]
1072026-01-07T21:59:15.102ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1082026-01-07T21:59:15.657ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1092026-01-07T21:59:15.662ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::e8f9:16ff:fe60:2b4a]
1102026-01-07T21:59:15.664ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1112026-01-07T21:59:16.430ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1122026-01-07T21:59:16.433ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::e8f9:16ff:fe60:2b4a]
1132026-01-07T21:59:16.437ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1142026-01-07T21:59:17.432ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1152026-01-07T21:59:17.436ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::e8f9:16ff:fe60:2b4a]
1162026-01-07T21:59:17.439ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1172026-01-07T21:59:18.796ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1182026-01-07T21:59:18.801ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::e8f9:16ff:fe60:2b4a]
1192026-01-07T21:59:18.804ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1202026-01-07T21:59:20.163ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1212026-01-07T21:59:20.168ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::e8f9:16ff:fe60:2b4a]
1222026-01-07T21:59:20.171ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1232026-01-07T21:59:21.273ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1242026-01-07T21:59:21.277ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::e8f9:16ff:fe60:2b4a]
1252026-01-07T21:59:21.280ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1262026-01-07T21:59:22.244ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1272026-01-07T21:59:22.248ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::e8f9:16ff:fe60:2b4a]
1282026-01-07T21:59:22.251ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1292026-01-07T21:59:22.949ZDEBGtfportd (kstat-sampler-worker): pulled samples from target id = 13744581995329210214 n_samples = 6 unit = oximeter
1302026-01-07T21:59:22.976ZDEBGtfportd (producer-registration-task): client response result = Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::c]:12221/metrics/producers", source: TimedOut }) unit = producer-server
1312026-01-07T21:59:22.977ZWARNtfportd (producer-registration-task): failed to register with Nexus, will retry count = 0 delay = 20.036953666s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers)" file = /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/64b40cb/oximeter/producer/src/lib.rs:371 unit = producer-server
1322026-01-07T21:59:23.156ZDEBGtfportd (internal-dns-resolver): lookup_socket_v6 srv dns_name = _nexus._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("12108c89-0861-4204-8076-0f158628b7df.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("72c00207-46dd-4ed5-9961-32d35103ed6b.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("ca0b25e5-526f-4db7-a30f-ebed4edf5319.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("ca0b25e5-526f-4db7-a30f-ebed4edf5319.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::b)) }], valid_until: Instant { tv_sec: 668, tv_nsec: 327627733 } }) unit = producer-server
1332026-01-07T21:59:23.157ZDEBGtfportd (producer-registration-task): will register with Nexus at [fd00:1122:3344:101::c]:12221 unit = producer-server
1342026-01-07T21:59:23.188ZDEBGtfportd (producer-registration-task): client request body = Some(Body) method = POST unit = producer-server uri = http://[fd00:1122:3344:101::c]:12221/metrics/producers
1352026-01-07T21:59:23.331ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1362026-01-07T21:59:23.335ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::e8f9:16ff:fe60:2b4a]
1372026-01-07T21:59:23.339ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1382026-01-07T21:59:23.500ZDEBGtfportd (kstat-sampler-worker): pulled samples from target id = 13745677142907762949 n_samples = 6 unit = oximeter
1392026-01-07T21:59:24.833ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1402026-01-07T21:59:24.833ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::e8f9:16ff:fe60:2b4a]
1412026-01-07T21:59:24.843ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1422026-01-07T21:59:26.185ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1432026-01-07T21:59:26.191ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::e8f9:16ff:fe60:2b4a]
1442026-01-07T21:59:26.195ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1452026-01-07T21:59:27.195ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1462026-01-07T21:59:27.199ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::e8f9:16ff:fe60:2b4a]
1472026-01-07T21:59:27.202ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1482026-01-07T21:59:27.735ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1492026-01-07T21:59:27.740ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::e8f9:16ff:fe60:2b4a]
1502026-01-07T21:59:27.743ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1512026-01-07T21:59:29.153ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1522026-01-07T21:59:29.160ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::e8f9:16ff:fe60:2b4a]
1532026-01-07T21:59:29.164ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1542026-01-07T21:59:30.452ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1552026-01-07T21:59:30.456ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::e8f9:16ff:fe60:2b4a]
1562026-01-07T21:59:30.459ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1572026-01-07T21:59:31.276ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1582026-01-07T21:59:31.283ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::e8f9:16ff:fe60:2b4a]
1592026-01-07T21:59:31.286ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1602026-01-07T21:59:32.678ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1612026-01-07T21:59:32.682ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::e8f9:16ff:fe60:2b4a]
1622026-01-07T21:59:32.685ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1632026-01-07T21:59:32.980ZDEBGtfportd (kstat-sampler-worker): pulled samples from target id = 13744581995329210214 n_samples = 6 unit = oximeter
1642026-01-07T21:59:33.272ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1652026-01-07T21:59:33.277ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::e8f9:16ff:fe60:2b4a]
1662026-01-07T21:59:33.280ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1672026-01-07T21:59:33.530ZDEBGtfportd (kstat-sampler-worker): pulled samples from target id = 13745677142907762949 n_samples = 6 unit = oximeter
1682026-01-07T21:59:34.535ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1692026-01-07T21:59:34.541ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::e8f9:16ff:fe60:2b4a]
1702026-01-07T21:59:34.544ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1712026-01-07T21:59:35.813ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1722026-01-07T21:59:35.818ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::e8f9:16ff:fe60:2b4a]
1732026-01-07T21:59:35.821ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1742026-01-07T21:59:36.882ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1752026-01-07T21:59:36.888ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::e8f9:16ff:fe60:2b4a]
1762026-01-07T21:59:36.891ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1772026-01-07T21:59:38.079ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1782026-01-07T21:59:38.086ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::e8f9:16ff:fe60:2b4a]
1792026-01-07T21:59:38.090ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1802026-01-07T21:59:38.228ZDEBGtfportd (producer-registration-task): client response result = Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::c]:12221/metrics/producers", source: TimedOut }) unit = producer-server
1812026-01-07T21:59:38.233ZWARNtfportd (producer-registration-task): failed to register with Nexus, will retry count = 1 delay = 35.255179517s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers)" file = /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/64b40cb/oximeter/producer/src/lib.rs:371 unit = producer-server
1822026-01-07T21:59:38.701ZDEBGtfportd (internal-dns-resolver): lookup_socket_v6 srv dns_name = _nexus._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("12108c89-0861-4204-8076-0f158628b7df.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("72c00207-46dd-4ed5-9961-32d35103ed6b.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("ca0b25e5-526f-4db7-a30f-ebed4edf5319.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("ca0b25e5-526f-4db7-a30f-ebed4edf5319.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::b)) }], valid_until: Instant { tv_sec: 683, tv_nsec: 834297378 } }) unit = producer-server
1832026-01-07T21:59:38.702ZDEBGtfportd (producer-registration-task): will register with Nexus at [fd00:1122:3344:101::c]:12221 unit = producer-server
1842026-01-07T21:59:38.757ZDEBGtfportd (producer-registration-task): client request body = Some(Body) method = POST unit = producer-server uri = http://[fd00:1122:3344:101::c]:12221/metrics/producers
1852026-01-07T21:59:39.494ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1862026-01-07T21:59:39.501ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::e8f9:16ff:fe60:2b4a]
1872026-01-07T21:59:39.505ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1882026-01-07T21:59:40.155ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1892026-01-07T21:59:40.159ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::e8f9:16ff:fe60:2b4a]
1902026-01-07T21:59:40.161ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1912026-01-07T21:59:41.498ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1922026-01-07T21:59:41.503ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::e8f9:16ff:fe60:2b4a]
1932026-01-07T21:59:41.506ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1942026-01-07T21:59:42.496ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1952026-01-07T21:59:42.501ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::e8f9:16ff:fe60:2b4a]
1962026-01-07T21:59:42.504ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1972026-01-07T21:59:42.993ZDEBGtfportd (kstat-sampler-worker): pulled samples from target id = 13744581995329210214 n_samples = 6 unit = oximeter
1982026-01-07T21:59:43.540ZDEBGtfportd (kstat-sampler-worker): pulled samples from target id = 13745677142907762949 n_samples = 6 unit = oximeter
1992026-01-07T21:59:43.842ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2002026-01-07T21:59:43.847ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::e8f9:16ff:fe60:2b4a]
2012026-01-07T21:59:43.850ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
2022026-01-07T21:59:44.969ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2032026-01-07T21:59:44.974ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::e8f9:16ff:fe60:2b4a]
2042026-01-07T21:59:44.977ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
2052026-01-07T21:59:46.241ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2062026-01-07T21:59:46.246ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::e8f9:16ff:fe60:2b4a]
2072026-01-07T21:59:46.249ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
2082026-01-07T21:59:47.697ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2092026-01-07T21:59:47.731ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::e8f9:16ff:fe60:2b4a]
2102026-01-07T21:59:47.784ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
2112026-01-07T21:59:49.168ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2122026-01-07T21:59:49.173ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::e8f9:16ff:fe60:2b4a]
2132026-01-07T21:59:49.175ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
2142026-01-07T21:59:50.090ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2152026-01-07T21:59:50.095ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::e8f9:16ff:fe60:2b4a]
2162026-01-07T21:59:50.098ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
2172026-01-07T21:59:51.382ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2182026-01-07T21:59:51.390ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::e8f9:16ff:fe60:2b4a]
2192026-01-07T21:59:51.394ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
2202026-01-07T21:59:51.947ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2212026-01-07T21:59:51.955ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::e8f9:16ff:fe60:2b4a]
2222026-01-07T21:59:51.960ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
2232026-01-07T21:59:52.734ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2242026-01-07T21:59:52.739ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::e8f9:16ff:fe60:2b4a]
2252026-01-07T21:59:52.741ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
2262026-01-07T21:59:52.997ZDEBGtfportd (kstat-sampler-worker): pulled samples from target id = 13744581995329210214 n_samples = 6 unit = oximeter
2272026-01-07T21:59:53.546ZDEBGtfportd (kstat-sampler-worker): pulled samples from target id = 13745677142907762949 n_samples = 6 unit = oximeter
2282026-01-07T21:59:53.760ZDEBGtfportd (producer-registration-task): client response result = Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::c]:12221/metrics/producers", source: TimedOut }) unit = producer-server
2292026-01-07T21:59:53.761ZWARNtfportd (producer-registration-task): failed to register with Nexus, will retry count = 2 delay = 50.781004746s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers)" file = /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/64b40cb/oximeter/producer/src/lib.rs:371 unit = producer-server
2302026-01-07T21:59:53.783ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2312026-01-07T21:59:53.787ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::e8f9:16ff:fe60:2b4a]
2322026-01-07T21:59:53.790ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
2332026-01-07T21:59:54.643ZDEBGtfportd (internal-dns-resolver): lookup_socket_v6 srv dns_name = _nexus._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("12108c89-0861-4204-8076-0f158628b7df.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("72c00207-46dd-4ed5-9961-32d35103ed6b.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("ca0b25e5-526f-4db7-a30f-ebed4edf5319.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("ca0b25e5-526f-4db7-a30f-ebed4edf5319.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::b)) }], valid_until: Instant { tv_sec: 699, tv_nsec: 775921923 } }) unit = producer-server
2342026-01-07T21:59:54.644ZDEBGtfportd (producer-registration-task): will register with Nexus at [fd00:1122:3344:101::c]:12221 unit = producer-server
2352026-01-07T21:59:54.675ZDEBGtfportd (producer-registration-task): client request body = Some(Body) method = POST unit = producer-server uri = http://[fd00:1122:3344:101::c]:12221/metrics/producers
2362026-01-07T21:59:54.726ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2372026-01-07T21:59:54.732ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::e8f9:16ff:fe60:2b4a]
2382026-01-07T21:59:54.735ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
2392026-01-07T21:59:55.853ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2402026-01-07T21:59:55.858ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::e8f9:16ff:fe60:2b4a]
2412026-01-07T21:59:55.861ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
2422026-01-07T21:59:56.820ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2432026-01-07T21:59:56.825ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::e8f9:16ff:fe60:2b4a]
2442026-01-07T21:59:56.829ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
2452026-01-07T21:59:57.984ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2462026-01-07T21:59:57.989ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::e8f9:16ff:fe60:2b4a]
2472026-01-07T21:59:57.993ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
2482026-01-07T21:59:59.040ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2492026-01-07T21:59:59.045ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::e8f9:16ff:fe60:2b4a]
2502026-01-07T21:59:59.048ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
2512026-01-07T21:59:59.697ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2522026-01-07T21:59:59.702ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::e8f9:16ff:fe60:2b4a]
2532026-01-07T21:59:59.705ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
2542026-01-07T22:00:00.926ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2552026-01-07T22:00:00.932ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::e8f9:16ff:fe60:2b4a]
2562026-01-07T22:00:00.935ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
2572026-01-07T22:00:01.479ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2582026-01-07T22:00:01.485ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::e8f9:16ff:fe60:2b4a]
2592026-01-07T22:00:01.488ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]