Max level shown:
1 [ Mar 7 01:43:40 Enabled. ]
2 [ Mar 7 01:43:40 Rereading configuration. ]
3 [ Mar 7 01:43:41 Rereading configuration. ]
4 [ Mar 7 01:44:02 Executing start method ("ctrun -l child -o noorphan,regent /opt/oxide/dendrite/bin/tfportd --sync-only &"). ]
5 [ Mar 7 01:44:02 Method "start" exited with status 0. ]
6 No non-localhost IPv6 addresses found in SMF properties
72026-03-07T01:44:02.227ZINFOtfportd: connecting to dpd host = [::1] port = 12224
82026-03-07T01:44:02.231ZINFOtfportd: connected to dpd running softnpu-f20f786e
92026-03-07T01:44:02.232ZINFOtfportd: running in sync-only mode
102026-03-07T01:44:02.232ZINFOtfportd: no non-localhost IPv6 listen address available, waiting for SMF update unit = oximeter
112026-03-07T01:44:02.234ZINFOtfportd: refreshing SMF config prior to waiting on signals unit = signal_handler
12 No non-localhost IPv6 addresses found in SMF properties
132026-03-07T01:44:02.236ZINFOtfportd: refreshed config: Config { log_file: None, log_format: Json, listen_addresses: [ [::1]:12231, ], pkt_source: None, dpd_host: "[::1]", dpd_port: 12224, vlan_link: None, vlan_data: None, techport0_prefix: None, techport1_prefix: None, rack_id: Some( 793748c4-4beb-4252-87b7-4659feafcbd4, ), sled_id: Some( 875a97ff-610f-4476-8995-d9ae9d1e17d2, ), sled_model: Some( "i86pc", ), sled_revision: Some( 0, ), sled_serial: Some( "buskin", ), }
142026-03-07T01:44:02.236ZINFOtfportd: no non-localhost IPv6 listen address available, waiting for SMF update unit = oximeter
152026-03-07T01:44:02.240ZINFOtfportd: added new NDP entry entry = tfportrear0_0:fe80::b0b9:3eff:fe61:d0d->b2:b9:3e:61:0d:0d interface = tfportrear0_0
162026-03-07T01:44:02.242ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
172026-03-07T01:44:02.247ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::b0b9:3eff:fe61:d0d]
182026-03-07T01:44:02.252ZWARNtfportd: tfportqsfp0_0 has no addrs
19 [ Mar 7 01:44:02 Rereading configuration. ]
20 [ Mar 7 01:44:02 Executing refresh method (:kill). ]
21 [ Mar 7 01:44:02 Stopping because all processes in service exited. ]
22 [ Mar 7 01:44:02 Executing stop method (:kill). ]
23 [ Mar 7 01:44:02 Executing start method ("ctrun -l child -o noorphan,regent /opt/oxide/dendrite/bin/tfportd --sync-only &"). ]
24 [ Mar 7 01:44:02 Method "start" exited with status 0. ]
252026-03-07T01:44:02.796ZINFOtfportd: connecting to dpd host = [::1] port = 12224
262026-03-07T01:44:02.797ZINFOtfportd: connected to dpd running softnpu-f20f786e
272026-03-07T01:44:02.797ZINFOtfportd: running in sync-only mode
282026-03-07T01:44:02.797ZINFOtfportd: found suitable IPv6 address for oximeter producer server address = fd00:1122:3344:101::2 unit = oximeter
292026-03-07T01:44:02.799ZINFOtfportd: refreshing SMF config prior to waiting on signals unit = signal_handler
302026-03-07T01:44:02.799ZDEBGtfportd: extracted oximeter configuration data config = OximeterConfig { listen_address: fd00:1122:3344:101::2, sled_identifiers: SledIdentifiers { rack_id: 793748c4-4beb-4252-87b7-4659feafcbd4, sled_id: 875a97ff-610f-4476-8995-d9ae9d1e17d2, model: "i86pc", revision: 0, serial: "buskin" }, switch_identifiers: SwitchIdentifiers { asic_backend: "softnpu", fab: None, lot: None, model: "FAKE_SIM_SIDECAR", revision: 0, serial: "SimSidecar0", sidecar_id: d40e0341-5ba2-4a24-baf0-23290e0910b6, slot: 0, wafer: None, wafer_loc: None } } unit = oximeter
312026-03-07T01:44:02.799ZDEBGtfportd: created producer ID producer_id = 2a63dc7a-7146-4c00-b4d2-bcde6db8ccb0 unit = oximeter
322026-03-07T01:44:02.800ZINFOtfportd: refreshed config: Config { log_file: None, log_format: Json, listen_addresses: [ [fd00:1122:3344:101::2]:12231, [::1]:12231, ], pkt_source: None, dpd_host: "[::1]", dpd_port: 12224, vlan_link: None, vlan_data: None, techport0_prefix: None, techport1_prefix: None, rack_id: Some( 793748c4-4beb-4252-87b7-4659feafcbd4, ), sled_id: Some( 875a97ff-610f-4476-8995-d9ae9d1e17d2, ), sled_model: Some( "i86pc", ), sled_revision: Some( 0, ), sled_serial: Some( "buskin", ), }
332026-03-07T01:44:02.808ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
342026-03-07T01:44:02.809ZINFOtfportd: added new NDP entry entry = tfportrear0_0:fe80::b0b9:3eff:fe61:d0d->b2:b9:3e:61:0d:0d interface = tfportrear0_0
352026-03-07T01:44:02.809ZINFOtfportd: added new NDP entry entry = tfportrear0_0:fe80::b0cd:28ff:fe2b:f2ac->b2:cd:28:2b:f2:ac interface = tfportrear0_0
362026-03-07T01:44:02.811ZDEBGtfportd: registered DTrace probes unit = producer-server
372026-03-07T01:44:02.812ZDEBGtfportd (dropshot): registered endpoint local_addr = [fd00:1122:3344:101::2]:33397 method = GET path = /{producer_id} unit = producer-server versions = all
382026-03-07T01:44:02.813ZINFOtfportd (dropshot): listening file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.7/src/server.rs:328 local_addr = [fd00:1122:3344:101::2]:33397 unit = producer-server
392026-03-07T01:44:02.813ZDEBGtfportd (dropshot): successfully registered DTrace USDT probes local_addr = [fd00:1122:3344:101::2]:33397 unit = producer-server
402026-03-07T01:44:02.813ZDEBGtfportd: Requested any available port, Dropshot server has been bound to [fd00:1122:3344:101::2]:33397 unit = producer-server
412026-03-07T01:44:02.813ZDEBGtfportd: Nexus IP not provided, will use DNS to resolve it unit = producer-server
422026-03-07T01:44:02.813ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::b0b9:3eff:fe61:d0d]
432026-03-07T01:44:02.813ZINFOtfportd (internal-dns-resolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53] file = /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/8e30605/internal-dns/resolver/src/resolver.rs:144 unit = producer-server
442026-03-07T01:44:02.814ZDEBGtfportd: starting producer registration task unit = producer-server
452026-03-07T01:44:02.814ZINFOtfportd: starting oximeter metric producer server address = [fd00:1122:3344:101::2]:33397 file = /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/8e30605/oximeter/producer/src/lib.rs:279 interval = 30s producer_id = 2a63dc7a-7146-4c00-b4d2-bcde6db8ccb0 unit = producer-server
462026-03-07T01:44:02.814ZINFOtfportd: created oximeter producer server unit = oximeter
472026-03-07T01:44:02.815ZDEBGtfportd (producer-registration-task): registering as oximeter producer with Nexus registration_count = 0 unit = producer-server
482026-03-07T01:44:02.815ZDEBGtfportd (kstat-sampler): adding new target unit = oximeter
492026-03-07T01:44:02.816ZDEBGtfportd (kstat-sampler-worker): storing new creation time for tracked target creation_time = 2026-03-07T01:43:36.211416095Z path = KstatPath { module: "link", instance: 0, name: "tfportrear0_0" } unit = oximeter
502026-03-07T01:44:02.816ZDEBGtfportd (kstat-sampler-worker): inserted empty per-target sample queue id = 1092800514650368985 unit = oximeter
512026-03-07T01:44:02.816ZDEBGtfportd: added new link to kstat sampler link_kind = switch-port-control link_name = tfportrear0_0 unit = metrics-task-tracker
522026-03-07T01:44:02.816ZWARNtfportd (producer-registration-task): failed to register with Nexus, will retry count = 0 delay = 1.518819ms error = "proto error: no records found for Query { name: Name(\\"_nexus._tcp.control-plane.oxide.internal.\\"), query_type: SRV, query_class: IN }" file = /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/8e30605/oximeter/producer/src/lib.rs:371 unit = producer-server
532026-03-07T01:44:02.816ZWARNtfportd: tfportqsfp0_0 has no addrs
542026-03-07T01:44:03.099ZDEBGtfportd (internal-dns-resolver): lookup_socket_v6 srv dns_name = _nexus._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("3b725edd-3111-4340-9cbb-becbd792b090.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("67fe79d4-b7f4-4e94-a920-250e511ba133.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("cfb08bc8-1733-4a6a-b6ce-e1345a48e55e.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("cfb08bc8-1733-4a6a-b6ce-e1345a48e55e.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::c)) }], valid_until: Instant { tv_sec: 1540, tv_nsec: 230071298 } }) unit = producer-server
552026-03-07T01:44:03.099ZDEBGtfportd (producer-registration-task): will register with Nexus at [fd00:1122:3344:101::a]:12221 unit = producer-server
562026-03-07T01:44:03.144ZDEBGtfportd (producer-registration-task): client request body = Some(Body) method = POST unit = producer-server uri = http://[fd00:1122:3344:101::a]:12221/metrics/producers
572026-03-07T01:44:03.677ZINFOtfportd: added new ARP entry entry = 10.151.1.174->96:be:78:d7:ad:d6 interface = tfportqsfp0_0
582026-03-07T01:44:04.142ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
592026-03-07T01:44:04.148ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::b0b9:3eff:fe61:d0d]
602026-03-07T01:44:04.162ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
612026-03-07T01:44:04.162ZDEBGtfportd (kstat-sampler): adding new target unit = oximeter
622026-03-07T01:44:04.163ZDEBGtfportd (kstat-sampler-worker): storing new creation time for tracked target creation_time = 2026-03-07T01:43:36.211277731Z path = KstatPath { module: "link", instance: 0, name: "tfportqsfp0_0" } unit = oximeter
632026-03-07T01:44:04.163ZDEBGtfportd (kstat-sampler-worker): inserted empty per-target sample queue id = 10573277605096125527 unit = oximeter
642026-03-07T01:44:04.163ZDEBGtfportd: added new link to kstat sampler link_kind = switch-port-control link_name = tfportqsfp0_0 unit = metrics-task-tracker
652026-03-07T01:44:05.317ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
662026-03-07T01:44:05.322ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::b0b9:3eff:fe61:d0d]
672026-03-07T01:44:05.325ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
682026-03-07T01:44:06.408ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
692026-03-07T01:44:06.412ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::b0b9:3eff:fe61:d0d]
702026-03-07T01:44:06.415ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
712026-03-07T01:44:07.318ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
722026-03-07T01:44:07.327ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::b0b9:3eff:fe61:d0d]
732026-03-07T01:44:07.330ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
742026-03-07T01:44:08.444ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
752026-03-07T01:44:08.449ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::b0b9:3eff:fe61:d0d]
762026-03-07T01:44:08.451ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
772026-03-07T01:44:09.069ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
782026-03-07T01:44:09.075ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::b0b9:3eff:fe61:d0d]
792026-03-07T01:44:09.079ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
802026-03-07T01:44:09.866ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
812026-03-07T01:44:09.870ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::b0b9:3eff:fe61:d0d]
822026-03-07T01:44:09.873ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
832026-03-07T01:44:10.682ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
842026-03-07T01:44:10.687ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::b0b9:3eff:fe61:d0d]
852026-03-07T01:44:10.691ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
862026-03-07T01:44:11.779ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
872026-03-07T01:44:11.784ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::b0b9:3eff:fe61:d0d]
882026-03-07T01:44:11.787ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
892026-03-07T01:44:12.821ZDEBGtfportd (kstat-sampler-worker): pulled samples from target id = 1092800514650368985 n_samples = 6 unit = oximeter
902026-03-07T01:44:12.888ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
912026-03-07T01:44:12.892ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::b0b9:3eff:fe61:d0d]
922026-03-07T01:44:12.895ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
932026-03-07T01:44:13.543ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
942026-03-07T01:44:13.549ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::b0b9:3eff:fe61:d0d]
952026-03-07T01:44:13.552ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
962026-03-07T01:44:14.166ZDEBGtfportd (kstat-sampler-worker): pulled samples from target id = 10573277605096125527 n_samples = 6 unit = oximeter
972026-03-07T01:44:14.212ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
982026-03-07T01:44:14.217ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::b0b9:3eff:fe61:d0d]
992026-03-07T01:44:14.220ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1002026-03-07T01:44:14.825ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1012026-03-07T01:44:14.830ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::b0b9:3eff:fe61:d0d]
1022026-03-07T01:44:14.832ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1032026-03-07T01:44:15.683ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1042026-03-07T01:44:15.688ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::b0b9:3eff:fe61:d0d]
1052026-03-07T01:44:15.690ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1062026-03-07T01:44:16.433ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1072026-03-07T01:44:16.437ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::b0b9:3eff:fe61:d0d]
1082026-03-07T01:44:16.440ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1092026-03-07T01:44:17.037ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1102026-03-07T01:44:17.043ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::b0b9:3eff:fe61:d0d]
1112026-03-07T01:44:17.045ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1122026-03-07T01:44:18.140ZDEBGtfportd (producer-registration-task): client response result = Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::a]:12221/metrics/producers", source: TimedOut }) unit = producer-server
1132026-03-07T01:44:18.141ZWARNtfportd (producer-registration-task): failed to register with Nexus, will retry count = 1 delay = 15.331965348s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers)" file = /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/8e30605/oximeter/producer/src/lib.rs:371 unit = producer-server
1142026-03-07T01:44:18.434ZDEBGtfportd (internal-dns-resolver): lookup_socket_v6 srv dns_name = _nexus._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("3b725edd-3111-4340-9cbb-becbd792b090.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("67fe79d4-b7f4-4e94-a920-250e511ba133.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("cfb08bc8-1733-4a6a-b6ce-e1345a48e55e.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("cfb08bc8-1733-4a6a-b6ce-e1345a48e55e.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::c)) }], valid_until: Instant { tv_sec: 1555, tv_nsec: 571566098 } }) unit = producer-server
1152026-03-07T01:44:18.434ZDEBGtfportd (producer-registration-task): will register with Nexus at [fd00:1122:3344:101::a]:12221 unit = producer-server
1162026-03-07T01:44:18.466ZDEBGtfportd (producer-registration-task): client request body = Some(Body) method = POST unit = producer-server uri = http://[fd00:1122:3344:101::a]:12221/metrics/producers
1172026-03-07T01:44:18.501ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1182026-03-07T01:44:18.507ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::b0b9:3eff:fe61:d0d]
1192026-03-07T01:44:18.511ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1202026-03-07T01:44:19.039ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1212026-03-07T01:44:19.053ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::b0b9:3eff:fe61:d0d]
1222026-03-07T01:44:19.159ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1232026-03-07T01:44:20.592ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1242026-03-07T01:44:20.597ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::b0b9:3eff:fe61:d0d]
1252026-03-07T01:44:20.600ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1262026-03-07T01:44:21.327ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1272026-03-07T01:44:21.333ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::b0b9:3eff:fe61:d0d]
1282026-03-07T01:44:21.337ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1292026-03-07T01:44:22.497ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1302026-03-07T01:44:22.507ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::b0b9:3eff:fe61:d0d]
1312026-03-07T01:44:22.514ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1322026-03-07T01:44:22.811ZDEBGtfportd (kstat-sampler-worker): pulled samples from target id = 1092800514650368985 n_samples = 6 unit = oximeter
1332026-03-07T01:44:23.281ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1342026-03-07T01:44:23.288ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::b0b9:3eff:fe61:d0d]
1352026-03-07T01:44:23.293ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1362026-03-07T01:44:24.158ZDEBGtfportd (kstat-sampler-worker): pulled samples from target id = 10573277605096125527 n_samples = 6 unit = oximeter
1372026-03-07T01:44:24.715ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1382026-03-07T01:44:24.723ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::b0b9:3eff:fe61:d0d]
1392026-03-07T01:44:24.726ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1402026-03-07T01:44:26.225ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1412026-03-07T01:44:26.230ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::b0b9:3eff:fe61:d0d]
1422026-03-07T01:44:26.233ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1432026-03-07T01:44:26.808ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1442026-03-07T01:44:26.814ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::b0b9:3eff:fe61:d0d]
1452026-03-07T01:44:26.816ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1462026-03-07T01:44:27.959ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1472026-03-07T01:44:27.964ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::b0b9:3eff:fe61:d0d]
1482026-03-07T01:44:27.967ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1492026-03-07T01:44:28.566ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1502026-03-07T01:44:28.573ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::b0b9:3eff:fe61:d0d]
1512026-03-07T01:44:28.577ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1522026-03-07T01:44:29.308ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1532026-03-07T01:44:29.315ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::b0b9:3eff:fe61:d0d]
1542026-03-07T01:44:29.319ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1552026-03-07T01:44:34.515ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1562026-03-07T01:44:34.543ZDEBGtfportd (kstat-sampler-worker): pulled samples from target id = 1092800514650368985 n_samples = 6 unit = oximeter
1572026-03-07T01:44:34.544ZDEBGtfportd (kstat-sampler-worker): pulled samples from target id = 10573277605096125527 n_samples = 6 unit = oximeter
1582026-03-07T01:44:34.550ZDEBGtfportd (producer-registration-task): client response result = Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::a]:12221/metrics/producers", source: TimedOut }) unit = producer-server
1592026-03-07T01:44:34.570ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::b0b9:3eff:fe61:d0d]
1602026-03-07T01:44:34.636ZWARNtfportd (producer-registration-task): failed to register with Nexus, will retry count = 2 delay = 31.830603386s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers)" file = /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/8e30605/oximeter/producer/src/lib.rs:371 unit = producer-server
1612026-03-07T01:44:34.645ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1622026-03-07T01:44:35.378ZDEBGtfportd (internal-dns-resolver): lookup_socket_v6 srv dns_name = _nexus._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("3b725edd-3111-4340-9cbb-becbd792b090.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("67fe79d4-b7f4-4e94-a920-250e511ba133.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("cfb08bc8-1733-4a6a-b6ce-e1345a48e55e.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("cfb08bc8-1733-4a6a-b6ce-e1345a48e55e.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::c)) }], valid_until: Instant { tv_sec: 1572, tv_nsec: 518462742 } }) unit = producer-server
1632026-03-07T01:44:35.579ZDEBGtfportd (producer-registration-task): will register with Nexus at [fd00:1122:3344:101::a]:12221 unit = producer-server
1642026-03-07T01:44:35.655ZDEBGtfportd (producer-registration-task): client request body = Some(Body) method = POST unit = producer-server uri = http://[fd00:1122:3344:101::a]:12221/metrics/producers
1652026-03-07T01:44:35.680ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1662026-03-07T01:44:35.688ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::b0b9:3eff:fe61:d0d]
1672026-03-07T01:44:35.693ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1682026-03-07T01:44:36.655ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1692026-03-07T01:44:36.662ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::b0b9:3eff:fe61:d0d]
1702026-03-07T01:44:36.666ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1712026-03-07T01:44:37.920ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1722026-03-07T01:44:37.925ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::b0b9:3eff:fe61:d0d]
1732026-03-07T01:44:37.930ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1742026-03-07T01:44:38.588ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1752026-03-07T01:44:38.595ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::b0b9:3eff:fe61:d0d]
1762026-03-07T01:44:38.600ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1772026-03-07T01:44:39.778ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1782026-03-07T01:44:39.784ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::b0b9:3eff:fe61:d0d]
1792026-03-07T01:44:39.788ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1802026-03-07T01:44:41.300ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1812026-03-07T01:44:41.307ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::b0b9:3eff:fe61:d0d]
1822026-03-07T01:44:41.311ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1832026-03-07T01:44:42.299ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1842026-03-07T01:44:42.305ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::b0b9:3eff:fe61:d0d]
1852026-03-07T01:44:42.308ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1862026-03-07T01:44:43.547ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1872026-03-07T01:44:43.553ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::b0b9:3eff:fe61:d0d]
1882026-03-07T01:44:43.557ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1892026-03-07T01:44:44.379ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1902026-03-07T01:44:44.384ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::b0b9:3eff:fe61:d0d]
1912026-03-07T01:44:44.386ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1922026-03-07T01:44:44.548ZDEBGtfportd (kstat-sampler-worker): pulled samples from target id = 1092800514650368985 n_samples = 6 unit = oximeter
1932026-03-07T01:44:44.548ZDEBGtfportd (kstat-sampler-worker): pulled samples from target id = 10573277605096125527 n_samples = 6 unit = oximeter
1942026-03-07T01:44:45.345ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1952026-03-07T01:44:45.350ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::b0b9:3eff:fe61:d0d]
1962026-03-07T01:44:45.353ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
1972026-03-07T01:44:45.918ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
1982026-03-07T01:44:45.922ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::b0b9:3eff:fe61:d0d]
1992026-03-07T01:44:45.925ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
2002026-03-07T01:44:47.330ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2012026-03-07T01:44:47.335ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::b0b9:3eff:fe61:d0d]
2022026-03-07T01:44:47.338ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
2032026-03-07T01:44:48.160ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2042026-03-07T01:44:48.165ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::b0b9:3eff:fe61:d0d]
2052026-03-07T01:44:48.167ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
2062026-03-07T01:44:49.251ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2072026-03-07T01:44:49.257ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::b0b9:3eff:fe61:d0d]
2082026-03-07T01:44:49.259ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
2092026-03-07T01:44:50.062ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2102026-03-07T01:44:50.068ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::b0b9:3eff:fe61:d0d]
2112026-03-07T01:44:50.071ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
2122026-03-07T01:44:50.657ZDEBGtfportd (producer-registration-task): client response result = Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::a]:12221/metrics/producers", source: TimedOut }) unit = producer-server
2132026-03-07T01:44:50.660ZWARNtfportd (producer-registration-task): failed to register with Nexus, will retry count = 3 delay = 47.856453816s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers)" file = /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/8e30605/oximeter/producer/src/lib.rs:371 unit = producer-server
2142026-03-07T01:44:51.401ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2152026-03-07T01:44:51.406ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::b0b9:3eff:fe61:d0d]
2162026-03-07T01:44:51.409ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
2172026-03-07T01:44:51.724ZDEBGtfportd (internal-dns-resolver): lookup_socket_v6 srv dns_name = _nexus._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("3b725edd-3111-4340-9cbb-becbd792b090.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("67fe79d4-b7f4-4e94-a920-250e511ba133.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("cfb08bc8-1733-4a6a-b6ce-e1345a48e55e.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("cfb08bc8-1733-4a6a-b6ce-e1345a48e55e.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::c)) }], valid_until: Instant { tv_sec: 1588, tv_nsec: 866879248 } }) unit = producer-server
2182026-03-07T01:44:51.725ZDEBGtfportd (producer-registration-task): will register with Nexus at [fd00:1122:3344:101::a]:12221 unit = producer-server
2192026-03-07T01:44:51.759ZDEBGtfportd (producer-registration-task): client request body = Some(Body) method = POST unit = producer-server uri = http://[fd00:1122:3344:101::a]:12221/metrics/producers
2202026-03-07T01:44:52.810ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2212026-03-07T01:44:52.816ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::b0b9:3eff:fe61:d0d]
2222026-03-07T01:44:52.819ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
2232026-03-07T01:44:53.528ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2242026-03-07T01:44:53.534ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::b0b9:3eff:fe61:d0d]
2252026-03-07T01:44:53.537ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
2262026-03-07T01:44:54.192ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2272026-03-07T01:44:54.198ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::b0b9:3eff:fe61:d0d]
2282026-03-07T01:44:54.201ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
2292026-03-07T01:44:54.549ZDEBGtfportd (kstat-sampler-worker): pulled samples from target id = 1092800514650368985 n_samples = 6 unit = oximeter
2302026-03-07T01:44:54.549ZDEBGtfportd (kstat-sampler-worker): pulled samples from target id = 10573277605096125527 n_samples = 6 unit = oximeter
2312026-03-07T01:44:55.372ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2322026-03-07T01:44:55.378ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::b0b9:3eff:fe61:d0d]
2332026-03-07T01:44:55.380ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
2342026-03-07T01:44:56.135ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2352026-03-07T01:44:56.140ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::b0b9:3eff:fe61:d0d]
2362026-03-07T01:44:56.143ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
2372026-03-07T01:44:57.168ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2382026-03-07T01:44:57.172ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::b0b9:3eff:fe61:d0d]
2392026-03-07T01:44:57.175ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
2402026-03-07T01:44:58.457ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2412026-03-07T01:44:58.462ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::b0b9:3eff:fe61:d0d]
2422026-03-07T01:44:58.465ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
2432026-03-07T01:44:59.364ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2442026-03-07T01:44:59.370ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::b0b9:3eff:fe61:d0d]
2452026-03-07T01:44:59.373ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]
2462026-03-07T01:45:00.499ZDEBGtfportd: found simports [ "tfportrear0_0", "tfportqsfp0_0", ]
2472026-03-07T01:45:00.505ZDEBGtfportd: sync tfportrear0_0 addrs [fe80::b0b9:3eff:fe61:d0d]
2482026-03-07T01:45:00.508ZDEBGtfportd: sync tfportqsfp0_0 addrs [10.151.1.174]