Max level shown:
1 [ Aug 31 16:14:07 Disabled. ]
2 [ Aug 31 16:14:07 Rereading configuration. ]
3 [ Aug 31 16:14:08 Enabled. ]
4 [ Aug 31 16:14:08 Rereading configuration. ]
5 [ Aug 31 16:14:08 Executing start method ("ctrun -l child -o noorphan,regent /opt/oxide/mg-ddm/pkg/ddm_method_script.sh &"). ]
6 [ Aug 31 16:14:08 Method "start" exited with status 0. ]
72026-08-31T16:14:08.411ZINFOslog-rs: signal handler waiting for context
82026-08-31T16:14:08.439ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
92026-08-31T16:14:08.439ZERROslog-rs: failed to start stats server: underlay address not found
102026-08-31T16:14:08.439ZINFOslog-rs: signal handler waiting got context
112026-08-31T16:14:08.441ZINFOslog-rs: admin: listening on [::]:8000
122026-08-31T16:14:09.440ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
132026-08-31T16:14:10.440ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
142026-08-31T16:14:11.440ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
152026-08-31T16:14:12.440ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
162026-08-31T16:14:13.444ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
172026-08-31T16:14:14.444ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
182026-08-31T16:14:15.444ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
192026-08-31T16:14:16.444ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
202026-08-31T16:14:17.444ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
212026-08-31T16:14:18.445ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
222026-08-31T16:14:19.445ZINFOslog-rs: [tfportrear0_0] sm initialized with addr fe80::e00f:9bff:fe32:f9a7 on if tfportrear0_0 index 3
232026-08-31T16:14:19.445ZDEBGslog-rs: [tfportrear0_0] starting discovery handler
242026-08-31T16:14:19.445ZINFOslog-rs: [tfportrear0_0] nbr is fe80::3846:3ff:feb6:19a3@buskin server
252026-08-31T16:14:19.445ZDEBGslog-rs: [tfportrear0_0] transition solicit -> exchange
262026-08-31T16:14:19.445ZINFOslog-rs: [tfportrear0_0] exchange: listening on [fe80::e00f:9bff:fe32:f9a7]:56797
272026-08-31T16:14:19.446ZINFOslog-rs: [tfportrear0_0] waiting for exchange server to start
282026-08-31T16:14:19.448ZWARNslog-rs: [tfportrear0_0] exchange pull: hyper client error: client error (Connect)
29 [ Aug 31 16:14:19 Rereading configuration. ]
30 [ Aug 31 16:14:19 Executing refresh method ("/opt/oxide/mg-ddm/pkg/refresh_ctrun_child.sh -c 214"). ]
31 + CONTRACT=''
32 + getopts c: o
33 + CONTRACT=214
34 + getopts c: o
35 + [[ -n 214 ]]
36 + pgrep -c 214
37 + PARENT_PID=2298
38 + [[ -n 2298 ]]
39 + pgrep -P 2298
40 + CHILD_PID=2299
41 + [[ -n 2299 ]]
42 + kill -USR1 2299
432026-08-31T16:14:19.920ZINFOslog-rs: handling smf refresh
44 [ Aug 31 16:14:19 Method "refresh" exited with status 0. ]
452026-08-31T16:14:19.922ZINFOslog-rs: starting stats server on smf refresh
46 Aug 31 16:14:19.923 DEBG registered DTrace probes
47 Aug 31 16:14:19.923 DEBG registered endpoint, versions: all, path: /{producer_id}, method: GET, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot
48 Aug 31 16:14:19.924 INFO listening, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:335
49 Aug 31 16:14:19.924 DEBG successfully registered DTrace USDT probes, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot
50 Aug 31 16:14:19.924 DEBG Nexus IP not provided, will use DNS to resolve it
51 Aug 31 16:14:19.924 INFO new DNS resolver, addresses: [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53], component: internal-dns-resolver, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/internal-dns/resolver/src/resolver.rs:144
52 Aug 31 16:14:19.924 DEBG starting producer registration task
53 Aug 31 16:14:19.924 INFO starting oximeter metric producer server, interval: 1s, address: [fd00:1122:3344:101::2]:8001, producer_id: e8b871cb-2bfa-4901-b0a3-2bf186b9a1e0, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/oximeter/producer/src/lib.rs:280
54 Aug 31 16:14:19.924 DEBG registering as oximeter producer with Nexus, registration_count: 0, component: producer-registration-task
55 Aug 31 16:14:19.926 WARN failed to register with Nexus, will retry, error: proto error: no records found for Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, delay: 1.679529ms, count: 0, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/oximeter/producer/src/lib.rs:376
56 Aug 31 16:14:20.184 WARN failed to register with Nexus, will retry, error: proto error: no records found for Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, delay: 259.415097ms, count: 1, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/oximeter/producer/src/lib.rs:376
57 Aug 31 16:14:20.926 WARN failed to register with Nexus, will retry, error: proto error: no records found for Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, delay: 1.001383654s, count: 2, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/oximeter/producer/src/lib.rs:376
582026-08-31T16:14:21.449ZINFOslog-rs: [tfportrear0_0] sending 5 routes to dendrite
592026-08-31T16:14:21.449ZDEBGslog-rs: [tfportrear0_0] sending to dpd host=localhost port=12224
602026-08-31T16:14:21.450ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:3:: -> fe80::3846:3ff:feb6:19a3 on port Rear(Rear("rear0"))/0
612026-08-31T16:14:21.452ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:2:: -> fe80::3846:3ff:feb6:19a3 on port Rear(Rear("rear0"))/0
622026-08-31T16:14:21.452ZINFOslog-rs: [tfportrear0_0] adding route fdb0:18c0:4d0d:9fb2:: -> fe80::3846:3ff:feb6:19a3 on port Rear(Rear("rear0"))/0
632026-08-31T16:14:21.453ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:1:: -> fe80::3846:3ff:feb6:19a3 on port Rear(Rear("rear0"))/0
642026-08-31T16:14:21.453ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:101:: -> fe80::3846:3ff:feb6:19a3 on port Rear(Rear("rear0"))/0
652026-08-31T16:14:21.453ZINFOslog-rs: [tfportrear0_0] removing routes 0 from dendrite
662026-08-31T16:14:21.454ZDEBGslog-rs: [tfportrear0_0] redistributing update to 0 peers
67 Aug 31 16:14:21.926 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("178c9f96-fe8f-4eda-925a-75c06c3797db.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("552dc43f-49ec-4676-adb1-9effed3aba6f.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("806197a2-8710-41a7-a117-e8820a80acd3.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("806197a2-8710-41a7-a117-e8820a80acd3.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::c)) }], valid_until: Instant { tv_sec: 740, tv_nsec: 422534677 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
68 Aug 31 16:14:21.927 DEBG will register with Nexus at [fd00:1122:3344:101::b]:12221, component: producer-registration-task
69 Aug 31 16:14:21.928 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::b]:12221/metrics/producers, method: POST, component: producer-registration-task
702026-08-31T16:14:22.083ZINFOslog-rs: advertise tunnel: { TunnelOrigin { overlay_prefix: V4( Ipv4Net { addr: 0.0.0.0, width: 0, }, ), boundary_addr: fd8f:e4d8:5972:c310::1, vni: 99, metric: 18446744073709551615, }, }
712026-08-31T16:14:29.449ZINFOslog-rs: [tfportrear0_0] sending 5 routes to dendrite
722026-08-31T16:14:29.449ZDEBGslog-rs: [tfportrear0_0] sending to dpd host=localhost port=12224
732026-08-31T16:14:29.450ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:1:: -> fe80::3846:3ff:feb6:19a3 on port Rear(Rear("rear0"))/0
742026-08-31T16:14:29.450ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:3:: -> fe80::3846:3ff:feb6:19a3 on port Rear(Rear("rear0"))/0
752026-08-31T16:14:29.450ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:2:: -> fe80::3846:3ff:feb6:19a3 on port Rear(Rear("rear0"))/0
762026-08-31T16:14:29.451ZINFOslog-rs: [tfportrear0_0] adding route fdb0:18c0:4d0d:9fb2:: -> fe80::3846:3ff:feb6:19a3 on port Rear(Rear("rear0"))/0
772026-08-31T16:14:29.451ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:101:: -> fe80::3846:3ff:feb6:19a3 on port Rear(Rear("rear0"))/0
782026-08-31T16:14:29.451ZINFOslog-rs: [tfportrear0_0] removing routes 0 from dendrite
79 Aug 31 16:14:36.929 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::b]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
80 Aug 31 16:14:36.929 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): operation timed out, delay: 17.004444122s, count: 3, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/oximeter/producer/src/lib.rs:376
812026-08-31T16:14:39.413ZINFOslog-rs: [tfportrear0_0] sending 5 routes to dendrite
822026-08-31T16:14:39.413ZDEBGslog-rs: [tfportrear0_0] sending to dpd host=localhost port=12224
832026-08-31T16:14:39.414ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:2:: -> fe80::3846:3ff:feb6:19a3 on port Rear(Rear("rear0"))/0
842026-08-31T16:14:39.415ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:101:: -> fe80::3846:3ff:feb6:19a3 on port Rear(Rear("rear0"))/0
852026-08-31T16:14:39.415ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:1:: -> fe80::3846:3ff:feb6:19a3 on port Rear(Rear("rear0"))/0
862026-08-31T16:14:39.415ZINFOslog-rs: [tfportrear0_0] adding route fdb0:18c0:4d0d:9fb2:: -> fe80::3846:3ff:feb6:19a3 on port Rear(Rear("rear0"))/0
872026-08-31T16:14:39.415ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:3:: -> fe80::3846:3ff:feb6:19a3 on port Rear(Rear("rear0"))/0
882026-08-31T16:14:39.415ZINFOslog-rs: [tfportrear0_0] removing routes 0 from dendrite
89 Aug 31 16:14:39.510 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("178c9f96-fe8f-4eda-925a-75c06c3797db.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("552dc43f-49ec-4676-adb1-9effed3aba6f.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("806197a2-8710-41a7-a117-e8820a80acd3.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("806197a2-8710-41a7-a117-e8820a80acd3.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::c)) }], valid_until: Instant { tv_sec: 758, tv_nsec: 41447357 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
90 Aug 31 16:14:39.511 DEBG will register with Nexus at [fd00:1122:3344:101::b]:12221, component: producer-registration-task
91 Aug 31 16:14:39.512 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::b]:12221/metrics/producers, method: POST, component: producer-registration-task
922026-08-31T16:14:43.827ZWARNslog-rs: [tfportrear0_0] neighbor buskin@fe80::3846:3ff:feb6:19a3 missed solicit interval
932026-08-31T16:14:45.827ZWARNslog-rs: [tfportrear0_0] neighbor buskin@fe80::3846:3ff:feb6:19a3 missed solicit interval
942026-08-31T16:14:47.828ZWARNslog-rs: [tfportrear0_0] neighbor buskin@fe80::3846:3ff:feb6:19a3 missed solicit interval
952026-08-31T16:14:49.809ZINFOslog-rs: [tfportrear0_0] sending 5 routes to dendrite
962026-08-31T16:14:49.809ZDEBGslog-rs: [tfportrear0_0] sending to dpd host=localhost port=12224
972026-08-31T16:14:49.810ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:101:: -> fe80::3846:3ff:feb6:19a3 on port Rear(Rear("rear0"))/0
982026-08-31T16:14:49.810ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:1:: -> fe80::3846:3ff:feb6:19a3 on port Rear(Rear("rear0"))/0
992026-08-31T16:14:49.811ZINFOslog-rs: [tfportrear0_0] adding route fdb0:18c0:4d0d:9fb2:: -> fe80::3846:3ff:feb6:19a3 on port Rear(Rear("rear0"))/0
1002026-08-31T16:14:49.811ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:3:: -> fe80::3846:3ff:feb6:19a3 on port Rear(Rear("rear0"))/0
1012026-08-31T16:14:49.811ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:2:: -> fe80::3846:3ff:feb6:19a3 on port Rear(Rear("rear0"))/0
1022026-08-31T16:14:49.811ZINFOslog-rs: [tfportrear0_0] removing routes 0 from dendrite
1032026-08-31T16:14:49.829ZWARNslog-rs: [tfportrear0_0] neighbor buskin@fe80::3846:3ff:feb6:19a3 missed solicit interval
1042026-08-31T16:14:51.829ZWARNslog-rs: [tfportrear0_0] neighbor buskin@fe80::3846:3ff:feb6:19a3 missed solicit interval
1052026-08-31T16:14:53.830ZWARNslog-rs: [tfportrear0_0] neighbor buskin@fe80::3846:3ff:feb6:19a3 missed solicit interval
106 Aug 31 16:14:54.894 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::b]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
107 Aug 31 16:14:54.894 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): operation timed out, delay: 34.623382833s, count: 4, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/oximeter/producer/src/lib.rs:376
1082026-08-31T16:14:55.831ZWARNslog-rs: [tfportrear0_0] neighbor buskin@fe80::3846:3ff:feb6:19a3 missed solicit interval
1092026-08-31T16:14:57.832ZWARNslog-rs: [tfportrear0_0] neighbor buskin@fe80::3846:3ff:feb6:19a3 missed solicit interval
110 Aug 31 16:14:58.825 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("178c9f96-fe8f-4eda-925a-75c06c3797db.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("552dc43f-49ec-4676-adb1-9effed3aba6f.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("806197a2-8710-41a7-a117-e8820a80acd3.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("806197a2-8710-41a7-a117-e8820a80acd3.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::c)) }], valid_until: Instant { tv_sec: 776, tv_nsec: 985675693 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
111 Aug 31 16:14:58.826 DEBG will register with Nexus at [fd00:1122:3344:101::b]:12221, component: producer-registration-task
112 Aug 31 16:14:58.827 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::b]:12221/metrics/producers, method: POST, component: producer-registration-task
1132026-08-31T16:14:59.810ZINFOslog-rs: [tfportrear0_0] sending 5 routes to dendrite
1142026-08-31T16:14:59.810ZDEBGslog-rs: [tfportrear0_0] sending to dpd host=localhost port=12224
1152026-08-31T16:14:59.811ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:2:: -> fe80::3846:3ff:feb6:19a3 on port Rear(Rear("rear0"))/0
1162026-08-31T16:14:59.812ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:101:: -> fe80::3846:3ff:feb6:19a3 on port Rear(Rear("rear0"))/0
1172026-08-31T16:14:59.812ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:3:: -> fe80::3846:3ff:feb6:19a3 on port Rear(Rear("rear0"))/0
1182026-08-31T16:14:59.812ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:1:: -> fe80::3846:3ff:feb6:19a3 on port Rear(Rear("rear0"))/0
1192026-08-31T16:14:59.812ZINFOslog-rs: [tfportrear0_0] adding route fdb0:18c0:4d0d:9fb2:: -> fe80::3846:3ff:feb6:19a3 on port Rear(Rear("rear0"))/0
1202026-08-31T16:14:59.812ZINFOslog-rs: [tfportrear0_0] removing routes 0 from dendrite
1212026-08-31T16:14:59.832ZWARNslog-rs: [tfportrear0_0] neighbor buskin@fe80::3846:3ff:feb6:19a3 missed solicit interval