Max level shown:
1 [ Jun 3 22:43:34 Disabled. ]
2 [ Jun 3 22:43:34 Rereading configuration. ]
3 [ Jun 3 22:43:35 Enabled. ]
4 [ Jun 3 22:43:35 Rereading configuration. ]
5 [ Jun 3 22:43:35 Executing start method ("ctrun -l child -o noorphan,regent /opt/oxide/mg-ddm/pkg/ddm_method_script.sh &"). ]
6 [ Jun 3 22:43:35 Method "start" exited with status 0. ]
72026-06-03T22:43:35.522ZINFOslog-rs: signal handler waiting for context
82026-06-03T22:43:35.544ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
92026-06-03T22:43:35.544ZERROslog-rs: failed to start stats server: underlay address not found
102026-06-03T22:43:35.544ZINFOslog-rs: signal handler waiting got context
112026-06-03T22:43:35.545ZINFOslog-rs: admin: listening on [::1]:8000
122026-06-03T22:43:35.546ZINFOslog-rs: admin: server started
132026-06-03T22:43:36.545ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
142026-06-03T22:43:37.545ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
152026-06-03T22:43:38.545ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
162026-06-03T22:43:39.545ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
172026-06-03T22:43:40.545ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
182026-06-03T22:43:41.572ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
192026-06-03T22:43:42.545ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
202026-06-03T22:43:43.545ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
212026-06-03T22:43:44.545ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
222026-06-03T22:43:45.545ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
232026-06-03T22:43:46.545ZINFOslog-rs: [tfportrear0_0] sm initialized with addr fe80::38d8:19ff:febb:30 on if tfportrear0_0 index 3
242026-06-03T22:43:46.545ZDEBGslog-rs: [tfportrear0_0] starting discovery handler
252026-06-03T22:43:46.546ZINFOslog-rs: [tfportrear0_0] nbr is fe80::647e:50ff:fe5a:943b@sock server
262026-06-03T22:43:46.546ZDEBGslog-rs: [tfportrear0_0] transition solicit -> exchange
272026-06-03T22:43:46.546ZINFOslog-rs: [tfportrear0_0] exchange: listening on [fe80::38d8:19ff:febb:30]:56797
282026-06-03T22:43:46.546ZINFOslog-rs: [tfportrear0_0] waiting for exchange server to start
29 [ Jun 3 22:43:47 Rereading configuration. ]
30 [ Jun 3 22:43:47 Executing refresh method ("/opt/oxide/mg-ddm/pkg/refresh_ctrun_child.sh -c 214"). ]
31 + CONTRACT=''
32 + getopts c: o
33 + CONTRACT=214
34 + getopts c: o
35 + [[ -n 214 ]]
36 + pgrep -c 214
37 + PARENT_PID=2290
38 + [[ -n 2290 ]]
39 + pgrep -P 2290
40 + CHILD_PID=2294
41 + [[ -n 2294 ]]
42 + kill -USR1 2294
432026-06-03T22:43:47.070ZINFOslog-rs: handling smf refresh
44 [ Jun 3 22:43:47 Method "refresh" exited with status 0. ]
452026-06-03T22:43:47.074ZINFOslog-rs: starting stats server on smf refresh
46 Jun 03 22:43:47.074 DEBG registered DTrace probes
47 Jun 03 22:43:47.075 DEBG registered endpoint, versions: all, path: /{producer_id}, method: GET, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot
48 Jun 03 22:43:47.075 INFO listening, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.7/src/server.rs:328
49 Jun 03 22:43:47.075 DEBG successfully registered DTrace USDT probes, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot
50 Jun 03 22:43:47.075 DEBG Nexus IP not provided, will use DNS to resolve it
51 Jun 03 22:43:47.075 INFO new DNS resolver, addresses: [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53], component: internal-dns-resolver, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/dfcd9ac/internal-dns/resolver/src/resolver.rs:144
52 Jun 03 22:43:47.075 DEBG starting producer registration task
53 Jun 03 22:43:47.076 INFO starting oximeter metric producer server, interval: 1s, address: [fd00:1122:3344:101::2]:8001, producer_id: 42cc7827-2424-4a23-8e8f-fdee14365bca, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/dfcd9ac/oximeter/producer/src/lib.rs:283
54 Jun 03 22:43:47.076 DEBG registering as oximeter producer with Nexus, registration_count: 0, component: producer-registration-task
552026-06-03T22:43:48.279ZINFOslog-rs: advertise tunnel: { TunnelOrigin { overlay_prefix: V4( Ipv4Net { addr: 0.0.0.0, width: 0, }, ), boundary_addr: fdae:6da4:f17f:3a03::1, vni: 99, metric: 18446744073709551615, }, }
562026-06-03T22:43:48.549ZWARNslog-rs: [tfportrear0_0] exchange pull: hyper client error: client error (Connect)
572026-06-03T22:43:48.549ZINFOslog-rs: [tfportrear0_0] sending 5 routes to dendrite
582026-06-03T22:43:48.550ZDEBGslog-rs: [tfportrear0_0] sending to dpd host=localhost port=12224
592026-06-03T22:43:48.550ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:101:: -> fe80::647e:50ff:fe5a:943b on port Rear(Rear("rear0"))/0
602026-06-03T22:43:48.551ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:1:: -> fe80::647e:50ff:fe5a:943b on port Rear(Rear("rear0"))/0
612026-06-03T22:43:48.551ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:2:: -> fe80::647e:50ff:fe5a:943b on port Rear(Rear("rear0"))/0
622026-06-03T22:43:48.552ZINFOslog-rs: [tfportrear0_0] adding route fdb0:18c0:4d0d:a02a:: -> fe80::647e:50ff:fe5a:943b on port Rear(Rear("rear0"))/0
632026-06-03T22:43:48.552ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:3:: -> fe80::647e:50ff:fe5a:943b on port Rear(Rear("rear0"))/0
642026-06-03T22:43:48.552ZINFOslog-rs: [tfportrear0_0] removing routes 0 from dendrite
652026-06-03T22:43:48.552ZDEBGslog-rs: [tfportrear0_0] redistributing update to 0 peers
66 Jun 03 22:43:52.079 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("561849dd-51cf-4a51-aa3d-460e709b1bc9.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("a50458cd-7bc4-4c94-8695-dfe49e9c4823.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("ec4d8e48-9030-409f-90f4-277006ddbf8e.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("ec4d8e48-9030-409f-90f4-277006ddbf8e.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::c)) }], valid_until: Instant { tv_sec: 712, tv_nsec: 999071543 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
67 Jun 03 22:43:52.080 DEBG will register with Nexus at [fd00:1122:3344:101::b]:12221, component: producer-registration-task
68 Jun 03 22:43:52.081 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::b]:12221/metrics/producers, method: POST, component: producer-registration-task
69 Jun 03 22:44:07.654 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::b]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
70 Jun 03 22:44:07.654 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): operation timed out, delay: 20.00729989s, count: 0, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/dfcd9ac/oximeter/producer/src/lib.rs:379
71 Jun 03 22:44:07.916 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("561849dd-51cf-4a51-aa3d-460e709b1bc9.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("a50458cd-7bc4-4c94-8695-dfe49e9c4823.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("ec4d8e48-9030-409f-90f4-277006ddbf8e.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("ec4d8e48-9030-409f-90f4-277006ddbf8e.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::c)) }], valid_until: Instant { tv_sec: 728, tv_nsec: 264559454 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
72 Jun 03 22:44:07.917 DEBG will register with Nexus at [fd00:1122:3344:101::b]:12221, component: producer-registration-task
73 Jun 03 22:44:07.918 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::b]:12221/metrics/producers, method: POST, component: producer-registration-task
74 Jun 03 22:44:22.961 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::b]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
75 Jun 03 22:44:22.961 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): operation timed out, delay: 35.27173504s, count: 1, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/dfcd9ac/oximeter/producer/src/lib.rs:379
76 Jun 03 22:44:23.595 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("561849dd-51cf-4a51-aa3d-460e709b1bc9.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("a50458cd-7bc4-4c94-8695-dfe49e9c4823.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("ec4d8e48-9030-409f-90f4-277006ddbf8e.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("ec4d8e48-9030-409f-90f4-277006ddbf8e.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::c)) }], valid_until: Instant { tv_sec: 743, tv_nsec: 900293627 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
77 Jun 03 22:44:23.596 DEBG will register with Nexus at [fd00:1122:3344:101::b]:12221, component: producer-registration-task
78 Jun 03 22:44:23.597 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::b]:12221/metrics/producers, method: POST, component: producer-registration-task
79 Jun 03 22:44:38.614 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::b]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
80 Jun 03 22:44:38.615 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): operation timed out, delay: 50.907299659s, count: 2, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/dfcd9ac/oximeter/producer/src/lib.rs:379
81 Jun 03 22:44:39.888 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("561849dd-51cf-4a51-aa3d-460e709b1bc9.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("a50458cd-7bc4-4c94-8695-dfe49e9c4823.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("ec4d8e48-9030-409f-90f4-277006ddbf8e.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("ec4d8e48-9030-409f-90f4-277006ddbf8e.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::c)) }], valid_until: Instant { tv_sec: 760, tv_nsec: 177013046 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
82 Jun 03 22:44:39.889 DEBG will register with Nexus at [fd00:1122:3344:101::b]:12221, component: producer-registration-task
83 Jun 03 22:44:39.891 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::b]:12221/metrics/producers, method: POST, component: producer-registration-task
842026-06-03T22:44:54.389ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::647e:50ff:fe5a:943b expire
852026-06-03T22:44:54.396ZWARNslog-rs: [tfportrear0_0] expiring peer fe80::647e:50ff:fe5a:943b due to discovery event
862026-06-03T22:44:54.396ZINFOslog-rs: [tfportrear0_0] nbr is fe80::647e:50ff:fe5a:943b@sock server
872026-06-03T22:44:54.396ZINFOslog-rs: [tfportrear0_0] removing routes 5 from dendrite
882026-06-03T22:44:54.812ZERROslog-rs: [tfportrear0_0] failed to remove tunnel routes: {} OPTE driver is not attached
892026-06-03T22:44:54.812ZDEBGslog-rs: [tfportrear0_0] redistributing expire to 0 peers
902026-06-03T22:44:54.813ZDEBGslog-rs: [tfportrear0_0] transition solicit -> exchange
912026-06-03T22:44:54.817ZINFOslog-rs: [tfportrear0_0] exchange: listening on [fe80::38d8:19ff:febb:30]:56797
922026-06-03T22:44:54.834ZINFOslog-rs: [tfportrear0_0] waiting for exchange server to start
932026-06-03T22:44:54.866ZINFOslog-rs: [tfportrear0_0] sending 5 routes to dendrite
942026-06-03T22:44:54.866ZDEBGslog-rs: [tfportrear0_0] sending to dpd host=localhost port=12224
952026-06-03T22:44:54.868ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:2:: -> fe80::647e:50ff:fe5a:943b on port Rear(Rear("rear0"))/0
962026-06-03T22:44:54.879ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:101:: -> fe80::647e:50ff:fe5a:943b on port Rear(Rear("rear0"))/0
972026-06-03T22:44:54.879ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:3:: -> fe80::647e:50ff:fe5a:943b on port Rear(Rear("rear0"))/0
982026-06-03T22:44:54.881ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:1:: -> fe80::647e:50ff:fe5a:943b on port Rear(Rear("rear0"))/0
992026-06-03T22:44:54.881ZINFOslog-rs: [tfportrear0_0] adding route fdb0:18c0:4d0d:a02a:: -> fe80::647e:50ff:fe5a:943b on port Rear(Rear("rear0"))/0
1002026-06-03T22:44:54.883ZINFOslog-rs: [tfportrear0_0] removing routes 0 from dendrite
1012026-06-03T22:44:54.883ZDEBGslog-rs: [tfportrear0_0] redistributing update to 0 peers
102 Jun 03 22:44:54.903 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::b]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
103 Jun 03 22:44:54.912 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): operation timed out, delay: 67.200229261s, count: 3, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/dfcd9ac/oximeter/producer/src/lib.rs:379
104 Jun 03 22:44:57.181 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("561849dd-51cf-4a51-aa3d-460e709b1bc9.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("a50458cd-7bc4-4c94-8695-dfe49e9c4823.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("ec4d8e48-9030-409f-90f4-277006ddbf8e.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("ec4d8e48-9030-409f-90f4-277006ddbf8e.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::c)) }], valid_until: Instant { tv_sec: 777, tv_nsec: 469302402 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
105 Jun 03 22:44:57.188 DEBG will register with Nexus at [fd00:1122:3344:101::b]:12221, component: producer-registration-task
106 Jun 03 22:44:57.200 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::b]:12221/metrics/producers, method: POST, component: producer-registration-task