Max level shown:
1 [ Aug 25 23:58:29 Disabled. ]
2 [ Aug 25 23:58:29 Rereading configuration. ]
3 [ Aug 25 23:58:30 Enabled. ]
4 [ Aug 25 23:58:30 Rereading configuration. ]
5 [ Aug 25 23:58:30 Executing start method ("ctrun -l child -o noorphan,regent /opt/oxide/mg-ddm/pkg/ddm_method_script.sh &"). ]
6 [ Aug 25 23:58:30 Method "start" exited with status 0. ]
72026-08-25T23:58:30.717ZINFOslog-rs: signal handler waiting for context
82026-08-25T23:58:30.740ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
92026-08-25T23:58:30.740ZERROslog-rs: failed to start stats server: underlay address not found
102026-08-25T23:58:30.740ZINFOslog-rs: signal handler waiting got context
112026-08-25T23:58:30.741ZINFOslog-rs: admin: listening on [::1]:8000
122026-08-25T23:58:31.740ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
132026-08-25T23:58:32.740ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
142026-08-25T23:58:33.740ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
152026-08-25T23:58:34.740ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
162026-08-25T23:58:35.743ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
172026-08-25T23:58:36.742ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
182026-08-25T23:58:37.742ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
192026-08-25T23:58:38.742ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
202026-08-25T23:58:39.742ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
212026-08-25T23:58:40.742ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
222026-08-25T23:58:41.742ZINFOslog-rs: [tfportrear0_0] sm initialized with addr fe80::5ce2:21ff:fe38:a06c on if tfportrear0_0 index 3
232026-08-25T23:58:41.742ZDEBGslog-rs: [tfportrear0_0] starting discovery handler
242026-08-25T23:58:41.743ZINFOslog-rs: [tfportrear0_0] nbr is fe80::a455:ddff:fe59:ea31@sock server
252026-08-25T23:58:41.743ZDEBGslog-rs: [tfportrear0_0] transition solicit -> exchange
262026-08-25T23:58:41.743ZINFOslog-rs: [tfportrear0_0] exchange: listening on [fe80::5ce2:21ff:fe38:a06c]:56797
272026-08-25T23:58:41.743ZINFOslog-rs: [tfportrear0_0] waiting for exchange server to start
28 [ Aug 25 23:58:42 Rereading configuration. ]
29 [ Aug 25 23:58:42 Executing refresh method ("/opt/oxide/mg-ddm/pkg/refresh_ctrun_child.sh -c 214"). ]
30 + CONTRACT=''
31 + getopts c: o
32 + CONTRACT=214
33 + getopts c: o
34 + [[ -n 214 ]]
35 + pgrep -c 214
36 + PARENT_PID=2293
37 + [[ -n 2293 ]]
38 + pgrep -P 2293
39 + CHILD_PID=2297
40 + [[ -n 2297 ]]
41 + kill -USR1 2297
422026-08-25T23:58:42.220ZINFOslog-rs: handling smf refresh
43 [ Aug 25 23:58:42 Method "refresh" exited with status 0. ]
442026-08-25T23:58:42.223ZINFOslog-rs: starting stats server on smf refresh
45 Aug 25 23:58:42.223 DEBG registered DTrace probes
46 Aug 25 23:58:42.224 DEBG registered endpoint, versions: all, path: /{producer_id}, method: GET, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot
47 Aug 25 23:58:42.224 INFO listening, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:335
48 Aug 25 23:58:42.224 DEBG successfully registered DTrace USDT probes, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot
49 Aug 25 23:58:42.224 DEBG Nexus IP not provided, will use DNS to resolve it
50 Aug 25 23:58:42.224 INFO new DNS resolver, addresses: [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53], component: internal-dns-resolver, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/internal-dns/resolver/src/resolver.rs:144
51 Aug 25 23:58:42.224 DEBG starting producer registration task
52 Aug 25 23:58:42.224 INFO starting oximeter metric producer server, interval: 1s, address: [fd00:1122:3344:101::2]:8001, producer_id: 681b16dc-7e23-4774-bf0f-5af36f151b38, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/oximeter/producer/src/lib.rs:280
53 Aug 25 23:58:42.225 DEBG registering as oximeter producer with Nexus, registration_count: 0, component: producer-registration-task
54 Aug 25 23:58:42.226 WARN failed to register with Nexus, will retry, error: proto error: no records found for Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, delay: 1.806075ms, count: 0, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/oximeter/producer/src/lib.rs:376
55 Aug 25 23:58:42.423 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("5cf4f225-06a4-46dd-8bde-e36aae204692.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("6d80c45d-0f75-4276-a842-872bd53173ae.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("b4d4e362-edf6-4910-b829-5347fa537781.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("b4d4e362-edf6-4910-b829-5347fa537781.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::c)) }], valid_until: Instant { tv_sec: 717, tv_nsec: 895344031 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
56 Aug 25 23:58:42.424 DEBG will register with Nexus at [fd00:1122:3344:101::a]:12221, component: producer-registration-task
57 Aug 25 23:58:42.426 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::a]:12221/metrics/producers, method: POST, component: producer-registration-task
582026-08-25T23:58:43.746ZWARNslog-rs: [tfportrear0_0] exchange pull: hyper client error: client error (Connect)
592026-08-25T23:58:43.747ZINFOslog-rs: [tfportrear0_0] sending 5 routes to dendrite
602026-08-25T23:58:43.747ZDEBGslog-rs: [tfportrear0_0] sending to dpd host=localhost port=12224
612026-08-25T23:58:43.749ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:2:: -> fe80::a455:ddff:fe59:ea31 on port Rear(Rear("rear0"))/0
622026-08-25T23:58:43.750ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:3:: -> fe80::a455:ddff:fe59:ea31 on port Rear(Rear("rear0"))/0
632026-08-25T23:58:43.750ZINFOslog-rs: [tfportrear0_0] adding route fdb0:18c0:4d0d:a02a:: -> fe80::a455:ddff:fe59:ea31 on port Rear(Rear("rear0"))/0
642026-08-25T23:58:43.751ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:1:: -> fe80::a455:ddff:fe59:ea31 on port Rear(Rear("rear0"))/0
652026-08-25T23:58:43.751ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:101:: -> fe80::a455:ddff:fe59:ea31 on port Rear(Rear("rear0"))/0
662026-08-25T23:58:43.751ZINFOslog-rs: [tfportrear0_0] removing routes 0 from dendrite
672026-08-25T23:58:43.752ZDEBGslog-rs: [tfportrear0_0] redistributing update to 0 peers
682026-08-25T23:58:44.605ZINFOslog-rs: advertise tunnel: { TunnelOrigin { overlay_prefix: V4( Ipv4Net { addr: 0.0.0.0, width: 0, }, ), boundary_addr: fd19:f7e0:1a67:879d::1, vni: 99, metric: 18446744073709551615, }, }
69 Aug 25 23:58:57.427 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::a]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
70 Aug 25 23:58:57.427 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers): operation timed out, delay: 15.20217624s, count: 1, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/oximeter/producer/src/lib.rs:376
71 Aug 25 23:58:57.844 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("5cf4f225-06a4-46dd-8bde-e36aae204692.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("6d80c45d-0f75-4276-a842-872bd53173ae.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("b4d4e362-edf6-4910-b829-5347fa537781.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("b4d4e362-edf6-4910-b829-5347fa537781.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::c)) }], valid_until: Instant { tv_sec: 733, tv_nsec: 316344060 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
72 Aug 25 23:58:57.845 DEBG will register with Nexus at [fd00:1122:3344:101::a]:12221, component: producer-registration-task
73 Aug 25 23:58:57.846 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::a]:12221/metrics/producers, method: POST, component: producer-registration-task
742026-08-25T23:59:03.387ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::a455:ddff:fe59:ea31 missed solicit interval
752026-08-25T23:59:07.400ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::a455:ddff:fe59:ea31 missed solicit interval
76 Aug 25 23:59:12.475 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::a]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
77 Aug 25 23:59:12.475 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers): operation timed out, delay: 30.622119486s, count: 2, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/oximeter/producer/src/lib.rs:376
78 Aug 25 23:59:13.136 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("5cf4f225-06a4-46dd-8bde-e36aae204692.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("6d80c45d-0f75-4276-a842-872bd53173ae.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("b4d4e362-edf6-4910-b829-5347fa537781.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("b4d4e362-edf6-4910-b829-5347fa537781.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::c)) }], valid_until: Instant { tv_sec: 748, tv_nsec: 979876789 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
79 Aug 25 23:59:13.136 DEBG will register with Nexus at [fd00:1122:3344:101::a]:12221, component: producer-registration-task
80 Aug 25 23:59:13.138 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::a]:12221/metrics/producers, method: POST, component: producer-registration-task
81 Aug 25 23:59:28.138 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::a]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
82 Aug 25 23:59:28.138 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers): operation timed out, delay: 46.286264535s, count: 3, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/oximeter/producer/src/lib.rs:376
83 Aug 25 23:59:29.613 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("5cf4f225-06a4-46dd-8bde-e36aae204692.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("6d80c45d-0f75-4276-a842-872bd53173ae.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("b4d4e362-edf6-4910-b829-5347fa537781.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("b4d4e362-edf6-4910-b829-5347fa537781.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::c)) }], valid_until: Instant { tv_sec: 765, tv_nsec: 458381950 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
84 Aug 25 23:59:29.614 DEBG will register with Nexus at [fd00:1122:3344:101::a]:12221, component: producer-registration-task
85 Aug 25 23:59:29.615 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::a]:12221/metrics/producers, method: POST, component: producer-registration-task
862026-08-25T23:59:44.437ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::a455:ddff:fe59:ea31 expire
872026-08-25T23:59:44.462ZWARNslog-rs: [tfportrear0_0] expiring peer fe80::a455:ddff:fe59:ea31 due to discovery event
882026-08-25T23:59:44.464ZINFOslog-rs: [tfportrear0_0] removing routes 5 from dendrite
892026-08-25T23:59:44.465ZINFOslog-rs: [tfportrear0_0] nbr is fe80::a455:ddff:fe59:ea31@sock server
90 Aug 25 23:59:44.624 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::a]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
91 Aug 25 23:59:44.631 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers): operation timed out, delay: 62.77774643s, count: 4, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/oximeter/producer/src/lib.rs:376
922026-08-25T23:59:45.021ZERROslog-rs: [tfportrear0_0] failed to remove tunnel routes: {} OPTE driver is not attached
932026-08-25T23:59:45.021ZDEBGslog-rs: [tfportrear0_0] redistributing expire to 0 peers
942026-08-25T23:59:45.049ZDEBGslog-rs: [tfportrear0_0] transition solicit -> exchange
952026-08-25T23:59:45.051ZINFOslog-rs: [tfportrear0_0] exchange: listening on [fe80::5ce2:21ff:fe38:a06c]:56797
962026-08-25T23:59:45.069ZINFOslog-rs: [tfportrear0_0] waiting for exchange server to start
972026-08-25T23:59:45.237ZINFOslog-rs: [tfportrear0_0] sending 5 routes to dendrite
982026-08-25T23:59:45.237ZDEBGslog-rs: [tfportrear0_0] sending to dpd host=localhost port=12224
992026-08-25T23:59:45.851ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:1:: -> fe80::a455:ddff:fe59:ea31 on port Rear(Rear("rear0"))/0
1002026-08-25T23:59:45.895ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:2:: -> fe80::a455:ddff:fe59:ea31 on port Rear(Rear("rear0"))/0
1012026-08-25T23:59:45.896ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:3:: -> fe80::a455:ddff:fe59:ea31 on port Rear(Rear("rear0"))/0
1022026-08-25T23:59:45.897ZINFOslog-rs: [tfportrear0_0] adding route fdb0:18c0:4d0d:a02a:: -> fe80::a455:ddff:fe59:ea31 on port Rear(Rear("rear0"))/0
1032026-08-25T23:59:45.897ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:101:: -> fe80::a455:ddff:fe59:ea31 on port Rear(Rear("rear0"))/0
1042026-08-25T23:59:45.898ZINFOslog-rs: [tfportrear0_0] removing routes 0 from dendrite
1052026-08-25T23:59:45.906ZDEBGslog-rs: [tfportrear0_0] redistributing update to 0 peers
106 Aug 25 23:59:49.635 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("5cf4f225-06a4-46dd-8bde-e36aae204692.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("6d80c45d-0f75-4276-a842-872bd53173ae.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("b4d4e362-edf6-4910-b829-5347fa537781.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("b4d4e362-edf6-4910-b829-5347fa537781.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::c)) }], valid_until: Instant { tv_sec: 785, tv_nsec: 479620137 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
107 Aug 25 23:59:49.636 DEBG will register with Nexus at [fd00:1122:3344:101::a]:12221, component: producer-registration-task
108 Aug 25 23:59:49.640 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::a]:12221/metrics/producers, method: POST, component: producer-registration-task