Max level shown:
1 [ Feb 14 20:02:42 Disabled. ]
2 [ Feb 14 20:02:42 Rereading configuration. ]
3 [ Feb 14 20:02:42 Enabled. ]
4 [ Feb 14 20:02:42 Rereading configuration. ]
5 [ Feb 14 20:02:43 Executing start method ("ctrun -l child -o noorphan,regent /opt/oxide/mgd/pkg/mgd_method_script.sh &"). ]
6 [ Feb 14 20:02:43 Method "start" exited with status 0. ]
72025-02-14T20:02:43.056ZINFOslog-rs: signal handler waiting for context
82025-02-14T20:02:43.078ZINFOslog-rs: signal handler waiting got context
92025-02-14T20:02:43.078ZINFOslog-rs: bgp routers: {}
102025-02-14T20:02:43.078ZINFOslog-rs: bfd peers: []
112025-02-14T20:02:43.078ZERROslog-rs: failed to start stats server: underlay address not found
122025-02-14T20:02:43.081ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = GET path = /bfd/peers unit = api-server
132025-02-14T20:02:43.081ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = PUT path = /bfd/peers unit = api-server
142025-02-14T20:02:43.081ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = DELETE path = /bfd/peers/{addr} unit = api-server
152025-02-14T20:02:43.081ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = POST path = /bgp/clear/neighbor unit = api-server
162025-02-14T20:02:43.081ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = DELETE path = /bgp/config/checker unit = api-server
172025-02-14T20:02:43.081ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = GET path = /bgp/config/checker unit = api-server
182025-02-14T20:02:43.081ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = POST path = /bgp/config/checker unit = api-server
192025-02-14T20:02:43.081ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = PUT path = /bgp/config/checker unit = api-server
202025-02-14T20:02:43.081ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = DELETE path = /bgp/config/neighbor unit = api-server
212025-02-14T20:02:43.081ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = GET path = /bgp/config/neighbor unit = api-server
222025-02-14T20:02:43.081ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = POST path = /bgp/config/neighbor unit = api-server
232025-02-14T20:02:43.081ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = PUT path = /bgp/config/neighbor unit = api-server
242025-02-14T20:02:43.081ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = GET path = /bgp/config/neighbors unit = api-server
252025-02-14T20:02:43.081ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = DELETE path = /bgp/config/origin4 unit = api-server
262025-02-14T20:02:43.081ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = GET path = /bgp/config/origin4 unit = api-server
272025-02-14T20:02:43.081ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = POST path = /bgp/config/origin4 unit = api-server
282025-02-14T20:02:43.081ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = PUT path = /bgp/config/origin4 unit = api-server
292025-02-14T20:02:43.081ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = DELETE path = /bgp/config/router unit = api-server
302025-02-14T20:02:43.081ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = GET path = /bgp/config/router unit = api-server
312025-02-14T20:02:43.082ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = POST path = /bgp/config/router unit = api-server
322025-02-14T20:02:43.082ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = PUT path = /bgp/config/router unit = api-server
332025-02-14T20:02:43.082ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = GET path = /bgp/config/routers unit = api-server
342025-02-14T20:02:43.082ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = DELETE path = /bgp/config/shaper unit = api-server
352025-02-14T20:02:43.082ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = GET path = /bgp/config/shaper unit = api-server
362025-02-14T20:02:43.082ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = POST path = /bgp/config/shaper unit = api-server
372025-02-14T20:02:43.082ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = PUT path = /bgp/config/shaper unit = api-server
382025-02-14T20:02:43.082ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = GET path = /bgp/message-history unit = api-server
392025-02-14T20:02:43.082ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = POST path = /bgp/omicron/apply unit = api-server
402025-02-14T20:02:43.082ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = GET path = /bgp/status/exported unit = api-server
412025-02-14T20:02:43.082ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = GET path = /bgp/status/imported unit = api-server
422025-02-14T20:02:43.082ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = GET path = /bgp/status/neighbors unit = api-server
432025-02-14T20:02:43.082ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = GET path = /bgp/status/selected unit = api-server
442025-02-14T20:02:43.082ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = DELETE path = /static/route4 unit = api-server
452025-02-14T20:02:43.082ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = GET path = /static/route4 unit = api-server
462025-02-14T20:02:43.082ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = PUT path = /static/route4 unit = api-server
472025-02-14T20:02:43.082ZINFOslog-rs: admin: listening on [::]:4676
482025-02-14T20:02:43.082ZINFOslog-rs: listening local_addr = [::]:4676 unit = api-server
492025-02-14T20:02:43.084ZDEBGslog-rs: successfully registered DTrace USDT probes local_addr = [::]:4676 unit = api-server
502025-02-14T20:02:43.158ZWARNslog-rs: failed to ensure TEP address fd05:146f:7918:7e79::1 on ASIC: Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6)
512025-02-14T20:02:44.160ZWARNslog-rs: failed to ensure TEP address fd05:146f:7918:7e79::1 on ASIC: Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6)
522025-02-14T20:02:45.160ZWARNslog-rs: failed to ensure TEP address fd05:146f:7918:7e79::1 on ASIC: Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6)
532025-02-14T20:02:46.161ZWARNslog-rs: failed to ensure TEP address fd05:146f:7918:7e79::1 on ASIC: Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6)
542025-02-14T20:02:47.161ZWARNslog-rs: failed to ensure TEP address fd05:146f:7918:7e79::1 on ASIC: Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6)
552025-02-14T20:02:48.162ZWARNslog-rs: failed to ensure TEP address fd05:146f:7918:7e79::1 on ASIC: Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6)
562025-02-14T20:02:49.162ZWARNslog-rs: failed to ensure TEP address fd05:146f:7918:7e79::1 on ASIC: Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6)
572025-02-14T20:02:50.163ZWARNslog-rs: failed to ensure TEP address fd05:146f:7918:7e79::1 on ASIC: Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6)
582025-02-14T20:02:51.163ZWARNslog-rs: failed to ensure TEP address fd05:146f:7918:7e79::1 on ASIC: Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6)
592025-02-14T20:02:52.164ZWARNslog-rs: failed to ensure TEP address fd05:146f:7918:7e79::1 on ASIC: Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6)
602025-02-14T20:02:53.165ZWARNslog-rs: failed to ensure TEP address fd05:146f:7918:7e79::1 on ASIC: Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6)
61 [ Feb 14 20:02:54 Rereading configuration. ]
62 [ Feb 14 20:02:54 Executing refresh method ("/opt/oxide/mgd/pkg/refresh_ctrun_child.sh -c 213"). ]
63 + CONTRACT=''
64 + getopts c: o
65 + CONTRACT=213
66 + getopts c: o
67 + [[ -n 213 ]]
68 + pgrep -c 213
69 + PARENT_PID=2258
70 + [[ -n 2258 ]]
71 + pgrep -P 2258
72 + CHILD_PID=2259
73 + [[ -n 2259 ]]
74 + kill -USR1 2259
752025-02-14T20:02:54.744ZINFOslog-rs: handling smf refresh
76 [ Feb 14 20:02:54 Method "refresh" exited with status 0. ]
772025-02-14T20:02:54.746ZINFOslog-rs: starting stats server on smf refresh
78 Feb 14 20:02:54.781 DEBG registered DTrace probes
79 Feb 14 20:02:54.781 DEBG registered endpoint, versions: all, path: /{producer_id}, method: GET, local_addr: [fd00:1122:3344:101::2]:4677, component: dropshot
80 Feb 14 20:02:54.781 INFO listening, local_addr: [fd00:1122:3344:101::2]:4677, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:328
81 Feb 14 20:02:54.781 DEBG successfully registered DTrace USDT probes, local_addr: [fd00:1122:3344:101::2]:4677, component: dropshot
82 Feb 14 20:02:54.782 DEBG Nexus IP not provided, will use DNS to resolve it
83 Feb 14 20:02:54.782 INFO new DNS resolver, addresses: [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53], component: internal-dns-resolver, file: /home/build/.cargo/git/checkouts/omicron-d039c41f152bda83/a68d462/internal-dns/resolver/src/resolver.rs:111
84 Feb 14 20:02:54.782 DEBG starting producer registration task
85 Feb 14 20:02:54.782 INFO starting oximeter metric producer server, interval: 1s, address: [fd00:1122:3344:101::2]:4677, producer_id: 173135d8-1db0-4042-8827-8fcecca36c90, file: /home/build/.cargo/git/checkouts/omicron-d039c41f152bda83/a68d462/oximeter/producer/src/lib.rs:279
86 Feb 14 20:02:54.782 DEBG registering / renewing oximeter producer lease with Nexus, component: producer-registration-task
872025-02-14T20:02:55.087ZINFOslog-rs: accepted connection local_addr = [::]:4676 remote_addr = [fd00:1122:3344:101::1]:55903 unit = api-server
882025-02-14T20:02:55.088ZINFOslog-rs: request completed latency_us = 384 local_addr = [::]:4676 method = PUT remote_addr = [fd00:1122:3344:101::1]:55903 req_id = c2fb7272-eb3e-4bb9-8fe3-2d1e932a094c response_code = 204 unit = api-server uri = /static/route4
892025-02-14T20:02:55.089ZERROslog-rs: handling change failed: libnet error route io error No such process (os error 3)
902025-02-14T20:02:55.089ZINFOslog-rs: restarting sync loop
91 Feb 14 20:03:04.784 WARN failed to lookup Nexus IP, will retry, error: "no record found for Query { name: Name(\"_nexus._tcp.control-plane.oxide.internal.\"), query_type: SRV, query_class: IN }", delay: 310.625237ms, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-d039c41f152bda83/a68d462/oximeter/producer/src/lib.rs:387
92 Feb 14 20:03:05.096 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("15c1e91b-5078-4620-9a5f-350a017795ec.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("2c84a910-1bb9-4b4d-9f4f-e640af248108.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("6efc27f1-6196-413c-be8b-46c95f2b6949.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("6efc27f1-6196-413c-be8b-46c95f2b6949.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::b))) }], valid_until: Instant { tv_sec: 652, tv_nsec: 352503031 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
93 Feb 14 20:03:05.097 DEBG using nexus address for registration, addr: [fd00:1122:3344:101::c]:12221, component: producer-registration-task
94 Feb 14 20:03:05.129 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::c]:12221/metrics/producers, method: POST, component: producer-registration-task
95 Feb 14 20:03:20.130 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::c]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
96 Feb 14 20:03:20.131 WARN failed to register as a producer with Nexus, will retry, error: "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers)", delay: 350.092932ms, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-d039c41f152bda83/a68d462/oximeter/producer/src/lib.rs:420
97 Feb 14 20:03:20.513 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::c]:12221/metrics/producers, method: POST, component: producer-registration-task
98 Feb 14 20:03:35.514 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::c]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
99 Feb 14 20:03:35.515 WARN failed to register as a producer with Nexus, will retry, error: "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers)", delay: 489.782223ms, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-d039c41f152bda83/a68d462/oximeter/producer/src/lib.rs:420
100 Feb 14 20:03:36.035 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::c]:12221/metrics/producers, method: POST, component: producer-registration-task
101 Feb 14 20:03:51.037 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::c]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
102 Feb 14 20:03:51.038 WARN failed to register as a producer with Nexus, will retry, error: "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers)", delay: 1.02098205s, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-d039c41f152bda83/a68d462/oximeter/producer/src/lib.rs:420
103 Feb 14 20:03:52.093 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::c]:12221/metrics/producers, method: POST, component: producer-registration-task
104 Feb 14 20:04:07.094 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::c]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
105 Feb 14 20:04:07.094 WARN failed to register as a producer with Nexus, will retry, error: "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers)", delay: 1.850707088s, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-d039c41f152bda83/a68d462/oximeter/producer/src/lib.rs:420
106 Feb 14 20:04:08.977 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::c]:12221/metrics/producers, method: POST, component: producer-registration-task
107 Feb 14 20:04:23.978 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::c]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
108 Feb 14 20:04:23.979 WARN failed to register as a producer with Nexus, will retry, error: "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers)", delay: 3.559600964s, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-d039c41f152bda83/a68d462/oximeter/producer/src/lib.rs:420
109 Feb 14 20:04:27.570 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::c]:12221/metrics/producers, method: POST, component: producer-registration-task
110 Feb 14 20:04:42.572 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::c]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
111 Feb 14 20:04:42.573 WARN failed to register as a producer with Nexus, will retry, error: "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers)", delay: 11.82466821s, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-d039c41f152bda83/a68d462/oximeter/producer/src/lib.rs:420
112 Feb 14 20:04:54.430 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::c]:12221/metrics/producers, method: POST, component: producer-registration-task
113 Feb 14 20:05:09.430 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::c]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
114 Feb 14 20:05:09.431 WARN failed to register as a producer with Nexus, will retry, error: "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers)", delay: 9.255921654s, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-d039c41f152bda83/a68d462/oximeter/producer/src/lib.rs:420
115 Feb 14 20:05:18.719 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::c]:12221/metrics/producers, method: POST, component: producer-registration-task
116 Feb 14 20:05:33.721 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::c]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
117 Feb 14 20:05:33.722 WARN failed to register as a producer with Nexus, will retry, error: "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers)", delay: 33.84721165s, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-d039c41f152bda83/a68d462/oximeter/producer/src/lib.rs:420
118 Feb 14 20:06:07.603 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::c]:12221/metrics/producers, method: POST, component: producer-registration-task
119 Feb 14 20:06:22.604 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::c]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
120 Feb 14 20:06:22.605 WARN failed to register as a producer with Nexus, will retry, error: "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers)", delay: 82.822076116s, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-d039c41f152bda83/a68d462/oximeter/producer/src/lib.rs:420
121 Feb 14 20:07:45.460 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::c]:12221/metrics/producers, method: POST, component: producer-registration-task
122 Feb 14 20:08:00.460 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::c]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
123 Feb 14 20:08:00.461 WARN failed to register as a producer with Nexus, will retry, error: "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers)", delay: 64.336412075s, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-d039c41f152bda83/a68d462/oximeter/producer/src/lib.rs:420
124 Feb 14 20:09:04.830 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::c]:12221/metrics/producers, method: POST, component: producer-registration-task
125 Feb 14 20:09:19.831 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::c]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
126 Feb 14 20:09:19.832 WARN failed to register as a producer with Nexus, will retry, error: "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers)", delay: 249.255335258s, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-d039c41f152bda83/a68d462/oximeter/producer/src/lib.rs:420