Max level shown:
1 [ May 13 22:58:45 Disabled. ]
2 [ May 13 22:58:45 Rereading configuration. ]
3 [ May 13 22:58:46 Enabled. ]
4 [ May 13 22:58:46 Rereading configuration. ]
5 [ May 13 22:58:46 Executing start method ("ctrun -l child -o noorphan,regent /opt/oxide/mgd/pkg/mgd_method_script.sh &"). ]
6 [ May 13 22:58:46 Method "start" exited with status 0. ]
72025-05-13T22:58:46.437ZINFOslog-rs: signal handler waiting for context
82025-05-13T22:58:46.457ZINFOslog-rs: signal handler waiting got context
92025-05-13T22:58:46.457ZINFOslog-rs: bgp routers: {}
102025-05-13T22:58:46.458ZINFOslog-rs: bfd peers: []
112025-05-13T22:58:46.458ZERROslog-rs: failed to start stats server: underlay address not found
122025-05-13T22:58:46.459ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = GET path = /bfd/peers unit = api-server versions = all
132025-05-13T22:58:46.459ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = PUT path = /bfd/peers unit = api-server versions = all
142025-05-13T22:58:46.459ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = DELETE path = /bfd/peers/{addr} unit = api-server versions = all
152025-05-13T22:58:46.459ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = POST path = /bgp/clear/neighbor unit = api-server versions = all
162025-05-13T22:58:46.459ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = DELETE path = /bgp/config/checker unit = api-server versions = all
172025-05-13T22:58:46.459ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = GET path = /bgp/config/checker unit = api-server versions = all
182025-05-13T22:58:46.459ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = POST path = /bgp/config/checker unit = api-server versions = all
192025-05-13T22:58:46.459ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = PUT path = /bgp/config/checker unit = api-server versions = all
202025-05-13T22:58:46.459ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = DELETE path = /bgp/config/neighbor unit = api-server versions = all
212025-05-13T22:58:46.459ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = GET path = /bgp/config/neighbor unit = api-server versions = all
222025-05-13T22:58:46.459ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = POST path = /bgp/config/neighbor unit = api-server versions = all
232025-05-13T22:58:46.459ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = PUT path = /bgp/config/neighbor unit = api-server versions = all
242025-05-13T22:58:46.459ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = GET path = /bgp/config/neighbors unit = api-server versions = all
252025-05-13T22:58:46.459ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = DELETE path = /bgp/config/origin4 unit = api-server versions = all
262025-05-13T22:58:46.459ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = GET path = /bgp/config/origin4 unit = api-server versions = all
272025-05-13T22:58:46.459ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = POST path = /bgp/config/origin4 unit = api-server versions = all
282025-05-13T22:58:46.459ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = PUT path = /bgp/config/origin4 unit = api-server versions = all
292025-05-13T22:58:46.459ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = DELETE path = /bgp/config/router unit = api-server versions = all
302025-05-13T22:58:46.459ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = GET path = /bgp/config/router unit = api-server versions = all
312025-05-13T22:58:46.459ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = POST path = /bgp/config/router unit = api-server versions = all
322025-05-13T22:58:46.459ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = PUT path = /bgp/config/router unit = api-server versions = all
332025-05-13T22:58:46.459ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = GET path = /bgp/config/routers unit = api-server versions = all
342025-05-13T22:58:46.459ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = DELETE path = /bgp/config/shaper unit = api-server versions = all
352025-05-13T22:58:46.459ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = GET path = /bgp/config/shaper unit = api-server versions = all
362025-05-13T22:58:46.459ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = POST path = /bgp/config/shaper unit = api-server versions = all
372025-05-13T22:58:46.459ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = PUT path = /bgp/config/shaper unit = api-server versions = all
382025-05-13T22:58:46.459ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = GET path = /bgp/message-history unit = api-server versions = all
392025-05-13T22:58:46.459ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = POST path = /bgp/omicron/apply unit = api-server versions = all
402025-05-13T22:58:46.459ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = GET path = /bgp/status/exported unit = api-server versions = all
412025-05-13T22:58:46.459ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = GET path = /bgp/status/imported unit = api-server versions = all
422025-05-13T22:58:46.459ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = GET path = /bgp/status/neighbors unit = api-server versions = all
432025-05-13T22:58:46.459ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = GET path = /bgp/status/selected unit = api-server versions = all
442025-05-13T22:58:46.460ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = DELETE path = /static/route4 unit = api-server versions = all
452025-05-13T22:58:46.460ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = GET path = /static/route4 unit = api-server versions = all
462025-05-13T22:58:46.460ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = PUT path = /static/route4 unit = api-server versions = all
472025-05-13T22:58:46.460ZINFOslog-rs: admin: listening on [::]:4676
482025-05-13T22:58:46.460ZINFOslog-rs: listening local_addr = [::]:4676 unit = api-server
492025-05-13T22:58:46.461ZDEBGslog-rs: successfully registered DTrace USDT probes local_addr = [::]:4676 unit = api-server
502025-05-13T22:58:46.461ZWARNslog-rs: failed to ensure TEP address fd30:23a6:d379:146::1 on ASIC: Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6)
512025-05-13T22:58:46.461ZERROslog-rs: get originated tunnel endpoints: Communication Error: error sending request for url (http://localhost:8000/originated_tunnel_endpoints)
522025-05-13T22:58:47.462ZWARNslog-rs: failed to ensure TEP address fd30:23a6:d379:146::1 on ASIC: Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6)
532025-05-13T22:58:48.464ZWARNslog-rs: failed to ensure TEP address fd30:23a6:d379:146::1 on ASIC: Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6)
542025-05-13T22:58:49.465ZWARNslog-rs: failed to ensure TEP address fd30:23a6:d379:146::1 on ASIC: Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6)
552025-05-13T22:58:50.465ZWARNslog-rs: failed to ensure TEP address fd30:23a6:d379:146::1 on ASIC: Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6)
562025-05-13T22:58:51.466ZWARNslog-rs: failed to ensure TEP address fd30:23a6:d379:146::1 on ASIC: Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6)
572025-05-13T22:58:52.466ZWARNslog-rs: failed to ensure TEP address fd30:23a6:d379:146::1 on ASIC: Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6)
582025-05-13T22:58:53.467ZWARNslog-rs: failed to ensure TEP address fd30:23a6:d379:146::1 on ASIC: Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6)
592025-05-13T22:58:54.467ZWARNslog-rs: failed to ensure TEP address fd30:23a6:d379:146::1 on ASIC: Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6)
602025-05-13T22:58:55.468ZWARNslog-rs: failed to ensure TEP address fd30:23a6:d379:146::1 on ASIC: Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6)
612025-05-13T22:58:56.468ZWARNslog-rs: failed to ensure TEP address fd30:23a6:d379:146::1 on ASIC: Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6)
62 [ May 13 22:58:57 Rereading configuration. ]
63 [ May 13 22:58:57 Executing refresh method ("/opt/oxide/mgd/pkg/refresh_ctrun_child.sh -c 214"). ]
64 + CONTRACT=''
65 + getopts c: o
66 + CONTRACT=214
67 + getopts c: o
68 + [[ -n 214 ]]
69 + pgrep -c 214
70 + PARENT_PID=2277
71 + [[ -n 2277 ]]
72 + pgrep -P 2277
73 + CHILD_PID=2278
74 + [[ -n 2278 ]]
75 + kill -USR1 2278
762025-05-13T22:58:57.824ZINFOslog-rs: handling smf refresh
772025-05-13T22:58:57.826ZINFOslog-rs: starting stats server on smf refresh
78 [ May 13 22:58:57 Method "refresh" exited with status 0. ]
79 May 13 22:58:57.827 DEBG registered DTrace probes
80 May 13 22:58:57.827 DEBG registered endpoint, versions: all, path: /{producer_id}, method: GET, local_addr: [fd00:1122:3344:101::2]:4677, component: dropshot
81 May 13 22:58:57.827 INFO listening, local_addr: [fd00:1122:3344:101::2]:4677, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:328
82 May 13 22:58:57.828 DEBG successfully registered DTrace USDT probes, local_addr: [fd00:1122:3344:101::2]:4677, component: dropshot
83 May 13 22:58:57.828 DEBG Nexus IP not provided, will use DNS to resolve it
84 May 13 22:58:57.828 INFO new DNS resolver, addresses: [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53], component: internal-dns-resolver, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/1d3958c/internal-dns/resolver/src/resolver.rs:111
85 May 13 22:58:57.828 DEBG starting producer registration task
86 May 13 22:58:57.828 INFO starting oximeter metric producer server, interval: 1s, address: [fd00:1122:3344:101::2]:4677, producer_id: 8f482d19-1e65-49c5-b0f2-508cd6cf85bd, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/1d3958c/oximeter/producer/src/lib.rs:279
87 May 13 22:58:57.828 DEBG registering as oximeter producer with Nexus, registration_count: 0, component: producer-registration-task
882025-05-13T22:58:58.094ZINFOslog-rs: accepted connection local_addr = [::]:4676 remote_addr = [fd00:1122:3344:101::1]:46579 unit = api-server
892025-05-13T22:58:58.094ZINFOslog-rs: request completed latency_us = 493 local_addr = [::]:4676 method = PUT remote_addr = [fd00:1122:3344:101::1]:46579 req_id = 4d75acb4-acd3-4dd6-ad1b-cc91d7460288 response_code = 204 unit = api-server uri = /static/route4
902025-05-13T22:58:58.095ZERROslog-rs: handling change failed: libnet error route io error No such process (os error 3)
912025-05-13T22:58:58.095ZINFOslog-rs: restarting sync loop
92 May 13 22:59:07.833 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("049171a7-8aff-47c3-bb96-a4154a077926.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("6f1779bd-e1a8-42f2-ab98-d5ed269e565e.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("ddc040c8-99a0-469b-88c2-2773cfc2e24b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("ddc040c8-99a0-469b-88c2-2773cfc2e24b.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::a))) }], valid_until: Instant { tv_sec: 832, tv_nsec: 527174588 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
93 May 13 22:59:07.833 DEBG will register with Nexus at [fd00:1122:3344:101::b]:12221, component: producer-registration-task
94 May 13 22:59:07.834 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::b]:12221/metrics/producers, method: POST, component: producer-registration-task
95 May 13 22:59:23.368 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::b]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
96 May 13 22:59:23.368 WARN failed to register with Nexus, will retry, error: "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers)", delay: 25.006529074s, count: 0, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/1d3958c/oximeter/producer/src/lib.rs:371
97 May 13 22:59:23.656 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("049171a7-8aff-47c3-bb96-a4154a077926.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("6f1779bd-e1a8-42f2-ab98-d5ed269e565e.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("ddc040c8-99a0-469b-88c2-2773cfc2e24b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("ddc040c8-99a0-469b-88c2-2773cfc2e24b.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::a))) }], valid_until: Instant { tv_sec: 847, tv_nsec: 815553951 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
98 May 13 22:59:23.656 DEBG will register with Nexus at [fd00:1122:3344:101::b]:12221, component: producer-registration-task
99 May 13 22:59:23.656 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::b]:12221/metrics/producers, method: POST, component: producer-registration-task
100 May 13 22:59:38.697 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::b]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
101 May 13 22:59:38.697 WARN failed to register with Nexus, will retry, error: "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers)", delay: 40.29406344s, count: 1, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/1d3958c/oximeter/producer/src/lib.rs:371
102 May 13 22:59:39.158 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("049171a7-8aff-47c3-bb96-a4154a077926.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("6f1779bd-e1a8-42f2-ab98-d5ed269e565e.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("ddc040c8-99a0-469b-88c2-2773cfc2e24b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("ddc040c8-99a0-469b-88c2-2773cfc2e24b.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::a))) }], valid_until: Instant { tv_sec: 863, tv_nsec: 275954629 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
103 May 13 22:59:39.158 DEBG will register with Nexus at [fd00:1122:3344:101::b]:12221, component: producer-registration-task
104 May 13 22:59:39.158 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::b]:12221/metrics/producers, method: POST, component: producer-registration-task
105 May 13 22:59:54.200 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::b]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
106 May 13 22:59:54.200 WARN failed to register with Nexus, will retry, error: "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers)", delay: 55.754965783s, count: 2, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/1d3958c/oximeter/producer/src/lib.rs:371
107 May 13 22:59:54.917 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("049171a7-8aff-47c3-bb96-a4154a077926.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("6f1779bd-e1a8-42f2-ab98-d5ed269e565e.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("ddc040c8-99a0-469b-88c2-2773cfc2e24b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("ddc040c8-99a0-469b-88c2-2773cfc2e24b.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::a))) }], valid_until: Instant { tv_sec: 878, tv_nsec: 992975257 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
108 May 13 22:59:54.918 DEBG will register with Nexus at [fd00:1122:3344:101::b]:12221, component: producer-registration-task
109 May 13 22:59:54.918 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::b]:12221/metrics/producers, method: POST, component: producer-registration-task