Max level shown:
1 [ Feb 22 00:54:33 Disabled. ]
2 [ Feb 22 00:54:33 Rereading configuration. ]
3 [ Feb 22 00:54:34 Enabled. ]
4 [ Feb 22 00:54:34 Rereading configuration. ]
5 [ Feb 22 00:54:34 Executing start method ("ctrun -l child -o noorphan,regent /opt/oxide/mgd/pkg/mgd_method_script.sh &"). ]
6 [ Feb 22 00:54:34 Method "start" exited with status 0. ]
72025-02-22T00:54:34.694ZINFOslog-rs: signal handler waiting for context
82025-02-22T00:54:34.714ZINFOslog-rs: signal handler waiting got context
92025-02-22T00:54:34.714ZINFOslog-rs: bgp routers: {}
102025-02-22T00:54:34.714ZINFOslog-rs: bfd peers: []
112025-02-22T00:54:34.715ZERROslog-rs: failed to start stats server: underlay address not found
122025-02-22T00:54:34.718ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = GET path = /bfd/peers unit = api-server
132025-02-22T00:54:34.718ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = PUT path = /bfd/peers unit = api-server
142025-02-22T00:54:34.718ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = DELETE path = /bfd/peers/{addr} unit = api-server
152025-02-22T00:54:34.718ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = POST path = /bgp/clear/neighbor unit = api-server
162025-02-22T00:54:34.718ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = DELETE path = /bgp/config/checker unit = api-server
172025-02-22T00:54:34.718ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = GET path = /bgp/config/checker unit = api-server
182025-02-22T00:54:34.718ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = POST path = /bgp/config/checker unit = api-server
192025-02-22T00:54:34.718ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = PUT path = /bgp/config/checker unit = api-server
202025-02-22T00:54:34.718ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = DELETE path = /bgp/config/neighbor unit = api-server
212025-02-22T00:54:34.718ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = GET path = /bgp/config/neighbor unit = api-server
222025-02-22T00:54:34.718ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = POST path = /bgp/config/neighbor unit = api-server
232025-02-22T00:54:34.718ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = PUT path = /bgp/config/neighbor unit = api-server
242025-02-22T00:54:34.718ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = GET path = /bgp/config/neighbors unit = api-server
252025-02-22T00:54:34.718ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = DELETE path = /bgp/config/origin4 unit = api-server
262025-02-22T00:54:34.718ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = GET path = /bgp/config/origin4 unit = api-server
272025-02-22T00:54:34.718ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = POST path = /bgp/config/origin4 unit = api-server
282025-02-22T00:54:34.718ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = PUT path = /bgp/config/origin4 unit = api-server
292025-02-22T00:54:34.718ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = DELETE path = /bgp/config/router unit = api-server
302025-02-22T00:54:34.718ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = GET path = /bgp/config/router unit = api-server
312025-02-22T00:54:34.718ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = POST path = /bgp/config/router unit = api-server
322025-02-22T00:54:34.718ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = PUT path = /bgp/config/router unit = api-server
332025-02-22T00:54:34.718ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = GET path = /bgp/config/routers unit = api-server
342025-02-22T00:54:34.719ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = DELETE path = /bgp/config/shaper unit = api-server
352025-02-22T00:54:34.719ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = GET path = /bgp/config/shaper unit = api-server
362025-02-22T00:54:34.719ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = POST path = /bgp/config/shaper unit = api-server
372025-02-22T00:54:34.719ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = PUT path = /bgp/config/shaper unit = api-server
382025-02-22T00:54:34.719ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = GET path = /bgp/message-history unit = api-server
392025-02-22T00:54:34.719ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = POST path = /bgp/omicron/apply unit = api-server
402025-02-22T00:54:34.719ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = GET path = /bgp/status/exported unit = api-server
412025-02-22T00:54:34.719ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = GET path = /bgp/status/imported unit = api-server
422025-02-22T00:54:34.719ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = GET path = /bgp/status/neighbors unit = api-server
432025-02-22T00:54:34.719ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = GET path = /bgp/status/selected unit = api-server
442025-02-22T00:54:34.719ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = DELETE path = /static/route4 unit = api-server
452025-02-22T00:54:34.719ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = GET path = /static/route4 unit = api-server
462025-02-22T00:54:34.719ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = PUT path = /static/route4 unit = api-server
472025-02-22T00:54:34.719ZINFOslog-rs: admin: listening on [::]:4676
482025-02-22T00:54:34.719ZINFOslog-rs: listening local_addr = [::]:4676 unit = api-server
492025-02-22T00:54:34.720ZDEBGslog-rs: successfully registered DTrace USDT probes local_addr = [::]:4676 unit = api-server
502025-02-22T00:54:34.788ZWARNslog-rs: failed to ensure TEP address fd24:d881:b234:bee8::1 on ASIC: Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6)
512025-02-22T00:54:35.789ZWARNslog-rs: failed to ensure TEP address fd24:d881:b234:bee8::1 on ASIC: Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6)
522025-02-22T00:54:36.790ZWARNslog-rs: failed to ensure TEP address fd24:d881:b234:bee8::1 on ASIC: Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6)
532025-02-22T00:54:37.790ZWARNslog-rs: failed to ensure TEP address fd24:d881:b234:bee8::1 on ASIC: Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6)
542025-02-22T00:54:38.791ZWARNslog-rs: failed to ensure TEP address fd24:d881:b234:bee8::1 on ASIC: Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6)
552025-02-22T00:54:39.791ZWARNslog-rs: failed to ensure TEP address fd24:d881:b234:bee8::1 on ASIC: Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6)
562025-02-22T00:54:40.792ZWARNslog-rs: failed to ensure TEP address fd24:d881:b234:bee8::1 on ASIC: Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6)
572025-02-22T00:54:41.792ZWARNslog-rs: failed to ensure TEP address fd24:d881:b234:bee8::1 on ASIC: Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6)
582025-02-22T00:54:42.793ZWARNslog-rs: failed to ensure TEP address fd24:d881:b234:bee8::1 on ASIC: Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6)
592025-02-22T00:54:43.793ZWARNslog-rs: failed to ensure TEP address fd24:d881:b234:bee8::1 on ASIC: Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6)
602025-02-22T00:54:44.794ZWARNslog-rs: failed to ensure TEP address fd24:d881:b234:bee8::1 on ASIC: Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6)
61 [ Feb 22 00:54:45 Rereading configuration. ]
62 [ Feb 22 00:54:45 Executing refresh method ("/opt/oxide/mgd/pkg/refresh_ctrun_child.sh -c 213"). ]
63 + CONTRACT=''
64 + getopts c: o
65 + CONTRACT=213
66 + getopts c: o
67 + [[ -n 213 ]]
68 + pgrep -c 213
69 + PARENT_PID=2259
70 + [[ -n 2259 ]]
71 + pgrep -P 2259
72 + CHILD_PID=2261
73 + [[ -n 2261 ]]
74 + kill -USR1 2261
752025-02-22T00:54:45.942ZINFOslog-rs: handling smf refresh
762025-02-22T00:54:45.943ZINFOslog-rs: starting stats server on smf refresh
77 [ Feb 22 00:54:45 Method "refresh" exited with status 0. ]
78 Feb 22 00:54:45.944 DEBG registered DTrace probes
79 Feb 22 00:54:45.945 DEBG registered endpoint, versions: all, path: /{producer_id}, method: GET, local_addr: [fd00:1122:3344:101::2]:4677, component: dropshot
80 Feb 22 00:54:45.945 INFO listening, local_addr: [fd00:1122:3344:101::2]:4677, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.15.1/src/server.rs:328
81 Feb 22 00:54:45.945 DEBG successfully registered DTrace USDT probes, local_addr: [fd00:1122:3344:101::2]:4677, component: dropshot
82 Feb 22 00:54:45.945 DEBG Nexus IP not provided, will use DNS to resolve it
83 Feb 22 00:54:45.945 INFO new DNS resolver, addresses: [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53], component: internal-dns-resolver, file: /home/build/.cargo/git/checkouts/omicron-d039c41f152bda83/78772ea/internal-dns/resolver/src/resolver.rs:111
84 Feb 22 00:54:45.945 DEBG starting producer registration task
85 Feb 22 00:54:45.945 INFO starting oximeter metric producer server, interval: 1s, address: [fd00:1122:3344:101::2]:4677, producer_id: 64905d67-52f4-4a79-98ad-67939d7343d8, file: /home/build/.cargo/git/checkouts/omicron-d039c41f152bda83/78772ea/oximeter/producer/src/lib.rs:279
86 Feb 22 00:54:45.945 DEBG registering / renewing oximeter producer lease with Nexus, component: producer-registration-task
872025-02-22T00:54:46.198ZINFOslog-rs: accepted connection local_addr = [::]:4676 remote_addr = [fd00:1122:3344:101::1]:46132 unit = api-server
882025-02-22T00:54:46.199ZINFOslog-rs: request completed latency_us = 482 local_addr = [::]:4676 method = PUT remote_addr = [fd00:1122:3344:101::1]:46132 req_id = fe432f03-55bc-4ad4-a5dd-e9fc0148080c response_code = 204 unit = api-server uri = /static/route4
892025-02-22T00:54:46.199ZERROslog-rs: handling change failed: libnet error route io error No such process (os error 3)
902025-02-22T00:54:46.199ZINFOslog-rs: restarting sync loop
91 Feb 22 00:54:55.948 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("407c6397-e5e8-474a-82a5-2ddcd6a5fbb1.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("4f2d90dc-0f1a-4fef-a6eb-7e9430c6f5cb.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("f1598ca1-8c9f-4830-9931-a5c2d7746097.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("f1598ca1-8c9f-4830-9931-a5c2d7746097.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::b))) }], valid_until: Instant { tv_sec: 594, tv_nsec: 809867652 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
92 Feb 22 00:54:55.949 DEBG using nexus address for registration, addr: [fd00:1122:3344:101::a]:12221, component: producer-registration-task
93 Feb 22 00:54:55.981 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::a]:12221/metrics/producers, method: POST, component: producer-registration-task
94 Feb 22 00:55:10.983 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::a]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
95 Feb 22 00:55:10.983 WARN failed to register as a producer with Nexus, will retry, error: "Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers)", delay: 263.691796ms, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-d039c41f152bda83/78772ea/oximeter/producer/src/lib.rs:420
96 Feb 22 00:55:11.278 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::a]:12221/metrics/producers, method: POST, component: producer-registration-task
97 Feb 22 00:55:26.279 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::a]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
98 Feb 22 00:55:26.280 WARN failed to register as a producer with Nexus, will retry, error: "Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers)", delay: 559.074938ms, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-d039c41f152bda83/78772ea/oximeter/producer/src/lib.rs:420
99 Feb 22 00:55:26.871 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::a]:12221/metrics/producers, method: POST, component: producer-registration-task
100 Feb 22 00:55:41.871 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::a]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
101 Feb 22 00:55:41.872 WARN failed to register as a producer with Nexus, will retry, error: "Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers)", delay: 1.45708905s, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-d039c41f152bda83/78772ea/oximeter/producer/src/lib.rs:420
102 Feb 22 00:55:43.360 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::a]:12221/metrics/producers, method: POST, component: producer-registration-task
103 Feb 22 00:55:58.361 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::a]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
104 Feb 22 00:55:58.362 WARN failed to register as a producer with Nexus, will retry, error: "Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers)", delay: 1.397518582s, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-d039c41f152bda83/78772ea/oximeter/producer/src/lib.rs:420
105 Feb 22 00:55:59.794 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::a]:12221/metrics/producers, method: POST, component: producer-registration-task
106 Feb 22 00:56:14.795 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::a]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
107 Feb 22 00:56:14.796 WARN failed to register as a producer with Nexus, will retry, error: "Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers)", delay: 2.62799139s, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-d039c41f152bda83/78772ea/oximeter/producer/src/lib.rs:420
108 Feb 22 00:56:17.455 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::a]:12221/metrics/producers, method: POST, component: producer-registration-task
109 Feb 22 00:56:32.456 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::a]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
110 Feb 22 00:56:32.457 WARN failed to register as a producer with Nexus, will retry, error: "Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers)", delay: 8.673204959s, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-d039c41f152bda83/78772ea/oximeter/producer/src/lib.rs:420
111 Feb 22 00:56:41.162 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::a]:12221/metrics/producers, method: POST, component: producer-registration-task
112 Feb 22 00:56:56.163 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::a]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
113 Feb 22 00:56:56.164 WARN failed to register as a producer with Nexus, will retry, error: "Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers)", delay: 22.886486069s, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-d039c41f152bda83/78772ea/oximeter/producer/src/lib.rs:420
114 Feb 22 00:57:19.083 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::a]:12221/metrics/producers, method: POST, component: producer-registration-task
115 Feb 22 00:57:34.084 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::a]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
116 Feb 22 00:57:34.086 WARN failed to register as a producer with Nexus, will retry, error: "Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers)", delay: 39.374057544s, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-d039c41f152bda83/78772ea/oximeter/producer/src/lib.rs:420
117 Feb 22 00:58:13.491 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::a]:12221/metrics/producers, method: POST, component: producer-registration-task
118 Feb 22 00:58:28.493 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::a]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
119 Feb 22 00:58:28.494 WARN failed to register as a producer with Nexus, will retry, error: "Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers)", delay: 55.266761927s, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-d039c41f152bda83/78772ea/oximeter/producer/src/lib.rs:420
120 Feb 22 00:59:23.792 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::a]:12221/metrics/producers, method: POST, component: producer-registration-task
121 Feb 22 00:59:38.794 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::a]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
122 Feb 22 00:59:38.795 WARN failed to register as a producer with Nexus, will retry, error: "Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers)", delay: 177.25910079s, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-d039c41f152bda83/78772ea/oximeter/producer/src/lib.rs:420