Max level shown:
1 [ Oct 16 04:19:11 Disabled. ]
2 [ Oct 16 04:19:11 Rereading configuration. ]
3 [ Oct 16 04:19:12 Enabled. ]
4 [ Oct 16 04:19:12 Executing start method ("ctrun -l child -o noorphan,regent /opt/oxide/mgd/pkg/mgd_method_script.sh &"). ]
5 [ Oct 16 04:19:12 Method "start" exited with status 0. ]
6 [ Oct 16 04:19:12 Rereading configuration. ]
7 [ Oct 16 04:19:12 Executing refresh method ("/opt/oxide/mgd/pkg/refresh_ctrun_child.sh -c 214"). ]
8 + CONTRACT=''
9 + getopts c: o
10 + CONTRACT=214
11 + getopts c: o
12 + [[ -n 214 ]]
13 + pgrep -c 214
14 + PARENT_PID=2283
15 + [[ -n 2283 ]]
16 + pgrep -P 2283
17 + CHILD_PID=2284
18 + [[ -n 2284 ]]
19 + kill -USR1 2284
20 [ Oct 16 04:19:12 Method "refresh" exited with status 0. ]
21 [ Oct 16 04:19:12 Stopping because all processes in service exited. ]
22 [ Oct 16 04:19:12 Executing stop method (:kill). ]
23 [ Oct 16 04:19:12 Executing start method ("ctrun -l child -o noorphan,regent /opt/oxide/mgd/pkg/mgd_method_script.sh &"). ]
24 [ Oct 16 04:19:12 Method "start" exited with status 0. ]
252024-10-16T04:19:12.226ZINFOslog-rs: signal handler waiting for context
262024-10-16T04:19:12.246ZINFOslog-rs: signal handler waiting got context
272024-10-16T04:19:12.246ZINFOslog-rs: bgp routers: {}
282024-10-16T04:19:12.246ZINFOslog-rs: bfd peers: []
292024-10-16T04:19:12.247ZERROslog-rs: failed to start stats server: underlay address not found
302024-10-16T04:19:12.250ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = GET path = /bfd/peers unit = api-server
312024-10-16T04:19:12.250ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = PUT path = /bfd/peers unit = api-server
322024-10-16T04:19:12.250ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = DELETE path = /bfd/peers/{addr} unit = api-server
332024-10-16T04:19:12.250ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = DELETE path = /bgp/config/checker unit = api-server
342024-10-16T04:19:12.250ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = GET path = /bgp/config/checker unit = api-server
352024-10-16T04:19:12.250ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = POST path = /bgp/config/checker unit = api-server
362024-10-16T04:19:12.250ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = PUT path = /bgp/config/checker unit = api-server
372024-10-16T04:19:12.250ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = DELETE path = /bgp/config/neighbor unit = api-server
382024-10-16T04:19:12.250ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = GET path = /bgp/config/neighbor unit = api-server
392024-10-16T04:19:12.251ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = POST path = /bgp/config/neighbor unit = api-server
402024-10-16T04:19:12.251ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = PUT path = /bgp/config/neighbor unit = api-server
412024-10-16T04:19:12.251ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = GET path = /bgp/config/neighbors unit = api-server
422024-10-16T04:19:12.251ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = DELETE path = /bgp/config/origin4 unit = api-server
432024-10-16T04:19:12.251ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = GET path = /bgp/config/origin4 unit = api-server
442024-10-16T04:19:12.251ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = POST path = /bgp/config/origin4 unit = api-server
452024-10-16T04:19:12.251ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = PUT path = /bgp/config/origin4 unit = api-server
462024-10-16T04:19:12.251ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = DELETE path = /bgp/config/router unit = api-server
472024-10-16T04:19:12.251ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = GET path = /bgp/config/router unit = api-server
482024-10-16T04:19:12.251ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = POST path = /bgp/config/router unit = api-server
492024-10-16T04:19:12.251ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = PUT path = /bgp/config/router unit = api-server
502024-10-16T04:19:12.251ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = GET path = /bgp/config/routers unit = api-server
512024-10-16T04:19:12.251ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = DELETE path = /bgp/config/shaper unit = api-server
522024-10-16T04:19:12.251ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = GET path = /bgp/config/shaper unit = api-server
532024-10-16T04:19:12.251ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = POST path = /bgp/config/shaper unit = api-server
542024-10-16T04:19:12.251ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = PUT path = /bgp/config/shaper unit = api-server
552024-10-16T04:19:12.251ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = GET path = /bgp/message-history unit = api-server
562024-10-16T04:19:12.251ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = POST path = /bgp/omicron/apply unit = api-server
572024-10-16T04:19:12.251ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = GET path = /bgp/status/exported unit = api-server
582024-10-16T04:19:12.251ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = GET path = /bgp/status/imported unit = api-server
592024-10-16T04:19:12.251ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = GET path = /bgp/status/neighbors unit = api-server
602024-10-16T04:19:12.251ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = GET path = /bgp/status/selected unit = api-server
612024-10-16T04:19:12.251ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = DELETE path = /static/route4 unit = api-server
622024-10-16T04:19:12.251ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = GET path = /static/route4 unit = api-server
632024-10-16T04:19:12.251ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = PUT path = /static/route4 unit = api-server
642024-10-16T04:19:12.251ZINFOslog-rs: admin: listening on [::]:4676
652024-10-16T04:19:12.251ZINFOslog-rs: listening local_addr = [::]:4676 unit = api-server
662024-10-16T04:19:12.252ZDEBGslog-rs: successfully registered DTrace USDT probes local_addr = [::]:4676 unit = api-server
672024-10-16T04:19:12.321ZWARNslog-rs: failed to ensure TEP address fd06:d34c:ac1f:c1dd::1 on ASIC: Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6)
682024-10-16T04:19:13.322ZWARNslog-rs: failed to ensure TEP address fd06:d34c:ac1f:c1dd::1 on ASIC: Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6)
692024-10-16T04:19:14.323ZWARNslog-rs: failed to ensure TEP address fd06:d34c:ac1f:c1dd::1 on ASIC: Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6)
702024-10-16T04:19:15.324ZWARNslog-rs: failed to ensure TEP address fd06:d34c:ac1f:c1dd::1 on ASIC: Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6)
712024-10-16T04:19:16.324ZWARNslog-rs: failed to ensure TEP address fd06:d34c:ac1f:c1dd::1 on ASIC: Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6)
722024-10-16T04:19:17.325ZWARNslog-rs: failed to ensure TEP address fd06:d34c:ac1f:c1dd::1 on ASIC: Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6)
732024-10-16T04:19:18.326ZWARNslog-rs: failed to ensure TEP address fd06:d34c:ac1f:c1dd::1 on ASIC: Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6)
742024-10-16T04:19:19.326ZWARNslog-rs: failed to ensure TEP address fd06:d34c:ac1f:c1dd::1 on ASIC: Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6)
752024-10-16T04:19:20.327ZWARNslog-rs: failed to ensure TEP address fd06:d34c:ac1f:c1dd::1 on ASIC: Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6)
762024-10-16T04:19:21.328ZWARNslog-rs: failed to ensure TEP address fd06:d34c:ac1f:c1dd::1 on ASIC: Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6)
772024-10-16T04:19:22.328ZWARNslog-rs: failed to ensure TEP address fd06:d34c:ac1f:c1dd::1 on ASIC: Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6)
782024-10-16T04:19:23.329ZWARNslog-rs: failed to ensure TEP address fd06:d34c:ac1f:c1dd::1 on ASIC: Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6)
79 [ Oct 16 04:19:23 Rereading configuration. ]
80 [ Oct 16 04:19:23 Executing refresh method ("/opt/oxide/mgd/pkg/refresh_ctrun_child.sh -c 219"). ]
81 + CONTRACT=''
82 + getopts c: o
83 + CONTRACT=219
84 + getopts c: o
85 + [[ -n 219 ]]
86 + pgrep -c 219
87 + PARENT_PID=2311
88 + [[ -n 2311 ]]
89 + pgrep -P 2311
90 + CHILD_PID=2312
91 + [[ -n 2312 ]]
92 + kill -USR1 2312
932024-10-16T04:19:23.414ZINFOslog-rs: handling smf refresh
942024-10-16T04:19:23.415ZINFOslog-rs: starting stats server on smf refresh
95 [ Oct 16 04:19:23 Method "refresh" exited with status 0. ]
96 Oct 16 04:19:23.416 DEBG registered DTrace probes
97 Oct 16 04:19:23.416 DEBG registered endpoint, path: /{producer_id}, method: GET, local_addr: [fd00:1122:3344:101::2]:4677, component: dropshot
98 Oct 16 04:19:23.416 INFO listening, local_addr: [fd00:1122:3344:101::2]:4677, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:197
99 Oct 16 04:19:23.416 DEBG successfully registered DTrace USDT probes, local_addr: [fd00:1122:3344:101::2]:4677, component: dropshot
100 Oct 16 04:19:23.416 DEBG Nexus IP not provided, will use DNS to resolve it
101 Oct 16 04:19:23.417 INFO new DNS resolver, addresses: [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53], component: internal-dns-resolver, file: /home/build/.cargo/git/checkouts/omicron-d039c41f152bda83/48125c8/internal-dns/src/resolver.rs:72
102 Oct 16 04:19:23.417 DEBG starting producer registration task
103 Oct 16 04:19:23.417 INFO starting oximeter metric producer server, interval: 1s, address: [fd00:1122:3344:101::2]:4677, producer_id: 090745fb-27c4-4326-bc56-585f1fc622fd, file: /home/build/.cargo/git/checkouts/omicron-d039c41f152bda83/48125c8/oximeter/producer/src/lib.rs:283
104 Oct 16 04:19:23.417 DEBG registering / renewing oximeter producer lease with Nexus, component: producer-registration-task
1052024-10-16T04:19:24.274ZINFOslog-rs: accepted connection local_addr = [::]:4676 remote_addr = [fd00:1122:3344:101::1]:65069 unit = api-server
1062024-10-16T04:19:24.275ZINFOslog-rs: request completed latency_us = 373 local_addr = [::]:4676 method = PUT remote_addr = [fd00:1122:3344:101::1]:65069 req_id = ef8bfd44-da56-4aa8-ac67-e62b3dfb2d63 response_code = 204 unit = api-server uri = /static/route4
1072024-10-16T04:19:24.276ZERROslog-rs: handling change failed: libnet error route io error No such process (os error 3)
1082024-10-16T04:19:24.276ZINFOslog-rs: restarting sync loop
109 Oct 16 04:19:28.419 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("02fb2c73-9c69-4c8c-a60d-a53672f9852d.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("602cad02-64fe-45d5-8a4e-f566b096d0ef.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("8a506f8b-a3bf-4ef1-bea0-8f247d059e28.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("8a506f8b-a3bf-4ef1-bea0-8f247d059e28.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::a))) }], valid_until: Instant { tv_sec: 526, tv_nsec: 26594150 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
110 Oct 16 04:19:28.420 DEBG using nexus address for registration, addr: [fd00:1122:3344:101::b]:12221, component: producer-registration-task
111 Oct 16 04:19:28.452 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::b]:12221/metrics/producers, method: POST, component: producer-registration-task
112 Oct 16 04:19:43.453 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::b]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
113 Oct 16 04:19:43.454 WARN failed to register as a producer with Nexus, will retry, error: "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers)", delay: 350.456929ms, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-d039c41f152bda83/48125c8/oximeter/producer/src/lib.rs:424
114 Oct 16 04:19:43.836 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::b]:12221/metrics/producers, method: POST, component: producer-registration-task
115 Oct 16 04:19:58.838 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::b]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
116 Oct 16 04:19:58.839 WARN failed to register as a producer with Nexus, will retry, error: "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers)", delay: 343.275138ms, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-d039c41f152bda83/48125c8/oximeter/producer/src/lib.rs:424
117 Oct 16 04:19:59.214 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::b]:12221/metrics/producers, method: POST, component: producer-registration-task
118 Oct 16 04:20:14.214 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::b]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
119 Oct 16 04:20:14.216 WARN failed to register as a producer with Nexus, will retry, error: "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers)", delay: 664.010567ms, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-d039c41f152bda83/48125c8/oximeter/producer/src/lib.rs:424
120 Oct 16 04:20:14.913 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::b]:12221/metrics/producers, method: POST, component: producer-registration-task
121 Oct 16 04:20:29.914 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::b]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
122 Oct 16 04:20:29.915 WARN failed to register as a producer with Nexus, will retry, error: "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers)", delay: 2.447748613s, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-d039c41f152bda83/48125c8/oximeter/producer/src/lib.rs:424
123 Oct 16 04:20:32.395 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::b]:12221/metrics/producers, method: POST, component: producer-registration-task
124 Oct 16 04:20:47.397 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::b]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
125 Oct 16 04:20:47.398 WARN failed to register as a producer with Nexus, will retry, error: "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers)", delay: 3.888701737s, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-d039c41f152bda83/48125c8/oximeter/producer/src/lib.rs:424
126 Oct 16 04:20:51.319 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::b]:12221/metrics/producers, method: POST, component: producer-registration-task
127 Oct 16 04:21:06.320 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::b]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
128 Oct 16 04:21:06.322 WARN failed to register as a producer with Nexus, will retry, error: "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers)", delay: 9.887669375s, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-d039c41f152bda83/48125c8/oximeter/producer/src/lib.rs:424
129 Oct 16 04:21:16.242 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::b]:12221/metrics/producers, method: POST, component: producer-registration-task
130 Oct 16 04:21:31.242 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::b]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
131 Oct 16 04:21:31.244 WARN failed to register as a producer with Nexus, will retry, error: "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers)", delay: 9.471025579s, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-d039c41f152bda83/48125c8/oximeter/producer/src/lib.rs:424
132 Oct 16 04:21:40.747 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::b]:12221/metrics/producers, method: POST, component: producer-registration-task
133 Oct 16 04:21:55.749 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::b]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
134 Oct 16 04:21:55.750 WARN failed to register as a producer with Nexus, will retry, error: "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers)", delay: 29.123816097s, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-d039c41f152bda83/48125c8/oximeter/producer/src/lib.rs:424
135 Oct 16 04:22:24.906 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::b]:12221/metrics/producers, method: POST, component: producer-registration-task
136 Oct 16 04:22:39.908 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::b]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
137 Oct 16 04:22:39.909 WARN failed to register as a producer with Nexus, will retry, error: "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers)", delay: 89.875548419s, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-d039c41f152bda83/48125c8/oximeter/producer/src/lib.rs:424
138 Oct 16 04:24:09.816 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::b]:12221/metrics/producers, method: POST, component: producer-registration-task
139 Oct 16 04:24:24.816 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::b]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
140 Oct 16 04:24:24.817 WARN failed to register as a producer with Nexus, will retry, error: "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers)", delay: 140.361599765s, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-d039c41f152bda83/48125c8/oximeter/producer/src/lib.rs:424
141 Oct 16 04:26:45.211 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::b]:12221/metrics/producers, method: POST, component: producer-registration-task
142 Oct 16 04:27:00.212 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::b]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
143 Oct 16 04:27:00.214 WARN failed to register as a producer with Nexus, will retry, error: "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers)", delay: 95.219079363s, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-d039c41f152bda83/48125c8/oximeter/producer/src/lib.rs:424