Max level shown:
1 [ May 14 04:08:37 Disabled. ]
2 [ May 14 04:08:37 Rereading configuration. ]
3 [ May 14 04:08:51 Rereading configuration. ]
4 [ May 14 04:08:51 Rereading configuration. ]
5 [ May 14 04:08:51 Enabled. ]
6 [ May 14 04:08:51 Executing start method ("ctrun -l child -o noorphan,regent /opt/oxide/mgd/pkg/mgd_method_script.sh &"). ]
7 [ May 14 04:08:51 Method "start" exited with status 0. ]
82024-05-14T04:08:51.217ZINFOslog-rs: signal handler waiting for context
92024-05-14T04:08:51.235ZINFOslog-rs: signal handler waiting got context
102024-05-14T04:08:51.235ZINFOslog-rs: bgp routers: {}
112024-05-14T04:08:51.235ZINFOslog-rs: bfd peers: []
122024-05-14T04:08:51.239ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = GET path = /bfd/peers unit = api-server
132024-05-14T04:08:51.239ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = PUT path = /bfd/peers unit = api-server
142024-05-14T04:08:51.239ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = DELETE path = /bfd/peers/{addr} unit = api-server
152024-05-14T04:08:51.239ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = DELETE path = /bgp/config/checker unit = api-server
162024-05-14T04:08:51.239ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = GET path = /bgp/config/checker unit = api-server
172024-05-14T04:08:51.239ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = POST path = /bgp/config/checker unit = api-server
182024-05-14T04:08:51.239ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = PUT path = /bgp/config/checker unit = api-server
192024-05-14T04:08:51.239ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = DELETE path = /bgp/config/neighbor unit = api-server
202024-05-14T04:08:51.239ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = GET path = /bgp/config/neighbor unit = api-server
212024-05-14T04:08:51.239ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = POST path = /bgp/config/neighbor unit = api-server
222024-05-14T04:08:51.239ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = PUT path = /bgp/config/neighbor unit = api-server
232024-05-14T04:08:51.239ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = GET path = /bgp/config/neighbors unit = api-server
242024-05-14T04:08:51.239ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = DELETE path = /bgp/config/origin4 unit = api-server
252024-05-14T04:08:51.239ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = GET path = /bgp/config/origin4 unit = api-server
262024-05-14T04:08:51.239ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = POST path = /bgp/config/origin4 unit = api-server
272024-05-14T04:08:51.239ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = PUT path = /bgp/config/origin4 unit = api-server
282024-05-14T04:08:51.239ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = DELETE path = /bgp/config/router unit = api-server
292024-05-14T04:08:51.239ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = GET path = /bgp/config/router unit = api-server
302024-05-14T04:08:51.239ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = POST path = /bgp/config/router unit = api-server
312024-05-14T04:08:51.239ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = PUT path = /bgp/config/router unit = api-server
322024-05-14T04:08:51.239ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = GET path = /bgp/config/routers unit = api-server
332024-05-14T04:08:51.239ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = DELETE path = /bgp/config/shaper unit = api-server
342024-05-14T04:08:51.239ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = GET path = /bgp/config/shaper unit = api-server
352024-05-14T04:08:51.239ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = POST path = /bgp/config/shaper unit = api-server
362024-05-14T04:08:51.239ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = PUT path = /bgp/config/shaper unit = api-server
372024-05-14T04:08:51.239ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = GET path = /bgp/message-history unit = api-server
382024-05-14T04:08:51.239ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = POST path = /bgp/omicron/apply unit = api-server
392024-05-14T04:08:51.240ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = GET path = /bgp/status/imported unit = api-server
402024-05-14T04:08:51.240ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = GET path = /bgp/status/neighbors unit = api-server
412024-05-14T04:08:51.240ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = GET path = /bgp/status/selected unit = api-server
422024-05-14T04:08:51.240ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = DELETE path = /static/route4 unit = api-server
432024-05-14T04:08:51.240ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = GET path = /static/route4 unit = api-server
442024-05-14T04:08:51.240ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = PUT path = /static/route4 unit = api-server
452024-05-14T04:08:51.240ZINFOslog-rs: admin: listening on [::]:4676
462024-05-14T04:08:51.240ZINFOslog-rs: listening local_addr = [::]:4676 unit = api-server
472024-05-14T04:08:51.240ZDEBGslog-rs: successfully registered DTrace USDT probes local_addr = [::]:4676 unit = api-server
482024-05-14T04:08:51.309ZERROslog-rs: get originated tunnel endpoints: Communication Error: error sending request for url (http://localhost:8000/originated_tunnel_endpoints): error trying to connect: tcp connect error: Connection refused (os error 146)
49 [ May 14 04:08:52 Rereading configuration. ]
50 [ May 14 04:08:52 Executing refresh method ("/opt/oxide/mgd/pkg/refresh_ctrun_child.sh -c 433"). ]
51 + CONTRACT=''
52 + getopts c: o
53 + CONTRACT=433
54 + getopts c: o
55 + [[ -n 433 ]]
56 + pgrep -c 433
57 + PARENT_PID=4215
58 + [[ -n 4215 ]]
59 + pgrep -P 4215
60 + CHILD_PID=4216
61 + [[ -n 4216 ]]
62 + kill -USR1 4216
632024-05-14T04:08:52.683ZINFOslog-rs: handling smf refresh
642024-05-14T04:08:52.684ZINFOslog-rs: starting stats server on smf refresh
65 [ May 14 04:08:52 Method "refresh" exited with status 0. ]
662024-05-14T04:08:52.685ZINFOslog-rs: resolving nexus
672024-05-14T04:08:52.685ZINFOslog-rs: new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53]
682024-05-14T04:08:52.687ZDEBGslog-rs: lookup_socket_v6 srv dns_name = _nexus._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("4db81567-f1e3-4377-871b-92d735132e59.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("d4fac703-f5ba-4b34-aac1-7fc066e8abb3.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("ef1c9f55-f835-41e6-8d3a-81e26da276c7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("ef1c9f55-f835-41e6-8d3a-81e26da276c7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:101::c)) }], valid_until: Instant { tv_sec: 6357, tv_nsec: 203323849 } })
69 May 14 04:08:52.688 DEBG registered DTrace probes
70 May 14 04:08:52.689 DEBG registered endpoint, path: /{producer_id}, method: GET, local_addr: [fd00:1122:3344:101::2]:4677, component: dropshot
71 May 14 04:08:52.689 INFO listening, local_addr: [fd00:1122:3344:101::2]:4677, component: dropshot, file: /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/33e158b/dropshot/src/server.rs:195
72 May 14 04:08:52.689 DEBG successfully registered DTrace USDT probes, local_addr: [fd00:1122:3344:101::2]:4677, component: dropshot
73 May 14 04:08:52.689 DEBG Nexus IP provided explicitly, will use it, addr: [fd00:1122:3344:101::a]:12221
74 May 14 04:08:52.689 DEBG starting producer registration task
75 May 14 04:08:52.689 INFO starting oximeter metric producer server, interval: 1s, address: [fd00:1122:3344:101::2]:4677, producer_id: 751546b3-1c48-433c-ac6c-ece0271284ad, route: //751546b3-1c48-433c-ac6c-ece0271284ad, file: /home/build/.cargo/git/checkouts/omicron-d039c41f152bda83/ee901b0/oximeter/producer/src/lib.rs:284
76 May 14 04:08:52.689 DEBG registering / renewing oximeter producer lease with Nexus, component: producer-registration-task
77 May 14 04:08:52.689 DEBG using nexus address for registration, addr: [fd00:1122:3344:101::a]:12221, component: producer-registration-task
78 May 14 04:08:52.721 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::a]:12221/metrics/producers, method: POST, component: producer-registration-task
792024-05-14T04:08:52.865ZINFOslog-rs: accepted connection local_addr = [::]:4676 remote_addr = [fd00:1122:3344:101::1]:46472 unit = api-server
802024-05-14T04:08:52.865ZINFOslog-rs: request completed latency_us = 332 local_addr = [::]:4676 method = PUT remote_addr = [fd00:1122:3344:101::1]:46472 req_id = 9da0fcb9-dd18-4126-84a3-edfa24718219 response_code = 204 unit = api-server uri = /static/route4
812024-05-14T04:08:52.866ZERROslog-rs: handling change failed: libnet error route io error No such process (os error 3)
822024-05-14T04:08:52.866ZINFOslog-rs: restarting sync loop
83 May 14 04:09:07.722 DEBG client response, result: Err(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::a)), port: Some(12221), path: "/metrics/producers", query: None, fragment: None }, source: TimedOut }), component: producer-registration-task
84 May 14 04:09:07.723 WARN failed to register as a producer with Nexus, will retry, error: "Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers): operation timed out", delay: 133.28506ms, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-d039c41f152bda83/ee901b0/oximeter/producer/src/lib.rs:426
85 May 14 04:09:07.886 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::a]:12221/metrics/producers, method: POST, component: producer-registration-task
86 May 14 04:09:23.089 DEBG client response, result: Err(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::a)), port: Some(12221), path: "/metrics/producers", query: None, fragment: None }, source: TimedOut }), component: producer-registration-task
87 May 14 04:09:23.090 WARN failed to register as a producer with Nexus, will retry, error: "Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers): operation timed out", delay: 470.543289ms, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-d039c41f152bda83/ee901b0/oximeter/producer/src/lib.rs:426
88 May 14 04:09:23.590 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::a]:12221/metrics/producers, method: POST, component: producer-registration-task
89 May 14 04:09:38.550 DEBG client response, result: Err(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::a)), port: Some(12221), path: "/metrics/producers", query: None, fragment: None }, source: TimedOut }), component: producer-registration-task
90 May 14 04:09:38.551 WARN failed to register as a producer with Nexus, will retry, error: "Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers): operation timed out", delay: 953.159838ms, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-d039c41f152bda83/ee901b0/oximeter/producer/src/lib.rs:426
91 May 14 04:09:39.531 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::a]:12221/metrics/producers, method: POST, component: producer-registration-task
92 May 14 04:09:54.492 DEBG client response, result: Err(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::a)), port: Some(12221), path: "/metrics/producers", query: None, fragment: None }, source: TimedOut }), component: producer-registration-task
93 May 14 04:09:54.493 WARN failed to register as a producer with Nexus, will retry, error: "Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers): operation timed out", delay: 2.622687413s, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-d039c41f152bda83/ee901b0/oximeter/producer/src/lib.rs:426
94 May 14 04:09:57.139 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::a]:12221/metrics/producers, method: POST, component: producer-registration-task
95 May 14 04:10:12.099 DEBG client response, result: Err(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::a)), port: Some(12221), path: "/metrics/producers", query: None, fragment: None }, source: TimedOut }), component: producer-registration-task
96 May 14 04:10:12.100 WARN failed to register as a producer with Nexus, will retry, error: "Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers): operation timed out", delay: 4.09628502s, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-d039c41f152bda83/ee901b0/oximeter/producer/src/lib.rs:426
97 May 14 04:10:16.217 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::a]:12221/metrics/producers, method: POST, component: producer-registration-task
98 May 14 04:10:31.177 DEBG client response, result: Err(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::a)), port: Some(12221), path: "/metrics/producers", query: None, fragment: None }, source: TimedOut }), component: producer-registration-task
99 May 14 04:10:31.178 WARN failed to register as a producer with Nexus, will retry, error: "Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers): operation timed out", delay: 7.449514755s, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-d039c41f152bda83/ee901b0/oximeter/producer/src/lib.rs:426
100 May 14 04:10:38.640 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::a]:12221/metrics/producers, method: POST, component: producer-registration-task
101 May 14 04:10:53.601 DEBG client response, result: Err(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::a)), port: Some(12221), path: "/metrics/producers", query: None, fragment: None }, source: TimedOut }), component: producer-registration-task
102 May 14 04:10:53.602 WARN failed to register as a producer with Nexus, will retry, error: "Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers): operation timed out", delay: 14.849377577s, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-d039c41f152bda83/ee901b0/oximeter/producer/src/lib.rs:426
103 May 14 04:11:08.442 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::a]:12221/metrics/producers, method: POST, component: producer-registration-task
104 May 14 04:11:23.404 DEBG client response, result: Err(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::a)), port: Some(12221), path: "/metrics/producers", query: None, fragment: None }, source: TimedOut }), component: producer-registration-task
105 May 14 04:11:23.405 WARN failed to register as a producer with Nexus, will retry, error: "Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers): operation timed out", delay: 44.572470509s, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-d039c41f152bda83/ee901b0/oximeter/producer/src/lib.rs:426
106 May 14 04:12:07.983 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::a]:12221/metrics/producers, method: POST, component: producer-registration-task
107 May 14 04:12:22.988 DEBG client response, result: Err(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::a)), port: Some(12221), path: "/metrics/producers", query: None, fragment: None }, source: TimedOut }), component: producer-registration-task
108 May 14 04:12:22.994 WARN failed to register as a producer with Nexus, will retry, error: "Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers): operation timed out", delay: 50.955626919s, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-d039c41f152bda83/ee901b0/oximeter/producer/src/lib.rs:426
109 May 14 04:13:14.083 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::a]:12221/metrics/producers, method: POST, component: producer-registration-task
110 May 14 04:13:15.067 DEBG client response, result: Err(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::a)), port: Some(12221), path: "/metrics/producers", query: None, fragment: None }, source: hyper::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) }), component: producer-registration-task
111 May 14 04:13:15.069 WARN failed to register as a producer with Nexus, will retry, error: "Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers): error trying to connect: tcp connect error: Connection refused (os error 146)", delay: 78.579533994s, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-d039c41f152bda83/ee901b0/oximeter/producer/src/lib.rs:426
1122024-05-14T04:13:24.694ZINFOslog-rs: accepted connection local_addr = [::]:4676 remote_addr = [fd00:1122:3344:101::b]:46474 unit = api-server
1132024-05-14T04:13:24.702ZINFOslog-rs: request completed latency_us = 6221 local_addr = [::]:4676 method = GET remote_addr = [fd00:1122:3344:101::b]:46474 req_id = 092eec17-4156-431c-b2bf-37a5c9552b3d response_code = 200 unit = api-server uri = /bfd/peers
1142024-05-14T04:13:24.764ZINFOslog-rs: accepted connection local_addr = [::]:4676 remote_addr = [fd00:1122:3344:101::c]:46788 unit = api-server
1152024-05-14T04:13:24.764ZINFOslog-rs: request completed latency_us = 32 local_addr = [::]:4676 method = GET remote_addr = [fd00:1122:3344:101::c]:46788 req_id = 853e4651-f5af-4cff-87f2-75e4faa30f21 response_code = 200 unit = api-server uri = /bfd/peers
1162024-05-14T04:13:24.814ZINFOslog-rs: accepted connection local_addr = [::]:4676 remote_addr = [fd00:1122:3344:101::a]:44303 unit = api-server
1172024-05-14T04:13:24.815ZINFOslog-rs: request completed latency_us = 242 local_addr = [::]:4676 method = GET remote_addr = [fd00:1122:3344:101::a]:44303 req_id = c485eca6-09a7-4c9e-80b2-495f36ae3d0f response_code = 200 unit = api-server uri = /bfd/peers
1182024-05-14T04:13:25.445ZINFOslog-rs: accepted connection local_addr = [::]:4676 remote_addr = [fd00:1122:3344:101::b]:49467 unit = api-server
1192024-05-14T04:13:25.445ZINFOslog-rs: request completed latency_us = 30 local_addr = [::]:4676 method = GET remote_addr = [fd00:1122:3344:101::b]:49467 req_id = 26590bd2-d5c9-476e-b9d3-17404aa5b38c response_code = 200 unit = api-server uri = /bfd/peers
1202024-05-14T04:13:54.711ZINFOslog-rs: accepted connection local_addr = [::]:4676 remote_addr = [fd00:1122:3344:101::c]:48701 unit = api-server
1212024-05-14T04:13:54.711ZINFOslog-rs: request completed latency_us = 27 local_addr = [::]:4676 method = GET remote_addr = [fd00:1122:3344:101::c]:48701 req_id = 7e557825-5d6e-4616-984e-06b362c54ecc response_code = 200 unit = api-server uri = /bfd/peers
1222024-05-14T04:13:54.786ZINFOslog-rs: accepted connection local_addr = [::]:4676 remote_addr = [fd00:1122:3344:101::a]:52744 unit = api-server
1232024-05-14T04:13:54.786ZINFOslog-rs: request completed latency_us = 23 local_addr = [::]:4676 method = GET remote_addr = [fd00:1122:3344:101::a]:52744 req_id = 757be793-47cf-44b8-bead-4d274be40058 response_code = 200 unit = api-server uri = /bfd/peers
1242024-05-14T04:13:54.795ZINFOslog-rs: accepted connection local_addr = [::]:4676 remote_addr = [fd00:1122:3344:101::c]:57722 unit = api-server
1252024-05-14T04:13:54.796ZINFOslog-rs: request completed error_message_external = Internal Server Error error_message_internal = key must be a string latency_us = 898 local_addr = [::]:4676 method = GET remote_addr = [fd00:1122:3344:101::c]:57722 req_id = cf132308-3ef7-4e81-870a-0e67bf034b01 response_code = 500 unit = api-server uri = /static/route4
1262024-05-14T04:13:54.805ZINFOslog-rs: accepted connection local_addr = [::]:4676 remote_addr = [fd00:1122:3344:101::b]:61014 unit = api-server
1272024-05-14T04:13:54.806ZINFOslog-rs: request completed latency_us = 15 local_addr = [::]:4676 method = GET remote_addr = [fd00:1122:3344:101::b]:61014 req_id = 1b92493c-08ca-4376-9f04-26efec1c1d4e response_code = 200 unit = api-server uri = /bfd/peers
1282024-05-14T04:13:54.858ZINFOslog-rs: accepted connection local_addr = [::]:4676 remote_addr = [fd00:1122:3344:101::a]:43028 unit = api-server
1292024-05-14T04:13:54.858ZINFOslog-rs: request completed error_message_external = Internal Server Error error_message_internal = key must be a string latency_us = 26 local_addr = [::]:4676 method = GET remote_addr = [fd00:1122:3344:101::a]:43028 req_id = 15ce9f20-cf77-4d67-ac03-ccb61ebfc389 response_code = 500 unit = api-server uri = /static/route4
1302024-05-14T04:13:54.973ZINFOslog-rs: accepted connection local_addr = [::]:4676 remote_addr = [fd00:1122:3344:101::b]:42701 unit = api-server
1312024-05-14T04:13:54.973ZINFOslog-rs: request completed error_message_external = Internal Server Error error_message_internal = key must be a string latency_us = 30 local_addr = [::]:4676 method = GET remote_addr = [fd00:1122:3344:101::b]:42701 req_id = 03e491af-1269-4145-aa2b-f64dd9655087 response_code = 500 unit = api-server uri = /static/route4
1322024-05-14T04:14:24.897ZINFOslog-rs: accepted connection local_addr = [::]:4676 remote_addr = [fd00:1122:3344:101::b]:46009 unit = api-server
1332024-05-14T04:14:24.898ZINFOslog-rs: request completed latency_us = 1093 local_addr = [::]:4676 method = GET remote_addr = [fd00:1122:3344:101::b]:46009 req_id = 5674cc72-d27b-4e3c-8e6a-bf1df756f92f response_code = 200 unit = api-server uri = /bfd/peers
1342024-05-14T04:14:25.020ZINFOslog-rs: accepted connection local_addr = [::]:4676 remote_addr = [fd00:1122:3344:101::c]:63317 unit = api-server
1352024-05-14T04:14:25.020ZINFOslog-rs: request completed latency_us = 24 local_addr = [::]:4676 method = GET remote_addr = [fd00:1122:3344:101::c]:63317 req_id = d6095b8f-e031-4ea7-9bb0-1d2ec22399df response_code = 200 unit = api-server uri = /bfd/peers
1362024-05-14T04:14:25.078ZINFOslog-rs: accepted connection local_addr = [::]:4676 remote_addr = [fd00:1122:3344:101::c]:63746 unit = api-server
1372024-05-14T04:14:25.078ZINFOslog-rs: request completed error_message_external = Internal Server Error error_message_internal = key must be a string latency_us = 22 local_addr = [::]:4676 method = GET remote_addr = [fd00:1122:3344:101::c]:63746 req_id = b810c969-0462-435f-befb-d5d30fe320e8 response_code = 500 unit = api-server uri = /static/route4
1382024-05-14T04:14:25.130ZINFOslog-rs: accepted connection local_addr = [::]:4676 remote_addr = [fd00:1122:3344:101::b]:34571 unit = api-server
1392024-05-14T04:14:25.130ZINFOslog-rs: request completed error_message_external = Internal Server Error error_message_internal = key must be a string latency_us = 28 local_addr = [::]:4676 method = GET remote_addr = [fd00:1122:3344:101::b]:34571 req_id = 2f9f75d9-91c5-460c-bbaf-6cafed82af50 response_code = 500 unit = api-server uri = /static/route4
1402024-05-14T04:14:25.157ZINFOslog-rs: accepted connection local_addr = [::]:4676 remote_addr = [fd00:1122:3344:101::a]:47324 unit = api-server
1412024-05-14T04:14:25.157ZINFOslog-rs: request completed latency_us = 18 local_addr = [::]:4676 method = GET remote_addr = [fd00:1122:3344:101::a]:47324 req_id = a133f44a-d7bb-49e9-991b-eba31fdb6638 response_code = 200 unit = api-server uri = /bfd/peers
1422024-05-14T04:14:25.270ZINFOslog-rs: accepted connection local_addr = [::]:4676 remote_addr = [fd00:1122:3344:101::a]:41369 unit = api-server
1432024-05-14T04:14:25.271ZINFOslog-rs: request completed error_message_external = Internal Server Error error_message_internal = key must be a string latency_us = 920 local_addr = [::]:4676 method = GET remote_addr = [fd00:1122:3344:101::a]:41369 req_id = 0fbe8d2d-4146-410b-b2f4-5b0dfa67d769 response_code = 500 unit = api-server uri = /static/route4
144 May 14 04:14:33.682 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::a]:12221/metrics/producers, method: POST, component: producer-registration-task
145 May 14 04:14:33.743 DEBG client response, result: Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::a)), port: Some(12221), path: "/metrics/producers", query: None, fragment: None }, status: 201, headers: {"content-type": "application/json", "x-request-id": "77d61aec-5738-4aab-beee-10390e19e6e9", "content-length": "41", "date": "Tue, 14 May 2024 04:14:33 GMT"} }), component: producer-registration-task
146 May 14 04:14:33.744 DEBG registered with nexus successfully, component: producer-registration-task
147 May 14 04:14:33.744 DEBG pausing until time to renew lease, wait_period: 150s, lease_duration: 600s, component: producer-registration-task
148 May 14 04:14:34.775 INFO accepted connection, remote_addr: [fd00:1122:3344:101::d]:55842, local_addr: [fd00:1122:3344:101::2]:4677, component: dropshot, file: /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/33e158b/dropshot/src/server.rs:765
149 May 14 04:14:34.784 INFO request completed, latency_us: 8616, response_code: 200, uri: //751546b3-1c48-433c-ac6c-ece0271284ad, method: GET, req_id: 99fbe2fc-1ee5-4443-a2b7-6667ced09a95, remote_addr: [fd00:1122:3344:101::d]:55842, local_addr: [fd00:1122:3344:101::2]:4677, component: dropshot, file: /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/33e158b/dropshot/src/server.rs:849
150 May 14 04:14:35.774 INFO request completed, latency_us: 357, response_code: 200, uri: //751546b3-1c48-433c-ac6c-ece0271284ad, method: GET, req_id: e8777dfe-07ec-4720-b545-b0adfd69252a, remote_addr: [fd00:1122:3344:101::d]:55842, local_addr: [fd00:1122:3344:101::2]:4677, component: dropshot, file: /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/33e158b/dropshot/src/server.rs:849
151 May 14 04:14:36.774 INFO request completed, latency_us: 112, response_code: 200, uri: //751546b3-1c48-433c-ac6c-ece0271284ad, method: GET, req_id: 0a5a9025-4ff3-400a-8620-b7612821a230, remote_addr: [fd00:1122:3344:101::d]:55842, local_addr: [fd00:1122:3344:101::2]:4677, component: dropshot, file: /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/33e158b/dropshot/src/server.rs:849
152 May 14 04:14:37.774 INFO request completed, latency_us: 100, response_code: 200, uri: //751546b3-1c48-433c-ac6c-ece0271284ad, method: GET, req_id: 76c2843d-7c44-4f25-898d-ab5304f38626, remote_addr: [fd00:1122:3344:101::d]:55842, local_addr: [fd00:1122:3344:101::2]:4677, component: dropshot, file: /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/33e158b/dropshot/src/server.rs:849
153 May 14 04:14:38.775 INFO request completed, latency_us: 138, response_code: 200, uri: //751546b3-1c48-433c-ac6c-ece0271284ad, method: GET, req_id: 57fb34ae-dfe4-4f18-a91c-a984f646aa07, remote_addr: [fd00:1122:3344:101::d]:55842, local_addr: [fd00:1122:3344:101::2]:4677, component: dropshot, file: /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/33e158b/dropshot/src/server.rs:849
154 May 14 04:14:39.774 INFO request completed, latency_us: 100, response_code: 200, uri: //751546b3-1c48-433c-ac6c-ece0271284ad, method: GET, req_id: f66e7e51-6419-4084-ae23-df8ae3990838, remote_addr: [fd00:1122:3344:101::d]:55842, local_addr: [fd00:1122:3344:101::2]:4677, component: dropshot, file: /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/33e158b/dropshot/src/server.rs:849
155 May 14 04:14:40.774 INFO request completed, latency_us: 140, response_code: 200, uri: //751546b3-1c48-433c-ac6c-ece0271284ad, method: GET, req_id: 8054a6ea-4434-43ec-9256-7ee577b35533, remote_addr: [fd00:1122:3344:101::d]:55842, local_addr: [fd00:1122:3344:101::2]:4677, component: dropshot, file: /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/33e158b/dropshot/src/server.rs:849
156 May 14 04:14:41.774 INFO request completed, latency_us: 139, response_code: 200, uri: //751546b3-1c48-433c-ac6c-ece0271284ad, method: GET, req_id: 1de4ac0d-927c-4c6c-ba9e-a31bcbf468d5, remote_addr: [fd00:1122:3344:101::d]:55842, local_addr: [fd00:1122:3344:101::2]:4677, component: dropshot, file: /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/33e158b/dropshot/src/server.rs:849
157 May 14 04:14:42.775 INFO request completed, latency_us: 131, response_code: 200, uri: //751546b3-1c48-433c-ac6c-ece0271284ad, method: GET, req_id: 67af7bfe-2a9a-4577-9962-36423372a0e4, remote_addr: [fd00:1122:3344:101::d]:55842, local_addr: [fd00:1122:3344:101::2]:4677, component: dropshot, file: /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/33e158b/dropshot/src/server.rs:849
158 May 14 04:14:43.774 INFO request completed, latency_us: 100, response_code: 200, uri: //751546b3-1c48-433c-ac6c-ece0271284ad, method: GET, req_id: f7275ecc-eefe-44d7-871c-e2e404683749, remote_addr: [fd00:1122:3344:101::d]:55842, local_addr: [fd00:1122:3344:101::2]:4677, component: dropshot, file: /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/33e158b/dropshot/src/server.rs:849
159 May 14 04:14:44.775 INFO request completed, latency_us: 591, response_code: 200, uri: //751546b3-1c48-433c-ac6c-ece0271284ad, method: GET, req_id: eeb811c7-17d1-44ea-8ed5-87ae9b7731a7, remote_addr: [fd00:1122:3344:101::d]:55842, local_addr: [fd00:1122:3344:101::2]:4677, component: dropshot, file: /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/33e158b/dropshot/src/server.rs:849
160 May 14 04:14:45.774 INFO request completed, latency_us: 97, response_code: 200, uri: //751546b3-1c48-433c-ac6c-ece0271284ad, method: GET, req_id: 2b8259a2-a058-4568-b524-b81a9e0f0f70, remote_addr: [fd00:1122:3344:101::d]:55842, local_addr: [fd00:1122:3344:101::2]:4677, component: dropshot, file: /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/33e158b/dropshot/src/server.rs:849
161 May 14 04:14:46.775 INFO request completed, latency_us: 141, response_code: 200, uri: //751546b3-1c48-433c-ac6c-ece0271284ad, method: GET, req_id: 7f48f394-61e0-40e5-8a9d-5844b9a35b42, remote_addr: [fd00:1122:3344:101::d]:55842, local_addr: [fd00:1122:3344:101::2]:4677, component: dropshot, file: /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/33e158b/dropshot/src/server.rs:849
162 May 14 04:14:47.774 INFO request completed, latency_us: 83, response_code: 200, uri: //751546b3-1c48-433c-ac6c-ece0271284ad, method: GET, req_id: 9c677860-268e-4bac-bbad-d08677c118d0, remote_addr: [fd00:1122:3344:101::d]:55842, local_addr: [fd00:1122:3344:101::2]:4677, component: dropshot, file: /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/33e158b/dropshot/src/server.rs:849
163 May 14 04:14:48.774 INFO request completed, latency_us: 99, response_code: 200, uri: //751546b3-1c48-433c-ac6c-ece0271284ad, method: GET, req_id: 40b7e414-54b1-4317-b285-919ab8138d6f, remote_addr: [fd00:1122:3344:101::d]:55842, local_addr: [fd00:1122:3344:101::2]:4677, component: dropshot, file: /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/33e158b/dropshot/src/server.rs:849
164 May 14 04:14:49.774 INFO request completed, latency_us: 99, response_code: 200, uri: //751546b3-1c48-433c-ac6c-ece0271284ad, method: GET, req_id: dff9382c-0473-47d1-aa2f-cb68da17ec4a, remote_addr: [fd00:1122:3344:101::d]:55842, local_addr: [fd00:1122:3344:101::2]:4677, component: dropshot, file: /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/33e158b/dropshot/src/server.rs:849
165 May 14 04:14:50.774 INFO request completed, latency_us: 88, response_code: 200, uri: //751546b3-1c48-433c-ac6c-ece0271284ad, method: GET, req_id: 81377c56-f37e-4adc-9196-3784dd7e932e, remote_addr: [fd00:1122:3344:101::d]:55842, local_addr: [fd00:1122:3344:101::2]:4677, component: dropshot, file: /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/33e158b/dropshot/src/server.rs:849
166 May 14 04:14:51.774 INFO request completed, latency_us: 109, response_code: 200, uri: //751546b3-1c48-433c-ac6c-ece0271284ad, method: GET, req_id: 5d18c993-6827-474e-bf50-d82c5dcdff8d, remote_addr: [fd00:1122:3344:101::d]:55842, local_addr: [fd00:1122:3344:101::2]:4677, component: dropshot, file: /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/33e158b/dropshot/src/server.rs:849
167 May 14 04:14:52.775 INFO request completed, latency_us: 784, response_code: 200, uri: //751546b3-1c48-433c-ac6c-ece0271284ad, method: GET, req_id: 07e0274c-0f12-4307-ba3d-c667fb16f113, remote_addr: [fd00:1122:3344:101::d]:55842, local_addr: [fd00:1122:3344:101::2]:4677, component: dropshot, file: /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/33e158b/dropshot/src/server.rs:849
168 May 14 04:14:53.774 INFO request completed, latency_us: 90, response_code: 200, uri: //751546b3-1c48-433c-ac6c-ece0271284ad, method: GET, req_id: 0024d5af-27a4-4edd-a8c3-d8427553d506, remote_addr: [fd00:1122:3344:101::d]:55842, local_addr: [fd00:1122:3344:101::2]:4677, component: dropshot, file: /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/33e158b/dropshot/src/server.rs:849
1692024-05-14T04:14:54.745ZINFOslog-rs: accepted connection local_addr = [::]:4676 remote_addr = [fd00:1122:3344:101::b]:32838 unit = api-server
1702024-05-14T04:14:54.745ZINFOslog-rs: request completed latency_us = 31 local_addr = [::]:4676 method = GET remote_addr = [fd00:1122:3344:101::b]:32838 req_id = 9ac738b3-32b2-4d33-8b0c-226d84c2a292 response_code = 200 unit = api-server uri = /bfd/peers
1712024-05-14T04:14:54.754ZINFOslog-rs: accepted connection local_addr = [::]:4676 remote_addr = [fd00:1122:3344:101::c]:35583 unit = api-server
1722024-05-14T04:14:54.754ZINFOslog-rs: request completed latency_us = 20 local_addr = [::]:4676 method = GET remote_addr = [fd00:1122:3344:101::c]:35583 req_id = a779f5d2-aaf6-484a-bde9-dc4f800179ae response_code = 200 unit = api-server uri = /bfd/peers
173 May 14 04:14:54.775 INFO request completed, latency_us: 79, response_code: 200, uri: //751546b3-1c48-433c-ac6c-ece0271284ad, method: GET, req_id: 65659b21-ac88-4766-b4db-10fb13cbefae, remote_addr: [fd00:1122:3344:101::d]:55842, local_addr: [fd00:1122:3344:101::2]:4677, component: dropshot, file: /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/33e158b/dropshot/src/server.rs:849
1742024-05-14T04:14:54.790ZINFOslog-rs: accepted connection local_addr = [::]:4676 remote_addr = [fd00:1122:3344:101::b]:33448 unit = api-server
1752024-05-14T04:14:54.790ZINFOslog-rs: request completed error_message_external = Internal Server Error error_message_internal = key must be a string latency_us = 23 local_addr = [::]:4676 method = GET remote_addr = [fd00:1122:3344:101::b]:33448 req_id = 45e95022-600f-4953-a8e4-3633b3920384 response_code = 500 unit = api-server uri = /static/route4
1762024-05-14T04:14:54.838ZINFOslog-rs: accepted connection local_addr = [::]:4676 remote_addr = [fd00:1122:3344:101::c]:33862 unit = api-server
1772024-05-14T04:14:54.838ZINFOslog-rs: request completed error_message_external = Internal Server Error error_message_internal = key must be a string latency_us = 29 local_addr = [::]:4676 method = GET remote_addr = [fd00:1122:3344:101::c]:33862 req_id = 199dfb63-28b5-4299-85e3-27c1cf42ef88 response_code = 500 unit = api-server uri = /static/route4
1782024-05-14T04:14:54.851ZINFOslog-rs: accepted connection local_addr = [::]:4676 remote_addr = [fd00:1122:3344:101::a]:39674 unit = api-server
1792024-05-14T04:14:54.851ZINFOslog-rs: request completed latency_us = 21 local_addr = [::]:4676 method = GET remote_addr = [fd00:1122:3344:101::a]:39674 req_id = b5d283e4-abbe-481f-9b4b-8d969c83d838 response_code = 200 unit = api-server uri = /bfd/peers
1802024-05-14T04:14:55.007ZINFOslog-rs: accepted connection local_addr = [::]:4676 remote_addr = [fd00:1122:3344:101::a]:40184 unit = api-server
1812024-05-14T04:14:55.007ZINFOslog-rs: request completed error_message_external = Internal Server Error error_message_internal = key must be a string latency_us = 29 local_addr = [::]:4676 method = GET remote_addr = [fd00:1122:3344:101::a]:40184 req_id = d9baef4d-968b-4ad2-a7f5-b230e9df7d04 response_code = 500 unit = api-server uri = /static/route4
182 May 14 04:14:55.775 INFO request completed, latency_us: 721, response_code: 200, uri: //751546b3-1c48-433c-ac6c-ece0271284ad, method: GET, req_id: dc1f89d9-af4f-4427-bea0-f2be9d208c0c, remote_addr: [fd00:1122:3344:101::d]:55842, local_addr: [fd00:1122:3344:101::2]:4677, component: dropshot, file: /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/33e158b/dropshot/src/server.rs:849
183 May 14 04:14:56.775 INFO request completed, latency_us: 594, response_code: 200, uri: //751546b3-1c48-433c-ac6c-ece0271284ad, method: GET, req_id: dec0845e-a5e3-4db2-a46c-62c44b0f9b8f, remote_addr: [fd00:1122:3344:101::d]:55842, local_addr: [fd00:1122:3344:101::2]:4677, component: dropshot, file: /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/33e158b/dropshot/src/server.rs:849
184 May 14 04:14:57.774 INFO request completed, latency_us: 123, response_code: 200, uri: //751546b3-1c48-433c-ac6c-ece0271284ad, method: GET, req_id: 7e164ea3-566c-41ac-adc7-e0353e119378, remote_addr: [fd00:1122:3344:101::d]:55842, local_addr: [fd00:1122:3344:101::2]:4677, component: dropshot, file: /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/33e158b/dropshot/src/server.rs:849
185 May 14 04:14:58.774 INFO request completed, latency_us: 96, response_code: 200, uri: //751546b3-1c48-433c-ac6c-ece0271284ad, method: GET, req_id: fa863ade-52d0-43d8-aec4-ed3264500217, remote_addr: [fd00:1122:3344:101::d]:55842, local_addr: [fd00:1122:3344:101::2]:4677, component: dropshot, file: /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/33e158b/dropshot/src/server.rs:849
186 May 14 04:14:59.775 INFO request completed, latency_us: 800, response_code: 200, uri: //751546b3-1c48-433c-ac6c-ece0271284ad, method: GET, req_id: b27464c0-051b-43a4-86c3-8eebb64741c0, remote_addr: [fd00:1122:3344:101::d]:55842, local_addr: [fd00:1122:3344:101::2]:4677, component: dropshot, file: /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/33e158b/dropshot/src/server.rs:849