Max level shown:
1 [ May 30 04:24:47 Disabled. ]
2 [ May 30 04:24:47 Rereading configuration. ]
3 [ May 30 04:25:00 Rereading configuration. ]
4 [ May 30 04:25:00 Rereading configuration. ]
5 [ May 30 04:25:00 Enabled. ]
6 [ May 30 04:25:00 Executing start method ("ctrun -l child -o noorphan,regent /opt/oxide/mgd/pkg/mgd_method_script.sh &"). ]
7 [ May 30 04:25:00 Method "start" exited with status 0. ]
82024-05-30T04:25:00.382ZINFOslog-rs: signal handler waiting for context
92024-05-30T04:25:00.406ZINFOslog-rs: signal handler waiting got context
102024-05-30T04:25:00.406ZINFOslog-rs: bgp routers: {}
112024-05-30T04:25:00.406ZINFOslog-rs: bfd peers: []
122024-05-30T04:25:00.410ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = GET path = /bfd/peers unit = api-server
132024-05-30T04:25:00.410ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = PUT path = /bfd/peers unit = api-server
142024-05-30T04:25:00.410ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = DELETE path = /bfd/peers/{addr} unit = api-server
152024-05-30T04:25:00.410ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = DELETE path = /bgp/config/checker unit = api-server
162024-05-30T04:25:00.410ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = GET path = /bgp/config/checker unit = api-server
172024-05-30T04:25:00.410ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = POST path = /bgp/config/checker unit = api-server
182024-05-30T04:25:00.410ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = PUT path = /bgp/config/checker unit = api-server
192024-05-30T04:25:00.410ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = DELETE path = /bgp/config/neighbor unit = api-server
202024-05-30T04:25:00.410ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = GET path = /bgp/config/neighbor unit = api-server
212024-05-30T04:25:00.410ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = POST path = /bgp/config/neighbor unit = api-server
222024-05-30T04:25:00.410ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = PUT path = /bgp/config/neighbor unit = api-server
232024-05-30T04:25:00.410ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = GET path = /bgp/config/neighbors unit = api-server
242024-05-30T04:25:00.410ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = DELETE path = /bgp/config/origin4 unit = api-server
252024-05-30T04:25:00.410ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = GET path = /bgp/config/origin4 unit = api-server
262024-05-30T04:25:00.410ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = POST path = /bgp/config/origin4 unit = api-server
272024-05-30T04:25:00.410ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = PUT path = /bgp/config/origin4 unit = api-server
282024-05-30T04:25:00.410ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = DELETE path = /bgp/config/router unit = api-server
292024-05-30T04:25:00.410ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = GET path = /bgp/config/router unit = api-server
302024-05-30T04:25:00.410ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = POST path = /bgp/config/router unit = api-server
312024-05-30T04:25:00.410ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = PUT path = /bgp/config/router unit = api-server
322024-05-30T04:25:00.410ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = GET path = /bgp/config/routers unit = api-server
332024-05-30T04:25:00.410ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = DELETE path = /bgp/config/shaper unit = api-server
342024-05-30T04:25:00.410ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = GET path = /bgp/config/shaper unit = api-server
352024-05-30T04:25:00.410ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = POST path = /bgp/config/shaper unit = api-server
362024-05-30T04:25:00.410ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = PUT path = /bgp/config/shaper unit = api-server
372024-05-30T04:25:00.410ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = GET path = /bgp/message-history unit = api-server
382024-05-30T04:25:00.410ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = POST path = /bgp/omicron/apply unit = api-server
392024-05-30T04:25:00.410ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = GET path = /bgp/status/imported unit = api-server
402024-05-30T04:25:00.410ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = GET path = /bgp/status/neighbors unit = api-server
412024-05-30T04:25:00.410ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = GET path = /bgp/status/selected unit = api-server
422024-05-30T04:25:00.410ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = DELETE path = /static/route4 unit = api-server
432024-05-30T04:25:00.410ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = GET path = /static/route4 unit = api-server
442024-05-30T04:25:00.410ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = PUT path = /static/route4 unit = api-server
452024-05-30T04:25:00.410ZINFOslog-rs: admin: listening on [::]:4676
462024-05-30T04:25:00.410ZINFOslog-rs: listening local_addr = [::]:4676 unit = api-server
472024-05-30T04:25:00.411ZDEBGslog-rs: successfully registered DTrace USDT probes local_addr = [::]:4676 unit = api-server
482024-05-30T04:25:00.480ZERROslog-rs: get originated tunnel endpoints: Communication Error: error sending request for url (http://localhost:8000/originated_tunnel_endpoints): error trying to connect: tcp connect error: Connection refused (os error 146)
49 [ May 30 04:25:01 Rereading configuration. ]
50 [ May 30 04:25:01 Executing refresh method ("/opt/oxide/mgd/pkg/refresh_ctrun_child.sh -c 434"). ]
51 + CONTRACT=''
52 + getopts c: o
53 + CONTRACT=434
54 + getopts c: o
55 + [[ -n 434 ]]
56 + pgrep -c 434
57 + PARENT_PID=4201
58 + [[ -n 4201 ]]
59 + pgrep -P 4201
60 + CHILD_PID=4202
61 + [[ -n 4202 ]]
62 + kill -USR1 4202
632024-05-30T04:25:01.839ZINFOslog-rs: handling smf refresh
64 [ May 30 04:25:01 Method "refresh" exited with status 0. ]
652024-05-30T04:25:01.840ZINFOslog-rs: starting stats server on smf refresh
662024-05-30T04:25:01.841ZINFOslog-rs: resolving nexus
672024-05-30T04:25:01.841ZINFOslog-rs: new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53]
682024-05-30T04:25:02.089ZINFOslog-rs: accepted connection local_addr = [::]:4676 remote_addr = [fd00:1122:3344:101::1]:51755 unit = api-server
692024-05-30T04:25:02.089ZINFOslog-rs: request completed latency_us = 338 local_addr = [::]:4676 method = PUT remote_addr = [fd00:1122:3344:101::1]:51755 req_id = b6ce9199-72e0-4179-a210-ff8f91c2280e response_code = 204 unit = api-server uri = /static/route4
702024-05-30T04:25:02.090ZERROslog-rs: handling change failed: libnet error route io error No such process (os error 3)
712024-05-30T04:25:02.090ZINFOslog-rs: restarting sync loop
722024-05-30T04:25:06.844ZWARNslog-rs: resolving nexus failed, no record found for Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } waiting 51.59624ms
732024-05-30T04:25:06.898ZDEBGslog-rs: lookup_socket_v6 srv dns_name = _nexus._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("31c86263-9349-4603-aaa0-6dc44827df57.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("7c996f98-0c86-4480-8700-580fc6f3590a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("b5bf7e90-33d3-4f67-b9d3-793143654e9a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("b5bf7e90-33d3-4f67-b9d3-793143654e9a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:101::b)) }], valid_until: Instant { tv_sec: 729, tv_nsec: 26505842 } })
74 May 30 04:25:06.900 DEBG registered DTrace probes
75 May 30 04:25:06.900 DEBG registered endpoint, path: /{producer_id}, method: GET, local_addr: [fd00:1122:3344:101::2]:4677, component: dropshot
76 May 30 04:25:06.900 INFO listening, local_addr: [fd00:1122:3344:101::2]:4677, component: dropshot, file: /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/33e158b/dropshot/src/server.rs:195
77 May 30 04:25:06.900 DEBG successfully registered DTrace USDT probes, local_addr: [fd00:1122:3344:101::2]:4677, component: dropshot
78 May 30 04:25:06.900 DEBG Nexus IP provided explicitly, will use it, addr: [fd00:1122:3344:101::c]:12221
79 May 30 04:25:06.900 DEBG starting producer registration task
80 May 30 04:25:06.900 INFO starting oximeter metric producer server, interval: 1s, address: [fd00:1122:3344:101::2]:4677, producer_id: 0cf65393-f6a2-480b-8f0b-78c9543ba992, route: //0cf65393-f6a2-480b-8f0b-78c9543ba992, file: /home/build/.cargo/git/checkouts/omicron-d039c41f152bda83/ee901b0/oximeter/producer/src/lib.rs:284
81 May 30 04:25:06.900 DEBG registering / renewing oximeter producer lease with Nexus, component: producer-registration-task
82 May 30 04:25:06.900 DEBG using nexus address for registration, addr: [fd00:1122:3344:101::c]:12221, component: producer-registration-task
83 May 30 04:25:06.932 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::c]:12221/metrics/producers, method: POST, component: producer-registration-task
84 May 30 04:25:21.941 DEBG client response, result: Err(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::c)), port: Some(12221), path: "/metrics/producers", query: None, fragment: None }, source: TimedOut }), component: producer-registration-task
85 May 30 04:25:21.942 WARN failed to register as a producer with Nexus, will retry, error: "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out", delay: 345.949048ms, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-d039c41f152bda83/ee901b0/oximeter/producer/src/lib.rs:426
86 May 30 04:25:22.319 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::c]:12221/metrics/producers, method: POST, component: producer-registration-task
87 May 30 04:25:37.324 DEBG client response, result: Err(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::c)), port: Some(12221), path: "/metrics/producers", query: None, fragment: None }, source: TimedOut }), component: producer-registration-task
88 May 30 04:25:37.325 WARN failed to register as a producer with Nexus, will retry, error: "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out", delay: 725.715649ms, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-d039c41f152bda83/ee901b0/oximeter/producer/src/lib.rs:426
89 May 30 04:25:38.082 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::c]:12221/metrics/producers, method: POST, component: producer-registration-task
90 May 30 04:25:53.082 DEBG client response, result: Err(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::c)), port: Some(12221), path: "/metrics/producers", query: None, fragment: None }, source: TimedOut }), component: producer-registration-task
91 May 30 04:25:53.083 WARN failed to register as a producer with Nexus, will retry, error: "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out", delay: 1.019188727s, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-d039c41f152bda83/ee901b0/oximeter/producer/src/lib.rs:426
92 May 30 04:25:54.136 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::c]:12221/metrics/producers, method: POST, component: producer-registration-task
93 May 30 04:26:09.137 DEBG client response, result: Err(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::c)), port: Some(12221), path: "/metrics/producers", query: None, fragment: None }, source: TimedOut }), component: producer-registration-task
94 May 30 04:26:09.138 WARN failed to register as a producer with Nexus, will retry, error: "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out", delay: 1.099670067s, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-d039c41f152bda83/ee901b0/oximeter/producer/src/lib.rs:426
95 May 30 04:26:10.271 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::c]:12221/metrics/producers, method: POST, component: producer-registration-task
96 May 30 04:26:25.272 DEBG client response, result: Err(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::c)), port: Some(12221), path: "/metrics/producers", query: None, fragment: None }, source: TimedOut }), component: producer-registration-task
97 May 30 04:26:25.273 WARN failed to register as a producer with Nexus, will retry, error: "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out", delay: 2.192021758s, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-d039c41f152bda83/ee901b0/oximeter/producer/src/lib.rs:426
98 May 30 04:26:27.514 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::c]:12221/metrics/producers, method: POST, component: producer-registration-task
99 May 30 04:26:42.513 DEBG client response, result: Err(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::c)), port: Some(12221), path: "/metrics/producers", query: None, fragment: None }, source: TimedOut }), component: producer-registration-task
100 May 30 04:26:42.514 WARN failed to register as a producer with Nexus, will retry, error: "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out", delay: 6.478234609s, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-d039c41f152bda83/ee901b0/oximeter/producer/src/lib.rs:426
101 May 30 04:26:49.023 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::c]:12221/metrics/producers, method: POST, component: producer-registration-task
102 May 30 04:27:04.026 DEBG client response, result: Err(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::c)), port: Some(12221), path: "/metrics/producers", query: None, fragment: None }, source: TimedOut }), component: producer-registration-task
103 May 30 04:27:04.029 WARN failed to register as a producer with Nexus, will retry, error: "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out", delay: 22.362795772s, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-d039c41f152bda83/ee901b0/oximeter/producer/src/lib.rs:426
104 May 30 04:27:28.073 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::c]:12221/metrics/producers, method: POST, component: producer-registration-task
105 May 30 04:27:43.079 DEBG client response, result: Err(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::c)), port: Some(12221), path: "/metrics/producers", query: None, fragment: None }, source: TimedOut }), component: producer-registration-task
106 May 30 04:27:43.084 WARN failed to register as a producer with Nexus, will retry, error: "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out", delay: 19.677580781s, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-d039c41f152bda83/ee901b0/oximeter/producer/src/lib.rs:426
1072024-05-30T04:28:02.340ZINFOslog-rs: accepted connection local_addr = [::]:4676 remote_addr = [fd00:1122:3344:101::b]:54293 unit = api-server
1082024-05-30T04:28:02.344ZINFOslog-rs: request completed latency_us = 4791 local_addr = [::]:4676 method = GET remote_addr = [fd00:1122:3344:101::b]:54293 req_id = 786471c8-67bd-47a8-a3cd-540aca30034f response_code = 200 unit = api-server uri = /bfd/peers
1092024-05-30T04:28:02.345ZINFOslog-rs: accepted connection local_addr = [::]:4676 remote_addr = [fd00:1122:3344:101::a]:35178 unit = api-server
1102024-05-30T04:28:02.345ZINFOslog-rs: request completed latency_us = 15 local_addr = [::]:4676 method = GET remote_addr = [fd00:1122:3344:101::a]:35178 req_id = 67283c10-39a5-4678-bc48-bd96ac007b72 response_code = 200 unit = api-server uri = /bfd/peers
1112024-05-30T04:28:02.456ZINFOslog-rs: accepted connection local_addr = [::]:4676 remote_addr = [fd00:1122:3344:101::c]:37081 unit = api-server
1122024-05-30T04:28:02.456ZINFOslog-rs: request completed latency_us = 29 local_addr = [::]:4676 method = GET remote_addr = [fd00:1122:3344:101::c]:37081 req_id = 18a1f349-01e1-4ef6-8fac-06a33105e039 response_code = 200 unit = api-server uri = /bfd/peers
113 May 30 04:28:02.795 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::c]:12221/metrics/producers, method: POST, component: producer-registration-task
114 May 30 04:28:02.818 DEBG client response, result: Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::c)), port: Some(12221), path: "/metrics/producers", query: None, fragment: None }, status: 503, headers: {"content-type": "application/json", "x-request-id": "150c2c05-a523-4a74-8b32-bd8831736824", "content-length": "133", "date": "Thu, 30 May 2024 04:28:02 GMT"} }), component: producer-registration-task
115 May 30 04:28:02.819 WARN failed to register as a producer with Nexus, will retry, error: "Error Response: status: 503 Service Unavailable; headers: {\"content-type\": \"application/json\", \"x-request-id\": \"150c2c05-a523-4a74-8b32-bd8831736824\", \"content-length\": \"133\", \"date\": \"Thu, 30 May 2024 04:28:02 GMT\"}; value: Error { error_code: Some(\"ServiceNotAvailable\"), message: \"Service Unavailable\", request_id: \"150c2c05-a523-4a74-8b32-bd8831736824\" }", delay: 95.975429456s, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-d039c41f152bda83/ee901b0/oximeter/producer/src/lib.rs:426
1162024-05-30T04:28:03.112ZINFOslog-rs: accepted connection local_addr = [::]:4676 remote_addr = [fd00:1122:3344:101::a]:40378 unit = api-server
1172024-05-30T04:28:03.112ZINFOslog-rs: request completed latency_us = 32 local_addr = [::]:4676 method = GET remote_addr = [fd00:1122:3344:101::a]:40378 req_id = 0bd4e9ca-7d4c-42d3-88c6-549a77c3ad4a response_code = 200 unit = api-server uri = /bfd/peers
1182024-05-30T04:28:32.331ZINFOslog-rs: accepted connection local_addr = [::]:4676 remote_addr = [fd00:1122:3344:101::b]:39225 unit = api-server
1192024-05-30T04:28:32.331ZINFOslog-rs: request completed latency_us = 31 local_addr = [::]:4676 method = GET remote_addr = [fd00:1122:3344:101::b]:39225 req_id = 8d817ef7-39f1-423d-bc9a-cf27dbc0dc75 response_code = 200 unit = api-server uri = /bfd/peers
1202024-05-30T04:28:32.337ZINFOslog-rs: accepted connection local_addr = [::]:4676 remote_addr = [fd00:1122:3344:101::c]:44283 unit = api-server
1212024-05-30T04:28:32.337ZINFOslog-rs: request completed latency_us = 16 local_addr = [::]:4676 method = GET remote_addr = [fd00:1122:3344:101::c]:44283 req_id = dc7790bc-5236-4129-83a7-4b738875a4ed response_code = 200 unit = api-server uri = /bfd/peers
1222024-05-30T04:28:32.417ZINFOslog-rs: accepted connection local_addr = [::]:4676 remote_addr = [fd00:1122:3344:101::a]:39516 unit = api-server
1232024-05-30T04:28:32.417ZINFOslog-rs: request completed latency_us = 26 local_addr = [::]:4676 method = GET remote_addr = [fd00:1122:3344:101::a]:39516 req_id = 1b93365c-da22-4997-8c01-bfa1dcba96b1 response_code = 200 unit = api-server uri = /bfd/peers
1242024-05-30T04:28:32.481ZINFOslog-rs: accepted connection local_addr = [::]:4676 remote_addr = [fd00:1122:3344:101::b]:45091 unit = api-server
1252024-05-30T04:28:32.482ZINFOslog-rs: request completed error_message_external = Internal Server Error error_message_internal = key must be a string latency_us = 1434 local_addr = [::]:4676 method = GET remote_addr = [fd00:1122:3344:101::b]:45091 req_id = 8d5a564a-effb-4b1a-9797-a716ecf63333 response_code = 500 unit = api-server uri = /static/route4
1262024-05-30T04:28:32.500ZINFOslog-rs: accepted connection local_addr = [::]:4676 remote_addr = [fd00:1122:3344:101::a]:37866 unit = api-server
1272024-05-30T04:28:32.500ZINFOslog-rs: request completed error_message_external = Internal Server Error error_message_internal = key must be a string latency_us = 21 local_addr = [::]:4676 method = GET remote_addr = [fd00:1122:3344:101::a]:37866 req_id = 02016b08-faf3-4a5d-9ebb-5fbfe70a56bd response_code = 500 unit = api-server uri = /static/route4
1282024-05-30T04:28:32.508ZINFOslog-rs: accepted connection local_addr = [::]:4676 remote_addr = [fd00:1122:3344:101::c]:35408 unit = api-server
1292024-05-30T04:28:32.508ZINFOslog-rs: request completed error_message_external = Internal Server Error error_message_internal = key must be a string latency_us = 23 local_addr = [::]:4676 method = GET remote_addr = [fd00:1122:3344:101::c]:35408 req_id = 9c8ac4f7-8ff3-414f-b6c3-57e959e8b883 response_code = 500 unit = api-server uri = /static/route4
1302024-05-30T04:29:02.522ZINFOslog-rs: accepted connection local_addr = [::]:4676 remote_addr = [fd00:1122:3344:101::b]:59159 unit = api-server
1312024-05-30T04:29:02.522ZINFOslog-rs: request completed latency_us = 36 local_addr = [::]:4676 method = GET remote_addr = [fd00:1122:3344:101::b]:59159 req_id = 65ae0b50-e088-445a-8afa-a2902a8ea968 response_code = 200 unit = api-server uri = /bfd/peers
1322024-05-30T04:29:02.607ZINFOslog-rs: accepted connection local_addr = [::]:4676 remote_addr = [fd00:1122:3344:101::c]:40876 unit = api-server
1332024-05-30T04:29:02.607ZINFOslog-rs: request completed latency_us = 23 local_addr = [::]:4676 method = GET remote_addr = [fd00:1122:3344:101::c]:40876 req_id = c48bf439-6584-4d41-a59b-c87b529833a9 response_code = 200 unit = api-server uri = /bfd/peers
1342024-05-30T04:29:02.636ZINFOslog-rs: accepted connection local_addr = [::]:4676 remote_addr = [fd00:1122:3344:101::a]:63339 unit = api-server
1352024-05-30T04:29:02.637ZINFOslog-rs: request completed latency_us = 19 local_addr = [::]:4676 method = GET remote_addr = [fd00:1122:3344:101::a]:63339 req_id = 3240bc6b-fc03-441e-8c5a-d271fdd3f582 response_code = 200 unit = api-server uri = /bfd/peers
1362024-05-30T04:29:02.716ZINFOslog-rs: accepted connection local_addr = [::]:4676 remote_addr = [fd00:1122:3344:101::b]:55631 unit = api-server
1372024-05-30T04:29:02.717ZINFOslog-rs: request completed error_message_external = Internal Server Error error_message_internal = key must be a string latency_us = 23 local_addr = [::]:4676 method = GET remote_addr = [fd00:1122:3344:101::b]:55631 req_id = ebdaa0ab-0092-43bd-8b28-6490ef928967 response_code = 500 unit = api-server uri = /static/route4
1382024-05-30T04:29:02.724ZINFOslog-rs: accepted connection local_addr = [::]:4676 remote_addr = [fd00:1122:3344:101::c]:44298 unit = api-server
1392024-05-30T04:29:02.724ZINFOslog-rs: request completed error_message_external = Internal Server Error error_message_internal = key must be a string latency_us = 16 local_addr = [::]:4676 method = GET remote_addr = [fd00:1122:3344:101::c]:44298 req_id = 7c65e161-6b59-478c-965b-fde082e7293b response_code = 500 unit = api-server uri = /static/route4
1402024-05-30T04:29:02.913ZINFOslog-rs: accepted connection local_addr = [::]:4676 remote_addr = [fd00:1122:3344:101::a]:64372 unit = api-server
1412024-05-30T04:29:02.914ZINFOslog-rs: request completed error_message_external = Internal Server Error error_message_internal = key must be a string latency_us = 688 local_addr = [::]:4676 method = GET remote_addr = [fd00:1122:3344:101::a]:64372 req_id = 10687ef8-e6b4-4d19-9273-2d8a975ae55d response_code = 500 unit = api-server uri = /static/route4
1422024-05-30T04:29:32.357ZINFOslog-rs: accepted connection local_addr = [::]:4676 remote_addr = [fd00:1122:3344:101::a]:40283 unit = api-server
1432024-05-30T04:29:32.357ZINFOslog-rs: request completed latency_us = 33 local_addr = [::]:4676 method = GET remote_addr = [fd00:1122:3344:101::a]:40283 req_id = 48eb146f-ee18-461c-aab2-8b706d12e0c6 response_code = 200 unit = api-server uri = /bfd/peers
1442024-05-30T04:29:32.401ZINFOslog-rs: accepted connection local_addr = [::]:4676 remote_addr = [fd00:1122:3344:101::c]:59856 unit = api-server
1452024-05-30T04:29:32.401ZINFOslog-rs: request completed latency_us = 20 local_addr = [::]:4676 method = GET remote_addr = [fd00:1122:3344:101::c]:59856 req_id = 1f564d2b-3eca-401a-8480-eaae3086c455 response_code = 200 unit = api-server uri = /bfd/peers
1462024-05-30T04:29:32.409ZINFOslog-rs: accepted connection local_addr = [::]:4676 remote_addr = [fd00:1122:3344:101::b]:50077 unit = api-server
1472024-05-30T04:29:32.409ZINFOslog-rs: request completed latency_us = 17 local_addr = [::]:4676 method = GET remote_addr = [fd00:1122:3344:101::b]:50077 req_id = 13e72cb2-b5db-4bbb-8e4c-d45254980a91 response_code = 200 unit = api-server uri = /bfd/peers
1482024-05-30T04:29:32.448ZINFOslog-rs: accepted connection local_addr = [::]:4676 remote_addr = [fd00:1122:3344:101::a]:55325 unit = api-server
1492024-05-30T04:29:32.448ZINFOslog-rs: request completed error_message_external = Internal Server Error error_message_internal = key must be a string latency_us = 19 local_addr = [::]:4676 method = GET remote_addr = [fd00:1122:3344:101::a]:55325 req_id = f9f38c4d-fe5c-4a48-84b1-fc8afdfeb066 response_code = 500 unit = api-server uri = /static/route4
1502024-05-30T04:29:32.471ZINFOslog-rs: accepted connection local_addr = [::]:4676 remote_addr = [fd00:1122:3344:101::c]:34583 unit = api-server
1512024-05-30T04:29:32.471ZINFOslog-rs: request completed error_message_external = Internal Server Error error_message_internal = key must be a string latency_us = 20 local_addr = [::]:4676 method = GET remote_addr = [fd00:1122:3344:101::c]:34583 req_id = 734c3d90-1cc6-4905-8723-9ee6c65505e4 response_code = 500 unit = api-server uri = /static/route4
1522024-05-30T04:29:32.478ZINFOslog-rs: accepted connection local_addr = [::]:4676 remote_addr = [fd00:1122:3344:101::b]:51958 unit = api-server
1532024-05-30T04:29:32.478ZINFOslog-rs: request completed error_message_external = Internal Server Error error_message_internal = key must be a string latency_us = 18 local_addr = [::]:4676 method = GET remote_addr = [fd00:1122:3344:101::b]:51958 req_id = 28b1792b-dfbb-4fcd-a6e4-e54738efed34 response_code = 500 unit = api-server uri = /static/route4
154 May 30 04:29:38.829 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::c]:12221/metrics/producers, method: POST, component: producer-registration-task
155 May 30 04:29:38.892 DEBG client response, result: Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::c)), port: Some(12221), path: "/metrics/producers", query: None, fragment: None }, status: 201, headers: {"content-type": "application/json", "x-request-id": "c547b960-812a-43f0-9418-7d583528a9f0", "content-length": "41", "date": "Thu, 30 May 2024 04:29:38 GMT"} }), component: producer-registration-task
156 May 30 04:29:38.893 DEBG registered with nexus successfully, component: producer-registration-task
157 May 30 04:29:38.893 DEBG pausing until time to renew lease, wait_period: 150s, lease_duration: 600s, component: producer-registration-task
158 May 30 04:29:39.922 INFO accepted connection, remote_addr: [fd00:1122:3344:101::d]:48781, local_addr: [fd00:1122:3344:101::2]:4677, component: dropshot, file: /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/33e158b/dropshot/src/server.rs:765
159 May 30 04:29:39.931 INFO request completed, latency_us: 8452, response_code: 200, uri: /0cf65393-f6a2-480b-8f0b-78c9543ba992, method: GET, req_id: 9dcf4297-bb56-4e5c-924e-41d2fbc7de00, remote_addr: [fd00:1122:3344:101::d]:48781, local_addr: [fd00:1122:3344:101::2]:4677, component: dropshot, file: /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/33e158b/dropshot/src/server.rs:849
160 May 30 04:29:40.922 INFO request completed, latency_us: 88, response_code: 200, uri: /0cf65393-f6a2-480b-8f0b-78c9543ba992, method: GET, req_id: ceae4fcb-4f0d-44ed-af5d-78d743802c04, remote_addr: [fd00:1122:3344:101::d]:48781, local_addr: [fd00:1122:3344:101::2]:4677, component: dropshot, file: /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/33e158b/dropshot/src/server.rs:849
161 May 30 04:29:41.922 INFO request completed, latency_us: 261, response_code: 200, uri: /0cf65393-f6a2-480b-8f0b-78c9543ba992, method: GET, req_id: 180dbeff-3020-4b99-a0f2-f5a249590356, remote_addr: [fd00:1122:3344:101::d]:48781, local_addr: [fd00:1122:3344:101::2]:4677, component: dropshot, file: /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/33e158b/dropshot/src/server.rs:849
162 May 30 04:29:42.922 INFO request completed, latency_us: 657, response_code: 200, uri: /0cf65393-f6a2-480b-8f0b-78c9543ba992, method: GET, req_id: 69700115-be20-4d7e-9df7-8fd91102be79, remote_addr: [fd00:1122:3344:101::d]:48781, local_addr: [fd00:1122:3344:101::2]:4677, component: dropshot, file: /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/33e158b/dropshot/src/server.rs:849
163 May 30 04:29:43.922 INFO request completed, latency_us: 96, response_code: 200, uri: /0cf65393-f6a2-480b-8f0b-78c9543ba992, method: GET, req_id: 1e8e11da-e081-4684-8df8-d374d558f031, remote_addr: [fd00:1122:3344:101::d]:48781, local_addr: [fd00:1122:3344:101::2]:4677, component: dropshot, file: /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/33e158b/dropshot/src/server.rs:849
164 May 30 04:29:44.922 INFO request completed, latency_us: 86, response_code: 200, uri: /0cf65393-f6a2-480b-8f0b-78c9543ba992, method: GET, req_id: 2362a7ef-1fbd-4771-ad42-02f6059be411, remote_addr: [fd00:1122:3344:101::d]:48781, local_addr: [fd00:1122:3344:101::2]:4677, component: dropshot, file: /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/33e158b/dropshot/src/server.rs:849
165 May 30 04:29:45.921 INFO request completed, latency_us: 89, response_code: 200, uri: /0cf65393-f6a2-480b-8f0b-78c9543ba992, method: GET, req_id: 1739ebe8-d6c4-42d8-9bbc-57e582dd62a0, remote_addr: [fd00:1122:3344:101::d]:48781, local_addr: [fd00:1122:3344:101::2]:4677, component: dropshot, file: /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/33e158b/dropshot/src/server.rs:849
166 May 30 04:29:46.922 INFO request completed, latency_us: 98, response_code: 200, uri: /0cf65393-f6a2-480b-8f0b-78c9543ba992, method: GET, req_id: 4c88e2f2-f947-4920-ae18-8e6fd475513a, remote_addr: [fd00:1122:3344:101::d]:48781, local_addr: [fd00:1122:3344:101::2]:4677, component: dropshot, file: /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/33e158b/dropshot/src/server.rs:849
167 May 30 04:29:47.921 INFO request completed, latency_us: 88, response_code: 200, uri: /0cf65393-f6a2-480b-8f0b-78c9543ba992, method: GET, req_id: 77218f94-e56e-44a5-84fa-15bd7c68001d, remote_addr: [fd00:1122:3344:101::d]:48781, local_addr: [fd00:1122:3344:101::2]:4677, component: dropshot, file: /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/33e158b/dropshot/src/server.rs:849
168 May 30 04:29:48.922 INFO request completed, latency_us: 131, response_code: 200, uri: /0cf65393-f6a2-480b-8f0b-78c9543ba992, method: GET, req_id: cec2812a-5837-4185-bb33-454c22580390, remote_addr: [fd00:1122:3344:101::d]:48781, local_addr: [fd00:1122:3344:101::2]:4677, component: dropshot, file: /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/33e158b/dropshot/src/server.rs:849
169 May 30 04:29:49.922 INFO request completed, latency_us: 91, response_code: 200, uri: /0cf65393-f6a2-480b-8f0b-78c9543ba992, method: GET, req_id: d30837af-8845-415c-9184-3f7913fbbcf4, remote_addr: [fd00:1122:3344:101::d]:48781, local_addr: [fd00:1122:3344:101::2]:4677, component: dropshot, file: /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/33e158b/dropshot/src/server.rs:849
170 May 30 04:29:50.921 INFO request completed, latency_us: 87, response_code: 200, uri: /0cf65393-f6a2-480b-8f0b-78c9543ba992, method: GET, req_id: 9394c8b1-53eb-46a8-926d-f8c7409476ae, remote_addr: [fd00:1122:3344:101::d]:48781, local_addr: [fd00:1122:3344:101::2]:4677, component: dropshot, file: /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/33e158b/dropshot/src/server.rs:849
171 May 30 04:29:51.922 INFO request completed, latency_us: 97, response_code: 200, uri: /0cf65393-f6a2-480b-8f0b-78c9543ba992, method: GET, req_id: ab722dd4-edfd-4aea-a614-793460a74d5a, remote_addr: [fd00:1122:3344:101::d]:48781, local_addr: [fd00:1122:3344:101::2]:4677, component: dropshot, file: /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/33e158b/dropshot/src/server.rs:849
172 May 30 04:29:52.922 INFO request completed, latency_us: 87, response_code: 200, uri: /0cf65393-f6a2-480b-8f0b-78c9543ba992, method: GET, req_id: 05206111-52f8-4c2d-a3d3-2b62291f9109, remote_addr: [fd00:1122:3344:101::d]:48781, local_addr: [fd00:1122:3344:101::2]:4677, component: dropshot, file: /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/33e158b/dropshot/src/server.rs:849
173 May 30 04:29:53.922 INFO request completed, latency_us: 83, response_code: 200, uri: /0cf65393-f6a2-480b-8f0b-78c9543ba992, method: GET, req_id: 35f4b8da-8305-4291-a7d2-9927c0dd14bb, remote_addr: [fd00:1122:3344:101::d]:48781, local_addr: [fd00:1122:3344:101::2]:4677, component: dropshot, file: /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/33e158b/dropshot/src/server.rs:849
174 May 30 04:29:54.922 INFO request completed, latency_us: 87, response_code: 200, uri: /0cf65393-f6a2-480b-8f0b-78c9543ba992, method: GET, req_id: 8fd62405-e980-4228-8770-32b34ed275cd, remote_addr: [fd00:1122:3344:101::d]:48781, local_addr: [fd00:1122:3344:101::2]:4677, component: dropshot, file: /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/33e158b/dropshot/src/server.rs:849
175 May 30 04:29:55.922 INFO request completed, latency_us: 89, response_code: 200, uri: /0cf65393-f6a2-480b-8f0b-78c9543ba992, method: GET, req_id: 2dbc844a-7abe-494e-87ac-5e301778078f, remote_addr: [fd00:1122:3344:101::d]:48781, local_addr: [fd00:1122:3344:101::2]:4677, component: dropshot, file: /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/33e158b/dropshot/src/server.rs:849
176 May 30 04:29:56.922 INFO request completed, latency_us: 98, response_code: 200, uri: /0cf65393-f6a2-480b-8f0b-78c9543ba992, method: GET, req_id: fbff3051-9a7d-458e-8ddb-2357f5af815c, remote_addr: [fd00:1122:3344:101::d]:48781, local_addr: [fd00:1122:3344:101::2]:4677, component: dropshot, file: /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/33e158b/dropshot/src/server.rs:849
177 May 30 04:29:57.922 INFO request completed, latency_us: 98, response_code: 200, uri: /0cf65393-f6a2-480b-8f0b-78c9543ba992, method: GET, req_id: 72986d3d-9770-4ebc-9d87-cb01bf1728e3, remote_addr: [fd00:1122:3344:101::d]:48781, local_addr: [fd00:1122:3344:101::2]:4677, component: dropshot, file: /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/33e158b/dropshot/src/server.rs:849
178 May 30 04:29:58.922 INFO request completed, latency_us: 88, response_code: 200, uri: /0cf65393-f6a2-480b-8f0b-78c9543ba992, method: GET, req_id: c52f1a3c-6ad3-4ee8-98f1-23906e000bf3, remote_addr: [fd00:1122:3344:101::d]:48781, local_addr: [fd00:1122:3344:101::2]:4677, component: dropshot, file: /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/33e158b/dropshot/src/server.rs:849
179 May 30 04:29:59.922 INFO request completed, latency_us: 87, response_code: 200, uri: /0cf65393-f6a2-480b-8f0b-78c9543ba992, method: GET, req_id: 05b76bdd-09a1-4375-bdd5-13ac291e078b, remote_addr: [fd00:1122:3344:101::d]:48781, local_addr: [fd00:1122:3344:101::2]:4677, component: dropshot, file: /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/33e158b/dropshot/src/server.rs:849