Max level shown:
1 [ Aug 28 00:51:38 Disabled. ]
2 [ Aug 28 00:51:38 Rereading configuration. ]
3 [ Aug 28 00:51:39 Enabled. ]
4 [ Aug 28 00:51:39 Rereading configuration. ]
5 [ Aug 28 00:51:39 Executing start method ("ctrun -l child -o noorphan,regent /opt/oxide/mgd/pkg/mgd_method_script.sh &"). ]
6 [ Aug 28 00:51:39 Method "start" exited with status 0. ]
72024-08-28T00:51:39.175ZINFOslog-rs: signal handler waiting for context
82024-08-28T00:51:39.196ZINFOslog-rs: signal handler waiting got context
92024-08-28T00:51:39.196ZINFOslog-rs: bgp routers: {}
102024-08-28T00:51:39.196ZINFOslog-rs: bfd peers: []
112024-08-28T00:51:39.197ZERROslog-rs: failed to start stats server: underlay address not found
122024-08-28T00:51:39.200ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = GET path = /bfd/peers unit = api-server
132024-08-28T00:51:39.200ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = PUT path = /bfd/peers unit = api-server
142024-08-28T00:51:39.200ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = DELETE path = /bfd/peers/{addr} unit = api-server
152024-08-28T00:51:39.201ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = DELETE path = /bgp/config/checker unit = api-server
162024-08-28T00:51:39.201ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = GET path = /bgp/config/checker unit = api-server
172024-08-28T00:51:39.201ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = POST path = /bgp/config/checker unit = api-server
182024-08-28T00:51:39.201ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = PUT path = /bgp/config/checker unit = api-server
192024-08-28T00:51:39.201ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = DELETE path = /bgp/config/neighbor unit = api-server
202024-08-28T00:51:39.201ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = GET path = /bgp/config/neighbor unit = api-server
212024-08-28T00:51:39.201ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = POST path = /bgp/config/neighbor unit = api-server
222024-08-28T00:51:39.201ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = PUT path = /bgp/config/neighbor unit = api-server
232024-08-28T00:51:39.201ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = GET path = /bgp/config/neighbors unit = api-server
242024-08-28T00:51:39.201ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = DELETE path = /bgp/config/origin4 unit = api-server
252024-08-28T00:51:39.201ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = GET path = /bgp/config/origin4 unit = api-server
262024-08-28T00:51:39.201ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = POST path = /bgp/config/origin4 unit = api-server
272024-08-28T00:51:39.201ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = PUT path = /bgp/config/origin4 unit = api-server
282024-08-28T00:51:39.201ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = DELETE path = /bgp/config/router unit = api-server
292024-08-28T00:51:39.201ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = GET path = /bgp/config/router unit = api-server
302024-08-28T00:51:39.201ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = POST path = /bgp/config/router unit = api-server
312024-08-28T00:51:39.201ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = PUT path = /bgp/config/router unit = api-server
322024-08-28T00:51:39.201ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = GET path = /bgp/config/routers unit = api-server
332024-08-28T00:51:39.201ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = DELETE path = /bgp/config/shaper unit = api-server
342024-08-28T00:51:39.201ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = GET path = /bgp/config/shaper unit = api-server
352024-08-28T00:51:39.201ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = POST path = /bgp/config/shaper unit = api-server
362024-08-28T00:51:39.201ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = PUT path = /bgp/config/shaper unit = api-server
372024-08-28T00:51:39.201ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = GET path = /bgp/message-history unit = api-server
382024-08-28T00:51:39.201ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = POST path = /bgp/omicron/apply unit = api-server
392024-08-28T00:51:39.201ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = GET path = /bgp/status/exported unit = api-server
402024-08-28T00:51:39.201ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = GET path = /bgp/status/imported unit = api-server
412024-08-28T00:51:39.201ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = GET path = /bgp/status/neighbors unit = api-server
422024-08-28T00:51:39.201ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = GET path = /bgp/status/selected unit = api-server
432024-08-28T00:51:39.201ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = DELETE path = /static/route4 unit = api-server
442024-08-28T00:51:39.201ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = GET path = /static/route4 unit = api-server
452024-08-28T00:51:39.201ZDEBGslog-rs: registered endpoint local_addr = [::]:4676 method = PUT path = /static/route4 unit = api-server
462024-08-28T00:51:39.201ZINFOslog-rs: admin: listening on [::]:4676
472024-08-28T00:51:39.201ZINFOslog-rs: listening local_addr = [::]:4676 unit = api-server
482024-08-28T00:51:39.202ZDEBGslog-rs: successfully registered DTrace USDT probes local_addr = [::]:4676 unit = api-server
492024-08-28T00:51:39.270ZWARNslog-rs: failed to ensure TEP address fde4:9bf9:2992:9055::1 on ASIC: Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6): error trying to connect: tcp connect error: Connection refused (os error 146)
502024-08-28T00:51:40.271ZWARNslog-rs: failed to ensure TEP address fde4:9bf9:2992:9055::1 on ASIC: Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6): error trying to connect: tcp connect error: Connection refused (os error 146)
512024-08-28T00:51:41.272ZWARNslog-rs: failed to ensure TEP address fde4:9bf9:2992:9055::1 on ASIC: Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6): error trying to connect: tcp connect error: Connection refused (os error 146)
522024-08-28T00:51:42.273ZWARNslog-rs: failed to ensure TEP address fde4:9bf9:2992:9055::1 on ASIC: Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6): error trying to connect: tcp connect error: Connection refused (os error 146)
532024-08-28T00:51:43.273ZWARNslog-rs: failed to ensure TEP address fde4:9bf9:2992:9055::1 on ASIC: Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6): error trying to connect: tcp connect error: Connection refused (os error 146)
542024-08-28T00:51:44.274ZWARNslog-rs: failed to ensure TEP address fde4:9bf9:2992:9055::1 on ASIC: Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6): error trying to connect: tcp connect error: Connection refused (os error 146)
552024-08-28T00:51:45.274ZWARNslog-rs: failed to ensure TEP address fde4:9bf9:2992:9055::1 on ASIC: Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6): error trying to connect: tcp connect error: Connection refused (os error 146)
562024-08-28T00:51:46.275ZWARNslog-rs: failed to ensure TEP address fde4:9bf9:2992:9055::1 on ASIC: Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6): error trying to connect: tcp connect error: Connection refused (os error 146)
572024-08-28T00:51:47.276ZWARNslog-rs: failed to ensure TEP address fde4:9bf9:2992:9055::1 on ASIC: Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6): error trying to connect: tcp connect error: Connection refused (os error 146)
582024-08-28T00:51:48.277ZWARNslog-rs: failed to ensure TEP address fde4:9bf9:2992:9055::1 on ASIC: Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6): error trying to connect: tcp connect error: Connection refused (os error 146)
592024-08-28T00:51:49.278ZWARNslog-rs: failed to ensure TEP address fde4:9bf9:2992:9055::1 on ASIC: Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6): error trying to connect: tcp connect error: Connection refused (os error 146)
602024-08-28T00:51:50.278ZWARNslog-rs: failed to ensure TEP address fde4:9bf9:2992:9055::1 on ASIC: Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6): error trying to connect: tcp connect error: Connection refused (os error 146)
61 [ Aug 28 00:51:50 Rereading configuration. ]
62 [ Aug 28 00:51:50 Executing refresh method ("/opt/oxide/mgd/pkg/refresh_ctrun_child.sh -c 214"). ]
63 + CONTRACT=''
64 + getopts c: o
65 + CONTRACT=214
66 + getopts c: o
67 + [[ -n 214 ]]
68 + pgrep -c 214
69 + PARENT_PID=2239
70 + [[ -n 2239 ]]
71 + pgrep -P 2239
72 + CHILD_PID=2241
73 + [[ -n 2241 ]]
74 + kill -USR1 2241
752024-08-28T00:51:50.396ZINFOslog-rs: handling smf refresh
762024-08-28T00:51:50.397ZINFOslog-rs: starting stats server on smf refresh
77 [ Aug 28 00:51:50 Method "refresh" exited with status 0. ]
78 Aug 28 00:51:50.398 DEBG registered DTrace probes
79 Aug 28 00:51:50.398 DEBG registered endpoint, path: /{producer_id}, method: GET, local_addr: [fd00:1122:3344:101::2]:4677, component: dropshot
80 Aug 28 00:51:50.398 INFO listening, local_addr: [fd00:1122:3344:101::2]:4677, component: dropshot, file: /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:205
81 Aug 28 00:51:50.399 DEBG successfully registered DTrace USDT probes, local_addr: [fd00:1122:3344:101::2]:4677, component: dropshot
82 Aug 28 00:51:50.399 DEBG Nexus IP not provided, will use DNS to resolve it
83 Aug 28 00:51:50.399 INFO new DNS resolver, addresses: [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53], component: internal-dns-resolver, file: /home/build/.cargo/git/checkouts/omicron-d039c41f152bda83/66ac7b3/internal-dns/src/resolver.rs:73
84 Aug 28 00:51:50.399 DEBG starting producer registration task
85 Aug 28 00:51:50.399 INFO starting oximeter metric producer server, interval: 1s, address: [fd00:1122:3344:101::2]:4677, producer_id: 0479269d-f2fd-4cd7-9a71-c7333514ed28, file: /home/build/.cargo/git/checkouts/omicron-d039c41f152bda83/66ac7b3/oximeter/producer/src/lib.rs:283
86 Aug 28 00:51:50.399 DEBG registering / renewing oximeter producer lease with Nexus, component: producer-registration-task
872024-08-28T00:51:50.769ZINFOslog-rs: accepted connection local_addr = [::]:4676 remote_addr = [fd00:1122:3344:101::1]:36674 unit = api-server
882024-08-28T00:51:50.770ZINFOslog-rs: request completed latency_us = 398 local_addr = [::]:4676 method = PUT remote_addr = [fd00:1122:3344:101::1]:36674 req_id = d03f9edc-89ef-4f3c-9e46-c46bf9a81d23 response_code = 204 unit = api-server uri = /static/route4
892024-08-28T00:51:50.771ZERROslog-rs: handling change failed: libnet error route io error No such process (os error 3)
902024-08-28T00:51:50.771ZINFOslog-rs: restarting sync loop
91 Aug 28 00:51:55.402 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("4ee19452-709a-4abd-a238-7af80f61d166.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("a5227819-155d-4a08-8e71-bd8a327ad739.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("f840e071-026c-488d-b1c9-2c4d1ce3e74d.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("f840e071-026c-488d-b1c9-2c4d1ce3e74d.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::b))) }], valid_until: Instant { tv_sec: 779, tv_nsec: 21799067 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
92 Aug 28 00:51:55.403 DEBG using nexus address for registration, addr: [fd00:1122:3344:101::c]:12221, component: producer-registration-task
93 Aug 28 00:51:55.436 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::c]:12221/metrics/producers, method: POST, component: producer-registration-task
94 Aug 28 00:52:10.914 DEBG client response, result: Err(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::c)), port: Some(12221), path: "/metrics/producers", query: None, fragment: None }, source: TimedOut }), component: producer-registration-task
95 Aug 28 00:52:10.915 WARN failed to register as a producer with Nexus, will retry, error: "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out", delay: 288.387073ms, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-d039c41f152bda83/66ac7b3/oximeter/producer/src/lib.rs:424
96 Aug 28 00:52:11.234 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::c]:12221/metrics/producers, method: POST, component: producer-registration-task
97 Aug 28 00:52:26.194 DEBG client response, result: Err(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::c)), port: Some(12221), path: "/metrics/producers", query: None, fragment: None }, source: TimedOut }), component: producer-registration-task
98 Aug 28 00:52:26.195 WARN failed to register as a producer with Nexus, will retry, error: "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out", delay: 635.877638ms, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-d039c41f152bda83/66ac7b3/oximeter/producer/src/lib.rs:424
99 Aug 28 00:52:26.862 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::c]:12221/metrics/producers, method: POST, component: producer-registration-task
100 Aug 28 00:52:41.822 DEBG client response, result: Err(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::c)), port: Some(12221), path: "/metrics/producers", query: None, fragment: None }, source: TimedOut }), component: producer-registration-task
101 Aug 28 00:52:41.823 WARN failed to register as a producer with Nexus, will retry, error: "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out", delay: 702.367579ms, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-d039c41f152bda83/66ac7b3/oximeter/producer/src/lib.rs:424
102 Aug 28 00:52:42.555 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::c]:12221/metrics/producers, method: POST, component: producer-registration-task
103 Aug 28 00:52:57.516 DEBG client response, result: Err(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::c)), port: Some(12221), path: "/metrics/producers", query: None, fragment: None }, source: TimedOut }), component: producer-registration-task
104 Aug 28 00:52:57.517 WARN failed to register as a producer with Nexus, will retry, error: "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out", delay: 2.076312551s, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-d039c41f152bda83/66ac7b3/oximeter/producer/src/lib.rs:424
105 Aug 28 00:52:59.619 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::c]:12221/metrics/producers, method: POST, component: producer-registration-task
106 Aug 28 00:53:14.580 DEBG client response, result: Err(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::c)), port: Some(12221), path: "/metrics/producers", query: None, fragment: None }, source: TimedOut }), component: producer-registration-task
107 Aug 28 00:53:14.581 WARN failed to register as a producer with Nexus, will retry, error: "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out", delay: 2.803438243s, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-d039c41f152bda83/66ac7b3/oximeter/producer/src/lib.rs:424
108 Aug 28 00:53:17.407 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::c]:12221/metrics/producers, method: POST, component: producer-registration-task
109 Aug 28 00:53:32.368 DEBG client response, result: Err(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::c)), port: Some(12221), path: "/metrics/producers", query: None, fragment: None }, source: TimedOut }), component: producer-registration-task
110 Aug 28 00:53:32.369 WARN failed to register as a producer with Nexus, will retry, error: "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out", delay: 9.786527105s, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-d039c41f152bda83/66ac7b3/oximeter/producer/src/lib.rs:424
111 Aug 28 00:53:42.163 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::c]:12221/metrics/producers, method: POST, component: producer-registration-task
112 Aug 28 00:53:57.124 DEBG client response, result: Err(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::c)), port: Some(12221), path: "/metrics/producers", query: None, fragment: None }, source: TimedOut }), component: producer-registration-task
113 Aug 28 00:53:57.125 WARN failed to register as a producer with Nexus, will retry, error: "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out", delay: 23.442725535s, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-d039c41f152bda83/66ac7b3/oximeter/producer/src/lib.rs:424
114 Aug 28 00:54:20.535 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::c]:12221/metrics/producers, method: POST, component: producer-registration-task
115 Aug 28 00:54:35.515 DEBG client response, result: Err(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::c)), port: Some(12221), path: "/metrics/producers", query: None, fragment: None }, source: TimedOut }), component: producer-registration-task
116 Aug 28 00:54:35.516 WARN failed to register as a producer with Nexus, will retry, error: "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out", delay: 19.866636352s, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-d039c41f152bda83/66ac7b3/oximeter/producer/src/lib.rs:424
117 Aug 28 00:54:55.437 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::c]:12221/metrics/producers, method: POST, component: producer-registration-task
118 Aug 28 00:55:10.438 DEBG client response, result: Err(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::c)), port: Some(12221), path: "/metrics/producers", query: None, fragment: None }, source: TimedOut }), component: producer-registration-task
119 Aug 28 00:55:10.442 WARN failed to register as a producer with Nexus, will retry, error: "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out", delay: 57.408106801s, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-d039c41f152bda83/66ac7b3/oximeter/producer/src/lib.rs:424
1202024-08-28T00:56:06.269ZINFOslog-rs: accepted connection local_addr = [::]:4676 remote_addr = [fd00:1122:3344:101::a]:64130 unit = api-server
1212024-08-28T00:56:06.277ZINFOslog-rs: request completed latency_us = 10050 local_addr = [::]:4676 method = GET remote_addr = [fd00:1122:3344:101::a]:64130 req_id = 8af91ee8-d6a9-4c23-90a6-4080bcdb0fdb response_code = 200 unit = api-server uri = /bfd/peers
1222024-08-28T00:56:07.003ZINFOslog-rs: accepted connection local_addr = [::]:4676 remote_addr = [fd00:1122:3344:101::c]:51976 unit = api-server
1232024-08-28T00:56:07.003ZINFOslog-rs: request completed latency_us = 40 local_addr = [::]:4676 method = GET remote_addr = [fd00:1122:3344:101::c]:51976 req_id = 6f329d28-ec58-4128-b12b-82734c3f3ead response_code = 200 unit = api-server uri = /bfd/peers
1242024-08-28T00:56:07.067ZINFOslog-rs: accepted connection local_addr = [::]:4676 remote_addr = [fd00:1122:3344:101::b]:36453 unit = api-server
1252024-08-28T00:56:07.067ZINFOslog-rs: request completed latency_us = 32 local_addr = [::]:4676 method = GET remote_addr = [fd00:1122:3344:101::b]:36453 req_id = b5ddfe8d-ad76-4dca-aacf-305eb3c5a3f7 response_code = 200 unit = api-server uri = /bfd/peers
126 Aug 28 00:56:07.891 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::c]:12221/metrics/producers, method: POST, component: producer-registration-task
127 Aug 28 00:56:07.904 DEBG client response, result: Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::c)), port: Some(12221), path: "/metrics/producers", query: None, fragment: None }, status: 503, headers: {"content-type": "application/json", "x-request-id": "88641281-6a22-4a6a-a0b4-90d7d00f3174", "content-length": "133", "date": "Wed, 28 Aug 2024 00:56:07 GMT"} }), component: producer-registration-task
128 Aug 28 00:56:07.905 WARN failed to register as a producer with Nexus, will retry, error: "Error Response: status: 503 Service Unavailable; headers: {\"content-type\": \"application/json\", \"x-request-id\": \"88641281-6a22-4a6a-a0b4-90d7d00f3174\", \"content-length\": \"133\", \"date\": \"Wed, 28 Aug 2024 00:56:07 GMT\"}; value: Error { error_code: Some(\"ServiceNotAvailable\"), message: \"Service Unavailable\", request_id: \"88641281-6a22-4a6a-a0b4-90d7d00f3174\" }", delay: 72.186685995s, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-d039c41f152bda83/66ac7b3/oximeter/producer/src/lib.rs:424
1292024-08-28T00:56:36.335ZINFOslog-rs: accepted connection local_addr = [::]:4676 remote_addr = [fd00:1122:3344:101::a]:56503 unit = api-server
1302024-08-28T00:56:36.335ZINFOslog-rs: request completed latency_us = 26 local_addr = [::]:4676 method = GET remote_addr = [fd00:1122:3344:101::a]:56503 req_id = 8d37ded6-e45f-47a9-9908-01ab531d0be7 response_code = 200 unit = api-server uri = /bfd/peers
1312024-08-28T00:56:36.516ZINFOslog-rs: accepted connection local_addr = [::]:4676 remote_addr = [fd00:1122:3344:101::a]:42711 unit = api-server
1322024-08-28T00:56:36.518ZINFOslog-rs: request completed latency_us = 2370 local_addr = [::]:4676 method = GET remote_addr = [fd00:1122:3344:101::a]:42711 req_id = 720fbcf3-63a1-43ba-9f55-35e4fcec8921 response_code = 200 unit = api-server uri = /static/route4
1332024-08-28T00:56:37.034ZINFOslog-rs: accepted connection local_addr = [::]:4676 remote_addr = [fd00:1122:3344:101::c]:57829 unit = api-server
1342024-08-28T00:56:37.034ZINFOslog-rs: request completed latency_us = 28 local_addr = [::]:4676 method = GET remote_addr = [fd00:1122:3344:101::c]:57829 req_id = bbd8da3f-dfc5-4123-b7e8-56f064869cf8 response_code = 200 unit = api-server uri = /bfd/peers
1352024-08-28T00:56:37.116ZINFOslog-rs: accepted connection local_addr = [::]:4676 remote_addr = [fd00:1122:3344:101::b]:56845 unit = api-server
1362024-08-28T00:56:37.116ZINFOslog-rs: request completed latency_us = 25 local_addr = [::]:4676 method = GET remote_addr = [fd00:1122:3344:101::b]:56845 req_id = 678cd43e-744a-4b7b-8f1e-2a8cf2ec5bc6 response_code = 200 unit = api-server uri = /bfd/peers
1372024-08-28T00:56:37.151ZINFOslog-rs: accepted connection local_addr = [::]:4676 remote_addr = [fd00:1122:3344:101::c]:47925 unit = api-server
1382024-08-28T00:56:37.151ZINFOslog-rs: request completed latency_us = 27 local_addr = [::]:4676 method = GET remote_addr = [fd00:1122:3344:101::c]:47925 req_id = da8fed96-681f-44b5-97b2-d5d61e324767 response_code = 200 unit = api-server uri = /static/route4
1392024-08-28T00:56:37.187ZINFOslog-rs: accepted connection local_addr = [::]:4676 remote_addr = [fd00:1122:3344:101::b]:50797 unit = api-server
1402024-08-28T00:56:37.187ZINFOslog-rs: request completed latency_us = 29 local_addr = [::]:4676 method = GET remote_addr = [fd00:1122:3344:101::b]:50797 req_id = 431bcd9a-ca71-42b7-a137-d74c6a60d937 response_code = 200 unit = api-server uri = /static/route4
1412024-08-28T00:57:06.323ZINFOslog-rs: accepted connection local_addr = [::]:4676 remote_addr = [fd00:1122:3344:101::a]:60078 unit = api-server
1422024-08-28T00:57:06.323ZINFOslog-rs: request completed latency_us = 28 local_addr = [::]:4676 method = GET remote_addr = [fd00:1122:3344:101::a]:60078 req_id = 1701e929-fc7d-4b03-b89f-1120bbfeba68 response_code = 200 unit = api-server uri = /bfd/peers
1432024-08-28T00:57:06.707ZINFOslog-rs: accepted connection local_addr = [::]:4676 remote_addr = [fd00:1122:3344:101::a]:44078 unit = api-server
1442024-08-28T00:57:06.707ZINFOslog-rs: request completed latency_us = 32 local_addr = [::]:4676 method = GET remote_addr = [fd00:1122:3344:101::a]:44078 req_id = 578b56fd-b46b-40d7-85d9-42ddf8416c94 response_code = 200 unit = api-server uri = /static/route4
1452024-08-28T00:57:07.015ZINFOslog-rs: accepted connection local_addr = [::]:4676 remote_addr = [fd00:1122:3344:101::b]:43006 unit = api-server
1462024-08-28T00:57:07.015ZINFOslog-rs: request completed latency_us = 26 local_addr = [::]:4676 method = GET remote_addr = [fd00:1122:3344:101::b]:43006 req_id = 8834428e-331f-4308-aaeb-191316140e21 response_code = 200 unit = api-server uri = /bfd/peers
1472024-08-28T00:57:07.185ZINFOslog-rs: accepted connection local_addr = [::]:4676 remote_addr = [fd00:1122:3344:101::c]:64826 unit = api-server
1482024-08-28T00:57:07.185ZINFOslog-rs: request completed latency_us = 34 local_addr = [::]:4676 method = GET remote_addr = [fd00:1122:3344:101::c]:64826 req_id = 271b7c31-18df-4ad5-ac37-1e75f9a2e60d response_code = 200 unit = api-server uri = /bfd/peers
1492024-08-28T00:57:07.480ZINFOslog-rs: accepted connection local_addr = [::]:4676 remote_addr = [fd00:1122:3344:101::b]:63598 unit = api-server
1502024-08-28T00:57:07.480ZINFOslog-rs: request completed latency_us = 31 local_addr = [::]:4676 method = GET remote_addr = [fd00:1122:3344:101::b]:63598 req_id = 1798295f-9078-4783-80b4-bb87ca75a687 response_code = 200 unit = api-server uri = /static/route4
1512024-08-28T00:57:07.503ZINFOslog-rs: accepted connection local_addr = [::]:4676 remote_addr = [fd00:1122:3344:101::c]:64086 unit = api-server
1522024-08-28T00:57:07.503ZINFOslog-rs: request completed latency_us = 27 local_addr = [::]:4676 method = GET remote_addr = [fd00:1122:3344:101::c]:64086 req_id = 2b4a517c-5429-4ecd-b4da-af51ba10fa14 response_code = 200 unit = api-server uri = /static/route4
153 Aug 28 00:57:20.124 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::c]:12221/metrics/producers, method: POST, component: producer-registration-task
154 Aug 28 00:57:20.128 DEBG client response, result: Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::c)), port: Some(12221), path: "/metrics/producers", query: None, fragment: None }, status: 503, headers: {"content-type": "application/json", "x-request-id": "fbfaf262-faf0-4e1b-8eb0-6c0bc17ea083", "content-length": "133", "date": "Wed, 28 Aug 2024 00:57:20 GMT"} }), component: producer-registration-task
155 Aug 28 00:57:20.129 WARN failed to register as a producer with Nexus, will retry, error: "Error Response: status: 503 Service Unavailable; headers: {\"content-type\": \"application/json\", \"x-request-id\": \"fbfaf262-faf0-4e1b-8eb0-6c0bc17ea083\", \"content-length\": \"133\", \"date\": \"Wed, 28 Aug 2024 00:57:20 GMT\"}; value: Error { error_code: Some(\"ServiceNotAvailable\"), message: \"Service Unavailable\", request_id: \"fbfaf262-faf0-4e1b-8eb0-6c0bc17ea083\" }", delay: 209.5358249s, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-d039c41f152bda83/66ac7b3/oximeter/producer/src/lib.rs:424
1562024-08-28T00:57:36.370ZINFOslog-rs: accepted connection local_addr = [::]:4676 remote_addr = [fd00:1122:3344:101::a]:60760 unit = api-server
1572024-08-28T00:57:36.371ZINFOslog-rs: request completed latency_us = 830 local_addr = [::]:4676 method = GET remote_addr = [fd00:1122:3344:101::a]:60760 req_id = ff56697d-e722-4bcf-a364-08f9f53cf4ad response_code = 200 unit = api-server uri = /bfd/peers
1582024-08-28T00:57:36.704ZINFOslog-rs: accepted connection local_addr = [::]:4676 remote_addr = [fd00:1122:3344:101::a]:52483 unit = api-server
1592024-08-28T00:57:36.704ZINFOslog-rs: request completed latency_us = 36 local_addr = [::]:4676 method = GET remote_addr = [fd00:1122:3344:101::a]:52483 req_id = d4f82e26-4192-4526-88a0-16316a7af006 response_code = 200 unit = api-server uri = /static/route4
1602024-08-28T00:57:37.086ZINFOslog-rs: accepted connection local_addr = [::]:4676 remote_addr = [fd00:1122:3344:101::b]:43191 unit = api-server
1612024-08-28T00:57:37.086ZINFOslog-rs: request completed latency_us = 30 local_addr = [::]:4676 method = GET remote_addr = [fd00:1122:3344:101::b]:43191 req_id = dea6f096-63a8-4cb2-9e2b-398f009b76cb response_code = 200 unit = api-server uri = /bfd/peers
1622024-08-28T00:57:37.147ZINFOslog-rs: accepted connection local_addr = [::]:4676 remote_addr = [fd00:1122:3344:101::c]:34667 unit = api-server
1632024-08-28T00:57:37.147ZINFOslog-rs: request completed latency_us = 29 local_addr = [::]:4676 method = GET remote_addr = [fd00:1122:3344:101::c]:34667 req_id = 20e1e284-e570-4dc9-ab70-671db7de477f response_code = 200 unit = api-server uri = /bfd/peers
1642024-08-28T00:57:37.300ZINFOslog-rs: accepted connection local_addr = [::]:4676 remote_addr = [fd00:1122:3344:101::c]:33908 unit = api-server
1652024-08-28T00:57:37.300ZINFOslog-rs: request completed latency_us = 37 local_addr = [::]:4676 method = GET remote_addr = [fd00:1122:3344:101::c]:33908 req_id = 1003647e-4605-4d57-a19a-7fa1fba78282 response_code = 200 unit = api-server uri = /static/route4
1662024-08-28T00:57:37.311ZINFOslog-rs: accepted connection local_addr = [::]:4676 remote_addr = [fd00:1122:3344:101::b]:35079 unit = api-server
1672024-08-28T00:57:37.311ZINFOslog-rs: request completed latency_us = 23 local_addr = [::]:4676 method = GET remote_addr = [fd00:1122:3344:101::b]:35079 req_id = 5e4b589d-1001-4ac8-9aae-39a5189906f4 response_code = 200 unit = api-server uri = /static/route4
1682024-08-28T00:58:06.563ZINFOslog-rs: accepted connection local_addr = [::]:4676 remote_addr = [fd00:1122:3344:101::a]:54855 unit = api-server
1692024-08-28T00:58:06.563ZINFOslog-rs: request completed latency_us = 30 local_addr = [::]:4676 method = GET remote_addr = [fd00:1122:3344:101::a]:54855 req_id = f721c704-f8a7-4f82-9faa-0c2d66cebdf7 response_code = 200 unit = api-server uri = /bfd/peers
1702024-08-28T00:58:06.615ZINFOslog-rs: accepted connection local_addr = [::]:4676 remote_addr = [fd00:1122:3344:101::a]:32886 unit = api-server
1712024-08-28T00:58:06.615ZINFOslog-rs: request completed latency_us = 34 local_addr = [::]:4676 method = GET remote_addr = [fd00:1122:3344:101::a]:32886 req_id = a094a7e1-844e-499e-b17e-160f617bfe28 response_code = 200 unit = api-server uri = /static/route4
1722024-08-28T00:58:07.148ZINFOslog-rs: accepted connection local_addr = [::]:4676 remote_addr = [fd00:1122:3344:101::b]:53663 unit = api-server
1732024-08-28T00:58:07.148ZINFOslog-rs: request completed latency_us = 28 local_addr = [::]:4676 method = GET remote_addr = [fd00:1122:3344:101::b]:53663 req_id = 9c10dba3-3065-4059-8d15-48397f3648fc response_code = 200 unit = api-server uri = /bfd/peers
1742024-08-28T00:58:07.363ZINFOslog-rs: accepted connection local_addr = [::]:4676 remote_addr = [fd00:1122:3344:101::c]:48090 unit = api-server
1752024-08-28T00:58:07.363ZINFOslog-rs: request completed latency_us = 29 local_addr = [::]:4676 method = GET remote_addr = [fd00:1122:3344:101::c]:48090 req_id = 68285fd0-a3b6-4f36-ae4b-ccef0eae57b1 response_code = 200 unit = api-server uri = /bfd/peers
1762024-08-28T00:58:07.483ZINFOslog-rs: accepted connection local_addr = [::]:4676 remote_addr = [fd00:1122:3344:101::c]:39003 unit = api-server
1772024-08-28T00:58:07.483ZINFOslog-rs: request completed latency_us = 33 local_addr = [::]:4676 method = GET remote_addr = [fd00:1122:3344:101::c]:39003 req_id = 59cc6216-9ebb-4a76-a54b-d4517646e8bc response_code = 200 unit = api-server uri = /static/route4
1782024-08-28T00:58:07.561ZINFOslog-rs: accepted connection local_addr = [::]:4676 remote_addr = [fd00:1122:3344:101::b]:41815 unit = api-server
1792024-08-28T00:58:07.561ZINFOslog-rs: request completed latency_us = 28 local_addr = [::]:4676 method = GET remote_addr = [fd00:1122:3344:101::b]:41815 req_id = 747cdc05-4cff-43e4-8a2d-54bfe285a727 response_code = 200 unit = api-server uri = /static/route4
1802024-08-28T00:58:36.428ZINFOslog-rs: accepted connection local_addr = [::]:4676 remote_addr = [fd00:1122:3344:101::a]:59465 unit = api-server
1812024-08-28T00:58:36.428ZINFOslog-rs: request completed latency_us = 32 local_addr = [::]:4676 method = GET remote_addr = [fd00:1122:3344:101::a]:59465 req_id = f02ccb21-490d-4fc7-a252-3c0d17249e9c response_code = 200 unit = api-server uri = /bfd/peers
1822024-08-28T00:58:36.618ZINFOslog-rs: accepted connection local_addr = [::]:4676 remote_addr = [fd00:1122:3344:101::a]:43686 unit = api-server
1832024-08-28T00:58:36.618ZINFOslog-rs: request completed latency_us = 38 local_addr = [::]:4676 method = GET remote_addr = [fd00:1122:3344:101::a]:43686 req_id = 03a6b76d-4f23-40dd-a714-504993c58dd0 response_code = 200 unit = api-server uri = /static/route4
1842024-08-28T00:58:37.067ZINFOslog-rs: accepted connection local_addr = [::]:4676 remote_addr = [fd00:1122:3344:101::c]:34528 unit = api-server
1852024-08-28T00:58:37.067ZINFOslog-rs: request completed latency_us = 30 local_addr = [::]:4676 method = GET remote_addr = [fd00:1122:3344:101::c]:34528 req_id = 4e99cbd8-5b42-46e7-b527-674c5936561b response_code = 200 unit = api-server uri = /bfd/peers
1862024-08-28T00:58:37.077ZINFOslog-rs: accepted connection local_addr = [::]:4676 remote_addr = [fd00:1122:3344:101::b]:60790 unit = api-server
1872024-08-28T00:58:37.077ZINFOslog-rs: request completed latency_us = 22 local_addr = [::]:4676 method = GET remote_addr = [fd00:1122:3344:101::b]:60790 req_id = 9dec626e-7357-4775-8b48-38caba3f07b0 response_code = 200 unit = api-server uri = /bfd/peers
1882024-08-28T00:58:37.179ZINFOslog-rs: accepted connection local_addr = [::]:4676 remote_addr = [fd00:1122:3344:101::b]:54833 unit = api-server
1892024-08-28T00:58:37.179ZINFOslog-rs: request completed latency_us = 34 local_addr = [::]:4676 method = GET remote_addr = [fd00:1122:3344:101::b]:54833 req_id = 955c77d3-ae5c-4a7f-93e3-c9bb5d3291aa response_code = 200 unit = api-server uri = /static/route4
1902024-08-28T00:58:37.290ZINFOslog-rs: accepted connection local_addr = [::]:4676 remote_addr = [fd00:1122:3344:101::c]:35641 unit = api-server
1912024-08-28T00:58:37.290ZINFOslog-rs: request completed latency_us = 30 local_addr = [::]:4676 method = GET remote_addr = [fd00:1122:3344:101::c]:35641 req_id = 78c1154c-f45a-494f-8236-32e81d143ca9 response_code = 200 unit = api-server uri = /static/route4
1922024-08-28T00:59:06.513ZINFOslog-rs: accepted connection local_addr = [::]:4676 remote_addr = [fd00:1122:3344:101::a]:33467 unit = api-server
1932024-08-28T00:59:06.513ZINFOslog-rs: request completed latency_us = 509 local_addr = [::]:4676 method = GET remote_addr = [fd00:1122:3344:101::a]:33467 req_id = 5bfab584-a381-42aa-95ad-4014a85ad5f4 response_code = 200 unit = api-server uri = /bfd/peers
1942024-08-28T00:59:06.697ZINFOslog-rs: accepted connection local_addr = [::]:4676 remote_addr = [fd00:1122:3344:101::a]:44673 unit = api-server
1952024-08-28T00:59:06.698ZINFOslog-rs: request completed latency_us = 43 local_addr = [::]:4676 method = GET remote_addr = [fd00:1122:3344:101::a]:44673 req_id = 18634cc3-989f-45ca-9097-7ac7081b195e response_code = 200 unit = api-server uri = /static/route4
1962024-08-28T00:59:07.135ZINFOslog-rs: accepted connection local_addr = [::]:4676 remote_addr = [fd00:1122:3344:101::b]:39117 unit = api-server
1972024-08-28T00:59:07.135ZINFOslog-rs: request completed latency_us = 30 local_addr = [::]:4676 method = GET remote_addr = [fd00:1122:3344:101::b]:39117 req_id = 04f18d83-1a63-4861-87a5-bf263a061ca9 response_code = 200 unit = api-server uri = /bfd/peers
1982024-08-28T00:59:07.378ZINFOslog-rs: accepted connection local_addr = [::]:4676 remote_addr = [fd00:1122:3344:101::c]:50539 unit = api-server
1992024-08-28T00:59:07.378ZINFOslog-rs: request completed latency_us = 26 local_addr = [::]:4676 method = GET remote_addr = [fd00:1122:3344:101::c]:50539 req_id = 1adda56a-f232-4ab6-8389-8fa08ab3286d response_code = 200 unit = api-server uri = /bfd/peers
2002024-08-28T00:59:07.404ZINFOslog-rs: accepted connection local_addr = [::]:4676 remote_addr = [fd00:1122:3344:101::b]:47959 unit = api-server
2012024-08-28T00:59:07.404ZINFOslog-rs: request completed latency_us = 24 local_addr = [::]:4676 method = GET remote_addr = [fd00:1122:3344:101::b]:47959 req_id = 9defb001-8337-4389-92a9-b00362fb3ff9 response_code = 200 unit = api-server uri = /static/route4
2022024-08-28T00:59:07.549ZINFOslog-rs: accepted connection local_addr = [::]:4676 remote_addr = [fd00:1122:3344:101::c]:49065 unit = api-server
2032024-08-28T00:59:07.549ZINFOslog-rs: request completed latency_us = 31 local_addr = [::]:4676 method = GET remote_addr = [fd00:1122:3344:101::c]:49065 req_id = f2904c8e-4cf6-4087-ae0d-cd05cdbfc261 response_code = 200 unit = api-server uri = /static/route4
2042024-08-28T00:59:36.550ZINFOslog-rs: accepted connection local_addr = [::]:4676 remote_addr = [fd00:1122:3344:101::a]:45047 unit = api-server
2052024-08-28T00:59:36.550ZINFOslog-rs: request completed latency_us = 27 local_addr = [::]:4676 method = GET remote_addr = [fd00:1122:3344:101::a]:45047 req_id = 6027e332-5841-4162-b62d-8dc535c48fc6 response_code = 200 unit = api-server uri = /bfd/peers
2062024-08-28T00:59:36.628ZINFOslog-rs: accepted connection local_addr = [::]:4676 remote_addr = [fd00:1122:3344:101::a]:56185 unit = api-server
2072024-08-28T00:59:36.628ZINFOslog-rs: request completed latency_us = 32 local_addr = [::]:4676 method = GET remote_addr = [fd00:1122:3344:101::a]:56185 req_id = 97267134-d1b9-4a47-af17-df1e4e40096a response_code = 200 unit = api-server uri = /static/route4
2082024-08-28T00:59:37.011ZINFOslog-rs: accepted connection local_addr = [::]:4676 remote_addr = [fd00:1122:3344:101::c]:63636 unit = api-server
2092024-08-28T00:59:37.011ZINFOslog-rs: request completed latency_us = 28 local_addr = [::]:4676 method = GET remote_addr = [fd00:1122:3344:101::c]:63636 req_id = 1dd672f1-9ce8-4093-b3d4-15fbfd06728d response_code = 200 unit = api-server uri = /bfd/peers
2102024-08-28T00:59:37.038ZINFOslog-rs: accepted connection local_addr = [::]:4676 remote_addr = [fd00:1122:3344:101::b]:59819 unit = api-server
2112024-08-28T00:59:37.038ZINFOslog-rs: request completed latency_us = 29 local_addr = [::]:4676 method = GET remote_addr = [fd00:1122:3344:101::b]:59819 req_id = d7c3543a-e385-4e18-a2b1-cebaf1035e72 response_code = 200 unit = api-server uri = /bfd/peers
2122024-08-28T00:59:37.126ZINFOslog-rs: accepted connection local_addr = [::]:4676 remote_addr = [fd00:1122:3344:101::b]:37789 unit = api-server
2132024-08-28T00:59:37.126ZINFOslog-rs: request completed latency_us = 44 local_addr = [::]:4676 method = GET remote_addr = [fd00:1122:3344:101::b]:37789 req_id = c8a0f3f5-34a1-4cd4-8bc6-6fafc784f093 response_code = 200 unit = api-server uri = /static/route4
2142024-08-28T00:59:37.359ZINFOslog-rs: accepted connection local_addr = [::]:4676 remote_addr = [fd00:1122:3344:101::c]:64843 unit = api-server
2152024-08-28T00:59:37.359ZINFOslog-rs: request completed latency_us = 34 local_addr = [::]:4676 method = GET remote_addr = [fd00:1122:3344:101::c]:64843 req_id = d2b9e38f-cfdf-4845-840e-2a11f5df509c response_code = 200 unit = api-server uri = /static/route4