Max level shown:
1 [ Mar 29 05:52:59 Disabled. ]
2 [ Mar 29 05:52:59 Rereading configuration. ]
3 [ Mar 29 05:53:11 Rereading configuration. ]
4 [ Mar 29 05:53:11 Rereading configuration. ]
5 [ Mar 29 05:53:11 Enabled. ]
6 [ Mar 29 05:53:11 Executing start method ("ctrun -l child -o noorphan,regent /opt/oxide/mg-ddm/pkg/ddm_method_script.sh &"). ]
7 [ Mar 29 05:53:11 Method "start" exited with status 0. ]
82024-03-29T05:53:12.037ZINFOslog-rs: signal handler waiting for context
92024-03-29T05:53:12.055ZINFOslog-rs: signal handler waiting got context
102024-03-29T05:53:12.056ZINFOslog-rs: [tfportrear0_0] sm initialized with addr fe80::183c:81ff:fed7:3feb on if tfportrear0_0 index 3
112024-03-29T05:53:12.056ZDEBGslog-rs: [tfportrear0_0] starting discovery handler
122024-03-29T05:53:12.056ZINFOslog-rs: admin: listening on [::1]:8000
13 [ Mar 29 05:53:13 Rereading configuration. ]
14 [ Mar 29 05:53:13 Executing refresh method ("/opt/oxide/mg-ddm/pkg/refresh_ctrun_child.sh -c 445"). ]
15 + CONTRACT=''
16 + getopts c: o
17 + CONTRACT=445
18 + getopts c: o
19 + [[ -n 445 ]]
20 + pgrep -c 445
21 + PARENT_PID=4267
22 + [[ -n 4267 ]]
23 + pgrep -P 4267
24 + CHILD_PID=4270
25 + [[ -n 4270 ]]
26 + kill -USR1 4270
272024-03-29T05:53:13.368ZINFOslog-rs: handling smf refresh
28 [ Mar 29 05:53:13 Method "refresh" exited with status 0. ]
292024-03-29T05:53:13.370ZINFOslog-rs: starting stats server on smf refresh
302024-03-29T05:53:13.370ZINFOslog-rs: resolving nexus
312024-03-29T05:53:13.370ZINFOslog-rs: new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53]
322024-03-29T05:53:13.445ZWARNslog-rs: [tfportrear0_0] admin event in solicit state: Announce(Underlay({Ipv6Prefix { addr: fd95:c037:d8f8:ba69::1, len: 64 }}))
332024-03-29T05:53:13.445ZINFOslog-rs: advertise tunnel: { TunnelOrigin { overlay_prefix: V4( Ipv4Prefix { addr: 0.0.0.0, len: 0, }, ), boundary_addr: fd95:c037:d8f8:ba69::1, vni: 99, metric: 100, }, }
342024-03-29T05:53:13.445ZWARNslog-rs: [tfportrear0_0] admin event in solicit state: Announce(Tunnel({TunnelOrigin { overlay_prefix: V4(Ipv4Prefix { addr: 0.0.0.0, len: 0 }), boundary_addr: fd95:c037:d8f8:ba69::1, vni: 99, metric: 100 }}))
352024-03-29T05:53:14.056ZINFOslog-rs: [tfportrear0_0] nbr is fe80::ac8a:b6ff:feaf:ec7e@buskin server
362024-03-29T05:53:14.056ZDEBGslog-rs: [tfportrear0_0] transition solicit -> exchange
372024-03-29T05:53:14.057ZINFOslog-rs: [tfportrear0_0] exchange: listening on [fe80::183c:81ff:fed7:3feb]:56797
382024-03-29T05:53:14.057ZINFOslog-rs: [tfportrear0_0] waiting for exchange server to start
392024-03-29T05:53:14.059ZINFOslog-rs: [tfportrear0_0] sending 5 routes to dendrite
402024-03-29T05:53:14.059ZDEBGslog-rs: [tfportrear0_0] sending to dpd host=localhost port=12224
412024-03-29T05:53:14.100ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:2:: -> fe80::ac8a:b6ff:feaf:ec7e on port Rear(RearPort(0))/0
422024-03-29T05:53:14.101ZINFOslog-rs: [tfportrear0_0] adding route fdb0:200:: -> fe80::ac8a:b6ff:feaf:ec7e on port Rear(RearPort(0))/0
432024-03-29T05:53:14.101ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:3:: -> fe80::ac8a:b6ff:feaf:ec7e on port Rear(RearPort(0))/0
442024-03-29T05:53:14.101ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:101:: -> fe80::ac8a:b6ff:feaf:ec7e on port Rear(RearPort(0))/0
452024-03-29T05:53:14.101ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:1:: -> fe80::ac8a:b6ff:feaf:ec7e on port Rear(RearPort(0))/0
462024-03-29T05:53:14.102ZINFOslog-rs: [tfportrear0_0] removing routes 0 from dendrite
472024-03-29T05:53:14.131ZDEBGslog-rs: [tfportrear0_0] redistributing update to 0 peers
482024-03-29T05:53:18.374ZDEBGslog-rs: lookup_socket_v6 srv dns_name = _nexus._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("00110943-032d-4124-9b1a-b12f6e105bb1.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("33404a22-f9b6-4bb7-9f67-120c445fb407.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("595220f7-5a03-4309-9899-d806864e6ebc.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("595220f7-5a03-4309-9899-d806864e6ebc.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:101::c)) }], valid_until: Instant { tv_sec: 732, tv_nsec: 267420655 } })
49 Mar 29 05:53:18.375 DEBG registered DTrace probes
50 Mar 29 05:53:18.376 DEBG registered endpoint, path: /collect/{producer_id}, method: GET, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot
51 Mar 29 05:53:18.376 INFO listening, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/2d9b906/dropshot/src/server.rs:195
52 Mar 29 05:53:18.376 DEBG successfully registered DTrace USDT probes, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot
53 Mar 29 05:53:18.376 DEBG registering metric server as a producer
54 Mar 29 05:53:18.406 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::a]:12221/metrics/producers, method: POST
55 Mar 29 05:53:33.407 DEBG client response, result: Err(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::a)), port: Some(12221), path: "/metrics/producers", query: None, fragment: None }, source: TimedOut })
56 Mar 29 05:53:33.408 INFO{"msg":"stats server not created yet, Error registering as metric producer: Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers): operation timed out waiting 62.213366ms","v":0,"name":"slog-rs","level":40,"time":"2024-03-29T05:53:33.408980388Z","hostname":"oxz_switch","pid":4270}
57 received request to begin graceful shutdown, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/2d9b906/dropshot/src/server.rs:266
58 Mar 29 05:53:33.473 DEBG registered DTrace probes
59 Mar 29 05:53:33.473 DEBG registered endpoint, path: /collect/{producer_id}, method: GET, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot
60 Mar 29 05:53:33.473 INFO listening, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/2d9b906/dropshot/src/server.rs:195
61 Mar 29 05:53:33.473 DEBG successfully registered DTrace USDT probes, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot
62 Mar 29 05:53:33.473 DEBG registering metric server as a producer
63 Mar 29 05:53:33.502 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::a]:12221/metrics/producers, method: POST
64 Mar 29 05:53:48.503 DEBG client response, result: Err(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::a)), port: Some(12221), path: "/metrics/producers", query: None, fragment: None }, source: TimedOut })
65 Mar 29 05:53:48.504 INFO received request to begin graceful shutdown, local_addr: [fd00:{"msg":"stats server not created yet, Error registering as metric producer: Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers): operation timed out waiting 73.9586ms","v":0,"name":"slog-rs","level":40,"time":"2024-03-29T05:53:48.504444324Z","hostname":"oxz_switch","pid":4270}
66 1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/2d9b906/dropshot/src/server.rs:266
67 Mar 29 05:53:48.579 DEBG registered DTrace probes
68 Mar 29 05:53:48.579 DEBG registered endpoint, path: /collect/{producer_id}, method: GET, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot
69 Mar 29 05:53:48.580 INFO listening, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/2d9b906/dropshot/src/server.rs:195
70 Mar 29 05:53:48.580 DEBG successfully registered DTrace USDT probes, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot
71 Mar 29 05:53:48.580 DEBG registering metric server as a producer
72 Mar 29 05:53:48.608 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::a]:12221/metrics/producers, method: POST
73 Mar 29 05:54:03.609 DEBG client response, result: Err(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::a)), port: Some(12221), path: "/metrics/producers", query: None, fragment: None }, source: TimedOut })
74 Mar 29 05:54:03.610 INFO received request to begin graceful shutdown, local_addr: [fd00:{"msg":"stats server not created yet, Error registering as metric producer: Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers): operation timed out waiting 138.526336ms","v":0,"name":"slog-rs","level":40,"time":"2024-03-29T05:54:03.610745931Z","hostname":"oxz_switch","pid":4270}
75 1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/2d9b906/dropshot/src/server.rs:266
76 Mar 29 05:54:03.751 DEBG registered DTrace probes
77 Mar 29 05:54:03.751 DEBG registered endpoint, path: /collect/{producer_id}, method: GET, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot
78 Mar 29 05:54:03.751 INFO listening, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/2d9b906/dropshot/src/server.rs:195
79 Mar 29 05:54:03.751 DEBG successfully registered DTrace USDT probes, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot
80 Mar 29 05:54:03.751 DEBG registering metric server as a producer
81 Mar 29 05:54:03.779 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::a]:12221/metrics/producers, method: POST
82 Mar 29 05:54:18.780 DEBG client response, result: Err(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::a)), port: Some(12221), path: "/metrics/producers", query: None, fragment: None }, source: TimedOut })
83 Mar 29 05:54:18.781 INFO received request to begin graceful shutdown, local_addr: [fd00:1122:{"msg":"stats server not created yet, Error registering as metric producer: Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers): operation timed out waiting 180.782011ms","v":0,"name":"slog-rs","level":40,"time":"2024-03-29T05:54:18.781871625Z","hostname":"oxz_switch","pid":4270}
84 3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/2d9b906/dropshot/src/server.rs:266
85 Mar 29 05:54:18.964 DEBG registered DTrace probes
86 Mar 29 05:54:18.964 DEBG registered endpoint, path: /collect/{producer_id}, method: GET, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot
87 Mar 29 05:54:18.964 INFO listening, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/2d9b906/dropshot/src/server.rs:195
88 Mar 29 05:54:18.964 DEBG successfully registered DTrace USDT probes, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot
89 Mar 29 05:54:18.964 DEBG registering metric server as a producer
90 Mar 29 05:54:18.993 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::a]:12221/metrics/producers, method: POST
91 Mar 29 05:54:33.995 DEBG client response, result: Err(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::a)), port: Some(12221), path: "/metrics/producers", query: None, fragment: None }, source: TimedOut })
92 Mar 29 05:54:33.996 INFO{"msg":"stats server not created yet, Error registering as metric producer: Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers): operation timed out waiting 110.389772ms","v":0,"name":"slog-rs","level":40,"time":"2024-03-29T05:54:33.996379805Z","hostname":"oxz_switch","pid":4270}
93 received request to begin graceful shutdown, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/2d9b906/dropshot/src/server.rs:266
94 Mar 29 05:54:34.107 DEBG registered DTrace probes
95 Mar 29 05:54:34.107 DEBG registered endpoint, path: /collect/{producer_id}, method: GET, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot
96 Mar 29 05:54:34.107 INFO listening, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/2d9b906/dropshot/src/server.rs:195
97 Mar 29 05:54:34.108 DEBG successfully registered DTrace USDT probes, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot
98 Mar 29 05:54:34.108 DEBG registering metric server as a producer
99 Mar 29 05:54:34.136 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::a]:12221/metrics/producers, method: POST
100 Mar 29 05:54:49.138 DEBG client response, result: Err(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::a)), port: Some(12221), path: "/metrics/producers", query: None, fragment: None }, source: TimedOut })
1012024-03-29T05:54:49.139ZWARNslog-rs: stats server not created yet, Error registering as metric producer: Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers): operation timed out waiting 278.219257ms
102 Mar 29 05:54:49.139 INFO received request to begin graceful shutdown, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/2d9b906/dropshot/src/server.rs:266
103 Mar 29 05:54:49.418 DEBG registered DTrace probes
104 Mar 29 05:54:49.418 DEBG registered endpoint, path: /collect/{producer_id}, method: GET, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot
105 Mar 29 05:54:49.418 INFO listening, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/2d9b906/dropshot/src/server.rs:195
106 Mar 29 05:54:49.419 DEBG successfully registered DTrace USDT probes, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot
107 Mar 29 05:54:49.419 DEBG registering metric server as a producer
108 Mar 29 05:54:49.447 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::a]:12221/metrics/producers, method: POST
109 Mar 29 05:55:04.448 DEBG client response, result: Err(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::a)), port: Some(12221), path: "/metrics/producers", query: None, fragment: None }, source: TimedOut })
110 Mar 29 05:55:04.450 INFO{"msg":"stats server not created yet, Error registering as metric producer: Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers): operation timed out waiting 271.142417ms","v":0,"name":"slog-rs","level":40,"time":"2024-03-29T05:55:04.450081604Z","hostname":"oxz_switch","pid":4270}
111 received request to begin graceful shutdown, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/2d9b906/dropshot/src/server.rs:266
112 Mar 29 05:55:04.722 DEBG registered DTrace probes
113 Mar 29 05:55:04.722 DEBG registered endpoint, path: /collect/{producer_id}, method: GET, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot
114 Mar 29 05:55:04.722 INFO listening, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/2d9b906/dropshot/src/server.rs:195
115 Mar 29 05:55:04.722 DEBG successfully registered DTrace USDT probes, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot
116 Mar 29 05:55:04.722 DEBG registering metric server as a producer
117 Mar 29 05:55:04.751 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::a]:12221/metrics/producers, method: POST
118 Mar 29 05:55:19.752 DEBG client response, result: Err(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::a)), port: Some(12221), path: "/metrics/producers", query: None, fragment: None }, source: TimedOut })
119 Mar 29 05:55:19.753{"msg":"stats server not created yet, Error registering as metric producer: Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers): operation timed out waiting 367.951867ms","v":0,"name":"slog-rs","level":40,"time":"2024-03-29T05:55:19.753895643Z","hostname":"oxz_switch","pid":4270}
120 INFO received request to begin graceful shutdown, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/2d9b906/dropshot/src/server.rs:266
121 Mar 29 05:55:20.123 DEBG registered DTrace probes
122 Mar 29 05:55:20.123 DEBG registered endpoint, path: /collect/{producer_id}, method: GET, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot
123 Mar 29 05:55:20.123 INFO listening, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/2d9b906/dropshot/src/server.rs:195
124 Mar 29 05:55:20.123 DEBG successfully registered DTrace USDT probes, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot
125 Mar 29 05:55:20.123 DEBG registering metric server as a producer
126 Mar 29 05:55:20.152 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::a]:12221/metrics/producers, method: POST
127 Mar 29 05:55:35.153 DEBG client response, result: Err(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::a)), port: Some(12221), path: "/metrics/producers", query: None, fragment: None }, source: TimedOut })
128 Mar 29 05:55:35.155 INFO received request to begin graceful shutdown, local_addr: {"msg":"stats server not created yet, Error registering as metric producer: Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers): operation timed out waiting 218.367019ms","v":0,"name":"slog-rs","level":40,"time":"2024-03-29T05:55:35.155018253Z","hostname":"oxz_switch","pid":4270}
129 [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/2d9b906/dropshot/src/server.rs:266
130 Mar 29 05:55:35.374 DEBG registered DTrace probes
131 Mar 29 05:55:35.374 DEBG registered endpoint, path: /collect/{producer_id}, method: GET, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot
132 Mar 29 05:55:35.374 INFO listening, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/2d9b906/dropshot/src/server.rs:195
133 Mar 29 05:55:35.374 DEBG successfully registered DTrace USDT probes, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot
134 Mar 29 05:55:35.374 DEBG registering metric server as a producer
135 Mar 29 05:55:35.403 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::a]:12221/metrics/producers, method: POST
136 Mar 29 05:55:50.403 DEBG client response, result: Err(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::a)), port: Some(12221), path: "/metrics/producers", query: None, fragment: None }, source: TimedOut })
137 Mar 29 05:55:50.404 INFO received request to begin graceful shutdown{"msg":"stats server not created yet, Error registering as metric producer: Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers): operation timed out waiting 697.770514ms","v":0,"name":"slog-rs","level":40,"time":"2024-03-29T05:55:50.404511912Z","hostname":"oxz_switch","pid":4270}
138 , local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/2d9b906/dropshot/src/server.rs:266
139 Mar 29 05:55:51.103 DEBG registered DTrace probes
140 Mar 29 05:55:51.103 DEBG registered endpoint, path: /collect/{producer_id}, method: GET, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot
141 Mar 29 05:55:51.104 INFO listening, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/2d9b906/dropshot/src/server.rs:195
142 Mar 29 05:55:51.104 DEBG successfully registered DTrace USDT probes, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot
143 Mar 29 05:55:51.104 DEBG registering metric server as a producer
144 Mar 29 05:55:51.133 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::a]:12221/metrics/producers, method: POST
145 Mar 29 05:56:06.134 DEBG client response, result: Err(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::a)), port: Some(12221), path: "/metrics/producers", query: None, fragment: None }, source: TimedOut })
146 Mar 29 05:56:06.135 INFO received request to begin graceful shutdown, local_addr: [fd00:1122:{"msg":"stats server not created yet, Error registering as metric producer: Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers): operation timed out waiting 844.110702ms","v":0,"name":"slog-rs","level":40,"time":"2024-03-29T05:56:06.135240488Z","hostname":"oxz_switch","pid":4270}
147 3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/2d9b906/dropshot/src/server.rs:266
148 Mar 29 05:56:06.980 DEBG registered DTrace probes
149 Mar 29 05:56:06.980 DEBG registered endpoint, path: /collect/{producer_id}, method: GET, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot
150 Mar 29 05:56:06.980 INFO listening, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/2d9b906/dropshot/src/server.rs:195
151 Mar 29 05:56:06.980 DEBG successfully registered DTrace USDT probes, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot
152 Mar 29 05:56:06.980 DEBG registering metric server as a producer
153 Mar 29 05:56:07.009 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::a]:12221/metrics/producers, method: POST
154 Mar 29 05:56:22.009 DEBG client response, result: Err(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::a)), port: Some(12221), path: "/metrics/producers", query: None, fragment: None }, source: TimedOut })
155 Mar 29 05:56:22.011 INFO received request to begin graceful shutdown, local_addr: [fd00:{"msg":"stats server not created yet, Error registering as metric producer: Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers): operation timed out waiting 543.73734ms","v":0,"name":"slog-rs","level":40,"time":"2024-03-29T05:56:22.010996813Z","hostname":"oxz_switch","pid":4270}
156 1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/2d9b906/dropshot/src/server.rs:266
157 Mar 29 05:56:22.555 DEBG registered DTrace probes
158 Mar 29 05:56:22.555 DEBG registered endpoint, path: /collect/{producer_id}, method: GET, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot
159 Mar 29 05:56:22.555 INFO listening, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/2d9b906/dropshot/src/server.rs:195
160 Mar 29 05:56:22.555 DEBG successfully registered DTrace USDT probes, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot
161 Mar 29 05:56:22.555 DEBG registering metric server as a producer
162 Mar 29 05:56:22.584 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::a]:12221/metrics/producers, method: POST
163 Mar 29 05:56:37.585 DEBG client response, result: Err(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::a)), port: Some(12221), path: "/metrics/producers", query: None, fragment: None }, source: TimedOut })
164 Mar 29 05:56:37.586 INFO received request to begin graceful shutdown, local_addr: [fd00:{"msg":"stats server not created yet, Error registering as metric producer: Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers): operation timed out waiting 1.259117973s","v":0,"name":"slog-rs","level":40,"time":"2024-03-29T05:56:37.586826808Z","hostname":"oxz_switch","pid":4270}
165 1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/2d9b906/dropshot/src/server.rs:266
166 Mar 29 05:56:38.846 DEBG registered DTrace probes
167 Mar 29 05:56:38.846 DEBG registered endpoint, path: /collect/{producer_id}, method: GET, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot
168 Mar 29 05:56:38.846 INFO listening, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/2d9b906/dropshot/src/server.rs:195
169 Mar 29 05:56:38.847 DEBG successfully registered DTrace USDT probes, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot
170 Mar 29 05:56:38.847 DEBG registering metric server as a producer
171 Mar 29 05:56:38.875 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::a]:12221/metrics/producers, method: POST
172 Mar 29 05:56:53.876 DEBG client response, result: Err(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::a)), port: Some(12221), path: "/metrics/producers", query: None, fragment: None }, source: TimedOut })
173 Mar 29 05:56:53.877 INFO {"msg":"stats server not created yet, Error registering as metric producer: Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers): operation timed out waiting 940.918832ms","v":0,"name":"slog-rs","level":40,"time":"2024-03-29T05:56:53.877418272Z","hostname":"oxz_switch","pid":4270}
174 received request to begin graceful shutdown, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/2d9b906/dropshot/src/server.rs:266
175 Mar 29 05:56:54.819 DEBG registered DTrace probes
176 Mar 29 05:56:54.820 DEBG registered endpoint, path: /collect/{producer_id}, method: GET, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot
177 Mar 29 05:56:54.820 INFO listening, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/2d9b906/dropshot/src/server.rs:195
178 Mar 29 05:56:54.820 DEBG successfully registered DTrace USDT probes, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot
179 Mar 29 05:56:54.820 DEBG registering metric server as a producer
180 Mar 29 05:56:54.849 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::a]:12221/metrics/producers, method: POST
181 Mar 29 05:57:09.850 DEBG client response, result: Err(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::a)), port: Some(12221), path: "/metrics/producers", query: None, fragment: None }, source: TimedOut })
182 Mar 29 05:57:09.851{"msg":"stats server not created yet, Error registering as metric producer: Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers): operation timed out waiting 1.656550103s","v":0,"name":"slog-rs","level":40,"time":"2024-03-29T05:57:09.85147984Z","hostname":"oxz_switch","pid":4270}
183 INFO received request to begin graceful shutdown, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/2d9b906/dropshot/src/server.rs:266
184 Mar 29 05:57:11.509 DEBG registered DTrace probes
185 Mar 29 05:57:11.509 DEBG registered endpoint, path: /collect/{producer_id}, method: GET, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot
186 Mar 29 05:57:11.509 INFO listening, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/2d9b906/dropshot/src/server.rs:195
187 Mar 29 05:57:11.509 DEBG successfully registered DTrace USDT probes, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot
188 Mar 29 05:57:11.509 DEBG registering metric server as a producer
189 Mar 29 05:57:11.537 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::a]:12221/metrics/producers, method: POST
190 Mar 29 05:57:26.538 DEBG client response, result: Err(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::a)), port: Some(12221), path: "/metrics/producers", query: None, fragment: None }, source: TimedOut })
191 Mar 29 05:57:26.539 INFO received request to begin graceful shutdown, local_addr{"msg":"stats server not created yet, Error registering as metric producer: Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers): operation timed out waiting 1.199508215s","v":0,"name":"slog-rs","level":40,"time":"2024-03-29T05:57:26.539938419Z","hostname":"oxz_switch","pid":4270}
192 : [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/2d9b906/dropshot/src/server.rs:266
193 Mar 29 05:57:27.740 DEBG registered DTrace probes
194 Mar 29 05:57:27.740 DEBG registered endpoint, path: /collect/{producer_id}, method: GET, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot
195 Mar 29 05:57:27.740 INFO listening, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/2d9b906/dropshot/src/server.rs:195
196 Mar 29 05:57:27.741 DEBG successfully registered DTrace USDT probes, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot
197 Mar 29 05:57:27.741 DEBG registering metric server as a producer
198 Mar 29 05:57:27.769 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::a]:12221/metrics/producers, method: POST
199 Mar 29 05:57:42.770 DEBG client response, result: Err(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::a)), port: Some(12221), path: "/metrics/producers", query: None, fragment: None }, source: TimedOut })
200 Mar 29 05:57:42.771{"msg":"stats server not created yet, Error registering as metric producer: Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers): operation timed out waiting 1.095685206s","v":0,"name":"slog-rs","level":40,"time":"2024-03-29T05:57:42.771664774Z","hostname":"oxz_switch","pid":4270}
201 INFO received request to begin graceful shutdown, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/2d9b906/dropshot/src/server.rs:266
202 Mar 29 05:57:43.868 DEBG registered DTrace probes
203 Mar 29 05:57:43.868 DEBG registered endpoint, path: /collect/{producer_id}, method: GET, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot
204 Mar 29 05:57:43.868 INFO listening, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/2d9b906/dropshot/src/server.rs:195
205 Mar 29 05:57:43.868 DEBG successfully registered DTrace USDT probes, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot
206 Mar 29 05:57:43.868 DEBG registering metric server as a producer
207 Mar 29 05:57:43.897 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::a]:12221/metrics/producers, method: POST
208 Mar 29 05:57:58.897 DEBG client response, result: Err(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::a)), port: Some(12221), path: "/metrics/producers", query: None, fragment: None }, source: TimedOut })
209 Mar 29 05:57:58.898 INFO received request to begin graceful shutdown, local_addr: [fd00:1122{"msg":"stats server not created yet, Error registering as metric producer: Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers): operation timed out waiting 1.59380555s","v":0,"name":"slog-rs","level":40,"time":"2024-03-29T05:57:58.898516658Z","hostname":"oxz_switch","pid":4270}
210 :3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/2d9b906/dropshot/src/server.rs:266
211 Mar 29 05:58:00.493 DEBG registered DTrace probes
212 Mar 29 05:58:00.493 DEBG registered endpoint, path: /collect/{producer_id}, method: GET, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot
213 Mar 29 05:58:00.494 INFO listening, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/2d9b906/dropshot/src/server.rs:195
214 Mar 29 05:58:00.494 DEBG successfully registered DTrace USDT probes, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot
215 Mar 29 05:58:00.494 DEBG registering metric server as a producer
216 Mar 29 05:58:00.522 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::a]:12221/metrics/producers, method: POST
217 Mar 29 05:58:15.523 DEBG client response, result: Err(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::a)), port: Some(12221), path: "/metrics/producers", query: None, fragment: None }, source: TimedOut })
218 Mar 29 05:58:15.524 INFO received request to begin graceful shutdown, local_addr: [fd00:{"msg":"stats server not created yet, Error registering as metric producer: Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers): operation timed out waiting 1.813276944s","v":0,"name":"slog-rs","level":40,"time":"2024-03-29T05:58:15.524688886Z","hostname":"oxz_switch","pid":4270}
219 1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/2d9b906/dropshot/src/server.rs:266
220 Mar 29 05:58:17.339 DEBG registered DTrace probes
221 Mar 29 05:58:17.339 DEBG registered endpoint, path: /collect/{producer_id}, method: GET, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot
222 Mar 29 05:58:17.339 INFO listening, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/2d9b906/dropshot/src/server.rs:195
223 Mar 29 05:58:17.339 DEBG successfully registered DTrace USDT probes, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot
224 Mar 29 05:58:17.339 DEBG registering metric server as a producer
225 Mar 29 05:58:17.368 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::a]:12221/metrics/producers, method: POST
226 Mar 29 05:58:32.370 DEBG client response, result: Err(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::a)), port: Some(12221), path: "/metrics/producers", query: None, fragment: None }, source: TimedOut })
227 Mar 29 05:58:32.370 INFO{"msg":"stats server not created yet, Error registering as metric producer: Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers): operation timed out waiting 3.318436526s","v":0,"name":"slog-rs","level":40,"time":"2024-03-29T05:58:32.370970549Z","hostname":"oxz_switch","pid":4270}
228 received request to begin graceful shutdown, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/2d9b906/dropshot/src/server.rs:266
229 Mar 29 05:58:35.690 DEBG registered DTrace probes
230 Mar 29 05:58:35.690 DEBG registered endpoint, path: /collect/{producer_id}, method: GET, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot
231 Mar 29 05:58:35.691 INFO listening, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/2d9b906/dropshot/src/server.rs:195
232 Mar 29 05:58:35.691 DEBG successfully registered DTrace USDT probes, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot
233 Mar 29 05:58:35.691 DEBG registering metric server as a producer
234 Mar 29 05:58:35.719 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::a]:12221/metrics/producers, method: POST