Max level shown:
1 [ Jun 20 21:28:37 Enabled. ]
2 [ Jun 20 21:28:37 Rereading configuration. ]
3 [ Jun 20 21:28:38 Rereading configuration. ]
4 [ Jun 20 21:28:41 Executing start method ("ctrun -l child -o noorphan,regent /opt/oxide/oximeter-collector/bin/oximeter run /var/svc/manifest/site/oximeter/config.toml --address [fd00:1122:3344:101::d]:12223 --id 83a45644-f784-4b40-a7e1-ec94b0dc35d4 &"). ]
5 [ Jun 20 21:28:41 Method "start" exited with status 0. ]
6 note: configured to log to "/dev/stdout"
72024-06-20T21:28:41.916ZDEBGoximeter: registered DTrace probes
82024-06-20T21:28:41.920ZINFOoximeter: starting oximeter server file = oximeter/collector/src/lib.rs:194
92024-06-20T21:28:41.924ZINFOoximeter (DnsResolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = internal-dns/src/resolver.rs:60
102024-06-20T21:28:41.926ZDEBGoximeter: creating ClickHouse client
112024-06-20T21:28:41.926ZDEBGoximeter (DnsResolver): lookup srv dns_name = _clickhouse._tcp.control-plane.oxide.internal
122024-06-20T21:28:42.304ZWARNoximeter (clickhouse-client): failed to read version collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d error = Telemetry database unavailable: error sending request for url (http://[fd00:1122:3344:101::e]:8123/?output_format_json_quote_64bit_integers=0&wait_end_of_query=1): error trying to connect: tcp connect error: Connection refused (os error 146) file = oximeter/db/src/client/mod.rs:683 id = 94f641cb-aa74-4ad1-bafc-9dfec5b222fa
132024-06-20T21:28:42.306ZWARNoximeter: failed to create ClickHouse client error = Database(DatabaseUnavailable("error sending request for url (http://[fd00:1122:3344:101::e]:8123/?output_format_json_quote_64bit_integers=0&wait_end_of_query=1): error trying to connect: tcp connect error: Connection refused (os error 146)")) file = oximeter/collector/src/lib.rs:216 retry_after = 313.028563ms
142024-06-20T21:28:42.620ZDEBGoximeter: creating ClickHouse client
152024-06-20T21:28:42.620ZDEBGoximeter (DnsResolver): lookup srv dns_name = _clickhouse._tcp.control-plane.oxide.internal
162024-06-20T21:28:42.670ZWARNoximeter (clickhouse-client): failed to read version collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d error = Telemetry database unavailable: error sending request for url (http://[fd00:1122:3344:101::e]:8123/?output_format_json_quote_64bit_integers=0&wait_end_of_query=1): error trying to connect: tcp connect error: Connection refused (os error 146) file = oximeter/db/src/client/mod.rs:683 id = 82ad2df6-0177-46ac-be82-ea9ff8e7abd7
172024-06-20T21:28:42.670ZWARNoximeter: failed to create ClickHouse client error = Database(DatabaseUnavailable("error sending request for url (http://[fd00:1122:3344:101::e]:8123/?output_format_json_quote_64bit_integers=0&wait_end_of_query=1): error trying to connect: tcp connect error: Connection refused (os error 146)")) file = oximeter/collector/src/lib.rs:216 retry_after = 554.393068ms
182024-06-20T21:28:43.226ZDEBGoximeter: creating ClickHouse client
192024-06-20T21:28:43.226ZDEBGoximeter (DnsResolver): lookup srv dns_name = _clickhouse._tcp.control-plane.oxide.internal
202024-06-20T21:28:43.259ZWARNoximeter (clickhouse-client): failed to read version collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d error = Telemetry database unavailable: error sending request for url (http://[fd00:1122:3344:101::e]:8123/?output_format_json_quote_64bit_integers=0&wait_end_of_query=1): error trying to connect: tcp connect error: Connection refused (os error 146) file = oximeter/db/src/client/mod.rs:683 id = 4ba59b06-8094-4263-91b0-835b74c47edc
212024-06-20T21:28:43.259ZWARNoximeter: failed to create ClickHouse client error = Database(DatabaseUnavailable("error sending request for url (http://[fd00:1122:3344:101::e]:8123/?output_format_json_quote_64bit_integers=0&wait_end_of_query=1): error trying to connect: tcp connect error: Connection refused (os error 146)")) file = oximeter/collector/src/lib.rs:216 retry_after = 1.471597142s
222024-06-20T21:28:44.732ZDEBGoximeter: creating ClickHouse client
232024-06-20T21:28:44.732ZDEBGoximeter (DnsResolver): lookup srv dns_name = _clickhouse._tcp.control-plane.oxide.internal
242024-06-20T21:28:44.804ZWARNoximeter (clickhouse-client): oximeter database does not exist, or is out-of-date collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d file = oximeter/db/src/client/mod.rs:679 id = 564220a6-751a-473a-9a3b-358bf07cde65
252024-06-20T21:28:44.804ZDEBGoximeter (oximeter-agent): oximeter database does not exist, creating collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d
262024-06-20T21:28:44.818ZINFOoximeter (clickhouse-client): reading db version collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d file = oximeter/db/src/client/mod.rs:617 id = 564220a6-751a-473a-9a3b-358bf07cde65
272024-06-20T21:28:44.819ZWARNoximeter (clickhouse-client): oximeter database does not exist, or is out-of-date collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d file = oximeter/db/src/client/mod.rs:679 id = 564220a6-751a-473a-9a3b-358bf07cde65
282024-06-20T21:28:44.819ZINFOoximeter (clickhouse-client): read oximeter database version collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d file = oximeter/db/src/client/mod.rs:621 id = 564220a6-751a-473a-9a3b-358bf07cde65 version = 0
292024-06-20T21:28:44.819ZINFOoximeter (clickhouse-client): wiping and re-initializing oximeter schema collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d file = oximeter/db/src/client/mod.rs:626 id = 564220a6-751a-473a-9a3b-358bf07cde65
302024-06-20T21:28:44.819ZDEBGoximeter (clickhouse-client): wiping ClickHouse database collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d id = 564220a6-751a-473a-9a3b-358bf07cde65
312024-06-20T21:28:44.820ZDEBGoximeter (clickhouse-client): initializing ClickHouse database collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d id = 564220a6-751a-473a-9a3b-358bf07cde65
322024-06-20T21:28:44.977ZINFOoximeter (clickhouse-client): inserting current version collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d file = oximeter/db/src/client/mod.rs:649 id = 564220a6-751a-473a-9a3b-358bf07cde65 version = 4
332024-06-20T21:28:44.995ZDEBGoximeter (dropshot): registered endpoint local_addr = [fd00:1122:3344:101::d]:12223 method = GET path = /info
342024-06-20T21:28:44.995ZDEBGoximeter (dropshot): registered endpoint local_addr = [fd00:1122:3344:101::d]:12223 method = GET path = /producers
352024-06-20T21:28:44.995ZDEBGoximeter (dropshot): registered endpoint local_addr = [fd00:1122:3344:101::d]:12223 method = POST path = /producers
362024-06-20T21:28:44.995ZDEBGoximeter (dropshot): registered endpoint local_addr = [fd00:1122:3344:101::d]:12223 method = DELETE path = /producers/{producer_id}
372024-06-20T21:28:44.995ZINFOoximeter (dropshot): listening file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:196 local_addr = [fd00:1122:3344:101::d]:12223
382024-06-20T21:28:44.995ZDEBGoximeter (dropshot): successfully registered DTrace USDT probes local_addr = [fd00:1122:3344:101::d]:12223
392024-06-20T21:28:44.995ZDEBGoximeter: contacting nexus
402024-06-20T21:28:44.995ZDEBGoximeter (DnsResolver): lookup_ipv6 srv dns_name = _nexus._tcp.control-plane.oxide.internal
412024-06-20T21:28:45.030ZDEBGoximeter: client request body = Some(Body) method = POST uri = http://[fd00:1122:3344:101::b]:12221/metrics/collectors
422024-06-20T21:28:45.030ZDEBGoximeter: client response result = Err(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::b)), port: Some(12221), path: "/metrics/collectors", query: None, fragment: None }, source: hyper::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
432024-06-20T21:28:45.031ZWARNoximeter: failed to contact nexus, will retry in 235.775033ms error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/collectors): error trying to connect: tcp connect error: Connection refused (os error 146)" file = oximeter/collector/src/lib.rs:284
442024-06-20T21:28:45.268ZDEBGoximeter: contacting nexus
452024-06-20T21:28:45.268ZDEBGoximeter (DnsResolver): lookup_ipv6 srv dns_name = _nexus._tcp.control-plane.oxide.internal
462024-06-20T21:28:45.297ZDEBGoximeter: client request body = Some(Body) method = POST uri = http://[fd00:1122:3344:101::b]:12221/metrics/collectors
472024-06-20T21:28:45.297ZDEBGoximeter: client response result = Err(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::b)), port: Some(12221), path: "/metrics/collectors", query: None, fragment: None }, source: hyper::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
482024-06-20T21:28:45.298ZWARNoximeter: failed to contact nexus, will retry in 631.745503ms error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/collectors): error trying to connect: tcp connect error: Connection refused (os error 146)" file = oximeter/collector/src/lib.rs:284
492024-06-20T21:28:45.931ZDEBGoximeter: contacting nexus
502024-06-20T21:28:45.931ZDEBGoximeter (DnsResolver): lookup_ipv6 srv dns_name = _nexus._tcp.control-plane.oxide.internal
512024-06-20T21:28:45.959ZDEBGoximeter: client request body = Some(Body) method = POST uri = http://[fd00:1122:3344:101::b]:12221/metrics/collectors
522024-06-20T21:28:45.959ZDEBGoximeter: client response result = Err(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::b)), port: Some(12221), path: "/metrics/collectors", query: None, fragment: None }, source: hyper::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
532024-06-20T21:28:45.960ZWARNoximeter: failed to contact nexus, will retry in 995.785893ms error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/collectors): error trying to connect: tcp connect error: Connection refused (os error 146)" file = oximeter/collector/src/lib.rs:284
542024-06-20T21:28:46.956ZDEBGoximeter: contacting nexus
552024-06-20T21:28:46.956ZDEBGoximeter (DnsResolver): lookup_ipv6 srv dns_name = _nexus._tcp.control-plane.oxide.internal
562024-06-20T21:28:46.986ZDEBGoximeter: client request body = Some(Body) method = POST uri = http://[fd00:1122:3344:101::b]:12221/metrics/collectors
572024-06-20T21:28:46.986ZDEBGoximeter: client response result = Err(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::b)), port: Some(12221), path: "/metrics/collectors", query: None, fragment: None }, source: hyper::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
582024-06-20T21:28:46.987ZWARNoximeter: failed to contact nexus, will retry in 1.25552964s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/collectors): error trying to connect: tcp connect error: Connection refused (os error 146)" file = oximeter/collector/src/lib.rs:284
592024-06-20T21:28:48.243ZDEBGoximeter: contacting nexus
602024-06-20T21:28:48.243ZDEBGoximeter (DnsResolver): lookup_ipv6 srv dns_name = _nexus._tcp.control-plane.oxide.internal
612024-06-20T21:28:48.273ZDEBGoximeter: client request body = Some(Body) method = POST uri = http://[fd00:1122:3344:101::b]:12221/metrics/collectors
622024-06-20T21:28:48.273ZDEBGoximeter: client response result = Err(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::b)), port: Some(12221), path: "/metrics/collectors", query: None, fragment: None }, source: hyper::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
632024-06-20T21:28:48.273ZWARNoximeter: failed to contact nexus, will retry in 5.572424737s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/collectors): error trying to connect: tcp connect error: Connection refused (os error 146)" file = oximeter/collector/src/lib.rs:284
642024-06-20T21:28:53.847ZDEBGoximeter: contacting nexus
652024-06-20T21:28:53.847ZDEBGoximeter (DnsResolver): lookup_ipv6 srv dns_name = _nexus._tcp.control-plane.oxide.internal
662024-06-20T21:28:53.878ZDEBGoximeter: client request body = Some(Body) method = POST uri = http://[fd00:1122:3344:101::b]:12221/metrics/collectors
672024-06-20T21:28:53.888ZDEBGoximeter: client response result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::b)), port: Some(12221), path: "/metrics/collectors", query: None, fragment: None }, status: 204, headers: {"x-request-id": "b46f327f-ca7c-4b16-a64d-5b640bda03d7", "date": "Thu, 20 Jun 2024 21:28:53 GMT"} })
682024-06-20T21:28:53.889ZINFOoximeter: oximeter registered with nexus file = oximeter/collector/src/lib.rs:302 id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4
692024-06-20T21:28:53.890ZINFOoximeter (oximeter-agent): refreshing list of producers from Nexus collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d file = oximeter/collector/src/agent.rs:757
702024-06-20T21:28:53.890ZDEBGoximeter (DnsResolver): lookup_ipv6 srv dns_name = _nexus._tcp.control-plane.oxide.internal
712024-06-20T21:28:53.926ZDEBGoximeter (oximeter-agent): client request body = None collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d method = GET uri = http://[fd00:1122:3344:101::b]:12221/metrics/collectors/83a45644-f784-4b40-a7e1-ec94b0dc35d4/producers?sort_by=id_ascending
722024-06-20T21:28:53.929ZDEBGoximeter (oximeter-agent): client response collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::b)), port: Some(12221), path: "/metrics/collectors/83a45644-f784-4b40-a7e1-ec94b0dc35d4/producers", query: Some("sort_by=id_ascending"), fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "478cd3a4-d932-46b6-a20b-9e8859a24f8d", "content-length": "29", "date": "Thu, 20 Jun 2024 21:28:53 GMT"} })
732024-06-20T21:28:53.930ZINFOoximeter (oximeter-agent): refreshed list of producers from Nexus collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d file = oximeter/collector/src/agent.rs:807 n_current_tasks = 0 n_pruned_tasks = 0
742024-06-20T21:29:32.335ZINFOoximeter (dropshot): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::d]:12223 remote_addr = [fd00:1122:3344:101::a]:65491
752024-06-20T21:29:32.341ZDEBGoximeter (oximeter-agent): registered new metric producer address = [fd00:1122:3344:101::a]:62763 collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d producer_id = 4c9d7efe-dded-4bfd-b6fb-594f5dd61d1c
762024-06-20T21:29:32.341ZINFOoximeter (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 2038 local_addr = [fd00:1122:3344:101::d]:12223 method = POST remote_addr = [fd00:1122:3344:101::a]:65491 req_id = 5fe71141-7b12-4a9f-adb0-3d85d5b7e4ba response_code = 204 uri = /producers
772024-06-20T21:29:32.374ZDEBGoximeter (collection-task): starting oximeter collection task address = [fd00:1122:3344:101::a]:62763 collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d interval = 10s producer_id = 4c9d7efe-dded-4bfd-b6fb-594f5dd61d1c
782024-06-20T21:29:42.373ZDEBGoximeter (collection-task): collecting from producer address = [fd00:1122:3344:101::a]:62763 collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d producer_id = 4c9d7efe-dded-4bfd-b6fb-594f5dd61d1c
792024-06-20T21:29:42.385ZDEBGoximeter (collection-task): collected results from producer address = [fd00:1122:3344:101::a]:62763 collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d n_results = 5 producer_id = 4c9d7efe-dded-4bfd-b6fb-594f5dd61d1c
802024-06-20T21:29:42.833ZINFOoximeter (dropshot): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::d]:12223 remote_addr = [fd00:1122:3344:101::a]:58481
812024-06-20T21:29:42.833ZDEBGoximeter (oximeter-agent): registered new metric producer address = [fd00:1122:3344:101::1]:8001 collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d producer_id = df63a4fe-3119-43c9-b4de-7606cf2321fe
822024-06-20T21:29:42.833ZINFOoximeter (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 126 local_addr = [fd00:1122:3344:101::d]:12223 method = POST remote_addr = [fd00:1122:3344:101::a]:58481 req_id = d853ca31-6745-4717-8fc2-dbee86be8e42 response_code = 204 uri = /producers
832024-06-20T21:29:42.865ZDEBGoximeter (collection-task): starting oximeter collection task address = [fd00:1122:3344:101::1]:8001 collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d interval = 1s producer_id = df63a4fe-3119-43c9-b4de-7606cf2321fe
842024-06-20T21:29:43.864ZDEBGoximeter (collection-task): collecting from producer address = [fd00:1122:3344:101::1]:8001 collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d producer_id = df63a4fe-3119-43c9-b4de-7606cf2321fe
852024-06-20T21:29:43.868ZDEBGoximeter (collection-task): collected results from producer address = [fd00:1122:3344:101::1]:8001 collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = df63a4fe-3119-43c9-b4de-7606cf2321fe
862024-06-20T21:29:44.865ZDEBGoximeter (collection-task): collecting from producer address = [fd00:1122:3344:101::1]:8001 collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d producer_id = df63a4fe-3119-43c9-b4de-7606cf2321fe
872024-06-20T21:29:44.866ZDEBGoximeter (collection-task): collected results from producer address = [fd00:1122:3344:101::1]:8001 collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = df63a4fe-3119-43c9-b4de-7606cf2321fe
882024-06-20T21:29:44.992ZDEBGoximeter (results-sink): inserting 90 samples into database collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d
892024-06-20T21:29:44.992ZDEBGoximeter (clickhouse-client): unrolling 90 total samples collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d id = 564220a6-751a-473a-9a3b-358bf07cde65
902024-06-20T21:29:45.008ZDEBGoximeter (clickhouse-client): retrieving timeseries schema from database collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d id = 564220a6-751a-473a-9a3b-358bf07cde65
912024-06-20T21:29:45.024ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d id = 564220a6-751a-473a-9a3b-358bf07cde65 schema = {"timeseries_name":"http_service:request_latency_histogram","fields.name":["id","method","name","route","status_code"],"fields.type":["Uuid","String","String","String","I64"],"fields.source":["Target","Metric","Target","Metric","Metric"],"datum_type":"HistogramF64","created":"2024-06-20 21:29:45.008680993"} timeseries_name = http_service:request_latency_histogram
922024-06-20T21:29:45.030ZDEBGoximeter (clickhouse-client): retrieving timeseries schema from database collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d id = 564220a6-751a-473a-9a3b-358bf07cde65
932024-06-20T21:29:45.056ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d id = 564220a6-751a-473a-9a3b-358bf07cde65 schema = {"timeseries_name":"collection_target:cpus_provisioned","fields.name":["id"],"fields.type":["Uuid"],"fields.source":["Target"],"datum_type":"I64","created":"2024-06-20 21:29:45.030469237"} timeseries_name = collection_target:cpus_provisioned
942024-06-20T21:29:45.056ZDEBGoximeter (clickhouse-client): retrieving timeseries schema from database collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d id = 564220a6-751a-473a-9a3b-358bf07cde65
952024-06-20T21:29:45.064ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d id = 564220a6-751a-473a-9a3b-358bf07cde65 schema = {"timeseries_name":"collection_target:ram_provisioned","fields.name":["id"],"fields.type":["Uuid"],"fields.source":["Target"],"datum_type":"I64","created":"2024-06-20 21:29:45.056831966"} timeseries_name = collection_target:ram_provisioned
962024-06-20T21:29:45.064ZDEBGoximeter (clickhouse-client): retrieving timeseries schema from database collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d id = 564220a6-751a-473a-9a3b-358bf07cde65
972024-06-20T21:29:45.066ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d id = 564220a6-751a-473a-9a3b-358bf07cde65 schema = {"timeseries_name":"collection_target:virtual_disk_space_provisioned","fields.name":["id"],"fields.type":["Uuid"],"fields.source":["Target"],"datum_type":"I64","created":"2024-06-20 21:29:45.064359998"} timeseries_name = collection_target:virtual_disk_space_provisioned
982024-06-20T21:29:45.066ZDEBGoximeter (clickhouse-client): retrieving timeseries schema from database collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d id = 564220a6-751a-473a-9a3b-358bf07cde65
992024-06-20T21:29:45.068ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d id = 564220a6-751a-473a-9a3b-358bf07cde65 schema = {"timeseries_name":"ddm_router:originated_underlay_prefixes","fields.name":["hostname","rack_id","sled_id"],"fields.type":["String","Uuid","Uuid"],"fields.source":["Target","Target","Target"],"datum_type":"U64","created":"2024-06-20 21:29:45.066816470"} timeseries_name = ddm_router:originated_underlay_prefixes
1002024-06-20T21:29:45.068ZDEBGoximeter (clickhouse-client): retrieving timeseries schema from database collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d id = 564220a6-751a-473a-9a3b-358bf07cde65
1012024-06-20T21:29:45.070ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d id = 564220a6-751a-473a-9a3b-358bf07cde65 schema = {"timeseries_name":"ddm_router:originated_tunnel_endpoints","fields.name":["hostname","rack_id","sled_id"],"fields.type":["String","Uuid","Uuid"],"fields.source":["Target","Target","Target"],"datum_type":"U64","created":"2024-06-20 21:29:45.068620196"} timeseries_name = ddm_router:originated_tunnel_endpoints
1022024-06-20T21:29:45.070ZDEBGoximeter (clickhouse-client): retrieving timeseries schema from database collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d id = 564220a6-751a-473a-9a3b-358bf07cde65
1032024-06-20T21:29:45.072ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d id = 564220a6-751a-473a-9a3b-358bf07cde65 schema = {"timeseries_name":"ddm_session:solicitations_sent","fields.name":["hostname","interface","rack_id","sled_id"],"fields.type":["String","String","Uuid","Uuid"],"fields.source":["Target","Target","Target","Target"],"datum_type":"CumulativeU64","created":"2024-06-20 21:29:45.070378471"} timeseries_name = ddm_session:solicitations_sent
1042024-06-20T21:29:45.072ZDEBGoximeter (clickhouse-client): retrieving timeseries schema from database collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d id = 564220a6-751a-473a-9a3b-358bf07cde65
1052024-06-20T21:29:45.073ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d id = 564220a6-751a-473a-9a3b-358bf07cde65 schema = {"timeseries_name":"ddm_session:solicitations_received","fields.name":["hostname","interface","rack_id","sled_id"],"fields.type":["String","String","Uuid","Uuid"],"fields.source":["Target","Target","Target","Target"],"datum_type":"CumulativeU64","created":"2024-06-20 21:29:45.072163096"} timeseries_name = ddm_session:solicitations_received
1062024-06-20T21:29:45.073ZDEBGoximeter (clickhouse-client): retrieving timeseries schema from database collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d id = 564220a6-751a-473a-9a3b-358bf07cde65
1072024-06-20T21:29:45.075ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d id = 564220a6-751a-473a-9a3b-358bf07cde65 schema = {"timeseries_name":"ddm_session:advertisements_sent","fields.name":["hostname","interface","rack_id","sled_id"],"fields.type":["String","String","Uuid","Uuid"],"fields.source":["Target","Target","Target","Target"],"datum_type":"CumulativeU64","created":"2024-06-20 21:29:45.073944381"} timeseries_name = ddm_session:advertisements_sent
1082024-06-20T21:29:45.075ZDEBGoximeter (clickhouse-client): retrieving timeseries schema from database collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d id = 564220a6-751a-473a-9a3b-358bf07cde65
1092024-06-20T21:29:45.077ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d id = 564220a6-751a-473a-9a3b-358bf07cde65 schema = {"timeseries_name":"ddm_session:advertisements_received","fields.name":["hostname","interface","rack_id","sled_id"],"fields.type":["String","String","Uuid","Uuid"],"fields.source":["Target","Target","Target","Target"],"datum_type":"CumulativeU64","created":"2024-06-20 21:29:45.075708886"} timeseries_name = ddm_session:advertisements_received
1102024-06-20T21:29:45.077ZDEBGoximeter (clickhouse-client): retrieving timeseries schema from database collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d id = 564220a6-751a-473a-9a3b-358bf07cde65
1112024-06-20T21:29:45.079ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d id = 564220a6-751a-473a-9a3b-358bf07cde65 schema = {"timeseries_name":"ddm_session:peer_expirations","fields.name":["hostname","interface","rack_id","sled_id"],"fields.type":["String","String","Uuid","Uuid"],"fields.source":["Target","Target","Target","Target"],"datum_type":"CumulativeU64","created":"2024-06-20 21:29:45.077449040"} timeseries_name = ddm_session:peer_expirations
1122024-06-20T21:29:45.079ZDEBGoximeter (clickhouse-client): retrieving timeseries schema from database collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d id = 564220a6-751a-473a-9a3b-358bf07cde65
1132024-06-20T21:29:45.080ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d id = 564220a6-751a-473a-9a3b-358bf07cde65 schema = {"timeseries_name":"ddm_session:peer_address_changes","fields.name":["hostname","interface","rack_id","sled_id"],"fields.type":["String","String","Uuid","Uuid"],"fields.source":["Target","Target","Target","Target"],"datum_type":"CumulativeU64","created":"2024-06-20 21:29:45.079216795"} timeseries_name = ddm_session:peer_address_changes
1142024-06-20T21:29:45.080ZDEBGoximeter (clickhouse-client): retrieving timeseries schema from database collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d id = 564220a6-751a-473a-9a3b-358bf07cde65
1152024-06-20T21:29:45.082ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d id = 564220a6-751a-473a-9a3b-358bf07cde65 schema = {"timeseries_name":"ddm_session:peer_sessions_established","fields.name":["hostname","interface","rack_id","sled_id"],"fields.type":["String","String","Uuid","Uuid"],"fields.source":["Target","Target","Target","Target"],"datum_type":"CumulativeU64","created":"2024-06-20 21:29:45.080970580"} timeseries_name = ddm_session:peer_sessions_established
1162024-06-20T21:29:45.082ZDEBGoximeter (clickhouse-client): retrieving timeseries schema from database collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d id = 564220a6-751a-473a-9a3b-358bf07cde65
1172024-06-20T21:29:45.084ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d id = 564220a6-751a-473a-9a3b-358bf07cde65 schema = {"timeseries_name":"ddm_session:updates_sent","fields.name":["hostname","interface","rack_id","sled_id"],"fields.type":["String","String","Uuid","Uuid"],"fields.source":["Target","Target","Target","Target"],"datum_type":"CumulativeU64","created":"2024-06-20 21:29:45.082757935"} timeseries_name = ddm_session:updates_sent
1182024-06-20T21:29:45.084ZDEBGoximeter (clickhouse-client): retrieving timeseries schema from database collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d id = 564220a6-751a-473a-9a3b-358bf07cde65
1192024-06-20T21:29:45.086ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d id = 564220a6-751a-473a-9a3b-358bf07cde65 schema = {"timeseries_name":"ddm_session:updates_received","fields.name":["hostname","interface","rack_id","sled_id"],"fields.type":["String","String","Uuid","Uuid"],"fields.source":["Target","Target","Target","Target"],"datum_type":"CumulativeU64","created":"2024-06-20 21:29:45.084564551"} timeseries_name = ddm_session:updates_received
1202024-06-20T21:29:45.086ZDEBGoximeter (clickhouse-client): retrieving timeseries schema from database collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d id = 564220a6-751a-473a-9a3b-358bf07cde65
1212024-06-20T21:29:45.088ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d id = 564220a6-751a-473a-9a3b-358bf07cde65 schema = {"timeseries_name":"ddm_session:update_send_fail","fields.name":["hostname","interface","rack_id","sled_id"],"fields.type":["String","String","Uuid","Uuid"],"fields.source":["Target","Target","Target","Target"],"datum_type":"CumulativeU64","created":"2024-06-20 21:29:45.086398918"} timeseries_name = ddm_session:update_send_fail
1222024-06-20T21:29:45.088ZDEBGoximeter (clickhouse-client): retrieving timeseries schema from database collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d id = 564220a6-751a-473a-9a3b-358bf07cde65
1232024-06-20T21:29:45.089ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d id = 564220a6-751a-473a-9a3b-358bf07cde65 schema = {"timeseries_name":"ddm_session:imported_underlay_prefixes","fields.name":["hostname","interface","rack_id","sled_id"],"fields.type":["String","String","Uuid","Uuid"],"fields.source":["Target","Target","Target","Target"],"datum_type":"U64","created":"2024-06-20 21:29:45.088208954"} timeseries_name = ddm_session:imported_underlay_prefixes
1242024-06-20T21:29:45.090ZDEBGoximeter (clickhouse-client): retrieving timeseries schema from database collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d id = 564220a6-751a-473a-9a3b-358bf07cde65
1252024-06-20T21:29:45.091ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d id = 564220a6-751a-473a-9a3b-358bf07cde65 schema = {"timeseries_name":"ddm_session:imported_tunnel_endpoints","fields.name":["hostname","interface","rack_id","sled_id"],"fields.type":["String","String","Uuid","Uuid"],"fields.source":["Target","Target","Target","Target"],"datum_type":"U64","created":"2024-06-20 21:29:45.090062091"} timeseries_name = ddm_session:imported_tunnel_endpoints
1262024-06-20T21:29:45.093ZDEBGoximeter (clickhouse-client): inserting 18 new timeseries schema collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d id = 564220a6-751a-473a-9a3b-358bf07cde65
1272024-06-20T21:29:45.119ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d id = 564220a6-751a-473a-9a3b-358bf07cde65 n_rows = 20 table_name = oximeter.fields_i64
1282024-06-20T21:29:45.123ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d id = 564220a6-751a-473a-9a3b-358bf07cde65 n_rows = 86 table_name = oximeter.fields_string
1292024-06-20T21:29:45.128ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d id = 564220a6-751a-473a-9a3b-358bf07cde65 n_rows = 58 table_name = oximeter.fields_uuid
1302024-06-20T21:29:45.137ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d id = 564220a6-751a-473a-9a3b-358bf07cde65 n_rows = 40 table_name = oximeter.measurements_cumulativeu64
1312024-06-20T21:29:45.145ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d id = 564220a6-751a-473a-9a3b-358bf07cde65 n_rows = 20 table_name = oximeter.measurements_histogramf64
1322024-06-20T21:29:45.147ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d id = 564220a6-751a-473a-9a3b-358bf07cde65 n_rows = 18 table_name = oximeter.measurements_i64
1332024-06-20T21:29:45.150ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d id = 564220a6-751a-473a-9a3b-358bf07cde65 n_rows = 12 table_name = oximeter.measurements_u64
1342024-06-20T21:29:45.865ZDEBGoximeter (collection-task): collecting from producer address = [fd00:1122:3344:101::1]:8001 collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d producer_id = df63a4fe-3119-43c9-b4de-7606cf2321fe
1352024-06-20T21:29:45.865ZDEBGoximeter (collection-task): collected results from producer address = [fd00:1122:3344:101::1]:8001 collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = df63a4fe-3119-43c9-b4de-7606cf2321fe
1362024-06-20T21:29:46.865ZDEBGoximeter (collection-task): collecting from producer address = [fd00:1122:3344:101::1]:8001 collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d producer_id = df63a4fe-3119-43c9-b4de-7606cf2321fe
1372024-06-20T21:29:46.865ZDEBGoximeter (collection-task): collected results from producer address = [fd00:1122:3344:101::1]:8001 collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = df63a4fe-3119-43c9-b4de-7606cf2321fe
1382024-06-20T21:29:47.864ZDEBGoximeter (collection-task): collecting from producer address = [fd00:1122:3344:101::1]:8001 collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d producer_id = df63a4fe-3119-43c9-b4de-7606cf2321fe
1392024-06-20T21:29:47.865ZDEBGoximeter (collection-task): collected results from producer address = [fd00:1122:3344:101::1]:8001 collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = df63a4fe-3119-43c9-b4de-7606cf2321fe
1402024-06-20T21:29:48.864ZDEBGoximeter (collection-task): collecting from producer address = [fd00:1122:3344:101::1]:8001 collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d producer_id = df63a4fe-3119-43c9-b4de-7606cf2321fe
1412024-06-20T21:29:48.865ZDEBGoximeter (collection-task): collected results from producer address = [fd00:1122:3344:101::1]:8001 collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = df63a4fe-3119-43c9-b4de-7606cf2321fe
1422024-06-20T21:29:49.865ZDEBGoximeter (collection-task): collecting from producer address = [fd00:1122:3344:101::1]:8001 collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d producer_id = df63a4fe-3119-43c9-b4de-7606cf2321fe
1432024-06-20T21:29:49.865ZDEBGoximeter (collection-task): collected results from producer address = [fd00:1122:3344:101::1]:8001 collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = df63a4fe-3119-43c9-b4de-7606cf2321fe
1442024-06-20T21:29:49.991ZDEBGoximeter (results-sink): inserting 130 samples into database collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d
1452024-06-20T21:29:49.991ZDEBGoximeter (clickhouse-client): unrolling 130 total samples collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d id = 564220a6-751a-473a-9a3b-358bf07cde65
1462024-06-20T21:29:49.999ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d id = 564220a6-751a-473a-9a3b-358bf07cde65 n_rows = 26 table_name = oximeter.fields_string
1472024-06-20T21:29:50.001ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d id = 564220a6-751a-473a-9a3b-358bf07cde65 n_rows = 28 table_name = oximeter.fields_uuid
1482024-06-20T21:29:50.004ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d id = 564220a6-751a-473a-9a3b-358bf07cde65 n_rows = 100 table_name = oximeter.measurements_cumulativeu64
1492024-06-20T21:29:50.006ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d id = 564220a6-751a-473a-9a3b-358bf07cde65 n_rows = 30 table_name = oximeter.measurements_u64
1502024-06-20T21:29:50.865ZDEBGoximeter (collection-task): collecting from producer address = [fd00:1122:3344:101::1]:8001 collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d producer_id = df63a4fe-3119-43c9-b4de-7606cf2321fe
1512024-06-20T21:29:50.866ZDEBGoximeter (collection-task): collected results from producer address = [fd00:1122:3344:101::1]:8001 collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = df63a4fe-3119-43c9-b4de-7606cf2321fe
1522024-06-20T21:29:51.772ZINFOoximeter (dropshot): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::d]:12223 remote_addr = [fd00:1122:3344:101::b]:55153
1532024-06-20T21:29:51.772ZDEBGoximeter (oximeter-agent): registered new metric producer address = [fd00:1122:3344:101::b]:33328 collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d producer_id = 8f64d0bd-6f7c-418c-a532-2ff9143ec947
1542024-06-20T21:29:51.772ZINFOoximeter (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 128 local_addr = [fd00:1122:3344:101::d]:12223 method = POST remote_addr = [fd00:1122:3344:101::b]:55153 req_id = a3b0f796-94fb-4d36-9eb8-536e748cd054 response_code = 204 uri = /producers
1552024-06-20T21:29:51.805ZDEBGoximeter (collection-task): starting oximeter collection task address = [fd00:1122:3344:101::b]:33328 collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d interval = 10s producer_id = 8f64d0bd-6f7c-418c-a532-2ff9143ec947
1562024-06-20T21:29:51.865ZDEBGoximeter (collection-task): collecting from producer address = [fd00:1122:3344:101::1]:8001 collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d producer_id = df63a4fe-3119-43c9-b4de-7606cf2321fe
1572024-06-20T21:29:51.866ZDEBGoximeter (collection-task): collected results from producer address = [fd00:1122:3344:101::1]:8001 collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = df63a4fe-3119-43c9-b4de-7606cf2321fe
1582024-06-20T21:29:52.373ZDEBGoximeter (collection-task): collecting from producer address = [fd00:1122:3344:101::a]:62763 collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d producer_id = 4c9d7efe-dded-4bfd-b6fb-594f5dd61d1c
1592024-06-20T21:29:52.375ZDEBGoximeter (collection-task): collected results from producer address = [fd00:1122:3344:101::a]:62763 collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d n_results = 5 producer_id = 4c9d7efe-dded-4bfd-b6fb-594f5dd61d1c
1602024-06-20T21:29:52.503ZINFOoximeter (dropshot): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::d]:12223 remote_addr = [fd00:1122:3344:101::c]:37471
1612024-06-20T21:29:52.503ZDEBGoximeter (oximeter-agent): registered new metric producer address = [fd00:1122:3344:101::c]:44151 collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d producer_id = 65b469d2-6eb7-4e46-8276-ad6d27b67bf9
1622024-06-20T21:29:52.503ZINFOoximeter (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 120 local_addr = [fd00:1122:3344:101::d]:12223 method = POST remote_addr = [fd00:1122:3344:101::c]:37471 req_id = 39f5cc69-3b12-4e5e-b173-ba493445c38e response_code = 204 uri = /producers
1632024-06-20T21:29:52.535ZDEBGoximeter (collection-task): starting oximeter collection task address = [fd00:1122:3344:101::c]:44151 collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d interval = 10s producer_id = 65b469d2-6eb7-4e46-8276-ad6d27b67bf9
1642024-06-20T21:29:52.864ZDEBGoximeter (collection-task): collecting from producer address = [fd00:1122:3344:101::1]:8001 collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d producer_id = df63a4fe-3119-43c9-b4de-7606cf2321fe
1652024-06-20T21:29:52.865ZDEBGoximeter (collection-task): collected results from producer address = [fd00:1122:3344:101::1]:8001 collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = df63a4fe-3119-43c9-b4de-7606cf2321fe
1662024-06-20T21:29:53.864ZDEBGoximeter (collection-task): collecting from producer address = [fd00:1122:3344:101::1]:8001 collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d producer_id = df63a4fe-3119-43c9-b4de-7606cf2321fe
1672024-06-20T21:29:53.865ZDEBGoximeter (collection-task): collected results from producer address = [fd00:1122:3344:101::1]:8001 collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = df63a4fe-3119-43c9-b4de-7606cf2321fe
1682024-06-20T21:29:54.865ZDEBGoximeter (collection-task): collecting from producer address = [fd00:1122:3344:101::1]:8001 collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d producer_id = df63a4fe-3119-43c9-b4de-7606cf2321fe
1692024-06-20T21:29:54.866ZDEBGoximeter (collection-task): collected results from producer address = [fd00:1122:3344:101::1]:8001 collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = df63a4fe-3119-43c9-b4de-7606cf2321fe
1702024-06-20T21:29:54.991ZDEBGoximeter (results-sink): inserting 151 samples into database collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d
1712024-06-20T21:29:54.991ZDEBGoximeter (clickhouse-client): unrolling 151 total samples collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d id = 564220a6-751a-473a-9a3b-358bf07cde65
1722024-06-20T21:29:55.001ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d id = 564220a6-751a-473a-9a3b-358bf07cde65 n_rows = 21 table_name = oximeter.fields_i64
1732024-06-20T21:29:55.003ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d id = 564220a6-751a-473a-9a3b-358bf07cde65 n_rows = 89 table_name = oximeter.fields_string
1742024-06-20T21:29:55.005ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d id = 564220a6-751a-473a-9a3b-358bf07cde65 n_rows = 49 table_name = oximeter.fields_uuid
1752024-06-20T21:29:55.007ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d id = 564220a6-751a-473a-9a3b-358bf07cde65 n_rows = 100 table_name = oximeter.measurements_cumulativeu64
1762024-06-20T21:29:55.010ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d id = 564220a6-751a-473a-9a3b-358bf07cde65 n_rows = 21 table_name = oximeter.measurements_histogramf64
1772024-06-20T21:29:55.012ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d id = 564220a6-751a-473a-9a3b-358bf07cde65 n_rows = 30 table_name = oximeter.measurements_u64
1782024-06-20T21:29:55.865ZDEBGoximeter (collection-task): collecting from producer address = [fd00:1122:3344:101::1]:8001 collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d producer_id = df63a4fe-3119-43c9-b4de-7606cf2321fe
1792024-06-20T21:29:55.865ZDEBGoximeter (collection-task): collected results from producer address = [fd00:1122:3344:101::1]:8001 collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = df63a4fe-3119-43c9-b4de-7606cf2321fe
1802024-06-20T21:29:56.864ZDEBGoximeter (collection-task): collecting from producer address = [fd00:1122:3344:101::1]:8001 collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d producer_id = df63a4fe-3119-43c9-b4de-7606cf2321fe
1812024-06-20T21:29:56.865ZDEBGoximeter (collection-task): collected results from producer address = [fd00:1122:3344:101::1]:8001 collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = df63a4fe-3119-43c9-b4de-7606cf2321fe
1822024-06-20T21:29:57.864ZDEBGoximeter (collection-task): collecting from producer address = [fd00:1122:3344:101::1]:8001 collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d producer_id = df63a4fe-3119-43c9-b4de-7606cf2321fe
1832024-06-20T21:29:57.865ZDEBGoximeter (collection-task): collected results from producer address = [fd00:1122:3344:101::1]:8001 collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = df63a4fe-3119-43c9-b4de-7606cf2321fe
1842024-06-20T21:29:58.865ZDEBGoximeter (collection-task): collecting from producer address = [fd00:1122:3344:101::1]:8001 collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d producer_id = df63a4fe-3119-43c9-b4de-7606cf2321fe
1852024-06-20T21:29:58.865ZDEBGoximeter (collection-task): collected results from producer address = [fd00:1122:3344:101::1]:8001 collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = df63a4fe-3119-43c9-b4de-7606cf2321fe
1862024-06-20T21:29:59.758ZINFOoximeter (dropshot): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::d]:12223 remote_addr = [fd00:1122:3344:101::a]:58627
1872024-06-20T21:29:59.758ZDEBGoximeter (oximeter-agent): registered new metric producer address = [fd00:1122:3344:101::2]:4677 collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d producer_id = e789eeb5-b9d0-4486-9315-eafe6d7ad753
1882024-06-20T21:29:59.759ZINFOoximeter (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 103 local_addr = [fd00:1122:3344:101::d]:12223 method = POST remote_addr = [fd00:1122:3344:101::a]:58627 req_id = 96ee9d3d-4612-4119-987a-2b1ce973201e response_code = 204 uri = /producers
1892024-06-20T21:29:59.791ZDEBGoximeter (collection-task): starting oximeter collection task address = [fd00:1122:3344:101::2]:4677 collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d interval = 1s producer_id = e789eeb5-b9d0-4486-9315-eafe6d7ad753
1902024-06-20T21:29:59.865ZDEBGoximeter (collection-task): collecting from producer address = [fd00:1122:3344:101::1]:8001 collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d producer_id = df63a4fe-3119-43c9-b4de-7606cf2321fe
1912024-06-20T21:29:59.865ZDEBGoximeter (collection-task): collected results from producer address = [fd00:1122:3344:101::1]:8001 collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = df63a4fe-3119-43c9-b4de-7606cf2321fe
1922024-06-20T21:29:59.992ZDEBGoximeter (results-sink): inserting 130 samples into database collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d
1932024-06-20T21:29:59.992ZDEBGoximeter (clickhouse-client): unrolling 130 total samples collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d id = 564220a6-751a-473a-9a3b-358bf07cde65
1942024-06-20T21:29:59.999ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d id = 564220a6-751a-473a-9a3b-358bf07cde65 n_rows = 26 table_name = oximeter.fields_string
1952024-06-20T21:30:00.002ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d id = 564220a6-751a-473a-9a3b-358bf07cde65 n_rows = 28 table_name = oximeter.fields_uuid
1962024-06-20T21:30:00.004ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d id = 564220a6-751a-473a-9a3b-358bf07cde65 n_rows = 100 table_name = oximeter.measurements_cumulativeu64
1972024-06-20T21:30:00.006ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d id = 564220a6-751a-473a-9a3b-358bf07cde65 n_rows = 30 table_name = oximeter.measurements_u64
1982024-06-20T21:30:00.790ZDEBGoximeter (collection-task): collecting from producer address = [fd00:1122:3344:101::2]:4677 collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d producer_id = e789eeb5-b9d0-4486-9315-eafe6d7ad753
1992024-06-20T21:30:00.807ZDEBGoximeter (collection-task): collected results from producer address = [fd00:1122:3344:101::2]:4677 collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = e789eeb5-b9d0-4486-9315-eafe6d7ad753
2002024-06-20T21:30:00.864ZDEBGoximeter (collection-task): collecting from producer address = [fd00:1122:3344:101::1]:8001 collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d producer_id = df63a4fe-3119-43c9-b4de-7606cf2321fe
2012024-06-20T21:30:00.865ZDEBGoximeter (collection-task): collected results from producer address = [fd00:1122:3344:101::1]:8001 collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = df63a4fe-3119-43c9-b4de-7606cf2321fe
2022024-06-20T21:30:01.791ZDEBGoximeter (collection-task): collecting from producer address = [fd00:1122:3344:101::2]:4677 collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d producer_id = e789eeb5-b9d0-4486-9315-eafe6d7ad753
2032024-06-20T21:30:01.793ZDEBGoximeter (collection-task): collected results from producer address = [fd00:1122:3344:101::2]:4677 collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = e789eeb5-b9d0-4486-9315-eafe6d7ad753
2042024-06-20T21:30:01.805ZDEBGoximeter (collection-task): collecting from producer address = [fd00:1122:3344:101::b]:33328 collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d producer_id = 8f64d0bd-6f7c-418c-a532-2ff9143ec947
2052024-06-20T21:30:01.816ZDEBGoximeter (collection-task): collected results from producer address = [fd00:1122:3344:101::b]:33328 collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d n_results = 5 producer_id = 8f64d0bd-6f7c-418c-a532-2ff9143ec947
2062024-06-20T21:30:01.865ZDEBGoximeter (collection-task): collecting from producer address = [fd00:1122:3344:101::1]:8001 collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d producer_id = df63a4fe-3119-43c9-b4de-7606cf2321fe
2072024-06-20T21:30:01.865ZDEBGoximeter (collection-task): collected results from producer address = [fd00:1122:3344:101::1]:8001 collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = df63a4fe-3119-43c9-b4de-7606cf2321fe
2082024-06-20T21:30:02.373ZDEBGoximeter (collection-task): collecting from producer address = [fd00:1122:3344:101::a]:62763 collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d producer_id = 4c9d7efe-dded-4bfd-b6fb-594f5dd61d1c
2092024-06-20T21:30:02.375ZDEBGoximeter (collection-task): collected results from producer address = [fd00:1122:3344:101::a]:62763 collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d n_results = 5 producer_id = 4c9d7efe-dded-4bfd-b6fb-594f5dd61d1c
2102024-06-20T21:30:02.535ZDEBGoximeter (collection-task): collecting from producer address = [fd00:1122:3344:101::c]:44151 collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d producer_id = 65b469d2-6eb7-4e46-8276-ad6d27b67bf9
2112024-06-20T21:30:02.540ZDEBGoximeter (collection-task): collected results from producer address = [fd00:1122:3344:101::c]:44151 collector_id = 83a45644-f784-4b40-a7e1-ec94b0dc35d4 collector_ip = fd00:1122:3344:101::d n_results = 5 producer_id = 65b469d2-6eb7-4e46-8276-ad6d27b67bf9