Max level shown:
1 [ Dec 6 17:42:10 Disabled. ]
2 [ Dec 6 17:42:10 Rereading configuration. ]
3 [ Dec 6 17:42:33 Rereading configuration. ]
4 [ Dec 6 17:42:33 Rereading configuration. ]
5 [ Dec 6 17:42:33 Enabled. ]
6 [ Dec 6 17:42:33 Executing start method ("ctrun -l child -o noorphan,regent /opt/oxide/oximeter/bin/oximeter run /var/svc/manifest/site/oximeter/config.toml --address [fd00:1122:3344:101::d]:12223 --id 358b59f5-3c51-4c63-92f1-1dcada1dc798 &"). ]
7 [ Dec 6 17:42:33 Method "start" exited with status 0. ]
8 note: configured to log to "/dev/stdout"
92023-12-06T17:42:33.532ZDEBGoximeter: registered DTrace probes
102023-12-06T17:42:33.533ZINFOoximeter: starting oximeter server file = oximeter/collector/src/lib.rs:192
112023-12-06T17:42:33.535ZINFOoximeter (DnsResolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = internal-dns/src/resolver.rs:60
122023-12-06T17:42:33.536ZDEBGoximeter: creating ClickHouse client
132023-12-06T17:42:33.536ZDEBGoximeter (DnsResolver): lookup srv dns_name = _clickhouse._tcp.control-plane.oxide.internal
142023-12-06T17:42:33.634ZWARNoximeter (clickhouse-client): oximeter database does not exist, or is out-of-date collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 file = oximeter/db/src/client.rs:658 id = 1cd5687c-e4c2-4772-ab30-533e410c62df
152023-12-06T17:42:33.634ZDEBGoximeter (oximeter-agent): oximeter database does not exist, creating collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798
162023-12-06T17:42:33.641ZINFOoximeter (clickhouse-client): reading db version collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 file = oximeter/db/src/client.rs:596 id = 1cd5687c-e4c2-4772-ab30-533e410c62df
172023-12-06T17:42:33.642ZWARNoximeter (clickhouse-client): oximeter database does not exist, or is out-of-date collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 file = oximeter/db/src/client.rs:658 id = 1cd5687c-e4c2-4772-ab30-533e410c62df
182023-12-06T17:42:33.642ZINFOoximeter (clickhouse-client): read oximeter database version collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 file = oximeter/db/src/client.rs:600 id = 1cd5687c-e4c2-4772-ab30-533e410c62df version = 0
192023-12-06T17:42:33.642ZINFOoximeter (clickhouse-client): wiping and re-initializing oximeter schema collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 file = oximeter/db/src/client.rs:605 id = 1cd5687c-e4c2-4772-ab30-533e410c62df
202023-12-06T17:42:33.642ZDEBGoximeter (clickhouse-client): wiping ClickHouse database collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 id = 1cd5687c-e4c2-4772-ab30-533e410c62df
212023-12-06T17:42:33.642ZDEBGoximeter (clickhouse-client): initializing ClickHouse database collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 id = 1cd5687c-e4c2-4772-ab30-533e410c62df
222023-12-06T17:42:33.788ZINFOoximeter (clickhouse-client): inserting current version collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 file = oximeter/db/src/client.rs:628 id = 1cd5687c-e4c2-4772-ab30-533e410c62df version = 4
232023-12-06T17:42:33.799ZDEBGoximeter (dropshot): registered endpoint local_addr = [fd00:1122:3344:101::d]:12223 method = GET path = /info
242023-12-06T17:42:33.799ZDEBGoximeter (dropshot): registered endpoint local_addr = [fd00:1122:3344:101::d]:12223 method = GET path = /producers
252023-12-06T17:42:33.799ZDEBGoximeter (dropshot): registered endpoint local_addr = [fd00:1122:3344:101::d]:12223 method = POST path = /producers
262023-12-06T17:42:33.799ZDEBGoximeter (dropshot): registered endpoint local_addr = [fd00:1122:3344:101::d]:12223 method = DELETE path = /producers/{producer_id}
272023-12-06T17:42:33.799ZINFOoximeter (dropshot): listening file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:195 local_addr = [fd00:1122:3344:101::d]:12223
282023-12-06T17:42:33.799ZDEBGoximeter (dropshot): successfully registered DTrace USDT probes local_addr = [fd00:1122:3344:101::d]:12223
292023-12-06T17:42:33.831ZDEBGoximeter: contacting nexus
302023-12-06T17:42:33.831ZDEBGoximeter (DnsResolver): lookup_ipv6 srv dns_name = _nexus._tcp.control-plane.oxide.internal
312023-12-06T17:42:33.833ZWARNoximeter: failed to contact nexus, will retry in 294.52878ms error = "error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/collectors): error trying to connect: tcp connect error: Connection refused (os error 146)" file = oximeter/collector/src/lib.rs:271
322023-12-06T17:42:34.129ZDEBGoximeter: contacting nexus
332023-12-06T17:42:34.129ZDEBGoximeter (DnsResolver): lookup_ipv6 srv dns_name = _nexus._tcp.control-plane.oxide.internal
342023-12-06T17:42:34.130ZWARNoximeter: failed to contact nexus, will retry in 445.859341ms error = "error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/collectors): error trying to connect: tcp connect error: Connection refused (os error 146)" file = oximeter/collector/src/lib.rs:271
352023-12-06T17:42:34.578ZDEBGoximeter: contacting nexus
362023-12-06T17:42:34.578ZDEBGoximeter (DnsResolver): lookup_ipv6 srv dns_name = _nexus._tcp.control-plane.oxide.internal
372023-12-06T17:42:34.578ZWARNoximeter: failed to contact nexus, will retry in 810.385557ms error = "error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/collectors): error trying to connect: tcp connect error: Connection refused (os error 146)" file = oximeter/collector/src/lib.rs:271
382023-12-06T17:42:35.390ZDEBGoximeter: contacting nexus
392023-12-06T17:42:35.390ZDEBGoximeter (DnsResolver): lookup_ipv6 srv dns_name = _nexus._tcp.control-plane.oxide.internal
402023-12-06T17:42:35.391ZWARNoximeter: failed to contact nexus, will retry in 1.230185962s error = "error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/collectors): error trying to connect: tcp connect error: Connection refused (os error 146)" file = oximeter/collector/src/lib.rs:271
412023-12-06T17:42:36.622ZDEBGoximeter: contacting nexus
422023-12-06T17:42:36.622ZDEBGoximeter (DnsResolver): lookup_ipv6 srv dns_name = _nexus._tcp.control-plane.oxide.internal
432023-12-06T17:42:36.623ZWARNoximeter: failed to contact nexus, will retry in 4.38119019s error = "error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/collectors): error trying to connect: tcp connect error: Connection refused (os error 146)" file = oximeter/collector/src/lib.rs:271
442023-12-06T17:42:41.005ZDEBGoximeter: contacting nexus
452023-12-06T17:42:41.005ZDEBGoximeter (DnsResolver): lookup_ipv6 srv dns_name = _nexus._tcp.control-plane.oxide.internal
462023-12-06T17:42:41.006ZWARNoximeter: failed to contact nexus, will retry in 5.474074398s error = "error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/collectors): error trying to connect: tcp connect error: Connection refused (os error 146)" file = oximeter/collector/src/lib.rs:271
472023-12-06T17:42:46.481ZDEBGoximeter: contacting nexus
482023-12-06T17:42:46.481ZDEBGoximeter (DnsResolver): lookup_ipv6 srv dns_name = _nexus._tcp.control-plane.oxide.internal
492023-12-06T17:42:46.561ZINFOoximeter: oximeter registered with nexus file = oximeter/collector/src/lib.rs:285 id = 358b59f5-3c51-4c63-92f1-1dcada1dc798
502023-12-06T17:43:19.866ZINFOoximeter (dropshot): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:769 local_addr = [fd00:1122:3344:101::d]:12223 remote_addr = [fd00:1122:3344:101::c]:62348
512023-12-06T17:43:19.873ZDEBGoximeter (oximeter-agent): registered new metric producer address = [fd00:1122:3344:101::1]:12345 collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 producer_id = 1e40a2ae-ffda-4a94-95f6-839f3349172b
522023-12-06T17:43:19.873ZINFOoximeter (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:853 latency_us = 6696 local_addr = [fd00:1122:3344:101::d]:12223 method = POST remote_addr = [fd00:1122:3344:101::c]:62348 req_id = 0ae6ef24-e732-4eee-91ec-c0991d0a6bcb response_code = 204 uri = /producers
532023-12-06T17:43:19.906ZDEBGoximeter (collection-task): starting oximeter collection task collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 interval = 30s producer_id = 1e40a2ae-ffda-4a94-95f6-839f3349172b
542023-12-06T17:43:38.290ZINFOoximeter (dropshot): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:769 local_addr = [fd00:1122:3344:101::d]:12223 remote_addr = [fd00:1122:3344:101::c]:53604
552023-12-06T17:43:38.291ZDEBGoximeter (oximeter-agent): registered new metric producer address = [fd00:1122:3344:101::c]:12221 collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 producer_id = 1dfcdbef-d660-4977-8363-0facffed5dd7
562023-12-06T17:43:38.291ZINFOoximeter (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:853 latency_us = 599 local_addr = [fd00:1122:3344:101::d]:12223 method = POST remote_addr = [fd00:1122:3344:101::c]:53604 req_id = faa0d493-8379-40e4-b0d8-372d35d46c53 response_code = 204 uri = /producers
572023-12-06T17:43:38.323ZDEBGoximeter (collection-task): starting oximeter collection task collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 interval = 10s producer_id = 1dfcdbef-d660-4977-8363-0facffed5dd7
582023-12-06T17:43:46.208ZINFOoximeter (dropshot): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:769 local_addr = [fd00:1122:3344:101::d]:12223 remote_addr = [fd00:1122:3344:101::b]:46713
592023-12-06T17:43:46.209ZDEBGoximeter (oximeter-agent): registered new metric producer address = [fd00:1122:3344:101::b]:12221 collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 producer_id = 9a25727f-e844-4dd1-b1b4-978eca7f5c90
602023-12-06T17:43:46.209ZINFOoximeter (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:853 latency_us = 708 local_addr = [fd00:1122:3344:101::d]:12223 method = POST remote_addr = [fd00:1122:3344:101::b]:46713 req_id = f3fde1f2-8825-4ee5-9d8d-3587cb60adfd response_code = 204 uri = /producers
612023-12-06T17:43:46.213ZINFOoximeter (dropshot): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:769 local_addr = [fd00:1122:3344:101::d]:12223 remote_addr = [fd00:1122:3344:101::a]:64250
622023-12-06T17:43:46.213ZDEBGoximeter (oximeter-agent): registered new metric producer address = [fd00:1122:3344:101::a]:12221 collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 producer_id = d2016ddd-0efc-4439-89aa-2f166d289e6f
632023-12-06T17:43:46.213ZINFOoximeter (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:853 latency_us = 92 local_addr = [fd00:1122:3344:101::d]:12223 method = POST remote_addr = [fd00:1122:3344:101::a]:64250 req_id = 751cda0c-187c-46da-98f4-36e62cc900d2 response_code = 204 uri = /producers
642023-12-06T17:43:46.321ZDEBGoximeter (collection-task): starting oximeter collection task collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 interval = 10s producer_id = 9a25727f-e844-4dd1-b1b4-978eca7f5c90
652023-12-06T17:43:46.327ZDEBGoximeter (collection-task): starting oximeter collection task collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 interval = 10s producer_id = d2016ddd-0efc-4439-89aa-2f166d289e6f
662023-12-06T17:43:48.324ZDEBGoximeter (collection-task): collecting from producer collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 producer_id = 1dfcdbef-d660-4977-8363-0facffed5dd7
672023-12-06T17:43:48.328ZDEBGoximeter (collection-task): collected results from producer collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 n_results = 4 producer_id = 1dfcdbef-d660-4977-8363-0facffed5dd7
682023-12-06T17:43:48.798ZDEBGoximeter (results-sink): inserting 57 samples into database collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798
692023-12-06T17:43:48.798ZDEBGoximeter (clickhouse-client): unrolling 57 total samples collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 id = 1cd5687c-e4c2-4772-ab30-533e410c62df
702023-12-06T17:43:48.815ZDEBGoximeter (clickhouse-client): retrieving timeseries schema from database collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 id = 1cd5687c-e4c2-4772-ab30-533e410c62df
712023-12-06T17:43:48.818ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 id = 1cd5687c-e4c2-4772-ab30-533e410c62df schema = {"timeseries_name":"http_service:request_latency_histogram","fields.name":["id","method","name","route","status_code"],"fields.type":["Uuid","String","String","String","I64"],"fields.source":["Target","Metric","Target","Metric","Metric"],"datum_type":"HistogramF64","created":"2023-12-06 17:43:48.800642867"} timeseries_name = http_service:request_latency_histogram
722023-12-06T17:43:48.822ZDEBGoximeter (clickhouse-client): retrieving timeseries schema from database collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 id = 1cd5687c-e4c2-4772-ab30-533e410c62df
732023-12-06T17:43:48.827ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 id = 1cd5687c-e4c2-4772-ab30-533e410c62df schema = {"timeseries_name":"collection_target:cpus_provisioned","fields.name":["id"],"fields.type":["Uuid"],"fields.source":["Target"],"datum_type":"I64","created":"2023-12-06 17:43:48.822781948"} timeseries_name = collection_target:cpus_provisioned
742023-12-06T17:43:48.827ZDEBGoximeter (clickhouse-client): retrieving timeseries schema from database collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 id = 1cd5687c-e4c2-4772-ab30-533e410c62df
752023-12-06T17:43:48.829ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 id = 1cd5687c-e4c2-4772-ab30-533e410c62df schema = {"timeseries_name":"collection_target:ram_provisioned","fields.name":["id"],"fields.type":["Uuid"],"fields.source":["Target"],"datum_type":"I64","created":"2023-12-06 17:43:48.827564504"} timeseries_name = collection_target:ram_provisioned
762023-12-06T17:43:48.829ZDEBGoximeter (clickhouse-client): retrieving timeseries schema from database collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 id = 1cd5687c-e4c2-4772-ab30-533e410c62df
772023-12-06T17:43:48.831ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 id = 1cd5687c-e4c2-4772-ab30-533e410c62df schema = {"timeseries_name":"collection_target:virtual_disk_space_provisioned","fields.name":["id"],"fields.type":["Uuid"],"fields.source":["Target"],"datum_type":"I64","created":"2023-12-06 17:43:48.829489292"} timeseries_name = collection_target:virtual_disk_space_provisioned
782023-12-06T17:43:48.832ZDEBGoximeter (clickhouse-client): inserting 4 new timeseries schema collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 id = 1cd5687c-e4c2-4772-ab30-533e410c62df
792023-12-06T17:43:48.840ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 id = 1cd5687c-e4c2-4772-ab30-533e410c62df n_rows = 36 table_name = oximeter.fields_i64
802023-12-06T17:43:48.841ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 id = 1cd5687c-e4c2-4772-ab30-533e410c62df n_rows = 108 table_name = oximeter.fields_string
812023-12-06T17:43:48.846ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 id = 1cd5687c-e4c2-4772-ab30-533e410c62df n_rows = 48 table_name = oximeter.fields_uuid
822023-12-06T17:43:48.849ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 id = 1cd5687c-e4c2-4772-ab30-533e410c62df n_rows = 36 table_name = oximeter.measurements_histogramf64
832023-12-06T17:43:48.851ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 id = 1cd5687c-e4c2-4772-ab30-533e410c62df n_rows = 21 table_name = oximeter.measurements_i64
842023-12-06T17:43:49.906ZDEBGoximeter (collection-task): collecting from producer collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 producer_id = 1e40a2ae-ffda-4a94-95f6-839f3349172b
852023-12-06T17:43:49.910ZDEBGoximeter (collection-task): collected results from producer collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 n_results = 1 producer_id = 1e40a2ae-ffda-4a94-95f6-839f3349172b
862023-12-06T17:43:53.798ZDEBGoximeter (results-sink): inserting 192 samples into database collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798
872023-12-06T17:43:53.798ZDEBGoximeter (clickhouse-client): unrolling 192 total samples collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 id = 1cd5687c-e4c2-4772-ab30-533e410c62df
882023-12-06T17:43:53.798ZDEBGoximeter (clickhouse-client): retrieving timeseries schema from database collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 id = 1cd5687c-e4c2-4772-ab30-533e410c62df
892023-12-06T17:43:53.805ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 id = 1cd5687c-e4c2-4772-ab30-533e410c62df schema = {"timeseries_name":"physical_data_link:errors_received","fields.name":["hostname","link_name","rack_id","serial","sled_id"],"fields.type":["String","String","Uuid","String","Uuid"],"fields.source":["Target","Target","Target","Target","Target"],"datum_type":"CumulativeU64","created":"2023-12-06 17:43:53.798776410"} timeseries_name = physical_data_link:errors_received
902023-12-06T17:43:53.805ZDEBGoximeter (clickhouse-client): retrieving timeseries schema from database collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 id = 1cd5687c-e4c2-4772-ab30-533e410c62df
912023-12-06T17:43:53.808ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 id = 1cd5687c-e4c2-4772-ab30-533e410c62df schema = {"timeseries_name":"physical_data_link:errors_sent","fields.name":["hostname","link_name","rack_id","serial","sled_id"],"fields.type":["String","String","Uuid","String","Uuid"],"fields.source":["Target","Target","Target","Target","Target"],"datum_type":"CumulativeU64","created":"2023-12-06 17:43:53.805622136"} timeseries_name = physical_data_link:errors_sent
922023-12-06T17:43:53.808ZDEBGoximeter (clickhouse-client): retrieving timeseries schema from database collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 id = 1cd5687c-e4c2-4772-ab30-533e410c62df
932023-12-06T17:43:53.810ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 id = 1cd5687c-e4c2-4772-ab30-533e410c62df schema = {"timeseries_name":"physical_data_link:bytes_received","fields.name":["hostname","link_name","rack_id","serial","sled_id"],"fields.type":["String","String","Uuid","String","Uuid"],"fields.source":["Target","Target","Target","Target","Target"],"datum_type":"CumulativeU64","created":"2023-12-06 17:43:53.808202108"} timeseries_name = physical_data_link:bytes_received
942023-12-06T17:43:53.810ZDEBGoximeter (clickhouse-client): retrieving timeseries schema from database collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 id = 1cd5687c-e4c2-4772-ab30-533e410c62df
952023-12-06T17:43:53.813ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 id = 1cd5687c-e4c2-4772-ab30-533e410c62df schema = {"timeseries_name":"physical_data_link:packets_received","fields.name":["hostname","link_name","rack_id","serial","sled_id"],"fields.type":["String","String","Uuid","String","Uuid"],"fields.source":["Target","Target","Target","Target","Target"],"datum_type":"CumulativeU64","created":"2023-12-06 17:43:53.810661657"} timeseries_name = physical_data_link:packets_received
962023-12-06T17:43:53.813ZDEBGoximeter (clickhouse-client): retrieving timeseries schema from database collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 id = 1cd5687c-e4c2-4772-ab30-533e410c62df
972023-12-06T17:43:53.815ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 id = 1cd5687c-e4c2-4772-ab30-533e410c62df schema = {"timeseries_name":"physical_data_link:bytes_sent","fields.name":["hostname","link_name","rack_id","serial","sled_id"],"fields.type":["String","String","Uuid","String","Uuid"],"fields.source":["Target","Target","Target","Target","Target"],"datum_type":"CumulativeU64","created":"2023-12-06 17:43:53.813309260"} timeseries_name = physical_data_link:bytes_sent
982023-12-06T17:43:53.815ZDEBGoximeter (clickhouse-client): retrieving timeseries schema from database collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 id = 1cd5687c-e4c2-4772-ab30-533e410c62df
992023-12-06T17:43:53.818ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 id = 1cd5687c-e4c2-4772-ab30-533e410c62df schema = {"timeseries_name":"physical_data_link:packets_sent","fields.name":["hostname","link_name","rack_id","serial","sled_id"],"fields.type":["String","String","Uuid","String","Uuid"],"fields.source":["Target","Target","Target","Target","Target"],"datum_type":"CumulativeU64","created":"2023-12-06 17:43:53.815873111"} timeseries_name = physical_data_link:packets_sent
1002023-12-06T17:43:53.827ZDEBGoximeter (clickhouse-client): inserting 6 new timeseries schema collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 id = 1cd5687c-e4c2-4772-ab30-533e410c62df
1012023-12-06T17:43:53.831ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 id = 1cd5687c-e4c2-4772-ab30-533e410c62df n_rows = 36 table_name = oximeter.fields_string
1022023-12-06T17:43:53.832ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 id = 1cd5687c-e4c2-4772-ab30-533e410c62df n_rows = 24 table_name = oximeter.fields_uuid
1032023-12-06T17:43:53.834ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 id = 1cd5687c-e4c2-4772-ab30-533e410c62df n_rows = 192 table_name = oximeter.measurements_cumulativeu64
1042023-12-06T17:43:56.321ZDEBGoximeter (collection-task): collecting from producer collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 producer_id = 9a25727f-e844-4dd1-b1b4-978eca7f5c90
1052023-12-06T17:43:56.323ZDEBGoximeter (collection-task): collected results from producer collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 n_results = 4 producer_id = 9a25727f-e844-4dd1-b1b4-978eca7f5c90
1062023-12-06T17:43:56.327ZDEBGoximeter (collection-task): collecting from producer collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 producer_id = d2016ddd-0efc-4439-89aa-2f166d289e6f
1072023-12-06T17:43:56.329ZDEBGoximeter (collection-task): collected results from producer collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 n_results = 4 producer_id = d2016ddd-0efc-4439-89aa-2f166d289e6f
1082023-12-06T17:43:56.983ZINFOoximeter (dropshot): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:769 local_addr = [fd00:1122:3344:101::d]:12223 remote_addr = [fd00:1122:3344:101::a]:64736
1092023-12-06T17:43:56.983ZDEBGoximeter (oximeter-agent): registered new metric producer address = [fd00:1122:3344:101::22]:37426 collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 producer_id = 798fd184-57e7-4c63-a223-b0a810e00937
1102023-12-06T17:43:56.983ZINFOoximeter (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:853 latency_us = 565 local_addr = [fd00:1122:3344:101::d]:12223 method = POST remote_addr = [fd00:1122:3344:101::a]:64736 req_id = b286579c-e9b1-4fd6-9c08-c6d867d07349 response_code = 204 uri = /producers
1112023-12-06T17:43:57.016ZDEBGoximeter (collection-task): starting oximeter collection task collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 interval = 30s producer_id = 798fd184-57e7-4c63-a223-b0a810e00937
1122023-12-06T17:43:58.324ZDEBGoximeter (collection-task): collecting from producer collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 producer_id = 1dfcdbef-d660-4977-8363-0facffed5dd7
1132023-12-06T17:43:58.327ZDEBGoximeter (collection-task): collected results from producer collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 n_results = 4 producer_id = 1dfcdbef-d660-4977-8363-0facffed5dd7
1142023-12-06T17:43:58.798ZDEBGoximeter (results-sink): inserting 57 samples into database collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798
1152023-12-06T17:43:58.798ZDEBGoximeter (clickhouse-client): unrolling 57 total samples collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 id = 1cd5687c-e4c2-4772-ab30-533e410c62df
1162023-12-06T17:43:58.804ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 id = 1cd5687c-e4c2-4772-ab30-533e410c62df n_rows = 45 table_name = oximeter.fields_i64
1172023-12-06T17:43:58.806ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 id = 1cd5687c-e4c2-4772-ab30-533e410c62df n_rows = 135 table_name = oximeter.fields_string
1182023-12-06T17:43:58.807ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 id = 1cd5687c-e4c2-4772-ab30-533e410c62df n_rows = 57 table_name = oximeter.fields_uuid
1192023-12-06T17:43:58.809ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 id = 1cd5687c-e4c2-4772-ab30-533e410c62df n_rows = 45 table_name = oximeter.measurements_histogramf64
1202023-12-06T17:43:58.810ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 id = 1cd5687c-e4c2-4772-ab30-533e410c62df n_rows = 12 table_name = oximeter.measurements_i64
1212023-12-06T17:44:06.321ZDEBGoximeter (collection-task): collecting from producer collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 producer_id = 9a25727f-e844-4dd1-b1b4-978eca7f5c90
1222023-12-06T17:44:06.321ZDEBGoximeter (collection-task): collected results from producer collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 n_results = 4 producer_id = 9a25727f-e844-4dd1-b1b4-978eca7f5c90
1232023-12-06T17:44:06.327ZDEBGoximeter (collection-task): collecting from producer collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 producer_id = d2016ddd-0efc-4439-89aa-2f166d289e6f
1242023-12-06T17:44:06.328ZDEBGoximeter (collection-task): collected results from producer collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 n_results = 4 producer_id = d2016ddd-0efc-4439-89aa-2f166d289e6f
1252023-12-06T17:44:08.324ZDEBGoximeter (collection-task): collecting from producer collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 producer_id = 1dfcdbef-d660-4977-8363-0facffed5dd7
1262023-12-06T17:44:08.327ZDEBGoximeter (collection-task): collected results from producer collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 n_results = 4 producer_id = 1dfcdbef-d660-4977-8363-0facffed5dd7
1272023-12-06T17:44:08.797ZDEBGoximeter (results-sink): inserting 47 samples into database collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798
1282023-12-06T17:44:08.797ZDEBGoximeter (clickhouse-client): unrolling 47 total samples collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 id = 1cd5687c-e4c2-4772-ab30-533e410c62df
1292023-12-06T17:44:08.803ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 id = 1cd5687c-e4c2-4772-ab30-533e410c62df n_rows = 47 table_name = oximeter.fields_i64
1302023-12-06T17:44:08.805ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 id = 1cd5687c-e4c2-4772-ab30-533e410c62df n_rows = 141 table_name = oximeter.fields_string
1312023-12-06T17:44:08.806ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 id = 1cd5687c-e4c2-4772-ab30-533e410c62df n_rows = 47 table_name = oximeter.fields_uuid
1322023-12-06T17:44:08.808ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 id = 1cd5687c-e4c2-4772-ab30-533e410c62df n_rows = 47 table_name = oximeter.measurements_histogramf64
1332023-12-06T17:44:16.321ZDEBGoximeter (collection-task): collecting from producer collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 producer_id = 9a25727f-e844-4dd1-b1b4-978eca7f5c90
1342023-12-06T17:44:16.322ZDEBGoximeter (collection-task): collected results from producer collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 n_results = 4 producer_id = 9a25727f-e844-4dd1-b1b4-978eca7f5c90
1352023-12-06T17:44:16.327ZDEBGoximeter (collection-task): collecting from producer collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 producer_id = d2016ddd-0efc-4439-89aa-2f166d289e6f
1362023-12-06T17:44:16.327ZDEBGoximeter (collection-task): collected results from producer collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 n_results = 4 producer_id = d2016ddd-0efc-4439-89aa-2f166d289e6f
1372023-12-06T17:44:18.324ZDEBGoximeter (collection-task): collecting from producer collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 producer_id = 1dfcdbef-d660-4977-8363-0facffed5dd7
1382023-12-06T17:44:18.327ZDEBGoximeter (collection-task): collected results from producer collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 n_results = 4 producer_id = 1dfcdbef-d660-4977-8363-0facffed5dd7
1392023-12-06T17:44:18.798ZDEBGoximeter (results-sink): inserting 47 samples into database collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798
1402023-12-06T17:44:18.798ZDEBGoximeter (clickhouse-client): unrolling 47 total samples collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 id = 1cd5687c-e4c2-4772-ab30-533e410c62df
1412023-12-06T17:44:18.803ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 id = 1cd5687c-e4c2-4772-ab30-533e410c62df n_rows = 47 table_name = oximeter.fields_i64
1422023-12-06T17:44:18.805ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 id = 1cd5687c-e4c2-4772-ab30-533e410c62df n_rows = 141 table_name = oximeter.fields_string
1432023-12-06T17:44:18.806ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 id = 1cd5687c-e4c2-4772-ab30-533e410c62df n_rows = 47 table_name = oximeter.fields_uuid
1442023-12-06T17:44:18.808ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 id = 1cd5687c-e4c2-4772-ab30-533e410c62df n_rows = 47 table_name = oximeter.measurements_histogramf64
1452023-12-06T17:44:19.906ZDEBGoximeter (collection-task): collecting from producer collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 producer_id = 1e40a2ae-ffda-4a94-95f6-839f3349172b
1462023-12-06T17:44:19.906ZDEBGoximeter (collection-task): collected results from producer collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 n_results = 1 producer_id = 1e40a2ae-ffda-4a94-95f6-839f3349172b
1472023-12-06T17:44:19.907ZDEBGoximeter (collection-task): reporting oximeter self-collection statistics collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 producer_id = 1e40a2ae-ffda-4a94-95f6-839f3349172b
1482023-12-06T17:44:23.798ZDEBGoximeter (results-sink): inserting 37 samples into database collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798
1492023-12-06T17:44:23.798ZDEBGoximeter (clickhouse-client): unrolling 37 total samples collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 id = 1cd5687c-e4c2-4772-ab30-533e410c62df
1502023-12-06T17:44:23.800ZDEBGoximeter (clickhouse-client): retrieving timeseries schema from database collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 id = 1cd5687c-e4c2-4772-ab30-533e410c62df
1512023-12-06T17:44:23.804ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 id = 1cd5687c-e4c2-4772-ab30-533e410c62df schema = {"timeseries_name":"oximeter_collector:collections","fields.name":["base_route","collector_id","collector_ip","collector_port","producer_id","producer_ip","producer_port"],"fields.type":["String","Uuid","IpAddr","U16","Uuid","IpAddr","U16"],"fields.source":["Metric","Target","Target","Target","Metric","Metric","Metric"],"datum_type":"CumulativeU64","created":"2023-12-06 17:44:23.800179980"} timeseries_name = oximeter_collector:collections
1522023-12-06T17:44:23.804ZDEBGoximeter (clickhouse-client): inserting 1 new timeseries schema collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 id = 1cd5687c-e4c2-4772-ab30-533e410c62df
1532023-12-06T17:44:23.807ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 id = 1cd5687c-e4c2-4772-ab30-533e410c62df n_rows = 2 table_name = oximeter.fields_ipaddr
1542023-12-06T17:44:23.808ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 id = 1cd5687c-e4c2-4772-ab30-533e410c62df n_rows = 37 table_name = oximeter.fields_string
1552023-12-06T17:44:23.810ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 id = 1cd5687c-e4c2-4772-ab30-533e410c62df n_rows = 2 table_name = oximeter.fields_u16
1562023-12-06T17:44:23.813ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 id = 1cd5687c-e4c2-4772-ab30-533e410c62df n_rows = 26 table_name = oximeter.fields_uuid
1572023-12-06T17:44:23.817ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 id = 1cd5687c-e4c2-4772-ab30-533e410c62df n_rows = 37 table_name = oximeter.measurements_cumulativeu64
1582023-12-06T17:44:26.320ZDEBGoximeter (collection-task): collecting from producer collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 producer_id = 9a25727f-e844-4dd1-b1b4-978eca7f5c90
1592023-12-06T17:44:26.321ZDEBGoximeter (collection-task): collected results from producer collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 n_results = 4 producer_id = 9a25727f-e844-4dd1-b1b4-978eca7f5c90
1602023-12-06T17:44:26.327ZDEBGoximeter (collection-task): collecting from producer collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 producer_id = d2016ddd-0efc-4439-89aa-2f166d289e6f
1612023-12-06T17:44:26.327ZDEBGoximeter (collection-task): collected results from producer collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 n_results = 4 producer_id = d2016ddd-0efc-4439-89aa-2f166d289e6f
1622023-12-06T17:44:27.016ZDEBGoximeter (collection-task): collecting from producer collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 producer_id = 798fd184-57e7-4c63-a223-b0a810e00937
1632023-12-06T17:44:27.017ZDEBGoximeter (collection-task): collected results from producer collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 n_results = 2 producer_id = 798fd184-57e7-4c63-a223-b0a810e00937
1642023-12-06T17:44:28.323ZDEBGoximeter (collection-task): collecting from producer collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 producer_id = 1dfcdbef-d660-4977-8363-0facffed5dd7
1652023-12-06T17:44:28.381ZDEBGoximeter (collection-task): collected results from producer collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 n_results = 4 producer_id = 1dfcdbef-d660-4977-8363-0facffed5dd7
1662023-12-06T17:44:28.798ZDEBGoximeter (results-sink): inserting 58 samples into database collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798
1672023-12-06T17:44:28.798ZDEBGoximeter (clickhouse-client): unrolling 58 total samples collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 id = 1cd5687c-e4c2-4772-ab30-533e410c62df
1682023-12-06T17:44:28.799ZDEBGoximeter (clickhouse-client): retrieving timeseries schema from database collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 id = 1cd5687c-e4c2-4772-ab30-533e410c62df
1692023-12-06T17:44:28.803ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 id = 1cd5687c-e4c2-4772-ab30-533e410c62df schema = {"timeseries_name":"instance_uuid:reset","fields.name":["uuid"],"fields.type":["Uuid"],"fields.source":["Target"],"datum_type":"CumulativeI64","created":"2023-12-06 17:44:28.799115939"} timeseries_name = instance_uuid:reset
1702023-12-06T17:44:28.803ZDEBGoximeter (clickhouse-client): retrieving timeseries schema from database collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 id = 1cd5687c-e4c2-4772-ab30-533e410c62df
1712023-12-06T17:44:28.806ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 id = 1cd5687c-e4c2-4772-ab30-533e410c62df schema = {"timeseries_name":"crucible_upstairs:activated","fields.name":["upstairs_uuid"],"fields.type":["Uuid"],"fields.source":["Target"],"datum_type":"CumulativeI64","created":"2023-12-06 17:44:28.803408945"} timeseries_name = crucible_upstairs:activated
1722023-12-06T17:44:28.806ZDEBGoximeter (clickhouse-client): retrieving timeseries schema from database collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 id = 1cd5687c-e4c2-4772-ab30-533e410c62df
1732023-12-06T17:44:28.810ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 id = 1cd5687c-e4c2-4772-ab30-533e410c62df schema = {"timeseries_name":"crucible_upstairs:flush","fields.name":["upstairs_uuid"],"fields.type":["Uuid"],"fields.source":["Target"],"datum_type":"CumulativeI64","created":"2023-12-06 17:44:28.806843663"} timeseries_name = crucible_upstairs:flush
1742023-12-06T17:44:28.810ZDEBGoximeter (clickhouse-client): retrieving timeseries schema from database collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 id = 1cd5687c-e4c2-4772-ab30-533e410c62df
1752023-12-06T17:44:28.813ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 id = 1cd5687c-e4c2-4772-ab30-533e410c62df schema = {"timeseries_name":"crucible_upstairs:write","fields.name":["upstairs_uuid"],"fields.type":["Uuid"],"fields.source":["Target"],"datum_type":"CumulativeI64","created":"2023-12-06 17:44:28.810349873"} timeseries_name = crucible_upstairs:write
1762023-12-06T17:44:28.813ZDEBGoximeter (clickhouse-client): retrieving timeseries schema from database collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 id = 1cd5687c-e4c2-4772-ab30-533e410c62df
1772023-12-06T17:44:28.817ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 id = 1cd5687c-e4c2-4772-ab30-533e410c62df schema = {"timeseries_name":"crucible_upstairs:write_bytes","fields.name":["upstairs_uuid"],"fields.type":["Uuid"],"fields.source":["Target"],"datum_type":"CumulativeI64","created":"2023-12-06 17:44:28.813841623"} timeseries_name = crucible_upstairs:write_bytes
1782023-12-06T17:44:28.817ZDEBGoximeter (clickhouse-client): retrieving timeseries schema from database collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 id = 1cd5687c-e4c2-4772-ab30-533e410c62df
1792023-12-06T17:44:28.820ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 id = 1cd5687c-e4c2-4772-ab30-533e410c62df schema = {"timeseries_name":"crucible_upstairs:read","fields.name":["upstairs_uuid"],"fields.type":["Uuid"],"fields.source":["Target"],"datum_type":"CumulativeI64","created":"2023-12-06 17:44:28.817287491"} timeseries_name = crucible_upstairs:read
1802023-12-06T17:44:28.820ZDEBGoximeter (clickhouse-client): retrieving timeseries schema from database collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 id = 1cd5687c-e4c2-4772-ab30-533e410c62df
1812023-12-06T17:44:28.824ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 id = 1cd5687c-e4c2-4772-ab30-533e410c62df schema = {"timeseries_name":"crucible_upstairs:read_bytes","fields.name":["upstairs_uuid"],"fields.type":["Uuid"],"fields.source":["Target"],"datum_type":"CumulativeI64","created":"2023-12-06 17:44:28.820858963"} timeseries_name = crucible_upstairs:read_bytes
1822023-12-06T17:44:28.824ZDEBGoximeter (clickhouse-client): retrieving timeseries schema from database collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 id = 1cd5687c-e4c2-4772-ab30-533e410c62df
1832023-12-06T17:44:28.827ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 id = 1cd5687c-e4c2-4772-ab30-533e410c62df schema = {"timeseries_name":"crucible_upstairs:flush_close","fields.name":["upstairs_uuid"],"fields.type":["Uuid"],"fields.source":["Target"],"datum_type":"CumulativeI64","created":"2023-12-06 17:44:28.824313211"} timeseries_name = crucible_upstairs:flush_close
1842023-12-06T17:44:28.827ZDEBGoximeter (clickhouse-client): retrieving timeseries schema from database collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 id = 1cd5687c-e4c2-4772-ab30-533e410c62df
1852023-12-06T17:44:28.831ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 id = 1cd5687c-e4c2-4772-ab30-533e410c62df schema = {"timeseries_name":"crucible_upstairs:extent_repair","fields.name":["upstairs_uuid"],"fields.type":["Uuid"],"fields.source":["Target"],"datum_type":"CumulativeI64","created":"2023-12-06 17:44:28.827758020"} timeseries_name = crucible_upstairs:extent_repair
1862023-12-06T17:44:28.831ZDEBGoximeter (clickhouse-client): retrieving timeseries schema from database collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 id = 1cd5687c-e4c2-4772-ab30-533e410c62df
1872023-12-06T17:44:28.834ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 id = 1cd5687c-e4c2-4772-ab30-533e410c62df schema = {"timeseries_name":"crucible_upstairs:extent_no_op","fields.name":["upstairs_uuid"],"fields.type":["Uuid"],"fields.source":["Target"],"datum_type":"CumulativeI64","created":"2023-12-06 17:44:28.831178738"} timeseries_name = crucible_upstairs:extent_no_op
1882023-12-06T17:44:28.834ZDEBGoximeter (clickhouse-client): retrieving timeseries schema from database collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 id = 1cd5687c-e4c2-4772-ab30-533e410c62df
1892023-12-06T17:44:28.837ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 id = 1cd5687c-e4c2-4772-ab30-533e410c62df schema = {"timeseries_name":"crucible_upstairs:extent_reopen","fields.name":["upstairs_uuid"],"fields.type":["Uuid"],"fields.source":["Target"],"datum_type":"CumulativeI64","created":"2023-12-06 17:44:28.834739949"} timeseries_name = crucible_upstairs:extent_reopen
1902023-12-06T17:44:28.840ZDEBGoximeter (clickhouse-client): inserting 11 new timeseries schema collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 id = 1cd5687c-e4c2-4772-ab30-533e410c62df
1912023-12-06T17:44:28.844ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 id = 1cd5687c-e4c2-4772-ab30-533e410c62df n_rows = 47 table_name = oximeter.fields_i64
1922023-12-06T17:44:28.845ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 id = 1cd5687c-e4c2-4772-ab30-533e410c62df n_rows = 141 table_name = oximeter.fields_string
1932023-12-06T17:44:28.846ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 id = 1cd5687c-e4c2-4772-ab30-533e410c62df n_rows = 58 table_name = oximeter.fields_uuid
1942023-12-06T17:44:28.848ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 id = 1cd5687c-e4c2-4772-ab30-533e410c62df n_rows = 11 table_name = oximeter.measurements_cumulativei64
1952023-12-06T17:44:28.849ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 id = 1cd5687c-e4c2-4772-ab30-533e410c62df n_rows = 47 table_name = oximeter.measurements_histogramf64
1962023-12-06T17:44:36.321ZDEBGoximeter (collection-task): collecting from producer collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 producer_id = 9a25727f-e844-4dd1-b1b4-978eca7f5c90
1972023-12-06T17:44:36.321ZDEBGoximeter (collection-task): collected results from producer collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 n_results = 4 producer_id = 9a25727f-e844-4dd1-b1b4-978eca7f5c90
1982023-12-06T17:44:36.326ZDEBGoximeter (collection-task): collecting from producer collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 producer_id = d2016ddd-0efc-4439-89aa-2f166d289e6f
1992023-12-06T17:44:36.327ZDEBGoximeter (collection-task): collected results from producer collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 n_results = 4 producer_id = d2016ddd-0efc-4439-89aa-2f166d289e6f
2002023-12-06T17:44:38.324ZDEBGoximeter (collection-task): collecting from producer collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 producer_id = 1dfcdbef-d660-4977-8363-0facffed5dd7
2012023-12-06T17:44:38.327ZDEBGoximeter (collection-task): collected results from producer collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 n_results = 4 producer_id = 1dfcdbef-d660-4977-8363-0facffed5dd7
2022023-12-06T17:44:38.327ZDEBGoximeter (collection-task): reporting oximeter self-collection statistics collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 producer_id = 1dfcdbef-d660-4977-8363-0facffed5dd7
2032023-12-06T17:44:38.798ZDEBGoximeter (results-sink): inserting 48 samples into database collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798
2042023-12-06T17:44:38.798ZDEBGoximeter (clickhouse-client): unrolling 48 total samples collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 id = 1cd5687c-e4c2-4772-ab30-533e410c62df
2052023-12-06T17:44:38.804ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 id = 1cd5687c-e4c2-4772-ab30-533e410c62df n_rows = 47 table_name = oximeter.fields_i64
2062023-12-06T17:44:38.807ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 id = 1cd5687c-e4c2-4772-ab30-533e410c62df n_rows = 2 table_name = oximeter.fields_ipaddr
2072023-12-06T17:44:38.809ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 id = 1cd5687c-e4c2-4772-ab30-533e410c62df n_rows = 142 table_name = oximeter.fields_string
2082023-12-06T17:44:38.810ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 id = 1cd5687c-e4c2-4772-ab30-533e410c62df n_rows = 2 table_name = oximeter.fields_u16
2092023-12-06T17:44:38.812ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 id = 1cd5687c-e4c2-4772-ab30-533e410c62df n_rows = 49 table_name = oximeter.fields_uuid
2102023-12-06T17:44:38.813ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 id = 1cd5687c-e4c2-4772-ab30-533e410c62df n_rows = 1 table_name = oximeter.measurements_cumulativeu64
2112023-12-06T17:44:38.815ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 id = 1cd5687c-e4c2-4772-ab30-533e410c62df n_rows = 47 table_name = oximeter.measurements_histogramf64
2122023-12-06T17:44:46.320ZDEBGoximeter (collection-task): collecting from producer collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 producer_id = 9a25727f-e844-4dd1-b1b4-978eca7f5c90
2132023-12-06T17:44:46.321ZDEBGoximeter (collection-task): collected results from producer collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 n_results = 4 producer_id = 9a25727f-e844-4dd1-b1b4-978eca7f5c90
2142023-12-06T17:44:46.321ZDEBGoximeter (collection-task): reporting oximeter self-collection statistics collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 producer_id = 9a25727f-e844-4dd1-b1b4-978eca7f5c90
2152023-12-06T17:44:46.326ZDEBGoximeter (collection-task): collecting from producer collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 producer_id = d2016ddd-0efc-4439-89aa-2f166d289e6f
2162023-12-06T17:44:46.327ZDEBGoximeter (collection-task): collected results from producer collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 n_results = 4 producer_id = d2016ddd-0efc-4439-89aa-2f166d289e6f
2172023-12-06T17:44:46.328ZDEBGoximeter (collection-task): reporting oximeter self-collection statistics collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 producer_id = d2016ddd-0efc-4439-89aa-2f166d289e6f
2182023-12-06T17:44:48.324ZDEBGoximeter (collection-task): collecting from producer collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 producer_id = 1dfcdbef-d660-4977-8363-0facffed5dd7
2192023-12-06T17:44:48.327ZDEBGoximeter (collection-task): collected results from producer collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 n_results = 4 producer_id = 1dfcdbef-d660-4977-8363-0facffed5dd7
2202023-12-06T17:44:48.797ZDEBGoximeter (results-sink): inserting 49 samples into database collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798
2212023-12-06T17:44:48.797ZDEBGoximeter (clickhouse-client): unrolling 49 total samples collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 id = 1cd5687c-e4c2-4772-ab30-533e410c62df
2222023-12-06T17:44:48.803ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 id = 1cd5687c-e4c2-4772-ab30-533e410c62df n_rows = 47 table_name = oximeter.fields_i64
2232023-12-06T17:44:48.804ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 id = 1cd5687c-e4c2-4772-ab30-533e410c62df n_rows = 4 table_name = oximeter.fields_ipaddr
2242023-12-06T17:44:48.806ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 id = 1cd5687c-e4c2-4772-ab30-533e410c62df n_rows = 143 table_name = oximeter.fields_string
2252023-12-06T17:44:48.807ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 id = 1cd5687c-e4c2-4772-ab30-533e410c62df n_rows = 4 table_name = oximeter.fields_u16
2262023-12-06T17:44:48.808ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 id = 1cd5687c-e4c2-4772-ab30-533e410c62df n_rows = 51 table_name = oximeter.fields_uuid
2272023-12-06T17:44:48.810ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 id = 1cd5687c-e4c2-4772-ab30-533e410c62df n_rows = 2 table_name = oximeter.measurements_cumulativeu64
2282023-12-06T17:44:48.812ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 id = 1cd5687c-e4c2-4772-ab30-533e410c62df n_rows = 47 table_name = oximeter.measurements_histogramf64
2292023-12-06T17:44:49.906ZDEBGoximeter (collection-task): collecting from producer collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 producer_id = 1e40a2ae-ffda-4a94-95f6-839f3349172b
2302023-12-06T17:44:49.906ZDEBGoximeter (collection-task): collected results from producer collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 n_results = 1 producer_id = 1e40a2ae-ffda-4a94-95f6-839f3349172b
2312023-12-06T17:44:53.798ZDEBGoximeter (results-sink): inserting 36 samples into database collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798
2322023-12-06T17:44:53.798ZDEBGoximeter (clickhouse-client): unrolling 36 total samples collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 id = 1cd5687c-e4c2-4772-ab30-533e410c62df
2332023-12-06T17:44:53.802ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 id = 1cd5687c-e4c2-4772-ab30-533e410c62df n_rows = 36 table_name = oximeter.fields_string
2342023-12-06T17:44:53.804ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 id = 1cd5687c-e4c2-4772-ab30-533e410c62df n_rows = 24 table_name = oximeter.fields_uuid
2352023-12-06T17:44:53.805ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 id = 1cd5687c-e4c2-4772-ab30-533e410c62df n_rows = 36 table_name = oximeter.measurements_cumulativeu64
2362023-12-06T17:44:56.320ZDEBGoximeter (collection-task): collecting from producer collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 producer_id = 9a25727f-e844-4dd1-b1b4-978eca7f5c90
2372023-12-06T17:44:56.321ZDEBGoximeter (collection-task): collected results from producer collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 n_results = 4 producer_id = 9a25727f-e844-4dd1-b1b4-978eca7f5c90
2382023-12-06T17:44:56.327ZDEBGoximeter (collection-task): collecting from producer collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 producer_id = d2016ddd-0efc-4439-89aa-2f166d289e6f
2392023-12-06T17:44:56.327ZDEBGoximeter (collection-task): collected results from producer collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 n_results = 4 producer_id = d2016ddd-0efc-4439-89aa-2f166d289e6f
2402023-12-06T17:44:57.016ZDEBGoximeter (collection-task): collecting from producer collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 producer_id = 798fd184-57e7-4c63-a223-b0a810e00937
2412023-12-06T17:44:57.016ZDEBGoximeter (collection-task): collected results from producer collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 n_results = 2 producer_id = 798fd184-57e7-4c63-a223-b0a810e00937
2422023-12-06T17:44:57.017ZDEBGoximeter (collection-task): reporting oximeter self-collection statistics collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 producer_id = 798fd184-57e7-4c63-a223-b0a810e00937
2432023-12-06T17:44:58.324ZDEBGoximeter (collection-task): collecting from producer collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 producer_id = 1dfcdbef-d660-4977-8363-0facffed5dd7
2442023-12-06T17:44:58.327ZDEBGoximeter (collection-task): collected results from producer collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 n_results = 4 producer_id = 1dfcdbef-d660-4977-8363-0facffed5dd7
2452023-12-06T17:44:58.797ZDEBGoximeter (results-sink): inserting 59 samples into database collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798
2462023-12-06T17:44:58.797ZDEBGoximeter (clickhouse-client): unrolling 59 total samples collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 id = 1cd5687c-e4c2-4772-ab30-533e410c62df
2472023-12-06T17:44:58.804ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 id = 1cd5687c-e4c2-4772-ab30-533e410c62df n_rows = 47 table_name = oximeter.fields_i64
2482023-12-06T17:44:58.805ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 id = 1cd5687c-e4c2-4772-ab30-533e410c62df n_rows = 2 table_name = oximeter.fields_ipaddr
2492023-12-06T17:44:58.807ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 id = 1cd5687c-e4c2-4772-ab30-533e410c62df n_rows = 142 table_name = oximeter.fields_string
2502023-12-06T17:44:58.810ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 id = 1cd5687c-e4c2-4772-ab30-533e410c62df n_rows = 2 table_name = oximeter.fields_u16
2512023-12-06T17:44:58.812ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 id = 1cd5687c-e4c2-4772-ab30-533e410c62df n_rows = 60 table_name = oximeter.fields_uuid
2522023-12-06T17:44:58.815ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 id = 1cd5687c-e4c2-4772-ab30-533e410c62df n_rows = 11 table_name = oximeter.measurements_cumulativei64
2532023-12-06T17:44:58.817ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 id = 1cd5687c-e4c2-4772-ab30-533e410c62df n_rows = 1 table_name = oximeter.measurements_cumulativeu64
2542023-12-06T17:44:58.821ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 358b59f5-3c51-4c63-92f1-1dcada1dc798 id = 1cd5687c-e4c2-4772-ab30-533e410c62df n_rows = 47 table_name = oximeter.measurements_histogramf64