Max level shown:
1 [ Jul 18 19:44:19 Enabled. ]
2 [ Jul 18 19:44:19 Rereading configuration. ]
3 [ Jul 18 19:44:21 Rereading configuration. ]
4 [ Jul 18 19:44:23 Executing start method ("ctrun -l child -o noorphan,regent /opt/oxide/oximeter-collector/bin/oximeter run /var/svc/manifest/site/oximeter/config.toml --address [fd00:1122:3344:101::d]:12223 --id 650a8687-e31d-4bdf-8da8-edf2f453515c &"). ]
5 [ Jul 18 19:44:23 Method "start" exited with status 0. ]
6 note: configured to log to "/dev/stdout"
72025-07-18T19:44:23.783ZDEBGoximeter: registered DTrace probes
82025-07-18T19:44:23.787ZINFOoximeter: starting oximeter server file = oximeter/collector/src/lib.rs:236
92025-07-18T19:44:23.793ZDEBGoximeter: creating ClickHouse client
102025-07-18T19:44:23.858ZDEBGoximeter (clickhouse-client): registered USDT probes collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d id = c6f04c80-a0b6-4ed1-9a43-264260a5e2a8
112025-07-18T19:44:31.085ZWARNoximeter (clickhouse-client): oximeter database does not exist, or is out-of-date collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d file = oximeter/db/src/client/mod.rs:902 id = c6f04c80-a0b6-4ed1-9a43-264260a5e2a8
122025-07-18T19:44:31.085ZDEBGoximeter (oximeter-agent): oximeter database does not exist, creating collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d
132025-07-18T19:44:31.085ZINFOoximeter (clickhouse-client): reading db version collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d file = oximeter/db/src/client/mod.rs:802 id = c6f04c80-a0b6-4ed1-9a43-264260a5e2a8
142025-07-18T19:44:31.087ZWARNoximeter (clickhouse-client): oximeter database does not exist, or is out-of-date collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d file = oximeter/db/src/client/mod.rs:902 id = c6f04c80-a0b6-4ed1-9a43-264260a5e2a8
152025-07-18T19:44:31.087ZINFOoximeter (clickhouse-client): read oximeter database version collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d file = oximeter/db/src/client/mod.rs:807 id = c6f04c80-a0b6-4ed1-9a43-264260a5e2a8 version = 0
162025-07-18T19:44:31.087ZINFOoximeter (clickhouse-client): initializing oximeter schema collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d file = oximeter/db/src/client/mod.rs:812 id = c6f04c80-a0b6-4ed1-9a43-264260a5e2a8
172025-07-18T19:44:31.087ZDEBGoximeter (clickhouse-client): initializing ClickHouse database collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d id = c6f04c80-a0b6-4ed1-9a43-264260a5e2a8
182025-07-18T19:44:31.305ZINFOoximeter (clickhouse-client): inserting current version collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d file = oximeter/db/src/client/mod.rs:833 id = c6f04c80-a0b6-4ed1-9a43-264260a5e2a8 version = 14
192025-07-18T19:44:31.354ZDEBGoximeter (clickhouse-client): registered USDT probes collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d id = a2ecfbe1-90dd-4552-8dab-3533b250999e
202025-07-18T19:44:31.376ZDEBGoximeter (dropshot): registered endpoint local_addr = [fd00:1122:3344:101::d]:12223 method = GET path = /info versions = all
212025-07-18T19:44:31.376ZDEBGoximeter (dropshot): registered endpoint local_addr = [fd00:1122:3344:101::d]:12223 method = GET path = /producers versions = all
222025-07-18T19:44:31.376ZDEBGoximeter (dropshot): registered endpoint local_addr = [fd00:1122:3344:101::d]:12223 method = DELETE path = /producers/{producer_id} versions = all
232025-07-18T19:44:31.377ZDEBGoximeter (dropshot): registered endpoint local_addr = [fd00:1122:3344:101::d]:12223 method = GET path = /producers/{producer_id} versions = all
242025-07-18T19:44:31.377ZINFOoximeter (dropshot): listening file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:328 local_addr = [fd00:1122:3344:101::d]:12223
252025-07-18T19:44:31.377ZDEBGoximeter (dropshot): successfully registered DTrace USDT probes local_addr = [fd00:1122:3344:101::d]:12223
262025-07-18T19:44:31.377ZDEBGoximeter: registered USDT probes
272025-07-18T19:44:31.377ZDEBGoximeter: contacting nexus
282025-07-18T19:44:31.554ZDEBGoximeter: client request body = Some(Body) method = POST uri = http://[fd00:1122:3344:101::b]:12221/metrics/collectors
292025-07-18T19:44:31.652ZDEBGoximeter: client response result = Ok(Response { url: "http://[fd00:1122:3344:101::b]:12221/metrics/collectors", status: 204, headers: {"x-request-id": "abad4677-5f0e-4657-8044-c4bd49e31ad4", "date": "Fri, 18 Jul 2025 19:44:31 GMT"} })
302025-07-18T19:44:31.656ZINFOoximeter: oximeter registered with nexus file = oximeter/collector/src/lib.rs:393 id = 650a8687-e31d-4bdf-8da8-edf2f453515c
312025-07-18T19:44:31.656ZINFOoximeter (oximeter-agent): starting refresh list task collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d file = oximeter/collector/src/agent.rs:559
322025-07-18T19:44:31.657ZINFOoximeter (oximeter-agent): refreshing list of producers from Nexus collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d file = oximeter/collector/src/agent.rs:562
332025-07-18T19:44:31.667ZDEBGoximeter: client request body = None method = GET uri = http://[fd00:1122:3344:101::b]:12221/metrics/collectors/650a8687-e31d-4bdf-8da8-edf2f453515c/producers?sort_by=id_ascending
342025-07-18T19:44:31.701ZDEBGoximeter: client response result = Ok(Response { url: "http://[fd00:1122:3344:101::b]:12221/metrics/collectors/650a8687-e31d-4bdf-8da8-edf2f453515c/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "03b96080-d7f9-4b7a-83f8-1c82d5542513", "content-length": "29", "date": "Fri, 18 Jul 2025 19:44:31 GMT"} })
352025-07-18T19:44:31.701ZINFOoximeter (oximeter-agent): refreshed list of producers from Nexus collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d file = oximeter/collector/src/agent.rs:637 n_current_tasks = 0 n_pruned_tasks = 0
362025-07-18T19:44:46.657ZINFOoximeter (oximeter-agent): refreshing list of producers from Nexus collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d file = oximeter/collector/src/agent.rs:562
372025-07-18T19:44:46.657ZDEBGoximeter: client request body = None method = GET uri = http://[fd00:1122:3344:101::c]:12221/metrics/collectors/650a8687-e31d-4bdf-8da8-edf2f453515c/producers?sort_by=id_ascending
382025-07-18T19:44:46.670ZDEBGoximeter: client response result = Ok(Response { url: "http://[fd00:1122:3344:101::c]:12221/metrics/collectors/650a8687-e31d-4bdf-8da8-edf2f453515c/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "2f82a165-57ed-4366-b0f3-bf1b182a1af4", "content-length": "721", "date": "Fri, 18 Jul 2025 19:44:46 GMT"} })
392025-07-18T19:44:46.673ZDEBGoximeter: client request body = None method = GET uri = http://[fd00:1122:3344:101::c]:12221/metrics/collectors/650a8687-e31d-4bdf-8da8-edf2f453515c/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJiMjU2OGQ2OC1jMmYxLTRlYzUtOWY0MC00M2VjZjkxYTg5OGEifX0%3D
402025-07-18T19:44:46.686ZDEBGoximeter: client response result = Ok(Response { url: "http://[fd00:1122:3344:101::c]:12221/metrics/collectors/650a8687-e31d-4bdf-8da8-edf2f453515c/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJiMjU2OGQ2OC1jMmYxLTRlYzUtOWY0MC00M2VjZjkxYTg5OGEifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "462c9caf-56ba-490a-a775-7acac7a0349d", "content-length": "29", "date": "Fri, 18 Jul 2025 19:44:46 GMT"} })
412025-07-18T19:44:46.689ZDEBGoximeter (oximeter-agent): registered new metric producer address = [fd00:1122:3344:101::2]:4677 collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 2ab58cea-1973-4965-8759-ba7489e0fe93
422025-07-18T19:44:46.691ZDEBGoximeter (oximeter-agent): registered new metric producer address = [fd00:1122:3344:101::2]:40407 collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 3f5838b1-1901-4ccc-8412-bc0c1b44e959
432025-07-18T19:44:46.693ZDEBGoximeter (oximeter-agent): registered new metric producer address = [fd00:1122:3344:101::1]:8001 collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 9dce788c-d2e1-4d5e-ac83-e6d8d56311b5
442025-07-18T19:44:46.695ZDEBGoximeter (oximeter-agent): registered new metric producer address = [fd00:1122:3344:101::c]:45058 collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = b2568d68-c2f1-4ec5-9f40-43ecf91a898a
452025-07-18T19:44:46.697ZINFOoximeter (oximeter-agent): refreshed list of producers from Nexus collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d file = oximeter/collector/src/agent.rs:637 n_current_tasks = 4 n_pruned_tasks = 0
462025-07-18T19:44:47.689ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 2ab58cea-1973-4965-8759-ba7489e0fe93
472025-07-18T19:44:47.689ZDEBGoximeter (collection-task): collecting from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 2ab58cea-1973-4965-8759-ba7489e0fe93
482025-07-18T19:44:47.694ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 9dce788c-d2e1-4d5e-ac83-e6d8d56311b5
492025-07-18T19:44:47.694ZDEBGoximeter (collection-task): collecting from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 9dce788c-d2e1-4d5e-ac83-e6d8d56311b5
502025-07-18T19:44:47.737ZDEBGoximeter (collection-task): collected results from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 2ab58cea-1973-4965-8759-ba7489e0fe93
512025-07-18T19:44:47.738ZDEBGoximeter (collection-task): collected results from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 9dce788c-d2e1-4d5e-ac83-e6d8d56311b5
522025-07-18T19:44:48.689ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 2ab58cea-1973-4965-8759-ba7489e0fe93
532025-07-18T19:44:48.689ZDEBGoximeter (collection-task): collecting from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 2ab58cea-1973-4965-8759-ba7489e0fe93
542025-07-18T19:44:48.690ZDEBGoximeter (collection-task): collected results from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 2ab58cea-1973-4965-8759-ba7489e0fe93
552025-07-18T19:44:48.694ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 9dce788c-d2e1-4d5e-ac83-e6d8d56311b5
562025-07-18T19:44:48.694ZDEBGoximeter (collection-task): collecting from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 9dce788c-d2e1-4d5e-ac83-e6d8d56311b5
572025-07-18T19:44:48.697ZDEBGoximeter (collection-task): collected results from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 9dce788c-d2e1-4d5e-ac83-e6d8d56311b5
582025-07-18T19:44:49.689ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 2ab58cea-1973-4965-8759-ba7489e0fe93
592025-07-18T19:44:49.689ZDEBGoximeter (collection-task): collecting from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 2ab58cea-1973-4965-8759-ba7489e0fe93
602025-07-18T19:44:49.690ZDEBGoximeter (collection-task): collected results from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 2ab58cea-1973-4965-8759-ba7489e0fe93
612025-07-18T19:44:49.694ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 9dce788c-d2e1-4d5e-ac83-e6d8d56311b5
622025-07-18T19:44:49.694ZDEBGoximeter (collection-task): collecting from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 9dce788c-d2e1-4d5e-ac83-e6d8d56311b5
632025-07-18T19:44:49.696ZDEBGoximeter (collection-task): collected results from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 9dce788c-d2e1-4d5e-ac83-e6d8d56311b5
642025-07-18T19:44:50.689ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 2ab58cea-1973-4965-8759-ba7489e0fe93
652025-07-18T19:44:50.689ZDEBGoximeter (collection-task): collecting from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 2ab58cea-1973-4965-8759-ba7489e0fe93
662025-07-18T19:44:50.690ZDEBGoximeter (collection-task): collected results from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 2ab58cea-1973-4965-8759-ba7489e0fe93
672025-07-18T19:44:50.694ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 9dce788c-d2e1-4d5e-ac83-e6d8d56311b5
682025-07-18T19:44:50.694ZDEBGoximeter (collection-task): collecting from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 9dce788c-d2e1-4d5e-ac83-e6d8d56311b5
692025-07-18T19:44:50.697ZDEBGoximeter (collection-task): collected results from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 9dce788c-d2e1-4d5e-ac83-e6d8d56311b5
702025-07-18T19:44:51.355ZDEBGoximeter (results-sink-cluster): inserting 120 samples into database collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d
712025-07-18T19:44:51.355ZDEBGoximeter (results-sink): inserting 120 samples into database collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d
722025-07-18T19:44:51.355ZDEBGoximeter (clickhouse-client): unrolling 120 total samples collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d id = c6f04c80-a0b6-4ed1-9a43-264260a5e2a8
732025-07-18T19:44:51.357ZDEBGoximeter (clickhouse-client): retrieving timeseries schema from database collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d id = c6f04c80-a0b6-4ed1-9a43-264260a5e2a8
742025-07-18T19:44:51.384ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d id = c6f04c80-a0b6-4ed1-9a43-264260a5e2a8 schema = TimeseriesSchema { timeseries_name: TimeseriesName("static_routing_config:static_routes"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "hostname", field_type: String, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }}, datum_type: CumulativeU64, version: 1, authz_scope: Fleet, units: Count, created: 2025-07-18T19:44:51.357956204Z } timeseries_name = static_routing_config:static_routes
752025-07-18T19:44:51.396ZDEBGoximeter (clickhouse-client): retrieving timeseries schema from database collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d id = c6f04c80-a0b6-4ed1-9a43-264260a5e2a8
762025-07-18T19:44:51.405ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d id = c6f04c80-a0b6-4ed1-9a43-264260a5e2a8 schema = TimeseriesSchema { timeseries_name: TimeseriesName("static_routing_config:static_nexthops"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "hostname", field_type: String, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }}, datum_type: CumulativeU64, version: 1, authz_scope: Fleet, units: Count, created: 2025-07-18T19:44:51.396628262Z } timeseries_name = static_routing_config:static_nexthops
772025-07-18T19:44:51.405ZDEBGoximeter (clickhouse-client): retrieving timeseries schema from database collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d id = c6f04c80-a0b6-4ed1-9a43-264260a5e2a8
782025-07-18T19:44:51.411ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d id = c6f04c80-a0b6-4ed1-9a43-264260a5e2a8 schema = TimeseriesSchema { timeseries_name: TimeseriesName("switch_rib:active_routes"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "hostname", field_type: String, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }}, datum_type: U64, version: 1, authz_scope: Fleet, units: Count, created: 2025-07-18T19:44:51.405871693Z } timeseries_name = switch_rib:active_routes
792025-07-18T19:44:51.411ZDEBGoximeter (clickhouse-client): retrieving timeseries schema from database collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d id = c6f04c80-a0b6-4ed1-9a43-264260a5e2a8
802025-07-18T19:44:51.412ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d id = c6f04c80-a0b6-4ed1-9a43-264260a5e2a8 schema = TimeseriesSchema { timeseries_name: TimeseriesName("mg_lower:routes_blocked_by_link_state"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "hostname", field_type: String, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }}, datum_type: U64, version: 1, authz_scope: Fleet, units: Count, created: 2025-07-18T19:44:51.411339760Z } timeseries_name = mg_lower:routes_blocked_by_link_state
812025-07-18T19:44:51.412ZDEBGoximeter (clickhouse-client): retrieving timeseries schema from database collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d id = c6f04c80-a0b6-4ed1-9a43-264260a5e2a8
822025-07-18T19:44:51.413ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d id = c6f04c80-a0b6-4ed1-9a43-264260a5e2a8 schema = TimeseriesSchema { timeseries_name: TimeseriesName("ddm_router:originated_underlay_prefixes"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "hostname", field_type: String, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }}, datum_type: U64, version: 1, authz_scope: Fleet, units: Count, created: 2025-07-18T19:44:51.412403295Z } timeseries_name = ddm_router:originated_underlay_prefixes
832025-07-18T19:44:51.413ZDEBGoximeter (clickhouse-client): retrieving timeseries schema from database collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d id = c6f04c80-a0b6-4ed1-9a43-264260a5e2a8
842025-07-18T19:44:51.414ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d id = c6f04c80-a0b6-4ed1-9a43-264260a5e2a8 schema = TimeseriesSchema { timeseries_name: TimeseriesName("ddm_router:originated_tunnel_endpoints"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "hostname", field_type: String, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }}, datum_type: U64, version: 1, authz_scope: Fleet, units: Count, created: 2025-07-18T19:44:51.413333148Z } timeseries_name = ddm_router:originated_tunnel_endpoints
852025-07-18T19:44:51.414ZDEBGoximeter (clickhouse-client): retrieving timeseries schema from database collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d id = c6f04c80-a0b6-4ed1-9a43-264260a5e2a8
862025-07-18T19:44:51.415ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d id = c6f04c80-a0b6-4ed1-9a43-264260a5e2a8 schema = TimeseriesSchema { timeseries_name: TimeseriesName("ddm_session:solicitations_sent"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "hostname", field_type: String, source: Target, description: "" }, FieldSchema { name: "interface", field_type: String, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }}, datum_type: CumulativeU64, version: 1, authz_scope: Fleet, units: Count, created: 2025-07-18T19:44:51.414238171Z } timeseries_name = ddm_session:solicitations_sent
872025-07-18T19:44:51.415ZDEBGoximeter (clickhouse-client): retrieving timeseries schema from database collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d id = c6f04c80-a0b6-4ed1-9a43-264260a5e2a8
882025-07-18T19:44:51.415ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d id = c6f04c80-a0b6-4ed1-9a43-264260a5e2a8 schema = TimeseriesSchema { timeseries_name: TimeseriesName("ddm_session:solicitations_received"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "hostname", field_type: String, source: Target, description: "" }, FieldSchema { name: "interface", field_type: String, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }}, datum_type: CumulativeU64, version: 1, authz_scope: Fleet, units: Count, created: 2025-07-18T19:44:51.415142514Z } timeseries_name = ddm_session:solicitations_received
892025-07-18T19:44:51.416ZDEBGoximeter (clickhouse-client): retrieving timeseries schema from database collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d id = c6f04c80-a0b6-4ed1-9a43-264260a5e2a8
902025-07-18T19:44:51.416ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d id = c6f04c80-a0b6-4ed1-9a43-264260a5e2a8 schema = TimeseriesSchema { timeseries_name: TimeseriesName("ddm_session:advertisements_sent"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "hostname", field_type: String, source: Target, description: "" }, FieldSchema { name: "interface", field_type: String, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }}, datum_type: CumulativeU64, version: 1, authz_scope: Fleet, units: Count, created: 2025-07-18T19:44:51.416040187Z } timeseries_name = ddm_session:advertisements_sent
912025-07-18T19:44:51.416ZDEBGoximeter (clickhouse-client): retrieving timeseries schema from database collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d id = c6f04c80-a0b6-4ed1-9a43-264260a5e2a8
922025-07-18T19:44:51.417ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d id = c6f04c80-a0b6-4ed1-9a43-264260a5e2a8 schema = TimeseriesSchema { timeseries_name: TimeseriesName("ddm_session:advertisements_received"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "hostname", field_type: String, source: Target, description: "" }, FieldSchema { name: "interface", field_type: String, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }}, datum_type: CumulativeU64, version: 1, authz_scope: Fleet, units: Count, created: 2025-07-18T19:44:51.416926759Z } timeseries_name = ddm_session:advertisements_received
932025-07-18T19:44:51.417ZDEBGoximeter (clickhouse-client): retrieving timeseries schema from database collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d id = c6f04c80-a0b6-4ed1-9a43-264260a5e2a8
942025-07-18T19:44:51.418ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d id = c6f04c80-a0b6-4ed1-9a43-264260a5e2a8 schema = TimeseriesSchema { timeseries_name: TimeseriesName("ddm_session:peer_expirations"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "hostname", field_type: String, source: Target, description: "" }, FieldSchema { name: "interface", field_type: String, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }}, datum_type: CumulativeU64, version: 1, authz_scope: Fleet, units: Count, created: 2025-07-18T19:44:51.417823752Z } timeseries_name = ddm_session:peer_expirations
952025-07-18T19:44:51.418ZDEBGoximeter (clickhouse-client): retrieving timeseries schema from database collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d id = c6f04c80-a0b6-4ed1-9a43-264260a5e2a8
962025-07-18T19:44:51.419ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d id = c6f04c80-a0b6-4ed1-9a43-264260a5e2a8 schema = TimeseriesSchema { timeseries_name: TimeseriesName("ddm_session:peer_address_changes"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "hostname", field_type: String, source: Target, description: "" }, FieldSchema { name: "interface", field_type: String, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }}, datum_type: CumulativeU64, version: 1, authz_scope: Fleet, units: Count, created: 2025-07-18T19:44:51.418710955Z } timeseries_name = ddm_session:peer_address_changes
972025-07-18T19:44:51.419ZDEBGoximeter (clickhouse-client): retrieving timeseries schema from database collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d id = c6f04c80-a0b6-4ed1-9a43-264260a5e2a8
982025-07-18T19:44:51.420ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d id = c6f04c80-a0b6-4ed1-9a43-264260a5e2a8 schema = TimeseriesSchema { timeseries_name: TimeseriesName("ddm_session:peer_sessions_established"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "hostname", field_type: String, source: Target, description: "" }, FieldSchema { name: "interface", field_type: String, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }}, datum_type: CumulativeU64, version: 1, authz_scope: Fleet, units: Count, created: 2025-07-18T19:44:51.419627018Z } timeseries_name = ddm_session:peer_sessions_established
992025-07-18T19:44:51.420ZDEBGoximeter (clickhouse-client): retrieving timeseries schema from database collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d id = c6f04c80-a0b6-4ed1-9a43-264260a5e2a8
1002025-07-18T19:44:51.421ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d id = c6f04c80-a0b6-4ed1-9a43-264260a5e2a8 schema = TimeseriesSchema { timeseries_name: TimeseriesName("ddm_session:updates_sent"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "hostname", field_type: String, source: Target, description: "" }, FieldSchema { name: "interface", field_type: String, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }}, datum_type: CumulativeU64, version: 1, authz_scope: Fleet, units: Count, created: 2025-07-18T19:44:51.420542300Z } timeseries_name = ddm_session:updates_sent
1012025-07-18T19:44:51.421ZDEBGoximeter (clickhouse-client): retrieving timeseries schema from database collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d id = c6f04c80-a0b6-4ed1-9a43-264260a5e2a8
1022025-07-18T19:44:51.422ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d id = c6f04c80-a0b6-4ed1-9a43-264260a5e2a8 schema = TimeseriesSchema { timeseries_name: TimeseriesName("ddm_session:updates_received"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "hostname", field_type: String, source: Target, description: "" }, FieldSchema { name: "interface", field_type: String, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }}, datum_type: CumulativeU64, version: 1, authz_scope: Fleet, units: Count, created: 2025-07-18T19:44:51.421452113Z } timeseries_name = ddm_session:updates_received
1032025-07-18T19:44:51.422ZDEBGoximeter (clickhouse-client): retrieving timeseries schema from database collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d id = c6f04c80-a0b6-4ed1-9a43-264260a5e2a8
1042025-07-18T19:44:51.423ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d id = c6f04c80-a0b6-4ed1-9a43-264260a5e2a8 schema = TimeseriesSchema { timeseries_name: TimeseriesName("ddm_session:update_send_fail"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "hostname", field_type: String, source: Target, description: "" }, FieldSchema { name: "interface", field_type: String, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }}, datum_type: CumulativeU64, version: 1, authz_scope: Fleet, units: Count, created: 2025-07-18T19:44:51.422381737Z } timeseries_name = ddm_session:update_send_fail
1052025-07-18T19:44:51.423ZDEBGoximeter (clickhouse-client): retrieving timeseries schema from database collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d id = c6f04c80-a0b6-4ed1-9a43-264260a5e2a8
1062025-07-18T19:44:51.424ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d id = c6f04c80-a0b6-4ed1-9a43-264260a5e2a8 schema = TimeseriesSchema { timeseries_name: TimeseriesName("ddm_session:imported_underlay_prefixes"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "hostname", field_type: String, source: Target, description: "" }, FieldSchema { name: "interface", field_type: String, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }}, datum_type: U64, version: 1, authz_scope: Fleet, units: Count, created: 2025-07-18T19:44:51.423276820Z } timeseries_name = ddm_session:imported_underlay_prefixes
1072025-07-18T19:44:51.424ZDEBGoximeter (clickhouse-client): retrieving timeseries schema from database collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d id = c6f04c80-a0b6-4ed1-9a43-264260a5e2a8
1082025-07-18T19:44:51.425ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d id = c6f04c80-a0b6-4ed1-9a43-264260a5e2a8 schema = TimeseriesSchema { timeseries_name: TimeseriesName("ddm_session:imported_tunnel_endpoints"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "hostname", field_type: String, source: Target, description: "" }, FieldSchema { name: "interface", field_type: String, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }}, datum_type: U64, version: 1, authz_scope: Fleet, units: Count, created: 2025-07-18T19:44:51.424224733Z } timeseries_name = ddm_session:imported_tunnel_endpoints
1092025-07-18T19:44:51.428ZDEBGoximeter (clickhouse-client): inserting 18 new timeseries schema collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d id = c6f04c80-a0b6-4ed1-9a43-264260a5e2a8
1102025-07-18T19:44:51.436ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d id = c6f04c80-a0b6-4ed1-9a43-264260a5e2a8 n_rows = 30 table_name = fields_string
1112025-07-18T19:44:51.438ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d id = c6f04c80-a0b6-4ed1-9a43-264260a5e2a8 n_rows = 36 table_name = fields_uuid
1122025-07-18T19:44:51.442ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d id = c6f04c80-a0b6-4ed1-9a43-264260a5e2a8 n_rows = 88 table_name = measurements_cumulativeu64
1132025-07-18T19:44:51.444ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d id = c6f04c80-a0b6-4ed1-9a43-264260a5e2a8 n_rows = 32 table_name = measurements_u64
1142025-07-18T19:44:51.456ZWARNoximeter (results-sink-cluster): failed to insert some results into metric DB collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
1152025-07-18T19:44:51.689ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 2ab58cea-1973-4965-8759-ba7489e0fe93
1162025-07-18T19:44:51.689ZDEBGoximeter (collection-task): collecting from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 2ab58cea-1973-4965-8759-ba7489e0fe93
1172025-07-18T19:44:51.690ZDEBGoximeter (collection-task): collected results from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 2ab58cea-1973-4965-8759-ba7489e0fe93
1182025-07-18T19:44:51.694ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 9dce788c-d2e1-4d5e-ac83-e6d8d56311b5
1192025-07-18T19:44:51.694ZDEBGoximeter (collection-task): collecting from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 9dce788c-d2e1-4d5e-ac83-e6d8d56311b5
1202025-07-18T19:44:51.696ZDEBGoximeter (collection-task): collected results from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 9dce788c-d2e1-4d5e-ac83-e6d8d56311b5
1212025-07-18T19:44:52.843ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 2ab58cea-1973-4965-8759-ba7489e0fe93
1222025-07-18T19:44:52.843ZDEBGoximeter (collection-task): collecting from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 2ab58cea-1973-4965-8759-ba7489e0fe93
1232025-07-18T19:44:52.843ZDEBGoximeter (collection-task): collected results from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 2ab58cea-1973-4965-8759-ba7489e0fe93
1242025-07-18T19:44:52.847ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 9dce788c-d2e1-4d5e-ac83-e6d8d56311b5
1252025-07-18T19:44:52.847ZDEBGoximeter (collection-task): collecting from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 9dce788c-d2e1-4d5e-ac83-e6d8d56311b5
1262025-07-18T19:44:52.849ZDEBGoximeter (collection-task): collected results from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 9dce788c-d2e1-4d5e-ac83-e6d8d56311b5
1272025-07-18T19:44:53.843ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 2ab58cea-1973-4965-8759-ba7489e0fe93
1282025-07-18T19:44:53.843ZDEBGoximeter (collection-task): collecting from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 2ab58cea-1973-4965-8759-ba7489e0fe93
1292025-07-18T19:44:53.843ZDEBGoximeter (collection-task): collected results from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 2ab58cea-1973-4965-8759-ba7489e0fe93
1302025-07-18T19:44:53.848ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 9dce788c-d2e1-4d5e-ac83-e6d8d56311b5
1312025-07-18T19:44:53.848ZDEBGoximeter (collection-task): collecting from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 9dce788c-d2e1-4d5e-ac83-e6d8d56311b5
1322025-07-18T19:44:53.850ZDEBGoximeter (collection-task): collected results from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 9dce788c-d2e1-4d5e-ac83-e6d8d56311b5
1332025-07-18T19:44:54.842ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 2ab58cea-1973-4965-8759-ba7489e0fe93
1342025-07-18T19:44:54.842ZDEBGoximeter (collection-task): collecting from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 2ab58cea-1973-4965-8759-ba7489e0fe93
1352025-07-18T19:44:54.843ZDEBGoximeter (collection-task): collected results from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 2ab58cea-1973-4965-8759-ba7489e0fe93
1362025-07-18T19:44:54.847ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 9dce788c-d2e1-4d5e-ac83-e6d8d56311b5
1372025-07-18T19:44:54.847ZDEBGoximeter (collection-task): collecting from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 9dce788c-d2e1-4d5e-ac83-e6d8d56311b5
1382025-07-18T19:44:54.850ZDEBGoximeter (collection-task): collected results from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 9dce788c-d2e1-4d5e-ac83-e6d8d56311b5
1392025-07-18T19:44:55.842ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 2ab58cea-1973-4965-8759-ba7489e0fe93
1402025-07-18T19:44:55.842ZDEBGoximeter (collection-task): collecting from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 2ab58cea-1973-4965-8759-ba7489e0fe93
1412025-07-18T19:44:55.843ZDEBGoximeter (collection-task): collected results from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 2ab58cea-1973-4965-8759-ba7489e0fe93
1422025-07-18T19:44:55.847ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 9dce788c-d2e1-4d5e-ac83-e6d8d56311b5
1432025-07-18T19:44:55.847ZDEBGoximeter (collection-task): collecting from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 9dce788c-d2e1-4d5e-ac83-e6d8d56311b5
1442025-07-18T19:44:55.849ZDEBGoximeter (collection-task): collected results from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 9dce788c-d2e1-4d5e-ac83-e6d8d56311b5
1452025-07-18T19:44:56.509ZDEBGoximeter (results-sink): inserting 150 samples into database collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d
1462025-07-18T19:44:56.509ZDEBGoximeter (results-sink-cluster): inserting 150 samples into database collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d
1472025-07-18T19:44:56.509ZDEBGoximeter (clickhouse-client): unrolling 150 total samples collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d id = c6f04c80-a0b6-4ed1-9a43-264260a5e2a8
1482025-07-18T19:44:56.516ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d id = c6f04c80-a0b6-4ed1-9a43-264260a5e2a8 n_rows = 30 table_name = fields_string
1492025-07-18T19:44:56.518ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d id = c6f04c80-a0b6-4ed1-9a43-264260a5e2a8 n_rows = 36 table_name = fields_uuid
1502025-07-18T19:44:56.520ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d id = c6f04c80-a0b6-4ed1-9a43-264260a5e2a8 n_rows = 110 table_name = measurements_cumulativeu64
1512025-07-18T19:44:56.521ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d id = c6f04c80-a0b6-4ed1-9a43-264260a5e2a8 n_rows = 40 table_name = measurements_u64
1522025-07-18T19:44:56.609ZWARNoximeter (results-sink-cluster): failed to insert some results into metric DB collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
1532025-07-18T19:44:56.843ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 2ab58cea-1973-4965-8759-ba7489e0fe93
1542025-07-18T19:44:56.843ZDEBGoximeter (collection-task): collecting from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 2ab58cea-1973-4965-8759-ba7489e0fe93
1552025-07-18T19:44:56.843ZDEBGoximeter (collection-task): collected results from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 2ab58cea-1973-4965-8759-ba7489e0fe93
1562025-07-18T19:44:56.844ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 3f5838b1-1901-4ccc-8412-bc0c1b44e959
1572025-07-18T19:44:56.845ZDEBGoximeter (collection-task): collecting from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 3f5838b1-1901-4ccc-8412-bc0c1b44e959
1582025-07-18T19:44:56.848ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 9dce788c-d2e1-4d5e-ac83-e6d8d56311b5
1592025-07-18T19:44:56.848ZDEBGoximeter (collection-task): collecting from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 9dce788c-d2e1-4d5e-ac83-e6d8d56311b5
1602025-07-18T19:44:56.849ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = b2568d68-c2f1-4ec5-9f40-43ecf91a898a
1612025-07-18T19:44:56.849ZDEBGoximeter (collection-task): collecting from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = b2568d68-c2f1-4ec5-9f40-43ecf91a898a
1622025-07-18T19:44:56.850ZDEBGoximeter (collection-task): collected results from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 9dce788c-d2e1-4d5e-ac83-e6d8d56311b5
1632025-07-18T19:44:56.853ZDEBGoximeter (collection-task): collected results from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d n_results = 5 producer_id = b2568d68-c2f1-4ec5-9f40-43ecf91a898a
1642025-07-18T19:44:56.853ZDEBGoximeter (collection-task): collected results from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d n_results = 2 producer_id = 3f5838b1-1901-4ccc-8412-bc0c1b44e959
1652025-07-18T19:44:57.842ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 2ab58cea-1973-4965-8759-ba7489e0fe93
1662025-07-18T19:44:57.842ZDEBGoximeter (collection-task): collecting from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 2ab58cea-1973-4965-8759-ba7489e0fe93
1672025-07-18T19:44:57.843ZDEBGoximeter (collection-task): collected results from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 2ab58cea-1973-4965-8759-ba7489e0fe93
1682025-07-18T19:44:57.847ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 9dce788c-d2e1-4d5e-ac83-e6d8d56311b5
1692025-07-18T19:44:57.847ZDEBGoximeter (collection-task): collecting from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 9dce788c-d2e1-4d5e-ac83-e6d8d56311b5
1702025-07-18T19:44:57.850ZDEBGoximeter (collection-task): collected results from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 9dce788c-d2e1-4d5e-ac83-e6d8d56311b5
1712025-07-18T19:44:58.843ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 2ab58cea-1973-4965-8759-ba7489e0fe93
1722025-07-18T19:44:58.843ZDEBGoximeter (collection-task): collecting from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 2ab58cea-1973-4965-8759-ba7489e0fe93
1732025-07-18T19:44:58.844ZDEBGoximeter (collection-task): collected results from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 2ab58cea-1973-4965-8759-ba7489e0fe93
1742025-07-18T19:44:58.848ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 9dce788c-d2e1-4d5e-ac83-e6d8d56311b5
1752025-07-18T19:44:58.848ZDEBGoximeter (collection-task): collecting from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 9dce788c-d2e1-4d5e-ac83-e6d8d56311b5
1762025-07-18T19:44:58.850ZDEBGoximeter (collection-task): collected results from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 9dce788c-d2e1-4d5e-ac83-e6d8d56311b5
1772025-07-18T19:44:59.842ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 2ab58cea-1973-4965-8759-ba7489e0fe93
1782025-07-18T19:44:59.842ZDEBGoximeter (collection-task): collecting from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 2ab58cea-1973-4965-8759-ba7489e0fe93
1792025-07-18T19:44:59.843ZDEBGoximeter (collection-task): collected results from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 2ab58cea-1973-4965-8759-ba7489e0fe93
1802025-07-18T19:44:59.847ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 9dce788c-d2e1-4d5e-ac83-e6d8d56311b5
1812025-07-18T19:44:59.847ZDEBGoximeter (collection-task): collecting from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 9dce788c-d2e1-4d5e-ac83-e6d8d56311b5
1822025-07-18T19:44:59.849ZDEBGoximeter (collection-task): collected results from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 9dce788c-d2e1-4d5e-ac83-e6d8d56311b5
1832025-07-18T19:45:00.843ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 2ab58cea-1973-4965-8759-ba7489e0fe93
1842025-07-18T19:45:00.843ZDEBGoximeter (collection-task): collecting from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 2ab58cea-1973-4965-8759-ba7489e0fe93
1852025-07-18T19:45:00.843ZDEBGoximeter (collection-task): collected results from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 2ab58cea-1973-4965-8759-ba7489e0fe93
1862025-07-18T19:45:00.848ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 9dce788c-d2e1-4d5e-ac83-e6d8d56311b5
1872025-07-18T19:45:00.848ZDEBGoximeter (collection-task): collecting from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 9dce788c-d2e1-4d5e-ac83-e6d8d56311b5
1882025-07-18T19:45:00.850ZDEBGoximeter (collection-task): collected results from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 9dce788c-d2e1-4d5e-ac83-e6d8d56311b5
1892025-07-18T19:45:01.509ZDEBGoximeter (results-sink): inserting 165 samples into database collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d
1902025-07-18T19:45:01.509ZDEBGoximeter (results-sink-cluster): inserting 165 samples into database collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d
1912025-07-18T19:45:01.509ZDEBGoximeter (clickhouse-client): unrolling 165 total samples collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d id = c6f04c80-a0b6-4ed1-9a43-264260a5e2a8
1922025-07-18T19:45:01.510ZDEBGoximeter (clickhouse-client): retrieving timeseries schema from database collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d id = c6f04c80-a0b6-4ed1-9a43-264260a5e2a8
1932025-07-18T19:45:01.564ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d id = c6f04c80-a0b6-4ed1-9a43-264260a5e2a8 schema = TimeseriesSchema { timeseries_name: TimeseriesName("http_service:request_latency_histogram"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "name", field_type: String, source: Target, description: "" }, FieldSchema { name: "operation_id", field_type: String, source: Metric, description: "" }, FieldSchema { name: "status_code", field_type: U16, source: Metric, description: "" }}, datum_type: HistogramU64, version: 1, authz_scope: Fleet, units: Count, created: 2025-07-18T19:45:01.510721712Z } timeseries_name = http_service:request_latency_histogram
1942025-07-18T19:45:01.564ZDEBGoximeter (clickhouse-client): retrieving timeseries schema from database collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d id = c6f04c80-a0b6-4ed1-9a43-264260a5e2a8
1952025-07-18T19:45:01.566ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d id = c6f04c80-a0b6-4ed1-9a43-264260a5e2a8 schema = TimeseriesSchema { timeseries_name: TimeseriesName("collection_target:cpus_provisioned"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "id", field_type: Uuid, source: Target, description: "" }}, datum_type: I64, version: 1, authz_scope: Fleet, units: Count, created: 2025-07-18T19:45:01.564765337Z } timeseries_name = collection_target:cpus_provisioned
1962025-07-18T19:45:01.566ZDEBGoximeter (clickhouse-client): retrieving timeseries schema from database collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d id = c6f04c80-a0b6-4ed1-9a43-264260a5e2a8
1972025-07-18T19:45:01.567ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d id = c6f04c80-a0b6-4ed1-9a43-264260a5e2a8 schema = TimeseriesSchema { timeseries_name: TimeseriesName("collection_target:ram_provisioned"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "id", field_type: Uuid, source: Target, description: "" }}, datum_type: I64, version: 1, authz_scope: Fleet, units: Count, created: 2025-07-18T19:45:01.566427401Z } timeseries_name = collection_target:ram_provisioned
1982025-07-18T19:45:01.567ZDEBGoximeter (clickhouse-client): retrieving timeseries schema from database collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d id = c6f04c80-a0b6-4ed1-9a43-264260a5e2a8
1992025-07-18T19:45:01.569ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d id = c6f04c80-a0b6-4ed1-9a43-264260a5e2a8 schema = TimeseriesSchema { timeseries_name: TimeseriesName("collection_target:virtual_disk_space_provisioned"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "id", field_type: Uuid, source: Target, description: "" }}, datum_type: I64, version: 1, authz_scope: Fleet, units: Count, created: 2025-07-18T19:45:01.567891442Z } timeseries_name = collection_target:virtual_disk_space_provisioned
2002025-07-18T19:45:01.573ZDEBGoximeter (clickhouse-client): inserting 4 new timeseries schema collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d id = c6f04c80-a0b6-4ed1-9a43-264260a5e2a8
2012025-07-18T19:45:01.577ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d id = c6f04c80-a0b6-4ed1-9a43-264260a5e2a8 n_rows = 54 table_name = fields_string
2022025-07-18T19:45:01.579ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d id = c6f04c80-a0b6-4ed1-9a43-264260a5e2a8 n_rows = 12 table_name = fields_u16
2032025-07-18T19:45:01.581ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d id = c6f04c80-a0b6-4ed1-9a43-264260a5e2a8 n_rows = 51 table_name = fields_uuid
2042025-07-18T19:45:01.582ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d id = c6f04c80-a0b6-4ed1-9a43-264260a5e2a8 n_rows = 110 table_name = measurements_cumulativeu64
2052025-07-18T19:45:01.590ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d id = c6f04c80-a0b6-4ed1-9a43-264260a5e2a8 n_rows = 12 table_name = measurements_histogramu64
2062025-07-18T19:45:01.591ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d id = c6f04c80-a0b6-4ed1-9a43-264260a5e2a8 n_rows = 3 table_name = measurements_i64
2072025-07-18T19:45:01.593ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d id = c6f04c80-a0b6-4ed1-9a43-264260a5e2a8 n_rows = 40 table_name = measurements_u64
2082025-07-18T19:45:01.609ZWARNoximeter (results-sink-cluster): failed to insert some results into metric DB collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
2092025-07-18T19:45:01.810ZINFOoximeter (oximeter-agent): refreshing list of producers from Nexus collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d file = oximeter/collector/src/agent.rs:562
2102025-07-18T19:45:01.810ZDEBGoximeter: client request body = None method = GET uri = http://[fd00:1122:3344:101::a]:12221/metrics/collectors/650a8687-e31d-4bdf-8da8-edf2f453515c/producers?sort_by=id_ascending
2112025-07-18T19:45:01.825ZDEBGoximeter: client response result = Ok(Response { url: "http://[fd00:1122:3344:101::a]:12221/metrics/collectors/650a8687-e31d-4bdf-8da8-edf2f453515c/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "34e77ec5-39a0-409c-bb91-d4bc0e95247b", "content-length": "721", "date": "Fri, 18 Jul 2025 19:45:01 GMT"} })
2122025-07-18T19:45:01.825ZDEBGoximeter: client request body = None method = GET uri = http://[fd00:1122:3344:101::a]:12221/metrics/collectors/650a8687-e31d-4bdf-8da8-edf2f453515c/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJiMjU2OGQ2OC1jMmYxLTRlYzUtOWY0MC00M2VjZjkxYTg5OGEifX0%3D
2132025-07-18T19:45:01.830ZDEBGoximeter: client response result = Ok(Response { url: "http://[fd00:1122:3344:101::a]:12221/metrics/collectors/650a8687-e31d-4bdf-8da8-edf2f453515c/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJiMjU2OGQ2OC1jMmYxLTRlYzUtOWY0MC00M2VjZjkxYTg5OGEifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "b0415956-ecf1-4362-8dfc-a5562f72031b", "content-length": "29", "date": "Fri, 18 Jul 2025 19:45:01 GMT"} })
2142025-07-18T19:45:01.830ZINFOoximeter (oximeter-agent): refreshed list of producers from Nexus collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d file = oximeter/collector/src/agent.rs:637 n_current_tasks = 4 n_pruned_tasks = 0
2152025-07-18T19:45:01.842ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 2ab58cea-1973-4965-8759-ba7489e0fe93
2162025-07-18T19:45:01.842ZDEBGoximeter (collection-task): collecting from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 2ab58cea-1973-4965-8759-ba7489e0fe93
2172025-07-18T19:45:01.843ZDEBGoximeter (collection-task): collected results from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 2ab58cea-1973-4965-8759-ba7489e0fe93
2182025-07-18T19:45:01.847ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 9dce788c-d2e1-4d5e-ac83-e6d8d56311b5
2192025-07-18T19:45:01.847ZDEBGoximeter (collection-task): collecting from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 9dce788c-d2e1-4d5e-ac83-e6d8d56311b5
2202025-07-18T19:45:01.850ZDEBGoximeter (collection-task): collected results from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 9dce788c-d2e1-4d5e-ac83-e6d8d56311b5
2212025-07-18T19:45:02.842ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 2ab58cea-1973-4965-8759-ba7489e0fe93
2222025-07-18T19:45:02.842ZDEBGoximeter (collection-task): collecting from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 2ab58cea-1973-4965-8759-ba7489e0fe93
2232025-07-18T19:45:02.843ZDEBGoximeter (collection-task): collected results from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 2ab58cea-1973-4965-8759-ba7489e0fe93
2242025-07-18T19:45:02.847ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 9dce788c-d2e1-4d5e-ac83-e6d8d56311b5
2252025-07-18T19:45:02.847ZDEBGoximeter (collection-task): collecting from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 9dce788c-d2e1-4d5e-ac83-e6d8d56311b5
2262025-07-18T19:45:02.850ZDEBGoximeter (collection-task): collected results from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 9dce788c-d2e1-4d5e-ac83-e6d8d56311b5
2272025-07-18T19:45:03.843ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 2ab58cea-1973-4965-8759-ba7489e0fe93
2282025-07-18T19:45:03.843ZDEBGoximeter (collection-task): collecting from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 2ab58cea-1973-4965-8759-ba7489e0fe93
2292025-07-18T19:45:03.844ZDEBGoximeter (collection-task): collected results from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 2ab58cea-1973-4965-8759-ba7489e0fe93
2302025-07-18T19:45:03.848ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 9dce788c-d2e1-4d5e-ac83-e6d8d56311b5
2312025-07-18T19:45:03.848ZDEBGoximeter (collection-task): collecting from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 9dce788c-d2e1-4d5e-ac83-e6d8d56311b5
2322025-07-18T19:45:03.851ZDEBGoximeter (collection-task): collected results from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 9dce788c-d2e1-4d5e-ac83-e6d8d56311b5
2332025-07-18T19:45:04.842ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 2ab58cea-1973-4965-8759-ba7489e0fe93
2342025-07-18T19:45:04.842ZDEBGoximeter (collection-task): collecting from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 2ab58cea-1973-4965-8759-ba7489e0fe93
2352025-07-18T19:45:04.843ZDEBGoximeter (collection-task): collected results from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 2ab58cea-1973-4965-8759-ba7489e0fe93
2362025-07-18T19:45:04.847ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 9dce788c-d2e1-4d5e-ac83-e6d8d56311b5
2372025-07-18T19:45:04.847ZDEBGoximeter (collection-task): collecting from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 9dce788c-d2e1-4d5e-ac83-e6d8d56311b5
2382025-07-18T19:45:04.849ZDEBGoximeter (collection-task): collected results from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 9dce788c-d2e1-4d5e-ac83-e6d8d56311b5
2392025-07-18T19:45:05.843ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 2ab58cea-1973-4965-8759-ba7489e0fe93
2402025-07-18T19:45:05.843ZDEBGoximeter (collection-task): collecting from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 2ab58cea-1973-4965-8759-ba7489e0fe93
2412025-07-18T19:45:05.843ZDEBGoximeter (collection-task): collected results from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 2ab58cea-1973-4965-8759-ba7489e0fe93
2422025-07-18T19:45:05.848ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 9dce788c-d2e1-4d5e-ac83-e6d8d56311b5
2432025-07-18T19:45:05.848ZDEBGoximeter (collection-task): collecting from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 9dce788c-d2e1-4d5e-ac83-e6d8d56311b5
2442025-07-18T19:45:05.850ZDEBGoximeter (collection-task): collected results from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 9dce788c-d2e1-4d5e-ac83-e6d8d56311b5
2452025-07-18T19:45:06.508ZDEBGoximeter (results-sink): inserting 150 samples into database collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d
2462025-07-18T19:45:06.508ZDEBGoximeter (results-sink-cluster): inserting 150 samples into database collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d
2472025-07-18T19:45:06.508ZDEBGoximeter (clickhouse-client): unrolling 150 total samples collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d id = c6f04c80-a0b6-4ed1-9a43-264260a5e2a8
2482025-07-18T19:45:06.516ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d id = c6f04c80-a0b6-4ed1-9a43-264260a5e2a8 n_rows = 30 table_name = fields_string
2492025-07-18T19:45:06.517ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d id = c6f04c80-a0b6-4ed1-9a43-264260a5e2a8 n_rows = 36 table_name = fields_uuid
2502025-07-18T19:45:06.519ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d id = c6f04c80-a0b6-4ed1-9a43-264260a5e2a8 n_rows = 110 table_name = measurements_cumulativeu64
2512025-07-18T19:45:06.521ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d id = c6f04c80-a0b6-4ed1-9a43-264260a5e2a8 n_rows = 40 table_name = measurements_u64
2522025-07-18T19:45:06.610ZWARNoximeter (results-sink-cluster): failed to insert some results into metric DB collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
2532025-07-18T19:45:06.843ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 2ab58cea-1973-4965-8759-ba7489e0fe93
2542025-07-18T19:45:06.843ZDEBGoximeter (collection-task): collecting from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 2ab58cea-1973-4965-8759-ba7489e0fe93
2552025-07-18T19:45:06.844ZDEBGoximeter (collection-task): collected results from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 2ab58cea-1973-4965-8759-ba7489e0fe93
2562025-07-18T19:45:06.845ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 3f5838b1-1901-4ccc-8412-bc0c1b44e959
2572025-07-18T19:45:06.845ZDEBGoximeter (collection-task): collecting from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 3f5838b1-1901-4ccc-8412-bc0c1b44e959
2582025-07-18T19:45:06.847ZDEBGoximeter (collection-task): collected results from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d n_results = 2 producer_id = 3f5838b1-1901-4ccc-8412-bc0c1b44e959
2592025-07-18T19:45:06.847ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 9dce788c-d2e1-4d5e-ac83-e6d8d56311b5
2602025-07-18T19:45:06.847ZDEBGoximeter (collection-task): collecting from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 9dce788c-d2e1-4d5e-ac83-e6d8d56311b5
2612025-07-18T19:45:06.849ZDEBGoximeter (collection-task): collected results from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 9dce788c-d2e1-4d5e-ac83-e6d8d56311b5
2622025-07-18T19:45:06.850ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = b2568d68-c2f1-4ec5-9f40-43ecf91a898a
2632025-07-18T19:45:06.850ZDEBGoximeter (collection-task): collecting from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = b2568d68-c2f1-4ec5-9f40-43ecf91a898a
2642025-07-18T19:45:06.850ZDEBGoximeter (collection-task): collected results from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d n_results = 5 producer_id = b2568d68-c2f1-4ec5-9f40-43ecf91a898a
2652025-07-18T19:45:07.843ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 2ab58cea-1973-4965-8759-ba7489e0fe93
2662025-07-18T19:45:07.843ZDEBGoximeter (collection-task): collecting from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 2ab58cea-1973-4965-8759-ba7489e0fe93
2672025-07-18T19:45:07.843ZDEBGoximeter (collection-task): collected results from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 2ab58cea-1973-4965-8759-ba7489e0fe93
2682025-07-18T19:45:07.848ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 9dce788c-d2e1-4d5e-ac83-e6d8d56311b5
2692025-07-18T19:45:07.848ZDEBGoximeter (collection-task): collecting from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 9dce788c-d2e1-4d5e-ac83-e6d8d56311b5
2702025-07-18T19:45:07.850ZDEBGoximeter (collection-task): collected results from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 9dce788c-d2e1-4d5e-ac83-e6d8d56311b5
2712025-07-18T19:45:08.843ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 2ab58cea-1973-4965-8759-ba7489e0fe93
2722025-07-18T19:45:08.843ZDEBGoximeter (collection-task): collecting from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 2ab58cea-1973-4965-8759-ba7489e0fe93
2732025-07-18T19:45:08.844ZDEBGoximeter (collection-task): collected results from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 2ab58cea-1973-4965-8759-ba7489e0fe93
2742025-07-18T19:45:08.848ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 9dce788c-d2e1-4d5e-ac83-e6d8d56311b5
2752025-07-18T19:45:08.848ZDEBGoximeter (collection-task): collecting from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 9dce788c-d2e1-4d5e-ac83-e6d8d56311b5
2762025-07-18T19:45:08.850ZDEBGoximeter (collection-task): collected results from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 9dce788c-d2e1-4d5e-ac83-e6d8d56311b5
2772025-07-18T19:45:09.842ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 2ab58cea-1973-4965-8759-ba7489e0fe93
2782025-07-18T19:45:09.842ZDEBGoximeter (collection-task): collecting from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 2ab58cea-1973-4965-8759-ba7489e0fe93
2792025-07-18T19:45:09.843ZDEBGoximeter (collection-task): collected results from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 2ab58cea-1973-4965-8759-ba7489e0fe93
2802025-07-18T19:45:09.847ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 9dce788c-d2e1-4d5e-ac83-e6d8d56311b5
2812025-07-18T19:45:09.847ZDEBGoximeter (collection-task): collecting from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 9dce788c-d2e1-4d5e-ac83-e6d8d56311b5
2822025-07-18T19:45:09.849ZDEBGoximeter (collection-task): collected results from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 9dce788c-d2e1-4d5e-ac83-e6d8d56311b5
2832025-07-18T19:45:10.843ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 2ab58cea-1973-4965-8759-ba7489e0fe93
2842025-07-18T19:45:10.843ZDEBGoximeter (collection-task): collecting from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 2ab58cea-1973-4965-8759-ba7489e0fe93
2852025-07-18T19:45:10.843ZDEBGoximeter (collection-task): collected results from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 2ab58cea-1973-4965-8759-ba7489e0fe93
2862025-07-18T19:45:10.847ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 9dce788c-d2e1-4d5e-ac83-e6d8d56311b5
2872025-07-18T19:45:10.847ZDEBGoximeter (collection-task): collecting from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 9dce788c-d2e1-4d5e-ac83-e6d8d56311b5
2882025-07-18T19:45:10.850ZDEBGoximeter (collection-task): collected results from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 9dce788c-d2e1-4d5e-ac83-e6d8d56311b5
2892025-07-18T19:45:11.508ZDEBGoximeter (results-sink): inserting 162 samples into database collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d
2902025-07-18T19:45:11.508ZDEBGoximeter (results-sink-cluster): inserting 162 samples into database collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d
2912025-07-18T19:45:11.508ZDEBGoximeter (clickhouse-client): unrolling 162 total samples collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d id = c6f04c80-a0b6-4ed1-9a43-264260a5e2a8
2922025-07-18T19:45:11.517ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d id = c6f04c80-a0b6-4ed1-9a43-264260a5e2a8 n_rows = 54 table_name = fields_string
2932025-07-18T19:45:11.519ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d id = c6f04c80-a0b6-4ed1-9a43-264260a5e2a8 n_rows = 12 table_name = fields_u16
2942025-07-18T19:45:11.520ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d id = c6f04c80-a0b6-4ed1-9a43-264260a5e2a8 n_rows = 48 table_name = fields_uuid
2952025-07-18T19:45:11.522ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d id = c6f04c80-a0b6-4ed1-9a43-264260a5e2a8 n_rows = 110 table_name = measurements_cumulativeu64
2962025-07-18T19:45:11.524ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d id = c6f04c80-a0b6-4ed1-9a43-264260a5e2a8 n_rows = 12 table_name = measurements_histogramu64
2972025-07-18T19:45:11.526ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d id = c6f04c80-a0b6-4ed1-9a43-264260a5e2a8 n_rows = 40 table_name = measurements_u64
2982025-07-18T19:45:11.610ZWARNoximeter (results-sink-cluster): failed to insert some results into metric DB collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
2992025-07-18T19:45:11.842ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 2ab58cea-1973-4965-8759-ba7489e0fe93
3002025-07-18T19:45:11.842ZDEBGoximeter (collection-task): collecting from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 2ab58cea-1973-4965-8759-ba7489e0fe93
3012025-07-18T19:45:11.843ZDEBGoximeter (collection-task): collected results from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 2ab58cea-1973-4965-8759-ba7489e0fe93
3022025-07-18T19:45:11.847ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 9dce788c-d2e1-4d5e-ac83-e6d8d56311b5
3032025-07-18T19:45:11.847ZDEBGoximeter (collection-task): collecting from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 9dce788c-d2e1-4d5e-ac83-e6d8d56311b5
3042025-07-18T19:45:11.849ZDEBGoximeter (collection-task): collected results from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 9dce788c-d2e1-4d5e-ac83-e6d8d56311b5
3052025-07-18T19:45:12.843ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 2ab58cea-1973-4965-8759-ba7489e0fe93
3062025-07-18T19:45:12.843ZDEBGoximeter (collection-task): collecting from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 2ab58cea-1973-4965-8759-ba7489e0fe93
3072025-07-18T19:45:12.843ZDEBGoximeter (collection-task): collected results from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 2ab58cea-1973-4965-8759-ba7489e0fe93
3082025-07-18T19:45:12.848ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 9dce788c-d2e1-4d5e-ac83-e6d8d56311b5
3092025-07-18T19:45:12.848ZDEBGoximeter (collection-task): collecting from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 9dce788c-d2e1-4d5e-ac83-e6d8d56311b5
3102025-07-18T19:45:12.850ZDEBGoximeter (collection-task): collected results from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 9dce788c-d2e1-4d5e-ac83-e6d8d56311b5
3112025-07-18T19:45:13.843ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 2ab58cea-1973-4965-8759-ba7489e0fe93
3122025-07-18T19:45:13.843ZDEBGoximeter (collection-task): collecting from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 2ab58cea-1973-4965-8759-ba7489e0fe93
3132025-07-18T19:45:13.844ZDEBGoximeter (collection-task): collected results from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 2ab58cea-1973-4965-8759-ba7489e0fe93
3142025-07-18T19:45:13.848ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 9dce788c-d2e1-4d5e-ac83-e6d8d56311b5
3152025-07-18T19:45:13.848ZDEBGoximeter (collection-task): collecting from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 9dce788c-d2e1-4d5e-ac83-e6d8d56311b5
3162025-07-18T19:45:13.850ZDEBGoximeter (collection-task): collected results from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 9dce788c-d2e1-4d5e-ac83-e6d8d56311b5
3172025-07-18T19:45:14.843ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 2ab58cea-1973-4965-8759-ba7489e0fe93
3182025-07-18T19:45:14.843ZDEBGoximeter (collection-task): collecting from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 2ab58cea-1973-4965-8759-ba7489e0fe93
3192025-07-18T19:45:14.844ZDEBGoximeter (collection-task): collected results from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 2ab58cea-1973-4965-8759-ba7489e0fe93
3202025-07-18T19:45:14.848ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 9dce788c-d2e1-4d5e-ac83-e6d8d56311b5
3212025-07-18T19:45:14.848ZDEBGoximeter (collection-task): collecting from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 9dce788c-d2e1-4d5e-ac83-e6d8d56311b5
3222025-07-18T19:45:14.850ZDEBGoximeter (collection-task): collected results from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 9dce788c-d2e1-4d5e-ac83-e6d8d56311b5
3232025-07-18T19:45:15.843ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 2ab58cea-1973-4965-8759-ba7489e0fe93
3242025-07-18T19:45:15.843ZDEBGoximeter (collection-task): collecting from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 2ab58cea-1973-4965-8759-ba7489e0fe93
3252025-07-18T19:45:15.843ZDEBGoximeter (collection-task): collected results from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 2ab58cea-1973-4965-8759-ba7489e0fe93
3262025-07-18T19:45:15.848ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 9dce788c-d2e1-4d5e-ac83-e6d8d56311b5
3272025-07-18T19:45:15.848ZDEBGoximeter (collection-task): collecting from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 9dce788c-d2e1-4d5e-ac83-e6d8d56311b5
3282025-07-18T19:45:15.850ZDEBGoximeter (collection-task): collected results from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 9dce788c-d2e1-4d5e-ac83-e6d8d56311b5
3292025-07-18T19:45:16.509ZDEBGoximeter (results-sink): inserting 150 samples into database collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d
3302025-07-18T19:45:16.509ZDEBGoximeter (results-sink-cluster): inserting 150 samples into database collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d
3312025-07-18T19:45:16.509ZDEBGoximeter (clickhouse-client): unrolling 150 total samples collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d id = c6f04c80-a0b6-4ed1-9a43-264260a5e2a8
3322025-07-18T19:45:16.517ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d id = c6f04c80-a0b6-4ed1-9a43-264260a5e2a8 n_rows = 30 table_name = fields_string
3332025-07-18T19:45:16.519ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d id = c6f04c80-a0b6-4ed1-9a43-264260a5e2a8 n_rows = 36 table_name = fields_uuid
3342025-07-18T19:45:16.521ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d id = c6f04c80-a0b6-4ed1-9a43-264260a5e2a8 n_rows = 110 table_name = measurements_cumulativeu64
3352025-07-18T19:45:16.523ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d id = c6f04c80-a0b6-4ed1-9a43-264260a5e2a8 n_rows = 40 table_name = measurements_u64
3362025-07-18T19:45:16.610ZWARNoximeter (results-sink-cluster): failed to insert some results into metric DB collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
3372025-07-18T19:45:16.811ZINFOoximeter (oximeter-agent): refreshing list of producers from Nexus collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d file = oximeter/collector/src/agent.rs:562
3382025-07-18T19:45:16.811ZDEBGoximeter: client request body = None method = GET uri = http://[fd00:1122:3344:101::c]:12221/metrics/collectors/650a8687-e31d-4bdf-8da8-edf2f453515c/producers?sort_by=id_ascending
3392025-07-18T19:45:16.815ZDEBGoximeter: client response result = Ok(Response { url: "http://[fd00:1122:3344:101::c]:12221/metrics/collectors/650a8687-e31d-4bdf-8da8-edf2f453515c/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "57184c47-ed6d-4af3-bd2b-d9b90bf717a4", "content-length": "857", "date": "Fri, 18 Jul 2025 19:45:16 GMT"} })
3402025-07-18T19:45:16.815ZDEBGoximeter: client request body = None method = GET uri = http://[fd00:1122:3344:101::c]:12221/metrics/collectors/650a8687-e31d-4bdf-8da8-edf2f453515c/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJiMjU2OGQ2OC1jMmYxLTRlYzUtOWY0MC00M2VjZjkxYTg5OGEifX0%3D
3412025-07-18T19:45:16.818ZDEBGoximeter: client response result = Ok(Response { url: "http://[fd00:1122:3344:101::c]:12221/metrics/collectors/650a8687-e31d-4bdf-8da8-edf2f453515c/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJiMjU2OGQ2OC1jMmYxLTRlYzUtOWY0MC00M2VjZjkxYTg5OGEifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "60cc3f54-bd73-4e12-9a6f-fb08c3c71d60", "content-length": "29", "date": "Fri, 18 Jul 2025 19:45:16 GMT"} })
3422025-07-18T19:45:16.818ZDEBGoximeter (oximeter-agent): registered new metric producer address = [fd00:1122:3344:101::2]:8001 collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 8c9d2fe9-a895-4c3f-a88b-f32ffc48b0d5
3432025-07-18T19:45:16.820ZINFOoximeter (oximeter-agent): refreshed list of producers from Nexus collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d file = oximeter/collector/src/agent.rs:637 n_current_tasks = 5 n_pruned_tasks = 0
3442025-07-18T19:45:16.843ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 2ab58cea-1973-4965-8759-ba7489e0fe93
3452025-07-18T19:45:16.843ZDEBGoximeter (collection-task): collecting from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 2ab58cea-1973-4965-8759-ba7489e0fe93
3462025-07-18T19:45:16.844ZDEBGoximeter (collection-task): collected results from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 2ab58cea-1973-4965-8759-ba7489e0fe93
3472025-07-18T19:45:16.845ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 3f5838b1-1901-4ccc-8412-bc0c1b44e959
3482025-07-18T19:45:16.845ZDEBGoximeter (collection-task): collecting from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 3f5838b1-1901-4ccc-8412-bc0c1b44e959
3492025-07-18T19:45:16.847ZDEBGoximeter (collection-task): collected results from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d n_results = 2 producer_id = 3f5838b1-1901-4ccc-8412-bc0c1b44e959
3502025-07-18T19:45:16.848ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 9dce788c-d2e1-4d5e-ac83-e6d8d56311b5
3512025-07-18T19:45:16.848ZDEBGoximeter (collection-task): collecting from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 9dce788c-d2e1-4d5e-ac83-e6d8d56311b5
3522025-07-18T19:45:16.850ZDEBGoximeter (collection-task): collected results from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 9dce788c-d2e1-4d5e-ac83-e6d8d56311b5
3532025-07-18T19:45:16.850ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = b2568d68-c2f1-4ec5-9f40-43ecf91a898a
3542025-07-18T19:45:16.850ZDEBGoximeter (collection-task): collecting from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = b2568d68-c2f1-4ec5-9f40-43ecf91a898a
3552025-07-18T19:45:16.851ZDEBGoximeter (collection-task): collected results from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d n_results = 5 producer_id = b2568d68-c2f1-4ec5-9f40-43ecf91a898a
3562025-07-18T19:45:17.819ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 8c9d2fe9-a895-4c3f-a88b-f32ffc48b0d5
3572025-07-18T19:45:17.819ZDEBGoximeter (collection-task): collecting from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 8c9d2fe9-a895-4c3f-a88b-f32ffc48b0d5
3582025-07-18T19:45:17.828ZDEBGoximeter (collection-task): collected results from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 8c9d2fe9-a895-4c3f-a88b-f32ffc48b0d5
3592025-07-18T19:45:17.842ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 2ab58cea-1973-4965-8759-ba7489e0fe93
3602025-07-18T19:45:17.842ZDEBGoximeter (collection-task): collecting from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 2ab58cea-1973-4965-8759-ba7489e0fe93
3612025-07-18T19:45:17.843ZDEBGoximeter (collection-task): collected results from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 2ab58cea-1973-4965-8759-ba7489e0fe93
3622025-07-18T19:45:17.848ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 9dce788c-d2e1-4d5e-ac83-e6d8d56311b5
3632025-07-18T19:45:17.848ZDEBGoximeter (collection-task): collecting from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 9dce788c-d2e1-4d5e-ac83-e6d8d56311b5
3642025-07-18T19:45:17.850ZDEBGoximeter (collection-task): collected results from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 9dce788c-d2e1-4d5e-ac83-e6d8d56311b5
3652025-07-18T19:45:18.819ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 8c9d2fe9-a895-4c3f-a88b-f32ffc48b0d5
3662025-07-18T19:45:18.819ZDEBGoximeter (collection-task): collecting from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 8c9d2fe9-a895-4c3f-a88b-f32ffc48b0d5
3672025-07-18T19:45:18.820ZDEBGoximeter (collection-task): collected results from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 8c9d2fe9-a895-4c3f-a88b-f32ffc48b0d5
3682025-07-18T19:45:18.843ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 2ab58cea-1973-4965-8759-ba7489e0fe93
3692025-07-18T19:45:18.843ZDEBGoximeter (collection-task): collecting from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 2ab58cea-1973-4965-8759-ba7489e0fe93
3702025-07-18T19:45:18.844ZDEBGoximeter (collection-task): collected results from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 2ab58cea-1973-4965-8759-ba7489e0fe93
3712025-07-18T19:45:18.848ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 9dce788c-d2e1-4d5e-ac83-e6d8d56311b5
3722025-07-18T19:45:18.848ZDEBGoximeter (collection-task): collecting from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 9dce788c-d2e1-4d5e-ac83-e6d8d56311b5
3732025-07-18T19:45:18.850ZDEBGoximeter (collection-task): collected results from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 9dce788c-d2e1-4d5e-ac83-e6d8d56311b5
3742025-07-18T19:45:19.818ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 8c9d2fe9-a895-4c3f-a88b-f32ffc48b0d5
3752025-07-18T19:45:19.818ZDEBGoximeter (collection-task): collecting from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 8c9d2fe9-a895-4c3f-a88b-f32ffc48b0d5
3762025-07-18T19:45:19.820ZDEBGoximeter (collection-task): collected results from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 8c9d2fe9-a895-4c3f-a88b-f32ffc48b0d5
3772025-07-18T19:45:19.843ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 2ab58cea-1973-4965-8759-ba7489e0fe93
3782025-07-18T19:45:19.843ZDEBGoximeter (collection-task): collecting from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 2ab58cea-1973-4965-8759-ba7489e0fe93
3792025-07-18T19:45:19.844ZDEBGoximeter (collection-task): collected results from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 2ab58cea-1973-4965-8759-ba7489e0fe93
3802025-07-18T19:45:19.848ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 9dce788c-d2e1-4d5e-ac83-e6d8d56311b5
3812025-07-18T19:45:19.848ZDEBGoximeter (collection-task): collecting from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 9dce788c-d2e1-4d5e-ac83-e6d8d56311b5
3822025-07-18T19:45:19.850ZDEBGoximeter (collection-task): collected results from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 9dce788c-d2e1-4d5e-ac83-e6d8d56311b5
3832025-07-18T19:45:20.819ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 8c9d2fe9-a895-4c3f-a88b-f32ffc48b0d5
3842025-07-18T19:45:20.819ZDEBGoximeter (collection-task): collecting from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 8c9d2fe9-a895-4c3f-a88b-f32ffc48b0d5
3852025-07-18T19:45:20.821ZDEBGoximeter (collection-task): collected results from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 8c9d2fe9-a895-4c3f-a88b-f32ffc48b0d5
3862025-07-18T19:45:20.843ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 2ab58cea-1973-4965-8759-ba7489e0fe93
3872025-07-18T19:45:20.843ZDEBGoximeter (collection-task): collecting from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 2ab58cea-1973-4965-8759-ba7489e0fe93
3882025-07-18T19:45:20.844ZDEBGoximeter (collection-task): collected results from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 2ab58cea-1973-4965-8759-ba7489e0fe93
3892025-07-18T19:45:20.848ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 9dce788c-d2e1-4d5e-ac83-e6d8d56311b5
3902025-07-18T19:45:20.848ZDEBGoximeter (collection-task): collecting from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 9dce788c-d2e1-4d5e-ac83-e6d8d56311b5
3912025-07-18T19:45:20.850ZDEBGoximeter (collection-task): collected results from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 9dce788c-d2e1-4d5e-ac83-e6d8d56311b5
3922025-07-18T19:45:21.509ZDEBGoximeter (results-sink): inserting 218 samples into database collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d
3932025-07-18T19:45:21.509ZDEBGoximeter (results-sink-cluster): inserting 218 samples into database collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d
3942025-07-18T19:45:21.509ZDEBGoximeter (clickhouse-client): unrolling 218 total samples collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d id = c6f04c80-a0b6-4ed1-9a43-264260a5e2a8
3952025-07-18T19:45:21.519ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d id = c6f04c80-a0b6-4ed1-9a43-264260a5e2a8 n_rows = 80 table_name = fields_string
3962025-07-18T19:45:21.521ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d id = c6f04c80-a0b6-4ed1-9a43-264260a5e2a8 n_rows = 12 table_name = fields_u16
3972025-07-18T19:45:21.522ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d id = c6f04c80-a0b6-4ed1-9a43-264260a5e2a8 n_rows = 76 table_name = fields_uuid
3982025-07-18T19:45:21.523ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d id = c6f04c80-a0b6-4ed1-9a43-264260a5e2a8 n_rows = 150 table_name = measurements_cumulativeu64
3992025-07-18T19:45:21.525ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d id = c6f04c80-a0b6-4ed1-9a43-264260a5e2a8 n_rows = 12 table_name = measurements_histogramu64
4002025-07-18T19:45:21.527ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d id = c6f04c80-a0b6-4ed1-9a43-264260a5e2a8 n_rows = 56 table_name = measurements_u64
4012025-07-18T19:45:21.610ZWARNoximeter (results-sink-cluster): failed to insert some results into metric DB collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
4022025-07-18T19:45:21.819ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 8c9d2fe9-a895-4c3f-a88b-f32ffc48b0d5
4032025-07-18T19:45:21.819ZDEBGoximeter (collection-task): collecting from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 8c9d2fe9-a895-4c3f-a88b-f32ffc48b0d5
4042025-07-18T19:45:21.820ZDEBGoximeter (collection-task): collected results from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 8c9d2fe9-a895-4c3f-a88b-f32ffc48b0d5
4052025-07-18T19:45:21.843ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 2ab58cea-1973-4965-8759-ba7489e0fe93
4062025-07-18T19:45:21.843ZDEBGoximeter (collection-task): collecting from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 2ab58cea-1973-4965-8759-ba7489e0fe93
4072025-07-18T19:45:21.843ZDEBGoximeter (collection-task): collected results from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 2ab58cea-1973-4965-8759-ba7489e0fe93
4082025-07-18T19:45:21.847ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 9dce788c-d2e1-4d5e-ac83-e6d8d56311b5
4092025-07-18T19:45:21.847ZDEBGoximeter (collection-task): collecting from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 9dce788c-d2e1-4d5e-ac83-e6d8d56311b5
4102025-07-18T19:45:21.850ZDEBGoximeter (collection-task): collected results from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 9dce788c-d2e1-4d5e-ac83-e6d8d56311b5
4112025-07-18T19:45:22.819ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 8c9d2fe9-a895-4c3f-a88b-f32ffc48b0d5
4122025-07-18T19:45:22.819ZDEBGoximeter (collection-task): collecting from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 8c9d2fe9-a895-4c3f-a88b-f32ffc48b0d5
4132025-07-18T19:45:22.820ZDEBGoximeter (collection-task): collected results from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 8c9d2fe9-a895-4c3f-a88b-f32ffc48b0d5
4142025-07-18T19:45:22.842ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 2ab58cea-1973-4965-8759-ba7489e0fe93
4152025-07-18T19:45:22.842ZDEBGoximeter (collection-task): collecting from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 2ab58cea-1973-4965-8759-ba7489e0fe93
4162025-07-18T19:45:22.843ZDEBGoximeter (collection-task): collected results from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 2ab58cea-1973-4965-8759-ba7489e0fe93
4172025-07-18T19:45:22.848ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 9dce788c-d2e1-4d5e-ac83-e6d8d56311b5
4182025-07-18T19:45:22.848ZDEBGoximeter (collection-task): collecting from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 9dce788c-d2e1-4d5e-ac83-e6d8d56311b5
4192025-07-18T19:45:22.850ZDEBGoximeter (collection-task): collected results from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 9dce788c-d2e1-4d5e-ac83-e6d8d56311b5
4202025-07-18T19:45:23.819ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 8c9d2fe9-a895-4c3f-a88b-f32ffc48b0d5
4212025-07-18T19:45:23.819ZDEBGoximeter (collection-task): collecting from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 8c9d2fe9-a895-4c3f-a88b-f32ffc48b0d5
4222025-07-18T19:45:23.820ZDEBGoximeter (collection-task): collected results from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 8c9d2fe9-a895-4c3f-a88b-f32ffc48b0d5
4232025-07-18T19:45:23.843ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 2ab58cea-1973-4965-8759-ba7489e0fe93
4242025-07-18T19:45:23.843ZDEBGoximeter (collection-task): collecting from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 2ab58cea-1973-4965-8759-ba7489e0fe93
4252025-07-18T19:45:23.844ZDEBGoximeter (collection-task): collected results from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 2ab58cea-1973-4965-8759-ba7489e0fe93
4262025-07-18T19:45:23.848ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 9dce788c-d2e1-4d5e-ac83-e6d8d56311b5
4272025-07-18T19:45:23.848ZDEBGoximeter (collection-task): collecting from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d producer_id = 9dce788c-d2e1-4d5e-ac83-e6d8d56311b5
4282025-07-18T19:45:23.850ZDEBGoximeter (collection-task): collected results from producer collector_id = 650a8687-e31d-4bdf-8da8-edf2f453515c collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 9dce788c-d2e1-4d5e-ac83-e6d8d56311b5