Max level shown:
1 [ Dec 16 09:14:11 Enabled. ]
2 [ Dec 16 09:14:11 Rereading configuration. ]
3 [ Dec 16 09:14:13 Rereading configuration. ]
4 [ Dec 16 09:14:17 Executing start method ("ctrun -l child -o noorphan,regent /opt/oxide/oximeter-collector/bin/oximeter run /var/svc/manifest/site/oximeter/config.toml --address [fd00:1122:3344:101::d]:12223 --id e0dc1a92-16e0-44d0-bf37-d9bb30e63782 &"). ]
5 [ Dec 16 09:14:17 Method "start" exited with status 0. ]
6 note: configured to log to "/dev/stdout"
72025-12-16T09:14:17.520ZDEBGoximeter: registered DTrace probes
82025-12-16T09:14:17.523ZINFOoximeter: starting oximeter server file = oximeter/collector/src/lib.rs:264
92025-12-16T09:14:17.531ZDEBGoximeter: creating ClickHouse client
102025-12-16T09:14:17.570ZDEBGoximeter (clickhouse-client): registered USDT probes collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d id = 2e611522-2449-41fe-882b-03e765f06c58
112025-12-16T09:14:21.435ZWARNoximeter (clickhouse-client): oximeter database does not exist, or is out-of-date collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d file = oximeter/db/src/client/mod.rs:900 id = 2e611522-2449-41fe-882b-03e765f06c58
122025-12-16T09:14:21.435ZDEBGoximeter (oximeter-agent): oximeter database does not exist, creating collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d
132025-12-16T09:14:21.435ZINFOoximeter (clickhouse-client): reading db version collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d file = oximeter/db/src/client/mod.rs:800 id = 2e611522-2449-41fe-882b-03e765f06c58
142025-12-16T09:14:21.436ZWARNoximeter (clickhouse-client): oximeter database does not exist, or is out-of-date collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d file = oximeter/db/src/client/mod.rs:900 id = 2e611522-2449-41fe-882b-03e765f06c58
152025-12-16T09:14:21.436ZINFOoximeter (clickhouse-client): read oximeter database version collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d file = oximeter/db/src/client/mod.rs:805 id = 2e611522-2449-41fe-882b-03e765f06c58 version = 0
162025-12-16T09:14:21.436ZINFOoximeter (clickhouse-client): initializing oximeter schema collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d file = oximeter/db/src/client/mod.rs:810 id = 2e611522-2449-41fe-882b-03e765f06c58
172025-12-16T09:14:21.436ZDEBGoximeter (clickhouse-client): initializing ClickHouse database collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d id = 2e611522-2449-41fe-882b-03e765f06c58
182025-12-16T09:14:21.695ZINFOoximeter (clickhouse-client): inserting current version collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d file = oximeter/db/src/client/mod.rs:831 id = 2e611522-2449-41fe-882b-03e765f06c58 version = 14
192025-12-16T09:14:21.725ZDEBGoximeter (clickhouse-client): registered USDT probes collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d id = 7d6fd972-7393-4cf0-929d-2cb2a0d74dd2
202025-12-16T09:14:21.749ZDEBGoximeter (dropshot): registered endpoint local_addr = [fd00:1122:3344:101::d]:12223 method = GET path = /info versions = all
212025-12-16T09:14:21.749ZDEBGoximeter (dropshot): registered endpoint local_addr = [fd00:1122:3344:101::d]:12223 method = GET path = /producers versions = all
222025-12-16T09:14:21.749ZDEBGoximeter (dropshot): registered endpoint local_addr = [fd00:1122:3344:101::d]:12223 method = DELETE path = /producers/{producer_id} versions = all
232025-12-16T09:14:21.749ZDEBGoximeter (dropshot): registered endpoint local_addr = [fd00:1122:3344:101::d]:12223 method = GET path = /producers/{producer_id} versions = all
242025-12-16T09:14:21.749ZINFOoximeter (dropshot): listening file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.6/src/server.rs:328 local_addr = [fd00:1122:3344:101::d]:12223
252025-12-16T09:14:21.760ZDEBGoximeter (dropshot): successfully registered DTrace USDT probes local_addr = [fd00:1122:3344:101::d]:12223
262025-12-16T09:14:21.760ZDEBGoximeter: registered USDT probes
272025-12-16T09:14:21.760ZDEBGoximeter: contacting nexus
282025-12-16T09:14:21.935ZDEBGoximeter: client request body = Some(Body) method = POST uri = http://[fd00:1122:3344:101::c]:12221/metrics/collectors
292025-12-16T09:14:21.937ZDEBGoximeter: client response result = Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::c]:12221/metrics/collectors", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", [fd00:1122:3344:101::c]:12221, Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
302025-12-16T09:14:21.937ZWARNoximeter: failed to contact nexus, will retry in 171.429395ms error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/collectors)" file = oximeter/collector/src/lib.rs:409
312025-12-16T09:14:22.110ZDEBGoximeter: contacting nexus
322025-12-16T09:14:22.110ZDEBGoximeter: client request body = Some(Body) method = POST uri = http://[fd00:1122:3344:101::c]:12221/metrics/collectors
332025-12-16T09:14:22.110ZDEBGoximeter: client response result = Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::c]:12221/metrics/collectors", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", [fd00:1122:3344:101::c]:12221, Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
342025-12-16T09:14:22.110ZWARNoximeter: failed to contact nexus, will retry in 530.267026ms error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/collectors)" file = oximeter/collector/src/lib.rs:409
352025-12-16T09:14:22.641ZDEBGoximeter: contacting nexus
362025-12-16T09:14:22.641ZDEBGoximeter: client request body = Some(Body) method = POST uri = http://[fd00:1122:3344:101::c]:12221/metrics/collectors
372025-12-16T09:14:22.641ZDEBGoximeter: client response result = Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::c]:12221/metrics/collectors", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", [fd00:1122:3344:101::c]:12221, Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
382025-12-16T09:14:22.642ZWARNoximeter: failed to contact nexus, will retry in 1.48900249s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/collectors)" file = oximeter/collector/src/lib.rs:409
392025-12-16T09:14:24.132ZDEBGoximeter: contacting nexus
402025-12-16T09:14:24.132ZDEBGoximeter: client request body = Some(Body) method = POST uri = http://[fd00:1122:3344:101::c]:12221/metrics/collectors
412025-12-16T09:14:24.276ZDEBGoximeter: client response result = Ok(Response { url: "http://[fd00:1122:3344:101::c]:12221/metrics/collectors", status: 204, headers: {"x-request-id": "59ddc7d2-9fdc-4560-b798-2157c78e735b", "date": "Tue, 16 Dec 2025 09:14:24 GMT"} })
422025-12-16T09:14:24.276ZINFOoximeter: oximeter registered with nexus file = oximeter/collector/src/lib.rs:427 id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782
432025-12-16T09:14:24.279ZINFOoximeter (oximeter-agent): starting refresh list task collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d file = oximeter/collector/src/agent.rs:554
442025-12-16T09:14:24.280ZINFOoximeter (oximeter-agent): refreshing list of producers from Nexus collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d file = oximeter/collector/src/agent.rs:557
452025-12-16T09:14:24.286ZDEBGoximeter: client request body = None method = GET uri = http://[fd00:1122:3344:101::c]:12221/metrics/collectors/e0dc1a92-16e0-44d0-bf37-d9bb30e63782/producers?sort_by=id_ascending
462025-12-16T09:14:24.388ZDEBGoximeter: client response result = Ok(Response { url: "http://[fd00:1122:3344:101::c]:12221/metrics/collectors/e0dc1a92-16e0-44d0-bf37-d9bb30e63782/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "04b675ba-da78-48de-b872-c9cdcb358a0f", "content-length": "29", "date": "Tue, 16 Dec 2025 09:14:24 GMT"} })
472025-12-16T09:14:24.388ZINFOoximeter (oximeter-agent): refreshed list of producers from Nexus collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d file = oximeter/collector/src/agent.rs:632 n_current_tasks = 0 n_pruned_tasks = 0
482025-12-16T09:14:39.280ZINFOoximeter (oximeter-agent): refreshing list of producers from Nexus collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d file = oximeter/collector/src/agent.rs:557
492025-12-16T09:14:39.280ZDEBGoximeter: client request body = None method = GET uri = http://[fd00:1122:3344:101::c]:12221/metrics/collectors/e0dc1a92-16e0-44d0-bf37-d9bb30e63782/producers?sort_by=id_ascending
502025-12-16T09:14:39.283ZDEBGoximeter: client response result = Ok(Response { url: "http://[fd00:1122:3344:101::c]:12221/metrics/collectors/e0dc1a92-16e0-44d0-bf37-d9bb30e63782/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "100f3ecc-609a-48f1-b7d6-324d872e9ae0", "content-length": "447", "date": "Tue, 16 Dec 2025 09:14:39 GMT"} })
512025-12-16T09:14:39.285ZDEBGoximeter: client request body = None method = GET uri = http://[fd00:1122:3344:101::c]:12221/metrics/collectors/e0dc1a92-16e0-44d0-bf37-d9bb30e63782/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiI5NGRhYzQ5Mi0yYTc5LTRkNjgtODlhMi1mNDhmZTUxMGEzZWIifX0%3D
522025-12-16T09:14:39.288ZDEBGoximeter: client response result = Ok(Response { url: "http://[fd00:1122:3344:101::c]:12221/metrics/collectors/e0dc1a92-16e0-44d0-bf37-d9bb30e63782/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiI5NGRhYzQ5Mi0yYTc5LTRkNjgtODlhMi1mNDhmZTUxMGEzZWIifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "5c28ff3d-f0d6-422e-85ef-e74670228612", "content-length": "29", "date": "Tue, 16 Dec 2025 09:14:39 GMT"} })
532025-12-16T09:14:39.288ZDEBGoximeter (oximeter-agent): registered new metric producer address = [fd00:1122:3344:101::2]:44059 collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d producer_id = 10373832-fc98-4a20-8740-604ad8bcd1a7
542025-12-16T09:14:39.288ZDEBGoximeter (oximeter-agent): registered new metric producer address = [fd00:1122:3344:101::1]:8001 collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d producer_id = 94dac492-2a79-4d68-89a2-f48fe510a3eb
552025-12-16T09:14:39.288ZINFOoximeter (oximeter-agent): refreshed list of producers from Nexus collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d file = oximeter/collector/src/agent.rs:632 n_current_tasks = 2 n_pruned_tasks = 0
562025-12-16T09:14:39.289ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d producer_id = 10373832-fc98-4a20-8740-604ad8bcd1a7 producer_ip = fd00:1122:3344:101::2 producer_port = 44059
572025-12-16T09:14:39.289ZDEBGoximeter (collection-task): reporting oximeter self-collection statistics collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d producer_id = 94dac492-2a79-4d68-89a2-f48fe510a3eb producer_ip = fd00:1122:3344:101::1 producer_port = 8001
582025-12-16T09:14:39.289ZDEBGoximeter (collection-task): reporting oximeter self-collection statistics collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d producer_id = 10373832-fc98-4a20-8740-604ad8bcd1a7 producer_ip = fd00:1122:3344:101::2 producer_port = 44059
592025-12-16T09:14:39.329ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d producer_id = 94dac492-2a79-4d68-89a2-f48fe510a3eb producer_ip = fd00:1122:3344:101::1 producer_port = 8001
602025-12-16T09:14:39.425ZDEBGoximeter (collection-task): collecting from producer collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d producer_id = 10373832-fc98-4a20-8740-604ad8bcd1a7 producer_ip = fd00:1122:3344:101::2 producer_port = 44059
612025-12-16T09:14:39.426ZDEBGoximeter (collection-task): collecting from producer collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d producer_id = 94dac492-2a79-4d68-89a2-f48fe510a3eb producer_ip = fd00:1122:3344:101::1 producer_port = 8001
622025-12-16T09:14:39.441ZDEBGoximeter (collection-task): collected results from producer collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 94dac492-2a79-4d68-89a2-f48fe510a3eb producer_ip = fd00:1122:3344:101::1 producer_port = 8001
632025-12-16T09:14:39.443ZDEBGoximeter (collection-task): collected results from producer collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d n_results = 2 producer_id = 10373832-fc98-4a20-8740-604ad8bcd1a7 producer_ip = fd00:1122:3344:101::2 producer_port = 44059
642025-12-16T09:14:40.289ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d producer_id = 94dac492-2a79-4d68-89a2-f48fe510a3eb producer_ip = fd00:1122:3344:101::1 producer_port = 8001
652025-12-16T09:14:40.289ZDEBGoximeter (collection-task): collecting from producer collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d producer_id = 94dac492-2a79-4d68-89a2-f48fe510a3eb producer_ip = fd00:1122:3344:101::1 producer_port = 8001
662025-12-16T09:14:40.291ZDEBGoximeter (collection-task): collected results from producer collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 94dac492-2a79-4d68-89a2-f48fe510a3eb producer_ip = fd00:1122:3344:101::1 producer_port = 8001
672025-12-16T09:14:41.289ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d producer_id = 94dac492-2a79-4d68-89a2-f48fe510a3eb producer_ip = fd00:1122:3344:101::1 producer_port = 8001
682025-12-16T09:14:41.289ZDEBGoximeter (collection-task): collecting from producer collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d producer_id = 94dac492-2a79-4d68-89a2-f48fe510a3eb producer_ip = fd00:1122:3344:101::1 producer_port = 8001
692025-12-16T09:14:41.292ZDEBGoximeter (collection-task): collected results from producer collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 94dac492-2a79-4d68-89a2-f48fe510a3eb producer_ip = fd00:1122:3344:101::1 producer_port = 8001
702025-12-16T09:14:41.725ZDEBGoximeter (results-sink): inserting 90 samples into database collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d
712025-12-16T09:14:41.725ZDEBGoximeter (results-sink-cluster): inserting 90 samples into database collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d
722025-12-16T09:14:41.726ZDEBGoximeter (clickhouse-client): unrolling 90 total samples collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d id = 2e611522-2449-41fe-882b-03e765f06c58
732025-12-16T09:14:41.728ZDEBGoximeter (clickhouse-client): retrieving timeseries schema from database collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d id = 2e611522-2449-41fe-882b-03e765f06c58
742025-12-16T09:14:41.732ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d id = 2e611522-2449-41fe-882b-03e765f06c58 schema = TimeseriesSchema { timeseries_name: TimeseriesName("oximeter_collector:collections"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "base_route", field_type: String, source: Metric, description: "" }, FieldSchema { name: "collector_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "collector_ip", field_type: IpAddr, source: Target, description: "" }, FieldSchema { name: "collector_port", field_type: U16, source: Target, description: "" }, FieldSchema { name: "producer_id", field_type: Uuid, source: Metric, description: "" }, FieldSchema { name: "producer_ip", field_type: IpAddr, source: Metric, description: "" }, FieldSchema { name: "producer_port", field_type: U16, source: Metric, description: "" }}, datum_type: CumulativeU64, version: 1, authz_scope: Fleet, units: Count, created: 2025-12-16T09:14:41.728742738Z } timeseries_name = oximeter_collector:collections
752025-12-16T09:14:41.736ZDEBGoximeter (clickhouse-client): retrieving timeseries schema from database collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d id = 2e611522-2449-41fe-882b-03e765f06c58
762025-12-16T09:14:41.746ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d id = 2e611522-2449-41fe-882b-03e765f06c58 schema = TimeseriesSchema { timeseries_name: TimeseriesName("ddm_router:originated_underlay_prefixes"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "hostname", field_type: String, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }}, datum_type: U64, version: 1, authz_scope: Fleet, units: Count, created: 2025-12-16T09:14:41.736879815Z } timeseries_name = ddm_router:originated_underlay_prefixes
772025-12-16T09:14:41.746ZDEBGoximeter (clickhouse-client): retrieving timeseries schema from database collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d id = 2e611522-2449-41fe-882b-03e765f06c58
782025-12-16T09:14:41.751ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d id = 2e611522-2449-41fe-882b-03e765f06c58 schema = TimeseriesSchema { timeseries_name: TimeseriesName("ddm_router:originated_tunnel_endpoints"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "hostname", field_type: String, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }}, datum_type: U64, version: 1, authz_scope: Fleet, units: Count, created: 2025-12-16T09:14:41.746467303Z } timeseries_name = ddm_router:originated_tunnel_endpoints
792025-12-16T09:14:41.751ZDEBGoximeter (clickhouse-client): retrieving timeseries schema from database collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d id = 2e611522-2449-41fe-882b-03e765f06c58
802025-12-16T09:14:41.752ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d id = 2e611522-2449-41fe-882b-03e765f06c58 schema = TimeseriesSchema { timeseries_name: TimeseriesName("ddm_session:solicitations_sent"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "hostname", field_type: String, source: Target, description: "" }, FieldSchema { name: "interface", field_type: String, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }}, datum_type: CumulativeU64, version: 1, authz_scope: Fleet, units: Count, created: 2025-12-16T09:14:41.751604037Z } timeseries_name = ddm_session:solicitations_sent
812025-12-16T09:14:41.752ZDEBGoximeter (clickhouse-client): retrieving timeseries schema from database collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d id = 2e611522-2449-41fe-882b-03e765f06c58
822025-12-16T09:14:41.753ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d id = 2e611522-2449-41fe-882b-03e765f06c58 schema = TimeseriesSchema { timeseries_name: TimeseriesName("ddm_session:solicitations_received"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "hostname", field_type: String, source: Target, description: "" }, FieldSchema { name: "interface", field_type: String, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }}, datum_type: CumulativeU64, version: 1, authz_scope: Fleet, units: Count, created: 2025-12-16T09:14:41.752599321Z } timeseries_name = ddm_session:solicitations_received
832025-12-16T09:14:41.753ZDEBGoximeter (clickhouse-client): retrieving timeseries schema from database collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d id = 2e611522-2449-41fe-882b-03e765f06c58
842025-12-16T09:14:41.754ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d id = 2e611522-2449-41fe-882b-03e765f06c58 schema = TimeseriesSchema { timeseries_name: TimeseriesName("ddm_session:advertisements_sent"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "hostname", field_type: String, source: Target, description: "" }, FieldSchema { name: "interface", field_type: String, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }}, datum_type: CumulativeU64, version: 1, authz_scope: Fleet, units: Count, created: 2025-12-16T09:14:41.753565805Z } timeseries_name = ddm_session:advertisements_sent
852025-12-16T09:14:41.754ZDEBGoximeter (clickhouse-client): retrieving timeseries schema from database collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d id = 2e611522-2449-41fe-882b-03e765f06c58
862025-12-16T09:14:41.755ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d id = 2e611522-2449-41fe-882b-03e765f06c58 schema = TimeseriesSchema { timeseries_name: TimeseriesName("ddm_session:advertisements_received"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "hostname", field_type: String, source: Target, description: "" }, FieldSchema { name: "interface", field_type: String, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }}, datum_type: CumulativeU64, version: 1, authz_scope: Fleet, units: Count, created: 2025-12-16T09:14:41.754564909Z } timeseries_name = ddm_session:advertisements_received
872025-12-16T09:14:41.755ZDEBGoximeter (clickhouse-client): retrieving timeseries schema from database collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d id = 2e611522-2449-41fe-882b-03e765f06c58
882025-12-16T09:14:41.756ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d id = 2e611522-2449-41fe-882b-03e765f06c58 schema = TimeseriesSchema { timeseries_name: TimeseriesName("ddm_session:peer_expirations"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "hostname", field_type: String, source: Target, description: "" }, FieldSchema { name: "interface", field_type: String, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }}, datum_type: CumulativeU64, version: 1, authz_scope: Fleet, units: Count, created: 2025-12-16T09:14:41.755495233Z } timeseries_name = ddm_session:peer_expirations
892025-12-16T09:14:41.756ZDEBGoximeter (clickhouse-client): retrieving timeseries schema from database collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d id = 2e611522-2449-41fe-882b-03e765f06c58
902025-12-16T09:14:41.757ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d id = 2e611522-2449-41fe-882b-03e765f06c58 schema = TimeseriesSchema { timeseries_name: TimeseriesName("ddm_session:peer_address_changes"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "hostname", field_type: String, source: Target, description: "" }, FieldSchema { name: "interface", field_type: String, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }}, datum_type: CumulativeU64, version: 1, authz_scope: Fleet, units: Count, created: 2025-12-16T09:14:41.756429526Z } timeseries_name = ddm_session:peer_address_changes
912025-12-16T09:14:41.757ZDEBGoximeter (clickhouse-client): retrieving timeseries schema from database collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d id = 2e611522-2449-41fe-882b-03e765f06c58
922025-12-16T09:14:41.758ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d id = 2e611522-2449-41fe-882b-03e765f06c58 schema = TimeseriesSchema { timeseries_name: TimeseriesName("ddm_session:peer_sessions_established"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "hostname", field_type: String, source: Target, description: "" }, FieldSchema { name: "interface", field_type: String, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }}, datum_type: CumulativeU64, version: 1, authz_scope: Fleet, units: Count, created: 2025-12-16T09:14:41.757364079Z } timeseries_name = ddm_session:peer_sessions_established
932025-12-16T09:14:41.758ZDEBGoximeter (clickhouse-client): retrieving timeseries schema from database collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d id = 2e611522-2449-41fe-882b-03e765f06c58
942025-12-16T09:14:41.759ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d id = 2e611522-2449-41fe-882b-03e765f06c58 schema = TimeseriesSchema { timeseries_name: TimeseriesName("ddm_session:updates_sent"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "hostname", field_type: String, source: Target, description: "" }, FieldSchema { name: "interface", field_type: String, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }}, datum_type: CumulativeU64, version: 1, authz_scope: Fleet, units: Count, created: 2025-12-16T09:14:41.758275303Z } timeseries_name = ddm_session:updates_sent
952025-12-16T09:14:41.759ZDEBGoximeter (clickhouse-client): retrieving timeseries schema from database collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d id = 2e611522-2449-41fe-882b-03e765f06c58
962025-12-16T09:14:41.760ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d id = 2e611522-2449-41fe-882b-03e765f06c58 schema = TimeseriesSchema { timeseries_name: TimeseriesName("ddm_session:updates_received"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "hostname", field_type: String, source: Target, description: "" }, FieldSchema { name: "interface", field_type: String, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }}, datum_type: CumulativeU64, version: 1, authz_scope: Fleet, units: Count, created: 2025-12-16T09:14:41.759257027Z } timeseries_name = ddm_session:updates_received
972025-12-16T09:14:41.760ZDEBGoximeter (clickhouse-client): retrieving timeseries schema from database collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d id = 2e611522-2449-41fe-882b-03e765f06c58
982025-12-16T09:14:41.761ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d id = 2e611522-2449-41fe-882b-03e765f06c58 schema = TimeseriesSchema { timeseries_name: TimeseriesName("ddm_session:update_send_fail"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "hostname", field_type: String, source: Target, description: "" }, FieldSchema { name: "interface", field_type: String, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }}, datum_type: CumulativeU64, version: 1, authz_scope: Fleet, units: Count, created: 2025-12-16T09:14:41.760191430Z } timeseries_name = ddm_session:update_send_fail
992025-12-16T09:14:41.761ZDEBGoximeter (clickhouse-client): retrieving timeseries schema from database collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d id = 2e611522-2449-41fe-882b-03e765f06c58
1002025-12-16T09:14:41.762ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d id = 2e611522-2449-41fe-882b-03e765f06c58 schema = TimeseriesSchema { timeseries_name: TimeseriesName("ddm_session:imported_underlay_prefixes"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "hostname", field_type: String, source: Target, description: "" }, FieldSchema { name: "interface", field_type: String, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }}, datum_type: U64, version: 1, authz_scope: Fleet, units: Count, created: 2025-12-16T09:14:41.761134034Z } timeseries_name = ddm_session:imported_underlay_prefixes
1012025-12-16T09:14:41.762ZDEBGoximeter (clickhouse-client): retrieving timeseries schema from database collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d id = 2e611522-2449-41fe-882b-03e765f06c58
1022025-12-16T09:14:41.762ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d id = 2e611522-2449-41fe-882b-03e765f06c58 schema = TimeseriesSchema { timeseries_name: TimeseriesName("ddm_session:imported_tunnel_endpoints"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "hostname", field_type: String, source: Target, description: "" }, FieldSchema { name: "interface", field_type: String, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }}, datum_type: U64, version: 1, authz_scope: Fleet, units: Count, created: 2025-12-16T09:14:41.762093408Z } timeseries_name = ddm_session:imported_tunnel_endpoints
1032025-12-16T09:14:41.763ZDEBGoximeter (clickhouse-client): retrieving timeseries schema from database collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d id = 2e611522-2449-41fe-882b-03e765f06c58
1042025-12-16T09:14:41.764ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d id = 2e611522-2449-41fe-882b-03e765f06c58 schema = TimeseriesSchema { timeseries_name: TimeseriesName("http_service:request_latency_histogram"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "name", field_type: String, source: Target, description: "" }, FieldSchema { name: "operation_id", field_type: String, source: Metric, description: "" }, FieldSchema { name: "status_code", field_type: U16, source: Metric, description: "" }}, datum_type: HistogramU64, version: 1, authz_scope: Fleet, units: Count, created: 2025-12-16T09:14:41.763538588Z } timeseries_name = http_service:request_latency_histogram
1052025-12-16T09:14:41.767ZDEBGoximeter (clickhouse-client): inserting 16 new timeseries schema collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d id = 2e611522-2449-41fe-882b-03e765f06c58
1062025-12-16T09:14:41.784ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d id = 2e611522-2449-41fe-882b-03e765f06c58 n_rows = 4 table_name = fields_ipaddr
1072025-12-16T09:14:41.786ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d id = 2e611522-2449-41fe-882b-03e765f06c58 n_rows = 48 table_name = fields_string
1082025-12-16T09:14:41.787ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d id = 2e611522-2449-41fe-882b-03e765f06c58 n_rows = 14 table_name = fields_u16
1092025-12-16T09:14:41.789ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d id = 2e611522-2449-41fe-882b-03e765f06c58 n_rows = 42 table_name = fields_uuid
1102025-12-16T09:14:41.797ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d id = 2e611522-2449-41fe-882b-03e765f06c58 n_rows = 62 table_name = measurements_cumulativeu64
1112025-12-16T09:14:41.803ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d id = 2e611522-2449-41fe-882b-03e765f06c58 n_rows = 10 table_name = measurements_histogramu64
1122025-12-16T09:14:41.804ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d id = 2e611522-2449-41fe-882b-03e765f06c58 n_rows = 18 table_name = measurements_u64
1132025-12-16T09:14:41.830ZWARNoximeter (results-sink-cluster): failed to insert some results into metric DB collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
1142025-12-16T09:14:42.289ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d producer_id = 94dac492-2a79-4d68-89a2-f48fe510a3eb producer_ip = fd00:1122:3344:101::1 producer_port = 8001
1152025-12-16T09:14:42.289ZDEBGoximeter (collection-task): collecting from producer collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d producer_id = 94dac492-2a79-4d68-89a2-f48fe510a3eb producer_ip = fd00:1122:3344:101::1 producer_port = 8001
1162025-12-16T09:14:42.291ZDEBGoximeter (collection-task): collected results from producer collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 94dac492-2a79-4d68-89a2-f48fe510a3eb producer_ip = fd00:1122:3344:101::1 producer_port = 8001
1172025-12-16T09:14:43.289ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d producer_id = 94dac492-2a79-4d68-89a2-f48fe510a3eb producer_ip = fd00:1122:3344:101::1 producer_port = 8001
1182025-12-16T09:14:43.289ZDEBGoximeter (collection-task): collecting from producer collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d producer_id = 94dac492-2a79-4d68-89a2-f48fe510a3eb producer_ip = fd00:1122:3344:101::1 producer_port = 8001
1192025-12-16T09:14:43.292ZDEBGoximeter (collection-task): collected results from producer collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 94dac492-2a79-4d68-89a2-f48fe510a3eb producer_ip = fd00:1122:3344:101::1 producer_port = 8001
1202025-12-16T09:14:44.289ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d producer_id = 94dac492-2a79-4d68-89a2-f48fe510a3eb producer_ip = fd00:1122:3344:101::1 producer_port = 8001
1212025-12-16T09:14:44.289ZDEBGoximeter (collection-task): collecting from producer collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d producer_id = 94dac492-2a79-4d68-89a2-f48fe510a3eb producer_ip = fd00:1122:3344:101::1 producer_port = 8001
1222025-12-16T09:14:44.292ZDEBGoximeter (collection-task): collected results from producer collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 94dac492-2a79-4d68-89a2-f48fe510a3eb producer_ip = fd00:1122:3344:101::1 producer_port = 8001
1232025-12-16T09:14:45.289ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d producer_id = 94dac492-2a79-4d68-89a2-f48fe510a3eb producer_ip = fd00:1122:3344:101::1 producer_port = 8001
1242025-12-16T09:14:45.289ZDEBGoximeter (collection-task): collecting from producer collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d producer_id = 94dac492-2a79-4d68-89a2-f48fe510a3eb producer_ip = fd00:1122:3344:101::1 producer_port = 8001
1252025-12-16T09:14:45.292ZDEBGoximeter (collection-task): collected results from producer collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 94dac492-2a79-4d68-89a2-f48fe510a3eb producer_ip = fd00:1122:3344:101::1 producer_port = 8001
1262025-12-16T09:14:46.289ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d producer_id = 94dac492-2a79-4d68-89a2-f48fe510a3eb producer_ip = fd00:1122:3344:101::1 producer_port = 8001
1272025-12-16T09:14:46.289ZDEBGoximeter (collection-task): collecting from producer collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d producer_id = 94dac492-2a79-4d68-89a2-f48fe510a3eb producer_ip = fd00:1122:3344:101::1 producer_port = 8001
1282025-12-16T09:14:46.292ZDEBGoximeter (collection-task): collected results from producer collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 94dac492-2a79-4d68-89a2-f48fe510a3eb producer_ip = fd00:1122:3344:101::1 producer_port = 8001
1292025-12-16T09:14:46.726ZDEBGoximeter (results-sink): inserting 130 samples into database collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d
1302025-12-16T09:14:46.726ZDEBGoximeter (results-sink-cluster): inserting 130 samples into database collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d
1312025-12-16T09:14:46.726ZDEBGoximeter (clickhouse-client): unrolling 130 total samples collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d id = 2e611522-2449-41fe-882b-03e765f06c58
1322025-12-16T09:14:46.733ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d id = 2e611522-2449-41fe-882b-03e765f06c58 n_rows = 26 table_name = fields_string
1332025-12-16T09:14:46.735ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d id = 2e611522-2449-41fe-882b-03e765f06c58 n_rows = 28 table_name = fields_uuid
1342025-12-16T09:14:46.737ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d id = 2e611522-2449-41fe-882b-03e765f06c58 n_rows = 100 table_name = measurements_cumulativeu64
1352025-12-16T09:14:46.738ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d id = 2e611522-2449-41fe-882b-03e765f06c58 n_rows = 30 table_name = measurements_u64
1362025-12-16T09:14:46.826ZWARNoximeter (results-sink-cluster): failed to insert some results into metric DB collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
1372025-12-16T09:14:47.288ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d producer_id = 94dac492-2a79-4d68-89a2-f48fe510a3eb producer_ip = fd00:1122:3344:101::1 producer_port = 8001
1382025-12-16T09:14:47.289ZDEBGoximeter (collection-task): collecting from producer collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d producer_id = 94dac492-2a79-4d68-89a2-f48fe510a3eb producer_ip = fd00:1122:3344:101::1 producer_port = 8001
1392025-12-16T09:14:47.291ZDEBGoximeter (collection-task): collected results from producer collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 94dac492-2a79-4d68-89a2-f48fe510a3eb producer_ip = fd00:1122:3344:101::1 producer_port = 8001
1402025-12-16T09:14:48.288ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d producer_id = 94dac492-2a79-4d68-89a2-f48fe510a3eb producer_ip = fd00:1122:3344:101::1 producer_port = 8001
1412025-12-16T09:14:48.289ZDEBGoximeter (collection-task): collecting from producer collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d producer_id = 94dac492-2a79-4d68-89a2-f48fe510a3eb producer_ip = fd00:1122:3344:101::1 producer_port = 8001
1422025-12-16T09:14:48.291ZDEBGoximeter (collection-task): collected results from producer collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 94dac492-2a79-4d68-89a2-f48fe510a3eb producer_ip = fd00:1122:3344:101::1 producer_port = 8001
1432025-12-16T09:14:49.289ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d producer_id = 10373832-fc98-4a20-8740-604ad8bcd1a7 producer_ip = fd00:1122:3344:101::2 producer_port = 44059
1442025-12-16T09:14:49.289ZDEBGoximeter (collection-task): collecting from producer collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d producer_id = 10373832-fc98-4a20-8740-604ad8bcd1a7 producer_ip = fd00:1122:3344:101::2 producer_port = 44059
1452025-12-16T09:14:49.289ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d producer_id = 94dac492-2a79-4d68-89a2-f48fe510a3eb producer_ip = fd00:1122:3344:101::1 producer_port = 8001
1462025-12-16T09:14:49.289ZDEBGoximeter (collection-task): collecting from producer collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d producer_id = 94dac492-2a79-4d68-89a2-f48fe510a3eb producer_ip = fd00:1122:3344:101::1 producer_port = 8001
1472025-12-16T09:14:49.291ZDEBGoximeter (collection-task): collected results from producer collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d n_results = 2 producer_id = 10373832-fc98-4a20-8740-604ad8bcd1a7 producer_ip = fd00:1122:3344:101::2 producer_port = 44059
1482025-12-16T09:14:49.292ZDEBGoximeter (collection-task): collected results from producer collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 94dac492-2a79-4d68-89a2-f48fe510a3eb producer_ip = fd00:1122:3344:101::1 producer_port = 8001
1492025-12-16T09:14:50.289ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d producer_id = 94dac492-2a79-4d68-89a2-f48fe510a3eb producer_ip = fd00:1122:3344:101::1 producer_port = 8001
1502025-12-16T09:14:50.289ZDEBGoximeter (collection-task): collecting from producer collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d producer_id = 94dac492-2a79-4d68-89a2-f48fe510a3eb producer_ip = fd00:1122:3344:101::1 producer_port = 8001
1512025-12-16T09:14:50.292ZDEBGoximeter (collection-task): collected results from producer collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 94dac492-2a79-4d68-89a2-f48fe510a3eb producer_ip = fd00:1122:3344:101::1 producer_port = 8001
1522025-12-16T09:14:51.289ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d producer_id = 94dac492-2a79-4d68-89a2-f48fe510a3eb producer_ip = fd00:1122:3344:101::1 producer_port = 8001
1532025-12-16T09:14:51.289ZDEBGoximeter (collection-task): collecting from producer collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d producer_id = 94dac492-2a79-4d68-89a2-f48fe510a3eb producer_ip = fd00:1122:3344:101::1 producer_port = 8001
1542025-12-16T09:14:51.292ZDEBGoximeter (collection-task): collected results from producer collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 94dac492-2a79-4d68-89a2-f48fe510a3eb producer_ip = fd00:1122:3344:101::1 producer_port = 8001
1552025-12-16T09:14:51.725ZDEBGoximeter (results-sink): inserting 140 samples into database collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d
1562025-12-16T09:14:51.726ZDEBGoximeter (results-sink-cluster): inserting 140 samples into database collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d
1572025-12-16T09:14:51.726ZDEBGoximeter (clickhouse-client): unrolling 140 total samples collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d id = 2e611522-2449-41fe-882b-03e765f06c58
1582025-12-16T09:14:51.733ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d id = 2e611522-2449-41fe-882b-03e765f06c58 n_rows = 46 table_name = fields_string
1592025-12-16T09:14:51.735ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d id = 2e611522-2449-41fe-882b-03e765f06c58 n_rows = 10 table_name = fields_u16
1602025-12-16T09:14:51.737ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d id = 2e611522-2449-41fe-882b-03e765f06c58 n_rows = 38 table_name = fields_uuid
1612025-12-16T09:14:51.739ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d id = 2e611522-2449-41fe-882b-03e765f06c58 n_rows = 100 table_name = measurements_cumulativeu64
1622025-12-16T09:14:51.741ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d id = 2e611522-2449-41fe-882b-03e765f06c58 n_rows = 10 table_name = measurements_histogramu64
1632025-12-16T09:14:51.742ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d id = 2e611522-2449-41fe-882b-03e765f06c58 n_rows = 30 table_name = measurements_u64
1642025-12-16T09:14:51.827ZWARNoximeter (results-sink-cluster): failed to insert some results into metric DB collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
1652025-12-16T09:14:52.289ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d producer_id = 94dac492-2a79-4d68-89a2-f48fe510a3eb producer_ip = fd00:1122:3344:101::1 producer_port = 8001
1662025-12-16T09:14:52.289ZDEBGoximeter (collection-task): collecting from producer collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d producer_id = 94dac492-2a79-4d68-89a2-f48fe510a3eb producer_ip = fd00:1122:3344:101::1 producer_port = 8001
1672025-12-16T09:14:52.291ZDEBGoximeter (collection-task): collected results from producer collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 94dac492-2a79-4d68-89a2-f48fe510a3eb producer_ip = fd00:1122:3344:101::1 producer_port = 8001
1682025-12-16T09:14:53.289ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d producer_id = 94dac492-2a79-4d68-89a2-f48fe510a3eb producer_ip = fd00:1122:3344:101::1 producer_port = 8001
1692025-12-16T09:14:53.289ZDEBGoximeter (collection-task): collecting from producer collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d producer_id = 94dac492-2a79-4d68-89a2-f48fe510a3eb producer_ip = fd00:1122:3344:101::1 producer_port = 8001
1702025-12-16T09:14:53.291ZDEBGoximeter (collection-task): collected results from producer collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 94dac492-2a79-4d68-89a2-f48fe510a3eb producer_ip = fd00:1122:3344:101::1 producer_port = 8001
1712025-12-16T09:14:54.279ZINFOoximeter (oximeter-agent): refreshing list of producers from Nexus collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d file = oximeter/collector/src/agent.rs:557
1722025-12-16T09:14:54.280ZDEBGoximeter: client request body = None method = GET uri = http://[fd00:1122:3344:101::b]:12221/metrics/collectors/e0dc1a92-16e0-44d0-bf37-d9bb30e63782/producers?sort_by=id_ascending
1732025-12-16T09:14:54.289ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d producer_id = 94dac492-2a79-4d68-89a2-f48fe510a3eb producer_ip = fd00:1122:3344:101::1 producer_port = 8001
1742025-12-16T09:14:54.289ZDEBGoximeter (collection-task): collecting from producer collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d producer_id = 94dac492-2a79-4d68-89a2-f48fe510a3eb producer_ip = fd00:1122:3344:101::1 producer_port = 8001
1752025-12-16T09:14:54.291ZDEBGoximeter (collection-task): collected results from producer collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 94dac492-2a79-4d68-89a2-f48fe510a3eb producer_ip = fd00:1122:3344:101::1 producer_port = 8001
1762025-12-16T09:14:54.318ZDEBGoximeter: client response result = Ok(Response { url: "http://[fd00:1122:3344:101::b]:12221/metrics/collectors/e0dc1a92-16e0-44d0-bf37-d9bb30e63782/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "28efe757-fc6f-4825-b8dd-f8e6366cc390", "content-length": "721", "date": "Tue, 16 Dec 2025 09:14:54 GMT"} })
1772025-12-16T09:14:54.318ZDEBGoximeter: client request body = None method = GET uri = http://[fd00:1122:3344:101::b]:12221/metrics/collectors/e0dc1a92-16e0-44d0-bf37-d9bb30e63782/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiI5NGRhYzQ5Mi0yYTc5LTRkNjgtODlhMi1mNDhmZTUxMGEzZWIifX0%3D
1782025-12-16T09:14:54.333ZDEBGoximeter: client response result = Ok(Response { url: "http://[fd00:1122:3344:101::b]:12221/metrics/collectors/e0dc1a92-16e0-44d0-bf37-d9bb30e63782/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiI5NGRhYzQ5Mi0yYTc5LTRkNjgtODlhMi1mNDhmZTUxMGEzZWIifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "e64ecb6d-702d-4502-8f9d-f2b541a7e6b5", "content-length": "29", "date": "Tue, 16 Dec 2025 09:14:54 GMT"} })
1792025-12-16T09:14:54.333ZDEBGoximeter (oximeter-agent): registered new metric producer address = [fd00:1122:3344:101::a]:35873 collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d producer_id = 3d8bfae9-c3c6-45e2-84f6-2622d0446577
1802025-12-16T09:14:54.333ZDEBGoximeter (oximeter-agent): registered new metric producer address = [fd00:1122:3344:101::2]:4677 collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d producer_id = 4913ebcb-614b-4662-b025-544257b8523d
1812025-12-16T09:14:54.333ZINFOoximeter (oximeter-agent): refreshed list of producers from Nexus collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d file = oximeter/collector/src/agent.rs:632 n_current_tasks = 4 n_pruned_tasks = 0
1822025-12-16T09:14:54.334ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d producer_id = 4913ebcb-614b-4662-b025-544257b8523d producer_ip = fd00:1122:3344:101::2 producer_port = 4677
1832025-12-16T09:14:54.334ZDEBGoximeter (collection-task): reporting oximeter self-collection statistics collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d producer_id = 4913ebcb-614b-4662-b025-544257b8523d producer_ip = fd00:1122:3344:101::2 producer_port = 4677
1842025-12-16T09:14:54.334ZDEBGoximeter (collection-task): reporting oximeter self-collection statistics collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d producer_id = 3d8bfae9-c3c6-45e2-84f6-2622d0446577 producer_ip = fd00:1122:3344:101::a producer_port = 35873
1852025-12-16T09:14:54.334ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d producer_id = 3d8bfae9-c3c6-45e2-84f6-2622d0446577 producer_ip = fd00:1122:3344:101::a producer_port = 35873
1862025-12-16T09:14:54.394ZDEBGoximeter (collection-task): collecting from producer collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d producer_id = 4913ebcb-614b-4662-b025-544257b8523d producer_ip = fd00:1122:3344:101::2 producer_port = 4677
1872025-12-16T09:14:54.394ZDEBGoximeter (collection-task): collecting from producer collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d producer_id = 3d8bfae9-c3c6-45e2-84f6-2622d0446577 producer_ip = fd00:1122:3344:101::a producer_port = 35873
1882025-12-16T09:14:54.407ZDEBGoximeter (collection-task): collected results from producer collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d n_results = 5 producer_id = 3d8bfae9-c3c6-45e2-84f6-2622d0446577 producer_ip = fd00:1122:3344:101::a producer_port = 35873
1892025-12-16T09:14:54.409ZDEBGoximeter (collection-task): collected results from producer collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 4913ebcb-614b-4662-b025-544257b8523d producer_ip = fd00:1122:3344:101::2 producer_port = 4677
1902025-12-16T09:14:55.289ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d producer_id = 94dac492-2a79-4d68-89a2-f48fe510a3eb producer_ip = fd00:1122:3344:101::1 producer_port = 8001
1912025-12-16T09:14:55.289ZDEBGoximeter (collection-task): collecting from producer collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d producer_id = 94dac492-2a79-4d68-89a2-f48fe510a3eb producer_ip = fd00:1122:3344:101::1 producer_port = 8001
1922025-12-16T09:14:55.291ZDEBGoximeter (collection-task): collected results from producer collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 94dac492-2a79-4d68-89a2-f48fe510a3eb producer_ip = fd00:1122:3344:101::1 producer_port = 8001
1932025-12-16T09:14:55.334ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d producer_id = 4913ebcb-614b-4662-b025-544257b8523d producer_ip = fd00:1122:3344:101::2 producer_port = 4677
1942025-12-16T09:14:55.334ZDEBGoximeter (collection-task): collecting from producer collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d producer_id = 4913ebcb-614b-4662-b025-544257b8523d producer_ip = fd00:1122:3344:101::2 producer_port = 4677
1952025-12-16T09:14:55.334ZDEBGoximeter (collection-task): collected results from producer collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 4913ebcb-614b-4662-b025-544257b8523d producer_ip = fd00:1122:3344:101::2 producer_port = 4677
1962025-12-16T09:14:56.289ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d producer_id = 94dac492-2a79-4d68-89a2-f48fe510a3eb producer_ip = fd00:1122:3344:101::1 producer_port = 8001
1972025-12-16T09:14:56.289ZDEBGoximeter (collection-task): collecting from producer collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d producer_id = 94dac492-2a79-4d68-89a2-f48fe510a3eb producer_ip = fd00:1122:3344:101::1 producer_port = 8001
1982025-12-16T09:14:56.291ZDEBGoximeter (collection-task): collected results from producer collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 94dac492-2a79-4d68-89a2-f48fe510a3eb producer_ip = fd00:1122:3344:101::1 producer_port = 8001
1992025-12-16T09:14:56.334ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d producer_id = 4913ebcb-614b-4662-b025-544257b8523d producer_ip = fd00:1122:3344:101::2 producer_port = 4677
2002025-12-16T09:14:56.334ZDEBGoximeter (collection-task): collecting from producer collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d producer_id = 4913ebcb-614b-4662-b025-544257b8523d producer_ip = fd00:1122:3344:101::2 producer_port = 4677
2012025-12-16T09:14:56.335ZDEBGoximeter (collection-task): collected results from producer collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 4913ebcb-614b-4662-b025-544257b8523d producer_ip = fd00:1122:3344:101::2 producer_port = 4677
2022025-12-16T09:14:56.726ZDEBGoximeter (results-sink): inserting 178 samples into database collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d
2032025-12-16T09:14:56.726ZDEBGoximeter (results-sink-cluster): inserting 178 samples into database collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d
2042025-12-16T09:14:56.726ZDEBGoximeter (clickhouse-client): unrolling 178 total samples collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d id = 2e611522-2449-41fe-882b-03e765f06c58
2052025-12-16T09:14:56.731ZDEBGoximeter (clickhouse-client): retrieving timeseries schema from database collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d id = 2e611522-2449-41fe-882b-03e765f06c58
2062025-12-16T09:14:56.772ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d id = 2e611522-2449-41fe-882b-03e765f06c58 schema = TimeseriesSchema { timeseries_name: TimeseriesName("collection_target:cpus_provisioned"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "id", field_type: Uuid, source: Target, description: "" }}, datum_type: I64, version: 1, authz_scope: Fleet, units: Count, created: 2025-12-16T09:14:56.731150201Z } timeseries_name = collection_target:cpus_provisioned
2072025-12-16T09:14:56.772ZDEBGoximeter (clickhouse-client): retrieving timeseries schema from database collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d id = 2e611522-2449-41fe-882b-03e765f06c58
2082025-12-16T09:14:56.774ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d id = 2e611522-2449-41fe-882b-03e765f06c58 schema = TimeseriesSchema { timeseries_name: TimeseriesName("collection_target:ram_provisioned"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "id", field_type: Uuid, source: Target, description: "" }}, datum_type: I64, version: 1, authz_scope: Fleet, units: Count, created: 2025-12-16T09:14:56.772683819Z } timeseries_name = collection_target:ram_provisioned
2092025-12-16T09:14:56.774ZDEBGoximeter (clickhouse-client): retrieving timeseries schema from database collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d id = 2e611522-2449-41fe-882b-03e765f06c58
2102025-12-16T09:14:56.775ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d id = 2e611522-2449-41fe-882b-03e765f06c58 schema = TimeseriesSchema { timeseries_name: TimeseriesName("collection_target:virtual_disk_space_provisioned"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "id", field_type: Uuid, source: Target, description: "" }}, datum_type: I64, version: 1, authz_scope: Fleet, units: Count, created: 2025-12-16T09:14:56.774418784Z } timeseries_name = collection_target:virtual_disk_space_provisioned
2112025-12-16T09:14:56.776ZDEBGoximeter (clickhouse-client): retrieving timeseries schema from database collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d id = 2e611522-2449-41fe-882b-03e765f06c58
2122025-12-16T09:14:56.777ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d id = 2e611522-2449-41fe-882b-03e765f06c58 schema = TimeseriesSchema { timeseries_name: TimeseriesName("static_routing_config:static_routes"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "hostname", field_type: String, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }}, datum_type: CumulativeU64, version: 1, authz_scope: Fleet, units: Count, created: 2025-12-16T09:14:56.776462733Z } timeseries_name = static_routing_config:static_routes
2132025-12-16T09:14:56.777ZDEBGoximeter (clickhouse-client): retrieving timeseries schema from database collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d id = 2e611522-2449-41fe-882b-03e765f06c58
2142025-12-16T09:14:56.779ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d id = 2e611522-2449-41fe-882b-03e765f06c58 schema = TimeseriesSchema { timeseries_name: TimeseriesName("static_routing_config:static_nexthops"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "hostname", field_type: String, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }}, datum_type: CumulativeU64, version: 1, authz_scope: Fleet, units: Count, created: 2025-12-16T09:14:56.777981215Z } timeseries_name = static_routing_config:static_nexthops
2152025-12-16T09:14:56.779ZDEBGoximeter (clickhouse-client): retrieving timeseries schema from database collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d id = 2e611522-2449-41fe-882b-03e765f06c58
2162025-12-16T09:14:56.780ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d id = 2e611522-2449-41fe-882b-03e765f06c58 schema = TimeseriesSchema { timeseries_name: TimeseriesName("switch_rib:active_routes"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "hostname", field_type: String, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }}, datum_type: U64, version: 1, authz_scope: Fleet, units: Count, created: 2025-12-16T09:14:56.779548058Z } timeseries_name = switch_rib:active_routes
2172025-12-16T09:14:56.780ZDEBGoximeter (clickhouse-client): retrieving timeseries schema from database collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d id = 2e611522-2449-41fe-882b-03e765f06c58
2182025-12-16T09:14:56.782ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d id = 2e611522-2449-41fe-882b-03e765f06c58 schema = TimeseriesSchema { timeseries_name: TimeseriesName("mg_lower:routes_blocked_by_link_state"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "hostname", field_type: String, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }}, datum_type: U64, version: 1, authz_scope: Fleet, units: Count, created: 2025-12-16T09:14:56.780949558Z } timeseries_name = mg_lower:routes_blocked_by_link_state
2192025-12-16T09:14:56.784ZDEBGoximeter (clickhouse-client): inserting 7 new timeseries schema collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d id = 2e611522-2449-41fe-882b-03e765f06c58
2202025-12-16T09:14:56.788ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d id = 2e611522-2449-41fe-882b-03e765f06c58 n_rows = 4 table_name = fields_ipaddr
2212025-12-16T09:14:56.790ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d id = 2e611522-2449-41fe-882b-03e765f06c58 n_rows = 70 table_name = fields_string
2222025-12-16T09:14:56.792ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d id = 2e611522-2449-41fe-882b-03e765f06c58 n_rows = 23 table_name = fields_u16
2232025-12-16T09:14:56.794ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d id = 2e611522-2449-41fe-882b-03e765f06c58 n_rows = 69 table_name = fields_uuid
2242025-12-16T09:14:56.795ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d id = 2e611522-2449-41fe-882b-03e765f06c58 n_rows = 108 table_name = measurements_cumulativeu64
2252025-12-16T09:14:56.798ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d id = 2e611522-2449-41fe-882b-03e765f06c58 n_rows = 19 table_name = measurements_histogramu64
2262025-12-16T09:14:56.800ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d id = 2e611522-2449-41fe-882b-03e765f06c58 n_rows = 15 table_name = measurements_i64
2272025-12-16T09:14:56.801ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d id = 2e611522-2449-41fe-882b-03e765f06c58 n_rows = 36 table_name = measurements_u64
2282025-12-16T09:14:56.827ZWARNoximeter (results-sink-cluster): failed to insert some results into metric DB collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
2292025-12-16T09:14:57.289ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d producer_id = 94dac492-2a79-4d68-89a2-f48fe510a3eb producer_ip = fd00:1122:3344:101::1 producer_port = 8001
2302025-12-16T09:14:57.289ZDEBGoximeter (collection-task): collecting from producer collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d producer_id = 94dac492-2a79-4d68-89a2-f48fe510a3eb producer_ip = fd00:1122:3344:101::1 producer_port = 8001
2312025-12-16T09:14:57.292ZDEBGoximeter (collection-task): collected results from producer collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 94dac492-2a79-4d68-89a2-f48fe510a3eb producer_ip = fd00:1122:3344:101::1 producer_port = 8001
2322025-12-16T09:14:57.334ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d producer_id = 4913ebcb-614b-4662-b025-544257b8523d producer_ip = fd00:1122:3344:101::2 producer_port = 4677
2332025-12-16T09:14:57.334ZDEBGoximeter (collection-task): collecting from producer collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d producer_id = 4913ebcb-614b-4662-b025-544257b8523d producer_ip = fd00:1122:3344:101::2 producer_port = 4677
2342025-12-16T09:14:57.335ZDEBGoximeter (collection-task): collected results from producer collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 4913ebcb-614b-4662-b025-544257b8523d producer_ip = fd00:1122:3344:101::2 producer_port = 4677
2352025-12-16T09:14:58.289ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d producer_id = 94dac492-2a79-4d68-89a2-f48fe510a3eb producer_ip = fd00:1122:3344:101::1 producer_port = 8001
2362025-12-16T09:14:58.289ZDEBGoximeter (collection-task): collecting from producer collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d producer_id = 94dac492-2a79-4d68-89a2-f48fe510a3eb producer_ip = fd00:1122:3344:101::1 producer_port = 8001
2372025-12-16T09:14:58.292ZDEBGoximeter (collection-task): collected results from producer collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 94dac492-2a79-4d68-89a2-f48fe510a3eb producer_ip = fd00:1122:3344:101::1 producer_port = 8001
2382025-12-16T09:14:58.334ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d producer_id = 4913ebcb-614b-4662-b025-544257b8523d producer_ip = fd00:1122:3344:101::2 producer_port = 4677
2392025-12-16T09:14:58.334ZDEBGoximeter (collection-task): collecting from producer collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d producer_id = 4913ebcb-614b-4662-b025-544257b8523d producer_ip = fd00:1122:3344:101::2 producer_port = 4677
2402025-12-16T09:14:58.335ZDEBGoximeter (collection-task): collected results from producer collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 4913ebcb-614b-4662-b025-544257b8523d producer_ip = fd00:1122:3344:101::2 producer_port = 4677
2412025-12-16T09:14:59.289ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d producer_id = 10373832-fc98-4a20-8740-604ad8bcd1a7 producer_ip = fd00:1122:3344:101::2 producer_port = 44059
2422025-12-16T09:14:59.289ZDEBGoximeter (collection-task): collecting from producer collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d producer_id = 10373832-fc98-4a20-8740-604ad8bcd1a7 producer_ip = fd00:1122:3344:101::2 producer_port = 44059
2432025-12-16T09:14:59.289ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d producer_id = 94dac492-2a79-4d68-89a2-f48fe510a3eb producer_ip = fd00:1122:3344:101::1 producer_port = 8001
2442025-12-16T09:14:59.289ZDEBGoximeter (collection-task): collecting from producer collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d producer_id = 94dac492-2a79-4d68-89a2-f48fe510a3eb producer_ip = fd00:1122:3344:101::1 producer_port = 8001
2452025-12-16T09:14:59.291ZDEBGoximeter (collection-task): collected results from producer collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 94dac492-2a79-4d68-89a2-f48fe510a3eb producer_ip = fd00:1122:3344:101::1 producer_port = 8001
2462025-12-16T09:14:59.292ZDEBGoximeter (collection-task): collected results from producer collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d n_results = 2 producer_id = 10373832-fc98-4a20-8740-604ad8bcd1a7 producer_ip = fd00:1122:3344:101::2 producer_port = 44059
2472025-12-16T09:14:59.333ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d producer_id = 4913ebcb-614b-4662-b025-544257b8523d producer_ip = fd00:1122:3344:101::2 producer_port = 4677
2482025-12-16T09:14:59.334ZDEBGoximeter (collection-task): collecting from producer collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d producer_id = 4913ebcb-614b-4662-b025-544257b8523d producer_ip = fd00:1122:3344:101::2 producer_port = 4677
2492025-12-16T09:14:59.335ZDEBGoximeter (collection-task): collected results from producer collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 4913ebcb-614b-4662-b025-544257b8523d producer_ip = fd00:1122:3344:101::2 producer_port = 4677
2502025-12-16T09:15:00.289ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d producer_id = 94dac492-2a79-4d68-89a2-f48fe510a3eb producer_ip = fd00:1122:3344:101::1 producer_port = 8001
2512025-12-16T09:15:00.289ZDEBGoximeter (collection-task): collecting from producer collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d producer_id = 94dac492-2a79-4d68-89a2-f48fe510a3eb producer_ip = fd00:1122:3344:101::1 producer_port = 8001
2522025-12-16T09:15:00.291ZDEBGoximeter (collection-task): collected results from producer collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 94dac492-2a79-4d68-89a2-f48fe510a3eb producer_ip = fd00:1122:3344:101::1 producer_port = 8001
2532025-12-16T09:15:00.333ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d producer_id = 4913ebcb-614b-4662-b025-544257b8523d producer_ip = fd00:1122:3344:101::2 producer_port = 4677
2542025-12-16T09:15:00.334ZDEBGoximeter (collection-task): collecting from producer collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d producer_id = 4913ebcb-614b-4662-b025-544257b8523d producer_ip = fd00:1122:3344:101::2 producer_port = 4677
2552025-12-16T09:15:00.334ZDEBGoximeter (collection-task): collected results from producer collector_id = e0dc1a92-16e0-44d0-bf37-d9bb30e63782 collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 4913ebcb-614b-4662-b025-544257b8523d producer_ip = fd00:1122:3344:101::2 producer_port = 4677