Max level shown:
1 [ Aug 26 21:28:49 Enabled. ]
2 [ Aug 26 21:28:49 Rereading configuration. ]
3 [ Aug 26 21:28:52 Rereading configuration. ]
4 [ Aug 26 21:28:58 Executing start method ("ctrun -l child -o noorphan,regent /opt/oxide/oximeter-collector/bin/oximeter run /var/svc/manifest/site/oximeter/config.toml --address [fd00:1122:3344:101::d]:12223 --id 8434ede6-327d-44e3-8722-6c9e65e5b476 &"). ]
5 [ Aug 26 21:28:58 Method "start" exited with status 0. ]
6 note: configured to log to "/dev/stdout"
72026-08-26T21:28:58.279ZDEBGoximeter: registered DTrace probes
82026-08-26T21:28:58.280ZINFOoximeter: starting oximeter server file = oximeter/collector/src/lib.rs:299
92026-08-26T21:28:58.299ZDEBGoximeter: creating ClickHouse client
102026-08-26T21:28:58.308ZDEBGoximeter (clickhouse-client): registered USDT probes collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d id = 686df668-58e8-4ea0-8f4e-b1d302e96d30
112026-08-26T21:29:09.150ZWARNoximeter (clickhouse-client): oximeter database does not exist, or is out-of-date collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d file = oximeter/db/src/client/mod.rs:888 id = 686df668-58e8-4ea0-8f4e-b1d302e96d30
122026-08-26T21:29:09.150ZDEBGoximeter (oximeter-agent): oximeter database does not exist, creating collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d
132026-08-26T21:29:09.150ZINFOoximeter (clickhouse-client): reading db version collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d file = oximeter/db/src/client/mod.rs:788 id = 686df668-58e8-4ea0-8f4e-b1d302e96d30
142026-08-26T21:29:09.152ZWARNoximeter (clickhouse-client): oximeter database does not exist, or is out-of-date collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d file = oximeter/db/src/client/mod.rs:888 id = 686df668-58e8-4ea0-8f4e-b1d302e96d30
152026-08-26T21:29:09.152ZINFOoximeter (clickhouse-client): read oximeter database version collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d file = oximeter/db/src/client/mod.rs:793 id = 686df668-58e8-4ea0-8f4e-b1d302e96d30 version = 0
162026-08-26T21:29:09.152ZINFOoximeter (clickhouse-client): initializing oximeter schema collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d file = oximeter/db/src/client/mod.rs:798 id = 686df668-58e8-4ea0-8f4e-b1d302e96d30
172026-08-26T21:29:09.152ZDEBGoximeter (clickhouse-client): initializing ClickHouse database collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d id = 686df668-58e8-4ea0-8f4e-b1d302e96d30
182026-08-26T21:29:09.779ZINFOoximeter (clickhouse-client): inserting current version collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d file = oximeter/db/src/client/mod.rs:819 id = 686df668-58e8-4ea0-8f4e-b1d302e96d30 version = 14
192026-08-26T21:29:09.810ZDEBGoximeter (clickhouse-client): registered USDT probes collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d id = d0e03fce-be30-4ca7-8f21-5ccbff02f5bb
202026-08-26T21:29:09.827ZDEBGoximeter (dropshot): registered endpoint local_addr = [fd00:1122:3344:101::d]:12223 method = GET path = /info versions = all
212026-08-26T21:29:09.827ZDEBGoximeter (dropshot): registered endpoint local_addr = [fd00:1122:3344:101::d]:12223 method = GET path = /producers versions = all
222026-08-26T21:29:09.827ZDEBGoximeter (dropshot): registered endpoint local_addr = [fd00:1122:3344:101::d]:12223 method = DELETE path = /producers/{producer_id} versions = all
232026-08-26T21:29:09.827ZDEBGoximeter (dropshot): registered endpoint local_addr = [fd00:1122:3344:101::d]:12223 method = GET path = /producers/{producer_id} versions = all
242026-08-26T21:29:09.828ZINFOoximeter (dropshot): listening file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:335 local_addr = [fd00:1122:3344:101::d]:12223
252026-08-26T21:29:09.834ZDEBGoximeter (dropshot): successfully registered DTrace USDT probes local_addr = [fd00:1122:3344:101::d]:12223
262026-08-26T21:29:09.834ZDEBGoximeter: registered USDT probes
272026-08-26T21:29:09.834ZDEBGoximeter: contacting nexus
282026-08-26T21:29:09.862ZDEBGoximeter: client request body = Some(Body) method = POST uri = http://[fd00:1122:3344:101::a]:12221/metrics/collectors
292026-08-26T21:29:09.906ZDEBGoximeter: client response result = Ok(Response { url: "http://[fd00:1122:3344:101::a]:12221/metrics/collectors", status: 204, headers: {"x-request-id": "c0199b8c-475f-4c92-ac34-0187c23f179c", "date": "Wed, 26 Aug 2026 21:29:09 GMT"} })
302026-08-26T21:29:09.906ZINFOoximeter: oximeter registered with nexus file = oximeter/collector/src/lib.rs:431 id = 8434ede6-327d-44e3-8722-6c9e65e5b476
312026-08-26T21:29:09.906ZINFOoximeter (oximeter-agent): starting refresh list task collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d file = oximeter/collector/src/agent.rs:626
322026-08-26T21:29:09.907ZINFOoximeter (oximeter-agent): refreshing list of producers from Nexus collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d file = oximeter/collector/src/agent.rs:629
332026-08-26T21:29:09.908ZDEBGoximeter: client request body = None method = GET uri = http://[fd00:1122:3344:101::a]:12221/metrics/collectors/8434ede6-327d-44e3-8722-6c9e65e5b476/producers?sort_by=id_ascending
342026-08-26T21:29:09.932ZDEBGoximeter: client response result = Ok(Response { url: "http://[fd00:1122:3344:101::a]:12221/metrics/collectors/8434ede6-327d-44e3-8722-6c9e65e5b476/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "fd2f457e-ae1d-4b79-9da1-baf36864aa1d", "content-length": "29", "date": "Wed, 26 Aug 2026 21:29:09 GMT"} })
352026-08-26T21:29:09.935ZINFOoximeter (oximeter-agent): refreshed list of producers from Nexus collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d file = oximeter/collector/src/agent.rs:704 n_current_tasks = 0 n_pruned_tasks = 0
362026-08-26T21:29:14.811ZDEBGoximeter (database-inserter): inserting samples into database collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d n_samples = 4
372026-08-26T21:29:14.811ZDEBGoximeter (database-inserter): inserting samples into database collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d n_samples = 4
382026-08-26T21:29:14.811ZDEBGoximeter (clickhouse-client): unrolling 4 total samples collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d id = 686df668-58e8-4ea0-8f4e-b1d302e96d30
392026-08-26T21:29:14.812ZDEBGoximeter (clickhouse-client): fetching timeseries schema from database collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d id = 686df668-58e8-4ea0-8f4e-b1d302e96d30
402026-08-26T21:29:14.813ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d id = 686df668-58e8-4ea0-8f4e-b1d302e96d30 schema = TimeseriesSchema { timeseries_name: TimeseriesName("oximeter_collector:database_queue_depth"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "collector_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "collector_ip", field_type: IpAddr, source: Target, description: "" }, FieldSchema { name: "collector_port", field_type: U16, source: Target, description: "" }, FieldSchema { name: "collector_sink", field_type: String, source: Metric, description: "" }}, datum_type: HistogramU64, version: 1, authz_scope: Fleet, units: Count, created: 2026-08-26T21:29:14.812318158Z } timeseries_name = oximeter_collector:database_queue_depth
412026-08-26T21:29:14.814ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d id = 686df668-58e8-4ea0-8f4e-b1d302e96d30 schema = TimeseriesSchema { timeseries_name: TimeseriesName("oximeter_collector:database_samples_dropped"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "collector_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "collector_ip", field_type: IpAddr, source: Target, description: "" }, FieldSchema { name: "collector_port", field_type: U16, source: Target, description: "" }, FieldSchema { name: "collector_sink", field_type: String, source: Metric, description: "" }}, datum_type: CumulativeU64, version: 1, authz_scope: Fleet, units: Count, created: 2026-08-26T21:29:14.812313088Z } timeseries_name = oximeter_collector:database_samples_dropped
422026-08-26T21:29:14.814ZDEBGoximeter (clickhouse-client): inserting 2 new timeseries schema collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d id = 686df668-58e8-4ea0-8f4e-b1d302e96d30
432026-08-26T21:29:14.818ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d id = 686df668-58e8-4ea0-8f4e-b1d302e96d30 n_rows = 4 table_name = fields_ipaddr
442026-08-26T21:29:14.820ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d id = 686df668-58e8-4ea0-8f4e-b1d302e96d30 n_rows = 4 table_name = fields_string
452026-08-26T21:29:14.822ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d id = 686df668-58e8-4ea0-8f4e-b1d302e96d30 n_rows = 4 table_name = fields_u16
462026-08-26T21:29:14.825ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d id = 686df668-58e8-4ea0-8f4e-b1d302e96d30 n_rows = 4 table_name = fields_uuid
472026-08-26T21:29:14.826ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d id = 686df668-58e8-4ea0-8f4e-b1d302e96d30 n_rows = 2 table_name = measurements_cumulativeu64
482026-08-26T21:29:14.829ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d id = 686df668-58e8-4ea0-8f4e-b1d302e96d30 n_rows = 2 table_name = measurements_histogramu64
492026-08-26T21:29:14.913ZWARNoximeter (database-inserter): failed to insert some results into metric DB collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:276
502026-08-26T21:29:19.811ZDEBGoximeter (database-inserter): batch interval expired, but no samples to insert collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d
512026-08-26T21:29:19.811ZDEBGoximeter (database-inserter): batch interval expired, but no samples to insert collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d
522026-08-26T21:29:24.811ZDEBGoximeter (database-inserter): batch interval expired, but no samples to insert collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d
532026-08-26T21:29:24.811ZDEBGoximeter (database-inserter): batch interval expired, but no samples to insert collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d
542026-08-26T21:29:24.907ZINFOoximeter (oximeter-agent): refreshing list of producers from Nexus collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d file = oximeter/collector/src/agent.rs:629
552026-08-26T21:29:24.908ZDEBGoximeter: client request body = None method = GET uri = http://[fd00:1122:3344:101::c]:12221/metrics/collectors/8434ede6-327d-44e3-8722-6c9e65e5b476/producers?sort_by=id_ascending
562026-08-26T21:29:24.918ZDEBGoximeter: client response result = Ok(Response { url: "http://[fd00:1122:3344:101::c]:12221/metrics/collectors/8434ede6-327d-44e3-8722-6c9e65e5b476/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "cbb51233-77c3-4141-8dc1-c9efa688230e", "content-length": "436", "date": "Wed, 26 Aug 2026 21:29:24 GMT"} })
572026-08-26T21:29:24.918ZDEBGoximeter: client request body = None method = GET uri = http://[fd00:1122:3344:101::c]:12221/metrics/collectors/8434ede6-327d-44e3-8722-6c9e65e5b476/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiI0MmUwMTI1Mi1iMWU2LTQ0YjUtOGU4MC1mZGEzODE3Nzg2MGIifX0%3D
582026-08-26T21:29:24.923ZDEBGoximeter: client response result = Ok(Response { url: "http://[fd00:1122:3344:101::c]:12221/metrics/collectors/8434ede6-327d-44e3-8722-6c9e65e5b476/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiI0MmUwMTI1Mi1iMWU2LTQ0YjUtOGU4MC1mZGEzODE3Nzg2MGIifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "83c26d60-0a4d-4fc8-bdca-aac7a6b2e330", "content-length": "29", "date": "Wed, 26 Aug 2026 21:29:24 GMT"} })
592026-08-26T21:29:24.923ZDEBGoximeter (oximeter-agent): registered new metric producer address = [fd00:1122:3344:101::2]:4677 collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 322a7ed9-abbd-411d-8ee5-1a1fdff896f1
602026-08-26T21:29:24.923ZDEBGoximeter (oximeter-agent): registered new metric producer address = [fd00:1122:3344:101::2]:38589 collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 42e01252-b1e6-44b5-8e80-fda38177860b
612026-08-26T21:29:24.923ZINFOoximeter (oximeter-agent): refreshed list of producers from Nexus collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d file = oximeter/collector/src/agent.rs:704 n_current_tasks = 2 n_pruned_tasks = 0
622026-08-26T21:29:25.411ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 322a7ed9-abbd-411d-8ee5-1a1fdff896f1 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
632026-08-26T21:29:25.411ZDEBGoximeter (collection-task): collecting from producer collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 322a7ed9-abbd-411d-8ee5-1a1fdff896f1 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
642026-08-26T21:29:25.428ZDEBGoximeter (collection-task): collected results from producer collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 4 producer_id = 322a7ed9-abbd-411d-8ee5-1a1fdff896f1 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
652026-08-26T21:29:26.411ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 322a7ed9-abbd-411d-8ee5-1a1fdff896f1 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
662026-08-26T21:29:26.411ZDEBGoximeter (collection-task): collecting from producer collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 322a7ed9-abbd-411d-8ee5-1a1fdff896f1 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
672026-08-26T21:29:26.412ZDEBGoximeter (collection-task): collected results from producer collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 4 producer_id = 322a7ed9-abbd-411d-8ee5-1a1fdff896f1 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
682026-08-26T21:29:27.411ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 322a7ed9-abbd-411d-8ee5-1a1fdff896f1 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
692026-08-26T21:29:27.412ZDEBGoximeter (collection-task): collecting from producer collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 322a7ed9-abbd-411d-8ee5-1a1fdff896f1 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
702026-08-26T21:29:27.412ZDEBGoximeter (collection-task): collected results from producer collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 4 producer_id = 322a7ed9-abbd-411d-8ee5-1a1fdff896f1 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
712026-08-26T21:29:28.411ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 322a7ed9-abbd-411d-8ee5-1a1fdff896f1 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
722026-08-26T21:29:28.411ZDEBGoximeter (collection-task): collecting from producer collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 322a7ed9-abbd-411d-8ee5-1a1fdff896f1 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
732026-08-26T21:29:28.412ZDEBGoximeter (collection-task): collected results from producer collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 4 producer_id = 322a7ed9-abbd-411d-8ee5-1a1fdff896f1 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
742026-08-26T21:29:29.411ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 322a7ed9-abbd-411d-8ee5-1a1fdff896f1 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
752026-08-26T21:29:29.411ZDEBGoximeter (collection-task): collecting from producer collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 322a7ed9-abbd-411d-8ee5-1a1fdff896f1 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
762026-08-26T21:29:29.412ZDEBGoximeter (collection-task): collected results from producer collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 4 producer_id = 322a7ed9-abbd-411d-8ee5-1a1fdff896f1 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
772026-08-26T21:29:29.811ZDEBGoximeter (database-inserter): inserting samples into database collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d n_samples = 20
782026-08-26T21:29:29.811ZDEBGoximeter (database-inserter): inserting samples into database collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d n_samples = 20
792026-08-26T21:29:29.811ZDEBGoximeter (clickhouse-client): unrolling 20 total samples collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d id = 686df668-58e8-4ea0-8f4e-b1d302e96d30
802026-08-26T21:29:29.811ZDEBGoximeter (clickhouse-client): fetching timeseries schema from database collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d id = 686df668-58e8-4ea0-8f4e-b1d302e96d30
812026-08-26T21:29:29.813ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d id = 686df668-58e8-4ea0-8f4e-b1d302e96d30 schema = TimeseriesSchema { timeseries_name: TimeseriesName("mg_lower:routes_blocked_by_link_state"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "hostname", field_type: String, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }}, datum_type: U64, version: 1, authz_scope: Fleet, units: Count, created: 2026-08-26T21:29:29.811356555Z } timeseries_name = mg_lower:routes_blocked_by_link_state
822026-08-26T21:29:29.813ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d id = 686df668-58e8-4ea0-8f4e-b1d302e96d30 schema = TimeseriesSchema { timeseries_name: TimeseriesName("static_routing_config:static_nexthops"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "hostname", field_type: String, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }}, datum_type: CumulativeU64, version: 1, authz_scope: Fleet, units: Count, created: 2026-08-26T21:29:29.811351615Z } timeseries_name = static_routing_config:static_nexthops
832026-08-26T21:29:29.813ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d id = 686df668-58e8-4ea0-8f4e-b1d302e96d30 schema = TimeseriesSchema { timeseries_name: TimeseriesName("static_routing_config:static_routes"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "hostname", field_type: String, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }}, datum_type: CumulativeU64, version: 1, authz_scope: Fleet, units: Count, created: 2026-08-26T21:29:29.811347534Z } timeseries_name = static_routing_config:static_routes
842026-08-26T21:29:29.813ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d id = 686df668-58e8-4ea0-8f4e-b1d302e96d30 schema = TimeseriesSchema { timeseries_name: TimeseriesName("switch_rib:active_routes"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "hostname", field_type: String, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }}, datum_type: U64, version: 1, authz_scope: Fleet, units: Count, created: 2026-08-26T21:29:29.811354544Z } timeseries_name = switch_rib:active_routes
852026-08-26T21:29:29.813ZDEBGoximeter (clickhouse-client): inserting 4 new timeseries schema collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d id = 686df668-58e8-4ea0-8f4e-b1d302e96d30
862026-08-26T21:29:29.817ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d id = 686df668-58e8-4ea0-8f4e-b1d302e96d30 n_rows = 4 table_name = fields_string
872026-08-26T21:29:29.819ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d id = 686df668-58e8-4ea0-8f4e-b1d302e96d30 n_rows = 8 table_name = fields_uuid
882026-08-26T21:29:29.821ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d id = 686df668-58e8-4ea0-8f4e-b1d302e96d30 n_rows = 10 table_name = measurements_cumulativeu64
892026-08-26T21:29:29.822ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d id = 686df668-58e8-4ea0-8f4e-b1d302e96d30 n_rows = 10 table_name = measurements_u64
902026-08-26T21:29:29.912ZWARNoximeter (database-inserter): failed to insert some results into metric DB collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:276
912026-08-26T21:29:30.411ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 322a7ed9-abbd-411d-8ee5-1a1fdff896f1 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
922026-08-26T21:29:30.411ZDEBGoximeter (collection-task): collecting from producer collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 322a7ed9-abbd-411d-8ee5-1a1fdff896f1 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
932026-08-26T21:29:30.412ZDEBGoximeter (collection-task): collected results from producer collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 4 producer_id = 322a7ed9-abbd-411d-8ee5-1a1fdff896f1 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
942026-08-26T21:29:31.411ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 322a7ed9-abbd-411d-8ee5-1a1fdff896f1 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
952026-08-26T21:29:31.411ZDEBGoximeter (collection-task): collecting from producer collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 322a7ed9-abbd-411d-8ee5-1a1fdff896f1 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
962026-08-26T21:29:31.412ZDEBGoximeter (collection-task): collected results from producer collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 4 producer_id = 322a7ed9-abbd-411d-8ee5-1a1fdff896f1 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
972026-08-26T21:29:32.411ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 322a7ed9-abbd-411d-8ee5-1a1fdff896f1 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
982026-08-26T21:29:32.411ZDEBGoximeter (collection-task): collecting from producer collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 322a7ed9-abbd-411d-8ee5-1a1fdff896f1 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
992026-08-26T21:29:32.412ZDEBGoximeter (collection-task): collected results from producer collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 4 producer_id = 322a7ed9-abbd-411d-8ee5-1a1fdff896f1 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
1002026-08-26T21:29:33.411ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 322a7ed9-abbd-411d-8ee5-1a1fdff896f1 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
1012026-08-26T21:29:33.411ZDEBGoximeter (collection-task): collecting from producer collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 322a7ed9-abbd-411d-8ee5-1a1fdff896f1 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
1022026-08-26T21:29:33.411ZDEBGoximeter (collection-task): collected results from producer collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 4 producer_id = 322a7ed9-abbd-411d-8ee5-1a1fdff896f1 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
1032026-08-26T21:29:34.412ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 322a7ed9-abbd-411d-8ee5-1a1fdff896f1 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
1042026-08-26T21:29:34.412ZDEBGoximeter (collection-task): collecting from producer collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 322a7ed9-abbd-411d-8ee5-1a1fdff896f1 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
1052026-08-26T21:29:34.413ZDEBGoximeter (collection-task): collected results from producer collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 4 producer_id = 322a7ed9-abbd-411d-8ee5-1a1fdff896f1 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
1062026-08-26T21:29:34.812ZDEBGoximeter (database-inserter): inserting samples into database collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d n_samples = 20
1072026-08-26T21:29:34.812ZDEBGoximeter (database-inserter): inserting samples into database collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d n_samples = 20
1082026-08-26T21:29:34.812ZDEBGoximeter (clickhouse-client): unrolling 20 total samples collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d id = 686df668-58e8-4ea0-8f4e-b1d302e96d30
1092026-08-26T21:29:34.812ZDEBGoximeter (clickhouse-client): fetching timeseries schema from database collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d id = 686df668-58e8-4ea0-8f4e-b1d302e96d30
1102026-08-26T21:29:34.820ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d id = 686df668-58e8-4ea0-8f4e-b1d302e96d30 n_rows = 4 table_name = fields_string
1112026-08-26T21:29:34.822ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d id = 686df668-58e8-4ea0-8f4e-b1d302e96d30 n_rows = 8 table_name = fields_uuid
1122026-08-26T21:29:34.823ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d id = 686df668-58e8-4ea0-8f4e-b1d302e96d30 n_rows = 10 table_name = measurements_cumulativeu64
1132026-08-26T21:29:34.825ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d id = 686df668-58e8-4ea0-8f4e-b1d302e96d30 n_rows = 10 table_name = measurements_u64
1142026-08-26T21:29:34.913ZWARNoximeter (database-inserter): failed to insert some results into metric DB collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:276
1152026-08-26T21:29:35.411ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 322a7ed9-abbd-411d-8ee5-1a1fdff896f1 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
1162026-08-26T21:29:35.411ZDEBGoximeter (collection-task): collecting from producer collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 322a7ed9-abbd-411d-8ee5-1a1fdff896f1 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
1172026-08-26T21:29:35.412ZDEBGoximeter (collection-task): collected results from producer collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 4 producer_id = 322a7ed9-abbd-411d-8ee5-1a1fdff896f1 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
1182026-08-26T21:29:36.412ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 322a7ed9-abbd-411d-8ee5-1a1fdff896f1 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
1192026-08-26T21:29:36.412ZDEBGoximeter (collection-task): collecting from producer collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 322a7ed9-abbd-411d-8ee5-1a1fdff896f1 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
1202026-08-26T21:29:36.412ZDEBGoximeter (collection-task): collected results from producer collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 4 producer_id = 322a7ed9-abbd-411d-8ee5-1a1fdff896f1 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
1212026-08-26T21:29:37.411ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 322a7ed9-abbd-411d-8ee5-1a1fdff896f1 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
1222026-08-26T21:29:37.411ZDEBGoximeter (collection-task): collecting from producer collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 322a7ed9-abbd-411d-8ee5-1a1fdff896f1 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
1232026-08-26T21:29:37.412ZDEBGoximeter (collection-task): collected results from producer collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 4 producer_id = 322a7ed9-abbd-411d-8ee5-1a1fdff896f1 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
1242026-08-26T21:29:38.411ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 322a7ed9-abbd-411d-8ee5-1a1fdff896f1 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
1252026-08-26T21:29:38.411ZDEBGoximeter (collection-task): collecting from producer collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 322a7ed9-abbd-411d-8ee5-1a1fdff896f1 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
1262026-08-26T21:29:38.412ZDEBGoximeter (collection-task): collected results from producer collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 4 producer_id = 322a7ed9-abbd-411d-8ee5-1a1fdff896f1 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
1272026-08-26T21:29:39.412ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 322a7ed9-abbd-411d-8ee5-1a1fdff896f1 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
1282026-08-26T21:29:39.412ZDEBGoximeter (collection-task): collecting from producer collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 322a7ed9-abbd-411d-8ee5-1a1fdff896f1 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
1292026-08-26T21:29:39.412ZDEBGoximeter (collection-task): collected results from producer collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 4 producer_id = 322a7ed9-abbd-411d-8ee5-1a1fdff896f1 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
1302026-08-26T21:29:39.811ZDEBGoximeter (database-inserter): inserting samples into database collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d n_samples = 20
1312026-08-26T21:29:39.812ZDEBGoximeter (database-inserter): inserting samples into database collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d n_samples = 20
1322026-08-26T21:29:39.812ZDEBGoximeter (clickhouse-client): unrolling 20 total samples collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d id = 686df668-58e8-4ea0-8f4e-b1d302e96d30
1332026-08-26T21:29:39.812ZDEBGoximeter (clickhouse-client): fetching timeseries schema from database collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d id = 686df668-58e8-4ea0-8f4e-b1d302e96d30
1342026-08-26T21:29:39.817ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d id = 686df668-58e8-4ea0-8f4e-b1d302e96d30 n_rows = 4 table_name = fields_string
1352026-08-26T21:29:39.818ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d id = 686df668-58e8-4ea0-8f4e-b1d302e96d30 n_rows = 8 table_name = fields_uuid
1362026-08-26T21:29:39.820ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d id = 686df668-58e8-4ea0-8f4e-b1d302e96d30 n_rows = 10 table_name = measurements_cumulativeu64
1372026-08-26T21:29:39.822ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d id = 686df668-58e8-4ea0-8f4e-b1d302e96d30 n_rows = 10 table_name = measurements_u64
1382026-08-26T21:29:39.907ZINFOoximeter (oximeter-agent): refreshing list of producers from Nexus collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d file = oximeter/collector/src/agent.rs:629
1392026-08-26T21:29:39.907ZDEBGoximeter: client request body = None method = GET uri = http://[fd00:1122:3344:101::a]:12221/metrics/collectors/8434ede6-327d-44e3-8722-6c9e65e5b476/producers?sort_by=id_ascending
1402026-08-26T21:29:39.911ZDEBGoximeter: client response result = Ok(Response { url: "http://[fd00:1122:3344:101::a]:12221/metrics/collectors/8434ede6-327d-44e3-8722-6c9e65e5b476/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "3f3ae090-34f1-4d54-b7da-d62e9bcb981a", "content-length": "572", "date": "Wed, 26 Aug 2026 21:29:39 GMT"} })
1412026-08-26T21:29:39.911ZDEBGoximeter: client request body = None method = GET uri = http://[fd00:1122:3344:101::a]:12221/metrics/collectors/8434ede6-327d-44e3-8722-6c9e65e5b476/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiI0MmUwMTI1Mi1iMWU2LTQ0YjUtOGU4MC1mZGEzODE3Nzg2MGIifX0%3D
1422026-08-26T21:29:39.912ZWARNoximeter (database-inserter): failed to insert some results into metric DB collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:276
1432026-08-26T21:29:39.915ZDEBGoximeter: client response result = Ok(Response { url: "http://[fd00:1122:3344:101::a]:12221/metrics/collectors/8434ede6-327d-44e3-8722-6c9e65e5b476/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiI0MmUwMTI1Mi1iMWU2LTQ0YjUtOGU4MC1mZGEzODE3Nzg2MGIifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "2ec70620-fd1d-4c98-83e4-27e0103daf0c", "content-length": "29", "date": "Wed, 26 Aug 2026 21:29:39 GMT"} })
1442026-08-26T21:29:39.915ZDEBGoximeter (oximeter-agent): registered new metric producer address = [fd00:1122:3344:101::1]:8001 collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 1132f55a-c26d-4393-a6a9-70855ee9911b
1452026-08-26T21:29:39.915ZINFOoximeter (oximeter-agent): refreshed list of producers from Nexus collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d file = oximeter/collector/src/agent.rs:704 n_current_tasks = 3 n_pruned_tasks = 0
1462026-08-26T21:29:40.019ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 1132f55a-c26d-4393-a6a9-70855ee9911b producer_ip = fd00:1122:3344:101::1 producer_port = 8001
1472026-08-26T21:29:40.019ZDEBGoximeter (collection-task): collecting from producer collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 1132f55a-c26d-4393-a6a9-70855ee9911b producer_ip = fd00:1122:3344:101::1 producer_port = 8001
1482026-08-26T21:29:40.025ZDEBGoximeter (collection-task): collected results from producer collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 26 producer_id = 1132f55a-c26d-4393-a6a9-70855ee9911b producer_ip = fd00:1122:3344:101::1 producer_port = 8001
1492026-08-26T21:29:40.412ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 322a7ed9-abbd-411d-8ee5-1a1fdff896f1 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
1502026-08-26T21:29:40.412ZDEBGoximeter (collection-task): collecting from producer collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 322a7ed9-abbd-411d-8ee5-1a1fdff896f1 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
1512026-08-26T21:29:40.413ZDEBGoximeter (collection-task): collected results from producer collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 4 producer_id = 322a7ed9-abbd-411d-8ee5-1a1fdff896f1 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
1522026-08-26T21:29:41.019ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 1132f55a-c26d-4393-a6a9-70855ee9911b producer_ip = fd00:1122:3344:101::1 producer_port = 8001
1532026-08-26T21:29:41.019ZDEBGoximeter (collection-task): collecting from producer collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 1132f55a-c26d-4393-a6a9-70855ee9911b producer_ip = fd00:1122:3344:101::1 producer_port = 8001
1542026-08-26T21:29:41.021ZDEBGoximeter (collection-task): collected results from producer collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 26 producer_id = 1132f55a-c26d-4393-a6a9-70855ee9911b producer_ip = fd00:1122:3344:101::1 producer_port = 8001
1552026-08-26T21:29:41.411ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 322a7ed9-abbd-411d-8ee5-1a1fdff896f1 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
1562026-08-26T21:29:41.411ZDEBGoximeter (collection-task): collecting from producer collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 322a7ed9-abbd-411d-8ee5-1a1fdff896f1 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
1572026-08-26T21:29:41.412ZDEBGoximeter (collection-task): collected results from producer collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 4 producer_id = 322a7ed9-abbd-411d-8ee5-1a1fdff896f1 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
1582026-08-26T21:29:41.524ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 42e01252-b1e6-44b5-8e80-fda38177860b producer_ip = fd00:1122:3344:101::2 producer_port = 38589
1592026-08-26T21:29:41.524ZDEBGoximeter (collection-task): collecting from producer collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 42e01252-b1e6-44b5-8e80-fda38177860b producer_ip = fd00:1122:3344:101::2 producer_port = 38589
1602026-08-26T21:29:41.543ZDEBGoximeter (collection-task): collected results from producer collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 228 producer_id = 42e01252-b1e6-44b5-8e80-fda38177860b producer_ip = fd00:1122:3344:101::2 producer_port = 38589
1612026-08-26T21:29:42.020ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 1132f55a-c26d-4393-a6a9-70855ee9911b producer_ip = fd00:1122:3344:101::1 producer_port = 8001
1622026-08-26T21:29:42.020ZDEBGoximeter (collection-task): collecting from producer collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 1132f55a-c26d-4393-a6a9-70855ee9911b producer_ip = fd00:1122:3344:101::1 producer_port = 8001
1632026-08-26T21:29:42.021ZDEBGoximeter (collection-task): collected results from producer collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 26 producer_id = 1132f55a-c26d-4393-a6a9-70855ee9911b producer_ip = fd00:1122:3344:101::1 producer_port = 8001
1642026-08-26T21:29:42.411ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 322a7ed9-abbd-411d-8ee5-1a1fdff896f1 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
1652026-08-26T21:29:42.411ZDEBGoximeter (collection-task): collecting from producer collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 322a7ed9-abbd-411d-8ee5-1a1fdff896f1 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
1662026-08-26T21:29:42.412ZDEBGoximeter (collection-task): collected results from producer collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 4 producer_id = 322a7ed9-abbd-411d-8ee5-1a1fdff896f1 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
1672026-08-26T21:29:43.019ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 1132f55a-c26d-4393-a6a9-70855ee9911b producer_ip = fd00:1122:3344:101::1 producer_port = 8001
1682026-08-26T21:29:43.019ZDEBGoximeter (collection-task): collecting from producer collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 1132f55a-c26d-4393-a6a9-70855ee9911b producer_ip = fd00:1122:3344:101::1 producer_port = 8001
1692026-08-26T21:29:43.022ZDEBGoximeter (collection-task): collected results from producer collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 26 producer_id = 1132f55a-c26d-4393-a6a9-70855ee9911b producer_ip = fd00:1122:3344:101::1 producer_port = 8001
1702026-08-26T21:29:43.411ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 322a7ed9-abbd-411d-8ee5-1a1fdff896f1 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
1712026-08-26T21:29:43.411ZDEBGoximeter (collection-task): collecting from producer collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 322a7ed9-abbd-411d-8ee5-1a1fdff896f1 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
1722026-08-26T21:29:43.412ZDEBGoximeter (collection-task): collected results from producer collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 4 producer_id = 322a7ed9-abbd-411d-8ee5-1a1fdff896f1 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
1732026-08-26T21:29:44.020ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 1132f55a-c26d-4393-a6a9-70855ee9911b producer_ip = fd00:1122:3344:101::1 producer_port = 8001
1742026-08-26T21:29:44.020ZDEBGoximeter (collection-task): collecting from producer collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 1132f55a-c26d-4393-a6a9-70855ee9911b producer_ip = fd00:1122:3344:101::1 producer_port = 8001
1752026-08-26T21:29:44.021ZDEBGoximeter (collection-task): collected results from producer collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 26 producer_id = 1132f55a-c26d-4393-a6a9-70855ee9911b producer_ip = fd00:1122:3344:101::1 producer_port = 8001
1762026-08-26T21:29:44.411ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 322a7ed9-abbd-411d-8ee5-1a1fdff896f1 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
1772026-08-26T21:29:44.411ZDEBGoximeter (collection-task): collecting from producer collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 322a7ed9-abbd-411d-8ee5-1a1fdff896f1 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
1782026-08-26T21:29:44.412ZDEBGoximeter (collection-task): collected results from producer collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 4 producer_id = 322a7ed9-abbd-411d-8ee5-1a1fdff896f1 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
1792026-08-26T21:29:44.811ZDEBGoximeter (database-inserter): inserting samples into database collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d n_samples = 378
1802026-08-26T21:29:44.811ZDEBGoximeter (database-inserter): inserting samples into database collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d n_samples = 378
1812026-08-26T21:29:44.811ZDEBGoximeter (clickhouse-client): unrolling 378 total samples collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d id = 686df668-58e8-4ea0-8f4e-b1d302e96d30
1822026-08-26T21:29:44.811ZDEBGoximeter (clickhouse-client): fetching timeseries schema from database collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d id = 686df668-58e8-4ea0-8f4e-b1d302e96d30
1832026-08-26T21:29:44.814ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d id = 686df668-58e8-4ea0-8f4e-b1d302e96d30 schema = TimeseriesSchema { timeseries_name: TimeseriesName("ddm_router:originated_tunnel_endpoints"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "hostname", field_type: String, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }}, datum_type: U64, version: 1, authz_scope: Fleet, units: Count, created: 2026-08-26T21:29:44.811731387Z } timeseries_name = ddm_router:originated_tunnel_endpoints
1842026-08-26T21:29:44.814ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d id = 686df668-58e8-4ea0-8f4e-b1d302e96d30 schema = TimeseriesSchema { timeseries_name: TimeseriesName("ddm_router:originated_underlay_prefixes"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "hostname", field_type: String, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }}, datum_type: U64, version: 1, authz_scope: Fleet, units: Count, created: 2026-08-26T21:29:44.811717696Z } timeseries_name = ddm_router:originated_underlay_prefixes
1852026-08-26T21:29:44.814ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d id = 686df668-58e8-4ea0-8f4e-b1d302e96d30 schema = TimeseriesSchema { timeseries_name: TimeseriesName("ddm_session:advertisements_received"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "hostname", field_type: String, source: Target, description: "" }, FieldSchema { name: "interface", field_type: String, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }}, datum_type: CumulativeU64, version: 1, authz_scope: Fleet, units: Count, created: 2026-08-26T21:29:44.811739967Z } timeseries_name = ddm_session:advertisements_received
1862026-08-26T21:29:44.814ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d id = 686df668-58e8-4ea0-8f4e-b1d302e96d30 schema = TimeseriesSchema { timeseries_name: TimeseriesName("ddm_session:advertisements_sent"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "hostname", field_type: String, source: Target, description: "" }, FieldSchema { name: "interface", field_type: String, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }}, datum_type: CumulativeU64, version: 1, authz_scope: Fleet, units: Count, created: 2026-08-26T21:29:44.811737757Z } timeseries_name = ddm_session:advertisements_sent
1872026-08-26T21:29:44.814ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d id = 686df668-58e8-4ea0-8f4e-b1d302e96d30 schema = TimeseriesSchema { timeseries_name: TimeseriesName("ddm_session:imported_tunnel_endpoints"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "hostname", field_type: String, source: Target, description: "" }, FieldSchema { name: "interface", field_type: String, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }}, datum_type: U64, version: 1, authz_scope: Fleet, units: Count, created: 2026-08-26T21:29:44.811753867Z } timeseries_name = ddm_session:imported_tunnel_endpoints
1882026-08-26T21:29:44.814ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d id = 686df668-58e8-4ea0-8f4e-b1d302e96d30 schema = TimeseriesSchema { timeseries_name: TimeseriesName("ddm_session:imported_underlay_prefixes"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "hostname", field_type: String, source: Target, description: "" }, FieldSchema { name: "interface", field_type: String, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }}, datum_type: U64, version: 1, authz_scope: Fleet, units: Count, created: 2026-08-26T21:29:44.811752297Z } timeseries_name = ddm_session:imported_underlay_prefixes
1892026-08-26T21:29:44.814ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d id = 686df668-58e8-4ea0-8f4e-b1d302e96d30 schema = TimeseriesSchema { timeseries_name: TimeseriesName("ddm_session:peer_address_changes"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "hostname", field_type: String, source: Target, description: "" }, FieldSchema { name: "interface", field_type: String, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }}, datum_type: CumulativeU64, version: 1, authz_scope: Fleet, units: Count, created: 2026-08-26T21:29:44.811743347Z } timeseries_name = ddm_session:peer_address_changes
1902026-08-26T21:29:44.814ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d id = 686df668-58e8-4ea0-8f4e-b1d302e96d30 schema = TimeseriesSchema { timeseries_name: TimeseriesName("ddm_session:peer_expirations"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "hostname", field_type: String, source: Target, description: "" }, FieldSchema { name: "interface", field_type: String, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }}, datum_type: CumulativeU64, version: 1, authz_scope: Fleet, units: Count, created: 2026-08-26T21:29:44.811741607Z } timeseries_name = ddm_session:peer_expirations
1912026-08-26T21:29:44.814ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d id = 686df668-58e8-4ea0-8f4e-b1d302e96d30 schema = TimeseriesSchema { timeseries_name: TimeseriesName("ddm_session:peer_sessions_established"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "hostname", field_type: String, source: Target, description: "" }, FieldSchema { name: "interface", field_type: String, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }}, datum_type: CumulativeU64, version: 1, authz_scope: Fleet, units: Count, created: 2026-08-26T21:29:44.811744997Z } timeseries_name = ddm_session:peer_sessions_established
1922026-08-26T21:29:44.814ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d id = 686df668-58e8-4ea0-8f4e-b1d302e96d30 schema = TimeseriesSchema { timeseries_name: TimeseriesName("ddm_session:solicitations_received"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "hostname", field_type: String, source: Target, description: "" }, FieldSchema { name: "interface", field_type: String, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }}, datum_type: CumulativeU64, version: 1, authz_scope: Fleet, units: Count, created: 2026-08-26T21:29:44.811735987Z } timeseries_name = ddm_session:solicitations_received
1932026-08-26T21:29:44.814ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d id = 686df668-58e8-4ea0-8f4e-b1d302e96d30 schema = TimeseriesSchema { timeseries_name: TimeseriesName("ddm_session:solicitations_sent"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "hostname", field_type: String, source: Target, description: "" }, FieldSchema { name: "interface", field_type: String, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }}, datum_type: CumulativeU64, version: 1, authz_scope: Fleet, units: Count, created: 2026-08-26T21:29:44.811733797Z } timeseries_name = ddm_session:solicitations_sent
1942026-08-26T21:29:44.814ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d id = 686df668-58e8-4ea0-8f4e-b1d302e96d30 schema = TimeseriesSchema { timeseries_name: TimeseriesName("ddm_session:update_send_fail"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "hostname", field_type: String, source: Target, description: "" }, FieldSchema { name: "interface", field_type: String, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }}, datum_type: CumulativeU64, version: 1, authz_scope: Fleet, units: Count, created: 2026-08-26T21:29:44.811749627Z } timeseries_name = ddm_session:update_send_fail
1952026-08-26T21:29:44.815ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d id = 686df668-58e8-4ea0-8f4e-b1d302e96d30 schema = TimeseriesSchema { timeseries_name: TimeseriesName("ddm_session:updates_received"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "hostname", field_type: String, source: Target, description: "" }, FieldSchema { name: "interface", field_type: String, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }}, datum_type: CumulativeU64, version: 1, authz_scope: Fleet, units: Count, created: 2026-08-26T21:29:44.811748137Z } timeseries_name = ddm_session:updates_received
1962026-08-26T21:29:44.815ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d id = 686df668-58e8-4ea0-8f4e-b1d302e96d30 schema = TimeseriesSchema { timeseries_name: TimeseriesName("ddm_session:updates_sent"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "hostname", field_type: String, source: Target, description: "" }, FieldSchema { name: "interface", field_type: String, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }}, datum_type: CumulativeU64, version: 1, authz_scope: Fleet, units: Count, created: 2026-08-26T21:29:44.811746867Z } timeseries_name = ddm_session:updates_sent
1972026-08-26T21:29:44.815ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d id = 686df668-58e8-4ea0-8f4e-b1d302e96d30 schema = TimeseriesSchema { timeseries_name: TimeseriesName("switch_port_control_data_link:bytes_received"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "asic_fab", field_type: String, source: Target, description: "" }, FieldSchema { name: "asic_lot", field_type: String, source: Target, description: "" }, FieldSchema { name: "asic_wafer", field_type: U8, source: Target, description: "" }, FieldSchema { name: "asic_wafer_loc_x", field_type: I16, source: Target, description: "" }, FieldSchema { name: "asic_wafer_loc_y", field_type: I16, source: Target, description: "" }, FieldSchema { name: "kind", field_type: String, source: Target, description: "" }, FieldSchema { name: "link_name", field_type: String, source: Target, description: "" }, FieldSchema { name: "model", field_type: String, source: Target, description: "" }, FieldSchema { name: "network", field_type: String, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_model", field_type: String, source: Target, description: "" }, FieldSchema { name: "sled_revision", field_type: U32, source: Target, description: "" }, FieldSchema { name: "sled_serial", field_type: String, source: Target, description: "" }, FieldSchema { name: "switch_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "switch_model", field_type: String, source: Target, description: "" }, FieldSchema { name: "switch_revision", field_type: U32, source: Target, description: "" }, FieldSchema { name: "switch_serial", field_type: String, source: Target, description: "" }, FieldSchema { name: "switch_slot", field_type: U16, source: Target, description: "" }}, datum_type: CumulativeU64, version: 1, authz_scope: Fleet, units: Count, created: 2026-08-26T21:29:44.811795148Z } timeseries_name = switch_port_control_data_link:bytes_received
1982026-08-26T21:29:44.815ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d id = 686df668-58e8-4ea0-8f4e-b1d302e96d30 schema = TimeseriesSchema { timeseries_name: TimeseriesName("switch_port_control_data_link:bytes_sent"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "asic_fab", field_type: String, source: Target, description: "" }, FieldSchema { name: "asic_lot", field_type: String, source: Target, description: "" }, FieldSchema { name: "asic_wafer", field_type: U8, source: Target, description: "" }, FieldSchema { name: "asic_wafer_loc_x", field_type: I16, source: Target, description: "" }, FieldSchema { name: "asic_wafer_loc_y", field_type: I16, source: Target, description: "" }, FieldSchema { name: "kind", field_type: String, source: Target, description: "" }, FieldSchema { name: "link_name", field_type: String, source: Target, description: "" }, FieldSchema { name: "model", field_type: String, source: Target, description: "" }, FieldSchema { name: "network", field_type: String, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_model", field_type: String, source: Target, description: "" }, FieldSchema { name: "sled_revision", field_type: U32, source: Target, description: "" }, FieldSchema { name: "sled_serial", field_type: String, source: Target, description: "" }, FieldSchema { name: "switch_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "switch_model", field_type: String, source: Target, description: "" }, FieldSchema { name: "switch_revision", field_type: U32, source: Target, description: "" }, FieldSchema { name: "switch_serial", field_type: String, source: Target, description: "" }, FieldSchema { name: "switch_slot", field_type: U16, source: Target, description: "" }}, datum_type: CumulativeU64, version: 1, authz_scope: Fleet, units: Count, created: 2026-08-26T21:29:44.811805218Z } timeseries_name = switch_port_control_data_link:bytes_sent
1992026-08-26T21:29:44.815ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d id = 686df668-58e8-4ea0-8f4e-b1d302e96d30 schema = TimeseriesSchema { timeseries_name: TimeseriesName("switch_port_control_data_link:errors_received"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "asic_fab", field_type: String, source: Target, description: "" }, FieldSchema { name: "asic_lot", field_type: String, source: Target, description: "" }, FieldSchema { name: "asic_wafer", field_type: U8, source: Target, description: "" }, FieldSchema { name: "asic_wafer_loc_x", field_type: I16, source: Target, description: "" }, FieldSchema { name: "asic_wafer_loc_y", field_type: I16, source: Target, description: "" }, FieldSchema { name: "kind", field_type: String, source: Target, description: "" }, FieldSchema { name: "link_name", field_type: String, source: Target, description: "" }, FieldSchema { name: "model", field_type: String, source: Target, description: "" }, FieldSchema { name: "network", field_type: String, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_model", field_type: String, source: Target, description: "" }, FieldSchema { name: "sled_revision", field_type: U32, source: Target, description: "" }, FieldSchema { name: "sled_serial", field_type: String, source: Target, description: "" }, FieldSchema { name: "switch_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "switch_model", field_type: String, source: Target, description: "" }, FieldSchema { name: "switch_revision", field_type: U32, source: Target, description: "" }, FieldSchema { name: "switch_serial", field_type: String, source: Target, description: "" }, FieldSchema { name: "switch_slot", field_type: U16, source: Target, description: "" }}, datum_type: CumulativeU64, version: 1, authz_scope: Fleet, units: Count, created: 2026-08-26T21:29:44.811779808Z } timeseries_name = switch_port_control_data_link:errors_received
2002026-08-26T21:29:44.815ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d id = 686df668-58e8-4ea0-8f4e-b1d302e96d30 schema = TimeseriesSchema { timeseries_name: TimeseriesName("switch_port_control_data_link:errors_sent"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "asic_fab", field_type: String, source: Target, description: "" }, FieldSchema { name: "asic_lot", field_type: String, source: Target, description: "" }, FieldSchema { name: "asic_wafer", field_type: U8, source: Target, description: "" }, FieldSchema { name: "asic_wafer_loc_x", field_type: I16, source: Target, description: "" }, FieldSchema { name: "asic_wafer_loc_y", field_type: I16, source: Target, description: "" }, FieldSchema { name: "kind", field_type: String, source: Target, description: "" }, FieldSchema { name: "link_name", field_type: String, source: Target, description: "" }, FieldSchema { name: "model", field_type: String, source: Target, description: "" }, FieldSchema { name: "network", field_type: String, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_model", field_type: String, source: Target, description: "" }, FieldSchema { name: "sled_revision", field_type: U32, source: Target, description: "" }, FieldSchema { name: "sled_serial", field_type: String, source: Target, description: "" }, FieldSchema { name: "switch_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "switch_model", field_type: String, source: Target, description: "" }, FieldSchema { name: "switch_revision", field_type: U32, source: Target, description: "" }, FieldSchema { name: "switch_serial", field_type: String, source: Target, description: "" }, FieldSchema { name: "switch_slot", field_type: U16, source: Target, description: "" }}, datum_type: CumulativeU64, version: 1, authz_scope: Fleet, units: Count, created: 2026-08-26T21:29:44.811785728Z } timeseries_name = switch_port_control_data_link:errors_sent
2012026-08-26T21:29:44.815ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d id = 686df668-58e8-4ea0-8f4e-b1d302e96d30 schema = TimeseriesSchema { timeseries_name: TimeseriesName("switch_port_control_data_link:packets_received"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "asic_fab", field_type: String, source: Target, description: "" }, FieldSchema { name: "asic_lot", field_type: String, source: Target, description: "" }, FieldSchema { name: "asic_wafer", field_type: U8, source: Target, description: "" }, FieldSchema { name: "asic_wafer_loc_x", field_type: I16, source: Target, description: "" }, FieldSchema { name: "asic_wafer_loc_y", field_type: I16, source: Target, description: "" }, FieldSchema { name: "kind", field_type: String, source: Target, description: "" }, FieldSchema { name: "link_name", field_type: String, source: Target, description: "" }, FieldSchema { name: "model", field_type: String, source: Target, description: "" }, FieldSchema { name: "network", field_type: String, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_model", field_type: String, source: Target, description: "" }, FieldSchema { name: "sled_revision", field_type: U32, source: Target, description: "" }, FieldSchema { name: "sled_serial", field_type: String, source: Target, description: "" }, FieldSchema { name: "switch_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "switch_model", field_type: String, source: Target, description: "" }, FieldSchema { name: "switch_revision", field_type: U32, source: Target, description: "" }, FieldSchema { name: "switch_serial", field_type: String, source: Target, description: "" }, FieldSchema { name: "switch_slot", field_type: U16, source: Target, description: "" }}, datum_type: CumulativeU64, version: 1, authz_scope: Fleet, units: Count, created: 2026-08-26T21:29:44.811800748Z } timeseries_name = switch_port_control_data_link:packets_received
2022026-08-26T21:29:44.815ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d id = 686df668-58e8-4ea0-8f4e-b1d302e96d30 schema = TimeseriesSchema { timeseries_name: TimeseriesName("switch_port_control_data_link:packets_sent"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "asic_fab", field_type: String, source: Target, description: "" }, FieldSchema { name: "asic_lot", field_type: String, source: Target, description: "" }, FieldSchema { name: "asic_wafer", field_type: U8, source: Target, description: "" }, FieldSchema { name: "asic_wafer_loc_x", field_type: I16, source: Target, description: "" }, FieldSchema { name: "asic_wafer_loc_y", field_type: I16, source: Target, description: "" }, FieldSchema { name: "kind", field_type: String, source: Target, description: "" }, FieldSchema { name: "link_name", field_type: String, source: Target, description: "" }, FieldSchema { name: "model", field_type: String, source: Target, description: "" }, FieldSchema { name: "network", field_type: String, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_model", field_type: String, source: Target, description: "" }, FieldSchema { name: "sled_revision", field_type: U32, source: Target, description: "" }, FieldSchema { name: "sled_serial", field_type: String, source: Target, description: "" }, FieldSchema { name: "switch_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "switch_model", field_type: String, source: Target, description: "" }, FieldSchema { name: "switch_revision", field_type: U32, source: Target, description: "" }, FieldSchema { name: "switch_serial", field_type: String, source: Target, description: "" }, FieldSchema { name: "switch_slot", field_type: U16, source: Target, description: "" }}, datum_type: CumulativeU64, version: 1, authz_scope: Fleet, units: Count, created: 2026-08-26T21:29:44.811811928Z } timeseries_name = switch_port_control_data_link:packets_sent
2032026-08-26T21:29:44.817ZDEBGoximeter (clickhouse-client): inserting 20 new timeseries schema collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d id = 686df668-58e8-4ea0-8f4e-b1d302e96d30
2042026-08-26T21:29:44.821ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d id = 686df668-58e8-4ea0-8f4e-b1d302e96d30 n_rows = 24 table_name = fields_i16
2052026-08-26T21:29:44.823ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d id = 686df668-58e8-4ea0-8f4e-b1d302e96d30 n_rows = 174 table_name = fields_string
2062026-08-26T21:29:44.825ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d id = 686df668-58e8-4ea0-8f4e-b1d302e96d30 n_rows = 12 table_name = fields_u16
2072026-08-26T21:29:44.827ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d id = 686df668-58e8-4ea0-8f4e-b1d302e96d30 n_rows = 24 table_name = fields_u32
2082026-08-26T21:29:44.828ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d id = 686df668-58e8-4ea0-8f4e-b1d302e96d30 n_rows = 12 table_name = fields_u8
2092026-08-26T21:29:44.830ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d id = 686df668-58e8-4ea0-8f4e-b1d302e96d30 n_rows = 96 table_name = fields_uuid
2102026-08-26T21:29:44.832ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d id = 686df668-58e8-4ea0-8f4e-b1d302e96d30 n_rows = 338 table_name = measurements_cumulativeu64
2112026-08-26T21:29:44.833ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d id = 686df668-58e8-4ea0-8f4e-b1d302e96d30 n_rows = 40 table_name = measurements_u64
2122026-08-26T21:29:44.912ZWARNoximeter (database-inserter): failed to insert some results into metric DB collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:276
2132026-08-26T21:29:45.019ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 1132f55a-c26d-4393-a6a9-70855ee9911b producer_ip = fd00:1122:3344:101::1 producer_port = 8001
2142026-08-26T21:29:45.019ZDEBGoximeter (collection-task): collecting from producer collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 1132f55a-c26d-4393-a6a9-70855ee9911b producer_ip = fd00:1122:3344:101::1 producer_port = 8001
2152026-08-26T21:29:45.021ZDEBGoximeter (collection-task): collected results from producer collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 26 producer_id = 1132f55a-c26d-4393-a6a9-70855ee9911b producer_ip = fd00:1122:3344:101::1 producer_port = 8001
2162026-08-26T21:29:45.411ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 322a7ed9-abbd-411d-8ee5-1a1fdff896f1 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
2172026-08-26T21:29:45.411ZDEBGoximeter (collection-task): collecting from producer collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 322a7ed9-abbd-411d-8ee5-1a1fdff896f1 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
2182026-08-26T21:29:45.412ZDEBGoximeter (collection-task): collected results from producer collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 4 producer_id = 322a7ed9-abbd-411d-8ee5-1a1fdff896f1 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
2192026-08-26T21:29:46.020ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 1132f55a-c26d-4393-a6a9-70855ee9911b producer_ip = fd00:1122:3344:101::1 producer_port = 8001
2202026-08-26T21:29:46.020ZDEBGoximeter (collection-task): collecting from producer collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 1132f55a-c26d-4393-a6a9-70855ee9911b producer_ip = fd00:1122:3344:101::1 producer_port = 8001
2212026-08-26T21:29:46.021ZDEBGoximeter (collection-task): collected results from producer collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 26 producer_id = 1132f55a-c26d-4393-a6a9-70855ee9911b producer_ip = fd00:1122:3344:101::1 producer_port = 8001
2222026-08-26T21:29:46.411ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 322a7ed9-abbd-411d-8ee5-1a1fdff896f1 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
2232026-08-26T21:29:46.412ZDEBGoximeter (collection-task): collecting from producer collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 322a7ed9-abbd-411d-8ee5-1a1fdff896f1 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
2242026-08-26T21:29:46.412ZDEBGoximeter (collection-task): collected results from producer collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 4 producer_id = 322a7ed9-abbd-411d-8ee5-1a1fdff896f1 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
2252026-08-26T21:29:47.019ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 1132f55a-c26d-4393-a6a9-70855ee9911b producer_ip = fd00:1122:3344:101::1 producer_port = 8001
2262026-08-26T21:29:47.019ZDEBGoximeter (collection-task): collecting from producer collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 1132f55a-c26d-4393-a6a9-70855ee9911b producer_ip = fd00:1122:3344:101::1 producer_port = 8001
2272026-08-26T21:29:47.021ZDEBGoximeter (collection-task): collected results from producer collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 26 producer_id = 1132f55a-c26d-4393-a6a9-70855ee9911b producer_ip = fd00:1122:3344:101::1 producer_port = 8001
2282026-08-26T21:29:47.411ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 322a7ed9-abbd-411d-8ee5-1a1fdff896f1 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
2292026-08-26T21:29:47.411ZDEBGoximeter (collection-task): collecting from producer collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 322a7ed9-abbd-411d-8ee5-1a1fdff896f1 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
2302026-08-26T21:29:47.412ZDEBGoximeter (collection-task): collected results from producer collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 4 producer_id = 322a7ed9-abbd-411d-8ee5-1a1fdff896f1 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
2312026-08-26T21:29:48.019ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 1132f55a-c26d-4393-a6a9-70855ee9911b producer_ip = fd00:1122:3344:101::1 producer_port = 8001
2322026-08-26T21:29:48.019ZDEBGoximeter (collection-task): collecting from producer collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 1132f55a-c26d-4393-a6a9-70855ee9911b producer_ip = fd00:1122:3344:101::1 producer_port = 8001
2332026-08-26T21:29:48.021ZDEBGoximeter (collection-task): collected results from producer collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 26 producer_id = 1132f55a-c26d-4393-a6a9-70855ee9911b producer_ip = fd00:1122:3344:101::1 producer_port = 8001
2342026-08-26T21:29:48.412ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 322a7ed9-abbd-411d-8ee5-1a1fdff896f1 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
2352026-08-26T21:29:48.412ZDEBGoximeter (collection-task): collecting from producer collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 322a7ed9-abbd-411d-8ee5-1a1fdff896f1 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
2362026-08-26T21:29:48.412ZDEBGoximeter (collection-task): collected results from producer collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 4 producer_id = 322a7ed9-abbd-411d-8ee5-1a1fdff896f1 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
2372026-08-26T21:29:49.020ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 1132f55a-c26d-4393-a6a9-70855ee9911b producer_ip = fd00:1122:3344:101::1 producer_port = 8001
2382026-08-26T21:29:49.020ZDEBGoximeter (collection-task): collecting from producer collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 1132f55a-c26d-4393-a6a9-70855ee9911b producer_ip = fd00:1122:3344:101::1 producer_port = 8001
2392026-08-26T21:29:49.021ZDEBGoximeter (collection-task): collected results from producer collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 26 producer_id = 1132f55a-c26d-4393-a6a9-70855ee9911b producer_ip = fd00:1122:3344:101::1 producer_port = 8001
2402026-08-26T21:29:49.411ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 322a7ed9-abbd-411d-8ee5-1a1fdff896f1 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
2412026-08-26T21:29:49.411ZDEBGoximeter (collection-task): collecting from producer collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 322a7ed9-abbd-411d-8ee5-1a1fdff896f1 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
2422026-08-26T21:29:49.412ZDEBGoximeter (collection-task): collected results from producer collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 4 producer_id = 322a7ed9-abbd-411d-8ee5-1a1fdff896f1 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
2432026-08-26T21:29:49.811ZDEBGoximeter (database-inserter): inserting samples into database collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d n_samples = 150
2442026-08-26T21:29:49.811ZDEBGoximeter (database-inserter): inserting samples into database collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d n_samples = 150
2452026-08-26T21:29:49.811ZDEBGoximeter (clickhouse-client): unrolling 150 total samples collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d id = 686df668-58e8-4ea0-8f4e-b1d302e96d30
2462026-08-26T21:29:49.811ZDEBGoximeter (clickhouse-client): fetching timeseries schema from database collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d id = 686df668-58e8-4ea0-8f4e-b1d302e96d30
2472026-08-26T21:29:49.821ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d id = 686df668-58e8-4ea0-8f4e-b1d302e96d30 n_rows = 54 table_name = fields_string
2482026-08-26T21:29:49.823ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d id = 686df668-58e8-4ea0-8f4e-b1d302e96d30 n_rows = 60 table_name = fields_uuid
2492026-08-26T21:29:49.825ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d id = 686df668-58e8-4ea0-8f4e-b1d302e96d30 n_rows = 110 table_name = measurements_cumulativeu64
2502026-08-26T21:29:49.826ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d id = 686df668-58e8-4ea0-8f4e-b1d302e96d30 n_rows = 40 table_name = measurements_u64
2512026-08-26T21:29:49.913ZWARNoximeter (database-inserter): failed to insert some results into metric DB collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:276
2522026-08-26T21:29:50.020ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 1132f55a-c26d-4393-a6a9-70855ee9911b producer_ip = fd00:1122:3344:101::1 producer_port = 8001
2532026-08-26T21:29:50.020ZDEBGoximeter (collection-task): collecting from producer collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 1132f55a-c26d-4393-a6a9-70855ee9911b producer_ip = fd00:1122:3344:101::1 producer_port = 8001
2542026-08-26T21:29:50.021ZDEBGoximeter (collection-task): collected results from producer collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 26 producer_id = 1132f55a-c26d-4393-a6a9-70855ee9911b producer_ip = fd00:1122:3344:101::1 producer_port = 8001
2552026-08-26T21:29:50.411ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 322a7ed9-abbd-411d-8ee5-1a1fdff896f1 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
2562026-08-26T21:29:50.411ZDEBGoximeter (collection-task): collecting from producer collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 322a7ed9-abbd-411d-8ee5-1a1fdff896f1 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
2572026-08-26T21:29:50.412ZDEBGoximeter (collection-task): collected results from producer collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 4 producer_id = 322a7ed9-abbd-411d-8ee5-1a1fdff896f1 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
2582026-08-26T21:29:51.019ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 1132f55a-c26d-4393-a6a9-70855ee9911b producer_ip = fd00:1122:3344:101::1 producer_port = 8001
2592026-08-26T21:29:51.019ZDEBGoximeter (collection-task): collecting from producer collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 1132f55a-c26d-4393-a6a9-70855ee9911b producer_ip = fd00:1122:3344:101::1 producer_port = 8001
2602026-08-26T21:29:51.021ZDEBGoximeter (collection-task): collected results from producer collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 26 producer_id = 1132f55a-c26d-4393-a6a9-70855ee9911b producer_ip = fd00:1122:3344:101::1 producer_port = 8001
2612026-08-26T21:29:51.411ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 322a7ed9-abbd-411d-8ee5-1a1fdff896f1 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
2622026-08-26T21:29:51.411ZDEBGoximeter (collection-task): collecting from producer collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 322a7ed9-abbd-411d-8ee5-1a1fdff896f1 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
2632026-08-26T21:29:51.412ZDEBGoximeter (collection-task): collected results from producer collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 4 producer_id = 322a7ed9-abbd-411d-8ee5-1a1fdff896f1 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
2642026-08-26T21:29:52.019ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 1132f55a-c26d-4393-a6a9-70855ee9911b producer_ip = fd00:1122:3344:101::1 producer_port = 8001
2652026-08-26T21:29:52.019ZDEBGoximeter (collection-task): collecting from producer collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 1132f55a-c26d-4393-a6a9-70855ee9911b producer_ip = fd00:1122:3344:101::1 producer_port = 8001
2662026-08-26T21:29:52.021ZDEBGoximeter (collection-task): collected results from producer collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 26 producer_id = 1132f55a-c26d-4393-a6a9-70855ee9911b producer_ip = fd00:1122:3344:101::1 producer_port = 8001
2672026-08-26T21:29:52.411ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 322a7ed9-abbd-411d-8ee5-1a1fdff896f1 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
2682026-08-26T21:29:52.411ZDEBGoximeter (collection-task): collecting from producer collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 322a7ed9-abbd-411d-8ee5-1a1fdff896f1 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
2692026-08-26T21:29:52.412ZDEBGoximeter (collection-task): collected results from producer collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 4 producer_id = 322a7ed9-abbd-411d-8ee5-1a1fdff896f1 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
2702026-08-26T21:29:52.712ZDEBGoximeter (collection-task): reporting oximeter self-collection statistics collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 322a7ed9-abbd-411d-8ee5-1a1fdff896f1 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
2712026-08-26T21:29:53.020ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 1132f55a-c26d-4393-a6a9-70855ee9911b producer_ip = fd00:1122:3344:101::1 producer_port = 8001
2722026-08-26T21:29:53.020ZDEBGoximeter (collection-task): collecting from producer collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 1132f55a-c26d-4393-a6a9-70855ee9911b producer_ip = fd00:1122:3344:101::1 producer_port = 8001
2732026-08-26T21:29:53.021ZDEBGoximeter (collection-task): collected results from producer collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 26 producer_id = 1132f55a-c26d-4393-a6a9-70855ee9911b producer_ip = fd00:1122:3344:101::1 producer_port = 8001
2742026-08-26T21:29:53.411ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 322a7ed9-abbd-411d-8ee5-1a1fdff896f1 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
2752026-08-26T21:29:53.411ZDEBGoximeter (collection-task): collecting from producer collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 322a7ed9-abbd-411d-8ee5-1a1fdff896f1 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
2762026-08-26T21:29:53.412ZDEBGoximeter (collection-task): collected results from producer collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 4 producer_id = 322a7ed9-abbd-411d-8ee5-1a1fdff896f1 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
2772026-08-26T21:29:54.019ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 1132f55a-c26d-4393-a6a9-70855ee9911b producer_ip = fd00:1122:3344:101::1 producer_port = 8001
2782026-08-26T21:29:54.019ZDEBGoximeter (collection-task): collecting from producer collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 1132f55a-c26d-4393-a6a9-70855ee9911b producer_ip = fd00:1122:3344:101::1 producer_port = 8001
2792026-08-26T21:29:54.021ZDEBGoximeter (collection-task): collected results from producer collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 26 producer_id = 1132f55a-c26d-4393-a6a9-70855ee9911b producer_ip = fd00:1122:3344:101::1 producer_port = 8001
2802026-08-26T21:29:54.411ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 322a7ed9-abbd-411d-8ee5-1a1fdff896f1 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
2812026-08-26T21:29:54.411ZDEBGoximeter (collection-task): collecting from producer collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 322a7ed9-abbd-411d-8ee5-1a1fdff896f1 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
2822026-08-26T21:29:54.411ZDEBGoximeter (collection-task): collected results from producer collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 4 producer_id = 322a7ed9-abbd-411d-8ee5-1a1fdff896f1 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
2832026-08-26T21:29:54.812ZDEBGoximeter (database-inserter): inserting samples into database collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d n_samples = 152
2842026-08-26T21:29:54.812ZDEBGoximeter (database-inserter): inserting samples into database collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d n_samples = 152
2852026-08-26T21:29:54.812ZDEBGoximeter (clickhouse-client): unrolling 152 total samples collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d id = 686df668-58e8-4ea0-8f4e-b1d302e96d30
2862026-08-26T21:29:54.812ZDEBGoximeter (clickhouse-client): fetching timeseries schema from database collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d id = 686df668-58e8-4ea0-8f4e-b1d302e96d30
2872026-08-26T21:29:54.815ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d id = 686df668-58e8-4ea0-8f4e-b1d302e96d30 schema = TimeseriesSchema { timeseries_name: TimeseriesName("oximeter_collector:collections"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "base_route", field_type: String, source: Metric, description: "" }, FieldSchema { name: "collector_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "collector_ip", field_type: IpAddr, source: Target, description: "" }, FieldSchema { name: "collector_port", field_type: U16, source: Target, description: "" }, FieldSchema { name: "producer_id", field_type: Uuid, source: Metric, description: "" }, FieldSchema { name: "producer_ip", field_type: IpAddr, source: Metric, description: "" }, FieldSchema { name: "producer_port", field_type: U16, source: Metric, description: "" }}, datum_type: CumulativeU64, version: 1, authz_scope: Fleet, units: Count, created: 2026-08-26T21:29:54.812227446Z } timeseries_name = oximeter_collector:collections
2882026-08-26T21:29:54.815ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d id = 686df668-58e8-4ea0-8f4e-b1d302e96d30 schema = TimeseriesSchema { timeseries_name: TimeseriesName("oximeter_collector:samples_collected"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "base_route", field_type: String, source: Metric, description: "" }, FieldSchema { name: "collector_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "collector_ip", field_type: IpAddr, source: Target, description: "" }, FieldSchema { name: "collector_port", field_type: U16, source: Target, description: "" }, FieldSchema { name: "producer_id", field_type: Uuid, source: Metric, description: "" }, FieldSchema { name: "producer_ip", field_type: IpAddr, source: Metric, description: "" }, FieldSchema { name: "producer_port", field_type: U16, source: Metric, description: "" }}, datum_type: CumulativeU64, version: 1, authz_scope: Fleet, units: Count, created: 2026-08-26T21:29:54.812229716Z } timeseries_name = oximeter_collector:samples_collected
2892026-08-26T21:29:54.815ZDEBGoximeter (clickhouse-client): inserting 2 new timeseries schema collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d id = 686df668-58e8-4ea0-8f4e-b1d302e96d30
2902026-08-26T21:29:54.818ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d id = 686df668-58e8-4ea0-8f4e-b1d302e96d30 n_rows = 4 table_name = fields_ipaddr
2912026-08-26T21:29:54.819ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d id = 686df668-58e8-4ea0-8f4e-b1d302e96d30 n_rows = 56 table_name = fields_string
2922026-08-26T21:29:54.820ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d id = 686df668-58e8-4ea0-8f4e-b1d302e96d30 n_rows = 4 table_name = fields_u16
2932026-08-26T21:29:54.822ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d id = 686df668-58e8-4ea0-8f4e-b1d302e96d30 n_rows = 64 table_name = fields_uuid
2942026-08-26T21:29:54.823ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d id = 686df668-58e8-4ea0-8f4e-b1d302e96d30 n_rows = 112 table_name = measurements_cumulativeu64
2952026-08-26T21:29:54.824ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d id = 686df668-58e8-4ea0-8f4e-b1d302e96d30 n_rows = 40 table_name = measurements_u64
2962026-08-26T21:29:54.907ZINFOoximeter (oximeter-agent): refreshing list of producers from Nexus collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d file = oximeter/collector/src/agent.rs:629
2972026-08-26T21:29:54.907ZDEBGoximeter: client request body = None method = GET uri = http://[fd00:1122:3344:101::c]:12221/metrics/collectors/8434ede6-327d-44e3-8722-6c9e65e5b476/producers?sort_by=id_ascending
2982026-08-26T21:29:54.913ZWARNoximeter (database-inserter): failed to insert some results into metric DB collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:276
2992026-08-26T21:29:54.913ZDEBGoximeter: client response result = Ok(Response { url: "http://[fd00:1122:3344:101::c]:12221/metrics/collectors/8434ede6-327d-44e3-8722-6c9e65e5b476/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "6f58d95d-d95d-4427-a6fd-fbc0cc492962", "content-length": "995", "date": "Wed, 26 Aug 2026 21:29:54 GMT"} })
3002026-08-26T21:29:54.913ZDEBGoximeter: client request body = None method = GET uri = http://[fd00:1122:3344:101::c]:12221/metrics/collectors/8434ede6-327d-44e3-8722-6c9e65e5b476/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiI5ZTlmNzEwZS0wMzAxLTQ0MTctYTI4Ni05NmRkZmI0YjlhM2QifX0%3D
3012026-08-26T21:29:54.916ZDEBGoximeter: client response result = Ok(Response { url: "http://[fd00:1122:3344:101::c]:12221/metrics/collectors/8434ede6-327d-44e3-8722-6c9e65e5b476/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiI5ZTlmNzEwZS0wMzAxLTQ0MTctYTI4Ni05NmRkZmI0YjlhM2QifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "906a4ba3-531c-409d-9e35-f4e82813f072", "content-length": "29", "date": "Wed, 26 Aug 2026 21:29:54 GMT"} })
3022026-08-26T21:29:54.916ZDEBGoximeter (oximeter-agent): registered new metric producer address = [fd00:1122:3344:101::2]:8001 collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 25e38706-5d92-4662-82c7-284daee2e3f5
3032026-08-26T21:29:54.916ZDEBGoximeter (oximeter-agent): received request to register existing metric producer, updating collection information address = [fd00:1122:3344:101::2]:35741 collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d interval = 10s producer_id = 42e01252-b1e6-44b5-8e80-fda38177860b
3042026-08-26T21:29:54.916ZDEBGoximeter (collection-task-handle): notified collection task of new producer endpoint collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 42e01252-b1e6-44b5-8e80-fda38177860b producer_ip = fd00:1122:3344:101::2 producer_port = 38589
3052026-08-26T21:29:54.916ZDEBGoximeter (oximeter-agent): registered new metric producer address = [fd00:1122:3344:101::2]:34121 collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 5f75ab69-29fe-41fd-8e57-fddae0ba2803
3062026-08-26T21:29:54.916ZDEBGoximeter (oximeter-agent): registered new metric producer address = [fd00:1122:3344:101::a]:36878 collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 9e9f710e-0301-4417-a286-96ddfb4b9a3d
3072026-08-26T21:29:54.916ZINFOoximeter (oximeter-agent): refreshed list of producers from Nexus collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d file = oximeter/collector/src/agent.rs:704 n_current_tasks = 6 n_pruned_tasks = 0
3082026-08-26T21:29:54.916ZDEBGoximeter (collection-task): collection task received request to update its producer information address = [fd00:1122:3344:101::2]:35741 collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d interval = 10s producer_id = 42e01252-b1e6-44b5-8e80-fda38177860b producer_ip = fd00:1122:3344:101::2 producer_port = 38589
3092026-08-26T21:29:55.019ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 1132f55a-c26d-4393-a6a9-70855ee9911b producer_ip = fd00:1122:3344:101::1 producer_port = 8001
3102026-08-26T21:29:55.019ZDEBGoximeter (collection-task): collecting from producer collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 1132f55a-c26d-4393-a6a9-70855ee9911b producer_ip = fd00:1122:3344:101::1 producer_port = 8001
3112026-08-26T21:29:55.021ZDEBGoximeter (collection-task): collected results from producer collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 26 producer_id = 1132f55a-c26d-4393-a6a9-70855ee9911b producer_ip = fd00:1122:3344:101::1 producer_port = 8001
3122026-08-26T21:29:55.034ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 25e38706-5d92-4662-82c7-284daee2e3f5 producer_ip = fd00:1122:3344:101::2 producer_port = 8001
3132026-08-26T21:29:55.034ZDEBGoximeter (collection-task): collecting from producer collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 25e38706-5d92-4662-82c7-284daee2e3f5 producer_ip = fd00:1122:3344:101::2 producer_port = 8001
3142026-08-26T21:29:55.041ZDEBGoximeter (collection-task): collected results from producer collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 14 producer_id = 25e38706-5d92-4662-82c7-284daee2e3f5 producer_ip = fd00:1122:3344:101::2 producer_port = 8001
3152026-08-26T21:29:55.411ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 322a7ed9-abbd-411d-8ee5-1a1fdff896f1 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
3162026-08-26T21:29:55.411ZDEBGoximeter (collection-task): collecting from producer collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 322a7ed9-abbd-411d-8ee5-1a1fdff896f1 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
3172026-08-26T21:29:55.412ZDEBGoximeter (collection-task): collected results from producer collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 4 producer_id = 322a7ed9-abbd-411d-8ee5-1a1fdff896f1 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
3182026-08-26T21:29:56.019ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 1132f55a-c26d-4393-a6a9-70855ee9911b producer_ip = fd00:1122:3344:101::1 producer_port = 8001
3192026-08-26T21:29:56.019ZDEBGoximeter (collection-task): collecting from producer collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 1132f55a-c26d-4393-a6a9-70855ee9911b producer_ip = fd00:1122:3344:101::1 producer_port = 8001
3202026-08-26T21:29:56.021ZDEBGoximeter (collection-task): collected results from producer collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 26 producer_id = 1132f55a-c26d-4393-a6a9-70855ee9911b producer_ip = fd00:1122:3344:101::1 producer_port = 8001
3212026-08-26T21:29:56.035ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 25e38706-5d92-4662-82c7-284daee2e3f5 producer_ip = fd00:1122:3344:101::2 producer_port = 8001
3222026-08-26T21:29:56.035ZDEBGoximeter (collection-task): collecting from producer collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 25e38706-5d92-4662-82c7-284daee2e3f5 producer_ip = fd00:1122:3344:101::2 producer_port = 8001
3232026-08-26T21:29:56.036ZDEBGoximeter (collection-task): collected results from producer collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 14 producer_id = 25e38706-5d92-4662-82c7-284daee2e3f5 producer_ip = fd00:1122:3344:101::2 producer_port = 8001
3242026-08-26T21:29:56.411ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 322a7ed9-abbd-411d-8ee5-1a1fdff896f1 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
3252026-08-26T21:29:56.411ZDEBGoximeter (collection-task): collecting from producer collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 322a7ed9-abbd-411d-8ee5-1a1fdff896f1 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
3262026-08-26T21:29:56.412ZDEBGoximeter (collection-task): collected results from producer collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 4 producer_id = 322a7ed9-abbd-411d-8ee5-1a1fdff896f1 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
3272026-08-26T21:29:57.019ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 1132f55a-c26d-4393-a6a9-70855ee9911b producer_ip = fd00:1122:3344:101::1 producer_port = 8001
3282026-08-26T21:29:57.019ZDEBGoximeter (collection-task): collecting from producer collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 1132f55a-c26d-4393-a6a9-70855ee9911b producer_ip = fd00:1122:3344:101::1 producer_port = 8001
3292026-08-26T21:29:57.021ZDEBGoximeter (collection-task): collected results from producer collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 26 producer_id = 1132f55a-c26d-4393-a6a9-70855ee9911b producer_ip = fd00:1122:3344:101::1 producer_port = 8001
3302026-08-26T21:29:57.034ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 25e38706-5d92-4662-82c7-284daee2e3f5 producer_ip = fd00:1122:3344:101::2 producer_port = 8001
3312026-08-26T21:29:57.034ZDEBGoximeter (collection-task): collecting from producer collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 25e38706-5d92-4662-82c7-284daee2e3f5 producer_ip = fd00:1122:3344:101::2 producer_port = 8001
3322026-08-26T21:29:57.035ZDEBGoximeter (collection-task): collected results from producer collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 14 producer_id = 25e38706-5d92-4662-82c7-284daee2e3f5 producer_ip = fd00:1122:3344:101::2 producer_port = 8001
3332026-08-26T21:29:57.412ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 322a7ed9-abbd-411d-8ee5-1a1fdff896f1 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
3342026-08-26T21:29:57.412ZDEBGoximeter (collection-task): collecting from producer collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 322a7ed9-abbd-411d-8ee5-1a1fdff896f1 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
3352026-08-26T21:29:57.412ZDEBGoximeter (collection-task): collected results from producer collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 4 producer_id = 322a7ed9-abbd-411d-8ee5-1a1fdff896f1 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
3362026-08-26T21:29:58.019ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 1132f55a-c26d-4393-a6a9-70855ee9911b producer_ip = fd00:1122:3344:101::1 producer_port = 8001
3372026-08-26T21:29:58.019ZDEBGoximeter (collection-task): collecting from producer collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 1132f55a-c26d-4393-a6a9-70855ee9911b producer_ip = fd00:1122:3344:101::1 producer_port = 8001
3382026-08-26T21:29:58.021ZDEBGoximeter (collection-task): collected results from producer collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 26 producer_id = 1132f55a-c26d-4393-a6a9-70855ee9911b producer_ip = fd00:1122:3344:101::1 producer_port = 8001
3392026-08-26T21:29:58.035ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 25e38706-5d92-4662-82c7-284daee2e3f5 producer_ip = fd00:1122:3344:101::2 producer_port = 8001
3402026-08-26T21:29:58.035ZDEBGoximeter (collection-task): collecting from producer collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 25e38706-5d92-4662-82c7-284daee2e3f5 producer_ip = fd00:1122:3344:101::2 producer_port = 8001
3412026-08-26T21:29:58.036ZDEBGoximeter (collection-task): collected results from producer collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 14 producer_id = 25e38706-5d92-4662-82c7-284daee2e3f5 producer_ip = fd00:1122:3344:101::2 producer_port = 8001
3422026-08-26T21:29:58.412ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 322a7ed9-abbd-411d-8ee5-1a1fdff896f1 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
3432026-08-26T21:29:58.412ZDEBGoximeter (collection-task): collecting from producer collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 322a7ed9-abbd-411d-8ee5-1a1fdff896f1 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
3442026-08-26T21:29:58.413ZDEBGoximeter (collection-task): collected results from producer collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 4 producer_id = 322a7ed9-abbd-411d-8ee5-1a1fdff896f1 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
3452026-08-26T21:29:59.020ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 1132f55a-c26d-4393-a6a9-70855ee9911b producer_ip = fd00:1122:3344:101::1 producer_port = 8001
3462026-08-26T21:29:59.020ZDEBGoximeter (collection-task): collecting from producer collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 1132f55a-c26d-4393-a6a9-70855ee9911b producer_ip = fd00:1122:3344:101::1 producer_port = 8001
3472026-08-26T21:29:59.022ZDEBGoximeter (collection-task): collected results from producer collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 26 producer_id = 1132f55a-c26d-4393-a6a9-70855ee9911b producer_ip = fd00:1122:3344:101::1 producer_port = 8001
3482026-08-26T21:29:59.034ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 25e38706-5d92-4662-82c7-284daee2e3f5 producer_ip = fd00:1122:3344:101::2 producer_port = 8001
3492026-08-26T21:29:59.034ZDEBGoximeter (collection-task): collecting from producer collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 25e38706-5d92-4662-82c7-284daee2e3f5 producer_ip = fd00:1122:3344:101::2 producer_port = 8001
3502026-08-26T21:29:59.035ZDEBGoximeter (collection-task): collected results from producer collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 14 producer_id = 25e38706-5d92-4662-82c7-284daee2e3f5 producer_ip = fd00:1122:3344:101::2 producer_port = 8001
3512026-08-26T21:29:59.411ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 322a7ed9-abbd-411d-8ee5-1a1fdff896f1 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
3522026-08-26T21:29:59.411ZDEBGoximeter (collection-task): collecting from producer collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 322a7ed9-abbd-411d-8ee5-1a1fdff896f1 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
3532026-08-26T21:29:59.412ZDEBGoximeter (collection-task): collected results from producer collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 4 producer_id = 322a7ed9-abbd-411d-8ee5-1a1fdff896f1 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
3542026-08-26T21:29:59.682ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 5f75ab69-29fe-41fd-8e57-fddae0ba2803 producer_ip = fd00:1122:3344:101::2 producer_port = 34121
3552026-08-26T21:29:59.682ZDEBGoximeter (collection-task): collecting from producer collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 5f75ab69-29fe-41fd-8e57-fddae0ba2803 producer_ip = fd00:1122:3344:101::2 producer_port = 34121
3562026-08-26T21:29:59.690ZDEBGoximeter (collection-task): collected results from producer collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d n_results = 2 n_samples = 11 producer_id = 5f75ab69-29fe-41fd-8e57-fddae0ba2803 producer_ip = fd00:1122:3344:101::2 producer_port = 34121
3572026-08-26T21:29:59.812ZDEBGoximeter (database-inserter): inserting samples into database collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d n_samples = 231
3582026-08-26T21:29:59.812ZDEBGoximeter (database-inserter): inserting samples into database collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d n_samples = 231
3592026-08-26T21:29:59.812ZDEBGoximeter (clickhouse-client): unrolling 231 total samples collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d id = 686df668-58e8-4ea0-8f4e-b1d302e96d30
3602026-08-26T21:29:59.812ZDEBGoximeter (clickhouse-client): fetching timeseries schema from database collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d id = 686df668-58e8-4ea0-8f4e-b1d302e96d30
3612026-08-26T21:29:59.815ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d id = 686df668-58e8-4ea0-8f4e-b1d302e96d30 schema = TimeseriesSchema { timeseries_name: TimeseriesName("http_service:request_latency_histogram"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "name", field_type: String, source: Target, description: "" }, FieldSchema { name: "operation_id", field_type: String, source: Metric, description: "" }, FieldSchema { name: "status_code", field_type: U16, source: Metric, description: "" }}, datum_type: HistogramU64, version: 1, authz_scope: Fleet, units: Count, created: 2026-08-26T21:29:59.812419574Z } timeseries_name = http_service:request_latency_histogram
3622026-08-26T21:29:59.816ZDEBGoximeter (clickhouse-client): inserting 1 new timeseries schema collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d id = 686df668-58e8-4ea0-8f4e-b1d302e96d30
3632026-08-26T21:29:59.819ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d id = 686df668-58e8-4ea0-8f4e-b1d302e96d30 n_rows = 102 table_name = fields_string
3642026-08-26T21:29:59.820ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d id = 686df668-58e8-4ea0-8f4e-b1d302e96d30 n_rows = 11 table_name = fields_u16
3652026-08-26T21:29:59.821ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d id = 686df668-58e8-4ea0-8f4e-b1d302e96d30 n_rows = 99 table_name = fields_uuid
3662026-08-26T21:29:59.822ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d id = 686df668-58e8-4ea0-8f4e-b1d302e96d30 n_rows = 160 table_name = measurements_cumulativeu64
3672026-08-26T21:29:59.824ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d id = 686df668-58e8-4ea0-8f4e-b1d302e96d30 n_rows = 11 table_name = measurements_histogramu64
3682026-08-26T21:29:59.825ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d id = 686df668-58e8-4ea0-8f4e-b1d302e96d30 n_rows = 60 table_name = measurements_u64
3692026-08-26T21:29:59.913ZWARNoximeter (database-inserter): failed to insert some results into metric DB collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:276
3702026-08-26T21:30:00.020ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 1132f55a-c26d-4393-a6a9-70855ee9911b producer_ip = fd00:1122:3344:101::1 producer_port = 8001
3712026-08-26T21:30:00.020ZDEBGoximeter (collection-task): collecting from producer collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 1132f55a-c26d-4393-a6a9-70855ee9911b producer_ip = fd00:1122:3344:101::1 producer_port = 8001
3722026-08-26T21:30:00.021ZDEBGoximeter (collection-task): collected results from producer collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 26 producer_id = 1132f55a-c26d-4393-a6a9-70855ee9911b producer_ip = fd00:1122:3344:101::1 producer_port = 8001
3732026-08-26T21:30:00.034ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 25e38706-5d92-4662-82c7-284daee2e3f5 producer_ip = fd00:1122:3344:101::2 producer_port = 8001
3742026-08-26T21:30:00.034ZDEBGoximeter (collection-task): collecting from producer collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 25e38706-5d92-4662-82c7-284daee2e3f5 producer_ip = fd00:1122:3344:101::2 producer_port = 8001
3752026-08-26T21:30:00.035ZDEBGoximeter (collection-task): collected results from producer collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 14 producer_id = 25e38706-5d92-4662-82c7-284daee2e3f5 producer_ip = fd00:1122:3344:101::2 producer_port = 8001
3762026-08-26T21:30:00.411ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 322a7ed9-abbd-411d-8ee5-1a1fdff896f1 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
3772026-08-26T21:30:00.411ZDEBGoximeter (collection-task): collecting from producer collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d producer_id = 322a7ed9-abbd-411d-8ee5-1a1fdff896f1 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
3782026-08-26T21:30:00.412ZDEBGoximeter (collection-task): collected results from producer collector_id = 8434ede6-327d-44e3-8722-6c9e65e5b476 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 4 producer_id = 322a7ed9-abbd-411d-8ee5-1a1fdff896f1 producer_ip = fd00:1122:3344:101::2 producer_port = 4677