Max level shown:
1 [ Jul 29 21:28:22 Enabled. ]
2 [ Jul 29 21:28:22 Rereading configuration. ]
3 [ Jul 29 21:28:23 Rereading configuration. ]
4 [ Jul 29 21:28:25 Executing start method ("ctrun -l child -o noorphan,regent /opt/oxide/oximeter-collector/bin/oximeter run /var/svc/manifest/site/oximeter/config.toml --address [fd00:1122:3344:101::d]:12223 --id fa84a12f-09b4-438f-8dee-6b54032162b4 &"). ]
5 [ Jul 29 21:28:25 Method "start" exited with status 0. ]
6 note: configured to log to "/dev/stdout"
72026-07-29T21:28:26.116ZDEBGoximeter: registered DTrace probes
82026-07-29T21:28:26.119ZINFOoximeter: starting oximeter server file = oximeter/collector/src/lib.rs:299
92026-07-29T21:28:26.119ZDEBGoximeter: creating ClickHouse client
102026-07-29T21:28:26.121ZDEBGoximeter (clickhouse-client): registered USDT probes collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21
112026-07-29T21:28:43.125ZWARNoximeter (clickhouse-client): oximeter database does not exist, or is out-of-date collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d file = oximeter/db/src/client/mod.rs:888 id = 23508800-ec0a-4945-b9e3-a3fdb28cba21
122026-07-29T21:28:43.126ZDEBGoximeter (oximeter-agent): oximeter database does not exist, creating collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d
132026-07-29T21:28:43.126ZINFOoximeter (clickhouse-client): reading db version collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d file = oximeter/db/src/client/mod.rs:788 id = 23508800-ec0a-4945-b9e3-a3fdb28cba21
142026-07-29T21:28:43.127ZWARNoximeter (clickhouse-client): oximeter database does not exist, or is out-of-date collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d file = oximeter/db/src/client/mod.rs:888 id = 23508800-ec0a-4945-b9e3-a3fdb28cba21
152026-07-29T21:28:43.127ZINFOoximeter (clickhouse-client): read oximeter database version collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d file = oximeter/db/src/client/mod.rs:793 id = 23508800-ec0a-4945-b9e3-a3fdb28cba21 version = 0
162026-07-29T21:28:43.129ZINFOoximeter (clickhouse-client): initializing oximeter schema collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d file = oximeter/db/src/client/mod.rs:798 id = 23508800-ec0a-4945-b9e3-a3fdb28cba21
172026-07-29T21:28:43.129ZDEBGoximeter (clickhouse-client): initializing ClickHouse database collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21
182026-07-29T21:28:43.320ZINFOoximeter (clickhouse-client): inserting current version collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d file = oximeter/db/src/client/mod.rs:819 id = 23508800-ec0a-4945-b9e3-a3fdb28cba21 version = 14
192026-07-29T21:28:43.347ZDEBGoximeter (clickhouse-client): registered USDT probes collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 942c9d8b-5004-4f34-bd93-98f8d4aedb71
202026-07-29T21:28:43.372ZDEBGoximeter (dropshot): registered endpoint local_addr = [fd00:1122:3344:101::d]:12223 method = GET path = /info versions = all
212026-07-29T21:28:43.372ZDEBGoximeter (dropshot): registered endpoint local_addr = [fd00:1122:3344:101::d]:12223 method = GET path = /producers versions = all
222026-07-29T21:28:43.372ZDEBGoximeter (dropshot): registered endpoint local_addr = [fd00:1122:3344:101::d]:12223 method = DELETE path = /producers/{producer_id} versions = all
232026-07-29T21:28:43.372ZDEBGoximeter (dropshot): registered endpoint local_addr = [fd00:1122:3344:101::d]:12223 method = GET path = /producers/{producer_id} versions = all
242026-07-29T21:28:43.372ZINFOoximeter (dropshot): listening file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:335 local_addr = [fd00:1122:3344:101::d]:12223
252026-07-29T21:28:43.374ZDEBGoximeter (dropshot): successfully registered DTrace USDT probes local_addr = [fd00:1122:3344:101::d]:12223
262026-07-29T21:28:43.377ZDEBGoximeter: registered USDT probes
272026-07-29T21:28:43.377ZDEBGoximeter: contacting nexus
282026-07-29T21:28:43.414ZDEBGoximeter: client request body = Some(Body) method = POST uri = http://[fd00:1122:3344:101::c]:12221/metrics/collectors
292026-07-29T21:28:43.462ZDEBGoximeter: client response result = Ok(Response { url: "http://[fd00:1122:3344:101::c]:12221/metrics/collectors", status: 204, headers: {"x-request-id": "a8845728-82cc-4964-b666-828134c1fbd2", "date": "Wed, 29 Jul 2026 21:28:43 GMT"} })
302026-07-29T21:28:43.462ZINFOoximeter: oximeter registered with nexus file = oximeter/collector/src/lib.rs:431 id = fa84a12f-09b4-438f-8dee-6b54032162b4
312026-07-29T21:28:43.462ZINFOoximeter (oximeter-agent): starting refresh list task collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d file = oximeter/collector/src/agent.rs:626
322026-07-29T21:28:43.463ZINFOoximeter (oximeter-agent): refreshing list of producers from Nexus collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d file = oximeter/collector/src/agent.rs:629
332026-07-29T21:28:43.465ZDEBGoximeter: client request body = None method = GET uri = http://[fd00:1122:3344:101::b]:12221/metrics/collectors/fa84a12f-09b4-438f-8dee-6b54032162b4/producers?sort_by=id_ascending
342026-07-29T21:28:43.472ZDEBGoximeter: client response result = Ok(Response { url: "http://[fd00:1122:3344:101::b]:12221/metrics/collectors/fa84a12f-09b4-438f-8dee-6b54032162b4/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "98087368-8258-40b8-8357-041e9291b56c", "content-length": "29", "date": "Wed, 29 Jul 2026 21:28:43 GMT"} })
352026-07-29T21:28:43.474ZINFOoximeter (oximeter-agent): refreshed list of producers from Nexus collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d file = oximeter/collector/src/agent.rs:704 n_current_tasks = 0 n_pruned_tasks = 0
362026-07-29T21:28:48.350ZDEBGoximeter (database-inserter): inserting samples into database collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d n_samples = 4
372026-07-29T21:28:48.350ZDEBGoximeter (database-inserter): inserting samples into database collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d n_samples = 4
382026-07-29T21:28:48.350ZDEBGoximeter (clickhouse-client): unrolling 4 total samples collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21
392026-07-29T21:28:48.354ZDEBGoximeter (clickhouse-client): fetching timeseries schema from database collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21
402026-07-29T21:28:48.356ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21 schema = TimeseriesSchema { timeseries_name: TimeseriesName("oximeter_collector:database_queue_depth"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "collector_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "collector_ip", field_type: IpAddr, source: Target, description: "" }, FieldSchema { name: "collector_port", field_type: U16, source: Target, description: "" }, FieldSchema { name: "collector_sink", field_type: String, source: Metric, description: "" }}, datum_type: HistogramU64, version: 1, authz_scope: Fleet, units: Count, created: 2026-07-29T21:28:48.352460958Z } timeseries_name = oximeter_collector:database_queue_depth
412026-07-29T21:28:48.356ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21 schema = TimeseriesSchema { timeseries_name: TimeseriesName("oximeter_collector:database_samples_dropped"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "collector_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "collector_ip", field_type: IpAddr, source: Target, description: "" }, FieldSchema { name: "collector_port", field_type: U16, source: Target, description: "" }, FieldSchema { name: "collector_sink", field_type: String, source: Metric, description: "" }}, datum_type: CumulativeU64, version: 1, authz_scope: Fleet, units: Count, created: 2026-07-29T21:28:48.352457048Z } timeseries_name = oximeter_collector:database_samples_dropped
422026-07-29T21:28:48.356ZDEBGoximeter (clickhouse-client): inserting 2 new timeseries schema collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21
432026-07-29T21:28:48.360ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21 n_rows = 4 table_name = fields_ipaddr
442026-07-29T21:28:48.362ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21 n_rows = 4 table_name = fields_string
452026-07-29T21:28:48.364ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21 n_rows = 4 table_name = fields_u16
462026-07-29T21:28:48.366ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21 n_rows = 4 table_name = fields_uuid
472026-07-29T21:28:48.368ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21 n_rows = 2 table_name = measurements_cumulativeu64
482026-07-29T21:28:48.370ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21 n_rows = 2 table_name = measurements_histogramu64
492026-07-29T21:28:48.452ZWARNoximeter (database-inserter): failed to insert some results into metric DB collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:276
502026-07-29T21:28:53.350ZDEBGoximeter (database-inserter): batch interval expired, but no samples to insert collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d
512026-07-29T21:28:53.350ZDEBGoximeter (database-inserter): batch interval expired, but no samples to insert collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d
522026-07-29T21:28:58.350ZDEBGoximeter (database-inserter): batch interval expired, but no samples to insert collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d
532026-07-29T21:28:58.350ZDEBGoximeter (database-inserter): batch interval expired, but no samples to insert collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d
542026-07-29T21:28:58.463ZINFOoximeter (oximeter-agent): refreshing list of producers from Nexus collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d file = oximeter/collector/src/agent.rs:629
552026-07-29T21:28:58.463ZDEBGoximeter: client request body = None method = GET uri = http://[fd00:1122:3344:101::a]:12221/metrics/collectors/fa84a12f-09b4-438f-8dee-6b54032162b4/producers?sort_by=id_ascending
562026-07-29T21:28:58.477ZDEBGoximeter: client response result = Ok(Response { url: "http://[fd00:1122:3344:101::a]:12221/metrics/collectors/fa84a12f-09b4-438f-8dee-6b54032162b4/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "fd096ce2-ae26-4cd5-867b-01982b7b903c", "content-length": "300", "date": "Wed, 29 Jul 2026 21:28:58 GMT"} })
572026-07-29T21:28:58.477ZDEBGoximeter: client request body = None method = GET uri = http://[fd00:1122:3344:101::a]:12221/metrics/collectors/fa84a12f-09b4-438f-8dee-6b54032162b4/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiIwNDgwODA4MC03ZTgxLTRhZDUtOTlkZC0xNmRmOWFiMGI4NDkifX0%3D
582026-07-29T21:28:58.484ZDEBGoximeter: client response result = Ok(Response { url: "http://[fd00:1122:3344:101::a]:12221/metrics/collectors/fa84a12f-09b4-438f-8dee-6b54032162b4/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiIwNDgwODA4MC03ZTgxLTRhZDUtOTlkZC0xNmRmOWFiMGI4NDkifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "dd34e26d-c0c1-46fd-9453-c880356cd70e", "content-length": "29", "date": "Wed, 29 Jul 2026 21:28:58 GMT"} })
592026-07-29T21:28:58.484ZDEBGoximeter (oximeter-agent): registered new metric producer address = [fd00:1122:3344:101::2]:59901 collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = 04808080-7e81-4ad5-99dd-16df9ab0b849
602026-07-29T21:28:58.484ZINFOoximeter (oximeter-agent): refreshed list of producers from Nexus collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d file = oximeter/collector/src/agent.rs:704 n_current_tasks = 1 n_pruned_tasks = 0
612026-07-29T21:29:00.133ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = 04808080-7e81-4ad5-99dd-16df9ab0b849 producer_ip = fd00:1122:3344:101::2 producer_port = 59901
622026-07-29T21:29:00.134ZDEBGoximeter (collection-task): collecting from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = 04808080-7e81-4ad5-99dd-16df9ab0b849 producer_ip = fd00:1122:3344:101::2 producer_port = 59901
632026-07-29T21:29:00.163ZDEBGoximeter (collection-task): collected results from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 140 producer_id = 04808080-7e81-4ad5-99dd-16df9ab0b849 producer_ip = fd00:1122:3344:101::2 producer_port = 59901
642026-07-29T21:29:03.350ZDEBGoximeter (database-inserter): inserting samples into database collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d n_samples = 140
652026-07-29T21:29:03.351ZDEBGoximeter (database-inserter): inserting samples into database collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d n_samples = 140
662026-07-29T21:29:03.351ZDEBGoximeter (clickhouse-client): unrolling 140 total samples collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21
672026-07-29T21:29:03.351ZDEBGoximeter (clickhouse-client): fetching timeseries schema from database collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21
682026-07-29T21:29:03.352ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21 schema = TimeseriesSchema { timeseries_name: TimeseriesName("dendrite:sample_collection_duration"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "asic_fab", field_type: String, source: Metric, description: "" }, FieldSchema { name: "asic_lot", field_type: String, source: Metric, description: "" }, FieldSchema { name: "asic_wafer", field_type: U8, source: Metric, description: "" }, FieldSchema { name: "asic_wafer_loc_x", field_type: I16, source: Metric, description: "" }, FieldSchema { name: "asic_wafer_loc_y", field_type: I16, source: Metric, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_model", field_type: String, source: Target, description: "" }, FieldSchema { name: "sled_revision", field_type: U32, source: Target, description: "" }, FieldSchema { name: "sled_serial", field_type: String, source: Target, description: "" }, FieldSchema { name: "switch_id", field_type: Uuid, source: Metric, description: "" }, FieldSchema { name: "switch_model", field_type: String, source: Metric, description: "" }, FieldSchema { name: "switch_revision", field_type: U32, source: Metric, description: "" }, FieldSchema { name: "switch_serial", field_type: String, source: Metric, description: "" }, FieldSchema { name: "switch_slot", field_type: U16, source: Metric, description: "" }}, datum_type: F64, version: 1, authz_scope: Fleet, units: Count, created: 2026-07-29T21:29:03.351042889Z } timeseries_name = dendrite:sample_collection_duration
692026-07-29T21:29:03.352ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21 schema = TimeseriesSchema { timeseries_name: TimeseriesName("switch_data_link:link_enabled"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "asic_fab", field_type: String, source: Target, description: "" }, FieldSchema { name: "asic_lot", field_type: String, source: Target, description: "" }, FieldSchema { name: "asic_wafer", field_type: U8, source: Target, description: "" }, FieldSchema { name: "asic_wafer_loc_x", field_type: I16, source: Target, description: "" }, FieldSchema { name: "asic_wafer_loc_y", field_type: I16, source: Target, description: "" }, FieldSchema { name: "kind", field_type: String, source: Target, description: "" }, FieldSchema { name: "link_id", field_type: U8, source: Metric, description: "" }, FieldSchema { name: "model", field_type: String, source: Target, description: "" }, FieldSchema { name: "network", field_type: String, source: Target, description: "" }, FieldSchema { name: "port_id", field_type: String, source: Metric, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_model", field_type: String, source: Target, description: "" }, FieldSchema { name: "sled_revision", field_type: U32, source: Target, description: "" }, FieldSchema { name: "sled_serial", field_type: String, source: Target, description: "" }, FieldSchema { name: "switch_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "switch_model", field_type: String, source: Target, description: "" }, FieldSchema { name: "switch_revision", field_type: U32, source: Target, description: "" }, FieldSchema { name: "switch_serial", field_type: String, source: Target, description: "" }, FieldSchema { name: "switch_slot", field_type: U16, source: Target, description: "" }}, datum_type: Bool, version: 1, authz_scope: Fleet, units: Count, created: 2026-07-29T21:29:03.350963718Z } timeseries_name = switch_data_link:link_enabled
702026-07-29T21:29:03.352ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21 schema = TimeseriesSchema { timeseries_name: TimeseriesName("switch_data_link:link_up"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "asic_fab", field_type: String, source: Target, description: "" }, FieldSchema { name: "asic_lot", field_type: String, source: Target, description: "" }, FieldSchema { name: "asic_wafer", field_type: U8, source: Target, description: "" }, FieldSchema { name: "asic_wafer_loc_x", field_type: I16, source: Target, description: "" }, FieldSchema { name: "asic_wafer_loc_y", field_type: I16, source: Target, description: "" }, FieldSchema { name: "kind", field_type: String, source: Target, description: "" }, FieldSchema { name: "link_id", field_type: U8, source: Metric, description: "" }, FieldSchema { name: "model", field_type: String, source: Target, description: "" }, FieldSchema { name: "network", field_type: String, source: Target, description: "" }, FieldSchema { name: "port_id", field_type: String, source: Metric, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_model", field_type: String, source: Target, description: "" }, FieldSchema { name: "sled_revision", field_type: U32, source: Target, description: "" }, FieldSchema { name: "sled_serial", field_type: String, source: Target, description: "" }, FieldSchema { name: "switch_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "switch_model", field_type: String, source: Target, description: "" }, FieldSchema { name: "switch_revision", field_type: U32, source: Target, description: "" }, FieldSchema { name: "switch_serial", field_type: String, source: Target, description: "" }, FieldSchema { name: "switch_slot", field_type: U16, source: Target, description: "" }}, datum_type: Bool, version: 1, authz_scope: Fleet, units: Count, created: 2026-07-29T21:29:03.350970888Z } timeseries_name = switch_data_link:link_up
712026-07-29T21:29:03.353ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21 schema = TimeseriesSchema { timeseries_name: TimeseriesName("switch_table:capacity"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "asic_fab", field_type: String, source: Target, description: "" }, FieldSchema { name: "asic_lot", field_type: String, source: Target, description: "" }, FieldSchema { name: "asic_wafer", field_type: U8, source: Target, description: "" }, FieldSchema { name: "asic_wafer_loc_x", field_type: I16, source: Target, description: "" }, FieldSchema { name: "asic_wafer_loc_y", field_type: I16, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "switch_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "switch_model", field_type: String, source: Target, description: "" }, FieldSchema { name: "switch_revision", field_type: U32, source: Target, description: "" }, FieldSchema { name: "switch_serial", field_type: String, source: Target, description: "" }, FieldSchema { name: "switch_slot", field_type: U16, source: Target, description: "" }, FieldSchema { name: "table", field_type: String, source: Target, description: "" }}, datum_type: U64, version: 1, authz_scope: Fleet, units: Count, created: 2026-07-29T21:29:03.350977948Z } timeseries_name = switch_table:capacity
722026-07-29T21:29:03.353ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21 schema = TimeseriesSchema { timeseries_name: TimeseriesName("switch_table:collisions"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "asic_fab", field_type: String, source: Target, description: "" }, FieldSchema { name: "asic_lot", field_type: String, source: Target, description: "" }, FieldSchema { name: "asic_wafer", field_type: U8, source: Target, description: "" }, FieldSchema { name: "asic_wafer_loc_x", field_type: I16, source: Target, description: "" }, FieldSchema { name: "asic_wafer_loc_y", field_type: I16, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "switch_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "switch_model", field_type: String, source: Target, description: "" }, FieldSchema { name: "switch_revision", field_type: U32, source: Target, description: "" }, FieldSchema { name: "switch_serial", field_type: String, source: Target, description: "" }, FieldSchema { name: "switch_slot", field_type: U16, source: Target, description: "" }, FieldSchema { name: "table", field_type: String, source: Target, description: "" }}, datum_type: CumulativeU64, version: 1, authz_scope: Fleet, units: Count, created: 2026-07-29T21:29:03.351005838Z } timeseries_name = switch_table:collisions
732026-07-29T21:29:03.353ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21 schema = TimeseriesSchema { timeseries_name: TimeseriesName("switch_table:delete_misses"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "asic_fab", field_type: String, source: Target, description: "" }, FieldSchema { name: "asic_lot", field_type: String, source: Target, description: "" }, FieldSchema { name: "asic_wafer", field_type: U8, source: Target, description: "" }, FieldSchema { name: "asic_wafer_loc_x", field_type: I16, source: Target, description: "" }, FieldSchema { name: "asic_wafer_loc_y", field_type: I16, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "switch_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "switch_model", field_type: String, source: Target, description: "" }, FieldSchema { name: "switch_revision", field_type: U32, source: Target, description: "" }, FieldSchema { name: "switch_serial", field_type: String, source: Target, description: "" }, FieldSchema { name: "switch_slot", field_type: U16, source: Target, description: "" }, FieldSchema { name: "table", field_type: String, source: Target, description: "" }}, datum_type: CumulativeU64, version: 1, authz_scope: Fleet, units: Count, created: 2026-07-29T21:29:03.351013918Z } timeseries_name = switch_table:delete_misses
742026-07-29T21:29:03.353ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21 schema = TimeseriesSchema { timeseries_name: TimeseriesName("switch_table:deletes"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "asic_fab", field_type: String, source: Target, description: "" }, FieldSchema { name: "asic_lot", field_type: String, source: Target, description: "" }, FieldSchema { name: "asic_wafer", field_type: U8, source: Target, description: "" }, FieldSchema { name: "asic_wafer_loc_x", field_type: I16, source: Target, description: "" }, FieldSchema { name: "asic_wafer_loc_y", field_type: I16, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "switch_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "switch_model", field_type: String, source: Target, description: "" }, FieldSchema { name: "switch_revision", field_type: U32, source: Target, description: "" }, FieldSchema { name: "switch_serial", field_type: String, source: Target, description: "" }, FieldSchema { name: "switch_slot", field_type: U16, source: Target, description: "" }, FieldSchema { name: "table", field_type: String, source: Target, description: "" }}, datum_type: CumulativeU64, version: 1, authz_scope: Fleet, units: Count, created: 2026-07-29T21:29:03.350988498Z } timeseries_name = switch_table:deletes
752026-07-29T21:29:03.353ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21 schema = TimeseriesSchema { timeseries_name: TimeseriesName("switch_table:exhaustion"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "asic_fab", field_type: String, source: Target, description: "" }, FieldSchema { name: "asic_lot", field_type: String, source: Target, description: "" }, FieldSchema { name: "asic_wafer", field_type: U8, source: Target, description: "" }, FieldSchema { name: "asic_wafer_loc_x", field_type: I16, source: Target, description: "" }, FieldSchema { name: "asic_wafer_loc_y", field_type: I16, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "switch_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "switch_model", field_type: String, source: Target, description: "" }, FieldSchema { name: "switch_revision", field_type: U32, source: Target, description: "" }, FieldSchema { name: "switch_serial", field_type: String, source: Target, description: "" }, FieldSchema { name: "switch_slot", field_type: U16, source: Target, description: "" }, FieldSchema { name: "table", field_type: String, source: Target, description: "" }}, datum_type: CumulativeU64, version: 1, authz_scope: Fleet, units: Count, created: 2026-07-29T21:29:03.351019138Z } timeseries_name = switch_table:exhaustion
762026-07-29T21:29:03.353ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21 schema = TimeseriesSchema { timeseries_name: TimeseriesName("switch_table:inserts"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "asic_fab", field_type: String, source: Target, description: "" }, FieldSchema { name: "asic_lot", field_type: String, source: Target, description: "" }, FieldSchema { name: "asic_wafer", field_type: U8, source: Target, description: "" }, FieldSchema { name: "asic_wafer_loc_x", field_type: I16, source: Target, description: "" }, FieldSchema { name: "asic_wafer_loc_y", field_type: I16, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "switch_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "switch_model", field_type: String, source: Target, description: "" }, FieldSchema { name: "switch_revision", field_type: U32, source: Target, description: "" }, FieldSchema { name: "switch_serial", field_type: String, source: Target, description: "" }, FieldSchema { name: "switch_slot", field_type: U16, source: Target, description: "" }, FieldSchema { name: "table", field_type: String, source: Target, description: "" }}, datum_type: CumulativeU64, version: 1, authz_scope: Fleet, units: Count, created: 2026-07-29T21:29:03.350985218Z } timeseries_name = switch_table:inserts
772026-07-29T21:29:03.353ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21 schema = TimeseriesSchema { timeseries_name: TimeseriesName("switch_table:occupancy"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "asic_fab", field_type: String, source: Target, description: "" }, FieldSchema { name: "asic_lot", field_type: String, source: Target, description: "" }, FieldSchema { name: "asic_wafer", field_type: U8, source: Target, description: "" }, FieldSchema { name: "asic_wafer_loc_x", field_type: I16, source: Target, description: "" }, FieldSchema { name: "asic_wafer_loc_y", field_type: I16, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "switch_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "switch_model", field_type: String, source: Target, description: "" }, FieldSchema { name: "switch_revision", field_type: U32, source: Target, description: "" }, FieldSchema { name: "switch_serial", field_type: String, source: Target, description: "" }, FieldSchema { name: "switch_slot", field_type: U16, source: Target, description: "" }, FieldSchema { name: "table", field_type: String, source: Target, description: "" }}, datum_type: U64, version: 1, authz_scope: Fleet, units: Count, created: 2026-07-29T21:29:03.350981718Z } timeseries_name = switch_table:occupancy
782026-07-29T21:29:03.353ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21 schema = TimeseriesSchema { timeseries_name: TimeseriesName("switch_table:update_misses"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "asic_fab", field_type: String, source: Target, description: "" }, FieldSchema { name: "asic_lot", field_type: String, source: Target, description: "" }, FieldSchema { name: "asic_wafer", field_type: U8, source: Target, description: "" }, FieldSchema { name: "asic_wafer_loc_x", field_type: I16, source: Target, description: "" }, FieldSchema { name: "asic_wafer_loc_y", field_type: I16, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "switch_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "switch_model", field_type: String, source: Target, description: "" }, FieldSchema { name: "switch_revision", field_type: U32, source: Target, description: "" }, FieldSchema { name: "switch_serial", field_type: String, source: Target, description: "" }, FieldSchema { name: "switch_slot", field_type: U16, source: Target, description: "" }, FieldSchema { name: "table", field_type: String, source: Target, description: "" }}, datum_type: CumulativeU64, version: 1, authz_scope: Fleet, units: Count, created: 2026-07-29T21:29:03.351010918Z } timeseries_name = switch_table:update_misses
792026-07-29T21:29:03.353ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21 schema = TimeseriesSchema { timeseries_name: TimeseriesName("switch_table:updates"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "asic_fab", field_type: String, source: Target, description: "" }, FieldSchema { name: "asic_lot", field_type: String, source: Target, description: "" }, FieldSchema { name: "asic_wafer", field_type: U8, source: Target, description: "" }, FieldSchema { name: "asic_wafer_loc_x", field_type: I16, source: Target, description: "" }, FieldSchema { name: "asic_wafer_loc_y", field_type: I16, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "switch_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "switch_model", field_type: String, source: Target, description: "" }, FieldSchema { name: "switch_revision", field_type: U32, source: Target, description: "" }, FieldSchema { name: "switch_serial", field_type: String, source: Target, description: "" }, FieldSchema { name: "switch_slot", field_type: U16, source: Target, description: "" }, FieldSchema { name: "table", field_type: String, source: Target, description: "" }}, datum_type: CumulativeU64, version: 1, authz_scope: Fleet, units: Count, created: 2026-07-29T21:29:03.351001548Z } timeseries_name = switch_table:updates
802026-07-29T21:29:03.355ZDEBGoximeter (clickhouse-client): inserting 12 new timeseries schema collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21
812026-07-29T21:29:03.360ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21 n_rows = 280 table_name = fields_i16
822026-07-29T21:29:03.362ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21 n_rows = 721 table_name = fields_string
832026-07-29T21:29:03.363ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21 n_rows = 140 table_name = fields_u16
842026-07-29T21:29:03.365ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21 n_rows = 145 table_name = fields_u32
852026-07-29T21:29:03.366ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21 n_rows = 144 table_name = fields_u8
862026-07-29T21:29:03.368ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21 n_rows = 420 table_name = fields_uuid
872026-07-29T21:29:03.369ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21 n_rows = 4 table_name = measurements_bool
882026-07-29T21:29:03.371ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21 n_rows = 105 table_name = measurements_cumulativeu64
892026-07-29T21:29:03.373ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21 n_rows = 1 table_name = measurements_f64
902026-07-29T21:29:03.374ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21 n_rows = 30 table_name = measurements_u64
912026-07-29T21:29:03.452ZWARNoximeter (database-inserter): failed to insert some results into metric DB collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:276
922026-07-29T21:29:08.351ZDEBGoximeter (database-inserter): batch interval expired, but no samples to insert collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d
932026-07-29T21:29:08.351ZDEBGoximeter (database-inserter): batch interval expired, but no samples to insert collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d
942026-07-29T21:29:10.134ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = 04808080-7e81-4ad5-99dd-16df9ab0b849 producer_ip = fd00:1122:3344:101::2 producer_port = 59901
952026-07-29T21:29:10.134ZDEBGoximeter (collection-task): collecting from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = 04808080-7e81-4ad5-99dd-16df9ab0b849 producer_ip = fd00:1122:3344:101::2 producer_port = 59901
962026-07-29T21:29:10.142ZDEBGoximeter (collection-task): collected results from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 140 producer_id = 04808080-7e81-4ad5-99dd-16df9ab0b849 producer_ip = fd00:1122:3344:101::2 producer_port = 59901
972026-07-29T21:29:13.350ZDEBGoximeter (database-inserter): inserting samples into database collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d n_samples = 140
982026-07-29T21:29:13.350ZDEBGoximeter (database-inserter): inserting samples into database collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d n_samples = 140
992026-07-29T21:29:13.350ZDEBGoximeter (clickhouse-client): unrolling 140 total samples collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21
1002026-07-29T21:29:13.350ZDEBGoximeter (clickhouse-client): fetching timeseries schema from database collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21
1012026-07-29T21:29:13.362ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21 n_rows = 280 table_name = fields_i16
1022026-07-29T21:29:13.364ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21 n_rows = 721 table_name = fields_string
1032026-07-29T21:29:13.365ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21 n_rows = 140 table_name = fields_u16
1042026-07-29T21:29:13.367ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21 n_rows = 145 table_name = fields_u32
1052026-07-29T21:29:13.368ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21 n_rows = 144 table_name = fields_u8
1062026-07-29T21:29:13.370ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21 n_rows = 420 table_name = fields_uuid
1072026-07-29T21:29:13.372ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21 n_rows = 4 table_name = measurements_bool
1082026-07-29T21:29:13.373ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21 n_rows = 105 table_name = measurements_cumulativeu64
1092026-07-29T21:29:13.375ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21 n_rows = 1 table_name = measurements_f64
1102026-07-29T21:29:13.376ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21 n_rows = 30 table_name = measurements_u64
1112026-07-29T21:29:13.451ZWARNoximeter (database-inserter): failed to insert some results into metric DB collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:276
1122026-07-29T21:29:13.463ZINFOoximeter (oximeter-agent): refreshing list of producers from Nexus collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d file = oximeter/collector/src/agent.rs:629
1132026-07-29T21:29:13.464ZDEBGoximeter: client request body = None method = GET uri = http://[fd00:1122:3344:101::b]:12221/metrics/collectors/fa84a12f-09b4-438f-8dee-6b54032162b4/producers?sort_by=id_ascending
1142026-07-29T21:29:13.478ZDEBGoximeter: client response result = Ok(Response { url: "http://[fd00:1122:3344:101::b]:12221/metrics/collectors/fa84a12f-09b4-438f-8dee-6b54032162b4/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "0f21f7d0-268d-4d5a-be1a-884fe82eea13", "content-length": "585", "date": "Wed, 29 Jul 2026 21:29:13 GMT"} })
1152026-07-29T21:29:13.478ZDEBGoximeter: client request body = None method = GET uri = http://[fd00:1122:3344:101::b]:12221/metrics/collectors/fa84a12f-09b4-438f-8dee-6b54032162b4/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNTNhZjRiZS00YzhmLTRkMDAtYTAzNS1jNTE0YjQ1N2M5Y2MifX0%3D
1162026-07-29T21:29:13.481ZDEBGoximeter: client response result = Ok(Response { url: "http://[fd00:1122:3344:101::b]:12221/metrics/collectors/fa84a12f-09b4-438f-8dee-6b54032162b4/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNTNhZjRiZS00YzhmLTRkMDAtYTAzNS1jNTE0YjQ1N2M5Y2MifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "de5e179f-9c4d-47b3-ab6c-112cdb907169", "content-length": "29", "date": "Wed, 29 Jul 2026 21:29:13 GMT"} })
1172026-07-29T21:29:13.481ZDEBGoximeter (oximeter-agent): received request to register existing metric producer, updating collection information address = [fd00:1122:3344:101::2]:63088 collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d interval = 30s producer_id = 04808080-7e81-4ad5-99dd-16df9ab0b849
1182026-07-29T21:29:13.481ZDEBGoximeter (collection-task-handle): notified collection task of new producer endpoint collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = 04808080-7e81-4ad5-99dd-16df9ab0b849 producer_ip = fd00:1122:3344:101::2 producer_port = 59901
1192026-07-29T21:29:13.481ZDEBGoximeter (oximeter-agent): registered new metric producer address = [fd00:1122:3344:101::2]:37767 collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = 6f2c93a7-3ce8-4cc1-8671-369f638cb080
1202026-07-29T21:29:13.481ZDEBGoximeter (collection-task): collection task received request to update its producer information address = [fd00:1122:3344:101::2]:63088 collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d interval = 30s producer_id = 04808080-7e81-4ad5-99dd-16df9ab0b849 producer_ip = fd00:1122:3344:101::2 producer_port = 59901
1212026-07-29T21:29:13.482ZDEBGoximeter (oximeter-agent): registered new metric producer address = [fd00:1122:3344:101::2]:8001 collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc
1222026-07-29T21:29:13.482ZINFOoximeter (oximeter-agent): refreshed list of producers from Nexus collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d file = oximeter/collector/src/agent.rs:704 n_current_tasks = 3 n_pruned_tasks = 0
1232026-07-29T21:29:14.392ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
1242026-07-29T21:29:14.393ZDEBGoximeter (collection-task): collecting from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
1252026-07-29T21:29:14.401ZDEBGoximeter (collection-task): collected results from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 14 producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
1262026-07-29T21:29:15.392ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
1272026-07-29T21:29:15.392ZDEBGoximeter (collection-task): collecting from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
1282026-07-29T21:29:15.393ZDEBGoximeter (collection-task): collected results from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 14 producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
1292026-07-29T21:29:16.392ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
1302026-07-29T21:29:16.392ZDEBGoximeter (collection-task): collecting from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
1312026-07-29T21:29:16.393ZDEBGoximeter (collection-task): collected results from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 14 producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
1322026-07-29T21:29:17.393ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
1332026-07-29T21:29:17.393ZDEBGoximeter (collection-task): collecting from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
1342026-07-29T21:29:17.394ZDEBGoximeter (collection-task): collected results from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 14 producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
1352026-07-29T21:29:18.350ZDEBGoximeter (database-inserter): inserting samples into database collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d n_samples = 56
1362026-07-29T21:29:18.351ZDEBGoximeter (database-inserter): inserting samples into database collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d n_samples = 56
1372026-07-29T21:29:18.351ZDEBGoximeter (clickhouse-client): unrolling 56 total samples collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21
1382026-07-29T21:29:18.351ZDEBGoximeter (clickhouse-client): fetching timeseries schema from database collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21
1392026-07-29T21:29:18.352ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21 schema = TimeseriesSchema { timeseries_name: TimeseriesName("ddm_router:originated_tunnel_endpoints"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "hostname", field_type: String, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }}, datum_type: U64, version: 1, authz_scope: Fleet, units: Count, created: 2026-07-29T21:29:18.351044186Z } timeseries_name = ddm_router:originated_tunnel_endpoints
1402026-07-29T21:29:18.352ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21 schema = TimeseriesSchema { timeseries_name: TimeseriesName("ddm_router:originated_underlay_prefixes"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "hostname", field_type: String, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }}, datum_type: U64, version: 1, authz_scope: Fleet, units: Count, created: 2026-07-29T21:29:18.351041116Z } timeseries_name = ddm_router:originated_underlay_prefixes
1412026-07-29T21:29:18.352ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21 schema = TimeseriesSchema { timeseries_name: TimeseriesName("ddm_session:advertisements_received"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "hostname", field_type: String, source: Target, description: "" }, FieldSchema { name: "interface", field_type: String, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }}, datum_type: CumulativeU64, version: 1, authz_scope: Fleet, units: Count, created: 2026-07-29T21:29:18.351051426Z } timeseries_name = ddm_session:advertisements_received
1422026-07-29T21:29:18.352ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21 schema = TimeseriesSchema { timeseries_name: TimeseriesName("ddm_session:advertisements_sent"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "hostname", field_type: String, source: Target, description: "" }, FieldSchema { name: "interface", field_type: String, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }}, datum_type: CumulativeU64, version: 1, authz_scope: Fleet, units: Count, created: 2026-07-29T21:29:18.351049816Z } timeseries_name = ddm_session:advertisements_sent
1432026-07-29T21:29:18.352ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21 schema = TimeseriesSchema { timeseries_name: TimeseriesName("ddm_session:imported_tunnel_endpoints"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "hostname", field_type: String, source: Target, description: "" }, FieldSchema { name: "interface", field_type: String, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }}, datum_type: U64, version: 1, authz_scope: Fleet, units: Count, created: 2026-07-29T21:29:18.351065936Z } timeseries_name = ddm_session:imported_tunnel_endpoints
1442026-07-29T21:29:18.352ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21 schema = TimeseriesSchema { timeseries_name: TimeseriesName("ddm_session:imported_underlay_prefixes"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "hostname", field_type: String, source: Target, description: "" }, FieldSchema { name: "interface", field_type: String, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }}, datum_type: U64, version: 1, authz_scope: Fleet, units: Count, created: 2026-07-29T21:29:18.351064296Z } timeseries_name = ddm_session:imported_underlay_prefixes
1452026-07-29T21:29:18.352ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21 schema = TimeseriesSchema { timeseries_name: TimeseriesName("ddm_session:peer_address_changes"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "hostname", field_type: String, source: Target, description: "" }, FieldSchema { name: "interface", field_type: String, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }}, datum_type: CumulativeU64, version: 1, authz_scope: Fleet, units: Count, created: 2026-07-29T21:29:18.351055026Z } timeseries_name = ddm_session:peer_address_changes
1462026-07-29T21:29:18.352ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21 schema = TimeseriesSchema { timeseries_name: TimeseriesName("ddm_session:peer_expirations"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "hostname", field_type: String, source: Target, description: "" }, FieldSchema { name: "interface", field_type: String, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }}, datum_type: CumulativeU64, version: 1, authz_scope: Fleet, units: Count, created: 2026-07-29T21:29:18.351053116Z } timeseries_name = ddm_session:peer_expirations
1472026-07-29T21:29:18.352ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21 schema = TimeseriesSchema { timeseries_name: TimeseriesName("ddm_session:peer_sessions_established"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "hostname", field_type: String, source: Target, description: "" }, FieldSchema { name: "interface", field_type: String, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }}, datum_type: CumulativeU64, version: 1, authz_scope: Fleet, units: Count, created: 2026-07-29T21:29:18.351056646Z } timeseries_name = ddm_session:peer_sessions_established
1482026-07-29T21:29:18.352ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21 schema = TimeseriesSchema { timeseries_name: TimeseriesName("ddm_session:solicitations_received"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "hostname", field_type: String, source: Target, description: "" }, FieldSchema { name: "interface", field_type: String, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }}, datum_type: CumulativeU64, version: 1, authz_scope: Fleet, units: Count, created: 2026-07-29T21:29:18.351047966Z } timeseries_name = ddm_session:solicitations_received
1492026-07-29T21:29:18.352ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21 schema = TimeseriesSchema { timeseries_name: TimeseriesName("ddm_session:solicitations_sent"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "hostname", field_type: String, source: Target, description: "" }, FieldSchema { name: "interface", field_type: String, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }}, datum_type: CumulativeU64, version: 1, authz_scope: Fleet, units: Count, created: 2026-07-29T21:29:18.351046466Z } timeseries_name = ddm_session:solicitations_sent
1502026-07-29T21:29:18.352ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21 schema = TimeseriesSchema { timeseries_name: TimeseriesName("ddm_session:update_send_fail"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "hostname", field_type: String, source: Target, description: "" }, FieldSchema { name: "interface", field_type: String, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }}, datum_type: CumulativeU64, version: 1, authz_scope: Fleet, units: Count, created: 2026-07-29T21:29:18.351061626Z } timeseries_name = ddm_session:update_send_fail
1512026-07-29T21:29:18.352ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21 schema = TimeseriesSchema { timeseries_name: TimeseriesName("ddm_session:updates_received"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "hostname", field_type: String, source: Target, description: "" }, FieldSchema { name: "interface", field_type: String, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }}, datum_type: CumulativeU64, version: 1, authz_scope: Fleet, units: Count, created: 2026-07-29T21:29:18.351060066Z } timeseries_name = ddm_session:updates_received
1522026-07-29T21:29:18.352ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21 schema = TimeseriesSchema { timeseries_name: TimeseriesName("ddm_session:updates_sent"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "hostname", field_type: String, source: Target, description: "" }, FieldSchema { name: "interface", field_type: String, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }}, datum_type: CumulativeU64, version: 1, authz_scope: Fleet, units: Count, created: 2026-07-29T21:29:18.351057996Z } timeseries_name = ddm_session:updates_sent
1532026-07-29T21:29:18.352ZDEBGoximeter (clickhouse-client): inserting 14 new timeseries schema collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21
1542026-07-29T21:29:18.355ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21 n_rows = 26 table_name = fields_string
1552026-07-29T21:29:18.356ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21 n_rows = 28 table_name = fields_uuid
1562026-07-29T21:29:18.357ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21 n_rows = 40 table_name = measurements_cumulativeu64
1572026-07-29T21:29:18.359ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21 n_rows = 16 table_name = measurements_u64
1582026-07-29T21:29:18.393ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
1592026-07-29T21:29:18.393ZDEBGoximeter (collection-task): collecting from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
1602026-07-29T21:29:18.394ZDEBGoximeter (collection-task): collected results from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 14 producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
1612026-07-29T21:29:18.452ZWARNoximeter (database-inserter): failed to insert some results into metric DB collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:276
1622026-07-29T21:29:18.636ZDEBGoximeter (collection-task): reporting oximeter self-collection statistics collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
1632026-07-29T21:29:19.392ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
1642026-07-29T21:29:19.392ZDEBGoximeter (collection-task): collecting from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
1652026-07-29T21:29:19.393ZDEBGoximeter (collection-task): collected results from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 14 producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
1662026-07-29T21:29:20.393ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
1672026-07-29T21:29:20.393ZDEBGoximeter (collection-task): collecting from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
1682026-07-29T21:29:20.394ZDEBGoximeter (collection-task): collected results from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 14 producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
1692026-07-29T21:29:21.393ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
1702026-07-29T21:29:21.393ZDEBGoximeter (collection-task): collecting from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
1712026-07-29T21:29:21.394ZDEBGoximeter (collection-task): collected results from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 14 producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
1722026-07-29T21:29:22.145ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = 6f2c93a7-3ce8-4cc1-8671-369f638cb080 producer_ip = fd00:1122:3344:101::2 producer_port = 37767
1732026-07-29T21:29:22.145ZDEBGoximeter (collection-task): collecting from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = 6f2c93a7-3ce8-4cc1-8671-369f638cb080 producer_ip = fd00:1122:3344:101::2 producer_port = 37767
1742026-07-29T21:29:22.153ZDEBGoximeter (collection-task): collected results from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d n_results = 2 n_samples = 11 producer_id = 6f2c93a7-3ce8-4cc1-8671-369f638cb080 producer_ip = fd00:1122:3344:101::2 producer_port = 37767
1752026-07-29T21:29:22.393ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
1762026-07-29T21:29:22.393ZDEBGoximeter (collection-task): collecting from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
1772026-07-29T21:29:22.394ZDEBGoximeter (collection-task): collected results from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 14 producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
1782026-07-29T21:29:23.352ZDEBGoximeter (database-inserter): inserting samples into database collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d n_samples = 83
1792026-07-29T21:29:23.352ZDEBGoximeter (database-inserter): inserting samples into database collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d n_samples = 83
1802026-07-29T21:29:23.352ZDEBGoximeter (clickhouse-client): unrolling 83 total samples collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21
1812026-07-29T21:29:23.352ZDEBGoximeter (clickhouse-client): fetching timeseries schema from database collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21
1822026-07-29T21:29:23.355ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21 schema = TimeseriesSchema { timeseries_name: TimeseriesName("http_service:request_latency_histogram"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "name", field_type: String, source: Target, description: "" }, FieldSchema { name: "operation_id", field_type: String, source: Metric, description: "" }, FieldSchema { name: "status_code", field_type: U16, source: Metric, description: "" }}, datum_type: HistogramU64, version: 1, authz_scope: Fleet, units: Count, created: 2026-07-29T21:29:23.352280834Z } timeseries_name = http_service:request_latency_histogram
1832026-07-29T21:29:23.356ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21 schema = TimeseriesSchema { timeseries_name: TimeseriesName("oximeter_collector:collections"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "base_route", field_type: String, source: Metric, description: "" }, FieldSchema { name: "collector_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "collector_ip", field_type: IpAddr, source: Target, description: "" }, FieldSchema { name: "collector_port", field_type: U16, source: Target, description: "" }, FieldSchema { name: "producer_id", field_type: Uuid, source: Metric, description: "" }, FieldSchema { name: "producer_ip", field_type: IpAddr, source: Metric, description: "" }, FieldSchema { name: "producer_port", field_type: U16, source: Metric, description: "" }}, datum_type: CumulativeU64, version: 1, authz_scope: Fleet, units: Count, created: 2026-07-29T21:29:23.352261394Z } timeseries_name = oximeter_collector:collections
1842026-07-29T21:29:23.356ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21 schema = TimeseriesSchema { timeseries_name: TimeseriesName("oximeter_collector:samples_collected"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "base_route", field_type: String, source: Metric, description: "" }, FieldSchema { name: "collector_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "collector_ip", field_type: IpAddr, source: Target, description: "" }, FieldSchema { name: "collector_port", field_type: U16, source: Target, description: "" }, FieldSchema { name: "producer_id", field_type: Uuid, source: Metric, description: "" }, FieldSchema { name: "producer_ip", field_type: IpAddr, source: Metric, description: "" }, FieldSchema { name: "producer_port", field_type: U16, source: Metric, description: "" }}, datum_type: CumulativeU64, version: 1, authz_scope: Fleet, units: Count, created: 2026-07-29T21:29:23.352263494Z } timeseries_name = oximeter_collector:samples_collected
1852026-07-29T21:29:23.356ZDEBGoximeter (clickhouse-client): inserting 3 new timeseries schema collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21
1862026-07-29T21:29:23.359ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21 n_rows = 4 table_name = fields_ipaddr
1872026-07-29T21:29:23.360ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21 n_rows = 50 table_name = fields_string
1882026-07-29T21:29:23.361ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21 n_rows = 15 table_name = fields_u16
1892026-07-29T21:29:23.362ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21 n_rows = 43 table_name = fields_uuid
1902026-07-29T21:29:23.363ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21 n_rows = 52 table_name = measurements_cumulativeu64
1912026-07-29T21:29:23.365ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21 n_rows = 11 table_name = measurements_histogramu64
1922026-07-29T21:29:23.366ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21 n_rows = 20 table_name = measurements_u64
1932026-07-29T21:29:23.394ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
1942026-07-29T21:29:23.394ZDEBGoximeter (collection-task): collecting from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
1952026-07-29T21:29:23.395ZDEBGoximeter (collection-task): collected results from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 14 producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
1962026-07-29T21:29:23.452ZWARNoximeter (database-inserter): failed to insert some results into metric DB collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:276
1972026-07-29T21:29:24.393ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
1982026-07-29T21:29:24.393ZDEBGoximeter (collection-task): collecting from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
1992026-07-29T21:29:24.394ZDEBGoximeter (collection-task): collected results from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 14 producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
2002026-07-29T21:29:25.393ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
2012026-07-29T21:29:25.393ZDEBGoximeter (collection-task): collecting from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
2022026-07-29T21:29:25.394ZDEBGoximeter (collection-task): collected results from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 14 producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
2032026-07-29T21:29:26.393ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
2042026-07-29T21:29:26.394ZDEBGoximeter (collection-task): collecting from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
2052026-07-29T21:29:26.395ZDEBGoximeter (collection-task): collected results from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 14 producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
2062026-07-29T21:29:27.394ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
2072026-07-29T21:29:27.394ZDEBGoximeter (collection-task): collecting from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
2082026-07-29T21:29:27.396ZDEBGoximeter (collection-task): collected results from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 14 producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
2092026-07-29T21:29:28.352ZDEBGoximeter (database-inserter): inserting samples into database collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d n_samples = 70
2102026-07-29T21:29:28.352ZDEBGoximeter (database-inserter): inserting samples into database collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d n_samples = 70
2112026-07-29T21:29:28.352ZDEBGoximeter (clickhouse-client): unrolling 70 total samples collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21
2122026-07-29T21:29:28.352ZDEBGoximeter (clickhouse-client): fetching timeseries schema from database collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21
2132026-07-29T21:29:28.356ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21 n_rows = 26 table_name = fields_string
2142026-07-29T21:29:28.358ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21 n_rows = 28 table_name = fields_uuid
2152026-07-29T21:29:28.359ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21 n_rows = 50 table_name = measurements_cumulativeu64
2162026-07-29T21:29:28.360ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21 n_rows = 20 table_name = measurements_u64
2172026-07-29T21:29:28.394ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
2182026-07-29T21:29:28.394ZDEBGoximeter (collection-task): collecting from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
2192026-07-29T21:29:28.395ZDEBGoximeter (collection-task): collected results from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 14 producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
2202026-07-29T21:29:28.453ZWARNoximeter (database-inserter): failed to insert some results into metric DB collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:276
2212026-07-29T21:29:28.465ZINFOoximeter (oximeter-agent): refreshing list of producers from Nexus collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d file = oximeter/collector/src/agent.rs:629
2222026-07-29T21:29:28.465ZDEBGoximeter: client request body = None method = GET uri = http://[fd00:1122:3344:101::a]:12221/metrics/collectors/fa84a12f-09b4-438f-8dee-6b54032162b4/producers?sort_by=id_ascending
2232026-07-29T21:29:28.468ZDEBGoximeter: client response result = Ok(Response { url: "http://[fd00:1122:3344:101::a]:12221/metrics/collectors/fa84a12f-09b4-438f-8dee-6b54032162b4/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "3938e228-101d-4bf6-82f8-4e6e59e82728", "content-length": "721", "date": "Wed, 29 Jul 2026 21:29:28 GMT"} })
2242026-07-29T21:29:28.468ZDEBGoximeter: client request body = None method = GET uri = http://[fd00:1122:3344:101::a]:12221/metrics/collectors/fa84a12f-09b4-438f-8dee-6b54032162b4/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNTNhZjRiZS00YzhmLTRkMDAtYTAzNS1jNTE0YjQ1N2M5Y2MifX0%3D
2252026-07-29T21:29:28.472ZDEBGoximeter: client response result = Ok(Response { url: "http://[fd00:1122:3344:101::a]:12221/metrics/collectors/fa84a12f-09b4-438f-8dee-6b54032162b4/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNTNhZjRiZS00YzhmLTRkMDAtYTAzNS1jNTE0YjQ1N2M5Y2MifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "5f96219e-4a60-44da-88a9-ad2710bd1095", "content-length": "29", "date": "Wed, 29 Jul 2026 21:29:28 GMT"} })
2262026-07-29T21:29:28.472ZDEBGoximeter (oximeter-agent): registered new metric producer address = [fd00:1122:3344:101::1]:8001 collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = 609be78f-16d5-4f8a-b3dd-f5270fe1de68
2272026-07-29T21:29:28.472ZINFOoximeter (oximeter-agent): refreshed list of producers from Nexus collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d file = oximeter/collector/src/agent.rs:704 n_current_tasks = 4 n_pruned_tasks = 0
2282026-07-29T21:29:29.394ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
2292026-07-29T21:29:29.394ZDEBGoximeter (collection-task): collecting from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
2302026-07-29T21:29:29.395ZDEBGoximeter (collection-task): collected results from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 14 producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
2312026-07-29T21:29:29.435ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = 609be78f-16d5-4f8a-b3dd-f5270fe1de68 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
2322026-07-29T21:29:29.435ZDEBGoximeter (collection-task): collecting from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = 609be78f-16d5-4f8a-b3dd-f5270fe1de68 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
2332026-07-29T21:29:29.440ZDEBGoximeter (collection-task): collected results from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 26 producer_id = 609be78f-16d5-4f8a-b3dd-f5270fe1de68 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
2342026-07-29T21:29:30.394ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
2352026-07-29T21:29:30.394ZDEBGoximeter (collection-task): collecting from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
2362026-07-29T21:29:30.395ZDEBGoximeter (collection-task): collected results from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 14 producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
2372026-07-29T21:29:30.435ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = 609be78f-16d5-4f8a-b3dd-f5270fe1de68 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
2382026-07-29T21:29:30.435ZDEBGoximeter (collection-task): collecting from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = 609be78f-16d5-4f8a-b3dd-f5270fe1de68 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
2392026-07-29T21:29:30.437ZDEBGoximeter (collection-task): collected results from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 26 producer_id = 609be78f-16d5-4f8a-b3dd-f5270fe1de68 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
2402026-07-29T21:29:31.394ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
2412026-07-29T21:29:31.394ZDEBGoximeter (collection-task): collecting from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
2422026-07-29T21:29:31.395ZDEBGoximeter (collection-task): collected results from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 14 producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
2432026-07-29T21:29:31.435ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = 609be78f-16d5-4f8a-b3dd-f5270fe1de68 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
2442026-07-29T21:29:31.435ZDEBGoximeter (collection-task): collecting from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = 609be78f-16d5-4f8a-b3dd-f5270fe1de68 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
2452026-07-29T21:29:31.437ZDEBGoximeter (collection-task): collected results from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 26 producer_id = 609be78f-16d5-4f8a-b3dd-f5270fe1de68 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
2462026-07-29T21:29:32.145ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = 6f2c93a7-3ce8-4cc1-8671-369f638cb080 producer_ip = fd00:1122:3344:101::2 producer_port = 37767
2472026-07-29T21:29:32.145ZDEBGoximeter (collection-task): collecting from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = 6f2c93a7-3ce8-4cc1-8671-369f638cb080 producer_ip = fd00:1122:3344:101::2 producer_port = 37767
2482026-07-29T21:29:32.147ZDEBGoximeter (collection-task): collected results from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d n_results = 2 n_samples = 11 producer_id = 6f2c93a7-3ce8-4cc1-8671-369f638cb080 producer_ip = fd00:1122:3344:101::2 producer_port = 37767
2492026-07-29T21:29:32.394ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
2502026-07-29T21:29:32.394ZDEBGoximeter (collection-task): collecting from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
2512026-07-29T21:29:32.395ZDEBGoximeter (collection-task): collected results from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 14 producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
2522026-07-29T21:29:32.435ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = 609be78f-16d5-4f8a-b3dd-f5270fe1de68 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
2532026-07-29T21:29:32.435ZDEBGoximeter (collection-task): collecting from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = 609be78f-16d5-4f8a-b3dd-f5270fe1de68 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
2542026-07-29T21:29:32.437ZDEBGoximeter (collection-task): collected results from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 26 producer_id = 609be78f-16d5-4f8a-b3dd-f5270fe1de68 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
2552026-07-29T21:29:33.352ZDEBGoximeter (database-inserter): inserting samples into database collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d n_samples = 185
2562026-07-29T21:29:33.352ZDEBGoximeter (database-inserter): inserting samples into database collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d n_samples = 185
2572026-07-29T21:29:33.352ZDEBGoximeter (clickhouse-client): unrolling 185 total samples collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21
2582026-07-29T21:29:33.352ZDEBGoximeter (clickhouse-client): fetching timeseries schema from database collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21
2592026-07-29T21:29:33.357ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21 n_rows = 98 table_name = fields_string
2602026-07-29T21:29:33.358ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21 n_rows = 11 table_name = fields_u16
2612026-07-29T21:29:33.359ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21 n_rows = 91 table_name = fields_uuid
2622026-07-29T21:29:33.361ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21 n_rows = 130 table_name = measurements_cumulativeu64
2632026-07-29T21:29:33.362ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21 n_rows = 11 table_name = measurements_histogramu64
2642026-07-29T21:29:33.364ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21 n_rows = 44 table_name = measurements_u64
2652026-07-29T21:29:33.394ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
2662026-07-29T21:29:33.394ZDEBGoximeter (collection-task): collecting from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
2672026-07-29T21:29:33.395ZDEBGoximeter (collection-task): collected results from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 14 producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
2682026-07-29T21:29:33.435ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = 609be78f-16d5-4f8a-b3dd-f5270fe1de68 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
2692026-07-29T21:29:33.435ZDEBGoximeter (collection-task): collecting from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = 609be78f-16d5-4f8a-b3dd-f5270fe1de68 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
2702026-07-29T21:29:33.436ZDEBGoximeter (collection-task): collected results from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 26 producer_id = 609be78f-16d5-4f8a-b3dd-f5270fe1de68 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
2712026-07-29T21:29:33.452ZWARNoximeter (database-inserter): failed to insert some results into metric DB collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:276
2722026-07-29T21:29:34.394ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
2732026-07-29T21:29:34.394ZDEBGoximeter (collection-task): collecting from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
2742026-07-29T21:29:34.395ZDEBGoximeter (collection-task): collected results from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 14 producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
2752026-07-29T21:29:34.435ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = 609be78f-16d5-4f8a-b3dd-f5270fe1de68 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
2762026-07-29T21:29:34.435ZDEBGoximeter (collection-task): collecting from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = 609be78f-16d5-4f8a-b3dd-f5270fe1de68 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
2772026-07-29T21:29:34.437ZDEBGoximeter (collection-task): collected results from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 26 producer_id = 609be78f-16d5-4f8a-b3dd-f5270fe1de68 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
2782026-07-29T21:29:35.394ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
2792026-07-29T21:29:35.394ZDEBGoximeter (collection-task): collecting from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
2802026-07-29T21:29:35.395ZDEBGoximeter (collection-task): collected results from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 14 producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
2812026-07-29T21:29:35.435ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = 609be78f-16d5-4f8a-b3dd-f5270fe1de68 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
2822026-07-29T21:29:35.435ZDEBGoximeter (collection-task): collecting from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = 609be78f-16d5-4f8a-b3dd-f5270fe1de68 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
2832026-07-29T21:29:35.437ZDEBGoximeter (collection-task): collected results from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 26 producer_id = 609be78f-16d5-4f8a-b3dd-f5270fe1de68 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
2842026-07-29T21:29:36.394ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
2852026-07-29T21:29:36.394ZDEBGoximeter (collection-task): collecting from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
2862026-07-29T21:29:36.395ZDEBGoximeter (collection-task): collected results from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 14 producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
2872026-07-29T21:29:36.434ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = 609be78f-16d5-4f8a-b3dd-f5270fe1de68 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
2882026-07-29T21:29:36.434ZDEBGoximeter (collection-task): collecting from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = 609be78f-16d5-4f8a-b3dd-f5270fe1de68 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
2892026-07-29T21:29:36.436ZDEBGoximeter (collection-task): collected results from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 26 producer_id = 609be78f-16d5-4f8a-b3dd-f5270fe1de68 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
2902026-07-29T21:29:37.394ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
2912026-07-29T21:29:37.394ZDEBGoximeter (collection-task): collecting from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
2922026-07-29T21:29:37.395ZDEBGoximeter (collection-task): collected results from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 14 producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
2932026-07-29T21:29:37.434ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = 609be78f-16d5-4f8a-b3dd-f5270fe1de68 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
2942026-07-29T21:29:37.434ZDEBGoximeter (collection-task): collecting from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = 609be78f-16d5-4f8a-b3dd-f5270fe1de68 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
2952026-07-29T21:29:37.437ZDEBGoximeter (collection-task): collected results from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 26 producer_id = 609be78f-16d5-4f8a-b3dd-f5270fe1de68 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
2962026-07-29T21:29:38.352ZDEBGoximeter (database-inserter): inserting samples into database collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d n_samples = 200
2972026-07-29T21:29:38.352ZDEBGoximeter (database-inserter): inserting samples into database collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d n_samples = 200
2982026-07-29T21:29:38.352ZDEBGoximeter (clickhouse-client): unrolling 200 total samples collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21
2992026-07-29T21:29:38.352ZDEBGoximeter (clickhouse-client): fetching timeseries schema from database collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21
3002026-07-29T21:29:38.357ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21 n_rows = 76 table_name = fields_string
3012026-07-29T21:29:38.359ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21 n_rows = 80 table_name = fields_uuid
3022026-07-29T21:29:38.360ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21 n_rows = 150 table_name = measurements_cumulativeu64
3032026-07-29T21:29:38.362ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21 n_rows = 50 table_name = measurements_u64
3042026-07-29T21:29:38.394ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
3052026-07-29T21:29:38.394ZDEBGoximeter (collection-task): collecting from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
3062026-07-29T21:29:38.395ZDEBGoximeter (collection-task): collected results from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 14 producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
3072026-07-29T21:29:38.435ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = 609be78f-16d5-4f8a-b3dd-f5270fe1de68 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
3082026-07-29T21:29:38.435ZDEBGoximeter (collection-task): collecting from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = 609be78f-16d5-4f8a-b3dd-f5270fe1de68 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
3092026-07-29T21:29:38.437ZDEBGoximeter (collection-task): collected results from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 26 producer_id = 609be78f-16d5-4f8a-b3dd-f5270fe1de68 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
3102026-07-29T21:29:38.453ZWARNoximeter (database-inserter): failed to insert some results into metric DB collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:276
3112026-07-29T21:29:39.394ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
3122026-07-29T21:29:39.394ZDEBGoximeter (collection-task): collecting from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
3132026-07-29T21:29:39.395ZDEBGoximeter (collection-task): collected results from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 14 producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
3142026-07-29T21:29:39.435ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = 609be78f-16d5-4f8a-b3dd-f5270fe1de68 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
3152026-07-29T21:29:39.435ZDEBGoximeter (collection-task): collecting from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = 609be78f-16d5-4f8a-b3dd-f5270fe1de68 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
3162026-07-29T21:29:39.437ZDEBGoximeter (collection-task): collected results from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 26 producer_id = 609be78f-16d5-4f8a-b3dd-f5270fe1de68 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
3172026-07-29T21:29:40.394ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
3182026-07-29T21:29:40.394ZDEBGoximeter (collection-task): collecting from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
3192026-07-29T21:29:40.395ZDEBGoximeter (collection-task): collected results from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 14 producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
3202026-07-29T21:29:40.435ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = 609be78f-16d5-4f8a-b3dd-f5270fe1de68 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
3212026-07-29T21:29:40.435ZDEBGoximeter (collection-task): collecting from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = 609be78f-16d5-4f8a-b3dd-f5270fe1de68 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
3222026-07-29T21:29:40.437ZDEBGoximeter (collection-task): collected results from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 26 producer_id = 609be78f-16d5-4f8a-b3dd-f5270fe1de68 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
3232026-07-29T21:29:40.647ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = 04808080-7e81-4ad5-99dd-16df9ab0b849 producer_ip = fd00:1122:3344:101::2 producer_port = 59901
3242026-07-29T21:29:40.647ZDEBGoximeter (collection-task): collecting from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = 04808080-7e81-4ad5-99dd-16df9ab0b849 producer_ip = fd00:1122:3344:101::2 producer_port = 59901
3252026-07-29T21:29:40.662ZDEBGoximeter (collection-task): collected results from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 246 producer_id = 04808080-7e81-4ad5-99dd-16df9ab0b849 producer_ip = fd00:1122:3344:101::2 producer_port = 59901
3262026-07-29T21:29:41.394ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
3272026-07-29T21:29:41.394ZDEBGoximeter (collection-task): collecting from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
3282026-07-29T21:29:41.395ZDEBGoximeter (collection-task): collected results from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 14 producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
3292026-07-29T21:29:41.435ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = 609be78f-16d5-4f8a-b3dd-f5270fe1de68 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
3302026-07-29T21:29:41.435ZDEBGoximeter (collection-task): collecting from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = 609be78f-16d5-4f8a-b3dd-f5270fe1de68 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
3312026-07-29T21:29:41.437ZDEBGoximeter (collection-task): collected results from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 26 producer_id = 609be78f-16d5-4f8a-b3dd-f5270fe1de68 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
3322026-07-29T21:29:42.145ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = 6f2c93a7-3ce8-4cc1-8671-369f638cb080 producer_ip = fd00:1122:3344:101::2 producer_port = 37767
3332026-07-29T21:29:42.145ZDEBGoximeter (collection-task): collecting from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = 6f2c93a7-3ce8-4cc1-8671-369f638cb080 producer_ip = fd00:1122:3344:101::2 producer_port = 37767
3342026-07-29T21:29:42.146ZDEBGoximeter (collection-task): collected results from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d n_results = 2 n_samples = 11 producer_id = 6f2c93a7-3ce8-4cc1-8671-369f638cb080 producer_ip = fd00:1122:3344:101::2 producer_port = 37767
3352026-07-29T21:29:42.394ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
3362026-07-29T21:29:42.394ZDEBGoximeter (collection-task): collecting from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
3372026-07-29T21:29:42.395ZDEBGoximeter (collection-task): collected results from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 14 producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
3382026-07-29T21:29:42.435ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = 609be78f-16d5-4f8a-b3dd-f5270fe1de68 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
3392026-07-29T21:29:42.435ZDEBGoximeter (collection-task): collecting from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = 609be78f-16d5-4f8a-b3dd-f5270fe1de68 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
3402026-07-29T21:29:42.436ZDEBGoximeter (collection-task): collected results from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 26 producer_id = 609be78f-16d5-4f8a-b3dd-f5270fe1de68 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
3412026-07-29T21:29:43.352ZDEBGoximeter (database-inserter): inserting samples into database collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d n_samples = 461
3422026-07-29T21:29:43.352ZDEBGoximeter (database-inserter): inserting samples into database collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d n_samples = 461
3432026-07-29T21:29:43.352ZDEBGoximeter (clickhouse-client): unrolling 461 total samples collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21
3442026-07-29T21:29:43.353ZDEBGoximeter (clickhouse-client): fetching timeseries schema from database collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21
3452026-07-29T21:29:43.356ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21 schema = TimeseriesSchema { timeseries_name: TimeseriesName("switch_port_control_data_link:bytes_received"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "asic_fab", field_type: String, source: Target, description: "" }, FieldSchema { name: "asic_lot", field_type: String, source: Target, description: "" }, FieldSchema { name: "asic_wafer", field_type: U8, source: Target, description: "" }, FieldSchema { name: "asic_wafer_loc_x", field_type: I16, source: Target, description: "" }, FieldSchema { name: "asic_wafer_loc_y", field_type: I16, source: Target, description: "" }, FieldSchema { name: "kind", field_type: String, source: Target, description: "" }, FieldSchema { name: "link_name", field_type: String, source: Target, description: "" }, FieldSchema { name: "model", field_type: String, source: Target, description: "" }, FieldSchema { name: "network", field_type: String, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_model", field_type: String, source: Target, description: "" }, FieldSchema { name: "sled_revision", field_type: U32, source: Target, description: "" }, FieldSchema { name: "sled_serial", field_type: String, source: Target, description: "" }, FieldSchema { name: "switch_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "switch_model", field_type: String, source: Target, description: "" }, FieldSchema { name: "switch_revision", field_type: U32, source: Target, description: "" }, FieldSchema { name: "switch_serial", field_type: String, source: Target, description: "" }, FieldSchema { name: "switch_slot", field_type: U16, source: Target, description: "" }}, datum_type: CumulativeU64, version: 1, authz_scope: Fleet, units: Count, created: 2026-07-29T21:29:43.352980302Z } timeseries_name = switch_port_control_data_link:bytes_received
3462026-07-29T21:29:43.356ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21 schema = TimeseriesSchema { timeseries_name: TimeseriesName("switch_port_control_data_link:bytes_sent"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "asic_fab", field_type: String, source: Target, description: "" }, FieldSchema { name: "asic_lot", field_type: String, source: Target, description: "" }, FieldSchema { name: "asic_wafer", field_type: U8, source: Target, description: "" }, FieldSchema { name: "asic_wafer_loc_x", field_type: I16, source: Target, description: "" }, FieldSchema { name: "asic_wafer_loc_y", field_type: I16, source: Target, description: "" }, FieldSchema { name: "kind", field_type: String, source: Target, description: "" }, FieldSchema { name: "link_name", field_type: String, source: Target, description: "" }, FieldSchema { name: "model", field_type: String, source: Target, description: "" }, FieldSchema { name: "network", field_type: String, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_model", field_type: String, source: Target, description: "" }, FieldSchema { name: "sled_revision", field_type: U32, source: Target, description: "" }, FieldSchema { name: "sled_serial", field_type: String, source: Target, description: "" }, FieldSchema { name: "switch_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "switch_model", field_type: String, source: Target, description: "" }, FieldSchema { name: "switch_revision", field_type: U32, source: Target, description: "" }, FieldSchema { name: "switch_serial", field_type: String, source: Target, description: "" }, FieldSchema { name: "switch_slot", field_type: U16, source: Target, description: "" }}, datum_type: CumulativeU64, version: 1, authz_scope: Fleet, units: Count, created: 2026-07-29T21:29:43.352990272Z } timeseries_name = switch_port_control_data_link:bytes_sent
3472026-07-29T21:29:43.356ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21 schema = TimeseriesSchema { timeseries_name: TimeseriesName("switch_port_control_data_link:errors_received"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "asic_fab", field_type: String, source: Target, description: "" }, FieldSchema { name: "asic_lot", field_type: String, source: Target, description: "" }, FieldSchema { name: "asic_wafer", field_type: U8, source: Target, description: "" }, FieldSchema { name: "asic_wafer_loc_x", field_type: I16, source: Target, description: "" }, FieldSchema { name: "asic_wafer_loc_y", field_type: I16, source: Target, description: "" }, FieldSchema { name: "kind", field_type: String, source: Target, description: "" }, FieldSchema { name: "link_name", field_type: String, source: Target, description: "" }, FieldSchema { name: "model", field_type: String, source: Target, description: "" }, FieldSchema { name: "network", field_type: String, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_model", field_type: String, source: Target, description: "" }, FieldSchema { name: "sled_revision", field_type: U32, source: Target, description: "" }, FieldSchema { name: "sled_serial", field_type: String, source: Target, description: "" }, FieldSchema { name: "switch_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "switch_model", field_type: String, source: Target, description: "" }, FieldSchema { name: "switch_revision", field_type: U32, source: Target, description: "" }, FieldSchema { name: "switch_serial", field_type: String, source: Target, description: "" }, FieldSchema { name: "switch_slot", field_type: U16, source: Target, description: "" }}, datum_type: CumulativeU64, version: 1, authz_scope: Fleet, units: Count, created: 2026-07-29T21:29:43.352968002Z } timeseries_name = switch_port_control_data_link:errors_received
3482026-07-29T21:29:43.356ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21 schema = TimeseriesSchema { timeseries_name: TimeseriesName("switch_port_control_data_link:errors_sent"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "asic_fab", field_type: String, source: Target, description: "" }, FieldSchema { name: "asic_lot", field_type: String, source: Target, description: "" }, FieldSchema { name: "asic_wafer", field_type: U8, source: Target, description: "" }, FieldSchema { name: "asic_wafer_loc_x", field_type: I16, source: Target, description: "" }, FieldSchema { name: "asic_wafer_loc_y", field_type: I16, source: Target, description: "" }, FieldSchema { name: "kind", field_type: String, source: Target, description: "" }, FieldSchema { name: "link_name", field_type: String, source: Target, description: "" }, FieldSchema { name: "model", field_type: String, source: Target, description: "" }, FieldSchema { name: "network", field_type: String, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_model", field_type: String, source: Target, description: "" }, FieldSchema { name: "sled_revision", field_type: U32, source: Target, description: "" }, FieldSchema { name: "sled_serial", field_type: String, source: Target, description: "" }, FieldSchema { name: "switch_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "switch_model", field_type: String, source: Target, description: "" }, FieldSchema { name: "switch_revision", field_type: U32, source: Target, description: "" }, FieldSchema { name: "switch_serial", field_type: String, source: Target, description: "" }, FieldSchema { name: "switch_slot", field_type: U16, source: Target, description: "" }}, datum_type: CumulativeU64, version: 1, authz_scope: Fleet, units: Count, created: 2026-07-29T21:29:43.352975232Z } timeseries_name = switch_port_control_data_link:errors_sent
3492026-07-29T21:29:43.356ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21 schema = TimeseriesSchema { timeseries_name: TimeseriesName("switch_port_control_data_link:packets_received"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "asic_fab", field_type: String, source: Target, description: "" }, FieldSchema { name: "asic_lot", field_type: String, source: Target, description: "" }, FieldSchema { name: "asic_wafer", field_type: U8, source: Target, description: "" }, FieldSchema { name: "asic_wafer_loc_x", field_type: I16, source: Target, description: "" }, FieldSchema { name: "asic_wafer_loc_y", field_type: I16, source: Target, description: "" }, FieldSchema { name: "kind", field_type: String, source: Target, description: "" }, FieldSchema { name: "link_name", field_type: String, source: Target, description: "" }, FieldSchema { name: "model", field_type: String, source: Target, description: "" }, FieldSchema { name: "network", field_type: String, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_model", field_type: String, source: Target, description: "" }, FieldSchema { name: "sled_revision", field_type: U32, source: Target, description: "" }, FieldSchema { name: "sled_serial", field_type: String, source: Target, description: "" }, FieldSchema { name: "switch_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "switch_model", field_type: String, source: Target, description: "" }, FieldSchema { name: "switch_revision", field_type: U32, source: Target, description: "" }, FieldSchema { name: "switch_serial", field_type: String, source: Target, description: "" }, FieldSchema { name: "switch_slot", field_type: U16, source: Target, description: "" }}, datum_type: CumulativeU64, version: 1, authz_scope: Fleet, units: Count, created: 2026-07-29T21:29:43.352985312Z } timeseries_name = switch_port_control_data_link:packets_received
3502026-07-29T21:29:43.356ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21 schema = TimeseriesSchema { timeseries_name: TimeseriesName("switch_port_control_data_link:packets_sent"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "asic_fab", field_type: String, source: Target, description: "" }, FieldSchema { name: "asic_lot", field_type: String, source: Target, description: "" }, FieldSchema { name: "asic_wafer", field_type: U8, source: Target, description: "" }, FieldSchema { name: "asic_wafer_loc_x", field_type: I16, source: Target, description: "" }, FieldSchema { name: "asic_wafer_loc_y", field_type: I16, source: Target, description: "" }, FieldSchema { name: "kind", field_type: String, source: Target, description: "" }, FieldSchema { name: "link_name", field_type: String, source: Target, description: "" }, FieldSchema { name: "model", field_type: String, source: Target, description: "" }, FieldSchema { name: "network", field_type: String, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_model", field_type: String, source: Target, description: "" }, FieldSchema { name: "sled_revision", field_type: U32, source: Target, description: "" }, FieldSchema { name: "sled_serial", field_type: String, source: Target, description: "" }, FieldSchema { name: "switch_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "switch_model", field_type: String, source: Target, description: "" }, FieldSchema { name: "switch_revision", field_type: U32, source: Target, description: "" }, FieldSchema { name: "switch_serial", field_type: String, source: Target, description: "" }, FieldSchema { name: "switch_slot", field_type: U16, source: Target, description: "" }}, datum_type: CumulativeU64, version: 1, authz_scope: Fleet, units: Count, created: 2026-07-29T21:29:43.352995002Z } timeseries_name = switch_port_control_data_link:packets_sent
3512026-07-29T21:29:43.359ZDEBGoximeter (clickhouse-client): inserting 6 new timeseries schema collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21
3522026-07-29T21:29:43.362ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21 n_rows = 24 table_name = fields_i16
3532026-07-29T21:29:43.363ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21 n_rows = 4 table_name = fields_ipaddr
3542026-07-29T21:29:43.364ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21 n_rows = 222 table_name = fields_string
3552026-07-29T21:29:43.365ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21 n_rows = 27 table_name = fields_u16
3562026-07-29T21:29:43.367ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21 n_rows = 24 table_name = fields_u32
3572026-07-29T21:29:43.368ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21 n_rows = 12 table_name = fields_u8
3582026-07-29T21:29:43.369ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21 n_rows = 131 table_name = fields_uuid
3592026-07-29T21:29:43.370ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21 n_rows = 398 table_name = measurements_cumulativeu64
3602026-07-29T21:29:43.372ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21 n_rows = 13 table_name = measurements_histogramu64
3612026-07-29T21:29:43.373ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21 n_rows = 50 table_name = measurements_u64
3622026-07-29T21:29:43.394ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
3632026-07-29T21:29:43.394ZDEBGoximeter (collection-task): collecting from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
3642026-07-29T21:29:43.395ZDEBGoximeter (collection-task): collected results from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 14 producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
3652026-07-29T21:29:43.435ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = 609be78f-16d5-4f8a-b3dd-f5270fe1de68 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
3662026-07-29T21:29:43.435ZDEBGoximeter (collection-task): collecting from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = 609be78f-16d5-4f8a-b3dd-f5270fe1de68 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
3672026-07-29T21:29:43.437ZDEBGoximeter (collection-task): collected results from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 26 producer_id = 609be78f-16d5-4f8a-b3dd-f5270fe1de68 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
3682026-07-29T21:29:43.454ZWARNoximeter (database-inserter): failed to insert some results into metric DB collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:276
3692026-07-29T21:29:43.465ZINFOoximeter (oximeter-agent): refreshing list of producers from Nexus collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d file = oximeter/collector/src/agent.rs:629
3702026-07-29T21:29:43.465ZDEBGoximeter: client request body = None method = GET uri = http://[fd00:1122:3344:101::a]:12221/metrics/collectors/fa84a12f-09b4-438f-8dee-6b54032162b4/producers?sort_by=id_ascending
3712026-07-29T21:29:43.469ZDEBGoximeter: client response result = Ok(Response { url: "http://[fd00:1122:3344:101::a]:12221/metrics/collectors/fa84a12f-09b4-438f-8dee-6b54032162b4/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "a255a0a3-430d-4934-b376-3e67d0b6baa7", "content-length": "1135", "date": "Wed, 29 Jul 2026 21:29:43 GMT"} })
3722026-07-29T21:29:43.469ZDEBGoximeter: client request body = None method = GET uri = http://[fd00:1122:3344:101::a]:12221/metrics/collectors/fa84a12f-09b4-438f-8dee-6b54032162b4/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJjNGQ1NDczYS00MTA0LTRjYzQtYThlMy01MDE0ZDVhOTU4ZGMifX0%3D
3732026-07-29T21:29:43.472ZDEBGoximeter: client response result = Ok(Response { url: "http://[fd00:1122:3344:101::a]:12221/metrics/collectors/fa84a12f-09b4-438f-8dee-6b54032162b4/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJjNGQ1NDczYS00MTA0LTRjYzQtYThlMy01MDE0ZDVhOTU4ZGMifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "aca095a5-55d5-45d8-a5d7-41cb2e65f5a1", "content-length": "29", "date": "Wed, 29 Jul 2026 21:29:43 GMT"} })
3742026-07-29T21:29:43.472ZDEBGoximeter (oximeter-agent): registered new metric producer address = [fd00:1122:3344:101::c]:37508 collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = 7495615a-bf4a-4adb-8f22-76b6ec0eeaf0
3752026-07-29T21:29:43.472ZDEBGoximeter (oximeter-agent): registered new metric producer address = [fd00:1122:3344:101::a]:33085 collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = 8a6d7931-75ff-4c25-aab4-0b7211fc2ced
3762026-07-29T21:29:43.472ZDEBGoximeter (oximeter-agent): registered new metric producer address = [fd00:1122:3344:101::b]:33382 collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = c4d5473a-4104-4cc4-a8e3-5014d5a958dc
3772026-07-29T21:29:43.472ZINFOoximeter (oximeter-agent): refreshed list of producers from Nexus collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d file = oximeter/collector/src/agent.rs:704 n_current_tasks = 7 n_pruned_tasks = 0
3782026-07-29T21:29:44.394ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
3792026-07-29T21:29:44.394ZDEBGoximeter (collection-task): collecting from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
3802026-07-29T21:29:44.395ZDEBGoximeter (collection-task): collected results from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 14 producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
3812026-07-29T21:29:44.436ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = 609be78f-16d5-4f8a-b3dd-f5270fe1de68 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
3822026-07-29T21:29:44.436ZDEBGoximeter (collection-task): collecting from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = 609be78f-16d5-4f8a-b3dd-f5270fe1de68 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
3832026-07-29T21:29:44.437ZDEBGoximeter (collection-task): collected results from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 26 producer_id = 609be78f-16d5-4f8a-b3dd-f5270fe1de68 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
3842026-07-29T21:29:44.744ZDEBGoximeter (collection-task): reporting oximeter self-collection statistics collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = 8a6d7931-75ff-4c25-aab4-0b7211fc2ced producer_ip = fd00:1122:3344:101::a producer_port = 33085
3852026-07-29T21:29:45.394ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
3862026-07-29T21:29:45.394ZDEBGoximeter (collection-task): collecting from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
3872026-07-29T21:29:45.396ZDEBGoximeter (collection-task): collected results from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 14 producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
3882026-07-29T21:29:45.435ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = 609be78f-16d5-4f8a-b3dd-f5270fe1de68 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
3892026-07-29T21:29:45.435ZDEBGoximeter (collection-task): collecting from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = 609be78f-16d5-4f8a-b3dd-f5270fe1de68 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
3902026-07-29T21:29:45.436ZDEBGoximeter (collection-task): collected results from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 26 producer_id = 609be78f-16d5-4f8a-b3dd-f5270fe1de68 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
3912026-07-29T21:29:46.394ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
3922026-07-29T21:29:46.394ZDEBGoximeter (collection-task): collecting from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
3932026-07-29T21:29:46.395ZDEBGoximeter (collection-task): collected results from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 14 producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
3942026-07-29T21:29:46.435ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = 609be78f-16d5-4f8a-b3dd-f5270fe1de68 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
3952026-07-29T21:29:46.435ZDEBGoximeter (collection-task): collecting from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = 609be78f-16d5-4f8a-b3dd-f5270fe1de68 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
3962026-07-29T21:29:46.436ZDEBGoximeter (collection-task): collected results from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 26 producer_id = 609be78f-16d5-4f8a-b3dd-f5270fe1de68 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
3972026-07-29T21:29:47.394ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
3982026-07-29T21:29:47.394ZDEBGoximeter (collection-task): collecting from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
3992026-07-29T21:29:47.395ZDEBGoximeter (collection-task): collected results from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 14 producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
4002026-07-29T21:29:47.435ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = 609be78f-16d5-4f8a-b3dd-f5270fe1de68 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
4012026-07-29T21:29:47.435ZDEBGoximeter (collection-task): collecting from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = 609be78f-16d5-4f8a-b3dd-f5270fe1de68 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
4022026-07-29T21:29:47.437ZDEBGoximeter (collection-task): collected results from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 26 producer_id = 609be78f-16d5-4f8a-b3dd-f5270fe1de68 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
4032026-07-29T21:29:48.352ZDEBGoximeter (database-inserter): inserting samples into database collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d n_samples = 202
4042026-07-29T21:29:48.352ZDEBGoximeter (database-inserter): inserting samples into database collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d n_samples = 202
4052026-07-29T21:29:48.352ZDEBGoximeter (clickhouse-client): unrolling 202 total samples collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21
4062026-07-29T21:29:48.352ZDEBGoximeter (clickhouse-client): fetching timeseries schema from database collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21
4072026-07-29T21:29:48.357ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21 n_rows = 4 table_name = fields_ipaddr
4082026-07-29T21:29:48.359ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21 n_rows = 78 table_name = fields_string
4092026-07-29T21:29:48.360ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21 n_rows = 4 table_name = fields_u16
4102026-07-29T21:29:48.361ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21 n_rows = 84 table_name = fields_uuid
4112026-07-29T21:29:48.362ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21 n_rows = 152 table_name = measurements_cumulativeu64
4122026-07-29T21:29:48.363ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21 n_rows = 50 table_name = measurements_u64
4132026-07-29T21:29:48.394ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
4142026-07-29T21:29:48.394ZDEBGoximeter (collection-task): collecting from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
4152026-07-29T21:29:48.395ZDEBGoximeter (collection-task): collected results from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 14 producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
4162026-07-29T21:29:48.422ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = 8a6d7931-75ff-4c25-aab4-0b7211fc2ced producer_ip = fd00:1122:3344:101::a producer_port = 33085
4172026-07-29T21:29:48.422ZDEBGoximeter (collection-task): collecting from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = 8a6d7931-75ff-4c25-aab4-0b7211fc2ced producer_ip = fd00:1122:3344:101::a producer_port = 33085
4182026-07-29T21:29:48.425ZDEBGoximeter (collection-task): collected results from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d n_results = 5 n_samples = 6 producer_id = 8a6d7931-75ff-4c25-aab4-0b7211fc2ced producer_ip = fd00:1122:3344:101::a producer_port = 33085
4192026-07-29T21:29:48.435ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = 609be78f-16d5-4f8a-b3dd-f5270fe1de68 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
4202026-07-29T21:29:48.435ZDEBGoximeter (collection-task): collecting from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = 609be78f-16d5-4f8a-b3dd-f5270fe1de68 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
4212026-07-29T21:29:48.436ZDEBGoximeter (collection-task): collected results from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 26 producer_id = 609be78f-16d5-4f8a-b3dd-f5270fe1de68 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
4222026-07-29T21:29:48.453ZWARNoximeter (database-inserter): failed to insert some results into metric DB collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:276
4232026-07-29T21:29:49.394ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
4242026-07-29T21:29:49.395ZDEBGoximeter (collection-task): collecting from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
4252026-07-29T21:29:49.396ZDEBGoximeter (collection-task): collected results from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 14 producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
4262026-07-29T21:29:49.435ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = 609be78f-16d5-4f8a-b3dd-f5270fe1de68 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
4272026-07-29T21:29:49.435ZDEBGoximeter (collection-task): collecting from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = 609be78f-16d5-4f8a-b3dd-f5270fe1de68 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
4282026-07-29T21:29:49.436ZDEBGoximeter (collection-task): collected results from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 26 producer_id = 609be78f-16d5-4f8a-b3dd-f5270fe1de68 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
4292026-07-29T21:29:49.724ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = c4d5473a-4104-4cc4-a8e3-5014d5a958dc producer_ip = fd00:1122:3344:101::b producer_port = 33382
4302026-07-29T21:29:49.724ZDEBGoximeter (collection-task): collecting from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = c4d5473a-4104-4cc4-a8e3-5014d5a958dc producer_ip = fd00:1122:3344:101::b producer_port = 33382
4312026-07-29T21:29:49.732ZDEBGoximeter (collection-task): collected results from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d n_results = 5 n_samples = 2 producer_id = c4d5473a-4104-4cc4-a8e3-5014d5a958dc producer_ip = fd00:1122:3344:101::b producer_port = 33382
4322026-07-29T21:29:50.394ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
4332026-07-29T21:29:50.394ZDEBGoximeter (collection-task): collecting from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
4342026-07-29T21:29:50.395ZDEBGoximeter (collection-task): collected results from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 14 producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
4352026-07-29T21:29:50.435ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = 609be78f-16d5-4f8a-b3dd-f5270fe1de68 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
4362026-07-29T21:29:50.435ZDEBGoximeter (collection-task): collecting from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = 609be78f-16d5-4f8a-b3dd-f5270fe1de68 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
4372026-07-29T21:29:50.436ZDEBGoximeter (collection-task): collected results from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 26 producer_id = 609be78f-16d5-4f8a-b3dd-f5270fe1de68 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
4382026-07-29T21:29:50.629ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = 7495615a-bf4a-4adb-8f22-76b6ec0eeaf0 producer_ip = fd00:1122:3344:101::c producer_port = 37508
4392026-07-29T21:29:50.629ZDEBGoximeter (collection-task): collecting from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = 7495615a-bf4a-4adb-8f22-76b6ec0eeaf0 producer_ip = fd00:1122:3344:101::c producer_port = 37508
4402026-07-29T21:29:50.634ZDEBGoximeter (collection-task): collected results from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d n_results = 5 n_samples = 50 producer_id = 7495615a-bf4a-4adb-8f22-76b6ec0eeaf0 producer_ip = fd00:1122:3344:101::c producer_port = 37508
4412026-07-29T21:29:51.394ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
4422026-07-29T21:29:51.394ZDEBGoximeter (collection-task): collecting from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
4432026-07-29T21:29:51.395ZDEBGoximeter (collection-task): collected results from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 14 producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
4442026-07-29T21:29:51.435ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = 609be78f-16d5-4f8a-b3dd-f5270fe1de68 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
4452026-07-29T21:29:51.435ZDEBGoximeter (collection-task): collecting from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = 609be78f-16d5-4f8a-b3dd-f5270fe1de68 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
4462026-07-29T21:29:51.436ZDEBGoximeter (collection-task): collected results from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 26 producer_id = 609be78f-16d5-4f8a-b3dd-f5270fe1de68 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
4472026-07-29T21:29:52.145ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = 6f2c93a7-3ce8-4cc1-8671-369f638cb080 producer_ip = fd00:1122:3344:101::2 producer_port = 37767
4482026-07-29T21:29:52.145ZDEBGoximeter (collection-task): collecting from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = 6f2c93a7-3ce8-4cc1-8671-369f638cb080 producer_ip = fd00:1122:3344:101::2 producer_port = 37767
4492026-07-29T21:29:52.147ZDEBGoximeter (collection-task): collected results from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d n_results = 2 n_samples = 11 producer_id = 6f2c93a7-3ce8-4cc1-8671-369f638cb080 producer_ip = fd00:1122:3344:101::2 producer_port = 37767
4502026-07-29T21:29:52.394ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
4512026-07-29T21:29:52.394ZDEBGoximeter (collection-task): collecting from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
4522026-07-29T21:29:52.395ZDEBGoximeter (collection-task): collected results from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 14 producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
4532026-07-29T21:29:52.435ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = 609be78f-16d5-4f8a-b3dd-f5270fe1de68 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
4542026-07-29T21:29:52.435ZDEBGoximeter (collection-task): collecting from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = 609be78f-16d5-4f8a-b3dd-f5270fe1de68 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
4552026-07-29T21:29:52.436ZDEBGoximeter (collection-task): collected results from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 26 producer_id = 609be78f-16d5-4f8a-b3dd-f5270fe1de68 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
4562026-07-29T21:29:52.843ZDEBGoximeter (collection-task): reporting oximeter self-collection statistics collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = 04808080-7e81-4ad5-99dd-16df9ab0b849 producer_ip = fd00:1122:3344:101::2 producer_port = 59901
4572026-07-29T21:29:53.352ZDEBGoximeter (database-inserter): inserting samples into database collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d n_samples = 271
4582026-07-29T21:29:53.352ZDEBGoximeter (database-inserter): inserting samples into database collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d n_samples = 271
4592026-07-29T21:29:53.352ZDEBGoximeter (clickhouse-client): unrolling 271 total samples collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21
4602026-07-29T21:29:53.352ZDEBGoximeter (clickhouse-client): fetching timeseries schema from database collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21
4612026-07-29T21:29:53.355ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21 schema = TimeseriesSchema { timeseries_name: TimeseriesName("collection_target:cpus_provisioned"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "id", field_type: Uuid, source: Target, description: "" }}, datum_type: I64, version: 1, authz_scope: Fleet, units: Count, created: 2026-07-29T21:29:53.352164217Z } timeseries_name = collection_target:cpus_provisioned
4622026-07-29T21:29:53.355ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21 schema = TimeseriesSchema { timeseries_name: TimeseriesName("collection_target:ram_provisioned"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "id", field_type: Uuid, source: Target, description: "" }}, datum_type: I64, version: 1, authz_scope: Fleet, units: Count, created: 2026-07-29T21:29:53.352165501Z } timeseries_name = collection_target:ram_provisioned
4632026-07-29T21:29:53.355ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21 schema = TimeseriesSchema { timeseries_name: TimeseriesName("collection_target:virtual_disk_space_provisioned"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "id", field_type: Uuid, source: Target, description: "" }}, datum_type: I64, version: 1, authz_scope: Fleet, units: Count, created: 2026-07-29T21:29:53.352166420Z } timeseries_name = collection_target:virtual_disk_space_provisioned
4642026-07-29T21:29:53.355ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21 schema = TimeseriesSchema { timeseries_name: TimeseriesName("database_transaction:retry_data"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "attempt", field_type: U32, source: Metric, description: "" }, FieldSchema { name: "name", field_type: String, source: Target, description: "" }}, datum_type: F64, version: 1, authz_scope: Fleet, units: Count, created: 2026-07-29T21:29:53.352127822Z } timeseries_name = database_transaction:retry_data
4652026-07-29T21:29:53.356ZDEBGoximeter (clickhouse-client): inserting 4 new timeseries schema collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21
4662026-07-29T21:29:53.359ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21 n_rows = 4 table_name = fields_ipaddr
4672026-07-29T21:29:53.360ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21 n_rows = 161 table_name = fields_string
4682026-07-29T21:29:53.362ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21 n_rows = 45 table_name = fields_u16
4692026-07-29T21:29:53.363ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21 n_rows = 1 table_name = fields_u32
4702026-07-29T21:29:53.364ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21 n_rows = 143 table_name = fields_uuid
4712026-07-29T21:29:53.365ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21 n_rows = 152 table_name = measurements_cumulativeu64
4722026-07-29T21:29:53.367ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21 n_rows = 1 table_name = measurements_f64
4732026-07-29T21:29:53.369ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21 n_rows = 41 table_name = measurements_histogramu64
4742026-07-29T21:29:53.370ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21 n_rows = 27 table_name = measurements_i64
4752026-07-29T21:29:53.371ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21 n_rows = 50 table_name = measurements_u64
4762026-07-29T21:29:53.395ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
4772026-07-29T21:29:53.395ZDEBGoximeter (collection-task): collecting from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
4782026-07-29T21:29:53.396ZDEBGoximeter (collection-task): collected results from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 14 producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
4792026-07-29T21:29:53.435ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = 609be78f-16d5-4f8a-b3dd-f5270fe1de68 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
4802026-07-29T21:29:53.435ZDEBGoximeter (collection-task): collecting from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = 609be78f-16d5-4f8a-b3dd-f5270fe1de68 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
4812026-07-29T21:29:53.436ZDEBGoximeter (collection-task): collected results from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 26 producer_id = 609be78f-16d5-4f8a-b3dd-f5270fe1de68 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
4822026-07-29T21:29:53.453ZWARNoximeter (database-inserter): failed to insert some results into metric DB collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:276
4832026-07-29T21:29:54.395ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
4842026-07-29T21:29:54.395ZDEBGoximeter (collection-task): collecting from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
4852026-07-29T21:29:54.396ZDEBGoximeter (collection-task): collected results from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 14 producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
4862026-07-29T21:29:54.435ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = 609be78f-16d5-4f8a-b3dd-f5270fe1de68 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
4872026-07-29T21:29:54.435ZDEBGoximeter (collection-task): collecting from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = 609be78f-16d5-4f8a-b3dd-f5270fe1de68 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
4882026-07-29T21:29:54.437ZDEBGoximeter (collection-task): collected results from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 26 producer_id = 609be78f-16d5-4f8a-b3dd-f5270fe1de68 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
4892026-07-29T21:29:55.394ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
4902026-07-29T21:29:55.394ZDEBGoximeter (collection-task): collecting from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
4912026-07-29T21:29:55.395ZDEBGoximeter (collection-task): collected results from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 14 producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
4922026-07-29T21:29:55.435ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = 609be78f-16d5-4f8a-b3dd-f5270fe1de68 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
4932026-07-29T21:29:55.435ZDEBGoximeter (collection-task): collecting from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = 609be78f-16d5-4f8a-b3dd-f5270fe1de68 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
4942026-07-29T21:29:55.437ZDEBGoximeter (collection-task): collected results from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 26 producer_id = 609be78f-16d5-4f8a-b3dd-f5270fe1de68 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
4952026-07-29T21:29:56.395ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
4962026-07-29T21:29:56.395ZDEBGoximeter (collection-task): collecting from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
4972026-07-29T21:29:56.396ZDEBGoximeter (collection-task): collected results from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 14 producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
4982026-07-29T21:29:56.435ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = 609be78f-16d5-4f8a-b3dd-f5270fe1de68 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
4992026-07-29T21:29:56.435ZDEBGoximeter (collection-task): collecting from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = 609be78f-16d5-4f8a-b3dd-f5270fe1de68 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
5002026-07-29T21:29:56.437ZDEBGoximeter (collection-task): collected results from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 26 producer_id = 609be78f-16d5-4f8a-b3dd-f5270fe1de68 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
5012026-07-29T21:29:57.394ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
5022026-07-29T21:29:57.394ZDEBGoximeter (collection-task): collecting from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
5032026-07-29T21:29:57.396ZDEBGoximeter (collection-task): collected results from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 14 producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
5042026-07-29T21:29:57.435ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = 609be78f-16d5-4f8a-b3dd-f5270fe1de68 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
5052026-07-29T21:29:57.435ZDEBGoximeter (collection-task): collecting from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = 609be78f-16d5-4f8a-b3dd-f5270fe1de68 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
5062026-07-29T21:29:57.436ZDEBGoximeter (collection-task): collected results from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 26 producer_id = 609be78f-16d5-4f8a-b3dd-f5270fe1de68 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
5072026-07-29T21:29:58.353ZDEBGoximeter (database-inserter): inserting samples into database collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d n_samples = 200
5082026-07-29T21:29:58.353ZDEBGoximeter (database-inserter): inserting samples into database collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d n_samples = 200
5092026-07-29T21:29:58.353ZDEBGoximeter (clickhouse-client): unrolling 200 total samples collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21
5102026-07-29T21:29:58.353ZDEBGoximeter (clickhouse-client): fetching timeseries schema from database collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21
5112026-07-29T21:29:58.358ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21 n_rows = 76 table_name = fields_string
5122026-07-29T21:29:58.359ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21 n_rows = 80 table_name = fields_uuid
5132026-07-29T21:29:58.360ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21 n_rows = 150 table_name = measurements_cumulativeu64
5142026-07-29T21:29:58.361ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d id = 23508800-ec0a-4945-b9e3-a3fdb28cba21 n_rows = 50 table_name = measurements_u64
5152026-07-29T21:29:58.395ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
5162026-07-29T21:29:58.395ZDEBGoximeter (collection-task): collecting from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
5172026-07-29T21:29:58.396ZDEBGoximeter (collection-task): collected results from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 14 producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
5182026-07-29T21:29:58.422ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = 8a6d7931-75ff-4c25-aab4-0b7211fc2ced producer_ip = fd00:1122:3344:101::a producer_port = 33085
5192026-07-29T21:29:58.422ZDEBGoximeter (collection-task): collecting from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = 8a6d7931-75ff-4c25-aab4-0b7211fc2ced producer_ip = fd00:1122:3344:101::a producer_port = 33085
5202026-07-29T21:29:58.423ZDEBGoximeter (collection-task): collected results from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d n_results = 5 n_samples = 5 producer_id = 8a6d7931-75ff-4c25-aab4-0b7211fc2ced producer_ip = fd00:1122:3344:101::a producer_port = 33085
5212026-07-29T21:29:58.435ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = 609be78f-16d5-4f8a-b3dd-f5270fe1de68 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
5222026-07-29T21:29:58.435ZDEBGoximeter (collection-task): collecting from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = 609be78f-16d5-4f8a-b3dd-f5270fe1de68 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
5232026-07-29T21:29:58.437ZDEBGoximeter (collection-task): collected results from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 26 producer_id = 609be78f-16d5-4f8a-b3dd-f5270fe1de68 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
5242026-07-29T21:29:58.453ZWARNoximeter (database-inserter): failed to insert some results into metric DB collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:276
5252026-07-29T21:29:58.465ZINFOoximeter (oximeter-agent): refreshing list of producers from Nexus collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d file = oximeter/collector/src/agent.rs:629
5262026-07-29T21:29:58.465ZDEBGoximeter: client request body = None method = GET uri = http://[fd00:1122:3344:101::a]:12221/metrics/collectors/fa84a12f-09b4-438f-8dee-6b54032162b4/producers?sort_by=id_ascending
5272026-07-29T21:29:58.469ZDEBGoximeter: client response result = Ok(Response { url: "http://[fd00:1122:3344:101::a]:12221/metrics/collectors/fa84a12f-09b4-438f-8dee-6b54032162b4/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "86a6e479-427c-47c3-b3de-8f5e0e430535", "content-length": "1271", "date": "Wed, 29 Jul 2026 21:29:58 GMT"} })
5282026-07-29T21:29:58.469ZDEBGoximeter: client request body = None method = GET uri = http://[fd00:1122:3344:101::a]:12221/metrics/collectors/fa84a12f-09b4-438f-8dee-6b54032162b4/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJjNGQ1NDczYS00MTA0LTRjYzQtYThlMy01MDE0ZDVhOTU4ZGMifX0%3D
5292026-07-29T21:29:58.472ZDEBGoximeter: client response result = Ok(Response { url: "http://[fd00:1122:3344:101::a]:12221/metrics/collectors/fa84a12f-09b4-438f-8dee-6b54032162b4/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJjNGQ1NDczYS00MTA0LTRjYzQtYThlMy01MDE0ZDVhOTU4ZGMifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "946b2d11-46bf-410c-a246-a0d64ab7a689", "content-length": "29", "date": "Wed, 29 Jul 2026 21:29:58 GMT"} })
5302026-07-29T21:29:58.472ZDEBGoximeter (oximeter-agent): registered new metric producer address = [fd00:1122:3344:101::2]:4677 collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = 4721f97d-d3dc-41fd-ad89-fc0a59cb66d8
5312026-07-29T21:29:58.472ZINFOoximeter (oximeter-agent): refreshed list of producers from Nexus collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d file = oximeter/collector/src/agent.rs:704 n_current_tasks = 8 n_pruned_tasks = 0
5322026-07-29T21:29:59.144ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = 4721f97d-d3dc-41fd-ad89-fc0a59cb66d8 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
5332026-07-29T21:29:59.144ZDEBGoximeter (collection-task): collecting from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = 4721f97d-d3dc-41fd-ad89-fc0a59cb66d8 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
5342026-07-29T21:29:59.152ZDEBGoximeter (collection-task): collected results from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 4 producer_id = 4721f97d-d3dc-41fd-ad89-fc0a59cb66d8 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
5352026-07-29T21:29:59.394ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
5362026-07-29T21:29:59.394ZDEBGoximeter (collection-task): collecting from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
5372026-07-29T21:29:59.395ZDEBGoximeter (collection-task): collected results from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 14 producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
5382026-07-29T21:29:59.435ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = 609be78f-16d5-4f8a-b3dd-f5270fe1de68 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
5392026-07-29T21:29:59.435ZDEBGoximeter (collection-task): collecting from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = 609be78f-16d5-4f8a-b3dd-f5270fe1de68 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
5402026-07-29T21:29:59.436ZDEBGoximeter (collection-task): collected results from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 26 producer_id = 609be78f-16d5-4f8a-b3dd-f5270fe1de68 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
5412026-07-29T21:29:59.724ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = c4d5473a-4104-4cc4-a8e3-5014d5a958dc producer_ip = fd00:1122:3344:101::b producer_port = 33382
5422026-07-29T21:29:59.724ZDEBGoximeter (collection-task): collecting from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = c4d5473a-4104-4cc4-a8e3-5014d5a958dc producer_ip = fd00:1122:3344:101::b producer_port = 33382
5432026-07-29T21:29:59.725ZDEBGoximeter (collection-task): collected results from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d n_results = 5 n_samples = 2 producer_id = c4d5473a-4104-4cc4-a8e3-5014d5a958dc producer_ip = fd00:1122:3344:101::b producer_port = 33382
5442026-07-29T21:30:00.144ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = 4721f97d-d3dc-41fd-ad89-fc0a59cb66d8 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
5452026-07-29T21:30:00.144ZDEBGoximeter (collection-task): collecting from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = 4721f97d-d3dc-41fd-ad89-fc0a59cb66d8 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
5462026-07-29T21:30:00.145ZDEBGoximeter (collection-task): collected results from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 4 producer_id = 4721f97d-d3dc-41fd-ad89-fc0a59cb66d8 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
5472026-07-29T21:30:00.394ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
5482026-07-29T21:30:00.394ZDEBGoximeter (collection-task): collecting from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
5492026-07-29T21:30:00.395ZDEBGoximeter (collection-task): collected results from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 14 producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
5502026-07-29T21:30:00.435ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = 609be78f-16d5-4f8a-b3dd-f5270fe1de68 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
5512026-07-29T21:30:00.435ZDEBGoximeter (collection-task): collecting from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = 609be78f-16d5-4f8a-b3dd-f5270fe1de68 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
5522026-07-29T21:30:00.437ZDEBGoximeter (collection-task): collected results from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 26 producer_id = 609be78f-16d5-4f8a-b3dd-f5270fe1de68 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
5532026-07-29T21:30:00.629ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = 7495615a-bf4a-4adb-8f22-76b6ec0eeaf0 producer_ip = fd00:1122:3344:101::c producer_port = 37508
5542026-07-29T21:30:00.629ZDEBGoximeter (collection-task): collecting from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = 7495615a-bf4a-4adb-8f22-76b6ec0eeaf0 producer_ip = fd00:1122:3344:101::c producer_port = 37508
5552026-07-29T21:30:00.632ZDEBGoximeter (collection-task): collected results from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d n_results = 5 n_samples = 23 producer_id = 7495615a-bf4a-4adb-8f22-76b6ec0eeaf0 producer_ip = fd00:1122:3344:101::c producer_port = 37508
5562026-07-29T21:30:01.144ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = 4721f97d-d3dc-41fd-ad89-fc0a59cb66d8 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
5572026-07-29T21:30:01.144ZDEBGoximeter (collection-task): collecting from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = 4721f97d-d3dc-41fd-ad89-fc0a59cb66d8 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
5582026-07-29T21:30:01.145ZDEBGoximeter (collection-task): collected results from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 4 producer_id = 4721f97d-d3dc-41fd-ad89-fc0a59cb66d8 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
5592026-07-29T21:30:01.256ZDEBGoximeter (collection-task): reporting oximeter self-collection statistics collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = c4d5473a-4104-4cc4-a8e3-5014d5a958dc producer_ip = fd00:1122:3344:101::b producer_port = 33382
5602026-07-29T21:30:01.394ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
5612026-07-29T21:30:01.394ZDEBGoximeter (collection-task): collecting from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
5622026-07-29T21:30:01.395ZDEBGoximeter (collection-task): collected results from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 14 producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
5632026-07-29T21:30:01.435ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = 609be78f-16d5-4f8a-b3dd-f5270fe1de68 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
5642026-07-29T21:30:01.435ZDEBGoximeter (collection-task): collecting from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = 609be78f-16d5-4f8a-b3dd-f5270fe1de68 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
5652026-07-29T21:30:01.437ZDEBGoximeter (collection-task): collected results from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 26 producer_id = 609be78f-16d5-4f8a-b3dd-f5270fe1de68 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
5662026-07-29T21:30:02.144ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = 4721f97d-d3dc-41fd-ad89-fc0a59cb66d8 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
5672026-07-29T21:30:02.144ZDEBGoximeter (collection-task): collecting from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = 4721f97d-d3dc-41fd-ad89-fc0a59cb66d8 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
5682026-07-29T21:30:02.145ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = 6f2c93a7-3ce8-4cc1-8671-369f638cb080 producer_ip = fd00:1122:3344:101::2 producer_port = 37767
5692026-07-29T21:30:02.145ZDEBGoximeter (collection-task): collecting from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = 6f2c93a7-3ce8-4cc1-8671-369f638cb080 producer_ip = fd00:1122:3344:101::2 producer_port = 37767
5702026-07-29T21:30:02.145ZDEBGoximeter (collection-task): collected results from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 4 producer_id = 4721f97d-d3dc-41fd-ad89-fc0a59cb66d8 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
5712026-07-29T21:30:02.147ZDEBGoximeter (collection-task): collected results from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d n_results = 2 n_samples = 11 producer_id = 6f2c93a7-3ce8-4cc1-8671-369f638cb080 producer_ip = fd00:1122:3344:101::2 producer_port = 37767
5722026-07-29T21:30:02.394ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
5732026-07-29T21:30:02.394ZDEBGoximeter (collection-task): collecting from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001
5742026-07-29T21:30:02.395ZDEBGoximeter (collection-task): collected results from producer collector_id = fa84a12f-09b4-438f-8dee-6b54032162b4 collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 14 producer_id = a53af4be-4c8f-4d00-a035-c514b457c9cc producer_ip = fd00:1122:3344:101::2 producer_port = 8001