Max level shown:
1 [ Jul 21 18:13:57 Enabled. ]
2 [ Jul 21 18:13:57 Rereading configuration. ]
3 [ Jul 21 18:13:58 Rereading configuration. ]
4 [ Jul 21 18:14:04 Executing start method ("ctrun -l child -o noorphan,regent /opt/oxide/oximeter-collector/bin/oximeter run /var/svc/manifest/site/oximeter/config.toml --address [fd00:1122:3344:101::d]:12223 --id 58217aed-c161-4122-ac86-7675725ee0be &"). ]
5 [ Jul 21 18:14:04 Method "start" exited with status 0. ]
6 note: configured to log to "/dev/stdout"
72026-07-21T18:14:04.878ZDEBGoximeter: registered DTrace probes
82026-07-21T18:14:04.879ZINFOoximeter: starting oximeter server file = oximeter/collector/src/lib.rs:299
92026-07-21T18:14:04.880ZDEBGoximeter: creating ClickHouse client
102026-07-21T18:14:04.899ZDEBGoximeter (clickhouse-client): registered USDT probes collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617
112026-07-21T18:14:07.450ZWARNoximeter (clickhouse-client): oximeter database does not exist, or is out-of-date collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d file = oximeter/db/src/client/mod.rs:888 id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617
122026-07-21T18:14:07.450ZDEBGoximeter (oximeter-agent): oximeter database does not exist, creating collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d
132026-07-21T18:14:07.450ZINFOoximeter (clickhouse-client): reading db version collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d file = oximeter/db/src/client/mod.rs:788 id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617
142026-07-21T18:14:07.451ZWARNoximeter (clickhouse-client): oximeter database does not exist, or is out-of-date collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d file = oximeter/db/src/client/mod.rs:888 id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617
152026-07-21T18:14:07.451ZINFOoximeter (clickhouse-client): read oximeter database version collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d file = oximeter/db/src/client/mod.rs:793 id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617 version = 0
162026-07-21T18:14:07.451ZINFOoximeter (clickhouse-client): initializing oximeter schema collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d file = oximeter/db/src/client/mod.rs:798 id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617
172026-07-21T18:14:07.451ZDEBGoximeter (clickhouse-client): initializing ClickHouse database collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617
182026-07-21T18:14:08.008ZINFOoximeter (clickhouse-client): inserting current version collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d file = oximeter/db/src/client/mod.rs:819 id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617 version = 14
192026-07-21T18:14:08.027ZDEBGoximeter (clickhouse-client): registered USDT probes collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = 2bb32db5-de7d-4275-891f-f05d19c6f8d2
202026-07-21T18:14:08.035ZDEBGoximeter (dropshot): registered endpoint local_addr = [fd00:1122:3344:101::d]:12223 method = GET path = /info versions = all
212026-07-21T18:14:08.035ZDEBGoximeter (dropshot): registered endpoint local_addr = [fd00:1122:3344:101::d]:12223 method = GET path = /producers versions = all
222026-07-21T18:14:08.035ZDEBGoximeter (dropshot): registered endpoint local_addr = [fd00:1122:3344:101::d]:12223 method = DELETE path = /producers/{producer_id} versions = all
232026-07-21T18:14:08.035ZDEBGoximeter (dropshot): registered endpoint local_addr = [fd00:1122:3344:101::d]:12223 method = GET path = /producers/{producer_id} versions = all
242026-07-21T18:14:08.035ZINFOoximeter (dropshot): listening file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:335 local_addr = [fd00:1122:3344:101::d]:12223
252026-07-21T18:14:08.038ZDEBGoximeter (dropshot): successfully registered DTrace USDT probes local_addr = [fd00:1122:3344:101::d]:12223
262026-07-21T18:14:08.038ZDEBGoximeter: registered USDT probes
272026-07-21T18:14:08.038ZDEBGoximeter: contacting nexus
282026-07-21T18:14:08.066ZDEBGoximeter: client request body = Some(Body) method = POST uri = http://[fd00:1122:3344:101::b]:12221/metrics/collectors
292026-07-21T18:14:08.067ZDEBGoximeter: client response result = Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::b]:12221/metrics/collectors", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", [fd00:1122:3344:101::b]:12221, Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
302026-07-21T18:14:08.067ZWARNoximeter: failed to contact nexus, will retry in 212.606242ms error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/collectors)" file = oximeter/collector/src/lib.rs:444
312026-07-21T18:14:08.280ZDEBGoximeter: contacting nexus
322026-07-21T18:14:08.280ZDEBGoximeter: client request body = Some(Body) method = POST uri = http://[fd00:1122:3344:101::a]:12221/metrics/collectors
332026-07-21T18:14:08.280ZDEBGoximeter: client response result = Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::a]:12221/metrics/collectors", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", [fd00:1122:3344:101::a]:12221, Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
342026-07-21T18:14:08.280ZWARNoximeter: failed to contact nexus, will retry in 731.855718ms error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/collectors)" file = oximeter/collector/src/lib.rs:444
352026-07-21T18:14:09.013ZDEBGoximeter: contacting nexus
362026-07-21T18:14:09.013ZDEBGoximeter: client request body = Some(Body) method = POST uri = http://[fd00:1122:3344:101::c]:12221/metrics/collectors
372026-07-21T18:14:09.014ZDEBGoximeter: client response result = Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::c]:12221/metrics/collectors", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", [fd00:1122:3344:101::c]:12221, Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
382026-07-21T18:14:09.014ZWARNoximeter: failed to contact nexus, will retry in 523.169237ms error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/collectors)" file = oximeter/collector/src/lib.rs:444
392026-07-21T18:14:09.538ZDEBGoximeter: contacting nexus
402026-07-21T18:14:09.538ZDEBGoximeter: client request body = Some(Body) method = POST uri = http://[fd00:1122:3344:101::a]:12221/metrics/collectors
412026-07-21T18:14:09.538ZDEBGoximeter: client response result = Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::a]:12221/metrics/collectors", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", [fd00:1122:3344:101::a]:12221, Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
422026-07-21T18:14:09.538ZWARNoximeter: failed to contact nexus, will retry in 1.920834907s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/collectors)" file = oximeter/collector/src/lib.rs:444
432026-07-21T18:14:11.460ZDEBGoximeter: contacting nexus
442026-07-21T18:14:11.460ZDEBGoximeter: client request body = Some(Body) method = POST uri = http://[fd00:1122:3344:101::c]:12221/metrics/collectors
452026-07-21T18:14:11.536ZDEBGoximeter: client response result = Ok(Response { url: "http://[fd00:1122:3344:101::c]:12221/metrics/collectors", status: 204, headers: {"x-request-id": "0dcaad85-34af-4d62-8d09-ffc620d8276c", "date": "Tue, 21 Jul 2026 18:14:11 GMT"} })
462026-07-21T18:14:11.537ZINFOoximeter: oximeter registered with nexus file = oximeter/collector/src/lib.rs:462 id = 58217aed-c161-4122-ac86-7675725ee0be
472026-07-21T18:14:11.537ZINFOoximeter (oximeter-agent): starting refresh list task collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d file = oximeter/collector/src/agent.rs:631
482026-07-21T18:14:11.538ZINFOoximeter (oximeter-agent): refreshing list of producers from Nexus collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d file = oximeter/collector/src/agent.rs:634
492026-07-21T18:14:11.544ZDEBGoximeter: client request body = None method = GET uri = http://[fd00:1122:3344:101::a]:12221/metrics/collectors/58217aed-c161-4122-ac86-7675725ee0be/producers?sort_by=id_ascending
502026-07-21T18:14:11.544ZDEBGoximeter: client response result = Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::a]:12221/metrics/collectors/58217aed-c161-4122-ac86-7675725ee0be/producers?sort_by=id_ascending", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", [fd00:1122:3344:101::a]:12221, Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
512026-07-21T18:14:11.545ZERROoximeter (oximeter-agent): error fetching next assigned producer, abandoning this refresh attempt collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d error = Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/collectors/58217aed-c161-4122-ac86-7675725ee0be/producers?sort_by=id_ascending): error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/collectors/58217aed-c161-4122-ac86-7675725ee0be/producers?sort_by=id_ascending): client error (Connect): tcp connect error: Connection refused (os error 146) file = oximeter/collector/src/agent.rs:672
522026-07-21T18:14:13.027ZDEBGoximeter (database-inserter): inserting samples into database collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d n_samples = 4
532026-07-21T18:14:13.027ZDEBGoximeter (clickhouse-client): unrolling 4 total samples collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617
542026-07-21T18:14:13.028ZDEBGoximeter (clickhouse-client): fetching timeseries schema from database collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617
552026-07-21T18:14:13.028ZDEBGoximeter (database-inserter): inserting samples into database collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d n_samples = 4
562026-07-21T18:14:13.030ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617 schema = TimeseriesSchema { timeseries_name: TimeseriesName("oximeter_collector:database_queue_depth"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "collector_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "collector_ip", field_type: IpAddr, source: Target, description: "" }, FieldSchema { name: "collector_port", field_type: U16, source: Target, description: "" }, FieldSchema { name: "collector_sink", field_type: String, source: Metric, description: "" }}, datum_type: HistogramU64, version: 1, authz_scope: Fleet, units: Count, created: 2026-07-21T18:14:13.028283511Z } timeseries_name = oximeter_collector:database_queue_depth
572026-07-21T18:14:13.030ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617 schema = TimeseriesSchema { timeseries_name: TimeseriesName("oximeter_collector:database_samples_dropped"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "collector_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "collector_ip", field_type: IpAddr, source: Target, description: "" }, FieldSchema { name: "collector_port", field_type: U16, source: Target, description: "" }, FieldSchema { name: "collector_sink", field_type: String, source: Metric, description: "" }}, datum_type: CumulativeU64, version: 1, authz_scope: Fleet, units: Count, created: 2026-07-21T18:14:13.028279120Z } timeseries_name = oximeter_collector:database_samples_dropped
582026-07-21T18:14:13.030ZDEBGoximeter (clickhouse-client): inserting 2 new timeseries schema collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617
592026-07-21T18:14:13.036ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617 n_rows = 4 table_name = fields_ipaddr
602026-07-21T18:14:13.038ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617 n_rows = 4 table_name = fields_string
612026-07-21T18:14:13.040ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617 n_rows = 4 table_name = fields_u16
622026-07-21T18:14:13.043ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617 n_rows = 4 table_name = fields_uuid
632026-07-21T18:14:13.045ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617 n_rows = 2 table_name = measurements_cumulativeu64
642026-07-21T18:14:13.048ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617 n_rows = 2 table_name = measurements_histogramu64
652026-07-21T18:14:13.129ZWARNoximeter (database-inserter): failed to insert some results into metric DB collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:276
662026-07-21T18:14:18.028ZDEBGoximeter (database-inserter): batch interval expired, but no samples to insert collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d
672026-07-21T18:14:18.029ZDEBGoximeter (database-inserter): batch interval expired, but no samples to insert collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d
682026-07-21T18:14:23.027ZDEBGoximeter (database-inserter): batch interval expired, but no samples to insert collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d
692026-07-21T18:14:23.028ZDEBGoximeter (database-inserter): batch interval expired, but no samples to insert collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d
702026-07-21T18:14:26.537ZINFOoximeter (oximeter-agent): refreshing list of producers from Nexus collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d file = oximeter/collector/src/agent.rs:634
712026-07-21T18:14:26.537ZDEBGoximeter: client request body = None method = GET uri = http://[fd00:1122:3344:101::c]:12221/metrics/collectors/58217aed-c161-4122-ac86-7675725ee0be/producers?sort_by=id_ascending
722026-07-21T18:14:26.543ZDEBGoximeter: client response result = Ok(Response { url: "http://[fd00:1122:3344:101::c]:12221/metrics/collectors/58217aed-c161-4122-ac86-7675725ee0be/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "a1957581-3862-4156-b821-11969eaf859c", "content-length": "572", "date": "Tue, 21 Jul 2026 18:14:26 GMT"} })
732026-07-21T18:14:26.545ZDEBGoximeter: client request body = None method = GET uri = http://[fd00:1122:3344:101::c]:12221/metrics/collectors/58217aed-c161-4122-ac86-7675725ee0be/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJlMTgxMWNhZC00MGFiLTQyY2ItYTFhMi01NTdhN2U0ZmEyYjkifX0%3D
742026-07-21T18:14:26.549ZDEBGoximeter: client response result = Ok(Response { url: "http://[fd00:1122:3344:101::c]:12221/metrics/collectors/58217aed-c161-4122-ac86-7675725ee0be/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJlMTgxMWNhZC00MGFiLTQyY2ItYTFhMi01NTdhN2U0ZmEyYjkifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "9d386d07-2ecd-45cc-bafb-35d62138bb3e", "content-length": "29", "date": "Tue, 21 Jul 2026 18:14:26 GMT"} })
752026-07-21T18:14:26.549ZDEBGoximeter (oximeter-agent): registered new metric producer address = [fd00:1122:3344:101::2]:8001 collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = 80daec9f-82ff-4a1e-a3a8-8d4bda5887d1
762026-07-21T18:14:26.549ZDEBGoximeter (oximeter-agent): registered new metric producer address = [fd00:1122:3344:101::2]:43808 collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = 85d0356f-079a-42e7-9fd9-5e576eef1fdd
772026-07-21T18:14:26.549ZDEBGoximeter (oximeter-agent): registered new metric producer address = [fd00:1122:3344:101::1]:8001 collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = e1811cad-40ab-42cb-a1a2-557a7e4fa2b9
782026-07-21T18:14:26.549ZINFOoximeter (oximeter-agent): refreshed list of producers from Nexus collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d file = oximeter/collector/src/agent.rs:709 n_current_tasks = 3 n_pruned_tasks = 0
792026-07-21T18:14:26.745ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = 80daec9f-82ff-4a1e-a3a8-8d4bda5887d1 producer_ip = fd00:1122:3344:101::2 producer_port = 8001
802026-07-21T18:14:26.745ZDEBGoximeter (collection-task): collecting from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = 80daec9f-82ff-4a1e-a3a8-8d4bda5887d1 producer_ip = fd00:1122:3344:101::2 producer_port = 8001
812026-07-21T18:14:26.757ZDEBGoximeter (collection-task): collected results from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 14 producer_id = 80daec9f-82ff-4a1e-a3a8-8d4bda5887d1 producer_ip = fd00:1122:3344:101::2 producer_port = 8001
822026-07-21T18:14:27.233ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = e1811cad-40ab-42cb-a1a2-557a7e4fa2b9 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
832026-07-21T18:14:27.233ZDEBGoximeter (collection-task): collecting from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = e1811cad-40ab-42cb-a1a2-557a7e4fa2b9 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
842026-07-21T18:14:27.239ZDEBGoximeter (collection-task): collected results from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 26 producer_id = e1811cad-40ab-42cb-a1a2-557a7e4fa2b9 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
852026-07-21T18:14:27.745ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = 80daec9f-82ff-4a1e-a3a8-8d4bda5887d1 producer_ip = fd00:1122:3344:101::2 producer_port = 8001
862026-07-21T18:14:27.745ZDEBGoximeter (collection-task): collecting from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = 80daec9f-82ff-4a1e-a3a8-8d4bda5887d1 producer_ip = fd00:1122:3344:101::2 producer_port = 8001
872026-07-21T18:14:27.746ZDEBGoximeter (collection-task): collected results from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 14 producer_id = 80daec9f-82ff-4a1e-a3a8-8d4bda5887d1 producer_ip = fd00:1122:3344:101::2 producer_port = 8001
882026-07-21T18:14:28.027ZDEBGoximeter (database-inserter): inserting samples into database collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d n_samples = 54
892026-07-21T18:14:28.027ZDEBGoximeter (clickhouse-client): unrolling 54 total samples collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617
902026-07-21T18:14:28.027ZDEBGoximeter (clickhouse-client): fetching timeseries schema from database collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617
912026-07-21T18:14:28.028ZDEBGoximeter (database-inserter): inserting samples into database collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d n_samples = 54
922026-07-21T18:14:28.029ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617 schema = TimeseriesSchema { timeseries_name: TimeseriesName("ddm_router:originated_tunnel_endpoints"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "hostname", field_type: String, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }}, datum_type: U64, version: 1, authz_scope: Fleet, units: Count, created: 2026-07-21T18:14:28.027813084Z } timeseries_name = ddm_router:originated_tunnel_endpoints
932026-07-21T18:14:28.029ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617 schema = TimeseriesSchema { timeseries_name: TimeseriesName("ddm_router:originated_underlay_prefixes"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "hostname", field_type: String, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }}, datum_type: U64, version: 1, authz_scope: Fleet, units: Count, created: 2026-07-21T18:14:28.027808654Z } timeseries_name = ddm_router:originated_underlay_prefixes
942026-07-21T18:14:28.029ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617 schema = TimeseriesSchema { timeseries_name: TimeseriesName("ddm_session:advertisements_received"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "hostname", field_type: String, source: Target, description: "" }, FieldSchema { name: "interface", field_type: String, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }}, datum_type: CumulativeU64, version: 1, authz_scope: Fleet, units: Count, created: 2026-07-21T18:14:28.027821094Z } timeseries_name = ddm_session:advertisements_received
952026-07-21T18:14:28.029ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617 schema = TimeseriesSchema { timeseries_name: TimeseriesName("ddm_session:advertisements_sent"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "hostname", field_type: String, source: Target, description: "" }, FieldSchema { name: "interface", field_type: String, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }}, datum_type: CumulativeU64, version: 1, authz_scope: Fleet, units: Count, created: 2026-07-21T18:14:28.027819124Z } timeseries_name = ddm_session:advertisements_sent
962026-07-21T18:14:28.029ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617 schema = TimeseriesSchema { timeseries_name: TimeseriesName("ddm_session:imported_tunnel_endpoints"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "hostname", field_type: String, source: Target, description: "" }, FieldSchema { name: "interface", field_type: String, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }}, datum_type: U64, version: 1, authz_scope: Fleet, units: Count, created: 2026-07-21T18:14:28.027835154Z } timeseries_name = ddm_session:imported_tunnel_endpoints
972026-07-21T18:14:28.029ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617 schema = TimeseriesSchema { timeseries_name: TimeseriesName("ddm_session:imported_underlay_prefixes"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "hostname", field_type: String, source: Target, description: "" }, FieldSchema { name: "interface", field_type: String, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }}, datum_type: U64, version: 1, authz_scope: Fleet, units: Count, created: 2026-07-21T18:14:28.027833674Z } timeseries_name = ddm_session:imported_underlay_prefixes
982026-07-21T18:14:28.029ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617 schema = TimeseriesSchema { timeseries_name: TimeseriesName("ddm_session:peer_address_changes"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "hostname", field_type: String, source: Target, description: "" }, FieldSchema { name: "interface", field_type: String, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }}, datum_type: CumulativeU64, version: 1, authz_scope: Fleet, units: Count, created: 2026-07-21T18:14:28.027825114Z } timeseries_name = ddm_session:peer_address_changes
992026-07-21T18:14:28.030ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617 schema = TimeseriesSchema { timeseries_name: TimeseriesName("ddm_session:peer_expirations"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "hostname", field_type: String, source: Target, description: "" }, FieldSchema { name: "interface", field_type: String, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }}, datum_type: CumulativeU64, version: 1, authz_scope: Fleet, units: Count, created: 2026-07-21T18:14:28.027823014Z } timeseries_name = ddm_session:peer_expirations
1002026-07-21T18:14:28.030ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617 schema = TimeseriesSchema { timeseries_name: TimeseriesName("ddm_session:peer_sessions_established"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "hostname", field_type: String, source: Target, description: "" }, FieldSchema { name: "interface", field_type: String, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }}, datum_type: CumulativeU64, version: 1, authz_scope: Fleet, units: Count, created: 2026-07-21T18:14:28.027826624Z } timeseries_name = ddm_session:peer_sessions_established
1012026-07-21T18:14:28.030ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617 schema = TimeseriesSchema { timeseries_name: TimeseriesName("ddm_session:solicitations_received"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "hostname", field_type: String, source: Target, description: "" }, FieldSchema { name: "interface", field_type: String, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }}, datum_type: CumulativeU64, version: 1, authz_scope: Fleet, units: Count, created: 2026-07-21T18:14:28.027817164Z } timeseries_name = ddm_session:solicitations_received
1022026-07-21T18:14:28.030ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617 schema = TimeseriesSchema { timeseries_name: TimeseriesName("ddm_session:solicitations_sent"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "hostname", field_type: String, source: Target, description: "" }, FieldSchema { name: "interface", field_type: String, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }}, datum_type: CumulativeU64, version: 1, authz_scope: Fleet, units: Count, created: 2026-07-21T18:14:28.027815504Z } timeseries_name = ddm_session:solicitations_sent
1032026-07-21T18:14:28.030ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617 schema = TimeseriesSchema { timeseries_name: TimeseriesName("ddm_session:update_send_fail"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "hostname", field_type: String, source: Target, description: "" }, FieldSchema { name: "interface", field_type: String, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }}, datum_type: CumulativeU64, version: 1, authz_scope: Fleet, units: Count, created: 2026-07-21T18:14:28.027830764Z } timeseries_name = ddm_session:update_send_fail
1042026-07-21T18:14:28.030ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617 schema = TimeseriesSchema { timeseries_name: TimeseriesName("ddm_session:updates_received"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "hostname", field_type: String, source: Target, description: "" }, FieldSchema { name: "interface", field_type: String, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }}, datum_type: CumulativeU64, version: 1, authz_scope: Fleet, units: Count, created: 2026-07-21T18:14:28.027829614Z } timeseries_name = ddm_session:updates_received
1052026-07-21T18:14:28.030ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617 schema = TimeseriesSchema { timeseries_name: TimeseriesName("ddm_session:updates_sent"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "hostname", field_type: String, source: Target, description: "" }, FieldSchema { name: "interface", field_type: String, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }}, datum_type: CumulativeU64, version: 1, authz_scope: Fleet, units: Count, created: 2026-07-21T18:14:28.027828384Z } timeseries_name = ddm_session:updates_sent
1062026-07-21T18:14:28.030ZDEBGoximeter (clickhouse-client): inserting 14 new timeseries schema collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617
1072026-07-21T18:14:28.033ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617 n_rows = 76 table_name = fields_string
1082026-07-21T18:14:28.035ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617 n_rows = 80 table_name = fields_uuid
1092026-07-21T18:14:28.036ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617 n_rows = 40 table_name = measurements_cumulativeu64
1102026-07-21T18:14:28.038ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617 n_rows = 14 table_name = measurements_u64
1112026-07-21T18:14:28.129ZWARNoximeter (database-inserter): failed to insert some results into metric DB collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:276
1122026-07-21T18:14:28.233ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = e1811cad-40ab-42cb-a1a2-557a7e4fa2b9 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
1132026-07-21T18:14:28.233ZDEBGoximeter (collection-task): collecting from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = e1811cad-40ab-42cb-a1a2-557a7e4fa2b9 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
1142026-07-21T18:14:28.235ZDEBGoximeter (collection-task): collected results from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 26 producer_id = e1811cad-40ab-42cb-a1a2-557a7e4fa2b9 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
1152026-07-21T18:14:28.745ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = 80daec9f-82ff-4a1e-a3a8-8d4bda5887d1 producer_ip = fd00:1122:3344:101::2 producer_port = 8001
1162026-07-21T18:14:28.745ZDEBGoximeter (collection-task): collecting from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = 80daec9f-82ff-4a1e-a3a8-8d4bda5887d1 producer_ip = fd00:1122:3344:101::2 producer_port = 8001
1172026-07-21T18:14:28.746ZDEBGoximeter (collection-task): collected results from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 14 producer_id = 80daec9f-82ff-4a1e-a3a8-8d4bda5887d1 producer_ip = fd00:1122:3344:101::2 producer_port = 8001
1182026-07-21T18:14:29.233ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = e1811cad-40ab-42cb-a1a2-557a7e4fa2b9 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
1192026-07-21T18:14:29.233ZDEBGoximeter (collection-task): collecting from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = e1811cad-40ab-42cb-a1a2-557a7e4fa2b9 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
1202026-07-21T18:14:29.235ZDEBGoximeter (collection-task): collected results from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 26 producer_id = e1811cad-40ab-42cb-a1a2-557a7e4fa2b9 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
1212026-07-21T18:14:29.744ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = 80daec9f-82ff-4a1e-a3a8-8d4bda5887d1 producer_ip = fd00:1122:3344:101::2 producer_port = 8001
1222026-07-21T18:14:29.744ZDEBGoximeter (collection-task): collecting from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = 80daec9f-82ff-4a1e-a3a8-8d4bda5887d1 producer_ip = fd00:1122:3344:101::2 producer_port = 8001
1232026-07-21T18:14:29.746ZDEBGoximeter (collection-task): collected results from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 14 producer_id = 80daec9f-82ff-4a1e-a3a8-8d4bda5887d1 producer_ip = fd00:1122:3344:101::2 producer_port = 8001
1242026-07-21T18:14:30.234ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = e1811cad-40ab-42cb-a1a2-557a7e4fa2b9 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
1252026-07-21T18:14:30.234ZDEBGoximeter (collection-task): collecting from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = e1811cad-40ab-42cb-a1a2-557a7e4fa2b9 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
1262026-07-21T18:14:30.236ZDEBGoximeter (collection-task): collected results from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 26 producer_id = e1811cad-40ab-42cb-a1a2-557a7e4fa2b9 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
1272026-07-21T18:14:30.744ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = 80daec9f-82ff-4a1e-a3a8-8d4bda5887d1 producer_ip = fd00:1122:3344:101::2 producer_port = 8001
1282026-07-21T18:14:30.744ZDEBGoximeter (collection-task): collecting from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = 80daec9f-82ff-4a1e-a3a8-8d4bda5887d1 producer_ip = fd00:1122:3344:101::2 producer_port = 8001
1292026-07-21T18:14:30.746ZDEBGoximeter (collection-task): collected results from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 14 producer_id = 80daec9f-82ff-4a1e-a3a8-8d4bda5887d1 producer_ip = fd00:1122:3344:101::2 producer_port = 8001
1302026-07-21T18:14:31.234ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = e1811cad-40ab-42cb-a1a2-557a7e4fa2b9 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
1312026-07-21T18:14:31.234ZDEBGoximeter (collection-task): collecting from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = e1811cad-40ab-42cb-a1a2-557a7e4fa2b9 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
1322026-07-21T18:14:31.236ZDEBGoximeter (collection-task): collected results from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 26 producer_id = e1811cad-40ab-42cb-a1a2-557a7e4fa2b9 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
1332026-07-21T18:14:31.744ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = 80daec9f-82ff-4a1e-a3a8-8d4bda5887d1 producer_ip = fd00:1122:3344:101::2 producer_port = 8001
1342026-07-21T18:14:31.744ZDEBGoximeter (collection-task): collecting from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = 80daec9f-82ff-4a1e-a3a8-8d4bda5887d1 producer_ip = fd00:1122:3344:101::2 producer_port = 8001
1352026-07-21T18:14:31.746ZDEBGoximeter (collection-task): collected results from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 14 producer_id = 80daec9f-82ff-4a1e-a3a8-8d4bda5887d1 producer_ip = fd00:1122:3344:101::2 producer_port = 8001
1362026-07-21T18:14:31.750ZDEBGoximeter (collection-task): reporting oximeter self-collection statistics collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = 80daec9f-82ff-4a1e-a3a8-8d4bda5887d1 producer_ip = fd00:1122:3344:101::2 producer_port = 8001
1372026-07-21T18:14:32.234ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = e1811cad-40ab-42cb-a1a2-557a7e4fa2b9 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
1382026-07-21T18:14:32.234ZDEBGoximeter (collection-task): collecting from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = e1811cad-40ab-42cb-a1a2-557a7e4fa2b9 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
1392026-07-21T18:14:32.236ZDEBGoximeter (collection-task): collected results from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 26 producer_id = e1811cad-40ab-42cb-a1a2-557a7e4fa2b9 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
1402026-07-21T18:14:32.745ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = 80daec9f-82ff-4a1e-a3a8-8d4bda5887d1 producer_ip = fd00:1122:3344:101::2 producer_port = 8001
1412026-07-21T18:14:32.745ZDEBGoximeter (collection-task): collecting from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = 80daec9f-82ff-4a1e-a3a8-8d4bda5887d1 producer_ip = fd00:1122:3344:101::2 producer_port = 8001
1422026-07-21T18:14:32.746ZDEBGoximeter (collection-task): collected results from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 14 producer_id = 80daec9f-82ff-4a1e-a3a8-8d4bda5887d1 producer_ip = fd00:1122:3344:101::2 producer_port = 8001
1432026-07-21T18:14:33.027ZDEBGoximeter (database-inserter): inserting samples into database collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d n_samples = 202
1442026-07-21T18:14:33.027ZDEBGoximeter (clickhouse-client): unrolling 202 total samples collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617
1452026-07-21T18:14:33.027ZDEBGoximeter (clickhouse-client): fetching timeseries schema from database collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617
1462026-07-21T18:14:33.028ZDEBGoximeter (database-inserter): inserting samples into database collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d n_samples = 202
1472026-07-21T18:14:33.043ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617 schema = TimeseriesSchema { timeseries_name: TimeseriesName("oximeter_collector:collections"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "base_route", field_type: String, source: Metric, description: "" }, FieldSchema { name: "collector_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "collector_ip", field_type: IpAddr, source: Target, description: "" }, FieldSchema { name: "collector_port", field_type: U16, source: Target, description: "" }, FieldSchema { name: "producer_id", field_type: Uuid, source: Metric, description: "" }, FieldSchema { name: "producer_ip", field_type: IpAddr, source: Metric, description: "" }, FieldSchema { name: "producer_port", field_type: U16, source: Metric, description: "" }}, datum_type: CumulativeU64, version: 1, authz_scope: Fleet, units: Count, created: 2026-07-21T18:14:33.027874644Z } timeseries_name = oximeter_collector:collections
1482026-07-21T18:14:33.043ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617 schema = TimeseriesSchema { timeseries_name: TimeseriesName("oximeter_collector:samples_collected"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "base_route", field_type: String, source: Metric, description: "" }, FieldSchema { name: "collector_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "collector_ip", field_type: IpAddr, source: Target, description: "" }, FieldSchema { name: "collector_port", field_type: U16, source: Target, description: "" }, FieldSchema { name: "producer_id", field_type: Uuid, source: Metric, description: "" }, FieldSchema { name: "producer_ip", field_type: IpAddr, source: Metric, description: "" }, FieldSchema { name: "producer_port", field_type: U16, source: Metric, description: "" }}, datum_type: CumulativeU64, version: 1, authz_scope: Fleet, units: Count, created: 2026-07-21T18:14:33.027877714Z } timeseries_name = oximeter_collector:samples_collected
1492026-07-21T18:14:33.043ZDEBGoximeter (clickhouse-client): inserting 2 new timeseries schema collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617
1502026-07-21T18:14:33.047ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617 n_rows = 4 table_name = fields_ipaddr
1512026-07-21T18:14:33.049ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617 n_rows = 78 table_name = fields_string
1522026-07-21T18:14:33.051ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617 n_rows = 4 table_name = fields_u16
1532026-07-21T18:14:33.053ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617 n_rows = 84 table_name = fields_uuid
1542026-07-21T18:14:33.054ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617 n_rows = 152 table_name = measurements_cumulativeu64
1552026-07-21T18:14:33.058ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617 n_rows = 50 table_name = measurements_u64
1562026-07-21T18:14:33.129ZWARNoximeter (database-inserter): failed to insert some results into metric DB collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:276
1572026-07-21T18:14:33.234ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = e1811cad-40ab-42cb-a1a2-557a7e4fa2b9 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
1582026-07-21T18:14:33.234ZDEBGoximeter (collection-task): collecting from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = e1811cad-40ab-42cb-a1a2-557a7e4fa2b9 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
1592026-07-21T18:14:33.235ZDEBGoximeter (collection-task): collected results from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 26 producer_id = e1811cad-40ab-42cb-a1a2-557a7e4fa2b9 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
1602026-07-21T18:14:33.744ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = 80daec9f-82ff-4a1e-a3a8-8d4bda5887d1 producer_ip = fd00:1122:3344:101::2 producer_port = 8001
1612026-07-21T18:14:33.744ZDEBGoximeter (collection-task): collecting from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = 80daec9f-82ff-4a1e-a3a8-8d4bda5887d1 producer_ip = fd00:1122:3344:101::2 producer_port = 8001
1622026-07-21T18:14:33.745ZDEBGoximeter (collection-task): collected results from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 14 producer_id = 80daec9f-82ff-4a1e-a3a8-8d4bda5887d1 producer_ip = fd00:1122:3344:101::2 producer_port = 8001
1632026-07-21T18:14:34.233ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = e1811cad-40ab-42cb-a1a2-557a7e4fa2b9 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
1642026-07-21T18:14:34.233ZDEBGoximeter (collection-task): collecting from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = e1811cad-40ab-42cb-a1a2-557a7e4fa2b9 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
1652026-07-21T18:14:34.236ZDEBGoximeter (collection-task): collected results from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 26 producer_id = e1811cad-40ab-42cb-a1a2-557a7e4fa2b9 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
1662026-07-21T18:14:34.745ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = 80daec9f-82ff-4a1e-a3a8-8d4bda5887d1 producer_ip = fd00:1122:3344:101::2 producer_port = 8001
1672026-07-21T18:14:34.745ZDEBGoximeter (collection-task): collecting from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = 80daec9f-82ff-4a1e-a3a8-8d4bda5887d1 producer_ip = fd00:1122:3344:101::2 producer_port = 8001
1682026-07-21T18:14:34.746ZDEBGoximeter (collection-task): collected results from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 14 producer_id = 80daec9f-82ff-4a1e-a3a8-8d4bda5887d1 producer_ip = fd00:1122:3344:101::2 producer_port = 8001
1692026-07-21T18:14:35.234ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = e1811cad-40ab-42cb-a1a2-557a7e4fa2b9 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
1702026-07-21T18:14:35.234ZDEBGoximeter (collection-task): collecting from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = e1811cad-40ab-42cb-a1a2-557a7e4fa2b9 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
1712026-07-21T18:14:35.235ZDEBGoximeter (collection-task): collected results from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 26 producer_id = e1811cad-40ab-42cb-a1a2-557a7e4fa2b9 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
1722026-07-21T18:14:35.642ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = 85d0356f-079a-42e7-9fd9-5e576eef1fdd producer_ip = fd00:1122:3344:101::2 producer_port = 43808
1732026-07-21T18:14:35.643ZDEBGoximeter (collection-task): collecting from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = 85d0356f-079a-42e7-9fd9-5e576eef1fdd producer_ip = fd00:1122:3344:101::2 producer_port = 43808
1742026-07-21T18:14:35.672ZDEBGoximeter (collection-task): collected results from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 140 producer_id = 85d0356f-079a-42e7-9fd9-5e576eef1fdd producer_ip = fd00:1122:3344:101::2 producer_port = 43808
1752026-07-21T18:14:35.745ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = 80daec9f-82ff-4a1e-a3a8-8d4bda5887d1 producer_ip = fd00:1122:3344:101::2 producer_port = 8001
1762026-07-21T18:14:35.745ZDEBGoximeter (collection-task): collecting from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = 80daec9f-82ff-4a1e-a3a8-8d4bda5887d1 producer_ip = fd00:1122:3344:101::2 producer_port = 8001
1772026-07-21T18:14:35.746ZDEBGoximeter (collection-task): collected results from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 14 producer_id = 80daec9f-82ff-4a1e-a3a8-8d4bda5887d1 producer_ip = fd00:1122:3344:101::2 producer_port = 8001
1782026-07-21T18:14:36.233ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = e1811cad-40ab-42cb-a1a2-557a7e4fa2b9 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
1792026-07-21T18:14:36.233ZDEBGoximeter (collection-task): collecting from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = e1811cad-40ab-42cb-a1a2-557a7e4fa2b9 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
1802026-07-21T18:14:36.235ZDEBGoximeter (collection-task): collected results from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 26 producer_id = e1811cad-40ab-42cb-a1a2-557a7e4fa2b9 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
1812026-07-21T18:14:36.745ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = 80daec9f-82ff-4a1e-a3a8-8d4bda5887d1 producer_ip = fd00:1122:3344:101::2 producer_port = 8001
1822026-07-21T18:14:36.745ZDEBGoximeter (collection-task): collecting from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = 80daec9f-82ff-4a1e-a3a8-8d4bda5887d1 producer_ip = fd00:1122:3344:101::2 producer_port = 8001
1832026-07-21T18:14:36.746ZDEBGoximeter (collection-task): collected results from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 14 producer_id = 80daec9f-82ff-4a1e-a3a8-8d4bda5887d1 producer_ip = fd00:1122:3344:101::2 producer_port = 8001
1842026-07-21T18:14:37.234ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = e1811cad-40ab-42cb-a1a2-557a7e4fa2b9 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
1852026-07-21T18:14:37.234ZDEBGoximeter (collection-task): collecting from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = e1811cad-40ab-42cb-a1a2-557a7e4fa2b9 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
1862026-07-21T18:14:37.235ZDEBGoximeter (collection-task): collected results from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 26 producer_id = e1811cad-40ab-42cb-a1a2-557a7e4fa2b9 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
1872026-07-21T18:14:37.745ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = 80daec9f-82ff-4a1e-a3a8-8d4bda5887d1 producer_ip = fd00:1122:3344:101::2 producer_port = 8001
1882026-07-21T18:14:37.745ZDEBGoximeter (collection-task): collecting from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = 80daec9f-82ff-4a1e-a3a8-8d4bda5887d1 producer_ip = fd00:1122:3344:101::2 producer_port = 8001
1892026-07-21T18:14:37.746ZDEBGoximeter (collection-task): collected results from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 14 producer_id = 80daec9f-82ff-4a1e-a3a8-8d4bda5887d1 producer_ip = fd00:1122:3344:101::2 producer_port = 8001
1902026-07-21T18:14:38.028ZDEBGoximeter (database-inserter): inserting samples into database collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d n_samples = 340
1912026-07-21T18:14:38.028ZDEBGoximeter (clickhouse-client): unrolling 340 total samples collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617
1922026-07-21T18:14:38.028ZDEBGoximeter (clickhouse-client): fetching timeseries schema from database collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617
1932026-07-21T18:14:38.029ZDEBGoximeter (database-inserter): inserting samples into database collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d n_samples = 340
1942026-07-21T18:14:38.031ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617 schema = TimeseriesSchema { timeseries_name: TimeseriesName("dendrite:sample_collection_duration"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "asic_fab", field_type: String, source: Metric, description: "" }, FieldSchema { name: "asic_lot", field_type: String, source: Metric, description: "" }, FieldSchema { name: "asic_wafer", field_type: U8, source: Metric, description: "" }, FieldSchema { name: "asic_wafer_loc_x", field_type: I16, source: Metric, description: "" }, FieldSchema { name: "asic_wafer_loc_y", field_type: I16, source: Metric, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_model", field_type: String, source: Target, description: "" }, FieldSchema { name: "sled_revision", field_type: U32, source: Target, description: "" }, FieldSchema { name: "sled_serial", field_type: String, source: Target, description: "" }, FieldSchema { name: "switch_id", field_type: Uuid, source: Metric, description: "" }, FieldSchema { name: "switch_model", field_type: String, source: Metric, description: "" }, FieldSchema { name: "switch_revision", field_type: U32, source: Metric, description: "" }, FieldSchema { name: "switch_serial", field_type: String, source: Metric, description: "" }, FieldSchema { name: "switch_slot", field_type: U16, source: Metric, description: "" }}, datum_type: F64, version: 1, authz_scope: Fleet, units: Count, created: 2026-07-21T18:14:38.028490194Z } timeseries_name = dendrite:sample_collection_duration
1952026-07-21T18:14:38.031ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617 schema = TimeseriesSchema { timeseries_name: TimeseriesName("switch_data_link:link_enabled"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "asic_fab", field_type: String, source: Target, description: "" }, FieldSchema { name: "asic_lot", field_type: String, source: Target, description: "" }, FieldSchema { name: "asic_wafer", field_type: U8, source: Target, description: "" }, FieldSchema { name: "asic_wafer_loc_x", field_type: I16, source: Target, description: "" }, FieldSchema { name: "asic_wafer_loc_y", field_type: I16, source: Target, description: "" }, FieldSchema { name: "kind", field_type: String, source: Target, description: "" }, FieldSchema { name: "link_id", field_type: U8, source: Metric, description: "" }, FieldSchema { name: "model", field_type: String, source: Target, description: "" }, FieldSchema { name: "network", field_type: String, source: Target, description: "" }, FieldSchema { name: "port_id", field_type: String, source: Metric, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_model", field_type: String, source: Target, description: "" }, FieldSchema { name: "sled_revision", field_type: U32, source: Target, description: "" }, FieldSchema { name: "sled_serial", field_type: String, source: Target, description: "" }, FieldSchema { name: "switch_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "switch_model", field_type: String, source: Target, description: "" }, FieldSchema { name: "switch_revision", field_type: U32, source: Target, description: "" }, FieldSchema { name: "switch_serial", field_type: String, source: Target, description: "" }, FieldSchema { name: "switch_slot", field_type: U16, source: Target, description: "" }}, datum_type: Bool, version: 1, authz_scope: Fleet, units: Count, created: 2026-07-21T18:14:38.028410313Z } timeseries_name = switch_data_link:link_enabled
1962026-07-21T18:14:38.031ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617 schema = TimeseriesSchema { timeseries_name: TimeseriesName("switch_data_link:link_up"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "asic_fab", field_type: String, source: Target, description: "" }, FieldSchema { name: "asic_lot", field_type: String, source: Target, description: "" }, FieldSchema { name: "asic_wafer", field_type: U8, source: Target, description: "" }, FieldSchema { name: "asic_wafer_loc_x", field_type: I16, source: Target, description: "" }, FieldSchema { name: "asic_wafer_loc_y", field_type: I16, source: Target, description: "" }, FieldSchema { name: "kind", field_type: String, source: Target, description: "" }, FieldSchema { name: "link_id", field_type: U8, source: Metric, description: "" }, FieldSchema { name: "model", field_type: String, source: Target, description: "" }, FieldSchema { name: "network", field_type: String, source: Target, description: "" }, FieldSchema { name: "port_id", field_type: String, source: Metric, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_model", field_type: String, source: Target, description: "" }, FieldSchema { name: "sled_revision", field_type: U32, source: Target, description: "" }, FieldSchema { name: "sled_serial", field_type: String, source: Target, description: "" }, FieldSchema { name: "switch_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "switch_model", field_type: String, source: Target, description: "" }, FieldSchema { name: "switch_revision", field_type: U32, source: Target, description: "" }, FieldSchema { name: "switch_serial", field_type: String, source: Target, description: "" }, FieldSchema { name: "switch_slot", field_type: U16, source: Target, description: "" }}, datum_type: Bool, version: 1, authz_scope: Fleet, units: Count, created: 2026-07-21T18:14:38.028414853Z } timeseries_name = switch_data_link:link_up
1972026-07-21T18:14:38.031ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617 schema = TimeseriesSchema { timeseries_name: TimeseriesName("switch_table:capacity"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "asic_fab", field_type: String, source: Target, description: "" }, FieldSchema { name: "asic_lot", field_type: String, source: Target, description: "" }, FieldSchema { name: "asic_wafer", field_type: U8, source: Target, description: "" }, FieldSchema { name: "asic_wafer_loc_x", field_type: I16, source: Target, description: "" }, FieldSchema { name: "asic_wafer_loc_y", field_type: I16, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "switch_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "switch_model", field_type: String, source: Target, description: "" }, FieldSchema { name: "switch_revision", field_type: U32, source: Target, description: "" }, FieldSchema { name: "switch_serial", field_type: String, source: Target, description: "" }, FieldSchema { name: "switch_slot", field_type: U16, source: Target, description: "" }, FieldSchema { name: "table", field_type: String, source: Target, description: "" }}, datum_type: U64, version: 1, authz_scope: Fleet, units: Count, created: 2026-07-21T18:14:38.028420013Z } timeseries_name = switch_table:capacity
1982026-07-21T18:14:38.031ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617 schema = TimeseriesSchema { timeseries_name: TimeseriesName("switch_table:collisions"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "asic_fab", field_type: String, source: Target, description: "" }, FieldSchema { name: "asic_lot", field_type: String, source: Target, description: "" }, FieldSchema { name: "asic_wafer", field_type: U8, source: Target, description: "" }, FieldSchema { name: "asic_wafer_loc_x", field_type: I16, source: Target, description: "" }, FieldSchema { name: "asic_wafer_loc_y", field_type: I16, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "switch_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "switch_model", field_type: String, source: Target, description: "" }, FieldSchema { name: "switch_revision", field_type: U32, source: Target, description: "" }, FieldSchema { name: "switch_serial", field_type: String, source: Target, description: "" }, FieldSchema { name: "switch_slot", field_type: U16, source: Target, description: "" }, FieldSchema { name: "table", field_type: String, source: Target, description: "" }}, datum_type: CumulativeU64, version: 1, authz_scope: Fleet, units: Count, created: 2026-07-21T18:14:38.028442693Z } timeseries_name = switch_table:collisions
1992026-07-21T18:14:38.031ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617 schema = TimeseriesSchema { timeseries_name: TimeseriesName("switch_table:delete_misses"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "asic_fab", field_type: String, source: Target, description: "" }, FieldSchema { name: "asic_lot", field_type: String, source: Target, description: "" }, FieldSchema { name: "asic_wafer", field_type: U8, source: Target, description: "" }, FieldSchema { name: "asic_wafer_loc_x", field_type: I16, source: Target, description: "" }, FieldSchema { name: "asic_wafer_loc_y", field_type: I16, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "switch_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "switch_model", field_type: String, source: Target, description: "" }, FieldSchema { name: "switch_revision", field_type: U32, source: Target, description: "" }, FieldSchema { name: "switch_serial", field_type: String, source: Target, description: "" }, FieldSchema { name: "switch_slot", field_type: U16, source: Target, description: "" }, FieldSchema { name: "table", field_type: String, source: Target, description: "" }}, datum_type: CumulativeU64, version: 1, authz_scope: Fleet, units: Count, created: 2026-07-21T18:14:38.028450983Z } timeseries_name = switch_table:delete_misses
2002026-07-21T18:14:38.031ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617 schema = TimeseriesSchema { timeseries_name: TimeseriesName("switch_table:deletes"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "asic_fab", field_type: String, source: Target, description: "" }, FieldSchema { name: "asic_lot", field_type: String, source: Target, description: "" }, FieldSchema { name: "asic_wafer", field_type: U8, source: Target, description: "" }, FieldSchema { name: "asic_wafer_loc_x", field_type: I16, source: Target, description: "" }, FieldSchema { name: "asic_wafer_loc_y", field_type: I16, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "switch_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "switch_model", field_type: String, source: Target, description: "" }, FieldSchema { name: "switch_revision", field_type: U32, source: Target, description: "" }, FieldSchema { name: "switch_serial", field_type: String, source: Target, description: "" }, FieldSchema { name: "switch_slot", field_type: U16, source: Target, description: "" }, FieldSchema { name: "table", field_type: String, source: Target, description: "" }}, datum_type: CumulativeU64, version: 1, authz_scope: Fleet, units: Count, created: 2026-07-21T18:14:38.028428593Z } timeseries_name = switch_table:deletes
2012026-07-21T18:14:38.031ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617 schema = TimeseriesSchema { timeseries_name: TimeseriesName("switch_table:exhaustion"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "asic_fab", field_type: String, source: Target, description: "" }, FieldSchema { name: "asic_lot", field_type: String, source: Target, description: "" }, FieldSchema { name: "asic_wafer", field_type: U8, source: Target, description: "" }, FieldSchema { name: "asic_wafer_loc_x", field_type: I16, source: Target, description: "" }, FieldSchema { name: "asic_wafer_loc_y", field_type: I16, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "switch_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "switch_model", field_type: String, source: Target, description: "" }, FieldSchema { name: "switch_revision", field_type: U32, source: Target, description: "" }, FieldSchema { name: "switch_serial", field_type: String, source: Target, description: "" }, FieldSchema { name: "switch_slot", field_type: U16, source: Target, description: "" }, FieldSchema { name: "table", field_type: String, source: Target, description: "" }}, datum_type: CumulativeU64, version: 1, authz_scope: Fleet, units: Count, created: 2026-07-21T18:14:38.028454233Z } timeseries_name = switch_table:exhaustion
2022026-07-21T18:14:38.031ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617 schema = TimeseriesSchema { timeseries_name: TimeseriesName("switch_table:inserts"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "asic_fab", field_type: String, source: Target, description: "" }, FieldSchema { name: "asic_lot", field_type: String, source: Target, description: "" }, FieldSchema { name: "asic_wafer", field_type: U8, source: Target, description: "" }, FieldSchema { name: "asic_wafer_loc_x", field_type: I16, source: Target, description: "" }, FieldSchema { name: "asic_wafer_loc_y", field_type: I16, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "switch_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "switch_model", field_type: String, source: Target, description: "" }, FieldSchema { name: "switch_revision", field_type: U32, source: Target, description: "" }, FieldSchema { name: "switch_serial", field_type: String, source: Target, description: "" }, FieldSchema { name: "switch_slot", field_type: U16, source: Target, description: "" }, FieldSchema { name: "table", field_type: String, source: Target, description: "" }}, datum_type: CumulativeU64, version: 1, authz_scope: Fleet, units: Count, created: 2026-07-21T18:14:38.028425503Z } timeseries_name = switch_table:inserts
2032026-07-21T18:14:38.032ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617 schema = TimeseriesSchema { timeseries_name: TimeseriesName("switch_table:occupancy"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "asic_fab", field_type: String, source: Target, description: "" }, FieldSchema { name: "asic_lot", field_type: String, source: Target, description: "" }, FieldSchema { name: "asic_wafer", field_type: U8, source: Target, description: "" }, FieldSchema { name: "asic_wafer_loc_x", field_type: I16, source: Target, description: "" }, FieldSchema { name: "asic_wafer_loc_y", field_type: I16, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "switch_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "switch_model", field_type: String, source: Target, description: "" }, FieldSchema { name: "switch_revision", field_type: U32, source: Target, description: "" }, FieldSchema { name: "switch_serial", field_type: String, source: Target, description: "" }, FieldSchema { name: "switch_slot", field_type: U16, source: Target, description: "" }, FieldSchema { name: "table", field_type: String, source: Target, description: "" }}, datum_type: U64, version: 1, authz_scope: Fleet, units: Count, created: 2026-07-21T18:14:38.028422673Z } timeseries_name = switch_table:occupancy
2042026-07-21T18:14:38.032ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617 schema = TimeseriesSchema { timeseries_name: TimeseriesName("switch_table:update_misses"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "asic_fab", field_type: String, source: Target, description: "" }, FieldSchema { name: "asic_lot", field_type: String, source: Target, description: "" }, FieldSchema { name: "asic_wafer", field_type: U8, source: Target, description: "" }, FieldSchema { name: "asic_wafer_loc_x", field_type: I16, source: Target, description: "" }, FieldSchema { name: "asic_wafer_loc_y", field_type: I16, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "switch_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "switch_model", field_type: String, source: Target, description: "" }, FieldSchema { name: "switch_revision", field_type: U32, source: Target, description: "" }, FieldSchema { name: "switch_serial", field_type: String, source: Target, description: "" }, FieldSchema { name: "switch_slot", field_type: U16, source: Target, description: "" }, FieldSchema { name: "table", field_type: String, source: Target, description: "" }}, datum_type: CumulativeU64, version: 1, authz_scope: Fleet, units: Count, created: 2026-07-21T18:14:38.028446353Z } timeseries_name = switch_table:update_misses
2052026-07-21T18:14:38.032ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617 schema = TimeseriesSchema { timeseries_name: TimeseriesName("switch_table:updates"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "asic_fab", field_type: String, source: Target, description: "" }, FieldSchema { name: "asic_lot", field_type: String, source: Target, description: "" }, FieldSchema { name: "asic_wafer", field_type: U8, source: Target, description: "" }, FieldSchema { name: "asic_wafer_loc_x", field_type: I16, source: Target, description: "" }, FieldSchema { name: "asic_wafer_loc_y", field_type: I16, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "switch_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "switch_model", field_type: String, source: Target, description: "" }, FieldSchema { name: "switch_revision", field_type: U32, source: Target, description: "" }, FieldSchema { name: "switch_serial", field_type: String, source: Target, description: "" }, FieldSchema { name: "switch_slot", field_type: U16, source: Target, description: "" }, FieldSchema { name: "table", field_type: String, source: Target, description: "" }}, datum_type: CumulativeU64, version: 1, authz_scope: Fleet, units: Count, created: 2026-07-21T18:14:38.028437413Z } timeseries_name = switch_table:updates
2062026-07-21T18:14:38.035ZDEBGoximeter (clickhouse-client): inserting 12 new timeseries schema collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617
2072026-07-21T18:14:38.039ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617 n_rows = 280 table_name = fields_i16
2082026-07-21T18:14:38.042ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617 n_rows = 797 table_name = fields_string
2092026-07-21T18:14:38.044ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617 n_rows = 140 table_name = fields_u16
2102026-07-21T18:14:38.045ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617 n_rows = 145 table_name = fields_u32
2112026-07-21T18:14:38.047ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617 n_rows = 144 table_name = fields_u8
2122026-07-21T18:14:38.049ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617 n_rows = 500 table_name = fields_uuid
2132026-07-21T18:14:38.051ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617 n_rows = 4 table_name = measurements_bool
2142026-07-21T18:14:38.053ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617 n_rows = 255 table_name = measurements_cumulativeu64
2152026-07-21T18:14:38.054ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617 n_rows = 1 table_name = measurements_f64
2162026-07-21T18:14:38.056ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617 n_rows = 80 table_name = measurements_u64
2172026-07-21T18:14:38.130ZWARNoximeter (database-inserter): failed to insert some results into metric DB collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:276
2182026-07-21T18:14:38.234ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = e1811cad-40ab-42cb-a1a2-557a7e4fa2b9 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
2192026-07-21T18:14:38.234ZDEBGoximeter (collection-task): collecting from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = e1811cad-40ab-42cb-a1a2-557a7e4fa2b9 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
2202026-07-21T18:14:38.236ZDEBGoximeter (collection-task): collected results from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 26 producer_id = e1811cad-40ab-42cb-a1a2-557a7e4fa2b9 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
2212026-07-21T18:14:38.745ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = 80daec9f-82ff-4a1e-a3a8-8d4bda5887d1 producer_ip = fd00:1122:3344:101::2 producer_port = 8001
2222026-07-21T18:14:38.745ZDEBGoximeter (collection-task): collecting from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = 80daec9f-82ff-4a1e-a3a8-8d4bda5887d1 producer_ip = fd00:1122:3344:101::2 producer_port = 8001
2232026-07-21T18:14:38.746ZDEBGoximeter (collection-task): collected results from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 14 producer_id = 80daec9f-82ff-4a1e-a3a8-8d4bda5887d1 producer_ip = fd00:1122:3344:101::2 producer_port = 8001
2242026-07-21T18:14:39.234ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = e1811cad-40ab-42cb-a1a2-557a7e4fa2b9 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
2252026-07-21T18:14:39.234ZDEBGoximeter (collection-task): collecting from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = e1811cad-40ab-42cb-a1a2-557a7e4fa2b9 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
2262026-07-21T18:14:39.236ZDEBGoximeter (collection-task): collected results from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 26 producer_id = e1811cad-40ab-42cb-a1a2-557a7e4fa2b9 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
2272026-07-21T18:14:39.745ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = 80daec9f-82ff-4a1e-a3a8-8d4bda5887d1 producer_ip = fd00:1122:3344:101::2 producer_port = 8001
2282026-07-21T18:14:39.745ZDEBGoximeter (collection-task): collecting from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = 80daec9f-82ff-4a1e-a3a8-8d4bda5887d1 producer_ip = fd00:1122:3344:101::2 producer_port = 8001
2292026-07-21T18:14:39.746ZDEBGoximeter (collection-task): collected results from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 14 producer_id = 80daec9f-82ff-4a1e-a3a8-8d4bda5887d1 producer_ip = fd00:1122:3344:101::2 producer_port = 8001
2302026-07-21T18:14:40.234ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = e1811cad-40ab-42cb-a1a2-557a7e4fa2b9 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
2312026-07-21T18:14:40.234ZDEBGoximeter (collection-task): collecting from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = e1811cad-40ab-42cb-a1a2-557a7e4fa2b9 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
2322026-07-21T18:14:40.235ZDEBGoximeter (collection-task): collected results from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 26 producer_id = e1811cad-40ab-42cb-a1a2-557a7e4fa2b9 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
2332026-07-21T18:14:40.745ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = 80daec9f-82ff-4a1e-a3a8-8d4bda5887d1 producer_ip = fd00:1122:3344:101::2 producer_port = 8001
2342026-07-21T18:14:40.745ZDEBGoximeter (collection-task): collecting from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = 80daec9f-82ff-4a1e-a3a8-8d4bda5887d1 producer_ip = fd00:1122:3344:101::2 producer_port = 8001
2352026-07-21T18:14:40.746ZDEBGoximeter (collection-task): collected results from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 14 producer_id = 80daec9f-82ff-4a1e-a3a8-8d4bda5887d1 producer_ip = fd00:1122:3344:101::2 producer_port = 8001
2362026-07-21T18:14:41.234ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = e1811cad-40ab-42cb-a1a2-557a7e4fa2b9 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
2372026-07-21T18:14:41.234ZDEBGoximeter (collection-task): collecting from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = e1811cad-40ab-42cb-a1a2-557a7e4fa2b9 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
2382026-07-21T18:14:41.235ZDEBGoximeter (collection-task): collected results from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 26 producer_id = e1811cad-40ab-42cb-a1a2-557a7e4fa2b9 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
2392026-07-21T18:14:41.538ZINFOoximeter (oximeter-agent): refreshing list of producers from Nexus collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d file = oximeter/collector/src/agent.rs:634
2402026-07-21T18:14:41.538ZDEBGoximeter: client request body = None method = GET uri = http://[fd00:1122:3344:101::a]:12221/metrics/collectors/58217aed-c161-4122-ac86-7675725ee0be/producers?sort_by=id_ascending
2412026-07-21T18:14:41.552ZDEBGoximeter: client response result = Ok(Response { url: "http://[fd00:1122:3344:101::a]:12221/metrics/collectors/58217aed-c161-4122-ac86-7675725ee0be/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "a1fc4ddf-c085-45cf-9aa1-3144cdc782e7", "content-length": "1000", "date": "Tue, 21 Jul 2026 18:14:41 GMT"} })
2422026-07-21T18:14:41.552ZDEBGoximeter: client request body = None method = GET uri = http://[fd00:1122:3344:101::a]:12221/metrics/collectors/58217aed-c161-4122-ac86-7675725ee0be/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJlMTgxMWNhZC00MGFiLTQyY2ItYTFhMi01NTdhN2U0ZmEyYjkifX0%3D
2432026-07-21T18:14:41.555ZDEBGoximeter: client response result = Ok(Response { url: "http://[fd00:1122:3344:101::a]:12221/metrics/collectors/58217aed-c161-4122-ac86-7675725ee0be/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJlMTgxMWNhZC00MGFiLTQyY2ItYTFhMi01NTdhN2U0ZmEyYjkifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "0ab62c9f-3b68-4f4f-a9db-4278eecaa440", "content-length": "29", "date": "Tue, 21 Jul 2026 18:14:41 GMT"} })
2442026-07-21T18:14:41.555ZDEBGoximeter (oximeter-agent): registered new metric producer address = [fd00:1122:3344:101::c]:49130 collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = 364a8dbf-2ce2-4052-a8d2-defa027122c2
2452026-07-21T18:14:41.555ZDEBGoximeter (oximeter-agent): registered new metric producer address = [fd00:1122:3344:101::1]:58482 collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = 6bfeba7a-1e40-4286-afda-7aad3c5c0e21
2462026-07-21T18:14:41.555ZDEBGoximeter (oximeter-agent): registered new metric producer address = [fd00:1122:3344:101::2]:65509 collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = 88fc664b-de3a-4fb3-a9e0-f8811a34a3de
2472026-07-21T18:14:41.555ZINFOoximeter (oximeter-agent): refreshed list of producers from Nexus collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d file = oximeter/collector/src/agent.rs:709 n_current_tasks = 6 n_pruned_tasks = 0
2482026-07-21T18:14:41.745ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = 80daec9f-82ff-4a1e-a3a8-8d4bda5887d1 producer_ip = fd00:1122:3344:101::2 producer_port = 8001
2492026-07-21T18:14:41.745ZDEBGoximeter (collection-task): collecting from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = 80daec9f-82ff-4a1e-a3a8-8d4bda5887d1 producer_ip = fd00:1122:3344:101::2 producer_port = 8001
2502026-07-21T18:14:41.746ZDEBGoximeter (collection-task): collected results from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 14 producer_id = 80daec9f-82ff-4a1e-a3a8-8d4bda5887d1 producer_ip = fd00:1122:3344:101::2 producer_port = 8001
2512026-07-21T18:14:42.234ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = e1811cad-40ab-42cb-a1a2-557a7e4fa2b9 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
2522026-07-21T18:14:42.234ZDEBGoximeter (collection-task): collecting from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = e1811cad-40ab-42cb-a1a2-557a7e4fa2b9 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
2532026-07-21T18:14:42.235ZDEBGoximeter (collection-task): collected results from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 26 producer_id = e1811cad-40ab-42cb-a1a2-557a7e4fa2b9 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
2542026-07-21T18:14:42.745ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = 80daec9f-82ff-4a1e-a3a8-8d4bda5887d1 producer_ip = fd00:1122:3344:101::2 producer_port = 8001
2552026-07-21T18:14:42.745ZDEBGoximeter (collection-task): collecting from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = 80daec9f-82ff-4a1e-a3a8-8d4bda5887d1 producer_ip = fd00:1122:3344:101::2 producer_port = 8001
2562026-07-21T18:14:42.746ZDEBGoximeter (collection-task): collected results from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 14 producer_id = 80daec9f-82ff-4a1e-a3a8-8d4bda5887d1 producer_ip = fd00:1122:3344:101::2 producer_port = 8001
2572026-07-21T18:14:43.028ZDEBGoximeter (database-inserter): inserting samples into database collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d n_samples = 200
2582026-07-21T18:14:43.028ZDEBGoximeter (clickhouse-client): unrolling 200 total samples collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617
2592026-07-21T18:14:43.028ZDEBGoximeter (clickhouse-client): fetching timeseries schema from database collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617
2602026-07-21T18:14:43.029ZDEBGoximeter (database-inserter): inserting samples into database collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d n_samples = 200
2612026-07-21T18:14:43.033ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617 n_rows = 76 table_name = fields_string
2622026-07-21T18:14:43.035ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617 n_rows = 80 table_name = fields_uuid
2632026-07-21T18:14:43.037ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617 n_rows = 150 table_name = measurements_cumulativeu64
2642026-07-21T18:14:43.038ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617 n_rows = 50 table_name = measurements_u64
2652026-07-21T18:14:43.130ZWARNoximeter (database-inserter): failed to insert some results into metric DB collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:276
2662026-07-21T18:14:43.234ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = e1811cad-40ab-42cb-a1a2-557a7e4fa2b9 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
2672026-07-21T18:14:43.234ZDEBGoximeter (collection-task): collecting from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = e1811cad-40ab-42cb-a1a2-557a7e4fa2b9 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
2682026-07-21T18:14:43.235ZDEBGoximeter (collection-task): collected results from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 26 producer_id = e1811cad-40ab-42cb-a1a2-557a7e4fa2b9 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
2692026-07-21T18:14:43.404ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = 364a8dbf-2ce2-4052-a8d2-defa027122c2 producer_ip = fd00:1122:3344:101::c producer_port = 49130
2702026-07-21T18:14:43.404ZDEBGoximeter (collection-task): collecting from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = 364a8dbf-2ce2-4052-a8d2-defa027122c2 producer_ip = fd00:1122:3344:101::c producer_port = 49130
2712026-07-21T18:14:43.413ZDEBGoximeter (collection-task): collected results from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d n_results = 5 n_samples = 12 producer_id = 364a8dbf-2ce2-4052-a8d2-defa027122c2 producer_ip = fd00:1122:3344:101::c producer_port = 49130
2722026-07-21T18:14:43.745ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = 80daec9f-82ff-4a1e-a3a8-8d4bda5887d1 producer_ip = fd00:1122:3344:101::2 producer_port = 8001
2732026-07-21T18:14:43.745ZDEBGoximeter (collection-task): collecting from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = 80daec9f-82ff-4a1e-a3a8-8d4bda5887d1 producer_ip = fd00:1122:3344:101::2 producer_port = 8001
2742026-07-21T18:14:43.746ZDEBGoximeter (collection-task): collected results from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 14 producer_id = 80daec9f-82ff-4a1e-a3a8-8d4bda5887d1 producer_ip = fd00:1122:3344:101::2 producer_port = 8001
2752026-07-21T18:14:44.234ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = e1811cad-40ab-42cb-a1a2-557a7e4fa2b9 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
2762026-07-21T18:14:44.234ZDEBGoximeter (collection-task): collecting from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = e1811cad-40ab-42cb-a1a2-557a7e4fa2b9 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
2772026-07-21T18:14:44.236ZDEBGoximeter (collection-task): collected results from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 26 producer_id = e1811cad-40ab-42cb-a1a2-557a7e4fa2b9 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
2782026-07-21T18:14:44.745ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = 80daec9f-82ff-4a1e-a3a8-8d4bda5887d1 producer_ip = fd00:1122:3344:101::2 producer_port = 8001
2792026-07-21T18:14:44.745ZDEBGoximeter (collection-task): collecting from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = 80daec9f-82ff-4a1e-a3a8-8d4bda5887d1 producer_ip = fd00:1122:3344:101::2 producer_port = 8001
2802026-07-21T18:14:44.746ZDEBGoximeter (collection-task): collected results from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 14 producer_id = 80daec9f-82ff-4a1e-a3a8-8d4bda5887d1 producer_ip = fd00:1122:3344:101::2 producer_port = 8001
2812026-07-21T18:14:45.234ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = e1811cad-40ab-42cb-a1a2-557a7e4fa2b9 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
2822026-07-21T18:14:45.234ZDEBGoximeter (collection-task): collecting from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = e1811cad-40ab-42cb-a1a2-557a7e4fa2b9 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
2832026-07-21T18:14:45.236ZDEBGoximeter (collection-task): collected results from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 26 producer_id = e1811cad-40ab-42cb-a1a2-557a7e4fa2b9 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
2842026-07-21T18:14:45.643ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = 85d0356f-079a-42e7-9fd9-5e576eef1fdd producer_ip = fd00:1122:3344:101::2 producer_port = 43808
2852026-07-21T18:14:45.643ZDEBGoximeter (collection-task): collecting from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = 85d0356f-079a-42e7-9fd9-5e576eef1fdd producer_ip = fd00:1122:3344:101::2 producer_port = 43808
2862026-07-21T18:14:45.657ZDEBGoximeter (collection-task): collected results from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 140 producer_id = 85d0356f-079a-42e7-9fd9-5e576eef1fdd producer_ip = fd00:1122:3344:101::2 producer_port = 43808
2872026-07-21T18:14:45.745ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = 80daec9f-82ff-4a1e-a3a8-8d4bda5887d1 producer_ip = fd00:1122:3344:101::2 producer_port = 8001
2882026-07-21T18:14:45.745ZDEBGoximeter (collection-task): collecting from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = 80daec9f-82ff-4a1e-a3a8-8d4bda5887d1 producer_ip = fd00:1122:3344:101::2 producer_port = 8001
2892026-07-21T18:14:45.747ZDEBGoximeter (collection-task): collected results from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 14 producer_id = 80daec9f-82ff-4a1e-a3a8-8d4bda5887d1 producer_ip = fd00:1122:3344:101::2 producer_port = 8001
2902026-07-21T18:14:46.234ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = e1811cad-40ab-42cb-a1a2-557a7e4fa2b9 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
2912026-07-21T18:14:46.234ZDEBGoximeter (collection-task): collecting from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = e1811cad-40ab-42cb-a1a2-557a7e4fa2b9 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
2922026-07-21T18:14:46.236ZDEBGoximeter (collection-task): collected results from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 26 producer_id = e1811cad-40ab-42cb-a1a2-557a7e4fa2b9 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
2932026-07-21T18:14:46.327ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = 88fc664b-de3a-4fb3-a9e0-f8811a34a3de producer_ip = fd00:1122:3344:101::2 producer_port = 65509
2942026-07-21T18:14:46.327ZDEBGoximeter (collection-task): collecting from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = 88fc664b-de3a-4fb3-a9e0-f8811a34a3de producer_ip = fd00:1122:3344:101::2 producer_port = 65509
2952026-07-21T18:14:46.336ZDEBGoximeter (collection-task): collected results from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d n_results = 2 n_samples = 11 producer_id = 88fc664b-de3a-4fb3-a9e0-f8811a34a3de producer_ip = fd00:1122:3344:101::2 producer_port = 65509
2962026-07-21T18:14:46.744ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = 80daec9f-82ff-4a1e-a3a8-8d4bda5887d1 producer_ip = fd00:1122:3344:101::2 producer_port = 8001
2972026-07-21T18:14:46.745ZDEBGoximeter (collection-task): collecting from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = 80daec9f-82ff-4a1e-a3a8-8d4bda5887d1 producer_ip = fd00:1122:3344:101::2 producer_port = 8001
2982026-07-21T18:14:46.746ZDEBGoximeter (collection-task): collected results from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 14 producer_id = 80daec9f-82ff-4a1e-a3a8-8d4bda5887d1 producer_ip = fd00:1122:3344:101::2 producer_port = 8001
2992026-07-21T18:14:47.234ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = e1811cad-40ab-42cb-a1a2-557a7e4fa2b9 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
3002026-07-21T18:14:47.234ZDEBGoximeter (collection-task): collecting from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = e1811cad-40ab-42cb-a1a2-557a7e4fa2b9 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
3012026-07-21T18:14:47.235ZDEBGoximeter (collection-task): collected results from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 26 producer_id = e1811cad-40ab-42cb-a1a2-557a7e4fa2b9 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
3022026-07-21T18:14:47.745ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = 80daec9f-82ff-4a1e-a3a8-8d4bda5887d1 producer_ip = fd00:1122:3344:101::2 producer_port = 8001
3032026-07-21T18:14:47.745ZDEBGoximeter (collection-task): collecting from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = 80daec9f-82ff-4a1e-a3a8-8d4bda5887d1 producer_ip = fd00:1122:3344:101::2 producer_port = 8001
3042026-07-21T18:14:47.746ZDEBGoximeter (collection-task): collected results from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 14 producer_id = 80daec9f-82ff-4a1e-a3a8-8d4bda5887d1 producer_ip = fd00:1122:3344:101::2 producer_port = 8001
3052026-07-21T18:14:48.028ZDEBGoximeter (database-inserter): inserting samples into database collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d n_samples = 363
3062026-07-21T18:14:48.028ZDEBGoximeter (clickhouse-client): unrolling 363 total samples collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617
3072026-07-21T18:14:48.028ZDEBGoximeter (clickhouse-client): fetching timeseries schema from database collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617
3082026-07-21T18:14:48.029ZDEBGoximeter (database-inserter): inserting samples into database collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d n_samples = 363
3092026-07-21T18:14:48.038ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617 schema = TimeseriesSchema { timeseries_name: TimeseriesName("collection_target:cpus_provisioned"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "id", field_type: Uuid, source: Target, description: "" }}, datum_type: I64, version: 1, authz_scope: Fleet, units: Count, created: 2026-07-21T18:14:48.028482362Z } timeseries_name = collection_target:cpus_provisioned
3102026-07-21T18:14:48.038ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617 schema = TimeseriesSchema { timeseries_name: TimeseriesName("collection_target:ram_provisioned"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "id", field_type: Uuid, source: Target, description: "" }}, datum_type: I64, version: 1, authz_scope: Fleet, units: Count, created: 2026-07-21T18:14:48.028483693Z } timeseries_name = collection_target:ram_provisioned
3112026-07-21T18:14:48.038ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617 schema = TimeseriesSchema { timeseries_name: TimeseriesName("collection_target:virtual_disk_space_provisioned"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "id", field_type: Uuid, source: Target, description: "" }}, datum_type: I64, version: 1, authz_scope: Fleet, units: Count, created: 2026-07-21T18:14:48.028485042Z } timeseries_name = collection_target:virtual_disk_space_provisioned
3122026-07-21T18:14:48.038ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617 schema = TimeseriesSchema { timeseries_name: TimeseriesName("http_service:request_latency_histogram"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "name", field_type: String, source: Target, description: "" }, FieldSchema { name: "operation_id", field_type: String, source: Metric, description: "" }, FieldSchema { name: "status_code", field_type: U16, source: Metric, description: "" }}, datum_type: HistogramU64, version: 1, authz_scope: Fleet, units: Count, created: 2026-07-21T18:14:48.028479222Z } timeseries_name = http_service:request_latency_histogram
3132026-07-21T18:14:48.041ZDEBGoximeter (clickhouse-client): inserting 4 new timeseries schema collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617
3142026-07-21T18:14:48.045ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617 n_rows = 280 table_name = fields_i16
3152026-07-21T18:14:48.048ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617 n_rows = 831 table_name = fields_string
3162026-07-21T18:14:48.049ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617 n_rows = 157 table_name = fields_u16
3172026-07-21T18:14:48.051ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617 n_rows = 145 table_name = fields_u32
3182026-07-21T18:14:48.053ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617 n_rows = 144 table_name = fields_u8
3192026-07-21T18:14:48.055ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617 n_rows = 523 table_name = fields_uuid
3202026-07-21T18:14:48.057ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617 n_rows = 4 table_name = measurements_bool
3212026-07-21T18:14:48.060ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617 n_rows = 255 table_name = measurements_cumulativeu64
3222026-07-21T18:14:48.062ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617 n_rows = 1 table_name = measurements_f64
3232026-07-21T18:14:48.070ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617 n_rows = 17 table_name = measurements_histogramu64
3242026-07-21T18:14:48.072ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617 n_rows = 6 table_name = measurements_i64
3252026-07-21T18:14:48.073ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617 n_rows = 80 table_name = measurements_u64
3262026-07-21T18:14:48.129ZWARNoximeter (database-inserter): failed to insert some results into metric DB collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:276
3272026-07-21T18:14:48.234ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = e1811cad-40ab-42cb-a1a2-557a7e4fa2b9 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
3282026-07-21T18:14:48.234ZDEBGoximeter (collection-task): collecting from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = e1811cad-40ab-42cb-a1a2-557a7e4fa2b9 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
3292026-07-21T18:14:48.236ZDEBGoximeter (collection-task): collected results from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 26 producer_id = e1811cad-40ab-42cb-a1a2-557a7e4fa2b9 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
3302026-07-21T18:14:48.596ZDEBGoximeter (collection-task): reporting oximeter self-collection statistics collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = 85d0356f-079a-42e7-9fd9-5e576eef1fdd producer_ip = fd00:1122:3344:101::2 producer_port = 43808
3312026-07-21T18:14:48.745ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = 80daec9f-82ff-4a1e-a3a8-8d4bda5887d1 producer_ip = fd00:1122:3344:101::2 producer_port = 8001
3322026-07-21T18:14:48.745ZDEBGoximeter (collection-task): collecting from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = 80daec9f-82ff-4a1e-a3a8-8d4bda5887d1 producer_ip = fd00:1122:3344:101::2 producer_port = 8001
3332026-07-21T18:14:48.746ZDEBGoximeter (collection-task): collected results from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 14 producer_id = 80daec9f-82ff-4a1e-a3a8-8d4bda5887d1 producer_ip = fd00:1122:3344:101::2 producer_port = 8001
3342026-07-21T18:14:48.775ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = 6bfeba7a-1e40-4286-afda-7aad3c5c0e21 producer_ip = fd00:1122:3344:101::1 producer_port = 58482
3352026-07-21T18:14:48.775ZDEBGoximeter (collection-task): collecting from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = 6bfeba7a-1e40-4286-afda-7aad3c5c0e21 producer_ip = fd00:1122:3344:101::1 producer_port = 58482
3362026-07-21T18:14:48.917ZDEBGoximeter (collection-task): collected results from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d n_results = 3 n_samples = 6107 producer_id = 6bfeba7a-1e40-4286-afda-7aad3c5c0e21 producer_ip = fd00:1122:3344:101::1 producer_port = 58482
3372026-07-21T18:14:48.949ZDEBGoximeter (database-inserter): inserting samples into database collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d n_samples = 6149
3382026-07-21T18:14:48.949ZDEBGoximeter (database-inserter): inserting samples into database collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d n_samples = 6149
3392026-07-21T18:14:48.949ZDEBGoximeter (clickhouse-client): unrolling 6149 total samples collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617
3402026-07-21T18:14:48.950ZDEBGoximeter (clickhouse-client): fetching timeseries schema from database collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617
3412026-07-21T18:14:48.956ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617 schema = TimeseriesSchema { timeseries_name: TimeseriesName("sled_cpu:cpu_nsec"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "cpu_id", field_type: U32, source: Metric, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_model", field_type: String, source: Target, description: "" }, FieldSchema { name: "sled_revision", field_type: U32, source: Target, description: "" }, FieldSchema { name: "sled_serial", field_type: String, source: Target, description: "" }, FieldSchema { name: "state", field_type: String, source: Metric, description: "" }}, datum_type: CumulativeU64, version: 1, authz_scope: Fleet, units: Count, created: 2026-07-21T18:14:48.949824725Z } timeseries_name = sled_cpu:cpu_nsec
3422026-07-21T18:14:48.956ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617 schema = TimeseriesSchema { timeseries_name: TimeseriesName("sled_data_link:bytes_received"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "kind", field_type: String, source: Target, description: "" }, FieldSchema { name: "link_name", field_type: String, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_model", field_type: String, source: Target, description: "" }, FieldSchema { name: "sled_revision", field_type: U32, source: Target, description: "" }, FieldSchema { name: "sled_serial", field_type: String, source: Target, description: "" }, FieldSchema { name: "zone_name", field_type: String, source: Target, description: "" }}, datum_type: CumulativeU64, version: 1, authz_scope: Fleet, units: Count, created: 2026-07-21T18:14:48.949443599Z } timeseries_name = sled_data_link:bytes_received
3432026-07-21T18:14:48.956ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617 schema = TimeseriesSchema { timeseries_name: TimeseriesName("sled_data_link:bytes_sent"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "kind", field_type: String, source: Target, description: "" }, FieldSchema { name: "link_name", field_type: String, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_model", field_type: String, source: Target, description: "" }, FieldSchema { name: "sled_revision", field_type: U32, source: Target, description: "" }, FieldSchema { name: "sled_serial", field_type: String, source: Target, description: "" }, FieldSchema { name: "zone_name", field_type: String, source: Target, description: "" }}, datum_type: CumulativeU64, version: 1, authz_scope: Fleet, units: Count, created: 2026-07-21T18:14:48.949446999Z } timeseries_name = sled_data_link:bytes_sent
3442026-07-21T18:14:48.956ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617 schema = TimeseriesSchema { timeseries_name: TimeseriesName("sled_data_link:errors_received"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "kind", field_type: String, source: Target, description: "" }, FieldSchema { name: "link_name", field_type: String, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_model", field_type: String, source: Target, description: "" }, FieldSchema { name: "sled_revision", field_type: U32, source: Target, description: "" }, FieldSchema { name: "sled_serial", field_type: String, source: Target, description: "" }, FieldSchema { name: "zone_name", field_type: String, source: Target, description: "" }}, datum_type: CumulativeU64, version: 1, authz_scope: Fleet, units: Count, created: 2026-07-21T18:14:48.949439739Z } timeseries_name = sled_data_link:errors_received
3452026-07-21T18:14:48.956ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617 schema = TimeseriesSchema { timeseries_name: TimeseriesName("sled_data_link:errors_sent"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "kind", field_type: String, source: Target, description: "" }, FieldSchema { name: "link_name", field_type: String, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_model", field_type: String, source: Target, description: "" }, FieldSchema { name: "sled_revision", field_type: U32, source: Target, description: "" }, FieldSchema { name: "sled_serial", field_type: String, source: Target, description: "" }, FieldSchema { name: "zone_name", field_type: String, source: Target, description: "" }}, datum_type: CumulativeU64, version: 1, authz_scope: Fleet, units: Count, created: 2026-07-21T18:14:48.949441729Z } timeseries_name = sled_data_link:errors_sent
3462026-07-21T18:14:48.956ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617 schema = TimeseriesSchema { timeseries_name: TimeseriesName("sled_data_link:packets_received"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "kind", field_type: String, source: Target, description: "" }, FieldSchema { name: "link_name", field_type: String, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_model", field_type: String, source: Target, description: "" }, FieldSchema { name: "sled_revision", field_type: U32, source: Target, description: "" }, FieldSchema { name: "sled_serial", field_type: String, source: Target, description: "" }, FieldSchema { name: "zone_name", field_type: String, source: Target, description: "" }}, datum_type: CumulativeU64, version: 1, authz_scope: Fleet, units: Count, created: 2026-07-21T18:14:48.949445459Z } timeseries_name = sled_data_link:packets_received
3472026-07-21T18:14:48.956ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617 schema = TimeseriesSchema { timeseries_name: TimeseriesName("sled_data_link:packets_sent"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "kind", field_type: String, source: Target, description: "" }, FieldSchema { name: "link_name", field_type: String, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_model", field_type: String, source: Target, description: "" }, FieldSchema { name: "sled_revision", field_type: U32, source: Target, description: "" }, FieldSchema { name: "sled_serial", field_type: String, source: Target, description: "" }, FieldSchema { name: "zone_name", field_type: String, source: Target, description: "" }}, datum_type: CumulativeU64, version: 1, authz_scope: Fleet, units: Count, created: 2026-07-21T18:14:48.949449359Z } timeseries_name = sled_data_link:packets_sent
3482026-07-21T18:14:48.956ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617 schema = TimeseriesSchema { timeseries_name: TimeseriesName("zfs_dataset:bytes_used"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "dataset_id", field_type: Uuid, source: Metric, description: "" }, FieldSchema { name: "dataset_kind", field_type: String, source: Metric, description: "" }, FieldSchema { name: "dataset_name", field_type: String, source: Metric, description: "" }, FieldSchema { name: "pool_id", field_type: Uuid, source: Metric, description: "" }, FieldSchema { name: "pool_kind", field_type: String, source: Metric, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_model", field_type: String, source: Target, description: "" }, FieldSchema { name: "sled_revision", field_type: U32, source: Target, description: "" }, FieldSchema { name: "sled_serial", field_type: String, source: Target, description: "" }, FieldSchema { name: "zone_name", field_type: String, source: Metric, description: "" }}, datum_type: U64, version: 1, authz_scope: Fleet, units: Count, created: 2026-07-21T18:14:48.950237712Z } timeseries_name = zfs_dataset:bytes_used
3492026-07-21T18:14:48.956ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617 schema = TimeseriesSchema { timeseries_name: TimeseriesName("zfs_pool:bytes_allocated"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "pool_id", field_type: Uuid, source: Metric, description: "" }, FieldSchema { name: "pool_kind", field_type: String, source: Metric, description: "" }, FieldSchema { name: "pool_name", field_type: String, source: Metric, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_model", field_type: String, source: Target, description: "" }, FieldSchema { name: "sled_revision", field_type: U32, source: Target, description: "" }, FieldSchema { name: "sled_serial", field_type: String, source: Target, description: "" }}, datum_type: U64, version: 1, authz_scope: Fleet, units: Count, created: 2026-07-21T18:14:48.950229182Z } timeseries_name = zfs_pool:bytes_allocated
3502026-07-21T18:14:48.956ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617 schema = TimeseriesSchema { timeseries_name: TimeseriesName("zfs_pool:bytes_total"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "pool_id", field_type: Uuid, source: Metric, description: "" }, FieldSchema { name: "pool_kind", field_type: String, source: Metric, description: "" }, FieldSchema { name: "pool_name", field_type: String, source: Metric, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_model", field_type: String, source: Target, description: "" }, FieldSchema { name: "sled_revision", field_type: U32, source: Target, description: "" }, FieldSchema { name: "sled_serial", field_type: String, source: Target, description: "" }}, datum_type: U64, version: 1, authz_scope: Fleet, units: Count, created: 2026-07-21T18:14:48.950231302Z } timeseries_name = zfs_pool:bytes_total
3512026-07-21T18:14:48.956ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617 schema = TimeseriesSchema { timeseries_name: TimeseriesName("zone:cpu_nsec"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_model", field_type: String, source: Target, description: "" }, FieldSchema { name: "sled_revision", field_type: U32, source: Target, description: "" }, FieldSchema { name: "sled_serial", field_type: String, source: Target, description: "" }, FieldSchema { name: "state", field_type: String, source: Metric, description: "" }, FieldSchema { name: "zone_id", field_type: Uuid, source: Metric, description: "" }, FieldSchema { name: "zone_name", field_type: String, source: Metric, description: "" }, FieldSchema { name: "zone_type", field_type: String, source: Metric, description: "" }}, datum_type: CumulativeU64, version: 1, authz_scope: Fleet, units: Count, created: 2026-07-21T18:14:48.949603732Z } timeseries_name = zone:cpu_nsec
3522026-07-21T18:14:48.985ZDEBGoximeter (clickhouse-client): inserting 11 new timeseries schema collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617
3532026-07-21T18:14:48.989ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617 n_rows = 4 table_name = fields_ipaddr
3542026-07-21T18:14:48.993ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617 n_rows = 3279 table_name = fields_string
3552026-07-21T18:14:48.994ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617 n_rows = 12 table_name = fields_u16
3562026-07-21T18:14:48.997ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617 n_rows = 953 table_name = fields_u32
3572026-07-21T18:14:48.999ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617 n_rows = 1889 table_name = fields_uuid
3582026-07-21T18:14:49.003ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617 n_rows = 5675 table_name = measurements_cumulativeu64
3592026-07-21T18:14:49.005ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617 n_rows = 8 table_name = measurements_histogramu64
3602026-07-21T18:14:49.007ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617 n_rows = 466 table_name = measurements_u64
3612026-07-21T18:14:49.050ZWARNoximeter (database-inserter): failed to insert some results into metric DB collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:276
3622026-07-21T18:14:49.234ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = e1811cad-40ab-42cb-a1a2-557a7e4fa2b9 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
3632026-07-21T18:14:49.234ZDEBGoximeter (collection-task): collecting from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = e1811cad-40ab-42cb-a1a2-557a7e4fa2b9 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
3642026-07-21T18:14:49.236ZDEBGoximeter (collection-task): collected results from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 26 producer_id = e1811cad-40ab-42cb-a1a2-557a7e4fa2b9 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
3652026-07-21T18:14:49.745ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = 80daec9f-82ff-4a1e-a3a8-8d4bda5887d1 producer_ip = fd00:1122:3344:101::2 producer_port = 8001
3662026-07-21T18:14:49.745ZDEBGoximeter (collection-task): collecting from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = 80daec9f-82ff-4a1e-a3a8-8d4bda5887d1 producer_ip = fd00:1122:3344:101::2 producer_port = 8001
3672026-07-21T18:14:49.746ZDEBGoximeter (collection-task): collected results from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 14 producer_id = 80daec9f-82ff-4a1e-a3a8-8d4bda5887d1 producer_ip = fd00:1122:3344:101::2 producer_port = 8001
3682026-07-21T18:14:50.234ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = e1811cad-40ab-42cb-a1a2-557a7e4fa2b9 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
3692026-07-21T18:14:50.234ZDEBGoximeter (collection-task): collecting from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = e1811cad-40ab-42cb-a1a2-557a7e4fa2b9 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
3702026-07-21T18:14:50.235ZDEBGoximeter (collection-task): collected results from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 26 producer_id = e1811cad-40ab-42cb-a1a2-557a7e4fa2b9 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
3712026-07-21T18:14:50.745ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = 80daec9f-82ff-4a1e-a3a8-8d4bda5887d1 producer_ip = fd00:1122:3344:101::2 producer_port = 8001
3722026-07-21T18:14:50.745ZDEBGoximeter (collection-task): collecting from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = 80daec9f-82ff-4a1e-a3a8-8d4bda5887d1 producer_ip = fd00:1122:3344:101::2 producer_port = 8001
3732026-07-21T18:14:50.746ZDEBGoximeter (collection-task): collected results from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 14 producer_id = 80daec9f-82ff-4a1e-a3a8-8d4bda5887d1 producer_ip = fd00:1122:3344:101::2 producer_port = 8001
3742026-07-21T18:14:51.234ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = e1811cad-40ab-42cb-a1a2-557a7e4fa2b9 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
3752026-07-21T18:14:51.234ZDEBGoximeter (collection-task): collecting from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = e1811cad-40ab-42cb-a1a2-557a7e4fa2b9 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
3762026-07-21T18:14:51.236ZDEBGoximeter (collection-task): collected results from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 26 producer_id = e1811cad-40ab-42cb-a1a2-557a7e4fa2b9 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
3772026-07-21T18:14:51.745ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = 80daec9f-82ff-4a1e-a3a8-8d4bda5887d1 producer_ip = fd00:1122:3344:101::2 producer_port = 8001
3782026-07-21T18:14:51.745ZDEBGoximeter (collection-task): collecting from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = 80daec9f-82ff-4a1e-a3a8-8d4bda5887d1 producer_ip = fd00:1122:3344:101::2 producer_port = 8001
3792026-07-21T18:14:51.746ZDEBGoximeter (collection-task): collected results from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 14 producer_id = 80daec9f-82ff-4a1e-a3a8-8d4bda5887d1 producer_ip = fd00:1122:3344:101::2 producer_port = 8001
3802026-07-21T18:14:52.234ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = e1811cad-40ab-42cb-a1a2-557a7e4fa2b9 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
3812026-07-21T18:14:52.234ZDEBGoximeter (collection-task): collecting from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = e1811cad-40ab-42cb-a1a2-557a7e4fa2b9 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
3822026-07-21T18:14:52.236ZDEBGoximeter (collection-task): collected results from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 26 producer_id = e1811cad-40ab-42cb-a1a2-557a7e4fa2b9 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
3832026-07-21T18:14:52.745ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = 80daec9f-82ff-4a1e-a3a8-8d4bda5887d1 producer_ip = fd00:1122:3344:101::2 producer_port = 8001
3842026-07-21T18:14:52.745ZDEBGoximeter (collection-task): collecting from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = 80daec9f-82ff-4a1e-a3a8-8d4bda5887d1 producer_ip = fd00:1122:3344:101::2 producer_port = 8001
3852026-07-21T18:14:52.746ZDEBGoximeter (collection-task): collected results from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 14 producer_id = 80daec9f-82ff-4a1e-a3a8-8d4bda5887d1 producer_ip = fd00:1122:3344:101::2 producer_port = 8001
3862026-07-21T18:14:53.028ZDEBGoximeter (database-inserter): inserting samples into database collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d n_samples = 160
3872026-07-21T18:14:53.028ZDEBGoximeter (clickhouse-client): unrolling 160 total samples collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617
3882026-07-21T18:14:53.028ZDEBGoximeter (clickhouse-client): fetching timeseries schema from database collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617
3892026-07-21T18:14:53.029ZDEBGoximeter (database-inserter): inserting samples into database collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d n_samples = 160
3902026-07-21T18:14:53.034ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617 n_rows = 76 table_name = fields_string
3912026-07-21T18:14:53.035ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617 n_rows = 80 table_name = fields_uuid
3922026-07-21T18:14:53.037ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617 n_rows = 120 table_name = measurements_cumulativeu64
3932026-07-21T18:14:53.038ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617 n_rows = 40 table_name = measurements_u64
3942026-07-21T18:14:53.131ZWARNoximeter (database-inserter): failed to insert some results into metric DB collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:276
3952026-07-21T18:14:53.234ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = e1811cad-40ab-42cb-a1a2-557a7e4fa2b9 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
3962026-07-21T18:14:53.234ZDEBGoximeter (collection-task): collecting from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = e1811cad-40ab-42cb-a1a2-557a7e4fa2b9 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
3972026-07-21T18:14:53.235ZDEBGoximeter (collection-task): collected results from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 26 producer_id = e1811cad-40ab-42cb-a1a2-557a7e4fa2b9 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
3982026-07-21T18:14:53.406ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = 364a8dbf-2ce2-4052-a8d2-defa027122c2 producer_ip = fd00:1122:3344:101::c producer_port = 49130
3992026-07-21T18:14:53.406ZDEBGoximeter (collection-task): collecting from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = 364a8dbf-2ce2-4052-a8d2-defa027122c2 producer_ip = fd00:1122:3344:101::c producer_port = 49130
4002026-07-21T18:14:53.408ZDEBGoximeter (collection-task): collected results from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d n_results = 5 n_samples = 39 producer_id = 364a8dbf-2ce2-4052-a8d2-defa027122c2 producer_ip = fd00:1122:3344:101::c producer_port = 49130
4012026-07-21T18:14:53.745ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = 80daec9f-82ff-4a1e-a3a8-8d4bda5887d1 producer_ip = fd00:1122:3344:101::2 producer_port = 8001
4022026-07-21T18:14:53.745ZDEBGoximeter (collection-task): collecting from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = 80daec9f-82ff-4a1e-a3a8-8d4bda5887d1 producer_ip = fd00:1122:3344:101::2 producer_port = 8001
4032026-07-21T18:14:53.746ZDEBGoximeter (collection-task): collected results from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 14 producer_id = 80daec9f-82ff-4a1e-a3a8-8d4bda5887d1 producer_ip = fd00:1122:3344:101::2 producer_port = 8001
4042026-07-21T18:14:54.234ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = e1811cad-40ab-42cb-a1a2-557a7e4fa2b9 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
4052026-07-21T18:14:54.234ZDEBGoximeter (collection-task): collecting from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = e1811cad-40ab-42cb-a1a2-557a7e4fa2b9 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
4062026-07-21T18:14:54.235ZDEBGoximeter (collection-task): collected results from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 26 producer_id = e1811cad-40ab-42cb-a1a2-557a7e4fa2b9 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
4072026-07-21T18:14:54.744ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = 80daec9f-82ff-4a1e-a3a8-8d4bda5887d1 producer_ip = fd00:1122:3344:101::2 producer_port = 8001
4082026-07-21T18:14:54.745ZDEBGoximeter (collection-task): collecting from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = 80daec9f-82ff-4a1e-a3a8-8d4bda5887d1 producer_ip = fd00:1122:3344:101::2 producer_port = 8001
4092026-07-21T18:14:54.746ZDEBGoximeter (collection-task): collected results from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 14 producer_id = 80daec9f-82ff-4a1e-a3a8-8d4bda5887d1 producer_ip = fd00:1122:3344:101::2 producer_port = 8001
4102026-07-21T18:14:55.234ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = e1811cad-40ab-42cb-a1a2-557a7e4fa2b9 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
4112026-07-21T18:14:55.234ZDEBGoximeter (collection-task): collecting from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = e1811cad-40ab-42cb-a1a2-557a7e4fa2b9 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
4122026-07-21T18:14:55.236ZDEBGoximeter (collection-task): collected results from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 26 producer_id = e1811cad-40ab-42cb-a1a2-557a7e4fa2b9 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
4132026-07-21T18:14:55.643ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = 85d0356f-079a-42e7-9fd9-5e576eef1fdd producer_ip = fd00:1122:3344:101::2 producer_port = 43808
4142026-07-21T18:14:55.643ZDEBGoximeter (collection-task): collecting from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = 85d0356f-079a-42e7-9fd9-5e576eef1fdd producer_ip = fd00:1122:3344:101::2 producer_port = 43808
4152026-07-21T18:14:55.652ZDEBGoximeter (collection-task): collected results from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 140 producer_id = 85d0356f-079a-42e7-9fd9-5e576eef1fdd producer_ip = fd00:1122:3344:101::2 producer_port = 43808
4162026-07-21T18:14:55.745ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = 80daec9f-82ff-4a1e-a3a8-8d4bda5887d1 producer_ip = fd00:1122:3344:101::2 producer_port = 8001
4172026-07-21T18:14:55.745ZDEBGoximeter (collection-task): collecting from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = 80daec9f-82ff-4a1e-a3a8-8d4bda5887d1 producer_ip = fd00:1122:3344:101::2 producer_port = 8001
4182026-07-21T18:14:55.746ZDEBGoximeter (collection-task): collected results from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 14 producer_id = 80daec9f-82ff-4a1e-a3a8-8d4bda5887d1 producer_ip = fd00:1122:3344:101::2 producer_port = 8001
4192026-07-21T18:14:56.234ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = e1811cad-40ab-42cb-a1a2-557a7e4fa2b9 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
4202026-07-21T18:14:56.234ZDEBGoximeter (collection-task): collecting from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = e1811cad-40ab-42cb-a1a2-557a7e4fa2b9 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
4212026-07-21T18:14:56.236ZDEBGoximeter (collection-task): collected results from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 26 producer_id = e1811cad-40ab-42cb-a1a2-557a7e4fa2b9 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
4222026-07-21T18:14:56.327ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = 88fc664b-de3a-4fb3-a9e0-f8811a34a3de producer_ip = fd00:1122:3344:101::2 producer_port = 65509
4232026-07-21T18:14:56.327ZDEBGoximeter (collection-task): collecting from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = 88fc664b-de3a-4fb3-a9e0-f8811a34a3de producer_ip = fd00:1122:3344:101::2 producer_port = 65509
4242026-07-21T18:14:56.329ZDEBGoximeter (collection-task): collected results from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d n_results = 2 n_samples = 11 producer_id = 88fc664b-de3a-4fb3-a9e0-f8811a34a3de producer_ip = fd00:1122:3344:101::2 producer_port = 65509
4252026-07-21T18:14:56.538ZINFOoximeter (oximeter-agent): refreshing list of producers from Nexus collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d file = oximeter/collector/src/agent.rs:634
4262026-07-21T18:14:56.538ZDEBGoximeter: client request body = None method = GET uri = http://[fd00:1122:3344:101::c]:12221/metrics/collectors/58217aed-c161-4122-ac86-7675725ee0be/producers?sort_by=id_ascending
4272026-07-21T18:14:56.552ZDEBGoximeter: client response result = Ok(Response { url: "http://[fd00:1122:3344:101::c]:12221/metrics/collectors/58217aed-c161-4122-ac86-7675725ee0be/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "7212e089-4eab-4db4-b6e8-27967c2d16d7", "content-length": "1136", "date": "Tue, 21 Jul 2026 18:14:56 GMT"} })
4282026-07-21T18:14:56.552ZDEBGoximeter: client request body = None method = GET uri = http://[fd00:1122:3344:101::c]:12221/metrics/collectors/58217aed-c161-4122-ac86-7675725ee0be/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJlMTgxMWNhZC00MGFiLTQyY2ItYTFhMi01NTdhN2U0ZmEyYjkifX0%3D
4292026-07-21T18:14:56.555ZDEBGoximeter: client response result = Ok(Response { url: "http://[fd00:1122:3344:101::c]:12221/metrics/collectors/58217aed-c161-4122-ac86-7675725ee0be/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJlMTgxMWNhZC00MGFiLTQyY2ItYTFhMi01NTdhN2U0ZmEyYjkifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "ebff272e-6a53-44c4-91b8-f0807d25a566", "content-length": "29", "date": "Tue, 21 Jul 2026 18:14:56 GMT"} })
4302026-07-21T18:14:56.555ZDEBGoximeter (oximeter-agent): received request to register existing metric producer, updating collection information address = [fd00:1122:3344:101::2]:36527 collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d interval = 30s producer_id = 85d0356f-079a-42e7-9fd9-5e576eef1fdd
4312026-07-21T18:14:56.555ZDEBGoximeter (collection-task-handle): notified collection task of new producer endpoint collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = 85d0356f-079a-42e7-9fd9-5e576eef1fdd producer_ip = fd00:1122:3344:101::2 producer_port = 43808
4322026-07-21T18:14:56.555ZDEBGoximeter (oximeter-agent): registered new metric producer address = [fd00:1122:3344:101::2]:4677 collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = baca509e-91d7-4e13-be71-35d7e6912078
4332026-07-21T18:14:56.555ZINFOoximeter (oximeter-agent): refreshed list of producers from Nexus collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d file = oximeter/collector/src/agent.rs:709 n_current_tasks = 7 n_pruned_tasks = 0
4342026-07-21T18:14:56.555ZDEBGoximeter (collection-task): collection task received request to update its producer information address = [fd00:1122:3344:101::2]:36527 collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d interval = 30s producer_id = 85d0356f-079a-42e7-9fd9-5e576eef1fdd producer_ip = fd00:1122:3344:101::2 producer_port = 43808
4352026-07-21T18:14:56.745ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = 80daec9f-82ff-4a1e-a3a8-8d4bda5887d1 producer_ip = fd00:1122:3344:101::2 producer_port = 8001
4362026-07-21T18:14:56.745ZDEBGoximeter (collection-task): collecting from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = 80daec9f-82ff-4a1e-a3a8-8d4bda5887d1 producer_ip = fd00:1122:3344:101::2 producer_port = 8001
4372026-07-21T18:14:56.746ZDEBGoximeter (collection-task): collected results from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 14 producer_id = 80daec9f-82ff-4a1e-a3a8-8d4bda5887d1 producer_ip = fd00:1122:3344:101::2 producer_port = 8001
4382026-07-21T18:14:56.791ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = baca509e-91d7-4e13-be71-35d7e6912078 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
4392026-07-21T18:14:56.791ZDEBGoximeter (collection-task): collecting from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = baca509e-91d7-4e13-be71-35d7e6912078 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
4402026-07-21T18:14:56.799ZDEBGoximeter (collection-task): collected results from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 4 producer_id = baca509e-91d7-4e13-be71-35d7e6912078 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
4412026-07-21T18:14:57.234ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = e1811cad-40ab-42cb-a1a2-557a7e4fa2b9 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
4422026-07-21T18:14:57.234ZDEBGoximeter (collection-task): collecting from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = e1811cad-40ab-42cb-a1a2-557a7e4fa2b9 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
4432026-07-21T18:14:57.236ZDEBGoximeter (collection-task): collected results from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 26 producer_id = e1811cad-40ab-42cb-a1a2-557a7e4fa2b9 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
4442026-07-21T18:14:57.745ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = 80daec9f-82ff-4a1e-a3a8-8d4bda5887d1 producer_ip = fd00:1122:3344:101::2 producer_port = 8001
4452026-07-21T18:14:57.745ZDEBGoximeter (collection-task): collecting from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = 80daec9f-82ff-4a1e-a3a8-8d4bda5887d1 producer_ip = fd00:1122:3344:101::2 producer_port = 8001
4462026-07-21T18:14:57.747ZDEBGoximeter (collection-task): collected results from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 14 producer_id = 80daec9f-82ff-4a1e-a3a8-8d4bda5887d1 producer_ip = fd00:1122:3344:101::2 producer_port = 8001
4472026-07-21T18:14:57.791ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = baca509e-91d7-4e13-be71-35d7e6912078 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
4482026-07-21T18:14:57.791ZDEBGoximeter (collection-task): collecting from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = baca509e-91d7-4e13-be71-35d7e6912078 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
4492026-07-21T18:14:57.792ZDEBGoximeter (collection-task): collected results from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 4 producer_id = baca509e-91d7-4e13-be71-35d7e6912078 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
4502026-07-21T18:14:58.028ZDEBGoximeter (database-inserter): inserting samples into database collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d n_samples = 398
4512026-07-21T18:14:58.028ZDEBGoximeter (clickhouse-client): unrolling 398 total samples collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617
4522026-07-21T18:14:58.028ZDEBGoximeter (clickhouse-client): fetching timeseries schema from database collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617
4532026-07-21T18:14:58.029ZDEBGoximeter (database-inserter): inserting samples into database collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d n_samples = 398
4542026-07-21T18:14:58.031ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617 schema = TimeseriesSchema { timeseries_name: TimeseriesName("mg_lower:routes_blocked_by_link_state"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "hostname", field_type: String, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }}, datum_type: U64, version: 1, authz_scope: Fleet, units: Count, created: 2026-07-21T18:14:58.028829359Z } timeseries_name = mg_lower:routes_blocked_by_link_state
4552026-07-21T18:14:58.031ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617 schema = TimeseriesSchema { timeseries_name: TimeseriesName("static_routing_config:static_nexthops"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "hostname", field_type: String, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }}, datum_type: CumulativeU64, version: 1, authz_scope: Fleet, units: Count, created: 2026-07-21T18:14:58.028826399Z } timeseries_name = static_routing_config:static_nexthops
4562026-07-21T18:14:58.031ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617 schema = TimeseriesSchema { timeseries_name: TimeseriesName("static_routing_config:static_routes"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "hostname", field_type: String, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }}, datum_type: CumulativeU64, version: 1, authz_scope: Fleet, units: Count, created: 2026-07-21T18:14:58.028825079Z } timeseries_name = static_routing_config:static_routes
4572026-07-21T18:14:58.031ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617 schema = TimeseriesSchema { timeseries_name: TimeseriesName("switch_rib:active_routes"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "hostname", field_type: String, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }}, datum_type: U64, version: 1, authz_scope: Fleet, units: Count, created: 2026-07-21T18:14:58.028827889Z } timeseries_name = switch_rib:active_routes
4582026-07-21T18:14:58.035ZDEBGoximeter (clickhouse-client): inserting 4 new timeseries schema collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617
4592026-07-21T18:14:58.038ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617 n_rows = 280 table_name = fields_i16
4602026-07-21T18:14:58.040ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617 n_rows = 871 table_name = fields_string
4612026-07-21T18:14:58.042ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617 n_rows = 175 table_name = fields_u16
4622026-07-21T18:14:58.044ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617 n_rows = 145 table_name = fields_u32
4632026-07-21T18:14:58.046ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617 n_rows = 144 table_name = fields_u8
4642026-07-21T18:14:58.047ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617 n_rows = 551 table_name = fields_uuid
4652026-07-21T18:14:58.049ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617 n_rows = 4 table_name = measurements_bool
4662026-07-21T18:14:58.050ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617 n_rows = 259 table_name = measurements_cumulativeu64
4672026-07-21T18:14:58.052ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617 n_rows = 1 table_name = measurements_f64
4682026-07-21T18:14:58.054ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617 n_rows = 35 table_name = measurements_histogramu64
4692026-07-21T18:14:58.056ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617 n_rows = 15 table_name = measurements_i64
4702026-07-21T18:14:58.057ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617 n_rows = 84 table_name = measurements_u64
4712026-07-21T18:14:58.130ZWARNoximeter (database-inserter): failed to insert some results into metric DB collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:276
4722026-07-21T18:14:58.234ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = e1811cad-40ab-42cb-a1a2-557a7e4fa2b9 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
4732026-07-21T18:14:58.234ZDEBGoximeter (collection-task): collecting from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = e1811cad-40ab-42cb-a1a2-557a7e4fa2b9 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
4742026-07-21T18:14:58.236ZDEBGoximeter (collection-task): collected results from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 26 producer_id = e1811cad-40ab-42cb-a1a2-557a7e4fa2b9 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
4752026-07-21T18:14:58.745ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = 80daec9f-82ff-4a1e-a3a8-8d4bda5887d1 producer_ip = fd00:1122:3344:101::2 producer_port = 8001
4762026-07-21T18:14:58.745ZDEBGoximeter (collection-task): collecting from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = 80daec9f-82ff-4a1e-a3a8-8d4bda5887d1 producer_ip = fd00:1122:3344:101::2 producer_port = 8001
4772026-07-21T18:14:58.746ZDEBGoximeter (collection-task): collected results from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 14 producer_id = 80daec9f-82ff-4a1e-a3a8-8d4bda5887d1 producer_ip = fd00:1122:3344:101::2 producer_port = 8001
4782026-07-21T18:14:58.791ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = baca509e-91d7-4e13-be71-35d7e6912078 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
4792026-07-21T18:14:58.791ZDEBGoximeter (collection-task): collecting from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = baca509e-91d7-4e13-be71-35d7e6912078 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
4802026-07-21T18:14:58.792ZDEBGoximeter (collection-task): collected results from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 4 producer_id = baca509e-91d7-4e13-be71-35d7e6912078 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
4812026-07-21T18:14:58.891ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = 85d0356f-079a-42e7-9fd9-5e576eef1fdd producer_ip = fd00:1122:3344:101::2 producer_port = 43808
4822026-07-21T18:14:58.891ZDEBGoximeter (collection-task): collecting from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = 85d0356f-079a-42e7-9fd9-5e576eef1fdd producer_ip = fd00:1122:3344:101::2 producer_port = 43808
4832026-07-21T18:14:58.905ZDEBGoximeter (collection-task): collected results from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 192 producer_id = 85d0356f-079a-42e7-9fd9-5e576eef1fdd producer_ip = fd00:1122:3344:101::2 producer_port = 43808
4842026-07-21T18:14:59.234ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = e1811cad-40ab-42cb-a1a2-557a7e4fa2b9 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
4852026-07-21T18:14:59.234ZDEBGoximeter (collection-task): collecting from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = e1811cad-40ab-42cb-a1a2-557a7e4fa2b9 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
4862026-07-21T18:14:59.235ZDEBGoximeter (collection-task): collected results from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 26 producer_id = e1811cad-40ab-42cb-a1a2-557a7e4fa2b9 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
4872026-07-21T18:14:59.745ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = 80daec9f-82ff-4a1e-a3a8-8d4bda5887d1 producer_ip = fd00:1122:3344:101::2 producer_port = 8001
4882026-07-21T18:14:59.745ZDEBGoximeter (collection-task): collecting from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = 80daec9f-82ff-4a1e-a3a8-8d4bda5887d1 producer_ip = fd00:1122:3344:101::2 producer_port = 8001
4892026-07-21T18:14:59.746ZDEBGoximeter (collection-task): collected results from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 14 producer_id = 80daec9f-82ff-4a1e-a3a8-8d4bda5887d1 producer_ip = fd00:1122:3344:101::2 producer_port = 8001
4902026-07-21T18:14:59.791ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = baca509e-91d7-4e13-be71-35d7e6912078 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
4912026-07-21T18:14:59.792ZDEBGoximeter (collection-task): collecting from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = baca509e-91d7-4e13-be71-35d7e6912078 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
4922026-07-21T18:14:59.792ZDEBGoximeter (collection-task): collected results from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 4 producer_id = baca509e-91d7-4e13-be71-35d7e6912078 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
4932026-07-21T18:15:00.234ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = e1811cad-40ab-42cb-a1a2-557a7e4fa2b9 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
4942026-07-21T18:15:00.234ZDEBGoximeter (collection-task): collecting from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = e1811cad-40ab-42cb-a1a2-557a7e4fa2b9 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
4952026-07-21T18:15:00.236ZDEBGoximeter (collection-task): collected results from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 26 producer_id = e1811cad-40ab-42cb-a1a2-557a7e4fa2b9 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
4962026-07-21T18:15:00.745ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = 80daec9f-82ff-4a1e-a3a8-8d4bda5887d1 producer_ip = fd00:1122:3344:101::2 producer_port = 8001
4972026-07-21T18:15:00.745ZDEBGoximeter (collection-task): collecting from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = 80daec9f-82ff-4a1e-a3a8-8d4bda5887d1 producer_ip = fd00:1122:3344:101::2 producer_port = 8001
4982026-07-21T18:15:00.746ZDEBGoximeter (collection-task): collected results from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 14 producer_id = 80daec9f-82ff-4a1e-a3a8-8d4bda5887d1 producer_ip = fd00:1122:3344:101::2 producer_port = 8001
4992026-07-21T18:15:00.791ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = baca509e-91d7-4e13-be71-35d7e6912078 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
5002026-07-21T18:15:00.791ZDEBGoximeter (collection-task): collecting from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = baca509e-91d7-4e13-be71-35d7e6912078 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
5012026-07-21T18:15:00.792ZDEBGoximeter (collection-task): collected results from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 4 producer_id = baca509e-91d7-4e13-be71-35d7e6912078 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
5022026-07-21T18:15:01.234ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = e1811cad-40ab-42cb-a1a2-557a7e4fa2b9 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
5032026-07-21T18:15:01.234ZDEBGoximeter (collection-task): collecting from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = e1811cad-40ab-42cb-a1a2-557a7e4fa2b9 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
5042026-07-21T18:15:01.236ZDEBGoximeter (collection-task): collected results from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 26 producer_id = e1811cad-40ab-42cb-a1a2-557a7e4fa2b9 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
5052026-07-21T18:15:01.745ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = 80daec9f-82ff-4a1e-a3a8-8d4bda5887d1 producer_ip = fd00:1122:3344:101::2 producer_port = 8001
5062026-07-21T18:15:01.745ZDEBGoximeter (collection-task): collecting from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = 80daec9f-82ff-4a1e-a3a8-8d4bda5887d1 producer_ip = fd00:1122:3344:101::2 producer_port = 8001
5072026-07-21T18:15:01.746ZDEBGoximeter (collection-task): collected results from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 14 producer_id = 80daec9f-82ff-4a1e-a3a8-8d4bda5887d1 producer_ip = fd00:1122:3344:101::2 producer_port = 8001
5082026-07-21T18:15:01.791ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = baca509e-91d7-4e13-be71-35d7e6912078 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
5092026-07-21T18:15:01.791ZDEBGoximeter (collection-task): collecting from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = baca509e-91d7-4e13-be71-35d7e6912078 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
5102026-07-21T18:15:01.791ZDEBGoximeter (collection-task): collected results from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 4 producer_id = baca509e-91d7-4e13-be71-35d7e6912078 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
5112026-07-21T18:15:02.234ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = e1811cad-40ab-42cb-a1a2-557a7e4fa2b9 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
5122026-07-21T18:15:02.234ZDEBGoximeter (collection-task): collecting from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = e1811cad-40ab-42cb-a1a2-557a7e4fa2b9 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
5132026-07-21T18:15:02.235ZDEBGoximeter (collection-task): collected results from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 26 producer_id = e1811cad-40ab-42cb-a1a2-557a7e4fa2b9 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
5142026-07-21T18:15:02.745ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = 80daec9f-82ff-4a1e-a3a8-8d4bda5887d1 producer_ip = fd00:1122:3344:101::2 producer_port = 8001
5152026-07-21T18:15:02.745ZDEBGoximeter (collection-task): collecting from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = 80daec9f-82ff-4a1e-a3a8-8d4bda5887d1 producer_ip = fd00:1122:3344:101::2 producer_port = 8001
5162026-07-21T18:15:02.746ZDEBGoximeter (collection-task): collected results from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 14 producer_id = 80daec9f-82ff-4a1e-a3a8-8d4bda5887d1 producer_ip = fd00:1122:3344:101::2 producer_port = 8001
5172026-07-21T18:15:02.791ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = baca509e-91d7-4e13-be71-35d7e6912078 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
5182026-07-21T18:15:02.791ZDEBGoximeter (collection-task): collecting from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = baca509e-91d7-4e13-be71-35d7e6912078 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
5192026-07-21T18:15:02.792ZDEBGoximeter (collection-task): collected results from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 4 producer_id = baca509e-91d7-4e13-be71-35d7e6912078 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
5202026-07-21T18:15:03.028ZDEBGoximeter (database-inserter): inserting samples into database collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d n_samples = 412
5212026-07-21T18:15:03.028ZDEBGoximeter (clickhouse-client): unrolling 412 total samples collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617
5222026-07-21T18:15:03.028ZDEBGoximeter (clickhouse-client): fetching timeseries schema from database collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617
5232026-07-21T18:15:03.029ZDEBGoximeter (database-inserter): inserting samples into database collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d n_samples = 412
5242026-07-21T18:15:03.031ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617 schema = TimeseriesSchema { timeseries_name: TimeseriesName("switch_port_control_data_link:bytes_received"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "asic_fab", field_type: String, source: Target, description: "" }, FieldSchema { name: "asic_lot", field_type: String, source: Target, description: "" }, FieldSchema { name: "asic_wafer", field_type: U8, source: Target, description: "" }, FieldSchema { name: "asic_wafer_loc_x", field_type: I16, source: Target, description: "" }, FieldSchema { name: "asic_wafer_loc_y", field_type: I16, source: Target, description: "" }, FieldSchema { name: "kind", field_type: String, source: Target, description: "" }, FieldSchema { name: "link_name", field_type: String, source: Target, description: "" }, FieldSchema { name: "model", field_type: String, source: Target, description: "" }, FieldSchema { name: "network", field_type: String, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_model", field_type: String, source: Target, description: "" }, FieldSchema { name: "sled_revision", field_type: U32, source: Target, description: "" }, FieldSchema { name: "sled_serial", field_type: String, source: Target, description: "" }, FieldSchema { name: "switch_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "switch_model", field_type: String, source: Target, description: "" }, FieldSchema { name: "switch_revision", field_type: U32, source: Target, description: "" }, FieldSchema { name: "switch_serial", field_type: String, source: Target, description: "" }, FieldSchema { name: "switch_slot", field_type: U16, source: Target, description: "" }}, datum_type: CumulativeU64, version: 1, authz_scope: Fleet, units: Count, created: 2026-07-21T18:15:03.028591776Z } timeseries_name = switch_port_control_data_link:bytes_received
5252026-07-21T18:15:03.031ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617 schema = TimeseriesSchema { timeseries_name: TimeseriesName("switch_port_control_data_link:bytes_sent"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "asic_fab", field_type: String, source: Target, description: "" }, FieldSchema { name: "asic_lot", field_type: String, source: Target, description: "" }, FieldSchema { name: "asic_wafer", field_type: U8, source: Target, description: "" }, FieldSchema { name: "asic_wafer_loc_x", field_type: I16, source: Target, description: "" }, FieldSchema { name: "asic_wafer_loc_y", field_type: I16, source: Target, description: "" }, FieldSchema { name: "kind", field_type: String, source: Target, description: "" }, FieldSchema { name: "link_name", field_type: String, source: Target, description: "" }, FieldSchema { name: "model", field_type: String, source: Target, description: "" }, FieldSchema { name: "network", field_type: String, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_model", field_type: String, source: Target, description: "" }, FieldSchema { name: "sled_revision", field_type: U32, source: Target, description: "" }, FieldSchema { name: "sled_serial", field_type: String, source: Target, description: "" }, FieldSchema { name: "switch_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "switch_model", field_type: String, source: Target, description: "" }, FieldSchema { name: "switch_revision", field_type: U32, source: Target, description: "" }, FieldSchema { name: "switch_serial", field_type: String, source: Target, description: "" }, FieldSchema { name: "switch_slot", field_type: U16, source: Target, description: "" }}, datum_type: CumulativeU64, version: 1, authz_scope: Fleet, units: Count, created: 2026-07-21T18:15:03.028601056Z } timeseries_name = switch_port_control_data_link:bytes_sent
5262026-07-21T18:15:03.031ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617 schema = TimeseriesSchema { timeseries_name: TimeseriesName("switch_port_control_data_link:errors_received"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "asic_fab", field_type: String, source: Target, description: "" }, FieldSchema { name: "asic_lot", field_type: String, source: Target, description: "" }, FieldSchema { name: "asic_wafer", field_type: U8, source: Target, description: "" }, FieldSchema { name: "asic_wafer_loc_x", field_type: I16, source: Target, description: "" }, FieldSchema { name: "asic_wafer_loc_y", field_type: I16, source: Target, description: "" }, FieldSchema { name: "kind", field_type: String, source: Target, description: "" }, FieldSchema { name: "link_name", field_type: String, source: Target, description: "" }, FieldSchema { name: "model", field_type: String, source: Target, description: "" }, FieldSchema { name: "network", field_type: String, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_model", field_type: String, source: Target, description: "" }, FieldSchema { name: "sled_revision", field_type: U32, source: Target, description: "" }, FieldSchema { name: "sled_serial", field_type: String, source: Target, description: "" }, FieldSchema { name: "switch_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "switch_model", field_type: String, source: Target, description: "" }, FieldSchema { name: "switch_revision", field_type: U32, source: Target, description: "" }, FieldSchema { name: "switch_serial", field_type: String, source: Target, description: "" }, FieldSchema { name: "switch_slot", field_type: U16, source: Target, description: "" }}, datum_type: CumulativeU64, version: 1, authz_scope: Fleet, units: Count, created: 2026-07-21T18:15:03.028581485Z } timeseries_name = switch_port_control_data_link:errors_received
5272026-07-21T18:15:03.031ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617 schema = TimeseriesSchema { timeseries_name: TimeseriesName("switch_port_control_data_link:errors_sent"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "asic_fab", field_type: String, source: Target, description: "" }, FieldSchema { name: "asic_lot", field_type: String, source: Target, description: "" }, FieldSchema { name: "asic_wafer", field_type: U8, source: Target, description: "" }, FieldSchema { name: "asic_wafer_loc_x", field_type: I16, source: Target, description: "" }, FieldSchema { name: "asic_wafer_loc_y", field_type: I16, source: Target, description: "" }, FieldSchema { name: "kind", field_type: String, source: Target, description: "" }, FieldSchema { name: "link_name", field_type: String, source: Target, description: "" }, FieldSchema { name: "model", field_type: String, source: Target, description: "" }, FieldSchema { name: "network", field_type: String, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_model", field_type: String, source: Target, description: "" }, FieldSchema { name: "sled_revision", field_type: U32, source: Target, description: "" }, FieldSchema { name: "sled_serial", field_type: String, source: Target, description: "" }, FieldSchema { name: "switch_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "switch_model", field_type: String, source: Target, description: "" }, FieldSchema { name: "switch_revision", field_type: U32, source: Target, description: "" }, FieldSchema { name: "switch_serial", field_type: String, source: Target, description: "" }, FieldSchema { name: "switch_slot", field_type: U16, source: Target, description: "" }}, datum_type: CumulativeU64, version: 1, authz_scope: Fleet, units: Count, created: 2026-07-21T18:15:03.028586885Z } timeseries_name = switch_port_control_data_link:errors_sent
5282026-07-21T18:15:03.031ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617 schema = TimeseriesSchema { timeseries_name: TimeseriesName("switch_port_control_data_link:packets_received"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "asic_fab", field_type: String, source: Target, description: "" }, FieldSchema { name: "asic_lot", field_type: String, source: Target, description: "" }, FieldSchema { name: "asic_wafer", field_type: U8, source: Target, description: "" }, FieldSchema { name: "asic_wafer_loc_x", field_type: I16, source: Target, description: "" }, FieldSchema { name: "asic_wafer_loc_y", field_type: I16, source: Target, description: "" }, FieldSchema { name: "kind", field_type: String, source: Target, description: "" }, FieldSchema { name: "link_name", field_type: String, source: Target, description: "" }, FieldSchema { name: "model", field_type: String, source: Target, description: "" }, FieldSchema { name: "network", field_type: String, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_model", field_type: String, source: Target, description: "" }, FieldSchema { name: "sled_revision", field_type: U32, source: Target, description: "" }, FieldSchema { name: "sled_serial", field_type: String, source: Target, description: "" }, FieldSchema { name: "switch_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "switch_model", field_type: String, source: Target, description: "" }, FieldSchema { name: "switch_revision", field_type: U32, source: Target, description: "" }, FieldSchema { name: "switch_serial", field_type: String, source: Target, description: "" }, FieldSchema { name: "switch_slot", field_type: U16, source: Target, description: "" }}, datum_type: CumulativeU64, version: 1, authz_scope: Fleet, units: Count, created: 2026-07-21T18:15:03.028597186Z } timeseries_name = switch_port_control_data_link:packets_received
5292026-07-21T18:15:03.031ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617 schema = TimeseriesSchema { timeseries_name: TimeseriesName("switch_port_control_data_link:packets_sent"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "asic_fab", field_type: String, source: Target, description: "" }, FieldSchema { name: "asic_lot", field_type: String, source: Target, description: "" }, FieldSchema { name: "asic_wafer", field_type: U8, source: Target, description: "" }, FieldSchema { name: "asic_wafer_loc_x", field_type: I16, source: Target, description: "" }, FieldSchema { name: "asic_wafer_loc_y", field_type: I16, source: Target, description: "" }, FieldSchema { name: "kind", field_type: String, source: Target, description: "" }, FieldSchema { name: "link_name", field_type: String, source: Target, description: "" }, FieldSchema { name: "model", field_type: String, source: Target, description: "" }, FieldSchema { name: "network", field_type: String, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_model", field_type: String, source: Target, description: "" }, FieldSchema { name: "sled_revision", field_type: U32, source: Target, description: "" }, FieldSchema { name: "sled_serial", field_type: String, source: Target, description: "" }, FieldSchema { name: "switch_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "switch_model", field_type: String, source: Target, description: "" }, FieldSchema { name: "switch_revision", field_type: U32, source: Target, description: "" }, FieldSchema { name: "switch_serial", field_type: String, source: Target, description: "" }, FieldSchema { name: "switch_slot", field_type: U16, source: Target, description: "" }}, datum_type: CumulativeU64, version: 1, authz_scope: Fleet, units: Count, created: 2026-07-21T18:15:03.028605356Z } timeseries_name = switch_port_control_data_link:packets_sent
5302026-07-21T18:15:03.034ZDEBGoximeter (clickhouse-client): inserting 6 new timeseries schema collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617
5312026-07-21T18:15:03.038ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617 n_rows = 24 table_name = fields_i16
5322026-07-21T18:15:03.040ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617 n_rows = 200 table_name = fields_string
5332026-07-21T18:15:03.042ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617 n_rows = 12 table_name = fields_u16
5342026-07-21T18:15:03.044ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617 n_rows = 24 table_name = fields_u32
5352026-07-21T18:15:03.047ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617 n_rows = 12 table_name = fields_u8
5362026-07-21T18:15:03.049ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617 n_rows = 124 table_name = fields_uuid
5372026-07-21T18:15:03.051ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617 n_rows = 352 table_name = measurements_cumulativeu64
5382026-07-21T18:15:03.053ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d id = fcc5a6b4-2b3b-4fa5-b88b-5c0235e44617 n_rows = 60 table_name = measurements_u64
5392026-07-21T18:15:03.130ZWARNoximeter (database-inserter): failed to insert some results into metric DB collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:276
5402026-07-21T18:15:03.234ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = e1811cad-40ab-42cb-a1a2-557a7e4fa2b9 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
5412026-07-21T18:15:03.234ZDEBGoximeter (collection-task): collecting from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = e1811cad-40ab-42cb-a1a2-557a7e4fa2b9 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
5422026-07-21T18:15:03.236ZDEBGoximeter (collection-task): collected results from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d n_results = 1 n_samples = 26 producer_id = e1811cad-40ab-42cb-a1a2-557a7e4fa2b9 producer_ip = fd00:1122:3344:101::1 producer_port = 8001
5432026-07-21T18:15:03.311ZDEBGoximeter (collection-task): reporting oximeter self-collection statistics collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = 6bfeba7a-1e40-4286-afda-7aad3c5c0e21 producer_ip = fd00:1122:3344:101::1 producer_port = 58482
5442026-07-21T18:15:03.375ZDEBGoximeter (collection-task): reporting oximeter self-collection statistics collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = baca509e-91d7-4e13-be71-35d7e6912078 producer_ip = fd00:1122:3344:101::2 producer_port = 4677
5452026-07-21T18:15:03.405ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = 364a8dbf-2ce2-4052-a8d2-defa027122c2 producer_ip = fd00:1122:3344:101::c producer_port = 49130
5462026-07-21T18:15:03.405ZDEBGoximeter (collection-task): collecting from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d producer_id = 364a8dbf-2ce2-4052-a8d2-defa027122c2 producer_ip = fd00:1122:3344:101::c producer_port = 49130
5472026-07-21T18:15:03.408ZDEBGoximeter (collection-task): collected results from producer collector_id = 58217aed-c161-4122-ac86-7675725ee0be collector_ip = fd00:1122:3344:101::d n_results = 5 n_samples = 26 producer_id = 364a8dbf-2ce2-4052-a8d2-defa027122c2 producer_ip = fd00:1122:3344:101::c producer_port = 49130