Max level shown:
1 [ Jul 23 10:13:48 Enabled. ]
2 [ Jul 23 10:13:48 Rereading configuration. ]
3 [ Jul 23 10:13:49 Rereading configuration. ]
4 [ Jul 23 10:13:54 Executing start method ("ctrun -l child -o noorphan,regent /opt/oxide/oximeter-collector/bin/oximeter run /var/svc/manifest/site/oximeter/config.toml --address [fd00:1122:3344:101::d]:12223 --id ebee0801-669b-46a1-909b-6aa260b2e1b6 &"). ]
5 [ Jul 23 10:13:54 Method "start" exited with status 0. ]
6 note: configured to log to "/dev/stdout"
72025-07-23T10:13:54.607ZDEBGoximeter: registered DTrace probes
82025-07-23T10:13:54.612ZINFOoximeter: starting oximeter server file = oximeter/collector/src/lib.rs:236
92025-07-23T10:13:54.617ZDEBGoximeter: creating ClickHouse client
102025-07-23T10:13:54.680ZDEBGoximeter (clickhouse-client): registered USDT probes collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d id = 95fa9b41-0e46-4735-9a42-3ca9ffb193f0
112025-07-23T10:14:00.514ZWARNoximeter (clickhouse-client): oximeter database does not exist, or is out-of-date collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d file = oximeter/db/src/client/mod.rs:902 id = 95fa9b41-0e46-4735-9a42-3ca9ffb193f0
122025-07-23T10:14:00.514ZDEBGoximeter (oximeter-agent): oximeter database does not exist, creating collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d
132025-07-23T10:14:00.514ZINFOoximeter (clickhouse-client): reading db version collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d file = oximeter/db/src/client/mod.rs:802 id = 95fa9b41-0e46-4735-9a42-3ca9ffb193f0
142025-07-23T10:14:00.516ZWARNoximeter (clickhouse-client): oximeter database does not exist, or is out-of-date collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d file = oximeter/db/src/client/mod.rs:902 id = 95fa9b41-0e46-4735-9a42-3ca9ffb193f0
152025-07-23T10:14:00.516ZINFOoximeter (clickhouse-client): read oximeter database version collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d file = oximeter/db/src/client/mod.rs:807 id = 95fa9b41-0e46-4735-9a42-3ca9ffb193f0 version = 0
162025-07-23T10:14:00.516ZINFOoximeter (clickhouse-client): initializing oximeter schema collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d file = oximeter/db/src/client/mod.rs:812 id = 95fa9b41-0e46-4735-9a42-3ca9ffb193f0
172025-07-23T10:14:00.516ZDEBGoximeter (clickhouse-client): initializing ClickHouse database collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d id = 95fa9b41-0e46-4735-9a42-3ca9ffb193f0
182025-07-23T10:14:01.369ZINFOoximeter (clickhouse-client): inserting current version collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d file = oximeter/db/src/client/mod.rs:833 id = 95fa9b41-0e46-4735-9a42-3ca9ffb193f0 version = 14
192025-07-23T10:14:01.396ZDEBGoximeter (clickhouse-client): registered USDT probes collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d id = f4381dd2-22a9-4df7-86b9-52021331e971
202025-07-23T10:14:01.416ZDEBGoximeter (dropshot): registered endpoint local_addr = [fd00:1122:3344:101::d]:12223 method = GET path = /info versions = all
212025-07-23T10:14:01.416ZDEBGoximeter (dropshot): registered endpoint local_addr = [fd00:1122:3344:101::d]:12223 method = GET path = /producers versions = all
222025-07-23T10:14:01.416ZDEBGoximeter (dropshot): registered endpoint local_addr = [fd00:1122:3344:101::d]:12223 method = DELETE path = /producers/{producer_id} versions = all
232025-07-23T10:14:01.416ZDEBGoximeter (dropshot): registered endpoint local_addr = [fd00:1122:3344:101::d]:12223 method = GET path = /producers/{producer_id} versions = all
242025-07-23T10:14:01.416ZINFOoximeter (dropshot): listening file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:328 local_addr = [fd00:1122:3344:101::d]:12223
252025-07-23T10:14:01.416ZDEBGoximeter (dropshot): successfully registered DTrace USDT probes local_addr = [fd00:1122:3344:101::d]:12223
262025-07-23T10:14:01.416ZDEBGoximeter: registered USDT probes
272025-07-23T10:14:01.416ZDEBGoximeter: contacting nexus
282025-07-23T10:14:01.523ZDEBGoximeter: client request body = Some(Body) method = POST uri = http://[fd00:1122:3344:101::b]:12221/metrics/collectors
292025-07-23T10:14:01.524ZDEBGoximeter: client response result = Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::b]:12221/metrics/collectors", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", [fd00:1122:3344:101::b]:12221, Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
302025-07-23T10:14:01.524ZWARNoximeter: failed to contact nexus, will retry in 134.15655ms error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/collectors)" file = oximeter/collector/src/lib.rs:375
312025-07-23T10:14:01.659ZDEBGoximeter: contacting nexus
322025-07-23T10:14:01.659ZDEBGoximeter: client request body = Some(Body) method = POST uri = http://[fd00:1122:3344:101::b]:12221/metrics/collectors
332025-07-23T10:14:01.659ZDEBGoximeter: client response result = Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::b]:12221/metrics/collectors", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", [fd00:1122:3344:101::b]:12221, Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
342025-07-23T10:14:01.659ZWARNoximeter: failed to contact nexus, will retry in 412.126973ms error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/collectors)" file = oximeter/collector/src/lib.rs:375
352025-07-23T10:14:02.072ZDEBGoximeter: contacting nexus
362025-07-23T10:14:02.072ZDEBGoximeter: client request body = Some(Body) method = POST uri = http://[fd00:1122:3344:101::a]:12221/metrics/collectors
372025-07-23T10:14:02.073ZDEBGoximeter: client response result = Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::a]:12221/metrics/collectors", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", [fd00:1122:3344:101::a]:12221, Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
382025-07-23T10:14:02.073ZWARNoximeter: failed to contact nexus, will retry in 912.432798ms error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/collectors)" file = oximeter/collector/src/lib.rs:375
392025-07-23T10:14:02.986ZDEBGoximeter: contacting nexus
402025-07-23T10:14:02.986ZDEBGoximeter: client request body = Some(Body) method = POST uri = http://[fd00:1122:3344:101::b]:12221/metrics/collectors
412025-07-23T10:14:02.986ZDEBGoximeter: client response result = Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::b]:12221/metrics/collectors", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", [fd00:1122:3344:101::b]:12221, Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
422025-07-23T10:14:02.986ZWARNoximeter: failed to contact nexus, will retry in 2.019231208s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/collectors)" file = oximeter/collector/src/lib.rs:375
432025-07-23T10:14:05.006ZDEBGoximeter: contacting nexus
442025-07-23T10:14:05.006ZDEBGoximeter: client request body = Some(Body) method = POST uri = http://[fd00:1122:3344:101::a]:12221/metrics/collectors
452025-07-23T10:14:05.188ZDEBGoximeter: client response result = Ok(Response { url: "http://[fd00:1122:3344:101::a]:12221/metrics/collectors", status: 204, headers: {"x-request-id": "8df0bb84-f3ce-4bc6-a22e-adfd3e896e32", "date": "Wed, 23 Jul 2025 10:14:05 GMT"} })
462025-07-23T10:14:05.201ZINFOoximeter: oximeter registered with nexus file = oximeter/collector/src/lib.rs:393 id = ebee0801-669b-46a1-909b-6aa260b2e1b6
472025-07-23T10:14:05.201ZINFOoximeter (oximeter-agent): starting refresh list task collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d file = oximeter/collector/src/agent.rs:559
482025-07-23T10:14:05.202ZINFOoximeter (oximeter-agent): refreshing list of producers from Nexus collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d file = oximeter/collector/src/agent.rs:562
492025-07-23T10:14:05.214ZDEBGoximeter: client request body = None method = GET uri = http://[fd00:1122:3344:101::c]:12221/metrics/collectors/ebee0801-669b-46a1-909b-6aa260b2e1b6/producers?sort_by=id_ascending
502025-07-23T10:14:05.384ZDEBGoximeter: client response result = Ok(Response { url: "http://[fd00:1122:3344:101::c]:12221/metrics/collectors/ebee0801-669b-46a1-909b-6aa260b2e1b6/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "f4f0eb9e-762a-4294-9db4-f6a4a11ee240", "content-length": "29", "date": "Wed, 23 Jul 2025 10:14:05 GMT"} })
512025-07-23T10:14:05.385ZINFOoximeter (oximeter-agent): refreshed list of producers from Nexus collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d file = oximeter/collector/src/agent.rs:637 n_current_tasks = 0 n_pruned_tasks = 0
522025-07-23T10:14:20.205ZINFOoximeter (oximeter-agent): refreshing list of producers from Nexus collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d file = oximeter/collector/src/agent.rs:562
532025-07-23T10:14:20.205ZDEBGoximeter: client request body = None method = GET uri = http://[fd00:1122:3344:101::b]:12221/metrics/collectors/ebee0801-669b-46a1-909b-6aa260b2e1b6/producers?sort_by=id_ascending
542025-07-23T10:14:20.217ZDEBGoximeter: client response result = Ok(Response { url: "http://[fd00:1122:3344:101::b]:12221/metrics/collectors/ebee0801-669b-46a1-909b-6aa260b2e1b6/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "34893627-2b88-4fa9-9146-38c80bb863c1", "content-length": "298", "date": "Wed, 23 Jul 2025 10:14:20 GMT"} })
552025-07-23T10:14:20.228ZDEBGoximeter: client request body = None method = GET uri = http://[fd00:1122:3344:101::b]:12221/metrics/collectors/ebee0801-669b-46a1-909b-6aa260b2e1b6/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiI4ZDY4Y2NkNy05MjM4LTRjZTQtOTNhZS02NDg4MjAxMmM1OGMifX0%3D
562025-07-23T10:14:20.231ZDEBGoximeter: client response result = Ok(Response { url: "http://[fd00:1122:3344:101::b]:12221/metrics/collectors/ebee0801-669b-46a1-909b-6aa260b2e1b6/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiI4ZDY4Y2NkNy05MjM4LTRjZTQtOTNhZS02NDg4MjAxMmM1OGMifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "7231d12d-d422-4c7c-9a00-12366da9ff2e", "content-length": "29", "date": "Wed, 23 Jul 2025 10:14:20 GMT"} })
572025-07-23T10:14:20.231ZDEBGoximeter (oximeter-agent): registered new metric producer address = [fd00:1122:3344:101::1]:8001 collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 8d68ccd7-9238-4ce4-93ae-64882012c58c
582025-07-23T10:14:20.233ZINFOoximeter (oximeter-agent): refreshed list of producers from Nexus collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d file = oximeter/collector/src/agent.rs:637 n_current_tasks = 1 n_pruned_tasks = 0
592025-07-23T10:14:21.232ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 8d68ccd7-9238-4ce4-93ae-64882012c58c
602025-07-23T10:14:21.232ZDEBGoximeter (collection-task): collecting from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 8d68ccd7-9238-4ce4-93ae-64882012c58c
612025-07-23T10:14:21.300ZDEBGoximeter (collection-task): collected results from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 8d68ccd7-9238-4ce4-93ae-64882012c58c
622025-07-23T10:14:21.404ZDEBGoximeter (results-sink-cluster): inserting 26 samples into database collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d
632025-07-23T10:14:21.404ZDEBGoximeter (results-sink): inserting 26 samples into database collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d
642025-07-23T10:14:21.404ZDEBGoximeter (clickhouse-client): unrolling 26 total samples collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d id = 95fa9b41-0e46-4735-9a42-3ca9ffb193f0
652025-07-23T10:14:21.409ZDEBGoximeter (clickhouse-client): retrieving timeseries schema from database collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d id = 95fa9b41-0e46-4735-9a42-3ca9ffb193f0
662025-07-23T10:14:21.441ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d id = 95fa9b41-0e46-4735-9a42-3ca9ffb193f0 schema = TimeseriesSchema { timeseries_name: TimeseriesName("ddm_router:originated_underlay_prefixes"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "hostname", field_type: String, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }}, datum_type: U64, version: 1, authz_scope: Fleet, units: Count, created: 2025-07-23T10:14:21.409751927Z } timeseries_name = ddm_router:originated_underlay_prefixes
672025-07-23T10:14:21.450ZDEBGoximeter (clickhouse-client): retrieving timeseries schema from database collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d id = 95fa9b41-0e46-4735-9a42-3ca9ffb193f0
682025-07-23T10:14:21.469ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d id = 95fa9b41-0e46-4735-9a42-3ca9ffb193f0 schema = TimeseriesSchema { timeseries_name: TimeseriesName("ddm_router:originated_tunnel_endpoints"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "hostname", field_type: String, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }}, datum_type: U64, version: 1, authz_scope: Fleet, units: Count, created: 2025-07-23T10:14:21.450963512Z } timeseries_name = ddm_router:originated_tunnel_endpoints
692025-07-23T10:14:21.469ZDEBGoximeter (clickhouse-client): retrieving timeseries schema from database collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d id = 95fa9b41-0e46-4735-9a42-3ca9ffb193f0
702025-07-23T10:14:21.472ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d id = 95fa9b41-0e46-4735-9a42-3ca9ffb193f0 schema = TimeseriesSchema { timeseries_name: TimeseriesName("ddm_session:solicitations_sent"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "hostname", field_type: String, source: Target, description: "" }, FieldSchema { name: "interface", field_type: String, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }}, datum_type: CumulativeU64, version: 1, authz_scope: Fleet, units: Count, created: 2025-07-23T10:14:21.469767208Z } timeseries_name = ddm_session:solicitations_sent
712025-07-23T10:14:21.472ZDEBGoximeter (clickhouse-client): retrieving timeseries schema from database collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d id = 95fa9b41-0e46-4735-9a42-3ca9ffb193f0
722025-07-23T10:14:21.473ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d id = 95fa9b41-0e46-4735-9a42-3ca9ffb193f0 schema = TimeseriesSchema { timeseries_name: TimeseriesName("ddm_session:solicitations_received"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "hostname", field_type: String, source: Target, description: "" }, FieldSchema { name: "interface", field_type: String, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }}, datum_type: CumulativeU64, version: 1, authz_scope: Fleet, units: Count, created: 2025-07-23T10:14:21.472801410Z } timeseries_name = ddm_session:solicitations_received
732025-07-23T10:14:21.473ZDEBGoximeter (clickhouse-client): retrieving timeseries schema from database collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d id = 95fa9b41-0e46-4735-9a42-3ca9ffb193f0
742025-07-23T10:14:21.474ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d id = 95fa9b41-0e46-4735-9a42-3ca9ffb193f0 schema = TimeseriesSchema { timeseries_name: TimeseriesName("ddm_session:advertisements_sent"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "hostname", field_type: String, source: Target, description: "" }, FieldSchema { name: "interface", field_type: String, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }}, datum_type: CumulativeU64, version: 1, authz_scope: Fleet, units: Count, created: 2025-07-23T10:14:21.473756949Z } timeseries_name = ddm_session:advertisements_sent
752025-07-23T10:14:21.474ZDEBGoximeter (clickhouse-client): retrieving timeseries schema from database collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d id = 95fa9b41-0e46-4735-9a42-3ca9ffb193f0
762025-07-23T10:14:21.475ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d id = 95fa9b41-0e46-4735-9a42-3ca9ffb193f0 schema = TimeseriesSchema { timeseries_name: TimeseriesName("ddm_session:advertisements_received"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "hostname", field_type: String, source: Target, description: "" }, FieldSchema { name: "interface", field_type: String, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }}, datum_type: CumulativeU64, version: 1, authz_scope: Fleet, units: Count, created: 2025-07-23T10:14:21.474689418Z } timeseries_name = ddm_session:advertisements_received
772025-07-23T10:14:21.475ZDEBGoximeter (clickhouse-client): retrieving timeseries schema from database collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d id = 95fa9b41-0e46-4735-9a42-3ca9ffb193f0
782025-07-23T10:14:21.476ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d id = 95fa9b41-0e46-4735-9a42-3ca9ffb193f0 schema = TimeseriesSchema { timeseries_name: TimeseriesName("ddm_session:peer_expirations"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "hostname", field_type: String, source: Target, description: "" }, FieldSchema { name: "interface", field_type: String, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }}, datum_type: CumulativeU64, version: 1, authz_scope: Fleet, units: Count, created: 2025-07-23T10:14:21.475682329Z } timeseries_name = ddm_session:peer_expirations
792025-07-23T10:14:21.476ZDEBGoximeter (clickhouse-client): retrieving timeseries schema from database collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d id = 95fa9b41-0e46-4735-9a42-3ca9ffb193f0
802025-07-23T10:14:21.477ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d id = 95fa9b41-0e46-4735-9a42-3ca9ffb193f0 schema = TimeseriesSchema { timeseries_name: TimeseriesName("ddm_session:peer_address_changes"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "hostname", field_type: String, source: Target, description: "" }, FieldSchema { name: "interface", field_type: String, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }}, datum_type: CumulativeU64, version: 1, authz_scope: Fleet, units: Count, created: 2025-07-23T10:14:21.476607508Z } timeseries_name = ddm_session:peer_address_changes
812025-07-23T10:14:21.477ZDEBGoximeter (clickhouse-client): retrieving timeseries schema from database collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d id = 95fa9b41-0e46-4735-9a42-3ca9ffb193f0
822025-07-23T10:14:21.478ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d id = 95fa9b41-0e46-4735-9a42-3ca9ffb193f0 schema = TimeseriesSchema { timeseries_name: TimeseriesName("ddm_session:peer_sessions_established"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "hostname", field_type: String, source: Target, description: "" }, FieldSchema { name: "interface", field_type: String, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }}, datum_type: CumulativeU64, version: 1, authz_scope: Fleet, units: Count, created: 2025-07-23T10:14:21.477522707Z } timeseries_name = ddm_session:peer_sessions_established
832025-07-23T10:14:21.478ZDEBGoximeter (clickhouse-client): retrieving timeseries schema from database collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d id = 95fa9b41-0e46-4735-9a42-3ca9ffb193f0
842025-07-23T10:14:21.479ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d id = 95fa9b41-0e46-4735-9a42-3ca9ffb193f0 schema = TimeseriesSchema { timeseries_name: TimeseriesName("ddm_session:updates_sent"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "hostname", field_type: String, source: Target, description: "" }, FieldSchema { name: "interface", field_type: String, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }}, datum_type: CumulativeU64, version: 1, authz_scope: Fleet, units: Count, created: 2025-07-23T10:14:21.478472596Z } timeseries_name = ddm_session:updates_sent
852025-07-23T10:14:21.479ZDEBGoximeter (clickhouse-client): retrieving timeseries schema from database collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d id = 95fa9b41-0e46-4735-9a42-3ca9ffb193f0
862025-07-23T10:14:21.480ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d id = 95fa9b41-0e46-4735-9a42-3ca9ffb193f0 schema = TimeseriesSchema { timeseries_name: TimeseriesName("ddm_session:updates_received"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "hostname", field_type: String, source: Target, description: "" }, FieldSchema { name: "interface", field_type: String, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }}, datum_type: CumulativeU64, version: 1, authz_scope: Fleet, units: Count, created: 2025-07-23T10:14:21.479416135Z } timeseries_name = ddm_session:updates_received
872025-07-23T10:14:21.480ZDEBGoximeter (clickhouse-client): retrieving timeseries schema from database collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d id = 95fa9b41-0e46-4735-9a42-3ca9ffb193f0
882025-07-23T10:14:21.481ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d id = 95fa9b41-0e46-4735-9a42-3ca9ffb193f0 schema = TimeseriesSchema { timeseries_name: TimeseriesName("ddm_session:update_send_fail"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "hostname", field_type: String, source: Target, description: "" }, FieldSchema { name: "interface", field_type: String, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }}, datum_type: CumulativeU64, version: 1, authz_scope: Fleet, units: Count, created: 2025-07-23T10:14:21.480386115Z } timeseries_name = ddm_session:update_send_fail
892025-07-23T10:14:21.481ZDEBGoximeter (clickhouse-client): retrieving timeseries schema from database collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d id = 95fa9b41-0e46-4735-9a42-3ca9ffb193f0
902025-07-23T10:14:21.482ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d id = 95fa9b41-0e46-4735-9a42-3ca9ffb193f0 schema = TimeseriesSchema { timeseries_name: TimeseriesName("ddm_session:imported_underlay_prefixes"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "hostname", field_type: String, source: Target, description: "" }, FieldSchema { name: "interface", field_type: String, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }}, datum_type: U64, version: 1, authz_scope: Fleet, units: Count, created: 2025-07-23T10:14:21.481331275Z } timeseries_name = ddm_session:imported_underlay_prefixes
912025-07-23T10:14:21.482ZDEBGoximeter (clickhouse-client): retrieving timeseries schema from database collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d id = 95fa9b41-0e46-4735-9a42-3ca9ffb193f0
922025-07-23T10:14:21.483ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d id = 95fa9b41-0e46-4735-9a42-3ca9ffb193f0 schema = TimeseriesSchema { timeseries_name: TimeseriesName("ddm_session:imported_tunnel_endpoints"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "hostname", field_type: String, source: Target, description: "" }, FieldSchema { name: "interface", field_type: String, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }}, datum_type: U64, version: 1, authz_scope: Fleet, units: Count, created: 2025-07-23T10:14:21.482251293Z } timeseries_name = ddm_session:imported_tunnel_endpoints
932025-07-23T10:14:21.483ZDEBGoximeter (clickhouse-client): inserting 14 new timeseries schema collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d id = 95fa9b41-0e46-4735-9a42-3ca9ffb193f0
942025-07-23T10:14:21.494ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d id = 95fa9b41-0e46-4735-9a42-3ca9ffb193f0 n_rows = 26 table_name = fields_string
952025-07-23T10:14:21.496ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d id = 95fa9b41-0e46-4735-9a42-3ca9ffb193f0 n_rows = 28 table_name = fields_uuid
962025-07-23T10:14:21.502ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d id = 95fa9b41-0e46-4735-9a42-3ca9ffb193f0 n_rows = 20 table_name = measurements_cumulativeu64
972025-07-23T10:14:21.503ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d id = 95fa9b41-0e46-4735-9a42-3ca9ffb193f0 n_rows = 6 table_name = measurements_u64
982025-07-23T10:14:21.506ZWARNoximeter (results-sink-cluster): failed to insert some results into metric DB collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
992025-07-23T10:14:22.232ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 8d68ccd7-9238-4ce4-93ae-64882012c58c
1002025-07-23T10:14:22.232ZDEBGoximeter (collection-task): collecting from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 8d68ccd7-9238-4ce4-93ae-64882012c58c
1012025-07-23T10:14:22.234ZDEBGoximeter (collection-task): collected results from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 8d68ccd7-9238-4ce4-93ae-64882012c58c
1022025-07-23T10:14:23.232ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 8d68ccd7-9238-4ce4-93ae-64882012c58c
1032025-07-23T10:14:23.232ZDEBGoximeter (collection-task): collecting from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 8d68ccd7-9238-4ce4-93ae-64882012c58c
1042025-07-23T10:14:23.235ZDEBGoximeter (collection-task): collected results from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 8d68ccd7-9238-4ce4-93ae-64882012c58c
1052025-07-23T10:14:24.232ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 8d68ccd7-9238-4ce4-93ae-64882012c58c
1062025-07-23T10:14:24.232ZDEBGoximeter (collection-task): collecting from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 8d68ccd7-9238-4ce4-93ae-64882012c58c
1072025-07-23T10:14:24.235ZDEBGoximeter (collection-task): collected results from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 8d68ccd7-9238-4ce4-93ae-64882012c58c
1082025-07-23T10:14:25.232ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 8d68ccd7-9238-4ce4-93ae-64882012c58c
1092025-07-23T10:14:25.232ZDEBGoximeter (collection-task): collecting from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 8d68ccd7-9238-4ce4-93ae-64882012c58c
1102025-07-23T10:14:25.234ZDEBGoximeter (collection-task): collected results from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 8d68ccd7-9238-4ce4-93ae-64882012c58c
1112025-07-23T10:14:26.233ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 8d68ccd7-9238-4ce4-93ae-64882012c58c
1122025-07-23T10:14:26.233ZDEBGoximeter (collection-task): collecting from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 8d68ccd7-9238-4ce4-93ae-64882012c58c
1132025-07-23T10:14:26.235ZDEBGoximeter (collection-task): collected results from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 8d68ccd7-9238-4ce4-93ae-64882012c58c
1142025-07-23T10:14:26.399ZDEBGoximeter (results-sink): inserting 130 samples into database collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d
1152025-07-23T10:14:26.399ZDEBGoximeter (results-sink-cluster): inserting 130 samples into database collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d
1162025-07-23T10:14:26.399ZDEBGoximeter (clickhouse-client): unrolling 130 total samples collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d id = 95fa9b41-0e46-4735-9a42-3ca9ffb193f0
1172025-07-23T10:14:26.406ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d id = 95fa9b41-0e46-4735-9a42-3ca9ffb193f0 n_rows = 26 table_name = fields_string
1182025-07-23T10:14:26.408ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d id = 95fa9b41-0e46-4735-9a42-3ca9ffb193f0 n_rows = 28 table_name = fields_uuid
1192025-07-23T10:14:26.410ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d id = 95fa9b41-0e46-4735-9a42-3ca9ffb193f0 n_rows = 100 table_name = measurements_cumulativeu64
1202025-07-23T10:14:26.412ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d id = 95fa9b41-0e46-4735-9a42-3ca9ffb193f0 n_rows = 30 table_name = measurements_u64
1212025-07-23T10:14:26.501ZWARNoximeter (results-sink-cluster): failed to insert some results into metric DB collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
1222025-07-23T10:14:27.233ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 8d68ccd7-9238-4ce4-93ae-64882012c58c
1232025-07-23T10:14:27.233ZDEBGoximeter (collection-task): collecting from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 8d68ccd7-9238-4ce4-93ae-64882012c58c
1242025-07-23T10:14:27.236ZDEBGoximeter (collection-task): collected results from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 8d68ccd7-9238-4ce4-93ae-64882012c58c
1252025-07-23T10:14:28.232ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 8d68ccd7-9238-4ce4-93ae-64882012c58c
1262025-07-23T10:14:28.232ZDEBGoximeter (collection-task): collecting from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 8d68ccd7-9238-4ce4-93ae-64882012c58c
1272025-07-23T10:14:28.235ZDEBGoximeter (collection-task): collected results from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 8d68ccd7-9238-4ce4-93ae-64882012c58c
1282025-07-23T10:14:29.232ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 8d68ccd7-9238-4ce4-93ae-64882012c58c
1292025-07-23T10:14:29.233ZDEBGoximeter (collection-task): collecting from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 8d68ccd7-9238-4ce4-93ae-64882012c58c
1302025-07-23T10:14:29.235ZDEBGoximeter (collection-task): collected results from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 8d68ccd7-9238-4ce4-93ae-64882012c58c
1312025-07-23T10:14:30.233ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 8d68ccd7-9238-4ce4-93ae-64882012c58c
1322025-07-23T10:14:30.233ZDEBGoximeter (collection-task): collecting from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 8d68ccd7-9238-4ce4-93ae-64882012c58c
1332025-07-23T10:14:30.236ZDEBGoximeter (collection-task): collected results from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 8d68ccd7-9238-4ce4-93ae-64882012c58c
1342025-07-23T10:14:31.232ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 8d68ccd7-9238-4ce4-93ae-64882012c58c
1352025-07-23T10:14:31.233ZDEBGoximeter (collection-task): collecting from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 8d68ccd7-9238-4ce4-93ae-64882012c58c
1362025-07-23T10:14:31.235ZDEBGoximeter (collection-task): collected results from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 8d68ccd7-9238-4ce4-93ae-64882012c58c
1372025-07-23T10:14:31.400ZDEBGoximeter (results-sink): inserting 130 samples into database collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d
1382025-07-23T10:14:31.400ZDEBGoximeter (results-sink-cluster): inserting 130 samples into database collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d
1392025-07-23T10:14:31.400ZDEBGoximeter (clickhouse-client): unrolling 130 total samples collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d id = 95fa9b41-0e46-4735-9a42-3ca9ffb193f0
1402025-07-23T10:14:31.407ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d id = 95fa9b41-0e46-4735-9a42-3ca9ffb193f0 n_rows = 26 table_name = fields_string
1412025-07-23T10:14:31.409ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d id = 95fa9b41-0e46-4735-9a42-3ca9ffb193f0 n_rows = 28 table_name = fields_uuid
1422025-07-23T10:14:31.411ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d id = 95fa9b41-0e46-4735-9a42-3ca9ffb193f0 n_rows = 100 table_name = measurements_cumulativeu64
1432025-07-23T10:14:31.412ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d id = 95fa9b41-0e46-4735-9a42-3ca9ffb193f0 n_rows = 30 table_name = measurements_u64
1442025-07-23T10:14:31.501ZWARNoximeter (results-sink-cluster): failed to insert some results into metric DB collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
1452025-07-23T10:14:32.233ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 8d68ccd7-9238-4ce4-93ae-64882012c58c
1462025-07-23T10:14:32.233ZDEBGoximeter (collection-task): collecting from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 8d68ccd7-9238-4ce4-93ae-64882012c58c
1472025-07-23T10:14:32.235ZDEBGoximeter (collection-task): collected results from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 8d68ccd7-9238-4ce4-93ae-64882012c58c
1482025-07-23T10:14:33.233ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 8d68ccd7-9238-4ce4-93ae-64882012c58c
1492025-07-23T10:14:33.233ZDEBGoximeter (collection-task): collecting from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 8d68ccd7-9238-4ce4-93ae-64882012c58c
1502025-07-23T10:14:33.235ZDEBGoximeter (collection-task): collected results from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 8d68ccd7-9238-4ce4-93ae-64882012c58c
1512025-07-23T10:14:34.233ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 8d68ccd7-9238-4ce4-93ae-64882012c58c
1522025-07-23T10:14:34.233ZDEBGoximeter (collection-task): collecting from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 8d68ccd7-9238-4ce4-93ae-64882012c58c
1532025-07-23T10:14:34.236ZDEBGoximeter (collection-task): collected results from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 8d68ccd7-9238-4ce4-93ae-64882012c58c
1542025-07-23T10:14:35.205ZINFOoximeter (oximeter-agent): refreshing list of producers from Nexus collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d file = oximeter/collector/src/agent.rs:562
1552025-07-23T10:14:35.205ZDEBGoximeter: client request body = None method = GET uri = http://[fd00:1122:3344:101::a]:12221/metrics/collectors/ebee0801-669b-46a1-909b-6aa260b2e1b6/producers?sort_by=id_ascending
1562025-07-23T10:14:35.223ZDEBGoximeter: client response result = Ok(Response { url: "http://[fd00:1122:3344:101::a]:12221/metrics/collectors/ebee0801-669b-46a1-909b-6aa260b2e1b6/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "624deff6-3b6a-41ba-8abc-f6024b5deeb3", "content-length": "721", "date": "Wed, 23 Jul 2025 10:14:35 GMT"} })
1572025-07-23T10:14:35.223ZDEBGoximeter: client request body = None method = GET uri = http://[fd00:1122:3344:101::a]:12221/metrics/collectors/ebee0801-669b-46a1-909b-6aa260b2e1b6/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJiMTc5MDU1Ni05YWY5LTQ2YmEtOGZiNy1kYTY0MWZhNWVlOWMifX0%3D
1582025-07-23T10:14:35.228ZDEBGoximeter: client response result = Ok(Response { url: "http://[fd00:1122:3344:101::a]:12221/metrics/collectors/ebee0801-669b-46a1-909b-6aa260b2e1b6/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJiMTc5MDU1Ni05YWY5LTQ2YmEtOGZiNy1kYTY0MWZhNWVlOWMifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "00b8110a-7725-4ac1-a50f-149d7782cfa7", "content-length": "29", "date": "Wed, 23 Jul 2025 10:14:35 GMT"} })
1592025-07-23T10:14:35.228ZDEBGoximeter (oximeter-agent): registered new metric producer address = [fd00:1122:3344:101::2]:4677 collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 882e3042-1c70-490e-9f1e-ab0530fdfade
1602025-07-23T10:14:35.230ZDEBGoximeter (oximeter-agent): registered new metric producer address = [fd00:1122:3344:101::2]:50643 collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 95583e8c-e579-4eba-a499-5e966e69e3b4
1612025-07-23T10:14:35.232ZDEBGoximeter (oximeter-agent): registered new metric producer address = [fd00:1122:3344:101::2]:33980 collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = b1790556-9af9-46ba-8fb7-da641fa5ee9c
1622025-07-23T10:14:35.233ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 8d68ccd7-9238-4ce4-93ae-64882012c58c
1632025-07-23T10:14:35.233ZDEBGoximeter (collection-task): collecting from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 8d68ccd7-9238-4ce4-93ae-64882012c58c
1642025-07-23T10:14:35.234ZINFOoximeter (oximeter-agent): refreshed list of producers from Nexus collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d file = oximeter/collector/src/agent.rs:637 n_current_tasks = 4 n_pruned_tasks = 0
1652025-07-23T10:14:35.236ZDEBGoximeter (collection-task): collected results from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 8d68ccd7-9238-4ce4-93ae-64882012c58c
1662025-07-23T10:14:36.229ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 882e3042-1c70-490e-9f1e-ab0530fdfade
1672025-07-23T10:14:36.229ZDEBGoximeter (collection-task): collecting from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 882e3042-1c70-490e-9f1e-ab0530fdfade
1682025-07-23T10:14:36.233ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 8d68ccd7-9238-4ce4-93ae-64882012c58c
1692025-07-23T10:14:36.233ZDEBGoximeter (collection-task): collecting from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 8d68ccd7-9238-4ce4-93ae-64882012c58c
1702025-07-23T10:14:36.236ZDEBGoximeter (collection-task): collected results from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 8d68ccd7-9238-4ce4-93ae-64882012c58c
1712025-07-23T10:14:36.240ZDEBGoximeter (collection-task): collected results from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 882e3042-1c70-490e-9f1e-ab0530fdfade
1722025-07-23T10:14:36.400ZDEBGoximeter (results-sink): inserting 134 samples into database collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d
1732025-07-23T10:14:36.400ZDEBGoximeter (results-sink-cluster): inserting 134 samples into database collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d
1742025-07-23T10:14:36.400ZDEBGoximeter (clickhouse-client): unrolling 134 total samples collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d id = 95fa9b41-0e46-4735-9a42-3ca9ffb193f0
1752025-07-23T10:14:36.405ZDEBGoximeter (clickhouse-client): retrieving timeseries schema from database collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d id = 95fa9b41-0e46-4735-9a42-3ca9ffb193f0
1762025-07-23T10:14:36.465ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d id = 95fa9b41-0e46-4735-9a42-3ca9ffb193f0 schema = TimeseriesSchema { timeseries_name: TimeseriesName("static_routing_config:static_routes"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "hostname", field_type: String, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }}, datum_type: CumulativeU64, version: 1, authz_scope: Fleet, units: Count, created: 2025-07-23T10:14:36.405105876Z } timeseries_name = static_routing_config:static_routes
1772025-07-23T10:14:36.465ZDEBGoximeter (clickhouse-client): retrieving timeseries schema from database collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d id = 95fa9b41-0e46-4735-9a42-3ca9ffb193f0
1782025-07-23T10:14:36.467ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d id = 95fa9b41-0e46-4735-9a42-3ca9ffb193f0 schema = TimeseriesSchema { timeseries_name: TimeseriesName("static_routing_config:static_nexthops"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "hostname", field_type: String, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }}, datum_type: CumulativeU64, version: 1, authz_scope: Fleet, units: Count, created: 2025-07-23T10:14:36.465609007Z } timeseries_name = static_routing_config:static_nexthops
1792025-07-23T10:14:36.467ZDEBGoximeter (clickhouse-client): retrieving timeseries schema from database collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d id = 95fa9b41-0e46-4735-9a42-3ca9ffb193f0
1802025-07-23T10:14:36.468ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d id = 95fa9b41-0e46-4735-9a42-3ca9ffb193f0 schema = TimeseriesSchema { timeseries_name: TimeseriesName("switch_rib:active_routes"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "hostname", field_type: String, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }}, datum_type: U64, version: 1, authz_scope: Fleet, units: Count, created: 2025-07-23T10:14:36.467392409Z } timeseries_name = switch_rib:active_routes
1812025-07-23T10:14:36.469ZDEBGoximeter (clickhouse-client): retrieving timeseries schema from database collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d id = 95fa9b41-0e46-4735-9a42-3ca9ffb193f0
1822025-07-23T10:14:36.470ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d id = 95fa9b41-0e46-4735-9a42-3ca9ffb193f0 schema = TimeseriesSchema { timeseries_name: TimeseriesName("mg_lower:routes_blocked_by_link_state"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "hostname", field_type: String, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_id", field_type: Uuid, source: Target, description: "" }}, datum_type: U64, version: 1, authz_scope: Fleet, units: Count, created: 2025-07-23T10:14:36.469001435Z } timeseries_name = mg_lower:routes_blocked_by_link_state
1832025-07-23T10:14:36.470ZDEBGoximeter (clickhouse-client): inserting 4 new timeseries schema collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d id = 95fa9b41-0e46-4735-9a42-3ca9ffb193f0
1842025-07-23T10:14:36.473ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d id = 95fa9b41-0e46-4735-9a42-3ca9ffb193f0 n_rows = 30 table_name = fields_string
1852025-07-23T10:14:36.475ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d id = 95fa9b41-0e46-4735-9a42-3ca9ffb193f0 n_rows = 36 table_name = fields_uuid
1862025-07-23T10:14:36.476ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d id = 95fa9b41-0e46-4735-9a42-3ca9ffb193f0 n_rows = 102 table_name = measurements_cumulativeu64
1872025-07-23T10:14:36.478ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d id = 95fa9b41-0e46-4735-9a42-3ca9ffb193f0 n_rows = 32 table_name = measurements_u64
1882025-07-23T10:14:36.501ZWARNoximeter (results-sink-cluster): failed to insert some results into metric DB collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
1892025-07-23T10:14:37.229ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 882e3042-1c70-490e-9f1e-ab0530fdfade
1902025-07-23T10:14:37.229ZDEBGoximeter (collection-task): collecting from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 882e3042-1c70-490e-9f1e-ab0530fdfade
1912025-07-23T10:14:37.230ZDEBGoximeter (collection-task): collected results from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 882e3042-1c70-490e-9f1e-ab0530fdfade
1922025-07-23T10:14:37.233ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 8d68ccd7-9238-4ce4-93ae-64882012c58c
1932025-07-23T10:14:37.233ZDEBGoximeter (collection-task): collecting from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 8d68ccd7-9238-4ce4-93ae-64882012c58c
1942025-07-23T10:14:37.235ZDEBGoximeter (collection-task): collected results from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 8d68ccd7-9238-4ce4-93ae-64882012c58c
1952025-07-23T10:14:38.229ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 882e3042-1c70-490e-9f1e-ab0530fdfade
1962025-07-23T10:14:38.229ZDEBGoximeter (collection-task): collecting from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 882e3042-1c70-490e-9f1e-ab0530fdfade
1972025-07-23T10:14:38.230ZDEBGoximeter (collection-task): collected results from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 882e3042-1c70-490e-9f1e-ab0530fdfade
1982025-07-23T10:14:38.233ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 8d68ccd7-9238-4ce4-93ae-64882012c58c
1992025-07-23T10:14:38.233ZDEBGoximeter (collection-task): collecting from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 8d68ccd7-9238-4ce4-93ae-64882012c58c
2002025-07-23T10:14:38.235ZDEBGoximeter (collection-task): collected results from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 8d68ccd7-9238-4ce4-93ae-64882012c58c
2012025-07-23T10:14:39.230ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 882e3042-1c70-490e-9f1e-ab0530fdfade
2022025-07-23T10:14:39.230ZDEBGoximeter (collection-task): collecting from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 882e3042-1c70-490e-9f1e-ab0530fdfade
2032025-07-23T10:14:39.231ZDEBGoximeter (collection-task): collected results from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 882e3042-1c70-490e-9f1e-ab0530fdfade
2042025-07-23T10:14:39.233ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 8d68ccd7-9238-4ce4-93ae-64882012c58c
2052025-07-23T10:14:39.233ZDEBGoximeter (collection-task): collecting from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 8d68ccd7-9238-4ce4-93ae-64882012c58c
2062025-07-23T10:14:39.235ZDEBGoximeter (collection-task): collected results from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 8d68ccd7-9238-4ce4-93ae-64882012c58c
2072025-07-23T10:14:40.229ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 882e3042-1c70-490e-9f1e-ab0530fdfade
2082025-07-23T10:14:40.229ZDEBGoximeter (collection-task): collecting from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 882e3042-1c70-490e-9f1e-ab0530fdfade
2092025-07-23T10:14:40.230ZDEBGoximeter (collection-task): collected results from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 882e3042-1c70-490e-9f1e-ab0530fdfade
2102025-07-23T10:14:40.233ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 8d68ccd7-9238-4ce4-93ae-64882012c58c
2112025-07-23T10:14:40.233ZDEBGoximeter (collection-task): collecting from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 8d68ccd7-9238-4ce4-93ae-64882012c58c
2122025-07-23T10:14:40.235ZDEBGoximeter (collection-task): collected results from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 8d68ccd7-9238-4ce4-93ae-64882012c58c
2132025-07-23T10:14:41.229ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 882e3042-1c70-490e-9f1e-ab0530fdfade
2142025-07-23T10:14:41.229ZDEBGoximeter (collection-task): collecting from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 882e3042-1c70-490e-9f1e-ab0530fdfade
2152025-07-23T10:14:41.230ZDEBGoximeter (collection-task): collected results from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 882e3042-1c70-490e-9f1e-ab0530fdfade
2162025-07-23T10:14:41.233ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 8d68ccd7-9238-4ce4-93ae-64882012c58c
2172025-07-23T10:14:41.233ZDEBGoximeter (collection-task): collecting from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 8d68ccd7-9238-4ce4-93ae-64882012c58c
2182025-07-23T10:14:41.235ZDEBGoximeter (collection-task): collected results from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 8d68ccd7-9238-4ce4-93ae-64882012c58c
2192025-07-23T10:14:41.400ZDEBGoximeter (results-sink): inserting 150 samples into database collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d
2202025-07-23T10:14:41.400ZDEBGoximeter (results-sink-cluster): inserting 150 samples into database collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d
2212025-07-23T10:14:41.400ZDEBGoximeter (clickhouse-client): unrolling 150 total samples collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d id = 95fa9b41-0e46-4735-9a42-3ca9ffb193f0
2222025-07-23T10:14:41.408ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d id = 95fa9b41-0e46-4735-9a42-3ca9ffb193f0 n_rows = 30 table_name = fields_string
2232025-07-23T10:14:41.410ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d id = 95fa9b41-0e46-4735-9a42-3ca9ffb193f0 n_rows = 36 table_name = fields_uuid
2242025-07-23T10:14:41.411ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d id = 95fa9b41-0e46-4735-9a42-3ca9ffb193f0 n_rows = 110 table_name = measurements_cumulativeu64
2252025-07-23T10:14:41.413ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d id = 95fa9b41-0e46-4735-9a42-3ca9ffb193f0 n_rows = 40 table_name = measurements_u64
2262025-07-23T10:14:41.501ZWARNoximeter (results-sink-cluster): failed to insert some results into metric DB collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
2272025-07-23T10:14:42.229ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 882e3042-1c70-490e-9f1e-ab0530fdfade
2282025-07-23T10:14:42.229ZDEBGoximeter (collection-task): collecting from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 882e3042-1c70-490e-9f1e-ab0530fdfade
2292025-07-23T10:14:42.230ZDEBGoximeter (collection-task): collected results from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 882e3042-1c70-490e-9f1e-ab0530fdfade
2302025-07-23T10:14:42.234ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 8d68ccd7-9238-4ce4-93ae-64882012c58c
2312025-07-23T10:14:42.234ZDEBGoximeter (collection-task): collecting from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 8d68ccd7-9238-4ce4-93ae-64882012c58c
2322025-07-23T10:14:42.237ZDEBGoximeter (collection-task): collected results from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 8d68ccd7-9238-4ce4-93ae-64882012c58c
2332025-07-23T10:14:43.229ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 882e3042-1c70-490e-9f1e-ab0530fdfade
2342025-07-23T10:14:43.229ZDEBGoximeter (collection-task): collecting from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 882e3042-1c70-490e-9f1e-ab0530fdfade
2352025-07-23T10:14:43.230ZDEBGoximeter (collection-task): collected results from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 882e3042-1c70-490e-9f1e-ab0530fdfade
2362025-07-23T10:14:43.233ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 8d68ccd7-9238-4ce4-93ae-64882012c58c
2372025-07-23T10:14:43.233ZDEBGoximeter (collection-task): collecting from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 8d68ccd7-9238-4ce4-93ae-64882012c58c
2382025-07-23T10:14:43.236ZDEBGoximeter (collection-task): collected results from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 8d68ccd7-9238-4ce4-93ae-64882012c58c
2392025-07-23T10:14:44.229ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 882e3042-1c70-490e-9f1e-ab0530fdfade
2402025-07-23T10:14:44.229ZDEBGoximeter (collection-task): collecting from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 882e3042-1c70-490e-9f1e-ab0530fdfade
2412025-07-23T10:14:44.230ZDEBGoximeter (collection-task): collected results from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 882e3042-1c70-490e-9f1e-ab0530fdfade
2422025-07-23T10:14:44.233ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 8d68ccd7-9238-4ce4-93ae-64882012c58c
2432025-07-23T10:14:44.233ZDEBGoximeter (collection-task): collecting from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 8d68ccd7-9238-4ce4-93ae-64882012c58c
2442025-07-23T10:14:44.235ZDEBGoximeter (collection-task): collected results from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 8d68ccd7-9238-4ce4-93ae-64882012c58c
2452025-07-23T10:14:45.229ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 882e3042-1c70-490e-9f1e-ab0530fdfade
2462025-07-23T10:14:45.229ZDEBGoximeter (collection-task): collecting from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 882e3042-1c70-490e-9f1e-ab0530fdfade
2472025-07-23T10:14:45.230ZDEBGoximeter (collection-task): collected results from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 882e3042-1c70-490e-9f1e-ab0530fdfade
2482025-07-23T10:14:45.231ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 95583e8c-e579-4eba-a499-5e966e69e3b4
2492025-07-23T10:14:45.231ZDEBGoximeter (collection-task): collecting from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 95583e8c-e579-4eba-a499-5e966e69e3b4
2502025-07-23T10:14:45.233ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 8d68ccd7-9238-4ce4-93ae-64882012c58c
2512025-07-23T10:14:45.233ZDEBGoximeter (collection-task): collecting from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 8d68ccd7-9238-4ce4-93ae-64882012c58c
2522025-07-23T10:14:45.236ZDEBGoximeter (collection-task): collected results from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 8d68ccd7-9238-4ce4-93ae-64882012c58c
2532025-07-23T10:14:45.246ZDEBGoximeter (collection-task): collected results from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d n_results = 2 producer_id = 95583e8c-e579-4eba-a499-5e966e69e3b4
2542025-07-23T10:14:46.230ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 882e3042-1c70-490e-9f1e-ab0530fdfade
2552025-07-23T10:14:46.230ZDEBGoximeter (collection-task): collecting from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 882e3042-1c70-490e-9f1e-ab0530fdfade
2562025-07-23T10:14:46.231ZDEBGoximeter (collection-task): collected results from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 882e3042-1c70-490e-9f1e-ab0530fdfade
2572025-07-23T10:14:46.234ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 8d68ccd7-9238-4ce4-93ae-64882012c58c
2582025-07-23T10:14:46.234ZDEBGoximeter (collection-task): collecting from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 8d68ccd7-9238-4ce4-93ae-64882012c58c
2592025-07-23T10:14:46.236ZDEBGoximeter (collection-task): collected results from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 8d68ccd7-9238-4ce4-93ae-64882012c58c
2602025-07-23T10:14:46.400ZDEBGoximeter (results-sink): inserting 160 samples into database collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d
2612025-07-23T10:14:46.400ZDEBGoximeter (results-sink-cluster): inserting 160 samples into database collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d
2622025-07-23T10:14:46.400ZDEBGoximeter (clickhouse-client): unrolling 160 total samples collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d id = 95fa9b41-0e46-4735-9a42-3ca9ffb193f0
2632025-07-23T10:14:46.405ZDEBGoximeter (clickhouse-client): retrieving timeseries schema from database collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d id = 95fa9b41-0e46-4735-9a42-3ca9ffb193f0
2642025-07-23T10:14:46.412ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d id = 95fa9b41-0e46-4735-9a42-3ca9ffb193f0 schema = TimeseriesSchema { timeseries_name: TimeseriesName("http_service:request_latency_histogram"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "name", field_type: String, source: Target, description: "" }, FieldSchema { name: "operation_id", field_type: String, source: Metric, description: "" }, FieldSchema { name: "status_code", field_type: U16, source: Metric, description: "" }}, datum_type: HistogramU64, version: 1, authz_scope: Fleet, units: Count, created: 2025-07-23T10:14:46.405244830Z } timeseries_name = http_service:request_latency_histogram
2652025-07-23T10:14:46.414ZDEBGoximeter (clickhouse-client): inserting 1 new timeseries schema collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d id = 95fa9b41-0e46-4735-9a42-3ca9ffb193f0
2662025-07-23T10:14:46.417ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d id = 95fa9b41-0e46-4735-9a42-3ca9ffb193f0 n_rows = 50 table_name = fields_string
2672025-07-23T10:14:46.419ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d id = 95fa9b41-0e46-4735-9a42-3ca9ffb193f0 n_rows = 10 table_name = fields_u16
2682025-07-23T10:14:46.420ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d id = 95fa9b41-0e46-4735-9a42-3ca9ffb193f0 n_rows = 46 table_name = fields_uuid
2692025-07-23T10:14:46.422ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d id = 95fa9b41-0e46-4735-9a42-3ca9ffb193f0 n_rows = 110 table_name = measurements_cumulativeu64
2702025-07-23T10:14:46.428ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d id = 95fa9b41-0e46-4735-9a42-3ca9ffb193f0 n_rows = 10 table_name = measurements_histogramu64
2712025-07-23T10:14:46.430ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d id = 95fa9b41-0e46-4735-9a42-3ca9ffb193f0 n_rows = 40 table_name = measurements_u64
2722025-07-23T10:14:46.501ZWARNoximeter (results-sink-cluster): failed to insert some results into metric DB collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
2732025-07-23T10:14:47.229ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 882e3042-1c70-490e-9f1e-ab0530fdfade
2742025-07-23T10:14:47.229ZDEBGoximeter (collection-task): collecting from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 882e3042-1c70-490e-9f1e-ab0530fdfade
2752025-07-23T10:14:47.230ZDEBGoximeter (collection-task): collected results from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 882e3042-1c70-490e-9f1e-ab0530fdfade
2762025-07-23T10:14:47.233ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 8d68ccd7-9238-4ce4-93ae-64882012c58c
2772025-07-23T10:14:47.233ZDEBGoximeter (collection-task): collecting from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 8d68ccd7-9238-4ce4-93ae-64882012c58c
2782025-07-23T10:14:47.236ZDEBGoximeter (collection-task): collected results from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 8d68ccd7-9238-4ce4-93ae-64882012c58c
2792025-07-23T10:14:48.230ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 882e3042-1c70-490e-9f1e-ab0530fdfade
2802025-07-23T10:14:48.230ZDEBGoximeter (collection-task): collecting from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 882e3042-1c70-490e-9f1e-ab0530fdfade
2812025-07-23T10:14:48.231ZDEBGoximeter (collection-task): collected results from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 882e3042-1c70-490e-9f1e-ab0530fdfade
2822025-07-23T10:14:48.234ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 8d68ccd7-9238-4ce4-93ae-64882012c58c
2832025-07-23T10:14:48.234ZDEBGoximeter (collection-task): collecting from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 8d68ccd7-9238-4ce4-93ae-64882012c58c
2842025-07-23T10:14:48.236ZDEBGoximeter (collection-task): collected results from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 8d68ccd7-9238-4ce4-93ae-64882012c58c
2852025-07-23T10:14:49.230ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 882e3042-1c70-490e-9f1e-ab0530fdfade
2862025-07-23T10:14:49.230ZDEBGoximeter (collection-task): collecting from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 882e3042-1c70-490e-9f1e-ab0530fdfade
2872025-07-23T10:14:49.231ZDEBGoximeter (collection-task): collected results from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 882e3042-1c70-490e-9f1e-ab0530fdfade
2882025-07-23T10:14:49.234ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 8d68ccd7-9238-4ce4-93ae-64882012c58c
2892025-07-23T10:14:49.234ZDEBGoximeter (collection-task): collecting from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 8d68ccd7-9238-4ce4-93ae-64882012c58c
2902025-07-23T10:14:49.237ZDEBGoximeter (collection-task): collected results from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 8d68ccd7-9238-4ce4-93ae-64882012c58c
2912025-07-23T10:14:50.206ZINFOoximeter (oximeter-agent): refreshing list of producers from Nexus collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d file = oximeter/collector/src/agent.rs:562
2922025-07-23T10:14:50.206ZDEBGoximeter: client request body = None method = GET uri = http://[fd00:1122:3344:101::c]:12221/metrics/collectors/ebee0801-669b-46a1-909b-6aa260b2e1b6/producers?sort_by=id_ascending
2932025-07-23T10:14:50.211ZDEBGoximeter: client response result = Ok(Response { url: "http://[fd00:1122:3344:101::c]:12221/metrics/collectors/ebee0801-669b-46a1-909b-6aa260b2e1b6/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "6e2c83e1-bc90-4611-a696-d3cdb2c1c70b", "content-length": "995", "date": "Wed, 23 Jul 2025 10:14:50 GMT"} })
2942025-07-23T10:14:50.211ZDEBGoximeter: client request body = None method = GET uri = http://[fd00:1122:3344:101::c]:12221/metrics/collectors/ebee0801-669b-46a1-909b-6aa260b2e1b6/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJiMTc5MDU1Ni05YWY5LTQ2YmEtOGZiNy1kYTY0MWZhNWVlOWMifX0%3D
2952025-07-23T10:14:50.214ZDEBGoximeter: client response result = Ok(Response { url: "http://[fd00:1122:3344:101::c]:12221/metrics/collectors/ebee0801-669b-46a1-909b-6aa260b2e1b6/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJiMTc5MDU1Ni05YWY5LTQ2YmEtOGZiNy1kYTY0MWZhNWVlOWMifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "2beb2f10-98d5-4bb2-b36d-384fb7dcb0f3", "content-length": "29", "date": "Wed, 23 Jul 2025 10:14:50 GMT"} })
2962025-07-23T10:14:50.214ZDEBGoximeter (oximeter-agent): registered new metric producer address = [fd00:1122:3344:101::a]:40700 collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 503f98a0-54fd-4258-b2d8-a761939d1032
2972025-07-23T10:14:50.216ZDEBGoximeter (oximeter-agent): registered new metric producer address = [fd00:1122:3344:101::2]:8001 collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = a2826c11-418c-4af4-a55c-46138ee2938d
2982025-07-23T10:14:50.218ZDEBGoximeter (oximeter-agent): received request to register existing metric producer, updating collection information address = [fd00:1122:3344:101::2]:37232 collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d interval = 10s producer_id = b1790556-9af9-46ba-8fb7-da641fa5ee9c
2992025-07-23T10:14:50.218ZINFOoximeter (oximeter-agent): refreshed list of producers from Nexus collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d file = oximeter/collector/src/agent.rs:637 n_current_tasks = 6 n_pruned_tasks = 0
3002025-07-23T10:14:50.218ZDEBGoximeter (collection-task): collection task received request to update its producer information address = [fd00:1122:3344:101::2]:37232 collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d interval = 10s producer_id = b1790556-9af9-46ba-8fb7-da641fa5ee9c
3012025-07-23T10:14:50.229ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 882e3042-1c70-490e-9f1e-ab0530fdfade
3022025-07-23T10:14:50.230ZDEBGoximeter (collection-task): collecting from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 882e3042-1c70-490e-9f1e-ab0530fdfade
3032025-07-23T10:14:50.230ZDEBGoximeter (collection-task): collected results from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 882e3042-1c70-490e-9f1e-ab0530fdfade
3042025-07-23T10:14:50.234ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 8d68ccd7-9238-4ce4-93ae-64882012c58c
3052025-07-23T10:14:50.234ZDEBGoximeter (collection-task): collecting from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 8d68ccd7-9238-4ce4-93ae-64882012c58c
3062025-07-23T10:14:50.236ZDEBGoximeter (collection-task): collected results from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 8d68ccd7-9238-4ce4-93ae-64882012c58c
3072025-07-23T10:14:51.217ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = a2826c11-418c-4af4-a55c-46138ee2938d
3082025-07-23T10:14:51.217ZDEBGoximeter (collection-task): collecting from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = a2826c11-418c-4af4-a55c-46138ee2938d
3092025-07-23T10:14:51.227ZDEBGoximeter (collection-task): collected results from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = a2826c11-418c-4af4-a55c-46138ee2938d
3102025-07-23T10:14:51.229ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 882e3042-1c70-490e-9f1e-ab0530fdfade
3112025-07-23T10:14:51.229ZDEBGoximeter (collection-task): collecting from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 882e3042-1c70-490e-9f1e-ab0530fdfade
3122025-07-23T10:14:51.230ZDEBGoximeter (collection-task): collected results from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 882e3042-1c70-490e-9f1e-ab0530fdfade
3132025-07-23T10:14:51.234ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 8d68ccd7-9238-4ce4-93ae-64882012c58c
3142025-07-23T10:14:51.234ZDEBGoximeter (collection-task): collecting from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 8d68ccd7-9238-4ce4-93ae-64882012c58c
3152025-07-23T10:14:51.237ZDEBGoximeter (collection-task): collected results from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 8d68ccd7-9238-4ce4-93ae-64882012c58c
3162025-07-23T10:14:51.401ZDEBGoximeter (results-sink): inserting 164 samples into database collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d
3172025-07-23T10:14:51.401ZDEBGoximeter (results-sink-cluster): inserting 164 samples into database collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d
3182025-07-23T10:14:51.401ZDEBGoximeter (clickhouse-client): unrolling 164 total samples collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d id = 95fa9b41-0e46-4735-9a42-3ca9ffb193f0
3192025-07-23T10:14:51.409ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d id = 95fa9b41-0e46-4735-9a42-3ca9ffb193f0 n_rows = 56 table_name = fields_string
3202025-07-23T10:14:51.411ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d id = 95fa9b41-0e46-4735-9a42-3ca9ffb193f0 n_rows = 64 table_name = fields_uuid
3212025-07-23T10:14:51.412ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d id = 95fa9b41-0e46-4735-9a42-3ca9ffb193f0 n_rows = 120 table_name = measurements_cumulativeu64
3222025-07-23T10:14:51.414ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d id = 95fa9b41-0e46-4735-9a42-3ca9ffb193f0 n_rows = 44 table_name = measurements_u64
3232025-07-23T10:14:51.502ZWARNoximeter (results-sink-cluster): failed to insert some results into metric DB collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
3242025-07-23T10:14:52.218ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = a2826c11-418c-4af4-a55c-46138ee2938d
3252025-07-23T10:14:52.218ZDEBGoximeter (collection-task): collecting from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = a2826c11-418c-4af4-a55c-46138ee2938d
3262025-07-23T10:14:52.220ZDEBGoximeter (collection-task): collected results from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = a2826c11-418c-4af4-a55c-46138ee2938d
3272025-07-23T10:14:52.230ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 882e3042-1c70-490e-9f1e-ab0530fdfade
3282025-07-23T10:14:52.230ZDEBGoximeter (collection-task): collecting from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 882e3042-1c70-490e-9f1e-ab0530fdfade
3292025-07-23T10:14:52.230ZDEBGoximeter (collection-task): collected results from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 882e3042-1c70-490e-9f1e-ab0530fdfade
3302025-07-23T10:14:52.234ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 8d68ccd7-9238-4ce4-93ae-64882012c58c
3312025-07-23T10:14:52.234ZDEBGoximeter (collection-task): collecting from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 8d68ccd7-9238-4ce4-93ae-64882012c58c
3322025-07-23T10:14:52.236ZDEBGoximeter (collection-task): collected results from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 8d68ccd7-9238-4ce4-93ae-64882012c58c
3332025-07-23T10:14:53.218ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = a2826c11-418c-4af4-a55c-46138ee2938d
3342025-07-23T10:14:53.218ZDEBGoximeter (collection-task): collecting from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = a2826c11-418c-4af4-a55c-46138ee2938d
3352025-07-23T10:14:53.220ZDEBGoximeter (collection-task): collected results from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = a2826c11-418c-4af4-a55c-46138ee2938d
3362025-07-23T10:14:53.230ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 882e3042-1c70-490e-9f1e-ab0530fdfade
3372025-07-23T10:14:53.230ZDEBGoximeter (collection-task): collecting from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 882e3042-1c70-490e-9f1e-ab0530fdfade
3382025-07-23T10:14:53.231ZDEBGoximeter (collection-task): collected results from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 882e3042-1c70-490e-9f1e-ab0530fdfade
3392025-07-23T10:14:53.234ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 8d68ccd7-9238-4ce4-93ae-64882012c58c
3402025-07-23T10:14:53.234ZDEBGoximeter (collection-task): collecting from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 8d68ccd7-9238-4ce4-93ae-64882012c58c
3412025-07-23T10:14:53.237ZDEBGoximeter (collection-task): collected results from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 8d68ccd7-9238-4ce4-93ae-64882012c58c
3422025-07-23T10:14:54.218ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = a2826c11-418c-4af4-a55c-46138ee2938d
3432025-07-23T10:14:54.218ZDEBGoximeter (collection-task): collecting from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = a2826c11-418c-4af4-a55c-46138ee2938d
3442025-07-23T10:14:54.219ZDEBGoximeter (collection-task): collected results from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = a2826c11-418c-4af4-a55c-46138ee2938d
3452025-07-23T10:14:54.230ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 882e3042-1c70-490e-9f1e-ab0530fdfade
3462025-07-23T10:14:54.230ZDEBGoximeter (collection-task): collecting from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 882e3042-1c70-490e-9f1e-ab0530fdfade
3472025-07-23T10:14:54.231ZDEBGoximeter (collection-task): collected results from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 882e3042-1c70-490e-9f1e-ab0530fdfade
3482025-07-23T10:14:54.234ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 8d68ccd7-9238-4ce4-93ae-64882012c58c
3492025-07-23T10:14:54.234ZDEBGoximeter (collection-task): collecting from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 8d68ccd7-9238-4ce4-93ae-64882012c58c
3502025-07-23T10:14:54.236ZDEBGoximeter (collection-task): collected results from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 8d68ccd7-9238-4ce4-93ae-64882012c58c
3512025-07-23T10:14:55.219ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = a2826c11-418c-4af4-a55c-46138ee2938d
3522025-07-23T10:14:55.219ZDEBGoximeter (collection-task): collecting from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = a2826c11-418c-4af4-a55c-46138ee2938d
3532025-07-23T10:14:55.220ZDEBGoximeter (collection-task): collected results from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = a2826c11-418c-4af4-a55c-46138ee2938d
3542025-07-23T10:14:55.230ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 882e3042-1c70-490e-9f1e-ab0530fdfade
3552025-07-23T10:14:55.230ZDEBGoximeter (collection-task): collecting from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 882e3042-1c70-490e-9f1e-ab0530fdfade
3562025-07-23T10:14:55.231ZDEBGoximeter (collection-task): collected results from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 882e3042-1c70-490e-9f1e-ab0530fdfade
3572025-07-23T10:14:55.232ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 95583e8c-e579-4eba-a499-5e966e69e3b4
3582025-07-23T10:14:55.232ZDEBGoximeter (collection-task): collecting from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 95583e8c-e579-4eba-a499-5e966e69e3b4
3592025-07-23T10:14:55.234ZDEBGoximeter (collection-task): collected results from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d n_results = 2 producer_id = 95583e8c-e579-4eba-a499-5e966e69e3b4
3602025-07-23T10:14:55.234ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 8d68ccd7-9238-4ce4-93ae-64882012c58c
3612025-07-23T10:14:55.234ZDEBGoximeter (collection-task): collecting from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 8d68ccd7-9238-4ce4-93ae-64882012c58c
3622025-07-23T10:14:55.237ZDEBGoximeter (collection-task): collected results from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 8d68ccd7-9238-4ce4-93ae-64882012c58c
3632025-07-23T10:14:56.218ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = a2826c11-418c-4af4-a55c-46138ee2938d
3642025-07-23T10:14:56.218ZDEBGoximeter (collection-task): collecting from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = a2826c11-418c-4af4-a55c-46138ee2938d
3652025-07-23T10:14:56.220ZDEBGoximeter (collection-task): collected results from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = a2826c11-418c-4af4-a55c-46138ee2938d
3662025-07-23T10:14:56.230ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 882e3042-1c70-490e-9f1e-ab0530fdfade
3672025-07-23T10:14:56.230ZDEBGoximeter (collection-task): collecting from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 882e3042-1c70-490e-9f1e-ab0530fdfade
3682025-07-23T10:14:56.231ZDEBGoximeter (collection-task): collected results from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 882e3042-1c70-490e-9f1e-ab0530fdfade
3692025-07-23T10:14:56.234ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 8d68ccd7-9238-4ce4-93ae-64882012c58c
3702025-07-23T10:14:56.234ZDEBGoximeter (collection-task): collecting from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 8d68ccd7-9238-4ce4-93ae-64882012c58c
3712025-07-23T10:14:56.237ZDEBGoximeter (collection-task): collected results from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 8d68ccd7-9238-4ce4-93ae-64882012c58c
3722025-07-23T10:14:56.401ZDEBGoximeter (results-sink): inserting 230 samples into database collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d
3732025-07-23T10:14:56.401ZDEBGoximeter (results-sink-cluster): inserting 230 samples into database collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d
3742025-07-23T10:14:56.401ZDEBGoximeter (clickhouse-client): unrolling 230 total samples collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d id = 95fa9b41-0e46-4735-9a42-3ca9ffb193f0
3752025-07-23T10:14:56.413ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d id = 95fa9b41-0e46-4735-9a42-3ca9ffb193f0 n_rows = 76 table_name = fields_string
3762025-07-23T10:14:56.414ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d id = 95fa9b41-0e46-4735-9a42-3ca9ffb193f0 n_rows = 10 table_name = fields_u16
3772025-07-23T10:14:56.415ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d id = 95fa9b41-0e46-4735-9a42-3ca9ffb193f0 n_rows = 74 table_name = fields_uuid
3782025-07-23T10:14:56.417ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d id = 95fa9b41-0e46-4735-9a42-3ca9ffb193f0 n_rows = 160 table_name = measurements_cumulativeu64
3792025-07-23T10:14:56.418ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d id = 95fa9b41-0e46-4735-9a42-3ca9ffb193f0 n_rows = 10 table_name = measurements_histogramu64
3802025-07-23T10:14:56.420ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d id = 95fa9b41-0e46-4735-9a42-3ca9ffb193f0 n_rows = 60 table_name = measurements_u64
3812025-07-23T10:14:56.502ZWARNoximeter (results-sink-cluster): failed to insert some results into metric DB collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
3822025-07-23T10:14:57.218ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = a2826c11-418c-4af4-a55c-46138ee2938d
3832025-07-23T10:14:57.218ZDEBGoximeter (collection-task): collecting from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = a2826c11-418c-4af4-a55c-46138ee2938d
3842025-07-23T10:14:57.220ZDEBGoximeter (collection-task): collected results from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = a2826c11-418c-4af4-a55c-46138ee2938d
3852025-07-23T10:14:57.231ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 882e3042-1c70-490e-9f1e-ab0530fdfade
3862025-07-23T10:14:57.231ZDEBGoximeter (collection-task): collecting from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 882e3042-1c70-490e-9f1e-ab0530fdfade
3872025-07-23T10:14:57.232ZDEBGoximeter (collection-task): collected results from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 882e3042-1c70-490e-9f1e-ab0530fdfade
3882025-07-23T10:14:57.234ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 8d68ccd7-9238-4ce4-93ae-64882012c58c
3892025-07-23T10:14:57.234ZDEBGoximeter (collection-task): collecting from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 8d68ccd7-9238-4ce4-93ae-64882012c58c
3902025-07-23T10:14:57.236ZDEBGoximeter (collection-task): collected results from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 8d68ccd7-9238-4ce4-93ae-64882012c58c
3912025-07-23T10:14:58.218ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = a2826c11-418c-4af4-a55c-46138ee2938d
3922025-07-23T10:14:58.218ZDEBGoximeter (collection-task): collecting from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = a2826c11-418c-4af4-a55c-46138ee2938d
3932025-07-23T10:14:58.220ZDEBGoximeter (collection-task): collected results from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = a2826c11-418c-4af4-a55c-46138ee2938d
3942025-07-23T10:14:58.230ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 882e3042-1c70-490e-9f1e-ab0530fdfade
3952025-07-23T10:14:58.230ZDEBGoximeter (collection-task): collecting from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 882e3042-1c70-490e-9f1e-ab0530fdfade
3962025-07-23T10:14:58.231ZDEBGoximeter (collection-task): collected results from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 882e3042-1c70-490e-9f1e-ab0530fdfade
3972025-07-23T10:14:58.234ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 8d68ccd7-9238-4ce4-93ae-64882012c58c
3982025-07-23T10:14:58.234ZDEBGoximeter (collection-task): collecting from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 8d68ccd7-9238-4ce4-93ae-64882012c58c
3992025-07-23T10:14:58.236ZDEBGoximeter (collection-task): collected results from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 8d68ccd7-9238-4ce4-93ae-64882012c58c
4002025-07-23T10:14:59.219ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = a2826c11-418c-4af4-a55c-46138ee2938d
4012025-07-23T10:14:59.219ZDEBGoximeter (collection-task): collecting from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = a2826c11-418c-4af4-a55c-46138ee2938d
4022025-07-23T10:14:59.220ZDEBGoximeter (collection-task): collected results from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = a2826c11-418c-4af4-a55c-46138ee2938d
4032025-07-23T10:14:59.230ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 882e3042-1c70-490e-9f1e-ab0530fdfade
4042025-07-23T10:14:59.230ZDEBGoximeter (collection-task): collecting from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 882e3042-1c70-490e-9f1e-ab0530fdfade
4052025-07-23T10:14:59.231ZDEBGoximeter (collection-task): collected results from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 882e3042-1c70-490e-9f1e-ab0530fdfade
4062025-07-23T10:14:59.234ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 8d68ccd7-9238-4ce4-93ae-64882012c58c
4072025-07-23T10:14:59.234ZDEBGoximeter (collection-task): collecting from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 8d68ccd7-9238-4ce4-93ae-64882012c58c
4082025-07-23T10:14:59.236ZDEBGoximeter (collection-task): collected results from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 8d68ccd7-9238-4ce4-93ae-64882012c58c
4092025-07-23T10:15:00.216ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 503f98a0-54fd-4258-b2d8-a761939d1032
4102025-07-23T10:15:00.216ZDEBGoximeter (collection-task): collecting from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 503f98a0-54fd-4258-b2d8-a761939d1032
4112025-07-23T10:15:00.219ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = a2826c11-418c-4af4-a55c-46138ee2938d
4122025-07-23T10:15:00.219ZDEBGoximeter (collection-task): collecting from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = a2826c11-418c-4af4-a55c-46138ee2938d
4132025-07-23T10:15:00.220ZDEBGoximeter (collection-task): collected results from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = a2826c11-418c-4af4-a55c-46138ee2938d
4142025-07-23T10:15:00.220ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = b1790556-9af9-46ba-8fb7-da641fa5ee9c
4152025-07-23T10:15:00.220ZDEBGoximeter (collection-task): collecting from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = b1790556-9af9-46ba-8fb7-da641fa5ee9c
4162025-07-23T10:15:00.230ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 882e3042-1c70-490e-9f1e-ab0530fdfade
4172025-07-23T10:15:00.230ZDEBGoximeter (collection-task): collecting from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 882e3042-1c70-490e-9f1e-ab0530fdfade
4182025-07-23T10:15:00.231ZDEBGoximeter (collection-task): collected results from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d n_results = 5 producer_id = 503f98a0-54fd-4258-b2d8-a761939d1032
4192025-07-23T10:15:00.231ZDEBGoximeter (collection-task): collected results from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 882e3042-1c70-490e-9f1e-ab0530fdfade
4202025-07-23T10:15:00.235ZDEBGoximeter (collection-task): sent timer-based collection request to the collection task collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 8d68ccd7-9238-4ce4-93ae-64882012c58c
4212025-07-23T10:15:00.235ZDEBGoximeter (collection-task): collecting from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d producer_id = 8d68ccd7-9238-4ce4-93ae-64882012c58c
4222025-07-23T10:15:00.237ZDEBGoximeter (collection-task): collected results from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = 8d68ccd7-9238-4ce4-93ae-64882012c58c
4232025-07-23T10:15:00.252ZDEBGoximeter (collection-task): collected results from producer collector_id = ebee0801-669b-46a1-909b-6aa260b2e1b6 collector_ip = fd00:1122:3344:101::d n_results = 1 producer_id = b1790556-9af9-46ba-8fb7-da641fa5ee9c