Max level shown:
1 [ Feb 15 05:58:36 Enabled. ]
2 [ Feb 15 05:58:36 Rereading configuration. ]
3 [ Feb 15 05:58:40 Rereading configuration. ]
4 [ Feb 15 05:58:44 Executing start method ("ctrun -l child -o noorphan,regent /opt/oxide/oximeter-collector/bin/oximeter run /var/svc/manifest/site/oximeter/config.toml --address [fd00:1122:3344:101::d]:12223 --id 07eefccc-a16d-41be-9df2-0faa4294ae1c &"). ]
5 [ Feb 15 05:58:44 Method "start" exited with status 0. ]
6 note: configured to log to "/dev/stdout"
72024-02-15T05:58:45.063ZDEBGoximeter: registered DTrace probes
82024-02-15T05:58:45.066ZINFOoximeter: starting oximeter server file = oximeter/collector/src/lib.rs:192
92024-02-15T05:58:45.067ZINFOoximeter (DnsResolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = internal-dns/src/resolver.rs:60
102024-02-15T05:58:45.070ZDEBGoximeter: creating ClickHouse client
112024-02-15T05:58:45.070ZDEBGoximeter (DnsResolver): lookup srv dns_name = _clickhouse._tcp.control-plane.oxide.internal
122024-02-15T05:58:45.143ZWARNoximeter (clickhouse-client): failed to read version collector_id = 07eefccc-a16d-41be-9df2-0faa4294ae1c error = Telemetry database unavailable: error sending request for url (http://[fd00:1122:3344:101::e]:8123/?output_format_json_quote_64bit_integers=0): error trying to connect: tcp connect error: Connection refused (os error 146) file = oximeter/db/src/client.rs:868 id = 9b120228-1138-4577-8450-0fb874464397
132024-02-15T05:58:45.143ZWARNoximeter: failed to create ClickHouse client error = Database(DatabaseUnavailable("error sending request for url (http://[fd00:1122:3344:101::e]:8123/?output_format_json_quote_64bit_integers=0): error trying to connect: tcp connect error: Connection refused (os error 146)")) file = oximeter/collector/src/lib.rs:213 retry_after = 143.070453ms
142024-02-15T05:58:45.288ZDEBGoximeter: creating ClickHouse client
152024-02-15T05:58:45.288ZDEBGoximeter (DnsResolver): lookup srv dns_name = _clickhouse._tcp.control-plane.oxide.internal
162024-02-15T05:58:45.321ZWARNoximeter (clickhouse-client): failed to read version collector_id = 07eefccc-a16d-41be-9df2-0faa4294ae1c error = Telemetry database unavailable: error sending request for url (http://[fd00:1122:3344:101::e]:8123/?output_format_json_quote_64bit_integers=0): error trying to connect: tcp connect error: Connection refused (os error 146) file = oximeter/db/src/client.rs:868 id = 27477fdc-bd97-43f6-a76a-6acd4402049d
172024-02-15T05:58:45.322ZWARNoximeter: failed to create ClickHouse client error = Database(DatabaseUnavailable("error sending request for url (http://[fd00:1122:3344:101::e]:8123/?output_format_json_quote_64bit_integers=0): error trying to connect: tcp connect error: Connection refused (os error 146)")) file = oximeter/collector/src/lib.rs:213 retry_after = 682.863552ms
182024-02-15T05:58:46.006ZDEBGoximeter: creating ClickHouse client
192024-02-15T05:58:46.006ZDEBGoximeter (DnsResolver): lookup srv dns_name = _clickhouse._tcp.control-plane.oxide.internal
202024-02-15T05:58:46.037ZWARNoximeter (clickhouse-client): failed to read version collector_id = 07eefccc-a16d-41be-9df2-0faa4294ae1c error = Telemetry database unavailable: error sending request for url (http://[fd00:1122:3344:101::e]:8123/?output_format_json_quote_64bit_integers=0): error trying to connect: tcp connect error: Connection refused (os error 146) file = oximeter/db/src/client.rs:868 id = fad61f5a-83fd-441d-83f6-ac141060a1d6
212024-02-15T05:58:46.038ZWARNoximeter: failed to create ClickHouse client error = Database(DatabaseUnavailable("error sending request for url (http://[fd00:1122:3344:101::e]:8123/?output_format_json_quote_64bit_integers=0): error trying to connect: tcp connect error: Connection refused (os error 146)")) file = oximeter/collector/src/lib.rs:213 retry_after = 1.220628462s
222024-02-15T05:58:47.260ZDEBGoximeter: creating ClickHouse client
232024-02-15T05:58:47.260ZDEBGoximeter (DnsResolver): lookup srv dns_name = _clickhouse._tcp.control-plane.oxide.internal
242024-02-15T05:58:47.317ZWARNoximeter (clickhouse-client): oximeter database does not exist, or is out-of-date collector_id = 07eefccc-a16d-41be-9df2-0faa4294ae1c file = oximeter/db/src/client.rs:864 id = 9e43e1fa-cabb-47e4-956f-d2dd5e6b7504
252024-02-15T05:58:47.317ZDEBGoximeter (oximeter-agent): oximeter database does not exist, creating collector_id = 07eefccc-a16d-41be-9df2-0faa4294ae1c
262024-02-15T05:58:47.324ZINFOoximeter (clickhouse-client): reading db version collector_id = 07eefccc-a16d-41be-9df2-0faa4294ae1c file = oximeter/db/src/client.rs:802 id = 9e43e1fa-cabb-47e4-956f-d2dd5e6b7504
272024-02-15T05:58:47.325ZWARNoximeter (clickhouse-client): oximeter database does not exist, or is out-of-date collector_id = 07eefccc-a16d-41be-9df2-0faa4294ae1c file = oximeter/db/src/client.rs:864 id = 9e43e1fa-cabb-47e4-956f-d2dd5e6b7504
282024-02-15T05:58:47.325ZINFOoximeter (clickhouse-client): read oximeter database version collector_id = 07eefccc-a16d-41be-9df2-0faa4294ae1c file = oximeter/db/src/client.rs:806 id = 9e43e1fa-cabb-47e4-956f-d2dd5e6b7504 version = 0
292024-02-15T05:58:47.325ZINFOoximeter (clickhouse-client): wiping and re-initializing oximeter schema collector_id = 07eefccc-a16d-41be-9df2-0faa4294ae1c file = oximeter/db/src/client.rs:811 id = 9e43e1fa-cabb-47e4-956f-d2dd5e6b7504
302024-02-15T05:58:47.325ZDEBGoximeter (clickhouse-client): wiping ClickHouse database collector_id = 07eefccc-a16d-41be-9df2-0faa4294ae1c id = 9e43e1fa-cabb-47e4-956f-d2dd5e6b7504
312024-02-15T05:58:47.327ZDEBGoximeter (clickhouse-client): initializing ClickHouse database collector_id = 07eefccc-a16d-41be-9df2-0faa4294ae1c id = 9e43e1fa-cabb-47e4-956f-d2dd5e6b7504
322024-02-15T05:58:47.479ZINFOoximeter (clickhouse-client): inserting current version collector_id = 07eefccc-a16d-41be-9df2-0faa4294ae1c file = oximeter/db/src/client.rs:834 id = 9e43e1fa-cabb-47e4-956f-d2dd5e6b7504 version = 4
332024-02-15T05:58:47.488ZDEBGoximeter (dropshot): registered endpoint local_addr = [fd00:1122:3344:101::d]:12223 method = GET path = /info
342024-02-15T05:58:47.488ZDEBGoximeter (dropshot): registered endpoint local_addr = [fd00:1122:3344:101::d]:12223 method = GET path = /producers
352024-02-15T05:58:47.488ZDEBGoximeter (dropshot): registered endpoint local_addr = [fd00:1122:3344:101::d]:12223 method = POST path = /producers
362024-02-15T05:58:47.488ZDEBGoximeter (dropshot): registered endpoint local_addr = [fd00:1122:3344:101::d]:12223 method = DELETE path = /producers/{producer_id}
372024-02-15T05:58:47.488ZINFOoximeter (dropshot): listening file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/711a749/dropshot/src/server.rs:195 local_addr = [fd00:1122:3344:101::d]:12223
382024-02-15T05:58:47.490ZDEBGoximeter (dropshot): successfully registered DTrace USDT probes local_addr = [fd00:1122:3344:101::d]:12223
392024-02-15T05:58:47.522ZDEBGoximeter: contacting nexus
402024-02-15T05:58:47.522ZDEBGoximeter (DnsResolver): lookup_ipv6 srv dns_name = _nexus._tcp.control-plane.oxide.internal
412024-02-15T05:59:12.686ZWARNoximeter: failed to contact nexus, will retry in 324.714669ms error = "error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/collectors): error trying to connect: tcp connect error: Connection refused (os error 146)" file = oximeter/collector/src/lib.rs:271
422024-02-15T05:59:13.011ZDEBGoximeter: contacting nexus
432024-02-15T05:59:13.011ZDEBGoximeter (DnsResolver): lookup_ipv6 srv dns_name = _nexus._tcp.control-plane.oxide.internal
442024-02-15T05:59:13.012ZWARNoximeter: failed to contact nexus, will retry in 263.88108ms error = "error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/collectors): error trying to connect: tcp connect error: Connection refused (os error 146)" file = oximeter/collector/src/lib.rs:271
452024-02-15T05:59:13.277ZDEBGoximeter: contacting nexus
462024-02-15T05:59:13.277ZDEBGoximeter (DnsResolver): lookup_ipv6 srv dns_name = _nexus._tcp.control-plane.oxide.internal
472024-02-15T05:59:13.277ZWARNoximeter: failed to contact nexus, will retry in 1.3784279s error = "error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/collectors): error trying to connect: tcp connect error: Connection refused (os error 146)" file = oximeter/collector/src/lib.rs:271
482024-02-15T05:59:14.658ZDEBGoximeter: contacting nexus
492024-02-15T05:59:14.658ZDEBGoximeter (DnsResolver): lookup_ipv6 srv dns_name = _nexus._tcp.control-plane.oxide.internal
502024-02-15T05:59:14.659ZWARNoximeter: failed to contact nexus, will retry in 2.69313475s error = "error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/collectors): error trying to connect: tcp connect error: Connection refused (os error 146)" file = oximeter/collector/src/lib.rs:271
512024-02-15T05:59:17.354ZDEBGoximeter: contacting nexus
522024-02-15T05:59:17.354ZDEBGoximeter (DnsResolver): lookup_ipv6 srv dns_name = _nexus._tcp.control-plane.oxide.internal
532024-02-15T05:59:17.354ZWARNoximeter: failed to contact nexus, will retry in 2.072123042s error = "error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/collectors): error trying to connect: tcp connect error: Connection refused (os error 146)" file = oximeter/collector/src/lib.rs:271
542024-02-15T05:59:19.427ZDEBGoximeter: contacting nexus
552024-02-15T05:59:19.427ZDEBGoximeter (DnsResolver): lookup_ipv6 srv dns_name = _nexus._tcp.control-plane.oxide.internal
562024-02-15T05:59:19.479ZINFOoximeter: oximeter registered with nexus file = oximeter/collector/src/lib.rs:285 id = 07eefccc-a16d-41be-9df2-0faa4294ae1c
572024-02-15T05:59:27.645ZINFOoximeter (dropshot): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/711a749/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::d]:12223 remote_addr = [fd00:1122:3344:101::c]:40361
582024-02-15T05:59:27.650ZDEBGoximeter (oximeter-agent): registered new metric producer address = [fd00:1122:3344:101::1]:12345 collector_id = 07eefccc-a16d-41be-9df2-0faa4294ae1c producer_id = ec2916b4-23be-4c03-a34b-23d4a0915a29
592024-02-15T05:59:27.650ZINFOoximeter (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/711a749/dropshot/src/server.rs:849 latency_us = 4236 local_addr = [fd00:1122:3344:101::d]:12223 method = POST remote_addr = [fd00:1122:3344:101::c]:40361 req_id = fd3560e7-053a-4f1b-ba08-3fb6d6cfdb95 response_code = 204 uri = /producers
602024-02-15T05:59:27.682ZDEBGoximeter (collection-task): starting oximeter collection task collector_id = 07eefccc-a16d-41be-9df2-0faa4294ae1c interval = 30s producer_id = ec2916b4-23be-4c03-a34b-23d4a0915a29
612024-02-15T05:59:32.072ZINFOoximeter (dropshot): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/711a749/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::d]:12223 remote_addr = [fd00:1122:3344:101::a]:62306
622024-02-15T05:59:32.073ZDEBGoximeter (oximeter-agent): registered new metric producer address = [fd00:1122:3344:101::a]:12221 collector_id = 07eefccc-a16d-41be-9df2-0faa4294ae1c producer_id = 28701ada-e899-402b-a2fd-f7a3fd7c41df
632024-02-15T05:59:32.073ZINFOoximeter (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/711a749/dropshot/src/server.rs:849 latency_us = 114 local_addr = [fd00:1122:3344:101::d]:12223 method = POST remote_addr = [fd00:1122:3344:101::a]:62306 req_id = 864fce74-1a7d-4641-863e-7c91edbe380c response_code = 204 uri = /producers
642024-02-15T05:59:32.105ZDEBGoximeter (collection-task): starting oximeter collection task collector_id = 07eefccc-a16d-41be-9df2-0faa4294ae1c interval = 10s producer_id = 28701ada-e899-402b-a2fd-f7a3fd7c41df
652024-02-15T05:59:42.105ZDEBGoximeter (collection-task): collecting from producer collector_id = 07eefccc-a16d-41be-9df2-0faa4294ae1c producer_id = 28701ada-e899-402b-a2fd-f7a3fd7c41df
662024-02-15T05:59:42.109ZDEBGoximeter (collection-task): collected results from producer collector_id = 07eefccc-a16d-41be-9df2-0faa4294ae1c n_results = 4 producer_id = 28701ada-e899-402b-a2fd-f7a3fd7c41df
672024-02-15T05:59:42.487ZDEBGoximeter (results-sink): inserting 47 samples into database collector_id = 07eefccc-a16d-41be-9df2-0faa4294ae1c
682024-02-15T05:59:42.487ZDEBGoximeter (clickhouse-client): unrolling 47 total samples collector_id = 07eefccc-a16d-41be-9df2-0faa4294ae1c id = 9e43e1fa-cabb-47e4-956f-d2dd5e6b7504
692024-02-15T05:59:42.493ZDEBGoximeter (clickhouse-client): retrieving timeseries schema from database collector_id = 07eefccc-a16d-41be-9df2-0faa4294ae1c id = 9e43e1fa-cabb-47e4-956f-d2dd5e6b7504
702024-02-15T05:59:42.496ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 07eefccc-a16d-41be-9df2-0faa4294ae1c id = 9e43e1fa-cabb-47e4-956f-d2dd5e6b7504 schema = {"timeseries_name":"http_service:request_latency_histogram","fields.name":["id","method","name","route","status_code"],"fields.type":["Uuid","String","String","String","I64"],"fields.source":["Target","Metric","Target","Metric","Metric"],"datum_type":"HistogramF64","created":"2024-02-15 05:59:42.493768771"} timeseries_name = http_service:request_latency_histogram
712024-02-15T05:59:42.499ZDEBGoximeter (clickhouse-client): retrieving timeseries schema from database collector_id = 07eefccc-a16d-41be-9df2-0faa4294ae1c id = 9e43e1fa-cabb-47e4-956f-d2dd5e6b7504
722024-02-15T05:59:42.507ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 07eefccc-a16d-41be-9df2-0faa4294ae1c id = 9e43e1fa-cabb-47e4-956f-d2dd5e6b7504 schema = {"timeseries_name":"collection_target:cpus_provisioned","fields.name":["id"],"fields.type":["Uuid"],"fields.source":["Target"],"datum_type":"I64","created":"2024-02-15 05:59:42.499752597"} timeseries_name = collection_target:cpus_provisioned
732024-02-15T05:59:42.507ZDEBGoximeter (clickhouse-client): retrieving timeseries schema from database collector_id = 07eefccc-a16d-41be-9df2-0faa4294ae1c id = 9e43e1fa-cabb-47e4-956f-d2dd5e6b7504
742024-02-15T05:59:42.508ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 07eefccc-a16d-41be-9df2-0faa4294ae1c id = 9e43e1fa-cabb-47e4-956f-d2dd5e6b7504 schema = {"timeseries_name":"collection_target:ram_provisioned","fields.name":["id"],"fields.type":["Uuid"],"fields.source":["Target"],"datum_type":"I64","created":"2024-02-15 05:59:42.507142580"} timeseries_name = collection_target:ram_provisioned
752024-02-15T05:59:42.509ZDEBGoximeter (clickhouse-client): retrieving timeseries schema from database collector_id = 07eefccc-a16d-41be-9df2-0faa4294ae1c id = 9e43e1fa-cabb-47e4-956f-d2dd5e6b7504
762024-02-15T05:59:42.510ZDEBGoximeter (clickhouse-client): new timeseries schema collector_id = 07eefccc-a16d-41be-9df2-0faa4294ae1c id = 9e43e1fa-cabb-47e4-956f-d2dd5e6b7504 schema = {"timeseries_name":"collection_target:virtual_disk_space_provisioned","fields.name":["id"],"fields.type":["Uuid"],"fields.source":["Target"],"datum_type":"I64","created":"2024-02-15 05:59:42.509024932"} timeseries_name = collection_target:virtual_disk_space_provisioned
772024-02-15T05:59:42.511ZDEBGoximeter (clickhouse-client): inserting 4 new timeseries schema collector_id = 07eefccc-a16d-41be-9df2-0faa4294ae1c id = 9e43e1fa-cabb-47e4-956f-d2dd5e6b7504
782024-02-15T05:59:42.518ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 07eefccc-a16d-41be-9df2-0faa4294ae1c id = 9e43e1fa-cabb-47e4-956f-d2dd5e6b7504 n_rows = 29 table_name = oximeter.fields_i64
792024-02-15T05:59:42.520ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 07eefccc-a16d-41be-9df2-0faa4294ae1c id = 9e43e1fa-cabb-47e4-956f-d2dd5e6b7504 n_rows = 87 table_name = oximeter.fields_string
802024-02-15T05:59:42.525ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 07eefccc-a16d-41be-9df2-0faa4294ae1c id = 9e43e1fa-cabb-47e4-956f-d2dd5e6b7504 n_rows = 39 table_name = oximeter.fields_uuid
812024-02-15T05:59:42.527ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 07eefccc-a16d-41be-9df2-0faa4294ae1c id = 9e43e1fa-cabb-47e4-956f-d2dd5e6b7504 n_rows = 29 table_name = oximeter.measurements_histogramf64
822024-02-15T05:59:42.529ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 07eefccc-a16d-41be-9df2-0faa4294ae1c id = 9e43e1fa-cabb-47e4-956f-d2dd5e6b7504 n_rows = 18 table_name = oximeter.measurements_i64
832024-02-15T05:59:52.106ZDEBGoximeter (collection-task): collecting from producer collector_id = 07eefccc-a16d-41be-9df2-0faa4294ae1c producer_id = 28701ada-e899-402b-a2fd-f7a3fd7c41df
842024-02-15T05:59:52.108ZDEBGoximeter (collection-task): collected results from producer collector_id = 07eefccc-a16d-41be-9df2-0faa4294ae1c n_results = 4 producer_id = 28701ada-e899-402b-a2fd-f7a3fd7c41df
852024-02-15T05:59:52.487ZDEBGoximeter (results-sink): inserting 31 samples into database collector_id = 07eefccc-a16d-41be-9df2-0faa4294ae1c
862024-02-15T05:59:52.487ZDEBGoximeter (clickhouse-client): unrolling 31 total samples collector_id = 07eefccc-a16d-41be-9df2-0faa4294ae1c id = 9e43e1fa-cabb-47e4-956f-d2dd5e6b7504
872024-02-15T05:59:52.492ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 07eefccc-a16d-41be-9df2-0faa4294ae1c id = 9e43e1fa-cabb-47e4-956f-d2dd5e6b7504 n_rows = 31 table_name = oximeter.fields_i64
882024-02-15T05:59:52.493ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 07eefccc-a16d-41be-9df2-0faa4294ae1c id = 9e43e1fa-cabb-47e4-956f-d2dd5e6b7504 n_rows = 93 table_name = oximeter.fields_string
892024-02-15T05:59:52.494ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 07eefccc-a16d-41be-9df2-0faa4294ae1c id = 9e43e1fa-cabb-47e4-956f-d2dd5e6b7504 n_rows = 31 table_name = oximeter.fields_uuid
902024-02-15T05:59:52.496ZDEBGoximeter (clickhouse-client): inserted rows into table collector_id = 07eefccc-a16d-41be-9df2-0faa4294ae1c id = 9e43e1fa-cabb-47e4-956f-d2dd5e6b7504 n_rows = 31 table_name = oximeter.measurements_histogramf64
912024-02-15T05:59:57.682ZDEBGoximeter (collection-task): collecting from producer collector_id = 07eefccc-a16d-41be-9df2-0faa4294ae1c producer_id = ec2916b4-23be-4c03-a34b-23d4a0915a29
922024-02-15T05:59:57.689ZDEBGoximeter (collection-task): collected results from producer collector_id = 07eefccc-a16d-41be-9df2-0faa4294ae1c n_results = 1 producer_id = ec2916b4-23be-4c03-a34b-23d4a0915a29
932024-02-15T06:00:02.105ZDEBGoximeter (collection-task): collecting from producer collector_id = 07eefccc-a16d-41be-9df2-0faa4294ae1c producer_id = 28701ada-e899-402b-a2fd-f7a3fd7c41df
942024-02-15T06:00:02.168ZDEBGoximeter (collection-task): collected results from producer collector_id = 07eefccc-a16d-41be-9df2-0faa4294ae1c n_results = 4 producer_id = 28701ada-e899-402b-a2fd-f7a3fd7c41df