2025.01.11 00:30:06.839079 [ 32661 ] {} Application: Will watch for the process with pid 32666 2025.01.11 00:30:06.839106 [ 32666 ] {} Application: Forked a child process to watch 2025.01.11 00:30:06.839276 [ 32666 ] {} SentryWriter: Sending crash reports is disabled 2025.01.11 00:30:06.839385 [ 32666 ] {} Pipe: Pipe capacity is 1.00 MiB 2025.01.11 00:30:07.690083 [ 32666 ] {} Application: Starting ClickHouse 23.8.7.24 (revision: 54477, git hash: 812b95e14ba8cf744bf1d70e6de607cf130a79fa, build id: 6484B75ECB16F4FF87A88E10249A45E1BDBDE2E5), PID 32666 2025.01.11 00:30:07.794108 [ 32666 ] {} Application: starting up 2025.01.11 00:30:07.794141 [ 32666 ] {} Application: OS name: Linux, version: 5.15.0-1015-aws, architecture: x86_64 2025.01.11 00:30:09.773534 [ 32666 ] {} Application: Available RAM: 30.68 GiB; physical cores: 8; logical cores: 8. 2025.01.11 00:30:09.795023 [ 32666 ] {} AsynchronousMetrics: Scanning /sys/class/thermal 2025.01.11 00:30:09.795042 [ 32666 ] {} AsynchronousMetrics: Scanning /sys/block 2025.01.11 00:30:09.795081 [ 32666 ] {} AsynchronousMetrics: Scanning /sys/devices/system/edac 2025.01.11 00:30:09.795095 [ 32666 ] {} AsynchronousMetrics: Scanning /sys/class/hwmon 2025.01.11 00:30:19.869478 [ 32666 ] {} Application: Integrity check of the executable successfully passed (checksum: 63E397DB90569A93F13765E06F4A1FBF) 2025.01.11 00:30:19.877334 [ 32666 ] {} Application: Set max number of file descriptors to 524288 (was 1024). 2025.01.11 00:30:19.877345 [ 32666 ] {} Application: rlimit on number of threads is 125612 2025.01.11 00:30:19.877351 [ 32666 ] {} Application: Initializing DateLUT. 2025.01.11 00:30:19.877355 [ 32666 ] {} Application: Initialized DateLUT with time zone 'UTC'. 2025.01.11 00:30:19.877372 [ 32666 ] {} Context: Setting up /var/tmp/omicron_tmp/clickward_test/clickhouse-1/data/tmp/ to store temporary data in it 2025.01.11 00:30:19.877630 [ 32666 ] {} Application: Initializing interserver credentials. 2025.01.11 00:30:19.878146 [ 32666 ] {} ConfigReloader: Loading config '/var/tmp/omicron_tmp/clickward_test/clickhouse-1/clickhouse-config.xml' 2025.01.11 00:30:19.878871 [ 32666 ] {} ConfigReloader: Loaded config '/var/tmp/omicron_tmp/clickward_test/clickhouse-1/clickhouse-config.xml', performing update on configuration 2025.01.11 00:30:19.879453 [ 32666 ] {} Application: Setting max_server_memory_usage was set to 27.61 GiB (30.68 GiB available * 0.90 max_server_memory_usage_to_ram_ratio) 2025.01.11 00:30:19.879463 [ 32666 ] {} Application: Setting merges_mutations_memory_usage_soft_limit was set to 15.34 GiB (30.68 GiB available * 0.50 merges_mutations_memory_usage_to_ram_ratio) 2025.01.11 00:30:19.879468 [ 32666 ] {} Application: Merges and mutations memory limit is set to 15.34 GiB 2025.01.11 00:30:19.879676 [ 32666 ] {} BackgroundSchedulePool/BgBufSchPool: Create BackgroundSchedulePool with 16 threads 2025.01.11 00:30:19.880450 [ 32666 ] {} BackgroundSchedulePool/BgSchPool: Create BackgroundSchedulePool with 128 threads 2025.01.11 00:30:19.887963 [ 32666 ] {} BackgroundSchedulePool/BgMBSchPool: Create BackgroundSchedulePool with 16 threads 2025.01.11 00:30:19.888730 [ 32666 ] {} BackgroundSchedulePool/BgDistSchPool: Create BackgroundSchedulePool with 16 threads 2025.01.11 00:30:19.898803 [ 32666 ] {} CertificateReloader: One of paths is empty. Cannot apply new configuration for certificates. Fill all paths and try again. 2025.01.11 00:30:19.898829 [ 32666 ] {} ConfigReloader: Loaded config '/var/tmp/omicron_tmp/clickward_test/clickhouse-1/clickhouse-config.xml', performed update on configuration 2025.01.11 00:30:19.898990 [ 32666 ] {} Application: Listening for replica communication (interserver): http://[::1]:29401 2025.01.11 00:30:19.900573 [ 32666 ] {} ConfigReloader: Loading config '/var/tmp/omicron_tmp/clickward_test/clickhouse-1/clickhouse-config.xml' 2025.01.11 00:30:19.901081 [ 32666 ] {} ConfigReloader: Loaded config '/var/tmp/omicron_tmp/clickward_test/clickhouse-1/clickhouse-config.xml', performing update on configuration 2025.01.11 00:30:19.901825 [ 32666 ] {} ConfigReloader: Loaded config '/var/tmp/omicron_tmp/clickward_test/clickhouse-1/clickhouse-config.xml', performed update on configuration 2025.01.11 00:30:19.902052 [ 32666 ] {} Access(user directories): Added users_xml access storage 'users_xml', path: /var/tmp/omicron_tmp/clickward_test/clickhouse-1/clickhouse-config.xml 2025.01.11 00:30:19.902787 [ 32666 ] {} Context: Initialized background executor for merges and mutations with num_threads=16, num_tasks=32, scheduling_policy=round_robin 2025.01.11 00:30:19.904013 [ 32666 ] {} Context: Initialized background executor for move operations with num_threads=8, num_tasks=8 2025.01.11 00:30:19.904318 [ 32666 ] {} Context: Initialized background executor for fetches with num_threads=8, num_tasks=8 2025.01.11 00:30:19.904681 [ 32666 ] {} Context: Initialized background executor for common operations (e.g. clearing old parts) with num_threads=8, num_tasks=8 2025.01.11 00:30:19.907709 [ 32666 ] {} DNSCacheUpdater: Update period 15 seconds 2025.01.11 00:30:19.907730 [ 32666 ] {} Application: Loading metadata from /var/tmp/omicron_tmp/clickward_test/clickhouse-1/data/ 2025.01.11 00:30:19.907915 [ 32822 ] {} DNSResolver: Updating DNS cache 2025.01.11 00:30:19.908071 [ 32822 ] {} DNSResolver: Updated DNS cache 2025.01.11 00:30:19.923312 [ 32666 ] {} DatabaseAtomic (system): Metadata processed, database system has 0 tables and 0 dictionaries in total. 2025.01.11 00:30:19.923322 [ 32666 ] {} TablesLoader: Parsed metadata of 0 tables in 1 databases in 4.792e-05 sec 2025.01.11 00:30:19.923326 [ 32666 ] {} ReferentialDeps: No tables 2025.01.11 00:30:19.923329 [ 32666 ] {} LoadingDeps: No tables 2025.01.11 00:30:19.923346 [ 32666 ] {} DatabaseAtomic (system): Starting up tables. 2025.01.11 00:30:19.923356 [ 32666 ] {} SystemLog: Not creating system.query_log since corresponding section 'query_log' is missing from config 2025.01.11 00:30:19.923361 [ 32666 ] {} SystemLog: Not creating system.query_thread_log since corresponding section 'query_thread_log' is missing from config 2025.01.11 00:30:19.923364 [ 32666 ] {} SystemLog: Not creating system.part_log since corresponding section 'part_log' is missing from config 2025.01.11 00:30:19.923368 [ 32666 ] {} SystemLog: Not creating system.trace_log since corresponding section 'trace_log' is missing from config 2025.01.11 00:30:19.923372 [ 32666 ] {} SystemLog: Not creating system.crash_log since corresponding section 'crash_log' is missing from config 2025.01.11 00:30:19.923377 [ 32666 ] {} SystemLog: Not creating system.text_log since corresponding section 'text_log' is missing from config 2025.01.11 00:30:19.923383 [ 32666 ] {} SystemLog: Creating system.metric_log from metric_log 2025.01.11 00:30:19.924920 [ 32666 ] {} SystemLog: Not creating system.filesystem_cache_log since corresponding section 'filesystem_cache_log' is missing from config 2025.01.11 00:30:19.924930 [ 32666 ] {} SystemLog: Not creating system.filesystem_read_prefetches_log since corresponding section 'filesystem_read_prefetches_log' is missing from config 2025.01.11 00:30:19.924934 [ 32666 ] {} SystemLog: Creating system.asynchronous_metric_log from asynchronous_metric_log 2025.01.11 00:30:19.925042 [ 32666 ] {} SystemLog: Creating system.opentelemetry_span_log from opentelemetry_span_log 2025.01.11 00:30:19.925171 [ 32666 ] {} SystemLog: Not creating system.query_views_log since corresponding section 'query_views_log' is missing from config 2025.01.11 00:30:19.925178 [ 32666 ] {} SystemLog: Not creating system.zookeeper_log since corresponding section 'zookeeper_log' is missing from config 2025.01.11 00:30:19.925186 [ 32666 ] {} SystemLog: Not creating system.session_log since corresponding section 'session_log' is missing from config 2025.01.11 00:30:19.925189 [ 32666 ] {} SystemLog: Not creating system.transactions_info_log since corresponding section 'transactions_info_log' is missing from config 2025.01.11 00:30:19.925192 [ 32666 ] {} SystemLog: Not creating system.processors_profile_log since corresponding section 'processors_profile_log' is missing from config 2025.01.11 00:30:19.925199 [ 32666 ] {} SystemLog: Not creating system.asynchronous_insert_log since corresponding section 'asynchronous_insert_log' is missing from config 2025.01.11 00:30:19.934264 [ 32666 ] {} DatabaseCatalog: Found 0 partially dropped tables. Will load them and retry removal. 2025.01.11 00:30:19.938092 [ 32666 ] {} DatabaseAtomic (default): Metadata processed, database default has 0 tables and 0 dictionaries in total. 2025.01.11 00:30:19.938104 [ 32666 ] {} TablesLoader: Parsed metadata of 0 tables in 1 databases in 3.68e-05 sec 2025.01.11 00:30:19.938110 [ 32666 ] {} ReferentialDeps: No tables 2025.01.11 00:30:19.938115 [ 32666 ] {} LoadingDeps: No tables 2025.01.11 00:30:19.938130 [ 32666 ] {} DatabaseAtomic (default): Starting up tables. 2025.01.11 00:30:19.938214 [ 32666 ] {} UserDefinedSQLObjectsLoaderFromDisk: Loading user defined objects from /var/tmp/omicron_tmp/clickward_test/clickhouse-1/data/user_defined/ 2025.01.11 00:30:19.938238 [ 32666 ] {} UserDefinedSQLObjectsLoaderFromDisk: User defined objects loaded 2025.01.11 00:30:19.938242 [ 32666 ] {} Application: Loaded metadata. 2025.01.11 00:30:19.938257 [ 32666 ] {} Application: Tasks stats provider: procfs 2025.01.11 00:30:19.938263 [ 32666 ] {} Application: It looks like the process has no CAP_SYS_NICE capability, the setting 'os_thread_priority' will have no effect. It could happen due to incorrect ClickHouse package installation. You could resolve the problem manually with 'sudo setcap cap_sys_nice=+ep /work/oxidecomputer/omicron/out/clickhouse/clickhouse'. Note that it will not work on 'nosuid' mounted filesystems. 2025.01.11 00:30:19.938335 [ 32666 ] {} CertificateReloader: One of paths is empty. Cannot apply new configuration for certificates. Fill all paths and try again. 2025.01.11 00:30:19.938704 [ 32666 ] {} AsynchronousMetrics: MemoryTracking: was 12.57 MiB, peak 13.26 MiB, free memory in arenas 22.28 MiB, will set to 385.19 MiB (RSS), difference: 372.61 MiB 2025.01.11 00:30:19.938995 [ 32666 ] {} AsynchronousMetrics: Update heavy metrics. Update period 1 sec. Update heavy metrics period 120 sec. Heavy metrics calculation elapsed: 1.09e-05 sec. 2025.01.11 00:30:19.940738 [ 32666 ] {} Application: Listening for http://[::1]:29301 2025.01.11 00:30:19.940743 [ 33250 ] {} DDLWorker: Starting DDLWorker thread 2025.01.11 00:30:19.940791 [ 33250 ] {} DDLWorker: Initializing DDLWorker thread 2025.01.11 00:30:19.940791 [ 33251 ] {} DDLWorker: Started DDLWorker cleanup thread 2025.01.11 00:30:19.940827 [ 32666 ] {} Application: Listening for native protocol (tcp): [::1]:29201 2025.01.11 00:30:19.948279 [ 33250 ] {} ZooKeeperClient: Connected to ZooKeeper at [::1]:29001 with session_id 2 2025.01.11 00:30:19.948686 [ 33250 ] {} ZooKeeperClient: Keeper feature flag FILTERED_LIST: enabled 2025.01.11 00:30:19.948695 [ 33250 ] {} ZooKeeperClient: Keeper feature flag MULTI_READ: enabled 2025.01.11 00:30:19.948698 [ 33250 ] {} ZooKeeperClient: Keeper feature flag CHECK_NOT_EXISTS: disabled 2025.01.11 00:30:19.948701 [ 33250 ] {} ZooKeeper: Initialized, hosts: [::1]:29001,[::1]:29002,[::1]:29003 2025.01.11 00:30:19.948723 [ 32666 ] {} Application: Ready for connections. 2025.01.11 00:30:19.966246 [ 33250 ] {} DDLWorker: Initialized DDLWorker thread 2025.01.11 00:30:19.966259 [ 33250 ] {} DDLWorker: Scheduling tasks 2025.01.11 00:30:19.966263 [ 33250 ] {} DDLWorker: Don't have unfinished tasks after restarting 2025.01.11 00:30:19.966269 [ 33251 ] {} DDLWorker: Cleaning queue 2025.01.11 00:30:19.966453 [ 33250 ] {} DDLWorker: scheduleTasks: initialized=true, size_before_filtering=0, queue_size=0, entries=none..none, first_failed_task_name=none, current_tasks_size=0, last_current_task=none, last_skipped_entry_name=none 2025.01.11 00:30:19.966465 [ 33250 ] {} DDLWorker: No tasks to schedule 2025.01.11 00:30:19.966470 [ 33250 ] {} DDLWorker: Waiting for queue updates 2025.01.11 00:30:27.425538 [ 33235 ] {} SystemLog (system.metric_log): Flushing system log, 8 entries to flush up to offset 8 2025.01.11 00:30:27.425589 [ 33235 ] {} SystemLog (system.metric_log): Creating new table system.metric_log for MetricLog 2025.01.11 00:30:27.425568 [ 33234 ] {} SystemLog (system.asynchronous_metric_log): Flushing system log, 1953 entries to flush up to offset 1953 2025.01.11 00:30:27.425636 [ 33234 ] {} SystemLog (system.asynchronous_metric_log): Creating new table system.asynchronous_metric_log for AsynchronousMetricLog 2025.01.11 00:30:27.506128 [ 33234 ] {} StoragePolicy (default): Storage policy default created, total volumes 1 2025.01.11 00:30:27.511573 [ 33234 ] {} system.asynchronous_metric_log (7c1cab3d-a87a-4959-8d08-f2ab3db7432e): Loading data parts 2025.01.11 00:30:27.512737 [ 33234 ] {} system.asynchronous_metric_log (7c1cab3d-a87a-4959-8d08-f2ab3db7432e): There are no data parts 2025.01.11 00:30:27.513735 [ 33235 ] {} system.metric_log (c926069d-e416-444a-b525-0c878d521725): Loading data parts 2025.01.11 00:30:27.513809 [ 33235 ] {} system.metric_log (c926069d-e416-444a-b525-0c878d521725): There are no data parts 2025.01.11 00:30:27.519280 [ 33234 ] {} system.asynchronous_metric_log (7c1cab3d-a87a-4959-8d08-f2ab3db7432e): Trying to reserve 1.00 MiB using storage policy from min volume index 0 2025.01.11 00:30:27.519298 [ 33234 ] {} DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 163.51 GiB. 2025.01.11 00:30:27.520700 [ 33234 ] {} MergedBlockOutputStream: filled checksums 202501_1_1_0 (state Temporary) 2025.01.11 00:30:27.521744 [ 33234 ] {} system.asynchronous_metric_log (7c1cab3d-a87a-4959-8d08-f2ab3db7432e): Renaming temporary part tmp_insert_202501_1_1_0 to 202501_1_1_0 with tid (1, 1, 00000000-0000-0000-0000-000000000000). 2025.01.11 00:30:27.521876 [ 33234 ] {} SystemLog (system.asynchronous_metric_log): Flushed system log up to offset 1953 2025.01.11 00:30:27.530516 [ 33235 ] {} system.metric_log (c926069d-e416-444a-b525-0c878d521725): Trying to reserve 1.00 MiB using storage policy from min volume index 0 2025.01.11 00:30:27.530532 [ 33235 ] {} DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 163.51 GiB. 2025.01.11 00:30:27.536373 [ 33235 ] {} MergedBlockOutputStream: filled checksums 202501_1_1_0 (state Temporary) 2025.01.11 00:30:27.540385 [ 33235 ] {} system.metric_log (c926069d-e416-444a-b525-0c878d521725): Renaming temporary part tmp_insert_202501_1_1_0 to 202501_1_1_0 with tid (1, 1, 00000000-0000-0000-0000-000000000000). 2025.01.11 00:30:27.541490 [ 33235 ] {} SystemLog (system.metric_log): Flushed system log up to offset 8 2025.01.11 00:30:28.000485 [ 33248 ] {} AsynchronousMetrics: MemoryTracking: was 389.58 MiB, peak 398.36 MiB, free memory in arenas 29.43 MiB, will set to 388.10 MiB (RSS), difference: -1.48 MiB 2025.01.11 00:30:34.908190 [ 32895 ] {} DNSResolver: Updating DNS cache 2025.01.11 00:30:34.908223 [ 32895 ] {} DNSResolver: Updated DNS cache 2025.01.11 00:30:35.021995 [ 33234 ] {} SystemLog (system.asynchronous_metric_log): Flushing system log, 1848 entries to flush up to offset 3801 2025.01.11 00:30:35.023484 [ 33234 ] {} system.asynchronous_metric_log (7c1cab3d-a87a-4959-8d08-f2ab3db7432e): Trying to reserve 1.00 MiB using storage policy from min volume index 0 2025.01.11 00:30:35.023504 [ 33234 ] {} DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 163.51 GiB. 2025.01.11 00:30:35.024014 [ 33234 ] {} MergedBlockOutputStream: filled checksums 202501_2_2_0 (state Temporary) 2025.01.11 00:30:35.024232 [ 33234 ] {} system.asynchronous_metric_log (7c1cab3d-a87a-4959-8d08-f2ab3db7432e): Renaming temporary part tmp_insert_202501_2_2_0 to 202501_2_2_0 with tid (1, 1, 00000000-0000-0000-0000-000000000000). 2025.01.11 00:30:35.024344 [ 33234 ] {} SystemLog (system.asynchronous_metric_log): Flushed system log up to offset 3801 2025.01.11 00:30:35.041580 [ 33235 ] {} SystemLog (system.metric_log): Flushing system log, 8 entries to flush up to offset 16 2025.01.11 00:30:35.052294 [ 33235 ] {} system.metric_log (c926069d-e416-444a-b525-0c878d521725): Trying to reserve 1.00 MiB using storage policy from min volume index 0 2025.01.11 00:30:35.052314 [ 33235 ] {} DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 163.51 GiB. 2025.01.11 00:30:35.057014 [ 33235 ] {} MergedBlockOutputStream: filled checksums 202501_2_2_0 (state Temporary) 2025.01.11 00:30:35.060957 [ 33235 ] {} system.metric_log (c926069d-e416-444a-b525-0c878d521725): Renaming temporary part tmp_insert_202501_2_2_0 to 202501_2_2_0 with tid (1, 1, 00000000-0000-0000-0000-000000000000). 2025.01.11 00:30:35.062021 [ 33235 ] {} SystemLog (system.metric_log): Flushed system log up to offset 16