Max level shown:
1 2025.01.11 00:30:06.839079 [ 32661 ] {} <Information> Application: Will watch for the process with pid 32666
2 2025.01.11 00:30:06.839106 [ 32666 ] {} <Information> Application: Forked a child process to watch
3 2025.01.11 00:30:06.839276 [ 32666 ] {} <Information> SentryWriter: Sending crash reports is disabled
4 2025.01.11 00:30:06.839385 [ 32666 ] {} <Trace> Pipe: Pipe capacity is 1.00 MiB
5 2025.01.11 00:30:07.690083 [ 32666 ] {} <Information> Application: Starting ClickHouse 23.8.7.24 (revision: 54477, git hash: 812b95e14ba8cf744bf1d70e6de607cf130a79fa, build id: 6484B75ECB16F4FF87A88E10249A45E1BDBDE2E5), PID 32666
6 2025.01.11 00:30:07.794108 [ 32666 ] {} <Information> Application: starting up
7 2025.01.11 00:30:07.794141 [ 32666 ] {} <Information> Application: OS name: Linux, version: 5.15.0-1015-aws, architecture: x86_64
8 2025.01.11 00:30:09.773534 [ 32666 ] {} <Information> Application: Available RAM: 30.68 GiB; physical cores: 8; logical cores: 8.
9 2025.01.11 00:30:09.795023 [ 32666 ] {} <Trace> AsynchronousMetrics: Scanning /sys/class/thermal
10 2025.01.11 00:30:09.795042 [ 32666 ] {} <Trace> AsynchronousMetrics: Scanning /sys/block
11 2025.01.11 00:30:09.795081 [ 32666 ] {} <Trace> AsynchronousMetrics: Scanning /sys/devices/system/edac
12 2025.01.11 00:30:09.795095 [ 32666 ] {} <Trace> AsynchronousMetrics: Scanning /sys/class/hwmon
13 2025.01.11 00:30:19.869478 [ 32666 ] {} <Information> Application: Integrity check of the executable successfully passed (checksum: 63E397DB90569A93F13765E06F4A1FBF)
14 2025.01.11 00:30:19.877334 [ 32666 ] {} <Debug> Application: Set max number of file descriptors to 524288 (was 1024).
15 2025.01.11 00:30:19.877345 [ 32666 ] {} <Debug> Application: rlimit on number of threads is 125612
16 2025.01.11 00:30:19.877351 [ 32666 ] {} <Debug> Application: Initializing DateLUT.
17 2025.01.11 00:30:19.877355 [ 32666 ] {} <Trace> Application: Initialized DateLUT with time zone 'UTC'.
18 2025.01.11 00:30:19.877372 [ 32666 ] {} <Debug> Context: Setting up /var/tmp/omicron_tmp/clickward_test/clickhouse-1/data/tmp/ to store temporary data in it
19 2025.01.11 00:30:19.877630 [ 32666 ] {} <Debug> Application: Initializing interserver credentials.
20 2025.01.11 00:30:19.878146 [ 32666 ] {} <Debug> ConfigReloader: Loading config '/var/tmp/omicron_tmp/clickward_test/clickhouse-1/clickhouse-config.xml'
21 2025.01.11 00:30:19.878871 [ 32666 ] {} <Debug> ConfigReloader: Loaded config '/var/tmp/omicron_tmp/clickward_test/clickhouse-1/clickhouse-config.xml', performing update on configuration
22 2025.01.11 00:30:19.879453 [ 32666 ] {} <Information> Application: Setting max_server_memory_usage was set to 27.61 GiB (30.68 GiB available * 0.90 max_server_memory_usage_to_ram_ratio)
23 2025.01.11 00:30:19.879463 [ 32666 ] {} <Information> Application: Setting merges_mutations_memory_usage_soft_limit was set to 15.34 GiB (30.68 GiB available * 0.50 merges_mutations_memory_usage_to_ram_ratio)
24 2025.01.11 00:30:19.879468 [ 32666 ] {} <Information> Application: Merges and mutations memory limit is set to 15.34 GiB
25 2025.01.11 00:30:19.879676 [ 32666 ] {} <Information> BackgroundSchedulePool/BgBufSchPool: Create BackgroundSchedulePool with 16 threads
26 2025.01.11 00:30:19.880450 [ 32666 ] {} <Information> BackgroundSchedulePool/BgSchPool: Create BackgroundSchedulePool with 128 threads
27 2025.01.11 00:30:19.887963 [ 32666 ] {} <Information> BackgroundSchedulePool/BgMBSchPool: Create BackgroundSchedulePool with 16 threads
28 2025.01.11 00:30:19.888730 [ 32666 ] {} <Information> BackgroundSchedulePool/BgDistSchPool: Create BackgroundSchedulePool with 16 threads
29 2025.01.11 00:30:19.898803 [ 32666 ] {} <Information> CertificateReloader: One of paths is empty. Cannot apply new configuration for certificates. Fill all paths and try again.
30 2025.01.11 00:30:19.898829 [ 32666 ] {} <Debug> ConfigReloader: Loaded config '/var/tmp/omicron_tmp/clickward_test/clickhouse-1/clickhouse-config.xml', performed update on configuration
31 2025.01.11 00:30:19.898990 [ 32666 ] {} <Information> Application: Listening for replica communication (interserver): http://[::1]:29401
32 2025.01.11 00:30:19.900573 [ 32666 ] {} <Debug> ConfigReloader: Loading config '/var/tmp/omicron_tmp/clickward_test/clickhouse-1/clickhouse-config.xml'
33 2025.01.11 00:30:19.901081 [ 32666 ] {} <Debug> ConfigReloader: Loaded config '/var/tmp/omicron_tmp/clickward_test/clickhouse-1/clickhouse-config.xml', performing update on configuration
34 2025.01.11 00:30:19.901825 [ 32666 ] {} <Debug> ConfigReloader: Loaded config '/var/tmp/omicron_tmp/clickward_test/clickhouse-1/clickhouse-config.xml', performed update on configuration
35 2025.01.11 00:30:19.902052 [ 32666 ] {} <Debug> Access(user directories): Added users_xml access storage 'users_xml', path: /var/tmp/omicron_tmp/clickward_test/clickhouse-1/clickhouse-config.xml
36 2025.01.11 00:30:19.902787 [ 32666 ] {} <Information> Context: Initialized background executor for merges and mutations with num_threads=16, num_tasks=32, scheduling_policy=round_robin
37 2025.01.11 00:30:19.904013 [ 32666 ] {} <Information> Context: Initialized background executor for move operations with num_threads=8, num_tasks=8
38 2025.01.11 00:30:19.904318 [ 32666 ] {} <Information> Context: Initialized background executor for fetches with num_threads=8, num_tasks=8
39 2025.01.11 00:30:19.904681 [ 32666 ] {} <Information> Context: Initialized background executor for common operations (e.g. clearing old parts) with num_threads=8, num_tasks=8
40 2025.01.11 00:30:19.907709 [ 32666 ] {} <Information> DNSCacheUpdater: Update period 15 seconds
41 2025.01.11 00:30:19.907730 [ 32666 ] {} <Information> Application: Loading metadata from /var/tmp/omicron_tmp/clickward_test/clickhouse-1/data/
42 2025.01.11 00:30:19.907915 [ 32822 ] {} <Debug> DNSResolver: Updating DNS cache
43 2025.01.11 00:30:19.908071 [ 32822 ] {} <Debug> DNSResolver: Updated DNS cache
44 2025.01.11 00:30:19.923312 [ 32666 ] {} <Information> DatabaseAtomic (system): Metadata processed, database system has 0 tables and 0 dictionaries in total.
45 2025.01.11 00:30:19.923322 [ 32666 ] {} <Information> TablesLoader: Parsed metadata of 0 tables in 1 databases in 4.792e-05 sec
46 2025.01.11 00:30:19.923326 [ 32666 ] {} <Trace> ReferentialDeps: No tables
47 2025.01.11 00:30:19.923329 [ 32666 ] {} <Trace> LoadingDeps: No tables
48 2025.01.11 00:30:19.923346 [ 32666 ] {} <Information> DatabaseAtomic (system): Starting up tables.
49 2025.01.11 00:30:19.923356 [ 32666 ] {} <Debug> SystemLog: Not creating system.query_log since corresponding section 'query_log' is missing from config
50 2025.01.11 00:30:19.923361 [ 32666 ] {} <Debug> SystemLog: Not creating system.query_thread_log since corresponding section 'query_thread_log' is missing from config
51 2025.01.11 00:30:19.923364 [ 32666 ] {} <Debug> SystemLog: Not creating system.part_log since corresponding section 'part_log' is missing from config
52 2025.01.11 00:30:19.923368 [ 32666 ] {} <Debug> SystemLog: Not creating system.trace_log since corresponding section 'trace_log' is missing from config
53 2025.01.11 00:30:19.923372 [ 32666 ] {} <Debug> SystemLog: Not creating system.crash_log since corresponding section 'crash_log' is missing from config
54 2025.01.11 00:30:19.923377 [ 32666 ] {} <Debug> SystemLog: Not creating system.text_log since corresponding section 'text_log' is missing from config
55 2025.01.11 00:30:19.923383 [ 32666 ] {} <Debug> SystemLog: Creating system.metric_log from metric_log
56 2025.01.11 00:30:19.924920 [ 32666 ] {} <Debug> SystemLog: Not creating system.filesystem_cache_log since corresponding section 'filesystem_cache_log' is missing from config
57 2025.01.11 00:30:19.924930 [ 32666 ] {} <Debug> SystemLog: Not creating system.filesystem_read_prefetches_log since corresponding section 'filesystem_read_prefetches_log' is missing from config
58 2025.01.11 00:30:19.924934 [ 32666 ] {} <Debug> SystemLog: Creating system.asynchronous_metric_log from asynchronous_metric_log
59 2025.01.11 00:30:19.925042 [ 32666 ] {} <Debug> SystemLog: Creating system.opentelemetry_span_log from opentelemetry_span_log
60 2025.01.11 00:30:19.925171 [ 32666 ] {} <Debug> SystemLog: Not creating system.query_views_log since corresponding section 'query_views_log' is missing from config
61 2025.01.11 00:30:19.925178 [ 32666 ] {} <Debug> SystemLog: Not creating system.zookeeper_log since corresponding section 'zookeeper_log' is missing from config
62 2025.01.11 00:30:19.925186 [ 32666 ] {} <Debug> SystemLog: Not creating system.session_log since corresponding section 'session_log' is missing from config
63 2025.01.11 00:30:19.925189 [ 32666 ] {} <Debug> SystemLog: Not creating system.transactions_info_log since corresponding section 'transactions_info_log' is missing from config
64 2025.01.11 00:30:19.925192 [ 32666 ] {} <Debug> SystemLog: Not creating system.processors_profile_log since corresponding section 'processors_profile_log' is missing from config
65 2025.01.11 00:30:19.925199 [ 32666 ] {} <Debug> SystemLog: Not creating system.asynchronous_insert_log since corresponding section 'asynchronous_insert_log' is missing from config
66 2025.01.11 00:30:19.934264 [ 32666 ] {} <Information> DatabaseCatalog: Found 0 partially dropped tables. Will load them and retry removal.
67 2025.01.11 00:30:19.938092 [ 32666 ] {} <Information> DatabaseAtomic (default): Metadata processed, database default has 0 tables and 0 dictionaries in total.
68 2025.01.11 00:30:19.938104 [ 32666 ] {} <Information> TablesLoader: Parsed metadata of 0 tables in 1 databases in 3.68e-05 sec
69 2025.01.11 00:30:19.938110 [ 32666 ] {} <Trace> ReferentialDeps: No tables
70 2025.01.11 00:30:19.938115 [ 32666 ] {} <Trace> LoadingDeps: No tables
71 2025.01.11 00:30:19.938130 [ 32666 ] {} <Information> DatabaseAtomic (default): Starting up tables.
72 2025.01.11 00:30:19.938214 [ 32666 ] {} <Information> UserDefinedSQLObjectsLoaderFromDisk: Loading user defined objects from /var/tmp/omicron_tmp/clickward_test/clickhouse-1/data/user_defined/
73 2025.01.11 00:30:19.938238 [ 32666 ] {} <Debug> UserDefinedSQLObjectsLoaderFromDisk: User defined objects loaded
74 2025.01.11 00:30:19.938242 [ 32666 ] {} <Debug> Application: Loaded metadata.
75 2025.01.11 00:30:19.938257 [ 32666 ] {} <Information> Application: Tasks stats provider: procfs
76 2025.01.11 00:30:19.938263 [ 32666 ] {} <Information> Application: It looks like the process has no CAP_SYS_NICE capability, the setting 'os_thread_priority' will have no effect. It could happen due to incorrect ClickHouse package installation. You could resolve the problem manually with 'sudo setcap cap_sys_nice=+ep /work/oxidecomputer/omicron/out/clickhouse/clickhouse'. Note that it will not work on 'nosuid' mounted filesystems.
77 2025.01.11 00:30:19.938335 [ 32666 ] {} <Information> CertificateReloader: One of paths is empty. Cannot apply new configuration for certificates. Fill all paths and try again.
78 2025.01.11 00:30:19.938704 [ 32666 ] {} <Trace> AsynchronousMetrics: MemoryTracking: was 12.57 MiB, peak 13.26 MiB, free memory in arenas 22.28 MiB, will set to 385.19 MiB (RSS), difference: 372.61 MiB
79 2025.01.11 00:30:19.938995 [ 32666 ] {} <Trace> AsynchronousMetrics: Update heavy metrics. Update period 1 sec. Update heavy metrics period 120 sec. Heavy metrics calculation elapsed: 1.09e-05 sec.
80 2025.01.11 00:30:19.940738 [ 32666 ] {} <Information> Application: Listening for http://[::1]:29301
81 2025.01.11 00:30:19.940743 [ 33250 ] {} <Debug> DDLWorker: Starting DDLWorker thread
82 2025.01.11 00:30:19.940791 [ 33250 ] {} <Debug> DDLWorker: Initializing DDLWorker thread
83 2025.01.11 00:30:19.940791 [ 33251 ] {} <Debug> DDLWorker: Started DDLWorker cleanup thread
84 2025.01.11 00:30:19.940827 [ 32666 ] {} <Information> Application: Listening for native protocol (tcp): [::1]:29201
85 2025.01.11 00:30:19.948279 [ 33250 ] {} <Information> ZooKeeperClient: Connected to ZooKeeper at [::1]:29001 with session_id 2
86 2025.01.11 00:30:19.948686 [ 33250 ] {} <Information> ZooKeeperClient: Keeper feature flag FILTERED_LIST: enabled
87 2025.01.11 00:30:19.948695 [ 33250 ] {} <Information> ZooKeeperClient: Keeper feature flag MULTI_READ: enabled
88 2025.01.11 00:30:19.948698 [ 33250 ] {} <Information> ZooKeeperClient: Keeper feature flag CHECK_NOT_EXISTS: disabled
89 2025.01.11 00:30:19.948701 [ 33250 ] {} <Trace> ZooKeeper: Initialized, hosts: [::1]:29001,[::1]:29002,[::1]:29003
90 2025.01.11 00:30:19.948723 [ 32666 ] {} <Information> Application: Ready for connections.
91 2025.01.11 00:30:19.966246 [ 33250 ] {} <Debug> DDLWorker: Initialized DDLWorker thread
92 2025.01.11 00:30:19.966259 [ 33250 ] {} <Debug> DDLWorker: Scheduling tasks
93 2025.01.11 00:30:19.966263 [ 33250 ] {} <Trace> DDLWorker: Don't have unfinished tasks after restarting
94 2025.01.11 00:30:19.966269 [ 33251 ] {} <Debug> DDLWorker: Cleaning queue
95 2025.01.11 00:30:19.966453 [ 33250 ] {} <Trace> DDLWorker: scheduleTasks: initialized=true, size_before_filtering=0, queue_size=0, entries=none..none, first_failed_task_name=none, current_tasks_size=0, last_current_task=none, last_skipped_entry_name=none
96 2025.01.11 00:30:19.966465 [ 33250 ] {} <Debug> DDLWorker: No tasks to schedule
97 2025.01.11 00:30:19.966470 [ 33250 ] {} <Debug> DDLWorker: Waiting for queue updates
98 2025.01.11 00:30:27.425538 [ 33235 ] {} <Trace> SystemLog (system.metric_log): Flushing system log, 8 entries to flush up to offset 8
99 2025.01.11 00:30:27.425589 [ 33235 ] {} <Debug> SystemLog (system.metric_log): Creating new table system.metric_log for MetricLog
100 2025.01.11 00:30:27.425568 [ 33234 ] {} <Trace> SystemLog (system.asynchronous_metric_log): Flushing system log, 1953 entries to flush up to offset 1953
101 2025.01.11 00:30:27.425636 [ 33234 ] {} <Debug> SystemLog (system.asynchronous_metric_log): Creating new table system.asynchronous_metric_log for AsynchronousMetricLog
102 2025.01.11 00:30:27.506128 [ 33234 ] {} <Trace> StoragePolicy (default): Storage policy default created, total volumes 1
103 2025.01.11 00:30:27.511573 [ 33234 ] {} <Debug> system.asynchronous_metric_log (7c1cab3d-a87a-4959-8d08-f2ab3db7432e): Loading data parts
104 2025.01.11 00:30:27.512737 [ 33234 ] {} <Debug> system.asynchronous_metric_log (7c1cab3d-a87a-4959-8d08-f2ab3db7432e): There are no data parts
105 2025.01.11 00:30:27.513735 [ 33235 ] {} <Debug> system.metric_log (c926069d-e416-444a-b525-0c878d521725): Loading data parts
106 2025.01.11 00:30:27.513809 [ 33235 ] {} <Debug> system.metric_log (c926069d-e416-444a-b525-0c878d521725): There are no data parts
107 2025.01.11 00:30:27.519280 [ 33234 ] {} <Trace> system.asynchronous_metric_log (7c1cab3d-a87a-4959-8d08-f2ab3db7432e): Trying to reserve 1.00 MiB using storage policy from min volume index 0
108 2025.01.11 00:30:27.519298 [ 33234 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 163.51 GiB.
109 2025.01.11 00:30:27.520700 [ 33234 ] {} <Trace> MergedBlockOutputStream: filled checksums 202501_1_1_0 (state Temporary)
110 2025.01.11 00:30:27.521744 [ 33234 ] {} <Trace> system.asynchronous_metric_log (7c1cab3d-a87a-4959-8d08-f2ab3db7432e): Renaming temporary part tmp_insert_202501_1_1_0 to 202501_1_1_0 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
111 2025.01.11 00:30:27.521876 [ 33234 ] {} <Trace> SystemLog (system.asynchronous_metric_log): Flushed system log up to offset 1953
112 2025.01.11 00:30:27.530516 [ 33235 ] {} <Trace> system.metric_log (c926069d-e416-444a-b525-0c878d521725): Trying to reserve 1.00 MiB using storage policy from min volume index 0
113 2025.01.11 00:30:27.530532 [ 33235 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 163.51 GiB.
114 2025.01.11 00:30:27.536373 [ 33235 ] {} <Trace> MergedBlockOutputStream: filled checksums 202501_1_1_0 (state Temporary)
115 2025.01.11 00:30:27.540385 [ 33235 ] {} <Trace> system.metric_log (c926069d-e416-444a-b525-0c878d521725): Renaming temporary part tmp_insert_202501_1_1_0 to 202501_1_1_0 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
116 2025.01.11 00:30:27.541490 [ 33235 ] {} <Trace> SystemLog (system.metric_log): Flushed system log up to offset 8
117 2025.01.11 00:30:28.000485 [ 33248 ] {} <Trace> AsynchronousMetrics: MemoryTracking: was 389.58 MiB, peak 398.36 MiB, free memory in arenas 29.43 MiB, will set to 388.10 MiB (RSS), difference: -1.48 MiB
118 2025.01.11 00:30:34.908190 [ 32895 ] {} <Debug> DNSResolver: Updating DNS cache
119 2025.01.11 00:30:34.908223 [ 32895 ] {} <Debug> DNSResolver: Updated DNS cache
120 2025.01.11 00:30:35.021995 [ 33234 ] {} <Trace> SystemLog (system.asynchronous_metric_log): Flushing system log, 1848 entries to flush up to offset 3801
121 2025.01.11 00:30:35.023484 [ 33234 ] {} <Trace> system.asynchronous_metric_log (7c1cab3d-a87a-4959-8d08-f2ab3db7432e): Trying to reserve 1.00 MiB using storage policy from min volume index 0
122 2025.01.11 00:30:35.023504 [ 33234 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 163.51 GiB.
123 2025.01.11 00:30:35.024014 [ 33234 ] {} <Trace> MergedBlockOutputStream: filled checksums 202501_2_2_0 (state Temporary)
124 2025.01.11 00:30:35.024232 [ 33234 ] {} <Trace> system.asynchronous_metric_log (7c1cab3d-a87a-4959-8d08-f2ab3db7432e): Renaming temporary part tmp_insert_202501_2_2_0 to 202501_2_2_0 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
125 2025.01.11 00:30:35.024344 [ 33234 ] {} <Trace> SystemLog (system.asynchronous_metric_log): Flushed system log up to offset 3801
126 2025.01.11 00:30:35.041580 [ 33235 ] {} <Trace> SystemLog (system.metric_log): Flushing system log, 8 entries to flush up to offset 16
127 2025.01.11 00:30:35.052294 [ 33235 ] {} <Trace> system.metric_log (c926069d-e416-444a-b525-0c878d521725): Trying to reserve 1.00 MiB using storage policy from min volume index 0
128 2025.01.11 00:30:35.052314 [ 33235 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 163.51 GiB.
129 2025.01.11 00:30:35.057014 [ 33235 ] {} <Trace> MergedBlockOutputStream: filled checksums 202501_2_2_0 (state Temporary)
130 2025.01.11 00:30:35.060957 [ 33235 ] {} <Trace> system.metric_log (c926069d-e416-444a-b525-0c878d521725): Renaming temporary part tmp_insert_202501_2_2_0 to 202501_2_2_0 with tid (1, 1, 00000000-0000-0000-0000-000000000000).
131 2025.01.11 00:30:35.062021 [ 33235 ] {} <Trace> SystemLog (system.metric_log): Flushed system log up to offset 16