1 | 2025.01.11 00:30:06.839079 [ 32661 ] {} <Information> Application: Will watch for the process with pid 32666 | ||
2 | 2025.01.11 00:30:06.839106 [ 32666 ] {} <Information> Application: Forked a child process to watch | ||
3 | 2025.01.11 00:30:06.839276 [ 32666 ] {} <Information> SentryWriter: Sending crash reports is disabled | ||
4 | 2025.01.11 00:30:06.839385 [ 32666 ] {} <Trace> Pipe: Pipe capacity is 1.00 MiB | ||
5 | 2025.01.11 00:30:07.690083 [ 32666 ] {} <Information> Application: Starting ClickHouse 23.8.7.24 (revision: 54477, git hash: 812b95e14ba8cf744bf1d70e6de607cf130a79fa, build id: 6484B75ECB16F4FF87A88E10249A45E1BDBDE2E5), PID 32666 | ||
6 | 2025.01.11 00:30:07.794108 [ 32666 ] {} <Information> Application: starting up | ||
7 | 2025.01.11 00:30:07.794141 [ 32666 ] {} <Information> Application: OS name: Linux, version: 5.15.0-1015-aws, architecture: x86_64 | ||
8 | 2025.01.11 00:30:09.773534 [ 32666 ] {} <Information> Application: Available RAM: 30.68 GiB; physical cores: 8; logical cores: 8. | ||
9 | 2025.01.11 00:30:09.795023 [ 32666 ] {} <Trace> AsynchronousMetrics: Scanning /sys/class/thermal | ||
10 | 2025.01.11 00:30:09.795042 [ 32666 ] {} <Trace> AsynchronousMetrics: Scanning /sys/block | ||
11 | 2025.01.11 00:30:09.795081 [ 32666 ] {} <Trace> AsynchronousMetrics: Scanning /sys/devices/system/edac | ||
12 | 2025.01.11 00:30:09.795095 [ 32666 ] {} <Trace> AsynchronousMetrics: Scanning /sys/class/hwmon | ||
13 | 2025.01.11 00:30:19.869478 [ 32666 ] {} <Information> Application: Integrity check of the executable successfully passed (checksum: 63E397DB90569A93F13765E06F4A1FBF) | ||
14 | 2025.01.11 00:30:19.877334 [ 32666 ] {} <Debug> Application: Set max number of file descriptors to 524288 (was 1024). | ||
15 | 2025.01.11 00:30:19.877345 [ 32666 ] {} <Debug> Application: rlimit on number of threads is 125612 | ||
16 | 2025.01.11 00:30:19.877351 [ 32666 ] {} <Debug> Application: Initializing DateLUT. | ||
17 | 2025.01.11 00:30:19.877355 [ 32666 ] {} <Trace> Application: Initialized DateLUT with time zone 'UTC'. | ||
18 | 2025.01.11 00:30:19.877372 [ 32666 ] {} <Debug> Context: Setting up /var/tmp/omicron_tmp/clickward_test/clickhouse-1/data/tmp/ to store temporary data in it | ||
19 | 2025.01.11 00:30:19.877630 [ 32666 ] {} <Debug> Application: Initializing interserver credentials. | ||
20 | 2025.01.11 00:30:19.878146 [ 32666 ] {} <Debug> ConfigReloader: Loading config '/var/tmp/omicron_tmp/clickward_test/clickhouse-1/clickhouse-config.xml' | ||
21 | 2025.01.11 00:30:19.878871 [ 32666 ] {} <Debug> ConfigReloader: Loaded config '/var/tmp/omicron_tmp/clickward_test/clickhouse-1/clickhouse-config.xml', performing update on configuration | ||
22 | 2025.01.11 00:30:19.879453 [ 32666 ] {} <Information> Application: Setting max_server_memory_usage was set to 27.61 GiB (30.68 GiB available * 0.90 max_server_memory_usage_to_ram_ratio) | ||
23 | 2025.01.11 00:30:19.879463 [ 32666 ] {} <Information> Application: Setting merges_mutations_memory_usage_soft_limit was set to 15.34 GiB (30.68 GiB available * 0.50 merges_mutations_memory_usage_to_ram_ratio) | ||
24 | 2025.01.11 00:30:19.879468 [ 32666 ] {} <Information> Application: Merges and mutations memory limit is set to 15.34 GiB | ||
25 | 2025.01.11 00:30:19.879676 [ 32666 ] {} <Information> BackgroundSchedulePool/BgBufSchPool: Create BackgroundSchedulePool with 16 threads | ||
26 | 2025.01.11 00:30:19.880450 [ 32666 ] {} <Information> BackgroundSchedulePool/BgSchPool: Create BackgroundSchedulePool with 128 threads | ||
27 | 2025.01.11 00:30:19.887963 [ 32666 ] {} <Information> BackgroundSchedulePool/BgMBSchPool: Create BackgroundSchedulePool with 16 threads | ||
28 | 2025.01.11 00:30:19.888730 [ 32666 ] {} <Information> BackgroundSchedulePool/BgDistSchPool: Create BackgroundSchedulePool with 16 threads | ||
29 | 2025.01.11 00:30:19.898803 [ 32666 ] {} <Information> CertificateReloader: One of paths is empty. Cannot apply new configuration for certificates. Fill all paths and try again. | ||
30 | 2025.01.11 00:30:19.898829 [ 32666 ] {} <Debug> ConfigReloader: Loaded config '/var/tmp/omicron_tmp/clickward_test/clickhouse-1/clickhouse-config.xml', performed update on configuration | ||
31 | 2025.01.11 00:30:19.898990 [ 32666 ] {} <Information> Application: Listening for replica communication (interserver): http://[::1]:29401 | ||
32 | 2025.01.11 00:30:19.900573 [ 32666 ] {} <Debug> ConfigReloader: Loading config '/var/tmp/omicron_tmp/clickward_test/clickhouse-1/clickhouse-config.xml' | ||
33 | 2025.01.11 00:30:19.901081 [ 32666 ] {} <Debug> ConfigReloader: Loaded config '/var/tmp/omicron_tmp/clickward_test/clickhouse-1/clickhouse-config.xml', performing update on configuration | ||
34 | 2025.01.11 00:30:19.901825 [ 32666 ] {} <Debug> ConfigReloader: Loaded config '/var/tmp/omicron_tmp/clickward_test/clickhouse-1/clickhouse-config.xml', performed update on configuration | ||
35 | 2025.01.11 00:30:19.902052 [ 32666 ] {} <Debug> Access(user directories): Added users_xml access storage 'users_xml', path: /var/tmp/omicron_tmp/clickward_test/clickhouse-1/clickhouse-config.xml | ||
36 | 2025.01.11 00:30:19.902787 [ 32666 ] {} <Information> Context: Initialized background executor for merges and mutations with num_threads=16, num_tasks=32, scheduling_policy=round_robin | ||
37 | 2025.01.11 00:30:19.904013 [ 32666 ] {} <Information> Context: Initialized background executor for move operations with num_threads=8, num_tasks=8 | ||
38 | 2025.01.11 00:30:19.904318 [ 32666 ] {} <Information> Context: Initialized background executor for fetches with num_threads=8, num_tasks=8 | ||
39 | 2025.01.11 00:30:19.904681 [ 32666 ] {} <Information> Context: Initialized background executor for common operations (e.g. clearing old parts) with num_threads=8, num_tasks=8 | ||
40 | 2025.01.11 00:30:19.907709 [ 32666 ] {} <Information> DNSCacheUpdater: Update period 15 seconds | ||
41 | 2025.01.11 00:30:19.907730 [ 32666 ] {} <Information> Application: Loading metadata from /var/tmp/omicron_tmp/clickward_test/clickhouse-1/data/ | ||
42 | 2025.01.11 00:30:19.907915 [ 32822 ] {} <Debug> DNSResolver: Updating DNS cache | ||
43 | 2025.01.11 00:30:19.908071 [ 32822 ] {} <Debug> DNSResolver: Updated DNS cache | ||
44 | 2025.01.11 00:30:19.923312 [ 32666 ] {} <Information> DatabaseAtomic (system): Metadata processed, database system has 0 tables and 0 dictionaries in total. | ||
45 | 2025.01.11 00:30:19.923322 [ 32666 ] {} <Information> TablesLoader: Parsed metadata of 0 tables in 1 databases in 4.792e-05 sec | ||
46 | 2025.01.11 00:30:19.923326 [ 32666 ] {} <Trace> ReferentialDeps: No tables | ||
47 | 2025.01.11 00:30:19.923329 [ 32666 ] {} <Trace> LoadingDeps: No tables | ||
48 | 2025.01.11 00:30:19.923346 [ 32666 ] {} <Information> DatabaseAtomic (system): Starting up tables. | ||
49 | 2025.01.11 00:30:19.923356 [ 32666 ] {} <Debug> SystemLog: Not creating system.query_log since corresponding section 'query_log' is missing from config | ||
50 | 2025.01.11 00:30:19.923361 [ 32666 ] {} <Debug> SystemLog: Not creating system.query_thread_log since corresponding section 'query_thread_log' is missing from config | ||
51 | 2025.01.11 00:30:19.923364 [ 32666 ] {} <Debug> SystemLog: Not creating system.part_log since corresponding section 'part_log' is missing from config | ||
52 | 2025.01.11 00:30:19.923368 [ 32666 ] {} <Debug> SystemLog: Not creating system.trace_log since corresponding section 'trace_log' is missing from config | ||
53 | 2025.01.11 00:30:19.923372 [ 32666 ] {} <Debug> SystemLog: Not creating system.crash_log since corresponding section 'crash_log' is missing from config | ||
54 | 2025.01.11 00:30:19.923377 [ 32666 ] {} <Debug> SystemLog: Not creating system.text_log since corresponding section 'text_log' is missing from config | ||
55 | 2025.01.11 00:30:19.923383 [ 32666 ] {} <Debug> SystemLog: Creating system.metric_log from metric_log | ||
56 | 2025.01.11 00:30:19.924920 [ 32666 ] {} <Debug> SystemLog: Not creating system.filesystem_cache_log since corresponding section 'filesystem_cache_log' is missing from config | ||
57 | 2025.01.11 00:30:19.924930 [ 32666 ] {} <Debug> SystemLog: Not creating system.filesystem_read_prefetches_log since corresponding section 'filesystem_read_prefetches_log' is missing from config | ||
58 | 2025.01.11 00:30:19.924934 [ 32666 ] {} <Debug> SystemLog: Creating system.asynchronous_metric_log from asynchronous_metric_log | ||
59 | 2025.01.11 00:30:19.925042 [ 32666 ] {} <Debug> SystemLog: Creating system.opentelemetry_span_log from opentelemetry_span_log | ||
60 | 2025.01.11 00:30:19.925171 [ 32666 ] {} <Debug> SystemLog: Not creating system.query_views_log since corresponding section 'query_views_log' is missing from config | ||
61 | 2025.01.11 00:30:19.925178 [ 32666 ] {} <Debug> SystemLog: Not creating system.zookeeper_log since corresponding section 'zookeeper_log' is missing from config | ||
62 | 2025.01.11 00:30:19.925186 [ 32666 ] {} <Debug> SystemLog: Not creating system.session_log since corresponding section 'session_log' is missing from config | ||
63 | 2025.01.11 00:30:19.925189 [ 32666 ] {} <Debug> SystemLog: Not creating system.transactions_info_log since corresponding section 'transactions_info_log' is missing from config | ||
64 | 2025.01.11 00:30:19.925192 [ 32666 ] {} <Debug> SystemLog: Not creating system.processors_profile_log since corresponding section 'processors_profile_log' is missing from config | ||
65 | 2025.01.11 00:30:19.925199 [ 32666 ] {} <Debug> SystemLog: Not creating system.asynchronous_insert_log since corresponding section 'asynchronous_insert_log' is missing from config | ||
66 | 2025.01.11 00:30:19.934264 [ 32666 ] {} <Information> DatabaseCatalog: Found 0 partially dropped tables. Will load them and retry removal. | ||
67 | 2025.01.11 00:30:19.938092 [ 32666 ] {} <Information> DatabaseAtomic (default): Metadata processed, database default has 0 tables and 0 dictionaries in total. | ||
68 | 2025.01.11 00:30:19.938104 [ 32666 ] {} <Information> TablesLoader: Parsed metadata of 0 tables in 1 databases in 3.68e-05 sec | ||
69 | 2025.01.11 00:30:19.938110 [ 32666 ] {} <Trace> ReferentialDeps: No tables | ||
70 | 2025.01.11 00:30:19.938115 [ 32666 ] {} <Trace> LoadingDeps: No tables | ||
71 | 2025.01.11 00:30:19.938130 [ 32666 ] {} <Information> DatabaseAtomic (default): Starting up tables. | ||
72 | 2025.01.11 00:30:19.938214 [ 32666 ] {} <Information> UserDefinedSQLObjectsLoaderFromDisk: Loading user defined objects from /var/tmp/omicron_tmp/clickward_test/clickhouse-1/data/user_defined/ | ||
73 | 2025.01.11 00:30:19.938238 [ 32666 ] {} <Debug> UserDefinedSQLObjectsLoaderFromDisk: User defined objects loaded | ||
74 | 2025.01.11 00:30:19.938242 [ 32666 ] {} <Debug> Application: Loaded metadata. | ||
75 | 2025.01.11 00:30:19.938257 [ 32666 ] {} <Information> Application: Tasks stats provider: procfs | ||
76 | 2025.01.11 00:30:19.938263 [ 32666 ] {} <Information> Application: It looks like the process has no CAP_SYS_NICE capability, the setting 'os_thread_priority' will have no effect. It could happen due to incorrect ClickHouse package installation. You could resolve the problem manually with 'sudo setcap cap_sys_nice=+ep /work/oxidecomputer/omicron/out/clickhouse/clickhouse'. Note that it will not work on 'nosuid' mounted filesystems. | ||
77 | 2025.01.11 00:30:19.938335 [ 32666 ] {} <Information> CertificateReloader: One of paths is empty. Cannot apply new configuration for certificates. Fill all paths and try again. | ||
78 | 2025.01.11 00:30:19.938704 [ 32666 ] {} <Trace> AsynchronousMetrics: MemoryTracking: was 12.57 MiB, peak 13.26 MiB, free memory in arenas 22.28 MiB, will set to 385.19 MiB (RSS), difference: 372.61 MiB | ||
79 | 2025.01.11 00:30:19.938995 [ 32666 ] {} <Trace> AsynchronousMetrics: Update heavy metrics. Update period 1 sec. Update heavy metrics period 120 sec. Heavy metrics calculation elapsed: 1.09e-05 sec. | ||
80 | 2025.01.11 00:30:19.940738 [ 32666 ] {} <Information> Application: Listening for http://[::1]:29301 | ||
81 | 2025.01.11 00:30:19.940743 [ 33250 ] {} <Debug> DDLWorker: Starting DDLWorker thread | ||
82 | 2025.01.11 00:30:19.940791 [ 33250 ] {} <Debug> DDLWorker: Initializing DDLWorker thread | ||
83 | 2025.01.11 00:30:19.940791 [ 33251 ] {} <Debug> DDLWorker: Started DDLWorker cleanup thread | ||
84 | 2025.01.11 00:30:19.940827 [ 32666 ] {} <Information> Application: Listening for native protocol (tcp): [::1]:29201 | ||
85 | 2025.01.11 00:30:19.948279 [ 33250 ] {} <Information> ZooKeeperClient: Connected to ZooKeeper at [::1]:29001 with session_id 2 | ||
86 | 2025.01.11 00:30:19.948686 [ 33250 ] {} <Information> ZooKeeperClient: Keeper feature flag FILTERED_LIST: enabled | ||
87 | 2025.01.11 00:30:19.948695 [ 33250 ] {} <Information> ZooKeeperClient: Keeper feature flag MULTI_READ: enabled | ||
88 | 2025.01.11 00:30:19.948698 [ 33250 ] {} <Information> ZooKeeperClient: Keeper feature flag CHECK_NOT_EXISTS: disabled | ||
89 | 2025.01.11 00:30:19.948701 [ 33250 ] {} <Trace> ZooKeeper: Initialized, hosts: [::1]:29001,[::1]:29002,[::1]:29003 | ||
90 | 2025.01.11 00:30:19.948723 [ 32666 ] {} <Information> Application: Ready for connections. | ||
91 | 2025.01.11 00:30:19.966246 [ 33250 ] {} <Debug> DDLWorker: Initialized DDLWorker thread | ||
92 | 2025.01.11 00:30:19.966259 [ 33250 ] {} <Debug> DDLWorker: Scheduling tasks | ||
93 | 2025.01.11 00:30:19.966263 [ 33250 ] {} <Trace> DDLWorker: Don't have unfinished tasks after restarting | ||
94 | 2025.01.11 00:30:19.966269 [ 33251 ] {} <Debug> DDLWorker: Cleaning queue | ||
95 | 2025.01.11 00:30:19.966453 [ 33250 ] {} <Trace> DDLWorker: scheduleTasks: initialized=true, size_before_filtering=0, queue_size=0, entries=none..none, first_failed_task_name=none, current_tasks_size=0, last_current_task=none, last_skipped_entry_name=none | ||
96 | 2025.01.11 00:30:19.966465 [ 33250 ] {} <Debug> DDLWorker: No tasks to schedule | ||
97 | 2025.01.11 00:30:19.966470 [ 33250 ] {} <Debug> DDLWorker: Waiting for queue updates | ||
98 | 2025.01.11 00:30:27.425538 [ 33235 ] {} <Trace> SystemLog (system.metric_log): Flushing system log, 8 entries to flush up to offset 8 | ||
99 | 2025.01.11 00:30:27.425589 [ 33235 ] {} <Debug> SystemLog (system.metric_log): Creating new table system.metric_log for MetricLog | ||
100 | 2025.01.11 00:30:27.425568 [ 33234 ] {} <Trace> SystemLog (system.asynchronous_metric_log): Flushing system log, 1953 entries to flush up to offset 1953 | ||
101 | 2025.01.11 00:30:27.425636 [ 33234 ] {} <Debug> SystemLog (system.asynchronous_metric_log): Creating new table system.asynchronous_metric_log for AsynchronousMetricLog | ||
102 | 2025.01.11 00:30:27.506128 [ 33234 ] {} <Trace> StoragePolicy (default): Storage policy default created, total volumes 1 | ||
103 | 2025.01.11 00:30:27.511573 [ 33234 ] {} <Debug> system.asynchronous_metric_log (7c1cab3d-a87a-4959-8d08-f2ab3db7432e): Loading data parts | ||
104 | 2025.01.11 00:30:27.512737 [ 33234 ] {} <Debug> system.asynchronous_metric_log (7c1cab3d-a87a-4959-8d08-f2ab3db7432e): There are no data parts | ||
105 | 2025.01.11 00:30:27.513735 [ 33235 ] {} <Debug> system.metric_log (c926069d-e416-444a-b525-0c878d521725): Loading data parts | ||
106 | 2025.01.11 00:30:27.513809 [ 33235 ] {} <Debug> system.metric_log (c926069d-e416-444a-b525-0c878d521725): There are no data parts | ||
107 | 2025.01.11 00:30:27.519280 [ 33234 ] {} <Trace> system.asynchronous_metric_log (7c1cab3d-a87a-4959-8d08-f2ab3db7432e): Trying to reserve 1.00 MiB using storage policy from min volume index 0 | ||
108 | 2025.01.11 00:30:27.519298 [ 33234 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 163.51 GiB. | ||
109 | 2025.01.11 00:30:27.520700 [ 33234 ] {} <Trace> MergedBlockOutputStream: filled checksums 202501_1_1_0 (state Temporary) | ||
110 | 2025.01.11 00:30:27.521744 [ 33234 ] {} <Trace> system.asynchronous_metric_log (7c1cab3d-a87a-4959-8d08-f2ab3db7432e): Renaming temporary part tmp_insert_202501_1_1_0 to 202501_1_1_0 with tid (1, 1, 00000000-0000-0000-0000-000000000000). | ||
111 | 2025.01.11 00:30:27.521876 [ 33234 ] {} <Trace> SystemLog (system.asynchronous_metric_log): Flushed system log up to offset 1953 | ||
112 | 2025.01.11 00:30:27.530516 [ 33235 ] {} <Trace> system.metric_log (c926069d-e416-444a-b525-0c878d521725): Trying to reserve 1.00 MiB using storage policy from min volume index 0 | ||
113 | 2025.01.11 00:30:27.530532 [ 33235 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 163.51 GiB. | ||
114 | 2025.01.11 00:30:27.536373 [ 33235 ] {} <Trace> MergedBlockOutputStream: filled checksums 202501_1_1_0 (state Temporary) | ||
115 | 2025.01.11 00:30:27.540385 [ 33235 ] {} <Trace> system.metric_log (c926069d-e416-444a-b525-0c878d521725): Renaming temporary part tmp_insert_202501_1_1_0 to 202501_1_1_0 with tid (1, 1, 00000000-0000-0000-0000-000000000000). | ||
116 | 2025.01.11 00:30:27.541490 [ 33235 ] {} <Trace> SystemLog (system.metric_log): Flushed system log up to offset 8 | ||
117 | 2025.01.11 00:30:28.000485 [ 33248 ] {} <Trace> AsynchronousMetrics: MemoryTracking: was 389.58 MiB, peak 398.36 MiB, free memory in arenas 29.43 MiB, will set to 388.10 MiB (RSS), difference: -1.48 MiB | ||
118 | 2025.01.11 00:30:34.908190 [ 32895 ] {} <Debug> DNSResolver: Updating DNS cache | ||
119 | 2025.01.11 00:30:34.908223 [ 32895 ] {} <Debug> DNSResolver: Updated DNS cache | ||
120 | 2025.01.11 00:30:35.021995 [ 33234 ] {} <Trace> SystemLog (system.asynchronous_metric_log): Flushing system log, 1848 entries to flush up to offset 3801 | ||
121 | 2025.01.11 00:30:35.023484 [ 33234 ] {} <Trace> system.asynchronous_metric_log (7c1cab3d-a87a-4959-8d08-f2ab3db7432e): Trying to reserve 1.00 MiB using storage policy from min volume index 0 | ||
122 | 2025.01.11 00:30:35.023504 [ 33234 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 163.51 GiB. | ||
123 | 2025.01.11 00:30:35.024014 [ 33234 ] {} <Trace> MergedBlockOutputStream: filled checksums 202501_2_2_0 (state Temporary) | ||
124 | 2025.01.11 00:30:35.024232 [ 33234 ] {} <Trace> system.asynchronous_metric_log (7c1cab3d-a87a-4959-8d08-f2ab3db7432e): Renaming temporary part tmp_insert_202501_2_2_0 to 202501_2_2_0 with tid (1, 1, 00000000-0000-0000-0000-000000000000). | ||
125 | 2025.01.11 00:30:35.024344 [ 33234 ] {} <Trace> SystemLog (system.asynchronous_metric_log): Flushed system log up to offset 3801 | ||
126 | 2025.01.11 00:30:35.041580 [ 33235 ] {} <Trace> SystemLog (system.metric_log): Flushing system log, 8 entries to flush up to offset 16 | ||
127 | 2025.01.11 00:30:35.052294 [ 33235 ] {} <Trace> system.metric_log (c926069d-e416-444a-b525-0c878d521725): Trying to reserve 1.00 MiB using storage policy from min volume index 0 | ||
128 | 2025.01.11 00:30:35.052314 [ 33235 ] {} <Trace> DiskLocal: Reserved 1.00 MiB on local disk `default`, having unreserved 163.51 GiB. | ||
129 | 2025.01.11 00:30:35.057014 [ 33235 ] {} <Trace> MergedBlockOutputStream: filled checksums 202501_2_2_0 (state Temporary) | ||
130 | 2025.01.11 00:30:35.060957 [ 33235 ] {} <Trace> system.metric_log (c926069d-e416-444a-b525-0c878d521725): Renaming temporary part tmp_insert_202501_2_2_0 to 202501_2_2_0 with tid (1, 1, 00000000-0000-0000-0000-000000000000). | ||
131 | 2025.01.11 00:30:35.062021 [ 33235 ] {} <Trace> SystemLog (system.metric_log): Flushed system log up to offset 16 |