| 1 | 2025.05.14 01:41:24.174429 [ 1 ] {} <Information> Application: Starting ClickHouse 23.8.7.1 (revision: 54477, git hash: 3042d295d963012962f5c683bd2776fa331a38c3, build id: <unknown>), PID 16487 | ||
| 2 | 2025.05.14 01:41:24.174764 [ 1 ] {} <Information> Application: starting up | ||
| 3 | 2025.05.14 01:41:24.174821 [ 1 ] {} <Information> Application: OS name: SunOS, version: 5.11, architecture: i86pc | ||
| 4 | 2025.05.14 01:41:24.195374 [ 1 ] {} <Information> Application: Available RAM: 31.99 GiB; physical cores: 8; logical cores: 8. | ||
| 5 | 2025.05.14 01:41:24.204001 [ 1 ] {} <Debug> Application: rlimit on number of file descriptors is 65536 | ||
| 6 | 2025.05.14 01:41:24.204121 [ 1 ] {} <Debug> Application: Initializing DateLUT. | ||
| 7 | 2025.05.14 01:41:24.204139 [ 1 ] {} <Trace> Application: Initialized DateLUT with time zone 'UTC'. | ||
| 8 | 2025.05.14 01:41:24.204210 [ 1 ] {} <Debug> Context: Setting up /var/tmp/omicron_tmp/oximeter_db-11d8f51cb75f441f-test_ensure_schema_walks_through_multiple_steps_replicated.16475.5-clickhouse-gCPVKS/tmp/ to store temporary data in it | ||
| 9 | 2025.05.14 01:41:24.205623 [ 1 ] {} <Debug> Application: Initializing interserver credentials. | ||
| 10 | 2025.05.14 01:41:24.206511 [ 1 ] {} <Information> SensitiveDataMaskerConfigRead: 1 query masking rules loaded. | ||
| 11 | 2025.05.14 01:41:24.211514 [ 1 ] {} <Debug> ConfigReloader: Loading config '/work/oxidecomputer/omicron/oximeter/db/src/configs/replica_config.xml' | ||
| 12 | 2025.05.14 01:41:24.216114 [ 1 ] {} <Debug> ConfigReloader: Loaded config '/work/oxidecomputer/omicron/oximeter/db/src/configs/replica_config.xml', performing update on configuration | ||
| 13 | 2025.05.14 01:41:24.219249 [ 1 ] {} <Information> Application: Setting max_server_memory_usage was set to 28.79 GiB (31.99 GiB available * 0.90 max_server_memory_usage_to_ram_ratio) | ||
| 14 | 2025.05.14 01:41:24.219324 [ 1 ] {} <Information> Application: Setting merges_mutations_memory_usage_soft_limit was set to 16.00 GiB (31.99 GiB available * 0.50 merges_mutations_memory_usage_to_ram_ratio) | ||
| 15 | 2025.05.14 01:41:24.219338 [ 1 ] {} <Information> Application: Merges and mutations memory limit is set to 16.00 GiB | ||
| 16 | 2025.05.14 01:41:24.222190 [ 1 ] {} <Information> BackgroundSchedulePool/BgBufSchPool: Create BackgroundSchedulePool with 16 threads | ||
| 17 | 2025.05.14 01:41:24.223381 [ 1 ] {} <Information> BackgroundSchedulePool/BgSchPool: Create BackgroundSchedulePool with 128 threads | ||
| 18 | 2025.05.14 01:41:24.268871 [ 1 ] {} <Information> BackgroundSchedulePool/BgMBSchPool: Create BackgroundSchedulePool with 16 threads | ||
| 19 | 2025.05.14 01:41:24.275726 [ 1 ] {} <Information> BackgroundSchedulePool/BgDistSchPool: Create BackgroundSchedulePool with 16 threads | ||
| 20 | 2025.05.14 01:41:24.277720 [ 1 ] {} <Information> CertificateReloader: One of paths is empty. Cannot apply new configuration for certificates. Fill all paths and try again. | ||
| 21 | 2025.05.14 01:41:24.277883 [ 1 ] {} <Debug> ConfigReloader: Loaded config '/work/oxidecomputer/omicron/oximeter/db/src/configs/replica_config.xml', performed update on configuration | ||
| 22 | 2025.05.14 01:41:24.279255 [ 1 ] {} <Information> Application: Listening for replica communication (interserver): http://[::]:9010 | ||
| 23 | 2025.05.14 01:41:24.280842 [ 1 ] {} <Debug> ConfigReloader: Loading config '/work/oxidecomputer/omicron/oximeter/db/src/configs/replica_config.xml' | ||
| 24 | 2025.05.14 01:41:24.289132 [ 1 ] {} <Debug> ConfigReloader: Loaded config '/work/oxidecomputer/omicron/oximeter/db/src/configs/replica_config.xml', performing update on configuration | ||
| 25 | 2025.05.14 01:41:24.293584 [ 1 ] {} <Debug> ConfigReloader: Loaded config '/work/oxidecomputer/omicron/oximeter/db/src/configs/replica_config.xml', performed update on configuration | ||
| 26 | 2025.05.14 01:41:24.295135 [ 1 ] {} <Debug> Access(user directories): Added users_xml access storage 'users_xml', path: /work/oxidecomputer/omicron/oximeter/db/src/configs/replica_config.xml | ||
| 27 | 2025.05.14 01:41:24.298505 [ 1 ] {} <Information> Context: Initialized background executor for merges and mutations with num_threads=16, num_tasks=32, scheduling_policy=round_robin | ||
| 28 | 2025.05.14 01:41:24.300427 [ 1 ] {} <Information> Context: Initialized background executor for move operations with num_threads=8, num_tasks=8 | ||
| 29 | 2025.05.14 01:41:24.301537 [ 1 ] {} <Information> Context: Initialized background executor for fetches with num_threads=8, num_tasks=8 | ||
| 30 | 2025.05.14 01:41:24.303486 [ 1 ] {} <Information> Context: Initialized background executor for common operations (e.g. clearing old parts) with num_threads=8, num_tasks=8 | ||
| 31 | 2025.05.14 01:41:24.306734 [ 1 ] {} <Information> DNSCacheUpdater: Update period 15 seconds | ||
| 32 | 2025.05.14 01:41:24.308560 [ 1 ] {} <Information> Application: Loading metadata from /var/tmp/omicron_tmp/oximeter_db-11d8f51cb75f441f-test_ensure_schema_walks_through_multiple_steps_replicated.16475.5-clickhouse-gCPVKS/datastore/ | ||
| 33 | 2025.05.14 01:41:24.308706 [ 150 ] {} <Debug> DNSResolver: Updating DNS cache | ||
| 34 | 2025.05.14 01:41:24.308845 [ 150 ] {} <Debug> DNSResolver: Updated DNS cache | ||
| 35 | 2025.05.14 01:41:24.311203 [ 1 ] {} <Information> DatabaseAtomic (system): Metadata processed, database system has 0 tables and 0 dictionaries in total. | ||
| 36 | 2025.05.14 01:41:24.311504 [ 1 ] {} <Information> TablesLoader: Parsed metadata of 0 tables in 1 databases in 0.000498505 sec | ||
| 37 | 2025.05.14 01:41:24.311528 [ 1 ] {} <Trace> ReferentialDeps: No tables | ||
| 38 | 2025.05.14 01:41:24.311537 [ 1 ] {} <Trace> LoadingDeps: No tables | ||
| 39 | 2025.05.14 01:41:24.311899 [ 1 ] {} <Information> DatabaseAtomic (system): Starting up tables. | ||
| 40 | 2025.05.14 01:41:24.312155 [ 1 ] {} <Debug> SystemLog: Creating system.query_log from query_log | ||
| 41 | 2025.05.14 01:41:24.331280 [ 1 ] {} <Debug> SystemLog: Creating system.query_thread_log from query_thread_log | ||
| 42 | 2025.05.14 01:41:24.332285 [ 1 ] {} <Debug> SystemLog: Creating system.part_log from part_log | ||
| 43 | 2025.05.14 01:41:24.332875 [ 1 ] {} <Debug> SystemLog: Creating system.trace_log from trace_log | ||
| 44 | 2025.05.14 01:41:24.333277 [ 1 ] {} <Debug> SystemLog: Creating system.crash_log from crash_log | ||
| 45 | 2025.05.14 01:41:24.335196 [ 1 ] {} <Debug> SystemLog: Not creating system.text_log since corresponding section 'text_log' is missing from config | ||
| 46 | 2025.05.14 01:41:24.335237 [ 1 ] {} <Debug> SystemLog: Creating system.metric_log from metric_log | ||
| 47 | 2025.05.14 01:41:24.344320 [ 1 ] {} <Debug> SystemLog: Not creating system.filesystem_cache_log since corresponding section 'filesystem_cache_log' is missing from config | ||
| 48 | 2025.05.14 01:41:24.344434 [ 1 ] {} <Debug> SystemLog: Not creating system.filesystem_read_prefetches_log since corresponding section 'filesystem_read_prefetches_log' is missing from config | ||
| 49 | 2025.05.14 01:41:24.344454 [ 1 ] {} <Debug> SystemLog: Creating system.asynchronous_metric_log from asynchronous_metric_log | ||
| 50 | 2025.05.14 01:41:24.345287 [ 1 ] {} <Debug> SystemLog: Not creating system.opentelemetry_span_log since corresponding section 'opentelemetry_span_log' is missing from config | ||
| 51 | 2025.05.14 01:41:24.345341 [ 1 ] {} <Debug> SystemLog: Creating system.query_views_log from query_views_log | ||
| 52 | 2025.05.14 01:41:24.345820 [ 1 ] {} <Debug> SystemLog: Not creating system.zookeeper_log since corresponding section 'zookeeper_log' is missing from config | ||
| 53 | 2025.05.14 01:41:24.345854 [ 1 ] {} <Debug> SystemLog: Not creating system.session_log since corresponding section 'session_log' is missing from config | ||
| 54 | 2025.05.14 01:41:24.345876 [ 1 ] {} <Debug> SystemLog: Not creating system.transactions_info_log since corresponding section 'transactions_info_log' is missing from config | ||
| 55 | 2025.05.14 01:41:24.345914 [ 1 ] {} <Debug> SystemLog: Creating system.processors_profile_log from processors_profile_log | ||
| 56 | 2025.05.14 01:41:24.346174 [ 1 ] {} <Debug> SystemLog: Not creating system.asynchronous_insert_log since corresponding section 'asynchronous_insert_log' is missing from config | ||
| 57 | 2025.05.14 01:41:24.441618 [ 1 ] {} <Information> DatabaseCatalog: Found 0 partially dropped tables. Will load them and retry removal. | ||
| 58 | 2025.05.14 01:41:24.443356 [ 1 ] {} <Information> DatabaseAtomic (default): Metadata processed, database default has 0 tables and 0 dictionaries in total. | ||
| 59 | 2025.05.14 01:41:24.443397 [ 1 ] {} <Information> TablesLoader: Parsed metadata of 0 tables in 1 databases in 0.000108101 sec | ||
| 60 | 2025.05.14 01:41:24.443411 [ 1 ] {} <Trace> ReferentialDeps: No tables | ||
| 61 | 2025.05.14 01:41:24.443421 [ 1 ] {} <Trace> LoadingDeps: No tables | ||
| 62 | 2025.05.14 01:41:24.443666 [ 1 ] {} <Information> DatabaseAtomic (default): Starting up tables. | ||
| 63 | 2025.05.14 01:41:24.444042 [ 1 ] {} <Information> UserDefinedSQLObjectsLoaderFromDisk: Loading user defined objects from /var/tmp/omicron_tmp/oximeter_db-11d8f51cb75f441f-test_ensure_schema_walks_through_multiple_steps_replicated.16475.5-clickhouse-gCPVKS/datastore/user_defined/ | ||
| 64 | 2025.05.14 01:41:24.444127 [ 1 ] {} <Debug> UserDefinedSQLObjectsLoaderFromDisk: User defined objects loaded | ||
| 65 | 2025.05.14 01:41:24.444155 [ 1 ] {} <Debug> Application: Loaded metadata. | ||
| 66 | 2025.05.14 01:41:24.444172 [ 1 ] {} <Information> Application: Query Profiler and TraceCollector are disabled because they require PHDR cache to be created (otherwise the function 'dl_iterate_phdr' is not lock free and not async-signal safe). | ||
| 67 | 2025.05.14 01:41:24.444183 [ 1 ] {} <Information> Application: TaskStats is not implemented for this OS. IO accounting will be disabled. | ||
| 68 | 2025.05.14 01:41:24.444601 [ 1 ] {} <Information> CertificateReloader: One of paths is empty. Cannot apply new configuration for certificates. Fill all paths and try again. | ||
| 69 | 2025.05.14 01:41:24.444876 [ 1 ] {} <Trace> AsynchronousMetrics: MemoryTracking: was 40.97 MiB, peak 40.97 MiB, free memory in arenas 0.00 B, will set to 145.30 MiB (RSS), difference: 104.33 MiB | ||
| 70 | 2025.05.14 01:41:24.445349 [ 1 ] {} <Trace> AsynchronousMetrics: Update heavy metrics. Update period 1 sec. Update heavy metrics period 120 sec. Heavy metrics calculation elapsed: 2.4563e-05 sec. | ||
| 71 | 2025.05.14 01:41:24.446529 [ 1 ] {} <Information> Application: Listening for http://[::]:8124 | ||
| 72 | 2025.05.14 01:41:24.446621 [ 1 ] {} <Information> Application: Listening for native protocol (tcp): [::]:9002 | ||
| 73 | 2025.05.14 01:41:24.446637 [ 1 ] {} <Information> Application: Ready for connections. | ||
| 74 | 2025.05.14 01:41:24.446985 [ 258 ] {} <Debug> DDLWorker: Started DDLWorker cleanup thread | ||
| 75 | 2025.05.14 01:41:24.447449 [ 257 ] {} <Debug> DDLWorker: Starting DDLWorker thread | ||
| 76 | 2025.05.14 01:41:24.447526 [ 257 ] {} <Debug> DDLWorker: Initializing DDLWorker thread | ||
| 77 | 2025.05.14 01:41:24.471444 [ 257 ] {} <Error> virtual bool DB::DDLWorker::initializeMainThread(): Code: 999. Coordination::Exception: All connection tries failed while connecting to ZooKeeper. nodes: [::1]:9181, [::1]:9183, [::1]:9182 | ||
| 78 | Poco::Exception. Code: 1000, e.code() = 146, Connection refused: [::1]:9181 (version 23.8.7.1), [::1]:9181 | ||
| 79 | Poco::Exception. Code: 1000, e.code() = 146, Connection refused: [::1]:9183 (version 23.8.7.1), [::1]:9183 | ||
| 80 | Poco::Exception. Code: 1000, e.code() = 146, Connection refused: [::1]:9182 (version 23.8.7.1), [::1]:9182 | ||
| 81 | Poco::Exception. Code: 1000, e.code() = 146, Connection refused: [::1]:9181 (version 23.8.7.1), [::1]:9181 | ||
| 82 | Poco::Exception. Code: 1000, e.code() = 146, Connection refused: [::1]:9183 (version 23.8.7.1), [::1]:9183 | ||
| 83 | Poco::Exception. Code: 1000, e.code() = 146, Connection refused: [::1]:9182 (version 23.8.7.1), [::1]:9182 | ||
| 84 | Poco::Exception. Code: 1000, e.code() = 146, Connection refused: [::1]:9181 (version 23.8.7.1), [::1]:9181 | ||
| 85 | Poco::Exception. Code: 1000, e.code() = 146, Connection refused: [::1]:9183 (version 23.8.7.1), [::1]:9183 | ||
| 86 | Poco::Exception. Code: 1000, e.code() = 146, Connection refused: [::1]:9182 (version 23.8.7.1), [::1]:9182 | ||
| 87 | . (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below): | ||
| 88 | |||
| 89 | 0. StackTrace::tryCapture() @ 0x000000000f9acdb7 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 90 | 1. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000f97f7df in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 91 | 2. DB::Exception::Exception<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&>(int, FormatStringHelperImpl<std::type_identity<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&>::type>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&) @ 0x000000000a7857f7 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 92 | 3. Coordination::Exception::Exception<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&>::type>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&) @ 0x0000000014e3842d in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 93 | 4. Coordination::ZooKeeper::connect(std::vector<Coordination::ZooKeeper::Node, std::allocator<Coordination::ZooKeeper::Node> > const&, Poco::Timespan) @ 0x000000001648290a in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 94 | 5. Coordination::ZooKeeper::ZooKeeper(std::vector<Coordination::ZooKeeper::Node, std::allocator<Coordination::ZooKeeper::Node> > const&, zkutil::ZooKeeperArgs const&, std::shared_ptr<DB::ZooKeeperLog>, std::optional<std::function<void (unsigned long, Coordination::ZooKeeper::Node const&)> >&&) @ 0x000000001648050b in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 95 | 6. zkutil::ZooKeeper::init(zkutil::ZooKeeperArgs) @ 0x0000000016449829 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 96 | 7. zkutil::ZooKeeper::ZooKeeper(Poco::Util::AbstractConfiguration const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<DB::ZooKeeperLog>) @ 0x000000001644b0a3 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 97 | 8. DB::Context::getZooKeeper() const @ 0x0000000014755b49 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 98 | 9. DB::DDLWorker::getAndSetZooKeeper() @ 0x0000000014792b4a in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 99 | 10. DB::DDLWorker::initializeMainThread() @ 0x00000000147a134c in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 100 | 11. DB::DDLWorker::runMainThread() @ 0x0000000014791391 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 101 | 12. ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<void (DB::DDLWorker::*)(), DB::DDLWorker*>(void (DB::DDLWorker::*&&)(), DB::DDLWorker*&&)::{lambda()#1}::operator()() @ 0x00000000147a2fd5 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 102 | 13. ThreadPoolImpl<std::thread>::worker(std::_List_iterator<std::thread>) @ 0x00000000182ede9c in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 103 | 14. ? @ 0x000000000014d750 in /usr/gcc/13/lib/amd64/libstdc++.so.6.0.32 | ||
| 104 | 15. _thrp_setup @ 0x0000000000116767 in /lib/amd64/libc.so.1 | ||
| 105 | (version 23.8.7.1) | ||
| 106 | 2025.05.14 01:41:25.000509 [ 254 ] {} <Trace> AsynchronousMetrics: MemoryTracking: was 145.32 MiB, peak 145.32 MiB, free memory in arenas 0.00 B, will set to 151.03 MiB (RSS), difference: 5.70 MiB | ||