2025.05.14 01:41:24.174429 [ 1 ] {} <Information> Application: Starting ClickHouse 23.8.7.1 (revision: 54477, git hash: 3042d295d963012962f5c683bd2776fa331a38c3, build id: <unknown>), PID 16487
2025.05.14 01:41:24.174764 [ 1 ] {} <Information> Application: starting up
2025.05.14 01:41:24.174821 [ 1 ] {} <Information> Application: OS name: SunOS, version: 5.11, architecture: i86pc
2025.05.14 01:41:24.195374 [ 1 ] {} <Information> Application: Available RAM: 31.99 GiB; physical cores: 8; logical cores: 8.
2025.05.14 01:41:24.204001 [ 1 ] {} <Debug> Application: rlimit on number of file descriptors is 65536
2025.05.14 01:41:24.204121 [ 1 ] {} <Debug> Application: Initializing DateLUT.
2025.05.14 01:41:24.204139 [ 1 ] {} <Trace> Application: Initialized DateLUT with time zone 'UTC'.
2025.05.14 01:41:24.204210 [ 1 ] {} <Debug> Context: Setting up /var/tmp/omicron_tmp/oximeter_db-11d8f51cb75f441f-test_ensure_schema_walks_through_multiple_steps_replicated.16475.5-clickhouse-gCPVKS/tmp/ to store temporary data in it
2025.05.14 01:41:24.205623 [ 1 ] {} <Debug> Application: Initializing interserver credentials.
2025.05.14 01:41:24.206511 [ 1 ] {} <Information> SensitiveDataMaskerConfigRead: 1 query masking rules loaded.
2025.05.14 01:41:24.211514 [ 1 ] {} <Debug> ConfigReloader: Loading config '/work/oxidecomputer/omicron/oximeter/db/src/configs/replica_config.xml'
2025.05.14 01:41:24.216114 [ 1 ] {} <Debug> ConfigReloader: Loaded config '/work/oxidecomputer/omicron/oximeter/db/src/configs/replica_config.xml', performing update on configuration
2025.05.14 01:41:24.219249 [ 1 ] {} <Information> Application: Setting max_server_memory_usage was set to 28.79 GiB (31.99 GiB available * 0.90 max_server_memory_usage_to_ram_ratio)
2025.05.14 01:41:24.219324 [ 1 ] {} <Information> Application: Setting merges_mutations_memory_usage_soft_limit was set to 16.00 GiB (31.99 GiB available * 0.50 merges_mutations_memory_usage_to_ram_ratio)
2025.05.14 01:41:24.219338 [ 1 ] {} <Information> Application: Merges and mutations memory limit is set to 16.00 GiB
2025.05.14 01:41:24.222190 [ 1 ] {} <Information> BackgroundSchedulePool/BgBufSchPool: Create BackgroundSchedulePool with 16 threads
2025.05.14 01:41:24.223381 [ 1 ] {} <Information> BackgroundSchedulePool/BgSchPool: Create BackgroundSchedulePool with 128 threads
2025.05.14 01:41:24.268871 [ 1 ] {} <Information> BackgroundSchedulePool/BgMBSchPool: Create BackgroundSchedulePool with 16 threads
2025.05.14 01:41:24.275726 [ 1 ] {} <Information> BackgroundSchedulePool/BgDistSchPool: Create BackgroundSchedulePool with 16 threads
2025.05.14 01:41:24.277720 [ 1 ] {} <Information> CertificateReloader: One of paths is empty. Cannot apply new configuration for certificates. Fill all paths and try again.
2025.05.14 01:41:24.277883 [ 1 ] {} <Debug> ConfigReloader: Loaded config '/work/oxidecomputer/omicron/oximeter/db/src/configs/replica_config.xml', performed update on configuration
2025.05.14 01:41:24.279255 [ 1 ] {} <Information> Application: Listening for replica communication (interserver): http://[::]:9010
2025.05.14 01:41:24.280842 [ 1 ] {} <Debug> ConfigReloader: Loading config '/work/oxidecomputer/omicron/oximeter/db/src/configs/replica_config.xml'
2025.05.14 01:41:24.289132 [ 1 ] {} <Debug> ConfigReloader: Loaded config '/work/oxidecomputer/omicron/oximeter/db/src/configs/replica_config.xml', performing update on configuration
2025.05.14 01:41:24.293584 [ 1 ] {} <Debug> ConfigReloader: Loaded config '/work/oxidecomputer/omicron/oximeter/db/src/configs/replica_config.xml', performed update on configuration
2025.05.14 01:41:24.295135 [ 1 ] {} <Debug> Access(user directories): Added users_xml access storage 'users_xml', path: /work/oxidecomputer/omicron/oximeter/db/src/configs/replica_config.xml
2025.05.14 01:41:24.298505 [ 1 ] {} <Information> Context: Initialized background executor for merges and mutations with num_threads=16, num_tasks=32, scheduling_policy=round_robin
2025.05.14 01:41:24.300427 [ 1 ] {} <Information> Context: Initialized background executor for move operations with num_threads=8, num_tasks=8
2025.05.14 01:41:24.301537 [ 1 ] {} <Information> Context: Initialized background executor for fetches with num_threads=8, num_tasks=8
2025.05.14 01:41:24.303486 [ 1 ] {} <Information> Context: Initialized background executor for common operations (e.g. clearing old parts) with num_threads=8, num_tasks=8
2025.05.14 01:41:24.306734 [ 1 ] {} <Information> DNSCacheUpdater: Update period 15 seconds
2025.05.14 01:41:24.308560 [ 1 ] {} <Information> Application: Loading metadata from /var/tmp/omicron_tmp/oximeter_db-11d8f51cb75f441f-test_ensure_schema_walks_through_multiple_steps_replicated.16475.5-clickhouse-gCPVKS/datastore/
2025.05.14 01:41:24.308706 [ 150 ] {} <Debug> DNSResolver: Updating DNS cache
2025.05.14 01:41:24.308845 [ 150 ] {} <Debug> DNSResolver: Updated DNS cache
2025.05.14 01:41:24.311203 [ 1 ] {} <Information> DatabaseAtomic (system): Metadata processed, database system has 0 tables and 0 dictionaries in total.
2025.05.14 01:41:24.311504 [ 1 ] {} <Information> TablesLoader: Parsed metadata of 0 tables in 1 databases in 0.000498505 sec
2025.05.14 01:41:24.311528 [ 1 ] {} <Trace> ReferentialDeps: No tables
2025.05.14 01:41:24.311537 [ 1 ] {} <Trace> LoadingDeps: No tables
2025.05.14 01:41:24.311899 [ 1 ] {} <Information> DatabaseAtomic (system): Starting up tables.
2025.05.14 01:41:24.312155 [ 1 ] {} <Debug> SystemLog: Creating system.query_log from query_log
2025.05.14 01:41:24.331280 [ 1 ] {} <Debug> SystemLog: Creating system.query_thread_log from query_thread_log
2025.05.14 01:41:24.332285 [ 1 ] {} <Debug> SystemLog: Creating system.part_log from part_log
2025.05.14 01:41:24.332875 [ 1 ] {} <Debug> SystemLog: Creating system.trace_log from trace_log
2025.05.14 01:41:24.333277 [ 1 ] {} <Debug> SystemLog: Creating system.crash_log from crash_log
2025.05.14 01:41:24.335196 [ 1 ] {} <Debug> SystemLog: Not creating system.text_log since corresponding section 'text_log' is missing from config
2025.05.14 01:41:24.335237 [ 1 ] {} <Debug> SystemLog: Creating system.metric_log from metric_log
2025.05.14 01:41:24.344320 [ 1 ] {} <Debug> SystemLog: Not creating system.filesystem_cache_log since corresponding section 'filesystem_cache_log' is missing from config
2025.05.14 01:41:24.344434 [ 1 ] {} <Debug> SystemLog: Not creating system.filesystem_read_prefetches_log since corresponding section 'filesystem_read_prefetches_log' is missing from config
2025.05.14 01:41:24.344454 [ 1 ] {} <Debug> SystemLog: Creating system.asynchronous_metric_log from asynchronous_metric_log
2025.05.14 01:41:24.345287 [ 1 ] {} <Debug> SystemLog: Not creating system.opentelemetry_span_log since corresponding section 'opentelemetry_span_log' is missing from config
2025.05.14 01:41:24.345341 [ 1 ] {} <Debug> SystemLog: Creating system.query_views_log from query_views_log
2025.05.14 01:41:24.345820 [ 1 ] {} <Debug> SystemLog: Not creating system.zookeeper_log since corresponding section 'zookeeper_log' is missing from config
2025.05.14 01:41:24.345854 [ 1 ] {} <Debug> SystemLog: Not creating system.session_log since corresponding section 'session_log' is missing from config
2025.05.14 01:41:24.345876 [ 1 ] {} <Debug> SystemLog: Not creating system.transactions_info_log since corresponding section 'transactions_info_log' is missing from config
2025.05.14 01:41:24.345914 [ 1 ] {} <Debug> SystemLog: Creating system.processors_profile_log from processors_profile_log
2025.05.14 01:41:24.346174 [ 1 ] {} <Debug> SystemLog: Not creating system.asynchronous_insert_log since corresponding section 'asynchronous_insert_log' is missing from config
2025.05.14 01:41:24.441618 [ 1 ] {} <Information> DatabaseCatalog: Found 0 partially dropped tables. Will load them and retry removal.
2025.05.14 01:41:24.443356 [ 1 ] {} <Information> DatabaseAtomic (default): Metadata processed, database default has 0 tables and 0 dictionaries in total.
2025.05.14 01:41:24.443397 [ 1 ] {} <Information> TablesLoader: Parsed metadata of 0 tables in 1 databases in 0.000108101 sec
2025.05.14 01:41:24.443411 [ 1 ] {} <Trace> ReferentialDeps: No tables
2025.05.14 01:41:24.443421 [ 1 ] {} <Trace> LoadingDeps: No tables
2025.05.14 01:41:24.443666 [ 1 ] {} <Information> DatabaseAtomic (default): Starting up tables.
2025.05.14 01:41:24.444042 [ 1 ] {} <Information> UserDefinedSQLObjectsLoaderFromDisk: Loading user defined objects from /var/tmp/omicron_tmp/oximeter_db-11d8f51cb75f441f-test_ensure_schema_walks_through_multiple_steps_replicated.16475.5-clickhouse-gCPVKS/datastore/user_defined/
2025.05.14 01:41:24.444127 [ 1 ] {} <Debug> UserDefinedSQLObjectsLoaderFromDisk: User defined objects loaded
2025.05.14 01:41:24.444155 [ 1 ] {} <Debug> Application: Loaded metadata.
2025.05.14 01:41:24.444172 [ 1 ] {} <Information> Application: Query Profiler and TraceCollector are disabled because they require PHDR cache to be created (otherwise the function 'dl_iterate_phdr' is not lock free and not async-signal safe).
2025.05.14 01:41:24.444183 [ 1 ] {} <Information> Application: TaskStats is not implemented for this OS. IO accounting will be disabled.
2025.05.14 01:41:24.444601 [ 1 ] {} <Information> CertificateReloader: One of paths is empty. Cannot apply new configuration for certificates. Fill all paths and try again.
2025.05.14 01:41:24.444876 [ 1 ] {} <Trace> AsynchronousMetrics: MemoryTracking: was 40.97 MiB, peak 40.97 MiB, free memory in arenas 0.00 B, will set to 145.30 MiB (RSS), difference: 104.33 MiB
2025.05.14 01:41:24.445349 [ 1 ] {} <Trace> AsynchronousMetrics: Update heavy metrics. Update period 1 sec. Update heavy metrics period 120 sec. Heavy metrics calculation elapsed: 2.4563e-05 sec.
2025.05.14 01:41:24.446529 [ 1 ] {} <Information> Application: Listening for http://[::]:8124
2025.05.14 01:41:24.446621 [ 1 ] {} <Information> Application: Listening for native protocol (tcp): [::]:9002
2025.05.14 01:41:24.446637 [ 1 ] {} <Information> Application: Ready for connections.
2025.05.14 01:41:24.446985 [ 258 ] {} <Debug> DDLWorker: Started DDLWorker cleanup thread
2025.05.14 01:41:24.447449 [ 257 ] {} <Debug> DDLWorker: Starting DDLWorker thread
2025.05.14 01:41:24.447526 [ 257 ] {} <Debug> DDLWorker: Initializing DDLWorker thread
2025.05.14 01:41:24.471444 [ 257 ] {} <Error> virtual bool DB::DDLWorker::initializeMainThread(): Code: 999. Coordination::Exception: All connection tries failed while connecting to ZooKeeper. nodes: [::1]:9181, [::1]:9183, [::1]:9182
Poco::Exception. Code: 1000, e.code() = 146, Connection refused: [::1]:9181 (version 23.8.7.1), [::1]:9181
Poco::Exception. Code: 1000, e.code() = 146, Connection refused: [::1]:9183 (version 23.8.7.1), [::1]:9183
Poco::Exception. Code: 1000, e.code() = 146, Connection refused: [::1]:9182 (version 23.8.7.1), [::1]:9182
Poco::Exception. Code: 1000, e.code() = 146, Connection refused: [::1]:9181 (version 23.8.7.1), [::1]:9181
Poco::Exception. Code: 1000, e.code() = 146, Connection refused: [::1]:9183 (version 23.8.7.1), [::1]:9183
Poco::Exception. Code: 1000, e.code() = 146, Connection refused: [::1]:9182 (version 23.8.7.1), [::1]:9182
Poco::Exception. Code: 1000, e.code() = 146, Connection refused: [::1]:9181 (version 23.8.7.1), [::1]:9181
Poco::Exception. Code: 1000, e.code() = 146, Connection refused: [::1]:9183 (version 23.8.7.1), [::1]:9183
Poco::Exception. Code: 1000, e.code() = 146, Connection refused: [::1]:9182 (version 23.8.7.1), [::1]:9182
. (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. StackTrace::tryCapture() @ 0x000000000f9acdb7 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
1. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000f97f7df in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
2. DB::Exception::Exception<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&>(int, FormatStringHelperImpl<std::type_identity<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&>::type>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&) @ 0x000000000a7857f7 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
3. Coordination::Exception::Exception<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&>::type>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&) @ 0x0000000014e3842d in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
4. Coordination::ZooKeeper::connect(std::vector<Coordination::ZooKeeper::Node, std::allocator<Coordination::ZooKeeper::Node> > const&, Poco::Timespan) @ 0x000000001648290a in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
5. Coordination::ZooKeeper::ZooKeeper(std::vector<Coordination::ZooKeeper::Node, std::allocator<Coordination::ZooKeeper::Node> > const&, zkutil::ZooKeeperArgs const&, std::shared_ptr<DB::ZooKeeperLog>, std::optional<std::function<void (unsigned long, Coordination::ZooKeeper::Node const&)> >&&) @ 0x000000001648050b in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
6. zkutil::ZooKeeper::init(zkutil::ZooKeeperArgs) @ 0x0000000016449829 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
7. zkutil::ZooKeeper::ZooKeeper(Poco::Util::AbstractConfiguration const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<DB::ZooKeeperLog>) @ 0x000000001644b0a3 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
8. DB::Context::getZooKeeper() const @ 0x0000000014755b49 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
9. DB::DDLWorker::getAndSetZooKeeper() @ 0x0000000014792b4a in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
10. DB::DDLWorker::initializeMainThread() @ 0x00000000147a134c in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
11. DB::DDLWorker::runMainThread() @ 0x0000000014791391 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
12. ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<void (DB::DDLWorker::*)(), DB::DDLWorker*>(void (DB::DDLWorker::*&&)(), DB::DDLWorker*&&)::{lambda()#1}::operator()() @ 0x00000000147a2fd5 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
13. ThreadPoolImpl<std::thread>::worker(std::_List_iterator<std::thread>) @ 0x00000000182ede9c in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
14. ? @ 0x000000000014d750 in /usr/gcc/13/lib/amd64/libstdc++.so.6.0.32
15. _thrp_setup @ 0x0000000000116767 in /lib/amd64/libc.so.1
 (version 23.8.7.1)
2025.05.14 01:41:25.000509 [ 254 ] {} <Trace> AsynchronousMetrics: MemoryTracking: was 145.32 MiB, peak 145.32 MiB, free memory in arenas 0.00 B, will set to 151.03 MiB (RSS), difference: 5.70 MiB