2025.05.14 01:41:23.425146 [ 1 ] {} <Information> Application: Starting ClickHouse 23.8.7.1 (revision: 54477, git hash: 3042d295d963012962f5c683bd2776fa331a38c3, build id: <unknown>), PID 16484
2025.05.14 01:41:23.425479 [ 1 ] {} <Information> Application: starting up
2025.05.14 01:41:23.425542 [ 1 ] {} <Information> Application: OS name: SunOS, version: 5.11, architecture: i86pc
2025.05.14 01:41:23.439115 [ 1 ] {} <Information> Application: Available RAM: 31.99 GiB; physical cores: 8; logical cores: 8.
2025.05.14 01:41:23.454465 [ 1 ] {} <Debug> Application: rlimit on number of file descriptors is 65536
2025.05.14 01:41:23.454568 [ 1 ] {} <Debug> Application: Initializing DateLUT.
2025.05.14 01:41:23.454680 [ 1 ] {} <Trace> Application: Initialized DateLUT with time zone 'UTC'.
2025.05.14 01:41:23.454851 [ 1 ] {} <Debug> Context: Setting up /var/tmp/omicron_tmp/oximeter_db-11d8f51cb75f441f-test_ensure_schema_walks_through_multiple_steps_replicated.16475.4-clickhouse-fgDTKD/tmp/ to store temporary data in it
2025.05.14 01:41:23.456042 [ 1 ] {} <Debug> Application: Initializing interserver credentials.
2025.05.14 01:41:23.457041 [ 1 ] {} <Information> SensitiveDataMaskerConfigRead: 1 query masking rules loaded.
2025.05.14 01:41:23.458098 [ 1 ] {} <Debug> ConfigReloader: Loading config '/work/oxidecomputer/omicron/oximeter/db/src/configs/replica_config.xml'
2025.05.14 01:41:23.462592 [ 1 ] {} <Debug> ConfigReloader: Loaded config '/work/oxidecomputer/omicron/oximeter/db/src/configs/replica_config.xml', performing update on configuration
2025.05.14 01:41:23.465421 [ 1 ] {} <Information> Application: Setting max_server_memory_usage was set to 28.79 GiB (31.99 GiB available * 0.90 max_server_memory_usage_to_ram_ratio)
2025.05.14 01:41:23.465515 [ 1 ] {} <Information> Application: Setting merges_mutations_memory_usage_soft_limit was set to 16.00 GiB (31.99 GiB available * 0.50 merges_mutations_memory_usage_to_ram_ratio)
2025.05.14 01:41:23.465542 [ 1 ] {} <Information> Application: Merges and mutations memory limit is set to 16.00 GiB
2025.05.14 01:41:23.467223 [ 1 ] {} <Information> BackgroundSchedulePool/BgBufSchPool: Create BackgroundSchedulePool with 16 threads
2025.05.14 01:41:23.472625 [ 1 ] {} <Information> BackgroundSchedulePool/BgSchPool: Create BackgroundSchedulePool with 128 threads
2025.05.14 01:41:23.492747 [ 1 ] {} <Information> BackgroundSchedulePool/BgMBSchPool: Create BackgroundSchedulePool with 16 threads
2025.05.14 01:41:23.493785 [ 1 ] {} <Information> BackgroundSchedulePool/BgDistSchPool: Create BackgroundSchedulePool with 16 threads
2025.05.14 01:41:23.515844 [ 1 ] {} <Information> CertificateReloader: One of paths is empty. Cannot apply new configuration for certificates. Fill all paths and try again.
2025.05.14 01:41:23.515951 [ 1 ] {} <Debug> ConfigReloader: Loaded config '/work/oxidecomputer/omicron/oximeter/db/src/configs/replica_config.xml', performed update on configuration
2025.05.14 01:41:23.517145 [ 1 ] {} <Information> Application: Listening for replica communication (interserver): http://[::]:9009
2025.05.14 01:41:23.518365 [ 1 ] {} <Debug> ConfigReloader: Loading config '/work/oxidecomputer/omicron/oximeter/db/src/configs/replica_config.xml'
2025.05.14 01:41:23.522129 [ 1 ] {} <Debug> ConfigReloader: Loaded config '/work/oxidecomputer/omicron/oximeter/db/src/configs/replica_config.xml', performing update on configuration
2025.05.14 01:41:23.525370 [ 1 ] {} <Debug> ConfigReloader: Loaded config '/work/oxidecomputer/omicron/oximeter/db/src/configs/replica_config.xml', performed update on configuration
2025.05.14 01:41:23.526326 [ 1 ] {} <Debug> Access(user directories): Added users_xml access storage 'users_xml', path: /work/oxidecomputer/omicron/oximeter/db/src/configs/replica_config.xml
2025.05.14 01:41:23.527885 [ 1 ] {} <Information> Context: Initialized background executor for merges and mutations with num_threads=16, num_tasks=32, scheduling_policy=round_robin
2025.05.14 01:41:23.528391 [ 1 ] {} <Information> Context: Initialized background executor for move operations with num_threads=8, num_tasks=8
2025.05.14 01:41:23.528847 [ 1 ] {} <Information> Context: Initialized background executor for fetches with num_threads=8, num_tasks=8
2025.05.14 01:41:23.529335 [ 1 ] {} <Information> Context: Initialized background executor for common operations (e.g. clearing old parts) with num_threads=8, num_tasks=8
2025.05.14 01:41:23.530825 [ 1 ] {} <Information> DNSCacheUpdater: Update period 15 seconds
2025.05.14 01:41:23.530861 [ 1 ] {} <Information> Application: Loading metadata from /var/tmp/omicron_tmp/oximeter_db-11d8f51cb75f441f-test_ensure_schema_walks_through_multiple_steps_replicated.16475.4-clickhouse-fgDTKD/datastore/
2025.05.14 01:41:23.533341 [ 1 ] {} <Information> DatabaseAtomic (system): Metadata processed, database system has 0 tables and 0 dictionaries in total.
2025.05.14 01:41:23.533372 [ 1 ] {} <Information> TablesLoader: Parsed metadata of 0 tables in 1 databases in 0.000239132 sec
2025.05.14 01:41:23.533385 [ 1 ] {} <Trace> ReferentialDeps: No tables
2025.05.14 01:41:23.533393 [ 1 ] {} <Trace> LoadingDeps: No tables
2025.05.14 01:41:23.533557 [ 39 ] {} <Debug> DNSResolver: Updating DNS cache
2025.05.14 01:41:23.533621 [ 1 ] {} <Information> DatabaseAtomic (system): Starting up tables.
2025.05.14 01:41:23.533682 [ 39 ] {} <Debug> DNSResolver: Updated DNS cache
2025.05.14 01:41:23.533708 [ 1 ] {} <Debug> SystemLog: Creating system.query_log from query_log
2025.05.14 01:41:23.536636 [ 1 ] {} <Debug> SystemLog: Creating system.query_thread_log from query_thread_log
2025.05.14 01:41:23.537147 [ 1 ] {} <Debug> SystemLog: Creating system.part_log from part_log
2025.05.14 01:41:23.537848 [ 1 ] {} <Debug> SystemLog: Creating system.trace_log from trace_log
2025.05.14 01:41:23.538438 [ 1 ] {} <Debug> SystemLog: Creating system.crash_log from crash_log
2025.05.14 01:41:23.538641 [ 1 ] {} <Debug> SystemLog: Not creating system.text_log since corresponding section 'text_log' is missing from config
2025.05.14 01:41:23.538810 [ 1 ] {} <Debug> SystemLog: Creating system.metric_log from metric_log
2025.05.14 01:41:23.543862 [ 1 ] {} <Debug> SystemLog: Not creating system.filesystem_cache_log since corresponding section 'filesystem_cache_log' is missing from config
2025.05.14 01:41:23.543961 [ 1 ] {} <Debug> SystemLog: Not creating system.filesystem_read_prefetches_log since corresponding section 'filesystem_read_prefetches_log' is missing from config
2025.05.14 01:41:23.543978 [ 1 ] {} <Debug> SystemLog: Creating system.asynchronous_metric_log from asynchronous_metric_log
2025.05.14 01:41:23.544336 [ 1 ] {} <Debug> SystemLog: Not creating system.opentelemetry_span_log since corresponding section 'opentelemetry_span_log' is missing from config
2025.05.14 01:41:23.544374 [ 1 ] {} <Debug> SystemLog: Creating system.query_views_log from query_views_log
2025.05.14 01:41:23.544771 [ 1 ] {} <Debug> SystemLog: Not creating system.zookeeper_log since corresponding section 'zookeeper_log' is missing from config
2025.05.14 01:41:23.544798 [ 1 ] {} <Debug> SystemLog: Not creating system.session_log since corresponding section 'session_log' is missing from config
2025.05.14 01:41:23.544820 [ 1 ] {} <Debug> SystemLog: Not creating system.transactions_info_log since corresponding section 'transactions_info_log' is missing from config
2025.05.14 01:41:23.544859 [ 1 ] {} <Debug> SystemLog: Creating system.processors_profile_log from processors_profile_log
2025.05.14 01:41:23.545140 [ 1 ] {} <Debug> SystemLog: Not creating system.asynchronous_insert_log since corresponding section 'asynchronous_insert_log' is missing from config
2025.05.14 01:41:23.594269 [ 1 ] {} <Information> DatabaseCatalog: Found 0 partially dropped tables. Will load them and retry removal.
2025.05.14 01:41:23.595202 [ 1 ] {} <Information> DatabaseAtomic (default): Metadata processed, database default has 0 tables and 0 dictionaries in total.
2025.05.14 01:41:23.595236 [ 1 ] {} <Information> TablesLoader: Parsed metadata of 0 tables in 1 databases in 9.5444e-05 sec
2025.05.14 01:41:23.595248 [ 1 ] {} <Trace> ReferentialDeps: No tables
2025.05.14 01:41:23.595256 [ 1 ] {} <Trace> LoadingDeps: No tables
2025.05.14 01:41:23.595340 [ 1 ] {} <Information> DatabaseAtomic (default): Starting up tables.
2025.05.14 01:41:23.595549 [ 1 ] {} <Information> UserDefinedSQLObjectsLoaderFromDisk: Loading user defined objects from /var/tmp/omicron_tmp/oximeter_db-11d8f51cb75f441f-test_ensure_schema_walks_through_multiple_steps_replicated.16475.4-clickhouse-fgDTKD/datastore/user_defined/
2025.05.14 01:41:23.595610 [ 1 ] {} <Debug> UserDefinedSQLObjectsLoaderFromDisk: User defined objects loaded
2025.05.14 01:41:23.595814 [ 1 ] {} <Debug> Application: Loaded metadata.
2025.05.14 01:41:23.595847 [ 1 ] {} <Information> Application: Query Profiler and TraceCollector are disabled because they require PHDR cache to be created (otherwise the function 'dl_iterate_phdr' is not lock free and not async-signal safe).
2025.05.14 01:41:23.595858 [ 1 ] {} <Information> Application: TaskStats is not implemented for this OS. IO accounting will be disabled.
2025.05.14 01:41:23.596183 [ 1 ] {} <Information> CertificateReloader: One of paths is empty. Cannot apply new configuration for certificates. Fill all paths and try again.
2025.05.14 01:41:23.596469 [ 1 ] {} <Trace> AsynchronousMetrics: MemoryTracking: was 40.98 MiB, peak 40.98 MiB, free memory in arenas 0.00 B, will set to 145.02 MiB (RSS), difference: 104.04 MiB
2025.05.14 01:41:23.596870 [ 1 ] {} <Trace> AsynchronousMetrics: Update heavy metrics. Update period 1 sec. Update heavy metrics period 120 sec. Heavy metrics calculation elapsed: 1.8409e-05 sec.
2025.05.14 01:41:23.597847 [ 1 ] {} <Information> Application: Listening for http://[::]:8123
2025.05.14 01:41:23.597931 [ 1 ] {} <Information> Application: Listening for native protocol (tcp): [::]:9001
2025.05.14 01:41:23.597945 [ 1 ] {} <Information> Application: Ready for connections.
2025.05.14 01:41:23.598318 [ 259 ] {} <Debug> DDLWorker: Started DDLWorker cleanup thread
2025.05.14 01:41:23.598618 [ 255 ] {} <Debug> DDLWorker: Starting DDLWorker thread
2025.05.14 01:41:23.598652 [ 255 ] {} <Debug> DDLWorker: Initializing DDLWorker thread
2025.05.14 01:41:23.611521 [ 255 ] {} <Error> virtual bool DB::DDLWorker::initializeMainThread(): Code: 999. Coordination::Exception: All connection tries failed while connecting to ZooKeeper. nodes: [::1]:9182, [::1]:9181, [::1]:9183
Poco::Exception. Code: 1000, e.code() = 146, Connection refused: [::1]:9182 (version 23.8.7.1), [::1]:9182
Poco::Exception. Code: 1000, e.code() = 146, Connection refused: [::1]:9181 (version 23.8.7.1), [::1]:9181
Poco::Exception. Code: 1000, e.code() = 146, Connection refused: [::1]:9183 (version 23.8.7.1), [::1]:9183
Poco::Exception. Code: 1000, e.code() = 146, Connection refused: [::1]:9182 (version 23.8.7.1), [::1]:9182
Poco::Exception. Code: 1000, e.code() = 146, Connection refused: [::1]:9181 (version 23.8.7.1), [::1]:9181
Poco::Exception. Code: 1000, e.code() = 146, Connection refused: [::1]:9183 (version 23.8.7.1), [::1]:9183
Poco::Exception. Code: 1000, e.code() = 146, Connection refused: [::1]:9182 (version 23.8.7.1), [::1]:9182
Poco::Exception. Code: 1000, e.code() = 146, Connection refused: [::1]:9181 (version 23.8.7.1), [::1]:9181
Poco::Exception. Code: 1000, e.code() = 146, Connection refused: [::1]:9183 (version 23.8.7.1), [::1]:9183
. (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. StackTrace::tryCapture() @ 0x000000000f9acdb7 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
1. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000f97f7df in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
2. DB::Exception::Exception<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&>(int, FormatStringHelperImpl<std::type_identity<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&>::type>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&) @ 0x000000000a7857f7 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
3. Coordination::Exception::Exception<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&>::type>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&) @ 0x0000000014e3842d in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
4. Coordination::ZooKeeper::connect(std::vector<Coordination::ZooKeeper::Node, std::allocator<Coordination::ZooKeeper::Node> > const&, Poco::Timespan) @ 0x000000001648290a in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
5. Coordination::ZooKeeper::ZooKeeper(std::vector<Coordination::ZooKeeper::Node, std::allocator<Coordination::ZooKeeper::Node> > const&, zkutil::ZooKeeperArgs const&, std::shared_ptr<DB::ZooKeeperLog>, std::optional<std::function<void (unsigned long, Coordination::ZooKeeper::Node const&)> >&&) @ 0x000000001648050b in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
6. zkutil::ZooKeeper::init(zkutil::ZooKeeperArgs) @ 0x0000000016449829 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
7. zkutil::ZooKeeper::ZooKeeper(Poco::Util::AbstractConfiguration const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<DB::ZooKeeperLog>) @ 0x000000001644b0a3 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
8. DB::Context::getZooKeeper() const @ 0x0000000014755b49 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
9. DB::DDLWorker::getAndSetZooKeeper() @ 0x0000000014792b4a in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
10. DB::DDLWorker::initializeMainThread() @ 0x00000000147a134c in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
11. DB::DDLWorker::runMainThread() @ 0x0000000014791391 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
12. ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<void (DB::DDLWorker::*)(), DB::DDLWorker*>(void (DB::DDLWorker::*&&)(), DB::DDLWorker*&&)::{lambda()#1}::operator()() @ 0x00000000147a2fd5 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
13. ThreadPoolImpl<std::thread>::worker(std::_List_iterator<std::thread>) @ 0x00000000182ede9c in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
14. ? @ 0x000000000014d750 in /usr/gcc/13/lib/amd64/libstdc++.so.6.0.32
15. _thrp_setup @ 0x0000000000116767 in /lib/amd64/libc.so.1
 (version 23.8.7.1)
2025.05.14 01:41:24.000556 [ 254 ] {} <Trace> AsynchronousMetrics: MemoryTracking: was 145.05 MiB, peak 145.05 MiB, free memory in arenas 0.00 B, will set to 150.77 MiB (RSS), difference: 5.72 MiB
2025.05.14 01:41:24.452653 [ 3 ] {} <Trace> TCPHandlerFactory: TCP Request. Address: [::1]:55581
2025.05.14 01:41:24.453868 [ 3 ] {} <Debug> TCPHandler: Connected oximeter version 23.8.0, revision: 54465, database: default, user: default.
2025.05.14 01:41:24.453987 [ 3 ] {} <Debug> TCP-Session: e9af9fc0-ee11-4d98-b1ae-215d2a9f6564 Authenticating user 'default' from [::1]:55581
2025.05.14 01:41:24.454031 [ 3 ] {} <Debug> TCP-Session: e9af9fc0-ee11-4d98-b1ae-215d2a9f6564 Authenticated with global context as user 94309d50-4f52-5250-31bd-74fecac179db
2025.05.14 01:41:24.454061 [ 3 ] {} <Debug> TCP-Session: e9af9fc0-ee11-4d98-b1ae-215d2a9f6564 Creating session context with user_id: 94309d50-4f52-5250-31bd-74fecac179db
2025.05.14 01:41:24.454419 [ 3 ] {} <Trace> ContextAccess (default): Settings: readonly = 0, allow_ddl = true, allow_introspection_functions = false
2025.05.14 01:41:24.454498 [ 3 ] {} <Trace> ContextAccess (default): List of all grants: GRANT SHOW, SELECT, INSERT, ALTER, CREATE, DROP, UNDROP TABLE, TRUNCATE, OPTIMIZE, BACKUP, KILL QUERY, KILL TRANSACTION, MOVE PARTITION BETWEEN SHARDS, SYSTEM, dictGet, displaySecretsInShowAndSelect, INTROSPECTION, SOURCES, CLUSTER ON *.*
2025.05.14 01:41:24.454528 [ 3 ] {} <Trace> ContextAccess (default): List of all grants including implicit: GRANT SHOW, SELECT, INSERT, ALTER, CREATE, DROP, UNDROP TABLE, TRUNCATE, OPTIMIZE, BACKUP, KILL QUERY, KILL TRANSACTION, MOVE PARTITION BETWEEN SHARDS, SYSTEM, dictGet, displaySecretsInShowAndSelect, INTROSPECTION, SOURCES, CLUSTER ON *.*
2025.05.14 01:41:24.455470 [ 4 ] {} <Trace> TCPHandlerFactory: TCP Request. Address: [::1]:46131
2025.05.14 01:41:24.455935 [ 5 ] {} <Trace> TCPHandlerFactory: TCP Request. Address: [::1]:54323
2025.05.14 01:41:24.456696 [ 5 ] {} <Debug> TCPHandler: Connected oximeter version 23.8.0, revision: 54465, database: default, user: default.
2025.05.14 01:41:24.456712 [ 4 ] {} <Debug> TCPHandler: Connected oximeter version 23.8.0, revision: 54465, database: default, user: default.
2025.05.14 01:41:24.456814 [ 5 ] {} <Debug> TCP-Session: 557d40c5-e419-49c7-a8d5-01263aa8bdea Authenticating user 'default' from [::1]:54323
2025.05.14 01:41:24.456868 [ 4 ] {} <Debug> TCP-Session: 255c3e70-b7cc-4508-8a56-cbc93c3d08d8 Authenticating user 'default' from [::1]:46131
2025.05.14 01:41:24.456874 [ 5 ] {} <Debug> TCP-Session: 557d40c5-e419-49c7-a8d5-01263aa8bdea Authenticated with global context as user 94309d50-4f52-5250-31bd-74fecac179db
2025.05.14 01:41:24.456907 [ 5 ] {} <Debug> TCP-Session: 557d40c5-e419-49c7-a8d5-01263aa8bdea Creating session context with user_id: 94309d50-4f52-5250-31bd-74fecac179db
2025.05.14 01:41:24.456959 [ 4 ] {} <Debug> TCP-Session: 255c3e70-b7cc-4508-8a56-cbc93c3d08d8 Authenticated with global context as user 94309d50-4f52-5250-31bd-74fecac179db
2025.05.14 01:41:24.456978 [ 4 ] {} <Debug> TCP-Session: 255c3e70-b7cc-4508-8a56-cbc93c3d08d8 Creating session context with user_id: 94309d50-4f52-5250-31bd-74fecac179db
2025.05.14 01:41:24.457095 [ 267 ] {} <Trace> TCPHandlerFactory: TCP Request. Address: [::1]:33790
2025.05.14 01:41:24.457545 [ 3 ] {} <Trace> TCP-Session: e9af9fc0-ee11-4d98-b1ae-215d2a9f6564 Creating query context from session context, user_id: 94309d50-4f52-5250-31bd-74fecac179db, parent context user: default
2025.05.14 01:41:24.457632 [ 267 ] {} <Debug> TCPHandler: Connected oximeter version 23.8.0, revision: 54465, database: default, user: default.
2025.05.14 01:41:24.457675 [ 268 ] {} <Trace> TCPHandlerFactory: TCP Request. Address: [::1]:48118
2025.05.14 01:41:24.457766 [ 266 ] {} <Trace> TCPHandlerFactory: TCP Request. Address: [::1]:41079
2025.05.14 01:41:24.457725 [ 267 ] {} <Debug> TCP-Session: a5e950da-365d-49a9-8e6d-2d47968a7a8b Authenticating user 'default' from [::1]:33790
2025.05.14 01:41:24.457902 [ 267 ] {} <Debug> TCP-Session: a5e950da-365d-49a9-8e6d-2d47968a7a8b Authenticated with global context as user 94309d50-4f52-5250-31bd-74fecac179db
2025.05.14 01:41:24.457918 [ 267 ] {} <Debug> TCP-Session: a5e950da-365d-49a9-8e6d-2d47968a7a8b Creating session context with user_id: 94309d50-4f52-5250-31bd-74fecac179db
2025.05.14 01:41:24.457682 [ 263 ] {} <Trace> TCPHandlerFactory: TCP Request. Address: [::1]:58062
2025.05.14 01:41:24.457992 [ 265 ] {} <Trace> TCPHandlerFactory: TCP Request. Address: [::1]:50686
2025.05.14 01:41:24.458092 [ 264 ] {} <Trace> TCPHandlerFactory: TCP Request. Address: [::1]:37237
2025.05.14 01:41:24.458589 [ 268 ] {} <Debug> TCPHandler: Connected oximeter version 23.8.0, revision: 54465, database: default, user: default.
2025.05.14 01:41:24.458689 [ 268 ] {} <Debug> TCP-Session: e4169bd1-742f-4d26-baaa-1ce2ee2f7990 Authenticating user 'default' from [::1]:48118
2025.05.14 01:41:24.458719 [ 268 ] {} <Debug> TCP-Session: e4169bd1-742f-4d26-baaa-1ce2ee2f7990 Authenticated with global context as user 94309d50-4f52-5250-31bd-74fecac179db
2025.05.14 01:41:24.458731 [ 268 ] {} <Debug> TCP-Session: e4169bd1-742f-4d26-baaa-1ce2ee2f7990 Creating session context with user_id: 94309d50-4f52-5250-31bd-74fecac179db
2025.05.14 01:41:24.458861 [ 266 ] {} <Debug> TCPHandler: Connected oximeter version 23.8.0, revision: 54465, database: default, user: default.
2025.05.14 01:41:24.458871 [ 3 ] {d4abc497-4df0-48a5-b355-fe4fba537595} <Debug> executeQuery: (from [::1]:55581) SELECT MAX(value) AS max_version FROM oximeter.version; (stage: Complete)
2025.05.14 01:41:24.458921 [ 266 ] {} <Debug> TCP-Session: dfc35a2c-059c-4569-a053-17c409e210ad Authenticating user 'default' from [::1]:41079
2025.05.14 01:41:24.458952 [ 266 ] {} <Debug> TCP-Session: dfc35a2c-059c-4569-a053-17c409e210ad Authenticated with global context as user 94309d50-4f52-5250-31bd-74fecac179db
2025.05.14 01:41:24.458985 [ 266 ] {} <Debug> TCP-Session: dfc35a2c-059c-4569-a053-17c409e210ad Creating session context with user_id: 94309d50-4f52-5250-31bd-74fecac179db
2025.05.14 01:41:24.459420 [ 265 ] {} <Debug> TCPHandler: Connected oximeter version 23.8.0, revision: 54465, database: default, user: default.
2025.05.14 01:41:24.459494 [ 265 ] {} <Debug> TCP-Session: d42900fb-75ec-4084-9f1f-3d35b61d89b0 Authenticating user 'default' from [::1]:50686
2025.05.14 01:41:24.459536 [ 265 ] {} <Debug> TCP-Session: d42900fb-75ec-4084-9f1f-3d35b61d89b0 Authenticated with global context as user 94309d50-4f52-5250-31bd-74fecac179db
2025.05.14 01:41:24.459548 [ 265 ] {} <Debug> TCP-Session: d42900fb-75ec-4084-9f1f-3d35b61d89b0 Creating session context with user_id: 94309d50-4f52-5250-31bd-74fecac179db
2025.05.14 01:41:24.459659 [ 264 ] {} <Debug> TCPHandler: Connected oximeter version 23.8.0, revision: 54465, database: default, user: default.
2025.05.14 01:41:24.459744 [ 264 ] {} <Debug> TCP-Session: a87fb864-862b-4cda-9ecb-a672bd01e095 Authenticating user 'default' from [::1]:37237
2025.05.14 01:41:24.459892 [ 264 ] {} <Debug> TCP-Session: a87fb864-862b-4cda-9ecb-a672bd01e095 Authenticated with global context as user 94309d50-4f52-5250-31bd-74fecac179db
2025.05.14 01:41:24.459924 [ 264 ] {} <Debug> TCP-Session: a87fb864-862b-4cda-9ecb-a672bd01e095 Creating session context with user_id: 94309d50-4f52-5250-31bd-74fecac179db
2025.05.14 01:41:24.460182 [ 263 ] {} <Debug> TCPHandler: Connected oximeter version 23.8.0, revision: 54465, database: default, user: default.
2025.05.14 01:41:24.460245 [ 263 ] {} <Debug> TCP-Session: a87f11b1-2c34-40e9-ac16-4258a164578e Authenticating user 'default' from [::1]:58062
2025.05.14 01:41:24.460269 [ 263 ] {} <Debug> TCP-Session: a87f11b1-2c34-40e9-ac16-4258a164578e Authenticated with global context as user 94309d50-4f52-5250-31bd-74fecac179db
2025.05.14 01:41:24.460284 [ 263 ] {} <Debug> TCP-Session: a87f11b1-2c34-40e9-ac16-4258a164578e Creating session context with user_id: 94309d50-4f52-5250-31bd-74fecac179db
2025.05.14 01:41:24.469509 [ 3 ] {d4abc497-4df0-48a5-b355-fe4fba537595} <Error> executeQuery: Code: 81. DB::Exception: Database oximeter does not exist. (UNKNOWN_DATABASE) (version 23.8.7.1) (from [::1]:55581) (in query: SELECT MAX(value) AS max_version FROM oximeter.version;), Stack trace (when copying this message, always include the lines below):

0. StackTrace::tryCapture() @ 0x000000000f9acdb7 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
1. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000f97f7df in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
2. DB::Exception::Exception<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >(int, FormatStringHelperImpl<std::type_identity<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >::type>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&&) @ 0x000000000a3c7817 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
3. DB::DatabaseCatalog::getDatabase(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const @ 0x00000000147ad091 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
4. DB::Context::resolveStorageID(DB::StorageID, DB::Context::StorageNamespace) const @ 0x000000001476347d in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
5. DB::JoinedTables::getLeftTableStorage() @ 0x00000000148ec08b in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
6. DB::InterpreterSelectQuery::InterpreterSelectQuery(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context> const&, std::optional<DB::Pipe>, std::shared_ptr<DB::IStorage> const&, DB::SelectQueryOptions const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::PreparedSets>) @ 0x0000000014866701 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
7. DB::InterpreterSelectQuery::InterpreterSelectQuery(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context> const&, DB::SelectQueryOptions const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) @ 0x0000000014864f66 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
8. DB::InterpreterSelectWithUnionQuery::buildCurrentChildInterpreter(std::shared_ptr<DB::IAST> const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) @ 0x00000000148b3d46 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
9. DB::InterpreterSelectWithUnionQuery::InterpreterSelectWithUnionQuery(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context>, DB::SelectQueryOptions const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) @ 0x00000000148b2a83 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
10. std::__detail::_MakeUniq<DB::InterpreterSelectWithUnionQuery>::__single_object std::make_unique<DB::InterpreterSelectWithUnionQuery, std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context>&, DB::SelectQueryOptions const&>(std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context>&, DB::SelectQueryOptions const&) @ 0x000000001483d407 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
11. DB::InterpreterFactory::get(std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context>, DB::SelectQueryOptions const&) @ 0x000000001483c8d5 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
12. DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x0000000014a5f216 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
13. DB::executeQuery(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum) @ 0x0000000014a5bd46 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
14. DB::TCPHandler::runImpl() @ 0x00000000153fbaaa in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
15. DB::TCPHandler::run() @ 0x000000001540bc5c in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
16. Poco::Net::TCPServerConnection::start() @ 0x00000000187a058c in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
17. Poco::Net::TCPServerDispatcher::run() @ 0x00000000187a0a61 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
18. Poco::PooledThread::run() @ 0x0000000018805972 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
19. Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000018803425 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
20. _thrp_setup @ 0x0000000000116767 in /lib/amd64/libc.so.1

2025.05.14 01:41:24.469951 [ 3 ] {d4abc497-4df0-48a5-b355-fe4fba537595} <Error> TCPHandler: Code: 81. DB::Exception: Database oximeter does not exist. (UNKNOWN_DATABASE), Stack trace (when copying this message, always include the lines below):

0. StackTrace::tryCapture() @ 0x000000000f9acdb7 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
1. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000f97f7df in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
2. DB::Exception::Exception<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >(int, FormatStringHelperImpl<std::type_identity<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >::type>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&&) @ 0x000000000a3c7817 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
3. DB::DatabaseCatalog::getDatabase(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const @ 0x00000000147ad091 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
4. DB::Context::resolveStorageID(DB::StorageID, DB::Context::StorageNamespace) const @ 0x000000001476347d in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
5. DB::JoinedTables::getLeftTableStorage() @ 0x00000000148ec08b in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
6. DB::InterpreterSelectQuery::InterpreterSelectQuery(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context> const&, std::optional<DB::Pipe>, std::shared_ptr<DB::IStorage> const&, DB::SelectQueryOptions const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::PreparedSets>) @ 0x0000000014866701 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
7. DB::InterpreterSelectQuery::InterpreterSelectQuery(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context> const&, DB::SelectQueryOptions const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) @ 0x0000000014864f66 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
8. DB::InterpreterSelectWithUnionQuery::buildCurrentChildInterpreter(std::shared_ptr<DB::IAST> const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) @ 0x00000000148b3d46 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
9. DB::InterpreterSelectWithUnionQuery::InterpreterSelectWithUnionQuery(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context>, DB::SelectQueryOptions const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) @ 0x00000000148b2a83 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
10. std::__detail::_MakeUniq<DB::InterpreterSelectWithUnionQuery>::__single_object std::make_unique<DB::InterpreterSelectWithUnionQuery, std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context>&, DB::SelectQueryOptions const&>(std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context>&, DB::SelectQueryOptions const&) @ 0x000000001483d407 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
11. DB::InterpreterFactory::get(std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context>, DB::SelectQueryOptions const&) @ 0x000000001483c8d5 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
12. DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x0000000014a5f216 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
13. DB::executeQuery(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum) @ 0x0000000014a5bd46 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
14. DB::TCPHandler::runImpl() @ 0x00000000153fbaaa in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
15. DB::TCPHandler::run() @ 0x000000001540bc5c in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
16. Poco::Net::TCPServerConnection::start() @ 0x00000000187a058c in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
17. Poco::Net::TCPServerDispatcher::run() @ 0x00000000187a0a61 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
18. Poco::PooledThread::run() @ 0x0000000018805972 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
19. Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000018803425 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
20. _thrp_setup @ 0x0000000000116767 in /lib/amd64/libc.so.1

2025.05.14 01:41:24.470204 [ 3 ] {d4abc497-4df0-48a5-b355-fe4fba537595} <Debug> TCPHandler: Processed in 0.013679404 sec.
2025.05.14 01:41:24.482518 [ 4 ] {} <Trace> TCP-Session: 255c3e70-b7cc-4508-8a56-cbc93c3d08d8 Creating query context from session context, user_id: 94309d50-4f52-5250-31bd-74fecac179db, parent context user: default
2025.05.14 01:41:24.482966 [ 4 ] {dd8889ed-26fb-46d1-949d-688d6752c1b6} <Debug> executeQuery: (from [::1]:46131) /* We purposefully split our DB schema into two *disjoint* files:
 * `db-init-1.sql` and `db-init-2.sql`. The purpose of this split is to shorten
 * the duration of our replicated tests. These tests only use a subset of the
 * tables defined in the full schema. We put the tables used by the replicated
 * tests in `db-init-1.sql`, and the remainder of the tables in `db-init-2.sql`.
 * This minimizes test time by reducing the cost to load a schema. In
 * production, we load `db-init-1.sql` followed by `db-init-2.sql` so we have
 * the full schema. If we end up needing to use more tables in replicated tests
 * we can go  ahead and move them into `db-init-1.sql`, removing them from
 * `db-init-2.sql`. Conversely, if we stop using given tables in our tests we
 * can move them from `db-init-1.sql` into `db-init-2.sql` and keep our test
 * times minimal.

 * The reason to keep the two files disjoint is so that we don't have to
 * maintain consistency between table definitions. All tables are defined
 * once. However, in order to write a test that ensures the tables are in fact
 * disjoint, we must create the `oximeter` database in both files so we can load
 * them in isolation.
 */
 CREATE DATABASE IF NOT EXISTS oximeter ON CLUSTER oximeter_cluster (stage: Complete)
2025.05.14 01:41:24.483312 [ 4 ] {dd8889ed-26fb-46d1-949d-688d6752c1b6} <Trace> ContextAccess (default): Access granted: CREATE DATABASE ON oximeter.*
2025.05.14 01:41:24.491611 [ 4 ] {dd8889ed-26fb-46d1-949d-688d6752c1b6} <Error> executeQuery: Code: 999. Coordination::Exception: All connection tries failed while connecting to ZooKeeper. nodes: [::1]:9181, [::1]:9183, [::1]:9182
Poco::Exception. Code: 1000, e.code() = 146, Connection refused: [::1]:9181 (version 23.8.7.1), [::1]:9181
Poco::Exception. Code: 1000, e.code() = 146, Connection refused: [::1]:9183 (version 23.8.7.1), [::1]:9183
Poco::Exception. Code: 1000, e.code() = 146, Connection refused: [::1]:9182 (version 23.8.7.1), [::1]:9182
Poco::Exception. Code: 1000, e.code() = 146, Connection refused: [::1]:9181 (version 23.8.7.1), [::1]:9181
Poco::Exception. Code: 1000, e.code() = 146, Connection refused: [::1]:9183 (version 23.8.7.1), [::1]:9183
Poco::Exception. Code: 1000, e.code() = 146, Connection refused: [::1]:9182 (version 23.8.7.1), [::1]:9182
Poco::Exception. Code: 1000, e.code() = 146, Connection refused: [::1]:9181 (version 23.8.7.1), [::1]:9181
Poco::Exception. Code: 1000, e.code() = 146, Connection refused: [::1]:9183 (version 23.8.7.1), [::1]:9183
Poco::Exception. Code: 1000, e.code() = 146, Connection refused: [::1]:9182 (version 23.8.7.1), [::1]:9182
. (KEEPER_EXCEPTION) (version 23.8.7.1) (from [::1]:46131) (in query: /* We purposefully split our DB schema into two *disjoint* files:
 * `db-init-1.sql` and `db-init-2.sql`. The purpose of this split is to shorten
 * the duration of our replicated tests. These tests only use a subset of the
 * tables defined in the full schema. We put the tables used by the replicated
 * tests in `db-init-1.sql`, and the remainder of the tables in `db-init-2.sql`.
 * This minimizes test time by reducing the cost to load a schema. In
 * production, we load `db-init-1.sql` followed by `db-init-2.sql` so we have
 * the full schema. If we end up needing to use more tables in replicated tests
 * we can go  ahead and move them into `db-init-1.sql`, removing them from
 * `db-init-2.sql`. Conversely, if we stop using given tables in our tests we
 * can move them from `db-init-1.sql` into `db-init-2.sql` and keep our test
 * times minimal.

 * The reason to keep the two files disjoint is so that we don't have to
 * maintain consistency between table definitions. All tables are defined
 * once. However, in order to write a test that ensures the tables are in fact
 * disjoint, we must create the `oximeter` database in both files so we can load
 * them in isolation.
 */
 CREATE DATABASE IF NOT EXISTS oximeter ON CLUSTER oximeter_cluster), Stack trace (when copying this message, always include the lines below):

0. StackTrace::tryCapture() @ 0x000000000f9acdb7 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
1. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000f97f7df in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
2. DB::Exception::Exception<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&>(int, FormatStringHelperImpl<std::type_identity<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&>::type>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&) @ 0x000000000a7857f7 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
3. Coordination::Exception::Exception<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&>::type>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&) @ 0x0000000014e3842d in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
4. Coordination::ZooKeeper::connect(std::vector<Coordination::ZooKeeper::Node, std::allocator<Coordination::ZooKeeper::Node> > const&, Poco::Timespan) @ 0x000000001648290a in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
5. Coordination::ZooKeeper::ZooKeeper(std::vector<Coordination::ZooKeeper::Node, std::allocator<Coordination::ZooKeeper::Node> > const&, zkutil::ZooKeeperArgs const&, std::shared_ptr<DB::ZooKeeperLog>, std::optional<std::function<void (unsigned long, Coordination::ZooKeeper::Node const&)> >&&) @ 0x000000001648050b in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
6. zkutil::ZooKeeper::init(zkutil::ZooKeeperArgs) @ 0x0000000016449829 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
7. zkutil::ZooKeeper::ZooKeeper(Poco::Util::AbstractConfiguration const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<DB::ZooKeeperLog>) @ 0x000000001644b0a3 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
8. DB::Context::getZooKeeper() const @ 0x0000000014755b49 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
9. DB::DDLWorker::getAndSetZooKeeper() @ 0x0000000014792b4a in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
10. DB::DDLWorker::enqueueQuery[abi:cxx11](DB::DDLLogEntry&) @ 0x00000000147a099c in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
11. DB::executeDDLQueryOnCluster(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context const>, DB::DDLQueryOnClusterParams const&) @ 0x0000000014a4b2e7 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
12. DB::InterpreterCreateQuery::executeQueryOnCluster(DB::ASTCreateQuery&) @ 0x00000000148189e1 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
13. DB::InterpreterCreateQuery::execute() @ 0x0000000014820819 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
14. DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x0000000014a5f949 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
15. DB::executeQuery(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum) @ 0x0000000014a5bd46 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
16. DB::TCPHandler::runImpl() @ 0x00000000153fbaaa in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
17. DB::TCPHandler::run() @ 0x000000001540bc5c in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
18. Poco::Net::TCPServerConnection::start() @ 0x00000000187a058c in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
19. Poco::Net::TCPServerDispatcher::run() @ 0x00000000187a0a61 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
20. Poco::PooledThread::run() @ 0x0000000018805972 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
21. Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000018803425 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
22. _thrp_setup @ 0x0000000000116767 in /lib/amd64/libc.so.1

2025.05.14 01:41:24.491909 [ 4 ] {dd8889ed-26fb-46d1-949d-688d6752c1b6} <Error> TCPHandler: Code: 999. Coordination::Exception: All connection tries failed while connecting to ZooKeeper. nodes: [::1]:9181, [::1]:9183, [::1]:9182
Poco::Exception. Code: 1000, e.code() = 146, Connection refused: [::1]:9181 (version 23.8.7.1), [::1]:9181
Poco::Exception. Code: 1000, e.code() = 146, Connection refused: [::1]:9183 (version 23.8.7.1), [::1]:9183
Poco::Exception. Code: 1000, e.code() = 146, Connection refused: [::1]:9182 (version 23.8.7.1), [::1]:9182
Poco::Exception. Code: 1000, e.code() = 146, Connection refused: [::1]:9181 (version 23.8.7.1), [::1]:9181
Poco::Exception. Code: 1000, e.code() = 146, Connection refused: [::1]:9183 (version 23.8.7.1), [::1]:9183
Poco::Exception. Code: 1000, e.code() = 146, Connection refused: [::1]:9182 (version 23.8.7.1), [::1]:9182
Poco::Exception. Code: 1000, e.code() = 146, Connection refused: [::1]:9181 (version 23.8.7.1), [::1]:9181
Poco::Exception. Code: 1000, e.code() = 146, Connection refused: [::1]:9183 (version 23.8.7.1), [::1]:9183
Poco::Exception. Code: 1000, e.code() = 146, Connection refused: [::1]:9182 (version 23.8.7.1), [::1]:9182
. (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. StackTrace::tryCapture() @ 0x000000000f9acdb7 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
1. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000f97f7df in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
2. DB::Exception::Exception<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&>(int, FormatStringHelperImpl<std::type_identity<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&>::type>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&) @ 0x000000000a7857f7 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
3. Coordination::Exception::Exception<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&>::type>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&) @ 0x0000000014e3842d in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
4. Coordination::ZooKeeper::connect(std::vector<Coordination::ZooKeeper::Node, std::allocator<Coordination::ZooKeeper::Node> > const&, Poco::Timespan) @ 0x000000001648290a in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
5. Coordination::ZooKeeper::ZooKeeper(std::vector<Coordination::ZooKeeper::Node, std::allocator<Coordination::ZooKeeper::Node> > const&, zkutil::ZooKeeperArgs const&, std::shared_ptr<DB::ZooKeeperLog>, std::optional<std::function<void (unsigned long, Coordination::ZooKeeper::Node const&)> >&&) @ 0x000000001648050b in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
6. zkutil::ZooKeeper::init(zkutil::ZooKeeperArgs) @ 0x0000000016449829 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
7. zkutil::ZooKeeper::ZooKeeper(Poco::Util::AbstractConfiguration const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<DB::ZooKeeperLog>) @ 0x000000001644b0a3 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
8. DB::Context::getZooKeeper() const @ 0x0000000014755b49 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
9. DB::DDLWorker::getAndSetZooKeeper() @ 0x0000000014792b4a in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
10. DB::DDLWorker::enqueueQuery[abi:cxx11](DB::DDLLogEntry&) @ 0x00000000147a099c in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
11. DB::executeDDLQueryOnCluster(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context const>, DB::DDLQueryOnClusterParams const&) @ 0x0000000014a4b2e7 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
12. DB::InterpreterCreateQuery::executeQueryOnCluster(DB::ASTCreateQuery&) @ 0x00000000148189e1 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
13. DB::InterpreterCreateQuery::execute() @ 0x0000000014820819 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
14. DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x0000000014a5f949 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
15. DB::executeQuery(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum) @ 0x0000000014a5bd46 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
16. DB::TCPHandler::runImpl() @ 0x00000000153fbaaa in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
17. DB::TCPHandler::run() @ 0x000000001540bc5c in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
18. Poco::Net::TCPServerConnection::start() @ 0x00000000187a058c in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
19. Poco::Net::TCPServerDispatcher::run() @ 0x00000000187a0a61 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
20. Poco::PooledThread::run() @ 0x0000000018805972 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
21. Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000018803425 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
22. _thrp_setup @ 0x0000000000116767 in /lib/amd64/libc.so.1

2025.05.14 01:41:24.492048 [ 4 ] {dd8889ed-26fb-46d1-949d-688d6752c1b6} <Debug> TCPHandler: Processed in 0.009853875 sec.