| 1 | 2025.05.14 01:41:23.425146 [ 1 ] {} <Information> Application: Starting ClickHouse 23.8.7.1 (revision: 54477, git hash: 3042d295d963012962f5c683bd2776fa331a38c3, build id: <unknown>), PID 16484 | ||
| 2 | 2025.05.14 01:41:23.425479 [ 1 ] {} <Information> Application: starting up | ||
| 3 | 2025.05.14 01:41:23.425542 [ 1 ] {} <Information> Application: OS name: SunOS, version: 5.11, architecture: i86pc | ||
| 4 | 2025.05.14 01:41:23.439115 [ 1 ] {} <Information> Application: Available RAM: 31.99 GiB; physical cores: 8; logical cores: 8. | ||
| 5 | 2025.05.14 01:41:23.454465 [ 1 ] {} <Debug> Application: rlimit on number of file descriptors is 65536 | ||
| 6 | 2025.05.14 01:41:23.454568 [ 1 ] {} <Debug> Application: Initializing DateLUT. | ||
| 7 | 2025.05.14 01:41:23.454680 [ 1 ] {} <Trace> Application: Initialized DateLUT with time zone 'UTC'. | ||
| 8 | 2025.05.14 01:41:23.454851 [ 1 ] {} <Debug> Context: Setting up /var/tmp/omicron_tmp/oximeter_db-11d8f51cb75f441f-test_ensure_schema_walks_through_multiple_steps_replicated.16475.4-clickhouse-fgDTKD/tmp/ to store temporary data in it | ||
| 9 | 2025.05.14 01:41:23.456042 [ 1 ] {} <Debug> Application: Initializing interserver credentials. | ||
| 10 | 2025.05.14 01:41:23.457041 [ 1 ] {} <Information> SensitiveDataMaskerConfigRead: 1 query masking rules loaded. | ||
| 11 | 2025.05.14 01:41:23.458098 [ 1 ] {} <Debug> ConfigReloader: Loading config '/work/oxidecomputer/omicron/oximeter/db/src/configs/replica_config.xml' | ||
| 12 | 2025.05.14 01:41:23.462592 [ 1 ] {} <Debug> ConfigReloader: Loaded config '/work/oxidecomputer/omicron/oximeter/db/src/configs/replica_config.xml', performing update on configuration | ||
| 13 | 2025.05.14 01:41:23.465421 [ 1 ] {} <Information> Application: Setting max_server_memory_usage was set to 28.79 GiB (31.99 GiB available * 0.90 max_server_memory_usage_to_ram_ratio) | ||
| 14 | 2025.05.14 01:41:23.465515 [ 1 ] {} <Information> Application: Setting merges_mutations_memory_usage_soft_limit was set to 16.00 GiB (31.99 GiB available * 0.50 merges_mutations_memory_usage_to_ram_ratio) | ||
| 15 | 2025.05.14 01:41:23.465542 [ 1 ] {} <Information> Application: Merges and mutations memory limit is set to 16.00 GiB | ||
| 16 | 2025.05.14 01:41:23.467223 [ 1 ] {} <Information> BackgroundSchedulePool/BgBufSchPool: Create BackgroundSchedulePool with 16 threads | ||
| 17 | 2025.05.14 01:41:23.472625 [ 1 ] {} <Information> BackgroundSchedulePool/BgSchPool: Create BackgroundSchedulePool with 128 threads | ||
| 18 | 2025.05.14 01:41:23.492747 [ 1 ] {} <Information> BackgroundSchedulePool/BgMBSchPool: Create BackgroundSchedulePool with 16 threads | ||
| 19 | 2025.05.14 01:41:23.493785 [ 1 ] {} <Information> BackgroundSchedulePool/BgDistSchPool: Create BackgroundSchedulePool with 16 threads | ||
| 20 | 2025.05.14 01:41:23.515844 [ 1 ] {} <Information> CertificateReloader: One of paths is empty. Cannot apply new configuration for certificates. Fill all paths and try again. | ||
| 21 | 2025.05.14 01:41:23.515951 [ 1 ] {} <Debug> ConfigReloader: Loaded config '/work/oxidecomputer/omicron/oximeter/db/src/configs/replica_config.xml', performed update on configuration | ||
| 22 | 2025.05.14 01:41:23.517145 [ 1 ] {} <Information> Application: Listening for replica communication (interserver): http://[::]:9009 | ||
| 23 | 2025.05.14 01:41:23.518365 [ 1 ] {} <Debug> ConfigReloader: Loading config '/work/oxidecomputer/omicron/oximeter/db/src/configs/replica_config.xml' | ||
| 24 | 2025.05.14 01:41:23.522129 [ 1 ] {} <Debug> ConfigReloader: Loaded config '/work/oxidecomputer/omicron/oximeter/db/src/configs/replica_config.xml', performing update on configuration | ||
| 25 | 2025.05.14 01:41:23.525370 [ 1 ] {} <Debug> ConfigReloader: Loaded config '/work/oxidecomputer/omicron/oximeter/db/src/configs/replica_config.xml', performed update on configuration | ||
| 26 | 2025.05.14 01:41:23.526326 [ 1 ] {} <Debug> Access(user directories): Added users_xml access storage 'users_xml', path: /work/oxidecomputer/omicron/oximeter/db/src/configs/replica_config.xml | ||
| 27 | 2025.05.14 01:41:23.527885 [ 1 ] {} <Information> Context: Initialized background executor for merges and mutations with num_threads=16, num_tasks=32, scheduling_policy=round_robin | ||
| 28 | 2025.05.14 01:41:23.528391 [ 1 ] {} <Information> Context: Initialized background executor for move operations with num_threads=8, num_tasks=8 | ||
| 29 | 2025.05.14 01:41:23.528847 [ 1 ] {} <Information> Context: Initialized background executor for fetches with num_threads=8, num_tasks=8 | ||
| 30 | 2025.05.14 01:41:23.529335 [ 1 ] {} <Information> Context: Initialized background executor for common operations (e.g. clearing old parts) with num_threads=8, num_tasks=8 | ||
| 31 | 2025.05.14 01:41:23.530825 [ 1 ] {} <Information> DNSCacheUpdater: Update period 15 seconds | ||
| 32 | 2025.05.14 01:41:23.530861 [ 1 ] {} <Information> Application: Loading metadata from /var/tmp/omicron_tmp/oximeter_db-11d8f51cb75f441f-test_ensure_schema_walks_through_multiple_steps_replicated.16475.4-clickhouse-fgDTKD/datastore/ | ||
| 33 | 2025.05.14 01:41:23.533341 [ 1 ] {} <Information> DatabaseAtomic (system): Metadata processed, database system has 0 tables and 0 dictionaries in total. | ||
| 34 | 2025.05.14 01:41:23.533372 [ 1 ] {} <Information> TablesLoader: Parsed metadata of 0 tables in 1 databases in 0.000239132 sec | ||
| 35 | 2025.05.14 01:41:23.533385 [ 1 ] {} <Trace> ReferentialDeps: No tables | ||
| 36 | 2025.05.14 01:41:23.533393 [ 1 ] {} <Trace> LoadingDeps: No tables | ||
| 37 | 2025.05.14 01:41:23.533557 [ 39 ] {} <Debug> DNSResolver: Updating DNS cache | ||
| 38 | 2025.05.14 01:41:23.533621 [ 1 ] {} <Information> DatabaseAtomic (system): Starting up tables. | ||
| 39 | 2025.05.14 01:41:23.533682 [ 39 ] {} <Debug> DNSResolver: Updated DNS cache | ||
| 40 | 2025.05.14 01:41:23.533708 [ 1 ] {} <Debug> SystemLog: Creating system.query_log from query_log | ||
| 41 | 2025.05.14 01:41:23.536636 [ 1 ] {} <Debug> SystemLog: Creating system.query_thread_log from query_thread_log | ||
| 42 | 2025.05.14 01:41:23.537147 [ 1 ] {} <Debug> SystemLog: Creating system.part_log from part_log | ||
| 43 | 2025.05.14 01:41:23.537848 [ 1 ] {} <Debug> SystemLog: Creating system.trace_log from trace_log | ||
| 44 | 2025.05.14 01:41:23.538438 [ 1 ] {} <Debug> SystemLog: Creating system.crash_log from crash_log | ||
| 45 | 2025.05.14 01:41:23.538641 [ 1 ] {} <Debug> SystemLog: Not creating system.text_log since corresponding section 'text_log' is missing from config | ||
| 46 | 2025.05.14 01:41:23.538810 [ 1 ] {} <Debug> SystemLog: Creating system.metric_log from metric_log | ||
| 47 | 2025.05.14 01:41:23.543862 [ 1 ] {} <Debug> SystemLog: Not creating system.filesystem_cache_log since corresponding section 'filesystem_cache_log' is missing from config | ||
| 48 | 2025.05.14 01:41:23.543961 [ 1 ] {} <Debug> SystemLog: Not creating system.filesystem_read_prefetches_log since corresponding section 'filesystem_read_prefetches_log' is missing from config | ||
| 49 | 2025.05.14 01:41:23.543978 [ 1 ] {} <Debug> SystemLog: Creating system.asynchronous_metric_log from asynchronous_metric_log | ||
| 50 | 2025.05.14 01:41:23.544336 [ 1 ] {} <Debug> SystemLog: Not creating system.opentelemetry_span_log since corresponding section 'opentelemetry_span_log' is missing from config | ||
| 51 | 2025.05.14 01:41:23.544374 [ 1 ] {} <Debug> SystemLog: Creating system.query_views_log from query_views_log | ||
| 52 | 2025.05.14 01:41:23.544771 [ 1 ] {} <Debug> SystemLog: Not creating system.zookeeper_log since corresponding section 'zookeeper_log' is missing from config | ||
| 53 | 2025.05.14 01:41:23.544798 [ 1 ] {} <Debug> SystemLog: Not creating system.session_log since corresponding section 'session_log' is missing from config | ||
| 54 | 2025.05.14 01:41:23.544820 [ 1 ] {} <Debug> SystemLog: Not creating system.transactions_info_log since corresponding section 'transactions_info_log' is missing from config | ||
| 55 | 2025.05.14 01:41:23.544859 [ 1 ] {} <Debug> SystemLog: Creating system.processors_profile_log from processors_profile_log | ||
| 56 | 2025.05.14 01:41:23.545140 [ 1 ] {} <Debug> SystemLog: Not creating system.asynchronous_insert_log since corresponding section 'asynchronous_insert_log' is missing from config | ||
| 57 | 2025.05.14 01:41:23.594269 [ 1 ] {} <Information> DatabaseCatalog: Found 0 partially dropped tables. Will load them and retry removal. | ||
| 58 | 2025.05.14 01:41:23.595202 [ 1 ] {} <Information> DatabaseAtomic (default): Metadata processed, database default has 0 tables and 0 dictionaries in total. | ||
| 59 | 2025.05.14 01:41:23.595236 [ 1 ] {} <Information> TablesLoader: Parsed metadata of 0 tables in 1 databases in 9.5444e-05 sec | ||
| 60 | 2025.05.14 01:41:23.595248 [ 1 ] {} <Trace> ReferentialDeps: No tables | ||
| 61 | 2025.05.14 01:41:23.595256 [ 1 ] {} <Trace> LoadingDeps: No tables | ||
| 62 | 2025.05.14 01:41:23.595340 [ 1 ] {} <Information> DatabaseAtomic (default): Starting up tables. | ||
| 63 | 2025.05.14 01:41:23.595549 [ 1 ] {} <Information> UserDefinedSQLObjectsLoaderFromDisk: Loading user defined objects from /var/tmp/omicron_tmp/oximeter_db-11d8f51cb75f441f-test_ensure_schema_walks_through_multiple_steps_replicated.16475.4-clickhouse-fgDTKD/datastore/user_defined/ | ||
| 64 | 2025.05.14 01:41:23.595610 [ 1 ] {} <Debug> UserDefinedSQLObjectsLoaderFromDisk: User defined objects loaded | ||
| 65 | 2025.05.14 01:41:23.595814 [ 1 ] {} <Debug> Application: Loaded metadata. | ||
| 66 | 2025.05.14 01:41:23.595847 [ 1 ] {} <Information> Application: Query Profiler and TraceCollector are disabled because they require PHDR cache to be created (otherwise the function 'dl_iterate_phdr' is not lock free and not async-signal safe). | ||
| 67 | 2025.05.14 01:41:23.595858 [ 1 ] {} <Information> Application: TaskStats is not implemented for this OS. IO accounting will be disabled. | ||
| 68 | 2025.05.14 01:41:23.596183 [ 1 ] {} <Information> CertificateReloader: One of paths is empty. Cannot apply new configuration for certificates. Fill all paths and try again. | ||
| 69 | 2025.05.14 01:41:23.596469 [ 1 ] {} <Trace> AsynchronousMetrics: MemoryTracking: was 40.98 MiB, peak 40.98 MiB, free memory in arenas 0.00 B, will set to 145.02 MiB (RSS), difference: 104.04 MiB | ||
| 70 | 2025.05.14 01:41:23.596870 [ 1 ] {} <Trace> AsynchronousMetrics: Update heavy metrics. Update period 1 sec. Update heavy metrics period 120 sec. Heavy metrics calculation elapsed: 1.8409e-05 sec. | ||
| 71 | 2025.05.14 01:41:23.597847 [ 1 ] {} <Information> Application: Listening for http://[::]:8123 | ||
| 72 | 2025.05.14 01:41:23.597931 [ 1 ] {} <Information> Application: Listening for native protocol (tcp): [::]:9001 | ||
| 73 | 2025.05.14 01:41:23.597945 [ 1 ] {} <Information> Application: Ready for connections. | ||
| 74 | 2025.05.14 01:41:23.598318 [ 259 ] {} <Debug> DDLWorker: Started DDLWorker cleanup thread | ||
| 75 | 2025.05.14 01:41:23.598618 [ 255 ] {} <Debug> DDLWorker: Starting DDLWorker thread | ||
| 76 | 2025.05.14 01:41:23.598652 [ 255 ] {} <Debug> DDLWorker: Initializing DDLWorker thread | ||
| 77 | 2025.05.14 01:41:23.611521 [ 255 ] {} <Error> virtual bool DB::DDLWorker::initializeMainThread(): Code: 999. Coordination::Exception: All connection tries failed while connecting to ZooKeeper. nodes: [::1]:9182, [::1]:9181, [::1]:9183 | ||
| 78 | Poco::Exception. Code: 1000, e.code() = 146, Connection refused: [::1]:9182 (version 23.8.7.1), [::1]:9182 | ||
| 79 | Poco::Exception. Code: 1000, e.code() = 146, Connection refused: [::1]:9181 (version 23.8.7.1), [::1]:9181 | ||
| 80 | Poco::Exception. Code: 1000, e.code() = 146, Connection refused: [::1]:9183 (version 23.8.7.1), [::1]:9183 | ||
| 81 | Poco::Exception. Code: 1000, e.code() = 146, Connection refused: [::1]:9182 (version 23.8.7.1), [::1]:9182 | ||
| 82 | Poco::Exception. Code: 1000, e.code() = 146, Connection refused: [::1]:9181 (version 23.8.7.1), [::1]:9181 | ||
| 83 | Poco::Exception. Code: 1000, e.code() = 146, Connection refused: [::1]:9183 (version 23.8.7.1), [::1]:9183 | ||
| 84 | Poco::Exception. Code: 1000, e.code() = 146, Connection refused: [::1]:9182 (version 23.8.7.1), [::1]:9182 | ||
| 85 | Poco::Exception. Code: 1000, e.code() = 146, Connection refused: [::1]:9181 (version 23.8.7.1), [::1]:9181 | ||
| 86 | Poco::Exception. Code: 1000, e.code() = 146, Connection refused: [::1]:9183 (version 23.8.7.1), [::1]:9183 | ||
| 87 | . (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below): | ||
| 88 | |||
| 89 | 0. StackTrace::tryCapture() @ 0x000000000f9acdb7 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 90 | 1. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000f97f7df in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 91 | 2. DB::Exception::Exception<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&>(int, FormatStringHelperImpl<std::type_identity<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&>::type>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&) @ 0x000000000a7857f7 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 92 | 3. Coordination::Exception::Exception<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&>::type>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&) @ 0x0000000014e3842d in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 93 | 4. Coordination::ZooKeeper::connect(std::vector<Coordination::ZooKeeper::Node, std::allocator<Coordination::ZooKeeper::Node> > const&, Poco::Timespan) @ 0x000000001648290a in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 94 | 5. Coordination::ZooKeeper::ZooKeeper(std::vector<Coordination::ZooKeeper::Node, std::allocator<Coordination::ZooKeeper::Node> > const&, zkutil::ZooKeeperArgs const&, std::shared_ptr<DB::ZooKeeperLog>, std::optional<std::function<void (unsigned long, Coordination::ZooKeeper::Node const&)> >&&) @ 0x000000001648050b in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 95 | 6. zkutil::ZooKeeper::init(zkutil::ZooKeeperArgs) @ 0x0000000016449829 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 96 | 7. zkutil::ZooKeeper::ZooKeeper(Poco::Util::AbstractConfiguration const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<DB::ZooKeeperLog>) @ 0x000000001644b0a3 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 97 | 8. DB::Context::getZooKeeper() const @ 0x0000000014755b49 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 98 | 9. DB::DDLWorker::getAndSetZooKeeper() @ 0x0000000014792b4a in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 99 | 10. DB::DDLWorker::initializeMainThread() @ 0x00000000147a134c in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 100 | 11. DB::DDLWorker::runMainThread() @ 0x0000000014791391 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 101 | 12. ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<void (DB::DDLWorker::*)(), DB::DDLWorker*>(void (DB::DDLWorker::*&&)(), DB::DDLWorker*&&)::{lambda()#1}::operator()() @ 0x00000000147a2fd5 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 102 | 13. ThreadPoolImpl<std::thread>::worker(std::_List_iterator<std::thread>) @ 0x00000000182ede9c in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 103 | 14. ? @ 0x000000000014d750 in /usr/gcc/13/lib/amd64/libstdc++.so.6.0.32 | ||
| 104 | 15. _thrp_setup @ 0x0000000000116767 in /lib/amd64/libc.so.1 | ||
| 105 | (version 23.8.7.1) | ||
| 106 | 2025.05.14 01:41:24.000556 [ 254 ] {} <Trace> AsynchronousMetrics: MemoryTracking: was 145.05 MiB, peak 145.05 MiB, free memory in arenas 0.00 B, will set to 150.77 MiB (RSS), difference: 5.72 MiB | ||
| 107 | 2025.05.14 01:41:24.452653 [ 3 ] {} <Trace> TCPHandlerFactory: TCP Request. Address: [::1]:55581 | ||
| 108 | 2025.05.14 01:41:24.453868 [ 3 ] {} <Debug> TCPHandler: Connected oximeter version 23.8.0, revision: 54465, database: default, user: default. | ||
| 109 | 2025.05.14 01:41:24.453987 [ 3 ] {} <Debug> TCP-Session: e9af9fc0-ee11-4d98-b1ae-215d2a9f6564 Authenticating user 'default' from [::1]:55581 | ||
| 110 | 2025.05.14 01:41:24.454031 [ 3 ] {} <Debug> TCP-Session: e9af9fc0-ee11-4d98-b1ae-215d2a9f6564 Authenticated with global context as user 94309d50-4f52-5250-31bd-74fecac179db | ||
| 111 | 2025.05.14 01:41:24.454061 [ 3 ] {} <Debug> TCP-Session: e9af9fc0-ee11-4d98-b1ae-215d2a9f6564 Creating session context with user_id: 94309d50-4f52-5250-31bd-74fecac179db | ||
| 112 | 2025.05.14 01:41:24.454419 [ 3 ] {} <Trace> ContextAccess (default): Settings: readonly = 0, allow_ddl = true, allow_introspection_functions = false | ||
| 113 | 2025.05.14 01:41:24.454498 [ 3 ] {} <Trace> ContextAccess (default): List of all grants: GRANT SHOW, SELECT, INSERT, ALTER, CREATE, DROP, UNDROP TABLE, TRUNCATE, OPTIMIZE, BACKUP, KILL QUERY, KILL TRANSACTION, MOVE PARTITION BETWEEN SHARDS, SYSTEM, dictGet, displaySecretsInShowAndSelect, INTROSPECTION, SOURCES, CLUSTER ON *.* | ||
| 114 | 2025.05.14 01:41:24.454528 [ 3 ] {} <Trace> ContextAccess (default): List of all grants including implicit: GRANT SHOW, SELECT, INSERT, ALTER, CREATE, DROP, UNDROP TABLE, TRUNCATE, OPTIMIZE, BACKUP, KILL QUERY, KILL TRANSACTION, MOVE PARTITION BETWEEN SHARDS, SYSTEM, dictGet, displaySecretsInShowAndSelect, INTROSPECTION, SOURCES, CLUSTER ON *.* | ||
| 115 | 2025.05.14 01:41:24.455470 [ 4 ] {} <Trace> TCPHandlerFactory: TCP Request. Address: [::1]:46131 | ||
| 116 | 2025.05.14 01:41:24.455935 [ 5 ] {} <Trace> TCPHandlerFactory: TCP Request. Address: [::1]:54323 | ||
| 117 | 2025.05.14 01:41:24.456696 [ 5 ] {} <Debug> TCPHandler: Connected oximeter version 23.8.0, revision: 54465, database: default, user: default. | ||
| 118 | 2025.05.14 01:41:24.456712 [ 4 ] {} <Debug> TCPHandler: Connected oximeter version 23.8.0, revision: 54465, database: default, user: default. | ||
| 119 | 2025.05.14 01:41:24.456814 [ 5 ] {} <Debug> TCP-Session: 557d40c5-e419-49c7-a8d5-01263aa8bdea Authenticating user 'default' from [::1]:54323 | ||
| 120 | 2025.05.14 01:41:24.456868 [ 4 ] {} <Debug> TCP-Session: 255c3e70-b7cc-4508-8a56-cbc93c3d08d8 Authenticating user 'default' from [::1]:46131 | ||
| 121 | 2025.05.14 01:41:24.456874 [ 5 ] {} <Debug> TCP-Session: 557d40c5-e419-49c7-a8d5-01263aa8bdea Authenticated with global context as user 94309d50-4f52-5250-31bd-74fecac179db | ||
| 122 | 2025.05.14 01:41:24.456907 [ 5 ] {} <Debug> TCP-Session: 557d40c5-e419-49c7-a8d5-01263aa8bdea Creating session context with user_id: 94309d50-4f52-5250-31bd-74fecac179db | ||
| 123 | 2025.05.14 01:41:24.456959 [ 4 ] {} <Debug> TCP-Session: 255c3e70-b7cc-4508-8a56-cbc93c3d08d8 Authenticated with global context as user 94309d50-4f52-5250-31bd-74fecac179db | ||
| 124 | 2025.05.14 01:41:24.456978 [ 4 ] {} <Debug> TCP-Session: 255c3e70-b7cc-4508-8a56-cbc93c3d08d8 Creating session context with user_id: 94309d50-4f52-5250-31bd-74fecac179db | ||
| 125 | 2025.05.14 01:41:24.457095 [ 267 ] {} <Trace> TCPHandlerFactory: TCP Request. Address: [::1]:33790 | ||
| 126 | 2025.05.14 01:41:24.457545 [ 3 ] {} <Trace> TCP-Session: e9af9fc0-ee11-4d98-b1ae-215d2a9f6564 Creating query context from session context, user_id: 94309d50-4f52-5250-31bd-74fecac179db, parent context user: default | ||
| 127 | 2025.05.14 01:41:24.457632 [ 267 ] {} <Debug> TCPHandler: Connected oximeter version 23.8.0, revision: 54465, database: default, user: default. | ||
| 128 | 2025.05.14 01:41:24.457675 [ 268 ] {} <Trace> TCPHandlerFactory: TCP Request. Address: [::1]:48118 | ||
| 129 | 2025.05.14 01:41:24.457766 [ 266 ] {} <Trace> TCPHandlerFactory: TCP Request. Address: [::1]:41079 | ||
| 130 | 2025.05.14 01:41:24.457725 [ 267 ] {} <Debug> TCP-Session: a5e950da-365d-49a9-8e6d-2d47968a7a8b Authenticating user 'default' from [::1]:33790 | ||
| 131 | 2025.05.14 01:41:24.457902 [ 267 ] {} <Debug> TCP-Session: a5e950da-365d-49a9-8e6d-2d47968a7a8b Authenticated with global context as user 94309d50-4f52-5250-31bd-74fecac179db | ||
| 132 | 2025.05.14 01:41:24.457918 [ 267 ] {} <Debug> TCP-Session: a5e950da-365d-49a9-8e6d-2d47968a7a8b Creating session context with user_id: 94309d50-4f52-5250-31bd-74fecac179db | ||
| 133 | 2025.05.14 01:41:24.457682 [ 263 ] {} <Trace> TCPHandlerFactory: TCP Request. Address: [::1]:58062 | ||
| 134 | 2025.05.14 01:41:24.457992 [ 265 ] {} <Trace> TCPHandlerFactory: TCP Request. Address: [::1]:50686 | ||
| 135 | 2025.05.14 01:41:24.458092 [ 264 ] {} <Trace> TCPHandlerFactory: TCP Request. Address: [::1]:37237 | ||
| 136 | 2025.05.14 01:41:24.458589 [ 268 ] {} <Debug> TCPHandler: Connected oximeter version 23.8.0, revision: 54465, database: default, user: default. | ||
| 137 | 2025.05.14 01:41:24.458689 [ 268 ] {} <Debug> TCP-Session: e4169bd1-742f-4d26-baaa-1ce2ee2f7990 Authenticating user 'default' from [::1]:48118 | ||
| 138 | 2025.05.14 01:41:24.458719 [ 268 ] {} <Debug> TCP-Session: e4169bd1-742f-4d26-baaa-1ce2ee2f7990 Authenticated with global context as user 94309d50-4f52-5250-31bd-74fecac179db | ||
| 139 | 2025.05.14 01:41:24.458731 [ 268 ] {} <Debug> TCP-Session: e4169bd1-742f-4d26-baaa-1ce2ee2f7990 Creating session context with user_id: 94309d50-4f52-5250-31bd-74fecac179db | ||
| 140 | 2025.05.14 01:41:24.458861 [ 266 ] {} <Debug> TCPHandler: Connected oximeter version 23.8.0, revision: 54465, database: default, user: default. | ||
| 141 | 2025.05.14 01:41:24.458871 [ 3 ] {d4abc497-4df0-48a5-b355-fe4fba537595} <Debug> executeQuery: (from [::1]:55581) SELECT MAX(value) AS max_version FROM oximeter.version; (stage: Complete) | ||
| 142 | 2025.05.14 01:41:24.458921 [ 266 ] {} <Debug> TCP-Session: dfc35a2c-059c-4569-a053-17c409e210ad Authenticating user 'default' from [::1]:41079 | ||
| 143 | 2025.05.14 01:41:24.458952 [ 266 ] {} <Debug> TCP-Session: dfc35a2c-059c-4569-a053-17c409e210ad Authenticated with global context as user 94309d50-4f52-5250-31bd-74fecac179db | ||
| 144 | 2025.05.14 01:41:24.458985 [ 266 ] {} <Debug> TCP-Session: dfc35a2c-059c-4569-a053-17c409e210ad Creating session context with user_id: 94309d50-4f52-5250-31bd-74fecac179db | ||
| 145 | 2025.05.14 01:41:24.459420 [ 265 ] {} <Debug> TCPHandler: Connected oximeter version 23.8.0, revision: 54465, database: default, user: default. | ||
| 146 | 2025.05.14 01:41:24.459494 [ 265 ] {} <Debug> TCP-Session: d42900fb-75ec-4084-9f1f-3d35b61d89b0 Authenticating user 'default' from [::1]:50686 | ||
| 147 | 2025.05.14 01:41:24.459536 [ 265 ] {} <Debug> TCP-Session: d42900fb-75ec-4084-9f1f-3d35b61d89b0 Authenticated with global context as user 94309d50-4f52-5250-31bd-74fecac179db | ||
| 148 | 2025.05.14 01:41:24.459548 [ 265 ] {} <Debug> TCP-Session: d42900fb-75ec-4084-9f1f-3d35b61d89b0 Creating session context with user_id: 94309d50-4f52-5250-31bd-74fecac179db | ||
| 149 | 2025.05.14 01:41:24.459659 [ 264 ] {} <Debug> TCPHandler: Connected oximeter version 23.8.0, revision: 54465, database: default, user: default. | ||
| 150 | 2025.05.14 01:41:24.459744 [ 264 ] {} <Debug> TCP-Session: a87fb864-862b-4cda-9ecb-a672bd01e095 Authenticating user 'default' from [::1]:37237 | ||
| 151 | 2025.05.14 01:41:24.459892 [ 264 ] {} <Debug> TCP-Session: a87fb864-862b-4cda-9ecb-a672bd01e095 Authenticated with global context as user 94309d50-4f52-5250-31bd-74fecac179db | ||
| 152 | 2025.05.14 01:41:24.459924 [ 264 ] {} <Debug> TCP-Session: a87fb864-862b-4cda-9ecb-a672bd01e095 Creating session context with user_id: 94309d50-4f52-5250-31bd-74fecac179db | ||
| 153 | 2025.05.14 01:41:24.460182 [ 263 ] {} <Debug> TCPHandler: Connected oximeter version 23.8.0, revision: 54465, database: default, user: default. | ||
| 154 | 2025.05.14 01:41:24.460245 [ 263 ] {} <Debug> TCP-Session: a87f11b1-2c34-40e9-ac16-4258a164578e Authenticating user 'default' from [::1]:58062 | ||
| 155 | 2025.05.14 01:41:24.460269 [ 263 ] {} <Debug> TCP-Session: a87f11b1-2c34-40e9-ac16-4258a164578e Authenticated with global context as user 94309d50-4f52-5250-31bd-74fecac179db | ||
| 156 | 2025.05.14 01:41:24.460284 [ 263 ] {} <Debug> TCP-Session: a87f11b1-2c34-40e9-ac16-4258a164578e Creating session context with user_id: 94309d50-4f52-5250-31bd-74fecac179db | ||
| 157 | 2025.05.14 01:41:24.469509 [ 3 ] {d4abc497-4df0-48a5-b355-fe4fba537595} <Error> executeQuery: Code: 81. DB::Exception: Database oximeter does not exist. (UNKNOWN_DATABASE) (version 23.8.7.1) (from [::1]:55581) (in query: SELECT MAX(value) AS max_version FROM oximeter.version;), Stack trace (when copying this message, always include the lines below): | ||
| 158 | |||
| 159 | 0. StackTrace::tryCapture() @ 0x000000000f9acdb7 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 160 | 1. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000f97f7df in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 161 | 2. DB::Exception::Exception<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >(int, FormatStringHelperImpl<std::type_identity<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >::type>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&&) @ 0x000000000a3c7817 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 162 | 3. DB::DatabaseCatalog::getDatabase(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const @ 0x00000000147ad091 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 163 | 4. DB::Context::resolveStorageID(DB::StorageID, DB::Context::StorageNamespace) const @ 0x000000001476347d in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 164 | 5. DB::JoinedTables::getLeftTableStorage() @ 0x00000000148ec08b in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 165 | 6. DB::InterpreterSelectQuery::InterpreterSelectQuery(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context> const&, std::optional<DB::Pipe>, std::shared_ptr<DB::IStorage> const&, DB::SelectQueryOptions const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::PreparedSets>) @ 0x0000000014866701 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 166 | 7. DB::InterpreterSelectQuery::InterpreterSelectQuery(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context> const&, DB::SelectQueryOptions const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) @ 0x0000000014864f66 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 167 | 8. DB::InterpreterSelectWithUnionQuery::buildCurrentChildInterpreter(std::shared_ptr<DB::IAST> const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) @ 0x00000000148b3d46 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 168 | 9. DB::InterpreterSelectWithUnionQuery::InterpreterSelectWithUnionQuery(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context>, DB::SelectQueryOptions const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) @ 0x00000000148b2a83 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 169 | 10. std::__detail::_MakeUniq<DB::InterpreterSelectWithUnionQuery>::__single_object std::make_unique<DB::InterpreterSelectWithUnionQuery, std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context>&, DB::SelectQueryOptions const&>(std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context>&, DB::SelectQueryOptions const&) @ 0x000000001483d407 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 170 | 11. DB::InterpreterFactory::get(std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context>, DB::SelectQueryOptions const&) @ 0x000000001483c8d5 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 171 | 12. DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x0000000014a5f216 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 172 | 13. DB::executeQuery(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum) @ 0x0000000014a5bd46 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 173 | 14. DB::TCPHandler::runImpl() @ 0x00000000153fbaaa in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 174 | 15. DB::TCPHandler::run() @ 0x000000001540bc5c in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 175 | 16. Poco::Net::TCPServerConnection::start() @ 0x00000000187a058c in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 176 | 17. Poco::Net::TCPServerDispatcher::run() @ 0x00000000187a0a61 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 177 | 18. Poco::PooledThread::run() @ 0x0000000018805972 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 178 | 19. Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000018803425 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 179 | 20. _thrp_setup @ 0x0000000000116767 in /lib/amd64/libc.so.1 | ||
| 180 | |||
| 181 | 2025.05.14 01:41:24.469951 [ 3 ] {d4abc497-4df0-48a5-b355-fe4fba537595} <Error> TCPHandler: Code: 81. DB::Exception: Database oximeter does not exist. (UNKNOWN_DATABASE), Stack trace (when copying this message, always include the lines below): | ||
| 182 | |||
| 183 | 0. StackTrace::tryCapture() @ 0x000000000f9acdb7 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 184 | 1. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000f97f7df in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 185 | 2. DB::Exception::Exception<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >(int, FormatStringHelperImpl<std::type_identity<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >::type>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&&) @ 0x000000000a3c7817 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 186 | 3. DB::DatabaseCatalog::getDatabase(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const @ 0x00000000147ad091 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 187 | 4. DB::Context::resolveStorageID(DB::StorageID, DB::Context::StorageNamespace) const @ 0x000000001476347d in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 188 | 5. DB::JoinedTables::getLeftTableStorage() @ 0x00000000148ec08b in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 189 | 6. DB::InterpreterSelectQuery::InterpreterSelectQuery(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context> const&, std::optional<DB::Pipe>, std::shared_ptr<DB::IStorage> const&, DB::SelectQueryOptions const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::PreparedSets>) @ 0x0000000014866701 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 190 | 7. DB::InterpreterSelectQuery::InterpreterSelectQuery(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context> const&, DB::SelectQueryOptions const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) @ 0x0000000014864f66 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 191 | 8. DB::InterpreterSelectWithUnionQuery::buildCurrentChildInterpreter(std::shared_ptr<DB::IAST> const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) @ 0x00000000148b3d46 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 192 | 9. DB::InterpreterSelectWithUnionQuery::InterpreterSelectWithUnionQuery(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context>, DB::SelectQueryOptions const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) @ 0x00000000148b2a83 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 193 | 10. std::__detail::_MakeUniq<DB::InterpreterSelectWithUnionQuery>::__single_object std::make_unique<DB::InterpreterSelectWithUnionQuery, std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context>&, DB::SelectQueryOptions const&>(std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context>&, DB::SelectQueryOptions const&) @ 0x000000001483d407 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 194 | 11. DB::InterpreterFactory::get(std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context>, DB::SelectQueryOptions const&) @ 0x000000001483c8d5 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 195 | 12. DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x0000000014a5f216 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 196 | 13. DB::executeQuery(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum) @ 0x0000000014a5bd46 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 197 | 14. DB::TCPHandler::runImpl() @ 0x00000000153fbaaa in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 198 | 15. DB::TCPHandler::run() @ 0x000000001540bc5c in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 199 | 16. Poco::Net::TCPServerConnection::start() @ 0x00000000187a058c in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 200 | 17. Poco::Net::TCPServerDispatcher::run() @ 0x00000000187a0a61 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 201 | 18. Poco::PooledThread::run() @ 0x0000000018805972 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 202 | 19. Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000018803425 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 203 | 20. _thrp_setup @ 0x0000000000116767 in /lib/amd64/libc.so.1 | ||
| 204 | |||
| 205 | 2025.05.14 01:41:24.470204 [ 3 ] {d4abc497-4df0-48a5-b355-fe4fba537595} <Debug> TCPHandler: Processed in 0.013679404 sec. | ||
| 206 | 2025.05.14 01:41:24.482518 [ 4 ] {} <Trace> TCP-Session: 255c3e70-b7cc-4508-8a56-cbc93c3d08d8 Creating query context from session context, user_id: 94309d50-4f52-5250-31bd-74fecac179db, parent context user: default | ||
| 207 | 2025.05.14 01:41:24.482966 [ 4 ] {dd8889ed-26fb-46d1-949d-688d6752c1b6} <Debug> executeQuery: (from [::1]:46131) /* We purposefully split our DB schema into two *disjoint* files: | ||
| 208 | * `db-init-1.sql` and `db-init-2.sql`. The purpose of this split is to shorten | ||
| 209 | * the duration of our replicated tests. These tests only use a subset of the | ||
| 210 | * tables defined in the full schema. We put the tables used by the replicated | ||
| 211 | * tests in `db-init-1.sql`, and the remainder of the tables in `db-init-2.sql`. | ||
| 212 | * This minimizes test time by reducing the cost to load a schema. In | ||
| 213 | * production, we load `db-init-1.sql` followed by `db-init-2.sql` so we have | ||
| 214 | * the full schema. If we end up needing to use more tables in replicated tests | ||
| 215 | * we can go ahead and move them into `db-init-1.sql`, removing them from | ||
| 216 | * `db-init-2.sql`. Conversely, if we stop using given tables in our tests we | ||
| 217 | * can move them from `db-init-1.sql` into `db-init-2.sql` and keep our test | ||
| 218 | * times minimal. | ||
| 219 | |||
| 220 | * The reason to keep the two files disjoint is so that we don't have to | ||
| 221 | * maintain consistency between table definitions. All tables are defined | ||
| 222 | * once. However, in order to write a test that ensures the tables are in fact | ||
| 223 | * disjoint, we must create the `oximeter` database in both files so we can load | ||
| 224 | * them in isolation. | ||
| 225 | */ | ||
| 226 | CREATE DATABASE IF NOT EXISTS oximeter ON CLUSTER oximeter_cluster (stage: Complete) | ||
| 227 | 2025.05.14 01:41:24.483312 [ 4 ] {dd8889ed-26fb-46d1-949d-688d6752c1b6} <Trace> ContextAccess (default): Access granted: CREATE DATABASE ON oximeter.* | ||
| 228 | 2025.05.14 01:41:24.491611 [ 4 ] {dd8889ed-26fb-46d1-949d-688d6752c1b6} <Error> executeQuery: Code: 999. Coordination::Exception: All connection tries failed while connecting to ZooKeeper. nodes: [::1]:9181, [::1]:9183, [::1]:9182 | ||
| 229 | Poco::Exception. Code: 1000, e.code() = 146, Connection refused: [::1]:9181 (version 23.8.7.1), [::1]:9181 | ||
| 230 | Poco::Exception. Code: 1000, e.code() = 146, Connection refused: [::1]:9183 (version 23.8.7.1), [::1]:9183 | ||
| 231 | Poco::Exception. Code: 1000, e.code() = 146, Connection refused: [::1]:9182 (version 23.8.7.1), [::1]:9182 | ||
| 232 | Poco::Exception. Code: 1000, e.code() = 146, Connection refused: [::1]:9181 (version 23.8.7.1), [::1]:9181 | ||
| 233 | Poco::Exception. Code: 1000, e.code() = 146, Connection refused: [::1]:9183 (version 23.8.7.1), [::1]:9183 | ||
| 234 | Poco::Exception. Code: 1000, e.code() = 146, Connection refused: [::1]:9182 (version 23.8.7.1), [::1]:9182 | ||
| 235 | Poco::Exception. Code: 1000, e.code() = 146, Connection refused: [::1]:9181 (version 23.8.7.1), [::1]:9181 | ||
| 236 | Poco::Exception. Code: 1000, e.code() = 146, Connection refused: [::1]:9183 (version 23.8.7.1), [::1]:9183 | ||
| 237 | Poco::Exception. Code: 1000, e.code() = 146, Connection refused: [::1]:9182 (version 23.8.7.1), [::1]:9182 | ||
| 238 | . (KEEPER_EXCEPTION) (version 23.8.7.1) (from [::1]:46131) (in query: /* We purposefully split our DB schema into two *disjoint* files: | ||
| 239 | * `db-init-1.sql` and `db-init-2.sql`. The purpose of this split is to shorten | ||
| 240 | * the duration of our replicated tests. These tests only use a subset of the | ||
| 241 | * tables defined in the full schema. We put the tables used by the replicated | ||
| 242 | * tests in `db-init-1.sql`, and the remainder of the tables in `db-init-2.sql`. | ||
| 243 | * This minimizes test time by reducing the cost to load a schema. In | ||
| 244 | * production, we load `db-init-1.sql` followed by `db-init-2.sql` so we have | ||
| 245 | * the full schema. If we end up needing to use more tables in replicated tests | ||
| 246 | * we can go ahead and move them into `db-init-1.sql`, removing them from | ||
| 247 | * `db-init-2.sql`. Conversely, if we stop using given tables in our tests we | ||
| 248 | * can move them from `db-init-1.sql` into `db-init-2.sql` and keep our test | ||
| 249 | * times minimal. | ||
| 250 | |||
| 251 | * The reason to keep the two files disjoint is so that we don't have to | ||
| 252 | * maintain consistency between table definitions. All tables are defined | ||
| 253 | * once. However, in order to write a test that ensures the tables are in fact | ||
| 254 | * disjoint, we must create the `oximeter` database in both files so we can load | ||
| 255 | * them in isolation. | ||
| 256 | */ | ||
| 257 | CREATE DATABASE IF NOT EXISTS oximeter ON CLUSTER oximeter_cluster), Stack trace (when copying this message, always include the lines below): | ||
| 258 | |||
| 259 | 0. StackTrace::tryCapture() @ 0x000000000f9acdb7 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 260 | 1. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000f97f7df in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 261 | 2. DB::Exception::Exception<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&>(int, FormatStringHelperImpl<std::type_identity<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&>::type>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&) @ 0x000000000a7857f7 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 262 | 3. Coordination::Exception::Exception<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&>::type>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&) @ 0x0000000014e3842d in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 263 | 4. Coordination::ZooKeeper::connect(std::vector<Coordination::ZooKeeper::Node, std::allocator<Coordination::ZooKeeper::Node> > const&, Poco::Timespan) @ 0x000000001648290a in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 264 | 5. Coordination::ZooKeeper::ZooKeeper(std::vector<Coordination::ZooKeeper::Node, std::allocator<Coordination::ZooKeeper::Node> > const&, zkutil::ZooKeeperArgs const&, std::shared_ptr<DB::ZooKeeperLog>, std::optional<std::function<void (unsigned long, Coordination::ZooKeeper::Node const&)> >&&) @ 0x000000001648050b in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 265 | 6. zkutil::ZooKeeper::init(zkutil::ZooKeeperArgs) @ 0x0000000016449829 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 266 | 7. zkutil::ZooKeeper::ZooKeeper(Poco::Util::AbstractConfiguration const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<DB::ZooKeeperLog>) @ 0x000000001644b0a3 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 267 | 8. DB::Context::getZooKeeper() const @ 0x0000000014755b49 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 268 | 9. DB::DDLWorker::getAndSetZooKeeper() @ 0x0000000014792b4a in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 269 | 10. DB::DDLWorker::enqueueQuery[abi:cxx11](DB::DDLLogEntry&) @ 0x00000000147a099c in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 270 | 11. DB::executeDDLQueryOnCluster(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context const>, DB::DDLQueryOnClusterParams const&) @ 0x0000000014a4b2e7 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 271 | 12. DB::InterpreterCreateQuery::executeQueryOnCluster(DB::ASTCreateQuery&) @ 0x00000000148189e1 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 272 | 13. DB::InterpreterCreateQuery::execute() @ 0x0000000014820819 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 273 | 14. DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x0000000014a5f949 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 274 | 15. DB::executeQuery(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum) @ 0x0000000014a5bd46 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 275 | 16. DB::TCPHandler::runImpl() @ 0x00000000153fbaaa in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 276 | 17. DB::TCPHandler::run() @ 0x000000001540bc5c in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 277 | 18. Poco::Net::TCPServerConnection::start() @ 0x00000000187a058c in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 278 | 19. Poco::Net::TCPServerDispatcher::run() @ 0x00000000187a0a61 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 279 | 20. Poco::PooledThread::run() @ 0x0000000018805972 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 280 | 21. Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000018803425 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 281 | 22. _thrp_setup @ 0x0000000000116767 in /lib/amd64/libc.so.1 | ||
| 282 | |||
| 283 | 2025.05.14 01:41:24.491909 [ 4 ] {dd8889ed-26fb-46d1-949d-688d6752c1b6} <Error> TCPHandler: Code: 999. Coordination::Exception: All connection tries failed while connecting to ZooKeeper. nodes: [::1]:9181, [::1]:9183, [::1]:9182 | ||
| 284 | Poco::Exception. Code: 1000, e.code() = 146, Connection refused: [::1]:9181 (version 23.8.7.1), [::1]:9181 | ||
| 285 | Poco::Exception. Code: 1000, e.code() = 146, Connection refused: [::1]:9183 (version 23.8.7.1), [::1]:9183 | ||
| 286 | Poco::Exception. Code: 1000, e.code() = 146, Connection refused: [::1]:9182 (version 23.8.7.1), [::1]:9182 | ||
| 287 | Poco::Exception. Code: 1000, e.code() = 146, Connection refused: [::1]:9181 (version 23.8.7.1), [::1]:9181 | ||
| 288 | Poco::Exception. Code: 1000, e.code() = 146, Connection refused: [::1]:9183 (version 23.8.7.1), [::1]:9183 | ||
| 289 | Poco::Exception. Code: 1000, e.code() = 146, Connection refused: [::1]:9182 (version 23.8.7.1), [::1]:9182 | ||
| 290 | Poco::Exception. Code: 1000, e.code() = 146, Connection refused: [::1]:9181 (version 23.8.7.1), [::1]:9181 | ||
| 291 | Poco::Exception. Code: 1000, e.code() = 146, Connection refused: [::1]:9183 (version 23.8.7.1), [::1]:9183 | ||
| 292 | Poco::Exception. Code: 1000, e.code() = 146, Connection refused: [::1]:9182 (version 23.8.7.1), [::1]:9182 | ||
| 293 | . (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below): | ||
| 294 | |||
| 295 | 0. StackTrace::tryCapture() @ 0x000000000f9acdb7 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 296 | 1. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000f97f7df in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 297 | 2. DB::Exception::Exception<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&>(int, FormatStringHelperImpl<std::type_identity<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&>::type>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&) @ 0x000000000a7857f7 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 298 | 3. Coordination::Exception::Exception<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&>::type>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&) @ 0x0000000014e3842d in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 299 | 4. Coordination::ZooKeeper::connect(std::vector<Coordination::ZooKeeper::Node, std::allocator<Coordination::ZooKeeper::Node> > const&, Poco::Timespan) @ 0x000000001648290a in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 300 | 5. Coordination::ZooKeeper::ZooKeeper(std::vector<Coordination::ZooKeeper::Node, std::allocator<Coordination::ZooKeeper::Node> > const&, zkutil::ZooKeeperArgs const&, std::shared_ptr<DB::ZooKeeperLog>, std::optional<std::function<void (unsigned long, Coordination::ZooKeeper::Node const&)> >&&) @ 0x000000001648050b in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 301 | 6. zkutil::ZooKeeper::init(zkutil::ZooKeeperArgs) @ 0x0000000016449829 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 302 | 7. zkutil::ZooKeeper::ZooKeeper(Poco::Util::AbstractConfiguration const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<DB::ZooKeeperLog>) @ 0x000000001644b0a3 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 303 | 8. DB::Context::getZooKeeper() const @ 0x0000000014755b49 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 304 | 9. DB::DDLWorker::getAndSetZooKeeper() @ 0x0000000014792b4a in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 305 | 10. DB::DDLWorker::enqueueQuery[abi:cxx11](DB::DDLLogEntry&) @ 0x00000000147a099c in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 306 | 11. DB::executeDDLQueryOnCluster(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context const>, DB::DDLQueryOnClusterParams const&) @ 0x0000000014a4b2e7 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 307 | 12. DB::InterpreterCreateQuery::executeQueryOnCluster(DB::ASTCreateQuery&) @ 0x00000000148189e1 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 308 | 13. DB::InterpreterCreateQuery::execute() @ 0x0000000014820819 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 309 | 14. DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x0000000014a5f949 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 310 | 15. DB::executeQuery(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum) @ 0x0000000014a5bd46 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 311 | 16. DB::TCPHandler::runImpl() @ 0x00000000153fbaaa in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 312 | 17. DB::TCPHandler::run() @ 0x000000001540bc5c in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 313 | 18. Poco::Net::TCPServerConnection::start() @ 0x00000000187a058c in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 314 | 19. Poco::Net::TCPServerDispatcher::run() @ 0x00000000187a0a61 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 315 | 20. Poco::PooledThread::run() @ 0x0000000018805972 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 316 | 21. Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000018803425 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 317 | 22. _thrp_setup @ 0x0000000000116767 in /lib/amd64/libc.so.1 | ||
| 318 | |||
| 319 | 2025.05.14 01:41:24.492048 [ 4 ] {dd8889ed-26fb-46d1-949d-688d6752c1b6} <Debug> TCPHandler: Processed in 0.009853875 sec. | ||