Max level shown:
1 [ Jan 12 09:27:38 Enabled. ]
2 [ Jan 12 09:27:38 Rereading configuration. ]
3 [ Jan 12 09:27:39 Rereading configuration. ]
4 [ Jan 12 09:27:44 Executing start method ("/opt/oxide/lib/svc/manifest/cockroach-admin.sh"). ]
5 + set -o errexit
6 + set -o pipefail
7 + . /lib/svc/share/smf_include.sh
8 ++ SMF_EXIT_OK=0
9 ++ SMF_EXIT_NODAEMON=94
10 ++ SMF_EXIT_ERR_FATAL=95
11 ++ SMF_EXIT_ERR_CONFIG=96
12 ++ SMF_EXIT_MON_DEGRADE=97
13 ++ SMF_EXIT_MON_OFFLINE=98
14 ++ SMF_EXIT_ERR_NOSMF=99
15 ++ SMF_EXIT_ERR_PERM=100
16 ++ svcprop -c -p config/zone_id svc:/oxide/cockroach-admin:default
17 + ZONE_ID=4de083cb-3d00-4701-ab2c-7a45ad2bb58b
18 ++ svcprop -c -p config/cockroach_address svc:/oxide/cockroach-admin:default
19 + COCKROACH_ADDR='[fd00:1122:3344:101::4]:32221'
20 ++ svcprop -c -p config/cockroach_http_address svc:/oxide/cockroach-admin:default
21 + COCKROACH_HTTP_ADDR=127.0.0.1:8080
22 ++ svcprop -c -p config/http_address svc:/oxide/cockroach-admin:default
23 + HTTP_ADDR='[fd00:1122:3344:101::4]:32222'
24 + args=('run' '--config-file-path' "/opt/oxide/lib/svc/cockroach-admin/config.toml" '--path-to-cockroach-binary' "/opt/oxide/cockroachdb/bin/cockroach" '--cockroach-address' "$COCKROACH_ADDR" '--cockroach-http-address' "$COCKROACH_HTTP_ADDR" '--http-address' "$HTTP_ADDR" '--zone-id' "$ZONE_ID")
25 + exec /opt/oxide/cockroach-admin/bin/cockroach-admin run --config-file-path /opt/oxide/lib/svc/cockroach-admin/config.toml --path-to-cockroach-binary /opt/oxide/cockroachdb/bin/cockroach --cockroach-address '[fd00:1122:3344:101::4]:32221' --cockroach-http-address 127.0.0.1:8080 --http-address '[fd00:1122:3344:101::4]:32222' --zone-id 4de083cb-3d00-4701-ab2c-7a45ad2bb58b
26 [ Jan 12 09:27:44 Method "start" exited with status 0. ]
27 note: configured to log to "/dev/stdout"
282026-01-12T09:27:44.222ZINFOcockroach-admin (dropshot): listening file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.6/src/server.rs:328 local_addr = [fd00:1122:3344:101::4]:32222
292026-01-12T09:27:44.260ZINFOcockroach-admin (dropshot): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.6/src/server.rs:1025 local_addr = [fd00:1122:3344:101::4]:32222 remote_addr = [fd00:1122:3344:101::1]:40956
302026-01-12T09:27:44.264ZINFOcockroach-admin (ServerContext): Initializing CRDB cluster file = cockroach-admin/src/http_entrypoints.rs:37
312026-01-12T09:27:55.098ZINFOcockroach-admin (ServerContext): CRDB cluster initialized - initializing Omicron schema file = cockroach-admin/src/http_entrypoints.rs:39
322026-01-12T09:27:55.878ZINFOcockroach-admin (dropshot): request completed error_message_external = Command [sql --file /opt/oxide/cockroachdb/sql/dbinit.sql --host [fd00:1122:3344:101::4]:32221 --insecure] executed and failed with status: exit status: 1 stdout: BEGIN\nCREATE DATABASE\nCREATE ROLE\nALTER DEFAULT PRIVILEGES\nCONFIGURE ZONE 1\n stderr: ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionAbortedError(ABORT_REASON_ABORT_SPAN): "sql txn" meta={id=39dcf476 key=/NamespaceTable/30/1/104/0/"public"/4/1 pri=0.01502088 epo=0 ts=1768210075.725559381,1 min=1768210075.691522728,0 seq=38} lock=true stat=ABORTED rts=1768210075.691522728,0 wto=false gul=1768210076.191522728,0\nSQLSTATE: 40001\nHINT: See: https://www.cockroachlabs.com/docs/v22.1/transaction-retry-error-reference.html#abort_reason_abort_span\nERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionAbortedError(ABORT_REASON_ABORT_SPAN): "sql txn" meta={id=39dcf476 key=/NamespaceTable/30/1/104/0/"public"/4/1 pri=0.01502088 epo=0 ts=1768210075.725559381,1 min=1768210075.691522728,0 seq=38} lock=true stat=ABORTED rts=1768210075.691522728,0 wto=false gul=1768210076.191522728,0\nSQLSTATE: 40001\nHINT: See: https://www.cockroachlabs.com/docs/v22.1/transaction-retry-error-reference.html#abort_reason_abort_span\nFailed running "sql"\n error_message_internal = Command [sql --file /opt/oxide/cockroachdb/sql/dbinit.sql --host [fd00:1122:3344:101::4]:32221 --insecure] executed and failed with status: exit status: 1 stdout: BEGIN\nCREATE DATABASE\nCREATE ROLE\nALTER DEFAULT PRIVILEGES\nCONFIGURE ZONE 1\n stderr: ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionAbortedError(ABORT_REASON_ABORT_SPAN): "sql txn" meta={id=39dcf476 key=/NamespaceTable/30/1/104/0/"public"/4/1 pri=0.01502088 epo=0 ts=1768210075.725559381,1 min=1768210075.691522728,0 seq=38} lock=true stat=ABORTED rts=1768210075.691522728,0 wto=false gul=1768210076.191522728,0\nSQLSTATE: 40001\nHINT: See: https://www.cockroachlabs.com/docs/v22.1/transaction-retry-error-reference.html#abort_reason_abort_span\nERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionAbortedError(ABORT_REASON_ABORT_SPAN): "sql txn" meta={id=39dcf476 key=/NamespaceTable/30/1/104/0/"public"/4/1 pri=0.01502088 epo=0 ts=1768210075.725559381,1 min=1768210075.691522728,0 seq=38} lock=true stat=ABORTED rts=1768210075.691522728,0 wto=false gul=1768210076.191522728,0\nSQLSTATE: 40001\nHINT: See: https://www.cockroachlabs.com/docs/v22.1/transaction-retry-error-reference.html#abort_reason_abort_span\nFailed running "sql"\n file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.6/src/server.rs:855 latency_us = 11613877 local_addr = [fd00:1122:3344:101::4]:32222 method = POST remote_addr = [fd00:1122:3344:101::1]:40956 req_id = a2ae7bb5-ea96-40b9-9dda-c292d89775c3 response_code = 500 uri = /cluster/init
332026-01-12T09:27:56.027ZINFOcockroach-admin (ServerContext): Initializing CRDB cluster file = cockroach-admin/src/http_entrypoints.rs:37
342026-01-12T09:27:56.187ZINFOcockroach-admin (ServerContext): CRDB cluster initialized - initializing Omicron schema file = cockroach-admin/src/http_entrypoints.rs:39
352026-01-12T09:28:02.866ZINFOcockroach-admin (ServerContext): Omicron schema initialized file = cockroach-admin/src/http_entrypoints.rs:41
362026-01-12T09:28:02.866ZINFOcockroach-admin (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.6/src/server.rs:867 latency_us = 6844031 local_addr = [fd00:1122:3344:101::4]:32222 method = POST remote_addr = [fd00:1122:3344:101::1]:40956 req_id = 4184ed2b-7abd-473d-a9d4-194c22fef704 response_code = 204 uri = /cluster/init
372026-01-12T09:28:57.541ZINFOcockroach-admin (dropshot): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.6/src/server.rs:1025 local_addr = [fd00:1122:3344:101::4]:32222 remote_addr = [fd00:1122:3344:101::b]:49760
382026-01-12T09:28:58.473ZINFOcockroach-admin (dropshot): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.6/src/server.rs:1025 local_addr = [fd00:1122:3344:101::4]:32222 remote_addr = [fd00:1122:3344:101::c]:48716
392026-01-12T09:28:58.743ZINFOcockroach-admin (dropshot): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.6/src/server.rs:1025 local_addr = [fd00:1122:3344:101::4]:32222 remote_addr = [fd00:1122:3344:101::a]:54148
402026-01-12T09:28:58.932ZINFOcockroach-admin (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.6/src/server.rs:867 latency_us = 189089 local_addr = [fd00:1122:3344:101::4]:32222 method = GET remote_addr = [fd00:1122:3344:101::a]:54148 req_id = 71674f38-d47b-4c78-97e2-b39ad1da9fa8 response_code = 200 uri = /node/id
412026-01-12T09:28:58.933ZINFOcockroach-admin (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.6/src/server.rs:867 latency_us = 458739 local_addr = [fd00:1122:3344:101::4]:32222 method = GET remote_addr = [fd00:1122:3344:101::c]:48716 req_id = d28be8ad-5775-4b9b-b28b-1c9a0c3adb47 response_code = 200 uri = /node/id
422026-01-12T09:28:58.933ZINFOcockroach-admin (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.6/src/server.rs:867 latency_us = 1357430 local_addr = [fd00:1122:3344:101::4]:32222 method = GET remote_addr = [fd00:1122:3344:101::b]:49760 req_id = e773b9af-3e39-4045-b383-a6b693368809 response_code = 200 uri = /node/id
432026-01-12T09:28:59.214ZINFOcockroach-admin (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.6/src/server.rs:867 latency_us = 265590 local_addr = [fd00:1122:3344:101::4]:32222 method = GET remote_addr = [fd00:1122:3344:101::b]:49760 req_id = a0f9f9a5-5bd7-4029-a483-121119ba704c response_code = 200 uri = /proxy/status/vars
442026-01-12T09:28:59.223ZINFOcockroach-admin (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.6/src/server.rs:867 latency_us = 275427 local_addr = [fd00:1122:3344:101::4]:32222 method = GET remote_addr = [fd00:1122:3344:101::c]:48716 req_id = 8e5afa7d-7df6-4cec-b2ab-c1255c90b6ab response_code = 200 uri = /proxy/status/vars
452026-01-12T09:28:59.227ZINFOcockroach-admin (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.6/src/server.rs:867 latency_us = 279719 local_addr = [fd00:1122:3344:101::4]:32222 method = GET remote_addr = [fd00:1122:3344:101::a]:54148 req_id = 739c0e8c-2c02-407b-a918-53e5e2d80179 response_code = 200 uri = /proxy/status/vars
462026-01-12T09:29:25.321ZINFOcockroach-admin (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.6/src/server.rs:867 latency_us = 66 local_addr = [fd00:1122:3344:101::4]:32222 method = GET remote_addr = [fd00:1122:3344:101::c]:48716 req_id = 56484158-c8ce-49df-947d-68cc6392d3d0 response_code = 200 uri = /node/id
472026-01-12T09:29:25.359ZINFOcockroach-admin (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.6/src/server.rs:867 latency_us = 38313 local_addr = [fd00:1122:3344:101::4]:32222 method = GET remote_addr = [fd00:1122:3344:101::c]:48716 req_id = fa65c191-d023-4ab2-bfce-83a2e82939b8 response_code = 200 uri = /proxy/status/vars
482026-01-12T09:29:34.020ZINFOcockroach-admin (dropshot): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.6/src/server.rs:1025 local_addr = [fd00:1122:3344:101::4]:32222 remote_addr = [fd00:1122:3344:101::c]:35094
492026-01-12T09:29:34.066ZINFOcockroach-admin (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.6/src/server.rs:867 latency_us = 72 local_addr = [fd00:1122:3344:101::4]:32222 method = GET remote_addr = [fd00:1122:3344:101::c]:48716 req_id = 44d89084-8eb9-4318-b093-5df6bf2071e4 response_code = 200 uri = /node/id
502026-01-12T09:29:34.104ZINFOcockroach-admin (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.6/src/server.rs:867 latency_us = 87 local_addr = [fd00:1122:3344:101::4]:32222 method = GET remote_addr = [fd00:1122:3344:101::c]:35094 req_id = e01c7f4d-34f2-4170-ad9c-5c6dcc038d27 response_code = 200 uri = /node/id
512026-01-12T09:29:34.108ZINFOcockroach-admin (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.6/src/server.rs:867 latency_us = 41800 local_addr = [fd00:1122:3344:101::4]:32222 method = GET remote_addr = [fd00:1122:3344:101::c]:48716 req_id = c182fc36-2724-416c-9a01-6f0d6f15c5af response_code = 200 uri = /proxy/status/vars
522026-01-12T09:29:35.288ZINFOcockroach-admin (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.6/src/server.rs:867 latency_us = 84 local_addr = [fd00:1122:3344:101::4]:32222 method = GET remote_addr = [fd00:1122:3344:101::c]:48716 req_id = b1683427-d78e-4fac-be7f-fb32133e0349 response_code = 200 uri = /node/id
532026-01-12T09:29:35.328ZINFOcockroach-admin (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.6/src/server.rs:867 latency_us = 39583 local_addr = [fd00:1122:3344:101::4]:32222 method = GET remote_addr = [fd00:1122:3344:101::c]:48716 req_id = a1522689-6c51-4871-bf1c-420ac3d5573c response_code = 200 uri = /proxy/status/vars
542026-01-12T09:29:37.992ZINFOcockroach-admin (dropshot): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.6/src/server.rs:1025 local_addr = [fd00:1122:3344:101::4]:32222 remote_addr = [fd00:1122:3344:101::b]:41570
552026-01-12T09:29:37.992ZINFOcockroach-admin (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.6/src/server.rs:867 latency_us = 48 local_addr = [fd00:1122:3344:101::4]:32222 method = GET remote_addr = [fd00:1122:3344:101::b]:41570 req_id = 983e6e79-c4bf-49a2-adbb-dbe9a056cd5b response_code = 200 uri = /node/id
562026-01-12T09:29:38.039ZINFOcockroach-admin (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.6/src/server.rs:867 latency_us = 46832 local_addr = [fd00:1122:3344:101::4]:32222 method = GET remote_addr = [fd00:1122:3344:101::b]:41570 req_id = ab59316e-e377-4d37-a47d-7cc5b5c11ee4 response_code = 200 uri = /proxy/status/vars
572026-01-12T09:29:39.218ZINFOcockroach-admin (dropshot): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.6/src/server.rs:1025 local_addr = [fd00:1122:3344:101::4]:32222 remote_addr = [fd00:1122:3344:101::a]:61810
582026-01-12T09:29:39.218ZINFOcockroach-admin (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.6/src/server.rs:867 latency_us = 54 local_addr = [fd00:1122:3344:101::4]:32222 method = GET remote_addr = [fd00:1122:3344:101::a]:61810 req_id = 3a9089a4-ac8f-4722-9c71-7725674e47e0 response_code = 200 uri = /node/id
592026-01-12T09:29:39.255ZINFOcockroach-admin (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.6/src/server.rs:867 latency_us = 36687 local_addr = [fd00:1122:3344:101::4]:32222 method = GET remote_addr = [fd00:1122:3344:101::a]:61810 req_id = 778ce235-5f9c-494a-81e4-aaf9fdbcf7b5 response_code = 200 uri = /proxy/status/vars
602026-01-12T09:29:58.837ZINFOcockroach-admin (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.6/src/server.rs:867 latency_us = 68 local_addr = [fd00:1122:3344:101::4]:32222 method = GET remote_addr = [fd00:1122:3344:101::b]:41570 req_id = ff8b28c0-2ab5-46f4-8ff6-7ee2404d7178 response_code = 200 uri = /node/id
612026-01-12T09:29:58.880ZINFOcockroach-admin (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.6/src/server.rs:867 latency_us = 41950 local_addr = [fd00:1122:3344:101::4]:32222 method = GET remote_addr = [fd00:1122:3344:101::b]:41570 req_id = ac16ce10-5429-4d81-a96d-8e047944f621 response_code = 200 uri = /proxy/status/vars
622026-01-12T09:29:59.142ZINFOcockroach-admin (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.6/src/server.rs:867 latency_us = 67 local_addr = [fd00:1122:3344:101::4]:32222 method = GET remote_addr = [fd00:1122:3344:101::c]:48716 req_id = 02a4ce06-8bd8-4173-9fd3-62fc6430186f response_code = 200 uri = /node/id
632026-01-12T09:29:59.183ZINFOcockroach-admin (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.6/src/server.rs:867 latency_us = 40436 local_addr = [fd00:1122:3344:101::4]:32222 method = GET remote_addr = [fd00:1122:3344:101::c]:48716 req_id = 063a4e54-2c23-4c78-94ec-8a8a85c746c8 response_code = 200 uri = /proxy/status/vars
642026-01-12T09:30:00.105ZINFOcockroach-admin (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.6/src/server.rs:867 latency_us = 57 local_addr = [fd00:1122:3344:101::4]:32222 method = GET remote_addr = [fd00:1122:3344:101::a]:61810 req_id = 5503aaa1-459a-4205-b5cf-a188b2be2e77 response_code = 200 uri = /node/id
652026-01-12T09:30:00.144ZINFOcockroach-admin (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.6/src/server.rs:867 latency_us = 38766 local_addr = [fd00:1122:3344:101::4]:32222 method = GET remote_addr = [fd00:1122:3344:101::a]:61810 req_id = 7aa720ab-a735-4d90-b486-ace0cfa7fd2c response_code = 200 uri = /proxy/status/vars