Max level shown:
1 [ Aug 22 03:57:19 Enabled. ]
2 [ Aug 22 03:57:19 Rereading configuration. ]
3 [ Aug 22 03:57:20 Rereading configuration. ]
4 [ Aug 22 03:57:23 Executing start method ("/opt/oxide/lib/svc/manifest/cockroach-admin.sh"). ]
5 + set -o errexit
6 + set -o pipefail
7 + . /lib/svc/share/smf_include.sh
8 ++ SMF_EXIT_OK=0
9 ++ SMF_EXIT_NODAEMON=94
10 ++ SMF_EXIT_ERR_FATAL=95
11 ++ SMF_EXIT_ERR_CONFIG=96
12 ++ SMF_EXIT_MON_DEGRADE=97
13 ++ SMF_EXIT_MON_OFFLINE=98
14 ++ SMF_EXIT_ERR_NOSMF=99
15 ++ SMF_EXIT_ERR_PERM=100
16 ++ svcprop -c -p config/zone_id svc:/oxide/cockroach-admin:default
17 + ZONE_ID=0080441c-f636-4df8-bf07-c409bb3a2f03
18 ++ svcprop -c -p config/cockroach_address svc:/oxide/cockroach-admin:default
19 + COCKROACH_ADDR='[fd00:1122:3344:101::3]:32221'
20 ++ svcprop -c -p config/cockroach_http_address svc:/oxide/cockroach-admin:default
21 + COCKROACH_HTTP_ADDR=127.0.0.1:8080
22 ++ svcprop -c -p config/http_address svc:/oxide/cockroach-admin:default
23 + HTTP_ADDR='[fd00:1122:3344:101::3]:32222'
24 + args=('run' '--config-file-path' "/opt/oxide/lib/svc/cockroach-admin/config.toml" '--path-to-cockroach-binary' "/opt/oxide/cockroachdb/bin/cockroach" '--cockroach-address' "$COCKROACH_ADDR" '--cockroach-http-address' "$COCKROACH_HTTP_ADDR" '--http-address' "$HTTP_ADDR" '--zone-id' "$ZONE_ID")
25 + exec /opt/oxide/cockroach-admin/bin/cockroach-admin run --config-file-path /opt/oxide/lib/svc/cockroach-admin/config.toml --path-to-cockroach-binary /opt/oxide/cockroachdb/bin/cockroach --cockroach-address '[fd00:1122:3344:101::3]:32221' --cockroach-http-address 127.0.0.1:8080 --http-address '[fd00:1122:3344:101::3]:32222' --zone-id 0080441c-f636-4df8-bf07-c409bb3a2f03
26 [ Aug 22 03:57:23 Method "start" exited with status 0. ]
27 note: configured to log to "/dev/stdout"
282026-08-22T03:57:23.802ZINFOcockroach-admin (dropshot): listening file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:335 local_addr = [fd00:1122:3344:101::3]:32222
292026-08-22T03:57:35.232ZINFOcockroach-admin (dropshot): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:1057 local_addr = [fd00:1122:3344:101::3]:32222 remote_addr = [fd00:1122:3344:101::1]:52391
302026-08-22T03:57:35.235ZINFOcockroach-admin (ServerContext): Initializing CRDB cluster file = cockroach-admin/src/http_entrypoints.rs:38
312026-08-22T03:57:43.911ZINFOcockroach-admin (ServerContext): CRDB cluster initialized - initializing Omicron schema file = cockroach-admin/src/http_entrypoints.rs:40
322026-08-22T03:57:47.817ZINFOcockroach-admin (dropshot): request completed error_message_external = Command [sql --file /opt/oxide/cockroachdb/sql/dbinit.sql --host [fd00:1122:3344:101::3]:32221 --insecure] executed and failed with status: exit status: 1 stdout: SET CLUSTER SETTING\nBEGIN\nCREATE DATABASE\nCREATE ROLE\nALTER DEFAULT PRIVILEGES\nCONFIGURE ZONE 1\nCREATE TYPE\nCREATE TABLE\nCREATE TYPE\nCREATE TABLE\nINSERT 1\nCREATE TABLE\nCREATE TYPE\nCREATE TYPE\nCREATE TYPE\nCREATE TABLE\nCREATE INDEX\nCREATE INDEX\nCREATE INDEX\nCREATE TYPE\nCREATE TABLE\nCREATE INDEX\nCREATE INDEX\nCREATE INDEX\nCREATE TABLE\nCREATE INDEX\nCREATE TABLE\nCREATE INDEX\nCREATE TYPE\nCREATE TYPE\nCREATE TYPE\nCREATE TYPE\nCREATE TABLE\nCREATE INDEX\nCREATE INDEX\nCREATE INDEX\nCREATE TABLE\nCREATE INDEX\nCREATE TABLE\nCREATE INDEX\nCREATE INDEX\nCREATE TABLE\nCREATE TABLE\nCREATE TABLE\nCREATE INDEX\nCREATE TYPE\nCREATE TABLE\nCREATE INDEX\nCREATE INDEX\nCREATE INDEX\nCREATE TYPE\nCREATE TABLE\nCREATE INDEX\nCREATE INDEX\nCREATE INDEX\nCREATE INDEX\nCREATE TABLE\nCREATE INDEX\nCREATE INDEX\nCREATE INDEX\nCREATE INDEX\nCREATE INDEX\nCREATE TABLE\nCREATE INDEX\nCREATE TYPE\nCREATE TYPE\nCREATE TABLE\nCREATE INDEX\nCREATE TABLE\nCREATE INDEX\nCREATE INDEX\nCREATE TABLE\nCREATE TABLE\nCREATE INDEX\nCREATE INDEX\n stderr: ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionAbortedError(ABORT_REASON_ABORT_SPAN): "sql txn" meta={id=394cbb41 key=/NamespaceTable/30/1/104/0/"public"/4/1 pri=0.02187528 epo=0 ts=1787371067.734022242,1 min=1787371067.324105254,0 seq=606} lock=true stat=ABORTED rts=1787371067.324105254,0 wto=false gul=1787371067.824105254,0\nSQLSTATE: 40001\nHINT: See: https://www.cockroachlabs.com/docs/v22.1/transaction-retry-error-reference.html#abort_reason_abort_span\nERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionAbortedError(ABORT_REASON_ABORT_SPAN): "sql txn" meta={id=394cbb41 key=/NamespaceTable/30/1/104/0/"public"/4/1 pri=0.02187528 epo=0 ts=1787371067.734022242,1 min=1787371067.324105254,0 seq=606} lock=true stat=ABORTED rts=1787371067.324105254,0 wto=false gul=1787371067.824105254,0\nSQLSTATE: 40001\nHINT: See: https://www.cockroachlabs.com/docs/v22.1/transaction-retry-error-reference.html#abort_reason_abort_span\nFailed running "sql"\n error_message_internal = Command [sql --file /opt/oxide/cockroachdb/sql/dbinit.sql --host [fd00:1122:3344:101::3]:32221 --insecure] executed and failed with status: exit status: 1 stdout: SET CLUSTER SETTING\nBEGIN\nCREATE DATABASE\nCREATE ROLE\nALTER DEFAULT PRIVILEGES\nCONFIGURE ZONE 1\nCREATE TYPE\nCREATE TABLE\nCREATE TYPE\nCREATE TABLE\nINSERT 1\nCREATE TABLE\nCREATE TYPE\nCREATE TYPE\nCREATE TYPE\nCREATE TABLE\nCREATE INDEX\nCREATE INDEX\nCREATE INDEX\nCREATE TYPE\nCREATE TABLE\nCREATE INDEX\nCREATE INDEX\nCREATE INDEX\nCREATE TABLE\nCREATE INDEX\nCREATE TABLE\nCREATE INDEX\nCREATE TYPE\nCREATE TYPE\nCREATE TYPE\nCREATE TYPE\nCREATE TABLE\nCREATE INDEX\nCREATE INDEX\nCREATE INDEX\nCREATE TABLE\nCREATE INDEX\nCREATE TABLE\nCREATE INDEX\nCREATE INDEX\nCREATE TABLE\nCREATE TABLE\nCREATE TABLE\nCREATE INDEX\nCREATE TYPE\nCREATE TABLE\nCREATE INDEX\nCREATE INDEX\nCREATE INDEX\nCREATE TYPE\nCREATE TABLE\nCREATE INDEX\nCREATE INDEX\nCREATE INDEX\nCREATE INDEX\nCREATE TABLE\nCREATE INDEX\nCREATE INDEX\nCREATE INDEX\nCREATE INDEX\nCREATE INDEX\nCREATE TABLE\nCREATE INDEX\nCREATE TYPE\nCREATE TYPE\nCREATE TABLE\nCREATE INDEX\nCREATE TABLE\nCREATE INDEX\nCREATE INDEX\nCREATE TABLE\nCREATE TABLE\nCREATE INDEX\nCREATE INDEX\n stderr: ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionAbortedError(ABORT_REASON_ABORT_SPAN): "sql txn" meta={id=394cbb41 key=/NamespaceTable/30/1/104/0/"public"/4/1 pri=0.02187528 epo=0 ts=1787371067.734022242,1 min=1787371067.324105254,0 seq=606} lock=true stat=ABORTED rts=1787371067.324105254,0 wto=false gul=1787371067.824105254,0\nSQLSTATE: 40001\nHINT: See: https://www.cockroachlabs.com/docs/v22.1/transaction-retry-error-reference.html#abort_reason_abort_span\nERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionAbortedError(ABORT_REASON_ABORT_SPAN): "sql txn" meta={id=394cbb41 key=/NamespaceTable/30/1/104/0/"public"/4/1 pri=0.02187528 epo=0 ts=1787371067.734022242,1 min=1787371067.324105254,0 seq=606} lock=true stat=ABORTED rts=1787371067.324105254,0 wto=false gul=1787371067.824105254,0\nSQLSTATE: 40001\nHINT: See: https://www.cockroachlabs.com/docs/v22.1/transaction-retry-error-reference.html#abort_reason_abort_span\nFailed running "sql"\n file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:862 latency_us = 12582315 local_addr = [fd00:1122:3344:101::3]:32222 method = POST remote_addr = [fd00:1122:3344:101::1]:52391 req_id = 3823e3b5-5f68-4d80-84ad-ac70d782a4cd response_code = 500 uri = /cluster/init
332026-08-22T03:57:47.949ZINFOcockroach-admin (ServerContext): Initializing CRDB cluster file = cockroach-admin/src/http_entrypoints.rs:38
342026-08-22T03:57:48.089ZINFOcockroach-admin (ServerContext): CRDB cluster initialized - initializing Omicron schema file = cockroach-admin/src/http_entrypoints.rs:40
352026-08-22T03:58:11.486ZINFOcockroach-admin (ServerContext): Omicron schema initialized file = cockroach-admin/src/http_entrypoints.rs:42
362026-08-22T03:58:11.487ZINFOcockroach-admin (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874 latency_us = 23536816 local_addr = [fd00:1122:3344:101::3]:32222 method = POST remote_addr = [fd00:1122:3344:101::1]:52391 req_id = 49c6359e-1038-4a60-b480-66bffc368e33 response_code = 204 uri = /cluster/init
372026-08-22T03:59:13.119ZINFOcockroach-admin (dropshot): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:1057 local_addr = [fd00:1122:3344:101::3]:32222 remote_addr = [fd00:1122:3344:101::b]:51024
382026-08-22T03:59:14.057ZINFOcockroach-admin (dropshot): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:1057 local_addr = [fd00:1122:3344:101::3]:32222 remote_addr = [fd00:1122:3344:101::c]:59507
392026-08-22T03:59:14.123ZINFOcockroach-admin (dropshot): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:1057 local_addr = [fd00:1122:3344:101::3]:32222 remote_addr = [fd00:1122:3344:101::a]:47870
402026-08-22T03:59:14.408ZINFOcockroach-admin (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874 latency_us = 1281606 local_addr = [fd00:1122:3344:101::3]:32222 method = GET remote_addr = [fd00:1122:3344:101::b]:51024 req_id = af0682c1-c8a9-409b-8541-7dc3f67b6ded response_code = 200 uri = /node/id
412026-08-22T03:59:14.408ZINFOcockroach-admin (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874 latency_us = 284849 local_addr = [fd00:1122:3344:101::3]:32222 method = GET remote_addr = [fd00:1122:3344:101::a]:47870 req_id = 2e879276-a1f5-4a8a-882e-c633618e5856 response_code = 200 uri = /node/id
422026-08-22T03:59:14.409ZINFOcockroach-admin (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874 latency_us = 351186 local_addr = [fd00:1122:3344:101::3]:32222 method = GET remote_addr = [fd00:1122:3344:101::c]:59507 req_id = e315a553-90fd-4d31-abcf-f03a4217cae0 response_code = 200 uri = /node/id
432026-08-22T03:59:14.527ZINFOcockroach-admin (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874 latency_us = 114373 local_addr = [fd00:1122:3344:101::3]:32222 method = GET remote_addr = [fd00:1122:3344:101::a]:47870 req_id = d7a5ac96-73de-48cd-adef-5ba4242a0641 response_code = 200 uri = /proxy/status/vars
442026-08-22T03:59:14.528ZINFOcockroach-admin (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874 latency_us = 114425 local_addr = [fd00:1122:3344:101::3]:32222 method = GET remote_addr = [fd00:1122:3344:101::c]:59507 req_id = bb6d18b0-c056-4bb3-be3f-7fec8bde3508 response_code = 200 uri = /proxy/status/vars
452026-08-22T03:59:14.531ZINFOcockroach-admin (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874 latency_us = 117534 local_addr = [fd00:1122:3344:101::3]:32222 method = GET remote_addr = [fd00:1122:3344:101::b]:51024 req_id = fe284747-d3e4-462a-8ea5-ba9072334678 response_code = 200 uri = /proxy/status/vars
462026-08-22T03:59:19.786ZINFOcockroach-admin (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874 latency_us = 64 local_addr = [fd00:1122:3344:101::3]:32222 method = GET remote_addr = [fd00:1122:3344:101::a]:47870 req_id = 59a6cf44-a3bc-4388-be97-fa27a65cee19 response_code = 200 uri = /node/id
472026-08-22T03:59:19.793ZINFOcockroach-admin (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874 latency_us = 7120 local_addr = [fd00:1122:3344:101::3]:32222 method = GET remote_addr = [fd00:1122:3344:101::a]:47870 req_id = f7bb6565-d42f-4881-9da1-0f1de08f216f response_code = 200 uri = /proxy/status/vars
482026-08-22T03:59:23.817ZINFOcockroach-admin (dropshot): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:1057 local_addr = [fd00:1122:3344:101::3]:32222 remote_addr = [fd00:1122:3344:101::a]:65037
492026-08-22T03:59:23.818ZINFOcockroach-admin (dropshot): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:1057 local_addr = [fd00:1122:3344:101::3]:32222 remote_addr = [fd00:1122:3344:101::c]:37910
502026-08-22T03:59:23.820ZINFOcockroach-admin (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874 latency_us = 70 local_addr = [fd00:1122:3344:101::3]:32222 method = GET remote_addr = [fd00:1122:3344:101::a]:65037 req_id = 55589d4a-af71-4e5e-983c-32701dd3a538 response_code = 200 uri = /node/id
512026-08-22T03:59:23.821ZINFOcockroach-admin (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874 latency_us = 94 local_addr = [fd00:1122:3344:101::3]:32222 method = GET remote_addr = [fd00:1122:3344:101::c]:37910 req_id = e59b0036-00ae-4bfc-ae8f-ce080ee1570a response_code = 200 uri = /node/id
522026-08-22T03:59:23.873ZINFOcockroach-admin (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874 latency_us = 61 local_addr = [fd00:1122:3344:101::3]:32222 method = GET remote_addr = [fd00:1122:3344:101::c]:59507 req_id = a8530ff0-70d5-40c6-97b3-b16bf08cabba response_code = 200 uri = /node/id
532026-08-22T03:59:23.882ZINFOcockroach-admin (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874 latency_us = 8742 local_addr = [fd00:1122:3344:101::3]:32222 method = GET remote_addr = [fd00:1122:3344:101::c]:59507 req_id = bcb75f88-0bb6-4f38-9d6d-365a76b24b02 response_code = 200 uri = /proxy/status/vars
542026-08-22T03:59:25.293ZINFOcockroach-admin (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874 latency_us = 60 local_addr = [fd00:1122:3344:101::3]:32222 method = GET remote_addr = [fd00:1122:3344:101::a]:47870 req_id = d5ef6f45-441f-449b-94e8-9d721dfdb3ed response_code = 200 uri = /node/id
552026-08-22T03:59:25.300ZINFOcockroach-admin (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874 latency_us = 7051 local_addr = [fd00:1122:3344:101::3]:32222 method = GET remote_addr = [fd00:1122:3344:101::a]:47870 req_id = 523b771c-cb1f-44c1-bd6a-4d53ce3e1226 response_code = 200 uri = /proxy/status/vars
562026-08-22T03:59:25.348ZINFOcockroach-admin (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874 latency_us = 72 local_addr = [fd00:1122:3344:101::3]:32222 method = GET remote_addr = [fd00:1122:3344:101::c]:59507 req_id = 40ed4c83-fb50-49f7-b4c5-c09ec244cee7 response_code = 200 uri = /node/id
572026-08-22T03:59:25.356ZINFOcockroach-admin (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874 latency_us = 7950 local_addr = [fd00:1122:3344:101::3]:32222 method = GET remote_addr = [fd00:1122:3344:101::c]:59507 req_id = cb6def0d-6530-4bea-956b-b566eeb7c78f response_code = 200 uri = /proxy/status/vars
582026-08-22T03:59:33.828ZINFOcockroach-admin (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874 latency_us = 66 local_addr = [fd00:1122:3344:101::3]:32222 method = GET remote_addr = [fd00:1122:3344:101::b]:51024 req_id = 7fa6e40c-c53f-4279-84d2-d9d86db9d357 response_code = 200 uri = /node/id
592026-08-22T03:59:33.838ZINFOcockroach-admin (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874 latency_us = 9641 local_addr = [fd00:1122:3344:101::3]:32222 method = GET remote_addr = [fd00:1122:3344:101::b]:51024 req_id = 5021f79e-cd11-42d7-b822-95baae0c6ffa response_code = 200 uri = /proxy/status/vars