Max level shown:
1 [ Jul 28 03:11:32 Enabled. ]
2 [ Jul 28 03:11:32 Rereading configuration. ]
3 [ Jul 28 03:11:32 Rereading configuration. ]
4 [ Jul 28 03:11:34 Executing start method ("/opt/oxide/lib/svc/manifest/cockroach-admin.sh"). ]
5 + set -o errexit
6 + set -o pipefail
7 + . /lib/svc/share/smf_include.sh
8 ++ SMF_EXIT_OK=0
9 ++ SMF_EXIT_NODAEMON=94
10 ++ SMF_EXIT_ERR_FATAL=95
11 ++ SMF_EXIT_ERR_CONFIG=96
12 ++ SMF_EXIT_MON_DEGRADE=97
13 ++ SMF_EXIT_MON_OFFLINE=98
14 ++ SMF_EXIT_ERR_NOSMF=99
15 ++ SMF_EXIT_ERR_PERM=100
16 ++ svcprop -c -p config/zone_id svc:/oxide/cockroach-admin:default
17 + ZONE_ID=174429bd-2fe0-4ec4-9ef9-5f7bb413f931
18 ++ svcprop -c -p config/cockroach_address svc:/oxide/cockroach-admin:default
19 + COCKROACH_ADDR='[fd00:1122:3344:101::3]:32221'
20 ++ svcprop -c -p config/cockroach_http_address svc:/oxide/cockroach-admin:default
21 + COCKROACH_HTTP_ADDR=127.0.0.1:8080
22 ++ svcprop -c -p config/http_address svc:/oxide/cockroach-admin:default
23 + HTTP_ADDR='[fd00:1122:3344:101::3]:32222'
24 + args=('run' '--config-file-path' "/opt/oxide/lib/svc/cockroach-admin/config.toml" '--path-to-cockroach-binary' "/opt/oxide/cockroachdb/bin/cockroach" '--cockroach-address' "$COCKROACH_ADDR" '--cockroach-http-address' "$COCKROACH_HTTP_ADDR" '--http-address' "$HTTP_ADDR" '--zone-id' "$ZONE_ID")
25 + exec /opt/oxide/cockroach-admin/bin/cockroach-admin run --config-file-path /opt/oxide/lib/svc/cockroach-admin/config.toml --path-to-cockroach-binary /opt/oxide/cockroachdb/bin/cockroach --cockroach-address '[fd00:1122:3344:101::3]:32221' --cockroach-http-address 127.0.0.1:8080 --http-address '[fd00:1122:3344:101::3]:32222' --zone-id 174429bd-2fe0-4ec4-9ef9-5f7bb413f931
26 [ Jul 28 03:11:34 Method "start" exited with status 0. ]
27 note: configured to log to "/dev/stdout"
282026-07-28T03:11:34.457ZINFOcockroach-admin (dropshot): listening file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:335 local_addr = [fd00:1122:3344:101::3]:32222
292026-07-28T03:11:44.944ZINFOcockroach-admin (dropshot): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:1057 local_addr = [fd00:1122:3344:101::3]:32222 remote_addr = [fd00:1122:3344:101::1]:57210
302026-07-28T03:11:44.956ZINFOcockroach-admin (ServerContext): Initializing CRDB cluster file = cockroach-admin/src/http_entrypoints.rs:38
312026-07-28T03:11:45.298ZINFOcockroach-admin (ServerContext): CRDB cluster initialized - initializing Omicron schema file = cockroach-admin/src/http_entrypoints.rs:40
322026-07-28T03:11:50.102ZINFOcockroach-admin (dropshot): request completed error_message_external = Command [sql --file /opt/oxide/cockroachdb/sql/dbinit.sql --host [fd00:1122:3344:101::3]:32221 --insecure] executed and failed with status: exit status: 1 stdout: SET CLUSTER SETTING\nBEGIN\nCREATE DATABASE\nCREATE ROLE\nALTER DEFAULT PRIVILEGES\nCONFIGURE ZONE 1\nCREATE TYPE\nCREATE TABLE\nCREATE TYPE\nCREATE TABLE\nINSERT 1\nCREATE TABLE\nCREATE TYPE\nCREATE TYPE\nCREATE TYPE\nCREATE TABLE\nCREATE INDEX\nCREATE INDEX\nCREATE INDEX\nCREATE TYPE\nCREATE TABLE\nCREATE INDEX\nCREATE INDEX\nCREATE INDEX\nCREATE TABLE\nCREATE INDEX\nCREATE TABLE\nCREATE INDEX\nCREATE TYPE\n stderr: ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionAbortedError(ABORT_REASON_ABORT_SPAN): "sql txn" meta={id=d06eadfb key=/NamespaceTable/30/1/104/0/"public"/4/1 pri=0.05038106 epo=0 ts=1785208309.659678951,1 min=1785208309.185293137,0 seq=235} lock=true stat=ABORTED rts=1785208309.185293137,0 wto=false gul=1785208309.685293137,0\nSQLSTATE: 40001\nHINT: See: https://www.cockroachlabs.com/docs/v22.1/transaction-retry-error-reference.html#abort_reason_abort_span\nERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionAbortedError(ABORT_REASON_ABORT_SPAN): "sql txn" meta={id=d06eadfb key=/NamespaceTable/30/1/104/0/"public"/4/1 pri=0.05038106 epo=0 ts=1785208309.659678951,1 min=1785208309.185293137,0 seq=235} lock=true stat=ABORTED rts=1785208309.185293137,0 wto=false gul=1785208309.685293137,0\nSQLSTATE: 40001\nHINT: See: https://www.cockroachlabs.com/docs/v22.1/transaction-retry-error-reference.html#abort_reason_abort_span\nFailed running "sql"\n error_message_internal = Command [sql --file /opt/oxide/cockroachdb/sql/dbinit.sql --host [fd00:1122:3344:101::3]:32221 --insecure] executed and failed with status: exit status: 1 stdout: SET CLUSTER SETTING\nBEGIN\nCREATE DATABASE\nCREATE ROLE\nALTER DEFAULT PRIVILEGES\nCONFIGURE ZONE 1\nCREATE TYPE\nCREATE TABLE\nCREATE TYPE\nCREATE TABLE\nINSERT 1\nCREATE TABLE\nCREATE TYPE\nCREATE TYPE\nCREATE TYPE\nCREATE TABLE\nCREATE INDEX\nCREATE INDEX\nCREATE INDEX\nCREATE TYPE\nCREATE TABLE\nCREATE INDEX\nCREATE INDEX\nCREATE INDEX\nCREATE TABLE\nCREATE INDEX\nCREATE TABLE\nCREATE INDEX\nCREATE TYPE\n stderr: ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionAbortedError(ABORT_REASON_ABORT_SPAN): "sql txn" meta={id=d06eadfb key=/NamespaceTable/30/1/104/0/"public"/4/1 pri=0.05038106 epo=0 ts=1785208309.659678951,1 min=1785208309.185293137,0 seq=235} lock=true stat=ABORTED rts=1785208309.185293137,0 wto=false gul=1785208309.685293137,0\nSQLSTATE: 40001\nHINT: See: https://www.cockroachlabs.com/docs/v22.1/transaction-retry-error-reference.html#abort_reason_abort_span\nERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionAbortedError(ABORT_REASON_ABORT_SPAN): "sql txn" meta={id=d06eadfb key=/NamespaceTable/30/1/104/0/"public"/4/1 pri=0.05038106 epo=0 ts=1785208309.659678951,1 min=1785208309.185293137,0 seq=235} lock=true stat=ABORTED rts=1785208309.185293137,0 wto=false gul=1785208309.685293137,0\nSQLSTATE: 40001\nHINT: See: https://www.cockroachlabs.com/docs/v22.1/transaction-retry-error-reference.html#abort_reason_abort_span\nFailed running "sql"\n file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:862 latency_us = 5150424 local_addr = [fd00:1122:3344:101::3]:32222 method = POST remote_addr = [fd00:1122:3344:101::1]:57210 req_id = 2595a0c0-1f06-4f55-a42a-5360ca58e76c response_code = 500 uri = /cluster/init
332026-07-28T03:11:50.191ZINFOcockroach-admin (ServerContext): Initializing CRDB cluster file = cockroach-admin/src/http_entrypoints.rs:38
342026-07-28T03:11:50.325ZINFOcockroach-admin (ServerContext): CRDB cluster initialized - initializing Omicron schema file = cockroach-admin/src/http_entrypoints.rs:40
352026-07-28T03:12:08.754ZINFOcockroach-admin (ServerContext): Omicron schema initialized file = cockroach-admin/src/http_entrypoints.rs:42
362026-07-28T03:12:08.754ZINFOcockroach-admin (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874 latency_us = 18563569 local_addr = [fd00:1122:3344:101::3]:32222 method = POST remote_addr = [fd00:1122:3344:101::1]:57210 req_id = c616aee2-8a31-40b3-b118-345a495ad6c4 response_code = 204 uri = /cluster/init
372026-07-28T03:13:02.008ZINFOcockroach-admin (dropshot): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:1057 local_addr = [fd00:1122:3344:101::3]:32222 remote_addr = [fd00:1122:3344:101::b]:33964
382026-07-28T03:13:02.046ZINFOcockroach-admin (dropshot): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:1057 local_addr = [fd00:1122:3344:101::3]:32222 remote_addr = [fd00:1122:3344:101::c]:55940
392026-07-28T03:13:02.172ZINFOcockroach-admin (dropshot): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:1057 local_addr = [fd00:1122:3344:101::3]:32222 remote_addr = [fd00:1122:3344:101::a]:37431
402026-07-28T03:13:03.013ZINFOcockroach-admin (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874 latency_us = 993383 local_addr = [fd00:1122:3344:101::3]:32222 method = GET remote_addr = [fd00:1122:3344:101::b]:33964 req_id = b8d5a778-c481-490f-a675-82ba33f4673e response_code = 200 uri = /node/id
412026-07-28T03:13:03.013ZINFOcockroach-admin (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874 latency_us = 840519 local_addr = [fd00:1122:3344:101::3]:32222 method = GET remote_addr = [fd00:1122:3344:101::a]:37431 req_id = b1079220-b6cb-4a30-8964-4f5e75359ffd response_code = 200 uri = /node/id
422026-07-28T03:13:03.013ZINFOcockroach-admin (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874 latency_us = 966522 local_addr = [fd00:1122:3344:101::3]:32222 method = GET remote_addr = [fd00:1122:3344:101::c]:55940 req_id = 9d56000d-0825-42ff-88ea-375c3d454af4 response_code = 200 uri = /node/id
432026-07-28T03:13:03.092ZINFOcockroach-admin (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874 latency_us = 78378 local_addr = [fd00:1122:3344:101::3]:32222 method = GET remote_addr = [fd00:1122:3344:101::b]:33964 req_id = c34dce59-71e5-4277-a767-a2f195d8946c response_code = 200 uri = /proxy/status/vars
442026-07-28T03:13:03.092ZINFOcockroach-admin (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874 latency_us = 78542 local_addr = [fd00:1122:3344:101::3]:32222 method = GET remote_addr = [fd00:1122:3344:101::c]:55940 req_id = e4f9a69f-a834-46bd-a149-88c5801e41ed response_code = 200 uri = /proxy/status/vars
452026-07-28T03:13:03.096ZINFOcockroach-admin (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874 latency_us = 82753 local_addr = [fd00:1122:3344:101::3]:32222 method = GET remote_addr = [fd00:1122:3344:101::a]:37431 req_id = 82b79285-4d26-4882-907a-e27305260b9c response_code = 200 uri = /proxy/status/vars
462026-07-28T03:13:03.175ZINFOcockroach-admin (dropshot): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:1057 local_addr = [fd00:1122:3344:101::3]:32222 remote_addr = [fd00:1122:3344:101::a]:41390
472026-07-28T03:13:03.177ZINFOcockroach-admin (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874 latency_us = 75 local_addr = [fd00:1122:3344:101::3]:32222 method = GET remote_addr = [fd00:1122:3344:101::a]:41390 req_id = f78e2d40-88cc-437f-8ef8-e8159a580c5d response_code = 200 uri = /node/id
482026-07-28T03:13:04.365ZINFOcockroach-admin (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874 latency_us = 62 local_addr = [fd00:1122:3344:101::3]:32222 method = GET remote_addr = [fd00:1122:3344:101::b]:33964 req_id = 6248c0ce-2193-4f72-ac96-5bcdf6688db6 response_code = 200 uri = /node/id
492026-07-28T03:13:04.374ZINFOcockroach-admin (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874 latency_us = 8555 local_addr = [fd00:1122:3344:101::3]:32222 method = GET remote_addr = [fd00:1122:3344:101::b]:33964 req_id = b55693f7-1a0c-44a2-8e4a-1a1870aee335 response_code = 200 uri = /proxy/status/vars
502026-07-28T03:13:04.479ZINFOcockroach-admin (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874 latency_us = 76 local_addr = [fd00:1122:3344:101::3]:32222 method = GET remote_addr = [fd00:1122:3344:101::a]:37431 req_id = bb4e6f38-5c16-4b5e-8b4f-62a8cd6c10dc response_code = 200 uri = /node/id
512026-07-28T03:13:04.496ZINFOcockroach-admin (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874 latency_us = 16065 local_addr = [fd00:1122:3344:101::3]:32222 method = GET remote_addr = [fd00:1122:3344:101::a]:37431 req_id = 66892180-77f6-4aa9-a708-99a878af620c response_code = 200 uri = /proxy/status/vars
522026-07-28T03:13:05.585ZINFOcockroach-admin (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874 latency_us = 63 local_addr = [fd00:1122:3344:101::3]:32222 method = GET remote_addr = [fd00:1122:3344:101::a]:37431 req_id = 2b73b099-938e-4af6-b166-9a0b3a303702 response_code = 200 uri = /node/id
532026-07-28T03:13:05.594ZINFOcockroach-admin (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874 latency_us = 8101 local_addr = [fd00:1122:3344:101::3]:32222 method = GET remote_addr = [fd00:1122:3344:101::a]:37431 req_id = ab78c5c3-351d-495f-9dee-d4fa907c7112 response_code = 200 uri = /proxy/status/vars
542026-07-28T03:13:12.490ZINFOcockroach-admin (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874 latency_us = 62 local_addr = [fd00:1122:3344:101::3]:32222 method = GET remote_addr = [fd00:1122:3344:101::b]:33964 req_id = 42cc0bbe-c83a-49f2-a3ae-c0494a4335ed response_code = 200 uri = /node/id
552026-07-28T03:13:12.498ZINFOcockroach-admin (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874 latency_us = 7731 local_addr = [fd00:1122:3344:101::3]:32222 method = GET remote_addr = [fd00:1122:3344:101::b]:33964 req_id = 8d4719f6-a135-4c62-9447-ff28ba1894f5 response_code = 200 uri = /proxy/status/vars
562026-07-28T03:13:12.556ZINFOcockroach-admin (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874 latency_us = 56 local_addr = [fd00:1122:3344:101::3]:32222 method = GET remote_addr = [fd00:1122:3344:101::c]:55940 req_id = e3386fb4-3ddc-4b98-a88a-58146068f13e response_code = 200 uri = /node/id
572026-07-28T03:13:12.568ZINFOcockroach-admin (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874 latency_us = 12475 local_addr = [fd00:1122:3344:101::3]:32222 method = GET remote_addr = [fd00:1122:3344:101::c]:55940 req_id = 8c51483b-afb9-4e46-bd30-2813a9b218c6 response_code = 200 uri = /proxy/status/vars
582026-07-28T03:14:03.220ZINFOcockroach-admin (dropshot): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:1057 local_addr = [fd00:1122:3344:101::3]:32222 remote_addr = [fd00:1122:3344:101::c]:57504
592026-07-28T03:14:03.220ZINFOcockroach-admin (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874 latency_us = 60 local_addr = [fd00:1122:3344:101::3]:32222 method = GET remote_addr = [fd00:1122:3344:101::c]:57504 req_id = c3056cab-1df8-46c1-a17b-edfe0b5a8860 response_code = 200 uri = /node/id
602026-07-28T03:14:03.229ZINFOcockroach-admin (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874 latency_us = 8267 local_addr = [fd00:1122:3344:101::3]:32222 method = GET remote_addr = [fd00:1122:3344:101::c]:57504 req_id = b8fbe8f5-52a8-415b-892f-9c9d6d12be9b response_code = 200 uri = /proxy/status/vars
612026-07-28T03:14:03.685ZINFOcockroach-admin (dropshot): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:1057 local_addr = [fd00:1122:3344:101::3]:32222 remote_addr = [fd00:1122:3344:101::b]:62700
622026-07-28T03:14:03.685ZINFOcockroach-admin (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874 latency_us = 51 local_addr = [fd00:1122:3344:101::3]:32222 method = GET remote_addr = [fd00:1122:3344:101::b]:62700 req_id = 53854d09-3b62-4230-94d7-ad08d1c98f56 response_code = 200 uri = /node/id
632026-07-28T03:14:03.692ZINFOcockroach-admin (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874 latency_us = 7444 local_addr = [fd00:1122:3344:101::3]:32222 method = GET remote_addr = [fd00:1122:3344:101::b]:62700 req_id = 0c5e07b6-36df-4390-9f61-c601a1ee6117 response_code = 200 uri = /proxy/status/vars
642026-07-28T03:14:04.230ZINFOcockroach-admin (dropshot): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:1057 local_addr = [fd00:1122:3344:101::3]:32222 remote_addr = [fd00:1122:3344:101::a]:52863
652026-07-28T03:14:04.230ZINFOcockroach-admin (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874 latency_us = 68 local_addr = [fd00:1122:3344:101::3]:32222 method = GET remote_addr = [fd00:1122:3344:101::a]:52863 req_id = 21312191-d582-443f-86b2-fce82d21fbe6 response_code = 200 uri = /node/id
662026-07-28T03:14:04.237ZINFOcockroach-admin (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874 latency_us = 6663 local_addr = [fd00:1122:3344:101::3]:32222 method = GET remote_addr = [fd00:1122:3344:101::a]:52863 req_id = 265608d9-6bd5-4498-b282-5c7d7da9003e response_code = 200 uri = /proxy/status/vars