Max level shown:
1 [ Jan 31 21:19:41 Enabled. ]
2 [ Jan 31 21:19:41 Rereading configuration. ]
3 [ Jan 31 21:19:43 Rereading configuration. ]
4 [ Jan 31 21:19:46 Executing start method ("/opt/oxide/lib/svc/manifest/crucible/agent.sh"). ]
5 + . /lib/svc/share/smf_include.sh
6 ++ SMF_EXIT_OK=0
7 ++ SMF_EXIT_NODAEMON=94
8 ++ SMF_EXIT_ERR_FATAL=95
9 ++ SMF_EXIT_ERR_CONFIG=96
10 ++ SMF_EXIT_MON_DEGRADE=97
11 ++ SMF_EXIT_MON_OFFLINE=98
12 ++ SMF_EXIT_ERR_NOSMF=99
13 ++ SMF_EXIT_ERR_PERM=100
14 ++ svcprop -c -p config/datalink svc:/oxide/crucible/agent:default
15 + DATALINK=oxControlService19
16 ++ svcprop -c -p config/gateway svc:/oxide/crucible/agent:default
17 + GATEWAY=fd00:1122:3344:101::1
18 ++ svcprop -c -p config/dataset svc:/oxide/crucible/agent:default
19 + DATASET=oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b/crucible
20 ++ svcprop -c -p config/listen_addr svc:/oxide/crucible/agent:default
21 + LISTEN_ADDR=fd00:1122:3344:101::18
22 ++ svcprop -c -p config/listen_port svc:/oxide/crucible/agent:default
23 + LISTEN_PORT=32345
24 ++ svcprop -c -p config/portbase svc:/oxide/crucible/agent:default
25 + PORTBASE=19000
26 ++ svcprop -c -p config/downstairs_prefix svc:/oxide/crucible/agent:default
27 + DOWNSTAIRS_PREFIX=downstairs
28 ++ svcprop -c -p config/snapshot_prefix svc:/oxide/crucible/agent:default
29 + SNAPSHOT_PREFIX=snapshot
30 + [[ oxControlService19 == unknown ]]
31 + [[ fd00:1122:3344:101::1 == unknown ]]
32 + ipadm delete-if oxControlService19
33 ipadm: Could not delete oxControlService19: Interface does not exist
34 + true
35 + ipadm create-if -t oxControlService19
36 + ipadm set-ifprop -t -p mtu=9000 -m ipv4 oxControlService19
37 + ipadm set-ifprop -t -p mtu=9000 -m ipv6 oxControlService19
38 + ipadm show-addr oxControlService19/ll
39 ipadm: Address object not found
40 + ipadm create-addr -t -T addrconf oxControlService19/ll
41 + ipadm show-addr oxControlService19/omicron6
42 ipadm: Address object not found
43 + ipadm create-addr -t -T static -a fd00:1122:3344:101::18 oxControlService19/omicron6
44 + route get -inet6 default -inet6 fd00:1122:3344:101::1
45 default: not in table
46 + route add -inet6 default -inet6 fd00:1122:3344:101::1
47 add net default: gateway fd00:1122:3344:101::1
48 + args=('-D' '/opt/oxide/crucible/bin/crucible-downstairs' '--dataset' "$DATASET" '-l' "[$LISTEN_ADDR]:$LISTEN_PORT" '-P' "$PORTBASE" '-p' "$DOWNSTAIRS_PREFIX" '-s' "$SNAPSHOT_PREFIX")
49 + exec /opt/oxide/crucible/bin/crucible-agent run -D /opt/oxide/crucible/bin/crucible-downstairs --dataset oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b/crucible -l '[fd00:1122:3344:101::18]:32345' -P 19000 -p downstairs -s snapshot
50 note: configured to log to "/dev/stdout"
512024-01-31T21:19:47.227ZINFOcrucible-agent: dataset: "oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b/crucible"
522024-01-31T21:19:47.245ZINFOcrucible-agent: listen IP: [fd00:1122:3344:101::18]:32345
532024-01-31T21:19:47.245ZINFOcrucible-agent: SMF instance name downstairs_prefix: "downstairs"
542024-01-31T21:19:47.472ZINFOcrucible-agent (dropshot): listening local_addr = [fd00:1122:3344:101::18]:32345
552024-01-31T21:21:10.890ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::18]:32345 remote_addr = [fd00:1122:3344:101::b]:43266
562024-01-31T21:21:10.926ZINFOcrucible-agent (datafile): region 20952443-a437-4525-ac69-361ffe9946b8 state: Requested
572024-01-31T21:21:10.926ZINFOcrucible-agent (dropshot): request completed latency_us = 2650 local_addr = [fd00:1122:3344:101::18]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:43266 req_id = b4769bbe-c657-4bfc-8c86-6f56d9d1ed09 response_code = 200 uri = /crucible/0/regions
582024-01-31T21:21:10.931ZINFOcrucible-agent (worker): Region size:1073741824 reservation:1342177280 quota:3221225472
592024-01-31T21:21:10.940ZINFOcrucible-agent (worker): zfs set reservation of 1342177280 for oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b/crucible/regions/20952443-a437-4525-ac69-361ffe9946b8
602024-01-31T21:21:10.940ZINFOcrucible-agent (worker): zfs set quota of 3221225472 for oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b/crucible/regions/20952443-a437-4525-ac69-361ffe9946b8
612024-01-31T21:21:10.998ZINFOcrucible-agent (worker): creating region Region { id: RegionId("20952443-a437-4525-ac69-361ffe9946b8"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 16, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None } at "/data/regions/20952443-a437-4525-ac69-361ffe9946b8" region = 20952443-a437-4525-ac69-361ffe9946b8
622024-01-31T21:21:11.237ZINFOcrucible-agent (worker): region files created ok region = 20952443-a437-4525-ac69-361ffe9946b8
632024-01-31T21:21:11.237ZINFOcrucible-agent (datafile): region 20952443-a437-4525-ac69-361ffe9946b8 state: Requested -> Created
642024-01-31T21:21:11.237ZINFOcrucible-agent (worker): applying SMF actions post create...
652024-01-31T21:21:11.238ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-20952443-a437-4525-ac69-361ffe9946b8
662024-01-31T21:21:11.241ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-20952443-a437-4525-ac69-361ffe9946b8
672024-01-31T21:21:11.248ZINFOcrucible-agent (worker): creating config property group
682024-01-31T21:21:11.251ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-20952443-a437-4525-ac69-361ffe9946b8
692024-01-31T21:21:11.252ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/20952443-a437-4525-ac69-361ffe9946b8
702024-01-31T21:21:11.252ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
712024-01-31T21:21:11.252ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::18
722024-01-31T21:21:11.252ZINFOcrucible-agent (worker): commit
732024-01-31T21:21:11.257ZINFOcrucible-agent (worker): ok!
742024-01-31T21:21:11.262ZINFOcrucible-agent (worker): SMF ok!
752024-01-31T21:21:11.288ZINFOcrucible-agent (dropshot): request completed latency_us = 79 local_addr = [fd00:1122:3344:101::18]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:43266 req_id = c20e82ad-aae6-42ee-8f97-baa2e1659f59 response_code = 200 uri = /crucible/0/regions
762024-01-31T21:21:11.904ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::18]:32345 remote_addr = [fd00:1122:3344:101::b]:61173
772024-01-31T21:21:11.904ZINFOcrucible-agent (dropshot): request completed latency_us = 70 local_addr = [fd00:1122:3344:101::18]:32345 method = GET remote_addr = [fd00:1122:3344:101::b]:61173 req_id = 948d9165-7f88-4e5f-928d-28a74c13bb4d response_code = 200 uri = /crucible/0/regions/20952443-a437-4525-ac69-361ffe9946b8
782024-01-31T21:21:11.904ZINFOcrucible-agent (datafile): path is "/data/regions/20952443-a437-4525-ac69-361ffe9946b8"
792024-01-31T21:21:11.913ZINFOcrucible-agent (datafile): dataset is oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b/crucible/regions/20952443-a437-4525-ac69-361ffe9946b8
802024-01-31T21:21:11.921ZINFOcrucible-agent (datafile): region 20952443-a437-4525-ac69-361ffe9946b8 state: Created -> Tombstoned
812024-01-31T21:21:11.921ZINFOcrucible-agent (worker): applying SMF actions before removal...
822024-01-31T21:21:11.921ZINFOcrucible-agent (dropshot): request completed latency_us = 17401 local_addr = [fd00:1122:3344:101::18]:32345 method = DELETE remote_addr = [fd00:1122:3344:101::b]:61173 req_id = 6698dc1c-2a7f-4f86-8781-92c39220dfa9 response_code = 204 uri = /crucible/0/regions/20952443-a437-4525-ac69-361ffe9946b8
832024-01-31T21:21:11.922ZINFOcrucible-agent (dropshot): request completed latency_us = 22 local_addr = [fd00:1122:3344:101::18]:32345 method = GET remote_addr = [fd00:1122:3344:101::b]:61173 req_id = 4fc2abfc-d79f-44fd-a1f1-b1a48dedf3df response_code = 200 uri = /crucible/0/regions/20952443-a437-4525-ac69-361ffe9946b8
842024-01-31T21:21:11.923ZINFOcrucible-agent (worker): disabling downstairs instance: downstairs-20952443-a437-4525-ac69-361ffe9946b8 (instance states: (Some(Online), None))
852024-01-31T21:21:11.926ZINFOcrucible-agent (worker): SMF ok!
862024-01-31T21:21:11.937ZINFOcrucible-agent (worker): deleting zfs dataset "oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b/crucible/regions/20952443-a437-4525-ac69-361ffe9946b8" region = 20952443-a437-4525-ac69-361ffe9946b8
872024-01-31T21:21:11.992ZINFOcrucible-agent (datafile): region 20952443-a437-4525-ac69-361ffe9946b8 state: Tombstoned -> Destroyed
882024-01-31T21:21:12.041ZINFOcrucible-agent (dropshot): request completed latency_us = 70 local_addr = [fd00:1122:3344:101::18]:32345 method = GET remote_addr = [fd00:1122:3344:101::b]:61173 req_id = 7ce0e62d-1cfa-494c-bf1b-e81cd4b40a0e response_code = 200 uri = /crucible/0/regions/20952443-a437-4525-ac69-361ffe9946b8
892024-01-31T21:26:57.577ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::18]:32345 remote_addr = [fd00:1122:3344:101::a]:60193
902024-01-31T21:26:57.577ZINFOcrucible-agent (datafile): region 32584863-4e48-4a15-ab20-9c922cbe24b4 state: Requested
912024-01-31T21:26:57.587ZINFOcrucible-agent (dropshot): request completed latency_us = 10103 local_addr = [fd00:1122:3344:101::18]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:60193 req_id = 25b69ab1-c0cb-4757-a1f3-68a15d50d740 response_code = 200 uri = /crucible/0/regions
922024-01-31T21:26:57.587ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
932024-01-31T21:26:57.599ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b/crucible/regions/32584863-4e48-4a15-ab20-9c922cbe24b4
942024-01-31T21:26:57.599ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b/crucible/regions/32584863-4e48-4a15-ab20-9c922cbe24b4
952024-01-31T21:26:57.666ZINFOcrucible-agent (worker): creating region Region { id: RegionId("32584863-4e48-4a15-ab20-9c922cbe24b4"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None } at "/data/regions/32584863-4e48-4a15-ab20-9c922cbe24b4" region = 32584863-4e48-4a15-ab20-9c922cbe24b4
962024-01-31T21:26:57.842ZINFOcrucible-agent (dropshot): request completed latency_us = 90 local_addr = [fd00:1122:3344:101::18]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:60193 req_id = 94c8aee3-d0ef-45c3-9de3-6ec6f823c07e response_code = 200 uri = /crucible/0/regions
972024-01-31T21:26:58.125ZINFOcrucible-agent (worker): region files created ok region = 32584863-4e48-4a15-ab20-9c922cbe24b4
982024-01-31T21:26:58.125ZINFOcrucible-agent (datafile): region 32584863-4e48-4a15-ab20-9c922cbe24b4 state: Requested -> Created
992024-01-31T21:26:58.126ZINFOcrucible-agent (worker): applying SMF actions post create...
1002024-01-31T21:26:58.127ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-32584863-4e48-4a15-ab20-9c922cbe24b4
1012024-01-31T21:26:58.135ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-32584863-4e48-4a15-ab20-9c922cbe24b4
1022024-01-31T21:26:58.169ZINFOcrucible-agent (worker): creating config property group
1032024-01-31T21:26:58.183ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-32584863-4e48-4a15-ab20-9c922cbe24b4
1042024-01-31T21:26:58.183ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/32584863-4e48-4a15-ab20-9c922cbe24b4
1052024-01-31T21:26:58.184ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
1062024-01-31T21:26:58.184ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::18
1072024-01-31T21:26:58.184ZINFOcrucible-agent (worker): commit
1082024-01-31T21:26:58.191ZINFOcrucible-agent (worker): ok!
1092024-01-31T21:26:58.200ZINFOcrucible-agent (worker): SMF ok!
1102024-01-31T21:26:58.476ZINFOcrucible-agent (dropshot): request completed latency_us = 58 local_addr = [fd00:1122:3344:101::18]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:60193 req_id = d54c0689-2ad2-4084-a678-d8fa74eb9432 response_code = 200 uri = /crucible/0/regions
1112024-01-31T21:27:04.390ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::18]:32345 remote_addr = [fd00:1122:3344:101::a]:41039
1122024-01-31T21:27:04.452ZINFOcrucible-agent (datafile): region f6af4d9f-b4e8-463a-8610-46ddfd398696 state: Requested
1132024-01-31T21:27:04.452ZINFOcrucible-agent (dropshot): request completed latency_us = 271 local_addr = [fd00:1122:3344:101::18]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:41039 req_id = f1f9d020-3900-44fd-91a7-3c665d662974 response_code = 200 uri = /crucible/0/regions
1142024-01-31T21:27:04.452ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
1152024-01-31T21:27:04.460ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b/crucible/regions/f6af4d9f-b4e8-463a-8610-46ddfd398696
1162024-01-31T21:27:04.460ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b/crucible/regions/f6af4d9f-b4e8-463a-8610-46ddfd398696
1172024-01-31T21:27:04.564ZINFOcrucible-agent (worker): creating region Region { id: RegionId("f6af4d9f-b4e8-463a-8610-46ddfd398696"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19001, cert_pem: None, key_pem: None, root_pem: None } at "/data/regions/f6af4d9f-b4e8-463a-8610-46ddfd398696" region = f6af4d9f-b4e8-463a-8610-46ddfd398696
1182024-01-31T21:27:04.777ZINFOcrucible-agent (dropshot): request completed latency_us = 680 local_addr = [fd00:1122:3344:101::18]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:41039 req_id = 47cdd2ef-bd3a-4cb8-afcb-4e7b98b19227 response_code = 200 uri = /crucible/0/regions
1192024-01-31T21:27:04.821ZINFOcrucible-agent (worker): region files created ok region = f6af4d9f-b4e8-463a-8610-46ddfd398696
1202024-01-31T21:27:04.821ZINFOcrucible-agent (datafile): region f6af4d9f-b4e8-463a-8610-46ddfd398696 state: Requested -> Created
1212024-01-31T21:27:04.821ZINFOcrucible-agent (worker): applying SMF actions post create...
1222024-01-31T21:27:04.823ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-f6af4d9f-b4e8-463a-8610-46ddfd398696
1232024-01-31T21:27:04.834ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-f6af4d9f-b4e8-463a-8610-46ddfd398696
1242024-01-31T21:27:04.860ZINFOcrucible-agent (worker): creating config property group
1252024-01-31T21:27:04.883ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-f6af4d9f-b4e8-463a-8610-46ddfd398696
1262024-01-31T21:27:04.883ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/f6af4d9f-b4e8-463a-8610-46ddfd398696
1272024-01-31T21:27:04.883ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
1282024-01-31T21:27:04.883ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::18
1292024-01-31T21:27:04.883ZINFOcrucible-agent (worker): commit
1302024-01-31T21:27:04.894ZINFOcrucible-agent (worker): ok!
1312024-01-31T21:27:04.902ZINFOcrucible-agent (worker): SMF ok!
1322024-01-31T21:27:05.316ZINFOcrucible-agent (dropshot): request completed latency_us = 69 local_addr = [fd00:1122:3344:101::18]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:41039 req_id = 76ed19d5-666b-4f55-870a-125a27965bbc response_code = 200 uri = /crucible/0/regions
1332024-01-31T21:27:38.732ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::18]:32345 remote_addr = [fd00:1122:3344:101::a]:46887
1342024-01-31T21:27:38.733ZINFOcrucible-agent (dropshot): request completed latency_us = 58 local_addr = [fd00:1122:3344:101::18]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:46887 req_id = af5775c3-e70a-4d70-b6fc-916fd97428cb response_code = 200 uri = /crucible/0/regions/f6af4d9f-b4e8-463a-8610-46ddfd398696
1352024-01-31T21:27:38.733ZINFOcrucible-agent (datafile): path is "/data/regions/f6af4d9f-b4e8-463a-8610-46ddfd398696"
1362024-01-31T21:27:38.746ZINFOcrucible-agent (datafile): dataset is oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b/crucible/regions/f6af4d9f-b4e8-463a-8610-46ddfd398696
1372024-01-31T21:27:38.754ZINFOcrucible-agent (datafile): region f6af4d9f-b4e8-463a-8610-46ddfd398696 state: Created -> Tombstoned
1382024-01-31T21:27:38.754ZINFOcrucible-agent (worker): applying SMF actions before removal...
1392024-01-31T21:27:38.754ZINFOcrucible-agent (dropshot): request completed latency_us = 21776 local_addr = [fd00:1122:3344:101::18]:32345 method = DELETE remote_addr = [fd00:1122:3344:101::a]:46887 req_id = 237541f3-a497-40d6-8df3-077a9cd31a43 response_code = 204 uri = /crucible/0/regions/f6af4d9f-b4e8-463a-8610-46ddfd398696
1402024-01-31T21:27:38.755ZINFOcrucible-agent (dropshot): request completed latency_us = 23 local_addr = [fd00:1122:3344:101::18]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:46887 req_id = 69ae6142-8a07-4e11-99b5-02e4d2bb0854 response_code = 200 uri = /crucible/0/regions/f6af4d9f-b4e8-463a-8610-46ddfd398696
1412024-01-31T21:27:38.756ZINFOcrucible-agent (worker): disabling downstairs instance: downstairs-f6af4d9f-b4e8-463a-8610-46ddfd398696 (instance states: (Some(Online), None))
1422024-01-31T21:27:38.770ZINFOcrucible-agent (worker): SMF ok!
1432024-01-31T21:27:38.781ZINFOcrucible-agent (worker): deleting zfs dataset "oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b/crucible/regions/f6af4d9f-b4e8-463a-8610-46ddfd398696" region = f6af4d9f-b4e8-463a-8610-46ddfd398696
1442024-01-31T21:27:38.842ZINFOcrucible-agent (datafile): region f6af4d9f-b4e8-463a-8610-46ddfd398696 state: Tombstoned -> Destroyed
1452024-01-31T21:27:38.889ZINFOcrucible-agent (dropshot): request completed latency_us = 96 local_addr = [fd00:1122:3344:101::18]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:46887 req_id = 2d864acb-493e-4187-9877-8911488cdb2f response_code = 200 uri = /crucible/0/regions/f6af4d9f-b4e8-463a-8610-46ddfd398696