Max level shown:
1 [ Jan 25 04:48:23 Enabled. ]
2 [ Jan 25 04:48:23 Rereading configuration. ]
3 [ Jan 25 04:48:25 Rereading configuration. ]
4 [ Jan 25 04:48:28 Executing start method ("/opt/oxide/lib/svc/manifest/crucible/agent.sh"). ]
5 + . /lib/svc/share/smf_include.sh
6 ++ SMF_EXIT_OK=0
7 ++ SMF_EXIT_NODAEMON=94
8 ++ SMF_EXIT_ERR_FATAL=95
9 ++ SMF_EXIT_ERR_CONFIG=96
10 ++ SMF_EXIT_MON_DEGRADE=97
11 ++ SMF_EXIT_MON_OFFLINE=98
12 ++ SMF_EXIT_ERR_NOSMF=99
13 ++ SMF_EXIT_ERR_PERM=100
14 ++ svcprop -c -p config/datalink svc:/oxide/crucible/agent:default
15 + DATALINK=oxControlService18
16 ++ svcprop -c -p config/gateway svc:/oxide/crucible/agent:default
17 + GATEWAY=fd00:1122:3344:101::1
18 ++ svcprop -c -p config/dataset svc:/oxide/crucible/agent:default
19 + DATASET=oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible
20 ++ svcprop -c -p config/listen_addr svc:/oxide/crucible/agent:default
21 + LISTEN_ADDR=fd00:1122:3344:101::13
22 ++ svcprop -c -p config/listen_port svc:/oxide/crucible/agent:default
23 + LISTEN_PORT=32345
24 ++ svcprop -c -p config/portbase svc:/oxide/crucible/agent:default
25 + PORTBASE=19000
26 ++ svcprop -c -p config/downstairs_prefix svc:/oxide/crucible/agent:default
27 + DOWNSTAIRS_PREFIX=downstairs
28 ++ svcprop -c -p config/snapshot_prefix svc:/oxide/crucible/agent:default
29 + SNAPSHOT_PREFIX=snapshot
30 + [[ oxControlService18 == unknown ]]
31 + [[ fd00:1122:3344:101::1 == unknown ]]
32 + ipadm delete-if oxControlService18
33 ipadm: Could not delete oxControlService18: Interface does not exist
34 + true
35 + ipadm create-if -t oxControlService18
36 + ipadm set-ifprop -t -p mtu=9000 -m ipv4 oxControlService18
37 + ipadm set-ifprop -t -p mtu=9000 -m ipv6 oxControlService18
38 + ipadm show-addr oxControlService18/ll
39 ipadm: Address object not found
40 + ipadm create-addr -t -T addrconf oxControlService18/ll
41 + ipadm show-addr oxControlService18/omicron6
42 ipadm: Address object not found
43 + ipadm create-addr -t -T static -a fd00:1122:3344:101::13 oxControlService18/omicron6
44 + route get -inet6 default -inet6 fd00:1122:3344:101::1
45 default: not in table
46 + route add -inet6 default -inet6 fd00:1122:3344:101::1
47 add net default: gateway fd00:1122:3344:101::1
48 + args=('-D' '/opt/oxide/crucible/bin/crucible-downstairs' '--dataset' "$DATASET" '-l' "[$LISTEN_ADDR]:$LISTEN_PORT" '-P' "$PORTBASE" '-p' "$DOWNSTAIRS_PREFIX" '-s' "$SNAPSHOT_PREFIX")
49 + exec /opt/oxide/crucible/bin/crucible-agent run -D /opt/oxide/crucible/bin/crucible-downstairs --dataset oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible -l '[fd00:1122:3344:101::13]:32345' -P 19000 -p downstairs -s snapshot
50 note: configured to log to "/dev/stdout"
512024-01-25T04:48:29.585ZINFOcrucible-agent: dataset: "oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible"
522024-01-25T04:48:29.588ZINFOcrucible-agent: listen IP: [fd00:1122:3344:101::13]:32345
532024-01-25T04:48:29.588ZINFOcrucible-agent: SMF instance name downstairs_prefix: "downstairs"
542024-01-25T04:48:29.901ZINFOcrucible-agent (dropshot): listening local_addr = [fd00:1122:3344:101::13]:32345
552024-01-25T04:50:07.795ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::13]:32345 remote_addr = [fd00:1122:3344:101::a]:55850
562024-01-25T04:50:07.861ZINFOcrucible-agent (datafile): region 5fbf2bed-d906-4e58-9d18-f9ec1d056b4b state: Requested
572024-01-25T04:50:07.861ZINFOcrucible-agent (dropshot): request completed latency_us = 2802 local_addr = [fd00:1122:3344:101::13]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:55850 req_id = 9827c269-6d37-468f-acae-3beb4364fe98 response_code = 200 uri = /crucible/0/regions
582024-01-25T04:50:07.865ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
592024-01-25T04:50:07.874ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible/regions/5fbf2bed-d906-4e58-9d18-f9ec1d056b4b
602024-01-25T04:50:07.874ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible/regions/5fbf2bed-d906-4e58-9d18-f9ec1d056b4b
612024-01-25T04:50:07.939ZINFOcrucible-agent (worker): creating region Region { id: RegionId("5fbf2bed-d906-4e58-9d18-f9ec1d056b4b"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None } at "/data/regions/5fbf2bed-d906-4e58-9d18-f9ec1d056b4b" region = 5fbf2bed-d906-4e58-9d18-f9ec1d056b4b
622024-01-25T04:50:08.072ZINFOcrucible-agent (dropshot): request completed latency_us = 80 local_addr = [fd00:1122:3344:101::13]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:55850 req_id = f08f5429-eff6-448d-93a7-1a3f2f5617f3 response_code = 200 uri = /crucible/0/regions
632024-01-25T04:50:08.275ZINFOcrucible-agent (worker): region files created ok region = 5fbf2bed-d906-4e58-9d18-f9ec1d056b4b
642024-01-25T04:50:08.275ZINFOcrucible-agent (datafile): region 5fbf2bed-d906-4e58-9d18-f9ec1d056b4b state: Requested -> Created
652024-01-25T04:50:08.276ZINFOcrucible-agent (worker): applying SMF actions post create...
662024-01-25T04:50:08.276ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-5fbf2bed-d906-4e58-9d18-f9ec1d056b4b
672024-01-25T04:50:08.281ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-5fbf2bed-d906-4e58-9d18-f9ec1d056b4b
682024-01-25T04:50:08.291ZINFOcrucible-agent (worker): creating config property group
692024-01-25T04:50:08.294ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-5fbf2bed-d906-4e58-9d18-f9ec1d056b4b
702024-01-25T04:50:08.295ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/5fbf2bed-d906-4e58-9d18-f9ec1d056b4b
712024-01-25T04:50:08.295ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
722024-01-25T04:50:08.295ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::13
732024-01-25T04:50:08.295ZINFOcrucible-agent (worker): commit
742024-01-25T04:50:08.300ZINFOcrucible-agent (worker): ok!
752024-01-25T04:50:08.305ZINFOcrucible-agent (worker): SMF ok!
762024-01-25T04:50:08.539ZINFOcrucible-agent (dropshot): request completed latency_us = 62 local_addr = [fd00:1122:3344:101::13]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:55850 req_id = 160c12a5-afa3-438b-a7a7-8ac936a14e8c response_code = 200 uri = /crucible/0/regions
772024-01-25T04:55:32.341ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::13]:32345 remote_addr = [fd00:1122:3344:101::a]:39533
782024-01-25T04:55:32.342ZINFOcrucible-agent (dropshot): request completed latency_us = 72 local_addr = [fd00:1122:3344:101::13]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:39533 req_id = 7c3bf177-5b41-41ba-90ee-dc0593f46f81 response_code = 200 uri = /crucible/0/regions/5fbf2bed-d906-4e58-9d18-f9ec1d056b4b
792024-01-25T04:55:32.342ZINFOcrucible-agent (datafile): path is "/data/regions/5fbf2bed-d906-4e58-9d18-f9ec1d056b4b"
802024-01-25T04:55:32.354ZINFOcrucible-agent (datafile): dataset is oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible/regions/5fbf2bed-d906-4e58-9d18-f9ec1d056b4b
812024-01-25T04:55:32.364ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible/regions/5fbf2bed-d906-4e58-9d18-f9ec1d056b4b@395a799e-3df0-44f7-a272-b33a4277dcc0
822024-01-25T04:55:32.364ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible/regions/5fbf2bed-d906-4e58-9d18-f9ec1d056b4b", "395a799e-3df0-44f7-a272-b33a4277dcc0"]
832024-01-25T04:55:32.374ZINFOcrucible-agent (dropshot): request completed latency_us = 32096 local_addr = [fd00:1122:3344:101::13]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:39533 req_id = 485ab806-b147-4a67-8e51-a024970d30a7 response_code = 200 uri = /crucible/0/regions/5fbf2bed-d906-4e58-9d18-f9ec1d056b4b/snapshots/395a799e-3df0-44f7-a272-b33a4277dcc0
842024-01-25T04:55:32.375ZINFOcrucible-agent (datafile): path is "/data/regions/5fbf2bed-d906-4e58-9d18-f9ec1d056b4b"
852024-01-25T04:55:32.383ZINFOcrucible-agent (datafile): dataset is oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible/regions/5fbf2bed-d906-4e58-9d18-f9ec1d056b4b
862024-01-25T04:55:32.392ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible/regions/5fbf2bed-d906-4e58-9d18-f9ec1d056b4b@395a799e-3df0-44f7-a272-b33a4277dcc0
872024-01-25T04:55:32.392ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible/regions/5fbf2bed-d906-4e58-9d18-f9ec1d056b4b", "395a799e-3df0-44f7-a272-b33a4277dcc0"]
882024-01-25T04:55:32.401ZINFOcrucible-agent (datafile): requesting running snapshot 5fbf2bed-d906-4e58-9d18-f9ec1d056b4b-395a799e-3df0-44f7-a272-b33a4277dcc0 state: Requested
892024-01-25T04:55:32.401ZINFOcrucible-agent (worker): applying SMF actions for region 5fbf2bed-d906-4e58-9d18-f9ec1d056b4b running snapshot 395a799e-3df0-44f7-a272-b33a4277dcc0 (state Requested)...
902024-01-25T04:55:32.401ZINFOcrucible-agent (dropshot): request completed latency_us = 27412 local_addr = [fd00:1122:3344:101::13]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:39533 req_id = 90be346e-a0f9-4735-ac1a-e0d021534ea0 response_code = 200 uri = /crucible/0/regions/5fbf2bed-d906-4e58-9d18-f9ec1d056b4b/snapshots/395a799e-3df0-44f7-a272-b33a4277dcc0/run
912024-01-25T04:55:32.408ZINFOcrucible-agent (worker): creating missing snapshot instance snapshot-5fbf2bed-d906-4e58-9d18-f9ec1d056b4b-395a799e-3df0-44f7-a272-b33a4277dcc0
922024-01-25T04:55:32.411ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:snapshot-5fbf2bed-d906-4e58-9d18-f9ec1d056b4b-395a799e-3df0-44f7-a272-b33a4277dcc0
932024-01-25T04:55:32.424ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:snapshot-5fbf2bed-d906-4e58-9d18-f9ec1d056b4b-395a799e-3df0-44f7-a272-b33a4277dcc0
942024-01-25T04:55:32.424ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/5fbf2bed-d906-4e58-9d18-f9ec1d056b4b/.zfs/snapshot/395a799e-3df0-44f7-a272-b33a4277dcc0
952024-01-25T04:55:32.424ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
962024-01-25T04:55:32.424ZINFOcrucible-agent (worker): ensure mode SCF_TYPE_ASTRING ro
972024-01-25T04:55:32.424ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::13
982024-01-25T04:55:32.424ZINFOcrucible-agent (worker): commit
992024-01-25T04:55:32.428ZINFOcrucible-agent (worker): ok!
1002024-01-25T04:55:32.433ZINFOcrucible-agent (worker): SMF ok!
1012024-01-25T04:55:32.433ZINFOcrucible-agent (datafile): region 5fbf2bed-d906-4e58-9d18-f9ec1d056b4b running snapshot 395a799e-3df0-44f7-a272-b33a4277dcc0 state: Requested -> Created
1022024-01-25T04:55:35.066ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::13]:32345 remote_addr = [fd00:1122:3344:101::a]:48656
1032024-01-25T04:55:35.067ZINFOcrucible-agent (datafile): region b546d202-3730-4095-8b35-7dc8cd5d5872 state: Requested
1042024-01-25T04:55:35.067ZINFOcrucible-agent (dropshot): request completed latency_us = 805 local_addr = [fd00:1122:3344:101::13]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:48656 req_id = 94209072-47bd-4773-819c-734fdad1b1dd response_code = 200 uri = /crucible/0/regions
1052024-01-25T04:55:35.067ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
1062024-01-25T04:55:35.076ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible/regions/b546d202-3730-4095-8b35-7dc8cd5d5872
1072024-01-25T04:55:35.076ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible/regions/b546d202-3730-4095-8b35-7dc8cd5d5872
1082024-01-25T04:55:35.145ZINFOcrucible-agent (worker): creating region Region { id: RegionId("b546d202-3730-4095-8b35-7dc8cd5d5872"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19002, cert_pem: None, key_pem: None, root_pem: None } at "/data/regions/b546d202-3730-4095-8b35-7dc8cd5d5872" region = b546d202-3730-4095-8b35-7dc8cd5d5872
1092024-01-25T04:55:35.215ZINFOcrucible-agent (worker): region files created ok region = b546d202-3730-4095-8b35-7dc8cd5d5872
1102024-01-25T04:55:35.216ZINFOcrucible-agent (datafile): region b546d202-3730-4095-8b35-7dc8cd5d5872 state: Requested -> Created
1112024-01-25T04:55:35.216ZINFOcrucible-agent (worker): applying SMF actions post create...
1122024-01-25T04:55:35.217ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-b546d202-3730-4095-8b35-7dc8cd5d5872
1132024-01-25T04:55:35.222ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-b546d202-3730-4095-8b35-7dc8cd5d5872
1142024-01-25T04:55:35.230ZINFOcrucible-agent (worker): creating config property group
1152024-01-25T04:55:35.232ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-b546d202-3730-4095-8b35-7dc8cd5d5872
1162024-01-25T04:55:35.232ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/b546d202-3730-4095-8b35-7dc8cd5d5872
1172024-01-25T04:55:35.232ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19002
1182024-01-25T04:55:35.232ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::13
1192024-01-25T04:55:35.233ZINFOcrucible-agent (worker): commit
1202024-01-25T04:55:35.236ZINFOcrucible-agent (worker): ok!
1212024-01-25T04:55:35.241ZINFOcrucible-agent (dropshot): request completed latency_us = 104 local_addr = [fd00:1122:3344:101::13]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:48656 req_id = e5830207-b1b1-4a76-a0c7-e212a56092f9 response_code = 200 uri = /crucible/0/regions
1222024-01-25T04:55:35.242ZINFOcrucible-agent (worker): SMF ok!
1232024-01-25T04:56:05.297ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::13]:32345 remote_addr = [fd00:1122:3344:101::a]:42211
1242024-01-25T04:56:05.297ZINFOcrucible-agent (dropshot): request completed latency_us = 76 local_addr = [fd00:1122:3344:101::13]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:42211 req_id = 8d0e0749-5acc-46b8-8f99-6a873f45552d response_code = 200 uri = /crucible/0/regions/b546d202-3730-4095-8b35-7dc8cd5d5872
1252024-01-25T04:56:05.298ZINFOcrucible-agent (datafile): path is "/data/regions/b546d202-3730-4095-8b35-7dc8cd5d5872"
1262024-01-25T04:56:05.310ZINFOcrucible-agent (datafile): dataset is oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible/regions/b546d202-3730-4095-8b35-7dc8cd5d5872
1272024-01-25T04:56:05.318ZINFOcrucible-agent (datafile): region b546d202-3730-4095-8b35-7dc8cd5d5872 state: Created -> Tombstoned
1282024-01-25T04:56:05.318ZINFOcrucible-agent (worker): applying SMF actions before removal...
1292024-01-25T04:56:05.318ZINFOcrucible-agent (dropshot): request completed latency_us = 20480 local_addr = [fd00:1122:3344:101::13]:32345 method = DELETE remote_addr = [fd00:1122:3344:101::a]:42211 req_id = 132546ae-0450-4506-8bd0-89de21cdfbe9 response_code = 204 uri = /crucible/0/regions/b546d202-3730-4095-8b35-7dc8cd5d5872
1302024-01-25T04:56:05.318ZINFOcrucible-agent (dropshot): request completed latency_us = 36 local_addr = [fd00:1122:3344:101::13]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:42211 req_id = 22ac1a9c-028b-4211-8164-2c468e7b23f9 response_code = 200 uri = /crucible/0/regions/b546d202-3730-4095-8b35-7dc8cd5d5872
1312024-01-25T04:56:05.320ZINFOcrucible-agent (worker): disabling downstairs instance: downstairs-b546d202-3730-4095-8b35-7dc8cd5d5872 (instance states: (Some(Online), None))
1322024-01-25T04:56:05.325ZINFOcrucible-agent (worker): SMF ok!
1332024-01-25T04:56:05.335ZINFOcrucible-agent (worker): deleting zfs dataset "oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible/regions/b546d202-3730-4095-8b35-7dc8cd5d5872" region = b546d202-3730-4095-8b35-7dc8cd5d5872
1342024-01-25T04:56:05.383ZINFOcrucible-agent (datafile): region b546d202-3730-4095-8b35-7dc8cd5d5872 state: Tombstoned -> Destroyed
1352024-01-25T04:56:05.432ZINFOcrucible-agent (dropshot): request completed latency_us = 68 local_addr = [fd00:1122:3344:101::13]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:42211 req_id = 2e25e11a-60f9-400a-b755-c3261cf16a69 response_code = 200 uri = /crucible/0/regions/b546d202-3730-4095-8b35-7dc8cd5d5872