Max level shown:
1 [ Jul 18 22:03:05 Enabled. ]
2 [ Jul 18 22:03:05 Rereading configuration. ]
3 [ Jul 18 22:03:07 Rereading configuration. ]
4 [ Jul 18 22:03:09 Executing start method ("/opt/oxide/crucible/bin/crucible-agent run -D /opt/oxide/crucible/bin/crucible-downstairs --dataset oxp_18924fa5-0339-4445-8677-913f8ef37483/crucible -l [fd00:1122:3344:101::12]:32345 -P 19000 -p downstairs -s snapshot"). ]
5 note: configured to log to "/dev/stdout"
62024-07-18T22:03:09.354ZINFOcrucible-agent: dataset: "oxp_18924fa5-0339-4445-8677-913f8ef37483/crucible"
72024-07-18T22:03:09.356ZINFOcrucible-agent: listen IP: [fd00:1122:3344:101::12]:32345
82024-07-18T22:03:09.356ZINFOcrucible-agent: SMF instance name downstairs_prefix: "downstairs"
92024-07-18T22:03:09.379ZINFOcrucible-agent (datafile): Using conf_path:"/data/crucible.json"
102024-07-18T22:03:09.470ZINFOcrucible-agent (dropshot): listening local_addr = [fd00:1122:3344:101::12]:32345
112024-07-18T22:04:25.435ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::12]:32345 remote_addr = [fd00:1122:3344:101::a]:54187
122024-07-18T22:04:25.454ZINFOcrucible-agent (datafile): region 6cdf6550-5363-4a93-9776-188667afbbd5 state: Requested
132024-07-18T22:04:25.461ZINFOcrucible-agent (dropshot): request completed latency_us = 14870 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:54187 req_id = 2e1869f4-015b-42c1-9cf7-9794dcdd50de response_code = 200 uri = /crucible/0/regions
142024-07-18T22:04:25.461ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
152024-07-18T22:04:25.474ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_18924fa5-0339-4445-8677-913f8ef37483/crucible/regions/6cdf6550-5363-4a93-9776-188667afbbd5
162024-07-18T22:04:25.474ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_18924fa5-0339-4445-8677-913f8ef37483/crucible/regions/6cdf6550-5363-4a93-9776-188667afbbd5
172024-07-18T22:04:25.535ZINFOcrucible-agent (worker): creating region Region { id: RegionId("6cdf6550-5363-4a93-9776-188667afbbd5"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/6cdf6550-5363-4a93-9776-188667afbbd5" region = 6cdf6550-5363-4a93-9776-188667afbbd5
182024-07-18T22:04:25.537ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "6cdf6550-5363-4a93-9776-188667afbbd5" "--data" "/data/regions/6cdf6550-5363-4a93-9776-188667afbbd5" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 6cdf6550-5363-4a93-9776-188667afbbd5
192024-07-18T22:04:25.663ZINFOcrucible-agent (worker): region files created ok region = 6cdf6550-5363-4a93-9776-188667afbbd5
202024-07-18T22:04:25.663ZINFOcrucible-agent (datafile): region 6cdf6550-5363-4a93-9776-188667afbbd5 state: Requested -> Created
212024-07-18T22:04:25.663ZINFOcrucible-agent (worker): applying SMF actions post create...
222024-07-18T22:04:25.664ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-6cdf6550-5363-4a93-9776-188667afbbd5
232024-07-18T22:04:25.672ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-6cdf6550-5363-4a93-9776-188667afbbd5
242024-07-18T22:04:25.689ZINFOcrucible-agent (worker): creating config property group
252024-07-18T22:04:25.692ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-6cdf6550-5363-4a93-9776-188667afbbd5
262024-07-18T22:04:25.692ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/6cdf6550-5363-4a93-9776-188667afbbd5
272024-07-18T22:04:25.697ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
282024-07-18T22:04:25.697ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::12
292024-07-18T22:04:25.697ZINFOcrucible-agent (worker): commit
302024-07-18T22:04:25.703ZINFOcrucible-agent (worker): ok!
312024-07-18T22:04:25.711ZINFOcrucible-agent (worker): SMF ok!
322024-07-18T22:04:25.777ZINFOcrucible-agent (dropshot): request completed latency_us = 642 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:54187 req_id = dec26cd9-1fb0-4ae3-b5e5-d4571671d014 response_code = 200 uri = /crucible/0/regions
332024-07-18T22:09:26.881ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::12]:32345 remote_addr = [fd00:1122:3344:101::a]:35882
342024-07-18T22:09:26.883ZINFOcrucible-agent (dropshot): request completed latency_us = 2302 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:35882 req_id = 38e8139f-d9c5-4991-a3e4-c00bc9060a60 response_code = 200 uri = /crucible/0/regions/6cdf6550-5363-4a93-9776-188667afbbd5
352024-07-18T22:09:26.885ZINFOcrucible-agent (datafile): path is "/data/regions/6cdf6550-5363-4a93-9776-188667afbbd5"
362024-07-18T22:09:26.895ZINFOcrucible-agent (datafile): dataset is oxp_18924fa5-0339-4445-8677-913f8ef37483/crucible/regions/6cdf6550-5363-4a93-9776-188667afbbd5
372024-07-18T22:09:26.905ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_18924fa5-0339-4445-8677-913f8ef37483/crucible/regions/6cdf6550-5363-4a93-9776-188667afbbd5@8ff1b748-9eef-44e1-9536-0ebb429402dc
382024-07-18T22:09:26.905ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_18924fa5-0339-4445-8677-913f8ef37483/crucible/regions/6cdf6550-5363-4a93-9776-188667afbbd5", "8ff1b748-9eef-44e1-9536-0ebb429402dc"]
392024-07-18T22:09:26.916ZINFOcrucible-agent (dropshot): request completed latency_us = 33048 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:35882 req_id = 322f299c-cf4e-4598-88d0-348196b519f1 response_code = 200 uri = /crucible/0/regions/6cdf6550-5363-4a93-9776-188667afbbd5/snapshots/8ff1b748-9eef-44e1-9536-0ebb429402dc
402024-07-18T22:09:26.917ZINFOcrucible-agent (datafile): path is "/data/regions/6cdf6550-5363-4a93-9776-188667afbbd5"
412024-07-18T22:09:26.926ZINFOcrucible-agent (datafile): dataset is oxp_18924fa5-0339-4445-8677-913f8ef37483/crucible/regions/6cdf6550-5363-4a93-9776-188667afbbd5
422024-07-18T22:09:26.934ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_18924fa5-0339-4445-8677-913f8ef37483/crucible/regions/6cdf6550-5363-4a93-9776-188667afbbd5@8ff1b748-9eef-44e1-9536-0ebb429402dc
432024-07-18T22:09:26.934ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_18924fa5-0339-4445-8677-913f8ef37483/crucible/regions/6cdf6550-5363-4a93-9776-188667afbbd5", "8ff1b748-9eef-44e1-9536-0ebb429402dc"]
442024-07-18T22:09:26.943ZINFOcrucible-agent (datafile): requesting running snapshot 6cdf6550-5363-4a93-9776-188667afbbd5-8ff1b748-9eef-44e1-9536-0ebb429402dc state: Requested
452024-07-18T22:09:26.944ZINFOcrucible-agent (worker): applying SMF actions for region 6cdf6550-5363-4a93-9776-188667afbbd5 running snapshot 8ff1b748-9eef-44e1-9536-0ebb429402dc (state Requested)...
462024-07-18T22:09:26.944ZINFOcrucible-agent (dropshot): request completed latency_us = 27302 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:35882 req_id = dc51d53b-0515-42da-a4c7-2a7877529cab response_code = 200 uri = /crucible/0/regions/6cdf6550-5363-4a93-9776-188667afbbd5/snapshots/8ff1b748-9eef-44e1-9536-0ebb429402dc/run
472024-07-18T22:09:26.951ZINFOcrucible-agent (worker): existing directory value /dev/null does not match /data/regions/6cdf6550-5363-4a93-9776-188667afbbd5
482024-07-18T22:09:26.952ZINFOcrucible-agent (worker): existing port value 0 does not match 19000
492024-07-18T22:09:26.952ZINFOcrucible-agent (worker): existing address value 0.0.0.0 does not match fd00:1122:3344:101::12
502024-07-18T22:09:26.952ZINFOcrucible-agent (worker): reconfig required
512024-07-18T22:09:26.952ZINFOcrucible-agent (worker): using existing config property group
522024-07-18T22:09:26.952ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-6cdf6550-5363-4a93-9776-188667afbbd5
532024-07-18T22:09:26.952ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/6cdf6550-5363-4a93-9776-188667afbbd5
542024-07-18T22:09:26.952ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
552024-07-18T22:09:26.952ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::12
562024-07-18T22:09:26.952ZINFOcrucible-agent (worker): commit
572024-07-18T22:09:26.959ZINFOcrucible-agent (worker): ok!
582024-07-18T22:09:26.960ZINFOcrucible-agent (worker): creating missing snapshot instance snapshot-6cdf6550-5363-4a93-9776-188667afbbd5-8ff1b748-9eef-44e1-9536-0ebb429402dc
592024-07-18T22:09:26.966ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:snapshot-6cdf6550-5363-4a93-9776-188667afbbd5-8ff1b748-9eef-44e1-9536-0ebb429402dc
602024-07-18T22:09:26.977ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:snapshot-6cdf6550-5363-4a93-9776-188667afbbd5-8ff1b748-9eef-44e1-9536-0ebb429402dc
612024-07-18T22:09:26.977ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/6cdf6550-5363-4a93-9776-188667afbbd5/.zfs/snapshot/8ff1b748-9eef-44e1-9536-0ebb429402dc
622024-07-18T22:09:26.977ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
632024-07-18T22:09:26.977ZINFOcrucible-agent (worker): ensure mode SCF_TYPE_ASTRING ro
642024-07-18T22:09:26.977ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::12
652024-07-18T22:09:26.977ZINFOcrucible-agent (worker): commit
662024-07-18T22:09:26.981ZINFOcrucible-agent (worker): ok!
672024-07-18T22:09:26.985ZINFOcrucible-agent (worker): SMF ok!
682024-07-18T22:09:26.985ZINFOcrucible-agent (datafile): region 6cdf6550-5363-4a93-9776-188667afbbd5 running snapshot 8ff1b748-9eef-44e1-9536-0ebb429402dc state: Requested -> Created
692024-07-18T22:09:29.945ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::12]:32345 remote_addr = [fd00:1122:3344:101::a]:42354
702024-07-18T22:09:29.945ZINFOcrucible-agent (datafile): region 6af44937-d40d-436d-8294-2377390fa49e state: Requested
712024-07-18T22:09:29.948ZINFOcrucible-agent (dropshot): request completed latency_us = 2377 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:42354 req_id = cfd94ac6-2813-4132-9cec-120b62cd9f2a response_code = 200 uri = /crucible/0/regions
722024-07-18T22:09:29.948ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
732024-07-18T22:09:29.957ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_18924fa5-0339-4445-8677-913f8ef37483/crucible/regions/6af44937-d40d-436d-8294-2377390fa49e
742024-07-18T22:09:29.957ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_18924fa5-0339-4445-8677-913f8ef37483/crucible/regions/6af44937-d40d-436d-8294-2377390fa49e
752024-07-18T22:09:30.020ZINFOcrucible-agent (worker): creating region Region { id: RegionId("6af44937-d40d-436d-8294-2377390fa49e"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19002, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/6af44937-d40d-436d-8294-2377390fa49e" region = 6af44937-d40d-436d-8294-2377390fa49e
762024-07-18T22:09:30.020ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "6af44937-d40d-436d-8294-2377390fa49e" "--data" "/data/regions/6af44937-d40d-436d-8294-2377390fa49e" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 6af44937-d40d-436d-8294-2377390fa49e
772024-07-18T22:09:30.112ZINFOcrucible-agent (worker): region files created ok region = 6af44937-d40d-436d-8294-2377390fa49e
782024-07-18T22:09:30.112ZINFOcrucible-agent (datafile): region 6af44937-d40d-436d-8294-2377390fa49e state: Requested -> Created
792024-07-18T22:09:30.112ZINFOcrucible-agent (worker): applying SMF actions post create...
802024-07-18T22:09:30.112ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-6af44937-d40d-436d-8294-2377390fa49e
812024-07-18T22:09:30.115ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-6af44937-d40d-436d-8294-2377390fa49e
822024-07-18T22:09:30.119ZINFOcrucible-agent (dropshot): request completed latency_us = 713 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:42354 req_id = 17593918-fc95-43d3-a60e-8094cd21832e response_code = 200 uri = /crucible/0/regions
832024-07-18T22:09:30.122ZINFOcrucible-agent (worker): creating config property group
842024-07-18T22:09:30.124ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-6af44937-d40d-436d-8294-2377390fa49e
852024-07-18T22:09:30.124ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/6af44937-d40d-436d-8294-2377390fa49e
862024-07-18T22:09:30.124ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19002
872024-07-18T22:09:30.124ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::12
882024-07-18T22:09:30.124ZINFOcrucible-agent (worker): commit
892024-07-18T22:09:30.128ZINFOcrucible-agent (worker): ok!
902024-07-18T22:09:30.134ZINFOcrucible-agent (worker): existing directory value /dev/null does not match /data/regions/6cdf6550-5363-4a93-9776-188667afbbd5
912024-07-18T22:09:30.134ZINFOcrucible-agent (worker): existing port value 0 does not match 19000
922024-07-18T22:09:30.134ZINFOcrucible-agent (worker): existing address value 0.0.0.0 does not match fd00:1122:3344:101::12
932024-07-18T22:09:30.134ZINFOcrucible-agent (worker): reconfig required
942024-07-18T22:09:30.134ZINFOcrucible-agent (worker): using existing config property group
952024-07-18T22:09:30.134ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-6cdf6550-5363-4a93-9776-188667afbbd5
962024-07-18T22:09:30.139ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/6cdf6550-5363-4a93-9776-188667afbbd5
972024-07-18T22:09:30.139ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
982024-07-18T22:09:30.139ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::12
992024-07-18T22:09:30.139ZINFOcrucible-agent (worker): commit
1002024-07-18T22:09:30.142ZINFOcrucible-agent (worker): ok!
1012024-07-18T22:09:30.144ZINFOcrucible-agent (worker): SMF ok!
1022024-07-18T22:09:54.259ZINFOcrucible-agent (dropshot): request completed latency_us = 56 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:42354 req_id = c2267d36-ecdb-44d8-84bf-0653f994c32a response_code = 200 uri = /crucible/0/regions/6af44937-d40d-436d-8294-2377390fa49e
1032024-07-18T22:09:54.261ZINFOcrucible-agent (datafile): path is "/data/regions/6af44937-d40d-436d-8294-2377390fa49e"
1042024-07-18T22:09:54.273ZINFOcrucible-agent (datafile): dataset is oxp_18924fa5-0339-4445-8677-913f8ef37483/crucible/regions/6af44937-d40d-436d-8294-2377390fa49e
1052024-07-18T22:09:54.281ZINFOcrucible-agent (datafile): region 6af44937-d40d-436d-8294-2377390fa49e state: Created -> Tombstoned
1062024-07-18T22:09:54.281ZINFOcrucible-agent (worker): applying SMF actions before removal...
1072024-07-18T22:09:54.281ZINFOcrucible-agent (dropshot): request completed latency_us = 19420 local_addr = [fd00:1122:3344:101::12]:32345 method = DELETE remote_addr = [fd00:1122:3344:101::a]:42354 req_id = 56e591de-bedd-4ace-b234-281f9027a245 response_code = 204 uri = /crucible/0/regions/6af44937-d40d-436d-8294-2377390fa49e
1082024-07-18T22:09:54.282ZINFOcrucible-agent (worker): disabling downstairs instance: downstairs-6af44937-d40d-436d-8294-2377390fa49e (instance states: (Some(Online), None))
1092024-07-18T22:09:54.283ZINFOcrucible-agent (dropshot): request completed latency_us = 25 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:42354 req_id = 024946dc-5312-40db-80dc-f2364bad2515 response_code = 200 uri = /crucible/0/regions/6af44937-d40d-436d-8294-2377390fa49e
1102024-07-18T22:09:54.286ZINFOcrucible-agent (worker): existing directory value /dev/null does not match /data/regions/6cdf6550-5363-4a93-9776-188667afbbd5
1112024-07-18T22:09:54.286ZINFOcrucible-agent (worker): existing port value 0 does not match 19000
1122024-07-18T22:09:54.286ZINFOcrucible-agent (worker): existing address value 0.0.0.0 does not match fd00:1122:3344:101::12
1132024-07-18T22:09:54.286ZINFOcrucible-agent (worker): reconfig required
1142024-07-18T22:09:54.286ZINFOcrucible-agent (worker): using existing config property group
1152024-07-18T22:09:54.286ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-6cdf6550-5363-4a93-9776-188667afbbd5
1162024-07-18T22:09:54.287ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/6cdf6550-5363-4a93-9776-188667afbbd5
1172024-07-18T22:09:54.287ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
1182024-07-18T22:09:54.287ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::12
1192024-07-18T22:09:54.287ZINFOcrucible-agent (worker): commit
1202024-07-18T22:09:54.290ZINFOcrucible-agent (worker): ok!
1212024-07-18T22:09:54.292ZINFOcrucible-agent (worker): SMF ok!
1222024-07-18T22:09:54.301ZINFOcrucible-agent (worker): deleting zfs dataset "oxp_18924fa5-0339-4445-8677-913f8ef37483/crucible/regions/6af44937-d40d-436d-8294-2377390fa49e" region = 6af44937-d40d-436d-8294-2377390fa49e
1232024-07-18T22:09:54.376ZINFOcrucible-agent (dropshot): request completed latency_us = 694 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:42354 req_id = 450cb98e-8b8e-4306-94fd-9e0ee1611530 response_code = 200 uri = /crucible/0/regions/6af44937-d40d-436d-8294-2377390fa49e
1242024-07-18T22:09:54.381ZINFOcrucible-agent (datafile): region 6af44937-d40d-436d-8294-2377390fa49e state: Tombstoned -> Destroyed
1252024-07-18T22:09:54.448ZINFOcrucible-agent (dropshot): request completed latency_us = 41 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:42354 req_id = 861e4363-9957-4c5b-acc1-d546cc027c18 response_code = 200 uri = /crucible/0/regions/6af44937-d40d-436d-8294-2377390fa49e