Max level shown:
12026-07-02T18:45:41.500ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::14]:32345 remote_addr = [fd00:1122:3344:101::b]:64955
22026-07-02T18:45:41.528ZINFOcrucible-agent (datafile): region 0b579936-85fb-4e7d-ac24-ff208948cffe state: Requested
32026-07-02T18:45:41.533ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
42026-07-02T18:45:41.535ZINFOcrucible-agent (dropshot): request completed latency_us = 15231 local_addr = [fd00:1122:3344:101::14]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:64955 req_id = 4dfaa4b0-854e-44e7-a99b-793931c8d20c response_code = 200 uri = /crucible/0/regions
52026-07-02T18:45:41.544ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_a17f5a7e-073e-4668-ada4-d2aeb9fea173/crucible/regions/0b579936-85fb-4e7d-ac24-ff208948cffe
62026-07-02T18:45:41.544ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_a17f5a7e-073e-4668-ada4-d2aeb9fea173/crucible/regions/0b579936-85fb-4e7d-ac24-ff208948cffe
72026-07-02T18:45:41.593ZINFOcrucible-agent (worker): creating region Region { id: RegionId("0b579936-85fb-4e7d-ac24-ff208948cffe"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/0b579936-85fb-4e7d-ac24-ff208948cffe" region = 0b579936-85fb-4e7d-ac24-ff208948cffe
82026-07-02T18:45:41.595ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "0b579936-85fb-4e7d-ac24-ff208948cffe" "--data" "/data/regions/0b579936-85fb-4e7d-ac24-ff208948cffe" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 0b579936-85fb-4e7d-ac24-ff208948cffe
92026-07-02T18:45:41.717ZINFOcrucible-agent (worker): region files created ok region = 0b579936-85fb-4e7d-ac24-ff208948cffe
102026-07-02T18:45:41.719ZINFOcrucible-agent (datafile): region 0b579936-85fb-4e7d-ac24-ff208948cffe state: Requested -> Created
112026-07-02T18:45:41.719ZINFOcrucible-agent (worker): applying SMF actions post create...
122026-07-02T18:45:41.721ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-0b579936-85fb-4e7d-ac24-ff208948cffe
132026-07-02T18:45:41.727ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-0b579936-85fb-4e7d-ac24-ff208948cffe
142026-07-02T18:45:41.737ZINFOcrucible-agent (worker): creating config property group
152026-07-02T18:45:41.738ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-0b579936-85fb-4e7d-ac24-ff208948cffe
162026-07-02T18:45:41.739ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/0b579936-85fb-4e7d-ac24-ff208948cffe
172026-07-02T18:45:41.741ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
182026-07-02T18:45:41.741ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::14
192026-07-02T18:45:41.741ZINFOcrucible-agent (worker): commit
202026-07-02T18:45:41.749ZINFOcrucible-agent (worker): ok!
212026-07-02T18:45:41.752ZINFOcrucible-agent (worker): SMF ok!
222026-07-02T18:45:41.921ZINFOcrucible-agent (dropshot): request completed latency_us = 86 local_addr = [fd00:1122:3344:101::14]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:64955 req_id = fa35167b-3f4a-43fb-a0c4-fc3d81345bea response_code = 200 uri = /crucible/0/regions
232026-07-02T18:56:45.254ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::14]:32345 remote_addr = [fd00:1122:3344:101::b]:40061
242026-07-02T18:56:45.255ZINFOcrucible-agent (dropshot): request completed latency_us = 52 local_addr = [fd00:1122:3344:101::14]:32345 method = GET remote_addr = [fd00:1122:3344:101::b]:40061 req_id = dcaf5ac8-4293-42c3-9a9d-fe1739eb52de response_code = 200 uri = /crucible/0/regions/0b579936-85fb-4e7d-ac24-ff208948cffe
252026-07-02T18:56:45.259ZINFOcrucible-agent (datafile): path is "/data/regions/0b579936-85fb-4e7d-ac24-ff208948cffe"
262026-07-02T18:56:45.266ZINFOcrucible-agent (datafile): dataset is oxp_a17f5a7e-073e-4668-ada4-d2aeb9fea173/crucible/regions/0b579936-85fb-4e7d-ac24-ff208948cffe
272026-07-02T18:56:45.272ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_a17f5a7e-073e-4668-ada4-d2aeb9fea173/crucible/regions/0b579936-85fb-4e7d-ac24-ff208948cffe@3a162d34-2c29-4971-b389-772810b80b52
282026-07-02T18:56:45.272ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_a17f5a7e-073e-4668-ada4-d2aeb9fea173/crucible/regions/0b579936-85fb-4e7d-ac24-ff208948cffe", "3a162d34-2c29-4971-b389-772810b80b52"]
292026-07-02T18:56:45.285ZINFOcrucible-agent (dropshot): request completed latency_us = 26180 local_addr = [fd00:1122:3344:101::14]:32345 method = GET remote_addr = [fd00:1122:3344:101::b]:40061 req_id = bad8af3d-8a32-4f5c-9211-2ae9e1b72cbe response_code = 200 uri = /crucible/0/regions/0b579936-85fb-4e7d-ac24-ff208948cffe/snapshots/3a162d34-2c29-4971-b389-772810b80b52
302026-07-02T18:56:45.288ZINFOcrucible-agent (datafile): path is "/data/regions/0b579936-85fb-4e7d-ac24-ff208948cffe"
312026-07-02T18:56:45.293ZINFOcrucible-agent (datafile): dataset is oxp_a17f5a7e-073e-4668-ada4-d2aeb9fea173/crucible/regions/0b579936-85fb-4e7d-ac24-ff208948cffe
322026-07-02T18:56:45.299ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_a17f5a7e-073e-4668-ada4-d2aeb9fea173/crucible/regions/0b579936-85fb-4e7d-ac24-ff208948cffe@3a162d34-2c29-4971-b389-772810b80b52
332026-07-02T18:56:45.299ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_a17f5a7e-073e-4668-ada4-d2aeb9fea173/crucible/regions/0b579936-85fb-4e7d-ac24-ff208948cffe", "3a162d34-2c29-4971-b389-772810b80b52"]
342026-07-02T18:56:45.306ZINFOcrucible-agent (datafile): requesting running snapshot 0b579936-85fb-4e7d-ac24-ff208948cffe-3a162d34-2c29-4971-b389-772810b80b52 state: Requested
352026-07-02T18:56:45.309ZINFOcrucible-agent (worker): applying SMF actions for region 0b579936-85fb-4e7d-ac24-ff208948cffe running snapshot 3a162d34-2c29-4971-b389-772810b80b52 (state Requested)...
362026-07-02T18:56:45.309ZINFOcrucible-agent (dropshot): request completed latency_us = 21133 local_addr = [fd00:1122:3344:101::14]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:40061 req_id = c136c7cb-1397-43bc-9a14-450a0dbbcf1c response_code = 200 uri = /crucible/0/regions/0b579936-85fb-4e7d-ac24-ff208948cffe/snapshots/3a162d34-2c29-4971-b389-772810b80b52/run
372026-07-02T18:56:45.318ZINFOcrucible-agent (worker): existing directory value /dev/null does not match /data/regions/0b579936-85fb-4e7d-ac24-ff208948cffe
382026-07-02T18:56:45.318ZINFOcrucible-agent (worker): existing port value 0 does not match 19000
392026-07-02T18:56:45.318ZINFOcrucible-agent (worker): existing address value 0.0.0.0 does not match fd00:1122:3344:101::14
402026-07-02T18:56:45.318ZINFOcrucible-agent (worker): reconfig required
412026-07-02T18:56:45.318ZINFOcrucible-agent (worker): using existing config property group
422026-07-02T18:56:45.318ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-0b579936-85fb-4e7d-ac24-ff208948cffe
432026-07-02T18:56:45.318ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/0b579936-85fb-4e7d-ac24-ff208948cffe
442026-07-02T18:56:45.318ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
452026-07-02T18:56:45.318ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::14
462026-07-02T18:56:45.318ZINFOcrucible-agent (worker): commit
472026-07-02T18:56:45.324ZINFOcrucible-agent (worker): ok!
482026-07-02T18:56:45.327ZINFOcrucible-agent (worker): creating missing snapshot instance snapshot-0b579936-85fb-4e7d-ac24-ff208948cffe-3a162d34-2c29-4971-b389-772810b80b52
492026-07-02T18:56:45.328ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:snapshot-0b579936-85fb-4e7d-ac24-ff208948cffe-3a162d34-2c29-4971-b389-772810b80b52
502026-07-02T18:56:45.335ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:snapshot-0b579936-85fb-4e7d-ac24-ff208948cffe-3a162d34-2c29-4971-b389-772810b80b52
512026-07-02T18:56:45.335ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/0b579936-85fb-4e7d-ac24-ff208948cffe/.zfs/snapshot/3a162d34-2c29-4971-b389-772810b80b52
522026-07-02T18:56:45.335ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
532026-07-02T18:56:45.335ZINFOcrucible-agent (worker): ensure mode SCF_TYPE_ASTRING ro
542026-07-02T18:56:45.335ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::14
552026-07-02T18:56:45.335ZINFOcrucible-agent (worker): commit
562026-07-02T18:56:45.337ZINFOcrucible-agent (worker): ok!
572026-07-02T18:56:45.340ZINFOcrucible-agent (worker): SMF ok!
582026-07-02T18:56:45.340ZINFOcrucible-agent (datafile): region 0b579936-85fb-4e7d-ac24-ff208948cffe running snapshot 3a162d34-2c29-4971-b389-772810b80b52 state: Requested -> Created
592026-07-02T18:56:45.567ZINFOcrucible-agent (datafile): path is "/data/regions/0b579936-85fb-4e7d-ac24-ff208948cffe"
602026-07-02T18:56:45.574ZINFOcrucible-agent (datafile): dataset is oxp_a17f5a7e-073e-4668-ada4-d2aeb9fea173/crucible/regions/0b579936-85fb-4e7d-ac24-ff208948cffe
612026-07-02T18:56:45.581ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_a17f5a7e-073e-4668-ada4-d2aeb9fea173/crucible/regions/0b579936-85fb-4e7d-ac24-ff208948cffe@3a162d34-2c29-4971-b389-772810b80b52
622026-07-02T18:56:45.581ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_a17f5a7e-073e-4668-ada4-d2aeb9fea173/crucible/regions/0b579936-85fb-4e7d-ac24-ff208948cffe", "3a162d34-2c29-4971-b389-772810b80b52"]
632026-07-02T18:56:45.587ZINFOcrucible-agent (dropshot): request completed latency_us = 19822 local_addr = [fd00:1122:3344:101::14]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:40061 req_id = d9fb2b85-5de8-4902-8b83-2223d5132dca response_code = 200 uri = /crucible/0/regions/0b579936-85fb-4e7d-ac24-ff208948cffe/snapshots/3a162d34-2c29-4971-b389-772810b80b52/run
642026-07-02T18:56:50.160ZINFOcrucible-agent (datafile): region 8886c387-54cf-48e9-b4d6-bf37c103eb28 state: Requested
652026-07-02T18:56:50.160ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
662026-07-02T18:56:50.160ZINFOcrucible-agent (dropshot): request completed latency_us = 318 local_addr = [fd00:1122:3344:101::14]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:40061 req_id = edb7e4d8-1472-4d8f-aed8-943bd243e545 response_code = 200 uri = /crucible/0/regions
672026-07-02T18:56:50.169ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_a17f5a7e-073e-4668-ada4-d2aeb9fea173/crucible/regions/8886c387-54cf-48e9-b4d6-bf37c103eb28
682026-07-02T18:56:50.169ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_a17f5a7e-073e-4668-ada4-d2aeb9fea173/crucible/regions/8886c387-54cf-48e9-b4d6-bf37c103eb28
692026-07-02T18:56:50.233ZINFOcrucible-agent (worker): creating region Region { id: RegionId("8886c387-54cf-48e9-b4d6-bf37c103eb28"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19002, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/8886c387-54cf-48e9-b4d6-bf37c103eb28" region = 8886c387-54cf-48e9-b4d6-bf37c103eb28
702026-07-02T18:56:50.233ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "8886c387-54cf-48e9-b4d6-bf37c103eb28" "--data" "/data/regions/8886c387-54cf-48e9-b4d6-bf37c103eb28" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 8886c387-54cf-48e9-b4d6-bf37c103eb28
712026-07-02T18:56:50.289ZINFOcrucible-agent (worker): region files created ok region = 8886c387-54cf-48e9-b4d6-bf37c103eb28
722026-07-02T18:56:50.289ZINFOcrucible-agent (datafile): region 8886c387-54cf-48e9-b4d6-bf37c103eb28 state: Requested -> Created
732026-07-02T18:56:50.289ZINFOcrucible-agent (worker): applying SMF actions post create...
742026-07-02T18:56:50.290ZINFOcrucible-agent (worker): existing directory value /dev/null does not match /data/regions/0b579936-85fb-4e7d-ac24-ff208948cffe
752026-07-02T18:56:50.290ZINFOcrucible-agent (worker): existing port value 0 does not match 19000
762026-07-02T18:56:50.290ZINFOcrucible-agent (worker): existing address value 0.0.0.0 does not match fd00:1122:3344:101::14
772026-07-02T18:56:50.290ZINFOcrucible-agent (worker): reconfig required
782026-07-02T18:56:50.290ZINFOcrucible-agent (worker): using existing config property group
792026-07-02T18:56:50.290ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-0b579936-85fb-4e7d-ac24-ff208948cffe
802026-07-02T18:56:50.290ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/0b579936-85fb-4e7d-ac24-ff208948cffe
812026-07-02T18:56:50.290ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
822026-07-02T18:56:50.290ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::14
832026-07-02T18:56:50.290ZINFOcrucible-agent (worker): commit
842026-07-02T18:56:50.293ZINFOcrucible-agent (worker): ok!
852026-07-02T18:56:50.293ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-8886c387-54cf-48e9-b4d6-bf37c103eb28
862026-07-02T18:56:50.303ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-8886c387-54cf-48e9-b4d6-bf37c103eb28
872026-07-02T18:56:50.306ZINFOcrucible-agent (worker): creating config property group
882026-07-02T18:56:50.308ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-8886c387-54cf-48e9-b4d6-bf37c103eb28
892026-07-02T18:56:50.308ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/8886c387-54cf-48e9-b4d6-bf37c103eb28
902026-07-02T18:56:50.308ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19002
912026-07-02T18:56:50.308ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::14
922026-07-02T18:56:50.308ZINFOcrucible-agent (worker): commit
932026-07-02T18:56:50.310ZINFOcrucible-agent (worker): ok!
942026-07-02T18:56:50.314ZINFOcrucible-agent (worker): SMF ok!
952026-07-02T18:56:50.474ZINFOcrucible-agent (dropshot): request completed latency_us = 67 local_addr = [fd00:1122:3344:101::14]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:40061 req_id = b7c9e9f7-3e95-431a-85b7-1cb37fdacaae response_code = 200 uri = /crucible/0/regions
962026-07-02T18:57:14.706ZINFOcrucible-agent (dropshot): request completed latency_us = 73 local_addr = [fd00:1122:3344:101::14]:32345 method = GET remote_addr = [fd00:1122:3344:101::b]:40061 req_id = f75a450f-7034-4c5d-90e4-03f1a37bbd06 response_code = 200 uri = /crucible/0/regions/8886c387-54cf-48e9-b4d6-bf37c103eb28
972026-07-02T18:57:14.710ZINFOcrucible-agent (datafile): path is "/data/regions/8886c387-54cf-48e9-b4d6-bf37c103eb28"
982026-07-02T18:57:14.717ZINFOcrucible-agent (datafile): dataset is oxp_a17f5a7e-073e-4668-ada4-d2aeb9fea173/crucible/regions/8886c387-54cf-48e9-b4d6-bf37c103eb28
992026-07-02T18:57:14.724ZINFOcrucible-agent (datafile): region 8886c387-54cf-48e9-b4d6-bf37c103eb28 state: Created -> Tombstoned
1002026-07-02T18:57:14.725ZINFOcrucible-agent (worker): applying SMF actions before removal...
1012026-07-02T18:57:14.727ZINFOcrucible-agent (dropshot): request completed latency_us = 17287 local_addr = [fd00:1122:3344:101::14]:32345 method = DELETE remote_addr = [fd00:1122:3344:101::b]:40061 req_id = 9750116f-a633-4e0d-aa9e-44a8b7675c82 response_code = 204 uri = /crucible/0/regions/8886c387-54cf-48e9-b4d6-bf37c103eb28
1022026-07-02T18:57:14.728ZINFOcrucible-agent (worker): disabling downstairs instance: downstairs-8886c387-54cf-48e9-b4d6-bf37c103eb28 (instance states: (Some(Online), None))
1032026-07-02T18:57:14.730ZINFOcrucible-agent (worker): existing directory value /dev/null does not match /data/regions/0b579936-85fb-4e7d-ac24-ff208948cffe
1042026-07-02T18:57:14.730ZINFOcrucible-agent (worker): existing port value 0 does not match 19000
1052026-07-02T18:57:14.730ZINFOcrucible-agent (worker): existing address value 0.0.0.0 does not match fd00:1122:3344:101::14
1062026-07-02T18:57:14.730ZINFOcrucible-agent (worker): reconfig required
1072026-07-02T18:57:14.730ZINFOcrucible-agent (worker): using existing config property group
1082026-07-02T18:57:14.731ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-0b579936-85fb-4e7d-ac24-ff208948cffe
1092026-07-02T18:57:14.731ZINFOcrucible-agent (dropshot): request completed latency_us = 34 local_addr = [fd00:1122:3344:101::14]:32345 method = GET remote_addr = [fd00:1122:3344:101::b]:40061 req_id = e6231d59-448c-4b4d-be66-33948422f8e9 response_code = 200 uri = /crucible/0/regions/8886c387-54cf-48e9-b4d6-bf37c103eb28
1102026-07-02T18:57:14.731ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/0b579936-85fb-4e7d-ac24-ff208948cffe
1112026-07-02T18:57:14.731ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
1122026-07-02T18:57:14.731ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::14
1132026-07-02T18:57:14.731ZINFOcrucible-agent (worker): commit
1142026-07-02T18:57:14.733ZINFOcrucible-agent (worker): ok!
1152026-07-02T18:57:14.734ZINFOcrucible-agent (worker): SMF ok!
1162026-07-02T18:57:14.743ZINFOcrucible-agent (worker): deleting zfs dataset "oxp_a17f5a7e-073e-4668-ada4-d2aeb9fea173/crucible/regions/8886c387-54cf-48e9-b4d6-bf37c103eb28" region = 8886c387-54cf-48e9-b4d6-bf37c103eb28
1172026-07-02T18:57:14.798ZINFOcrucible-agent (datafile): region 8886c387-54cf-48e9-b4d6-bf37c103eb28 state: Tombstoned -> Destroyed
1182026-07-02T18:57:14.799ZINFOcrucible-agent (dropshot): request completed latency_us = 50 local_addr = [fd00:1122:3344:101::14]:32345 method = GET remote_addr = [fd00:1122:3344:101::b]:40061 req_id = d13cac26-ef11-4f06-ba51-d655416cae69 response_code = 200 uri = /crucible/0/regions/8886c387-54cf-48e9-b4d6-bf37c103eb28