Max level shown:
1 [ Aug 15 07:31:02 Enabled. ]
2 [ Aug 15 07:31:02 Rereading configuration. ]
3 [ Aug 15 07:31:07 Rereading configuration. ]
4 [ Aug 15 07:31:11 Executing start method ("/opt/oxide/crucible/bin/crucible-agent run -D /opt/oxide/crucible/bin/crucible-downstairs --dataset oxp_44320b94-d85e-41fa-8002-b60346d6c092/crucible -l [fd00:1122:3344:101::14]:32345 -P 19000 -p downstairs -s snapshot"). ]
5 note: configured to log to "/dev/stdout"
62026-08-15T07:31:11.288ZINFOcrucible-agent: Crucible Version: Crucible Version: 0.0.1 Commit SHA: ad8a31742adc45e925e63443a5b43c8e30604022 Commit timestamp: 2026-07-28T19:41:59.000000000Z branch: main rustc: 1.90.0 stable x86_64-unknown-illumos Cargo: x86_64-unknown-illumos Debug: false Opt level: 3
72026-08-15T07:31:11.288ZINFOcrucible-agent: dataset: "oxp_44320b94-d85e-41fa-8002-b60346d6c092/crucible"
82026-08-15T07:31:11.288ZINFOcrucible-agent: listen IP: [fd00:1122:3344:101::14]:32345
92026-08-15T07:31:11.288ZINFOcrucible-agent: SMF instance name downstairs_prefix: "downstairs"
102026-08-15T07:31:11.307ZINFOcrucible-agent (datafile): Using conf_path:"/data/crucible.json"
112026-08-15T07:31:11.431ZINFOcrucible-agent (dropshot): listening local_addr = [fd00:1122:3344:101::14]:32345
122026-08-15T07:31:47.576ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::14]:32345 remote_addr = [fd00:1122:3344:101::b]:48456
132026-08-15T07:31:47.608ZINFOcrucible-agent (datafile): region d1d008d6-90b9-4f07-b4af-c58b6acb6d3d state: Requested
142026-08-15T07:31:47.612ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
152026-08-15T07:31:47.616ZINFOcrucible-agent (dropshot): request completed latency_us = 15156 local_addr = [fd00:1122:3344:101::14]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:48456 req_id = 83bdbf8c-fe0a-47bc-9d25-08b408f30c97 response_code = 200 uri = /crucible/0/regions
162026-08-15T07:31:47.620ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_44320b94-d85e-41fa-8002-b60346d6c092/crucible/regions/d1d008d6-90b9-4f07-b4af-c58b6acb6d3d
172026-08-15T07:31:47.620ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_44320b94-d85e-41fa-8002-b60346d6c092/crucible/regions/d1d008d6-90b9-4f07-b4af-c58b6acb6d3d
182026-08-15T07:31:47.751ZINFOcrucible-agent (worker): creating region Region { id: RegionId("d1d008d6-90b9-4f07-b4af-c58b6acb6d3d"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/d1d008d6-90b9-4f07-b4af-c58b6acb6d3d" region = d1d008d6-90b9-4f07-b4af-c58b6acb6d3d
192026-08-15T07:31:47.753ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "d1d008d6-90b9-4f07-b4af-c58b6acb6d3d" "--data" "/data/regions/d1d008d6-90b9-4f07-b4af-c58b6acb6d3d" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = d1d008d6-90b9-4f07-b4af-c58b6acb6d3d
202026-08-15T07:31:47.827ZINFOcrucible-agent (dropshot): request completed latency_us = 62 local_addr = [fd00:1122:3344:101::14]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:48456 req_id = 595f6e36-ceee-414b-8249-dce928ac973c response_code = 200 uri = /crucible/0/regions
212026-08-15T07:31:48.003ZINFOcrucible-agent (worker): region files created ok region = d1d008d6-90b9-4f07-b4af-c58b6acb6d3d
222026-08-15T07:31:48.007ZINFOcrucible-agent (datafile): region d1d008d6-90b9-4f07-b4af-c58b6acb6d3d state: Requested -> Created
232026-08-15T07:31:48.008ZINFOcrucible-agent (worker): applying SMF actions post create...
242026-08-15T07:31:48.010ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-d1d008d6-90b9-4f07-b4af-c58b6acb6d3d
252026-08-15T07:31:48.028ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-d1d008d6-90b9-4f07-b4af-c58b6acb6d3d
262026-08-15T07:31:48.061ZINFOcrucible-agent (worker): creating config property group
272026-08-15T07:31:48.065ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-d1d008d6-90b9-4f07-b4af-c58b6acb6d3d
282026-08-15T07:31:48.066ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/d1d008d6-90b9-4f07-b4af-c58b6acb6d3d
292026-08-15T07:31:48.066ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
302026-08-15T07:31:48.066ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::14
312026-08-15T07:31:48.066ZINFOcrucible-agent (worker): commit
322026-08-15T07:31:48.084ZINFOcrucible-agent (worker): ok!
332026-08-15T07:31:48.111ZINFOcrucible-agent (worker): SMF ok!
342026-08-15T07:31:48.237ZINFOcrucible-agent (dropshot): request completed latency_us = 71 local_addr = [fd00:1122:3344:101::14]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:48456 req_id = 30843b1a-fd22-4640-839f-55ace1d46e54 response_code = 200 uri = /crucible/0/regions
352026-08-15T07:42:49.931ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::14]:32345 remote_addr = [fd00:1122:3344:101::b]:49053
362026-08-15T07:42:49.932ZINFOcrucible-agent (datafile): region 0adbff53-18c6-4e88-a79c-2a41aaa68fc0 state: Requested
372026-08-15T07:42:49.933ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
382026-08-15T07:42:49.934ZINFOcrucible-agent (dropshot): request completed latency_us = 1428 local_addr = [fd00:1122:3344:101::14]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:49053 req_id = 1e559490-acc5-44a3-875c-7c7e1a685d9d response_code = 200 uri = /crucible/0/regions
392026-08-15T07:42:49.941ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_44320b94-d85e-41fa-8002-b60346d6c092/crucible/regions/0adbff53-18c6-4e88-a79c-2a41aaa68fc0
402026-08-15T07:42:49.941ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_44320b94-d85e-41fa-8002-b60346d6c092/crucible/regions/0adbff53-18c6-4e88-a79c-2a41aaa68fc0
412026-08-15T07:42:50.075ZINFOcrucible-agent (worker): creating region Region { id: RegionId("0adbff53-18c6-4e88-a79c-2a41aaa68fc0"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19001, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/0adbff53-18c6-4e88-a79c-2a41aaa68fc0" region = 0adbff53-18c6-4e88-a79c-2a41aaa68fc0
422026-08-15T07:42:50.075ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "0adbff53-18c6-4e88-a79c-2a41aaa68fc0" "--data" "/data/regions/0adbff53-18c6-4e88-a79c-2a41aaa68fc0" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 0adbff53-18c6-4e88-a79c-2a41aaa68fc0
432026-08-15T07:42:50.214ZINFOcrucible-agent (dropshot): request completed latency_us = 62 local_addr = [fd00:1122:3344:101::14]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:49053 req_id = 954357f3-5a82-4124-b57c-b3ece1574623 response_code = 200 uri = /crucible/0/regions
442026-08-15T07:42:50.270ZINFOcrucible-agent (worker): region files created ok region = 0adbff53-18c6-4e88-a79c-2a41aaa68fc0
452026-08-15T07:42:50.270ZINFOcrucible-agent (datafile): region 0adbff53-18c6-4e88-a79c-2a41aaa68fc0 state: Requested -> Created
462026-08-15T07:42:50.270ZINFOcrucible-agent (worker): applying SMF actions post create...
472026-08-15T07:42:50.271ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-0adbff53-18c6-4e88-a79c-2a41aaa68fc0
482026-08-15T07:42:50.300ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-0adbff53-18c6-4e88-a79c-2a41aaa68fc0
492026-08-15T07:42:50.359ZINFOcrucible-agent (worker): creating config property group
502026-08-15T07:42:50.366ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-0adbff53-18c6-4e88-a79c-2a41aaa68fc0
512026-08-15T07:42:50.367ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/0adbff53-18c6-4e88-a79c-2a41aaa68fc0
522026-08-15T07:42:50.367ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
532026-08-15T07:42:50.367ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::14
542026-08-15T07:42:50.367ZINFOcrucible-agent (worker): commit
552026-08-15T07:42:50.393ZINFOcrucible-agent (worker): ok!
562026-08-15T07:42:50.413ZINFOcrucible-agent (worker): SMF ok!
572026-08-15T07:42:50.750ZINFOcrucible-agent (dropshot): request completed latency_us = 70 local_addr = [fd00:1122:3344:101::14]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:49053 req_id = 71143028-b697-46c9-8fbf-79be94afe786 response_code = 200 uri = /crucible/0/regions
582026-08-15T07:42:52.247ZINFOcrucible-agent (dropshot): request completed latency_us = 75 local_addr = [fd00:1122:3344:101::14]:32345 method = GET remote_addr = [fd00:1122:3344:101::b]:49053 req_id = 80d217fd-6326-402d-ac2f-16338e3ea23c response_code = 200 uri = /crucible/0/regions/d1d008d6-90b9-4f07-b4af-c58b6acb6d3d
592026-08-15T07:42:52.251ZINFOcrucible-agent (datafile): path is "/data/regions/d1d008d6-90b9-4f07-b4af-c58b6acb6d3d"
602026-08-15T07:42:52.257ZINFOcrucible-agent (datafile): dataset is oxp_44320b94-d85e-41fa-8002-b60346d6c092/crucible/regions/d1d008d6-90b9-4f07-b4af-c58b6acb6d3d
612026-08-15T07:42:52.262ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_44320b94-d85e-41fa-8002-b60346d6c092/crucible/regions/d1d008d6-90b9-4f07-b4af-c58b6acb6d3d@444a1022-a703-41dd-b916-d7bd97e459c4
622026-08-15T07:42:52.262ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_44320b94-d85e-41fa-8002-b60346d6c092/crucible/regions/d1d008d6-90b9-4f07-b4af-c58b6acb6d3d", "444a1022-a703-41dd-b916-d7bd97e459c4"]
632026-08-15T07:42:52.273ZINFOcrucible-agent (dropshot): request completed latency_us = 21887 local_addr = [fd00:1122:3344:101::14]:32345 method = GET remote_addr = [fd00:1122:3344:101::b]:49053 req_id = 5ad3c78e-343a-4121-9378-ceeb341702a5 response_code = 200 uri = /crucible/0/regions/d1d008d6-90b9-4f07-b4af-c58b6acb6d3d/snapshots/444a1022-a703-41dd-b916-d7bd97e459c4
642026-08-15T07:42:52.276ZINFOcrucible-agent (datafile): path is "/data/regions/d1d008d6-90b9-4f07-b4af-c58b6acb6d3d"
652026-08-15T07:42:52.283ZINFOcrucible-agent (datafile): dataset is oxp_44320b94-d85e-41fa-8002-b60346d6c092/crucible/regions/d1d008d6-90b9-4f07-b4af-c58b6acb6d3d
662026-08-15T07:42:52.288ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_44320b94-d85e-41fa-8002-b60346d6c092/crucible/regions/d1d008d6-90b9-4f07-b4af-c58b6acb6d3d@444a1022-a703-41dd-b916-d7bd97e459c4
672026-08-15T07:42:52.288ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_44320b94-d85e-41fa-8002-b60346d6c092/crucible/regions/d1d008d6-90b9-4f07-b4af-c58b6acb6d3d", "444a1022-a703-41dd-b916-d7bd97e459c4"]
682026-08-15T07:42:52.299ZINFOcrucible-agent (datafile): requesting running snapshot d1d008d6-90b9-4f07-b4af-c58b6acb6d3d-444a1022-a703-41dd-b916-d7bd97e459c4 state: Requested
692026-08-15T07:42:52.304ZINFOcrucible-agent (worker): applying SMF actions for region d1d008d6-90b9-4f07-b4af-c58b6acb6d3d running snapshot 444a1022-a703-41dd-b916-d7bd97e459c4 (state Requested)...
702026-08-15T07:42:52.304ZINFOcrucible-agent (dropshot): request completed latency_us = 27379 local_addr = [fd00:1122:3344:101::14]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:49053 req_id = 1fd6e1f7-f08d-4c88-8577-48ba051d675d response_code = 200 uri = /crucible/0/regions/d1d008d6-90b9-4f07-b4af-c58b6acb6d3d/snapshots/444a1022-a703-41dd-b916-d7bd97e459c4/run
712026-08-15T07:42:52.306ZINFOcrucible-agent (worker): creating missing snapshot instance snapshot-d1d008d6-90b9-4f07-b4af-c58b6acb6d3d-444a1022-a703-41dd-b916-d7bd97e459c4
722026-08-15T07:42:52.323ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:snapshot-d1d008d6-90b9-4f07-b4af-c58b6acb6d3d-444a1022-a703-41dd-b916-d7bd97e459c4
732026-08-15T07:42:52.350ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:snapshot-d1d008d6-90b9-4f07-b4af-c58b6acb6d3d-444a1022-a703-41dd-b916-d7bd97e459c4
742026-08-15T07:42:52.350ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/d1d008d6-90b9-4f07-b4af-c58b6acb6d3d/.zfs/snapshot/444a1022-a703-41dd-b916-d7bd97e459c4
752026-08-15T07:42:52.350ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19002
762026-08-15T07:42:52.350ZINFOcrucible-agent (worker): ensure mode SCF_TYPE_ASTRING ro
772026-08-15T07:42:52.350ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::14
782026-08-15T07:42:52.350ZINFOcrucible-agent (worker): commit
792026-08-15T07:42:52.356ZINFOcrucible-agent (worker): ok!
802026-08-15T07:42:52.369ZINFOcrucible-agent (worker): SMF ok!
812026-08-15T07:42:52.369ZINFOcrucible-agent (datafile): region d1d008d6-90b9-4f07-b4af-c58b6acb6d3d running snapshot 444a1022-a703-41dd-b916-d7bd97e459c4 state: Requested -> Created
822026-08-15T07:42:52.554ZINFOcrucible-agent (datafile): path is "/data/regions/d1d008d6-90b9-4f07-b4af-c58b6acb6d3d"
832026-08-15T07:42:52.561ZINFOcrucible-agent (datafile): dataset is oxp_44320b94-d85e-41fa-8002-b60346d6c092/crucible/regions/d1d008d6-90b9-4f07-b4af-c58b6acb6d3d
842026-08-15T07:42:52.567ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_44320b94-d85e-41fa-8002-b60346d6c092/crucible/regions/d1d008d6-90b9-4f07-b4af-c58b6acb6d3d@444a1022-a703-41dd-b916-d7bd97e459c4
852026-08-15T07:42:52.567ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_44320b94-d85e-41fa-8002-b60346d6c092/crucible/regions/d1d008d6-90b9-4f07-b4af-c58b6acb6d3d", "444a1022-a703-41dd-b916-d7bd97e459c4"]
862026-08-15T07:42:52.572ZINFOcrucible-agent (dropshot): request completed latency_us = 17580 local_addr = [fd00:1122:3344:101::14]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:49053 req_id = 550b71aa-273b-48f2-a6cc-fc424b0ef1f8 response_code = 200 uri = /crucible/0/regions/d1d008d6-90b9-4f07-b4af-c58b6acb6d3d/snapshots/444a1022-a703-41dd-b916-d7bd97e459c4/run
872026-08-15T07:42:58.260ZINFOcrucible-agent (datafile): region 4038f538-3551-48e2-bcaa-ab96e52db4ae state: Requested
882026-08-15T07:42:58.260ZINFOcrucible-agent (dropshot): request completed latency_us = 222 local_addr = [fd00:1122:3344:101::14]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:49053 req_id = 664a9386-245f-4bda-85f3-d62b3b48eab7 response_code = 200 uri = /crucible/0/regions
892026-08-15T07:42:58.260ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
902026-08-15T07:42:58.266ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_44320b94-d85e-41fa-8002-b60346d6c092/crucible/regions/4038f538-3551-48e2-bcaa-ab96e52db4ae
912026-08-15T07:42:58.266ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_44320b94-d85e-41fa-8002-b60346d6c092/crucible/regions/4038f538-3551-48e2-bcaa-ab96e52db4ae
922026-08-15T07:42:58.334ZINFOcrucible-agent (worker): creating region Region { id: RegionId("4038f538-3551-48e2-bcaa-ab96e52db4ae"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19003, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/4038f538-3551-48e2-bcaa-ab96e52db4ae" region = 4038f538-3551-48e2-bcaa-ab96e52db4ae
932026-08-15T07:42:58.334ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "4038f538-3551-48e2-bcaa-ab96e52db4ae" "--data" "/data/regions/4038f538-3551-48e2-bcaa-ab96e52db4ae" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 4038f538-3551-48e2-bcaa-ab96e52db4ae
942026-08-15T07:42:58.446ZINFOcrucible-agent (worker): region files created ok region = 4038f538-3551-48e2-bcaa-ab96e52db4ae
952026-08-15T07:42:58.446ZINFOcrucible-agent (datafile): region 4038f538-3551-48e2-bcaa-ab96e52db4ae state: Requested -> Created
962026-08-15T07:42:58.446ZINFOcrucible-agent (worker): applying SMF actions post create...
972026-08-15T07:42:58.447ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-4038f538-3551-48e2-bcaa-ab96e52db4ae
982026-08-15T07:42:58.454ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-4038f538-3551-48e2-bcaa-ab96e52db4ae
992026-08-15T07:42:58.465ZINFOcrucible-agent (worker): creating config property group
1002026-08-15T07:42:58.471ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-4038f538-3551-48e2-bcaa-ab96e52db4ae
1012026-08-15T07:42:58.471ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/4038f538-3551-48e2-bcaa-ab96e52db4ae
1022026-08-15T07:42:58.471ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19003
1032026-08-15T07:42:58.471ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::14
1042026-08-15T07:42:58.472ZINFOcrucible-agent (worker): commit
1052026-08-15T07:42:58.478ZINFOcrucible-agent (worker): ok!
1062026-08-15T07:42:58.496ZINFOcrucible-agent (worker): SMF ok!
1072026-08-15T07:42:58.639ZINFOcrucible-agent (dropshot): request completed latency_us = 57 local_addr = [fd00:1122:3344:101::14]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:49053 req_id = 0aad79b0-8288-45fa-827c-8ca23e04f92e response_code = 200 uri = /crucible/0/regions
1082026-08-15T07:43:25.096ZINFOcrucible-agent (dropshot): request completed latency_us = 51 local_addr = [fd00:1122:3344:101::14]:32345 method = GET remote_addr = [fd00:1122:3344:101::b]:49053 req_id = 277b491c-5fa5-4f6f-8366-61d0b5a8d90d response_code = 200 uri = /crucible/0/regions/4038f538-3551-48e2-bcaa-ab96e52db4ae
1092026-08-15T07:43:25.100ZINFOcrucible-agent (datafile): path is "/data/regions/4038f538-3551-48e2-bcaa-ab96e52db4ae"
1102026-08-15T07:43:25.110ZINFOcrucible-agent (datafile): dataset is oxp_44320b94-d85e-41fa-8002-b60346d6c092/crucible/regions/4038f538-3551-48e2-bcaa-ab96e52db4ae
1112026-08-15T07:43:25.118ZINFOcrucible-agent (datafile): region 4038f538-3551-48e2-bcaa-ab96e52db4ae state: Created -> Tombstoned
1122026-08-15T07:43:25.118ZINFOcrucible-agent (worker): applying SMF actions before removal...
1132026-08-15T07:43:25.118ZINFOcrucible-agent (dropshot): request completed latency_us = 18042 local_addr = [fd00:1122:3344:101::14]:32345 method = DELETE remote_addr = [fd00:1122:3344:101::b]:49053 req_id = 16d67f92-0924-4048-bf35-034cc97df351 response_code = 204 uri = /crucible/0/regions/4038f538-3551-48e2-bcaa-ab96e52db4ae
1142026-08-15T07:43:25.119ZINFOcrucible-agent (worker): disabling downstairs instance: downstairs-4038f538-3551-48e2-bcaa-ab96e52db4ae (instance states: (Some(Online), None))
1152026-08-15T07:43:25.121ZINFOcrucible-agent (dropshot): request completed latency_us = 64 local_addr = [fd00:1122:3344:101::14]:32345 method = GET remote_addr = [fd00:1122:3344:101::b]:49053 req_id = 682d0eb6-ff0a-4d60-a3c4-35be4ba4aa91 response_code = 200 uri = /crucible/0/regions/4038f538-3551-48e2-bcaa-ab96e52db4ae
1162026-08-15T07:43:25.129ZINFOcrucible-agent (worker): SMF ok!
1172026-08-15T07:43:25.138ZINFOcrucible-agent (worker): deleting zfs dataset "oxp_44320b94-d85e-41fa-8002-b60346d6c092/crucible/regions/4038f538-3551-48e2-bcaa-ab96e52db4ae" region = 4038f538-3551-48e2-bcaa-ab96e52db4ae
1182026-08-15T07:43:25.195ZINFOcrucible-agent (datafile): region 4038f538-3551-48e2-bcaa-ab96e52db4ae state: Tombstoned -> Destroyed
1192026-08-15T07:43:25.245ZINFOcrucible-agent (dropshot): request completed latency_us = 50 local_addr = [fd00:1122:3344:101::14]:32345 method = GET remote_addr = [fd00:1122:3344:101::b]:49053 req_id = 308c118c-8ff1-46d4-8772-b58608e25825 response_code = 200 uri = /crucible/0/regions/4038f538-3551-48e2-bcaa-ab96e52db4ae