Max level shown:
12026-06-22T21:00:31.893ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::14]:32345 remote_addr = [fd00:1122:3344:101::c]:47066
22026-06-22T21:00:31.927ZINFOcrucible-agent (datafile): region 4fca0064-3aed-4f1a-a315-49e5fc913e38 state: Requested
32026-06-22T21:00:31.931ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
42026-06-22T21:00:31.931ZINFOcrucible-agent (dropshot): request completed latency_us = 13232 local_addr = [fd00:1122:3344:101::14]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:47066 req_id = ae0708e2-8082-4cd4-aad5-f0a786afbe9d response_code = 200 uri = /crucible/0/regions
52026-06-22T21:00:31.937ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_96ad0d72-7678-45ca-a377-d07769acd0e0/crucible/regions/4fca0064-3aed-4f1a-a315-49e5fc913e38
62026-06-22T21:00:31.937ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_96ad0d72-7678-45ca-a377-d07769acd0e0/crucible/regions/4fca0064-3aed-4f1a-a315-49e5fc913e38
72026-06-22T21:00:31.989ZINFOcrucible-agent (worker): creating region Region { id: RegionId("4fca0064-3aed-4f1a-a315-49e5fc913e38"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/4fca0064-3aed-4f1a-a315-49e5fc913e38" region = 4fca0064-3aed-4f1a-a315-49e5fc913e38
82026-06-22T21:00:31.991ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "4fca0064-3aed-4f1a-a315-49e5fc913e38" "--data" "/data/regions/4fca0064-3aed-4f1a-a315-49e5fc913e38" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 4fca0064-3aed-4f1a-a315-49e5fc913e38
92026-06-22T21:00:32.105ZINFOcrucible-agent (worker): region files created ok region = 4fca0064-3aed-4f1a-a315-49e5fc913e38
102026-06-22T21:00:32.105ZINFOcrucible-agent (datafile): region 4fca0064-3aed-4f1a-a315-49e5fc913e38 state: Requested -> Created
112026-06-22T21:00:32.105ZINFOcrucible-agent (worker): applying SMF actions post create...
122026-06-22T21:00:32.108ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-4fca0064-3aed-4f1a-a315-49e5fc913e38
132026-06-22T21:00:32.112ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-4fca0064-3aed-4f1a-a315-49e5fc913e38
142026-06-22T21:00:32.125ZINFOcrucible-agent (worker): creating config property group
152026-06-22T21:00:32.127ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-4fca0064-3aed-4f1a-a315-49e5fc913e38
162026-06-22T21:00:32.128ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/4fca0064-3aed-4f1a-a315-49e5fc913e38
172026-06-22T21:00:32.128ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
182026-06-22T21:00:32.128ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::14
192026-06-22T21:00:32.128ZINFOcrucible-agent (worker): commit
202026-06-22T21:00:32.138ZINFOcrucible-agent (worker): ok!
212026-06-22T21:00:32.141ZINFOcrucible-agent (worker): SMF ok!
222026-06-22T21:00:32.315ZINFOcrucible-agent (dropshot): request completed latency_us = 70 local_addr = [fd00:1122:3344:101::14]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:47066 req_id = 6a149779-bc26-4a34-a916-06bcd83b85fa response_code = 200 uri = /crucible/0/regions
232026-06-22T21:11:37.322ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::14]:32345 remote_addr = [fd00:1122:3344:101::c]:48354
242026-06-22T21:11:37.322ZINFOcrucible-agent (dropshot): request completed latency_us = 87 local_addr = [fd00:1122:3344:101::14]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:48354 req_id = 4686653f-f10d-4294-9b15-7922da37a9b5 response_code = 200 uri = /crucible/0/regions/4fca0064-3aed-4f1a-a315-49e5fc913e38
252026-06-22T21:11:37.327ZINFOcrucible-agent (datafile): path is "/data/regions/4fca0064-3aed-4f1a-a315-49e5fc913e38"
262026-06-22T21:11:37.333ZINFOcrucible-agent (datafile): dataset is oxp_96ad0d72-7678-45ca-a377-d07769acd0e0/crucible/regions/4fca0064-3aed-4f1a-a315-49e5fc913e38
272026-06-22T21:11:37.340ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_96ad0d72-7678-45ca-a377-d07769acd0e0/crucible/regions/4fca0064-3aed-4f1a-a315-49e5fc913e38@c4125642-dafd-4e03-a862-e83165c96356
282026-06-22T21:11:37.340ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_96ad0d72-7678-45ca-a377-d07769acd0e0/crucible/regions/4fca0064-3aed-4f1a-a315-49e5fc913e38", "c4125642-dafd-4e03-a862-e83165c96356"]
292026-06-22T21:11:37.350ZINFOcrucible-agent (dropshot): request completed latency_us = 23025 local_addr = [fd00:1122:3344:101::14]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:48354 req_id = 3ef0a0e7-22f2-4d93-b71b-fb338c6356f8 response_code = 200 uri = /crucible/0/regions/4fca0064-3aed-4f1a-a315-49e5fc913e38/snapshots/c4125642-dafd-4e03-a862-e83165c96356
302026-06-22T21:11:37.356ZINFOcrucible-agent (datafile): path is "/data/regions/4fca0064-3aed-4f1a-a315-49e5fc913e38"
312026-06-22T21:11:37.361ZINFOcrucible-agent (datafile): dataset is oxp_96ad0d72-7678-45ca-a377-d07769acd0e0/crucible/regions/4fca0064-3aed-4f1a-a315-49e5fc913e38
322026-06-22T21:11:37.367ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_96ad0d72-7678-45ca-a377-d07769acd0e0/crucible/regions/4fca0064-3aed-4f1a-a315-49e5fc913e38@c4125642-dafd-4e03-a862-e83165c96356
332026-06-22T21:11:37.367ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_96ad0d72-7678-45ca-a377-d07769acd0e0/crucible/regions/4fca0064-3aed-4f1a-a315-49e5fc913e38", "c4125642-dafd-4e03-a862-e83165c96356"]
342026-06-22T21:11:37.375ZINFOcrucible-agent (datafile): requesting running snapshot 4fca0064-3aed-4f1a-a315-49e5fc913e38-c4125642-dafd-4e03-a862-e83165c96356 state: Requested
352026-06-22T21:11:37.378ZINFOcrucible-agent (worker): applying SMF actions for region 4fca0064-3aed-4f1a-a315-49e5fc913e38 running snapshot c4125642-dafd-4e03-a862-e83165c96356 (state Requested)...
362026-06-22T21:11:37.378ZINFOcrucible-agent (dropshot): request completed latency_us = 22249 local_addr = [fd00:1122:3344:101::14]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:48354 req_id = 18d9570f-ca10-44c5-b292-4a672e71b61a response_code = 200 uri = /crucible/0/regions/4fca0064-3aed-4f1a-a315-49e5fc913e38/snapshots/c4125642-dafd-4e03-a862-e83165c96356/run
372026-06-22T21:11:37.386ZINFOcrucible-agent (worker): existing directory value /dev/null does not match /data/regions/4fca0064-3aed-4f1a-a315-49e5fc913e38
382026-06-22T21:11:37.386ZINFOcrucible-agent (worker): existing port value 0 does not match 19000
392026-06-22T21:11:37.386ZINFOcrucible-agent (worker): existing address value 0.0.0.0 does not match fd00:1122:3344:101::14
402026-06-22T21:11:37.386ZINFOcrucible-agent (worker): reconfig required
412026-06-22T21:11:37.386ZINFOcrucible-agent (worker): using existing config property group
422026-06-22T21:11:37.386ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-4fca0064-3aed-4f1a-a315-49e5fc913e38
432026-06-22T21:11:37.387ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/4fca0064-3aed-4f1a-a315-49e5fc913e38
442026-06-22T21:11:37.387ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
452026-06-22T21:11:37.387ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::14
462026-06-22T21:11:37.387ZINFOcrucible-agent (worker): commit
472026-06-22T21:11:37.393ZINFOcrucible-agent (worker): ok!
482026-06-22T21:11:37.396ZINFOcrucible-agent (worker): creating missing snapshot instance snapshot-4fca0064-3aed-4f1a-a315-49e5fc913e38-c4125642-dafd-4e03-a862-e83165c96356
492026-06-22T21:11:37.397ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:snapshot-4fca0064-3aed-4f1a-a315-49e5fc913e38-c4125642-dafd-4e03-a862-e83165c96356
502026-06-22T21:11:37.405ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:snapshot-4fca0064-3aed-4f1a-a315-49e5fc913e38-c4125642-dafd-4e03-a862-e83165c96356
512026-06-22T21:11:37.406ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/4fca0064-3aed-4f1a-a315-49e5fc913e38/.zfs/snapshot/c4125642-dafd-4e03-a862-e83165c96356
522026-06-22T21:11:37.406ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
532026-06-22T21:11:37.406ZINFOcrucible-agent (worker): ensure mode SCF_TYPE_ASTRING ro
542026-06-22T21:11:37.406ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::14
552026-06-22T21:11:37.406ZINFOcrucible-agent (worker): commit
562026-06-22T21:11:37.409ZINFOcrucible-agent (worker): ok!
572026-06-22T21:11:37.413ZINFOcrucible-agent (worker): SMF ok!
582026-06-22T21:11:37.413ZINFOcrucible-agent (datafile): region 4fca0064-3aed-4f1a-a315-49e5fc913e38 running snapshot c4125642-dafd-4e03-a862-e83165c96356 state: Requested -> Created
592026-06-22T21:11:37.668ZINFOcrucible-agent (datafile): path is "/data/regions/4fca0064-3aed-4f1a-a315-49e5fc913e38"
602026-06-22T21:11:37.675ZINFOcrucible-agent (datafile): dataset is oxp_96ad0d72-7678-45ca-a377-d07769acd0e0/crucible/regions/4fca0064-3aed-4f1a-a315-49e5fc913e38
612026-06-22T21:11:37.682ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_96ad0d72-7678-45ca-a377-d07769acd0e0/crucible/regions/4fca0064-3aed-4f1a-a315-49e5fc913e38@c4125642-dafd-4e03-a862-e83165c96356
622026-06-22T21:11:37.682ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_96ad0d72-7678-45ca-a377-d07769acd0e0/crucible/regions/4fca0064-3aed-4f1a-a315-49e5fc913e38", "c4125642-dafd-4e03-a862-e83165c96356"]
632026-06-22T21:11:37.688ZINFOcrucible-agent (dropshot): request completed latency_us = 19991 local_addr = [fd00:1122:3344:101::14]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:48354 req_id = 426f4e69-40ab-41c0-adcb-05503ba19e0f response_code = 200 uri = /crucible/0/regions/4fca0064-3aed-4f1a-a315-49e5fc913e38/snapshots/c4125642-dafd-4e03-a862-e83165c96356/run
642026-06-22T21:11:43.191ZINFOcrucible-agent (datafile): region 443635f2-a331-4861-8fb4-e4e27a59941d state: Requested
652026-06-22T21:11:43.191ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
662026-06-22T21:11:43.191ZINFOcrucible-agent (dropshot): request completed latency_us = 274 local_addr = [fd00:1122:3344:101::14]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:48354 req_id = 352c7747-5dc0-48e7-ae38-4cc6a8331d98 response_code = 200 uri = /crucible/0/regions
672026-06-22T21:11:43.198ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_96ad0d72-7678-45ca-a377-d07769acd0e0/crucible/regions/443635f2-a331-4861-8fb4-e4e27a59941d
682026-06-22T21:11:43.198ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_96ad0d72-7678-45ca-a377-d07769acd0e0/crucible/regions/443635f2-a331-4861-8fb4-e4e27a59941d
692026-06-22T21:11:43.256ZINFOcrucible-agent (worker): creating region Region { id: RegionId("443635f2-a331-4861-8fb4-e4e27a59941d"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19002, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/443635f2-a331-4861-8fb4-e4e27a59941d" region = 443635f2-a331-4861-8fb4-e4e27a59941d
702026-06-22T21:11:43.256ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "443635f2-a331-4861-8fb4-e4e27a59941d" "--data" "/data/regions/443635f2-a331-4861-8fb4-e4e27a59941d" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 443635f2-a331-4861-8fb4-e4e27a59941d
712026-06-22T21:11:43.310ZINFOcrucible-agent (worker): region files created ok region = 443635f2-a331-4861-8fb4-e4e27a59941d
722026-06-22T21:11:43.310ZINFOcrucible-agent (datafile): region 443635f2-a331-4861-8fb4-e4e27a59941d state: Requested -> Created
732026-06-22T21:11:43.310ZINFOcrucible-agent (worker): applying SMF actions post create...
742026-06-22T21:11:43.311ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-443635f2-a331-4861-8fb4-e4e27a59941d
752026-06-22T21:11:43.324ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-443635f2-a331-4861-8fb4-e4e27a59941d
762026-06-22T21:11:43.330ZINFOcrucible-agent (worker): creating config property group
772026-06-22T21:11:43.331ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-443635f2-a331-4861-8fb4-e4e27a59941d
782026-06-22T21:11:43.332ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/443635f2-a331-4861-8fb4-e4e27a59941d
792026-06-22T21:11:43.332ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19002
802026-06-22T21:11:43.332ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::14
812026-06-22T21:11:43.332ZINFOcrucible-agent (worker): commit
822026-06-22T21:11:43.334ZINFOcrucible-agent (worker): ok!
832026-06-22T21:11:43.337ZINFOcrucible-agent (worker): existing directory value /dev/null does not match /data/regions/4fca0064-3aed-4f1a-a315-49e5fc913e38
842026-06-22T21:11:43.337ZINFOcrucible-agent (worker): existing port value 0 does not match 19000
852026-06-22T21:11:43.337ZINFOcrucible-agent (worker): existing address value 0.0.0.0 does not match fd00:1122:3344:101::14
862026-06-22T21:11:43.337ZINFOcrucible-agent (worker): reconfig required
872026-06-22T21:11:43.337ZINFOcrucible-agent (worker): using existing config property group
882026-06-22T21:11:43.337ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-4fca0064-3aed-4f1a-a315-49e5fc913e38
892026-06-22T21:11:43.340ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/4fca0064-3aed-4f1a-a315-49e5fc913e38
902026-06-22T21:11:43.340ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
912026-06-22T21:11:43.340ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::14
922026-06-22T21:11:43.340ZINFOcrucible-agent (worker): commit
932026-06-22T21:11:43.343ZINFOcrucible-agent (worker): ok!
942026-06-22T21:11:43.345ZINFOcrucible-agent (worker): SMF ok!
952026-06-22T21:11:43.436ZINFOcrucible-agent (dropshot): request completed latency_us = 64 local_addr = [fd00:1122:3344:101::14]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:48354 req_id = a1393a4e-d6ab-4441-b8b8-9a5a5511498f response_code = 200 uri = /crucible/0/regions
962026-06-22T21:12:08.528ZINFOcrucible-agent (dropshot): request completed latency_us = 72 local_addr = [fd00:1122:3344:101::14]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:48354 req_id = bc69b2c4-7559-472b-9a6c-c4db894de217 response_code = 200 uri = /crucible/0/regions/443635f2-a331-4861-8fb4-e4e27a59941d
972026-06-22T21:12:08.531ZINFOcrucible-agent (datafile): path is "/data/regions/443635f2-a331-4861-8fb4-e4e27a59941d"
982026-06-22T21:12:08.540ZINFOcrucible-agent (datafile): dataset is oxp_96ad0d72-7678-45ca-a377-d07769acd0e0/crucible/regions/443635f2-a331-4861-8fb4-e4e27a59941d
992026-06-22T21:12:08.548ZINFOcrucible-agent (datafile): region 443635f2-a331-4861-8fb4-e4e27a59941d state: Created -> Tombstoned
1002026-06-22T21:12:08.548ZINFOcrucible-agent (worker): applying SMF actions before removal...
1012026-06-22T21:12:08.548ZINFOcrucible-agent (dropshot): request completed latency_us = 17556 local_addr = [fd00:1122:3344:101::14]:32345 method = DELETE remote_addr = [fd00:1122:3344:101::c]:48354 req_id = 29723c49-0f93-4ea9-b00e-6ed9130cb5a6 response_code = 204 uri = /crucible/0/regions/443635f2-a331-4861-8fb4-e4e27a59941d
1022026-06-22T21:12:08.549ZINFOcrucible-agent (worker): disabling downstairs instance: downstairs-443635f2-a331-4861-8fb4-e4e27a59941d (instance states: (Some(Online), None))
1032026-06-22T21:12:08.553ZINFOcrucible-agent (worker): existing directory value /dev/null does not match /data/regions/4fca0064-3aed-4f1a-a315-49e5fc913e38
1042026-06-22T21:12:08.553ZINFOcrucible-agent (worker): existing port value 0 does not match 19000
1052026-06-22T21:12:08.553ZINFOcrucible-agent (worker): existing address value 0.0.0.0 does not match fd00:1122:3344:101::14
1062026-06-22T21:12:08.553ZINFOcrucible-agent (worker): reconfig required
1072026-06-22T21:12:08.553ZINFOcrucible-agent (worker): using existing config property group
1082026-06-22T21:12:08.553ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-4fca0064-3aed-4f1a-a315-49e5fc913e38
1092026-06-22T21:12:08.553ZINFOcrucible-agent (dropshot): request completed latency_us = 37 local_addr = [fd00:1122:3344:101::14]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:48354 req_id = 30755c2f-8507-4d01-8965-e4f5843abd0d response_code = 200 uri = /crucible/0/regions/443635f2-a331-4861-8fb4-e4e27a59941d
1102026-06-22T21:12:08.553ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/4fca0064-3aed-4f1a-a315-49e5fc913e38
1112026-06-22T21:12:08.553ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
1122026-06-22T21:12:08.553ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::14
1132026-06-22T21:12:08.554ZINFOcrucible-agent (worker): commit
1142026-06-22T21:12:08.557ZINFOcrucible-agent (worker): ok!
1152026-06-22T21:12:08.559ZINFOcrucible-agent (worker): SMF ok!
1162026-06-22T21:12:08.568ZINFOcrucible-agent (worker): deleting zfs dataset "oxp_96ad0d72-7678-45ca-a377-d07769acd0e0/crucible/regions/443635f2-a331-4861-8fb4-e4e27a59941d" region = 443635f2-a331-4861-8fb4-e4e27a59941d
1172026-06-22T21:12:08.616ZINFOcrucible-agent (dropshot): request completed latency_us = 48 local_addr = [fd00:1122:3344:101::14]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:48354 req_id = 69227c88-31ae-4d7b-b5cc-05a5263865d2 response_code = 200 uri = /crucible/0/regions/443635f2-a331-4861-8fb4-e4e27a59941d
1182026-06-22T21:12:08.637ZINFOcrucible-agent (datafile): region 443635f2-a331-4861-8fb4-e4e27a59941d state: Tombstoned -> Destroyed
1192026-06-22T21:12:08.735ZINFOcrucible-agent (dropshot): request completed latency_us = 48 local_addr = [fd00:1122:3344:101::14]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:48354 req_id = 52d8ecac-fdef-4026-bbfc-7638da24ce92 response_code = 200 uri = /crucible/0/regions/443635f2-a331-4861-8fb4-e4e27a59941d