Max level shown:
12026-07-02T18:45:38.867ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::18]:32345 remote_addr = [fd00:1122:3344:101::b]:56726
22026-07-02T18:45:38.897ZINFOcrucible-agent (datafile): region 4125e9cc-a33b-48ee-96a2-aa56eeb291bb state: Requested
32026-07-02T18:45:38.901ZINFOcrucible-agent (worker): Region size:1073741824 reservation:1342177280 quota:3221225472
42026-07-02T18:45:38.903ZINFOcrucible-agent (dropshot): request completed latency_us = 15906 local_addr = [fd00:1122:3344:101::18]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:56726 req_id = ceaf41b9-7a97-4e7b-be3c-8ee5addfee23 response_code = 200 uri = /crucible/0/regions
52026-07-02T18:45:38.913ZINFOcrucible-agent (worker): zfs set reservation of 1342177280 for oxp_8fa5211b-4f32-4a50-bba3-dd5e3d80ed45/crucible/regions/4125e9cc-a33b-48ee-96a2-aa56eeb291bb
62026-07-02T18:45:38.913ZINFOcrucible-agent (worker): zfs set quota of 3221225472 for oxp_8fa5211b-4f32-4a50-bba3-dd5e3d80ed45/crucible/regions/4125e9cc-a33b-48ee-96a2-aa56eeb291bb
72026-07-02T18:45:38.959ZINFOcrucible-agent (worker): creating region Region { id: RegionId("4125e9cc-a33b-48ee-96a2-aa56eeb291bb"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 16, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/4125e9cc-a33b-48ee-96a2-aa56eeb291bb" region = 4125e9cc-a33b-48ee-96a2-aa56eeb291bb
82026-07-02T18:45:38.961ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "4125e9cc-a33b-48ee-96a2-aa56eeb291bb" "--data" "/data/regions/4125e9cc-a33b-48ee-96a2-aa56eeb291bb" "--block-size" "512" "--extent-size" "131072" "--extent-count" "16" "--encrypted" region = 4125e9cc-a33b-48ee-96a2-aa56eeb291bb
92026-07-02T18:45:39.073ZINFOcrucible-agent (worker): region files created ok region = 4125e9cc-a33b-48ee-96a2-aa56eeb291bb
102026-07-02T18:45:39.075ZINFOcrucible-agent (datafile): region 4125e9cc-a33b-48ee-96a2-aa56eeb291bb state: Requested -> Created
112026-07-02T18:45:39.075ZINFOcrucible-agent (worker): applying SMF actions post create...
122026-07-02T18:45:39.077ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-4125e9cc-a33b-48ee-96a2-aa56eeb291bb
132026-07-02T18:45:39.084ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-4125e9cc-a33b-48ee-96a2-aa56eeb291bb
142026-07-02T18:45:39.094ZINFOcrucible-agent (worker): creating config property group
152026-07-02T18:45:39.095ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-4125e9cc-a33b-48ee-96a2-aa56eeb291bb
162026-07-02T18:45:39.096ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/4125e9cc-a33b-48ee-96a2-aa56eeb291bb
172026-07-02T18:45:39.098ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
182026-07-02T18:45:39.098ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::18
192026-07-02T18:45:39.098ZINFOcrucible-agent (worker): commit
202026-07-02T18:45:39.106ZINFOcrucible-agent (worker): ok!
212026-07-02T18:45:39.109ZINFOcrucible-agent (worker): SMF ok!
222026-07-02T18:45:39.245ZINFOcrucible-agent (dropshot): request completed latency_us = 73 local_addr = [fd00:1122:3344:101::18]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:56726 req_id = 818e22d2-e32d-4141-9f7a-b9fe9d09a044 response_code = 200 uri = /crucible/0/regions
232026-07-02T18:45:39.591ZINFOcrucible-agent (dropshot): request completed latency_us = 82 local_addr = [fd00:1122:3344:101::18]:32345 method = GET remote_addr = [fd00:1122:3344:101::b]:56726 req_id = 56e3ded7-514c-40b0-a9dd-4f1cb3b93b26 response_code = 200 uri = /crucible/0/regions/4125e9cc-a33b-48ee-96a2-aa56eeb291bb
242026-07-02T18:45:39.593ZINFOcrucible-agent (datafile): path is "/data/regions/4125e9cc-a33b-48ee-96a2-aa56eeb291bb"
252026-07-02T18:45:39.601ZINFOcrucible-agent (datafile): dataset is oxp_8fa5211b-4f32-4a50-bba3-dd5e3d80ed45/crucible/regions/4125e9cc-a33b-48ee-96a2-aa56eeb291bb
262026-07-02T18:45:39.608ZINFOcrucible-agent (datafile): region 4125e9cc-a33b-48ee-96a2-aa56eeb291bb state: Created -> Tombstoned
272026-07-02T18:45:39.608ZINFOcrucible-agent (worker): applying SMF actions before removal...
282026-07-02T18:45:39.608ZINFOcrucible-agent (dropshot): request completed latency_us = 14539 local_addr = [fd00:1122:3344:101::18]:32345 method = DELETE remote_addr = [fd00:1122:3344:101::b]:56726 req_id = e241a34d-e2e3-4bc9-99aa-879329a2eadb response_code = 204 uri = /crucible/0/regions/4125e9cc-a33b-48ee-96a2-aa56eeb291bb
292026-07-02T18:45:39.609ZINFOcrucible-agent (worker): disabling downstairs instance: downstairs-4125e9cc-a33b-48ee-96a2-aa56eeb291bb (instance states: (Some(Online), None))
302026-07-02T18:45:39.612ZINFOcrucible-agent (worker): SMF ok!
312026-07-02T18:45:39.613ZINFOcrucible-agent (dropshot): request completed latency_us = 71 local_addr = [fd00:1122:3344:101::18]:32345 method = GET remote_addr = [fd00:1122:3344:101::b]:56726 req_id = 9faccf8a-123d-4b8f-86f6-d446cbbc114d response_code = 200 uri = /crucible/0/regions/4125e9cc-a33b-48ee-96a2-aa56eeb291bb
322026-07-02T18:45:39.620ZINFOcrucible-agent (worker): deleting zfs dataset "oxp_8fa5211b-4f32-4a50-bba3-dd5e3d80ed45/crucible/regions/4125e9cc-a33b-48ee-96a2-aa56eeb291bb" region = 4125e9cc-a33b-48ee-96a2-aa56eeb291bb
332026-07-02T18:45:39.683ZINFOcrucible-agent (datafile): region 4125e9cc-a33b-48ee-96a2-aa56eeb291bb state: Tombstoned -> Destroyed
342026-07-02T18:45:39.695ZINFOcrucible-agent (dropshot): request completed latency_us = 46 local_addr = [fd00:1122:3344:101::18]:32345 method = GET remote_addr = [fd00:1122:3344:101::b]:56726 req_id = 56f77469-c253-413e-a8c9-8ca6bf39723c response_code = 200 uri = /crucible/0/regions/4125e9cc-a33b-48ee-96a2-aa56eeb291bb
352026-07-02T18:45:41.499ZINFOcrucible-agent (datafile): region 3010b456-b277-474f-8ebf-d2d5f6694bfa state: Requested
362026-07-02T18:45:41.499ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
372026-07-02T18:45:41.499ZINFOcrucible-agent (dropshot): request completed latency_us = 192 local_addr = [fd00:1122:3344:101::18]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:56726 req_id = 9b8f70b6-1e5b-4155-9975-09f07b4cfed3 response_code = 200 uri = /crucible/0/regions
382026-07-02T18:45:41.506ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_8fa5211b-4f32-4a50-bba3-dd5e3d80ed45/crucible/regions/3010b456-b277-474f-8ebf-d2d5f6694bfa
392026-07-02T18:45:41.506ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_8fa5211b-4f32-4a50-bba3-dd5e3d80ed45/crucible/regions/3010b456-b277-474f-8ebf-d2d5f6694bfa
402026-07-02T18:45:41.547ZINFOcrucible-agent (worker): creating region Region { id: RegionId("3010b456-b277-474f-8ebf-d2d5f6694bfa"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/3010b456-b277-474f-8ebf-d2d5f6694bfa" region = 3010b456-b277-474f-8ebf-d2d5f6694bfa
412026-07-02T18:45:41.547ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "3010b456-b277-474f-8ebf-d2d5f6694bfa" "--data" "/data/regions/3010b456-b277-474f-8ebf-d2d5f6694bfa" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 3010b456-b277-474f-8ebf-d2d5f6694bfa
422026-07-02T18:45:41.597ZINFOcrucible-agent (worker): region files created ok region = 3010b456-b277-474f-8ebf-d2d5f6694bfa
432026-07-02T18:45:41.597ZINFOcrucible-agent (datafile): region 3010b456-b277-474f-8ebf-d2d5f6694bfa state: Requested -> Created
442026-07-02T18:45:41.597ZINFOcrucible-agent (worker): applying SMF actions post create...
452026-07-02T18:45:41.599ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-3010b456-b277-474f-8ebf-d2d5f6694bfa
462026-07-02T18:45:41.600ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-3010b456-b277-474f-8ebf-d2d5f6694bfa
472026-07-02T18:45:41.604ZINFOcrucible-agent (worker): creating config property group
482026-07-02T18:45:41.606ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-3010b456-b277-474f-8ebf-d2d5f6694bfa
492026-07-02T18:45:41.606ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/3010b456-b277-474f-8ebf-d2d5f6694bfa
502026-07-02T18:45:41.606ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
512026-07-02T18:45:41.606ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::18
522026-07-02T18:45:41.606ZINFOcrucible-agent (worker): commit
532026-07-02T18:45:41.608ZINFOcrucible-agent (worker): ok!
542026-07-02T18:45:41.611ZINFOcrucible-agent (worker): SMF ok!
552026-07-02T18:45:41.685ZINFOcrucible-agent (dropshot): request completed latency_us = 69 local_addr = [fd00:1122:3344:101::18]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:56726 req_id = 349e86be-0b69-4e85-9e74-88ecd254842d response_code = 200 uri = /crucible/0/regions
562026-07-02T18:56:43.392ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::18]:32345 remote_addr = [fd00:1122:3344:101::b]:47339
572026-07-02T18:56:43.393ZINFOcrucible-agent (datafile): region 26a89e3c-69d0-467a-97d3-75d2c2e8c116 state: Requested
582026-07-02T18:56:43.393ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
592026-07-02T18:56:43.393ZINFOcrucible-agent (dropshot): request completed latency_us = 930 local_addr = [fd00:1122:3344:101::18]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:47339 req_id = d1ec2dbd-d5c8-4bdd-b06b-a41db17bbf29 response_code = 200 uri = /crucible/0/regions
602026-07-02T18:56:43.402ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_8fa5211b-4f32-4a50-bba3-dd5e3d80ed45/crucible/regions/26a89e3c-69d0-467a-97d3-75d2c2e8c116
612026-07-02T18:56:43.402ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_8fa5211b-4f32-4a50-bba3-dd5e3d80ed45/crucible/regions/26a89e3c-69d0-467a-97d3-75d2c2e8c116
622026-07-02T18:56:43.437ZINFOcrucible-agent (worker): creating region Region { id: RegionId("26a89e3c-69d0-467a-97d3-75d2c2e8c116"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19001, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/26a89e3c-69d0-467a-97d3-75d2c2e8c116" region = 26a89e3c-69d0-467a-97d3-75d2c2e8c116
632026-07-02T18:56:43.438ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "26a89e3c-69d0-467a-97d3-75d2c2e8c116" "--data" "/data/regions/26a89e3c-69d0-467a-97d3-75d2c2e8c116" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 26a89e3c-69d0-467a-97d3-75d2c2e8c116
642026-07-02T18:56:43.484ZINFOcrucible-agent (worker): region files created ok region = 26a89e3c-69d0-467a-97d3-75d2c2e8c116
652026-07-02T18:56:43.484ZINFOcrucible-agent (datafile): region 26a89e3c-69d0-467a-97d3-75d2c2e8c116 state: Requested -> Created
662026-07-02T18:56:43.484ZINFOcrucible-agent (worker): applying SMF actions post create...
672026-07-02T18:56:43.485ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-26a89e3c-69d0-467a-97d3-75d2c2e8c116
682026-07-02T18:56:43.500ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-26a89e3c-69d0-467a-97d3-75d2c2e8c116
692026-07-02T18:56:43.508ZINFOcrucible-agent (worker): creating config property group
702026-07-02T18:56:43.509ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-26a89e3c-69d0-467a-97d3-75d2c2e8c116
712026-07-02T18:56:43.509ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/26a89e3c-69d0-467a-97d3-75d2c2e8c116
722026-07-02T18:56:43.509ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
732026-07-02T18:56:43.509ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::18
742026-07-02T18:56:43.510ZINFOcrucible-agent (worker): commit
752026-07-02T18:56:43.511ZINFOcrucible-agent (worker): ok!
762026-07-02T18:56:43.515ZINFOcrucible-agent (worker): SMF ok!
772026-07-02T18:56:43.639ZINFOcrucible-agent (dropshot): request completed latency_us = 94 local_addr = [fd00:1122:3344:101::18]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:47339 req_id = 0fe26463-65b4-4734-b99b-959813366a6a response_code = 200 uri = /crucible/0/regions
782026-07-02T18:56:44.692ZINFOcrucible-agent (dropshot): request completed latency_us = 52 local_addr = [fd00:1122:3344:101::18]:32345 method = GET remote_addr = [fd00:1122:3344:101::b]:47339 req_id = 886bacae-22d8-409a-8184-f88f17c6d39a response_code = 200 uri = /crucible/0/regions/3010b456-b277-474f-8ebf-d2d5f6694bfa
792026-07-02T18:56:44.697ZINFOcrucible-agent (datafile): path is "/data/regions/3010b456-b277-474f-8ebf-d2d5f6694bfa"
802026-07-02T18:56:44.704ZINFOcrucible-agent (datafile): dataset is oxp_8fa5211b-4f32-4a50-bba3-dd5e3d80ed45/crucible/regions/3010b456-b277-474f-8ebf-d2d5f6694bfa
812026-07-02T18:56:44.710ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_8fa5211b-4f32-4a50-bba3-dd5e3d80ed45/crucible/regions/3010b456-b277-474f-8ebf-d2d5f6694bfa@3a162d34-2c29-4971-b389-772810b80b52
822026-07-02T18:56:44.710ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_8fa5211b-4f32-4a50-bba3-dd5e3d80ed45/crucible/regions/3010b456-b277-474f-8ebf-d2d5f6694bfa", "3a162d34-2c29-4971-b389-772810b80b52"]
832026-07-02T18:56:44.723ZINFOcrucible-agent (dropshot): request completed latency_us = 25507 local_addr = [fd00:1122:3344:101::18]:32345 method = GET remote_addr = [fd00:1122:3344:101::b]:47339 req_id = f4b74540-7f3f-47cb-8415-88bb6ac63558 response_code = 200 uri = /crucible/0/regions/3010b456-b277-474f-8ebf-d2d5f6694bfa/snapshots/3a162d34-2c29-4971-b389-772810b80b52
842026-07-02T18:56:44.729ZINFOcrucible-agent (datafile): path is "/data/regions/3010b456-b277-474f-8ebf-d2d5f6694bfa"
852026-07-02T18:56:44.736ZINFOcrucible-agent (datafile): dataset is oxp_8fa5211b-4f32-4a50-bba3-dd5e3d80ed45/crucible/regions/3010b456-b277-474f-8ebf-d2d5f6694bfa
862026-07-02T18:56:44.742ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_8fa5211b-4f32-4a50-bba3-dd5e3d80ed45/crucible/regions/3010b456-b277-474f-8ebf-d2d5f6694bfa@3a162d34-2c29-4971-b389-772810b80b52
872026-07-02T18:56:44.742ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_8fa5211b-4f32-4a50-bba3-dd5e3d80ed45/crucible/regions/3010b456-b277-474f-8ebf-d2d5f6694bfa", "3a162d34-2c29-4971-b389-772810b80b52"]
882026-07-02T18:56:44.750ZINFOcrucible-agent (datafile): requesting running snapshot 3010b456-b277-474f-8ebf-d2d5f6694bfa-3a162d34-2c29-4971-b389-772810b80b52 state: Requested
892026-07-02T18:56:44.753ZINFOcrucible-agent (worker): applying SMF actions for region 3010b456-b277-474f-8ebf-d2d5f6694bfa running snapshot 3a162d34-2c29-4971-b389-772810b80b52 (state Requested)...
902026-07-02T18:56:44.753ZINFOcrucible-agent (dropshot): request completed latency_us = 23935 local_addr = [fd00:1122:3344:101::18]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:47339 req_id = a979cf79-eb2d-4b95-ab92-9fad86915819 response_code = 200 uri = /crucible/0/regions/3010b456-b277-474f-8ebf-d2d5f6694bfa/snapshots/3a162d34-2c29-4971-b389-772810b80b52/run
912026-07-02T18:56:44.758ZINFOcrucible-agent (worker): creating missing snapshot instance snapshot-3010b456-b277-474f-8ebf-d2d5f6694bfa-3a162d34-2c29-4971-b389-772810b80b52
922026-07-02T18:56:44.759ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:snapshot-3010b456-b277-474f-8ebf-d2d5f6694bfa-3a162d34-2c29-4971-b389-772810b80b52
932026-07-02T18:56:44.765ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:snapshot-3010b456-b277-474f-8ebf-d2d5f6694bfa-3a162d34-2c29-4971-b389-772810b80b52
942026-07-02T18:56:44.765ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/3010b456-b277-474f-8ebf-d2d5f6694bfa/.zfs/snapshot/3a162d34-2c29-4971-b389-772810b80b52
952026-07-02T18:56:44.765ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19002
962026-07-02T18:56:44.765ZINFOcrucible-agent (worker): ensure mode SCF_TYPE_ASTRING ro
972026-07-02T18:56:44.765ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::18
982026-07-02T18:56:44.765ZINFOcrucible-agent (worker): commit
992026-07-02T18:56:44.767ZINFOcrucible-agent (worker): ok!
1002026-07-02T18:56:44.770ZINFOcrucible-agent (worker): SMF ok!
1012026-07-02T18:56:44.770ZINFOcrucible-agent (datafile): region 3010b456-b277-474f-8ebf-d2d5f6694bfa running snapshot 3a162d34-2c29-4971-b389-772810b80b52 state: Requested -> Created
1022026-07-02T18:56:44.969ZINFOcrucible-agent (datafile): path is "/data/regions/3010b456-b277-474f-8ebf-d2d5f6694bfa"
1032026-07-02T18:56:44.976ZINFOcrucible-agent (datafile): dataset is oxp_8fa5211b-4f32-4a50-bba3-dd5e3d80ed45/crucible/regions/3010b456-b277-474f-8ebf-d2d5f6694bfa
1042026-07-02T18:56:44.982ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_8fa5211b-4f32-4a50-bba3-dd5e3d80ed45/crucible/regions/3010b456-b277-474f-8ebf-d2d5f6694bfa@3a162d34-2c29-4971-b389-772810b80b52
1052026-07-02T18:56:44.982ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_8fa5211b-4f32-4a50-bba3-dd5e3d80ed45/crucible/regions/3010b456-b277-474f-8ebf-d2d5f6694bfa", "3a162d34-2c29-4971-b389-772810b80b52"]
1062026-07-02T18:56:44.988ZINFOcrucible-agent (dropshot): request completed latency_us = 18885 local_addr = [fd00:1122:3344:101::18]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:47339 req_id = 4d69a816-92b1-4f98-86bc-de8317e97f95 response_code = 200 uri = /crucible/0/regions/3010b456-b277-474f-8ebf-d2d5f6694bfa/snapshots/3a162d34-2c29-4971-b389-772810b80b52/run