Max level shown:
12026-05-19T00:00:07.711ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::17]:32345 remote_addr = [fd00:1122:3344:101::a]:64670
22026-05-19T00:00:07.758ZINFOcrucible-agent (datafile): region b422453a-f725-47dc-9cfb-f3f967fb9594 state: Requested
32026-05-19T00:00:07.761ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
42026-05-19T00:00:07.761ZINFOcrucible-agent (dropshot): request completed latency_us = 20501 local_addr = [fd00:1122:3344:101::17]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:64670 req_id = eb39ca5f-8997-4363-8cc1-70f84ef95912 response_code = 200 uri = /crucible/0/regions
52026-05-19T00:00:07.769ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_a0d4bba2-a17b-41da-aa32-bb2b7de0796b/crucible/regions/b422453a-f725-47dc-9cfb-f3f967fb9594
62026-05-19T00:00:07.769ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_a0d4bba2-a17b-41da-aa32-bb2b7de0796b/crucible/regions/b422453a-f725-47dc-9cfb-f3f967fb9594
72026-05-19T00:00:07.811ZINFOcrucible-agent (worker): creating region Region { id: RegionId("b422453a-f725-47dc-9cfb-f3f967fb9594"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/b422453a-f725-47dc-9cfb-f3f967fb9594" region = b422453a-f725-47dc-9cfb-f3f967fb9594
82026-05-19T00:00:07.814ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "b422453a-f725-47dc-9cfb-f3f967fb9594" "--data" "/data/regions/b422453a-f725-47dc-9cfb-f3f967fb9594" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = b422453a-f725-47dc-9cfb-f3f967fb9594
92026-05-19T00:00:08.074ZINFOcrucible-agent (dropshot): request completed latency_us = 58 local_addr = [fd00:1122:3344:101::17]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:64670 req_id = 436a357e-3d2d-4085-9b4a-12e2afe524c3 response_code = 200 uri = /crucible/0/regions
102026-05-19T00:00:08.152ZINFOcrucible-agent (worker): region files created ok region = b422453a-f725-47dc-9cfb-f3f967fb9594
112026-05-19T00:00:08.152ZINFOcrucible-agent (datafile): region b422453a-f725-47dc-9cfb-f3f967fb9594 state: Requested -> Created
122026-05-19T00:00:08.152ZINFOcrucible-agent (worker): applying SMF actions post create...
132026-05-19T00:00:08.152ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-b422453a-f725-47dc-9cfb-f3f967fb9594
142026-05-19T00:00:08.157ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-b422453a-f725-47dc-9cfb-f3f967fb9594
152026-05-19T00:00:08.167ZINFOcrucible-agent (worker): creating config property group
162026-05-19T00:00:08.169ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-b422453a-f725-47dc-9cfb-f3f967fb9594
172026-05-19T00:00:08.169ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/b422453a-f725-47dc-9cfb-f3f967fb9594
182026-05-19T00:00:08.169ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
192026-05-19T00:00:08.169ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::17
202026-05-19T00:00:08.169ZINFOcrucible-agent (worker): commit
212026-05-19T00:00:08.171ZINFOcrucible-agent (worker): ok!
222026-05-19T00:00:08.174ZINFOcrucible-agent (worker): SMF ok!
232026-05-19T00:00:08.627ZINFOcrucible-agent (dropshot): request completed latency_us = 62 local_addr = [fd00:1122:3344:101::17]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:64670 req_id = 4406667d-bd72-48e1-a00d-abdefc0ec923 response_code = 200 uri = /crucible/0/regions
242026-05-19T00:11:12.768ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::17]:32345 remote_addr = [fd00:1122:3344:101::a]:46412
252026-05-19T00:11:12.768ZINFOcrucible-agent (datafile): region d5af5b06-1881-4664-b0f9-82e4128b62aa state: Requested
262026-05-19T00:11:12.769ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
272026-05-19T00:11:12.769ZINFOcrucible-agent (dropshot): request completed latency_us = 810 local_addr = [fd00:1122:3344:101::17]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:46412 req_id = a2705d11-064f-4109-a470-7331befcf34e response_code = 200 uri = /crucible/0/regions
282026-05-19T00:11:12.779ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_a0d4bba2-a17b-41da-aa32-bb2b7de0796b/crucible/regions/d5af5b06-1881-4664-b0f9-82e4128b62aa
292026-05-19T00:11:12.779ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_a0d4bba2-a17b-41da-aa32-bb2b7de0796b/crucible/regions/d5af5b06-1881-4664-b0f9-82e4128b62aa
302026-05-19T00:11:12.840ZINFOcrucible-agent (worker): creating region Region { id: RegionId("d5af5b06-1881-4664-b0f9-82e4128b62aa"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19001, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/d5af5b06-1881-4664-b0f9-82e4128b62aa" region = d5af5b06-1881-4664-b0f9-82e4128b62aa
312026-05-19T00:11:12.840ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "d5af5b06-1881-4664-b0f9-82e4128b62aa" "--data" "/data/regions/d5af5b06-1881-4664-b0f9-82e4128b62aa" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = d5af5b06-1881-4664-b0f9-82e4128b62aa
322026-05-19T00:11:12.897ZINFOcrucible-agent (worker): region files created ok region = d5af5b06-1881-4664-b0f9-82e4128b62aa
332026-05-19T00:11:12.897ZINFOcrucible-agent (datafile): region d5af5b06-1881-4664-b0f9-82e4128b62aa state: Requested -> Created
342026-05-19T00:11:12.897ZINFOcrucible-agent (worker): applying SMF actions post create...
352026-05-19T00:11:12.909ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-d5af5b06-1881-4664-b0f9-82e4128b62aa
362026-05-19T00:11:12.913ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-d5af5b06-1881-4664-b0f9-82e4128b62aa
372026-05-19T00:11:12.919ZINFOcrucible-agent (worker): creating config property group
382026-05-19T00:11:12.920ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-d5af5b06-1881-4664-b0f9-82e4128b62aa
392026-05-19T00:11:12.921ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/d5af5b06-1881-4664-b0f9-82e4128b62aa
402026-05-19T00:11:12.921ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
412026-05-19T00:11:12.921ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::17
422026-05-19T00:11:12.921ZINFOcrucible-agent (worker): commit
432026-05-19T00:11:12.923ZINFOcrucible-agent (worker): ok!
442026-05-19T00:11:12.926ZINFOcrucible-agent (worker): SMF ok!
452026-05-19T00:11:12.968ZINFOcrucible-agent (dropshot): request completed latency_us = 62 local_addr = [fd00:1122:3344:101::17]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:46412 req_id = 73d17d73-30d6-40d7-bc2c-f280719e9198 response_code = 200 uri = /crucible/0/regions
462026-05-19T00:11:13.627ZINFOcrucible-agent (dropshot): request completed latency_us = 88 local_addr = [fd00:1122:3344:101::17]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:46412 req_id = 29d342c0-a133-4e62-965f-000b2bbe5548 response_code = 200 uri = /crucible/0/regions/b422453a-f725-47dc-9cfb-f3f967fb9594
472026-05-19T00:11:13.630ZINFOcrucible-agent (datafile): path is "/data/regions/b422453a-f725-47dc-9cfb-f3f967fb9594"
482026-05-19T00:11:13.638ZINFOcrucible-agent (datafile): dataset is oxp_a0d4bba2-a17b-41da-aa32-bb2b7de0796b/crucible/regions/b422453a-f725-47dc-9cfb-f3f967fb9594
492026-05-19T00:11:13.646ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_a0d4bba2-a17b-41da-aa32-bb2b7de0796b/crucible/regions/b422453a-f725-47dc-9cfb-f3f967fb9594@bb8e8d19-d8ae-479e-9cea-8198c6ddf4aa
502026-05-19T00:11:13.646ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_a0d4bba2-a17b-41da-aa32-bb2b7de0796b/crucible/regions/b422453a-f725-47dc-9cfb-f3f967fb9594", "bb8e8d19-d8ae-479e-9cea-8198c6ddf4aa"]
512026-05-19T00:11:13.656ZINFOcrucible-agent (dropshot): request completed latency_us = 25291 local_addr = [fd00:1122:3344:101::17]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:46412 req_id = e5835a4b-0c58-4322-b3bd-305a53b5b09c response_code = 200 uri = /crucible/0/regions/b422453a-f725-47dc-9cfb-f3f967fb9594/snapshots/bb8e8d19-d8ae-479e-9cea-8198c6ddf4aa
522026-05-19T00:11:13.659ZINFOcrucible-agent (datafile): path is "/data/regions/b422453a-f725-47dc-9cfb-f3f967fb9594"
532026-05-19T00:11:13.666ZINFOcrucible-agent (datafile): dataset is oxp_a0d4bba2-a17b-41da-aa32-bb2b7de0796b/crucible/regions/b422453a-f725-47dc-9cfb-f3f967fb9594
542026-05-19T00:11:13.672ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_a0d4bba2-a17b-41da-aa32-bb2b7de0796b/crucible/regions/b422453a-f725-47dc-9cfb-f3f967fb9594@bb8e8d19-d8ae-479e-9cea-8198c6ddf4aa
552026-05-19T00:11:13.672ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_a0d4bba2-a17b-41da-aa32-bb2b7de0796b/crucible/regions/b422453a-f725-47dc-9cfb-f3f967fb9594", "bb8e8d19-d8ae-479e-9cea-8198c6ddf4aa"]
562026-05-19T00:11:13.678ZINFOcrucible-agent (datafile): requesting running snapshot b422453a-f725-47dc-9cfb-f3f967fb9594-bb8e8d19-d8ae-479e-9cea-8198c6ddf4aa state: Requested
572026-05-19T00:11:13.678ZINFOcrucible-agent (worker): applying SMF actions for region b422453a-f725-47dc-9cfb-f3f967fb9594 running snapshot bb8e8d19-d8ae-479e-9cea-8198c6ddf4aa (state Requested)...
582026-05-19T00:11:13.678ZINFOcrucible-agent (dropshot): request completed latency_us = 18871 local_addr = [fd00:1122:3344:101::17]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:46412 req_id = 255edc08-fb7c-4263-9561-b7610cb4ab1d response_code = 200 uri = /crucible/0/regions/b422453a-f725-47dc-9cfb-f3f967fb9594/snapshots/bb8e8d19-d8ae-479e-9cea-8198c6ddf4aa/run
592026-05-19T00:11:13.683ZINFOcrucible-agent (worker): creating missing snapshot instance snapshot-b422453a-f725-47dc-9cfb-f3f967fb9594-bb8e8d19-d8ae-479e-9cea-8198c6ddf4aa
602026-05-19T00:11:13.687ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:snapshot-b422453a-f725-47dc-9cfb-f3f967fb9594-bb8e8d19-d8ae-479e-9cea-8198c6ddf4aa
612026-05-19T00:11:13.692ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:snapshot-b422453a-f725-47dc-9cfb-f3f967fb9594-bb8e8d19-d8ae-479e-9cea-8198c6ddf4aa
622026-05-19T00:11:13.693ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/b422453a-f725-47dc-9cfb-f3f967fb9594/.zfs/snapshot/bb8e8d19-d8ae-479e-9cea-8198c6ddf4aa
632026-05-19T00:11:13.693ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19002
642026-05-19T00:11:13.693ZINFOcrucible-agent (worker): ensure mode SCF_TYPE_ASTRING ro
652026-05-19T00:11:13.693ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::17
662026-05-19T00:11:13.693ZINFOcrucible-agent (worker): commit
672026-05-19T00:11:13.695ZINFOcrucible-agent (worker): ok!
682026-05-19T00:11:13.698ZINFOcrucible-agent (worker): SMF ok!
692026-05-19T00:11:13.698ZINFOcrucible-agent (datafile): region b422453a-f725-47dc-9cfb-f3f967fb9594 running snapshot bb8e8d19-d8ae-479e-9cea-8198c6ddf4aa state: Requested -> Created
702026-05-19T00:11:14.009ZINFOcrucible-agent (datafile): path is "/data/regions/b422453a-f725-47dc-9cfb-f3f967fb9594"
712026-05-19T00:11:14.015ZINFOcrucible-agent (datafile): dataset is oxp_a0d4bba2-a17b-41da-aa32-bb2b7de0796b/crucible/regions/b422453a-f725-47dc-9cfb-f3f967fb9594
722026-05-19T00:11:14.021ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_a0d4bba2-a17b-41da-aa32-bb2b7de0796b/crucible/regions/b422453a-f725-47dc-9cfb-f3f967fb9594@bb8e8d19-d8ae-479e-9cea-8198c6ddf4aa
732026-05-19T00:11:14.021ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_a0d4bba2-a17b-41da-aa32-bb2b7de0796b/crucible/regions/b422453a-f725-47dc-9cfb-f3f967fb9594", "bb8e8d19-d8ae-479e-9cea-8198c6ddf4aa"]
742026-05-19T00:11:14.025ZINFOcrucible-agent (dropshot): request completed latency_us = 16242 local_addr = [fd00:1122:3344:101::17]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:46412 req_id = b799f4d9-481c-4e8f-990b-4a02ce81c1bf response_code = 200 uri = /crucible/0/regions/b422453a-f725-47dc-9cfb-f3f967fb9594/snapshots/bb8e8d19-d8ae-479e-9cea-8198c6ddf4aa/run
752026-05-19T00:11:18.206ZINFOcrucible-agent (datafile): region d8b88646-4532-42c7-8c3f-61943c1824c3 state: Requested
762026-05-19T00:11:18.206ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
772026-05-19T00:11:18.206ZINFOcrucible-agent (dropshot): request completed latency_us = 187 local_addr = [fd00:1122:3344:101::17]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:46412 req_id = 56cd45d2-f9e7-4bd7-81e0-aa60292e0482 response_code = 200 uri = /crucible/0/regions
782026-05-19T00:11:18.212ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_a0d4bba2-a17b-41da-aa32-bb2b7de0796b/crucible/regions/d8b88646-4532-42c7-8c3f-61943c1824c3
792026-05-19T00:11:18.212ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_a0d4bba2-a17b-41da-aa32-bb2b7de0796b/crucible/regions/d8b88646-4532-42c7-8c3f-61943c1824c3
802026-05-19T00:11:18.255ZINFOcrucible-agent (worker): creating region Region { id: RegionId("d8b88646-4532-42c7-8c3f-61943c1824c3"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19003, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/d8b88646-4532-42c7-8c3f-61943c1824c3" region = d8b88646-4532-42c7-8c3f-61943c1824c3
812026-05-19T00:11:18.255ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "d8b88646-4532-42c7-8c3f-61943c1824c3" "--data" "/data/regions/d8b88646-4532-42c7-8c3f-61943c1824c3" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = d8b88646-4532-42c7-8c3f-61943c1824c3
822026-05-19T00:11:18.303ZINFOcrucible-agent (worker): region files created ok region = d8b88646-4532-42c7-8c3f-61943c1824c3
832026-05-19T00:11:18.303ZINFOcrucible-agent (datafile): region d8b88646-4532-42c7-8c3f-61943c1824c3 state: Requested -> Created
842026-05-19T00:11:18.303ZINFOcrucible-agent (worker): applying SMF actions post create...
852026-05-19T00:11:18.304ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-d8b88646-4532-42c7-8c3f-61943c1824c3
862026-05-19T00:11:18.306ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-d8b88646-4532-42c7-8c3f-61943c1824c3
872026-05-19T00:11:18.309ZINFOcrucible-agent (worker): creating config property group
882026-05-19T00:11:18.310ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-d8b88646-4532-42c7-8c3f-61943c1824c3
892026-05-19T00:11:18.311ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/d8b88646-4532-42c7-8c3f-61943c1824c3
902026-05-19T00:11:18.311ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19003
912026-05-19T00:11:18.311ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::17
922026-05-19T00:11:18.311ZINFOcrucible-agent (worker): commit
932026-05-19T00:11:18.312ZINFOcrucible-agent (worker): ok!
942026-05-19T00:11:18.316ZINFOcrucible-agent (worker): SMF ok!
952026-05-19T00:11:18.509ZINFOcrucible-agent (dropshot): request completed latency_us = 64 local_addr = [fd00:1122:3344:101::17]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:46412 req_id = dffbbe3b-badf-4fae-ac51-142ffc1b2e44 response_code = 200 uri = /crucible/0/regions
962026-05-19T00:11:43.105ZINFOcrucible-agent (dropshot): request completed latency_us = 77 local_addr = [fd00:1122:3344:101::17]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:46412 req_id = c8119641-b5f5-4dfe-b4a9-2569cd6ca296 response_code = 200 uri = /crucible/0/regions/d8b88646-4532-42c7-8c3f-61943c1824c3
972026-05-19T00:11:43.109ZINFOcrucible-agent (datafile): path is "/data/regions/d8b88646-4532-42c7-8c3f-61943c1824c3"
982026-05-19T00:11:43.118ZINFOcrucible-agent (datafile): dataset is oxp_a0d4bba2-a17b-41da-aa32-bb2b7de0796b/crucible/regions/d8b88646-4532-42c7-8c3f-61943c1824c3
992026-05-19T00:11:43.125ZINFOcrucible-agent (datafile): region d8b88646-4532-42c7-8c3f-61943c1824c3 state: Created -> Tombstoned
1002026-05-19T00:11:43.125ZINFOcrucible-agent (worker): applying SMF actions before removal...
1012026-05-19T00:11:43.125ZINFOcrucible-agent (dropshot): request completed latency_us = 16656 local_addr = [fd00:1122:3344:101::17]:32345 method = DELETE remote_addr = [fd00:1122:3344:101::a]:46412 req_id = 58d5adf4-c5cc-46fd-b7e4-d8f719b6b136 response_code = 204 uri = /crucible/0/regions/d8b88646-4532-42c7-8c3f-61943c1824c3
1022026-05-19T00:11:43.126ZINFOcrucible-agent (worker): disabling downstairs instance: downstairs-d8b88646-4532-42c7-8c3f-61943c1824c3 (instance states: (Some(Online), None))
1032026-05-19T00:11:43.129ZINFOcrucible-agent (dropshot): request completed latency_us = 48 local_addr = [fd00:1122:3344:101::17]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:46412 req_id = f7e1f509-1620-454d-9e96-12620405b8f0 response_code = 200 uri = /crucible/0/regions/d8b88646-4532-42c7-8c3f-61943c1824c3
1042026-05-19T00:11:43.131ZINFOcrucible-agent (worker): SMF ok!
1052026-05-19T00:11:43.140ZINFOcrucible-agent (worker): deleting zfs dataset "oxp_a0d4bba2-a17b-41da-aa32-bb2b7de0796b/crucible/regions/d8b88646-4532-42c7-8c3f-61943c1824c3" region = d8b88646-4532-42c7-8c3f-61943c1824c3
1062026-05-19T00:11:43.189ZINFOcrucible-agent (datafile): region d8b88646-4532-42c7-8c3f-61943c1824c3 state: Tombstoned -> Destroyed
1072026-05-19T00:11:43.268ZINFOcrucible-agent (dropshot): request completed latency_us = 74 local_addr = [fd00:1122:3344:101::17]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:46412 req_id = 8650fb38-f6be-42e2-9da2-fc3ac393219e response_code = 200 uri = /crucible/0/regions/d8b88646-4532-42c7-8c3f-61943c1824c3