Max level shown:
12026-03-18T02:36:52.778ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::18]:32345 remote_addr = [fd00:1122:3344:101::a]:50128
22026-03-18T02:36:52.778ZINFOcrucible-agent (datafile): region 4853d6e5-b86a-4e36-a69d-a3f84f142400 state: Requested
32026-03-18T02:36:52.779ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
42026-03-18T02:36:52.779ZINFOcrucible-agent (dropshot): request completed latency_us = 623 local_addr = [fd00:1122:3344:101::18]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:50128 req_id = 14621649-c87f-4314-975f-2661cd34c989 response_code = 200 uri = /crucible/0/regions
52026-03-18T02:36:52.788ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_db4dd703-156b-43de-bbbc-68cc9e8e4692/crucible/regions/4853d6e5-b86a-4e36-a69d-a3f84f142400
62026-03-18T02:36:52.788ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_db4dd703-156b-43de-bbbc-68cc9e8e4692/crucible/regions/4853d6e5-b86a-4e36-a69d-a3f84f142400
72026-03-18T02:36:52.842ZINFOcrucible-agent (worker): creating region Region { id: RegionId("4853d6e5-b86a-4e36-a69d-a3f84f142400"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19001, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/4853d6e5-b86a-4e36-a69d-a3f84f142400" region = 4853d6e5-b86a-4e36-a69d-a3f84f142400
82026-03-18T02:36:52.842ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "4853d6e5-b86a-4e36-a69d-a3f84f142400" "--data" "/data/regions/4853d6e5-b86a-4e36-a69d-a3f84f142400" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 4853d6e5-b86a-4e36-a69d-a3f84f142400
92026-03-18T02:36:52.922ZINFOcrucible-agent (worker): region files created ok region = 4853d6e5-b86a-4e36-a69d-a3f84f142400
102026-03-18T02:36:52.923ZINFOcrucible-agent (datafile): region 4853d6e5-b86a-4e36-a69d-a3f84f142400 state: Requested -> Created
112026-03-18T02:36:52.923ZINFOcrucible-agent (worker): applying SMF actions post create...
122026-03-18T02:36:52.940ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-4853d6e5-b86a-4e36-a69d-a3f84f142400
132026-03-18T02:36:52.946ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-4853d6e5-b86a-4e36-a69d-a3f84f142400
142026-03-18T02:36:52.958ZINFOcrucible-agent (worker): creating config property group
152026-03-18T02:36:52.962ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-4853d6e5-b86a-4e36-a69d-a3f84f142400
162026-03-18T02:36:52.962ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/4853d6e5-b86a-4e36-a69d-a3f84f142400
172026-03-18T02:36:52.962ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
182026-03-18T02:36:52.962ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::18
192026-03-18T02:36:52.962ZINFOcrucible-agent (worker): commit
202026-03-18T02:36:52.967ZINFOcrucible-agent (worker): ok!
212026-03-18T02:36:52.972ZINFOcrucible-agent (worker): SMF ok!
222026-03-18T02:36:53.041ZINFOcrucible-agent (dropshot): request completed latency_us = 55 local_addr = [fd00:1122:3344:101::18]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:50128 req_id = 08e1df4d-9e2c-4017-8c63-a9268ec007ad response_code = 200 uri = /crucible/0/regions
232026-03-18T02:36:54.373ZINFOcrucible-agent (dropshot): request completed latency_us = 70 local_addr = [fd00:1122:3344:101::18]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:50128 req_id = acf4dcfc-2887-42a2-b907-891fae21e0f8 response_code = 200 uri = /crucible/0/regions/28a1f8f3-8e91-42b7-9195-43678684a5e6
242026-03-18T02:36:54.376ZINFOcrucible-agent (datafile): path is "/data/regions/28a1f8f3-8e91-42b7-9195-43678684a5e6"
252026-03-18T02:36:54.383ZINFOcrucible-agent (datafile): dataset is oxp_db4dd703-156b-43de-bbbc-68cc9e8e4692/crucible/regions/28a1f8f3-8e91-42b7-9195-43678684a5e6
262026-03-18T02:36:54.389ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_db4dd703-156b-43de-bbbc-68cc9e8e4692/crucible/regions/28a1f8f3-8e91-42b7-9195-43678684a5e6@65e984d6-cd93-4417-9651-c0a4280e7d60
272026-03-18T02:36:54.389ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_db4dd703-156b-43de-bbbc-68cc9e8e4692/crucible/regions/28a1f8f3-8e91-42b7-9195-43678684a5e6", "65e984d6-cd93-4417-9651-c0a4280e7d60"]
282026-03-18T02:36:54.398ZINFOcrucible-agent (dropshot): request completed latency_us = 21545 local_addr = [fd00:1122:3344:101::18]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:50128 req_id = 36e336a4-186d-4e1a-a081-817585de72f5 response_code = 200 uri = /crucible/0/regions/28a1f8f3-8e91-42b7-9195-43678684a5e6/snapshots/65e984d6-cd93-4417-9651-c0a4280e7d60
292026-03-18T02:36:54.401ZINFOcrucible-agent (datafile): path is "/data/regions/28a1f8f3-8e91-42b7-9195-43678684a5e6"
302026-03-18T02:36:54.407ZINFOcrucible-agent (datafile): dataset is oxp_db4dd703-156b-43de-bbbc-68cc9e8e4692/crucible/regions/28a1f8f3-8e91-42b7-9195-43678684a5e6
312026-03-18T02:36:54.413ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_db4dd703-156b-43de-bbbc-68cc9e8e4692/crucible/regions/28a1f8f3-8e91-42b7-9195-43678684a5e6@65e984d6-cd93-4417-9651-c0a4280e7d60
322026-03-18T02:36:54.413ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_db4dd703-156b-43de-bbbc-68cc9e8e4692/crucible/regions/28a1f8f3-8e91-42b7-9195-43678684a5e6", "65e984d6-cd93-4417-9651-c0a4280e7d60"]
332026-03-18T02:36:54.422ZINFOcrucible-agent (datafile): requesting running snapshot 28a1f8f3-8e91-42b7-9195-43678684a5e6-65e984d6-cd93-4417-9651-c0a4280e7d60 state: Requested
342026-03-18T02:36:54.422ZINFOcrucible-agent (worker): applying SMF actions for region 28a1f8f3-8e91-42b7-9195-43678684a5e6 running snapshot 65e984d6-cd93-4417-9651-c0a4280e7d60 (state Requested)...
352026-03-18T02:36:54.422ZINFOcrucible-agent (dropshot): request completed latency_us = 20566 local_addr = [fd00:1122:3344:101::18]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:50128 req_id = 0c10d6c5-2b83-4e79-a400-a0fc2ae64149 response_code = 200 uri = /crucible/0/regions/28a1f8f3-8e91-42b7-9195-43678684a5e6/snapshots/65e984d6-cd93-4417-9651-c0a4280e7d60/run
362026-03-18T02:36:54.426ZINFOcrucible-agent (worker): creating missing snapshot instance snapshot-28a1f8f3-8e91-42b7-9195-43678684a5e6-65e984d6-cd93-4417-9651-c0a4280e7d60
372026-03-18T02:36:54.440ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:snapshot-28a1f8f3-8e91-42b7-9195-43678684a5e6-65e984d6-cd93-4417-9651-c0a4280e7d60
382026-03-18T02:36:54.450ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:snapshot-28a1f8f3-8e91-42b7-9195-43678684a5e6-65e984d6-cd93-4417-9651-c0a4280e7d60
392026-03-18T02:36:54.451ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/28a1f8f3-8e91-42b7-9195-43678684a5e6/.zfs/snapshot/65e984d6-cd93-4417-9651-c0a4280e7d60
402026-03-18T02:36:54.451ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19002
412026-03-18T02:36:54.451ZINFOcrucible-agent (worker): ensure mode SCF_TYPE_ASTRING ro
422026-03-18T02:36:54.451ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::18
432026-03-18T02:36:54.451ZINFOcrucible-agent (worker): commit
442026-03-18T02:36:54.454ZINFOcrucible-agent (worker): ok!
452026-03-18T02:36:54.459ZINFOcrucible-agent (worker): SMF ok!
462026-03-18T02:36:54.459ZINFOcrucible-agent (datafile): region 28a1f8f3-8e91-42b7-9195-43678684a5e6 running snapshot 65e984d6-cd93-4417-9651-c0a4280e7d60 state: Requested -> Created
472026-03-18T02:36:54.788ZINFOcrucible-agent (datafile): path is "/data/regions/28a1f8f3-8e91-42b7-9195-43678684a5e6"
482026-03-18T02:36:54.795ZINFOcrucible-agent (datafile): dataset is oxp_db4dd703-156b-43de-bbbc-68cc9e8e4692/crucible/regions/28a1f8f3-8e91-42b7-9195-43678684a5e6
492026-03-18T02:36:54.800ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_db4dd703-156b-43de-bbbc-68cc9e8e4692/crucible/regions/28a1f8f3-8e91-42b7-9195-43678684a5e6@65e984d6-cd93-4417-9651-c0a4280e7d60
502026-03-18T02:36:54.800ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_db4dd703-156b-43de-bbbc-68cc9e8e4692/crucible/regions/28a1f8f3-8e91-42b7-9195-43678684a5e6", "65e984d6-cd93-4417-9651-c0a4280e7d60"]
512026-03-18T02:36:54.805ZINFOcrucible-agent (dropshot): request completed latency_us = 17196 local_addr = [fd00:1122:3344:101::18]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:50128 req_id = 0d99eb19-f74c-4218-8fa0-432507418087 response_code = 200 uri = /crucible/0/regions/28a1f8f3-8e91-42b7-9195-43678684a5e6/snapshots/65e984d6-cd93-4417-9651-c0a4280e7d60/run