Max level shown:
12026-05-13T21:07:10.576ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::1a]:32345 remote_addr = [fd00:1122:3344:101::b]:45434
22026-05-13T21:07:10.576ZINFOcrucible-agent (datafile): region 8524b423-5ded-4664-a6bf-0ca51b037ffb state: Requested
32026-05-13T21:07:10.577ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
42026-05-13T21:07:10.577ZINFOcrucible-agent (dropshot): request completed latency_us = 857 local_addr = [fd00:1122:3344:101::1a]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:45434 req_id = 74ef8486-6210-47b5-b71e-b010e8e4f78e response_code = 200 uri = /crucible/0/regions
52026-05-13T21:07:10.587ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_5027f210-ceed-41c7-b783-966d70e827f3/crucible/regions/8524b423-5ded-4664-a6bf-0ca51b037ffb
62026-05-13T21:07:10.587ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_5027f210-ceed-41c7-b783-966d70e827f3/crucible/regions/8524b423-5ded-4664-a6bf-0ca51b037ffb
72026-05-13T21:07:10.636ZINFOcrucible-agent (worker): creating region Region { id: RegionId("8524b423-5ded-4664-a6bf-0ca51b037ffb"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19001, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/8524b423-5ded-4664-a6bf-0ca51b037ffb" region = 8524b423-5ded-4664-a6bf-0ca51b037ffb
82026-05-13T21:07:10.637ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "8524b423-5ded-4664-a6bf-0ca51b037ffb" "--data" "/data/regions/8524b423-5ded-4664-a6bf-0ca51b037ffb" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 8524b423-5ded-4664-a6bf-0ca51b037ffb
92026-05-13T21:07:10.696ZINFOcrucible-agent (worker): region files created ok region = 8524b423-5ded-4664-a6bf-0ca51b037ffb
102026-05-13T21:07:10.696ZINFOcrucible-agent (datafile): region 8524b423-5ded-4664-a6bf-0ca51b037ffb state: Requested -> Created
112026-05-13T21:07:10.696ZINFOcrucible-agent (worker): applying SMF actions post create...
122026-05-13T21:07:10.696ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-8524b423-5ded-4664-a6bf-0ca51b037ffb
132026-05-13T21:07:10.703ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-8524b423-5ded-4664-a6bf-0ca51b037ffb
142026-05-13T21:07:10.721ZINFOcrucible-agent (worker): creating config property group
152026-05-13T21:07:10.722ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-8524b423-5ded-4664-a6bf-0ca51b037ffb
162026-05-13T21:07:10.722ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/8524b423-5ded-4664-a6bf-0ca51b037ffb
172026-05-13T21:07:10.722ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
182026-05-13T21:07:10.723ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::1a
192026-05-13T21:07:10.723ZINFOcrucible-agent (worker): commit
202026-05-13T21:07:10.725ZINFOcrucible-agent (worker): ok!
212026-05-13T21:07:10.737ZINFOcrucible-agent (worker): SMF ok!
222026-05-13T21:07:10.822ZINFOcrucible-agent (dropshot): request completed latency_us = 61 local_addr = [fd00:1122:3344:101::1a]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:45434 req_id = 707e6982-94a0-41c5-a6b8-12330d632d9b response_code = 200 uri = /crucible/0/regions
232026-05-13T21:07:11.653ZINFOcrucible-agent (dropshot): request completed latency_us = 89 local_addr = [fd00:1122:3344:101::1a]:32345 method = GET remote_addr = [fd00:1122:3344:101::b]:45434 req_id = c2794bba-c8af-46f8-a91f-429da66b4e63 response_code = 200 uri = /crucible/0/regions/d404f4de-40ad-4af9-a00e-3c7a15eb3dc5
242026-05-13T21:07:11.668ZINFOcrucible-agent (datafile): path is "/data/regions/d404f4de-40ad-4af9-a00e-3c7a15eb3dc5"
252026-05-13T21:07:11.676ZINFOcrucible-agent (datafile): dataset is oxp_5027f210-ceed-41c7-b783-966d70e827f3/crucible/regions/d404f4de-40ad-4af9-a00e-3c7a15eb3dc5
262026-05-13T21:07:11.685ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_5027f210-ceed-41c7-b783-966d70e827f3/crucible/regions/d404f4de-40ad-4af9-a00e-3c7a15eb3dc5@2a6b22e7-5d26-4932-af30-d008c708ec8b
272026-05-13T21:07:11.685ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_5027f210-ceed-41c7-b783-966d70e827f3/crucible/regions/d404f4de-40ad-4af9-a00e-3c7a15eb3dc5", "2a6b22e7-5d26-4932-af30-d008c708ec8b"]
282026-05-13T21:07:11.698ZINFOcrucible-agent (dropshot): request completed latency_us = 29870 local_addr = [fd00:1122:3344:101::1a]:32345 method = GET remote_addr = [fd00:1122:3344:101::b]:45434 req_id = 208bee9f-f508-43f4-b424-9647e0b2a427 response_code = 200 uri = /crucible/0/regions/d404f4de-40ad-4af9-a00e-3c7a15eb3dc5/snapshots/2a6b22e7-5d26-4932-af30-d008c708ec8b
292026-05-13T21:07:11.713ZINFOcrucible-agent (datafile): path is "/data/regions/d404f4de-40ad-4af9-a00e-3c7a15eb3dc5"
302026-05-13T21:07:11.723ZINFOcrucible-agent (datafile): dataset is oxp_5027f210-ceed-41c7-b783-966d70e827f3/crucible/regions/d404f4de-40ad-4af9-a00e-3c7a15eb3dc5
312026-05-13T21:07:11.732ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_5027f210-ceed-41c7-b783-966d70e827f3/crucible/regions/d404f4de-40ad-4af9-a00e-3c7a15eb3dc5@2a6b22e7-5d26-4932-af30-d008c708ec8b
322026-05-13T21:07:11.732ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_5027f210-ceed-41c7-b783-966d70e827f3/crucible/regions/d404f4de-40ad-4af9-a00e-3c7a15eb3dc5", "2a6b22e7-5d26-4932-af30-d008c708ec8b"]
332026-05-13T21:07:11.743ZINFOcrucible-agent (datafile): requesting running snapshot d404f4de-40ad-4af9-a00e-3c7a15eb3dc5-2a6b22e7-5d26-4932-af30-d008c708ec8b state: Requested
342026-05-13T21:07:11.743ZINFOcrucible-agent (worker): applying SMF actions for region d404f4de-40ad-4af9-a00e-3c7a15eb3dc5 running snapshot 2a6b22e7-5d26-4932-af30-d008c708ec8b (state Requested)...
352026-05-13T21:07:11.743ZINFOcrucible-agent (dropshot): request completed latency_us = 29871 local_addr = [fd00:1122:3344:101::1a]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:45434 req_id = 549feb49-ed01-4529-80f6-ee308ef06bba response_code = 200 uri = /crucible/0/regions/d404f4de-40ad-4af9-a00e-3c7a15eb3dc5/snapshots/2a6b22e7-5d26-4932-af30-d008c708ec8b/run
362026-05-13T21:07:11.750ZINFOcrucible-agent (worker): creating missing snapshot instance snapshot-d404f4de-40ad-4af9-a00e-3c7a15eb3dc5-2a6b22e7-5d26-4932-af30-d008c708ec8b
372026-05-13T21:07:11.755ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:snapshot-d404f4de-40ad-4af9-a00e-3c7a15eb3dc5-2a6b22e7-5d26-4932-af30-d008c708ec8b
382026-05-13T21:07:11.763ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:snapshot-d404f4de-40ad-4af9-a00e-3c7a15eb3dc5-2a6b22e7-5d26-4932-af30-d008c708ec8b
392026-05-13T21:07:11.763ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/d404f4de-40ad-4af9-a00e-3c7a15eb3dc5/.zfs/snapshot/2a6b22e7-5d26-4932-af30-d008c708ec8b
402026-05-13T21:07:11.764ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19002
412026-05-13T21:07:11.764ZINFOcrucible-agent (worker): ensure mode SCF_TYPE_ASTRING ro
422026-05-13T21:07:11.764ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::1a
432026-05-13T21:07:11.764ZINFOcrucible-agent (worker): commit
442026-05-13T21:07:11.766ZINFOcrucible-agent (worker): ok!
452026-05-13T21:07:11.771ZINFOcrucible-agent (worker): SMF ok!
462026-05-13T21:07:11.771ZINFOcrucible-agent (datafile): region d404f4de-40ad-4af9-a00e-3c7a15eb3dc5 running snapshot 2a6b22e7-5d26-4932-af30-d008c708ec8b state: Requested -> Created
472026-05-13T21:07:12.062ZINFOcrucible-agent (datafile): path is "/data/regions/d404f4de-40ad-4af9-a00e-3c7a15eb3dc5"
482026-05-13T21:07:12.069ZINFOcrucible-agent (datafile): dataset is oxp_5027f210-ceed-41c7-b783-966d70e827f3/crucible/regions/d404f4de-40ad-4af9-a00e-3c7a15eb3dc5
492026-05-13T21:07:12.076ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_5027f210-ceed-41c7-b783-966d70e827f3/crucible/regions/d404f4de-40ad-4af9-a00e-3c7a15eb3dc5@2a6b22e7-5d26-4932-af30-d008c708ec8b
502026-05-13T21:07:12.076ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_5027f210-ceed-41c7-b783-966d70e827f3/crucible/regions/d404f4de-40ad-4af9-a00e-3c7a15eb3dc5", "2a6b22e7-5d26-4932-af30-d008c708ec8b"]
512026-05-13T21:07:12.082ZINFOcrucible-agent (dropshot): request completed latency_us = 19862 local_addr = [fd00:1122:3344:101::1a]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:45434 req_id = 442b8a35-512b-43bc-86fd-0b30e3b1c6e0 response_code = 200 uri = /crucible/0/regions/d404f4de-40ad-4af9-a00e-3c7a15eb3dc5/snapshots/2a6b22e7-5d26-4932-af30-d008c708ec8b/run