Max level shown:
12026-05-07T01:09:29.180ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::14]:32345 remote_addr = [fd00:1122:3344:101::a]:60310
22026-05-07T01:09:29.180ZINFOcrucible-agent (datafile): region aa445f22-49f5-4cb1-8247-c46ae358d139 state: Requested
32026-05-07T01:09:29.181ZINFOcrucible-agent (dropshot): request completed latency_us = 907 local_addr = [fd00:1122:3344:101::14]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:60310 req_id = 79eff3aa-e8ef-4fbc-a47d-315d68e1aece response_code = 200 uri = /crucible/0/regions
42026-05-07T01:09:29.181ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
52026-05-07T01:09:29.191ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_34341938-9ad5-498f-b135-ee7e060624e8/crucible/regions/aa445f22-49f5-4cb1-8247-c46ae358d139
62026-05-07T01:09:29.191ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_34341938-9ad5-498f-b135-ee7e060624e8/crucible/regions/aa445f22-49f5-4cb1-8247-c46ae358d139
72026-05-07T01:09:29.248ZINFOcrucible-agent (worker): creating region Region { id: RegionId("aa445f22-49f5-4cb1-8247-c46ae358d139"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19001, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/aa445f22-49f5-4cb1-8247-c46ae358d139" region = aa445f22-49f5-4cb1-8247-c46ae358d139
82026-05-07T01:09:29.248ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "aa445f22-49f5-4cb1-8247-c46ae358d139" "--data" "/data/regions/aa445f22-49f5-4cb1-8247-c46ae358d139" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = aa445f22-49f5-4cb1-8247-c46ae358d139
92026-05-07T01:09:29.323ZINFOcrucible-agent (worker): region files created ok region = aa445f22-49f5-4cb1-8247-c46ae358d139
102026-05-07T01:09:29.323ZINFOcrucible-agent (datafile): region aa445f22-49f5-4cb1-8247-c46ae358d139 state: Requested -> Created
112026-05-07T01:09:29.323ZINFOcrucible-agent (worker): applying SMF actions post create...
122026-05-07T01:09:29.343ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-aa445f22-49f5-4cb1-8247-c46ae358d139
132026-05-07T01:09:29.351ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-aa445f22-49f5-4cb1-8247-c46ae358d139
142026-05-07T01:09:29.360ZINFOcrucible-agent (worker): creating config property group
152026-05-07T01:09:29.364ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-aa445f22-49f5-4cb1-8247-c46ae358d139
162026-05-07T01:09:29.364ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/aa445f22-49f5-4cb1-8247-c46ae358d139
172026-05-07T01:09:29.364ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
182026-05-07T01:09:29.364ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::14
192026-05-07T01:09:29.364ZINFOcrucible-agent (worker): commit
202026-05-07T01:09:29.367ZINFOcrucible-agent (worker): ok!
212026-05-07T01:09:29.372ZINFOcrucible-agent (worker): SMF ok!
222026-05-07T01:09:29.523ZINFOcrucible-agent (dropshot): request completed latency_us = 58 local_addr = [fd00:1122:3344:101::14]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:60310 req_id = 583b020e-5bd9-48f1-aa85-ffc0737f5cb2 response_code = 200 uri = /crucible/0/regions
232026-05-07T01:09:30.696ZINFOcrucible-agent (dropshot): request completed latency_us = 93 local_addr = [fd00:1122:3344:101::14]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:60310 req_id = d49afd31-3858-42d0-ae02-c753a3c24ccc response_code = 200 uri = /crucible/0/regions/1db20b9b-5d48-4d10-a263-9f0d44305e79
242026-05-07T01:09:30.703ZINFOcrucible-agent (datafile): path is "/data/regions/1db20b9b-5d48-4d10-a263-9f0d44305e79"
252026-05-07T01:09:30.714ZINFOcrucible-agent (datafile): dataset is oxp_34341938-9ad5-498f-b135-ee7e060624e8/crucible/regions/1db20b9b-5d48-4d10-a263-9f0d44305e79
262026-05-07T01:09:30.725ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_34341938-9ad5-498f-b135-ee7e060624e8/crucible/regions/1db20b9b-5d48-4d10-a263-9f0d44305e79@05dc2b1a-27f0-400b-ac8b-abe73f6a3b29
272026-05-07T01:09:30.725ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_34341938-9ad5-498f-b135-ee7e060624e8/crucible/regions/1db20b9b-5d48-4d10-a263-9f0d44305e79", "05dc2b1a-27f0-400b-ac8b-abe73f6a3b29"]
282026-05-07T01:09:30.739ZINFOcrucible-agent (dropshot): request completed latency_us = 36417 local_addr = [fd00:1122:3344:101::14]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:60310 req_id = fd3433e7-f1b3-4271-a8f5-cfc3730bd2d5 response_code = 200 uri = /crucible/0/regions/1db20b9b-5d48-4d10-a263-9f0d44305e79/snapshots/05dc2b1a-27f0-400b-ac8b-abe73f6a3b29
292026-05-07T01:09:30.752ZINFOcrucible-agent (datafile): path is "/data/regions/1db20b9b-5d48-4d10-a263-9f0d44305e79"
302026-05-07T01:09:30.761ZINFOcrucible-agent (datafile): dataset is oxp_34341938-9ad5-498f-b135-ee7e060624e8/crucible/regions/1db20b9b-5d48-4d10-a263-9f0d44305e79
312026-05-07T01:09:30.771ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_34341938-9ad5-498f-b135-ee7e060624e8/crucible/regions/1db20b9b-5d48-4d10-a263-9f0d44305e79@05dc2b1a-27f0-400b-ac8b-abe73f6a3b29
322026-05-07T01:09:30.771ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_34341938-9ad5-498f-b135-ee7e060624e8/crucible/regions/1db20b9b-5d48-4d10-a263-9f0d44305e79", "05dc2b1a-27f0-400b-ac8b-abe73f6a3b29"]
332026-05-07T01:09:30.782ZINFOcrucible-agent (datafile): requesting running snapshot 1db20b9b-5d48-4d10-a263-9f0d44305e79-05dc2b1a-27f0-400b-ac8b-abe73f6a3b29 state: Requested
342026-05-07T01:09:30.782ZINFOcrucible-agent (worker): applying SMF actions for region 1db20b9b-5d48-4d10-a263-9f0d44305e79 running snapshot 05dc2b1a-27f0-400b-ac8b-abe73f6a3b29 (state Requested)...
352026-05-07T01:09:30.782ZINFOcrucible-agent (dropshot): request completed latency_us = 30335 local_addr = [fd00:1122:3344:101::14]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:60310 req_id = 69924c2b-c129-4372-adc3-bce7115e4c08 response_code = 200 uri = /crucible/0/regions/1db20b9b-5d48-4d10-a263-9f0d44305e79/snapshots/05dc2b1a-27f0-400b-ac8b-abe73f6a3b29/run
362026-05-07T01:09:30.787ZINFOcrucible-agent (worker): creating missing snapshot instance snapshot-1db20b9b-5d48-4d10-a263-9f0d44305e79-05dc2b1a-27f0-400b-ac8b-abe73f6a3b29
372026-05-07T01:09:30.793ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:snapshot-1db20b9b-5d48-4d10-a263-9f0d44305e79-05dc2b1a-27f0-400b-ac8b-abe73f6a3b29
382026-05-07T01:09:30.805ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:snapshot-1db20b9b-5d48-4d10-a263-9f0d44305e79-05dc2b1a-27f0-400b-ac8b-abe73f6a3b29
392026-05-07T01:09:30.805ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/1db20b9b-5d48-4d10-a263-9f0d44305e79/.zfs/snapshot/05dc2b1a-27f0-400b-ac8b-abe73f6a3b29
402026-05-07T01:09:30.805ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19002
412026-05-07T01:09:30.805ZINFOcrucible-agent (worker): ensure mode SCF_TYPE_ASTRING ro
422026-05-07T01:09:30.805ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::14
432026-05-07T01:09:30.806ZINFOcrucible-agent (worker): commit
442026-05-07T01:09:30.809ZINFOcrucible-agent (worker): ok!
452026-05-07T01:09:30.814ZINFOcrucible-agent (worker): SMF ok!
462026-05-07T01:09:30.814ZINFOcrucible-agent (datafile): region 1db20b9b-5d48-4d10-a263-9f0d44305e79 running snapshot 05dc2b1a-27f0-400b-ac8b-abe73f6a3b29 state: Requested -> Created
472026-05-07T01:09:31.077ZINFOcrucible-agent (datafile): path is "/data/regions/1db20b9b-5d48-4d10-a263-9f0d44305e79"
482026-05-07T01:09:31.087ZINFOcrucible-agent (datafile): dataset is oxp_34341938-9ad5-498f-b135-ee7e060624e8/crucible/regions/1db20b9b-5d48-4d10-a263-9f0d44305e79
492026-05-07T01:09:31.099ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_34341938-9ad5-498f-b135-ee7e060624e8/crucible/regions/1db20b9b-5d48-4d10-a263-9f0d44305e79@05dc2b1a-27f0-400b-ac8b-abe73f6a3b29
502026-05-07T01:09:31.099ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_34341938-9ad5-498f-b135-ee7e060624e8/crucible/regions/1db20b9b-5d48-4d10-a263-9f0d44305e79", "05dc2b1a-27f0-400b-ac8b-abe73f6a3b29"]
512026-05-07T01:09:31.109ZINFOcrucible-agent (dropshot): request completed latency_us = 31859 local_addr = [fd00:1122:3344:101::14]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:60310 req_id = cbe6fa95-d25b-4262-b437-8d672ff62c7a response_code = 200 uri = /crucible/0/regions/1db20b9b-5d48-4d10-a263-9f0d44305e79/snapshots/05dc2b1a-27f0-400b-ac8b-abe73f6a3b29/run