Max level shown:
12026-08-28T00:31:49.846ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::16]:32345 remote_addr = [fd00:1122:3344:101::a]:60321
22026-08-28T00:31:49.847ZINFOcrucible-agent (datafile): region 21718215-c72a-417f-9358-70ff623f09e6 state: Requested
32026-08-28T00:31:49.848ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
42026-08-28T00:31:49.848ZINFOcrucible-agent (dropshot): request completed latency_us = 694 local_addr = [fd00:1122:3344:101::16]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:60321 req_id = b2525ce4-7682-4f16-887a-8a2a416fcc0c response_code = 200 uri = /crucible/0/regions
52026-08-28T00:31:49.855ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_0725dbfd-bf89-4b83-ba2e-e2cdc6ec41e2/crucible/regions/21718215-c72a-417f-9358-70ff623f09e6
62026-08-28T00:31:49.855ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_0725dbfd-bf89-4b83-ba2e-e2cdc6ec41e2/crucible/regions/21718215-c72a-417f-9358-70ff623f09e6
72026-08-28T00:31:49.982ZINFOcrucible-agent (worker): creating region Region { id: RegionId("21718215-c72a-417f-9358-70ff623f09e6"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19001, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/21718215-c72a-417f-9358-70ff623f09e6" region = 21718215-c72a-417f-9358-70ff623f09e6
82026-08-28T00:31:49.982ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "21718215-c72a-417f-9358-70ff623f09e6" "--data" "/data/regions/21718215-c72a-417f-9358-70ff623f09e6" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 21718215-c72a-417f-9358-70ff623f09e6
92026-08-28T00:31:50.060ZINFOcrucible-agent (worker): region files created ok region = 21718215-c72a-417f-9358-70ff623f09e6
102026-08-28T00:31:50.060ZINFOcrucible-agent (datafile): region 21718215-c72a-417f-9358-70ff623f09e6 state: Requested -> Created
112026-08-28T00:31:50.060ZINFOcrucible-agent (worker): applying SMF actions post create...
122026-08-28T00:31:50.061ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-21718215-c72a-417f-9358-70ff623f09e6
132026-08-28T00:31:50.078ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-21718215-c72a-417f-9358-70ff623f09e6
142026-08-28T00:31:50.089ZINFOcrucible-agent (worker): creating config property group
152026-08-28T00:31:50.092ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-21718215-c72a-417f-9358-70ff623f09e6
162026-08-28T00:31:50.092ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/21718215-c72a-417f-9358-70ff623f09e6
172026-08-28T00:31:50.092ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
182026-08-28T00:31:50.092ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::16
192026-08-28T00:31:50.092ZINFOcrucible-agent (worker): commit
202026-08-28T00:31:50.096ZINFOcrucible-agent (worker): ok!
212026-08-28T00:31:50.103ZINFOcrucible-agent (worker): SMF ok!
222026-08-28T00:31:50.128ZINFOcrucible-agent (dropshot): request completed latency_us = 61 local_addr = [fd00:1122:3344:101::16]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:60321 req_id = 36925ad7-6e48-4566-8d9f-0cf24215fe5b response_code = 200 uri = /crucible/0/regions
232026-08-28T00:31:51.521ZINFOcrucible-agent (dropshot): request completed latency_us = 59 local_addr = [fd00:1122:3344:101::16]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:60321 req_id = 9548e1a0-7d0c-4e17-b511-45a4b04ca5e1 response_code = 200 uri = /crucible/0/regions/ca35f608-dc59-47b1-8f1a-74e1ad9d5402
242026-08-28T00:31:51.525ZINFOcrucible-agent (datafile): path is "/data/regions/ca35f608-dc59-47b1-8f1a-74e1ad9d5402"
252026-08-28T00:31:51.531ZINFOcrucible-agent (datafile): dataset is oxp_0725dbfd-bf89-4b83-ba2e-e2cdc6ec41e2/crucible/regions/ca35f608-dc59-47b1-8f1a-74e1ad9d5402
262026-08-28T00:31:51.535ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_0725dbfd-bf89-4b83-ba2e-e2cdc6ec41e2/crucible/regions/ca35f608-dc59-47b1-8f1a-74e1ad9d5402@f818b14e-9613-47d0-ad78-d8df6f923a22
272026-08-28T00:31:51.535ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_0725dbfd-bf89-4b83-ba2e-e2cdc6ec41e2/crucible/regions/ca35f608-dc59-47b1-8f1a-74e1ad9d5402", "f818b14e-9613-47d0-ad78-d8df6f923a22"]
282026-08-28T00:31:51.544ZINFOcrucible-agent (dropshot): request completed latency_us = 18740 local_addr = [fd00:1122:3344:101::16]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:60321 req_id = 8a773050-2d2a-4e3b-9887-22e24607aeb3 response_code = 200 uri = /crucible/0/regions/ca35f608-dc59-47b1-8f1a-74e1ad9d5402/snapshots/f818b14e-9613-47d0-ad78-d8df6f923a22
292026-08-28T00:31:51.547ZINFOcrucible-agent (datafile): path is "/data/regions/ca35f608-dc59-47b1-8f1a-74e1ad9d5402"
302026-08-28T00:31:51.552ZINFOcrucible-agent (datafile): dataset is oxp_0725dbfd-bf89-4b83-ba2e-e2cdc6ec41e2/crucible/regions/ca35f608-dc59-47b1-8f1a-74e1ad9d5402
312026-08-28T00:31:51.557ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_0725dbfd-bf89-4b83-ba2e-e2cdc6ec41e2/crucible/regions/ca35f608-dc59-47b1-8f1a-74e1ad9d5402@f818b14e-9613-47d0-ad78-d8df6f923a22
322026-08-28T00:31:51.557ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_0725dbfd-bf89-4b83-ba2e-e2cdc6ec41e2/crucible/regions/ca35f608-dc59-47b1-8f1a-74e1ad9d5402", "f818b14e-9613-47d0-ad78-d8df6f923a22"]
332026-08-28T00:31:51.563ZINFOcrucible-agent (datafile): requesting running snapshot ca35f608-dc59-47b1-8f1a-74e1ad9d5402-f818b14e-9613-47d0-ad78-d8df6f923a22 state: Requested
342026-08-28T00:31:51.568ZINFOcrucible-agent (worker): applying SMF actions for region ca35f608-dc59-47b1-8f1a-74e1ad9d5402 running snapshot f818b14e-9613-47d0-ad78-d8df6f923a22 (state Requested)...
352026-08-28T00:31:51.568ZINFOcrucible-agent (dropshot): request completed latency_us = 21207 local_addr = [fd00:1122:3344:101::16]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:60321 req_id = 479311fd-61ec-43f9-ac93-cba93ea795b6 response_code = 200 uri = /crucible/0/regions/ca35f608-dc59-47b1-8f1a-74e1ad9d5402/snapshots/f818b14e-9613-47d0-ad78-d8df6f923a22/run
362026-08-28T00:31:51.571ZINFOcrucible-agent (worker): creating missing snapshot instance snapshot-ca35f608-dc59-47b1-8f1a-74e1ad9d5402-f818b14e-9613-47d0-ad78-d8df6f923a22
372026-08-28T00:31:51.573ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:snapshot-ca35f608-dc59-47b1-8f1a-74e1ad9d5402-f818b14e-9613-47d0-ad78-d8df6f923a22
382026-08-28T00:31:51.582ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:snapshot-ca35f608-dc59-47b1-8f1a-74e1ad9d5402-f818b14e-9613-47d0-ad78-d8df6f923a22
392026-08-28T00:31:51.582ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/ca35f608-dc59-47b1-8f1a-74e1ad9d5402/.zfs/snapshot/f818b14e-9613-47d0-ad78-d8df6f923a22
402026-08-28T00:31:51.582ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19002
412026-08-28T00:31:51.583ZINFOcrucible-agent (worker): ensure mode SCF_TYPE_ASTRING ro
422026-08-28T00:31:51.583ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::16
432026-08-28T00:31:51.583ZINFOcrucible-agent (worker): commit
442026-08-28T00:31:51.586ZINFOcrucible-agent (worker): ok!
452026-08-28T00:31:51.590ZINFOcrucible-agent (worker): SMF ok!
462026-08-28T00:31:51.590ZINFOcrucible-agent (datafile): region ca35f608-dc59-47b1-8f1a-74e1ad9d5402 running snapshot f818b14e-9613-47d0-ad78-d8df6f923a22 state: Requested -> Created
472026-08-28T00:31:51.734ZINFOcrucible-agent (datafile): path is "/data/regions/ca35f608-dc59-47b1-8f1a-74e1ad9d5402"
482026-08-28T00:31:51.742ZINFOcrucible-agent (datafile): dataset is oxp_0725dbfd-bf89-4b83-ba2e-e2cdc6ec41e2/crucible/regions/ca35f608-dc59-47b1-8f1a-74e1ad9d5402
492026-08-28T00:31:51.749ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_0725dbfd-bf89-4b83-ba2e-e2cdc6ec41e2/crucible/regions/ca35f608-dc59-47b1-8f1a-74e1ad9d5402@f818b14e-9613-47d0-ad78-d8df6f923a22
502026-08-28T00:31:51.749ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_0725dbfd-bf89-4b83-ba2e-e2cdc6ec41e2/crucible/regions/ca35f608-dc59-47b1-8f1a-74e1ad9d5402", "f818b14e-9613-47d0-ad78-d8df6f923a22"]
512026-08-28T00:31:51.755ZINFOcrucible-agent (dropshot): request completed latency_us = 20448 local_addr = [fd00:1122:3344:101::16]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:60321 req_id = b366ef2c-3fcf-4783-8111-522166374fd3 response_code = 200 uri = /crucible/0/regions/ca35f608-dc59-47b1-8f1a-74e1ad9d5402/snapshots/f818b14e-9613-47d0-ad78-d8df6f923a22/run