Max level shown:
12026-01-02T18:50:06.166ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::16]:32345 remote_addr = [fd00:1122:3344:101::a]:63031
22026-01-02T18:50:06.166ZINFOcrucible-agent (datafile): region 8437bab4-cf3b-4902-894e-ef72c8e50c79 state: Requested
32026-01-02T18:50:06.167ZINFOcrucible-agent (dropshot): request completed latency_us = 1071 local_addr = [fd00:1122:3344:101::16]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:63031 req_id = 48cde168-2a0e-4550-9faf-dcd41fc83820 response_code = 200 uri = /crucible/0/regions
42026-01-02T18:50:06.167ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
52026-01-02T18:50:06.176ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_005a40e9-b3c8-48d8-b861-eb6c993b0888/crucible/regions/8437bab4-cf3b-4902-894e-ef72c8e50c79
62026-01-02T18:50:06.176ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_005a40e9-b3c8-48d8-b861-eb6c993b0888/crucible/regions/8437bab4-cf3b-4902-894e-ef72c8e50c79
72026-01-02T18:50:06.236ZINFOcrucible-agent (worker): creating region Region { id: RegionId("8437bab4-cf3b-4902-894e-ef72c8e50c79"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19001, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/8437bab4-cf3b-4902-894e-ef72c8e50c79" region = 8437bab4-cf3b-4902-894e-ef72c8e50c79
82026-01-02T18:50:06.236ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "8437bab4-cf3b-4902-894e-ef72c8e50c79" "--data" "/data/regions/8437bab4-cf3b-4902-894e-ef72c8e50c79" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 8437bab4-cf3b-4902-894e-ef72c8e50c79
92026-01-02T18:50:06.330ZINFOcrucible-agent (worker): region files created ok region = 8437bab4-cf3b-4902-894e-ef72c8e50c79
102026-01-02T18:50:06.330ZINFOcrucible-agent (datafile): region 8437bab4-cf3b-4902-894e-ef72c8e50c79 state: Requested -> Created
112026-01-02T18:50:06.330ZINFOcrucible-agent (worker): applying SMF actions post create...
122026-01-02T18:50:06.330ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-8437bab4-cf3b-4902-894e-ef72c8e50c79
132026-01-02T18:50:06.341ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-8437bab4-cf3b-4902-894e-ef72c8e50c79
142026-01-02T18:50:06.362ZINFOcrucible-agent (worker): creating config property group
152026-01-02T18:50:06.365ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-8437bab4-cf3b-4902-894e-ef72c8e50c79
162026-01-02T18:50:06.365ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/8437bab4-cf3b-4902-894e-ef72c8e50c79
172026-01-02T18:50:06.366ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
182026-01-02T18:50:06.366ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::16
192026-01-02T18:50:06.366ZINFOcrucible-agent (worker): commit
202026-01-02T18:50:06.369ZINFOcrucible-agent (worker): ok!
212026-01-02T18:50:06.386ZINFOcrucible-agent (worker): SMF ok!
222026-01-02T18:50:06.418ZINFOcrucible-agent (dropshot): request completed latency_us = 51 local_addr = [fd00:1122:3344:101::16]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:63031 req_id = 82fc3e9e-3cbe-40b6-8393-8ea3050bc542 response_code = 200 uri = /crucible/0/regions
232026-01-02T18:50:07.456ZINFOcrucible-agent (dropshot): request completed latency_us = 91 local_addr = [fd00:1122:3344:101::16]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:63031 req_id = 17c8860c-eb7e-4d1d-bf8b-0104ffff9512 response_code = 200 uri = /crucible/0/regions/c86d3697-1673-40a7-b241-449243d7c581
242026-01-02T18:50:07.468ZINFOcrucible-agent (datafile): path is "/data/regions/c86d3697-1673-40a7-b241-449243d7c581"
252026-01-02T18:50:07.477ZINFOcrucible-agent (datafile): dataset is oxp_005a40e9-b3c8-48d8-b861-eb6c993b0888/crucible/regions/c86d3697-1673-40a7-b241-449243d7c581
262026-01-02T18:50:07.487ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_005a40e9-b3c8-48d8-b861-eb6c993b0888/crucible/regions/c86d3697-1673-40a7-b241-449243d7c581@8c056911-d541-470e-96fa-50edd75a9b88
272026-01-02T18:50:07.487ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_005a40e9-b3c8-48d8-b861-eb6c993b0888/crucible/regions/c86d3697-1673-40a7-b241-449243d7c581", "8c056911-d541-470e-96fa-50edd75a9b88"]
282026-01-02T18:50:07.499ZINFOcrucible-agent (dropshot): request completed latency_us = 31349 local_addr = [fd00:1122:3344:101::16]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:63031 req_id = 8df4b5d5-b032-408b-8654-b98786889276 response_code = 200 uri = /crucible/0/regions/c86d3697-1673-40a7-b241-449243d7c581/snapshots/8c056911-d541-470e-96fa-50edd75a9b88
292026-01-02T18:50:07.511ZINFOcrucible-agent (datafile): path is "/data/regions/c86d3697-1673-40a7-b241-449243d7c581"
302026-01-02T18:50:07.520ZINFOcrucible-agent (datafile): dataset is oxp_005a40e9-b3c8-48d8-b861-eb6c993b0888/crucible/regions/c86d3697-1673-40a7-b241-449243d7c581
312026-01-02T18:50:07.528ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_005a40e9-b3c8-48d8-b861-eb6c993b0888/crucible/regions/c86d3697-1673-40a7-b241-449243d7c581@8c056911-d541-470e-96fa-50edd75a9b88
322026-01-02T18:50:07.528ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_005a40e9-b3c8-48d8-b861-eb6c993b0888/crucible/regions/c86d3697-1673-40a7-b241-449243d7c581", "8c056911-d541-470e-96fa-50edd75a9b88"]
332026-01-02T18:50:07.539ZINFOcrucible-agent (datafile): requesting running snapshot c86d3697-1673-40a7-b241-449243d7c581-8c056911-d541-470e-96fa-50edd75a9b88 state: Requested
342026-01-02T18:50:07.539ZINFOcrucible-agent (dropshot): request completed latency_us = 28586 local_addr = [fd00:1122:3344:101::16]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:63031 req_id = 52e9ad64-2f37-419a-926a-358f457bf729 response_code = 200 uri = /crucible/0/regions/c86d3697-1673-40a7-b241-449243d7c581/snapshots/8c056911-d541-470e-96fa-50edd75a9b88/run
352026-01-02T18:50:07.539ZINFOcrucible-agent (worker): applying SMF actions for region c86d3697-1673-40a7-b241-449243d7c581 running snapshot 8c056911-d541-470e-96fa-50edd75a9b88 (state Requested)...
362026-01-02T18:50:07.545ZINFOcrucible-agent (worker): creating missing snapshot instance snapshot-c86d3697-1673-40a7-b241-449243d7c581-8c056911-d541-470e-96fa-50edd75a9b88
372026-01-02T18:50:07.555ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:snapshot-c86d3697-1673-40a7-b241-449243d7c581-8c056911-d541-470e-96fa-50edd75a9b88
382026-01-02T18:50:07.568ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:snapshot-c86d3697-1673-40a7-b241-449243d7c581-8c056911-d541-470e-96fa-50edd75a9b88
392026-01-02T18:50:07.568ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/c86d3697-1673-40a7-b241-449243d7c581/.zfs/snapshot/8c056911-d541-470e-96fa-50edd75a9b88
402026-01-02T18:50:07.568ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19002
412026-01-02T18:50:07.568ZINFOcrucible-agent (worker): ensure mode SCF_TYPE_ASTRING ro
422026-01-02T18:50:07.568ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::16
432026-01-02T18:50:07.568ZINFOcrucible-agent (worker): commit
442026-01-02T18:50:07.573ZINFOcrucible-agent (worker): ok!
452026-01-02T18:50:07.579ZINFOcrucible-agent (worker): SMF ok!
462026-01-02T18:50:07.579ZINFOcrucible-agent (datafile): region c86d3697-1673-40a7-b241-449243d7c581 running snapshot 8c056911-d541-470e-96fa-50edd75a9b88 state: Requested -> Created
472026-01-02T18:50:07.920ZINFOcrucible-agent (datafile): path is "/data/regions/c86d3697-1673-40a7-b241-449243d7c581"
482026-01-02T18:50:07.926ZINFOcrucible-agent (datafile): dataset is oxp_005a40e9-b3c8-48d8-b861-eb6c993b0888/crucible/regions/c86d3697-1673-40a7-b241-449243d7c581
492026-01-02T18:50:07.931ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_005a40e9-b3c8-48d8-b861-eb6c993b0888/crucible/regions/c86d3697-1673-40a7-b241-449243d7c581@8c056911-d541-470e-96fa-50edd75a9b88
502026-01-02T18:50:07.931ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_005a40e9-b3c8-48d8-b861-eb6c993b0888/crucible/regions/c86d3697-1673-40a7-b241-449243d7c581", "8c056911-d541-470e-96fa-50edd75a9b88"]
512026-01-02T18:50:07.936ZINFOcrucible-agent (dropshot): request completed latency_us = 15871 local_addr = [fd00:1122:3344:101::16]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:63031 req_id = 94d7fc0e-855e-4129-8415-8f17641a43ba response_code = 200 uri = /crucible/0/regions/c86d3697-1673-40a7-b241-449243d7c581/snapshots/8c056911-d541-470e-96fa-50edd75a9b88/run