Max level shown:
12025-02-28T11:19:25.088ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::19]:32345 remote_addr = [fd00:1122:3344:101::b]:38461
22025-02-28T11:19:25.088ZINFOcrucible-agent (datafile): region 2277fcdb-32e6-4eb2-985c-59b62f533fcd state: Requested
32025-02-28T11:19:25.089ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
42025-02-28T11:19:25.089ZINFOcrucible-agent (dropshot): request completed latency_us = 589 local_addr = [fd00:1122:3344:101::19]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:38461 req_id = 1232bfea-38e6-4032-beb9-06c5427afb5a response_code = 200 uri = /crucible/0/regions
52025-02-28T11:19:25.099ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_b7be4b87-5b67-4526-86cc-07dffc1acb59/crucible/regions/2277fcdb-32e6-4eb2-985c-59b62f533fcd
62025-02-28T11:19:25.099ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_b7be4b87-5b67-4526-86cc-07dffc1acb59/crucible/regions/2277fcdb-32e6-4eb2-985c-59b62f533fcd
72025-02-28T11:19:25.156ZINFOcrucible-agent (worker): creating region Region { id: RegionId("2277fcdb-32e6-4eb2-985c-59b62f533fcd"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19001, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/2277fcdb-32e6-4eb2-985c-59b62f533fcd" region = 2277fcdb-32e6-4eb2-985c-59b62f533fcd
82025-02-28T11:19:25.156ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "2277fcdb-32e6-4eb2-985c-59b62f533fcd" "--data" "/data/regions/2277fcdb-32e6-4eb2-985c-59b62f533fcd" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 2277fcdb-32e6-4eb2-985c-59b62f533fcd
92025-02-28T11:19:25.236ZINFOcrucible-agent (worker): region files created ok region = 2277fcdb-32e6-4eb2-985c-59b62f533fcd
102025-02-28T11:19:25.236ZINFOcrucible-agent (datafile): region 2277fcdb-32e6-4eb2-985c-59b62f533fcd state: Requested -> Created
112025-02-28T11:19:25.236ZINFOcrucible-agent (worker): applying SMF actions post create...
122025-02-28T11:19:25.237ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-2277fcdb-32e6-4eb2-985c-59b62f533fcd
132025-02-28T11:19:25.242ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-2277fcdb-32e6-4eb2-985c-59b62f533fcd
142025-02-28T11:19:25.255ZINFOcrucible-agent (worker): creating config property group
152025-02-28T11:19:25.258ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-2277fcdb-32e6-4eb2-985c-59b62f533fcd
162025-02-28T11:19:25.259ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/2277fcdb-32e6-4eb2-985c-59b62f533fcd
172025-02-28T11:19:25.259ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
182025-02-28T11:19:25.259ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::19
192025-02-28T11:19:25.259ZINFOcrucible-agent (worker): commit
202025-02-28T11:19:25.263ZINFOcrucible-agent (worker): ok!
212025-02-28T11:19:25.268ZINFOcrucible-agent (worker): SMF ok!
222025-02-28T11:19:25.418ZINFOcrucible-agent (dropshot): request completed latency_us = 74 local_addr = [fd00:1122:3344:101::19]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:38461 req_id = a5ea7f32-2a84-4ab5-bd37-50b3e1d09139 response_code = 200 uri = /crucible/0/regions
232025-02-28T11:19:27.077ZINFOcrucible-agent (dropshot): request completed latency_us = 3028 local_addr = [fd00:1122:3344:101::19]:32345 method = GET remote_addr = [fd00:1122:3344:101::b]:38461 req_id = 2ec65f9d-592a-4adf-8d71-4dd0981795ff response_code = 200 uri = /crucible/0/regions/7bdc2f8f-7958-4c7f-b33e-c5825535a329
242025-02-28T11:19:27.083ZINFOcrucible-agent (datafile): path is "/data/regions/7bdc2f8f-7958-4c7f-b33e-c5825535a329"
252025-02-28T11:19:27.093ZINFOcrucible-agent (datafile): dataset is oxp_b7be4b87-5b67-4526-86cc-07dffc1acb59/crucible/regions/7bdc2f8f-7958-4c7f-b33e-c5825535a329
262025-02-28T11:19:27.101ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_b7be4b87-5b67-4526-86cc-07dffc1acb59/crucible/regions/7bdc2f8f-7958-4c7f-b33e-c5825535a329@421df0b3-15da-4055-97ce-50c89a55dd58
272025-02-28T11:19:27.101ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_b7be4b87-5b67-4526-86cc-07dffc1acb59/crucible/regions/7bdc2f8f-7958-4c7f-b33e-c5825535a329", "421df0b3-15da-4055-97ce-50c89a55dd58"]
282025-02-28T11:19:27.112ZINFOcrucible-agent (dropshot): request completed latency_us = 31732 local_addr = [fd00:1122:3344:101::19]:32345 method = GET remote_addr = [fd00:1122:3344:101::b]:38461 req_id = 51857b8e-4e4c-4f5f-8bb7-d76881529332 response_code = 200 uri = /crucible/0/regions/7bdc2f8f-7958-4c7f-b33e-c5825535a329/snapshots/421df0b3-15da-4055-97ce-50c89a55dd58
292025-02-28T11:19:27.116ZINFOcrucible-agent (datafile): path is "/data/regions/7bdc2f8f-7958-4c7f-b33e-c5825535a329"
302025-02-28T11:19:27.124ZINFOcrucible-agent (datafile): dataset is oxp_b7be4b87-5b67-4526-86cc-07dffc1acb59/crucible/regions/7bdc2f8f-7958-4c7f-b33e-c5825535a329
312025-02-28T11:19:27.132ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_b7be4b87-5b67-4526-86cc-07dffc1acb59/crucible/regions/7bdc2f8f-7958-4c7f-b33e-c5825535a329@421df0b3-15da-4055-97ce-50c89a55dd58
322025-02-28T11:19:27.132ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_b7be4b87-5b67-4526-86cc-07dffc1acb59/crucible/regions/7bdc2f8f-7958-4c7f-b33e-c5825535a329", "421df0b3-15da-4055-97ce-50c89a55dd58"]
332025-02-28T11:19:27.140ZINFOcrucible-agent (datafile): requesting running snapshot 7bdc2f8f-7958-4c7f-b33e-c5825535a329-421df0b3-15da-4055-97ce-50c89a55dd58 state: Requested
342025-02-28T11:19:27.140ZINFOcrucible-agent (worker): applying SMF actions for region 7bdc2f8f-7958-4c7f-b33e-c5825535a329 running snapshot 421df0b3-15da-4055-97ce-50c89a55dd58 (state Requested)...
352025-02-28T11:19:27.140ZINFOcrucible-agent (dropshot): request completed latency_us = 24284 local_addr = [fd00:1122:3344:101::19]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:38461 req_id = ddf12c32-685b-45a6-bdfd-61978cda9b0d response_code = 200 uri = /crucible/0/regions/7bdc2f8f-7958-4c7f-b33e-c5825535a329/snapshots/421df0b3-15da-4055-97ce-50c89a55dd58/run
362025-02-28T11:19:27.142ZINFOcrucible-agent (worker): creating missing snapshot instance snapshot-7bdc2f8f-7958-4c7f-b33e-c5825535a329-421df0b3-15da-4055-97ce-50c89a55dd58
372025-02-28T11:19:27.152ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:snapshot-7bdc2f8f-7958-4c7f-b33e-c5825535a329-421df0b3-15da-4055-97ce-50c89a55dd58
382025-02-28T11:19:27.162ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:snapshot-7bdc2f8f-7958-4c7f-b33e-c5825535a329-421df0b3-15da-4055-97ce-50c89a55dd58
392025-02-28T11:19:27.162ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/7bdc2f8f-7958-4c7f-b33e-c5825535a329/.zfs/snapshot/421df0b3-15da-4055-97ce-50c89a55dd58
402025-02-28T11:19:27.162ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19002
412025-02-28T11:19:27.162ZINFOcrucible-agent (worker): ensure mode SCF_TYPE_ASTRING ro
422025-02-28T11:19:27.162ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::19
432025-02-28T11:19:27.162ZINFOcrucible-agent (worker): commit
442025-02-28T11:19:27.166ZINFOcrucible-agent (worker): ok!
452025-02-28T11:19:27.170ZINFOcrucible-agent (worker): SMF ok!
462025-02-28T11:19:27.170ZINFOcrucible-agent (datafile): region 7bdc2f8f-7958-4c7f-b33e-c5825535a329 running snapshot 421df0b3-15da-4055-97ce-50c89a55dd58 state: Requested -> Created
472025-02-28T11:19:27.304ZINFOcrucible-agent (datafile): path is "/data/regions/7bdc2f8f-7958-4c7f-b33e-c5825535a329"
482025-02-28T11:19:27.313ZINFOcrucible-agent (datafile): dataset is oxp_b7be4b87-5b67-4526-86cc-07dffc1acb59/crucible/regions/7bdc2f8f-7958-4c7f-b33e-c5825535a329
492025-02-28T11:19:27.321ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_b7be4b87-5b67-4526-86cc-07dffc1acb59/crucible/regions/7bdc2f8f-7958-4c7f-b33e-c5825535a329@421df0b3-15da-4055-97ce-50c89a55dd58
502025-02-28T11:19:27.321ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_b7be4b87-5b67-4526-86cc-07dffc1acb59/crucible/regions/7bdc2f8f-7958-4c7f-b33e-c5825535a329", "421df0b3-15da-4055-97ce-50c89a55dd58"]
512025-02-28T11:19:27.329ZINFOcrucible-agent (dropshot): request completed latency_us = 25392 local_addr = [fd00:1122:3344:101::19]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:38461 req_id = 4f156fcd-22a9-408a-a9e0-2643df3b58a4 response_code = 200 uri = /crucible/0/regions/7bdc2f8f-7958-4c7f-b33e-c5825535a329/snapshots/421df0b3-15da-4055-97ce-50c89a55dd58/run