Max level shown:
12025-12-19T04:24:33.016ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::19]:32345 remote_addr = [fd00:1122:3344:101::b]:36131
22025-12-19T04:24:33.016ZINFOcrucible-agent (datafile): region 60dc7013-f77c-45db-a8a8-b5603bb44f45 state: Requested
32025-12-19T04:24:33.016ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
42025-12-19T04:24:33.016ZINFOcrucible-agent (dropshot): request completed latency_us = 620 local_addr = [fd00:1122:3344:101::19]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:36131 req_id = 0c05fb16-5184-482c-9842-12c7c0376797 response_code = 200 uri = /crucible/0/regions
52025-12-19T04:24:33.028ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_0775ebfc-8730-4593-b887-e1c4a88c6e27/crucible/regions/60dc7013-f77c-45db-a8a8-b5603bb44f45
62025-12-19T04:24:33.028ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_0775ebfc-8730-4593-b887-e1c4a88c6e27/crucible/regions/60dc7013-f77c-45db-a8a8-b5603bb44f45
72025-12-19T04:24:33.087ZINFOcrucible-agent (worker): creating region Region { id: RegionId("60dc7013-f77c-45db-a8a8-b5603bb44f45"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19001, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/60dc7013-f77c-45db-a8a8-b5603bb44f45" region = 60dc7013-f77c-45db-a8a8-b5603bb44f45
82025-12-19T04:24:33.087ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "60dc7013-f77c-45db-a8a8-b5603bb44f45" "--data" "/data/regions/60dc7013-f77c-45db-a8a8-b5603bb44f45" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 60dc7013-f77c-45db-a8a8-b5603bb44f45
92025-12-19T04:24:33.214ZINFOcrucible-agent (worker): region files created ok region = 60dc7013-f77c-45db-a8a8-b5603bb44f45
102025-12-19T04:24:33.214ZINFOcrucible-agent (datafile): region 60dc7013-f77c-45db-a8a8-b5603bb44f45 state: Requested -> Created
112025-12-19T04:24:33.215ZINFOcrucible-agent (worker): applying SMF actions post create...
122025-12-19T04:24:33.233ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-60dc7013-f77c-45db-a8a8-b5603bb44f45
132025-12-19T04:24:33.240ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-60dc7013-f77c-45db-a8a8-b5603bb44f45
142025-12-19T04:24:33.251ZINFOcrucible-agent (worker): creating config property group
152025-12-19T04:24:33.254ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-60dc7013-f77c-45db-a8a8-b5603bb44f45
162025-12-19T04:24:33.255ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/60dc7013-f77c-45db-a8a8-b5603bb44f45
172025-12-19T04:24:33.255ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
182025-12-19T04:24:33.255ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::19
192025-12-19T04:24:33.255ZINFOcrucible-agent (worker): commit
202025-12-19T04:24:33.259ZINFOcrucible-agent (worker): ok!
212025-12-19T04:24:33.264ZINFOcrucible-agent (worker): SMF ok!
222025-12-19T04:24:33.303ZINFOcrucible-agent (dropshot): request completed latency_us = 54 local_addr = [fd00:1122:3344:101::19]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:36131 req_id = 0854c0d4-a7f3-437b-9c9f-efccd1076a74 response_code = 200 uri = /crucible/0/regions
232025-12-19T04:24:33.794ZINFOcrucible-agent (dropshot): request completed latency_us = 72 local_addr = [fd00:1122:3344:101::19]:32345 method = GET remote_addr = [fd00:1122:3344:101::b]:36131 req_id = e3e005b7-f5d4-4d83-ba68-6a9bca83e99c response_code = 200 uri = /crucible/0/regions/3ef57f6e-8525-400a-8729-be239bb6563f
242025-12-19T04:24:33.799ZINFOcrucible-agent (datafile): path is "/data/regions/3ef57f6e-8525-400a-8729-be239bb6563f"
252025-12-19T04:24:33.806ZINFOcrucible-agent (datafile): dataset is oxp_0775ebfc-8730-4593-b887-e1c4a88c6e27/crucible/regions/3ef57f6e-8525-400a-8729-be239bb6563f
262025-12-19T04:24:33.812ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_0775ebfc-8730-4593-b887-e1c4a88c6e27/crucible/regions/3ef57f6e-8525-400a-8729-be239bb6563f@88ae9946-f8e6-4568-9a00-28d43135de8c
272025-12-19T04:24:33.812ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_0775ebfc-8730-4593-b887-e1c4a88c6e27/crucible/regions/3ef57f6e-8525-400a-8729-be239bb6563f", "88ae9946-f8e6-4568-9a00-28d43135de8c"]
282025-12-19T04:24:33.821ZINFOcrucible-agent (dropshot): request completed latency_us = 21371 local_addr = [fd00:1122:3344:101::19]:32345 method = GET remote_addr = [fd00:1122:3344:101::b]:36131 req_id = d6fe86a7-2438-4840-aa1d-4ca065d1aad6 response_code = 200 uri = /crucible/0/regions/3ef57f6e-8525-400a-8729-be239bb6563f/snapshots/88ae9946-f8e6-4568-9a00-28d43135de8c
292025-12-19T04:24:33.828ZINFOcrucible-agent (datafile): path is "/data/regions/3ef57f6e-8525-400a-8729-be239bb6563f"
302025-12-19T04:24:33.833ZINFOcrucible-agent (datafile): dataset is oxp_0775ebfc-8730-4593-b887-e1c4a88c6e27/crucible/regions/3ef57f6e-8525-400a-8729-be239bb6563f
312025-12-19T04:24:33.839ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_0775ebfc-8730-4593-b887-e1c4a88c6e27/crucible/regions/3ef57f6e-8525-400a-8729-be239bb6563f@88ae9946-f8e6-4568-9a00-28d43135de8c
322025-12-19T04:24:33.839ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_0775ebfc-8730-4593-b887-e1c4a88c6e27/crucible/regions/3ef57f6e-8525-400a-8729-be239bb6563f", "88ae9946-f8e6-4568-9a00-28d43135de8c"]
332025-12-19T04:24:33.845ZINFOcrucible-agent (datafile): requesting running snapshot 3ef57f6e-8525-400a-8729-be239bb6563f-88ae9946-f8e6-4568-9a00-28d43135de8c state: Requested
342025-12-19T04:24:33.845ZINFOcrucible-agent (worker): applying SMF actions for region 3ef57f6e-8525-400a-8729-be239bb6563f running snapshot 88ae9946-f8e6-4568-9a00-28d43135de8c (state Requested)...
352025-12-19T04:24:33.845ZINFOcrucible-agent (dropshot): request completed latency_us = 16713 local_addr = [fd00:1122:3344:101::19]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:36131 req_id = 6a9cf4a6-ba2d-46f5-b5b1-534ad4eb30c9 response_code = 200 uri = /crucible/0/regions/3ef57f6e-8525-400a-8729-be239bb6563f/snapshots/88ae9946-f8e6-4568-9a00-28d43135de8c/run
362025-12-19T04:24:33.850ZINFOcrucible-agent (worker): creating missing snapshot instance snapshot-3ef57f6e-8525-400a-8729-be239bb6563f-88ae9946-f8e6-4568-9a00-28d43135de8c
372025-12-19T04:24:33.859ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:snapshot-3ef57f6e-8525-400a-8729-be239bb6563f-88ae9946-f8e6-4568-9a00-28d43135de8c
382025-12-19T04:24:33.870ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:snapshot-3ef57f6e-8525-400a-8729-be239bb6563f-88ae9946-f8e6-4568-9a00-28d43135de8c
392025-12-19T04:24:33.870ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/3ef57f6e-8525-400a-8729-be239bb6563f/.zfs/snapshot/88ae9946-f8e6-4568-9a00-28d43135de8c
402025-12-19T04:24:33.870ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19002
412025-12-19T04:24:33.870ZINFOcrucible-agent (worker): ensure mode SCF_TYPE_ASTRING ro
422025-12-19T04:24:33.870ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::19
432025-12-19T04:24:33.870ZINFOcrucible-agent (worker): commit
442025-12-19T04:24:33.874ZINFOcrucible-agent (worker): ok!
452025-12-19T04:24:33.879ZINFOcrucible-agent (worker): SMF ok!
462025-12-19T04:24:33.879ZINFOcrucible-agent (datafile): region 3ef57f6e-8525-400a-8729-be239bb6563f running snapshot 88ae9946-f8e6-4568-9a00-28d43135de8c state: Requested -> Created
472025-12-19T04:24:34.126ZINFOcrucible-agent (datafile): path is "/data/regions/3ef57f6e-8525-400a-8729-be239bb6563f"
482025-12-19T04:24:34.133ZINFOcrucible-agent (datafile): dataset is oxp_0775ebfc-8730-4593-b887-e1c4a88c6e27/crucible/regions/3ef57f6e-8525-400a-8729-be239bb6563f
492025-12-19T04:24:34.140ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_0775ebfc-8730-4593-b887-e1c4a88c6e27/crucible/regions/3ef57f6e-8525-400a-8729-be239bb6563f@88ae9946-f8e6-4568-9a00-28d43135de8c
502025-12-19T04:24:34.140ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_0775ebfc-8730-4593-b887-e1c4a88c6e27/crucible/regions/3ef57f6e-8525-400a-8729-be239bb6563f", "88ae9946-f8e6-4568-9a00-28d43135de8c"]
512025-12-19T04:24:34.146ZINFOcrucible-agent (dropshot): request completed latency_us = 20046 local_addr = [fd00:1122:3344:101::19]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:36131 req_id = 977c7488-1ed6-428a-9696-b74548c6da40 response_code = 200 uri = /crucible/0/regions/3ef57f6e-8525-400a-8729-be239bb6563f/snapshots/88ae9946-f8e6-4568-9a00-28d43135de8c/run