Max level shown:
12025-02-06T05:22:22.387ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::18]:32345 remote_addr = [fd00:1122:3344:101::a]:34054
22025-02-06T05:22:22.387ZINFOcrucible-agent (dropshot): request completed latency_us = 55 local_addr = [fd00:1122:3344:101::18]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:34054 req_id = 34e4aa01-5528-43ea-a77a-8fea61da924f response_code = 200 uri = /crucible/0/regions/ee71f648-8fae-476d-81f3-424b9e0c059b
32025-02-06T05:22:22.391ZINFOcrucible-agent (datafile): path is "/data/regions/ee71f648-8fae-476d-81f3-424b9e0c059b"
42025-02-06T05:22:22.400ZINFOcrucible-agent (datafile): dataset is oxp_1ebf15f8-05d3-41b9-90d8-afe2e5dd874b/crucible/regions/ee71f648-8fae-476d-81f3-424b9e0c059b
52025-02-06T05:22:22.408ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_1ebf15f8-05d3-41b9-90d8-afe2e5dd874b/crucible/regions/ee71f648-8fae-476d-81f3-424b9e0c059b@254c67cc-cf19-479d-83e2-9a737a2d0f62
62025-02-06T05:22:22.408ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_1ebf15f8-05d3-41b9-90d8-afe2e5dd874b/crucible/regions/ee71f648-8fae-476d-81f3-424b9e0c059b", "254c67cc-cf19-479d-83e2-9a737a2d0f62"]
72025-02-06T05:22:22.419ZINFOcrucible-agent (dropshot): request completed latency_us = 27957 local_addr = [fd00:1122:3344:101::18]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:34054 req_id = 126f16c9-7837-4ced-a762-c2e72aa580df response_code = 200 uri = /crucible/0/regions/ee71f648-8fae-476d-81f3-424b9e0c059b/snapshots/254c67cc-cf19-479d-83e2-9a737a2d0f62
82025-02-06T05:22:22.422ZINFOcrucible-agent (datafile): path is "/data/regions/ee71f648-8fae-476d-81f3-424b9e0c059b"
92025-02-06T05:22:22.429ZINFOcrucible-agent (datafile): dataset is oxp_1ebf15f8-05d3-41b9-90d8-afe2e5dd874b/crucible/regions/ee71f648-8fae-476d-81f3-424b9e0c059b
102025-02-06T05:22:22.437ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_1ebf15f8-05d3-41b9-90d8-afe2e5dd874b/crucible/regions/ee71f648-8fae-476d-81f3-424b9e0c059b@254c67cc-cf19-479d-83e2-9a737a2d0f62
112025-02-06T05:22:22.437ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_1ebf15f8-05d3-41b9-90d8-afe2e5dd874b/crucible/regions/ee71f648-8fae-476d-81f3-424b9e0c059b", "254c67cc-cf19-479d-83e2-9a737a2d0f62"]
122025-02-06T05:22:22.445ZINFOcrucible-agent (datafile): requesting running snapshot ee71f648-8fae-476d-81f3-424b9e0c059b-254c67cc-cf19-479d-83e2-9a737a2d0f62 state: Requested
132025-02-06T05:22:22.446ZINFOcrucible-agent (worker): applying SMF actions for region ee71f648-8fae-476d-81f3-424b9e0c059b running snapshot 254c67cc-cf19-479d-83e2-9a737a2d0f62 (state Requested)...
142025-02-06T05:22:22.446ZINFOcrucible-agent (dropshot): request completed latency_us = 23739 local_addr = [fd00:1122:3344:101::18]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:34054 req_id = 4f3d55c4-2307-4f75-95a1-05950cc2836a response_code = 200 uri = /crucible/0/regions/ee71f648-8fae-476d-81f3-424b9e0c059b/snapshots/254c67cc-cf19-479d-83e2-9a737a2d0f62/run
152025-02-06T05:22:22.452ZINFOcrucible-agent (worker): creating missing snapshot instance snapshot-ee71f648-8fae-476d-81f3-424b9e0c059b-254c67cc-cf19-479d-83e2-9a737a2d0f62
162025-02-06T05:22:22.462ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:snapshot-ee71f648-8fae-476d-81f3-424b9e0c059b-254c67cc-cf19-479d-83e2-9a737a2d0f62
172025-02-06T05:22:22.493ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:snapshot-ee71f648-8fae-476d-81f3-424b9e0c059b-254c67cc-cf19-479d-83e2-9a737a2d0f62
182025-02-06T05:22:22.493ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/ee71f648-8fae-476d-81f3-424b9e0c059b/.zfs/snapshot/254c67cc-cf19-479d-83e2-9a737a2d0f62
192025-02-06T05:22:22.493ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
202025-02-06T05:22:22.493ZINFOcrucible-agent (worker): ensure mode SCF_TYPE_ASTRING ro
212025-02-06T05:22:22.493ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::18
222025-02-06T05:22:22.493ZINFOcrucible-agent (worker): commit
232025-02-06T05:22:22.500ZINFOcrucible-agent (worker): ok!
242025-02-06T05:22:22.528ZINFOcrucible-agent (worker): SMF ok!
252025-02-06T05:22:22.528ZINFOcrucible-agent (datafile): region ee71f648-8fae-476d-81f3-424b9e0c059b running snapshot 254c67cc-cf19-479d-83e2-9a737a2d0f62 state: Requested -> Created
262025-02-06T05:22:22.589ZINFOcrucible-agent (datafile): path is "/data/regions/ee71f648-8fae-476d-81f3-424b9e0c059b"
272025-02-06T05:22:22.598ZINFOcrucible-agent (datafile): dataset is oxp_1ebf15f8-05d3-41b9-90d8-afe2e5dd874b/crucible/regions/ee71f648-8fae-476d-81f3-424b9e0c059b
282025-02-06T05:22:22.607ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_1ebf15f8-05d3-41b9-90d8-afe2e5dd874b/crucible/regions/ee71f648-8fae-476d-81f3-424b9e0c059b@254c67cc-cf19-479d-83e2-9a737a2d0f62
292025-02-06T05:22:22.607ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_1ebf15f8-05d3-41b9-90d8-afe2e5dd874b/crucible/regions/ee71f648-8fae-476d-81f3-424b9e0c059b", "254c67cc-cf19-479d-83e2-9a737a2d0f62"]
302025-02-06T05:22:22.615ZINFOcrucible-agent (dropshot): request completed latency_us = 25287 local_addr = [fd00:1122:3344:101::18]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:34054 req_id = 21b929f3-0bc6-46fa-b167-fbc066acec28 response_code = 200 uri = /crucible/0/regions/ee71f648-8fae-476d-81f3-424b9e0c059b/snapshots/254c67cc-cf19-479d-83e2-9a737a2d0f62/run
312025-02-06T05:22:26.143ZINFOcrucible-agent (datafile): region 402e387d-c534-4adb-b060-bf37002f00b7 state: Requested
322025-02-06T05:22:26.143ZINFOcrucible-agent (dropshot): request completed latency_us = 234 local_addr = [fd00:1122:3344:101::18]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:34054 req_id = 9b5da8a9-379f-4050-b200-d58ead2e895b response_code = 200 uri = /crucible/0/regions
332025-02-06T05:22:26.143ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
342025-02-06T05:22:26.153ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_1ebf15f8-05d3-41b9-90d8-afe2e5dd874b/crucible/regions/402e387d-c534-4adb-b060-bf37002f00b7
352025-02-06T05:22:26.153ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_1ebf15f8-05d3-41b9-90d8-afe2e5dd874b/crucible/regions/402e387d-c534-4adb-b060-bf37002f00b7
362025-02-06T05:22:26.205ZINFOcrucible-agent (worker): creating region Region { id: RegionId("402e387d-c534-4adb-b060-bf37002f00b7"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19002, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/402e387d-c534-4adb-b060-bf37002f00b7" region = 402e387d-c534-4adb-b060-bf37002f00b7
372025-02-06T05:22:26.207ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "402e387d-c534-4adb-b060-bf37002f00b7" "--data" "/data/regions/402e387d-c534-4adb-b060-bf37002f00b7" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 402e387d-c534-4adb-b060-bf37002f00b7
382025-02-06T05:22:26.302ZINFOcrucible-agent (worker): region files created ok region = 402e387d-c534-4adb-b060-bf37002f00b7
392025-02-06T05:22:26.302ZINFOcrucible-agent (datafile): region 402e387d-c534-4adb-b060-bf37002f00b7 state: Requested -> Created
402025-02-06T05:22:26.302ZINFOcrucible-agent (worker): applying SMF actions post create...
412025-02-06T05:22:26.303ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-402e387d-c534-4adb-b060-bf37002f00b7
422025-02-06T05:22:26.306ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-402e387d-c534-4adb-b060-bf37002f00b7
432025-02-06T05:22:26.314ZINFOcrucible-agent (worker): creating config property group
442025-02-06T05:22:26.317ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-402e387d-c534-4adb-b060-bf37002f00b7
452025-02-06T05:22:26.317ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/402e387d-c534-4adb-b060-bf37002f00b7
462025-02-06T05:22:26.317ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19002
472025-02-06T05:22:26.317ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::18
482025-02-06T05:22:26.317ZINFOcrucible-agent (worker): commit
492025-02-06T05:22:26.320ZINFOcrucible-agent (worker): ok!
502025-02-06T05:22:26.331ZINFOcrucible-agent (worker): SMF ok!
512025-02-06T05:22:26.423ZINFOcrucible-agent (dropshot): request completed latency_us = 62 local_addr = [fd00:1122:3344:101::18]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:34054 req_id = f083f817-455b-4dc8-9ddb-ce9884a35f27 response_code = 200 uri = /crucible/0/regions
522025-02-06T05:22:50.994ZINFOcrucible-agent (dropshot): request completed latency_us = 50 local_addr = [fd00:1122:3344:101::18]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:34054 req_id = 047b6e5c-267f-4bb7-8001-6c4533200b5c response_code = 200 uri = /crucible/0/regions/402e387d-c534-4adb-b060-bf37002f00b7
532025-02-06T05:22:50.998ZINFOcrucible-agent (datafile): path is "/data/regions/402e387d-c534-4adb-b060-bf37002f00b7"
542025-02-06T05:22:51.011ZINFOcrucible-agent (datafile): dataset is oxp_1ebf15f8-05d3-41b9-90d8-afe2e5dd874b/crucible/regions/402e387d-c534-4adb-b060-bf37002f00b7
552025-02-06T05:22:51.021ZINFOcrucible-agent (datafile): region 402e387d-c534-4adb-b060-bf37002f00b7 state: Created -> Tombstoned
562025-02-06T05:22:51.021ZINFOcrucible-agent (worker): applying SMF actions before removal...
572025-02-06T05:22:51.021ZINFOcrucible-agent (dropshot): request completed latency_us = 22934 local_addr = [fd00:1122:3344:101::18]:32345 method = DELETE remote_addr = [fd00:1122:3344:101::a]:34054 req_id = 563ba197-f47e-4897-9814-1d2d4642ae80 response_code = 204 uri = /crucible/0/regions/402e387d-c534-4adb-b060-bf37002f00b7
582025-02-06T05:22:51.022ZINFOcrucible-agent (worker): disabling downstairs instance: downstairs-402e387d-c534-4adb-b060-bf37002f00b7 (instance states: (Some(Online), None))
592025-02-06T05:22:51.024ZINFOcrucible-agent (dropshot): request completed latency_us = 35 local_addr = [fd00:1122:3344:101::18]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:34054 req_id = 98489234-9143-4baf-89e7-2160e8eed20c response_code = 200 uri = /crucible/0/regions/402e387d-c534-4adb-b060-bf37002f00b7
602025-02-06T05:22:51.030ZINFOcrucible-agent (worker): SMF ok!
612025-02-06T05:22:51.041ZINFOcrucible-agent (worker): deleting zfs dataset "oxp_1ebf15f8-05d3-41b9-90d8-afe2e5dd874b/crucible/regions/402e387d-c534-4adb-b060-bf37002f00b7" region = 402e387d-c534-4adb-b060-bf37002f00b7
622025-02-06T05:22:51.097ZINFOcrucible-agent (datafile): region 402e387d-c534-4adb-b060-bf37002f00b7 state: Tombstoned -> Destroyed
632025-02-06T05:22:51.143ZINFOcrucible-agent (dropshot): request completed latency_us = 62 local_addr = [fd00:1122:3344:101::18]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:34054 req_id = 9d1c02eb-712b-41f9-b759-bc839d2538d8 response_code = 200 uri = /crucible/0/regions/402e387d-c534-4adb-b060-bf37002f00b7