Max level shown:
12025-03-03T05:35:36.507ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::15]:32345 remote_addr = [fd00:1122:3344:101::b]:52587
22025-03-03T05:35:36.507ZINFOcrucible-agent (datafile): region 4737fc2d-2806-495e-9822-37e9db0f5868 state: Requested
32025-03-03T05:35:36.508ZINFOcrucible-agent (dropshot): request completed latency_us = 927 local_addr = [fd00:1122:3344:101::15]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:52587 req_id = 5bea1e66-8d81-4967-9950-44042a4d0905 response_code = 200 uri = /crucible/0/regions
42025-03-03T05:35:36.508ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
52025-03-03T05:35:36.521ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_fd854483-4975-4105-b707-5010fa5c5d88/crucible/regions/4737fc2d-2806-495e-9822-37e9db0f5868
62025-03-03T05:35:36.521ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_fd854483-4975-4105-b707-5010fa5c5d88/crucible/regions/4737fc2d-2806-495e-9822-37e9db0f5868
72025-03-03T05:35:36.598ZINFOcrucible-agent (worker): creating region Region { id: RegionId("4737fc2d-2806-495e-9822-37e9db0f5868"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19001, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/4737fc2d-2806-495e-9822-37e9db0f5868" region = 4737fc2d-2806-495e-9822-37e9db0f5868
82025-03-03T05:35:36.600ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "4737fc2d-2806-495e-9822-37e9db0f5868" "--data" "/data/regions/4737fc2d-2806-495e-9822-37e9db0f5868" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 4737fc2d-2806-495e-9822-37e9db0f5868
92025-03-03T05:35:36.683ZINFOcrucible-agent (worker): region files created ok region = 4737fc2d-2806-495e-9822-37e9db0f5868
102025-03-03T05:35:36.683ZINFOcrucible-agent (datafile): region 4737fc2d-2806-495e-9822-37e9db0f5868 state: Requested -> Created
112025-03-03T05:35:36.683ZINFOcrucible-agent (worker): applying SMF actions post create...
122025-03-03T05:35:36.684ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-4737fc2d-2806-495e-9822-37e9db0f5868
132025-03-03T05:35:36.690ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-4737fc2d-2806-495e-9822-37e9db0f5868
142025-03-03T05:35:36.702ZINFOcrucible-agent (worker): creating config property group
152025-03-03T05:35:36.705ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-4737fc2d-2806-495e-9822-37e9db0f5868
162025-03-03T05:35:36.706ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/4737fc2d-2806-495e-9822-37e9db0f5868
172025-03-03T05:35:36.706ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
182025-03-03T05:35:36.706ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::15
192025-03-03T05:35:36.706ZINFOcrucible-agent (worker): commit
202025-03-03T05:35:36.710ZINFOcrucible-agent (worker): ok!
212025-03-03T05:35:36.716ZINFOcrucible-agent (worker): SMF ok!
222025-03-03T05:35:36.758ZINFOcrucible-agent (dropshot): request completed latency_us = 70 local_addr = [fd00:1122:3344:101::15]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:52587 req_id = 1b2746bf-9a7b-4570-ae80-6ba14f6322a0 response_code = 200 uri = /crucible/0/regions
232025-03-03T05:35:37.664ZINFOcrucible-agent (dropshot): request completed latency_us = 3376 local_addr = [fd00:1122:3344:101::15]:32345 method = GET remote_addr = [fd00:1122:3344:101::b]:52587 req_id = 38f4d898-3a51-4972-b91c-f6bcbab1f4ff response_code = 200 uri = /crucible/0/regions/79b4ac45-dba6-4f56-a9fc-3a17adbbc612
242025-03-03T05:35:37.670ZINFOcrucible-agent (datafile): path is "/data/regions/79b4ac45-dba6-4f56-a9fc-3a17adbbc612"
252025-03-03T05:35:37.679ZINFOcrucible-agent (datafile): dataset is oxp_fd854483-4975-4105-b707-5010fa5c5d88/crucible/regions/79b4ac45-dba6-4f56-a9fc-3a17adbbc612
262025-03-03T05:35:37.688ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_fd854483-4975-4105-b707-5010fa5c5d88/crucible/regions/79b4ac45-dba6-4f56-a9fc-3a17adbbc612@a66257f2-19e5-45b9-ab36-1aeffae8ef9c
272025-03-03T05:35:37.688ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_fd854483-4975-4105-b707-5010fa5c5d88/crucible/regions/79b4ac45-dba6-4f56-a9fc-3a17adbbc612", "a66257f2-19e5-45b9-ab36-1aeffae8ef9c"]
282025-03-03T05:35:37.699ZINFOcrucible-agent (dropshot): request completed latency_us = 31250 local_addr = [fd00:1122:3344:101::15]:32345 method = GET remote_addr = [fd00:1122:3344:101::b]:52587 req_id = 955a4def-b85b-4d3d-a2ef-a39d55392e66 response_code = 200 uri = /crucible/0/regions/79b4ac45-dba6-4f56-a9fc-3a17adbbc612/snapshots/a66257f2-19e5-45b9-ab36-1aeffae8ef9c
292025-03-03T05:35:37.702ZINFOcrucible-agent (datafile): path is "/data/regions/79b4ac45-dba6-4f56-a9fc-3a17adbbc612"
302025-03-03T05:35:37.710ZINFOcrucible-agent (datafile): dataset is oxp_fd854483-4975-4105-b707-5010fa5c5d88/crucible/regions/79b4ac45-dba6-4f56-a9fc-3a17adbbc612
312025-03-03T05:35:37.718ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_fd854483-4975-4105-b707-5010fa5c5d88/crucible/regions/79b4ac45-dba6-4f56-a9fc-3a17adbbc612@a66257f2-19e5-45b9-ab36-1aeffae8ef9c
322025-03-03T05:35:37.718ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_fd854483-4975-4105-b707-5010fa5c5d88/crucible/regions/79b4ac45-dba6-4f56-a9fc-3a17adbbc612", "a66257f2-19e5-45b9-ab36-1aeffae8ef9c"]
332025-03-03T05:35:37.726ZINFOcrucible-agent (datafile): requesting running snapshot 79b4ac45-dba6-4f56-a9fc-3a17adbbc612-a66257f2-19e5-45b9-ab36-1aeffae8ef9c state: Requested
342025-03-03T05:35:37.727ZINFOcrucible-agent (worker): applying SMF actions for region 79b4ac45-dba6-4f56-a9fc-3a17adbbc612 running snapshot a66257f2-19e5-45b9-ab36-1aeffae8ef9c (state Requested)...
352025-03-03T05:35:37.727ZINFOcrucible-agent (dropshot): request completed latency_us = 24570 local_addr = [fd00:1122:3344:101::15]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:52587 req_id = a6dff534-76c1-4b5c-95cc-5797c3da1edf response_code = 200 uri = /crucible/0/regions/79b4ac45-dba6-4f56-a9fc-3a17adbbc612/snapshots/a66257f2-19e5-45b9-ab36-1aeffae8ef9c/run
362025-03-03T05:35:37.729ZINFOcrucible-agent (worker): creating missing snapshot instance snapshot-79b4ac45-dba6-4f56-a9fc-3a17adbbc612-a66257f2-19e5-45b9-ab36-1aeffae8ef9c
372025-03-03T05:35:37.739ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:snapshot-79b4ac45-dba6-4f56-a9fc-3a17adbbc612-a66257f2-19e5-45b9-ab36-1aeffae8ef9c
382025-03-03T05:35:37.750ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:snapshot-79b4ac45-dba6-4f56-a9fc-3a17adbbc612-a66257f2-19e5-45b9-ab36-1aeffae8ef9c
392025-03-03T05:35:37.750ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/79b4ac45-dba6-4f56-a9fc-3a17adbbc612/.zfs/snapshot/a66257f2-19e5-45b9-ab36-1aeffae8ef9c
402025-03-03T05:35:37.750ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19002
412025-03-03T05:35:37.750ZINFOcrucible-agent (worker): ensure mode SCF_TYPE_ASTRING ro
422025-03-03T05:35:37.750ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::15
432025-03-03T05:35:37.750ZINFOcrucible-agent (worker): commit
442025-03-03T05:35:37.754ZINFOcrucible-agent (worker): ok!
452025-03-03T05:35:37.758ZINFOcrucible-agent (worker): SMF ok!
462025-03-03T05:35:37.758ZINFOcrucible-agent (datafile): region 79b4ac45-dba6-4f56-a9fc-3a17adbbc612 running snapshot a66257f2-19e5-45b9-ab36-1aeffae8ef9c state: Requested -> Created
472025-03-03T05:35:37.895ZINFOcrucible-agent (datafile): path is "/data/regions/79b4ac45-dba6-4f56-a9fc-3a17adbbc612"
482025-03-03T05:35:37.904ZINFOcrucible-agent (datafile): dataset is oxp_fd854483-4975-4105-b707-5010fa5c5d88/crucible/regions/79b4ac45-dba6-4f56-a9fc-3a17adbbc612
492025-03-03T05:35:37.912ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_fd854483-4975-4105-b707-5010fa5c5d88/crucible/regions/79b4ac45-dba6-4f56-a9fc-3a17adbbc612@a66257f2-19e5-45b9-ab36-1aeffae8ef9c
502025-03-03T05:35:37.912ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_fd854483-4975-4105-b707-5010fa5c5d88/crucible/regions/79b4ac45-dba6-4f56-a9fc-3a17adbbc612", "a66257f2-19e5-45b9-ab36-1aeffae8ef9c"]
512025-03-03T05:35:37.921ZINFOcrucible-agent (dropshot): request completed latency_us = 26011 local_addr = [fd00:1122:3344:101::15]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:52587 req_id = 954ba3f3-5509-45c8-8cd7-61123951ae75 response_code = 200 uri = /crucible/0/regions/79b4ac45-dba6-4f56-a9fc-3a17adbbc612/snapshots/a66257f2-19e5-45b9-ab36-1aeffae8ef9c/run