Max level shown:
12026-07-08T00:45:20.681ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::16]:32345 remote_addr = [fd00:1122:3344:101::a]:53685
22026-07-08T00:45:20.709ZINFOcrucible-agent (datafile): region 36a3eecd-8291-41c7-bd8a-0547e83efebf state: Requested
32026-07-08T00:45:20.713ZINFOcrucible-agent (worker): Region size:1073741824 reservation:1342177280 quota:3221225472
42026-07-08T00:45:20.716ZINFOcrucible-agent (dropshot): request completed latency_us = 15910 local_addr = [fd00:1122:3344:101::16]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:53685 req_id = 0acf5e20-80f4-4d18-9e34-38f911793aed response_code = 200 uri = /crucible/0/regions
52026-07-08T00:45:20.721ZINFOcrucible-agent (worker): zfs set reservation of 1342177280 for oxp_4c58c6df-060c-4308-88cf-2fe47d29a322/crucible/regions/36a3eecd-8291-41c7-bd8a-0547e83efebf
62026-07-08T00:45:20.721ZINFOcrucible-agent (worker): zfs set quota of 3221225472 for oxp_4c58c6df-060c-4308-88cf-2fe47d29a322/crucible/regions/36a3eecd-8291-41c7-bd8a-0547e83efebf
72026-07-08T00:45:20.792ZINFOcrucible-agent (worker): creating region Region { id: RegionId("36a3eecd-8291-41c7-bd8a-0547e83efebf"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 16, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/36a3eecd-8291-41c7-bd8a-0547e83efebf" region = 36a3eecd-8291-41c7-bd8a-0547e83efebf
82026-07-08T00:45:20.796ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "36a3eecd-8291-41c7-bd8a-0547e83efebf" "--data" "/data/regions/36a3eecd-8291-41c7-bd8a-0547e83efebf" "--block-size" "512" "--extent-size" "131072" "--extent-count" "16" "--encrypted" region = 36a3eecd-8291-41c7-bd8a-0547e83efebf
92026-07-08T00:45:20.880ZINFOcrucible-agent (dropshot): request completed latency_us = 68 local_addr = [fd00:1122:3344:101::16]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:53685 req_id = 60055a21-6c7d-4c2d-a719-a475670da5be response_code = 200 uri = /crucible/0/regions
102026-07-08T00:45:20.919ZINFOcrucible-agent (worker): region files created ok region = 36a3eecd-8291-41c7-bd8a-0547e83efebf
112026-07-08T00:45:20.923ZINFOcrucible-agent (datafile): region 36a3eecd-8291-41c7-bd8a-0547e83efebf state: Requested -> Created
122026-07-08T00:45:20.923ZINFOcrucible-agent (worker): applying SMF actions post create...
132026-07-08T00:45:20.926ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-36a3eecd-8291-41c7-bd8a-0547e83efebf
142026-07-08T00:45:20.932ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-36a3eecd-8291-41c7-bd8a-0547e83efebf
152026-07-08T00:45:20.943ZINFOcrucible-agent (worker): creating config property group
162026-07-08T00:45:20.944ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-36a3eecd-8291-41c7-bd8a-0547e83efebf
172026-07-08T00:45:20.945ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/36a3eecd-8291-41c7-bd8a-0547e83efebf
182026-07-08T00:45:20.947ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
192026-07-08T00:45:20.947ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::16
202026-07-08T00:45:20.947ZINFOcrucible-agent (worker): commit
212026-07-08T00:45:20.956ZINFOcrucible-agent (worker): ok!
222026-07-08T00:45:20.960ZINFOcrucible-agent (worker): SMF ok!
232026-07-08T00:45:21.346ZINFOcrucible-agent (dropshot): request completed latency_us = 85 local_addr = [fd00:1122:3344:101::16]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:53685 req_id = 37cf0542-eea6-4e65-ad23-e59381901379 response_code = 200 uri = /crucible/0/regions
242026-07-08T00:45:21.889ZINFOcrucible-agent (dropshot): request completed latency_us = 138 local_addr = [fd00:1122:3344:101::16]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:53685 req_id = f30d76db-3448-4a01-bd35-72e0333fec97 response_code = 200 uri = /crucible/0/regions/36a3eecd-8291-41c7-bd8a-0547e83efebf
252026-07-08T00:45:21.892ZINFOcrucible-agent (datafile): path is "/data/regions/36a3eecd-8291-41c7-bd8a-0547e83efebf"
262026-07-08T00:45:21.900ZINFOcrucible-agent (datafile): dataset is oxp_4c58c6df-060c-4308-88cf-2fe47d29a322/crucible/regions/36a3eecd-8291-41c7-bd8a-0547e83efebf
272026-07-08T00:45:21.907ZINFOcrucible-agent (datafile): region 36a3eecd-8291-41c7-bd8a-0547e83efebf state: Created -> Tombstoned
282026-07-08T00:45:21.908ZINFOcrucible-agent (worker): applying SMF actions before removal...
292026-07-08T00:45:21.908ZINFOcrucible-agent (dropshot): request completed latency_us = 15318 local_addr = [fd00:1122:3344:101::16]:32345 method = DELETE remote_addr = [fd00:1122:3344:101::a]:53685 req_id = b826effc-10ab-49d3-89e3-6bdc47e5b8e8 response_code = 204 uri = /crucible/0/regions/36a3eecd-8291-41c7-bd8a-0547e83efebf
302026-07-08T00:45:21.908ZINFOcrucible-agent (worker): disabling downstairs instance: downstairs-36a3eecd-8291-41c7-bd8a-0547e83efebf (instance states: (Some(Online), None))
312026-07-08T00:45:21.911ZINFOcrucible-agent (dropshot): request completed latency_us = 58 local_addr = [fd00:1122:3344:101::16]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:53685 req_id = 1aa8229c-f188-43eb-b0da-a1bec6730a6c response_code = 200 uri = /crucible/0/regions/36a3eecd-8291-41c7-bd8a-0547e83efebf
322026-07-08T00:45:21.911ZINFOcrucible-agent (worker): SMF ok!
332026-07-08T00:45:21.920ZINFOcrucible-agent (worker): deleting zfs dataset "oxp_4c58c6df-060c-4308-88cf-2fe47d29a322/crucible/regions/36a3eecd-8291-41c7-bd8a-0547e83efebf" region = 36a3eecd-8291-41c7-bd8a-0547e83efebf
342026-07-08T00:45:21.994ZINFOcrucible-agent (datafile): region 36a3eecd-8291-41c7-bd8a-0547e83efebf state: Tombstoned -> Destroyed
352026-07-08T00:45:22.004ZINFOcrucible-agent (dropshot): request completed latency_us = 56 local_addr = [fd00:1122:3344:101::16]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:53685 req_id = 45ea7a33-12ea-4e3c-b1e2-c744925db3a9 response_code = 200 uri = /crucible/0/regions/36a3eecd-8291-41c7-bd8a-0547e83efebf
362026-07-08T00:45:24.117ZINFOcrucible-agent (datafile): region de1f9f5d-7cdf-493e-960a-01252222acf8 state: Requested
372026-07-08T00:45:24.118ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
382026-07-08T00:45:24.118ZINFOcrucible-agent (dropshot): request completed latency_us = 274 local_addr = [fd00:1122:3344:101::16]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:53685 req_id = 7b65711c-dec9-4e98-931d-69d22bc74fa4 response_code = 200 uri = /crucible/0/regions
392026-07-08T00:45:24.124ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_4c58c6df-060c-4308-88cf-2fe47d29a322/crucible/regions/de1f9f5d-7cdf-493e-960a-01252222acf8
402026-07-08T00:45:24.124ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_4c58c6df-060c-4308-88cf-2fe47d29a322/crucible/regions/de1f9f5d-7cdf-493e-960a-01252222acf8
412026-07-08T00:45:24.199ZINFOcrucible-agent (worker): creating region Region { id: RegionId("de1f9f5d-7cdf-493e-960a-01252222acf8"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/de1f9f5d-7cdf-493e-960a-01252222acf8" region = de1f9f5d-7cdf-493e-960a-01252222acf8
422026-07-08T00:45:24.199ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "de1f9f5d-7cdf-493e-960a-01252222acf8" "--data" "/data/regions/de1f9f5d-7cdf-493e-960a-01252222acf8" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = de1f9f5d-7cdf-493e-960a-01252222acf8
432026-07-08T00:45:24.253ZINFOcrucible-agent (worker): region files created ok region = de1f9f5d-7cdf-493e-960a-01252222acf8
442026-07-08T00:45:24.253ZINFOcrucible-agent (datafile): region de1f9f5d-7cdf-493e-960a-01252222acf8 state: Requested -> Created
452026-07-08T00:45:24.253ZINFOcrucible-agent (worker): applying SMF actions post create...
462026-07-08T00:45:24.255ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-de1f9f5d-7cdf-493e-960a-01252222acf8
472026-07-08T00:45:24.256ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-de1f9f5d-7cdf-493e-960a-01252222acf8
482026-07-08T00:45:24.261ZINFOcrucible-agent (worker): creating config property group
492026-07-08T00:45:24.263ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-de1f9f5d-7cdf-493e-960a-01252222acf8
502026-07-08T00:45:24.263ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/de1f9f5d-7cdf-493e-960a-01252222acf8
512026-07-08T00:45:24.264ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
522026-07-08T00:45:24.264ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::16
532026-07-08T00:45:24.264ZINFOcrucible-agent (worker): commit
542026-07-08T00:45:24.267ZINFOcrucible-agent (worker): ok!
552026-07-08T00:45:24.270ZINFOcrucible-agent (worker): SMF ok!
562026-07-08T00:45:24.419ZINFOcrucible-agent (dropshot): request completed latency_us = 56 local_addr = [fd00:1122:3344:101::16]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:53685 req_id = 05d93140-9d92-4277-a261-0bf62af419b8 response_code = 200 uri = /crucible/0/regions
572026-07-08T00:56:26.224ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::16]:32345 remote_addr = [fd00:1122:3344:101::a]:39818
582026-07-08T00:56:26.224ZINFOcrucible-agent (datafile): region b0456164-967e-47f5-b4c3-21246991d0ac state: Requested
592026-07-08T00:56:26.225ZINFOcrucible-agent (dropshot): request completed latency_us = 766 local_addr = [fd00:1122:3344:101::16]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:39818 req_id = 27225386-403b-4f0a-ad5d-27a06b8eee13 response_code = 200 uri = /crucible/0/regions
602026-07-08T00:56:26.225ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
612026-07-08T00:56:26.235ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_4c58c6df-060c-4308-88cf-2fe47d29a322/crucible/regions/b0456164-967e-47f5-b4c3-21246991d0ac
622026-07-08T00:56:26.235ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_4c58c6df-060c-4308-88cf-2fe47d29a322/crucible/regions/b0456164-967e-47f5-b4c3-21246991d0ac
632026-07-08T00:56:26.294ZINFOcrucible-agent (worker): creating region Region { id: RegionId("b0456164-967e-47f5-b4c3-21246991d0ac"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19001, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/b0456164-967e-47f5-b4c3-21246991d0ac" region = b0456164-967e-47f5-b4c3-21246991d0ac
642026-07-08T00:56:26.294ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "b0456164-967e-47f5-b4c3-21246991d0ac" "--data" "/data/regions/b0456164-967e-47f5-b4c3-21246991d0ac" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = b0456164-967e-47f5-b4c3-21246991d0ac
652026-07-08T00:56:26.349ZINFOcrucible-agent (worker): region files created ok region = b0456164-967e-47f5-b4c3-21246991d0ac
662026-07-08T00:56:26.349ZINFOcrucible-agent (datafile): region b0456164-967e-47f5-b4c3-21246991d0ac state: Requested -> Created
672026-07-08T00:56:26.349ZINFOcrucible-agent (worker): applying SMF actions post create...
682026-07-08T00:56:26.350ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-b0456164-967e-47f5-b4c3-21246991d0ac
692026-07-08T00:56:26.370ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-b0456164-967e-47f5-b4c3-21246991d0ac
702026-07-08T00:56:26.383ZINFOcrucible-agent (worker): creating config property group
712026-07-08T00:56:26.384ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-b0456164-967e-47f5-b4c3-21246991d0ac
722026-07-08T00:56:26.385ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/b0456164-967e-47f5-b4c3-21246991d0ac
732026-07-08T00:56:26.385ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
742026-07-08T00:56:26.385ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::16
752026-07-08T00:56:26.385ZINFOcrucible-agent (worker): commit
762026-07-08T00:56:26.387ZINFOcrucible-agent (worker): ok!
772026-07-08T00:56:26.390ZINFOcrucible-agent (worker): SMF ok!
782026-07-08T00:56:26.391ZINFOcrucible-agent (dropshot): request completed latency_us = 66 local_addr = [fd00:1122:3344:101::16]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:39818 req_id = e0a62007-788f-46d1-b920-0e040c210f49 response_code = 200 uri = /crucible/0/regions
792026-07-08T00:56:28.098ZINFOcrucible-agent (dropshot): request completed latency_us = 52 local_addr = [fd00:1122:3344:101::16]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:39818 req_id = f6d324ad-8bd1-4d57-9bfb-7e01e93c4d62 response_code = 200 uri = /crucible/0/regions/de1f9f5d-7cdf-493e-960a-01252222acf8
802026-07-08T00:56:28.102ZINFOcrucible-agent (datafile): path is "/data/regions/de1f9f5d-7cdf-493e-960a-01252222acf8"
812026-07-08T00:56:28.109ZINFOcrucible-agent (datafile): dataset is oxp_4c58c6df-060c-4308-88cf-2fe47d29a322/crucible/regions/de1f9f5d-7cdf-493e-960a-01252222acf8
822026-07-08T00:56:28.115ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_4c58c6df-060c-4308-88cf-2fe47d29a322/crucible/regions/de1f9f5d-7cdf-493e-960a-01252222acf8@e12fb882-5493-463a-a447-273c03e9b64a
832026-07-08T00:56:28.115ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_4c58c6df-060c-4308-88cf-2fe47d29a322/crucible/regions/de1f9f5d-7cdf-493e-960a-01252222acf8", "e12fb882-5493-463a-a447-273c03e9b64a"]
842026-07-08T00:56:28.127ZINFOcrucible-agent (dropshot): request completed latency_us = 24801 local_addr = [fd00:1122:3344:101::16]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:39818 req_id = a008404c-4de3-4145-a3b7-5f2218fd7610 response_code = 200 uri = /crucible/0/regions/de1f9f5d-7cdf-493e-960a-01252222acf8/snapshots/e12fb882-5493-463a-a447-273c03e9b64a
852026-07-08T00:56:28.130ZINFOcrucible-agent (datafile): path is "/data/regions/de1f9f5d-7cdf-493e-960a-01252222acf8"
862026-07-08T00:56:28.136ZINFOcrucible-agent (datafile): dataset is oxp_4c58c6df-060c-4308-88cf-2fe47d29a322/crucible/regions/de1f9f5d-7cdf-493e-960a-01252222acf8
872026-07-08T00:56:28.141ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_4c58c6df-060c-4308-88cf-2fe47d29a322/crucible/regions/de1f9f5d-7cdf-493e-960a-01252222acf8@e12fb882-5493-463a-a447-273c03e9b64a
882026-07-08T00:56:28.141ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_4c58c6df-060c-4308-88cf-2fe47d29a322/crucible/regions/de1f9f5d-7cdf-493e-960a-01252222acf8", "e12fb882-5493-463a-a447-273c03e9b64a"]
892026-07-08T00:56:28.149ZINFOcrucible-agent (datafile): requesting running snapshot de1f9f5d-7cdf-493e-960a-01252222acf8-e12fb882-5493-463a-a447-273c03e9b64a state: Requested
902026-07-08T00:56:28.152ZINFOcrucible-agent (worker): applying SMF actions for region de1f9f5d-7cdf-493e-960a-01252222acf8 running snapshot e12fb882-5493-463a-a447-273c03e9b64a (state Requested)...
912026-07-08T00:56:28.152ZINFOcrucible-agent (dropshot): request completed latency_us = 22116 local_addr = [fd00:1122:3344:101::16]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:39818 req_id = bc5a6e1a-fce2-49e2-bc88-50b6d5af5872 response_code = 200 uri = /crucible/0/regions/de1f9f5d-7cdf-493e-960a-01252222acf8/snapshots/e12fb882-5493-463a-a447-273c03e9b64a/run
922026-07-08T00:56:28.157ZINFOcrucible-agent (worker): creating missing snapshot instance snapshot-de1f9f5d-7cdf-493e-960a-01252222acf8-e12fb882-5493-463a-a447-273c03e9b64a
932026-07-08T00:56:28.158ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:snapshot-de1f9f5d-7cdf-493e-960a-01252222acf8-e12fb882-5493-463a-a447-273c03e9b64a
942026-07-08T00:56:28.163ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:snapshot-de1f9f5d-7cdf-493e-960a-01252222acf8-e12fb882-5493-463a-a447-273c03e9b64a
952026-07-08T00:56:28.163ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/de1f9f5d-7cdf-493e-960a-01252222acf8/.zfs/snapshot/e12fb882-5493-463a-a447-273c03e9b64a
962026-07-08T00:56:28.163ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19002
972026-07-08T00:56:28.163ZINFOcrucible-agent (worker): ensure mode SCF_TYPE_ASTRING ro
982026-07-08T00:56:28.163ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::16
992026-07-08T00:56:28.163ZINFOcrucible-agent (worker): commit
1002026-07-08T00:56:28.165ZINFOcrucible-agent (worker): ok!
1012026-07-08T00:56:28.168ZINFOcrucible-agent (worker): SMF ok!
1022026-07-08T00:56:28.168ZINFOcrucible-agent (datafile): region de1f9f5d-7cdf-493e-960a-01252222acf8 running snapshot e12fb882-5493-463a-a447-273c03e9b64a state: Requested -> Created
1032026-07-08T00:56:28.396ZINFOcrucible-agent (datafile): path is "/data/regions/de1f9f5d-7cdf-493e-960a-01252222acf8"
1042026-07-08T00:56:28.404ZINFOcrucible-agent (datafile): dataset is oxp_4c58c6df-060c-4308-88cf-2fe47d29a322/crucible/regions/de1f9f5d-7cdf-493e-960a-01252222acf8
1052026-07-08T00:56:28.408ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_4c58c6df-060c-4308-88cf-2fe47d29a322/crucible/regions/de1f9f5d-7cdf-493e-960a-01252222acf8@e12fb882-5493-463a-a447-273c03e9b64a
1062026-07-08T00:56:28.408ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_4c58c6df-060c-4308-88cf-2fe47d29a322/crucible/regions/de1f9f5d-7cdf-493e-960a-01252222acf8", "e12fb882-5493-463a-a447-273c03e9b64a"]
1072026-07-08T00:56:28.414ZINFOcrucible-agent (dropshot): request completed latency_us = 18260 local_addr = [fd00:1122:3344:101::16]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:39818 req_id = 8effac18-3c54-48b9-9590-f2378f87d698 response_code = 200 uri = /crucible/0/regions/de1f9f5d-7cdf-493e-960a-01252222acf8/snapshots/e12fb882-5493-463a-a447-273c03e9b64a/run