Max level shown:
12026-02-24T02:34:30.065ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::13]:32345 remote_addr = [fd00:1122:3344:101::a]:52656
22026-02-24T02:34:30.065ZINFOcrucible-agent (datafile): region 3d591af9-7ea5-4683-b5d9-a0b82162059a state: Requested
32026-02-24T02:34:30.066ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
42026-02-24T02:34:30.066ZINFOcrucible-agent (dropshot): request completed latency_us = 1379 local_addr = [fd00:1122:3344:101::13]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:52656 req_id = 58b6d1c0-4a23-48b8-93b3-feaf519eeba6 response_code = 200 uri = /crucible/0/regions
52026-02-24T02:34:30.075ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_4b75335b-3adf-4120-b9a1-846e41996b37/crucible/regions/3d591af9-7ea5-4683-b5d9-a0b82162059a
62026-02-24T02:34:30.075ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_4b75335b-3adf-4120-b9a1-846e41996b37/crucible/regions/3d591af9-7ea5-4683-b5d9-a0b82162059a
72026-02-24T02:34:30.132ZINFOcrucible-agent (worker): creating region Region { id: RegionId("3d591af9-7ea5-4683-b5d9-a0b82162059a"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19001, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/3d591af9-7ea5-4683-b5d9-a0b82162059a" region = 3d591af9-7ea5-4683-b5d9-a0b82162059a
82026-02-24T02:34:30.132ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "3d591af9-7ea5-4683-b5d9-a0b82162059a" "--data" "/data/regions/3d591af9-7ea5-4683-b5d9-a0b82162059a" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 3d591af9-7ea5-4683-b5d9-a0b82162059a
92026-02-24T02:34:30.207ZINFOcrucible-agent (worker): region files created ok region = 3d591af9-7ea5-4683-b5d9-a0b82162059a
102026-02-24T02:34:30.207ZINFOcrucible-agent (datafile): region 3d591af9-7ea5-4683-b5d9-a0b82162059a state: Requested -> Created
112026-02-24T02:34:30.207ZINFOcrucible-agent (worker): applying SMF actions post create...
122026-02-24T02:34:30.227ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-3d591af9-7ea5-4683-b5d9-a0b82162059a
132026-02-24T02:34:30.233ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-3d591af9-7ea5-4683-b5d9-a0b82162059a
142026-02-24T02:34:30.246ZINFOcrucible-agent (worker): creating config property group
152026-02-24T02:34:30.249ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-3d591af9-7ea5-4683-b5d9-a0b82162059a
162026-02-24T02:34:30.250ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/3d591af9-7ea5-4683-b5d9-a0b82162059a
172026-02-24T02:34:30.250ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
182026-02-24T02:34:30.250ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::13
192026-02-24T02:34:30.250ZINFOcrucible-agent (worker): commit
202026-02-24T02:34:30.254ZINFOcrucible-agent (worker): ok!
212026-02-24T02:34:30.258ZINFOcrucible-agent (worker): SMF ok!
222026-02-24T02:34:30.420ZINFOcrucible-agent (dropshot): request completed latency_us = 85 local_addr = [fd00:1122:3344:101::13]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:52656 req_id = e0120fe5-273b-48cf-aa68-3719b2db65c9 response_code = 200 uri = /crucible/0/regions
232026-02-24T02:34:31.042ZINFOcrucible-agent (dropshot): request completed latency_us = 77 local_addr = [fd00:1122:3344:101::13]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:52656 req_id = 1e781a1e-fb19-4f47-9177-51334b019f7a response_code = 200 uri = /crucible/0/regions/06879ddb-4f33-49f1-975c-bfa40b6a9e13
242026-02-24T02:34:31.045ZINFOcrucible-agent (datafile): path is "/data/regions/06879ddb-4f33-49f1-975c-bfa40b6a9e13"
252026-02-24T02:34:31.051ZINFOcrucible-agent (datafile): dataset is oxp_4b75335b-3adf-4120-b9a1-846e41996b37/crucible/regions/06879ddb-4f33-49f1-975c-bfa40b6a9e13
262026-02-24T02:34:31.056ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_4b75335b-3adf-4120-b9a1-846e41996b37/crucible/regions/06879ddb-4f33-49f1-975c-bfa40b6a9e13@53a934f6-d542-4134-bed9-a4f304b74bc8
272026-02-24T02:34:31.056ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_4b75335b-3adf-4120-b9a1-846e41996b37/crucible/regions/06879ddb-4f33-49f1-975c-bfa40b6a9e13", "53a934f6-d542-4134-bed9-a4f304b74bc8"]
282026-02-24T02:34:31.065ZINFOcrucible-agent (dropshot): request completed latency_us = 19778 local_addr = [fd00:1122:3344:101::13]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:52656 req_id = 3c836229-e1ae-45f3-93e8-f627244e2345 response_code = 200 uri = /crucible/0/regions/06879ddb-4f33-49f1-975c-bfa40b6a9e13/snapshots/53a934f6-d542-4134-bed9-a4f304b74bc8
292026-02-24T02:34:31.069ZINFOcrucible-agent (datafile): path is "/data/regions/06879ddb-4f33-49f1-975c-bfa40b6a9e13"
302026-02-24T02:34:31.076ZINFOcrucible-agent (datafile): dataset is oxp_4b75335b-3adf-4120-b9a1-846e41996b37/crucible/regions/06879ddb-4f33-49f1-975c-bfa40b6a9e13
312026-02-24T02:34:31.082ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_4b75335b-3adf-4120-b9a1-846e41996b37/crucible/regions/06879ddb-4f33-49f1-975c-bfa40b6a9e13@53a934f6-d542-4134-bed9-a4f304b74bc8
322026-02-24T02:34:31.082ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_4b75335b-3adf-4120-b9a1-846e41996b37/crucible/regions/06879ddb-4f33-49f1-975c-bfa40b6a9e13", "53a934f6-d542-4134-bed9-a4f304b74bc8"]
332026-02-24T02:34:31.089ZINFOcrucible-agent (datafile): requesting running snapshot 06879ddb-4f33-49f1-975c-bfa40b6a9e13-53a934f6-d542-4134-bed9-a4f304b74bc8 state: Requested
342026-02-24T02:34:31.089ZINFOcrucible-agent (worker): applying SMF actions for region 06879ddb-4f33-49f1-975c-bfa40b6a9e13 running snapshot 53a934f6-d542-4134-bed9-a4f304b74bc8 (state Requested)...
352026-02-24T02:34:31.089ZINFOcrucible-agent (dropshot): request completed latency_us = 20428 local_addr = [fd00:1122:3344:101::13]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:52656 req_id = 0bb1c25a-0d66-47ff-a51e-eb575d8616b1 response_code = 200 uri = /crucible/0/regions/06879ddb-4f33-49f1-975c-bfa40b6a9e13/snapshots/53a934f6-d542-4134-bed9-a4f304b74bc8/run
362026-02-24T02:34:31.093ZINFOcrucible-agent (worker): creating missing snapshot instance snapshot-06879ddb-4f33-49f1-975c-bfa40b6a9e13-53a934f6-d542-4134-bed9-a4f304b74bc8
372026-02-24T02:34:31.106ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:snapshot-06879ddb-4f33-49f1-975c-bfa40b6a9e13-53a934f6-d542-4134-bed9-a4f304b74bc8
382026-02-24T02:34:31.116ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:snapshot-06879ddb-4f33-49f1-975c-bfa40b6a9e13-53a934f6-d542-4134-bed9-a4f304b74bc8
392026-02-24T02:34:31.116ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/06879ddb-4f33-49f1-975c-bfa40b6a9e13/.zfs/snapshot/53a934f6-d542-4134-bed9-a4f304b74bc8
402026-02-24T02:34:31.116ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19002
412026-02-24T02:34:31.116ZINFOcrucible-agent (worker): ensure mode SCF_TYPE_ASTRING ro
422026-02-24T02:34:31.116ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::13
432026-02-24T02:34:31.117ZINFOcrucible-agent (worker): commit
442026-02-24T02:34:31.121ZINFOcrucible-agent (worker): ok!
452026-02-24T02:34:31.125ZINFOcrucible-agent (worker): SMF ok!
462026-02-24T02:34:31.125ZINFOcrucible-agent (datafile): region 06879ddb-4f33-49f1-975c-bfa40b6a9e13 running snapshot 53a934f6-d542-4134-bed9-a4f304b74bc8 state: Requested -> Created
472026-02-24T02:34:31.281ZINFOcrucible-agent (datafile): path is "/data/regions/06879ddb-4f33-49f1-975c-bfa40b6a9e13"
482026-02-24T02:34:31.290ZINFOcrucible-agent (datafile): dataset is oxp_4b75335b-3adf-4120-b9a1-846e41996b37/crucible/regions/06879ddb-4f33-49f1-975c-bfa40b6a9e13
492026-02-24T02:34:31.299ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_4b75335b-3adf-4120-b9a1-846e41996b37/crucible/regions/06879ddb-4f33-49f1-975c-bfa40b6a9e13@53a934f6-d542-4134-bed9-a4f304b74bc8
502026-02-24T02:34:31.299ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_4b75335b-3adf-4120-b9a1-846e41996b37/crucible/regions/06879ddb-4f33-49f1-975c-bfa40b6a9e13", "53a934f6-d542-4134-bed9-a4f304b74bc8"]
512026-02-24T02:34:31.307ZINFOcrucible-agent (dropshot): request completed latency_us = 26076 local_addr = [fd00:1122:3344:101::13]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:52656 req_id = ed4a9b10-8590-4b25-ab75-c61814c2374e response_code = 200 uri = /crucible/0/regions/06879ddb-4f33-49f1-975c-bfa40b6a9e13/snapshots/53a934f6-d542-4134-bed9-a4f304b74bc8/run