Max level shown:
12026-11-04T00:38:32.535ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::1a]:32345 remote_addr = [fd00:1122:3344:101::c]:54975
22026-11-04T00:38:32.536ZINFOcrucible-agent (datafile): region 06cf8768-b8b3-4389-96fb-0652ed4dd8f3 state: Requested
32026-11-04T00:38:32.536ZINFOcrucible-agent (dropshot): request completed latency_us = 774 local_addr = [fd00:1122:3344:101::1a]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:54975 req_id = 91012681-5356-4ad8-b2d6-517cceba9537 response_code = 200 uri = /crucible/0/regions
42026-11-04T00:38:32.536ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
52026-11-04T00:38:32.546ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_dcecfb60-8c6f-4ad8-9d24-50569e8a4e20/crucible/regions/06cf8768-b8b3-4389-96fb-0652ed4dd8f3
62026-11-04T00:38:32.546ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_dcecfb60-8c6f-4ad8-9d24-50569e8a4e20/crucible/regions/06cf8768-b8b3-4389-96fb-0652ed4dd8f3
72026-11-04T00:38:32.584ZINFOcrucible-agent (worker): creating region Region { id: RegionId("06cf8768-b8b3-4389-96fb-0652ed4dd8f3"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19001, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/06cf8768-b8b3-4389-96fb-0652ed4dd8f3" region = 06cf8768-b8b3-4389-96fb-0652ed4dd8f3
82026-11-04T00:38:32.584ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "06cf8768-b8b3-4389-96fb-0652ed4dd8f3" "--data" "/data/regions/06cf8768-b8b3-4389-96fb-0652ed4dd8f3" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 06cf8768-b8b3-4389-96fb-0652ed4dd8f3
92026-11-04T00:38:32.632ZINFOcrucible-agent (worker): region files created ok region = 06cf8768-b8b3-4389-96fb-0652ed4dd8f3
102026-11-04T00:38:32.632ZINFOcrucible-agent (datafile): region 06cf8768-b8b3-4389-96fb-0652ed4dd8f3 state: Requested -> Created
112026-11-04T00:38:32.633ZINFOcrucible-agent (worker): applying SMF actions post create...
122026-11-04T00:38:32.633ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-06cf8768-b8b3-4389-96fb-0652ed4dd8f3
132026-11-04T00:38:32.646ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-06cf8768-b8b3-4389-96fb-0652ed4dd8f3
142026-11-04T00:38:32.663ZINFOcrucible-agent (worker): creating config property group
152026-11-04T00:38:32.664ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-06cf8768-b8b3-4389-96fb-0652ed4dd8f3
162026-11-04T00:38:32.665ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/06cf8768-b8b3-4389-96fb-0652ed4dd8f3
172026-11-04T00:38:32.665ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
182026-11-04T00:38:32.665ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::1a
192026-11-04T00:38:32.665ZINFOcrucible-agent (worker): commit
202026-11-04T00:38:32.667ZINFOcrucible-agent (worker): ok!
212026-11-04T00:38:32.670ZINFOcrucible-agent (worker): SMF ok!
222026-11-04T00:38:32.861ZINFOcrucible-agent (dropshot): request completed latency_us = 60 local_addr = [fd00:1122:3344:101::1a]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:54975 req_id = 8875a827-96e3-45bd-97bf-4bc313d02cb3 response_code = 200 uri = /crucible/0/regions
232026-11-04T00:38:33.657ZINFOcrucible-agent (dropshot): request completed latency_us = 53 local_addr = [fd00:1122:3344:101::1a]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:54975 req_id = 12859051-5dd5-4737-8e28-5bfca356a5d0 response_code = 200 uri = /crucible/0/regions/b42400db-0ce0-44de-9bdd-8c3811bf8507
242026-11-04T00:38:33.666ZINFOcrucible-agent (datafile): path is "/data/regions/b42400db-0ce0-44de-9bdd-8c3811bf8507"
252026-11-04T00:38:33.672ZINFOcrucible-agent (datafile): dataset is oxp_dcecfb60-8c6f-4ad8-9d24-50569e8a4e20/crucible/regions/b42400db-0ce0-44de-9bdd-8c3811bf8507
262026-11-04T00:38:33.677ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_dcecfb60-8c6f-4ad8-9d24-50569e8a4e20/crucible/regions/b42400db-0ce0-44de-9bdd-8c3811bf8507@39f5dcea-a288-4fa1-b158-348721670201
272026-11-04T00:38:33.677ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_dcecfb60-8c6f-4ad8-9d24-50569e8a4e20/crucible/regions/b42400db-0ce0-44de-9bdd-8c3811bf8507", "39f5dcea-a288-4fa1-b158-348721670201"]
282026-11-04T00:38:33.687ZINFOcrucible-agent (dropshot): request completed latency_us = 23806 local_addr = [fd00:1122:3344:101::1a]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:54975 req_id = c83b49d4-6b36-44ab-b1ad-3027a4f76e86 response_code = 200 uri = /crucible/0/regions/b42400db-0ce0-44de-9bdd-8c3811bf8507/snapshots/39f5dcea-a288-4fa1-b158-348721670201
292026-11-04T00:38:33.693ZINFOcrucible-agent (datafile): path is "/data/regions/b42400db-0ce0-44de-9bdd-8c3811bf8507"
302026-11-04T00:38:33.700ZINFOcrucible-agent (datafile): dataset is oxp_dcecfb60-8c6f-4ad8-9d24-50569e8a4e20/crucible/regions/b42400db-0ce0-44de-9bdd-8c3811bf8507
312026-11-04T00:38:33.705ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_dcecfb60-8c6f-4ad8-9d24-50569e8a4e20/crucible/regions/b42400db-0ce0-44de-9bdd-8c3811bf8507@39f5dcea-a288-4fa1-b158-348721670201
322026-11-04T00:38:33.705ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_dcecfb60-8c6f-4ad8-9d24-50569e8a4e20/crucible/regions/b42400db-0ce0-44de-9bdd-8c3811bf8507", "39f5dcea-a288-4fa1-b158-348721670201"]
332026-11-04T00:38:33.711ZINFOcrucible-agent (datafile): requesting running snapshot b42400db-0ce0-44de-9bdd-8c3811bf8507-39f5dcea-a288-4fa1-b158-348721670201 state: Requested
342026-11-04T00:38:33.711ZINFOcrucible-agent (worker): applying SMF actions for region b42400db-0ce0-44de-9bdd-8c3811bf8507 running snapshot 39f5dcea-a288-4fa1-b158-348721670201 (state Requested)...
352026-11-04T00:38:33.711ZINFOcrucible-agent (dropshot): request completed latency_us = 18027 local_addr = [fd00:1122:3344:101::1a]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:54975 req_id = 735c95f5-c8b2-4e5a-89d1-4bebbdd97533 response_code = 200 uri = /crucible/0/regions/b42400db-0ce0-44de-9bdd-8c3811bf8507/snapshots/39f5dcea-a288-4fa1-b158-348721670201/run
362026-11-04T00:38:33.714ZINFOcrucible-agent (worker): creating missing snapshot instance snapshot-b42400db-0ce0-44de-9bdd-8c3811bf8507-39f5dcea-a288-4fa1-b158-348721670201
372026-11-04T00:38:33.715ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:snapshot-b42400db-0ce0-44de-9bdd-8c3811bf8507-39f5dcea-a288-4fa1-b158-348721670201
382026-11-04T00:38:33.720ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:snapshot-b42400db-0ce0-44de-9bdd-8c3811bf8507-39f5dcea-a288-4fa1-b158-348721670201
392026-11-04T00:38:33.720ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/b42400db-0ce0-44de-9bdd-8c3811bf8507/.zfs/snapshot/39f5dcea-a288-4fa1-b158-348721670201
402026-11-04T00:38:33.720ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19002
412026-11-04T00:38:33.720ZINFOcrucible-agent (worker): ensure mode SCF_TYPE_ASTRING ro
422026-11-04T00:38:33.720ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::1a
432026-11-04T00:38:33.720ZINFOcrucible-agent (worker): commit
442026-11-04T00:38:33.722ZINFOcrucible-agent (worker): ok!
452026-11-04T00:38:33.725ZINFOcrucible-agent (worker): SMF ok!
462026-11-04T00:38:33.725ZINFOcrucible-agent (datafile): region b42400db-0ce0-44de-9bdd-8c3811bf8507 running snapshot 39f5dcea-a288-4fa1-b158-348721670201 state: Requested -> Created
472026-11-04T00:38:33.845ZINFOcrucible-agent (datafile): path is "/data/regions/b42400db-0ce0-44de-9bdd-8c3811bf8507"
482026-11-04T00:38:33.851ZINFOcrucible-agent (datafile): dataset is oxp_dcecfb60-8c6f-4ad8-9d24-50569e8a4e20/crucible/regions/b42400db-0ce0-44de-9bdd-8c3811bf8507
492026-11-04T00:38:33.858ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_dcecfb60-8c6f-4ad8-9d24-50569e8a4e20/crucible/regions/b42400db-0ce0-44de-9bdd-8c3811bf8507@39f5dcea-a288-4fa1-b158-348721670201
502026-11-04T00:38:33.858ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_dcecfb60-8c6f-4ad8-9d24-50569e8a4e20/crucible/regions/b42400db-0ce0-44de-9bdd-8c3811bf8507", "39f5dcea-a288-4fa1-b158-348721670201"]
512026-11-04T00:38:33.864ZINFOcrucible-agent (dropshot): request completed latency_us = 18735 local_addr = [fd00:1122:3344:101::1a]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:54975 req_id = a2288d57-d23d-4267-a763-9b7020ef46f5 response_code = 200 uri = /crucible/0/regions/b42400db-0ce0-44de-9bdd-8c3811bf8507/snapshots/39f5dcea-a288-4fa1-b158-348721670201/run