Max level shown:
12026-03-24T15:38:12.939ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::12]:32345 remote_addr = [fd00:1122:3344:101::c]:50735
22026-03-24T15:38:12.940ZINFOcrucible-agent (datafile): region ddcd0022-57bf-4bb6-9f9b-3546b51b215a state: Requested
32026-03-24T15:38:12.940ZINFOcrucible-agent (dropshot): request completed latency_us = 769 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:50735 req_id = 7348680d-b1a1-42e1-a40e-fcde2368d58d response_code = 200 uri = /crucible/0/regions
42026-03-24T15:38:12.940ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
52026-03-24T15:38:12.950ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_1fc7a4a2-1bda-4116-a50a-1b8a72f173aa/crucible/regions/ddcd0022-57bf-4bb6-9f9b-3546b51b215a
62026-03-24T15:38:12.951ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_1fc7a4a2-1bda-4116-a50a-1b8a72f173aa/crucible/regions/ddcd0022-57bf-4bb6-9f9b-3546b51b215a
72026-03-24T15:38:13.006ZINFOcrucible-agent (worker): creating region Region { id: RegionId("ddcd0022-57bf-4bb6-9f9b-3546b51b215a"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19001, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/ddcd0022-57bf-4bb6-9f9b-3546b51b215a" region = ddcd0022-57bf-4bb6-9f9b-3546b51b215a
82026-03-24T15:38:13.006ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "ddcd0022-57bf-4bb6-9f9b-3546b51b215a" "--data" "/data/regions/ddcd0022-57bf-4bb6-9f9b-3546b51b215a" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = ddcd0022-57bf-4bb6-9f9b-3546b51b215a
92026-03-24T15:38:13.123ZINFOcrucible-agent (worker): region files created ok region = ddcd0022-57bf-4bb6-9f9b-3546b51b215a
102026-03-24T15:38:13.123ZINFOcrucible-agent (datafile): region ddcd0022-57bf-4bb6-9f9b-3546b51b215a state: Requested -> Created
112026-03-24T15:38:13.123ZINFOcrucible-agent (worker): applying SMF actions post create...
122026-03-24T15:38:13.143ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-ddcd0022-57bf-4bb6-9f9b-3546b51b215a
132026-03-24T15:38:13.149ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-ddcd0022-57bf-4bb6-9f9b-3546b51b215a
142026-03-24T15:38:13.160ZINFOcrucible-agent (worker): creating config property group
152026-03-24T15:38:13.164ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-ddcd0022-57bf-4bb6-9f9b-3546b51b215a
162026-03-24T15:38:13.164ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/ddcd0022-57bf-4bb6-9f9b-3546b51b215a
172026-03-24T15:38:13.164ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
182026-03-24T15:38:13.164ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::12
192026-03-24T15:38:13.164ZINFOcrucible-agent (worker): commit
202026-03-24T15:38:13.168ZINFOcrucible-agent (worker): ok!
212026-03-24T15:38:13.173ZINFOcrucible-agent (worker): SMF ok!
222026-03-24T15:38:13.288ZINFOcrucible-agent (dropshot): request completed latency_us = 65 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:50735 req_id = 84787d74-b261-4ebf-a4b7-69bc5fad898e response_code = 200 uri = /crucible/0/regions
232026-03-24T15:38:14.439ZINFOcrucible-agent (dropshot): request completed latency_us = 85 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:50735 req_id = 329ea49e-24e4-4a58-9733-3fd4aab43e9c response_code = 200 uri = /crucible/0/regions/4356bdaf-4f57-413e-b5fb-080585002d02
242026-03-24T15:38:14.443ZINFOcrucible-agent (datafile): path is "/data/regions/4356bdaf-4f57-413e-b5fb-080585002d02"
252026-03-24T15:38:14.450ZINFOcrucible-agent (datafile): dataset is oxp_1fc7a4a2-1bda-4116-a50a-1b8a72f173aa/crucible/regions/4356bdaf-4f57-413e-b5fb-080585002d02
262026-03-24T15:38:14.455ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_1fc7a4a2-1bda-4116-a50a-1b8a72f173aa/crucible/regions/4356bdaf-4f57-413e-b5fb-080585002d02@04f0b6c6-5b1f-41ce-8991-67ed112946a3
272026-03-24T15:38:14.455ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_1fc7a4a2-1bda-4116-a50a-1b8a72f173aa/crucible/regions/4356bdaf-4f57-413e-b5fb-080585002d02", "04f0b6c6-5b1f-41ce-8991-67ed112946a3"]
282026-03-24T15:38:14.464ZINFOcrucible-agent (dropshot): request completed latency_us = 20665 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:50735 req_id = 392c98b8-333b-45c1-b5c2-614972d32584 response_code = 200 uri = /crucible/0/regions/4356bdaf-4f57-413e-b5fb-080585002d02/snapshots/04f0b6c6-5b1f-41ce-8991-67ed112946a3
292026-03-24T15:38:14.467ZINFOcrucible-agent (datafile): path is "/data/regions/4356bdaf-4f57-413e-b5fb-080585002d02"
302026-03-24T15:38:14.473ZINFOcrucible-agent (datafile): dataset is oxp_1fc7a4a2-1bda-4116-a50a-1b8a72f173aa/crucible/regions/4356bdaf-4f57-413e-b5fb-080585002d02
312026-03-24T15:38:14.479ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_1fc7a4a2-1bda-4116-a50a-1b8a72f173aa/crucible/regions/4356bdaf-4f57-413e-b5fb-080585002d02@04f0b6c6-5b1f-41ce-8991-67ed112946a3
322026-03-24T15:38:14.479ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_1fc7a4a2-1bda-4116-a50a-1b8a72f173aa/crucible/regions/4356bdaf-4f57-413e-b5fb-080585002d02", "04f0b6c6-5b1f-41ce-8991-67ed112946a3"]
332026-03-24T15:38:14.486ZINFOcrucible-agent (datafile): requesting running snapshot 4356bdaf-4f57-413e-b5fb-080585002d02-04f0b6c6-5b1f-41ce-8991-67ed112946a3 state: Requested
342026-03-24T15:38:14.486ZINFOcrucible-agent (dropshot): request completed latency_us = 18786 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:50735 req_id = f7c4a636-c7d1-4b6c-85ce-bed73da43b0c response_code = 200 uri = /crucible/0/regions/4356bdaf-4f57-413e-b5fb-080585002d02/snapshots/04f0b6c6-5b1f-41ce-8991-67ed112946a3/run
352026-03-24T15:38:14.486ZINFOcrucible-agent (worker): applying SMF actions for region 4356bdaf-4f57-413e-b5fb-080585002d02 running snapshot 04f0b6c6-5b1f-41ce-8991-67ed112946a3 (state Requested)...
362026-03-24T15:38:14.490ZINFOcrucible-agent (worker): creating missing snapshot instance snapshot-4356bdaf-4f57-413e-b5fb-080585002d02-04f0b6c6-5b1f-41ce-8991-67ed112946a3
372026-03-24T15:38:14.504ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:snapshot-4356bdaf-4f57-413e-b5fb-080585002d02-04f0b6c6-5b1f-41ce-8991-67ed112946a3
382026-03-24T15:38:14.513ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:snapshot-4356bdaf-4f57-413e-b5fb-080585002d02-04f0b6c6-5b1f-41ce-8991-67ed112946a3
392026-03-24T15:38:14.513ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/4356bdaf-4f57-413e-b5fb-080585002d02/.zfs/snapshot/04f0b6c6-5b1f-41ce-8991-67ed112946a3
402026-03-24T15:38:14.513ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19002
412026-03-24T15:38:14.513ZINFOcrucible-agent (worker): ensure mode SCF_TYPE_ASTRING ro
422026-03-24T15:38:14.513ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::12
432026-03-24T15:38:14.514ZINFOcrucible-agent (worker): commit
442026-03-24T15:38:14.518ZINFOcrucible-agent (worker): ok!
452026-03-24T15:38:14.522ZINFOcrucible-agent (worker): SMF ok!
462026-03-24T15:38:14.523ZINFOcrucible-agent (datafile): region 4356bdaf-4f57-413e-b5fb-080585002d02 running snapshot 04f0b6c6-5b1f-41ce-8991-67ed112946a3 state: Requested -> Created
472026-03-24T15:38:14.639ZINFOcrucible-agent (datafile): path is "/data/regions/4356bdaf-4f57-413e-b5fb-080585002d02"
482026-03-24T15:38:14.649ZINFOcrucible-agent (datafile): dataset is oxp_1fc7a4a2-1bda-4116-a50a-1b8a72f173aa/crucible/regions/4356bdaf-4f57-413e-b5fb-080585002d02
492026-03-24T15:38:14.658ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_1fc7a4a2-1bda-4116-a50a-1b8a72f173aa/crucible/regions/4356bdaf-4f57-413e-b5fb-080585002d02@04f0b6c6-5b1f-41ce-8991-67ed112946a3
502026-03-24T15:38:14.658ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_1fc7a4a2-1bda-4116-a50a-1b8a72f173aa/crucible/regions/4356bdaf-4f57-413e-b5fb-080585002d02", "04f0b6c6-5b1f-41ce-8991-67ed112946a3"]
512026-03-24T15:38:14.668ZINFOcrucible-agent (dropshot): request completed latency_us = 29166 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:50735 req_id = 4fc82c38-ca21-4c83-88cc-728254161bbb response_code = 200 uri = /crucible/0/regions/4356bdaf-4f57-413e-b5fb-080585002d02/snapshots/04f0b6c6-5b1f-41ce-8991-67ed112946a3/run