Max level shown:
1 [ May 19 00:00:04 Executing start method ("/opt/oxide/crucible/bin/crucible-agent run -D /opt/oxide/crucible/bin/crucible-downstairs --dataset oxp_cea2a7a2-9e32-4b98-9aa9-9a98cf4a8fa3/crucible -l [fd00:1122:3344:101::13]:32345 -P 19000 -p downstairs -s snapshot"). ]
2 note: configured to log to "/dev/stdout"
32026-05-19T00:00:04.559ZINFOcrucible-agent: Crucible Version: Crucible Version: 0.0.1 Commit SHA: 7103cd3a3d7b0112d2949dd135db06fef0c156bb Commit timestamp: 2025-11-10T22:24:53.000000000Z branch: main rustc: 1.90.0 stable x86_64-unknown-illumos Cargo: x86_64-unknown-illumos Debug: false Opt level: 3
42026-05-19T00:00:04.560ZINFOcrucible-agent: dataset: "oxp_cea2a7a2-9e32-4b98-9aa9-9a98cf4a8fa3/crucible"
52026-05-19T00:00:04.560ZINFOcrucible-agent: listen IP: [fd00:1122:3344:101::13]:32345
62026-05-19T00:00:04.560ZINFOcrucible-agent: SMF instance name downstairs_prefix: "downstairs"
72026-05-19T00:00:04.574ZINFOcrucible-agent (datafile): Using conf_path:"/data/crucible.json"
82026-05-19T00:00:04.636ZINFOcrucible-agent (dropshot): listening local_addr = [fd00:1122:3344:101::13]:32345
92026-05-19T00:00:51.272ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::13]:32345 remote_addr = [fd00:1122:3344:101::c]:54195
102026-05-19T00:00:51.301ZINFOcrucible-agent (datafile): region d40d9bc7-40ad-4ff3-9313-06817baf37a9 state: Requested
112026-05-19T00:00:51.304ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
122026-05-19T00:00:51.304ZINFOcrucible-agent (dropshot): request completed latency_us = 16642 local_addr = [fd00:1122:3344:101::13]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:54195 req_id = 737b536a-a690-4674-966b-950d4e3a81c4 response_code = 200 uri = /crucible/0/regions
132026-05-19T00:00:51.310ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_cea2a7a2-9e32-4b98-9aa9-9a98cf4a8fa3/crucible/regions/d40d9bc7-40ad-4ff3-9313-06817baf37a9
142026-05-19T00:00:51.310ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_cea2a7a2-9e32-4b98-9aa9-9a98cf4a8fa3/crucible/regions/d40d9bc7-40ad-4ff3-9313-06817baf37a9
152026-05-19T00:00:51.362ZINFOcrucible-agent (worker): creating region Region { id: RegionId("d40d9bc7-40ad-4ff3-9313-06817baf37a9"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/d40d9bc7-40ad-4ff3-9313-06817baf37a9" region = d40d9bc7-40ad-4ff3-9313-06817baf37a9
162026-05-19T00:00:51.362ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "d40d9bc7-40ad-4ff3-9313-06817baf37a9" "--data" "/data/regions/d40d9bc7-40ad-4ff3-9313-06817baf37a9" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = d40d9bc7-40ad-4ff3-9313-06817baf37a9
172026-05-19T00:00:51.636ZINFOcrucible-agent (dropshot): request completed latency_us = 58 local_addr = [fd00:1122:3344:101::13]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:54195 req_id = fd78e4b6-f37e-47ce-b97a-52ff30e8d6ff response_code = 200 uri = /crucible/0/regions
182026-05-19T00:00:51.694ZINFOcrucible-agent (worker): region files created ok region = d40d9bc7-40ad-4ff3-9313-06817baf37a9
192026-05-19T00:00:51.694ZINFOcrucible-agent (datafile): region d40d9bc7-40ad-4ff3-9313-06817baf37a9 state: Requested -> Created
202026-05-19T00:00:51.695ZINFOcrucible-agent (worker): applying SMF actions post create...
212026-05-19T00:00:51.695ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-d40d9bc7-40ad-4ff3-9313-06817baf37a9
222026-05-19T00:00:51.702ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-d40d9bc7-40ad-4ff3-9313-06817baf37a9
232026-05-19T00:00:51.716ZINFOcrucible-agent (worker): creating config property group
242026-05-19T00:00:51.719ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-d40d9bc7-40ad-4ff3-9313-06817baf37a9
252026-05-19T00:00:51.720ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/d40d9bc7-40ad-4ff3-9313-06817baf37a9
262026-05-19T00:00:51.720ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
272026-05-19T00:00:51.720ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::13
282026-05-19T00:00:51.720ZINFOcrucible-agent (worker): commit
292026-05-19T00:00:51.724ZINFOcrucible-agent (worker): ok!
302026-05-19T00:00:51.729ZINFOcrucible-agent (worker): SMF ok!
312026-05-19T00:00:52.015ZINFOcrucible-agent (dropshot): request completed latency_us = 58 local_addr = [fd00:1122:3344:101::13]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:54195 req_id = f2e624c0-a963-4e5e-8cb4-b0a4c43d0be5 response_code = 200 uri = /crucible/0/regions
322026-05-19T00:11:32.407ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::13]:32345 remote_addr = [fd00:1122:3344:101::c]:51704
332026-05-19T00:11:32.407ZINFOcrucible-agent (dropshot): request completed latency_us = 54 local_addr = [fd00:1122:3344:101::13]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:51704 req_id = 3968707e-48d5-49ed-bf37-9ccfeb40338c response_code = 200 uri = /crucible/0/regions/d40d9bc7-40ad-4ff3-9313-06817baf37a9
342026-05-19T00:11:32.415ZINFOcrucible-agent (datafile): path is "/data/regions/d40d9bc7-40ad-4ff3-9313-06817baf37a9"
352026-05-19T00:11:32.422ZINFOcrucible-agent (datafile): dataset is oxp_cea2a7a2-9e32-4b98-9aa9-9a98cf4a8fa3/crucible/regions/d40d9bc7-40ad-4ff3-9313-06817baf37a9
362026-05-19T00:11:32.428ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_cea2a7a2-9e32-4b98-9aa9-9a98cf4a8fa3/crucible/regions/d40d9bc7-40ad-4ff3-9313-06817baf37a9@bc1d8e8b-0159-4283-ac58-b90610b76f31
372026-05-19T00:11:32.428ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_cea2a7a2-9e32-4b98-9aa9-9a98cf4a8fa3/crucible/regions/d40d9bc7-40ad-4ff3-9313-06817baf37a9", "bc1d8e8b-0159-4283-ac58-b90610b76f31"]
382026-05-19T00:11:32.443ZINFOcrucible-agent (dropshot): request completed latency_us = 27635 local_addr = [fd00:1122:3344:101::13]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:51704 req_id = 28c47a89-0db8-4d9d-ad8d-f9f0db4e726b response_code = 200 uri = /crucible/0/regions/d40d9bc7-40ad-4ff3-9313-06817baf37a9/snapshots/bc1d8e8b-0159-4283-ac58-b90610b76f31
392026-05-19T00:11:32.449ZINFOcrucible-agent (datafile): path is "/data/regions/d40d9bc7-40ad-4ff3-9313-06817baf37a9"
402026-05-19T00:11:32.455ZINFOcrucible-agent (datafile): dataset is oxp_cea2a7a2-9e32-4b98-9aa9-9a98cf4a8fa3/crucible/regions/d40d9bc7-40ad-4ff3-9313-06817baf37a9
412026-05-19T00:11:32.461ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_cea2a7a2-9e32-4b98-9aa9-9a98cf4a8fa3/crucible/regions/d40d9bc7-40ad-4ff3-9313-06817baf37a9@bc1d8e8b-0159-4283-ac58-b90610b76f31
422026-05-19T00:11:32.461ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_cea2a7a2-9e32-4b98-9aa9-9a98cf4a8fa3/crucible/regions/d40d9bc7-40ad-4ff3-9313-06817baf37a9", "bc1d8e8b-0159-4283-ac58-b90610b76f31"]
432026-05-19T00:11:32.476ZINFOcrucible-agent (datafile): requesting running snapshot d40d9bc7-40ad-4ff3-9313-06817baf37a9-bc1d8e8b-0159-4283-ac58-b90610b76f31 state: Requested
442026-05-19T00:11:32.476ZINFOcrucible-agent (worker): applying SMF actions for region d40d9bc7-40ad-4ff3-9313-06817baf37a9 running snapshot bc1d8e8b-0159-4283-ac58-b90610b76f31 (state Requested)...
452026-05-19T00:11:32.476ZINFOcrucible-agent (dropshot): request completed latency_us = 27384 local_addr = [fd00:1122:3344:101::13]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:51704 req_id = d01d3e1a-3335-4a8e-8b79-5aff866a7ee2 response_code = 200 uri = /crucible/0/regions/d40d9bc7-40ad-4ff3-9313-06817baf37a9/snapshots/bc1d8e8b-0159-4283-ac58-b90610b76f31/run
462026-05-19T00:11:32.497ZINFOcrucible-agent (worker): creating missing snapshot instance snapshot-d40d9bc7-40ad-4ff3-9313-06817baf37a9-bc1d8e8b-0159-4283-ac58-b90610b76f31
472026-05-19T00:11:32.514ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:snapshot-d40d9bc7-40ad-4ff3-9313-06817baf37a9-bc1d8e8b-0159-4283-ac58-b90610b76f31
482026-05-19T00:11:32.544ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:snapshot-d40d9bc7-40ad-4ff3-9313-06817baf37a9-bc1d8e8b-0159-4283-ac58-b90610b76f31
492026-05-19T00:11:32.544ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/d40d9bc7-40ad-4ff3-9313-06817baf37a9/.zfs/snapshot/bc1d8e8b-0159-4283-ac58-b90610b76f31
502026-05-19T00:11:32.544ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
512026-05-19T00:11:32.544ZINFOcrucible-agent (worker): ensure mode SCF_TYPE_ASTRING ro
522026-05-19T00:11:32.544ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::13
532026-05-19T00:11:32.544ZINFOcrucible-agent (worker): commit
542026-05-19T00:11:32.555ZINFOcrucible-agent (worker): ok!
552026-05-19T00:11:32.561ZINFOcrucible-agent (worker): SMF ok!
562026-05-19T00:11:32.561ZINFOcrucible-agent (datafile): region d40d9bc7-40ad-4ff3-9313-06817baf37a9 running snapshot bc1d8e8b-0159-4283-ac58-b90610b76f31 state: Requested -> Created
572026-05-19T00:11:32.846ZINFOcrucible-agent (datafile): path is "/data/regions/d40d9bc7-40ad-4ff3-9313-06817baf37a9"
582026-05-19T00:11:32.853ZINFOcrucible-agent (datafile): dataset is oxp_cea2a7a2-9e32-4b98-9aa9-9a98cf4a8fa3/crucible/regions/d40d9bc7-40ad-4ff3-9313-06817baf37a9
592026-05-19T00:11:32.859ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_cea2a7a2-9e32-4b98-9aa9-9a98cf4a8fa3/crucible/regions/d40d9bc7-40ad-4ff3-9313-06817baf37a9@bc1d8e8b-0159-4283-ac58-b90610b76f31
602026-05-19T00:11:32.859ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_cea2a7a2-9e32-4b98-9aa9-9a98cf4a8fa3/crucible/regions/d40d9bc7-40ad-4ff3-9313-06817baf37a9", "bc1d8e8b-0159-4283-ac58-b90610b76f31"]
612026-05-19T00:11:32.864ZINFOcrucible-agent (dropshot): request completed latency_us = 18186 local_addr = [fd00:1122:3344:101::13]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:51704 req_id = 823581b7-a40f-41c7-b97f-fd790eb7561e response_code = 200 uri = /crucible/0/regions/d40d9bc7-40ad-4ff3-9313-06817baf37a9/snapshots/bc1d8e8b-0159-4283-ac58-b90610b76f31/run
622026-05-19T00:11:39.955ZINFOcrucible-agent (datafile): region 57240288-fad9-4149-bd5e-502de4b15348 state: Requested
632026-05-19T00:11:39.955ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
642026-05-19T00:11:39.955ZINFOcrucible-agent (dropshot): request completed latency_us = 203 local_addr = [fd00:1122:3344:101::13]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:51704 req_id = 4331865c-d409-4e1c-bd0b-bf5f86b28acd response_code = 200 uri = /crucible/0/regions
652026-05-19T00:11:39.961ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_cea2a7a2-9e32-4b98-9aa9-9a98cf4a8fa3/crucible/regions/57240288-fad9-4149-bd5e-502de4b15348
662026-05-19T00:11:39.961ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_cea2a7a2-9e32-4b98-9aa9-9a98cf4a8fa3/crucible/regions/57240288-fad9-4149-bd5e-502de4b15348
672026-05-19T00:11:40.166ZINFOcrucible-agent (worker): creating region Region { id: RegionId("57240288-fad9-4149-bd5e-502de4b15348"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19002, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/57240288-fad9-4149-bd5e-502de4b15348" region = 57240288-fad9-4149-bd5e-502de4b15348
682026-05-19T00:11:40.166ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "57240288-fad9-4149-bd5e-502de4b15348" "--data" "/data/regions/57240288-fad9-4149-bd5e-502de4b15348" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 57240288-fad9-4149-bd5e-502de4b15348
692026-05-19T00:11:40.188ZINFOcrucible-agent (dropshot): request completed latency_us = 51 local_addr = [fd00:1122:3344:101::13]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:51704 req_id = 0da5db64-0ec4-44d0-b657-2a4e6b8e4a2c response_code = 200 uri = /crucible/0/regions
702026-05-19T00:11:40.515ZINFOcrucible-agent (dropshot): request completed latency_us = 56 local_addr = [fd00:1122:3344:101::13]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:51704 req_id = b2255f79-bb64-4d0a-9d83-ea7a5cb9359f response_code = 200 uri = /crucible/0/regions
712026-05-19T00:11:40.547ZINFOcrucible-agent (worker): region files created ok region = 57240288-fad9-4149-bd5e-502de4b15348
722026-05-19T00:11:40.547ZINFOcrucible-agent (datafile): region 57240288-fad9-4149-bd5e-502de4b15348 state: Requested -> Created
732026-05-19T00:11:40.547ZINFOcrucible-agent (worker): applying SMF actions post create...
742026-05-19T00:11:40.547ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-57240288-fad9-4149-bd5e-502de4b15348
752026-05-19T00:11:40.576ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-57240288-fad9-4149-bd5e-502de4b15348
762026-05-19T00:11:40.676ZINFOcrucible-agent (worker): creating config property group
772026-05-19T00:11:40.728ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-57240288-fad9-4149-bd5e-502de4b15348
782026-05-19T00:11:40.728ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/57240288-fad9-4149-bd5e-502de4b15348
792026-05-19T00:11:40.728ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19002
802026-05-19T00:11:40.728ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::13
812026-05-19T00:11:40.728ZINFOcrucible-agent (worker): commit
822026-05-19T00:11:40.784ZINFOcrucible-agent (worker): ok!
832026-05-19T00:11:40.882ZINFOcrucible-agent (worker): SMF ok!
842026-05-19T00:11:41.166ZINFOcrucible-agent (dropshot): request completed latency_us = 61 local_addr = [fd00:1122:3344:101::13]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:51704 req_id = 5150ff9a-c58e-49fd-b5e5-942f35bb7cfc response_code = 200 uri = /crucible/0/regions
852026-05-19T00:12:16.473ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::13]:32345 remote_addr = [fd00:1122:3344:101::c]:38745
862026-05-19T00:12:16.474ZINFOcrucible-agent (dropshot): request completed latency_us = 69 local_addr = [fd00:1122:3344:101::13]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:38745 req_id = 26dee761-81cf-4758-807e-71d394d2802a response_code = 200 uri = /crucible/0/regions/57240288-fad9-4149-bd5e-502de4b15348
872026-05-19T00:12:16.476ZINFOcrucible-agent (datafile): path is "/data/regions/57240288-fad9-4149-bd5e-502de4b15348"
882026-05-19T00:12:16.486ZINFOcrucible-agent (datafile): dataset is oxp_cea2a7a2-9e32-4b98-9aa9-9a98cf4a8fa3/crucible/regions/57240288-fad9-4149-bd5e-502de4b15348
892026-05-19T00:12:16.494ZINFOcrucible-agent (datafile): region 57240288-fad9-4149-bd5e-502de4b15348 state: Created -> Tombstoned
902026-05-19T00:12:16.495ZINFOcrucible-agent (worker): applying SMF actions before removal...
912026-05-19T00:12:16.495ZINFOcrucible-agent (dropshot): request completed latency_us = 18361 local_addr = [fd00:1122:3344:101::13]:32345 method = DELETE remote_addr = [fd00:1122:3344:101::c]:38745 req_id = 105e13d2-110b-444a-af20-9826a823c1c3 response_code = 204 uri = /crucible/0/regions/57240288-fad9-4149-bd5e-502de4b15348
922026-05-19T00:12:16.495ZINFOcrucible-agent (worker): disabling downstairs instance: downstairs-57240288-fad9-4149-bd5e-502de4b15348 (instance states: (Some(Online), None))
932026-05-19T00:12:16.499ZINFOcrucible-agent (dropshot): request completed latency_us = 35 local_addr = [fd00:1122:3344:101::13]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:38745 req_id = 6efedc3d-7e81-46ab-8b66-ec5d1115c25b response_code = 200 uri = /crucible/0/regions/57240288-fad9-4149-bd5e-502de4b15348
942026-05-19T00:12:16.514ZINFOcrucible-agent (worker): SMF ok!
952026-05-19T00:12:16.521ZINFOcrucible-agent (worker): deleting zfs dataset "oxp_cea2a7a2-9e32-4b98-9aa9-9a98cf4a8fa3/crucible/regions/57240288-fad9-4149-bd5e-502de4b15348" region = 57240288-fad9-4149-bd5e-502de4b15348
962026-05-19T00:12:16.597ZINFOcrucible-agent (datafile): region 57240288-fad9-4149-bd5e-502de4b15348 state: Tombstoned -> Destroyed
972026-05-19T00:12:16.649ZINFOcrucible-agent (dropshot): request completed latency_us = 47 local_addr = [fd00:1122:3344:101::13]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:38745 req_id = 633f1622-27cc-4f74-b8f5-4e069ec75f54 response_code = 200 uri = /crucible/0/regions/57240288-fad9-4149-bd5e-502de4b15348