Max level shown:
1 [ May 8 03:00:06 Enabled. ]
2 [ May 8 03:00:06 Rereading configuration. ]
3 [ May 8 03:00:07 Rereading configuration. ]
4 [ May 8 03:00:10 Executing start method ("/opt/oxide/crucible/bin/crucible-agent run -D /opt/oxide/crucible/bin/crucible-downstairs --dataset oxp_8cd0f983-f129-4a01-ab5a-495470988dc8/crucible -l [fd00:1122:3344:101::1a]:32345 -P 19000 -p downstairs -s snapshot"). ]
5 note: configured to log to "/dev/stdout"
62026-05-08T03:00:10.380ZINFOcrucible-agent: Crucible Version: Crucible Version: 0.0.1 Commit SHA: 7103cd3a3d7b0112d2949dd135db06fef0c156bb Commit timestamp: 2025-11-10T22:24:53.000000000Z branch: main rustc: 1.90.0 stable x86_64-unknown-illumos Cargo: x86_64-unknown-illumos Debug: false Opt level: 3
72026-05-08T03:00:10.381ZINFOcrucible-agent: dataset: "oxp_8cd0f983-f129-4a01-ab5a-495470988dc8/crucible"
82026-05-08T03:00:10.381ZINFOcrucible-agent: listen IP: [fd00:1122:3344:101::1a]:32345
92026-05-08T03:00:10.381ZINFOcrucible-agent: SMF instance name downstairs_prefix: "downstairs"
102026-05-08T03:00:10.402ZINFOcrucible-agent (datafile): Using conf_path:"/data/crucible.json"
112026-05-08T03:00:10.470ZINFOcrucible-agent (dropshot): listening local_addr = [fd00:1122:3344:101::1a]:32345
122026-05-08T03:01:17.749ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::1a]:32345 remote_addr = [fd00:1122:3344:101::b]:59162
132026-05-08T03:01:17.793ZINFOcrucible-agent (datafile): region 537b346e-285f-49e3-925d-3c21b1e0ba47 state: Requested
142026-05-08T03:01:17.794ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
152026-05-08T03:01:17.794ZINFOcrucible-agent (dropshot): request completed latency_us = 18949 local_addr = [fd00:1122:3344:101::1a]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:59162 req_id = 145c2232-de37-4089-997e-fb93be86b5e1 response_code = 200 uri = /crucible/0/regions
162026-05-08T03:01:17.803ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_8cd0f983-f129-4a01-ab5a-495470988dc8/crucible/regions/537b346e-285f-49e3-925d-3c21b1e0ba47
172026-05-08T03:01:17.803ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_8cd0f983-f129-4a01-ab5a-495470988dc8/crucible/regions/537b346e-285f-49e3-925d-3c21b1e0ba47
182026-05-08T03:01:17.842ZINFOcrucible-agent (worker): creating region Region { id: RegionId("537b346e-285f-49e3-925d-3c21b1e0ba47"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/537b346e-285f-49e3-925d-3c21b1e0ba47" region = 537b346e-285f-49e3-925d-3c21b1e0ba47
192026-05-08T03:01:17.844ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "537b346e-285f-49e3-925d-3c21b1e0ba47" "--data" "/data/regions/537b346e-285f-49e3-925d-3c21b1e0ba47" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 537b346e-285f-49e3-925d-3c21b1e0ba47
202026-05-08T03:01:18.129ZINFOcrucible-agent (dropshot): request completed latency_us = 58 local_addr = [fd00:1122:3344:101::1a]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:59162 req_id = a0bb87bd-a3a8-4bf2-a61e-c651292c0e38 response_code = 200 uri = /crucible/0/regions
212026-05-08T03:01:18.156ZINFOcrucible-agent (worker): region files created ok region = 537b346e-285f-49e3-925d-3c21b1e0ba47
222026-05-08T03:01:18.156ZINFOcrucible-agent (datafile): region 537b346e-285f-49e3-925d-3c21b1e0ba47 state: Requested -> Created
232026-05-08T03:01:18.156ZINFOcrucible-agent (worker): applying SMF actions post create...
242026-05-08T03:01:18.157ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-537b346e-285f-49e3-925d-3c21b1e0ba47
252026-05-08T03:01:18.162ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-537b346e-285f-49e3-925d-3c21b1e0ba47
262026-05-08T03:01:18.170ZINFOcrucible-agent (worker): creating config property group
272026-05-08T03:01:18.171ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-537b346e-285f-49e3-925d-3c21b1e0ba47
282026-05-08T03:01:18.172ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/537b346e-285f-49e3-925d-3c21b1e0ba47
292026-05-08T03:01:18.172ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
302026-05-08T03:01:18.172ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::1a
312026-05-08T03:01:18.172ZINFOcrucible-agent (worker): commit
322026-05-08T03:01:18.174ZINFOcrucible-agent (worker): ok!
332026-05-08T03:01:18.177ZINFOcrucible-agent (worker): SMF ok!
342026-05-08T03:01:18.659ZINFOcrucible-agent (dropshot): request completed latency_us = 70 local_addr = [fd00:1122:3344:101::1a]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:59162 req_id = 05a6c7e9-9845-4552-b44b-615ac41f6fdc response_code = 200 uri = /crucible/0/regions
352026-05-08T03:12:17.526ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::1a]:32345 remote_addr = [fd00:1122:3344:101::b]:64611
362026-05-08T03:12:17.526ZINFOcrucible-agent (dropshot): request completed latency_us = 53 local_addr = [fd00:1122:3344:101::1a]:32345 method = GET remote_addr = [fd00:1122:3344:101::b]:64611 req_id = d0a8f6f2-a72b-4621-b76a-28d7a43268a6 response_code = 200 uri = /crucible/0/regions/537b346e-285f-49e3-925d-3c21b1e0ba47
372026-05-08T03:12:17.529ZINFOcrucible-agent (datafile): path is "/data/regions/537b346e-285f-49e3-925d-3c21b1e0ba47"
382026-05-08T03:12:17.535ZINFOcrucible-agent (datafile): dataset is oxp_8cd0f983-f129-4a01-ab5a-495470988dc8/crucible/regions/537b346e-285f-49e3-925d-3c21b1e0ba47
392026-05-08T03:12:17.540ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_8cd0f983-f129-4a01-ab5a-495470988dc8/crucible/regions/537b346e-285f-49e3-925d-3c21b1e0ba47@defa2aee-9ac4-419c-86ae-2ad870019525
402026-05-08T03:12:17.540ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_8cd0f983-f129-4a01-ab5a-495470988dc8/crucible/regions/537b346e-285f-49e3-925d-3c21b1e0ba47", "defa2aee-9ac4-419c-86ae-2ad870019525"]
412026-05-08T03:12:17.551ZINFOcrucible-agent (dropshot): request completed latency_us = 22359 local_addr = [fd00:1122:3344:101::1a]:32345 method = GET remote_addr = [fd00:1122:3344:101::b]:64611 req_id = ad132a42-e1b7-46f4-a034-709274aa1d7a response_code = 200 uri = /crucible/0/regions/537b346e-285f-49e3-925d-3c21b1e0ba47/snapshots/defa2aee-9ac4-419c-86ae-2ad870019525
422026-05-08T03:12:17.556ZINFOcrucible-agent (datafile): path is "/data/regions/537b346e-285f-49e3-925d-3c21b1e0ba47"
432026-05-08T03:12:17.561ZINFOcrucible-agent (datafile): dataset is oxp_8cd0f983-f129-4a01-ab5a-495470988dc8/crucible/regions/537b346e-285f-49e3-925d-3c21b1e0ba47
442026-05-08T03:12:17.566ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_8cd0f983-f129-4a01-ab5a-495470988dc8/crucible/regions/537b346e-285f-49e3-925d-3c21b1e0ba47@defa2aee-9ac4-419c-86ae-2ad870019525
452026-05-08T03:12:17.566ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_8cd0f983-f129-4a01-ab5a-495470988dc8/crucible/regions/537b346e-285f-49e3-925d-3c21b1e0ba47", "defa2aee-9ac4-419c-86ae-2ad870019525"]
462026-05-08T03:12:17.573ZINFOcrucible-agent (datafile): requesting running snapshot 537b346e-285f-49e3-925d-3c21b1e0ba47-defa2aee-9ac4-419c-86ae-2ad870019525 state: Requested
472026-05-08T03:12:17.573ZINFOcrucible-agent (worker): applying SMF actions for region 537b346e-285f-49e3-925d-3c21b1e0ba47 running snapshot defa2aee-9ac4-419c-86ae-2ad870019525 (state Requested)...
482026-05-08T03:12:17.573ZINFOcrucible-agent (dropshot): request completed latency_us = 17762 local_addr = [fd00:1122:3344:101::1a]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:64611 req_id = 73ee47b9-3b26-47db-8db4-8e4726861ae4 response_code = 200 uri = /crucible/0/regions/537b346e-285f-49e3-925d-3c21b1e0ba47/snapshots/defa2aee-9ac4-419c-86ae-2ad870019525/run
492026-05-08T03:12:17.588ZINFOcrucible-agent (worker): creating missing snapshot instance snapshot-537b346e-285f-49e3-925d-3c21b1e0ba47-defa2aee-9ac4-419c-86ae-2ad870019525
502026-05-08T03:12:17.591ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:snapshot-537b346e-285f-49e3-925d-3c21b1e0ba47-defa2aee-9ac4-419c-86ae-2ad870019525
512026-05-08T03:12:17.598ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:snapshot-537b346e-285f-49e3-925d-3c21b1e0ba47-defa2aee-9ac4-419c-86ae-2ad870019525
522026-05-08T03:12:17.599ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/537b346e-285f-49e3-925d-3c21b1e0ba47/.zfs/snapshot/defa2aee-9ac4-419c-86ae-2ad870019525
532026-05-08T03:12:17.599ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
542026-05-08T03:12:17.599ZINFOcrucible-agent (worker): ensure mode SCF_TYPE_ASTRING ro
552026-05-08T03:12:17.599ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::1a
562026-05-08T03:12:17.599ZINFOcrucible-agent (worker): commit
572026-05-08T03:12:17.601ZINFOcrucible-agent (worker): ok!
582026-05-08T03:12:17.604ZINFOcrucible-agent (worker): SMF ok!
592026-05-08T03:12:17.604ZINFOcrucible-agent (datafile): region 537b346e-285f-49e3-925d-3c21b1e0ba47 running snapshot defa2aee-9ac4-419c-86ae-2ad870019525 state: Requested -> Created
602026-05-08T03:12:17.706ZINFOcrucible-agent (datafile): path is "/data/regions/537b346e-285f-49e3-925d-3c21b1e0ba47"
612026-05-08T03:12:17.712ZINFOcrucible-agent (datafile): dataset is oxp_8cd0f983-f129-4a01-ab5a-495470988dc8/crucible/regions/537b346e-285f-49e3-925d-3c21b1e0ba47
622026-05-08T03:12:17.718ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_8cd0f983-f129-4a01-ab5a-495470988dc8/crucible/regions/537b346e-285f-49e3-925d-3c21b1e0ba47@defa2aee-9ac4-419c-86ae-2ad870019525
632026-05-08T03:12:17.718ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_8cd0f983-f129-4a01-ab5a-495470988dc8/crucible/regions/537b346e-285f-49e3-925d-3c21b1e0ba47", "defa2aee-9ac4-419c-86ae-2ad870019525"]
642026-05-08T03:12:17.724ZINFOcrucible-agent (dropshot): request completed latency_us = 17691 local_addr = [fd00:1122:3344:101::1a]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:64611 req_id = c0cf9272-81b2-41fd-89d9-dc52480c2c31 response_code = 200 uri = /crucible/0/regions/537b346e-285f-49e3-925d-3c21b1e0ba47/snapshots/defa2aee-9ac4-419c-86ae-2ad870019525/run
652026-05-08T03:12:22.367ZINFOcrucible-agent (datafile): region 3fa6fff7-6117-4d3e-a09a-ebf07bf6ceda state: Requested
662026-05-08T03:12:22.368ZINFOcrucible-agent (dropshot): request completed latency_us = 218 local_addr = [fd00:1122:3344:101::1a]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:64611 req_id = b9f6ae58-3e86-4ef5-846a-51e808a2e222 response_code = 200 uri = /crucible/0/regions
672026-05-08T03:12:22.368ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
682026-05-08T03:12:22.375ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_8cd0f983-f129-4a01-ab5a-495470988dc8/crucible/regions/3fa6fff7-6117-4d3e-a09a-ebf07bf6ceda
692026-05-08T03:12:22.375ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_8cd0f983-f129-4a01-ab5a-495470988dc8/crucible/regions/3fa6fff7-6117-4d3e-a09a-ebf07bf6ceda
702026-05-08T03:12:22.415ZINFOcrucible-agent (worker): creating region Region { id: RegionId("3fa6fff7-6117-4d3e-a09a-ebf07bf6ceda"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19002, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/3fa6fff7-6117-4d3e-a09a-ebf07bf6ceda" region = 3fa6fff7-6117-4d3e-a09a-ebf07bf6ceda
712026-05-08T03:12:22.415ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "3fa6fff7-6117-4d3e-a09a-ebf07bf6ceda" "--data" "/data/regions/3fa6fff7-6117-4d3e-a09a-ebf07bf6ceda" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 3fa6fff7-6117-4d3e-a09a-ebf07bf6ceda
722026-05-08T03:12:22.560ZINFOcrucible-agent (worker): region files created ok region = 3fa6fff7-6117-4d3e-a09a-ebf07bf6ceda
732026-05-08T03:12:22.560ZINFOcrucible-agent (datafile): region 3fa6fff7-6117-4d3e-a09a-ebf07bf6ceda state: Requested -> Created
742026-05-08T03:12:22.560ZINFOcrucible-agent (worker): applying SMF actions post create...
752026-05-08T03:12:22.560ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-3fa6fff7-6117-4d3e-a09a-ebf07bf6ceda
762026-05-08T03:12:22.570ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::1a]:32345 remote_addr = [fd00:1122:3344:101::b]:56415
772026-05-08T03:12:22.570ZINFOcrucible-agent (dropshot): request completed latency_us = 63 local_addr = [fd00:1122:3344:101::1a]:32345 method = GET remote_addr = [fd00:1122:3344:101::b]:56415 req_id = 89eac038-4388-4e8f-b480-854d79d3e47b response_code = 200 uri = /crucible/0/regions/3fa6fff7-6117-4d3e-a09a-ebf07bf6ceda
782026-05-08T03:12:22.580ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-3fa6fff7-6117-4d3e-a09a-ebf07bf6ceda
792026-05-08T03:12:22.586ZINFOcrucible-agent (worker): creating config property group
802026-05-08T03:12:22.588ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-3fa6fff7-6117-4d3e-a09a-ebf07bf6ceda
812026-05-08T03:12:22.588ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/3fa6fff7-6117-4d3e-a09a-ebf07bf6ceda
822026-05-08T03:12:22.588ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19002
832026-05-08T03:12:22.588ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::1a
842026-05-08T03:12:22.588ZINFOcrucible-agent (worker): commit
852026-05-08T03:12:22.591ZINFOcrucible-agent (worker): ok!
862026-05-08T03:12:22.600ZINFOcrucible-agent (worker): SMF ok!
872026-05-08T03:12:22.660ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::1a]:32345 remote_addr = [fd00:1122:3344:101::a]:45431
882026-05-08T03:12:22.660ZINFOcrucible-agent (dropshot): request completed latency_us = 65 local_addr = [fd00:1122:3344:101::1a]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:45431 req_id = 90f754e5-a4c5-43e6-abc4-c9d7a6ffd04f response_code = 200 uri = /crucible/0/regions/3fa6fff7-6117-4d3e-a09a-ebf07bf6ceda
892026-05-08T03:12:22.698ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::1a]:32345 remote_addr = [fd00:1122:3344:101::c]:57368
902026-05-08T03:12:22.701ZINFOcrucible-agent (dropshot): request completed latency_us = 67 local_addr = [fd00:1122:3344:101::1a]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:57368 req_id = cd5ab854-d802-4da2-8c0c-d11973bdef44 response_code = 200 uri = /crucible/0/regions/3fa6fff7-6117-4d3e-a09a-ebf07bf6ceda
912026-05-08T03:12:22.721ZINFOcrucible-agent (dropshot): request completed latency_us = 96 local_addr = [fd00:1122:3344:101::1a]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:64611 req_id = 97e20787-cb27-4cd0-a04f-1adafbacd986 response_code = 200 uri = /crucible/0/regions
922026-05-08T03:12:47.708ZINFOcrucible-agent (dropshot): request completed latency_us = 52 local_addr = [fd00:1122:3344:101::1a]:32345 method = GET remote_addr = [fd00:1122:3344:101::b]:64611 req_id = 5fe1af50-1722-45b6-a378-5ca4916b9fef response_code = 200 uri = /crucible/0/regions/3fa6fff7-6117-4d3e-a09a-ebf07bf6ceda
932026-05-08T03:12:47.712ZINFOcrucible-agent (datafile): path is "/data/regions/3fa6fff7-6117-4d3e-a09a-ebf07bf6ceda"
942026-05-08T03:12:47.721ZINFOcrucible-agent (datafile): dataset is oxp_8cd0f983-f129-4a01-ab5a-495470988dc8/crucible/regions/3fa6fff7-6117-4d3e-a09a-ebf07bf6ceda
952026-05-08T03:12:47.729ZINFOcrucible-agent (datafile): region 3fa6fff7-6117-4d3e-a09a-ebf07bf6ceda state: Created -> Tombstoned
962026-05-08T03:12:47.729ZINFOcrucible-agent (worker): applying SMF actions before removal...
972026-05-08T03:12:47.729ZINFOcrucible-agent (dropshot): request completed latency_us = 17498 local_addr = [fd00:1122:3344:101::1a]:32345 method = DELETE remote_addr = [fd00:1122:3344:101::b]:64611 req_id = 558bdb33-7aa6-44b0-b646-ec5b8f589d8f response_code = 204 uri = /crucible/0/regions/3fa6fff7-6117-4d3e-a09a-ebf07bf6ceda
982026-05-08T03:12:47.730ZINFOcrucible-agent (worker): disabling downstairs instance: downstairs-3fa6fff7-6117-4d3e-a09a-ebf07bf6ceda (instance states: (Some(Online), None))
992026-05-08T03:12:47.734ZINFOcrucible-agent (dropshot): request completed latency_us = 70 local_addr = [fd00:1122:3344:101::1a]:32345 method = GET remote_addr = [fd00:1122:3344:101::b]:64611 req_id = 740a1a53-20a0-4428-b373-01996d1b91d4 response_code = 200 uri = /crucible/0/regions/3fa6fff7-6117-4d3e-a09a-ebf07bf6ceda
1002026-05-08T03:12:47.735ZINFOcrucible-agent (worker): SMF ok!
1012026-05-08T03:12:47.744ZINFOcrucible-agent (worker): deleting zfs dataset "oxp_8cd0f983-f129-4a01-ab5a-495470988dc8/crucible/regions/3fa6fff7-6117-4d3e-a09a-ebf07bf6ceda" region = 3fa6fff7-6117-4d3e-a09a-ebf07bf6ceda
1022026-05-08T03:12:47.790ZINFOcrucible-agent (datafile): region 3fa6fff7-6117-4d3e-a09a-ebf07bf6ceda state: Tombstoned -> Destroyed
1032026-05-08T03:12:47.823ZINFOcrucible-agent (dropshot): request completed latency_us = 54 local_addr = [fd00:1122:3344:101::1a]:32345 method = GET remote_addr = [fd00:1122:3344:101::b]:64611 req_id = 64e14478-61db-4ec3-a3b3-8baa7ef2cf75 response_code = 200 uri = /crucible/0/regions/3fa6fff7-6117-4d3e-a09a-ebf07bf6ceda