Max level shown:
1 [ Mar 12 16:15:54 Enabled. ]
2 [ Mar 12 16:15:54 Rereading configuration. ]
3 [ Mar 12 16:15:56 Rereading configuration. ]
4 [ Mar 12 16:16:01 Executing start method ("/opt/oxide/crucible/bin/crucible-agent run -D /opt/oxide/crucible/bin/crucible-downstairs --dataset oxp_e06bdc7e-b17a-4859-b4ab-79eac4284485/crucible -l [fd00:1122:3344:101::17]:32345 -P 19000 -p downstairs -s snapshot"). ]
5 note: configured to log to "/dev/stdout"
62026-03-12T16:16:01.825ZINFOcrucible-agent: Crucible Version: Crucible Version: 0.0.1 Commit SHA: 7103cd3a3d7b0112d2949dd135db06fef0c156bb Commit timestamp: 2025-11-10T22:24:53.000000000Z branch: main rustc: 1.90.0 stable x86_64-unknown-illumos Cargo: x86_64-unknown-illumos Debug: false Opt level: 3
72026-03-12T16:16:01.825ZINFOcrucible-agent: dataset: "oxp_e06bdc7e-b17a-4859-b4ab-79eac4284485/crucible"
82026-03-12T16:16:01.825ZINFOcrucible-agent: listen IP: [fd00:1122:3344:101::17]:32345
92026-03-12T16:16:01.825ZINFOcrucible-agent: SMF instance name downstairs_prefix: "downstairs"
102026-03-12T16:16:01.839ZINFOcrucible-agent (datafile): Using conf_path:"/data/crucible.json"
112026-03-12T16:16:01.942ZINFOcrucible-agent (dropshot): listening local_addr = [fd00:1122:3344:101::17]:32345
122026-03-12T16:16:49.400ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::17]:32345 remote_addr = [fd00:1122:3344:101::b]:37493
132026-03-12T16:16:49.427ZINFOcrucible-agent (datafile): region f7ca51e9-6ebf-4a7d-bf85-00c79c4a05af state: Requested
142026-03-12T16:16:49.430ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
152026-03-12T16:16:49.430ZINFOcrucible-agent (dropshot): request completed latency_us = 14539 local_addr = [fd00:1122:3344:101::17]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:37493 req_id = 93e65600-06b1-432e-b8a3-5519df001826 response_code = 200 uri = /crucible/0/regions
162026-03-12T16:16:49.439ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_e06bdc7e-b17a-4859-b4ab-79eac4284485/crucible/regions/f7ca51e9-6ebf-4a7d-bf85-00c79c4a05af
172026-03-12T16:16:49.439ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_e06bdc7e-b17a-4859-b4ab-79eac4284485/crucible/regions/f7ca51e9-6ebf-4a7d-bf85-00c79c4a05af
182026-03-12T16:16:49.506ZINFOcrucible-agent (worker): creating region Region { id: RegionId("f7ca51e9-6ebf-4a7d-bf85-00c79c4a05af"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/f7ca51e9-6ebf-4a7d-bf85-00c79c4a05af" region = f7ca51e9-6ebf-4a7d-bf85-00c79c4a05af
192026-03-12T16:16:49.508ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "f7ca51e9-6ebf-4a7d-bf85-00c79c4a05af" "--data" "/data/regions/f7ca51e9-6ebf-4a7d-bf85-00c79c4a05af" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = f7ca51e9-6ebf-4a7d-bf85-00c79c4a05af
202026-03-12T16:16:49.628ZINFOcrucible-agent (dropshot): request completed latency_us = 63 local_addr = [fd00:1122:3344:101::17]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:37493 req_id = f68bc843-2018-4a92-a82f-494c370dfd27 response_code = 200 uri = /crucible/0/regions
212026-03-12T16:16:49.876ZINFOcrucible-agent (worker): region files created ok region = f7ca51e9-6ebf-4a7d-bf85-00c79c4a05af
222026-03-12T16:16:49.876ZINFOcrucible-agent (datafile): region f7ca51e9-6ebf-4a7d-bf85-00c79c4a05af state: Requested -> Created
232026-03-12T16:16:49.876ZINFOcrucible-agent (worker): applying SMF actions post create...
242026-03-12T16:16:49.877ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-f7ca51e9-6ebf-4a7d-bf85-00c79c4a05af
252026-03-12T16:16:49.882ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-f7ca51e9-6ebf-4a7d-bf85-00c79c4a05af
262026-03-12T16:16:49.900ZINFOcrucible-agent (worker): creating config property group
272026-03-12T16:16:49.903ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-f7ca51e9-6ebf-4a7d-bf85-00c79c4a05af
282026-03-12T16:16:49.904ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/f7ca51e9-6ebf-4a7d-bf85-00c79c4a05af
292026-03-12T16:16:49.904ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
302026-03-12T16:16:49.904ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::17
312026-03-12T16:16:49.904ZINFOcrucible-agent (worker): commit
322026-03-12T16:16:49.909ZINFOcrucible-agent (worker): ok!
332026-03-12T16:16:49.914ZINFOcrucible-agent (worker): SMF ok!
342026-03-12T16:16:50.180ZINFOcrucible-agent (dropshot): request completed latency_us = 61 local_addr = [fd00:1122:3344:101::17]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:37493 req_id = 68a00e68-a61e-4959-b528-0a3b199fbecf response_code = 200 uri = /crucible/0/regions
352026-03-12T16:27:55.326ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::17]:32345 remote_addr = [fd00:1122:3344:101::b]:60914
362026-03-12T16:27:55.327ZINFOcrucible-agent (dropshot): request completed latency_us = 82 local_addr = [fd00:1122:3344:101::17]:32345 method = GET remote_addr = [fd00:1122:3344:101::b]:60914 req_id = c77db6c4-4c81-4e32-b165-58650be747a3 response_code = 200 uri = /crucible/0/regions/f7ca51e9-6ebf-4a7d-bf85-00c79c4a05af
372026-03-12T16:27:55.332ZINFOcrucible-agent (datafile): path is "/data/regions/f7ca51e9-6ebf-4a7d-bf85-00c79c4a05af"
382026-03-12T16:27:55.343ZINFOcrucible-agent (datafile): dataset is oxp_e06bdc7e-b17a-4859-b4ab-79eac4284485/crucible/regions/f7ca51e9-6ebf-4a7d-bf85-00c79c4a05af
392026-03-12T16:27:55.352ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_e06bdc7e-b17a-4859-b4ab-79eac4284485/crucible/regions/f7ca51e9-6ebf-4a7d-bf85-00c79c4a05af@93db9137-4507-4fbd-9812-efd7964088c5
402026-03-12T16:27:55.352ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_e06bdc7e-b17a-4859-b4ab-79eac4284485/crucible/regions/f7ca51e9-6ebf-4a7d-bf85-00c79c4a05af", "93db9137-4507-4fbd-9812-efd7964088c5"]
412026-03-12T16:27:55.365ZINFOcrucible-agent (dropshot): request completed latency_us = 33175 local_addr = [fd00:1122:3344:101::17]:32345 method = GET remote_addr = [fd00:1122:3344:101::b]:60914 req_id = c213c6eb-db32-4233-bcec-850545fc660d response_code = 200 uri = /crucible/0/regions/f7ca51e9-6ebf-4a7d-bf85-00c79c4a05af/snapshots/93db9137-4507-4fbd-9812-efd7964088c5
422026-03-12T16:27:55.370ZINFOcrucible-agent (datafile): path is "/data/regions/f7ca51e9-6ebf-4a7d-bf85-00c79c4a05af"
432026-03-12T16:27:55.377ZINFOcrucible-agent (datafile): dataset is oxp_e06bdc7e-b17a-4859-b4ab-79eac4284485/crucible/regions/f7ca51e9-6ebf-4a7d-bf85-00c79c4a05af
442026-03-12T16:27:55.385ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_e06bdc7e-b17a-4859-b4ab-79eac4284485/crucible/regions/f7ca51e9-6ebf-4a7d-bf85-00c79c4a05af@93db9137-4507-4fbd-9812-efd7964088c5
452026-03-12T16:27:55.385ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_e06bdc7e-b17a-4859-b4ab-79eac4284485/crucible/regions/f7ca51e9-6ebf-4a7d-bf85-00c79c4a05af", "93db9137-4507-4fbd-9812-efd7964088c5"]
462026-03-12T16:27:55.396ZINFOcrucible-agent (datafile): requesting running snapshot f7ca51e9-6ebf-4a7d-bf85-00c79c4a05af-93db9137-4507-4fbd-9812-efd7964088c5 state: Requested
472026-03-12T16:27:55.396ZINFOcrucible-agent (dropshot): request completed latency_us = 26813 local_addr = [fd00:1122:3344:101::17]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:60914 req_id = b529fc78-c485-4b44-8834-8d3d4549d9e8 response_code = 200 uri = /crucible/0/regions/f7ca51e9-6ebf-4a7d-bf85-00c79c4a05af/snapshots/93db9137-4507-4fbd-9812-efd7964088c5/run
482026-03-12T16:27:55.396ZINFOcrucible-agent (worker): applying SMF actions for region f7ca51e9-6ebf-4a7d-bf85-00c79c4a05af running snapshot 93db9137-4507-4fbd-9812-efd7964088c5 (state Requested)...
492026-03-12T16:27:55.427ZINFOcrucible-agent (worker): creating missing snapshot instance snapshot-f7ca51e9-6ebf-4a7d-bf85-00c79c4a05af-93db9137-4507-4fbd-9812-efd7964088c5
502026-03-12T16:27:55.436ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:snapshot-f7ca51e9-6ebf-4a7d-bf85-00c79c4a05af-93db9137-4507-4fbd-9812-efd7964088c5
512026-03-12T16:27:55.451ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:snapshot-f7ca51e9-6ebf-4a7d-bf85-00c79c4a05af-93db9137-4507-4fbd-9812-efd7964088c5
522026-03-12T16:27:55.451ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/f7ca51e9-6ebf-4a7d-bf85-00c79c4a05af/.zfs/snapshot/93db9137-4507-4fbd-9812-efd7964088c5
532026-03-12T16:27:55.451ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
542026-03-12T16:27:55.451ZINFOcrucible-agent (worker): ensure mode SCF_TYPE_ASTRING ro
552026-03-12T16:27:55.451ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::17
562026-03-12T16:27:55.452ZINFOcrucible-agent (worker): commit
572026-03-12T16:27:55.456ZINFOcrucible-agent (worker): ok!
582026-03-12T16:27:55.462ZINFOcrucible-agent (worker): SMF ok!
592026-03-12T16:27:55.462ZINFOcrucible-agent (datafile): region f7ca51e9-6ebf-4a7d-bf85-00c79c4a05af running snapshot 93db9137-4507-4fbd-9812-efd7964088c5 state: Requested -> Created
602026-03-12T16:27:55.537ZINFOcrucible-agent (datafile): path is "/data/regions/f7ca51e9-6ebf-4a7d-bf85-00c79c4a05af"
612026-03-12T16:27:55.543ZINFOcrucible-agent (datafile): dataset is oxp_e06bdc7e-b17a-4859-b4ab-79eac4284485/crucible/regions/f7ca51e9-6ebf-4a7d-bf85-00c79c4a05af
622026-03-12T16:27:55.549ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_e06bdc7e-b17a-4859-b4ab-79eac4284485/crucible/regions/f7ca51e9-6ebf-4a7d-bf85-00c79c4a05af@93db9137-4507-4fbd-9812-efd7964088c5
632026-03-12T16:27:55.549ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_e06bdc7e-b17a-4859-b4ab-79eac4284485/crucible/regions/f7ca51e9-6ebf-4a7d-bf85-00c79c4a05af", "93db9137-4507-4fbd-9812-efd7964088c5"]
642026-03-12T16:27:55.555ZINFOcrucible-agent (dropshot): request completed latency_us = 18322 local_addr = [fd00:1122:3344:101::17]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:60914 req_id = 43512596-5aad-4d48-bd81-cad67d38890f response_code = 200 uri = /crucible/0/regions/f7ca51e9-6ebf-4a7d-bf85-00c79c4a05af/snapshots/93db9137-4507-4fbd-9812-efd7964088c5/run
652026-03-12T16:27:59.797ZINFOcrucible-agent (datafile): region 6627b7e6-3382-4897-adcd-f7ae263cbf25 state: Requested
662026-03-12T16:27:59.797ZINFOcrucible-agent (dropshot): request completed latency_us = 215 local_addr = [fd00:1122:3344:101::17]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:60914 req_id = 5e0556f9-ea7c-4ab8-bbbb-f9d6cc5a2cef response_code = 200 uri = /crucible/0/regions
672026-03-12T16:27:59.797ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
682026-03-12T16:27:59.804ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_e06bdc7e-b17a-4859-b4ab-79eac4284485/crucible/regions/6627b7e6-3382-4897-adcd-f7ae263cbf25
692026-03-12T16:27:59.804ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_e06bdc7e-b17a-4859-b4ab-79eac4284485/crucible/regions/6627b7e6-3382-4897-adcd-f7ae263cbf25
702026-03-12T16:27:59.853ZINFOcrucible-agent (worker): creating region Region { id: RegionId("6627b7e6-3382-4897-adcd-f7ae263cbf25"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19002, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/6627b7e6-3382-4897-adcd-f7ae263cbf25" region = 6627b7e6-3382-4897-adcd-f7ae263cbf25
712026-03-12T16:27:59.853ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "6627b7e6-3382-4897-adcd-f7ae263cbf25" "--data" "/data/regions/6627b7e6-3382-4897-adcd-f7ae263cbf25" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 6627b7e6-3382-4897-adcd-f7ae263cbf25
722026-03-12T16:27:59.937ZINFOcrucible-agent (dropshot): request completed latency_us = 50 local_addr = [fd00:1122:3344:101::17]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:60914 req_id = ddd5eb76-0573-48d3-85d3-86baba2313ef response_code = 200 uri = /crucible/0/regions
732026-03-12T16:27:59.944ZINFOcrucible-agent (worker): region files created ok region = 6627b7e6-3382-4897-adcd-f7ae263cbf25
742026-03-12T16:27:59.944ZINFOcrucible-agent (datafile): region 6627b7e6-3382-4897-adcd-f7ae263cbf25 state: Requested -> Created
752026-03-12T16:27:59.944ZINFOcrucible-agent (worker): applying SMF actions post create...
762026-03-12T16:27:59.944ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-6627b7e6-3382-4897-adcd-f7ae263cbf25
772026-03-12T16:27:59.958ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-6627b7e6-3382-4897-adcd-f7ae263cbf25
782026-03-12T16:27:59.965ZINFOcrucible-agent (worker): creating config property group
792026-03-12T16:27:59.968ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-6627b7e6-3382-4897-adcd-f7ae263cbf25
802026-03-12T16:27:59.968ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/6627b7e6-3382-4897-adcd-f7ae263cbf25
812026-03-12T16:27:59.969ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19002
822026-03-12T16:27:59.969ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::17
832026-03-12T16:27:59.969ZINFOcrucible-agent (worker): commit
842026-03-12T16:27:59.972ZINFOcrucible-agent (worker): ok!
852026-03-12T16:27:59.982ZINFOcrucible-agent (worker): SMF ok!
862026-03-12T16:28:00.436ZINFOcrucible-agent (dropshot): request completed latency_us = 62 local_addr = [fd00:1122:3344:101::17]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:60914 req_id = 2fbf1281-1ed0-4535-bc0e-74de41e1828b response_code = 200 uri = /crucible/0/regions
872026-03-12T16:28:24.862ZINFOcrucible-agent (dropshot): request completed latency_us = 57 local_addr = [fd00:1122:3344:101::17]:32345 method = GET remote_addr = [fd00:1122:3344:101::b]:60914 req_id = a0550ce8-d0fc-4761-92da-ab405fa64c61 response_code = 200 uri = /crucible/0/regions/6627b7e6-3382-4897-adcd-f7ae263cbf25
882026-03-12T16:28:24.865ZINFOcrucible-agent (datafile): path is "/data/regions/6627b7e6-3382-4897-adcd-f7ae263cbf25"
892026-03-12T16:28:24.877ZINFOcrucible-agent (datafile): dataset is oxp_e06bdc7e-b17a-4859-b4ab-79eac4284485/crucible/regions/6627b7e6-3382-4897-adcd-f7ae263cbf25
902026-03-12T16:28:24.884ZINFOcrucible-agent (datafile): region 6627b7e6-3382-4897-adcd-f7ae263cbf25 state: Created -> Tombstoned
912026-03-12T16:28:24.884ZINFOcrucible-agent (dropshot): request completed latency_us = 19089 local_addr = [fd00:1122:3344:101::17]:32345 method = DELETE remote_addr = [fd00:1122:3344:101::b]:60914 req_id = 469fcf25-c16b-4a90-b9ad-e4fc68a4411a response_code = 204 uri = /crucible/0/regions/6627b7e6-3382-4897-adcd-f7ae263cbf25
922026-03-12T16:28:24.884ZINFOcrucible-agent (worker): applying SMF actions before removal...
932026-03-12T16:28:24.886ZINFOcrucible-agent (worker): disabling downstairs instance: downstairs-6627b7e6-3382-4897-adcd-f7ae263cbf25 (instance states: (Some(Online), None))
942026-03-12T16:28:24.888ZINFOcrucible-agent (dropshot): request completed latency_us = 32 local_addr = [fd00:1122:3344:101::17]:32345 method = GET remote_addr = [fd00:1122:3344:101::b]:60914 req_id = 67ad1053-5d6c-4b06-9345-4ab38f6d7892 response_code = 200 uri = /crucible/0/regions/6627b7e6-3382-4897-adcd-f7ae263cbf25
952026-03-12T16:28:24.901ZINFOcrucible-agent (worker): SMF ok!
962026-03-12T16:28:24.908ZINFOcrucible-agent (worker): deleting zfs dataset "oxp_e06bdc7e-b17a-4859-b4ab-79eac4284485/crucible/regions/6627b7e6-3382-4897-adcd-f7ae263cbf25" region = 6627b7e6-3382-4897-adcd-f7ae263cbf25
972026-03-12T16:28:24.950ZINFOcrucible-agent (dropshot): request completed latency_us = 36 local_addr = [fd00:1122:3344:101::17]:32345 method = GET remote_addr = [fd00:1122:3344:101::b]:60914 req_id = 729f3eb6-3aa1-4065-bdbd-aac8209f8c67 response_code = 200 uri = /crucible/0/regions/6627b7e6-3382-4897-adcd-f7ae263cbf25
982026-03-12T16:28:24.955ZINFOcrucible-agent (datafile): region 6627b7e6-3382-4897-adcd-f7ae263cbf25 state: Tombstoned -> Destroyed
992026-03-12T16:28:25.133ZINFOcrucible-agent (dropshot): request completed latency_us = 56 local_addr = [fd00:1122:3344:101::17]:32345 method = GET remote_addr = [fd00:1122:3344:101::b]:60914 req_id = 13a06610-708b-4926-8b6a-f2bcb6a740e6 response_code = 200 uri = /crucible/0/regions/6627b7e6-3382-4897-adcd-f7ae263cbf25