Max level shown:
1 [ Apr 7 02:47:30 Enabled. ]
2 [ Apr 7 02:47:30 Rereading configuration. ]
3 [ Apr 7 02:47:32 Rereading configuration. ]
4 [ Apr 7 02:47:37 Executing start method ("/opt/oxide/crucible/bin/crucible-agent run -D /opt/oxide/crucible/bin/crucible-downstairs --dataset oxp_f45535cb-c78c-4fd3-bcd3-4692b7a79434/crucible -l [fd00:1122:3344:101::17]:32345 -P 19000 -p downstairs -s snapshot"). ]
5 note: configured to log to "/dev/stdout"
62026-04-07T02:47:37.751ZINFOcrucible-agent: Crucible Version: Crucible Version: 0.0.1 Commit SHA: 7103cd3a3d7b0112d2949dd135db06fef0c156bb Commit timestamp: 2025-11-10T22:24:53.000000000Z branch: main rustc: 1.90.0 stable x86_64-unknown-illumos Cargo: x86_64-unknown-illumos Debug: false Opt level: 3
72026-04-07T02:47:37.752ZINFOcrucible-agent: dataset: "oxp_f45535cb-c78c-4fd3-bcd3-4692b7a79434/crucible"
82026-04-07T02:47:37.752ZINFOcrucible-agent: listen IP: [fd00:1122:3344:101::17]:32345
92026-04-07T02:47:37.752ZINFOcrucible-agent: SMF instance name downstairs_prefix: "downstairs"
102026-04-07T02:47:37.763ZINFOcrucible-agent (datafile): Using conf_path:"/data/crucible.json"
112026-04-07T02:47:37.884ZINFOcrucible-agent (dropshot): listening local_addr = [fd00:1122:3344:101::17]:32345
122026-04-07T02:48:16.846ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::17]:32345 remote_addr = [fd00:1122:3344:101::b]:59939
132026-04-07T02:48:16.874ZINFOcrucible-agent (datafile): region 9e74d982-d004-46c2-8d77-0cce2edb2805 state: Requested
142026-04-07T02:48:16.877ZINFOcrucible-agent (worker): Region size:1073741824 reservation:1342177280 quota:3221225472
152026-04-07T02:48:16.877ZINFOcrucible-agent (dropshot): request completed latency_us = 15976 local_addr = [fd00:1122:3344:101::17]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:59939 req_id = f71ded40-dd0b-4706-a651-feb7946a47cb response_code = 200 uri = /crucible/0/regions
162026-04-07T02:48:16.885ZINFOcrucible-agent (worker): zfs set reservation of 1342177280 for oxp_f45535cb-c78c-4fd3-bcd3-4692b7a79434/crucible/regions/9e74d982-d004-46c2-8d77-0cce2edb2805
172026-04-07T02:48:16.885ZINFOcrucible-agent (worker): zfs set quota of 3221225472 for oxp_f45535cb-c78c-4fd3-bcd3-4692b7a79434/crucible/regions/9e74d982-d004-46c2-8d77-0cce2edb2805
182026-04-07T02:48:16.948ZINFOcrucible-agent (worker): creating region Region { id: RegionId("9e74d982-d004-46c2-8d77-0cce2edb2805"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 16, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/9e74d982-d004-46c2-8d77-0cce2edb2805" region = 9e74d982-d004-46c2-8d77-0cce2edb2805
192026-04-07T02:48:16.950ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "9e74d982-d004-46c2-8d77-0cce2edb2805" "--data" "/data/regions/9e74d982-d004-46c2-8d77-0cce2edb2805" "--block-size" "512" "--extent-size" "131072" "--extent-count" "16" "--encrypted" region = 9e74d982-d004-46c2-8d77-0cce2edb2805
202026-04-07T02:48:17.046ZINFOcrucible-agent (dropshot): request completed latency_us = 60 local_addr = [fd00:1122:3344:101::17]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:59939 req_id = c7101065-39f0-43c2-9e46-af0f58b5b839 response_code = 200 uri = /crucible/0/regions
212026-04-07T02:48:17.300ZINFOcrucible-agent (worker): region files created ok region = 9e74d982-d004-46c2-8d77-0cce2edb2805
222026-04-07T02:48:17.300ZINFOcrucible-agent (datafile): region 9e74d982-d004-46c2-8d77-0cce2edb2805 state: Requested -> Created
232026-04-07T02:48:17.300ZINFOcrucible-agent (worker): applying SMF actions post create...
242026-04-07T02:48:17.301ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-9e74d982-d004-46c2-8d77-0cce2edb2805
252026-04-07T02:48:17.310ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-9e74d982-d004-46c2-8d77-0cce2edb2805
262026-04-07T02:48:17.322ZINFOcrucible-agent (worker): creating config property group
272026-04-07T02:48:17.325ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-9e74d982-d004-46c2-8d77-0cce2edb2805
282026-04-07T02:48:17.326ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/9e74d982-d004-46c2-8d77-0cce2edb2805
292026-04-07T02:48:17.326ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
302026-04-07T02:48:17.326ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::17
312026-04-07T02:48:17.326ZINFOcrucible-agent (worker): commit
322026-04-07T02:48:17.331ZINFOcrucible-agent (worker): ok!
332026-04-07T02:48:17.336ZINFOcrucible-agent (worker): SMF ok!
342026-04-07T02:48:17.690ZINFOcrucible-agent (dropshot): request completed latency_us = 56 local_addr = [fd00:1122:3344:101::17]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:59939 req_id = 1f15365c-3fba-4205-81ee-93512cbe3d1b response_code = 200 uri = /crucible/0/regions
352026-04-07T02:48:17.963ZINFOcrucible-agent (dropshot): request completed latency_us = 75 local_addr = [fd00:1122:3344:101::17]:32345 method = GET remote_addr = [fd00:1122:3344:101::b]:59939 req_id = 1981f280-f207-4586-bd1b-0c99713e468c response_code = 200 uri = /crucible/0/regions/9e74d982-d004-46c2-8d77-0cce2edb2805
362026-04-07T02:48:17.967ZINFOcrucible-agent (datafile): path is "/data/regions/9e74d982-d004-46c2-8d77-0cce2edb2805"
372026-04-07T02:48:17.975ZINFOcrucible-agent (datafile): dataset is oxp_f45535cb-c78c-4fd3-bcd3-4692b7a79434/crucible/regions/9e74d982-d004-46c2-8d77-0cce2edb2805
382026-04-07T02:48:17.982ZINFOcrucible-agent (datafile): region 9e74d982-d004-46c2-8d77-0cce2edb2805 state: Created -> Tombstoned
392026-04-07T02:48:17.982ZINFOcrucible-agent (worker): applying SMF actions before removal...
402026-04-07T02:48:17.982ZINFOcrucible-agent (dropshot): request completed latency_us = 15206 local_addr = [fd00:1122:3344:101::17]:32345 method = DELETE remote_addr = [fd00:1122:3344:101::b]:59939 req_id = 4173bf52-6abc-492b-af04-3b2c4fd5635e response_code = 204 uri = /crucible/0/regions/9e74d982-d004-46c2-8d77-0cce2edb2805
412026-04-07T02:48:17.983ZINFOcrucible-agent (worker): disabling downstairs instance: downstairs-9e74d982-d004-46c2-8d77-0cce2edb2805 (instance states: (Some(Online), None))
422026-04-07T02:48:17.985ZINFOcrucible-agent (dropshot): request completed latency_us = 25 local_addr = [fd00:1122:3344:101::17]:32345 method = GET remote_addr = [fd00:1122:3344:101::b]:59939 req_id = 3ca5fe83-d0b3-44fe-a134-fb8384bbfba6 response_code = 200 uri = /crucible/0/regions/9e74d982-d004-46c2-8d77-0cce2edb2805
432026-04-07T02:48:17.988ZINFOcrucible-agent (worker): SMF ok!
442026-04-07T02:48:17.996ZINFOcrucible-agent (worker): deleting zfs dataset "oxp_f45535cb-c78c-4fd3-bcd3-4692b7a79434/crucible/regions/9e74d982-d004-46c2-8d77-0cce2edb2805" region = 9e74d982-d004-46c2-8d77-0cce2edb2805
452026-04-07T02:48:18.070ZINFOcrucible-agent (datafile): region 9e74d982-d004-46c2-8d77-0cce2edb2805 state: Tombstoned -> Destroyed
462026-04-07T02:48:18.100ZINFOcrucible-agent (dropshot): request completed latency_us = 44 local_addr = [fd00:1122:3344:101::17]:32345 method = GET remote_addr = [fd00:1122:3344:101::b]:59939 req_id = 0035848a-31e8-412e-9290-7d59839c9072 response_code = 200 uri = /crucible/0/regions/9e74d982-d004-46c2-8d77-0cce2edb2805
472026-04-07T02:48:21.633ZINFOcrucible-agent (datafile): region 3e17f998-c85d-4a00-86e6-a1d5b6f672fe state: Requested
482026-04-07T02:48:21.633ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
492026-04-07T02:48:21.633ZINFOcrucible-agent (dropshot): request completed latency_us = 272 local_addr = [fd00:1122:3344:101::17]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:59939 req_id = f7eeb9ec-e7bc-4f8a-a6f5-16afc91f6624 response_code = 200 uri = /crucible/0/regions
502026-04-07T02:48:21.639ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_f45535cb-c78c-4fd3-bcd3-4692b7a79434/crucible/regions/3e17f998-c85d-4a00-86e6-a1d5b6f672fe
512026-04-07T02:48:21.639ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_f45535cb-c78c-4fd3-bcd3-4692b7a79434/crucible/regions/3e17f998-c85d-4a00-86e6-a1d5b6f672fe
522026-04-07T02:48:21.703ZINFOcrucible-agent (worker): creating region Region { id: RegionId("3e17f998-c85d-4a00-86e6-a1d5b6f672fe"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/3e17f998-c85d-4a00-86e6-a1d5b6f672fe" region = 3e17f998-c85d-4a00-86e6-a1d5b6f672fe
532026-04-07T02:48:21.703ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "3e17f998-c85d-4a00-86e6-a1d5b6f672fe" "--data" "/data/regions/3e17f998-c85d-4a00-86e6-a1d5b6f672fe" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 3e17f998-c85d-4a00-86e6-a1d5b6f672fe
542026-04-07T02:48:21.767ZINFOcrucible-agent (dropshot): request completed latency_us = 74 local_addr = [fd00:1122:3344:101::17]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:59939 req_id = 4f349919-7e70-46dc-afe5-2eb96dfd9fc0 response_code = 200 uri = /crucible/0/regions
552026-04-07T02:48:21.779ZINFOcrucible-agent (worker): region files created ok region = 3e17f998-c85d-4a00-86e6-a1d5b6f672fe
562026-04-07T02:48:21.779ZINFOcrucible-agent (datafile): region 3e17f998-c85d-4a00-86e6-a1d5b6f672fe state: Requested -> Created
572026-04-07T02:48:21.779ZINFOcrucible-agent (worker): applying SMF actions post create...
582026-04-07T02:48:21.780ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-3e17f998-c85d-4a00-86e6-a1d5b6f672fe
592026-04-07T02:48:21.783ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-3e17f998-c85d-4a00-86e6-a1d5b6f672fe
602026-04-07T02:48:21.791ZINFOcrucible-agent (worker): creating config property group
612026-04-07T02:48:21.794ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-3e17f998-c85d-4a00-86e6-a1d5b6f672fe
622026-04-07T02:48:21.794ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/3e17f998-c85d-4a00-86e6-a1d5b6f672fe
632026-04-07T02:48:21.794ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
642026-04-07T02:48:21.794ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::17
652026-04-07T02:48:21.794ZINFOcrucible-agent (worker): commit
662026-04-07T02:48:21.798ZINFOcrucible-agent (worker): ok!
672026-04-07T02:48:21.802ZINFOcrucible-agent (worker): SMF ok!
682026-04-07T02:48:22.364ZINFOcrucible-agent (dropshot): request completed latency_us = 62 local_addr = [fd00:1122:3344:101::17]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:59939 req_id = 73356e35-36c2-4309-863e-d1d02ebeed58 response_code = 200 uri = /crucible/0/regions
692026-04-07T02:58:58.820ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::17]:32345 remote_addr = [fd00:1122:3344:101::b]:43995
702026-04-07T02:58:58.821ZINFOcrucible-agent (dropshot): request completed latency_us = 35 local_addr = [fd00:1122:3344:101::17]:32345 method = GET remote_addr = [fd00:1122:3344:101::b]:43995 req_id = 6a7af673-c266-42e3-a872-6748a49efaac response_code = 200 uri = /crucible/0/regions/3e17f998-c85d-4a00-86e6-a1d5b6f672fe
712026-04-07T02:58:58.824ZINFOcrucible-agent (datafile): path is "/data/regions/3e17f998-c85d-4a00-86e6-a1d5b6f672fe"
722026-04-07T02:58:58.830ZINFOcrucible-agent (datafile): dataset is oxp_f45535cb-c78c-4fd3-bcd3-4692b7a79434/crucible/regions/3e17f998-c85d-4a00-86e6-a1d5b6f672fe
732026-04-07T02:58:58.836ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_f45535cb-c78c-4fd3-bcd3-4692b7a79434/crucible/regions/3e17f998-c85d-4a00-86e6-a1d5b6f672fe@d85562e7-4d7e-4a5a-9bf5-4c1f300d0489
742026-04-07T02:58:58.836ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_f45535cb-c78c-4fd3-bcd3-4692b7a79434/crucible/regions/3e17f998-c85d-4a00-86e6-a1d5b6f672fe", "d85562e7-4d7e-4a5a-9bf5-4c1f300d0489"]
752026-04-07T02:58:58.844ZINFOcrucible-agent (dropshot): request completed latency_us = 20417 local_addr = [fd00:1122:3344:101::17]:32345 method = GET remote_addr = [fd00:1122:3344:101::b]:43995 req_id = 30519c59-ac1e-4185-8ca2-5ea17d41fc3d response_code = 200 uri = /crucible/0/regions/3e17f998-c85d-4a00-86e6-a1d5b6f672fe/snapshots/d85562e7-4d7e-4a5a-9bf5-4c1f300d0489
762026-04-07T02:58:58.847ZINFOcrucible-agent (datafile): path is "/data/regions/3e17f998-c85d-4a00-86e6-a1d5b6f672fe"
772026-04-07T02:58:58.853ZINFOcrucible-agent (datafile): dataset is oxp_f45535cb-c78c-4fd3-bcd3-4692b7a79434/crucible/regions/3e17f998-c85d-4a00-86e6-a1d5b6f672fe
782026-04-07T02:58:58.859ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_f45535cb-c78c-4fd3-bcd3-4692b7a79434/crucible/regions/3e17f998-c85d-4a00-86e6-a1d5b6f672fe@d85562e7-4d7e-4a5a-9bf5-4c1f300d0489
792026-04-07T02:58:58.859ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_f45535cb-c78c-4fd3-bcd3-4692b7a79434/crucible/regions/3e17f998-c85d-4a00-86e6-a1d5b6f672fe", "d85562e7-4d7e-4a5a-9bf5-4c1f300d0489"]
802026-04-07T02:58:58.867ZINFOcrucible-agent (datafile): requesting running snapshot 3e17f998-c85d-4a00-86e6-a1d5b6f672fe-d85562e7-4d7e-4a5a-9bf5-4c1f300d0489 state: Requested
812026-04-07T02:58:58.868ZINFOcrucible-agent (worker): applying SMF actions for region 3e17f998-c85d-4a00-86e6-a1d5b6f672fe running snapshot d85562e7-4d7e-4a5a-9bf5-4c1f300d0489 (state Requested)...
822026-04-07T02:58:58.868ZINFOcrucible-agent (dropshot): request completed latency_us = 20317 local_addr = [fd00:1122:3344:101::17]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:43995 req_id = b8bb9076-e932-4c49-8437-37f325e0d5d4 response_code = 200 uri = /crucible/0/regions/3e17f998-c85d-4a00-86e6-a1d5b6f672fe/snapshots/d85562e7-4d7e-4a5a-9bf5-4c1f300d0489/run
832026-04-07T02:58:58.889ZINFOcrucible-agent (worker): creating missing snapshot instance snapshot-3e17f998-c85d-4a00-86e6-a1d5b6f672fe-d85562e7-4d7e-4a5a-9bf5-4c1f300d0489
842026-04-07T02:58:58.907ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:snapshot-3e17f998-c85d-4a00-86e6-a1d5b6f672fe-d85562e7-4d7e-4a5a-9bf5-4c1f300d0489
852026-04-07T02:58:58.929ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:snapshot-3e17f998-c85d-4a00-86e6-a1d5b6f672fe-d85562e7-4d7e-4a5a-9bf5-4c1f300d0489
862026-04-07T02:58:58.929ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/3e17f998-c85d-4a00-86e6-a1d5b6f672fe/.zfs/snapshot/d85562e7-4d7e-4a5a-9bf5-4c1f300d0489
872026-04-07T02:58:58.929ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
882026-04-07T02:58:58.929ZINFOcrucible-agent (worker): ensure mode SCF_TYPE_ASTRING ro
892026-04-07T02:58:58.929ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::17
902026-04-07T02:58:58.929ZINFOcrucible-agent (worker): commit
912026-04-07T02:58:58.933ZINFOcrucible-agent (worker): ok!
922026-04-07T02:58:58.941ZINFOcrucible-agent (worker): SMF ok!
932026-04-07T02:58:58.941ZINFOcrucible-agent (datafile): region 3e17f998-c85d-4a00-86e6-a1d5b6f672fe running snapshot d85562e7-4d7e-4a5a-9bf5-4c1f300d0489 state: Requested -> Created
942026-04-07T02:58:59.171ZINFOcrucible-agent (datafile): path is "/data/regions/3e17f998-c85d-4a00-86e6-a1d5b6f672fe"
952026-04-07T02:58:59.177ZINFOcrucible-agent (datafile): dataset is oxp_f45535cb-c78c-4fd3-bcd3-4692b7a79434/crucible/regions/3e17f998-c85d-4a00-86e6-a1d5b6f672fe
962026-04-07T02:58:59.183ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_f45535cb-c78c-4fd3-bcd3-4692b7a79434/crucible/regions/3e17f998-c85d-4a00-86e6-a1d5b6f672fe@d85562e7-4d7e-4a5a-9bf5-4c1f300d0489
972026-04-07T02:58:59.183ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_f45535cb-c78c-4fd3-bcd3-4692b7a79434/crucible/regions/3e17f998-c85d-4a00-86e6-a1d5b6f672fe", "d85562e7-4d7e-4a5a-9bf5-4c1f300d0489"]
982026-04-07T02:58:59.190ZINFOcrucible-agent (dropshot): request completed latency_us = 19067 local_addr = [fd00:1122:3344:101::17]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:43995 req_id = aaf3009b-ef9b-4819-8844-d8f5ba6e9e39 response_code = 200 uri = /crucible/0/regions/3e17f998-c85d-4a00-86e6-a1d5b6f672fe/snapshots/d85562e7-4d7e-4a5a-9bf5-4c1f300d0489/run