Max level shown:
1 [ May 28 17:30:48 Enabled. ]
2 [ May 28 17:30:48 Rereading configuration. ]
3 [ May 28 17:30:50 Rereading configuration. ]
4 [ May 28 17:30:55 Executing start method ("/opt/oxide/crucible/bin/crucible-agent run -D /opt/oxide/crucible/bin/crucible-downstairs --dataset oxp_39194572-7e6e-4ea1-92a4-b67c0c4e1811/crucible -l [fd00:1122:3344:101::15]:32345 -P 19000 -p downstairs -s snapshot"). ]
5 note: configured to log to "/dev/stdout"
62026-05-28T17:30:56.132ZINFOcrucible-agent: Crucible Version: Crucible Version: 0.0.1 Commit SHA: bd9a0e2abe6b6b89aec8c85f4ee57474144ed150 Commit timestamp: 2026-05-14T15:48:00.000000000Z branch: main rustc: 1.90.0 stable x86_64-unknown-illumos Cargo: x86_64-unknown-illumos Debug: false Opt level: 3
72026-05-28T17:30:56.132ZINFOcrucible-agent: dataset: "oxp_39194572-7e6e-4ea1-92a4-b67c0c4e1811/crucible"
82026-05-28T17:30:56.132ZINFOcrucible-agent: listen IP: [fd00:1122:3344:101::15]:32345
92026-05-28T17:30:56.132ZINFOcrucible-agent: SMF instance name downstairs_prefix: "downstairs"
102026-05-28T17:30:56.147ZINFOcrucible-agent (datafile): Using conf_path:"/data/crucible.json"
112026-05-28T17:30:56.357ZINFOcrucible-agent (dropshot): listening local_addr = [fd00:1122:3344:101::15]:32345
122026-05-28T17:31:26.081ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::15]:32345 remote_addr = [fd00:1122:3344:101::c]:36216
132026-05-28T17:31:26.100ZINFOcrucible-agent (datafile): region 5e2bc0c4-64ef-4592-ae7e-8afa80c7ffda state: Requested
142026-05-28T17:31:26.105ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
152026-05-28T17:31:26.105ZINFOcrucible-agent (dropshot): request completed latency_us = 11887 local_addr = [fd00:1122:3344:101::15]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:36216 req_id = 01d266ab-07d9-457a-9208-ec4e0f7f0328 response_code = 200 uri = /crucible/0/regions
162026-05-28T17:31:26.115ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_39194572-7e6e-4ea1-92a4-b67c0c4e1811/crucible/regions/5e2bc0c4-64ef-4592-ae7e-8afa80c7ffda
172026-05-28T17:31:26.115ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_39194572-7e6e-4ea1-92a4-b67c0c4e1811/crucible/regions/5e2bc0c4-64ef-4592-ae7e-8afa80c7ffda
182026-05-28T17:31:26.186ZINFOcrucible-agent (worker): creating region Region { id: RegionId("5e2bc0c4-64ef-4592-ae7e-8afa80c7ffda"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/5e2bc0c4-64ef-4592-ae7e-8afa80c7ffda" region = 5e2bc0c4-64ef-4592-ae7e-8afa80c7ffda
192026-05-28T17:31:26.186ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "5e2bc0c4-64ef-4592-ae7e-8afa80c7ffda" "--data" "/data/regions/5e2bc0c4-64ef-4592-ae7e-8afa80c7ffda" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 5e2bc0c4-64ef-4592-ae7e-8afa80c7ffda
202026-05-28T17:31:26.431ZINFOcrucible-agent (dropshot): request completed latency_us = 63 local_addr = [fd00:1122:3344:101::15]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:36216 req_id = 9e40cf77-e130-44ca-bf59-fa285c76ef89 response_code = 200 uri = /crucible/0/regions
212026-05-28T17:31:26.541ZINFOcrucible-agent (worker): region files created ok region = 5e2bc0c4-64ef-4592-ae7e-8afa80c7ffda
222026-05-28T17:31:26.541ZINFOcrucible-agent (datafile): region 5e2bc0c4-64ef-4592-ae7e-8afa80c7ffda state: Requested -> Created
232026-05-28T17:31:26.542ZINFOcrucible-agent (worker): applying SMF actions post create...
242026-05-28T17:31:26.544ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-5e2bc0c4-64ef-4592-ae7e-8afa80c7ffda
252026-05-28T17:31:26.550ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-5e2bc0c4-64ef-4592-ae7e-8afa80c7ffda
262026-05-28T17:31:26.565ZINFOcrucible-agent (worker): creating config property group
272026-05-28T17:31:26.568ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-5e2bc0c4-64ef-4592-ae7e-8afa80c7ffda
282026-05-28T17:31:26.569ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/5e2bc0c4-64ef-4592-ae7e-8afa80c7ffda
292026-05-28T17:31:26.569ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
302026-05-28T17:31:26.569ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::15
312026-05-28T17:31:26.569ZINFOcrucible-agent (worker): commit
322026-05-28T17:31:26.579ZINFOcrucible-agent (worker): ok!
332026-05-28T17:31:26.584ZINFOcrucible-agent (worker): SMF ok!
342026-05-28T17:31:26.716ZINFOcrucible-agent (dropshot): request completed latency_us = 61 local_addr = [fd00:1122:3344:101::15]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:36216 req_id = a3dc784b-5822-4dfc-b777-0b133dd74fd9 response_code = 200 uri = /crucible/0/regions
352026-05-28T17:41:10.461ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::15]:32345 remote_addr = [fd00:1122:3344:101::c]:50012
362026-05-28T17:41:10.462ZINFOcrucible-agent (datafile): region 2c4a4a7d-f8cc-4b82-aefc-d64946f5efe1 state: Requested
372026-05-28T17:41:10.465ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
382026-05-28T17:41:10.465ZINFOcrucible-agent (dropshot): request completed latency_us = 4727 local_addr = [fd00:1122:3344:101::15]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:50012 req_id = 5de297d6-1c2d-4c7d-9ce0-e137f69a0e55 response_code = 200 uri = /crucible/0/regions
392026-05-28T17:41:10.474ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_39194572-7e6e-4ea1-92a4-b67c0c4e1811/crucible/regions/2c4a4a7d-f8cc-4b82-aefc-d64946f5efe1
402026-05-28T17:41:10.474ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_39194572-7e6e-4ea1-92a4-b67c0c4e1811/crucible/regions/2c4a4a7d-f8cc-4b82-aefc-d64946f5efe1
412026-05-28T17:41:10.535ZINFOcrucible-agent (worker): creating region Region { id: RegionId("2c4a4a7d-f8cc-4b82-aefc-d64946f5efe1"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19001, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/2c4a4a7d-f8cc-4b82-aefc-d64946f5efe1" region = 2c4a4a7d-f8cc-4b82-aefc-d64946f5efe1
422026-05-28T17:41:10.539ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "2c4a4a7d-f8cc-4b82-aefc-d64946f5efe1" "--data" "/data/regions/2c4a4a7d-f8cc-4b82-aefc-d64946f5efe1" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 2c4a4a7d-f8cc-4b82-aefc-d64946f5efe1
432026-05-28T17:41:10.664ZINFOcrucible-agent (dropshot): request completed latency_us = 51 local_addr = [fd00:1122:3344:101::15]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:50012 req_id = 389afb47-3052-424e-9c3a-76a8d7ddc842 response_code = 200 uri = /crucible/0/regions
442026-05-28T17:41:10.666ZINFOcrucible-agent (worker): region files created ok region = 2c4a4a7d-f8cc-4b82-aefc-d64946f5efe1
452026-05-28T17:41:10.666ZINFOcrucible-agent (datafile): region 2c4a4a7d-f8cc-4b82-aefc-d64946f5efe1 state: Requested -> Created
462026-05-28T17:41:10.666ZINFOcrucible-agent (worker): applying SMF actions post create...
472026-05-28T17:41:10.667ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-2c4a4a7d-f8cc-4b82-aefc-d64946f5efe1
482026-05-28T17:41:10.676ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-2c4a4a7d-f8cc-4b82-aefc-d64946f5efe1
492026-05-28T17:41:10.694ZINFOcrucible-agent (worker): creating config property group
502026-05-28T17:41:10.697ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-2c4a4a7d-f8cc-4b82-aefc-d64946f5efe1
512026-05-28T17:41:10.697ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/2c4a4a7d-f8cc-4b82-aefc-d64946f5efe1
522026-05-28T17:41:10.697ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
532026-05-28T17:41:10.698ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::15
542026-05-28T17:41:10.698ZINFOcrucible-agent (worker): commit
552026-05-28T17:41:10.701ZINFOcrucible-agent (worker): ok!
562026-05-28T17:41:10.715ZINFOcrucible-agent (worker): existing directory value /dev/null does not match /data/regions/5e2bc0c4-64ef-4592-ae7e-8afa80c7ffda
572026-05-28T17:41:10.715ZINFOcrucible-agent (worker): existing port value 0 does not match 19000
582026-05-28T17:41:10.716ZINFOcrucible-agent (worker): existing address value 0.0.0.0 does not match fd00:1122:3344:101::15
592026-05-28T17:41:10.716ZINFOcrucible-agent (worker): reconfig required
602026-05-28T17:41:10.716ZINFOcrucible-agent (worker): using existing config property group
612026-05-28T17:41:10.716ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-5e2bc0c4-64ef-4592-ae7e-8afa80c7ffda
622026-05-28T17:41:10.716ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/5e2bc0c4-64ef-4592-ae7e-8afa80c7ffda
632026-05-28T17:41:10.716ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
642026-05-28T17:41:10.716ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::15
652026-05-28T17:41:10.716ZINFOcrucible-agent (worker): commit
662026-05-28T17:41:10.721ZINFOcrucible-agent (worker): ok!
672026-05-28T17:41:10.722ZINFOcrucible-agent (worker): SMF ok!
682026-05-28T17:41:11.348ZINFOcrucible-agent (dropshot): request completed latency_us = 74 local_addr = [fd00:1122:3344:101::15]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:50012 req_id = e762b839-af95-412f-abbe-94b3af3d08b5 response_code = 200 uri = /crucible/0/regions
692026-05-28T17:41:11.692ZINFOcrucible-agent (dropshot): request completed latency_us = 77 local_addr = [fd00:1122:3344:101::15]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:50012 req_id = fb0a7ea9-038f-43bb-9d38-8160ee2a2c66 response_code = 200 uri = /crucible/0/regions/5e2bc0c4-64ef-4592-ae7e-8afa80c7ffda
702026-05-28T17:41:11.696ZINFOcrucible-agent (datafile): path is "/data/regions/5e2bc0c4-64ef-4592-ae7e-8afa80c7ffda"
712026-05-28T17:41:11.702ZINFOcrucible-agent (datafile): dataset is oxp_39194572-7e6e-4ea1-92a4-b67c0c4e1811/crucible/regions/5e2bc0c4-64ef-4592-ae7e-8afa80c7ffda
722026-05-28T17:41:11.707ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_39194572-7e6e-4ea1-92a4-b67c0c4e1811/crucible/regions/5e2bc0c4-64ef-4592-ae7e-8afa80c7ffda@d33ad71a-9e33-4bc1-be64-e6f2b8de52c4
732026-05-28T17:41:11.707ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_39194572-7e6e-4ea1-92a4-b67c0c4e1811/crucible/regions/5e2bc0c4-64ef-4592-ae7e-8afa80c7ffda", "d33ad71a-9e33-4bc1-be64-e6f2b8de52c4"]
742026-05-28T17:41:11.714ZINFOcrucible-agent (dropshot): request completed latency_us = 18103 local_addr = [fd00:1122:3344:101::15]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:50012 req_id = 9733442b-532b-46e9-89a2-c25ae647d64f response_code = 200 uri = /crucible/0/regions/5e2bc0c4-64ef-4592-ae7e-8afa80c7ffda/snapshots/d33ad71a-9e33-4bc1-be64-e6f2b8de52c4
752026-05-28T17:41:11.722ZINFOcrucible-agent (datafile): path is "/data/regions/5e2bc0c4-64ef-4592-ae7e-8afa80c7ffda"
762026-05-28T17:41:11.726ZINFOcrucible-agent (datafile): dataset is oxp_39194572-7e6e-4ea1-92a4-b67c0c4e1811/crucible/regions/5e2bc0c4-64ef-4592-ae7e-8afa80c7ffda
772026-05-28T17:41:11.731ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_39194572-7e6e-4ea1-92a4-b67c0c4e1811/crucible/regions/5e2bc0c4-64ef-4592-ae7e-8afa80c7ffda@d33ad71a-9e33-4bc1-be64-e6f2b8de52c4
782026-05-28T17:41:11.731ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_39194572-7e6e-4ea1-92a4-b67c0c4e1811/crucible/regions/5e2bc0c4-64ef-4592-ae7e-8afa80c7ffda", "d33ad71a-9e33-4bc1-be64-e6f2b8de52c4"]
792026-05-28T17:41:11.738ZINFOcrucible-agent (datafile): requesting running snapshot 5e2bc0c4-64ef-4592-ae7e-8afa80c7ffda-d33ad71a-9e33-4bc1-be64-e6f2b8de52c4 state: Requested
802026-05-28T17:41:11.738ZINFOcrucible-agent (worker): applying SMF actions for region 5e2bc0c4-64ef-4592-ae7e-8afa80c7ffda running snapshot d33ad71a-9e33-4bc1-be64-e6f2b8de52c4 (state Requested)...
812026-05-28T17:41:11.738ZINFOcrucible-agent (dropshot): request completed latency_us = 16096 local_addr = [fd00:1122:3344:101::15]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:50012 req_id = fc42c774-ebef-490e-baf0-54775bc60085 response_code = 200 uri = /crucible/0/regions/5e2bc0c4-64ef-4592-ae7e-8afa80c7ffda/snapshots/d33ad71a-9e33-4bc1-be64-e6f2b8de52c4/run
822026-05-28T17:41:11.739ZINFOcrucible-agent (worker): existing directory value /dev/null does not match /data/regions/5e2bc0c4-64ef-4592-ae7e-8afa80c7ffda
832026-05-28T17:41:11.739ZINFOcrucible-agent (worker): existing port value 0 does not match 19000
842026-05-28T17:41:11.740ZINFOcrucible-agent (worker): existing address value 0.0.0.0 does not match fd00:1122:3344:101::15
852026-05-28T17:41:11.740ZINFOcrucible-agent (worker): reconfig required
862026-05-28T17:41:11.740ZINFOcrucible-agent (worker): using existing config property group
872026-05-28T17:41:11.740ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-5e2bc0c4-64ef-4592-ae7e-8afa80c7ffda
882026-05-28T17:41:11.740ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/5e2bc0c4-64ef-4592-ae7e-8afa80c7ffda
892026-05-28T17:41:11.740ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
902026-05-28T17:41:11.740ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::15
912026-05-28T17:41:11.740ZINFOcrucible-agent (worker): commit
922026-05-28T17:41:11.743ZINFOcrucible-agent (worker): ok!
932026-05-28T17:41:11.746ZINFOcrucible-agent (worker): creating missing snapshot instance snapshot-5e2bc0c4-64ef-4592-ae7e-8afa80c7ffda-d33ad71a-9e33-4bc1-be64-e6f2b8de52c4
942026-05-28T17:41:11.756ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:snapshot-5e2bc0c4-64ef-4592-ae7e-8afa80c7ffda-d33ad71a-9e33-4bc1-be64-e6f2b8de52c4
952026-05-28T17:41:11.765ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:snapshot-5e2bc0c4-64ef-4592-ae7e-8afa80c7ffda-d33ad71a-9e33-4bc1-be64-e6f2b8de52c4
962026-05-28T17:41:11.766ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/5e2bc0c4-64ef-4592-ae7e-8afa80c7ffda/.zfs/snapshot/d33ad71a-9e33-4bc1-be64-e6f2b8de52c4
972026-05-28T17:41:11.766ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19002
982026-05-28T17:41:11.766ZINFOcrucible-agent (worker): ensure mode SCF_TYPE_ASTRING ro
992026-05-28T17:41:11.766ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::15
1002026-05-28T17:41:11.766ZINFOcrucible-agent (worker): commit
1012026-05-28T17:41:11.770ZINFOcrucible-agent (worker): ok!
1022026-05-28T17:41:11.773ZINFOcrucible-agent (worker): SMF ok!
1032026-05-28T17:41:11.773ZINFOcrucible-agent (datafile): region 5e2bc0c4-64ef-4592-ae7e-8afa80c7ffda running snapshot d33ad71a-9e33-4bc1-be64-e6f2b8de52c4 state: Requested -> Created
1042026-05-28T17:41:11.992ZINFOcrucible-agent (datafile): path is "/data/regions/5e2bc0c4-64ef-4592-ae7e-8afa80c7ffda"
1052026-05-28T17:41:11.998ZINFOcrucible-agent (datafile): dataset is oxp_39194572-7e6e-4ea1-92a4-b67c0c4e1811/crucible/regions/5e2bc0c4-64ef-4592-ae7e-8afa80c7ffda
1062026-05-28T17:41:12.005ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_39194572-7e6e-4ea1-92a4-b67c0c4e1811/crucible/regions/5e2bc0c4-64ef-4592-ae7e-8afa80c7ffda@d33ad71a-9e33-4bc1-be64-e6f2b8de52c4
1072026-05-28T17:41:12.005ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_39194572-7e6e-4ea1-92a4-b67c0c4e1811/crucible/regions/5e2bc0c4-64ef-4592-ae7e-8afa80c7ffda", "d33ad71a-9e33-4bc1-be64-e6f2b8de52c4"]
1082026-05-28T17:41:12.012ZINFOcrucible-agent (dropshot): request completed latency_us = 19887 local_addr = [fd00:1122:3344:101::15]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:50012 req_id = 6527c047-7067-4929-bee3-2d8d0ebc293c response_code = 200 uri = /crucible/0/regions/5e2bc0c4-64ef-4592-ae7e-8afa80c7ffda/snapshots/d33ad71a-9e33-4bc1-be64-e6f2b8de52c4/run