Max level shown:
1 [ Aug 25 23:02:02 Enabled. ]
2 [ Aug 25 23:02:02 Rereading configuration. ]
3 [ Aug 25 23:02:04 Rereading configuration. ]
4 [ Aug 25 23:02:08 Executing start method ("/opt/oxide/crucible/bin/crucible-agent run -D /opt/oxide/crucible/bin/crucible-downstairs --dataset oxp_ff8b6744-39e1-4a1f-b1b7-d67351673376/crucible -l [fd00:1122:3344:101::16]:32345 -P 19000 -p downstairs -s snapshot"). ]
5 note: configured to log to "/dev/stdout"
62026-08-25T23:02:08.713ZINFOcrucible-agent: Crucible Version: Crucible Version: 0.0.1 Commit SHA: ad8a31742adc45e925e63443a5b43c8e30604022 Commit timestamp: 2026-07-28T19:41:59.000000000Z branch: main rustc: 1.90.0 stable x86_64-unknown-illumos Cargo: x86_64-unknown-illumos Debug: false Opt level: 3
72026-08-25T23:02:08.713ZINFOcrucible-agent: dataset: "oxp_ff8b6744-39e1-4a1f-b1b7-d67351673376/crucible"
82026-08-25T23:02:08.713ZINFOcrucible-agent: listen IP: [fd00:1122:3344:101::16]:32345
92026-08-25T23:02:08.713ZINFOcrucible-agent: SMF instance name downstairs_prefix: "downstairs"
102026-08-25T23:02:08.733ZINFOcrucible-agent (datafile): Using conf_path:"/data/crucible.json"
112026-08-25T23:02:08.824ZINFOcrucible-agent (dropshot): listening local_addr = [fd00:1122:3344:101::16]:32345
122026-08-25T23:03:23.598ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::16]:32345 remote_addr = [fd00:1122:3344:101::c]:40950
132026-08-25T23:03:23.630ZINFOcrucible-agent (datafile): region 0746fbd6-da07-4199-800a-6a17daf63713 state: Requested
142026-08-25T23:03:23.635ZINFOcrucible-agent (worker): Region size:1073741824 reservation:1342177280 quota:3221225472
152026-08-25T23:03:23.639ZINFOcrucible-agent (dropshot): request completed latency_us = 14882 local_addr = [fd00:1122:3344:101::16]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:40950 req_id = d73aa329-cdc1-4f49-bc48-743e0c683162 response_code = 200 uri = /crucible/0/regions
162026-08-25T23:03:23.642ZINFOcrucible-agent (worker): zfs set reservation of 1342177280 for oxp_ff8b6744-39e1-4a1f-b1b7-d67351673376/crucible/regions/0746fbd6-da07-4199-800a-6a17daf63713
172026-08-25T23:03:23.642ZINFOcrucible-agent (worker): zfs set quota of 3221225472 for oxp_ff8b6744-39e1-4a1f-b1b7-d67351673376/crucible/regions/0746fbd6-da07-4199-800a-6a17daf63713
182026-08-25T23:03:23.693ZINFOcrucible-agent (worker): creating region Region { id: RegionId("0746fbd6-da07-4199-800a-6a17daf63713"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 16, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/0746fbd6-da07-4199-800a-6a17daf63713" region = 0746fbd6-da07-4199-800a-6a17daf63713
192026-08-25T23:03:23.695ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "0746fbd6-da07-4199-800a-6a17daf63713" "--data" "/data/regions/0746fbd6-da07-4199-800a-6a17daf63713" "--block-size" "512" "--extent-size" "131072" "--extent-count" "16" "--encrypted" region = 0746fbd6-da07-4199-800a-6a17daf63713
202026-08-25T23:03:23.828ZINFOcrucible-agent (worker): region files created ok region = 0746fbd6-da07-4199-800a-6a17daf63713
212026-08-25T23:03:23.830ZINFOcrucible-agent (datafile): region 0746fbd6-da07-4199-800a-6a17daf63713 state: Requested -> Created
222026-08-25T23:03:23.830ZINFOcrucible-agent (worker): applying SMF actions post create...
232026-08-25T23:03:23.833ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-0746fbd6-da07-4199-800a-6a17daf63713
242026-08-25T23:03:23.839ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-0746fbd6-da07-4199-800a-6a17daf63713
252026-08-25T23:03:23.848ZINFOcrucible-agent (worker): creating config property group
262026-08-25T23:03:23.850ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-0746fbd6-da07-4199-800a-6a17daf63713
272026-08-25T23:03:23.850ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/0746fbd6-da07-4199-800a-6a17daf63713
282026-08-25T23:03:23.850ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
292026-08-25T23:03:23.850ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::16
302026-08-25T23:03:23.850ZINFOcrucible-agent (worker): commit
312026-08-25T23:03:23.853ZINFOcrucible-agent (worker): ok!
322026-08-25T23:03:23.855ZINFOcrucible-agent (worker): SMF ok!
332026-08-25T23:03:23.933ZINFOcrucible-agent (dropshot): request completed latency_us = 88 local_addr = [fd00:1122:3344:101::16]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:40950 req_id = 5c66bc72-f01b-4be7-a5f9-197b9b3c3ceb response_code = 200 uri = /crucible/0/regions
342026-08-25T23:03:24.384ZINFOcrucible-agent (dropshot): request completed latency_us = 71 local_addr = [fd00:1122:3344:101::16]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:40950 req_id = ee2e7568-dcf8-435e-8d19-b9eb845abfe4 response_code = 200 uri = /crucible/0/regions/0746fbd6-da07-4199-800a-6a17daf63713
352026-08-25T23:03:24.386ZINFOcrucible-agent (datafile): path is "/data/regions/0746fbd6-da07-4199-800a-6a17daf63713"
362026-08-25T23:03:24.395ZINFOcrucible-agent (datafile): dataset is oxp_ff8b6744-39e1-4a1f-b1b7-d67351673376/crucible/regions/0746fbd6-da07-4199-800a-6a17daf63713
372026-08-25T23:03:24.403ZINFOcrucible-agent (datafile): region 0746fbd6-da07-4199-800a-6a17daf63713 state: Created -> Tombstoned
382026-08-25T23:03:24.404ZINFOcrucible-agent (worker): applying SMF actions before removal...
392026-08-25T23:03:24.404ZINFOcrucible-agent (dropshot): request completed latency_us = 17142 local_addr = [fd00:1122:3344:101::16]:32345 method = DELETE remote_addr = [fd00:1122:3344:101::c]:40950 req_id = 0524cab7-3172-432a-9fac-d3b2c9dc2ef3 response_code = 204 uri = /crucible/0/regions/0746fbd6-da07-4199-800a-6a17daf63713
402026-08-25T23:03:24.405ZINFOcrucible-agent (worker): disabling downstairs instance: downstairs-0746fbd6-da07-4199-800a-6a17daf63713 (instance states: (Some(Online), None))
412026-08-25T23:03:24.408ZINFOcrucible-agent (worker): SMF ok!
422026-08-25T23:03:24.408ZINFOcrucible-agent (dropshot): request completed latency_us = 48 local_addr = [fd00:1122:3344:101::16]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:40950 req_id = d3281d72-1241-4446-a5d3-a9ebb2ed9f6c response_code = 200 uri = /crucible/0/regions/0746fbd6-da07-4199-800a-6a17daf63713
432026-08-25T23:03:24.417ZINFOcrucible-agent (worker): deleting zfs dataset "oxp_ff8b6744-39e1-4a1f-b1b7-d67351673376/crucible/regions/0746fbd6-da07-4199-800a-6a17daf63713" region = 0746fbd6-da07-4199-800a-6a17daf63713
442026-08-25T23:03:24.472ZINFOcrucible-agent (dropshot): request completed latency_us = 53 local_addr = [fd00:1122:3344:101::16]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:40950 req_id = c6a3b676-c63c-4d7b-8702-0256eb43f9f6 response_code = 200 uri = /crucible/0/regions/0746fbd6-da07-4199-800a-6a17daf63713
452026-08-25T23:03:24.472ZINFOcrucible-agent (datafile): region 0746fbd6-da07-4199-800a-6a17daf63713 state: Tombstoned -> Destroyed
462026-08-25T23:03:24.629ZINFOcrucible-agent (dropshot): request completed latency_us = 33 local_addr = [fd00:1122:3344:101::16]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:40950 req_id = 82f0925c-2248-40a7-96df-727eafaa4cb3 response_code = 200 uri = /crucible/0/regions/0746fbd6-da07-4199-800a-6a17daf63713
472026-08-25T23:14:25.765ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::16]:32345 remote_addr = [fd00:1122:3344:101::c]:39982
482026-08-25T23:14:25.765ZINFOcrucible-agent (datafile): region 7a6484a3-2d7a-4b97-962b-e759a5a8460d state: Requested
492026-08-25T23:14:25.766ZINFOcrucible-agent (dropshot): request completed latency_us = 858 local_addr = [fd00:1122:3344:101::16]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:39982 req_id = f8e8fa8a-fc6e-4d16-839b-720b093ac6a7 response_code = 200 uri = /crucible/0/regions
502026-08-25T23:14:25.766ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
512026-08-25T23:14:25.777ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_ff8b6744-39e1-4a1f-b1b7-d67351673376/crucible/regions/7a6484a3-2d7a-4b97-962b-e759a5a8460d
522026-08-25T23:14:25.777ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_ff8b6744-39e1-4a1f-b1b7-d67351673376/crucible/regions/7a6484a3-2d7a-4b97-962b-e759a5a8460d
532026-08-25T23:14:25.820ZINFOcrucible-agent (worker): creating region Region { id: RegionId("7a6484a3-2d7a-4b97-962b-e759a5a8460d"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/7a6484a3-2d7a-4b97-962b-e759a5a8460d" region = 7a6484a3-2d7a-4b97-962b-e759a5a8460d
542026-08-25T23:14:25.820ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "7a6484a3-2d7a-4b97-962b-e759a5a8460d" "--data" "/data/regions/7a6484a3-2d7a-4b97-962b-e759a5a8460d" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 7a6484a3-2d7a-4b97-962b-e759a5a8460d
552026-08-25T23:14:25.869ZINFOcrucible-agent (worker): region files created ok region = 7a6484a3-2d7a-4b97-962b-e759a5a8460d
562026-08-25T23:14:25.869ZINFOcrucible-agent (datafile): region 7a6484a3-2d7a-4b97-962b-e759a5a8460d state: Requested -> Created
572026-08-25T23:14:25.869ZINFOcrucible-agent (worker): applying SMF actions post create...
582026-08-25T23:14:25.870ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-7a6484a3-2d7a-4b97-962b-e759a5a8460d
592026-08-25T23:14:25.877ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-7a6484a3-2d7a-4b97-962b-e759a5a8460d
602026-08-25T23:14:25.891ZINFOcrucible-agent (worker): creating config property group
612026-08-25T23:14:25.892ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-7a6484a3-2d7a-4b97-962b-e759a5a8460d
622026-08-25T23:14:25.893ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/7a6484a3-2d7a-4b97-962b-e759a5a8460d
632026-08-25T23:14:25.893ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
642026-08-25T23:14:25.893ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::16
652026-08-25T23:14:25.893ZINFOcrucible-agent (worker): commit
662026-08-25T23:14:25.894ZINFOcrucible-agent (worker): ok!
672026-08-25T23:14:25.897ZINFOcrucible-agent (worker): SMF ok!
682026-08-25T23:14:25.967ZINFOcrucible-agent (dropshot): request completed latency_us = 55 local_addr = [fd00:1122:3344:101::16]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:39982 req_id = 25a42978-d5a2-4e03-9c22-7aecce4cc0ef response_code = 200 uri = /crucible/0/regions
692026-08-25T23:14:32.109ZINFOcrucible-agent (datafile): region d297d779-9d2d-47d5-9bac-58fec1ef04a2 state: Requested
702026-08-25T23:14:32.109ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
712026-08-25T23:14:32.109ZINFOcrucible-agent (dropshot): request completed latency_us = 231 local_addr = [fd00:1122:3344:101::16]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:39982 req_id = 95156071-b2de-4634-b328-d43bd0a1ffe6 response_code = 200 uri = /crucible/0/regions
722026-08-25T23:14:32.116ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_ff8b6744-39e1-4a1f-b1b7-d67351673376/crucible/regions/d297d779-9d2d-47d5-9bac-58fec1ef04a2
732026-08-25T23:14:32.116ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_ff8b6744-39e1-4a1f-b1b7-d67351673376/crucible/regions/d297d779-9d2d-47d5-9bac-58fec1ef04a2
742026-08-25T23:14:32.160ZINFOcrucible-agent (worker): creating region Region { id: RegionId("d297d779-9d2d-47d5-9bac-58fec1ef04a2"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19001, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/d297d779-9d2d-47d5-9bac-58fec1ef04a2" region = d297d779-9d2d-47d5-9bac-58fec1ef04a2
752026-08-25T23:14:32.160ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "d297d779-9d2d-47d5-9bac-58fec1ef04a2" "--data" "/data/regions/d297d779-9d2d-47d5-9bac-58fec1ef04a2" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = d297d779-9d2d-47d5-9bac-58fec1ef04a2
762026-08-25T23:14:32.209ZINFOcrucible-agent (worker): region files created ok region = d297d779-9d2d-47d5-9bac-58fec1ef04a2
772026-08-25T23:14:32.209ZINFOcrucible-agent (datafile): region d297d779-9d2d-47d5-9bac-58fec1ef04a2 state: Requested -> Created
782026-08-25T23:14:32.209ZINFOcrucible-agent (worker): applying SMF actions post create...
792026-08-25T23:14:32.210ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-d297d779-9d2d-47d5-9bac-58fec1ef04a2
802026-08-25T23:14:32.224ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-d297d779-9d2d-47d5-9bac-58fec1ef04a2
812026-08-25T23:14:32.227ZINFOcrucible-agent (worker): creating config property group
822026-08-25T23:14:32.228ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-d297d779-9d2d-47d5-9bac-58fec1ef04a2
832026-08-25T23:14:32.229ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/d297d779-9d2d-47d5-9bac-58fec1ef04a2
842026-08-25T23:14:32.229ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
852026-08-25T23:14:32.229ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::16
862026-08-25T23:14:32.229ZINFOcrucible-agent (worker): commit
872026-08-25T23:14:32.231ZINFOcrucible-agent (worker): ok!
882026-08-25T23:14:32.234ZINFOcrucible-agent (worker): SMF ok!
892026-08-25T23:14:32.371ZINFOcrucible-agent (dropshot): request completed latency_us = 64 local_addr = [fd00:1122:3344:101::16]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:39982 req_id = 88e0d575-2d04-4e4d-9372-23749daca474 response_code = 200 uri = /crucible/0/regions
902026-08-25T23:14:57.913ZINFOcrucible-agent (dropshot): request completed latency_us = 53 local_addr = [fd00:1122:3344:101::16]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:39982 req_id = 345010c0-b04f-4b37-b22b-072030ce0712 response_code = 200 uri = /crucible/0/regions/d297d779-9d2d-47d5-9bac-58fec1ef04a2
912026-08-25T23:14:57.917ZINFOcrucible-agent (datafile): path is "/data/regions/d297d779-9d2d-47d5-9bac-58fec1ef04a2"
922026-08-25T23:14:57.927ZINFOcrucible-agent (datafile): dataset is oxp_ff8b6744-39e1-4a1f-b1b7-d67351673376/crucible/regions/d297d779-9d2d-47d5-9bac-58fec1ef04a2
932026-08-25T23:14:57.933ZINFOcrucible-agent (datafile): region d297d779-9d2d-47d5-9bac-58fec1ef04a2 state: Created -> Tombstoned
942026-08-25T23:14:57.933ZINFOcrucible-agent (worker): applying SMF actions before removal...
952026-08-25T23:14:57.933ZINFOcrucible-agent (dropshot): request completed latency_us = 16303 local_addr = [fd00:1122:3344:101::16]:32345 method = DELETE remote_addr = [fd00:1122:3344:101::c]:39982 req_id = 37e6a4d4-1991-4586-9374-8d0fd9e7b7b5 response_code = 204 uri = /crucible/0/regions/d297d779-9d2d-47d5-9bac-58fec1ef04a2
962026-08-25T23:14:57.935ZINFOcrucible-agent (worker): disabling downstairs instance: downstairs-d297d779-9d2d-47d5-9bac-58fec1ef04a2 (instance states: (Some(Online), None))
972026-08-25T23:14:57.936ZINFOcrucible-agent (dropshot): request completed latency_us = 41 local_addr = [fd00:1122:3344:101::16]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:39982 req_id = 0ac4efce-43cf-42be-a543-031886a17242 response_code = 200 uri = /crucible/0/regions/d297d779-9d2d-47d5-9bac-58fec1ef04a2
982026-08-25T23:14:57.938ZINFOcrucible-agent (worker): SMF ok!
992026-08-25T23:14:57.946ZINFOcrucible-agent (worker): deleting zfs dataset "oxp_ff8b6744-39e1-4a1f-b1b7-d67351673376/crucible/regions/d297d779-9d2d-47d5-9bac-58fec1ef04a2" region = d297d779-9d2d-47d5-9bac-58fec1ef04a2
1002026-08-25T23:14:57.992ZINFOcrucible-agent (datafile): region d297d779-9d2d-47d5-9bac-58fec1ef04a2 state: Tombstoned -> Destroyed
1012026-08-25T23:14:58.088ZINFOcrucible-agent (dropshot): request completed latency_us = 52 local_addr = [fd00:1122:3344:101::16]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:39982 req_id = 72e6e08d-0afa-42a3-a5ec-63645c9183e2 response_code = 200 uri = /crucible/0/regions/d297d779-9d2d-47d5-9bac-58fec1ef04a2