Max level shown:
1 [ Jul 2 04:15:16 Enabled. ]
2 [ Jul 2 04:15:16 Rereading configuration. ]
3 [ Jul 2 04:15:17 Rereading configuration. ]
4 [ Jul 2 04:15:20 Executing start method ("/opt/oxide/crucible/bin/crucible-agent run -D /opt/oxide/crucible/bin/crucible-downstairs --dataset oxp_b8508b78-4ea7-43a3-8927-5a33e998cf65/crucible -l [fd00:1122:3344:101::15]:32345 -P 19000 -p downstairs -s snapshot"). ]
5 note: configured to log to "/dev/stdout"
62026-07-02T04:15:20.704ZINFOcrucible-agent: Crucible Version: Crucible Version: 0.0.1 Commit SHA: 2bfe090eb5318ec8c467157018db2429d4df535b Commit timestamp: 2026-06-25T19:33:01.000000000Z branch: main rustc: 1.90.0 stable x86_64-unknown-illumos Cargo: x86_64-unknown-illumos Debug: false Opt level: 3
72026-07-02T04:15:20.704ZINFOcrucible-agent: dataset: "oxp_b8508b78-4ea7-43a3-8927-5a33e998cf65/crucible"
82026-07-02T04:15:20.704ZINFOcrucible-agent: listen IP: [fd00:1122:3344:101::15]:32345
92026-07-02T04:15:20.704ZINFOcrucible-agent: SMF instance name downstairs_prefix: "downstairs"
102026-07-02T04:15:20.722ZINFOcrucible-agent (datafile): Using conf_path:"/data/crucible.json"
112026-07-02T04:15:20.786ZINFOcrucible-agent (dropshot): listening local_addr = [fd00:1122:3344:101::15]:32345
122026-07-02T04:27:33.507ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::15]:32345 remote_addr = [fd00:1122:3344:101::b]:34188
132026-07-02T04:27:33.551ZINFOcrucible-agent (datafile): region 8ec8ced8-bd38-4fab-b716-e4a25108daad state: Requested
142026-07-02T04:27:33.557ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
152026-07-02T04:27:33.561ZINFOcrucible-agent (dropshot): request completed latency_us = 21831 local_addr = [fd00:1122:3344:101::15]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:34188 req_id = e109bc81-fffb-4bc8-b507-a1f70f15e5ef response_code = 200 uri = /crucible/0/regions
162026-07-02T04:27:33.568ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_b8508b78-4ea7-43a3-8927-5a33e998cf65/crucible/regions/8ec8ced8-bd38-4fab-b716-e4a25108daad
172026-07-02T04:27:33.568ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_b8508b78-4ea7-43a3-8927-5a33e998cf65/crucible/regions/8ec8ced8-bd38-4fab-b716-e4a25108daad
182026-07-02T04:27:33.610ZINFOcrucible-agent (worker): creating region Region { id: RegionId("8ec8ced8-bd38-4fab-b716-e4a25108daad"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/8ec8ced8-bd38-4fab-b716-e4a25108daad" region = 8ec8ced8-bd38-4fab-b716-e4a25108daad
192026-07-02T04:27:33.613ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "8ec8ced8-bd38-4fab-b716-e4a25108daad" "--data" "/data/regions/8ec8ced8-bd38-4fab-b716-e4a25108daad" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 8ec8ced8-bd38-4fab-b716-e4a25108daad
202026-07-02T04:27:33.822ZINFOcrucible-agent (dropshot): request completed latency_us = 58 local_addr = [fd00:1122:3344:101::15]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:34188 req_id = 81b6d802-563e-4aec-b35f-6380be6f4f36 response_code = 200 uri = /crucible/0/regions
212026-07-02T04:27:33.964ZINFOcrucible-agent (worker): region files created ok region = 8ec8ced8-bd38-4fab-b716-e4a25108daad
222026-07-02T04:27:33.968ZINFOcrucible-agent (datafile): region 8ec8ced8-bd38-4fab-b716-e4a25108daad state: Requested -> Created
232026-07-02T04:27:33.968ZINFOcrucible-agent (worker): applying SMF actions post create...
242026-07-02T04:27:33.972ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-8ec8ced8-bd38-4fab-b716-e4a25108daad
252026-07-02T04:27:33.981ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-8ec8ced8-bd38-4fab-b716-e4a25108daad
262026-07-02T04:27:33.993ZINFOcrucible-agent (worker): creating config property group
272026-07-02T04:27:33.995ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-8ec8ced8-bd38-4fab-b716-e4a25108daad
282026-07-02T04:27:33.995ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/8ec8ced8-bd38-4fab-b716-e4a25108daad
292026-07-02T04:27:33.998ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
302026-07-02T04:27:33.998ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::15
312026-07-02T04:27:33.998ZINFOcrucible-agent (worker): commit
322026-07-02T04:27:34.004ZINFOcrucible-agent (worker): ok!
332026-07-02T04:27:34.006ZINFOcrucible-agent (worker): SMF ok!
342026-07-02T04:27:34.333ZINFOcrucible-agent (dropshot): request completed latency_us = 96 local_addr = [fd00:1122:3344:101::15]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:34188 req_id = a8112845-2502-48e1-ab4c-cd2db6ced612 response_code = 200 uri = /crucible/0/regions
352026-07-02T04:27:40.487ZINFOcrucible-agent (datafile): region 71d1399d-865c-46b5-b1ca-df518690829a state: Requested
362026-07-02T04:27:40.487ZINFOcrucible-agent (dropshot): request completed latency_us = 227 local_addr = [fd00:1122:3344:101::15]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:34188 req_id = 703bbf80-409a-40e4-8f00-1ebd6ec01367 response_code = 200 uri = /crucible/0/regions
372026-07-02T04:27:40.488ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
382026-07-02T04:27:40.493ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_b8508b78-4ea7-43a3-8927-5a33e998cf65/crucible/regions/71d1399d-865c-46b5-b1ca-df518690829a
392026-07-02T04:27:40.493ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_b8508b78-4ea7-43a3-8927-5a33e998cf65/crucible/regions/71d1399d-865c-46b5-b1ca-df518690829a
402026-07-02T04:27:40.533ZINFOcrucible-agent (worker): creating region Region { id: RegionId("71d1399d-865c-46b5-b1ca-df518690829a"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19001, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/71d1399d-865c-46b5-b1ca-df518690829a" region = 71d1399d-865c-46b5-b1ca-df518690829a
412026-07-02T04:27:40.533ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "71d1399d-865c-46b5-b1ca-df518690829a" "--data" "/data/regions/71d1399d-865c-46b5-b1ca-df518690829a" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 71d1399d-865c-46b5-b1ca-df518690829a
422026-07-02T04:27:40.580ZINFOcrucible-agent (worker): region files created ok region = 71d1399d-865c-46b5-b1ca-df518690829a
432026-07-02T04:27:40.580ZINFOcrucible-agent (datafile): region 71d1399d-865c-46b5-b1ca-df518690829a state: Requested -> Created
442026-07-02T04:27:40.580ZINFOcrucible-agent (worker): applying SMF actions post create...
452026-07-02T04:27:40.581ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-71d1399d-865c-46b5-b1ca-df518690829a
462026-07-02T04:27:40.582ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-71d1399d-865c-46b5-b1ca-df518690829a
472026-07-02T04:27:40.586ZINFOcrucible-agent (worker): creating config property group
482026-07-02T04:27:40.587ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-71d1399d-865c-46b5-b1ca-df518690829a
492026-07-02T04:27:40.587ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/71d1399d-865c-46b5-b1ca-df518690829a
502026-07-02T04:27:40.587ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
512026-07-02T04:27:40.587ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::15
522026-07-02T04:27:40.587ZINFOcrucible-agent (worker): commit
532026-07-02T04:27:40.589ZINFOcrucible-agent (worker): ok!
542026-07-02T04:27:40.593ZINFOcrucible-agent (worker): existing directory value /dev/null does not match /data/regions/8ec8ced8-bd38-4fab-b716-e4a25108daad
552026-07-02T04:27:40.593ZINFOcrucible-agent (worker): existing port value 0 does not match 19000
562026-07-02T04:27:40.593ZINFOcrucible-agent (worker): existing address value 0.0.0.0 does not match fd00:1122:3344:101::15
572026-07-02T04:27:40.593ZINFOcrucible-agent (worker): reconfig required
582026-07-02T04:27:40.593ZINFOcrucible-agent (worker): using existing config property group
592026-07-02T04:27:40.593ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-8ec8ced8-bd38-4fab-b716-e4a25108daad
602026-07-02T04:27:40.593ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/8ec8ced8-bd38-4fab-b716-e4a25108daad
612026-07-02T04:27:40.593ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
622026-07-02T04:27:40.593ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::15
632026-07-02T04:27:40.593ZINFOcrucible-agent (worker): commit
642026-07-02T04:27:40.598ZINFOcrucible-agent (worker): ok!
652026-07-02T04:27:40.598ZINFOcrucible-agent (worker): SMF ok!
662026-07-02T04:27:40.806ZINFOcrucible-agent (dropshot): request completed latency_us = 83 local_addr = [fd00:1122:3344:101::15]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:34188 req_id = bfd48bba-84e3-4de8-a4ed-63c59dc9ad7d response_code = 200 uri = /crucible/0/regions
672026-07-02T04:28:05.934ZINFOcrucible-agent (dropshot): request completed latency_us = 74 local_addr = [fd00:1122:3344:101::15]:32345 method = GET remote_addr = [fd00:1122:3344:101::b]:34188 req_id = 62d7c0cf-589c-4532-9558-12f1e72c64de response_code = 200 uri = /crucible/0/regions/71d1399d-865c-46b5-b1ca-df518690829a
682026-07-02T04:28:05.938ZINFOcrucible-agent (datafile): path is "/data/regions/71d1399d-865c-46b5-b1ca-df518690829a"
692026-07-02T04:28:05.948ZINFOcrucible-agent (datafile): dataset is oxp_b8508b78-4ea7-43a3-8927-5a33e998cf65/crucible/regions/71d1399d-865c-46b5-b1ca-df518690829a
702026-07-02T04:28:05.956ZINFOcrucible-agent (datafile): region 71d1399d-865c-46b5-b1ca-df518690829a state: Created -> Tombstoned
712026-07-02T04:28:05.956ZINFOcrucible-agent (worker): applying SMF actions before removal...
722026-07-02T04:28:05.956ZINFOcrucible-agent (dropshot): request completed latency_us = 18403 local_addr = [fd00:1122:3344:101::15]:32345 method = DELETE remote_addr = [fd00:1122:3344:101::b]:34188 req_id = ac0e0fe9-0373-408a-9dfc-b84bb0e43088 response_code = 204 uri = /crucible/0/regions/71d1399d-865c-46b5-b1ca-df518690829a
732026-07-02T04:28:05.957ZINFOcrucible-agent (worker): disabling downstairs instance: downstairs-71d1399d-865c-46b5-b1ca-df518690829a (instance states: (Some(Online), None))
742026-07-02T04:28:05.960ZINFOcrucible-agent (worker): existing directory value /dev/null does not match /data/regions/8ec8ced8-bd38-4fab-b716-e4a25108daad
752026-07-02T04:28:05.960ZINFOcrucible-agent (worker): existing port value 0 does not match 19000
762026-07-02T04:28:05.960ZINFOcrucible-agent (dropshot): request completed latency_us = 35 local_addr = [fd00:1122:3344:101::15]:32345 method = GET remote_addr = [fd00:1122:3344:101::b]:34188 req_id = a7a177f2-54e2-43f2-bd97-16bf877bc09c response_code = 200 uri = /crucible/0/regions/71d1399d-865c-46b5-b1ca-df518690829a
772026-07-02T04:28:05.960ZINFOcrucible-agent (worker): existing address value 0.0.0.0 does not match fd00:1122:3344:101::15
782026-07-02T04:28:05.960ZINFOcrucible-agent (worker): reconfig required
792026-07-02T04:28:05.960ZINFOcrucible-agent (worker): using existing config property group
802026-07-02T04:28:05.960ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-8ec8ced8-bd38-4fab-b716-e4a25108daad
812026-07-02T04:28:05.960ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/8ec8ced8-bd38-4fab-b716-e4a25108daad
822026-07-02T04:28:05.960ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
832026-07-02T04:28:05.960ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::15
842026-07-02T04:28:05.960ZINFOcrucible-agent (worker): commit
852026-07-02T04:28:05.962ZINFOcrucible-agent (worker): ok!
862026-07-02T04:28:05.962ZINFOcrucible-agent (worker): SMF ok!
872026-07-02T04:28:05.971ZINFOcrucible-agent (worker): deleting zfs dataset "oxp_b8508b78-4ea7-43a3-8927-5a33e998cf65/crucible/regions/71d1399d-865c-46b5-b1ca-df518690829a" region = 71d1399d-865c-46b5-b1ca-df518690829a
882026-07-02T04:28:06.011ZINFOcrucible-agent (datafile): region 71d1399d-865c-46b5-b1ca-df518690829a state: Tombstoned -> Destroyed
892026-07-02T04:28:06.113ZINFOcrucible-agent (dropshot): request completed latency_us = 48 local_addr = [fd00:1122:3344:101::15]:32345 method = GET remote_addr = [fd00:1122:3344:101::b]:34188 req_id = 391287ed-6634-4c99-99e9-21ead64d9bb3 response_code = 200 uri = /crucible/0/regions/71d1399d-865c-46b5-b1ca-df518690829a