Max level shown:
1 [ May 1 17:47:08 Enabled. ]
2 [ May 1 17:47:08 Rereading configuration. ]
3 [ May 1 17:47:10 Rereading configuration. ]
4 [ May 1 17:47:14 Executing start method ("/opt/oxide/crucible/bin/crucible-agent run -D /opt/oxide/crucible/bin/crucible-downstairs --dataset oxp_6b66cf61-6fec-4687-9f98-c6973910c3a7/crucible -l [fd00:1122:3344:101::17]:32345 -P 19000 -p downstairs -s snapshot"). ]
5 note: configured to log to "/dev/stdout"
62026-05-01T17:47:14.260ZINFOcrucible-agent: Crucible Version: Crucible Version: 0.0.1 Commit SHA: ae1da83e66c648574827298f4bc444632bf4d047 Commit timestamp: 2026-03-31T21:17:13.000000000Z branch: main rustc: 1.90.0 stable x86_64-unknown-illumos Cargo: x86_64-unknown-illumos Debug: false Opt level: 3
72026-05-01T17:47:14.264ZINFOcrucible-agent: dataset: "oxp_6b66cf61-6fec-4687-9f98-c6973910c3a7/crucible"
82026-05-01T17:47:14.264ZINFOcrucible-agent: listen IP: [fd00:1122:3344:101::17]:32345
92026-05-01T17:47:14.264ZINFOcrucible-agent: SMF instance name downstairs_prefix: "downstairs"
102026-05-01T17:47:14.272ZINFOcrucible-agent (datafile): Using conf_path:"/data/crucible.json"
112026-05-01T17:47:14.356ZINFOcrucible-agent (dropshot): listening local_addr = [fd00:1122:3344:101::17]:32345
122026-05-01T17:58:34.215ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::17]:32345 remote_addr = [fd00:1122:3344:101::c]:56839
132026-05-01T17:58:34.269ZINFOcrucible-agent (datafile): region 7e57b96c-574e-4dde-b56e-db7709ffa813 state: Requested
142026-05-01T17:58:34.270ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
152026-05-01T17:58:34.270ZINFOcrucible-agent (dropshot): request completed latency_us = 27591 local_addr = [fd00:1122:3344:101::17]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:56839 req_id = fd80abe7-45ec-488a-a69a-1fa1d7a33347 response_code = 200 uri = /crucible/0/regions
162026-05-01T17:58:34.277ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_6b66cf61-6fec-4687-9f98-c6973910c3a7/crucible/regions/7e57b96c-574e-4dde-b56e-db7709ffa813
172026-05-01T17:58:34.277ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_6b66cf61-6fec-4687-9f98-c6973910c3a7/crucible/regions/7e57b96c-574e-4dde-b56e-db7709ffa813
182026-05-01T17:58:34.331ZINFOcrucible-agent (worker): creating region Region { id: RegionId("7e57b96c-574e-4dde-b56e-db7709ffa813"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/7e57b96c-574e-4dde-b56e-db7709ffa813" region = 7e57b96c-574e-4dde-b56e-db7709ffa813
192026-05-01T17:58:34.336ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "7e57b96c-574e-4dde-b56e-db7709ffa813" "--data" "/data/regions/7e57b96c-574e-4dde-b56e-db7709ffa813" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 7e57b96c-574e-4dde-b56e-db7709ffa813
202026-05-01T17:58:34.443ZINFOcrucible-agent (dropshot): request completed latency_us = 66 local_addr = [fd00:1122:3344:101::17]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:56839 req_id = 3feed581-2481-4f2a-93aa-5233a3fbf3ce response_code = 200 uri = /crucible/0/regions
212026-05-01T17:58:34.636ZINFOcrucible-agent (worker): region files created ok region = 7e57b96c-574e-4dde-b56e-db7709ffa813
222026-05-01T17:58:34.636ZINFOcrucible-agent (datafile): region 7e57b96c-574e-4dde-b56e-db7709ffa813 state: Requested -> Created
232026-05-01T17:58:34.636ZINFOcrucible-agent (worker): applying SMF actions post create...
242026-05-01T17:58:34.636ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-7e57b96c-574e-4dde-b56e-db7709ffa813
252026-05-01T17:58:34.646ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-7e57b96c-574e-4dde-b56e-db7709ffa813
262026-05-01T17:58:34.667ZINFOcrucible-agent (worker): creating config property group
272026-05-01T17:58:34.670ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-7e57b96c-574e-4dde-b56e-db7709ffa813
282026-05-01T17:58:34.670ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/7e57b96c-574e-4dde-b56e-db7709ffa813
292026-05-01T17:58:34.673ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
302026-05-01T17:58:34.673ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::17
312026-05-01T17:58:34.673ZINFOcrucible-agent (worker): commit
322026-05-01T17:58:34.689ZINFOcrucible-agent (worker): ok!
332026-05-01T17:58:34.692ZINFOcrucible-agent (worker): SMF ok!
342026-05-01T17:58:34.942ZINFOcrucible-agent (dropshot): request completed latency_us = 84 local_addr = [fd00:1122:3344:101::17]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:56839 req_id = 72feb40d-ee4e-4adf-886f-e3b68a6f71c6 response_code = 200 uri = /crucible/0/regions
352026-05-01T17:58:40.565ZINFOcrucible-agent (datafile): region 316899de-4b14-4ab0-9f6a-6a946da72258 state: Requested
362026-05-01T17:58:40.565ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
372026-05-01T17:58:40.565ZINFOcrucible-agent (dropshot): request completed latency_us = 233 local_addr = [fd00:1122:3344:101::17]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:56839 req_id = fafb78e9-514f-4c07-a70e-635cf065ff42 response_code = 200 uri = /crucible/0/regions
382026-05-01T17:58:40.572ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_6b66cf61-6fec-4687-9f98-c6973910c3a7/crucible/regions/316899de-4b14-4ab0-9f6a-6a946da72258
392026-05-01T17:58:40.572ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_6b66cf61-6fec-4687-9f98-c6973910c3a7/crucible/regions/316899de-4b14-4ab0-9f6a-6a946da72258
402026-05-01T17:58:40.621ZINFOcrucible-agent (worker): creating region Region { id: RegionId("316899de-4b14-4ab0-9f6a-6a946da72258"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19001, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/316899de-4b14-4ab0-9f6a-6a946da72258" region = 316899de-4b14-4ab0-9f6a-6a946da72258
412026-05-01T17:58:40.622ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "316899de-4b14-4ab0-9f6a-6a946da72258" "--data" "/data/regions/316899de-4b14-4ab0-9f6a-6a946da72258" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 316899de-4b14-4ab0-9f6a-6a946da72258
422026-05-01T17:58:40.699ZINFOcrucible-agent (worker): region files created ok region = 316899de-4b14-4ab0-9f6a-6a946da72258
432026-05-01T17:58:40.699ZINFOcrucible-agent (datafile): region 316899de-4b14-4ab0-9f6a-6a946da72258 state: Requested -> Created
442026-05-01T17:58:40.699ZINFOcrucible-agent (worker): applying SMF actions post create...
452026-05-01T17:58:40.700ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-316899de-4b14-4ab0-9f6a-6a946da72258
462026-05-01T17:58:40.703ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-316899de-4b14-4ab0-9f6a-6a946da72258
472026-05-01T17:58:40.713ZINFOcrucible-agent (worker): creating config property group
482026-05-01T17:58:40.718ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-316899de-4b14-4ab0-9f6a-6a946da72258
492026-05-01T17:58:40.719ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/316899de-4b14-4ab0-9f6a-6a946da72258
502026-05-01T17:58:40.719ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
512026-05-01T17:58:40.719ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::17
522026-05-01T17:58:40.719ZINFOcrucible-agent (worker): commit
532026-05-01T17:58:40.724ZINFOcrucible-agent (worker): ok!
542026-05-01T17:58:40.729ZINFOcrucible-agent (worker): existing directory value /dev/null does not match /data/regions/7e57b96c-574e-4dde-b56e-db7709ffa813
552026-05-01T17:58:40.729ZINFOcrucible-agent (worker): existing port value 0 does not match 19000
562026-05-01T17:58:40.729ZINFOcrucible-agent (worker): existing address value 0.0.0.0 does not match fd00:1122:3344:101::17
572026-05-01T17:58:40.729ZINFOcrucible-agent (worker): reconfig required
582026-05-01T17:58:40.729ZINFOcrucible-agent (worker): using existing config property group
592026-05-01T17:58:40.729ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-7e57b96c-574e-4dde-b56e-db7709ffa813
602026-05-01T17:58:40.729ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/7e57b96c-574e-4dde-b56e-db7709ffa813
612026-05-01T17:58:40.729ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
622026-05-01T17:58:40.729ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::17
632026-05-01T17:58:40.729ZINFOcrucible-agent (worker): commit
642026-05-01T17:58:40.737ZINFOcrucible-agent (worker): ok!
652026-05-01T17:58:40.738ZINFOcrucible-agent (worker): SMF ok!
662026-05-01T17:58:40.801ZINFOcrucible-agent (dropshot): request completed latency_us = 61 local_addr = [fd00:1122:3344:101::17]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:56839 req_id = ecdb3a33-a73d-4007-8ffa-6ffa8ba33e7a response_code = 200 uri = /crucible/0/regions
672026-05-01T17:59:06.656ZINFOcrucible-agent (dropshot): request completed latency_us = 2135 local_addr = [fd00:1122:3344:101::17]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:56839 req_id = 13b2ec24-c395-469e-969a-28c646e569c5 response_code = 200 uri = /crucible/0/regions/316899de-4b14-4ab0-9f6a-6a946da72258
682026-05-01T17:59:06.659ZINFOcrucible-agent (datafile): path is "/data/regions/316899de-4b14-4ab0-9f6a-6a946da72258"
692026-05-01T17:59:06.668ZINFOcrucible-agent (datafile): dataset is oxp_6b66cf61-6fec-4687-9f98-c6973910c3a7/crucible/regions/316899de-4b14-4ab0-9f6a-6a946da72258
702026-05-01T17:59:06.675ZINFOcrucible-agent (datafile): region 316899de-4b14-4ab0-9f6a-6a946da72258 state: Created -> Tombstoned
712026-05-01T17:59:06.675ZINFOcrucible-agent (dropshot): request completed latency_us = 16058 local_addr = [fd00:1122:3344:101::17]:32345 method = DELETE remote_addr = [fd00:1122:3344:101::c]:56839 req_id = 10ce7cff-d6d6-43a0-b283-01143639cd2e response_code = 204 uri = /crucible/0/regions/316899de-4b14-4ab0-9f6a-6a946da72258
722026-05-01T17:59:06.675ZINFOcrucible-agent (worker): applying SMF actions before removal...
732026-05-01T17:59:06.676ZINFOcrucible-agent (worker): disabling downstairs instance: downstairs-316899de-4b14-4ab0-9f6a-6a946da72258 (instance states: (Some(Online), None))
742026-05-01T17:59:06.678ZINFOcrucible-agent (dropshot): request completed latency_us = 31 local_addr = [fd00:1122:3344:101::17]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:56839 req_id = b1fc72e2-af4c-4451-9951-389e47117ce8 response_code = 200 uri = /crucible/0/regions/316899de-4b14-4ab0-9f6a-6a946da72258
752026-05-01T17:59:06.689ZINFOcrucible-agent (worker): existing directory value /dev/null does not match /data/regions/7e57b96c-574e-4dde-b56e-db7709ffa813
762026-05-01T17:59:06.689ZINFOcrucible-agent (worker): existing port value 0 does not match 19000
772026-05-01T17:59:06.689ZINFOcrucible-agent (worker): existing address value 0.0.0.0 does not match fd00:1122:3344:101::17
782026-05-01T17:59:06.689ZINFOcrucible-agent (worker): reconfig required
792026-05-01T17:59:06.689ZINFOcrucible-agent (worker): using existing config property group
802026-05-01T17:59:06.689ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-7e57b96c-574e-4dde-b56e-db7709ffa813
812026-05-01T17:59:06.689ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/7e57b96c-574e-4dde-b56e-db7709ffa813
822026-05-01T17:59:06.689ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
832026-05-01T17:59:06.689ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::17
842026-05-01T17:59:06.689ZINFOcrucible-agent (worker): commit
852026-05-01T17:59:06.693ZINFOcrucible-agent (worker): ok!
862026-05-01T17:59:06.693ZINFOcrucible-agent (worker): SMF ok!
872026-05-01T17:59:06.701ZINFOcrucible-agent (worker): deleting zfs dataset "oxp_6b66cf61-6fec-4687-9f98-c6973910c3a7/crucible/regions/316899de-4b14-4ab0-9f6a-6a946da72258" region = 316899de-4b14-4ab0-9f6a-6a946da72258
882026-05-01T17:59:06.771ZINFOcrucible-agent (datafile): region 316899de-4b14-4ab0-9f6a-6a946da72258 state: Tombstoned -> Destroyed
892026-05-01T17:59:06.792ZINFOcrucible-agent (dropshot): request completed latency_us = 54 local_addr = [fd00:1122:3344:101::17]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:56839 req_id = ea650d44-c0cf-4d4d-8509-83c91c808c72 response_code = 200 uri = /crucible/0/regions/316899de-4b14-4ab0-9f6a-6a946da72258