Max level shown:
1 [ Mar 10 17:17:00 Enabled. ]
2 [ Mar 10 17:17:00 Rereading configuration. ]
3 [ Mar 10 17:17:02 Rereading configuration. ]
4 [ Mar 10 17:17:08 Executing start method ("/opt/oxide/crucible/bin/crucible-agent run -D /opt/oxide/crucible/bin/crucible-downstairs --dataset oxp_78d3a28c-d3c3-4210-9456-e2684c7a4890/crucible -l [fd00:1122:3344:101::17]:32345 -P 19000 -p downstairs -s snapshot"). ]
5 note: configured to log to "/dev/stdout"
62026-03-10T17:17:08.880ZINFOcrucible-agent: Crucible Version: Crucible Version: 0.0.1 Commit SHA: 7103cd3a3d7b0112d2949dd135db06fef0c156bb Commit timestamp: 2025-11-10T22:24:53.000000000Z branch: main rustc: 1.90.0 stable x86_64-unknown-illumos Cargo: x86_64-unknown-illumos Debug: false Opt level: 3
72026-03-10T17:17:08.881ZINFOcrucible-agent: dataset: "oxp_78d3a28c-d3c3-4210-9456-e2684c7a4890/crucible"
82026-03-10T17:17:08.881ZINFOcrucible-agent: listen IP: [fd00:1122:3344:101::17]:32345
92026-03-10T17:17:08.881ZINFOcrucible-agent: SMF instance name downstairs_prefix: "downstairs"
102026-03-10T17:17:08.895ZINFOcrucible-agent (datafile): Using conf_path:"/data/crucible.json"
112026-03-10T17:17:09.015ZINFOcrucible-agent (dropshot): listening local_addr = [fd00:1122:3344:101::17]:32345
122026-03-10T17:17:34.966ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::17]:32345 remote_addr = [fd00:1122:3344:101::c]:42243
132026-03-10T17:17:34.994ZINFOcrucible-agent (datafile): region f827cf6b-da29-41bc-b57d-afb1fde9226d state: Requested
142026-03-10T17:17:34.997ZINFOcrucible-agent (worker): Region size:1073741824 reservation:1342177280 quota:3221225472
152026-03-10T17:17:34.997ZINFOcrucible-agent (dropshot): request completed latency_us = 16801 local_addr = [fd00:1122:3344:101::17]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:42243 req_id = 447fd9b5-75a1-448f-8e32-eca5997f509c response_code = 200 uri = /crucible/0/regions
162026-03-10T17:17:35.010ZINFOcrucible-agent (worker): zfs set reservation of 1342177280 for oxp_78d3a28c-d3c3-4210-9456-e2684c7a4890/crucible/regions/f827cf6b-da29-41bc-b57d-afb1fde9226d
172026-03-10T17:17:35.010ZINFOcrucible-agent (worker): zfs set quota of 3221225472 for oxp_78d3a28c-d3c3-4210-9456-e2684c7a4890/crucible/regions/f827cf6b-da29-41bc-b57d-afb1fde9226d
182026-03-10T17:17:35.104ZINFOcrucible-agent (worker): creating region Region { id: RegionId("f827cf6b-da29-41bc-b57d-afb1fde9226d"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 16, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/f827cf6b-da29-41bc-b57d-afb1fde9226d" region = f827cf6b-da29-41bc-b57d-afb1fde9226d
192026-03-10T17:17:35.106ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "f827cf6b-da29-41bc-b57d-afb1fde9226d" "--data" "/data/regions/f827cf6b-da29-41bc-b57d-afb1fde9226d" "--block-size" "512" "--extent-size" "131072" "--extent-count" "16" "--encrypted" region = f827cf6b-da29-41bc-b57d-afb1fde9226d
202026-03-10T17:17:35.172ZINFOcrucible-agent (dropshot): request completed latency_us = 76 local_addr = [fd00:1122:3344:101::17]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:42243 req_id = 1ad96e41-56ea-4f26-9a67-37558c0f651c response_code = 200 uri = /crucible/0/regions
212026-03-10T17:17:35.493ZINFOcrucible-agent (worker): region files created ok region = f827cf6b-da29-41bc-b57d-afb1fde9226d
222026-03-10T17:17:35.493ZINFOcrucible-agent (datafile): region f827cf6b-da29-41bc-b57d-afb1fde9226d state: Requested -> Created
232026-03-10T17:17:35.493ZINFOcrucible-agent (worker): applying SMF actions post create...
242026-03-10T17:17:35.494ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-f827cf6b-da29-41bc-b57d-afb1fde9226d
252026-03-10T17:17:35.509ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-f827cf6b-da29-41bc-b57d-afb1fde9226d
262026-03-10T17:17:35.529ZINFOcrucible-agent (worker): creating config property group
272026-03-10T17:17:35.534ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-f827cf6b-da29-41bc-b57d-afb1fde9226d
282026-03-10T17:17:35.535ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/f827cf6b-da29-41bc-b57d-afb1fde9226d
292026-03-10T17:17:35.535ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
302026-03-10T17:17:35.535ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::17
312026-03-10T17:17:35.535ZINFOcrucible-agent (worker): commit
322026-03-10T17:17:35.540ZINFOcrucible-agent (worker): ok!
332026-03-10T17:17:35.547ZINFOcrucible-agent (worker): SMF ok!
342026-03-10T17:17:35.684ZINFOcrucible-agent (dropshot): request completed latency_us = 71 local_addr = [fd00:1122:3344:101::17]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:42243 req_id = 1bae6963-b081-45d5-a24c-ac5a672d7038 response_code = 200 uri = /crucible/0/regions
352026-03-10T17:17:37.065ZINFOcrucible-agent (dropshot): request completed latency_us = 88 local_addr = [fd00:1122:3344:101::17]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:42243 req_id = 6675573a-2f58-4a19-8420-5b41070d61f4 response_code = 200 uri = /crucible/0/regions/f827cf6b-da29-41bc-b57d-afb1fde9226d
362026-03-10T17:17:37.068ZINFOcrucible-agent (datafile): path is "/data/regions/f827cf6b-da29-41bc-b57d-afb1fde9226d"
372026-03-10T17:17:37.077ZINFOcrucible-agent (datafile): dataset is oxp_78d3a28c-d3c3-4210-9456-e2684c7a4890/crucible/regions/f827cf6b-da29-41bc-b57d-afb1fde9226d
382026-03-10T17:17:37.084ZINFOcrucible-agent (datafile): region f827cf6b-da29-41bc-b57d-afb1fde9226d state: Created -> Tombstoned
392026-03-10T17:17:37.084ZINFOcrucible-agent (worker): applying SMF actions before removal...
402026-03-10T17:17:37.084ZINFOcrucible-agent (dropshot): request completed latency_us = 16115 local_addr = [fd00:1122:3344:101::17]:32345 method = DELETE remote_addr = [fd00:1122:3344:101::c]:42243 req_id = 0b5feee6-338a-4de2-b8d1-0c3ce43e9a6e response_code = 204 uri = /crucible/0/regions/f827cf6b-da29-41bc-b57d-afb1fde9226d
412026-03-10T17:17:37.085ZINFOcrucible-agent (worker): disabling downstairs instance: downstairs-f827cf6b-da29-41bc-b57d-afb1fde9226d (instance states: (Some(Online), None))
422026-03-10T17:17:37.087ZINFOcrucible-agent (dropshot): request completed latency_us = 32 local_addr = [fd00:1122:3344:101::17]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:42243 req_id = 26805552-2c13-4408-a9e1-4b4eb2a1bc31 response_code = 200 uri = /crucible/0/regions/f827cf6b-da29-41bc-b57d-afb1fde9226d
432026-03-10T17:17:37.102ZINFOcrucible-agent (worker): SMF ok!
442026-03-10T17:17:37.111ZINFOcrucible-agent (worker): deleting zfs dataset "oxp_78d3a28c-d3c3-4210-9456-e2684c7a4890/crucible/regions/f827cf6b-da29-41bc-b57d-afb1fde9226d" region = f827cf6b-da29-41bc-b57d-afb1fde9226d
452026-03-10T17:17:37.152ZINFOcrucible-agent (dropshot): request completed latency_us = 56 local_addr = [fd00:1122:3344:101::17]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:42243 req_id = 64fac788-398d-43ce-936e-80a0ee6cb97b response_code = 200 uri = /crucible/0/regions/f827cf6b-da29-41bc-b57d-afb1fde9226d
462026-03-10T17:17:37.235ZINFOcrucible-agent (dropshot): request completed latency_us = 44 local_addr = [fd00:1122:3344:101::17]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:42243 req_id = 16a94da9-cb5d-4097-af05-b6f7b0a0edb5 response_code = 200 uri = /crucible/0/regions/f827cf6b-da29-41bc-b57d-afb1fde9226d
472026-03-10T17:17:37.318ZINFOcrucible-agent (datafile): region f827cf6b-da29-41bc-b57d-afb1fde9226d state: Tombstoned -> Destroyed
482026-03-10T17:17:37.327ZINFOcrucible-agent (dropshot): request completed latency_us = 35 local_addr = [fd00:1122:3344:101::17]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:42243 req_id = ff43e8a2-8e0d-4433-ac2e-be2b4d5e9a05 response_code = 200 uri = /crucible/0/regions/f827cf6b-da29-41bc-b57d-afb1fde9226d
492026-03-10T17:28:45.750ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::17]:32345 remote_addr = [fd00:1122:3344:101::c]:40269
502026-03-10T17:28:45.750ZINFOcrucible-agent (datafile): region b99c0ab8-b993-43bb-b23f-a945f85dd38c state: Requested
512026-03-10T17:28:45.750ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
522026-03-10T17:28:45.750ZINFOcrucible-agent (dropshot): request completed latency_us = 634 local_addr = [fd00:1122:3344:101::17]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:40269 req_id = 216a48af-9d4e-4ddc-a9a5-2b431289aef6 response_code = 200 uri = /crucible/0/regions
532026-03-10T17:28:45.758ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_78d3a28c-d3c3-4210-9456-e2684c7a4890/crucible/regions/b99c0ab8-b993-43bb-b23f-a945f85dd38c
542026-03-10T17:28:45.758ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_78d3a28c-d3c3-4210-9456-e2684c7a4890/crucible/regions/b99c0ab8-b993-43bb-b23f-a945f85dd38c
552026-03-10T17:28:45.806ZINFOcrucible-agent (worker): creating region Region { id: RegionId("b99c0ab8-b993-43bb-b23f-a945f85dd38c"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/b99c0ab8-b993-43bb-b23f-a945f85dd38c" region = b99c0ab8-b993-43bb-b23f-a945f85dd38c
562026-03-10T17:28:45.806ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "b99c0ab8-b993-43bb-b23f-a945f85dd38c" "--data" "/data/regions/b99c0ab8-b993-43bb-b23f-a945f85dd38c" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = b99c0ab8-b993-43bb-b23f-a945f85dd38c
572026-03-10T17:28:45.947ZINFOcrucible-agent (dropshot): request completed latency_us = 62 local_addr = [fd00:1122:3344:101::17]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:40269 req_id = aa6f828e-843e-4fea-8af4-c03bc225611e response_code = 200 uri = /crucible/0/regions
582026-03-10T17:28:46.175ZINFOcrucible-agent (worker): region files created ok region = b99c0ab8-b993-43bb-b23f-a945f85dd38c
592026-03-10T17:28:46.175ZINFOcrucible-agent (datafile): region b99c0ab8-b993-43bb-b23f-a945f85dd38c state: Requested -> Created
602026-03-10T17:28:46.175ZINFOcrucible-agent (worker): applying SMF actions post create...
612026-03-10T17:28:46.176ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-b99c0ab8-b993-43bb-b23f-a945f85dd38c
622026-03-10T17:28:46.185ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-b99c0ab8-b993-43bb-b23f-a945f85dd38c
632026-03-10T17:28:46.209ZINFOcrucible-agent (worker): creating config property group
642026-03-10T17:28:46.212ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-b99c0ab8-b993-43bb-b23f-a945f85dd38c
652026-03-10T17:28:46.212ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/b99c0ab8-b993-43bb-b23f-a945f85dd38c
662026-03-10T17:28:46.212ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
672026-03-10T17:28:46.212ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::17
682026-03-10T17:28:46.212ZINFOcrucible-agent (worker): commit
692026-03-10T17:28:46.216ZINFOcrucible-agent (worker): ok!
702026-03-10T17:28:46.221ZINFOcrucible-agent (worker): SMF ok!
712026-03-10T17:28:46.318ZINFOcrucible-agent (dropshot): request completed latency_us = 66 local_addr = [fd00:1122:3344:101::17]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:40269 req_id = 154c7200-1ff8-41a7-89a8-4bc09c7565ff response_code = 200 uri = /crucible/0/regions
722026-03-10T17:28:52.467ZINFOcrucible-agent (datafile): region 571cd6cb-8e8b-469e-a5ca-ceb56c01eef3 state: Requested
732026-03-10T17:28:52.467ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
742026-03-10T17:28:52.467ZINFOcrucible-agent (dropshot): request completed latency_us = 219 local_addr = [fd00:1122:3344:101::17]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:40269 req_id = a4ab2c39-76c5-488f-b8cb-9adb5d3673ee response_code = 200 uri = /crucible/0/regions
752026-03-10T17:28:52.475ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_78d3a28c-d3c3-4210-9456-e2684c7a4890/crucible/regions/571cd6cb-8e8b-469e-a5ca-ceb56c01eef3
762026-03-10T17:28:52.475ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_78d3a28c-d3c3-4210-9456-e2684c7a4890/crucible/regions/571cd6cb-8e8b-469e-a5ca-ceb56c01eef3
772026-03-10T17:28:52.522ZINFOcrucible-agent (worker): creating region Region { id: RegionId("571cd6cb-8e8b-469e-a5ca-ceb56c01eef3"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19001, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/571cd6cb-8e8b-469e-a5ca-ceb56c01eef3" region = 571cd6cb-8e8b-469e-a5ca-ceb56c01eef3
782026-03-10T17:28:52.522ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "571cd6cb-8e8b-469e-a5ca-ceb56c01eef3" "--data" "/data/regions/571cd6cb-8e8b-469e-a5ca-ceb56c01eef3" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 571cd6cb-8e8b-469e-a5ca-ceb56c01eef3
792026-03-10T17:28:52.596ZINFOcrucible-agent (worker): region files created ok region = 571cd6cb-8e8b-469e-a5ca-ceb56c01eef3
802026-03-10T17:28:52.596ZINFOcrucible-agent (datafile): region 571cd6cb-8e8b-469e-a5ca-ceb56c01eef3 state: Requested -> Created
812026-03-10T17:28:52.597ZINFOcrucible-agent (worker): applying SMF actions post create...
822026-03-10T17:28:52.597ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-571cd6cb-8e8b-469e-a5ca-ceb56c01eef3
832026-03-10T17:28:52.606ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-571cd6cb-8e8b-469e-a5ca-ceb56c01eef3
842026-03-10T17:28:52.613ZINFOcrucible-agent (worker): creating config property group
852026-03-10T17:28:52.616ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-571cd6cb-8e8b-469e-a5ca-ceb56c01eef3
862026-03-10T17:28:52.616ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/571cd6cb-8e8b-469e-a5ca-ceb56c01eef3
872026-03-10T17:28:52.616ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
882026-03-10T17:28:52.616ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::17
892026-03-10T17:28:52.617ZINFOcrucible-agent (worker): commit
902026-03-10T17:28:52.620ZINFOcrucible-agent (worker): ok!
912026-03-10T17:28:52.630ZINFOcrucible-agent (worker): SMF ok!
922026-03-10T17:28:52.697ZINFOcrucible-agent (dropshot): request completed latency_us = 73 local_addr = [fd00:1122:3344:101::17]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:40269 req_id = 76c08f0d-e1d7-4e61-b248-22781a5d4e90 response_code = 200 uri = /crucible/0/regions
932026-03-10T17:29:19.487ZINFOcrucible-agent (dropshot): request completed latency_us = 70 local_addr = [fd00:1122:3344:101::17]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:40269 req_id = a1e463b2-a9e3-49c7-8e21-c8ae9a5044f9 response_code = 200 uri = /crucible/0/regions/571cd6cb-8e8b-469e-a5ca-ceb56c01eef3
942026-03-10T17:29:19.490ZINFOcrucible-agent (datafile): path is "/data/regions/571cd6cb-8e8b-469e-a5ca-ceb56c01eef3"
952026-03-10T17:29:19.501ZINFOcrucible-agent (datafile): dataset is oxp_78d3a28c-d3c3-4210-9456-e2684c7a4890/crucible/regions/571cd6cb-8e8b-469e-a5ca-ceb56c01eef3
962026-03-10T17:29:19.508ZINFOcrucible-agent (datafile): region 571cd6cb-8e8b-469e-a5ca-ceb56c01eef3 state: Created -> Tombstoned
972026-03-10T17:29:19.508ZINFOcrucible-agent (worker): applying SMF actions before removal...
982026-03-10T17:29:19.508ZINFOcrucible-agent (dropshot): request completed latency_us = 18712 local_addr = [fd00:1122:3344:101::17]:32345 method = DELETE remote_addr = [fd00:1122:3344:101::c]:40269 req_id = 704cc151-1843-4827-9a0b-0ac99f133e1d response_code = 204 uri = /crucible/0/regions/571cd6cb-8e8b-469e-a5ca-ceb56c01eef3
992026-03-10T17:29:19.510ZINFOcrucible-agent (worker): disabling downstairs instance: downstairs-571cd6cb-8e8b-469e-a5ca-ceb56c01eef3 (instance states: (Some(Online), None))
1002026-03-10T17:29:19.512ZINFOcrucible-agent (dropshot): request completed latency_us = 48 local_addr = [fd00:1122:3344:101::17]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:40269 req_id = 4abdf0d3-a83b-4132-81be-4a143d015229 response_code = 200 uri = /crucible/0/regions/571cd6cb-8e8b-469e-a5ca-ceb56c01eef3
1012026-03-10T17:29:19.529ZINFOcrucible-agent (worker): SMF ok!
1022026-03-10T17:29:19.537ZINFOcrucible-agent (worker): deleting zfs dataset "oxp_78d3a28c-d3c3-4210-9456-e2684c7a4890/crucible/regions/571cd6cb-8e8b-469e-a5ca-ceb56c01eef3" region = 571cd6cb-8e8b-469e-a5ca-ceb56c01eef3
1032026-03-10T17:29:19.622ZINFOcrucible-agent (datafile): region 571cd6cb-8e8b-469e-a5ca-ceb56c01eef3 state: Tombstoned -> Destroyed
1042026-03-10T17:29:19.627ZINFOcrucible-agent (dropshot): request completed latency_us = 46 local_addr = [fd00:1122:3344:101::17]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:40269 req_id = b1334c89-17a4-4438-817a-3b4179d2fc8e response_code = 200 uri = /crucible/0/regions/571cd6cb-8e8b-469e-a5ca-ceb56c01eef3