Max level shown:
1 [ Mar 4 04:29:57 Enabled. ]
2 [ Mar 4 04:29:57 Rereading configuration. ]
3 [ Mar 4 04:30:00 Rereading configuration. ]
4 [ Mar 4 04:30:05 Executing start method ("/opt/oxide/crucible/bin/crucible-agent run -D /opt/oxide/crucible/bin/crucible-downstairs --dataset oxp_2a3e78bc-d612-43fc-a619-b5e212bdc7e5/crucible -l [fd00:1122:3344:101::1a]:32345 -P 19000 -p downstairs -s snapshot"). ]
5 note: configured to log to "/dev/stdout"
62026-03-04T04:30:05.789ZINFOcrucible-agent: Crucible Version: Crucible Version: 0.0.1 Commit SHA: 7103cd3a3d7b0112d2949dd135db06fef0c156bb Commit timestamp: 2025-11-10T22:24:53.000000000Z branch: main rustc: 1.90.0 stable x86_64-unknown-illumos Cargo: x86_64-unknown-illumos Debug: false Opt level: 3
72026-03-04T04:30:05.790ZINFOcrucible-agent: dataset: "oxp_2a3e78bc-d612-43fc-a619-b5e212bdc7e5/crucible"
82026-03-04T04:30:05.790ZINFOcrucible-agent: listen IP: [fd00:1122:3344:101::1a]:32345
92026-03-04T04:30:05.790ZINFOcrucible-agent: SMF instance name downstairs_prefix: "downstairs"
102026-03-04T04:30:05.804ZINFOcrucible-agent (datafile): Using conf_path:"/data/crucible.json"
112026-03-04T04:30:05.925ZINFOcrucible-agent (dropshot): listening local_addr = [fd00:1122:3344:101::1a]:32345
122026-03-04T04:30:44.369ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::1a]:32345 remote_addr = [fd00:1122:3344:101::c]:64881
132026-03-04T04:30:44.394ZINFOcrucible-agent (datafile): region 4ab05022-f5fc-42d2-ada3-5a6e2547549a state: Requested
142026-03-04T04:30:44.397ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
152026-03-04T04:30:44.397ZINFOcrucible-agent (dropshot): request completed latency_us = 14289 local_addr = [fd00:1122:3344:101::1a]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:64881 req_id = 542ddad5-185f-4a0f-b9a4-e0d4585a0449 response_code = 200 uri = /crucible/0/regions
162026-03-04T04:30:44.406ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_2a3e78bc-d612-43fc-a619-b5e212bdc7e5/crucible/regions/4ab05022-f5fc-42d2-ada3-5a6e2547549a
172026-03-04T04:30:44.406ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_2a3e78bc-d612-43fc-a619-b5e212bdc7e5/crucible/regions/4ab05022-f5fc-42d2-ada3-5a6e2547549a
182026-03-04T04:30:44.477ZINFOcrucible-agent (worker): creating region Region { id: RegionId("4ab05022-f5fc-42d2-ada3-5a6e2547549a"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/4ab05022-f5fc-42d2-ada3-5a6e2547549a" region = 4ab05022-f5fc-42d2-ada3-5a6e2547549a
192026-03-04T04:30:44.480ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "4ab05022-f5fc-42d2-ada3-5a6e2547549a" "--data" "/data/regions/4ab05022-f5fc-42d2-ada3-5a6e2547549a" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 4ab05022-f5fc-42d2-ada3-5a6e2547549a
202026-03-04T04:30:44.563ZINFOcrucible-agent (dropshot): request completed latency_us = 65 local_addr = [fd00:1122:3344:101::1a]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:64881 req_id = 62821c11-2887-46d3-be46-4f7b086545bb response_code = 200 uri = /crucible/0/regions
212026-03-04T04:30:44.852ZINFOcrucible-agent (worker): region files created ok region = 4ab05022-f5fc-42d2-ada3-5a6e2547549a
222026-03-04T04:30:44.852ZINFOcrucible-agent (datafile): region 4ab05022-f5fc-42d2-ada3-5a6e2547549a state: Requested -> Created
232026-03-04T04:30:44.852ZINFOcrucible-agent (worker): applying SMF actions post create...
242026-03-04T04:30:44.852ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-4ab05022-f5fc-42d2-ada3-5a6e2547549a
252026-03-04T04:30:44.859ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-4ab05022-f5fc-42d2-ada3-5a6e2547549a
262026-03-04T04:30:44.876ZINFOcrucible-agent (worker): creating config property group
272026-03-04T04:30:44.881ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-4ab05022-f5fc-42d2-ada3-5a6e2547549a
282026-03-04T04:30:44.882ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/4ab05022-f5fc-42d2-ada3-5a6e2547549a
292026-03-04T04:30:44.882ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
302026-03-04T04:30:44.882ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::1a
312026-03-04T04:30:44.882ZINFOcrucible-agent (worker): commit
322026-03-04T04:30:44.887ZINFOcrucible-agent (worker): ok!
332026-03-04T04:30:44.893ZINFOcrucible-agent (worker): SMF ok!
342026-03-04T04:30:45.180ZINFOcrucible-agent (dropshot): request completed latency_us = 86 local_addr = [fd00:1122:3344:101::1a]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:64881 req_id = f53f930f-9838-462e-af7e-ac59f6558bc5 response_code = 200 uri = /crucible/0/regions
352026-03-04T04:42:12.604ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::1a]:32345 remote_addr = [fd00:1122:3344:101::c]:38152
362026-03-04T04:42:12.607ZINFOcrucible-agent (dropshot): request completed latency_us = 62 local_addr = [fd00:1122:3344:101::1a]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:38152 req_id = a965282a-1f9e-4252-b2da-331fa8208aef response_code = 200 uri = /crucible/0/regions/4ab05022-f5fc-42d2-ada3-5a6e2547549a
372026-03-04T04:42:12.608ZINFOcrucible-agent (datafile): path is "/data/regions/4ab05022-f5fc-42d2-ada3-5a6e2547549a"
382026-03-04T04:42:12.614ZINFOcrucible-agent (datafile): dataset is oxp_2a3e78bc-d612-43fc-a619-b5e212bdc7e5/crucible/regions/4ab05022-f5fc-42d2-ada3-5a6e2547549a
392026-03-04T04:42:12.621ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_2a3e78bc-d612-43fc-a619-b5e212bdc7e5/crucible/regions/4ab05022-f5fc-42d2-ada3-5a6e2547549a@c83ec41c-5341-4ce4-8349-d181265001b0
402026-03-04T04:42:12.621ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_2a3e78bc-d612-43fc-a619-b5e212bdc7e5/crucible/regions/4ab05022-f5fc-42d2-ada3-5a6e2547549a", "c83ec41c-5341-4ce4-8349-d181265001b0"]
412026-03-04T04:42:12.630ZINFOcrucible-agent (dropshot): request completed latency_us = 22469 local_addr = [fd00:1122:3344:101::1a]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:38152 req_id = cc1c0455-4a73-436c-bb5d-c1b320c36254 response_code = 200 uri = /crucible/0/regions/4ab05022-f5fc-42d2-ada3-5a6e2547549a/snapshots/c83ec41c-5341-4ce4-8349-d181265001b0
422026-03-04T04:42:12.634ZINFOcrucible-agent (datafile): path is "/data/regions/4ab05022-f5fc-42d2-ada3-5a6e2547549a"
432026-03-04T04:42:12.640ZINFOcrucible-agent (datafile): dataset is oxp_2a3e78bc-d612-43fc-a619-b5e212bdc7e5/crucible/regions/4ab05022-f5fc-42d2-ada3-5a6e2547549a
442026-03-04T04:42:12.646ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_2a3e78bc-d612-43fc-a619-b5e212bdc7e5/crucible/regions/4ab05022-f5fc-42d2-ada3-5a6e2547549a@c83ec41c-5341-4ce4-8349-d181265001b0
452026-03-04T04:42:12.646ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_2a3e78bc-d612-43fc-a619-b5e212bdc7e5/crucible/regions/4ab05022-f5fc-42d2-ada3-5a6e2547549a", "c83ec41c-5341-4ce4-8349-d181265001b0"]
462026-03-04T04:42:12.653ZINFOcrucible-agent (datafile): requesting running snapshot 4ab05022-f5fc-42d2-ada3-5a6e2547549a-c83ec41c-5341-4ce4-8349-d181265001b0 state: Requested
472026-03-04T04:42:12.654ZINFOcrucible-agent (dropshot): request completed latency_us = 19900 local_addr = [fd00:1122:3344:101::1a]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:38152 req_id = 0248b422-d073-4217-88bd-b44a40873415 response_code = 200 uri = /crucible/0/regions/4ab05022-f5fc-42d2-ada3-5a6e2547549a/snapshots/c83ec41c-5341-4ce4-8349-d181265001b0/run
482026-03-04T04:42:12.654ZINFOcrucible-agent (worker): applying SMF actions for region 4ab05022-f5fc-42d2-ada3-5a6e2547549a running snapshot c83ec41c-5341-4ce4-8349-d181265001b0 (state Requested)...
492026-03-04T04:42:12.678ZINFOcrucible-agent (worker): creating missing snapshot instance snapshot-4ab05022-f5fc-42d2-ada3-5a6e2547549a-c83ec41c-5341-4ce4-8349-d181265001b0
502026-03-04T04:42:12.692ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:snapshot-4ab05022-f5fc-42d2-ada3-5a6e2547549a-c83ec41c-5341-4ce4-8349-d181265001b0
512026-03-04T04:42:12.720ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:snapshot-4ab05022-f5fc-42d2-ada3-5a6e2547549a-c83ec41c-5341-4ce4-8349-d181265001b0
522026-03-04T04:42:12.721ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/4ab05022-f5fc-42d2-ada3-5a6e2547549a/.zfs/snapshot/c83ec41c-5341-4ce4-8349-d181265001b0
532026-03-04T04:42:12.721ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
542026-03-04T04:42:12.721ZINFOcrucible-agent (worker): ensure mode SCF_TYPE_ASTRING ro
552026-03-04T04:42:12.721ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::1a
562026-03-04T04:42:12.721ZINFOcrucible-agent (worker): commit
572026-03-04T04:42:12.730ZINFOcrucible-agent (worker): ok!
582026-03-04T04:42:12.741ZINFOcrucible-agent (worker): SMF ok!
592026-03-04T04:42:12.741ZINFOcrucible-agent (datafile): region 4ab05022-f5fc-42d2-ada3-5a6e2547549a running snapshot c83ec41c-5341-4ce4-8349-d181265001b0 state: Requested -> Created
602026-03-04T04:42:12.951ZINFOcrucible-agent (datafile): path is "/data/regions/4ab05022-f5fc-42d2-ada3-5a6e2547549a"
612026-03-04T04:42:12.957ZINFOcrucible-agent (datafile): dataset is oxp_2a3e78bc-d612-43fc-a619-b5e212bdc7e5/crucible/regions/4ab05022-f5fc-42d2-ada3-5a6e2547549a
622026-03-04T04:42:12.962ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_2a3e78bc-d612-43fc-a619-b5e212bdc7e5/crucible/regions/4ab05022-f5fc-42d2-ada3-5a6e2547549a@c83ec41c-5341-4ce4-8349-d181265001b0
632026-03-04T04:42:12.962ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_2a3e78bc-d612-43fc-a619-b5e212bdc7e5/crucible/regions/4ab05022-f5fc-42d2-ada3-5a6e2547549a", "c83ec41c-5341-4ce4-8349-d181265001b0"]
642026-03-04T04:42:12.968ZINFOcrucible-agent (dropshot): request completed latency_us = 17184 local_addr = [fd00:1122:3344:101::1a]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:38152 req_id = 01a17edd-c30e-4c1b-9c8d-db4a96c66255 response_code = 200 uri = /crucible/0/regions/4ab05022-f5fc-42d2-ada3-5a6e2547549a/snapshots/c83ec41c-5341-4ce4-8349-d181265001b0/run
652026-03-04T04:42:18.379ZINFOcrucible-agent (datafile): region 1fbfc2d8-2775-47c1-b8f9-4744d6a0f6d7 state: Requested
662026-03-04T04:42:18.379ZINFOcrucible-agent (dropshot): request completed latency_us = 249 local_addr = [fd00:1122:3344:101::1a]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:38152 req_id = bfdce96f-bff8-4e6a-addc-fff1e6aa6664 response_code = 200 uri = /crucible/0/regions
672026-03-04T04:42:18.379ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
682026-03-04T04:42:18.386ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_2a3e78bc-d612-43fc-a619-b5e212bdc7e5/crucible/regions/1fbfc2d8-2775-47c1-b8f9-4744d6a0f6d7
692026-03-04T04:42:18.386ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_2a3e78bc-d612-43fc-a619-b5e212bdc7e5/crucible/regions/1fbfc2d8-2775-47c1-b8f9-4744d6a0f6d7
702026-03-04T04:42:18.454ZINFOcrucible-agent (worker): creating region Region { id: RegionId("1fbfc2d8-2775-47c1-b8f9-4744d6a0f6d7"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19002, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/1fbfc2d8-2775-47c1-b8f9-4744d6a0f6d7" region = 1fbfc2d8-2775-47c1-b8f9-4744d6a0f6d7
712026-03-04T04:42:18.454ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "1fbfc2d8-2775-47c1-b8f9-4744d6a0f6d7" "--data" "/data/regions/1fbfc2d8-2775-47c1-b8f9-4744d6a0f6d7" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 1fbfc2d8-2775-47c1-b8f9-4744d6a0f6d7
722026-03-04T04:42:18.637ZINFOcrucible-agent (worker): region files created ok region = 1fbfc2d8-2775-47c1-b8f9-4744d6a0f6d7
732026-03-04T04:42:18.637ZINFOcrucible-agent (datafile): region 1fbfc2d8-2775-47c1-b8f9-4744d6a0f6d7 state: Requested -> Created
742026-03-04T04:42:18.637ZINFOcrucible-agent (worker): applying SMF actions post create...
752026-03-04T04:42:18.638ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-1fbfc2d8-2775-47c1-b8f9-4744d6a0f6d7
762026-03-04T04:42:18.679ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-1fbfc2d8-2775-47c1-b8f9-4744d6a0f6d7
772026-03-04T04:42:18.688ZINFOcrucible-agent (dropshot): request completed latency_us = 78 local_addr = [fd00:1122:3344:101::1a]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:38152 req_id = 543245b5-4ab7-4a1e-966c-b305cec1ef94 response_code = 200 uri = /crucible/0/regions
782026-03-04T04:42:18.702ZINFOcrucible-agent (worker): creating config property group
792026-03-04T04:42:18.715ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-1fbfc2d8-2775-47c1-b8f9-4744d6a0f6d7
802026-03-04T04:42:18.715ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/1fbfc2d8-2775-47c1-b8f9-4744d6a0f6d7
812026-03-04T04:42:18.715ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19002
822026-03-04T04:42:18.715ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::1a
832026-03-04T04:42:18.715ZINFOcrucible-agent (worker): commit
842026-03-04T04:42:18.730ZINFOcrucible-agent (worker): ok!
852026-03-04T04:42:18.751ZINFOcrucible-agent (worker): SMF ok!
862026-03-04T04:42:48.890ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::1a]:32345 remote_addr = [fd00:1122:3344:101::c]:46657
872026-03-04T04:42:48.890ZINFOcrucible-agent (dropshot): request completed latency_us = 41 local_addr = [fd00:1122:3344:101::1a]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:46657 req_id = c7f5e7e9-bae6-4255-9291-b0261adb11c7 response_code = 200 uri = /crucible/0/regions/1fbfc2d8-2775-47c1-b8f9-4744d6a0f6d7
882026-03-04T04:42:48.893ZINFOcrucible-agent (datafile): path is "/data/regions/1fbfc2d8-2775-47c1-b8f9-4744d6a0f6d7"
892026-03-04T04:42:48.903ZINFOcrucible-agent (datafile): dataset is oxp_2a3e78bc-d612-43fc-a619-b5e212bdc7e5/crucible/regions/1fbfc2d8-2775-47c1-b8f9-4744d6a0f6d7
902026-03-04T04:42:48.910ZINFOcrucible-agent (datafile): region 1fbfc2d8-2775-47c1-b8f9-4744d6a0f6d7 state: Created -> Tombstoned
912026-03-04T04:42:48.911ZINFOcrucible-agent (dropshot): request completed latency_us = 17165 local_addr = [fd00:1122:3344:101::1a]:32345 method = DELETE remote_addr = [fd00:1122:3344:101::c]:46657 req_id = 73ce52f2-d47f-49b0-bb46-87e35321921a response_code = 204 uri = /crucible/0/regions/1fbfc2d8-2775-47c1-b8f9-4744d6a0f6d7
922026-03-04T04:42:48.911ZINFOcrucible-agent (worker): applying SMF actions before removal...
932026-03-04T04:42:48.911ZINFOcrucible-agent (worker): disabling downstairs instance: downstairs-1fbfc2d8-2775-47c1-b8f9-4744d6a0f6d7 (instance states: (Some(Online), None))
942026-03-04T04:42:48.914ZINFOcrucible-agent (dropshot): request completed latency_us = 43 local_addr = [fd00:1122:3344:101::1a]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:46657 req_id = 170003ae-745f-4ee9-8019-bf3b67fc9318 response_code = 200 uri = /crucible/0/regions/1fbfc2d8-2775-47c1-b8f9-4744d6a0f6d7
952026-03-04T04:42:48.923ZINFOcrucible-agent (worker): SMF ok!
962026-03-04T04:42:48.932ZINFOcrucible-agent (worker): deleting zfs dataset "oxp_2a3e78bc-d612-43fc-a619-b5e212bdc7e5/crucible/regions/1fbfc2d8-2775-47c1-b8f9-4744d6a0f6d7" region = 1fbfc2d8-2775-47c1-b8f9-4744d6a0f6d7
972026-03-04T04:42:48.977ZINFOcrucible-agent (dropshot): request completed latency_us = 62 local_addr = [fd00:1122:3344:101::1a]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:46657 req_id = af454654-20f9-478e-9a63-9bb3827913d4 response_code = 200 uri = /crucible/0/regions/1fbfc2d8-2775-47c1-b8f9-4744d6a0f6d7
982026-03-04T04:42:48.987ZINFOcrucible-agent (datafile): region 1fbfc2d8-2775-47c1-b8f9-4744d6a0f6d7 state: Tombstoned -> Destroyed
992026-03-04T04:42:49.122ZINFOcrucible-agent (dropshot): request completed latency_us = 50 local_addr = [fd00:1122:3344:101::1a]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:46657 req_id = 0b4f50f7-6be2-4d42-88c8-14b8a2b14838 response_code = 200 uri = /crucible/0/regions/1fbfc2d8-2775-47c1-b8f9-4744d6a0f6d7