Max level shown:
1 [ Jul 4 15:31:17 Enabled. ]
2 [ Jul 4 15:31:17 Rereading configuration. ]
3 [ Jul 4 15:31:22 Rereading configuration. ]
4 [ Jul 4 15:31:28 Executing start method ("/opt/oxide/crucible/bin/crucible-agent run -D /opt/oxide/crucible/bin/crucible-downstairs --dataset oxp_05e5d694-3136-4088-be79-073f4866dc49/crucible -l [fd00:1122:3344:101::15]:32345 -P 19000 -p downstairs -s snapshot"). ]
5 note: configured to log to "/dev/stdout"
62026-07-04T15:31:28.718ZINFOcrucible-agent: Crucible Version: Crucible Version: 0.0.1 Commit SHA: 2bfe090eb5318ec8c467157018db2429d4df535b Commit timestamp: 2026-06-25T19:33:01.000000000Z branch: main rustc: 1.90.0 stable x86_64-unknown-illumos Cargo: x86_64-unknown-illumos Debug: false Opt level: 3
72026-07-04T15:31:28.718ZINFOcrucible-agent: dataset: "oxp_05e5d694-3136-4088-be79-073f4866dc49/crucible"
82026-07-04T15:31:28.719ZINFOcrucible-agent: listen IP: [fd00:1122:3344:101::15]:32345
92026-07-04T15:31:28.719ZINFOcrucible-agent: SMF instance name downstairs_prefix: "downstairs"
102026-07-04T15:31:28.740ZINFOcrucible-agent (datafile): Using conf_path:"/data/crucible.json"
112026-07-04T15:31:29.351ZINFOcrucible-agent (dropshot): listening local_addr = [fd00:1122:3344:101::15]:32345
122026-07-04T15:31:54.940ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::15]:32345 remote_addr = [fd00:1122:3344:101::c]:48267
132026-07-04T15:31:54.969ZINFOcrucible-agent (datafile): region e549c8bb-a8bf-4d0c-a0d1-eeaf8ae1b5af state: Requested
142026-07-04T15:31:54.974ZINFOcrucible-agent (worker): Region size:1073741824 reservation:1342177280 quota:3221225472
152026-07-04T15:31:54.976ZINFOcrucible-agent (dropshot): request completed latency_us = 15418 local_addr = [fd00:1122:3344:101::15]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:48267 req_id = 0fd0aa7d-46a5-444e-987e-2a1874704fca response_code = 200 uri = /crucible/0/regions
162026-07-04T15:31:54.983ZINFOcrucible-agent (worker): zfs set reservation of 1342177280 for oxp_05e5d694-3136-4088-be79-073f4866dc49/crucible/regions/e549c8bb-a8bf-4d0c-a0d1-eeaf8ae1b5af
172026-07-04T15:31:54.983ZINFOcrucible-agent (worker): zfs set quota of 3221225472 for oxp_05e5d694-3136-4088-be79-073f4866dc49/crucible/regions/e549c8bb-a8bf-4d0c-a0d1-eeaf8ae1b5af
182026-07-04T15:31:55.037ZINFOcrucible-agent (worker): creating region Region { id: RegionId("e549c8bb-a8bf-4d0c-a0d1-eeaf8ae1b5af"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 16, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/e549c8bb-a8bf-4d0c-a0d1-eeaf8ae1b5af" region = e549c8bb-a8bf-4d0c-a0d1-eeaf8ae1b5af
192026-07-04T15:31:55.039ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "e549c8bb-a8bf-4d0c-a0d1-eeaf8ae1b5af" "--data" "/data/regions/e549c8bb-a8bf-4d0c-a0d1-eeaf8ae1b5af" "--block-size" "512" "--extent-size" "131072" "--extent-count" "16" "--encrypted" region = e549c8bb-a8bf-4d0c-a0d1-eeaf8ae1b5af
202026-07-04T15:31:55.121ZINFOcrucible-agent (dropshot): request completed latency_us = 72 local_addr = [fd00:1122:3344:101::15]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:48267 req_id = 091f8e5f-dfb5-4914-b349-6072ae61badd response_code = 200 uri = /crucible/0/regions
212026-07-04T15:31:55.171ZINFOcrucible-agent (worker): region files created ok region = e549c8bb-a8bf-4d0c-a0d1-eeaf8ae1b5af
222026-07-04T15:31:55.175ZINFOcrucible-agent (datafile): region e549c8bb-a8bf-4d0c-a0d1-eeaf8ae1b5af state: Requested -> Created
232026-07-04T15:31:55.175ZINFOcrucible-agent (worker): applying SMF actions post create...
242026-07-04T15:31:55.178ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-e549c8bb-a8bf-4d0c-a0d1-eeaf8ae1b5af
252026-07-04T15:31:55.185ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-e549c8bb-a8bf-4d0c-a0d1-eeaf8ae1b5af
262026-07-04T15:31:55.199ZINFOcrucible-agent (worker): creating config property group
272026-07-04T15:31:55.202ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-e549c8bb-a8bf-4d0c-a0d1-eeaf8ae1b5af
282026-07-04T15:31:55.203ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/e549c8bb-a8bf-4d0c-a0d1-eeaf8ae1b5af
292026-07-04T15:31:55.205ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
302026-07-04T15:31:55.205ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::15
312026-07-04T15:31:55.205ZINFOcrucible-agent (worker): commit
322026-07-04T15:31:55.214ZINFOcrucible-agent (worker): ok!
332026-07-04T15:31:55.219ZINFOcrucible-agent (worker): SMF ok!
342026-07-04T15:31:55.499ZINFOcrucible-agent (dropshot): request completed latency_us = 86 local_addr = [fd00:1122:3344:101::15]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:48267 req_id = d2081b5b-0d5b-4d94-a03b-9a1157afa360 response_code = 200 uri = /crucible/0/regions
352026-07-04T15:31:55.801ZINFOcrucible-agent (dropshot): request completed latency_us = 78 local_addr = [fd00:1122:3344:101::15]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:48267 req_id = e407def2-95f3-483e-8d33-f64c77a77ca6 response_code = 200 uri = /crucible/0/regions/e549c8bb-a8bf-4d0c-a0d1-eeaf8ae1b5af
362026-07-04T15:31:55.806ZINFOcrucible-agent (datafile): path is "/data/regions/e549c8bb-a8bf-4d0c-a0d1-eeaf8ae1b5af"
372026-07-04T15:31:55.813ZINFOcrucible-agent (datafile): dataset is oxp_05e5d694-3136-4088-be79-073f4866dc49/crucible/regions/e549c8bb-a8bf-4d0c-a0d1-eeaf8ae1b5af
382026-07-04T15:31:55.821ZINFOcrucible-agent (datafile): region e549c8bb-a8bf-4d0c-a0d1-eeaf8ae1b5af state: Created -> Tombstoned
392026-07-04T15:31:55.821ZINFOcrucible-agent (worker): applying SMF actions before removal...
402026-07-04T15:31:55.821ZINFOcrucible-agent (dropshot): request completed latency_us = 14857 local_addr = [fd00:1122:3344:101::15]:32345 method = DELETE remote_addr = [fd00:1122:3344:101::c]:48267 req_id = b2c489ac-1497-4b3b-9f8f-68ac88c2712f response_code = 204 uri = /crucible/0/regions/e549c8bb-a8bf-4d0c-a0d1-eeaf8ae1b5af
412026-07-04T15:31:55.822ZINFOcrucible-agent (worker): disabling downstairs instance: downstairs-e549c8bb-a8bf-4d0c-a0d1-eeaf8ae1b5af (instance states: (Some(Online), None))
422026-07-04T15:31:55.825ZINFOcrucible-agent (dropshot): request completed latency_us = 47 local_addr = [fd00:1122:3344:101::15]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:48267 req_id = cc48c2cf-4925-4953-b691-d5a1ec110d6e response_code = 200 uri = /crucible/0/regions/e549c8bb-a8bf-4d0c-a0d1-eeaf8ae1b5af
432026-07-04T15:31:55.826ZINFOcrucible-agent (worker): SMF ok!
442026-07-04T15:31:55.834ZINFOcrucible-agent (worker): deleting zfs dataset "oxp_05e5d694-3136-4088-be79-073f4866dc49/crucible/regions/e549c8bb-a8bf-4d0c-a0d1-eeaf8ae1b5af" region = e549c8bb-a8bf-4d0c-a0d1-eeaf8ae1b5af
452026-07-04T15:31:55.897ZINFOcrucible-agent (dropshot): request completed latency_us = 52 local_addr = [fd00:1122:3344:101::15]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:48267 req_id = 3aac63d0-1386-4779-82ec-0c2e4c09e1c5 response_code = 200 uri = /crucible/0/regions/e549c8bb-a8bf-4d0c-a0d1-eeaf8ae1b5af
462026-07-04T15:31:55.909ZINFOcrucible-agent (datafile): region e549c8bb-a8bf-4d0c-a0d1-eeaf8ae1b5af state: Tombstoned -> Destroyed
472026-07-04T15:31:56.016ZINFOcrucible-agent (dropshot): request completed latency_us = 48 local_addr = [fd00:1122:3344:101::15]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:48267 req_id = 45db4541-647e-4787-88c5-3a83daf58f0e response_code = 200 uri = /crucible/0/regions/e549c8bb-a8bf-4d0c-a0d1-eeaf8ae1b5af
482026-07-04T15:31:58.783ZINFOcrucible-agent (datafile): region e6d66ddd-58af-4aee-8795-0577a7f42b68 state: Requested
492026-07-04T15:31:58.783ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
502026-07-04T15:31:58.783ZINFOcrucible-agent (dropshot): request completed latency_us = 275 local_addr = [fd00:1122:3344:101::15]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:48267 req_id = e8d22a17-49f5-49be-8e72-4628640610e2 response_code = 200 uri = /crucible/0/regions
512026-07-04T15:31:58.790ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_05e5d694-3136-4088-be79-073f4866dc49/crucible/regions/e6d66ddd-58af-4aee-8795-0577a7f42b68
522026-07-04T15:31:58.790ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_05e5d694-3136-4088-be79-073f4866dc49/crucible/regions/e6d66ddd-58af-4aee-8795-0577a7f42b68
532026-07-04T15:31:58.864ZINFOcrucible-agent (worker): creating region Region { id: RegionId("e6d66ddd-58af-4aee-8795-0577a7f42b68"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/e6d66ddd-58af-4aee-8795-0577a7f42b68" region = e6d66ddd-58af-4aee-8795-0577a7f42b68
542026-07-04T15:31:58.864ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "e6d66ddd-58af-4aee-8795-0577a7f42b68" "--data" "/data/regions/e6d66ddd-58af-4aee-8795-0577a7f42b68" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = e6d66ddd-58af-4aee-8795-0577a7f42b68
552026-07-04T15:31:58.919ZINFOcrucible-agent (dropshot): request completed latency_us = 50 local_addr = [fd00:1122:3344:101::15]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:48267 req_id = d3e8d056-6569-477c-9097-73d856a90214 response_code = 200 uri = /crucible/0/regions
562026-07-04T15:31:58.942ZINFOcrucible-agent (worker): region files created ok region = e6d66ddd-58af-4aee-8795-0577a7f42b68
572026-07-04T15:31:58.942ZINFOcrucible-agent (datafile): region e6d66ddd-58af-4aee-8795-0577a7f42b68 state: Requested -> Created
582026-07-04T15:31:58.943ZINFOcrucible-agent (worker): applying SMF actions post create...
592026-07-04T15:31:58.944ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-e6d66ddd-58af-4aee-8795-0577a7f42b68
602026-07-04T15:31:58.947ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-e6d66ddd-58af-4aee-8795-0577a7f42b68
612026-07-04T15:31:58.956ZINFOcrucible-agent (worker): creating config property group
622026-07-04T15:31:58.959ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-e6d66ddd-58af-4aee-8795-0577a7f42b68
632026-07-04T15:31:58.959ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/e6d66ddd-58af-4aee-8795-0577a7f42b68
642026-07-04T15:31:58.959ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
652026-07-04T15:31:58.959ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::15
662026-07-04T15:31:58.959ZINFOcrucible-agent (worker): commit
672026-07-04T15:31:58.964ZINFOcrucible-agent (worker): ok!
682026-07-04T15:31:58.969ZINFOcrucible-agent (worker): SMF ok!
692026-07-04T15:31:59.300ZINFOcrucible-agent (dropshot): request completed latency_us = 57 local_addr = [fd00:1122:3344:101::15]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:48267 req_id = 244abf6c-7d8b-4136-8870-ab335ed641f6 response_code = 200 uri = /crucible/0/regions
702026-07-04T15:43:10.984ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::15]:32345 remote_addr = [fd00:1122:3344:101::c]:35937
712026-07-04T15:43:10.984ZINFOcrucible-agent (dropshot): request completed latency_us = 40 local_addr = [fd00:1122:3344:101::15]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:35937 req_id = e8411e26-bbc5-431e-a700-2bb22da69548 response_code = 200 uri = /crucible/0/regions/e6d66ddd-58af-4aee-8795-0577a7f42b68
722026-07-04T15:43:10.988ZINFOcrucible-agent (datafile): path is "/data/regions/e6d66ddd-58af-4aee-8795-0577a7f42b68"
732026-07-04T15:43:10.994ZINFOcrucible-agent (datafile): dataset is oxp_05e5d694-3136-4088-be79-073f4866dc49/crucible/regions/e6d66ddd-58af-4aee-8795-0577a7f42b68
742026-07-04T15:43:11.001ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_05e5d694-3136-4088-be79-073f4866dc49/crucible/regions/e6d66ddd-58af-4aee-8795-0577a7f42b68@a3409a38-1846-43c8-9a16-d8f3b352cbc5
752026-07-04T15:43:11.001ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_05e5d694-3136-4088-be79-073f4866dc49/crucible/regions/e6d66ddd-58af-4aee-8795-0577a7f42b68", "a3409a38-1846-43c8-9a16-d8f3b352cbc5"]
762026-07-04T15:43:11.036ZINFOcrucible-agent (dropshot): request completed latency_us = 48410 local_addr = [fd00:1122:3344:101::15]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:35937 req_id = d2f9ba1f-a8c1-4ddc-ab25-088cb07a1767 response_code = 200 uri = /crucible/0/regions/e6d66ddd-58af-4aee-8795-0577a7f42b68/snapshots/a3409a38-1846-43c8-9a16-d8f3b352cbc5
772026-07-04T15:43:11.040ZINFOcrucible-agent (datafile): path is "/data/regions/e6d66ddd-58af-4aee-8795-0577a7f42b68"
782026-07-04T15:43:11.047ZINFOcrucible-agent (datafile): dataset is oxp_05e5d694-3136-4088-be79-073f4866dc49/crucible/regions/e6d66ddd-58af-4aee-8795-0577a7f42b68
792026-07-04T15:43:11.052ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_05e5d694-3136-4088-be79-073f4866dc49/crucible/regions/e6d66ddd-58af-4aee-8795-0577a7f42b68@a3409a38-1846-43c8-9a16-d8f3b352cbc5
802026-07-04T15:43:11.052ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_05e5d694-3136-4088-be79-073f4866dc49/crucible/regions/e6d66ddd-58af-4aee-8795-0577a7f42b68", "a3409a38-1846-43c8-9a16-d8f3b352cbc5"]
812026-07-04T15:43:11.064ZINFOcrucible-agent (datafile): requesting running snapshot e6d66ddd-58af-4aee-8795-0577a7f42b68-a3409a38-1846-43c8-9a16-d8f3b352cbc5 state: Requested
822026-07-04T15:43:11.067ZINFOcrucible-agent (worker): applying SMF actions for region e6d66ddd-58af-4aee-8795-0577a7f42b68 running snapshot a3409a38-1846-43c8-9a16-d8f3b352cbc5 (state Requested)...
832026-07-04T15:43:11.067ZINFOcrucible-agent (dropshot): request completed latency_us = 27333 local_addr = [fd00:1122:3344:101::15]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:35937 req_id = ea50130f-fd62-4de0-8e0a-59a4203af011 response_code = 200 uri = /crucible/0/regions/e6d66ddd-58af-4aee-8795-0577a7f42b68/snapshots/a3409a38-1846-43c8-9a16-d8f3b352cbc5/run
842026-07-04T15:43:11.080ZINFOcrucible-agent (worker): creating missing snapshot instance snapshot-e6d66ddd-58af-4aee-8795-0577a7f42b68-a3409a38-1846-43c8-9a16-d8f3b352cbc5
852026-07-04T15:43:11.099ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:snapshot-e6d66ddd-58af-4aee-8795-0577a7f42b68-a3409a38-1846-43c8-9a16-d8f3b352cbc5
862026-07-04T15:43:11.159ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:snapshot-e6d66ddd-58af-4aee-8795-0577a7f42b68-a3409a38-1846-43c8-9a16-d8f3b352cbc5
872026-07-04T15:43:11.160ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/e6d66ddd-58af-4aee-8795-0577a7f42b68/.zfs/snapshot/a3409a38-1846-43c8-9a16-d8f3b352cbc5
882026-07-04T15:43:11.160ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
892026-07-04T15:43:11.160ZINFOcrucible-agent (worker): ensure mode SCF_TYPE_ASTRING ro
902026-07-04T15:43:11.160ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::15
912026-07-04T15:43:11.160ZINFOcrucible-agent (worker): commit
922026-07-04T15:43:11.187ZINFOcrucible-agent (worker): ok!
932026-07-04T15:43:11.213ZINFOcrucible-agent (worker): SMF ok!
942026-07-04T15:43:11.213ZINFOcrucible-agent (datafile): region e6d66ddd-58af-4aee-8795-0577a7f42b68 running snapshot a3409a38-1846-43c8-9a16-d8f3b352cbc5 state: Requested -> Created
952026-07-04T15:43:11.281ZINFOcrucible-agent (datafile): path is "/data/regions/e6d66ddd-58af-4aee-8795-0577a7f42b68"
962026-07-04T15:43:11.289ZINFOcrucible-agent (datafile): dataset is oxp_05e5d694-3136-4088-be79-073f4866dc49/crucible/regions/e6d66ddd-58af-4aee-8795-0577a7f42b68
972026-07-04T15:43:11.295ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_05e5d694-3136-4088-be79-073f4866dc49/crucible/regions/e6d66ddd-58af-4aee-8795-0577a7f42b68@a3409a38-1846-43c8-9a16-d8f3b352cbc5
982026-07-04T15:43:11.295ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_05e5d694-3136-4088-be79-073f4866dc49/crucible/regions/e6d66ddd-58af-4aee-8795-0577a7f42b68", "a3409a38-1846-43c8-9a16-d8f3b352cbc5"]
992026-07-04T15:43:11.302ZINFOcrucible-agent (dropshot): request completed latency_us = 21254 local_addr = [fd00:1122:3344:101::15]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:35937 req_id = 092a7541-708f-44fe-941c-ba721e0cdf58 response_code = 200 uri = /crucible/0/regions/e6d66ddd-58af-4aee-8795-0577a7f42b68/snapshots/a3409a38-1846-43c8-9a16-d8f3b352cbc5/run