Max level shown:
12026-07-15T04:15:26.243ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::19]:32345 remote_addr = [fd00:1122:3344:101::a]:40382
22026-07-15T04:15:26.288ZINFOcrucible-agent (datafile): region 3065ee3b-4e66-4df9-8caf-500cfd851d86 state: Requested
32026-07-15T04:15:26.291ZINFOcrucible-agent (worker): Region size:1073741824 reservation:1342177280 quota:3221225472
42026-07-15T04:15:26.293ZINFOcrucible-agent (dropshot): request completed latency_us = 17754 local_addr = [fd00:1122:3344:101::19]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:40382 req_id = afc0c632-2486-4df2-9cc5-0923230e4b00 response_code = 200 uri = /crucible/0/regions
52026-07-15T04:15:26.300ZINFOcrucible-agent (worker): zfs set reservation of 1342177280 for oxp_11db365b-d983-4b65-b3de-d90504558774/crucible/regions/3065ee3b-4e66-4df9-8caf-500cfd851d86
62026-07-15T04:15:26.300ZINFOcrucible-agent (worker): zfs set quota of 3221225472 for oxp_11db365b-d983-4b65-b3de-d90504558774/crucible/regions/3065ee3b-4e66-4df9-8caf-500cfd851d86
72026-07-15T04:15:26.370ZINFOcrucible-agent (worker): creating region Region { id: RegionId("3065ee3b-4e66-4df9-8caf-500cfd851d86"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 16, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/3065ee3b-4e66-4df9-8caf-500cfd851d86" region = 3065ee3b-4e66-4df9-8caf-500cfd851d86
82026-07-15T04:15:26.370ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "3065ee3b-4e66-4df9-8caf-500cfd851d86" "--data" "/data/regions/3065ee3b-4e66-4df9-8caf-500cfd851d86" "--block-size" "512" "--extent-size" "131072" "--extent-count" "16" "--encrypted" region = 3065ee3b-4e66-4df9-8caf-500cfd851d86
92026-07-15T04:15:26.437ZINFOcrucible-agent (dropshot): request completed latency_us = 67 local_addr = [fd00:1122:3344:101::19]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:40382 req_id = 0e944888-c514-423a-a874-a0ebc6d75576 response_code = 200 uri = /crucible/0/regions
102026-07-15T04:15:26.682ZINFOcrucible-agent (worker): region files created ok region = 3065ee3b-4e66-4df9-8caf-500cfd851d86
112026-07-15T04:15:26.686ZINFOcrucible-agent (datafile): region 3065ee3b-4e66-4df9-8caf-500cfd851d86 state: Requested -> Created
122026-07-15T04:15:26.686ZINFOcrucible-agent (worker): applying SMF actions post create...
132026-07-15T04:15:26.689ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-3065ee3b-4e66-4df9-8caf-500cfd851d86
142026-07-15T04:15:26.697ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-3065ee3b-4e66-4df9-8caf-500cfd851d86
152026-07-15T04:15:26.710ZINFOcrucible-agent (worker): creating config property group
162026-07-15T04:15:26.714ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-3065ee3b-4e66-4df9-8caf-500cfd851d86
172026-07-15T04:15:26.714ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/3065ee3b-4e66-4df9-8caf-500cfd851d86
182026-07-15T04:15:26.717ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
192026-07-15T04:15:26.717ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::19
202026-07-15T04:15:26.717ZINFOcrucible-agent (worker): commit
212026-07-15T04:15:26.729ZINFOcrucible-agent (worker): ok!
222026-07-15T04:15:26.734ZINFOcrucible-agent (worker): SMF ok!
232026-07-15T04:15:27.121ZINFOcrucible-agent (dropshot): request completed latency_us = 73 local_addr = [fd00:1122:3344:101::19]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:40382 req_id = 19a38cf9-cbf1-4fc3-b779-9377811a4979 response_code = 200 uri = /crucible/0/regions
242026-07-15T04:15:27.470ZINFOcrucible-agent (dropshot): request completed latency_us = 71 local_addr = [fd00:1122:3344:101::19]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:40382 req_id = 26159151-8072-4b9a-a9bc-34ca9b623e3f response_code = 200 uri = /crucible/0/regions/3065ee3b-4e66-4df9-8caf-500cfd851d86
252026-07-15T04:15:27.474ZINFOcrucible-agent (datafile): path is "/data/regions/3065ee3b-4e66-4df9-8caf-500cfd851d86"
262026-07-15T04:15:27.481ZINFOcrucible-agent (datafile): dataset is oxp_11db365b-d983-4b65-b3de-d90504558774/crucible/regions/3065ee3b-4e66-4df9-8caf-500cfd851d86
272026-07-15T04:15:27.489ZINFOcrucible-agent (datafile): region 3065ee3b-4e66-4df9-8caf-500cfd851d86 state: Created -> Tombstoned
282026-07-15T04:15:27.489ZINFOcrucible-agent (worker): applying SMF actions before removal...
292026-07-15T04:15:27.489ZINFOcrucible-agent (dropshot): request completed latency_us = 15539 local_addr = [fd00:1122:3344:101::19]:32345 method = DELETE remote_addr = [fd00:1122:3344:101::a]:40382 req_id = 540e8a7e-86bc-4eb8-8e6d-e1c0fa96961b response_code = 204 uri = /crucible/0/regions/3065ee3b-4e66-4df9-8caf-500cfd851d86
302026-07-15T04:15:27.490ZINFOcrucible-agent (worker): disabling downstairs instance: downstairs-3065ee3b-4e66-4df9-8caf-500cfd851d86 (instance states: (Some(Online), None))
312026-07-15T04:15:27.492ZINFOcrucible-agent (dropshot): request completed latency_us = 43 local_addr = [fd00:1122:3344:101::19]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:40382 req_id = b7ba592a-ea39-4fb9-b0dc-b33474e1992a response_code = 200 uri = /crucible/0/regions/3065ee3b-4e66-4df9-8caf-500cfd851d86
322026-07-15T04:15:27.495ZINFOcrucible-agent (worker): SMF ok!
332026-07-15T04:15:27.505ZINFOcrucible-agent (worker): deleting zfs dataset "oxp_11db365b-d983-4b65-b3de-d90504558774/crucible/regions/3065ee3b-4e66-4df9-8caf-500cfd851d86" region = 3065ee3b-4e66-4df9-8caf-500cfd851d86
342026-07-15T04:15:27.577ZINFOcrucible-agent (datafile): region 3065ee3b-4e66-4df9-8caf-500cfd851d86 state: Tombstoned -> Destroyed
352026-07-15T04:15:27.618ZINFOcrucible-agent (dropshot): request completed latency_us = 61 local_addr = [fd00:1122:3344:101::19]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:40382 req_id = 4cc46fdb-30f5-4a00-97c7-cd8670773874 response_code = 200 uri = /crucible/0/regions/3065ee3b-4e66-4df9-8caf-500cfd851d86
362026-07-15T04:15:30.608ZINFOcrucible-agent (datafile): region fb7ba080-8a27-4917-aa84-857397837d75 state: Requested
372026-07-15T04:15:30.608ZINFOcrucible-agent (dropshot): request completed latency_us = 232 local_addr = [fd00:1122:3344:101::19]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:40382 req_id = f78c11be-ec83-4760-bd37-d1aff8c9961d response_code = 200 uri = /crucible/0/regions
382026-07-15T04:15:30.608ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
392026-07-15T04:15:30.615ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_11db365b-d983-4b65-b3de-d90504558774/crucible/regions/fb7ba080-8a27-4917-aa84-857397837d75
402026-07-15T04:15:30.615ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_11db365b-d983-4b65-b3de-d90504558774/crucible/regions/fb7ba080-8a27-4917-aa84-857397837d75
412026-07-15T04:15:30.684ZINFOcrucible-agent (worker): creating region Region { id: RegionId("fb7ba080-8a27-4917-aa84-857397837d75"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/fb7ba080-8a27-4917-aa84-857397837d75" region = fb7ba080-8a27-4917-aa84-857397837d75
422026-07-15T04:15:30.684ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "fb7ba080-8a27-4917-aa84-857397837d75" "--data" "/data/regions/fb7ba080-8a27-4917-aa84-857397837d75" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = fb7ba080-8a27-4917-aa84-857397837d75
432026-07-15T04:15:30.763ZINFOcrucible-agent (worker): region files created ok region = fb7ba080-8a27-4917-aa84-857397837d75
442026-07-15T04:15:30.763ZINFOcrucible-agent (datafile): region fb7ba080-8a27-4917-aa84-857397837d75 state: Requested -> Created
452026-07-15T04:15:30.764ZINFOcrucible-agent (worker): applying SMF actions post create...
462026-07-15T04:15:30.765ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-fb7ba080-8a27-4917-aa84-857397837d75
472026-07-15T04:15:30.769ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-fb7ba080-8a27-4917-aa84-857397837d75
482026-07-15T04:15:30.779ZINFOcrucible-agent (worker): creating config property group
492026-07-15T04:15:30.783ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-fb7ba080-8a27-4917-aa84-857397837d75
502026-07-15T04:15:30.783ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/fb7ba080-8a27-4917-aa84-857397837d75
512026-07-15T04:15:30.783ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
522026-07-15T04:15:30.783ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::19
532026-07-15T04:15:30.783ZINFOcrucible-agent (worker): commit
542026-07-15T04:15:30.788ZINFOcrucible-agent (worker): ok!
552026-07-15T04:15:30.795ZINFOcrucible-agent (worker): SMF ok!
562026-07-15T04:15:30.931ZINFOcrucible-agent (dropshot): request completed latency_us = 59 local_addr = [fd00:1122:3344:101::19]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:40382 req_id = b8cbaf63-313f-43ee-8648-bdc16bc3bbed response_code = 200 uri = /crucible/0/regions
572026-07-15T04:26:40.910ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::19]:32345 remote_addr = [fd00:1122:3344:101::a]:36854
582026-07-15T04:26:40.910ZINFOcrucible-agent (datafile): region 15ea0204-4ecb-4e10-b1a0-be7ea73c1ce5 state: Requested
592026-07-15T04:26:40.911ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
602026-07-15T04:26:40.911ZINFOcrucible-agent (dropshot): request completed latency_us = 976 local_addr = [fd00:1122:3344:101::19]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:36854 req_id = 1e319c69-0896-4c4d-a502-c106ef6c9914 response_code = 200 uri = /crucible/0/regions
612026-07-15T04:26:40.921ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_11db365b-d983-4b65-b3de-d90504558774/crucible/regions/15ea0204-4ecb-4e10-b1a0-be7ea73c1ce5
622026-07-15T04:26:40.921ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_11db365b-d983-4b65-b3de-d90504558774/crucible/regions/15ea0204-4ecb-4e10-b1a0-be7ea73c1ce5
632026-07-15T04:26:40.975ZINFOcrucible-agent (worker): creating region Region { id: RegionId("15ea0204-4ecb-4e10-b1a0-be7ea73c1ce5"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19001, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/15ea0204-4ecb-4e10-b1a0-be7ea73c1ce5" region = 15ea0204-4ecb-4e10-b1a0-be7ea73c1ce5
642026-07-15T04:26:40.975ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "15ea0204-4ecb-4e10-b1a0-be7ea73c1ce5" "--data" "/data/regions/15ea0204-4ecb-4e10-b1a0-be7ea73c1ce5" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 15ea0204-4ecb-4e10-b1a0-be7ea73c1ce5
652026-07-15T04:26:41.052ZINFOcrucible-agent (worker): region files created ok region = 15ea0204-4ecb-4e10-b1a0-be7ea73c1ce5
662026-07-15T04:26:41.052ZINFOcrucible-agent (datafile): region 15ea0204-4ecb-4e10-b1a0-be7ea73c1ce5 state: Requested -> Created
672026-07-15T04:26:41.052ZINFOcrucible-agent (worker): applying SMF actions post create...
682026-07-15T04:26:41.053ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-15ea0204-4ecb-4e10-b1a0-be7ea73c1ce5
692026-07-15T04:26:41.064ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-15ea0204-4ecb-4e10-b1a0-be7ea73c1ce5
702026-07-15T04:26:41.077ZINFOcrucible-agent (worker): creating config property group
712026-07-15T04:26:41.081ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-15ea0204-4ecb-4e10-b1a0-be7ea73c1ce5
722026-07-15T04:26:41.081ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/15ea0204-4ecb-4e10-b1a0-be7ea73c1ce5
732026-07-15T04:26:41.081ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
742026-07-15T04:26:41.081ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::19
752026-07-15T04:26:41.081ZINFOcrucible-agent (worker): commit
762026-07-15T04:26:41.085ZINFOcrucible-agent (worker): ok!
772026-07-15T04:26:41.091ZINFOcrucible-agent (worker): SMF ok!
782026-07-15T04:26:41.275ZINFOcrucible-agent (dropshot): request completed latency_us = 58 local_addr = [fd00:1122:3344:101::19]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:36854 req_id = 2653c5d7-9396-4212-9f35-8fdfe2dfda0a response_code = 200 uri = /crucible/0/regions
792026-07-15T04:26:42.482ZINFOcrucible-agent (dropshot): request completed latency_us = 53 local_addr = [fd00:1122:3344:101::19]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:36854 req_id = 1d7edee9-c8da-4d32-bb12-43e4aea213e0 response_code = 200 uri = /crucible/0/regions/fb7ba080-8a27-4917-aa84-857397837d75
802026-07-15T04:26:42.486ZINFOcrucible-agent (datafile): path is "/data/regions/fb7ba080-8a27-4917-aa84-857397837d75"
812026-07-15T04:26:42.492ZINFOcrucible-agent (datafile): dataset is oxp_11db365b-d983-4b65-b3de-d90504558774/crucible/regions/fb7ba080-8a27-4917-aa84-857397837d75
822026-07-15T04:26:42.497ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_11db365b-d983-4b65-b3de-d90504558774/crucible/regions/fb7ba080-8a27-4917-aa84-857397837d75@1aa6248d-c8ac-4c78-933b-b86a534670bc
832026-07-15T04:26:42.497ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_11db365b-d983-4b65-b3de-d90504558774/crucible/regions/fb7ba080-8a27-4917-aa84-857397837d75", "1aa6248d-c8ac-4c78-933b-b86a534670bc"]
842026-07-15T04:26:42.507ZINFOcrucible-agent (dropshot): request completed latency_us = 21249 local_addr = [fd00:1122:3344:101::19]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:36854 req_id = ecaebc56-6dc2-4189-bfc4-0ab1ff1bd155 response_code = 200 uri = /crucible/0/regions/fb7ba080-8a27-4917-aa84-857397837d75/snapshots/1aa6248d-c8ac-4c78-933b-b86a534670bc
852026-07-15T04:26:42.511ZINFOcrucible-agent (datafile): path is "/data/regions/fb7ba080-8a27-4917-aa84-857397837d75"
862026-07-15T04:26:42.515ZINFOcrucible-agent (datafile): dataset is oxp_11db365b-d983-4b65-b3de-d90504558774/crucible/regions/fb7ba080-8a27-4917-aa84-857397837d75
872026-07-15T04:26:42.520ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_11db365b-d983-4b65-b3de-d90504558774/crucible/regions/fb7ba080-8a27-4917-aa84-857397837d75@1aa6248d-c8ac-4c78-933b-b86a534670bc
882026-07-15T04:26:42.520ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_11db365b-d983-4b65-b3de-d90504558774/crucible/regions/fb7ba080-8a27-4917-aa84-857397837d75", "1aa6248d-c8ac-4c78-933b-b86a534670bc"]
892026-07-15T04:26:42.526ZINFOcrucible-agent (datafile): requesting running snapshot fb7ba080-8a27-4917-aa84-857397837d75-1aa6248d-c8ac-4c78-933b-b86a534670bc state: Requested
902026-07-15T04:26:42.528ZINFOcrucible-agent (worker): applying SMF actions for region fb7ba080-8a27-4917-aa84-857397837d75 running snapshot 1aa6248d-c8ac-4c78-933b-b86a534670bc (state Requested)...
912026-07-15T04:26:42.528ZINFOcrucible-agent (dropshot): request completed latency_us = 17900 local_addr = [fd00:1122:3344:101::19]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:36854 req_id = c511794b-0016-4aed-a27c-990bc90c7f67 response_code = 200 uri = /crucible/0/regions/fb7ba080-8a27-4917-aa84-857397837d75/snapshots/1aa6248d-c8ac-4c78-933b-b86a534670bc/run
922026-07-15T04:26:42.533ZINFOcrucible-agent (worker): creating missing snapshot instance snapshot-fb7ba080-8a27-4917-aa84-857397837d75-1aa6248d-c8ac-4c78-933b-b86a534670bc
932026-07-15T04:26:42.537ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:snapshot-fb7ba080-8a27-4917-aa84-857397837d75-1aa6248d-c8ac-4c78-933b-b86a534670bc
942026-07-15T04:26:42.549ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:snapshot-fb7ba080-8a27-4917-aa84-857397837d75-1aa6248d-c8ac-4c78-933b-b86a534670bc
952026-07-15T04:26:42.549ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/fb7ba080-8a27-4917-aa84-857397837d75/.zfs/snapshot/1aa6248d-c8ac-4c78-933b-b86a534670bc
962026-07-15T04:26:42.549ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19002
972026-07-15T04:26:42.549ZINFOcrucible-agent (worker): ensure mode SCF_TYPE_ASTRING ro
982026-07-15T04:26:42.549ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::19
992026-07-15T04:26:42.549ZINFOcrucible-agent (worker): commit
1002026-07-15T04:26:42.553ZINFOcrucible-agent (worker): ok!
1012026-07-15T04:26:42.558ZINFOcrucible-agent (worker): SMF ok!
1022026-07-15T04:26:42.558ZINFOcrucible-agent (datafile): region fb7ba080-8a27-4917-aa84-857397837d75 running snapshot 1aa6248d-c8ac-4c78-933b-b86a534670bc state: Requested -> Created
1032026-07-15T04:26:42.889ZINFOcrucible-agent (datafile): path is "/data/regions/fb7ba080-8a27-4917-aa84-857397837d75"
1042026-07-15T04:26:42.896ZINFOcrucible-agent (datafile): dataset is oxp_11db365b-d983-4b65-b3de-d90504558774/crucible/regions/fb7ba080-8a27-4917-aa84-857397837d75
1052026-07-15T04:26:42.902ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_11db365b-d983-4b65-b3de-d90504558774/crucible/regions/fb7ba080-8a27-4917-aa84-857397837d75@1aa6248d-c8ac-4c78-933b-b86a534670bc
1062026-07-15T04:26:42.902ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_11db365b-d983-4b65-b3de-d90504558774/crucible/regions/fb7ba080-8a27-4917-aa84-857397837d75", "1aa6248d-c8ac-4c78-933b-b86a534670bc"]
1072026-07-15T04:26:42.908ZINFOcrucible-agent (dropshot): request completed latency_us = 19136 local_addr = [fd00:1122:3344:101::19]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:36854 req_id = 8aeaa8b7-aa71-4b21-827c-f96dd9b6bbe6 response_code = 200 uri = /crucible/0/regions/fb7ba080-8a27-4917-aa84-857397837d75/snapshots/1aa6248d-c8ac-4c78-933b-b86a534670bc/run