Max level shown:
12026-01-30T03:45:32.848ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::12]:32345 remote_addr = [fd00:1122:3344:101::c]:63490
22026-01-30T03:45:32.911ZINFOcrucible-agent (datafile): region 6818fd22-eb0e-46cc-9b86-3fe6ce7562e4 state: Requested
32026-01-30T03:45:32.928ZINFOcrucible-agent (worker): Region size:1073741824 reservation:1342177280 quota:3221225472
42026-01-30T03:45:32.928ZINFOcrucible-agent (dropshot): request completed latency_us = 54569 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:63490 req_id = 4bfbd050-1639-4621-bb44-06bf46bc2de0 response_code = 200 uri = /crucible/0/regions
52026-01-30T03:45:32.936ZINFOcrucible-agent (worker): zfs set reservation of 1342177280 for oxp_3c8f7726-4fba-4e58-a7e5-1ee10d70743e/crucible/regions/6818fd22-eb0e-46cc-9b86-3fe6ce7562e4
62026-01-30T03:45:32.936ZINFOcrucible-agent (worker): zfs set quota of 3221225472 for oxp_3c8f7726-4fba-4e58-a7e5-1ee10d70743e/crucible/regions/6818fd22-eb0e-46cc-9b86-3fe6ce7562e4
72026-01-30T03:45:33.117ZINFOcrucible-agent (dropshot): request completed latency_us = 56 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:63490 req_id = 7e2a0de9-00a7-493b-8f1c-451515ac76ea response_code = 200 uri = /crucible/0/regions
82026-01-30T03:45:33.135ZINFOcrucible-agent (worker): creating region Region { id: RegionId("6818fd22-eb0e-46cc-9b86-3fe6ce7562e4"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 16, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/6818fd22-eb0e-46cc-9b86-3fe6ce7562e4" region = 6818fd22-eb0e-46cc-9b86-3fe6ce7562e4
92026-01-30T03:45:33.137ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "6818fd22-eb0e-46cc-9b86-3fe6ce7562e4" "--data" "/data/regions/6818fd22-eb0e-46cc-9b86-3fe6ce7562e4" "--block-size" "512" "--extent-size" "131072" "--extent-count" "16" "--encrypted" region = 6818fd22-eb0e-46cc-9b86-3fe6ce7562e4
102026-01-30T03:45:33.418ZINFOcrucible-agent (worker): region files created ok region = 6818fd22-eb0e-46cc-9b86-3fe6ce7562e4
112026-01-30T03:45:33.418ZINFOcrucible-agent (datafile): region 6818fd22-eb0e-46cc-9b86-3fe6ce7562e4 state: Requested -> Created
122026-01-30T03:45:33.418ZINFOcrucible-agent (worker): applying SMF actions post create...
132026-01-30T03:45:33.419ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-6818fd22-eb0e-46cc-9b86-3fe6ce7562e4
142026-01-30T03:45:33.428ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-6818fd22-eb0e-46cc-9b86-3fe6ce7562e4
152026-01-30T03:45:33.475ZINFOcrucible-agent (worker): creating config property group
162026-01-30T03:45:33.502ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-6818fd22-eb0e-46cc-9b86-3fe6ce7562e4
172026-01-30T03:45:33.503ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/6818fd22-eb0e-46cc-9b86-3fe6ce7562e4
182026-01-30T03:45:33.503ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
192026-01-30T03:45:33.503ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::12
202026-01-30T03:45:33.503ZINFOcrucible-agent (worker): commit
212026-01-30T03:45:33.530ZINFOcrucible-agent (worker): ok!
222026-01-30T03:45:33.555ZINFOcrucible-agent (worker): SMF ok!
232026-01-30T03:45:33.781ZINFOcrucible-agent (dropshot): request completed latency_us = 66 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:63490 req_id = 37f78840-ee00-4857-a1dd-df4a7f85e762 response_code = 200 uri = /crucible/0/regions
242026-01-30T03:45:36.221ZINFOcrucible-agent (dropshot): request completed latency_us = 71 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:63490 req_id = 7b913294-d77a-4f6a-8987-9cfb75946ca3 response_code = 200 uri = /crucible/0/regions/6818fd22-eb0e-46cc-9b86-3fe6ce7562e4
252026-01-30T03:45:36.225ZINFOcrucible-agent (datafile): path is "/data/regions/6818fd22-eb0e-46cc-9b86-3fe6ce7562e4"
262026-01-30T03:45:36.233ZINFOcrucible-agent (datafile): dataset is oxp_3c8f7726-4fba-4e58-a7e5-1ee10d70743e/crucible/regions/6818fd22-eb0e-46cc-9b86-3fe6ce7562e4
272026-01-30T03:45:36.241ZINFOcrucible-agent (datafile): region 6818fd22-eb0e-46cc-9b86-3fe6ce7562e4 state: Created -> Tombstoned
282026-01-30T03:45:36.241ZINFOcrucible-agent (worker): applying SMF actions before removal...
292026-01-30T03:45:36.241ZINFOcrucible-agent (dropshot): request completed latency_us = 15884 local_addr = [fd00:1122:3344:101::12]:32345 method = DELETE remote_addr = [fd00:1122:3344:101::c]:63490 req_id = b5a7b4ca-f2ee-4a2f-900d-cac97da0aebb response_code = 204 uri = /crucible/0/regions/6818fd22-eb0e-46cc-9b86-3fe6ce7562e4
302026-01-30T03:45:36.242ZINFOcrucible-agent (worker): disabling downstairs instance: downstairs-6818fd22-eb0e-46cc-9b86-3fe6ce7562e4 (instance states: (Some(Online), None))
312026-01-30T03:45:36.244ZINFOcrucible-agent (dropshot): request completed latency_us = 27 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:63490 req_id = cb8c55e3-229b-4e8f-b1a9-d4ca4e641b60 response_code = 200 uri = /crucible/0/regions/6818fd22-eb0e-46cc-9b86-3fe6ce7562e4
322026-01-30T03:45:36.263ZINFOcrucible-agent (worker): SMF ok!
332026-01-30T03:45:36.273ZINFOcrucible-agent (worker): deleting zfs dataset "oxp_3c8f7726-4fba-4e58-a7e5-1ee10d70743e/crucible/regions/6818fd22-eb0e-46cc-9b86-3fe6ce7562e4" region = 6818fd22-eb0e-46cc-9b86-3fe6ce7562e4
342026-01-30T03:45:36.383ZINFOcrucible-agent (dropshot): request completed latency_us = 57 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:63490 req_id = 0cc6a3f7-414f-478c-b394-a877a58cdf44 response_code = 200 uri = /crucible/0/regions/6818fd22-eb0e-46cc-9b86-3fe6ce7562e4
352026-01-30T03:45:36.454ZINFOcrucible-agent (dropshot): request completed latency_us = 33 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:63490 req_id = ccef4207-b562-4c1b-9e03-a62a670ca0bc response_code = 200 uri = /crucible/0/regions/6818fd22-eb0e-46cc-9b86-3fe6ce7562e4
362026-01-30T03:45:36.462ZINFOcrucible-agent (datafile): region 6818fd22-eb0e-46cc-9b86-3fe6ce7562e4 state: Tombstoned -> Destroyed
372026-01-30T03:45:36.610ZINFOcrucible-agent (dropshot): request completed latency_us = 32 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:63490 req_id = e561d75d-c407-485d-b544-0049054777a6 response_code = 200 uri = /crucible/0/regions/6818fd22-eb0e-46cc-9b86-3fe6ce7562e4
382026-01-30T03:57:04.558ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::12]:32345 remote_addr = [fd00:1122:3344:101::c]:57567
392026-01-30T03:57:04.558ZINFOcrucible-agent (datafile): region d6c54ece-9459-4d87-95b1-79477ce96d64 state: Requested
402026-01-30T03:57:04.559ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
412026-01-30T03:57:04.559ZINFOcrucible-agent (dropshot): request completed latency_us = 630 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:57567 req_id = 117b9696-d038-43a3-8e50-5af5c57631d7 response_code = 200 uri = /crucible/0/regions
422026-01-30T03:57:04.566ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_3c8f7726-4fba-4e58-a7e5-1ee10d70743e/crucible/regions/d6c54ece-9459-4d87-95b1-79477ce96d64
432026-01-30T03:57:04.566ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_3c8f7726-4fba-4e58-a7e5-1ee10d70743e/crucible/regions/d6c54ece-9459-4d87-95b1-79477ce96d64
442026-01-30T03:57:04.637ZINFOcrucible-agent (worker): creating region Region { id: RegionId("d6c54ece-9459-4d87-95b1-79477ce96d64"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/d6c54ece-9459-4d87-95b1-79477ce96d64" region = d6c54ece-9459-4d87-95b1-79477ce96d64
452026-01-30T03:57:04.637ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "d6c54ece-9459-4d87-95b1-79477ce96d64" "--data" "/data/regions/d6c54ece-9459-4d87-95b1-79477ce96d64" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = d6c54ece-9459-4d87-95b1-79477ce96d64
462026-01-30T03:57:04.781ZINFOcrucible-agent (worker): region files created ok region = d6c54ece-9459-4d87-95b1-79477ce96d64
472026-01-30T03:57:04.781ZINFOcrucible-agent (datafile): region d6c54ece-9459-4d87-95b1-79477ce96d64 state: Requested -> Created
482026-01-30T03:57:04.781ZINFOcrucible-agent (worker): applying SMF actions post create...
492026-01-30T03:57:04.782ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-d6c54ece-9459-4d87-95b1-79477ce96d64
502026-01-30T03:57:04.800ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-d6c54ece-9459-4d87-95b1-79477ce96d64
512026-01-30T03:57:04.829ZINFOcrucible-agent (worker): creating config property group
522026-01-30T03:57:04.837ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-d6c54ece-9459-4d87-95b1-79477ce96d64
532026-01-30T03:57:04.837ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/d6c54ece-9459-4d87-95b1-79477ce96d64
542026-01-30T03:57:04.837ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
552026-01-30T03:57:04.837ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::12
562026-01-30T03:57:04.837ZINFOcrucible-agent (worker): commit
572026-01-30T03:57:04.848ZINFOcrucible-agent (worker): ok!
582026-01-30T03:57:04.860ZINFOcrucible-agent (worker): SMF ok!
592026-01-30T03:57:04.870ZINFOcrucible-agent (dropshot): request completed latency_us = 56 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:57567 req_id = 0050ff46-0f13-482b-b96d-273588de77fe response_code = 200 uri = /crucible/0/regions
602026-01-30T03:57:34.455ZINFOcrucible-agent (dropshot): request completed latency_us = 53 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:57567 req_id = d08920b4-22a6-4f4c-823f-844fb70a6c28 response_code = 200 uri = /crucible/0/regions/d6c54ece-9459-4d87-95b1-79477ce96d64
612026-01-30T03:57:34.458ZINFOcrucible-agent (datafile): path is "/data/regions/d6c54ece-9459-4d87-95b1-79477ce96d64"
622026-01-30T03:57:34.471ZINFOcrucible-agent (datafile): dataset is oxp_3c8f7726-4fba-4e58-a7e5-1ee10d70743e/crucible/regions/d6c54ece-9459-4d87-95b1-79477ce96d64
632026-01-30T03:57:34.478ZINFOcrucible-agent (datafile): region d6c54ece-9459-4d87-95b1-79477ce96d64 state: Created -> Tombstoned
642026-01-30T03:57:34.479ZINFOcrucible-agent (worker): applying SMF actions before removal...
652026-01-30T03:57:34.479ZINFOcrucible-agent (dropshot): request completed latency_us = 20126 local_addr = [fd00:1122:3344:101::12]:32345 method = DELETE remote_addr = [fd00:1122:3344:101::c]:57567 req_id = ca9c588c-ae1d-47f1-a5c7-78eef0d23eb4 response_code = 204 uri = /crucible/0/regions/d6c54ece-9459-4d87-95b1-79477ce96d64
662026-01-30T03:57:34.480ZINFOcrucible-agent (worker): disabling downstairs instance: downstairs-d6c54ece-9459-4d87-95b1-79477ce96d64 (instance states: (Some(Online), None))
672026-01-30T03:57:34.481ZINFOcrucible-agent (dropshot): request completed latency_us = 46 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:57567 req_id = 8b8802c2-878b-48d3-8ca2-4b9e3fac3309 response_code = 200 uri = /crucible/0/regions/d6c54ece-9459-4d87-95b1-79477ce96d64
682026-01-30T03:57:34.491ZINFOcrucible-agent (worker): SMF ok!
692026-01-30T03:57:34.500ZINFOcrucible-agent (worker): deleting zfs dataset "oxp_3c8f7726-4fba-4e58-a7e5-1ee10d70743e/crucible/regions/d6c54ece-9459-4d87-95b1-79477ce96d64" region = d6c54ece-9459-4d87-95b1-79477ce96d64
702026-01-30T03:57:34.573ZINFOcrucible-agent (datafile): region d6c54ece-9459-4d87-95b1-79477ce96d64 state: Tombstoned -> Destroyed
712026-01-30T03:57:34.603ZINFOcrucible-agent (dropshot): request completed latency_us = 54 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:57567 req_id = 7e430810-e8ff-4f74-a11c-6d9fed398ff0 response_code = 200 uri = /crucible/0/regions/d6c54ece-9459-4d87-95b1-79477ce96d64