Max level shown:
12026-08-06T20:45:42.878ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::18]:32345 remote_addr = [fd00:1122:3344:101::c]:49907
22026-08-06T20:45:42.878ZINFOcrucible-agent (datafile): region 7481668c-1402-4067-a755-c74ca4afb8f3 state: Requested
32026-08-06T20:45:42.879ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
42026-08-06T20:45:42.879ZINFOcrucible-agent (dropshot): request completed latency_us = 746 local_addr = [fd00:1122:3344:101::18]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:49907 req_id = b5a7ddbb-3f85-40e0-8797-0177dd7f71bb response_code = 200 uri = /crucible/0/regions
52026-08-06T20:45:42.887ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_02559026-0512-40f2-9ab0-7d621162eac2/crucible/regions/7481668c-1402-4067-a755-c74ca4afb8f3
62026-08-06T20:45:42.887ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_02559026-0512-40f2-9ab0-7d621162eac2/crucible/regions/7481668c-1402-4067-a755-c74ca4afb8f3
72026-08-06T20:45:42.967ZINFOcrucible-agent (worker): creating region Region { id: RegionId("7481668c-1402-4067-a755-c74ca4afb8f3"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19001, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/7481668c-1402-4067-a755-c74ca4afb8f3" region = 7481668c-1402-4067-a755-c74ca4afb8f3
82026-08-06T20:45:42.967ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "7481668c-1402-4067-a755-c74ca4afb8f3" "--data" "/data/regions/7481668c-1402-4067-a755-c74ca4afb8f3" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 7481668c-1402-4067-a755-c74ca4afb8f3
92026-08-06T20:45:43.029ZINFOcrucible-agent (dropshot): request completed latency_us = 53 local_addr = [fd00:1122:3344:101::18]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:49907 req_id = 8a77e0ac-4222-4ace-8ada-779a271cfc31 response_code = 200 uri = /crucible/0/regions
102026-08-06T20:45:43.124ZINFOcrucible-agent (worker): region files created ok region = 7481668c-1402-4067-a755-c74ca4afb8f3
112026-08-06T20:45:43.124ZINFOcrucible-agent (datafile): region 7481668c-1402-4067-a755-c74ca4afb8f3 state: Requested -> Created
122026-08-06T20:45:43.124ZINFOcrucible-agent (worker): applying SMF actions post create...
132026-08-06T20:45:43.140ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-7481668c-1402-4067-a755-c74ca4afb8f3
142026-08-06T20:45:43.149ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-7481668c-1402-4067-a755-c74ca4afb8f3
152026-08-06T20:45:43.174ZINFOcrucible-agent (worker): creating config property group
162026-08-06T20:45:43.186ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-7481668c-1402-4067-a755-c74ca4afb8f3
172026-08-06T20:45:43.186ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/7481668c-1402-4067-a755-c74ca4afb8f3
182026-08-06T20:45:43.186ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
192026-08-06T20:45:43.186ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::18
202026-08-06T20:45:43.186ZINFOcrucible-agent (worker): commit
212026-08-06T20:45:43.196ZINFOcrucible-agent (worker): ok!
222026-08-06T20:45:43.206ZINFOcrucible-agent (worker): SMF ok!
232026-08-06T20:45:43.513ZINFOcrucible-agent (dropshot): request completed latency_us = 58 local_addr = [fd00:1122:3344:101::18]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:49907 req_id = 4d3c594d-33b8-4f89-9842-087e705a996a response_code = 200 uri = /crucible/0/regions
242026-08-06T20:45:44.384ZINFOcrucible-agent (dropshot): request completed latency_us = 78 local_addr = [fd00:1122:3344:101::18]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:49907 req_id = b64c7a26-d240-4378-affd-c5b565a69eea response_code = 200 uri = /crucible/0/regions/6859aca2-1703-4676-94dc-a1bf1b9ccd97
252026-08-06T20:45:44.388ZINFOcrucible-agent (datafile): path is "/data/regions/6859aca2-1703-4676-94dc-a1bf1b9ccd97"
262026-08-06T20:45:44.395ZINFOcrucible-agent (datafile): dataset is oxp_02559026-0512-40f2-9ab0-7d621162eac2/crucible/regions/6859aca2-1703-4676-94dc-a1bf1b9ccd97
272026-08-06T20:45:44.402ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_02559026-0512-40f2-9ab0-7d621162eac2/crucible/regions/6859aca2-1703-4676-94dc-a1bf1b9ccd97@66f9f246-93a0-4fd0-857e-7c149cb681dc
282026-08-06T20:45:44.402ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_02559026-0512-40f2-9ab0-7d621162eac2/crucible/regions/6859aca2-1703-4676-94dc-a1bf1b9ccd97", "66f9f246-93a0-4fd0-857e-7c149cb681dc"]
292026-08-06T20:45:44.414ZINFOcrucible-agent (dropshot): request completed latency_us = 26356 local_addr = [fd00:1122:3344:101::18]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:49907 req_id = aaee2079-f1c8-4afc-9fe0-a85f0d51531f response_code = 200 uri = /crucible/0/regions/6859aca2-1703-4676-94dc-a1bf1b9ccd97/snapshots/66f9f246-93a0-4fd0-857e-7c149cb681dc
302026-08-06T20:45:44.423ZINFOcrucible-agent (datafile): path is "/data/regions/6859aca2-1703-4676-94dc-a1bf1b9ccd97"
312026-08-06T20:45:44.430ZINFOcrucible-agent (datafile): dataset is oxp_02559026-0512-40f2-9ab0-7d621162eac2/crucible/regions/6859aca2-1703-4676-94dc-a1bf1b9ccd97
322026-08-06T20:45:44.437ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_02559026-0512-40f2-9ab0-7d621162eac2/crucible/regions/6859aca2-1703-4676-94dc-a1bf1b9ccd97@66f9f246-93a0-4fd0-857e-7c149cb681dc
332026-08-06T20:45:44.437ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_02559026-0512-40f2-9ab0-7d621162eac2/crucible/regions/6859aca2-1703-4676-94dc-a1bf1b9ccd97", "66f9f246-93a0-4fd0-857e-7c149cb681dc"]
342026-08-06T20:45:44.445ZINFOcrucible-agent (datafile): requesting running snapshot 6859aca2-1703-4676-94dc-a1bf1b9ccd97-66f9f246-93a0-4fd0-857e-7c149cb681dc state: Requested
352026-08-06T20:45:44.450ZINFOcrucible-agent (worker): applying SMF actions for region 6859aca2-1703-4676-94dc-a1bf1b9ccd97 running snapshot 66f9f246-93a0-4fd0-857e-7c149cb681dc (state Requested)...
362026-08-06T20:45:44.450ZINFOcrucible-agent (dropshot): request completed latency_us = 26810 local_addr = [fd00:1122:3344:101::18]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:49907 req_id = 5d6be4c8-e584-4466-a53b-baa0c2198b38 response_code = 200 uri = /crucible/0/regions/6859aca2-1703-4676-94dc-a1bf1b9ccd97/snapshots/66f9f246-93a0-4fd0-857e-7c149cb681dc/run
372026-08-06T20:45:44.452ZINFOcrucible-agent (worker): creating missing snapshot instance snapshot-6859aca2-1703-4676-94dc-a1bf1b9ccd97-66f9f246-93a0-4fd0-857e-7c149cb681dc
382026-08-06T20:45:44.468ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:snapshot-6859aca2-1703-4676-94dc-a1bf1b9ccd97-66f9f246-93a0-4fd0-857e-7c149cb681dc
392026-08-06T20:45:44.490ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:snapshot-6859aca2-1703-4676-94dc-a1bf1b9ccd97-66f9f246-93a0-4fd0-857e-7c149cb681dc
402026-08-06T20:45:44.491ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/6859aca2-1703-4676-94dc-a1bf1b9ccd97/.zfs/snapshot/66f9f246-93a0-4fd0-857e-7c149cb681dc
412026-08-06T20:45:44.491ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19002
422026-08-06T20:45:44.491ZINFOcrucible-agent (worker): ensure mode SCF_TYPE_ASTRING ro
432026-08-06T20:45:44.491ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::18
442026-08-06T20:45:44.491ZINFOcrucible-agent (worker): commit
452026-08-06T20:45:44.498ZINFOcrucible-agent (worker): ok!
462026-08-06T20:45:44.507ZINFOcrucible-agent (worker): SMF ok!
472026-08-06T20:45:44.507ZINFOcrucible-agent (datafile): region 6859aca2-1703-4676-94dc-a1bf1b9ccd97 running snapshot 66f9f246-93a0-4fd0-857e-7c149cb681dc state: Requested -> Created
482026-08-06T20:45:44.597ZINFOcrucible-agent (datafile): path is "/data/regions/6859aca2-1703-4676-94dc-a1bf1b9ccd97"
492026-08-06T20:45:44.604ZINFOcrucible-agent (datafile): dataset is oxp_02559026-0512-40f2-9ab0-7d621162eac2/crucible/regions/6859aca2-1703-4676-94dc-a1bf1b9ccd97
502026-08-06T20:45:44.610ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_02559026-0512-40f2-9ab0-7d621162eac2/crucible/regions/6859aca2-1703-4676-94dc-a1bf1b9ccd97@66f9f246-93a0-4fd0-857e-7c149cb681dc
512026-08-06T20:45:44.610ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_02559026-0512-40f2-9ab0-7d621162eac2/crucible/regions/6859aca2-1703-4676-94dc-a1bf1b9ccd97", "66f9f246-93a0-4fd0-857e-7c149cb681dc"]
522026-08-06T20:45:44.615ZINFOcrucible-agent (dropshot): request completed latency_us = 18019 local_addr = [fd00:1122:3344:101::18]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:49907 req_id = 3977b643-c4c7-4b36-b24e-9922f9dfb8b6 response_code = 200 uri = /crucible/0/regions/6859aca2-1703-4676-94dc-a1bf1b9ccd97/snapshots/66f9f246-93a0-4fd0-857e-7c149cb681dc/run
532026-08-06T20:45:50.552ZINFOcrucible-agent (datafile): region 2f09af95-da8f-4754-9cdf-52727c88e7f4 state: Requested
542026-08-06T20:45:50.552ZINFOcrucible-agent (dropshot): request completed latency_us = 238 local_addr = [fd00:1122:3344:101::18]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:49907 req_id = 65447071-e13f-486f-b4ff-5aa70cbb52dc response_code = 200 uri = /crucible/0/regions
552026-08-06T20:45:50.552ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
562026-08-06T20:45:50.559ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_02559026-0512-40f2-9ab0-7d621162eac2/crucible/regions/2f09af95-da8f-4754-9cdf-52727c88e7f4
572026-08-06T20:45:50.559ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_02559026-0512-40f2-9ab0-7d621162eac2/crucible/regions/2f09af95-da8f-4754-9cdf-52727c88e7f4
582026-08-06T20:45:50.622ZINFOcrucible-agent (worker): creating region Region { id: RegionId("2f09af95-da8f-4754-9cdf-52727c88e7f4"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19003, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/2f09af95-da8f-4754-9cdf-52727c88e7f4" region = 2f09af95-da8f-4754-9cdf-52727c88e7f4
592026-08-06T20:45:50.622ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "2f09af95-da8f-4754-9cdf-52727c88e7f4" "--data" "/data/regions/2f09af95-da8f-4754-9cdf-52727c88e7f4" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 2f09af95-da8f-4754-9cdf-52727c88e7f4
602026-08-06T20:45:50.771ZINFOcrucible-agent (worker): region files created ok region = 2f09af95-da8f-4754-9cdf-52727c88e7f4
612026-08-06T20:45:50.771ZINFOcrucible-agent (datafile): region 2f09af95-da8f-4754-9cdf-52727c88e7f4 state: Requested -> Created
622026-08-06T20:45:50.771ZINFOcrucible-agent (worker): applying SMF actions post create...
632026-08-06T20:45:50.771ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-2f09af95-da8f-4754-9cdf-52727c88e7f4
642026-08-06T20:45:50.780ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-2f09af95-da8f-4754-9cdf-52727c88e7f4
652026-08-06T20:45:50.794ZINFOcrucible-agent (worker): creating config property group
662026-08-06T20:45:50.798ZINFOcrucible-agent (dropshot): request completed latency_us = 56 local_addr = [fd00:1122:3344:101::18]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:49907 req_id = 17ed6573-c3bf-47b7-8bcd-496693abb6e1 response_code = 200 uri = /crucible/0/regions
672026-08-06T20:45:50.801ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-2f09af95-da8f-4754-9cdf-52727c88e7f4
682026-08-06T20:45:50.801ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/2f09af95-da8f-4754-9cdf-52727c88e7f4
692026-08-06T20:45:50.801ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19003
702026-08-06T20:45:50.801ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::18
712026-08-06T20:45:50.801ZINFOcrucible-agent (worker): commit
722026-08-06T20:45:50.813ZINFOcrucible-agent (worker): ok!
732026-08-06T20:45:50.829ZINFOcrucible-agent (worker): SMF ok!
742026-08-06T20:46:18.319ZINFOcrucible-agent (dropshot): request completed latency_us = 52 local_addr = [fd00:1122:3344:101::18]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:49907 req_id = 58da1a56-13a6-4ca0-9c63-c45bcb164740 response_code = 200 uri = /crucible/0/regions/2f09af95-da8f-4754-9cdf-52727c88e7f4
752026-08-06T20:46:18.322ZINFOcrucible-agent (datafile): path is "/data/regions/2f09af95-da8f-4754-9cdf-52727c88e7f4"
762026-08-06T20:46:18.332ZINFOcrucible-agent (datafile): dataset is oxp_02559026-0512-40f2-9ab0-7d621162eac2/crucible/regions/2f09af95-da8f-4754-9cdf-52727c88e7f4
772026-08-06T20:46:18.339ZINFOcrucible-agent (datafile): region 2f09af95-da8f-4754-9cdf-52727c88e7f4 state: Created -> Tombstoned
782026-08-06T20:46:18.339ZINFOcrucible-agent (worker): applying SMF actions before removal...
792026-08-06T20:46:18.339ZINFOcrucible-agent (dropshot): request completed latency_us = 17077 local_addr = [fd00:1122:3344:101::18]:32345 method = DELETE remote_addr = [fd00:1122:3344:101::c]:49907 req_id = c3fb9da2-11d5-4871-845d-0d0febf2b98c response_code = 204 uri = /crucible/0/regions/2f09af95-da8f-4754-9cdf-52727c88e7f4
802026-08-06T20:46:18.340ZINFOcrucible-agent (worker): disabling downstairs instance: downstairs-2f09af95-da8f-4754-9cdf-52727c88e7f4 (instance states: (Some(Online), None))
812026-08-06T20:46:18.342ZINFOcrucible-agent (dropshot): request completed latency_us = 45 local_addr = [fd00:1122:3344:101::18]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:49907 req_id = 77bcbccb-5eec-466c-9f80-b17121a721f8 response_code = 200 uri = /crucible/0/regions/2f09af95-da8f-4754-9cdf-52727c88e7f4
822026-08-06T20:46:18.347ZINFOcrucible-agent (worker): SMF ok!
832026-08-06T20:46:18.356ZINFOcrucible-agent (worker): deleting zfs dataset "oxp_02559026-0512-40f2-9ab0-7d621162eac2/crucible/regions/2f09af95-da8f-4754-9cdf-52727c88e7f4" region = 2f09af95-da8f-4754-9cdf-52727c88e7f4
842026-08-06T20:46:18.403ZINFOcrucible-agent (datafile): region 2f09af95-da8f-4754-9cdf-52727c88e7f4 state: Tombstoned -> Destroyed
852026-08-06T20:46:18.472ZINFOcrucible-agent (dropshot): request completed latency_us = 87 local_addr = [fd00:1122:3344:101::18]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:49907 req_id = 4cce9bb3-168d-40e0-92fe-c2d69ab08f17 response_code = 200 uri = /crucible/0/regions/2f09af95-da8f-4754-9cdf-52727c88e7f4