Max level shown:
12026-07-08T00:45:24.123ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::18]:32345 remote_addr = [fd00:1122:3344:101::a]:44767
22026-07-08T00:45:24.164ZINFOcrucible-agent (datafile): region 37bda5dd-4d8e-446d-aceb-41bc38642e07 state: Requested
32026-07-08T00:45:24.170ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
42026-07-08T00:45:24.172ZINFOcrucible-agent (dropshot): request completed latency_us = 22470 local_addr = [fd00:1122:3344:101::18]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:44767 req_id = 86174f50-ee61-4c0c-a22b-aea086d933bb response_code = 200 uri = /crucible/0/regions
52026-07-08T00:45:24.181ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_7b9bb7ab-3a2c-4057-abf2-fb555560ba4b/crucible/regions/37bda5dd-4d8e-446d-aceb-41bc38642e07
62026-07-08T00:45:24.181ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_7b9bb7ab-3a2c-4057-abf2-fb555560ba4b/crucible/regions/37bda5dd-4d8e-446d-aceb-41bc38642e07
72026-07-08T00:45:24.237ZINFOcrucible-agent (worker): creating region Region { id: RegionId("37bda5dd-4d8e-446d-aceb-41bc38642e07"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/37bda5dd-4d8e-446d-aceb-41bc38642e07" region = 37bda5dd-4d8e-446d-aceb-41bc38642e07
82026-07-08T00:45:24.240ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "37bda5dd-4d8e-446d-aceb-41bc38642e07" "--data" "/data/regions/37bda5dd-4d8e-446d-aceb-41bc38642e07" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 37bda5dd-4d8e-446d-aceb-41bc38642e07
92026-07-08T00:45:24.413ZINFOcrucible-agent (dropshot): request completed latency_us = 59 local_addr = [fd00:1122:3344:101::18]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:44767 req_id = 7b6e4d51-d6b1-44a9-9f15-c32c76418449 response_code = 200 uri = /crucible/0/regions
102026-07-08T00:45:24.465ZINFOcrucible-agent (worker): region files created ok region = 37bda5dd-4d8e-446d-aceb-41bc38642e07
112026-07-08T00:45:24.469ZINFOcrucible-agent (datafile): region 37bda5dd-4d8e-446d-aceb-41bc38642e07 state: Requested -> Created
122026-07-08T00:45:24.469ZINFOcrucible-agent (worker): applying SMF actions post create...
132026-07-08T00:45:24.472ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-37bda5dd-4d8e-446d-aceb-41bc38642e07
142026-07-08T00:45:24.477ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-37bda5dd-4d8e-446d-aceb-41bc38642e07
152026-07-08T00:45:24.491ZINFOcrucible-agent (worker): creating config property group
162026-07-08T00:45:24.493ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-37bda5dd-4d8e-446d-aceb-41bc38642e07
172026-07-08T00:45:24.494ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/37bda5dd-4d8e-446d-aceb-41bc38642e07
182026-07-08T00:45:24.496ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
192026-07-08T00:45:24.496ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::18
202026-07-08T00:45:24.496ZINFOcrucible-agent (worker): commit
212026-07-08T00:45:24.506ZINFOcrucible-agent (worker): ok!
222026-07-08T00:45:24.509ZINFOcrucible-agent (worker): SMF ok!
232026-07-08T00:45:24.679ZINFOcrucible-agent (dropshot): request completed latency_us = 84 local_addr = [fd00:1122:3344:101::18]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:44767 req_id = 8fb74b78-0792-4362-99dd-01e77223405b response_code = 200 uri = /crucible/0/regions
242026-07-08T00:56:27.877ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::18]:32345 remote_addr = [fd00:1122:3344:101::a]:64094
252026-07-08T00:56:27.878ZINFOcrucible-agent (dropshot): request completed latency_us = 125 local_addr = [fd00:1122:3344:101::18]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:64094 req_id = 2aa791d8-a70c-4d01-8da8-7e02205e7b3d response_code = 200 uri = /crucible/0/regions/37bda5dd-4d8e-446d-aceb-41bc38642e07
262026-07-08T00:56:27.881ZINFOcrucible-agent (datafile): path is "/data/regions/37bda5dd-4d8e-446d-aceb-41bc38642e07"
272026-07-08T00:56:27.887ZINFOcrucible-agent (datafile): dataset is oxp_7b9bb7ab-3a2c-4057-abf2-fb555560ba4b/crucible/regions/37bda5dd-4d8e-446d-aceb-41bc38642e07
282026-07-08T00:56:27.894ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_7b9bb7ab-3a2c-4057-abf2-fb555560ba4b/crucible/regions/37bda5dd-4d8e-446d-aceb-41bc38642e07@e12fb882-5493-463a-a447-273c03e9b64a
292026-07-08T00:56:27.894ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_7b9bb7ab-3a2c-4057-abf2-fb555560ba4b/crucible/regions/37bda5dd-4d8e-446d-aceb-41bc38642e07", "e12fb882-5493-463a-a447-273c03e9b64a"]
302026-07-08T00:56:27.906ZINFOcrucible-agent (dropshot): request completed latency_us = 24609 local_addr = [fd00:1122:3344:101::18]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:64094 req_id = ff561cb4-3fb5-45ec-8890-7053ed28bee0 response_code = 200 uri = /crucible/0/regions/37bda5dd-4d8e-446d-aceb-41bc38642e07/snapshots/e12fb882-5493-463a-a447-273c03e9b64a
312026-07-08T00:56:27.909ZINFOcrucible-agent (datafile): path is "/data/regions/37bda5dd-4d8e-446d-aceb-41bc38642e07"
322026-07-08T00:56:27.914ZINFOcrucible-agent (datafile): dataset is oxp_7b9bb7ab-3a2c-4057-abf2-fb555560ba4b/crucible/regions/37bda5dd-4d8e-446d-aceb-41bc38642e07
332026-07-08T00:56:27.920ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_7b9bb7ab-3a2c-4057-abf2-fb555560ba4b/crucible/regions/37bda5dd-4d8e-446d-aceb-41bc38642e07@e12fb882-5493-463a-a447-273c03e9b64a
342026-07-08T00:56:27.920ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_7b9bb7ab-3a2c-4057-abf2-fb555560ba4b/crucible/regions/37bda5dd-4d8e-446d-aceb-41bc38642e07", "e12fb882-5493-463a-a447-273c03e9b64a"]
352026-07-08T00:56:27.927ZINFOcrucible-agent (datafile): requesting running snapshot 37bda5dd-4d8e-446d-aceb-41bc38642e07-e12fb882-5493-463a-a447-273c03e9b64a state: Requested
362026-07-08T00:56:27.930ZINFOcrucible-agent (dropshot): request completed latency_us = 21621 local_addr = [fd00:1122:3344:101::18]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:64094 req_id = 209343cc-2be2-4b9b-9c4b-af98a1603fee response_code = 200 uri = /crucible/0/regions/37bda5dd-4d8e-446d-aceb-41bc38642e07/snapshots/e12fb882-5493-463a-a447-273c03e9b64a/run
372026-07-08T00:56:27.930ZINFOcrucible-agent (worker): applying SMF actions for region 37bda5dd-4d8e-446d-aceb-41bc38642e07 running snapshot e12fb882-5493-463a-a447-273c03e9b64a (state Requested)...
382026-07-08T00:56:27.947ZINFOcrucible-agent (worker): existing directory value /dev/null does not match /data/regions/37bda5dd-4d8e-446d-aceb-41bc38642e07
392026-07-08T00:56:27.947ZINFOcrucible-agent (worker): existing port value 0 does not match 19000
402026-07-08T00:56:27.947ZINFOcrucible-agent (worker): existing address value 0.0.0.0 does not match fd00:1122:3344:101::18
412026-07-08T00:56:27.947ZINFOcrucible-agent (worker): reconfig required
422026-07-08T00:56:27.947ZINFOcrucible-agent (worker): using existing config property group
432026-07-08T00:56:27.947ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-37bda5dd-4d8e-446d-aceb-41bc38642e07
442026-07-08T00:56:27.947ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/37bda5dd-4d8e-446d-aceb-41bc38642e07
452026-07-08T00:56:27.947ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
462026-07-08T00:56:27.947ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::18
472026-07-08T00:56:27.947ZINFOcrucible-agent (worker): commit
482026-07-08T00:56:27.953ZINFOcrucible-agent (worker): ok!
492026-07-08T00:56:27.956ZINFOcrucible-agent (worker): creating missing snapshot instance snapshot-37bda5dd-4d8e-446d-aceb-41bc38642e07-e12fb882-5493-463a-a447-273c03e9b64a
502026-07-08T00:56:27.957ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:snapshot-37bda5dd-4d8e-446d-aceb-41bc38642e07-e12fb882-5493-463a-a447-273c03e9b64a
512026-07-08T00:56:27.966ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:snapshot-37bda5dd-4d8e-446d-aceb-41bc38642e07-e12fb882-5493-463a-a447-273c03e9b64a
522026-07-08T00:56:27.966ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/37bda5dd-4d8e-446d-aceb-41bc38642e07/.zfs/snapshot/e12fb882-5493-463a-a447-273c03e9b64a
532026-07-08T00:56:27.966ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
542026-07-08T00:56:27.966ZINFOcrucible-agent (worker): ensure mode SCF_TYPE_ASTRING ro
552026-07-08T00:56:27.966ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::18
562026-07-08T00:56:27.966ZINFOcrucible-agent (worker): commit
572026-07-08T00:56:27.968ZINFOcrucible-agent (worker): ok!
582026-07-08T00:56:27.971ZINFOcrucible-agent (worker): SMF ok!
592026-07-08T00:56:27.971ZINFOcrucible-agent (datafile): region 37bda5dd-4d8e-446d-aceb-41bc38642e07 running snapshot e12fb882-5493-463a-a447-273c03e9b64a state: Requested -> Created
602026-07-08T00:56:28.071ZINFOcrucible-agent (datafile): path is "/data/regions/37bda5dd-4d8e-446d-aceb-41bc38642e07"
612026-07-08T00:56:28.077ZINFOcrucible-agent (datafile): dataset is oxp_7b9bb7ab-3a2c-4057-abf2-fb555560ba4b/crucible/regions/37bda5dd-4d8e-446d-aceb-41bc38642e07
622026-07-08T00:56:28.083ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_7b9bb7ab-3a2c-4057-abf2-fb555560ba4b/crucible/regions/37bda5dd-4d8e-446d-aceb-41bc38642e07@e12fb882-5493-463a-a447-273c03e9b64a
632026-07-08T00:56:28.083ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_7b9bb7ab-3a2c-4057-abf2-fb555560ba4b/crucible/regions/37bda5dd-4d8e-446d-aceb-41bc38642e07", "e12fb882-5493-463a-a447-273c03e9b64a"]
642026-07-08T00:56:28.089ZINFOcrucible-agent (dropshot): request completed latency_us = 17504 local_addr = [fd00:1122:3344:101::18]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:64094 req_id = fbbd5d29-5a70-49d8-97dd-984ffd50977c response_code = 200 uri = /crucible/0/regions/37bda5dd-4d8e-446d-aceb-41bc38642e07/snapshots/e12fb882-5493-463a-a447-273c03e9b64a/run
652026-07-08T00:56:32.838ZINFOcrucible-agent (datafile): region ee51bdd0-5388-4e13-a93c-12b9909de10d state: Requested
662026-07-08T00:56:32.838ZINFOcrucible-agent (dropshot): request completed latency_us = 233 local_addr = [fd00:1122:3344:101::18]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:64094 req_id = 4e86e786-ed1b-48b9-af6a-cf3b28bb0ee9 response_code = 200 uri = /crucible/0/regions
672026-07-08T00:56:32.838ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
682026-07-08T00:56:32.844ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_7b9bb7ab-3a2c-4057-abf2-fb555560ba4b/crucible/regions/ee51bdd0-5388-4e13-a93c-12b9909de10d
692026-07-08T00:56:32.844ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_7b9bb7ab-3a2c-4057-abf2-fb555560ba4b/crucible/regions/ee51bdd0-5388-4e13-a93c-12b9909de10d
702026-07-08T00:56:32.884ZINFOcrucible-agent (worker): creating region Region { id: RegionId("ee51bdd0-5388-4e13-a93c-12b9909de10d"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19002, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/ee51bdd0-5388-4e13-a93c-12b9909de10d" region = ee51bdd0-5388-4e13-a93c-12b9909de10d
712026-07-08T00:56:32.884ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "ee51bdd0-5388-4e13-a93c-12b9909de10d" "--data" "/data/regions/ee51bdd0-5388-4e13-a93c-12b9909de10d" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = ee51bdd0-5388-4e13-a93c-12b9909de10d
722026-07-08T00:56:32.935ZINFOcrucible-agent (worker): region files created ok region = ee51bdd0-5388-4e13-a93c-12b9909de10d
732026-07-08T00:56:32.935ZINFOcrucible-agent (datafile): region ee51bdd0-5388-4e13-a93c-12b9909de10d state: Requested -> Created
742026-07-08T00:56:32.935ZINFOcrucible-agent (worker): applying SMF actions post create...
752026-07-08T00:56:32.936ZINFOcrucible-agent (worker): existing directory value /dev/null does not match /data/regions/37bda5dd-4d8e-446d-aceb-41bc38642e07
762026-07-08T00:56:32.936ZINFOcrucible-agent (worker): existing port value 0 does not match 19000
772026-07-08T00:56:32.936ZINFOcrucible-agent (worker): existing address value 0.0.0.0 does not match fd00:1122:3344:101::18
782026-07-08T00:56:32.936ZINFOcrucible-agent (worker): reconfig required
792026-07-08T00:56:32.936ZINFOcrucible-agent (worker): using existing config property group
802026-07-08T00:56:32.936ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-37bda5dd-4d8e-446d-aceb-41bc38642e07
812026-07-08T00:56:32.936ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/37bda5dd-4d8e-446d-aceb-41bc38642e07
822026-07-08T00:56:32.936ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
832026-07-08T00:56:32.937ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::18
842026-07-08T00:56:32.937ZINFOcrucible-agent (worker): commit
852026-07-08T00:56:32.939ZINFOcrucible-agent (worker): ok!
862026-07-08T00:56:32.939ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-ee51bdd0-5388-4e13-a93c-12b9909de10d
872026-07-08T00:56:32.949ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-ee51bdd0-5388-4e13-a93c-12b9909de10d
882026-07-08T00:56:32.952ZINFOcrucible-agent (worker): creating config property group
892026-07-08T00:56:32.954ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-ee51bdd0-5388-4e13-a93c-12b9909de10d
902026-07-08T00:56:32.954ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/ee51bdd0-5388-4e13-a93c-12b9909de10d
912026-07-08T00:56:32.954ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19002
922026-07-08T00:56:32.954ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::18
932026-07-08T00:56:32.954ZINFOcrucible-agent (worker): commit
942026-07-08T00:56:32.956ZINFOcrucible-agent (worker): ok!
952026-07-08T00:56:32.960ZINFOcrucible-agent (worker): SMF ok!
962026-07-08T00:56:33.020ZINFOcrucible-agent (dropshot): request completed latency_us = 53 local_addr = [fd00:1122:3344:101::18]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:64094 req_id = 12fef76b-87d9-47f1-878a-a2d04bb467c7 response_code = 200 uri = /crucible/0/regions
972026-07-08T00:56:58.191ZINFOcrucible-agent (dropshot): request completed latency_us = 52 local_addr = [fd00:1122:3344:101::18]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:64094 req_id = 88a1ed7f-0969-4f4c-b168-8342267a97f1 response_code = 200 uri = /crucible/0/regions/ee51bdd0-5388-4e13-a93c-12b9909de10d
982026-07-08T00:56:58.193ZINFOcrucible-agent (datafile): path is "/data/regions/ee51bdd0-5388-4e13-a93c-12b9909de10d"
992026-07-08T00:56:58.205ZINFOcrucible-agent (datafile): dataset is oxp_7b9bb7ab-3a2c-4057-abf2-fb555560ba4b/crucible/regions/ee51bdd0-5388-4e13-a93c-12b9909de10d
1002026-07-08T00:56:58.212ZINFOcrucible-agent (datafile): region ee51bdd0-5388-4e13-a93c-12b9909de10d state: Created -> Tombstoned
1012026-07-08T00:56:58.212ZINFOcrucible-agent (worker): applying SMF actions before removal...
1022026-07-08T00:56:58.214ZINFOcrucible-agent (dropshot): request completed latency_us = 21158 local_addr = [fd00:1122:3344:101::18]:32345 method = DELETE remote_addr = [fd00:1122:3344:101::a]:64094 req_id = 6991d093-871b-404e-9be0-3580fe0a6bfe response_code = 204 uri = /crucible/0/regions/ee51bdd0-5388-4e13-a93c-12b9909de10d
1032026-07-08T00:56:58.215ZINFOcrucible-agent (worker): disabling downstairs instance: downstairs-ee51bdd0-5388-4e13-a93c-12b9909de10d (instance states: (Some(Online), None))
1042026-07-08T00:56:58.218ZINFOcrucible-agent (worker): existing directory value /dev/null does not match /data/regions/37bda5dd-4d8e-446d-aceb-41bc38642e07
1052026-07-08T00:56:58.219ZINFOcrucible-agent (worker): existing port value 0 does not match 19000
1062026-07-08T00:56:58.219ZINFOcrucible-agent (worker): existing address value 0.0.0.0 does not match fd00:1122:3344:101::18
1072026-07-08T00:56:58.219ZINFOcrucible-agent (worker): reconfig required
1082026-07-08T00:56:58.219ZINFOcrucible-agent (worker): using existing config property group
1092026-07-08T00:56:58.219ZINFOcrucible-agent (dropshot): request completed latency_us = 38 local_addr = [fd00:1122:3344:101::18]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:64094 req_id = 76459ef7-89f4-4592-8f41-7381d84a380b response_code = 200 uri = /crucible/0/regions/ee51bdd0-5388-4e13-a93c-12b9909de10d
1102026-07-08T00:56:58.219ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-37bda5dd-4d8e-446d-aceb-41bc38642e07
1112026-07-08T00:56:58.219ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/37bda5dd-4d8e-446d-aceb-41bc38642e07
1122026-07-08T00:56:58.219ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
1132026-07-08T00:56:58.219ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::18
1142026-07-08T00:56:58.219ZINFOcrucible-agent (worker): commit
1152026-07-08T00:56:58.222ZINFOcrucible-agent (worker): ok!
1162026-07-08T00:56:58.223ZINFOcrucible-agent (worker): SMF ok!
1172026-07-08T00:56:58.232ZINFOcrucible-agent (worker): deleting zfs dataset "oxp_7b9bb7ab-3a2c-4057-abf2-fb555560ba4b/crucible/regions/ee51bdd0-5388-4e13-a93c-12b9909de10d" region = ee51bdd0-5388-4e13-a93c-12b9909de10d
1182026-07-08T00:56:58.281ZINFOcrucible-agent (datafile): region ee51bdd0-5388-4e13-a93c-12b9909de10d state: Tombstoned -> Destroyed
1192026-07-08T00:56:58.320ZINFOcrucible-agent (dropshot): request completed latency_us = 51 local_addr = [fd00:1122:3344:101::18]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:64094 req_id = d80a52c9-b043-4f4d-bc2f-e59be06a0559 response_code = 200 uri = /crucible/0/regions/ee51bdd0-5388-4e13-a93c-12b9909de10d