Max level shown:
12024-11-07T00:00:24.600ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::13]:32345 remote_addr = [fd00:1122:3344:101::a]:61257
22024-11-07T00:00:24.634ZINFOcrucible-agent (datafile): region 4ba15b31-2d99-4340-bde8-9e0899af3bb9 state: Requested
32024-11-07T00:00:24.637ZINFOcrucible-agent (worker): Region size:1073741824 reservation:1342177280 quota:3221225472
42024-11-07T00:00:24.641ZINFOcrucible-agent (dropshot): request completed latency_us = 19791 local_addr = [fd00:1122:3344:101::13]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:61257 req_id = 6a1bb2eb-6841-4f65-a1e0-78f31b7c8ddf response_code = 200 uri = /crucible/0/regions
52024-11-07T00:00:24.647ZINFOcrucible-agent (worker): zfs set reservation of 1342177280 for oxp_737b2345-38a9-43c1-8e04-028013083dab/crucible/regions/4ba15b31-2d99-4340-bde8-9e0899af3bb9
62024-11-07T00:00:24.647ZINFOcrucible-agent (worker): zfs set quota of 3221225472 for oxp_737b2345-38a9-43c1-8e04-028013083dab/crucible/regions/4ba15b31-2d99-4340-bde8-9e0899af3bb9
72024-11-07T00:00:24.697ZINFOcrucible-agent (worker): creating region Region { id: RegionId("4ba15b31-2d99-4340-bde8-9e0899af3bb9"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 16, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/4ba15b31-2d99-4340-bde8-9e0899af3bb9" region = 4ba15b31-2d99-4340-bde8-9e0899af3bb9
82024-11-07T00:00:24.701ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "4ba15b31-2d99-4340-bde8-9e0899af3bb9" "--data" "/data/regions/4ba15b31-2d99-4340-bde8-9e0899af3bb9" "--block-size" "512" "--extent-size" "131072" "--extent-count" "16" "--encrypted" region = 4ba15b31-2d99-4340-bde8-9e0899af3bb9
92024-11-07T00:00:24.845ZINFOcrucible-agent (dropshot): request completed latency_us = 69 local_addr = [fd00:1122:3344:101::13]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:61257 req_id = 014536ff-bd45-4075-a5a5-dbb42346d330 response_code = 200 uri = /crucible/0/regions
102024-11-07T00:00:24.847ZINFOcrucible-agent (worker): region files created ok region = 4ba15b31-2d99-4340-bde8-9e0899af3bb9
112024-11-07T00:00:24.847ZINFOcrucible-agent (datafile): region 4ba15b31-2d99-4340-bde8-9e0899af3bb9 state: Requested -> Created
122024-11-07T00:00:24.847ZINFOcrucible-agent (worker): applying SMF actions post create...
132024-11-07T00:00:24.847ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-4ba15b31-2d99-4340-bde8-9e0899af3bb9
142024-11-07T00:00:24.852ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-4ba15b31-2d99-4340-bde8-9e0899af3bb9
152024-11-07T00:00:24.869ZINFOcrucible-agent (worker): creating config property group
162024-11-07T00:00:24.872ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-4ba15b31-2d99-4340-bde8-9e0899af3bb9
172024-11-07T00:00:24.873ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/4ba15b31-2d99-4340-bde8-9e0899af3bb9
182024-11-07T00:00:24.876ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
192024-11-07T00:00:24.876ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::13
202024-11-07T00:00:24.876ZINFOcrucible-agent (worker): commit
212024-11-07T00:00:24.885ZINFOcrucible-agent (worker): ok!
222024-11-07T00:00:24.890ZINFOcrucible-agent (worker): SMF ok!
232024-11-07T00:00:25.206ZINFOcrucible-agent (dropshot): request completed latency_us = 60 local_addr = [fd00:1122:3344:101::13]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:61257 req_id = 48fd9f5e-dfe8-4b9f-bef1-6b986492ca3e response_code = 200 uri = /crucible/0/regions
242024-11-07T00:00:25.825ZINFOcrucible-agent (dropshot): request completed latency_us = 84 local_addr = [fd00:1122:3344:101::13]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:61257 req_id = 8a02bd13-2c51-44a4-bd7a-97e35ccbbf3d response_code = 200 uri = /crucible/0/regions/4ba15b31-2d99-4340-bde8-9e0899af3bb9
252024-11-07T00:00:25.828ZINFOcrucible-agent (datafile): path is "/data/regions/4ba15b31-2d99-4340-bde8-9e0899af3bb9"
262024-11-07T00:00:25.839ZINFOcrucible-agent (datafile): dataset is oxp_737b2345-38a9-43c1-8e04-028013083dab/crucible/regions/4ba15b31-2d99-4340-bde8-9e0899af3bb9
272024-11-07T00:00:25.849ZINFOcrucible-agent (datafile): region 4ba15b31-2d99-4340-bde8-9e0899af3bb9 state: Created -> Tombstoned
282024-11-07T00:00:25.849ZINFOcrucible-agent (worker): applying SMF actions before removal...
292024-11-07T00:00:25.850ZINFOcrucible-agent (worker): disabling downstairs instance: downstairs-4ba15b31-2d99-4340-bde8-9e0899af3bb9 (instance states: (Some(Online), None))
302024-11-07T00:00:25.856ZINFOcrucible-agent (dropshot): request completed latency_us = 28315 local_addr = [fd00:1122:3344:101::13]:32345 method = DELETE remote_addr = [fd00:1122:3344:101::a]:61257 req_id = 09b38112-48b4-4b48-b0ac-c2124205a5eb response_code = 204 uri = /crucible/0/regions/4ba15b31-2d99-4340-bde8-9e0899af3bb9
312024-11-07T00:00:25.857ZINFOcrucible-agent (worker): SMF ok!
322024-11-07T00:00:25.860ZINFOcrucible-agent (dropshot): request completed latency_us = 145 local_addr = [fd00:1122:3344:101::13]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:61257 req_id = 7bc04a59-ebb0-4e18-b6df-8f018f07cfbd response_code = 200 uri = /crucible/0/regions/4ba15b31-2d99-4340-bde8-9e0899af3bb9
332024-11-07T00:00:25.868ZINFOcrucible-agent (worker): deleting zfs dataset "oxp_737b2345-38a9-43c1-8e04-028013083dab/crucible/regions/4ba15b31-2d99-4340-bde8-9e0899af3bb9" region = 4ba15b31-2d99-4340-bde8-9e0899af3bb9
342024-11-07T00:00:25.930ZINFOcrucible-agent (datafile): region 4ba15b31-2d99-4340-bde8-9e0899af3bb9 state: Tombstoned -> Destroyed
352024-11-07T00:00:25.979ZINFOcrucible-agent (dropshot): request completed latency_us = 66 local_addr = [fd00:1122:3344:101::13]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:61257 req_id = 785639a2-da8b-483e-8617-550e745ccafd response_code = 200 uri = /crucible/0/regions/4ba15b31-2d99-4340-bde8-9e0899af3bb9
362024-11-07T00:00:28.093ZINFOcrucible-agent (datafile): region 157f3169-ebe8-419b-9078-464900e6cf90 state: Requested
372024-11-07T00:00:28.093ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
382024-11-07T00:00:28.093ZINFOcrucible-agent (dropshot): request completed latency_us = 202 local_addr = [fd00:1122:3344:101::13]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:61257 req_id = 0db252b4-047c-4465-ac03-efe0a3219934 response_code = 200 uri = /crucible/0/regions
392024-11-07T00:00:28.102ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_737b2345-38a9-43c1-8e04-028013083dab/crucible/regions/157f3169-ebe8-419b-9078-464900e6cf90
402024-11-07T00:00:28.102ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_737b2345-38a9-43c1-8e04-028013083dab/crucible/regions/157f3169-ebe8-419b-9078-464900e6cf90
412024-11-07T00:00:28.154ZINFOcrucible-agent (worker): creating region Region { id: RegionId("157f3169-ebe8-419b-9078-464900e6cf90"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/157f3169-ebe8-419b-9078-464900e6cf90" region = 157f3169-ebe8-419b-9078-464900e6cf90
422024-11-07T00:00:28.156ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "157f3169-ebe8-419b-9078-464900e6cf90" "--data" "/data/regions/157f3169-ebe8-419b-9078-464900e6cf90" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 157f3169-ebe8-419b-9078-464900e6cf90
432024-11-07T00:00:28.221ZINFOcrucible-agent (worker): region files created ok region = 157f3169-ebe8-419b-9078-464900e6cf90
442024-11-07T00:00:28.221ZINFOcrucible-agent (datafile): region 157f3169-ebe8-419b-9078-464900e6cf90 state: Requested -> Created
452024-11-07T00:00:28.221ZINFOcrucible-agent (worker): applying SMF actions post create...
462024-11-07T00:00:28.223ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-157f3169-ebe8-419b-9078-464900e6cf90
472024-11-07T00:00:28.226ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-157f3169-ebe8-419b-9078-464900e6cf90
482024-11-07T00:00:28.232ZINFOcrucible-agent (worker): creating config property group
492024-11-07T00:00:28.236ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-157f3169-ebe8-419b-9078-464900e6cf90
502024-11-07T00:00:28.236ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/157f3169-ebe8-419b-9078-464900e6cf90
512024-11-07T00:00:28.236ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
522024-11-07T00:00:28.236ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::13
532024-11-07T00:00:28.236ZINFOcrucible-agent (worker): commit
542024-11-07T00:00:28.240ZINFOcrucible-agent (worker): ok!
552024-11-07T00:00:28.244ZINFOcrucible-agent (worker): SMF ok!
562024-11-07T00:00:28.418ZINFOcrucible-agent (dropshot): request completed latency_us = 66 local_addr = [fd00:1122:3344:101::13]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:61257 req_id = 317146c3-6414-469d-9175-bcad9efc9183 response_code = 200 uri = /crucible/0/regions
572024-11-07T00:11:12.951ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::13]:32345 remote_addr = [fd00:1122:3344:101::a]:51329
582024-11-07T00:11:12.952ZINFOcrucible-agent (datafile): region d9e0b93e-884e-4f52-a3aa-841ed56b2fe7 state: Requested
592024-11-07T00:11:12.953ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
602024-11-07T00:11:12.953ZINFOcrucible-agent (dropshot): request completed latency_us = 1511 local_addr = [fd00:1122:3344:101::13]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:51329 req_id = ce92e965-f054-4519-85ac-afdebd0ce164 response_code = 200 uri = /crucible/0/regions
612024-11-07T00:11:12.962ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_737b2345-38a9-43c1-8e04-028013083dab/crucible/regions/d9e0b93e-884e-4f52-a3aa-841ed56b2fe7
622024-11-07T00:11:12.962ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_737b2345-38a9-43c1-8e04-028013083dab/crucible/regions/d9e0b93e-884e-4f52-a3aa-841ed56b2fe7
632024-11-07T00:11:13.018ZINFOcrucible-agent (worker): creating region Region { id: RegionId("d9e0b93e-884e-4f52-a3aa-841ed56b2fe7"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19001, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/d9e0b93e-884e-4f52-a3aa-841ed56b2fe7" region = d9e0b93e-884e-4f52-a3aa-841ed56b2fe7
642024-11-07T00:11:13.018ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "d9e0b93e-884e-4f52-a3aa-841ed56b2fe7" "--data" "/data/regions/d9e0b93e-884e-4f52-a3aa-841ed56b2fe7" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = d9e0b93e-884e-4f52-a3aa-841ed56b2fe7
652024-11-07T00:11:13.095ZINFOcrucible-agent (dropshot): request completed latency_us = 97 local_addr = [fd00:1122:3344:101::13]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:51329 req_id = 058cf7c2-eb72-4487-a8e8-2a124c463754 response_code = 200 uri = /crucible/0/regions
662024-11-07T00:11:13.141ZINFOcrucible-agent (worker): region files created ok region = d9e0b93e-884e-4f52-a3aa-841ed56b2fe7
672024-11-07T00:11:13.141ZINFOcrucible-agent (datafile): region d9e0b93e-884e-4f52-a3aa-841ed56b2fe7 state: Requested -> Created
682024-11-07T00:11:13.141ZINFOcrucible-agent (worker): applying SMF actions post create...
692024-11-07T00:11:13.150ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-d9e0b93e-884e-4f52-a3aa-841ed56b2fe7
702024-11-07T00:11:13.156ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-d9e0b93e-884e-4f52-a3aa-841ed56b2fe7
712024-11-07T00:11:13.165ZINFOcrucible-agent (worker): creating config property group
722024-11-07T00:11:13.168ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-d9e0b93e-884e-4f52-a3aa-841ed56b2fe7
732024-11-07T00:11:13.168ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/d9e0b93e-884e-4f52-a3aa-841ed56b2fe7
742024-11-07T00:11:13.168ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
752024-11-07T00:11:13.169ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::13
762024-11-07T00:11:13.169ZINFOcrucible-agent (worker): commit
772024-11-07T00:11:13.172ZINFOcrucible-agent (worker): ok!
782024-11-07T00:11:13.176ZINFOcrucible-agent (worker): SMF ok!
792024-11-07T00:11:13.704ZINFOcrucible-agent (dropshot): request completed latency_us = 50 local_addr = [fd00:1122:3344:101::13]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:51329 req_id = b6c7c7ae-83d2-4f89-8b05-76a1da7bb248 response_code = 200 uri = /crucible/0/regions
802024-11-07T00:11:14.913ZINFOcrucible-agent (dropshot): request completed latency_us = 56 local_addr = [fd00:1122:3344:101::13]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:51329 req_id = d2fc0d94-547b-4643-94b1-623c8fc2fc86 response_code = 200 uri = /crucible/0/regions/157f3169-ebe8-419b-9078-464900e6cf90
812024-11-07T00:11:14.917ZINFOcrucible-agent (datafile): path is "/data/regions/157f3169-ebe8-419b-9078-464900e6cf90"
822024-11-07T00:11:14.927ZINFOcrucible-agent (datafile): dataset is oxp_737b2345-38a9-43c1-8e04-028013083dab/crucible/regions/157f3169-ebe8-419b-9078-464900e6cf90
832024-11-07T00:11:14.936ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_737b2345-38a9-43c1-8e04-028013083dab/crucible/regions/157f3169-ebe8-419b-9078-464900e6cf90@93736f55-8d99-452a-9e65-823dc6e47b2c
842024-11-07T00:11:14.936ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_737b2345-38a9-43c1-8e04-028013083dab/crucible/regions/157f3169-ebe8-419b-9078-464900e6cf90", "93736f55-8d99-452a-9e65-823dc6e47b2c"]
852024-11-07T00:11:14.947ZINFOcrucible-agent (dropshot): request completed latency_us = 30480 local_addr = [fd00:1122:3344:101::13]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:51329 req_id = 8d763c68-2134-470f-adc4-51da3173693b response_code = 200 uri = /crucible/0/regions/157f3169-ebe8-419b-9078-464900e6cf90/snapshots/93736f55-8d99-452a-9e65-823dc6e47b2c
862024-11-07T00:11:14.951ZINFOcrucible-agent (datafile): path is "/data/regions/157f3169-ebe8-419b-9078-464900e6cf90"
872024-11-07T00:11:14.960ZINFOcrucible-agent (datafile): dataset is oxp_737b2345-38a9-43c1-8e04-028013083dab/crucible/regions/157f3169-ebe8-419b-9078-464900e6cf90
882024-11-07T00:11:14.968ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_737b2345-38a9-43c1-8e04-028013083dab/crucible/regions/157f3169-ebe8-419b-9078-464900e6cf90@93736f55-8d99-452a-9e65-823dc6e47b2c
892024-11-07T00:11:14.968ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_737b2345-38a9-43c1-8e04-028013083dab/crucible/regions/157f3169-ebe8-419b-9078-464900e6cf90", "93736f55-8d99-452a-9e65-823dc6e47b2c"]
902024-11-07T00:11:14.976ZINFOcrucible-agent (datafile): requesting running snapshot 157f3169-ebe8-419b-9078-464900e6cf90-93736f55-8d99-452a-9e65-823dc6e47b2c state: Requested
912024-11-07T00:11:14.976ZINFOcrucible-agent (worker): applying SMF actions for region 157f3169-ebe8-419b-9078-464900e6cf90 running snapshot 93736f55-8d99-452a-9e65-823dc6e47b2c (state Requested)...
922024-11-07T00:11:14.976ZINFOcrucible-agent (dropshot): request completed latency_us = 24904 local_addr = [fd00:1122:3344:101::13]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:51329 req_id = e733ada1-a46f-49dd-872e-99e32b3a327a response_code = 200 uri = /crucible/0/regions/157f3169-ebe8-419b-9078-464900e6cf90/snapshots/93736f55-8d99-452a-9e65-823dc6e47b2c/run
932024-11-07T00:11:14.979ZINFOcrucible-agent (worker): creating missing snapshot instance snapshot-157f3169-ebe8-419b-9078-464900e6cf90-93736f55-8d99-452a-9e65-823dc6e47b2c
942024-11-07T00:11:14.982ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:snapshot-157f3169-ebe8-419b-9078-464900e6cf90-93736f55-8d99-452a-9e65-823dc6e47b2c
952024-11-07T00:11:14.991ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:snapshot-157f3169-ebe8-419b-9078-464900e6cf90-93736f55-8d99-452a-9e65-823dc6e47b2c
962024-11-07T00:11:14.991ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/157f3169-ebe8-419b-9078-464900e6cf90/.zfs/snapshot/93736f55-8d99-452a-9e65-823dc6e47b2c
972024-11-07T00:11:14.991ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19002
982024-11-07T00:11:14.991ZINFOcrucible-agent (worker): ensure mode SCF_TYPE_ASTRING ro
992024-11-07T00:11:14.991ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::13
1002024-11-07T00:11:14.991ZINFOcrucible-agent (worker): commit
1012024-11-07T00:11:14.994ZINFOcrucible-agent (worker): ok!
1022024-11-07T00:11:14.999ZINFOcrucible-agent (worker): SMF ok!
1032024-11-07T00:11:14.999ZINFOcrucible-agent (datafile): region 157f3169-ebe8-419b-9078-464900e6cf90 running snapshot 93736f55-8d99-452a-9e65-823dc6e47b2c state: Requested -> Created
1042024-11-07T00:11:15.229ZINFOcrucible-agent (datafile): path is "/data/regions/157f3169-ebe8-419b-9078-464900e6cf90"
1052024-11-07T00:11:15.237ZINFOcrucible-agent (datafile): dataset is oxp_737b2345-38a9-43c1-8e04-028013083dab/crucible/regions/157f3169-ebe8-419b-9078-464900e6cf90
1062024-11-07T00:11:15.247ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_737b2345-38a9-43c1-8e04-028013083dab/crucible/regions/157f3169-ebe8-419b-9078-464900e6cf90@93736f55-8d99-452a-9e65-823dc6e47b2c
1072024-11-07T00:11:15.247ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_737b2345-38a9-43c1-8e04-028013083dab/crucible/regions/157f3169-ebe8-419b-9078-464900e6cf90", "93736f55-8d99-452a-9e65-823dc6e47b2c"]
1082024-11-07T00:11:15.255ZINFOcrucible-agent (dropshot): request completed latency_us = 25933 local_addr = [fd00:1122:3344:101::13]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:51329 req_id = d7a6bfa6-4b91-4348-a453-974476583895 response_code = 200 uri = /crucible/0/regions/157f3169-ebe8-419b-9078-464900e6cf90/snapshots/93736f55-8d99-452a-9e65-823dc6e47b2c/run