Max level shown:
12024-12-06T21:15:10.908ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::14]:32345 remote_addr = [fd00:1122:3344:101::a]:56794
22024-12-06T21:15:10.924ZINFOcrucible-agent (datafile): region 7400ed25-c4be-4189-a432-3cab1060eae5 state: Requested
32024-12-06T21:15:10.926ZINFOcrucible-agent (worker): Region size:1073741824 reservation:1342177280 quota:3221225472
42024-12-06T21:15:10.926ZINFOcrucible-agent (dropshot): request completed latency_us = 4469 local_addr = [fd00:1122:3344:101::14]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:56794 req_id = 0f4ddfff-5d80-4fa8-a4dc-63908c802136 response_code = 200 uri = /crucible/0/regions
52024-12-06T21:15:10.936ZINFOcrucible-agent (worker): zfs set reservation of 1342177280 for oxp_afed364c-1eb2-4c62-a6a7-b86cf13ba8c1/crucible/regions/7400ed25-c4be-4189-a432-3cab1060eae5
62024-12-06T21:15:10.936ZINFOcrucible-agent (worker): zfs set quota of 3221225472 for oxp_afed364c-1eb2-4c62-a6a7-b86cf13ba8c1/crucible/regions/7400ed25-c4be-4189-a432-3cab1060eae5
72024-12-06T21:15:10.999ZINFOcrucible-agent (worker): creating region Region { id: RegionId("7400ed25-c4be-4189-a432-3cab1060eae5"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 16, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/7400ed25-c4be-4189-a432-3cab1060eae5" region = 7400ed25-c4be-4189-a432-3cab1060eae5
82024-12-06T21:15:11.001ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "7400ed25-c4be-4189-a432-3cab1060eae5" "--data" "/data/regions/7400ed25-c4be-4189-a432-3cab1060eae5" "--block-size" "512" "--extent-size" "131072" "--extent-count" "16" "--encrypted" region = 7400ed25-c4be-4189-a432-3cab1060eae5
92024-12-06T21:15:11.109ZINFOcrucible-agent (worker): region files created ok region = 7400ed25-c4be-4189-a432-3cab1060eae5
102024-12-06T21:15:11.109ZINFOcrucible-agent (datafile): region 7400ed25-c4be-4189-a432-3cab1060eae5 state: Requested -> Created
112024-12-06T21:15:11.109ZINFOcrucible-agent (worker): applying SMF actions post create...
122024-12-06T21:15:11.110ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-7400ed25-c4be-4189-a432-3cab1060eae5
132024-12-06T21:15:11.116ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-7400ed25-c4be-4189-a432-3cab1060eae5
142024-12-06T21:15:11.125ZINFOcrucible-agent (worker): creating config property group
152024-12-06T21:15:11.129ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-7400ed25-c4be-4189-a432-3cab1060eae5
162024-12-06T21:15:11.129ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/7400ed25-c4be-4189-a432-3cab1060eae5
172024-12-06T21:15:11.130ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
182024-12-06T21:15:11.130ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::14
192024-12-06T21:15:11.130ZINFOcrucible-agent (worker): commit
202024-12-06T21:15:11.134ZINFOcrucible-agent (worker): ok!
212024-12-06T21:15:11.138ZINFOcrucible-agent (worker): SMF ok!
222024-12-06T21:15:11.296ZINFOcrucible-agent (dropshot): request completed latency_us = 62 local_addr = [fd00:1122:3344:101::14]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:56794 req_id = 94bb704f-99d8-4e1d-a0ea-20bafc1e1f0d response_code = 200 uri = /crucible/0/regions
232024-12-06T21:15:12.034ZINFOcrucible-agent (dropshot): request completed latency_us = 71 local_addr = [fd00:1122:3344:101::14]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:56794 req_id = 012c9fd9-dd2a-4e03-b025-f2c72ad2b11d response_code = 200 uri = /crucible/0/regions/7400ed25-c4be-4189-a432-3cab1060eae5
242024-12-06T21:15:12.037ZINFOcrucible-agent (datafile): path is "/data/regions/7400ed25-c4be-4189-a432-3cab1060eae5"
252024-12-06T21:15:12.048ZINFOcrucible-agent (datafile): dataset is oxp_afed364c-1eb2-4c62-a6a7-b86cf13ba8c1/crucible/regions/7400ed25-c4be-4189-a432-3cab1060eae5
262024-12-06T21:15:12.057ZINFOcrucible-agent (datafile): region 7400ed25-c4be-4189-a432-3cab1060eae5 state: Created -> Tombstoned
272024-12-06T21:15:12.058ZINFOcrucible-agent (worker): applying SMF actions before removal...
282024-12-06T21:15:12.058ZINFOcrucible-agent (dropshot): request completed latency_us = 20775 local_addr = [fd00:1122:3344:101::14]:32345 method = DELETE remote_addr = [fd00:1122:3344:101::a]:56794 req_id = d7fc7023-8ffb-49c7-8922-48688e592515 response_code = 204 uri = /crucible/0/regions/7400ed25-c4be-4189-a432-3cab1060eae5
292024-12-06T21:15:12.061ZINFOcrucible-agent (worker): disabling downstairs instance: downstairs-7400ed25-c4be-4189-a432-3cab1060eae5 (instance states: (Some(Online), None))
302024-12-06T21:15:12.061ZINFOcrucible-agent (dropshot): request completed latency_us = 36 local_addr = [fd00:1122:3344:101::14]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:56794 req_id = e2eff7f9-00e2-4472-8f44-120915c9c3af response_code = 200 uri = /crucible/0/regions/7400ed25-c4be-4189-a432-3cab1060eae5
312024-12-06T21:15:12.065ZINFOcrucible-agent (worker): SMF ok!
322024-12-06T21:15:12.076ZINFOcrucible-agent (worker): deleting zfs dataset "oxp_afed364c-1eb2-4c62-a6a7-b86cf13ba8c1/crucible/regions/7400ed25-c4be-4189-a432-3cab1060eae5" region = 7400ed25-c4be-4189-a432-3cab1060eae5
332024-12-06T21:15:12.129ZINFOcrucible-agent (dropshot): request completed latency_us = 84 local_addr = [fd00:1122:3344:101::14]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:56794 req_id = 818a4511-7c06-4ac0-a29d-3da7cd11ce29 response_code = 200 uri = /crucible/0/regions/7400ed25-c4be-4189-a432-3cab1060eae5
342024-12-06T21:15:12.146ZINFOcrucible-agent (datafile): region 7400ed25-c4be-4189-a432-3cab1060eae5 state: Tombstoned -> Destroyed
352024-12-06T21:15:12.195ZINFOcrucible-agent (dropshot): request completed latency_us = 22 local_addr = [fd00:1122:3344:101::14]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:56794 req_id = d0bcb1da-dd97-484f-8f60-0d045d0245ca response_code = 200 uri = /crucible/0/regions/7400ed25-c4be-4189-a432-3cab1060eae5
362024-12-06T21:15:14.061ZINFOcrucible-agent (datafile): region 960301fa-fe60-4a7e-85bb-83848cd35a2c state: Requested
372024-12-06T21:15:14.062ZINFOcrucible-agent (dropshot): request completed latency_us = 221 local_addr = [fd00:1122:3344:101::14]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:56794 req_id = 575d9e47-274b-40b1-b656-d7efb31ad0a1 response_code = 200 uri = /crucible/0/regions
382024-12-06T21:15:14.062ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
392024-12-06T21:15:14.073ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_afed364c-1eb2-4c62-a6a7-b86cf13ba8c1/crucible/regions/960301fa-fe60-4a7e-85bb-83848cd35a2c
402024-12-06T21:15:14.073ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_afed364c-1eb2-4c62-a6a7-b86cf13ba8c1/crucible/regions/960301fa-fe60-4a7e-85bb-83848cd35a2c
412024-12-06T21:15:14.129ZINFOcrucible-agent (worker): creating region Region { id: RegionId("960301fa-fe60-4a7e-85bb-83848cd35a2c"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/960301fa-fe60-4a7e-85bb-83848cd35a2c" region = 960301fa-fe60-4a7e-85bb-83848cd35a2c
422024-12-06T21:15:14.131ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "960301fa-fe60-4a7e-85bb-83848cd35a2c" "--data" "/data/regions/960301fa-fe60-4a7e-85bb-83848cd35a2c" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 960301fa-fe60-4a7e-85bb-83848cd35a2c
432024-12-06T21:15:14.214ZINFOcrucible-agent (worker): region files created ok region = 960301fa-fe60-4a7e-85bb-83848cd35a2c
442024-12-06T21:15:14.214ZINFOcrucible-agent (datafile): region 960301fa-fe60-4a7e-85bb-83848cd35a2c state: Requested -> Created
452024-12-06T21:15:14.214ZINFOcrucible-agent (worker): applying SMF actions post create...
462024-12-06T21:15:14.215ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-960301fa-fe60-4a7e-85bb-83848cd35a2c
472024-12-06T21:15:14.220ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-960301fa-fe60-4a7e-85bb-83848cd35a2c
482024-12-06T21:15:14.237ZINFOcrucible-agent (worker): creating config property group
492024-12-06T21:15:14.238ZINFOcrucible-agent (dropshot): request completed latency_us = 71 local_addr = [fd00:1122:3344:101::14]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:56794 req_id = be8dee0c-82d0-427b-ba4e-b97b2dc3212c response_code = 200 uri = /crucible/0/regions
502024-12-06T21:15:14.246ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-960301fa-fe60-4a7e-85bb-83848cd35a2c
512024-12-06T21:15:14.246ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/960301fa-fe60-4a7e-85bb-83848cd35a2c
522024-12-06T21:15:14.246ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
532024-12-06T21:15:14.246ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::14
542024-12-06T21:15:14.246ZINFOcrucible-agent (worker): commit
552024-12-06T21:15:14.254ZINFOcrucible-agent (worker): ok!
562024-12-06T21:15:14.259ZINFOcrucible-agent (worker): SMF ok!
572024-12-06T21:25:34.927ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::14]:32345 remote_addr = [fd00:1122:3344:101::a]:34806
582024-12-06T21:25:34.927ZINFOcrucible-agent (datafile): region 68621ad6-9576-49f2-9782-e2e2ec9abfcd state: Requested
592024-12-06T21:25:34.928ZINFOcrucible-agent (dropshot): request completed latency_us = 1128 local_addr = [fd00:1122:3344:101::14]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:34806 req_id = d778ca80-9245-4ef6-80a3-2b89646d7dbf response_code = 200 uri = /crucible/0/regions
602024-12-06T21:25:34.928ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
612024-12-06T21:25:34.937ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_afed364c-1eb2-4c62-a6a7-b86cf13ba8c1/crucible/regions/68621ad6-9576-49f2-9782-e2e2ec9abfcd
622024-12-06T21:25:34.937ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_afed364c-1eb2-4c62-a6a7-b86cf13ba8c1/crucible/regions/68621ad6-9576-49f2-9782-e2e2ec9abfcd
632024-12-06T21:25:34.991ZINFOcrucible-agent (worker): creating region Region { id: RegionId("68621ad6-9576-49f2-9782-e2e2ec9abfcd"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19001, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/68621ad6-9576-49f2-9782-e2e2ec9abfcd" region = 68621ad6-9576-49f2-9782-e2e2ec9abfcd
642024-12-06T21:25:34.991ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "68621ad6-9576-49f2-9782-e2e2ec9abfcd" "--data" "/data/regions/68621ad6-9576-49f2-9782-e2e2ec9abfcd" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 68621ad6-9576-49f2-9782-e2e2ec9abfcd
652024-12-06T21:25:35.092ZINFOcrucible-agent (worker): region files created ok region = 68621ad6-9576-49f2-9782-e2e2ec9abfcd
662024-12-06T21:25:35.092ZINFOcrucible-agent (datafile): region 68621ad6-9576-49f2-9782-e2e2ec9abfcd state: Requested -> Created
672024-12-06T21:25:35.092ZINFOcrucible-agent (worker): applying SMF actions post create...
682024-12-06T21:25:35.094ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-68621ad6-9576-49f2-9782-e2e2ec9abfcd
692024-12-06T21:25:35.102ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-68621ad6-9576-49f2-9782-e2e2ec9abfcd
702024-12-06T21:25:35.115ZINFOcrucible-agent (worker): creating config property group
712024-12-06T21:25:35.118ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-68621ad6-9576-49f2-9782-e2e2ec9abfcd
722024-12-06T21:25:35.119ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/68621ad6-9576-49f2-9782-e2e2ec9abfcd
732024-12-06T21:25:35.119ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
742024-12-06T21:25:35.119ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::14
752024-12-06T21:25:35.119ZINFOcrucible-agent (worker): commit
762024-12-06T21:25:35.123ZINFOcrucible-agent (worker): ok!
772024-12-06T21:25:35.128ZINFOcrucible-agent (worker): SMF ok!
782024-12-06T21:25:35.292ZINFOcrucible-agent (dropshot): request completed latency_us = 67 local_addr = [fd00:1122:3344:101::14]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:34806 req_id = 987b0c83-2ff1-4575-b43d-e8d0cddfbf48 response_code = 200 uri = /crucible/0/regions
792024-12-06T21:25:37.073ZINFOcrucible-agent (dropshot): request completed latency_us = 57 local_addr = [fd00:1122:3344:101::14]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:34806 req_id = 38408c65-f920-46d2-8840-46793d9d9c80 response_code = 200 uri = /crucible/0/regions/960301fa-fe60-4a7e-85bb-83848cd35a2c
802024-12-06T21:25:37.077ZINFOcrucible-agent (datafile): path is "/data/regions/960301fa-fe60-4a7e-85bb-83848cd35a2c"
812024-12-06T21:25:37.085ZINFOcrucible-agent (datafile): dataset is oxp_afed364c-1eb2-4c62-a6a7-b86cf13ba8c1/crucible/regions/960301fa-fe60-4a7e-85bb-83848cd35a2c
822024-12-06T21:25:37.095ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_afed364c-1eb2-4c62-a6a7-b86cf13ba8c1/crucible/regions/960301fa-fe60-4a7e-85bb-83848cd35a2c@b60da084-ea61-49d4-b78f-f89aa537f063
832024-12-06T21:25:37.095ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_afed364c-1eb2-4c62-a6a7-b86cf13ba8c1/crucible/regions/960301fa-fe60-4a7e-85bb-83848cd35a2c", "b60da084-ea61-49d4-b78f-f89aa537f063"]
842024-12-06T21:25:37.107ZINFOcrucible-agent (dropshot): request completed latency_us = 30411 local_addr = [fd00:1122:3344:101::14]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:34806 req_id = d07e8579-8ad1-455a-af4a-e431612db4f7 response_code = 200 uri = /crucible/0/regions/960301fa-fe60-4a7e-85bb-83848cd35a2c/snapshots/b60da084-ea61-49d4-b78f-f89aa537f063
852024-12-06T21:25:37.110ZINFOcrucible-agent (datafile): path is "/data/regions/960301fa-fe60-4a7e-85bb-83848cd35a2c"
862024-12-06T21:25:37.118ZINFOcrucible-agent (datafile): dataset is oxp_afed364c-1eb2-4c62-a6a7-b86cf13ba8c1/crucible/regions/960301fa-fe60-4a7e-85bb-83848cd35a2c
872024-12-06T21:25:37.126ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_afed364c-1eb2-4c62-a6a7-b86cf13ba8c1/crucible/regions/960301fa-fe60-4a7e-85bb-83848cd35a2c@b60da084-ea61-49d4-b78f-f89aa537f063
882024-12-06T21:25:37.126ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_afed364c-1eb2-4c62-a6a7-b86cf13ba8c1/crucible/regions/960301fa-fe60-4a7e-85bb-83848cd35a2c", "b60da084-ea61-49d4-b78f-f89aa537f063"]
892024-12-06T21:25:37.135ZINFOcrucible-agent (datafile): requesting running snapshot 960301fa-fe60-4a7e-85bb-83848cd35a2c-b60da084-ea61-49d4-b78f-f89aa537f063 state: Requested
902024-12-06T21:25:37.135ZINFOcrucible-agent (worker): applying SMF actions for region 960301fa-fe60-4a7e-85bb-83848cd35a2c running snapshot b60da084-ea61-49d4-b78f-f89aa537f063 (state Requested)...
912024-12-06T21:25:37.135ZINFOcrucible-agent (dropshot): request completed latency_us = 24981 local_addr = [fd00:1122:3344:101::14]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:34806 req_id = 5ce36f3b-fdf7-43da-8ef8-af4bf329d5dc response_code = 200 uri = /crucible/0/regions/960301fa-fe60-4a7e-85bb-83848cd35a2c/snapshots/b60da084-ea61-49d4-b78f-f89aa537f063/run
922024-12-06T21:25:37.138ZINFOcrucible-agent (worker): creating missing snapshot instance snapshot-960301fa-fe60-4a7e-85bb-83848cd35a2c-b60da084-ea61-49d4-b78f-f89aa537f063
932024-12-06T21:25:37.141ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:snapshot-960301fa-fe60-4a7e-85bb-83848cd35a2c-b60da084-ea61-49d4-b78f-f89aa537f063
942024-12-06T21:25:37.152ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:snapshot-960301fa-fe60-4a7e-85bb-83848cd35a2c-b60da084-ea61-49d4-b78f-f89aa537f063
952024-12-06T21:25:37.152ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/960301fa-fe60-4a7e-85bb-83848cd35a2c/.zfs/snapshot/b60da084-ea61-49d4-b78f-f89aa537f063
962024-12-06T21:25:37.152ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19002
972024-12-06T21:25:37.152ZINFOcrucible-agent (worker): ensure mode SCF_TYPE_ASTRING ro
982024-12-06T21:25:37.152ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::14
992024-12-06T21:25:37.152ZINFOcrucible-agent (worker): commit
1002024-12-06T21:25:37.157ZINFOcrucible-agent (worker): ok!
1012024-12-06T21:25:37.162ZINFOcrucible-agent (worker): SMF ok!
1022024-12-06T21:25:37.162ZINFOcrucible-agent (datafile): region 960301fa-fe60-4a7e-85bb-83848cd35a2c running snapshot b60da084-ea61-49d4-b78f-f89aa537f063 state: Requested -> Created
1032024-12-06T21:25:37.449ZINFOcrucible-agent (datafile): path is "/data/regions/960301fa-fe60-4a7e-85bb-83848cd35a2c"
1042024-12-06T21:25:37.458ZINFOcrucible-agent (datafile): dataset is oxp_afed364c-1eb2-4c62-a6a7-b86cf13ba8c1/crucible/regions/960301fa-fe60-4a7e-85bb-83848cd35a2c
1052024-12-06T21:25:37.467ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_afed364c-1eb2-4c62-a6a7-b86cf13ba8c1/crucible/regions/960301fa-fe60-4a7e-85bb-83848cd35a2c@b60da084-ea61-49d4-b78f-f89aa537f063
1062024-12-06T21:25:37.467ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_afed364c-1eb2-4c62-a6a7-b86cf13ba8c1/crucible/regions/960301fa-fe60-4a7e-85bb-83848cd35a2c", "b60da084-ea61-49d4-b78f-f89aa537f063"]
1072024-12-06T21:25:37.476ZINFOcrucible-agent (dropshot): request completed latency_us = 26951 local_addr = [fd00:1122:3344:101::14]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:34806 req_id = 31a0c306-9a94-4ec7-8157-5e5d99044eac response_code = 200 uri = /crucible/0/regions/960301fa-fe60-4a7e-85bb-83848cd35a2c/snapshots/b60da084-ea61-49d4-b78f-f89aa537f063/run