Max level shown:
12024-02-28T07:45:30.862ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::12]:32345 remote_addr = [fd00:1122:3344:101::a]:54414
22024-02-28T07:45:30.901ZINFOcrucible-agent (datafile): region 3fe5eee2-ce80-498c-b063-786e50cf10b7 state: Requested
32024-02-28T07:45:30.901ZINFOcrucible-agent (dropshot): request completed latency_us = 5237 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:54414 req_id = dd282daf-a1e7-4023-b1c0-5bd7dccf69bd response_code = 200 uri = /crucible/0/regions
42024-02-28T07:45:30.902ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
52024-02-28T07:45:30.911ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible/regions/3fe5eee2-ce80-498c-b063-786e50cf10b7
62024-02-28T07:45:30.911ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible/regions/3fe5eee2-ce80-498c-b063-786e50cf10b7
72024-02-28T07:45:30.972ZINFOcrucible-agent (worker): creating region Region { id: RegionId("3fe5eee2-ce80-498c-b063-786e50cf10b7"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None } at "/data/regions/3fe5eee2-ce80-498c-b063-786e50cf10b7" region = 3fe5eee2-ce80-498c-b063-786e50cf10b7
82024-02-28T07:45:31.115ZINFOcrucible-agent (worker): region files created ok region = 3fe5eee2-ce80-498c-b063-786e50cf10b7
92024-02-28T07:45:31.116ZINFOcrucible-agent (datafile): region 3fe5eee2-ce80-498c-b063-786e50cf10b7 state: Requested -> Created
102024-02-28T07:45:31.116ZINFOcrucible-agent (worker): applying SMF actions post create...
112024-02-28T07:45:31.116ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-3fe5eee2-ce80-498c-b063-786e50cf10b7
122024-02-28T07:45:31.118ZINFOcrucible-agent (dropshot): request completed latency_us = 69 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:54414 req_id = 95824352-d100-4d27-9cb2-3c1e2911a0e9 response_code = 200 uri = /crucible/0/regions
132024-02-28T07:45:31.122ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-3fe5eee2-ce80-498c-b063-786e50cf10b7
142024-02-28T07:45:31.133ZINFOcrucible-agent (worker): creating config property group
152024-02-28T07:45:31.136ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-3fe5eee2-ce80-498c-b063-786e50cf10b7
162024-02-28T07:45:31.137ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/3fe5eee2-ce80-498c-b063-786e50cf10b7
172024-02-28T07:45:31.137ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
182024-02-28T07:45:31.137ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::12
192024-02-28T07:45:31.137ZINFOcrucible-agent (worker): commit
202024-02-28T07:45:31.141ZINFOcrucible-agent (worker): ok!
212024-02-28T07:45:31.146ZINFOcrucible-agent (worker): SMF ok!
222024-02-28T07:50:59.672ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::12]:32345 remote_addr = [fd00:1122:3344:101::a]:42498
232024-02-28T07:50:59.702ZINFOcrucible-agent (datafile): region 0b638ac9-2cdf-4431-865c-af4eb389969a state: Requested
242024-02-28T07:50:59.703ZINFOcrucible-agent (dropshot): request completed latency_us = 686 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:42498 req_id = 9719813d-b7c3-4aba-91ca-d14d50df35d9 response_code = 200 uri = /crucible/0/regions
252024-02-28T07:50:59.703ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
262024-02-28T07:50:59.713ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible/regions/0b638ac9-2cdf-4431-865c-af4eb389969a
272024-02-28T07:50:59.713ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible/regions/0b638ac9-2cdf-4431-865c-af4eb389969a
282024-02-28T07:50:59.786ZINFOcrucible-agent (worker): creating region Region { id: RegionId("0b638ac9-2cdf-4431-865c-af4eb389969a"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19001, cert_pem: None, key_pem: None, root_pem: None } at "/data/regions/0b638ac9-2cdf-4431-865c-af4eb389969a" region = 0b638ac9-2cdf-4431-865c-af4eb389969a
292024-02-28T07:50:59.913ZINFOcrucible-agent (dropshot): request completed latency_us = 83 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:42498 req_id = 5b03ab70-7a35-458d-8d2b-dc1e22471cfa response_code = 200 uri = /crucible/0/regions
302024-02-28T07:50:59.914ZINFOcrucible-agent (worker): region files created ok region = 0b638ac9-2cdf-4431-865c-af4eb389969a
312024-02-28T07:50:59.914ZINFOcrucible-agent (datafile): region 0b638ac9-2cdf-4431-865c-af4eb389969a state: Requested -> Created
322024-02-28T07:50:59.914ZINFOcrucible-agent (worker): applying SMF actions post create...
332024-02-28T07:50:59.914ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-0b638ac9-2cdf-4431-865c-af4eb389969a
342024-02-28T07:50:59.926ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-0b638ac9-2cdf-4431-865c-af4eb389969a
352024-02-28T07:50:59.938ZINFOcrucible-agent (worker): creating config property group
362024-02-28T07:50:59.945ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-0b638ac9-2cdf-4431-865c-af4eb389969a
372024-02-28T07:50:59.945ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/0b638ac9-2cdf-4431-865c-af4eb389969a
382024-02-28T07:50:59.945ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
392024-02-28T07:50:59.945ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::12
402024-02-28T07:50:59.945ZINFOcrucible-agent (worker): commit
412024-02-28T07:50:59.957ZINFOcrucible-agent (worker): ok!
422024-02-28T07:50:59.970ZINFOcrucible-agent (worker): SMF ok!
432024-02-28T07:51:00.279ZINFOcrucible-agent (dropshot): request completed latency_us = 82 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:42498 req_id = 17ddf576-613e-40c2-aa9b-70924bec3a1a response_code = 200 uri = /crucible/0/regions
442024-02-28T07:51:01.004ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::12]:32345 remote_addr = [fd00:1122:3344:101::a]:46713
452024-02-28T07:51:01.009ZINFOcrucible-agent (dropshot): request completed latency_us = 5219 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:46713 req_id = 784444b4-6551-47a7-8fa1-55ad788c401e response_code = 200 uri = /crucible/0/regions/3fe5eee2-ce80-498c-b063-786e50cf10b7
462024-02-28T07:51:01.010ZINFOcrucible-agent (datafile): path is "/data/regions/3fe5eee2-ce80-498c-b063-786e50cf10b7"
472024-02-28T07:51:01.018ZINFOcrucible-agent (datafile): dataset is oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible/regions/3fe5eee2-ce80-498c-b063-786e50cf10b7
482024-02-28T07:51:01.027ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible/regions/3fe5eee2-ce80-498c-b063-786e50cf10b7@519ce304-61ed-4030-b292-02f079a1f21c
492024-02-28T07:51:01.027ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible/regions/3fe5eee2-ce80-498c-b063-786e50cf10b7", "519ce304-61ed-4030-b292-02f079a1f21c"]
502024-02-28T07:51:01.038ZINFOcrucible-agent (dropshot): request completed latency_us = 27842 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:46713 req_id = c79306c0-6991-452c-b505-7f2d42edc44f response_code = 200 uri = /crucible/0/regions/3fe5eee2-ce80-498c-b063-786e50cf10b7/snapshots/519ce304-61ed-4030-b292-02f079a1f21c
512024-02-28T07:51:01.038ZINFOcrucible-agent (datafile): path is "/data/regions/3fe5eee2-ce80-498c-b063-786e50cf10b7"
522024-02-28T07:51:01.045ZINFOcrucible-agent (datafile): dataset is oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible/regions/3fe5eee2-ce80-498c-b063-786e50cf10b7
532024-02-28T07:51:01.053ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible/regions/3fe5eee2-ce80-498c-b063-786e50cf10b7@519ce304-61ed-4030-b292-02f079a1f21c
542024-02-28T07:51:01.053ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible/regions/3fe5eee2-ce80-498c-b063-786e50cf10b7", "519ce304-61ed-4030-b292-02f079a1f21c"]
552024-02-28T07:51:01.062ZINFOcrucible-agent (datafile): requesting running snapshot 3fe5eee2-ce80-498c-b063-786e50cf10b7-519ce304-61ed-4030-b292-02f079a1f21c state: Requested
562024-02-28T07:51:01.062ZINFOcrucible-agent (worker): applying SMF actions for region 3fe5eee2-ce80-498c-b063-786e50cf10b7 running snapshot 519ce304-61ed-4030-b292-02f079a1f21c (state Requested)...
572024-02-28T07:51:01.062ZINFOcrucible-agent (dropshot): request completed latency_us = 24303 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:46713 req_id = 75bd988d-95b4-4549-96b9-750bd49ecada response_code = 200 uri = /crucible/0/regions/3fe5eee2-ce80-498c-b063-786e50cf10b7/snapshots/519ce304-61ed-4030-b292-02f079a1f21c/run
582024-02-28T07:51:01.064ZINFOcrucible-agent (worker): creating missing snapshot instance snapshot-3fe5eee2-ce80-498c-b063-786e50cf10b7-519ce304-61ed-4030-b292-02f079a1f21c
592024-02-28T07:51:01.077ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:snapshot-3fe5eee2-ce80-498c-b063-786e50cf10b7-519ce304-61ed-4030-b292-02f079a1f21c
602024-02-28T07:51:01.099ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:snapshot-3fe5eee2-ce80-498c-b063-786e50cf10b7-519ce304-61ed-4030-b292-02f079a1f21c
612024-02-28T07:51:01.099ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/3fe5eee2-ce80-498c-b063-786e50cf10b7/.zfs/snapshot/519ce304-61ed-4030-b292-02f079a1f21c
622024-02-28T07:51:01.099ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19002
632024-02-28T07:51:01.099ZINFOcrucible-agent (worker): ensure mode SCF_TYPE_ASTRING ro
642024-02-28T07:51:01.099ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::12
652024-02-28T07:51:01.099ZINFOcrucible-agent (worker): commit
662024-02-28T07:51:01.106ZINFOcrucible-agent (worker): ok!
672024-02-28T07:51:01.114ZINFOcrucible-agent (worker): SMF ok!
682024-02-28T07:51:01.114ZINFOcrucible-agent (datafile): region 3fe5eee2-ce80-498c-b063-786e50cf10b7 running snapshot 519ce304-61ed-4030-b292-02f079a1f21c state: Requested -> Created
692024-02-28T07:51:03.989ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::12]:32345 remote_addr = [fd00:1122:3344:101::a]:55672
702024-02-28T07:51:03.989ZINFOcrucible-agent (datafile): region 1d62302a-ed51-40dc-bda4-0cbbda7f1b53 state: Requested
712024-02-28T07:51:03.989ZINFOcrucible-agent (dropshot): request completed latency_us = 257 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:55672 req_id = c333fae9-2a5e-4bf5-85c0-47e42a56399f response_code = 200 uri = /crucible/0/regions
722024-02-28T07:51:03.989ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
732024-02-28T07:51:03.998ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible/regions/1d62302a-ed51-40dc-bda4-0cbbda7f1b53
742024-02-28T07:51:03.998ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible/regions/1d62302a-ed51-40dc-bda4-0cbbda7f1b53
752024-02-28T07:51:04.077ZINFOcrucible-agent (worker): creating region Region { id: RegionId("1d62302a-ed51-40dc-bda4-0cbbda7f1b53"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19003, cert_pem: None, key_pem: None, root_pem: None } at "/data/regions/1d62302a-ed51-40dc-bda4-0cbbda7f1b53" region = 1d62302a-ed51-40dc-bda4-0cbbda7f1b53
762024-02-28T07:51:04.167ZINFOcrucible-agent (dropshot): request completed latency_us = 84 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:55672 req_id = 4192e71f-97c4-4427-9284-d46557580714 response_code = 200 uri = /crucible/0/regions
772024-02-28T07:51:04.170ZINFOcrucible-agent (worker): region files created ok region = 1d62302a-ed51-40dc-bda4-0cbbda7f1b53
782024-02-28T07:51:04.170ZINFOcrucible-agent (datafile): region 1d62302a-ed51-40dc-bda4-0cbbda7f1b53 state: Requested -> Created
792024-02-28T07:51:04.170ZINFOcrucible-agent (worker): applying SMF actions post create...
802024-02-28T07:51:04.171ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-1d62302a-ed51-40dc-bda4-0cbbda7f1b53
812024-02-28T07:51:04.177ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-1d62302a-ed51-40dc-bda4-0cbbda7f1b53
822024-02-28T07:51:04.187ZINFOcrucible-agent (worker): creating config property group
832024-02-28T07:51:04.191ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-1d62302a-ed51-40dc-bda4-0cbbda7f1b53
842024-02-28T07:51:04.191ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/1d62302a-ed51-40dc-bda4-0cbbda7f1b53
852024-02-28T07:51:04.191ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19003
862024-02-28T07:51:04.191ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::12
872024-02-28T07:51:04.191ZINFOcrucible-agent (worker): commit
882024-02-28T07:51:04.197ZINFOcrucible-agent (worker): ok!
892024-02-28T07:51:04.212ZINFOcrucible-agent (worker): SMF ok!
902024-02-28T07:51:04.521ZINFOcrucible-agent (dropshot): request completed latency_us = 73 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:55672 req_id = 1a19327e-de3f-42e7-abde-234260bf9c8a response_code = 200 uri = /crucible/0/regions
912024-02-28T07:51:39.366ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::12]:32345 remote_addr = [fd00:1122:3344:101::a]:38883
922024-02-28T07:51:39.366ZINFOcrucible-agent (dropshot): request completed latency_us = 458 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:38883 req_id = e9fcea62-53d9-436d-b414-ea9ad559d30c response_code = 200 uri = /crucible/0/regions/1d62302a-ed51-40dc-bda4-0cbbda7f1b53
932024-02-28T07:51:39.366ZINFOcrucible-agent (datafile): path is "/data/regions/1d62302a-ed51-40dc-bda4-0cbbda7f1b53"
942024-02-28T07:51:39.379ZINFOcrucible-agent (datafile): dataset is oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible/regions/1d62302a-ed51-40dc-bda4-0cbbda7f1b53
952024-02-28T07:51:39.391ZINFOcrucible-agent (datafile): region 1d62302a-ed51-40dc-bda4-0cbbda7f1b53 state: Created -> Tombstoned
962024-02-28T07:51:39.391ZINFOcrucible-agent (worker): applying SMF actions before removal...
972024-02-28T07:51:39.391ZINFOcrucible-agent (dropshot): request completed latency_us = 24597 local_addr = [fd00:1122:3344:101::12]:32345 method = DELETE remote_addr = [fd00:1122:3344:101::a]:38883 req_id = 01d72855-5e7c-4aff-a0ae-a3858615591a response_code = 204 uri = /crucible/0/regions/1d62302a-ed51-40dc-bda4-0cbbda7f1b53
982024-02-28T07:51:39.391ZINFOcrucible-agent (dropshot): request completed latency_us = 29 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:38883 req_id = ce6686c2-cead-49cb-b153-84ddbfcf5cd4 response_code = 200 uri = /crucible/0/regions/1d62302a-ed51-40dc-bda4-0cbbda7f1b53
992024-02-28T07:51:39.392ZINFOcrucible-agent (worker): disabling downstairs instance: downstairs-1d62302a-ed51-40dc-bda4-0cbbda7f1b53 (instance states: (Some(Online), None))
1002024-02-28T07:51:39.399ZINFOcrucible-agent (worker): SMF ok!
1012024-02-28T07:51:39.408ZINFOcrucible-agent (worker): deleting zfs dataset "oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible/regions/1d62302a-ed51-40dc-bda4-0cbbda7f1b53" region = 1d62302a-ed51-40dc-bda4-0cbbda7f1b53
1022024-02-28T07:51:39.443ZINFOcrucible-agent (dropshot): request completed latency_us = 63 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:38883 req_id = d1252f4f-08ef-4739-85d3-8702502c2191 response_code = 200 uri = /crucible/0/regions/1d62302a-ed51-40dc-bda4-0cbbda7f1b53
1032024-02-28T07:51:39.465ZINFOcrucible-agent (datafile): region 1d62302a-ed51-40dc-bda4-0cbbda7f1b53 state: Tombstoned -> Destroyed
1042024-02-28T07:51:39.544ZINFOcrucible-agent (dropshot): request completed latency_us = 46 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:38883 req_id = 8993cd3e-9600-4969-a6d1-8899f6b8ce00 response_code = 200 uri = /crucible/0/regions/1d62302a-ed51-40dc-bda4-0cbbda7f1b53