Max level shown:
12024-05-14T09:46:05.522ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::17]:32345 remote_addr = [fd00:1122:3344:101::a]:38803
22024-05-14T09:46:05.527ZINFOcrucible-agent (datafile): region fba13271-1916-46b8-9686-889130427141 state: Requested
32024-05-14T09:46:05.528ZINFOcrucible-agent (dropshot): request completed latency_us = 580 local_addr = [fd00:1122:3344:101::17]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:38803 req_id = a8191d02-eb3a-4436-812e-548acdb7066f response_code = 200 uri = /crucible/0/regions
42024-05-14T09:46:05.530ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
52024-05-14T09:46:05.540ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_ae2f89f7-35c8-4c05-940c-cdee6d8f8a12/crucible/regions/fba13271-1916-46b8-9686-889130427141
62024-05-14T09:46:05.541ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_ae2f89f7-35c8-4c05-940c-cdee6d8f8a12/crucible/regions/fba13271-1916-46b8-9686-889130427141
72024-05-14T09:46:05.595ZINFOcrucible-agent (worker): creating region Region { id: RegionId("fba13271-1916-46b8-9686-889130427141"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/fba13271-1916-46b8-9686-889130427141" region = fba13271-1916-46b8-9686-889130427141
82024-05-14T09:46:05.595ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "fba13271-1916-46b8-9686-889130427141" "--data" "/data/regions/fba13271-1916-46b8-9686-889130427141" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = fba13271-1916-46b8-9686-889130427141
92024-05-14T09:46:05.664ZINFOcrucible-agent (dropshot): request completed latency_us = 76 local_addr = [fd00:1122:3344:101::17]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:38803 req_id = f9faa6c7-0e47-40fa-b36b-3cb5a45c2c14 response_code = 200 uri = /crucible/0/regions
102024-05-14T09:46:05.754ZINFOcrucible-agent (worker): region files created ok region = fba13271-1916-46b8-9686-889130427141
112024-05-14T09:46:05.754ZINFOcrucible-agent (datafile): region fba13271-1916-46b8-9686-889130427141 state: Requested -> Created
122024-05-14T09:46:05.754ZINFOcrucible-agent (worker): applying SMF actions post create...
132024-05-14T09:46:05.755ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-fba13271-1916-46b8-9686-889130427141
142024-05-14T09:46:05.759ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-fba13271-1916-46b8-9686-889130427141
152024-05-14T09:46:05.769ZINFOcrucible-agent (worker): creating config property group
162024-05-14T09:46:05.772ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-fba13271-1916-46b8-9686-889130427141
172024-05-14T09:46:05.773ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/fba13271-1916-46b8-9686-889130427141
182024-05-14T09:46:05.773ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
192024-05-14T09:46:05.773ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::17
202024-05-14T09:46:05.773ZINFOcrucible-agent (worker): commit
212024-05-14T09:46:05.778ZINFOcrucible-agent (worker): ok!
222024-05-14T09:46:05.782ZINFOcrucible-agent (worker): SMF ok!
232024-05-14T09:46:06.081ZINFOcrucible-agent (dropshot): request completed latency_us = 83 local_addr = [fd00:1122:3344:101::17]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:38803 req_id = 6ee0c4ff-7134-476b-94de-4e749e7c3af6 response_code = 200 uri = /crucible/0/regions
242024-05-14T09:51:01.706ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::17]:32345 remote_addr = [fd00:1122:3344:101::a]:52743
252024-05-14T09:51:01.706ZINFOcrucible-agent (datafile): region c44b10f3-6767-4b47-9978-2057dfeaaf13 state: Requested
262024-05-14T09:51:01.710ZINFOcrucible-agent (dropshot): request completed latency_us = 4201 local_addr = [fd00:1122:3344:101::17]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:52743 req_id = 663f3f86-13f2-4518-8e9e-f3197510d0f6 response_code = 200 uri = /crucible/0/regions
272024-05-14T09:51:01.710ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
282024-05-14T09:51:01.720ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_ae2f89f7-35c8-4c05-940c-cdee6d8f8a12/crucible/regions/c44b10f3-6767-4b47-9978-2057dfeaaf13
292024-05-14T09:51:01.720ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_ae2f89f7-35c8-4c05-940c-cdee6d8f8a12/crucible/regions/c44b10f3-6767-4b47-9978-2057dfeaaf13
302024-05-14T09:51:01.781ZINFOcrucible-agent (worker): creating region Region { id: RegionId("c44b10f3-6767-4b47-9978-2057dfeaaf13"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19001, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/c44b10f3-6767-4b47-9978-2057dfeaaf13" region = c44b10f3-6767-4b47-9978-2057dfeaaf13
312024-05-14T09:51:01.781ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "c44b10f3-6767-4b47-9978-2057dfeaaf13" "--data" "/data/regions/c44b10f3-6767-4b47-9978-2057dfeaaf13" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = c44b10f3-6767-4b47-9978-2057dfeaaf13
322024-05-14T09:51:01.839ZINFOcrucible-agent (dropshot): request completed latency_us = 79 local_addr = [fd00:1122:3344:101::17]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:52743 req_id = a519c0b0-8f36-4f24-90d6-9b9eee99db18 response_code = 200 uri = /crucible/0/regions
332024-05-14T09:51:01.888ZINFOcrucible-agent (worker): region files created ok region = c44b10f3-6767-4b47-9978-2057dfeaaf13
342024-05-14T09:51:01.888ZINFOcrucible-agent (datafile): region c44b10f3-6767-4b47-9978-2057dfeaaf13 state: Requested -> Created
352024-05-14T09:51:01.888ZINFOcrucible-agent (worker): applying SMF actions post create...
362024-05-14T09:51:01.888ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-c44b10f3-6767-4b47-9978-2057dfeaaf13
372024-05-14T09:51:01.904ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-c44b10f3-6767-4b47-9978-2057dfeaaf13
382024-05-14T09:51:01.920ZINFOcrucible-agent (worker): creating config property group
392024-05-14T09:51:01.924ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-c44b10f3-6767-4b47-9978-2057dfeaaf13
402024-05-14T09:51:01.924ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/c44b10f3-6767-4b47-9978-2057dfeaaf13
412024-05-14T09:51:01.924ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
422024-05-14T09:51:01.924ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::17
432024-05-14T09:51:01.924ZINFOcrucible-agent (worker): commit
442024-05-14T09:51:01.928ZINFOcrucible-agent (worker): ok!
452024-05-14T09:51:01.935ZINFOcrucible-agent (worker): SMF ok!
462024-05-14T09:51:02.094ZINFOcrucible-agent (dropshot): request completed latency_us = 50 local_addr = [fd00:1122:3344:101::17]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:52743 req_id = bf52a36a-6aec-4819-a52a-020ba53e37c6 response_code = 200 uri = /crucible/0/regions
472024-05-14T09:51:03.302ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::17]:32345 remote_addr = [fd00:1122:3344:101::a]:32856
482024-05-14T09:51:03.302ZINFOcrucible-agent (dropshot): request completed latency_us = 61 local_addr = [fd00:1122:3344:101::17]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:32856 req_id = 04a7bc11-85cd-4d5f-8229-6ebae02f9dea response_code = 200 uri = /crucible/0/regions/fba13271-1916-46b8-9686-889130427141
492024-05-14T09:51:03.302ZINFOcrucible-agent (datafile): path is "/data/regions/fba13271-1916-46b8-9686-889130427141"
502024-05-14T09:51:03.311ZINFOcrucible-agent (datafile): dataset is oxp_ae2f89f7-35c8-4c05-940c-cdee6d8f8a12/crucible/regions/fba13271-1916-46b8-9686-889130427141
512024-05-14T09:51:03.319ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_ae2f89f7-35c8-4c05-940c-cdee6d8f8a12/crucible/regions/fba13271-1916-46b8-9686-889130427141@ad51f756-5c32-4544-8d55-47c59456e030
522024-05-14T09:51:03.319ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_ae2f89f7-35c8-4c05-940c-cdee6d8f8a12/crucible/regions/fba13271-1916-46b8-9686-889130427141", "ad51f756-5c32-4544-8d55-47c59456e030"]
532024-05-14T09:51:03.334ZINFOcrucible-agent (dropshot): request completed latency_us = 31632 local_addr = [fd00:1122:3344:101::17]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:32856 req_id = 86c8835c-e0c8-466a-8b98-54c768e78d0b response_code = 200 uri = /crucible/0/regions/fba13271-1916-46b8-9686-889130427141/snapshots/ad51f756-5c32-4544-8d55-47c59456e030
542024-05-14T09:51:03.334ZINFOcrucible-agent (datafile): path is "/data/regions/fba13271-1916-46b8-9686-889130427141"
552024-05-14T09:51:03.343ZINFOcrucible-agent (datafile): dataset is oxp_ae2f89f7-35c8-4c05-940c-cdee6d8f8a12/crucible/regions/fba13271-1916-46b8-9686-889130427141
562024-05-14T09:51:03.351ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_ae2f89f7-35c8-4c05-940c-cdee6d8f8a12/crucible/regions/fba13271-1916-46b8-9686-889130427141@ad51f756-5c32-4544-8d55-47c59456e030
572024-05-14T09:51:03.351ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_ae2f89f7-35c8-4c05-940c-cdee6d8f8a12/crucible/regions/fba13271-1916-46b8-9686-889130427141", "ad51f756-5c32-4544-8d55-47c59456e030"]
582024-05-14T09:51:03.360ZINFOcrucible-agent (datafile): requesting running snapshot fba13271-1916-46b8-9686-889130427141-ad51f756-5c32-4544-8d55-47c59456e030 state: Requested
592024-05-14T09:51:03.361ZINFOcrucible-agent (worker): applying SMF actions for region fba13271-1916-46b8-9686-889130427141 running snapshot ad51f756-5c32-4544-8d55-47c59456e030 (state Requested)...
602024-05-14T09:51:03.361ZINFOcrucible-agent (dropshot): request completed latency_us = 26593 local_addr = [fd00:1122:3344:101::17]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:32856 req_id = d54d5f5a-8b60-4f0b-84e7-432e76af5beb response_code = 200 uri = /crucible/0/regions/fba13271-1916-46b8-9686-889130427141/snapshots/ad51f756-5c32-4544-8d55-47c59456e030/run
612024-05-14T09:51:03.363ZINFOcrucible-agent (worker): creating missing snapshot instance snapshot-fba13271-1916-46b8-9686-889130427141-ad51f756-5c32-4544-8d55-47c59456e030
622024-05-14T09:51:03.368ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:snapshot-fba13271-1916-46b8-9686-889130427141-ad51f756-5c32-4544-8d55-47c59456e030
632024-05-14T09:51:03.377ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:snapshot-fba13271-1916-46b8-9686-889130427141-ad51f756-5c32-4544-8d55-47c59456e030
642024-05-14T09:51:03.377ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/fba13271-1916-46b8-9686-889130427141/.zfs/snapshot/ad51f756-5c32-4544-8d55-47c59456e030
652024-05-14T09:51:03.377ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19002
662024-05-14T09:51:03.377ZINFOcrucible-agent (worker): ensure mode SCF_TYPE_ASTRING ro
672024-05-14T09:51:03.377ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::17
682024-05-14T09:51:03.377ZINFOcrucible-agent (worker): commit
692024-05-14T09:51:03.380ZINFOcrucible-agent (worker): ok!
702024-05-14T09:51:03.385ZINFOcrucible-agent (worker): SMF ok!
712024-05-14T09:51:03.385ZINFOcrucible-agent (datafile): region fba13271-1916-46b8-9686-889130427141 running snapshot ad51f756-5c32-4544-8d55-47c59456e030 state: Requested -> Created
722024-05-14T09:51:06.565ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::17]:32345 remote_addr = [fd00:1122:3344:101::a]:62969
732024-05-14T09:51:06.565ZINFOcrucible-agent (datafile): region edd0f016-41d3-4812-a61b-301f985b822e state: Requested
742024-05-14T09:51:06.565ZINFOcrucible-agent (dropshot): request completed latency_us = 211 local_addr = [fd00:1122:3344:101::17]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:62969 req_id = 46ee1296-1a4e-49e6-92bc-b4cdc731985b response_code = 200 uri = /crucible/0/regions
752024-05-14T09:51:06.565ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
762024-05-14T09:51:06.575ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_ae2f89f7-35c8-4c05-940c-cdee6d8f8a12/crucible/regions/edd0f016-41d3-4812-a61b-301f985b822e
772024-05-14T09:51:06.575ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_ae2f89f7-35c8-4c05-940c-cdee6d8f8a12/crucible/regions/edd0f016-41d3-4812-a61b-301f985b822e
782024-05-14T09:51:06.641ZINFOcrucible-agent (worker): creating region Region { id: RegionId("edd0f016-41d3-4812-a61b-301f985b822e"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19003, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/edd0f016-41d3-4812-a61b-301f985b822e" region = edd0f016-41d3-4812-a61b-301f985b822e
792024-05-14T09:51:06.643ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "edd0f016-41d3-4812-a61b-301f985b822e" "--data" "/data/regions/edd0f016-41d3-4812-a61b-301f985b822e" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = edd0f016-41d3-4812-a61b-301f985b822e
802024-05-14T09:51:06.705ZINFOcrucible-agent (worker): region files created ok region = edd0f016-41d3-4812-a61b-301f985b822e
812024-05-14T09:51:06.705ZINFOcrucible-agent (datafile): region edd0f016-41d3-4812-a61b-301f985b822e state: Requested -> Created
822024-05-14T09:51:06.705ZINFOcrucible-agent (worker): applying SMF actions post create...
832024-05-14T09:51:06.706ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-edd0f016-41d3-4812-a61b-301f985b822e
842024-05-14T09:51:06.709ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-edd0f016-41d3-4812-a61b-301f985b822e
852024-05-14T09:51:06.717ZINFOcrucible-agent (worker): creating config property group
862024-05-14T09:51:06.722ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-edd0f016-41d3-4812-a61b-301f985b822e
872024-05-14T09:51:06.722ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/edd0f016-41d3-4812-a61b-301f985b822e
882024-05-14T09:51:06.722ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19003
892024-05-14T09:51:06.722ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::17
902024-05-14T09:51:06.722ZINFOcrucible-agent (worker): commit
912024-05-14T09:51:06.726ZINFOcrucible-agent (worker): ok!
922024-05-14T09:51:06.736ZINFOcrucible-agent (worker): SMF ok!
932024-05-14T09:51:06.841ZINFOcrucible-agent (dropshot): request completed latency_us = 76 local_addr = [fd00:1122:3344:101::17]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:62969 req_id = 191f2466-1155-4fc3-9512-632263f0c62c response_code = 200 uri = /crucible/0/regions
942024-05-14T09:51:33.135ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::17]:32345 remote_addr = [fd00:1122:3344:101::a]:58061
952024-05-14T09:51:33.136ZINFOcrucible-agent (dropshot): request completed latency_us = 552 local_addr = [fd00:1122:3344:101::17]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:58061 req_id = 8bf5a75f-7e6f-4105-8e87-10be29347e1d response_code = 200 uri = /crucible/0/regions/edd0f016-41d3-4812-a61b-301f985b822e
962024-05-14T09:51:33.136ZINFOcrucible-agent (datafile): path is "/data/regions/edd0f016-41d3-4812-a61b-301f985b822e"
972024-05-14T09:51:33.155ZINFOcrucible-agent (datafile): dataset is oxp_ae2f89f7-35c8-4c05-940c-cdee6d8f8a12/crucible/regions/edd0f016-41d3-4812-a61b-301f985b822e
982024-05-14T09:51:33.164ZINFOcrucible-agent (datafile): region edd0f016-41d3-4812-a61b-301f985b822e state: Created -> Tombstoned
992024-05-14T09:51:33.165ZINFOcrucible-agent (worker): applying SMF actions before removal...
1002024-05-14T09:51:33.166ZINFOcrucible-agent (worker): disabling downstairs instance: downstairs-edd0f016-41d3-4812-a61b-301f985b822e (instance states: (Some(Online), None))
1012024-05-14T09:51:33.169ZINFOcrucible-agent (dropshot): request completed latency_us = 33229 local_addr = [fd00:1122:3344:101::17]:32345 method = DELETE remote_addr = [fd00:1122:3344:101::a]:58061 req_id = aeae532f-dff0-4f3f-b06f-16969182ca3b response_code = 204 uri = /crucible/0/regions/edd0f016-41d3-4812-a61b-301f985b822e
1022024-05-14T09:51:33.169ZINFOcrucible-agent (dropshot): request completed latency_us = 19 local_addr = [fd00:1122:3344:101::17]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:58061 req_id = cbad85eb-b0b7-4aa6-bf88-ba34fd360a04 response_code = 200 uri = /crucible/0/regions/edd0f016-41d3-4812-a61b-301f985b822e
1032024-05-14T09:51:33.187ZINFOcrucible-agent (worker): SMF ok!
1042024-05-14T09:51:33.196ZINFOcrucible-agent (worker): deleting zfs dataset "oxp_ae2f89f7-35c8-4c05-940c-cdee6d8f8a12/crucible/regions/edd0f016-41d3-4812-a61b-301f985b822e" region = edd0f016-41d3-4812-a61b-301f985b822e
1052024-05-14T09:51:33.279ZINFOcrucible-agent (dropshot): request completed latency_us = 109 local_addr = [fd00:1122:3344:101::17]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:58061 req_id = dab86d0c-625c-4f5e-809d-05f56beb6836 response_code = 200 uri = /crucible/0/regions/edd0f016-41d3-4812-a61b-301f985b822e
1062024-05-14T09:51:33.293ZINFOcrucible-agent (datafile): region edd0f016-41d3-4812-a61b-301f985b822e state: Tombstoned -> Destroyed
1072024-05-14T09:51:33.421ZINFOcrucible-agent (dropshot): request completed latency_us = 1119 local_addr = [fd00:1122:3344:101::17]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:58061 req_id = d72f185e-3984-4261-8d58-aaca69a69074 response_code = 200 uri = /crucible/0/regions/edd0f016-41d3-4812-a61b-301f985b822e