Max level shown:
12024-02-15T22:30:15.059ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::19]:32345 remote_addr = [fd00:1122:3344:101::a]:60923
22024-02-15T22:30:15.097ZINFOcrucible-agent (datafile): region ddd17ee8-a338-4135-8709-754cdab526f1 state: Requested
32024-02-15T22:30:15.099ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
42024-02-15T22:30:15.099ZINFOcrucible-agent (dropshot): request completed latency_us = 8796 local_addr = [fd00:1122:3344:101::19]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:60923 req_id = 77442d99-c7cb-41be-8367-5c8cdb930da5 response_code = 200 uri = /crucible/0/regions
52024-02-15T22:30:15.108ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible/regions/ddd17ee8-a338-4135-8709-754cdab526f1
62024-02-15T22:30:15.108ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible/regions/ddd17ee8-a338-4135-8709-754cdab526f1
72024-02-15T22:30:15.163ZINFOcrucible-agent (worker): creating region Region { id: RegionId("ddd17ee8-a338-4135-8709-754cdab526f1"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None } at "/data/regions/ddd17ee8-a338-4135-8709-754cdab526f1" region = ddd17ee8-a338-4135-8709-754cdab526f1
82024-02-15T22:30:15.253ZINFOcrucible-agent (dropshot): request completed latency_us = 79 local_addr = [fd00:1122:3344:101::19]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:60923 req_id = 6594e52e-d683-45f1-bacd-0f1f5409730b response_code = 200 uri = /crucible/0/regions
92024-02-15T22:30:15.305ZINFOcrucible-agent (worker): region files created ok region = ddd17ee8-a338-4135-8709-754cdab526f1
102024-02-15T22:30:15.305ZINFOcrucible-agent (datafile): region ddd17ee8-a338-4135-8709-754cdab526f1 state: Requested -> Created
112024-02-15T22:30:15.305ZINFOcrucible-agent (worker): applying SMF actions post create...
122024-02-15T22:30:15.306ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-ddd17ee8-a338-4135-8709-754cdab526f1
132024-02-15T22:30:15.313ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-ddd17ee8-a338-4135-8709-754cdab526f1
142024-02-15T22:30:15.323ZINFOcrucible-agent (worker): creating config property group
152024-02-15T22:30:15.326ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-ddd17ee8-a338-4135-8709-754cdab526f1
162024-02-15T22:30:15.327ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/ddd17ee8-a338-4135-8709-754cdab526f1
172024-02-15T22:30:15.327ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
182024-02-15T22:30:15.327ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::19
192024-02-15T22:30:15.327ZINFOcrucible-agent (worker): commit
202024-02-15T22:30:15.331ZINFOcrucible-agent (worker): ok!
212024-02-15T22:30:15.335ZINFOcrucible-agent (worker): SMF ok!
222024-02-15T22:30:15.998ZINFOcrucible-agent (dropshot): request completed latency_us = 84 local_addr = [fd00:1122:3344:101::19]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:60923 req_id = aef68c9d-ad73-4033-99ef-78a3144c0376 response_code = 200 uri = /crucible/0/regions
232024-02-15T22:35:51.274ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::19]:32345 remote_addr = [fd00:1122:3344:101::a]:60556
242024-02-15T22:35:51.304ZINFOcrucible-agent (datafile): region b4b38917-4310-4291-989f-bdde70d84919 state: Requested
252024-02-15T22:35:51.305ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
262024-02-15T22:35:51.305ZINFOcrucible-agent (dropshot): request completed latency_us = 1040 local_addr = [fd00:1122:3344:101::19]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:60556 req_id = f79be197-d317-4415-9188-a9dce5decd40 response_code = 200 uri = /crucible/0/regions
272024-02-15T22:35:51.315ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible/regions/b4b38917-4310-4291-989f-bdde70d84919
282024-02-15T22:35:51.315ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible/regions/b4b38917-4310-4291-989f-bdde70d84919
292024-02-15T22:35:51.377ZINFOcrucible-agent (worker): creating region Region { id: RegionId("b4b38917-4310-4291-989f-bdde70d84919"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19001, cert_pem: None, key_pem: None, root_pem: None } at "/data/regions/b4b38917-4310-4291-989f-bdde70d84919" region = b4b38917-4310-4291-989f-bdde70d84919
302024-02-15T22:35:51.488ZINFOcrucible-agent (worker): region files created ok region = b4b38917-4310-4291-989f-bdde70d84919
312024-02-15T22:35:51.489ZINFOcrucible-agent (datafile): region b4b38917-4310-4291-989f-bdde70d84919 state: Requested -> Created
322024-02-15T22:35:51.489ZINFOcrucible-agent (worker): applying SMF actions post create...
332024-02-15T22:35:51.489ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-b4b38917-4310-4291-989f-bdde70d84919
342024-02-15T22:35:51.499ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-b4b38917-4310-4291-989f-bdde70d84919
352024-02-15T22:35:51.522ZINFOcrucible-agent (worker): creating config property group
362024-02-15T22:35:51.529ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-b4b38917-4310-4291-989f-bdde70d84919
372024-02-15T22:35:51.529ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/b4b38917-4310-4291-989f-bdde70d84919
382024-02-15T22:35:51.529ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
392024-02-15T22:35:51.529ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::19
402024-02-15T22:35:51.529ZINFOcrucible-agent (worker): commit
412024-02-15T22:35:51.531ZINFOcrucible-agent (dropshot): request completed latency_us = 90 local_addr = [fd00:1122:3344:101::19]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:60556 req_id = 593501a9-b76f-4ec0-91ad-e075c3bcfd78 response_code = 200 uri = /crucible/0/regions
422024-02-15T22:35:51.537ZINFOcrucible-agent (worker): ok!
432024-02-15T22:35:51.552ZINFOcrucible-agent (worker): SMF ok!
442024-02-15T22:35:52.767ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::19]:32345 remote_addr = [fd00:1122:3344:101::a]:58148
452024-02-15T22:35:52.767ZINFOcrucible-agent (dropshot): request completed latency_us = 83 local_addr = [fd00:1122:3344:101::19]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:58148 req_id = 252065be-91e5-44c2-b3ce-de107ce7d570 response_code = 200 uri = /crucible/0/regions/ddd17ee8-a338-4135-8709-754cdab526f1
462024-02-15T22:35:52.768ZINFOcrucible-agent (datafile): path is "/data/regions/ddd17ee8-a338-4135-8709-754cdab526f1"
472024-02-15T22:35:52.776ZINFOcrucible-agent (datafile): dataset is oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible/regions/ddd17ee8-a338-4135-8709-754cdab526f1
482024-02-15T22:35:52.785ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible/regions/ddd17ee8-a338-4135-8709-754cdab526f1@a886df7f-b8f2-4e61-87bd-1070ae592dc9
492024-02-15T22:35:52.785ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible/regions/ddd17ee8-a338-4135-8709-754cdab526f1", "a886df7f-b8f2-4e61-87bd-1070ae592dc9"]
502024-02-15T22:35:52.796ZINFOcrucible-agent (dropshot): request completed latency_us = 28290 local_addr = [fd00:1122:3344:101::19]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:58148 req_id = 920c2ce6-657d-4882-96ef-91c117d6c3b9 response_code = 200 uri = /crucible/0/regions/ddd17ee8-a338-4135-8709-754cdab526f1/snapshots/a886df7f-b8f2-4e61-87bd-1070ae592dc9
512024-02-15T22:35:52.799ZINFOcrucible-agent (datafile): path is "/data/regions/ddd17ee8-a338-4135-8709-754cdab526f1"
522024-02-15T22:35:52.806ZINFOcrucible-agent (datafile): dataset is oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible/regions/ddd17ee8-a338-4135-8709-754cdab526f1
532024-02-15T22:35:52.814ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible/regions/ddd17ee8-a338-4135-8709-754cdab526f1@a886df7f-b8f2-4e61-87bd-1070ae592dc9
542024-02-15T22:35:52.814ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible/regions/ddd17ee8-a338-4135-8709-754cdab526f1", "a886df7f-b8f2-4e61-87bd-1070ae592dc9"]
552024-02-15T22:35:52.822ZINFOcrucible-agent (datafile): requesting running snapshot ddd17ee8-a338-4135-8709-754cdab526f1-a886df7f-b8f2-4e61-87bd-1070ae592dc9 state: Requested
562024-02-15T22:35:52.823ZINFOcrucible-agent (worker): applying SMF actions for region ddd17ee8-a338-4135-8709-754cdab526f1 running snapshot a886df7f-b8f2-4e61-87bd-1070ae592dc9 (state Requested)...
572024-02-15T22:35:52.823ZINFOcrucible-agent (dropshot): request completed latency_us = 23793 local_addr = [fd00:1122:3344:101::19]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:58148 req_id = c412266e-51c1-4a19-8fc9-744cc7c1fb2a response_code = 200 uri = /crucible/0/regions/ddd17ee8-a338-4135-8709-754cdab526f1/snapshots/a886df7f-b8f2-4e61-87bd-1070ae592dc9/run
582024-02-15T22:35:52.825ZINFOcrucible-agent (worker): creating missing snapshot instance snapshot-ddd17ee8-a338-4135-8709-754cdab526f1-a886df7f-b8f2-4e61-87bd-1070ae592dc9
592024-02-15T22:35:52.834ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:snapshot-ddd17ee8-a338-4135-8709-754cdab526f1-a886df7f-b8f2-4e61-87bd-1070ae592dc9
602024-02-15T22:35:52.852ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:snapshot-ddd17ee8-a338-4135-8709-754cdab526f1-a886df7f-b8f2-4e61-87bd-1070ae592dc9
612024-02-15T22:35:52.852ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/ddd17ee8-a338-4135-8709-754cdab526f1/.zfs/snapshot/a886df7f-b8f2-4e61-87bd-1070ae592dc9
622024-02-15T22:35:52.852ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19002
632024-02-15T22:35:52.852ZINFOcrucible-agent (worker): ensure mode SCF_TYPE_ASTRING ro
642024-02-15T22:35:52.852ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::19
652024-02-15T22:35:52.852ZINFOcrucible-agent (worker): commit
662024-02-15T22:35:52.859ZINFOcrucible-agent (worker): ok!
672024-02-15T22:35:52.866ZINFOcrucible-agent (worker): SMF ok!
682024-02-15T22:35:52.866ZINFOcrucible-agent (datafile): region ddd17ee8-a338-4135-8709-754cdab526f1 running snapshot a886df7f-b8f2-4e61-87bd-1070ae592dc9 state: Requested -> Created
692024-02-15T22:35:55.906ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::19]:32345 remote_addr = [fd00:1122:3344:101::a]:39791
702024-02-15T22:35:55.966ZINFOcrucible-agent (datafile): region f3efd9b0-3125-4f07-94d9-b0e9965a6fd9 state: Requested
712024-02-15T22:35:55.967ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
722024-02-15T22:35:55.967ZINFOcrucible-agent (dropshot): request completed latency_us = 342 local_addr = [fd00:1122:3344:101::19]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:39791 req_id = a3083bb6-c366-4c66-b611-f343b057826f response_code = 200 uri = /crucible/0/regions
732024-02-15T22:35:55.976ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible/regions/f3efd9b0-3125-4f07-94d9-b0e9965a6fd9
742024-02-15T22:35:55.976ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible/regions/f3efd9b0-3125-4f07-94d9-b0e9965a6fd9
752024-02-15T22:35:56.067ZINFOcrucible-agent (worker): creating region Region { id: RegionId("f3efd9b0-3125-4f07-94d9-b0e9965a6fd9"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19003, cert_pem: None, key_pem: None, root_pem: None } at "/data/regions/f3efd9b0-3125-4f07-94d9-b0e9965a6fd9" region = f3efd9b0-3125-4f07-94d9-b0e9965a6fd9
762024-02-15T22:35:56.096ZINFOcrucible-agent (dropshot): request completed latency_us = 91 local_addr = [fd00:1122:3344:101::19]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:39791 req_id = 9ff1d938-b924-43a5-bdc5-7900f05fb1a9 response_code = 200 uri = /crucible/0/regions
772024-02-15T22:35:56.175ZINFOcrucible-agent (worker): region files created ok region = f3efd9b0-3125-4f07-94d9-b0e9965a6fd9
782024-02-15T22:35:56.175ZINFOcrucible-agent (datafile): region f3efd9b0-3125-4f07-94d9-b0e9965a6fd9 state: Requested -> Created
792024-02-15T22:35:56.175ZINFOcrucible-agent (worker): applying SMF actions post create...
802024-02-15T22:35:56.177ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-f3efd9b0-3125-4f07-94d9-b0e9965a6fd9
812024-02-15T22:35:56.180ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-f3efd9b0-3125-4f07-94d9-b0e9965a6fd9
822024-02-15T22:35:56.197ZINFOcrucible-agent (worker): creating config property group
832024-02-15T22:35:56.203ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-f3efd9b0-3125-4f07-94d9-b0e9965a6fd9
842024-02-15T22:35:56.203ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/f3efd9b0-3125-4f07-94d9-b0e9965a6fd9
852024-02-15T22:35:56.203ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19003
862024-02-15T22:35:56.204ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::19
872024-02-15T22:35:56.204ZINFOcrucible-agent (worker): commit
882024-02-15T22:35:56.210ZINFOcrucible-agent (worker): ok!
892024-02-15T22:35:56.219ZINFOcrucible-agent (worker): SMF ok!
902024-02-15T22:35:56.494ZINFOcrucible-agent (dropshot): request completed latency_us = 66 local_addr = [fd00:1122:3344:101::19]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:39791 req_id = 7ca81eec-9a27-49c1-8bfb-436578836291 response_code = 200 uri = /crucible/0/regions
912024-02-15T22:36:27.359ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::19]:32345 remote_addr = [fd00:1122:3344:101::a]:35888
922024-02-15T22:36:27.359ZINFOcrucible-agent (dropshot): request completed latency_us = 52 local_addr = [fd00:1122:3344:101::19]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:35888 req_id = 0ed32d9e-f481-4c82-998f-4909e25531d6 response_code = 200 uri = /crucible/0/regions/f3efd9b0-3125-4f07-94d9-b0e9965a6fd9
932024-02-15T22:36:27.359ZINFOcrucible-agent (datafile): path is "/data/regions/f3efd9b0-3125-4f07-94d9-b0e9965a6fd9"
942024-02-15T22:36:27.370ZINFOcrucible-agent (datafile): dataset is oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible/regions/f3efd9b0-3125-4f07-94d9-b0e9965a6fd9
952024-02-15T22:36:27.379ZINFOcrucible-agent (datafile): region f3efd9b0-3125-4f07-94d9-b0e9965a6fd9 state: Created -> Tombstoned
962024-02-15T22:36:27.379ZINFOcrucible-agent (worker): applying SMF actions before removal...
972024-02-15T22:36:27.379ZINFOcrucible-agent (dropshot): request completed latency_us = 19467 local_addr = [fd00:1122:3344:101::19]:32345 method = DELETE remote_addr = [fd00:1122:3344:101::a]:35888 req_id = 77c7686c-6c93-4c15-ab58-6507e4ca5a79 response_code = 204 uri = /crucible/0/regions/f3efd9b0-3125-4f07-94d9-b0e9965a6fd9
982024-02-15T22:36:27.379ZINFOcrucible-agent (dropshot): request completed latency_us = 23 local_addr = [fd00:1122:3344:101::19]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:35888 req_id = 9fdfc83d-41d0-498b-9216-0f852334bf47 response_code = 200 uri = /crucible/0/regions/f3efd9b0-3125-4f07-94d9-b0e9965a6fd9
992024-02-15T22:36:27.383ZINFOcrucible-agent (worker): disabling downstairs instance: downstairs-f3efd9b0-3125-4f07-94d9-b0e9965a6fd9 (instance states: (Some(Online), None))
1002024-02-15T22:36:27.391ZINFOcrucible-agent (worker): SMF ok!
1012024-02-15T22:36:27.401ZINFOcrucible-agent (worker): deleting zfs dataset "oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible/regions/f3efd9b0-3125-4f07-94d9-b0e9965a6fd9" region = f3efd9b0-3125-4f07-94d9-b0e9965a6fd9
1022024-02-15T22:36:27.448ZINFOcrucible-agent (datafile): region f3efd9b0-3125-4f07-94d9-b0e9965a6fd9 state: Tombstoned -> Destroyed
1032024-02-15T22:36:27.458ZINFOcrucible-agent (dropshot): request completed latency_us = 622 local_addr = [fd00:1122:3344:101::19]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:35888 req_id = 3881dc20-91b9-4edb-a449-8c089d385a7c response_code = 200 uri = /crucible/0/regions/f3efd9b0-3125-4f07-94d9-b0e9965a6fd9