Max level shown:
12024-02-10T01:00:53.241ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::16]:32345 remote_addr = [fd00:1122:3344:101::a]:48581
22024-02-10T01:00:53.277ZINFOcrucible-agent (datafile): region b4dd44b7-14fd-4fd1-ae5c-8b9cc2a7ad36 state: Requested
32024-02-10T01:00:53.284ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
42024-02-10T01:00:53.290ZINFOcrucible-agent (dropshot): request completed latency_us = 27618 local_addr = [fd00:1122:3344:101::16]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:48581 req_id = 110ff252-daab-4a07-9647-2f3a40fef9e0 response_code = 200 uri = /crucible/0/regions
52024-02-10T01:00:53.294ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crucible/regions/b4dd44b7-14fd-4fd1-ae5c-8b9cc2a7ad36
62024-02-10T01:00:53.294ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crucible/regions/b4dd44b7-14fd-4fd1-ae5c-8b9cc2a7ad36
72024-02-10T01:00:53.347ZINFOcrucible-agent (worker): creating region Region { id: RegionId("b4dd44b7-14fd-4fd1-ae5c-8b9cc2a7ad36"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None } at "/data/regions/b4dd44b7-14fd-4fd1-ae5c-8b9cc2a7ad36" region = b4dd44b7-14fd-4fd1-ae5c-8b9cc2a7ad36
82024-02-10T01:00:53.499ZINFOcrucible-agent (worker): region files created ok region = b4dd44b7-14fd-4fd1-ae5c-8b9cc2a7ad36
92024-02-10T01:00:53.499ZINFOcrucible-agent (datafile): region b4dd44b7-14fd-4fd1-ae5c-8b9cc2a7ad36 state: Requested -> Created
102024-02-10T01:00:53.499ZINFOcrucible-agent (worker): applying SMF actions post create...
112024-02-10T01:00:53.499ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-b4dd44b7-14fd-4fd1-ae5c-8b9cc2a7ad36
122024-02-10T01:00:53.508ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-b4dd44b7-14fd-4fd1-ae5c-8b9cc2a7ad36
132024-02-10T01:00:53.527ZINFOcrucible-agent (worker): creating config property group
142024-02-10T01:00:53.531ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-b4dd44b7-14fd-4fd1-ae5c-8b9cc2a7ad36
152024-02-10T01:00:53.532ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/b4dd44b7-14fd-4fd1-ae5c-8b9cc2a7ad36
162024-02-10T01:00:53.532ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
172024-02-10T01:00:53.532ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::16
182024-02-10T01:00:53.532ZINFOcrucible-agent (worker): commit
192024-02-10T01:00:53.536ZINFOcrucible-agent (worker): ok!
202024-02-10T01:00:53.548ZINFOcrucible-agent (dropshot): request completed latency_us = 874 local_addr = [fd00:1122:3344:101::16]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:48581 req_id = 1f9cf16b-74aa-415b-8f34-586ffa4c89ba response_code = 200 uri = /crucible/0/regions
212024-02-10T01:00:53.548ZINFOcrucible-agent (worker): SMF ok!
222024-02-10T01:06:24.045ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::16]:32345 remote_addr = [fd00:1122:3344:101::a]:58290
232024-02-10T01:06:24.045ZINFOcrucible-agent (datafile): region 1ec6b31b-12de-4a44-b129-b992b581f7a7 state: Requested
242024-02-10T01:06:24.052ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
252024-02-10T01:06:24.052ZINFOcrucible-agent (dropshot): request completed latency_us = 7009 local_addr = [fd00:1122:3344:101::16]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:58290 req_id = 33ca8870-2cda-4d99-8ddc-0b4053322b02 response_code = 200 uri = /crucible/0/regions
262024-02-10T01:06:24.062ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crucible/regions/1ec6b31b-12de-4a44-b129-b992b581f7a7
272024-02-10T01:06:24.062ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crucible/regions/1ec6b31b-12de-4a44-b129-b992b581f7a7
282024-02-10T01:06:24.216ZINFOcrucible-agent (worker): creating region Region { id: RegionId("1ec6b31b-12de-4a44-b129-b992b581f7a7"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19001, cert_pem: None, key_pem: None, root_pem: None } at "/data/regions/1ec6b31b-12de-4a44-b129-b992b581f7a7" region = 1ec6b31b-12de-4a44-b129-b992b581f7a7
292024-02-10T01:06:24.377ZINFOcrucible-agent (dropshot): request completed latency_us = 159 local_addr = [fd00:1122:3344:101::16]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:58290 req_id = 9183fa43-398c-46b6-81c0-ee41e1e33912 response_code = 200 uri = /crucible/0/regions
302024-02-10T01:06:24.622ZINFOcrucible-agent (worker): region files created ok region = 1ec6b31b-12de-4a44-b129-b992b581f7a7
312024-02-10T01:06:24.622ZINFOcrucible-agent (datafile): region 1ec6b31b-12de-4a44-b129-b992b581f7a7 state: Requested -> Created
322024-02-10T01:06:24.622ZINFOcrucible-agent (worker): applying SMF actions post create...
332024-02-10T01:06:24.622ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-1ec6b31b-12de-4a44-b129-b992b581f7a7
342024-02-10T01:06:24.680ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-1ec6b31b-12de-4a44-b129-b992b581f7a7
352024-02-10T01:06:24.736ZINFOcrucible-agent (worker): creating config property group
362024-02-10T01:06:24.758ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-1ec6b31b-12de-4a44-b129-b992b581f7a7
372024-02-10T01:06:24.758ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/1ec6b31b-12de-4a44-b129-b992b581f7a7
382024-02-10T01:06:24.758ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
392024-02-10T01:06:24.758ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::16
402024-02-10T01:06:24.758ZINFOcrucible-agent (worker): commit
412024-02-10T01:06:24.774ZINFOcrucible-agent (worker): ok!
422024-02-10T01:06:24.804ZINFOcrucible-agent (worker): SMF ok!
432024-02-10T01:06:25.025ZINFOcrucible-agent (dropshot): request completed latency_us = 100 local_addr = [fd00:1122:3344:101::16]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:58290 req_id = c2bf17ae-b1d8-4c51-b0b7-cf88939b45fe response_code = 200 uri = /crucible/0/regions
442024-02-10T01:06:26.037ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::16]:32345 remote_addr = [fd00:1122:3344:101::a]:39227
452024-02-10T01:06:26.037ZINFOcrucible-agent (dropshot): request completed latency_us = 84 local_addr = [fd00:1122:3344:101::16]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:39227 req_id = 58db79fb-f32e-427d-8d6e-77e694f8aa11 response_code = 200 uri = /crucible/0/regions/b4dd44b7-14fd-4fd1-ae5c-8b9cc2a7ad36
462024-02-10T01:06:26.037ZINFOcrucible-agent (datafile): path is "/data/regions/b4dd44b7-14fd-4fd1-ae5c-8b9cc2a7ad36"
472024-02-10T01:06:26.046ZINFOcrucible-agent (datafile): dataset is oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crucible/regions/b4dd44b7-14fd-4fd1-ae5c-8b9cc2a7ad36
482024-02-10T01:06:26.055ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crucible/regions/b4dd44b7-14fd-4fd1-ae5c-8b9cc2a7ad36@fb4d62a3-a3c9-48d2-a9b8-a73ced585137
492024-02-10T01:06:26.055ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crucible/regions/b4dd44b7-14fd-4fd1-ae5c-8b9cc2a7ad36", "fb4d62a3-a3c9-48d2-a9b8-a73ced585137"]
502024-02-10T01:06:26.065ZINFOcrucible-agent (dropshot): request completed latency_us = 28061 local_addr = [fd00:1122:3344:101::16]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:39227 req_id = eac22f8e-f36e-42c2-b813-d703ea7c4a14 response_code = 200 uri = /crucible/0/regions/b4dd44b7-14fd-4fd1-ae5c-8b9cc2a7ad36/snapshots/fb4d62a3-a3c9-48d2-a9b8-a73ced585137
512024-02-10T01:06:26.065ZINFOcrucible-agent (datafile): path is "/data/regions/b4dd44b7-14fd-4fd1-ae5c-8b9cc2a7ad36"
522024-02-10T01:06:26.073ZINFOcrucible-agent (datafile): dataset is oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crucible/regions/b4dd44b7-14fd-4fd1-ae5c-8b9cc2a7ad36
532024-02-10T01:06:26.080ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crucible/regions/b4dd44b7-14fd-4fd1-ae5c-8b9cc2a7ad36@fb4d62a3-a3c9-48d2-a9b8-a73ced585137
542024-02-10T01:06:26.080ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crucible/regions/b4dd44b7-14fd-4fd1-ae5c-8b9cc2a7ad36", "fb4d62a3-a3c9-48d2-a9b8-a73ced585137"]
552024-02-10T01:06:26.087ZINFOcrucible-agent (datafile): requesting running snapshot b4dd44b7-14fd-4fd1-ae5c-8b9cc2a7ad36-fb4d62a3-a3c9-48d2-a9b8-a73ced585137 state: Requested
562024-02-10T01:06:26.087ZINFOcrucible-agent (worker): applying SMF actions for region b4dd44b7-14fd-4fd1-ae5c-8b9cc2a7ad36 running snapshot fb4d62a3-a3c9-48d2-a9b8-a73ced585137 (state Requested)...
572024-02-10T01:06:26.088ZINFOcrucible-agent (dropshot): request completed latency_us = 22335 local_addr = [fd00:1122:3344:101::16]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:39227 req_id = fbeb278a-2333-4dd7-b248-47dbf2b7d461 response_code = 200 uri = /crucible/0/regions/b4dd44b7-14fd-4fd1-ae5c-8b9cc2a7ad36/snapshots/fb4d62a3-a3c9-48d2-a9b8-a73ced585137/run
582024-02-10T01:06:26.090ZINFOcrucible-agent (worker): creating missing snapshot instance snapshot-b4dd44b7-14fd-4fd1-ae5c-8b9cc2a7ad36-fb4d62a3-a3c9-48d2-a9b8-a73ced585137
592024-02-10T01:06:26.113ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:snapshot-b4dd44b7-14fd-4fd1-ae5c-8b9cc2a7ad36-fb4d62a3-a3c9-48d2-a9b8-a73ced585137
602024-02-10T01:06:26.142ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:snapshot-b4dd44b7-14fd-4fd1-ae5c-8b9cc2a7ad36-fb4d62a3-a3c9-48d2-a9b8-a73ced585137
612024-02-10T01:06:26.142ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/b4dd44b7-14fd-4fd1-ae5c-8b9cc2a7ad36/.zfs/snapshot/fb4d62a3-a3c9-48d2-a9b8-a73ced585137
622024-02-10T01:06:26.142ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19002
632024-02-10T01:06:26.142ZINFOcrucible-agent (worker): ensure mode SCF_TYPE_ASTRING ro
642024-02-10T01:06:26.143ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::16
652024-02-10T01:06:26.143ZINFOcrucible-agent (worker): commit
662024-02-10T01:06:26.166ZINFOcrucible-agent (worker): ok!
672024-02-10T01:06:26.179ZINFOcrucible-agent (worker): SMF ok!
682024-02-10T01:06:26.179ZINFOcrucible-agent (datafile): region b4dd44b7-14fd-4fd1-ae5c-8b9cc2a7ad36 running snapshot fb4d62a3-a3c9-48d2-a9b8-a73ced585137 state: Requested -> Created
692024-02-10T01:06:31.109ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::16]:32345 remote_addr = [fd00:1122:3344:101::a]:38165
702024-02-10T01:06:31.139ZINFOcrucible-agent (datafile): region 41296c4c-138c-485e-b2e9-ace4530b10d5 state: Requested
712024-02-10T01:06:31.139ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
722024-02-10T01:06:31.139ZINFOcrucible-agent (dropshot): request completed latency_us = 747 local_addr = [fd00:1122:3344:101::16]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:38165 req_id = b5ea2834-86e8-46c4-aca9-ce7cf72ca0ec response_code = 200 uri = /crucible/0/regions
732024-02-10T01:06:31.148ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crucible/regions/41296c4c-138c-485e-b2e9-ace4530b10d5
742024-02-10T01:06:31.148ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crucible/regions/41296c4c-138c-485e-b2e9-ace4530b10d5
752024-02-10T01:06:31.333ZINFOcrucible-agent (worker): creating region Region { id: RegionId("41296c4c-138c-485e-b2e9-ace4530b10d5"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19003, cert_pem: None, key_pem: None, root_pem: None } at "/data/regions/41296c4c-138c-485e-b2e9-ace4530b10d5" region = 41296c4c-138c-485e-b2e9-ace4530b10d5
762024-02-10T01:06:31.512ZINFOcrucible-agent (dropshot): request completed latency_us = 78 local_addr = [fd00:1122:3344:101::16]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:38165 req_id = 4358a125-f299-491b-b843-ef89d73011a1 response_code = 200 uri = /crucible/0/regions
772024-02-10T01:06:31.725ZINFOcrucible-agent (worker): region files created ok region = 41296c4c-138c-485e-b2e9-ace4530b10d5
782024-02-10T01:06:31.725ZINFOcrucible-agent (datafile): region 41296c4c-138c-485e-b2e9-ace4530b10d5 state: Requested -> Created
792024-02-10T01:06:31.725ZINFOcrucible-agent (worker): applying SMF actions post create...
802024-02-10T01:06:31.726ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-41296c4c-138c-485e-b2e9-ace4530b10d5
812024-02-10T01:06:31.748ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-41296c4c-138c-485e-b2e9-ace4530b10d5
822024-02-10T01:06:31.771ZINFOcrucible-agent (worker): creating config property group
832024-02-10T01:06:31.783ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-41296c4c-138c-485e-b2e9-ace4530b10d5
842024-02-10T01:06:31.783ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/41296c4c-138c-485e-b2e9-ace4530b10d5
852024-02-10T01:06:31.783ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19003
862024-02-10T01:06:31.783ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::16
872024-02-10T01:06:31.783ZINFOcrucible-agent (worker): commit
882024-02-10T01:06:31.791ZINFOcrucible-agent (worker): ok!
892024-02-10T01:06:31.809ZINFOcrucible-agent (worker): SMF ok!
902024-02-10T01:06:32.030ZINFOcrucible-agent (dropshot): request completed latency_us = 65 local_addr = [fd00:1122:3344:101::16]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:38165 req_id = c8be24ae-5e07-452d-8ff7-9075ed871392 response_code = 200 uri = /crucible/0/regions
912024-02-10T01:07:03.560ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::16]:32345 remote_addr = [fd00:1122:3344:101::a]:50175
922024-02-10T01:07:03.560ZINFOcrucible-agent (dropshot): request completed latency_us = 48 local_addr = [fd00:1122:3344:101::16]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:50175 req_id = d27eeec1-4fb7-4214-b71f-02b65878b458 response_code = 200 uri = /crucible/0/regions/41296c4c-138c-485e-b2e9-ace4530b10d5
932024-02-10T01:07:03.560ZINFOcrucible-agent (datafile): path is "/data/regions/41296c4c-138c-485e-b2e9-ace4530b10d5"
942024-02-10T01:07:03.571ZINFOcrucible-agent (datafile): dataset is oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crucible/regions/41296c4c-138c-485e-b2e9-ace4530b10d5
952024-02-10T01:07:03.579ZINFOcrucible-agent (datafile): region 41296c4c-138c-485e-b2e9-ace4530b10d5 state: Created -> Tombstoned
962024-02-10T01:07:03.579ZINFOcrucible-agent (worker): applying SMF actions before removal...
972024-02-10T01:07:03.579ZINFOcrucible-agent (dropshot): request completed latency_us = 18824 local_addr = [fd00:1122:3344:101::16]:32345 method = DELETE remote_addr = [fd00:1122:3344:101::a]:50175 req_id = 2eaca449-1c30-408d-861a-070ed89d2015 response_code = 204 uri = /crucible/0/regions/41296c4c-138c-485e-b2e9-ace4530b10d5
982024-02-10T01:07:03.579ZINFOcrucible-agent (dropshot): request completed latency_us = 27 local_addr = [fd00:1122:3344:101::16]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:50175 req_id = 6319571f-6c5d-48d4-b1df-a1a03f21977e response_code = 200 uri = /crucible/0/regions/41296c4c-138c-485e-b2e9-ace4530b10d5
992024-02-10T01:07:03.583ZINFOcrucible-agent (worker): disabling downstairs instance: downstairs-41296c4c-138c-485e-b2e9-ace4530b10d5 (instance states: (Some(Online), None))
1002024-02-10T01:07:03.591ZINFOcrucible-agent (worker): SMF ok!
1012024-02-10T01:07:03.601ZINFOcrucible-agent (worker): deleting zfs dataset "oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crucible/regions/41296c4c-138c-485e-b2e9-ace4530b10d5" region = 41296c4c-138c-485e-b2e9-ace4530b10d5
1022024-02-10T01:07:03.659ZINFOcrucible-agent (datafile): region 41296c4c-138c-485e-b2e9-ace4530b10d5 state: Tombstoned -> Destroyed
1032024-02-10T01:07:03.663ZINFOcrucible-agent (dropshot): request completed latency_us = 84 local_addr = [fd00:1122:3344:101::16]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:50175 req_id = 85a16b86-6bbd-4ba6-ba64-30dfce4c20e4 response_code = 200 uri = /crucible/0/regions/41296c4c-138c-485e-b2e9-ace4530b10d5