Max level shown:
12024-06-12T21:33:32.416ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::12]:32345 remote_addr = [fd00:1122:3344:101::a]:63914
22024-06-12T21:33:32.416ZINFOcrucible-agent (datafile): region 6da0f4fa-395c-4789-ae43-f9fed76a2935 state: Requested
32024-06-12T21:33:32.417ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
42024-06-12T21:33:32.417ZINFOcrucible-agent (dropshot): request completed latency_us = 945 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:63914 req_id = 0499a1f8-bf6e-4a5a-ba08-cda79aaecd8f response_code = 200 uri = /crucible/0/regions
52024-06-12T21:33:32.431ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_fdc5c351-bd0c-490e-a824-bb2f2b967954/crucible/regions/6da0f4fa-395c-4789-ae43-f9fed76a2935
62024-06-12T21:33:32.431ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_fdc5c351-bd0c-490e-a824-bb2f2b967954/crucible/regions/6da0f4fa-395c-4789-ae43-f9fed76a2935
72024-06-12T21:33:32.490ZINFOcrucible-agent (worker): creating region Region { id: RegionId("6da0f4fa-395c-4789-ae43-f9fed76a2935"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19001, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/6da0f4fa-395c-4789-ae43-f9fed76a2935" region = 6da0f4fa-395c-4789-ae43-f9fed76a2935
82024-06-12T21:33:32.490ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "6da0f4fa-395c-4789-ae43-f9fed76a2935" "--data" "/data/regions/6da0f4fa-395c-4789-ae43-f9fed76a2935" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 6da0f4fa-395c-4789-ae43-f9fed76a2935
92024-06-12T21:33:32.553ZINFOcrucible-agent (worker): region files created ok region = 6da0f4fa-395c-4789-ae43-f9fed76a2935
102024-06-12T21:33:32.553ZINFOcrucible-agent (datafile): region 6da0f4fa-395c-4789-ae43-f9fed76a2935 state: Requested -> Created
112024-06-12T21:33:32.553ZINFOcrucible-agent (worker): applying SMF actions post create...
122024-06-12T21:33:32.554ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-6da0f4fa-395c-4789-ae43-f9fed76a2935
132024-06-12T21:33:32.560ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-6da0f4fa-395c-4789-ae43-f9fed76a2935
142024-06-12T21:33:32.571ZINFOcrucible-agent (worker): creating config property group
152024-06-12T21:33:32.575ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-6da0f4fa-395c-4789-ae43-f9fed76a2935
162024-06-12T21:33:32.575ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/6da0f4fa-395c-4789-ae43-f9fed76a2935
172024-06-12T21:33:32.575ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
182024-06-12T21:33:32.575ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::12
192024-06-12T21:33:32.575ZINFOcrucible-agent (worker): commit
202024-06-12T21:33:32.578ZINFOcrucible-agent (worker): ok!
212024-06-12T21:33:32.583ZINFOcrucible-agent (worker): existing directory value /dev/null does not match /data/regions/9a30626b-f599-4b7f-9fd7-a6694dac1be3
222024-06-12T21:33:32.583ZINFOcrucible-agent (worker): existing port value 0 does not match 19000
232024-06-12T21:33:32.583ZINFOcrucible-agent (worker): existing address value 0.0.0.0 does not match fd00:1122:3344:101::12
242024-06-12T21:33:32.583ZINFOcrucible-agent (worker): reconfig required
252024-06-12T21:33:32.583ZINFOcrucible-agent (worker): using existing config property group
262024-06-12T21:33:32.583ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-9a30626b-f599-4b7f-9fd7-a6694dac1be3
272024-06-12T21:33:32.584ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/9a30626b-f599-4b7f-9fd7-a6694dac1be3
282024-06-12T21:33:32.584ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
292024-06-12T21:33:32.584ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::12
302024-06-12T21:33:32.584ZINFOcrucible-agent (worker): commit
312024-06-12T21:33:32.597ZINFOcrucible-agent (worker): ok!
322024-06-12T21:33:32.597ZINFOcrucible-agent (worker): SMF ok!
332024-06-12T21:33:32.605ZINFOcrucible-agent (dropshot): request completed latency_us = 92 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:63914 req_id = eefc1f80-fa52-49ee-a586-0aded79e1d30 response_code = 200 uri = /crucible/0/regions
342024-06-12T21:33:33.369ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::12]:32345 remote_addr = [fd00:1122:3344:101::a]:61935
352024-06-12T21:33:33.372ZINFOcrucible-agent (dropshot): request completed latency_us = 2600 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:61935 req_id = fae6fb52-4226-4cee-ac1f-ea94f979c15a response_code = 200 uri = /crucible/0/regions/9a30626b-f599-4b7f-9fd7-a6694dac1be3
362024-06-12T21:33:33.372ZINFOcrucible-agent (datafile): path is "/data/regions/9a30626b-f599-4b7f-9fd7-a6694dac1be3"
372024-06-12T21:33:33.381ZINFOcrucible-agent (datafile): dataset is oxp_fdc5c351-bd0c-490e-a824-bb2f2b967954/crucible/regions/9a30626b-f599-4b7f-9fd7-a6694dac1be3
382024-06-12T21:33:33.389ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_fdc5c351-bd0c-490e-a824-bb2f2b967954/crucible/regions/9a30626b-f599-4b7f-9fd7-a6694dac1be3@8024e4d8-277d-43cb-b8aa-5dbd6ebde952
392024-06-12T21:33:33.389ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_fdc5c351-bd0c-490e-a824-bb2f2b967954/crucible/regions/9a30626b-f599-4b7f-9fd7-a6694dac1be3", "8024e4d8-277d-43cb-b8aa-5dbd6ebde952"]
402024-06-12T21:33:33.403ZINFOcrucible-agent (dropshot): request completed latency_us = 31533 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:61935 req_id = e3cc9213-9247-4ff7-9847-25fb8ab69522 response_code = 200 uri = /crucible/0/regions/9a30626b-f599-4b7f-9fd7-a6694dac1be3/snapshots/8024e4d8-277d-43cb-b8aa-5dbd6ebde952
412024-06-12T21:33:33.403ZINFOcrucible-agent (datafile): path is "/data/regions/9a30626b-f599-4b7f-9fd7-a6694dac1be3"
422024-06-12T21:33:33.412ZINFOcrucible-agent (datafile): dataset is oxp_fdc5c351-bd0c-490e-a824-bb2f2b967954/crucible/regions/9a30626b-f599-4b7f-9fd7-a6694dac1be3
432024-06-12T21:33:33.420ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_fdc5c351-bd0c-490e-a824-bb2f2b967954/crucible/regions/9a30626b-f599-4b7f-9fd7-a6694dac1be3@8024e4d8-277d-43cb-b8aa-5dbd6ebde952
442024-06-12T21:33:33.420ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_fdc5c351-bd0c-490e-a824-bb2f2b967954/crucible/regions/9a30626b-f599-4b7f-9fd7-a6694dac1be3", "8024e4d8-277d-43cb-b8aa-5dbd6ebde952"]
452024-06-12T21:33:33.430ZINFOcrucible-agent (datafile): requesting running snapshot 9a30626b-f599-4b7f-9fd7-a6694dac1be3-8024e4d8-277d-43cb-b8aa-5dbd6ebde952 state: Requested
462024-06-12T21:33:33.430ZINFOcrucible-agent (worker): applying SMF actions for region 9a30626b-f599-4b7f-9fd7-a6694dac1be3 running snapshot 8024e4d8-277d-43cb-b8aa-5dbd6ebde952 (state Requested)...
472024-06-12T21:33:33.430ZINFOcrucible-agent (dropshot): request completed latency_us = 26631 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:61935 req_id = b425009a-8b04-43cc-bf5e-ca62b65f5f3f response_code = 200 uri = /crucible/0/regions/9a30626b-f599-4b7f-9fd7-a6694dac1be3/snapshots/8024e4d8-277d-43cb-b8aa-5dbd6ebde952/run
482024-06-12T21:33:33.432ZINFOcrucible-agent (worker): existing directory value /dev/null does not match /data/regions/9a30626b-f599-4b7f-9fd7-a6694dac1be3
492024-06-12T21:33:33.432ZINFOcrucible-agent (worker): existing port value 0 does not match 19000
502024-06-12T21:33:33.432ZINFOcrucible-agent (worker): existing address value 0.0.0.0 does not match fd00:1122:3344:101::12
512024-06-12T21:33:33.432ZINFOcrucible-agent (worker): reconfig required
522024-06-12T21:33:33.432ZINFOcrucible-agent (worker): using existing config property group
532024-06-12T21:33:33.432ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-9a30626b-f599-4b7f-9fd7-a6694dac1be3
542024-06-12T21:33:33.432ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/9a30626b-f599-4b7f-9fd7-a6694dac1be3
552024-06-12T21:33:33.432ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
562024-06-12T21:33:33.432ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::12
572024-06-12T21:33:33.432ZINFOcrucible-agent (worker): commit
582024-06-12T21:33:33.436ZINFOcrucible-agent (worker): ok!
592024-06-12T21:33:33.437ZINFOcrucible-agent (worker): creating missing snapshot instance snapshot-9a30626b-f599-4b7f-9fd7-a6694dac1be3-8024e4d8-277d-43cb-b8aa-5dbd6ebde952
602024-06-12T21:33:33.443ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:snapshot-9a30626b-f599-4b7f-9fd7-a6694dac1be3-8024e4d8-277d-43cb-b8aa-5dbd6ebde952
612024-06-12T21:33:33.452ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:snapshot-9a30626b-f599-4b7f-9fd7-a6694dac1be3-8024e4d8-277d-43cb-b8aa-5dbd6ebde952
622024-06-12T21:33:33.452ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/9a30626b-f599-4b7f-9fd7-a6694dac1be3/.zfs/snapshot/8024e4d8-277d-43cb-b8aa-5dbd6ebde952
632024-06-12T21:33:33.452ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19002
642024-06-12T21:33:33.452ZINFOcrucible-agent (worker): ensure mode SCF_TYPE_ASTRING ro
652024-06-12T21:33:33.452ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::12
662024-06-12T21:33:33.452ZINFOcrucible-agent (worker): commit
672024-06-12T21:33:33.456ZINFOcrucible-agent (worker): ok!
682024-06-12T21:33:33.459ZINFOcrucible-agent (worker): SMF ok!
692024-06-12T21:33:33.460ZINFOcrucible-agent (datafile): region 9a30626b-f599-4b7f-9fd7-a6694dac1be3 running snapshot 8024e4d8-277d-43cb-b8aa-5dbd6ebde952 state: Requested -> Created
702024-06-12T21:33:36.219ZINFOcrucible-agent (datafile): region cb444619-1fda-4184-9f95-f2240c35461a state: Requested
712024-06-12T21:33:36.220ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
722024-06-12T21:33:36.220ZINFOcrucible-agent (dropshot): request completed latency_us = 216 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:63914 req_id = 3f464a43-b538-4bd1-8607-792ac38450cb response_code = 200 uri = /crucible/0/regions
732024-06-12T21:33:36.229ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_fdc5c351-bd0c-490e-a824-bb2f2b967954/crucible/regions/cb444619-1fda-4184-9f95-f2240c35461a
742024-06-12T21:33:36.229ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_fdc5c351-bd0c-490e-a824-bb2f2b967954/crucible/regions/cb444619-1fda-4184-9f95-f2240c35461a
752024-06-12T21:33:36.290ZINFOcrucible-agent (worker): creating region Region { id: RegionId("cb444619-1fda-4184-9f95-f2240c35461a"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19003, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/cb444619-1fda-4184-9f95-f2240c35461a" region = cb444619-1fda-4184-9f95-f2240c35461a
762024-06-12T21:33:36.292ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "cb444619-1fda-4184-9f95-f2240c35461a" "--data" "/data/regions/cb444619-1fda-4184-9f95-f2240c35461a" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = cb444619-1fda-4184-9f95-f2240c35461a
772024-06-12T21:33:36.352ZINFOcrucible-agent (worker): region files created ok region = cb444619-1fda-4184-9f95-f2240c35461a
782024-06-12T21:33:36.352ZINFOcrucible-agent (datafile): region cb444619-1fda-4184-9f95-f2240c35461a state: Requested -> Created
792024-06-12T21:33:36.353ZINFOcrucible-agent (worker): applying SMF actions post create...
802024-06-12T21:33:36.354ZINFOcrucible-agent (worker): existing directory value /dev/null does not match /data/regions/9a30626b-f599-4b7f-9fd7-a6694dac1be3
812024-06-12T21:33:36.354ZINFOcrucible-agent (worker): existing port value 0 does not match 19000
822024-06-12T21:33:36.354ZINFOcrucible-agent (worker): existing address value 0.0.0.0 does not match fd00:1122:3344:101::12
832024-06-12T21:33:36.354ZINFOcrucible-agent (worker): reconfig required
842024-06-12T21:33:36.354ZINFOcrucible-agent (worker): using existing config property group
852024-06-12T21:33:36.354ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-9a30626b-f599-4b7f-9fd7-a6694dac1be3
862024-06-12T21:33:36.355ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/9a30626b-f599-4b7f-9fd7-a6694dac1be3
872024-06-12T21:33:36.355ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
882024-06-12T21:33:36.355ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::12
892024-06-12T21:33:36.355ZINFOcrucible-agent (worker): commit
902024-06-12T21:33:36.359ZINFOcrucible-agent (worker): ok!
912024-06-12T21:33:36.360ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-cb444619-1fda-4184-9f95-f2240c35461a
922024-06-12T21:33:36.363ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-cb444619-1fda-4184-9f95-f2240c35461a
932024-06-12T21:33:36.371ZINFOcrucible-agent (worker): creating config property group
942024-06-12T21:33:36.374ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-cb444619-1fda-4184-9f95-f2240c35461a
952024-06-12T21:33:36.375ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/cb444619-1fda-4184-9f95-f2240c35461a
962024-06-12T21:33:36.375ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19003
972024-06-12T21:33:36.375ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::12
982024-06-12T21:33:36.375ZINFOcrucible-agent (worker): commit
992024-06-12T21:33:36.378ZINFOcrucible-agent (worker): ok!
1002024-06-12T21:33:36.383ZINFOcrucible-agent (worker): SMF ok!
1012024-06-12T21:33:36.507ZINFOcrucible-agent (dropshot): request completed latency_us = 62 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:63914 req_id = b4f7ad78-a934-40e3-abca-77ceba52f086 response_code = 200 uri = /crucible/0/regions
1022024-06-12T21:34:01.074ZINFOcrucible-agent (dropshot): request completed latency_us = 747 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:63914 req_id = 0d0cc3fd-8e27-487d-bd8c-2de6f7b429c5 response_code = 200 uri = /crucible/0/regions/cb444619-1fda-4184-9f95-f2240c35461a
1032024-06-12T21:34:01.076ZINFOcrucible-agent (datafile): path is "/data/regions/cb444619-1fda-4184-9f95-f2240c35461a"
1042024-06-12T21:34:01.089ZINFOcrucible-agent (datafile): dataset is oxp_fdc5c351-bd0c-490e-a824-bb2f2b967954/crucible/regions/cb444619-1fda-4184-9f95-f2240c35461a
1052024-06-12T21:34:01.098ZINFOcrucible-agent (datafile): region cb444619-1fda-4184-9f95-f2240c35461a state: Created -> Tombstoned
1062024-06-12T21:34:01.098ZINFOcrucible-agent (worker): applying SMF actions before removal...
1072024-06-12T21:34:01.101ZINFOcrucible-agent (dropshot): request completed latency_us = 24409 local_addr = [fd00:1122:3344:101::12]:32345 method = DELETE remote_addr = [fd00:1122:3344:101::a]:63914 req_id = 311b920f-3288-4b34-b514-9ffbfe15ee3c response_code = 204 uri = /crucible/0/regions/cb444619-1fda-4184-9f95-f2240c35461a
1082024-06-12T21:34:01.102ZINFOcrucible-agent (worker): disabling downstairs instance: downstairs-cb444619-1fda-4184-9f95-f2240c35461a (instance states: (Some(Online), None))
1092024-06-12T21:34:01.104ZINFOcrucible-agent (dropshot): request completed latency_us = 26 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:63914 req_id = 46068de5-54d1-4c5c-b670-2983fca2ba6f response_code = 200 uri = /crucible/0/regions/cb444619-1fda-4184-9f95-f2240c35461a
1102024-06-12T21:34:01.108ZINFOcrucible-agent (worker): existing directory value /dev/null does not match /data/regions/9a30626b-f599-4b7f-9fd7-a6694dac1be3
1112024-06-12T21:34:01.108ZINFOcrucible-agent (worker): existing port value 0 does not match 19000
1122024-06-12T21:34:01.108ZINFOcrucible-agent (worker): existing address value 0.0.0.0 does not match fd00:1122:3344:101::12
1132024-06-12T21:34:01.108ZINFOcrucible-agent (worker): reconfig required
1142024-06-12T21:34:01.108ZINFOcrucible-agent (worker): using existing config property group
1152024-06-12T21:34:01.108ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-9a30626b-f599-4b7f-9fd7-a6694dac1be3
1162024-06-12T21:34:01.108ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/9a30626b-f599-4b7f-9fd7-a6694dac1be3
1172024-06-12T21:34:01.109ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
1182024-06-12T21:34:01.109ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::12
1192024-06-12T21:34:01.109ZINFOcrucible-agent (worker): commit
1202024-06-12T21:34:01.113ZINFOcrucible-agent (worker): ok!
1212024-06-12T21:34:01.114ZINFOcrucible-agent (worker): SMF ok!
1222024-06-12T21:34:01.126ZINFOcrucible-agent (worker): deleting zfs dataset "oxp_fdc5c351-bd0c-490e-a824-bb2f2b967954/crucible/regions/cb444619-1fda-4184-9f95-f2240c35461a" region = cb444619-1fda-4184-9f95-f2240c35461a
1232024-06-12T21:34:01.175ZINFOcrucible-agent (dropshot): request completed latency_us = 81 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:63914 req_id = 6bd36cfd-54c2-4d89-857d-370cd3aba9fc response_code = 200 uri = /crucible/0/regions/cb444619-1fda-4184-9f95-f2240c35461a
1242024-06-12T21:34:01.193ZINFOcrucible-agent (datafile): region cb444619-1fda-4184-9f95-f2240c35461a state: Tombstoned -> Destroyed
1252024-06-12T21:34:01.273ZINFOcrucible-agent (dropshot): request completed latency_us = 30 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:63914 req_id = 71633d81-5300-4eda-b723-2b5d74631385 response_code = 200 uri = /crucible/0/regions/cb444619-1fda-4184-9f95-f2240c35461a