Max level shown:
12024-01-11T01:01:58.135ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::13]:32345 remote_addr = [fd00:1122:3344:101::a]:52576
22024-01-11T01:01:58.166ZINFOcrucible-agent (datafile): region b799dd57-f706-4779-b0c9-256fbc6d6f08 state: Requested
32024-01-11T01:01:58.166ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
42024-01-11T01:01:58.166ZINFOcrucible-agent (dropshot): request completed latency_us = 743 local_addr = [fd00:1122:3344:101::13]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:52576 req_id = 406bff14-5d92-41fd-b3b1-3df2e9bf3abd response_code = 200 uri = /crucible/0/regions
52024-01-11T01:01:58.177ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible/regions/b799dd57-f706-4779-b0c9-256fbc6d6f08
62024-01-11T01:01:58.177ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible/regions/b799dd57-f706-4779-b0c9-256fbc6d6f08
72024-01-11T01:01:58.240ZINFOcrucible-agent (worker): creating region Region { id: RegionId("b799dd57-f706-4779-b0c9-256fbc6d6f08"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19001, cert_pem: None, key_pem: None, root_pem: None } at "/data/regions/b799dd57-f706-4779-b0c9-256fbc6d6f08" region = b799dd57-f706-4779-b0c9-256fbc6d6f08
82024-01-11T01:01:58.322ZINFOcrucible-agent (worker): region files created ok region = b799dd57-f706-4779-b0c9-256fbc6d6f08
92024-01-11T01:01:58.322ZINFOcrucible-agent (datafile): region b799dd57-f706-4779-b0c9-256fbc6d6f08 state: Requested -> Created
102024-01-11T01:01:58.322ZINFOcrucible-agent (worker): applying SMF actions post create...
112024-01-11T01:01:58.329ZINFOcrucible-agent (worker): existing directory value /dev/null does not match /data/regions/ade455a2-80ad-4e67-a5cd-94e1a13604e5
122024-01-11T01:01:58.329ZINFOcrucible-agent (worker): existing port value 0 does not match 19000
132024-01-11T01:01:58.329ZINFOcrucible-agent (worker): existing address value 0.0.0.0 does not match fd00:1122:3344:101::13
142024-01-11T01:01:58.329ZINFOcrucible-agent (worker): reconfig required
152024-01-11T01:01:58.330ZINFOcrucible-agent (worker): using existing config property group
162024-01-11T01:01:58.330ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-ade455a2-80ad-4e67-a5cd-94e1a13604e5
172024-01-11T01:01:58.330ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/ade455a2-80ad-4e67-a5cd-94e1a13604e5
182024-01-11T01:01:58.330ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
192024-01-11T01:01:58.330ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::13
202024-01-11T01:01:58.330ZINFOcrucible-agent (worker): commit
212024-01-11T01:01:58.339ZINFOcrucible-agent (worker): ok!
222024-01-11T01:01:58.340ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-b799dd57-f706-4779-b0c9-256fbc6d6f08
232024-01-11T01:01:58.343ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-b799dd57-f706-4779-b0c9-256fbc6d6f08
242024-01-11T01:01:58.353ZINFOcrucible-agent (worker): creating config property group
252024-01-11T01:01:58.356ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-b799dd57-f706-4779-b0c9-256fbc6d6f08
262024-01-11T01:01:58.356ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/b799dd57-f706-4779-b0c9-256fbc6d6f08
272024-01-11T01:01:58.356ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
282024-01-11T01:01:58.356ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::13
292024-01-11T01:01:58.356ZINFOcrucible-agent (worker): commit
302024-01-11T01:01:58.360ZINFOcrucible-agent (worker): ok!
312024-01-11T01:01:58.365ZINFOcrucible-agent (worker): SMF ok!
322024-01-11T01:01:58.382ZINFOcrucible-agent (dropshot): request completed latency_us = 73 local_addr = [fd00:1122:3344:101::13]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:52576 req_id = f4a314a8-daf1-4537-b62a-055a4c851748 response_code = 200 uri = /crucible/0/regions
332024-01-11T01:01:58.915ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::13]:32345 remote_addr = [fd00:1122:3344:101::a]:36838
342024-01-11T01:01:58.915ZINFOcrucible-agent (dropshot): request completed latency_us = 87 local_addr = [fd00:1122:3344:101::13]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:36838 req_id = 1ea21aa7-a0a6-4373-9a14-9bf6d51ae71e response_code = 200 uri = /crucible/0/regions/ade455a2-80ad-4e67-a5cd-94e1a13604e5
352024-01-11T01:01:58.917ZINFOcrucible-agent (datafile): path is "/data/regions/ade455a2-80ad-4e67-a5cd-94e1a13604e5"
362024-01-11T01:01:58.927ZINFOcrucible-agent (datafile): dataset is oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible/regions/ade455a2-80ad-4e67-a5cd-94e1a13604e5
372024-01-11T01:01:58.936ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible/regions/ade455a2-80ad-4e67-a5cd-94e1a13604e5@3904e5c5-8a8d-4d9e-9f60-5b9f2fbf39cb
382024-01-11T01:01:58.936ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible/regions/ade455a2-80ad-4e67-a5cd-94e1a13604e5", "3904e5c5-8a8d-4d9e-9f60-5b9f2fbf39cb"]
392024-01-11T01:01:58.948ZINFOcrucible-agent (dropshot): request completed latency_us = 30739 local_addr = [fd00:1122:3344:101::13]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:36838 req_id = c7e13405-c99a-4b1f-b741-4b3c27ea6b84 response_code = 200 uri = /crucible/0/regions/ade455a2-80ad-4e67-a5cd-94e1a13604e5/snapshots/3904e5c5-8a8d-4d9e-9f60-5b9f2fbf39cb
402024-01-11T01:01:58.950ZINFOcrucible-agent (datafile): path is "/data/regions/ade455a2-80ad-4e67-a5cd-94e1a13604e5"
412024-01-11T01:01:58.959ZINFOcrucible-agent (datafile): dataset is oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible/regions/ade455a2-80ad-4e67-a5cd-94e1a13604e5
422024-01-11T01:01:58.967ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible/regions/ade455a2-80ad-4e67-a5cd-94e1a13604e5@3904e5c5-8a8d-4d9e-9f60-5b9f2fbf39cb
432024-01-11T01:01:58.967ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible/regions/ade455a2-80ad-4e67-a5cd-94e1a13604e5", "3904e5c5-8a8d-4d9e-9f60-5b9f2fbf39cb"]
442024-01-11T01:01:58.976ZINFOcrucible-agent (datafile): requesting running snapshot ade455a2-80ad-4e67-a5cd-94e1a13604e5-3904e5c5-8a8d-4d9e-9f60-5b9f2fbf39cb state: Requested
452024-01-11T01:01:58.977ZINFOcrucible-agent (worker): applying SMF actions for region ade455a2-80ad-4e67-a5cd-94e1a13604e5 running snapshot 3904e5c5-8a8d-4d9e-9f60-5b9f2fbf39cb (state Requested)...
462024-01-11T01:01:58.977ZINFOcrucible-agent (dropshot): request completed latency_us = 26278 local_addr = [fd00:1122:3344:101::13]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:36838 req_id = 6a96d47f-0579-4574-89cb-55d9d57f6360 response_code = 200 uri = /crucible/0/regions/ade455a2-80ad-4e67-a5cd-94e1a13604e5/snapshots/3904e5c5-8a8d-4d9e-9f60-5b9f2fbf39cb/run
472024-01-11T01:01:58.977ZINFOcrucible-agent (worker): existing directory value /dev/null does not match /data/regions/ade455a2-80ad-4e67-a5cd-94e1a13604e5
482024-01-11T01:01:58.977ZINFOcrucible-agent (worker): existing port value 0 does not match 19000
492024-01-11T01:01:58.977ZINFOcrucible-agent (worker): existing address value 0.0.0.0 does not match fd00:1122:3344:101::13
502024-01-11T01:01:58.977ZINFOcrucible-agent (worker): reconfig required
512024-01-11T01:01:58.977ZINFOcrucible-agent (worker): using existing config property group
522024-01-11T01:01:58.978ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-ade455a2-80ad-4e67-a5cd-94e1a13604e5
532024-01-11T01:01:58.978ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/ade455a2-80ad-4e67-a5cd-94e1a13604e5
542024-01-11T01:01:58.978ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
552024-01-11T01:01:58.978ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::13
562024-01-11T01:01:58.978ZINFOcrucible-agent (worker): commit
572024-01-11T01:01:58.983ZINFOcrucible-agent (worker): ok!
582024-01-11T01:01:58.984ZINFOcrucible-agent (worker): creating missing snapshot instance snapshot-ade455a2-80ad-4e67-a5cd-94e1a13604e5-3904e5c5-8a8d-4d9e-9f60-5b9f2fbf39cb
592024-01-11T01:01:58.987ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:snapshot-ade455a2-80ad-4e67-a5cd-94e1a13604e5-3904e5c5-8a8d-4d9e-9f60-5b9f2fbf39cb
602024-01-11T01:01:58.997ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:snapshot-ade455a2-80ad-4e67-a5cd-94e1a13604e5-3904e5c5-8a8d-4d9e-9f60-5b9f2fbf39cb
612024-01-11T01:01:58.997ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/ade455a2-80ad-4e67-a5cd-94e1a13604e5/.zfs/snapshot/3904e5c5-8a8d-4d9e-9f60-5b9f2fbf39cb
622024-01-11T01:01:58.997ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19002
632024-01-11T01:01:58.997ZINFOcrucible-agent (worker): ensure mode SCF_TYPE_ASTRING ro
642024-01-11T01:01:58.997ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::13
652024-01-11T01:01:58.997ZINFOcrucible-agent (worker): commit
662024-01-11T01:01:59.000ZINFOcrucible-agent (worker): ok!
672024-01-11T01:01:59.004ZINFOcrucible-agent (worker): SMF ok!
682024-01-11T01:01:59.005ZINFOcrucible-agent (datafile): region ade455a2-80ad-4e67-a5cd-94e1a13604e5 running snapshot 3904e5c5-8a8d-4d9e-9f60-5b9f2fbf39cb state: Requested -> Created
692024-01-11T01:02:01.397ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::13]:32345 remote_addr = [fd00:1122:3344:101::a]:46796
702024-01-11T01:02:01.428ZINFOcrucible-agent (datafile): region d161fa2e-d751-4c1c-9b9b-f07e812b1309 state: Requested
712024-01-11T01:02:01.428ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
722024-01-11T01:02:01.428ZINFOcrucible-agent (dropshot): request completed latency_us = 287 local_addr = [fd00:1122:3344:101::13]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:46796 req_id = a8016d85-4512-4683-addd-c633842080c3 response_code = 200 uri = /crucible/0/regions
732024-01-11T01:02:01.438ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible/regions/d161fa2e-d751-4c1c-9b9b-f07e812b1309
742024-01-11T01:02:01.438ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible/regions/d161fa2e-d751-4c1c-9b9b-f07e812b1309
752024-01-11T01:02:01.498ZINFOcrucible-agent (worker): creating region Region { id: RegionId("d161fa2e-d751-4c1c-9b9b-f07e812b1309"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19003, cert_pem: None, key_pem: None, root_pem: None } at "/data/regions/d161fa2e-d751-4c1c-9b9b-f07e812b1309" region = d161fa2e-d751-4c1c-9b9b-f07e812b1309
762024-01-11T01:02:01.571ZINFOcrucible-agent (worker): region files created ok region = d161fa2e-d751-4c1c-9b9b-f07e812b1309
772024-01-11T01:02:01.571ZINFOcrucible-agent (datafile): region d161fa2e-d751-4c1c-9b9b-f07e812b1309 state: Requested -> Created
782024-01-11T01:02:01.571ZINFOcrucible-agent (worker): applying SMF actions post create...
792024-01-11T01:02:01.572ZINFOcrucible-agent (worker): existing directory value /dev/null does not match /data/regions/ade455a2-80ad-4e67-a5cd-94e1a13604e5
802024-01-11T01:02:01.572ZINFOcrucible-agent (worker): existing port value 0 does not match 19000
812024-01-11T01:02:01.572ZINFOcrucible-agent (worker): existing address value 0.0.0.0 does not match fd00:1122:3344:101::13
822024-01-11T01:02:01.572ZINFOcrucible-agent (worker): reconfig required
832024-01-11T01:02:01.572ZINFOcrucible-agent (worker): using existing config property group
842024-01-11T01:02:01.572ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-ade455a2-80ad-4e67-a5cd-94e1a13604e5
852024-01-11T01:02:01.572ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/ade455a2-80ad-4e67-a5cd-94e1a13604e5
862024-01-11T01:02:01.573ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
872024-01-11T01:02:01.573ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::13
882024-01-11T01:02:01.573ZINFOcrucible-agent (worker): commit
892024-01-11T01:02:01.577ZINFOcrucible-agent (worker): ok!
902024-01-11T01:02:01.578ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-d161fa2e-d751-4c1c-9b9b-f07e812b1309
912024-01-11T01:02:01.583ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-d161fa2e-d751-4c1c-9b9b-f07e812b1309
922024-01-11T01:02:01.590ZINFOcrucible-agent (worker): creating config property group
932024-01-11T01:02:01.594ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-d161fa2e-d751-4c1c-9b9b-f07e812b1309
942024-01-11T01:02:01.594ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/d161fa2e-d751-4c1c-9b9b-f07e812b1309
952024-01-11T01:02:01.594ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19003
962024-01-11T01:02:01.594ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::13
972024-01-11T01:02:01.594ZINFOcrucible-agent (worker): commit
982024-01-11T01:02:01.598ZINFOcrucible-agent (worker): ok!
992024-01-11T01:02:01.604ZINFOcrucible-agent (worker): SMF ok!
1002024-01-11T01:02:01.777ZINFOcrucible-agent (dropshot): request completed latency_us = 107 local_addr = [fd00:1122:3344:101::13]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:46796 req_id = 99d044c3-25a0-4e5a-9e34-5d1e99ce0026 response_code = 200 uri = /crucible/0/regions
1012024-01-11T01:02:31.704ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::13]:32345 remote_addr = [fd00:1122:3344:101::a]:35672
1022024-01-11T01:02:31.704ZINFOcrucible-agent (dropshot): request completed latency_us = 61 local_addr = [fd00:1122:3344:101::13]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:35672 req_id = 89e52640-e597-4735-a8a0-2f8c43a0a523 response_code = 200 uri = /crucible/0/regions/d161fa2e-d751-4c1c-9b9b-f07e812b1309
1032024-01-11T01:02:31.704ZINFOcrucible-agent (datafile): path is "/data/regions/d161fa2e-d751-4c1c-9b9b-f07e812b1309"
1042024-01-11T01:02:31.716ZINFOcrucible-agent (datafile): dataset is oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible/regions/d161fa2e-d751-4c1c-9b9b-f07e812b1309
1052024-01-11T01:02:31.726ZINFOcrucible-agent (datafile): region d161fa2e-d751-4c1c-9b9b-f07e812b1309 state: Created -> Tombstoned
1062024-01-11T01:02:31.726ZINFOcrucible-agent (worker): applying SMF actions before removal...
1072024-01-11T01:02:31.726ZINFOcrucible-agent (dropshot): request completed latency_us = 21583 local_addr = [fd00:1122:3344:101::13]:32345 method = DELETE remote_addr = [fd00:1122:3344:101::a]:35672 req_id = 74f8e207-8b30-45cb-9e72-db33e3e14bf1 response_code = 204 uri = /crucible/0/regions/d161fa2e-d751-4c1c-9b9b-f07e812b1309
1082024-01-11T01:02:31.726ZINFOcrucible-agent (dropshot): request completed latency_us = 24 local_addr = [fd00:1122:3344:101::13]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:35672 req_id = 394820e0-a22e-4a86-b6a5-d169799e753b response_code = 200 uri = /crucible/0/regions/d161fa2e-d751-4c1c-9b9b-f07e812b1309
1092024-01-11T01:02:31.727ZINFOcrucible-agent (worker): disabling downstairs instance: downstairs-d161fa2e-d751-4c1c-9b9b-f07e812b1309 (instance states: (Some(Online), None))
1102024-01-11T01:02:31.732ZINFOcrucible-agent (worker): existing directory value /dev/null does not match /data/regions/ade455a2-80ad-4e67-a5cd-94e1a13604e5
1112024-01-11T01:02:31.732ZINFOcrucible-agent (worker): existing port value 0 does not match 19000
1122024-01-11T01:02:31.732ZINFOcrucible-agent (worker): existing address value 0.0.0.0 does not match fd00:1122:3344:101::13
1132024-01-11T01:02:31.732ZINFOcrucible-agent (worker): reconfig required
1142024-01-11T01:02:31.733ZINFOcrucible-agent (worker): using existing config property group
1152024-01-11T01:02:31.733ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-ade455a2-80ad-4e67-a5cd-94e1a13604e5
1162024-01-11T01:02:31.733ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/ade455a2-80ad-4e67-a5cd-94e1a13604e5
1172024-01-11T01:02:31.733ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
1182024-01-11T01:02:31.733ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::13
1192024-01-11T01:02:31.733ZINFOcrucible-agent (worker): commit
1202024-01-11T01:02:31.737ZINFOcrucible-agent (worker): ok!
1212024-01-11T01:02:31.740ZINFOcrucible-agent (worker): SMF ok!
1222024-01-11T01:02:31.750ZINFOcrucible-agent (worker): deleting zfs dataset "oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible/regions/d161fa2e-d751-4c1c-9b9b-f07e812b1309" region = d161fa2e-d751-4c1c-9b9b-f07e812b1309
1232024-01-11T01:02:31.793ZINFOcrucible-agent (datafile): region d161fa2e-d751-4c1c-9b9b-f07e812b1309 state: Tombstoned -> Destroyed
1242024-01-11T01:02:31.871ZINFOcrucible-agent (dropshot): request completed latency_us = 71 local_addr = [fd00:1122:3344:101::13]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:35672 req_id = 4ba750f9-3d4c-4e49-a8b3-92be3e8fd3f0 response_code = 200 uri = /crucible/0/regions/d161fa2e-d751-4c1c-9b9b-f07e812b1309