Max level shown:
12024-06-12T21:33:32.415ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::16]:32345 remote_addr = [fd00:1122:3344:101::a]:56790
22024-06-12T21:33:32.416ZINFOcrucible-agent (datafile): region b770e10c-d241-434c-9138-e4323d556466 state: Requested
32024-06-12T21:33:32.416ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
42024-06-12T21:33:32.416ZINFOcrucible-agent (dropshot): request completed latency_us = 639 local_addr = [fd00:1122:3344:101::16]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:56790 req_id = 2f740c43-9a4e-44c6-8605-0296703e1e66 response_code = 200 uri = /crucible/0/regions
52024-06-12T21:33:32.426ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_8043b355-0fd3-40fc-8db4-22fae40be7a4/crucible/regions/b770e10c-d241-434c-9138-e4323d556466
62024-06-12T21:33:32.426ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_8043b355-0fd3-40fc-8db4-22fae40be7a4/crucible/regions/b770e10c-d241-434c-9138-e4323d556466
72024-06-12T21:33:32.482ZINFOcrucible-agent (worker): creating region Region { id: RegionId("b770e10c-d241-434c-9138-e4323d556466"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19001, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/b770e10c-d241-434c-9138-e4323d556466" region = b770e10c-d241-434c-9138-e4323d556466
82024-06-12T21:33:32.483ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "b770e10c-d241-434c-9138-e4323d556466" "--data" "/data/regions/b770e10c-d241-434c-9138-e4323d556466" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = b770e10c-d241-434c-9138-e4323d556466
92024-06-12T21:33:32.548ZINFOcrucible-agent (worker): region files created ok region = b770e10c-d241-434c-9138-e4323d556466
102024-06-12T21:33:32.548ZINFOcrucible-agent (datafile): region b770e10c-d241-434c-9138-e4323d556466 state: Requested -> Created
112024-06-12T21:33:32.548ZINFOcrucible-agent (worker): applying SMF actions post create...
122024-06-12T21:33:32.548ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-b770e10c-d241-434c-9138-e4323d556466
132024-06-12T21:33:32.553ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-b770e10c-d241-434c-9138-e4323d556466
142024-06-12T21:33:32.567ZINFOcrucible-agent (worker): creating config property group
152024-06-12T21:33:32.570ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-b770e10c-d241-434c-9138-e4323d556466
162024-06-12T21:33:32.570ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/b770e10c-d241-434c-9138-e4323d556466
172024-06-12T21:33:32.570ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
182024-06-12T21:33:32.570ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::16
192024-06-12T21:33:32.570ZINFOcrucible-agent (worker): commit
202024-06-12T21:33:32.573ZINFOcrucible-agent (worker): ok!
212024-06-12T21:33:32.578ZINFOcrucible-agent (worker): existing directory value /dev/null does not match /data/regions/db49ef5b-f68d-4c3a-b793-d0a46be672f2
222024-06-12T21:33:32.578ZINFOcrucible-agent (worker): existing port value 0 does not match 19000
232024-06-12T21:33:32.579ZINFOcrucible-agent (worker): existing address value 0.0.0.0 does not match fd00:1122:3344:101::16
242024-06-12T21:33:32.579ZINFOcrucible-agent (worker): reconfig required
252024-06-12T21:33:32.579ZINFOcrucible-agent (worker): using existing config property group
262024-06-12T21:33:32.579ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-db49ef5b-f68d-4c3a-b793-d0a46be672f2
272024-06-12T21:33:32.579ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/db49ef5b-f68d-4c3a-b793-d0a46be672f2
282024-06-12T21:33:32.579ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
292024-06-12T21:33:32.579ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::16
302024-06-12T21:33:32.579ZINFOcrucible-agent (worker): commit
312024-06-12T21:33:32.593ZINFOcrucible-agent (worker): ok!
322024-06-12T21:33:32.593ZINFOcrucible-agent (worker): SMF ok!
332024-06-12T21:33:32.624ZINFOcrucible-agent (dropshot): request completed latency_us = 81 local_addr = [fd00:1122:3344:101::16]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:56790 req_id = be5528db-e455-4475-af5d-5bdcb718b62c response_code = 200 uri = /crucible/0/regions
342024-06-12T21:33:33.271ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::16]:32345 remote_addr = [fd00:1122:3344:101::a]:45614
352024-06-12T21:33:33.275ZINFOcrucible-agent (dropshot): request completed latency_us = 3968 local_addr = [fd00:1122:3344:101::16]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:45614 req_id = 6c25a20a-b733-4529-9e62-0698e8793207 response_code = 200 uri = /crucible/0/regions/db49ef5b-f68d-4c3a-b793-d0a46be672f2
362024-06-12T21:33:33.276ZINFOcrucible-agent (datafile): path is "/data/regions/db49ef5b-f68d-4c3a-b793-d0a46be672f2"
372024-06-12T21:33:33.284ZINFOcrucible-agent (datafile): dataset is oxp_8043b355-0fd3-40fc-8db4-22fae40be7a4/crucible/regions/db49ef5b-f68d-4c3a-b793-d0a46be672f2
382024-06-12T21:33:33.294ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_8043b355-0fd3-40fc-8db4-22fae40be7a4/crucible/regions/db49ef5b-f68d-4c3a-b793-d0a46be672f2@8024e4d8-277d-43cb-b8aa-5dbd6ebde952
392024-06-12T21:33:33.294ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_8043b355-0fd3-40fc-8db4-22fae40be7a4/crucible/regions/db49ef5b-f68d-4c3a-b793-d0a46be672f2", "8024e4d8-277d-43cb-b8aa-5dbd6ebde952"]
402024-06-12T21:33:33.307ZINFOcrucible-agent (dropshot): request completed latency_us = 31658 local_addr = [fd00:1122:3344:101::16]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:45614 req_id = 50cc1182-9f4e-4f89-8763-2e5450138326 response_code = 200 uri = /crucible/0/regions/db49ef5b-f68d-4c3a-b793-d0a46be672f2/snapshots/8024e4d8-277d-43cb-b8aa-5dbd6ebde952
412024-06-12T21:33:33.307ZINFOcrucible-agent (datafile): path is "/data/regions/db49ef5b-f68d-4c3a-b793-d0a46be672f2"
422024-06-12T21:33:33.316ZINFOcrucible-agent (datafile): dataset is oxp_8043b355-0fd3-40fc-8db4-22fae40be7a4/crucible/regions/db49ef5b-f68d-4c3a-b793-d0a46be672f2
432024-06-12T21:33:33.324ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_8043b355-0fd3-40fc-8db4-22fae40be7a4/crucible/regions/db49ef5b-f68d-4c3a-b793-d0a46be672f2@8024e4d8-277d-43cb-b8aa-5dbd6ebde952
442024-06-12T21:33:33.324ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_8043b355-0fd3-40fc-8db4-22fae40be7a4/crucible/regions/db49ef5b-f68d-4c3a-b793-d0a46be672f2", "8024e4d8-277d-43cb-b8aa-5dbd6ebde952"]
452024-06-12T21:33:33.333ZINFOcrucible-agent (datafile): requesting running snapshot db49ef5b-f68d-4c3a-b793-d0a46be672f2-8024e4d8-277d-43cb-b8aa-5dbd6ebde952 state: Requested
462024-06-12T21:33:33.333ZINFOcrucible-agent (worker): applying SMF actions for region db49ef5b-f68d-4c3a-b793-d0a46be672f2 running snapshot 8024e4d8-277d-43cb-b8aa-5dbd6ebde952 (state Requested)...
472024-06-12T21:33:33.333ZINFOcrucible-agent (dropshot): request completed latency_us = 25782 local_addr = [fd00:1122:3344:101::16]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:45614 req_id = b2edc013-4ccd-4764-a84d-2aeca1cf76dd response_code = 200 uri = /crucible/0/regions/db49ef5b-f68d-4c3a-b793-d0a46be672f2/snapshots/8024e4d8-277d-43cb-b8aa-5dbd6ebde952/run
482024-06-12T21:33:33.335ZINFOcrucible-agent (worker): existing directory value /dev/null does not match /data/regions/db49ef5b-f68d-4c3a-b793-d0a46be672f2
492024-06-12T21:33:33.335ZINFOcrucible-agent (worker): existing port value 0 does not match 19000
502024-06-12T21:33:33.335ZINFOcrucible-agent (worker): existing address value 0.0.0.0 does not match fd00:1122:3344:101::16
512024-06-12T21:33:33.335ZINFOcrucible-agent (worker): reconfig required
522024-06-12T21:33:33.335ZINFOcrucible-agent (worker): using existing config property group
532024-06-12T21:33:33.335ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-db49ef5b-f68d-4c3a-b793-d0a46be672f2
542024-06-12T21:33:33.335ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/db49ef5b-f68d-4c3a-b793-d0a46be672f2
552024-06-12T21:33:33.335ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
562024-06-12T21:33:33.335ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::16
572024-06-12T21:33:33.336ZINFOcrucible-agent (worker): commit
582024-06-12T21:33:33.340ZINFOcrucible-agent (worker): ok!
592024-06-12T21:33:33.340ZINFOcrucible-agent (worker): creating missing snapshot instance snapshot-db49ef5b-f68d-4c3a-b793-d0a46be672f2-8024e4d8-277d-43cb-b8aa-5dbd6ebde952
602024-06-12T21:33:33.345ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:snapshot-db49ef5b-f68d-4c3a-b793-d0a46be672f2-8024e4d8-277d-43cb-b8aa-5dbd6ebde952
612024-06-12T21:33:33.355ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:snapshot-db49ef5b-f68d-4c3a-b793-d0a46be672f2-8024e4d8-277d-43cb-b8aa-5dbd6ebde952
622024-06-12T21:33:33.355ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/db49ef5b-f68d-4c3a-b793-d0a46be672f2/.zfs/snapshot/8024e4d8-277d-43cb-b8aa-5dbd6ebde952
632024-06-12T21:33:33.355ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19002
642024-06-12T21:33:33.355ZINFOcrucible-agent (worker): ensure mode SCF_TYPE_ASTRING ro
652024-06-12T21:33:33.355ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::16
662024-06-12T21:33:33.355ZINFOcrucible-agent (worker): commit
672024-06-12T21:33:33.358ZINFOcrucible-agent (worker): ok!
682024-06-12T21:33:33.362ZINFOcrucible-agent (worker): SMF ok!
692024-06-12T21:33:33.362ZINFOcrucible-agent (datafile): region db49ef5b-f68d-4c3a-b793-d0a46be672f2 running snapshot 8024e4d8-277d-43cb-b8aa-5dbd6ebde952 state: Requested -> Created
702024-06-12T21:33:36.220ZINFOcrucible-agent (datafile): region 22c02cd4-6422-4fdc-be9a-61f9f54e7578 state: Requested
712024-06-12T21:33:36.220ZINFOcrucible-agent (dropshot): request completed latency_us = 218 local_addr = [fd00:1122:3344:101::16]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:56790 req_id = 1d767517-0aca-425e-867d-4f684c00ac54 response_code = 200 uri = /crucible/0/regions
722024-06-12T21:33:36.220ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
732024-06-12T21:33:36.230ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_8043b355-0fd3-40fc-8db4-22fae40be7a4/crucible/regions/22c02cd4-6422-4fdc-be9a-61f9f54e7578
742024-06-12T21:33:36.230ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_8043b355-0fd3-40fc-8db4-22fae40be7a4/crucible/regions/22c02cd4-6422-4fdc-be9a-61f9f54e7578
752024-06-12T21:33:36.288ZINFOcrucible-agent (worker): creating region Region { id: RegionId("22c02cd4-6422-4fdc-be9a-61f9f54e7578"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19003, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/22c02cd4-6422-4fdc-be9a-61f9f54e7578" region = 22c02cd4-6422-4fdc-be9a-61f9f54e7578
762024-06-12T21:33:36.290ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "22c02cd4-6422-4fdc-be9a-61f9f54e7578" "--data" "/data/regions/22c02cd4-6422-4fdc-be9a-61f9f54e7578" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 22c02cd4-6422-4fdc-be9a-61f9f54e7578
772024-06-12T21:33:36.352ZINFOcrucible-agent (worker): region files created ok region = 22c02cd4-6422-4fdc-be9a-61f9f54e7578
782024-06-12T21:33:36.352ZINFOcrucible-agent (datafile): region 22c02cd4-6422-4fdc-be9a-61f9f54e7578 state: Requested -> Created
792024-06-12T21:33:36.352ZINFOcrucible-agent (worker): applying SMF actions post create...
802024-06-12T21:33:36.352ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-22c02cd4-6422-4fdc-be9a-61f9f54e7578
812024-06-12T21:33:36.355ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-22c02cd4-6422-4fdc-be9a-61f9f54e7578
822024-06-12T21:33:36.364ZINFOcrucible-agent (worker): creating config property group
832024-06-12T21:33:36.369ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-22c02cd4-6422-4fdc-be9a-61f9f54e7578
842024-06-12T21:33:36.369ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/22c02cd4-6422-4fdc-be9a-61f9f54e7578
852024-06-12T21:33:36.369ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19003
862024-06-12T21:33:36.369ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::16
872024-06-12T21:33:36.369ZINFOcrucible-agent (worker): commit
882024-06-12T21:33:36.373ZINFOcrucible-agent (worker): ok!
892024-06-12T21:33:36.378ZINFOcrucible-agent (worker): existing directory value /dev/null does not match /data/regions/db49ef5b-f68d-4c3a-b793-d0a46be672f2
902024-06-12T21:33:36.378ZINFOcrucible-agent (worker): existing port value 0 does not match 19000
912024-06-12T21:33:36.378ZINFOcrucible-agent (worker): existing address value 0.0.0.0 does not match fd00:1122:3344:101::16
922024-06-12T21:33:36.378ZINFOcrucible-agent (worker): reconfig required
932024-06-12T21:33:36.378ZINFOcrucible-agent (worker): using existing config property group
942024-06-12T21:33:36.378ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-db49ef5b-f68d-4c3a-b793-d0a46be672f2
952024-06-12T21:33:36.382ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/db49ef5b-f68d-4c3a-b793-d0a46be672f2
962024-06-12T21:33:36.382ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
972024-06-12T21:33:36.382ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::16
982024-06-12T21:33:36.382ZINFOcrucible-agent (worker): commit
992024-06-12T21:33:36.385ZINFOcrucible-agent (worker): ok!
1002024-06-12T21:33:36.387ZINFOcrucible-agent (worker): SMF ok!
1012024-06-12T21:33:36.528ZINFOcrucible-agent (dropshot): request completed latency_us = 87 local_addr = [fd00:1122:3344:101::16]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:56790 req_id = f8192bf8-935c-4944-92e5-4afe403a1d2a response_code = 200 uri = /crucible/0/regions
1022024-06-12T21:34:00.863ZINFOcrucible-agent (dropshot): request completed latency_us = 67 local_addr = [fd00:1122:3344:101::16]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:56790 req_id = 21121127-e57e-45ce-8372-05d6033c7aea response_code = 200 uri = /crucible/0/regions/22c02cd4-6422-4fdc-be9a-61f9f54e7578
1032024-06-12T21:34:00.865ZINFOcrucible-agent (datafile): path is "/data/regions/22c02cd4-6422-4fdc-be9a-61f9f54e7578"
1042024-06-12T21:34:00.878ZINFOcrucible-agent (datafile): dataset is oxp_8043b355-0fd3-40fc-8db4-22fae40be7a4/crucible/regions/22c02cd4-6422-4fdc-be9a-61f9f54e7578
1052024-06-12T21:34:00.886ZINFOcrucible-agent (datafile): region 22c02cd4-6422-4fdc-be9a-61f9f54e7578 state: Created -> Tombstoned
1062024-06-12T21:34:00.887ZINFOcrucible-agent (worker): applying SMF actions before removal...
1072024-06-12T21:34:00.889ZINFOcrucible-agent (dropshot): request completed latency_us = 23586 local_addr = [fd00:1122:3344:101::16]:32345 method = DELETE remote_addr = [fd00:1122:3344:101::a]:56790 req_id = 107375d2-5c97-4b7d-85f2-4a280f0a7ef0 response_code = 204 uri = /crucible/0/regions/22c02cd4-6422-4fdc-be9a-61f9f54e7578
1082024-06-12T21:34:00.890ZINFOcrucible-agent (worker): disabling downstairs instance: downstairs-22c02cd4-6422-4fdc-be9a-61f9f54e7578 (instance states: (Some(Online), None))
1092024-06-12T21:34:00.892ZINFOcrucible-agent (dropshot): request completed latency_us = 29 local_addr = [fd00:1122:3344:101::16]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:56790 req_id = 0bb3453f-8e27-415d-8fd7-0fff5761aaac response_code = 200 uri = /crucible/0/regions/22c02cd4-6422-4fdc-be9a-61f9f54e7578
1102024-06-12T21:34:00.896ZINFOcrucible-agent (worker): existing directory value /dev/null does not match /data/regions/db49ef5b-f68d-4c3a-b793-d0a46be672f2
1112024-06-12T21:34:00.896ZINFOcrucible-agent (worker): existing port value 0 does not match 19000
1122024-06-12T21:34:00.896ZINFOcrucible-agent (worker): existing address value 0.0.0.0 does not match fd00:1122:3344:101::16
1132024-06-12T21:34:00.896ZINFOcrucible-agent (worker): reconfig required
1142024-06-12T21:34:00.896ZINFOcrucible-agent (worker): using existing config property group
1152024-06-12T21:34:00.896ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-db49ef5b-f68d-4c3a-b793-d0a46be672f2
1162024-06-12T21:34:00.896ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/db49ef5b-f68d-4c3a-b793-d0a46be672f2
1172024-06-12T21:34:00.896ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
1182024-06-12T21:34:00.896ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::16
1192024-06-12T21:34:00.896ZINFOcrucible-agent (worker): commit
1202024-06-12T21:34:00.900ZINFOcrucible-agent (worker): ok!
1212024-06-12T21:34:00.901ZINFOcrucible-agent (worker): SMF ok!
1222024-06-12T21:34:00.911ZINFOcrucible-agent (worker): deleting zfs dataset "oxp_8043b355-0fd3-40fc-8db4-22fae40be7a4/crucible/regions/22c02cd4-6422-4fdc-be9a-61f9f54e7578" region = 22c02cd4-6422-4fdc-be9a-61f9f54e7578
1232024-06-12T21:34:00.948ZINFOcrucible-agent (dropshot): request completed latency_us = 103 local_addr = [fd00:1122:3344:101::16]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:56790 req_id = 36764e20-2cab-4e2e-be6c-6d776f05a3e9 response_code = 200 uri = /crucible/0/regions/22c02cd4-6422-4fdc-be9a-61f9f54e7578
1242024-06-12T21:34:00.962ZINFOcrucible-agent (datafile): region 22c02cd4-6422-4fdc-be9a-61f9f54e7578 state: Tombstoned -> Destroyed
1252024-06-12T21:34:01.053ZINFOcrucible-agent (dropshot): request completed latency_us = 27 local_addr = [fd00:1122:3344:101::16]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:56790 req_id = 08c770fa-377c-41cc-affe-8432e6ce9704 response_code = 200 uri = /crucible/0/regions/22c02cd4-6422-4fdc-be9a-61f9f54e7578