Max level shown:
12024-08-06T04:01:07.766ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::12]:32345 remote_addr = [fd00:1122:3344:101::a]:62451
22024-08-06T04:01:07.787ZINFOcrucible-agent (datafile): region 4fa30645-53fd-4130-8ad0-a8c170b57d25 state: Requested
32024-08-06T04:01:07.792ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
42024-08-06T04:01:07.796ZINFOcrucible-agent (dropshot): request completed latency_us = 15515 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:62451 req_id = 5fd6edd0-76d5-43a1-9f68-681c4e5d2c07 response_code = 200 uri = /crucible/0/regions
52024-08-06T04:01:07.805ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_6380009d-c432-4baf-a967-a7a8d15c0531/crucible/regions/4fa30645-53fd-4130-8ad0-a8c170b57d25
62024-08-06T04:01:07.805ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_6380009d-c432-4baf-a967-a7a8d15c0531/crucible/regions/4fa30645-53fd-4130-8ad0-a8c170b57d25
72024-08-06T04:01:07.872ZINFOcrucible-agent (worker): creating region Region { id: RegionId("4fa30645-53fd-4130-8ad0-a8c170b57d25"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/4fa30645-53fd-4130-8ad0-a8c170b57d25" region = 4fa30645-53fd-4130-8ad0-a8c170b57d25
82024-08-06T04:01:07.874ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "4fa30645-53fd-4130-8ad0-a8c170b57d25" "--data" "/data/regions/4fa30645-53fd-4130-8ad0-a8c170b57d25" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 4fa30645-53fd-4130-8ad0-a8c170b57d25
92024-08-06T04:01:08.000ZINFOcrucible-agent (worker): region files created ok region = 4fa30645-53fd-4130-8ad0-a8c170b57d25
102024-08-06T04:01:08.000ZINFOcrucible-agent (datafile): region 4fa30645-53fd-4130-8ad0-a8c170b57d25 state: Requested -> Created
112024-08-06T04:01:08.000ZINFOcrucible-agent (worker): applying SMF actions post create...
122024-08-06T04:01:08.001ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-4fa30645-53fd-4130-8ad0-a8c170b57d25
132024-08-06T04:01:08.005ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-4fa30645-53fd-4130-8ad0-a8c170b57d25
142024-08-06T04:01:08.021ZINFOcrucible-agent (worker): creating config property group
152024-08-06T04:01:08.024ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-4fa30645-53fd-4130-8ad0-a8c170b57d25
162024-08-06T04:01:08.032ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/4fa30645-53fd-4130-8ad0-a8c170b57d25
172024-08-06T04:01:08.034ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
182024-08-06T04:01:08.034ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::12
192024-08-06T04:01:08.034ZINFOcrucible-agent (worker): commit
202024-08-06T04:01:08.038ZINFOcrucible-agent (worker): ok!
212024-08-06T04:01:08.043ZINFOcrucible-agent (worker): SMF ok!
222024-08-06T04:01:08.138ZINFOcrucible-agent (dropshot): request completed latency_us = 85 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:62451 req_id = 2bbbb9bd-c9c8-4ded-9ffe-8a098919f2b4 response_code = 200 uri = /crucible/0/regions
232024-08-06T04:06:31.691ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::12]:32345 remote_addr = [fd00:1122:3344:101::a]:50248
242024-08-06T04:06:31.691ZINFOcrucible-agent (dropshot): request completed latency_us = 68 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:50248 req_id = 733738e5-c9ea-41f1-a8b9-fa40ca5005fb response_code = 200 uri = /crucible/0/regions/4fa30645-53fd-4130-8ad0-a8c170b57d25
252024-08-06T04:06:31.695ZINFOcrucible-agent (datafile): path is "/data/regions/4fa30645-53fd-4130-8ad0-a8c170b57d25"
262024-08-06T04:06:31.705ZINFOcrucible-agent (datafile): dataset is oxp_6380009d-c432-4baf-a967-a7a8d15c0531/crucible/regions/4fa30645-53fd-4130-8ad0-a8c170b57d25
272024-08-06T04:06:31.714ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_6380009d-c432-4baf-a967-a7a8d15c0531/crucible/regions/4fa30645-53fd-4130-8ad0-a8c170b57d25@71195216-cfec-47d4-9ea7-f59b10c8f429
282024-08-06T04:06:31.714ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_6380009d-c432-4baf-a967-a7a8d15c0531/crucible/regions/4fa30645-53fd-4130-8ad0-a8c170b57d25", "71195216-cfec-47d4-9ea7-f59b10c8f429"]
292024-08-06T04:06:31.726ZINFOcrucible-agent (dropshot): request completed latency_us = 34395 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:50248 req_id = 587f3cc2-d8e6-485a-8193-f4110a01d0cb response_code = 200 uri = /crucible/0/regions/4fa30645-53fd-4130-8ad0-a8c170b57d25/snapshots/71195216-cfec-47d4-9ea7-f59b10c8f429
302024-08-06T04:06:31.726ZINFOcrucible-agent (datafile): path is "/data/regions/4fa30645-53fd-4130-8ad0-a8c170b57d25"
312024-08-06T04:06:31.735ZINFOcrucible-agent (datafile): dataset is oxp_6380009d-c432-4baf-a967-a7a8d15c0531/crucible/regions/4fa30645-53fd-4130-8ad0-a8c170b57d25
322024-08-06T04:06:31.743ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_6380009d-c432-4baf-a967-a7a8d15c0531/crucible/regions/4fa30645-53fd-4130-8ad0-a8c170b57d25@71195216-cfec-47d4-9ea7-f59b10c8f429
332024-08-06T04:06:31.743ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_6380009d-c432-4baf-a967-a7a8d15c0531/crucible/regions/4fa30645-53fd-4130-8ad0-a8c170b57d25", "71195216-cfec-47d4-9ea7-f59b10c8f429"]
342024-08-06T04:06:31.753ZINFOcrucible-agent (datafile): requesting running snapshot 4fa30645-53fd-4130-8ad0-a8c170b57d25-71195216-cfec-47d4-9ea7-f59b10c8f429 state: Requested
352024-08-06T04:06:31.753ZINFOcrucible-agent (worker): applying SMF actions for region 4fa30645-53fd-4130-8ad0-a8c170b57d25 running snapshot 71195216-cfec-47d4-9ea7-f59b10c8f429 (state Requested)...
362024-08-06T04:06:31.753ZINFOcrucible-agent (dropshot): request completed latency_us = 27702 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:50248 req_id = cffc2e5d-c3fb-4f6b-b810-3e800ae69b2f response_code = 200 uri = /crucible/0/regions/4fa30645-53fd-4130-8ad0-a8c170b57d25/snapshots/71195216-cfec-47d4-9ea7-f59b10c8f429/run
372024-08-06T04:06:31.758ZINFOcrucible-agent (worker): existing directory value /dev/null does not match /data/regions/4fa30645-53fd-4130-8ad0-a8c170b57d25
382024-08-06T04:06:31.758ZINFOcrucible-agent (worker): existing port value 0 does not match 19000
392024-08-06T04:06:31.759ZINFOcrucible-agent (worker): existing address value 0.0.0.0 does not match fd00:1122:3344:101::12
402024-08-06T04:06:31.759ZINFOcrucible-agent (worker): reconfig required
412024-08-06T04:06:31.759ZINFOcrucible-agent (worker): using existing config property group
422024-08-06T04:06:31.759ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-4fa30645-53fd-4130-8ad0-a8c170b57d25
432024-08-06T04:06:31.759ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/4fa30645-53fd-4130-8ad0-a8c170b57d25
442024-08-06T04:06:31.759ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
452024-08-06T04:06:31.759ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::12
462024-08-06T04:06:31.759ZINFOcrucible-agent (worker): commit
472024-08-06T04:06:31.767ZINFOcrucible-agent (worker): ok!
482024-08-06T04:06:31.768ZINFOcrucible-agent (worker): creating missing snapshot instance snapshot-4fa30645-53fd-4130-8ad0-a8c170b57d25-71195216-cfec-47d4-9ea7-f59b10c8f429
492024-08-06T04:06:31.774ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:snapshot-4fa30645-53fd-4130-8ad0-a8c170b57d25-71195216-cfec-47d4-9ea7-f59b10c8f429
502024-08-06T04:06:31.786ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:snapshot-4fa30645-53fd-4130-8ad0-a8c170b57d25-71195216-cfec-47d4-9ea7-f59b10c8f429
512024-08-06T04:06:31.786ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/4fa30645-53fd-4130-8ad0-a8c170b57d25/.zfs/snapshot/71195216-cfec-47d4-9ea7-f59b10c8f429
522024-08-06T04:06:31.786ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
532024-08-06T04:06:31.786ZINFOcrucible-agent (worker): ensure mode SCF_TYPE_ASTRING ro
542024-08-06T04:06:31.786ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::12
552024-08-06T04:06:31.786ZINFOcrucible-agent (worker): commit
562024-08-06T04:06:31.790ZINFOcrucible-agent (worker): ok!
572024-08-06T04:06:31.794ZINFOcrucible-agent (worker): SMF ok!
582024-08-06T04:06:31.794ZINFOcrucible-agent (datafile): region 4fa30645-53fd-4130-8ad0-a8c170b57d25 running snapshot 71195216-cfec-47d4-9ea7-f59b10c8f429 state: Requested -> Created
592024-08-06T04:06:34.593ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::12]:32345 remote_addr = [fd00:1122:3344:101::a]:47626
602024-08-06T04:06:34.593ZINFOcrucible-agent (datafile): region c727c466-0991-488d-b84d-8031783c1585 state: Requested
612024-08-06T04:06:34.593ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
622024-08-06T04:06:34.593ZINFOcrucible-agent (dropshot): request completed latency_us = 211 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:47626 req_id = c4bccb68-5b09-4c33-b462-681de6baaa0a response_code = 200 uri = /crucible/0/regions
632024-08-06T04:06:34.602ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_6380009d-c432-4baf-a967-a7a8d15c0531/crucible/regions/c727c466-0991-488d-b84d-8031783c1585
642024-08-06T04:06:34.602ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_6380009d-c432-4baf-a967-a7a8d15c0531/crucible/regions/c727c466-0991-488d-b84d-8031783c1585
652024-08-06T04:06:34.666ZINFOcrucible-agent (worker): creating region Region { id: RegionId("c727c466-0991-488d-b84d-8031783c1585"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19002, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/c727c466-0991-488d-b84d-8031783c1585" region = c727c466-0991-488d-b84d-8031783c1585
662024-08-06T04:06:34.668ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "c727c466-0991-488d-b84d-8031783c1585" "--data" "/data/regions/c727c466-0991-488d-b84d-8031783c1585" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = c727c466-0991-488d-b84d-8031783c1585
672024-08-06T04:06:34.735ZINFOcrucible-agent (worker): region files created ok region = c727c466-0991-488d-b84d-8031783c1585
682024-08-06T04:06:34.735ZINFOcrucible-agent (datafile): region c727c466-0991-488d-b84d-8031783c1585 state: Requested -> Created
692024-08-06T04:06:34.735ZINFOcrucible-agent (worker): applying SMF actions post create...
702024-08-06T04:06:34.736ZINFOcrucible-agent (worker): existing directory value /dev/null does not match /data/regions/4fa30645-53fd-4130-8ad0-a8c170b57d25
712024-08-06T04:06:34.736ZINFOcrucible-agent (worker): existing port value 0 does not match 19000
722024-08-06T04:06:34.736ZINFOcrucible-agent (worker): existing address value 0.0.0.0 does not match fd00:1122:3344:101::12
732024-08-06T04:06:34.736ZINFOcrucible-agent (worker): reconfig required
742024-08-06T04:06:34.736ZINFOcrucible-agent (worker): using existing config property group
752024-08-06T04:06:34.736ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-4fa30645-53fd-4130-8ad0-a8c170b57d25
762024-08-06T04:06:34.736ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/4fa30645-53fd-4130-8ad0-a8c170b57d25
772024-08-06T04:06:34.736ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
782024-08-06T04:06:34.736ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::12
792024-08-06T04:06:34.736ZINFOcrucible-agent (worker): commit
802024-08-06T04:06:34.740ZINFOcrucible-agent (worker): ok!
812024-08-06T04:06:34.741ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-c727c466-0991-488d-b84d-8031783c1585
822024-08-06T04:06:34.744ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-c727c466-0991-488d-b84d-8031783c1585
832024-08-06T04:06:34.750ZINFOcrucible-agent (worker): creating config property group
842024-08-06T04:06:34.753ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-c727c466-0991-488d-b84d-8031783c1585
852024-08-06T04:06:34.754ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/c727c466-0991-488d-b84d-8031783c1585
862024-08-06T04:06:34.754ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19002
872024-08-06T04:06:34.754ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::12
882024-08-06T04:06:34.754ZINFOcrucible-agent (worker): commit
892024-08-06T04:06:34.758ZINFOcrucible-agent (worker): ok!
902024-08-06T04:06:34.764ZINFOcrucible-agent (worker): SMF ok!
912024-08-06T04:06:34.797ZINFOcrucible-agent (dropshot): request completed latency_us = 68 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:47626 req_id = 3ff8a6ae-7e7a-4a6a-99fc-f0fb6c021659 response_code = 200 uri = /crucible/0/regions
922024-08-06T04:06:59.251ZINFOcrucible-agent (dropshot): request completed latency_us = 67 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:47626 req_id = 3faf5a06-1cd7-4961-a916-a31277a226bf response_code = 200 uri = /crucible/0/regions/c727c466-0991-488d-b84d-8031783c1585
932024-08-06T04:06:59.254ZINFOcrucible-agent (datafile): path is "/data/regions/c727c466-0991-488d-b84d-8031783c1585"
942024-08-06T04:06:59.266ZINFOcrucible-agent (datafile): dataset is oxp_6380009d-c432-4baf-a967-a7a8d15c0531/crucible/regions/c727c466-0991-488d-b84d-8031783c1585
952024-08-06T04:06:59.274ZINFOcrucible-agent (datafile): region c727c466-0991-488d-b84d-8031783c1585 state: Created -> Tombstoned
962024-08-06T04:06:59.275ZINFOcrucible-agent (worker): applying SMF actions before removal...
972024-08-06T04:06:59.275ZINFOcrucible-agent (dropshot): request completed latency_us = 20870 local_addr = [fd00:1122:3344:101::12]:32345 method = DELETE remote_addr = [fd00:1122:3344:101::a]:47626 req_id = 5dc290ed-f36a-40ee-89f5-8a0eb4095070 response_code = 204 uri = /crucible/0/regions/c727c466-0991-488d-b84d-8031783c1585
982024-08-06T04:06:59.276ZINFOcrucible-agent (worker): disabling downstairs instance: downstairs-c727c466-0991-488d-b84d-8031783c1585 (instance states: (Some(Online), None))
992024-08-06T04:06:59.277ZINFOcrucible-agent (dropshot): request completed latency_us = 25 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:47626 req_id = 07a368e6-2a58-4f8b-86c0-b220c0cf7296 response_code = 200 uri = /crucible/0/regions/c727c466-0991-488d-b84d-8031783c1585
1002024-08-06T04:06:59.281ZINFOcrucible-agent (worker): existing directory value /dev/null does not match /data/regions/4fa30645-53fd-4130-8ad0-a8c170b57d25
1012024-08-06T04:06:59.281ZINFOcrucible-agent (worker): existing port value 0 does not match 19000
1022024-08-06T04:06:59.281ZINFOcrucible-agent (worker): existing address value 0.0.0.0 does not match fd00:1122:3344:101::12
1032024-08-06T04:06:59.281ZINFOcrucible-agent (worker): reconfig required
1042024-08-06T04:06:59.281ZINFOcrucible-agent (worker): using existing config property group
1052024-08-06T04:06:59.281ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-4fa30645-53fd-4130-8ad0-a8c170b57d25
1062024-08-06T04:06:59.281ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/4fa30645-53fd-4130-8ad0-a8c170b57d25
1072024-08-06T04:06:59.282ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
1082024-08-06T04:06:59.282ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::12
1092024-08-06T04:06:59.282ZINFOcrucible-agent (worker): commit
1102024-08-06T04:06:59.286ZINFOcrucible-agent (worker): ok!
1112024-08-06T04:06:59.287ZINFOcrucible-agent (worker): SMF ok!
1122024-08-06T04:06:59.297ZINFOcrucible-agent (worker): deleting zfs dataset "oxp_6380009d-c432-4baf-a967-a7a8d15c0531/crucible/regions/c727c466-0991-488d-b84d-8031783c1585" region = c727c466-0991-488d-b84d-8031783c1585
1132024-08-06T04:06:59.350ZINFOcrucible-agent (dropshot): request completed latency_us = 69 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:47626 req_id = 797cbe36-6426-423b-8103-857409a2cb27 response_code = 200 uri = /crucible/0/regions/c727c466-0991-488d-b84d-8031783c1585
1142024-08-06T04:06:59.353ZINFOcrucible-agent (datafile): region c727c466-0991-488d-b84d-8031783c1585 state: Tombstoned -> Destroyed
1152024-08-06T04:06:59.519ZINFOcrucible-agent (dropshot): request completed latency_us = 23 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:47626 req_id = 984885e5-305e-4b9d-a277-c66ae4ce9ca7 response_code = 200 uri = /crucible/0/regions/c727c466-0991-488d-b84d-8031783c1585