Max level shown:
12024-09-05T00:48:50.976ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::13]:32345 remote_addr = [fd00:1122:3344:101::a]:57235
22024-09-05T00:48:50.976ZINFOcrucible-agent (datafile): region d073ba9e-ae06-4505-a8dd-115bbda8b66b state: Requested
32024-09-05T00:48:50.977ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
42024-09-05T00:48:50.977ZINFOcrucible-agent (dropshot): request completed latency_us = 695 local_addr = [fd00:1122:3344:101::13]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:57235 req_id = bc83f2c3-9fd1-4d61-9aad-27155f0a842e response_code = 200 uri = /crucible/0/regions
52024-09-05T00:48:50.987ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_3149c6f5-adf9-4e42-9a6f-ea817157070d/crucible/regions/d073ba9e-ae06-4505-a8dd-115bbda8b66b
62024-09-05T00:48:50.987ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_3149c6f5-adf9-4e42-9a6f-ea817157070d/crucible/regions/d073ba9e-ae06-4505-a8dd-115bbda8b66b
72024-09-05T00:48:51.059ZINFOcrucible-agent (worker): creating region Region { id: RegionId("d073ba9e-ae06-4505-a8dd-115bbda8b66b"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19001, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/d073ba9e-ae06-4505-a8dd-115bbda8b66b" region = d073ba9e-ae06-4505-a8dd-115bbda8b66b
82024-09-05T00:48:51.059ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "d073ba9e-ae06-4505-a8dd-115bbda8b66b" "--data" "/data/regions/d073ba9e-ae06-4505-a8dd-115bbda8b66b" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = d073ba9e-ae06-4505-a8dd-115bbda8b66b
92024-09-05T00:48:51.173ZINFOcrucible-agent (worker): region files created ok region = d073ba9e-ae06-4505-a8dd-115bbda8b66b
102024-09-05T00:48:51.173ZINFOcrucible-agent (datafile): region d073ba9e-ae06-4505-a8dd-115bbda8b66b state: Requested -> Created
112024-09-05T00:48:51.173ZINFOcrucible-agent (worker): applying SMF actions post create...
122024-09-05T00:48:51.178ZINFOcrucible-agent (worker): existing directory value /dev/null does not match /data/regions/0cdc26ae-935c-4875-88e3-fd0a9abc45a0
132024-09-05T00:48:51.178ZINFOcrucible-agent (worker): existing port value 0 does not match 19000
142024-09-05T00:48:51.178ZINFOcrucible-agent (worker): existing address value 0.0.0.0 does not match fd00:1122:3344:101::13
152024-09-05T00:48:51.178ZINFOcrucible-agent (worker): reconfig required
162024-09-05T00:48:51.178ZINFOcrucible-agent (worker): using existing config property group
172024-09-05T00:48:51.178ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-0cdc26ae-935c-4875-88e3-fd0a9abc45a0
182024-09-05T00:48:51.179ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/0cdc26ae-935c-4875-88e3-fd0a9abc45a0
192024-09-05T00:48:51.179ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
202024-09-05T00:48:51.179ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::13
212024-09-05T00:48:51.179ZINFOcrucible-agent (worker): commit
222024-09-05T00:48:51.191ZINFOcrucible-agent (worker): ok!
232024-09-05T00:48:51.192ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-d073ba9e-ae06-4505-a8dd-115bbda8b66b
242024-09-05T00:48:51.197ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-d073ba9e-ae06-4505-a8dd-115bbda8b66b
252024-09-05T00:48:51.210ZINFOcrucible-agent (worker): creating config property group
262024-09-05T00:48:51.218ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-d073ba9e-ae06-4505-a8dd-115bbda8b66b
272024-09-05T00:48:51.219ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/d073ba9e-ae06-4505-a8dd-115bbda8b66b
282024-09-05T00:48:51.219ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
292024-09-05T00:48:51.219ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::13
302024-09-05T00:48:51.219ZINFOcrucible-agent (worker): commit
312024-09-05T00:48:51.225ZINFOcrucible-agent (worker): ok!
322024-09-05T00:48:51.234ZINFOcrucible-agent (worker): SMF ok!
332024-09-05T00:48:51.278ZINFOcrucible-agent (dropshot): request completed latency_us = 76 local_addr = [fd00:1122:3344:101::13]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:57235 req_id = 06b16fa2-545b-4f59-9f4e-1369e8e9a0e7 response_code = 200 uri = /crucible/0/regions
342024-09-05T00:48:52.880ZINFOcrucible-agent (dropshot): request completed latency_us = 57 local_addr = [fd00:1122:3344:101::13]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:57235 req_id = cbc9f51b-0f71-4bbc-81e9-0bf1f0e97d02 response_code = 200 uri = /crucible/0/regions/0cdc26ae-935c-4875-88e3-fd0a9abc45a0
352024-09-05T00:48:52.887ZINFOcrucible-agent (datafile): path is "/data/regions/0cdc26ae-935c-4875-88e3-fd0a9abc45a0"
362024-09-05T00:48:52.895ZINFOcrucible-agent (datafile): dataset is oxp_3149c6f5-adf9-4e42-9a6f-ea817157070d/crucible/regions/0cdc26ae-935c-4875-88e3-fd0a9abc45a0
372024-09-05T00:48:52.904ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_3149c6f5-adf9-4e42-9a6f-ea817157070d/crucible/regions/0cdc26ae-935c-4875-88e3-fd0a9abc45a0@27849cd6-3ba0-4d2d-9220-da51501c52b2
382024-09-05T00:48:52.904ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_3149c6f5-adf9-4e42-9a6f-ea817157070d/crucible/regions/0cdc26ae-935c-4875-88e3-fd0a9abc45a0", "27849cd6-3ba0-4d2d-9220-da51501c52b2"]
392024-09-05T00:48:52.914ZINFOcrucible-agent (dropshot): request completed latency_us = 30758 local_addr = [fd00:1122:3344:101::13]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:57235 req_id = 3f38e923-5943-44de-a41b-5c7c3fe938bb response_code = 200 uri = /crucible/0/regions/0cdc26ae-935c-4875-88e3-fd0a9abc45a0/snapshots/27849cd6-3ba0-4d2d-9220-da51501c52b2
402024-09-05T00:48:52.917ZINFOcrucible-agent (datafile): path is "/data/regions/0cdc26ae-935c-4875-88e3-fd0a9abc45a0"
412024-09-05T00:48:52.925ZINFOcrucible-agent (datafile): dataset is oxp_3149c6f5-adf9-4e42-9a6f-ea817157070d/crucible/regions/0cdc26ae-935c-4875-88e3-fd0a9abc45a0
422024-09-05T00:48:52.933ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_3149c6f5-adf9-4e42-9a6f-ea817157070d/crucible/regions/0cdc26ae-935c-4875-88e3-fd0a9abc45a0@27849cd6-3ba0-4d2d-9220-da51501c52b2
432024-09-05T00:48:52.933ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_3149c6f5-adf9-4e42-9a6f-ea817157070d/crucible/regions/0cdc26ae-935c-4875-88e3-fd0a9abc45a0", "27849cd6-3ba0-4d2d-9220-da51501c52b2"]
442024-09-05T00:48:52.941ZINFOcrucible-agent (datafile): requesting running snapshot 0cdc26ae-935c-4875-88e3-fd0a9abc45a0-27849cd6-3ba0-4d2d-9220-da51501c52b2 state: Requested
452024-09-05T00:48:52.942ZINFOcrucible-agent (worker): applying SMF actions for region 0cdc26ae-935c-4875-88e3-fd0a9abc45a0 running snapshot 27849cd6-3ba0-4d2d-9220-da51501c52b2 (state Requested)...
462024-09-05T00:48:52.942ZINFOcrucible-agent (dropshot): request completed latency_us = 24184 local_addr = [fd00:1122:3344:101::13]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:57235 req_id = 0effa58f-2b75-40c4-b2fe-2bd231a8da2a response_code = 200 uri = /crucible/0/regions/0cdc26ae-935c-4875-88e3-fd0a9abc45a0/snapshots/27849cd6-3ba0-4d2d-9220-da51501c52b2/run
472024-09-05T00:48:52.942ZINFOcrucible-agent (worker): existing directory value /dev/null does not match /data/regions/0cdc26ae-935c-4875-88e3-fd0a9abc45a0
482024-09-05T00:48:52.942ZINFOcrucible-agent (worker): existing port value 0 does not match 19000
492024-09-05T00:48:52.942ZINFOcrucible-agent (worker): existing address value 0.0.0.0 does not match fd00:1122:3344:101::13
502024-09-05T00:48:52.942ZINFOcrucible-agent (worker): reconfig required
512024-09-05T00:48:52.942ZINFOcrucible-agent (worker): using existing config property group
522024-09-05T00:48:52.942ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-0cdc26ae-935c-4875-88e3-fd0a9abc45a0
532024-09-05T00:48:52.943ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/0cdc26ae-935c-4875-88e3-fd0a9abc45a0
542024-09-05T00:48:52.943ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
552024-09-05T00:48:52.943ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::13
562024-09-05T00:48:52.943ZINFOcrucible-agent (worker): commit
572024-09-05T00:48:52.948ZINFOcrucible-agent (worker): ok!
582024-09-05T00:48:52.950ZINFOcrucible-agent (worker): creating missing snapshot instance snapshot-0cdc26ae-935c-4875-88e3-fd0a9abc45a0-27849cd6-3ba0-4d2d-9220-da51501c52b2
592024-09-05T00:48:52.959ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:snapshot-0cdc26ae-935c-4875-88e3-fd0a9abc45a0-27849cd6-3ba0-4d2d-9220-da51501c52b2
602024-09-05T00:48:52.976ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:snapshot-0cdc26ae-935c-4875-88e3-fd0a9abc45a0-27849cd6-3ba0-4d2d-9220-da51501c52b2
612024-09-05T00:48:52.976ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/0cdc26ae-935c-4875-88e3-fd0a9abc45a0/.zfs/snapshot/27849cd6-3ba0-4d2d-9220-da51501c52b2
622024-09-05T00:48:52.976ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19002
632024-09-05T00:48:52.976ZINFOcrucible-agent (worker): ensure mode SCF_TYPE_ASTRING ro
642024-09-05T00:48:52.976ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::13
652024-09-05T00:48:52.977ZINFOcrucible-agent (worker): commit
662024-09-05T00:48:52.982ZINFOcrucible-agent (worker): ok!
672024-09-05T00:48:52.991ZINFOcrucible-agent (worker): SMF ok!
682024-09-05T00:48:52.991ZINFOcrucible-agent (datafile): region 0cdc26ae-935c-4875-88e3-fd0a9abc45a0 running snapshot 27849cd6-3ba0-4d2d-9220-da51501c52b2 state: Requested -> Created
692024-09-05T00:48:53.161ZINFOcrucible-agent (datafile): path is "/data/regions/0cdc26ae-935c-4875-88e3-fd0a9abc45a0"
702024-09-05T00:48:53.171ZINFOcrucible-agent (datafile): dataset is oxp_3149c6f5-adf9-4e42-9a6f-ea817157070d/crucible/regions/0cdc26ae-935c-4875-88e3-fd0a9abc45a0
712024-09-05T00:48:53.180ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_3149c6f5-adf9-4e42-9a6f-ea817157070d/crucible/regions/0cdc26ae-935c-4875-88e3-fd0a9abc45a0@27849cd6-3ba0-4d2d-9220-da51501c52b2
722024-09-05T00:48:53.180ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_3149c6f5-adf9-4e42-9a6f-ea817157070d/crucible/regions/0cdc26ae-935c-4875-88e3-fd0a9abc45a0", "27849cd6-3ba0-4d2d-9220-da51501c52b2"]
732024-09-05T00:48:53.188ZINFOcrucible-agent (dropshot): request completed latency_us = 26758 local_addr = [fd00:1122:3344:101::13]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:57235 req_id = c83a0494-3a72-4fb3-a3ac-ae78fb92693a response_code = 200 uri = /crucible/0/regions/0cdc26ae-935c-4875-88e3-fd0a9abc45a0/snapshots/27849cd6-3ba0-4d2d-9220-da51501c52b2/run