Max level shown:
12024-09-10T21:31:07.929ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::13]:32345 remote_addr = [fd00:1122:3344:101::a]:54225
22024-09-10T21:31:07.929ZINFOcrucible-agent (datafile): region 932faeca-2cb4-4ee4-b151-90aa6b4f6a96 state: Requested
32024-09-10T21:31:07.929ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
42024-09-10T21:31:07.929ZINFOcrucible-agent (dropshot): request completed latency_us = 655 local_addr = [fd00:1122:3344:101::13]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:54225 req_id = 80c564ba-14bd-4c40-ba76-42fc3fe07f4b response_code = 200 uri = /crucible/0/regions
52024-09-10T21:31:07.940ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_1d09a0b5-64fe-49da-9f0c-2aeb05367248/crucible/regions/932faeca-2cb4-4ee4-b151-90aa6b4f6a96
62024-09-10T21:31:07.940ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_1d09a0b5-64fe-49da-9f0c-2aeb05367248/crucible/regions/932faeca-2cb4-4ee4-b151-90aa6b4f6a96
72024-09-10T21:31:07.990ZINFOcrucible-agent (worker): creating region Region { id: RegionId("932faeca-2cb4-4ee4-b151-90aa6b4f6a96"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19001, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/932faeca-2cb4-4ee4-b151-90aa6b4f6a96" region = 932faeca-2cb4-4ee4-b151-90aa6b4f6a96
82024-09-10T21:31:07.992ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "932faeca-2cb4-4ee4-b151-90aa6b4f6a96" "--data" "/data/regions/932faeca-2cb4-4ee4-b151-90aa6b4f6a96" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 932faeca-2cb4-4ee4-b151-90aa6b4f6a96
92024-09-10T21:31:08.061ZINFOcrucible-agent (worker): region files created ok region = 932faeca-2cb4-4ee4-b151-90aa6b4f6a96
102024-09-10T21:31:08.061ZINFOcrucible-agent (datafile): region 932faeca-2cb4-4ee4-b151-90aa6b4f6a96 state: Requested -> Created
112024-09-10T21:31:08.061ZINFOcrucible-agent (worker): applying SMF actions post create...
122024-09-10T21:31:08.062ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-932faeca-2cb4-4ee4-b151-90aa6b4f6a96
132024-09-10T21:31:08.066ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-932faeca-2cb4-4ee4-b151-90aa6b4f6a96
142024-09-10T21:31:08.079ZINFOcrucible-agent (worker): creating config property group
152024-09-10T21:31:08.083ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-932faeca-2cb4-4ee4-b151-90aa6b4f6a96
162024-09-10T21:31:08.083ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/932faeca-2cb4-4ee4-b151-90aa6b4f6a96
172024-09-10T21:31:08.083ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
182024-09-10T21:31:08.083ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::13
192024-09-10T21:31:08.083ZINFOcrucible-agent (worker): commit
202024-09-10T21:31:08.087ZINFOcrucible-agent (worker): ok!
212024-09-10T21:31:08.092ZINFOcrucible-agent (worker): existing directory value /dev/null does not match /data/regions/a6fdfa2d-8d70-48b1-9240-60453da5b0a0
222024-09-10T21:31:08.092ZINFOcrucible-agent (worker): existing port value 0 does not match 19000
232024-09-10T21:31:08.092ZINFOcrucible-agent (worker): existing address value 0.0.0.0 does not match fd00:1122:3344:101::13
242024-09-10T21:31:08.092ZINFOcrucible-agent (worker): reconfig required
252024-09-10T21:31:08.092ZINFOcrucible-agent (worker): using existing config property group
262024-09-10T21:31:08.092ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-a6fdfa2d-8d70-48b1-9240-60453da5b0a0
272024-09-10T21:31:08.092ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/a6fdfa2d-8d70-48b1-9240-60453da5b0a0
282024-09-10T21:31:08.095ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
292024-09-10T21:31:08.095ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::13
302024-09-10T21:31:08.095ZINFOcrucible-agent (worker): commit
312024-09-10T21:31:08.102ZINFOcrucible-agent (worker): ok!
322024-09-10T21:31:08.102ZINFOcrucible-agent (worker): SMF ok!
332024-09-10T21:31:08.216ZINFOcrucible-agent (dropshot): request completed latency_us = 72 local_addr = [fd00:1122:3344:101::13]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:54225 req_id = 2b1cd6f6-0c76-4746-8409-02dbe156901a response_code = 200 uri = /crucible/0/regions
342024-09-10T21:31:09.797ZINFOcrucible-agent (dropshot): request completed latency_us = 2703 local_addr = [fd00:1122:3344:101::13]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:54225 req_id = b2b034df-0c7d-4ad6-b4f0-9bb657cd69cd response_code = 200 uri = /crucible/0/regions/a6fdfa2d-8d70-48b1-9240-60453da5b0a0
352024-09-10T21:31:09.802ZINFOcrucible-agent (datafile): path is "/data/regions/a6fdfa2d-8d70-48b1-9240-60453da5b0a0"
362024-09-10T21:31:09.811ZINFOcrucible-agent (datafile): dataset is oxp_1d09a0b5-64fe-49da-9f0c-2aeb05367248/crucible/regions/a6fdfa2d-8d70-48b1-9240-60453da5b0a0
372024-09-10T21:31:09.820ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_1d09a0b5-64fe-49da-9f0c-2aeb05367248/crucible/regions/a6fdfa2d-8d70-48b1-9240-60453da5b0a0@ef68791a-37f5-4179-9c0a-3d6af1978a0e
382024-09-10T21:31:09.820ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_1d09a0b5-64fe-49da-9f0c-2aeb05367248/crucible/regions/a6fdfa2d-8d70-48b1-9240-60453da5b0a0", "ef68791a-37f5-4179-9c0a-3d6af1978a0e"]
392024-09-10T21:31:09.831ZINFOcrucible-agent (dropshot): request completed latency_us = 29115 local_addr = [fd00:1122:3344:101::13]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:54225 req_id = 3ba257bc-a379-4797-bf49-cc9dcb9df538 response_code = 200 uri = /crucible/0/regions/a6fdfa2d-8d70-48b1-9240-60453da5b0a0/snapshots/ef68791a-37f5-4179-9c0a-3d6af1978a0e
402024-09-10T21:31:09.835ZINFOcrucible-agent (datafile): path is "/data/regions/a6fdfa2d-8d70-48b1-9240-60453da5b0a0"
412024-09-10T21:31:09.842ZINFOcrucible-agent (datafile): dataset is oxp_1d09a0b5-64fe-49da-9f0c-2aeb05367248/crucible/regions/a6fdfa2d-8d70-48b1-9240-60453da5b0a0
422024-09-10T21:31:09.849ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_1d09a0b5-64fe-49da-9f0c-2aeb05367248/crucible/regions/a6fdfa2d-8d70-48b1-9240-60453da5b0a0@ef68791a-37f5-4179-9c0a-3d6af1978a0e
432024-09-10T21:31:09.849ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_1d09a0b5-64fe-49da-9f0c-2aeb05367248/crucible/regions/a6fdfa2d-8d70-48b1-9240-60453da5b0a0", "ef68791a-37f5-4179-9c0a-3d6af1978a0e"]
442024-09-10T21:31:09.859ZINFOcrucible-agent (datafile): requesting running snapshot a6fdfa2d-8d70-48b1-9240-60453da5b0a0-ef68791a-37f5-4179-9c0a-3d6af1978a0e state: Requested
452024-09-10T21:31:09.859ZINFOcrucible-agent (worker): applying SMF actions for region a6fdfa2d-8d70-48b1-9240-60453da5b0a0 running snapshot ef68791a-37f5-4179-9c0a-3d6af1978a0e (state Requested)...
462024-09-10T21:31:09.859ZINFOcrucible-agent (dropshot): request completed latency_us = 24554 local_addr = [fd00:1122:3344:101::13]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:54225 req_id = 35de494b-5fc3-48b2-ab96-a118b7130f78 response_code = 200 uri = /crucible/0/regions/a6fdfa2d-8d70-48b1-9240-60453da5b0a0/snapshots/ef68791a-37f5-4179-9c0a-3d6af1978a0e/run
472024-09-10T21:31:09.861ZINFOcrucible-agent (worker): existing directory value /dev/null does not match /data/regions/a6fdfa2d-8d70-48b1-9240-60453da5b0a0
482024-09-10T21:31:09.861ZINFOcrucible-agent (worker): existing port value 0 does not match 19000
492024-09-10T21:31:09.861ZINFOcrucible-agent (worker): existing address value 0.0.0.0 does not match fd00:1122:3344:101::13
502024-09-10T21:31:09.861ZINFOcrucible-agent (worker): reconfig required
512024-09-10T21:31:09.861ZINFOcrucible-agent (worker): using existing config property group
522024-09-10T21:31:09.861ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-a6fdfa2d-8d70-48b1-9240-60453da5b0a0
532024-09-10T21:31:09.861ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/a6fdfa2d-8d70-48b1-9240-60453da5b0a0
542024-09-10T21:31:09.861ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
552024-09-10T21:31:09.861ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::13
562024-09-10T21:31:09.861ZINFOcrucible-agent (worker): commit
572024-09-10T21:31:09.865ZINFOcrucible-agent (worker): ok!
582024-09-10T21:31:09.865ZINFOcrucible-agent (worker): creating missing snapshot instance snapshot-a6fdfa2d-8d70-48b1-9240-60453da5b0a0-ef68791a-37f5-4179-9c0a-3d6af1978a0e
592024-09-10T21:31:09.870ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:snapshot-a6fdfa2d-8d70-48b1-9240-60453da5b0a0-ef68791a-37f5-4179-9c0a-3d6af1978a0e
602024-09-10T21:31:09.881ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:snapshot-a6fdfa2d-8d70-48b1-9240-60453da5b0a0-ef68791a-37f5-4179-9c0a-3d6af1978a0e
612024-09-10T21:31:09.881ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/a6fdfa2d-8d70-48b1-9240-60453da5b0a0/.zfs/snapshot/ef68791a-37f5-4179-9c0a-3d6af1978a0e
622024-09-10T21:31:09.881ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19002
632024-09-10T21:31:09.881ZINFOcrucible-agent (worker): ensure mode SCF_TYPE_ASTRING ro
642024-09-10T21:31:09.881ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::13
652024-09-10T21:31:09.881ZINFOcrucible-agent (worker): commit
662024-09-10T21:31:09.885ZINFOcrucible-agent (worker): ok!
672024-09-10T21:31:09.890ZINFOcrucible-agent (worker): SMF ok!
682024-09-10T21:31:09.890ZINFOcrucible-agent (datafile): region a6fdfa2d-8d70-48b1-9240-60453da5b0a0 running snapshot ef68791a-37f5-4179-9c0a-3d6af1978a0e state: Requested -> Created
692024-09-10T21:31:10.015ZINFOcrucible-agent (datafile): path is "/data/regions/a6fdfa2d-8d70-48b1-9240-60453da5b0a0"
702024-09-10T21:31:10.023ZINFOcrucible-agent (datafile): dataset is oxp_1d09a0b5-64fe-49da-9f0c-2aeb05367248/crucible/regions/a6fdfa2d-8d70-48b1-9240-60453da5b0a0
712024-09-10T21:31:10.031ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_1d09a0b5-64fe-49da-9f0c-2aeb05367248/crucible/regions/a6fdfa2d-8d70-48b1-9240-60453da5b0a0@ef68791a-37f5-4179-9c0a-3d6af1978a0e
722024-09-10T21:31:10.031ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_1d09a0b5-64fe-49da-9f0c-2aeb05367248/crucible/regions/a6fdfa2d-8d70-48b1-9240-60453da5b0a0", "ef68791a-37f5-4179-9c0a-3d6af1978a0e"]
732024-09-10T21:31:10.040ZINFOcrucible-agent (dropshot): request completed latency_us = 25188 local_addr = [fd00:1122:3344:101::13]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:54225 req_id = c4308739-18bb-4e68-9576-ed03b4ed3f58 response_code = 200 uri = /crucible/0/regions/a6fdfa2d-8d70-48b1-9240-60453da5b0a0/snapshots/ef68791a-37f5-4179-9c0a-3d6af1978a0e/run