Max level shown:
12024-11-07T22:54:32.475ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::18]:32345 remote_addr = [fd00:1122:3344:101::a]:42111
22024-11-07T22:54:32.475ZINFOcrucible-agent (datafile): region 5a14485e-ae30-476d-b789-f3498800a567 state: Requested
32024-11-07T22:54:32.476ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
42024-11-07T22:54:32.476ZINFOcrucible-agent (dropshot): request completed latency_us = 862 local_addr = [fd00:1122:3344:101::18]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:42111 req_id = 9c3f295b-382d-4b26-8b2a-f4c2b9ec2199 response_code = 200 uri = /crucible/0/regions
52024-11-07T22:54:32.486ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_c38ffc78-518d-45b9-9391-e3e1820de913/crucible/regions/5a14485e-ae30-476d-b789-f3498800a567
62024-11-07T22:54:32.486ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_c38ffc78-518d-45b9-9391-e3e1820de913/crucible/regions/5a14485e-ae30-476d-b789-f3498800a567
72024-11-07T22:54:32.540ZINFOcrucible-agent (worker): creating region Region { id: RegionId("5a14485e-ae30-476d-b789-f3498800a567"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19001, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/5a14485e-ae30-476d-b789-f3498800a567" region = 5a14485e-ae30-476d-b789-f3498800a567
82024-11-07T22:54:32.540ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "5a14485e-ae30-476d-b789-f3498800a567" "--data" "/data/regions/5a14485e-ae30-476d-b789-f3498800a567" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 5a14485e-ae30-476d-b789-f3498800a567
92024-11-07T22:54:32.609ZINFOcrucible-agent (worker): region files created ok region = 5a14485e-ae30-476d-b789-f3498800a567
102024-11-07T22:54:32.609ZINFOcrucible-agent (datafile): region 5a14485e-ae30-476d-b789-f3498800a567 state: Requested -> Created
112024-11-07T22:54:32.609ZINFOcrucible-agent (worker): applying SMF actions post create...
122024-11-07T22:54:32.619ZINFOcrucible-agent (worker): existing directory value /dev/null does not match /data/regions/1ea419dc-b796-48b5-bd44-6cb31c067703
132024-11-07T22:54:32.620ZINFOcrucible-agent (worker): existing port value 0 does not match 19000
142024-11-07T22:54:32.620ZINFOcrucible-agent (worker): existing address value 0.0.0.0 does not match fd00:1122:3344:101::18
152024-11-07T22:54:32.620ZINFOcrucible-agent (worker): reconfig required
162024-11-07T22:54:32.620ZINFOcrucible-agent (worker): using existing config property group
172024-11-07T22:54:32.620ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-1ea419dc-b796-48b5-bd44-6cb31c067703
182024-11-07T22:54:32.620ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/1ea419dc-b796-48b5-bd44-6cb31c067703
192024-11-07T22:54:32.620ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
202024-11-07T22:54:32.620ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::18
212024-11-07T22:54:32.620ZINFOcrucible-agent (worker): commit
222024-11-07T22:54:32.628ZINFOcrucible-agent (worker): ok!
232024-11-07T22:54:32.628ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-5a14485e-ae30-476d-b789-f3498800a567
242024-11-07T22:54:32.632ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-5a14485e-ae30-476d-b789-f3498800a567
252024-11-07T22:54:32.640ZINFOcrucible-agent (worker): creating config property group
262024-11-07T22:54:32.644ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-5a14485e-ae30-476d-b789-f3498800a567
272024-11-07T22:54:32.644ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/5a14485e-ae30-476d-b789-f3498800a567
282024-11-07T22:54:32.644ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
292024-11-07T22:54:32.644ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::18
302024-11-07T22:54:32.644ZINFOcrucible-agent (worker): commit
312024-11-07T22:54:32.648ZINFOcrucible-agent (worker): ok!
322024-11-07T22:54:32.652ZINFOcrucible-agent (worker): SMF ok!
332024-11-07T22:54:32.738ZINFOcrucible-agent (dropshot): request completed latency_us = 74 local_addr = [fd00:1122:3344:101::18]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:42111 req_id = f43242ea-3292-42bc-8b62-7233a7c1b015 response_code = 200 uri = /crucible/0/regions
342024-11-07T22:54:33.878ZINFOcrucible-agent (dropshot): request completed latency_us = 5356 local_addr = [fd00:1122:3344:101::18]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:42111 req_id = 3d9ba2e9-8065-429f-80b1-41a478a44f61 response_code = 200 uri = /crucible/0/regions/1ea419dc-b796-48b5-bd44-6cb31c067703
352024-11-07T22:54:33.882ZINFOcrucible-agent (datafile): path is "/data/regions/1ea419dc-b796-48b5-bd44-6cb31c067703"
362024-11-07T22:54:33.890ZINFOcrucible-agent (datafile): dataset is oxp_c38ffc78-518d-45b9-9391-e3e1820de913/crucible/regions/1ea419dc-b796-48b5-bd44-6cb31c067703
372024-11-07T22:54:33.898ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_c38ffc78-518d-45b9-9391-e3e1820de913/crucible/regions/1ea419dc-b796-48b5-bd44-6cb31c067703@8a3c1b13-7dc3-4c4a-a191-8182c96f5eeb
382024-11-07T22:54:33.898ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_c38ffc78-518d-45b9-9391-e3e1820de913/crucible/regions/1ea419dc-b796-48b5-bd44-6cb31c067703", "8a3c1b13-7dc3-4c4a-a191-8182c96f5eeb"]
392024-11-07T22:54:33.908ZINFOcrucible-agent (dropshot): request completed latency_us = 26727 local_addr = [fd00:1122:3344:101::18]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:42111 req_id = 8310fe70-93a0-411a-9afa-277a5f3eace8 response_code = 200 uri = /crucible/0/regions/1ea419dc-b796-48b5-bd44-6cb31c067703/snapshots/8a3c1b13-7dc3-4c4a-a191-8182c96f5eeb
402024-11-07T22:54:33.912ZINFOcrucible-agent (datafile): path is "/data/regions/1ea419dc-b796-48b5-bd44-6cb31c067703"
412024-11-07T22:54:33.921ZINFOcrucible-agent (datafile): dataset is oxp_c38ffc78-518d-45b9-9391-e3e1820de913/crucible/regions/1ea419dc-b796-48b5-bd44-6cb31c067703
422024-11-07T22:54:33.929ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_c38ffc78-518d-45b9-9391-e3e1820de913/crucible/regions/1ea419dc-b796-48b5-bd44-6cb31c067703@8a3c1b13-7dc3-4c4a-a191-8182c96f5eeb
432024-11-07T22:54:33.929ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_c38ffc78-518d-45b9-9391-e3e1820de913/crucible/regions/1ea419dc-b796-48b5-bd44-6cb31c067703", "8a3c1b13-7dc3-4c4a-a191-8182c96f5eeb"]
442024-11-07T22:54:33.939ZINFOcrucible-agent (datafile): requesting running snapshot 1ea419dc-b796-48b5-bd44-6cb31c067703-8a3c1b13-7dc3-4c4a-a191-8182c96f5eeb state: Requested
452024-11-07T22:54:33.939ZINFOcrucible-agent (worker): applying SMF actions for region 1ea419dc-b796-48b5-bd44-6cb31c067703 running snapshot 8a3c1b13-7dc3-4c4a-a191-8182c96f5eeb (state Requested)...
462024-11-07T22:54:33.939ZINFOcrucible-agent (dropshot): request completed latency_us = 26803 local_addr = [fd00:1122:3344:101::18]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:42111 req_id = 9327ad33-ed75-4020-be30-ada8107bd966 response_code = 200 uri = /crucible/0/regions/1ea419dc-b796-48b5-bd44-6cb31c067703/snapshots/8a3c1b13-7dc3-4c4a-a191-8182c96f5eeb/run
472024-11-07T22:54:33.939ZINFOcrucible-agent (worker): existing directory value /dev/null does not match /data/regions/1ea419dc-b796-48b5-bd44-6cb31c067703
482024-11-07T22:54:33.940ZINFOcrucible-agent (worker): existing port value 0 does not match 19000
492024-11-07T22:54:33.940ZINFOcrucible-agent (worker): existing address value 0.0.0.0 does not match fd00:1122:3344:101::18
502024-11-07T22:54:33.940ZINFOcrucible-agent (worker): reconfig required
512024-11-07T22:54:33.940ZINFOcrucible-agent (worker): using existing config property group
522024-11-07T22:54:33.940ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-1ea419dc-b796-48b5-bd44-6cb31c067703
532024-11-07T22:54:33.940ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/1ea419dc-b796-48b5-bd44-6cb31c067703
542024-11-07T22:54:33.940ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
552024-11-07T22:54:33.940ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::18
562024-11-07T22:54:33.940ZINFOcrucible-agent (worker): commit
572024-11-07T22:54:33.944ZINFOcrucible-agent (worker): ok!
582024-11-07T22:54:33.945ZINFOcrucible-agent (worker): creating missing snapshot instance snapshot-1ea419dc-b796-48b5-bd44-6cb31c067703-8a3c1b13-7dc3-4c4a-a191-8182c96f5eeb
592024-11-07T22:54:33.950ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:snapshot-1ea419dc-b796-48b5-bd44-6cb31c067703-8a3c1b13-7dc3-4c4a-a191-8182c96f5eeb
602024-11-07T22:54:33.959ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:snapshot-1ea419dc-b796-48b5-bd44-6cb31c067703-8a3c1b13-7dc3-4c4a-a191-8182c96f5eeb
612024-11-07T22:54:33.960ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/1ea419dc-b796-48b5-bd44-6cb31c067703/.zfs/snapshot/8a3c1b13-7dc3-4c4a-a191-8182c96f5eeb
622024-11-07T22:54:33.960ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19002
632024-11-07T22:54:33.960ZINFOcrucible-agent (worker): ensure mode SCF_TYPE_ASTRING ro
642024-11-07T22:54:33.960ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::18
652024-11-07T22:54:33.960ZINFOcrucible-agent (worker): commit
662024-11-07T22:54:33.964ZINFOcrucible-agent (worker): ok!
672024-11-07T22:54:33.968ZINFOcrucible-agent (worker): SMF ok!
682024-11-07T22:54:33.968ZINFOcrucible-agent (datafile): region 1ea419dc-b796-48b5-bd44-6cb31c067703 running snapshot 8a3c1b13-7dc3-4c4a-a191-8182c96f5eeb state: Requested -> Created
692024-11-07T22:54:34.260ZINFOcrucible-agent (datafile): path is "/data/regions/1ea419dc-b796-48b5-bd44-6cb31c067703"
702024-11-07T22:54:34.269ZINFOcrucible-agent (datafile): dataset is oxp_c38ffc78-518d-45b9-9391-e3e1820de913/crucible/regions/1ea419dc-b796-48b5-bd44-6cb31c067703
712024-11-07T22:54:34.277ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_c38ffc78-518d-45b9-9391-e3e1820de913/crucible/regions/1ea419dc-b796-48b5-bd44-6cb31c067703@8a3c1b13-7dc3-4c4a-a191-8182c96f5eeb
722024-11-07T22:54:34.277ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_c38ffc78-518d-45b9-9391-e3e1820de913/crucible/regions/1ea419dc-b796-48b5-bd44-6cb31c067703", "8a3c1b13-7dc3-4c4a-a191-8182c96f5eeb"]
732024-11-07T22:54:34.286ZINFOcrucible-agent (dropshot): request completed latency_us = 26201 local_addr = [fd00:1122:3344:101::18]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:42111 req_id = 8d3b14bb-814e-477c-883e-5da3e0036847 response_code = 200 uri = /crucible/0/regions/1ea419dc-b796-48b5-bd44-6cb31c067703/snapshots/8a3c1b13-7dc3-4c4a-a191-8182c96f5eeb/run