Max level shown:
12024-10-30T14:16:38.166ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::13]:32345 remote_addr = [fd00:1122:3344:101::a]:64649
22024-10-30T14:16:38.166ZINFOcrucible-agent (datafile): region f6cf64d9-5bec-477a-ad24-e3c94cffdcae state: Requested
32024-10-30T14:16:38.167ZINFOcrucible-agent (dropshot): request completed latency_us = 640 local_addr = [fd00:1122:3344:101::13]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:64649 req_id = a93da201-f9e4-44ef-a7bf-96dc54eae31d response_code = 200 uri = /crucible/0/regions
42024-10-30T14:16:38.167ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
52024-10-30T14:16:38.177ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_068f3094-32ac-46d9-a211-9e63ff96772e/crucible/regions/f6cf64d9-5bec-477a-ad24-e3c94cffdcae
62024-10-30T14:16:38.177ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_068f3094-32ac-46d9-a211-9e63ff96772e/crucible/regions/f6cf64d9-5bec-477a-ad24-e3c94cffdcae
72024-10-30T14:16:38.258ZINFOcrucible-agent (worker): creating region Region { id: RegionId("f6cf64d9-5bec-477a-ad24-e3c94cffdcae"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19001, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/f6cf64d9-5bec-477a-ad24-e3c94cffdcae" region = f6cf64d9-5bec-477a-ad24-e3c94cffdcae
82024-10-30T14:16:38.258ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "f6cf64d9-5bec-477a-ad24-e3c94cffdcae" "--data" "/data/regions/f6cf64d9-5bec-477a-ad24-e3c94cffdcae" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = f6cf64d9-5bec-477a-ad24-e3c94cffdcae
92024-10-30T14:16:38.414ZINFOcrucible-agent (worker): region files created ok region = f6cf64d9-5bec-477a-ad24-e3c94cffdcae
102024-10-30T14:16:38.414ZINFOcrucible-agent (datafile): region f6cf64d9-5bec-477a-ad24-e3c94cffdcae state: Requested -> Created
112024-10-30T14:16:38.414ZINFOcrucible-agent (worker): applying SMF actions post create...
122024-10-30T14:16:38.421ZINFOcrucible-agent (worker): existing directory value /dev/null does not match /data/regions/db61be8b-bba3-4efd-8443-9ac11888ecda
132024-10-30T14:16:38.421ZINFOcrucible-agent (worker): existing port value 0 does not match 19000
142024-10-30T14:16:38.422ZINFOcrucible-agent (worker): existing address value 0.0.0.0 does not match fd00:1122:3344:101::13
152024-10-30T14:16:38.422ZINFOcrucible-agent (worker): reconfig required
162024-10-30T14:16:38.422ZINFOcrucible-agent (worker): using existing config property group
172024-10-30T14:16:38.422ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-db61be8b-bba3-4efd-8443-9ac11888ecda
182024-10-30T14:16:38.422ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/db61be8b-bba3-4efd-8443-9ac11888ecda
192024-10-30T14:16:38.422ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
202024-10-30T14:16:38.422ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::13
212024-10-30T14:16:38.422ZINFOcrucible-agent (worker): commit
222024-10-30T14:16:38.433ZINFOcrucible-agent (worker): ok!
232024-10-30T14:16:38.433ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-f6cf64d9-5bec-477a-ad24-e3c94cffdcae
242024-10-30T14:16:38.441ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-f6cf64d9-5bec-477a-ad24-e3c94cffdcae
252024-10-30T14:16:38.456ZINFOcrucible-agent (worker): creating config property group
262024-10-30T14:16:38.463ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-f6cf64d9-5bec-477a-ad24-e3c94cffdcae
272024-10-30T14:16:38.463ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/f6cf64d9-5bec-477a-ad24-e3c94cffdcae
282024-10-30T14:16:38.463ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
292024-10-30T14:16:38.463ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::13
302024-10-30T14:16:38.463ZINFOcrucible-agent (worker): commit
312024-10-30T14:16:38.470ZINFOcrucible-agent (worker): ok!
322024-10-30T14:16:38.477ZINFOcrucible-agent (worker): SMF ok!
332024-10-30T14:16:38.627ZINFOcrucible-agent (dropshot): request completed latency_us = 65 local_addr = [fd00:1122:3344:101::13]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:64649 req_id = d7ffa77d-f5df-42a2-92fd-2f89aeb8a779 response_code = 200 uri = /crucible/0/regions
342024-10-30T14:16:40.085ZINFOcrucible-agent (dropshot): request completed latency_us = 2653 local_addr = [fd00:1122:3344:101::13]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:64649 req_id = a1c7666f-36f5-440e-849c-7d1ad038b6a0 response_code = 200 uri = /crucible/0/regions/db61be8b-bba3-4efd-8443-9ac11888ecda
352024-10-30T14:16:40.089ZINFOcrucible-agent (datafile): path is "/data/regions/db61be8b-bba3-4efd-8443-9ac11888ecda"
362024-10-30T14:16:40.098ZINFOcrucible-agent (datafile): dataset is oxp_068f3094-32ac-46d9-a211-9e63ff96772e/crucible/regions/db61be8b-bba3-4efd-8443-9ac11888ecda
372024-10-30T14:16:40.107ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_068f3094-32ac-46d9-a211-9e63ff96772e/crucible/regions/db61be8b-bba3-4efd-8443-9ac11888ecda@19898bc9-a062-4926-b165-e67066c9c552
382024-10-30T14:16:40.107ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_068f3094-32ac-46d9-a211-9e63ff96772e/crucible/regions/db61be8b-bba3-4efd-8443-9ac11888ecda", "19898bc9-a062-4926-b165-e67066c9c552"]
392024-10-30T14:16:40.120ZINFOcrucible-agent (dropshot): request completed latency_us = 31368 local_addr = [fd00:1122:3344:101::13]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:64649 req_id = c6e4478a-a138-4e07-9b2b-dbcb037273e0 response_code = 200 uri = /crucible/0/regions/db61be8b-bba3-4efd-8443-9ac11888ecda/snapshots/19898bc9-a062-4926-b165-e67066c9c552
402024-10-30T14:16:40.124ZINFOcrucible-agent (datafile): path is "/data/regions/db61be8b-bba3-4efd-8443-9ac11888ecda"
412024-10-30T14:16:40.132ZINFOcrucible-agent (datafile): dataset is oxp_068f3094-32ac-46d9-a211-9e63ff96772e/crucible/regions/db61be8b-bba3-4efd-8443-9ac11888ecda
422024-10-30T14:16:40.140ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_068f3094-32ac-46d9-a211-9e63ff96772e/crucible/regions/db61be8b-bba3-4efd-8443-9ac11888ecda@19898bc9-a062-4926-b165-e67066c9c552
432024-10-30T14:16:40.140ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_068f3094-32ac-46d9-a211-9e63ff96772e/crucible/regions/db61be8b-bba3-4efd-8443-9ac11888ecda", "19898bc9-a062-4926-b165-e67066c9c552"]
442024-10-30T14:16:40.154ZINFOcrucible-agent (datafile): requesting running snapshot db61be8b-bba3-4efd-8443-9ac11888ecda-19898bc9-a062-4926-b165-e67066c9c552 state: Requested
452024-10-30T14:16:40.155ZINFOcrucible-agent (worker): applying SMF actions for region db61be8b-bba3-4efd-8443-9ac11888ecda running snapshot 19898bc9-a062-4926-b165-e67066c9c552 (state Requested)...
462024-10-30T14:16:40.155ZINFOcrucible-agent (dropshot): request completed latency_us = 30922 local_addr = [fd00:1122:3344:101::13]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:64649 req_id = 5bf729a6-01cf-4e97-adaa-c3a97547eb41 response_code = 200 uri = /crucible/0/regions/db61be8b-bba3-4efd-8443-9ac11888ecda/snapshots/19898bc9-a062-4926-b165-e67066c9c552/run
472024-10-30T14:16:40.155ZINFOcrucible-agent (worker): existing directory value /dev/null does not match /data/regions/db61be8b-bba3-4efd-8443-9ac11888ecda
482024-10-30T14:16:40.155ZINFOcrucible-agent (worker): existing port value 0 does not match 19000
492024-10-30T14:16:40.156ZINFOcrucible-agent (worker): existing address value 0.0.0.0 does not match fd00:1122:3344:101::13
502024-10-30T14:16:40.156ZINFOcrucible-agent (worker): reconfig required
512024-10-30T14:16:40.156ZINFOcrucible-agent (worker): using existing config property group
522024-10-30T14:16:40.156ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-db61be8b-bba3-4efd-8443-9ac11888ecda
532024-10-30T14:16:40.156ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/db61be8b-bba3-4efd-8443-9ac11888ecda
542024-10-30T14:16:40.156ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
552024-10-30T14:16:40.156ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::13
562024-10-30T14:16:40.156ZINFOcrucible-agent (worker): commit
572024-10-30T14:16:40.165ZINFOcrucible-agent (worker): ok!
582024-10-30T14:16:40.167ZINFOcrucible-agent (worker): creating missing snapshot instance snapshot-db61be8b-bba3-4efd-8443-9ac11888ecda-19898bc9-a062-4926-b165-e67066c9c552
592024-10-30T14:16:40.181ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:snapshot-db61be8b-bba3-4efd-8443-9ac11888ecda-19898bc9-a062-4926-b165-e67066c9c552
602024-10-30T14:16:40.201ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:snapshot-db61be8b-bba3-4efd-8443-9ac11888ecda-19898bc9-a062-4926-b165-e67066c9c552
612024-10-30T14:16:40.202ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/db61be8b-bba3-4efd-8443-9ac11888ecda/.zfs/snapshot/19898bc9-a062-4926-b165-e67066c9c552
622024-10-30T14:16:40.202ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19002
632024-10-30T14:16:40.202ZINFOcrucible-agent (worker): ensure mode SCF_TYPE_ASTRING ro
642024-10-30T14:16:40.202ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::13
652024-10-30T14:16:40.202ZINFOcrucible-agent (worker): commit
662024-10-30T14:16:40.219ZINFOcrucible-agent (worker): ok!
672024-10-30T14:16:40.228ZINFOcrucible-agent (worker): SMF ok!
682024-10-30T14:16:40.228ZINFOcrucible-agent (datafile): region db61be8b-bba3-4efd-8443-9ac11888ecda running snapshot 19898bc9-a062-4926-b165-e67066c9c552 state: Requested -> Created
692024-10-30T14:16:40.508ZINFOcrucible-agent (datafile): path is "/data/regions/db61be8b-bba3-4efd-8443-9ac11888ecda"
702024-10-30T14:16:40.517ZINFOcrucible-agent (datafile): dataset is oxp_068f3094-32ac-46d9-a211-9e63ff96772e/crucible/regions/db61be8b-bba3-4efd-8443-9ac11888ecda
712024-10-30T14:16:40.525ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_068f3094-32ac-46d9-a211-9e63ff96772e/crucible/regions/db61be8b-bba3-4efd-8443-9ac11888ecda@19898bc9-a062-4926-b165-e67066c9c552
722024-10-30T14:16:40.525ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_068f3094-32ac-46d9-a211-9e63ff96772e/crucible/regions/db61be8b-bba3-4efd-8443-9ac11888ecda", "19898bc9-a062-4926-b165-e67066c9c552"]
732024-10-30T14:16:40.534ZINFOcrucible-agent (dropshot): request completed latency_us = 26015 local_addr = [fd00:1122:3344:101::13]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:64649 req_id = a195ee75-3a2f-441d-a78b-967cb6cac8c2 response_code = 200 uri = /crucible/0/regions/db61be8b-bba3-4efd-8443-9ac11888ecda/snapshots/19898bc9-a062-4926-b165-e67066c9c552/run