Max level shown:
12024-09-10T21:31:07.928ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::19]:32345 remote_addr = [fd00:1122:3344:101::a]:48057
22024-09-10T21:31:07.928ZINFOcrucible-agent (datafile): region d4aa5b05-a4a6-45d6-8bab-e877455195b5 state: Requested
32024-09-10T21:31:07.929ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
42024-09-10T21:31:07.929ZINFOcrucible-agent (dropshot): request completed latency_us = 666 local_addr = [fd00:1122:3344:101::19]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:48057 req_id = 884c253d-0bc9-4a61-a5b5-5b080e3faa6d response_code = 200 uri = /crucible/0/regions
52024-09-10T21:31:07.941ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_e82c3049-23d1-4237-ad1f-14e37401e51a/crucible/regions/d4aa5b05-a4a6-45d6-8bab-e877455195b5
62024-09-10T21:31:07.941ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_e82c3049-23d1-4237-ad1f-14e37401e51a/crucible/regions/d4aa5b05-a4a6-45d6-8bab-e877455195b5
72024-09-10T21:31:07.989ZINFOcrucible-agent (worker): creating region Region { id: RegionId("d4aa5b05-a4a6-45d6-8bab-e877455195b5"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19001, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/d4aa5b05-a4a6-45d6-8bab-e877455195b5" region = d4aa5b05-a4a6-45d6-8bab-e877455195b5
82024-09-10T21:31:07.991ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "d4aa5b05-a4a6-45d6-8bab-e877455195b5" "--data" "/data/regions/d4aa5b05-a4a6-45d6-8bab-e877455195b5" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = d4aa5b05-a4a6-45d6-8bab-e877455195b5
92024-09-10T21:31:08.103ZINFOcrucible-agent (worker): region files created ok region = d4aa5b05-a4a6-45d6-8bab-e877455195b5
102024-09-10T21:31:08.103ZINFOcrucible-agent (datafile): region d4aa5b05-a4a6-45d6-8bab-e877455195b5 state: Requested -> Created
112024-09-10T21:31:08.103ZINFOcrucible-agent (worker): applying SMF actions post create...
122024-09-10T21:31:08.103ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-d4aa5b05-a4a6-45d6-8bab-e877455195b5
132024-09-10T21:31:08.108ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-d4aa5b05-a4a6-45d6-8bab-e877455195b5
142024-09-10T21:31:08.118ZINFOcrucible-agent (worker): creating config property group
152024-09-10T21:31:08.122ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-d4aa5b05-a4a6-45d6-8bab-e877455195b5
162024-09-10T21:31:08.122ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/d4aa5b05-a4a6-45d6-8bab-e877455195b5
172024-09-10T21:31:08.122ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
182024-09-10T21:31:08.122ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::19
192024-09-10T21:31:08.122ZINFOcrucible-agent (worker): commit
202024-09-10T21:31:08.126ZINFOcrucible-agent (worker): ok!
212024-09-10T21:31:08.131ZINFOcrucible-agent (worker): existing directory value /dev/null does not match /data/regions/e8249a2c-c480-4ff7-ba26-7184920e5082
222024-09-10T21:31:08.131ZINFOcrucible-agent (worker): existing port value 0 does not match 19000
232024-09-10T21:31:08.131ZINFOcrucible-agent (worker): existing address value 0.0.0.0 does not match fd00:1122:3344:101::19
242024-09-10T21:31:08.131ZINFOcrucible-agent (worker): reconfig required
252024-09-10T21:31:08.131ZINFOcrucible-agent (worker): using existing config property group
262024-09-10T21:31:08.131ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-e8249a2c-c480-4ff7-ba26-7184920e5082
272024-09-10T21:31:08.131ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/e8249a2c-c480-4ff7-ba26-7184920e5082
282024-09-10T21:31:08.136ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
292024-09-10T21:31:08.136ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::19
302024-09-10T21:31:08.136ZINFOcrucible-agent (worker): commit
312024-09-10T21:31:08.136ZINFOcrucible-agent (dropshot): request completed latency_us = 64 local_addr = [fd00:1122:3344:101::19]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:48057 req_id = 0c3dac4e-0ac2-4dcb-84dd-bc91f84ef655 response_code = 200 uri = /crucible/0/regions
322024-09-10T21:31:08.143ZINFOcrucible-agent (worker): ok!
332024-09-10T21:31:08.143ZINFOcrucible-agent (worker): SMF ok!
342024-09-10T21:31:08.984ZINFOcrucible-agent (dropshot): request completed latency_us = 2543 local_addr = [fd00:1122:3344:101::19]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:48057 req_id = a8a1d7ac-e952-43d4-ad13-299d8739c77e response_code = 200 uri = /crucible/0/regions/e8249a2c-c480-4ff7-ba26-7184920e5082
352024-09-10T21:31:08.987ZINFOcrucible-agent (datafile): path is "/data/regions/e8249a2c-c480-4ff7-ba26-7184920e5082"
362024-09-10T21:31:08.996ZINFOcrucible-agent (datafile): dataset is oxp_e82c3049-23d1-4237-ad1f-14e37401e51a/crucible/regions/e8249a2c-c480-4ff7-ba26-7184920e5082
372024-09-10T21:31:09.004ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_e82c3049-23d1-4237-ad1f-14e37401e51a/crucible/regions/e8249a2c-c480-4ff7-ba26-7184920e5082@ef68791a-37f5-4179-9c0a-3d6af1978a0e
382024-09-10T21:31:09.004ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_e82c3049-23d1-4237-ad1f-14e37401e51a/crucible/regions/e8249a2c-c480-4ff7-ba26-7184920e5082", "ef68791a-37f5-4179-9c0a-3d6af1978a0e"]
392024-09-10T21:31:09.014ZINFOcrucible-agent (dropshot): request completed latency_us = 27359 local_addr = [fd00:1122:3344:101::19]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:48057 req_id = 66f623d0-dd44-4af4-8ba2-9c0f0b157e50 response_code = 200 uri = /crucible/0/regions/e8249a2c-c480-4ff7-ba26-7184920e5082/snapshots/ef68791a-37f5-4179-9c0a-3d6af1978a0e
402024-09-10T21:31:09.018ZINFOcrucible-agent (datafile): path is "/data/regions/e8249a2c-c480-4ff7-ba26-7184920e5082"
412024-09-10T21:31:09.026ZINFOcrucible-agent (datafile): dataset is oxp_e82c3049-23d1-4237-ad1f-14e37401e51a/crucible/regions/e8249a2c-c480-4ff7-ba26-7184920e5082
422024-09-10T21:31:09.033ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_e82c3049-23d1-4237-ad1f-14e37401e51a/crucible/regions/e8249a2c-c480-4ff7-ba26-7184920e5082@ef68791a-37f5-4179-9c0a-3d6af1978a0e
432024-09-10T21:31:09.033ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_e82c3049-23d1-4237-ad1f-14e37401e51a/crucible/regions/e8249a2c-c480-4ff7-ba26-7184920e5082", "ef68791a-37f5-4179-9c0a-3d6af1978a0e"]
442024-09-10T21:31:09.041ZINFOcrucible-agent (datafile): requesting running snapshot e8249a2c-c480-4ff7-ba26-7184920e5082-ef68791a-37f5-4179-9c0a-3d6af1978a0e state: Requested
452024-09-10T21:31:09.041ZINFOcrucible-agent (worker): applying SMF actions for region e8249a2c-c480-4ff7-ba26-7184920e5082 running snapshot ef68791a-37f5-4179-9c0a-3d6af1978a0e (state Requested)...
462024-09-10T21:31:09.041ZINFOcrucible-agent (dropshot): request completed latency_us = 22767 local_addr = [fd00:1122:3344:101::19]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:48057 req_id = 68b2e4fb-f266-4005-ab6a-4df8b9d61fd6 response_code = 200 uri = /crucible/0/regions/e8249a2c-c480-4ff7-ba26-7184920e5082/snapshots/ef68791a-37f5-4179-9c0a-3d6af1978a0e/run
472024-09-10T21:31:09.042ZINFOcrucible-agent (worker): existing directory value /dev/null does not match /data/regions/e8249a2c-c480-4ff7-ba26-7184920e5082
482024-09-10T21:31:09.042ZINFOcrucible-agent (worker): existing port value 0 does not match 19000
492024-09-10T21:31:09.043ZINFOcrucible-agent (worker): existing address value 0.0.0.0 does not match fd00:1122:3344:101::19
502024-09-10T21:31:09.043ZINFOcrucible-agent (worker): reconfig required
512024-09-10T21:31:09.043ZINFOcrucible-agent (worker): using existing config property group
522024-09-10T21:31:09.043ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-e8249a2c-c480-4ff7-ba26-7184920e5082
532024-09-10T21:31:09.043ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/e8249a2c-c480-4ff7-ba26-7184920e5082
542024-09-10T21:31:09.043ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
552024-09-10T21:31:09.043ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::19
562024-09-10T21:31:09.043ZINFOcrucible-agent (worker): commit
572024-09-10T21:31:09.047ZINFOcrucible-agent (worker): ok!
582024-09-10T21:31:09.047ZINFOcrucible-agent (worker): creating missing snapshot instance snapshot-e8249a2c-c480-4ff7-ba26-7184920e5082-ef68791a-37f5-4179-9c0a-3d6af1978a0e
592024-09-10T21:31:09.054ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:snapshot-e8249a2c-c480-4ff7-ba26-7184920e5082-ef68791a-37f5-4179-9c0a-3d6af1978a0e
602024-09-10T21:31:09.067ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:snapshot-e8249a2c-c480-4ff7-ba26-7184920e5082-ef68791a-37f5-4179-9c0a-3d6af1978a0e
612024-09-10T21:31:09.067ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/e8249a2c-c480-4ff7-ba26-7184920e5082/.zfs/snapshot/ef68791a-37f5-4179-9c0a-3d6af1978a0e
622024-09-10T21:31:09.067ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19002
632024-09-10T21:31:09.067ZINFOcrucible-agent (worker): ensure mode SCF_TYPE_ASTRING ro
642024-09-10T21:31:09.067ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::19
652024-09-10T21:31:09.067ZINFOcrucible-agent (worker): commit
662024-09-10T21:31:09.078ZINFOcrucible-agent (worker): ok!
672024-09-10T21:31:09.088ZINFOcrucible-agent (worker): SMF ok!
682024-09-10T21:31:09.088ZINFOcrucible-agent (datafile): region e8249a2c-c480-4ff7-ba26-7184920e5082 running snapshot ef68791a-37f5-4179-9c0a-3d6af1978a0e state: Requested -> Created
692024-09-10T21:31:09.360ZINFOcrucible-agent (datafile): path is "/data/regions/e8249a2c-c480-4ff7-ba26-7184920e5082"
702024-09-10T21:31:09.370ZINFOcrucible-agent (datafile): dataset is oxp_e82c3049-23d1-4237-ad1f-14e37401e51a/crucible/regions/e8249a2c-c480-4ff7-ba26-7184920e5082
712024-09-10T21:31:09.377ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_e82c3049-23d1-4237-ad1f-14e37401e51a/crucible/regions/e8249a2c-c480-4ff7-ba26-7184920e5082@ef68791a-37f5-4179-9c0a-3d6af1978a0e
722024-09-10T21:31:09.377ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_e82c3049-23d1-4237-ad1f-14e37401e51a/crucible/regions/e8249a2c-c480-4ff7-ba26-7184920e5082", "ef68791a-37f5-4179-9c0a-3d6af1978a0e"]
732024-09-10T21:31:09.386ZINFOcrucible-agent (dropshot): request completed latency_us = 25648 local_addr = [fd00:1122:3344:101::19]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:48057 req_id = a20986b5-481a-4cd7-96d1-d53556be5585 response_code = 200 uri = /crucible/0/regions/e8249a2c-c480-4ff7-ba26-7184920e5082/snapshots/ef68791a-37f5-4179-9c0a-3d6af1978a0e/run