Max level shown:
12024-12-17T08:23:02.182ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::12]:32345 remote_addr = [fd00:1122:3344:101::a]:57467
22024-12-17T08:23:02.182ZINFOcrucible-agent (datafile): region 3a735328-b2c5-4c25-b078-d38d3ff71e29 state: Requested
32024-12-17T08:23:02.184ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
42024-12-17T08:23:02.184ZINFOcrucible-agent (dropshot): request completed latency_us = 1344 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:57467 req_id = 2f2bb52a-5c0f-4846-b5cf-703df6d4b786 response_code = 200 uri = /crucible/0/regions
52024-12-17T08:23:02.195ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_f5f8cb88-f4e6-4642-9e86-74cec4dcf84f/crucible/regions/3a735328-b2c5-4c25-b078-d38d3ff71e29
62024-12-17T08:23:02.195ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_f5f8cb88-f4e6-4642-9e86-74cec4dcf84f/crucible/regions/3a735328-b2c5-4c25-b078-d38d3ff71e29
72024-12-17T08:23:02.263ZINFOcrucible-agent (worker): creating region Region { id: RegionId("3a735328-b2c5-4c25-b078-d38d3ff71e29"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19001, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/3a735328-b2c5-4c25-b078-d38d3ff71e29" region = 3a735328-b2c5-4c25-b078-d38d3ff71e29
82024-12-17T08:23:02.263ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "3a735328-b2c5-4c25-b078-d38d3ff71e29" "--data" "/data/regions/3a735328-b2c5-4c25-b078-d38d3ff71e29" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 3a735328-b2c5-4c25-b078-d38d3ff71e29
92024-12-17T08:23:02.334ZINFOcrucible-agent (worker): region files created ok region = 3a735328-b2c5-4c25-b078-d38d3ff71e29
102024-12-17T08:23:02.334ZINFOcrucible-agent (datafile): region 3a735328-b2c5-4c25-b078-d38d3ff71e29 state: Requested -> Created
112024-12-17T08:23:02.334ZINFOcrucible-agent (worker): applying SMF actions post create...
122024-12-17T08:23:02.335ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-3a735328-b2c5-4c25-b078-d38d3ff71e29
132024-12-17T08:23:02.341ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-3a735328-b2c5-4c25-b078-d38d3ff71e29
142024-12-17T08:23:02.355ZINFOcrucible-agent (worker): creating config property group
152024-12-17T08:23:02.358ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-3a735328-b2c5-4c25-b078-d38d3ff71e29
162024-12-17T08:23:02.359ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/3a735328-b2c5-4c25-b078-d38d3ff71e29
172024-12-17T08:23:02.359ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
182024-12-17T08:23:02.359ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::12
192024-12-17T08:23:02.359ZINFOcrucible-agent (worker): commit
202024-12-17T08:23:02.363ZINFOcrucible-agent (worker): ok!
212024-12-17T08:23:02.372ZINFOcrucible-agent (worker): SMF ok!
222024-12-17T08:23:02.483ZINFOcrucible-agent (dropshot): request completed latency_us = 80 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:57467 req_id = 57dbad7a-ef29-4466-9978-33c04afc2364 response_code = 200 uri = /crucible/0/regions
232024-12-17T08:23:04.332ZINFOcrucible-agent (dropshot): request completed latency_us = 2263 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:57467 req_id = 360b3300-aadb-4ff9-a820-a5dae8c8d496 response_code = 200 uri = /crucible/0/regions/9b58f204-fe6d-4cee-b2a4-988be0e2232b
242024-12-17T08:23:04.336ZINFOcrucible-agent (datafile): path is "/data/regions/9b58f204-fe6d-4cee-b2a4-988be0e2232b"
252024-12-17T08:23:04.344ZINFOcrucible-agent (datafile): dataset is oxp_f5f8cb88-f4e6-4642-9e86-74cec4dcf84f/crucible/regions/9b58f204-fe6d-4cee-b2a4-988be0e2232b
262024-12-17T08:23:04.352ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_f5f8cb88-f4e6-4642-9e86-74cec4dcf84f/crucible/regions/9b58f204-fe6d-4cee-b2a4-988be0e2232b@8c08efe9-6bd3-47a6-81a0-501c2168fbb5
272024-12-17T08:23:04.352ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_f5f8cb88-f4e6-4642-9e86-74cec4dcf84f/crucible/regions/9b58f204-fe6d-4cee-b2a4-988be0e2232b", "8c08efe9-6bd3-47a6-81a0-501c2168fbb5"]
282024-12-17T08:23:04.362ZINFOcrucible-agent (dropshot): request completed latency_us = 26605 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:57467 req_id = 98f39902-515e-4d95-8afc-75b7f7d572cf response_code = 200 uri = /crucible/0/regions/9b58f204-fe6d-4cee-b2a4-988be0e2232b/snapshots/8c08efe9-6bd3-47a6-81a0-501c2168fbb5
292024-12-17T08:23:04.366ZINFOcrucible-agent (datafile): path is "/data/regions/9b58f204-fe6d-4cee-b2a4-988be0e2232b"
302024-12-17T08:23:04.374ZINFOcrucible-agent (datafile): dataset is oxp_f5f8cb88-f4e6-4642-9e86-74cec4dcf84f/crucible/regions/9b58f204-fe6d-4cee-b2a4-988be0e2232b
312024-12-17T08:23:04.382ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_f5f8cb88-f4e6-4642-9e86-74cec4dcf84f/crucible/regions/9b58f204-fe6d-4cee-b2a4-988be0e2232b@8c08efe9-6bd3-47a6-81a0-501c2168fbb5
322024-12-17T08:23:04.382ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_f5f8cb88-f4e6-4642-9e86-74cec4dcf84f/crucible/regions/9b58f204-fe6d-4cee-b2a4-988be0e2232b", "8c08efe9-6bd3-47a6-81a0-501c2168fbb5"]
332024-12-17T08:23:04.391ZINFOcrucible-agent (datafile): requesting running snapshot 9b58f204-fe6d-4cee-b2a4-988be0e2232b-8c08efe9-6bd3-47a6-81a0-501c2168fbb5 state: Requested
342024-12-17T08:23:04.391ZINFOcrucible-agent (worker): applying SMF actions for region 9b58f204-fe6d-4cee-b2a4-988be0e2232b running snapshot 8c08efe9-6bd3-47a6-81a0-501c2168fbb5 (state Requested)...
352024-12-17T08:23:04.391ZINFOcrucible-agent (dropshot): request completed latency_us = 25791 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:57467 req_id = 87ad7323-7289-4795-9ab6-e7e7f4b134fc response_code = 200 uri = /crucible/0/regions/9b58f204-fe6d-4cee-b2a4-988be0e2232b/snapshots/8c08efe9-6bd3-47a6-81a0-501c2168fbb5/run
362024-12-17T08:23:04.394ZINFOcrucible-agent (worker): creating missing snapshot instance snapshot-9b58f204-fe6d-4cee-b2a4-988be0e2232b-8c08efe9-6bd3-47a6-81a0-501c2168fbb5
372024-12-17T08:23:04.404ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:snapshot-9b58f204-fe6d-4cee-b2a4-988be0e2232b-8c08efe9-6bd3-47a6-81a0-501c2168fbb5
382024-12-17T08:23:04.415ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:snapshot-9b58f204-fe6d-4cee-b2a4-988be0e2232b-8c08efe9-6bd3-47a6-81a0-501c2168fbb5
392024-12-17T08:23:04.415ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/9b58f204-fe6d-4cee-b2a4-988be0e2232b/.zfs/snapshot/8c08efe9-6bd3-47a6-81a0-501c2168fbb5
402024-12-17T08:23:04.415ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19002
412024-12-17T08:23:04.415ZINFOcrucible-agent (worker): ensure mode SCF_TYPE_ASTRING ro
422024-12-17T08:23:04.415ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::12
432024-12-17T08:23:04.416ZINFOcrucible-agent (worker): commit
442024-12-17T08:23:04.420ZINFOcrucible-agent (worker): ok!
452024-12-17T08:23:04.424ZINFOcrucible-agent (worker): SMF ok!
462024-12-17T08:23:04.424ZINFOcrucible-agent (datafile): region 9b58f204-fe6d-4cee-b2a4-988be0e2232b running snapshot 8c08efe9-6bd3-47a6-81a0-501c2168fbb5 state: Requested -> Created
472024-12-17T08:23:04.717ZINFOcrucible-agent (datafile): path is "/data/regions/9b58f204-fe6d-4cee-b2a4-988be0e2232b"
482024-12-17T08:23:04.726ZINFOcrucible-agent (datafile): dataset is oxp_f5f8cb88-f4e6-4642-9e86-74cec4dcf84f/crucible/regions/9b58f204-fe6d-4cee-b2a4-988be0e2232b
492024-12-17T08:23:04.734ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_f5f8cb88-f4e6-4642-9e86-74cec4dcf84f/crucible/regions/9b58f204-fe6d-4cee-b2a4-988be0e2232b@8c08efe9-6bd3-47a6-81a0-501c2168fbb5
502024-12-17T08:23:04.734ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_f5f8cb88-f4e6-4642-9e86-74cec4dcf84f/crucible/regions/9b58f204-fe6d-4cee-b2a4-988be0e2232b", "8c08efe9-6bd3-47a6-81a0-501c2168fbb5"]
512024-12-17T08:23:04.743ZINFOcrucible-agent (dropshot): request completed latency_us = 25680 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:57467 req_id = 03b28b22-b9c4-450f-8b05-bf0839dd5385 response_code = 200 uri = /crucible/0/regions/9b58f204-fe6d-4cee-b2a4-988be0e2232b/snapshots/8c08efe9-6bd3-47a6-81a0-501c2168fbb5/run
522024-12-17T08:23:08.939ZINFOcrucible-agent (datafile): region 6ca16eda-8882-4385-aca7-7e610560c56a state: Requested
532024-12-17T08:23:08.939ZINFOcrucible-agent (dropshot): request completed latency_us = 831 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:57467 req_id = 3c05fbe3-2969-4e03-af46-9329b39afe28 response_code = 200 uri = /crucible/0/regions
542024-12-17T08:23:08.939ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
552024-12-17T08:23:08.949ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_f5f8cb88-f4e6-4642-9e86-74cec4dcf84f/crucible/regions/6ca16eda-8882-4385-aca7-7e610560c56a
562024-12-17T08:23:08.949ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_f5f8cb88-f4e6-4642-9e86-74cec4dcf84f/crucible/regions/6ca16eda-8882-4385-aca7-7e610560c56a
572024-12-17T08:23:09.003ZINFOcrucible-agent (worker): creating region Region { id: RegionId("6ca16eda-8882-4385-aca7-7e610560c56a"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19003, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/6ca16eda-8882-4385-aca7-7e610560c56a" region = 6ca16eda-8882-4385-aca7-7e610560c56a
582024-12-17T08:23:09.003ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "6ca16eda-8882-4385-aca7-7e610560c56a" "--data" "/data/regions/6ca16eda-8882-4385-aca7-7e610560c56a" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 6ca16eda-8882-4385-aca7-7e610560c56a
592024-12-17T08:23:09.075ZINFOcrucible-agent (worker): region files created ok region = 6ca16eda-8882-4385-aca7-7e610560c56a
602024-12-17T08:23:09.075ZINFOcrucible-agent (datafile): region 6ca16eda-8882-4385-aca7-7e610560c56a state: Requested -> Created
612024-12-17T08:23:09.075ZINFOcrucible-agent (worker): applying SMF actions post create...
622024-12-17T08:23:09.076ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-6ca16eda-8882-4385-aca7-7e610560c56a
632024-12-17T08:23:09.079ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-6ca16eda-8882-4385-aca7-7e610560c56a
642024-12-17T08:23:09.086ZINFOcrucible-agent (worker): creating config property group
652024-12-17T08:23:09.089ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-6ca16eda-8882-4385-aca7-7e610560c56a
662024-12-17T08:23:09.090ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/6ca16eda-8882-4385-aca7-7e610560c56a
672024-12-17T08:23:09.090ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19003
682024-12-17T08:23:09.090ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::12
692024-12-17T08:23:09.090ZINFOcrucible-agent (worker): commit
702024-12-17T08:23:09.093ZINFOcrucible-agent (worker): ok!
712024-12-17T08:23:09.102ZINFOcrucible-agent (dropshot): request completed latency_us = 61 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:57467 req_id = 316bc57d-b7a6-4dbf-9c3b-c944dd0141f9 response_code = 200 uri = /crucible/0/regions
722024-12-17T08:23:09.104ZINFOcrucible-agent (worker): SMF ok!
732024-12-17T08:23:34.903ZINFOcrucible-agent (dropshot): request completed latency_us = 108 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:57467 req_id = cba9933a-ebda-4b54-9330-4c6937237204 response_code = 200 uri = /crucible/0/regions/6ca16eda-8882-4385-aca7-7e610560c56a
742024-12-17T08:23:34.906ZINFOcrucible-agent (datafile): path is "/data/regions/6ca16eda-8882-4385-aca7-7e610560c56a"
752024-12-17T08:23:34.918ZINFOcrucible-agent (datafile): dataset is oxp_f5f8cb88-f4e6-4642-9e86-74cec4dcf84f/crucible/regions/6ca16eda-8882-4385-aca7-7e610560c56a
762024-12-17T08:23:34.927ZINFOcrucible-agent (datafile): region 6ca16eda-8882-4385-aca7-7e610560c56a state: Created -> Tombstoned
772024-12-17T08:23:34.927ZINFOcrucible-agent (worker): applying SMF actions before removal...
782024-12-17T08:23:34.928ZINFOcrucible-agent (dropshot): request completed latency_us = 21192 local_addr = [fd00:1122:3344:101::12]:32345 method = DELETE remote_addr = [fd00:1122:3344:101::a]:57467 req_id = 4f5d17af-631d-476c-8c87-6bf0e236ed6d response_code = 204 uri = /crucible/0/regions/6ca16eda-8882-4385-aca7-7e610560c56a
792024-12-17T08:23:34.931ZINFOcrucible-agent (dropshot): request completed latency_us = 30 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:57467 req_id = 28050606-c0b1-44bf-b56f-ddcb1efa9c0b response_code = 200 uri = /crucible/0/regions/6ca16eda-8882-4385-aca7-7e610560c56a
802024-12-17T08:23:34.935ZINFOcrucible-agent (worker): disabling downstairs instance: downstairs-6ca16eda-8882-4385-aca7-7e610560c56a (instance states: (Some(Online), None))
812024-12-17T08:23:34.963ZINFOcrucible-agent (worker): SMF ok!
822024-12-17T08:23:34.973ZINFOcrucible-agent (worker): deleting zfs dataset "oxp_f5f8cb88-f4e6-4642-9e86-74cec4dcf84f/crucible/regions/6ca16eda-8882-4385-aca7-7e610560c56a" region = 6ca16eda-8882-4385-aca7-7e610560c56a
832024-12-17T08:23:35.023ZINFOcrucible-agent (dropshot): request completed latency_us = 93 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:57467 req_id = 7d8391f9-2e7a-444b-aa99-49fc3efb9e64 response_code = 200 uri = /crucible/0/regions/6ca16eda-8882-4385-aca7-7e610560c56a
842024-12-17T08:23:35.067ZINFOcrucible-agent (datafile): region 6ca16eda-8882-4385-aca7-7e610560c56a state: Tombstoned -> Destroyed
852024-12-17T08:23:35.158ZINFOcrucible-agent (dropshot): request completed latency_us = 48 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:57467 req_id = 1ce062be-7c18-4c66-a878-d672d5900059 response_code = 200 uri = /crucible/0/regions/6ca16eda-8882-4385-aca7-7e610560c56a