Max level shown:
12024-07-19T23:03:10.794ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::18]:32345 remote_addr = [fd00:1122:3344:101::a]:38083
22024-07-19T23:03:10.817ZINFOcrucible-agent (datafile): region 6341e33d-f9c6-4d7d-a567-8f58cb653e37 state: Requested
32024-07-19T23:03:10.827ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
42024-07-19T23:03:10.827ZINFOcrucible-agent (dropshot): request completed latency_us = 18906 local_addr = [fd00:1122:3344:101::18]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:38083 req_id = 21307521-01a0-4ee6-a4f4-071cb9da9228 response_code = 200 uri = /crucible/0/regions
52024-07-19T23:03:10.843ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_5e9012da-baa1-440f-b7c2-376e7117cb03/crucible/regions/6341e33d-f9c6-4d7d-a567-8f58cb653e37
62024-07-19T23:03:10.843ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_5e9012da-baa1-440f-b7c2-376e7117cb03/crucible/regions/6341e33d-f9c6-4d7d-a567-8f58cb653e37
72024-07-19T23:03:10.910ZINFOcrucible-agent (worker): creating region Region { id: RegionId("6341e33d-f9c6-4d7d-a567-8f58cb653e37"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/6341e33d-f9c6-4d7d-a567-8f58cb653e37" region = 6341e33d-f9c6-4d7d-a567-8f58cb653e37
82024-07-19T23:03:10.913ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "6341e33d-f9c6-4d7d-a567-8f58cb653e37" "--data" "/data/regions/6341e33d-f9c6-4d7d-a567-8f58cb653e37" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 6341e33d-f9c6-4d7d-a567-8f58cb653e37
92024-07-19T23:03:10.970ZINFOcrucible-agent (dropshot): request completed latency_us = 584 local_addr = [fd00:1122:3344:101::18]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:38083 req_id = d1a9febd-5abe-46d7-8d1a-d6e071bc41ef response_code = 200 uri = /crucible/0/regions
102024-07-19T23:03:11.215ZINFOcrucible-agent (worker): region files created ok region = 6341e33d-f9c6-4d7d-a567-8f58cb653e37
112024-07-19T23:03:11.215ZINFOcrucible-agent (datafile): region 6341e33d-f9c6-4d7d-a567-8f58cb653e37 state: Requested -> Created
122024-07-19T23:03:11.215ZINFOcrucible-agent (worker): applying SMF actions post create...
132024-07-19T23:03:11.215ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-6341e33d-f9c6-4d7d-a567-8f58cb653e37
142024-07-19T23:03:11.224ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-6341e33d-f9c6-4d7d-a567-8f58cb653e37
152024-07-19T23:03:11.243ZINFOcrucible-agent (worker): creating config property group
162024-07-19T23:03:11.246ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-6341e33d-f9c6-4d7d-a567-8f58cb653e37
172024-07-19T23:03:11.247ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/6341e33d-f9c6-4d7d-a567-8f58cb653e37
182024-07-19T23:03:11.252ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
192024-07-19T23:03:11.252ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::18
202024-07-19T23:03:11.252ZINFOcrucible-agent (worker): commit
212024-07-19T23:03:11.261ZINFOcrucible-agent (worker): ok!
222024-07-19T23:03:11.269ZINFOcrucible-agent (worker): SMF ok!
232024-07-19T23:03:11.304ZINFOcrucible-agent (dropshot): request completed latency_us = 86 local_addr = [fd00:1122:3344:101::18]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:38083 req_id = 8c3fa905-fbb1-4851-a40f-868daacd431b response_code = 200 uri = /crucible/0/regions
242024-07-19T23:03:14.906ZINFOcrucible-agent (datafile): region c0fef4c4-820b-482b-8d7c-07512e2caaae state: Requested
252024-07-19T23:03:14.909ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
262024-07-19T23:03:14.909ZINFOcrucible-agent (dropshot): request completed latency_us = 2611 local_addr = [fd00:1122:3344:101::18]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:38083 req_id = 3ab8f747-adf5-4c0f-8d96-4d2267cc5d15 response_code = 200 uri = /crucible/0/regions
272024-07-19T23:03:14.918ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_5e9012da-baa1-440f-b7c2-376e7117cb03/crucible/regions/c0fef4c4-820b-482b-8d7c-07512e2caaae
282024-07-19T23:03:14.918ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_5e9012da-baa1-440f-b7c2-376e7117cb03/crucible/regions/c0fef4c4-820b-482b-8d7c-07512e2caaae
292024-07-19T23:03:14.982ZINFOcrucible-agent (worker): creating region Region { id: RegionId("c0fef4c4-820b-482b-8d7c-07512e2caaae"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19001, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/c0fef4c4-820b-482b-8d7c-07512e2caaae" region = c0fef4c4-820b-482b-8d7c-07512e2caaae
302024-07-19T23:03:14.982ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "c0fef4c4-820b-482b-8d7c-07512e2caaae" "--data" "/data/regions/c0fef4c4-820b-482b-8d7c-07512e2caaae" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = c0fef4c4-820b-482b-8d7c-07512e2caaae
312024-07-19T23:03:15.061ZINFOcrucible-agent (worker): region files created ok region = c0fef4c4-820b-482b-8d7c-07512e2caaae
322024-07-19T23:03:15.061ZINFOcrucible-agent (datafile): region c0fef4c4-820b-482b-8d7c-07512e2caaae state: Requested -> Created
332024-07-19T23:03:15.061ZINFOcrucible-agent (worker): applying SMF actions post create...
342024-07-19T23:03:15.062ZINFOcrucible-agent (worker): existing directory value /dev/null does not match /data/regions/6341e33d-f9c6-4d7d-a567-8f58cb653e37
352024-07-19T23:03:15.063ZINFOcrucible-agent (worker): existing port value 0 does not match 19000
362024-07-19T23:03:15.063ZINFOcrucible-agent (worker): existing address value 0.0.0.0 does not match fd00:1122:3344:101::18
372024-07-19T23:03:15.063ZINFOcrucible-agent (worker): reconfig required
382024-07-19T23:03:15.063ZINFOcrucible-agent (worker): using existing config property group
392024-07-19T23:03:15.063ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-6341e33d-f9c6-4d7d-a567-8f58cb653e37
402024-07-19T23:03:15.063ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/6341e33d-f9c6-4d7d-a567-8f58cb653e37
412024-07-19T23:03:15.063ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
422024-07-19T23:03:15.063ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::18
432024-07-19T23:03:15.063ZINFOcrucible-agent (worker): commit
442024-07-19T23:03:15.068ZINFOcrucible-agent (worker): ok!
452024-07-19T23:03:15.069ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-c0fef4c4-820b-482b-8d7c-07512e2caaae
462024-07-19T23:03:15.072ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-c0fef4c4-820b-482b-8d7c-07512e2caaae
472024-07-19T23:03:15.081ZINFOcrucible-agent (worker): creating config property group
482024-07-19T23:03:15.085ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-c0fef4c4-820b-482b-8d7c-07512e2caaae
492024-07-19T23:03:15.085ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/c0fef4c4-820b-482b-8d7c-07512e2caaae
502024-07-19T23:03:15.085ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
512024-07-19T23:03:15.085ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::18
522024-07-19T23:03:15.085ZINFOcrucible-agent (worker): commit
532024-07-19T23:03:15.090ZINFOcrucible-agent (worker): ok!
542024-07-19T23:03:15.095ZINFOcrucible-agent (worker): SMF ok!
552024-07-19T23:03:15.135ZINFOcrucible-agent (dropshot): request completed latency_us = 73 local_addr = [fd00:1122:3344:101::18]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:38083 req_id = 6e75e142-4fc2-432c-8397-0b09fb74af12 response_code = 200 uri = /crucible/0/regions
562024-07-19T23:03:39.672ZINFOcrucible-agent (dropshot): request completed latency_us = 79 local_addr = [fd00:1122:3344:101::18]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:38083 req_id = a04b4c03-a51d-48ca-ae8b-bbd14b55efc9 response_code = 200 uri = /crucible/0/regions/c0fef4c4-820b-482b-8d7c-07512e2caaae
572024-07-19T23:03:39.674ZINFOcrucible-agent (datafile): path is "/data/regions/c0fef4c4-820b-482b-8d7c-07512e2caaae"
582024-07-19T23:03:39.685ZINFOcrucible-agent (datafile): dataset is oxp_5e9012da-baa1-440f-b7c2-376e7117cb03/crucible/regions/c0fef4c4-820b-482b-8d7c-07512e2caaae
592024-07-19T23:03:39.694ZINFOcrucible-agent (datafile): region c0fef4c4-820b-482b-8d7c-07512e2caaae state: Created -> Tombstoned
602024-07-19T23:03:39.695ZINFOcrucible-agent (worker): applying SMF actions before removal...
612024-07-19T23:03:39.695ZINFOcrucible-agent (dropshot): request completed latency_us = 20244 local_addr = [fd00:1122:3344:101::18]:32345 method = DELETE remote_addr = [fd00:1122:3344:101::a]:38083 req_id = 8eda5de2-1c43-4c18-b8ea-e035ce66e582 response_code = 204 uri = /crucible/0/regions/c0fef4c4-820b-482b-8d7c-07512e2caaae
622024-07-19T23:03:39.696ZINFOcrucible-agent (worker): disabling downstairs instance: downstairs-c0fef4c4-820b-482b-8d7c-07512e2caaae (instance states: (Some(Online), None))
632024-07-19T23:03:39.697ZINFOcrucible-agent (dropshot): request completed latency_us = 24 local_addr = [fd00:1122:3344:101::18]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:38083 req_id = 9004dfaa-e786-4dba-a516-93f20ced38a5 response_code = 200 uri = /crucible/0/regions/c0fef4c4-820b-482b-8d7c-07512e2caaae
642024-07-19T23:03:39.701ZINFOcrucible-agent (worker): existing directory value /dev/null does not match /data/regions/6341e33d-f9c6-4d7d-a567-8f58cb653e37
652024-07-19T23:03:39.701ZINFOcrucible-agent (worker): existing port value 0 does not match 19000
662024-07-19T23:03:39.701ZINFOcrucible-agent (worker): existing address value 0.0.0.0 does not match fd00:1122:3344:101::18
672024-07-19T23:03:39.701ZINFOcrucible-agent (worker): reconfig required
682024-07-19T23:03:39.701ZINFOcrucible-agent (worker): using existing config property group
692024-07-19T23:03:39.701ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-6341e33d-f9c6-4d7d-a567-8f58cb653e37
702024-07-19T23:03:39.701ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/6341e33d-f9c6-4d7d-a567-8f58cb653e37
712024-07-19T23:03:39.702ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
722024-07-19T23:03:39.702ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::18
732024-07-19T23:03:39.702ZINFOcrucible-agent (worker): commit
742024-07-19T23:03:39.706ZINFOcrucible-agent (worker): ok!
752024-07-19T23:03:39.707ZINFOcrucible-agent (worker): SMF ok!
762024-07-19T23:03:39.717ZINFOcrucible-agent (worker): deleting zfs dataset "oxp_5e9012da-baa1-440f-b7c2-376e7117cb03/crucible/regions/c0fef4c4-820b-482b-8d7c-07512e2caaae" region = c0fef4c4-820b-482b-8d7c-07512e2caaae
772024-07-19T23:03:39.767ZINFOcrucible-agent (datafile): region c0fef4c4-820b-482b-8d7c-07512e2caaae state: Tombstoned -> Destroyed
782024-07-19T23:03:39.815ZINFOcrucible-agent (dropshot): request completed latency_us = 60 local_addr = [fd00:1122:3344:101::18]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:38083 req_id = 46a13483-a23d-4ad2-8bbf-490628f49871 response_code = 200 uri = /crucible/0/regions/c0fef4c4-820b-482b-8d7c-07512e2caaae