/input/rbuild/work Create region with ES:16384 EC:640 BS:4096 Creating region directory at: /var/tmp/dsc Creating dsc directory at: /tmp/dsc Downstairs region 0 created at /var/tmp/dsc/8810 in 0.9494018 Downstairs region 1 created at /var/tmp/dsc/8820 in 0.9561012 Downstairs region 2 created at /var/tmp/dsc/8830 in 0.7439662 Region set with 3 regions was created Using existing output directory "/tmp/dsc" start ds: 8810 start ds: 8820 start access at:127.0.0.1:9998 Starting downstairs at port 8810 Starting downstairs at port 8820 start ds: 8830 Make output file at "/tmp/dsc/downstairs-8820.txt" Make output file at "/tmp/dsc/downstairs-8810.txt" Starting downstairs at port 8830 Make output file at "/tmp/dsc/downstairs-8830.txt" [8820][1] reports Starting [8810][0] reports Starting [8830][2] reports Starting Downstairs /var/tmp/dsc/8810 port 8810 PID:Some(1153) [8810][0] reports Running Downstairs /var/tmp/dsc/8820 port 8820 PID:Some(1154) [8820][1] reports Running Downstairs /var/tmp/dsc/8830 port 8830 PID:Some(1155) [8830][2] reports Running Feb 28 22:13:30.650 INFO listening, local_addr: 127.0.0.1:9998 Control access at:127.0.0.1:9998 1153 1154 1155 Feb 28 22:13:35.688 INFO accepted connection, remote_addr: 127.0.0.1:37756, local_addr: 127.0.0.1:9998 Main task has work to do, go find it got dsc DisableRestartAll disable restart on all: 3 Feb 28 22:13:35.689 INFO request completed, latency_us[8810] Disable keep_running :[8820] Disable keep_running 164[8830] Disable keep_running , response_code: 204, uri: /disablerestart/all, method: POST, req_id: 4c885b78-3a22-4366-94ee-14b2d33ac16f, remote_addr: 127.0.0.1:37756, local_addr: 127.0.0.1:9998 A new loop begins at February 28, 2024 at 10:13:35 PM UTC /var/tmp/bins/crutest perf -g 1 -t 127.0.0.1:8810 -t 127.0.0.1:8820 -t 127.0.0.1:8830 -c 30000 -q --write-loops 2 --read-loops 2 --perf-out /tmp/perf-ES-16384-EC-640.csv {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:13:35.71175462Z","hostname":"ip-10-150-1-6.us-west-2.compute.internal","pid":1163} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"80b6f734175c2bc8e222c8d6f7875e53cbd5c9ca\",\n git_commit_timestamp: \"2024-02-28T21:51:33.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.75.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"82e1608dfa6e0b5569232559e3d385fea5a93112\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: false,\n opt_level: 3,\n}","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:13:35.711975747Z","hostname":"ip-10-150-1-6.us-west-2.compute.internal","pid":1163} {"msg":"Upstairs <-> Downstairs Message Version: 5","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:13:35.711987157Z","hostname":"ip-10-150-1-6.us-west-2.compute.internal","pid"Crucible runtime is spawned :1163} {"msg":"Crucible 218fbf7f-5eda-49b1-96b5-b7c75729b2ee has session id: d80ea41b-2c37-49ee-a643-a76f75b5e972","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:13:35.711999206Z","hostname":"ip-10-150-1-6.us-west-2.compute.internal","pid":1163,"session_id":"d80ea41b-2c37-49ee-a643-a76f75b5e972"} {"msg":"Upstairs opts: Upstairs UUID: 218fbf7f-5eda-49b1-96b5-b7c75729b2ee, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:13:35.712010035Z","hostname":"ip-10-150-1-6.us-west-2.compute.internal","pid":1163,"session_id":"d80ea41b-2c37-49ee-a643-a76f75b5e972"} {"msg":"Crucible stats registered with UUID: 218fbf7f-5eda-49b1-96b5-b7c75729b2ee","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:13:35.712019485Z","hostname":"ip-10-150-1-6.us-west-2.compute.internal","pid":1163,"session_id":"d80ea41b-2c37-49ee-a643-a76f75b5e972"} {"msg":"The guest has requested activation with gen:1","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:13:35.712027324Z","hostname":"ip-10-150-1-6.us-west-2.compute.internal","pid":1163} {"msg":"218fbf7f-5eda-49b1-96b5-b7c75729b2ee active request set","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:13:35.712282089Z","hostname":"ip-10-150-1-6.us-west-2.compute.internal","pid":1163,"session_id":"d80ea41b-2c37-49ee-a643-a76f75b5e972"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:13:35.712297799Z","hostname":"ip-10-150-1-6.us-west-2.compute.internal","pid":1163,"client":"0","":"downstairs","session_id":"d80ea41b-2c37-49ee-a643-a76f75b5e972"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:13:35.712303748Z","hostname":"ip-10-150-1-6.us-west-2.compute.internal","pid":1163,"client":"0","":"downstairs","session_id":"d80ea41b-2c37-49ee-a643-a76f75b5e972"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:13:35.712308938Z","hostname":"ip-10-150-1-6.us-west-2.compute.internal","pid":1163,"client":"1","":"downstairs","session_id":"d80ea41b-2c37-49ee-a643-a76f75b5e972"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:13:35.712314418Z","hostname":"ip-10-150-1-6.us-west-2.compute.internal","pid":1163,"client":"1","":"downstairs","session_id":"d80ea41b-2c37-49ee-a643-a76f75b5e972"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:13:35.712319407Z","hostname":"ip-10-150-1-6.us-west-2.compute.internal","pid":1163,"client":"2","":"downstairs","session_id":"d80ea41b-2c37-49ee-a643-a76f75b5e972"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:13:35.712327077Z","hostname":"ip-10-150-1-6.us-west-2.compute.internal","pid":1163,"client":"2","":"downstairs","session_id":"d80ea41b-2c37-49ee-a643-a76f75b5e972"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:13:35.712348866Z","hostname":"ip-10-150-1-6.us-west-2.compute.internal","pid":1163,"":"io task","client":"0","":"downstairs","session_id":"d80ea41b-2c37-49ee-a643-a76f75b5e972"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:13:35.712367364Z","hostname":"ip-10-150-1-6.us-west-2.compute.internal","pid":1163,"":"io task","client":"1","":"downstairs","session_id":"d80ea41b-2c37-49ee-a643-a76f75b5e972"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:13:35.712378484Z","hostname":"ip-10-150-1-6.us-west-2.compute.internal","pid":1163,"":"io task","client":"2","":"downstairs","session_id":"d80ea41b-2c37-49ee-a643-a76f75b5e972"} {"msg":"ds_connection connected","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:13:35.712518686Z","hostname":"ip-10-150-1-6.us-west-2.compute.internal","pid":1163,"":"io task","client":"2","":"downstairs","session_id":"d80ea41b-2c37-49ee-a643-a76f75b5e972"} {"msg":"ds_connection connected","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:13:35.712546344Z","hostname":"ip-10-150-1-6.us-west-2.compute.internal","pid":1163,"":"io task","client":"1","":"downstairs","session_id":"d80ea41b-2c37-49ee-a643-a76f75b5e972"} {"msg":"ds_connection connected","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:13:35.712556884Z","hostname":"ip-10-150-1-6.us-west-2.compute.internal","pid":1163,"":"io task","client":"0","":"downstairs","session_id":"d80ea41b-2c37-49ee-a643-a76f75b5e972"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:13:35.7131317Z","hostname":"ip-10-150-1-6.us-west-2.compute.internal","pid":1163,"client":"2","":"downstairs","session_id":"d80ea41b-2c37-49ee-a643-a76f75b5e972"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:13:35.713148479Z","hostname":"ip-10-150-1-6.us-west-2.compute.internal","pid":1163,"client":"2","":"downstairs","session_id":"d80ea41b-2c37-49ee-a643-a76f75b5e972"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:13:35.713156189Z","hostname":"ip-10-150-1-6.us-west-2.compute.internal","pid":1163,"client":"0","":"downstairs","session_id":"d80ea41b-2c37-49ee-a643-a76f75b5e972"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:13:35.713163109Z","hostname":"ip-10-150-1-6.us-west-2.compute.internal","pid":1163,"client":"0","":"downstairs","session_id":"d80ea41b-2c37-49ee-a643-a76f75b5e972"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:13:35.713167378Z","hostname":"ip-10-150-1-6.us-west-2.compute.internal","pid":1163,"client":"1","":"downstairs","session_id":"d80ea41b-2c37-49ee-a643-a76f75b5e972"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:13:35.713171508Z","hostname":"ip-10-150-1-6.us-west-2.compute.internal","pid":1163,"client":"1","":"downstairs","session_id":"d80ea41b-2c37-49ee-a643-a76f75b5e972"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:13:35.713390105Z","hostname":"ip-10-150-1-6.us-west-2.compute.internal","pid":1163,"client":"2","":"downstairs","session_id":"d80ea41b-2c37-49ee-a643-a76f75b5e972"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:13:35.713407404Z","hostname":"ip-10-150-1-6.us-west-2.compute.internal","pid":1163,"client":"0","":"downstairs","session_id":"d80ea41b-2c37-49ee-a643-a76f75b5e972"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:13:35.713415104Z","hostname":"ip-10-150-1-6.us-west-2.compute.internal","pid":1163,"client":"1","":"downstairs","session_id":"d80ea41b-2c37-49ee-a643-a76f75b5e972"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:13:35.715686723Z","hostname":"ip-10-150-1-6.us-west-2.compute.internal","pid":1163,"client":"1","":"downstairs","session_id":"d80ea41b-2c37-49ee-a643-a76f75b5e972"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:13:35.715708302Z","hostname":"ip-10-150-1-6.us-west-2.compute.internal","pid":1163,"client":"1","":"downstairs","session_id":"d80ea41b-2c37-49ee-a643-a76f75b5e972"} {"msg":"Waiting for true more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:13:35.715714581Z","hostname":"ip-10-150-1-6.us-west-2.compute.internal","pid":1163,"session_id":"d80ea41b-2c37-49ee-a643-a76f75b5e972"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:13:35.716552543Z","hostname":"ip-10-150-1-6.us-west-2.compute.internal","pid":1163,"client":"0","":"downstairs","session_id":"d80ea41b-2c37-49ee-a643-a76f75b5e972"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:13:35.716569342Z","hostname":"ip-10-150-1-6.us-west-2.compute.internal","pid":1163,"client":"0","":"downstairs","session_id":"d80ea41b-2c37-49ee-a643-a76f75b5e972"} {"msg":"Waiting for true more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:13:35.716574261Z","hostname":"ip-10-150-1-6.us-west-2.compute.internal","pid":1163,"session_id":"d80ea41b-2c37-49ee-a643-a76f75b5e972"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:13:35.716585251Z","hostname":"ip-10-150-1-6.us-west-2.compute.internal","pid":1163,"client":"2","":"downstairs","session_id":"d80ea41b-2c37-49ee-a643-a76f75b5e972"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:13:35.7165909Z","hostname":"ip-10-150-1-6.us-west-2.compute.internal","pid":1163,"client":"2","":"downstairs","session_id":"d80ea41b-2c37-49ee-a643-a76f75b5e972"} {"msg":"[0]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:13:35.71659566Z","hostname":"ip-10-150-1-6.us-west-2.compute.internal","pid":1163,"":"downstairs","":"downstairs","session_id":"d80ea41b-2c37-49ee-a643-a76f75b5e972"} {"msg":"[0]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:13:35.71660046Z","hostname":"ip-10-150-1-6.us-west-2.compute.internal","pid":1163,"":"downstairs","":"downstairs","session_id":"d80ea41b-2c37-49ee-a643-a76f75b5e972"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:13:35.71660589Z","hostname":"ip-10-150-1-6.us-west-2.compute.internal","pid":1163,"":"downstairs","":"downstairs","session_id":"d80ea41b-2c37-49ee-a643-a76f75b5e972"} {"msg":"[1]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:13:35.716611279Z","hostname":"ip-10-150-1-6.us-west-2.compute.internal","pid":1163,"":"downstairs","":"downstairs","session_id":"d80ea41b-2c37-49ee-a643-a76f75b5e972"} {"msg":"[1]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:13:35.716615559Z","hostname":"ip-10-150-1-6.us-west-2.compute.internal","pid":1163,"":"downstairs","":"downstairs","session_id":"d80ea41b-2c37-49ee-a643-a76f75b5e972"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:13:35.716619919Z","hostname":"ip-10-150-1-6.us-west-2.compute.internal","pid":1163,"":"downstairs","":"downstairs","session_id":"d80ea41b-2c37-49ee-a643-a76f75b5e972"} {"msg":"[2]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:13:35.716624109Z","hostname":"ip-10-150-1-6.us-west-2.compute.internal","pid":1163,"":"downstairs","":"downstairs","session_id":"d80ea41b-2c37-49ee-a643-a76f75b5e972"} {"msg":"[2]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:13:35.716628768Z","hostname":"ip-10-150-1-6.us-west-2.compute.internal","pid":1163,"":"downstairs","":"downstairs","session_id":"d80ea41b-2c37-49ee-a643-a76f75b5e972"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:13:35.716633408Z","hostname":"ip-10-150-1-6.us-west-2.compute.internal","pid":1163,"":"downstairs","":"downstairs","session_id":"d80ea41b-2c37-49ee-a643-a76f75b5e972"} {"msg":"Max found gen is 1","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:13:35.716637818Z","hostname":"ip-10-150-1-6.us-west-2.compute.internal","pid":1163,"":"downstairs","":"downstairs","session_id":"d80ea41b-2c37-49ee-a643-a76f75b5e972"} {"msg":"Generation requested: 1 >= found:1","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:13:35.716644027Z","hostname":"ip-10-150-1-6.us-west-2.compute.internal","pid":1163,"":"downstairs","":"downstairs","session_id":"d80ea41b-2c37-49ee-a643-a76f75b5e972"} {"msg":"Next flush: 1","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:13:35.716652037Z","hostname":"ip-10-150-1-6.us-west-2.compute.internal","pid":1163,"":"downstairs","":"downstairs","session_id":"d80ea41b-2c37-49ee-a643-a76f75b5e972"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:13:35.716662326Z","hostname":"ip-10-150-1-6.us-west-2.compute.internal","pid":1163,"":"downstairs","":"downstairs","session_id":"d80ea41b-2c37-49ee-a643-a76f75b5e972"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:13:35.716677435Z","hostname":"ip-10-150-1-6.us-west-2.compute.internal","pid":1163,"session_id":"d80ea41b-2c37-49ee-a643-a76f75b5e972"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"Wait for a query_work_queue command to finish before sending IO 2024-02-28T22:13:35.716684285Z","hostname":"ip-10-150-1-6.us-west-2.compute.internal","pid":1163,"client":"0","":"downstairs","session_id":"d80ea41b-2c37-49ee-a643-a76f75b5e972"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:13:35.716695214Z","hostname":"ip-10-150-1-6.us-west-2.compute.internal","pid":1163,"client":"1","":"downstairs","session_id":"d80ea41b-2c37-49ee-a643-a76f75b5e972"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:13:35.716702484Z","hostname":"ip-10-150-1-6.us-west-2.compute.internal","pid":1163,"client":"2","":"downstairs","session_id":"d80ea41b-2c37-49ee-a643-a76f75b5e972"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:13:35.716731462Z","hostname":"ip-10-150-1-6.us-west-2.compute.internal","pid":1163,"session_id":"d80ea41b-2c37-49ee-a643-a76f75b5e972"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:13:35.716738932Z","hostname":"ip-10-150-1-6.us-west-2.compute.internal","pid":1163,"session_id":"d80ea41b-2c37-49ee-a643-a76f75b5e972"} {"msg":"218fbf7f-5eda-49b1-96b5-b7c75729b2ee is now active with session: d80ea41b-2c37-49ee-a643-a76f75b5e972","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:13:35.716745611Z","hostname":"ip-10-150-1-6.us-west-2.compute.internal","pid":1163,"session_id":"d80ea41b-2c37-49ee-a643-a76f75b5e972"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:13:35.716750811Z","hostname":"ip-10-150-1-6.us-west-2.compute.internal","pid":1163,"session_id":"d80ea41b-2c37-49ee-a643-a76f75b5e972"} {"msg":"The guest has finished waiting for activation with:1","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:13:35.716756081Z","hostname":"ip-10-150-1-6.us-west-2.compute.internal","pid":1163} Region: es:16384 ec:640 bs:4096 ts:42949672960 tb:10485760 max_io:256 or 1048576 Perf test TEST SECONDS COUNT DPTH IOPS MEAN P95 P99 MAX ES EC rwrites 17.56 30000 1 1708.83 0.00058 0.00138 0.00177 0.05111 16384 640 rwrites 7.80 30000 1 3848.34 0.00026 0.00062 0.00140 0.05703 16384 640 rreads 5.59 30000 1 5362.43 0.00019 0.00023 0.00029 0.01332 16384 640 rreads 5.53 30000 1 5429.56 0.00019 0.00023 0.00028 0.01332 16384 640 Perf test completed, stop all downstairs Feb 28 22:14:17.622 INFO accepted connection, remote_addr: 127.0.0.1:45775, local_addr: 127.0.0.1:9998 Main task has work to do, go find it got dsc Shutdown Shutdown Feb 28 22:14:17.623Shut it down INFO request completed, latency_us: 183[8810] Disable keep_running , [8810] Got stop action so:false kr:false response_code[8820] Disable keep_running :[8820] Got stop action so:false kr:false 204, uri: /shutdown, method: POST, req_id: fe864a66-674a-48ac-9751-7c2e21905549, remote_addr: 127.0.0.1:45775, local_addr: 127.0.0.1:9998 [8830] Disable keep_running [8830] Got stop action so:false kr:false [8830] Exited with: ExitStatus(unix_wait_status(9)) [8830][2] reports Exit [8810] Exited with: ExitStatus(unix_wait_status(9)) [8820] Exited with: ExitStatus(unix_wait_status(9)) [8810][0] reports Exit [8820][1] reports Exit Create region with ES:16384 EC:1280 BS:4096 Removing existing dsc directory "/tmp/dsc" Removing existing region "/var/tmp/dsc" Creating region directory at: /var/tmp/dsc Creating dsc directory at: /tmp/dsc Downstairs region 0 created at /var/tmp/dsc/8810 in 1.1390874 Downstairs region 1 created at /var/tmp/dsc/8820 in 1.1960447 Downstairs region 2 created at /var/tmp/dsc/8830 in 1.1679354 Region set with 3 regions was created Using existing output directory "/tmp/dsc" start ds: 8810 start ds: 8820 start ds: 8830 start access at:127.0.0.1:9998 Starting downstairs at port 8810 Make output file at "/tmp/dsc/downstairs-8810.txt" Starting downstairs at port 8820 Make output file at "/tmp/dsc/downstairs-8820.txt" [8810][0] reports Starting Starting downstairs at port 8830 [8820][1] reports Starting Make output file at "/tmp/dsc/downstairs-8830.txt" [8830][2] reports Starting Downstairs /var/tmp/dsc/8820 port 8820 PID:Some(1172) [8820][1] reports Running Downstairs /var/tmp/dsc/8830 port 8830 PID:Some(1173) [8830][2] reports Running Downstairs /var/tmp/dsc/8810 port 8810 PID:Some(1174) [8810][0] reports Running Feb 28 22:14:22.189 INFO listening, local_addr: 127.0.0.1:9998 Control access at:127.0.0.1:9998 1174 1172 1173 Feb 28 22:14:27.226 INFO accepted connection, remote_addr: 127.0.0.1:61919, local_addr: 127.0.0.1:9998 Main task has work to do, go find it got dsc DisableRestartAll disable restart on all: 3 Feb 28 22:14:27.227 INFO request completed, latency_us: 117, response_code: 204, uri: /disablerestart/all[8810] Disable keep_running , [8820] Disable keep_running method: POST, req_id: 92c0d925-fce9-485d-b2a5-9c73788c4a3c, remote_addr: 127.0.0[8830] Disable keep_running .1:61919, local_addr: 127.0.0.1:9998 A new loop begins at February 28, 2024 at 10:14:27 PM UTC /var/tmp/bins/crutest perf -g 1 -t 127.0.0.1:8810 -t 127.0.0.1:8820 -t 127.0.0.1:8830 -c 30000 -q --write-loops 2 --read-loops 2 --perf-out /tmp/perf-ES-16384-EC-1280.csv {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:14:27.247914576Z","hostname":"ip-10-150-1-6.us-west-2.compute.internal","pid":1182} {Crucible runtime is spawned "msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"80b6f734175c2bc8e222c8d6f7875e53cbd5c9ca\",\n git_commit_timestamp: \"2024-02-28T21:51:33.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.75.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"82e1608dfa6e0b5569232559e3d385fea5a93112\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: false,\n opt_level: 3,\n}","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:14:27.248081706Z","hostname":"ip-10-150-1-6.us-west-2.compute.internal","pid":1182} {"msg":"Upstairs <-> Downstairs Message Version: 5","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:14:27.248093785Z","hostname":"ip-10-150-1-6.us-west-2.compute.internal","pid":1182} {"msg":"Crucible aa343aac-bd0c-453b-ac90-3a44d872949e has session id: b68e5376-5f14-46b9-8914-7483734fa9b6","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:14:27.248102145Z","hostname":"ip-10-150-1-6.us-west-2.compute.internal","pid":1182,"session_id":"b68e5376-5f14-46b9-8914-7483734fa9b6"} {"msg":"Upstairs opts: Upstairs UUID: aa343aac-bd0c-453b-ac90-3a44d872949e, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:14:27.248127353Z","hostname":"ip-10-150-1-6.us-west-2.compute.internal","pid":1182,"session_id":"b68e5376-5f14-46b9-8914-7483734fa9b6"} {"msg":"Crucible stats registered with UUID: aa343aac-bd0c-453b-ac90-3a44d872949e","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:14:27.248136333Z","hostname":"ip-10-150-1-6.us-west-2.compute.internal","pid":1182,"session_id":"b68e5376-5f14-46b9-8914-7483734fa9b6"} {"msg":"The guest has requested activation with gen:1","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:14:27.248143972Z","hostname":"ip-10-150-1-6.us-west-2.compute.internal","pid":1182} {"msg":"aa343aac-bd0c-453b-ac90-3a44d872949e active request set","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:14:27.248153312Z","hostname":"ip-10-150-1-6.us-west-2.compute.internal","pid":1182,"session_id":"b68e5376-5f14-46b9-8914-7483734fa9b6"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:14:27.248161021Z","hostname":"ip-10-150-1-6.us-west-2.compute.internal","pid":1182,"client":"0","":"downstairs","session_id":"b68e5376-5f14-46b9-8914-7483734fa9b6"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:14:27.24818512Z","hostname":"ip-10-150-1-6.us-west-2.compute.internal","pid":1182,"client":"0","":"downstairs","session_id":"b68e5376-5f14-46b9-8914-7483734fa9b6"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:14:27.248196569Z","hostname":"ip-10-150-1-6.us-west-2.compute.internal","pid":1182,"client":"1","":"downstairs","session_id":"b68e5376-5f14-46b9-8914-7483734fa9b6"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:14:27.248205639Z","hostname":"ip-10-150-1-6.us-west-2.compute.internal","pid":1182,"client":"1","":"downstairs","session_id":"b68e5376-5f14-46b9-8914-7483734fa9b6"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:14:27.248217168Z","hostname":"ip-10-150-1-6.us-west-2.compute.internal","pid":1182,"client":"2","":"downstairs","session_id":"b68e5376-5f14-46b9-8914-7483734fa9b6"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:14:27.248228577Z","hostname":"ip-10-150-1-6.us-west-2.compute.internal","pid":1182,"client":"2","":"downstairs","session_id":"b68e5376-5f14-46b9-8914-7483734fa9b6"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:14:27.248237937Z","hostname":"ip-10-150-1-6.us-west-2.compute.internal","pid":1182,"":"io task","client":"2","":"downstairs","session_id":"b68e5376-5f14-46b9-8914-7483734fa9b6"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:14:27.248262606Z","hostname":"ip-10-150-1-6.us-west-2.compute.internal","pid":1182,"":"io task","client":"0","":"downstairs","session_id":"b68e5376-5f14-46b9-8914-7483734fa9b6"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:14:27.248272775Z","hostname":"ip-10-150-1-6.us-west-2.compute.internal","pid":1182,"":"io task","client":"1","":"downstairs","session_id":"b68e5376-5f14-46b9-8914-7483734fa9b6"} {"msg":"ds_connection connected","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:14:27.248333751Z","hostname":"ip-10-150-1-6.us-west-2.compute.internal","pid":1182,"":"io task","client":"2","":"downstairs","session_id":"b68e5376-5f14-46b9-8914-7483734fa9b6"} {"msg":"ds_connection connected","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:14:27.248343991Z","hostname":"ip-10-150-1-6.us-west-2.compute.internal","pid":1182,"":"io task","client":"0","":"downstairs","session_id":"b68e5376-5f14-46b9-8914-7483734fa9b6"} {"msg":"ds_connection connected","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:14:27.248426816Z","hostname":"ip-10-150-1-6.us-west-2.compute.internal","pid":1182,"":"io task","client":"1","":"downstairs","session_id":"b68e5376-5f14-46b9-8914-7483734fa9b6"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:14:27.248797875Z","hostname":"ip-10-150-1-6.us-west-2.compute.internal","pid":1182,"client":"2","":"downstairs","session_id":"b68e5376-5f14-46b9-8914-7483734fa9b6"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:14:27.248818353Z","hostname":"ip-10-150-1-6.us-west-2.compute.internal","pid":1182,"client":"2","":"downstairs","session_id":"b68e5376-5f14-46b9-8914-7483734fa9b6"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:14:27.248825333Z","hostname":"ip-10-150-1-6.us-west-2.compute.internal","pid":1182,"client":"1","":"downstairs","session_id":"b68e5376-5f14-46b9-8914-7483734fa9b6"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:14:27.248831633Z","hostname":"ip-10-150-1-6.us-west-2.compute.internal","pid":1182,"client":"1","":"downstairs","session_id":"b68e5376-5f14-46b9-8914-7483734fa9b6"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:14:27.248838302Z","hostname":"ip-10-150-1-6.us-west-2.compute.internal","pid":1182,"client":"0","":"downstairs","session_id":"b68e5376-5f14-46b9-8914-7483734fa9b6"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:14:27.248844182Z","hostname":"ip-10-150-1-6.us-west-2.compute.internal","pid":1182,"client":"0","":"downstairs","session_id":"b68e5376-5f14-46b9-8914-7483734fa9b6"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:14:27.249026791Z","hostname":"ip-10-150-1-6.us-west-2.compute.internal","pid":1182,"client":"1","":"downstairs","session_id":"b68e5376-5f14-46b9-8914-7483734fa9b6"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:14:27.24904483Z","hostname":"ip-10-150-1-6.us-west-2.compute.internal","pid":1182,"client":"2","":"downstairs","session_id":"b68e5376-5f14-46b9-8914-7483734fa9b6"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:14:27.24905031Z","hostname":"ip-10-150-1-6.us-west-2.compute.internal","pid":1182,"client":"0","":"downstairs","session_id":"b68e5376-5f14-46b9-8914-7483734fa9b6"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:14:27.254944579Z","hostname":"ip-10-150-1-6.us-west-2.compute.internal","pid":1182,"client":"0","":"downstairs","session_id":"b68e5376-5f14-46b9-8914-7483734fa9b6"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:14:27.254963718Z","hostname":"ip-10-150-1-6.us-west-2.compute.internal","pid":1182,"client":"0","":"downstairs","session_id":"b68e5376-5f14-46b9-8914-7483734fa9b6"} {"msg":"Waiting for true more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:14:27.254968928Z","hostname":"ip-10-150-1-6.us-west-2.compute.internal","pid":1182,"session_id":"b68e5376-5f14-46b9-8914-7483734fa9b6"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:14:27.256043146Z","hostname":"ip-10-150-1-6.us-west-2.compute.internal","pid":1182,"client":"2","":"downstairs","session_id":"b68e5376-5f14-46b9-8914-7483734fa9b6"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:14:27.256058635Z","hostname":"ip-10-150-1-6.us-west-2.compute.internal","pid":1182,"client":"2","":"downstairs","session_id":"b68e5376-5f14-46b9-8914-7483734fa9b6"} {"msg":"Waiting for true more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:14:27.256064564Z","hostname":"ip-10-150-1-6.us-west-2.compute.internal","pid":1182,"session_id":"b68e5376-5f14-46b9-8914-7483734fa9b6"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:14:27.256121921Z","hostname":"ip-10-150-1-6.us-west-2.compute.internal","pid":1182,"client":"1","":"downstairs","session_id":"b68e5376-5f14-46b9-8914-7483734fa9b6"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:14:27.25613498Z","hostname":"ip-10-150-1-6.us-west-2.compute.internal","pid":1182,"client":"1","":"downstairs","session_id":"b68e5376-5f14-46b9-8914-7483734fa9b6"} {"msg":"[0]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:14:27.25613995Z","hostname":"ip-10-150-1-6.us-west-2.compute.internal","pid":1182,"":"downstairs","":"downstairs","session_id":"b68e5376-5f14-46b9-8914-7483734fa9b6"} {"msg":"[0]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:14:27.25614454Z","hostname":"ip-10-150-1-6.us-west-2.compute.internal","pid":1182,"":"downstairs","":"downstairs","session_id":"b68e5376-5f14-46b9-8914-7483734fa9b6"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:14:27.25614902Z","hostname":"ip-10-150-1-6.us-west-2.compute.internal","pid":1182,"":"downstairs","":"downstairs","session_id":"b68e5376-5f14-46b9-8914-7483734fa9b6"} {"msg":"[1]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:14:27.256154549Z","hostname":"ip-10-150-1-6.us-west-2.compute.internal","pid":1182,"":"downstairs","":"downstairs","session_id":"b68e5376-5f14-46b9-8914-7483734fa9b6"} {"msg":"[1]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:14:27.256159829Z","hostname":"ip-10-150-1-6.us-west-2.compute.internal","pid":1182,"":"downstairs","":"downstairs","session_id":"b68e5376-5f14-46b9-8914-7483734fa9b6"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:14:27.256164379Z","hostname":"ip-10-150-1-6.us-west-2.compute.internal","pid":1182,"":"downstairs","":"downstairs","session_id":"b68e5376-5f14-46b9-8914-7483734fa9b6"} {"msg":"[2]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:14:27.256168809Z","hostname":"ip-10-150-1-6.us-west-2.compute.internal","pid":1182,"":"downstairs","":"downstairs","session_id":"b68e5376-5f14-46b9-8914-7483734fa9b6"} {"msg":"[2]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:14:27.256173418Z","hostname":"ip-10-150-1-6.us-west-2.compute.internal","pid":1182,"":"downstairs","":"downstairs","session_id":"b68e5376-5f14-46b9-8914-7483734fa9b6"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:14:27.256177708Z","hostname":"ip-10-150-1-6.us-west-2.compute.internal","pid":1182,"":"downstairs","":"downstairs","session_id":"b68e5376-5f14-46b9-8914-7483734fa9b6"} {"msg":"Max found gen is 1","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:14:27.256182158Z","hostname":"ip-10-150-1-6.us-west-2.compute.internal","pid":1182,"":"downstairs","":"downstairs","session_id":"b68e5376-5f14-46b9-8914-7483734fa9b6"} {"msg":"Generation requested: 1 >= found:1","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:14:27.256187457Z","hostname":"ip-10-150-1-6.us-west-2.compute.internal","pid":1182,"":"downstairs","":"downstairs","session_id":"b68e5376-5f14-46b9-8914-7483734fa9b6"} {"msg":"Next flush: 1","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:14:27.256191877Z","hostname":"ip-10-150-1-6.us-west-2.compute.internal","pid":1182,"":"downstairs","":"downstairs","session_id":"b68e5376-5f14-46b9-8914-7483734fa9b6"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:14:27.256197207Z","hostname":"ip-10-150-1-6.us-west-2.compute.internal","pid":1182,"":"downstairs","":"downstairs","session_id":"b68e5376-5f14-46b9-8914-7483734fa9b6"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:14:27.256203036Z","hostname":"ip-10-150-1-6.us-west-2.compute.internal","pid":1182,"session_id":"b68e5376-5f14-46b9-8914-7483734fa9b6"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:14:27.256210176Z","hostname":"ip-10-150-1-6.us-west-2.compute.internal","pid":1182,"client":"0","":"downstairs","session_id":"b68e5376-5f14-46b9-8914-7483734fa9b6"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:14:27.256217756Z","hostname":"ip-10-150-1-6.us-west-2.compute.internal","pid":1182,"client":"1","":"downstairs","session_id":"b68e5376-5f14-46b9-8914-7483734fa9b6"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:14:27.256224845Z","hostname":"Wait for a query_work_queue command to finish before sending IO ip-10-150-1-6.us-west-2.compute.internal","pid":1182,"client":"2","":"downstairs","session_id":"b68e5376-5f14-46b9-8914-7483734fa9b6"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:14:27.256235645Z","hostname":"ip-10-150-1-6.us-west-2.compute.internal","pid":1182,"session_id":"b68e5376-5f14-46b9-8914-7483734fa9b6"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:14:27.256241884Z","hostname":"ip-10-150-1-6.us-west-2.compute.internal","pid":1182,"session_id":"b68e5376-5f14-46b9-8914-7483734fa9b6"} {"msg":"aa343aac-bd0c-453b-ac90-3a44d872949e is now active with session: b68e5376-5f14-46b9-8914-7483734fa9b6","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:14:27.256257204Z","hostname":"ip-10-150-1-6.us-west-2.compute.internal","pid":1182,"session_id":"b68e5376-5f14-46b9-8914-7483734fa9b6"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:14:27.256261323Z","hostname":"ip-10-150-1-6.us-west-2.compute.internal","pid":1182,"session_id":"b68e5376-5f14-46b9-8914-7483734fa9b6"} {"msg":"The guest has finished waiting for activation with:1","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:14:27.256265183Z","hostname":"ip-10-150-1-6.us-west-2.compute.internal","pid":1182} Region: es:16384 ec:1280 bs:4096 ts:85899345920 tb:20971520 max_io:256 or 1048576 Perf test TEST SECONDS COUNT DPTH IOPS MEAN P95 P99 MAX ES EC rwrites 8.46 30000 1 3544.48 0.00028 0.00064 0.00135 0.07161 16384 1280 rwrites 8.33 30000 1 3602.67 0.00028 0.00062 0.00114 0.07011 16384 1280 rreads 5.47 30000 1 5481.78 0.00018 0.00022 0.00027 0.01316 16384 1280 rreads 5.60 30000 1 5357.81 0.00018 0.00023 0.00027 0.01323 16384 1280 Perf test completed, stop all downstairs Feb 28 22:15:03.875 INFO accepted connection, remote_addr: 127.0.0.1:51779, local_addr: 127.0.0.1:9998 Main task has work to do, go find it got dsc Shutdown Shutdown Feb 28 22:15:03.876Shut it down INFO request completed[8820] Disable keep_running , [8820] Got stop action so:false kr:false latency_us[8830] Disable keep_running : 62, response_code: 204, uri: /shutdown, method: POST, req_id: 2a35d02d-4453-4e1d-a94a-381915ef02bc, remote_addr: 127.0.0.1:51779[8810] Disable keep_running , [8810] Got stop action so:false kr:false local_addr[8830] Got stop action so:false kr:false : 127.0.0.1:9998 [8810] Exited with: ExitStatus(unix_wait_status(9)) [8820] Exited with: ExitStatus(unix_wait_status(9)) [8830] Exited with: ExitStatus(unix_wait_status(9)) [8810][0] reports Exit [8820][1] reports Exit [8830][2] reports Exit TEST SECONDS COUNT DPTH IOPS MEAN P95 P99 MAX ES EC rwrites 17.56 30000 1 1708.83 0.00058 0.00138 0.00177 0.05111 16384 640 rwrites 7.80 30000 1 3848.34 0.00026 0.00062 0.00140 0.05703 16384 640 rwrites 8.46 30000 1 3544.48 0.00028 0.00064 0.00135 0.07161 16384 1280 rwrites 8.33 30000 1 3602.67 0.00028 0.00062 0.00114 0.07011 16384 1280 TEST SECONDS COUNT DPTH IOPS MEAN P95 P99 MAX ES EC rreads 5.59 30000 1 5362.43 0.00019 0.00023 0.00029 0.01332 16384 640 rreads 5.53 30000 1 5429.56 0.00019 0.00023 0.00028 0.01332 16384 640 rreads 5.47 30000 1 5481.78 0.00018 0.00022 0.00027 0.01316 16384 1280 rreads 5.60 30000 1 5357.81 0.00018 0.00023 0.00027 0.01323 16384 1280 Perf test finished on February 28, 2024 at 10:15:05 PM UTC