/input/rbuild/work Create region with ES:16384 EC:640 BS:4096 Creating region directory at: /var/tmp/dsc Creating dsc directory at: /tmp/dsc Downstairs region 0 created at /var/tmp/dsc/8810 in 0.98304206 Downstairs region 1 created at /var/tmp/dsc/8820 in 0.7370447 Downstairs region 2 created at /var/tmp/dsc/8830 in 0.6961141 Region set with 3 regions was created Using existing output directory "/tmp/dsc" start ds: 8810 start ds: 8820 start ds: 8830 Starting downstairs at port 8830 Starting downstairs at port 8810 start access at:127.0.0.1:9998 Starting downstairs at port 8820 [8830][2] reports Starting [8820][1] reports Starting Make output file at "/tmp/dsc/downstairs-8810.txt" Make output file at "/tmp/dsc/downstairs-8820.txt" Make output file at "/tmp/dsc/downstairs-8830.txt" [8810][0] reports Starting Downstairs /var/tmp/dsc/8810 port 8810 PID:Some(1192) [8810][0] reports Running Downstairs /var/tmp/dsc/8820 port 8820 PID:Some(1193) [8820][1] reports Running Downstairs /var/tmp/dsc/8830 port 8830 PID:Some(1194) [8830][2] reports Running Aug 08 15:43:38.506 INFO listening, local_addr: 127.0.0.1:9998 Control access at:127.0.0.1:9998 1192 1193 1194 Aug 08 15:43:43.544 INFO accepted connection, remote_addr: 127.0.0.1:46120, local_addr: 127.0.0.1:9998 Main task has work to do, go find it got dsc DisableRestartAll disable restart on all: 3 Aug 08 15:43:43.545 INFO request completed, latency_us: 164, response_code[8810] Disable keep_running :[8820] Disable keep_running 204, uri: /disablerestart/all, method: POST, req_id: 2b059f36-589c-4d6f-8b39-58a3cbc88bd9, remote_addr:[8830] Disable keep_running 127.0.0.1:46120, local_addr: 127.0.0.1:9998 A new loop begins at August 8, 2024 at 03:43:43 PM UTC /var/tmp/bins/crutest perf -g 1 -t 127.0.0.1:8810 -t 127.0.0.1:8820 -t 127.0.0.1:8830 -c 30000 -q --write-loops 2 --read-loops 2 --perf-out /tmp/perf-ES-16384-EC-640.csv {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-08-08T15:43:43.568426565Z","hostname":"w-01J4SAFH2PQH83DF5HAX003QMF","pid":1205} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"361584843ef1ddcec7585daa2b2efecd4dcd6a18\",\n git_commit_timestamp: \"2024-08-08T15:17:39.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: false,\n opt_level: 3,\n}","v":0,"name":"crucible","level":30,"time":"2024-08-08T15:43:43.568607176Z","hostname":"w-01J4SAFH2PQH83DF5HAX003QMF","pid":1205} {"msg":"Upstairs <-> Downstairs Message Version: 11","v":0,"name":"crucible","level":30,"time":"2024-08-08T15:43:43.568614116Z","hostname":"w-01J4SAFH2PQH83DF5HAX003QMF","pid":1205} {"msg":"Crucible 72905c1b-6eef-476a-9bae-90719f88986a has session id: 8acb7884-acb1-4570-ae49-fa2833b5fc21","v":0,"name":"crucible","level":30,"time":"2024-08-08T15:43:43.56902295Z","hostname":"w-01J4SAFH2PQH83DF5HAX003QMF","pid":1205,"session_id":"8acb7884-acb1-4570-ae49-fa2833b5fc21"} {"msg":"Upstairs opts: Upstairs UUID: 72905c1b-6eef-476a-9bae-90719f88986a, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-08-08T15:43:43.569110395Z","hostname":"w-01J4SAFH2PQH83DF5HAX003QMF","pid":1205,"session_id":"8acb7884-acb1-4570-ae49-fa2833b5fc21"} {"msg":"Crucible stats registered with UUID: 72905c1b-6eef-476a-9bae-90719f88986a","v":0,"name":"crucible","level":30,"time":"2024-08-08T15:43:43.569124196Z","hostname":"w-01J4SAFH2PQH83DF5HAX003QMF","pid":1205,"session_id":"8acb7884-acb1-4570-ae49-fa2833b5fc21"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:1","v":0,"name":"crucible","level":30,"time":"2024-08-08T15:43:43.606200945Z","hostname":"w-01J4SAFH2PQH83DF5HAX003QMF","pid":1205} {"msg":"72905c1b-6eef-476a-9bae-90719f88986a active request set","v":0,"name":"crucible","level":30,"time":"2024-08-08T15:43:43.606261788Z","hostname":"w-01J4SAFH2PQH83DF5HAX003QMF","pid":1205,"session_id":"8acb7884-acb1-4570-ae49-fa2833b5fc21"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-08-08T15:43:43.606282389Z","hostname":"w-01J4SAFH2PQH83DF5HAX003QMF","pid":1205,"client":"0","":"downstairs","session_id":"8acb7884-acb1-4570-ae49-fa2833b5fc21"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-08-08T15:43:43.606300471Z","hostname":"w-01J4SAFH2PQH83DF5HAX003QMF","pid":1205,"client":"0","":"downstairs","session_id":"8acb7884-acb1-4570-ae49-fa2833b5fc21"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-08-08T15:43:43.606319702Z","hostname":"w-01J4SAFH2PQH83DF5HAX003QMF","pid":1205,"client":"1","":"downstairs","session_id":"8acb7884-acb1-4570-ae49-fa2833b5fc21"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-08-08T15:43:43.606329332Z","hostname":"w-01J4SAFH2PQH83DF5HAX003QMF","pid":1205,"client":"1","":"downstairs","session_id":"8acb7884-acb1-4570-ae49-fa2833b5fc21"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-08-08T15:43:43.606348793Z","hostname":"w-01J4SAFH2PQH83DF5HAX003QMF","pid":1205,"client":"2","":"downstairs","session_id":"8acb7884-acb1-4570-ae49-fa2833b5fc21"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-08-08T15:43:43.606359594Z","hostname":"w-01J4SAFH2PQH83DF5HAX003QMF","pid":1205,"client":"2","":"downstairs","session_id":"8acb7884-acb1-4570-ae49-fa2833b5fc21"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-08-08T15:43:43.606517974Z","hostname":"w-01J4SAFH2PQH83DF5HAX003QMF","pid":1205,"":"io task","client":"1","":"downstairs","session_id":"8acb7884-acb1-4570-ae49-fa2833b5fc21"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-08-08T15:43:43.606538915Z","hostname":"w-01J4SAFH2PQH83DF5HAX003QMF","pid":1205,"":"io task","client":"2","":"downstairs","session_id":"8acb7884-acb1-4570-ae49-fa2833b5fc21"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-08-08T15:43:43.606547555Z","hostname":"w-01J4SAFH2PQH83DF5HAX003QMF","pid":1205,"":"io task","client":"0","":"downstairs","session_id":"8acb7884-acb1-4570-ae49-fa2833b5fc21"} {"msg":"ds_connection connected from Ok(127.0.0.1:36422)","v":0,"name":"crucible","level":30,"time":"2024-08-08T15:43:43.606691424Z","hostname":"w-01J4SAFH2PQH83DF5HAX003QMF","pid":1205,"":"io task","client":"1","":"downstairs","session_id":"8acb7884-acb1-4570-ae49-fa2833b5fc21"} {"msg":"ds_connection connected from Ok(127.0.0.1:53402)","v":0,"name":"crucible","level":30,"time":"2024-08-08T15:43:43.606703995Z","hostname":"w-01J4SAFH2PQH83DF5HAX003QMF","pid":1205,"":"io task","client":"2","":"downstairs","session_id":"8acb7884-acb1-4570-ae49-fa2833b5fc21"} {"msg":"ds_connection connected from Ok(127.0.0.1:53574)","v":0,"name":"crucible","level":30,"time":"2024-08-08T15:43:43.606711505Z","hostname":"w-01J4SAFH2PQH83DF5HAX003QMF","pid":1205,"":"io task","client":"0","":"downstairs","session_id":"8acb7884-acb1-4570-ae49-fa2833b5fc21"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-08-08T15:43:43.607573837Z","hostname":"w-01J4SAFH2PQH83DF5HAX003QMF","pid":1205,"client":"0","":"downstairs","session_id":"8acb7884-acb1-4570-ae49-fa2833b5fc21"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-08-08T15:43:43.607590827Z","hostname":"w-01J4SAFH2PQH83DF5HAX003QMF","pid":1205,"client":"0","":"downstairs","session_id":"8acb7884-acb1-4570-ae49-fa2833b5fc21"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-08-08T15:43:43.607595968Z","hostname":"w-01J4SAFH2PQH83DF5HAX003QMF","pid":1205,"client":"1","":"downstairs","session_id":"8acb7884-acb1-4570-ae49-fa2833b5fc21"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-08-08T15:43:43.607603578Z","hostname":"w-01J4SAFH2PQH83DF5HAX003QMF","pid":1205,"client":"1","":"downstairs","session_id":"8acb7884-acb1-4570-ae49-fa2833b5fc21"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-08-08T15:43:43.607610499Z","hostname":"w-01J4SAFH2PQH83DF5HAX003QMF","pid":1205,"client":"2","":"downstairs","session_id":"8acb7884-acb1-4570-ae49-fa2833b5fc21"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-08-08T15:43:43.60763178Z","hostname":"w-01J4SAFH2PQH83DF5HAX003QMF","pid":1205,"client":"2","":"downstairs","session_id":"8acb7884-acb1-4570-ae49-fa2833b5fc21"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-08-08T15:43:43.632597767Z","hostname":"w-01J4SAFH2PQH83DF5HAX003QMF","pid":1205,"client":"2","":"downstairs","session_id":"8acb7884-acb1-4570-ae49-fa2833b5fc21"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-08-08T15:43:43.635755885Z","hostname":"w-01J4SAFH2PQH83DF5HAX003QMF","pid":1205,"client":"1","":"downstairs","session_id":"8acb7884-acb1-4570-ae49-fa2833b5fc21"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-08-08T15:43:43.636296867Z","hostname":"w-01J4SAFH2PQH83DF5HAX003QMF","pid":1205,"client":"2","":"downstairs","session_id":"8acb7884-acb1-4570-ae49-fa2833b5fc21"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-08-08T15:43:43.636313998Z","hostname":"w-01J4SAFH2PQH83DF5HAX003QMF","pid":1205,"client":"2","":"downstairs","session_id":"8acb7884-acb1-4570-ae49-fa2833b5fc21"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-08-08T15:43:43.636319709Z","hostname":"w-01J4SAFH2PQH83DF5HAX003QMF","pid":1205,"session_id":"8acb7884-acb1-4570-ae49-fa2833b5fc21"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-08-08T15:43:43.637307357Z","hostname":"w-01J4SAFH2PQH83DF5HAX003QMF","pid":1205,"client":"0","":"downstairs","session_id":"8acb7884-acb1-4570-ae49-fa2833b5fc21"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-08-08T15:43:43.638238133Z","hostname":"w-01J4SAFH2PQH83DF5HAX003QMF","pid":1205,"client":"1","":"downstairs","session_id":"8acb7884-acb1-4570-ae49-fa2833b5fc21"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-08-08T15:43:43.638252304Z","hostname":"w-01J4SAFH2PQH83DF5HAX003QMF","pid":1205,"client":"1","":"downstairs","session_id":"8acb7884-acb1-4570-ae49-fa2833b5fc21"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-08-08T15:43:43.638257714Z","hostname":"w-01J4SAFH2PQH83DF5HAX003QMF","pid":1205,"session_id":"8acb7884-acb1-4570-ae49-fa2833b5fc21"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-08-08T15:43:43.639502298Z","hostname":"w-01J4SAFH2PQH83DF5HAX003QMF","pid":1205,"client":"0","":"downstairs","session_id":"8acb7884-acb1-4570-ae49-fa2833b5fc21"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-08-08T15:43:43.639515939Z","hostname":"w-01J4SAFH2PQH83DF5HAX003QMF","pid":1205,"client":"0","":"downstairs","session_id":"8acb7884-acb1-4570-ae49-fa2833b5fc21"} {"msg":"[0]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-08-08T15:43:43.639520909Z","hostname":"w-01J4SAFH2PQH83DF5HAX003QMF","pid":1205,"":"downstairs","":"downstairs","session_id":"8acb7884-acb1-4570-ae49-fa2833b5fc21"} {"msg":"[0]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-08-08T15:43:43.63952626Z","hostname":"w-01J4SAFH2PQH83DF5HAX003QMF","pid":1205,"":"downstairs","":"downstairs","session_id":"8acb7884-acb1-4570-ae49-fa2833b5fc21"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-08-08T15:43:43.63953147Z","hostname":"w-01J4SAFH2PQH83DF5HAX003QMF","pid":1205,"":"downstairs","":"downstairs","session_id":"8acb7884-acb1-4570-ae49-fa2833b5fc21"} {"msg":"[1]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-08-08T15:43:43.63953609Z","hostname":"w-01J4SAFH2PQH83DF5HAX003QMF","pid":1205,"":"downstairs","":"downstairs","session_id":"8acb7884-acb1-4570-ae49-fa2833b5fc21"} {"msg":"[1]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-08-08T15:43:43.63954122Z","hostname":"w-01J4SAFH2PQH83DF5HAX003QMF","pid":1205,"":"downstairs","":"downstairs","session_id":"8acb7884-acb1-4570-ae49-fa2833b5fc21"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-08-08T15:43:43.639546081Z","hostname":"w-01J4SAFH2PQH83DF5HAX003QMF","pid":1205,"":"downstairs","":"downstairs","session_id":"8acb7884-acb1-4570-ae49-fa2833b5fc21"} {"msg":"[2]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-08-08T15:43:43.639550861Z","hostname":"w-01J4SAFH2PQH83DF5HAX003QMF","pid":1205,"":"downstairs","":"downstairs","session_id":"8acb7884-acb1-4570-ae49-fa2833b5fc21"} {"msg":"[2]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-08-08T15:43:43.639555731Z","hostname":"w-01J4SAFH2PQH83DF5HAX003QMF","pid":1205,"":"downstairs","":"downstairs","session_id":"8acb7884-acb1-4570-ae49-fa2833b5fc21"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-08-08T15:43:43.639560262Z","hostname":"w-01J4SAFH2PQH83DF5HAX003QMF","pid":1205,"":"downstairs","":"downstairs","session_id":"8acb7884-acb1-4570-ae49-fa2833b5fc21"} {"msg":"Max found gen is 1","v":0,"name":"crucible","level":30,"time":"2024-08-08T15:43:43.639564722Z","hostname":"w-01J4SAFH2PQH83DF5HAX003QMF","pid":1205,"":"downstairs","":"downstairs","session_id":"8acb7884-acb1-4570-ae49-fa2833b5fc21"} {"msg":"Generation requested: 1 >= found:1","v":0,"name":"crucible","level":30,"time":"2024-08-08T15:43:43.639569862Z","hostname":"w-01J4SAFH2PQH83DF5HAX003QMF","pid":1205,"":"downstairs","":"downstairs","session_id":"8acb7884-acb1-4570-ae49-fa2833b5fc21"} {"msg":"Next flush: 1","v":0,"name":"crucible","level":30,"time":"2024-08-08T15:43:43.639574312Z","hostname":"w-01J4SAFH2PQH83DF5HAX003QMF","pid":1205,"":"downstairs","":"downstairs","session_id":"8acb7884-acb1-4570-ae49-fa2833b5fc21"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-08-08T15:43:43.639578773Z","hostname":"w-01J4SAFH2PQH83DF5HAX003QMF","pid":1205,"":"downstairs","":"downstairs","session_id":"8acb7884-acb1-4570-ae49-fa2833b5fc21"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-08-08T15:43:43.639583863Z","hostname":"w-01J4SAFH2PQH83DF5HAX003QMF","pid":1205,"session_id":"8acb7884-acb1-4570-ae49-fa2833b5fc21"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-08-08T15:43:43.639587743Z","hostname":"w-01J4SAFH2PQH83DF5HAX003QMF","pid":1205,"client":"0","":"downstairs","session_id":"8acb7884-acb1-4570-ae49-fa2833b5fc21"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-08-08T15:43:43.639592253Z","hostname":"w-01J4SAFH2PQH83DF5HAX003QMF","pid":1205,"client":"1","":"downstairs","session_id":"8acb7884-acb1-4570-ae49-fa2833b5fc21"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-08-08T15:43:43.639597404Z","hostname":"w-01J4SAFH2PQH83DF5HAX003QMF","pid":1205,"client":"2","":"downstairs","session_id":"8acb7884-acb1-4570-ae49-fa2833b5fc21"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-08-08T15:43:43.639601844Z","hostname":"w-01J4SAFH2PQH83DF5HAX003QMF","pid":1205,"session_id":"8acb7884-acb1-4570-ae49-fa2833b5fc21"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-08-08T15:43:43.639605684Z","hostname":"w-01J4SAFH2PQH83DF5HAX003QMF","pid":1205,"session_id":"8acb7884-acb1-4570-ae49-fa2833b5fc21"} {"msg":"72905c1b-6eef-476a-9bae-90719f88986a is now active with session: 8acb7884-acb1-4570-ae49-fa2833b5fc21","v":0,"name":"crucible","level":30,"time":"2024-08-08T15:43:43.639610345Z","hostname":"w-01J4SAFH2PQH83DF5HAX003QMF","pid":1205,"session_id":"8acb7884-acb1-4570-ae49-fa2833b5fc21"} Wait for a query_work_queue command to finish before sending IO {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-08-08T15:43:43.639617695Z","hostname":"w-01J4SAFH2PQH83DF5HAX003QMF","pid":1205,"session_id":"8acb7884-acb1-4570-ae49-fa2833b5fc21"} {"msg":"The guest has finished waiting for activation with:1","v":0,"name":"crucible","level":30,"time":"2024-08-08T15:43:43.639622125Z","hostname":"w-01J4SAFH2PQH83DF5HAX003QMF","pid":1205} Region: es:16384 ec:640 bs:4096 ts:42949672960 tb:10485760 max_io:256 or 1048576 Perf test TEST SECONDS COUNT DPTH IOPS MEAN P95 P99 MAX ES EC rwrites 4.96 30000 1 6048.16 0.00017 0.00040 0.00130 0.05595 16384 640 rwrites 5.10 30000 1 5882.22 0.00017 0.00046 0.00140 0.02869 16384 640 rreads 5.58 30000 1 5372.36 0.00019 0.00026 0.00030 0.01439 16384 640 rreads 5.53 30000 1 5423.08 0.00018 0.00025 0.00029 0.01439 16384 640 Perf test completed, stop all downstairs Aug 08 15:44:07.385 INFO accepted connection, remote_addr: 127.0.0.1:37254, local_addr: 127.0.0.1:9998 Main task has work to do, go find it got dsc Shutdown Shutdown Shut it down Aug 08 15:44:07.386 INFO request completed, latency_us: 74, response_code:[8820] Disable keep_running [8820] Got stop action so:false kr:false 204[8830] Disable keep_running , [8830] Got stop action so:false kr:false uri[8810] Disable keep_running :[8810] Got stop action so:false kr:false /shutdown, method: POST, req_id: 490ed4a3-13fe-4af2-8dcc-dd8443b55ac9, remote_addr: 127.0.0.1:37254, local_addr: 127.0.0.1:9998 [8830] Exited with: ExitStatus(unix_wait_status(9)) [8820] Exited with: ExitStatus(unix_wait_status(9)) [8810] Exited with: ExitStatus(unix_wait_status(9)) [8830][2] reports Exit [8820][1] reports Exit [8810][0] reports Exit Create region with ES:16384 EC:1280 BS:4096 Removing existing dsc directory "/tmp/dsc" Removing existing region "/var/tmp/dsc" Creating region directory at: /var/tmp/dsc Creating dsc directory at: /tmp/dsc Downstairs region 0 created at /var/tmp/dsc/8810 in 1.3806494 Downstairs region 1 created at /var/tmp/dsc/8820 in 1.4211277 Downstairs region 2 created at /var/tmp/dsc/8830 in 1.3678087 Region set with 3 regions was created Using existing output directory "/tmp/dsc" start ds: 8810 start ds: 8820 start ds: 8830 start access at:127.0.0.1:9998 Starting downstairs at port 8810 Make output file at "/tmp/dsc/downstairs-8810.txt" Starting downstairs at port 8820 Make output file at "/tmp/dsc/downstairs-8820.txt" [8810][0] reports Starting [8820][1] reports Starting Starting downstairs at port 8830 Make output file at "/tmp/dsc/downstairs-8830.txt" [8830][2] reports Starting Downstairs /var/tmp/dsc/8820 port 8820 PID:Some(1220) [8820][1] reports Running Downstairs /var/tmp/dsc/8830 port 8830 PID:Some(1221) [8830][2] reports Running Downstairs /var/tmp/dsc/8810 port 8810 PID:Some(1222) [8810][0] reports Running Aug 08 15:44:12.624 INFO listening, local_addr: 127.0.0.1:9998 Control access at:127.0.0.1:9998 1220 1221 1222 Aug 08 15:44:17.665 INFO accepted connection, remote_addr: 127.0.0.1:38247, local_addr: 127.0.0.1:9998 Main task has work to do, go find it got dsc DisableRestartAll disable restart on all: 3 Aug 08 15:44:17.665 INFO request completed, latency_us: 126, response_code: 204, uri: /disablerestart/all, method: POST, req_id: 70e2472b-8fac-4753-98db-82c7b3ca446e, remote_addr[8820] Disable keep_running :[8810] Disable keep_running 127.0.0.1:38247, local_addr: 127.0.0[8830] Disable keep_running .1:9998 A new loop begins at August 8, 2024 at 03:44:17 PM UTC /var/tmp/bins/crutest perf -g 1 -t 127.0.0.1:8810 -t 127.0.0.1:8820 -t 127.0.0.1:8830 -c 30000 -q --write-loops 2 --read-loops 2 --perf-out /tmp/perf-ES-16384-EC-1280.csv {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-08-08T15:44:17.68649688Z","hostname":"w-01J4SAFH2PQH83DF5HAX003QMF","pid":1233} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"361584843ef1ddcec7585daa2b2efecd4dcd6a18\",\n git_commit_timestamp: \"2024-08-08T15:17:39.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: false,\n opt_level: 3,\n}","v":0,"name":"crucible","level":30,"time":"2024-08-08T15:44:17.686653929Z","hostname":"w-01J4SAFH2PQH83DF5HAX003QMF","pid":1233} {"msg":"Upstairs <-> Downstairs Message Version: 11","v":0,"name":"crucible","level":30,"time":"2024-08-08T15:44:17.68666251Z","hostname":"w-01J4SAFH2PQH83DF5HAX003QMF","pid":1233} {"msg":"Crucible f35ebbf2-b3d9-4557-a7c7-8535e4257519 has session id: 57d19fac-856e-453c-969e-e9969759be51","v":0,"name":"crucible","level":30,"time":"2024-08-08T15:44:17.686974028Z","hostname":"w-01J4SAFH2PQH83DF5HAX003QMF","pid":1233,"session_id":"57d19fac-856e-453c-969e-e9969759be51"} {"msg":"Upstairs opts: Upstairs UUID: f35ebbf2-b3d9-4557-a7c7-8535e4257519, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-08-08T15:44:17.687013391Z","hostname":"w-01J4SAFH2PQH83DF5HAX003QMF","pid":1233,"session_id":"57d19fac-856e-453c-969e-e9969759be51"} {"msg":"Crucible stats registered with UUID: f35ebbf2-b3d9-4557-a7c7-8535e4257519","v":0,"name":"crucible","level":30,"time":"2024-08-08T15:44:17.687025521Z","hostname":"w-01J4SAFH2PQH83DF5HAX003QMF","pid":1233,"session_id":"57d19fac-856e-453c-969e-e9969759be51"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:1","v":0,"name":"crucible","level":30,"time":"2024-08-08T15:44:17.72562664Z","hostname":"w-01J4SAFH2PQH83DF5HAX003QMF","pid":1233} {"msg":"f35ebbf2-b3d9-4557-a7c7-8535e4257519 active request set","v":0,"name":"crucible","level":30,"time":"2024-08-08T15:44:17.725684774Z","hostname":"w-01J4SAFH2PQH83DF5HAX003QMF","pid":1233,"session_id":"57d19fac-856e-453c-969e-e9969759be51"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-08-08T15:44:17.725701655Z","hostname":"w-01J4SAFH2PQH83DF5HAX003QMF","pid":1233,"client":"0","":"downstairs","session_id":"57d19fac-856e-453c-969e-e9969759be51"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-08-08T15:44:17.725716026Z","hostname":"w-01J4SAFH2PQH83DF5HAX003QMF","pid":1233,"client":"0","":"downstairs","session_id":"57d19fac-856e-453c-969e-e9969759be51"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-08-08T15:44:17.725724486Z","hostname":"w-01J4SAFH2PQH83DF5HAX003QMF","pid":1233,"client":"1","":"downstairs","session_id":"57d19fac-856e-453c-969e-e9969759be51"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-08-08T15:44:17.725732687Z","hostname":"w-01J4SAFH2PQH83DF5HAX003QMF","pid":1233,"client":"1","":"downstairs","session_id":"57d19fac-856e-453c-969e-e9969759be51"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-08-08T15:44:17.725739587Z","hostname":"w-01J4SAFH2PQH83DF5HAX003QMF","pid":1233,"client":"2","":"downstairs","session_id":"57d19fac-856e-453c-969e-e9969759be51"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-08-08T15:44:17.725747147Z","hostname":"w-01J4SAFH2PQH83DF5HAX003QMF","pid":1233,"client":"2","":"downstairs","session_id":"57d19fac-856e-453c-969e-e9969759be51"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-08-08T15:44:17.725837723Z","hostname":"w-01J4SAFH2PQH83DF5HAX003QMF","pid":1233,"":"io task","client":"0","":"downstairs","session_id":"57d19fac-856e-453c-969e-e9969759be51"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-08-08T15:44:17.725865805Z","hostname":"w-01J4SAFH2PQH83DF5HAX003QMF","pid":1233,"":"io task","client":"1","":"downstairs","session_id":"57d19fac-856e-453c-969e-e9969759be51"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-08-08T15:44:17.725878415Z","hostname":"w-01J4SAFH2PQH83DF5HAX003QMF","pid":1233,"":"io task","client":"2","":"downstairs","session_id":"57d19fac-856e-453c-969e-e9969759be51"} {"msg":"ds_connection connected from Ok(127.0.0.1:56593)","v":0,"name":"crucible","level":30,"time":"2024-08-08T15:44:17.725984282Z","hostname":"w-01J4SAFH2PQH83DF5HAX003QMF","pid":1233,"":"io task","client":"1","":"downstairs","session_id":"57d19fac-856e-453c-969e-e9969759be51"} {"msg":"ds_connection connected from Ok(127.0.0.1:46345)","v":0,"name":"crucible","level":30,"time":"2024-08-08T15:44:17.726016874Z","hostname":"w-01J4SAFH2PQH83DF5HAX003QMF","pid":1233,"":"io task","client":"2","":"downstairs","session_id":"57d19fac-856e-453c-969e-e9969759be51"} {"msg":"ds_connection connected from Ok(127.0.0.1:53021)","v":0,"name":"crucible","level":30,"time":"2024-08-08T15:44:17.726032475Z","hostname":"w-01J4SAFH2PQH83DF5HAX003QMF","pid":1233,"":"io task","client":"0","":"downstairs","session_id":"57d19fac-856e-453c-969e-e9969759be51"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-08-08T15:44:17.726537255Z","hostname":"w-01J4SAFH2PQH83DF5HAX003QMF","pid":1233,"client":"1","":"downstairs","session_id":"57d19fac-856e-453c-969e-e9969759be51"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-08-08T15:44:17.726557126Z","hostname":"w-01J4SAFH2PQH83DF5HAX003QMF","pid":1233,"client":"1","":"downstairs","session_id":"57d19fac-856e-453c-969e-e9969759be51"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-08-08T15:44:17.726565826Z","hostname":"w-01J4SAFH2PQH83DF5HAX003QMF","pid":1233,"client":"2","":"downstairs","session_id":"57d19fac-856e-453c-969e-e9969759be51"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-08-08T15:44:17.726577067Z","hostname":"w-01J4SAFH2PQH83DF5HAX003QMF","pid":1233,"client":"2","":"downstairs","session_id":"57d19fac-856e-453c-969e-e9969759be51"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-08-08T15:44:17.72661867Z","hostname":"w-01J4SAFH2PQH83DF5HAX003QMF","pid":1233,"client":"0","":"downstairs","session_id":"57d19fac-856e-453c-969e-e9969759be51"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-08-08T15:44:17.726641521Z","hostname":"w-01J4SAFH2PQH83DF5HAX003QMF","pid":1233,"client":"0","":"downstairs","session_id":"57d19fac-856e-453c-969e-e9969759be51"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-08-08T15:44:17.759451705Z","hostname":"w-01J4SAFH2PQH83DF5HAX003QMF","pid":1233,"client":"1","":"downstairs","session_id":"57d19fac-856e-453c-969e-e9969759be51"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-08-08T15:44:17.764102952Z","hostname":"w-01J4SAFH2PQH83DF5HAX003QMF","pid":1233,"client":"0","":"downstairs","session_id":"57d19fac-856e-453c-969e-e9969759be51"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-08-08T15:44:17.766657594Z","hostname":"w-01J4SAFH2PQH83DF5HAX003QMF","pid":1233,"client":"2","":"downstairs","session_id":"57d19fac-856e-453c-969e-e9969759be51"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-08-08T15:44:17.767337825Z","hostname":"w-01J4SAFH2PQH83DF5HAX003QMF","pid":1233,"client":"1","":"downstairs","session_id":"57d19fac-856e-453c-969e-e9969759be51"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-08-08T15:44:17.767356586Z","hostname":"w-01J4SAFH2PQH83DF5HAX003QMF","pid":1233,"client":"1","":"downstairs","session_id":"57d19fac-856e-453c-969e-e9969759be51"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-08-08T15:44:17.767363556Z","hostname":"w-01J4SAFH2PQH83DF5HAX003QMF","pid":1233,"session_id":"57d19fac-856e-453c-969e-e9969759be51"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-08-08T15:44:17.769742238Z","hostname":"w-01J4SAFH2PQH83DF5HAX003QMF","pid":1233,"client":"0","":"downstairs","session_id":"57d19fac-856e-453c-969e-e9969759be51"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-08-08T15:44:17.769761329Z","hostname":"w-01J4SAFH2PQH83DF5HAX003QMF","pid":1233,"client":"0","":"downstairs","session_id":"57d19fac-856e-453c-969e-e9969759be51"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-08-08T15:44:17.769768379Z","hostname":"w-01J4SAFH2PQH83DF5HAX003QMF","pid":1233,"session_id":"57d19fac-856e-453c-969e-e9969759be51"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-08-08T15:44:17.772095538Z","hostname":"w-01J4SAFH2PQH83DF5HAX003QMF","pid":1233,"client":"2","":"downstairs","session_id":"57d19fac-856e-453c-969e-e9969759be51"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-08-08T15:44:17.772111209Z","hostname":"w-01J4SAFH2PQH83DF5HAX003QMF","pid":1233,"client":"2","":"downstairs","session_id":"57d19fac-856e-453c-969e-e9969759be51"} {"msg":"[0]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-08-08T15:44:17.772117699Z","hostname":"w-01J4SAFH2PQH83DF5HAX003QMF","pid":1233,"":"downstairs","":"downstairs","session_id":"57d19fac-856e-453c-969e-e9969759be51"} {"msg":"[0]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-08-08T15:44:17.77212412Z","hostname":"w-01J4SAFH2PQH83DF5HAX003QMF","pid":1233,"":"downstairs","":"downstairs","session_id":"57d19fac-856e-453c-969e-e9969759be51"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-08-08T15:44:17.77213076Z","hostname":"w-01J4SAFH2PQH83DF5HAX003QMF","pid":1233,"":"downstairs","":"downstairs","session_id":"57d19fac-856e-453c-969e-e9969759be51"} {"msg":"[1]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-08-08T15:44:17.772136741Z","hostname":"w-01J4SAFH2PQH83DF5HAX003QMF","pid":1233,"":"downstairs","":"downstairs","session_id":"57d19fac-856e-453c-969e-e9969759be51"} {"msg":"[1]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-08-08T15:44:17.772143511Z","hostname":"w-01J4SAFH2PQH83DF5HAX003QMF","pid":1233,"":"downstairs","":"downstairs","session_id":"57d19fac-856e-453c-969e-e9969759be51"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-08-08T15:44:17.772149651Z","hostname":"w-01J4SAFH2PQH83DF5HAX003QMF","pid":1233,"":"downstairs","":"downstairs","session_id":"57d19fac-856e-453c-969e-e9969759be51"} {"msg":"[2]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-08-08T15:44:17.772155631Z","hostname":"w-01J4SAFH2PQH83DF5HAX003QMF","pid":1233,"":"downstairs","":"downstairs","session_id":"57d19fac-856e-453c-969e-e9969759be51"} {"msg":"[2]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-08-08T15:44:17.772161652Z","hostname":"w-01J4SAFH2PQH83DF5HAX003QMF","pid":1233,"":"downstairs","":"downstairs","session_id":"57d19fac-856e-453c-969e-e9969759be51"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-08-08T15:44:17.772168162Z","hostname":"w-01J4SAFH2PQH83DF5HAX003QMF","pid":1233,"":"downstairs","":"downstairs","session_id":"57d19fac-856e-453c-969e-e9969759be51"} {"msg":"Max found gen is 1","v":0,"name":"crucible","level":30,"time":"2024-08-08T15:44:17.772173993Z","hostname":"w-01J4SAFH2PQH83DF5HAX003QMF","pid":1233,"":"downstairs","":"downstairs","session_id":"57d19fac-856e-453c-969e-e9969759be51"} {"msg":"Generation requested: 1 >= found:1","v":0,"name":"crucible","level":30,"time":"2024-08-08T15:44:17.772180163Z","hostname":"w-01J4SAFH2PQH83DF5HAX003QMF","pid":1233,"":"downstairs","":"downstairs","session_id":"57d19fac-856e-453c-969e-e9969759be51"} {"msg":"Next flush: 1","v":0,"name":"crucible","level":30,"time":"2024-08-08T15:44:17.772186913Z","hostname":"w-01J4SAFH2PQH83DF5HAX003QMF","pid":1233,"":"downstairs","":"downstairs","session_id":"57d19fac-856e-453c-969e-e9969759be51"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-08-08T15:44:17.772192964Z","hostname":"w-01J4SAFH2PQH83DF5HAX003QMF","pid":1233,"":"downstairs","":"downstairs","session_id":"57d19fac-856e-453c-969e-e9969759be51"} Wait for a query_work_queue command to finish before sending IO {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-08-08T15:44:17.772202284Z","hostname":"w-01J4SAFH2PQH83DF5HAX003QMF","pid":1233,"session_id":"57d19fac-856e-453c-969e-e9969759be51"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-08-08T15:44:17.772207835Z","hostname":"w-01J4SAFH2PQH83DF5HAX003QMF","pid":1233,"client":"0","":"downstairs","session_id":"57d19fac-856e-453c-969e-e9969759be51"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-08-08T15:44:17.772213875Z","hostname":"w-01J4SAFH2PQH83DF5HAX003QMF","pid":1233,"client":"1","":"downstairs","session_id":"57d19fac-856e-453c-969e-e9969759be51"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-08-08T15:44:17.772219835Z","hostname":"w-01J4SAFH2PQH83DF5HAX003QMF","pid":1233,"client":"2","":"downstairs","session_id":"57d19fac-856e-453c-969e-e9969759be51"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-08-08T15:44:17.772226956Z","hostname":"w-01J4SAFH2PQH83DF5HAX003QMF","pid":1233,"session_id":"57d19fac-856e-453c-969e-e9969759be51"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-08-08T15:44:17.772231956Z","hostname":"w-01J4SAFH2PQH83DF5HAX003QMF","pid":1233,"session_id":"57d19fac-856e-453c-969e-e9969759be51"} {"msg":"f35ebbf2-b3d9-4557-a7c7-8535e4257519 is now active with session: 57d19fac-856e-453c-969e-e9969759be51","v":0,"name":"crucible","level":30,"time":"2024-08-08T15:44:17.772238076Z","hostname":"w-01J4SAFH2PQH83DF5HAX003QMF","pid":1233,"session_id":"57d19fac-856e-453c-969e-e9969759be51"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-08-08T15:44:17.772243277Z","hostname":"w-01J4SAFH2PQH83DF5HAX003QMF","pid":1233,"session_id":"57d19fac-856e-453c-969e-e9969759be51"} {"msg":"The guest has finished waiting for activation with:1","v":0,"name":"crucible","level":30,"time":"2024-08-08T15:44:17.772248347Z","hostname":"w-01J4SAFH2PQH83DF5HAX003QMF","pid":1233} Region: es:16384 ec:1280 bs:4096 ts:85899345920 tb:20971520 max_io:256 or 1048576 Perf test TEST SECONDS COUNT DPTH IOPS MEAN P95 P99 MAX ES EC rwrites 6.67 30000 1 4498.38 0.00022 0.00117 0.00150 0.02551 16384 1280 rwrites 6.57 30000 1 4567.51 0.00022 0.00116 0.00154 0.03056 16384 1280 rreads 5.38 30000 1 5580.91 0.00018 0.00021 0.00028 0.01518 16384 1280 rreads 5.31 30000 1 5648.35 0.00018 0.00021 0.00027 0.01518 16384 1280 Perf test completed, stop all downstairs Aug 08 15:44:44.326 INFO accepted connection, remote_addr: 127.0.0.1:52909, local_addr: 127.0.0.1:9998 Main task has work to do, go find it got dsc Shutdown Shutdown Aug 08 15:44:44.327 INFO Shut it down request completed[8810] Disable keep_running , [8810] Got stop action so:false kr:false latency_us[8820] Disable keep_running :[8820] Got stop action so:false kr:false 61, response_code: 204, uri: /shutdown, method: POST, req_id: c863244b-96bb-4513-8a79-299da881a636, remote_addr: 127.0.0.1:52909, local_addr: 127.0.[8830] Disable keep_running 0[8830] Got stop action so:false kr:false .1:9998 [8830] Exited with: ExitStatus(unix_wait_status(9)) [8820] Exited with: ExitStatus(unix_wait_status(9)) [8830][2] reports Exit [8820][1] reports Exit [8810] Exited with: ExitStatus(unix_wait_status(9)) [8810][0] reports Exit TEST SECONDS COUNT DPTH IOPS MEAN P95 P99 MAX ES EC rwrites 4.96 30000 1 6048.16 0.00017 0.00040 0.00130 0.05595 16384 640 rwrites 5.10 30000 1 5882.22 0.00017 0.00046 0.00140 0.02869 16384 640 rwrites 6.67 30000 1 4498.38 0.00022 0.00117 0.00150 0.02551 16384 1280 rwrites 6.57 30000 1 4567.51 0.00022 0.00116 0.00154 0.03056 16384 1280 TEST SECONDS COUNT DPTH IOPS MEAN P95 P99 MAX ES EC rreads 5.58 30000 1 5372.36 0.00019 0.00026 0.00030 0.01439 16384 640 rreads 5.53 30000 1 5423.08 0.00018 0.00025 0.00029 0.01439 16384 640 rreads 5.38 30000 1 5580.91 0.00018 0.00021 0.00028 0.01518 16384 1280 rreads 5.31 30000 1 5648.35 0.00018 0.00021 0.00027 0.01518 16384 1280 Perf test finished on August 8, 2024 at 03:44:45 PM UTC