/input/rbuild/work Create region with ES:16384 EC:640 BS:4096 Creating region directory at: /var/tmp/dsc Creating dsc directory at: /tmp/dsc Downstairs region 0 created at /var/tmp/dsc/8810 in 1.1422219 Downstairs region 1 created at /var/tmp/dsc/8820 in 1.1668293 Downstairs region 2 created at /var/tmp/dsc/8830 in 0.7839185 Region set with 3 regions was created Using existing output directory "/tmp/dsc" start ds: 8810 start ds: 8820 start ds: 8830 start access at:127.0.0.1:9998 Starting downstairs at port 8810 Starting downstairs at port 8820 Make output file at "/tmp/dsc/downstairs-8810.txt" Starting downstairs at port 8830 Make output file at "/tmp/dsc/downstairs-8820.txt" Make output file at "/tmp/dsc/downstairs-8830.txt" [8810][0] reports Starting [8820][1] reports Starting [8830][2] reports Starting Downstairs /var/tmp/dsc/8810 port 8810 PID:Some(1150) [8810][0] reports Running Downstairs /var/tmp/dsc/8820 port 8820 PID:Some(1151) [8820][1] reports Running Downstairs /var/tmp/dsc/8830 port 8830 PID:Some(1152) Feb 27 21:45:41.017 INFO[8830][2] reports Running listening, local_addr: 127.0.0.1:9998 Control access at:127.0.0.1:9998 1150 1151 1152 Feb 27 21:45:46.055 INFO accepted connection, remote_addr: 127.0.0.1:52230, local_addr: 127.0.0.1:9998 Main task has work to do, go find it got dsc DisableRestartAll disable restart on all: 3 Feb 27 21:45:46.055 INFO request completed, latency_us: [8820] Disable keep_running [8830] Disable keep_running 145, response_code: 204, uri: /disablerestart/all, method: POST, [8810] Disable keep_running req_id: 9f94922d-2ddf-4b8b-8ccb-815531adb21b, remote_addr: 127.0.0.1:52230, local_addr: 127.0.0.1:9998 A new loop begins at February 27, 2024 at 09:45:46 PM UTC /var/tmp/bins/crutest perf -g 1 -t 127.0.0.1:8810 -t 127.0.0.1:8820 -t 127.0.0.1:8830 -c 30000 -q --write-loops 2 --read-loops 2 --perf-out /tmp/perf-ES-16384-EC-640.csv {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:45:46.077579307Z","hostname":"ip-10-150-1-161.us-west-2.compute.internal","pid":1160} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"f6c40444ff68361c835e88ee91c409e8deb48d7d\",\n git_commit_timestamp: \"2024-02-27T21:23:06.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.75.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"82e1608dfa6e0b5569232559e3d385fea5a93112\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: false,\n opt_level: 3,\n}","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:45:46.077755505Z","hostname":"ip-10-150-1-161.us-west-2.compute.internal","pid":1160} {"msg":"Upstairs <-> Downstairs Message Version: 5","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:45:46.077763645Z","hostname":"ip-10-150-1-161.us-west-2.compute.internal","pid":1160} {"msg":"Crucible d207b9f7-0efd-4c78-a024-e8bee511253e has session id: b6eb08b5-0e06-49c0-b7a1-a74a4345bf8f","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:45:46.077770184Z","hostname":"ip-10-150-1-161.us-west-2.compute.internal","pid":1160,"session_id":"b6eb08b5-0e06-49c0-b7a1-a74a4345bf8f"} {"msg":"Upstairs opts: Upstairs UUID: d207b9f7-0efd-4c78-a024-e8bee511253e, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:45:46.077823581Z","hostname":"ip-10-150-1-161.us-west-2.compute.internal","pid":1160,"session_id":"b6eb08b5-0e06-49c0-b7a1-a74a4345bf8f"} {"msg":"Crucible stats registered with UUID: d207b9f7-0efd-4c78-a024-e8bee511253e","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:45:46.0778298Z","hostname":"ip-10-150-1-161.us-west-2.compute.internal","pid":1160,"session_id":"b6eb08b5-0e06-49c0-b7a1-a74a4345bf8f"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:1","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:45:46.077923514Z","hostname":"ip-10-150-1-161.us-west-2.compute.internal","pid":1160} {"msg":"d207b9f7-0efd-4c78-a024-e8bee511253e active request set","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:45:46.078152728Z","hostname":"ip-10-150-1-161.us-west-2.compute.internal","pid":1160,"session_id":"b6eb08b5-0e06-49c0-b7a1-a74a4345bf8f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:45:46.078174467Z","hostname":"ip-10-150-1-161.us-west-2.compute.internal","pid":1160,"client":"0","":"downstairs","session_id":"b6eb08b5-0e06-49c0-b7a1-a74a4345bf8f"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:45:46.078221334Z","hostname":"ip-10-150-1-161.us-west-2.compute.internal","pid":1160,"client":"0","":"downstairs","session_id":"b6eb08b5-0e06-49c0-b7a1-a74a4345bf8f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:45:46.078232063Z","hostname":"ip-10-150-1-161.us-west-2.compute.internal","pid":1160,"client":"1","":"downstairs","session_id":"b6eb08b5-0e06-49c0-b7a1-a74a4345bf8f"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:45:46.078253182Z","hostname":"ip-10-150-1-161.us-west-2.compute.internal","pid":1160,"client":"1","":"downstairs","session_id":"b6eb08b5-0e06-49c0-b7a1-a74a4345bf8f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:45:46.078262961Z","hostname":"ip-10-150-1-161.us-west-2.compute.internal","pid":1160,"client":"2","":"downstairs","session_id":"b6eb08b5-0e06-49c0-b7a1-a74a4345bf8f"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:45:46.07828282Z","hostname":"ip-10-150-1-161.us-west-2.compute.internal","pid":1160,"client":"2","":"downstairs","session_id":"b6eb08b5-0e06-49c0-b7a1-a74a4345bf8f"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:45:46.078291949Z","hostname":"ip-10-150-1-161.us-west-2.compute.internal","pid":1160,"":"io task","client":"2","":"downstairs","session_id":"b6eb08b5-0e06-49c0-b7a1-a74a4345bf8f"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:45:46.078302569Z","hostname":"ip-10-150-1-161.us-west-2.compute.internal","pid":1160,"":"io task","client":"0","":"downstairs","session_id":"b6eb08b5-0e06-49c0-b7a1-a74a4345bf8f"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:45:46.078312028Z","hostname":"ip-10-150-1-161.us-west-2.compute.internal","pid":1160,"":"io task","client":"1","":"downstairs","session_id":"b6eb08b5-0e06-49c0-b7a1-a74a4345bf8f"} {"msg":"ds_connection connected","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:45:46.078407461Z","hostname":"ip-10-150-1-161.us-west-2.compute.internal","pid":1160,"":"io task","client":"0","":"downstairs","session_id":"b6eb08b5-0e06-49c0-b7a1-a74a4345bf8f"} {"msg":"ds_connection connected","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:45:46.07842803Z","hostname":"ip-10-150-1-161.us-west-2.compute.internal","pid":1160,"":"io task","client":"2","":"downstairs","session_id":"b6eb08b5-0e06-49c0-b7a1-a74a4345bf8f"} {"msg":"ds_connection connected","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:45:46.078599869Z","hostname":"ip-10-150-1-161.us-west-2.compute.internal","pid":1160,"":"io task","client":"1","":"downstairs","session_id":"b6eb08b5-0e06-49c0-b7a1-a74a4345bf8f"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:45:46.078947945Z","hostname":"ip-10-150-1-161.us-west-2.compute.internal","pid":1160,"client":"1","":"downstairs","session_id":"b6eb08b5-0e06-49c0-b7a1-a74a4345bf8f"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:45:46.078968334Z","hostname":"ip-10-150-1-161.us-west-2.compute.internal","pid":1160,"client":"1","":"downstairs","session_id":"b6eb08b5-0e06-49c0-b7a1-a74a4345bf8f"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:45:46.078976763Z","hostname":"ip-10-150-1-161.us-west-2.compute.internal","pid":1160,"client":"0","":"downstairs","session_id":"b6eb08b5-0e06-49c0-b7a1-a74a4345bf8f"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:45:46.078984413Z","hostname":"ip-10-150-1-161.us-west-2.compute.internal","pid":1160,"client":"0","":"downstairs","session_id":"b6eb08b5-0e06-49c0-b7a1-a74a4345bf8f"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:45:46.078992952Z","hostname":"ip-10-150-1-161.us-west-2.compute.internal","pid":1160,"client":"2","":"downstairs","session_id":"b6eb08b5-0e06-49c0-b7a1-a74a4345bf8f"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:45:46.079000272Z","hostname":"ip-10-150-1-161.us-west-2.compute.internal","pid":1160,"client":"2","":"downstairs","session_id":"b6eb08b5-0e06-49c0-b7a1-a74a4345bf8f"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:45:46.079221787Z","hostname":"ip-10-150-1-161.us-west-2.compute.internal","pid":1160,"client":"0","":"downstairs","session_id":"b6eb08b5-0e06-49c0-b7a1-a74a4345bf8f"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:45:46.079238056Z","hostname":"ip-10-150-1-161.us-west-2.compute.internal","pid":1160,"client":"1","":"downstairs","session_id":"b6eb08b5-0e06-49c0-b7a1-a74a4345bf8f"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:45:46.079245985Z","hostname":"ip-10-150-1-161.us-west-2.compute.internal","pid":1160,"client":"2","":"downstairs","session_id":"b6eb08b5-0e06-49c0-b7a1-a74a4345bf8f"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:45:46.081592449Z","hostname":"ip-10-150-1-161.us-west-2.compute.internal","pid":1160,"client":"1","":"downstairs","session_id":"b6eb08b5-0e06-49c0-b7a1-a74a4345bf8f"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:45:46.081616147Z","hostname":"ip-10-150-1-161.us-west-2.compute.internal","pid":1160,"client":"1","":"downstairs","session_id":"b6eb08b5-0e06-49c0-b7a1-a74a4345bf8f"} {"msg":"Waiting for true more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:45:46.081624487Z","hostname":"ip-10-150-1-161.us-west-2.compute.internal","pid":1160,"session_id":"b6eb08b5-0e06-49c0-b7a1-a74a4345bf8f"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:45:46.08247273Z","hostname":"ip-10-150-1-161.us-west-2.compute.internal","pid":1160,"client":"0","":"downstairs","session_id":"b6eb08b5-0e06-49c0-b7a1-a74a4345bf8f"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:45:46.082489089Z","hostname":"ip-10-150-1-161.us-west-2.compute.internal","pid":1160,"client":"0","":"downstairs","session_id":"b6eb08b5-0e06-49c0-b7a1-a74a4345bf8f"} {"msg":"Waiting for true more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:45:46.082498008Z","hostname":"ip-10-150-1-161.us-west-2.compute.internal","pid":1160,"session_id":"b6eb08b5-0e06-49c0-b7a1-a74a4345bf8f"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:45:46.082504068Z","hostname":"ip-10-150-1-161.us-west-2.compute.internal","pid":1160,"client":"2","":"downstairs","session_id":"b6eb08b5-0e06-49c0-b7a1-a74a4345bf8f"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:45:46.082509397Z","hostname":"ip-10-150-1-161.us-west-2.compute.internal","pid":1160,"client":"2","":"downstairs","session_id":"b6eb08b5-0e06-49c0-b7a1-a74a4345bf8f"} {"msg":"[0]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:45:46.082537955Z","hostname":"ip-10-150-1-161.us-west-2.compute.internal","pid":1160,"":"downstairs","":"downstairs","session_id":"b6eb08b5-0e06-49c0-b7a1-a74a4345bf8f"} {"msg":"[0]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:45:46.082544925Z","hostname":"ip-10-150-1-161.us-west-2.compute.internal","pid":1160,"":"downstairs","":"downstairs","session_id":"b6eb08b5-0e06-49c0-b7a1-a74a4345bf8f"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:45:46.082552334Z","hostname":"ip-10-150-1-161.us-west-2.compute.internal","pid":1160,"":"downstairs","":"downstairs","session_id":"b6eb08b5-0e06-49c0-b7a1-a74a4345bf8f"} {"msg":"[1]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:45:46.082561944Z","hostname":"ip-10-150-1-161.us-west-2.compute.internal","pid":1160,"":"downstairs","":"downstairs","session_id":"b6eb08b5-0e06-49c0-b7a1-a74a4345bf8f"} {"msg":"[1]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:45:46.082581923Z","hostname":"ip-10-150-1-161.us-west-2.compute.internal","pid":1160,"":"downstairs","":"downstairs","session_id":"b6eb08b5-0e06-49c0-b7a1-a74a4345bf8f"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:45:46.082587662Z","hostname":"ip-10-150-1-161.us-west-2.compute.internal","pid":1160,"":"downstairs","":"downstairs","session_id":"b6eb08b5-0e06-49c0-b7a1-a74a4345bf8f"} {"msg":"[2]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:45:46.082593242Z","hostname":"ip-10-150-1-161.us-west-2.compute.internal","pid":1160,"":"downstairs","":"downstairs","session_id":"b6eb08b5-0e06-49c0-b7a1-a74a4345bf8f"} {"msg":"[2]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:45:46.082598442Z","hostname":"ip-10-150-1-161.us-west-2.compute.internal","pid":1160,"":"downstairs","":"downstairs","session_id":"b6eb08b5-0e06-49c0-b7a1-a74a4345bf8f"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:45:46.082603581Z","hostname":"ip-10-150-1-161.us-west-2.compute.internal","pid":1160,"":"downstairs","":"downstairs","session_id":"b6eb08b5-0e06-49c0-b7a1-a74a4345bf8f"} {"msg":"Max found gen is 1","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:45:46.082608951Z","hostname":"ip-10-150-1-161.us-west-2.compute.internal","pid":1160,"":"downstairs","":"downstairs","session_id":"b6eb08b5-0e06-49c0-b7a1-a74a4345bf8f"} {"msg":"Generation requested: 1 >= found:1","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:45:46.08261591Z"Wait for a query_work_queue command to finish before sending IO ,"hostname":"ip-10-150-1-161.us-west-2.compute.internal","pid":1160,"":"downstairs","":"downstairs","session_id":"b6eb08b5-0e06-49c0-b7a1-a74a4345bf8f"} {"msg":"Next flush: 1","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:45:46.082688775Z","hostname":"ip-10-150-1-161.us-west-2.compute.internal","pid":1160,"":"downstairs","":"downstairs","session_id":"b6eb08b5-0e06-49c0-b7a1-a74a4345bf8f"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:45:46.082695055Z","hostname":"ip-10-150-1-161.us-west-2.compute.internal","pid":1160,"":"downstairs","":"downstairs","session_id":"b6eb08b5-0e06-49c0-b7a1-a74a4345bf8f"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:45:46.082701744Z","hostname":"ip-10-150-1-161.us-west-2.compute.internal","pid":1160,"session_id":"b6eb08b5-0e06-49c0-b7a1-a74a4345bf8f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:45:46.082706494Z","hostname":"ip-10-150-1-161.us-west-2.compute.internal","pid":1160,"client":"0","":"downstairs","session_id":"b6eb08b5-0e06-49c0-b7a1-a74a4345bf8f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:45:46.082711894Z","hostname":"ip-10-150-1-161.us-west-2.compute.internal","pid":1160,"client":"1","":"downstairs","session_id":"b6eb08b5-0e06-49c0-b7a1-a74a4345bf8f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:45:46.082717043Z","hostname":"ip-10-150-1-161.us-west-2.compute.internal","pid":1160,"client":"2","":"downstairs","session_id":"b6eb08b5-0e06-49c0-b7a1-a74a4345bf8f"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:45:46.082723493Z","hostname":"ip-10-150-1-161.us-west-2.compute.internal","pid":1160,"session_id":"b6eb08b5-0e06-49c0-b7a1-a74a4345bf8f"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:45:46.082728333Z","hostname":"ip-10-150-1-161.us-west-2.compute.internal","pid":1160,"session_id":"b6eb08b5-0e06-49c0-b7a1-a74a4345bf8f"} {"msg":"d207b9f7-0efd-4c78-a024-e8bee511253e is now active with session: b6eb08b5-0e06-49c0-b7a1-a74a4345bf8f","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:45:46.082733393Z","hostname":"ip-10-150-1-161.us-west-2.compute.internal","pid":1160,"session_id":"b6eb08b5-0e06-49c0-b7a1-a74a4345bf8f"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:45:46.082739072Z","hostname":"ip-10-150-1-161.us-west-2.compute.internal","pid":1160,"session_id":"b6eb08b5-0e06-49c0-b7a1-a74a4345bf8f"} {"msg":"The guest has finished waiting for activation with:1","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:45:46.082743862Z","hostname":"ip-10-150-1-161.us-west-2.compute.internal","pid":1160} Region: es:16384 ec:640 bs:4096 ts:42949672960 tb:10485760 max_io:256 or 1048576 Perf test TEST SECONDS COUNT DPTH IOPS MEAN P95 P99 MAX ES EC rwrites 17.14 30000 1 1749.80 0.00057 0.00137 0.00187 0.03067 16384 640 rwrites 8.47 30000 1 3541.61 0.00028 0.00096 0.00155 0.06994 16384 640 rreads 5.69 30000 1 5275.30 0.00019 0.00026 0.00031 0.01379 16384 640 rreads 5.77 30000 1 5199.26 0.00019 0.00027 0.00031 0.01379 16384 640 Perf test completed, stop all downstairs Feb 27 21:46:35.439 INFO accepted connection, remote_addr: 127.0.0.1:33696, local_addr: 127.0.0.1:9998 Main task has work to do, go find it got dsc Shutdown Shutdown Shut it down Feb 27 21:46:35.440 INFO[8810] Disable keep_running [8810] Got stop action so:false kr:false request completed, latency_us: 202, response_code: 204, uri: /shutdown[8830] Disable keep_running , method: POST, req_id: 51abd867-631f-4e7b-8314-0112209ae986, remote_addr: 127.0.0.1:33696, local_addr: 127.0.0.1:9998[8830] Got stop action so:false kr:false [8820] Disable keep_running [8820] Got stop action so:false kr:false [8820] Exited with: ExitStatus(unix_wait_status(9)) [8820][1] reports Exit [8810] Exited with: ExitStatus(unix_wait_status(9)) [8810][0] reports Exit [8830] Exited with: ExitStatus(unix_wait_status(9)) [8830][2] reports Exit Create region with ES:16384 EC:1280 BS:4096 Removing existing dsc directory "/tmp/dsc" Removing existing region "/var/tmp/dsc" Creating region directory at: /var/tmp/dsc Creating dsc directory at: /tmp/dsc Downstairs region 0 created at /var/tmp/dsc/8810 in 1.3140421 Downstairs region 1 created at /var/tmp/dsc/8820 in 1.3155131 Downstairs region 2 created at /var/tmp/dsc/8830 in 1.283302 Region set with 3 regions was created Using existing output directory "/tmp/dsc" start ds: 8810 start ds: 8820 start ds: 8830 start access at:127.0.0.1:9998 Starting downstairs at port 8810 Starting downstairs at port 8820 Make output file at "/tmp/dsc/downstairs-8810.txt" Make output file at "/tmp/dsc/downstairs-8820.txt" [8810][0] reports Starting Starting downstairs at port 8830 [8820][1] reports Starting Make output file at "/tmp/dsc/downstairs-8830.txt" [8830][2] reports Starting Downstairs /var/tmp/dsc/8810 port 8810 PID:Some(1169) [8810][0] reports Running Downstairs /var/tmp/dsc/8830 port 8830 PID:Some(1170) [8830][2] reports Running Control access at:127.0.0.1:9998 Downstairs /var/tmp/dsc/8820 port 8820 PID:Some(1171) [8820][1] reports Running Feb 27 21:46:40.419 INFO listening, local_addr: 127.0.0.1:9998 1171 1169 1170 Feb 27 21:46:45.457 INFO accepted connection, remote_addr: 127.0.0.1:39783, local_addr: 127.0.0.1:9998 Main task has work to do, go find it got dsc DisableRestartAll disable restart on all: 3 Feb 27 21:46:45.457 INFO request completed, latency_us: [8810] Disable keep_running 127, response_code[8820] Disable keep_running :[8830] Disable keep_running 204, uri: /disablerestart/all, method: POST, req_id: 5f71e3b7-a4eb-44cb-985e-1082296fa55f, remote_addr: 127.0.0.1:39783, local_addr: 127.0.0.1:9998 A new loop begins at February 27, 2024 at 09:46:45 PM UTC /var/tmp/bins/crutest perf -g 1 -t 127.0.0.1:8810 -t 127.0.0.1:8820 -t 127.0.0.1:8830 -c 30000 -q --write-loops 2 --read-loops 2 --perf-out /tmp/perf-ES-16384-EC-1280.csv {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:46:45.478676786Z","hostname":"ip-10-150-1-161.us-west-2.compute.internal","pid":1179} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"f6c40444ff68361c835e88ee91c409e8deb48d7d\",\n git_commit_timestamp: \"2024-02-27T21:23:06.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.75.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"82e1608dfa6e0b5569232559e3d385fea5a93112\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: false,\n opt_level: 3,\n}","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:46:45.478803698Z","hostname":"ip-10-150-1-161.us-west-2.compute.internal","pid":1179} {"msg":"Upstairs <-> Downstairs Message Version: 5","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:46:45.478810398Z","hostname":"ip-10-150-1-161.us-west-2.compute.internal","pid":1179} {"msg":"Crucible 7fabf0cd-1e43-4e28-b43d-cb4ebabbbe8f has session id: 47c3f618-85f0-4eda-a99a-434f13c6c99d","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:46:45.478815087Z","hostname":"ip-10-150-1-161.us-west-2.compute.internal","pid":1179,"session_id":"47c3f618-85f0-4eda-a99a-434f13c6c99d"} {"msg":"Upstairs opts: Upstairs UUID: 7fabf0cd-1e43-4e28-b43d-cb4ebabbbe8f, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:46:45.478825766Z","hostname":"ip-10-150-1-161.us-west-2.compute.internal","pid":1179,"session_id":"47c3f618-85f0-4eda-a99a-434f13c6c99d"} {"msg":"Crucible stats registered with UUID: 7fabf0cd-1e43-4e28-b43d-cb4ebabbbe8f","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:46:45.478830236Z","hostname":"ip-10-150-1-161.us-west-2.compute.internal","pid":1179,"session_id":"47c3f618-85f0-4eda-a99a-434f13c6c99d"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:1","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:46:45.478896072Z","hostname":"ip-10-150-1-161.us-west-2.compute.internal","pid":1179} {"msg":"7fabf0cd-1e43-4e28-b43d-cb4ebabbbe8f active request set","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:46:45.478956108Z","hostname":"ip-10-150-1-161.us-west-2.compute.internal","pid":1179,"session_id":"47c3f618-85f0-4eda-a99a-434f13c6c99d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:46:45.478968387Z","hostname":"ip-10-150-1-161.us-west-2.compute.internal","pid":1179,"client":"0","":"downstairs","session_id":"47c3f618-85f0-4eda-a99a-434f13c6c99d"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:46:45.478978256Z","hostname":"ip-10-150-1-161.us-west-2.compute.internal","pid":1179,"client":"0","":"downstairs","session_id":"47c3f618-85f0-4eda-a99a-434f13c6c99d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:46:45.478982946Z","hostname":"ip-10-150-1-161.us-west-2.compute.internal","pid":1179,"client":"1","":"downstairs","session_id":"47c3f618-85f0-4eda-a99a-434f13c6c99d"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:46:45.478987376Z","hostname":"ip-10-150-1-161.us-west-2.compute.internal","pid":1179,"client":"1","":"downstairs","session_id":"47c3f618-85f0-4eda-a99a-434f13c6c99d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:46:45.478991665Z","hostname":"ip-10-150-1-161.us-west-2.compute.internal","pid":1179,"client":"2","":"downstairs","session_id":"47c3f618-85f0-4eda-a99a-434f13c6c99d"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:46:45.478996185Z","hostname":"ip-10-150-1-161.us-west-2.compute.internal","pid":1179,"client":"2","":"downstairs","session_id":"47c3f618-85f0-4eda-a99a-434f13c6c99d"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:46:45.479003614Z","hostname":"ip-10-150-1-161.us-west-2.compute.internal","pid":1179,"":"io task","client":"2","":"downstairs","session_id":"47c3f618-85f0-4eda-a99a-434f13c6c99d"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:46:45.479020004Z","hostname":"ip-10-150-1-161.us-west-2.compute.internal","pid":1179,"":"io task","client":"0","":"downstairs","session_id":"47c3f618-85f0-4eda-a99a-434f13c6c99d"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:46:45.479052201Z","hostname":"ip-10-150-1-161.us-west-2.compute.internal","pid":1179,"":"io task","client":"1","":"downstairs","session_id":"47c3f618-85f0-4eda-a99a-434f13c6c99d"} {"msg":"ds_connection connected","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:46:45.479198092Z","hostname":"ip-10-150-1-161.us-west-2.compute.internal","pid":1179,"":"io task","client":"2","":"downstairs","session_id":"47c3f618-85f0-4eda-a99a-434f13c6c99d"} {"msg":"ds_connection connected","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:46:45.47921837Z","hostname":"ip-10-150-1-161.us-west-2.compute.internal","pid":1179,"":"io task","client":"1","":"downstairs","session_id":"47c3f618-85f0-4eda-a99a-434f13c6c99d"} {"msg":"ds_connection connected","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:46:45.47922572Z","hostname":"ip-10-150-1-161.us-west-2.compute.internal","pid":1179,"":"io task","client":"0","":"downstairs","session_id":"47c3f618-85f0-4eda-a99a-434f13c6c99d"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:46:45.479632483Z","hostname":"ip-10-150-1-161.us-west-2.compute.internal","pid":1179,"client":"0","":"downstairs","session_id":"47c3f618-85f0-4eda-a99a-434f13c6c99d"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:46:45.479652951Z","hostname":"ip-10-150-1-161.us-west-2.compute.internal","pid":1179,"client":"0","":"downstairs","session_id":"47c3f618-85f0-4eda-a99a-434f13c6c99d"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:46:45.479660481Z","hostname":"ip-10-150-1-161.us-west-2.compute.internal","pid":1179,"client":"1","":"downstairs","session_id":"47c3f618-85f0-4eda-a99a-434f13c6c99d"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:46:45.47966742Z","hostname":"ip-10-150-1-161.us-west-2.compute.internal","pid":1179,"client":"1","":"downstairs","session_id":"47c3f618-85f0-4eda-a99a-434f13c6c99d"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:46:45.47967401Z","hostname":"ip-10-150-1-161.us-west-2.compute.internal","pid":1179,"client":"2","":"downstairs","session_id":"47c3f618-85f0-4eda-a99a-434f13c6c99d"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:46:45.479680139Z","hostname":"ip-10-150-1-161.us-west-2.compute.internal","pid":1179,"client":"2","":"downstairs","session_id":"47c3f618-85f0-4eda-a99a-434f13c6c99d"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:46:45.479945542Z","hostname":"ip-10-150-1-161.us-west-2.compute.internal","pid":1179,"client":"2","":"downstairs","session_id":"47c3f618-85f0-4eda-a99a-434f13c6c99d"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:46:45.479957921Z","hostname":"ip-10-150-1-161.us-west-2.compute.internal","pid":1179,"client":"1","":"downstairs","session_id":"47c3f618-85f0-4eda-a99a-434f13c6c99d"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:46:45.479982779Z","hostname":"ip-10-150-1-161.us-west-2.compute.internal","pid":1179,"client":"0","":"downstairs","session_id":"47c3f618-85f0-4eda-a99a-434f13c6c99d"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:46:45.485490221Z","hostname":"ip-10-150-1-161.us-west-2.compute.internal","pid":1179,"client":"1","":"downstairs","session_id":"47c3f618-85f0-4eda-a99a-434f13c6c99d"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:46:45.48551056Z","hostname":"ip-10-150-1-161.us-west-2.compute.internal","pid":1179,"client":"1","":"downstairs","session_id":"47c3f618-85f0-4eda-a99a-434f13c6c99d"} {"msg":"Waiting for true more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:46:45.485518959Z","hostname":"ip-10-150-1-161.us-west-2.compute.internal","pid":1179,"session_id":"47c3f618-85f0-4eda-a99a-434f13c6c99d"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:46:45.486142528Z","hostname":"ip-10-150-1-161.us-west-2.compute.internal","pid":1179,"client":"0","":"downstairs","session_id":"47c3f618-85f0-4eda-a99a-434f13c6c99d"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:46:45.486160707Z","hostname":"ip-10-150-1-161.us-west-2.compute.internal","pid":1179,"client":"0","":"downstairs","session_id":"47c3f618-85f0-4eda-a99a-434f13c6c99d"} {"msg":"Waiting for true more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:46:45.486166296Z","hostname":"ip-10-150-1-161.us-west-2.compute.internal","pid":1179,"session_id":"47c3f618-85f0-4eda-a99a-434f13c6c99d"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:46:45.493250943Z","hostname":"ip-10-150-1-161.us-west-2.compute.internal","pid":1179,"client":"2","":"downstairs","session_id":"47c3f618-85f0-4eda-a99a-434f13c6c99d"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:46:45.493274311Z","hostname":"ip-10-150-1-161.us-west-2.compute.internal","pid":1179,"client":"2","":"downstairs","session_id":"47c3f618-85f0-4eda-a99a-434f13c6c99d"} {"msg":"[0]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:46:45.493281611Z","hostname":"ip-10-150-1-161.us-west-2.compute.internal","pid":1179,"":"downstairs","":"downstairs","session_id":"47c3f618-85f0-4eda-a99a-434f13c6c99d"} {"msg":"[0]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:46:45.49328765Z","hostname":"ip-10-150-1-161.us-west-2.compute.internal","pid":1179,"":"downstairs","":"downstairs","session_id":"47c3f618-85f0-4eda-a99a-434f13c6c99d"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:46:45.49329364Z","hostname":"ip-10-150-1-161.us-west-2.compute.internal","pid":1179,"":"downstairs","":"downstairs","session_id":"47c3f618-85f0-4eda-a99a-434f13c6c99d"} {"msg":"[1]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:46:45.493299399Z","hostname":"ip-10-150-1-161.us-west-2.compute.internal","pid":1179,"":"downstairs","":"downstairs","session_id":"47c3f618-85f0-4eda-a99a-434f13c6c99d"} {"msg":"[1]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:46:45.493305459Z","hostname":"ip-10-150-1-161.us-west-2.compute.internal","pid":1179,"":"downstairs","":"downstairs","session_id":"47c3f618-85f0-4eda-a99a-434f13c6c99d"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:46:45.493310639Z","hostname":"ip-10-150-1-161.us-west-2.compute.internal","pid":1179,"":"downstairs","":"downstairs","session_id":"47c3f618-85f0-4eda-a99a-434f13c6c99d"} {"msg":"[2]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:46:45.493316228Z","hostname":"ip-10-150-1-161.us-west-2.compute.internal","pid":1179,"":"downstairs","":"downstairs","session_id":"47c3f618-85f0-4eda-a99a-434f13c6c99d"} {"msg":"[2]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:46:45.493321958Z","hostname":"ip-10-150-1-161.us-west-2.compute.internal","pid":1179,"":"downstairs","":"downstairs","session_id":"47c3f618-85f0-4eda-a99a-434f13c6c99d"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:46:45.493327608Z","hostname":"ip-10-150-1-161.us-west-2.compute.internal","pid":1179,"":"downstairs","":"downstairs","session_id":"47c3f618-85f0-4eda-a99a-434f13c6c99d"} {"msg":"Max found gen is 1","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:46:45.493333607Z","hostname":"ip-10-150-1-161.us-west-2.compute.internal","pid":1179,"":"downstairs","":"downstairs","session_id":"47c3f618-85f0-4eda-a99a-434f13c6c99d"} {"msg":"Generation requested: 1 >= found:1","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:46:45.493340257Z","hostname":"ip-10-150-1-161.us-west-2.compute.internal","pid":1179,"":"downstairs","":"downstairs","session_id":"47c3f618-85f0-4eda-a99a-434f13c6c99d"} {"msg":"Next flush: 1","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:46:45.493346106Z","hostname":"ip-10-150-1-161.us-west-2.compute.internal","pid":1179,"":"downstairs","":"downstairs","session_id":"47c3f618-85f0-4eda-a99a-434f13c6c99d"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:46:45.493351626Z","hostname":"ip-10-150-1-161.us-west-2.compute.internal","pid":1179,"":"downstairs","":"downstairs","session_id":"47c3f618-85f0-4eda-a99a-434f13c6c99d"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:46:45.493357806Z","hostname":"ip-10-150-1-161.us-west-2.compute.internal","pid":1179,"session_id":"47c3f618-85f0-4eda-a99a-434f13c6c99d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:46:45.493362725Z","hostname":"ip-10-150-1-161.us-west-2.compute.internal","pid":1179,"client":"0","":"downstairs","session_id":"47c3f618-85f0-4eda-a99a-434f13c6c99d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:46:45.493367935Z","hostname":"ip-10-150-1-161.us-west-2.compute.internal","pid":1179,"client":"1","":"downstairs","session_id":"47c3f618-85f0-4eda-a99a-434f13c6c99d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:46:45.493373105Z","hostname":Wait for a query_work_queue command to finish before sending IO "ip-10-150-1-161.us-west-2.compute.internal","pid":1179,"client":"2","":"downstairs","session_id":"47c3f618-85f0-4eda-a99a-434f13c6c99d"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:46:45.493382264Z","hostname":"ip-10-150-1-161.us-west-2.compute.internal","pid":1179,"session_id":"47c3f618-85f0-4eda-a99a-434f13c6c99d"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:46:45.493386864Z","hostname":"ip-10-150-1-161.us-west-2.compute.internal","pid":1179,"session_id":"47c3f618-85f0-4eda-a99a-434f13c6c99d"} {"msg":"7fabf0cd-1e43-4e28-b43d-cb4ebabbbe8f is now active with session: 47c3f618-85f0-4eda-a99a-434f13c6c99d","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:46:45.493391433Z","hostname":"ip-10-150-1-161.us-west-2.compute.internal","pid":1179,"session_id":"47c3f618-85f0-4eda-a99a-434f13c6c99d"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:46:45.493395813Z","hostname":"ip-10-150-1-161.us-west-2.compute.internal","pid":1179,"session_id":"47c3f618-85f0-4eda-a99a-434f13c6c99d"} {"msg":"The guest has finished waiting for activation with:1","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:46:45.493410632Z","hostname":"ip-10-150-1-161.us-west-2.compute.internal","pid":1179} Region: es:16384 ec:1280 bs:4096 ts:85899345920 tb:20971520 max_io:256 or 1048576 Perf test TEST SECONDS COUNT DPTH IOPS MEAN P95 P99 MAX ES EC rwrites 8.34 30000 1 3596.87 0.00028 0.00059 0.00118 0.08937 16384 1280 rwrites 8.34 30000 1 3595.02 0.00028 0.00062 0.00113 0.07626 16384 1280 rreads 5.47 30000 1 5484.73 0.00018 0.00023 0.00027 0.01292 16384 1280 rreads 5.49 30000 1 5464.09 0.00018 0.00023 0.00027 0.01499 16384 1280 Perf test completed, stop all downstairs Feb 27 21:47:28.599 INFO accepted connection, remote_addr: 127.0.0.1:44940, local_addr: 127.0.0.1:9998 Main task has work to do, go find it got dsc Shutdown Shutdown Feb 27 21:47:28.600 INFO Shut it down request completed[8810] Disable keep_running , [8820] Disable keep_running latency_us[8820] Got stop action so:false kr:false :[8810] Got stop action so:false kr:false [8830] Disable keep_running 55[8830] Got stop action so:false kr:false , response_code: 204, uri: /shutdown, method: POST, req_id: 415b536f-f716-4398-8257-4a40341fdbdf, remote_addr: 127.0.0.1:44940, local_addr: 127.0.0.1:9998 [8820] Exited with: ExitStatus(unix_wait_status(9)) [8820][1] reports Exit [8810] Exited with: ExitStatus(unix_wait_status(9)) [8810][0] reports Exit [8830] Exited with: ExitStatus(unix_wait_status(9)) [8830][2] reports Exit TEST SECONDS COUNT DPTH IOPS MEAN P95 P99 MAX ES EC rwrites 17.14 30000 1 1749.80 0.00057 0.00137 0.00187 0.03067 16384 640 rwrites 8.47 30000 1 3541.61 0.00028 0.00096 0.00155 0.06994 16384 640 rwrites 8.34 30000 1 3596.87 0.00028 0.00059 0.00118 0.08937 16384 1280 rwrites 8.34 30000 1 3595.02 0.00028 0.00062 0.00113 0.07626 16384 1280 TEST SECONDS COUNT DPTH IOPS MEAN P95 P99 MAX ES EC rreads 5.69 30000 1 5275.30 0.00019 0.00026 0.00031 0.01379 16384 640 rreads 5.77 30000 1 5199.26 0.00019 0.00027 0.00031 0.01379 16384 640 rreads 5.47 30000 1 5484.73 0.00018 0.00023 0.00027 0.01292 16384 1280 rreads 5.49 30000 1 5464.09 0.00018 0.00023 0.00027 0.01499 16384 1280 Perf test finished on February 27, 2024 at 09:47:29 PM UTC