/input/rbuild/work Create region with ES:16384 EC:640 BS:4096 Creating region directory at: /var/tmp/dsc Creating dsc directory at: /tmp/dsc Downstairs region 0 created at /var/tmp/dsc/8810 in 0.8427732 Downstairs region 1 created at /var/tmp/dsc/8820 in 0.66911954 Downstairs region 2 created at /var/tmp/dsc/8830 in 0.7045589 Region set with 3 regions was created Using existing output directory "/tmp/dsc" start ds: 8810 start access at:127.0.0.1:9998 start ds: 8820 start ds: 8830 Starting downstairs at port 8810 Make output file at "/tmp/dsc/downstairs-8810.txt" Starting downstairs at port 8820 Make output file at "/tmp/dsc/downstairs-8820.txt" Starting downstairs at port 8830 Make output file at "/tmp/dsc/downstairs-8830.txt" [8810][0] reports Starting [8820][1] reports Starting [8830][2] reports Starting Downstairs /var/tmp/dsc/8810 port 8810 PID:Some(1150) [8810][0] reports Running Downstairs /var/tmp/dsc/8820 port 8820 PID:Some(1151) [8820][1] reports Running Downstairs /var/tmp/dsc/8830 port 8830 PID:Some(1152) [8830][2] reports Running Feb 07 16:27:58.858 INFO listening, local_addr: 127.0.0.1:9998 Control access at:127.0.0.1:9998 1150 1151 1152 Feb 07 16:28:03.896 INFO accepted connection, remote_addr: 127.0.0.1:38430, local_addr: 127.0.0.1:9998 Main task has work to do, go find it got dsc DisableRestartAll disable restart on all: 3 Feb 07 16:28:03.897 INFO request completed, latency_us:[8810] Disable keep_running [8820] Disable keep_running [8830] Disable keep_running 423, response_code: 204, uri: /disablerestart/all, method: POST, req_id: 688bd12d-f284-410f-8e41-f2d3df3db53e, remote_addr: 127.0.0.1:38430, local_addr: 127.0.0.1:9998 A new loop begins at February 7, 2024 at 04:28:03 PM UTC /var/tmp/bins/crutest perf -g 1 -t 127.0.0.1:8810 -t 127.0.0.1:8820 -t 127.0.0.1:8830 -c 30000 -q --write-loops 2 --read-loops 2 --perf-out /tmp/perf-ES-16384-EC-640.csv {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-02-07T16:28:03.917976474Z","hostname":"ip-10-150-1-55.us-west-2.compute.internal","pid":1160} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"24836e2c4d76be332fe33bc8f22f62e25ba0a96e\",\n git_commit_timestamp: \"2024-02-07T16:05:39.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.75.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"82e1608dfa6e0b5569232559e3d385fea5a93112\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: false,\n opt_level: 3,\n}","v":0,"name":"crucible","level":30,"time":"2024-02-07T16:28:03.918173117Z","hostname":"ip-10-150-1-55.us-west-2.compute.internal","pid":1160} {"msg":"Upstairs <-> Downstairs Message Version: 5","v":0,"name":"crucible","level":30,"time":"2024-02-07T16:28:03.918181756Z","hostname":"ip-10-150-1-55.us-west-2.compute.internal","pid":1160} {"msg":"Crucible 5f1dcce7-a8e0-45b6-851d-bf7f3282ccb8 has session id: 4df6c04e-69a1-4c62-adbd-fc792f51bbe4","v":0,"name":"crucible","level":30,"time":"2024-02-07T16:28:03.918187115Z","hostname":"ip-10-150-1-55.us-west-2.compute.internal","pid":1160,"session_id":"4df6c04e-69a1-4c62-adbd-fc792f51bbe4"} {"msg":"Upstairs opts: Upstairs UUID: 5f1dcce7-a8e0-45b6-851d-bf7f3282ccb8, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-02-07T16:28:03.918202394Z","hostname":"ip-10-150-1-55.us-west-2.compute.internal","pid":1160,"session_id":"4df6c04e-69a1-4c62-adbd-fc792f51bbe4"} {"msg":"Crucible stats registered with UUID: 5f1dcce7-a8e0-45b6-851d-bf7f3282ccb8","v":0,"name":"crucible","level":30,"time":"2024-02-07T16:28:03.918209904Z","hostname":"ip-10-150-1-55.us-west-2.compute.internal","pid":1160,"session_id":"4df6c04e-69a1-4c62-adbd-fc792f51bbe4"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:1","v":0,"name":"crucible","level":30,"time":"2024-02-07T16:28:03.918343842Z","hostname":"ip-10-150-1-55.us-west-2.compute.internal","pid":1160} {"msg":"5f1dcce7-a8e0-45b6-851d-bf7f3282ccb8 active request set","v":0,"name":"crucible","level":30,"time":"2024-02-07T16:28:03.918543994Z","hostname":"ip-10-150-1-55.us-west-2.compute.internal","pid":1160,"session_id":"4df6c04e-69a1-4c62-adbd-fc792f51bbe4"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-02-07T16:28:03.918560063Z","hostname":"ip-10-150-1-55.us-west-2.compute.internal","pid":1160,"client":"0","":"downstairs","session_id":"4df6c04e-69a1-4c62-adbd-fc792f51bbe4"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-02-07T16:28:03.918566132Z","hostname":"ip-10-150-1-55.us-west-2.compute.internal","pid":1160,"client":"0","":"downstairs","session_id":"4df6c04e-69a1-4c62-adbd-fc792f51bbe4"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-02-07T16:28:03.918570982Z","hostname":"ip-10-150-1-55.us-west-2.compute.internal","pid":1160,"client":"1","":"downstairs","session_id":"4df6c04e-69a1-4c62-adbd-fc792f51bbe4"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-02-07T16:28:03.918575861Z","hostname":"ip-10-150-1-55.us-west-2.compute.internal","pid":1160,"client":"1","":"downstairs","session_id":"4df6c04e-69a1-4c62-adbd-fc792f51bbe4"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-02-07T16:28:03.918581211Z","hostname":"ip-10-150-1-55.us-west-2.compute.internal","pid":1160,"client":"2","":"downstairs","session_id":"4df6c04e-69a1-4c62-adbd-fc792f51bbe4"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-02-07T16:28:03.918585721Z","hostname":"ip-10-150-1-55.us-west-2.compute.internal","pid":1160,"client":"2","":"downstairs","session_id":"4df6c04e-69a1-4c62-adbd-fc792f51bbe4"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-02-07T16:28:03.91859029Z","hostname":"ip-10-150-1-55.us-west-2.compute.internal","pid":1160,"":"io task","client":"2","":"downstairs","session_id":"4df6c04e-69a1-4c62-adbd-fc792f51bbe4"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-07T16:28:03.918598Z","hostname":"ip-10-150-1-55.us-west-2.compute.internal","pid":1160,"":"io task","client":"0","":"downstairs","session_id":"4df6c04e-69a1-4c62-adbd-fc792f51bbe4"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-02-07T16:28:03.918610569Z","hostname":"ip-10-150-1-55.us-west-2.compute.internal","pid":1160,"":"io task","client":"1","":"downstairs","session_id":"4df6c04e-69a1-4c62-adbd-fc792f51bbe4"} {"msg":"ds_connection connected","v":0,"name":"crucible","level":30,"time":"2024-02-07T16:28:03.918713669Z","hostname":"ip-10-150-1-55.us-west-2.compute.internal","pid":1160,"":"io task","client":"2","":"downstairs","session_id":"4df6c04e-69a1-4c62-adbd-fc792f51bbe4"} {"msg":"ds_connection connected","v":0,"name":"crucible","level":30,"time":"2024-02-07T16:28:03.918724399Z","hostname":"ip-10-150-1-55.us-west-2.compute.internal","pid":1160,"":"io task","client":"1","":"downstairs","session_id":"4df6c04e-69a1-4c62-adbd-fc792f51bbe4"} {"msg":"ds_connection connected","v":0,"name":"crucible","level":30,"time":"2024-02-07T16:28:03.918733558Z","hostname":"ip-10-150-1-55.us-west-2.compute.internal","pid":1160,"":"io task","client":"0","":"downstairs","session_id":"4df6c04e-69a1-4c62-adbd-fc792f51bbe4"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-02-07T16:28:03.919326606Z","hostname":"ip-10-150-1-55.us-west-2.compute.internal","pid":1160,"client":"1","":"downstairs","session_id":"4df6c04e-69a1-4c62-adbd-fc792f51bbe4"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-02-07T16:28:03.919343414Z","hostname":"ip-10-150-1-55.us-west-2.compute.internal","pid":1160,"client":"1","":"downstairs","session_id":"4df6c04e-69a1-4c62-adbd-fc792f51bbe4"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-02-07T16:28:03.919348464Z","hostname":"ip-10-150-1-55.us-west-2.compute.internal","pid":1160,"client":"0","":"downstairs","session_id":"4df6c04e-69a1-4c62-adbd-fc792f51bbe4"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-02-07T16:28:03.919352644Z","hostname":"ip-10-150-1-55.us-west-2.compute.internal","pid":1160,"client":"0","":"downstairs","session_id":"4df6c04e-69a1-4c62-adbd-fc792f51bbe4"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-02-07T16:28:03.919356763Z","hostname":"ip-10-150-1-55.us-west-2.compute.internal","pid":1160,"client":"2","":"downstairs","session_id":"4df6c04e-69a1-4c62-adbd-fc792f51bbe4"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-02-07T16:28:03.919361283Z","hostname":"ip-10-150-1-55.us-west-2.compute.internal","pid":1160,"client":"2","":"downstairs","session_id":"4df6c04e-69a1-4c62-adbd-fc792f51bbe4"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-02-07T16:28:03.919838221Z","hostname":"ip-10-150-1-55.us-west-2.compute.internal","pid":1160,"client":"2","":"downstairs","session_id":"4df6c04e-69a1-4c62-adbd-fc792f51bbe4"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-02-07T16:28:03.919855Z","hostname":"ip-10-150-1-55.us-west-2.compute.internal","pid":1160,"client":"1","":"downstairs","session_id":"4df6c04e-69a1-4c62-adbd-fc792f51bbe4"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-02-07T16:28:03.919860239Z","hostname":"ip-10-150-1-55.us-west-2.compute.internal","pid":1160,"client":"0","":"downstairs","session_id":"4df6c04e-69a1-4c62-adbd-fc792f51bbe4"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-02-07T16:28:03.924183031Z","hostname":"ip-10-150-1-55.us-west-2.compute.internal","pid":1160,"client":"2","":"downstairs","session_id":"4df6c04e-69a1-4c62-adbd-fc792f51bbe4"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-02-07T16:28:03.92420264Z","hostname":"ip-10-150-1-55.us-west-2.compute.internal","pid":1160,"client":"2","":"downstairs","session_id":"4df6c04e-69a1-4c62-adbd-fc792f51bbe4"} {"msg":"Waiting for true more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-02-07T16:28:03.924210909Z","hostname":"ip-10-150-1-55.us-west-2.compute.internal","pid":1160,"session_id":"4df6c04e-69a1-4c62-adbd-fc792f51bbe4"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-02-07T16:28:03.92545524Z","hostname":"ip-10-150-1-55.us-west-2.compute.internal","pid":1160,"client":"1","":"downstairs","session_id":"4df6c04e-69a1-4c62-adbd-fc792f51bbe4"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-02-07T16:28:03.925473399Z","hostname":"ip-10-150-1-55.us-west-2.compute.internal","pid":1160,"client":"1","":"downstairs","session_id":"4df6c04e-69a1-4c62-adbd-fc792f51bbe4"} {"msg":"Waiting for true more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-02-07T16:28:03.925478948Z","hostname":"ip-10-150-1-55.us-west-2.compute.internal","pid":1160,"session_id":"4df6c04e-69a1-4c62-adbd-fc792f51bbe4"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-02-07T16:28:03.927757209Z","hostname":"ip-10-150-1-55.us-west-2.compute.internal","pid":1160,"client":"0","":"downstairs","session_id":"4df6c04e-69a1-4c62-adbd-fc792f51bbe4"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-02-07T16:28:03.927782607Z","hostname":"ip-10-150-1-55.us-west-2.compute.internal","pid":1160,"client":"0","":"downstairs","session_id":"4df6c04e-69a1-4c62-adbd-fc792f51bbe4"} {"msg":"[0]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-02-07T16:28:03.927823373Z","hostname":"ip-10-150-1-55.us-west-2.compute.internal","pid":1160,"":"downstairs","":"downstairs","session_id":"4df6c04e-69a1-4c62-adbd-fc792f51bbe4"} {"msg":"[0]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-02-07T16:28:03.927840172Z","hostname":"ip-10-150-1-55.us-west-2.compute.internal","pid":1160,"":"downstairs","":"downstairs","session_id":"4df6c04e-69a1-4c62-adbd-fc792f51bbe4"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-02-07T16:28:03.927845731Z","hostname":"ip-10-150-1-55.us-west-2.compute.internal","pid":1160,"":"downstairs","":"downstairs","session_id":"4df6c04e-69a1-4c62-adbd-fc792f51bbe4"} {"msg":"[1]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-02-07T16:28:03.927850301Z","hostname":"ip-10-150-1-55.us-west-2.compute.internal","pid":1160,"":"downstairs","":"downstairs","session_id":"4df6c04e-69a1-4c62-adbd-fc792f51bbe4"} {"msg":"[1]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-02-07T16:28:03.927854831Z","hostname":"ip-10-150-1-55.us-west-2.compute.internal","pid":1160,"":"downstairs","":"downstairs","session_id":"4df6c04e-69a1-4c62-adbd-fc792f51bbe4"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-02-07T16:28:03.92785909Z","hostname":"ip-10-150-1-55.us-west-2.compute.internal","pid":1160,"":"downstairs","":"downstairs","session_id":"4df6c04e-69a1-4c62-adbd-fc792f51bbe4"} {"msg":"[2]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-02-07T16:28:03.92786358Z","hostname":"ip-10-150-1-55.us-west-2.compute.internal","pid":1160,"":"downstairs","":"downstairs","session_id":"4df6c04e-69a1-4c62-adbd-fc792f51bbe4"} {"msg":"[2]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-02-07T16:28:03.927869349Z","hostname":"ip-10-150-1-55.us-west-2.compute.internal","pid":1160,"":"downstairs","":"downstairs","session_id":"4df6c04e-69a1-4c62-adbd-fc792f51bbe4"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-02-07T16:28:03.927873819Z","hostname":"ip-10-150-1-55.us-west-2.compute.internal","pid":1160,"":"downstairs","":"downstairs","session_id":"4df6c04e-69a1-4c62-adbd-fc792f51bbe4"} {"msg":"Max found gen is 1","v":0,"name":"crucible","level":30,"time":"2024-02-07T16:28:03.927878229Z","hostname":"ip-10-150-1-55.us-west-2.compute.internal","pid":1160,"":"downstairs","":"downstairs","session_id":"4df6c04e-69a1-4c62-adbd-fc792f51bbe4"} {"msg":"Generation requested: 1 >= found:1","v":0,"name":"crucible","level":30,"time":"2024-02-07T16:28:03.927883548Z","hostname":"ip-10-150-1-55.us-west-2.compute.internal","pid":1160,"":"downstairs","":"downstairs","session_id":"4df6c04e-69a1-4c62-adbd-fc792f51bbe4"} {"msg":"Next flush: 1","v":0,"name":"crucible","level":30,"time":"2024-02-07T16:28:03.927887778Z","hostname":"ip-10-150-1-55.us-west-2.compute.internal","pid":1160,"":"downstairs","":"downstairs","session_id":"4df6c04e-69a1-4c62-adbd-fc792f51bbe4"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-02-07T16:28:03.927891978Z","hostname":"ip-10-150-1-55.us-west-2.compute.internal","pid":1160,"":"downstairs","":"downstairs","session_id":"4df6c04e-69a1-4c62-adbd-fc792f51bbe4"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-02-07T16:28:03.927896917Z","hostname":"ip-10-150-1-55.us-west-2.compute.internal","pid":1160,"session_id":"4df6c04e-69a1-4c62-adbd-fc792f51bbe4"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible"Wait for a query_work_queue command to finish before sending IO ,"level":30,"time":"2024-02-07T16:28:03.927903606Z","hostname":"ip-10-150-1-55.us-west-2.compute.internal","pid":1160,"client":"0","":"downstairs","session_id":"4df6c04e-69a1-4c62-adbd-fc792f51bbe4"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-02-07T16:28:03.927908056Z","hostname":"ip-10-150-1-55.us-west-2.compute.internal","pid":1160,"client":"1","":"downstairs","session_id":"4df6c04e-69a1-4c62-adbd-fc792f51bbe4"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-02-07T16:28:03.927920245Z","hostname":"ip-10-150-1-55.us-west-2.compute.internal","pid":1160,"client":"2","":"downstairs","session_id":"4df6c04e-69a1-4c62-adbd-fc792f51bbe4"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-02-07T16:28:03.927927564Z","hostname":"ip-10-150-1-55.us-west-2.compute.internal","pid":1160,"session_id":"4df6c04e-69a1-4c62-adbd-fc792f51bbe4"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-02-07T16:28:03.927931944Z","hostname":"ip-10-150-1-55.us-west-2.compute.internal","pid":1160,"session_id":"4df6c04e-69a1-4c62-adbd-fc792f51bbe4"} {"msg":"5f1dcce7-a8e0-45b6-851d-bf7f3282ccb8 is now active with session: 4df6c04e-69a1-4c62-adbd-fc792f51bbe4","v":0,"name":"crucible","level":30,"time":"2024-02-07T16:28:03.927935844Z","hostname":"ip-10-150-1-55.us-west-2.compute.internal","pid":1160,"session_id":"4df6c04e-69a1-4c62-adbd-fc792f51bbe4"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-02-07T16:28:03.927939473Z","hostname":"ip-10-150-1-55.us-west-2.compute.internal","pid":1160,"session_id":"4df6c04e-69a1-4c62-adbd-fc792f51bbe4"} {"msg":"The guest has finished waiting for activation with:1","v":0,"name":"crucible","level":30,"time":"2024-02-07T16:28:03.927943423Z","hostname":"ip-10-150-1-55.us-west-2.compute.internal","pid":1160} Region: es:16384 ec:640 bs:4096 ts:42949672960 tb:10485760 max_io:256 or 1048576 Perf test TEST SECONDS COUNT DPTH IOPS MEAN P95 P99 MAX ES EC rwrites 20.92 30000 1 1433.86 0.00070 0.00145 0.00195 0.03599 16384 640 rwrites 6.88 30000 1 4363.02 0.00023 0.00049 0.00130 0.06747 16384 640 rreads 5.57 30000 1 5382.03 0.00019 0.00023 0.00029 0.01289 16384 640 rreads 5.52 30000 1 5431.08 0.00018 0.00023 0.00028 0.01302 16384 640 Perf test completed, stop all downstairs Feb 07 16:28:48.898 INFO accepted connection, remote_addr: 127.0.0.1:47496, local_addr: 127.0.0.1:9998 Main task has work to do, go find it got dsc Shutdown Shutdown Feb 07 16:28:48.898Shut it down INFO request completed[8810] Disable keep_running , [8810] Got stop action so:false kr:false latency_us[8820] Disable keep_running :[8820] Got stop action so:false kr:false [8830] Disable keep_running 47[8830] Got stop action so:false kr:false , response_code: 204, uri: /shutdown, method: POST, req_id: f51267f9-f25e-4a3f-aeb5-fa1a605d60ba, remote_addr: 127.0.0.1:47496, local_addr: 127.0.0.1:9998 [8810] Exited with: ExitStatus(unix_wait_status(9)) [8820] Exited with: ExitStatus(unix_wait_status(9)) [8810][0] reports Exit [8820][1] reports Exit [8830] Exited with: ExitStatus(unix_wait_status(9)) [8830][2] reports Exit Create region with ES:16384 EC:1280 BS:4096 Removing existing dsc directory "/tmp/dsc" Removing existing region "/var/tmp/dsc" Creating region directory at: /var/tmp/dsc Creating dsc directory at: /tmp/dsc Downstairs region 0 created at /var/tmp/dsc/8810 in 1.2616823 Downstairs region 1 created at /var/tmp/dsc/8820 in 1.2789001 Downstairs region 2 created at /var/tmp/dsc/8830 in 1.1911438 Region set with 3 regions was created Using existing output directory "/tmp/dsc" start ds: 8810 start ds: 8820 start ds: 8830 start access at:127.0.0.1:9998 Starting downstairs at port 8810 Make output file at "/tmp/dsc/downstairs-8810.txt" [8810][0] reports Starting Starting downstairs at port 8830 Starting downstairs at port 8820 Make output file at "/tmp/dsc/downstairs-8830.txt" Make output file at "/tmp/dsc/downstairs-8820.txt" [8830][2] reports Starting [8820][1] reports Starting Downstairs /var/tmp/dsc/8810 port 8810 PID:Some(1169) [8810][0] reports Running Downstairs /var/tmp/dsc/8820 port 8820 PID:Some(1170) [8820][1] reports Running Feb 07 16:28:53.695 INFO listening, local_addr: 127.0.0.1:9998 Downstairs /var/tmp/dsc/8830 port 8830 PID:Some(1171) [8830][2] reports Running Control access at:127.0.0.1:9998 1171 1170 1169 Feb 07 16:28:58.732 INFO accepted connection, remote_addr: 127.0.0.1:33387, local_addr: 127.0.0.1:9998 Main task has work to do, go find it got dsc DisableRestartAll disable restart on all: 3 Feb 07 16:28:58.733 INFO request completed, latency_us: 116[8810] Disable keep_running , [8820] Disable keep_running response_code: [8830] Disable keep_running 204, uri: /disablerestart/all, method: POST, req_id: 6d213147-0cf9-421b-9105-bd068eee5e46, remote_addr: 127.0.0.1:33387, local_addr: 127.0.0.1:9998 A new loop begins at February 7, 2024 at 04:28:58 PM UTC /var/tmp/bins/crutest perf -g 1 -t 127.0.0.1:8810 -t 127.0.0.1:8820 -t 127.0.0.1:8830 -c 30000 -q --write-loops 2 --read-loops 2 --perf-out /tmp/perf-ES-16384-EC-1280.csv {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-02-07T16:28:58.752570282Z","hostname":"ip-10-150-1-55.us-west-2.compute.internal","pid":1179} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"24836e2c4d76be332fe33bc8f22f62e25ba0a96e\",\n git_commit_timestamp: \"2024-02-07T16:05:39.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.75.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"82e1608dfa6e0b5569232559e3d385fea5a93112\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: false,\n opt_level: 3,\n}","v":0,"name":"crucible","level":30,"time":"2024-02-07T16:28:58.75271314Z","hostname":"ip-10-150-1-55.us-west-2.compute.internal","pid":1179} {"msg":"Upstairs <-> Downstairs Message Version: 5","v":0,"name":"crucible","level":30,"time":"2024-02-07T16:28:58.752722239Z","hostname":"ip-10-150-1-55.us-west-2.compute.internal","pid":1179} Crucible runtime is spawned {"msg":"Crucible 7248620c-f5cf-4f15-b784-caded6f3b574 has session id: 9c2fdd86-af20-4906-8572-67080448a597","v":0,"name":"crucible","level":30,"time":"2024-02-07T16:28:58.752741867Z","hostname":"ip-10-150-1-55.us-west-2.compute.internal","pid":1179,"session_id":"9c2fdd86-af20-4906-8572-67080448a597"} {"msg":"Upstairs opts: Upstairs UUID: 7248620c-f5cf-4f15-b784-caded6f3b574, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-02-07T16:28:58.752763475Z","hostname":"ip-10-150-1-55.us-west-2.compute.internal","pid":1179,"session_id":"9c2fdd86-af20-4906-8572-67080448a597"} {"msg":"Crucible stats registered with UUID: 7248620c-f5cf-4f15-b784-caded6f3b574","v":0,"name":"crucible","level":30,"time":"2024-02-07T16:28:58.752771165Z","hostname":"ip-10-150-1-55.us-west-2.compute.internal","pid":1179,"session_id":"9c2fdd86-af20-4906-8572-67080448a597"} {"msg":"The guest has requested activation with gen:1","v":0,"name":"crucible","level":30,"time":"2024-02-07T16:28:58.752777694Z","hostname":"ip-10-150-1-55.us-west-2.compute.internal","pid":1179} {"msg":"7248620c-f5cf-4f15-b784-caded6f3b574 active request set","v":0,"name":"crucible","level":30,"time":"2024-02-07T16:28:58.752830299Z","hostname":"ip-10-150-1-55.us-west-2.compute.internal","pid":1179,"session_id":"9c2fdd86-af20-4906-8572-67080448a597"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-02-07T16:28:58.752851527Z","hostname":"ip-10-150-1-55.us-west-2.compute.internal","pid":1179,"client":"0","":"downstairs","session_id":"9c2fdd86-af20-4906-8572-67080448a597"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-02-07T16:28:58.752867476Z","hostname":"ip-10-150-1-55.us-west-2.compute.internal","pid":1179,"client":"0","":"downstairs","session_id":"9c2fdd86-af20-4906-8572-67080448a597"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-02-07T16:28:58.752877325Z","hostname":"ip-10-150-1-55.us-west-2.compute.internal","pid":1179,"client":"1","":"downstairs","session_id":"9c2fdd86-af20-4906-8572-67080448a597"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-02-07T16:28:58.752886555Z","hostname":"ip-10-150-1-55.us-west-2.compute.internal","pid":1179,"client":"1","":"downstairs","session_id":"9c2fdd86-af20-4906-8572-67080448a597"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-02-07T16:28:58.752895934Z","hostname":"ip-10-150-1-55.us-west-2.compute.internal","pid":1179,"client":"2","":"downstairs","session_id":"9c2fdd86-af20-4906-8572-67080448a597"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-02-07T16:28:58.752905953Z","hostname":"ip-10-150-1-55.us-west-2.compute.internal","pid":1179,"client":"2","":"downstairs","session_id":"9c2fdd86-af20-4906-8572-67080448a597"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-02-07T16:28:58.752916282Z","hostname":"ip-10-150-1-55.us-west-2.compute.internal","pid":1179,"":"io task","client":"2","":"downstairs","session_id":"9c2fdd86-af20-4906-8572-67080448a597"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-07T16:28:58.752943979Z","hostname":"ip-10-150-1-55.us-west-2.compute.internal","pid":1179,"":"io task","client":"0","":"downstairs","session_id":"9c2fdd86-af20-4906-8572-67080448a597"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-02-07T16:28:58.752953389Z","hostname":"ip-10-150-1-55.us-west-2.compute.internal","pid":1179,"":"io task","client":"1","":"downstairs","session_id":"9c2fdd86-af20-4906-8572-67080448a597"} {"msg":"ds_connection connected","v":0,"name":"crucible","level":30,"time":"2024-02-07T16:28:58.752991215Z","hostname":"ip-10-150-1-55.us-west-2.compute.internal","pid":1179,"":"io task","client":"0","":"downstairs","session_id":"9c2fdd86-af20-4906-8572-67080448a597"} {"msg":"ds_connection connected","v":0,"name":"crucible","level":30,"time":"2024-02-07T16:28:58.753014883Z","hostname":"ip-10-150-1-55.us-west-2.compute.internal","pid":1179,"":"io task","client":"2","":"downstairs","session_id":"9c2fdd86-af20-4906-8572-67080448a597"} {"msg":"ds_connection connected","v":0,"name":"crucible","level":30,"time":"2024-02-07T16:28:58.753024772Z","hostname":"ip-10-150-1-55.us-west-2.compute.internal","pid":1179,"":"io task","client":"1","":"downstairs","session_id":"9c2fdd86-af20-4906-8572-67080448a597"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-02-07T16:28:58.753316767Z","hostname":"ip-10-150-1-55.us-west-2.compute.internal","pid":1179,"client":"1","":"downstairs","session_id":"9c2fdd86-af20-4906-8572-67080448a597"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-02-07T16:28:58.753341565Z","hostname":"ip-10-150-1-55.us-west-2.compute.internal","pid":1179,"client":"1","":"downstairs","session_id":"9c2fdd86-af20-4906-8572-67080448a597"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-02-07T16:28:58.753352624Z","hostname":"ip-10-150-1-55.us-west-2.compute.internal","pid":1179,"client":"2","":"downstairs","session_id":"9c2fdd86-af20-4906-8572-67080448a597"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-02-07T16:28:58.753362003Z","hostname":"ip-10-150-1-55.us-west-2.compute.internal","pid":1179,"client":"2","":"downstairs","session_id":"9c2fdd86-af20-4906-8572-67080448a597"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-02-07T16:28:58.753371532Z","hostname":"ip-10-150-1-55.us-west-2.compute.internal","pid":1179,"client":"0","":"downstairs","session_id":"9c2fdd86-af20-4906-8572-67080448a597"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-02-07T16:28:58.753380281Z","hostname":"ip-10-150-1-55.us-west-2.compute.internal","pid":1179,"client":"0","":"downstairs","session_id":"9c2fdd86-af20-4906-8572-67080448a597"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-02-07T16:28:58.753758938Z","hostname":"ip-10-150-1-55.us-west-2.compute.internal","pid":1179,"client":"1","":"downstairs","session_id":"9c2fdd86-af20-4906-8572-67080448a597"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-02-07T16:28:58.753780286Z","hostname":"ip-10-150-1-55.us-west-2.compute.internal","pid":1179,"client":"2","":"downstairs","session_id":"9c2fdd86-af20-4906-8572-67080448a597"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-02-07T16:28:58.753788246Z","hostname":"ip-10-150-1-55.us-west-2.compute.internal","pid":1179,"client":"0","":"downstairs","session_id":"9c2fdd86-af20-4906-8572-67080448a597"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-02-07T16:28:58.759894122Z","hostname":"ip-10-150-1-55.us-west-2.compute.internal","pid":1179,"client":"1","":"downstairs","session_id":"9c2fdd86-af20-4906-8572-67080448a597"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-02-07T16:28:58.75991894Z","hostname":"ip-10-150-1-55.us-west-2.compute.internal","pid":1179,"client":"1","":"downstairs","session_id":"9c2fdd86-af20-4906-8572-67080448a597"} {"msg":"Waiting for true more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-02-07T16:28:58.759927649Z","hostname":"ip-10-150-1-55.us-west-2.compute.internal","pid":1179,"session_id":"9c2fdd86-af20-4906-8572-67080448a597"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-02-07T16:28:58.761485693Z","hostname":"ip-10-150-1-55.us-west-2.compute.internal","pid":1179,"client":"2","":"downstairs","session_id":"9c2fdd86-af20-4906-8572-67080448a597"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-02-07T16:28:58.76151769Z","hostname":"ip-10-150-1-55.us-west-2.compute.internal","pid":1179,"client":"2","":"downstairs","session_id":"9c2fdd86-af20-4906-8572-67080448a597"} {"msg":"Waiting for true more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-02-07T16:28:58.761528319Z","hostname":"ip-10-150-1-55.us-west-2.compute.internal","pid":1179,"session_id":"9c2fdd86-af20-4906-8572-67080448a597"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-02-07T16:28:58.761594774Z","hostname":"ip-10-150-1-55.us-west-2.compute.internal","pid":1179,"client":"0","":"downstairs","session_id":"9c2fdd86-af20-4906-8572-67080448a597"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-02-07T16:28:58.761610652Z","hostname":"ip-10-150-1-55.us-west-2.compute.internal","pid":1179,"client":"0","":"downstairs","session_id":"9c2fdd86-af20-4906-8572-67080448a597"} {"msg":"[0]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-02-07T16:28:58.761616422Z","hostname":"ip-10-150-1-55.us-west-2.compute.internal","pid":1179,"":"downstairs","":"downstairs","session_id":"9c2fdd86-af20-4906-8572-67080448a597"} {"msg":"[0]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-02-07T16:28:58.761622031Z","hostname":"ip-10-150-1-55.us-west-2.compute.internal","pid":1179,"":"downstairs","":"downstairs","session_id":"9c2fdd86-af20-4906-8572-67080448a597"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-02-07T16:28:58.761627711Z","hostname":"ip-10-150-1-55.us-west-2.compute.internal","pid":1179,"":"downstairs","":"downstairs","session_id":"9c2fdd86-af20-4906-8572-67080448a597"} {"msg":"[1]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-02-07T16:28:58.76163324Z","hostname":"ip-10-150-1-55.us-west-2.compute.internal","pid":1179,"":"downstairs","":"downstairs","session_id":"9c2fdd86-af20-4906-8572-67080448a597"} {"msg":"[1]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-02-07T16:28:58.76163857Z","hostname":"ip-10-150-1-55.us-west-2.compute.internal","pid":1179,"":"downstairs","":"downstairs","session_id":"9c2fdd86-af20-4906-8572-67080448a597"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-02-07T16:28:58.761644599Z","hostname":"ip-10-150-1-55.us-west-2.compute.internal","pid":1179,"":"downstairs","":"downstairs","session_id":"9c2fdd86-af20-4906-8572-67080448a597"} {"msg":"[2]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-02-07T16:28:58.761650239Z","hostname":"ip-10-150-1-55.us-west-2.compute.internal","pid":1179,"":"downstairs","":"downstairs","session_id":"9c2fdd86-af20-4906-8572-67080448a597"} {"msg":"[2]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-02-07T16:28:58.761655338Z","hostname":"ip-10-150-1-55.us-west-2.compute.internal","pid":1179,"":"downstairs","":"downstairs","session_id":"9c2fdd86-af20-4906-8572-67080448a597"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-02-07T16:28:58.761663868Z","hostname":"ip-10-150-1-55.us-west-2.compute.internal","pid":1179,"":"downstairs","":"downstairs","session_id":"9c2fdd86-af20-4906-8572-67080448a597"} {"msg":"Max found gen is 1","v":0,"name":"crucible","level":30,"time":"2024-02-07T16:28:58.761683026Z","hostname":"ip-10-150-1-55.us-west-2.compute.internal","pid":1179,"":"downstairs","":"downstairs","session_id":"9c2fdd86-af20-4906-8572-67080448a597"} {"msg":"Generation requested: 1 >= found:1","v":0,"name":"crucible","level":30,"time":"2024-02-07T16:28:58.761691835Z","hostname":"ip-10-150-1-55.us-west-2.compute.internal","pid":1179,"":"downstairs","":"downstairs","session_id":"9c2fdd86-af20-4906-8572-67080448a597"} {"msg":"Next flush: 1","v":0,"name":"crucible","level":30,"time":"2024-02-07T16:28:58.761697274Z","hostname":"ip-10-150-1-55.us-west-2.compute.internal","pid":1179,"":"downstairs","":"downstairs","session_id":"9c2fdd86-af20-4906-8572-67080448a597"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-02-07T16:28:58.761703014Z","hostname":"ip-10-150-1-55.us-west-2.compute.internal","pid":1179,"":"downstairs","":"downstairs","session_id":"9c2fdd86-af20-4906-8572-67080448a597"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-02-07T16:28:58.761709293Z","hostname":"ip-10-150-1-55.us-west-2.compute.internal","pid":1179,"session_id":"9c2fdd86-af20-4906-8572-67080448a597"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-02-07T16:28:58.761713933Z","hostname":"ip-10-150-1-55.us-west-2.compute.internal","pid":1179,"client":"0","":"downstairs","session_id":"9c2fdd86-af20-4906-8572-67080448a597"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-02-07T16:28:58.761719013Z","hostname":"ip-10-150-1-55.us-west-2.compute.internal","pid":1179,"client":"1","":"downstairs","session_id":"Wait for a query_work_queue command to finish before sending IO 9c2fdd86-af20-4906-8572-67080448a597"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-02-07T16:28:58.761727172Z","hostname":"ip-10-150-1-55.us-west-2.compute.internal","pid":1179,"client":"2","":"downstairs","session_id":"9c2fdd86-af20-4906-8572-67080448a597"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-02-07T16:28:58.761732851Z","hostname":"ip-10-150-1-55.us-west-2.compute.internal","pid":1179,"session_id":"9c2fdd86-af20-4906-8572-67080448a597"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-02-07T16:28:58.761738431Z","hostname":"ip-10-150-1-55.us-west-2.compute.internal","pid":1179,"session_id":"9c2fdd86-af20-4906-8572-67080448a597"} {"msg":"7248620c-f5cf-4f15-b784-caded6f3b574 is now active with session: 9c2fdd86-af20-4906-8572-67080448a597","v":0,"name":"crucible","level":30,"time":"2024-02-07T16:28:58.761743121Z","hostname":"ip-10-150-1-55.us-west-2.compute.internal","pid":1179,"session_id":"9c2fdd86-af20-4906-8572-67080448a597"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-02-07T16:28:58.76174832Z","hostname":"ip-10-150-1-55.us-west-2.compute.internal","pid":1179,"session_id":"9c2fdd86-af20-4906-8572-67080448a597"} {"msg":"The guest has finished waiting for activation with:1","v":0,"name":"crucible","level":30,"time":"2024-02-07T16:28:58.76175316Z","hostname":"ip-10-150-1-55.us-west-2.compute.internal","pid":1179} Region: es:16384 ec:1280 bs:4096 ts:85899345920 tb:20971520 max_io:256 or 1048576 Perf test TEST SECONDS COUNT DPTH IOPS MEAN P95 P99 MAX ES EC rwrites 7.96 30000 1 3767.42 0.00027 0.00059 0.00135 0.06537 16384 1280 rwrites 8.00 30000 1 3750.52 0.00027 0.00059 0.00114 0.05919 16384 1280 rreads 5.64 30000 1 5321.60 0.00019 0.00025 0.00030 0.01597 16384 1280 rreads 5.62 30000 1 5338.86 0.00019 0.00025 0.00030 0.01597 16384 1280 Perf test completed, stop all downstairs Feb 07 16:29:34.894 INFO accepted connection, remote_addr: 127.0.0.1:52877, local_addr: 127.0.0.1:9998 Main task has work to do, go find it got dsc Shutdown Shutdown [8810] Disable keep_running Feb 07 16:29:34.894 INFOShut it down request completed, latency_us: 56, response_code: 204, uri: /shutdown, method: POST, req_id: 4a9df6dd-e6f0-45fe-92f6-c0995627eaac, remote_addr[8810] Got stop action so:false kr:false : 127.0.0.1:[8820] Disable keep_running 52877, local_addr: 127.0.0.1:9998 [8820] Got stop action so:false kr:false [8830] Disable keep_running [8830] Got stop action so:false kr:false [8820] Exited with: ExitStatus(unix_wait_status(9)) [8810] Exited with: ExitStatus(unix_wait_status(9)) [8830] Exited with: ExitStatus(unix_wait_status(9)) [8820][1] reports Exit [8810][0] reports Exit [8830][2] reports Exit TEST SECONDS COUNT DPTH IOPS MEAN P95 P99 MAX ES EC rwrites 20.92 30000 1 1433.86 0.00070 0.00145 0.00195 0.03599 16384 640 rwrites 6.88 30000 1 4363.02 0.00023 0.00049 0.00130 0.06747 16384 640 rwrites 7.96 30000 1 3767.42 0.00027 0.00059 0.00135 0.06537 16384 1280 rwrites 8.00 30000 1 3750.52 0.00027 0.00059 0.00114 0.05919 16384 1280 TEST SECONDS COUNT DPTH IOPS MEAN P95 P99 MAX ES EC rreads 5.57 30000 1 5382.03 0.00019 0.00023 0.00029 0.01289 16384 640 rreads 5.52 30000 1 5431.08 0.00018 0.00023 0.00028 0.01302 16384 640 rreads 5.64 30000 1 5321.60 0.00019 0.00025 0.00030 0.01597 16384 1280 rreads 5.62 30000 1 5338.86 0.00019 0.00025 0.00030 0.01597 16384 1280 Perf test finished on February 7, 2024 at 04:29:36 PM UTC