/input/rbuild/work Create region with ES:16384 EC:640 BS:4096 Creating region directory at: /var/tmp/dsc Creating dsc directory at: /tmp/dsc Downstairs region 0 created at /var/tmp/dsc/8810 in 1.0964825 Downstairs region 1 created at /var/tmp/dsc/8820 in 0.83165824 Downstairs region 2 created at /var/tmp/dsc/8830 in 0.7344011 Region set with 3 regions was created Using existing output directory "/tmp/dsc" start ds: 8810 start ds: 8820 start ds: 8830 start access at:127.0.0.1:9998 Starting downstairs at port 8810 Starting downstairs at port 8820 Make output file at "/tmp/dsc/downstairs-8810.txt" Starting downstairs at port 8830 Make output file at "/tmp/dsc/downstairs-8820.txt" Make output file at "/tmp/dsc/downstairs-8830.txt" [8810][0] reports Starting [8820][1] reports Starting [8830][2] reports Starting Downstairs /var/tmp/dsc/8810 port 8810 PID:Some(1150) [8810][0] reports Running Downstairs /var/tmp/dsc/8820 port 8820 PID:Some(1151) [8820][1] reports Running Downstairs /var/tmp/dsc/8830 port 8830 PID:Some(1152) [8830][2] reports Running Feb 08 06:41:00.976 INFO listening, local_addr: 127.0.0.1:9998 Control access at:127.0.0.1:9998 1150 1151 1152 Feb 08 06:41:06.013 INFO accepted connection, remote_addr: 127.0.0.1:41064, local_addr: 127.0.0.1:9998 Main task has work to do, go find it got dsc DisableRestartAll disable restart on all: 3 Feb 08 06:41:06.014 [8810] Disable keep_running INFO[8820] Disable keep_running request completed, latency_us: 141, response_code: 204, uri:[8830] Disable keep_running /disablerestart/all, method: POST, req_id: a5f4e617-f8d7-4724-8599-0de1b7d6fb50, remote_addr: 127.0.0.1:41064, local_addr: 127.0.0.1:9998 A new loop begins at February 8, 2024 at 06:41:06 AM UTC /var/tmp/bins/crutest perf -g 1 -t 127.0.0.1:8810 -t 127.0.0.1:8820 -t 127.0.0.1:8830 -c 30000 -q --write-loops 2 --read-loops 2 --perf-out /tmp/perf-ES-16384-EC-640.csv {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-02-08T06:41:06.035313724Z","hostname":"ip-10-150-1-144.us-west-2.compute.internal","pid":1160} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"ebef7f4917e579ad4649dc0478073aaafe487733\",\n git_commit_timestamp: \"2024-02-08T06:19:45.000000000Z\",\n git_branch: \"alan/downstairs-clone\",\n rustc_semver: \"1.75.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"82e1608dfa6e0b5569232559e3d385fea5a93112\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: false,\n opt_level: 3,\n}","v":0,"name":"crucible","level":30,"time":"2024-02-08T06:41:06.035479683Z","hostname":"ip-10-150-1-144.us-west-2.compute.internal","pid":1160} {"msg":"Upstairs <-> Downstairs Message Version: 5","v":0,"name":"crucible","level":30,"time":"2024-02-08T06:41:06.035486732Z","hostname":"ip-10-150-1-144.us-west-2.compute.internal","pid":1160} {"msg":"Crucible 8d570b26-0c2b-459c-8e05-53b63cc86f26 has session id: 74a5d181-442c-4791-b3eb-f18780922e37","v":0,"name":"crucible","level":30,"time":"2024-02-08T06:41:06.035533143Z","hostname":"ip-10-150-1-144.us-west-2.compute.internal","pid":1160,"session_id":"74a5d181-442c-4791-b3eb-f18780922e37"} {"msg":"Upstairs opts: Upstairs UUID: 8d570b26-0c2b-459c-8e05-53b63cc86f26, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-02-08T06:41:06.035570727Z","hostname":"ip-10-150-1-144.us-west-2.compute.internal","pid":1160,"session_id":"74a5d181-442c-4791-b3eb-f18780922e37"} {"msg":"Crucible stats registered with UUID: 8d570b26-0c2b-459c-8e05-53b63cc86f26","v":0,"name":"crucible","level":30,"time":"2024-02-08T06:41:06.035576515Z","hostname":"ip-10-150-1-144.us-west-2.compute.internal","pid":1160,"session_id":"74a5d181-442c-4791-b3eb-f18780922e37"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:1","v":0,"name":"crucible","level":30,"time":"2024-02-08T06:41:06.035711421Z","hostname":"ip-10-150-1-144.us-west-2.compute.internal","pid":1160} {"msg":"8d570b26-0c2b-459c-8e05-53b63cc86f26 active request set","v":0,"name":"crucible","level":30,"time":"2024-02-08T06:41:06.03587817Z","hostname":"ip-10-150-1-144.us-west-2.compute.internal","pid":1160,"session_id":"74a5d181-442c-4791-b3eb-f18780922e37"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-02-08T06:41:06.035895807Z","hostname":"ip-10-150-1-144.us-west-2.compute.internal","pid":1160,"client":"0","":"downstairs","session_id":"74a5d181-442c-4791-b3eb-f18780922e37"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-02-08T06:41:06.035902096Z","hostname":"ip-10-150-1-144.us-west-2.compute.internal","pid":1160,"client":"0","":"downstairs","session_id":"74a5d181-442c-4791-b3eb-f18780922e37"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-02-08T06:41:06.035907305Z","hostname":"ip-10-150-1-144.us-west-2.compute.internal","pid":1160,"client":"1","":"downstairs","session_id":"74a5d181-442c-4791-b3eb-f18780922e37"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-02-08T06:41:06.035912164Z","hostname":"ip-10-150-1-144.us-west-2.compute.internal","pid":1160,"client":"1","":"downstairs","session_id":"74a5d181-442c-4791-b3eb-f18780922e37"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-02-08T06:41:06.035917283Z","hostname":"ip-10-150-1-144.us-west-2.compute.internal","pid":1160,"client":"2","":"downstairs","session_id":"74a5d181-442c-4791-b3eb-f18780922e37"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-02-08T06:41:06.035921932Z","hostname":"ip-10-150-1-144.us-west-2.compute.internal","pid":1160,"client":"2","":"downstairs","session_id":"74a5d181-442c-4791-b3eb-f18780922e37"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-02-08T06:41:06.035959215Z","hostname":"ip-10-150-1-144.us-west-2.compute.internal","pid":1160,"":"io task","client":"2","":"downstairs","session_id":"74a5d181-442c-4791-b3eb-f18780922e37"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-08T06:41:06.036006197Z","hostname":"ip-10-150-1-144.us-west-2.compute.internal","pid":1160,"":"io task","client":"0","":"downstairs","session_id":"74a5d181-442c-4791-b3eb-f18780922e37"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-02-08T06:41:06.036016045Z","hostname":"ip-10-150-1-144.us-west-2.compute.internal","pid":1160,"":"io task","client":"1","":"downstairs","session_id":"74a5d181-442c-4791-b3eb-f18780922e37"} {"msg":"ds_connection connected","v":0,"name":"crucible","level":30,"time":"2024-02-08T06:41:06.036069725Z","hostname":"ip-10-150-1-144.us-west-2.compute.internal","pid":1160,"":"io task","client":"1","":"downstairs","session_id":"74a5d181-442c-4791-b3eb-f18780922e37"} {"msg":"ds_connection connected","v":0,"name":"crucible","level":30,"time":"2024-02-08T06:41:06.036085032Z","hostname":"ip-10-150-1-144.us-west-2.compute.internal","pid":1160,"":"io task","client":"0","":"downstairs","session_id":"74a5d181-442c-4791-b3eb-f18780922e37"} {"msg":"ds_connection connected","v":0,"name":"crucible","level":30,"time":"2024-02-08T06:41:06.036118936Z","hostname":"ip-10-150-1-144.us-west-2.compute.internal","pid":1160,"":"io task","client":"2","":"downstairs","session_id":"74a5d181-442c-4791-b3eb-f18780922e37"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-02-08T06:41:06.036608336Z","hostname":"ip-10-150-1-144.us-west-2.compute.internal","pid":1160,"client":"1","":"downstairs","session_id":"74a5d181-442c-4791-b3eb-f18780922e37"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-02-08T06:41:06.036626103Z","hostname":"ip-10-150-1-144.us-west-2.compute.internal","pid":1160,"client":"1","":"downstairs","session_id":"74a5d181-442c-4791-b3eb-f18780922e37"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-02-08T06:41:06.036633001Z","hostname":"ip-10-150-1-144.us-west-2.compute.internal","pid":1160,"client":"2","":"downstairs","session_id":"74a5d181-442c-4791-b3eb-f18780922e37"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-02-08T06:41:06.03664083Z","hostname":"ip-10-150-1-144.us-west-2.compute.internal","pid":1160,"client":"2","":"downstairs","session_id":"74a5d181-442c-4791-b3eb-f18780922e37"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-02-08T06:41:06.036645709Z","hostname":"ip-10-150-1-144.us-west-2.compute.internal","pid":1160,"client":"0","":"downstairs","session_id":"74a5d181-442c-4791-b3eb-f18780922e37"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-02-08T06:41:06.036650258Z","hostname":"ip-10-150-1-144.us-west-2.compute.internal","pid":1160,"client":"0","":"downstairs","session_id":"74a5d181-442c-4791-b3eb-f18780922e37"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-02-08T06:41:06.036936916Z","hostname":"ip-10-150-1-144.us-west-2.compute.internal","pid":1160,"client":"1","":"downstairs","session_id":"74a5d181-442c-4791-b3eb-f18780922e37"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-02-08T06:41:06.036953183Z","hostname":"ip-10-150-1-144.us-west-2.compute.internal","pid":1160,"client":"2","":"downstairs","session_id":"74a5d181-442c-4791-b3eb-f18780922e37"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-02-08T06:41:06.036959652Z","hostname":"ip-10-150-1-144.us-west-2.compute.internal","pid":1160,"client":"0","":"downstairs","session_id":"74a5d181-442c-4791-b3eb-f18780922e37"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-02-08T06:41:06.039732723Z","hostname":"ip-10-150-1-144.us-west-2.compute.internal","pid":1160,"client":"1","":"downstairs","session_id":"74a5d181-442c-4791-b3eb-f18780922e37"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-02-08T06:41:06.039749769Z","hostname":"ip-10-150-1-144.us-west-2.compute.internal","pid":1160,"client":"1","":"downstairs","session_id":"74a5d181-442c-4791-b3eb-f18780922e37"} {"msg":"Waiting for true more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-02-08T06:41:06.039754879Z","hostname":"ip-10-150-1-144.us-west-2.compute.internal","pid":1160,"session_id":"74a5d181-442c-4791-b3eb-f18780922e37"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-02-08T06:41:06.040416327Z","hostname":"ip-10-150-1-144.us-west-2.compute.internal","pid":1160,"client":"0","":"downstairs","session_id":"74a5d181-442c-4791-b3eb-f18780922e37"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-02-08T06:41:06.040428945Z","hostname":"ip-10-150-1-144.us-west-2.compute.internal","pid":1160,"client":"0","":"downstairs","session_id":"74a5d181-442c-4791-b3eb-f18780922e37"} {"msg":"Waiting for true more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-02-08T06:41:06.040433744Z","hostname":"ip-10-150-1-144.us-west-2.compute.internal","pid":1160,"session_id":"74a5d181-442c-4791-b3eb-f18780922e37"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-02-08T06:41:06.040668001Z","hostname":"ip-10-150-1-144.us-west-2.compute.internal","pid":1160,"client":"2","":"downstairs","session_id":"74a5d181-442c-4791-b3eb-f18780922e37"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-02-08T06:41:06.040687057Z","hostname":"ip-10-150-1-144.us-west-2.compute.internal","pid":1160,"client":"2","":"downstairs","session_id":"74a5d181-442c-4791-b3eb-f18780922e37"} {"msg":"[0]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-02-08T06:41:06.040704904Z","hostname":"ip-10-150-1-144.us-west-2.compute.internal","pid":1160,"":"downstairs","":"downstairs","session_id":"74a5d181-442c-4791-b3eb-f18780922e37"} {"msg":"[0]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-02-08T06:41:06.040716532Z","hostname":"ip-10-150-1-144.us-west-2.compute.internal","pid":1160,"":"downstairs","":"downstairs","session_id":"74a5d181-442c-4791-b3eb-f18780922e37"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-02-08T06:41:06.040721411Z","hostname":"ip-10-150-1-144.us-west-2.compute.internal","pid":1160,"":"downstairs","":"downstairs","session_id":"74a5d181-442c-4791-b3eb-f18780922e37"} {"msg":"[1]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-02-08T06:41:06.04072589Z","hostname":"ip-10-150-1-144.us-west-2.compute.internal","pid":1160,"":"downstairs","":"downstairs","session_id":"74a5d181-442c-4791-b3eb-f18780922e37"} {"msg":"[1]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-02-08T06:41:06.04073041Z","hostname":"ip-10-150-1-144.us-west-2.compute.internal","pid":1160,"":"downstairs","":"downstairs","session_id":"74a5d181-442c-4791-b3eb-f18780922e37"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-02-08T06:41:06.040735539Z","hostname":"ip-10-150-1-144.us-west-2.compute.internal","pid":1160,"":"downstairs","":"downstairs","session_id":"74a5d181-442c-4791-b3eb-f18780922e37"} {"msg":"[2]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-02-08T06:41:06.040739798Z","hostname":"ip-10-150-1-144.us-west-2.compute.internal","pid":1160,"":"downstairs","":"downstairs","session_id":"74a5d181-442c-4791-b3eb-f18780922e37"} {"msg":"[2]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-02-08T06:41:06.040744407Z","hostname":"ip-10-150-1-144.us-west-2.compute.internal","pid":1160,"":"downstairs","":"downstairs","session_id":"74a5d181-442c-4791-b3eb-f18780922e37"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-02-08T06:41:06.040750136Z","hostname":"ip-10-150-1-144.us-west-2.compute.internal","pid":1160,"":"downstairs","":"downstairs","session_id":"74a5d181-442c-4791-b3eb-f18780922e37"} {"msg":"Max found gen is 1","v":0,"name":"crucible","level":30,"time":"2024-02-08T06:41:06.040755175Z","hostname":"ip-10-150-1-144.us-west-2.compute.internal","pid":1160,"":"downstairs","":"downstairs","session_id":"74a5d181-442c-4791-b3eb-f18780922e37"} {"msg":"Generation requested: 1 >= found:1","v":0,"name":"crucible","level":30,"time":"2024-02-08T06:41:06.040760564Z","hostname":"ip-10-150-1-144.us-west-2.compute.internal","pid":1160,"":"downstairs","":"downstairs","session_id":"74a5d181-442c-4791-b3eb-f18780922e37"} {"msg":"Next flush: 1","v":0,"name":"crucible","level":30,"time":"2024-02-08T06:41:06.040764993Z","hostname":"ip-10-150-1-144.us-west-2.compute.internal","pid":1160,"":"downstairs","":"downstairs","session_id":"74a5d181-442c-4791-b3eb-f18780922e37"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-02-08T06:41:06.040769442Z","hostname":"ip-10-150-1-144.us-west-2.compute.internal","pid":1160,"":"downstairs","":"downstairs","session_id":"74a5d181-442c-4791-b3eb-f18780922e37"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-02-08T06:41:06.040774481Z","hostname":"ip-10-150-1-144.us-west-2.compute.internal","pid":1160,"session_id":"74a5d181-442c-4791-b3eb-f18780922e37"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-02-08T06:41:06.040778521Z","hostname":"ip-10-150-1-144.us-west-2.compute.internal","pid":1160,"client":"0","":"downstairs","session_id":"74a5d181-442c-4791-b3eb-f18780922e37"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-02-08T06:41:06.0407829Z","hostname":"ip-10-150-1-144.us-west-2.compute.internal","pid":1160,"client":"1","":"downstairs","session_id":"74a5d181-442c-4791-b3eb-f18780922e37"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2024-02-08T06:41:06.040787329Z","hostname":"ip-10-150-1-144.us-west-2.compute.internal","pid":1160,"client":"2","":"downstairs","session_id":"74a5d181-442c-4791-b3eb-f18780922e37"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-02-08T06:41:06.040795748Z","hostname":"ip-10-150-1-144.us-west-2.compute.internal","pid":1160,"session_id":"74a5d181-442c-4791-b3eb-f18780922e37"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-02-08T06:41:06.040800217Z","hostname":"ip-10-150-1-144.us-west-2.compute.internal","pid":1160,"session_id":"74a5d181-442c-4791-b3eb-f18780922e37"} {"msg":"8d570b26-0c2b-459c-8e05-53b63cc86f26 is now active with session: 74a5d181-442c-4791-b3eb-f18780922e37","v":0,"name":"crucible","level":30,"time":"2024-02-08T06:41:06.040804236Z","hostname":"ip-10-150-1-144.us-west-2.compute.internal","pid":1160,"session_id":"74a5d181-442c-4791-b3eb-f18780922e37"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-02-08T06:41:06.040808535Z","hostname":"ip-10-150-1-144.us-west-2.compute.internal","pid":1160,"session_id":"74a5d181-442c-4791-b3eb-f18780922e37"} {"msg":"The guest has finished waiting for activation with:1","v":0,"name":"crucible","level":30,"time":"2024-02-08T06:41:06.040812425Z","hostname":"ip-10-150-1-144.us-west-2.compute.internal","pid":1160} Region: es:16384 ec:640 bs:4096 ts:42949672960 tb:10485760 max_io:256 or 1048576 Perf test TEST SECONDS COUNT DPTH IOPS MEAN P95 P99 MAX ES EC rwrites 23.41 30000 1 1281.29 0.00078 0.00134 0.00203 0.02881 16384 640 rwrites 9.53 30000 1 3146.40 0.00032 0.00109 0.00177 0.08593 16384 640 rreads 5.66 30000 1 5298.08 0.00019 0.00026 0.00030 0.01252 16384 640 rreads 5.69 30000 1 5271.51 0.00019 0.00026 0.00030 0.01282 16384 640 Perf test completed, stop all downstairs Feb 08 06:42:02.452 INFO accepted connection, remote_addr: 127.0.0.1:56160, local_addr: 127.0.0.1:9998 Main task has work to do, go find it got dsc Shutdown Shutdown Feb 08 06:42:02.453 INFOShut it down request completed, latency_us[8810] Disable keep_running :[8810] Got stop action so:false kr:false [8830] Disable keep_running 48[8830] Got stop action so:false kr:false , response_code: 204, uri: /shutdown, method: POST, [8820] Disable keep_running req_id[8820] Got stop action so:false kr:false : 29494bbd-c941-45de-84bb-6364ab13f3a2, remote_addr: 127.0.0.1:56160, local_addr: 127.0.0.1:9998 [8830] Exited with: ExitStatus(unix_wait_status(9)) [8810] Exited with: ExitStatus(unix_wait_status(9)) [8830][2] reports Exit [8810][0] reports Exit [8820] Exited with: ExitStatus(unix_wait_status(9)) [8820][1] reports Exit Create region with ES:16384 EC:1280 BS:4096 Removing existing dsc directory "/tmp/dsc" Removing existing region "/var/tmp/dsc" Creating region directory at: /var/tmp/dsc Creating dsc directory at: /tmp/dsc Downstairs region 0 created at /var/tmp/dsc/8810 in 1.0672753 Downstairs region 1 created at /var/tmp/dsc/8820 in 1.1167414 Downstairs region 2 created at /var/tmp/dsc/8830 in 1.0574695 Region set with 3 regions was created Using existing output directory "/tmp/dsc" start ds: 8810 start ds: 8820 start ds: 8830 start access at:127.0.0.1:9998 Starting downstairs at port 8810 Make output file at "/tmp/dsc/downstairs-8810.txt" Starting downstairs at port 8820 Make output file at "/tmp/dsc/downstairs-8820.txt" [8810][0] reports Starting Starting downstairs at port 8830 [8820][1] reports Starting Make output file at "/tmp/dsc/downstairs-8830.txt" [8830][2] reports Starting Downstairs /var/tmp/dsc/8810 port 8810 PID:Some(1169) [8810][0] reports Running Downstairs /var/tmp/dsc/8830 port 8830 PID:Some(1170) [8830][2] reports Running Downstairs /var/tmp/dsc/8820 port 8820 PID:Some(1171) Feb 08 06:42:06.755 INFO listening, local_addr: 127.0.0.1:9998 [8820][1] reports Running Control access at:127.0.0.1:9998 1169 1170 1171 Feb 08 06:42:11.793 INFO accepted connection, remote_addr: 127.0.0.1:46217, local_addr: 127.0.0.1:9998 Main task has work to do, go find it got dsc DisableRestartAll disable restart on all: 3 Feb 08 06:42:11.794 INFO request completed, latency_us: 111, response_code: 204, uri: /disablerestart/all, method:[8810] Disable keep_running [8820] Disable keep_running POST, [8830] Disable keep_running req_id: ed31147a-90c9-4e11-87e3-3293d78c6349, remote_addr: 127.0.0.1:46217, local_addr: 127.0.0.1:9998 A new loop begins at February 8, 2024 at 06:42:11 AM UTC /var/tmp/bins/crutest perf -g 1 -t 127.0.0.1:8810 -t 127.0.0.1:8820 -t 127.0.0.1:8830 -c 30000 -q --write-loops 2 --read-loops 2 --perf-out /tmp/perf-ES-16384-EC-1280.csv {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-02-08T06:42:11.813210825Z","hostname":"ip-10-150-1-144.us-west-2.compute.internal","pid":1179} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"ebef7f4917e579ad4649dc0478073aaafe487733\",\n git_commit_timestamp: \"2024-02-08T06:19:45.000000000Z\",\n git_branch: \"alan/downstairs-clone\",\n rustc_semver: \"1.75.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"82e1608dfa6e0b5569232559e3d385fea5a93112\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: false,\n opt_level: 3,\n}","v":0,"name":"crucible","level":30,"time":"2024-02-08T06:42:11.813366056Z","hostname":"ip-10-150-1-144.us-west-2.compute.internal","pid":1179} {"msg":"Upstairs <-> Downstairs Message Version: 5","v":0,"name":"crucible","level":30,"time":"2024-02-08T06:42:11.813373155Z","hostname":"ip-10-150-1-144.us-west-2.compute.internal","pid":1179} {"msg":"Crucible 826fc6a9-3bbe-457f-8d06-9b13f7c5c5b7 has session id: a83bd733-c5b5-4c97-adc8-cee03da7e0a0","v":0,"name":"crucible","level":30,"time":"2024-02-08T06:42:11.813377764Z","hostname":"ip-10-150-1-144.us-west-2.compute.internal","pid":1179Crucible runtime is spawned ,"session_id":"a83bd733-c5b5-4c97-adc8-cee03da7e0a0"} {"msg":"Upstairs opts: Upstairs UUID: 826fc6a9-3bbe-457f-8d06-9b13f7c5c5b7, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-02-08T06:42:11.813393821Z","hostname":"ip-10-150-1-144.us-west-2.compute.internal","pid":1179,"session_id":"a83bd733-c5b5-4c97-adc8-cee03da7e0a0"} {"msg":"Crucible stats registered with UUID: 826fc6a9-3bbe-457f-8d06-9b13f7c5c5b7","v":0,"name":"crucible","level":30,"time":"2024-02-08T06:42:11.81339842Z","hostname":"ip-10-150-1-144.us-west-2.compute.internal","pid":1179,"session_id":"a83bd733-c5b5-4c97-adc8-cee03da7e0a0"} {"msg":"The guest has requested activation with gen:1","v":0,"name":"crucible","level":30,"time":"2024-02-08T06:42:11.813405909Z","hostname":"ip-10-150-1-144.us-west-2.compute.internal","pid":1179} {"msg":"826fc6a9-3bbe-457f-8d06-9b13f7c5c5b7 active request set","v":0,"name":"crucible","level":30,"time":"2024-02-08T06:42:11.813473097Z","hostname":"ip-10-150-1-144.us-west-2.compute.internal","pid":1179,"session_id":"a83bd733-c5b5-4c97-adc8-cee03da7e0a0"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-02-08T06:42:11.813487754Z","hostname":"ip-10-150-1-144.us-west-2.compute.internal","pid":1179,"client":"0","":"downstairs","session_id":"a83bd733-c5b5-4c97-adc8-cee03da7e0a0"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-02-08T06:42:11.813498042Z","hostname":"ip-10-150-1-144.us-west-2.compute.internal","pid":1179,"client":"0","":"downstairs","session_id":"a83bd733-c5b5-4c97-adc8-cee03da7e0a0"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-02-08T06:42:11.813503201Z","hostname":"ip-10-150-1-144.us-west-2.compute.internal","pid":1179,"client":"1","":"downstairs","session_id":"a83bd733-c5b5-4c97-adc8-cee03da7e0a0"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-02-08T06:42:11.81350831Z","hostname":"ip-10-150-1-144.us-west-2.compute.internal","pid":1179,"client":"1","":"downstairs","session_id":"a83bd733-c5b5-4c97-adc8-cee03da7e0a0"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-02-08T06:42:11.813513969Z","hostname":"ip-10-150-1-144.us-west-2.compute.internal","pid":1179,"client":"2","":"downstairs","session_id":"a83bd733-c5b5-4c97-adc8-cee03da7e0a0"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-02-08T06:42:11.813518578Z","hostname":"ip-10-150-1-144.us-west-2.compute.internal","pid":1179,"client":"2","":"downstairs","session_id":"a83bd733-c5b5-4c97-adc8-cee03da7e0a0"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-02-08T06:42:11.813522998Z","hostname":"ip-10-150-1-144.us-west-2.compute.internal","pid":1179,"":"io task","client":"2","":"downstairs","session_id":"a83bd733-c5b5-4c97-adc8-cee03da7e0a0"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-08T06:42:11.813536625Z","hostname":"ip-10-150-1-144.us-west-2.compute.internal","pid":1179,"":"io task","client":"0","":"downstairs","session_id":"a83bd733-c5b5-4c97-adc8-cee03da7e0a0"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-02-08T06:42:11.813541534Z","hostname":"ip-10-150-1-144.us-west-2.compute.internal","pid":1179,"":"io task","client":"1","":"downstairs","session_id":"a83bd733-c5b5-4c97-adc8-cee03da7e0a0"} {"msg":"ds_connection connected","v":0,"name":"crucible","level":30,"time":"2024-02-08T06:42:11.813692826Z","hostname":"ip-10-150-1-144.us-west-2.compute.internal","pid":1179,"":"io task","client":"0","":"downstairs","session_id":"a83bd733-c5b5-4c97-adc8-cee03da7e0a0"} {"msg":"ds_connection connected","v":0,"name":"crucible","level":30,"time":"2024-02-08T06:42:11.813706844Z","hostname":"ip-10-150-1-144.us-west-2.compute.internal","pid":1179,"":"io task","client":"2","":"downstairs","session_id":"a83bd733-c5b5-4c97-adc8-cee03da7e0a0"} {"msg":"ds_connection connected","v":0,"name":"crucible","level":30,"time":"2024-02-08T06:42:11.813769492Z","hostname":"ip-10-150-1-144.us-west-2.compute.internal","pid":1179,"":"io task","client":"1","":"downstairs","session_id":"a83bd733-c5b5-4c97-adc8-cee03da7e0a0"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-02-08T06:42:11.813973335Z","hostname":"ip-10-150-1-144.us-west-2.compute.internal","pid":1179,"client":"1","":"downstairs","session_id":"a83bd733-c5b5-4c97-adc8-cee03da7e0a0"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-02-08T06:42:11.813990602Z","hostname":"ip-10-150-1-144.us-west-2.compute.internal","pid":1179,"client":"1","":"downstairs","session_id":"a83bd733-c5b5-4c97-adc8-cee03da7e0a0"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-02-08T06:42:11.813996301Z","hostname":"ip-10-150-1-144.us-west-2.compute.internal","pid":1179,"client":"2","":"downstairs","session_id":"a83bd733-c5b5-4c97-adc8-cee03da7e0a0"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-02-08T06:42:11.81400076Z","hostname":"ip-10-150-1-144.us-west-2.compute.internal","pid":1179,"client":"2","":"downstairs","session_id":"a83bd733-c5b5-4c97-adc8-cee03da7e0a0"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-02-08T06:42:11.814062888Z","hostname":"ip-10-150-1-144.us-west-2.compute.internal","pid":1179,"client":"0","":"downstairs","session_id":"a83bd733-c5b5-4c97-adc8-cee03da7e0a0"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-02-08T06:42:11.814077846Z","hostname":"ip-10-150-1-144.us-west-2.compute.internal","pid":1179,"client":"0","":"downstairs","session_id":"a83bd733-c5b5-4c97-adc8-cee03da7e0a0"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-02-08T06:42:11.814462495Z","hostname":"ip-10-150-1-144.us-west-2.compute.internal","pid":1179,"client":"1","":"downstairs","session_id":"a83bd733-c5b5-4c97-adc8-cee03da7e0a0"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-02-08T06:42:11.814480332Z","hostname":"ip-10-150-1-144.us-west-2.compute.internal","pid":1179,"client":"0","":"downstairs","session_id":"a83bd733-c5b5-4c97-adc8-cee03da7e0a0"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-02-08T06:42:11.814519375Z","hostname":"ip-10-150-1-144.us-west-2.compute.internal","pid":1179,"client":"2","":"downstairs","session_id":"a83bd733-c5b5-4c97-adc8-cee03da7e0a0"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-02-08T06:42:11.820779646Z","hostname":"ip-10-150-1-144.us-west-2.compute.internal","pid":1179,"client":"1","":"downstairs","session_id":"a83bd733-c5b5-4c97-adc8-cee03da7e0a0"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-02-08T06:42:11.820797823Z","hostname":"ip-10-150-1-144.us-west-2.compute.internal","pid":1179,"client":"1","":"downstairs","session_id":"a83bd733-c5b5-4c97-adc8-cee03da7e0a0"} {"msg":"Waiting for true more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-02-08T06:42:11.820803331Z","hostname":"ip-10-150-1-144.us-west-2.compute.internal","pid":1179,"session_id":"a83bd733-c5b5-4c97-adc8-cee03da7e0a0"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-02-08T06:42:11.821907139Z","hostname":"ip-10-150-1-144.us-west-2.compute.internal","pid":1179,"client":"2","":"downstairs","session_id":"a83bd733-c5b5-4c97-adc8-cee03da7e0a0"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-02-08T06:42:11.821929565Z","hostname":"ip-10-150-1-144.us-west-2.compute.internal","pid":1179,"client":"2","":"downstairs","session_id":"a83bd733-c5b5-4c97-adc8-cee03da7e0a0"} {"msg":"Waiting for true more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-02-08T06:42:11.821937363Z","hostname":"ip-10-150-1-144.us-west-2.compute.internal","pid":1179,"session_id":"a83bd733-c5b5-4c97-adc8-cee03da7e0a0"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-02-08T06:42:11.821966238Z","hostname":"ip-10-150-1-144.us-west-2.compute.internal","pid":1179,"client":"0","":"downstairs","session_id":"a83bd733-c5b5-4c97-adc8-cee03da7e0a0"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-02-08T06:42:11.821979286Z","hostname":"ip-10-150-1-144.us-west-2.compute.internal","pid":1179,"client":"0","":"downstairs","session_id":"a83bd733-c5b5-4c97-adc8-cee03da7e0a0"} {"msg":"[0]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-02-08T06:42:11.821983945Z","hostname":"ip-10-150-1-144.us-west-2.compute.internal","pid":1179,"":"downstairs","":"downstairs","session_id":"a83bd733-c5b5-4c97-adc8-cee03da7e0a0"} {"msg":"[0]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-02-08T06:42:11.821988694Z","hostname":"ip-10-150-1-144.us-west-2.compute.internal","pid":1179,"":"downstairs","":"downstairs","session_id":"a83bd733-c5b5-4c97-adc8-cee03da7e0a0"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-02-08T06:42:11.821993293Z","hostname":"ip-10-150-1-144.us-west-2.compute.internal","pid":1179,"":"downstairs","":"downstairs","session_id":"a83bd733-c5b5-4c97-adc8-cee03da7e0a0"} {"msg":"[1]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-02-08T06:42:11.821998512Z","hostname":"ip-10-150-1-144.us-west-2.compute.internal","pid":1179,"":"downstairs","":"downstairs","session_id":"a83bd733-c5b5-4c97-adc8-cee03da7e0a0"} {"msg":"[1]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-02-08T06:42:11.822003651Z","hostname":"ip-10-150-1-144.us-west-2.compute.internal","pid":1179,"":"downstairs","":"downstairs","session_id":"a83bd733-c5b5-4c97-adc8-cee03da7e0a0"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-02-08T06:42:11.82200876Z","hostname":"ip-10-150-1-144.us-west-2.compute.internal","pid":1179,"":"downstairs","":"downstairs","session_id":"a83bd733-c5b5-4c97-adc8-cee03da7e0a0"} {"msg":"[2]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-02-08T06:42:11.822013589Z","hostname":"ip-10-150-1-144.us-west-2.compute.internal","pid":1179,"":"downstairs","":"downstairs","session_id":"a83bd733-c5b5-4c97-adc8-cee03da7e0a0"} {"msg":"[2]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-02-08T06:42:11.822018499Z","hostname":"ip-10-150-1-144.us-west-2.compute.internal","pid":1179,"":"downstairs","":"downstairs","session_id":"a83bd733-c5b5-4c97-adc8-cee03da7e0a0"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-02-08T06:42:11.822023227Z","hostname":"ip-10-150-1-144.us-west-2.compute.internal","pid":1179,"":"downstairs","":"downstairs","session_id":"a83bd733-c5b5-4c97-adc8-cee03da7e0a0"} {"msg":"Max found gen is 1","v":0,"name":"crucible","level":30,"time":"2024-02-08T06:42:11.822028726Z","hostname":"ip-10-150-1-144.us-west-2.compute.internal","pid":1179,"":"downstairs","":"downstairs","session_id":"a83bd733-c5b5-4c97-adc8-cee03da7e0a0"} {"msg":"Generation requested: 1 >= found:1","v":0,"name":"crucible","level":30,"time":"2024-02-08T06:42:11.822034185Z","hostname":"ip-10-150-1-144.us-west-2.compute.internal","pid":1179,"":"downstairs","":"downstairs","session_id":"a83bd733-c5b5-4c97-adc8-cee03da7e0a0"} {"msg":"Next flush: 1","v":0,"name":"crucible","level":30,"time":"2024-02-08T06:42:11.822038555Z","hostname":"ip-10-150-1-144.us-west-2.compute.internal","pid":1179,"":"downstairs","":"downstairs","session_id":"a83bd733-c5b5-4c97-adc8-cee03da7e0a0"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-02-08T06:42:11.822043704Z","hostname":"ip-10-150-1-144.us-west-2.compute.internal","pid":1179,"":"downstairs","":"downstairs","session_id":"a83bd733-c5b5-4c97-adc8-cee03da7e0a0"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-02-08T06:42:11.822048353Z","hostname":"ip-10-150-1-144.us-west-2.compute.internal","pid":1179,"session_id":"a83bd733-c5b5-4c97-adc8-cee03da7e0a0"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-02-08T06:42:11.822052202Z","hostname":"ip-10-150-1-144.us-west-2.compute.internal","pid":1179,"client":"0","":"downstairs","session_id":"a83bd733-c5b5-4c97-adc8-cee03da7e0a0"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-02-08T06:42:11.822057091Z","hostname":"ip-10-150-1-144.us-west-2.compute.internal","pid":1179,"client":"1","":"downstairs","session_id":"a83bd733-c5b5-4c97-adc8-cee03da7e0a0"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-02-08T06:42:11.822061431Z","hostname":"ip-10-150-1-144.us-west-2.compute.internal","pid":1179,"client":"2","":"downstairs","session_id":"a83bd733-c5b5-4c97-adc8-cee03da7e0a0"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-02-08T06:42:11.82206589Z","hostname":"ip-10-150-1-144.us-west-2.compute.internal","pid":1179,"session_id":"a83bd733-c5b5-4c97-adc8-cee03da7e0a0"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-02-08T06:42:11.822069799Z","hostname":"ip-10-150-1-144.us-west-2.compute.internal","pid":1179,"session_id":"a83bd733-c5b5-4c97-adc8-cee03da7e0a0"} {"msg":"826fc6a9-3bbe-457f-8d06-9b13f7c5c5b7 is now active with session: a83bd733-c5b5-4c97-adc8-cee03da7e0a0","v":0,"name":"crucible","level":30,"time":"2024-02-08T06:42:11.822073678Z","hostname":"ip-10-150-1-144.us-west-2.compute.internal","pid":1179,"session_id":"a83bd733-c5b5-4c97-adc8-cee03da7e0a0"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-02-08T06:42:11.822077388Z","hostname":"ip-10-150-1-144.us-west-2.compute.internal","pid":1179,"session_id":"a83bd733-c5b5-4c97-adc8-cee03da7e0a0"} {"msg":"The guest has finished waiting for activation with:1","v":0,"name":"crucible","level":30,"time":"2024-02-08T06:42:11.822087386Z","hostname":"ip-10-150-1-144.us-west-2.compute.internal","pid":1179} Wait for a query_work_queue command to finish before sending IO Region: es:16384 ec:1280 bs:4096 ts:85899345920 tb:20971520 max_io:256 or 1048576 Perf test TEST SECONDS COUNT DPTH IOPS MEAN P95 P99 MAX ES EC rwrites 7.78 30000 1 3855.08 0.00026 0.00058 0.00116 0.07638 16384 1280 rwrites 7.83 30000 1 3831.92 0.00026 0.00059 0.00117 0.06639 16384 1280 rreads 5.52 30000 1 5431.48 0.00018 0.00024 0.00029 0.01379 16384 1280 rreads 5.54 30000 1 5412.60 0.00018 0.00025 0.00029 0.01379 16384 1280 Perf test completed, stop all downstairs Feb 08 06:42:45.665 INFO accepted connection, remote_addr: 127.0.0.1:37942, local_addr: 127.0.0.1:9998 Main task has work to do, go find it got dsc Shutdown Shutdown Feb 08 06:42:45.665 INFO request completed[8810] Disable keep_running , Shut it down latency_us[8830] Disable keep_running :[8830] Got stop action so:false kr:false 57, response_code: 204, uri: /shutdown, method: POST, req_id: [8810] Got stop action so:false kr:false b108b076-8408-4da1-a27c-b505838e28be[8820] Disable keep_running , [8820] Got stop action so:false kr:false remote_addr: 127.0.0.1:37942, local_addr: 127.0.0.1:9998 [8820] Exited with: ExitStatus(unix_wait_status(9)) [8810] Exited with: ExitStatus(unix_wait_status(9)) [8820][1] reports Exit [8810][0] reports Exit [8830] Exited with: ExitStatus(unix_wait_status(9)) [8830][2] reports Exit TEST SECONDS COUNT DPTH IOPS MEAN P95 P99 MAX ES EC rwrites 23.41 30000 1 1281.29 0.00078 0.00134 0.00203 0.02881 16384 640 rwrites 9.53 30000 1 3146.40 0.00032 0.00109 0.00177 0.08593 16384 640 rwrites 7.78 30000 1 3855.08 0.00026 0.00058 0.00116 0.07638 16384 1280 rwrites 7.83 30000 1 3831.92 0.00026 0.00059 0.00117 0.06639 16384 1280 TEST SECONDS COUNT DPTH IOPS MEAN P95 P99 MAX ES EC rreads 5.66 30000 1 5298.08 0.00019 0.00026 0.00030 0.01252 16384 640 rreads 5.69 30000 1 5271.51 0.00019 0.00026 0.00030 0.01282 16384 640 rreads 5.52 30000 1 5431.48 0.00018 0.00024 0.00029 0.01379 16384 1280 rreads 5.54 30000 1 5412.60 0.00018 0.00025 0.00029 0.01379 16384 1280 Perf test finished on February 8, 2024 at 06:42:46 AM UTC