/input/rbuild/work Create region with ES:16384 EC:640 BS:4096 Creating region directory at: /var/tmp/dsc Creating dsc directory at: /tmp/dsc Downstairs region 0 created at /var/tmp/dsc/8810 in 1.2626936 Downstairs region 1 created at /var/tmp/dsc/8820 in 3.283981 Downstairs region 2 created at /var/tmp/dsc/8830 in 0.83388114 Region set with 3 regions was created Using existing output directory "/tmp/dsc" start ds: 8810 start ds: 8820 start ds: 8830 start access at:127.0.0.1:9998 Starting downstairs at port 8820 Starting downstairs at port 8810 Starting downstairs at port 8830 Make output file at "/tmp/dsc/downstairs-8810.txt" Make output file at "/tmp/dsc/downstairs-8820.txt" [8810][0] reports Starting [8820][1] reports Starting [8830][2] reports Starting Make output file at "/tmp/dsc/downstairs-8830.txt" Downstairs /var/tmp/dsc/8810 port 8810 PID:Some(1144) [8810][0] reports Running Downstairs /var/tmp/dsc/8820 port 8820 PID:Some(1145) [8820][1] reports Running Downstairs /var/tmp/dsc/8830 port 8830 PID:Some(1146) [8830][2] reports Running Feb 04 00:40:19.693 INFO listening, local_addr: 127.0.0.1:9998 Control access at:127.0.0.1:9998 1144 1145 1146 Feb 04 00:40:24.762 INFO accepted connection, remote_addr: 127.0.0.1:39608, local_addr: 127.0.0.1:9998 Main task has work to do, go find it got dsc DisableRestartAll disable restart on all: 3 [8810] Disable keep_running [8820] Disable keep_running Feb 04 00:40:24.763 INFO request completed, latency_us: [8830] Disable keep_running 292, response_code: 204, uri: /disablerestart/all, method: POST, req_id: 4916e793-9e12-4b23-8a9f-c78ec4f4427d, remote_addr: 127.0.0.1:39608, local_addr: 127.0.0.1:9998 A new loop begins at February 4, 2024 at 12:40:24 AM UTC /var/tmp/bins/crutest perf -g 1 -t 127.0.0.1:8810 -t 127.0.0.1:8820 -t 127.0.0.1:8830 -c 30000 -q --write-loops 2 --read-loops 2 --perf-out /tmp/perf-ES-16384-EC-640.csv {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-02-04T00:40:24.795282098Z","hostname":"ip-10-150-1-94.us-west-2.compute.internal","pid":1154} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"b0b91d339dac5ce12b22413acece6aaa5611cd02\",\n git_commit_timestamp: \"2024-02-03T15:21:01.000000000Z\",\n git_branch: \"renovate/tokio-1.x\",\n rustc_semver: \"1.75.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"82e1608dfa6e0b5569232559e3d385fea5a93112\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: false,\n opt_level: 3,\n}","v":0,"name":"crucible","level":30,"time":"2024-02-04T00:40:24.795574553Z","hostname":"ip-10-150-1-94.us-west-2.compute.internal","pid":1154} {"msg":"Upstairs <-> Downstairs Message Version: 5","v":0,"name":"crucible","level":30,"time":"2024-02-04T00:40:24.795585268Z","hostname":"ip-10-150-1-94.us-west-2.compute.internal","pid":1154} {"msg":"Crucible d2217d0a-9f32-49cd-8fd7-c4c76f9f6e28 has session id: dde2cb48-a8e7-4895-8cf7-96ba08f99ab4","v":0,"name":"crucible","level":30,"time":"2024-02-04T00:40:24.79559231Z","hostname":"ip-10-150-1-94.us-west-2.compute.internal","pid":1154,"session_id":"dde2cb48-a8e7-4895-8cf7-96ba08f99ab4"} {"msg":"Upstairs opts: Upstairs UUID: d2217d0a-9f32-49cd-8fd7-c4c76f9f6e28, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-02-04T00:40:24.795712874Z","hostname":"ip-10-150-1-94.us-west-2.compute.internal","pid":1154,"session_id":"dde2cb48-a8e7-4895-8cf7-96ba08f99ab4"} {"msg":"Crucible stats registered with UUID: d2217d0a-9f32-49cd-8fd7-c4c76f9f6e28","v":0,"name":"crucible","level":30,"time":"2024-02-04T00:40:24.795743215Z","hostname":"ip-10-150-1-94.us-west-2.compute.internal","pid":1154,"session_id":"dde2cb48-a8e7-4895-8cf7-96ba08f99ab4"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:1","v":0,"name":"crucible","level":30,"time":"2024-02-04T00:40:24.79602133Z","hostname":"ip-10-150-1-94.us-west-2.compute.internal","pid":1154} {"msg":"d2217d0a-9f32-49cd-8fd7-c4c76f9f6e28 active request set","v":0,"name":"crucible","level":30,"time":"2024-02-04T00:40:24.79627216Z","hostname":"ip-10-150-1-94.us-west-2.compute.internal","pid":1154,"session_id":"dde2cb48-a8e7-4895-8cf7-96ba08f99ab4"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-02-04T00:40:24.796314089Z","hostname":"ip-10-150-1-94.us-west-2.compute.internal","pid":1154,"client":"0","":"downstairs","session_id":"dde2cb48-a8e7-4895-8cf7-96ba08f99ab4"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-02-04T00:40:24.796328485Z","hostname":"ip-10-150-1-94.us-west-2.compute.internal","pid":1154,"client":"0","":"downstairs","session_id":"dde2cb48-a8e7-4895-8cf7-96ba08f99ab4"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-02-04T00:40:24.796341617Z","hostname":"ip-10-150-1-94.us-west-2.compute.internal","pid":1154,"client":"1","":"downstairs","session_id":"dde2cb48-a8e7-4895-8cf7-96ba08f99ab4"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-02-04T00:40:24.79637688Z","hostname":"ip-10-150-1-94.us-west-2.compute.internal","pid":1154,"client":"1","":"downstairs","session_id":"dde2cb48-a8e7-4895-8cf7-96ba08f99ab4"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-02-04T00:40:24.796389868Z","hostname":"ip-10-150-1-94.us-west-2.compute.internal","pid":1154,"client":"2","":"downstairs","session_id":"dde2cb48-a8e7-4895-8cf7-96ba08f99ab4"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-02-04T00:40:24.796428393Z","hostname":"ip-10-150-1-94.us-west-2.compute.internal","pid":1154,"client":"2","":"downstairs","session_id":"dde2cb48-a8e7-4895-8cf7-96ba08f99ab4"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-04T00:40:24.796441749Z","hostname":"ip-10-150-1-94.us-west-2.compute.internal","pid":1154,"":"io task","client":"0","":"downstairs","session_id":"dde2cb48-a8e7-4895-8cf7-96ba08f99ab4"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-02-04T00:40:24.796499646Z","hostname":"ip-10-150-1-94.us-west-2.compute.internal","pid":1154,"":"io task","client":"1","":"downstairs","session_id":"dde2cb48-a8e7-4895-8cf7-96ba08f99ab4"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-02-04T00:40:24.796620496Z","hostname":"ip-10-150-1-94.us-west-2.compute.internal","pid":1154,"":"io task","client":"2","":"downstairs","session_id":"dde2cb48-a8e7-4895-8cf7-96ba08f99ab4"} {"msg":"ds_connection connected","v":0,"name":"crucible","level":30,"time":"2024-02-04T00:40:24.79667645Z","hostname":"ip-10-150-1-94.us-west-2.compute.internal","pid":1154,"":"io task","client":"0","":"downstairs","session_id":"dde2cb48-a8e7-4895-8cf7-96ba08f99ab4"} {"msg":"ds_connection connected","v":0,"name":"crucible","level":30,"time":"2024-02-04T00:40:24.796690988Z","hostname":"ip-10-150-1-94.us-west-2.compute.internal","pid":1154,"":"io task","client":"1","":"downstairs","session_id":"dde2cb48-a8e7-4895-8cf7-96ba08f99ab4"} {"msg":"ds_connection connected","v":0,"name":"crucible","level":30,"time":"2024-02-04T00:40:24.796805972Z","hostname":"ip-10-150-1-94.us-west-2.compute.internal","pid":1154,"":"io task","client":"2","":"downstairs","session_id":"dde2cb48-a8e7-4895-8cf7-96ba08f99ab4"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-02-04T00:40:24.797926726Z","hostname":"ip-10-150-1-94.us-west-2.compute.internal","pid":1154,"client":"2","":"downstairs","session_id":"dde2cb48-a8e7-4895-8cf7-96ba08f99ab4"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-02-04T00:40:24.797971132Z","hostname":"ip-10-150-1-94.us-west-2.compute.internal","pid":1154,"client":"2","":"downstairs","session_id":"dde2cb48-a8e7-4895-8cf7-96ba08f99ab4"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-02-04T00:40:24.79797957Z","hostname":"ip-10-150-1-94.us-west-2.compute.internal","pid":1154,"client":"0","":"downstairs","session_id":"dde2cb48-a8e7-4895-8cf7-96ba08f99ab4"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-02-04T00:40:24.797987417Z","hostname":"ip-10-150-1-94.us-west-2.compute.internal","pid":1154,"client":"0","":"downstairs","session_id":"dde2cb48-a8e7-4895-8cf7-96ba08f99ab4"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-02-04T00:40:24.798020166Z","hostname":"ip-10-150-1-94.us-west-2.compute.internal","pid":1154,"client":"1","":"downstairs","session_id":"dde2cb48-a8e7-4895-8cf7-96ba08f99ab4"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-02-04T00:40:24.798065683Z","hostname":"ip-10-150-1-94.us-west-2.compute.internal","pid":1154,"client":"1","":"downstairs","session_id":"dde2cb48-a8e7-4895-8cf7-96ba08f99ab4"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-02-04T00:40:24.798700491Z","hostname":"ip-10-150-1-94.us-west-2.compute.internal","pid":1154,"client":"2","":"downstairs","session_id":"dde2cb48-a8e7-4895-8cf7-96ba08f99ab4"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-02-04T00:40:24.798824491Z","hostname":"ip-10-150-1-94.us-west-2.compute.internal","pid":1154,"client":"1","":"downstairs","session_id":"dde2cb48-a8e7-4895-8cf7-96ba08f99ab4"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-02-04T00:40:24.798857906Z","hostname":"ip-10-150-1-94.us-west-2.compute.internal","pid":1154,"client":"0","":"downstairs","session_id":"dde2cb48-a8e7-4895-8cf7-96ba08f99ab4"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-02-04T00:40:24.803749804Z","hostname":"ip-10-150-1-94.us-west-2.compute.internal","pid":1154,"client":"1","":"downstairs","session_id":"dde2cb48-a8e7-4895-8cf7-96ba08f99ab4"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-02-04T00:40:24.803788398Z","hostname":"ip-10-150-1-94.us-west-2.compute.internal","pid":1154,"client":"1","":"downstairs","session_id":"dde2cb48-a8e7-4895-8cf7-96ba08f99ab4"} {"msg":"Waiting for true more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-02-04T00:40:24.803798315Z","hostname":"ip-10-150-1-94.us-west-2.compute.internal","pid":1154,"session_id":"dde2cb48-a8e7-4895-8cf7-96ba08f99ab4"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-02-04T00:40:24.803883342Z","hostname":"ip-10-150-1-94.us-west-2.compute.internal","pid":1154,"client":"2","":"downstairs","session_id":"dde2cb48-a8e7-4895-8cf7-96ba08f99ab4"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-02-04T00:40:24.803956525Z","hostname":"ip-10-150-1-94.us-west-2.compute.internal","pid":1154,"client":"2","":"downstairs","session_id":"dde2cb48-a8e7-4895-8cf7-96ba08f99ab4"} {"msg":"Waiting for true more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-02-04T00:40:24.803964649Z","hostname":"ip-10-150-1-94.us-west-2.compute.internal","pid":1154,"session_id":"dde2cb48-a8e7-4895-8cf7-96ba08f99ab4"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-02-04T00:40:24.803971528Z","hostname":"ip-10-150-1-94.us-west-2.compute.internal","pid":1154,"client":"0","":"downstairs","session_id":"dde2cb48-a8e7-4895-8cf7-96ba08f99ab4"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-02-04T00:40:24.803978771Z","hostname":"ip-10-150-1-94.us-west-2.compute.internal","pid":1154,"client":"0","":"downstairs","session_id":"dde2cb48-a8e7-4895-8cf7-96ba08f99ab4"} {"msg":"[0]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-02-04T00:40:24.804064326Z","hostname":"ip-10-150-1-94.us-west-2.compute.internal","pid":1154,"":"downstairs","":"downstairs","session_id":"dde2cb48-a8e7-4895-8cf7-96ba08f99ab4"} {"msg":"[0]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-02-04T00:40:24.804091429Z","hostname":"ip-10-150-1-94.us-west-2.compute.internal","pid":1154,"":"downstairs","":"downstairs","session_id":"dde2cb48-a8e7-4895-8cf7-96ba08f99ab4"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-02-04T00:40:24.804099082Z","hostname":"ip-10-150-1-94.us-west-2.compute.internal","pid":1154,"":"downstairs","":"downstairs","session_id":"dde2cb48-a8e7-4895-8cf7-96ba08f99ab4"} {"msg":"[1]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-02-04T00:40:24.804107208Z","hostname":"ip-10-150-1-94.us-west-2.compute.internal","pid":1154,"":"downstairs","":"downstairs","session_id":"dde2cb48-a8e7-4895-8cf7-96ba08f99ab4"} {"msg":"[1]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-02-04T00:40:24.804114311Z","hostname":"ip-10-150-1-94.us-west-2.compute.internal","pid":1154,"":"downstairs","":"downstairs","session_id":"dde2cb48-a8e7-4895-8cf7-96ba08f99ab4"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-02-04T00:40:24.804121523Z","hostname":"ip-10-150-1-94.us-west-2.compute.internal","pid":1154,"":"downstairs","":"downstairs","session_id":"dde2cb48-a8e7-4895-8cf7-96ba08f99ab4"} {"msg":"[2]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-02-04T00:40:24.804129091Z","hostname":"ip-10-150-1-94.us-west-2.compute.internal","pid":1154,"":"downstairs","":"downstairs","session_id":"dde2cb48-a8e7-4895-8cf7-96ba08f99ab4"} {"msg":"[2]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-02-04T00:40:24.804136419Z","hostname":"ip-10-150-1-94.us-west-2.compute.internal","pid":1154,"":"downstairs","":"downstairs","session_id":"dde2cb48-a8e7-4895-8cf7-96ba08f99ab4"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-02-04T00:40:24.804143537Z","hostname":"ip-10-150-1-94.us-west-2.compute.internal","pid":1154,"":"downstairs","":"downstairs","session_id":"dde2cb48-a8e7-4895-8cf7-96ba08f99ab4"} {"msg":"Max found gen is 1","v":0,"name":"crucible","level":30,"time":"2024-02-04T00:40:24.804151342Z","hostname":"ip-10-150-1-94.us-west-2.compute.internal","pid":1154,"":"downstairs","":"downstairs","session_id":"dde2cb48-a8e7-4895-8cf7-96ba08f99ab4"} {"msg":"Generation requested: 1 >= found:1","v":0,"name":"crucible","level":30,"time":"2024-02-04T00:40:24.804158819Z","hostname":"ip-10-150-1-94.us-west-2.compute.internal","pid":1154,"":"downstairs","":"downstairs","session_id":"dde2cb48-a8e7-4895-8cf7-96ba08f99ab4"} {"msg":"Next flush: 1","v":0,"name":"crucible","level":30,"time":"2024-02-04T00:40:24.804183595Z","hostname":"ip-10-150-1-94.us-west-2.compute.internal","pid":1154,"":"downstairs","":"downstairs","session_id":"dde2cb48-a8e7-4895-8cf7-96ba08f99ab4"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-02-04T00:40:24.804191261Z","hostname":"ip-10-150-1-94.us-west-2.compute.internal","pid":1154,"":"downstairs","":"downstairs","session_id":"dde2cb48-a8e7-4895-8cf7-96ba08f99ab4"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2024-02-04T00:40:24.804198501Z","hostname":"ip-10-150-1-94.us-west-2.compute.internal","pid":1154,"session_id":"dde2cb48-a8e7-4895-8cf7-96ba08f99ab4"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-02-04T00:40:24.804209268Z","hostname":"ip-10-150-1-94.us-west-2.compute.internal","pid":1154,"client":"0","":"downstairs","session_id":"dde2cb48-a8e7-4895-8cf7-96ba08f99ab4"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-02-04T00:40:24.804216448Z","hostname":"ip-10-150-1-94.us-west-2.compute.internal","pid":1154,"client":"1","":"downstairs","session_id":"dde2cb48-a8e7-4895-8cf7-96ba08f99ab4"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-02-04T00:40:24.804223547Z","hostname":"ip-10-150-1-94.us-west-2.compute.internal","pid":1154,"client":"2","":"downstairs","session_id":"dde2cb48-a8e7-4895-8cf7-96ba08f99ab4"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-02-04T00:40:24.804247764Z","hostname":"ip-10-150-1-94.us-west-2.compute.internal","pid":1154,"session_id":"dde2cb48-a8e7-4895-8cf7-96ba08f99ab4"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-02-04T00:40:24.804254349Z","hostname":"ip-10-150-1-94.us-west-2.compute.internal","pid":1154,"session_id":"dde2cb48-a8e7-4895-8cf7-96ba08f99ab4"} {"msg":"d2217d0a-9f32-49cd-8fd7-c4c76f9f6e28 is now active with session: dde2cb48-a8e7-4895-8cf7-96ba08f99ab4","v":0,"name":"crucible","level":30,"time":"2024-02-04T00:40:24.804260927Z","hostname":"ip-10-150-1-94.us-west-2.compute.internal","pid":1154,"session_id":"dde2cb48-a8e7-4895-8cf7-96ba08f99ab4"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-02-04T00:40:24.804267933Z","hostname":"ip-10-150-1-94.us-west-2.compute.internal","pid":1154,"session_id":"dde2cb48-a8e7-4895-8cf7-96ba08f99ab4"} {"msg":"The guest has finished waiting for activation with:1","v":0,"name":"crucible","level":30,"time":"2024-02-04T00:40:24.804274753Z","hostname":"ip-10-150-1-94.us-west-2.compute.internal","pid":1154} Region: es:16384 ec:640 bs:4096 ts:42949672960 tb:10485760 max_io:256 or 1048576 Perf test TEST SECONDS COUNT DPTH IOPS MEAN P95 P99 MAX ES EC rwrites 16.85 30000 1 1780.30 0.00056 0.00149 0.00230 0.07616 16384 640 rwrites 14.39 30000 1 2084.81 0.00048 0.00104 0.00237 0.07736 16384 640 rreads 10.70 30000 1 2802.74 0.00036 0.00047 0.00062 0.01743 16384 640 rreads 10.63 30000 1 2821.25 0.00036 0.00047 0.00061 0.01747 16384 640 Perf test completed, stop all downstairs Feb 04 00:41:20.090 INFO accepted connection, remote_addr: 127.0.0.1:33048, local_addr: 127.0.0.1:9998 Main task has work to do, go find it got dsc Shutdown Shutdown Feb 04 00:41:20.091Shut it down INFO request completed, latency_us:[8830] Disable keep_running [8830] Got stop action so:false kr:false 74, response_code: 204, uri: /shutdown, method: POST, req_id: fbb7db83-10f9-4bc4-b082-9e7670a47b61, [8820] Disable keep_running remote_addr: 127.0.0.1:33048, local_addr: 127.0.0.1:9998 [8820] Got stop action so:false kr:false [8810] Disable keep_running [8810] Got stop action so:false kr:false [8830] Exited with: ExitStatus(unix_wait_status(9)) [8830][2] reports Exit [8820] Exited with: ExitStatus(unix_wait_status(9)) [8820][1] reports Exit [8810] Exited with: ExitStatus(unix_wait_status(9)) [8810][0] reports Exit Create region with ES:16384 EC:1280 BS:4096 Removing existing dsc directory "/tmp/dsc" Removing existing region "/var/tmp/dsc" Creating region directory at: /var/tmp/dsc Creating dsc directory at: /tmp/dsc Downstairs region 0 created at /var/tmp/dsc/8810 in 1.5934072 Downstairs region 1 created at /var/tmp/dsc/8820 in 1.728643 Downstairs region 2 created at /var/tmp/dsc/8830 in 1.5997896 Region set with 3 regions was created Using existing output directory "/tmp/dsc" start ds: 8810 start ds: 8820 start ds: 8830 start access at:127.0.0.1:9998 Starting downstairs at port 8810 Make output file at "/tmp/dsc/downstairs-8810.txt" [8810][0] reports Starting Starting downstairs at port 8830 Starting downstairs at port 8820 Make output file at "/tmp/dsc/downstairs-8820.txt" [8830][2] reports Starting Make output file at "/tmp/dsc/downstairs-8830.txt" [8820][1] reports Starting Downstairs /var/tmp/dsc/8810 port 8810 PID:Some(1163) [8810][0] reports Running Downstairs /var/tmp/dsc/8820 port 8820 PID:Some(1164) [8820][1] reports Running Downstairs /var/tmp/dsc/8830 port 8830 PID:Some(1165) [8830][2] reports Running Feb 04 00:41:26.120 INFO listening, local_addr: 127.0.0.1:9998 Control access at:127.0.0.1:9998 1163 1164 1165 Feb 04 00:41:31.189 INFO accepted connection, remote_addr: 127.0.0.1:35367, local_addr: 127.0.0.1:9998 Main task has work to do, go find it got dsc DisableRestartAll disable restart on all: 3 Feb 04 00:41:31.190 INFO request completed, latency_us: 154, response_code: 204, uri: /disablerestart/all, method[8810] Disable keep_running :[8830] Disable keep_running [8820] Disable keep_running POST, req_id: 4423e72d-dbd0-4957-b286-73305c4918eb, remote_addr: 127.0.0.1:35367, local_addr: 127.0.0.1:9998 A new loop begins at February 4, 2024 at 12:41:31 AM UTC /var/tmp/bins/crutest perf -g 1 -t 127.0.0.1:8810 -t 127.0.0.1:8820 -t 127.0.0.1:8830 -c 30000 -q --write-loops 2 --read-loops 2 --perf-out /tmp/perf-ES-16384-EC-1280.csv {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-02-04T00:41:31.217924234Z","hostname":"ip-10-150-1-94.us-west-2.compute.internal","pid":1173} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"b0b91d339dac5ce12b22413acece6aaa5611cd02\",\n git_commit_timestamp: \"2024-02-03T15:21:01.000000000Z\",\n git_branch: \"renovate/tokio-1.x\",\n rustc_semver: \"1.75.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"82e1608dfa6e0b5569232559e3d385fea5a93112\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: false,\n opt_level: 3,\n}","v":0,"name":"crucible","level":30,"time":"2024-02-04T00:41:31.218162304Z","hostname":"ip-10-150-1-94.us-west-2.compute.internal","pid":1173} {"msg":"Upstairs <-> Downstairs Message Version: 5","v":0,"name":"crucible","level":30,"time":"2024-02-04T00:41:31.218173232Z","hostname":"ip-10-150-1-94.us-west-2.compute.internal","pid":1173} {"msg":"Crucible f559163c-f50c-4ffa-9ee2-b7e6726706c7 has session id: 232a65ef-d055-429d-b95e-38159aad2e16","v":0,"name":"crucible","level":30,"time":"2024-02-04T00:41:31.21818034Z","hostname":"ip-10-150-1-94.us-west-2.compute.internal","pid":1173,"session_id":"232a65ef-d055-429d-b95e-38159aad2e16"} {"msg":"Upstairs opts: Upstairs UUID: f559163c-f50c-4ffa-9ee2-b7e6726706c7, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-02-04T00:41:31.218197943Z","hostname":"ip-10-150-1-94.us-west-2.compute.internal","pid":1173,"session_id":"232a65ef-d055-429d-b95e-38159aad2e16"} {"msg":"Crucible stats registered with UUID: f559163c-f50c-4ffa-9ee2-b7e6726706c7","v":0,"name":"crucible","level":30,"time":"2024-02-04T00:41:31.218205556Z","hostname":"ip-10-150-1-94.us-west-2.compute.internal","pid":1173,"session_id":"232a65ef-d055-429d-b95e-38159aad2e16"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:1","v":0,"name":"crucible","level":30,"time":"2024-02-04T00:41:31.218328821Z","hostname":"ip-10-150-1-94.us-west-2.compute.internal","pid":1173} {"msg":"f559163c-f50c-4ffa-9ee2-b7e6726706c7 active request set","v":0,"name":"crucible","level":30,"time":"2024-02-04T00:41:31.218363401Z","hostname":"ip-10-150-1-94.us-west-2.compute.internal","pid":1173,"session_id":"232a65ef-d055-429d-b95e-38159aad2e16"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-02-04T00:41:31.21837048Z","hostname":"ip-10-150-1-94.us-west-2.compute.internal","pid":1173,"client":"0","":"downstairs","session_id":"232a65ef-d055-429d-b95e-38159aad2e16"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-02-04T00:41:31.218384862Z","hostname":"ip-10-150-1-94.us-west-2.compute.internal","pid":1173,"client":"0","":"downstairs","session_id":"232a65ef-d055-429d-b95e-38159aad2e16"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-02-04T00:41:31.218392193Z","hostname":"ip-10-150-1-94.us-west-2.compute.internal","pid":1173,"client":"1","":"downstairs","session_id":"232a65ef-d055-429d-b95e-38159aad2e16"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-02-04T00:41:31.218401202Z","hostname":"ip-10-150-1-94.us-west-2.compute.internal","pid":1173,"client":"1","":"downstairs","session_id":"232a65ef-d055-429d-b95e-38159aad2e16"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-02-04T00:41:31.218408825Z","hostname":"ip-10-150-1-94.us-west-2.compute.internal","pid":1173,"client":"2","":"downstairs","session_id":"232a65ef-d055-429d-b95e-38159aad2e16"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-02-04T00:41:31.218417819Z","hostname":"ip-10-150-1-94.us-west-2.compute.internal","pid":1173,"client":"2","":"downstairs","session_id":"232a65ef-d055-429d-b95e-38159aad2e16"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-04T00:41:31.21847253Z","hostname":"ip-10-150-1-94.us-west-2.compute.internal","pid":1173,"":"io task","client":"0","":"downstairs","session_id":"232a65ef-d055-429d-b95e-38159aad2e16"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-02-04T00:41:31.21851129Z","hostname":"ip-10-150-1-94.us-west-2.compute.internal","pid":1173,"":"io task","client":"2","":"downstairs","session_id":"232a65ef-d055-429d-b95e-38159aad2e16"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-02-04T00:41:31.218519965Z","hostname":"ip-10-150-1-94.us-west-2.compute.internal","pid":1173,"":"io task","client":"1","":"downstairs","session_id":"232a65ef-d055-429d-b95e-38159aad2e16"} {"msg":"ds_connection connected","v":0,"name":"crucible","level":30,"time":"2024-02-04T00:41:31.218678077Z","hostname":"ip-10-150-1-94.us-west-2.compute.internal","pid":1173,"":"io task","client":"1","":"downstairs","session_id":"232a65ef-d055-429d-b95e-38159aad2e16"} {"msg":"ds_connection connected","v":0,"name":"crucible","level":30,"time":"2024-02-04T00:41:31.21870297Z","hostname":"ip-10-150-1-94.us-west-2.compute.internal","pid":1173,"":"io task","client":"2","":"downstairs","session_id":"232a65ef-d055-429d-b95e-38159aad2e16"} {"msg":"ds_connection connected","v":0,"name":"crucible","level":30,"time":"2024-02-04T00:41:31.218728931Z","hostname":"ip-10-150-1-94.us-west-2.compute.internal","pid":1173,"":"io task","client":"0","":"downstairs","session_id":"232a65ef-d055-429d-b95e-38159aad2e16"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-02-04T00:41:31.219223556Z","hostname":"ip-10-150-1-94.us-west-2.compute.internal","pid":1173,"client":"1","":"downstairs","session_id":"232a65ef-d055-429d-b95e-38159aad2e16"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-02-04T00:41:31.219266944Z","hostname":"ip-10-150-1-94.us-west-2.compute.internal","pid":1173,"client":"1","":"downstairs","session_id":"232a65ef-d055-429d-b95e-38159aad2e16"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-02-04T00:41:31.219324064Z","hostname":"ip-10-150-1-94.us-west-2.compute.internal","pid":1173,"client":"2","":"downstairs","session_id":"232a65ef-d055-429d-b95e-38159aad2e16"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-02-04T00:41:31.219428912Z","hostname":"ip-10-150-1-94.us-west-2.compute.internal","pid":1173,"client":"2","":"downstairs","session_id":"232a65ef-d055-429d-b95e-38159aad2e16"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-02-04T00:41:31.219456008Z","hostname":"ip-10-150-1-94.us-west-2.compute.internal","pid":1173,"client":"0","":"downstairs","session_id":"232a65ef-d055-429d-b95e-38159aad2e16"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-02-04T00:41:31.21946361Z","hostname":"ip-10-150-1-94.us-west-2.compute.internal","pid":1173,"client":"0","":"downstairs","session_id":"232a65ef-d055-429d-b95e-38159aad2e16"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-02-04T00:41:31.220152948Z","hostname":"ip-10-150-1-94.us-west-2.compute.internal","pid":1173,"client":"1","":"downstairs","session_id":"232a65ef-d055-429d-b95e-38159aad2e16"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-02-04T00:41:31.220308481Z","hostname":"ip-10-150-1-94.us-west-2.compute.internal","pid":1173,"client":"2","":"downstairs","session_id":"232a65ef-d055-429d-b95e-38159aad2e16"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-02-04T00:41:31.220353889Z","hostname":"ip-10-150-1-94.us-west-2.compute.internal","pid":1173,"client":"0","":"downstairs","session_id":"232a65ef-d055-429d-b95e-38159aad2e16"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-02-04T00:41:31.230392152Z","hostname":"ip-10-150-1-94.us-west-2.compute.internal","pid":1173,"client":"1","":"downstairs","session_id":"232a65ef-d055-429d-b95e-38159aad2e16"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-02-04T00:41:31.230415634Z","hostname":"ip-10-150-1-94.us-west-2.compute.internal","pid":1173,"client":"1","":"downstairs","session_id":"232a65ef-d055-429d-b95e-38159aad2e16"} {"msg":"Waiting for true more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-02-04T00:41:31.230428726Z","hostname":"ip-10-150-1-94.us-west-2.compute.internal","pid":1173,"session_id":"232a65ef-d055-429d-b95e-38159aad2e16"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-02-04T00:41:31.230511674Z","hostname":"ip-10-150-1-94.us-west-2.compute.internal","pid":1173,"client":"2","":"downstairs","session_id":"232a65ef-d055-429d-b95e-38159aad2e16"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-02-04T00:41:31.230532343Z","hostname":"ip-10-150-1-94.us-west-2.compute.internal","pid":1173,"client":"2","":"downstairs","session_id":"232a65ef-d055-429d-b95e-38159aad2e16"} {"msg":"Waiting for true more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-02-04T00:41:31.23054475Z","hostname":"ip-10-150-1-94.us-west-2.compute.internal","pid":1173,"session_id":"232a65ef-d055-429d-b95e-38159aad2e16"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-02-04T00:41:31.230663994Z","hostname":"ip-10-150-1-94.us-west-2.compute.internal","pid":1173,"client":"0","":"downstairs","session_id":"232a65ef-d055-429d-b95e-38159aad2e16"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-02-04T00:41:31.230707304Z","hostname":"ip-10-150-1-94.us-west-2.compute.internal","pid":1173,"client":"0","":"downstairs","session_id":"232a65ef-d055-429d-b95e-38159aad2e16"} {"msg":"[0]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-02-04T00:41:31.230720663Z","hostname":"ip-10-150-1-94.us-west-2.compute.internal","pid":1173,"":"downstairs","":"downstairs","session_id":"232a65ef-d055-429d-b95e-38159aad2e16"} {"msg":"[0]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-02-04T00:41:31.23073298Z","hostname":"ip-10-150-1-94.us-west-2.compute.internal","pid":1173,"":"downstairs","":"downstairs","session_id":"232a65ef-d055-429d-b95e-38159aad2e16"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-02-04T00:41:31.230747121Z","hostname":"ip-10-150-1-94.us-west-2.compute.internal","pid":1173,"":"downstairs","":"downstairs","session_id":"232a65ef-d055-429d-b95e-38159aad2e16"} {"msg":"[1]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-02-04T00:41:31.230759608Z","hostname":"ip-10-150-1-94.us-west-2.compute.internal","pid":1173,"":"downstairs","":"downstairs","session_id":"232a65ef-d055-429d-b95e-38159aad2e16"} {"msg":"[1]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-02-04T00:41:31.230772124Z","hostname":"ip-10-150-1-94.us-west-2.compute.internal","pid":1173,"":"downstairs","":"downstairs","session_id":"232a65ef-d055-429d-b95e-38159aad2e16"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-02-04T00:41:31.230786334Z","hostname":"ip-10-150-1-94.us-west-2.compute.internal","pid":1173,"":"downstairs","":"downstairs","session_id":"Wait for a query_work_queue command to finish before sending IO 232a65ef-d055-429d-b95e-38159aad2e16"} {"msg":"[2]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-02-04T00:41:31.230803216Z","hostname":"ip-10-150-1-94.us-west-2.compute.internal","pid":1173,"":"downstairs","":"downstairs","session_id":"232a65ef-d055-429d-b95e-38159aad2e16"} {"msg":"[2]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-02-04T00:41:31.230817106Z","hostname":"ip-10-150-1-94.us-west-2.compute.internal","pid":1173,"":"downstairs","":"downstairs","session_id":"232a65ef-d055-429d-b95e-38159aad2e16"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-02-04T00:41:31.230831815Z","hostname":"ip-10-150-1-94.us-west-2.compute.internal","pid":1173,"":"downstairs","":"downstairs","session_id":"232a65ef-d055-429d-b95e-38159aad2e16"} {"msg":"Max found gen is 1","v":0,"name":"crucible","level":30,"time":"2024-02-04T00:41:31.230844438Z","hostname":"ip-10-150-1-94.us-west-2.compute.internal","pid":1173,"":"downstairs","":"downstairs","session_id":"232a65ef-d055-429d-b95e-38159aad2e16"} {"msg":"Generation requested: 1 >= found:1","v":0,"name":"crucible","level":30,"time":"2024-02-04T00:41:31.230879159Z","hostname":"ip-10-150-1-94.us-west-2.compute.internal","pid":1173,"":"downstairs","":"downstairs","session_id":"232a65ef-d055-429d-b95e-38159aad2e16"} {"msg":"Next flush: 1","v":0,"name":"crucible","level":30,"time":"2024-02-04T00:41:31.230893098Z","hostname":"ip-10-150-1-94.us-west-2.compute.internal","pid":1173,"":"downstairs","":"downstairs","session_id":"232a65ef-d055-429d-b95e-38159aad2e16"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-02-04T00:41:31.230934098Z","hostname":"ip-10-150-1-94.us-west-2.compute.internal","pid":1173,"":"downstairs","":"downstairs","session_id":"232a65ef-d055-429d-b95e-38159aad2e16"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-02-04T00:41:31.230947672Z","hostname":"ip-10-150-1-94.us-west-2.compute.internal","pid":1173,"session_id":"232a65ef-d055-429d-b95e-38159aad2e16"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-02-04T00:41:31.230959004Z","hostname":"ip-10-150-1-94.us-west-2.compute.internal","pid":1173,"client":"0","":"downstairs","session_id":"232a65ef-d055-429d-b95e-38159aad2e16"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-02-04T00:41:31.231005093Z","hostname":"ip-10-150-1-94.us-west-2.compute.internal","pid":1173,"client":"1","":"downstairs","session_id":"232a65ef-d055-429d-b95e-38159aad2e16"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-02-04T00:41:31.231018622Z","hostname":"ip-10-150-1-94.us-west-2.compute.internal","pid":1173,"client":"2","":"downstairs","session_id":"232a65ef-d055-429d-b95e-38159aad2e16"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-02-04T00:41:31.23103064Z","hostname":"ip-10-150-1-94.us-west-2.compute.internal","pid":1173,"session_id":"232a65ef-d055-429d-b95e-38159aad2e16"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-02-04T00:41:31.231041573Z","hostname":"ip-10-150-1-94.us-west-2.compute.internal","pid":1173,"session_id":"232a65ef-d055-429d-b95e-38159aad2e16"} {"msg":"f559163c-f50c-4ffa-9ee2-b7e6726706c7 is now active with session: 232a65ef-d055-429d-b95e-38159aad2e16","v":0,"name":"crucible","level":30,"time":"2024-02-04T00:41:31.231049077Z","hostname":"ip-10-150-1-94.us-west-2.compute.internal","pid":1173,"session_id":"232a65ef-d055-429d-b95e-38159aad2e16"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-02-04T00:41:31.231061628Z","hostname":"ip-10-150-1-94.us-west-2.compute.internal","pid":1173,"session_id":"232a65ef-d055-429d-b95e-38159aad2e16"} {"msg":"The guest has finished waiting for activation with:1","v":0,"name":"crucible","level":30,"time":"2024-02-04T00:41:31.231068274Z","hostname":"ip-10-150-1-94.us-west-2.compute.internal","pid":1173} Region: es:16384 ec:1280 bs:4096 ts:85899345920 tb:20971520 max_io:256 or 1048576 Perf test TEST SECONDS COUNT DPTH IOPS MEAN P95 P99 MAX ES EC rwrites 17.33 30000 1 1731.16 0.00058 0.00136 0.00289 0.07455 16384 1280 rwrites 17.33 30000 1 1731.48 0.00058 0.00140 0.00316 0.04627 16384 1280 rreads 10.66 30000 1 2814.21 0.00036 0.00047 0.00062 0.01764 16384 1280 rreads 10.58 30000 1 2835.04 0.00035 0.00047 0.00061 0.01764 16384 1280 Perf test completed, stop all downstairs Feb 04 00:42:29.726 INFO accepted connection, remote_addr: 127.0.0.1:63571, local_addr: 127.0.0.1:9998 Main task has work to do, go find it got dsc Shutdown Shutdown Shut it down Feb 04 00:42:29.727 INFO request completed, latency_us: 76, response_code: 204, uri: /shutdown, method[8820] Disable keep_running :[8820] Got stop action so:false kr:false [8830] Disable keep_running POST, req_id: d1542ef0-3ca5-4ed2-b2dd-3fab37c3bbc2, remote_addr: 127.0.0.1:63571, local_addr: 127.0.0.1:9998 [8830] Got stop action so:false kr:false [8810] Disable keep_running [8810] Got stop action so:false kr:false [8830] Exited with: ExitStatus(unix_wait_status(9)) [8830][2] reports Exit [8810] Exited with: ExitStatus(unix_wait_status(9)) [8810][0] reports Exit [8820] Exited with: ExitStatus(unix_wait_status(9)) [8820][1] reports Exit TEST SECONDS COUNT DPTH IOPS MEAN P95 P99 MAX ES EC rwrites 16.85 30000 1 1780.30 0.00056 0.00149 0.00230 0.07616 16384 640 rwrites 14.39 30000 1 2084.81 0.00048 0.00104 0.00237 0.07736 16384 640 rwrites 17.33 30000 1 1731.16 0.00058 0.00136 0.00289 0.07455 16384 1280 rwrites 17.33 30000 1 1731.48 0.00058 0.00140 0.00316 0.04627 16384 1280 TEST SECONDS COUNT DPTH IOPS MEAN P95 P99 MAX ES EC rreads 10.70 30000 1 2802.74 0.00036 0.00047 0.00062 0.01743 16384 640 rreads 10.63 30000 1 2821.25 0.00036 0.00047 0.00061 0.01747 16384 640 rreads 10.66 30000 1 2814.21 0.00036 0.00047 0.00062 0.01764 16384 1280 rreads 10.58 30000 1 2835.04 0.00035 0.00047 0.00061 0.01764 16384 1280 Perf test finished on February 4, 2024 at 12:42:30 AM UTC