/input/rbuild/work Create region with ES:16384 EC:640 BS:4096 Creating region directory at: /var/tmp/dsc Creating dsc directory at: /tmp/dsc Downstairs region 0 created at /var/tmp/dsc/8810 in 1.0414768 Downstairs region 1 created at /var/tmp/dsc/8820 in 0.8335423 Downstairs region 2 created at /var/tmp/dsc/8830 in 0.94837004 Region set with 3 regions was created Using existing output directory "/tmp/dsc" start ds: 8810 start ds: 8820 start ds: 8830 start access at:127.0.0.1:9998 Starting downstairs at port 8810 Starting downstairs at port 8820 Make output file at "/tmp/dsc/downstairs-8810.txt" [8810][0] reports Starting Starting downstairs at port 8830 Make output file at "/tmp/dsc/downstairs-8820.txt" Make output file at "/tmp/dsc/downstairs-8830.txt" [8820][1] reports Starting [8830][2] reports Starting Downstairs /var/tmp/dsc/8820 port 8820 PID:Some(1184) [8820][1] reports Running Downstairs /var/tmp/dsc/8830 port 8830 PID:Some(1185) [8830][2] reports Running Downstairs /var/tmp/dsc/8810 port 8810 PID:Some(1186) [8810][0] reports Running Jul 10 15:29:14.591 INFO listening, local_addr: 127.0.0.1:9998 Control access at:127.0.0.1:9998 1184 1185 1186 Jul 10 15:29:19.629 INFO accepted connection, remote_addr: 127.0.0.1:34321, local_addr: 127.0.0.1:9998 Main task has work to do, go find it got dsc DisableRestartAll disable restart on all: 3 [8810] Disable keep_running [8820] Disable keep_running [8830] Disable keep_running Jul 10 15:29:19.630 INFO request completed, latency_us: 142, response_code: 204, uri: /disablerestart/all, method: POST, req_id: b9955556-02df-4774-95f7-1d42bb8171d6, remote_addr: 127.0.0.1:34321, local_addr: 127.0.0.1:9998 A new loop begins at July 10, 2024 at 03:29:19 PM UTC /var/tmp/bins/crutest perf -g 1 -t 127.0.0.1:8810 -t 127.0.0.1:8820 -t 127.0.0.1:8830 -c 30000 -q --write-loops 2 --read-loops 2 --perf-out /tmp/perf-ES-16384-EC-640.csv {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-10T15:29:19.653395431Z","hostname":"w-01J2EM4K1MH12Q344X8FMJ1HFS","pid":1194} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"6b815a43cfc10bcc32090edef97cce2a57967dc5\",\n git_commit_timestamp: \"2024-07-10T15:11:32.000000000Z\",\n git_branch: \"renovate/nbd-0.x\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: false,\n opt_level: 3,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-10T15:29:19.6535617Z","hostname":"w-01J2EM4K1MH12Q344X8FMJ1HFS","pid":1194} {"msg":"Upstairs <-> Downstairs Message Version: 8","v":0,"name":"crucible","level":30,"time":"2024-07-10T15:29:19.653572029Z","hostname":"w-01J2EM4K1MH12Q344X8FMJ1HFS","pid":1194} {"msg":"Crucible 5cbcdbb1-5123-4b51-82e8-0a2a74b2aa20 has session id: 2f2a6eae-8fac-4333-89dc-964a09fafbe3","v":0,"name":"crucible","level":30,"time":"2024-07-10T15:29:19.653862012Z","hostname":"w-01J2EM4K1MH12Q344X8FMJ1HFS","pid":1194,"session_id":"2f2a6eae-8fac-4333-89dc-964a09fafbe3"} {"msg":"Upstairs opts: Upstairs UUID: 5cbcdbb1-5123-4b51-82e8-0a2a74b2aa20, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-10T15:29:19.653936563Z","hostname":"w-01J2EM4K1MH12Q344X8FMJ1HFS","pid":1194,"session_id":"2f2a6eae-8fac-4333-89dc-964a09fafbe3"} {"msg":"Crucible stats registered with UUID: 5cbcdbb1-5123-4b51-82e8-0a2a74b2aa20","v":0,"name":"crucible","level":30,"time":"2024-07-10T15:29:19.653951252Z","hostname":"w-01J2EM4K1MH12Q344X8FMJ1HFS","pid":1194,"session_id":"2f2a6eae-8fac-4333-89dc-964a09fafbe3"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:1","v":0,"name":"crucible","level":30,"time":"2024-07-10T15:29:19.690908306Z","hostname":"w-01J2EM4K1MH12Q344X8FMJ1HFS","pid":1194} {"msg":"5cbcdbb1-5123-4b51-82e8-0a2a74b2aa20 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-10T15:29:19.691008994Z","hostname":"w-01J2EM4K1MH12Q344X8FMJ1HFS","pid":1194,"session_id":"2f2a6eae-8fac-4333-89dc-964a09fafbe3"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-10T15:29:19.691028531Z","hostname":"w-01J2EM4K1MH12Q344X8FMJ1HFS","pid":1194,"client":"0","":"downstairs","session_id":"2f2a6eae-8fac-4333-89dc-964a09fafbe3"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-10T15:29:19.691079395Z","hostname":"w-01J2EM4K1MH12Q344X8FMJ1HFS","pid":1194,"client":"0","":"downstairs","session_id":"2f2a6eae-8fac-4333-89dc-964a09fafbe3"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-10T15:29:19.691088914Z","hostname":"w-01J2EM4K1MH12Q344X8FMJ1HFS","pid":1194,"client":"1","":"downstairs","session_id":"2f2a6eae-8fac-4333-89dc-964a09fafbe3"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-10T15:29:19.691096143Z","hostname":"w-01J2EM4K1MH12Q344X8FMJ1HFS","pid":1194,"client":"1","":"downstairs","session_id":"2f2a6eae-8fac-4333-89dc-964a09fafbe3"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-10T15:29:19.691103692Z","hostname":"w-01J2EM4K1MH12Q344X8FMJ1HFS","pid":1194,"client":"2","":"downstairs","session_id":"2f2a6eae-8fac-4333-89dc-964a09fafbe3"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-10T15:29:19.691112111Z","hostname":"w-01J2EM4K1MH12Q344X8FMJ1HFS","pid":1194,"client":"2","":"downstairs","session_id":"2f2a6eae-8fac-4333-89dc-964a09fafbe3"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-10T15:29:19.691209349Z","hostname":"w-01J2EM4K1MH12Q344X8FMJ1HFS","pid":1194,"":"io task","client":"0","":"downstairs","session_id":"2f2a6eae-8fac-4333-89dc-964a09fafbe3"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-10T15:29:19.691225287Z","hostname":"w-01J2EM4K1MH12Q344X8FMJ1HFS","pid":1194,"":"io task","client":"2","":"downstairs","session_id":"2f2a6eae-8fac-4333-89dc-964a09fafbe3"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-10T15:29:19.691272251Z","hostname":"w-01J2EM4K1MH12Q344X8FMJ1HFS","pid":1194,"":"io task","client":"1","":"downstairs","session_id":"2f2a6eae-8fac-4333-89dc-964a09fafbe3"} {"msg":"ds_connection connected from Ok(127.0.0.1:36091)","v":0,"name":"crucible","level":30,"time":"2024-07-10T15:29:19.691416123Z","hostname":"w-01J2EM4K1MH12Q344X8FMJ1HFS","pid":1194,"":"io task","client":"1","":"downstairs","session_id":"2f2a6eae-8fac-4333-89dc-964a09fafbe3"} {"msg":"ds_connection connected from Ok(127.0.0.1:55059)","v":0,"name":"crucible","level":30,"time":"2024-07-10T15:29:19.691433831Z","hostname":"w-01J2EM4K1MH12Q344X8FMJ1HFS","pid":1194,"":"io task","client":"0","":"downstairs","session_id":"2f2a6eae-8fac-4333-89dc-964a09fafbe3"} {"msg":"ds_connection connected from Ok(127.0.0.1:62547)","v":0,"name":"crucible","level":30,"time":"2024-07-10T15:29:19.69144334Z","hostname":"w-01J2EM4K1MH12Q344X8FMJ1HFS","pid":1194,"":"io task","client":"2","":"downstairs","session_id":"2f2a6eae-8fac-4333-89dc-964a09fafbe3"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-10T15:29:19.692101788Z","hostname":"w-01J2EM4K1MH12Q344X8FMJ1HFS","pid":1194,"client":"1","":"downstairs","session_id":"2f2a6eae-8fac-4333-89dc-964a09fafbe3"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-10T15:29:19.692111766Z","hostname":"w-01J2EM4K1MH12Q344X8FMJ1HFS","pid":1194,"client":"1","":"downstairs","session_id":"2f2a6eae-8fac-4333-89dc-964a09fafbe3"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-10T15:29:19.692156811Z","hostname":"w-01J2EM4K1MH12Q344X8FMJ1HFS","pid":1194,"client":"2","":"downstairs","session_id":"2f2a6eae-8fac-4333-89dc-964a09fafbe3"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-10T15:29:19.69216249Z","hostname":"w-01J2EM4K1MH12Q344X8FMJ1HFS","pid":1194,"client":"2","":"downstairs","session_id":"2f2a6eae-8fac-4333-89dc-964a09fafbe3"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-10T15:29:19.692169489Z","hostname":"w-01J2EM4K1MH12Q344X8FMJ1HFS","pid":1194,"client":"0","":"downstairs","session_id":"2f2a6eae-8fac-4333-89dc-964a09fafbe3"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-10T15:29:19.692177028Z","hostname":"w-01J2EM4K1MH12Q344X8FMJ1HFS","pid":1194,"client":"0","":"downstairs","session_id":"2f2a6eae-8fac-4333-89dc-964a09fafbe3"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-10T15:29:19.692421578Z","hostname":"w-01J2EM4K1MH12Q344X8FMJ1HFS","pid":1194,"client":"0","":"downstairs","session_id":"2f2a6eae-8fac-4333-89dc-964a09fafbe3"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-10T15:29:19.692451704Z","hostname":"w-01J2EM4K1MH12Q344X8FMJ1HFS","pid":1194,"client":"1","":"downstairs","session_id":"2f2a6eae-8fac-4333-89dc-964a09fafbe3"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-10T15:29:19.692498208Z","hostname":"w-01J2EM4K1MH12Q344X8FMJ1HFS","pid":1194,"client":"2","":"downstairs","session_id":"2f2a6eae-8fac-4333-89dc-964a09fafbe3"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-10T15:29:19.694604746Z","hostname":"w-01J2EM4K1MH12Q344X8FMJ1HFS","pid":1194,"client":"1","":"downstairs","session_id":"2f2a6eae-8fac-4333-89dc-964a09fafbe3"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-10T15:29:19.694627183Z","hostname":"w-01J2EM4K1MH12Q344X8FMJ1HFS","pid":1194,"client":"1","":"downstairs","session_id":"2f2a6eae-8fac-4333-89dc-964a09fafbe3"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-10T15:29:19.694636652Z","hostname":"w-01J2EM4K1MH12Q344X8FMJ1HFS","pid":1194,"session_id":"2f2a6eae-8fac-4333-89dc-964a09fafbe3"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-10T15:29:19.694707543Z","hostname":"w-01J2EM4K1MH12Q344X8FMJ1HFS","pid":1194,"client":"0","":"downstairs","session_id":"2f2a6eae-8fac-4333-89dc-964a09fafbe3"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-10T15:29:19.694726011Z","hostname":"w-01J2EM4K1MH12Q344X8FMJ1HFS","pid":1194,"client":"0","":"downstairs","session_id":"2f2a6eae-8fac-4333-89dc-964a09fafbe3"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-10T15:29:19.69473139Z","hostname":"w-01J2EM4K1MH12Q344X8FMJ1HFS","pid":1194,"session_id":"2f2a6eae-8fac-4333-89dc-964a09fafbe3"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-10T15:29:19.694806471Z","hostname":"w-01J2EM4K1MH12Q344X8FMJ1HFS","pid":1194,"client":"2","":"downstairs","session_id":"2f2a6eae-8fac-4333-89dc-964a09fafbe3"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-10T15:29:19.694820279Z","hostname":"w-01J2EM4K1MH12Q344X8FMJ1HFS","pid":1194,"client":"2","":"downstairs","session_id":"2f2a6eae-8fac-4333-89dc-964a09fafbe3"} {"msg":"[0]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-07-10T15:29:19.694824958Z","hostname":"w-01J2EM4K1MH12Q344X8FMJ1HFS","pid":1194,"":"downstairs","":"downstairs","session_id":"2f2a6eae-8fac-4333-89dc-964a09fafbe3"} {"msg":"[0]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-07-10T15:29:19.694829838Z","hostname":"w-01J2EM4K1MH12Q344X8FMJ1HFS","pid":1194,"":"downstairs","":"downstairs","session_id":"2f2a6eae-8fac-4333-89dc-964a09fafbe3"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-10T15:29:19.694834297Z","hostname":"w-01J2EM4K1MH12Q344X8FMJ1HFS","pid":1194,"":"downstairs","":"downstairs","session_id":"2f2a6eae-8fac-4333-89dc-964a09fafbe3"} {"msg":"[1]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-07-10T15:29:19.694839426Z","hostname":"w-01J2EM4K1MH12Q344X8FMJ1HFS","pid":1194,"":"downstairs","":"downstairs","session_id":"2f2a6eae-8fac-4333-89dc-964a09fafbe3"} {"msg":"[1]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-07-10T15:29:19.694845166Z","hostname":"w-01J2EM4K1MH12Q344X8FMJ1HFS","pid":1194,"":"downstairs","":"downstairs","session_id":"2f2a6eae-8fac-4333-89dc-964a09fafbe3"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-10T15:29:19.694849545Z","hostname":"w-01J2EM4K1MH12Q344X8FMJ1HFS","pid":1194,"":"downstairs","":"downstairs","session_id":"2f2a6eae-8fac-4333-89dc-964a09fafbe3"} {"msg":"[2]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-07-10T15:29:19.694853805Z","hostname":"w-01J2EM4K1MH12Q344X8FMJ1HFS","pid":1194,"":"downstairs","":"downstairs","session_id":"2f2a6eae-8fac-4333-89dc-964a09fafbe3"} {"msg":"[2]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-07-10T15:29:19.694858454Z","hostname":"w-01J2EM4K1MH12Q344X8FMJ1HFS","pid":1194,"":"downstairs","":"downstairs","session_id":"2f2a6eae-8fac-4333-89dc-964a09fafbe3"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-10T15:29:19.694862834Z","hostname":"w-01J2EM4K1MH12Q344X8FMJ1HFS","pid":1194,"":"downstairs","":"downstairs","session_id":"2f2a6eae-8fac-4333-89dc-964a09fafbe3"} {"msg":"Max found gen is 1","v":0,"name":"crucible","level":30,"time":"2024-07-10T15:29:19.694867013Z","hostname":"w-01J2EM4K1MH12Q344X8FMJ1HFS","pid":1194,"":"downstairs","":"downstairs","session_id":"2f2a6eae-8fac-4333-89dc-964a09fafbe3"} {"msg":"Generation requested: 1 >= found:1","v":0,"name":"crucible","level":30,"time":"2024-07-10T15:29:19.694872113Z","hostname":"w-01J2EM4K1MH12Q344X8FMJ1HFS","pid":1194,"":"downstairs","":"downstairs","session_id":"2f2a6eae-8fac-4333-89dc-964a09fafbe3"} {"msg":"Next flush: 1","v":0,"name":"crucible","level":30,"time":"2024-07-10T15:29:19.694876492Z","hostname":"w-01J2EM4K1MH12Q344X8FMJ1HFS","pid":1194,"":"downstairs","":"downstairs","session_id":"2f2a6eae-8fac-4333-89dc-964a09fafbe3"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-10T15:29:19.694880701Z","hostname":"w-01J2EM4K1MH12Q344X8FMJ1HFS","pid":1194,"":"downstairs","":"downstairs","session_id":"2f2a6eae-8fac-4333-89dc-964a09fafbe3"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-10T15:29:19.694885451Z","hostname":"w-01J2EM4K1MH12Q344X8FMJ1HFS","pid":1194,"session_id":"2f2a6eae-8fac-4333-89dc-964a09fafbe3"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-10T15:29:19.69488954Z","hostname":"w-01J2EM4K1MH12Q344X8FMJ1HFS","pid":1194,"client":"0","":"downstairs","session_id":"2f2a6eae-8fac-4333-89dc-964a09fafbe3"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-10T15:29:19.69489391Z","hostname":"w-01J2EM4K1MH12Q344X8FMJ1HFS","pid":1194,"client":"1","":"downstairs","session_id":"2f2a6eae-8fac-4333-89dc-964a09fafbe3"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-10T15:29:19.694898549Z","hostname":"w-01J2EM4K1MH12Q344X8FMJ1HFS","pid":1194,"client":"2","":"downstairs","session_id":"2f2a6eae-8fac-4333-89dc-964a09fafbe3"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-10T15:29:19.694902809Z","hostname":"w-01J2EM4K1MH12Q344X8FMJ1HFS","pid":1194,"session_id":"2f2a6eae-8fac-4333-89dc-964a09fafbe3"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-10T15:29:19.694906528Z","hostname":"w-01J2EM4K1MH12Q344X8FMJ1HFS","pid":1194,"session_id":"2f2a6eae-8fac-4333-89dc-964a09fafbe3"} {"msg":"5cbcdbb1-5123-4b51-82e8-0a2a74b2aa20 is now active with session: 2f2a6eae-8fac-4333-89dc-964a09fafbe3","v":0,"name":"crucible","level":30,"time":"2024-07-10T15:29:19.694910318Z","hostname":"w-01J2EM4K1MH12Q344X8FMJ1HFS","pid":1194,"session_id":"2f2a6eae-8fac-4333-89dc-964a09fafbe3"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-10T15:29:19.694914377Z","hostname":"w-01J2EM4K1MH12Q344X8FMJ1HFS","pid":1194,"session_id":"2f2a6eae-8fac-4333-89dc-964a09fafbe3"} {"msg":"The guest has finished waiting for activation with:1","v":0,"name":"crucible"Wait for a query_work_queue command to finish before sending IO ,"level":30,"time":"2024-07-10T15:29:19.694928666Z","hostname":"w-01J2EM4K1MH12Q344X8FMJ1HFS","pid":1194} Region: es:16384 ec:640 bs:4096 ts:42949672960 tb:10485760 max_io:256 or 1048576 Perf test TEST SECONDS COUNT DPTH IOPS MEAN P95 P99 MAX ES EC rwrites 6.86 30000 1 4374.94 0.00023 0.00107 0.00168 0.05400 16384 640 rwrites 6.85 30000 1 4382.66 0.00023 0.00116 0.00177 0.03472 16384 640 rreads 5.60 30000 1 5361.33 0.00019 0.00023 0.00028 0.01326 16384 640 rreads 5.56 30000 1 5395.30 0.00019 0.00023 0.00028 0.01355 16384 640 Perf test completed, stop all downstairs Jul 10 15:29:49.165 INFO accepted connection, remote_addr: 127.0.0.1:35611, local_addr: 127.0.0.1:9998 Main task has work to do, go find it got dsc Shutdown Shutdown Jul 10 15:29:49.166 INFO request completed, latency_us: 62, [8810] Disable keep_running response_code[8810] Got stop action so:false kr:false :[8820] Disable keep_running [8820] Got stop action so:false kr:false 204, uri: /shutdown, method: POST, req_id: c0cd78c7-3055-46dc-8809-f43ae413705f, remote_addr: 127.0.0.1:35611, Shut it down local_addr: 127.0.0.1:9998 [8830] Disable keep_running [8830] Got stop action so:false kr:false [8820] Exited with: ExitStatus(unix_wait_status(9)) [8820][1] reports Exit [8810] Exited with: ExitStatus(unix_wait_status(9)) [8830] Exited with: ExitStatus(unix_wait_status(9)) [8810][0] reports Exit [8830][2] reports Exit Create region with ES:16384 EC:1280 BS:4096 Removing existing dsc directory "/tmp/dsc" Removing existing region "/var/tmp/dsc" Creating region directory at: /var/tmp/dsc Creating dsc directory at: /tmp/dsc Downstairs region 0 created at /var/tmp/dsc/8810 in 1.4701059 Downstairs region 1 created at /var/tmp/dsc/8820 in 1.4853597 Downstairs region 2 created at /var/tmp/dsc/8830 in 1.5423983 Region set with 3 regions was created Using existing output directory "/tmp/dsc" start ds: 8810 start ds: 8820 start ds: 8830 start access at:127.0.0.1:9998 Starting downstairs at port 8810 Make output file at "/tmp/dsc/downstairs-8810.txt" Starting downstairs at port 8820 Make output file at "/tmp/dsc/downstairs-8820.txt" [8810][0] reports Starting Starting downstairs at port 8830 [8820][1] reports Starting Make output file at "/tmp/dsc/downstairs-8830.txt" [8830][2] reports Starting Downstairs /var/tmp/dsc/8810 port 8810 PID:Some(1203) [8810][0] reports Running Downstairs /var/tmp/dsc/8830 port 8830 PID:Some(1204) [8830][2] reports Running Downstairs /var/tmp/dsc/8820 port 8820 PID:Some(1205) [8820][1] reports Running Jul 10 15:29:54.725 INFO listening, local_addr: 127.0.0.1:9998 Control access at:127.0.0.1:9998 1205 1204 1203 Jul 10 15:29:59.763 INFO accepted connection, remote_addr: 127.0.0.1:55064, local_addr: 127.0.0.1:9998 Main task has work to do, go find it got dsc DisableRestartAll disable restart on all: 3 [8820] Disable keep_running Jul 10 15:29:59.764[8810] Disable keep_running INFO request completed, [8830] Disable keep_running latency_us: 114, response_code: 204, uri: /disablerestart/all, method: POST, req_id: 701a2338-ff71-439d-a734-c23cb7f61877, remote_addr: 127.0.0.1:55064, local_addr: 127.0.0.1:9998 A new loop begins at July 10, 2024 at 03:29:59 PM UTC /var/tmp/bins/crutest perf -g 1 -t 127.0.0.1:8810 -t 127.0.0.1:8820 -t 127.0.0.1:8830 -c 30000 -q --write-loops 2 --read-loops 2 --perf-out /tmp/perf-ES-16384-EC-1280.csv {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-10T15:29:59.784834563Z","hostname":"w-01J2EM4K1MH12Q344X8FMJ1HFS","pid":1213} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"6b815a43cfc10bcc32090edef97cce2a57967dc5\",\n git_commit_timestamp: \"2024-07-10T15:11:32.000000000Z\",\n git_branch: \"renovate/nbd-0.x\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: false,\n opt_level: 3,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-10T15:29:59.785163992Z","hostname":"w-01J2EM4K1MH12Q344X8FMJ1HFS","pid":1213} {"msg":"Upstairs <-> Downstairs Message Version: 8","v":0,"name":"crucible","level":30,"time":"2024-07-10T15:29:59.785171941Z","hostname":"w-01J2EM4K1MH12Q344X8FMJ1HFS","pid":1213} {"msg":"Crucible 7e0a62dc-0149-4084-8c20-de906fb01d88 has session id: d6aa4ac0-f1ee-4340-9766-c27ef6d81a82","v":0,"name":"crucible","level":30,"time":"2024-07-10T15:29:59.785497381Z","hostname":"w-01J2EM4K1MH12Q344X8FMJ1HFS","pid":1213,"session_id":"d6aa4ac0-f1ee-4340-9766-c27ef6d81a82"} {"msg":"Upstairs opts: Upstairs UUID: 7e0a62dc-0149-4084-8c20-de906fb01d88, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-10T15:29:59.785535206Z","hostname":"w-01J2EM4K1MH12Q344X8FMJ1HFS","pid":1213,"session_id":"d6aa4ac0-f1ee-4340-9766-c27ef6d81a82"} {"msg":"Crucible stats registered with UUID: 7e0a62dc-0149-4084-8c20-de906fb01d88","v":0,"name":"crucible","level":30,"time":"2024-07-10T15:29:59.785542095Z","hostname":"w-01J2EM4K1MH12Q344X8FMJ1HFS","pid":1213,"session_id":"d6aa4ac0-f1ee-4340-9766-c27ef6d81a82"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:1","v":0,"name":"crucible","level":30,"time":"2024-07-10T15:29:59.822390474Z","hostname":"w-01J2EM4K1MH12Q344X8FMJ1HFS","pid":1213} {"msg":"7e0a62dc-0149-4084-8c20-de906fb01d88 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-10T15:29:59.822450596Z","hostname":"w-01J2EM4K1MH12Q344X8FMJ1HFS","pid":1213,"session_id":"d6aa4ac0-f1ee-4340-9766-c27ef6d81a82"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-10T15:29:59.822464794Z","hostname":"w-01J2EM4K1MH12Q344X8FMJ1HFS","pid":1213,"client":"0","":"downstairs","session_id":"d6aa4ac0-f1ee-4340-9766-c27ef6d81a82"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-10T15:29:59.822480562Z","hostname":"w-01J2EM4K1MH12Q344X8FMJ1HFS","pid":1213,"client":"0","":"downstairs","session_id":"d6aa4ac0-f1ee-4340-9766-c27ef6d81a82"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-10T15:29:59.822486202Z","hostname":"w-01J2EM4K1MH12Q344X8FMJ1HFS","pid":1213,"client":"1","":"downstairs","session_id":"d6aa4ac0-f1ee-4340-9766-c27ef6d81a82"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-10T15:29:59.822490771Z","hostname":"w-01J2EM4K1MH12Q344X8FMJ1HFS","pid":1213,"client":"1","":"downstairs","session_id":"d6aa4ac0-f1ee-4340-9766-c27ef6d81a82"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-10T15:29:59.822495921Z","hostname":"w-01J2EM4K1MH12Q344X8FMJ1HFS","pid":1213,"client":"2","":"downstairs","session_id":"d6aa4ac0-f1ee-4340-9766-c27ef6d81a82"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-10T15:29:59.82250027Z","hostname":"w-01J2EM4K1MH12Q344X8FMJ1HFS","pid":1213,"client":"2","":"downstairs","session_id":"d6aa4ac0-f1ee-4340-9766-c27ef6d81a82"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-10T15:29:59.822569831Z","hostname":"w-01J2EM4K1MH12Q344X8FMJ1HFS","pid":1213,"":"io task","client":"2","":"downstairs","session_id":"d6aa4ac0-f1ee-4340-9766-c27ef6d81a82"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-10T15:29:59.82258288Z","hostname":"w-01J2EM4K1MH12Q344X8FMJ1HFS","pid":1213,"":"io task","client":"0","":"downstairs","session_id":"d6aa4ac0-f1ee-4340-9766-c27ef6d81a82"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-10T15:29:59.822587599Z","hostname":"w-01J2EM4K1MH12Q344X8FMJ1HFS","pid":1213,"":"io task","client":"1","":"downstairs","session_id":"d6aa4ac0-f1ee-4340-9766-c27ef6d81a82"} {"msg":"ds_connection connected from Ok(127.0.0.1:34456)","v":0,"name":"crucible","level":30,"time":"2024-07-10T15:29:59.822731291Z","hostname":"w-01J2EM4K1MH12Q344X8FMJ1HFS","pid":1213,"":"io task","client":"1","":"downstairs","session_id":"d6aa4ac0-f1ee-4340-9766-c27ef6d81a82"} {"msg":"ds_connection connected from Ok(127.0.0.1:60624)","v":0,"name":"crucible","level":30,"time":"2024-07-10T15:29:59.822745769Z","hostname":"w-01J2EM4K1MH12Q344X8FMJ1HFS","pid":1213,"":"io task","client":"2","":"downstairs","session_id":"d6aa4ac0-f1ee-4340-9766-c27ef6d81a82"} {"msg":"ds_connection connected from Ok(127.0.0.1:36130)","v":0,"name":"crucible","level":30,"time":"2024-07-10T15:29:59.822754388Z","hostname":"w-01J2EM4K1MH12Q344X8FMJ1HFS","pid":1213,"":"io task","client":"0","":"downstairs","session_id":"d6aa4ac0-f1ee-4340-9766-c27ef6d81a82"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-10T15:29:59.823181935Z","hostname":"w-01J2EM4K1MH12Q344X8FMJ1HFS","pid":1213,"client":"1","":"downstairs","session_id":"d6aa4ac0-f1ee-4340-9766-c27ef6d81a82"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-10T15:29:59.823203332Z","hostname":"w-01J2EM4K1MH12Q344X8FMJ1HFS","pid":1213,"client":"1","":"downstairs","session_id":"d6aa4ac0-f1ee-4340-9766-c27ef6d81a82"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-10T15:29:59.823212981Z","hostname":"w-01J2EM4K1MH12Q344X8FMJ1HFS","pid":1213,"client":"2","":"downstairs","session_id":"d6aa4ac0-f1ee-4340-9766-c27ef6d81a82"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-10T15:29:59.823234798Z","hostname":"w-01J2EM4K1MH12Q344X8FMJ1HFS","pid":1213,"client":"2","":"downstairs","session_id":"d6aa4ac0-f1ee-4340-9766-c27ef6d81a82"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-10T15:29:59.823243767Z","hostname":"w-01J2EM4K1MH12Q344X8FMJ1HFS","pid":1213,"client":"0","":"downstairs","session_id":"d6aa4ac0-f1ee-4340-9766-c27ef6d81a82"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-10T15:29:59.823271624Z","hostname":"w-01J2EM4K1MH12Q344X8FMJ1HFS","pid":1213,"client":"0","":"downstairs","session_id":"d6aa4ac0-f1ee-4340-9766-c27ef6d81a82"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-10T15:29:59.823480438Z","hostname":"w-01J2EM4K1MH12Q344X8FMJ1HFS","pid":1213,"client":"1","":"downstairs","session_id":"d6aa4ac0-f1ee-4340-9766-c27ef6d81a82"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-10T15:29:59.823506664Z","hostname":"w-01J2EM4K1MH12Q344X8FMJ1HFS","pid":1213,"client":"2","":"downstairs","session_id":"d6aa4ac0-f1ee-4340-9766-c27ef6d81a82"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-10T15:29:59.823562477Z","hostname":"w-01J2EM4K1MH12Q344X8FMJ1HFS","pid":1213,"client":"0","":"downstairs","session_id":"d6aa4ac0-f1ee-4340-9766-c27ef6d81a82"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-10T15:29:59.828844619Z","hostname":"w-01J2EM4K1MH12Q344X8FMJ1HFS","pid":1213,"client":"1","":"downstairs","session_id":"d6aa4ac0-f1ee-4340-9766-c27ef6d81a82"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-10T15:29:59.828856538Z","hostname":"w-01J2EM4K1MH12Q344X8FMJ1HFS","pid":1213,"client":"1","":"downstairs","session_id":"d6aa4ac0-f1ee-4340-9766-c27ef6d81a82"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-10T15:29:59.828864117Z","hostname":"w-01J2EM4K1MH12Q344X8FMJ1HFS","pid":1213,"session_id":"d6aa4ac0-f1ee-4340-9766-c27ef6d81a82"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-10T15:29:59.828868916Z","hostname":"w-01J2EM4K1MH12Q344X8FMJ1HFS","pid":1213,"client":"2","":"downstairs","session_id":"d6aa4ac0-f1ee-4340-9766-c27ef6d81a82"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-10T15:29:59.828873356Z","hostname":"w-01J2EM4K1MH12Q344X8FMJ1HFS","pid":1213,"client":"2","":"downstairs","session_id":"d6aa4ac0-f1ee-4340-9766-c27ef6d81a82"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-10T15:29:59.828879605Z","hostname":"w-01J2EM4K1MH12Q344X8FMJ1HFS","pid":1213,"session_id":"d6aa4ac0-f1ee-4340-9766-c27ef6d81a82"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-10T15:29:59.828893433Z","hostname":"w-01J2EM4K1MH12Q344X8FMJ1HFS","pid":1213,"client":"0","":"downstairs","session_id":"d6aa4ac0-f1ee-4340-9766-c27ef6d81a82"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-10T15:29:59.828906692Z","hostname":"w-01J2EM4K1MH12Q344X8FMJ1HFS","pid":1213,"client":"0","":"downstairs","session_id":"d6aa4ac0-f1ee-4340-9766-c27ef6d81a82"} {"msg":"[0]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-07-10T15:29:59.828911241Z","hostname":"w-01J2EM4K1MH12Q344X8FMJ1HFS","pid":1213,"":"downstairs","":"downstairs","session_id":"d6aa4ac0-f1ee-4340-9766-c27ef6d81a82"} {"msg":"[0]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-07-10T15:29:59.82891684Z","hostname":"w-01J2EM4K1MH12Q344X8FMJ1HFS","pid":1213,"":"downstairs","":"downstairs","session_id":"d6aa4ac0-f1ee-4340-9766-c27ef6d81a82"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-10T15:29:59.82892186Z","hostname":"w-01J2EM4K1MH12Q344X8FMJ1HFS","pid":1213,"":"downstairs","":"downstairs","session_id":"d6aa4ac0-f1ee-4340-9766-c27ef6d81a82"} {"msg":"[1]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-07-10T15:29:59.828926019Z","hostname":"w-01J2EM4K1MH12Q344X8FMJ1HFS","pid":1213,"":"downstairs","":"downstairs","session_id":"d6aa4ac0-f1ee-4340-9766-c27ef6d81a82"} {"msg":"[1]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-07-10T15:29:59.828930969Z","hostname":"w-01J2EM4K1MH12Q344X8FMJ1HFS","pid":1213,"":"downstairs","":"downstairs","session_id":"d6aa4ac0-f1ee-4340-9766-c27ef6d81a82"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-10T15:29:59.828935238Z","hostname":"w-01J2EM4K1MH12Q344X8FMJ1HFS","pid":1213,"":"downstairs","":"downstairs","session_id":"d6aa4ac0-f1ee-4340-9766-c27ef6d81a82"} {"msg":"[2]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-07-10T15:29:59.828939427Z","hostname":"w-01J2EM4K1MH12Q344X8FMJ1HFS","pid":1213,"":"downstairs","":"downstairs","session_id":"d6aa4ac0-f1ee-4340-9766-c27ef6d81a82"} {"msg":"[2]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-07-10T15:29:59.828943537Z","hostname":"w-01J2EM4K1MH12Q344X8FMJ1HFS","pid":1213,"":"downstairs","":"downstairs","session_id":"d6aa4ac0-f1ee-4340-9766-c27ef6d81a82"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-10T15:29:59.828947917Z","hostname":"w-01J2EM4K1MH12Q344X8FMJ1HFS","pid":1213,"":"downstairs","":"downstairs","session_id":"d6aa4ac0-f1ee-4340-9766-c27ef6d81a82"} {"msg":"Max found gen is 1","v":0,"name":"crucible","level":30,"time":"2024-07-10T15:29:59.828952176Z","hostname":"w-01J2EM4K1MH12Q344X8FMJ1HFS","pid":1213,"":"downstairs","":"downstairs","session_id":"d6aa4ac0-f1ee-4340-9766-c27ef6d81a82"} {"msg":"Generation requested: 1 >= found:1","v":0,"name":"crucible","level":30,"time":"2024-07-10T15:29:59.828957095Z","hostname":"w-01J2EM4K1MH12Q344X8FMJ1HFS","pid":1213,"":"downstairs","":"downstairs","session_id":"d6aa4ac0-f1ee-4340-9766-c27ef6d81a82"} {"msg":"Next flush: 1","v":0,"name":"crucible","level":30,"time":"2024-07-10T15:29:59.828961515Z","hostname":"w-01J2EM4K1MH12Q344X8FMJ1HFS","pid":1213,"":"downstairs","":"downstairs","session_id":"d6aa4ac0-f1ee-4340-9766-c27ef6d81a82"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-10T15:29:59.828965634Z","hostname":"w-01J2EM4K1MH12Q344X8FMJ1HFS","pid":1213,"":"downstairs","":"downstairs","session_id":"d6aa4ac0-f1ee-4340-9766-c27ef6d81a82"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-10T15:29:59.828970234Z","hostname":"w-01J2EM4K1MH12Q344X8FMJ1HFS","pid":1213,"session_id":"d6aa4ac0-f1ee-4340-9766-c27ef6d81a82"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-10T15:29:59.828973823Z","hostname":"w-01J2EM4K1MH12Q344X8FMJ1HFS","pid":1213,"client":"0","":"downstairs","session_id":"d6aa4ac0-f1ee-4340-9766-c27ef6d81a82"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-10T15:29:59.828977923Z","hostname":"w-01J2EM4K1MH12Q344X8FMJ1HFS","pid":1213,"client":"1","":"downstairs","session_id":"d6aa4ac0-f1ee-4340-9766-c27ef6d81a82"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-10T15:29:59.828981952Z","hostname":"w-01J2EM4K1MH12Q344X8FMJ1HFS","pid":1213,"client":"2","":"downstairs","session_id":"d6aa4ac0-f1ee-4340-9766-c27ef6d81a82"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-10T15:29:59.828985972Z","hostname":"w-01J2EM4K1MH12Q344X8FMJ1HFS","pid":1213,"session_id":"d6aa4ac0-f1ee-4340-9766-c27ef6d81a82"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2024-07-10T15:29:59.828991821Z","hostname":"w-01J2EM4K1MH12Q344X8FMJ1HFS","pid":1213,"session_id":"d6aa4ac0-f1ee-4340-9766-c27ef6d81a82"} {"msg":"7e0a62dc-0149-4084-8c20-de906fb01d88 is now active with session: d6aa4ac0-f1ee-4340-9766-c27ef6d81a82","v":0,"name":"crucible","level":30,"time":"2024-07-10T15:29:59.82900261Z","hostname":"w-01J2EM4K1MH12Q344X8FMJ1HFS","pid":1213,"session_id":"d6aa4ac0-f1ee-4340-9766-c27ef6d81a82"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-10T15:29:59.829010229Z","hostname":"w-01J2EM4K1MH12Q344X8FMJ1HFS","pid":1213,"session_id":"d6aa4ac0-f1ee-4340-9766-c27ef6d81a82"} {"msg":"The guest has finished waiting for activation with:1","v":0,"name":"crucible","level":30,"time":"2024-07-10T15:29:59.829014168Z","hostname":"w-01J2EM4K1MH12Q344X8FMJ1HFS","pid":1213} Region: es:16384 ec:1280 bs:4096 ts:85899345920 tb:20971520 max_io:256 or 1048576 Perf test TEST SECONDS COUNT DPTH IOPS MEAN P95 P99 MAX ES EC rwrites 9.36 30000 1 3203.74 0.00031 0.00125 0.00184 0.07250 16384 1280 rwrites 8.92 30000 1 3365.07 0.00030 0.00123 0.00176 0.04342 16384 1280 rreads 5.63 30000 1 5326.02 0.00019 0.00023 0.00029 0.01320 16384 1280 rreads 5.66 30000 1 5304.95 0.00019 0.00023 0.00029 0.01320 16384 1280 Perf test completed, stop all downstairs Jul 10 15:30:32.162 INFO accepted connection, remote_addr: 127.0.0.1:65424, local_addr: 127.0.0.1:9998 Main task has work to do, go find it got dsc Shutdown Shutdown Jul 10 15:30:32.163 [8810] Disable keep_running INFOShut it down [8830] Disable keep_running request completed, latency_us: 71, response_code[8830] Got stop action so:false kr:false : 204, uri: /shutdown, method: POST, req_id: f41db537-13ee-43a5-bc95-7fbb014a6079, remote_addr: 127.0.[8820] Disable keep_running 0[8820] Got stop action so:false kr:false .[8810] Got stop action so:false kr:false 1:65424, local_addr: 127.0.0.1:9998 [8810] Exited with: ExitStatus(unix_wait_status(9)) [8820] Exited with: ExitStatus(unix_wait_status(9)) [8810][0] reports Exit [8820][1] reports Exit [8830] Exited with: ExitStatus(unix_wait_status(9)) [8830][2] reports Exit TEST SECONDS COUNT DPTH IOPS MEAN P95 P99 MAX ES EC rwrites 6.86 30000 1 4374.94 0.00023 0.00107 0.00168 0.05400 16384 640 rwrites 6.85 30000 1 4382.66 0.00023 0.00116 0.00177 0.03472 16384 640 rwrites 9.36 30000 1 3203.74 0.00031 0.00125 0.00184 0.07250 16384 1280 rwrites 8.92 30000 1 3365.07 0.00030 0.00123 0.00176 0.04342 16384 1280 TEST SECONDS COUNT DPTH IOPS MEAN P95 P99 MAX ES EC rreads 5.60 30000 1 5361.33 0.00019 0.00023 0.00028 0.01326 16384 640 rreads 5.56 30000 1 5395.30 0.00019 0.00023 0.00028 0.01355 16384 640 rreads 5.63 30000 1 5326.02 0.00019 0.00023 0.00029 0.01320 16384 1280 rreads 5.66 30000 1 5304.95 0.00019 0.00023 0.00029 0.01320 16384 1280 Perf test finished on July 10, 2024 at 03:30:33 PM UTC