/input/rbuild/work Create region with ES:16384 EC:640 BS:4096 Creating region directory at: /var/tmp/dsc Creating dsc directory at: /tmp/dsc Downstairs region 0 created at /var/tmp/dsc/8810 in 0.9162714 Downstairs region 1 created at /var/tmp/dsc/8820 in 0.6094762 Downstairs region 2 created at /var/tmp/dsc/8830 in 0.6940147 Created 3 regions Using existing output directory "/tmp/dsc" Update our region info with: Some(RegionExtentInfo { block_size: 4096, blocks_per_extent: 16384, extent_count: 640 }) start ds: 8810 start access at:127.0.0.1:9998 start ds: 8820 start ds: 8830 Starting downstairs at port 8810 Starting downstairs at port 8820 Make output file at "/tmp/dsc/downstairs-8820.txt" Make output file at "/tmp/dsc/downstairs-8810.txt" Starting downstairs at port 8830 [8810][0] initial start wait reports Starting [8820][1] initial start wait reports Starting [8830][2] initial start wait reports Starting Make output file at "/tmp/dsc/downstairs-8830.txt" Downstairs /var/tmp/dsc/8820 port 8820 PID:Some(1184) [8820][1] initial start wait reports Running Downstairs /var/tmp/dsc/8830 port 8830 PID:Some(1185) [8830][2] initial start wait reports Running Downstairs /var/tmp/dsc/8810 port 8810 PID:Some(1186) [8810][0] initial start wait reports Running All downstairs are running Sep 27 17:13:55.328 INFO listening, local_addr: 127.0.0.1:9998 Control access at:127.0.0.1:9998 1184 1185 1186 Sep 27 17:14:00.366 INFO accepted connection, remote_addr: 127.0.0.1:45893, local_addr: 127.0.0.1:9998 Main task has work to do, go find it got dsc DisableRestartAll disable restart on all: 3 Sep 27 17:14:00.367 INFO request completed, latency_us: 127, response_code: 204, uri: /disablerestart/all, method: POST, req_id: c846331f-01f4-4314-b27c-fb989738d1fe, remote_addr: 127.0.0.1:45893, local_addr: [8810] Disable keep_running 127[8820] Disable keep_running .0.0.1:9998 [8830] Disable keep_running A new loop begins at September 27, 2024 at 05:14:00 PM UTC /var/tmp/bins/crutest perf -g 1 -t 127.0.0.1:8810 -t 127.0.0.1:8820 -t 127.0.0.1:8830 -c 30000 -q --write-loops 2 --read-loops 2 --perf-out /tmp/perf-ES-16384-EC-640.csv {"msg":"look at: http://127.0.0.1:12810 ","v":0,"name":"crucible","level":30,"time":"2024-09-27T17:14:00.390924038Z","hostname":"w-01J8T7H5Q4H83A62H00MD9AFWN","pid":1194,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 16384, shift: 12 }, extent_count: 640, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2024-09-27T17:14:00.431362595Z","hostname":"w-01J8T7H5Q4H83A62H00MD9AFWN","pid":1194,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-09-27T17:14:00.431915407Z","hostname":"w-01J8T7H5Q4H83A62H00MD9AFWN","pid":1194} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"93b12f70ee7c1993c73ac4256d99ff80ccbe4bac\",\n git_commit_timestamp: \"2024-09-27T16:52:31.000000000Z\",\n git_branch: \"update-omicron-common\",\n rustc_semver: \"1.80.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"051478957371ee0084a7c0913941d2a8c4757bb9\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: false,\n opt_level: 3,\n}","v":0,"name":"crucible","level":30,"time":"2024-09-27T17:14:00.431944596Z","hostname":"w-01J8T7H5Q4H83A62H00MD9AFWN","pid":1194} {"msg":"Upstairs <-> Downstairs Message Version: 11","v":0,"name":"crucible","level":30,"time":"2024-09-27T17:14:00.431951645Z","hostname":"w-01J8T7H5Q4H83A62H00MD9AFWN","pid":1194} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 16384, shift: 12 }, extent_count: 640, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2024-09-27T17:14:00.431957095Z","hostname":"w-01J8T7H5Q4H83A62H00MD9AFWN","pid":1194} {"msg":"Crucible a1b1819d-a688-4d52-8fa9-9156dbc6e7b4 has session id: 79ad00af-3093-426c-94d7-4aa80060138e","v":0,"name":"crucible","level":30,"time":"2024-09-27T17:14:00.432364137Z","hostname":"w-01J8T7H5Q4H83A62H00MD9AFWN","pid":1194,"session_id":"79ad00af-3093-426c-94d7-4aa80060138e"} {"msg":"Upstairs opts: Upstairs UUID: a1b1819d-a688-4d52-8fa9-9156dbc6e7b4, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-09-27T17:14:00.432455851Z","hostname":"w-01J8T7H5Q4H83A62H00MD9AFWN","pid":1194,"session_id":"79ad00af-3093-426c-94d7-4aa80060138e"} {"msg":"Crucible stats registered with UUID: a1b1819d-a688-4d52-8fa9-9156dbc6e7b4","v":0,"name":"crucible","level":30,"time":"2024-09-27T17:14:00.432476239Z","hostname":"w-01J8T7H5Q4H83A62H00MD9AFWN","pid":1194,"session_id":"79ad00af-3093-426c-94d7-4aa80060138e"} {"msg":"The guest has requested activation with gen:1","v":0,"name":"crucible","level":30,"time":"2024-09-27T17:14:00.46272267Z","hostname":"w-01J8T7H5Q4H83A62H00MD9AFWN","pid":1194} {"msg":"a1b1819d-a688-4d52-8fa9-9156dbc6e7b4 active request set","v":0,"name":"crucible","level":30,"time":"2024-09-27T17:14:00.462743289Z","hostname":"w-01J8T7H5Q4H83A62H00MD9AFWN","pid":1194,"session_id":"79ad00af-3093-426c-94d7-4aa80060138e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-09-27T17:14:00.462814934Z","hostname":"w-01J8T7H5Q4H83A62H00MD9AFWN","pid":1194,"client":"0","":"downstairs","session_id":"79ad00af-3093-426c-94d7-4aa80060138e"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-09-27T17:14:00.462853421Z","hostname":"w-01J8T7H5Q4H83A62H00MD9AFWN","pid":1194,"client":"0","":"downstairs","session_id":"79ad00af-3093-426c-94d7-4aa80060138e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-09-27T17:14:00.462860051Z","hostname":"w-01J8T7H5Q4H83A62H00MD9AFWN","pid":1194,"client":"1","":"downstairs","session_id":"79ad00af-3093-426c-94d7-4aa80060138e"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-09-27T17:14:00.4628656Z","hostname":"w-01J8T7H5Q4H83A62H00MD9AFWN","pid":1194,"client":"1","":"downstairs","session_id":"79ad00af-3093-426c-94d7-4aa80060138e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-09-27T17:14:00.46287104Z","hostname":"w-01J8T7H5Q4H83A62H00MD9AFWN","pid":1194,"client":"2","":"downstairs","session_id":"79ad00af-3093-426c-94d7-4aa80060138e"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-09-27T17:14:00.462876639Z","hostname":"w-01J8T7H5Q4H83A62H00MD9AFWN","pid":1194,"client":"2","":"downstairs","session_id":"79ad00af-3093-426c-94d7-4aa80060138e"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-09-27T17:14:00.462881989Z","hostname":"w-01J8T7H5Q4H83A62H00MD9AFWN","pid":1194,"":"io task","client":"2","":"downstairs","session_id":"79ad00af-3093-426c-94d7-4aa80060138e"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-09-27T17:14:00.462888369Z","hostname":"w-01J8T7H5Q4H83A62H00MD9AFWN","pid":1194,"":"io task","client":"1","":"downstairs","session_id":"79ad00af-3093-426c-94d7-4aa80060138e"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-09-27T17:14:00.462950505Z","hostname":"w-01J8T7H5Q4H83A62H00MD9AFWN","pid":1194,"":"io task","client":"0","":"downstairs","session_id":"79ad00af-3093-426c-94d7-4aa80060138e"} {"msg":"ds_connection connected from Ok(127.0.0.1:42711)","v":0,"name":"crucible","level":30,"time":"2024-09-27T17:14:00.463040118Z","hostname":"w-01J8T7H5Q4H83A62H00MD9AFWN","pid":1194,"":"io task","client":"2","":"downstairs","session_id":"79ad00af-3093-426c-94d7-4aa80060138e"} {"msg":"ds_connection connected from Ok(127.0.0.1:39937)","v":0,"name":"crucible","level":30,"time":"2024-09-27T17:14:00.463054537Z","hostname":"w-01J8T7H5Q4H83A62H00MD9AFWN","pid":1194,"":"io task","client":"1","":"downstairs","session_id":"79ad00af-3093-426c-94d7-4aa80060138e"} {"msg":"ds_connection connected from Ok(127.0.0.1:37709)","v":0,"name":"crucible","level":30,"time":"2024-09-27T17:14:00.463088725Z","hostname":"w-01J8T7H5Q4H83A62H00MD9AFWN","pid":1194,"":"io task","client":"0","":"downstairs","session_id":"79ad00af-3093-426c-94d7-4aa80060138e"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-09-27T17:14:00.46346257Z","hostname":"w-01J8T7H5Q4H83A62H00MD9AFWN","pid":1194,"client":"0","":"downstairs","session_id":"79ad00af-3093-426c-94d7-4aa80060138e"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-09-27T17:14:00.463484878Z","hostname":"w-01J8T7H5Q4H83A62H00MD9AFWN","pid":1194,"client":"0","":"downstairs","session_id":"79ad00af-3093-426c-94d7-4aa80060138e"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-09-27T17:14:00.463493728Z","hostname":"w-01J8T7H5Q4H83A62H00MD9AFWN","pid":1194,"client":"1","":"downstairs","session_id":"79ad00af-3093-426c-94d7-4aa80060138e"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-09-27T17:14:00.463502367Z","hostname":"w-01J8T7H5Q4H83A62H00MD9AFWN","pid":1194,"client":"1","":"downstairs","session_id":"79ad00af-3093-426c-94d7-4aa80060138e"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-09-27T17:14:00.463534715Z","hostname":"w-01J8T7H5Q4H83A62H00MD9AFWN","pid":1194,"client":"2","":"downstairs","session_id":"79ad00af-3093-426c-94d7-4aa80060138e"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-09-27T17:14:00.463544864Z","hostname":"w-01J8T7H5Q4H83A62H00MD9AFWN","pid":1194,"client":"2","":"downstairs","session_id":"79ad00af-3093-426c-94d7-4aa80060138e"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-09-27T17:14:00.463732221Z","hostname":"w-01J8T7H5Q4H83A62H00MD9AFWN","pid":1194,"client":"0","":"downstairs","session_id":"79ad00af-3093-426c-94d7-4aa80060138e"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-09-27T17:14:00.46374984Z","hostname":"w-01J8T7H5Q4H83A62H00MD9AFWN","pid":1194,"client":"1","":"downstairs","session_id":"79ad00af-3093-426c-94d7-4aa80060138e"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-09-27T17:14:00.46375533Z","hostname":"w-01J8T7H5Q4H83A62H00MD9AFWN","pid":1194,"client":"2","":"downstairs","session_id":"79ad00af-3093-426c-94d7-4aa80060138e"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-09-27T17:14:00.46596158Z","hostname":"w-01J8T7H5Q4H83A62H00MD9AFWN","pid":1194,"client":"1","":"downstairs","session_id":"79ad00af-3093-426c-94d7-4aa80060138e"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-09-27T17:14:00.465983118Z","hostname":"w-01J8T7H5Q4H83A62H00MD9AFWN","pid":1194,"client":"1","":"downstairs","session_id":"79ad00af-3093-426c-94d7-4aa80060138e"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-09-27T17:14:00.465988968Z","hostname":"w-01J8T7H5Q4H83A62H00MD9AFWN","pid":1194,"session_id":"79ad00af-3093-426c-94d7-4aa80060138e"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-09-27T17:14:00.465993017Z","hostname":"w-01J8T7H5Q4H83A62H00MD9AFWN","pid":1194,"client":"0","":"downstairs","session_id":"79ad00af-3093-426c-94d7-4aa80060138e"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-09-27T17:14:00.466013976Z","hostname":"w-01J8T7H5Q4H83A62H00MD9AFWN","pid":1194,"client":"0","":"downstairs","session_id":"79ad00af-3093-426c-94d7-4aa80060138e"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-09-27T17:14:00.466018366Z","hostname":"w-01J8T7H5Q4H83A62H00MD9AFWN","pid":1194,"session_id":"79ad00af-3093-426c-94d7-4aa80060138e"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-09-27T17:14:00.46624115Z","hostname":"w-01J8T7H5Q4H83A62H00MD9AFWN","pid":1194,"client":"2","":"downstairs","session_id":"79ad00af-3093-426c-94d7-4aa80060138e"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-09-27T17:14:00.466254649Z","hostname":"w-01J8T7H5Q4H83A62H00MD9AFWN","pid":1194,"client":"2","":"downstairs","session_id":"79ad00af-3093-426c-94d7-4aa80060138e"} {"msg":"[0]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-09-27T17:14:00.466327244Z","hostname":"w-01J8T7H5Q4H83A62H00MD9AFWN","pid":1194,"":"downstairs","":"downstairs","session_id":"79ad00af-3093-426c-94d7-4aa80060138e"} {"msg":"[0]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-09-27T17:14:00.466339254Z","hostname":"w-01J8T7H5Q4H83A62H00MD9AFWN","pid":1194,"":"downstairs","":"downstairs","session_id":"79ad00af-3093-426c-94d7-4aa80060138e"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-09-27T17:14:00.466343774Z","hostname":"w-01J8T7H5Q4H83A62H00MD9AFWN","pid":1194,"":"downstairs","":"downstairs","session_id":"79ad00af-3093-426c-94d7-4aa80060138e"} {"msg":"[1]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-09-27T17:14:00.466347993Z","hostname":"w-01J8T7H5Q4H83A62H00MD9AFWN","pid":1194,"":"downstairs","":"downstairs","session_id":"79ad00af-3093-426c-94d7-4aa80060138e"} {"msg":"[1]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-09-27T17:14:00.466353893Z","hostname":"w-01J8T7H5Q4H83A62H00MD9AFWN","pid":1194,"":"downstairs","":"downstairs","session_id":"79ad00af-3093-426c-94d7-4aa80060138e"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-09-27T17:14:00.466358282Z","hostname":"w-01J8T7H5Q4H83A62H00MD9AFWN","pid":1194,"":"downstairs","":"downstairs","session_id":"79ad00af-3093-426c-94d7-4aa80060138e"} {"msg":"[2]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-09-27T17:14:00.466362492Z","hostname":"w-01J8T7H5Q4H83A62H00MD9AFWN","pid":1194,"":"downstairs","":"downstairs","session_id":"79ad00af-3093-426c-94d7-4aa80060138e"} {"msg":"[2]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-09-27T17:14:00.466367062Z","hostname":"w-01J8T7H5Q4H83A62H00MD9AFWN","pid":1194,"":"downstairs","":"downstairs","session_id":"79ad00af-3093-426c-94d7-4aa80060138e"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-09-27T17:14:00.466371812Z","hostname":"w-01J8T7H5Q4H83A62H00MD9AFWN","pid":1194,"":"downstairs","":"downstairs","session_id":"79ad00af-3093-426c-94d7-4aa80060138e"} {"msg":"Max found gen is 1","v":0,"name":"crucible","level":30,"time":"2024-09-27T17:14:00.466375981Z","hostname":"w-01J8T7H5Q4H83A62H00MD9AFWN","pid":1194,"":"downstairs","":"downstairs","session_id":"79ad00af-3093-426c-94d7-4aa80060138e"} {"msg":"Generation requested: 1 >= found:1","v":0,"name":"crucible","level":30,"time":"2024-09-27T17:14:00.466380631Z","hostname":"w-01J8T7H5Q4H83A62H00MD9AFWN","pid":1194,"":"downstairs","":"downstairs","session_id":"79ad00af-3093-426c-94d7-4aa80060138e"} {"msg":"Next flush: 1","v":0,"name":"crucible","level":30,"time":"2024-09-27T17:14:00.466385201Z","hostname":"w-01J8T7H5Q4H83A62H00MD9AFWN","pid":1194,"":"downstairs","":"downstairs","session_id":"79ad00af-3093-426c-94d7-4aa80060138e"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-09-27T17:14:00.4663893Z","hostname":"w-01J8T7H5Q4H83A62H00MD9AFWN","pid":1194,"":"downstairs","":"downstairs","session_id":"79ad00af-3093-426c-94d7-4aa80060138e"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-09-27T17:14:00.46639349Z","hostname":"w-01J8T7H5Q4H83A62H00MD9AFWN","pid":1194,"session_id":"79ad00af-3093-426c-94d7-4aa80060138e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-09-27T17:14:00.46639766Z","hostname":"w-01J8T7H5Q4H83A62H00MD9AFWN","pid":1194,"client":"0","":"downstairs","session_id":"79ad00af-3093-426c-94d7-4aa80060138e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-09-27T17:14:00.46640231Z","hostname":"w-01J8T7H5Q4H83A62H00MD9AFWN","pid":1194,"client":"1","":"downstairs","session_id":"79ad00af-3093-426c-94d7-4aa80060138e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-09-27T17:14:00.466406339Z","hostname":"w-01J8T7H5Q4H83A62H00MD9AFWN","pid":1194,"client":"2","":"downstairs","session_id":"79ad00af-3093-426c-94d7-4aa80060138e"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-09-27T17:14:00.466410869Z","hostname":"w-01J8T7H5Q4H83A62H00MD9AFWN","pid":1194,"session_id":"79ad00af-3093-426c-94d7-4aa80060138e"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-09-27T17:14:00.466414549Z","hostname":"w-01J8T7H5Q4H83A62H00MD9AFWN","pid":1194,"session_id":"79ad00af-3093-426c-94d7-4aa80060138e"} {"msg":"a1b1819d-a688-4d52-8fa9-9156dbc6e7b4 is now active with session: 79ad00af-3093-426c-94d7-4aa80060138e","v":0,"name":"crucible","level":30,"time":"2024-09-27T17:14:00.466418158Z","hostname":"w-01J8T7H5Q4H83A62H00MD9AFWN","pid":1194,"session_id":"79ad00af-3093-426c-94d7-4aa80060138e"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-09-27T17:14:00.466421688Z","hostname":"w-01J8T7H5Q4H83A62H00MD9AFWN","pid":1194,"session_id":"79ad00af-3093-426c-94d7-4aa80060138e"} {"msg":"The guest has finished waiting for activation with:1","v":0,"name":"crucible","level":30,"time":"2024-09-27T17:14:00.466425728Z","hostname":"w-01J8T7H5Q4H83A62H00MD9AFWN","pid":1194} Wait for a query_work_queue command to finish before sending IO Region: es:16384 ec:640 bs:4096 ts:42949672960 tb:10485760 max_io:256 or 1048576 Perf test TEST SECONDS COUNT DPTH IOPS MEAN P95 P99 MAX ES EC rwrites 5.41 30000 1 5548.50 0.00018 0.00047 0.00127 0.07029 16384 640 rwrites 5.41 30000 1 5542.36 0.00018 0.00058 0.00126 0.06491 16384 640 rreads 5.54 30000 1 5412.32 0.00018 0.00023 0.00029 0.00631 16384 640 rreads 5.48 30000 1 5473.24 0.00018 0.00023 0.00028 0.00651 16384 640 Perf test completed, stop all downstairs Sep 27 17:14:24.754 INFO accepted connection, remote_addr: 127.0.0.1:55237, local_addr: 127.0.0.1:9998 Main task has work to do, go find it got dsc Shutdown Shutdown Shut it down [8810] Disable keep_running Sep 27 17:14:24.755[8810] Got stop action so:false kr:false [8820] Disable keep_running INFO[8820] Got stop action so:false kr:false request completed, latency_us: 57, response_code: 204, uri: /shutdown, method: POST, req_id: 3ecf39a6-433e-4e5d-b862-3bbcbc73aac2, remote_addr: 127.0.0.1:55237, local_addr: 127.0.0.1:9998 [8830] Disable keep_running [8830] Got stop action so:false kr:false [8810] Exited with: ExitStatus(unix_wait_status(9)) [8830] Exited with: ExitStatus(unix_wait_status(9)) [8820] Exited with: ExitStatus(unix_wait_status(9)) [8810][0] reports Exit [8830][2] reports Exit [8820][1] reports Exit Create region with ES:16384 EC:1280 BS:4096 Removing existing dsc directory "/tmp/dsc" Removing existing region "/var/tmp/dsc" Creating region directory at: /var/tmp/dsc Creating dsc directory at: /tmp/dsc Downstairs region 0 created at /var/tmp/dsc/8810 in 1.1565613 Downstairs region 1 created at /var/tmp/dsc/8820 in 1.2067287 Downstairs region 2 created at /var/tmp/dsc/8830 in 1.0872108 Created 3 regions Using existing output directory "/tmp/dsc" Update our region info with: Some(RegionExtentInfo { block_size: 4096, blocks_per_extent: 16384, extent_count: 1280 }) start ds: 8810 start ds: 8820 start ds: 8830 start access at:127.0.0.1:9998 Starting downstairs at port 8810 Starting downstairs at port 8820 Make output file at "/tmp/dsc/downstairs-8820.txt" Make output file at "/tmp/dsc/downstairs-8810.txt" [8810][0] initial start wait reports Starting Starting downstairs at port 8830 [8820][1] initial start wait reports Starting [8830][2] initial start wait reports Starting Make output file at "/tmp/dsc/downstairs-8830.txt" Downstairs /var/tmp/dsc/8810 port 8810 PID:Some(1203) [8810][0] initial start wait reports Running Downstairs /var/tmp/dsc/8830 port 8830 PID:Some(1204) [8830][2] initial start wait reports Running Downstairs /var/tmp/dsc/8820 port 8820 PID:Some(1205) [8820][1] initial start wait reports Running All downstairs are running Sep 27 17:14:29.270 INFO listening, local_addr: 127.0.0.1:9998 Control access at:127.0.0.1:9998 1205 1203 1204 Sep 27 17:14:34.308 INFO accepted connection, remote_addr: 127.0.0.1:53302, local_addr: 127.0.0.1:9998 Main task has work to do, go find it got dsc DisableRestartAll disable restart on all: 3 Sep 27 17:14:34.309 INFO request completed, latency_us: 116, response_code: 204, uri: /disablerestart/all, method: POST, req_id: 42f44a76-807d-45d0-b6f8-ae430704f2e5, remote_addr: [8810] Disable keep_running [8830] Disable keep_running [8820] Disable keep_running 127.0.0.1:53302, local_addr: 127.0.0.1:9998 A new loop begins at September 27, 2024 at 05:14:34 PM UTC /var/tmp/bins/crutest perf -g 1 -t 127.0.0.1:8810 -t 127.0.0.1:8820 -t 127.0.0.1:8830 -c 30000 -q --write-loops 2 --read-loops 2 --perf-out /tmp/perf-ES-16384-EC-1280.csv {"msg":"look at: http://127.0.0.1:12810 ","v":0,"name":"crucible","level":30,"time":"2024-09-27T17:14:34.330184021Z","hostname":"w-01J8T7H5Q4H83A62H00MD9AFWN","pid":1213,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 16384, shift: 12 }, extent_count: 1280, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2024-09-27T17:14:34.370647876Z","hostname":"w-01J8T7H5Q4H83A62H00MD9AFWN","pid":1213,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-09-27T17:14:34.371083506Z","hostname":"w-01J8T7H5Q4H83A62H00MD9AFWN","pid":1213} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"93b12f70ee7c1993c73ac4256d99ff80ccbe4bac\",\n git_commit_timestamp: \"2024-09-27T16:52:31.000000000Z\",\n git_branch: \"update-omicron-common\",\n rustc_semver: \"1.80.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"051478957371ee0084a7c0913941d2a8c4757bb9\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: false,\n opt_level: 3,\n}","v":0,"name":"crucible","level":30,"time":"2024-09-27T17:14:34.371109684Z","hostname":"w-01J8T7H5Q4H83A62H00MD9AFWN","pid":1213} {"msg":"Upstairs <-> Downstairs Message Version: 11","v":0,"name":"crucible","level":30,"time":"2024-09-27T17:14:34.371116624Z","hostname":"w-01J8T7H5Q4H83A62H00MD9AFWN","pid":1213} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 16384, shift: 12 }, extent_count: 1280, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2024-09-27T17:14:34.371122213Z","hostname":"w-01J8T7H5Q4H83A62H00MD9AFWN","pid":1213} {"msg":"Crucible b19f0c8e-cec9-49fd-8388-a7de4c0993f4 has session id: 1e6ac906-7158-4c9e-82fa-e80b88058d96","v":0,"name":"crucible","level":30,"time":"2024-09-27T17:14:34.371456341Z","hostname":"w-01J8T7H5Q4H83A62H00MD9AFWN","pid":1213,"session_id":"1e6ac906-7158-4c9e-82fa-e80b88058d96"} {"msg":"Upstairs opts: Upstairs UUID: b19f0c8e-cec9-49fd-8388-a7de4c0993f4, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-09-27T17:14:34.371488048Z","hostname":"w-01J8T7H5Q4H83A62H00MD9AFWN","pid":1213,"session_id":"1e6ac906-7158-4c9e-82fa-e80b88058d96"} {"msg":"Crucible stats registered with UUID: b19f0c8e-cec9-49fd-8388-a7de4c0993f4","v":0,"name":"crucible","level":30,"time":"2024-09-27T17:14:34.371498628Z","hostname":"w-01J8T7H5Q4H83A62H00MD9AFWN","pid":1213,"session_id":"1e6ac906-7158-4c9e-82fa-e80b88058d96"} {"msg":"The guest has requested activation with gen:1","v":0,"name":"crucible","level":30,"time":"2024-09-27T17:14:34.401781306Z","hostname":"w-01J8T7H5Q4H83A62H00MD9AFWN","pid":1213} {"msg":"b19f0c8e-cec9-49fd-8388-a7de4c0993f4 active request set","v":0,"name":"crucible","level":30,"time":"2024-09-27T17:14:34.401800205Z","hostname":"w-01J8T7H5Q4H83A62H00MD9AFWN","pid":1213,"session_id":"1e6ac906-7158-4c9e-82fa-e80b88058d96"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-09-27T17:14:34.401806074Z","hostname":"w-01J8T7H5Q4H83A62H00MD9AFWN","pid":1213,"client":"0","":"downstairs","session_id":"1e6ac906-7158-4c9e-82fa-e80b88058d96"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-09-27T17:14:34.401818043Z","hostname":"w-01J8T7H5Q4H83A62H00MD9AFWN","pid":1213,"client":"0","":"downstairs","session_id":"1e6ac906-7158-4c9e-82fa-e80b88058d96"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-09-27T17:14:34.401824783Z","hostname":"w-01J8T7H5Q4H83A62H00MD9AFWN","pid":1213,"client":"1","":"downstairs","session_id":"1e6ac906-7158-4c9e-82fa-e80b88058d96"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-09-27T17:14:34.401830213Z","hostname":"w-01J8T7H5Q4H83A62H00MD9AFWN","pid":1213,"client":"1","":"downstairs","session_id":"1e6ac906-7158-4c9e-82fa-e80b88058d96"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-09-27T17:14:34.401838772Z","hostname":"w-01J8T7H5Q4H83A62H00MD9AFWN","pid":1213,"client":"2","":"downstairs","session_id":"1e6ac906-7158-4c9e-82fa-e80b88058d96"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-09-27T17:14:34.401848092Z","hostname":"w-01J8T7H5Q4H83A62H00MD9AFWN","pid":1213,"client":"2","":"downstairs","session_id":"1e6ac906-7158-4c9e-82fa-e80b88058d96"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-09-27T17:14:34.401857931Z","hostname":"w-01J8T7H5Q4H83A62H00MD9AFWN","pid":1213,"":"io task","client":"2","":"downstairs","session_id":"1e6ac906-7158-4c9e-82fa-e80b88058d96"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-09-27T17:14:34.401900248Z","hostname":"w-01J8T7H5Q4H83A62H00MD9AFWN","pid":1213,"":"io task","client":"0","":"downstairs","session_id":"1e6ac906-7158-4c9e-82fa-e80b88058d96"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-09-27T17:14:34.401920336Z","hostname":"w-01J8T7H5Q4H83A62H00MD9AFWN","pid":1213,"":"io task","client":"1","":"downstairs","session_id":"1e6ac906-7158-4c9e-82fa-e80b88058d96"} {"msg":"ds_connection connected from Ok(127.0.0.1:43554)","v":0,"name":"crucible","level":30,"time":"2024-09-27T17:14:34.401980662Z","hostname":"w-01J8T7H5Q4H83A62H00MD9AFWN","pid":1213,"":"io task","client":"0","":"downstairs","session_id":"1e6ac906-7158-4c9e-82fa-e80b88058d96"} {"msg":"ds_connection connected from Ok(127.0.0.1:44131)","v":0,"name":"crucible","level":30,"time":"2024-09-27T17:14:34.402076696Z","hostname":"w-01J8T7H5Q4H83A62H00MD9AFWN","pid":1213,"":"io task","client":"1","":"downstairs","session_id":"1e6ac906-7158-4c9e-82fa-e80b88058d96"} {"msg":"ds_connection connected from Ok(127.0.0.1:37212)","v":0,"name":"crucible","level":30,"time":"2024-09-27T17:14:34.402089545Z","hostname":"w-01J8T7H5Q4H83A62H00MD9AFWN","pid":1213,"":"io task","client":"2","":"downstairs","session_id":"1e6ac906-7158-4c9e-82fa-e80b88058d96"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-09-27T17:14:34.402303161Z","hostname":"w-01J8T7H5Q4H83A62H00MD9AFWN","pid":1213,"client":"0","":"downstairs","session_id":"1e6ac906-7158-4c9e-82fa-e80b88058d96"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-09-27T17:14:34.402321669Z","hostname":"w-01J8T7H5Q4H83A62H00MD9AFWN","pid":1213,"client":"0","":"downstairs","session_id":"1e6ac906-7158-4c9e-82fa-e80b88058d96"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-09-27T17:14:34.402467819Z","hostname":"w-01J8T7H5Q4H83A62H00MD9AFWN","pid":1213,"client":"1","":"downstairs","session_id":"1e6ac906-7158-4c9e-82fa-e80b88058d96"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-09-27T17:14:34.402488818Z","hostname":"w-01J8T7H5Q4H83A62H00MD9AFWN","pid":1213,"client":"1","":"downstairs","session_id":"1e6ac906-7158-4c9e-82fa-e80b88058d96"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-09-27T17:14:34.402495317Z","hostname":"w-01J8T7H5Q4H83A62H00MD9AFWN","pid":1213,"client":"2","":"downstairs","session_id":"1e6ac906-7158-4c9e-82fa-e80b88058d96"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-09-27T17:14:34.402499647Z","hostname":"w-01J8T7H5Q4H83A62H00MD9AFWN","pid":1213,"client":"2","":"downstairs","session_id":"1e6ac906-7158-4c9e-82fa-e80b88058d96"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-09-27T17:14:34.402630898Z","hostname":"w-01J8T7H5Q4H83A62H00MD9AFWN","pid":1213,"client":"0","":"downstairs","session_id":"1e6ac906-7158-4c9e-82fa-e80b88058d96"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-09-27T17:14:34.402651607Z","hostname":"w-01J8T7H5Q4H83A62H00MD9AFWN","pid":1213,"client":"1","":"downstairs","session_id":"1e6ac906-7158-4c9e-82fa-e80b88058d96"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-09-27T17:14:34.402669696Z","hostname":"w-01J8T7H5Q4H83A62H00MD9AFWN","pid":1213,"client":"2","":"downstairs","session_id":"1e6ac906-7158-4c9e-82fa-e80b88058d96"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-09-27T17:14:34.408095266Z","hostname":"w-01J8T7H5Q4H83A62H00MD9AFWN","pid":1213,"client":"0","":"downstairs","session_id":"1e6ac906-7158-4c9e-82fa-e80b88058d96"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-09-27T17:14:34.408108185Z","hostname":"w-01J8T7H5Q4H83A62H00MD9AFWN","pid":1213,"client":"0","":"downstairs","session_id":"1e6ac906-7158-4c9e-82fa-e80b88058d96"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-09-27T17:14:34.408116165Z","hostname":"w-01J8T7H5Q4H83A62H00MD9AFWN","pid":1213,"session_id":"1e6ac906-7158-4c9e-82fa-e80b88058d96"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-09-27T17:14:34.408122294Z","hostname":"w-01J8T7H5Q4H83A62H00MD9AFWN","pid":1213,"client":"2","":"downstairs","session_id":"1e6ac906-7158-4c9e-82fa-e80b88058d96"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-09-27T17:14:34.408129774Z","hostname":"w-01J8T7H5Q4H83A62H00MD9AFWN","pid":1213,"client":"2","":"downstairs","session_id":"1e6ac906-7158-4c9e-82fa-e80b88058d96"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-09-27T17:14:34.408136443Z","hostname":"w-01J8T7H5Q4H83A62H00MD9AFWN","pid":1213,"session_id":"1e6ac906-7158-4c9e-82fa-e80b88058d96"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-09-27T17:14:34.408142953Z","hostname":"w-01J8T7H5Q4H83A62H00MD9AFWN","pid":1213,"client":"1","":"downstairs","session_id":"1e6ac906-7158-4c9e-82fa-e80b88058d96"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-09-27T17:14:34.408151152Z","hostname":"w-01J8T7H5Q4H83A62H00MD9AFWN","pid":1213,"client":"1","":"downstairs","session_id":"1e6ac906-7158-4c9e-82fa-e80b88058d96"} {"msg":"[0]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-09-27T17:14:34.408155632Z","hostname":"w-01J8T7H5Q4H83A62H00MD9AFWN","pid":1213,"":"downstairs","":"downstairs","session_id":"1e6ac906-7158-4c9e-82fa-e80b88058d96"} {"msg":"[0]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-09-27T17:14:34.408160202Z","hostname":"w-01J8T7H5Q4H83A62H00MD9AFWN","pid":1213,"":"downstairs","":"downstairs","session_id":"1e6ac906-7158-4c9e-82fa-e80b88058d96"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-09-27T17:14:34.408173751Z","hostname":"w-01J8T7H5Q4H83A62H00MD9AFWN","pid":1213,"":"downstairs","":"downstairs","session_id":"1e6ac906-7158-4c9e-82fa-e80b88058d96"} {"msg":"[1]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-09-27T17:14:34.40817837Z","hostname":"w-01J8T7H5Q4H83A62H00MD9AFWN","pid":1213,"":"downstairs","":"downstairs","session_id":"1e6ac906-7158-4c9e-82fa-e80b88058d96"} {"msg":"[1]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-09-27T17:14:34.40818266Z","hostname":"w-01J8T7H5Q4H83A62H00MD9AFWN","pid":1213,"":"downstairs","":"downstairs","session_id":"1e6ac906-7158-4c9e-82fa-e80b88058d96"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-09-27T17:14:34.4081874Z","hostname":"w-01J8T7H5Q4H83A62H00MD9AFWN","pid":1213,"":"downstairs","":"downstairs","session_id":"1e6ac906-7158-4c9e-82fa-e80b88058d96"} {"msg":"[2]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-09-27T17:14:34.40819184Z","hostname":"w-01J8T7H5Q4H83A62H00MD9AFWN","pid":1213,"":"downstairs","":"downstairs","session_id":"1e6ac906-7158-4c9e-82fa-e80b88058d96"} {"msg":"[2]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-09-27T17:14:34.408196569Z","hostname":"w-01J8T7H5Q4H83A62H00MD9AFWN","pid":1213,"":"downstairs","":"downstairs","session_id":"1e6ac906-7158-4c9e-82fa-e80b88058d96"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-09-27T17:14:34.408200779Z","hostname":"w-01J8T7H5Q4H83A62H00MD9AFWN","pid":1213,"":"downstairs","":"downstairs","session_id":"1e6ac906-7158-4c9e-82fa-e80b88058d96"} {"msg":"Max found gen is 1","v":0,"name":"crucible","level":30,"time":"2024-09-27T17:14:34.408204869Z","hostname":"w-01J8T7H5Q4H83A62H00MD9AFWN","pid":1213,"":"downstairs","":"downstairs","session_id":"1e6ac906-7158-4c9e-82fa-e80b88058d96"} {"msg":"Generation requested: 1 >= found:1","v":0,"name":"crucible","level":30,"time":"2024-09-27T17:14:34.408208968Z","hostname":"w-01J8T7H5Q4H83A62H00MD9AFWN","pid":1213,"":"downstairs","":"downstairs","session_id":"1e6ac906-7158-4c9e-82fa-e80b88058d96"} {"msg":"Next flush: 1","v":0,"name":"crucible","level":30,"time":"2024-09-27T17:14:34.408213618Z","hostname":"w-01J8T7H5Q4H83A62H00MD9AFWN","pid":1213,"":"downstairs","":"downstairs","session_id":"1e6ac906-7158-4c9e-82fa-e80b88058d96"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-09-27T17:14:34.408217708Z","hostname":"w-01J8T7H5Q4H83A62H00MD9AFWN","pid":1213,"":"downstairs","":"downstairs","session_id":"1e6ac906-7158-4c9e-82fa-e80b88058d96"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-09-27T17:14:34.408221627Z","hostname":"w-01J8T7H5Q4H83A62H00MD9AFWN","pid":1213,"session_id":"1e6ac906-7158-4c9e-82fa-e80b88058d96"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-09-27T17:14:34.408225037Z","hostname":"w-01J8T7H5Q4H83A62H00MD9AFWN","pid":1213,"client":"0","":"downstairs","session_id":"1e6ac906-7158-4c9e-82fa-e80b88058d96"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-09-27T17:14:34.408229067Z","hostname":"w-01J8T7H5Q4H83A62H00MD9AFWN","pid":1213,"client":"1","":"downstairs","session_id":"1e6ac906-7158-4c9e-82fa-e80b88058d96"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-09-27T17:14:34.408233007Z","hostname":"w-01J8T7H5Q4H83A62H00MD9AFWN","pid":1213,"client":"2","":"downstairs","session_id":"1e6ac906-7158-4c9e-82fa-e80b88058d96"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-09-27T17:14:34.408246396Z","hostname":"w-01J8T7H5Q4H83A62H00MD9AFWN","pid":1213,"session_id":"1e6ac906-7158-4c9e-82fa-e80b88058d96"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-09-27T17:14:34.408252965Z","hostname":"w-01J8T7H5Q4H83A62H00MD9AFWN","pid":1213,"session_id":"1e6ac906-7158-4c9e-82fa-e80b88058d96"} {"msg":"b19f0c8e-cec9-49fd-8388-a7de4c0993f4 is now active with session: 1e6ac906-7158-4c9e-82fa-e80b88058d96"Wait for a query_work_queue command to finish before sending IO ,"v":0,"name":"crucible","level":30,"time":"2024-09-27T17:14:34.408274824Z","hostname":"w-01J8T7H5Q4H83A62H00MD9AFWN","pid":1213,"session_id":"1e6ac906-7158-4c9e-82fa-e80b88058d96"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-09-27T17:14:34.408279014Z","hostname":"w-01J8T7H5Q4H83A62H00MD9AFWN","pid":1213,"session_id":"1e6ac906-7158-4c9e-82fa-e80b88058d96"} {"msg":"The guest has finished waiting for activation with:1","v":0,"name":"crucible","level":30,"time":"2024-09-27T17:14:34.408283713Z","hostname":"w-01J8T7H5Q4H83A62H00MD9AFWN","pid":1213} Region: es:16384 ec:1280 bs:4096 ts:85899345920 tb:20971520 max_io:256 or 1048576 Perf test TEST SECONDS COUNT DPTH IOPS MEAN P95 P99 MAX ES EC rwrites 7.51 30000 1 3993.29 0.00025 0.00117 0.00144 0.06479 16384 1280 rwrites 8.89 30000 1 3374.64 0.00030 0.00119 0.00154 0.07806 16384 1280 rreads 5.54 30000 1 5415.89 0.00018 0.00025 0.00030 0.00712 16384 1280 rreads 5.55 30000 1 5409.03 0.00018 0.00025 0.00030 0.00712 16384 1280 Perf test completed, stop all downstairs Sep 27 17:15:04.430 INFO accepted connection, remote_addr: 127.0.0.1:61171, local_addr: 127.0.0.1:9998 Main task has work to do, go find it got dsc Shutdown Shutdown Sep 27 17:15:04.431 INFO request completed, [8810] Disable keep_running latency_usShut it down : 77, response_code: 204, uri: [8820] Disable keep_running /shutdown[8820] Got stop action so:false kr:false , method: POST, req_id: dda9b7bd-bb31-4470-8c7f-7b727a7ec11b[8810] Got stop action so:false kr:false , remote_addr: 127.0.[8830] Disable keep_running 0.1:61171, local_addr: 127.0.0.1:9998 [8830] Got stop action so:false kr:false [8820] Exited with: ExitStatus(unix_wait_status(9)) [8820][1] reports Exit [8830] Exited with: ExitStatus(unix_wait_status(9)) [8810] Exited with: ExitStatus(unix_wait_status(9)) [8830][2] reports Exit [8810][0] reports Exit TEST SECONDS COUNT DPTH IOPS MEAN P95 P99 MAX ES EC rwrites 5.41 30000 1 5548.50 0.00018 0.00047 0.00127 0.07029 16384 640 rwrites 5.41 30000 1 5542.36 0.00018 0.00058 0.00126 0.06491 16384 640 rwrites 7.51 30000 1 3993.29 0.00025 0.00117 0.00144 0.06479 16384 1280 rwrites 8.89 30000 1 3374.64 0.00030 0.00119 0.00154 0.07806 16384 1280 TEST SECONDS COUNT DPTH IOPS MEAN P95 P99 MAX ES EC rreads 5.54 30000 1 5412.32 0.00018 0.00023 0.00029 0.00631 16384 640 rreads 5.48 30000 1 5473.24 0.00018 0.00023 0.00028 0.00651 16384 640 rreads 5.54 30000 1 5415.89 0.00018 0.00025 0.00030 0.00712 16384 1280 rreads 5.55 30000 1 5409.03 0.00018 0.00025 0.00030 0.00712 16384 1280 Perf test finished on September 27, 2024 at 05:15:05 PM UTC