/input/rbuild/work Create region with ES:16384 EC:640 BS:4096 Creating region directory at: /var/tmp/dsc Creating dsc directory at: /tmp/dsc Downstairs region 0 created at /var/tmp/dsc/8810 in 0.6677032 Downstairs region 1 created at /var/tmp/dsc/8820 in 0.7409872 Downstairs region 2 created at /var/tmp/dsc/8830 in 0.588912 Region set with 3 regions was created Using existing output directory "/tmp/dsc" start ds: 8810 start ds: 8820 start ds: 8830 Starting downstairs at port 8810 Starting downstairs at port 8820 Starting downstairs at port 8830 Make output file at "/tmp/dsc/downstairs-8810.txt" start access at:127.0.0.1:9998 Make output file at "/tmp/dsc/downstairs-8820.txt" Make output file at "/tmp/dsc/downstairs-8830.txt" [8830][2] reports Starting [8810][0] reports Starting [8820][1] reports Starting Downstairs /var/tmp/dsc/8810 port 8810 PID:Some(1182) [8810][0] reports Running Downstairs /var/tmp/dsc/8820 port 8820 PID:Some(1183) [8820][1] reports Running Downstairs /var/tmp/dsc/8830 port 8830 PID:Some(1184) [8830][2] reports Running Apr 20 01:04:25.165 INFO listening, local_addr: 127.0.0.1:9998 Control access at:127.0.0.1:9998 1182 1183 1184 Apr 20 01:04:30.202 INFO accepted connection, remote_addr: 127.0.0.1:54598, local_addr: 127.0.0.1:9998 Main task has work to do, go find it got dsc DisableRestartAll disable restart on all: 3 Apr 20 01:04:30.203 INFO[8810] Disable keep_running [8820] Disable keep_running request completed[8830] Disable keep_running , latency_us: 133, response_code: 204, uri: /disablerestart/all, method: POST, req_id: f1041321-a265-4837-9295-fb421359d552, remote_addr: 127.0.0.1:54598, local_addr: 127.0.0.1:9998 A new loop begins at April 20, 2024 at 01:04:30 AM UTC /var/tmp/bins/crutest perf -g 1 -t 127.0.0.1:8810 -t 127.0.0.1:8820 -t 127.0.0.1:8830 -c 30000 -q --write-loops 2 --read-loops 2 --perf-out /tmp/perf-ES-16384-EC-640.csv {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-04-20T01:04:30.22451792Z","hostname":"w-01HVWGBHJ2K16BGQZGJEFZFPS8","pid":1192} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"b97e434eec561fa80f965a2f40ec87c83a3a439e\",\n git_commit_timestamp: \"2024-04-20T00:34:22.000000000Z\",\n git_branch: \"artemis/remove-reqwest-block-io\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: false,\n opt_level: 3,\n}","v":0,"name":"crucible","level":30,"time":"2024-04-20T01:04:30.224717107Z","hostname":"w-01HVWGBHJ2K16BGQZGJEFZFPS8","pid":1192} {"msg":"Upstairs <-> Downstairs Message Version: 6","v":0,"name":"crucible","level":30,"time":"2024-04-20T01:04:30.224723276Z","hostname":"w-01HVWGBHJ2K16BGQZGJEFZFPS8","pid":1192} {"msg":"Crucible f3e88e50-0682-46ea-a497-ae5a7fcd620a has session id: 07abe336-2aff-49af-bdea-6d1e3473f6d6","v":0,"name":"crucible","level":30,"time":"2024-04-20T01:04:30.224727606Z","hostname":"w-01HVWGBHJ2K16BGQZGJEFZFPS8","pid":1192,"session_id":"07abe336-2aff-49af-bdea-6d1e3473f6d6"} {"msg":"Upstairs opts: Upstairs UUID: f3e88e50-0682-46ea-a497-ae5a7fcd620a, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-04-20T01:04:30.224777023Z","hostname":"w-01HVWGBHJ2K16BGQZGJEFZFPS8","pid":1192,"session_id":"07abe336-2aff-49af-bdea-6d1e3473f6d6"} {"msg":"Crucible stats registered with UUID: f3e88e50-0682-46ea-a497-ae5a7fcd620a","v":0,"name":"crucible","level":30,"time":"2024-04-20T01:04:30.224790922Z","hostname":"w-01HVWGBHJ2K16BGQZGJEFZFPS8","pid":1192,"session_id":"07abe336-2aff-49af-bdea-6d1e3473f6d6"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:1","v":0,"name":"crucible","level":30,"time":"2024-04-20T01:04:30.224985479Z","hostname":"w-01HVWGBHJ2K16BGQZGJEFZFPS8","pid":1192} {"msg":"f3e88e50-0682-46ea-a497-ae5a7fcd620a active request set","v":0,"name":"crucible","level":30,"time":"2024-04-20T01:04:30.225175696Z","hostname":"w-01HVWGBHJ2K16BGQZGJEFZFPS8","pid":1192,"session_id":"07abe336-2aff-49af-bdea-6d1e3473f6d6"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-04-20T01:04:30.225190505Z","hostname":"w-01HVWGBHJ2K16BGQZGJEFZFPS8","pid":1192,"client":"0","":"downstairs","session_id":"07abe336-2aff-49af-bdea-6d1e3473f6d6"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-04-20T01:04:30.225196044Z","hostname":"w-01HVWGBHJ2K16BGQZGJEFZFPS8","pid":1192,"client":"0","":"downstairs","session_id":"07abe336-2aff-49af-bdea-6d1e3473f6d6"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-04-20T01:04:30.225201464Z","hostname":"w-01HVWGBHJ2K16BGQZGJEFZFPS8","pid":1192,"client":"1","":"downstairs","session_id":"07abe336-2aff-49af-bdea-6d1e3473f6d6"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-04-20T01:04:30.225205964Z","hostname":"w-01HVWGBHJ2K16BGQZGJEFZFPS8","pid":1192,"client":"1","":"downstairs","session_id":"07abe336-2aff-49af-bdea-6d1e3473f6d6"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-04-20T01:04:30.225210763Z","hostname":"w-01HVWGBHJ2K16BGQZGJEFZFPS8","pid":1192,"client":"2","":"downstairs","session_id":"07abe336-2aff-49af-bdea-6d1e3473f6d6"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-04-20T01:04:30.225215303Z","hostname":"w-01HVWGBHJ2K16BGQZGJEFZFPS8","pid":1192,"client":"2","":"downstairs","session_id":"07abe336-2aff-49af-bdea-6d1e3473f6d6"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-04-20T01:04:30.225266899Z","hostname":"w-01HVWGBHJ2K16BGQZGJEFZFPS8","pid":1192,"":"io task","client":"0","":"downstairs","session_id":"07abe336-2aff-49af-bdea-6d1e3473f6d6"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-04-20T01:04:30.225287678Z","hostname":"w-01HVWGBHJ2K16BGQZGJEFZFPS8","pid":1192,"":"io task","client":"2","":"downstairs","session_id":"07abe336-2aff-49af-bdea-6d1e3473f6d6"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-04-20T01:04:30.225296317Z","hostname":"w-01HVWGBHJ2K16BGQZGJEFZFPS8","pid":1192,"":"io task","client":"1","":"downstairs","session_id":"07abe336-2aff-49af-bdea-6d1e3473f6d6"} {"msg":"ds_connection connected from Ok(127.0.0.1:36340)","v":0,"name":"crucible","level":30,"time":"2024-04-20T01:04:30.225391551Z","hostname":"w-01HVWGBHJ2K16BGQZGJEFZFPS8","pid":1192,"":"io task","client":"1","":"downstairs","session_id":"07abe336-2aff-49af-bdea-6d1e3473f6d6"} {"msg":"ds_connection connected from Ok(127.0.0.1:52689)","v":0,"name":"crucible","level":30,"time":"2024-04-20T01:04:30.22541392Z","hostname":"w-01HVWGBHJ2K16BGQZGJEFZFPS8","pid":1192,"":"io task","client":"0","":"downstairs","session_id":"07abe336-2aff-49af-bdea-6d1e3473f6d6"} {"msg":"ds_connection connected from Ok(127.0.0.1:36231)","v":0,"name":"crucible","level":30,"time":"2024-04-20T01:04:30.225422489Z","hostname":"w-01HVWGBHJ2K16BGQZGJEFZFPS8","pid":1192,"":"io task","client":"2","":"downstairs","session_id":"07abe336-2aff-49af-bdea-6d1e3473f6d6"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-04-20T01:04:30.225977861Z","hostname":"w-01HVWGBHJ2K16BGQZGJEFZFPS8","pid":1192,"client":"1","":"downstairs","session_id":"07abe336-2aff-49af-bdea-6d1e3473f6d6"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-04-20T01:04:30.2259903Z","hostname":"w-01HVWGBHJ2K16BGQZGJEFZFPS8","pid":1192,"client":"1","":"downstairs","session_id":"07abe336-2aff-49af-bdea-6d1e3473f6d6"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-04-20T01:04:30.22599835Z","hostname":"w-01HVWGBHJ2K16BGQZGJEFZFPS8","pid":1192,"client":"2","":"downstairs","session_id":"07abe336-2aff-49af-bdea-6d1e3473f6d6"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-04-20T01:04:30.226005809Z","hostname":"w-01HVWGBHJ2K16BGQZGJEFZFPS8","pid":1192,"client":"2","":"downstairs","session_id":"07abe336-2aff-49af-bdea-6d1e3473f6d6"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-04-20T01:04:30.226012549Z","hostname":"w-01HVWGBHJ2K16BGQZGJEFZFPS8","pid":1192,"client":"0","":"downstairs","session_id":"07abe336-2aff-49af-bdea-6d1e3473f6d6"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-04-20T01:04:30.226017128Z","hostname":"w-01HVWGBHJ2K16BGQZGJEFZFPS8","pid":1192,"client":"0","":"downstairs","session_id":"07abe336-2aff-49af-bdea-6d1e3473f6d6"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-04-20T01:04:30.226265582Z","hostname":"w-01HVWGBHJ2K16BGQZGJEFZFPS8","pid":1192,"client":"0","":"downstairs","session_id":"07abe336-2aff-49af-bdea-6d1e3473f6d6"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-04-20T01:04:30.22629376Z","hostname":"w-01HVWGBHJ2K16BGQZGJEFZFPS8","pid":1192,"client":"1","":"downstairs","session_id":"07abe336-2aff-49af-bdea-6d1e3473f6d6"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-04-20T01:04:30.226302149Z","hostname":"w-01HVWGBHJ2K16BGQZGJEFZFPS8","pid":1192,"client":"2","":"downstairs","session_id":"07abe336-2aff-49af-bdea-6d1e3473f6d6"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-04-20T01:04:30.228487971Z","hostname":"w-01HVWGBHJ2K16BGQZGJEFZFPS8","pid":1192,"client":"1","":"downstairs","session_id":"07abe336-2aff-49af-bdea-6d1e3473f6d6"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-04-20T01:04:30.22850445Z","hostname":"w-01HVWGBHJ2K16BGQZGJEFZFPS8","pid":1192,"client":"1","":"downstairs","session_id":"07abe336-2aff-49af-bdea-6d1e3473f6d6"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-04-20T01:04:30.228512109Z","hostname":"w-01HVWGBHJ2K16BGQZGJEFZFPS8","pid":1192,"session_id":"07abe336-2aff-49af-bdea-6d1e3473f6d6"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-04-20T01:04:30.22923717Z","hostname":"w-01HVWGBHJ2K16BGQZGJEFZFPS8","pid":1192,"client":"0","":"downstairs","session_id":"07abe336-2aff-49af-bdea-6d1e3473f6d6"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-04-20T01:04:30.229259428Z","hostname":"w-01HVWGBHJ2K16BGQZGJEFZFPS8","pid":1192,"client":"0","":"downstairs","session_id":"07abe336-2aff-49af-bdea-6d1e3473f6d6"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-04-20T01:04:30.229267798Z","hostname":"w-01HVWGBHJ2K16BGQZGJEFZFPS8","pid":1192,"session_id":"07abe336-2aff-49af-bdea-6d1e3473f6d6"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-04-20T01:04:30.229475184Z","hostname":"w-01HVWGBHJ2K16BGQZGJEFZFPS8","pid":1192,"client":"2","":"downstairs","session_id":"07abe336-2aff-49af-bdea-6d1e3473f6d6"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-04-20T01:04:30.229483833Z","hostname":"w-01HVWGBHJ2K16BGQZGJEFZFPS8","pid":1192,"client":"2","":"downstairs","session_id":"07abe336-2aff-49af-bdea-6d1e3473f6d6"} {"msg":"[0]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-04-20T01:04:30.229556538Z","hostname":"w-01HVWGBHJ2K16BGQZGJEFZFPS8","pid":1192,"":"downstairs","":"downstairs","session_id":"07abe336-2aff-49af-bdea-6d1e3473f6d6"} {"msg":"[0]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-04-20T01:04:30.229577637Z","hostname":"w-01HVWGBHJ2K16BGQZGJEFZFPS8","pid":1192,"":"downstairs","":"downstairs","session_id":"07abe336-2aff-49af-bdea-6d1e3473f6d6"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-04-20T01:04:30.229585616Z","hostname":"w-01HVWGBHJ2K16BGQZGJEFZFPS8","pid":1192,"":"downstairs","":"downstairs","session_id":"07abe336-2aff-49af-bdea-6d1e3473f6d6"} {"msg":"[1]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-04-20T01:04:30.229593086Z","hostname":"w-01HVWGBHJ2K16BGQZGJEFZFPS8","pid":1192,"":"downstairs","":"downstairs","session_id":"07abe336-2aff-49af-bdea-6d1e3473f6d6"} {"msg":"[1]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-04-20T01:04:30.229600125Z","hostname":"w-01HVWGBHJ2K16BGQZGJEFZFPS8","pid":1192,"":"downstairs","":"downstairs","session_id":"07abe336-2aff-49af-bdea-6d1e3473f6d6"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-04-20T01:04:30.229606825Z","hostname":"w-01HVWGBHJ2K16BGQZGJEFZFPS8","pid":1192,"":"downstairs","":"downstairs","session_id":"07abe336-2aff-49af-bdea-6d1e3473f6d6"} {"msg":"[2]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-04-20T01:04:30.229613794Z","hostname":"w-01HVWGBHJ2K16BGQZGJEFZFPS8","pid":1192,"":"downstairs","":"downstairs","session_id":"07abe336-2aff-49af-bdea-6d1e3473f6d6"} {"msg":"[2]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-04-20T01:04:30.229620264Z","hostname":"w-01HVWGBHJ2K16BGQZGJEFZFPS8","pid":1192,"":"downstairs","":"downstairs","session_id":"07abe336-2aff-49af-bdea-6d1e3473f6d6"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-04-20T01:04:30.229627353Z","hostname":"w-01HVWGBHJ2K16BGQZGJEFZFPS8","pid":1192,"":"downstairs","":"downstairs","session_id":"07abe336-2aff-49af-bdea-6d1e3473f6d6"} {"msg":"Max found gen is 1","v":0,"name":"crucible","level":30,"time":"2024-04-20T01:04:30.229634913Z","hostname":"w-01HVWGBHJ2K16BGQZGJEFZFPS8","pid":1192,"":"downstairs","":"downstairs","session_id":"07abe336-2aff-49af-bdea-6d1e3473f6d6"} {"msg":"Generation requested: 1 >= found:1","v":0,"name":"crucible","level":30,"time":"2024-04-20T01:04:30.229641042Z","hostname":"w-01HVWGBHJ2K16BGQZGJEFZFPS8","pid":1192,"":Wait for a query_work_queue command to finish before sending IO "downstairs","":"downstairs","session_id":"07abe336-2aff-49af-bdea-6d1e3473f6d6"} {"msg":"Next flush: 1","v":0,"name":"crucible","level":30,"time":"2024-04-20T01:04:30.229650612Z","hostname":"w-01HVWGBHJ2K16BGQZGJEFZFPS8","pid":1192,"":"downstairs","":"downstairs","session_id":"07abe336-2aff-49af-bdea-6d1e3473f6d6"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-04-20T01:04:30.229666561Z","hostname":"w-01HVWGBHJ2K16BGQZGJEFZFPS8","pid":1192,"":"downstairs","":"downstairs","session_id":"07abe336-2aff-49af-bdea-6d1e3473f6d6"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-04-20T01:04:30.22967267Z","hostname":"w-01HVWGBHJ2K16BGQZGJEFZFPS8","pid":1192,"session_id":"07abe336-2aff-49af-bdea-6d1e3473f6d6"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-04-20T01:04:30.22967809Z","hostname":"w-01HVWGBHJ2K16BGQZGJEFZFPS8","pid":1192,"client":"0","":"downstairs","session_id":"07abe336-2aff-49af-bdea-6d1e3473f6d6"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-04-20T01:04:30.229685639Z","hostname":"w-01HVWGBHJ2K16BGQZGJEFZFPS8","pid":1192,"client":"1","":"downstairs","session_id":"07abe336-2aff-49af-bdea-6d1e3473f6d6"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-04-20T01:04:30.229691709Z","hostname":"w-01HVWGBHJ2K16BGQZGJEFZFPS8","pid":1192,"client":"2","":"downstairs","session_id":"07abe336-2aff-49af-bdea-6d1e3473f6d6"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-04-20T01:04:30.229699158Z","hostname":"w-01HVWGBHJ2K16BGQZGJEFZFPS8","pid":1192,"session_id":"07abe336-2aff-49af-bdea-6d1e3473f6d6"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-04-20T01:04:30.229704368Z","hostname":"w-01HVWGBHJ2K16BGQZGJEFZFPS8","pid":1192,"session_id":"07abe336-2aff-49af-bdea-6d1e3473f6d6"} {"msg":"f3e88e50-0682-46ea-a497-ae5a7fcd620a is now active with session: 07abe336-2aff-49af-bdea-6d1e3473f6d6","v":0,"name":"crucible","level":30,"time":"2024-04-20T01:04:30.229709638Z","hostname":"w-01HVWGBHJ2K16BGQZGJEFZFPS8","pid":1192,"session_id":"07abe336-2aff-49af-bdea-6d1e3473f6d6"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-04-20T01:04:30.229715017Z","hostname":"w-01HVWGBHJ2K16BGQZGJEFZFPS8","pid":1192,"session_id":"07abe336-2aff-49af-bdea-6d1e3473f6d6"} {"msg":"The guest has finished waiting for activation with:1","v":0,"name":"crucible","level":30,"time":"2024-04-20T01:04:30.229721357Z","hostname":"w-01HVWGBHJ2K16BGQZGJEFZFPS8","pid":1192} Region: es:16384 ec:640 bs:4096 ts:42949672960 tb:10485760 max_io:256 or 1048576 Perf test TEST SECONDS COUNT DPTH IOPS MEAN P95 P99 MAX ES EC rwrites 10.34 30000 1 2902.46 0.00034 0.00122 0.00184 0.03628 16384 640 rwrites 8.68 30000 1 3454.53 0.00029 0.00098 0.00176 0.06408 16384 640 rreads 5.79 30000 1 5179.06 0.00019 0.00027 0.00031 0.01259 16384 640 rreads 5.74 30000 1 5223.21 0.00019 0.00027 0.00031 0.01261 16384 640 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-04-20T01:05:05.589045313Z","hostname":"w-01HVWGBHJ2K16BGQZGJEFZFPS8","pid":1192} {"msg":"client_stop_rx closed unexpectedly: RecvError(())","v":0,"name":"crucible","level":40,"time":"2024-04-20T01:05:05.589070081Z","hostname":"w-01HVWGBHJ2K16BGQZGJEFZFPS8","pid":1192,"":"io task","client":"0","":"downstairs","session_id":"07abe336-2aff-49af-bdea-6d1e3473f6d6"} {"msg":"client_stop_rx closed unexpectedly: RecvError(())","v":0,"name":"crucible","level":40,"time":"2024-04-20T01:05:05.589104449Z","hostname":"w-01HVWGBHJ2K16BGQZGJEFZFPS8","pid":1192,"":"io task","client":"1","":"downstairs","session_id":"07abe336-2aff-49af-bdea-6d1e3473f6d6"} {"msg":"client task is sending Done(QueueClosed)","v":0,"name":"crucible","level":40,"time":"2024-04-20T01:05:05.589134107Z","hostname":"w-01HVWGBHJ2K16BGQZGJEFZFPS8","pid":1192,"":"io task","client":"1","":"downstairs","session_id":"07abe336-2aff-49af-bdea-6d1e3473f6d6"} {"msg":"client task is sending Done(QueueClosed)","v":0,"name":"crucible","level":40,"time":"2024-04-20T01:05:05.589142866Z","hostname":"w-01HVWGBHJ2K16BGQZGJEFZFPS8","pid":1192,"":"io task","client":"0","":"downstairs","session_id":"07abe336-2aff-49af-bdea-6d1e3473f6d6"} {"msg":"client task could not reply to main task; shutting down?","v":0,"name":"crucible","level":40,"time":"2024-04-20T01:05:05.589150246Z","hostname":"w-01HVWGBHJ2K16BGQZGJEFZFPS8","pid":1192,"":"io task","client":"1","":"downstairs","session_id":"07abe336-2aff-49af-bdea-6d1e3473f6d6"} {"msg":"client task could not reply to main task; shutting down?","v":0,"name":"crucible","level":40,"time":"2024-04-20T01:05:05.589165505Z","hostname":"w-01HVWGBHJ2K16BGQZGJEFZFPS8","pid":1192,"":"io task","client":"0","":"downstairs","session_id":"07abe336-2aff-49af-bdea-6d1e3473f6d6"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-04-20T01:05:05.589172815Z","hostname":"w-01HVWGBHJ2K16BGQZGJEFZFPS8","pid":1192,"":"io task","client":"1","":"downstairs","session_id":"07abe336-2aff-49af-bdea-6d1e3473f6d6"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-04-20T01:05:05.589180784Z","hostname":"w-01HVWGBHJ2K16BGQZGJEFZFPS8","pid":1192,"":"io task","client":"0","":"downstairs","session_id":"07abe336-2aff-49af-bdea-6d1e3473f6d6"} Perf test completed, stop all downstairs Apr 20 01:05:05.655 INFO accepted connection, remote_addr: 127.0.0.1:47382, local_addr: 127.0.0.1:9998 Main task has work to do, go find it got dsc Shutdown Shutdown Apr 20 01:05:05.655 INFO request completed, latency_us: 56, Shut it down response_code[8810] Disable keep_running :[8810] Got stop action so:false kr:false 204, uri: /shutdown, method: POST, req_id: 8b5c9661-6957-4601-821a-6f8d8d0d9e3c, remote_addr: 127.0.0.1:47382, local_addr: [8820] Disable keep_running 127[8820] Got stop action so:false kr:false .[8830] Disable keep_running 0.0.1:9998 [8830] Got stop action so:false kr:false [8810] Exited with: ExitStatus(unix_wait_status(9)) [8810][0] reports Exit [8820] Exited with: ExitStatus(unix_wait_status(9)) [8820][1] reports Exit [8830] Exited with: ExitStatus(unix_wait_status(9)) [8830][2] reports Exit Create region with ES:16384 EC:1280 BS:4096 Removing existing dsc directory "/tmp/dsc" Removing existing region "/var/tmp/dsc" Creating region directory at: /var/tmp/dsc Creating dsc directory at: /tmp/dsc Downstairs region 0 created at /var/tmp/dsc/8810 in 1.3970155 Downstairs region 1 created at /var/tmp/dsc/8820 in 1.4235969 Downstairs region 2 created at /var/tmp/dsc/8830 in 1.3895521 Region set with 3 regions was created Using existing output directory "/tmp/dsc" start ds: 8810 start ds: 8820 start ds: 8830 start access at:127.0.0.1:9998 Starting downstairs at port 8810 Make output file at "/tmp/dsc/downstairs-8810.txt" [8810][0] reports Starting Starting downstairs at port 8820 Starting downstairs at port 8830 Make output file at "/tmp/dsc/downstairs-8820.txt" Make output file at "/tmp/dsc/downstairs-8830.txt" [8820][1] reports Starting [8830][2] reports Starting Downstairs /var/tmp/dsc/8810 port 8810 PID:Some(1201) [8810][0] reports Running Downstairs /var/tmp/dsc/8820 port 8820 PID:Some(1202) [8820][1] reports Running Downstairs /var/tmp/dsc/8830 port 8830 PID:Some(1203) [8830][2] reports Running Apr 20 01:05:10.926 INFO listening, local_addr: 127.0.0.1:9998 Control access at:127.0.0.1:9998 1203 1202 1201 Apr 20 01:05:15.963 INFO accepted connection, remote_addr: 127.0.0.1:36409, local_addr: 127.0.0.1:9998 Main task has work to do, go find it got dsc DisableRestartAll disable restart on all: 3 Apr 20 01:05:15.963[8810] Disable keep_running [8820] Disable keep_running INFO request completed, latency_us: 100[8830] Disable keep_running , response_code: 204, uri: /disablerestart/all, method: POST, req_id: cee55714-da70-4cb1-9734-bef415356c3e, remote_addr: 127.0.0.1:36409, local_addr: 127.0.0.1:9998 A new loop begins at April 20, 2024 at 01:05:15 AM UTC /var/tmp/bins/crutest perf -g 1 -t 127.0.0.1:8810 -t 127.0.0.1:8820 -t 127.0.0.1:8830 -c 30000 -q --write-loops 2 --read-loops 2 --perf-out /tmp/perf-ES-16384-EC-1280.csv {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-04-20T01:05:15.982314029Z","hostname":"w-01HVWGBHJ2K16BGQZGJEFZFPS8","pid":1211} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"b97e434eec561fa80f965a2f40ec87c83a3a439e\",\n git_commit_timestamp: \"2024-04-20T00:34:22.000000000Z\",\n git_branch: \"artemis/remove-reqwest-block-io\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: false,\n opt_level: 3,\n}","v":0,"name":"crucible","level":30,"time":"2024-04-20T01:05:15.982435061Z","hostname":"w-01HVWGBHJ2K16BGQZGJEFZFPS8","pid":1211} {"msg":"Upstairs <-> Downstairs Message Version: 6","v":0,"name":"crucible","level":30,"time":"2024-04-20T01:05:15.982441191Z","hostname":"w-01HVWGBHJ2K16BGQZGJEFZFPS8","pid":1211} {"msg":"Crucible 4dc09763-cbfb-4df1-a8e3-8c33e813bf58 has session id: 4202398b-8c62-4e26-8923-4b357aa6da82","v":0,"name":"crucible","level":30,"time":"2024-04-20T01:05:15.98244585Z","hostname":"w-01HVWGBHJ2K16BGQZGJEFZFPS8","pid":1211,"session_id":"4202398b-8c62-4e26-8923-4b357aa6da82"} {"msg":"Upstairs opts: Upstairs UUID: 4dc09763-cbfb-4df1-a8e3-8c33e813bf58, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-04-20T01:05:15.98245504Z","hostname":"w-01HVWGBHJ2K16BGQZGJEFZFPS8","pid":1211,"session_id":"4202398b-8c62-4e26-8923-4b357aa6da82"} {"msg":"Crucible stats registered with UUID: 4dc09763-cbfb-4df1-a8e3-8c33e813bf58","v":0,"name":"crucible","level":30,"time":"2024-04-20T01:05:15.982459479Z","hostname":"w-01HVWGBHJ2K16BGQZGJEFZFPS8","pid":1211,"session_id":"4202398b-8c62-4e26-8923-4b357aa6da82"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:1","v":0,"name":"crucible","level":30,"time":"2024-04-20T01:05:15.982495387Z","hostname":"w-01HVWGBHJ2K16BGQZGJEFZFPS8","pid":1211} {"msg":"4dc09763-cbfb-4df1-a8e3-8c33e813bf58 active request set","v":0,"name":"crucible","level":30,"time":"2024-04-20T01:05:15.982589021Z","hostname":"w-01HVWGBHJ2K16BGQZGJEFZFPS8","pid":1211,"session_id":"4202398b-8c62-4e26-8923-4b357aa6da82"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-04-20T01:05:15.982627788Z","hostname":"w-01HVWGBHJ2K16BGQZGJEFZFPS8","pid":1211,"client":"0","":"downstairs","session_id":"4202398b-8c62-4e26-8923-4b357aa6da82"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-04-20T01:05:15.982640747Z","hostname":"w-01HVWGBHJ2K16BGQZGJEFZFPS8","pid":1211,"client":"0","":"downstairs","session_id":"4202398b-8c62-4e26-8923-4b357aa6da82"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-04-20T01:05:15.982678195Z","hostname":"w-01HVWGBHJ2K16BGQZGJEFZFPS8","pid":1211,"client":"1","":"downstairs","session_id":"4202398b-8c62-4e26-8923-4b357aa6da82"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-04-20T01:05:15.982711072Z","hostname":"w-01HVWGBHJ2K16BGQZGJEFZFPS8","pid":1211,"client":"1","":"downstairs","session_id":"4202398b-8c62-4e26-8923-4b357aa6da82"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-04-20T01:05:15.982723012Z","hostname":"w-01HVWGBHJ2K16BGQZGJEFZFPS8","pid":1211,"client":"2","":"downstairs","session_id":"4202398b-8c62-4e26-8923-4b357aa6da82"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-04-20T01:05:15.982730441Z","hostname":"w-01HVWGBHJ2K16BGQZGJEFZFPS8","pid":1211,"client":"2","":"downstairs","session_id":"4202398b-8c62-4e26-8923-4b357aa6da82"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-04-20T01:05:15.98274872Z","hostname":"w-01HVWGBHJ2K16BGQZGJEFZFPS8","pid":1211,"":"io task","client":"2","":"downstairs","session_id":"4202398b-8c62-4e26-8923-4b357aa6da82"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-04-20T01:05:15.982756659Z","hostname":"w-01HVWGBHJ2K16BGQZGJEFZFPS8","pid":1211,"":"io task","client":"0","":"downstairs","session_id":"4202398b-8c62-4e26-8923-4b357aa6da82"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-04-20T01:05:15.982764389Z","hostname":"w-01HVWGBHJ2K16BGQZGJEFZFPS8","pid":1211,"":"io task","client":"1","":"downstairs","session_id":"4202398b-8c62-4e26-8923-4b357aa6da82"} {"msg":"ds_connection connected from Ok(127.0.0.1:59916)","v":0,"name":"crucible","level":30,"time":"2024-04-20T01:05:15.982778748Z","hostname":"w-01HVWGBHJ2K16BGQZGJEFZFPS8","pid":1211,"":"io task","client":"0","":"downstairs","session_id":"4202398b-8c62-4e26-8923-4b357aa6da82"} {"msg":"ds_connection connected from Ok(127.0.0.1:55436)","v":0,"name":"crucible","level":30,"time":"2024-04-20T01:05:15.982787497Z","hostname":"w-01HVWGBHJ2K16BGQZGJEFZFPS8","pid":1211,"":"io task","client":"2","":"downstairs","session_id":"4202398b-8c62-4e26-8923-4b357aa6da82"} {"msg":"ds_connection connected from Ok(127.0.0.1:60470)","v":0,"name":"crucible","level":30,"time":"2024-04-20T01:05:15.982871041Z","hostname":"w-01HVWGBHJ2K16BGQZGJEFZFPS8","pid":1211,"":"io task","client":"1","":"downstairs","session_id":"4202398b-8c62-4e26-8923-4b357aa6da82"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-04-20T01:05:15.983117075Z","hostname":"w-01HVWGBHJ2K16BGQZGJEFZFPS8","pid":1211,"client":"1","":"downstairs","session_id":"4202398b-8c62-4e26-8923-4b357aa6da82"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-04-20T01:05:15.983138243Z","hostname":"w-01HVWGBHJ2K16BGQZGJEFZFPS8","pid":1211,"client":"1","":"downstairs","session_id":"4202398b-8c62-4e26-8923-4b357aa6da82"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-04-20T01:05:15.983146463Z","hostname":"w-01HVWGBHJ2K16BGQZGJEFZFPS8","pid":1211,"client":"2","":"downstairs","session_id":"4202398b-8c62-4e26-8923-4b357aa6da82"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-04-20T01:05:15.983154012Z","hostname":"w-01HVWGBHJ2K16BGQZGJEFZFPS8","pid":1211,"client":"2","":"downstairs","session_id":"4202398b-8c62-4e26-8923-4b357aa6da82"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-04-20T01:05:15.983161952Z","hostname":"w-01HVWGBHJ2K16BGQZGJEFZFPS8","pid":1211,"client":"0","":"downstairs","session_id":"4202398b-8c62-4e26-8923-4b357aa6da82"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-04-20T01:05:15.983169041Z","hostname":"w-01HVWGBHJ2K16BGQZGJEFZFPS8","pid":1211,"client":"0","":"downstairs","session_id":"4202398b-8c62-4e26-8923-4b357aa6da82"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-04-20T01:05:15.983402275Z","hostname":"w-01HVWGBHJ2K16BGQZGJEFZFPS8","pid":1211,"client":"1","":"downstairs","session_id":"4202398b-8c62-4e26-8923-4b357aa6da82"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-04-20T01:05:15.983418894Z","hostname":"w-01HVWGBHJ2K16BGQZGJEFZFPS8","pid":1211,"client":"2","":"downstairs","session_id":"4202398b-8c62-4e26-8923-4b357aa6da82"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-04-20T01:05:15.983423814Z","hostname":"w-01HVWGBHJ2K16BGQZGJEFZFPS8","pid":1211,"client":"0","":"downstairs","session_id":"4202398b-8c62-4e26-8923-4b357aa6da82"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-04-20T01:05:15.988874104Z","hostname":"w-01HVWGBHJ2K16BGQZGJEFZFPS8","pid":1211,"client":"2","":"downstairs","session_id":"4202398b-8c62-4e26-8923-4b357aa6da82"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-04-20T01:05:15.988891553Z","hostname":"w-01HVWGBHJ2K16BGQZGJEFZFPS8","pid":1211,"client":"2","":"downstairs","session_id":"4202398b-8c62-4e26-8923-4b357aa6da82"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-04-20T01:05:15.988896392Z","hostname":"w-01HVWGBHJ2K16BGQZGJEFZFPS8","pid":1211,"session_id":"4202398b-8c62-4e26-8923-4b357aa6da82"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-04-20T01:05:15.989999657Z","hostname":"w-01HVWGBHJ2K16BGQZGJEFZFPS8","pid":1211,"client":"0","":"downstairs","session_id":"4202398b-8c62-4e26-8923-4b357aa6da82"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-04-20T01:05:15.990015966Z","hostname":"w-01HVWGBHJ2K16BGQZGJEFZFPS8","pid":1211,"client":"0","":"downstairs","session_id":"4202398b-8c62-4e26-8923-4b357aa6da82"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-04-20T01:05:15.990023326Z","hostname":"w-01HVWGBHJ2K16BGQZGJEFZFPS8","pid":1211,"session_id":"4202398b-8c62-4e26-8923-4b357aa6da82"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-04-20T01:05:15.990058963Z","hostname":"w-01HVWGBHJ2K16BGQZGJEFZFPS8","pid":1211,"client":"1","":"downstairs","session_id":"4202398b-8c62-4e26-8923-4b357aa6da82"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-04-20T01:05:15.990077682Z","hostname":"w-01HVWGBHJ2K16BGQZGJEFZFPS8","pid":1211,"client":"1","":"downstairs","session_id":"4202398b-8c62-4e26-8923-4b357aa6da82"} {"msg":"[0]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-04-20T01:05:15.990083132Z","hostname":"w-01HVWGBHJ2K16BGQZGJEFZFPS8","pid":1211,"":"downstairs","":"downstairs","session_id":"4202398b-8c62-4e26-8923-4b357aa6da82"} {"msg":"[0]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-04-20T01:05:15.990088291Z","hostname":"w-01HVWGBHJ2K16BGQZGJEFZFPS8","pid":1211,"":"downstairs","":"downstairs","session_id":"4202398b-8c62-4e26-8923-4b357aa6da82"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-04-20T01:05:15.990093121Z","hostname":"w-01HVWGBHJ2K16BGQZGJEFZFPS8","pid":1211,"":"downstairs","":"downstairs","session_id":"4202398b-8c62-4e26-8923-4b357aa6da82"} {"msg":"[1]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-04-20T01:05:15.990098231Z","hostname":"w-01HVWGBHJ2K16BGQZGJEFZFPS8","pid":1211,"":"downstairs","":"downstairs","session_id":"4202398b-8c62-4e26-8923-4b357aa6da82"} {"msg":"[1]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-04-20T01:05:15.99010285Z","hostname":"w-01HVWGBHJ2K16BGQZGJEFZFPS8","pid":1211,"":"downstairs","":"downstairs","session_id":"4202398b-8c62-4e26-8923-4b357aa6da82"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-04-20T01:05:15.99010786Z","hostname":"w-01HVWGBHJ2K16BGQZGJEFZFPS8","pid":1211,"":"downstairs","":"downstairs","session_id":"4202398b-8c62-4e26-8923-4b357aa6da82"} {"msg":"[2]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-04-20T01:05:15.99011257Z","hostname":"w-01HVWGBHJ2K16BGQZGJEFZFPS8","pid":1211,"":"downstairs","":"downstairs","session_id":"4202398b-8c62-4e26-8923-4b357aa6da82"} {"msg":"[2]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-04-20T01:05:15.990116809Z","hostname":"w-01HVWGBHJ2K16BGQZGJEFZFPS8","pid":1211,"":"downstairs","":"downstairs","session_id":"4202398b-8c62-4e26-8923-4b357aa6da82"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-04-20T01:05:15.990121689Z","hostname":"w-01HVWGBHJ2K16BGQZGJEFZFPS8","pid":1211,"":"downstairs","":"downstairs","session_id":"4202398b-8c62-4e26-8923-4b357aa6da82"} {"msg":"Max found gen is 1","v":0,"name":"crucible","level":30,"time":"2024-04-20T01:05:15.990125949Z","hostname":"w-01HVWGBHJ2K16BGQZGJEFZFPS8","pid":1211,"":"downstairs","":"downstairs","session_id":"4202398b-8c62-4e26-8923-4b357aa6da82"} {"msg":"Generation requested: 1 >= found:1","v":0,"name":"crucible","level":30,"time":"2024-04-20T01:05:15.990130369Z","hostname":"w-01HVWGBHJ2K16BGQZGJEFZFPS8","pid":1211,"":"downstairs","":"downstairs","session_id":"4202398b-8c62-4e26-8923-4b357aa6da82"} {"msg":"Next flush: 1","v":0,"name":"crucible","level":30,"time"Wait for a query_work_queue command to finish before sending IO :"2024-04-20T01:05:15.990135168Z","hostname":"w-01HVWGBHJ2K16BGQZGJEFZFPS8","pid":1211,"":"downstairs","":"downstairs","session_id":"4202398b-8c62-4e26-8923-4b357aa6da82"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-04-20T01:05:15.990142657Z","hostname":"w-01HVWGBHJ2K16BGQZGJEFZFPS8","pid":1211,"":"downstairs","":"downstairs","session_id":"4202398b-8c62-4e26-8923-4b357aa6da82"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-04-20T01:05:15.990146977Z","hostname":"w-01HVWGBHJ2K16BGQZGJEFZFPS8","pid":1211,"session_id":"4202398b-8c62-4e26-8923-4b357aa6da82"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-04-20T01:05:15.990150687Z","hostname":"w-01HVWGBHJ2K16BGQZGJEFZFPS8","pid":1211,"client":"0","":"downstairs","session_id":"4202398b-8c62-4e26-8923-4b357aa6da82"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-04-20T01:05:15.990155297Z","hostname":"w-01HVWGBHJ2K16BGQZGJEFZFPS8","pid":1211,"client":"1","":"downstairs","session_id":"4202398b-8c62-4e26-8923-4b357aa6da82"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-04-20T01:05:15.990159336Z","hostname":"w-01HVWGBHJ2K16BGQZGJEFZFPS8","pid":1211,"client":"2","":"downstairs","session_id":"4202398b-8c62-4e26-8923-4b357aa6da82"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-04-20T01:05:15.990163516Z","hostname":"w-01HVWGBHJ2K16BGQZGJEFZFPS8","pid":1211,"session_id":"4202398b-8c62-4e26-8923-4b357aa6da82"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-04-20T01:05:15.990167806Z","hostname":"w-01HVWGBHJ2K16BGQZGJEFZFPS8","pid":1211,"session_id":"4202398b-8c62-4e26-8923-4b357aa6da82"} {"msg":"4dc09763-cbfb-4df1-a8e3-8c33e813bf58 is now active with session: 4202398b-8c62-4e26-8923-4b357aa6da82","v":0,"name":"crucible","level":30,"time":"2024-04-20T01:05:15.990171536Z","hostname":"w-01HVWGBHJ2K16BGQZGJEFZFPS8","pid":1211,"session_id":"4202398b-8c62-4e26-8923-4b357aa6da82"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-04-20T01:05:15.990175155Z","hostname":"w-01HVWGBHJ2K16BGQZGJEFZFPS8","pid":1211,"session_id":"4202398b-8c62-4e26-8923-4b357aa6da82"} {"msg":"The guest has finished waiting for activation with:1","v":0,"name":"crucible","level":30,"time":"2024-04-20T01:05:15.990180885Z","hostname":"w-01HVWGBHJ2K16BGQZGJEFZFPS8","pid":1211} Region: es:16384 ec:1280 bs:4096 ts:85899345920 tb:20971520 max_io:256 or 1048576 Perf test TEST SECONDS COUNT DPTH IOPS MEAN P95 P99 MAX ES EC rwrites 8.96 30000 1 3347.55 0.00030 0.00091 0.00181 0.04789 16384 1280 rwrites 8.87 30000 1 3383.59 0.00030 0.00082 0.00175 0.06067 16384 1280 rreads 5.63 30000 1 5327.01 0.00019 0.00023 0.00028 0.01262 16384 1280 rreads 5.57 30000 1 5390.40 0.00019 0.00023 0.00028 0.01409 16384 1280 Perf test completed, stop all downstairs Apr 20 01:05:51.614 INFO accepted connection, remote_addr: 127.0.0.1:40250, local_addr: 127.0.0.1:9998 Apr 20 01:05:51.615 Main task has work to do, go find it INFOgot dsc Shutdown Shutdown request completed, latency_us: 51, response_codeShut it down : 204, uri[8810] Disable keep_running :[8810] Got stop action so:false kr:false [8830] Disable keep_running /shutdown, method: POST, req_id: 04b171ed-c1eb-4c92-93d3-cba3cdf8b7f3, remote_addr:[8830] Got stop action so:false kr:false [8820] Disable keep_running 127.0.0.1:40250, local_addr: 127.0.0.1:9998 [8820] Got stop action so:false kr:false [8830] Exited with: ExitStatus(unix_wait_status(9)) [8830][2] reports Exit [8810] Exited with: ExitStatus(unix_wait_status(9)) [8810][0] reports Exit [8820] Exited with: ExitStatus(unix_wait_status(9)) [8820][1] reports Exit TEST SECONDS COUNT DPTH IOPS MEAN P95 P99 MAX ES EC rwrites 10.34 30000 1 2902.46 0.00034 0.00122 0.00184 0.03628 16384 640 rwrites 8.68 30000 1 3454.53 0.00029 0.00098 0.00176 0.06408 16384 640 rwrites 8.96 30000 1 3347.55 0.00030 0.00091 0.00181 0.04789 16384 1280 rwrites 8.87 30000 1 3383.59 0.00030 0.00082 0.00175 0.06067 16384 1280 TEST SECONDS COUNT DPTH IOPS MEAN P95 P99 MAX ES EC rreads 5.79 30000 1 5179.06 0.00019 0.00027 0.00031 0.01259 16384 640 rreads 5.74 30000 1 5223.21 0.00019 0.00027 0.00031 0.01261 16384 640 rreads 5.63 30000 1 5327.01 0.00019 0.00023 0.00028 0.01262 16384 1280 rreads 5.57 30000 1 5390.40 0.00019 0.00023 0.00028 0.01409 16384 1280 Perf test finished on April 20, 2024 at 01:05:52 AM UTC