/input/rbuild/work Create region with ES:16384 EC:640 BS:4096 Creating region directory at: /var/tmp/dsc Creating dsc directory at: /tmp/dsc Downstairs region 0 created at /var/tmp/dsc/8810 in 1.0960379 Downstairs region 1 created at /var/tmp/dsc/8820 in 0.64201087 Downstairs region 2 created at /var/tmp/dsc/8830 in 0.54345745 Region set with 3 regions was created Using existing output directory "/tmp/dsc" start ds: 8810 start ds: 8820 start ds: 8830 Starting downstairs at port 8810 Starting downstairs at port 8820 Starting downstairs at port 8830 Make output file at "/tmp/dsc/downstairs-8820.txt" Make output file at "/tmp/dsc/downstairs-8830.txt" start access at:127.0.0.1:9998 Make output file at "/tmp/dsc/downstairs-8810.txt" [8810][0] reports Starting [8820][1] reports Starting [8830][2] reports Starting Downstairs /var/tmp/dsc/8830 port 8830 PID:Some(1182) [8830][2] reports Running Downstairs /var/tmp/dsc/8810 port 8810 PID:Some(1183) [8810][0] reports Running Downstairs /var/tmp/dsc/8820 port 8820 PID:Some(1184) [8820][1] reports Running Apr 17 17:13:55.899 INFO listening, local_addr: 127.0.0.1:9998 Control access at:127.0.0.1:9998 1182 1183 1184 Apr 17 17:14:00.938 INFO accepted connection, remote_addr: 127.0.0.1:54812, local_addr: 127.0.0.1:9998 Main task has work to do, go find it got dsc DisableRestartAll disable restart on all: 3 Apr 17 17:14:00.938 INFO request completed, latency_us: 139, response_code: 204, uri:[8810] Disable keep_running [8820] Disable keep_running /disablerestart/all[8830] Disable keep_running , method: POST, req_id: b09eae8e-ac7f-4162-861b-bf316942c5a7, remote_addr: 127.0.0.1:54812, local_addr: 127.0.0.1:9998 A new loop begins at April 17, 2024 at 05:14:00 PM UTC /var/tmp/bins/crutest perf -g 1 -t 127.0.0.1:8810 -t 127.0.0.1:8820 -t 127.0.0.1:8830 -c 30000 -q --write-loops 2 --read-loops 2 --perf-out /tmp/perf-ES-16384-EC-640.csv {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-04-17T17:14:00.959792464Z","hostname":"w-01HVPGMT7GXTS2BQW5HJHZHK9D","pid":1192} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"9dd6771ac5213adb2b97af7a72c1375fe36d2636\",\n git_commit_timestamp: \"2024-04-17T16:00:39.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: false,\n opt_level: 3,\n}","v":0,"name":"crucible","level":30,"time":"2024-04-17T17:14:00.960006051Z","hostname":"w-01HVPGMT7GXTS2BQW5HJHZHK9D","pid":1192} {"msg":"Upstairs <-> Downstairs Message Version: 6","v":0,"name":"crucible","level":30,"time":"2024-04-17T17:14:00.9600137Z","hostname":"w-01HVPGMT7GXTS2BQW5HJHZHK9D","pid":1192} {"msg":"Crucible 8af3f883-cb3d-444d-b479-a75b52bbe518 has session id: 2cc58e3a-bc6a-49a6-8671-87173741b601","v":0,"name":"crucible","level":30,"time":"2024-04-17T17:14:00.96001906Z","hostname":"w-01HVPGMT7GXTS2BQW5HJHZHK9D","pid":1192,"session_id":"2cc58e3a-bc6a-49a6-8671-87173741b601"} {"msg":"Upstairs opts: Upstairs UUID: 8af3f883-cb3d-444d-b479-a75b52bbe518, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-04-17T17:14:00.9600256Z","hostname":"w-01HVPGMT7GXTS2BQW5HJHZHK9D","pid":1192,"session_id":"2cc58e3a-bc6a-49a6-8671-87173741b601"} {"msg":"Crucible stats registered with UUID: 8af3f883-cb3d-444d-b479-a75b52bbe518","v":0,"name":"crucible","level":30,"time":"2024-04-17T17:14:00.96003205Z","hostname":"w-01HVPGMT7GXTS2BQW5HJHZHK9D","pid":1192,"session_id":"2cc58e3a-bc6a-49a6-8671-87173741b601"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:1","v":0,"name":"crucible","level":30,"time":"2024-04-17T17:14:00.960238697Z","hostname":"w-01HVPGMT7GXTS2BQW5HJHZHK9D","pid":1192} {"msg":"8af3f883-cb3d-444d-b479-a75b52bbe518 active request set","v":0,"name":"crucible","level":30,"time":"2024-04-17T17:14:00.960408394Z","hostname":"w-01HVPGMT7GXTS2BQW5HJHZHK9D","pid":1192,"session_id":"2cc58e3a-bc6a-49a6-8671-87173741b601"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-04-17T17:14:00.960424314Z","hostname":"w-01HVPGMT7GXTS2BQW5HJHZHK9D","pid":1192,"client":"0","":"downstairs","session_id":"2cc58e3a-bc6a-49a6-8671-87173741b601"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-04-17T17:14:00.960469313Z","hostname":"w-01HVPGMT7GXTS2BQW5HJHZHK9D","pid":1192,"client":"0","":"downstairs","session_id":"2cc58e3a-bc6a-49a6-8671-87173741b601"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-04-17T17:14:00.960479063Z","hostname":"w-01HVPGMT7GXTS2BQW5HJHZHK9D","pid":1192,"client":"1","":"downstairs","session_id":"2cc58e3a-bc6a-49a6-8671-87173741b601"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-04-17T17:14:00.960487263Z","hostname":"w-01HVPGMT7GXTS2BQW5HJHZHK9D","pid":1192,"client":"1","":"downstairs","session_id":"2cc58e3a-bc6a-49a6-8671-87173741b601"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-04-17T17:14:00.960496232Z","hostname":"w-01HVPGMT7GXTS2BQW5HJHZHK9D","pid":1192,"client":"2","":"downstairs","session_id":"2cc58e3a-bc6a-49a6-8671-87173741b601"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-04-17T17:14:00.960504002Z","hostname":"w-01HVPGMT7GXTS2BQW5HJHZHK9D","pid":1192,"client":"2","":"downstairs","session_id":"2cc58e3a-bc6a-49a6-8671-87173741b601"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-04-17T17:14:00.960512032Z","hostname":"w-01HVPGMT7GXTS2BQW5HJHZHK9D","pid":1192,"":"io task","client":"2","":"downstairs","session_id":"2cc58e3a-bc6a-49a6-8671-87173741b601"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-04-17T17:14:00.960530372Z","hostname":"w-01HVPGMT7GXTS2BQW5HJHZHK9D","pid":1192,"":"io task","client":"1","":"downstairs","session_id":"2cc58e3a-bc6a-49a6-8671-87173741b601"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-04-17T17:14:00.960548151Z","hostname":"w-01HVPGMT7GXTS2BQW5HJHZHK9D","pid":1192,"":"io task","client":"0","":"downstairs","session_id":"2cc58e3a-bc6a-49a6-8671-87173741b601"} {"msg":"ds_connection connected from Ok(127.0.0.1:49731)","v":0,"name":"crucible","level":30,"time":"2024-04-17T17:14:00.96066168Z","hostname":"w-01HVPGMT7GXTS2BQW5HJHZHK9D","pid":1192,"":"io task","client":"0","":"downstairs","session_id":"2cc58e3a-bc6a-49a6-8671-87173741b601"} {"msg":"ds_connection connected from Ok(127.0.0.1:51931)","v":0,"name":"crucible","level":30,"time":"2024-04-17T17:14:00.96067832Z","hostname":"w-01HVPGMT7GXTS2BQW5HJHZHK9D","pid":1192,"":"io task","client":"1","":"downstairs","session_id":"2cc58e3a-bc6a-49a6-8671-87173741b601"} {"msg":"ds_connection connected from Ok(127.0.0.1:40446)","v":0,"name":"crucible","level":30,"time":"2024-04-17T17:14:00.960688709Z","hostname":"w-01HVPGMT7GXTS2BQW5HJHZHK9D","pid":1192,"":"io task","client":"2","":"downstairs","session_id":"2cc58e3a-bc6a-49a6-8671-87173741b601"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-04-17T17:14:00.961178731Z","hostname":"w-01HVPGMT7GXTS2BQW5HJHZHK9D","pid":1192,"client":"1","":"downstairs","session_id":"2cc58e3a-bc6a-49a6-8671-87173741b601"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-04-17T17:14:00.961204071Z","hostname":"w-01HVPGMT7GXTS2BQW5HJHZHK9D","pid":1192,"client":"1","":"downstairs","session_id":"2cc58e3a-bc6a-49a6-8671-87173741b601"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-04-17T17:14:00.961213511Z","hostname":"w-01HVPGMT7GXTS2BQW5HJHZHK9D","pid":1192,"client":"2","":"downstairs","session_id":"2cc58e3a-bc6a-49a6-8671-87173741b601"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-04-17T17:14:00.96122102Z","hostname":"w-01HVPGMT7GXTS2BQW5HJHZHK9D","pid":1192,"client":"2","":"downstairs","session_id":"2cc58e3a-bc6a-49a6-8671-87173741b601"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-04-17T17:14:00.961228771Z","hostname":"w-01HVPGMT7GXTS2BQW5HJHZHK9D","pid":1192,"client":"0","":"downstairs","session_id":"2cc58e3a-bc6a-49a6-8671-87173741b601"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-04-17T17:14:00.96123358Z","hostname":"w-01HVPGMT7GXTS2BQW5HJHZHK9D","pid":1192,"client":"0","":"downstairs","session_id":"2cc58e3a-bc6a-49a6-8671-87173741b601"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-04-17T17:14:00.961432687Z","hostname":"w-01HVPGMT7GXTS2BQW5HJHZHK9D","pid":1192,"client":"0","":"downstairs","session_id":"2cc58e3a-bc6a-49a6-8671-87173741b601"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-04-17T17:14:00.961454697Z","hostname":"w-01HVPGMT7GXTS2BQW5HJHZHK9D","pid":1192,"client":"1","":"downstairs","session_id":"2cc58e3a-bc6a-49a6-8671-87173741b601"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-04-17T17:14:00.961465947Z","hostname":"w-01HVPGMT7GXTS2BQW5HJHZHK9D","pid":1192,"client":"2","":"downstairs","session_id":"2cc58e3a-bc6a-49a6-8671-87173741b601"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-04-17T17:14:00.96371441Z","hostname":"w-01HVPGMT7GXTS2BQW5HJHZHK9D","pid":1192,"client":"0","":"downstairs","session_id":"2cc58e3a-bc6a-49a6-8671-87173741b601"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-04-17T17:14:00.963733079Z","hostname":"w-01HVPGMT7GXTS2BQW5HJHZHK9D","pid":1192,"client":"0","":"downstairs","session_id":"2cc58e3a-bc6a-49a6-8671-87173741b601"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-04-17T17:14:00.963741239Z","hostname":"w-01HVPGMT7GXTS2BQW5HJHZHK9D","pid":1192,"session_id":"2cc58e3a-bc6a-49a6-8671-87173741b601"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-04-17T17:14:00.963895857Z","hostname":"w-01HVPGMT7GXTS2BQW5HJHZHK9D","pid":1192,"client":"2","":"downstairs","session_id":"2cc58e3a-bc6a-49a6-8671-87173741b601"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-04-17T17:14:00.963909696Z","hostname":"w-01HVPGMT7GXTS2BQW5HJHZHK9D","pid":1192,"client":"2","":"downstairs","session_id":"2cc58e3a-bc6a-49a6-8671-87173741b601"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-04-17T17:14:00.963916656Z","hostname":"w-01HVPGMT7GXTS2BQW5HJHZHK9D","pid":1192,"session_id":"2cc58e3a-bc6a-49a6-8671-87173741b601"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-04-17T17:14:00.963925156Z","hostname":"w-01HVPGMT7GXTS2BQW5HJHZHK9D","pid":1192,"client":"1","":"downstairs","session_id":"2cc58e3a-bc6a-49a6-8671-87173741b601"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-04-17T17:14:00.963931906Z","hostname":"w-01HVPGMT7GXTS2BQW5HJHZHK9D","pid":1192,"client":"1","":"downstairs","session_id":"2cc58e3a-bc6a-49a6-8671-87173741b601"} {"msg":"[0]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-04-17T17:14:00.963939846Z","hostname":"w-01HVPGMT7GXTS2BQW5HJHZHK9D","pid":1192,"":"downstairs","":"downstairs","session_id":"2cc58e3a-bc6a-49a6-8671-87173741b601"} {"msg":"[0]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-04-17T17:14:00.964019885Z","hostname":"w-01HVPGMT7GXTS2BQW5HJHZHK9D","pid":1192,"":"downstairs","":"downstairs","session_id":"2cc58e3a-bc6a-49a6-8671-87173741b601"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-04-17T17:14:00.964041074Z","hostname":"w-01HVPGMT7GXTS2BQW5HJHZHK9D","pid":1192,"":"downstairs","":"downstairs","session_id":"2cc58e3a-bc6a-49a6-8671-87173741b601"} {"msg":"[1]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-04-17T17:14:00.964057274Z","hostname":"w-01HVPGMT7GXTS2BQW5HJHZHK9D","pid":1192,"":"downstairs","":"downstairs","session_id":"2cc58e3a-bc6a-49a6-8671-87173741b601"} {"msg":"[1]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2024-04-17T17:14:00.964065194Z","hostname":"w-01HVPGMT7GXTS2BQW5HJHZHK9D","pid":1192,"":"downstairs","":"downstairs","session_id":"2cc58e3a-bc6a-49a6-8671-87173741b601"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-04-17T17:14:00.964073924Z","hostname":"w-01HVPGMT7GXTS2BQW5HJHZHK9D","pid":1192,"":"downstairs","":"downstairs","session_id":"2cc58e3a-bc6a-49a6-8671-87173741b601"} {"msg":"[2]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-04-17T17:14:00.964089773Z","hostname":"w-01HVPGMT7GXTS2BQW5HJHZHK9D","pid":1192,"":"downstairs","":"downstairs","session_id":"2cc58e3a-bc6a-49a6-8671-87173741b601"} {"msg":"[2]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-04-17T17:14:00.964096284Z","hostname":"w-01HVPGMT7GXTS2BQW5HJHZHK9D","pid":1192,"":"downstairs","":"downstairs","session_id":"2cc58e3a-bc6a-49a6-8671-87173741b601"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-04-17T17:14:00.964100683Z","hostname":"w-01HVPGMT7GXTS2BQW5HJHZHK9D","pid":1192,"":"downstairs","":"downstairs","session_id":"2cc58e3a-bc6a-49a6-8671-87173741b601"} {"msg":"Max found gen is 1","v":0,"name":"crucible","level":30,"time":"2024-04-17T17:14:00.964106903Z","hostname":"w-01HVPGMT7GXTS2BQW5HJHZHK9D","pid":1192,"":"downstairs","":"downstairs","session_id":"2cc58e3a-bc6a-49a6-8671-87173741b601"} {"msg":"Generation requested: 1 >= found:1","v":0,"name":"crucible","level":30,"time":"2024-04-17T17:14:00.964114863Z","hostname":"w-01HVPGMT7GXTS2BQW5HJHZHK9D","pid":1192,"":"downstairs","":"downstairs","session_id":"2cc58e3a-bc6a-49a6-8671-87173741b601"} {"msg":"Next flush: 1","v":0,"name":"crucible","level":30,"time":"2024-04-17T17:14:00.964121943Z","hostname":"w-01HVPGMT7GXTS2BQW5HJHZHK9D","pid":1192,"":"downstairs","":"downstairs","session_id":"2cc58e3a-bc6a-49a6-8671-87173741b601"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-04-17T17:14:00.964135903Z","hostname":"w-01HVPGMT7GXTS2BQW5HJHZHK9D","pid":1192,"":"downstairs","":"downstairs","session_id":"2cc58e3a-bc6a-49a6-8671-87173741b601"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-04-17T17:14:00.964140463Z","hostname":"w-01HVPGMT7GXTS2BQW5HJHZHK9D","pid":1192,"session_id":"2cc58e3a-bc6a-49a6-8671-87173741b601"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-04-17T17:14:00.964145113Z","hostname":"w-01HVPGMT7GXTS2BQW5HJHZHK9D","pid":1192,"client":"0","":"downstairs","session_id":"2cc58e3a-bc6a-49a6-8671-87173741b601"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-04-17T17:14:00.964152043Z","hostname":"w-01HVPGMT7GXTS2BQW5HJHZHK9D","pid":1192,"client":"1","":"downstairs","session_id":"2cc58e3a-bc6a-49a6-8671-87173741b601"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-04-17T17:14:00.964159552Z","hostname":"w-01HVPGMT7GXTS2BQW5HJHZHK9D","pid":1192,"client":"2","":"downstairs","session_id":"2cc58e3a-bc6a-49a6-8671-87173741b601"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-04-17T17:14:00.964166732Z","hostname":"w-01HVPGMT7GXTS2BQW5HJHZHK9D","pid":1192,"session_id":"2cc58e3a-bc6a-49a6-8671-87173741b601"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-04-17T17:14:00.964174502Z","hostname":"w-01HVPGMT7GXTS2BQW5HJHZHK9D","pid":1192,"session_id":"2cc58e3a-bc6a-49a6-8671-87173741b601"} {"msg":"8af3f883-cb3d-444d-b479-a75b52bbe518 is now active with session: 2cc58e3a-bc6a-49a6-8671-87173741b601","v":0,"name":"crucible","level":30,"time":"2024-04-17T17:14:00.964181122Z","hostname":"w-01HVPGMT7GXTS2BQW5HJHZHK9D","pid":1192,"session_id":"2cc58e3a-bc6a-49a6-8671-87173741b601"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-04-17T17:14:00.964187442Z","hostname":"w-01HVPGMT7GXTS2BQW5HJHZHK9D","pid":1192,"session_id":"2cc58e3a-bc6a-49a6-8671-87173741b601"} {"msg":"The guest has finished waiting for activation with:1","v":0,"name":"crucible","level":30,"time":"2024-04-17T17:14:00.964194392Z","hostname":"w-01HVPGMT7GXTS2BQW5HJHZHK9D","pid":1192} Region: es:16384 ec:640 bs:4096 ts:42949672960 tb:10485760 max_io:256 or 1048576 Perf test TEST SECONDS COUNT DPTH IOPS MEAN P95 P99 MAX ES EC rwrites 11.23 30000 1 2672.11 0.00037 0.00130 0.00178 0.06248 16384 640 rwrites 7.56 30000 1 3970.14 0.00025 0.00058 0.00138 0.06452 16384 640 rreads 5.60 30000 1 5353.97 0.00019 0.00023 0.00029 0.01276 16384 640 rreads 5.54 30000 1 5417.91 0.00019 0.00023 0.00028 0.01314 16384 640 Perf test completed, stop all downstairs Apr 17 17:14:37.015 INFO accepted connection, remote_addr: 127.0.0.1:56869, local_addr: 127.0.0.1:9998 Main task has work to do, go find it got dsc Shutdown Shutdown Shut it down Apr 17 17:14:37.015 INFO request completed[8810] Disable keep_running , [8810] Got stop action so:false kr:false latency_us[8830] Disable keep_running : 54, response_code: 204, uri: /shutdown, method: POST, req_id: 62b90361-da9c-40b8-999b-ccdbab5b3edd, remote_addr: [8830] Got stop action so:false kr:false 127.0.0.1:56869, local_addr: 127.0.0.1:9998 [8820] Disable keep_running [8820] Got stop action so:false kr:false [8830] Exited with: ExitStatus(unix_wait_status(9)) [8810] Exited with: ExitStatus(unix_wait_status(9)) [8830][2] reports Exit [8810][0] reports Exit [8820] Exited with: ExitStatus(unix_wait_status(9)) [8820][1] reports Exit Create region with ES:16384 EC:1280 BS:4096 Removing existing dsc directory "/tmp/dsc" Removing existing region "/var/tmp/dsc" Creating region directory at: /var/tmp/dsc Creating dsc directory at: /tmp/dsc Downstairs region 0 created at /var/tmp/dsc/8810 in 1.2483962 Downstairs region 1 created at /var/tmp/dsc/8820 in 1.32739 Downstairs region 2 created at /var/tmp/dsc/8830 in 1.36992 Region set with 3 regions was created Using existing output directory "/tmp/dsc" start ds: 8810 start ds: 8820 start ds: 8830 start access at:127.0.0.1:9998 Starting downstairs at port 8810 Make output file at "/tmp/dsc/downstairs-8810.txt" [8810][0] reports Starting Starting downstairs at port 8820 Starting downstairs at port 8830 Make output file at "/tmp/dsc/downstairs-8830.txt" Make output file at "/tmp/dsc/downstairs-8820.txt" [8820][1] reports Starting [8830][2] reports Starting Downstairs /var/tmp/dsc/8810 port 8810 PID:Some(1201) [8810][0] reports Running Downstairs /var/tmp/dsc/8830 port 8830 PID:Some(1202) [8830][2] reports Running Downstairs /var/tmp/dsc/8820 port 8820 PID:Some(1203) [8820][1] reports Running Apr 17 17:14:42.025 INFO listening, local_addr: 127.0.0.1:9998 Control access at:127.0.0.1:9998 1202 1201 1203 Apr 17 17:14:47.063 INFO accepted connection, remote_addr: 127.0.0.1:58831, local_addr: 127.0.0.1:9998 Main task has work to do, go find it got dsc DisableRestartAll disable restart on all: 3 Apr 17 17:14:47.063 INFO request completed, latency_us: [8810] Disable keep_running 110[8820] Disable keep_running , response_code: 204, uri: /disablerestart/all, method[8830] Disable keep_running : POST, req_id: 9b1d4c20-ccd5-484c-9b16-4b447e4797dd, remote_addr: 127.0.0.1:58831, local_addr: 127.0.0.1:9998 A new loop begins at April 17, 2024 at 05:14:47 PM UTC /var/tmp/bins/crutest perf -g 1 -t 127.0.0.1:8810 -t 127.0.0.1:8820 -t 127.0.0.1:8830 -c 30000 -q --write-loops 2 --read-loops 2 --perf-out /tmp/perf-ES-16384-EC-1280.csv {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-04-17T17:14:47.082478322Z","hostname":"w-01HVPGMT7GXTS2BQW5HJHZHK9D","pid":1211} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"9dd6771ac5213adb2b97af7a72c1375fe36d2636\",\n git_commit_timestamp: \"2024-04-17T16:00:39.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: false,\n opt_level: 3,\n}","v":0,"name":"crucible","level":30,"time":"2024-04-17T17:14:47.08261517Z","hostname":"w-01HVPGMT7GXTS2BQW5HJHZHK9D","pid":1211} {"msg":"Upstairs <-> Downstairs Message Version: 6","v":0,"name":"crucible","level":30,"time":"2024-04-17T17:14:47.08262262Z","hostname":"w-01HVPGMT7GXTS2BQW5HJHZHK9D","pid":1211} {"msg":"Crucible ce881bdd-42ed-478a-b5a4-0e8b983cb73f has session id: ae8a9c72-d647-4955-b25b-d1dafe978c35","v":0,"name":"crucible","level":30,"time":"2024-04-17T17:14:47.08262736Z","hostname":"w-01HVPGMT7GXTS2BQW5HJHZHK9D","pid":1211,"session_id":"ae8a9c72-d647-4955-b25b-d1dafe978c35"} {"msg":"Upstairs opts: Upstairs UUID: ce881bdd-42ed-478a-b5a4-0e8b983cb73f, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-04-17T17:14:47.08263719Z","hostname":"w-01HVPGMT7GXTS2BQW5HJHZHK9D","pid":1211,"session_id":"ae8a9c72-d647-4955-b25b-d1dafe978c35"} {"msg":"Crucible stats registered with UUID: ce881bdd-42ed-478a-b5a4-0e8b983cb73f","v":0,"name":"crucible","level":30,"time":"2024-04-17T17:14:47.082642639Z","hostname":"w-01HVPGMT7GXTS2BQW5HJHZHK9D","pid":1211,"session_id":"ae8a9c72-d647-4955-b25b-d1dafe978c35"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:1","v":0,"name":"crucible","level":30,"time":"2024-04-17T17:14:47.082727828Z","hostname":"w-01HVPGMT7GXTS2BQW5HJHZHK9D","pid":1211} {"msg":"ce881bdd-42ed-478a-b5a4-0e8b983cb73f active request set","v":0,"name":"crucible","level":30,"time":"2024-04-17T17:14:47.082743748Z","hostname":"w-01HVPGMT7GXTS2BQW5HJHZHK9D","pid":1211,"session_id":"ae8a9c72-d647-4955-b25b-d1dafe978c35"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-04-17T17:14:47.082748568Z","hostname":"w-01HVPGMT7GXTS2BQW5HJHZHK9D","pid":1211,"client":"0","":"downstairs","session_id":"ae8a9c72-d647-4955-b25b-d1dafe978c35"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-04-17T17:14:47.082757788Z","hostname":"w-01HVPGMT7GXTS2BQW5HJHZHK9D","pid":1211,"client":"0","":"downstairs","session_id":"ae8a9c72-d647-4955-b25b-d1dafe978c35"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-04-17T17:14:47.082764037Z","hostname":"w-01HVPGMT7GXTS2BQW5HJHZHK9D","pid":1211,"client":"1","":"downstairs","session_id":"ae8a9c72-d647-4955-b25b-d1dafe978c35"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-04-17T17:14:47.082771657Z","hostname":"w-01HVPGMT7GXTS2BQW5HJHZHK9D","pid":1211,"client":"1","":"downstairs","session_id":"ae8a9c72-d647-4955-b25b-d1dafe978c35"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-04-17T17:14:47.082788187Z","hostname":"w-01HVPGMT7GXTS2BQW5HJHZHK9D","pid":1211,"client":"2","":"downstairs","session_id":"ae8a9c72-d647-4955-b25b-d1dafe978c35"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-04-17T17:14:47.082795967Z","hostname":"w-01HVPGMT7GXTS2BQW5HJHZHK9D","pid":1211,"client":"2","":"downstairs","session_id":"ae8a9c72-d647-4955-b25b-d1dafe978c35"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-04-17T17:14:47.082803707Z","hostname":"w-01HVPGMT7GXTS2BQW5HJHZHK9D","pid":1211,"":"io task","client":"2","":"downstairs","session_id":"ae8a9c72-d647-4955-b25b-d1dafe978c35"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-04-17T17:14:47.082842816Z","hostname":"w-01HVPGMT7GXTS2BQW5HJHZHK9D","pid":1211,"":"io task","client":"0","":"downstairs","session_id":"ae8a9c72-d647-4955-b25b-d1dafe978c35"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-04-17T17:14:47.082852196Z","hostname":"w-01HVPGMT7GXTS2BQW5HJHZHK9D","pid":1211,"":"io task","client":"1","":"downstairs","session_id":"ae8a9c72-d647-4955-b25b-d1dafe978c35"} {"msg":"ds_connection connected from Ok(127.0.0.1:41214)","v":0,"name":"crucible","level":30,"time":"2024-04-17T17:14:47.082949445Z","hostname":"w-01HVPGMT7GXTS2BQW5HJHZHK9D","pid":1211,"":"io task","client":"2","":"downstairs","session_id":"ae8a9c72-d647-4955-b25b-d1dafe978c35"} {"msg":"ds_connection connected from Ok(127.0.0.1:64122)","v":0,"name":"crucible","level":30,"time":"2024-04-17T17:14:47.082961644Z","hostname":"w-01HVPGMT7GXTS2BQW5HJHZHK9D","pid":1211,"":"io task","client":"0","":"downstairs","session_id":"ae8a9c72-d647-4955-b25b-d1dafe978c35"} {"msg":"ds_connection connected from Ok(127.0.0.1:33025)","v":0,"name":"crucible","level":30,"time":"2024-04-17T17:14:47.083058133Z","hostname":"w-01HVPGMT7GXTS2BQW5HJHZHK9D","pid":1211,"":"io task","client":"1","":"downstairs","session_id":"ae8a9c72-d647-4955-b25b-d1dafe978c35"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-04-17T17:14:47.083368408Z","hostname":"w-01HVPGMT7GXTS2BQW5HJHZHK9D","pid":1211,"client":"1","":"downstairs","session_id":"ae8a9c72-d647-4955-b25b-d1dafe978c35"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-04-17T17:14:47.083384317Z","hostname":"w-01HVPGMT7GXTS2BQW5HJHZHK9D","pid":1211,"client":"1","":"downstairs","session_id":"ae8a9c72-d647-4955-b25b-d1dafe978c35"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-04-17T17:14:47.083393257Z","hostname":"w-01HVPGMT7GXTS2BQW5HJHZHK9D","pid":1211,"client":"2","":"downstairs","session_id":"ae8a9c72-d647-4955-b25b-d1dafe978c35"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-04-17T17:14:47.083397827Z","hostname":"w-01HVPGMT7GXTS2BQW5HJHZHK9D","pid":1211,"client":"2","":"downstairs","session_id":"ae8a9c72-d647-4955-b25b-d1dafe978c35"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-04-17T17:14:47.083437346Z","hostname":"w-01HVPGMT7GXTS2BQW5HJHZHK9D","pid":1211,"client":"0","":"downstairs","session_id":"ae8a9c72-d647-4955-b25b-d1dafe978c35"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-04-17T17:14:47.083459096Z","hostname":"w-01HVPGMT7GXTS2BQW5HJHZHK9D","pid":1211,"client":"0","":"downstairs","session_id":"ae8a9c72-d647-4955-b25b-d1dafe978c35"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-04-17T17:14:47.083640783Z","hostname":"w-01HVPGMT7GXTS2BQW5HJHZHK9D","pid":1211,"client":"1","":"downstairs","session_id":"ae8a9c72-d647-4955-b25b-d1dafe978c35"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-04-17T17:14:47.083660193Z","hostname":"w-01HVPGMT7GXTS2BQW5HJHZHK9D","pid":1211,"client":"0","":"downstairs","session_id":"ae8a9c72-d647-4955-b25b-d1dafe978c35"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-04-17T17:14:47.083668393Z","hostname":"w-01HVPGMT7GXTS2BQW5HJHZHK9D","pid":1211,"client":"2","":"downstairs","session_id":"ae8a9c72-d647-4955-b25b-d1dafe978c35"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-04-17T17:14:47.089526017Z","hostname":"w-01HVPGMT7GXTS2BQW5HJHZHK9D","pid":1211,"client":"1","":"downstairs","session_id":"ae8a9c72-d647-4955-b25b-d1dafe978c35"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-04-17T17:14:47.089543846Z","hostname":"w-01HVPGMT7GXTS2BQW5HJHZHK9D","pid":1211,"client":"1","":"downstairs","session_id":"ae8a9c72-d647-4955-b25b-d1dafe978c35"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-04-17T17:14:47.089549517Z","hostname":"w-01HVPGMT7GXTS2BQW5HJHZHK9D","pid":1211,"session_id":"ae8a9c72-d647-4955-b25b-d1dafe978c35"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-04-17T17:14:47.090014599Z","hostname":"w-01HVPGMT7GXTS2BQW5HJHZHK9D","pid":1211,"client":"0","":"downstairs","session_id":"ae8a9c72-d647-4955-b25b-d1dafe978c35"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-04-17T17:14:47.090030509Z","hostname":"w-01HVPGMT7GXTS2BQW5HJHZHK9D","pid":1211,"client":"0","":"downstairs","session_id":"ae8a9c72-d647-4955-b25b-d1dafe978c35"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-04-17T17:14:47.090035409Z","hostname":"w-01HVPGMT7GXTS2BQW5HJHZHK9D","pid":1211,"session_id":"ae8a9c72-d647-4955-b25b-d1dafe978c35"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-04-17T17:14:47.090205376Z","hostname":"w-01HVPGMT7GXTS2BQW5HJHZHK9D","pid":1211,"client":"2","":"downstairs","session_id":"ae8a9c72-d647-4955-b25b-d1dafe978c35"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-04-17T17:14:47.090222886Z","hostname":"w-01HVPGMT7GXTS2BQW5HJHZHK9D","pid":1211,"client":"2","":"downstairs","session_id":"ae8a9c72-d647-4955-b25b-d1dafe978c35"} {"msg":"[0]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-04-17T17:14:47.090228035Z","hostname":"w-01HVPGMT7GXTS2BQW5HJHZHK9D","pid":1211,"":"downstairs","":"downstairs","session_id":"ae8a9c72-d647-4955-b25b-d1dafe978c35"} {"msg":"[0]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-04-17T17:14:47.090232785Z","hostname":"w-01HVPGMT7GXTS2BQW5HJHZHK9D","pid":1211,"":"downstairs","":"downstairs","session_id":"ae8a9c72-d647-4955-b25b-d1dafe978c35"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-04-17T17:14:47.090237295Z","hostname":"w-01HVPGMT7GXTS2BQW5HJHZHK9D","pid":1211,"":"downstairs","":"downstairs","session_id":"ae8a9c72-d647-4955-b25b-d1dafe978c35"} {"msg":"[1]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-04-17T17:14:47.090242675Z","hostname":"w-01HVPGMT7GXTS2BQW5HJHZHK9D","pid":1211,"":"downstairs","":"downstairs","session_id":"ae8a9c72-d647-4955-b25b-d1dafe978c35"} {"msg":"[1]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-04-17T17:14:47.090247085Z","hostname":"w-01HVPGMT7GXTS2BQW5HJHZHK9D","pid":1211,"":"downstairs","":"downstairs","session_id":"ae8a9c72-d647-4955-b25b-d1dafe978c35"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-04-17T17:14:47.090251555Z","hostname":"w-01HVPGMT7GXTS2BQW5HJHZHK9D","pid":1211,"":"downstairs","":"downstairs","session_id":"ae8a9c72-d647-4955-b25b-d1dafe978c35"} {"msg":"[2]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-04-17T17:14:47.090255865Z","hostname":"w-01HVPGMT7GXTS2BQW5HJHZHK9D","pid":1211,"":"downstairs","":"downstairs","session_id":"ae8a9c72-d647-4955-b25b-d1dafe978c35"} {"msg":"[2]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-04-17T17:14:47.090260175Z","hostname":"w-01HVPGMT7GXTS2BQW5HJHZHK9D","pid":1211,"":"downstairs","":"downstairs","session_id":"ae8a9c72-d647-4955-b25b-d1dafe978c35"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-04-17T17:14:47.090264895Z","hostname":"w-01HVPGMT7GXTS2BQW5HJHZHK9D","pid":1211,"":"downstairs","":"downstairs","session_id":"ae8a9c72-d647-4955-b25b-d1dafe978c35"} {"msg":"Max found gen is 1","v":0,"name":"crucible","level":30,"time":"2024-04-17T17:14:47.090269025Z","hostname":"w-01HVPGMT7GXTS2BQW5HJHZHK9D","pid":1211,"":"downstairs","":"downstairs","session_id":"ae8a9c72-d647-4955-b25b-d1dafe978c35"} {"msg":"Generation requested: 1 >= found:1","v":0,"name":"crucible","level":30,"time":"2024-04-17T17:14:47.090273285Z","hostname":"w-01HVPGMT7GXTS2BQW5HJHZHK9D","pid":1211,"":"downstairs","":"downstairs","session_id":"ae8a9c72-d647-4955-b25b-d1dafe978c35"} {"msg":"Next flush: 1","v":0,"name":"crucible","level":30,"time":"2024-04-17T17:14:47.090277894Z","hostname":"w-01HVPGMT7GXTS2BQW5HJHZHK9D","pid":1211,"":"downstairs","":"downstairs","session_id":"ae8a9c72-d647-4955-b25b-d1dafe978c35"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-04-17T17:14:47.090281974Z","hostname":"w-01HVPGMT7GXTS2BQW5HJHZHK9D","pid":1211,"":"downstairs","":"downstairs","session_id":"ae8a9c72-d647-4955-b25b-d1dafe978c35"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-04-17T17:14:47.090286184Z","hostname":"w-01HVPGMT7GXTS2BQW5HJHZHK9D","pid":1211,"session_id":"ae8a9c72-d647-4955-b25b-d1dafe978c35"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-04-17T17:14:47.090290234Z","hostname":"w-01HVPGMT7GXTS2BQW5HJHZHK9D","pid":1211,"client":"0","":"downstairs","session_id":"ae8a9c72-d647-4955-b25b-d1dafe978c35"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-04-17T17:14:47.090294994Z","hostname":"w-01HVPGMT7GXTS2BQW5HJHZHK9D","pid":1211,"client":"1","":"downstairs","session_id":"ae8a9c72-d647-4955-b25b-d1dafe978c35"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-04-17T17:14:47.090299144Z","hostname":"w-01HVPGMT7GXTS2BQW5HJHZHK9D","pid":1211,"client":"2","":"downstairs","session_id":"ae8a9c72-d647-4955-b25b-d1dafe978c35"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-04-17T17:14:47.090303834Z","hostname":"w-01HVPGMT7GXTS2BQW5HJHZHK9D","pid":1211,"session_id":"ae8a9c72-d647-4955-b25b-d1dafe978c35"} Wait for a query_work_queue command to finish before sending IO {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-04-17T17:14:47.090309544Z","hostname":"w-01HVPGMT7GXTS2BQW5HJHZHK9D","pid":1211,"session_id":"ae8a9c72-d647-4955-b25b-d1dafe978c35"} {"msg":"ce881bdd-42ed-478a-b5a4-0e8b983cb73f is now active with session: ae8a9c72-d647-4955-b25b-d1dafe978c35","v":0,"name":"crucible","level":30,"time":"2024-04-17T17:14:47.090313354Z","hostname":"w-01HVPGMT7GXTS2BQW5HJHZHK9D","pid":1211,"session_id":"ae8a9c72-d647-4955-b25b-d1dafe978c35"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-04-17T17:14:47.090317524Z","hostname":"w-01HVPGMT7GXTS2BQW5HJHZHK9D","pid":1211,"session_id":"ae8a9c72-d647-4955-b25b-d1dafe978c35"} {"msg":"The guest has finished waiting for activation with:1","v":0,"name":"crucible","level":30,"time":"2024-04-17T17:14:47.090321164Z","hostname":"w-01HVPGMT7GXTS2BQW5HJHZHK9D","pid":1211} Region: es:16384 ec:1280 bs:4096 ts:85899345920 tb:20971520 max_io:256 or 1048576 Perf test TEST SECONDS COUNT DPTH IOPS MEAN P95 P99 MAX ES EC rwrites 8.27 30000 1 3627.90 0.00028 0.00059 0.00120 0.06959 16384 1280 rwrites 8.00 30000 1 3747.70 0.00027 0.00056 0.00114 0.09276 16384 1280 rreads 5.49 30000 1 5462.19 0.00018 0.00022 0.00027 0.01301 16384 1280 rreads 5.52 30000 1 5430.36 0.00018 0.00022 0.00027 0.01301 16384 1280 Perf test completed, stop all downstairs Apr 17 17:15:25.620 INFO accepted connection, remote_addr: 127.0.0.1:46472, local_addr: 127.0.0.1:9998 Main task has work to do, go find it got dsc Shutdown Shutdown Shut it down Apr 17 17:15:25.621 INFO [8810] Disable keep_running request completed[8820] Disable keep_running , latency_us: 49, [8820] Got stop action so:false kr:false response_code: [8830] Disable keep_running 204, uri: /shutdown, method:[8810] Got stop action so:false kr:false POST, req_id: 56eae7c6-ca21-4736-9ace-41f9576ad824, remote_addr: 127.0.0.1:46472, local_addr: 127.0.0.1:9998 [8830] Got stop action so:false kr:false [8820] Exited with: ExitStatus(unix_wait_status(9)) [8820][1] reports Exit [8810] Exited with: ExitStatus(unix_wait_status(9)) [8810][0] reports Exit [8830] Exited with: ExitStatus(unix_wait_status(9)) [8830][2] reports Exit TEST SECONDS COUNT DPTH IOPS MEAN P95 P99 MAX ES EC rwrites 11.23 30000 1 2672.11 0.00037 0.00130 0.00178 0.06248 16384 640 rwrites 7.56 30000 1 3970.14 0.00025 0.00058 0.00138 0.06452 16384 640 rwrites 8.27 30000 1 3627.90 0.00028 0.00059 0.00120 0.06959 16384 1280 rwrites 8.00 30000 1 3747.70 0.00027 0.00056 0.00114 0.09276 16384 1280 TEST SECONDS COUNT DPTH IOPS MEAN P95 P99 MAX ES EC rreads 5.60 30000 1 5353.97 0.00019 0.00023 0.00029 0.01276 16384 640 rreads 5.54 30000 1 5417.91 0.00019 0.00023 0.00028 0.01314 16384 640 rreads 5.49 30000 1 5462.19 0.00018 0.00022 0.00027 0.01301 16384 1280 rreads 5.52 30000 1 5430.36 0.00018 0.00022 0.00027 0.01301 16384 1280 Perf test finished on April 17, 2024 at 05:15:26 PM UTC