/input/rbuild/work Create region with ES:16384 EC:640 BS:4096 Creating region directory at: /var/tmp/dsc Creating dsc directory at: /tmp/dsc Downstairs region 0 created at /var/tmp/dsc/8810 in 0.7444764 Downstairs region 1 created at /var/tmp/dsc/8820 in 0.59082025 Downstairs region 2 created at /var/tmp/dsc/8830 in 0.6550009 Region set with 3 regions was created Using existing output directory "/tmp/dsc" start ds: 8810 start ds: 8820 start ds: 8830 Starting downstairs at port 8810 start access at:127.0.0.1:9998 Starting downstairs at port 8820 Starting downstairs at port 8830 Make output file at "/tmp/dsc/downstairs-8810.txt" [8810][0] reports Starting Make output file at "/tmp/dsc/downstairs-8830.txt" Make output file at "/tmp/dsc/downstairs-8820.txt" [8830][2] reports Starting [8820][1] reports Starting Downstairs /var/tmp/dsc/8810 port 8810 PID:Some(1151) [8810][0] reports Running Downstairs /var/tmp/dsc/8820 port 8820 PID:Some(1152) [8820][1] reports Running Downstairs /var/tmp/dsc/8830 port 8830 PID:Some(1153) [8830][2] reports Running Mar 06 21:01:15.168 INFO listening, local_addr: 127.0.0.1:9998 Control access at:127.0.0.1:9998 1151 1152 1153 Mar 06 21:01:20.206 INFO accepted connection, remote_addr: 127.0.0.1:45861, local_addr: 127.0.0.1:9998 Main task has work to do, go find it got dsc DisableRestartAll disable restart on all: 3 Mar 06 21:01:20.207 INFO request completed, latency_us: 125, response_code: 204[8820] Disable keep_running , [8810] Disable keep_running uri: /disablerestart/all, method: POST[8830] Disable keep_running , req_id: 37f24b55-75d2-4e60-bbd1-5190cf7e8e88, remote_addr: 127.0.0.1:45861, local_addr: 127.0.0.1:9998 A new loop begins at March 6, 2024 at 09:01:20 PM UTC /var/tmp/bins/crutest perf -g 1 -t 127.0.0.1:8810 -t 127.0.0.1:8820 -t 127.0.0.1:8830 -c 30000 -q --write-loops 2 --read-loops 2 --perf-out /tmp/perf-ES-16384-EC-640.csv {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:01:20.229073391Z","hostname":"w-01HRAS08V1Y203C0H5ESGPJW6B","pid":1161} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"27c060b02133783737d505ea2610b3d179ebd892\",\n git_commit_timestamp: \"2024-03-06T20:36:21.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: false,\n opt_level: 3,\n}","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:01:20.229285589Z","hostname":"w-01HRAS08V1Y203C0H5ESGPJW6B","pid":1161} {"msg":"Upstairs <-> Downstairs Message Version: 5","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:01:20.229293559Z","hostname":"w-01HRAS08V1Y203C0H5ESGPJW6B","pid":1161} {"msg":"Crucible 917949c9-6306-4f97-942b-e83e15dbc96d has session id: 9d94f108-6c0d-4446-80de-31f7a3b575b6","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:01:20.229299118Z","hostname":"w-01HRAS08V1Y203C0H5ESGPJW6B","pid":1161,"session_id":"9d94f108-6c0d-4446-80de-31f7a3b575b6"} {"msg":"Upstairs opts: Upstairs UUID: 917949c9-6306-4f97-942b-e83e15dbc96d, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:01:20.229310908Z","hostname":"w-01HRAS08V1Y203C0H5ESGPJW6B","pid":1161,"session_id":"9d94f108-6c0d-4446-80de-31f7a3b575b6"} {"msg":"Crucible stats registered with UUID: 917949c9-6306-4f97-942b-e83e15dbc96d","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:01:20.229316537Z","hostname":"w-01HRAS08V1Y203C0H5ESGPJW6B","pid":1161,"session_id":"9d94f108-6c0d-4446-80de-31f7a3b575b6"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:1","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:01:20.229486438Z","hostname":"w-01HRAS08V1Y203C0H5ESGPJW6B","pid":1161} {"msg":"917949c9-6306-4f97-942b-e83e15dbc96d active request set","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:01:20.229660108Z","hostname":"w-01HRAS08V1Y203C0H5ESGPJW6B","pid":1161,"session_id":"9d94f108-6c0d-4446-80de-31f7a3b575b6"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:01:20.229677396Z","hostname":"w-01HRAS08V1Y203C0H5ESGPJW6B","pid":1161,"client":"0","":"downstairs","session_id":"9d94f108-6c0d-4446-80de-31f7a3b575b6"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:01:20.229696815Z","hostname":"w-01HRAS08V1Y203C0H5ESGPJW6B","pid":1161,"client":"0","":"downstairs","session_id":"9d94f108-6c0d-4446-80de-31f7a3b575b6"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:01:20.229704805Z","hostname":"w-01HRAS08V1Y203C0H5ESGPJW6B","pid":1161,"client":"1","":"downstairs","session_id":"9d94f108-6c0d-4446-80de-31f7a3b575b6"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:01:20.229721154Z","hostname":"w-01HRAS08V1Y203C0H5ESGPJW6B","pid":1161,"client":"1","":"downstairs","session_id":"9d94f108-6c0d-4446-80de-31f7a3b575b6"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:01:20.229729073Z","hostname":"w-01HRAS08V1Y203C0H5ESGPJW6B","pid":1161,"client":"2","":"downstairs","session_id":"9d94f108-6c0d-4446-80de-31f7a3b575b6"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:01:20.229736853Z","hostname":"w-01HRAS08V1Y203C0H5ESGPJW6B","pid":1161,"client":"2","":"downstairs","session_id":"9d94f108-6c0d-4446-80de-31f7a3b575b6"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:01:20.229755282Z","hostname":"w-01HRAS08V1Y203C0H5ESGPJW6B","pid":1161,"":"io task","client":"0","":"downstairs","session_id":"9d94f108-6c0d-4446-80de-31f7a3b575b6"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:01:20.229769741Z","hostname":"w-01HRAS08V1Y203C0H5ESGPJW6B","pid":1161,"":"io task","client":"2","":"downstairs","session_id":"9d94f108-6c0d-4446-80de-31f7a3b575b6"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:01:20.229778141Z","hostname":"w-01HRAS08V1Y203C0H5ESGPJW6B","pid":1161,"":"io task","client":"1","":"downstairs","session_id":"9d94f108-6c0d-4446-80de-31f7a3b575b6"} {"msg":"ds_connection connected","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:01:20.22996365Z","hostname":"w-01HRAS08V1Y203C0H5ESGPJW6B","pid":1161,"":"io task","client":"1","":"downstairs","session_id":"9d94f108-6c0d-4446-80de-31f7a3b575b6"} {"msg":"ds_connection connected","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:01:20.230027246Z","hostname":"w-01HRAS08V1Y203C0H5ESGPJW6B","pid":1161,"":"io task","client":"0","":"downstairs","session_id":"9d94f108-6c0d-4446-80de-31f7a3b575b6"} {"msg":"ds_connection connected","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:01:20.230058685Z","hostname":"w-01HRAS08V1Y203C0H5ESGPJW6B","pid":1161,"":"io task","client":"2","":"downstairs","session_id":"9d94f108-6c0d-4446-80de-31f7a3b575b6"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:01:20.230588154Z","hostname":"w-01HRAS08V1Y203C0H5ESGPJW6B","pid":1161,"client":"1","":"downstairs","session_id":"9d94f108-6c0d-4446-80de-31f7a3b575b6"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:01:20.230605523Z","hostname":"w-01HRAS08V1Y203C0H5ESGPJW6B","pid":1161,"client":"1","":"downstairs","session_id":"9d94f108-6c0d-4446-80de-31f7a3b575b6"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:01:20.230610773Z","hostname":"w-01HRAS08V1Y203C0H5ESGPJW6B","pid":1161,"client":"2","":"downstairs","session_id":"9d94f108-6c0d-4446-80de-31f7a3b575b6"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:01:20.230615122Z","hostname":"w-01HRAS08V1Y203C0H5ESGPJW6B","pid":1161,"client":"2","":"downstairs","session_id":"9d94f108-6c0d-4446-80de-31f7a3b575b6"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:01:20.230622272Z","hostname":"w-01HRAS08V1Y203C0H5ESGPJW6B","pid":1161,"client":"0","":"downstairs","session_id":"9d94f108-6c0d-4446-80de-31f7a3b575b6"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:01:20.230626692Z","hostname":"w-01HRAS08V1Y203C0H5ESGPJW6B","pid":1161,"client":"0","":"downstairs","session_id":"9d94f108-6c0d-4446-80de-31f7a3b575b6"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:01:20.230927254Z","hostname":"w-01HRAS08V1Y203C0H5ESGPJW6B","pid":1161,"client":"2","":"downstairs","session_id":"9d94f108-6c0d-4446-80de-31f7a3b575b6"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:01:20.230947943Z","hostname":"w-01HRAS08V1Y203C0H5ESGPJW6B","pid":1161,"client":"0","":"downstairs","session_id":"9d94f108-6c0d-4446-80de-31f7a3b575b6"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:01:20.230956902Z","hostname":"w-01HRAS08V1Y203C0H5ESGPJW6B","pid":1161,"client":"1","":"downstairs","session_id":"9d94f108-6c0d-4446-80de-31f7a3b575b6"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:01:20.233289298Z","hostname":"w-01HRAS08V1Y203C0H5ESGPJW6B","pid":1161,"client":"1","":"downstairs","session_id":"9d94f108-6c0d-4446-80de-31f7a3b575b6"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:01:20.233307037Z","hostname":"w-01HRAS08V1Y203C0H5ESGPJW6B","pid":1161,"client":"1","":"downstairs","session_id":"9d94f108-6c0d-4446-80de-31f7a3b575b6"} {"msg":"Waiting for true more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:01:20.233314976Z","hostname":"w-01HRAS08V1Y203C0H5ESGPJW6B","pid":1161,"session_id":"9d94f108-6c0d-4446-80de-31f7a3b575b6"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:01:20.233320746Z","hostname":"w-01HRAS08V1Y203C0H5ESGPJW6B","pid":1161,"client":"0","":"downstairs","session_id":"9d94f108-6c0d-4446-80de-31f7a3b575b6"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:01:20.233327526Z","hostname":"w-01HRAS08V1Y203C0H5ESGPJW6B","pid":1161,"client":"0","":"downstairs","session_id":"9d94f108-6c0d-4446-80de-31f7a3b575b6"} {"msg":"Waiting for true more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:01:20.233333875Z","hostname":"w-01HRAS08V1Y203C0H5ESGPJW6B","pid":1161,"session_id":"9d94f108-6c0d-4446-80de-31f7a3b575b6"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:01:20.233385552Z","hostname":"w-01HRAS08V1Y203C0H5ESGPJW6B","pid":1161,"client":"2","":"downstairs","session_id":"9d94f108-6c0d-4446-80de-31f7a3b575b6"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:01:20.233398541Z","hostname":"w-01HRAS08V1Y203C0H5ESGPJW6B","pid":1161,"client":"2","":"downstairs","session_id":"9d94f108-6c0d-4446-80de-31f7a3b575b6"} {"msg":"[0]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:01:20.233403001Z","hostname":"w-01HRAS08V1Y203C0H5ESGPJW6B","pid":1161,"":"downstairs","":"downstairs","session_id":"9d94f108-6c0d-4446-80de-31f7a3b575b6"} {"msg":"[0]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:01:20.233409151Z","hostname":"w-01HRAS08V1Y203C0H5ESGPJW6B","pid":1161,"":"downstairs","":"downstairs","session_id":"9d94f108-6c0d-4446-80de-31f7a3b575b6"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:01:20.233417Z","hostname":"w-01HRAS08V1Y203C0H5ESGPJW6B","pid":1161,"":"downstairs","":"downstairs","session_id":"9d94f108-6c0d-4446-80de-31f7a3b575b6"} {"msg":"[1]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:01:20.23342456Z","hostname":"w-01HRAS08V1Y203C0H5ESGPJW6B","pid":1161,"":"downstairs","":"downstairs","session_id":"9d94f108-6c0d-4446-80de-31f7a3b575b6"} {"msg":"[1]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:01:20.233481507Z","hostname":"w-01HRAS08V1Y203C0H5ESGPJW6B","pid":1161,"":"downstairs","":"downstairs","session_id":"9d94f108-6c0d-4446-80de-31f7a3b575b6"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:01:20.233494616Z","hostname":"w-01HRAS08V1Y203C0H5ESGPJW6B","pid":1161,"":"downstairs","":"downstairs","session_id":"9d94f108-6c0d-4446-80de-31f7a3b575b6"} {"msg":"[2]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:01:20.233502606Z","hostname":"w-01HRAS08V1Y203C0H5ESGPJW6B","pid":1161,"":"downstairs","":"downstairs","session_id":"9d94f108-6c0d-4446-80de-31f7a3b575b6"} {"msg":"[2]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:01:20.233530864Z","hostname":"w-01HRAS08V1Y203C0H5ESGPJW6B","pid":1161,"":"downstairs"Wait for a query_work_queue command to finish before sending IO ,"":"downstairs","session_id":"9d94f108-6c0d-4446-80de-31f7a3b575b6"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:01:20.233545323Z","hostname":"w-01HRAS08V1Y203C0H5ESGPJW6B","pid":1161,"":"downstairs","":"downstairs","session_id":"9d94f108-6c0d-4446-80de-31f7a3b575b6"} {"msg":"Max found gen is 1","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:01:20.233556012Z","hostname":"w-01HRAS08V1Y203C0H5ESGPJW6B","pid":1161,"":"downstairs","":"downstairs","session_id":"9d94f108-6c0d-4446-80de-31f7a3b575b6"} {"msg":"Generation requested: 1 >= found:1","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:01:20.233563662Z","hostname":"w-01HRAS08V1Y203C0H5ESGPJW6B","pid":1161,"":"downstairs","":"downstairs","session_id":"9d94f108-6c0d-4446-80de-31f7a3b575b6"} {"msg":"Next flush: 1","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:01:20.233570511Z","hostname":"w-01HRAS08V1Y203C0H5ESGPJW6B","pid":1161,"":"downstairs","":"downstairs","session_id":"9d94f108-6c0d-4446-80de-31f7a3b575b6"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:01:20.233578251Z","hostname":"w-01HRAS08V1Y203C0H5ESGPJW6B","pid":1161,"":"downstairs","":"downstairs","session_id":"9d94f108-6c0d-4446-80de-31f7a3b575b6"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:01:20.233585151Z","hostname":"w-01HRAS08V1Y203C0H5ESGPJW6B","pid":1161,"session_id":"9d94f108-6c0d-4446-80de-31f7a3b575b6"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:01:20.23359127Z","hostname":"w-01HRAS08V1Y203C0H5ESGPJW6B","pid":1161,"client":"0","":"downstairs","session_id":"9d94f108-6c0d-4446-80de-31f7a3b575b6"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:01:20.23359861Z","hostname":"w-01HRAS08V1Y203C0H5ESGPJW6B","pid":1161,"client":"1","":"downstairs","session_id":"9d94f108-6c0d-4446-80de-31f7a3b575b6"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:01:20.23360642Z","hostname":"w-01HRAS08V1Y203C0H5ESGPJW6B","pid":1161,"client":"2","":"downstairs","session_id":"9d94f108-6c0d-4446-80de-31f7a3b575b6"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:01:20.233613289Z","hostname":"w-01HRAS08V1Y203C0H5ESGPJW6B","pid":1161,"session_id":"9d94f108-6c0d-4446-80de-31f7a3b575b6"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:01:20.233619329Z","hostname":"w-01HRAS08V1Y203C0H5ESGPJW6B","pid":1161,"session_id":"9d94f108-6c0d-4446-80de-31f7a3b575b6"} {"msg":"917949c9-6306-4f97-942b-e83e15dbc96d is now active with session: 9d94f108-6c0d-4446-80de-31f7a3b575b6","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:01:20.233626738Z","hostname":"w-01HRAS08V1Y203C0H5ESGPJW6B","pid":1161,"session_id":"9d94f108-6c0d-4446-80de-31f7a3b575b6"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:01:20.233632708Z","hostname":"w-01HRAS08V1Y203C0H5ESGPJW6B","pid":1161,"session_id":"9d94f108-6c0d-4446-80de-31f7a3b575b6"} {"msg":"The guest has finished waiting for activation with:1","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:01:20.233638638Z","hostname":"w-01HRAS08V1Y203C0H5ESGPJW6B","pid":1161} Region: es:16384 ec:640 bs:4096 ts:42949672960 tb:10485760 max_io:256 or 1048576 Perf test TEST SECONDS COUNT DPTH IOPS MEAN P95 P99 MAX ES EC rwrites 17.37 30000 1 1726.63 0.00058 0.00144 0.00192 0.08547 16384 640 rwrites 8.50 30000 1 3530.51 0.00028 0.00067 0.00140 0.05179 16384 640 rreads 5.73 30000 1 5233.20 0.00019 0.00027 0.00031 0.01241 16384 640 rreads 5.69 30000 1 5268.23 0.00019 0.00026 0.00030 0.01241 16384 640 Perf test completed, stop all downstairs Mar 06 21:02:06.791 INFO accepted connection, remote_addr: 127.0.0.1:39884, local_addr: 127.0.0.1:9998 Main task has work to do, go find it got dsc Shutdown Shutdown Mar 06 21:02:06.791 INFO request completed, latency_usShut it down : [8810] Disable keep_running 50[8810] Got stop action so:false kr:false , [8830] Disable keep_running response_code:[8820] Disable keep_running 204, uri: /shutdown, method[8830] Got stop action so:false kr:false :[8820] Got stop action so:false kr:false POST, req_id: 28877ed5-eb20-4795-a077-c971a15b7b71, remote_addr: 127.0.0.1:39884, local_addr: 127.0.0.1:9998 [8810] Exited with: ExitStatus(unix_wait_status(9)) [8810][0] reports Exit [8830] Exited with: ExitStatus(unix_wait_status(9)) [8820] Exited with: ExitStatus(unix_wait_status(9)) [8830][2] reports Exit [8820][1] reports Exit Create region with ES:16384 EC:1280 BS:4096 Removing existing dsc directory "/tmp/dsc" Removing existing region "/var/tmp/dsc" Creating region directory at: /var/tmp/dsc Creating dsc directory at: /tmp/dsc Downstairs region 0 created at /var/tmp/dsc/8810 in 1.1841003 Downstairs region 1 created at /var/tmp/dsc/8820 in 1.1767057 Downstairs region 2 created at /var/tmp/dsc/8830 in 1.1488472 Region set with 3 regions was created Using existing output directory "/tmp/dsc" start ds: 8810 start ds: 8820 start ds: 8830 start access at:127.0.0.1:9998 Starting downstairs at port 8810 Make output file at "/tmp/dsc/downstairs-8810.txt" Starting downstairs at port 8820 [8810][0] reports Starting Make output file at "/tmp/dsc/downstairs-8820.txt" [8820][1] reports Starting Starting downstairs at port 8830 Make output file at "/tmp/dsc/downstairs-8830.txt" [8830][2] reports Starting Downstairs /var/tmp/dsc/8820 port 8820 PID:Some(1170) [8820][1] reports Running Downstairs /var/tmp/dsc/8810 port 8810 PID:Some(1171) [8810][0] reports Running Downstairs /var/tmp/dsc/8830 port 8830 PID:Some(1172) [8830][2] reports Running Mar 06 21:02:11.363 INFO listening, local_addr: 127.0.0.1:9998 Control access at:127.0.0.1:9998 1171 1170 1172 Mar 06 21:02:16.402 INFO accepted connection, remote_addr: 127.0.0.1:53759, local_addr: 127.0.0.1:9998 Main task has work to do, go find it got dsc DisableRestartAll disable restart on all: 3 Mar 06 21:02:16.402 INFO request completed, latency_us[8820] Disable keep_running : [8810] Disable keep_running 105[8830] Disable keep_running , response_code: 204, uri: /disablerestart/all, method: POST, req_id: 25d3fa16-ed2e-464c-a6df-cf63825dd943, remote_addr: 127.0.0.1:53759, local_addr: 127.0.0.1:9998 A new loop begins at March 6, 2024 at 09:02:16 PM UTC /var/tmp/bins/crutest perf -g 1 -t 127.0.0.1:8810 -t 127.0.0.1:8820 -t 127.0.0.1:8830 -c 30000 -q --write-loops 2 --read-loops 2 --perf-out /tmp/perf-ES-16384-EC-1280.csv {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:02:16.421363206Z","hostname":"w-01HRAS08V1Y203C0H5ESGPJW6B","pid":1180} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"27c060b02133783737d505ea2610b3d179ebd892\",\n git_commit_timestamp: \"2024-03-06T20:36:21.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: false,\n opt_level: 3,\n}","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:02:16.421491998Z","hostname":"w-01HRAS08V1Y203C0H5ESGPJW6B","pid":1180} Crucible runtime is spawned {"msg":"Upstairs <-> Downstairs Message Version: 5","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:02:16.421517797Z","hostname":"w-01HRAS08V1Y203C0H5ESGPJW6B","pid":1180} {"msg":"Crucible 471f7779-16bd-42a6-8bfc-4673544d5108 has session id: dc8bf16e-64ea-429f-ade3-9550007f0cca","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:02:16.421536886Z","hostname":"w-01HRAS08V1Y203C0H5ESGPJW6B","pid":1180,"session_id":"dc8bf16e-64ea-429f-ade3-9550007f0cca"} {"msg":"Upstairs opts: Upstairs UUID: 471f7779-16bd-42a6-8bfc-4673544d5108, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:02:16.421551295Z","hostname":"w-01HRAS08V1Y203C0H5ESGPJW6B","pid":1180,"session_id":"dc8bf16e-64ea-429f-ade3-9550007f0cca"} {"msg":"Crucible stats registered with UUID: 471f7779-16bd-42a6-8bfc-4673544d5108","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:02:16.421561654Z","hostname":"w-01HRAS08V1Y203C0H5ESGPJW6B","pid":1180,"session_id":"dc8bf16e-64ea-429f-ade3-9550007f0cca"} {"msg":"The guest has requested activation with gen:1","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:02:16.421570934Z","hostname":"w-01HRAS08V1Y203C0H5ESGPJW6B","pid":1180} {"msg":"471f7779-16bd-42a6-8bfc-4673544d5108 active request set","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:02:16.421578784Z","hostname":"w-01HRAS08V1Y203C0H5ESGPJW6B","pid":1180,"session_id":"dc8bf16e-64ea-429f-ade3-9550007f0cca"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:02:16.421586523Z","hostname":"w-01HRAS08V1Y203C0H5ESGPJW6B","pid":1180,"client":"0","":"downstairs","session_id":"dc8bf16e-64ea-429f-ade3-9550007f0cca"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:02:16.421598303Z","hostname":"w-01HRAS08V1Y203C0H5ESGPJW6B","pid":1180,"client":"0","":"downstairs","session_id":"dc8bf16e-64ea-429f-ade3-9550007f0cca"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:02:16.421606282Z","hostname":"w-01HRAS08V1Y203C0H5ESGPJW6B","pid":1180,"client":"1","":"downstairs","session_id":"dc8bf16e-64ea-429f-ade3-9550007f0cca"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:02:16.421621941Z","hostname":"w-01HRAS08V1Y203C0H5ESGPJW6B","pid":1180,"client":"1","":"downstairs","session_id":"dc8bf16e-64ea-429f-ade3-9550007f0cca"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:02:16.421628821Z","hostname":"w-01HRAS08V1Y203C0H5ESGPJW6B","pid":1180,"client":"2","":"downstairs","session_id":"dc8bf16e-64ea-429f-ade3-9550007f0cca"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:02:16.42163731Z","hostname":"w-01HRAS08V1Y203C0H5ESGPJW6B","pid":1180,"client":"2","":"downstairs","session_id":"dc8bf16e-64ea-429f-ade3-9550007f0cca"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:02:16.4216451Z","hostname":"w-01HRAS08V1Y203C0H5ESGPJW6B","pid":1180,"":"io task","client":"0","":"downstairs","session_id":"dc8bf16e-64ea-429f-ade3-9550007f0cca"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:02:16.421657169Z","hostname":"w-01HRAS08V1Y203C0H5ESGPJW6B","pid":1180,"":"io task","client":"2","":"downstairs","session_id":"dc8bf16e-64ea-429f-ade3-9550007f0cca"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:02:16.421665728Z","hostname":"w-01HRAS08V1Y203C0H5ESGPJW6B","pid":1180,"":"io task","client":"1","":"downstairs","session_id":"dc8bf16e-64ea-429f-ade3-9550007f0cca"} {"msg":"ds_connection connected","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:02:16.421776342Z","hostname":"w-01HRAS08V1Y203C0H5ESGPJW6B","pid":1180,"":"io task","client":"0","":"downstairs","session_id":"dc8bf16e-64ea-429f-ade3-9550007f0cca"} {"msg":"ds_connection connected","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:02:16.421789811Z","hostname":"w-01HRAS08V1Y203C0H5ESGPJW6B","pid":1180,"":"io task","client":"2","":"downstairs","session_id":"dc8bf16e-64ea-429f-ade3-9550007f0cca"} {"msg":"ds_connection connected","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:02:16.421797531Z","hostname":"w-01HRAS08V1Y203C0H5ESGPJW6B","pid":1180,"":"io task","client":"1","":"downstairs","session_id":"dc8bf16e-64ea-429f-ade3-9550007f0cca"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:02:16.422070355Z","hostname":"w-01HRAS08V1Y203C0H5ESGPJW6B","pid":1180,"client":"2","":"downstairs","session_id":"dc8bf16e-64ea-429f-ade3-9550007f0cca"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:02:16.422087254Z","hostname":"w-01HRAS08V1Y203C0H5ESGPJW6B","pid":1180,"client":"2","":"downstairs","session_id":"dc8bf16e-64ea-429f-ade3-9550007f0cca"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:02:16.422092624Z","hostname":"w-01HRAS08V1Y203C0H5ESGPJW6B","pid":1180,"client":"0","":"downstairs","session_id":"dc8bf16e-64ea-429f-ade3-9550007f0cca"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:02:16.422097064Z","hostname":"w-01HRAS08V1Y203C0H5ESGPJW6B","pid":1180,"client":"0","":"downstairs","session_id":"dc8bf16e-64ea-429f-ade3-9550007f0cca"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:02:16.422101923Z","hostname":"w-01HRAS08V1Y203C0H5ESGPJW6B","pid":1180,"client":"1","":"downstairs","session_id":"dc8bf16e-64ea-429f-ade3-9550007f0cca"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:02:16.422106033Z","hostname":"w-01HRAS08V1Y203C0H5ESGPJW6B","pid":1180,"client":"1","":"downstairs","session_id":"dc8bf16e-64ea-429f-ade3-9550007f0cca"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:02:16.42233647Z","hostname":"w-01HRAS08V1Y203C0H5ESGPJW6B","pid":1180,"client":"1","":"downstairs","session_id":"dc8bf16e-64ea-429f-ade3-9550007f0cca"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:02:16.422353079Z","hostname":"w-01HRAS08V1Y203C0H5ESGPJW6B","pid":1180,"client":"0","":"downstairs","session_id":"dc8bf16e-64ea-429f-ade3-9550007f0cca"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:02:16.422358339Z","hostname":"w-01HRAS08V1Y203C0H5ESGPJW6B","pid":1180,"client":"2","":"downstairs","session_id":"dc8bf16e-64ea-429f-ade3-9550007f0cca"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:02:16.427819143Z","hostname":"w-01HRAS08V1Y203C0H5ESGPJW6B","pid":1180,"client":"2","":"downstairs","session_id":"dc8bf16e-64ea-429f-ade3-9550007f0cca"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:02:16.427835832Z","hostname":"w-01HRAS08V1Y203C0H5ESGPJW6B","pid":1180,"client":"2","":"downstairs","session_id":"dc8bf16e-64ea-429f-ade3-9550007f0cca"} {"msg":"Waiting for true more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:02:16.427841052Z","hostname":"w-01HRAS08V1Y203C0H5ESGPJW6B","pid":1180,"session_id":"dc8bf16e-64ea-429f-ade3-9550007f0cca"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:02:16.428194422Z","hostname":"w-01HRAS08V1Y203C0H5ESGPJW6B","pid":1180,"client":"1","":"downstairs","session_id":"dc8bf16e-64ea-429f-ade3-9550007f0cca"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:02:16.428207351Z","hostname":"w-01HRAS08V1Y203C0H5ESGPJW6B","pid":1180,"client":"1","":"downstairs","session_id":"dc8bf16e-64ea-429f-ade3-9550007f0cca"} {"msg":"Waiting for true more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:02:16.42821521Z","hostname":"w-01HRAS08V1Y203C0H5ESGPJW6B","pid":1180,"session_id":"dc8bf16e-64ea-429f-ade3-9550007f0cca"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:02:16.428252328Z","hostname":"w-01HRAS08V1Y203C0H5ESGPJW6B","pid":1180,"client":"0","":"downstairs","session_id":"dc8bf16e-64ea-429f-ade3-9550007f0cca"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:02:16.428270747Z","hostname":"w-01HRAS08V1Y203C0H5ESGPJW6B","pid":1180,"client":"0","":"downstairs","session_id":"dc8bf16e-64ea-429f-ade3-9550007f0cca"} {"msg":"[0]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:02:16.428276217Z","hostname":"w-01HRAS08V1Y203C0H5ESGPJW6B","pid":1180,"":"downstairs","":"downstairs","session_id":"dc8bf16e-64ea-429f-ade3-9550007f0cca"} {"msg":"[0]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:02:16.428280957Z","hostname":"w-01HRAS08V1Y203C0H5ESGPJW6B","pid":1180,"":"downstairs","":"downstairs","session_id":"dc8bf16e-64ea-429f-ade3-9550007f0cca"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:02:16.428285976Z","hostname":"w-01HRAS08V1Y203C0H5ESGPJW6B","pid":1180,"":"downstairs","":"downstairs","session_id":"dc8bf16e-64ea-429f-ade3-9550007f0cca"} {"msg":"[1]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:02:16.428290366Z","hostname":"w-01HRAS08V1Y203C0H5ESGPJW6B","pid":1180,"":"downstairs","":"downstairs","session_id":"dc8bf16e-64ea-429f-ade3-9550007f0cca"} {"msg":"[1]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:02:16.428294526Z","hostname":"w-01HRAS08V1Y203C0H5ESGPJW6B","pid":1180,"":"downstairs","":"downstairs","session_id":"dc8bf16e-64ea-429f-ade3-9550007f0cca"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:02:16.428299415Z","hostname":"w-01HRAS08V1Y203C0H5ESGPJW6B","pid":1180,"":"downstairs","":"downstairs","session_id":"dc8bf16e-64ea-429f-ade3-9550007f0cca"} {"msg":"[2]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:02:16.428303785Z","hostname":"w-01HRAS08V1Y203C0H5ESGPJW6B","pid":1180,"":"downstairs","":"downstairs","session_id":"dc8bf16e-64ea-429f-ade3-9550007f0cca"} {"msg":"[2]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:02:16.428308045Z","hostname":"w-01HRAS08V1Y203C0H5ESGPJW6B","pid":1180,"":"downstairs","":"downstairs","session_id":"dc8bf16e-64ea-429f-ade3-9550007f0cca"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:02:16.428312365Z","hostname":"w-01HRAS08V1Y203C0H5ESGPJW6B","pid":1180,"":"downstairs","":"downstairs","session_id":"dc8bf16e-64ea-429f-ade3-9550007f0cca"} {"msg":"Max found gen is 1","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:02:16.428317394Z","hostname":"w-01HRAS08V1Y203C0H5ESGPJW6B","pid":1180,"":"downstairs","":"downstairs","session_id":"dc8bf16e-64ea-429f-ade3-9550007f0cca"} {"msg":"Generation requested: 1 >= found:1","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:02:16.428322134Z","hostname":"w-01HRAS08V1Y203C0H5ESGPJW6B","pid":1180,"":"downstairs","":"downstairs","session_id":"dc8bf16e-64ea-429f-ade3-9550007f0cca"} {"msg":"Next flush: 1","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:02:16.428326264Z","hostname":"w-01HRAS08V1Y203C0H5ESGPJW6B","pid":1180,"":"downstairs","":"downstairs","session_id":"dc8bf16e-64ea-429f-ade3-9550007f0cca"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:02:16.428331154Z","hostname":"w-01HRAS08V1Y203C0H5ESGPJW6B","pid":1180,"":"downstairs","":"downstairs","session_id":"dc8bf16e-64ea-429f-ade3-9550007f0cca"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":Wait for a query_work_queue command to finish before sending IO "2024-03-06T21:02:16.428335443Z","hostname":"w-01HRAS08V1Y203C0H5ESGPJW6B","pid":1180,"session_id":"dc8bf16e-64ea-429f-ade3-9550007f0cca"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:02:16.428342953Z","hostname":"w-01HRAS08V1Y203C0H5ESGPJW6B","pid":1180,"client":"0","":"downstairs","session_id":"dc8bf16e-64ea-429f-ade3-9550007f0cca"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:02:16.428348442Z","hostname":"w-01HRAS08V1Y203C0H5ESGPJW6B","pid":1180,"client":"1","":"downstairs","session_id":"dc8bf16e-64ea-429f-ade3-9550007f0cca"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:02:16.428352932Z","hostname":"w-01HRAS08V1Y203C0H5ESGPJW6B","pid":1180,"client":"2","":"downstairs","session_id":"dc8bf16e-64ea-429f-ade3-9550007f0cca"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:02:16.428357662Z","hostname":"w-01HRAS08V1Y203C0H5ESGPJW6B","pid":1180,"session_id":"dc8bf16e-64ea-429f-ade3-9550007f0cca"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:02:16.428361372Z","hostname":"w-01HRAS08V1Y203C0H5ESGPJW6B","pid":1180,"session_id":"dc8bf16e-64ea-429f-ade3-9550007f0cca"} {"msg":"471f7779-16bd-42a6-8bfc-4673544d5108 is now active with session: dc8bf16e-64ea-429f-ade3-9550007f0cca","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:02:16.428365012Z","hostname":"w-01HRAS08V1Y203C0H5ESGPJW6B","pid":1180,"session_id":"dc8bf16e-64ea-429f-ade3-9550007f0cca"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:02:16.428369191Z","hostname":"w-01HRAS08V1Y203C0H5ESGPJW6B","pid":1180,"session_id":"dc8bf16e-64ea-429f-ade3-9550007f0cca"} {"msg":"The guest has finished waiting for activation with:1","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:02:16.428373221Z","hostname":"w-01HRAS08V1Y203C0H5ESGPJW6B","pid":1180} Region: es:16384 ec:1280 bs:4096 ts:85899345920 tb:20971520 max_io:256 or 1048576 Perf test TEST SECONDS COUNT DPTH IOPS MEAN P95 P99 MAX ES EC rwrites 9.62 30000 1 3119.56 0.00032 0.00070 0.00139 0.05333 16384 1280 rwrites 9.76 30000 1 3073.08 0.00033 0.00073 0.00143 0.06361 16384 1280 rreads 5.56 30000 1 5393.72 0.00019 0.00023 0.00028 0.01291 16384 1280 rreads 5.50 30000 1 5451.10 0.00018 0.00023 0.00027 0.01310 16384 1280 Perf test completed, stop all downstairs Mar 06 21:02:49.547 INFO accepted connection, remote_addr: 127.0.0.1:47194, local_addr: 127.0.0.1:9998 Main task has work to do, go find it got dsc Shutdown Shutdown Shut it down Mar 06 21:02:49.547 INFO request completed, latency_us: 55, response_code: 204[8810] Disable keep_running , [8810] Got stop action so:false kr:false uri[8820] Disable keep_running :[8830] Disable keep_running [8830] Got stop action so:false kr:false /shutdown, method: POST, req_id: 9882b864-a2a0-4e6c-9148-40fc7a4de810, remote_addr: 127.0.0.1:47194, local_addr: 127.0.0.1:9998 [8820] Got stop action so:false kr:false [8820] Exited with: ExitStatus(unix_wait_status(9)) [8820][1] reports Exit [8830] Exited with: ExitStatus(unix_wait_status(9)) [8830][2] reports Exit [8810] Exited with: ExitStatus(unix_wait_status(9)) [8810][0] reports Exit TEST SECONDS COUNT DPTH IOPS MEAN P95 P99 MAX ES EC rwrites 17.37 30000 1 1726.63 0.00058 0.00144 0.00192 0.08547 16384 640 rwrites 8.50 30000 1 3530.51 0.00028 0.00067 0.00140 0.05179 16384 640 rwrites 9.62 30000 1 3119.56 0.00032 0.00070 0.00139 0.05333 16384 1280 rwrites 9.76 30000 1 3073.08 0.00033 0.00073 0.00143 0.06361 16384 1280 TEST SECONDS COUNT DPTH IOPS MEAN P95 P99 MAX ES EC rreads 5.73 30000 1 5233.20 0.00019 0.00027 0.00031 0.01241 16384 640 rreads 5.69 30000 1 5268.23 0.00019 0.00026 0.00030 0.01241 16384 640 rreads 5.56 30000 1 5393.72 0.00019 0.00023 0.00028 0.01291 16384 1280 rreads 5.50 30000 1 5451.10 0.00018 0.00023 0.00027 0.01310 16384 1280 Perf test finished on March 6, 2024 at 09:02:50 PM UTC