/input/rbuild/work Create region with ES:16384 EC:640 BS:4096 Creating region directory at: /var/tmp/dsc Creating dsc directory at: /tmp/dsc Downstairs region 0 created at /var/tmp/dsc/8810 in 1.6108084 Downstairs region 1 created at /var/tmp/dsc/8820 in 0.76261973 Downstairs region 2 created at /var/tmp/dsc/8830 in 0.74895966 Region set with 3 regions was created Using existing output directory "/tmp/dsc" start ds: 8810 start ds: 8820 start ds: 8830 Starting downstairs at port 8820 start access at:127.0.0.1:9998 Starting downstairs at port 8810 Starting downstairs at port 8830 [8820][1] reports Starting Make output file at "/tmp/dsc/downstairs-8820.txt" Make output file at "/tmp/dsc/downstairs-8830.txt" Make output file at "/tmp/dsc/downstairs-8810.txt" [8810][0] reports Starting [8830][2] reports Starting Downstairs /var/tmp/dsc/8830 port 8830 PID:Some(1170) [8830][2] reports Running Downstairs /var/tmp/dsc/8820 port 8820 PID:Some(1171) [8820][1] reports Running Downstairs /var/tmp/dsc/8810 port 8810 PID:Some(1172) [8810][0] reports Running Jan 10 17:59:10.884 INFO listening, local_addr: 127.0.0.1:9998 Control access at:127.0.0.1:9998 1170 1171 1172 Jan 10 17:59:15.957 INFO accepted connection, remote_addr: 127.0.0.1:44339, local_addr: 127.0.0.1:9998 Main task has work to do, go find it got dsc DisableRestartAll disable restart on all: 3 Jan 10 17:59:15.958 INFO request completed, latency_us[8810] Disable keep_running :[8830] Disable keep_running [8820] Disable keep_running 244, response_code: 204, uri: /disablerestart/all, method: POST, req_id: b9cde78b-010d-4723-8e65-fc61fbf0f41e, remote_addr: 127.0.0.1:44339, local_addr: 127.0.0.1:9998 A new loop begins at January 10, 2024 at 05:59:15 PM UTC /var/tmp/bins/crutest perf -g 1 -t 127.0.0.1:8810 -t 127.0.0.1:8820 -t 127.0.0.1:8830 -c 30000 -q --write-loops 2 --read-loops 2 --perf-out /tmp/perf-ES-16384-EC-640.csv {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-01-10T17:59:15.989141195Z","hostname":"ip-10-150-1-239.us-west-2.compute.internal","pid":1180} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"99c957c6a74e6d7d6180d15bada5890df4de8432\",\n git_commit_timestamp: \"2024-01-10T17:30:23.000000000Z\",\n git_branch: \"renovate/rust-1.x\",\n rustc_semver: \"1.75.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"82e1608dfa6e0b5569232559e3d385fea5a93112\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: false,\n opt_level: 3,\n}","v":0,"name":"crucible","level":30,"time":"2024-01-10T17:59:15.989414666Z","hostname":"ip-10-150-1-239.us-west-2.compute.internal","pid":1180} {"msg":"Upstairs <-> Downstairs Message Version: 5","v":0,"name":"crucible","level":30,"time":"2024-01-10T17:59:15.989424801Z","hostname":"ip-10-150-1-239.us-west-2.compute.internal","pid":1180} {"msg":"Crucible 8d79c52f-a574-4c85-b1c3-74a29498ce4b has session id: e65e80a2-7e9c-44cc-969b-c3b5d1ddec78","v":0,"name":"crucible","level":30,"time":"2024-01-10T17:59:15.989431907Z","hostname":"ip-10-150-1-239.us-west-2.compute.internal","pid":1180,"session_id":"e65e80a2-7e9c-44cc-969b-c3b5d1ddec78"} {"msg":"Upstairs opts: Upstairs UUID: 8d79c52f-a574-4c85-b1c3-74a29498ce4b, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-01-10T17:59:15.989443046Z","hostname":"ip-10-150-1-239.us-west-2.compute.internal","pid":1180,"session_id":"e65e80a2-7e9c-44cc-969b-c3b5d1ddec78"} {"msg":"Crucible stats registered with UUID: 8d79c52f-a574-4c85-b1c3-74a29498ce4b","v":0,"name":"crucible","level":30,"time":"2024-01-10T17:59:15.989451289Z","hostname":"ip-10-150-1-239.us-west-2.compute.internal","pid":1180,"session_id":"e65e80a2-7e9c-44cc-969b-c3b5d1ddec78"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:1","v":0,"name":"crucible","level":30,"time":"2024-01-10T17:59:15.989830583Z","hostname":"ip-10-150-1-239.us-west-2.compute.internal","pid":1180} {"msg":"8d79c52f-a574-4c85-b1c3-74a29498ce4b active request set","v":0,"name":"crucible","level":30,"time":"2024-01-10T17:59:15.990077509Z","hostname":"ip-10-150-1-239.us-west-2.compute.internal","pid":1180,"session_id":"e65e80a2-7e9c-44cc-969b-c3b5d1ddec78"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-01-10T17:59:15.990111015Z","hostname":"ip-10-150-1-239.us-west-2.compute.internal","pid":1180,"client":"0","":"downstairs","session_id":"e65e80a2-7e9c-44cc-969b-c3b5d1ddec78"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-01-10T17:59:15.990123959Z","hostname":"ip-10-150-1-239.us-west-2.compute.internal","pid":1180,"client":"0","":"downstairs","session_id":"e65e80a2-7e9c-44cc-969b-c3b5d1ddec78"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-01-10T17:59:15.990169266Z","hostname":"ip-10-150-1-239.us-west-2.compute.internal","pid":1180,"client":"1","":"downstairs","session_id":"e65e80a2-7e9c-44cc-969b-c3b5d1ddec78"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-01-10T17:59:15.990182936Z","hostname":"ip-10-150-1-239.us-west-2.compute.internal","pid":1180,"client":"1","":"downstairs","session_id":"e65e80a2-7e9c-44cc-969b-c3b5d1ddec78"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-01-10T17:59:15.990219869Z","hostname":"ip-10-150-1-239.us-west-2.compute.internal","pid":1180,"client":"2","":"downstairs","session_id":"e65e80a2-7e9c-44cc-969b-c3b5d1ddec78"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-01-10T17:59:15.990230709Z","hostname":"ip-10-150-1-239.us-west-2.compute.internal","pid":1180,"client":"2","":"downstairs","session_id":"e65e80a2-7e9c-44cc-969b-c3b5d1ddec78"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-01-10T17:59:15.99028891Z","hostname":"ip-10-150-1-239.us-west-2.compute.internal","pid":1180,"":"io task","client":"2","":"downstairs","session_id":"e65e80a2-7e9c-44cc-969b-c3b5d1ddec78"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-01-10T17:59:15.990329103Z","hostname":"ip-10-150-1-239.us-west-2.compute.internal","pid":1180,"":"io task","client":"1","":"downstairs","session_id":"e65e80a2-7e9c-44cc-969b-c3b5d1ddec78"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-10T17:59:15.99033814Z","hostname":"ip-10-150-1-239.us-west-2.compute.internal","pid":1180,"":"io task","client":"0","":"downstairs","session_id":"e65e80a2-7e9c-44cc-969b-c3b5d1ddec78"} {"msg":"ds_connection connected","v":0,"name":"crucible","level":30,"time":"2024-01-10T17:59:15.9905315Z","hostname":"ip-10-150-1-239.us-west-2.compute.internal","pid":1180,"":"io task","client":"0","":"downstairs","session_id":"e65e80a2-7e9c-44cc-969b-c3b5d1ddec78"} {"msg":"ds_connection connected","v":0,"name":"crucible","level":30,"time":"2024-01-10T17:59:15.990600097Z","hostname":"ip-10-150-1-239.us-west-2.compute.internal","pid":1180,"":"io task","client":"1","":"downstairs","session_id":"e65e80a2-7e9c-44cc-969b-c3b5d1ddec78"} {"msg":"ds_connection connected","v":0,"name":"crucible","level":30,"time":"2024-01-10T17:59:15.99061518Z","hostname":"ip-10-150-1-239.us-west-2.compute.internal","pid":1180,"":"io task","client":"2","":"downstairs","session_id":"e65e80a2-7e9c-44cc-969b-c3b5d1ddec78"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-01-10T17:59:15.991903874Z","hostname":"ip-10-150-1-239.us-west-2.compute.internal","pid":1180,"client":"0","":"downstairs","session_id":"e65e80a2-7e9c-44cc-969b-c3b5d1ddec78"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-01-10T17:59:15.991951243Z","hostname":"ip-10-150-1-239.us-west-2.compute.internal","pid":1180,"client":"0","":"downstairs","session_id":"e65e80a2-7e9c-44cc-969b-c3b5d1ddec78"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-01-10T17:59:15.991964787Z","hostname":"ip-10-150-1-239.us-west-2.compute.internal","pid":1180,"client":"1","":"downstairs","session_id":"e65e80a2-7e9c-44cc-969b-c3b5d1ddec78"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-01-10T17:59:15.991977641Z","hostname":"ip-10-150-1-239.us-west-2.compute.internal","pid":1180,"client":"1","":"downstairs","session_id":"e65e80a2-7e9c-44cc-969b-c3b5d1ddec78"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-01-10T17:59:15.992018665Z","hostname":"ip-10-150-1-239.us-west-2.compute.internal","pid":1180,"client":"2","":"downstairs","session_id":"e65e80a2-7e9c-44cc-969b-c3b5d1ddec78"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-01-10T17:59:15.992032212Z","hostname":"ip-10-150-1-239.us-west-2.compute.internal","pid":1180,"client":"2","":"downstairs","session_id":"e65e80a2-7e9c-44cc-969b-c3b5d1ddec78"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-01-10T17:59:15.992497455Z","hostname":"ip-10-150-1-239.us-west-2.compute.internal","pid":1180,"client":"0","":"downstairs","session_id":"e65e80a2-7e9c-44cc-969b-c3b5d1ddec78"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-01-10T17:59:15.992807165Z","hostname":"ip-10-150-1-239.us-west-2.compute.internal","pid":1180,"client":"1","":"downstairs","session_id":"e65e80a2-7e9c-44cc-969b-c3b5d1ddec78"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-01-10T17:59:15.992838368Z","hostname":"ip-10-150-1-239.us-west-2.compute.internal","pid":1180,"client":"2","":"downstairs","session_id":"e65e80a2-7e9c-44cc-969b-c3b5d1ddec78"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-01-10T17:59:15.997457401Z","hostname":"ip-10-150-1-239.us-west-2.compute.internal","pid":1180,"client":"0","":"downstairs","session_id":"e65e80a2-7e9c-44cc-969b-c3b5d1ddec78"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-01-10T17:59:15.997510552Z","hostname":"ip-10-150-1-239.us-west-2.compute.internal","pid":1180,"client":"0","":"downstairs","session_id":"e65e80a2-7e9c-44cc-969b-c3b5d1ddec78"} {"msg":"Waiting for true more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-01-10T17:59:15.997525396Z","hostname":"ip-10-150-1-239.us-west-2.compute.internal","pid":1180,"session_id":"e65e80a2-7e9c-44cc-969b-c3b5d1ddec78"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-01-10T17:59:15.998985789Z","hostname":"ip-10-150-1-239.us-west-2.compute.internal","pid":1180,"client":"1","":"downstairs","session_id":"e65e80a2-7e9c-44cc-969b-c3b5d1ddec78"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-01-10T17:59:15.99902708Z","hostname":"ip-10-150-1-239.us-west-2.compute.internal","pid":1180,"client":"1","":"downstairs","session_id":"e65e80a2-7e9c-44cc-969b-c3b5d1ddec78"} {"msg":"Waiting for true more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-01-10T17:59:15.999037109Z","hostname":"ip-10-150-1-239.us-west-2.compute.internal","pid":1180,"session_id":"e65e80a2-7e9c-44cc-969b-c3b5d1ddec78"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-01-10T17:59:15.999240236Z","hostname":"ip-10-150-1-239.us-west-2.compute.internal","pid":1180,"client":"2","":"downstairs","session_id":"e65e80a2-7e9c-44cc-969b-c3b5d1ddec78"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-01-10T17:59:15.999272977Z","hostname":"ip-10-150-1-239.us-west-2.compute.internal","pid":1180,"client":"2","":"downstairs","session_id":"e65e80a2-7e9c-44cc-969b-c3b5d1ddec78"} {"msg":"[0]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-01-10T17:59:15.999281571Z","hostname":"ip-10-150-1-239.us-west-2.compute.internal","pid":1180,"":"downstairs","":"downstairs","session_id":"e65e80a2-7e9c-44cc-969b-c3b5d1ddec78"} {"msg":"[0]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-01-10T17:59:15.99928944Z","hostname":"ip-10-150-1-239.us-west-2.compute.internal","pid":1180,"":"downstairs","":"downstairs","session_id":"e65e80a2-7e9c-44cc-969b-c3b5d1ddec78"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-01-10T17:59:15.999297184Z","hostname":"ip-10-150-1-239.us-west-2.compute.internal","pid":1180,"":"downstairs","":"downstairs","session_id":"e65e80a2-7e9c-44cc-969b-c3b5d1ddec78"} {"msg":"[1]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-01-10T17:59:15.999306326Z","hostname":"ip-10-150-1-239.us-west-2.compute.internal","pid":1180,"":"downstairs","":"downstairs","session_id":"e65e80a2-7e9c-44cc-969b-c3b5d1ddec78"} {"msg":"[1]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-01-10T17:59:15.999313538Z","hostname":"ip-10-150-1-239.us-west-2.compute.internal","pid":1180,"":"downstairs","":"downstairs","session_id":"e65e80a2-7e9c-44cc-969b-c3b5d1ddec78"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-01-10T17:59:15.999320743Z","hostname":"ip-10-150-1-239.us-west-2.compute.internal","pid":1180,"":"downstairs","":"downstairs","session_id":"e65e80a2-7e9c-44cc-969b-c3b5d1ddec78"} {"msg":"[2]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-01-10T17:59:15.999328178Z","hostname":"ip-10-150-1-239.us-west-2.compute.internal","pid":1180,"":"downstairs","":"downstairs","session_id":"e65e80a2-7e9c-44cc-969b-c3b5d1ddec78"} {"msg":"[2]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-01-10T17:59:15.999335234Z","hostname":"ip-10-150-1-239.us-west-2.compute.internal","pid":1180,"":"downstairs","":"downstairs","session_id":"e65e80a2-7e9c-44cc-969b-c3b5d1ddec78"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-01-10T17:59:15.999342421Z","hostname":"ip-10-150-1-239.us-west-2.compute.internal","pid":1180,"":"downstairs","":"downstairs","session_id":"e65e80a2-7e9c-44cc-969b-c3b5d1ddec78"} {"msg":"Max found gen is 1","v":0,"name":"crucible","level":30,"time":"2024-01-10T17:59:15.999350077Z","hostname":"ip-10-150-1-239.us-west-2.compute.internal","pid":1180,"":"downstairs","":"downstairs","session_id":"e65e80a2-7e9c-44cc-969b-c3b5d1ddec78"} {"msg":"Generation requested: 1 >= found:1","v":0,"name":"crucible","level":30,"time":"2024-01-10T17:59:15.999379798Z","hostname":"ip-10-150-1-239.us-west-2.compute.internal","pid":1180,"":"downstairs","":"downstairs","session_id":"e65e80a2-7e9c-44cc-969b-c3b5d1ddec78"} {"msg":"Next flush: 1","v":0,"name":"crucible","level":30,"time":"2024-01-10T17:59:15.999391567Z","hostname":"ip-10-150-1-239.us-west-2.compute.internal","pid":1180,"":"downstairs","":"downstairs","session_id":"e65e80a2-7e9c-44cc-969b-c3b5d1ddec78"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-01-10T17:59:15.999431174Z","hostname":"ip-10-150-1-239.us-west-2.compute.internal","pid":1180,"":"downstairs","":"downstairs","session_id":"e65e80a2-7e9c-44cc-969b-c3b5d1ddec78"} {"msg":"No downstairs repair required","v":0,"name":"crucible","level":30,"time":"2024-01-10T17:59:15.999441898Z","hostname":"ip-10-150-1-239.us-west-2.compute.internal","pid":1180,"session_id":"e65e80a2-7e9c-44cc-969b-c3b5d1ddec78"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-01-10T17:59:15.999448196Z","hostname":"ip-10-150-1-239.us-west-2.compute.internal","pid":1180,"client":"0","":"downstairs","session_id":"e65e80a2-7e9c-44cc-969b-c3b5d1ddec78"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-01-10T17:59:15.999455703Z","hostname":"ip-10-150-1-239.us-west-2.compute.internal","pid":1180,"client":"1","":"downstairs","session_id":"e65e80a2-7e9c-44cc-969b-c3b5d1ddec78"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-01-10T17:59:15.999468725Z","hostname":"ip-10-150-1-239.us-west-2.compute.internal","pid":1180,"client":"2","":"downstairs","session_id":"e65e80a2-7e9c-44cc-969b-c3b5d1ddec78"} {"msg":"All required repair work is completed","v":0,"name":"crucible","level":30,"time":"2024-01-10T17:59:15.999481258Z","hostname":"ip-10-150-1-239.us-west-2.compute.internal","pid":1180,"session_id":"e65e80a2-7e9c-44cc-969b-c3b5d1ddec78"} {"msg":"Set Downstairs and Upstairs active after repairs","v":0,"name":"crucible","level":30,"time":"2024-01-10T17:59:15.999542498Z","hostname":"ip-10-150-1-239.us-west-2.compute.internal","pid":1180,"session_id":"e65e80a2-7e9c-44cc-969b-c3b5d1ddec78"} {"msg":"8d79c52f-a574-4c85-b1c3-74a29498ce4b is now active with session: e65e80a2-7e9c-44cc-969b-c3b5d1ddec78","v":0,"name":"crucible","level":30,"time":"2024-01-10T17:59:15.999594305Z","hostname":"ip-10-150-1-239.us-west-2.compute.internal","pid":1180,"session_id":"e65e80a2-7e9c-44cc-969b-c3b5d1ddec78"} {"msg":"Set Active after no repair","v":0,"name":"crucible","level":30,"time":"2024-01-10T17:59:15.99964422Z","hostname":"ip-10-150-1-239.us-west-2.compute.internal","pid":1180,"session_id":"e65e80a2-7e9c-44cc-969b-c3b5d1ddec78"} {"msg":"The guest has finished waiting for activation with:1","v":0,"name":"crucible","level":30,"time":"2024-01-10T17:59:15.99970829Z","hostname":"ip-10-150-1-239.us-west-2.compute.internal","pid":1180} Wait for a query_work_queue command to finish before sending IO Region: es:16384 ec:640 bs:4096 ts:42949672960 tb:10485760 max_io:256 or 1048576 Perf test TEST SECONDS COUNT DPTH IOPS MEAN P95 P99 MAX ES EC rwrites 19.10 30000 1 1571.00 0.00064 0.00164 0.00267 0.06572 16384 640 rwrites 15.48 30000 1 1937.79 0.00052 0.00119 0.00248 0.08789 16384 640 rreads 14.37 30000 1 2087.11 0.00048 0.00061 0.00075 0.01378 16384 640 rreads 14.26 30000 1 2104.22 0.00048 0.00061 0.00074 0.01378 16384 640 Perf test completed, stop all downstairs Jan 10 18:00:21.679 INFO accepted connection, remote_addr: 127.0.0.1:61796, local_addr: 127.0.0.1:9998 Main task has work to do, go find it got dsc Shutdown Shutdown Jan 10 18:00:21.679 INFO request completed, latency_us: 69, response_code: 204[8810] Disable keep_running , [8820] Disable keep_running uri[8820] Got stop action so:false kr:false : /shutdown, method: POST, req_id:[8830] Disable keep_running [8830] Got stop action so:false kr:false 78e85bc8-5dd2-4654-8829-7d578a95c2e0, remote_addr: 127.0.0.1:61796, local_addr: 127.0.0.1:9998 [8810] Got stop action so:false kr:false Shut it down [8820] Exited with: ExitStatus(unix_wait_status(9)) [8820][1] reports Exit [8830] Exited with: ExitStatus(unix_wait_status(9)) [8830][2] reports Exit [8810] Exited with: ExitStatus(unix_wait_status(9)) [8810][0] reports Exit Create region with ES:16384 EC:1280 BS:4096 Removing existing dsc directory "/tmp/dsc" Removing existing region "/var/tmp/dsc" Creating region directory at: /var/tmp/dsc Creating dsc directory at: /tmp/dsc Downstairs region 0 created at /var/tmp/dsc/8810 in 1.566569 Downstairs region 1 created at /var/tmp/dsc/8820 in 1.5330013 Downstairs region 2 created at /var/tmp/dsc/8830 in 1.6307387 Region set with 3 regions was created Using existing output directory "/tmp/dsc" start ds: 8810 start ds: 8820 start ds: 8830 start access at:127.0.0.1:9998 Starting downstairs at port 8810 Make output file at "/tmp/dsc/downstairs-8810.txt" Starting downstairs at port 8820 Make output file at "/tmp/dsc/downstairs-8820.txt" Starting downstairs at port 8830 Make output file at "/tmp/dsc/downstairs-8830.txt" [8810][0] reports Starting [8820][1] reports Starting [8830][2] reports Starting Downstairs /var/tmp/dsc/8820 port 8820 PID:Some(1189) [8820][1] reports Running Downstairs /var/tmp/dsc/8810 port 8810 PID:Some(1190) [8810][0] reports Running Jan 10 18:00:27.508 INFO listening, local_addr: 127.0.0.1:9998 Control access at:127.0.0.1:9998 Downstairs /var/tmp/dsc/8830 port 8830 PID:Some(1191) [8830][2] reports Running 1189 1190 1191 Jan 10 18:00:32.582 INFO accepted connection, remote_addr: 127.0.0.1:41543, local_addr: 127.0.0.1:9998 Main task has work to do, go find it got dsc DisableRestartAll disable restart on all: 3 Jan 10 18:00:32.583 INFO request completed, latency_us: 159, response_code: 204, uri: /disablerestart/all, method: POST, req_id: bf13b88c-36b2-48e6-8765-e9bc0d3cd1ae[8810] Disable keep_running , remote_addr: 127.0.0.1:41543, local_addr: 127.0.0[8820] Disable keep_running .[8830] Disable keep_running 1:9998 A new loop begins at January 10, 2024 at 06:00:32 PM UTC /var/tmp/bins/crutest perf -g 1 -t 127.0.0.1:8810 -t 127.0.0.1:8820 -t 127.0.0.1:8830 -c 30000 -q --write-loops 2 --read-loops 2 --perf-out /tmp/perf-ES-16384-EC-1280.csv {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-01-10T18:00:32.609340962Z","hostname":"ip-10-150-1-239.us-west-2.compute.internal","pid":1199} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"99c957c6a74e6d7d6180d15bada5890df4de8432\",\n git_commit_timestamp: \"2024-01-10T17:30:23.000000000Z\",\n git_branch: \"renovate/rust-1.x\",\n rustc_semver: \"1.75.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"82e1608dfa6e0b5569232559e3d385fea5a93112\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: false,\n opt_level: 3,\n}","v":0,"name":"crucible","level":30,"time":"2024-01-10T18:00:32.609615464Z","hostname":"ip-10-150-1-239.us-west-2.compute.internal","pid":1199} {"msg":"Upstairs <-> Downstairs Message Version: 5","v":0,"name":"crucible","level":30,"time":"2024-01-10T18:00:32.609625811Z","hostname":"ip-10-150-1-239.us-west-2.compute.internal","pid":1199} {"msg":"Crucible 89fc5d24-7897-4278-b70f-7ba4bd85d3c6 has session id: 7d3a8767-b0e4-4cec-a03d-525f17aabe4b","v":0,"name":"crucible","level":30,"time":"2024-01-10T18:00:32.609632596Z","hostname":"ip-10-150-1-239.us-west-2.compute.internal","pid":1199,"session_id":"7d3a8767-b0e4-4cec-a03d-525f17aabe4b"} {"msg":"Upstairs opts: Upstairs UUID: 89fc5d24-7897-4278-b70f-7ba4bd85d3c6, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v"Crucible runtime is spawned :0,"name":"crucible","level":30,"time":"2024-01-10T18:00:32.609743295Z","hostname":"ip-10-150-1-239.us-west-2.compute.internal","pid":1199,"session_id":"7d3a8767-b0e4-4cec-a03d-525f17aabe4b"} {"msg":"Crucible stats registered with UUID: 89fc5d24-7897-4278-b70f-7ba4bd85d3c6","v":0,"name":"crucible","level":30,"time":"2024-01-10T18:00:32.609809709Z","hostname":"ip-10-150-1-239.us-west-2.compute.internal","pid":1199,"session_id":"7d3a8767-b0e4-4cec-a03d-525f17aabe4b"} {"msg":"The guest has requested activation with gen:1","v":0,"name":"crucible","level":30,"time":"2024-01-10T18:00:32.609822295Z","hostname":"ip-10-150-1-239.us-west-2.compute.internal","pid":1199} {"msg":"89fc5d24-7897-4278-b70f-7ba4bd85d3c6 active request set","v":0,"name":"crucible","level":30,"time":"2024-01-10T18:00:32.609832375Z","hostname":"ip-10-150-1-239.us-west-2.compute.internal","pid":1199,"session_id":"7d3a8767-b0e4-4cec-a03d-525f17aabe4b"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-01-10T18:00:32.609869901Z","hostname":"ip-10-150-1-239.us-west-2.compute.internal","pid":1199,"client":"0","":"downstairs","session_id":"7d3a8767-b0e4-4cec-a03d-525f17aabe4b"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-01-10T18:00:32.609891927Z","hostname":"ip-10-150-1-239.us-west-2.compute.internal","pid":1199,"client":"0","":"downstairs","session_id":"7d3a8767-b0e4-4cec-a03d-525f17aabe4b"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-01-10T18:00:32.609907196Z","hostname":"ip-10-150-1-239.us-west-2.compute.internal","pid":1199,"client":"1","":"downstairs","session_id":"7d3a8767-b0e4-4cec-a03d-525f17aabe4b"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-01-10T18:00:32.609925519Z","hostname":"ip-10-150-1-239.us-west-2.compute.internal","pid":1199,"client":"1","":"downstairs","session_id":"7d3a8767-b0e4-4cec-a03d-525f17aabe4b"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-01-10T18:00:32.610015721Z","hostname":"ip-10-150-1-239.us-west-2.compute.internal","pid":1199,"client":"2","":"downstairs","session_id":"7d3a8767-b0e4-4cec-a03d-525f17aabe4b"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-01-10T18:00:32.610031507Z","hostname":"ip-10-150-1-239.us-west-2.compute.internal","pid":1199,"client":"2","":"downstairs","session_id":"7d3a8767-b0e4-4cec-a03d-525f17aabe4b"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-10T18:00:32.61004353Z","hostname":"ip-10-150-1-239.us-west-2.compute.internal","pid":1199,"":"io task","client":"0","":"downstairs","session_id":"7d3a8767-b0e4-4cec-a03d-525f17aabe4b"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-01-10T18:00:32.610066512Z","hostname":"ip-10-150-1-239.us-west-2.compute.internal","pid":1199,"":"io task","client":"2","":"downstairs","session_id":"7d3a8767-b0e4-4cec-a03d-525f17aabe4b"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-01-10T18:00:32.610080173Z","hostname":"ip-10-150-1-239.us-west-2.compute.internal","pid":1199,"":"io task","client":"1","":"downstairs","session_id":"7d3a8767-b0e4-4cec-a03d-525f17aabe4b"} {"msg":"ds_connection connected","v":0,"name":"crucible","level":30,"time":"2024-01-10T18:00:32.610093075Z","hostname":"ip-10-150-1-239.us-west-2.compute.internal","pid":1199,"":"io task","client":"0","":"downstairs","session_id":"7d3a8767-b0e4-4cec-a03d-525f17aabe4b"} {"msg":"ds_connection connected","v":0,"name":"crucible","level":30,"time":"2024-01-10T18:00:32.610138115Z","hostname":"ip-10-150-1-239.us-west-2.compute.internal","pid":1199,"":"io task","client":"1","":"downstairs","session_id":"7d3a8767-b0e4-4cec-a03d-525f17aabe4b"} {"msg":"ds_connection connected","v":0,"name":"crucible","level":30,"time":"2024-01-10T18:00:32.610173092Z","hostname":"ip-10-150-1-239.us-west-2.compute.internal","pid":1199,"":"io task","client":"2","":"downstairs","session_id":"7d3a8767-b0e4-4cec-a03d-525f17aabe4b"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-01-10T18:00:32.610608905Z","hostname":"ip-10-150-1-239.us-west-2.compute.internal","pid":1199,"client":"2","":"downstairs","session_id":"7d3a8767-b0e4-4cec-a03d-525f17aabe4b"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-01-10T18:00:32.61064673Z","hostname":"ip-10-150-1-239.us-west-2.compute.internal","pid":1199,"client":"2","":"downstairs","session_id":"7d3a8767-b0e4-4cec-a03d-525f17aabe4b"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-01-10T18:00:32.610659223Z","hostname":"ip-10-150-1-239.us-west-2.compute.internal","pid":1199,"client":"1","":"downstairs","session_id":"7d3a8767-b0e4-4cec-a03d-525f17aabe4b"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-01-10T18:00:32.610666788Z","hostname":"ip-10-150-1-239.us-west-2.compute.internal","pid":1199,"client":"1","":"downstairs","session_id":"7d3a8767-b0e4-4cec-a03d-525f17aabe4b"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-01-10T18:00:32.610832194Z","hostname":"ip-10-150-1-239.us-west-2.compute.internal","pid":1199,"client":"0","":"downstairs","session_id":"7d3a8767-b0e4-4cec-a03d-525f17aabe4b"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-01-10T18:00:32.610848795Z","hostname":"ip-10-150-1-239.us-west-2.compute.internal","pid":1199,"client":"0","":"downstairs","session_id":"7d3a8767-b0e4-4cec-a03d-525f17aabe4b"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-01-10T18:00:32.611379763Z","hostname":"ip-10-150-1-239.us-west-2.compute.internal","pid":1199,"client":"1","":"downstairs","session_id":"7d3a8767-b0e4-4cec-a03d-525f17aabe4b"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-01-10T18:00:32.61148267Z","hostname":"ip-10-150-1-239.us-west-2.compute.internal","pid":1199,"client":"2","":"downstairs","session_id":"7d3a8767-b0e4-4cec-a03d-525f17aabe4b"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-01-10T18:00:32.611564137Z","hostname":"ip-10-150-1-239.us-west-2.compute.internal","pid":1199,"client":"0","":"downstairs","session_id":"7d3a8767-b0e4-4cec-a03d-525f17aabe4b"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-01-10T18:00:32.621181218Z","hostname":"ip-10-150-1-239.us-west-2.compute.internal","pid":1199,"client":"1","":"downstairs","session_id":"7d3a8767-b0e4-4cec-a03d-525f17aabe4b"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-01-10T18:00:32.621216611Z","hostname":"ip-10-150-1-239.us-west-2.compute.internal","pid":1199,"client":"1","":"downstairs","session_id":"7d3a8767-b0e4-4cec-a03d-525f17aabe4b"} {"msg":"Waiting for true more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-01-10T18:00:32.621225114Z","hostname":"ip-10-150-1-239.us-west-2.compute.internal","pid":1199,"session_id":"7d3a8767-b0e4-4cec-a03d-525f17aabe4b"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-01-10T18:00:32.621394079Z","hostname":"ip-10-150-1-239.us-west-2.compute.internal","pid":1199,"client":"0","":"downstairs","session_id":"7d3a8767-b0e4-4cec-a03d-525f17aabe4b"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-01-10T18:00:32.621406592Z","hostname":"ip-10-150-1-239.us-west-2.compute.internal","pid":1199,"client":"0","":"downstairs","session_id":"7d3a8767-b0e4-4cec-a03d-525f17aabe4b"} {"msg":"Waiting for true more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-01-10T18:00:32.621413961Z","hostname":"ip-10-150-1-239.us-west-2.compute.internal","pid":1199,"session_id":"7d3a8767-b0e4-4cec-a03d-525f17aabe4b"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-01-10T18:00:32.621741184Z","hostname":"ip-10-150-1-239.us-west-2.compute.internal","pid":1199,"client":"2","":"downstairs","session_id":"7d3a8767-b0e4-4cec-a03d-525f17aabe4b"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-01-10T18:00:32.621791241Z","hostname":"ip-10-150-1-239.us-west-2.compute.internal","pid":1199,"client":"2","":"downstairs","session_id":"7d3a8767-b0e4-4cec-a03d-525f17aabe4b"} {"msg":"[0]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-01-10T18:00:32.621805791Z","hostname":"ip-10-150-1-239.us-west-2.compute.internal","pid":1199,"":"downstairs","":"downstairs","session_id":"7d3a8767-b0e4-4cec-a03d-525f17aabe4b"} {"msg":"[0]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-01-10T18:00:32.621818589Z","hostname":"ip-10-150-1-239.us-west-2.compute.internal","pid":1199,"":"downstairs","":"downstairs","session_id":"7d3a8767-b0e4-4cec-a03d-525f17aabe4b"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-01-10T18:00:32.621830865Z","hostname":"ip-10-150-1-239.us-west-2.compute.internal","pid":1199,"":"downstairs","":"downstairs","session_id":"7d3a8767-b0e4-4cec-a03d-525f17aabe4b"} {"msg":"[1]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-01-10T18:00:32.621843785Z","hostname":"ip-10-150-1-239.us-west-2.compute.internal","pid":1199,"":"downstairs","":"downstairs","session_id":"7d3a8767-b0e4-4cec-a03d-525f17aabe4b"} {"msg":"[1]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-01-10T18:00:32.621889871Z","hostname":"ip-10-150-1-239.us-west-2.compute.internal","pid":1199,"":"downstairs","":"downstairs","session_id":"7d3a8767-b0e4-4cec-a03d-525f17aabe4b"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-01-10T18:00:32.621904614Z","hostname":"ip-10-150-1-239.us-west-2.compute.internal","pid":1199,"":"downstairs","":"downstairs","session_id":"7d3a8767-b0e4-4cec-a03d-525f17aabe4b"} {"msg":"[2]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-01-10T18:00:32.621959762Z","hostname":"ip-10-150-1-239.us-west-2.compute.internal","pid":1199,"":"downstairs","":"downstairs","session_id":"7d3a8767-b0e4-4cec-a03d-525f17aabe4b"} {"msg":"[2]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-01-10T18:00:32.621968371Z","hostname":"ip-10-150-1-239.us-west-2.compute.internal","pid":1199,"":"downstairs","":"downstairs","session_id":"7d3a8767-b0e4-4cec-a03d-525f17aabe4b"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-01-10T18:00:32.621975536Z","hostname":"ip-10-150-1-239.us-west-2.compute.internal","pid":1199,"":"downstairs","":"downstairs","session_id":"7d3a8767-b0e4-4cec-a03d-525f17aabe4b"} {"msg":"Max found gen is 1","v":0,"name":"crucible","level":30,"time":"2024-01-10T18:00:32.62198286Z","hostname":"ip-10-150-1-239.us-west-2.compute.internal","pid":1199,"":"downstairs","":"downstairs","session_id":"7d3a8767-b0e4-4cec-a03d-525f17aabe4b"} {"msg":"Generation requested: 1 >= found:1","v":0,"name":"crucible","level":30,"time":"2024-01-10T18:00:32.621990217Z","hostname":"ip-10-150-1-239.us-west-2.compute.internal","pid":1199,"":"downstairs","":"downstairs","session_id":"7d3a8767-b0e4-4cec-a03d-525f17aabe4b"} {"msg":"Next flush: 1","v":0,"name":"crucible","level":30,"time":"2024-01-10T18:00:32.621997103Z","hostname":"ip-10-150-1-239.us-west-2.compute.internal","pid":1199,"":"downstairs","":"downstairs","session_id":"7d3a8767-b0e4-4cec-a03d-525f17aabe4b"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-01-10T18:00:32.622004284Z","hostname":"ip-10-150-1-239.us-west-2.compute.internal","pid":1199,"":"downstairs","":"downstairs","session_id":"7d3a8767-b0e4-4cec-a03d-525f17aabe4b"} {"msg":"No downstairs repair required","v":0,"name":"crucible","level":30,"time":"2024-01-10T18:00:32.622011298Z","hostname":"ip-10-150-1-239.us-west-2.compute.internal","pid":1199,"session_id":"7d3a8767-b0e4-4cec-a03d-525f17aabe4b"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-01-10T18:00:32.622017577Z","hostname":"ip-10-150-1-239.us-west-2.compute.internal","pid":1199,"client":"0","":"downstairs","session_id":"7d3a8767-b0e4-4cec-a03d-525f17aabe4b"} {"Wait for a query_work_queue command to finish before sending IO msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-01-10T18:00:32.622030283Z","hostname":"ip-10-150-1-239.us-west-2.compute.internal","pid":1199,"client":"1","":"downstairs","session_id":"7d3a8767-b0e4-4cec-a03d-525f17aabe4b"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-01-10T18:00:32.622037482Z","hostname":"ip-10-150-1-239.us-west-2.compute.internal","pid":1199,"client":"2","":"downstairs","session_id":"7d3a8767-b0e4-4cec-a03d-525f17aabe4b"} {"msg":"All required repair work is completed","v":0,"name":"crucible","level":30,"time":"2024-01-10T18:00:32.622044366Z","hostname":"ip-10-150-1-239.us-west-2.compute.internal","pid":1199,"session_id":"7d3a8767-b0e4-4cec-a03d-525f17aabe4b"} {"msg":"Set Downstairs and Upstairs active after repairs","v":0,"name":"crucible","level":30,"time":"2024-01-10T18:00:32.622050993Z","hostname":"ip-10-150-1-239.us-west-2.compute.internal","pid":1199,"session_id":"7d3a8767-b0e4-4cec-a03d-525f17aabe4b"} {"msg":"89fc5d24-7897-4278-b70f-7ba4bd85d3c6 is now active with session: 7d3a8767-b0e4-4cec-a03d-525f17aabe4b","v":0,"name":"crucible","level":30,"time":"2024-01-10T18:00:32.622061581Z","hostname":"ip-10-150-1-239.us-west-2.compute.internal","pid":1199,"session_id":"7d3a8767-b0e4-4cec-a03d-525f17aabe4b"} {"msg":"Set Active after no repair","v":0,"name":"crucible","level":30,"time":"2024-01-10T18:00:32.622073383Z","hostname":"ip-10-150-1-239.us-west-2.compute.internal","pid":1199,"session_id":"7d3a8767-b0e4-4cec-a03d-525f17aabe4b"} {"msg":"The guest has finished waiting for activation with:1","v":0,"name":"crucible","level":30,"time":"2024-01-10T18:00:32.622084483Z","hostname":"ip-10-150-1-239.us-west-2.compute.internal","pid":1199} Region: es:16384 ec:1280 bs:4096 ts:85899345920 tb:20971520 max_io:256 or 1048576 Perf test TEST SECONDS COUNT DPTH IOPS MEAN P95 P99 MAX ES EC rwrites 17.82 30000 1 1683.82 0.00059 0.00144 0.00254 0.05003 16384 1280 rwrites 17.77 30000 1 1688.16 0.00059 0.00145 0.00257 0.06338 16384 1280 rreads 14.27 30000 1 2102.71 0.00048 0.00060 0.00073 0.01351 16384 1280 rreads 14.13 30000 1 2123.77 0.00047 0.00060 0.00072 0.01377 16384 1280 Perf test completed, stop all downstairs Jan 10 18:01:44.000 INFO accepted connection, remote_addr: 127.0.0.1:37908, local_addr: 127.0.0.1:9998 Main task has work to do, go find it got dsc Shutdown Shutdown Jan 10 18:01:44.000 INFO request completed, latency_us: Shut it down 71, response_code: 204, uri: /shutdown, method: POST, req_id: cb21362c-377b-4324-bc32-00e1059540f6, remote_addr: 127.0.0[8820] Disable keep_running .1:37908, local_addr: [8830] Disable keep_running 127[8830] Got stop action so:false kr:false .0.0.1:9998 [8820] Got stop action so:false kr:false [8810] Disable keep_running [8810] Got stop action so:false kr:false [8830] Exited with: ExitStatus(unix_wait_status(9)) [8830][2] reports Exit [8820] Exited with: ExitStatus(unix_wait_status(9)) [8820][1] reports Exit [8810] Exited with: ExitStatus(unix_wait_status(9)) [8810][0] reports Exit TEST SECONDS COUNT DPTH IOPS MEAN P95 P99 MAX ES EC rwrites 19.10 30000 1 1571.00 0.00064 0.00164 0.00267 0.06572 16384 640 rwrites 15.48 30000 1 1937.79 0.00052 0.00119 0.00248 0.08789 16384 640 rwrites 17.82 30000 1 1683.82 0.00059 0.00144 0.00254 0.05003 16384 1280 rwrites 17.77 30000 1 1688.16 0.00059 0.00145 0.00257 0.06338 16384 1280 TEST SECONDS COUNT DPTH IOPS MEAN P95 P99 MAX ES EC rreads 14.37 30000 1 2087.11 0.00048 0.00061 0.00075 0.01378 16384 640 rreads 14.26 30000 1 2104.22 0.00048 0.00061 0.00074 0.01378 16384 640 rreads 14.27 30000 1 2102.71 0.00048 0.00060 0.00073 0.01351 16384 1280 rreads 14.13 30000 1 2123.77 0.00047 0.00060 0.00072 0.01377 16384 1280 Perf test finished on January 10, 2024 at 06:01:45 PM UTC