Using existing output directory "/tmp/dsc" start ds: 8810 start ds: 8820 start ds: 8830 start access at:127.0.0.1:9998 Starting downstairs at port 8810 Starting downstairs at port 8820 Make output file at "/tmp/dsc/downstairs-8820.txt" Make output file at "/tmp/dsc/downstairs-8810.txt" Starting downstairs at port 8830 [8810][0] reports Starting Make output file at "/tmp/dsc/downstairs-8830.txt" [8820][1] reports Starting [8830][2] reports Starting Downstairs /var/tmp/dsc/8820 port 8820 PID:Some(1475) [8820][1] reports Running Downstairs /var/tmp/dsc/8810 port 8810 PID:Some(1476) [8810][0] reports Running Downstairs /var/tmp/dsc/8830 port 8830 PID:Some(1477) [8830][2] reports Running Sep 22 23:34:39.704 INFO listening, local_addr: 127.0.0.1:9998 Control access at:127.0.0.1:9998 Sep 22 23:34:44.776 INFO accepted connection, remote_addr: 127.0.0.1:63555, local_addr: 127.0.0.1:9998 Main task has work to do, go find it got dsc DisableRestartAll disable restart on all: 3 Sep 22 23:34:44.777[8810] Disable keep_running [8820] Disable keep_running INFO request completed, latency_us: 250, response_code: 204, uri: /disablerestart/all, method: POST, req_id: 4021e260-d32f-4a6b-a1a5-a88d9b675730, remote_addr: 127.0.0.1:[8830] Disable keep_running 63555, local_addr: 127.0.0.1:9998 Sep 22 23:51:17.331 INFO accepted connection, remote_addr: 127.0.0.1:42508, local_addr: 127.0.0.1:9998 Main task has work to do, go find it got dsc Shutdown Shutdown Sep 22 23:51:17.331 INFO request completed, latency_usShut it down :[8810] Disable keep_running 76, response_code: 204, [8810] Got stop action so:false kr:false uri: /shutdown, method[8830] Disable keep_running : POST, req_id: e6ab8629-f8e9-435d-a041-4c8dae544490, [8830] Got stop action so:false kr:false remote_addr[8820] Disable keep_running :[8820] Got stop action so:false kr:false 127.0.0.1:42508, local_addr: 127.0.0.1:9998 [8810] Exited with: ExitStatus(unix_wait_status(9)) [8810][0] reports Exit [8820] Exited with: ExitStatus(unix_wait_status(9)) [8830] Exited with: ExitStatus(unix_wait_status(9)) [8820][1] reports Exit [8830][2] reports Exit 61161Z","hostname":"ip-10-150-1-233.us-west-2.compute.internal","pid":1480} {"msg":"Upstairs <-> Downstairs Message Version: 4","v":0,"name":"crucible","level":30,"time":"2023-09-22T23:34:44.796473813Z","hostname":"ip-10-150-1-233.us-west-2.compute.internal","pid":1480} {"msg":"Crucible stats registered with UUID: a8244bfc-797a-4ad8-b523-e623c9bc817c","v":0,"name":"crucible","level":30,"time":"2023-09-22T23:34:44.796480608Z","hostname":"ip-10-150-1-233.us-west-2.compute.internal","pid":1480} {"msg":"Crucible a8244bfc-797a-4ad8-b523-e623c9bc817c has session id: 311d579a-5037-4b91-bf2b-b51f579c2366","v":0,"name":"crucible","level":30,"time":"2023-09-22T23:34:44.796487178Z","hostname":"ip-10-150-1-233.us-west-2.compute.internal","pid":1480} Crucible runtime is spawned The guest has requested activation with gen:1 {"msg":"[0] connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2023-09-22T23:34:44.796683421Z","hostname":"ip-10-150-1-233.us-west-2.compute.internal","pid":1480,"looper":"0"} {"msg":"[1] connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2023-09-22T23:34:44.796702907Z","hostname":"ip-10-150-1-233.us-west-2.compute.internal","pid":1480,"looper":"1"} {"msg":"[2] connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2023-09-22T23:34:44.796727458Z","hostname":"ip-10-150-1-233.us-west-2.compute.internal","pid":1480,"looper":"2"} {"msg":"up_listen starts","v":0,"name":"crucible","level":30,"time":"2023-09-22T23:34:44.796869406Z","hostname":"ip-10-150-1-233.us-west-2.compute.internal","pid":1480,"task":"up_listen"} {"msg":"Wait for all three downstairs to come online","v":0,"name":"crucible","level":30,"time":"2023-09-22T23:34:44.796927813Z","hostname":"ip-10-150-1-233.us-west-2.compute.internal","pid":1480} {"msg":"[1] a8244bfc-797a-4ad8-b523-e623c9bc817c looper connected","v":0,"name":"crucible","level":30,"time":"2023-09-22T23:34:44.796935158Z","hostname":"ip-10-150-1-233.us-west-2.compute.internal","pid":1480,"looper":"1"} {"msg":"[0] a8244bfc-797a-4ad8-b523-e623c9bc817c looper connected","v":0,"name":"crucible","level":30,"time":"2023-09-22T23:34:44.796947286Z","hostname":"ip-10-150-1-233.us-west-2.compute.internal","pid":1480,"looper":"0"} {"msg":"Flush timeout: 0.5","v":0,"name":"crucible","level":30,"time":"2023-09-22T23:34:44.796954412Z","hostname":"ip-10-150-1-233.us-west-2.compute.internal","pid":1480} {"msg":"[2] a8244bfc-797a-4ad8-b523-e623c9bc817c looper connected","v":0,"name":"crucible","level":30,"time":"2023-09-22T23:34:44.796965293Z","hostname":"ip-10-150-1-233.us-west-2.compute.internal","pid":1480,"looper":"2"} {"msg":"[0] Proc runs for 127.0.0.1:8810 in state New","v":0,"name":"crucible","level":30,"time":"2023-09-22T23:34:44.797096686Z","hostname":"ip-10-150-1-233.us-west-2.compute.internal","pid":1480} {"msg":"a8244bfc-797a-4ad8-b523-e623c9bc817c active request set","v":0,"name":"crucible","level":30,"time":"2023-09-22T23:34:44.797106224Z","hostname":"ip-10-150-1-233.us-west-2.compute.internal","pid":1480} {"msg":"Set desired generation to :1","v":0,"name":"crucible","level":30,"time":"2023-09-22T23:34:44.797209364Z","hostname":"ip-10-150-1-233.us-west-2.compute.internal","pid":1480} {"msg":"[1] Proc runs for 127.0.0.1:8820 in state New","v":0,"name":"crucible","level":30,"time":"2023-09-22T23:34:44.797221555Z","hostname":"ip-10-150-1-233.us-west-2.compute.internal","pid":1480} {"msg":"[2] Proc runs for 127.0.0.1:8830 in state New","v":0,"name":"crucible","level":30,"time":"2023-09-22T23:34:44.797235892Z","hostname":"ip-10-150-1-233.us-west-2.compute.internal","pid":1480} {"msg":"[0] a8244bfc-797a-4ad8-b523-e623c9bc817c (a6449fa7-364a-4994-9672-be07e3b19dc1) New New New ds_transition to WaitActive","v":0,"name":"crucible","level":30,"time":"2023-09-22T23:34:44.797609044Z","hostname":"ip-10-150-1-233.us-west-2.compute.internal","pid":1480} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2023-09-22T23:34:44.797624767Z","hostname":"ip-10-150-1-233.us-west-2.compute.internal","pid":1480} {"msg":"[2] a8244bfc-797a-4ad8-b523-e623c9bc817c (a6449fa7-364a-4994-9672-be07e3b19dc1) WaitActive New New ds_transition to WaitActive","v":0,"name":"crucible","level":30,"time":"2023-09-22T23:34:44.797635547Z","hostname":"ip-10-150-1-233.us-west-2.compute.internal","pid":1480} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2023-09-22T23:34:44.797644499Z","hostname":"ip-10-150-1-233.us-west-2.compute.internal","pid":1480} {"msg":"[0] client is_active_req TRUE, promote! session a6449fa7-364a-4994-9672-be07e3b19dc1","v":0,"name":"crucible","level":30,"time":"2023-09-22T23:34:44.79765093Z","hostname":"ip-10-150-1-233.us-west-2.compute.internal","pid":1480} {"msg":"[2] client is_active_req TRUE, promote! session a6449fa7-364a-4994-9672-be07e3b19dc1","v":0,"name":"crucible","level":30,"time":"2023-09-22T23:34:44.797657108Z","hostname":"ip-10-150-1-233.us-west-2.compute.internal","pid":1480} {"msg":"[1] a8244bfc-797a-4ad8-b523-e623c9bc817c (a6449fa7-364a-4994-9672-be07e3b19dc1) WaitActive New WaitActive ds_transition to WaitActive","v":0,"name":"crucible","level":30,"time":"2023-09-22T23:34:44.797675099Z","hostname":"ip-10-150-1-233.us-west-2.compute.internal","pid":1480} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2023-09-22T23:34:44.79768112Z","hostname":"ip-10-150-1-233.us-west-2.compute.internal","pid":1480} {"msg":"[1] client is_active_req TRUE, promote! session a6449fa7-364a-4994-9672-be07e3b19dc1","v":0,"name":"crucible","level":30,"time":"2023-09-22T23:34:44.797687021Z","hostname":"ip-10-150-1-233.us-west-2.compute.internal","pid":1480} {"msg":"[0] downstairs client at 127.0.0.1:8810 has UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2023-09-22T23:34:44.79873052Z","hostname":"ip-10-150-1-233.us-west-2.compute.internal","pid":1480} {"msg":"[0] Got region def RegionDefinition { block_size: 4096, extent_size: Block { value: 16384, shift: 12 }, extent_count: 1600, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2023-09-22T23:34:44.798745272Z","hostname":"ip-10-150-1-233.us-west-2.compute.internal","pid":1480} {"msg":"a8244bfc-797a-4ad8-b523-e623c9bc817c WaitActive WaitActive WaitActive","v":0,"name":"crucible","level":30,"time":"2023-09-22T23:34:44.798849517Z","hostname":"ip-10-150-1-233.us-west-2.compute.internal","pid":1480} {"msg":"[1] downstairs client at 127.0.0.1:8820 has UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2023-09-22T23:34:44.798868018Z","hostname":"ip-10-150-1-233.us-west-2.compute.internal","pid":1480} {"msg":"[2] downstairs client at 127.0.0.1:8830 has UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2023-09-22T23:34:44.798958299Z","hostname":"ip-10-150-1-233.us-west-2.compute.internal","pid":1480} {"msg":"[2] Got region def RegionDefinition { block_size: 4096, extent_size: Block { value: 16384, shift: 12 }, extent_count: 1600, uuid: 12345678-0000-0000-0000-000000008830, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2023-09-22T23:34:44.79896688Z","hostname":"ip-10-150-1-233.us-west-2.compute.internal","pid":1480} {"msg":"[1] Got region def RegionDefinition { block_size: 4096, extent_size: Block { value: 16384, shift: 12 }, extent_count: 1600, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2023-09-22T23:34:44.798973154Z","hostname":"ip-10-150-1-233.us-west-2.compute.internal","pid":1480} {"msg":"a8244bfc-797a-4ad8-b523-e623c9bc817c WaitActive WaitActive WaitActive","v":0,"name":"crucible","level":30,"time":"2023-09-22T23:34:44.798979064Z","hostname":"ip-10-150-1-233.us-west-2.compute.internal","pid":1480} {"msg":"a8244bfc-797a-4ad8-b523-e623c9bc817c WaitActive WaitActive WaitActive","v":0,"name":"crucible","level":30,"time":"2023-09-22T23:34:44.798984838Z","hostname":"ip-10-150-1-233.us-west-2.compute.internal","pid":1480} {"msg":"[1] a8244bfc-797a-4ad8-b523-e623c9bc817c (a6449fa7-364a-4994-9672-be07e3b19dc1) WaitActive WaitActive WaitActive ds_transition to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2023-09-22T23:34:44.880845451Z","hostname":"ip-10-150-1-233.us-west-2.compute.internal","pid":1480} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2023-09-22T23:34:44.880865923Z","hostname":"ip-10-150-1-233.us-west-2.compute.internal","pid":1480} {"msg":"[1] new RM replaced this: None","v":0,"name":"crucible","level":40,"time":"2023-09-22T23:34:44.880872861Z","hostname":"ip-10-150-1-233.us-west-2.compute.internal","pid":1480} {"msg":"[1] Starts reconcile loop","v":0,"name":"crucible","level":30,"time":"2023-09-22T23:34:44.880882197Z","hostname":"ip-10-150-1-233.us-west-2.compute.internal","pid":1480} {"msg":"[1] 127.0.0.1:8820 task reports connection:true","v":0,"name":"crucible","level":30,"time":"2023-09-22T23:34:44.880968267Z","hostname":"ip-10-150-1-233.us-west-2.compute.internal","pid":1480} {"msg":"a8244bfc-797a-4ad8-b523-e623c9bc817c WaitActive WaitQuorum WaitActive","v":0,"name":"crucible","level":30,"time":"2023-09-22T23:34:44.880976893Z","hostname":"ip-10-150-1-233.us-west-2.compute.internal","pid":1480} {"msg":"Waiting for 2 more clients to be ready","v":0,"name":"crucible","level":30,"time":"2023-09-22T23:34:44.881033187Z","hostname":"ip-10-150-1-233.us-west-2.compute.internal","pid":1480} {"msg":"Set check for repair","v":0,"name":"crucible","level":30,"time":"2023-09-22T23:34:44.881046016Z","hostname":"ip-10-150-1-233.us-west-2.compute.internal","pid":1480} {"msg":"[0] a8244bfc-797a-4ad8-b523-e623c9bc817c (a6449fa7-364a-4994-9672-be07e3b19dc1) WaitActive WaitQuorum WaitActive ds_transition to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2023-09-22T23:34:44.882367072Z","hostname":"ip-10-150-1-233.us-west-2.compute.internal","pid":1480} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2023-09-22T23:34:44.88238709Z","hostname":"ip-10-150-1-233.us-west-2.compute.internal","pid":1480} {"msg":"[0] new RM replaced this: None","v":0,"name":"crucible","level":40,"time":"2023-09-22T23:34:44.882393791Z","hostname":"ip-10-150-1-233.us-west-2.compute.internal","pid":1480} {"msg":"[0] Starts reconcile loop","v":0,"name":"crucible","level":30,"time":"2023-09-22T23:34:44.88239986Z","hostname":"ip-10-150-1-233.us-west-2.compute.internal","pid":1480} {"msg":"[0] 127.0.0.1:8810 task reports connection:true","v":0,"name":"crucible","level":30,"time":"2023-09-22T23:34:44.882405832Z","hostname":"ip-10-150-1-233.us-west-2.compute.internal","pid":1480} {"msg":"a8244bfc-797a-4ad8-b523-e623c9bc817c WaitQuorum WaitQuorum WaitActive","v":0,"name":"crucible","level":30,"time":"2023-09-22T23:34:44.882412002Z","hostname":"ip-10-150-1-233.us-west-2.compute.internal","pid":1480} {"msg":"Waiting for 1 more clients to be ready","v":0,"name":"crucible","level":30,"time":"2023-09-22T23:34:44.882417823Z","hostname":"ip-10-150-1-233.us-west-2.compute.internal","pid":1480} {"msg":"Set check for repair","v":0,"name":"crucible","level":30,"time":"2023-09-22T23:34:44.882423652Z","hostname":"ip-10-150-1-233.us-west-2.compute.internal","pid":1480} {"msg":"[2] a8244bfc-797a-4ad8-b523-e623c9bc817c (a6449fa7-364a-4994-9672-be07e3b19dc1) WaitQuorum WaitQuorum WaitActive ds_transition to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2023-09-22T23:34:44.88517015Z","hostname":"ip-10-150-1-233.us-west-2.compute.internal","pid":1480} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2023-09-22T23:34:44.885185165Z","hostname":"ip-10-150-1-233.us-west-2.compute.internal","pid":1480} {"msg":"[2] new RM replaced this: None","v":0,"name":"crucible","level":40,"time":"2023-09-22T23:34:44.885191877Z","hostname":"ip-10-150-1-233.us-west-2.compute.internal","pid":1480} {"msg":"[2] Starts reconcile loop","v":0,"name":"crucible","level":30,"time":"2023-09-22T23:34:44.885197792Z","hostname":"ip-10-150-1-233.us-west-2.compute.internal","pid":1480} {"msg":"[2] 127.0.0.1:8830 task reports connection:true","v":0,"name":"crucible","level":30,"time":"2023-09-22T23:34:44.885203747Z","hostname":"ip-10-150-1-233.us-west-2.compute.internal","pid":1480} {"msg":"a8244bfc-797a-4ad8-b523-e623c9bc817c WaitQuorum WaitQuorum WaitQuorum","v":0,"name":"crucible","level":30,"time":"2023-09-22T23:34:44.885209851Z","hostname":"ip-10-150-1-233.us-west-2.compute.internal","pid":1480} {"msg":"[0]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2023-09-22T23:34:44.885215841Z","hostname":"ip-10-150-1-233.us-west-2.compute.internal","pid":1480} {"msg":"[0]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2023-09-22T23:34:44.885221779Z","hostname":"ip-10-150-1-233.us-west-2.compute.internal","pid":1480} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2023-09-22T23:34:44.885227904Z","hostname":"ip-10-150-1-233.us-west-2.compute.internal","pid":1480} {"msg":"[1]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2023-09-22T23:34:44.885233867Z","hostname":"ip-10-150-1-233.us-west-2.compute.internal","pid":1480} {"msg":"[1]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2023-09-22T23:34:44.885239648Z","hostname":"ip-10-150-1-233.us-west-2.compute.internal","pid":1480} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2023-09-22T23:34:44.885245474Z","hostname":"ip-10-150-1-233.us-west-2.compute.internal","pid":1480} {"msg":"[2]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2023-09-22T23:34:44.885251487Z","hostname":"ip-10-150-1-233.us-west-2.compute.internal","pid":1480} {"msg":"[2]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2023-09-22T23:34:44.885262378Z","hostname":"ip-10-150-1-233.us-west-2.compute.internal","pid":1480} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2023-09-22T23:34:44.885268314Z","hostname":"ip-10-150-1-233.us-west-2.compute.internal","pid":1480} {"msg":"Max found gen is 1","v":0,"name":"crucible","level":30,"time":"2023-09-22T23:34:44.88527413Z","hostname":"ip-10-150-1-233.us-west-2.compute.internal","pid":1480} {"msg":"Generation requested: 1 >= found:1","v":0,"name":"crucible","level":30,"time":"2023-09-22T23:34:44.885280006Z","hostname":"ip-10-150-1-233.us-west-2.compute.internal","pid":1480} {"msg":"Next flush: 1","v":0,"name":"crucible","level":30,"time":"2023-09-22T23:34:44.885285889Z","hostname":"ip-10-150-1-233.us-west-2.compute.internal","pid":1480} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2023-09-22T23:34:44.885291737Z","hostname":"ip-10-150-1-233.us-west-2.compute.internal","pid":1480} {"msg":"No downstairs repair required","v":0,"name":"crucible","level":30,"time":"2023-09-22T23:34:44.885297508Z","hostname":"ip-10-150-1-233.us-west-2.compute.internal","pid":1480} {"msg":"No initial repair work was required","v":0,"name":"crucible","level":30,"time":"2023-09-22T23:34:44.885303234Z","hostname":"ip-10-150-1-233.us-west-2.compute.internal","pid":1480} {"msg":"Set Downstairs and Upstairs active","v":0,"name":"crucible","level":30,"time":"2023-09-22T23:34:44.885309016Z","hostname":"ip-10-150-1-233.us-west-2.compute.internal","pid":1480} The guest has finished waiting for activation with:1 {"msg":"a8244bfc-797a-4ad8-b523-e623c9bc817c is now active with session: a6449fa7-364a-4994-9672-be07e3b19dc1","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2023-09-22T23:34:44.885366397Z","hostname":"ip-10-150-1-233.us-west-2.compute.internal","pid":1480} {"msg":"a8244bfc-797a-4ad8-b523-e623c9bc817c Set Active after no repair","v":0,"name":"crucible","level":30,"time":"2023-09-22T23:34:44.885376422Z","hostname":"ip-10-150-1-233.us-west-2.compute.internal","pid":1480} {"msg":"Notify all downstairs, region set compare is done.","v":0,"name":"crucible","level":30,"time":"2023-09-22T23:34:44.885382426Z","hostname":"ip-10-150-1-233.us-west-2.compute.internal","pid":1480} {"msg":"Set check for repair","v":0,"name":"crucible","level":30,"time":"2023-09-22T23:34:44.885388476Z","hostname":"ip-10-150-1-233.us-west-2.compute.internal","pid":1480} {"msg":"[2] received reconcile message","v":0,"name":"crucible","level":30,"time":"2023-09-22T23:34:44.885462935Z","hostname":"ip-10-150-1-233.us-west-2.compute.internal","pid":1480} {"msg":"[0] received reconcile message","v":0,"name":"crucible","level":30,"time":"2023-09-22T23:34:44.885470502Z","hostname":"ip-10-150-1-233.us-west-2.compute.internal","pid":1480} {"msg":"[2] All repairs completed, exit","v":0,"name":"crucible","level":30,"time":"2023-09-22T23:34:44.885476429Z","hostname":"ip-10-150-1-233.us-west-2.compute.internal","pid":1480} {"msg":"[2] Starts cmd_loop","v":0,"name":"crucible","level":30,"time":"2023-09-22T23:34:44.885486351Z","hostname":"ip-10-150-1-233.us-west-2.compute.internal","pid":1480} {"msg":"[1] received reconcile message","v":0,"name":"crucible","level":30,"time":"2023-09-22T23:34:44.885549809Z","hostname":"ip-10-150-1-233.us-west-2.compute.internal","pid":1480} {"msg":"[0] All repairs completed, exit","v":0,"name":"crucible","level":30,"time":"2023-09-22T23:34:44.885563098Z","hostname":"ip-10-150-1-233.us-west-2.compute.internal","pid":1480} {"msg":"[1] All repairs completed, exit","v":0,"name":"crucible","level":30,"time":"2023-09-22T23:34:44.885569513Z","hostname":"ip-10-150-1-233.us-west-2.compute.internal","pid":1480} {"msg":"[0] Starts cmd_loop","v":0,"name":"crucible","level":30,"time":"2023-09-22T23:34:44.885579691Z","hostname":"ip-10-150-1-233.us-west-2.compute.internal","pid":1480} {"msg":"[1] Starts cmd_loop","v":0,"name":"crucible","level":30,"time":"2023-09-22T23:34:44.885585753Z","hostname":"ip-10-150-1-233.us-west-2.compute.internal","pid":1480} Region: es:16384 ec:1600 bs:4096 ts:107374182400 tb:26214400 max_io:256 or 1048576 Fill test {"msg":"Checking if live repair is needed","v":0,"name":"crucible","level":30,"time":"2023-09-22T23:34:45.887205797Z","hostname":"ip-10-150-1-233.us-west-2.compute.internal","pid":1480} {"msg":"No Live Repair required at this time","v":0,"name":"crucible","level":30,"time":"2023-09-22T23:34:45.887227988Z","hostname":"ip-10-150-1-233.us-west-2.compute.internal","pid":1480} {"msg":"[1] flow control start ","v":0,"name":"crucible","level":40,"time":"2023-09-22T23:34:57.582063823Z","hostname":"ip-10-150-1-233.us-west-2.compute.internal","pid":1480} {"msg":"[2] flow control start ","v":0,"name":"crucible","level":40,"time":"2023-09-22T23:34:57.597970532Z","hostname":"ip-10-150-1-233.us-west-2.compute.internal","pid":1480} {"msg":"[0] flow control start ","v":0,"name":"crucible","level":40,"time":"2023-09-22T23:34:57.624748014Z","hostname":"ip-10-150-1-233.us-west-2.compute.internal","pid":1480} {"msg":"[1] flow control end ","v":0,"name":"crucible","level":40,"time":"2023-09-22T23:50:51.901677703Z","hostname":"ip-10-150-1-233.us-west-2.compute.internal","pid":1480} {"msg":"[0] flow control end ","v":0,"name":"crucible","level":40,"time":"2023-09-22T23:50:52.002806826Z","hostname":"ip-10-150-1-233.us-west-2.compute.internal","pid":1480} {"msg":"[2] flow control end ","v":0,"name":"crucible","level":40,"time":"2023-09-22T23:50:52.031809816Z","hostname":"ip-10-150-1-233.us-west-2.compute.internal","pid":1480} CLIENT: Tests done. All submitted work has been ACK'd ---------------------------------------------------------------- Crucible gen:1 GIO:true work queues: Upstairs:0 downstairs:206 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY 263629 Acked 264628 Write 100 Done Done Done false 263630 Acked 264629 Write 100 Done Done Done false 263631 Acked 264630 Write 100 Done Done Done false 263632 Acked 264631 Write 100 Done Done Done false 263633 Acked 264632 Write 100 Done Done Done false 263634 Acked 264633 Write 100 Done Done Done false 263635 Acked 264634 Write 100 Done Done Done false 263636 Acked 264635 Write 100 Done Done Done false 263637 Acked 264636 Write 100 Done Done Done false 263638 Acked 264637 Write 100 Done Done Done false 263639 Acked 264638 Write 100 Done Done Done false 263640 Acked 264639 Write 100 Done Done Done false 263641 Acked 264640 Write 100 Done Done Done false 263642 Acked 264641 Write 100 Done Done Done false 263643 Acked 264642 Write 100 Done Done Done false 263644 Acked 264643 Write 100 Done Done Done false 263645 Acked 264644 Write 100 Done Done Done false 263646 Acked 264645 Write 100 Done Done Done false 263647 Acked 264646 Write 100 Done Done Done false 263648 Acked 264647 Write 100 Done Done Done false 263649 Acked 264648 Write 100 Done Done Done false 263650 Acked 264649 Write 100 Done Done Done false 263651 Acked 264650 Write 100 Done Done Done false 263652 Acked 264651 Write 100 Done Done Done false 263653 Acked 264652 Write 100 Done Done Done false 263654 Acked 264653 Write 100 Done Done Done false 263655 Acked 264654 Write 100 Done Done Done false 263656 Acked 264655 Write 100 Done Done Done false 263657 Acked 264656 Write 100 Done Done Done false 263658 Acked 264657 Write 100 Done Done Done false 263659 Acked 264658 Write 100 Done Done Done false 263660 Acked 264659 Write 100 Done Done Done false 263661 Acked 264660 Write 100 Done Done Done false 263662 Acked 264661 Write 100 Done Done Done false 263663 Acked 264662 Write 100 Done Done Done false 263664 Acked 264663 Write 100 Done Done Done false 263665 Acked 264664 Write 100 Done Done Done false 263666 Acked 264665 Write 100 Done Done Done false 263667 Acked 264666 Write 100 Done Done Done false 263668 Acked 264667 Write 100 Done Done Done false 263669 Acked 264668 Write 100 Done Done Done false 263670 Acked 264669 Write 100 Done Done Done false 263671 Acked 264670 Write 100 Done Done Done false 263672 Acked 264671 Write 100 Done Done Done false 263673 Acked 264672 Write 100 Done Done Done false 263674 Acked 264673 Write 100 Done Done Done false 263675 Acked 264674 Write 100 Done Done Done false 263676 Acked 264675 Write 100 Done Done Done false 263677 Acked 264676 Write 100 Done Done Done false 263678 Acked 264677 Write 100 Done Done Done false 263679 Acked 264678 Write 100 Done Done Done false 263680 Acked 264679 Write 100 Done Done Done false 263681 Acked 264680 Write 100 Done Done Done false 263682 Acked 264681 Write 100 Done Done Done false 263683 Acked 264682 Write 100 Done Done Done false 263684 Acked 264683 Write 100 Done Done Done false 263685 Acked 264684 Write 100 Done Done Done false 263686 Acked 264685 Write 100 Done Done Done false 263687 Acked 264686 Write 100 Done Done Done false 263688 Acked 264687 Write 100 Done Done Done false 263689 Acked 264688 Write 100 Done Done Done false 263690 Acked 264689 Write 100 Done Done Done false 263691 Acked 264690 Write 100 Done Done Done false 263692 Acked 264691 Write 100 Done Done Done false 263693 Acked 264692 Write 100 Done Done Done false 263694 Acked 264693 Write 100 Done Done Done false 263695 Acked 264694 Write 100 Done Done Done false 263696 Acked 264695 Write 100 Done Done Done false 263697 Acked 264696 Write 100 Done Done Done false 263698 Acked 264697 Write 100 Done Done Done false 263699 Acked 264698 Write 100 Done Done Done false 263700 Acked 264699 Write 100 Done Done Done false 263701 Acked 264700 Write 100 Done Done Done false 263702 Acked 264701 Write 100 Done Done Done false 263703 Acked 264702 Write 100 Done Done Done false 263704 Acked 264703 Write 100 Done Done Done false 263705 Acked 264704 Write 100 Done Done Done false 263706 Acked 264705 Write 100 Done Done Done false 263707 Acked 264706 Write 100 Done Done Done false 263708 Acked 264707 Write 100 Done Done Done false 263709 Acked 264708 Write 100 Done Done Done false 263710 Acked 264709 Write 100 Done Done Done false 263711 Acked 264710 Write 100 Done Done Done false 263712 Acked 264711 Write 100 Done Done Done false 263713 Acked 264712 Write 100 Done Done Done false 263714 Acked 264713 Write 100 Done Done Done false 263715 Acked 264714 Write 100 Done Done Done false 263716 Acked 264715 Write 100 Done Done Done false 263717 Acked 264716 Write 100 Done Done Done false 263718 Acked 264717 Write 100 Done Done Done false 263719 Acked 264718 Write 100 Done Done Done false 263720 Acked 264719 Write 100 Done Done Done false 263721 Acked 264720 Write 100 Done Done Done false 263722 Acked 264721 Write 100 Done Done Done false 263723 Acked 264722 Write 100 Done Done Done false 263724 Acked 264723 Write 100 Done Done Done false 263725 Acked 264724 Write 100 Done Done Done false 263726 Acked 264725 Write 100 Done Done Done false 263727 Acked 264726 Write 100 Done Done Done false 263728 Acked 264727 Write 100 Done Done Done false 263729 Acked 264728 Write 100 Done Done Done false 263730 Acked 264729 Write 100 Done Done Done false 263731 Acked 264730 Write 100 Done Done Done false 263732 Acked 264731 Write 100 Done Done Done false 263733 Acked 264732 Write 100 Done Done Done false 263734 Acked 264733 Write 100 Done Done Done false 263735 Acked 264734 Write 100 Done Done Done false 263736 Acked 264735 Write 100 Done Done Done false 263737 Acked 264736 Write 100 Done Done Done false 263738 Acked 264737 Write 100 Done Done Done false 263739 Acked 264738 Write 100 Done Done Done false 263740 Acked 264739 Write 100 Done Done Done false 263741 Acked 264740 Write 100 Done Done Done false 263742 Acked 264741 Write 100 Done Done Done false 263743 Acked 264742 Write 100 Done Done Done false 263744 Acked 264743 Write 100 Done Done Done false 263745 Acked 264744 Write 100 Done Done Done false 263746 Acked 264745 Write 100 Done Done Done false 263747 Acked 264746 Write 100 Done Done Done false 263748 Acked 264747 Write 100 Done Done Done false 263749 Acked 264748 Write 100 Done Done Done false 263750 Acked 264749 Write 100 Done Done Done false 263751 Acked 264750 Write 100 Done Done Done false 263752 Acked 264751 Write 100 Done Done Done false 263753 Acked 264752 Write 100 Done Done Done false 263754 Acked 264753 Write 100 Done Done Done false 263755 Acked 264754 Write 100 Done Done Done false 263756 Acked 264755 Write 100 Done Done Done false 263757 Acked 264756 Write 100 Done Done Done false 263758 Acked 264757 Write 100 Done Done Done false 263759 Acked 264758 Write 100 Done Done Done false 263760 Acked 264759 Write 100 Done Done Done false 263761 Acked 264760 Write 100 Done Done Done false 263762 Acked 264761 Write 100 Done Done Done false 263763 Acked 264762 Write 100 Done Done Done false 263764 Acked 264763 Write 100 Done Done Done false 263765 Acked 264764 Write 100 Done Done Done false 263766 Acked 264765 Write 100 Done Done Done false 263767 Acked 264766 Write 100 Done Done Done false 263768 Acked 264767 Write 100 Done Done Done false 263769 Acked 264768 Write 100 Done Done Done false 263770 Acked 264769 Write 100 Done Done Done false 263771 Acked 264770 Write 100 Done Done Done false 263772 Acked 264771 Write 100 Done Done Done false 263773 Acked 264772 Write 100 Done Done Done false 263774 Acked 264773 Write 100 Done Sent Done false 263775 Acked 264774 Write 100 Done Sent Done false 263776 Acked 264775 Write 100 Done Sent Done false 263777 Acked 264776 Flush 0 Done Sent Done false 263778 Acked 264777 Write 100 Done Sent Done false 263779 Acked 264778 Write 100 Done Sent Done false 263780 Acked 264779 Write 100 Done Sent Done false 263781 Acked 264780 Write 100 Done Sent Done false 263782 Acked 264781 Write 100 Done Sent Done false 263783 Acked 264782 Write 100 Done Sent Done false 263784 Acked 264783 Write 100 Done Sent Done false 263785 Acked 264784 Write 100 Done Sent Done false 263786 Acked 264785 Write 100 Done Sent Done false 263787 Acked 264786 Write 100 Done Sent Done false 263788 Acked 264787 Write 100 Done Sent Done false 263789 Acked 264788 Write 100 Done Sent Done false 263790 Acked 264789 Write 100 Done Sent Done false 263791 Acked 264790 Write 100 Done Sent Done false 263792 Acked 264791 Write 100 Done Sent Done false 263793 Acked 264792 Write 100 Done Sent Done false 263794 Acked 264793 Write 100 Done Sent Done false 263795 Acked 264794 Write 100 Done Sent Done false 263796 Acked 264795 Write 100 Done Sent Done false 263797 Acked 264796 Write 100 Done Sent Done false 263798 Acked 264797 Write 100 Done Sent Done false 263799 Acked 264798 Write 100 Done Sent Done false 263800 Acked 264799 Write 100 Done Sent Done false 263801 Acked 264800 Write 100 Done Sent Done false 263802 Acked 264801 Write 100 Done Sent Done false 263803 Acked 264802 Write 100 Done Sent Done false 263804 Acked 264803 Write 100 Done Sent Done false 263805 Acked 264804 Write 100 Done Sent Done false 263806 Acked 264805 Write 100 Done Sent Done false 263807 Acked 264806 Write 100 Done Sent Done false 263808 Acked 264807 Write 100 Done Sent Done false 263809 Acked 264808 Write 100 Done Sent Done false 263810 Acked 264809 Write 100 Done Sent Done false 263811 Acked 264810 Write 100 Done Sent Done false 263812 Acked 264811 Write 100 Done Sent Done false 263813 Acked 264812 Write 100 Done Sent Done false 263814 Acked 264813 Write 100 Done Sent Done false 263815 Acked 264814 Write 100 Done Sent Done false 263816 Acked 264815 Write 100 Done Sent Done false 263817 Acked 264816 Write 100 Done Sent Done false 263818 Acked 264817 Write 100 Done Sent Done false 263819 Acked 264818 Write 100 Done Sent Done false 263820 Acked 264819 Write 100 Done Sent Done false 263821 Acked 264820 Write 100 Done Sent Done false 263822 Acked 264821 Write 100 Done Sent Done false 263823 Acked 264822 Write 100 Done Sent Done false 263824 Acked 264823 Write 100 Done Sent Done false 263825 Acked 264824 Write 100 Done Sent Done false 263826 Acked 264825 Write 100 Done Sent Done false 263827 Acked 264826 Write 100 Done Sent Done false 263828 Acked 264827 Write 100 Done Sent Done false 263829 Acked 264828 Write 100 Done Sent Done false 263830 Acked 264829 Write 100 Done Sent Done false 263831 Acked 264830 Write 100 Done Sent Done false 263832 Acked 264831 Write 100 Done Sent Done false 263833 Acked 264832 Write 100 Done Sent Done false 263834 Acked 264833 Flush 0 Done Sent Done false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 0 61 0 61 Done 206 145 206 557 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 264833 264627 264833 Downstairs last five completed: 264627 264626 264625 264624 264623 264622 Upstairs last five completed: 263834 263777 263628 263486 263331 263185 CLIENT: Up:0 ds:206 act:3 ---------------------------------------------------------------- Crucible gen:1 GIO:true work queues: Upstairs:0 downstairs:0 Downstairs last five completed: 264833 264832 264831 264830 264829 264828 Upstairs last five completed: 263834 263777 263628 263486 263331 263185 CLIENT: Up:0 ds:0 act:3 CLIENT: All crucible jobs finished, exiting program