Max level shown:
1 starting February 15, 2024 at 02:57:24 AM UTC
2 Creating downstairs regions
3 Creating region directory at: /var/tmp/dsc/region
4 Creating dsc directory at: /tmp/dsc
5 Downstairs region 0 created at /var/tmp/dsc/region/8810 in 0.21686967
6 Downstairs region 1 created at /var/tmp/dsc/region/8820 in 0.06259794
7 Downstairs region 2 created at /var/tmp/dsc/region/8830 in 0.06430918
8 Region set with 3 regions was created
9 Starting downstairs
10 Using existing output directory "/tmp/dsc"
11 start ds: 8810
12 start ds: 8820
13 start ds: 8830
14 Starting downstairs at port 8820
15 start access at:127.0.0.1:9998
16 Starting downstairs at port 8810
17 Starting downstairs at port 8830
18 Make output file at "/tmp/dsc/downstairs-8820.txt"
19 Make output file at "/tmp/dsc/downstairs-8810.txt"
20 Make output file at "/tmp/dsc/downstairs-8830.txt"
21 [8820][1] reports Starting
22 [8810][0] reports Starting
23 [8830][2] reports Starting
24 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1157)
25 [8830][2] reports Running
26 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1158)
27 [8810][0] reports Running
28 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1159)
29 [8820][1] reports Running
30 Feb 15 02:57:24.515 INFO listening, local_addr: 127.0.0.1:9998
31 Control access at:127.0.0.1:9998
32 Running initial fill
332024-02-15T02:57:29.554ZINFOcrucible: Upstairs starts
342024-02-15T02:57:29.555ZINFOcrucible: Crucible Version: BuildInfo { version: "0.0.1", git_sha: "56eb785b9570b1bd468a7aafe9af80e337a7fae9", git_commit_timestamp: "2024-02-15T02:33:40.000000000Z", git_branch: "main", rustc_semver: "1.75.0", rustc_channel: "stable", rustc_host_triple: "x86_64-unknown-illumos", rustc_commit_sha: "82e1608dfa6e0b5569232559e3d385fea5a93112", cargo_triple: "x86_64-unknown-illumos", debug: true, opt_level: 0, }
352024-02-15T02:57:29.555ZINFOcrucible: Upstairs <-> Downstairs Message Version: 5
362024-02-15T02:57:29.555ZINFOcrucible: Crucible 71d71e80-f3c3-438b-aa3d-1674a086f76a has session id: 305f25d3-39ad-40f9-999f-686ef9804043 session_id = 305f25d3-39ad-40f9-999f-686ef9804043
372024-02-15T02:57:29.555ZINFOcrucible: Upstairs opts: Upstairs UUID: 71d71e80-f3c3-438b-aa3d-1674a086f76a, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false session_id = 305f25d3-39ad-40f9-999f-686ef9804043
382024-02-15T02:57:29.555ZINFOcrucible: Crucible stats registered with UUID: 71d71e80-f3c3-438b-aa3d-1674a086f76a session_id = 305f25d3-39ad-40f9-999f-686ef9804043
39 Crucible runtime is spawned
402024-02-15T02:57:29.556ZINFOcrucible: 71d71e80-f3c3-438b-aa3d-1674a086f76a active request set session_id = 305f25d3-39ad-40f9-999f-686ef9804043
412024-02-15T02:57:29.556ZINFOcrucible: sending connect oneshot to client = downstairs client = 0 session_id = 305f25d3-39ad-40f9-999f-686ef9804043
422024-02-15T02:57:29.556ZINFOcrucible: The guest has requested activation with gen:1
432024-02-15T02:57:29.556ZINFOcrucible: client set_active_request while in New; waiting... = downstairs client = 0 session_id = 305f25d3-39ad-40f9-999f-686ef9804043
442024-02-15T02:57:29.556ZINFOcrucible: sending connect oneshot to client = downstairs client = 1 session_id = 305f25d3-39ad-40f9-999f-686ef9804043
452024-02-15T02:57:29.556ZINFOcrucible: client set_active_request while in New; waiting... = downstairs client = 1 session_id = 305f25d3-39ad-40f9-999f-686ef9804043
462024-02-15T02:57:29.556ZINFOcrucible: sending connect oneshot to client = downstairs client = 2 session_id = 305f25d3-39ad-40f9-999f-686ef9804043
472024-02-15T02:57:29.556ZINFOcrucible: client set_active_request while in New; waiting... = downstairs client = 2 session_id = 305f25d3-39ad-40f9-999f-686ef9804043
482024-02-15T02:57:29.556ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 305f25d3-39ad-40f9-999f-686ef9804043
492024-02-15T02:57:29.556ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 305f25d3-39ad-40f9-999f-686ef9804043
502024-02-15T02:57:29.556ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 305f25d3-39ad-40f9-999f-686ef9804043
512024-02-15T02:57:29.557ZINFOcrucible: ds_connection connected = io task client = 1 session_id = 305f25d3-39ad-40f9-999f-686ef9804043
522024-02-15T02:57:29.557ZINFOcrucible: ds_connection connected = io task client = 2 session_id = 305f25d3-39ad-40f9-999f-686ef9804043
532024-02-15T02:57:29.557ZINFOcrucible: ds_connection connected = io task client = 0 session_id = 305f25d3-39ad-40f9-999f-686ef9804043
542024-02-15T02:57:29.559ZINFOcrucible: ds_transition from New to WaitActive = downstairs client = 0 session_id = 305f25d3-39ad-40f9-999f-686ef9804043
552024-02-15T02:57:29.559ZINFOcrucible: [0] Transition from New to WaitActive = downstairs client = 0 session_id = 305f25d3-39ad-40f9-999f-686ef9804043
562024-02-15T02:57:29.559ZINFOcrucible: ds_transition from New to WaitActive = downstairs client = 1 session_id = 305f25d3-39ad-40f9-999f-686ef9804043
572024-02-15T02:57:29.559ZINFOcrucible: [1] Transition from New to WaitActive = downstairs client = 1 session_id = 305f25d3-39ad-40f9-999f-686ef9804043
582024-02-15T02:57:29.559ZINFOcrucible: ds_transition from New to WaitActive = downstairs client = 2 session_id = 305f25d3-39ad-40f9-999f-686ef9804043
592024-02-15T02:57:29.559ZINFOcrucible: [2] Transition from New to WaitActive = downstairs client = 2 session_id = 305f25d3-39ad-40f9-999f-686ef9804043
602024-02-15T02:57:29.561ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 305f25d3-39ad-40f9-999f-686ef9804043
612024-02-15T02:57:29.561ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 305f25d3-39ad-40f9-999f-686ef9804043
622024-02-15T02:57:29.561ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 305f25d3-39ad-40f9-999f-686ef9804043
632024-02-15T02:57:29.562ZINFOcrucible: ds_transition from WaitActive to WaitQuorum = downstairs client = 1 session_id = 305f25d3-39ad-40f9-999f-686ef9804043
642024-02-15T02:57:29.562ZINFOcrucible: [1] Transition from WaitActive to WaitQuorum = downstairs client = 1 session_id = 305f25d3-39ad-40f9-999f-686ef9804043
652024-02-15T02:57:29.562ZINFOcrucible: Waiting for true more clients to be ready session_id = 305f25d3-39ad-40f9-999f-686ef9804043
662024-02-15T02:57:29.562ZINFOcrucible: ds_transition from WaitActive to WaitQuorum = downstairs client = 0 session_id = 305f25d3-39ad-40f9-999f-686ef9804043
672024-02-15T02:57:29.562ZINFOcrucible: [0] Transition from WaitActive to WaitQuorum = downstairs client = 0 session_id = 305f25d3-39ad-40f9-999f-686ef9804043
682024-02-15T02:57:29.562ZINFOcrucible: Waiting for true more clients to be ready session_id = 305f25d3-39ad-40f9-999f-686ef9804043
692024-02-15T02:57:29.562ZINFOcrucible: ds_transition from WaitActive to WaitQuorum = downstairs client = 2 session_id = 305f25d3-39ad-40f9-999f-686ef9804043
702024-02-15T02:57:29.562ZINFOcrucible: [2] Transition from WaitActive to WaitQuorum = downstairs client = 2 session_id = 305f25d3-39ad-40f9-999f-686ef9804043
712024-02-15T02:57:29.563ZINFOcrucible: [0]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] = downstairs session_id = 305f25d3-39ad-40f9-999f-686ef9804043
722024-02-15T02:57:29.563ZINFOcrucible: [0]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] = downstairs session_id = 305f25d3-39ad-40f9-999f-686ef9804043
732024-02-15T02:57:29.563ZINFOcrucible: [0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = 305f25d3-39ad-40f9-999f-686ef9804043
742024-02-15T02:57:29.563ZINFOcrucible: [1]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] = downstairs session_id = 305f25d3-39ad-40f9-999f-686ef9804043
752024-02-15T02:57:29.563ZINFOcrucible: [1]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] = downstairs session_id = 305f25d3-39ad-40f9-999f-686ef9804043
762024-02-15T02:57:29.563ZINFOcrucible: [1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = 305f25d3-39ad-40f9-999f-686ef9804043
772024-02-15T02:57:29.563ZINFOcrucible: [2]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] = downstairs session_id = 305f25d3-39ad-40f9-999f-686ef9804043
782024-02-15T02:57:29.563ZINFOcrucible: [2]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] = downstairs session_id = 305f25d3-39ad-40f9-999f-686ef9804043
792024-02-15T02:57:29.563ZINFOcrucible: [2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = 305f25d3-39ad-40f9-999f-686ef9804043
802024-02-15T02:57:29.563ZINFOcrucible: Max found gen is 1 = downstairs session_id = 305f25d3-39ad-40f9-999f-686ef9804043
812024-02-15T02:57:29.563ZINFOcrucible: Generation requested: 1 >= found:1 = downstairs session_id = 305f25d3-39ad-40f9-999f-686ef9804043
822024-02-15T02:57:29.563ZINFOcrucible: Next flush: 1 = downstairs session_id = 305f25d3-39ad-40f9-999f-686ef9804043
832024-02-15T02:57:29.563ZINFOcrucible: All extents match = downstairs session_id = 305f25d3-39ad-40f9-999f-686ef9804043
842024-02-15T02:57:29.563ZINFOcrucible: No downstairs reconciliation required session_id = 305f25d3-39ad-40f9-999f-686ef9804043
852024-02-15T02:57:29.563ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 0 session_id = 305f25d3-39ad-40f9-999f-686ef9804043
862024-02-15T02:57:29.564ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 1 session_id = 305f25d3-39ad-40f9-999f-686ef9804043
87 {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30Activate successful
88 Wait for a query_work_queue command to finish before sending IO
89 ,"time":"2024-02-15T02:57:29.56402059Z","hostname":"ip-10-150-1-67.us-west-2.compute.internal","pid":1163,"client":"2","":"downstairs","session_id":"305f25d3-39ad-40f9-999f-686ef9804043"}
902024-02-15T02:57:29.564ZINFOcrucible: All required reconciliation work is completed session_id = 305f25d3-39ad-40f9-999f-686ef9804043
912024-02-15T02:57:29.564ZINFOcrucible: Set Downstairs and Upstairs active after reconciliation session_id = 305f25d3-39ad-40f9-999f-686ef9804043
922024-02-15T02:57:29.564ZINFOcrucible: 71d71e80-f3c3-438b-aa3d-1674a086f76a is now active with session: 305f25d3-39ad-40f9-999f-686ef9804043 session_id = 305f25d3-39ad-40f9-999f-686ef9804043
932024-02-15T02:57:29.564ZINFOcrucible: Set Active after no reconciliation session_id = 305f25d3-39ad-40f9-999f-686ef9804043
942024-02-15T02:57:29.564ZINFOcrucible: The guest has finished waiting for activation with:1
95 Region: es:100 ec:50 bs:4096 ts:20480000 tb:5000 max_io:256 or 1048576
96 Fill test
97 Read and Verify all blocks (0..5000 range:false)
98 Wrote out file "alan"
99 CLIENT: Tests done. All submitted work has been ACK'd
100 ----------------------------------------------------------------
101 Crucible gen:1 GIO:true work queues: Upstairs:0 downstairs:38
102 States: Active Active Active
103 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
104 65 Acked 1064 Read 100 Done Done Done false
105 66 Acked 1065 Read 100 Done Done Done false
106 67 Acked 1066 Read 100 Done Done Done false
107 68 Acked 1067 Read 100 Done Done Done false
108 69 Acked 1068 Read 100 Done Done Done false
109 70 Acked 1069 Read 100 Done Done Done false
110 71 Acked 1070 Read 100 Done Done Done false
111 72 Acked 1071 Read 100 Done Done Done false
112 73 Acked 1072 Read 100 Done Done Done false
113 74 Acked 1073 Read 100 Done Done Done false
114 75 Acked 1074 Read 100 Done Done Done false
115 76 Acked 1075 Read 100 Done Done Done false
116 77 Acked 1076 Read 100 Done Done Done false
117 78 Acked 1077 Read 100 Done Done Done false
118 79 Acked 1078 Read 100 Done Done Done false
119 80 Acked 1079 Read 100 Done Done Done false
120 81 Acked 1080 Read 100 Done Done Done false
121 82 Acked 1081 Read 100 Done Done Done false
122 83 Acked 1082 Read 100 Done Done Done false
123 84 Acked 1083 Read 100 Done Done Done false
124 85 Acked 1084 Read 100 Done Done Done false
125 86 Acked 1085 Read 100 Done Done Done false
126 87 Acked 1086 Read 100 Done Done Done false
127 88 Acked 1087 Read 100 Done Done Done false
128 89 Acked 1088 Read 100 Done Done Done false
129 90 Acked 1089 Read 100 Done Done Done false
130 91 Acked 1090 Read 100 Done Done Done false
131 92 Acked 1091 Read 100 Done Done Done false
132 93 Acked 1092 Read 100 Done Done Done false
133 94 Acked 1093 Read 100 Done Done Done false
134 95 Acked 1094 Read 100 Done Done Done false
135 96 Acked 1095 Read 100 Done Done Done false
136 97 Acked 1096 Read 100 Done Done Done false
137 98 Acked 1097 Read 100 Done Done Done false
138 99 Acked 1098 Read 100 Done Done Done false
139 100 Acked 1099 Read 100 Done Done Done false
140 101 Acked 1100 Read 100 Done Done Done false
141 102 Acked 1101 Read 100 Done Done Done false
142 STATES DS:0 DS:1 DS:2 TOTAL
143 New 0 0 0 0
144 Sent 0 0 0 0
145 Done 38 38 38 114
146 Skipped 0 0 0 0
147 Error 0 0 0 0
148 Last Flush: 1063 1063 1063
149 Downstairs last five completed: 1063 1062 1061 1060 1059
150 Upstairs last five completed: 102 101 100 99 98
151 CLIENT: Up:0 ds:38 act:3
152 ----------------------------------------------------------------
153 Crucible gen:1 GIO:true work queues: Upstairs:0 downstairs:0
154 Downstairs last five completed: 1102 1101 1100 1099 1098
155 Upstairs last five completed: 103 102 101 100 99
156 CLIENT: Up:0 ds:0 act:3
157 CLIENT: All crucible jobs finished, exiting program
1582024-02-15T02:57:34.481ZWARNcrucible: Guest handle has been dropped
1592024-02-15T02:57:34.481ZWARNcrucible: client_stop_rx closed unexpectedly: RecvError(()) = io task client = 0 session_id = 305f25d3-39ad-40f9-999f-686ef9804043
1602024-02-15T02:57:34.481ZWARNcrucible: client request queue closed unexpectedly; is the program exiting? = io task client = 2 session_id = 305f25d3-39ad-40f9-999f-686ef9804043
1612024-02-15T02:57:34.481ZWARNcrucible: client_stop_rx closed unexpectedly: RecvError(()) = io task client = 1 session_id = 305f25d3-39ad-40f9-999f-686ef9804043
1622024-02-15T02:57:34.481ZWARNcrucible: client task is sending Done(QueueClosed) = io task client = 0 session_id = 305f25d3-39ad-40f9-999f-686ef9804043
1632024-02-15T02:57:34.481ZWARNcrucible: client task is sending Done(QueueClosed) = io task client = 1 session_id = 305f25d3-39ad-40f9-999f-686ef9804043
1642024-02-15T02:57:34.481ZWARNcrucible: client task could not reply to main task; shutting down? = io task client = 0 session_id = 305f25d3-39ad-40f9-999f-686ef9804043
1652024-02-15T02:57:34.481ZWARNcrucible: client task could not reply to main task; shutting down? = io task client = 1 session_id = 305f25d3-39ad-40f9-999f-686ef9804043
1662024-02-15T02:57:34.481ZINFOcrucible: client task is exiting = io task client = 0 session_id = 305f25d3-39ad-40f9-999f-686ef9804043
1672024-02-15T02:57:34.481ZINFOcrucible: client task is exiting = io task client = 1 session_id = 305f25d3-39ad-40f9-999f-686ef9804043
1682024-02-15T02:57:34.481ZWARNcrucible: client task is sending Done(QueueClosed) = io task client = 2 session_id = 305f25d3-39ad-40f9-999f-686ef9804043
1692024-02-15T02:57:34.481ZWARNcrucible: client task could not reply to main task; shutting down? = io task client = 2 session_id = 305f25d3-39ad-40f9-999f-686ef9804043
1702024-02-15T02:57:34.481ZINFOcrucible: client task is exiting = io task client = 2 session_id = 305f25d3-39ad-40f9-999f-686ef9804043
171 Replay loop starts now February 15, 2024 at 02:57:34 AM UTC
1722024-02-15T02:57:34.506ZINFOcrucible: Upstairs starts
173 {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"56eb785b9570b1bd468a7aafe9af80e337a7fae9\",\n git_commit_timestamp: \"2024-02-15T02:33:40.000000000Z\",\n git_branch: \"main\",\nCrucible runtime is spawned
174 rustc_semver: \"1.75.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"82e1608dfa6e0b5569232559e3d385fea5a93112\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-02-15T02:57:34.506384985Z","hostname":"ip-10-150-1-67.us-west-2.compute.internal","pid":1167}
1752024-02-15T02:57:34.506ZINFOcrucible: Upstairs <-> Downstairs Message Version: 5
1762024-02-15T02:57:34.506ZINFOcrucible: Crucible a966a5ab-893a-4cac-98f0-7b0ed9bb3f5a has session id: 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
1772024-02-15T02:57:34.506ZINFOcrucible: Upstairs opts: Upstairs UUID: a966a5ab-893a-4cac-98f0-7b0ed9bb3f5a, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
1782024-02-15T02:57:34.506ZINFOcrucible: Crucible stats registered with UUID: a966a5ab-893a-4cac-98f0-7b0ed9bb3f5a session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
1792024-02-15T02:57:34.506ZINFOcrucible: The guest has requested activation with gen:2
1802024-02-15T02:57:34.506ZINFOcrucible: a966a5ab-893a-4cac-98f0-7b0ed9bb3f5a active request set session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
1812024-02-15T02:57:34.506ZINFOcrucible: sending connect oneshot to client = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
1822024-02-15T02:57:34.506ZINFOcrucible: client set_active_request while in New; waiting... = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
1832024-02-15T02:57:34.506ZINFOcrucible: sending connect oneshot to client = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
1842024-02-15T02:57:34.506ZINFOcrucible: client set_active_request while in New; waiting... = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
1852024-02-15T02:57:34.506ZINFOcrucible: sending connect oneshot to client = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
1862024-02-15T02:57:34.506ZINFOcrucible: client set_active_request while in New; waiting... = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
1872024-02-15T02:57:34.506ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
1882024-02-15T02:57:34.506ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
1892024-02-15T02:57:34.506ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
1902024-02-15T02:57:34.506ZINFOcrucible: ds_connection connected = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
1912024-02-15T02:57:34.506ZINFOcrucible: ds_connection connected = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
1922024-02-15T02:57:34.506ZINFOcrucible: ds_connection connected = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
1932024-02-15T02:57:34.507ZINFOcrucible: ds_transition from New to WaitActive = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
1942024-02-15T02:57:34.507ZINFOcrucible: [2] Transition from New to WaitActive = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
1952024-02-15T02:57:34.507ZINFOcrucible: ds_transition from New to WaitActive = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
1962024-02-15T02:57:34.507ZINFOcrucible: [0] Transition from New to WaitActive = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
1972024-02-15T02:57:34.507ZINFOcrucible: ds_transition from New to WaitActive = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
1982024-02-15T02:57:34.507ZINFOcrucible: [1] Transition from New to WaitActive = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
1992024-02-15T02:57:34.508ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
2002024-02-15T02:57:34.508ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
2012024-02-15T02:57:34.508ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
2022024-02-15T02:57:34.509ZINFOcrucible: ds_transition from WaitActive to WaitQuorum = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
2032024-02-15T02:57:34.509ZINFOcrucible: [2] Transition from WaitActive to WaitQuorum = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
2042024-02-15T02:57:34.509ZINFOcrucible: Waiting for true more clients to be ready session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
2052024-02-15T02:57:34.509ZINFOcrucible: ds_transition from WaitActive to WaitQuorum = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
2062024-02-15T02:57:34.509ZINFOcrucible: [1] Transition from WaitActive to WaitQuorum = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
2072024-02-15T02:57:34.509ZINFOcrucible: Waiting for true more clients to be ready session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
2082024-02-15T02:57:34.509ZINFOcrucible: ds_transition from WaitActive to WaitQuorum = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
2092024-02-15T02:57:34.509ZINFOcrucible: [0] Transition from WaitActive to WaitQuorum = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
2102024-02-15T02:57:34.509ZINFOcrucible: [0]R flush_numbers[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
2112024-02-15T02:57:34.509ZINFOcrucible: [0]R generation[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
2122024-02-15T02:57:34.509ZINFOcrucible: [0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
2132024-02-15T02:57:34.509ZINFOcrucible: [1]R flush_numbers[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
2142024-02-15T02:57:34.509ZINFOcrucible: [1]R generation[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
2152024-02-15T02:57:34.509ZINFOcrucible: [1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
2162024-02-15T02:57:34.509ZINFOcrucible: [2]R flush_numbers[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
2172024-02-15T02:57:34.509ZINFOcrucible: [2]R generation[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
2182024-02-15T02:57:34.509ZINFOcrucible: [2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
2192024-02-15T02:57:34.509ZINFOcrucible: Max found gen is 2 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
220 {"msg":"Generation requested: 2 >= found:2"Activate successful
221 ,Wait for a query_work_queue command to finish before sending IO
222 "v":0,"name":"crucible","level":30,"time":"2024-02-15T02:57:34.50953112Z","hostname":"ip-10-150-1-67.us-west-2.compute.internal","pid":1167,"":"downstairs","":"downstairs","session_id":"4f5a63e5-aa00-4890-9f6b-96c2e069fdb2"}
2232024-02-15T02:57:34.509ZINFOcrucible: Next flush: 2 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
2242024-02-15T02:57:34.509ZINFOcrucible: All extents match = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
2252024-02-15T02:57:34.509ZINFOcrucible: No downstairs reconciliation required session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
2262024-02-15T02:57:34.509ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
2272024-02-15T02:57:34.509ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
2282024-02-15T02:57:34.509ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
2292024-02-15T02:57:34.509ZINFOcrucible: All required reconciliation work is completed session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
2302024-02-15T02:57:34.509ZINFOcrucible: Set Downstairs and Upstairs active after reconciliation session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
2312024-02-15T02:57:34.509ZINFOcrucible: a966a5ab-893a-4cac-98f0-7b0ed9bb3f5a is now active with session: 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
2322024-02-15T02:57:34.509ZINFOcrucible: Set Active after no reconciliation session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
2332024-02-15T02:57:34.509ZINFOcrucible: The guest has finished waiting for activation with:2
234 Region: es:100 ec:50 bs:4096 ts:20480000 tb:5000 max_io:256 or 1048576
235 Loading write count information from file "alan"
236 Feb 15 02:57:34.560 INFO accepted connection, remote_addr: 127.0.0.1:54744, local_addr: 127.0.0.1:9998
237 Main task has work to do, go find it
238 got dsc Stop(1)
239 stop 1
240 Feb 15 02:57:34.561 INFO request completed, latency_us: 419, response_code: 204, [8820] Got stop action so:false kr:false
241 uri: /stop/cid/1, method: POST, req_id: 126ddb1a-bacb-43dd-892b-81aaa2b6175f, remote_addr: 127.0.0.1:54744, local_addr: 127.0.0.1:9998
2422024-02-15T02:57:34.562ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
2432024-02-15T02:57:34.562ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
2442024-02-15T02:57:34.562ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
2452024-02-15T02:57:34.562ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
2462024-02-15T02:57:34.562ZINFOcrucible: [1] client re-new 0 jobs since flush 0 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
2472024-02-15T02:57:34.562ZINFOcrucible: client task is exiting = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
248 Feb 15 02:57:34.563 INFO request completed, latency_us: 207, response_code: 200, uri: /state/cid/1, method: GET, req_id: 35c04321-89a2-4178-a0c9-81d44ba1df3c, remote_addr: 127.0.0.1:54744, local_addr: 127.0.0.1:9998
249 [8820] Exited with: ExitStatus(unix_wait_status(9))
250 [8820][1] reports Exit
2512024-02-15T02:57:35.563ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
2522024-02-15T02:57:35.563ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
2532024-02-15T02:57:35.564ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
2542024-02-15T02:57:35.564ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
2552024-02-15T02:57:35.564ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
2562024-02-15T02:57:35.564ZINFOcrucible: [1] client re-new 0 jobs since flush 0 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
2572024-02-15T02:57:35.564ZINFOcrucible: client task is exiting = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
2582024-02-15T02:57:36.565ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
2592024-02-15T02:57:36.565ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
2602024-02-15T02:57:36.565ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
2612024-02-15T02:57:36.565ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
2622024-02-15T02:57:36.565ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
2632024-02-15T02:57:36.565ZINFOcrucible: [1] client re-new 0 jobs since flush 0 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
2642024-02-15T02:57:36.565ZINFOcrucible: client task is exiting = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
2652024-02-15T02:57:37.567ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
2662024-02-15T02:57:37.567ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
2672024-02-15T02:57:37.567ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
2682024-02-15T02:57:37.567ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
2692024-02-15T02:57:37.567ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
2702024-02-15T02:57:37.567ZINFOcrucible: [1] client re-new 0 jobs since flush 0 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
2712024-02-15T02:57:37.567ZINFOcrucible: client task is exiting = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
272 Feb 15 02:57:38.565 INFO request completed, latency_us: 198, response_code: 200, uri: /state/cid/1, method: GET, req_id: 9a26d7d4-79b0-4628-a0b8-b3addbcbb129, remote_addr: 127.0.0.1:54744, local_addr: 127.0.0.1:9998
273 001/300 Write block 1500 len 8192 data: 2 2
274 002/300 Write block 4624 len 8192 data: 2 2
275 003/300 Read block 2608 len 16384
2762024-02-15T02:57:38.568ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
2772024-02-15T02:57:38.568ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
2782024-02-15T02:57:38.568ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
2792024-02-15T02:57:38.568ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
2802024-02-15T02:57:38.568ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
2812024-02-15T02:57:38.568ZINFOcrucible: [1] client re-new 3 jobs since flush 0 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
2822024-02-15T02:57:38.569ZINFOcrucible: client task is exiting = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
283 004/300 Write block 370 len 20480 data: 2 2 2 2 2
284 005/300 Write block 2435 len 20480 data: 2 2 2 2 2
285 006/300 Write block 1916 len 16384 data: 2 2 2 2
286 007/300 Read block 3849 len 20480
287 008/300 Read block 180 len 12288
288 009/300 Write block 836 len 36864 data: 2 2 2 2 2 2 2 2 2
289 010/300 Read block 2322 len 20480
290 011/300 Read block 1575 len 16384
291 012/300 Write block 2618 len 16384 data: 2 2 2 2
292 013/300 Read block 3344 len 36864
293 014/300 Write block 4977 len 40960 data: 2 2 2 2 2 2 2 2 2 2
294 015/300 Read block 1747 len 24576
295 016/300 Read block 2772 len 36864
296 017/300 Read block 948 len 12288
297 018/300 Read block 3825 len 40960
298 019/300 Write block 3410 len 24576 data: 2 2 2 2 2 2
299 020/300 Read block 3622 len 36864
300 021/300 Write block 3694 len 20480 data: 2 2 2 2 2
301 022/300 Read block 4905 len 28672
302 023/300 Write block 3040 len 28672 data: 2 2 2 2 2 2 2
303 024/300 Write block 3296 len 12288 data: 2 2 2
304 025/300 Write block 1909 len 36864 data: 2 2 2 2 2 2 2 3 3
305 026/300 Write block 590 len 8192 data: 2 2
306 027/300 Flush
307 028/300 Flush
308 029/300 Flush
309 030/300 Read block 936 len 28672
310 031/300 Read block 3016 len 16384
311 032/300 Write block 589 len 16384 data: 2 3 3 2
312 033/300 Read block 1548 len 24576
313 034/300 Write block 1620 len 28672 data: 2 2 2 2 2 2 2
314 035/300 Write block 4019 len 20480 data: 2 2 2 2 2
315 036/300 Write block 4417 len 16384 data: 2 2 2 2
316 037/300 Write block 342 len 28672 data: 2 2 2 2 2 2 2
317 038/300 Write block 4439 len 32768 data: 2 2 2 2 2 2 2 2
318 039/300 Write block 2180 len 4096 data: 2
319 040/300 Flush
320 041/300 Read block 341 len 28672
321 042/300 Read block 4686 len 8192
322 043/300 Read block 3130 len 32768
323 044/300 Read block 811 len 16384
324 045/300 Write block 649 len 20480 data: 2 2 2 2 2
325 046/300 Read block 2187 len 16384
326 047/300 Read block 749 len 4096
327 048/300 Read block 4144 len 12288
328 049/300 Write block 3729 len 8192 data: 2 2
329 050/300 Write block 2291 len 32768 data: 2 2 2 2 2 2 2 2
330 051/300 Read block 2130 len 28672
331 052/300 Write block 1144 len 40960 data: 2 2 2 2 2 2 2 2 2 2
332 053/300 Write block 3463 len 8192 data: 2 2
333 054/300 Write block 547 len 8192 data: 2 2
334 055/300 Read block 4052 len 16384
335 056/300 Read block 1262 len 4096
336 057/300 Flush
337 058/300 Read block 3249 len 28672
338 059/300 Read block 4766 len 32768
339 060/300 Write block 1508 len 24576 data: 2 2 2 2 2 2
340 061/300 Write block 3531 len 36864 data: 2 2 2 2 2 2 2 2 2
341 062/300 Write block 4648 len 16384 data: 2 2 2 2
342 063/300 Flush
343 064/300 Read block 4883 len 24576
344 065/300 Write block 2543 len 8192 data: 2 2
345 066/300 Write block 4055 len 4096 data: 2
346 067/300 Read block 780 len 40960
347 068/300 Read block 2802 len 40960
348 069/300 Read block 704 len 36864
349 070/300 Flush
350 071/300 Read block 3942 len 24576
351 072/300 Write block 810 len 36864 data: 2 2 2 2 2 2 2 2 2
352 073/300 Read block 1050 len 16384
353 074/300 Flush
354 075/300 Write block 1009 len 28672 data: 2 2 2 2 2 2 2
355 076/300 Write block 2859 len 24576 data: 2 2 2 2 2 2
356 077/300 Write block 4772 len 24576 data: 2 2 2 2 2 2
357 078/300 Read block 4173 len 8192
358 079/300 Write block 1493 len 12288 data: 2 2 2
359 080/300 Write block 91 len 20480 data: 2 2 2 2 2
360 081/300 Write block 2741 len 36864 data: 2 2 2 2 2 2 2 2 2
361 082/300 Write block 651 len 12288 data: 3 3 3
362 083/300 Read block 1850 len 12288
363 084/300 Read block 1919 len 32768
364 085/300 Read block 3891 len 40960
365 086/300 Read block 3359 len 36864
366 087/300 Read block 4686 len 28672
367 088/300 Read block 3960 len 20480
368 089/300 Write block 1527 len 20480 data: 2 2 2 2 2
369 090/300 Read block 4466 len 28672
370 091/300 Read block 3528 len 12288
371 092/300 Read block 2502 len 40960
372 093/300 Write block 2899 len 28672 data: 2 2 2 2 2 2 2
373 094/300 Read block 83 len 28672
374 095/300 Read block 432 len 24576
375 096/300 Read block 755 len 12288
376 097/300 Flush
377 098/300 Write block 4759 len 24576 data: 2 2 2 2 2 2
378 099/300 Write block 162 len 28672 data: 2 2 2 2 2 2 2
379 100/300 Write block 988 len 8192 data: 2 2
380 101/300 Write block 2117 len 40960 data: 2 2 2 2 2 2 2 2 2 2
381 102/300 Write block 3058 len 40960 data: 2 2 2 2 2 2 2 2 2 2
382 103/300 Read block 857 len 12288
383 104/300 Write block 1810 len 32768 data: 2 2 2 2 2 2 2 2
384 105/300 Read block 4216 len 12288
385 106/300 Write block 222 len 40960 data: 2 2 2 2 2 2 2 2 2 2
386 107/300 Flush
387 108/300 Write block 4906 len 24576 data: 2 2 2 2 2 2
388 109/300 Read block 4162 len 24576
389 110/300 Read block 2425 len 32768
390 111/300 Read block 1084 len 36864
391 112/300 Read block 4754 len 4096
392 113/300 Write block 3512 len 8192 data: 2 2
393 114/300 Write block 4685 len 32768 data: 2 2 2 2 2 2 2 2
394 115/300 Read block 4500 len 36864
395 116/300 Read block 3227 len 28672
396 117/300 Write block 509 len 36864 data: 2 2 2 2 2 2 2 2 2
397 118/300 Read block 1438 len 40960
398 119/300 Read block 4232 len 8192
399 120/300 Write block 1528 len 16384 data: 3 3 3 3
400 121/300 Write block 2440 len 8192 data: 2 2
401 122/300 Read block 4782 len 32768
402 123/300 Read block 1044 len 8192
403 124/300 Write block 4768 len 4096 data: 2
404 125/300 Write block 2708 len 24576 data: 2 2 2 2 2 2
405 126/300 Read block 1864 len 16384
406 127/300 Write block 3154 len 36864 data: 2 2 2 2 2 2 2 2 2
407 128/300 Read block 4863 len 12288
408 129/300 Write block 3671 len 32768 data: 2 2 2 2 2 2 2 2
409 130/300 Write block 4844 len 12288 data: 2 2 2
410 131/300 Write block 377 len 16384 data: 2 2 2 2
411 132/300 Flush
412 133/300 Read block 273 len 12288
413 134/300 Write block 2083 len 24576 data: 2 2 2 2 2 2
414 135/300 Write block 4371 len 4096 data: 2
415 136/300 Write block 258 len 40960 data: 2 2 2 2 2 2 2 2 2 2
416 137/300 Write block 3127 len 16384 data: 2 2 2 2
417 138/300 Write block 4152 len 4096 data: 2
418 139/300 Read block 2159 len 36864
419 140/300 Read block 5 len 12288
420 141/300 Read block 4412 len 12288
421 142/300 Read block 3861 len 28672
422 143/300 Write block 1973 len 12288 data: 2 2 2
423 144/300 Write block 1673 len 24576 data: 2 2 2 2 2 2
424 145/300 Flush
425 146/300 Write block 3448 len 36864 data: 2 2 2 2 2 2 2 2 2
426 147/300 Write block 26 len 8192 data: 2 2
427 148/300 Read block 248 len 12288
428 149/300 Read block 3780 len 16384
429 150/300 Write block 1202 len 28672 data: 2 2 2 2 2 2 2
430 151/300 Read block 4286 len 32768
431 152/300 Read block 1494 len 32768
432 153/300 Read block 2909 len 36864
433 154/300 Flush
434 155/300 Write block 4629 len 4096 data: 2
435 156/300 Read block 3264 len 40960
436 157/300 Read block 2061 len 16384
437 158/300 Read block 2894 len 28672
438 159/300 Read block 972 len 16384
439 160/300 Read block 3016 len 40960
440 161/300 Write block 969 len 28672 data: 2 2 2 2 2 2 2
441 162/300 Write block 2158 len 24576 data: 2 2 2 2 2 2
442 163/300 Write block 142 len 16384 data: 2 2 2 2
443 164/300 Write block 4257 len 8192 data: 2 2
444 165/300 Flush
445 166/300 Read block 182 len 16384
446 167/300 Read block 4313 len 8192
447 168/300 Write block 3981 len 36864 data: 2 2 2 2 2 2 2 2 2
448 169/300 Write block 1480 len 40960 data: 2 2 2 2 2 2 2 2 2 2
449 170/300 Write block 3303 len 32768 data: 2 2 2 2 2 2 2 2
450 171/300 Write block 2270 len 8192 data: 2 2
451 172/300 Read block 3110 len 36864
452 173/300 Read block 4004 len 28672
453 174/300 Flush
454 175/300 Read block 4029 len 28672
455 176/300 Read block 4818 len 40960
456 177/300 Write block 787 len 16384 data: 2 2 2 2
457 178/300 Flush
458 179/300 Write block 3497 len 24576 data: 2 2 2 2 2 2
459 180/300 Write block 4373 len 24576 data: 2 2 2 2 2 2
460 181/300 Read block 617 len 16384
461 182/300 Write block 4838 len 20480 data: 2 2 2 2 2
462 183/300 Read block 3087 len 8192
463 184/300 Write block 4931 len 16384 data: 2 2 2 2
464 185/300 Read block 1998 len 32768
465 186/300 Read block 4737 len 8192
466 187/300 Write block 2974 len 8192 data: 2 2
467 188/300 Write block 4666 len 40960 data: 2 2 2 2 2 2 2 2 2 2
468 189/300 Write block 3362 len 4096 data: 2
469 190/300 Read block 4535 len 8192
470 191/300 Read block 4635 len 24576
471 192/300 Read block 4268 len 24576
472 193/300 Read block 3211 len 32768
473 194/300 Read block 3643 len 32768
474 195/300 Read block 3756 len 4096
475 196/300 Write block 796 len 12288 data: 2 2 2
476 197/300 Read block 1940 len 16384
477 198/300 Write block 2106 len 4096 data: 2
478 199/300 Write block 2101 len 4096 data: 2
479 200/300 Write block 783 len 4096 data: 2
480 201/300 Read block 2546 len 40960
481 202/300 Read block 2364 len 24576
482 203/300 Write block 338 len 8192 data: 2 2
483 204/300 Write block 21 len 20480 data: 2 2 2 2 2
484 205/300 Read block 20 len 36864
485 206/300 Read block 2083 len 28672
486 207/300 Read block 4926 len 32768
487 208/300 Write block 3024 len 40960 data: 2 2 2 2 2 2 2 2 2 2
488 209/300 Write block 4247 len 36864 data: 2 2 2 2 2 2 2 2 2
489 210/300 Read block 2249 len 32768
490 211/300 Read block 2465 len 36864
491 212/300 Read block 2781 len 8192
492 213/300 Write block 2018 len 8192 data: 2 2
493 214/300 Read block 3450 len 4096
494 215/300 Write block 513 len 16384 data: 3 3 3 3
495 216/300 Write block 1722 len 36864 data: 2 2 2 2 2 2 2 2 2
496 217/300 Read block 924 len 12288
497 218/300 Read block 3692 len 20480
498 219/300 Read block 2387 len 36864
499 220/300 Write block 4768 len 8192 data: 3 2
500 221/300 Read block 165 len 28672
501 222/300 Read block 3634 len 12288
502 223/300 Read block 2808 len 40960
503 224/300 Read block 1841 len 20480
504 225/300 Write block 1119 len 24576 data: 2 2 2 2 2 2
505 226/300 Write block 4885 len 16384 data: 2 2 2 2
506 227/300 Write block 4712 len 16384 data: 2 2 2 2
507 228/300 Flush
508 229/300 Flush
509 230/300 Write block 1684 len 36864 data: 2 2 2 2 2 2 2 2 2
510 231/300 Read block 3580 len 36864
511 232/300 Read block 4268 len 28672
512 233/300 Read block 1559 len 20480
513 234/300 Read block 4637 len 32768
514 235/300 Read block 3712 len 28672
515 236/300 Read block 3054 len 4096
516 237/300 Flush
517 238/300 Flush
518 239/300 Read block 3315 len 12288
519 240/300 Write block 3951 len 24576 data: 2 2 2 2 2 2
520 241/300 Write block 1244 len 32768 data: 2 2 2 2 2 2 2 2
521 242/300 Read block 4252 len 32768
522 243/300 Read block 3286 len 4096
523 244/300 Write block 2188 len 36864 data: 2 2 2 2 2 2 2 2 2
524 245/300 Write block 2664 len 28672 data: 2 2 2 2 2 2 2
525 246/300 Read block 1752 len 28672
526 247/300 Write block 1544 len 28672 data: 2 2 2 2 2 2 2
527 248/300 Read block 2547 len 28672
528 249/300 Read block 3554 len 12288
529 250/300 Read block 2123 len 4096
530 251/300 Read block 2086 len 16384
531 252/300 Write block 4423 len 8192 data: 2 2
532 253/300 Write block 1998 len 16384 data: 2 2 2 2
533 254/300 Read block 618 len 36864
534 255/300 Read block 2924 len 20480
535 256/300 Flush
536 257/300 Write block 531 len 28672 data: 2 2 2 2 2 2 2
537 258/300 Read block 1286 len 40960
538 259/300 Read block 251 len 4096
539 260/300 Write block 1177 len 40960 data: 2 2 2 2 2 2 2 2 2 2
540 261/300 Read block 879 len 36864
541 262/300 Write block 2133 len 4096 data: 2
542 263/300 Write block 3490 len 24576 data: 2 2 2 2 2 2
543 264/300 Read block 3163 len 16384
544 265/300 Flush
545 266/300 Write block 3806 len 8192 data: 2 2
546 267/300 Write block 484 len 36864 data: 2 2 2 2 2 2 2 2 2
547 268/300 Read block 1439 len 12288
548 269/300 Read block 992 len 40960
549 270/300 Write block 4621 len 8192 data: 2 2
550 271/300 Write block 2850 len 4096 data: 2
551 272/300 Write block 1882 len 8192 data: 2 2
552 273/300 Read block 2098 len 28672
553 274/300 Write block 851 len 40960 data: 2 2 2 2 2 2 2 2 2 2
554 275/300 Read block 2708 len 4096
555 276/300 Write block 321 len 40960 data: 2 2 2 2 2 2 2 2 2 2
556 277/300 Read block 3863 len 12288
557 278/300 Flush
558 279/300 Write block 794 len 24576 data: 2 2 3 3 3 2
559 280/300 Read block 1858 len 28672
560 281/300 Write block 2581 len 40960 data: 2 2 2 2 2 2 2 2 2 2
561 282/300 Write block 249 len 40960 data: 2 2 2 2 2 2 2 2 2 3
562 283/300 Write block 734 len 28672 data: 2 2 2 2 2 2 2
563 284/300 Read block 4506 len 12288
564 285/300 Read block 1431 len 28672
565 286/300 Write block 1641 len 24576 data: 2 2 2 2 2 2
566 287/300 Read block 3861 len 28672
567 288/300 Write block 1622 len 28672 data: 3 3 3 3 3 2 2
568 289/300 Read block 983 len 16384
569 290/300 Write block 3028 len 40960 data: 3 3 3 3 3 3 2 2 2 2
570 291/300 Read block 2003 len 32768
571 292/300 Read block 810 len 28672
572 293/300 Write block 2221 len 28672 data: 2 2 2 2 2 2 2
573 294/300 Write block 1150 len 20480 data: 3 3 3 3 2
574 295/300 Read block 1578 len 24576
575 296/300 Write block 2823 len 16384 data: 2 2 2 2
576 297/300 Read block 603 len 20480
577 298/300 Read block 4854 len 36864
578 299/300 Read block 1310 len 32768
579 300/300 Flush
580 Main task has work to do, go find it
581 got dsc Start(1)
582 start 1
583 Feb 15 02:57:38.905 INFO request completed, latency_us: 219, response_code: 204, uri: /start/cid/1, method: POST, req_id: 2bb1367b-f521-4cf8-a530-25f3d3365c32, remote_addr: 127.0.0.1:54744, local_addr: 127.0.0.1:9998
584 Replay: started 1, returned:Ok(())
585 ----------------------------------------------------------------
586 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:300
587 States: Active Offline Active
588 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
589 1 Acked 1000 Write 2 Done New Done true
590 2 Acked 1001 Write 2 Done New Done true
591 3 Acked 1002 Read 4 Done New Done true
592 4 Acked 1003 Write 5 Done New Done false
593 5 Acked 1004 Write 5 Done New Done false
594 6 Acked 1005 Write 4 Done New Done false
595 7 Acked 1006 Read 5 Done New Done false
596 8 Acked 1007 Read 3 Done New Done false
597 9 Acked 1008 Write 9 Done New Done false
598 10 Acked 1009 Read 5 Done New Done false
599 11 Acked 1010 Read 4 Done New Done false
600 12 Acked 1011 Write 4 Done New Done false
601 13 Acked 1012 Read 9 Done New Done false
602 14 Acked 1013 Write 10 Done New Done false
603 15 Acked 1014 Read 6 Done New Done false
604 16 Acked 1015 Read 9 Done New Done false
605 17 Acked 1016 Read 3 Done New Done false
606 18 Acked 1017 Read 10 Done New Done false
607 19 Acked 1018 Write 6 Done New Done false
608 20 Acked 1019 Read 9 Done New Done false
609 21 Acked 1020 Write 5 Done New Done false
610 22 Acked 1021 Read 7 Done New Done false
611 23 Acked 1022 Write 7 Done New Done false
612 24 Acked 1023 Write 3 Done New Done false
613 25 Acked 1024 Write 9 Done New Done false
614 26 Acked 1025 Write 2 Done New Done false
615 27 Acked 1026 Flush 0 Done New Done false
616 28 Acked 1027 Flush 0 Done New Done false
617 29 Acked 1028 Flush 0 Done New Done false
618 30 Acked 1029 Read 7 Done New Done false
619 31 Acked 1030 Read 4 Done New Done false
620 32 Acked 1031 Write 4 Done New Done false
621 33 Acked 1032 Read 6 Done New Done false
622 34 Acked 1033 Write 7 Done New Done false
623 35 Acked 1034 Write 5 Done New Done false
624 36 Acked 1035 Write 4 Done New Done false
625 37 Acked 1036 Write 7 Done New Done false
626 38 Acked 1037 Write 8 Done New Done false
627 39 Acked 1038 Write 1 Done New Done false
628 40 Acked 1039 Flush 0 Done New Done false
629 41 Acked 1040 Read 7 Done New Done false
630 42 Acked 1041 Read 2 Done New Done false
631 43 Acked 1042 Read 8 Done New Done false
632 44 Acked 1043 Read 4 Done New Done false
633 45 Acked 1044 Write 5 Done New Done false
634 46 Acked 1045 Read 4 Done New Done false
635 47 Acked 1046 Read 1 Done New Done false
636 48 Acked 1047 Read 3 Done New Done false
637 49 Acked 1048 Write 2 Done New Done false
638 50 Acked 1049 Write 8 Done New Done false
639 51 Acked 1050 Read 7 Done New Done false
640 52 Acked 1051 Write 10 Done New Done false
641 53 Acked 1052 Write 2 Done New Done false
642 54 Acked 1053 Write 2 Done New Done false
643 55 Acked 1054 Read 4 Done New Done false
644 56 Acked 1055 Read 1 Done New Done false
645 57 Acked 1056 Flush 0 Done New Done false
646 58 Acked 1057 Read 7 Done New Done false
647 59 Acked 1058 Read 8 Done New Done false
648 60 Acked 1059 Write 6 Done New Done false
649 61 Acked 1060 Write 9 Done New Done false
650 62 Acked 1061 Write 4 Done New Done false
651 63 Acked 1062 Flush 0 Done New Done false
652 64 Acked 1063 Read 6 Done New Done false
653 65 Acked 1064 Write 2 Done New Done false
654 66 Acked 1065 Write 1 Done New Done false
655 67 Acked 1066 Read 10 Done New Done false
656 68 Acked 1067 Read 10 Done New Done false
657 69 Acked 1068 Read 9 Done New Done false
658 70 Acked 1069 Flush 0 Done New Done false
659 71 Acked 1070 Read 6 Done New Done false
660 72 Acked 1071 Write 9 Done New Done false
661 73 Acked 1072 Read 4 Done New Done false
662 74 Acked 1073 Flush 0 Done New Done false
663 75 Acked 1074 Write 7 Done New Done false
664 76 Acked 1075 Write 6 Done New Done false
665 77 Acked 1076 Write 6 Done New Done false
666 78 Acked 1077 Read 2 Done New Done false
667 79 Acked 1078 Write 3 Done New Done false
668 80 Acked 1079 Write 5 Done New Done false
669 81 Acked 1080 Write 9 Done New Done false
670 82 Acked 1081 Write 3 Done New Done false
671 83 Acked 1082 Read 3 Done New Done false
672 84 Acked 1083 Read 8 Done New Done false
673 85 Acked 1084 Read 10 Done New Done false
674 86 Acked 1085 Read 9 Done New Done false
675 87 Acked 1086 Read 7 Done New Done false
676 88 Acked 1087 Read 5 Done New Done false
677 89 Acked 1088 Write 5 Done New Done false
678 90 Acked 1089 Read 7 Done New Done false
679 91 Acked 1090 Read 3 Done New Done false
680 92 Acked 1091 Read 10 Done New Done false
681 93 Acked 1092 Write 7 Done New Done false
682 94 Acked 1093 Read 7 Done New Done false
683 95 Acked 1094 Read 6 Done New Done false
684 96 Acked 1095 Read 3 Done New Done false
685 97 Acked 1096 Flush 0 Done New Done false
686 98 Acked 1097 Write 6 Done New Done false
687 99 Acked 1098 Write 7 Done New Done false
688 100 Acked 1099 Write 2 Done New Done false
689 101 Acked 1100 Write 10 Done New Done false
690 102 Acked 1101 Write 10 Done New Done false
691 103 Acked 1102 Read 3 Done New Done false
692 104 Acked 1103 Write 8 Done New Done false
693 105 Acked 1104 Read 3 Done New Done false
694 106 Acked 1105 Write 10 Done New Done false
695 107 Acked 1106 Flush 0 Done New Done false
696 108 Acked 1107 Write 6 Done New Done false
697 109 Acked 1108 Read 6 Done New Done false
698 110 Acked 1109 Read 8 Done New Done false
699 111 Acked 1110 Read 9 Done New Done false
700 112 Acked 1111 Read 1 Done New Done false
701 113 Acked 1112 Write 2 Done New Done false
702 114 Acked 1113 Write 8 Done New Done false
703 115 Acked 1114 Read 9 Done New Done false
704 116 Acked 1115 Read 7 Done New Done false
705 117 Acked 1116 Write 9 Done New Done false
706 118 Acked 1117 Read 10 Done New Done false
707 119 Acked 1118 Read 2 Done New Done false
708 120 Acked 1119 Write 4 Done New Done false
709 121 Acked 1120 Write 2 Done New Done false
710 122 Acked 1121 Read 8 Done New Done false
711 123 Acked 1122 Read 2 Done New Done false
712 124 Acked 1123 Write 1 Done New Done false
713 125 Acked 1124 Write 6 Done New Done false
714 126 Acked 1125 Read 4 Done New Done false
715 127 Acked 1126 Write 9 Done New Done false
716 128 Acked 1127 Read 3 Done New Done false
717 129 Acked 1128 Write 8 Done New Done false
718 130 Acked 1129 Write 3 Done New Done false
719 131 Acked 1130 Write 4 Done New Done false
720 132 Acked 1131 Flush 0 Done New Done false
721 133 Acked 1132 Read 3 Done New Done false
722 134 Acked 1133 Write 6 Done New Done false
723 135 Acked 1134 Write 1 Done New Done false
724 136 Acked 1135 Write 10 Done New Done false
725 137 Acked 1136 Write 4 Done New Done false
726 138 Acked 1137 Write 1 Done New Done false
727 139 Acked 1138 Read 9 Done New Done false
728 140 Acked 1139 Read 3 Done New Done false
729 141 Acked 1140 Read 3 Done New Done false
730 142 Acked 1141 Read 7 Done New Done false
731 143 Acked 1142 Write 3 Done New Done false
732 144 Acked 1143 Write 6 Done New Done false
733 145 Acked 1144 Flush 0 Done New Done false
734 146 Acked 1145 Write 9 Done New Done false
735 147 Acked 1146 Write 2 Done New Done false
736 148 Acked 1147 Read 3 Done New Done false
737 149 Acked 1148 Read 4 Done New Done false
738 150 Acked 1149 Write 7 Done New Done false
739 151 Acked 1150 Read 8 Done New Done false
740 152 Acked 1151 Read 8 Done New Done false
741 153 Acked 1152 Read 9 Done New Done false
742 154 Acked 1153 Flush 0 Done New Done false
743 155 Acked 1154 Write 1 Done New Done false
744 156 Acked 1155 Read 10 Done New Done false
745 157 Acked 1156 Read 4 Done New Done false
746 158 Acked 1157 Read 7 Done New Done false
747 159 Acked 1158 Read 4 Done New Done false
748 160 Acked 1159 Read 10 Done New Done false
749 161 Acked 1160 Write 7 Done New Done false
750 162 Acked 1161 Write 6 Done New Done false
751 163 Acked 1162 Write 4 Done New Done false
752 164 Acked 1163 Write 2 Done New Done false
753 165 Acked 1164 Flush 0 Done New Done false
754 166 Acked 1165 Read 4 Done New Done false
755 167 Acked 1166 Read 2 Done New Done false
756 168 Acked 1167 Write 9 Done New Done false
757 169 Acked 1168 Write 10 Done New Done false
758 170 Acked 1169 Write 8 Done New Done false
759 171 Acked 1170 Write 2 Done New Done false
760 172 Acked 1171 Read 9 Done New Done false
761 173 Acked 1172 Read 7 Done New Done false
762 174 Acked 1173 Flush 0 Done New Done false
763 175 Acked 1174 Read 7 Done New Done false
764 176 Acked 1175 Read 10 Done New Done false
765 177 Acked 1176 Write 4 Done New Done false
766 178 Acked 1177 Flush 0 Done New Done false
767 179 Acked 1178 Write 6 Done New Done false
768 180 Acked 1179 Write 6 Done New Done false
769 181 Acked 1180 Read 4 Done New Done false
770 182 Acked 1181 Write 5 Done New Done false
771 183 Acked 1182 Read 2 Done New Done false
772 184 Acked 1183 Write 4 Done New Done false
773 185 Acked 1184 Read 8 Done New Done false
774 186 Acked 1185 Read 2 Done New Done false
775 187 Acked 1186 Write 2 Done New Done false
776 188 Acked 1187 Write 10 Done New Done false
777 189 Acked 1188 Write 1 Done New Done false
778 190 Acked 1189 Read 2 Done New Done false
779 191 Acked 1190 Read 6 Done New Done false
780 192 Acked 1191 Read 6 Done New Done false
781 193 Acked 1192 Read 8 Done New Done false
782 194 Acked 1193 Read 8 Done New Done false
783 195 Acked 1194 Read 1 Done New Done false
784 196 Acked 1195 Write 3 Done New Done false
785 197 Acked 1196 Read 4 Done New Done false
786 198 Acked 1197 Write 1 Done New Done false
787 199 Acked 1198 Write 1 Done New Done false
788 200 Acked 1199 Write 1 Done New Done false
789 201 Acked 1200 Read 10 Done New Done false
790 202 Acked 1201 Read 6 Done New Done false
791 203 Acked 1202 Write 2 Done New Done false
792 204 Acked 1203 Write 5 Done New Done false
793 205 Acked 1204 Read 9 Done New Done false
794 206 Acked 1205 Read 7 Done New Done false
795 207 Acked 1206 Read 8 Done New Done false
796 208 Acked 1207 Write 10 Done New Done false
797 209 Acked 1208 Write 9 Done New Done false
798 210 Acked 1209 Read 8 Done New Done false
799 211 Acked 1210 Read 9 Done New Done false
800 212 Acked 1211 Read 2 Done New Done false
801 213 Acked 1212 Write 2 Done New Done false
802 214 Acked 1213 Read 1 Done New Done false
803 215 Acked 1214 Write 4 Done New Done false
804 216 Acked 1215 Write 9 Done New Done false
805 217 Acked 1216 Read 3 Done New Done false
806 218 Acked 1217 Read 5 Done New Done false
807 219 Acked 1218 Read 9 Done New Done false
808 220 Acked 1219 Write 2 Done New Done false
809 221 Acked 1220 Read 7 Done New Done false
810 222 Acked 1221 Read 3 Done New Done false
811 223 Acked 1222 Read 10 Done New Done false
812 224 Acked 1223 Read 5 Done New Done false
813 225 Acked 1224 Write 6 Done New Done false
814 226 Acked 1225 Write 4 Done New Done false
815 227 Acked 1226 Write 4 Done New Done false
816 228 Acked 1227 Flush 0 Done New Done false
817 229 Acked 1228 Flush 0 Done New Done false
818 230 Acked 1229 Write 9 Done New Done false
819 231 Acked 1230 Read 9 Done New Done false
820 232 Acked 1231 Read 7 Done New Done false
821 233 Acked 1232 Read 5 Done New Done false
822 234 Acked 1233 Read 8 Done New Done false
823 235 Acked 1234 Read 7 Done New Done false
824 236 Acked 1235 Read 1 Done New Done false
825 237 Acked 1236 Flush 0 Done New Done false
826 238 Acked 1237 Flush 0 Done New Done false
827 239 Acked 1238 Read 3 Done New Done false
828 240 Acked 1239 Write 6 Done New Done false
829 241 Acked 1240 Write 8 Done New Done false
830 242 Acked 1241 Read 8 Done New Done false
831 243 Acked 1242 Read 1 Done New Done false
832 244 Acked 1243 Write 9 Done New Done false
833 245 Acked 1244 Write 7 Done New Done false
834 246 Acked 1245 Read 7 Done New Done false
835 247 Acked 1246 Write 7 Done New Done false
836 248 Acked 1247 Read 7 Done New Done false
837 249 Acked 1248 Read 3 Done New Done false
838 250 Acked 1249 Read 1 Done New Done false
839 251 Acked 1250 Read 4 Done New Done false
840 252 Acked 1251 Write 2 Done New Done false
841 253 Acked 1252 Write 4 Done New Done false
842 254 Acked 1253 Read 9 Done New Done false
843 255 Acked 1254 Read 5 Done New Done false
844 256 Acked 1255 Flush 0 Done New Done false
845 257 Acked 1256 Write 7 Done New Done false
846 258 Acked 1257 Read 10 Done New Done false
847 259 Acked 1258 Read 1 Done New Done false
848 260 Acked 1259 Write 10 Done New Done false
849 261 Acked 1260 Read 9 Done New Done false
850 262 Acked 1261 Write 1 Done New Done false
851 263 Acked 1262 Write 6 Done New Done false
852 264 Acked 1263 Read 4 Done New Done false
853 265 Acked 1264 Flush 0 Done New Done false
854 266 Acked 1265 Write 2 Done New Done false
855 267 Acked 1266 Write 9 Done New Done false
856 268 Acked 1267 Read 3 Done New Done false
857 269 Acked 1268 Read 10 Done New Done false
858 270 Acked 1269 Write 2 Done New Done false
859 271 Acked 1270 Write 1 Done New Done false
860 272 Acked 1271 Write 2 Done New Done false
861 273 Acked 1272 Read 7 Done New Done false
862 274 Acked 1273 Write 10 Done New Done false
863 275 Acked 1274 Read 1 Done New Done false
864 276 Acked 1275 Write 10 Done New Done false
865 277 Acked 1276 Read 3 Done New Done false
866 278 Acked 1277 Flush 0 Done New Done false
867 279 Acked 1278 Write 6 Done New Done false
868 280 Acked 1279 Read 7 Done New Done false
869 281 Acked 1280 Write 10 Done New Done false
870 282 Acked 1281 Write 10 Done New Done false
871 283 Acked 1282 Write 7 Done New Done false
872 284 Acked 1283 Read 3 Done New Done false
873 285 Acked 1284 Read 7 Done New Done false
874 286 Acked 1285 Write 6 Done New Done false
875 287 Acked 1286 Read 7 Done New Done false
876 288 Acked 1287 Write 7 Done New Done false
877 289 Acked 1288 Read 4 Done New Done false
878 290 Acked 1289 Write 10 Done New Done false
879 291 Acked 1290 Read 8 Done New Done false
880 292 Acked 1291 Read 7 Done New Done false
881 293 Acked 1292 Write 7 Done New Done false
882 294 Acked 1293 Write 5 Done New Done false
883 295 Acked 1294 Read 6 Done New Done false
884 296 Acked 1295 Write 4 Done New Done false
885 297 Acked 1296 Read 5 Done New Done false
886 298 Acked 1297 Read 9 Done New Done false
887 299 Acked 1298 Read 8 Done New Done false
888 300 Acked 1299 Flush 0 Done New Done false
889 STATES DS:0 DS:1 DS:2 TOTAL
890 New 0 300 0 300
891 Sent 0 0 0 0
892 Done 300 0 300 600
893 Skipped 0 0 0 0
894 Error 0 0 0 0
895 Last Flush: 1299 0 1299
896 Downstairs last five completed:
897 Upstairs last five completed: 300 299 298 297 296
898 CLIENT: Up:0 ds:300 act:2
8992024-02-15T02:57:39.569ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
9002024-02-15T02:57:39.569ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
9012024-02-15T02:57:39.569ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
9022024-02-15T02:57:39.569ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
9032024-02-15T02:57:39.569ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
9042024-02-15T02:57:39.569ZINFOcrucible: [1] client re-new 300 jobs since flush 0 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
9052024-02-15T02:57:39.569ZINFOcrucible: client task is exiting = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
906 [8820] got start action
907 [8820] I am going to restart
908 Starting downstairs at port 8820
909 Make output file at "/tmp/dsc/downstairs-8820.txt"
910 [8820][1] reports Starting
9112024-02-15T02:57:40.570ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
9122024-02-15T02:57:40.570ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
9132024-02-15T02:57:40.570ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
9142024-02-15T02:57:40.570ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
9152024-02-15T02:57:40.570ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
9162024-02-15T02:57:40.570ZINFOcrucible: [1] client re-new 300 jobs since flush 0 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
9172024-02-15T02:57:40.570ZINFOcrucible: client task is exiting = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
918 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1168)
919 [8820][1] reports Running
9202024-02-15T02:57:41.571ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
9212024-02-15T02:57:41.571ZINFOcrucible: ds_connection connected = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
9222024-02-15T02:57:41.573ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
9232024-02-15T02:57:41.573ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
9242024-02-15T02:57:41.573ZINFOcrucible: send last flush ID to this DS: 0 = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
9252024-02-15T02:57:41.573ZINFOcrucible: Replied this last flush ID: 0 = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
9262024-02-15T02:57:41.573ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
9272024-02-15T02:57:41.573ZINFOcrucible: [1] Transition from Offline to Replay = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
9282024-02-15T02:57:41.573ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
9292024-02-15T02:57:41.573ZINFOcrucible: [1] Transition from Replay to Active = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
930 ----------------------------------------------------------------
931 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
932 Downstairs last five completed: 1299 1298 1297 1296 1295
933 Upstairs last five completed: 300 299 298 297 296
934 CLIENT: Up:0 ds:0 act:3
935 Replay: All jobs finished, all DS active.
936 Main task has work to do, go find it
937 got dsc Stop(1)
938 stop 1
939 [8820] Got stop action so:false kr:false
940 Feb 15 02:57:42.907 INFO request completed, latency_us: 365, response_code: 204, uri: /stop/cid/1, method: POST, req_id: 15ed5c56-a74c-44e8-8b8b-f65b8e04e638, remote_addr: 127.0.0.1:54744, local_addr: 127.0.0.1:9998
9412024-02-15T02:57:42.908ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
9422024-02-15T02:57:42.908ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
9432024-02-15T02:57:42.908ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
944 Feb 15 02:57:42.908 INFO request completed, latency_us: 144, response_code:{"msg":"Gone missing, transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-02-15T02:57:42.908759416Z","hostname":"ip-10-150-1-67.us-west-2.compute.internal","pid":1167,"client":"1","":"downstairs","session_id":"4f5a63e5-aa00-4890-9f6b-96c2e069fdb2"}
945 200, uri: /state/cid/1, method: GET, req_id: 420a978f-dc4e-45e1-9da1-b9be2e18bdcf, remote_addr: 127.0.0.1:54744, local_addr: 127.0.0.1:9998
9462024-02-15T02:57:42.908ZINFOcrucible: [1] client re-new 0 jobs since flush 1299 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
9472024-02-15T02:57:42.908ZINFOcrucible: client task is exiting = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
948 [8820] Exited with: ExitStatus(unix_wait_status(9))
949 [8820][1] reports Exit
9502024-02-15T02:57:43.910ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
9512024-02-15T02:57:43.910ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
9522024-02-15T02:57:43.910ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
9532024-02-15T02:57:43.910ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
9542024-02-15T02:57:43.910ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
9552024-02-15T02:57:43.910ZINFOcrucible: [1] client re-new 0 jobs since flush 1299 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
9562024-02-15T02:57:43.910ZINFOcrucible: client task is exiting = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
9572024-02-15T02:57:44.912ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
9582024-02-15T02:57:44.912ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
9592024-02-15T02:57:44.912ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
9602024-02-15T02:57:44.912ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
9612024-02-15T02:57:44.912ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
9622024-02-15T02:57:44.912ZINFOcrucible: [1] client re-new 0 jobs since flush 1299 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
9632024-02-15T02:57:44.912ZINFOcrucible: client task is exiting = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
9642024-02-15T02:57:45.913ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
9652024-02-15T02:57:45.913ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
9662024-02-15T02:57:45.913ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
9672024-02-15T02:57:45.913ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
9682024-02-15T02:57:45.913ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
9692024-02-15T02:57:45.913ZINFOcrucible: [1] client re-new 0 jobs since flush 1299 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
9702024-02-15T02:57:45.913ZINFOcrucible: client task is exiting = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
971 Feb 15 02:57:46.910 INFO request completed, latency_us: 199, response_code: 200, uri: /state/cid/1, method: GET, req_id: 11d31db1-5bd0-4832-9d7e-7ff23577c618, remote_addr: 127.0.0.1:54744, local_addr: 127.0.0.1:9998
972 001/300 Write block 610 len 36864 data: 2 2 2 2 2 2 2 2 2
973 002/300 Write block 1957 len 16384 data: 2 2 2 2
974 003/300 Read block 1309 len 4096
975 004/300 Read block 2493 len 12288
976 005/300 Read block 1235 len 36864
9772024-02-15T02:57:46.914ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
9782024-02-15T02:57:46.914ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
9792024-02-15T02:57:46.914ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
9802024-02-15T02:57:46.914ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
9812024-02-15T02:57:46.914ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
9822024-02-15T02:57:46.914ZINFOcrucible: [1] client re-new 5 jobs since flush 1299 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
9832024-02-15T02:57:46.914ZINFOcrucible: client task is exiting = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
984 006/300 Write block 3081 len 40960 data: 2 2 2 2 2 2 2 2 2 2
985 007/300 Read block 264 len 36864
986 008/300 Write block 3730 len 32768 data: 3 2 2 2 2 2 2 2
987 009/300 Read block 291 len 20480
988 010/300 Read block 1017 len 20480
989 011/300 Write block 3519 len 20480 data: 2 2 2 2 2
990 012/300 Write block 4953 len 24576 data: 2 2 2 2 2 2
991 013/300 Read block 2900 len 40960
992 014/300 Read block 1342 len 4096
993 015/300 Write block 2661 len 4096 data: 2
994 016/300 Read block 2622 len 36864
995 017/300 Read block 3178 len 24576
996 018/300 Read block 3501 len 36864
997 019/300 Read block 1187 len 8192
998 020/300 Read block 3824 len 36864
999 021/300 Read block 4333 len 36864
1000 022/300 Read block 2992 len 4096
1001 023/300 Read block 3083 len 20480
1002 024/300 Read block 1118 len 28672
1003 025/300 Flush
1004 026/300 Write block 1052 len 8192 data: 2 2
1005 027/300 Read block 4767 len 32768
1006 028/300 Read block 166 len 4096
1007 029/300 Write block 2052 len 36864 data: 2 2 2 2 2 2 2 2 2
1008 030/300 Read block 3583 len 4096
1009 031/300 Write block 1783 len 24576 data: 2 2 2 2 2 2
1010 032/300 Flush
1011 033/300 Flush
1012 034/300 Write block 4863 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1013 035/300 Read block 186 len 8192
1014 036/300 Read block 2682 len 36864
1015 037/300 Write block 3490 len 12288 data: 3 3 3
1016 038/300 Write block 3667 len 16384 data: 2 2 2 2
1017 039/300 Read block 4347 len 16384
1018 040/300 Write block 3196 len 8192 data: 2 2
1019 041/300 Write block 1240 len 24576 data: 2 2 2 2 3 3
1020 042/300 Write block 3970 len 8192 data: 2 2
1021 043/300 Read block 3700 len 12288
1022 044/300 Write block 4213 len 28672 data: 2 2 2 2 2 2 2
1023 045/300 Read block 2083 len 24576
1024 046/300 Read block 643 len 20480
1025 047/300 Read block 2637 len 28672
1026 048/300 Read block 2409 len 4096
1027 049/300 Read block 2756 len 36864
1028 050/300 Write block 2105 len 4096 data: 2
1029 051/300 Write block 3690 len 20480 data: 2 2 2 2 3
1030 052/300 Read block 2788 len 16384
1031 053/300 Read block 2709 len 8192
1032 054/300 Read block 2104 len 20480
1033 055/300 Write block 4950 len 40960 data: 2 2 2 3 3 3 3 3 3 2
1034 056/300 Read block 2703 len 20480
1035 057/300 Read block 1061 len 40960
1036 058/300 Write block 3533 len 8192 data: 3 3
1037 059/300 Read block 3632 len 32768
1038 060/300 Write block 2081 len 20480 data: 2 2 3 3 3
1039 061/300 Write block 219 len 20480 data: 2 2 2 3 3
1040 062/300 Read block 4011 len 4096
1041 063/300 Read block 4068 len 40960
1042 064/300 Write block 4904 len 28672 data: 2 2 3 3 3 3 3
1043 065/300 Read block 2923 len 40960
1044 066/300 Read block 4302 len 16384
1045 067/300 Read block 4960 len 40960
1046 068/300 Read block 2699 len 32768
1047 069/300 Write block 4834 len 20480 data: 2 2 2 2 3
1048 070/300 Write block 1131 len 8192 data: 2 2
1049 071/300 Write block 1828 len 24576 data: 2 2 2 2 2 2
1050 072/300 Read block 4173 len 32768
1051 073/300 Read block 1565 len 8192
1052 074/300 Read block 632 len 16384
1053 075/300 Read block 2577 len 40960
1054 076/300 Flush
1055 077/300 Write block 3593 len 28672 data: 2 2 2 2 2 2 2
1056 078/300 Flush
1057 079/300 Read block 819 len 28672
1058 080/300 Write block 1536 len 16384 data: 2 2 2 2
1059 081/300 Read block 545 len 28672
1060 082/300 Read block 4306 len 20480
1061 083/300 Write block 1939 len 4096 data: 2
1062 084/300 Write block 4965 len 4096 data: 2
1063 085/300 Read block 4003 len 4096
1064 086/300 Write block 617 len 12288 data: 3 3 2
1065 087/300 Read block 578 len 8192
1066 088/300 Read block 2382 len 36864
1067 089/300 Read block 4097 len 4096
1068 090/300 Write block 427 len 8192 data: 2 2
1069 091/300 Write block 529 len 36864 data: 2 2 3 3 3 3 3 3 3
1070 092/300 Read block 2601 len 32768
1071 093/300 Read block 4980 len 8192
1072 094/300 Write block 4941 len 40960 data: 2 2 2 2 2 2 2 2 2 3
1073 095/300 Read block 2418 len 16384
1074 096/300 Read block 1891 len 32768
1075 097/300 Read block 3853 len 20480
1076 098/300 Write block 982 len 40960 data: 2 2 2 2 2 2 3 3 2 2
1077 099/300 Write block 1554 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1078 100/300 Read block 2444 len 4096
1079 101/300 Read block 2079 len 24576
1080 102/300 Write block 469 len 8192 data: 2 2
1081 103/300 Flush
1082 104/300 Read block 387 len 28672
1083 105/300 Read block 3763 len 8192
1084 106/300 Read block 1513 len 40960
1085 107/300 Read block 4927 len 24576
1086 108/300 Write block 3149 len 12288 data: 2 2 2
1087 109/300 Read block 1465 len 24576
1088 110/300 Read block 924 len 36864
1089 111/300 Write block 2249 len 36864 data: 2 2 2 2 2 2 2 2 2
1090 112/300 Flush
1091 113/300 Write block 558 len 16384 data: 2 2 2 2
1092 114/300 Read block 377 len 16384
1093 115/300 Flush
1094 116/300 Flush
1095 117/300 Write block 956 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1096 118/300 Write block 2237 len 32768 data: 2 2 2 2 2 2 2 2
1097 119/300 Read block 4268 len 32768
1098 120/300 Write block 3158 len 40960 data: 3 3 3 3 3 2 2 2 2 2
1099 121/300 Write block 4948 len 8192 data: 3 3
1100 122/300 Read block 4288 len 36864
1101 123/300 Write block 4012 len 40960 data: 2 2 2 2 2 2 2 3 3 3
1102 124/300 Write block 1290 len 16384 data: 2 2 2 2
1103 125/300 Flush
1104 126/300 Write block 4040 len 24576 data: 2 2 2 2 2 2
1105 127/300 Read block 2152 len 8192
1106 128/300 Write block 3161 len 36864 data: 4 4 3 3 3 3 3 2 2
1107 129/300 Read block 3519 len 28672
1108 130/300 Flush
1109 131/300 Write block 1081 len 24576 data: 2 2 2 2 2 2
1110 132/300 Read block 1092 len 36864
1111 133/300 Flush
1112 134/300 Write block 2913 len 16384 data: 2 2 2 2
1113 135/300 Read block 4738 len 36864
1114 136/300 Read block 3375 len 28672
1115 137/300 Read block 1943 len 32768
1116 138/300 Read block 3066 len 16384
1117 139/300 Write block 4111 len 32768 data: 2 2 2 2 2 2 2 2
1118 140/300 Read block 1078 len 20480
1119 141/300 Read block 605 len 4096
1120 142/300 Write block 437 len 24576 data: 2 2 2 2 2 2
1121 143/300 Flush
1122 144/300 Read block 3779 len 40960
1123 145/300 Write block 3150 len 4096 data: 3
1124 146/300 Flush
1125 147/300 Read block 2478 len 32768
1126 148/300 Flush
1127 149/300 Flush
1128 150/300 Write block 1514 len 16384 data: 2 2 2 2
1129 151/300 Write block 2880 len 36864 data: 2 2 2 2 2 2 2 2 2
1130 152/300 Read block 1838 len 12288
1131 153/300 Write block 732 len 32768 data: 2 2 3 3 3 3 3 3
1132 154/300 Write block 176 len 8192 data: 2 2
1133 155/300 Read block 3678 len 16384
1134 156/300 Read block 2089 len 28672
1135 157/300 Read block 2673 len 20480
1136 158/300 Read block 4931 len 12288
1137 159/300 Write block 4136 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1138 160/300 Read block 4958 len 12288
1139 161/300 Read block 1471 len 36864
1140 162/300 Read block 1426 len 20480
1141 163/300 Read block 592 len 16384
1142 164/300 Read block 4907 len 20480
1143 165/300 Read block 4069 len 36864
1144 166/300 Flush
1145 167/300 Flush
1146 168/300 Flush
1147 169/300 Write block 628 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1148 170/300 Read block 396 len 8192
1149 171/300 Read block 3441 len 36864
1150 172/300 Write block 1020 len 20480 data: 2 2 2 2 2
1151 173/300 Write block 1059 len 4096 data: 2
1152 174/300 Read block 3873 len 28672
1153 175/300 Flush
1154 176/300 Read block 337 len 20480
1155 177/300 Read block 517 len 16384
1156 178/300 Write block 3930 len 8192 data: 2 2
1157 179/300 Write block 4809 len 20480 data: 2 2 2 2 2
1158 180/300 Read block 2641 len 40960
1159 181/300 Flush
1160 182/300 Write block 2622 len 36864 data: 2 2 2 2 2 2 2 2 2
1161 183/300 Read block 4027 len 12288
1162 184/300 Read block 1241 len 32768
1163 185/300 Write block 1437 len 24576 data: 2 2 2 2 2 2
1164 186/300 Read block 1694 len 16384
1165 187/300 Read block 1559 len 16384
1166 188/300 Read block 4554 len 40960
1167 189/300 Read block 4661 len 8192
1168 190/300 Read block 838 len 8192
1169 191/300 Read block 4187 len 20480
1170 192/300 Read block 3384 len 12288
1171 193/300 Flush
1172 194/300 Read block 1661 len 4096
1173 195/300 Write block 615 len 28672 data: 3 3 4 4 3 2 2
1174 196/300 Write block 2850 len 36864 data: 3 2 2 2 2 2 2 2 2
1175 197/300 Write block 3863 len 36864 data: 2 2 2 2 2 2 2 2 2
1176 198/300 Write block 755 len 12288 data: 2 2 2
1177 199/300 Write block 2213 len 32768 data: 2 2 2 2 2 2 2 2
1178 200/300 Read block 1423 len 40960
1179 201/300 Write block 2152 len 32768 data: 2 2 2 2 2 2 3 3
1180 202/300 Write block 4858 len 32768 data: 2 2 2 2 2 3 3 3
1181 203/300 Write block 402 len 8192 data: 2 2
1182 204/300 Read block 4770 len 32768
1183 205/300 Write block 1012 len 12288 data: 3 3 3
1184 206/300 Flush
1185 207/300 Read block 1750 len 32768
1186 208/300 Read block 3839 len 40960
1187 209/300 Write block 3158 len 40960 data: 4 4 4 5 5 4 4 4 4 4
1188 210/300 Read block 2166 len 28672
1189 211/300 Read block 3442 len 36864
1190 212/300 Read block 3752 len 16384
1191 213/300 Read block 2441 len 24576
1192 214/300 Read block 1252 len 12288
1193 215/300 Read block 580 len 16384
1194 216/300 Flush
1195 217/300 Read block 4628 len 12288
1196 218/300 Write block 2072 len 16384 data: 2 2 2 2
1197 219/300 Read block 3897 len 28672
1198 220/300 Write block 601 len 28672 data: 2 2 2 2 2 2 2
1199 221/300 Write block 1660 len 4096 data: 2
1200 222/300 Read block 1602 len 40960
1201 223/300 Write block 11 len 32768 data: 2 2 2 2 2 2 2 2
1202 224/300 Write block 1316 len 36864 data: 2 2 2 2 2 2 2 2 2
1203 225/300 Read block 2402 len 12288
1204 226/300 Read block 3980 len 4096
1205 227/300 Read block 2337 len 36864
1206 228/300 Read block 2437 len 40960
1207 229/300 Read block 938 len 36864
1208 230/300 Write block 2073 len 20480 data: 3 3 3 2 2
1209 231/300 Read block 3724 len 20480
1210 232/300 Read block 1262 len 16384
1211 233/300 Read block 653 len 32768
1212 234/300 Flush
1213 235/300 Read block 1747 len 8192
1214 236/300 Read block 3944 len 4096
1215 237/300 Write block 957 len 32768 data: 3 3 3 3 3 3 3 3
1216 238/300 Write block 720 len 36864 data: 2 2 2 2 2 2 2 2 2
1217 239/300 Read block 205 len 40960
1218 240/300 Read block 733 len 24576
1219 241/300 Read block 3399 len 40960
1220 242/300 Read block 4191 len 40960
1221 243/300 Write block 3009 len 12288 data: 2 2 2
1222 244/300 Flush
1223 245/300 Read block 4042 len 28672
1224 246/300 Read block 4480 len 32768
1225 247/300 Read block 3384 len 40960
1226 248/300 Flush
1227 249/300 Read block 3691 len 28672
1228 250/300 Read block 4543 len 40960
1229 251/300 Write block 1559 len 28672 data: 3 3 3 3 3 2 2
1230 252/300 Write block 3954 len 16384 data: 3 3 3 2
1231 253/300 Write block 612 len 36864 data: 3 3 3 4 4 5 5 4 3
1232 254/300 Flush
1233 255/300 Write block 4792 len 4096 data: 2
1234 256/300 Write block 3089 len 4096 data: 3
1235 257/300 Read block 1002 len 12288
1236 258/300 Write block 468 len 12288 data: 2 3 3
1237 259/300 Read block 1795 len 8192
1238 260/300 Write block 4402 len 20480 data: 2 2 2 2 2
1239 261/300 Read block 1371 len 4096
1240 262/300 Write block 3533 len 32768 data: 4 4 3 3 3 3 3 2
1241 263/300 Write block 282 len 20480 data: 2 2 2 2 2
1242 264/300 Write block 1367 len 12288 data: 2 2 2
1243 265/300 Write block 4134 len 28672 data: 2 2 3 3 3 3 3
1244 266/300 Write block 3695 len 28672 data: 3 3 3 3 2 2 2
1245 267/300 Write block 1149 len 24576 data: 3 4 4 4 4 3
1246 268/300 Read block 4235 len 40960
1247 269/300 Write block 3278 len 8192 data: 2 2
1248 270/300 Read block 1369 len 20480
1249 271/300 Read block 1903 len 16384
1250 272/300 Read block 1409 len 24576
1251 273/300 Read block 3166 len 8192
1252 274/300 Read block 792 len 16384
1253 275/300 Write block 3439 len 20480 data: 2 2 2 2 2
1254 276/300 Write block 2058 len 20480 data: 3 3 3 2 2
1255 277/300 Read block 3112 len 28672
1256 278/300 Write block 3395 len 36864 data: 2 2 2 2 2 2 2 2 2
1257 279/300 Read block 4834 len 20480
1258 280/300 Read block 2003 len 24576
1259 281/300 Write block 1092 len 20480 data: 2 2 2 2 2
1260 282/300 Write block 580 len 24576 data: 2 2 2 2 2 2
1261 283/300 Read block 2007 len 12288
1262 284/300 Flush
1263 285/300 Read block 727 len 8192
1264 286/300 Read block 4556 len 16384
1265 287/300 Read block 1634 len 20480
1266 288/300 Read block 1055 len 16384
1267 289/300 Read block 3263 len 8192
1268 290/300 Read block 3897 len 40960
1269 291/300 Read block 794 len 4096
1270 292/300 Read block 663 len 24576
1271 293/300 Read block 3489 len 28672
1272 294/300 Write block 4203 len 16384 data: 2 2 2 2
1273 295/300 Write block 3751 len 12288 data: 2 2 2
1274 296/300 Read block 4753 len 12288
1275 297/300 Read block 74 len 20480
1276 298/300 Read block 4375 len 16384
1277 299/300 Read block 2802 len 36864
1278 300/300 Write block 1392 len 20480 data: 2 2 2 2 2
1279 Main task has work to do, go find it
1280 got dsc Start(1)
1281 start 1
1282 Feb 15 02:57:47.260 INFO request completed, latency_us: 199, response_code: 204, uri: /start/cid/1, method: POST, req_id: 224912e7-7502-4dcf-983e-507182b5fe48, remote_addr: 127.0.0.1:54744, local_addr: 127.0.0.1:9998
1283 Replay: started 1, returned:Ok(())
1284 ----------------------------------------------------------------
1285 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
1286 States: Active Offline Active
1287 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
1288 301 Acked 1300 Write 9 Done New Done true
1289 302 Acked 1301 Write 4 Done New Done true
1290 303 Acked 1302 Read 1 Done New Done true
1291 304 Acked 1303 Read 3 Done New Done true
1292 305 Acked 1304 Read 9 Done New Done true
1293 306 Acked 1305 Write 10 Done New Done false
1294 307 Acked 1306 Read 9 Done New Done false
1295 308 Acked 1307 Write 8 Done New Done false
1296 309 Acked 1308 Read 5 Done New Done false
1297 310 Acked 1309 Read 5 Done New Done false
1298 311 Acked 1310 Write 5 Done New Done false
1299 312 Acked 1311 Write 6 Done New Done false
1300 313 Acked 1312 Read 10 Done New Done false
1301 314 Acked 1313 Read 1 Done New Done false
1302 315 Acked 1314 Write 1 Done New Done false
1303 316 Acked 1315 Read 9 Done New Done false
1304 317 Acked 1316 Read 6 Done New Done false
1305 318 Acked 1317 Read 9 Done New Done false
1306 319 Acked 1318 Read 2 Done New Done false
1307 320 Acked 1319 Read 9 Done New Done false
1308 321 Acked 1320 Read 9 Done New Done false
1309 322 Acked 1321 Read 1 Done New Done false
1310 323 Acked 1322 Read 5 Done New Done false
1311 324 Acked 1323 Read 7 Done New Done false
1312 325 Acked 1324 Flush 0 Done New Done false
1313 326 Acked 1325 Write 2 Done New Done false
1314 327 Acked 1326 Read 8 Done New Done false
1315 328 Acked 1327 Read 1 Done New Done false
1316 329 Acked 1328 Write 9 Done New Done false
1317 330 Acked 1329 Read 1 Done New Done false
1318 331 Acked 1330 Write 6 Done New Done false
1319 332 Acked 1331 Flush 0 Done New Done false
1320 333 Acked 1332 Flush 0 Done New Done false
1321 334 Acked 1333 Write 10 Done New Done false
1322 335 Acked 1334 Read 2 Done New Done false
1323 336 Acked 1335 Read 9 Done New Done false
1324 337 Acked 1336 Write 3 Done New Done false
1325 338 Acked 1337 Write 4 Done New Done false
1326 339 Acked 1338 Read 4 Done New Done false
1327 340 Acked 1339 Write 2 Done New Done false
1328 341 Acked 1340 Write 6 Done New Done false
1329 342 Acked 1341 Write 2 Done New Done false
1330 343 Acked 1342 Read 3 Done New Done false
1331 344 Acked 1343 Write 7 Done New Done false
1332 345 Acked 1344 Read 6 Done New Done false
1333 346 Acked 1345 Read 5 Done New Done false
1334 347 Acked 1346 Read 7 Done New Done false
1335 348 Acked 1347 Read 1 Done New Done false
1336 349 Acked 1348 Read 9 Done New Done false
1337 350 Acked 1349 Write 1 Done New Done false
1338 351 Acked 1350 Write 5 Done New Done false
1339 352 Acked 1351 Read 4 Done New Done false
1340 353 Acked 1352 Read 2 Done New Done false
1341 354 Acked 1353 Read 5 Done New Done false
1342 355 Acked 1354 Write 10 Done New Done false
1343 356 Acked 1355 Read 5 Done New Done false
1344 357 Acked 1356 Read 10 Done New Done false
1345 358 Acked 1357 Write 2 Done New Done false
1346 359 Acked 1358 Read 8 Done New Done false
1347 360 Acked 1359 Write 5 Done New Done false
1348 361 Acked 1360 Write 5 Done New Done false
1349 362 Acked 1361 Read 1 Done New Done false
1350 363 Acked 1362 Read 10 Done New Done false
1351 364 Acked 1363 Write 7 Done New Done false
1352 365 Acked 1364 Read 10 Done New Done false
1353 366 Acked 1365 Read 4 Done New Done false
1354 367 Acked 1366 Read 10 Done New Done false
1355 368 Acked 1367 Read 8 Done New Done false
1356 369 Acked 1368 Write 5 Done New Done false
1357 370 Acked 1369 Write 2 Done New Done false
1358 371 Acked 1370 Write 6 Done New Done false
1359 372 Acked 1371 Read 8 Done New Done false
1360 373 Acked 1372 Read 2 Done New Done false
1361 374 Acked 1373 Read 4 Done New Done false
1362 375 Acked 1374 Read 10 Done New Done false
1363 376 Acked 1375 Flush 0 Done New Done false
1364 377 Acked 1376 Write 7 Done New Done false
1365 378 Acked 1377 Flush 0 Done New Done false
1366 379 Acked 1378 Read 7 Done New Done false
1367 380 Acked 1379 Write 4 Done New Done false
1368 381 Acked 1380 Read 7 Done New Done false
1369 382 Acked 1381 Read 5 Done New Done false
1370 383 Acked 1382 Write 1 Done New Done false
1371 384 Acked 1383 Write 1 Done New Done false
1372 385 Acked 1384 Read 1 Done New Done false
1373 386 Acked 1385 Write 3 Done New Done false
1374 387 Acked 1386 Read 2 Done New Done false
1375 388 Acked 1387 Read 9 Done New Done false
1376 389 Acked 1388 Read 1 Done New Done false
1377 390 Acked 1389 Write 2 Done New Done false
1378 391 Acked 1390 Write 9 Done New Done false
1379 392 Acked 1391 Read 8 Done New Done false
1380 393 Acked 1392 Read 2 Done New Done false
1381 394 Acked 1393 Write 10 Done New Done false
1382 395 Acked 1394 Read 4 Done New Done false
1383 396 Acked 1395 Read 8 Done New Done false
1384 397 Acked 1396 Read 5 Done New Done false
1385 398 Acked 1397 Write 10 Done New Done false
1386 399 Acked 1398 Write 10 Done New Done false
1387 400 Acked 1399 Read 1 Done New Done false
1388 401 Acked 1400 Read 6 Done New Done false
1389 402 Acked 1401 Write 2 Done New Done false
1390 403 Acked 1402 Flush 0 Done New Done false
1391 404 Acked 1403 Read 7 Done New Done false
1392 405 Acked 1404 Read 2 Done New Done false
1393 406 Acked 1405 Read 10 Done New Done false
1394 407 Acked 1406 Read 6 Done New Done false
1395 408 Acked 1407 Write 3 Done New Done false
1396 409 Acked 1408 Read 6 Done New Done false
1397 410 Acked 1409 Read 9 Done New Done false
1398 411 Acked 1410 Flush 0 Done New Done false
1399 412 Acked 1411 Write 9 Done New Done false
1400 413 Acked 1412 Flush 0 Done New Done false
1401 414 Acked 1413 Write 4 Done New Done false
1402 415 Acked 1414 Read 4 Done New Done false
1403 416 Acked 1415 Flush 0 Done New Done false
1404 417 Acked 1416 Flush 0 Done New Done false
1405 418 Acked 1417 Write 10 Done New Done false
1406 419 Acked 1418 Write 8 Done New Done false
1407 420 Acked 1419 Read 8 Done New Done false
1408 421 Acked 1420 Write 10 Done New Done false
1409 422 Acked 1421 Write 2 Done New Done false
1410 423 Acked 1422 Read 9 Done New Done false
1411 424 Acked 1423 Write 10 Done New Done false
1412 425 Acked 1424 Write 4 Done New Done false
1413 426 Acked 1425 Flush 0 Done New Done false
1414 427 Acked 1426 Write 6 Done New Done false
1415 428 Acked 1427 Read 2 Done New Done false
1416 429 Acked 1428 Write 9 Done New Done false
1417 430 Acked 1429 Read 7 Done New Done false
1418 431 Acked 1430 Flush 0 Done New Done false
1419 432 Acked 1431 Write 6 Done New Done false
1420 433 Acked 1432 Read 9 Done New Done false
1421 434 Acked 1433 Flush 0 Done New Done false
1422 435 Acked 1434 Write 4 Done New Done false
1423 436 Acked 1435 Read 9 Done New Done false
1424 437 Acked 1436 Read 7 Done New Done false
1425 438 Acked 1437 Read 8 Done New Done false
1426 439 Acked 1438 Read 4 Done New Done false
1427 440 Acked 1439 Write 8 Done New Done false
1428 441 Acked 1440 Read 5 Done New Done false
1429 442 Acked 1441 Read 1 Done New Done false
1430 443 Acked 1442 Write 6 Done New Done false
1431 444 Acked 1443 Flush 0 Done New Done false
1432 445 Acked 1444 Read 10 Done New Done false
1433 446 Acked 1445 Write 1 Done New Done false
1434 447 Acked 1446 Flush 0 Done New Done false
1435 448 Acked 1447 Read 8 Done New Done false
1436 449 Acked 1448 Flush 0 Done New Done false
1437 450 Acked 1449 Flush 0 Done New Done false
1438 451 Acked 1450 Write 4 Done New Done false
1439 452 Acked 1451 Write 9 Done New Done false
1440 453 Acked 1452 Read 3 Done New Done false
1441 454 Acked 1453 Write 8 Done New Done false
1442 455 Acked 1454 Write 2 Done New Done false
1443 456 Acked 1455 Read 4 Done New Done false
1444 457 Acked 1456 Read 7 Done New Done false
1445 458 Acked 1457 Read 5 Done New Done false
1446 459 Acked 1458 Read 3 Done New Done false
1447 460 Acked 1459 Write 10 Done New Done false
1448 461 Acked 1460 Read 3 Done New Done false
1449 462 Acked 1461 Read 9 Done New Done false
1450 463 Acked 1462 Read 5 Done New Done false
1451 464 Acked 1463 Read 4 Done New Done false
1452 465 Acked 1464 Read 5 Done New Done false
1453 466 Acked 1465 Read 9 Done New Done false
1454 467 Acked 1466 Flush 0 Done New Done false
1455 468 Acked 1467 Flush 0 Done New Done false
1456 469 Acked 1468 Flush 0 Done New Done false
1457 470 Acked 1469 Write 10 Done New Done false
1458 471 Acked 1470 Read 2 Done New Done false
1459 472 Acked 1471 Read 9 Done New Done false
1460 473 Acked 1472 Write 5 Done New Done false
1461 474 Acked 1473 Write 1 Done New Done false
1462 475 Acked 1474 Read 7 Done New Done false
1463 476 Acked 1475 Flush 0 Done New Done false
1464 477 Acked 1476 Read 5 Done New Done false
1465 478 Acked 1477 Read 4 Done New Done false
1466 479 Acked 1478 Write 2 Done New Done false
1467 480 Acked 1479 Write 5 Done New Done false
1468 481 Acked 1480 Read 10 Done New Done false
1469 482 Acked 1481 Flush 0 Done New Done false
1470 483 Acked 1482 Write 9 Done New Done false
1471 484 Acked 1483 Read 3 Done New Done false
1472 485 Acked 1484 Read 8 Done New Done false
1473 486 Acked 1485 Write 6 Done New Done false
1474 487 Acked 1486 Read 4 Done New Done false
1475 488 Acked 1487 Read 4 Done New Done false
1476 489 Acked 1488 Read 10 Done New Done false
1477 490 Acked 1489 Read 2 Done New Done false
1478 491 Acked 1490 Read 2 Done New Done false
1479 492 Acked 1491 Read 5 Done New Done false
1480 493 Acked 1492 Read 3 Done New Done false
1481 494 Acked 1493 Flush 0 Done New Done false
1482 495 Acked 1494 Read 1 Done New Done false
1483 496 Acked 1495 Write 7 Done New Done false
1484 497 Acked 1496 Write 9 Done New Done false
1485 498 Acked 1497 Write 9 Done New Done false
1486 499 Acked 1498 Write 3 Done New Done false
1487 500 Acked 1499 Write 8 Done New Done false
1488 501 Acked 1500 Read 10 Done New Done false
1489 502 Acked 1501 Write 8 Done New Done false
1490 503 Acked 1502 Write 8 Done New Done false
1491 504 Acked 1503 Write 2 Done New Done false
1492 505 Acked 1504 Read 8 Done New Done false
1493 506 Acked 1505 Write 3 Done New Done false
1494 507 Acked 1506 Flush 0 Done New Done false
1495 508 Acked 1507 Read 8 Done New Done false
1496 509 Acked 1508 Read 10 Done New Done false
1497 510 Acked 1509 Write 10 Done New Done false
1498 511 Acked 1510 Read 7 Done New Done false
1499 512 Acked 1511 Read 9 Done New Done false
1500 513 Acked 1512 Read 4 Done New Done false
1501 514 Acked 1513 Read 6 Done New Done false
1502 515 Acked 1514 Read 3 Done New Done false
1503 516 Acked 1515 Read 4 Done New Done false
1504 517 Acked 1516 Flush 0 Done New Done false
1505 518 Acked 1517 Read 3 Done New Done false
1506 519 Acked 1518 Write 4 Done New Done false
1507 520 Acked 1519 Read 7 Done New Done false
1508 521 Acked 1520 Write 7 Done New Done false
1509 522 Acked 1521 Write 1 Done New Done false
1510 523 Acked 1522 Read 10 Done New Done false
1511 524 Acked 1523 Write 8 Done New Done false
1512 525 Acked 1524 Write 9 Done New Done false
1513 526 Acked 1525 Read 3 Done New Done false
1514 527 Acked 1526 Read 1 Done New Done false
1515 528 Acked 1527 Read 9 Done New Done false
1516 529 Acked 1528 Read 10 Done New Done false
1517 530 Acked 1529 Read 9 Done New Done false
1518 531 Acked 1530 Write 5 Done New Done false
1519 532 Acked 1531 Read 5 Done New Done false
1520 533 Acked 1532 Read 4 Done New Done false
1521 534 Acked 1533 Read 8 Done New Done false
1522 535 Acked 1534 Flush 0 Done New Done false
1523 536 Acked 1535 Read 2 Done New Done false
1524 537 Acked 1536 Read 1 Done New Done false
1525 538 Acked 1537 Write 8 Done New Done false
1526 539 Acked 1538 Write 9 Done New Done false
1527 540 Acked 1539 Read 10 Done New Done false
1528 541 Acked 1540 Read 6 Done New Done false
1529 542 Acked 1541 Read 10 Done New Done false
1530 543 Acked 1542 Read 10 Done New Done false
1531 544 Acked 1543 Write 3 Done New Done false
1532 545 Acked 1544 Flush 0 Done New Done false
1533 546 Acked 1545 Read 7 Done New Done false
1534 547 Acked 1546 Read 8 Done New Done false
1535 548 Acked 1547 Read 10 Done New Done false
1536 549 Acked 1548 Flush 0 Done New Done false
1537 550 Acked 1549 Read 7 Done New Done false
1538 551 Acked 1550 Read 10 Done New Done false
1539 552 Acked 1551 Write 7 Done New Done false
1540 553 Acked 1552 Write 4 Done New Done false
1541 554 Acked 1553 Write 9 Done New Done false
1542 555 Acked 1554 Flush 0 Done New Done false
1543 556 Acked 1555 Write 1 Done New Done false
1544 557 Acked 1556 Write 1 Done New Done false
1545 558 Acked 1557 Read 3 Done New Done false
1546 559 Acked 1558 Write 3 Done New Done false
1547 560 Acked 1559 Read 2 Done New Done false
1548 561 Acked 1560 Write 5 Done New Done false
1549 562 Acked 1561 Read 1 Done New Done false
1550 563 Acked 1562 Write 8 Done New Done false
1551 564 Acked 1563 Write 5 Done New Done false
1552 565 Acked 1564 Write 3 Done New Done false
1553 566 Acked 1565 Write 7 Done New Done false
1554 567 Acked 1566 Write 7 Done New Done false
1555 568 Acked 1567 Write 6 Done New Done false
1556 569 Acked 1568 Read 10 Done New Done false
1557 570 Acked 1569 Write 2 Done New Done false
1558 571 Acked 1570 Read 5 Done New Done false
1559 572 Acked 1571 Read 4 Done New Done false
1560 573 Acked 1572 Read 6 Done New Done false
1561 574 Acked 1573 Read 2 Done New Done false
1562 575 Acked 1574 Read 4 Done New Done false
1563 576 Acked 1575 Write 5 Done New Done false
1564 577 Acked 1576 Write 5 Done New Done false
1565 578 Acked 1577 Read 7 Done New Done false
1566 579 Acked 1578 Write 9 Done New Done false
1567 580 Acked 1579 Read 5 Done New Done false
1568 581 Acked 1580 Read 6 Done New Done false
1569 582 Acked 1581 Write 5 Done New Done false
1570 583 Acked 1582 Write 6 Done New Done false
1571 584 Acked 1583 Read 3 Done New Done false
1572 585 Acked 1584 Flush 0 Done New Done false
1573 586 Acked 1585 Read 2 Done New Done false
1574 587 Acked 1586 Read 4 Done New Done false
1575 588 Acked 1587 Read 5 Done New Done false
1576 589 Acked 1588 Read 4 Done New Done false
1577 590 Acked 1589 Read 2 Done New Done false
1578 591 Acked 1590 Read 10 Done New Done false
1579 592 Acked 1591 Read 1 Done New Done false
1580 593 Acked 1592 Read 6 Done New Done false
1581 594 Acked 1593 Read 7 Done New Done false
1582 595 Acked 1594 Write 4 Done New Done false
1583 596 Acked 1595 Write 3 Done New Done false
1584 597 Acked 1596 Read 3 Done New Done false
1585 598 Acked 1597 Read 5 Done New Done false
1586 599 Acked 1598 Read 4 Done New Done false
1587 600 Acked 1599 Read 9 Done New Done false
1588 601 Acked 1600 Write 5 Done New Done false
1589 STATES DS:0 DS:1 DS:2 TOTAL
1590 New 0 301 0 301
1591 Sent 0 0 0 0
1592 Done 301 0 301 602
1593 Skipped 0 0 0 0
1594 Error 0 0 0 0
1595 Last Flush: 1584 1299 1584
1596 Downstairs last five completed: 1299 1298 1297 1296 1295
1597 Upstairs last five completed: 601 600 599 598 597
1598 CLIENT: Up:0 ds:301 act:2
15992024-02-15T02:57:47.916ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
16002024-02-15T02:57:47.916ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
16012024-02-15T02:57:47.916ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
16022024-02-15T02:57:47.916ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
16032024-02-15T02:57:47.916ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
16042024-02-15T02:57:47.916ZINFOcrucible: [1] client re-new 302 jobs since flush 1299 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
16052024-02-15T02:57:47.916ZINFOcrucible: client task is exiting = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
1606 [8820] got start action
1607 [8820] I am going to restart
1608 Starting downstairs at port 8820
1609 Make output file at "/tmp/dsc/downstairs-8820.txt"
1610 [8820][1] reports Starting
16112024-02-15T02:57:48.917ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
16122024-02-15T02:57:48.917ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
16132024-02-15T02:57:48.917ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
16142024-02-15T02:57:48.917ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
16152024-02-15T02:57:48.917ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
16162024-02-15T02:57:48.917ZINFOcrucible: [1] client re-new 302 jobs since flush 1299 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
16172024-02-15T02:57:48.917ZINFOcrucible: client task is exiting = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
1618 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1169)
1619 [8820][1] reports Running
16202024-02-15T02:57:49.918ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
16212024-02-15T02:57:49.919ZINFOcrucible: ds_connection connected = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
16222024-02-15T02:57:49.920ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
16232024-02-15T02:57:49.920ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
16242024-02-15T02:57:49.920ZINFOcrucible: send last flush ID to this DS: 1299 = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
16252024-02-15T02:57:49.921ZINFOcrucible: Replied this last flush ID: 1299 = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
16262024-02-15T02:57:49.921ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
16272024-02-15T02:57:49.921ZINFOcrucible: [1] Transition from Offline to Replay = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
16282024-02-15T02:57:49.921ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
16292024-02-15T02:57:49.921ZINFOcrucible: [1] Transition from Replay to Active = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
1630 ----------------------------------------------------------------
1631 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
1632 Downstairs last five completed: 1601 1600 1599 1598 1597
1633 Upstairs last five completed: 602 601 600 599 598
1634 CLIENT: Up:0 ds:0 act:3
1635 Replay: All jobs finished, all DS active.
1636 Main task has work to do, go find it
1637 got dsc Stop(2)
1638 stop 2
1639 Feb 15 02:57:51.264 INFO request completed, latency_us: 351, response_code: 204, uri[8830] Got stop action so:false kr:false
1640 : /stop/cid/2, method: POST, req_id: b6c05408-9bde-4462-8aa4-76af6c512272, remote_addr: 127.0.0.1:54744, local_addr: 127.0.0.1:9998
16412024-02-15T02:57:51.264ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
16422024-02-15T02:57:51.264ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
16432024-02-15T02:57:51.264ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
16442024-02-15T02:57:51.264ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
1645 Feb 15 02:57:51.264 INFO request completed, latency_us: {"msg":"[2] client re-new 0 jobs since flush 1601","v":0,"name":"crucible","level":30,"time":"2024-02-15T02:57:51.26496213Z","hostname":"ip-10-150-1-67.us-west-2.compute.internal","pid":1167,"":"downstairs","":"downstairs","session_id":"4f5a63e5-aa00-4890-9f6b-96c2e069fdb2"}
1646 101, response_code: 200, uri: /state/cid/2, method: GET, req_id: b45d67ab-a6cd-414c-ae6d-fbd22cb0b9ff, remote_addr:{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-15T02:57:51.265003805Z","hostname":"ip-10-150-1-67.us-west-2.compute.internal","pid":1167,"":"io task","client":"2","":"downstairs","session_id":"4f5a63e5-aa00-4890-9f6b-96c2e069fdb2"}
1647 127.0.0.1:54744, local_addr: 127.0.0.1:9998
1648 [8830] Exited with: ExitStatus(unix_wait_status(9))
1649 [8830][2] reports Exit
16502024-02-15T02:57:52.266ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
16512024-02-15T02:57:52.266ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
16522024-02-15T02:57:52.266ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
16532024-02-15T02:57:52.266ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
16542024-02-15T02:57:52.266ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
16552024-02-15T02:57:52.266ZINFOcrucible: [2] client re-new 0 jobs since flush 1601 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
16562024-02-15T02:57:52.266ZINFOcrucible: client task is exiting = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
16572024-02-15T02:57:53.268ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
16582024-02-15T02:57:53.268ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
16592024-02-15T02:57:53.268ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
16602024-02-15T02:57:53.268ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
16612024-02-15T02:57:53.268ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
16622024-02-15T02:57:53.268ZINFOcrucible: [2] client re-new 0 jobs since flush 1601 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
16632024-02-15T02:57:53.268ZINFOcrucible: client task is exiting = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
16642024-02-15T02:57:54.269ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
16652024-02-15T02:57:54.269ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
16662024-02-15T02:57:54.269ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
16672024-02-15T02:57:54.269ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
16682024-02-15T02:57:54.269ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
16692024-02-15T02:57:54.269ZINFOcrucible: [2] client re-new 0 jobs since flush 1601 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
16702024-02-15T02:57:54.269ZINFOcrucible: client task is exiting = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
1671 Feb 15 02:57:55.266 INFO request completed, latency_us: 232, response_code: 200, uri: /state/cid/2, method: GET, req_id: aca4917b-487c-4b8a-b21a-159a1f8f4b43, remote_addr: 127.0.0.1:54744, local_addr: 127.0.0.1:9998
1672 001/300 Write block 4238 len 16384 data: 2 2 2 2
1673 002/300 Read block 4344 len 8192
1674 003/300 Read block 1487 len 8192
1675 004/300 Read block 1454 len 40960
16762024-02-15T02:57:55.270ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
16772024-02-15T02:57:55.270ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
16782024-02-15T02:57:55.270ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
16792024-02-15T02:57:55.270ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
16802024-02-15T02:57:55.270ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
16812024-02-15T02:57:55.270ZINFOcrucible: [2] client re-new 4 jobs since flush 1601 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
16822024-02-15T02:57:55.270ZINFOcrucible: client task is exiting = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
1683 005/300 Flush
1684 006/300 Flush
1685 007/300 Write block 3766 len 20480 data: 2 2 2 2 2
1686 008/300 Read block 452 len 36864
1687 009/300 Read block 1794 len 28672
1688 010/300 Write block 4788 len 36864 data: 2 2 2 2 3 2 2 2 2
1689 011/300 Flush
1690 012/300 Read block 3602 len 32768
1691 013/300 Read block 3731 len 40960
1692 014/300 Write block 534 len 24576 data: 4 4 4 4 2 2
1693 015/300 Read block 24 len 36864
1694 016/300 Flush
1695 017/300 Write block 3807 len 8192 data: 3 2
1696 018/300 Write block 2127 len 40960 data: 2 2 2 2 2 2 3 2 2 2
1697 019/300 Write block 1411 len 12288 data: 2 2 2
1698 020/300 Flush
1699 021/300 Write block 3623 len 24576 data: 2 2 2 2 2 2
1700 022/300 Read block 4987 len 28672
1701 023/300 Read block 4271 len 36864
1702 024/300 Write block 584 len 12288 data: 3 3 2
1703 025/300 Write block 4592 len 20480 data: 2 2 2 2 2
1704 026/300 Write block 1730 len 36864 data: 3 2 2 2 2 2 2 2 2
1705 027/300 Write block 3666 len 36864 data: 2 3 3 3 3 3 3 3 3
1706 028/300 Write block 3253 len 32768 data: 2 2 2 2 2 2 2 2
1707 029/300 Read block 3029 len 28672
1708 030/300 Read block 1667 len 36864
1709 031/300 Read block 3728 len 28672
1710 032/300 Read block 980 len 4096
1711 033/300 Write block 1526 len 28672 data: 2 3 4 4 4 4 2
1712 034/300 Write block 1164 len 20480 data: 2 2 2 2 2
1713 035/300 Write block 3690 len 20480 data: 3 3 3 3 4
1714 036/300 Read block 4249 len 40960
1715 037/300 Read block 1076 len 28672
1716 038/300 Read block 3000 len 32768
1717 039/300 Read block 2955 len 16384
1718 040/300 Write block 699 len 36864 data: 2 2 2 2 2 2 2 2 2
1719 041/300 Read block 1815 len 36864
1720 042/300 Read block 652 len 12288
1721 043/300 Write block 4283 len 12288 data: 2 2 2
1722 044/300 Write block 3687 len 24576 data: 2 2 2 4 4 4
1723 045/300 Flush
1724 046/300 Read block 3593 len 32768
1725 047/300 Write block 1650 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1726 048/300 Read block 4718 len 12288
1727 049/300 Read block 4887 len 36864
1728 050/300 Read block 964 len 4096
1729 051/300 Read block 2196 len 8192
1730 052/300 Read block 1123 len 24576
1731 053/300 Write block 3870 len 16384 data: 3 3 2 2
1732 054/300 Read block 1262 len 8192
1733 055/300 Read block 2228 len 24576
1734 056/300 Flush
1735 057/300 Flush
1736 058/300 Read block 4142 len 8192
1737 059/300 Write block 120 len 12288 data: 2 2 2
1738 060/300 Write block 3768 len 16384 data: 3 3 3 2
1739 061/300 Flush
1740 062/300 Write block 2325 len 24576 data: 2 2 2 2 2 2
1741 063/300 Write block 2183 len 24576 data: 2 2 2 2 2 3
1742 064/300 Write block 1590 len 28672 data: 2 2 2 2 2 2 2
1743 065/300 Write block 1133 len 28672 data: 2 2 2 2 2 2 2
1744 066/300 Write block 1538 len 16384 data: 3 3 2 2
1745 067/300 Write block 973 len 32768 data: 3 3 3 2 2 2 2 2
1746 068/300 Write block 4995 len 4096 data: 2
1747 069/300 Read block 3571 len 40960
1748 070/300 Read block 4627 len 24576
1749 071/300 Write block 2639 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1750 072/300 Read block 4744 len 40960
1751 073/300 Read block 3799 len 16384
1752 074/300 Read block 4344 len 28672
1753 075/300 Read block 223 len 8192
1754 076/300 Read block 1017 len 36864
1755 077/300 Read block 170 len 28672
1756 078/300 Write block 1729 len 24576 data: 3 4 3 3 3 3
1757 079/300 Read block 1146 len 4096
1758 080/300 Write block 3188 len 16384 data: 2 2 2 2
1759 081/300 Read block 1740 len 32768
1760 082/300 Read block 3330 len 28672
1761 083/300 Read block 4943 len 4096
1762 084/300 Read block 4382 len 36864
1763 085/300 Flush
1764 086/300 Read block 758 len 36864
1765 087/300 Write block 4443 len 32768 data: 3 3 3 3 2 2 2 2
1766 088/300 Write block 52 len 28672 data: 2 2 2 2 2 2 2
1767 089/300 Flush
1768 090/300 Read block 4431 len 4096
1769 091/300 Write block 4178 len 28672 data: 2 2 2 2 2 2 2
1770 092/300 Write block 52 len 40960 data: 3 3 3 3 3 3 3 2 2 2
1771 093/300 Write block 3236 len 24576 data: 2 2 2 2 2 2
1772 094/300 Write block 4021 len 40960 data: 4 3 3 2 2 2 2 2 2 2
1773 095/300 Flush
1774 096/300 Write block 4022 len 32768 data: 4 4 3 3 3 3 3 3
1775 097/300 Read block 421 len 40960
1776 098/300 Write block 3213 len 4096 data: 2
1777 099/300 Read block 2092 len 8192
1778 100/300 Read block 2890 len 4096
1779 101/300 Read block 457 len 8192
1780 102/300 Write block 1566 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1781 103/300 Write block 4248 len 20480 data: 3 3 3 3 3
1782 104/300 Read block 4804 len 28672
1783 105/300 Read block 2006 len 28672
1784 106/300 Write block 3584 len 12288 data: 2 2 2
1785 107/300 Read block 3788 len 20480
1786 108/300 Write block 2511 len 20480 data: 2 2 2 2 2
1787 109/300 Write block 20 len 40960 data: 2 3 3 3 3 3 3 3 2 2
1788 110/300 Read block 2719 len 40960
1789 111/300 Write block 3085 len 32768 data: 3 3 3 3 4 3 2 2
1790 112/300 Write block 4259 len 28672 data: 2 2 2 2 2 2 2
1791 113/300 Write block 2546 len 16384 data: 2 2 2 2
1792 114/300 Read block 3799 len 28672
1793 115/300 Read block 3213 len 32768
1794 116/300 Flush
1795 117/300 Flush
1796 118/300 Read block 1835 len 36864
1797 119/300 Read block 3006 len 28672
1798 120/300 Write block 3480 len 32768 data: 2 2 2 2 2 2 2 2
1799 121/300 Read block 2006 len 20480
1800 122/300 Write block 4152 len 36864 data: 3 2 2 2 2 2 2 2 2
1801 123/300 Read block 4898 len 24576
1802 124/300 Read block 1698 len 36864
1803 125/300 Write block 1379 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1804 126/300 Write block 3262 len 8192 data: 2 2
1805 127/300 Read block 4744 len 40960
1806 128/300 Write block 3491 len 24576 data: 4 4 3 3 3 2
1807 129/300 Read block 3730 len 12288
1808 130/300 Read block 4930 len 8192
1809 131/300 Read block 4022 len 24576
1810 132/300 Read block 1244 len 28672
1811 133/300 Write block 2970 len 8192 data: 2 2
1812 134/300 Write block 3086 len 20480 data: 4 4 4 5 4
1813 135/300 Read block 1739 len 32768
1814 136/300 Write block 4739 len 12288 data: 2 2 2
1815 137/300 Read block 3764 len 4096
1816 138/300 Read block 1143 len 28672
1817 139/300 Write block 2085 len 12288 data: 4 3 3
1818 140/300 Read block 2242 len 28672
1819 141/300 Write block 2446 len 32768 data: 2 2 2 2 2 2 2 2
1820 142/300 Write block 3210 len 36864 data: 2 2 2 3 2 2 2 2 2
1821 143/300 Flush
1822 144/300 Write block 639 len 28672 data: 2 2 2 2 2 2 2
1823 145/300 Read block 3000 len 28672
1824 146/300 Write block 1649 len 4096 data: 2
1825 147/300 Read block 4239 len 20480
1826 148/300 Read block 3226 len 28672
1827 149/300 Read block 453 len 8192
1828 150/300 Write block 1554 len 8192 data: 3 3
1829 151/300 Write block 923 len 8192 data: 2 2
1830 152/300 Write block 1592 len 8192 data: 3 3
1831 153/300 Read block 3709 len 4096
1832 154/300 Read block 117 len 28672
1833 155/300 Read block 1360 len 36864
1834 156/300 Read block 4821 len 40960
1835 157/300 Flush
1836 158/300 Write block 4767 len 32768 data: 2 4 3 2 2 3 3 3
1837 159/300 Read block 4830 len 40960
1838 160/300 Read block 4906 len 4096
1839 161/300 Write block 1875 len 16384 data: 2 2 2 2
1840 162/300 Write block 819 len 20480 data: 2 2 2 2 2
1841 163/300 Write block 226 len 4096 data: 3
1842 164/300 Flush
1843 165/300 Read block 2796 len 28672
1844 166/300 Write block 731 len 36864 data: 2 3 3 4 4 4 4 4 4
1845 167/300 Write block 255 len 32768 data: 3 3 3 4 3 3 3 3
1846 168/300 Read block 2974 len 28672
1847 169/300 Read block 2771 len 36864
1848 170/300 Read block 1525 len 32768
1849 171/300 Read block 4050 len 4096
1850 172/300 Read block 396 len 20480
1851 173/300 Read block 2108 len 20480
1852 174/300 Write block 3610 len 20480 data: 2 2 2 2 2
1853 175/300 Read block 4681 len 16384
1854 176/300 Read block 3888 len 40960
1855 177/300 Flush
1856 178/300 Read block 2906 len 20480
1857 179/300 Write block 4905 len 16384 data: 3 4 4 4
1858 180/300 Read block 1190 len 4096
1859 181/300 Read block 4936 len 36864
1860 182/300 Write block 4025 len 4096 data: 4
1861 183/300 Write block 1335 len 24576 data: 2 2 2 2 2 2
1862 184/300 Read block 3678 len 16384
1863 185/300 Write block 1581 len 16384 data: 2 2 2 2
1864 186/300 Read block 3978 len 20480
1865 187/300 Read block 3205 len 4096
1866 188/300 Flush
1867 189/300 Read block 2391 len 24576
1868 190/300 Read block 2948 len 8192
1869 191/300 Read block 1053 len 40960
1870 192/300 Write block 492 len 28672 data: 3 2 2 2 2 2 2
1871 193/300 Read block 862 len 16384
1872 194/300 Read block 3940 len 40960
1873 195/300 Flush
1874 196/300 Read block 3428 len 4096
1875 197/300 Write block 4167 len 16384 data: 2 2 2 2
1876 198/300 Read block 3286 len 8192
1877 199/300 Write block 4675 len 28672 data: 3 2 2 2 2 2 2
1878 200/300 Flush
1879 201/300 Write block 3032 len 28672 data: 4 4 3 3 3 3 2
1880 202/300 Read block 965 len 4096
1881 203/300 Read block 481 len 36864
1882 204/300 Read block 3368 len 8192
1883 205/300 Write block 2958 len 4096 data: 2
1884 206/300 Read block 876 len 36864
1885 207/300 Read block 1233 len 4096
1886 208/300 Read block 3800 len 32768
1887 209/300 Read block 1937 len 28672
1888 210/300 Read block 2297 len 24576
1889 211/300 Read block 4569 len 4096
1890 212/300 Flush
1891 213/300 Read block 1776 len 24576
1892 214/300 Write block 1796 len 16384 data: 2 2 2 2
1893 215/300 Read block 1561 len 4096
1894 216/300 Write block 847 len 28672 data: 2 2 2 2 3 3 3
1895 217/300 Write block 4708 len 8192 data: 2 2
1896 218/300 Read block 441 len 4096
1897 219/300 Write block 3402 len 12288 data: 3 3 2
1898 220/300 Read block 3612 len 32768
1899 221/300 Read block 972 len 16384
1900 222/300 Flush
1901 223/300 Write block 4052 len 20480 data: 2 2 2 3 2
1902 224/300 Write block 2218 len 12288 data: 3 3 3
1903 225/300 Read block 514 len 32768
1904 226/300 Write block 3175 len 36864 data: 2 2 2 2 2 2 2 2 2
1905 227/300 Write block 4739 len 28672 data: 3 3 3 2 2 2 2
1906 228/300 Write block 2879 len 24576 data: 2 3 3 3 3 3
1907 229/300 Read block 4219 len 32768
1908 230/300 Read block 3846 len 20480
1909 231/300 Read block 2686 len 8192
1910 232/300 Write block 3727 len 8192 data: 2 2
1911 233/300 Read block 881 len 20480
1912 234/300 Write block 3020 len 12288 data: 2 2 2
1913 235/300 Read block 4958 len 32768
1914 236/300 Write block 1059 len 24576 data: 3 2 2 2 2 2
1915 237/300 Write block 4028 len 36864 data: 4 4 3 2 2 2 2 2 2
1916 238/300 Read block 4854 len 28672
1917 239/300 Read block 1728 len 12288
1918 240/300 Read block 3663 len 4096
1919 241/300 Read block 1659 len 36864
1920 242/300 Read block 53 len 8192
1921 243/300 Write block 2700 len 28672 data: 2 2 2 2 2 2 2
1922 244/300 Write block 4449 len 12288 data: 3 3 2
1923 245/300 Write block 4236 len 8192 data: 2 2
1924 246/300 Flush
1925 247/300 Write block 844 len 24576 data: 3 2 2 3 3 3
1926 248/300 Write block 4982 len 36864 data: 3 3 3 3 3 2 2 2 2
1927 249/300 Read block 1684 len 8192
1928 250/300 Write block 4502 len 36864 data: 2 2 2 2 2 2 2 2 2
1929 251/300 Read block 4537 len 32768
1930 252/300 Read block 1710 len 16384
1931 253/300 Read block 4335 len 24576
1932 254/300 Read block 3147 len 4096
1933 255/300 Read block 1776 len 16384
1934 256/300 Write block 519 len 24576 data: 2 2 2 2 2 2
1935 257/300 Read block 4578 len 28672
1936 258/300 Read block 1720 len 8192
1937 259/300 Write block 4428 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1938 260/300 Write block 2240 len 12288 data: 3 3 3
1939 261/300 Write block 249 len 4096 data: 3
1940 262/300 Read block 800 len 24576
1941 263/300 Read block 1725 len 20480
1942 264/300 Read block 3651 len 12288
1943 265/300 Write block 148 len 28672 data: 2 2 2 2 2 2 2
1944 266/300 Write block 2001 len 28672 data: 3 2 2 2 2 2 2
1945 267/300 Read block 4437 len 40960
1946 268/300 Read block 3271 len 36864
1947 269/300 Read block 1549 len 8192
1948 270/300 Read block 1382 len 4096
1949 271/300 Read block 4161 len 16384
1950 272/300 Read block 3548 len 12288
1951 273/300 Write block 2430 len 12288 data: 2 2 2
1952 274/300 Read block 20 len 8192
1953 275/300 Read block 4626 len 28672
1954 276/300 Read block 828 len 20480
1955 277/300 Write block 4987 len 28672 data: 3 3 3 3 2 2 2
1956 278/300 Read block 4847 len 16384
1957 279/300 Flush
1958 280/300 Read block 4171 len 12288
1959 281/300 Write block 2575 len 32768 data: 2 2 2 2 2 2 3 3
1960 282/300 Write block 1344 len 24576 data: 2 2 2 2 2 2
1961 283/300 Read block 3907 len 4096
1962 284/300 Write block 4672 len 20480 data: 3 3 3 4 3
1963 285/300 Write block 1170 len 20480 data: 2 2 2 2 2
1964 286/300 Write block 2205 len 12288 data: 2 2 2
1965 287/300 Read block 3689 len 32768
1966 288/300 Write block 2205 len 20480 data: 3 3 3 2 2
1967 289/300 Write block 4051 len 28672 data: 2 3 3 3 4 3 2
1968 290/300 Flush
1969 291/300 Write block 3410 len 8192 data: 3 3
1970 292/300 Read block 747 len 28672
1971 293/300 Read block 3140 len 20480
1972 294/300 Flush
1973 295/300 Flush
1974 296/300 Read block 746 len 8192
1975 297/300 Read block 1762 len 28672
1976 298/300 Read block 1741 len 32768
1977 299/300 Write block 236 len 32768 data: 2 2 2 2 2 2 2 2
1978 300/300 Read block 3093 len 8192
1979 Main task has work to do, go find it
1980 got dsc Start(2)
1981 start 2
1982 Feb 15 02:57:55.601 INFO request completed, latency_us: 198, response_code: 204, uri: /start/cid/2, method: POST, req_id: ef23e301-9a8b-48e9-bbad-f4c0c814ef18, remote_addr: 127.0.0.1:54744, local_addr: 127.0.0.1:9998
1983 Replay: started 2, returned:Ok(())
1984 ----------------------------------------------------------------
1985 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
1986 States: Active Active Offline
1987 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
1988 603 Acked 1602 Write 4 Done Done New true
1989 604 Acked 1603 Read 2 Done Done New true
1990 605 Acked 1604 Read 2 Done Done New true
1991 606 Acked 1605 Read 10 Done Done New true
1992 607 Acked 1606 Flush 0 Done Done New false
1993 608 Acked 1607 Flush 0 Done Done New false
1994 609 Acked 1608 Write 5 Done Done New false
1995 610 Acked 1609 Read 9 Done Done New false
1996 611 Acked 1610 Read 7 Done Done New false
1997 612 Acked 1611 Write 9 Done Done New false
1998 613 Acked 1612 Flush 0 Done Done New false
1999 614 Acked 1613 Read 8 Done Done New false
2000 615 Acked 1614 Read 10 Done Done New false
2001 616 Acked 1615 Write 6 Done Done New false
2002 617 Acked 1616 Read 9 Done Done New false
2003 618 Acked 1617 Flush 0 Done Done New false
2004 619 Acked 1618 Write 2 Done Done New false
2005 620 Acked 1619 Write 10 Done Done New false
2006 621 Acked 1620 Write 3 Done Done New false
2007 622 Acked 1621 Flush 0 Done Done New false
2008 623 Acked 1622 Write 6 Done Done New false
2009 624 Acked 1623 Read 7 Done Done New false
2010 625 Acked 1624 Read 9 Done Done New false
2011 626 Acked 1625 Write 3 Done Done New false
2012 627 Acked 1626 Write 5 Done Done New false
2013 628 Acked 1627 Write 9 Done Done New false
2014 629 Acked 1628 Write 9 Done Done New false
2015 630 Acked 1629 Write 8 Done Done New false
2016 631 Acked 1630 Read 7 Done Done New false
2017 632 Acked 1631 Read 9 Done Done New false
2018 633 Acked 1632 Read 7 Done Done New false
2019 634 Acked 1633 Read 1 Done Done New false
2020 635 Acked 1634 Write 7 Done Done New false
2021 636 Acked 1635 Write 5 Done Done New false
2022 637 Acked 1636 Write 5 Done Done New false
2023 638 Acked 1637 Read 10 Done Done New false
2024 639 Acked 1638 Read 7 Done Done New false
2025 640 Acked 1639 Read 8 Done Done New false
2026 641 Acked 1640 Read 4 Done Done New false
2027 642 Acked 1641 Write 9 Done Done New false
2028 643 Acked 1642 Read 9 Done Done New false
2029 644 Acked 1643 Read 3 Done Done New false
2030 645 Acked 1644 Write 3 Done Done New false
2031 646 Acked 1645 Write 6 Done Done New false
2032 647 Acked 1646 Flush 0 Done Done New false
2033 648 Acked 1647 Read 8 Done Done New false
2034 649 Acked 1648 Write 10 Done Done New false
2035 650 Acked 1649 Read 3 Done Done New false
2036 651 Acked 1650 Read 9 Done Done New false
2037 652 Acked 1651 Read 1 Done Done New false
2038 653 Acked 1652 Read 2 Done Done New false
2039 654 Acked 1653 Read 6 Done Done New false
2040 655 Acked 1654 Write 4 Done Done New false
2041 656 Acked 1655 Read 2 Done Done New false
2042 657 Acked 1656 Read 6 Done Done New false
2043 658 Acked 1657 Flush 0 Done Done New false
2044 659 Acked 1658 Flush 0 Done Done New false
2045 660 Acked 1659 Read 2 Done Done New false
2046 661 Acked 1660 Write 3 Done Done New false
2047 662 Acked 1661 Write 4 Done Done New false
2048 663 Acked 1662 Flush 0 Done Done New false
2049 664 Acked 1663 Write 6 Done Done New false
2050 665 Acked 1664 Write 6 Done Done New false
2051 666 Acked 1665 Write 7 Done Done New false
2052 667 Acked 1666 Write 7 Done Done New false
2053 668 Acked 1667 Write 4 Done Done New false
2054 669 Acked 1668 Write 8 Done Done New false
2055 670 Acked 1669 Write 1 Done Done New false
2056 671 Acked 1670 Read 10 Done Done New false
2057 672 Acked 1671 Read 6 Done Done New false
2058 673 Acked 1672 Write 10 Done Done New false
2059 674 Acked 1673 Read 10 Done Done New false
2060 675 Acked 1674 Read 4 Done Done New false
2061 676 Acked 1675 Read 7 Done Done New false
2062 677 Acked 1676 Read 2 Done Done New false
2063 678 Acked 1677 Read 9 Done Done New false
2064 679 Acked 1678 Read 7 Done Done New false
2065 680 Acked 1679 Write 6 Done Done New false
2066 681 Acked 1680 Read 1 Done Done New false
2067 682 Acked 1681 Write 4 Done Done New false
2068 683 Acked 1682 Read 8 Done Done New false
2069 684 Acked 1683 Read 7 Done Done New false
2070 685 Acked 1684 Read 1 Done Done New false
2071 686 Acked 1685 Read 9 Done Done New false
2072 687 Acked 1686 Flush 0 Done Done New false
2073 688 Acked 1687 Read 9 Done Done New false
2074 689 Acked 1688 Write 8 Done Done New false
2075 690 Acked 1689 Write 7 Done Done New false
2076 691 Acked 1690 Flush 0 Done Done New false
2077 692 Acked 1691 Read 1 Done Done New false
2078 693 Acked 1692 Write 7 Done Done New false
2079 694 Acked 1693 Write 10 Done Done New false
2080 695 Acked 1694 Write 6 Done Done New false
2081 696 Acked 1695 Write 10 Done Done New false
2082 697 Acked 1696 Flush 0 Done Done New false
2083 698 Acked 1697 Write 8 Done Done New false
2084 699 Acked 1698 Read 10 Done Done New false
2085 700 Acked 1699 Write 1 Done Done New false
2086 701 Acked 1700 Read 2 Done Done New false
2087 702 Acked 1701 Read 1 Done Done New false
2088 703 Acked 1702 Read 2 Done Done New false
2089 704 Acked 1703 Write 10 Done Done New false
2090 705 Acked 1704 Write 5 Done Done New false
2091 706 Acked 1705 Read 7 Done Done New false
2092 707 Acked 1706 Read 7 Done Done New false
2093 708 Acked 1707 Write 3 Done Done New false
2094 709 Acked 1708 Read 5 Done Done New false
2095 710 Acked 1709 Write 5 Done Done New false
2096 711 Acked 1710 Write 10 Done Done New false
2097 712 Acked 1711 Read 10 Done Done New false
2098 713 Acked 1712 Write 8 Done Done New false
2099 714 Acked 1713 Write 7 Done Done New false
2100 715 Acked 1714 Write 4 Done Done New false
2101 716 Acked 1715 Read 7 Done Done New false
2102 717 Acked 1716 Read 8 Done Done New false
2103 718 Acked 1717 Flush 0 Done Done New false
2104 719 Acked 1718 Flush 0 Done Done New false
2105 720 Acked 1719 Read 9 Done Done New false
2106 721 Acked 1720 Read 7 Done Done New false
2107 722 Acked 1721 Write 8 Done Done New false
2108 723 Acked 1722 Read 5 Done Done New false
2109 724 Acked 1723 Write 9 Done Done New false
2110 725 Acked 1724 Read 6 Done Done New false
2111 726 Acked 1725 Read 9 Done Done New false
2112 727 Acked 1726 Write 10 Done Done New false
2113 728 Acked 1727 Write 2 Done Done New false
2114 729 Acked 1728 Read 10 Done Done New false
2115 730 Acked 1729 Write 6 Done Done New false
2116 731 Acked 1730 Read 3 Done Done New false
2117 732 Acked 1731 Read 2 Done Done New false
2118 733 Acked 1732 Read 6 Done Done New false
2119 734 Acked 1733 Read 7 Done Done New false
2120 735 Acked 1734 Write 2 Done Done New false
2121 736 Acked 1735 Write 5 Done Done New false
2122 737 Acked 1736 Read 8 Done Done New false
2123 738 Acked 1737 Write 3 Done Done New false
2124 739 Acked 1738 Read 1 Done Done New false
2125 740 Acked 1739 Read 7 Done Done New false
2126 741 Acked 1740 Write 3 Done Done New false
2127 742 Acked 1741 Read 7 Done Done New false
2128 743 Acked 1742 Write 8 Done Done New false
2129 744 Acked 1743 Write 9 Done Done New false
2130 745 Acked 1744 Flush 0 Done Done New false
2131 746 Acked 1745 Write 7 Done Done New false
2132 747 Acked 1746 Read 7 Done Done New false
2133 748 Acked 1747 Write 1 Done Done New false
2134 749 Acked 1748 Read 5 Done Done New false
2135 750 Acked 1749 Read 7 Done Done New false
2136 751 Acked 1750 Read 2 Done Done New false
2137 752 Acked 1751 Write 2 Done Done New false
2138 753 Acked 1752 Write 2 Done Done New false
2139 754 Acked 1753 Write 2 Done Done New false
2140 755 Acked 1754 Read 1 Done Done New false
2141 756 Acked 1755 Read 7 Done Done New false
2142 757 Acked 1756 Read 9 Done Done New false
2143 758 Acked 1757 Read 10 Done Done New false
2144 759 Acked 1758 Flush 0 Done Done New false
2145 760 Acked 1759 Write 8 Done Done New false
2146 761 Acked 1760 Read 10 Done Done New false
2147 762 Acked 1761 Read 1 Done Done New false
2148 763 Acked 1762 Write 4 Done Done New false
2149 764 Acked 1763 Write 5 Done Done New false
2150 765 Acked 1764 Write 1 Done Done New false
2151 766 Acked 1765 Flush 0 Done Done New false
2152 767 Acked 1766 Read 7 Done Done New false
2153 768 Acked 1767 Write 9 Done Done New false
2154 769 Acked 1768 Write 8 Done Done New false
2155 770 Acked 1769 Read 7 Done Done New false
2156 771 Acked 1770 Read 9 Done Done New false
2157 772 Acked 1771 Read 8 Done Done New false
2158 773 Acked 1772 Read 1 Done Done New false
2159 774 Acked 1773 Read 5 Done Done New false
2160 775 Acked 1774 Read 5 Done Done New false
2161 776 Acked 1775 Write 5 Done Done New false
2162 777 Acked 1776 Read 4 Done Done New false
2163 778 Acked 1777 Read 10 Done Done New false
2164 779 Acked 1778 Flush 0 Done Done New false
2165 780 Acked 1779 Read 5 Done Done New false
2166 781 Acked 1780 Write 4 Done Done New false
2167 782 Acked 1781 Read 1 Done Done New false
2168 783 Acked 1782 Read 9 Done Done New false
2169 784 Acked 1783 Write 1 Done Done New false
2170 785 Acked 1784 Write 6 Done Done New false
2171 786 Acked 1785 Read 4 Done Done New false
2172 787 Acked 1786 Write 4 Done Done New false
2173 788 Acked 1787 Read 5 Done Done New false
2174 789 Acked 1788 Read 1 Done Done New false
2175 790 Acked 1789 Flush 0 Done Done New false
2176 791 Acked 1790 Read 6 Done Done New false
2177 792 Acked 1791 Read 2 Done Done New false
2178 793 Acked 1792 Read 10 Done Done New false
2179 794 Acked 1793 Write 7 Done Done New false
2180 795 Acked 1794 Read 4 Done Done New false
2181 796 Acked 1795 Read 10 Done Done New false
2182 797 Acked 1796 Flush 0 Done Done New false
2183 798 Acked 1797 Read 1 Done Done New false
2184 799 Acked 1798 Write 4 Done Done New false
2185 800 Acked 1799 Read 2 Done Done New false
2186 801 Acked 1800 Write 7 Done Done New false
2187 802 Acked 1801 Flush 0 Done Done New false
2188 803 Acked 1802 Write 7 Done Done New false
2189 804 Acked 1803 Read 1 Done Done New false
2190 805 Acked 1804 Read 9 Done Done New false
2191 806 Acked 1805 Read 2 Done Done New false
2192 807 Acked 1806 Write 1 Done Done New false
2193 808 Acked 1807 Read 9 Done Done New false
2194 809 Acked 1808 Read 1 Done Done New false
2195 810 Acked 1809 Read 8 Done Done New false
2196 811 Acked 1810 Read 7 Done Done New false
2197 812 Acked 1811 Read 6 Done Done New false
2198 813 Acked 1812 Read 1 Done Done New false
2199 814 Acked 1813 Flush 0 Done Done New false
2200 815 Acked 1814 Read 6 Done Done New false
2201 816 Acked 1815 Write 4 Done Done New false
2202 817 Acked 1816 Read 1 Done Done New false
2203 818 Acked 1817 Write 7 Done Done New false
2204 819 Acked 1818 Write 2 Done Done New false
2205 820 Acked 1819 Read 1 Done Done New false
2206 821 Acked 1820 Write 3 Done Done New false
2207 822 Acked 1821 Read 8 Done Done New false
2208 823 Acked 1822 Read 4 Done Done New false
2209 824 Acked 1823 Flush 0 Done Done New false
2210 825 Acked 1824 Write 5 Done Done New false
2211 826 Acked 1825 Write 3 Done Done New false
2212 827 Acked 1826 Read 8 Done Done New false
2213 828 Acked 1827 Write 9 Done Done New false
2214 829 Acked 1828 Write 7 Done Done New false
2215 830 Acked 1829 Write 6 Done Done New false
2216 831 Acked 1830 Read 8 Done Done New false
2217 832 Acked 1831 Read 5 Done Done New false
2218 833 Acked 1832 Read 2 Done Done New false
2219 834 Acked 1833 Write 2 Done Done New false
2220 835 Acked 1834 Read 5 Done Done New false
2221 836 Acked 1835 Write 3 Done Done New false
2222 837 Acked 1836 Read 8 Done Done New false
2223 838 Acked 1837 Write 6 Done Done New false
2224 839 Acked 1838 Write 9 Done Done New false
2225 840 Acked 1839 Read 7 Done Done New false
2226 841 Acked 1840 Read 3 Done Done New false
2227 842 Acked 1841 Read 1 Done Done New false
2228 843 Acked 1842 Read 9 Done Done New false
2229 844 Acked 1843 Read 2 Done Done New false
2230 845 Acked 1844 Write 7 Done Done New false
2231 846 Acked 1845 Write 3 Done Done New false
2232 847 Acked 1846 Write 2 Done Done New false
2233 848 Acked 1847 Flush 0 Done Done New false
2234 849 Acked 1848 Write 6 Done Done New false
2235 850 Acked 1849 Write 9 Done Done New false
2236 851 Acked 1850 Read 2 Done Done New false
2237 852 Acked 1851 Write 9 Done Done New false
2238 853 Acked 1852 Read 8 Done Done New false
2239 854 Acked 1853 Read 4 Done Done New false
2240 855 Acked 1854 Read 6 Done Done New false
2241 856 Acked 1855 Read 1 Done Done New false
2242 857 Acked 1856 Read 4 Done Done New false
2243 858 Acked 1857 Flush 0 Done Done New false
2244 859 Acked 1858 Write 6 Done Done New false
2245 860 Acked 1859 Read 7 Done Done New false
2246 861 Acked 1860 Read 2 Done Done New false
2247 862 Acked 1861 Write 10 Done Done New false
2248 863 Acked 1862 Write 3 Done Done New false
2249 864 Acked 1863 Write 1 Done Done New false
2250 865 Acked 1864 Read 6 Done Done New false
2251 866 Acked 1865 Read 5 Done Done New false
2252 867 Acked 1866 Read 3 Done Done New false
2253 868 Acked 1867 Write 7 Done Done New false
2254 869 Acked 1868 Write 7 Done Done New false
2255 870 Acked 1869 Read 10 Done Done New false
2256 871 Acked 1870 Read 9 Done Done New false
2257 872 Acked 1871 Read 2 Done Done New false
2258 873 Acked 1872 Read 1 Done Done New false
2259 874 Acked 1873 Read 4 Done Done New false
2260 875 Acked 1874 Read 3 Done Done New false
2261 876 Acked 1875 Write 3 Done Done New false
2262 877 Acked 1876 Read 2 Done Done New false
2263 878 Acked 1877 Read 7 Done Done New false
2264 879 Acked 1878 Read 5 Done Done New false
2265 880 Acked 1879 Write 7 Done Done New false
2266 881 Acked 1880 Read 4 Done Done New false
2267 882 Acked 1881 Flush 0 Done Done New false
2268 883 Acked 1882 Read 3 Done Done New false
2269 884 Acked 1883 Write 8 Done Done New false
2270 885 Acked 1884 Write 6 Done Done New false
2271 886 Acked 1885 Read 1 Done Done New false
2272 887 Acked 1886 Write 5 Done Done New false
2273 888 Acked 1887 Write 5 Done Done New false
2274 889 Acked 1888 Write 3 Done Done New false
2275 890 Acked 1889 Read 8 Done Done New false
2276 891 Acked 1890 Write 5 Done Done New false
2277 892 Acked 1891 Write 7 Done Done New false
2278 893 Acked 1892 Flush 0 Done Done New false
2279 894 Acked 1893 Write 2 Done Done New false
2280 895 Acked 1894 Read 7 Done Done New false
2281 896 Acked 1895 Read 5 Done Done New false
2282 897 Acked 1896 Flush 0 Done Done New false
2283 898 Acked 1897 Flush 0 Done Done New false
2284 899 Acked 1898 Read 2 Done Done New false
2285 900 Acked 1899 Read 7 Done Done New false
2286 901 Acked 1900 Read 8 Done Done New false
2287 902 Acked 1901 Write 8 Done Done New false
2288 903 Acked 1902 Read 2 Done Done New false
2289 STATES DS:0 DS:1 DS:2 TOTAL
2290 New 0 0 301 301
2291 Sent 0 0 0 0
2292 Done 301 301 0 602
2293 Skipped 0 0 0 0
2294 Error 0 0 0 0
2295 Last Flush: 1897 1897 1601
2296 Downstairs last five completed: 1601 1600 1599 1598 1597
2297 Upstairs last five completed: 903 902 901 900 899
2298 CLIENT: Up:0 ds:301 act:2
22992024-02-15T02:57:56.272ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
23002024-02-15T02:57:56.272ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
23012024-02-15T02:57:56.272ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
23022024-02-15T02:57:56.272ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
23032024-02-15T02:57:56.272ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
23042024-02-15T02:57:56.272ZINFOcrucible: [2] client re-new 302 jobs since flush 1601 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
23052024-02-15T02:57:56.272ZINFOcrucible: client task is exiting = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
23062024-02-15T02:57:57.273ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
23072024-02-15T02:57:57.273ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
23082024-02-15T02:57:57.273ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
23092024-02-15T02:57:57.273ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
23102024-02-15T02:57:57.273ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
23112024-02-15T02:57:57.273ZINFOcrucible: [2] client re-new 302 jobs since flush 1601 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
23122024-02-15T02:57:57.274ZINFOcrucible: client task is exiting = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
23132024-02-15T02:57:58.274ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
23142024-02-15T02:57:58.275ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
23152024-02-15T02:57:58.275ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
23162024-02-15T02:57:58.275ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
23172024-02-15T02:57:58.275ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
23182024-02-15T02:57:58.275ZINFOcrucible: [2] client re-new 302 jobs since flush 1601 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
23192024-02-15T02:57:58.275ZINFOcrucible: client task is exiting = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
2320 [8830] got start action
2321 [8830] I am going to restart
2322 Starting downstairs at port 8830
2323 Make output file at "/tmp/dsc/downstairs-8830.txt"
2324 [8830][2] reports Starting
2325 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1170)
2326 [8830][2] reports Running
23272024-02-15T02:57:59.276ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
23282024-02-15T02:57:59.276ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
23292024-02-15T02:57:59.276ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
23302024-02-15T02:57:59.276ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
23312024-02-15T02:57:59.276ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
23322024-02-15T02:57:59.276ZINFOcrucible: [2] client re-new 302 jobs since flush 1601 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
23332024-02-15T02:57:59.276ZINFOcrucible: client task is exiting = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
2334 ----------------------------------------------------------------
2335 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
2336 States: Active Active Offline
2337 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
2338 603 Acked 1602 Write 4 Done Done New true
2339 604 Acked 1603 Read 2 Done Done New true
2340 605 Acked 1604 Read 2 Done Done New true
2341 606 Acked 1605 Read 10 Done Done New true
2342 607 Acked 1606 Flush 0 Done Done New true
2343 608 Acked 1607 Flush 0 Done Done New true
2344 609 Acked 1608 Write 5 Done Done New true
2345 610 Acked 1609 Read 9 Done Done New true
2346 611 Acked 1610 Read 7 Done Done New true
2347 612 Acked 1611 Write 9 Done Done New true
2348 613 Acked 1612 Flush 0 Done Done New true
2349 614 Acked 1613 Read 8 Done Done New true
2350 615 Acked 1614 Read 10 Done Done New true
2351 616 Acked 1615 Write 6 Done Done New true
2352 617 Acked 1616 Read 9 Done Done New true
2353 618 Acked 1617 Flush 0 Done Done New true
2354 619 Acked 1618 Write 2 Done Done New true
2355 620 Acked 1619 Write 10 Done Done New true
2356 621 Acked 1620 Write 3 Done Done New true
2357 622 Acked 1621 Flush 0 Done Done New true
2358 623 Acked 1622 Write 6 Done Done New true
2359 624 Acked 1623 Read 7 Done Done New true
2360 625 Acked 1624 Read 9 Done Done New true
2361 626 Acked 1625 Write 3 Done Done New true
2362 627 Acked 1626 Write 5 Done Done New true
2363 628 Acked 1627 Write 9 Done Done New true
2364 629 Acked 1628 Write 9 Done Done New true
2365 630 Acked 1629 Write 8 Done Done New true
2366 631 Acked 1630 Read 7 Done Done New true
2367 632 Acked 1631 Read 9 Done Done New true
2368 633 Acked 1632 Read 7 Done Done New true
2369 634 Acked 1633 Read 1 Done Done New true
2370 635 Acked 1634 Write 7 Done Done New true
2371 636 Acked 1635 Write 5 Done Done New true
2372 637 Acked 1636 Write 5 Done Done New true
2373 638 Acked 1637 Read 10 Done Done New true
2374 639 Acked 1638 Read 7 Done Done New true
2375 640 Acked 1639 Read 8 Done Done New true
2376 641 Acked 1640 Read 4 Done Done New true
2377 642 Acked 1641 Write 9 Done Done New true
2378 643 Acked 1642 Read 9 Done Done New true
2379 644 Acked 1643 Read 3 Done Done New true
2380 645 Acked 1644 Write 3 Done Done New true
2381 646 Acked 1645 Write 6 Done Done New true
2382 647 Acked 1646 Flush 0 Done Done New true
2383 648 Acked 1647 Read 8 Done Done New true
2384 649 Acked 1648 Write 10 Done Done New true
2385 650 Acked 1649 Read 3 Done Done New true
2386 651 Acked 1650 Read 9 Done Done New true
2387 652 Acked 1651 Read 1 Done Done New true
2388 653 Acked 1652 Read 2 Done Done New true
2389 654 Acked 1653 Read 6 Done Done New true
2390 655 Acked 1654 Write 4 Done Done New true
2391 656 Acked 1655 Read 2 Done Done New true
2392 657 Acked 1656 Read 6 Done Done New true
2393 658 Acked 1657 Flush 0 Done Done New true
2394 659 Acked 1658 Flush 0 Done Done New true
2395 660 Acked 1659 Read 2 Done Done New true
2396 661 Acked 1660 Write 3 Done Done New true
2397 662 Acked 1661 Write 4 Done Done New true
2398 663 Acked 1662 Flush 0 Done Done New true
2399 664 Acked 1663 Write 6 Done Done New true
2400 665 Acked 1664 Write 6 Done Done New true
2401 666 Acked 1665 Write 7 Done Done New true
2402 667 Acked 1666 Write 7 Done Done New true
2403 668 Acked 1667 Write 4 Done Done New true
2404 669 Acked 1668 Write 8 Done Done New true
2405 670 Acked 1669 Write 1 Done Done New true
2406 671 Acked 1670 Read 10 Done Done New true
2407 672 Acked 1671 Read 6 Done Done New true
2408 673 Acked 1672 Write 10 Done Done New true
2409 674 Acked 1673 Read 10 Done Done New true
2410 675 Acked 1674 Read 4 Done Done New true
2411 676 Acked 1675 Read 7 Done Done New true
2412 677 Acked 1676 Read 2 Done Done New true
2413 678 Acked 1677 Read 9 Done Done New true
2414 679 Acked 1678 Read 7 Done Done New true
2415 680 Acked 1679 Write 6 Done Done New true
2416 681 Acked 1680 Read 1 Done Done New true
2417 682 Acked 1681 Write 4 Done Done New true
2418 683 Acked 1682 Read 8 Done Done New true
2419 684 Acked 1683 Read 7 Done Done New true
2420 685 Acked 1684 Read 1 Done Done New true
2421 686 Acked 1685 Read 9 Done Done New true
2422 687 Acked 1686 Flush 0 Done Done New true
2423 688 Acked 1687 Read 9 Done Done New true
2424 689 Acked 1688 Write 8 Done Done New true
2425 690 Acked 1689 Write 7 Done Done New true
2426 691 Acked 1690 Flush 0 Done Done New true
2427 692 Acked 1691 Read 1 Done Done New true
2428 693 Acked 1692 Write 7 Done Done New true
2429 694 Acked 1693 Write 10 Done Done New true
2430 695 Acked 1694 Write 6 Done Done New true
2431 696 Acked 1695 Write 10 Done Done New true
2432 697 Acked 1696 Flush 0 Done Done New true
2433 698 Acked 1697 Write 8 Done Done New true
2434 699 Acked 1698 Read 10 Done Done New true
2435 700 Acked 1699 Write 1 Done Done New true
2436 701 Acked 1700 Read 2 Done Done New true
2437 702 Acked 1701 Read 1 Done Done New true
2438 703 Acked 1702 Read 2 Done Done New true
2439 704 Acked 1703 Write 10 Done Done New true
2440 705 Acked 1704 Write 5 Done Done New true
2441 706 Acked 1705 Read 7 Done Done New true
2442 707 Acked 1706 Read 7 Done Done New true
2443 708 Acked 1707 Write 3 Done Done New true
2444 709 Acked 1708 Read 5 Done Done New true
2445 710 Acked 1709 Write 5 Done Done New true
2446 711 Acked 1710 Write 10 Done Done New true
2447 712 Acked 1711 Read 10 Done Done New true
2448 713 Acked 1712 Write 8 Done Done New true
2449 714 Acked 1713 Write 7 Done Done New true
2450 715 Acked 1714 Write 4 Done Done New true
2451 716 Acked 1715 Read 7 Done Done New true
2452 717 Acked 1716 Read 8 Done Done New true
2453 718 Acked 1717 Flush 0 Done Done New true
2454 719 Acked 1718 Flush 0 Done Done New true
2455 720 Acked 1719 Read 9 Done Done New true
2456 721 Acked 1720 Read 7 Done Done New true
2457 722 Acked 1721 Write 8 Done Done New true
2458 723 Acked 1722 Read 5 Done Done New true
2459 724 Acked 1723 Write 9 Done Done New true
2460 725 Acked 1724 Read 6 Done Done New true
2461 726 Acked 1725 Read 9 Done Done New true
2462 727 Acked 1726 Write 10 Done Done New true
2463 728 Acked 1727 Write 2 Done Done New true
2464 729 Acked 1728 Read 10 Done Done New true
2465 730 Acked 1729 Write 6 Done Done New true
2466 731 Acked 1730 Read 3 Done Done New true
2467 732 Acked 1731 Read 2 Done Done New true
2468 733 Acked 1732 Read 6 Done Done New true
2469 734 Acked 1733 Read 7 Done Done New true
2470 735 Acked 1734 Write 2 Done Done New true
2471 736 Acked 1735 Write 5 Done Done New true
2472 737 Acked 1736 Read 8 Done Done New true
2473 738 Acked 1737 Write 3 Done Done New true
2474 739 Acked 1738 Read 1 Done Done New true
2475 740 Acked 1739 Read 7 Done Done New true
2476 741 Acked 1740 Write 3 Done Done New true
2477 742 Acked 1741 Read 7 Done Done New true
2478 743 Acked 1742 Write 8 Done Done New true
2479 744 Acked 1743 Write 9 Done Done New true
2480 745 Acked 1744 Flush 0 Done Done New true
2481 746 Acked 1745 Write 7 Done Done New true
2482 747 Acked 1746 Read 7 Done Done New true
2483 748 Acked 1747 Write 1 Done Done New true
2484 749 Acked 1748 Read 5 Done Done New true
2485 750 Acked 1749 Read 7 Done Done New true
2486 751 Acked 1750 Read 2 Done Done New true
2487 752 Acked 1751 Write 2 Done Done New true
2488 753 Acked 1752 Write 2 Done Done New true
2489 754 Acked 1753 Write 2 Done Done New true
2490 755 Acked 1754 Read 1 Done Done New true
2491 756 Acked 1755 Read 7 Done Done New true
2492 757 Acked 1756 Read 9 Done Done New true
2493 758 Acked 1757 Read 10 Done Done New true
2494 759 Acked 1758 Flush 0 Done Done New true
2495 760 Acked 1759 Write 8 Done Done New true
2496 761 Acked 1760 Read 10 Done Done New true
2497 762 Acked 1761 Read 1 Done Done New true
2498 763 Acked 1762 Write 4 Done Done New true
2499 764 Acked 1763 Write 5 Done Done New true
2500 765 Acked 1764 Write 1 Done Done New true
2501 766 Acked 1765 Flush 0 Done Done New true
2502 767 Acked 1766 Read 7 Done Done New true
2503 768 Acked 1767 Write 9 Done Done New true
2504 769 Acked 1768 Write 8 Done Done New true
2505 770 Acked 1769 Read 7 Done Done New true
2506 771 Acked 1770 Read 9 Done Done New true
2507 772 Acked 1771 Read 8 Done Done New true
2508 773 Acked 1772 Read 1 Done Done New true
2509 774 Acked 1773 Read 5 Done Done New true
2510 775 Acked 1774 Read 5 Done Done New true
2511 776 Acked 1775 Write 5 Done Done New true
2512 777 Acked 1776 Read 4 Done Done New true
2513 778 Acked 1777 Read 10 Done Done New true
2514 779 Acked 1778 Flush 0 Done Done New true
2515 780 Acked 1779 Read 5 Done Done New true
2516 781 Acked 1780 Write 4 Done Done New true
2517 782 Acked 1781 Read 1 Done Done New true
2518 783 Acked 1782 Read 9 Done Done New true
2519 784 Acked 1783 Write 1 Done Done New true
2520 785 Acked 1784 Write 6 Done Done New true
2521 786 Acked 1785 Read 4 Done Done New true
2522 787 Acked 1786 Write 4 Done Done New true
2523 788 Acked 1787 Read 5 Done Done New true
2524 789 Acked 1788 Read 1 Done Done New true
2525 790 Acked 1789 Flush 0 Done Done New true
2526 791 Acked 1790 Read 6 Done Done New true
2527 792 Acked 1791 Read 2 Done Done New true
2528 793 Acked 1792 Read 10 Done Done New true
2529 794 Acked 1793 Write 7 Done Done New true
2530 795 Acked 1794 Read 4 Done Done New true
2531 796 Acked 1795 Read 10 Done Done New true
2532 797 Acked 1796 Flush 0 Done Done New true
2533 798 Acked 1797 Read 1 Done Done New true
2534 799 Acked 1798 Write 4 Done Done New true
2535 800 Acked 1799 Read 2 Done Done New true
2536 801 Acked 1800 Write 7 Done Done New true
2537 802 Acked 1801 Flush 0 Done Done New true
2538 803 Acked 1802 Write 7 Done Done New true
2539 804 Acked 1803 Read 1 Done Done New true
2540 805 Acked 1804 Read 9 Done Done New true
2541 806 Acked 1805 Read 2 Done Done New true
2542 807 Acked 1806 Write 1 Done Done New true
2543 808 Acked 1807 Read 9 Done Done New true
2544 809 Acked 1808 Read 1 Done Done New true
2545 810 Acked 1809 Read 8 Done Done New true
2546 811 Acked 1810 Read 7 Done Done New true
2547 812 Acked 1811 Read 6 Done Done New true
2548 813 Acked 1812 Read 1 Done Done New true
2549 814 Acked 1813 Flush 0 Done Done New true
2550 815 Acked 1814 Read 6 Done Done New true
2551 816 Acked 1815 Write 4 Done Done New true
2552 817 Acked 1816 Read 1 Done Done New true
2553 818 Acked 1817 Write 7 Done Done New true
2554 819 Acked 1818 Write 2 Done Done New true
2555 820 Acked 1819 Read 1 Done Done New true
2556 821 Acked 1820 Write 3 Done Done New true
2557 822 Acked 1821 Read 8 Done Done New true
2558 823 Acked 1822 Read 4 Done Done New true
2559 824 Acked 1823 Flush 0 Done Done New true
2560 825 Acked 1824 Write 5 Done Done New true
2561 826 Acked 1825 Write 3 Done Done New true
2562 827 Acked 1826 Read 8 Done Done New true
2563 828 Acked 1827 Write 9 Done Done New true
2564 829 Acked 1828 Write 7 Done Done New true
2565 830 Acked 1829 Write 6 Done Done New true
2566 831 Acked 1830 Read 8 Done Done New true
2567 832 Acked 1831 Read 5 Done Done New true
2568 833 Acked 1832 Read 2 Done Done New true
2569 834 Acked 1833 Write 2 Done Done New true
2570 835 Acked 1834 Read 5 Done Done New true
2571 836 Acked 1835 Write 3 Done Done New true
2572 837 Acked 1836 Read 8 Done Done New true
2573 838 Acked 1837 Write 6 Done Done New true
2574 839 Acked 1838 Write 9 Done Done New true
2575 840 Acked 1839 Read 7 Done Done New true
2576 841 Acked 1840 Read 3 Done Done New true
2577 842 Acked 1841 Read 1 Done Done New true
2578 843 Acked 1842 Read 9 Done Done New true
2579 844 Acked 1843 Read 2 Done Done New true
2580 845 Acked 1844 Write 7 Done Done New true
2581 846 Acked 1845 Write 3 Done Done New true
2582 847 Acked 1846 Write 2 Done Done New true
2583 848 Acked 1847 Flush 0 Done Done New true
2584 849 Acked 1848 Write 6 Done Done New true
2585 850 Acked 1849 Write 9 Done Done New true
2586 851 Acked 1850 Read 2 Done Done New true
2587 852 Acked 1851 Write 9 Done Done New true
2588 853 Acked 1852 Read 8 Done Done New true
2589 854 Acked 1853 Read 4 Done Done New true
2590 855 Acked 1854 Read 6 Done Done New true
2591 856 Acked 1855 Read 1 Done Done New true
2592 857 Acked 1856 Read 4 Done Done New true
2593 858 Acked 1857 Flush 0 Done Done New true
2594 859 Acked 1858 Write 6 Done Done New true
2595 860 Acked 1859 Read 7 Done Done New true
2596 861 Acked 1860 Read 2 Done Done New true
2597 862 Acked 1861 Write 10 Done Done New true
2598 863 Acked 1862 Write 3 Done Done New true
2599 864 Acked 1863 Write 1 Done Done New true
2600 865 Acked 1864 Read 6 Done Done New true
2601 866 Acked 1865 Read 5 Done Done New true
2602 867 Acked 1866 Read 3 Done Done New true
2603 868 Acked 1867 Write 7 Done Done New true
2604 869 Acked 1868 Write 7 Done Done New true
2605 870 Acked 1869 Read 10 Done Done New true
2606 871 Acked 1870 Read 9 Done Done New true
2607 872 Acked 1871 Read 2 Done Done New true
2608 873 Acked 1872 Read 1 Done Done New true
2609 874 Acked 1873 Read 4 Done Done New true
2610 875 Acked 1874 Read 3 Done Done New true
2611 876 Acked 1875 Write 3 Done Done New true
2612 877 Acked 1876 Read 2 Done Done New true
2613 878 Acked 1877 Read 7 Done Done New true
2614 879 Acked 1878 Read 5 Done Done New true
2615 880 Acked 1879 Write 7 Done Done New true
2616 881 Acked 1880 Read 4 Done Done New true
2617 882 Acked 1881 Flush 0 Done Done New true
2618 883 Acked 1882 Read 3 Done Done New true
2619 884 Acked 1883 Write 8 Done Done New true
2620 885 Acked 1884 Write 6 Done Done New true
2621 886 Acked 1885 Read 1 Done Done New true
2622 887 Acked 1886 Write 5 Done Done New true
2623 888 Acked 1887 Write 5 Done Done New true
2624 889 Acked 1888 Write 3 Done Done New true
2625 890 Acked 1889 Read 8 Done Done New true
2626 891 Acked 1890 Write 5 Done Done New true
2627 892 Acked 1891 Write 7 Done Done New true
2628 893 Acked 1892 Flush 0 Done Done New true
2629 894 Acked 1893 Write 2 Done Done New true
2630 895 Acked 1894 Read 7 Done Done New true
2631 896 Acked 1895 Read 5 Done Done New true
2632 897 Acked 1896 Flush 0 Done Done New true
2633 898 Acked 1897 Flush 0 Done Done New true
2634 899 Acked 1898 Read 2 Done Done New true
2635 900 Acked 1899 Read 7 Done Done New true
2636 901 Acked 1900 Read 8 Done Done New true
2637 902 Acked 1901 Write 8 Done Done New true
2638 903 Acked 1902 Read 2 Done Done New true
2639 904 Acked 1903 Flush 0 Done Done New true
2640 STATES DS:0 DS:1 DS:2 TOTAL
2641 New 0 0 302 302
2642 Sent 0 0 0 0
2643 Done 302 302 0 604
2644 Skipped 0 0 0 0
2645 Error 0 0 0 0
2646 Last Flush: 1903 1903 1601
2647 Downstairs last five completed: 1601 1600 1599 1598 1597
2648 Upstairs last five completed: 904 903 902 901 900
2649 CLIENT: Up:0 ds:302 act:2
26502024-02-15T02:58:00.278ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
26512024-02-15T02:58:00.278ZINFOcrucible: ds_connection connected = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
26522024-02-15T02:58:00.279ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
26532024-02-15T02:58:00.279ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
26542024-02-15T02:58:00.279ZINFOcrucible: send last flush ID to this DS: 1601 = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
26552024-02-15T02:58:00.279ZINFOcrucible: Replied this last flush ID: 1601 = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
26562024-02-15T02:58:00.279ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
26572024-02-15T02:58:00.279ZINFOcrucible: [2] Transition from Offline to Replay = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
26582024-02-15T02:58:00.279ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
26592024-02-15T02:58:00.279ZINFOcrucible: [2] Transition from Replay to Active = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
2660 ----------------------------------------------------------------
2661 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
2662 Downstairs last five completed: 1903 1902 1901 1900 1899
2663 Upstairs last five completed: 904 903 902 901 900
2664 CLIENT: Up:0 ds:0 act:3
2665 Replay: All jobs finished, all DS active.
2666 Main task has work to do, go find it
2667 got dsc Stop(2)
2668 stop 2
2669 Feb 15 02:58:03.607 INFO request completed, latency_us: 333, response_code: 204, uri: /stop/cid/2, method: POST, [8830] Got stop action so:false kr:false
2670 req_id: adaa1419-67ed-4f36-b114-4de41ffd5bb5, remote_addr: 127.0.0.1:54744, local_addr: 127.0.0.1:9998
26712024-02-15T02:58:03.608ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
26722024-02-15T02:58:03.608ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
26732024-02-15T02:58:03.608ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
26742024-02-15T02:58:03.608ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
26752024-02-15T02:58:03.608ZINFOcrucible: [2] client re-new 0 jobs since flush 1903 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
26762024-02-15T02:58:03.608ZINFOcrucible: client task is exiting = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
2677 Feb 15 02:58:03.608 INFO request completed, latency_us: 180, response_code: 200, uri: /state/cid/2, method: GET, req_id: f90a9cbc-50ca-452f-8a4f-0530da38a25d, remote_addr: 127.0.0.1:54744, local_addr: 127.0.0.1:9998
2678 [8830] Exited with: ExitStatus(unix_wait_status(9))
2679 [8830][2] reports Exit
26802024-02-15T02:58:04.608ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
26812024-02-15T02:58:04.609ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
26822024-02-15T02:58:04.609ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
26832024-02-15T02:58:04.609ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
26842024-02-15T02:58:04.609ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
26852024-02-15T02:58:04.609ZINFOcrucible: [2] client re-new 0 jobs since flush 1903 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
26862024-02-15T02:58:04.609ZINFOcrucible: client task is exiting = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
26872024-02-15T02:58:05.610ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
26882024-02-15T02:58:05.610ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
26892024-02-15T02:58:05.610ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
26902024-02-15T02:58:05.610ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
26912024-02-15T02:58:05.610ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
26922024-02-15T02:58:05.610ZINFOcrucible: [2] client re-new 0 jobs since flush 1903 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
26932024-02-15T02:58:05.610ZINFOcrucible: client task is exiting = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
26942024-02-15T02:58:06.611ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
26952024-02-15T02:58:06.611ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
26962024-02-15T02:58:06.612ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
26972024-02-15T02:58:06.612ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
26982024-02-15T02:58:06.612ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
26992024-02-15T02:58:06.612ZINFOcrucible: [2] client re-new 0 jobs since flush 1903 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
27002024-02-15T02:58:06.612ZINFOcrucible: client task is exiting = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
2701 Feb 15 02:58:07.610 INFO request completed, latency_us: 210, response_code: 200, uri: /state/cid/2, method: GET, req_id: a88eb8b8-752e-44a0-9023-93367a39e0ed, remote_addr: 127.0.0.1:54744, local_addr: 127.0.0.1:9998
2702 001/300 Write block 2507 len 4096 data: 2
2703 002/300 Write block 284 len 40960 data: 3 3 3 2 2 2 2 2 2 2
27042024-02-15T02:58:07.612ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
27052024-02-15T02:58:07.612ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
27062024-02-15T02:58:07.612ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
2707 {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","003/300level" Write block 988 len 12288 data:: 440 4 3
2708 ,"time":"2024-02-15T02:58:07.612809882Z","hostname":"ip-10-150-1-67.us-west-2.compute.internal","pid":1167,"session_id":"4f5a63e5-aa00-4890-9f6b-96c2e069fdb2"}
27092024-02-15T02:58:07.612ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
27102024-02-15T02:58:07.612ZINFOcrucible: [2] client re-new 2 jobs since flush 1903 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
27112024-02-15T02:58:07.612ZINFOcrucible: client task is exiting = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
2712 004/300 Write block 1583 len 32768 data: 3 3 2 2 2 2 2 3
2713 005/300 Read block 3921 len 20480
2714 006/300 Write block 3511 len 4096 data: 2
2715 007/300 Read block 2425 len 32768
2716 008/300 Read block 3340 len 40960
2717 009/300 Flush
2718 010/300 Flush
2719 011/300 Write block 4617 len 32768 data: 2 2 2 2 3 3 2 3
2720 012/300 Write block 3865 len 36864 data: 3 3 3 3 3 4 4 3 3
2721 013/300 Read block 527 len 16384
2722 014/300 Write block 64 len 8192 data: 2 2
2723 015/300 Write block 4737 len 36864 data: 2 2 4 4 4 3 3 3 3
2724 016/300 Read block 1840 len 4096
2725 017/300 Read block 3680 len 40960
2726 018/300 Flush
2727 019/300 Write block 1188 len 32768 data: 2 2 2 2 2 2 2 2
2728 020/300 Write block 4314 len 8192 data: 2 2
2729 021/300 Write block 43 len 28672 data: 2 2 2 2 2 2 2
2730 022/300 Read block 1312 len 36864
2731 023/300 Write block 3190 len 12288 data: 3 3 2
2732 024/300 Write block 445 len 16384 data: 2 2 2 2
2733 025/300 Write block 4332 len 28672 data: 2 2 2 2 2 2 2
2734 026/300 Write block 2421 len 28672 data: 2 2 2 2 2 2 2
2735 027/300 Read block 3017 len 12288
2736 028/300 Read block 3228 len 40960
2737 029/300 Read block 884 len 16384
2738 030/300 Read block 2144 len 20480
2739 031/300 Read block 4226 len 24576
2740 032/300 Read block 2799 len 32768
2741 033/300 Flush
2742 034/300 Write block 1610 len 36864 data: 2 2 2 2 2 2 2 2 2
2743 035/300 Flush
2744 036/300 Write block 739 len 24576 data: 5 3 2 2 2 2
2745 037/300 Read block 444 len 12288
2746 038/300 Write block 1621 len 36864 data: 3 4 4 4 4 4 3 3 2
2747 039/300 Read block 3879 len 12288
2748 040/300 Flush
2749 041/300 Read block 980 len 36864
2750 042/300 Write block 2019 len 28672 data: 3 2 2 2 2 2 2
2751 043/300 Read block 4873 len 12288
2752 044/300 Write block 2605 len 4096 data: 2
2753 045/300 Flush
2754 046/300 Read block 1502 len 16384
2755 047/300 Write block 2739 len 12288 data: 2 2 3
2756 048/300 Read block 521 len 32768
2757 049/300 Read block 204 len 28672
2758 050/300 Flush
2759 051/300 Read block 4416 len 20480
2760 052/300 Read block 1630 len 12288
2761 053/300 Write block 56 len 12288 data: 4 4 4
2762 054/300 Read block 3712 len 4096
2763 055/300 Read block 3934 len 24576
2764 056/300 Write block 3825 len 8192 data: 2 2
2765 057/300 Write block 1777 len 28672 data: 2 2 2 2 2 2 3
2766 058/300 Read block 884 len 40960
2767 059/300 Read block 1696 len 8192
2768 060/300 Read block 1848 len 28672
2769 061/300 Read block 1721 len 32768
2770 062/300 Write block 578 len 28672 data: 2 2 3 3 3 3 4
2771 063/300 Read block 1029 len 12288
2772 064/300 Write block 3740 len 24576 data: 2 2 2 2 2 2
2773 065/300 Write block 857 len 40960 data: 3 3 3 3 2 2 2 2 2 2
2774 066/300 Read block 4138 len 40960
2775 067/300 Flush
2776 068/300 Write block 4603 len 8192 data: 2 2
2777 069/300 Write block 3460 len 12288 data: 2 2 2
2778 070/300 Write block 1785 len 20480 data: 3 3 3 3 2
2779 071/300 Read block 2689 len 16384
2780 072/300 Read block 3977 len 16384
2781 073/300 Write block 4778 len 16384 data: 2 2 2 2
2782 074/300 Read block 3500 len 12288
2783 075/300 Write block 1415 len 32768 data: 2 2 2 2 2 2 2 2
2784 076/300 Write block 1542 len 28672 data: 2 2 3 3 3 3 3
2785 077/300 Read block 1775 len 16384
2786 078/300 Read block 2031 len 4096
2787 079/300 Flush
2788 080/300 Read block 4232 len 24576
2789 081/300 Read block 3352 len 28672
2790 082/300 Write block 1135 len 16384 data: 3 3 3 3
2791 083/300 Write block 188 len 40960 data: 2 2 2 2 2 2 2 2 2 2
2792 084/300 Read block 3767 len 4096
2793 085/300 Flush
2794 086/300 Read block 2607 len 16384
2795 087/300 Read block 3384 len 32768
2796 088/300 Read block 1625 len 4096
2797 089/300 Write block 1017 len 28672 data: 2 2 2 3 3 3 3
2798 090/300 Read block 2243 len 16384
2799 091/300 Read block 1992 len 24576
2800 092/300 Flush
2801 093/300 Read block 4485 len 28672
2802 094/300 Write block 205 len 12288 data: 2 2 2
2803 095/300 Read block 3726 len 12288
2804 096/300 Read block 1367 len 4096
2805 097/300 Read block 30 len 28672
2806 098/300 Write block 2206 len 36864 data: 4 4 3 3 2 2 2 3 3
2807 099/300 Write block 1874 len 4096 data: 2
2808 100/300 Read block 823 len 4096
2809 101/300 Write block 2924 len 12288 data: 2 2 2
2810 102/300 Read block 3964 len 40960
2811 103/300 Read block 3464 len 8192
2812 104/300 Write block 4695 len 12288 data: 2 2 2
2813 105/300 Read block 2081 len 12288
2814 106/300 Read block 2868 len 12288
2815 107/300 Read block 665 len 8192
2816 108/300 Write block 1221 len 20480 data: 2 2 2 2 2
2817 109/300 Read block 1846 len 28672
2818 110/300 Read block 4935 len 12288
2819 111/300 Read block 3714 len 32768
2820 112/300 Write block 1722 len 4096 data: 3
2821 113/300 Write block 3111 len 32768 data: 2 2 2 2 2 2 2 2
2822 114/300 Read block 4519 len 4096
2823 115/300 Write block 789 len 36864 data: 3 3 2 2 2 3 3 4 4
2824 116/300 Read block 4160 len 12288
2825 117/300 Read block 1312 len 32768
2826 118/300 Write block 718 len 16384 data: 2 2 3 3
2827 119/300 Read block 3101 len 16384
2828 120/300 Read block 2961 len 16384
2829 121/300 Write block 1076 len 24576 data: 2 2 2 2 2 3
2830 122/300 Write block 3716 len 4096 data: 2
2831 123/300 Read block 390 len 40960
2832 124/300 Read block 1856 len 16384
2833 125/300 Read block 3559 len 16384
2834 126/300 Write block 23 len 32768 data: 4 4 4 4 4 3 3 2
2835 127/300 Read block 3906 len 12288
2836 128/300 Read block 4773 len 4096
2837 129/300 Write block 4111 len 32768 data: 3 3 3 3 3 3 3 3
2838 130/300 Read block 1430 len 20480
2839 131/300 Read block 2513 len 16384
2840 132/300 Write block 1977 len 24576 data: 2 2 2 2 2 2
2841 133/300 Read block 1807 len 20480
2842 134/300 Write block 2555 len 4096 data: 2
2843 135/300 Write block 357 len 12288 data: 2 2 2
2844 136/300 Write block 3014 len 20480 data: 2 2 2 2 2
2845 137/300 Flush
2846 138/300 Read block 961 len 20480
2847 139/300 Flush
2848 140/300 Flush
2849 141/300 Write block 4862 len 28672 data: 3 4 4 4 3 3 3
2850 142/300 Read block 4661 len 28672
2851 143/300 Read block 4118 len 36864
2852 144/300 Read block 107 len 28672
2853 145/300 Flush
2854 146/300 Read block 4779 len 40960
2855 147/300 Read block 528 len 16384
2856 148/300 Read block 2358 len 4096
2857 149/300 Read block 3532 len 40960
2858 150/300 Read block 3721 len 16384
2859 151/300 Write block 384 len 28672 data: 2 2 2 2 2 2 2
2860 152/300 Write block 195 len 20480 data: 3 3 3 2 2
2861 153/300 Read block 2495 len 40960
2862 154/300 Read block 2868 len 4096
2863 155/300 Read block 3318 len 36864
2864 156/300 Write block 2002 len 40960 data: 3 3 3 3 3 3 2 2 2 2
2865 157/300 Read block 3969 len 36864
2866 158/300 Write block 4922 len 28672 data: 2 2 2 2 2 2 2
2867 159/300 Flush
2868 160/300 Read block 3550 len 32768
2869 161/300 Read block 563 len 20480
2870 162/300 Read block 1236 len 12288
2871 163/300 Read block 4003 len 16384
2872 164/300 Write block 4044 len 4096 data: 3
2873 165/300 Write block 4768 len 8192 data: 5 4
2874 166/300 Write block 626 len 28672 data: 2 2 3 3 3 3 3
2875 167/300 Read block 4774 len 36864
2876 168/300 Write block 2732 len 16384 data: 2 2 2 2
2877 169/300 Read block 958 len 28672
2878 170/300 Read block 4090 len 28672
2879 171/300 Flush
2880 172/300 Write block 2001 len 12288 data: 4 4 4
2881 173/300 Read block 3495 len 20480
2882 174/300 Write block 2308 len 4096 data: 2
2883 175/300 Read block 1158 len 32768
2884 176/300 Write block 4691 len 16384 data: 3 3 2 2
2885 177/300 Write block 73 len 12288 data: 2 2 2
2886 178/300 Read block 2212 len 28672
2887 179/300 Read block 2440 len 20480
2888 180/300 Write block 713 len 16384 data: 2 2 2 2
2889 181/300 Read block 3347 len 16384
2890 182/300 Write block 288 len 32768 data: 3 3 3 3 3 3 2 2
2891 183/300 Write block 1823 len 36864 data: 2 2 2 2 2 3 3 3 3
2892 184/300 Read block 2033 len 4096
2893 185/300 Write block 1567 len 32768 data: 3 3 3 3 3 3 3 3
2894 186/300 Write block 2059 len 36864 data: 4 4 3 3 2 2 2 2 2
2895 187/300 Flush
2896 188/300 Read block 128 len 40960
2897 189/300 Read block 4377 len 12288
2898 190/300 Flush
2899 191/300 Write block 3712 len 36864 data: 2 2 2 2 3 2 2 2 2
2900 192/300 Read block 3143 len 32768
2901 193/300 Write block 4059 len 40960 data: 2 2 2 2 2 2 2 2 2 2
2902 194/300 Read block 3613 len 4096
2903 195/300 Flush
2904 196/300 Write block 1958 len 40960 data: 3 3 3 2 2 2 2 2 2 2
2905 197/300 Read block 4376 len 12288
2906 198/300 Read block 4722 len 32768
2907 199/300 Read block 1302 len 16384
2908 200/300 Write block 4478 len 8192 data: 2 2
2909 201/300 Read block 4711 len 24576
2910 202/300 Read block 3420 len 12288
2911 203/300 Flush
2912 204/300 Read block 1323 len 24576
2913 205/300 Read block 139 len 4096
2914 206/300 Flush
2915 207/300 Read block 4058 len 40960
2916 208/300 Read block 4663 len 8192
2917 209/300 Read block 1146 len 4096
2918 210/300 Write block 431 len 32768 data: 2 2 2 2 2 2 3 3
2919 211/300 Read block 2269 len 12288
2920 212/300 Read block 4976 len 24576
2921 213/300 Read block 3765 len 24576
2922 214/300 Write block 3108 len 32768 data: 2 2 2 3 3 3 3 3
2923 215/300 Read block 2784 len 16384
2924 216/300 Read block 909 len 4096
2925 217/300 Flush
2926 218/300 Flush
2927 219/300 Write block 4860 len 36864 data: 3 3 4 5 5 5 4 4 4
2928 220/300 Read block 159 len 28672
2929 221/300 Read block 685 len 16384
2930 222/300 Read block 602 len 32768
2931 223/300 Read block 4906 len 4096
2932 224/300 Read block 3642 len 36864
2933 225/300 Read block 1547 len 8192
2934 226/300 Read block 482 len 8192
2935 227/300 Write block 3052 len 32768 data: 2 2 2 2 2 2 3 3
2936 228/300 Write block 887 len 20480 data: 2 2 2 2 2
2937 229/300 Write block 2064 len 36864 data: 3 3 3 3 2 2 2 2 3
2938 230/300 Flush
2939 231/300 Write block 304 len 32768 data: 2 2 2 2 2 2 2 2
2940 232/300 Read block 2305 len 36864
2941 233/300 Read block 3598 len 20480
2942 234/300 Write block 4904 len 16384 data: 3 4 5 5
2943 235/300 Read block 4644 len 16384
2944 236/300 Read block 4705 len 40960
2945 237/300 Read block 259 len 16384
2946 238/300 Write block 4044 len 32768 data: 4 3 2 2 2 2 2 3
2947 239/300 Read block 793 len 32768
2948 240/300 Write block 4588 len 40960 data: 2 2 2 2 3 3 3 3 3 2
2949 241/300 Read block 2502 len 36864
2950 242/300 Read block 2677 len 32768
2951 243/300 Flush
2952 244/300 Read block 383 len 36864
2953 245/300 Write block 4277 len 4096 data: 2
2954 246/300 Read block 916 len 16384
2955 247/300 Write block 1537 len 24576 data: 3 4 4 3 3 3
2956 248/300 Write block 566 len 20480 data: 2 2 2 2 2
2957 249/300 Read block 1714 len 4096
2958 250/300 Write block 4011 len 20480 data: 2 3 3 3 3
2959 251/300 Write block 1498 len 4096 data: 2
2960 252/300 Write block 1 len 24576 data: 2 2 2 2 2 2
2961 253/300 Write block 4468 len 36864 data: 2 2 2 2 2 2 2 2 2
2962 254/300 Read block 1809 len 12288
2963 255/300 Read block 2439 len 12288
2964 256/300 Read block 3297 len 40960
2965 257/300 Read block 1078 len 8192
2966 258/300 Read block 724 len 32768
2967 259/300 Read block 534 len 12288
2968 260/300 Read block 3455 len 4096
2969 261/300 Read block 1418 len 28672
2970 262/300 Read block 410 len 16384
2971 263/300 Flush
2972 264/300 Read block 1530 len 32768
2973 265/300 Read block 3010 len 40960
2974 266/300 Write block 4261 len 12288 data: 3 3 3
2975 267/300 Read block 3651 len 16384
2976 268/300 Write block 373 len 24576 data: 3 3 2 2 3 3
2977 269/300 Flush
2978 270/300 Read block 601 len 36864
2979 271/300 Read block 2494 len 20480
2980 272/300 Read block 4424 len 36864
2981 273/300 Write block 2953 len 32768 data: 2 2 2 2 2 3 2 2
2982 274/300 Read block 1767 len 36864
2983 275/300 Read block 3028 len 28672
2984 276/300 Read block 4489 len 16384
2985 277/300 Write block 4569 len 8192 data: 2 2
2986 278/300 Read block 3624 len 16384
2987 279/300 Write block 4816 len 20480 data: 2 2 2 2 2
2988 280/300 Flush
2989 281/300 Flush
2990 282/300 Flush
2991 283/300 Read block 202 len 24576
2992 284/300 Read block 4545 len 12288
2993 285/300 Read block 2510 len 8192
2994 286/300 Write block 1737 len 20480 data: 3 3 2 2 2
2995 287/300 Read block 16 len 24576
2996 288/300 Read block 4938 len 24576
2997 289/300 Write block 4324 len 40960 data: 2 2 2 2 2 2 2 2 3 3
2998 290/300 Read block 2575 len 12288
2999 291/300 Read block 2160 len 12288
3000 292/300 Write block 4453 len 40960 data: 2 2 2 2 2 2 2 2 2 2
3001 293/300 Write block 2003 len 24576 data: 5 4 4 4 4 3
3002 294/300 Write block 2989 len 8192 data: 2 2
3003 295/300 Read block 4660 len 28672
3004 296/300 Read block 4401 len 4096
3005 297/300 Read block 4993 len 12288
3006 298/300 Write block 965 len 36864 data: 3 2 2 2 3 3 3 3 4
3007 299/300 Write block 4640 len 32768 data: 2 2 2 2 2 2 2 2
3008 300/300 Read block 1220 len 4096
3009 Main task has work to do, go find it
3010 got dsc Start(2)
3011 start 2
3012 Feb 15 02:58:07.947 INFO request completed, latency_us: 198, response_code: 204, uri: /start/cid/2, method: POST, req_id: c8e5817c-95f7-4bff-bfbf-4d7297d3b547, remote_addr: 127.0.0.1:54744, local_addr: 127.0.0.1:9998
3013 Replay: started 2, returned:Ok(())
3014 ----------------------------------------------------------------
3015 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:300
3016 States: Active Active Offline
3017 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
3018 905 Acked 1904 Write 1 Done Done New true
3019 906 Acked 1905 Write 10 Done Done New true
3020 907 Acked 1906 Write 3 Done Done New false
3021 908 Acked 1907 Write 8 Done Done New false
3022 909 Acked 1908 Read 5 Done Done New false
3023 910 Acked 1909 Write 1 Done Done New false
3024 911 Acked 1910 Read 8 Done Done New false
3025 912 Acked 1911 Read 10 Done Done New false
3026 913 Acked 1912 Flush 0 Done Done New false
3027 914 Acked 1913 Flush 0 Done Done New false
3028 915 Acked 1914 Write 8 Done Done New false
3029 916 Acked 1915 Write 9 Done Done New false
3030 917 Acked 1916 Read 4 Done Done New false
3031 918 Acked 1917 Write 2 Done Done New false
3032 919 Acked 1918 Write 9 Done Done New false
3033 920 Acked 1919 Read 1 Done Done New false
3034 921 Acked 1920 Read 10 Done Done New false
3035 922 Acked 1921 Flush 0 Done Done New false
3036 923 Acked 1922 Write 8 Done Done New false
3037 924 Acked 1923 Write 2 Done Done New false
3038 925 Acked 1924 Write 7 Done Done New false
3039 926 Acked 1925 Read 9 Done Done New false
3040 927 Acked 1926 Write 3 Done Done New false
3041 928 Acked 1927 Write 4 Done Done New false
3042 929 Acked 1928 Write 7 Done Done New false
3043 930 Acked 1929 Write 7 Done Done New false
3044 931 Acked 1930 Read 3 Done Done New false
3045 932 Acked 1931 Read 10 Done Done New false
3046 933 Acked 1932 Read 4 Done Done New false
3047 934 Acked 1933 Read 5 Done Done New false
3048 935 Acked 1934 Read 6 Done Done New false
3049 936 Acked 1935 Read 8 Done Done New false
3050 937 Acked 1936 Flush 0 Done Done New false
3051 938 Acked 1937 Write 9 Done Done New false
3052 939 Acked 1938 Flush 0 Done Done New false
3053 940 Acked 1939 Write 6 Done Done New false
3054 941 Acked 1940 Read 3 Done Done New false
3055 942 Acked 1941 Write 9 Done Done New false
3056 943 Acked 1942 Read 3 Done Done New false
3057 944 Acked 1943 Flush 0 Done Done New false
3058 945 Acked 1944 Read 9 Done Done New false
3059 946 Acked 1945 Write 7 Done Done New false
3060 947 Acked 1946 Read 3 Done Done New false
3061 948 Acked 1947 Write 1 Done Done New false
3062 949 Acked 1948 Flush 0 Done Done New false
3063 950 Acked 1949 Read 4 Done Done New false
3064 951 Acked 1950 Write 3 Done Done New false
3065 952 Acked 1951 Read 8 Done Done New false
3066 953 Acked 1952 Read 7 Done Done New false
3067 954 Acked 1953 Flush 0 Done Done New false
3068 955 Acked 1954 Read 5 Done Done New false
3069 956 Acked 1955 Read 3 Done Done New false
3070 957 Acked 1956 Write 3 Done Done New false
3071 958 Acked 1957 Read 1 Done Done New false
3072 959 Acked 1958 Read 6 Done Done New false
3073 960 Acked 1959 Write 2 Done Done New false
3074 961 Acked 1960 Write 7 Done Done New false
3075 962 Acked 1961 Read 10 Done Done New false
3076 963 Acked 1962 Read 2 Done Done New false
3077 964 Acked 1963 Read 7 Done Done New false
3078 965 Acked 1964 Read 8 Done Done New false
3079 966 Acked 1965 Write 7 Done Done New false
3080 967 Acked 1966 Read 3 Done Done New false
3081 968 Acked 1967 Write 6 Done Done New false
3082 969 Acked 1968 Write 10 Done Done New false
3083 970 Acked 1969 Read 10 Done Done New false
3084 971 Acked 1970 Flush 0 Done Done New false
3085 972 Acked 1971 Write 2 Done Done New false
3086 973 Acked 1972 Write 3 Done Done New false
3087 974 Acked 1973 Write 5 Done Done New false
3088 975 Acked 1974 Read 4 Done Done New false
3089 976 Acked 1975 Read 4 Done Done New false
3090 977 Acked 1976 Write 4 Done Done New false
3091 978 Acked 1977 Read 3 Done Done New false
3092 979 Acked 1978 Write 8 Done Done New false
3093 980 Acked 1979 Write 7 Done Done New false
3094 981 Acked 1980 Read 4 Done Done New false
3095 982 Acked 1981 Read 1 Done Done New false
3096 983 Acked 1982 Flush 0 Done Done New false
3097 984 Acked 1983 Read 6 Done Done New false
3098 985 Acked 1984 Read 7 Done Done New false
3099 986 Acked 1985 Write 4 Done Done New false
3100 987 Acked 1986 Write 10 Done Done New false
3101 988 Acked 1987 Read 1 Done Done New false
3102 989 Acked 1988 Flush 0 Done Done New false
3103 990 Acked 1989 Read 4 Done Done New false
3104 991 Acked 1990 Read 8 Done Done New false
3105 992 Acked 1991 Read 1 Done Done New false
3106 993 Acked 1992 Write 7 Done Done New false
3107 994 Acked 1993 Read 4 Done Done New false
3108 995 Acked 1994 Read 6 Done Done New false
3109 996 Acked 1995 Flush 0 Done Done New false
3110 997 Acked 1996 Read 7 Done Done New false
3111 998 Acked 1997 Write 3 Done Done New false
3112 999 Acked 1998 Read 3 Done Done New false
3113 1000 Acked 1999 Read 1 Done Done New false
3114 1001 Acked 2000 Read 7 Done Done New false
3115 1002 Acked 2001 Write 9 Done Done New false
3116 1003 Acked 2002 Write 1 Done Done New false
3117 1004 Acked 2003 Read 1 Done Done New false
3118 1005 Acked 2004 Write 3 Done Done New false
3119 1006 Acked 2005 Read 10 Done Done New false
3120 1007 Acked 2006 Read 2 Done Done New false
3121 1008 Acked 2007 Write 3 Done Done New false
3122 1009 Acked 2008 Read 3 Done Done New false
3123 1010 Acked 2009 Read 3 Done Done New false
3124 1011 Acked 2010 Read 2 Done Done New false
3125 1012 Acked 2011 Write 5 Done Done New false
3126 1013 Acked 2012 Read 7 Done Done New false
3127 1014 Acked 2013 Read 3 Done Done New false
3128 1015 Acked 2014 Read 8 Done Done New false
3129 1016 Acked 2015 Write 1 Done Done New false
3130 1017 Acked 2016 Write 8 Done Done New false
3131 1018 Acked 2017 Read 1 Done Done New false
3132 1019 Acked 2018 Write 9 Done Done New false
3133 1020 Acked 2019 Read 3 Done Done New false
3134 1021 Acked 2020 Read 8 Done Done New false
3135 1022 Acked 2021 Write 4 Done Done New false
3136 1023 Acked 2022 Read 4 Done Done New false
3137 1024 Acked 2023 Read 4 Done Done New false
3138 1025 Acked 2024 Write 6 Done Done New false
3139 1026 Acked 2025 Write 1 Done Done New false
3140 1027 Acked 2026 Read 10 Done Done New false
3141 1028 Acked 2027 Read 4 Done Done New false
3142 1029 Acked 2028 Read 4 Done Done New false
3143 1030 Acked 2029 Write 8 Done Done New false
3144 1031 Acked 2030 Read 3 Done Done New false
3145 1032 Acked 2031 Read 1 Done Done New false
3146 1033 Acked 2032 Write 8 Done Done New false
3147 1034 Acked 2033 Read 5 Done Done New false
3148 1035 Acked 2034 Read 4 Done Done New false
3149 1036 Acked 2035 Write 6 Done Done New false
3150 1037 Acked 2036 Read 5 Done Done New false
3151 1038 Acked 2037 Write 1 Done Done New false
3152 1039 Acked 2038 Write 3 Done Done New false
3153 1040 Acked 2039 Write 5 Done Done New false
3154 1041 Acked 2040 Flush 0 Done Done New false
3155 1042 Acked 2041 Read 5 Done Done New false
3156 1043 Acked 2042 Flush 0 Done Done New false
3157 1044 Acked 2043 Flush 0 Done Done New false
3158 1045 Acked 2044 Write 7 Done Done New false
3159 1046 Acked 2045 Read 7 Done Done New false
3160 1047 Acked 2046 Read 9 Done Done New false
3161 1048 Acked 2047 Read 7 Done Done New false
3162 1049 Acked 2048 Flush 0 Done Done New false
3163 1050 Acked 2049 Read 10 Done Done New false
3164 1051 Acked 2050 Read 4 Done Done New false
3165 1052 Acked 2051 Read 1 Done Done New false
3166 1053 Acked 2052 Read 10 Done Done New false
3167 1054 Acked 2053 Read 4 Done Done New false
3168 1055 Acked 2054 Write 7 Done Done New false
3169 1056 Acked 2055 Write 5 Done Done New false
3170 1057 Acked 2056 Read 10 Done Done New false
3171 1058 Acked 2057 Read 1 Done Done New false
3172 1059 Acked 2058 Read 9 Done Done New false
3173 1060 Acked 2059 Write 10 Done Done New false
3174 1061 Acked 2060 Read 9 Done Done New false
3175 1062 Acked 2061 Write 7 Done Done New false
3176 1063 Acked 2062 Flush 0 Done Done New false
3177 1064 Acked 2063 Read 8 Done Done New false
3178 1065 Acked 2064 Read 5 Done Done New false
3179 1066 Acked 2065 Read 3 Done Done New false
3180 1067 Acked 2066 Read 4 Done Done New false
3181 1068 Acked 2067 Write 1 Done Done New false
3182 1069 Acked 2068 Write 2 Done Done New false
3183 1070 Acked 2069 Write 7 Done Done New false
3184 1071 Acked 2070 Read 9 Done Done New false
3185 1072 Acked 2071 Write 4 Done Done New false
3186 1073 Acked 2072 Read 7 Done Done New false
3187 1074 Acked 2073 Read 7 Done Done New false
3188 1075 Acked 2074 Flush 0 Done Done New false
3189 1076 Acked 2075 Write 3 Done Done New false
3190 1077 Acked 2076 Read 5 Done Done New false
3191 1078 Acked 2077 Write 1 Done Done New false
3192 1079 Acked 2078 Read 8 Done Done New false
3193 1080 Acked 2079 Write 4 Done Done New false
3194 1081 Acked 2080 Write 3 Done Done New false
3195 1082 Acked 2081 Read 7 Done Done New false
3196 1083 Acked 2082 Read 5 Done Done New false
3197 1084 Acked 2083 Write 4 Done Done New false
3198 1085 Acked 2084 Read 4 Done Done New false
3199 1086 Acked 2085 Write 8 Done Done New false
3200 1087 Acked 2086 Write 9 Done Done New false
3201 1088 Acked 2087 Read 1 Done Done New false
3202 1089 Acked 2088 Write 8 Done Done New false
3203 1090 Acked 2089 Write 9 Done Done New false
3204 1091 Acked 2090 Flush 0 Done Done New false
3205 1092 Acked 2091 Read 10 Done Done New false
3206 1093 Acked 2092 Read 3 Done Done New false
3207 1094 Acked 2093 Flush 0 Done Done New false
3208 1095 Acked 2094 Write 9 Done Done New false
3209 1096 Acked 2095 Read 8 Done Done New false
3210 1097 Acked 2096 Write 10 Done Done New false
3211 1098 Acked 2097 Read 1 Done Done New false
3212 1099 Acked 2098 Flush 0 Done Done New false
3213 1100 Acked 2099 Write 10 Done Done New false
3214 1101 Acked 2100 Read 3 Done Done New false
3215 1102 Acked 2101 Read 8 Done Done New false
3216 1103 Acked 2102 Read 4 Done Done New false
3217 1104 Acked 2103 Write 2 Done Done New false
3218 1105 Acked 2104 Read 6 Done Done New false
3219 1106 Acked 2105 Read 3 Done Done New false
3220 1107 Acked 2106 Flush 0 Done Done New false
3221 1108 Acked 2107 Read 6 Done Done New false
3222 1109 Acked 2108 Read 1 Done Done New false
3223 1110 Acked 2109 Flush 0 Done Done New false
3224 1111 Acked 2110 Read 10 Done Done New false
3225 1112 Acked 2111 Read 2 Done Done New false
3226 1113 Acked 2112 Read 1 Done Done New false
3227 1114 Acked 2113 Write 8 Done Done New false
3228 1115 Acked 2114 Read 3 Done Done New false
3229 1116 Acked 2115 Read 6 Done Done New false
3230 1117 Acked 2116 Read 6 Done Done New false
3231 1118 Acked 2117 Write 8 Done Done New false
3232 1119 Acked 2118 Read 4 Done Done New false
3233 1120 Acked 2119 Read 1 Done Done New false
3234 1121 Acked 2120 Flush 0 Done Done New false
3235 1122 Acked 2121 Flush 0 Done Done New false
3236 1123 Acked 2122 Write 9 Done Done New false
3237 1124 Acked 2123 Read 7 Done Done New false
3238 1125 Acked 2124 Read 4 Done Done New false
3239 1126 Acked 2125 Read 8 Done Done New false
3240 1127 Acked 2126 Read 1 Done Done New false
3241 1128 Acked 2127 Read 9 Done Done New false
3242 1129 Acked 2128 Read 2 Done Done New false
3243 1130 Acked 2129 Read 2 Done Done New false
3244 1131 Acked 2130 Write 8 Done Done New false
3245 1132 Acked 2131 Write 5 Done Done New false
3246 1133 Acked 2132 Write 9 Done Done New false
3247 1134 Acked 2133 Flush 0 Done Done New false
3248 1135 Acked 2134 Write 8 Done Done New false
3249 1136 Acked 2135 Read 9 Done Done New false
3250 1137 Acked 2136 Read 5 Done Done New false
3251 1138 Acked 2137 Write 4 Done Done New false
3252 1139 Acked 2138 Read 4 Done Done New false
3253 1140 Acked 2139 Read 10 Done Done New false
3254 1141 Acked 2140 Read 4 Done Done New false
3255 1142 Acked 2141 Write 8 Done Done New false
3256 1143 Acked 2142 Read 8 Done Done New false
3257 1144 Acked 2143 Write 10 Done Done New false
3258 1145 Acked 2144 Read 9 Done Done New false
3259 1146 Acked 2145 Read 8 Done Done New false
3260 1147 Acked 2146 Flush 0 Done Done New false
3261 1148 Acked 2147 Read 9 Done Done New false
3262 1149 Acked 2148 Write 1 Done Done New false
3263 1150 Acked 2149 Read 4 Done Done New false
3264 1151 Acked 2150 Write 6 Done Done New false
3265 1152 Acked 2151 Write 5 Done Done New false
3266 1153 Acked 2152 Read 1 Done Done New false
3267 1154 Acked 2153 Write 5 Done Done New false
3268 1155 Acked 2154 Write 1 Done Done New false
3269 1156 Acked 2155 Write 6 Done Done New false
3270 1157 Acked 2156 Write 9 Done Done New false
3271 1158 Acked 2157 Read 3 Done Done New false
3272 1159 Acked 2158 Read 3 Done Done New false
3273 1160 Acked 2159 Read 10 Done Done New false
3274 1161 Acked 2160 Read 2 Done Done New false
3275 1162 Acked 2161 Read 8 Done Done New false
3276 1163 Acked 2162 Read 3 Done Done New false
3277 1164 Acked 2163 Read 1 Done Done New false
3278 1165 Acked 2164 Read 7 Done Done New false
3279 1166 Acked 2165 Read 4 Done Done New false
3280 1167 Acked 2166 Flush 0 Done Done New false
3281 1168 Acked 2167 Read 8 Done Done New false
3282 1169 Acked 2168 Read 10 Done Done New false
3283 1170 Acked 2169 Write 3 Done Done New false
3284 1171 Acked 2170 Read 4 Done Done New false
3285 1172 Acked 2171 Write 6 Done Done New false
3286 1173 Acked 2172 Flush 0 Done Done New false
3287 1174 Acked 2173 Read 9 Done Done New false
3288 1175 Acked 2174 Read 5 Done Done New false
3289 1176 Acked 2175 Read 9 Done Done New false
3290 1177 Acked 2176 Write 8 Done Done New false
3291 1178 Acked 2177 Read 9 Done Done New false
3292 1179 Acked 2178 Read 7 Done Done New false
3293 1180 Acked 2179 Read 4 Done Done New false
3294 1181 Acked 2180 Write 2 Done Done New false
3295 1182 Acked 2181 Read 4 Done Done New false
3296 1183 Acked 2182 Write 5 Done Done New false
3297 1184 Acked 2183 Flush 0 Done Done New false
3298 1185 Acked 2184 Flush 0 Done Done New false
3299 1186 Acked 2185 Flush 0 Done Done New false
3300 1187 Acked 2186 Read 6 Done Done New false
3301 1188 Acked 2187 Read 3 Done Done New false
3302 1189 Acked 2188 Read 2 Done Done New false
3303 1190 Acked 2189 Write 5 Done Done New false
3304 1191 Acked 2190 Read 6 Done Done New false
3305 1192 Acked 2191 Read 6 Done Done New false
3306 1193 Acked 2192 Write 10 Done Done New false
3307 1194 Acked 2193 Read 3 Done Done New false
3308 1195 Acked 2194 Read 3 Done Done New false
3309 1196 Acked 2195 Write 10 Done Done New false
3310 1197 Acked 2196 Write 6 Done Done New false
3311 1198 Acked 2197 Write 2 Done Done New false
3312 1199 Acked 2198 Read 7 Done Done New false
3313 1200 Acked 2199 Read 1 Done Done New false
3314 1201 Acked 2200 Read 3 Done Done New false
3315 1202 Acked 2201 Write 9 Done Done New false
3316 1203 Acked 2202 Write 8 Done Done New false
3317 1204 Acked 2203 Read 1 Done Done New false
3318 STATES DS:0 DS:1 DS:2 TOTAL
3319 New 0 0 300 300
3320 Sent 0 0 0 0
3321 Done 300 300 0 600
3322 Skipped 0 0 0 0
3323 Error 0 0 0 0
3324 Last Flush: 2185 2185 1903
3325 Downstairs last five completed: 1903 1902 1901 1900 1899
3326 Upstairs last five completed: 1204 1203 1202 1201 1200
3327 CLIENT: Up:0 ds:300 act:2
33282024-02-15T02:58:08.614ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
33292024-02-15T02:58:08.614ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
33302024-02-15T02:58:08.614ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
33312024-02-15T02:58:08.614ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
33322024-02-15T02:58:08.614ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
33332024-02-15T02:58:08.614ZINFOcrucible: [2] client re-new 301 jobs since flush 1903 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
33342024-02-15T02:58:08.614ZINFOcrucible: client task is exiting = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
33352024-02-15T02:58:09.616ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
33362024-02-15T02:58:09.616ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
33372024-02-15T02:58:09.616ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
33382024-02-15T02:58:09.616ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
33392024-02-15T02:58:09.616ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
33402024-02-15T02:58:09.616ZINFOcrucible: [2] client re-new 301 jobs since flush 1903 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
33412024-02-15T02:58:09.616ZINFOcrucible: client task is exiting = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
33422024-02-15T02:58:10.618ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
33432024-02-15T02:58:10.618ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
33442024-02-15T02:58:10.618ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
33452024-02-15T02:58:10.618ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
33462024-02-15T02:58:10.618ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
33472024-02-15T02:58:10.618ZINFOcrucible: [2] client re-new 301 jobs since flush 1903 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
33482024-02-15T02:58:10.618ZINFOcrucible: client task is exiting = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
33492024-02-15T02:58:11.620ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
33502024-02-15T02:58:11.620ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
33512024-02-15T02:58:11.620ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
33522024-02-15T02:58:11.620ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
33532024-02-15T02:58:11.620ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
33542024-02-15T02:58:11.620ZINFOcrucible: [2] client re-new 301 jobs since flush 1903 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
33552024-02-15T02:58:11.620ZINFOcrucible: client task is exiting = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
3356 ----------------------------------------------------------------
3357 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
3358 States: Active Active Offline
3359 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
3360 905 Acked 1904 Write 1 Done Done New true
3361 906 Acked 1905 Write 10 Done Done New true
3362 907 Acked 1906 Write 3 Done Done New true
3363 908 Acked 1907 Write 8 Done Done New true
3364 909 Acked 1908 Read 5 Done Done New true
3365 910 Acked 1909 Write 1 Done Done New true
3366 911 Acked 1910 Read 8 Done Done New true
3367 912 Acked 1911 Read 10 Done Done New true
3368 913 Acked 1912 Flush 0 Done Done New true
3369 914 Acked 1913 Flush 0 Done Done New true
3370 915 Acked 1914 Write 8 Done Done New true
3371 916 Acked 1915 Write 9 Done Done New true
3372 917 Acked 1916 Read 4 Done Done New true
3373 918 Acked 1917 Write 2 Done Done New true
3374 919 Acked 1918 Write 9 Done Done New true
3375 920 Acked 1919 Read 1 Done Done New true
3376 921 Acked 1920 Read 10 Done Done New true
3377 922 Acked 1921 Flush 0 Done Done New true
3378 923 Acked 1922 Write 8 Done Done New true
3379 924 Acked 1923 Write 2 Done Done New true
3380 925 Acked 1924 Write 7 Done Done New true
3381 926 Acked 1925 Read 9 Done Done New true
3382 927 Acked 1926 Write 3 Done Done New true
3383 928 Acked 1927 Write 4 Done Done New true
3384 929 Acked 1928 Write 7 Done Done New true
3385 930 Acked 1929 Write 7 Done Done New true
3386 931 Acked 1930 Read 3 Done Done New true
3387 932 Acked 1931 Read 10 Done Done New true
3388 933 Acked 1932 Read 4 Done Done New true
3389 934 Acked 1933 Read 5 Done Done New true
3390 935 Acked 1934 Read 6 Done Done New true
3391 936 Acked 1935 Read 8 Done Done New true
3392 937 Acked 1936 Flush 0 Done Done New true
3393 938 Acked 1937 Write 9 Done Done New true
3394 939 Acked 1938 Flush 0 Done Done New true
3395 940 Acked 1939 Write 6 Done Done New true
3396 941 Acked 1940 Read 3 Done Done New true
3397 942 Acked 1941 Write 9 Done Done New true
3398 943 Acked 1942 Read 3 Done Done New true
3399 944 Acked 1943 Flush 0 Done Done New true
3400 945 Acked 1944 Read 9 Done Done New true
3401 946 Acked 1945 Write 7 Done Done New true
3402 947 Acked 1946 Read 3 Done Done New true
3403 948 Acked 1947 Write 1 Done Done New true
3404 949 Acked 1948 Flush 0 Done Done New true
3405 950 Acked 1949 Read 4 Done Done New true
3406 951 Acked 1950 Write 3 Done Done New true
3407 952 Acked 1951 Read 8 Done Done New true
3408 953 Acked 1952 Read 7 Done Done New true
3409 954 Acked 1953 Flush 0 Done Done New true
3410 955 Acked 1954 Read 5 Done Done New true
3411 956 Acked 1955 Read 3 Done Done New true
3412 957 Acked 1956 Write 3 Done Done New true
3413 958 Acked 1957 Read 1 Done Done New true
3414 959 Acked 1958 Read 6 Done Done New true
3415 960 Acked 1959 Write 2 Done Done New true
3416 961 Acked 1960 Write 7 Done Done New true
3417 962 Acked 1961 Read 10 Done Done New true
3418 963 Acked 1962 Read 2 Done Done New true
3419 964 Acked 1963 Read 7 Done Done New true
3420 965 Acked 1964 Read 8 Done Done New true
3421 966 Acked 1965 Write 7 Done Done New true
3422 967 Acked 1966 Read 3 Done Done New true
3423 968 Acked 1967 Write 6 Done Done New true
3424 969 Acked 1968 Write 10 Done Done New true
3425 970 Acked 1969 Read 10 Done Done New true
3426 971 Acked 1970 Flush 0 Done Done New true
3427 972 Acked 1971 Write 2 Done Done New true
3428 973 Acked 1972 Write 3 Done Done New true
3429 974 Acked 1973 Write 5 Done Done New true
3430 975 Acked 1974 Read 4 Done Done New true
3431 976 Acked 1975 Read 4 Done Done New true
3432 977 Acked 1976 Write 4 Done Done New true
3433 978 Acked 1977 Read 3 Done Done New true
3434 979 Acked 1978 Write 8 Done Done New true
3435 980 Acked 1979 Write 7 Done Done New true
3436 981 Acked 1980 Read 4 Done Done New true
3437 982 Acked 1981 Read 1 Done Done New true
3438 983 Acked 1982 Flush 0 Done Done New true
3439 984 Acked 1983 Read 6 Done Done New true
3440 985 Acked 1984 Read 7 Done Done New true
3441 986 Acked 1985 Write 4 Done Done New true
3442 987 Acked 1986 Write 10 Done Done New true
3443 988 Acked 1987 Read 1 Done Done New true
3444 989 Acked 1988 Flush 0 Done Done New true
3445 990 Acked 1989 Read 4 Done Done New true
3446 991 Acked 1990 Read 8 Done Done New true
3447 992 Acked 1991 Read 1 Done Done New true
3448 993 Acked 1992 Write 7 Done Done New true
3449 994 Acked 1993 Read 4 Done Done New true
3450 995 Acked 1994 Read 6 Done Done New true
3451 996 Acked 1995 Flush 0 Done Done New true
3452 997 Acked 1996 Read 7 Done Done New true
3453 998 Acked 1997 Write 3 Done Done New true
3454 999 Acked 1998 Read 3 Done Done New true
3455 1000 Acked 1999 Read 1 Done Done New true
3456 1001 Acked 2000 Read 7 Done Done New true
3457 1002 Acked 2001 Write 9 Done Done New true
3458 1003 Acked 2002 Write 1 Done Done New true
3459 1004 Acked 2003 Read 1 Done Done New true
3460 1005 Acked 2004 Write 3 Done Done New true
3461 1006 Acked 2005 Read 10 Done Done New true
3462 1007 Acked 2006 Read 2 Done Done New true
3463 1008 Acked 2007 Write 3 Done Done New true
3464 1009 Acked 2008 Read 3 Done Done New true
3465 1010 Acked 2009 Read 3 Done Done New true
3466 1011 Acked 2010 Read 2 Done Done New true
3467 1012 Acked 2011 Write 5 Done Done New true
3468 1013 Acked 2012 Read 7 Done Done New true
3469 1014 Acked 2013 Read 3 Done Done New true
3470 1015 Acked 2014 Read 8 Done Done New true
3471 1016 Acked 2015 Write 1 Done Done New true
3472 1017 Acked 2016 Write 8 Done Done New true
3473 1018 Acked 2017 Read 1 Done Done New true
3474 1019 Acked 2018 Write 9 Done Done New true
3475 1020 Acked 2019 Read 3 Done Done New true
3476 1021 Acked 2020 Read 8 Done Done New true
3477 1022 Acked 2021 Write 4 Done Done New true
3478 1023 Acked 2022 Read 4 Done Done New true
3479 1024 Acked 2023 Read 4 Done Done New true
3480 1025 Acked 2024 Write 6 Done Done New true
3481 1026 Acked 2025 Write 1 Done Done New true
3482 1027 Acked 2026 Read 10 Done Done New true
3483 1028 Acked 2027 Read 4 Done Done New true
3484 1029 Acked 2028 Read 4 Done Done New true
3485 1030 Acked 2029 Write 8 Done Done New true
3486 1031 Acked 2030 Read 3 Done Done New true
3487 1032 Acked 2031 Read 1 Done Done New true
3488 1033 Acked 2032 Write 8 Done Done New true
3489 1034 Acked 2033 Read 5 Done Done New true
3490 1035 Acked 2034 Read 4 Done Done New true
3491 1036 Acked 2035 Write 6 Done Done New true
3492 1037 Acked 2036 Read 5 Done Done New true
3493 1038 Acked 2037 Write 1 Done Done New true
3494 1039 Acked 2038 Write 3 Done Done New true
3495 1040 Acked 2039 Write 5 Done Done New true
3496 1041 Acked 2040 Flush 0 Done Done New true
3497 1042 Acked 2041 Read 5 Done Done New true
3498 1043 Acked 2042 Flush 0 Done Done New true
3499 1044 Acked 2043 Flush 0 Done Done New true
3500 1045 Acked 2044 Write 7 Done Done New true
3501 1046 Acked 2045 Read 7 Done Done New true
3502 1047 Acked 2046 Read 9 Done Done New true
3503 1048 Acked 2047 Read 7 Done Done New true
3504 1049 Acked 2048 Flush 0 Done Done New true
3505 1050 Acked 2049 Read 10 Done Done New true
3506 1051 Acked 2050 Read 4 Done Done New true
3507 1052 Acked 2051 Read 1 Done Done New true
3508 1053 Acked 2052 Read 10 Done Done New true
3509 1054 Acked 2053 Read 4 Done Done New true
3510 1055 Acked 2054 Write 7 Done Done New true
3511 1056 Acked 2055 Write 5 Done Done New true
3512 1057 Acked 2056 Read 10 Done Done New true
3513 1058 Acked 2057 Read 1 Done Done New true
3514 1059 Acked 2058 Read 9 Done Done New true
3515 1060 Acked 2059 Write 10 Done Done New true
3516 1061 Acked 2060 Read 9 Done Done New true
3517 1062 Acked 2061 Write 7 Done Done New true
3518 1063 Acked 2062 Flush 0 Done Done New true
3519 1064 Acked 2063 Read 8 Done Done New true
3520 1065 Acked 2064 Read 5 Done Done New true
3521 1066 Acked 2065 Read 3 Done Done New true
3522 1067 Acked 2066 Read 4 Done Done New true
3523 1068 Acked 2067 Write 1 Done Done New true
3524 1069 Acked 2068 Write 2 Done Done New true
3525 1070 Acked 2069 Write 7 Done Done New true
3526 1071 Acked 2070 Read 9 Done Done New true
3527 1072 Acked 2071 Write 4 Done Done New true
3528 1073 Acked 2072 Read 7 Done Done New true
3529 1074 Acked 2073 Read 7 Done Done New true
3530 1075 Acked 2074 Flush 0 Done Done New true
3531 1076 Acked 2075 Write 3 Done Done New true
3532 1077 Acked 2076 Read 5 Done Done New true
3533 1078 Acked 2077 Write 1 Done Done New true
3534 1079 Acked 2078 Read 8 Done Done New true
3535 1080 Acked 2079 Write 4 Done Done New true
3536 1081 Acked 2080 Write 3 Done Done New true
3537 1082 Acked 2081 Read 7 Done Done New true
3538 1083 Acked 2082 Read 5 Done Done New true
3539 1084 Acked 2083 Write 4 Done Done New true
3540 1085 Acked 2084 Read 4 Done Done New true
3541 1086 Acked 2085 Write 8 Done Done New true
3542 1087 Acked 2086 Write 9 Done Done New true
3543 1088 Acked 2087 Read 1 Done Done New true
3544 1089 Acked 2088 Write 8 Done Done New true
3545 1090 Acked 2089 Write 9 Done Done New true
3546 1091 Acked 2090 Flush 0 Done Done New true
3547 1092 Acked 2091 Read 10 Done Done New true
3548 1093 Acked 2092 Read 3 Done Done New true
3549 1094 Acked 2093 Flush 0 Done Done New true
3550 1095 Acked 2094 Write 9 Done Done New true
3551 1096 Acked 2095 Read 8 Done Done New true
3552 1097 Acked 2096 Write 10 Done Done New true
3553 1098 Acked 2097 Read 1 Done Done New true
3554 1099 Acked 2098 Flush 0 Done Done New true
3555 1100 Acked 2099 Write 10 Done Done New true
3556 1101 Acked 2100 Read 3 Done Done New true
3557 1102 Acked 2101 Read 8 Done Done New true
3558 1103 Acked 2102 Read 4 Done Done New true
3559 1104 Acked 2103 Write 2 Done Done New true
3560 1105 Acked 2104 Read 6 Done Done New true
3561 1106 Acked 2105 Read 3 Done Done New true
3562 1107 Acked 2106 Flush 0 Done Done New true
3563 1108 Acked 2107 Read 6 Done Done New true
3564 1109 Acked 2108 Read 1 Done Done New true
3565 1110 Acked 2109 Flush 0 Done Done New true
3566 1111 Acked 2110 Read 10 Done Done New true
3567 1112 Acked 2111 Read 2 Done Done New true
3568 1113 Acked 2112 Read 1 Done Done New true
3569 1114 Acked 2113 Write 8 Done Done New true
3570 1115 Acked 2114 Read 3 Done Done New true
3571 1116 Acked 2115 Read 6 Done Done New true
3572 1117 Acked 2116 Read 6 Done Done New true
3573 1118 Acked 2117 Write 8 Done Done New true
3574 1119 Acked 2118 Read 4 Done Done New true
3575 1120 Acked 2119 Read 1 Done Done New true
3576 1121 Acked 2120 Flush 0 Done Done New true
3577 1122 Acked 2121 Flush 0 Done Done New true
3578 1123 Acked 2122 Write 9 Done Done New true
3579 1124 Acked 2123 Read 7 Done Done New true
3580 1125 Acked 2124 Read 4 Done Done New true
3581 1126 Acked 2125 Read 8 Done Done New true
3582 1127 Acked 2126 Read 1 Done Done New true
3583 1128 Acked 2127 Read 9 Done Done New true
3584 1129 Acked 2128 Read 2 Done Done New true
3585 1130 Acked 2129 Read 2 Done Done New true
3586 1131 Acked 2130 Write 8 Done Done New true
3587 1132 Acked 2131 Write 5 Done Done New true
3588 1133 Acked 2132 Write 9 Done Done New true
3589 1134 Acked 2133 Flush 0 Done Done New true
3590 1135 Acked 2134 Write 8 Done Done New true
3591 1136 Acked 2135 Read 9 Done Done New true
3592 1137 Acked 2136 Read 5 Done Done New true
3593 1138 Acked 2137 Write 4 Done Done New true
3594 1139 Acked 2138 Read 4 Done Done New true
3595 1140 Acked 2139 Read 10 Done Done New true
3596 1141 Acked 2140 Read 4 Done Done New true
3597 1142 Acked 2141 Write 8 Done Done New true
3598 1143 Acked 2142 Read 8 Done Done New true
3599 1144 Acked 2143 Write 10 Done Done New true
3600 1145 Acked 2144 Read 9 Done Done New true
3601 1146 Acked 2145 Read 8 Done Done New true
3602 1147 Acked 2146 Flush 0 Done Done New true
3603 1148 Acked 2147 Read 9 Done Done New true
3604 1149 Acked 2148 Write 1 Done Done New true
3605 1150 Acked 2149 Read 4 Done Done New true
3606 1151 Acked 2150 Write 6 Done Done New true
3607 1152 Acked 2151 Write 5 Done Done New true
3608 1153 Acked 2152 Read 1 Done Done New true
3609 1154 Acked 2153 Write 5 Done Done New true
3610 1155 Acked 2154 Write 1 Done Done New true
3611 1156 Acked 2155 Write 6 Done Done New true
3612 1157 Acked 2156 Write 9 Done Done New true
3613 1158 Acked 2157 Read 3 Done Done New true
3614 1159 Acked 2158 Read 3 Done Done New true
3615 1160 Acked 2159 Read 10 Done Done New true
3616 1161 Acked 2160 Read 2 Done Done New true
3617 1162 Acked 2161 Read 8 Done Done New true
3618 1163 Acked 2162 Read 3 Done Done New true
3619 1164 Acked 2163 Read 1 Done Done New true
3620 1165 Acked 2164 Read 7 Done Done New true
3621 1166 Acked 2165 Read 4 Done Done New true
3622 1167 Acked 2166 Flush 0 Done Done New true
3623 1168 Acked 2167 Read 8 Done Done New true
3624 1169 Acked 2168 Read 10 Done Done New true
3625 1170 Acked 2169 Write 3 Done Done New true
3626 1171 Acked 2170 Read 4 Done Done New true
3627 1172 Acked 2171 Write 6 Done Done New true
3628 1173 Acked 2172 Flush 0 Done Done New true
3629 1174 Acked 2173 Read 9 Done Done New true
3630 1175 Acked 2174 Read 5 Done Done New true
3631 1176 Acked 2175 Read 9 Done Done New true
3632 1177 Acked 2176 Write 8 Done Done New true
3633 1178 Acked 2177 Read 9 Done Done New true
3634 1179 Acked 2178 Read 7 Done Done New true
3635 1180 Acked 2179 Read 4 Done Done New true
3636 1181 Acked 2180 Write 2 Done Done New true
3637 1182 Acked 2181 Read 4 Done Done New true
3638 1183 Acked 2182 Write 5 Done Done New true
3639 1184 Acked 2183 Flush 0 Done Done New true
3640 1185 Acked 2184 Flush 0 Done Done New true
3641 1186 Acked 2185 Flush 0 Done Done New true
3642 1187 Acked 2186 Read 6 Done Done New true
3643 1188 Acked 2187 Read 3 Done Done New true
3644 1189 Acked 2188 Read 2 Done Done New true
3645 1190 Acked 2189 Write 5 Done Done New true
3646 1191 Acked 2190 Read 6 Done Done New true
3647 1192 Acked 2191 Read 6 Done Done New true
3648 1193 Acked 2192 Write 10 Done Done New true
3649 1194 Acked 2193 Read 3 Done Done New true
3650 1195 Acked 2194 Read 3 Done Done New true
3651 1196 Acked 2195 Write 10 Done Done New true
3652 1197 Acked 2196 Write 6 Done Done New true
3653 1198 Acked 2197 Write 2 Done Done New true
3654 1199 Acked 2198 Read 7 Done Done New true
3655 1200 Acked 2199 Read 1 Done Done New true
3656 1201 Acked 2200 Read 3 Done Done New true
3657 1202 Acked 2201 Write 9 Done Done New true
3658 1203 Acked 2202 Write 8 Done Done New true
3659 1204 Acked 2203 Read 1 Done Done New true
3660 1205 Acked 2204 Flush 0 Done Done New true
3661 STATES DS:0 DS:1 DS:2 TOTAL
3662 New 0 0 301 301
3663 Sent 0 0 0 0
3664 Done 301 301 0 602
3665 Skipped 0 0 0 0
3666 Error 0 0 0 0
3667 Last Flush: 2204 2204 1903
3668 Downstairs last five completed: 1903 1902 1901 1900 1899
3669 Upstairs last five completed: 1205 1204 1203 1202 1201
3670 CLIENT: Up:0 ds:301 act:2
36712024-02-15T02:58:12.621ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
36722024-02-15T02:58:12.621ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
36732024-02-15T02:58:12.621ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
36742024-02-15T02:58:12.621ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
36752024-02-15T02:58:12.621ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
36762024-02-15T02:58:12.621ZINFOcrucible: [2] client re-new 301 jobs since flush 1903 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
36772024-02-15T02:58:12.621ZINFOcrucible: client task is exiting = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
3678 [8830] got start action
3679 [8830] I am going to restart
3680 Starting downstairs at port 8830
3681 Make output file at "/tmp/dsc/downstairs-8830.txt"
3682 [8830][2] reports Starting
3683 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1171)
3684 [8830][2] reports Running
36852024-02-15T02:58:13.622ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
36862024-02-15T02:58:13.622ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
36872024-02-15T02:58:13.622ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
36882024-02-15T02:58:13.622ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
36892024-02-15T02:58:13.622ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
36902024-02-15T02:58:13.622ZINFOcrucible: [2] client re-new 301 jobs since flush 1903 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
36912024-02-15T02:58:13.622ZINFOcrucible: client task is exiting = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
36922024-02-15T02:58:14.624ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
36932024-02-15T02:58:14.624ZINFOcrucible: ds_connection connected = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
36942024-02-15T02:58:14.626ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
36952024-02-15T02:58:14.626ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
36962024-02-15T02:58:14.626ZINFOcrucible: send last flush ID to this DS: 1903 = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
36972024-02-15T02:58:14.626ZINFOcrucible: Replied this last flush ID: 1903 = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
36982024-02-15T02:58:14.626ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
36992024-02-15T02:58:14.626ZINFOcrucible: [2] Transition from Offline to Replay = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
37002024-02-15T02:58:14.626ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
37012024-02-15T02:58:14.626ZINFOcrucible: [2] Transition from Replay to Active = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
3702 ----------------------------------------------------------------
3703 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
3704 Downstairs last five completed: 2204 2203 2202 2201 2200
3705 Upstairs last five completed: 1205 1204 1203 1202 1201
3706 CLIENT: Up:0 ds:0 act:3
3707 Replay: All jobs finished, all DS active.
3708 Main task has work to do, go find it
3709 got dsc Stop(1)
3710 stop 1
3711 Feb 15 02:58:15.953 INFO request completed, latency_us: 337, response_code:[8820] Got stop action so:false kr:false
3712 204, uri: /stop/cid/1, method: POST, req_id: 11c8b1cb-7e03-4914-9443-d6414ec3ddba, remote_addr: 127.0.0.1:54744, local_addr: 127.0.0.1:9998
37132024-02-15T02:58:15.954ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
37142024-02-15T02:58:15.954ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
37152024-02-15T02:58:15.954ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
37162024-02-15T02:58:15.954ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
37172024-02-15T02:58:15.954ZINFOcrucible: [1] client re-new 0 jobs since flush 2204 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
3718 Feb 15 02:58:15.954 INFO request completed, latency_us: 132, response_code: {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-15T02:58:15.954845362Z","hostname":"ip-10-150-1-67.us-west-2.compute.internal","pid":1167,"":"io task","client":"1","":"downstairs","session_id":"4f5a63e5-aa00-4890-9f6b-96c2e069fdb2"}
3719 200, uri: /state/cid/1, method: GET, req_id: 189cf8a4-e4d8-45cb-87c9-fc4c030a0171, remote_addr: 127.0.0.1:54744, local_addr: 127.0.0.1:9998
3720 [8820] Exited with: ExitStatus(unix_wait_status(9))
3721 [8820][1] reports Exit
37222024-02-15T02:58:16.955ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
37232024-02-15T02:58:16.955ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
37242024-02-15T02:58:16.955ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
37252024-02-15T02:58:16.955ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
37262024-02-15T02:58:16.955ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
37272024-02-15T02:58:16.955ZINFOcrucible: [1] client re-new 0 jobs since flush 2204 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
37282024-02-15T02:58:16.955ZINFOcrucible: client task is exiting = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
37292024-02-15T02:58:17.956ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
37302024-02-15T02:58:17.956ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
37312024-02-15T02:58:17.956ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
37322024-02-15T02:58:17.956ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
37332024-02-15T02:58:17.956ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
37342024-02-15T02:58:17.956ZINFOcrucible: [1] client re-new 0 jobs since flush 2204 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
37352024-02-15T02:58:17.956ZINFOcrucible: client task is exiting = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
37362024-02-15T02:58:18.958ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
37372024-02-15T02:58:18.958ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
37382024-02-15T02:58:18.958ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
37392024-02-15T02:58:18.958ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
37402024-02-15T02:58:18.958ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
37412024-02-15T02:58:18.958ZINFOcrucible: [1] client re-new 0 jobs since flush 2204 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
37422024-02-15T02:58:18.958ZINFOcrucible: client task is exiting = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
3743 Feb 15 02:58:19.957 INFO request completed, latency_us: 382, response_code: 200, uri: /state/cid/1, method: GET, req_id: aed75c02-557c-4868-99e1-00ea7ac024e4, remote_addr: 127.0.0.1:54744, local_addr: 127.0.0.1:9998
3744 001/300 Read block 2148 len 8192
3745 002/300 Read block 2776 len 28672
37462024-02-15T02:58:19.958ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
37472024-02-15T02:58:19.958ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
37482024-02-15T02:58:19.958ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
37492024-02-15T02:58:19.958ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
37502024-02-15T02:58:19.958ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
37512024-02-15T02:58:19.958ZINFOcrucible: [1] client re-new 2 jobs since flush 2204 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
37522024-02-15T02:58:19.958ZINFOcrucible: client task is exiting = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
3753 003/300 Flush
3754 004/300 Read block 4127 len 12288
3755 005/300 Read block 2295 len 24576
3756 006/300 Write block 4305 len 36864 data: 2 2 2 2 2 2 2 2 2
3757 007/300 Read block 992 len 20480
3758 008/300 Read block 3554 len 12288
3759 009/300 Read block 1613 len 36864
3760 010/300 Read block 4431 len 36864
3761 011/300 Flush
3762 012/300 Read block 2016 len 28672
3763 013/300 Flush
3764 014/300 Read block 1866 len 32768
3765 015/300 Read block 596 len 36864
3766 016/300 Write block 4548 len 28672 data: 2 2 2 2 2 2 2
3767 017/300 Read block 4587 len 16384
3768 018/300 Flush
3769 019/300 Flush
3770 020/300 Write block 1885 len 16384 data: 2 2 2 2
3771 021/300 Read block 4769 len 40960
3772 022/300 Read block 2226 len 32768
3773 023/300 Write block 1951 len 24576 data: 2 2 2 2 2 2
3774 024/300 Read block 4872 len 24576
3775 025/300 Read block 697 len 20480
3776 026/300 Read block 16 len 40960
3777 027/300 Read block 2124 len 32768
3778 028/300 Write block 1734 len 40960 data: 4 3 3 4 4 3 3 3 2 2
3779 029/300 Write block 4033 len 24576 data: 3 3 3 3 2 2
3780 030/300 Read block 3029 len 24576
3781 031/300 Read block 4770 len 28672
3782 032/300 Flush
3783 033/300 Read block 2114 len 12288
3784 034/300 Read block 480 len 40960
3785 035/300 Read block 2918 len 4096
3786 036/300 Read block 3516 len 24576
3787 037/300 Write block 3046 len 16384 data: 3 2 2 2
3788 038/300 Read block 950 len 24576
3789 039/300 Write block 1523 len 20480 data: 2 2 2 3 4
3790 040/300 Read block 4393 len 28672
3791 041/300 Read block 2190 len 20480
3792 042/300 Read block 1327 len 8192
3793 043/300 Write block 2866 len 4096 data: 2
3794 044/300 Write block 1462 len 8192 data: 2 2
3795 045/300 Write block 1903 len 40960 data: 2 2 2 2 2 2 3 3 3 3
3796 046/300 Read block 3900 len 32768
3797 047/300 Write block 863 len 36864 data: 3 3 3 3 2 2 2 2 2
3798 048/300 Write block 794 len 24576 data: 4 4 5 5 4 3
3799 049/300 Read block 4672 len 8192
3800 050/300 Read block 2191 len 40960
3801 051/300 Write block 4569 len 4096 data: 3
3802 052/300 Read block 4472 len 40960
3803 053/300 Flush
3804 054/300 Read block 441 len 36864
3805 055/300 Write block 3521 len 40960 data: 3 3 3 2 2 2 2 2 2 2
3806 056/300 Write block 4477 len 40960 data: 2 3 3 2 2 2 2 2 2 2
3807 057/300 Read block 795 len 36864
3808 058/300 Write block 2962 len 4096 data: 2
3809 059/300 Read block 1504 len 8192
3810 060/300 Write block 1587 len 16384 data: 3 3 3 4
3811 061/300 Read block 2340 len 36864
3812 062/300 Write block 901 len 12288 data: 2 2 2
3813 063/300 Write block 1995 len 32768 data: 2 2 2 3 3 3 5 5
3814 064/300 Write block 3540 len 32768 data: 3 2 2 2 2 2 2 2
3815 065/300 Write block 780 len 24576 data: 2 2 2 3 2 2
3816 066/300 Read block 699 len 4096
3817 067/300 Write block 3880 len 36864 data: 2 2 2 2 2 2 2 2 2
3818 068/300 Flush
3819 069/300 Read block 3057 len 28672
3820 070/300 Read block 1788 len 40960
3821 071/300 Read block 4782 len 20480
3822 072/300 Write block 4248 len 4096 data: 4
3823 073/300 Read block 1716 len 4096
3824 074/300 Write block 4280 len 28672 data: 2 2 2 3 3 3 2
3825 075/300 Write block 3908 len 36864 data: 2 2 2 2 2 2 2 2 2
3826 076/300 Read block 4416 len 12288
3827 077/300 Read block 2450 len 40960
3828 078/300 Write block 226 len 32768 data: 4 3 3 3 3 3 2 2
3829 079/300 Write block 2959 len 16384 data: 3 3 2 3
3830 080/300 Read block 2038 len 24576
3831 081/300 Flush
3832 082/300 Read block 4356 len 36864
3833 083/300 Read block 2059 len 20480
3834 084/300 Read block 485 len 28672
3835 085/300 Write block 952 len 40960 data: 2 2 2 2 3 4 4 4 4 4
3836 086/300 Read block 3228 len 8192
3837 087/300 Read block 2170 len 16384
3838 088/300 Flush
3839 089/300 Read block 4264 len 40960
3840 090/300 Write block 1099 len 20480 data: 2 2 2 2 2
3841 091/300 Write block 4200 len 32768 data: 2 2 2 3 3 3 3 2
3842 092/300 Write block 4601 len 12288 data: 2 2 3
3843 093/300 Flush
3844 094/300 Read block 129 len 36864
3845 095/300 Write block 3704 len 4096 data: 2
3846 096/300 Flush
3847 097/300 Read block 1943 len 12288
3848 098/300 Read block 421 len 20480
3849 099/300 Write block 512 len 16384 data: 3 4 4 4
3850 100/300 Read block 1968 len 4096
3851 101/300 Read block 4113 len 36864
3852 102/300 Read block 2813 len 28672
3853 103/300 Write block 3407 len 36864 data: 2 2 2 4 4 3 3 3 3
3854 104/300 Read block 868 len 16384
3855 105/300 Write block 1642 len 32768 data: 3 3 3 3 3 2 2 3
3856 106/300 Flush
3857 107/300 Read block 773 len 16384
3858 108/300 Write block 865 len 16384 data: 4 4 3 3
3859 109/300 Read block 4681 len 28672
3860 110/300 Write block 2908 len 20480 data: 2 2 2 2 2
3861 111/300 Flush
3862 112/300 Write block 4414 len 20480 data: 2 2 2 3 3
3863 113/300 Write block 1779 len 36864 data: 3 3 3 3 4 3 4 4 4
3864 114/300 Write block 3150 len 36864 data: 4 3 2 2 3 3 3 3 5
3865 115/300 Write block 4702 len 4096 data: 2
3866 116/300 Write block 362 len 8192 data: 2 2
3867 117/300 Write block 1418 len 24576 data: 3 3 3 3 3 2
3868 118/300 Write block 909 len 40960 data: 2 2 2 2 2 2 2 2 2 2
3869 119/300 Write block 1951 len 4096 data: 3
3870 120/300 Flush
3871 121/300 Write block 1834 len 20480 data: 2 2 2 2 2
3872 122/300 Flush
3873 123/300 Read block 32 len 12288
3874 124/300 Read block 619 len 36864
3875 125/300 Read block 2543 len 12288
3876 126/300 Read block 4966 len 16384
3877 127/300 Read block 1246 len 28672
3878 128/300 Write block 1856 len 8192 data: 2 2
3879 129/300 Write block 4042 len 28672 data: 3 3 5 4 3 3 3
3880 130/300 Write block 3728 len 28672 data: 3 3 4 3 3 3 3
3881 131/300 Read block 3166 len 40960
3882 132/300 Read block 1979 len 8192
3883 133/300 Write block 617 len 16384 data: 6 6 5 4
3884 134/300 Write block 2632 len 36864 data: 2 2 2 2 2 2 2 3 3
3885 135/300 Read block 2732 len 28672
3886 136/300 Read block 4266 len 8192
3887 137/300 Read block 3688 len 36864
3888 138/300 Read block 2869 len 40960
3889 139/300 Flush
3890 140/300 Write block 2856 len 24576 data: 3 3 3 3 3 3
3891 141/300 Read block 3456 len 36864
3892 142/300 Read block 3155 len 16384
3893 143/300 Flush
3894 144/300 Read block 3759 len 28672
3895 145/300 Write block 4731 len 36864 data: 2 2 2 2 2 2 3 3 5
3896 146/300 Write block 3880 len 20480 data: 3 3 3 3 3
3897 147/300 Flush
3898 148/300 Write block 157 len 32768 data: 2 2 2 2 2 3 3 3
3899 149/300 Write block 2663 len 32768 data: 2 3 3 3 3 3 3 3
3900 150/300 Read block 3426 len 12288
3901 151/300 Read block 1368 len 4096
3902 152/300 Read block 2916 len 28672
3903 153/300 Read block 3413 len 28672
3904 154/300 Read block 1144 len 40960
3905 155/300 Flush
3906 156/300 Read block 4269 len 28672
3907 157/300 Flush
3908 158/300 Read block 2180 len 28672
3909 159/300 Write block 3347 len 20480 data: 2 2 2 2 2
3910 160/300 Write block 3655 len 36864 data: 2 2 2 2 2 2 2 2 2
3911 161/300 Flush
3912 162/300 Write block 4713 len 20480 data: 3 3 3 2 2
3913 163/300 Read block 3146 len 24576
3914 164/300 Read block 2780 len 28672
3915 165/300 Read block 1162 len 4096
3916 166/300 Read block 3740 len 16384
3917 167/300 Write block 4221 len 4096 data: 2
3918 168/300 Flush
3919 169/300 Read block 515 len 4096
3920 170/300 Read block 2927 len 40960
3921 171/300 Read block 4629 len 12288
3922 172/300 Read block 4040 len 4096
3923 173/300 Read block 3568 len 32768
3924 174/300 Write block 667 len 28672 data: 2 2 2 2 2 2 2
3925 175/300 Read block 1695 len 4096
3926 176/300 Flush
3927 177/300 Write block 4675 len 32768 data: 5 4 3 3 3 3 3 2
3928 178/300 Read block 2434 len 16384
3929 179/300 Read block 791 len 28672
3930 180/300 Flush
3931 181/300 Read block 4260 len 4096
3932 182/300 Read block 658 len 12288
3933 183/300 Write block 2287 len 36864 data: 2 2 2 2 3 3 3 3 3
3934 184/300 Read block 2939 len 16384
3935 185/300 Read block 2687 len 36864
3936 186/300 Read block 1763 len 4096
3937 187/300 Read block 3811 len 8192
3938 188/300 Write block 4834 len 20480 data: 3 3 3 3 4
3939 189/300 Write block 3185 len 16384 data: 2 2 2 3
3940 190/300 Write block 2294 len 4096 data: 4
3941 191/300 Read block 735 len 36864
3942 192/300 Read block 3917 len 8192
3943 193/300 Read block 1889 len 20480
3944 194/300 Read block 819 len 20480
3945 195/300 Read block 3114 len 8192
3946 196/300 Read block 3830 len 36864
3947 197/300 Read block 3472 len 28672
3948 198/300 Read block 2931 len 24576
3949 199/300 Read block 822 len 8192
3950 200/300 Write block 2376 len 16384 data: 2 2 2 2
3951 201/300 Read block 1429 len 36864
3952 202/300 Read block 575 len 36864
3953 203/300 Write block 913 len 12288 data: 3 3 3
3954 204/300 Flush
3955 205/300 Read block 198 len 40960
3956 206/300 Read block 654 len 24576
3957 207/300 Read block 4839 len 32768
3958 208/300 Read block 1333 len 4096
3959 209/300 Read block 715 len 32768
3960 210/300 Write block 1404 len 32768 data: 2 2 2 2 2 2 2 3
3961 211/300 Write block 2314 len 4096 data: 2
3962 212/300 Write block 2218 len 8192 data: 4 4
3963 213/300 Write block 741 len 4096 data: 3
3964 214/300 Write block 2478 len 36864 data: 2 2 2 2 2 2 2 2 2
3965 215/300 Read block 548 len 32768
3966 216/300 Read block 4356 len 40960
3967 217/300 Read block 3102 len 36864
3968 218/300 Write block 4163 len 36864 data: 2 2 2 2 3 3 3 3 2
3969 219/300 Read block 1229 len 8192
3970 220/300 Write block 3635 len 4096 data: 2
3971 221/300 Read block 610 len 8192
3972 222/300 Write block 721 len 24576 data: 4 3 3 3 3 3
3973 223/300 Write block 514 len 32768 data: 5 5 4 3 2 3 3 3
3974 224/300 Write block 1458 len 28672 data: 2 2 2 2 3 3 2
3975 225/300 Read block 49 len 32768
3976 226/300 Read block 3627 len 20480
3977 227/300 Read block 2120 len 32768
3978 228/300 Read block 2967 len 4096
3979 229/300 Write block 3617 len 40960 data: 2 2 2 2 2 2 3 3 3 3
3980 230/300 Read block 1452 len 12288
3981 231/300 Flush
3982 232/300 Write block 3789 len 4096 data: 2
3983 233/300 Read block 3219 len 32768
3984 234/300 Read block 2811 len 12288
3985 235/300 Read block 741 len 12288
3986 236/300 Write block 814 len 32768 data: 3 3 3 3 3 3 3 3
3987 237/300 Flush
3988 238/300 Read block 539 len 36864
3989 239/300 Write block 4527 len 40960 data: 2 2 2 2 2 2 2 2 2 2
3990 240/300 Write block 1478 len 36864 data: 2 2 3 3 3 3 3 3 3
3991 241/300 Read block 351 len 20480
3992 242/300 Write block 507 len 8192 data: 2 2
3993 243/300 Write block 2291 len 12288 data: 4 4 4
3994 244/300 Read block 4475 len 4096
3995 245/300 Flush
3996 246/300 Flush
3997 247/300 Write block 478 len 4096 data: 2
3998 248/300 Read block 3615 len 16384
3999 249/300 Read block 3497 len 32768
4000 250/300 Read block 2404 len 36864
4001 251/300 Flush
4002 252/300 Read block 3379 len 20480
4003 253/300 Read block 2401 len 32768
4004 254/300 Write block 1977 len 12288 data: 3 3 3
4005 255/300 Write block 3109 len 24576 data: 3 3 4 4 4 4
4006 256/300 Write block 4480 len 12288 data: 3 3 3
4007 257/300 Read block 2200 len 8192
4008 258/300 Write block 3134 len 36864 data: 2 2 2 2 2 2 2 2 2
4009 259/300 Read block 420 len 8192
4010 260/300 Write block 3868 len 4096 data: 4
4011 261/300 Read block 2679 len 12288
4012 262/300 Read block 3646 len 4096
4013 263/300 Write block 234 len 28672 data: 2 2 3 3 3 3 3
4014 264/300 Write block 135 len 8192 data: 2 2
4015 265/300 Read block 1105 len 4096
4016 266/300 Write block 3240 len 36864 data: 3 3 2 2 2 2 2 2 2
4017 267/300 Flush
4018 268/300 Flush
4019 269/300 Read block 3052 len 28672
4020 270/300 Read block 4785 len 24576
4021 271/300 Read block 2691 len 32768
4022 272/300 Read block 391 len 20480
4023 273/300 Flush
4024 274/300 Read block 2081 len 8192
4025 275/300 Write block 2820 len 36864 data: 2 2 2 3 3 3 3 2 2
4026 276/300 Write block 2276 len 16384 data: 2 2 2 2
4027 277/300 Read block 1217 len 8192
4028 278/300 Write block 3104 len 32768 data: 2 2 2 2 3 4 4 5
4029 279/300 Read block 3057 len 36864
4030 280/300 Write block 3666 len 24576 data: 3 4 4 4 4 4
4031 281/300 Flush
4032 282/300 Read block 4810 len 28672
4033 283/300 Read block 2151 len 20480
4034 284/300 Write block 2353 len 28672 data: 2 2 2 2 2 2 2
4035 285/300 Read block 3398 len 16384
4036 286/300 Write block 1641 len 36864 data: 3 4 4 4 4 4 3 3 4
4037 287/300 Read block 1396 len 36864
4038 288/300 Read block 3018 len 40960
4039 289/300 Read block 3748 len 12288
4040 290/300 Flush
4041 291/300 Read block 2539 len 32768
4042 292/300 Read block 3575 len 24576
4043 293/300 Read block 4791 len 16384
4044 294/300 Write block 2257 len 36864 data: 3 2 2 2 2 2 2 2 2
4045 295/300 Write block 598 len 36864 data: 2 2 2 3 3 3 3 3 3
4046 296/300 Read block 2598 len 4096
4047 297/300 Write block 2047 len 12288 data: 2 2 2
4048 298/300 Read block 127 len 20480
4049 299/300 Write block 2374 len 36864 data: 2 2 3 3 3 3 2 2 2
4050 300/300 Flush
4051 Main task has work to do, go find it
4052 got dsc Start(1)
4053 start 1
4054 Feb 15 02:58:20.299 INFO request completed, latency_us: 227, response_code: 204, uri: /start/cid/1, method: POST, req_id: 99a17189-6251-44ee-9a30-1fcc9e932658, remote_addr: 127.0.0.1:54744, local_addr: 127.0.0.1:9998
4055 Replay: started 1, returned:Ok(())
4056 ----------------------------------------------------------------
4057 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
4058 States: Active Offline Active
4059 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
4060 1206 Acked 2205 Read 2 Done New Done true
4061 1207 Acked 2206 Read 7 Done New Done true
4062 1208 Acked 2207 Flush 0 Done New Done false
4063 1209 Acked 2208 Read 3 Done New Done false
4064 1210 Acked 2209 Read 6 Done New Done false
4065 1211 Acked 2210 Write 9 Done New Done false
4066 1212 Acked 2211 Read 5 Done New Done false
4067 1213 Acked 2212 Read 3 Done New Done false
4068 1214 Acked 2213 Read 9 Done New Done false
4069 1215 Acked 2214 Read 9 Done New Done false
4070 1216 Acked 2215 Flush 0 Done New Done false
4071 1217 Acked 2216 Read 7 Done New Done false
4072 1218 Acked 2217 Flush 0 Done New Done false
4073 1219 Acked 2218 Read 8 Done New Done false
4074 1220 Acked 2219 Read 9 Done New Done false
4075 1221 Acked 2220 Write 7 Done New Done false
4076 1222 Acked 2221 Read 4 Done New Done false
4077 1223 Acked 2222 Flush 0 Done New Done false
4078 1224 Acked 2223 Flush 0 Done New Done false
4079 1225 Acked 2224 Write 4 Done New Done false
4080 1226 Acked 2225 Read 10 Done New Done false
4081 1227 Acked 2226 Read 8 Done New Done false
4082 1228 Acked 2227 Write 6 Done New Done false
4083 1229 Acked 2228 Read 6 Done New Done false
4084 1230 Acked 2229 Read 5 Done New Done false
4085 1231 Acked 2230 Read 10 Done New Done false
4086 1232 Acked 2231 Read 8 Done New Done false
4087 1233 Acked 2232 Write 10 Done New Done false
4088 1234 Acked 2233 Write 6 Done New Done false
4089 1235 Acked 2234 Read 6 Done New Done false
4090 1236 Acked 2235 Read 7 Done New Done false
4091 1237 Acked 2236 Flush 0 Done New Done false
4092 1238 Acked 2237 Read 3 Done New Done false
4093 1239 Acked 2238 Read 10 Done New Done false
4094 1240 Acked 2239 Read 1 Done New Done false
4095 1241 Acked 2240 Read 6 Done New Done false
4096 1242 Acked 2241 Write 4 Done New Done false
4097 1243 Acked 2242 Read 6 Done New Done false
4098 1244 Acked 2243 Write 5 Done New Done false
4099 1245 Acked 2244 Read 7 Done New Done false
4100 1246 Acked 2245 Read 5 Done New Done false
4101 1247 Acked 2246 Read 2 Done New Done false
4102 1248 Acked 2247 Write 1 Done New Done false
4103 1249 Acked 2248 Write 2 Done New Done false
4104 1250 Acked 2249 Write 10 Done New Done false
4105 1251 Acked 2250 Read 8 Done New Done false
4106 1252 Acked 2251 Write 9 Done New Done false
4107 1253 Acked 2252 Write 6 Done New Done false
4108 1254 Acked 2253 Read 2 Done New Done false
4109 1255 Acked 2254 Read 10 Done New Done false
4110 1256 Acked 2255 Write 1 Done New Done false
4111 1257 Acked 2256 Read 10 Done New Done false
4112 1258 Acked 2257 Flush 0 Done New Done false
4113 1259 Acked 2258 Read 9 Done New Done false
4114 1260 Acked 2259 Write 10 Done New Done false
4115 1261 Acked 2260 Write 10 Done New Done false
4116 1262 Acked 2261 Read 9 Done New Done false
4117 1263 Acked 2262 Write 1 Done New Done false
4118 1264 Acked 2263 Read 2 Done New Done false
4119 1265 Acked 2264 Write 4 Done New Done false
4120 1266 Acked 2265 Read 9 Done New Done false
4121 1267 Acked 2266 Write 3 Done New Done false
4122 1268 Acked 2267 Write 8 Done New Done false
4123 1269 Acked 2268 Write 8 Done New Done false
4124 1270 Acked 2269 Write 6 Done New Done false
4125 1271 Acked 2270 Read 1 Done New Done false
4126 1272 Acked 2271 Write 9 Done New Done false
4127 1273 Acked 2272 Flush 0 Done New Done false
4128 1274 Acked 2273 Read 7 Done New Done false
4129 1275 Acked 2274 Read 10 Done New Done false
4130 1276 Acked 2275 Read 5 Done New Done false
4131 1277 Acked 2276 Write 1 Done New Done false
4132 1278 Acked 2277 Read 1 Done New Done false
4133 1279 Acked 2278 Write 7 Done New Done false
4134 1280 Acked 2279 Write 9 Done New Done false
4135 1281 Acked 2280 Read 3 Done New Done false
4136 1282 Acked 2281 Read 10 Done New Done false
4137 1283 Acked 2282 Write 8 Done New Done false
4138 1284 Acked 2283 Write 4 Done New Done false
4139 1285 Acked 2284 Read 6 Done New Done false
4140 1286 Acked 2285 Flush 0 Done New Done false
4141 1287 Acked 2286 Read 9 Done New Done false
4142 1288 Acked 2287 Read 5 Done New Done false
4143 1289 Acked 2288 Read 7 Done New Done false
4144 1290 Acked 2289 Write 10 Done New Done false
4145 1291 Acked 2290 Read 2 Done New Done false
4146 1292 Acked 2291 Read 4 Done New Done false
4147 1293 Acked 2292 Flush 0 Done New Done false
4148 1294 Acked 2293 Read 10 Done New Done false
4149 1295 Acked 2294 Write 5 Done New Done false
4150 1296 Acked 2295 Write 8 Done New Done false
4151 1297 Acked 2296 Write 3 Done New Done false
4152 1298 Acked 2297 Flush 0 Done New Done false
4153 1299 Acked 2298 Read 9 Done New Done false
4154 1300 Acked 2299 Write 1 Done New Done false
4155 1301 Acked 2300 Flush 0 Done New Done false
4156 1302 Acked 2301 Read 3 Done New Done false
4157 1303 Acked 2302 Read 5 Done New Done false
4158 1304 Acked 2303 Write 4 Done New Done false
4159 1305 Acked 2304 Read 1 Done New Done false
4160 1306 Acked 2305 Read 9 Done New Done false
4161 1307 Acked 2306 Read 7 Done New Done false
4162 1308 Acked 2307 Write 9 Done New Done false
4163 1309 Acked 2308 Read 4 Done New Done false
4164 1310 Acked 2309 Write 8 Done New Done false
4165 1311 Acked 2310 Flush 0 Done New Done false
4166 1312 Acked 2311 Read 4 Done New Done false
4167 1313 Acked 2312 Write 4 Done New Done false
4168 1314 Acked 2313 Read 7 Done New Done false
4169 1315 Acked 2314 Write 5 Done New Done false
4170 1316 Acked 2315 Flush 0 Done New Done false
4171 1317 Acked 2316 Write 5 Done New Done false
4172 1318 Acked 2317 Write 9 Done New Done false
4173 1319 Acked 2318 Write 9 Done New Done false
4174 1320 Acked 2319 Write 1 Done New Done false
4175 1321 Acked 2320 Write 2 Done New Done false
4176 1322 Acked 2321 Write 6 Done New Done false
4177 1323 Acked 2322 Write 10 Done New Done false
4178 1324 Acked 2323 Write 1 Done New Done false
4179 1325 Acked 2324 Flush 0 Done New Done false
4180 1326 Acked 2325 Write 5 Done New Done false
4181 1327 Acked 2326 Flush 0 Done New Done false
4182 1328 Acked 2327 Read 3 Done New Done false
4183 1329 Acked 2328 Read 9 Done New Done false
4184 1330 Acked 2329 Read 3 Done New Done false
4185 1331 Acked 2330 Read 4 Done New Done false
4186 1332 Acked 2331 Read 7 Done New Done false
4187 1333 Acked 2332 Write 2 Done New Done false
4188 1334 Acked 2333 Write 7 Done New Done false
4189 1335 Acked 2334 Write 7 Done New Done false
4190 1336 Acked 2335 Read 10 Done New Done false
4191 1337 Acked 2336 Read 2 Done New Done false
4192 1338 Acked 2337 Write 4 Done New Done false
4193 1339 Acked 2338 Flush 0 Done New Done false
4194 1340 Acked 2339 Write 9 Done New Done false
4195 1341 Acked 2340 Read 7 Done New Done false
4196 1342 Acked 2341 Read 2 Done New Done false
4197 1343 Acked 2342 Read 9 Done New Done false
4198 1344 Acked 2343 Read 10 Done New Done false
4199 1345 Acked 2344 Flush 0 Done New Done false
4200 1346 Acked 2345 Write 6 Done New Done false
4201 1347 Acked 2346 Read 9 Done New Done false
4202 1348 Acked 2347 Read 4 Done New Done false
4203 1349 Acked 2348 Flush 0 Done New Done false
4204 1350 Acked 2349 Read 7 Done New Done false
4205 1351 Acked 2350 Write 9 Done New Done false
4206 1352 Acked 2351 Write 5 Done New Done false
4207 1353 Acked 2352 Flush 0 Done New Done false
4208 1354 Acked 2353 Write 8 Done New Done false
4209 1355 Acked 2354 Write 8 Done New Done false
4210 1356 Acked 2355 Read 3 Done New Done false
4211 1357 Acked 2356 Read 1 Done New Done false
4212 1358 Acked 2357 Read 7 Done New Done false
4213 1359 Acked 2358 Read 7 Done New Done false
4214 1360 Acked 2359 Read 10 Done New Done false
4215 1361 Acked 2360 Flush 0 Done New Done false
4216 1362 Acked 2361 Read 7 Done New Done false
4217 1363 Acked 2362 Flush 0 Done New Done false
4218 1364 Acked 2363 Read 7 Done New Done false
4219 1365 Acked 2364 Write 5 Done New Done false
4220 1366 Acked 2365 Write 9 Done New Done false
4221 1367 Acked 2366 Flush 0 Done New Done false
4222 1368 Acked 2367 Write 5 Done New Done false
4223 1369 Acked 2368 Read 6 Done New Done false
4224 1370 Acked 2369 Read 7 Done New Done false
4225 1371 Acked 2370 Read 1 Done New Done false
4226 1372 Acked 2371 Read 4 Done New Done false
4227 1373 Acked 2372 Write 1 Done New Done false
4228 1374 Acked 2373 Flush 0 Done New Done false
4229 1375 Acked 2374 Read 1 Done New Done false
4230 1376 Acked 2375 Read 10 Done New Done false
4231 1377 Acked 2376 Read 3 Done New Done false
4232 1378 Acked 2377 Read 1 Done New Done false
4233 1379 Acked 2378 Read 8 Done New Done false
4234 1380 Acked 2379 Write 7 Done New Done false
4235 1381 Acked 2380 Read 1 Done New Done false
4236 1382 Acked 2381 Flush 0 Done New Done false
4237 1383 Acked 2382 Write 8 Done New Done false
4238 1384 Acked 2383 Read 4 Done New Done false
4239 1385 Acked 2384 Read 7 Done New Done false
4240 1386 Acked 2385 Flush 0 Done New Done false
4241 1387 Acked 2386 Read 1 Done New Done false
4242 1388 Acked 2387 Read 3 Done New Done false
4243 1389 Acked 2388 Write 9 Done New Done false
4244 1390 Acked 2389 Read 4 Done New Done false
4245 1391 Acked 2390 Read 9 Done New Done false
4246 1392 Acked 2391 Read 1 Done New Done false
4247 1393 Acked 2392 Read 2 Done New Done false
4248 1394 Acked 2393 Write 5 Done New Done false
4249 1395 Acked 2394 Write 4 Done New Done false
4250 1396 Acked 2395 Write 1 Done New Done false
4251 1397 Acked 2396 Read 9 Done New Done false
4252 1398 Acked 2397 Read 2 Done New Done false
4253 1399 Acked 2398 Read 5 Done New Done false
4254 1400 Acked 2399 Read 5 Done New Done false
4255 1401 Acked 2400 Read 2 Done New Done false
4256 1402 Acked 2401 Read 9 Done New Done false
4257 1403 Acked 2402 Read 7 Done New Done false
4258 1404 Acked 2403 Read 6 Done New Done false
4259 1405 Acked 2404 Read 2 Done New Done false
4260 1406 Acked 2405 Write 4 Done New Done false
4261 1407 Acked 2406 Read 9 Done New Done false
4262 1408 Acked 2407 Read 9 Done New Done false
4263 1409 Acked 2408 Write 3 Done New Done false
4264 1410 Acked 2409 Flush 0 Done New Done false
4265 1411 Acked 2410 Read 10 Done New Done false
4266 1412 Acked 2411 Read 6 Done New Done false
4267 1413 Acked 2412 Read 8 Done New Done false
4268 1414 Acked 2413 Read 1 Done New Done false
4269 1415 Acked 2414 Read 8 Done New Done false
4270 1416 Acked 2415 Write 8 Done New Done false
4271 1417 Acked 2416 Write 1 Done New Done false
4272 1418 Acked 2417 Write 2 Done New Done false
4273 1419 Acked 2418 Write 1 Done New Done false
4274 1420 Acked 2419 Write 9 Done New Done false
4275 1421 Acked 2420 Read 8 Done New Done false
4276 1422 Acked 2421 Read 10 Done New Done false
4277 1423 Acked 2422 Read 9 Done New Done false
4278 1424 Acked 2423 Write 9 Done New Done false
4279 1425 Acked 2424 Read 2 Done New Done false
4280 1426 Acked 2425 Write 1 Done New Done false
4281 1427 Acked 2426 Read 2 Done New Done false
4282 1428 Acked 2427 Write 6 Done New Done false
4283 1429 Acked 2428 Write 8 Done New Done false
4284 1430 Acked 2429 Write 7 Done New Done false
4285 1431 Acked 2430 Read 8 Done New Done false
4286 1432 Acked 2431 Read 5 Done New Done false
4287 1433 Acked 2432 Read 8 Done New Done false
4288 1434 Acked 2433 Read 1 Done New Done false
4289 1435 Acked 2434 Write 10 Done New Done false
4290 1436 Acked 2435 Read 3 Done New Done false
4291 1437 Acked 2436 Flush 0 Done New Done false
4292 1438 Acked 2437 Write 1 Done New Done false
4293 1439 Acked 2438 Read 8 Done New Done false
4294 1440 Acked 2439 Read 3 Done New Done false
4295 1441 Acked 2440 Read 3 Done New Done false
4296 1442 Acked 2441 Write 8 Done New Done false
4297 1443 Acked 2442 Flush 0 Done New Done false
4298 1444 Acked 2443 Read 9 Done New Done false
4299 1445 Acked 2444 Write 10 Done New Done false
4300 1446 Acked 2445 Write 9 Done New Done false
4301 1447 Acked 2446 Read 5 Done New Done false
4302 1448 Acked 2447 Write 2 Done New Done false
4303 1449 Acked 2448 Write 3 Done New Done false
4304 1450 Acked 2449 Read 1 Done New Done false
4305 1451 Acked 2450 Flush 0 Done New Done false
4306 1452 Acked 2451 Flush 0 Done New Done false
4307 1453 Acked 2452 Write 1 Done New Done false
4308 1454 Acked 2453 Read 4 Done New Done false
4309 1455 Acked 2454 Read 8 Done New Done false
4310 1456 Acked 2455 Read 9 Done New Done false
4311 1457 Acked 2456 Flush 0 Done New Done false
4312 1458 Acked 2457 Read 5 Done New Done false
4313 1459 Acked 2458 Read 8 Done New Done false
4314 1460 Acked 2459 Write 3 Done New Done false
4315 1461 Acked 2460 Write 6 Done New Done false
4316 1462 Acked 2461 Write 3 Done New Done false
4317 1463 Acked 2462 Read 2 Done New Done false
4318 1464 Acked 2463 Write 9 Done New Done false
4319 1465 Acked 2464 Read 2 Done New Done false
4320 1466 Acked 2465 Write 1 Done New Done false
4321 1467 Acked 2466 Read 3 Done New Done false
4322 1468 Acked 2467 Read 1 Done New Done false
4323 1469 Acked 2468 Write 7 Done New Done false
4324 1470 Acked 2469 Write 2 Done New Done false
4325 1471 Acked 2470 Read 1 Done New Done false
4326 1472 Acked 2471 Write 9 Done New Done false
4327 1473 Acked 2472 Flush 0 Done New Done false
4328 1474 Acked 2473 Flush 0 Done New Done false
4329 1475 Acked 2474 Read 7 Done New Done false
4330 1476 Acked 2475 Read 6 Done New Done false
4331 1477 Acked 2476 Read 8 Done New Done false
4332 1478 Acked 2477 Read 5 Done New Done false
4333 1479 Acked 2478 Flush 0 Done New Done false
4334 1480 Acked 2479 Read 2 Done New Done false
4335 1481 Acked 2480 Write 9 Done New Done false
4336 1482 Acked 2481 Write 4 Done New Done false
4337 1483 Acked 2482 Read 2 Done New Done false
4338 1484 Acked 2483 Write 8 Done New Done false
4339 1485 Acked 2484 Read 9 Done New Done false
4340 1486 Acked 2485 Write 6 Done New Done false
4341 1487 Acked 2486 Flush 0 Done New Done false
4342 1488 Acked 2487 Read 7 Done New Done false
4343 1489 Acked 2488 Read 5 Done New Done false
4344 1490 Acked 2489 Write 7 Done New Done false
4345 1491 Acked 2490 Read 4 Done New Done false
4346 1492 Acked 2491 Write 9 Done New Done false
4347 1493 Acked 2492 Read 9 Done New Done false
4348 1494 Acked 2493 Read 10 Done New Done false
4349 1495 Acked 2494 Read 3 Done New Done false
4350 1496 Acked 2495 Flush 0 Done New Done false
4351 1497 Acked 2496 Read 8 Done New Done false
4352 1498 Acked 2497 Read 6 Done New Done false
4353 1499 Acked 2498 Read 4 Done New Done false
4354 1500 Acked 2499 Write 9 Done New Done false
4355 1501 Acked 2500 Write 9 Done New Done false
4356 1502 Acked 2501 Read 1 Done New Done false
4357 1503 Acked 2502 Write 3 Done New Done false
4358 1504 Acked 2503 Read 5 Done New Done false
4359 1505 Acked 2504 Write 9 Done New Done false
4360 1506 Acked 2505 Flush 0 Done New Done false
4361 STATES DS:0 DS:1 DS:2 TOTAL
4362 New 0 301 0 301
4363 Sent 0 0 0 0
4364 Done 301 0 301 602
4365 Skipped 0 0 0 0
4366 Error 0 0 0 0
4367 Last Flush: 2505 2204 2505
4368 Downstairs last five completed: 2204 2203 2202 2201 2200
4369 Upstairs last five completed: 1506 1505 1504 1503 1502
4370 CLIENT: Up:0 ds:301 act:2
43712024-02-15T02:58:20.959ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
43722024-02-15T02:58:20.959ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
43732024-02-15T02:58:20.959ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
43742024-02-15T02:58:20.959ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
43752024-02-15T02:58:20.959ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
43762024-02-15T02:58:20.960ZINFOcrucible: [1] client re-new 301 jobs since flush 2204 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
43772024-02-15T02:58:20.960ZINFOcrucible: client task is exiting = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
43782024-02-15T02:58:21.961ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
43792024-02-15T02:58:21.961ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
43802024-02-15T02:58:21.961ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
43812024-02-15T02:58:21.961ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
43822024-02-15T02:58:21.961ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
43832024-02-15T02:58:21.961ZINFOcrucible: [1] client re-new 301 jobs since flush 2204 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
43842024-02-15T02:58:21.961ZINFOcrucible: client task is exiting = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
43852024-02-15T02:58:22.962ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
43862024-02-15T02:58:22.962ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
43872024-02-15T02:58:22.962ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
43882024-02-15T02:58:22.962ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
43892024-02-15T02:58:22.962ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
43902024-02-15T02:58:22.962ZINFOcrucible: [1] client re-new 301 jobs since flush 2204 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
43912024-02-15T02:58:22.962ZINFOcrucible: client task is exiting = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
43922024-02-15T02:58:23.963ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
43932024-02-15T02:58:23.963ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
43942024-02-15T02:58:23.964ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
43952024-02-15T02:58:23.964ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
43962024-02-15T02:58:23.964ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
43972024-02-15T02:58:23.964ZINFOcrucible: [1] client re-new 301 jobs since flush 2204 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
43982024-02-15T02:58:23.964ZINFOcrucible: client task is exiting = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
4399 ----------------------------------------------------------------
4400 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
4401 States: Active Offline Active
4402 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
4403 1206 Acked 2205 Read 2 Done New Done true
4404 1207 Acked 2206 Read 7 Done New Done true
4405 1208 Acked 2207 Flush 0 Done New Done true
4406 1209 Acked 2208 Read 3 Done New Done true
4407 1210 Acked 2209 Read 6 Done New Done true
4408 1211 Acked 2210 Write 9 Done New Done true
4409 1212 Acked 2211 Read 5 Done New Done true
4410 1213 Acked 2212 Read 3 Done New Done true
4411 1214 Acked 2213 Read 9 Done New Done true
4412 1215 Acked 2214 Read 9 Done New Done true
4413 1216 Acked 2215 Flush 0 Done New Done true
4414 1217 Acked 2216 Read 7 Done New Done true
4415 1218 Acked 2217 Flush 0 Done New Done true
4416 1219 Acked 2218 Read 8 Done New Done true
4417 1220 Acked 2219 Read 9 Done New Done true
4418 1221 Acked 2220 Write 7 Done New Done true
4419 1222 Acked 2221 Read 4 Done New Done true
4420 1223 Acked 2222 Flush 0 Done New Done true
4421 1224 Acked 2223 Flush 0 Done New Done true
4422 1225 Acked 2224 Write 4 Done New Done true
4423 1226 Acked 2225 Read 10 Done New Done true
4424 1227 Acked 2226 Read 8 Done New Done true
4425 1228 Acked 2227 Write 6 Done New Done true
4426 1229 Acked 2228 Read 6 Done New Done true
4427 1230 Acked 2229 Read 5 Done New Done true
4428 1231 Acked 2230 Read 10 Done New Done true
4429 1232 Acked 2231 Read 8 Done New Done true
4430 1233 Acked 2232 Write 10 Done New Done true
4431 1234 Acked 2233 Write 6 Done New Done true
4432 1235 Acked 2234 Read 6 Done New Done true
4433 1236 Acked 2235 Read 7 Done New Done true
4434 1237 Acked 2236 Flush 0 Done New Done true
4435 1238 Acked 2237 Read 3 Done New Done true
4436 1239 Acked 2238 Read 10 Done New Done true
4437 1240 Acked 2239 Read 1 Done New Done true
4438 1241 Acked 2240 Read 6 Done New Done true
4439 1242 Acked 2241 Write 4 Done New Done true
4440 1243 Acked 2242 Read 6 Done New Done true
4441 1244 Acked 2243 Write 5 Done New Done true
4442 1245 Acked 2244 Read 7 Done New Done true
4443 1246 Acked 2245 Read 5 Done New Done true
4444 1247 Acked 2246 Read 2 Done New Done true
4445 1248 Acked 2247 Write 1 Done New Done true
4446 1249 Acked 2248 Write 2 Done New Done true
4447 1250 Acked 2249 Write 10 Done New Done true
4448 1251 Acked 2250 Read 8 Done New Done true
4449 1252 Acked 2251 Write 9 Done New Done true
4450 1253 Acked 2252 Write 6 Done New Done true
4451 1254 Acked 2253 Read 2 Done New Done true
4452 1255 Acked 2254 Read 10 Done New Done true
4453 1256 Acked 2255 Write 1 Done New Done true
4454 1257 Acked 2256 Read 10 Done New Done true
4455 1258 Acked 2257 Flush 0 Done New Done true
4456 1259 Acked 2258 Read 9 Done New Done true
4457 1260 Acked 2259 Write 10 Done New Done true
4458 1261 Acked 2260 Write 10 Done New Done true
4459 1262 Acked 2261 Read 9 Done New Done true
4460 1263 Acked 2262 Write 1 Done New Done true
4461 1264 Acked 2263 Read 2 Done New Done true
4462 1265 Acked 2264 Write 4 Done New Done true
4463 1266 Acked 2265 Read 9 Done New Done true
4464 1267 Acked 2266 Write 3 Done New Done true
4465 1268 Acked 2267 Write 8 Done New Done true
4466 1269 Acked 2268 Write 8 Done New Done true
4467 1270 Acked 2269 Write 6 Done New Done true
4468 1271 Acked 2270 Read 1 Done New Done true
4469 1272 Acked 2271 Write 9 Done New Done true
4470 1273 Acked 2272 Flush 0 Done New Done true
4471 1274 Acked 2273 Read 7 Done New Done true
4472 1275 Acked 2274 Read 10 Done New Done true
4473 1276 Acked 2275 Read 5 Done New Done true
4474 1277 Acked 2276 Write 1 Done New Done true
4475 1278 Acked 2277 Read 1 Done New Done true
4476 1279 Acked 2278 Write 7 Done New Done true
4477 1280 Acked 2279 Write 9 Done New Done true
4478 1281 Acked 2280 Read 3 Done New Done true
4479 1282 Acked 2281 Read 10 Done New Done true
4480 1283 Acked 2282 Write 8 Done New Done true
4481 1284 Acked 2283 Write 4 Done New Done true
4482 1285 Acked 2284 Read 6 Done New Done true
4483 1286 Acked 2285 Flush 0 Done New Done true
4484 1287 Acked 2286 Read 9 Done New Done true
4485 1288 Acked 2287 Read 5 Done New Done true
4486 1289 Acked 2288 Read 7 Done New Done true
4487 1290 Acked 2289 Write 10 Done New Done true
4488 1291 Acked 2290 Read 2 Done New Done true
4489 1292 Acked 2291 Read 4 Done New Done true
4490 1293 Acked 2292 Flush 0 Done New Done true
4491 1294 Acked 2293 Read 10 Done New Done true
4492 1295 Acked 2294 Write 5 Done New Done true
4493 1296 Acked 2295 Write 8 Done New Done true
4494 1297 Acked 2296 Write 3 Done New Done true
4495 1298 Acked 2297 Flush 0 Done New Done true
4496 1299 Acked 2298 Read 9 Done New Done true
4497 1300 Acked 2299 Write 1 Done New Done true
4498 1301 Acked 2300 Flush 0 Done New Done true
4499 1302 Acked 2301 Read 3 Done New Done true
4500 1303 Acked 2302 Read 5 Done New Done true
4501 1304 Acked 2303 Write 4 Done New Done true
4502 1305 Acked 2304 Read 1 Done New Done true
4503 1306 Acked 2305 Read 9 Done New Done true
4504 1307 Acked 2306 Read 7 Done New Done true
4505 1308 Acked 2307 Write 9 Done New Done true
4506 1309 Acked 2308 Read 4 Done New Done true
4507 1310 Acked 2309 Write 8 Done New Done true
4508 1311 Acked 2310 Flush 0 Done New Done true
4509 1312 Acked 2311 Read 4 Done New Done true
4510 1313 Acked 2312 Write 4 Done New Done true
4511 1314 Acked 2313 Read 7 Done New Done true
4512 1315 Acked 2314 Write 5 Done New Done true
4513 1316 Acked 2315 Flush 0 Done New Done true
4514 1317 Acked 2316 Write 5 Done New Done true
4515 1318 Acked 2317 Write 9 Done New Done true
4516 1319 Acked 2318 Write 9 Done New Done true
4517 1320 Acked 2319 Write 1 Done New Done true
4518 1321 Acked 2320 Write 2 Done New Done true
4519 1322 Acked 2321 Write 6 Done New Done true
4520 1323 Acked 2322 Write 10 Done New Done true
4521 1324 Acked 2323 Write 1 Done New Done true
4522 1325 Acked 2324 Flush 0 Done New Done true
4523 1326 Acked 2325 Write 5 Done New Done true
4524 1327 Acked 2326 Flush 0 Done New Done true
4525 1328 Acked 2327 Read 3 Done New Done true
4526 1329 Acked 2328 Read 9 Done New Done true
4527 1330 Acked 2329 Read 3 Done New Done true
4528 1331 Acked 2330 Read 4 Done New Done true
4529 1332 Acked 2331 Read 7 Done New Done true
4530 1333 Acked 2332 Write 2 Done New Done true
4531 1334 Acked 2333 Write 7 Done New Done true
4532 1335 Acked 2334 Write 7 Done New Done true
4533 1336 Acked 2335 Read 10 Done New Done true
4534 1337 Acked 2336 Read 2 Done New Done true
4535 1338 Acked 2337 Write 4 Done New Done true
4536 1339 Acked 2338 Flush 0 Done New Done true
4537 1340 Acked 2339 Write 9 Done New Done true
4538 1341 Acked 2340 Read 7 Done New Done true
4539 1342 Acked 2341 Read 2 Done New Done true
4540 1343 Acked 2342 Read 9 Done New Done true
4541 1344 Acked 2343 Read 10 Done New Done true
4542 1345 Acked 2344 Flush 0 Done New Done true
4543 1346 Acked 2345 Write 6 Done New Done true
4544 1347 Acked 2346 Read 9 Done New Done true
4545 1348 Acked 2347 Read 4 Done New Done true
4546 1349 Acked 2348 Flush 0 Done New Done true
4547 1350 Acked 2349 Read 7 Done New Done true
4548 1351 Acked 2350 Write 9 Done New Done true
4549 1352 Acked 2351 Write 5 Done New Done true
4550 1353 Acked 2352 Flush 0 Done New Done true
4551 1354 Acked 2353 Write 8 Done New Done true
4552 1355 Acked 2354 Write 8 Done New Done true
4553 1356 Acked 2355 Read 3 Done New Done true
4554 1357 Acked 2356 Read 1 Done New Done true
4555 1358 Acked 2357 Read 7 Done New Done true
4556 1359 Acked 2358 Read 7 Done New Done true
4557 1360 Acked 2359 Read 10 Done New Done true
4558 1361 Acked 2360 Flush 0 Done New Done true
4559 1362 Acked 2361 Read 7 Done New Done true
4560 1363 Acked 2362 Flush 0 Done New Done true
4561 1364 Acked 2363 Read 7 Done New Done true
4562 1365 Acked 2364 Write 5 Done New Done true
4563 1366 Acked 2365 Write 9 Done New Done true
4564 1367 Acked 2366 Flush 0 Done New Done true
4565 1368 Acked 2367 Write 5 Done New Done true
4566 1369 Acked 2368 Read 6 Done New Done true
4567 1370 Acked 2369 Read 7 Done New Done true
4568 1371 Acked 2370 Read 1 Done New Done true
4569 1372 Acked 2371 Read 4 Done New Done true
4570 1373 Acked 2372 Write 1 Done New Done true
4571 1374 Acked 2373 Flush 0 Done New Done true
4572 1375 Acked 2374 Read 1 Done New Done true
4573 1376 Acked 2375 Read 10 Done New Done true
4574 1377 Acked 2376 Read 3 Done New Done true
4575 1378 Acked 2377 Read 1 Done New Done true
4576 1379 Acked 2378 Read 8 Done New Done true
4577 1380 Acked 2379 Write 7 Done New Done true
4578 1381 Acked 2380 Read 1 Done New Done true
4579 1382 Acked 2381 Flush 0 Done New Done true
4580 1383 Acked 2382 Write 8 Done New Done true
4581 1384 Acked 2383 Read 4 Done New Done true
4582 1385 Acked 2384 Read 7 Done New Done true
4583 1386 Acked 2385 Flush 0 Done New Done true
4584 1387 Acked 2386 Read 1 Done New Done true
4585 1388 Acked 2387 Read 3 Done New Done true
4586 1389 Acked 2388 Write 9 Done New Done true
4587 1390 Acked 2389 Read 4 Done New Done true
4588 1391 Acked 2390 Read 9 Done New Done true
4589 1392 Acked 2391 Read 1 Done New Done true
4590 1393 Acked 2392 Read 2 Done New Done true
4591 1394 Acked 2393 Write 5 Done New Done true
4592 1395 Acked 2394 Write 4 Done New Done true
4593 1396 Acked 2395 Write 1 Done New Done true
4594 1397 Acked 2396 Read 9 Done New Done true
4595 1398 Acked 2397 Read 2 Done New Done true
4596 1399 Acked 2398 Read 5 Done New Done true
4597 1400 Acked 2399 Read 5 Done New Done true
4598 1401 Acked 2400 Read 2 Done New Done true
4599 1402 Acked 2401 Read 9 Done New Done true
4600 1403 Acked 2402 Read 7 Done New Done true
4601 1404 Acked 2403 Read 6 Done New Done true
4602 1405 Acked 2404 Read 2 Done New Done true
4603 1406 Acked 2405 Write 4 Done New Done true
4604 1407 Acked 2406 Read 9 Done New Done true
4605 1408 Acked 2407 Read 9 Done New Done true
4606 1409 Acked 2408 Write 3 Done New Done true
4607 1410 Acked 2409 Flush 0 Done New Done true
4608 1411 Acked 2410 Read 10 Done New Done true
4609 1412 Acked 2411 Read 6 Done New Done true
4610 1413 Acked 2412 Read 8 Done New Done true
4611 1414 Acked 2413 Read 1 Done New Done true
4612 1415 Acked 2414 Read 8 Done New Done true
4613 1416 Acked 2415 Write 8 Done New Done true
4614 1417 Acked 2416 Write 1 Done New Done true
4615 1418 Acked 2417 Write 2 Done New Done true
4616 1419 Acked 2418 Write 1 Done New Done true
4617 1420 Acked 2419 Write 9 Done New Done true
4618 1421 Acked 2420 Read 8 Done New Done true
4619 1422 Acked 2421 Read 10 Done New Done true
4620 1423 Acked 2422 Read 9 Done New Done true
4621 1424 Acked 2423 Write 9 Done New Done true
4622 1425 Acked 2424 Read 2 Done New Done true
4623 1426 Acked 2425 Write 1 Done New Done true
4624 1427 Acked 2426 Read 2 Done New Done true
4625 1428 Acked 2427 Write 6 Done New Done true
4626 1429 Acked 2428 Write 8 Done New Done true
4627 1430 Acked 2429 Write 7 Done New Done true
4628 1431 Acked 2430 Read 8 Done New Done true
4629 1432 Acked 2431 Read 5 Done New Done true
4630 1433 Acked 2432 Read 8 Done New Done true
4631 1434 Acked 2433 Read 1 Done New Done true
4632 1435 Acked 2434 Write 10 Done New Done true
4633 1436 Acked 2435 Read 3 Done New Done true
4634 1437 Acked 2436 Flush 0 Done New Done true
4635 1438 Acked 2437 Write 1 Done New Done true
4636 1439 Acked 2438 Read 8 Done New Done true
4637 1440 Acked 2439 Read 3 Done New Done true
4638 1441 Acked 2440 Read 3 Done New Done true
4639 1442 Acked 2441 Write 8 Done New Done true
4640 1443 Acked 2442 Flush 0 Done New Done true
4641 1444 Acked 2443 Read 9 Done New Done true
4642 1445 Acked 2444 Write 10 Done New Done true
4643 1446 Acked 2445 Write 9 Done New Done true
4644 1447 Acked 2446 Read 5 Done New Done true
4645 1448 Acked 2447 Write 2 Done New Done true
4646 1449 Acked 2448 Write 3 Done New Done true
4647 1450 Acked 2449 Read 1 Done New Done true
4648 1451 Acked 2450 Flush 0 Done New Done true
4649 1452 Acked 2451 Flush 0 Done New Done true
4650 1453 Acked 2452 Write 1 Done New Done true
4651 1454 Acked 2453 Read 4 Done New Done true
4652 1455 Acked 2454 Read 8 Done New Done true
4653 1456 Acked 2455 Read 9 Done New Done true
4654 1457 Acked 2456 Flush 0 Done New Done true
4655 1458 Acked 2457 Read 5 Done New Done true
4656 1459 Acked 2458 Read 8 Done New Done true
4657 1460 Acked 2459 Write 3 Done New Done true
4658 1461 Acked 2460 Write 6 Done New Done true
4659 1462 Acked 2461 Write 3 Done New Done true
4660 1463 Acked 2462 Read 2 Done New Done true
4661 1464 Acked 2463 Write 9 Done New Done true
4662 1465 Acked 2464 Read 2 Done New Done true
4663 1466 Acked 2465 Write 1 Done New Done true
4664 1467 Acked 2466 Read 3 Done New Done true
4665 1468 Acked 2467 Read 1 Done New Done true
4666 1469 Acked 2468 Write 7 Done New Done true
4667 1470 Acked 2469 Write 2 Done New Done true
4668 1471 Acked 2470 Read 1 Done New Done true
4669 1472 Acked 2471 Write 9 Done New Done true
4670 1473 Acked 2472 Flush 0 Done New Done true
4671 1474 Acked 2473 Flush 0 Done New Done true
4672 1475 Acked 2474 Read 7 Done New Done true
4673 1476 Acked 2475 Read 6 Done New Done true
4674 1477 Acked 2476 Read 8 Done New Done true
4675 1478 Acked 2477 Read 5 Done New Done true
4676 1479 Acked 2478 Flush 0 Done New Done true
4677 1480 Acked 2479 Read 2 Done New Done true
4678 1481 Acked 2480 Write 9 Done New Done true
4679 1482 Acked 2481 Write 4 Done New Done true
4680 1483 Acked 2482 Read 2 Done New Done true
4681 1484 Acked 2483 Write 8 Done New Done true
4682 1485 Acked 2484 Read 9 Done New Done true
4683 1486 Acked 2485 Write 6 Done New Done true
4684 1487 Acked 2486 Flush 0 Done New Done true
4685 1488 Acked 2487 Read 7 Done New Done true
4686 1489 Acked 2488 Read 5 Done New Done true
4687 1490 Acked 2489 Write 7 Done New Done true
4688 1491 Acked 2490 Read 4 Done New Done true
4689 1492 Acked 2491 Write 9 Done New Done true
4690 1493 Acked 2492 Read 9 Done New Done true
4691 1494 Acked 2493 Read 10 Done New Done true
4692 1495 Acked 2494 Read 3 Done New Done true
4693 1496 Acked 2495 Flush 0 Done New Done true
4694 1497 Acked 2496 Read 8 Done New Done true
4695 1498 Acked 2497 Read 6 Done New Done true
4696 1499 Acked 2498 Read 4 Done New Done true
4697 1500 Acked 2499 Write 9 Done New Done true
4698 1501 Acked 2500 Write 9 Done New Done true
4699 1502 Acked 2501 Read 1 Done New Done true
4700 1503 Acked 2502 Write 3 Done New Done true
4701 1504 Acked 2503 Read 5 Done New Done true
4702 1505 Acked 2504 Write 9 Done New Done true
4703 1506 Acked 2505 Flush 0 Done New Done true
4704 STATES DS:0 DS:1 DS:2 TOTAL
4705 New 0 301 0 301
4706 Sent 0 0 0 0
4707 Done 301 0 301 602
4708 Skipped 0 0 0 0
4709 Error 0 0 0 0
4710 Last Flush: 2505 2204 2505
4711 Downstairs last five completed: 2204 2203 2202 2201 2200
4712 Upstairs last five completed: 1506 1505 1504 1503 1502
4713 CLIENT: Up:0 ds:301 act:2
47142024-02-15T02:58:24.964ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
47152024-02-15T02:58:24.964ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
47162024-02-15T02:58:24.964ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
47172024-02-15T02:58:24.964ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
47182024-02-15T02:58:24.964ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
47192024-02-15T02:58:24.964ZINFOcrucible: [1] client re-new 301 jobs since flush 2204 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
47202024-02-15T02:58:24.965ZINFOcrucible: client task is exiting = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
47212024-02-15T02:58:25.965ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
47222024-02-15T02:58:25.966ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
47232024-02-15T02:58:25.966ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
47242024-02-15T02:58:25.966ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
47252024-02-15T02:58:25.966ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
47262024-02-15T02:58:25.966ZINFOcrucible: [1] client re-new 301 jobs since flush 2204 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
47272024-02-15T02:58:25.966ZINFOcrucible: client task is exiting = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
47282024-02-15T02:58:26.967ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
47292024-02-15T02:58:26.967ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
47302024-02-15T02:58:26.967ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
47312024-02-15T02:58:26.967ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
47322024-02-15T02:58:26.967ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
47332024-02-15T02:58:26.967ZINFOcrucible: [1] client re-new 301 jobs since flush 2204 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
47342024-02-15T02:58:26.967ZINFOcrucible: client task is exiting = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
47352024-02-15T02:58:27.969ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
47362024-02-15T02:58:27.969ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
47372024-02-15T02:58:27.969ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
47382024-02-15T02:58:27.969ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
47392024-02-15T02:58:27.969ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
47402024-02-15T02:58:27.969ZINFOcrucible: [1] client re-new 301 jobs since flush 2204 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
47412024-02-15T02:58:27.969ZINFOcrucible: client task is exiting = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
4742 ----------------------------------------------------------------
4743 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
4744 States: Active Offline Active
4745 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
4746 1206 Acked 2205 Read 2 Done New Done true
4747 1207 Acked 2206 Read 7 Done New Done true
4748 1208 Acked 2207 Flush 0 Done New Done true
4749 1209 Acked 2208 Read 3 Done New Done true
4750 1210 Acked 2209 Read 6 Done New Done true
4751 1211 Acked 2210 Write 9 Done New Done true
4752 1212 Acked 2211 Read 5 Done New Done true
4753 1213 Acked 2212 Read 3 Done New Done true
4754 1214 Acked 2213 Read 9 Done New Done true
4755 1215 Acked 2214 Read 9 Done New Done true
4756 1216 Acked 2215 Flush 0 Done New Done true
4757 1217 Acked 2216 Read 7 Done New Done true
4758 1218 Acked 2217 Flush 0 Done New Done true
4759 1219 Acked 2218 Read 8 Done New Done true
4760 1220 Acked 2219 Read 9 Done New Done true
4761 1221 Acked 2220 Write 7 Done New Done true
4762 1222 Acked 2221 Read 4 Done New Done true
4763 1223 Acked 2222 Flush 0 Done New Done true
4764 1224 Acked 2223 Flush 0 Done New Done true
4765 1225 Acked 2224 Write 4 Done New Done true
4766 1226 Acked 2225 Read 10 Done New Done true
4767 1227 Acked 2226 Read 8 Done New Done true
4768 1228 Acked 2227 Write 6 Done New Done true
4769 1229 Acked 2228 Read 6 Done New Done true
4770 1230 Acked 2229 Read 5 Done New Done true
4771 1231 Acked 2230 Read 10 Done New Done true
4772 1232 Acked 2231 Read 8 Done New Done true
4773 1233 Acked 2232 Write 10 Done New Done true
4774 1234 Acked 2233 Write 6 Done New Done true
4775 1235 Acked 2234 Read 6 Done New Done true
4776 1236 Acked 2235 Read 7 Done New Done true
4777 1237 Acked 2236 Flush 0 Done New Done true
4778 1238 Acked 2237 Read 3 Done New Done true
4779 1239 Acked 2238 Read 10 Done New Done true
4780 1240 Acked 2239 Read 1 Done New Done true
4781 1241 Acked 2240 Read 6 Done New Done true
4782 1242 Acked 2241 Write 4 Done New Done true
4783 1243 Acked 2242 Read 6 Done New Done true
4784 1244 Acked 2243 Write 5 Done New Done true
4785 1245 Acked 2244 Read 7 Done New Done true
4786 1246 Acked 2245 Read 5 Done New Done true
4787 1247 Acked 2246 Read 2 Done New Done true
4788 1248 Acked 2247 Write 1 Done New Done true
4789 1249 Acked 2248 Write 2 Done New Done true
4790 1250 Acked 2249 Write 10 Done New Done true
4791 1251 Acked 2250 Read 8 Done New Done true
4792 1252 Acked 2251 Write 9 Done New Done true
4793 1253 Acked 2252 Write 6 Done New Done true
4794 1254 Acked 2253 Read 2 Done New Done true
4795 1255 Acked 2254 Read 10 Done New Done true
4796 1256 Acked 2255 Write 1 Done New Done true
4797 1257 Acked 2256 Read 10 Done New Done true
4798 1258 Acked 2257 Flush 0 Done New Done true
4799 1259 Acked 2258 Read 9 Done New Done true
4800 1260 Acked 2259 Write 10 Done New Done true
4801 1261 Acked 2260 Write 10 Done New Done true
4802 1262 Acked 2261 Read 9 Done New Done true
4803 1263 Acked 2262 Write 1 Done New Done true
4804 1264 Acked 2263 Read 2 Done New Done true
4805 1265 Acked 2264 Write 4 Done New Done true
4806 1266 Acked 2265 Read 9 Done New Done true
4807 1267 Acked 2266 Write 3 Done New Done true
4808 1268 Acked 2267 Write 8 Done New Done true
4809 1269 Acked 2268 Write 8 Done New Done true
4810 1270 Acked 2269 Write 6 Done New Done true
4811 1271 Acked 2270 Read 1 Done New Done true
4812 1272 Acked 2271 Write 9 Done New Done true
4813 1273 Acked 2272 Flush 0 Done New Done true
4814 1274 Acked 2273 Read 7 Done New Done true
4815 1275 Acked 2274 Read 10 Done New Done true
4816 1276 Acked 2275 Read 5 Done New Done true
4817 1277 Acked 2276 Write 1 Done New Done true
4818 1278 Acked 2277 Read 1 Done New Done true
4819 1279 Acked 2278 Write 7 Done New Done true
4820 1280 Acked 2279 Write 9 Done New Done true
4821 1281 Acked 2280 Read 3 Done New Done true
4822 1282 Acked 2281 Read 10 Done New Done true
4823 1283 Acked 2282 Write 8 Done New Done true
4824 1284 Acked 2283 Write 4 Done New Done true
4825 1285 Acked 2284 Read 6 Done New Done true
4826 1286 Acked 2285 Flush 0 Done New Done true
4827 1287 Acked 2286 Read 9 Done New Done true
4828 1288 Acked 2287 Read 5 Done New Done true
4829 1289 Acked 2288 Read 7 Done New Done true
4830 1290 Acked 2289 Write 10 Done New Done true
4831 1291 Acked 2290 Read 2 Done New Done true
4832 1292 Acked 2291 Read 4 Done New Done true
4833 1293 Acked 2292 Flush 0 Done New Done true
4834 1294 Acked 2293 Read 10 Done New Done true
4835 1295 Acked 2294 Write 5 Done New Done true
4836 1296 Acked 2295 Write 8 Done New Done true
4837 1297 Acked 2296 Write 3 Done New Done true
4838 1298 Acked 2297 Flush 0 Done New Done true
4839 1299 Acked 2298 Read 9 Done New Done true
4840 1300 Acked 2299 Write 1 Done New Done true
4841 1301 Acked 2300 Flush 0 Done New Done true
4842 1302 Acked 2301 Read 3 Done New Done true
4843 1303 Acked 2302 Read 5 Done New Done true
4844 1304 Acked 2303 Write 4 Done New Done true
4845 1305 Acked 2304 Read 1 Done New Done true
4846 1306 Acked 2305 Read 9 Done New Done true
4847 1307 Acked 2306 Read 7 Done New Done true
4848 1308 Acked 2307 Write 9 Done New Done true
4849 1309 Acked 2308 Read 4 Done New Done true
4850 1310 Acked 2309 Write 8 Done New Done true
4851 1311 Acked 2310 Flush 0 Done New Done true
4852 1312 Acked 2311 Read 4 Done New Done true
4853 1313 Acked 2312 Write 4 Done New Done true
4854 1314 Acked 2313 Read 7 Done New Done true
4855 1315 Acked 2314 Write 5 Done New Done true
4856 1316 Acked 2315 Flush 0 Done New Done true
4857 1317 Acked 2316 Write 5 Done New Done true
4858 1318 Acked 2317 Write 9 Done New Done true
4859 1319 Acked 2318 Write 9 Done New Done true
4860 1320 Acked 2319 Write 1 Done New Done true
4861 1321 Acked 2320 Write 2 Done New Done true
4862 1322 Acked 2321 Write 6 Done New Done true
4863 1323 Acked 2322 Write 10 Done New Done true
4864 1324 Acked 2323 Write 1 Done New Done true
4865 1325 Acked 2324 Flush 0 Done New Done true
4866 1326 Acked 2325 Write 5 Done New Done true
4867 1327 Acked 2326 Flush 0 Done New Done true
4868 1328 Acked 2327 Read 3 Done New Done true
4869 1329 Acked 2328 Read 9 Done New Done true
4870 1330 Acked 2329 Read 3 Done New Done true
4871 1331 Acked 2330 Read 4 Done New Done true
4872 1332 Acked 2331 Read 7 Done New Done true
4873 1333 Acked 2332 Write 2 Done New Done true
4874 1334 Acked 2333 Write 7 Done New Done true
4875 1335 Acked 2334 Write 7 Done New Done true
4876 1336 Acked 2335 Read 10 Done New Done true
4877 1337 Acked 2336 Read 2 Done New Done true
4878 1338 Acked 2337 Write 4 Done New Done true
4879 1339 Acked 2338 Flush 0 Done New Done true
4880 1340 Acked 2339 Write 9 Done New Done true
4881 1341 Acked 2340 Read 7 Done New Done true
4882 1342 Acked 2341 Read 2 Done New Done true
4883 1343 Acked 2342 Read 9 Done New Done true
4884 1344 Acked 2343 Read 10 Done New Done true
4885 1345 Acked 2344 Flush 0 Done New Done true
4886 1346 Acked 2345 Write 6 Done New Done true
4887 1347 Acked 2346 Read 9 Done New Done true
4888 1348 Acked 2347 Read 4 Done New Done true
4889 1349 Acked 2348 Flush 0 Done New Done true
4890 1350 Acked 2349 Read 7 Done New Done true
4891 1351 Acked 2350 Write 9 Done New Done true
4892 1352 Acked 2351 Write 5 Done New Done true
4893 1353 Acked 2352 Flush 0 Done New Done true
4894 1354 Acked 2353 Write 8 Done New Done true
4895 1355 Acked 2354 Write 8 Done New Done true
4896 1356 Acked 2355 Read 3 Done New Done true
4897 1357 Acked 2356 Read 1 Done New Done true
4898 1358 Acked 2357 Read 7 Done New Done true
4899 1359 Acked 2358 Read 7 Done New Done true
4900 1360 Acked 2359 Read 10 Done New Done true
4901 1361 Acked 2360 Flush 0 Done New Done true
4902 1362 Acked 2361 Read 7 Done New Done true
4903 1363 Acked 2362 Flush 0 Done New Done true
4904 1364 Acked 2363 Read 7 Done New Done true
4905 1365 Acked 2364 Write 5 Done New Done true
4906 1366 Acked 2365 Write 9 Done New Done true
4907 1367 Acked 2366 Flush 0 Done New Done true
4908 1368 Acked 2367 Write 5 Done New Done true
4909 1369 Acked 2368 Read 6 Done New Done true
4910 1370 Acked 2369 Read 7 Done New Done true
4911 1371 Acked 2370 Read 1 Done New Done true
4912 1372 Acked 2371 Read 4 Done New Done true
4913 1373 Acked 2372 Write 1 Done New Done true
4914 1374 Acked 2373 Flush 0 Done New Done true
4915 1375 Acked 2374 Read 1 Done New Done true
4916 1376 Acked 2375 Read 10 Done New Done true
4917 1377 Acked 2376 Read 3 Done New Done true
4918 1378 Acked 2377 Read 1 Done New Done true
4919 1379 Acked 2378 Read 8 Done New Done true
4920 1380 Acked 2379 Write 7 Done New Done true
4921 1381 Acked 2380 Read 1 Done New Done true
4922 1382 Acked 2381 Flush 0 Done New Done true
4923 1383 Acked 2382 Write 8 Done New Done true
4924 1384 Acked 2383 Read 4 Done New Done true
4925 1385 Acked 2384 Read 7 Done New Done true
4926 1386 Acked 2385 Flush 0 Done New Done true
4927 1387 Acked 2386 Read 1 Done New Done true
4928 1388 Acked 2387 Read 3 Done New Done true
4929 1389 Acked 2388 Write 9 Done New Done true
4930 1390 Acked 2389 Read 4 Done New Done true
4931 1391 Acked 2390 Read 9 Done New Done true
4932 1392 Acked 2391 Read 1 Done New Done true
4933 1393 Acked 2392 Read 2 Done New Done true
4934 1394 Acked 2393 Write 5 Done New Done true
4935 1395 Acked 2394 Write 4 Done New Done true
4936 1396 Acked 2395 Write 1 Done New Done true
4937 1397 Acked 2396 Read 9 Done New Done true
4938 1398 Acked 2397 Read 2 Done New Done true
4939 1399 Acked 2398 Read 5 Done New Done true
4940 1400 Acked 2399 Read 5 Done New Done true
4941 1401 Acked 2400 Read 2 Done New Done true
4942 1402 Acked 2401 Read 9 Done New Done true
4943 1403 Acked 2402 Read 7 Done New Done true
4944 1404 Acked 2403 Read 6 Done New Done true
4945 1405 Acked 2404 Read 2 Done New Done true
4946 1406 Acked 2405 Write 4 Done New Done true
4947 1407 Acked 2406 Read 9 Done New Done true
4948 1408 Acked 2407 Read 9 Done New Done true
4949 1409 Acked 2408 Write 3 Done New Done true
4950 1410 Acked 2409 Flush 0 Done New Done true
4951 1411 Acked 2410 Read 10 Done New Done true
4952 1412 Acked 2411 Read 6 Done New Done true
4953 1413 Acked 2412 Read 8 Done New Done true
4954 1414 Acked 2413 Read 1 Done New Done true
4955 1415 Acked 2414 Read 8 Done New Done true
4956 1416 Acked 2415 Write 8 Done New Done true
4957 1417 Acked 2416 Write 1 Done New Done true
4958 1418 Acked 2417 Write 2 Done New Done true
4959 1419 Acked 2418 Write 1 Done New Done true
4960 1420 Acked 2419 Write 9 Done New Done true
4961 1421 Acked 2420 Read 8 Done New Done true
4962 1422 Acked 2421 Read 10 Done New Done true
4963 1423 Acked 2422 Read 9 Done New Done true
4964 1424 Acked 2423 Write 9 Done New Done true
4965 1425 Acked 2424 Read 2 Done New Done true
4966 1426 Acked 2425 Write 1 Done New Done true
4967 1427 Acked 2426 Read 2 Done New Done true
4968 1428 Acked 2427 Write 6 Done New Done true
4969 1429 Acked 2428 Write 8 Done New Done true
4970 1430 Acked 2429 Write 7 Done New Done true
4971 1431 Acked 2430 Read 8 Done New Done true
4972 1432 Acked 2431 Read 5 Done New Done true
4973 1433 Acked 2432 Read 8 Done New Done true
4974 1434 Acked 2433 Read 1 Done New Done true
4975 1435 Acked 2434 Write 10 Done New Done true
4976 1436 Acked 2435 Read 3 Done New Done true
4977 1437 Acked 2436 Flush 0 Done New Done true
4978 1438 Acked 2437 Write 1 Done New Done true
4979 1439 Acked 2438 Read 8 Done New Done true
4980 1440 Acked 2439 Read 3 Done New Done true
4981 1441 Acked 2440 Read 3 Done New Done true
4982 1442 Acked 2441 Write 8 Done New Done true
4983 1443 Acked 2442 Flush 0 Done New Done true
4984 1444 Acked 2443 Read 9 Done New Done true
4985 1445 Acked 2444 Write 10 Done New Done true
4986 1446 Acked 2445 Write 9 Done New Done true
4987 1447 Acked 2446 Read 5 Done New Done true
4988 1448 Acked 2447 Write 2 Done New Done true
4989 1449 Acked 2448 Write 3 Done New Done true
4990 1450 Acked 2449 Read 1 Done New Done true
4991 1451 Acked 2450 Flush 0 Done New Done true
4992 1452 Acked 2451 Flush 0 Done New Done true
4993 1453 Acked 2452 Write 1 Done New Done true
4994 1454 Acked 2453 Read 4 Done New Done true
4995 1455 Acked 2454 Read 8 Done New Done true
4996 1456 Acked 2455 Read 9 Done New Done true
4997 1457 Acked 2456 Flush 0 Done New Done true
4998 1458 Acked 2457 Read 5 Done New Done true
4999 1459 Acked 2458 Read 8 Done New Done true
5000 1460 Acked 2459 Write 3 Done New Done true
5001 1461 Acked 2460 Write 6 Done New Done true
5002 1462 Acked 2461 Write 3 Done New Done true
5003 1463 Acked 2462 Read 2 Done New Done true
5004 1464 Acked 2463 Write 9 Done New Done true
5005 1465 Acked 2464 Read 2 Done New Done true
5006 1466 Acked 2465 Write 1 Done New Done true
5007 1467 Acked 2466 Read 3 Done New Done true
5008 1468 Acked 2467 Read 1 Done New Done true
5009 1469 Acked 2468 Write 7 Done New Done true
5010 1470 Acked 2469 Write 2 Done New Done true
5011 1471 Acked 2470 Read 1 Done New Done true
5012 1472 Acked 2471 Write 9 Done New Done true
5013 1473 Acked 2472 Flush 0 Done New Done true
5014 1474 Acked 2473 Flush 0 Done New Done true
5015 1475 Acked 2474 Read 7 Done New Done true
5016 1476 Acked 2475 Read 6 Done New Done true
5017 1477 Acked 2476 Read 8 Done New Done true
5018 1478 Acked 2477 Read 5 Done New Done true
5019 1479 Acked 2478 Flush 0 Done New Done true
5020 1480 Acked 2479 Read 2 Done New Done true
5021 1481 Acked 2480 Write 9 Done New Done true
5022 1482 Acked 2481 Write 4 Done New Done true
5023 1483 Acked 2482 Read 2 Done New Done true
5024 1484 Acked 2483 Write 8 Done New Done true
5025 1485 Acked 2484 Read 9 Done New Done true
5026 1486 Acked 2485 Write 6 Done New Done true
5027 1487 Acked 2486 Flush 0 Done New Done true
5028 1488 Acked 2487 Read 7 Done New Done true
5029 1489 Acked 2488 Read 5 Done New Done true
5030 1490 Acked 2489 Write 7 Done New Done true
5031 1491 Acked 2490 Read 4 Done New Done true
5032 1492 Acked 2491 Write 9 Done New Done true
5033 1493 Acked 2492 Read 9 Done New Done true
5034 1494 Acked 2493 Read 10 Done New Done true
5035 1495 Acked 2494 Read 3 Done New Done true
5036 1496 Acked 2495 Flush 0 Done New Done true
5037 1497 Acked 2496 Read 8 Done New Done true
5038 1498 Acked 2497 Read 6 Done New Done true
5039 1499 Acked 2498 Read 4 Done New Done true
5040 1500 Acked 2499 Write 9 Done New Done true
5041 1501 Acked 2500 Write 9 Done New Done true
5042 1502 Acked 2501 Read 1 Done New Done true
5043 1503 Acked 2502 Write 3 Done New Done true
5044 1504 Acked 2503 Read 5 Done New Done true
5045 1505 Acked 2504 Write 9 Done New Done true
5046 1506 Acked 2505 Flush 0 Done New Done true
5047 STATES DS:0 DS:1 DS:2 TOTAL
5048 New 0 301 0 301
5049 Sent 0 0 0 0
5050 Done 301 0 301 602
5051 Skipped 0 0 0 0
5052 Error 0 0 0 0
5053 Last Flush: 2505 2204 2505
5054 Downstairs last five completed: 2204 2203 2202 2201 2200
5055 Upstairs last five completed: 1506 1505 1504 1503 1502
5056 CLIENT: Up:0 ds:301 act:2
50572024-02-15T02:58:28.970ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
50582024-02-15T02:58:28.970ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
50592024-02-15T02:58:28.970ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
50602024-02-15T02:58:28.970ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
50612024-02-15T02:58:28.970ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
50622024-02-15T02:58:28.970ZINFOcrucible: [1] client re-new 301 jobs since flush 2204 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
50632024-02-15T02:58:28.970ZINFOcrucible: client task is exiting = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
50642024-02-15T02:58:29.972ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
50652024-02-15T02:58:29.972ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
50662024-02-15T02:58:29.972ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
50672024-02-15T02:58:29.972ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
50682024-02-15T02:58:29.972ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
50692024-02-15T02:58:29.972ZINFOcrucible: [1] client re-new 301 jobs since flush 2204 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
50702024-02-15T02:58:29.972ZINFOcrucible: client task is exiting = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
50712024-02-15T02:58:30.973ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
50722024-02-15T02:58:30.973ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
50732024-02-15T02:58:30.973ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
50742024-02-15T02:58:30.973ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
50752024-02-15T02:58:30.973ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
50762024-02-15T02:58:30.973ZINFOcrucible: [1] client re-new 301 jobs since flush 2204 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
50772024-02-15T02:58:30.974ZINFOcrucible: client task is exiting = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
50782024-02-15T02:58:31.974ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
50792024-02-15T02:58:31.975ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
50802024-02-15T02:58:31.975ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
50812024-02-15T02:58:31.975ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
50822024-02-15T02:58:31.975ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
50832024-02-15T02:58:31.975ZINFOcrucible: [1] client re-new 301 jobs since flush 2204 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
50842024-02-15T02:58:31.975ZINFOcrucible: client task is exiting = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
5085 ----------------------------------------------------------------
5086 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
5087 States: Active Offline Active
5088 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
5089 1206 Acked 2205 Read 2 Done New Done true
5090 1207 Acked 2206 Read 7 Done New Done true
5091 1208 Acked 2207 Flush 0 Done New Done true
5092 1209 Acked 2208 Read 3 Done New Done true
5093 1210 Acked 2209 Read 6 Done New Done true
5094 1211 Acked 2210 Write 9 Done New Done true
5095 1212 Acked 2211 Read 5 Done New Done true
5096 1213 Acked 2212 Read 3 Done New Done true
5097 1214 Acked 2213 Read 9 Done New Done true
5098 1215 Acked 2214 Read 9 Done New Done true
5099 1216 Acked 2215 Flush 0 Done New Done true
5100 1217 Acked 2216 Read 7 Done New Done true
5101 1218 Acked 2217 Flush 0 Done New Done true
5102 1219 Acked 2218 Read 8 Done New Done true
5103 1220 Acked 2219 Read 9 Done New Done true
5104 1221 Acked 2220 Write 7 Done New Done true
5105 1222 Acked 2221 Read 4 Done New Done true
5106 1223 Acked 2222 Flush 0 Done New Done true
5107 1224 Acked 2223 Flush 0 Done New Done true
5108 1225 Acked 2224 Write 4 Done New Done true
5109 1226 Acked 2225 Read 10 Done New Done true
5110 1227 Acked 2226 Read 8 Done New Done true
5111 1228 Acked 2227 Write 6 Done New Done true
5112 1229 Acked 2228 Read 6 Done New Done true
5113 1230 Acked 2229 Read 5 Done New Done true
5114 1231 Acked 2230 Read 10 Done New Done true
5115 1232 Acked 2231 Read 8 Done New Done true
5116 1233 Acked 2232 Write 10 Done New Done true
5117 1234 Acked 2233 Write 6 Done New Done true
5118 1235 Acked 2234 Read 6 Done New Done true
5119 1236 Acked 2235 Read 7 Done New Done true
5120 1237 Acked 2236 Flush 0 Done New Done true
5121 1238 Acked 2237 Read 3 Done New Done true
5122 1239 Acked 2238 Read 10 Done New Done true
5123 1240 Acked 2239 Read 1 Done New Done true
5124 1241 Acked 2240 Read 6 Done New Done true
5125 1242 Acked 2241 Write 4 Done New Done true
5126 1243 Acked 2242 Read 6 Done New Done true
5127 1244 Acked 2243 Write 5 Done New Done true
5128 1245 Acked 2244 Read 7 Done New Done true
5129 1246 Acked 2245 Read 5 Done New Done true
5130 1247 Acked 2246 Read 2 Done New Done true
5131 1248 Acked 2247 Write 1 Done New Done true
5132 1249 Acked 2248 Write 2 Done New Done true
5133 1250 Acked 2249 Write 10 Done New Done true
5134 1251 Acked 2250 Read 8 Done New Done true
5135 1252 Acked 2251 Write 9 Done New Done true
5136 1253 Acked 2252 Write 6 Done New Done true
5137 1254 Acked 2253 Read 2 Done New Done true
5138 1255 Acked 2254 Read 10 Done New Done true
5139 1256 Acked 2255 Write 1 Done New Done true
5140 1257 Acked 2256 Read 10 Done New Done true
5141 1258 Acked 2257 Flush 0 Done New Done true
5142 1259 Acked 2258 Read 9 Done New Done true
5143 1260 Acked 2259 Write 10 Done New Done true
5144 1261 Acked 2260 Write 10 Done New Done true
5145 1262 Acked 2261 Read 9 Done New Done true
5146 1263 Acked 2262 Write 1 Done New Done true
5147 1264 Acked 2263 Read 2 Done New Done true
5148 1265 Acked 2264 Write 4 Done New Done true
5149 1266 Acked 2265 Read 9 Done New Done true
5150 1267 Acked 2266 Write 3 Done New Done true
5151 1268 Acked 2267 Write 8 Done New Done true
5152 1269 Acked 2268 Write 8 Done New Done true
5153 1270 Acked 2269 Write 6 Done New Done true
5154 1271 Acked 2270 Read 1 Done New Done true
5155 1272 Acked 2271 Write 9 Done New Done true
5156 1273 Acked 2272 Flush 0 Done New Done true
5157 1274 Acked 2273 Read 7 Done New Done true
5158 1275 Acked 2274 Read 10 Done New Done true
5159 1276 Acked 2275 Read 5 Done New Done true
5160 1277 Acked 2276 Write 1 Done New Done true
5161 1278 Acked 2277 Read 1 Done New Done true
5162 1279 Acked 2278 Write 7 Done New Done true
5163 1280 Acked 2279 Write 9 Done New Done true
5164 1281 Acked 2280 Read 3 Done New Done true
5165 1282 Acked 2281 Read 10 Done New Done true
5166 1283 Acked 2282 Write 8 Done New Done true
5167 1284 Acked 2283 Write 4 Done New Done true
5168 1285 Acked 2284 Read 6 Done New Done true
5169 1286 Acked 2285 Flush 0 Done New Done true
5170 1287 Acked 2286 Read 9 Done New Done true
5171 1288 Acked 2287 Read 5 Done New Done true
5172 1289 Acked 2288 Read 7 Done New Done true
5173 1290 Acked 2289 Write 10 Done New Done true
5174 1291 Acked 2290 Read 2 Done New Done true
5175 1292 Acked 2291 Read 4 Done New Done true
5176 1293 Acked 2292 Flush 0 Done New Done true
5177 1294 Acked 2293 Read 10 Done New Done true
5178 1295 Acked 2294 Write 5 Done New Done true
5179 1296 Acked 2295 Write 8 Done New Done true
5180 1297 Acked 2296 Write 3 Done New Done true
5181 1298 Acked 2297 Flush 0 Done New Done true
5182 1299 Acked 2298 Read 9 Done New Done true
5183 1300 Acked 2299 Write 1 Done New Done true
5184 1301 Acked 2300 Flush 0 Done New Done true
5185 1302 Acked 2301 Read 3 Done New Done true
5186 1303 Acked 2302 Read 5 Done New Done true
5187 1304 Acked 2303 Write 4 Done New Done true
5188 1305 Acked 2304 Read 1 Done New Done true
5189 1306 Acked 2305 Read 9 Done New Done true
5190 1307 Acked 2306 Read 7 Done New Done true
5191 1308 Acked 2307 Write 9 Done New Done true
5192 1309 Acked 2308 Read 4 Done New Done true
5193 1310 Acked 2309 Write 8 Done New Done true
5194 1311 Acked 2310 Flush 0 Done New Done true
5195 1312 Acked 2311 Read 4 Done New Done true
5196 1313 Acked 2312 Write 4 Done New Done true
5197 1314 Acked 2313 Read 7 Done New Done true
5198 1315 Acked 2314 Write 5 Done New Done true
5199 1316 Acked 2315 Flush 0 Done New Done true
5200 1317 Acked 2316 Write 5 Done New Done true
5201 1318 Acked 2317 Write 9 Done New Done true
5202 1319 Acked 2318 Write 9 Done New Done true
5203 1320 Acked 2319 Write 1 Done New Done true
5204 1321 Acked 2320 Write 2 Done New Done true
5205 1322 Acked 2321 Write 6 Done New Done true
5206 1323 Acked 2322 Write 10 Done New Done true
5207 1324 Acked 2323 Write 1 Done New Done true
5208 1325 Acked 2324 Flush 0 Done New Done true
5209 1326 Acked 2325 Write 5 Done New Done true
5210 1327 Acked 2326 Flush 0 Done New Done true
5211 1328 Acked 2327 Read 3 Done New Done true
5212 1329 Acked 2328 Read 9 Done New Done true
5213 1330 Acked 2329 Read 3 Done New Done true
5214 1331 Acked 2330 Read 4 Done New Done true
5215 1332 Acked 2331 Read 7 Done New Done true
5216 1333 Acked 2332 Write 2 Done New Done true
5217 1334 Acked 2333 Write 7 Done New Done true
5218 1335 Acked 2334 Write 7 Done New Done true
5219 1336 Acked 2335 Read 10 Done New Done true
5220 1337 Acked 2336 Read 2 Done New Done true
5221 1338 Acked 2337 Write 4 Done New Done true
5222 1339 Acked 2338 Flush 0 Done New Done true
5223 1340 Acked 2339 Write 9 Done New Done true
5224 1341 Acked 2340 Read 7 Done New Done true
5225 1342 Acked 2341 Read 2 Done New Done true
5226 1343 Acked 2342 Read 9 Done New Done true
5227 1344 Acked 2343 Read 10 Done New Done true
5228 1345 Acked 2344 Flush 0 Done New Done true
5229 1346 Acked 2345 Write 6 Done New Done true
5230 1347 Acked 2346 Read 9 Done New Done true
5231 1348 Acked 2347 Read 4 Done New Done true
5232 1349 Acked 2348 Flush 0 Done New Done true
5233 1350 Acked 2349 Read 7 Done New Done true
5234 1351 Acked 2350 Write 9 Done New Done true
5235 1352 Acked 2351 Write 5 Done New Done true
5236 1353 Acked 2352 Flush 0 Done New Done true
5237 1354 Acked 2353 Write 8 Done New Done true
5238 1355 Acked 2354 Write 8 Done New Done true
5239 1356 Acked 2355 Read 3 Done New Done true
5240 1357 Acked 2356 Read 1 Done New Done true
5241 1358 Acked 2357 Read 7 Done New Done true
5242 1359 Acked 2358 Read 7 Done New Done true
5243 1360 Acked 2359 Read 10 Done New Done true
5244 1361 Acked 2360 Flush 0 Done New Done true
5245 1362 Acked 2361 Read 7 Done New Done true
5246 1363 Acked 2362 Flush 0 Done New Done true
5247 1364 Acked 2363 Read 7 Done New Done true
5248 1365 Acked 2364 Write 5 Done New Done true
5249 1366 Acked 2365 Write 9 Done New Done true
5250 1367 Acked 2366 Flush 0 Done New Done true
5251 1368 Acked 2367 Write 5 Done New Done true
5252 1369 Acked 2368 Read 6 Done New Done true
5253 1370 Acked 2369 Read 7 Done New Done true
5254 1371 Acked 2370 Read 1 Done New Done true
5255 1372 Acked 2371 Read 4 Done New Done true
5256 1373 Acked 2372 Write 1 Done New Done true
5257 1374 Acked 2373 Flush 0 Done New Done true
5258 1375 Acked 2374 Read 1 Done New Done true
5259 1376 Acked 2375 Read 10 Done New Done true
5260 1377 Acked 2376 Read 3 Done New Done true
5261 1378 Acked 2377 Read 1 Done New Done true
5262 1379 Acked 2378 Read 8 Done New Done true
5263 1380 Acked 2379 Write 7 Done New Done true
5264 1381 Acked 2380 Read 1 Done New Done true
5265 1382 Acked 2381 Flush 0 Done New Done true
5266 1383 Acked 2382 Write 8 Done New Done true
5267 1384 Acked 2383 Read 4 Done New Done true
5268 1385 Acked 2384 Read 7 Done New Done true
5269 1386 Acked 2385 Flush 0 Done New Done true
5270 1387 Acked 2386 Read 1 Done New Done true
5271 1388 Acked 2387 Read 3 Done New Done true
5272 1389 Acked 2388 Write 9 Done New Done true
5273 1390 Acked 2389 Read 4 Done New Done true
5274 1391 Acked 2390 Read 9 Done New Done true
5275 1392 Acked 2391 Read 1 Done New Done true
5276 1393 Acked 2392 Read 2 Done New Done true
5277 1394 Acked 2393 Write 5 Done New Done true
5278 1395 Acked 2394 Write 4 Done New Done true
5279 1396 Acked 2395 Write 1 Done New Done true
5280 1397 Acked 2396 Read 9 Done New Done true
5281 1398 Acked 2397 Read 2 Done New Done true
5282 1399 Acked 2398 Read 5 Done New Done true
5283 1400 Acked 2399 Read 5 Done New Done true
5284 1401 Acked 2400 Read 2 Done New Done true
5285 1402 Acked 2401 Read 9 Done New Done true
5286 1403 Acked 2402 Read 7 Done New Done true
5287 1404 Acked 2403 Read 6 Done New Done true
5288 1405 Acked 2404 Read 2 Done New Done true
5289 1406 Acked 2405 Write 4 Done New Done true
5290 1407 Acked 2406 Read 9 Done New Done true
5291 1408 Acked 2407 Read 9 Done New Done true
5292 1409 Acked 2408 Write 3 Done New Done true
5293 1410 Acked 2409 Flush 0 Done New Done true
5294 1411 Acked 2410 Read 10 Done New Done true
5295 1412 Acked 2411 Read 6 Done New Done true
5296 1413 Acked 2412 Read 8 Done New Done true
5297 1414 Acked 2413 Read 1 Done New Done true
5298 1415 Acked 2414 Read 8 Done New Done true
5299 1416 Acked 2415 Write 8 Done New Done true
5300 1417 Acked 2416 Write 1 Done New Done true
5301 1418 Acked 2417 Write 2 Done New Done true
5302 1419 Acked 2418 Write 1 Done New Done true
5303 1420 Acked 2419 Write 9 Done New Done true
5304 1421 Acked 2420 Read 8 Done New Done true
5305 1422 Acked 2421 Read 10 Done New Done true
5306 1423 Acked 2422 Read 9 Done New Done true
5307 1424 Acked 2423 Write 9 Done New Done true
5308 1425 Acked 2424 Read 2 Done New Done true
5309 1426 Acked 2425 Write 1 Done New Done true
5310 1427 Acked 2426 Read 2 Done New Done true
5311 1428 Acked 2427 Write 6 Done New Done true
5312 1429 Acked 2428 Write 8 Done New Done true
5313 1430 Acked 2429 Write 7 Done New Done true
5314 1431 Acked 2430 Read 8 Done New Done true
5315 1432 Acked 2431 Read 5 Done New Done true
5316 1433 Acked 2432 Read 8 Done New Done true
5317 1434 Acked 2433 Read 1 Done New Done true
5318 1435 Acked 2434 Write 10 Done New Done true
5319 1436 Acked 2435 Read 3 Done New Done true
5320 1437 Acked 2436 Flush 0 Done New Done true
5321 1438 Acked 2437 Write 1 Done New Done true
5322 1439 Acked 2438 Read 8 Done New Done true
5323 1440 Acked 2439 Read 3 Done New Done true
5324 1441 Acked 2440 Read 3 Done New Done true
5325 1442 Acked 2441 Write 8 Done New Done true
5326 1443 Acked 2442 Flush 0 Done New Done true
5327 1444 Acked 2443 Read 9 Done New Done true
5328 1445 Acked 2444 Write 10 Done New Done true
5329 1446 Acked 2445 Write 9 Done New Done true
5330 1447 Acked 2446 Read 5 Done New Done true
5331 1448 Acked 2447 Write 2 Done New Done true
5332 1449 Acked 2448 Write 3 Done New Done true
5333 1450 Acked 2449 Read 1 Done New Done true
5334 1451 Acked 2450 Flush 0 Done New Done true
5335 1452 Acked 2451 Flush 0 Done New Done true
5336 1453 Acked 2452 Write 1 Done New Done true
5337 1454 Acked 2453 Read 4 Done New Done true
5338 1455 Acked 2454 Read 8 Done New Done true
5339 1456 Acked 2455 Read 9 Done New Done true
5340 1457 Acked 2456 Flush 0 Done New Done true
5341 1458 Acked 2457 Read 5 Done New Done true
5342 1459 Acked 2458 Read 8 Done New Done true
5343 1460 Acked 2459 Write 3 Done New Done true
5344 1461 Acked 2460 Write 6 Done New Done true
5345 1462 Acked 2461 Write 3 Done New Done true
5346 1463 Acked 2462 Read 2 Done New Done true
5347 1464 Acked 2463 Write 9 Done New Done true
5348 1465 Acked 2464 Read 2 Done New Done true
5349 1466 Acked 2465 Write 1 Done New Done true
5350 1467 Acked 2466 Read 3 Done New Done true
5351 1468 Acked 2467 Read 1 Done New Done true
5352 1469 Acked 2468 Write 7 Done New Done true
5353 1470 Acked 2469 Write 2 Done New Done true
5354 1471 Acked 2470 Read 1 Done New Done true
5355 1472 Acked 2471 Write 9 Done New Done true
5356 1473 Acked 2472 Flush 0 Done New Done true
5357 1474 Acked 2473 Flush 0 Done New Done true
5358 1475 Acked 2474 Read 7 Done New Done true
5359 1476 Acked 2475 Read 6 Done New Done true
5360 1477 Acked 2476 Read 8 Done New Done true
5361 1478 Acked 2477 Read 5 Done New Done true
5362 1479 Acked 2478 Flush 0 Done New Done true
5363 1480 Acked 2479 Read 2 Done New Done true
5364 1481 Acked 2480 Write 9 Done New Done true
5365 1482 Acked 2481 Write 4 Done New Done true
5366 1483 Acked 2482 Read 2 Done New Done true
5367 1484 Acked 2483 Write 8 Done New Done true
5368 1485 Acked 2484 Read 9 Done New Done true
5369 1486 Acked 2485 Write 6 Done New Done true
5370 1487 Acked 2486 Flush 0 Done New Done true
5371 1488 Acked 2487 Read 7 Done New Done true
5372 1489 Acked 2488 Read 5 Done New Done true
5373 1490 Acked 2489 Write 7 Done New Done true
5374 1491 Acked 2490 Read 4 Done New Done true
5375 1492 Acked 2491 Write 9 Done New Done true
5376 1493 Acked 2492 Read 9 Done New Done true
5377 1494 Acked 2493 Read 10 Done New Done true
5378 1495 Acked 2494 Read 3 Done New Done true
5379 1496 Acked 2495 Flush 0 Done New Done true
5380 1497 Acked 2496 Read 8 Done New Done true
5381 1498 Acked 2497 Read 6 Done New Done true
5382 1499 Acked 2498 Read 4 Done New Done true
5383 1500 Acked 2499 Write 9 Done New Done true
5384 1501 Acked 2500 Write 9 Done New Done true
5385 1502 Acked 2501 Read 1 Done New Done true
5386 1503 Acked 2502 Write 3 Done New Done true
5387 1504 Acked 2503 Read 5 Done New Done true
5388 1505 Acked 2504 Write 9 Done New Done true
5389 1506 Acked 2505 Flush 0 Done New Done true
5390 STATES DS:0 DS:1 DS:2 TOTAL
5391 New 0 301 0 301
5392 Sent 0 0 0 0
5393 Done 301 0 301 602
5394 Skipped 0 0 0 0
5395 Error 0 0 0 0
5396 Last Flush: 2505 2204 2505
5397 Downstairs last five completed: 2204 2203 2202 2201 2200
5398 Upstairs last five completed: 1506 1505 1504 1503 1502
5399 CLIENT: Up:0 ds:301 act:2
54002024-02-15T02:58:32.975ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
54012024-02-15T02:58:32.975ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
54022024-02-15T02:58:32.975ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
54032024-02-15T02:58:32.975ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
54042024-02-15T02:58:32.975ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
54052024-02-15T02:58:32.975ZINFOcrucible: [1] client re-new 301 jobs since flush 2204 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
54062024-02-15T02:58:32.975ZINFOcrucible: client task is exiting = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
54072024-02-15T02:58:33.976ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
54082024-02-15T02:58:33.976ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
54092024-02-15T02:58:33.976ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
54102024-02-15T02:58:33.976ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
54112024-02-15T02:58:33.976ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
54122024-02-15T02:58:33.976ZINFOcrucible: [1] client re-new 301 jobs since flush 2204 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
54132024-02-15T02:58:33.977ZINFOcrucible: client task is exiting = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
54142024-02-15T02:58:34.977ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
54152024-02-15T02:58:34.977ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
54162024-02-15T02:58:34.977ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
54172024-02-15T02:58:34.977ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
54182024-02-15T02:58:34.977ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
54192024-02-15T02:58:34.977ZINFOcrucible: [1] client re-new 301 jobs since flush 2204 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
54202024-02-15T02:58:34.978ZINFOcrucible: client task is exiting = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
54212024-02-15T02:58:35.978ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
54222024-02-15T02:58:35.978ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
54232024-02-15T02:58:35.979ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
54242024-02-15T02:58:35.979ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
54252024-02-15T02:58:35.979ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
54262024-02-15T02:58:35.979ZINFOcrucible: [1] client re-new 301 jobs since flush 2204 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
54272024-02-15T02:58:35.979ZINFOcrucible: client task is exiting = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
5428 ----------------------------------------------------------------
5429 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
5430 States: Active Offline Active
5431 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
5432 1206 Acked 2205 Read 2 Done New Done true
5433 1207 Acked 2206 Read 7 Done New Done true
5434 1208 Acked 2207 Flush 0 Done New Done true
5435 1209 Acked 2208 Read 3 Done New Done true
5436 1210 Acked 2209 Read 6 Done New Done true
5437 1211 Acked 2210 Write 9 Done New Done true
5438 1212 Acked 2211 Read 5 Done New Done true
5439 1213 Acked 2212 Read 3 Done New Done true
5440 1214 Acked 2213 Read 9 Done New Done true
5441 1215 Acked 2214 Read 9 Done New Done true
5442 1216 Acked 2215 Flush 0 Done New Done true
5443 1217 Acked 2216 Read 7 Done New Done true
5444 1218 Acked 2217 Flush 0 Done New Done true
5445 1219 Acked 2218 Read 8 Done New Done true
5446 1220 Acked 2219 Read 9 Done New Done true
5447 1221 Acked 2220 Write 7 Done New Done true
5448 1222 Acked 2221 Read 4 Done New Done true
5449 1223 Acked 2222 Flush 0 Done New Done true
5450 1224 Acked 2223 Flush 0 Done New Done true
5451 1225 Acked 2224 Write 4 Done New Done true
5452 1226 Acked 2225 Read 10 Done New Done true
5453 1227 Acked 2226 Read 8 Done New Done true
5454 1228 Acked 2227 Write 6 Done New Done true
5455 1229 Acked 2228 Read 6 Done New Done true
5456 1230 Acked 2229 Read 5 Done New Done true
5457 1231 Acked 2230 Read 10 Done New Done true
5458 1232 Acked 2231 Read 8 Done New Done true
5459 1233 Acked 2232 Write 10 Done New Done true
5460 1234 Acked 2233 Write 6 Done New Done true
5461 1235 Acked 2234 Read 6 Done New Done true
5462 1236 Acked 2235 Read 7 Done New Done true
5463 1237 Acked 2236 Flush 0 Done New Done true
5464 1238 Acked 2237 Read 3 Done New Done true
5465 1239 Acked 2238 Read 10 Done New Done true
5466 1240 Acked 2239 Read 1 Done New Done true
5467 1241 Acked 2240 Read 6 Done New Done true
5468 1242 Acked 2241 Write 4 Done New Done true
5469 1243 Acked 2242 Read 6 Done New Done true
5470 1244 Acked 2243 Write 5 Done New Done true
5471 1245 Acked 2244 Read 7 Done New Done true
5472 1246 Acked 2245 Read 5 Done New Done true
5473 1247 Acked 2246 Read 2 Done New Done true
5474 1248 Acked 2247 Write 1 Done New Done true
5475 1249 Acked 2248 Write 2 Done New Done true
5476 1250 Acked 2249 Write 10 Done New Done true
5477 1251 Acked 2250 Read 8 Done New Done true
5478 1252 Acked 2251 Write 9 Done New Done true
5479 1253 Acked 2252 Write 6 Done New Done true
5480 1254 Acked 2253 Read 2 Done New Done true
5481 1255 Acked 2254 Read 10 Done New Done true
5482 1256 Acked 2255 Write 1 Done New Done true
5483 1257 Acked 2256 Read 10 Done New Done true
5484 1258 Acked 2257 Flush 0 Done New Done true
5485 1259 Acked 2258 Read 9 Done New Done true
5486 1260 Acked 2259 Write 10 Done New Done true
5487 1261 Acked 2260 Write 10 Done New Done true
5488 1262 Acked 2261 Read 9 Done New Done true
5489 1263 Acked 2262 Write 1 Done New Done true
5490 1264 Acked 2263 Read 2 Done New Done true
5491 1265 Acked 2264 Write 4 Done New Done true
5492 1266 Acked 2265 Read 9 Done New Done true
5493 1267 Acked 2266 Write 3 Done New Done true
5494 1268 Acked 2267 Write 8 Done New Done true
5495 1269 Acked 2268 Write 8 Done New Done true
5496 1270 Acked 2269 Write 6 Done New Done true
5497 1271 Acked 2270 Read 1 Done New Done true
5498 1272 Acked 2271 Write 9 Done New Done true
5499 1273 Acked 2272 Flush 0 Done New Done true
5500 1274 Acked 2273 Read 7 Done New Done true
5501 1275 Acked 2274 Read 10 Done New Done true
5502 1276 Acked 2275 Read 5 Done New Done true
5503 1277 Acked 2276 Write 1 Done New Done true
5504 1278 Acked 2277 Read 1 Done New Done true
5505 1279 Acked 2278 Write 7 Done New Done true
5506 1280 Acked 2279 Write 9 Done New Done true
5507 1281 Acked 2280 Read 3 Done New Done true
5508 1282 Acked 2281 Read 10 Done New Done true
5509 1283 Acked 2282 Write 8 Done New Done true
5510 1284 Acked 2283 Write 4 Done New Done true
5511 1285 Acked 2284 Read 6 Done New Done true
5512 1286 Acked 2285 Flush 0 Done New Done true
5513 1287 Acked 2286 Read 9 Done New Done true
5514 1288 Acked 2287 Read 5 Done New Done true
5515 1289 Acked 2288 Read 7 Done New Done true
5516 1290 Acked 2289 Write 10 Done New Done true
5517 1291 Acked 2290 Read 2 Done New Done true
5518 1292 Acked 2291 Read 4 Done New Done true
5519 1293 Acked 2292 Flush 0 Done New Done true
5520 1294 Acked 2293 Read 10 Done New Done true
5521 1295 Acked 2294 Write 5 Done New Done true
5522 1296 Acked 2295 Write 8 Done New Done true
5523 1297 Acked 2296 Write 3 Done New Done true
5524 1298 Acked 2297 Flush 0 Done New Done true
5525 1299 Acked 2298 Read 9 Done New Done true
5526 1300 Acked 2299 Write 1 Done New Done true
5527 1301 Acked 2300 Flush 0 Done New Done true
5528 1302 Acked 2301 Read 3 Done New Done true
5529 1303 Acked 2302 Read 5 Done New Done true
5530 1304 Acked 2303 Write 4 Done New Done true
5531 1305 Acked 2304 Read 1 Done New Done true
5532 1306 Acked 2305 Read 9 Done New Done true
5533 1307 Acked 2306 Read 7 Done New Done true
5534 1308 Acked 2307 Write 9 Done New Done true
5535 1309 Acked 2308 Read 4 Done New Done true
5536 1310 Acked 2309 Write 8 Done New Done true
5537 1311 Acked 2310 Flush 0 Done New Done true
5538 1312 Acked 2311 Read 4 Done New Done true
5539 1313 Acked 2312 Write 4 Done New Done true
5540 1314 Acked 2313 Read 7 Done New Done true
5541 1315 Acked 2314 Write 5 Done New Done true
5542 1316 Acked 2315 Flush 0 Done New Done true
5543 1317 Acked 2316 Write 5 Done New Done true
5544 1318 Acked 2317 Write 9 Done New Done true
5545 1319 Acked 2318 Write 9 Done New Done true
5546 1320 Acked 2319 Write 1 Done New Done true
5547 1321 Acked 2320 Write 2 Done New Done true
5548 1322 Acked 2321 Write 6 Done New Done true
5549 1323 Acked 2322 Write 10 Done New Done true
5550 1324 Acked 2323 Write 1 Done New Done true
5551 1325 Acked 2324 Flush 0 Done New Done true
5552 1326 Acked 2325 Write 5 Done New Done true
5553 1327 Acked 2326 Flush 0 Done New Done true
5554 1328 Acked 2327 Read 3 Done New Done true
5555 1329 Acked 2328 Read 9 Done New Done true
5556 1330 Acked 2329 Read 3 Done New Done true
5557 1331 Acked 2330 Read 4 Done New Done true
5558 1332 Acked 2331 Read 7 Done New Done true
5559 1333 Acked 2332 Write 2 Done New Done true
5560 1334 Acked 2333 Write 7 Done New Done true
5561 1335 Acked 2334 Write 7 Done New Done true
5562 1336 Acked 2335 Read 10 Done New Done true
5563 1337 Acked 2336 Read 2 Done New Done true
5564 1338 Acked 2337 Write 4 Done New Done true
5565 1339 Acked 2338 Flush 0 Done New Done true
5566 1340 Acked 2339 Write 9 Done New Done true
5567 1341 Acked 2340 Read 7 Done New Done true
5568 1342 Acked 2341 Read 2 Done New Done true
5569 1343 Acked 2342 Read 9 Done New Done true
5570 1344 Acked 2343 Read 10 Done New Done true
5571 1345 Acked 2344 Flush 0 Done New Done true
5572 1346 Acked 2345 Write 6 Done New Done true
5573 1347 Acked 2346 Read 9 Done New Done true
5574 1348 Acked 2347 Read 4 Done New Done true
5575 1349 Acked 2348 Flush 0 Done New Done true
5576 1350 Acked 2349 Read 7 Done New Done true
5577 1351 Acked 2350 Write 9 Done New Done true
5578 1352 Acked 2351 Write 5 Done New Done true
5579 1353 Acked 2352 Flush 0 Done New Done true
5580 1354 Acked 2353 Write 8 Done New Done true
5581 1355 Acked 2354 Write 8 Done New Done true
5582 1356 Acked 2355 Read 3 Done New Done true
5583 1357 Acked 2356 Read 1 Done New Done true
5584 1358 Acked 2357 Read 7 Done New Done true
5585 1359 Acked 2358 Read 7 Done New Done true
5586 1360 Acked 2359 Read 10 Done New Done true
5587 1361 Acked 2360 Flush 0 Done New Done true
5588 1362 Acked 2361 Read 7 Done New Done true
5589 1363 Acked 2362 Flush 0 Done New Done true
5590 1364 Acked 2363 Read 7 Done New Done true
5591 1365 Acked 2364 Write 5 Done New Done true
5592 1366 Acked 2365 Write 9 Done New Done true
5593 1367 Acked 2366 Flush 0 Done New Done true
5594 1368 Acked 2367 Write 5 Done New Done true
5595 1369 Acked 2368 Read 6 Done New Done true
5596 1370 Acked 2369 Read 7 Done New Done true
5597 1371 Acked 2370 Read 1 Done New Done true
5598 1372 Acked 2371 Read 4 Done New Done true
5599 1373 Acked 2372 Write 1 Done New Done true
5600 1374 Acked 2373 Flush 0 Done New Done true
5601 1375 Acked 2374 Read 1 Done New Done true
5602 1376 Acked 2375 Read 10 Done New Done true
5603 1377 Acked 2376 Read 3 Done New Done true
5604 1378 Acked 2377 Read 1 Done New Done true
5605 1379 Acked 2378 Read 8 Done New Done true
5606 1380 Acked 2379 Write 7 Done New Done true
5607 1381 Acked 2380 Read 1 Done New Done true
5608 1382 Acked 2381 Flush 0 Done New Done true
5609 1383 Acked 2382 Write 8 Done New Done true
5610 1384 Acked 2383 Read 4 Done New Done true
5611 1385 Acked 2384 Read 7 Done New Done true
5612 1386 Acked 2385 Flush 0 Done New Done true
5613 1387 Acked 2386 Read 1 Done New Done true
5614 1388 Acked 2387 Read 3 Done New Done true
5615 1389 Acked 2388 Write 9 Done New Done true
5616 1390 Acked 2389 Read 4 Done New Done true
5617 1391 Acked 2390 Read 9 Done New Done true
5618 1392 Acked 2391 Read 1 Done New Done true
5619 1393 Acked 2392 Read 2 Done New Done true
5620 1394 Acked 2393 Write 5 Done New Done true
5621 1395 Acked 2394 Write 4 Done New Done true
5622 1396 Acked 2395 Write 1 Done New Done true
5623 1397 Acked 2396 Read 9 Done New Done true
5624 1398 Acked 2397 Read 2 Done New Done true
5625 1399 Acked 2398 Read 5 Done New Done true
5626 1400 Acked 2399 Read 5 Done New Done true
5627 1401 Acked 2400 Read 2 Done New Done true
5628 1402 Acked 2401 Read 9 Done New Done true
5629 1403 Acked 2402 Read 7 Done New Done true
5630 1404 Acked 2403 Read 6 Done New Done true
5631 1405 Acked 2404 Read 2 Done New Done true
5632 1406 Acked 2405 Write 4 Done New Done true
5633 1407 Acked 2406 Read 9 Done New Done true
5634 1408 Acked 2407 Read 9 Done New Done true
5635 1409 Acked 2408 Write 3 Done New Done true
5636 1410 Acked 2409 Flush 0 Done New Done true
5637 1411 Acked 2410 Read 10 Done New Done true
5638 1412 Acked 2411 Read 6 Done New Done true
5639 1413 Acked 2412 Read 8 Done New Done true
5640 1414 Acked 2413 Read 1 Done New Done true
5641 1415 Acked 2414 Read 8 Done New Done true
5642 1416 Acked 2415 Write 8 Done New Done true
5643 1417 Acked 2416 Write 1 Done New Done true
5644 1418 Acked 2417 Write 2 Done New Done true
5645 1419 Acked 2418 Write 1 Done New Done true
5646 1420 Acked 2419 Write 9 Done New Done true
5647 1421 Acked 2420 Read 8 Done New Done true
5648 1422 Acked 2421 Read 10 Done New Done true
5649 1423 Acked 2422 Read 9 Done New Done true
5650 1424 Acked 2423 Write 9 Done New Done true
5651 1425 Acked 2424 Read 2 Done New Done true
5652 1426 Acked 2425 Write 1 Done New Done true
5653 1427 Acked 2426 Read 2 Done New Done true
5654 1428 Acked 2427 Write 6 Done New Done true
5655 1429 Acked 2428 Write 8 Done New Done true
5656 1430 Acked 2429 Write 7 Done New Done true
5657 1431 Acked 2430 Read 8 Done New Done true
5658 1432 Acked 2431 Read 5 Done New Done true
5659 1433 Acked 2432 Read 8 Done New Done true
5660 1434 Acked 2433 Read 1 Done New Done true
5661 1435 Acked 2434 Write 10 Done New Done true
5662 1436 Acked 2435 Read 3 Done New Done true
5663 1437 Acked 2436 Flush 0 Done New Done true
5664 1438 Acked 2437 Write 1 Done New Done true
5665 1439 Acked 2438 Read 8 Done New Done true
5666 1440 Acked 2439 Read 3 Done New Done true
5667 1441 Acked 2440 Read 3 Done New Done true
5668 1442 Acked 2441 Write 8 Done New Done true
5669 1443 Acked 2442 Flush 0 Done New Done true
5670 1444 Acked 2443 Read 9 Done New Done true
5671 1445 Acked 2444 Write 10 Done New Done true
5672 1446 Acked 2445 Write 9 Done New Done true
5673 1447 Acked 2446 Read 5 Done New Done true
5674 1448 Acked 2447 Write 2 Done New Done true
5675 1449 Acked 2448 Write 3 Done New Done true
5676 1450 Acked 2449 Read 1 Done New Done true
5677 1451 Acked 2450 Flush 0 Done New Done true
5678 1452 Acked 2451 Flush 0 Done New Done true
5679 1453 Acked 2452 Write 1 Done New Done true
5680 1454 Acked 2453 Read 4 Done New Done true
5681 1455 Acked 2454 Read 8 Done New Done true
5682 1456 Acked 2455 Read 9 Done New Done true
5683 1457 Acked 2456 Flush 0 Done New Done true
5684 1458 Acked 2457 Read 5 Done New Done true
5685 1459 Acked 2458 Read 8 Done New Done true
5686 1460 Acked 2459 Write 3 Done New Done true
5687 1461 Acked 2460 Write 6 Done New Done true
5688 1462 Acked 2461 Write 3 Done New Done true
5689 1463 Acked 2462 Read 2 Done New Done true
5690 1464 Acked 2463 Write 9 Done New Done true
5691 1465 Acked 2464 Read 2 Done New Done true
5692 1466 Acked 2465 Write 1 Done New Done true
5693 1467 Acked 2466 Read 3 Done New Done true
5694 1468 Acked 2467 Read 1 Done New Done true
5695 1469 Acked 2468 Write 7 Done New Done true
5696 1470 Acked 2469 Write 2 Done New Done true
5697 1471 Acked 2470 Read 1 Done New Done true
5698 1472 Acked 2471 Write 9 Done New Done true
5699 1473 Acked 2472 Flush 0 Done New Done true
5700 1474 Acked 2473 Flush 0 Done New Done true
5701 1475 Acked 2474 Read 7 Done New Done true
5702 1476 Acked 2475 Read 6 Done New Done true
5703 1477 Acked 2476 Read 8 Done New Done true
5704 1478 Acked 2477 Read 5 Done New Done true
5705 1479 Acked 2478 Flush 0 Done New Done true
5706 1480 Acked 2479 Read 2 Done New Done true
5707 1481 Acked 2480 Write 9 Done New Done true
5708 1482 Acked 2481 Write 4 Done New Done true
5709 1483 Acked 2482 Read 2 Done New Done true
5710 1484 Acked 2483 Write 8 Done New Done true
5711 1485 Acked 2484 Read 9 Done New Done true
5712 1486 Acked 2485 Write 6 Done New Done true
5713 1487 Acked 2486 Flush 0 Done New Done true
5714 1488 Acked 2487 Read 7 Done New Done true
5715 1489 Acked 2488 Read 5 Done New Done true
5716 1490 Acked 2489 Write 7 Done New Done true
5717 1491 Acked 2490 Read 4 Done New Done true
5718 1492 Acked 2491 Write 9 Done New Done true
5719 1493 Acked 2492 Read 9 Done New Done true
5720 1494 Acked 2493 Read 10 Done New Done true
5721 1495 Acked 2494 Read 3 Done New Done true
5722 1496 Acked 2495 Flush 0 Done New Done true
5723 1497 Acked 2496 Read 8 Done New Done true
5724 1498 Acked 2497 Read 6 Done New Done true
5725 1499 Acked 2498 Read 4 Done New Done true
5726 1500 Acked 2499 Write 9 Done New Done true
5727 1501 Acked 2500 Write 9 Done New Done true
5728 1502 Acked 2501 Read 1 Done New Done true
5729 1503 Acked 2502 Write 3 Done New Done true
5730 1504 Acked 2503 Read 5 Done New Done true
5731 1505 Acked 2504 Write 9 Done New Done true
5732 1506 Acked 2505 Flush 0 Done New Done true
5733 STATES DS:0 DS:1 DS:2 TOTAL
5734 New 0 301 0 301
5735 Sent 0 0 0 0
5736 Done 301 0 301 602
5737 Skipped 0 0 0 0
5738 Error 0 0 0 0
5739 Last Flush: 2505 2204 2505
5740 Downstairs last five completed: 2204 2203 2202 2201 2200
5741 Upstairs last five completed: 1506 1505 1504 1503 1502
5742 CLIENT: Up:0 ds:301 act:2
57432024-02-15T02:58:36.979ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
57442024-02-15T02:58:36.979ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
57452024-02-15T02:58:36.980ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
57462024-02-15T02:58:36.980ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
57472024-02-15T02:58:36.980ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
57482024-02-15T02:58:36.980ZINFOcrucible: [1] client re-new 301 jobs since flush 2204 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
57492024-02-15T02:58:36.980ZINFOcrucible: client task is exiting = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
57502024-02-15T02:58:37.981ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
57512024-02-15T02:58:37.981ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
57522024-02-15T02:58:37.981ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
57532024-02-15T02:58:37.981ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
57542024-02-15T02:58:37.981ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
57552024-02-15T02:58:37.981ZINFOcrucible: [1] client re-new 301 jobs since flush 2204 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
57562024-02-15T02:58:37.981ZINFOcrucible: client task is exiting = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
57572024-02-15T02:58:38.982ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
57582024-02-15T02:58:38.982ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
57592024-02-15T02:58:38.982ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
57602024-02-15T02:58:38.982ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
57612024-02-15T02:58:38.982ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
57622024-02-15T02:58:38.982ZINFOcrucible: [1] client re-new 301 jobs since flush 2204 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
57632024-02-15T02:58:38.982ZINFOcrucible: client task is exiting = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
57642024-02-15T02:58:39.983ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
57652024-02-15T02:58:39.983ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
57662024-02-15T02:58:39.984ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
57672024-02-15T02:58:39.984ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
57682024-02-15T02:58:39.984ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
57692024-02-15T02:58:39.984ZINFOcrucible: [1] client re-new 301 jobs since flush 2204 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
57702024-02-15T02:58:39.984ZINFOcrucible: client task is exiting = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
5771 ----------------------------------------------------------------
5772 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
5773 States: Active Offline Active
5774 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
5775 1206 Acked 2205 Read 2 Done New Done true
5776 1207 Acked 2206 Read 7 Done New Done true
5777 1208 Acked 2207 Flush 0 Done New Done true
5778 1209 Acked 2208 Read 3 Done New Done true
5779 1210 Acked 2209 Read 6 Done New Done true
5780 1211 Acked 2210 Write 9 Done New Done true
5781 1212 Acked 2211 Read 5 Done New Done true
5782 1213 Acked 2212 Read 3 Done New Done true
5783 1214 Acked 2213 Read 9 Done New Done true
5784 1215 Acked 2214 Read 9 Done New Done true
5785 1216 Acked 2215 Flush 0 Done New Done true
5786 1217 Acked 2216 Read 7 Done New Done true
5787 1218 Acked 2217 Flush 0 Done New Done true
5788 1219 Acked 2218 Read 8 Done New Done true
5789 1220 Acked 2219 Read 9 Done New Done true
5790 1221 Acked 2220 Write 7 Done New Done true
5791 1222 Acked 2221 Read 4 Done New Done true
5792 1223 Acked 2222 Flush 0 Done New Done true
5793 1224 Acked 2223 Flush 0 Done New Done true
5794 1225 Acked 2224 Write 4 Done New Done true
5795 1226 Acked 2225 Read 10 Done New Done true
5796 1227 Acked 2226 Read 8 Done New Done true
5797 1228 Acked 2227 Write 6 Done New Done true
5798 1229 Acked 2228 Read 6 Done New Done true
5799 1230 Acked 2229 Read 5 Done New Done true
5800 1231 Acked 2230 Read 10 Done New Done true
5801 1232 Acked 2231 Read 8 Done New Done true
5802 1233 Acked 2232 Write 10 Done New Done true
5803 1234 Acked 2233 Write 6 Done New Done true
5804 1235 Acked 2234 Read 6 Done New Done true
5805 1236 Acked 2235 Read 7 Done New Done true
5806 1237 Acked 2236 Flush 0 Done New Done true
5807 1238 Acked 2237 Read 3 Done New Done true
5808 1239 Acked 2238 Read 10 Done New Done true
5809 1240 Acked 2239 Read 1 Done New Done true
5810 1241 Acked 2240 Read 6 Done New Done true
5811 1242 Acked 2241 Write 4 Done New Done true
5812 1243 Acked 2242 Read 6 Done New Done true
5813 1244 Acked 2243 Write 5 Done New Done true
5814 1245 Acked 2244 Read 7 Done New Done true
5815 1246 Acked 2245 Read 5 Done New Done true
5816 1247 Acked 2246 Read 2 Done New Done true
5817 1248 Acked 2247 Write 1 Done New Done true
5818 1249 Acked 2248 Write 2 Done New Done true
5819 1250 Acked 2249 Write 10 Done New Done true
5820 1251 Acked 2250 Read 8 Done New Done true
5821 1252 Acked 2251 Write 9 Done New Done true
5822 1253 Acked 2252 Write 6 Done New Done true
5823 1254 Acked 2253 Read 2 Done New Done true
5824 1255 Acked 2254 Read 10 Done New Done true
5825 1256 Acked 2255 Write 1 Done New Done true
5826 1257 Acked 2256 Read 10 Done New Done true
5827 1258 Acked 2257 Flush 0 Done New Done true
5828 1259 Acked 2258 Read 9 Done New Done true
5829 1260 Acked 2259 Write 10 Done New Done true
5830 1261 Acked 2260 Write 10 Done New Done true
5831 1262 Acked 2261 Read 9 Done New Done true
5832 1263 Acked 2262 Write 1 Done New Done true
5833 1264 Acked 2263 Read 2 Done New Done true
5834 1265 Acked 2264 Write 4 Done New Done true
5835 1266 Acked 2265 Read 9 Done New Done true
5836 1267 Acked 2266 Write 3 Done New Done true
5837 1268 Acked 2267 Write 8 Done New Done true
5838 1269 Acked 2268 Write 8 Done New Done true
5839 1270 Acked 2269 Write 6 Done New Done true
5840 1271 Acked 2270 Read 1 Done New Done true
5841 1272 Acked 2271 Write 9 Done New Done true
5842 1273 Acked 2272 Flush 0 Done New Done true
5843 1274 Acked 2273 Read 7 Done New Done true
5844 1275 Acked 2274 Read 10 Done New Done true
5845 1276 Acked 2275 Read 5 Done New Done true
5846 1277 Acked 2276 Write 1 Done New Done true
5847 1278 Acked 2277 Read 1 Done New Done true
5848 1279 Acked 2278 Write 7 Done New Done true
5849 1280 Acked 2279 Write 9 Done New Done true
5850 1281 Acked 2280 Read 3 Done New Done true
5851 1282 Acked 2281 Read 10 Done New Done true
5852 1283 Acked 2282 Write 8 Done New Done true
5853 1284 Acked 2283 Write 4 Done New Done true
5854 1285 Acked 2284 Read 6 Done New Done true
5855 1286 Acked 2285 Flush 0 Done New Done true
5856 1287 Acked 2286 Read 9 Done New Done true
5857 1288 Acked 2287 Read 5 Done New Done true
5858 1289 Acked 2288 Read 7 Done New Done true
5859 1290 Acked 2289 Write 10 Done New Done true
5860 1291 Acked 2290 Read 2 Done New Done true
5861 1292 Acked 2291 Read 4 Done New Done true
5862 1293 Acked 2292 Flush 0 Done New Done true
5863 1294 Acked 2293 Read 10 Done New Done true
5864 1295 Acked 2294 Write 5 Done New Done true
5865 1296 Acked 2295 Write 8 Done New Done true
5866 1297 Acked 2296 Write 3 Done New Done true
5867 1298 Acked 2297 Flush 0 Done New Done true
5868 1299 Acked 2298 Read 9 Done New Done true
5869 1300 Acked 2299 Write 1 Done New Done true
5870 1301 Acked 2300 Flush 0 Done New Done true
5871 1302 Acked 2301 Read 3 Done New Done true
5872 1303 Acked 2302 Read 5 Done New Done true
5873 1304 Acked 2303 Write 4 Done New Done true
5874 1305 Acked 2304 Read 1 Done New Done true
5875 1306 Acked 2305 Read 9 Done New Done true
5876 1307 Acked 2306 Read 7 Done New Done true
5877 1308 Acked 2307 Write 9 Done New Done true
5878 1309 Acked 2308 Read 4 Done New Done true
5879 1310 Acked 2309 Write 8 Done New Done true
5880 1311 Acked 2310 Flush 0 Done New Done true
5881 1312 Acked 2311 Read 4 Done New Done true
5882 1313 Acked 2312 Write 4 Done New Done true
5883 1314 Acked 2313 Read 7 Done New Done true
5884 1315 Acked 2314 Write 5 Done New Done true
5885 1316 Acked 2315 Flush 0 Done New Done true
5886 1317 Acked 2316 Write 5 Done New Done true
5887 1318 Acked 2317 Write 9 Done New Done true
5888 1319 Acked 2318 Write 9 Done New Done true
5889 1320 Acked 2319 Write 1 Done New Done true
5890 1321 Acked 2320 Write 2 Done New Done true
5891 1322 Acked 2321 Write 6 Done New Done true
5892 1323 Acked 2322 Write 10 Done New Done true
5893 1324 Acked 2323 Write 1 Done New Done true
5894 1325 Acked 2324 Flush 0 Done New Done true
5895 1326 Acked 2325 Write 5 Done New Done true
5896 1327 Acked 2326 Flush 0 Done New Done true
5897 1328 Acked 2327 Read 3 Done New Done true
5898 1329 Acked 2328 Read 9 Done New Done true
5899 1330 Acked 2329 Read 3 Done New Done true
5900 1331 Acked 2330 Read 4 Done New Done true
5901 1332 Acked 2331 Read 7 Done New Done true
5902 1333 Acked 2332 Write 2 Done New Done true
5903 1334 Acked 2333 Write 7 Done New Done true
5904 1335 Acked 2334 Write 7 Done New Done true
5905 1336 Acked 2335 Read 10 Done New Done true
5906 1337 Acked 2336 Read 2 Done New Done true
5907 1338 Acked 2337 Write 4 Done New Done true
5908 1339 Acked 2338 Flush 0 Done New Done true
5909 1340 Acked 2339 Write 9 Done New Done true
5910 1341 Acked 2340 Read 7 Done New Done true
5911 1342 Acked 2341 Read 2 Done New Done true
5912 1343 Acked 2342 Read 9 Done New Done true
5913 1344 Acked 2343 Read 10 Done New Done true
5914 1345 Acked 2344 Flush 0 Done New Done true
5915 1346 Acked 2345 Write 6 Done New Done true
5916 1347 Acked 2346 Read 9 Done New Done true
5917 1348 Acked 2347 Read 4 Done New Done true
5918 1349 Acked 2348 Flush 0 Done New Done true
5919 1350 Acked 2349 Read 7 Done New Done true
5920 1351 Acked 2350 Write 9 Done New Done true
5921 1352 Acked 2351 Write 5 Done New Done true
5922 1353 Acked 2352 Flush 0 Done New Done true
5923 1354 Acked 2353 Write 8 Done New Done true
5924 1355 Acked 2354 Write 8 Done New Done true
5925 1356 Acked 2355 Read 3 Done New Done true
5926 1357 Acked 2356 Read 1 Done New Done true
5927 1358 Acked 2357 Read 7 Done New Done true
5928 1359 Acked 2358 Read 7 Done New Done true
5929 1360 Acked 2359 Read 10 Done New Done true
5930 1361 Acked 2360 Flush 0 Done New Done true
5931 1362 Acked 2361 Read 7 Done New Done true
5932 1363 Acked 2362 Flush 0 Done New Done true
5933 1364 Acked 2363 Read 7 Done New Done true
5934 1365 Acked 2364 Write 5 Done New Done true
5935 1366 Acked 2365 Write 9 Done New Done true
5936 1367 Acked 2366 Flush 0 Done New Done true
5937 1368 Acked 2367 Write 5 Done New Done true
5938 1369 Acked 2368 Read 6 Done New Done true
5939 1370 Acked 2369 Read 7 Done New Done true
5940 1371 Acked 2370 Read 1 Done New Done true
5941 1372 Acked 2371 Read 4 Done New Done true
5942 1373 Acked 2372 Write 1 Done New Done true
5943 1374 Acked 2373 Flush 0 Done New Done true
5944 1375 Acked 2374 Read 1 Done New Done true
5945 1376 Acked 2375 Read 10 Done New Done true
5946 1377 Acked 2376 Read 3 Done New Done true
5947 1378 Acked 2377 Read 1 Done New Done true
5948 1379 Acked 2378 Read 8 Done New Done true
5949 1380 Acked 2379 Write 7 Done New Done true
5950 1381 Acked 2380 Read 1 Done New Done true
5951 1382 Acked 2381 Flush 0 Done New Done true
5952 1383 Acked 2382 Write 8 Done New Done true
5953 1384 Acked 2383 Read 4 Done New Done true
5954 1385 Acked 2384 Read 7 Done New Done true
5955 1386 Acked 2385 Flush 0 Done New Done true
5956 1387 Acked 2386 Read 1 Done New Done true
5957 1388 Acked 2387 Read 3 Done New Done true
5958 1389 Acked 2388 Write 9 Done New Done true
5959 1390 Acked 2389 Read 4 Done New Done true
5960 1391 Acked 2390 Read 9 Done New Done true
5961 1392 Acked 2391 Read 1 Done New Done true
5962 1393 Acked 2392 Read 2 Done New Done true
5963 1394 Acked 2393 Write 5 Done New Done true
5964 1395 Acked 2394 Write 4 Done New Done true
5965 1396 Acked 2395 Write 1 Done New Done true
5966 1397 Acked 2396 Read 9 Done New Done true
5967 1398 Acked 2397 Read 2 Done New Done true
5968 1399 Acked 2398 Read 5 Done New Done true
5969 1400 Acked 2399 Read 5 Done New Done true
5970 1401 Acked 2400 Read 2 Done New Done true
5971 1402 Acked 2401 Read 9 Done New Done true
5972 1403 Acked 2402 Read 7 Done New Done true
5973 1404 Acked 2403 Read 6 Done New Done true
5974 1405 Acked 2404 Read 2 Done New Done true
5975 1406 Acked 2405 Write 4 Done New Done true
5976 1407 Acked 2406 Read 9 Done New Done true
5977 1408 Acked 2407 Read 9 Done New Done true
5978 1409 Acked 2408 Write 3 Done New Done true
5979 1410 Acked 2409 Flush 0 Done New Done true
5980 1411 Acked 2410 Read 10 Done New Done true
5981 1412 Acked 2411 Read 6 Done New Done true
5982 1413 Acked 2412 Read 8 Done New Done true
5983 1414 Acked 2413 Read 1 Done New Done true
5984 1415 Acked 2414 Read 8 Done New Done true
5985 1416 Acked 2415 Write 8 Done New Done true
5986 1417 Acked 2416 Write 1 Done New Done true
5987 1418 Acked 2417 Write 2 Done New Done true
5988 1419 Acked 2418 Write 1 Done New Done true
5989 1420 Acked 2419 Write 9 Done New Done true
5990 1421 Acked 2420 Read 8 Done New Done true
5991 1422 Acked 2421 Read 10 Done New Done true
5992 1423 Acked 2422 Read 9 Done New Done true
5993 1424 Acked 2423 Write 9 Done New Done true
5994 1425 Acked 2424 Read 2 Done New Done true
5995 1426 Acked 2425 Write 1 Done New Done true
5996 1427 Acked 2426 Read 2 Done New Done true
5997 1428 Acked 2427 Write 6 Done New Done true
5998 1429 Acked 2428 Write 8 Done New Done true
5999 1430 Acked 2429 Write 7 Done New Done true
6000 1431 Acked 2430 Read 8 Done New Done true
6001 1432 Acked 2431 Read 5 Done New Done true
6002 1433 Acked 2432 Read 8 Done New Done true
6003 1434 Acked 2433 Read 1 Done New Done true
6004 1435 Acked 2434 Write 10 Done New Done true
6005 1436 Acked 2435 Read 3 Done New Done true
6006 1437 Acked 2436 Flush 0 Done New Done true
6007 1438 Acked 2437 Write 1 Done New Done true
6008 1439 Acked 2438 Read 8 Done New Done true
6009 1440 Acked 2439 Read 3 Done New Done true
6010 1441 Acked 2440 Read 3 Done New Done true
6011 1442 Acked 2441 Write 8 Done New Done true
6012 1443 Acked 2442 Flush 0 Done New Done true
6013 1444 Acked 2443 Read 9 Done New Done true
6014 1445 Acked 2444 Write 10 Done New Done true
6015 1446 Acked 2445 Write 9 Done New Done true
6016 1447 Acked 2446 Read 5 Done New Done true
6017 1448 Acked 2447 Write 2 Done New Done true
6018 1449 Acked 2448 Write 3 Done New Done true
6019 1450 Acked 2449 Read 1 Done New Done true
6020 1451 Acked 2450 Flush 0 Done New Done true
6021 1452 Acked 2451 Flush 0 Done New Done true
6022 1453 Acked 2452 Write 1 Done New Done true
6023 1454 Acked 2453 Read 4 Done New Done true
6024 1455 Acked 2454 Read 8 Done New Done true
6025 1456 Acked 2455 Read 9 Done New Done true
6026 1457 Acked 2456 Flush 0 Done New Done true
6027 1458 Acked 2457 Read 5 Done New Done true
6028 1459 Acked 2458 Read 8 Done New Done true
6029 1460 Acked 2459 Write 3 Done New Done true
6030 1461 Acked 2460 Write 6 Done New Done true
6031 1462 Acked 2461 Write 3 Done New Done true
6032 1463 Acked 2462 Read 2 Done New Done true
6033 1464 Acked 2463 Write 9 Done New Done true
6034 1465 Acked 2464 Read 2 Done New Done true
6035 1466 Acked 2465 Write 1 Done New Done true
6036 1467 Acked 2466 Read 3 Done New Done true
6037 1468 Acked 2467 Read 1 Done New Done true
6038 1469 Acked 2468 Write 7 Done New Done true
6039 1470 Acked 2469 Write 2 Done New Done true
6040 1471 Acked 2470 Read 1 Done New Done true
6041 1472 Acked 2471 Write 9 Done New Done true
6042 1473 Acked 2472 Flush 0 Done New Done true
6043 1474 Acked 2473 Flush 0 Done New Done true
6044 1475 Acked 2474 Read 7 Done New Done true
6045 1476 Acked 2475 Read 6 Done New Done true
6046 1477 Acked 2476 Read 8 Done New Done true
6047 1478 Acked 2477 Read 5 Done New Done true
6048 1479 Acked 2478 Flush 0 Done New Done true
6049 1480 Acked 2479 Read 2 Done New Done true
6050 1481 Acked 2480 Write 9 Done New Done true
6051 1482 Acked 2481 Write 4 Done New Done true
6052 1483 Acked 2482 Read 2 Done New Done true
6053 1484 Acked 2483 Write 8 Done New Done true
6054 1485 Acked 2484 Read 9 Done New Done true
6055 1486 Acked 2485 Write 6 Done New Done true
6056 1487 Acked 2486 Flush 0 Done New Done true
6057 1488 Acked 2487 Read 7 Done New Done true
6058 1489 Acked 2488 Read 5 Done New Done true
6059 1490 Acked 2489 Write 7 Done New Done true
6060 1491 Acked 2490 Read 4 Done New Done true
6061 1492 Acked 2491 Write 9 Done New Done true
6062 1493 Acked 2492 Read 9 Done New Done true
6063 1494 Acked 2493 Read 10 Done New Done true
6064 1495 Acked 2494 Read 3 Done New Done true
6065 1496 Acked 2495 Flush 0 Done New Done true
6066 1497 Acked 2496 Read 8 Done New Done true
6067 1498 Acked 2497 Read 6 Done New Done true
6068 1499 Acked 2498 Read 4 Done New Done true
6069 1500 Acked 2499 Write 9 Done New Done true
6070 1501 Acked 2500 Write 9 Done New Done true
6071 1502 Acked 2501 Read 1 Done New Done true
6072 1503 Acked 2502 Write 3 Done New Done true
6073 1504 Acked 2503 Read 5 Done New Done true
6074 1505 Acked 2504 Write 9 Done New Done true
6075 1506 Acked 2505 Flush 0 Done New Done true
6076 STATES DS:0 DS:1 DS:2 TOTAL
6077 New 0 301 0 301
6078 Sent 0 0 0 0
6079 Done 301 0 301 602
6080 Skipped 0 0 0 0
6081 Error 0 0 0 0
6082 Last Flush: 2505 2204 2505
6083 Downstairs last five completed: 2204 2203 2202 2201 2200
6084 Upstairs last five completed: 1506 1505 1504 1503 1502
6085 CLIENT: Up:0 ds:301 act:2
60862024-02-15T02:58:40.984ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
60872024-02-15T02:58:40.984ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
60882024-02-15T02:58:40.984ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
60892024-02-15T02:58:40.984ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
60902024-02-15T02:58:40.984ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
60912024-02-15T02:58:40.984ZINFOcrucible: [1] client re-new 301 jobs since flush 2204 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
60922024-02-15T02:58:40.985ZINFOcrucible: client task is exiting = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
60932024-02-15T02:58:41.986ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
60942024-02-15T02:58:41.986ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
60952024-02-15T02:58:41.986ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
60962024-02-15T02:58:41.986ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
60972024-02-15T02:58:41.986ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
60982024-02-15T02:58:41.986ZINFOcrucible: [1] client re-new 301 jobs since flush 2204 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
60992024-02-15T02:58:41.986ZINFOcrucible: client task is exiting = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
61002024-02-15T02:58:42.987ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
61012024-02-15T02:58:42.987ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
61022024-02-15T02:58:42.987ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
61032024-02-15T02:58:42.987ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
61042024-02-15T02:58:42.987ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
61052024-02-15T02:58:42.987ZINFOcrucible: [1] client re-new 301 jobs since flush 2204 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
61062024-02-15T02:58:42.987ZINFOcrucible: client task is exiting = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
6107 [8820] got start action
6108 [8820] I am going to restart
6109 Starting downstairs at port 8820
6110 Make output file at "/tmp/dsc/downstairs-8820.txt"
6111 [8820][1] reports Starting
6112 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1172)
6113 [8820][1] reports Running
61142024-02-15T02:58:43.989ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
61152024-02-15T02:58:43.989ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
61162024-02-15T02:58:43.989ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
61172024-02-15T02:58:43.989ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
61182024-02-15T02:58:43.989ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
61192024-02-15T02:58:43.989ZINFOcrucible: [1] client re-new 301 jobs since flush 2204 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
61202024-02-15T02:58:43.989ZINFOcrucible: client task is exiting = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
6121 ----------------------------------------------------------------
6122 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
6123 States: Active Offline Active
6124 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
6125 1206 Acked 2205 Read 2 Done New Done true
6126 1207 Acked 2206 Read 7 Done New Done true
6127 1208 Acked 2207 Flush 0 Done New Done true
6128 1209 Acked 2208 Read 3 Done New Done true
6129 1210 Acked 2209 Read 6 Done New Done true
6130 1211 Acked 2210 Write 9 Done New Done true
6131 1212 Acked 2211 Read 5 Done New Done true
6132 1213 Acked 2212 Read 3 Done New Done true
6133 1214 Acked 2213 Read 9 Done New Done true
6134 1215 Acked 2214 Read 9 Done New Done true
6135 1216 Acked 2215 Flush 0 Done New Done true
6136 1217 Acked 2216 Read 7 Done New Done true
6137 1218 Acked 2217 Flush 0 Done New Done true
6138 1219 Acked 2218 Read 8 Done New Done true
6139 1220 Acked 2219 Read 9 Done New Done true
6140 1221 Acked 2220 Write 7 Done New Done true
6141 1222 Acked 2221 Read 4 Done New Done true
6142 1223 Acked 2222 Flush 0 Done New Done true
6143 1224 Acked 2223 Flush 0 Done New Done true
6144 1225 Acked 2224 Write 4 Done New Done true
6145 1226 Acked 2225 Read 10 Done New Done true
6146 1227 Acked 2226 Read 8 Done New Done true
6147 1228 Acked 2227 Write 6 Done New Done true
6148 1229 Acked 2228 Read 6 Done New Done true
6149 1230 Acked 2229 Read 5 Done New Done true
6150 1231 Acked 2230 Read 10 Done New Done true
6151 1232 Acked 2231 Read 8 Done New Done true
6152 1233 Acked 2232 Write 10 Done New Done true
6153 1234 Acked 2233 Write 6 Done New Done true
6154 1235 Acked 2234 Read 6 Done New Done true
6155 1236 Acked 2235 Read 7 Done New Done true
6156 1237 Acked 2236 Flush 0 Done New Done true
6157 1238 Acked 2237 Read 3 Done New Done true
6158 1239 Acked 2238 Read 10 Done New Done true
6159 1240 Acked 2239 Read 1 Done New Done true
6160 1241 Acked 2240 Read 6 Done New Done true
6161 1242 Acked 2241 Write 4 Done New Done true
6162 1243 Acked 2242 Read 6 Done New Done true
6163 1244 Acked 2243 Write 5 Done New Done true
6164 1245 Acked 2244 Read 7 Done New Done true
6165 1246 Acked 2245 Read 5 Done New Done true
6166 1247 Acked 2246 Read 2 Done New Done true
6167 1248 Acked 2247 Write 1 Done New Done true
6168 1249 Acked 2248 Write 2 Done New Done true
6169 1250 Acked 2249 Write 10 Done New Done true
6170 1251 Acked 2250 Read 8 Done New Done true
6171 1252 Acked 2251 Write 9 Done New Done true
6172 1253 Acked 2252 Write 6 Done New Done true
6173 1254 Acked 2253 Read 2 Done New Done true
6174 1255 Acked 2254 Read 10 Done New Done true
6175 1256 Acked 2255 Write 1 Done New Done true
6176 1257 Acked 2256 Read 10 Done New Done true
6177 1258 Acked 2257 Flush 0 Done New Done true
6178 1259 Acked 2258 Read 9 Done New Done true
6179 1260 Acked 2259 Write 10 Done New Done true
6180 1261 Acked 2260 Write 10 Done New Done true
6181 1262 Acked 2261 Read 9 Done New Done true
6182 1263 Acked 2262 Write 1 Done New Done true
6183 1264 Acked 2263 Read 2 Done New Done true
6184 1265 Acked 2264 Write 4 Done New Done true
6185 1266 Acked 2265 Read 9 Done New Done true
6186 1267 Acked 2266 Write 3 Done New Done true
6187 1268 Acked 2267 Write 8 Done New Done true
6188 1269 Acked 2268 Write 8 Done New Done true
6189 1270 Acked 2269 Write 6 Done New Done true
6190 1271 Acked 2270 Read 1 Done New Done true
6191 1272 Acked 2271 Write 9 Done New Done true
6192 1273 Acked 2272 Flush 0 Done New Done true
6193 1274 Acked 2273 Read 7 Done New Done true
6194 1275 Acked 2274 Read 10 Done New Done true
6195 1276 Acked 2275 Read 5 Done New Done true
6196 1277 Acked 2276 Write 1 Done New Done true
6197 1278 Acked 2277 Read 1 Done New Done true
6198 1279 Acked 2278 Write 7 Done New Done true
6199 1280 Acked 2279 Write 9 Done New Done true
6200 1281 Acked 2280 Read 3 Done New Done true
6201 1282 Acked 2281 Read 10 Done New Done true
6202 1283 Acked 2282 Write 8 Done New Done true
6203 1284 Acked 2283 Write 4 Done New Done true
6204 1285 Acked 2284 Read 6 Done New Done true
6205 1286 Acked 2285 Flush 0 Done New Done true
6206 1287 Acked 2286 Read 9 Done New Done true
6207 1288 Acked 2287 Read 5 Done New Done true
6208 1289 Acked 2288 Read 7 Done New Done true
6209 1290 Acked 2289 Write 10 Done New Done true
6210 1291 Acked 2290 Read 2 Done New Done true
6211 1292 Acked 2291 Read 4 Done New Done true
6212 1293 Acked 2292 Flush 0 Done New Done true
6213 1294 Acked 2293 Read 10 Done New Done true
6214 1295 Acked 2294 Write 5 Done New Done true
6215 1296 Acked 2295 Write 8 Done New Done true
6216 1297 Acked 2296 Write 3 Done New Done true
6217 1298 Acked 2297 Flush 0 Done New Done true
6218 1299 Acked 2298 Read 9 Done New Done true
6219 1300 Acked 2299 Write 1 Done New Done true
6220 1301 Acked 2300 Flush 0 Done New Done true
6221 1302 Acked 2301 Read 3 Done New Done true
6222 1303 Acked 2302 Read 5 Done New Done true
6223 1304 Acked 2303 Write 4 Done New Done true
6224 1305 Acked 2304 Read 1 Done New Done true
6225 1306 Acked 2305 Read 9 Done New Done true
6226 1307 Acked 2306 Read 7 Done New Done true
6227 1308 Acked 2307 Write 9 Done New Done true
6228 1309 Acked 2308 Read 4 Done New Done true
6229 1310 Acked 2309 Write 8 Done New Done true
6230 1311 Acked 2310 Flush 0 Done New Done true
6231 1312 Acked 2311 Read 4 Done New Done true
6232 1313 Acked 2312 Write 4 Done New Done true
6233 1314 Acked 2313 Read 7 Done New Done true
6234 1315 Acked 2314 Write 5 Done New Done true
6235 1316 Acked 2315 Flush 0 Done New Done true
6236 1317 Acked 2316 Write 5 Done New Done true
6237 1318 Acked 2317 Write 9 Done New Done true
6238 1319 Acked 2318 Write 9 Done New Done true
6239 1320 Acked 2319 Write 1 Done New Done true
6240 1321 Acked 2320 Write 2 Done New Done true
6241 1322 Acked 2321 Write 6 Done New Done true
6242 1323 Acked 2322 Write 10 Done New Done true
6243 1324 Acked 2323 Write 1 Done New Done true
6244 1325 Acked 2324 Flush 0 Done New Done true
6245 1326 Acked 2325 Write 5 Done New Done true
6246 1327 Acked 2326 Flush 0 Done New Done true
6247 1328 Acked 2327 Read 3 Done New Done true
6248 1329 Acked 2328 Read 9 Done New Done true
6249 1330 Acked 2329 Read 3 Done New Done true
6250 1331 Acked 2330 Read 4 Done New Done true
6251 1332 Acked 2331 Read 7 Done New Done true
6252 1333 Acked 2332 Write 2 Done New Done true
6253 1334 Acked 2333 Write 7 Done New Done true
6254 1335 Acked 2334 Write 7 Done New Done true
6255 1336 Acked 2335 Read 10 Done New Done true
6256 1337 Acked 2336 Read 2 Done New Done true
6257 1338 Acked 2337 Write 4 Done New Done true
6258 1339 Acked 2338 Flush 0 Done New Done true
6259 1340 Acked 2339 Write 9 Done New Done true
6260 1341 Acked 2340 Read 7 Done New Done true
6261 1342 Acked 2341 Read 2 Done New Done true
6262 1343 Acked 2342 Read 9 Done New Done true
6263 1344 Acked 2343 Read 10 Done New Done true
6264 1345 Acked 2344 Flush 0 Done New Done true
6265 1346 Acked 2345 Write 6 Done New Done true
6266 1347 Acked 2346 Read 9 Done New Done true
6267 1348 Acked 2347 Read 4 Done New Done true
6268 1349 Acked 2348 Flush 0 Done New Done true
6269 1350 Acked 2349 Read 7 Done New Done true
6270 1351 Acked 2350 Write 9 Done New Done true
6271 1352 Acked 2351 Write 5 Done New Done true
6272 1353 Acked 2352 Flush 0 Done New Done true
6273 1354 Acked 2353 Write 8 Done New Done true
6274 1355 Acked 2354 Write 8 Done New Done true
6275 1356 Acked 2355 Read 3 Done New Done true
6276 1357 Acked 2356 Read 1 Done New Done true
6277 1358 Acked 2357 Read 7 Done New Done true
6278 1359 Acked 2358 Read 7 Done New Done true
6279 1360 Acked 2359 Read 10 Done New Done true
6280 1361 Acked 2360 Flush 0 Done New Done true
6281 1362 Acked 2361 Read 7 Done New Done true
6282 1363 Acked 2362 Flush 0 Done New Done true
6283 1364 Acked 2363 Read 7 Done New Done true
6284 1365 Acked 2364 Write 5 Done New Done true
6285 1366 Acked 2365 Write 9 Done New Done true
6286 1367 Acked 2366 Flush 0 Done New Done true
6287 1368 Acked 2367 Write 5 Done New Done true
6288 1369 Acked 2368 Read 6 Done New Done true
6289 1370 Acked 2369 Read 7 Done New Done true
6290 1371 Acked 2370 Read 1 Done New Done true
6291 1372 Acked 2371 Read 4 Done New Done true
6292 1373 Acked 2372 Write 1 Done New Done true
6293 1374 Acked 2373 Flush 0 Done New Done true
6294 1375 Acked 2374 Read 1 Done New Done true
6295 1376 Acked 2375 Read 10 Done New Done true
6296 1377 Acked 2376 Read 3 Done New Done true
6297 1378 Acked 2377 Read 1 Done New Done true
6298 1379 Acked 2378 Read 8 Done New Done true
6299 1380 Acked 2379 Write 7 Done New Done true
6300 1381 Acked 2380 Read 1 Done New Done true
6301 1382 Acked 2381 Flush 0 Done New Done true
6302 1383 Acked 2382 Write 8 Done New Done true
6303 1384 Acked 2383 Read 4 Done New Done true
6304 1385 Acked 2384 Read 7 Done New Done true
6305 1386 Acked 2385 Flush 0 Done New Done true
6306 1387 Acked 2386 Read 1 Done New Done true
6307 1388 Acked 2387 Read 3 Done New Done true
6308 1389 Acked 2388 Write 9 Done New Done true
6309 1390 Acked 2389 Read 4 Done New Done true
6310 1391 Acked 2390 Read 9 Done New Done true
6311 1392 Acked 2391 Read 1 Done New Done true
6312 1393 Acked 2392 Read 2 Done New Done true
6313 1394 Acked 2393 Write 5 Done New Done true
6314 1395 Acked 2394 Write 4 Done New Done true
6315 1396 Acked 2395 Write 1 Done New Done true
6316 1397 Acked 2396 Read 9 Done New Done true
6317 1398 Acked 2397 Read 2 Done New Done true
6318 1399 Acked 2398 Read 5 Done New Done true
6319 1400 Acked 2399 Read 5 Done New Done true
6320 1401 Acked 2400 Read 2 Done New Done true
6321 1402 Acked 2401 Read 9 Done New Done true
6322 1403 Acked 2402 Read 7 Done New Done true
6323 1404 Acked 2403 Read 6 Done New Done true
6324 1405 Acked 2404 Read 2 Done New Done true
6325 1406 Acked 2405 Write 4 Done New Done true
6326 1407 Acked 2406 Read 9 Done New Done true
6327 1408 Acked 2407 Read 9 Done New Done true
6328 1409 Acked 2408 Write 3 Done New Done true
6329 1410 Acked 2409 Flush 0 Done New Done true
6330 1411 Acked 2410 Read 10 Done New Done true
6331 1412 Acked 2411 Read 6 Done New Done true
6332 1413 Acked 2412 Read 8 Done New Done true
6333 1414 Acked 2413 Read 1 Done New Done true
6334 1415 Acked 2414 Read 8 Done New Done true
6335 1416 Acked 2415 Write 8 Done New Done true
6336 1417 Acked 2416 Write 1 Done New Done true
6337 1418 Acked 2417 Write 2 Done New Done true
6338 1419 Acked 2418 Write 1 Done New Done true
6339 1420 Acked 2419 Write 9 Done New Done true
6340 1421 Acked 2420 Read 8 Done New Done true
6341 1422 Acked 2421 Read 10 Done New Done true
6342 1423 Acked 2422 Read 9 Done New Done true
6343 1424 Acked 2423 Write 9 Done New Done true
6344 1425 Acked 2424 Read 2 Done New Done true
6345 1426 Acked 2425 Write 1 Done New Done true
6346 1427 Acked 2426 Read 2 Done New Done true
6347 1428 Acked 2427 Write 6 Done New Done true
6348 1429 Acked 2428 Write 8 Done New Done true
6349 1430 Acked 2429 Write 7 Done New Done true
6350 1431 Acked 2430 Read 8 Done New Done true
6351 1432 Acked 2431 Read 5 Done New Done true
6352 1433 Acked 2432 Read 8 Done New Done true
6353 1434 Acked 2433 Read 1 Done New Done true
6354 1435 Acked 2434 Write 10 Done New Done true
6355 1436 Acked 2435 Read 3 Done New Done true
6356 1437 Acked 2436 Flush 0 Done New Done true
6357 1438 Acked 2437 Write 1 Done New Done true
6358 1439 Acked 2438 Read 8 Done New Done true
6359 1440 Acked 2439 Read 3 Done New Done true
6360 1441 Acked 2440 Read 3 Done New Done true
6361 1442 Acked 2441 Write 8 Done New Done true
6362 1443 Acked 2442 Flush 0 Done New Done true
6363 1444 Acked 2443 Read 9 Done New Done true
6364 1445 Acked 2444 Write 10 Done New Done true
6365 1446 Acked 2445 Write 9 Done New Done true
6366 1447 Acked 2446 Read 5 Done New Done true
6367 1448 Acked 2447 Write 2 Done New Done true
6368 1449 Acked 2448 Write 3 Done New Done true
6369 1450 Acked 2449 Read 1 Done New Done true
6370 1451 Acked 2450 Flush 0 Done New Done true
6371 1452 Acked 2451 Flush 0 Done New Done true
6372 1453 Acked 2452 Write 1 Done New Done true
6373 1454 Acked 2453 Read 4 Done New Done true
6374 1455 Acked 2454 Read 8 Done New Done true
6375 1456 Acked 2455 Read 9 Done New Done true
6376 1457 Acked 2456 Flush 0 Done New Done true
6377 1458 Acked 2457 Read 5 Done New Done true
6378 1459 Acked 2458 Read 8 Done New Done true
6379 1460 Acked 2459 Write 3 Done New Done true
6380 1461 Acked 2460 Write 6 Done New Done true
6381 1462 Acked 2461 Write 3 Done New Done true
6382 1463 Acked 2462 Read 2 Done New Done true
6383 1464 Acked 2463 Write 9 Done New Done true
6384 1465 Acked 2464 Read 2 Done New Done true
6385 1466 Acked 2465 Write 1 Done New Done true
6386 1467 Acked 2466 Read 3 Done New Done true
6387 1468 Acked 2467 Read 1 Done New Done true
6388 1469 Acked 2468 Write 7 Done New Done true
6389 1470 Acked 2469 Write 2 Done New Done true
6390 1471 Acked 2470 Read 1 Done New Done true
6391 1472 Acked 2471 Write 9 Done New Done true
6392 1473 Acked 2472 Flush 0 Done New Done true
6393 1474 Acked 2473 Flush 0 Done New Done true
6394 1475 Acked 2474 Read 7 Done New Done true
6395 1476 Acked 2475 Read 6 Done New Done true
6396 1477 Acked 2476 Read 8 Done New Done true
6397 1478 Acked 2477 Read 5 Done New Done true
6398 1479 Acked 2478 Flush 0 Done New Done true
6399 1480 Acked 2479 Read 2 Done New Done true
6400 1481 Acked 2480 Write 9 Done New Done true
6401 1482 Acked 2481 Write 4 Done New Done true
6402 1483 Acked 2482 Read 2 Done New Done true
6403 1484 Acked 2483 Write 8 Done New Done true
6404 1485 Acked 2484 Read 9 Done New Done true
6405 1486 Acked 2485 Write 6 Done New Done true
6406 1487 Acked 2486 Flush 0 Done New Done true
6407 1488 Acked 2487 Read 7 Done New Done true
6408 1489 Acked 2488 Read 5 Done New Done true
6409 1490 Acked 2489 Write 7 Done New Done true
6410 1491 Acked 2490 Read 4 Done New Done true
6411 1492 Acked 2491 Write 9 Done New Done true
6412 1493 Acked 2492 Read 9 Done New Done true
6413 1494 Acked 2493 Read 10 Done New Done true
6414 1495 Acked 2494 Read 3 Done New Done true
6415 1496 Acked 2495 Flush 0 Done New Done true
6416 1497 Acked 2496 Read 8 Done New Done true
6417 1498 Acked 2497 Read 6 Done New Done true
6418 1499 Acked 2498 Read 4 Done New Done true
6419 1500 Acked 2499 Write 9 Done New Done true
6420 1501 Acked 2500 Write 9 Done New Done true
6421 1502 Acked 2501 Read 1 Done New Done true
6422 1503 Acked 2502 Write 3 Done New Done true
6423 1504 Acked 2503 Read 5 Done New Done true
6424 1505 Acked 2504 Write 9 Done New Done true
6425 1506 Acked 2505 Flush 0 Done New Done true
6426 STATES DS:0 DS:1 DS:2 TOTAL
6427 New 0 301 0 301
6428 Sent 0 0 0 0
6429 Done 301 0 301 602
6430 Skipped 0 0 0 0
6431 Error 0 0 0 0
6432 Last Flush: 2505 2204 2505
6433 Downstairs last five completed: 2204 2203 2202 2201 2200
6434 Upstairs last five completed: 1506 1505 1504 1503 1502
6435 CLIENT: Up:0 ds:301 act:2
64362024-02-15T02:58:44.990ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
64372024-02-15T02:58:44.990ZINFOcrucible: ds_connection connected = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
64382024-02-15T02:58:44.992ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
64392024-02-15T02:58:44.992ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
64402024-02-15T02:58:44.992ZINFOcrucible: send last flush ID to this DS: 2204 = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
64412024-02-15T02:58:44.992ZINFOcrucible: Replied this last flush ID: 2204 = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
64422024-02-15T02:58:44.992ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
64432024-02-15T02:58:44.992ZINFOcrucible: [1] Transition from Offline to Replay = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
64442024-02-15T02:58:44.992ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
64452024-02-15T02:58:44.992ZINFOcrucible: [1] Transition from Replay to Active = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
6446 ----------------------------------------------------------------
6447 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
6448 Downstairs last five completed: 2505 2504 2503 2502 2501
6449 Upstairs last five completed: 1506 1505 1504 1503 1502
6450 CLIENT: Up:0 ds:0 act:3
6451 Replay: All jobs finished, all DS active.
6452 Main task has work to do, go find it
6453 got dsc Stop(1)
6454 stop 1
6455 Feb 15 02:58:48.316 INFO request completed, latency_us:[8820] Got stop action so:false kr:false
6456 335, response_code: 204, uri: /stop/cid/1, method: POST, req_id: 054b5fdb-4ac3-4159-9961-5b0958bdb79b, remote_addr: 127.0.0.1:54744, local_addr: 127.0.0.1:9998
64572024-02-15T02:58:48.316ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
64582024-02-15T02:58:48.317ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
64592024-02-15T02:58:48.317ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
64602024-02-15T02:58:48.317ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
64612024-02-15T02:58:48.317ZINFOcrucible: [1] client re-new 0 jobs since flush 2505 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
64622024-02-15T02:58:48.317ZINFOcrucible: client task is exiting = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
6463 Feb 15 02:58:48.317 INFO request completed, latency_us: 133, response_code: 200, uri: /state/cid/1, method: GET, req_id: f282ddb1-cda9-400f-b86d-6d9066fc97b9, remote_addr: 127.0.0.1:54744, local_addr: 127.0.0.1:9998
6464 [8820] Exited with: ExitStatus(unix_wait_status(9))
6465 [8820][1] reports Exit
64662024-02-15T02:58:49.318ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
64672024-02-15T02:58:49.318ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
64682024-02-15T02:58:49.318ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
64692024-02-15T02:58:49.318ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
64702024-02-15T02:58:49.318ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
64712024-02-15T02:58:49.318ZINFOcrucible: [1] client re-new 0 jobs since flush 2505 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
64722024-02-15T02:58:49.318ZINFOcrucible: client task is exiting = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
64732024-02-15T02:58:50.319ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
64742024-02-15T02:58:50.319ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
64752024-02-15T02:58:50.319ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
64762024-02-15T02:58:50.319ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
64772024-02-15T02:58:50.319ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
64782024-02-15T02:58:50.319ZINFOcrucible: [1] client re-new 0 jobs since flush 2505 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
64792024-02-15T02:58:50.319ZINFOcrucible: client task is exiting = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
64802024-02-15T02:58:51.320ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
64812024-02-15T02:58:51.320ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
64822024-02-15T02:58:51.320ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
64832024-02-15T02:58:51.320ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
64842024-02-15T02:58:51.320ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
64852024-02-15T02:58:51.320ZINFOcrucible: [1] client re-new 0 jobs since flush 2505 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
64862024-02-15T02:58:51.320ZINFOcrucible: client task is exiting = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
6487 Feb 15 02:58:52.319 INFO request completed, latency_us: 247, response_code: 200, uri: /state/cid/1, method: GET, req_id: 83c4a776-c654-4cad-8286-387080a4bfbd, remote_addr: 127.0.0.1:54744, local_addr: 127.0.0.1:9998
6488 001/300 Write block 3728 len 32768 data: 4 4 5 4 4 4 4 3
6489 002/300 Write block 4678 len 12288 data: 4 4 4
64902024-02-15T02:58:52.321ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
64912024-02-15T02:58:52.321ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
64922024-02-15T02:58:52.321ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
64932024-02-15T02:58:52.321ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
64942024-02-15T02:58:52.321ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
64952024-02-15T02:58:52.321ZINFOcrucible: [1] client re-new 2 jobs since flush 2505 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
64962024-02-15T02:58:52.321ZINFOcrucible: client task is exiting = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
6497 003/300 Write block 4376 len 32768 data: 3 3 3 2 2 2 2 2
6498 004/300 Read block 1136 len 28672
6499 005/300 Write block 3769 len 12288 data: 4 4 3
6500 006/300 Write block 376 len 36864 data: 3 4 4 3 3 2 2 2 3
6501 007/300 Read block 561 len 28672
6502 008/300 Read block 2219 len 20480
6503 009/300 Write block 367 len 24576 data: 2 2 2 3 3 3
6504 010/300 Read block 1534 len 8192
6505 011/300 Read block 3809 len 24576
6506 012/300 Read block 2575 len 20480
6507 013/300 Read block 125 len 32768
6508 014/300 Read block 3360 len 32768
6509 015/300 Write block 2127 len 8192 data: 3 3
6510 016/300 Write block 4139 len 28672 data: 4 4 3 3 3 3 3
6511 017/300 Read block 1885 len 24576
6512 018/300 Write block 3135 len 12288 data: 3 3 3
6513 019/300 Write block 3303 len 8192 data: 3 3
6514 020/300 Read block 1904 len 40960
6515 021/300 Read block 1181 len 28672
6516 022/300 Read block 804 len 36864
6517 023/300 Write block 4187 len 36864 data: 2 2 2 2 2 2 2 2 2
6518 024/300 Write block 2889 len 4096 data: 2
6519 025/300 Write block 2410 len 8192 data: 2 2
6520 026/300 Read block 1488 len 32768
6521 027/300 Write block 1037 len 24576 data: 2 2 2 2 2 2
6522 028/300 Write block 3907 len 4096 data: 2
6523 029/300 Read block 3734 len 16384
6524 030/300 Read block 2740 len 20480
6525 031/300 Write block 2612 len 16384 data: 2 2 2 2
6526 032/300 Write block 902 len 8192 data: 3 3
6527 033/300 Read block 113 len 8192
6528 034/300 Write block 4015 len 24576 data: 4 3 3 3 4 4
6529 035/300 Read block 655 len 8192
6530 036/300 Read block 4491 len 28672
6531 037/300 Write block 4425 len 12288 data: 2 2 2
6532 038/300 Write block 1026 len 40960 data: 2 2 2 2 2 2 2 2 2 2
6533 039/300 Write block 4639 len 16384 data: 2 3 3 3
6534 040/300 Flush
6535 041/300 Write block 2542 len 40960 data: 2 3 3 2 3 3 3 3 2 2
6536 042/300 Read block 1760 len 4096
6537 043/300 Read block 1683 len 24576
6538 044/300 Write block 443 len 40960 data: 2 2 3 3 3 3 2 2 2 2
6539 045/300 Read block 837 len 8192
6540 046/300 Write block 3967 len 40960 data: 2 2 2 3 3 2 2 2 2 2
6541 047/300 Write block 2881 len 16384 data: 4 4 4 4
6542 048/300 Read block 1097 len 28672
6543 049/300 Flush
6544 050/300 Read block 1292 len 20480
6545 051/300 Write block 3267 len 40960 data: 2 2 2 2 2 2 2 2 2 2
6546 052/300 Write block 2935 len 28672 data: 2 2 2 2 2 2 2
6547 053/300 Flush
6548 054/300 Read block 2935 len 28672
6549 055/300 Flush
6550 056/300 Read block 4812 len 40960
6551 057/300 Write block 4725 len 20480 data: 2 2 2 2 2
6552 058/300 Read block 311 len 20480
6553 059/300 Flush
6554 060/300 Write block 977 len 28672 data: 3 3 3 3 2 3 3
6555 061/300 Write block 4746 len 40960 data: 2 2 2 2 2 2 2 2 2 2
6556 062/300 Write block 2082 len 16384 data: 3 4 4 5
6557 063/300 Read block 97 len 36864
6558 064/300 Read block 3722 len 20480
6559 065/300 Read block 4459 len 8192
6560 066/300 Read block 429 len 32768
6561 067/300 Write block 270 len 36864 data: 2 2 2 2 2 2 2 2 2
6562 068/300 Write block 3134 len 28672 data: 3 4 4 4 3 3 3
6563 069/300 Read block 4353 len 36864
6564 070/300 Read block 1169 len 28672
6565 071/300 Write block 570 len 16384 data: 3 2 2 2
6566 072/300 Read block 905 len 24576
6567 073/300 Write block 70 len 36864 data: 2 2 2 3 3 3 2 2 2
6568 074/300 Read block 4821 len 4096
6569 075/300 Read block 331 len 20480
6570 076/300 Read block 3771 len 20480
6571 077/300 Read block 960 len 20480
6572 078/300 Flush
6573 079/300 Read block 1764 len 28672
6574 080/300 Read block 172 len 40960
6575 081/300 Flush
6576 082/300 Write block 959 len 8192 data: 5 5
6577 083/300 Read block 2954 len 40960
6578 084/300 Write block 1086 len 40960 data: 3 2 2 2 2 2 3 3 3 3
6579 085/300 Write block 3823 len 12288 data: 2 2 3
6580 086/300 Write block 2620 len 4096 data: 3
6581 087/300 Read block 102 len 28672
6582 088/300 Read block 1050 len 4096
6583 089/300 Read block 2996 len 4096
6584 090/300 Read block 1781 len 4096
6585 091/300 Write block 263 len 32768 data: 3 3 3 3 3 2 2 3
6586 092/300 Write block 4773 len 8192 data: 4 4
6587 093/300 Flush
6588 094/300 Write block 492 len 32768 data: 4 3 3 3 3 3 3 2
6589 095/300 Write block 3773 len 28672 data: 2 2 2 2 2 2 2
6590 096/300 Write block 3529 len 28672 data: 3 3 3 3 5 5 4
6591 097/300 Read block 4125 len 20480
6592 098/300 Read block 4902 len 4096
6593 099/300 Write block 3595 len 24576 data: 3 3 3 3 3 2
6594 100/300 Write block 1825 len 24576 data: 3 3 3 4 4 4
6595 101/300 Read block 1430 len 20480
6596 102/300 Read block 2101 len 24576
6597 103/300 Read block 3562 len 16384
6598 104/300 Write block 3720 len 28672 data: 3 2 2 2 2 2 2
6599 105/300 Write block 2448 len 8192 data: 3 3
6600 106/300 Read block 3755 len 8192
6601 107/300 Write block 2223 len 36864 data: 3 3 3 3 3 2 2 2 2
6602 108/300 Read block 1609 len 40960
6603 109/300 Write block 4718 len 28672 data: 2 2 2 2 2 2 2
6604 110/300 Write block 1176 len 16384 data: 2 3 3 3
6605 111/300 Write block 4932 len 12288 data: 3 3 3
6606 112/300 Write block 4214 len 40960 data: 3 3 3 3 3 3 2 3 2 2
6607 113/300 Read block 3549 len 20480
6608 114/300 Read block 4688 len 36864
6609 115/300 Read block 2791 len 28672
6610 116/300 Read block 813 len 8192
6611 117/300 Write block 4929 len 28672 data: 2 2 3 4 4 4 2
6612 118/300 Flush
6613 119/300 Read block 4481 len 24576
6614 120/300 Read block 1164 len 4096
6615 121/300 Read block 4043 len 28672
6616 122/300 Read block 1757 len 4096
6617 123/300 Flush
6618 124/300 Read block 3621 len 16384
6619 125/300 Read block 496 len 4096
6620 126/300 Write block 4815 len 12288 data: 2 3 3
6621 127/300 Write block 1366 len 32768 data: 2 3 3 3 2 2 2 2
6622 128/300 Read block 3967 len 28672
6623 129/300 Read block 2732 len 20480
6624 130/300 Write block 4711 len 16384 data: 2 3 4 4
6625 131/300 Write block 826 len 12288 data: 2 2 2
6626 132/300 Read block 4542 len 12288
6627 133/300 Read block 3649 len 16384
6628 134/300 Read block 4682 len 8192
6629 135/300 Read block 2770 len 36864
6630 136/300 Write block 1820 len 28672 data: 2 2 2 3 3 4 4
6631 137/300 Write block 4870 len 32768 data: 3 3 3 2 2 2 2 2
6632 138/300 Write block 3920 len 8192 data: 2 2
6633 139/300 Write block 707 len 40960 data: 3 2 2 2 2 2 3 3 3 3
6634 140/300 Read block 1349 len 24576
6635 141/300 Read block 4988 len 28672
6636 142/300 Write block 838 len 40960 data: 3 3 3 3 3 3 4 3 3 4
6637 143/300 Write block 165 len 40960 data: 3 3 3 3 2 2 2 2 2 2
6638 144/300 Read block 4431 len 40960
6639 145/300 Read block 265 len 40960
6640 146/300 Write block 4231 len 36864 data: 2 2 2 2 2 3 3 3 3
6641 147/300 Read block 3123 len 8192
6642 148/300 Read block 4165 len 40960
6643 149/300 Read block 1557 len 4096
6644 150/300 Read block 198 len 24576
6645 151/300 Read block 827 len 20480
6646 152/300 Read block 2791 len 4096
6647 153/300 Write block 4501 len 8192 data: 2 3
6648 154/300 Write block 4235 len 28672 data: 3 4 4 4 4 3 3
6649 155/300 Read block 3029 len 4096
6650 156/300 Read block 2142 len 4096
6651 157/300 Read block 2520 len 28672
6652 158/300 Write block 369 len 16384 data: 3 4 4 4
6653 159/300 Read block 159 len 28672
6654 160/300 Read block 4388 len 32768
6655 161/300 Read block 551 len 24576
6656 162/300 Read block 2110 len 24576
6657 163/300 Write block 51 len 20480 data: 2 4 4 4 4
6658 164/300 Read block 1201 len 40960
6659 165/300 Read block 2854 len 24576
6660 166/300 Write block 2225 len 40960 data: 4 4 4 3 3 3 3 2 2 2
6661 167/300 Write block 1158 len 8192 data: 2 2
6662 168/300 Read block 2405 len 8192
6663 169/300 Read block 2075 len 8192
6664 170/300 Read block 998 len 32768
6665 171/300 Flush
6666 172/300 Read block 2947 len 16384
6667 173/300 Read block 3100 len 16384
6668 174/300 Write block 3413 len 12288 data: 4 4 4
6669 175/300 Write block 300 len 24576 data: 2 2 2 2 3 3
6670 176/300 Read block 675 len 40960
6671 177/300 Read block 1032 len 36864
6672 178/300 Read block 1627 len 12288
6673 179/300 Write block 227 len 4096 data: 4
6674 180/300 Read block 1593 len 8192
6675 181/300 Write block 4375 len 16384 data: 3 4 4 4
6676 182/300 Write block 2143 len 32768 data: 2 2 2 2 2 2 2 2
6677 183/300 Read block 4011 len 24576
6678 184/300 Write block 3155 len 36864 data: 4 4 4 6 5 5 6 6 5
6679 185/300 Write block 4089 len 24576 data: 2 2 2 2 2 2
6680 186/300 Read block 429 len 8192
6681 187/300 Flush
6682 188/300 Read block 4692 len 32768
6683 189/300 Read block 4575 len 16384
6684 190/300 Read block 568 len 40960
6685 191/300 Read block 2517 len 16384
6686 192/300 Write block 2867 len 4096 data: 2
6687 193/300 Read block 33 len 16384
6688 194/300 Write block 4762 len 16384 data: 3 3 3 2
6689 195/300 Write block 719 len 24576 data: 3 4 5 4 4 4
6690 196/300 Write block 1625 len 24576 data: 5 5 4 4 3 2
6691 197/300 Read block 2838 len 20480
6692 198/300 Write block 38 len 8192 data: 2 2
6693 199/300 Write block 2192 len 36864 data: 3 3 3 3 3 2 2 2 2
6694 200/300 Read block 4967 len 4096
6695 201/300 Read block 4249 len 36864
6696 202/300 Read block 709 len 40960
6697 203/300 Write block 3559 len 32768 data: 2 2 2 2 2 2 2 2
6698 204/300 Write block 2477 len 24576 data: 2 3 3 3 3 3
6699 205/300 Read block 3038 len 12288
6700 206/300 Write block 2352 len 4096 data: 2
6701 207/300 Read block 2184 len 28672
6702 208/300 Read block 4120 len 36864
6703 209/300 Read block 3630 len 4096
6704 210/300 Write block 1139 len 12288 data: 3 2 2
6705 211/300 Flush
6706 212/300 Write block 1210 len 24576 data: 2 2 2 2 2 2
6707 213/300 Read block 4867 len 20480
6708 214/300 Read block 1285 len 40960
6709 215/300 Read block 2267 len 20480
6710 216/300 Write block 3248 len 24576 data: 3 2 2 2 2 3
6711 217/300 Write block 161 len 4096 data: 3
6712 218/300 Read block 695 len 36864
6713 219/300 Write block 2901 len 4096 data: 3
6714 220/300 Write block 3572 len 16384 data: 2 2 2 2
6715 221/300 Flush
6716 222/300 Write block 2126 len 32768 data: 3 4 4 3 3 3 3 4
6717 223/300 Read block 2259 len 36864
6718 224/300 Write block 4433 len 32768 data: 3 3 3 3 3 2 3 3
6719 225/300 Write block 4358 len 8192 data: 2 2
6720 226/300 Flush
6721 227/300 Read block 631 len 4096
6722 228/300 Write block 3774 len 40960 data: 3 3 3 3 3 3 2 2 2 2
6723 229/300 Read block 1080 len 8192
6724 230/300 Write block 4812 len 20480 data: 3 3 2 3 4
6725 231/300 Read block 2730 len 4096
6726 232/300 Read block 3721 len 20480
6727 233/300 Read block 3713 len 24576
6728 234/300 Write block 2520 len 36864 data: 2 2 2 2 2 2 2 2 2
6729 235/300 Write block 3141 len 36864 data: 3 3 2 2 2 2 2 2 3
6730 236/300 Read block 4293 len 36864
6731 237/300 Write block 3565 len 32768 data: 3 3 2 2 2 2 2 3
6732 238/300 Read block 3654 len 24576
6733 239/300 Read block 2816 len 4096
6734 240/300 Read block 893 len 36864
6735 241/300 Read block 3968 len 8192
6736 242/300 Write block 4066 len 32768 data: 3 3 3 2 2 2 2 2
6737 243/300 Read block 581 len 4096
6738 244/300 Write block 3192 len 36864 data: 3 2 2 2 3 3 2 2 2
6739 245/300 Write block 2945 len 8192 data: 2 2
6740 246/300 Write block 4089 len 28672 data: 3 3 3 3 3 3 2
6741 247/300 Read block 1118 len 32768
6742 248/300 Read block 486 len 36864
6743 249/300 Write block 3945 len 32768 data: 2 2 2 2 2 2 3 3
6744 250/300 Read block 4622 len 24576
6745 251/300 Write block 4931 len 16384 data: 4 5 5 5
6746 252/300 Read block 2768 len 16384
6747 253/300 Write block 2164 len 8192 data: 2 2
6748 254/300 Write block 3827 len 28672 data: 2 2 2 2 2 2 2
6749 255/300 Read block 4882 len 8192
6750 256/300 Flush
6751 257/300 Read block 3946 len 20480
6752 258/300 Write block 350 len 32768 data: 2 2 2 2 2 2 2 3
6753 259/300 Write block 174 len 36864 data: 3 2 3 3 2 2 2 2 2
6754 260/300 Read block 1694 len 40960
6755 261/300 Read block 2994 len 40960
6756 262/300 Flush
6757 263/300 Flush
6758 264/300 Read block 3307 len 4096
6759 265/300 Read block 267 len 36864
6760 266/300 Read block 1139 len 28672
6761 267/300 Write block 2204 len 4096 data: 2
6762 268/300 Write block 4432 len 12288 data: 3 4 4
6763 269/300 Write block 4238 len 8192 data: 5 5
6764 270/300 Write block 1234 len 4096 data: 2
6765 271/300 Read block 2067 len 40960
6766 272/300 Write block 3517 len 4096 data: 2
6767 273/300 Read block 2101 len 32768
6768 274/300 Write block 2366 len 8192 data: 2 2
6769 275/300 Write block 3156 len 16384 data: 5 5 7 6
6770 276/300 Write block 1365 len 20480 data: 2 3 4 4 4
6771 277/300 Write block 1319 len 16384 data: 3 3 3 3
6772 278/300 Read block 1601 len 16384
6773 279/300 Write block 1349 len 8192 data: 3 2
6774 280/300 Write block 4268 len 12288 data: 2 2 2
6775 281/300 Read block 4903 len 16384
6776 282/300 Write block 2631 len 8192 data: 2 3
6777 283/300 Read block 1024 len 4096
6778 284/300 Read block 1333 len 28672
6779 285/300 Read block 1672 len 24576
6780 286/300 Read block 3370 len 20480
6781 287/300 Read block 1866 len 20480
6782 288/300 Write block 4266 len 20480 data: 2 2 3 3 3
6783 289/300 Write block 4750 len 36864 data: 3 3 3 3 3 3 2 2 2
6784 290/300 Read block 3310 len 36864
6785 291/300 Write block 2593 len 40960 data: 2 2 2 2 2 2 2 2 2 2
6786 292/300 Write block 4792 len 16384 data: 4 3 3 3
6787 293/300 Write block 1712 len 8192 data: 2 2
6788 294/300 Read block 1816 len 12288
6789 295/300 Read block 3842 len 4096
6790 296/300 Flush
6791 297/300 Write block 2747 len 32768 data: 3 3 3 2 2 2 2 2
6792 298/300 Write block 3530 len 4096 data: 4
6793 299/300 Read block 4828 len 36864
6794 300/300 Read block 3077 len 16384
6795 Main task has work to do, go find it
6796 got dsc Start(1)
6797 start 1
6798 Feb 15 02:58:52.648 INFO request completed, latency_us: 206, response_code: 204, uri: /start/cid/1, method: POST, req_id: 9b7a560f-9b82-40d1-940f-715750afb982, remote_addr: 127.0.0.1:54744, local_addr: 127.0.0.1:9998
6799 Replay: started 1, returned:Ok(())
6800 ----------------------------------------------------------------
6801 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:300
6802 States: Active Offline Active
6803 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
6804 1507 Acked 2506 Write 8 Done New Done true
6805 1508 Acked 2507 Write 3 Done New Done true
6806 1509 Acked 2508 Write 8 Done New Done false
6807 1510 Acked 2509 Read 7 Done New Done false
6808 1511 Acked 2510 Write 3 Done New Done false
6809 1512 Acked 2511 Write 9 Done New Done false
6810 1513 Acked 2512 Read 7 Done New Done false
6811 1514 Acked 2513 Read 5 Done New Done false
6812 1515 Acked 2514 Write 6 Done New Done false
6813 1516 Acked 2515 Read 2 Done New Done false
6814 1517 Acked 2516 Read 6 Done New Done false
6815 1518 Acked 2517 Read 5 Done New Done false
6816 1519 Acked 2518 Read 8 Done New Done false
6817 1520 Acked 2519 Read 8 Done New Done false
6818 1521 Acked 2520 Write 2 Done New Done false
6819 1522 Acked 2521 Write 7 Done New Done false
6820 1523 Acked 2522 Read 6 Done New Done false
6821 1524 Acked 2523 Write 3 Done New Done false
6822 1525 Acked 2524 Write 2 Done New Done false
6823 1526 Acked 2525 Read 10 Done New Done false
6824 1527 Acked 2526 Read 7 Done New Done false
6825 1528 Acked 2527 Read 9 Done New Done false
6826 1529 Acked 2528 Write 9 Done New Done false
6827 1530 Acked 2529 Write 1 Done New Done false
6828 1531 Acked 2530 Write 2 Done New Done false
6829 1532 Acked 2531 Read 8 Done New Done false
6830 1533 Acked 2532 Write 6 Done New Done false
6831 1534 Acked 2533 Write 1 Done New Done false
6832 1535 Acked 2534 Read 4 Done New Done false
6833 1536 Acked 2535 Read 5 Done New Done false
6834 1537 Acked 2536 Write 4 Done New Done false
6835 1538 Acked 2537 Write 2 Done New Done false
6836 1539 Acked 2538 Read 2 Done New Done false
6837 1540 Acked 2539 Write 6 Done New Done false
6838 1541 Acked 2540 Read 2 Done New Done false
6839 1542 Acked 2541 Read 7 Done New Done false
6840 1543 Acked 2542 Write 3 Done New Done false
6841 1544 Acked 2543 Write 10 Done New Done false
6842 1545 Acked 2544 Write 4 Done New Done false
6843 1546 Acked 2545 Flush 0 Done New Done false
6844 1547 Acked 2546 Write 10 Done New Done false
6845 1548 Acked 2547 Read 1 Done New Done false
6846 1549 Acked 2548 Read 6 Done New Done false
6847 1550 Acked 2549 Write 10 Done New Done false
6848 1551 Acked 2550 Read 2 Done New Done false
6849 1552 Acked 2551 Write 10 Done New Done false
6850 1553 Acked 2552 Write 4 Done New Done false
6851 1554 Acked 2553 Read 7 Done New Done false
6852 1555 Acked 2554 Flush 0 Done New Done false
6853 1556 Acked 2555 Read 5 Done New Done false
6854 1557 Acked 2556 Write 10 Done New Done false
6855 1558 Acked 2557 Write 7 Done New Done false
6856 1559 Acked 2558 Flush 0 Done New Done false
6857 1560 Acked 2559 Read 7 Done New Done false
6858 1561 Acked 2560 Flush 0 Done New Done false
6859 1562 Acked 2561 Read 10 Done New Done false
6860 1563 Acked 2562 Write 5 Done New Done false
6861 1564 Acked 2563 Read 5 Done New Done false
6862 1565 Acked 2564 Flush 0 Done New Done false
6863 1566 Acked 2565 Write 7 Done New Done false
6864 1567 Acked 2566 Write 10 Done New Done false
6865 1568 Acked 2567 Write 4 Done New Done false
6866 1569 Acked 2568 Read 9 Done New Done false
6867 1570 Acked 2569 Read 5 Done New Done false
6868 1571 Acked 2570 Read 2 Done New Done false
6869 1572 Acked 2571 Read 8 Done New Done false
6870 1573 Acked 2572 Write 9 Done New Done false
6871 1574 Acked 2573 Write 7 Done New Done false
6872 1575 Acked 2574 Read 9 Done New Done false
6873 1576 Acked 2575 Read 7 Done New Done false
6874 1577 Acked 2576 Write 4 Done New Done false
6875 1578 Acked 2577 Read 6 Done New Done false
6876 1579 Acked 2578 Write 9 Done New Done false
6877 1580 Acked 2579 Read 1 Done New Done false
6878 1581 Acked 2580 Read 5 Done New Done false
6879 1582 Acked 2581 Read 5 Done New Done false
6880 1583 Acked 2582 Read 5 Done New Done false
6881 1584 Acked 2583 Flush 0 Done New Done false
6882 1585 Acked 2584 Read 7 Done New Done false
6883 1586 Acked 2585 Read 10 Done New Done false
6884 1587 Acked 2586 Flush 0 Done New Done false
6885 1588 Acked 2587 Write 2 Done New Done false
6886 1589 Acked 2588 Read 10 Done New Done false
6887 1590 Acked 2589 Write 10 Done New Done false
6888 1591 Acked 2590 Write 3 Done New Done false
6889 1592 Acked 2591 Write 1 Done New Done false
6890 1593 Acked 2592 Read 7 Done New Done false
6891 1594 Acked 2593 Read 1 Done New Done false
6892 1595 Acked 2594 Read 1 Done New Done false
6893 1596 Acked 2595 Read 1 Done New Done false
6894 1597 Acked 2596 Write 8 Done New Done false
6895 1598 Acked 2597 Write 2 Done New Done false
6896 1599 Acked 2598 Flush 0 Done New Done false
6897 1600 Acked 2599 Write 8 Done New Done false
6898 1601 Acked 2600 Write 7 Done New Done false
6899 1602 Acked 2601 Write 7 Done New Done false
6900 1603 Acked 2602 Read 5 Done New Done false
6901 1604 Acked 2603 Read 1 Done New Done false
6902 1605 Acked 2604 Write 6 Done New Done false
6903 1606 Acked 2605 Write 6 Done New Done false
6904 1607 Acked 2606 Read 5 Done New Done false
6905 1608 Acked 2607 Read 6 Done New Done false
6906 1609 Acked 2608 Read 4 Done New Done false
6907 1610 Acked 2609 Write 7 Done New Done false
6908 1611 Acked 2610 Write 2 Done New Done false
6909 1612 Acked 2611 Read 2 Done New Done false
6910 1613 Acked 2612 Write 9 Done New Done false
6911 1614 Acked 2613 Read 10 Done New Done false
6912 1615 Acked 2614 Write 7 Done New Done false
6913 1616 Acked 2615 Write 4 Done New Done false
6914 1617 Acked 2616 Write 3 Done New Done false
6915 1618 Acked 2617 Write 10 Done New Done false
6916 1619 Acked 2618 Read 5 Done New Done false
6917 1620 Acked 2619 Read 9 Done New Done false
6918 1621 Acked 2620 Read 7 Done New Done false
6919 1622 Acked 2621 Read 2 Done New Done false
6920 1623 Acked 2622 Write 7 Done New Done false
6921 1624 Acked 2623 Flush 0 Done New Done false
6922 1625 Acked 2624 Read 6 Done New Done false
6923 1626 Acked 2625 Read 1 Done New Done false
6924 1627 Acked 2626 Read 7 Done New Done false
6925 1628 Acked 2627 Read 1 Done New Done false
6926 1629 Acked 2628 Flush 0 Done New Done false
6927 1630 Acked 2629 Read 4 Done New Done false
6928 1631 Acked 2630 Read 1 Done New Done false
6929 1632 Acked 2631 Write 3 Done New Done false
6930 1633 Acked 2632 Write 8 Done New Done false
6931 1634 Acked 2633 Read 7 Done New Done false
6932 1635 Acked 2634 Read 5 Done New Done false
6933 1636 Acked 2635 Write 4 Done New Done false
6934 1637 Acked 2636 Write 3 Done New Done false
6935 1638 Acked 2637 Read 3 Done New Done false
6936 1639 Acked 2638 Read 4 Done New Done false
6937 1640 Acked 2639 Read 2 Done New Done false
6938 1641 Acked 2640 Read 9 Done New Done false
6939 1642 Acked 2641 Write 7 Done New Done false
6940 1643 Acked 2642 Write 8 Done New Done false
6941 1644 Acked 2643 Write 2 Done New Done false
6942 1645 Acked 2644 Write 10 Done New Done false
6943 1646 Acked 2645 Read 6 Done New Done false
6944 1647 Acked 2646 Read 7 Done New Done false
6945 1648 Acked 2647 Write 10 Done New Done false
6946 1649 Acked 2648 Write 10 Done New Done false
6947 1650 Acked 2649 Read 10 Done New Done false
6948 1651 Acked 2650 Read 10 Done New Done false
6949 1652 Acked 2651 Write 9 Done New Done false
6950 1653 Acked 2652 Read 2 Done New Done false
6951 1654 Acked 2653 Read 10 Done New Done false
6952 1655 Acked 2654 Read 1 Done New Done false
6953 1656 Acked 2655 Read 6 Done New Done false
6954 1657 Acked 2656 Read 5 Done New Done false
6955 1658 Acked 2657 Read 1 Done New Done false
6956 1659 Acked 2658 Write 2 Done New Done false
6957 1660 Acked 2659 Write 7 Done New Done false
6958 1661 Acked 2660 Read 1 Done New Done false
6959 1662 Acked 2661 Read 1 Done New Done false
6960 1663 Acked 2662 Read 7 Done New Done false
6961 1664 Acked 2663 Write 4 Done New Done false
6962 1665 Acked 2664 Read 7 Done New Done false
6963 1666 Acked 2665 Read 8 Done New Done false
6964 1667 Acked 2666 Read 6 Done New Done false
6965 1668 Acked 2667 Read 6 Done New Done false
6966 1669 Acked 2668 Write 5 Done New Done false
6967 1670 Acked 2669 Read 10 Done New Done false
6968 1671 Acked 2670 Read 6 Done New Done false
6969 1672 Acked 2671 Write 10 Done New Done false
6970 1673 Acked 2672 Write 2 Done New Done false
6971 1674 Acked 2673 Read 2 Done New Done false
6972 1675 Acked 2674 Read 2 Done New Done false
6973 1676 Acked 2675 Read 8 Done New Done false
6974 1677 Acked 2676 Flush 0 Done New Done false
6975 1678 Acked 2677 Read 4 Done New Done false
6976 1679 Acked 2678 Read 4 Done New Done false
6977 1680 Acked 2679 Write 3 Done New Done false
6978 1681 Acked 2680 Write 6 Done New Done false
6979 1682 Acked 2681 Read 10 Done New Done false
6980 1683 Acked 2682 Read 9 Done New Done false
6981 1684 Acked 2683 Read 3 Done New Done false
6982 1685 Acked 2684 Write 1 Done New Done false
6983 1686 Acked 2685 Read 2 Done New Done false
6984 1687 Acked 2686 Write 4 Done New Done false
6985 1688 Acked 2687 Write 8 Done New Done false
6986 1689 Acked 2688 Read 6 Done New Done false
6987 1690 Acked 2689 Write 9 Done New Done false
6988 1691 Acked 2690 Write 6 Done New Done false
6989 1692 Acked 2691 Read 2 Done New Done false
6990 1693 Acked 2692 Flush 0 Done New Done false
6991 1694 Acked 2693 Read 8 Done New Done false
6992 1695 Acked 2694 Read 4 Done New Done false
6993 1696 Acked 2695 Read 10 Done New Done false
6994 1697 Acked 2696 Read 4 Done New Done false
6995 1698 Acked 2697 Write 1 Done New Done false
6996 1699 Acked 2698 Read 4 Done New Done false
6997 1700 Acked 2699 Write 4 Done New Done false
6998 1701 Acked 2700 Write 6 Done New Done false
6999 1702 Acked 2701 Write 6 Done New Done false
7000 1703 Acked 2702 Read 5 Done New Done false
7001 1704 Acked 2703 Write 2 Done New Done false
7002 1705 Acked 2704 Write 9 Done New Done false
7003 1706 Acked 2705 Read 1 Done New Done false
7004 1707 Acked 2706 Read 9 Done New Done false
7005 1708 Acked 2707 Read 10 Done New Done false
7006 1709 Acked 2708 Write 8 Done New Done false
7007 1710 Acked 2709 Write 6 Done New Done false
7008 1711 Acked 2710 Read 3 Done New Done false
7009 1712 Acked 2711 Write 1 Done New Done false
7010 1713 Acked 2712 Read 7 Done New Done false
7011 1714 Acked 2713 Read 9 Done New Done false
7012 1715 Acked 2714 Read 1 Done New Done false
7013 1716 Acked 2715 Write 3 Done New Done false
7014 1717 Acked 2716 Flush 0 Done New Done false
7015 1718 Acked 2717 Write 6 Done New Done false
7016 1719 Acked 2718 Read 5 Done New Done false
7017 1720 Acked 2719 Read 10 Done New Done false
7018 1721 Acked 2720 Read 5 Done New Done false
7019 1722 Acked 2721 Write 6 Done New Done false
7020 1723 Acked 2722 Write 1 Done New Done false
7021 1724 Acked 2723 Read 9 Done New Done false
7022 1725 Acked 2724 Write 1 Done New Done false
7023 1726 Acked 2725 Write 4 Done New Done false
7024 1727 Acked 2726 Flush 0 Done New Done false
7025 1728 Acked 2727 Write 8 Done New Done false
7026 1729 Acked 2728 Read 9 Done New Done false
7027 1730 Acked 2729 Write 8 Done New Done false
7028 1731 Acked 2730 Write 2 Done New Done false
7029 1732 Acked 2731 Flush 0 Done New Done false
7030 1733 Acked 2732 Read 1 Done New Done false
7031 1734 Acked 2733 Write 10 Done New Done false
7032 1735 Acked 2734 Read 2 Done New Done false
7033 1736 Acked 2735 Write 5 Done New Done false
7034 1737 Acked 2736 Read 1 Done New Done false
7035 1738 Acked 2737 Read 5 Done New Done false
7036 1739 Acked 2738 Read 6 Done New Done false
7037 1740 Acked 2739 Write 9 Done New Done false
7038 1741 Acked 2740 Write 9 Done New Done false
7039 1742 Acked 2741 Read 9 Done New Done false
7040 1743 Acked 2742 Write 8 Done New Done false
7041 1744 Acked 2743 Read 6 Done New Done false
7042 1745 Acked 2744 Read 1 Done New Done false
7043 1746 Acked 2745 Read 9 Done New Done false
7044 1747 Acked 2746 Read 2 Done New Done false
7045 1748 Acked 2747 Write 8 Done New Done false
7046 1749 Acked 2748 Read 1 Done New Done false
7047 1750 Acked 2749 Write 9 Done New Done false
7048 1751 Acked 2750 Write 2 Done New Done false
7049 1752 Acked 2751 Write 7 Done New Done false
7050 1753 Acked 2752 Read 8 Done New Done false
7051 1754 Acked 2753 Read 9 Done New Done false
7052 1755 Acked 2754 Write 8 Done New Done false
7053 1756 Acked 2755 Read 6 Done New Done false
7054 1757 Acked 2756 Write 4 Done New Done false
7055 1758 Acked 2757 Read 4 Done New Done false
7056 1759 Acked 2758 Write 2 Done New Done false
7057 1760 Acked 2759 Write 7 Done New Done false
7058 1761 Acked 2760 Read 2 Done New Done false
7059 1762 Acked 2761 Flush 0 Done New Done false
7060 1763 Acked 2762 Read 5 Done New Done false
7061 1764 Acked 2763 Write 8 Done New Done false
7062 1765 Acked 2764 Write 9 Done New Done false
7063 1766 Acked 2765 Read 10 Done New Done false
7064 1767 Acked 2766 Read 10 Done New Done false
7065 1768 Acked 2767 Flush 0 Done New Done false
7066 1769 Acked 2768 Flush 0 Done New Done false
7067 1770 Acked 2769 Read 1 Done New Done false
7068 1771 Acked 2770 Read 9 Done New Done false
7069 1772 Acked 2771 Read 7 Done New Done false
7070 1773 Acked 2772 Write 1 Done New Done false
7071 1774 Acked 2773 Write 3 Done New Done false
7072 1775 Acked 2774 Write 2 Done New Done false
7073 1776 Acked 2775 Write 1 Done New Done false
7074 1777 Acked 2776 Read 10 Done New Done false
7075 1778 Acked 2777 Write 1 Done New Done false
7076 1779 Acked 2778 Read 8 Done New Done false
7077 1780 Acked 2779 Write 2 Done New Done false
7078 1781 Acked 2780 Write 4 Done New Done false
7079 1782 Acked 2781 Write 5 Done New Done false
7080 1783 Acked 2782 Write 4 Done New Done false
7081 1784 Acked 2783 Read 4 Done New Done false
7082 1785 Acked 2784 Write 2 Done New Done false
7083 1786 Acked 2785 Write 3 Done New Done false
7084 1787 Acked 2786 Read 4 Done New Done false
7085 1788 Acked 2787 Write 2 Done New Done false
7086 1789 Acked 2788 Read 1 Done New Done false
7087 1790 Acked 2789 Read 7 Done New Done false
7088 1791 Acked 2790 Read 6 Done New Done false
7089 1792 Acked 2791 Read 5 Done New Done false
7090 1793 Acked 2792 Read 5 Done New Done false
7091 1794 Acked 2793 Write 5 Done New Done false
7092 1795 Acked 2794 Write 9 Done New Done false
7093 1796 Acked 2795 Read 9 Done New Done false
7094 1797 Acked 2796 Write 10 Done New Done false
7095 1798 Acked 2797 Write 4 Done New Done false
7096 1799 Acked 2798 Write 2 Done New Done false
7097 1800 Acked 2799 Read 3 Done New Done false
7098 1801 Acked 2800 Read 1 Done New Done false
7099 1802 Acked 2801 Flush 0 Done New Done false
7100 1803 Acked 2802 Write 8 Done New Done false
7101 1804 Acked 2803 Write 1 Done New Done false
7102 1805 Acked 2804 Read 9 Done New Done false
7103 1806 Acked 2805 Read 4 Done New Done false
7104 STATES DS:0 DS:1 DS:2 TOTAL
7105 New 0 300 0 300
7106 Sent 0 0 0 0
7107 Done 300 0 300 600
7108 Skipped 0 0 0 0
7109 Error 0 0 0 0
7110 Last Flush: 2801 2505 2801
7111 Downstairs last five completed: 2505 2504 2503 2502 2501
7112 Upstairs last five completed: 1806 1805 1804 1803 1802
7113 CLIENT: Up:0 ds:300 act:2
71142024-02-15T02:58:53.323ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
71152024-02-15T02:58:53.323ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
71162024-02-15T02:58:53.323ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
71172024-02-15T02:58:53.323ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
71182024-02-15T02:58:53.323ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
71192024-02-15T02:58:53.323ZINFOcrucible: [1] client re-new 301 jobs since flush 2505 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
71202024-02-15T02:58:53.323ZINFOcrucible: client task is exiting = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
71212024-02-15T02:58:54.324ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
71222024-02-15T02:58:54.324ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
71232024-02-15T02:58:54.324ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
71242024-02-15T02:58:54.324ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
71252024-02-15T02:58:54.324ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
71262024-02-15T02:58:54.324ZINFOcrucible: [1] client re-new 301 jobs since flush 2505 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
71272024-02-15T02:58:54.324ZINFOcrucible: client task is exiting = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
71282024-02-15T02:58:55.325ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
71292024-02-15T02:58:55.325ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
71302024-02-15T02:58:55.325ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
71312024-02-15T02:58:55.325ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
71322024-02-15T02:58:55.325ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
71332024-02-15T02:58:55.325ZINFOcrucible: [1] client re-new 301 jobs since flush 2505 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
71342024-02-15T02:58:55.325ZINFOcrucible: client task is exiting = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
7135 [8820] got start action
7136 [8820] I am going to restart
7137 Starting downstairs at port 8820
7138 Make output file at "/tmp/dsc/downstairs-8820.txt"
71392024-02-15T02:58:56.326ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
7140 [8820][1] reports Starting
71412024-02-15T02:58:56.326ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
71422024-02-15T02:58:56.326ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
71432024-02-15T02:58:56.326ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
71442024-02-15T02:58:56.326ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
71452024-02-15T02:58:56.326ZINFOcrucible: [1] client re-new 301 jobs since flush 2505 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
71462024-02-15T02:58:56.327ZINFOcrucible: client task is exiting = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
7147 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1173)
7148 [8820][1] reports Running
7149 ----------------------------------------------------------------
7150 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
7151 States: Active Offline Active
7152 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
7153 1507 Acked 2506 Write 8 Done New Done true
7154 1508 Acked 2507 Write 3 Done New Done true
7155 1509 Acked 2508 Write 8 Done New Done true
7156 1510 Acked 2509 Read 7 Done New Done true
7157 1511 Acked 2510 Write 3 Done New Done true
7158 1512 Acked 2511 Write 9 Done New Done true
7159 1513 Acked 2512 Read 7 Done New Done true
7160 1514 Acked 2513 Read 5 Done New Done true
7161 1515 Acked 2514 Write 6 Done New Done true
7162 1516 Acked 2515 Read 2 Done New Done true
7163 1517 Acked 2516 Read 6 Done New Done true
7164 1518 Acked 2517 Read 5 Done New Done true
7165 1519 Acked 2518 Read 8 Done New Done true
7166 1520 Acked 2519 Read 8 Done New Done true
7167 1521 Acked 2520 Write 2 Done New Done true
7168 1522 Acked 2521 Write 7 Done New Done true
7169 1523 Acked 2522 Read 6 Done New Done true
7170 1524 Acked 2523 Write 3 Done New Done true
7171 1525 Acked 2524 Write 2 Done New Done true
7172 1526 Acked 2525 Read 10 Done New Done true
7173 1527 Acked 2526 Read 7 Done New Done true
7174 1528 Acked 2527 Read 9 Done New Done true
7175 1529 Acked 2528 Write 9 Done New Done true
7176 1530 Acked 2529 Write 1 Done New Done true
7177 1531 Acked 2530 Write 2 Done New Done true
7178 1532 Acked 2531 Read 8 Done New Done true
7179 1533 Acked 2532 Write 6 Done New Done true
7180 1534 Acked 2533 Write 1 Done New Done true
7181 1535 Acked 2534 Read 4 Done New Done true
7182 1536 Acked 2535 Read 5 Done New Done true
7183 1537 Acked 2536 Write 4 Done New Done true
7184 1538 Acked 2537 Write 2 Done New Done true
7185 1539 Acked 2538 Read 2 Done New Done true
7186 1540 Acked 2539 Write 6 Done New Done true
7187 1541 Acked 2540 Read 2 Done New Done true
7188 1542 Acked 2541 Read 7 Done New Done true
7189 1543 Acked 2542 Write 3 Done New Done true
7190 1544 Acked 2543 Write 10 Done New Done true
7191 1545 Acked 2544 Write 4 Done New Done true
7192 1546 Acked 2545 Flush 0 Done New Done true
7193 1547 Acked 2546 Write 10 Done New Done true
7194 1548 Acked 2547 Read 1 Done New Done true
7195 1549 Acked 2548 Read 6 Done New Done true
7196 1550 Acked 2549 Write 10 Done New Done true
7197 1551 Acked 2550 Read 2 Done New Done true
7198 1552 Acked 2551 Write 10 Done New Done true
7199 1553 Acked 2552 Write 4 Done New Done true
7200 1554 Acked 2553 Read 7 Done New Done true
7201 1555 Acked 2554 Flush 0 Done New Done true
7202 1556 Acked 2555 Read 5 Done New Done true
7203 1557 Acked 2556 Write 10 Done New Done true
7204 1558 Acked 2557 Write 7 Done New Done true
7205 1559 Acked 2558 Flush 0 Done New Done true
7206 1560 Acked 2559 Read 7 Done New Done true
7207 1561 Acked 2560 Flush 0 Done New Done true
7208 1562 Acked 2561 Read 10 Done New Done true
7209 1563 Acked 2562 Write 5 Done New Done true
7210 1564 Acked 2563 Read 5 Done New Done true
7211 1565 Acked 2564 Flush 0 Done New Done true
7212 1566 Acked 2565 Write 7 Done New Done true
7213 1567 Acked 2566 Write 10 Done New Done true
7214 1568 Acked 2567 Write 4 Done New Done true
7215 1569 Acked 2568 Read 9 Done New Done true
7216 1570 Acked 2569 Read 5 Done New Done true
7217 1571 Acked 2570 Read 2 Done New Done true
7218 1572 Acked 2571 Read 8 Done New Done true
7219 1573 Acked 2572 Write 9 Done New Done true
7220 1574 Acked 2573 Write 7 Done New Done true
7221 1575 Acked 2574 Read 9 Done New Done true
7222 1576 Acked 2575 Read 7 Done New Done true
7223 1577 Acked 2576 Write 4 Done New Done true
7224 1578 Acked 2577 Read 6 Done New Done true
7225 1579 Acked 2578 Write 9 Done New Done true
7226 1580 Acked 2579 Read 1 Done New Done true
7227 1581 Acked 2580 Read 5 Done New Done true
7228 1582 Acked 2581 Read 5 Done New Done true
7229 1583 Acked 2582 Read 5 Done New Done true
7230 1584 Acked 2583 Flush 0 Done New Done true
7231 1585 Acked 2584 Read 7 Done New Done true
7232 1586 Acked 2585 Read 10 Done New Done true
7233 1587 Acked 2586 Flush 0 Done New Done true
7234 1588 Acked 2587 Write 2 Done New Done true
7235 1589 Acked 2588 Read 10 Done New Done true
7236 1590 Acked 2589 Write 10 Done New Done true
7237 1591 Acked 2590 Write 3 Done New Done true
7238 1592 Acked 2591 Write 1 Done New Done true
7239 1593 Acked 2592 Read 7 Done New Done true
7240 1594 Acked 2593 Read 1 Done New Done true
7241 1595 Acked 2594 Read 1 Done New Done true
7242 1596 Acked 2595 Read 1 Done New Done true
7243 1597 Acked 2596 Write 8 Done New Done true
7244 1598 Acked 2597 Write 2 Done New Done true
7245 1599 Acked 2598 Flush 0 Done New Done true
7246 1600 Acked 2599 Write 8 Done New Done true
7247 1601 Acked 2600 Write 7 Done New Done true
7248 1602 Acked 2601 Write 7 Done New Done true
7249 1603 Acked 2602 Read 5 Done New Done true
7250 1604 Acked 2603 Read 1 Done New Done true
7251 1605 Acked 2604 Write 6 Done New Done true
7252 1606 Acked 2605 Write 6 Done New Done true
7253 1607 Acked 2606 Read 5 Done New Done true
7254 1608 Acked 2607 Read 6 Done New Done true
7255 1609 Acked 2608 Read 4 Done New Done true
7256 1610 Acked 2609 Write 7 Done New Done true
7257 1611 Acked 2610 Write 2 Done New Done true
7258 1612 Acked 2611 Read 2 Done New Done true
7259 1613 Acked 2612 Write 9 Done New Done true
7260 1614 Acked 2613 Read 10 Done New Done true
7261 1615 Acked 2614 Write 7 Done New Done true
7262 1616 Acked 2615 Write 4 Done New Done true
7263 1617 Acked 2616 Write 3 Done New Done true
7264 1618 Acked 2617 Write 10 Done New Done true
7265 1619 Acked 2618 Read 5 Done New Done true
7266 1620 Acked 2619 Read 9 Done New Done true
7267 1621 Acked 2620 Read 7 Done New Done true
7268 1622 Acked 2621 Read 2 Done New Done true
7269 1623 Acked 2622 Write 7 Done New Done true
7270 1624 Acked 2623 Flush 0 Done New Done true
7271 1625 Acked 2624 Read 6 Done New Done true
7272 1626 Acked 2625 Read 1 Done New Done true
7273 1627 Acked 2626 Read 7 Done New Done true
7274 1628 Acked 2627 Read 1 Done New Done true
7275 1629 Acked 2628 Flush 0 Done New Done true
7276 1630 Acked 2629 Read 4 Done New Done true
7277 1631 Acked 2630 Read 1 Done New Done true
7278 1632 Acked 2631 Write 3 Done New Done true
7279 1633 Acked 2632 Write 8 Done New Done true
7280 1634 Acked 2633 Read 7 Done New Done true
7281 1635 Acked 2634 Read 5 Done New Done true
7282 1636 Acked 2635 Write 4 Done New Done true
7283 1637 Acked 2636 Write 3 Done New Done true
7284 1638 Acked 2637 Read 3 Done New Done true
7285 1639 Acked 2638 Read 4 Done New Done true
7286 1640 Acked 2639 Read 2 Done New Done true
7287 1641 Acked 2640 Read 9 Done New Done true
7288 1642 Acked 2641 Write 7 Done New Done true
7289 1643 Acked 2642 Write 8 Done New Done true
7290 1644 Acked 2643 Write 2 Done New Done true
7291 1645 Acked 2644 Write 10 Done New Done true
7292 1646 Acked 2645 Read 6 Done New Done true
7293 1647 Acked 2646 Read 7 Done New Done true
7294 1648 Acked 2647 Write 10 Done New Done true
7295 1649 Acked 2648 Write 10 Done New Done true
7296 1650 Acked 2649 Read 10 Done New Done true
7297 1651 Acked 2650 Read 10 Done New Done true
7298 1652 Acked 2651 Write 9 Done New Done true
7299 1653 Acked 2652 Read 2 Done New Done true
7300 1654 Acked 2653 Read 10 Done New Done true
7301 1655 Acked 2654 Read 1 Done New Done true
7302 1656 Acked 2655 Read 6 Done New Done true
7303 1657 Acked 2656 Read 5 Done New Done true
7304 1658 Acked 2657 Read 1 Done New Done true
7305 1659 Acked 2658 Write 2 Done New Done true
7306 1660 Acked 2659 Write 7 Done New Done true
7307 1661 Acked 2660 Read 1 Done New Done true
7308 1662 Acked 2661 Read 1 Done New Done true
7309 1663 Acked 2662 Read 7 Done New Done true
7310 1664 Acked 2663 Write 4 Done New Done true
7311 1665 Acked 2664 Read 7 Done New Done true
7312 1666 Acked 2665 Read 8 Done New Done true
7313 1667 Acked 2666 Read 6 Done New Done true
7314 1668 Acked 2667 Read 6 Done New Done true
7315 1669 Acked 2668 Write 5 Done New Done true
7316 1670 Acked 2669 Read 10 Done New Done true
7317 1671 Acked 2670 Read 6 Done New Done true
7318 1672 Acked 2671 Write 10 Done New Done true
7319 1673 Acked 2672 Write 2 Done New Done true
7320 1674 Acked 2673 Read 2 Done New Done true
7321 1675 Acked 2674 Read 2 Done New Done true
7322 1676 Acked 2675 Read 8 Done New Done true
7323 1677 Acked 2676 Flush 0 Done New Done true
7324 1678 Acked 2677 Read 4 Done New Done true
7325 1679 Acked 2678 Read 4 Done New Done true
7326 1680 Acked 2679 Write 3 Done New Done true
7327 1681 Acked 2680 Write 6 Done New Done true
7328 1682 Acked 2681 Read 10 Done New Done true
7329 1683 Acked 2682 Read 9 Done New Done true
7330 1684 Acked 2683 Read 3 Done New Done true
7331 1685 Acked 2684 Write 1 Done New Done true
7332 1686 Acked 2685 Read 2 Done New Done true
7333 1687 Acked 2686 Write 4 Done New Done true
7334 1688 Acked 2687 Write 8 Done New Done true
7335 1689 Acked 2688 Read 6 Done New Done true
7336 1690 Acked 2689 Write 9 Done New Done true
7337 1691 Acked 2690 Write 6 Done New Done true
7338 1692 Acked 2691 Read 2 Done New Done true
7339 1693 Acked 2692 Flush 0 Done New Done true
7340 1694 Acked 2693 Read 8 Done New Done true
7341 1695 Acked 2694 Read 4 Done New Done true
7342 1696 Acked 2695 Read 10 Done New Done true
7343 1697 Acked 2696 Read 4 Done New Done true
7344 1698 Acked 2697 Write 1 Done New Done true
7345 1699 Acked 2698 Read 4 Done New Done true
7346 1700 Acked 2699 Write 4 Done New Done true
7347 1701 Acked 2700 Write 6 Done New Done true
7348 1702 Acked 2701 Write 6 Done New Done true
7349 1703 Acked 2702 Read 5 Done New Done true
7350 1704 Acked 2703 Write 2 Done New Done true
7351 1705 Acked 2704 Write 9 Done New Done true
7352 1706 Acked 2705 Read 1 Done New Done true
7353 1707 Acked 2706 Read 9 Done New Done true
7354 1708 Acked 2707 Read 10 Done New Done true
7355 1709 Acked 2708 Write 8 Done New Done true
7356 1710 Acked 2709 Write 6 Done New Done true
7357 1711 Acked 2710 Read 3 Done New Done true
7358 1712 Acked 2711 Write 1 Done New Done true
7359 1713 Acked 2712 Read 7 Done New Done true
7360 1714 Acked 2713 Read 9 Done New Done true
7361 1715 Acked 2714 Read 1 Done New Done true
7362 1716 Acked 2715 Write 3 Done New Done true
7363 1717 Acked 2716 Flush 0 Done New Done true
7364 1718 Acked 2717 Write 6 Done New Done true
7365 1719 Acked 2718 Read 5 Done New Done true
7366 1720 Acked 2719 Read 10 Done New Done true
7367 1721 Acked 2720 Read 5 Done New Done true
7368 1722 Acked 2721 Write 6 Done New Done true
7369 1723 Acked 2722 Write 1 Done New Done true
7370 1724 Acked 2723 Read 9 Done New Done true
7371 1725 Acked 2724 Write 1 Done New Done true
7372 1726 Acked 2725 Write 4 Done New Done true
7373 1727 Acked 2726 Flush 0 Done New Done true
7374 1728 Acked 2727 Write 8 Done New Done true
7375 1729 Acked 2728 Read 9 Done New Done true
7376 1730 Acked 2729 Write 8 Done New Done true
7377 1731 Acked 2730 Write 2 Done New Done true
7378 1732 Acked 2731 Flush 0 Done New Done true
7379 1733 Acked 2732 Read 1 Done New Done true
7380 1734 Acked 2733 Write 10 Done New Done true
7381 1735 Acked 2734 Read 2 Done New Done true
7382 1736 Acked 2735 Write 5 Done New Done true
7383 1737 Acked 2736 Read 1 Done New Done true
7384 1738 Acked 2737 Read 5 Done New Done true
7385 1739 Acked 2738 Read 6 Done New Done true
7386 1740 Acked 2739 Write 9 Done New Done true
7387 1741 Acked 2740 Write 9 Done New Done true
7388 1742 Acked 2741 Read 9 Done New Done true
7389 1743 Acked 2742 Write 8 Done New Done true
7390 1744 Acked 2743 Read 6 Done New Done true
7391 1745 Acked 2744 Read 1 Done New Done true
7392 1746 Acked 2745 Read 9 Done New Done true
7393 1747 Acked 2746 Read 2 Done New Done true
7394 1748 Acked 2747 Write 8 Done New Done true
7395 1749 Acked 2748 Read 1 Done New Done true
7396 1750 Acked 2749 Write 9 Done New Done true
7397 1751 Acked 2750 Write 2 Done New Done true
7398 1752 Acked 2751 Write 7 Done New Done true
7399 1753 Acked 2752 Read 8 Done New Done true
7400 1754 Acked 2753 Read 9 Done New Done true
7401 1755 Acked 2754 Write 8 Done New Done true
7402 1756 Acked 2755 Read 6 Done New Done true
7403 1757 Acked 2756 Write 4 Done New Done true
7404 1758 Acked 2757 Read 4 Done New Done true
7405 1759 Acked 2758 Write 2 Done New Done true
7406 1760 Acked 2759 Write 7 Done New Done true
7407 1761 Acked 2760 Read 2 Done New Done true
7408 1762 Acked 2761 Flush 0 Done New Done true
7409 1763 Acked 2762 Read 5 Done New Done true
7410 1764 Acked 2763 Write 8 Done New Done true
7411 1765 Acked 2764 Write 9 Done New Done true
7412 1766 Acked 2765 Read 10 Done New Done true
7413 1767 Acked 2766 Read 10 Done New Done true
7414 1768 Acked 2767 Flush 0 Done New Done true
7415 1769 Acked 2768 Flush 0 Done New Done true
7416 1770 Acked 2769 Read 1 Done New Done true
7417 1771 Acked 2770 Read 9 Done New Done true
7418 1772 Acked 2771 Read 7 Done New Done true
7419 1773 Acked 2772 Write 1 Done New Done true
7420 1774 Acked 2773 Write 3 Done New Done true
7421 1775 Acked 2774 Write 2 Done New Done true
7422 1776 Acked 2775 Write 1 Done New Done true
7423 1777 Acked 2776 Read 10 Done New Done true
7424 1778 Acked 2777 Write 1 Done New Done true
7425 1779 Acked 2778 Read 8 Done New Done true
7426 1780 Acked 2779 Write 2 Done New Done true
7427 1781 Acked 2780 Write 4 Done New Done true
7428 1782 Acked 2781 Write 5 Done New Done true
7429 1783 Acked 2782 Write 4 Done New Done true
7430 1784 Acked 2783 Read 4 Done New Done true
7431 1785 Acked 2784 Write 2 Done New Done true
7432 1786 Acked 2785 Write 3 Done New Done true
7433 1787 Acked 2786 Read 4 Done New Done true
7434 1788 Acked 2787 Write 2 Done New Done true
7435 1789 Acked 2788 Read 1 Done New Done true
7436 1790 Acked 2789 Read 7 Done New Done true
7437 1791 Acked 2790 Read 6 Done New Done true
7438 1792 Acked 2791 Read 5 Done New Done true
7439 1793 Acked 2792 Read 5 Done New Done true
7440 1794 Acked 2793 Write 5 Done New Done true
7441 1795 Acked 2794 Write 9 Done New Done true
7442 1796 Acked 2795 Read 9 Done New Done true
7443 1797 Acked 2796 Write 10 Done New Done true
7444 1798 Acked 2797 Write 4 Done New Done true
7445 1799 Acked 2798 Write 2 Done New Done true
7446 1800 Acked 2799 Read 3 Done New Done true
7447 1801 Acked 2800 Read 1 Done New Done true
7448 1802 Acked 2801 Flush 0 Done New Done true
7449 1803 Acked 2802 Write 8 Done New Done true
7450 1804 Acked 2803 Write 1 Done New Done true
7451 1805 Acked 2804 Read 9 Done New Done true
7452 1806 Acked 2805 Read 4 Done New Done true
7453 1807 Acked 2806 Flush 0 Done New Done true
7454 STATES DS:0 DS:1 DS:2 TOTAL
7455 New 0 301 0 301
7456 Sent 0 0 0 0
7457 Done 301 0 301 602
7458 Skipped 0 0 0 0
7459 Error 0 0 0 0
7460 Last Flush: 2806 2505 2806
7461 Downstairs last five completed: 2505 2504 2503 2502 2501
7462 Upstairs last five completed: 1807 1806 1805 1804 1803
7463 CLIENT: Up:0 ds:301 act:2
74642024-02-15T02:58:57.328ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
74652024-02-15T02:58:57.328ZINFOcrucible: ds_connection connected = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
74662024-02-15T02:58:57.329ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
74672024-02-15T02:58:57.329ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
74682024-02-15T02:58:57.329ZINFOcrucible: send last flush ID to this DS: 2505 = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
74692024-02-15T02:58:57.330ZINFOcrucible: Replied this last flush ID: 2505 = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
74702024-02-15T02:58:57.330ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
74712024-02-15T02:58:57.330ZINFOcrucible: [1] Transition from Offline to Replay = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
74722024-02-15T02:58:57.330ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
74732024-02-15T02:58:57.330ZINFOcrucible: [1] Transition from Replay to Active = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
7474 ----------------------------------------------------------------
7475 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
7476 Downstairs last five completed: 2806 2805 2804 2803 2802
7477 Upstairs last five completed: 1807 1806 1805 1804 1803
7478 CLIENT: Up:0 ds:0 act:3
7479 Replay: All jobs finished, all DS active.
7480 Main task has work to do, go find it
7481 got dsc Stop(0)
7482 stop 0
7483 Feb 15 02:59:00.653 INFO request completed, latency_us: 341, response_code: 204, [8810] Got stop action so:false kr:false
7484 uri: /stop/cid/0, method: POST, req_id: bfd0bd79-d5bd-4397-bb25-b420827e6e95, remote_addr: 127.0.0.1:54744, local_addr: 127.0.0.1:9998
74852024-02-15T02:59:00.654ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
74862024-02-15T02:59:00.654ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
74872024-02-15T02:59:00.654ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
74882024-02-15T02:59:00.654ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
74892024-02-15T02:59:00.654ZINFOcrucible: [0] client re-new 0 jobs since flush 2806 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
74902024-02-15T02:59:00.654ZINFOcrucible: client task is exiting = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
7491 Feb 15 02:59:00.654 INFO request completed, latency_us: 184, response_code: 200, uri: /state/cid/0, method: GET, req_id: eb78f1e8-5201-43c7-9dcf-7ef4b8f55faa, remote_addr: 127.0.0.1:54744, local_addr: 127.0.0.1:9998
7492 [8810] Exited with: ExitStatus(unix_wait_status(9))
7493 [8810][0] reports Exit
74942024-02-15T02:59:01.655ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
74952024-02-15T02:59:01.655ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
74962024-02-15T02:59:01.656ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
74972024-02-15T02:59:01.656ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
74982024-02-15T02:59:01.656ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
74992024-02-15T02:59:01.656ZINFOcrucible: [0] client re-new 0 jobs since flush 2806 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
75002024-02-15T02:59:01.656ZINFOcrucible: client task is exiting = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
75012024-02-15T02:59:02.656ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
75022024-02-15T02:59:02.657ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
75032024-02-15T02:59:02.657ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
75042024-02-15T02:59:02.657ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
75052024-02-15T02:59:02.657ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
75062024-02-15T02:59:02.657ZINFOcrucible: [0] client re-new 0 jobs since flush 2806 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
75072024-02-15T02:59:02.657ZINFOcrucible: client task is exiting = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
75082024-02-15T02:59:03.657ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
75092024-02-15T02:59:03.658ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
75102024-02-15T02:59:03.658ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
75112024-02-15T02:59:03.658ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
75122024-02-15T02:59:03.658ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
75132024-02-15T02:59:03.658ZINFOcrucible: [0] client re-new 0 jobs since flush 2806 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
75142024-02-15T02:59:03.658ZINFOcrucible: client task is exiting = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
7515 Feb 15 02:59:04.656 INFO request completed, latency_us: 195, response_code: 200, uri: /state/cid/0, method: GET, req_id: 1fe0f82d-ec71-4759-9f9e-edd6b905dc51, remote_addr: 127.0.0.1:54744, local_addr: 127.0.0.1:9998
7516 001/300 Read block 2881 len 32768
7517 002/300 Write block 1425 len 36864 data: 2 2 2 2 2 2 2 2 2
7518 003/300 Flush
75192024-02-15T02:59:04.659ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
75202024-02-15T02:59:04.659ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
75212024-02-15T02:59:04.659ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
75222024-02-15T02:59:04.659ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
75232024-02-15T02:59:04.659ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
75242024-02-15T02:59:04.659ZINFOcrucible: [0] client re-new 3 jobs since flush 2806 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
75252024-02-15T02:59:04.660ZINFOcrucible: client task is exiting = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
7526 004/300 Read block 3831 len 12288
7527 005/300 Flush
7528 006/300 Read block 4161 len 36864
7529 007/300 Read block 2903 len 20480
7530 008/300 Read block 4675 len 36864
7531 009/300 Read block 1109 len 16384
7532 010/300 Read block 413 len 4096
7533 011/300 Read block 1673 len 12288
7534 012/300 Write block 4779 len 16384 data: 3 3 3 2
7535 013/300 Flush
7536 014/300 Read block 41 len 24576
7537 015/300 Write block 4789 len 24576 data: 3 3 3 5 4 4
7538 016/300 Read block 3597 len 16384
7539 017/300 Flush
7540 018/300 Read block 879 len 8192
7541 019/300 Read block 566 len 16384
7542 020/300 Flush
7543 021/300 Read block 1526 len 16384
7544 022/300 Read block 1497 len 40960
7545 023/300 Read block 199 len 28672
7546 024/300 Read block 935 len 12288
7547 025/300 Write block 1524 len 40960 data: 3 3 4 5 5 5 5 5 3 2
7548 026/300 Write block 4550 len 24576 data: 3 3 3 3 3 2
7549 027/300 Read block 2665 len 40960
7550 028/300 Write block 774 len 20480 data: 2 2 2 2 2
7551 029/300 Flush
7552 030/300 Write block 1681 len 4096 data: 2
7553 031/300 Write block 3122 len 20480 data: 2 2 2 2 2
7554 032/300 Read block 1595 len 8192
7555 033/300 Write block 2506 len 4096 data: 2
7556 034/300 Write block 76 len 32768 data: 3 3 3 2 2 2 2 2
7557 035/300 Read block 1002 len 16384
7558 036/300 Read block 1203 len 28672
7559 037/300 Read block 3817 len 32768
7560 038/300 Write block 160 len 40960 data: 3 4 4 4 4 4 4 4 4 3
7561 039/300 Read block 85 len 8192
7562 040/300 Write block 4951 len 20480 data: 3 3 4 4 4
7563 041/300 Write block 2277 len 12288 data: 3 3 3
7564 042/300 Write block 3504 len 40960 data: 2 2 2 2 2 2 2 3 3 3
7565 043/300 Write block 4331 len 32768 data: 3 4 4 3 3 3 3 3
7566 044/300 Write block 3200 len 36864 data: 3 2 2 2 2 2 2 2 2
7567 045/300 Write block 1735 len 28672 data: 4 4 5 5 4 4 4
7568 046/300 Flush
7569 047/300 Read block 1392 len 24576
7570 048/300 Read block 356 len 8192
7571 049/300 Read block 3349 len 4096
7572 050/300 Write block 4885 len 12288 data: 3 3 3
7573 051/300 Read block 4077 len 40960
7574 052/300 Write block 3512 len 4096 data: 4
7575 053/300 Read block 4444 len 36864
7576 054/300 Write block 3494 len 24576 data: 4 4 3 3 3 3
7577 055/300 Write block 1581 len 32768 data: 3 3 4 4 3 3 4 4
7578 056/300 Read block 4575 len 24576
7579 057/300 Read block 2444 len 40960
7580 058/300 Write block 4636 len 40960 data: 2 2 2 3 4 4 4 3 3 3
7581 059/300 Write block 2088 len 4096 data: 3
7582 060/300 Read block 2590 len 32768
7583 061/300 Write block 4401 len 20480 data: 2 3 3 3 3
7584 062/300 Flush
7585 063/300 Read block 4765 len 20480
7586 064/300 Read block 2551 len 28672
7587 065/300 Read block 4305 len 32768
7588 066/300 Write block 2861 len 36864 data: 4 3 3 3 2 3 3 2 2
7589 067/300 Read block 4353 len 36864
7590 068/300 Read block 875 len 40960
7591 069/300 Read block 4180 len 20480
7592 070/300 Read block 3756 len 28672
7593 071/300 Write block 1639 len 36864 data: 2 2 4 5 5 5 5 5 4
7594 072/300 Write block 1309 len 32768 data: 2 2 2 2 2 2 2 3
7595 073/300 Read block 2163 len 32768
7596 074/300 Flush
7597 075/300 Write block 290 len 32768 data: 4 4 4 4 3 3 2 2
7598 076/300 Write block 4821 len 40960 data: 2 2 2 2 2 2 2 2 2 2
7599 077/300 Read block 1670 len 4096
7600 078/300 Flush
7601 079/300 Flush
7602 080/300 Read block 288 len 20480
7603 081/300 Write block 508 len 24576 data: 3 3 3 3 4 5
7604 082/300 Read block 1547 len 12288
7605 083/300 Flush
7606 084/300 Write block 565 len 8192 data: 2 3
7607 085/300 Read block 1433 len 40960
7608 086/300 Write block 574 len 24576 data: 2 2 2 2 3 3
7609 087/300 Write block 261 len 4096 data: 4
7610 088/300 Write block 3948 len 28672 data: 3 3 3 4 4 3 4
7611 089/300 Read block 541 len 40960
7612 090/300 Write block 3298 len 4096 data: 3
7613 091/300 Read block 1422 len 28672
7614 092/300 Read block 3396 len 36864
7615 093/300 Read block 2314 len 12288
7616 094/300 Read block 4042 len 4096
7617 095/300 Write block 3614 len 24576 data: 3 2 2 3 3 3
7618 096/300 Write block 2923 len 36864 data: 2 3 3 3 2 2 2 2 2
7619 097/300 Read block 4119 len 32768
7620 098/300 Read block 2972 len 12288
7621 099/300 Write block 2209 len 28672 data: 4 3 3 3 4 4 3
7622 100/300 Read block 1563 len 8192
7623 101/300 Read block 3171 len 12288
7624 102/300 Read block 3733 len 12288
7625 103/300 Flush
7626 104/300 Read block 2798 len 4096
7627 105/300 Write block 4418 len 40960 data: 4 3 3 2 2 3 3 3 3 3
7628 106/300 Flush
7629 107/300 Write block 2574 len 20480 data: 2 3 3 3 3
7630 108/300 Read block 2034 len 32768
7631 109/300 Read block 2183 len 4096
7632 110/300 Read block 4077 len 32768
7633 111/300 Read block 3431 len 4096
7634 112/300 Write block 4888 len 32768 data: 3 2 2 2 2 2 2 2
7635 113/300 Read block 2995 len 12288
7636 114/300 Read block 3565 len 8192
7637 115/300 Read block 43 len 12288
7638 116/300 Read block 252 len 24576
7639 117/300 Read block 3203 len 36864
7640 118/300 Read block 961 len 16384
7641 119/300 Read block 1791 len 32768
7642 120/300 Write block 4064 len 12288 data: 3 3 4
7643 121/300 Read block 4282 len 32768
7644 122/300 Write block 4960 len 40960 data: 2 2 2 2 2 3 2 2 2 2
7645 123/300 Write block 3735 len 4096 data: 4
7646 124/300 Write block 394 len 8192 data: 2 2
7647 125/300 Read block 11 len 24576
7648 126/300 Read block 3766 len 16384
7649 127/300 Read block 4117 len 28672
7650 128/300 Read block 4625 len 24576
7651 129/300 Write block 4661 len 8192 data: 2 2
7652 130/300 Read block 1481 len 4096
7653 131/300 Write block 3995 len 8192 data: 2 2
7654 132/300 Flush
7655 133/300 Read block 705 len 32768
7656 134/300 Read block 3684 len 8192
7657 135/300 Write block 3956 len 32768 data: 4 3 2 2 2 2 2 2
7658 136/300 Read block 4590 len 24576
7659 137/300 Read block 2666 len 32768
7660 138/300 Write block 3052 len 40960 data: 3 3 3 3 3 3 4 4 3 3
7661 139/300 Write block 1742 len 32768 data: 3 3 2 2 2 2 2 2
7662 140/300 Write block 670 len 40960 data: 3 3 3 3 2 2 2 2 2 2
7663 141/300 Write block 3833 len 24576 data: 3 2 2 2 2 2
7664 142/300 Read block 2026 len 20480
7665 143/300 Read block 2153 len 8192
7666 144/300 Read block 4253 len 8192
7667 145/300 Read block 2692 len 20480
7668 146/300 Read block 1753 len 16384
7669 147/300 Read block 1572 len 40960
7670 148/300 Write block 1139 len 16384 data: 4 3 3 2
7671 149/300 Read block 240 len 24576
7672 150/300 Flush
7673 151/300 Read block 2676 len 20480
7674 152/300 Read block 4365 len 12288
7675 153/300 Read block 2659 len 4096
7676 154/300 Read block 770 len 20480
7677 155/300 Write block 3416 len 40960 data: 2 2 2 2 2 2 2 2 2 2
7678 156/300 Read block 1615 len 28672
7679 157/300 Read block 700 len 40960
7680 158/300 Write block 4123 len 24576 data: 2 2 2 2 2 2
7681 159/300 Write block 305 len 28672 data: 4 3 3 3 3 3 3
7682 160/300 Write block 3075 len 4096 data: 2
7683 161/300 Read block 4247 len 12288
7684 162/300 Write block 1250 len 16384 data: 3 3 2 2
7685 163/300 Write block 506 len 16384 data: 2 3 4 4
7686 164/300 Read block 4479 len 40960
7687 165/300 Write block 1928 len 4096 data: 2
7688 166/300 Read block 581 len 12288
7689 167/300 Write block 2472 len 20480 data: 2 2 2 2 2
7690 168/300 Write block 2499 len 28672 data: 2 2 2 2 2 2 2
7691 169/300 Read block 4664 len 36864
7692 170/300 Write block 674 len 4096 data: 3
7693 171/300 Read block 214 len 24576
7694 172/300 Read block 1147 len 12288
7695 173/300 Write block 4681 len 40960 data: 4 3 2 2 3 3 3 3 3 3
7696 174/300 Write block 999 len 40960 data: 2 2 2 2 2 2 2 2 2 2
7697 175/300 Read block 797 len 28672
7698 176/300 Read block 117 len 16384
7699 177/300 Write block 4907 len 36864 data: 6 5 4 4 3 2 2 2 2
7700 178/300 Write block 566 len 28672 data: 4 3 3 3 4 3 3
7701 179/300 Write block 554 len 24576 data: 2 2 2 2 3 3
7702 180/300 Read block 272 len 16384
7703 181/300 Write block 1154 len 24576 data: 4 2 2 2 3 3
7704 182/300 Read block 3171 len 12288
7705 183/300 Read block 893 len 36864
7706 184/300 Read block 337 len 24576
7707 185/300 Read block 2093 len 12288
7708 186/300 Read block 1865 len 24576
7709 187/300 Read block 4790 len 16384
7710 188/300 Read block 2597 len 16384
7711 189/300 Read block 4238 len 20480
7712 190/300 Read block 1368 len 36864
7713 191/300 Read block 3325 len 20480
7714 192/300 Read block 1212 len 28672
7715 193/300 Write block 753 len 36864 data: 2 2 3 3 3 2 2 2 2
7716 194/300 Write block 979 len 12288 data: 4 4 3
7717 195/300 Flush
7718 196/300 Read block 2239 len 32768
7719 197/300 Read block 861 len 32768
7720 198/300 Read block 4487 len 24576
7721 199/300 Read block 1913 len 4096
7722 200/300 Write block 2444 len 20480 data: 2 2 3 3 4
7723 201/300 Write block 4900 len 12288 data: 2 2 2
7724 202/300 Read block 1812 len 12288
7725 203/300 Read block 1645 len 12288
7726 204/300 Write block 1285 len 40960 data: 2 2 2 2 2 3 3 3 3 2
7727 205/300 Write block 3398 len 24576 data: 3 3 3 3 4 4
7728 206/300 Read block 3985 len 24576
7729 207/300 Flush
7730 208/300 Read block 473 len 16384
7731 209/300 Read block 2715 len 8192
7732 210/300 Read block 3082 len 8192
7733 211/300 Read block 1936 len 20480
7734 212/300 Read block 3452 len 8192
7735 213/300 Read block 1536 len 32768
7736 214/300 Write block 361 len 16384 data: 2 3 3 2
7737 215/300 Write block 3874 len 12288 data: 2 2 2
7738 216/300 Flush
7739 217/300 Read block 4408 len 4096
7740 218/300 Read block 1859 len 16384
7741 219/300 Write block 900 len 4096 data: 2
7742 220/300 Write block 2042 len 4096 data: 2
7743 221/300 Write block 997 len 28672 data: 2 2 3 3 3 3 3
7744 222/300 Write block 1044 len 16384 data: 2 2 2 2
7745 223/300 Read block 2101 len 32768
7746 224/300 Write block 4215 len 16384 data: 4 4 4 4
7747 225/300 Read block 4249 len 36864
7748 226/300 Write block 3105 len 8192 data: 3 3
7749 227/300 Write block 3649 len 28672 data: 2 2 2 2 2 2 3
7750 228/300 Read block 3098 len 12288
7751 229/300 Write block 1254 len 32768 data: 2 2 2 2 2 2 2 2
7752 230/300 Flush
7753 231/300 Write block 2966 len 20480 data: 2 2 2 2 3
7754 232/300 Write block 3156 len 24576 data: 6 6 8 7 6 7
7755 233/300 Read block 2722 len 16384
7756 234/300 Read block 4351 len 24576
7757 235/300 Write block 3620 len 32768 data: 3 3 3 4 4 4 4 3
7758 236/300 Read block 4443 len 12288
7759 237/300 Write block 832 len 12288 data: 2 2 2
7760 238/300 Flush
7761 239/300 Read block 2853 len 28672
7762 240/300 Write block 4259 len 16384 data: 3 3 4 4
7763 241/300 Flush
7764 242/300 Write block 1567 len 12288 data: 4 4 4
7765 243/300 Read block 2104 len 28672
7766 244/300 Flush
7767 245/300 Read block 3014 len 40960
7768 246/300 Read block 2340 len 4096
7769 247/300 Write block 689 len 32768 data: 2 2 2 2 2 2 2 2
7770 248/300 Write block 1604 len 28672 data: 2 2 2 2 2 2 3
7771 249/300 Write block 3187 len 20480 data: 3 4 3 4 4
7772 250/300 Write block 318 len 12288 data: 2 2 2
7773 251/300 Read block 3454 len 4096
7774 252/300 Read block 26 len 32768
7775 253/300 Read block 3078 len 4096
7776 254/300 Write block 4113 len 28672 data: 4 4 4 4 4 4 2
7777 255/300 Read block 2377 len 8192
7778 256/300 Flush
7779 257/300 Write block 4628 len 12288 data: 2 3 2
7780 258/300 Write block 4682 len 12288 data: 4 3 3
7781 259/300 Write block 3618 len 4096 data: 4
7782 260/300 Read block 2073 len 4096
7783 261/300 Read block 2067 len 4096
7784 262/300 Read block 2275 len 28672
7785 263/300 Read block 2154 len 4096
7786 264/300 Read block 1211 len 28672
7787 265/300 Read block 90 len 20480
7788 266/300 Read block 4644 len 4096
7789 267/300 Read block 379 len 36864
7790 268/300 Read block 4179 len 20480
7791 269/300 Write block 3868 len 12288 data: 5 4 5
7792 270/300 Read block 1860 len 4096
7793 271/300 Write block 2304 len 28672 data: 2 2 2 2 3 2 2
7794 272/300 Write block 2239 len 36864 data: 3 4 4 4 3 3 2 2 2
7795 273/300 Read block 4266 len 16384
7796 274/300 Write block 2011 len 12288 data: 3 2 2
7797 275/300 Read block 4518 len 24576
7798 276/300 Write block 3083 len 8192 data: 3 3
7799 277/300 Write block 2894 len 8192 data: 2 2
7800 278/300 Write block 1255 len 32768 data: 3 3 3 3 3 3 3 2
7801 279/300 Write block 4969 len 40960 data: 3 2 2 2 2 2 2 2 3 3
7802 280/300 Flush
7803 281/300 Write block 4346 len 12288 data: 2 2 2
7804 282/300 Write block 4795 len 12288 data: 4 3 2
7805 283/300 Read block 4507 len 28672
7806 284/300 Read block 4222 len 8192
7807 285/300 Write block 2836 len 28672 data: 2 2 2 2 2 2 2
7808 286/300 Read block 4880 len 40960
7809 287/300 Write block 4723 len 4096 data: 3
7810 288/300 Write block 4319 len 4096 data: 2
7811 289/300 Read block 1410 len 32768
7812 290/300 Write block 2642 len 32768 data: 3 3 3 3 3 3 3 2
7813 291/300 Read block 1116 len 12288
7814 292/300 Write block 1504 len 12288 data: 2 2 2
7815 293/300 Write block 954 len 32768 data: 3 3 4 5 5 6 6 5
7816 294/300 Write block 1524 len 24576 data: 4 4 5 6 6 6
7817 295/300 Write block 199 len 28672 data: 3 2 2 2 2 2 3
7818 296/300 Read block 3756 len 24576
7819 297/300 Read block 2766 len 12288
7820 298/300 Flush
7821 299/300 Write block 2093 len 36864 data: 2 2 2 2 2 2 2 2 3
7822 300/300 Write block 4794 len 8192 data: 5 5
7823 Main task has work to do, go find it
7824 got dsc Start(0)
7825 start 0
7826 Feb 15 02:59:04.983 INFO request completed, latency_us: 221, response_code: 204, uri: /start/cid/0, method: POST, req_id: 942a5a2e-cb7e-403e-b82e-19b0d96ebf9b, remote_addr: 127.0.0.1:54744, local_addr: 127.0.0.1:9998
7827 Replay: started 0, returned:Ok(())
7828 ----------------------------------------------------------------
7829 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
7830 States: Offline Active Active
7831 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
7832 1808 Acked 2807 Read 8 New Done Done true
7833 1809 Acked 2808 Write 9 New Done Done true
7834 1810 Acked 2809 Flush 0 New Done Done true
7835 1811 Acked 2810 Read 3 New Done Done false
7836 1812 Acked 2811 Flush 0 New Done Done false
7837 1813 Acked 2812 Read 9 New Done Done false
7838 1814 Acked 2813 Read 5 New Done Done false
7839 1815 Acked 2814 Read 9 New Done Done false
7840 1816 Acked 2815 Read 4 New Done Done false
7841 1817 Acked 2816 Read 1 New Done Done false
7842 1818 Acked 2817 Read 3 New Done Done false
7843 1819 Acked 2818 Write 4 New Done Done false
7844 1820 Acked 2819 Flush 0 New Done Done false
7845 1821 Acked 2820 Read 6 New Done Done false
7846 1822 Acked 2821 Write 6 New Done Done false
7847 1823 Acked 2822 Read 4 New Done Done false
7848 1824 Acked 2823 Flush 0 New Done Done false
7849 1825 Acked 2824 Read 2 New Done Done false
7850 1826 Acked 2825 Read 4 New Done Done false
7851 1827 Acked 2826 Flush 0 New Done Done false
7852 1828 Acked 2827 Read 4 New Done Done false
7853 1829 Acked 2828 Read 10 New Done Done false
7854 1830 Acked 2829 Read 7 New Done Done false
7855 1831 Acked 2830 Read 3 New Done Done false
7856 1832 Acked 2831 Write 10 New Done Done false
7857 1833 Acked 2832 Write 6 New Done Done false
7858 1834 Acked 2833 Read 10 New Done Done false
7859 1835 Acked 2834 Write 5 New Done Done false
7860 1836 Acked 2835 Flush 0 New Done Done false
7861 1837 Acked 2836 Write 1 New Done Done false
7862 1838 Acked 2837 Write 5 New Done Done false
7863 1839 Acked 2838 Read 2 New Done Done false
7864 1840 Acked 2839 Write 1 New Done Done false
7865 1841 Acked 2840 Write 8 New Done Done false
7866 1842 Acked 2841 Read 4 New Done Done false
7867 1843 Acked 2842 Read 7 New Done Done false
7868 1844 Acked 2843 Read 8 New Done Done false
7869 1845 Acked 2844 Write 10 New Done Done false
7870 1846 Acked 2845 Read 2 New Done Done false
7871 1847 Acked 2846 Write 5 New Done Done false
7872 1848 Acked 2847 Write 3 New Done Done false
7873 1849 Acked 2848 Write 10 New Done Done false
7874 1850 Acked 2849 Write 8 New Done Done false
7875 1851 Acked 2850 Write 9 New Done Done false
7876 1852 Acked 2851 Write 7 New Done Done false
7877 1853 Acked 2852 Flush 0 New Done Done false
7878 1854 Acked 2853 Read 6 New Done Done false
7879 1855 Acked 2854 Read 2 New Done Done false
7880 1856 Acked 2855 Read 1 New Done Done false
7881 1857 Acked 2856 Flush 0 New Done Done false
7882 1858 Acked 2857 Write 3 New Done Done false
7883 1859 Acked 2858 Read 10 New Done Done false
7884 1860 Acked 2859 Write 1 New Done Done false
7885 1861 Acked 2860 Read 9 New Done Done false
7886 1862 Acked 2861 Write 6 New Done Done false
7887 1863 Acked 2862 Write 8 New Done Done false
7888 1864 Acked 2863 Read 6 New Done Done false
7889 1865 Acked 2864 Read 10 New Done Done false
7890 1866 Acked 2865 Write 10 New Done Done false
7891 1867 Acked 2866 Write 1 New Done Done false
7892 1868 Acked 2867 Read 8 New Done Done false
7893 1869 Acked 2868 Write 5 New Done Done false
7894 1870 Acked 2869 Flush 0 New Done Done false
7895 1871 Acked 2870 Read 5 New Done Done false
7896 1872 Acked 2871 Read 7 New Done Done false
7897 1873 Acked 2872 Read 8 New Done Done false
7898 1874 Acked 2873 Write 9 New Done Done false
7899 1875 Acked 2874 Read 9 New Done Done false
7900 1876 Acked 2875 Read 10 New Done Done false
7901 1877 Acked 2876 Read 5 New Done Done false
7902 1878 Acked 2877 Read 7 New Done Done false
7903 1879 Acked 2878 Write 9 New Done Done false
7904 1880 Acked 2879 Write 8 New Done Done false
7905 1881 Acked 2880 Read 8 New Done Done false
7906 1882 Acked 2881 Flush 0 New Done Done false
7907 1883 Acked 2882 Write 8 New Done Done false
7908 1884 Acked 2883 Write 10 New Done Done false
7909 1885 Acked 2884 Read 1 New Done Done false
7910 1886 Acked 2885 Flush 0 New Done Done false
7911 1887 Acked 2886 Flush 0 New Done Done false
7912 1888 Acked 2887 Read 5 New Done Done false
7913 1889 Acked 2888 Write 6 New Done Done false
7914 1890 Acked 2889 Read 3 New Done Done false
7915 1891 Acked 2890 Flush 0 New Done Done false
7916 1892 Acked 2891 Write 2 New Done Done false
7917 1893 Acked 2892 Read 10 New Done Done false
7918 1894 Acked 2893 Write 6 New Done Done false
7919 1895 Acked 2894 Write 1 New Done Done false
7920 1896 Acked 2895 Write 7 New Done Done false
7921 1897 Acked 2896 Read 10 New Done Done false
7922 1898 Acked 2897 Write 1 New Done Done false
7923 1899 Acked 2898 Read 7 New Done Done false
7924 1900 Acked 2899 Read 9 New Done Done false
7925 1901 Acked 2900 Read 3 New Done Done false
7926 1902 Acked 2901 Read 1 New Done Done false
7927 1903 Acked 2902 Write 6 New Done Done false
7928 1904 Acked 2903 Write 9 New Done Done false
7929 1905 Acked 2904 Read 8 New Done Done false
7930 1906 Acked 2905 Read 3 New Done Done false
7931 1907 Acked 2906 Write 7 New Done Done false
7932 1908 Acked 2907 Read 2 New Done Done false
7933 1909 Acked 2908 Read 3 New Done Done false
7934 1910 Acked 2909 Read 3 New Done Done false
7935 1911 Acked 2910 Flush 0 New Done Done false
7936 1912 Acked 2911 Read 1 New Done Done false
7937 1913 Acked 2912 Write 10 New Done Done false
7938 1914 Acked 2913 Flush 0 New Done Done false
7939 1915 Acked 2914 Write 5 New Done Done false
7940 1916 Acked 2915 Read 8 New Done Done false
7941 1917 Acked 2916 Read 1 New Done Done false
7942 1918 Acked 2917 Read 8 New Done Done false
7943 1919 Acked 2918 Read 1 New Done Done false
7944 1920 Acked 2919 Write 8 New Done Done false
7945 1921 Acked 2920 Read 3 New Done Done false
7946 1922 Acked 2921 Read 2 New Done Done false
7947 1923 Acked 2922 Read 3 New Done Done false
7948 1924 Acked 2923 Read 6 New Done Done false
7949 1925 Acked 2924 Read 9 New Done Done false
7950 1926 Acked 2925 Read 4 New Done Done false
7951 1927 Acked 2926 Read 8 New Done Done false
7952 1928 Acked 2927 Write 3 New Done Done false
7953 1929 Acked 2928 Read 8 New Done Done false
7954 1930 Acked 2929 Write 10 New Done Done false
7955 1931 Acked 2930 Write 1 New Done Done false
7956 1932 Acked 2931 Write 2 New Done Done false
7957 1933 Acked 2932 Read 6 New Done Done false
7958 1934 Acked 2933 Read 4 New Done Done false
7959 1935 Acked 2934 Read 7 New Done Done false
7960 1936 Acked 2935 Read 6 New Done Done false
7961 1937 Acked 2936 Write 2 New Done Done false
7962 1938 Acked 2937 Read 1 New Done Done false
7963 1939 Acked 2938 Write 2 New Done Done false
7964 1940 Acked 2939 Flush 0 New Done Done false
7965 1941 Acked 2940 Read 8 New Done Done false
7966 1942 Acked 2941 Read 2 New Done Done false
7967 1943 Acked 2942 Write 8 New Done Done false
7968 1944 Acked 2943 Read 6 New Done Done false
7969 1945 Acked 2944 Read 8 New Done Done false
7970 1946 Acked 2945 Write 10 New Done Done false
7971 1947 Acked 2946 Write 8 New Done Done false
7972 1948 Acked 2947 Write 10 New Done Done false
7973 1949 Acked 2948 Write 6 New Done Done false
7974 1950 Acked 2949 Read 5 New Done Done false
7975 1951 Acked 2950 Read 2 New Done Done false
7976 1952 Acked 2951 Read 2 New Done Done false
7977 1953 Acked 2952 Read 5 New Done Done false
7978 1954 Acked 2953 Read 4 New Done Done false
7979 1955 Acked 2954 Read 10 New Done Done false
7980 1956 Acked 2955 Write 4 New Done Done false
7981 1957 Acked 2956 Read 6 New Done Done false
7982 1958 Acked 2957 Flush 0 New Done Done false
7983 1959 Acked 2958 Read 5 New Done Done false
7984 1960 Acked 2959 Read 3 New Done Done false
7985 1961 Acked 2960 Read 1 New Done Done false
7986 1962 Acked 2961 Read 5 New Done Done false
7987 1963 Acked 2962 Write 10 New Done Done false
7988 1964 Acked 2963 Read 7 New Done Done false
7989 1965 Acked 2964 Read 10 New Done Done false
7990 1966 Acked 2965 Write 6 New Done Done false
7991 1967 Acked 2966 Write 7 New Done Done false
7992 1968 Acked 2967 Write 1 New Done Done false
7993 1969 Acked 2968 Read 3 New Done Done false
7994 1970 Acked 2969 Write 4 New Done Done false
7995 1971 Acked 2970 Write 4 New Done Done false
7996 1972 Acked 2971 Read 10 New Done Done false
7997 1973 Acked 2972 Write 1 New Done Done false
7998 1974 Acked 2973 Read 3 New Done Done false
7999 1975 Acked 2974 Write 5 New Done Done false
8000 1976 Acked 2975 Write 7 New Done Done false
8001 1977 Acked 2976 Read 9 New Done Done false
8002 1978 Acked 2977 Write 1 New Done Done false
8003 1979 Acked 2978 Read 6 New Done Done false
8004 1980 Acked 2979 Read 3 New Done Done false
8005 1981 Acked 2980 Write 10 New Done Done false
8006 1982 Acked 2981 Write 10 New Done Done false
8007 1983 Acked 2982 Read 7 New Done Done false
8008 1984 Acked 2983 Read 4 New Done Done false
8009 1985 Acked 2984 Write 9 New Done Done false
8010 1986 Acked 2985 Write 7 New Done Done false
8011 1987 Acked 2986 Write 6 New Done Done false
8012 1988 Acked 2987 Read 4 New Done Done false
8013 1989 Acked 2988 Write 6 New Done Done false
8014 1990 Acked 2989 Read 3 New Done Done false
8015 1991 Acked 2990 Read 9 New Done Done false
8016 1992 Acked 2991 Read 6 New Done Done false
8017 1993 Acked 2992 Read 3 New Done Done false
8018 1994 Acked 2993 Read 6 New Done Done false
8019 1995 Acked 2994 Read 4 New Done Done false
8020 1996 Acked 2995 Read 4 New Done Done false
8021 1997 Acked 2996 Read 5 New Done Done false
8022 1998 Acked 2997 Read 9 New Done Done false
8023 1999 Acked 2998 Read 5 New Done Done false
8024 2000 Acked 2999 Read 7 New Done Done false
8025 2001 Acked 3000 Write 9 New Done Done false
8026 2002 Acked 3001 Write 3 New Done Done false
8027 2003 Acked 3002 Flush 0 New Done Done false
8028 2004 Acked 3003 Read 8 New Done Done false
8029 2005 Acked 3004 Read 8 New Done Done false
8030 2006 Acked 3005 Read 6 New Done Done false
8031 2007 Acked 3006 Read 1 New Done Done false
8032 2008 Acked 3007 Write 5 New Done Done false
8033 2009 Acked 3008 Write 3 New Done Done false
8034 2010 Acked 3009 Read 3 New Done Done false
8035 2011 Acked 3010 Read 3 New Done Done false
8036 2012 Acked 3011 Write 10 New Done Done false
8037 2013 Acked 3012 Write 6 New Done Done false
8038 2014 Acked 3013 Read 6 New Done Done false
8039 2015 Acked 3014 Flush 0 New Done Done false
8040 2016 Acked 3015 Read 4 New Done Done false
8041 2017 Acked 3016 Read 2 New Done Done false
8042 2018 Acked 3017 Read 2 New Done Done false
8043 2019 Acked 3018 Read 5 New Done Done false
8044 2020 Acked 3019 Read 2 New Done Done false
8045 2021 Acked 3020 Read 8 New Done Done false
8046 2022 Acked 3021 Write 4 New Done Done false
8047 2023 Acked 3022 Write 3 New Done Done false
8048 2024 Acked 3023 Flush 0 New Done Done false
8049 2025 Acked 3024 Read 1 New Done Done false
8050 2026 Acked 3025 Read 4 New Done Done false
8051 2027 Acked 3026 Write 1 New Done Done false
8052 2028 Acked 3027 Write 1 New Done Done false
8053 2029 Acked 3028 Write 7 New Done Done false
8054 2030 Acked 3029 Write 4 New Done Done false
8055 2031 Acked 3030 Read 8 New Done Done false
8056 2032 Acked 3031 Write 4 New Done Done false
8057 2033 Acked 3032 Read 9 New Done Done false
8058 2034 Acked 3033 Write 2 New Done Done false
8059 2035 Acked 3034 Write 7 New Done Done false
8060 2036 Acked 3035 Read 3 New Done Done false
8061 2037 Acked 3036 Write 8 New Done Done false
8062 2038 Acked 3037 Flush 0 New Done Done false
8063 2039 Acked 3038 Write 5 New Done Done false
8064 2040 Acked 3039 Write 6 New Done Done false
8065 2041 Acked 3040 Read 4 New Done Done false
8066 2042 Acked 3041 Read 6 New Done Done false
8067 2043 Acked 3042 Write 8 New Done Done false
8068 2044 Acked 3043 Read 3 New Done Done false
8069 2045 Acked 3044 Write 3 New Done Done false
8070 2046 Acked 3045 Flush 0 New Done Done false
8071 2047 Acked 3046 Read 7 New Done Done false
8072 2048 Acked 3047 Write 4 New Done Done false
8073 2049 Acked 3048 Flush 0 New Done Done false
8074 2050 Acked 3049 Write 3 New Done Done false
8075 2051 Acked 3050 Read 7 New Done Done false
8076 2052 Acked 3051 Flush 0 New Done Done false
8077 2053 Acked 3052 Read 10 New Done Done false
8078 2054 Acked 3053 Read 1 New Done Done false
8079 2055 Acked 3054 Write 8 New Done Done false
8080 2056 Acked 3055 Write 7 New Done Done false
8081 2057 Acked 3056 Write 5 New Done Done false
8082 2058 Acked 3057 Write 3 New Done Done false
8083 2059 Acked 3058 Read 1 New Done Done false
8084 2060 Acked 3059 Read 8 New Done Done false
8085 2061 Acked 3060 Read 1 New Done Done false
8086 2062 Acked 3061 Write 7 New Done Done false
8087 2063 Acked 3062 Read 2 New Done Done false
8088 2064 Acked 3063 Flush 0 New Done Done false
8089 2065 Acked 3064 Write 3 New Done Done false
8090 2066 Acked 3065 Write 3 New Done Done false
8091 2067 Acked 3066 Write 1 New Done Done false
8092 2068 Acked 3067 Read 1 New Done Done false
8093 2069 Acked 3068 Read 1 New Done Done false
8094 2070 Acked 3069 Read 7 New Done Done false
8095 2071 Acked 3070 Read 1 New Done Done false
8096 2072 Acked 3071 Read 7 New Done Done false
8097 2073 Acked 3072 Read 5 New Done Done false
8098 2074 Acked 3073 Read 1 New Done Done false
8099 2075 Acked 3074 Read 9 New Done Done false
8100 2076 Acked 3075 Read 5 New Done Done false
8101 2077 Acked 3076 Write 3 New Done Done false
8102 2078 Acked 3077 Read 1 New Done Done false
8103 2079 Acked 3078 Write 7 New Done Done false
8104 2080 Acked 3079 Write 9 New Done Done false
8105 2081 Acked 3080 Read 4 New Done Done false
8106 2082 Acked 3081 Write 3 New Done Done false
8107 2083 Acked 3082 Read 6 New Done Done false
8108 2084 Acked 3083 Write 2 New Done Done false
8109 2085 Acked 3084 Write 2 New Done Done false
8110 2086 Acked 3085 Write 8 New Done Done false
8111 2087 Acked 3086 Write 10 New Done Done false
8112 2088 Acked 3087 Flush 0 New Done Done false
8113 2089 Acked 3088 Write 3 New Done Done false
8114 2090 Acked 3089 Write 3 New Done Done false
8115 2091 Acked 3090 Read 7 New Done Done false
8116 2092 Acked 3091 Read 2 New Done Done false
8117 2093 Acked 3092 Write 7 New Done Done false
8118 2094 Acked 3093 Read 10 New Done Done false
8119 2095 Acked 3094 Write 1 New Done Done false
8120 2096 Acked 3095 Write 1 New Done Done false
8121 2097 Acked 3096 Read 8 New Done Done false
8122 2098 Acked 3097 Write 8 New Done Done false
8123 2099 Acked 3098 Read 3 New Done Done false
8124 2100 Acked 3099 Write 3 New Done Done false
8125 2101 Acked 3100 Write 8 New Done Done false
8126 2102 Acked 3101 Write 6 New Done Done false
8127 2103 Acked 3102 Write 7 New Done Done false
8128 2104 Acked 3103 Read 6 New Done Done false
8129 2105 Acked 3104 Read 3 New Done Done false
8130 2106 Acked 3105 Flush 0 New Done Done false
8131 2107 Acked 3106 Write 9 New Done Done false
8132 2108 Acked 3107 Write 2 New Done Done false
8133 STATES DS:0 DS:1 DS:2 TOTAL
8134 New 301 0 0 301
8135 Sent 0 0 0 0
8136 Done 0 301 301 602
8137 Skipped 0 0 0 0
8138 Error 0 0 0 0
8139 Last Flush: 2806 3105 3105
8140 Downstairs last five completed: 2806 2805 2804 2803 2802
8141 Upstairs last five completed: 2108 2107 2106 2105 2104
8142 CLIENT: Up:0 ds:301 act:2
81432024-02-15T02:59:05.660ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
81442024-02-15T02:59:05.660ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
81452024-02-15T02:59:05.660ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
81462024-02-15T02:59:05.660ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
81472024-02-15T02:59:05.660ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
81482024-02-15T02:59:05.660ZINFOcrucible: [0] client re-new 302 jobs since flush 2806 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
81492024-02-15T02:59:05.660ZINFOcrucible: client task is exiting = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
81502024-02-15T02:59:06.661ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
81512024-02-15T02:59:06.661ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
81522024-02-15T02:59:06.661ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
81532024-02-15T02:59:06.661ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
81542024-02-15T02:59:06.661ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
81552024-02-15T02:59:06.661ZINFOcrucible: [0] client re-new 302 jobs since flush 2806 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
81562024-02-15T02:59:06.662ZINFOcrucible: client task is exiting = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
81572024-02-15T02:59:07.662ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
81582024-02-15T02:59:07.662ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
81592024-02-15T02:59:07.662ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
81602024-02-15T02:59:07.662ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
81612024-02-15T02:59:07.662ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
81622024-02-15T02:59:07.662ZINFOcrucible: [0] client re-new 302 jobs since flush 2806 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
81632024-02-15T02:59:07.662ZINFOcrucible: client task is exiting = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
8164 [8810] got start action
8165 [8810] I am going to restart
8166 Starting downstairs at port 8810
8167 Make output file at "/tmp/dsc/downstairs-8810.txt"
8168 [8810][0] reports Starting
81692024-02-15T02:59:08.663ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
81702024-02-15T02:59:08.663ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
81712024-02-15T02:59:08.663ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
81722024-02-15T02:59:08.663ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
81732024-02-15T02:59:08.663ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
81742024-02-15T02:59:08.663ZINFOcrucible: [0] client re-new 302 jobs since flush 2806 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
81752024-02-15T02:59:08.663ZINFOcrucible: client task is exiting = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
8176 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1174)
8177 [8810][0] reports Running
8178 ----------------------------------------------------------------
8179 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
8180 States: Offline Active Active
8181 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
8182 1808 Acked 2807 Read 8 New Done Done true
8183 1809 Acked 2808 Write 9 New Done Done true
8184 1810 Acked 2809 Flush 0 New Done Done true
8185 1811 Acked 2810 Read 3 New Done Done true
8186 1812 Acked 2811 Flush 0 New Done Done true
8187 1813 Acked 2812 Read 9 New Done Done true
8188 1814 Acked 2813 Read 5 New Done Done true
8189 1815 Acked 2814 Read 9 New Done Done true
8190 1816 Acked 2815 Read 4 New Done Done true
8191 1817 Acked 2816 Read 1 New Done Done true
8192 1818 Acked 2817 Read 3 New Done Done true
8193 1819 Acked 2818 Write 4 New Done Done true
8194 1820 Acked 2819 Flush 0 New Done Done true
8195 1821 Acked 2820 Read 6 New Done Done true
8196 1822 Acked 2821 Write 6 New Done Done true
8197 1823 Acked 2822 Read 4 New Done Done true
8198 1824 Acked 2823 Flush 0 New Done Done true
8199 1825 Acked 2824 Read 2 New Done Done true
8200 1826 Acked 2825 Read 4 New Done Done true
8201 1827 Acked 2826 Flush 0 New Done Done true
8202 1828 Acked 2827 Read 4 New Done Done true
8203 1829 Acked 2828 Read 10 New Done Done true
8204 1830 Acked 2829 Read 7 New Done Done true
8205 1831 Acked 2830 Read 3 New Done Done true
8206 1832 Acked 2831 Write 10 New Done Done true
8207 1833 Acked 2832 Write 6 New Done Done true
8208 1834 Acked 2833 Read 10 New Done Done true
8209 1835 Acked 2834 Write 5 New Done Done true
8210 1836 Acked 2835 Flush 0 New Done Done true
8211 1837 Acked 2836 Write 1 New Done Done true
8212 1838 Acked 2837 Write 5 New Done Done true
8213 1839 Acked 2838 Read 2 New Done Done true
8214 1840 Acked 2839 Write 1 New Done Done true
8215 1841 Acked 2840 Write 8 New Done Done true
8216 1842 Acked 2841 Read 4 New Done Done true
8217 1843 Acked 2842 Read 7 New Done Done true
8218 1844 Acked 2843 Read 8 New Done Done true
8219 1845 Acked 2844 Write 10 New Done Done true
8220 1846 Acked 2845 Read 2 New Done Done true
8221 1847 Acked 2846 Write 5 New Done Done true
8222 1848 Acked 2847 Write 3 New Done Done true
8223 1849 Acked 2848 Write 10 New Done Done true
8224 1850 Acked 2849 Write 8 New Done Done true
8225 1851 Acked 2850 Write 9 New Done Done true
8226 1852 Acked 2851 Write 7 New Done Done true
8227 1853 Acked 2852 Flush 0 New Done Done true
8228 1854 Acked 2853 Read 6 New Done Done true
8229 1855 Acked 2854 Read 2 New Done Done true
8230 1856 Acked 2855 Read 1 New Done Done true
8231 1857 Acked 2856 Flush 0 New Done Done true
8232 1858 Acked 2857 Write 3 New Done Done true
8233 1859 Acked 2858 Read 10 New Done Done true
8234 1860 Acked 2859 Write 1 New Done Done true
8235 1861 Acked 2860 Read 9 New Done Done true
8236 1862 Acked 2861 Write 6 New Done Done true
8237 1863 Acked 2862 Write 8 New Done Done true
8238 1864 Acked 2863 Read 6 New Done Done true
8239 1865 Acked 2864 Read 10 New Done Done true
8240 1866 Acked 2865 Write 10 New Done Done true
8241 1867 Acked 2866 Write 1 New Done Done true
8242 1868 Acked 2867 Read 8 New Done Done true
8243 1869 Acked 2868 Write 5 New Done Done true
8244 1870 Acked 2869 Flush 0 New Done Done true
8245 1871 Acked 2870 Read 5 New Done Done true
8246 1872 Acked 2871 Read 7 New Done Done true
8247 1873 Acked 2872 Read 8 New Done Done true
8248 1874 Acked 2873 Write 9 New Done Done true
8249 1875 Acked 2874 Read 9 New Done Done true
8250 1876 Acked 2875 Read 10 New Done Done true
8251 1877 Acked 2876 Read 5 New Done Done true
8252 1878 Acked 2877 Read 7 New Done Done true
8253 1879 Acked 2878 Write 9 New Done Done true
8254 1880 Acked 2879 Write 8 New Done Done true
8255 1881 Acked 2880 Read 8 New Done Done true
8256 1882 Acked 2881 Flush 0 New Done Done true
8257 1883 Acked 2882 Write 8 New Done Done true
8258 1884 Acked 2883 Write 10 New Done Done true
8259 1885 Acked 2884 Read 1 New Done Done true
8260 1886 Acked 2885 Flush 0 New Done Done true
8261 1887 Acked 2886 Flush 0 New Done Done true
8262 1888 Acked 2887 Read 5 New Done Done true
8263 1889 Acked 2888 Write 6 New Done Done true
8264 1890 Acked 2889 Read 3 New Done Done true
8265 1891 Acked 2890 Flush 0 New Done Done true
8266 1892 Acked 2891 Write 2 New Done Done true
8267 1893 Acked 2892 Read 10 New Done Done true
8268 1894 Acked 2893 Write 6 New Done Done true
8269 1895 Acked 2894 Write 1 New Done Done true
8270 1896 Acked 2895 Write 7 New Done Done true
8271 1897 Acked 2896 Read 10 New Done Done true
8272 1898 Acked 2897 Write 1 New Done Done true
8273 1899 Acked 2898 Read 7 New Done Done true
8274 1900 Acked 2899 Read 9 New Done Done true
8275 1901 Acked 2900 Read 3 New Done Done true
8276 1902 Acked 2901 Read 1 New Done Done true
8277 1903 Acked 2902 Write 6 New Done Done true
8278 1904 Acked 2903 Write 9 New Done Done true
8279 1905 Acked 2904 Read 8 New Done Done true
8280 1906 Acked 2905 Read 3 New Done Done true
8281 1907 Acked 2906 Write 7 New Done Done true
8282 1908 Acked 2907 Read 2 New Done Done true
8283 1909 Acked 2908 Read 3 New Done Done true
8284 1910 Acked 2909 Read 3 New Done Done true
8285 1911 Acked 2910 Flush 0 New Done Done true
8286 1912 Acked 2911 Read 1 New Done Done true
8287 1913 Acked 2912 Write 10 New Done Done true
8288 1914 Acked 2913 Flush 0 New Done Done true
8289 1915 Acked 2914 Write 5 New Done Done true
8290 1916 Acked 2915 Read 8 New Done Done true
8291 1917 Acked 2916 Read 1 New Done Done true
8292 1918 Acked 2917 Read 8 New Done Done true
8293 1919 Acked 2918 Read 1 New Done Done true
8294 1920 Acked 2919 Write 8 New Done Done true
8295 1921 Acked 2920 Read 3 New Done Done true
8296 1922 Acked 2921 Read 2 New Done Done true
8297 1923 Acked 2922 Read 3 New Done Done true
8298 1924 Acked 2923 Read 6 New Done Done true
8299 1925 Acked 2924 Read 9 New Done Done true
8300 1926 Acked 2925 Read 4 New Done Done true
8301 1927 Acked 2926 Read 8 New Done Done true
8302 1928 Acked 2927 Write 3 New Done Done true
8303 1929 Acked 2928 Read 8 New Done Done true
8304 1930 Acked 2929 Write 10 New Done Done true
8305 1931 Acked 2930 Write 1 New Done Done true
8306 1932 Acked 2931 Write 2 New Done Done true
8307 1933 Acked 2932 Read 6 New Done Done true
8308 1934 Acked 2933 Read 4 New Done Done true
8309 1935 Acked 2934 Read 7 New Done Done true
8310 1936 Acked 2935 Read 6 New Done Done true
8311 1937 Acked 2936 Write 2 New Done Done true
8312 1938 Acked 2937 Read 1 New Done Done true
8313 1939 Acked 2938 Write 2 New Done Done true
8314 1940 Acked 2939 Flush 0 New Done Done true
8315 1941 Acked 2940 Read 8 New Done Done true
8316 1942 Acked 2941 Read 2 New Done Done true
8317 1943 Acked 2942 Write 8 New Done Done true
8318 1944 Acked 2943 Read 6 New Done Done true
8319 1945 Acked 2944 Read 8 New Done Done true
8320 1946 Acked 2945 Write 10 New Done Done true
8321 1947 Acked 2946 Write 8 New Done Done true
8322 1948 Acked 2947 Write 10 New Done Done true
8323 1949 Acked 2948 Write 6 New Done Done true
8324 1950 Acked 2949 Read 5 New Done Done true
8325 1951 Acked 2950 Read 2 New Done Done true
8326 1952 Acked 2951 Read 2 New Done Done true
8327 1953 Acked 2952 Read 5 New Done Done true
8328 1954 Acked 2953 Read 4 New Done Done true
8329 1955 Acked 2954 Read 10 New Done Done true
8330 1956 Acked 2955 Write 4 New Done Done true
8331 1957 Acked 2956 Read 6 New Done Done true
8332 1958 Acked 2957 Flush 0 New Done Done true
8333 1959 Acked 2958 Read 5 New Done Done true
8334 1960 Acked 2959 Read 3 New Done Done true
8335 1961 Acked 2960 Read 1 New Done Done true
8336 1962 Acked 2961 Read 5 New Done Done true
8337 1963 Acked 2962 Write 10 New Done Done true
8338 1964 Acked 2963 Read 7 New Done Done true
8339 1965 Acked 2964 Read 10 New Done Done true
8340 1966 Acked 2965 Write 6 New Done Done true
8341 1967 Acked 2966 Write 7 New Done Done true
8342 1968 Acked 2967 Write 1 New Done Done true
8343 1969 Acked 2968 Read 3 New Done Done true
8344 1970 Acked 2969 Write 4 New Done Done true
8345 1971 Acked 2970 Write 4 New Done Done true
8346 1972 Acked 2971 Read 10 New Done Done true
8347 1973 Acked 2972 Write 1 New Done Done true
8348 1974 Acked 2973 Read 3 New Done Done true
8349 1975 Acked 2974 Write 5 New Done Done true
8350 1976 Acked 2975 Write 7 New Done Done true
8351 1977 Acked 2976 Read 9 New Done Done true
8352 1978 Acked 2977 Write 1 New Done Done true
8353 1979 Acked 2978 Read 6 New Done Done true
8354 1980 Acked 2979 Read 3 New Done Done true
8355 1981 Acked 2980 Write 10 New Done Done true
8356 1982 Acked 2981 Write 10 New Done Done true
8357 1983 Acked 2982 Read 7 New Done Done true
8358 1984 Acked 2983 Read 4 New Done Done true
8359 1985 Acked 2984 Write 9 New Done Done true
8360 1986 Acked 2985 Write 7 New Done Done true
8361 1987 Acked 2986 Write 6 New Done Done true
8362 1988 Acked 2987 Read 4 New Done Done true
8363 1989 Acked 2988 Write 6 New Done Done true
8364 1990 Acked 2989 Read 3 New Done Done true
8365 1991 Acked 2990 Read 9 New Done Done true
8366 1992 Acked 2991 Read 6 New Done Done true
8367 1993 Acked 2992 Read 3 New Done Done true
8368 1994 Acked 2993 Read 6 New Done Done true
8369 1995 Acked 2994 Read 4 New Done Done true
8370 1996 Acked 2995 Read 4 New Done Done true
8371 1997 Acked 2996 Read 5 New Done Done true
8372 1998 Acked 2997 Read 9 New Done Done true
8373 1999 Acked 2998 Read 5 New Done Done true
8374 2000 Acked 2999 Read 7 New Done Done true
8375 2001 Acked 3000 Write 9 New Done Done true
8376 2002 Acked 3001 Write 3 New Done Done true
8377 2003 Acked 3002 Flush 0 New Done Done true
8378 2004 Acked 3003 Read 8 New Done Done true
8379 2005 Acked 3004 Read 8 New Done Done true
8380 2006 Acked 3005 Read 6 New Done Done true
8381 2007 Acked 3006 Read 1 New Done Done true
8382 2008 Acked 3007 Write 5 New Done Done true
8383 2009 Acked 3008 Write 3 New Done Done true
8384 2010 Acked 3009 Read 3 New Done Done true
8385 2011 Acked 3010 Read 3 New Done Done true
8386 2012 Acked 3011 Write 10 New Done Done true
8387 2013 Acked 3012 Write 6 New Done Done true
8388 2014 Acked 3013 Read 6 New Done Done true
8389 2015 Acked 3014 Flush 0 New Done Done true
8390 2016 Acked 3015 Read 4 New Done Done true
8391 2017 Acked 3016 Read 2 New Done Done true
8392 2018 Acked 3017 Read 2 New Done Done true
8393 2019 Acked 3018 Read 5 New Done Done true
8394 2020 Acked 3019 Read 2 New Done Done true
8395 2021 Acked 3020 Read 8 New Done Done true
8396 2022 Acked 3021 Write 4 New Done Done true
8397 2023 Acked 3022 Write 3 New Done Done true
8398 2024 Acked 3023 Flush 0 New Done Done true
8399 2025 Acked 3024 Read 1 New Done Done true
8400 2026 Acked 3025 Read 4 New Done Done true
8401 2027 Acked 3026 Write 1 New Done Done true
8402 2028 Acked 3027 Write 1 New Done Done true
8403 2029 Acked 3028 Write 7 New Done Done true
8404 2030 Acked 3029 Write 4 New Done Done true
8405 2031 Acked 3030 Read 8 New Done Done true
8406 2032 Acked 3031 Write 4 New Done Done true
8407 2033 Acked 3032 Read 9 New Done Done true
8408 2034 Acked 3033 Write 2 New Done Done true
8409 2035 Acked 3034 Write 7 New Done Done true
8410 2036 Acked 3035 Read 3 New Done Done true
8411 2037 Acked 3036 Write 8 New Done Done true
8412 2038 Acked 3037 Flush 0 New Done Done true
8413 2039 Acked 3038 Write 5 New Done Done true
8414 2040 Acked 3039 Write 6 New Done Done true
8415 2041 Acked 3040 Read 4 New Done Done true
8416 2042 Acked 3041 Read 6 New Done Done true
8417 2043 Acked 3042 Write 8 New Done Done true
8418 2044 Acked 3043 Read 3 New Done Done true
8419 2045 Acked 3044 Write 3 New Done Done true
8420 2046 Acked 3045 Flush 0 New Done Done true
8421 2047 Acked 3046 Read 7 New Done Done true
8422 2048 Acked 3047 Write 4 New Done Done true
8423 2049 Acked 3048 Flush 0 New Done Done true
8424 2050 Acked 3049 Write 3 New Done Done true
8425 2051 Acked 3050 Read 7 New Done Done true
8426 2052 Acked 3051 Flush 0 New Done Done true
8427 2053 Acked 3052 Read 10 New Done Done true
8428 2054 Acked 3053 Read 1 New Done Done true
8429 2055 Acked 3054 Write 8 New Done Done true
8430 2056 Acked 3055 Write 7 New Done Done true
8431 2057 Acked 3056 Write 5 New Done Done true
8432 2058 Acked 3057 Write 3 New Done Done true
8433 2059 Acked 3058 Read 1 New Done Done true
8434 2060 Acked 3059 Read 8 New Done Done true
8435 2061 Acked 3060 Read 1 New Done Done true
8436 2062 Acked 3061 Write 7 New Done Done true
8437 2063 Acked 3062 Read 2 New Done Done true
8438 2064 Acked 3063 Flush 0 New Done Done true
8439 2065 Acked 3064 Write 3 New Done Done true
8440 2066 Acked 3065 Write 3 New Done Done true
8441 2067 Acked 3066 Write 1 New Done Done true
8442 2068 Acked 3067 Read 1 New Done Done true
8443 2069 Acked 3068 Read 1 New Done Done true
8444 2070 Acked 3069 Read 7 New Done Done true
8445 2071 Acked 3070 Read 1 New Done Done true
8446 2072 Acked 3071 Read 7 New Done Done true
8447 2073 Acked 3072 Read 5 New Done Done true
8448 2074 Acked 3073 Read 1 New Done Done true
8449 2075 Acked 3074 Read 9 New Done Done true
8450 2076 Acked 3075 Read 5 New Done Done true
8451 2077 Acked 3076 Write 3 New Done Done true
8452 2078 Acked 3077 Read 1 New Done Done true
8453 2079 Acked 3078 Write 7 New Done Done true
8454 2080 Acked 3079 Write 9 New Done Done true
8455 2081 Acked 3080 Read 4 New Done Done true
8456 2082 Acked 3081 Write 3 New Done Done true
8457 2083 Acked 3082 Read 6 New Done Done true
8458 2084 Acked 3083 Write 2 New Done Done true
8459 2085 Acked 3084 Write 2 New Done Done true
8460 2086 Acked 3085 Write 8 New Done Done true
8461 2087 Acked 3086 Write 10 New Done Done true
8462 2088 Acked 3087 Flush 0 New Done Done true
8463 2089 Acked 3088 Write 3 New Done Done true
8464 2090 Acked 3089 Write 3 New Done Done true
8465 2091 Acked 3090 Read 7 New Done Done true
8466 2092 Acked 3091 Read 2 New Done Done true
8467 2093 Acked 3092 Write 7 New Done Done true
8468 2094 Acked 3093 Read 10 New Done Done true
8469 2095 Acked 3094 Write 1 New Done Done true
8470 2096 Acked 3095 Write 1 New Done Done true
8471 2097 Acked 3096 Read 8 New Done Done true
8472 2098 Acked 3097 Write 8 New Done Done true
8473 2099 Acked 3098 Read 3 New Done Done true
8474 2100 Acked 3099 Write 3 New Done Done true
8475 2101 Acked 3100 Write 8 New Done Done true
8476 2102 Acked 3101 Write 6 New Done Done true
8477 2103 Acked 3102 Write 7 New Done Done true
8478 2104 Acked 3103 Read 6 New Done Done true
8479 2105 Acked 3104 Read 3 New Done Done true
8480 2106 Acked 3105 Flush 0 New Done Done true
8481 2107 Acked 3106 Write 9 New Done Done true
8482 2108 Acked 3107 Write 2 New Done Done true
8483 2109 Acked 3108 Flush 0 New Done Done true
8484 STATES DS:0 DS:1 DS:2 TOTAL
8485 New 302 0 0 302
8486 Sent 0 0 0 0
8487 Done 0 302 302 604
8488 Skipped 0 0 0 0
8489 Error 0 0 0 0
8490 Last Flush: 2806 3108 3108
8491 Downstairs last five completed: 2806 2805 2804 2803 2802
8492 Upstairs last five completed: 2109 2108 2107 2106 2105
8493 CLIENT: Up:0 ds:302 act:2
84942024-02-15T02:59:09.665ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
84952024-02-15T02:59:09.665ZINFOcrucible: ds_connection connected = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
84962024-02-15T02:59:09.666ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
84972024-02-15T02:59:09.666ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
84982024-02-15T02:59:09.666ZINFOcrucible: send last flush ID to this DS: 2806 = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
84992024-02-15T02:59:09.666ZINFOcrucible: Replied this last flush ID: 2806 = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
85002024-02-15T02:59:09.666ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
85012024-02-15T02:59:09.666ZINFOcrucible: [0] Transition from Offline to Replay = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
85022024-02-15T02:59:09.666ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
85032024-02-15T02:59:09.666ZINFOcrucible: [0] Transition from Replay to Active = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
8504 ----------------------------------------------------------------
8505 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
8506 Downstairs last five completed: 3108 3107 3106 3105 3104
8507 Upstairs last five completed: 2109 2108 2107 2106 2105
8508 CLIENT: Up:0 ds:0 act:3
8509 Replay: All jobs finished, all DS active.
8510 Main task has work to do, go find it
8511 got dsc Stop(0)
8512 stop 0
8513 Feb 15 02:59:12.988 INFO request completed, latency_us: 361, response_code: 204, uri: /stop/cid/0, method: POST, req_id: ffd342ef-84e7-457d-9d06-75b4437ebca8, remote_addr: 127[8810] Got stop action so:false kr:false
8514 .0.0.1:54744, local_addr: 127.0.0.1:9998
8515 Feb 15 02:59:12.989 INFO request completed, latency_us: {"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2024-02-15T02:59:12.989328249Z","hostname":"ip-10-150-1-67.us-west-2.compute.internal","pid":1167,"":"io task","client":"0","":"downstairs","session_id":"4f5a63e5-aa00-4890-9f6b-96c2e069fdb2"}
8516 100, response_code: 200, uri: /state/cid/0, method: GET, req_id: a31e9f00-7368-4bc2-b1a8-ea7ea5106f40, remote_addr: 127.0.0.1:54744, local_addr: 127.0.0.1:9998{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2024-02-15T02:59:12.989429586Z","hostname":"ip-10-150-1-67.us-west-2.compute.internal","pid":1167,"":"io task","client":"0","":"downstairs","session_id":"4f5a63e5-aa00-4890-9f6b-96c2e069fdb2"}
8517 
85182024-02-15T02:59:12.989ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
85192024-02-15T02:59:12.989ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
85202024-02-15T02:59:12.989ZINFOcrucible: [0] client re-new 0 jobs since flush 3108 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
85212024-02-15T02:59:12.989ZINFOcrucible: client task is exiting = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
8522 [8810] Exited with: ExitStatus(unix_wait_status(9))
8523 [8810][0] reports Exit
85242024-02-15T02:59:13.990ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
85252024-02-15T02:59:13.990ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
85262024-02-15T02:59:13.990ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
85272024-02-15T02:59:13.990ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
85282024-02-15T02:59:13.990ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
85292024-02-15T02:59:13.990ZINFOcrucible: [0] client re-new 0 jobs since flush 3108 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
85302024-02-15T02:59:13.991ZINFOcrucible: client task is exiting = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
85312024-02-15T02:59:14.992ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
85322024-02-15T02:59:14.992ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
85332024-02-15T02:59:14.992ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
85342024-02-15T02:59:14.992ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
85352024-02-15T02:59:14.992ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
85362024-02-15T02:59:14.992ZINFOcrucible: [0] client re-new 0 jobs since flush 3108 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
85372024-02-15T02:59:14.992ZINFOcrucible: client task is exiting = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
85382024-02-15T02:59:15.994ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
85392024-02-15T02:59:15.994ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
85402024-02-15T02:59:15.994ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
85412024-02-15T02:59:15.994ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
85422024-02-15T02:59:15.994ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
85432024-02-15T02:59:15.994ZINFOcrucible: [0] client re-new 0 jobs since flush 3108 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
85442024-02-15T02:59:15.994ZINFOcrucible: client task is exiting = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
8545 Feb 15 02:59:16.991 INFO request completed, latency_us: 311, response_code: 200, uri: /state/cid/0, method: GET, req_id: 2bb471d7-f1a6-4747-b793-ce2eaecf0cb0, remote_addr: 127.0.0.1:54744, local_addr: 127.0.0.1:9998
8546 001/300 Read block 1827 len 4096
8547 002/300 Flush
8548 003/300 Write block 33 len 20480 data: 2 2 2 2 2
8549 004/300 Write block 3628 len 20480 data: 3 2 2 2 2
8550 005/300 Write block 180 len 4096 data: 3
8551 006/300 Read block 3044 len 16384
85522024-02-15T02:59:16.995ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
85532024-02-15T02:59:16.995ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
85542024-02-15T02:59:16.995ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
85552024-02-15T02:59:16.995ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
85562024-02-15T02:59:16.995ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
85572024-02-15T02:59:16.995ZINFOcrucible: [0] client re-new 6 jobs since flush 3108 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
85582024-02-15T02:59:16.995ZINFOcrucible: client task is exiting = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
8559 007/300 Flush
8560 008/300 Flush
8561 009/300 Read block 4025 len 40960
8562 010/300 Write block 1577 len 4096 data: 2
8563 011/300 Read block 3885 len 24576
8564 012/300 Flush
8565 013/300 Read block 2687 len 16384
8566 014/300 Read block 3064 len 36864
8567 015/300 Read block 4199 len 20480
8568 016/300 Read block 3719 len 24576
8569 017/300 Write block 1642 len 36864 data: 6 6 6 6 6 5 4 5 3
8570 018/300 Write block 3048 len 40960 data: 3 3 2 2 4 4 4 4 4 4
8571 019/300 Read block 2381 len 24576
8572 020/300 Read block 4051 len 20480
8573 021/300 Write block 1191 len 36864 data: 3 3 3 3 3 2 2 2 2
8574 022/300 Read block 637 len 36864
8575 023/300 Read block 4635 len 20480
8576 024/300 Read block 1454 len 28672
8577 025/300 Read block 4914 len 12288
8578 026/300 Read block 3394 len 28672
8579 027/300 Read block 2558 len 20480
8580 028/300 Write block 4423 len 28672 data: 4 4 4 4 4 3 3
8581 029/300 Read block 121 len 36864
8582 030/300 Read block 1270 len 36864
8583 031/300 Read block 1052 len 28672
8584 032/300 Read block 4578 len 8192
8585 033/300 Read block 1572 len 4096
8586 034/300 Read block 2928 len 24576
8587 035/300 Read block 3841 len 32768
8588 036/300 Flush
8589 037/300 Flush
8590 038/300 Write block 1838 len 12288 data: 3 2 2
8591 039/300 Read block 388 len 40960
8592 040/300 Read block 366 len 28672
8593 041/300 Write block 2232 len 20480 data: 3 3 3 2 2
8594 042/300 Read block 4896 len 12288
8595 043/300 Read block 2074 len 8192
8596 044/300 Flush
8597 045/300 Read block 4786 len 20480
8598 046/300 Read block 2304 len 32768
8599 047/300 Read block 204 len 40960
8600 048/300 Read block 4258 len 24576
8601 049/300 Read block 452 len 28672
8602 050/300 Read block 2610 len 12288
8603 051/300 Write block 3284 len 32768 data: 2 2 2 2 2 2 2 2
8604 052/300 Read block 918 len 36864
8605 053/300 Write block 3662 len 12288 data: 3 3 2
8606 054/300 Read block 1460 len 36864
8607 055/300 Read block 3017 len 36864
8608 056/300 Write block 834 len 16384 data: 3 2 3 3
8609 057/300 Flush
8610 058/300 Read block 1703 len 28672
8611 059/300 Write block 3795 len 20480 data: 2 2 2 2 2
8612 060/300 Write block 2677 len 16384 data: 2 2 2 2
8613 061/300 Write block 2354 len 32768 data: 3 3 3 3 3 3 2 2
8614 062/300 Write block 899 len 28672 data: 2 3 3 4 4 2 2
8615 063/300 Read block 1333 len 28672
8616 064/300 Write block 3304 len 8192 data: 4 3
8617 065/300 Read block 1730 len 40960
8618 066/300 Write block 3142 len 28672 data: 4 3 3 3 3 3 3
8619 067/300 Write block 2158 len 40960 data: 4 4 3 3 3 3 3 3 2 2
8620 068/300 Read block 4129 len 28672
8621 069/300 Write block 584 len 36864 data: 5 4 3 2 2 3 4 4 3
8622 070/300 Read block 4356 len 4096
8623 071/300 Write block 2080 len 20480 data: 2 3 4 5 5
8624 072/300 Read block 2417 len 4096
8625 073/300 Write block 3710 len 4096 data: 2
8626 074/300 Write block 918 len 8192 data: 3 2
8627 075/300 Read block 610 len 8192
8628 076/300 Read block 1541 len 24576
8629 077/300 Read block 3363 len 16384
8630 078/300 Read block 2441 len 12288
8631 079/300 Read block 2633 len 28672
8632 080/300 Write block 3756 len 4096 data: 2
8633 081/300 Write block 3049 len 16384 data: 4 3 3 5
8634 082/300 Read block 3246 len 8192
8635 083/300 Write block 489 len 28672 data: 3 3 3 5 4 4 4
8636 084/300 Read block 289 len 8192
8637 085/300 Write block 3408 len 20480 data: 3 3 5 5 4
8638 086/300 Read block 383 len 16384
8639 087/300 Write block 2984 len 28672 data: 2 2 2 2 2 3 3
8640 088/300 Read block 4417 len 32768
8641 089/300 Read block 1978 len 32768
8642 090/300 Read block 3640 len 24576
8643 091/300 Write block 4323 len 32768 data: 2 3 3 3 3 3 3 3
8644 092/300 Read block 2252 len 32768
8645 093/300 Write block 547 len 36864 data: 3 3 2 2 2 2 2 3 3
8646 094/300 Write block 4418 len 20480 data: 5 4 4 3 3
8647 095/300 Write block 529 len 32768 data: 3 3 4 4 4 5 5 5
8648 096/300 Write block 4198 len 4096 data: 2
8649 097/300 Write block 3034 len 4096 data: 4
8650 098/300 Read block 2541 len 8192
8651 099/300 Write block 3327 len 36864 data: 2 2 2 2 2 2 2 2 2
8652 100/300 Write block 1197 len 20480 data: 3 3 3 2 2
8653 101/300 Write block 3964 len 20480 data: 2 2 2 3 3
8654 102/300 Write block 2585 len 4096 data: 3
8655 103/300 Read block 1845 len 4096
8656 104/300 Flush
8657 105/300 Write block 3338 len 40960 data: 2 2 2 2 2 2 2 2 2 3
8658 106/300 Write block 4844 len 24576 data: 3 3 3 2 2 2
8659 107/300 Write block 4964 len 36864 data: 3 4 3 3 3 4 3 3 3
8660 108/300 Write block 990 len 24576 data: 4 3 2 2 2 2
8661 109/300 Write block 680 len 16384 data: 2 2 2 2
8662 110/300 Write block 3557 len 36864 data: 2 2 3 3 3 3 3 3 4
8663 111/300 Write block 2490 len 16384 data: 2 2 2 2
8664 112/300 Read block 3642 len 24576
8665 113/300 Read block 1065 len 4096
8666 114/300 Read block 4319 len 4096
8667 115/300 Write block 1506 len 12288 data: 3 2 3
8668 116/300 Read block 3008 len 40960
8669 117/300 Read block 55 len 28672
8670 118/300 Read block 3490 len 16384
8671 119/300 Read block 4560 len 36864
8672 120/300 Write block 3820 len 12288 data: 2 2 2
8673 121/300 Read block 965 len 28672
8674 122/300 Read block 4864 len 36864
8675 123/300 Flush
8676 124/300 Read block 3983 len 4096
8677 125/300 Write block 206 len 36864 data: 3 3 2 2 2 2 2 2 2
8678 126/300 Read block 4026 len 4096
8679 127/300 Read block 206 len 4096
8680 128/300 Read block 342 len 28672
8681 129/300 Flush
8682 130/300 Read block 3456 len 28672
8683 131/300 Write block 3767 len 8192 data: 3 4
8684 132/300 Write block 4494 len 16384 data: 2 2 2 2
8685 133/300 Read block 1274 len 36864
8686 134/300 Write block 1654 len 8192 data: 3 3
8687 135/300 Read block 3902 len 24576
8688 136/300 Flush
8689 137/300 Read block 2931 len 4096
8690 138/300 Write block 936 len 16384 data: 2 2 2 2
8691 139/300 Flush
8692 140/300 Flush
8693 141/300 Read block 2033 len 20480
8694 142/300 Write block 2424 len 40960 data: 3 3 3 3 2 2 3 3 3 2
8695 143/300 Write block 102 len 20480 data: 2 2 2 2 2
8696 144/300 Flush
8697 145/300 Read block 550 len 20480
8698 146/300 Flush
8699 147/300 Read block 486 len 20480
8700 148/300 Read block 296 len 28672
8701 149/300 Read block 1471 len 20480
8702 150/300 Write block 2235 len 20480 data: 3 3 3 3 4
8703 151/300 Write block 4753 len 40960 data: 4 4 4 3 3 3 3 3 3 4
8704 152/300 Flush
8705 153/300 Read block 359 len 32768
8706 154/300 Read block 1987 len 12288
8707 155/300 Write block 1973 len 4096 data: 3
8708 156/300 Read block 3477 len 24576
8709 157/300 Write block 1884 len 12288 data: 2 3 3
8710 158/300 Write block 4965 len 36864 data: 5 4 4 4 5 4 4 4 3
8711 159/300 Read block 2222 len 20480
8712 160/300 Write block 1328 len 40960 data: 2 2 2 2 2 2 2 3 3 3
8713 161/300 Write block 3519 len 24576 data: 3 3 4 4 4 3
8714 162/300 Read block 700 len 24576
8715 163/300 Read block 4428 len 16384
8716 164/300 Write block 1872 len 28672 data: 2 2 3 3 3 3 3
8717 165/300 Read block 419 len 40960
8718 166/300 Read block 4345 len 28672
8719 167/300 Write block 1099 len 40960 data: 3 3 3 3 3 2 2 2 2 2
8720 168/300 Read block 1062 len 24576
8721 169/300 Read block 782 len 12288
8722 170/300 Read block 3784 len 8192
8723 171/300 Write block 3876 len 32768 data: 3 2 2 2 4 4 4 4
8724 172/300 Read block 1947 len 8192
8725 173/300 Write block 2608 len 40960 data: 2 2 2 2 3 3 3 3 2 2
8726 174/300 Read block 4100 len 20480
8727 175/300 Write block 4312 len 36864 data: 3 3 3 3 2 2 2 3 2
8728 176/300 Write block 4898 len 12288 data: 2 2 3
8729 177/300 Read block 712 len 32768
8730 178/300 Read block 889 len 4096
8731 179/300 Read block 425 len 24576
8732 180/300 Read block 3665 len 16384
8733 181/300 Read block 4987 len 8192
8734 182/300 Write block 3987 len 32768 data: 3 3 3 2 2 2 2 2
8735 183/300 Write block 873 len 28672 data: 2 2 2 2 2 2 2
8736 184/300 Write block 2140 len 32768 data: 2 2 2 3 3 3 3 3
8737 185/300 Read block 1676 len 28672
8738 186/300 Flush
8739 187/300 Flush
8740 188/300 Read block 3508 len 16384
8741 189/300 Read block 1943 len 40960
8742 190/300 Write block 4546 len 20480 data: 2 2 3 3 4
8743 191/300 Read block 3454 len 16384
8744 192/300 Flush
8745 193/300 Write block 3325 len 20480 data: 2 2 3 3 3
8746 194/300 Flush
8747 195/300 Read block 1788 len 4096
8748 196/300 Write block 4383 len 40960 data: 3 2 2 2 2 2 2 2 2 2
8749 197/300 Read block 1809 len 24576
8750 198/300 Write block 3994 len 40960 data: 3 3 3 2 2 2 2 2 2 2
8751 199/300 Read block 4623 len 36864
8752 200/300 Read block 2133 len 4096
8753 201/300 Read block 4733 len 28672
8754 202/300 Write block 631 len 12288 data: 4 4 3
8755 203/300 Write block 2159 len 20480 data: 5 4 4 4 4
8756 204/300 Write block 3955 len 12288 data: 4 5 4
8757 205/300 Read block 1321 len 12288
8758 206/300 Read block 359 len 4096
8759 207/300 Read block 4780 len 12288
8760 208/300 Read block 2793 len 16384
8761 209/300 Write block 4432 len 28672 data: 4 5 5 4 4 4 3
8762 210/300 Read block 2824 len 32768
8763 211/300 Read block 4222 len 28672
8764 212/300 Flush
8765 213/300 Read block 4195 len 12288
8766 214/300 Write block 2501 len 36864 data: 3 3 3 3 3 3 3 2 2
8767 215/300 Read block 2191 len 12288
8768 216/300 Read block 2976 len 12288
8769 217/300 Write block 4210 len 12288 data: 2 2 2
8770 218/300 Flush
8771 219/300 Read block 3175 len 24576
8772 220/300 Write block 252 len 36864 data: 3 3 3 4 4 4 5 4 4
8773 221/300 Read block 3110 len 28672
8774 222/300 Read block 4384 len 28672
8775 223/300 Write block 2640 len 36864 data: 4 3 4 4 4 4 4 4 4
8776 224/300 Read block 1395 len 12288
8777 225/300 Write block 1955 len 16384 data: 3 3 3 4
8778 226/300 Read block 2400 len 32768
8779 227/300 Write block 915 len 16384 data: 4 3 3 4
8780 228/300 Write block 1810 len 36864 data: 3 3 3 3 3 3 3 3 2
8781 229/300 Read block 4240 len 20480
8782 230/300 Write block 749 len 16384 data: 2 2 2 2
8783 231/300 Read block 2585 len 36864
8784 232/300 Read block 3217 len 20480
8785 233/300 Read block 4939 len 40960
8786 234/300 Read block 1503 len 28672
8787 235/300 Write block 1285 len 20480 data: 3 3 3 3 3
8788 236/300 Read block 4634 len 4096
8789 237/300 Write block 2818 len 12288 data: 2 2 3
8790 238/300 Write block 2634 len 16384 data: 3 3 3 3
8791 239/300 Read block 478 len 24576
8792 240/300 Read block 2774 len 36864
8793 241/300 Write block 1718 len 8192 data: 2 2
8794 242/300 Read block 673 len 12288
8795 243/300 Read block 4643 len 32768
8796 244/300 Flush
8797 245/300 Read block 1991 len 28672
8798 246/300 Write block 3496 len 32768 data: 4 4 4 4 3 3 3 2
8799 247/300 Write block 510 len 20480 data: 4 4 5 6 6
8800 248/300 Write block 4678 len 12288 data: 5 5 5
8801 249/300 Flush
8802 250/300 Write block 4505 len 20480 data: 3 3 3 3 3
8803 251/300 Write block 3051 len 28672 data: 4 6 5 5 5 5 5
8804 252/300 Read block 68 len 36864
8805 253/300 Read block 407 len 16384
8806 254/300 Read block 1026 len 4096
8807 255/300 Write block 4758 len 20480 data: 4 4 4 4 5
8808 256/300 Write block 579 len 12288 data: 4 4 4
8809 257/300 Write block 1280 len 12288 data: 2 2 2
8810 258/300 Read block 1503 len 36864
8811 259/300 Read block 4944 len 20480
8812 260/300 Write block 3584 len 28672 data: 3 3 3 2 2 2 2
8813 261/300 Read block 1644 len 32768
8814 262/300 Write block 21 len 36864 data: 4 4 5 5 5 5 5 4 4
8815 263/300 Read block 950 len 16384
8816 264/300 Write block 4691 len 4096 data: 4
8817 265/300 Write block 1940 len 16384 data: 2 2 2 2
8818 266/300 Write block 2367 len 12288 data: 3 2 2
8819 267/300 Write block 350 len 24576 data: 3 3 3 3 3 3
8820 268/300 Read block 4297 len 16384
8821 269/300 Read block 4083 len 20480
8822 270/300 Read block 710 len 12288
8823 271/300 Write block 193 len 40960 data: 3 3 4 4 4 3 4 3 3 3
8824 272/300 Read block 831 len 4096
8825 273/300 Read block 2268 len 28672
8826 274/300 Read block 2606 len 32768
8827 275/300 Read block 619 len 24576
8828 276/300 Read block 1606 len 32768
8829 277/300 Write block 2910 len 16384 data: 3 3 3 3
8830 278/300 Read block 4118 len 4096
8831 279/300 Write block 1884 len 8192 data: 3 4
8832 280/300 Read block 1337 len 24576
8833 281/300 Write block 1889 len 16384 data: 2 2 2 2
8834 282/300 Read block 287 len 20480
8835 283/300 Write block 4105 len 40960 data: 2 2 2 2 2 2 4 4 5 5
8836 284/300 Read block 2938 len 40960
8837 285/300 Flush
8838 286/300 Read block 3125 len 28672
8839 287/300 Read block 3414 len 16384
8840 288/300 Read block 4071 len 36864
8841 289/300 Read block 2809 len 20480
8842 290/300 Flush
8843 291/300 Write block 3729 len 12288 data: 5 6 5
8844 292/300 Write block 1341 len 12288 data: 2 2 2
8845 293/300 Write block 4661 len 40960 data: 3 3 2 2 2 3 3 3 3 3
8846 294/300 Read block 3855 len 40960
8847 295/300 Write block 3207 len 16384 data: 3 3 2 3
8848 296/300 Write block 4250 len 8192 data: 4 4
8849 297/300 Flush
8850 298/300 Write block 4981 len 36864 data: 3 4 4 4 4 4 4 4 4
8851 299/300 Read block 3797 len 16384
8852 300/300 Read block 474 len 36864
8853 Main task has work to do, go find it
8854 got dsc Start(0)
8855 start 0
8856 Feb 15 02:59:17.329 INFO request completed, latency_us: 220, response_code: 204, uri: /start/cid/0, method: POST, req_id: e55cbf55-7881-4fd7-838b-6fdee753d9a5, remote_addr: 127.0.0.1:54744, local_addr: 127.0.0.1:9998
8857 Replay: started 0, returned:Ok(())
8858 ----------------------------------------------------------------
8859 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
8860 States: Offline Active Active
8861 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
8862 2110 Acked 3109 Read 1 New Done Done true
8863 2111 Acked 3110 Flush 0 New Done Done true
8864 2112 Acked 3111 Write 5 New Done Done true
8865 2113 Acked 3112 Write 5 New Done Done true
8866 2114 Acked 3113 Write 1 New Done Done true
8867 2115 Acked 3114 Read 4 New Done Done true
8868 2116 Acked 3115 Flush 0 New Done Done false
8869 2117 Acked 3116 Flush 0 New Done Done false
8870 2118 Acked 3117 Read 10 New Done Done false
8871 2119 Acked 3118 Write 1 New Done Done false
8872 2120 Acked 3119 Read 6 New Done Done false
8873 2121 Acked 3120 Flush 0 New Done Done false
8874 2122 Acked 3121 Read 4 New Done Done false
8875 2123 Acked 3122 Read 9 New Done Done false
8876 2124 Acked 3123 Read 5 New Done Done false
8877 2125 Acked 3124 Read 6 New Done Done false
8878 2126 Acked 3125 Write 9 New Done Done false
8879 2127 Acked 3126 Write 10 New Done Done false
8880 2128 Acked 3127 Read 6 New Done Done false
8881 2129 Acked 3128 Read 5 New Done Done false
8882 2130 Acked 3129 Write 9 New Done Done false
8883 2131 Acked 3130 Read 9 New Done Done false
8884 2132 Acked 3131 Read 5 New Done Done false
8885 2133 Acked 3132 Read 7 New Done Done false
8886 2134 Acked 3133 Read 3 New Done Done false
8887 2135 Acked 3134 Read 7 New Done Done false
8888 2136 Acked 3135 Read 5 New Done Done false
8889 2137 Acked 3136 Write 7 New Done Done false
8890 2138 Acked 3137 Read 9 New Done Done false
8891 2139 Acked 3138 Read 9 New Done Done false
8892 2140 Acked 3139 Read 7 New Done Done false
8893 2141 Acked 3140 Read 2 New Done Done false
8894 2142 Acked 3141 Read 1 New Done Done false
8895 2143 Acked 3142 Read 6 New Done Done false
8896 2144 Acked 3143 Read 8 New Done Done false
8897 2145 Acked 3144 Flush 0 New Done Done false
8898 2146 Acked 3145 Flush 0 New Done Done false
8899 2147 Acked 3146 Write 3 New Done Done false
8900 2148 Acked 3147 Read 10 New Done Done false
8901 2149 Acked 3148 Read 7 New Done Done false
8902 2150 Acked 3149 Write 5 New Done Done false
8903 2151 Acked 3150 Read 3 New Done Done false
8904 2152 Acked 3151 Read 2 New Done Done false
8905 2153 Acked 3152 Flush 0 New Done Done false
8906 2154 Acked 3153 Read 5 New Done Done false
8907 2155 Acked 3154 Read 8 New Done Done false
8908 2156 Acked 3155 Read 10 New Done Done false
8909 2157 Acked 3156 Read 6 New Done Done false
8910 2158 Acked 3157 Read 7 New Done Done false
8911 2159 Acked 3158 Read 3 New Done Done false
8912 2160 Acked 3159 Write 8 New Done Done false
8913 2161 Acked 3160 Read 9 New Done Done false
8914 2162 Acked 3161 Write 3 New Done Done false
8915 2163 Acked 3162 Read 9 New Done Done false
8916 2164 Acked 3163 Read 9 New Done Done false
8917 2165 Acked 3164 Write 4 New Done Done false
8918 2166 Acked 3165 Flush 0 New Done Done false
8919 2167 Acked 3166 Read 7 New Done Done false
8920 2168 Acked 3167 Write 5 New Done Done false
8921 2169 Acked 3168 Write 4 New Done Done false
8922 2170 Acked 3169 Write 8 New Done Done false
8923 2171 Acked 3170 Write 7 New Done Done false
8924 2172 Acked 3171 Read 7 New Done Done false
8925 2173 Acked 3172 Write 2 New Done Done false
8926 2174 Acked 3173 Read 10 New Done Done false
8927 2175 Acked 3174 Write 7 New Done Done false
8928 2176 Acked 3175 Write 10 New Done Done false
8929 2177 Acked 3176 Read 7 New Done Done false
8930 2178 Acked 3177 Write 9 New Done Done false
8931 2179 Acked 3178 Read 1 New Done Done false
8932 2180 Acked 3179 Write 5 New Done Done false
8933 2181 Acked 3180 Read 1 New Done Done false
8934 2182 Acked 3181 Write 1 New Done Done false
8935 2183 Acked 3182 Write 2 New Done Done false
8936 2184 Acked 3183 Read 2 New Done Done false
8937 2185 Acked 3184 Read 6 New Done Done false
8938 2186 Acked 3185 Read 4 New Done Done false
8939 2187 Acked 3186 Read 3 New Done Done false
8940 2188 Acked 3187 Read 7 New Done Done false
8941 2189 Acked 3188 Write 1 New Done Done false
8942 2190 Acked 3189 Write 4 New Done Done false
8943 2191 Acked 3190 Read 2 New Done Done false
8944 2192 Acked 3191 Write 7 New Done Done false
8945 2193 Acked 3192 Read 2 New Done Done false
8946 2194 Acked 3193 Write 5 New Done Done false
8947 2195 Acked 3194 Read 4 New Done Done false
8948 2196 Acked 3195 Write 7 New Done Done false
8949 2197 Acked 3196 Read 8 New Done Done false
8950 2198 Acked 3197 Read 8 New Done Done false
8951 2199 Acked 3198 Read 6 New Done Done false
8952 2200 Acked 3199 Write 8 New Done Done false
8953 2201 Acked 3200 Read 8 New Done Done false
8954 2202 Acked 3201 Write 9 New Done Done false
8955 2203 Acked 3202 Write 5 New Done Done false
8956 2204 Acked 3203 Write 8 New Done Done false
8957 2205 Acked 3204 Write 1 New Done Done false
8958 2206 Acked 3205 Write 1 New Done Done false
8959 2207 Acked 3206 Read 2 New Done Done false
8960 2208 Acked 3207 Write 9 New Done Done false
8961 2209 Acked 3208 Write 5 New Done Done false
8962 2210 Acked 3209 Write 5 New Done Done false
8963 2211 Acked 3210 Write 1 New Done Done false
8964 2212 Acked 3211 Read 1 New Done Done false
8965 2213 Acked 3212 Flush 0 New Done Done false
8966 2214 Acked 3213 Write 10 New Done Done false
8967 2215 Acked 3214 Write 6 New Done Done false
8968 2216 Acked 3215 Write 9 New Done Done false
8969 2217 Acked 3216 Write 6 New Done Done false
8970 2218 Acked 3217 Write 4 New Done Done false
8971 2219 Acked 3218 Write 9 New Done Done false
8972 2220 Acked 3219 Write 4 New Done Done false
8973 2221 Acked 3220 Read 6 New Done Done false
8974 2222 Acked 3221 Read 1 New Done Done false
8975 2223 Acked 3222 Read 1 New Done Done false
8976 2224 Acked 3223 Write 3 New Done Done false
8977 2225 Acked 3224 Read 10 New Done Done false
8978 2226 Acked 3225 Read 7 New Done Done false
8979 2227 Acked 3226 Read 4 New Done Done false
8980 2228 Acked 3227 Read 9 New Done Done false
8981 2229 Acked 3228 Write 3 New Done Done false
8982 2230 Acked 3229 Read 7 New Done Done false
8983 2231 Acked 3230 Read 9 New Done Done false
8984 2232 Acked 3231 Flush 0 New Done Done false
8985 2233 Acked 3232 Read 1 New Done Done false
8986 2234 Acked 3233 Write 9 New Done Done false
8987 2235 Acked 3234 Read 1 New Done Done false
8988 2236 Acked 3235 Read 1 New Done Done false
8989 2237 Acked 3236 Read 7 New Done Done false
8990 2238 Acked 3237 Flush 0 New Done Done false
8991 2239 Acked 3238 Read 7 New Done Done false
8992 2240 Acked 3239 Write 2 New Done Done false
8993 2241 Acked 3240 Write 4 New Done Done false
8994 2242 Acked 3241 Read 9 New Done Done false
8995 2243 Acked 3242 Write 2 New Done Done false
8996 2244 Acked 3243 Read 6 New Done Done false
8997 2245 Acked 3244 Flush 0 New Done Done false
8998 2246 Acked 3245 Read 1 New Done Done false
8999 2247 Acked 3246 Write 4 New Done Done false
9000 2248 Acked 3247 Flush 0 New Done Done false
9001 2249 Acked 3248 Flush 0 New Done Done false
9002 2250 Acked 3249 Read 5 New Done Done false
9003 2251 Acked 3250 Write 10 New Done Done false
9004 2252 Acked 3251 Write 5 New Done Done false
9005 2253 Acked 3252 Flush 0 New Done Done false
9006 2254 Acked 3253 Read 5 New Done Done false
9007 2255 Acked 3254 Flush 0 New Done Done false
9008 2256 Acked 3255 Read 5 New Done Done false
9009 2257 Acked 3256 Read 7 New Done Done false
9010 2258 Acked 3257 Read 5 New Done Done false
9011 2259 Acked 3258 Write 5 New Done Done false
9012 2260 Acked 3259 Write 10 New Done Done false
9013 2261 Acked 3260 Flush 0 New Done Done false
9014 2262 Acked 3261 Read 8 New Done Done false
9015 2263 Acked 3262 Read 3 New Done Done false
9016 2264 Acked 3263 Write 1 New Done Done false
9017 2265 Acked 3264 Read 6 New Done Done false
9018 2266 Acked 3265 Write 3 New Done Done false
9019 2267 Acked 3266 Write 9 New Done Done false
9020 2268 Acked 3267 Read 5 New Done Done false
9021 2269 Acked 3268 Write 10 New Done Done false
9022 2270 Acked 3269 Write 6 New Done Done false
9023 2271 Acked 3270 Read 6 New Done Done false
9024 2272 Acked 3271 Read 4 New Done Done false
9025 2273 Acked 3272 Write 7 New Done Done false
9026 2274 Acked 3273 Read 10 New Done Done false
9027 2275 Acked 3274 Read 7 New Done Done false
9028 2276 Acked 3275 Write 10 New Done Done false
9029 2277 Acked 3276 Read 6 New Done Done false
9030 2278 Acked 3277 Read 3 New Done Done false
9031 2279 Acked 3278 Read 2 New Done Done false
9032 2280 Acked 3279 Write 8 New Done Done false
9033 2281 Acked 3280 Read 2 New Done Done false
9034 2282 Acked 3281 Write 10 New Done Done false
9035 2283 Acked 3282 Read 5 New Done Done false
9036 2284 Acked 3283 Write 9 New Done Done false
9037 2285 Acked 3284 Write 3 New Done Done false
9038 2286 Acked 3285 Read 8 New Done Done false
9039 2287 Acked 3286 Read 1 New Done Done false
9040 2288 Acked 3287 Read 6 New Done Done false
9041 2289 Acked 3288 Read 4 New Done Done false
9042 2290 Acked 3289 Read 2 New Done Done false
9043 2291 Acked 3290 Write 8 New Done Done false
9044 2292 Acked 3291 Write 7 New Done Done false
9045 2293 Acked 3292 Write 8 New Done Done false
9046 2294 Acked 3293 Read 7 New Done Done false
9047 2295 Acked 3294 Flush 0 New Done Done false
9048 2296 Acked 3295 Flush 0 New Done Done false
9049 2297 Acked 3296 Read 4 New Done Done false
9050 2298 Acked 3297 Read 10 New Done Done false
9051 2299 Acked 3298 Write 5 New Done Done false
9052 2300 Acked 3299 Read 4 New Done Done false
9053 2301 Acked 3300 Flush 0 New Done Done false
9054 2302 Acked 3301 Write 5 New Done Done false
9055 2303 Acked 3302 Flush 0 New Done Done false
9056 2304 Acked 3303 Read 1 New Done Done false
9057 2305 Acked 3304 Write 10 New Done Done false
9058 2306 Acked 3305 Read 6 New Done Done false
9059 2307 Acked 3306 Write 10 New Done Done false
9060 2308 Acked 3307 Read 9 New Done Done false
9061 2309 Acked 3308 Read 1 New Done Done false
9062 2310 Acked 3309 Read 7 New Done Done false
9063 2311 Acked 3310 Write 3 New Done Done false
9064 2312 Acked 3311 Write 5 New Done Done false
9065 2313 Acked 3312 Write 3 New Done Done false
9066 2314 Acked 3313 Read 3 New Done Done false
9067 2315 Acked 3314 Read 1 New Done Done false
9068 2316 Acked 3315 Read 3 New Done Done false
9069 2317 Acked 3316 Read 4 New Done Done false
9070 2318 Acked 3317 Write 7 New Done Done false
9071 2319 Acked 3318 Read 8 New Done Done false
9072 2320 Acked 3319 Read 7 New Done Done false
9073 2321 Acked 3320 Flush 0 New Done Done false
9074 2322 Acked 3321 Read 3 New Done Done false
9075 2323 Acked 3322 Write 9 New Done Done false
9076 2324 Acked 3323 Read 3 New Done Done false
9077 2325 Acked 3324 Read 3 New Done Done false
9078 2326 Acked 3325 Write 3 New Done Done false
9079 2327 Acked 3326 Flush 0 New Done Done false
9080 2328 Acked 3327 Read 6 New Done Done false
9081 2329 Acked 3328 Write 9 New Done Done false
9082 2330 Acked 3329 Read 7 New Done Done false
9083 2331 Acked 3330 Read 7 New Done Done false
9084 2332 Acked 3331 Flush 0 New Done Done false
9085 2333 Acked 3332 Write 9 New Done Done false
9086 2334 Acked 3333 Read 3 New Done Done false
9087 2335 Acked 3334 Write 4 New Done Done false
9088 2336 Acked 3335 Read 8 New Done Done false
9089 2337 Acked 3336 Write 4 New Done Done false
9090 2338 Acked 3337 Write 9 New Done Done false
9091 2339 Acked 3338 Read 5 New Done Done false
9092 2340 Acked 3339 Write 4 New Done Done false
9093 2341 Acked 3340 Read 9 New Done Done false
9094 2342 Acked 3341 Read 5 New Done Done false
9095 2343 Acked 3342 Read 10 New Done Done false
9096 2344 Acked 3343 Read 7 New Done Done false
9097 2345 Acked 3344 Write 5 New Done Done false
9098 2346 Acked 3345 Read 1 New Done Done false
9099 2347 Acked 3346 Write 3 New Done Done false
9100 2348 Acked 3347 Write 4 New Done Done false
9101 2349 Acked 3348 Read 6 New Done Done false
9102 2350 Acked 3349 Read 9 New Done Done false
9103 2351 Acked 3350 Write 2 New Done Done false
9104 2352 Acked 3351 Read 3 New Done Done false
9105 2353 Acked 3352 Read 8 New Done Done false
9106 2354 Acked 3353 Flush 0 New Done Done false
9107 2355 Acked 3354 Read 7 New Done Done false
9108 2356 Acked 3355 Write 8 New Done Done false
9109 2357 Acked 3356 Write 5 New Done Done false
9110 2358 Acked 3357 Write 3 New Done Done false
9111 2359 Acked 3358 Flush 0 New Done Done false
9112 2360 Acked 3359 Write 5 New Done Done false
9113 2361 Acked 3360 Write 7 New Done Done false
9114 2362 Acked 3361 Read 9 New Done Done false
9115 2363 Acked 3362 Read 4 New Done Done false
9116 2364 Acked 3363 Read 1 New Done Done false
9117 2365 Acked 3364 Write 5 New Done Done false
9118 2366 Acked 3365 Write 3 New Done Done false
9119 2367 Acked 3366 Write 3 New Done Done false
9120 2368 Acked 3367 Read 9 New Done Done false
9121 2369 Acked 3368 Read 5 New Done Done false
9122 2370 Acked 3369 Write 7 New Done Done false
9123 2371 Acked 3370 Read 8 New Done Done false
9124 2372 Acked 3371 Write 9 New Done Done false
9125 2373 Acked 3372 Read 4 New Done Done false
9126 2374 Acked 3373 Write 1 New Done Done false
9127 2375 Acked 3374 Write 4 New Done Done false
9128 2376 Acked 3375 Write 3 New Done Done false
9129 2377 Acked 3376 Write 6 New Done Done false
9130 2378 Acked 3377 Read 4 New Done Done false
9131 2379 Acked 3378 Read 5 New Done Done false
9132 2380 Acked 3379 Read 3 New Done Done false
9133 2381 Acked 3380 Write 10 New Done Done false
9134 2382 Acked 3381 Read 1 New Done Done false
9135 2383 Acked 3382 Read 7 New Done Done false
9136 2384 Acked 3383 Read 8 New Done Done false
9137 2385 Acked 3384 Read 6 New Done Done false
9138 2386 Acked 3385 Read 8 New Done Done false
9139 2387 Acked 3386 Write 4 New Done Done false
9140 2388 Acked 3387 Read 1 New Done Done false
9141 2389 Acked 3388 Write 2 New Done Done false
9142 2390 Acked 3389 Read 6 New Done Done false
9143 2391 Acked 3390 Write 4 New Done Done false
9144 2392 Acked 3391 Read 5 New Done Done false
9145 2393 Acked 3392 Write 10 New Done Done false
9146 2394 Acked 3393 Read 10 New Done Done false
9147 2395 Acked 3394 Flush 0 New Done Done false
9148 2396 Acked 3395 Read 7 New Done Done false
9149 2397 Acked 3396 Read 4 New Done Done false
9150 2398 Acked 3397 Read 9 New Done Done false
9151 2399 Acked 3398 Read 5 New Done Done false
9152 2400 Acked 3399 Flush 0 New Done Done false
9153 2401 Acked 3400 Write 3 New Done Done false
9154 2402 Acked 3401 Write 3 New Done Done false
9155 2403 Acked 3402 Write 10 New Done Done false
9156 2404 Acked 3403 Read 10 New Done Done false
9157 2405 Acked 3404 Write 4 New Done Done false
9158 2406 Acked 3405 Write 2 New Done Done false
9159 2407 Acked 3406 Flush 0 New Done Done false
9160 2408 Acked 3407 Write 9 New Done Done false
9161 2409 Acked 3408 Read 4 New Done Done false
9162 2410 Acked 3409 Read 9 New Done Done false
9163 STATES DS:0 DS:1 DS:2 TOTAL
9164 New 301 0 0 301
9165 Sent 0 0 0 0
9166 Done 0 301 301 602
9167 Skipped 0 0 0 0
9168 Error 0 0 0 0
9169 Last Flush: 3108 3406 3406
9170 Downstairs last five completed: 3108 3107 3106 3105 3104
9171 Upstairs last five completed: 2410 2409 2408 2407 2406
9172 CLIENT: Up:0 ds:301 act:2
91732024-02-15T02:59:17.996ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
91742024-02-15T02:59:17.996ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
91752024-02-15T02:59:17.996ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
91762024-02-15T02:59:17.996ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
91772024-02-15T02:59:17.996ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
91782024-02-15T02:59:17.996ZINFOcrucible: [0] client re-new 302 jobs since flush 3108 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
91792024-02-15T02:59:17.996ZINFOcrucible: client task is exiting = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
9180 [8810] got start action
9181 [8810] I am going to restart
9182 Starting downstairs at port 8810
9183 Make output file at "/tmp/dsc/downstairs-8810.txt"
9184 [8810][0] reports Starting
91852024-02-15T02:59:18.997ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
91862024-02-15T02:59:18.997ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
91872024-02-15T02:59:18.997ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
91882024-02-15T02:59:18.997ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
91892024-02-15T02:59:18.997ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
91902024-02-15T02:59:18.997ZINFOcrucible: [0] client re-new 302 jobs since flush 3108 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
91912024-02-15T02:59:18.998ZINFOcrucible: client task is exiting = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
9192 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1175)
9193 [8810][0] reports Running
91942024-02-15T02:59:19.998ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
91952024-02-15T02:59:19.998ZINFOcrucible: ds_connection connected = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
91962024-02-15T02:59:20.000ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
91972024-02-15T02:59:20.000ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
91982024-02-15T02:59:20.000ZINFOcrucible: send last flush ID to this DS: 3108 = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
91992024-02-15T02:59:20.000ZINFOcrucible: Replied this last flush ID: 3108 = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
92002024-02-15T02:59:20.000ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
92012024-02-15T02:59:20.000ZINFOcrucible: [0] Transition from Offline to Replay = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
92022024-02-15T02:59:20.000ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
92032024-02-15T02:59:20.000ZINFOcrucible: [0] Transition from Replay to Active = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
9204 ----------------------------------------------------------------
9205 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
9206 Downstairs last five completed: 3410 3409 3408 3407 3406
9207 Upstairs last five completed: 2411 2410 2409 2408 2407
9208 CLIENT: Up:0 ds:0 act:3
9209 Replay: All jobs finished, all DS active.
9210 Main task has work to do, go find it
9211 got dsc Stop(2)
9212 stop 2
9213 Feb 15 02:59:21.332 INFO request completed, latency_us: 320, response_code: 204, [8830] Got stop action so:false kr:false
9214 uri: /stop/cid/2, method: POST, req_id: a3ea7e24-fce5-450c-9e44-5fd5f99d0089, remote_addr: 127.0.0.1:54744, local_addr: 127.0.0.1:9998
92152024-02-15T02:59:21.333ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
92162024-02-15T02:59:21.333ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
92172024-02-15T02:59:21.333ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
92182024-02-15T02:59:21.333ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
92192024-02-15T02:59:21.333ZINFOcrucible: [2] client re-new 0 jobs since flush 3410 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
92202024-02-15T02:59:21.333ZINFOcrucible: client task is exiting = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
9221 Feb 15 02:59:21.333 INFO request completed, latency_us: 122, response_code: 200, uri: /state/cid/2, method: GET, req_id: 7d2d5d37-7b8e-427f-9b50-e26bcbf551a1, remote_addr: 127.0.0.1:54744, local_addr: 127.0.0.1:9998
9222 [8830] Exited with: ExitStatus(unix_wait_status(9))
9223 [8830][2] reports Exit
92242024-02-15T02:59:22.335ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
92252024-02-15T02:59:22.335ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
92262024-02-15T02:59:22.335ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
92272024-02-15T02:59:22.335ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
92282024-02-15T02:59:22.335ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
92292024-02-15T02:59:22.335ZINFOcrucible: [2] client re-new 0 jobs since flush 3410 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
92302024-02-15T02:59:22.335ZINFOcrucible: client task is exiting = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
92312024-02-15T02:59:23.336ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
92322024-02-15T02:59:23.336ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
92332024-02-15T02:59:23.336ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
92342024-02-15T02:59:23.336ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
92352024-02-15T02:59:23.336ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
92362024-02-15T02:59:23.336ZINFOcrucible: [2] client re-new 0 jobs since flush 3410 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
92372024-02-15T02:59:23.336ZINFOcrucible: client task is exiting = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
92382024-02-15T02:59:24.337ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
92392024-02-15T02:59:24.337ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
92402024-02-15T02:59:24.337ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
92412024-02-15T02:59:24.337ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
92422024-02-15T02:59:24.337ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
92432024-02-15T02:59:24.337ZINFOcrucible: [2] client re-new 0 jobs since flush 3410 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
92442024-02-15T02:59:24.337ZINFOcrucible: client task is exiting = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
9245 Feb 15 02:59:25.336 INFO request completed, latency_us: 191, response_code: 200, uri: /state/cid/2, method: GET, req_id: e2614ec9-ace1-4fd1-8039-ecb45096077f, remote_addr: 127.0.0.1:54744, local_addr: 127.0.0.1:9998
9246 001/300 Flush
9247 002/300 Write block 2951 len 40960 data: 2 2 3 3 3 3 3 4 4 4
9248 003/300 Write block 376 len 32768 data: 4 5 5 4 4 3 3 3
9249 004/300 Read block 566 len 4096
92502024-02-15T02:59:25.339ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
92512024-02-15T02:59:25.339ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
92522024-02-15T02:59:25.339ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
92532024-02-15T02:59:25.339ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
92542024-02-15T02:59:25.339ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
92552024-02-15T02:59:25.339ZINFOcrucible: [2] client re-new 4 jobs since flush 3410 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
92562024-02-15T02:59:25.339ZINFOcrucible: client task is exiting = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
9257 005/300 Read block 1237 len 40960
9258 006/300 Flush
9259 007/300 Read block 1930 len 12288
9260 008/300 Write block 3263 len 8192 data: 3 2
9261 009/300 Read block 3840 len 20480
9262 010/300 Write block 4331 len 4096 data: 4
9263 011/300 Write block 4994 len 12288 data: 2 3 2
9264 012/300 Write block 2577 len 16384 data: 4 4 3 3
9265 013/300 Write block 3068 len 32768 data: 2 2 2 2 2 2 2 3
9266 014/300 Flush
9267 015/300 Read block 261 len 12288
9268 016/300 Read block 2354 len 28672
9269 017/300 Write block 2310 len 40960 data: 3 2 2 2 3 2 2 2 2 2
9270 018/300 Read block 1706 len 8192
9271 019/300 Write block 810 len 12288 data: 3 3 3
9272 020/300 Write block 4810 len 12288 data: 3 3 4
9273 021/300 Read block 3887 len 4096
9274 022/300 Flush
9275 023/300 Write block 2170 len 24576 data: 2 2 2 2 2 2
9276 024/300 Write block 4529 len 24576 data: 3 3 3 3 3 3
9277 025/300 Write block 596 len 4096 data: 2
9278 026/300 Read block 919 len 20480
9279 027/300 Read block 3048 len 28672
9280 028/300 Read block 2885 len 36864
9281 029/300 Write block 4670 len 28672 data: 4 3 4 4 4 6 5
9282 030/300 Read block 3053 len 8192
9283 031/300 Read block 849 len 8192
9284 032/300 Write block 3095 len 28672 data: 2 2 2 2 2 2 2
9285 033/300 Read block 150 len 24576
9286 034/300 Read block 3221 len 40960
9287 035/300 Write block 1729 len 24576 data: 4 5 4 4 4 5
9288 036/300 Flush
9289 037/300 Read block 1202 len 32768
9290 038/300 Read block 1335 len 12288
9291 039/300 Read block 2225 len 16384
9292 040/300 Read block 1517 len 24576
9293 041/300 Write block 4287 len 16384 data: 2 2 2 2
9294 042/300 Read block 3217 len 28672
9295 043/300 Read block 4769 len 24576
9296 044/300 Write block 4111 len 28672 data: 5 5 6 6 5 5 5
9297 045/300 Read block 4283 len 4096
9298 046/300 Read block 959 len 4096
9299 047/300 Read block 3991 len 40960
9300 048/300 Write block 1119 len 36864 data: 3 3 3 3 3 3 2 2 2
9301 049/300 Read block 713 len 16384
9302 050/300 Read block 4538 len 16384
9303 051/300 Write block 2811 len 40960 data: 2 2 2 2 2 2 2 3 3 4
9304 052/300 Write block 2800 len 24576 data: 2 2 2 2 2 2
9305 053/300 Read block 4510 len 4096
9306 054/300 Write block 4417 len 8192 data: 4 6
9307 055/300 Read block 4459 len 36864
9308 056/300 Write block 1122 len 36864 data: 4 4 4 3 3 3 2 2 2
9309 057/300 Flush
9310 058/300 Write block 4687 len 32768 data: 4 4 4 4 5 4 3 3
9311 059/300 Read block 176 len 32768
9312 060/300 Read block 748 len 20480
9313 061/300 Write block 4307 len 24576 data: 3 3 3 3 3 4
9314 062/300 Write block 1 len 40960 data: 3 3 3 3 3 3 2 2 2 2
9315 063/300 Read block 1518 len 8192
9316 064/300 Read block 3809 len 8192
9317 065/300 Read block 266 len 28672
9318 066/300 Read block 2859 len 32768
9319 067/300 Write block 2343 len 16384 data: 2 2 2 2
9320 068/300 Read block 3013 len 32768
9321 069/300 Read block 3069 len 40960
9322 070/300 Read block 1397 len 8192
9323 071/300 Read block 207 len 32768
9324 072/300 Write block 3904 len 24576 data: 2 2 2 3 3 3
9325 073/300 Flush
9326 074/300 Read block 2208 len 20480
9327 075/300 Write block 2460 len 40960 data: 2 2 2 2 2 2 2 2 2 2
9328 076/300 Flush
9329 077/300 Read block 2671 len 4096
9330 078/300 Read block 4047 len 4096
9331 079/300 Flush
9332 080/300 Read block 1017 len 16384
9333 081/300 Read block 2765 len 28672
9334 082/300 Read block 3701 len 20480
9335 083/300 Read block 2708 len 8192
9336 084/300 Read block 1840 len 12288
9337 085/300 Write block 1449 len 12288 data: 2 2 2
9338 086/300 Write block 4798 len 32768 data: 2 2 2 2 2 2 2 2
9339 087/300 Read block 3555 len 36864
9340 088/300 Read block 2671 len 24576
9341 089/300 Write block 2822 len 28672 data: 3 4 4 4 4 3 3
9342 090/300 Write block 3320 len 12288 data: 2 2 2
9343 091/300 Read block 1318 len 36864
9344 092/300 Flush
9345 093/300 Write block 4723 len 24576 data: 4 3 3 3 3 3
9346 094/300 Read block 4149 len 40960
9347 095/300 Flush
9348 096/300 Flush
9349 097/300 Write block 2414 len 16384 data: 2 2 2 2
9350 098/300 Read block 2209 len 32768
9351 099/300 Read block 4945 len 28672
9352 100/300 Write block 1756 len 36864 data: 2 2 2 2 2 2 2 2 2
9353 101/300 Write block 1004 len 36864 data: 3 3 3 3 3 3 3 3 4
9354 102/300 Read block 2336 len 32768
9355 103/300 Read block 2774 len 20480
9356 104/300 Write block 368 len 16384 data: 3 4 5 5
9357 105/300 Read block 3961 len 8192
9358 106/300 Write block 1838 len 36864 data: 4 3 3 2 2 2 2 2 2
9359 107/300 Read block 2547 len 24576
9360 108/300 Write block 3425 len 32768 data: 3 2 2 2 2 2 2 2
9361 109/300 Read block 4374 len 20480
9362 110/300 Write block 1878 len 4096 data: 4
9363 111/300 Write block 3326 len 8192 data: 3 4
9364 112/300 Write block 727 len 12288 data: 3 3 2
9365 113/300 Read block 2106 len 4096
9366 114/300 Write block 4898 len 8192 data: 3 3
9367 115/300 Write block 1248 len 40960 data: 3 3 4 4 3 3 3 4 4 4
9368 116/300 Read block 1375 len 4096
9369 117/300 Read block 1873 len 8192
9370 118/300 Read block 4646 len 12288
9371 119/300 Write block 1639 len 24576 data: 3 3 5 7 7 7
9372 120/300 Write block 669 len 24576 data: 3 4 4 4 4 4
9373 121/300 Write block 2929 len 24576 data: 3 3 3 2 2 2
9374 122/300 Write block 4317 len 4096 data: 3
9375 123/300 Read block 771 len 40960
9376 124/300 Read block 2443 len 8192
9377 125/300 Write block 2418 len 8192 data: 2 2
9378 126/300 Flush
9379 127/300 Write block 1827 len 12288 data: 4 5 5
9380 128/300 Write block 2119 len 8192 data: 3 3
9381 129/300 Read block 4831 len 12288
9382 130/300 Read block 4112 len 12288
9383 131/300 Read block 1301 len 24576
9384 132/300 Write block 1827 len 4096 data: 5
9385 133/300 Write block 2126 len 8192 data: 4 5
9386 134/300 Flush
9387 135/300 Write block 3152 len 8192 data: 3 3
9388 136/300 Read block 2700 len 16384
9389 137/300 Write block 3266 len 40960 data: 2 3 3 3 3 3 3 3 3 3
9390 138/300 Write block 4871 len 12288 data: 4 4 3
9391 139/300 Read block 3610 len 32768
9392 140/300 Write block 818 len 36864 data: 4 4 4 4 3 3 2 2 3
9393 141/300 Read block 1099 len 36864
9394 142/300 Write block 4418 len 4096 data: 7
9395 143/300 Write block 2340 len 32768 data: 2 2 2 3 3 3 3 2
9396 144/300 Write block 4889 len 32768 data: 3 3 3 3 3 3 3 2
9397 145/300 Read block 2080 len 24576
9398 146/300 Read block 1887 len 12288
9399 147/300 Write block 350 len 24576 data: 4 4 4 4 4 4
9400 148/300 Read block 2211 len 12288
9401 149/300 Flush
9402 150/300 Write block 3526 len 16384 data: 3 3 3 4
9403 151/300 Write block 3639 len 36864 data: 2 2 2 2 2 2 2 2 2
9404 152/300 Write block 4024 len 36864 data: 4 5 4 4 5 5 4 3 3
9405 153/300 Read block 4104 len 40960
9406 154/300 Read block 799 len 12288
9407 155/300 Write block 2535 len 4096 data: 2
9408 156/300 Write block 4170 len 24576 data: 4 3 2 2 2 2
9409 157/300 Read block 4734 len 40960
9410 158/300 Read block 1200 len 20480
9411 159/300 Read block 1447 len 20480
9412 160/300 Write block 4674 len 12288 data: 5 7 6
9413 161/300 Read block 3125 len 36864
9414 162/300 Read block 3572 len 40960
9415 163/300 Read block 1558 len 40960
9416 164/300 Write block 1219 len 12288 data: 2 2 3
9417 165/300 Read block 778 len 36864
9418 166/300 Read block 4937 len 12288
9419 167/300 Write block 2814 len 20480 data: 3 3 3 3 4
9420 168/300 Write block 2166 len 28672 data: 3 3 2 2 3 3 3
9421 169/300 Read block 2746 len 28672
9422 170/300 Read block 3113 len 36864
9423 171/300 Read block 1701 len 12288
9424 172/300 Flush
9425 173/300 Write block 1489 len 32768 data: 3 2 2 2 3 3 3 2
9426 174/300 Write block 3549 len 36864 data: 2 2 2 2 2 2 2 2 3
9427 175/300 Write block 4424 len 32768 data: 5 5 5 5 4 4 3 3
9428 176/300 Write block 1421 len 12288 data: 4 4 3
9429 177/300 Write block 682 len 40960 data: 3 3 2 2 2 2 2 3 3 3
9430 178/300 Write block 3347 len 36864 data: 4 3 3 3 3 2 2 2 2
9431 179/300 Read block 3502 len 20480
9432 180/300 Write block 484 len 12288 data: 3 3 3
9433 181/300 Read block 1080 len 36864
9434 182/300 Write block 2948 len 28672 data: 2 2 2 3 3 4 4
9435 183/300 Read block 1099 len 36864
9436 184/300 Read block 398 len 36864
9437 185/300 Read block 3748 len 16384
9438 186/300 Read block 1163 len 28672
9439 187/300 Write block 1115 len 32768 data: 2 2 2 2 4 4 4 5
9440 188/300 Read block 3250 len 8192
9441 189/300 Read block 651 len 8192
9442 190/300 Read block 2010 len 12288
9443 191/300 Write block 2725 len 28672 data: 2 2 2 2 2 2 2
9444 192/300 Flush
9445 193/300 Write block 523 len 24576 data: 3 3 2 2 2 2
9446 194/300 Read block 1230 len 24576
9447 195/300 Flush
9448 196/300 Read block 1934 len 28672
9449 197/300 Read block 349 len 36864
9450 198/300 Read block 695 len 12288
9451 199/300 Read block 4404 len 16384
9452 200/300 Write block 2634 len 20480 data: 4 4 4 4 3
9453 201/300 Read block 2394 len 24576
9454 202/300 Read block 1997 len 32768
9455 203/300 Flush
9456 204/300 Write block 1192 len 40960 data: 4 4 4 4 3 4 4 4 3 3
9457 205/300 Write block 48 len 28672 data: 3 3 2 3 5 5 5
9458 206/300 Write block 442 len 32768 data: 3 3 3 4 4 4 4 3
9459 207/300 Flush
9460 208/300 Write block 616 len 12288 data: 5 7 7
9461 209/300 Read block 4411 len 12288
9462 210/300 Write block 3733 len 36864 data: 5 5 5 3 3 2 2 3 3
9463 211/300 Flush
9464 212/300 Write block 4446 len 20480 data: 4 3 3 4 4
9465 213/300 Write block 2939 len 8192 data: 3 3
9466 214/300 Read block 2929 len 24576
9467 215/300 Write block 1701 len 4096 data: 2
9468 216/300 Read block 3236 len 12288
9469 217/300 Read block 1328 len 8192
9470 218/300 Write block 1218 len 40960 data: 2 3 3 4 3 3 3 3 2 2
9471 219/300 Write block 2555 len 8192 data: 3 2
9472 220/300 Flush
9473 221/300 Write block 3902 len 24576 data: 2 2 3 3 3 4
9474 222/300 Read block 4034 len 20480
9475 223/300 Write block 2057 len 36864 data: 3 4 5 5 4 4 3 4 4
9476 224/300 Read block 867 len 16384
9477 225/300 Read block 2203 len 8192
9478 226/300 Read block 3266 len 40960
9479 227/300 Flush
9480 228/300 Write block 3917 len 16384 data: 2 2 2 3
9481 229/300 Write block 4468 len 8192 data: 3 3
9482 230/300 Write block 3830 len 20480 data: 3 3 3 4 3
9483 231/300 Flush
9484 232/300 Read block 2936 len 24576
9485 233/300 Read block 3589 len 28672
9486 234/300 Read block 3906 len 24576
9487 235/300 Write block 3657 len 36864 data: 3 3 3 3 3 4 4 3 2
9488 236/300 Write block 3483 len 24576 data: 3 3 3 3 3 2
9489 237/300 Read block 3434 len 24576
9490 238/300 Read block 4776 len 16384
9491 239/300 Write block 4328 len 40960 data: 4 4 4 5 5 5 4 4 4 4
9492 240/300 Read block 422 len 8192
9493 241/300 Flush
9494 242/300 Write block 437 len 16384 data: 4 4 3 3
9495 243/300 Write block 1705 len 12288 data: 2 2 2
9496 244/300 Read block 4105 len 40960
9497 245/300 Read block 2724 len 40960
9498 246/300 Write block 4189 len 40960 data: 3 3 3 3 3 3 3 2 2 3
9499 247/300 Read block 215 len 24576
9500 248/300 Read block 2496 len 20480
9501 249/300 Read block 3283 len 28672
9502 250/300 Flush
9503 251/300 Read block 635 len 12288
9504 252/300 Read block 1596 len 8192
9505 253/300 Read block 1298 len 40960
9506 254/300 Read block 3769 len 16384
9507 255/300 Flush
9508 256/300 Read block 2165 len 24576
9509 257/300 Flush
9510 258/300 Write block 4307 len 16384 data: 4 4 4 4
9511 259/300 Write block 4002 len 12288 data: 3 3 2
9512 260/300 Read block 3048 len 4096
9513 261/300 Write block 3244 len 40960 data: 3 3 3 3 4 3 3 3 3 4
9514 262/300 Read block 4346 len 20480
9515 263/300 Read block 1413 len 40960
9516 264/300 Write block 3285 len 16384 data: 3 3 3 3
9517 265/300 Read block 1140 len 8192
9518 266/300 Write block 963 len 20480 data: 4 4 4 3 3
9519 267/300 Read block 1389 len 24576
9520 268/300 Read block 4790 len 32768
9521 269/300 Read block 3103 len 32768
9522 270/300 Write block 3102 len 12288 data: 2 2 3
9523 271/300 Read block 2048 len 32768
9524 272/300 Read block 3795 len 12288
9525 273/300 Read block 3097 len 36864
9526 274/300 Write block 4138 len 36864 data: 4 5 5 4 4 4 4 4 2
9527 275/300 Write block 1792 len 36864 data: 2 2 2 2 3 3 3 3 2
9528 276/300 Read block 1114 len 12288
9529 277/300 Read block 4028 len 36864
9530 278/300 Read block 4751 len 28672
9531 279/300 Read block 4463 len 28672
9532 280/300 Write block 3714 len 20480 data: 3 3 4 3 3
9533 281/300 Write block 3397 len 32768 data: 3 4 4 4 4 5 5 3
9534 282/300 Read block 2366 len 20480
9535 283/300 Write block 1763 len 20480 data: 3 3 2 2 2
9536 284/300 Read block 4272 len 40960
9537 285/300 Write block 1188 len 4096 data: 3
9538 286/300 Read block 4009 len 32768
9539 287/300 Write block 1435 len 4096 data: 2
9540 288/300 Read block 3538 len 40960
9541 289/300 Write block 946 len 24576 data: 2 2 2 2 2 2
9542 290/300 Read block 498 len 20480
9543 291/300 Read block 3161 len 12288
9544 292/300 Read block 4978 len 4096
9545 293/300 Read block 2031 len 4096
9546 294/300 Read block 51 len 28672
9547 295/300 Read block 522 len 36864
9548 296/300 Write block 3275 len 16384 data: 4 3 2 3
9549 297/300 Read block 4056 len 32768
9550 298/300 Read block 2499 len 20480
9551 299/300 Read block 1145 len 32768
9552 300/300 Read block 1073 len 8192
9553 Main task has work to do, go find it
9554 got dsc Start(2)
9555 start 2
9556 Feb 15 02:59:25.673 INFO request completed, latency_us: 218, response_code: 204, uri: /start/cid/2, method: POST, req_id: 5e39e6fb-8bdb-4433-b116-0f217736455f, remote_addr: 127.0.0.1:54744, local_addr: 127.0.0.1:9998
9557 Replay: started 2, returned:Ok(())
9558 ----------------------------------------------------------------
9559 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:300
9560 States: Active Active Offline
9561 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
9562 2412 Acked 3411 Flush 0 Done Done New true
9563 2413 Acked 3412 Write 10 Done Done New true
9564 2414 Acked 3413 Write 8 Done Done New true
9565 2415 Acked 3414 Read 1 Done Done New true
9566 2416 Acked 3415 Read 10 Done Done New false
9567 2417 Acked 3416 Flush 0 Done Done New false
9568 2418 Acked 3417 Read 3 Done Done New false
9569 2419 Acked 3418 Write 2 Done Done New false
9570 2420 Acked 3419 Read 5 Done Done New false
9571 2421 Acked 3420 Write 1 Done Done New false
9572 2422 Acked 3421 Write 3 Done Done New false
9573 2423 Acked 3422 Write 4 Done Done New false
9574 2424 Acked 3423 Write 8 Done Done New false
9575 2425 Acked 3424 Flush 0 Done Done New false
9576 2426 Acked 3425 Read 3 Done Done New false
9577 2427 Acked 3426 Read 7 Done Done New false
9578 2428 Acked 3427 Write 10 Done Done New false
9579 2429 Acked 3428 Read 2 Done Done New false
9580 2430 Acked 3429 Write 3 Done Done New false
9581 2431 Acked 3430 Write 3 Done Done New false
9582 2432 Acked 3431 Read 1 Done Done New false
9583 2433 Acked 3432 Flush 0 Done Done New false
9584 2434 Acked 3433 Write 6 Done Done New false
9585 2435 Acked 3434 Write 6 Done Done New false
9586 2436 Acked 3435 Write 1 Done Done New false
9587 2437 Acked 3436 Read 5 Done Done New false
9588 2438 Acked 3437 Read 7 Done Done New false
9589 2439 Acked 3438 Read 9 Done Done New false
9590 2440 Acked 3439 Write 7 Done Done New false
9591 2441 Acked 3440 Read 2 Done Done New false
9592 2442 Acked 3441 Read 2 Done Done New false
9593 2443 Acked 3442 Write 7 Done Done New false
9594 2444 Acked 3443 Read 6 Done Done New false
9595 2445 Acked 3444 Read 10 Done Done New false
9596 2446 Acked 3445 Write 6 Done Done New false
9597 2447 Acked 3446 Flush 0 Done Done New false
9598 2448 Acked 3447 Read 8 Done Done New false
9599 2449 Acked 3448 Read 3 Done Done New false
9600 2450 Acked 3449 Read 4 Done Done New false
9601 2451 Acked 3450 Read 6 Done Done New false
9602 2452 Acked 3451 Write 4 Done Done New false
9603 2453 Acked 3452 Read 7 Done Done New false
9604 2454 Acked 3453 Read 6 Done Done New false
9605 2455 Acked 3454 Write 7 Done Done New false
9606 2456 Acked 3455 Read 1 Done Done New false
9607 2457 Acked 3456 Read 1 Done Done New false
9608 2458 Acked 3457 Read 10 Done Done New false
9609 2459 Acked 3458 Write 9 Done Done New false
9610 2460 Acked 3459 Read 4 Done Done New false
9611 2461 Acked 3460 Read 4 Done Done New false
9612 2462 Acked 3461 Write 10 Done Done New false
9613 2463 Acked 3462 Write 6 Done Done New false
9614 2464 Acked 3463 Read 1 Done Done New false
9615 2465 Acked 3464 Write 2 Done Done New false
9616 2466 Acked 3465 Read 9 Done Done New false
9617 2467 Acked 3466 Write 9 Done Done New false
9618 2468 Acked 3467 Flush 0 Done Done New false
9619 2469 Acked 3468 Write 8 Done Done New false
9620 2470 Acked 3469 Read 8 Done Done New false
9621 2471 Acked 3470 Read 5 Done Done New false
9622 2472 Acked 3471 Write 6 Done Done New false
9623 2473 Acked 3472 Write 10 Done Done New false
9624 2474 Acked 3473 Read 2 Done Done New false
9625 2475 Acked 3474 Read 2 Done Done New false
9626 2476 Acked 3475 Read 7 Done Done New false
9627 2477 Acked 3476 Read 8 Done Done New false
9628 2478 Acked 3477 Write 4 Done Done New false
9629 2479 Acked 3478 Read 8 Done Done New false
9630 2480 Acked 3479 Read 10 Done Done New false
9631 2481 Acked 3480 Read 2 Done Done New false
9632 2482 Acked 3481 Read 8 Done Done New false
9633 2483 Acked 3482 Write 6 Done Done New false
9634 2484 Acked 3483 Flush 0 Done Done New false
9635 2485 Acked 3484 Read 5 Done Done New false
9636 2486 Acked 3485 Write 10 Done Done New false
9637 2487 Acked 3486 Flush 0 Done Done New false
9638 2488 Acked 3487 Read 1 Done Done New false
9639 2489 Acked 3488 Read 1 Done Done New false
9640 2490 Acked 3489 Flush 0 Done Done New false
9641 2491 Acked 3490 Read 4 Done Done New false
9642 2492 Acked 3491 Read 7 Done Done New false
9643 2493 Acked 3492 Read 5 Done Done New false
9644 2494 Acked 3493 Read 2 Done Done New false
9645 2495 Acked 3494 Read 3 Done Done New false
9646 2496 Acked 3495 Write 3 Done Done New false
9647 2497 Acked 3496 Write 8 Done Done New false
9648 2498 Acked 3497 Read 9 Done Done New false
9649 2499 Acked 3498 Read 6 Done Done New false
9650 2500 Acked 3499 Write 7 Done Done New false
9651 2501 Acked 3500 Write 3 Done Done New false
9652 2502 Acked 3501 Read 9 Done Done New false
9653 2503 Acked 3502 Flush 0 Done Done New false
9654 2504 Acked 3503 Write 6 Done Done New false
9655 2505 Acked 3504 Read 10 Done Done New false
9656 2506 Acked 3505 Flush 0 Done Done New false
9657 2507 Acked 3506 Flush 0 Done Done New false
9658 2508 Acked 3507 Write 4 Done Done New false
9659 2509 Acked 3508 Read 8 Done Done New false
9660 2510 Acked 3509 Read 7 Done Done New false
9661 2511 Acked 3510 Write 9 Done Done New false
9662 2512 Acked 3511 Write 9 Done Done New false
9663 2513 Acked 3512 Read 8 Done Done New false
9664 2514 Acked 3513 Read 5 Done Done New false
9665 2515 Acked 3514 Write 4 Done Done New false
9666 2516 Acked 3515 Read 2 Done Done New false
9667 2517 Acked 3516 Write 9 Done Done New false
9668 2518 Acked 3517 Read 6 Done Done New false
9669 2519 Acked 3518 Write 8 Done Done New false
9670 2520 Acked 3519 Read 5 Done Done New false
9671 2521 Acked 3520 Write 1 Done Done New false
9672 2522 Acked 3521 Write 2 Done Done New false
9673 2523 Acked 3522 Write 3 Done Done New false
9674 2524 Acked 3523 Read 1 Done Done New false
9675 2525 Acked 3524 Write 2 Done Done New false
9676 2526 Acked 3525 Write 10 Done Done New false
9677 2527 Acked 3526 Read 1 Done Done New false
9678 2528 Acked 3527 Read 2 Done Done New false
9679 2529 Acked 3528 Read 3 Done Done New false
9680 2530 Acked 3529 Write 6 Done Done New false
9681 2531 Acked 3530 Write 6 Done Done New false
9682 2532 Acked 3531 Write 6 Done Done New false
9683 2533 Acked 3532 Write 1 Done Done New false
9684 2534 Acked 3533 Read 10 Done Done New false
9685 2535 Acked 3534 Read 2 Done Done New false
9686 2536 Acked 3535 Write 2 Done Done New false
9687 2537 Acked 3536 Flush 0 Done Done New false
9688 2538 Acked 3537 Write 3 Done Done New false
9689 2539 Acked 3538 Write 2 Done Done New false
9690 2540 Acked 3539 Read 3 Done Done New false
9691 2541 Acked 3540 Read 3 Done Done New false
9692 2542 Acked 3541 Read 6 Done Done New false
9693 2543 Acked 3542 Write 1 Done Done New false
9694 2544 Acked 3543 Write 2 Done Done New false
9695 2545 Acked 3544 Flush 0 Done Done New false
9696 2546 Acked 3545 Write 2 Done Done New false
9697 2547 Acked 3546 Read 4 Done Done New false
9698 2548 Acked 3547 Write 10 Done Done New false
9699 2549 Acked 3548 Write 3 Done Done New false
9700 2550 Acked 3549 Read 8 Done Done New false
9701 2551 Acked 3550 Write 9 Done Done New false
9702 2552 Acked 3551 Read 9 Done Done New false
9703 2553 Acked 3552 Write 1 Done Done New false
9704 2554 Acked 3553 Write 8 Done Done New false
9705 2555 Acked 3554 Write 8 Done Done New false
9706 2556 Acked 3555 Read 6 Done Done New false
9707 2557 Acked 3556 Read 3 Done Done New false
9708 2558 Acked 3557 Write 6 Done Done New false
9709 2559 Acked 3558 Read 3 Done Done New false
9710 2560 Acked 3559 Flush 0 Done Done New false
9711 2561 Acked 3560 Write 4 Done Done New false
9712 2562 Acked 3561 Write 9 Done Done New false
9713 2563 Acked 3562 Write 9 Done Done New false
9714 2564 Acked 3563 Read 10 Done Done New false
9715 2565 Acked 3564 Read 3 Done Done New false
9716 2566 Acked 3565 Write 1 Done Done New false
9717 2567 Acked 3566 Write 6 Done Done New false
9718 2568 Acked 3567 Read 10 Done Done New false
9719 2569 Acked 3568 Read 5 Done Done New false
9720 2570 Acked 3569 Read 5 Done Done New false
9721 2571 Acked 3570 Write 3 Done Done New false
9722 2572 Acked 3571 Read 9 Done Done New false
9723 2573 Acked 3572 Read 10 Done Done New false
9724 2574 Acked 3573 Read 10 Done Done New false
9725 2575 Acked 3574 Write 3 Done Done New false
9726 2576 Acked 3575 Read 9 Done Done New false
9727 2577 Acked 3576 Read 3 Done Done New false
9728 2578 Acked 3577 Write 5 Done Done New false
9729 2579 Acked 3578 Write 7 Done Done New false
9730 2580 Acked 3579 Read 7 Done Done New false
9731 2581 Acked 3580 Read 9 Done Done New false
9732 2582 Acked 3581 Read 3 Done Done New false
9733 2583 Acked 3582 Flush 0 Done Done New false
9734 2584 Acked 3583 Write 8 Done Done New false
9735 2585 Acked 3584 Write 9 Done Done New false
9736 2586 Acked 3585 Write 8 Done Done New false
9737 2587 Acked 3586 Write 3 Done Done New false
9738 2588 Acked 3587 Write 10 Done Done New false
9739 2589 Acked 3588 Write 9 Done Done New false
9740 2590 Acked 3589 Read 5 Done Done New false
9741 2591 Acked 3590 Write 3 Done Done New false
9742 2592 Acked 3591 Read 9 Done Done New false
9743 2593 Acked 3592 Write 7 Done Done New false
9744 2594 Acked 3593 Read 9 Done Done New false
9745 2595 Acked 3594 Read 9 Done Done New false
9746 2596 Acked 3595 Read 4 Done Done New false
9747 2597 Acked 3596 Read 7 Done Done New false
9748 2598 Acked 3597 Write 8 Done Done New false
9749 2599 Acked 3598 Read 2 Done Done New false
9750 2600 Acked 3599 Read 2 Done Done New false
9751 2601 Acked 3600 Read 3 Done Done New false
9752 2602 Acked 3601 Write 7 Done Done New false
9753 2603 Acked 3602 Flush 0 Done Done New false
9754 2604 Acked 3603 Write 6 Done Done New false
9755 2605 Acked 3604 Read 6 Done Done New false
9756 2606 Acked 3605 Flush 0 Done Done New false
9757 2607 Acked 3606 Read 7 Done Done New false
9758 2608 Acked 3607 Read 9 Done Done New false
9759 2609 Acked 3608 Read 3 Done Done New false
9760 2610 Acked 3609 Read 4 Done Done New false
9761 2611 Acked 3610 Write 5 Done Done New false
9762 2612 Acked 3611 Read 6 Done Done New false
9763 2613 Acked 3612 Read 8 Done Done New false
9764 2614 Acked 3613 Flush 0 Done Done New false
9765 2615 Acked 3614 Write 10 Done Done New false
9766 2616 Acked 3615 Write 7 Done Done New false
9767 2617 Acked 3616 Write 8 Done Done New false
9768 2618 Acked 3617 Flush 0 Done Done New false
9769 2619 Acked 3618 Write 3 Done Done New false
9770 2620 Acked 3619 Read 3 Done Done New false
9771 2621 Acked 3620 Write 9 Done Done New false
9772 2622 Acked 3621 Flush 0 Done Done New false
9773 2623 Acked 3622 Write 5 Done Done New false
9774 2624 Acked 3623 Write 2 Done Done New false
9775 2625 Acked 3624 Read 6 Done Done New false
9776 2626 Acked 3625 Write 1 Done Done New false
9777 2627 Acked 3626 Read 3 Done Done New false
9778 2628 Acked 3627 Read 2 Done Done New false
9779 2629 Acked 3628 Write 10 Done Done New false
9780 2630 Acked 3629 Write 2 Done Done New false
9781 2631 Acked 3630 Flush 0 Done Done New false
9782 2632 Acked 3631 Write 6 Done Done New false
9783 2633 Acked 3632 Read 5 Done Done New false
9784 2634 Acked 3633 Write 9 Done Done New false
9785 2635 Acked 3634 Read 4 Done Done New false
9786 2636 Acked 3635 Read 2 Done Done New false
9787 2637 Acked 3636 Read 10 Done Done New false
9788 2638 Acked 3637 Flush 0 Done Done New false
9789 2639 Acked 3638 Write 4 Done Done New false
9790 2640 Acked 3639 Write 2 Done Done New false
9791 2641 Acked 3640 Write 5 Done Done New false
9792 2642 Acked 3641 Flush 0 Done Done New false
9793 2643 Acked 3642 Read 6 Done Done New false
9794 2644 Acked 3643 Read 7 Done Done New false
9795 2645 Acked 3644 Read 6 Done Done New false
9796 2646 Acked 3645 Write 9 Done Done New false
9797 2647 Acked 3646 Write 6 Done Done New false
9798 2648 Acked 3647 Read 6 Done Done New false
9799 2649 Acked 3648 Read 4 Done Done New false
9800 2650 Acked 3649 Write 10 Done Done New false
9801 2651 Acked 3650 Read 2 Done Done New false
9802 2652 Acked 3651 Flush 0 Done Done New false
9803 2653 Acked 3652 Write 4 Done Done New false
9804 2654 Acked 3653 Write 3 Done Done New false
9805 2655 Acked 3654 Read 10 Done Done New false
9806 2656 Acked 3655 Read 10 Done Done New false
9807 2657 Acked 3656 Write 10 Done Done New false
9808 2658 Acked 3657 Read 6 Done Done New false
9809 2659 Acked 3658 Read 5 Done Done New false
9810 2660 Acked 3659 Read 7 Done Done New false
9811 2661 Acked 3660 Flush 0 Done Done New false
9812 2662 Acked 3661 Read 3 Done Done New false
9813 2663 Acked 3662 Read 2 Done Done New false
9814 2664 Acked 3663 Read 10 Done Done New false
9815 2665 Acked 3664 Read 4 Done Done New false
9816 2666 Acked 3665 Flush 0 Done Done New false
9817 2667 Acked 3666 Read 6 Done Done New false
9818 2668 Acked 3667 Flush 0 Done Done New false
9819 2669 Acked 3668 Write 4 Done Done New false
9820 2670 Acked 3669 Write 3 Done Done New false
9821 2671 Acked 3670 Read 1 Done Done New false
9822 2672 Acked 3671 Write 10 Done Done New false
9823 2673 Acked 3672 Read 5 Done Done New false
9824 2674 Acked 3673 Read 10 Done Done New false
9825 2675 Acked 3674 Write 4 Done Done New false
9826 2676 Acked 3675 Read 2 Done Done New false
9827 2677 Acked 3676 Write 5 Done Done New false
9828 2678 Acked 3677 Read 6 Done Done New false
9829 2679 Acked 3678 Read 8 Done Done New false
9830 2680 Acked 3679 Read 8 Done Done New false
9831 2681 Acked 3680 Write 3 Done Done New false
9832 2682 Acked 3681 Read 8 Done Done New false
9833 2683 Acked 3682 Read 3 Done Done New false
9834 2684 Acked 3683 Read 9 Done Done New false
9835 2685 Acked 3684 Write 9 Done Done New false
9836 2686 Acked 3685 Write 9 Done Done New false
9837 2687 Acked 3686 Read 3 Done Done New false
9838 2688 Acked 3687 Read 9 Done Done New false
9839 2689 Acked 3688 Read 7 Done Done New false
9840 2690 Acked 3689 Read 7 Done Done New false
9841 2691 Acked 3690 Write 5 Done Done New false
9842 2692 Acked 3691 Write 8 Done Done New false
9843 2693 Acked 3692 Read 5 Done Done New false
9844 2694 Acked 3693 Write 5 Done Done New false
9845 2695 Acked 3694 Read 10 Done Done New false
9846 2696 Acked 3695 Write 1 Done Done New false
9847 2697 Acked 3696 Read 8 Done Done New false
9848 2698 Acked 3697 Write 1 Done Done New false
9849 2699 Acked 3698 Read 10 Done Done New false
9850 2700 Acked 3699 Write 6 Done Done New false
9851 2701 Acked 3700 Read 5 Done Done New false
9852 2702 Acked 3701 Read 3 Done Done New false
9853 2703 Acked 3702 Read 1 Done Done New false
9854 2704 Acked 3703 Read 1 Done Done New false
9855 2705 Acked 3704 Read 7 Done Done New false
9856 2706 Acked 3705 Read 9 Done Done New false
9857 2707 Acked 3706 Write 4 Done Done New false
9858 2708 Acked 3707 Read 8 Done Done New false
9859 2709 Acked 3708 Read 5 Done Done New false
9860 2710 Acked 3709 Read 8 Done Done New false
9861 2711 Acked 3710 Read 2 Done Done New false
9862 STATES DS:0 DS:1 DS:2 TOTAL
9863 New 0 0 300 300
9864 Sent 0 0 0 0
9865 Done 300 300 0 600
9866 Skipped 0 0 0 0
9867 Error 0 0 0 0
9868 Last Flush: 3667 3667 3410
9869 Downstairs last five completed: 3410 3409 3408 3407 3406
9870 Upstairs last five completed: 2711 2710 2709 2708 2707
9871 CLIENT: Up:0 ds:300 act:2
98722024-02-15T02:59:26.340ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
98732024-02-15T02:59:26.340ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
98742024-02-15T02:59:26.340ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
98752024-02-15T02:59:26.340ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
98762024-02-15T02:59:26.340ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
98772024-02-15T02:59:26.340ZINFOcrucible: [2] client re-new 301 jobs since flush 3410 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
98782024-02-15T02:59:26.340ZINFOcrucible: client task is exiting = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
9879 [8830] got start action
9880 [8830] I am going to restart
9881 Starting downstairs at port 8830
9882 Make output file at "/tmp/dsc/downstairs-8830.txt"
9883 [8830][2] reports Starting
98842024-02-15T02:59:27.342ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
98852024-02-15T02:59:27.342ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
98862024-02-15T02:59:27.342ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
98872024-02-15T02:59:27.342ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
98882024-02-15T02:59:27.342ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
98892024-02-15T02:59:27.342ZINFOcrucible: [2] client re-new 301 jobs since flush 3410 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
98902024-02-15T02:59:27.342ZINFOcrucible: client task is exiting = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
9891 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1176)
9892 [8830][2] reports Running
98932024-02-15T02:59:28.343ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
98942024-02-15T02:59:28.343ZINFOcrucible: ds_connection connected = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
98952024-02-15T02:59:28.345ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
98962024-02-15T02:59:28.345ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
98972024-02-15T02:59:28.345ZINFOcrucible: send last flush ID to this DS: 3410 = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
98982024-02-15T02:59:28.345ZINFOcrucible: Replied this last flush ID: 3410 = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
98992024-02-15T02:59:28.345ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
99002024-02-15T02:59:28.345ZINFOcrucible: [2] Transition from Offline to Replay = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
99012024-02-15T02:59:28.345ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
99022024-02-15T02:59:28.345ZINFOcrucible: [2] Transition from Replay to Active = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
9903 ----------------------------------------------------------------
9904 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
9905 Downstairs last five completed: 3711 3710 3709 3708 3707
9906 Upstairs last five completed: 2712 2711 2710 2709 2708
9907 CLIENT: Up:0 ds:0 act:3
9908 Replay: All jobs finished, all DS active.
9909 Main task has work to do, go find it
9910 got dsc Stop(0)
9911 stop 0
9912 Feb 15 02:59:29.677 INFO request completed, latency_us: 375, response_code: 204, uri: /stop/cid/0, method: POST, req_id: 260272de-fa34-45ae-bd3a-f96516414932, remote_addr[8810] Got stop action so:false kr:false
9913 : 127.0.0.1:54744, local_addr: 127.0.0.1:9998
99142024-02-15T02:59:29.677ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
99152024-02-15T02:59:29.677ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
9916 Feb 15 02:59:29.677 INFO request completed, {"msg":"downstairs task for 0 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-02-15T02:59:29.677827318Z","hostname":"ip-10-150-1-67.us-west-2.compute.internal","pid":1167,"session_id":"4f5a63e5-aa00-4890-9f6b-96c2e069fdb2"}
9917 latency_us: 102, response_code: 200, uri: /state/cid/0, method{"msg":"Gone missing, transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-02-15T02:59:29.677843956Z","hostname":"ip-10-150-1-67.us-west-2.compute.internal","pid":1167,"client":"0","":"downstairs","session_id":"4f5a63e5-aa00-4890-9f6b-96c2e069fdb2"}
9918 : GET, req_id: a5a3352a-32df-4d5e-a738-c917fd4f9dac, remote_addr: 127.0.0.1:54744, local_addr: 127.0.0.1:9998
99192024-02-15T02:59:29.677ZINFOcrucible: [0] client re-new 0 jobs since flush 3711 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
99202024-02-15T02:59:29.677ZINFOcrucible: client task is exiting = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
9921 [8810] Exited with: ExitStatus(unix_wait_status(9))
9922 [8810][0] reports Exit
99232024-02-15T02:59:30.679ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
99242024-02-15T02:59:30.679ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
99252024-02-15T02:59:30.679ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
99262024-02-15T02:59:30.679ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
99272024-02-15T02:59:30.679ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
99282024-02-15T02:59:30.679ZINFOcrucible: [0] client re-new 0 jobs since flush 3711 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
99292024-02-15T02:59:30.679ZINFOcrucible: client task is exiting = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
99302024-02-15T02:59:31.680ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
99312024-02-15T02:59:31.680ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
99322024-02-15T02:59:31.680ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
99332024-02-15T02:59:31.680ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
99342024-02-15T02:59:31.680ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
99352024-02-15T02:59:31.680ZINFOcrucible: [0] client re-new 0 jobs since flush 3711 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
99362024-02-15T02:59:31.680ZINFOcrucible: client task is exiting = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
99372024-02-15T02:59:32.681ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
99382024-02-15T02:59:32.682ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
99392024-02-15T02:59:32.682ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
99402024-02-15T02:59:32.682ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
99412024-02-15T02:59:32.682ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
99422024-02-15T02:59:32.682ZINFOcrucible: [0] client re-new 0 jobs since flush 3711 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
99432024-02-15T02:59:32.682ZINFOcrucible: client task is exiting = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
9944 Feb 15 02:59:33.679 INFO request completed, latency_us: 195, response_code: 200, uri: /state/cid/0, method: GET, req_id: 68d69334-f06f-456e-a5e1-6ac194af8526, remote_addr: 127.0.0.1:54744, local_addr: 127.0.0.1:9998
9945 001/300 Write block 371 len 40960 data: 6 5 4 4 3 5 6 6 5 5
9946 002/300 Flush
99472024-02-15T02:59:33.683ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
99482024-02-15T02:59:33.683ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
99492024-02-15T02:59:33.683ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
99502024-02-15T02:59:33.683ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
99512024-02-15T02:59:33.683ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
99522024-02-15T02:59:33.683ZINFOcrucible: [0] client re-new 2 jobs since flush 3711 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
99532024-02-15T02:59:33.683ZINFOcrucible: client task is exiting = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
9954 003/300 Read block 2645 len 36864
9955 004/300 Read block 529 len 4096
9956 005/300 Read block 3599 len 16384
9957 006/300 Write block 16 len 36864 data: 3 3 3 2 3 5 5 6 6
9958 007/300 Write block 881 len 16384 data: 2 2 2 2
9959 008/300 Write block 799 len 32768 data: 4 2 2 2 2 2 2 2
9960 009/300 Read block 3039 len 32768
9961 010/300 Read block 4961 len 36864
9962 011/300 Write block 498 len 16384 data: 4 3 2 2
9963 012/300 Read block 1920 len 4096
9964 013/300 Read block 1111 len 36864
9965 014/300 Read block 374 len 12288
9966 015/300 Write block 2015 len 32768 data: 2 2 2 3 4 3 3 3
9967 016/300 Read block 524 len 16384
9968 017/300 Read block 4235 len 28672
9969 018/300 Write block 86 len 24576 data: 2 2 2 2 2 3
9970 019/300 Read block 446 len 8192
9971 020/300 Read block 3912 len 32768
9972 021/300 Read block 2270 len 8192
9973 022/300 Read block 2770 len 28672
9974 023/300 Write block 4605 len 8192 data: 2 2
9975 024/300 Write block 3983 len 28672 data: 3 3 3 3 4 4 4
9976 025/300 Write block 4629 len 40960 data: 4 3 2 2 2 2 2 3 3 3
9977 026/300 Read block 1928 len 24576
9978 027/300 Write block 141 len 32768 data: 2 3 3 3 3 2 2 3
9979 028/300 Write block 694 len 40960 data: 3 3 3 2 2 3 3 3 3 3
9980 029/300 Flush
9981 030/300 Write block 3813 len 12288 data: 2 2 2
9982 031/300 Write block 3643 len 40960 data: 3 3 3 3 3 2 3 3 3 3
9983 032/300 Read block 3889 len 28672
9984 033/300 Write block 744 len 28672 data: 3 2 2 2 2 3 3
9985 034/300 Read block 776 len 8192
9986 035/300 Read block 4508 len 4096
9987 036/300 Read block 2862 len 4096
9988 037/300 Write block 1039 len 20480 data: 3 3 3 3 2
9989 038/300 Write block 2380 len 20480 data: 3 3 3 2 2
9990 039/300 Write block 4332 len 12288 data: 6 6 5
9991 040/300 Read block 1456 len 24576
9992 041/300 Read block 857 len 24576
9993 042/300 Flush
9994 043/300 Write block 1691 len 36864 data: 3 3 2 2 2 2 2 2 2
9995 044/300 Write block 4298 len 4096 data: 2
9996 045/300 Read block 3712 len 24576
9997 046/300 Read block 4545 len 16384
9998 047/300 Read block 777 len 40960
9999 048/300 Write block 4259 len 20480 data: 4 4 5 5 4
10000 049/300 Read block 572 len 20480
10001 050/300 Read block 1907 len 28672
10002 051/300 Read block 3819 len 32768
10003 052/300 Write block 3607 len 32768 data: 2 2 2 3 3 3 3 4
10004 053/300 Write block 745 len 32768 data: 3 3 3 3 4 4 3 3
10005 054/300 Write block 1264 len 28672 data: 2 2 2 2 2 2 2
10006 055/300 Read block 117 len 12288
10007 056/300 Write block 136 len 12288 data: 3 2 2
10008 057/300 Write block 601 len 32768 data: 4 4 4 4 4 4 3 2
10009 058/300 Read block 4001 len 8192
10010 059/300 Write block 2559 len 24576 data: 2 2 2 2 2 2
10011 060/300 Read block 79 len 16384
10012 061/300 Write block 2835 len 28672 data: 2 3 3 3 3 3 3
10013 062/300 Read block 2714 len 12288
10014 063/300 Read block 351 len 32768
10015 064/300 Write block 299 len 16384 data: 2 3 3 3
10016 065/300 Write block 3860 len 28672 data: 2 2 2 3 3 4 4
10017 066/300 Write block 2132 len 20480 data: 4 5 3 3 3
10018 067/300 Write block 1403 len 24576 data: 2 3 3 3 3 3
10019 068/300 Write block 4968 len 16384 data: 5 6 5 5
10020 069/300 Write block 3305 len 32768 data: 4 3 3 3 3 3 2 2
10021 070/300 Write block 751 len 28672 data: 4 4 3 3 4 4 4
10022 071/300 Read block 8 len 12288
10023 072/300 Read block 4390 len 36864
10024 073/300 Write block 4748 len 24576 data: 3 3 4 4 4 5
10025 074/300 Write block 1991 len 20480 data: 2 2 2 2 3
10026 075/300 Flush
10027 076/300 Read block 1856 len 4096
10028 077/300 Flush
10029 078/300 Write block 2778 len 32768 data: 2 2 2 2 2 2 2 2
10030 079/300 Read block 3840 len 16384
10031 080/300 Write block 4549 len 4096 data: 4
10032 081/300 Read block 491 len 24576
10033 082/300 Write block 4080 len 16384 data: 2 2 2 2
10034 083/300 Write block 1189 len 16384 data: 3 3 4 5
10035 084/300 Write block 3265 len 16384 data: 2 3 4 4
10036 085/300 Write block 2010 len 12288 data: 3 4 3
10037 086/300 Read block 4923 len 20480
10038 087/300 Read block 3064 len 40960
10039 088/300 Read block 3782 len 4096
10040 089/300 Read block 3089 len 32768
10041 090/300 Write block 3296 len 20480 data: 3 3 4 2 2
10042 091/300 Write block 4455 len 16384 data: 3 3 3 3
10043 092/300 Read block 3494 len 8192
10044 093/300 Read block 1933 len 36864
10045 094/300 Flush
10046 095/300 Read block 4985 len 20480
10047 096/300 Read block 494 len 24576
10048 097/300 Write block 1054 len 32768 data: 2 2 2 2 2 4 3 3
10049 098/300 Write block 584 len 20480 data: 6 5 4 3 3
10050 099/300 Write block 481 len 28672 data: 2 2 2 4 4 4 3
10051 100/300 Read block 856 len 24576
10052 101/300 Flush
10053 102/300 Flush
10054 103/300 Write block 3812 len 20480 data: 2 3 3 3 2
10055 104/300 Write block 4003 len 12288 data: 4 3 2
10056 105/300 Write block 1514 len 32768 data: 3 3 3 3 2 2 2 2
10057 106/300 Read block 4666 len 36864
10058 107/300 Read block 3576 len 12288
10059 108/300 Write block 4040 len 16384 data: 3 3 4 4
10060 109/300 Read block 3204 len 40960
10061 110/300 Write block 2453 len 36864 data: 3 2 2 2 2 2 2 3 3
10062 111/300 Read block 2850 len 16384
10063 112/300 Read block 1993 len 40960
10064 113/300 Write block 711 len 40960 data: 3 3 4 4 4 4 2 3 4 5
10065 114/300 Flush
10066 115/300 Read block 3151 len 32768
10067 116/300 Read block 4595 len 12288
10068 117/300 Flush
10069 118/300 Write block 4524 len 4096 data: 2
10070 119/300 Read block 2721 len 28672
10071 120/300 Flush
10072 121/300 Read block 41 len 36864
10073 122/300 Write block 97 len 8192 data: 2 2
10074 123/300 Read block 1864 len 8192
10075 124/300 Write block 2142 len 8192 data: 3 4
10076 125/300 Write block 784 len 20480 data: 3 3 2 3 3
10077 126/300 Read block 270 len 24576
10078 127/300 Read block 3850 len 12288
10079 128/300 Read block 2940 len 20480
10080 129/300 Read block 1952 len 40960
10081 130/300 Read block 3856 len 28672
10082 131/300 Write block 3781 len 40960 data: 3 3 3 2 2 2 2 2 3 2
10083 132/300 Write block 2292 len 40960 data: 5 5 5 4 3 3 3 2 2 2
10084 133/300 Write block 3619 len 36864 data: 4 4 4 4 5 5 5 5 4
10085 134/300 Write block 883 len 36864 data: 3 3 2 2 3 3 3 3 3
10086 135/300 Read block 4002 len 20480
10087 136/300 Write block 3872 len 12288 data: 4 4 3
10088 137/300 Write block 451 len 28672 data: 3 3 2 2 2 2 2
10089 138/300 Read block 422 len 32768
10090 139/300 Write block 2123 len 32768 data: 3 3 3 5 6 5 4 4
10091 140/300 Write block 2023 len 28672 data: 3 3 3 2 2 2 2
10092 141/300 Write block 3626 len 8192 data: 6 5
10093 142/300 Write block 3145 len 24576 data: 4 4 4 4 4 5
10094 143/300 Read block 1000 len 16384
10095 144/300 Write block 3762 len 8192 data: 2 2
10096 145/300 Read block 2147 len 36864
10097 146/300 Read block 425 len 16384
10098 147/300 Write block 4247 len 24576 data: 3 5 4 5 5 4
10099 148/300 Flush
10100 149/300 Read block 4909 len 20480
10101 150/300 Read block 2434 len 16384
10102 151/300 Read block 4429 len 16384
10103 152/300 Write block 1288 len 8192 data: 4 4
10104 153/300 Read block 3935 len 28672
10105 154/300 Read block 2063 len 4096
10106 155/300 Write block 4591 len 24576 data: 3 4 4 4 4 4
10107 156/300 Read block 1435 len 40960
10108 157/300 Write block 4944 len 40960 data: 3 3 3 3 4 4 4 4 4 5
10109 158/300 Write block 181 len 12288 data: 3 3 2
10110 159/300 Read block 3876 len 36864
10111 160/300 Write block 1452 len 16384 data: 2 2 2 2
10112 161/300 Write block 1953 len 40960 data: 3 3 4 4 4 5 4 4 3 3
10113 162/300 Read block 2492 len 4096
10114 163/300 Read block 2151 len 12288
10115 164/300 Write block 2613 len 32768 data: 4 4 4 3 3 3 3 4
10116 165/300 Read block 2872 len 32768
10117 166/300 Write block 3600 len 8192 data: 3 2
10118 167/300 Read block 4549 len 32768
10119 168/300 Write block 4203 len 32768 data: 4 4 4 4 3 2 2 3
10120 169/300 Write block 1233 len 12288 data: 2 3 2
10121 170/300 Write block 3371 len 28672 data: 2 2 2 2 2 2 2
10122 171/300 Read block 3743 len 4096
10123 172/300 Flush
10124 173/300 Flush
10125 174/300 Read block 1099 len 40960
10126 175/300 Write block 1005 len 16384 data: 4 4 4 4
10127 176/300 Write block 4844 len 4096 data: 4
10128 177/300 Flush
10129 178/300 Write block 2202 len 20480 data: 2 2 3 4 5
10130 179/300 Write block 3667 len 20480 data: 5 5 5 5 5
10131 180/300 Write block 2870 len 8192 data: 2 2
10132 181/300 Read block 3565 len 40960
10133 182/300 Read block 550 len 32768
10134 183/300 Write block 3757 len 40960 data: 2 2 2 2 2 3 3 2 2 3
10135 184/300 Read block 160 len 32768
10136 185/300 Write block 3034 len 20480 data: 5 4 4 4 3
10137 186/300 Read block 541 len 20480
10138 187/300 Write block 1254 len 28672 data: 4 5 5 5 4 4 4
10139 188/300 Flush
10140 189/300 Write block 2446 len 40960 data: 4 4 5 4 3 3 3 4 3 3
10141 190/300 Write block 972 len 12288 data: 4 5 4
10142 191/300 Write block 4744 len 4096 data: 4
10143 192/300 Read block 29 len 8192
10144 193/300 Read block 1524 len 16384
10145 194/300 Flush
10146 195/300 Read block 3058 len 24576
10147 196/300 Write block 2255 len 40960 data: 3 3 4 3 3 3 3 3 3 3
10148 197/300 Read block 4726 len 4096
10149 198/300 Read block 502 len 4096
10150 199/300 Write block 1961 len 36864 data: 4 4 3 3 3 3 3 2 2
10151 200/300 Read block 583 len 32768
10152 201/300 Write block 1873 len 28672 data: 3 4 4 4 4 5 2
10153 202/300 Read block 2622 len 16384
10154 203/300 Read block 4297 len 4096
10155 204/300 Read block 3563 len 32768
10156 205/300 Write block 4383 len 20480 data: 4 3 3 3 3
10157 206/300 Write block 4881 len 12288 data: 2 2 2
10158 207/300 Read block 588 len 20480
10159 208/300 Read block 3342 len 24576
10160 209/300 Write block 2671 len 20480 data: 2 2 2 2 2
10161 210/300 Read block 2999 len 28672
10162 211/300 Flush
10163 212/300 Read block 1822 len 20480
10164 213/300 Read block 3502 len 28672
10165 214/300 Write block 937 len 4096 data: 3
10166 215/300 Read block 4752 len 36864
10167 216/300 Write block 1188 len 16384 data: 4 4 4 5
10168 217/300 Flush
10169 218/300 Write block 2514 len 16384 data: 3 3 2 2
10170 219/300 Read block 2236 len 28672
10171 220/300 Write block 3730 len 28672 data: 7 6 5 6 6 6 4
10172 221/300 Read block 3089 len 40960
10173 222/300 Write block 3846 len 28672 data: 2 2 2 2 2 2 2
10174 223/300 Write block 2582 len 16384 data: 4 3 3 4
10175 224/300 Flush
10176 225/300 Write block 3756 len 8192 data: 3 3
10177 226/300 Read block 1168 len 12288
10178 227/300 Read block 1048 len 40960
10179 228/300 Write block 4375 len 20480 data: 4 5 5 5 3
10180 229/300 Flush
10181 230/300 Write block 2343 len 20480 data: 4 4 4 4 3
10182 231/300 Flush
10183 232/300 Read block 2238 len 8192
10184 233/300 Write block 1677 len 16384 data: 3 3 2 2
10185 234/300 Write block 861 len 20480 data: 3 3 4 4 5
10186 235/300 Write block 4408 len 12288 data: 2 2 2
10187 236/300 Read block 3025 len 8192
10188 237/300 Write block 4736 len 24576 data: 3 4 4 6 5 5
10189 238/300 Write block 3079 len 40960 data: 2 2 3 3 4 4 4 5 5 5
10190 239/300 Read block 1294 len 40960
10191 240/300 Read block 315 len 20480
10192 241/300 Write block 4110 len 36864 data: 3 6 6 7 7 6 6 6 5
10193 242/300 Read block 2403 len 8192
10194 243/300 Write block 101 len 40960 data: 2 3 3 3 3 3 2 2 2 2
10195 244/300 Read block 1625 len 20480
10196 245/300 Write block 3799 len 28672 data: 3 2 2 2 2 2 2
10197 246/300 Read block 3500 len 12288
10198 247/300 Read block 3801 len 20480
10199 248/300 Read block 3263 len 16384
10200 249/300 Write block 3694 len 20480 data: 5 4 4 4 4
10201 250/300 Read block 1005 len 36864
10202 251/300 Write block 2745 len 4096 data: 3
10203 252/300 Read block 3910 len 12288
10204 253/300 Read block 1551 len 24576
10205 254/300 Read block 1887 len 16384
10206 255/300 Write block 4532 len 16384 data: 4 4 4 3
10207 256/300 Read block 311 len 40960
10208 257/300 Write block 4124 len 28672 data: 3 3 3 3 3 2 2
10209 258/300 Write block 159 len 24576 data: 3 4 5 5 5 5
10210 259/300 Read block 2789 len 36864
10211 260/300 Read block 1939 len 36864
10212 261/300 Read block 1820 len 16384
10213 262/300 Read block 3796 len 24576
10214 263/300 Read block 3994 len 12288
10215 264/300 Write block 3164 len 24576 data: 5 5 5 5 3 3
10216 265/300 Write block 4344 len 20480 data: 2 2 3 3 3
10217 266/300 Read block 4388 len 12288
10218 267/300 Write block 3662 len 8192 data: 5 5
10219 268/300 Write block 3170 len 8192 data: 2 2
10220 269/300 Read block 1318 len 32768
10221 270/300 Read block 1289 len 24576
10222 271/300 Read block 1937 len 16384
10223 272/300 Write block 4639 len 32768 data: 4 5 5 5 4 4 4 3
10224 273/300 Read block 4655 len 4096
10225 274/300 Read block 3480 len 4096
10226 275/300 Write block 2984 len 36864 data: 3 3 3 3 3 4 4 2 2
10227 276/300 Read block 3349 len 36864
10228 277/300 Read block 904 len 12288
10229 278/300 Read block 2024 len 8192
10230 279/300 Write block 3246 len 4096 data: 4
10231 280/300 Read block 3919 len 40960
10232 281/300 Flush
10233 282/300 Read block 3936 len 16384
10234 283/300 Read block 1327 len 20480
10235 284/300 Write block 3848 len 4096 data: 3
10236 285/300 Read block 2347 len 20480
10237 286/300 Read block 159 len 16384
10238 287/300 Write block 1506 len 12288 data: 4 3 4
10239 288/300 Write block 2764 len 16384 data: 2 2 2 2
10240 289/300 Read block 1779 len 28672
10241 290/300 Write block 3125 len 28672 data: 3 3 3 3 3 3 2
10242 291/300 Write block 4143 len 40960 data: 5 5 5 3 2 2 2 2 2 4
10243 292/300 Read block 2735 len 8192
10244 293/300 Write block 3848 len 20480 data: 4 3 3 3 3
10245 294/300 Read block 1761 len 16384
10246 295/300 Read block 4716 len 8192
10247 296/300 Read block 4900 len 24576
10248 297/300 Read block 4181 len 32768
10249 298/300 Write block 2984 len 20480 data: 4 4 4 4 4
10250 299/300 Read block 2562 len 20480
10251 300/300 Write block 389 len 4096 data: 3
10252 Main task has work to do, go find it
10253 got dsc Start(0)
10254 start 0
10255 Feb 15 02:59:34.013 INFO request completed, latency_us: 212, response_code: 204, uri: /start/cid/0, method: POST, req_id: 59821991-b715-43c1-8af5-9728ce7ec0d9, remote_addr: 127.0.0.1:54744, local_addr: 127.0.0.1:9998
10256 Replay: started 0, returned:Ok(())
10257 ----------------------------------------------------------------
10258 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
10259 States: Offline Active Active
10260 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
10261 2713 Acked 3712 Write 10 New Done Done true
10262 2714 Acked 3713 Flush 0 New Done Done true
10263 2715 Acked 3714 Read 9 New Done Done false
10264 2716 Acked 3715 Read 1 New Done Done false
10265 2717 Acked 3716 Read 4 New Done Done false
10266 2718 Acked 3717 Write 9 New Done Done false
10267 2719 Acked 3718 Write 4 New Done Done false
10268 2720 Acked 3719 Write 8 New Done Done false
10269 2721 Acked 3720 Read 8 New Done Done false
10270 2722 Acked 3721 Read 9 New Done Done false
10271 2723 Acked 3722 Write 4 New Done Done false
10272 2724 Acked 3723 Read 1 New Done Done false
10273 2725 Acked 3724 Read 9 New Done Done false
10274 2726 Acked 3725 Read 3 New Done Done false
10275 2727 Acked 3726 Write 8 New Done Done false
10276 2728 Acked 3727 Read 4 New Done Done false
10277 2729 Acked 3728 Read 7 New Done Done false
10278 2730 Acked 3729 Write 6 New Done Done false
10279 2731 Acked 3730 Read 2 New Done Done false
10280 2732 Acked 3731 Read 8 New Done Done false
10281 2733 Acked 3732 Read 2 New Done Done false
10282 2734 Acked 3733 Read 7 New Done Done false
10283 2735 Acked 3734 Write 2 New Done Done false
10284 2736 Acked 3735 Write 7 New Done Done false
10285 2737 Acked 3736 Write 10 New Done Done false
10286 2738 Acked 3737 Read 6 New Done Done false
10287 2739 Acked 3738 Write 8 New Done Done false
10288 2740 Acked 3739 Write 10 New Done Done false
10289 2741 Acked 3740 Flush 0 New Done Done false
10290 2742 Acked 3741 Write 3 New Done Done false
10291 2743 Acked 3742 Write 10 New Done Done false
10292 2744 Acked 3743 Read 7 New Done Done false
10293 2745 Acked 3744 Write 7 New Done Done false
10294 2746 Acked 3745 Read 2 New Done Done false
10295 2747 Acked 3746 Read 1 New Done Done false
10296 2748 Acked 3747 Read 1 New Done Done false
10297 2749 Acked 3748 Write 5 New Done Done false
10298 2750 Acked 3749 Write 5 New Done Done false
10299 2751 Acked 3750 Write 3 New Done Done false
10300 2752 Acked 3751 Read 6 New Done Done false
10301 2753 Acked 3752 Read 6 New Done Done false
10302 2754 Acked 3753 Flush 0 New Done Done false
10303 2755 Acked 3754 Write 9 New Done Done false
10304 2756 Acked 3755 Write 1 New Done Done false
10305 2757 Acked 3756 Read 6 New Done Done false
10306 2758 Acked 3757 Read 4 New Done Done false
10307 2759 Acked 3758 Read 10 New Done Done false
10308 2760 Acked 3759 Write 5 New Done Done false
10309 2761 Acked 3760 Read 5 New Done Done false
10310 2762 Acked 3761 Read 7 New Done Done false
10311 2763 Acked 3762 Read 8 New Done Done false
10312 2764 Acked 3763 Write 8 New Done Done false
10313 2765 Acked 3764 Write 8 New Done Done false
10314 2766 Acked 3765 Write 7 New Done Done false
10315 2767 Acked 3766 Read 3 New Done Done false
10316 2768 Acked 3767 Write 3 New Done Done false
10317 2769 Acked 3768 Write 8 New Done Done false
10318 2770 Acked 3769 Read 2 New Done Done false
10319 2771 Acked 3770 Write 6 New Done Done false
10320 2772 Acked 3771 Read 4 New Done Done false
10321 2773 Acked 3772 Write 7 New Done Done false
10322 2774 Acked 3773 Read 3 New Done Done false
10323 2775 Acked 3774 Read 8 New Done Done false
10324 2776 Acked 3775 Write 4 New Done Done false
10325 2777 Acked 3776 Write 7 New Done Done false
10326 2778 Acked 3777 Write 5 New Done Done false
10327 2779 Acked 3778 Write 6 New Done Done false
10328 2780 Acked 3779 Write 4 New Done Done false
10329 2781 Acked 3780 Write 8 New Done Done false
10330 2782 Acked 3781 Write 7 New Done Done false
10331 2783 Acked 3782 Read 3 New Done Done false
10332 2784 Acked 3783 Read 9 New Done Done false
10333 2785 Acked 3784 Write 6 New Done Done false
10334 2786 Acked 3785 Write 5 New Done Done false
10335 2787 Acked 3786 Flush 0 New Done Done false
10336 2788 Acked 3787 Read 1 New Done Done false
10337 2789 Acked 3788 Flush 0 New Done Done false
10338 2790 Acked 3789 Write 8 New Done Done false
10339 2791 Acked 3790 Read 4 New Done Done false
10340 2792 Acked 3791 Write 1 New Done Done false
10341 2793 Acked 3792 Read 6 New Done Done false
10342 2794 Acked 3793 Write 4 New Done Done false
10343 2795 Acked 3794 Write 4 New Done Done false
10344 2796 Acked 3795 Write 4 New Done Done false
10345 2797 Acked 3796 Write 3 New Done Done false
10346 2798 Acked 3797 Read 5 New Done Done false
10347 2799 Acked 3798 Read 10 New Done Done false
10348 2800 Acked 3799 Read 1 New Done Done false
10349 2801 Acked 3800 Read 8 New Done Done false
10350 2802 Acked 3801 Write 5 New Done Done false
10351 2803 Acked 3802 Write 4 New Done Done false
10352 2804 Acked 3803 Read 2 New Done Done false
10353 2805 Acked 3804 Read 9 New Done Done false
10354 2806 Acked 3805 Flush 0 New Done Done false
10355 2807 Acked 3806 Read 5 New Done Done false
10356 2808 Acked 3807 Read 6 New Done Done false
10357 2809 Acked 3808 Write 8 New Done Done false
10358 2810 Acked 3809 Flush 0 New Done Done false
10359 2811 Acked 3810 Write 5 New Done Done false
10360 2812 Acked 3811 Write 7 New Done Done false
10361 2813 Acked 3812 Read 6 New Done Done false
10362 2814 Acked 3813 Flush 0 New Done Done false
10363 2815 Acked 3814 Flush 0 New Done Done false
10364 2816 Acked 3815 Write 5 New Done Done false
10365 2817 Acked 3816 Write 3 New Done Done false
10366 2818 Acked 3817 Write 8 New Done Done false
10367 2819 Acked 3818 Read 9 New Done Done false
10368 2820 Acked 3819 Read 3 New Done Done false
10369 2821 Acked 3820 Write 4 New Done Done false
10370 2822 Acked 3821 Read 10 New Done Done false
10371 2823 Acked 3822 Write 9 New Done Done false
10372 2824 Acked 3823 Read 4 New Done Done false
10373 2825 Acked 3824 Read 10 New Done Done false
10374 2826 Acked 3825 Write 10 New Done Done false
10375 2827 Acked 3826 Flush 0 New Done Done false
10376 2828 Acked 3827 Read 8 New Done Done false
10377 2829 Acked 3828 Read 3 New Done Done false
10378 2830 Acked 3829 Flush 0 New Done Done false
10379 2831 Acked 3830 Write 1 New Done Done false
10380 2832 Acked 3831 Read 7 New Done Done false
10381 2833 Acked 3832 Flush 0 New Done Done false
10382 2834 Acked 3833 Read 9 New Done Done false
10383 2835 Acked 3834 Write 2 New Done Done false
10384 2836 Acked 3835 Read 2 New Done Done false
10385 2837 Acked 3836 Write 2 New Done Done false
10386 2838 Acked 3837 Write 5 New Done Done false
10387 2839 Acked 3838 Read 6 New Done Done false
10388 2840 Acked 3839 Read 3 New Done Done false
10389 2841 Acked 3840 Read 5 New Done Done false
10390 2842 Acked 3841 Read 10 New Done Done false
10391 2843 Acked 3842 Read 7 New Done Done false
10392 2844 Acked 3843 Write 10 New Done Done false
10393 2845 Acked 3844 Write 10 New Done Done false
10394 2846 Acked 3845 Write 9 New Done Done false
10395 2847 Acked 3846 Write 9 New Done Done false
10396 2848 Acked 3847 Read 5 New Done Done false
10397 2849 Acked 3848 Write 3 New Done Done false
10398 2850 Acked 3849 Write 7 New Done Done false
10399 2851 Acked 3850 Read 8 New Done Done false
10400 2852 Acked 3851 Write 8 New Done Done false
10401 2853 Acked 3852 Write 7 New Done Done false
10402 2854 Acked 3853 Write 2 New Done Done false
10403 2855 Acked 3854 Write 6 New Done Done false
10404 2856 Acked 3855 Read 4 New Done Done false
10405 2857 Acked 3856 Write 2 New Done Done false
10406 2858 Acked 3857 Read 9 New Done Done false
10407 2859 Acked 3858 Read 4 New Done Done false
10408 2860 Acked 3859 Write 6 New Done Done false
10409 2861 Acked 3860 Flush 0 New Done Done false
10410 2862 Acked 3861 Read 5 New Done Done false
10411 2863 Acked 3862 Read 4 New Done Done false
10412 2864 Acked 3863 Read 4 New Done Done false
10413 2865 Acked 3864 Write 2 New Done Done false
10414 2866 Acked 3865 Read 7 New Done Done false
10415 2867 Acked 3866 Read 1 New Done Done false
10416 2868 Acked 3867 Write 6 New Done Done false
10417 2869 Acked 3868 Read 10 New Done Done false
10418 2870 Acked 3869 Write 10 New Done Done false
10419 2871 Acked 3870 Write 3 New Done Done false
10420 2872 Acked 3871 Read 9 New Done Done false
10421 2873 Acked 3872 Write 4 New Done Done false
10422 2874 Acked 3873 Write 10 New Done Done false
10423 2875 Acked 3874 Read 1 New Done Done false
10424 2876 Acked 3875 Read 3 New Done Done false
10425 2877 Acked 3876 Write 8 New Done Done false
10426 2878 Acked 3877 Read 8 New Done Done false
10427 2879 Acked 3878 Write 2 New Done Done false
10428 2880 Acked 3879 Read 8 New Done Done false
10429 2881 Acked 3880 Write 8 New Done Done false
10430 2882 Acked 3881 Write 3 New Done Done false
10431 2883 Acked 3882 Write 7 New Done Done false
10432 2884 Acked 3883 Read 1 New Done Done false
10433 2885 Acked 3884 Flush 0 New Done Done false
10434 2886 Acked 3885 Flush 0 New Done Done false
10435 2887 Acked 3886 Read 10 New Done Done false
10436 2888 Acked 3887 Write 4 New Done Done false
10437 2889 Acked 3888 Write 1 New Done Done false
10438 2890 Acked 3889 Flush 0 New Done Done false
10439 2891 Acked 3890 Write 5 New Done Done false
10440 2892 Acked 3891 Write 5 New Done Done false
10441 2893 Acked 3892 Write 2 New Done Done false
10442 2894 Acked 3893 Read 10 New Done Done false
10443 2895 Acked 3894 Read 8 New Done Done false
10444 2896 Acked 3895 Write 10 New Done Done false
10445 2897 Acked 3896 Read 8 New Done Done false
10446 2898 Acked 3897 Write 5 New Done Done false
10447 2899 Acked 3898 Read 5 New Done Done false
10448 2900 Acked 3899 Write 7 New Done Done false
10449 2901 Acked 3900 Flush 0 New Done Done false
10450 2902 Acked 3901 Write 10 New Done Done false
10451 2903 Acked 3902 Write 3 New Done Done false
10452 2904 Acked 3903 Write 1 New Done Done false
10453 2905 Acked 3904 Read 2 New Done Done false
10454 2906 Acked 3905 Read 4 New Done Done false
10455 2907 Acked 3906 Flush 0 New Done Done false
10456 2908 Acked 3907 Read 6 New Done Done false
10457 2909 Acked 3908 Write 10 New Done Done false
10458 2910 Acked 3909 Read 1 New Done Done false
10459 2911 Acked 3910 Read 1 New Done Done false
10460 2912 Acked 3911 Write 9 New Done Done false
10461 2913 Acked 3912 Read 8 New Done Done false
10462 2914 Acked 3913 Write 7 New Done Done false
10463 2915 Acked 3914 Read 4 New Done Done false
10464 2916 Acked 3915 Read 1 New Done Done false
10465 2917 Acked 3916 Read 8 New Done Done false
10466 2918 Acked 3917 Write 5 New Done Done false
10467 2919 Acked 3918 Write 3 New Done Done false
10468 2920 Acked 3919 Read 5 New Done Done false
10469 2921 Acked 3920 Read 6 New Done Done false
10470 2922 Acked 3921 Write 5 New Done Done false
10471 2923 Acked 3922 Read 7 New Done Done false
10472 2924 Acked 3923 Flush 0 New Done Done false
10473 2925 Acked 3924 Read 5 New Done Done false
10474 2926 Acked 3925 Read 7 New Done Done false
10475 2927 Acked 3926 Write 1 New Done Done false
10476 2928 Acked 3927 Read 9 New Done Done false
10477 2929 Acked 3928 Write 4 New Done Done false
10478 2930 Acked 3929 Flush 0 New Done Done false
10479 2931 Acked 3930 Write 4 New Done Done false
10480 2932 Acked 3931 Read 7 New Done Done false
10481 2933 Acked 3932 Write 7 New Done Done false
10482 2934 Acked 3933 Read 10 New Done Done false
10483 2935 Acked 3934 Write 7 New Done Done false
10484 2936 Acked 3935 Write 4 New Done Done false
10485 2937 Acked 3936 Flush 0 New Done Done false
10486 2938 Acked 3937 Write 2 New Done Done false
10487 2939 Acked 3938 Read 3 New Done Done false
10488 2940 Acked 3939 Read 10 New Done Done false
10489 2941 Acked 3940 Write 5 New Done Done false
10490 2942 Acked 3941 Flush 0 New Done Done false
10491 2943 Acked 3942 Write 5 New Done Done false
10492 2944 Acked 3943 Flush 0 New Done Done false
10493 2945 Acked 3944 Read 2 New Done Done false
10494 2946 Acked 3945 Write 4 New Done Done false
10495 2947 Acked 3946 Write 5 New Done Done false
10496 2948 Acked 3947 Write 3 New Done Done false
10497 2949 Acked 3948 Read 2 New Done Done false
10498 2950 Acked 3949 Write 6 New Done Done false
10499 2951 Acked 3950 Write 10 New Done Done false
10500 2952 Acked 3951 Read 10 New Done Done false
10501 2953 Acked 3952 Read 5 New Done Done false
10502 2954 Acked 3953 Write 9 New Done Done false
10503 2955 Acked 3954 Read 2 New Done Done false
10504 2956 Acked 3955 Write 10 New Done Done false
10505 2957 Acked 3956 Read 5 New Done Done false
10506 2958 Acked 3957 Write 7 New Done Done false
10507 2959 Acked 3958 Read 3 New Done Done false
10508 2960 Acked 3959 Read 5 New Done Done false
10509 2961 Acked 3960 Read 4 New Done Done false
10510 2962 Acked 3961 Write 5 New Done Done false
10511 2963 Acked 3962 Read 9 New Done Done false
10512 2964 Acked 3963 Write 1 New Done Done false
10513 2965 Acked 3964 Read 3 New Done Done false
10514 2966 Acked 3965 Read 6 New Done Done false
10515 2967 Acked 3966 Read 4 New Done Done false
10516 2968 Acked 3967 Write 4 New Done Done false
10517 2969 Acked 3968 Read 10 New Done Done false
10518 2970 Acked 3969 Write 7 New Done Done false
10519 2971 Acked 3970 Write 6 New Done Done false
10520 2972 Acked 3971 Read 9 New Done Done false
10521 2973 Acked 3972 Read 9 New Done Done false
10522 2974 Acked 3973 Read 4 New Done Done false
10523 2975 Acked 3974 Read 6 New Done Done false
10524 2976 Acked 3975 Read 3 New Done Done false
10525 2977 Acked 3976 Write 6 New Done Done false
10526 2978 Acked 3977 Write 5 New Done Done false
10527 2979 Acked 3978 Read 3 New Done Done false
10528 2980 Acked 3979 Write 2 New Done Done false
10529 2981 Acked 3980 Write 2 New Done Done false
10530 2982 Acked 3981 Read 8 New Done Done false
10531 2983 Acked 3982 Read 6 New Done Done false
10532 2984 Acked 3983 Read 4 New Done Done false
10533 2985 Acked 3984 Write 8 New Done Done false
10534 2986 Acked 3985 Read 1 New Done Done false
10535 2987 Acked 3986 Read 1 New Done Done false
10536 2988 Acked 3987 Write 9 New Done Done false
10537 2989 Acked 3988 Read 9 New Done Done false
10538 2990 Acked 3989 Read 3 New Done Done false
10539 2991 Acked 3990 Read 2 New Done Done false
10540 2992 Acked 3991 Write 1 New Done Done false
10541 2993 Acked 3992 Read 10 New Done Done false
10542 2994 Acked 3993 Flush 0 New Done Done false
10543 2995 Acked 3994 Read 4 New Done Done false
10544 2996 Acked 3995 Read 5 New Done Done false
10545 2997 Acked 3996 Write 1 New Done Done false
10546 2998 Acked 3997 Read 5 New Done Done false
10547 2999 Acked 3998 Read 4 New Done Done false
10548 3000 Acked 3999 Write 3 New Done Done false
10549 3001 Acked 4000 Write 4 New Done Done false
10550 3002 Acked 4001 Read 7 New Done Done false
10551 3003 Acked 4002 Write 7 New Done Done false
10552 3004 Acked 4003 Write 10 New Done Done false
10553 3005 Acked 4004 Read 2 New Done Done false
10554 3006 Acked 4005 Write 5 New Done Done false
10555 3007 Acked 4006 Read 4 New Done Done false
10556 3008 Acked 4007 Read 2 New Done Done false
10557 3009 Acked 4008 Read 6 New Done Done false
10558 3010 Acked 4009 Read 8 New Done Done false
10559 3011 Acked 4010 Write 5 New Done Done false
10560 3012 Acked 4011 Read 5 New Done Done false
10561 3013 Acked 4012 Write 1 New Done Done false
10562 STATES DS:0 DS:1 DS:2 TOTAL
10563 New 301 0 0 301
10564 Sent 0 0 0 0
10565 Done 0 301 301 602
10566 Skipped 0 0 0 0
10567 Error 0 0 0 0
10568 Last Flush: 3711 3993 3993
10569 Downstairs last five completed: 3711 3710 3709 3708 3707
10570 Upstairs last five completed: 3013 3012 3011 3010 3009
10571 CLIENT: Up:0 ds:301 act:2
105722024-02-15T02:59:34.684ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
105732024-02-15T02:59:34.684ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
105742024-02-15T02:59:34.684ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
105752024-02-15T02:59:34.684ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
105762024-02-15T02:59:34.684ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
105772024-02-15T02:59:34.684ZINFOcrucible: [0] client re-new 302 jobs since flush 3711 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
105782024-02-15T02:59:34.684ZINFOcrucible: client task is exiting = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
10579 [8810] got start action
10580 [8810] I am going to restart
10581 Starting downstairs at port 8810
10582 Make output file at "/tmp/dsc/downstairs-8810.txt"
10583 [8810][0] reports Starting
105842024-02-15T02:59:35.686ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
105852024-02-15T02:59:35.686ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
105862024-02-15T02:59:35.686ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
105872024-02-15T02:59:35.686ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
105882024-02-15T02:59:35.686ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
105892024-02-15T02:59:35.686ZINFOcrucible: [0] client re-new 302 jobs since flush 3711 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
105902024-02-15T02:59:35.686ZINFOcrucible: client task is exiting = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
10591 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1177)
10592 [8810][0] reports Running
105932024-02-15T02:59:36.687ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
105942024-02-15T02:59:36.687ZINFOcrucible: ds_connection connected = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
105952024-02-15T02:59:36.689ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
105962024-02-15T02:59:36.689ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
105972024-02-15T02:59:36.689ZINFOcrucible: send last flush ID to this DS: 3711 = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
105982024-02-15T02:59:36.689ZINFOcrucible: Replied this last flush ID: 3711 = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
105992024-02-15T02:59:36.689ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
106002024-02-15T02:59:36.689ZINFOcrucible: [0] Transition from Offline to Replay = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
106012024-02-15T02:59:36.689ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
106022024-02-15T02:59:36.689ZINFOcrucible: [0] Transition from Replay to Active = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
10603 ----------------------------------------------------------------
10604 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
10605 Downstairs last five completed: 4013 4012 4011 4010 4009
10606 Upstairs last five completed: 3014 3013 3012 3011 3010
10607 CLIENT: Up:0 ds:0 act:3
10608 Replay: All jobs finished, all DS active.
10609 Main task has work to do, go find it
10610 got dsc Stop(1)
10611 stop 1
10612 Feb 15 02:59:38.015 INFO request completed, latency_us: 344[8820] Got stop action so:false kr:false
10613 , response_code: 204, uri: /stop/cid/1, method: POST, req_id: 0187db54-16fd-4b5d-9513-4ca86b75effe, remote_addr: 127.0.0.1:54744, local_addr: 127.0.0.1:9998
106142024-02-15T02:59:38.016ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
106152024-02-15T02:59:38.016ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
106162024-02-15T02:59:38.016ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
106172024-02-15T02:59:38.016ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
106182024-02-15T02:59:38.016ZINFOcrucible: [1] client re-new 0 jobs since flush 4013 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
106192024-02-15T02:59:38.016ZINFOcrucible: client task is exiting = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
10620 Feb 15 02:59:38.016 INFO request completed, latency_us: 124, response_code: 200, uri: /state/cid/1, method: GET, req_id: 02f9d38c-2d7e-46f0-aebc-8c42dd3512fd, remote_addr: 127.0.0.1:54744, local_addr: 127.0.0.1:9998
10621 [8820] Exited with: ExitStatus(unix_wait_status(9))
10622 [8820][1] reports Exit
106232024-02-15T02:59:39.018ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
106242024-02-15T02:59:39.018ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
106252024-02-15T02:59:39.018ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
106262024-02-15T02:59:39.018ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
106272024-02-15T02:59:39.018ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
106282024-02-15T02:59:39.018ZINFOcrucible: [1] client re-new 0 jobs since flush 4013 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
106292024-02-15T02:59:39.018ZINFOcrucible: client task is exiting = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
106302024-02-15T02:59:40.019ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
106312024-02-15T02:59:40.019ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
106322024-02-15T02:59:40.019ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
106332024-02-15T02:59:40.019ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
106342024-02-15T02:59:40.019ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
106352024-02-15T02:59:40.020ZINFOcrucible: [1] client re-new 0 jobs since flush 4013 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
106362024-02-15T02:59:40.020ZINFOcrucible: client task is exiting = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
106372024-02-15T02:59:41.021ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
106382024-02-15T02:59:41.021ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
106392024-02-15T02:59:41.021ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
106402024-02-15T02:59:41.021ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
106412024-02-15T02:59:41.021ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
106422024-02-15T02:59:41.021ZINFOcrucible: [1] client re-new 0 jobs since flush 4013 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
106432024-02-15T02:59:41.021ZINFOcrucible: client task is exiting = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
10644 Feb 15 02:59:42.019 INFO request completed, latency_us: 308, response_code: 200, uri: /state/cid/1, method: GET, req_id: d7eb42f7-53a4-40e0-acb4-83914515a6ba, remote_addr: 127.0.0.1:54744, local_addr: 127.0.0.1:9998
10645 001/300 Write block 1604 len 12288 data: 3 3 3
10646 002/300 Read block 3097 len 32768
10647 003/300 Read block 4095 len 40960
106482024-02-15T02:59:42.022ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
106492024-02-15T02:59:42.022ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
106502024-02-15T02:59:42.022ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
106512024-02-15T02:59:42.022ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
106522024-02-15T02:59:42.022ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
106532024-02-15T02:59:42.022ZINFOcrucible: [1] client re-new 3 jobs since flush 4013 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
106542024-02-15T02:59:42.022ZINFOcrucible: client task is exiting = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
10655 004/300 Write block 3853 len 32768 data: 2 2 2 2 2 2 2 3
10656 005/300 Read block 4266 len 8192
10657 006/300 Read block 4647 len 12288
10658 007/300 Write block 3623 len 36864 data: 6 6 6 7 6 4 3 3 3
10659 008/300 Read block 306 len 16384
10660 009/300 Read block 2074 len 16384
10661 010/300 Read block 1204 len 24576
10662 011/300 Read block 1302 len 16384
10663 012/300 Read block 3905 len 20480
10664 013/300 Flush
10665 014/300 Write block 3833 len 20480 data: 5 4 3 3 3
10666 015/300 Write block 537 len 20480 data: 5 3 3 2 2
10667 016/300 Flush
10668 017/300 Write block 308 len 28672 data: 4 4 4 4 2 2 2
10669 018/300 Write block 2659 len 12288 data: 2 2 3
10670 019/300 Write block 654 len 28672 data: 2 2 2 2 2 2 2
10671 020/300 Write block 1402 len 24576 data: 2 3 4 4 4 4
10672 021/300 Write block 2320 len 8192 data: 2 2
10673 022/300 Read block 3143 len 4096
10674 023/300 Read block 384 len 36864
10675 024/300 Write block 1803 len 36864 data: 2 2 2 2 2 2 2 4 4
10676 025/300 Read block 392 len 24576
10677 026/300 Read block 1541 len 8192
10678 027/300 Read block 3356 len 16384
10679 028/300 Read block 2888 len 4096
10680 029/300 Read block 2536 len 4096
10681 030/300 Write block 1396 len 28672 data: 3 2 2 2 2 2 3
10682 031/300 Read block 4844 len 8192
10683 032/300 Write block 2195 len 20480 data: 4 4 3 3 3
10684 033/300 Write block 2819 len 12288 data: 4 5 3
10685 034/300 Read block 441 len 16384
10686 035/300 Read block 4683 len 36864
10687 036/300 Read block 1197 len 4096
10688 037/300 Read block 2077 len 40960
10689 038/300 Write block 1284 len 32768 data: 2 4 4 4 5 5 4 4
10690 039/300 Read block 4702 len 8192
10691 040/300 Write block 148 len 4096 data: 4
10692 041/300 Write block 2757 len 8192 data: 2 2
10693 042/300 Flush
10694 043/300 Write block 711 len 12288 data: 4 4 5
10695 044/300 Write block 1793 len 12288 data: 3 3 3
10696 045/300 Read block 618 len 40960
10697 046/300 Read block 588 len 40960
10698 047/300 Write block 3447 len 20480 data: 2 3 3 3 3
10699 048/300 Read block 4326 len 28672
10700 049/300 Flush
10701 050/300 Write block 4937 len 8192 data: 2 2
10702 051/300 Read block 1646 len 8192
10703 052/300 Read block 2563 len 32768
10704 053/300 Read block 337 len 4096
10705 054/300 Write block 2521 len 20480 data: 3 3 3 3 3
10706 055/300 Read block 4634 len 36864
10707 056/300 Read block 4083 len 24576
10708 057/300 Read block 1542 len 12288
10709 058/300 Read block 3757 len 32768
10710 059/300 Read block 1165 len 24576
10711 060/300 Read block 350 len 28672
10712 061/300 Write block 1320 len 40960 data: 4 4 4 3 3 2 2 2 3 3
10713 062/300 Write block 3845 len 16384 data: 2 3 3 5
10714 063/300 Read block 1712 len 40960
10715 064/300 Read block 2752 len 32768
10716 065/300 Write block 3663 len 24576 data: 6 4 3 4 6 6
10717 066/300 Write block 477 len 8192 data: 2 3
10718 067/300 Flush
10719 068/300 Read block 1856 len 8192
10720 069/300 Write block 4564 len 12288 data: 2 2 2
10721 070/300 Write block 2130 len 16384 data: 5 4 5 6
10722 071/300 Write block 2568 len 36864 data: 2 2 2 2 2 2 3 4 4
10723 072/300 Read block 2673 len 16384
10724 073/300 Write block 992 len 36864 data: 3 3 3 3 2 3 3 4 4
10725 074/300 Read block 4673 len 20480
10726 075/300 Write block 3563 len 4096 data: 4
10727 076/300 Flush
10728 077/300 Read block 3237 len 24576
10729 078/300 Write block 4623 len 24576 data: 3 4 3 2 2 3
10730 079/300 Write block 4109 len 20480 data: 3 4 7 7 8
10731 080/300 Read block 2344 len 4096
10732 081/300 Write block 4306 len 24576 data: 3 5 5 5 5 4
10733 082/300 Write block 54 len 12288 data: 6 5 5
10734 083/300 Write block 3346 len 32768 data: 3 5 4 4 4 4 3 3
10735 084/300 Read block 4414 len 40960
10736 085/300 Read block 3487 len 4096
10737 086/300 Read block 4877 len 36864
10738 087/300 Read block 1847 len 32768
10739 088/300 Read block 2946 len 4096
10740 089/300 Write block 906 len 20480 data: 2 2 2 3 3
10741 090/300 Read block 342 len 12288
10742 091/300 Write block 2017 len 12288 data: 3 4 5
10743 092/300 Read block 1037 len 24576
10744 093/300 Read block 4536 len 4096
10745 094/300 Flush
10746 095/300 Flush
10747 096/300 Read block 3860 len 32768
10748 097/300 Flush
10749 098/300 Write block 1558 len 20480 data: 3 4 4 4 4
10750 099/300 Read block 4627 len 24576
10751 100/300 Read block 2155 len 28672
10752 101/300 Write block 1302 len 36864 data: 2 2 2 2 2 2 2 3 3
10753 102/300 Write block 437 len 28672 data: 5 5 4 4 3 4 4
10754 103/300 Write block 1515 len 8192 data: 4 4
10755 104/300 Read block 2276 len 32768
10756 105/300 Flush
10757 106/300 Write block 938 len 28672 data: 3 3 2 2 2 2 2
10758 107/300 Read block 2654 len 8192
10759 108/300 Flush
10760 109/300 Write block 3076 len 24576 data: 2 2 2 3 3 4
10761 110/300 Write block 2976 len 16384 data: 2 2 2 2
10762 111/300 Write block 2479 len 24576 data: 4 4 4 4 3 3
10763 112/300 Read block 2979 len 32768
10764 113/300 Write block 1579 len 12288 data: 2 2 4
10765 114/300 Read block 339 len 40960
10766 115/300 Read block 1960 len 20480
10767 116/300 Read block 370 len 36864
10768 117/300 Read block 4877 len 32768
10769 118/300 Read block 2416 len 28672
10770 119/300 Write block 2936 len 8192 data: 3 3
10771 120/300 Read block 3003 len 32768
10772 121/300 Write block 1823 len 24576 data: 4 4 5 5 6 6
10773 122/300 Write block 185 len 20480 data: 2 2 2 3 3
10774 123/300 Write block 3027 len 12288 data: 3 4 4
10775 124/300 Write block 2455 len 28672 data: 4 3 3 3 3 4 4
10776 125/300 Read block 4725 len 40960
10777 126/300 Read block 4081 len 32768
10778 127/300 Write block 424 len 4096 data: 2
10779 128/300 Read block 1223 len 36864
10780 129/300 Write block 1465 len 36864 data: 2 2 2 2 2 2 2 2 2
10781 130/300 Flush
10782 131/300 Write block 4885 len 40960 data: 4 4 4 4 4 4 4 4 4 4
10783 132/300 Read block 4045 len 8192
10784 133/300 Write block 2391 len 24576 data: 2 2 2 2 2 2
10785 134/300 Write block 360 len 16384 data: 2 3 4 4
10786 135/300 Write block 289 len 16384 data: 4 5 5 5
10787 136/300 Read block 4329 len 28672
10788 137/300 Read block 593 len 8192
10789 138/300 Write block 1179 len 20480 data: 4 3 3 3 3
10790 139/300 Read block 3215 len 24576
10791 140/300 Read block 4268 len 36864
10792 141/300 Read block 4050 len 40960
10793 142/300 Read block 3722 len 8192
10794 143/300 Flush
10795 144/300 Write block 4449 len 16384 data: 5 5 3 2
10796 145/300 Write block 4242 len 12288 data: 2 2 2
10797 146/300 Read block 2143 len 4096
10798 147/300 Write block 2778 len 36864 data: 3 3 3 3 3 3 3 3 2
10799 148/300 Write block 1106 len 24576 data: 3 3 3 2 2 2
10800 149/300 Write block 3062 len 4096 data: 3
10801 150/300 Read block 363 len 28672
10802 151/300 Flush
10803 152/300 Read block 4739 len 36864
10804 153/300 Read block 1946 len 40960
10805 154/300 Read block 42 len 8192
10806 155/300 Read block 4118 len 32768
10807 156/300 Write block 3708 len 36864 data: 2 2 3 2 3 3 4 4 5
10808 157/300 Write block 2483 len 40960 data: 4 4 3 3 2 2 2 3 3 3
10809 158/300 Read block 3790 len 20480
10810 159/300 Flush
10811 160/300 Write block 1235 len 8192 data: 3 2
10812 161/300 Write block 988 len 20480 data: 5 5 5 4 4
10813 162/300 Write block 3563 len 20480 data: 5 4 5 4 3
10814 163/300 Read block 1091 len 24576
10815 164/300 Read block 1206 len 36864
10816 165/300 Read block 208 len 12288
10817 166/300 Read block 1508 len 36864
10818 167/300 Write block 1278 len 16384 data: 2 2 3 3
10819 168/300 Write block 4788 len 32768 data: 3 4 4 4 6 5 6 6
10820 169/300 Flush
10821 170/300 Read block 3525 len 16384
10822 171/300 Write block 1113 len 28672 data: 2 2 3 3 3 3 5
10823 172/300 Write block 446 len 20480 data: 5 5 5 4 3
10824 173/300 Read block 455 len 16384
10825 174/300 Write block 2230 len 32768 data: 4 4 4 4 4 4 4 4
10826 175/300 Read block 3968 len 40960
10827 176/300 Write block 90 len 32768 data: 3 4 3 3 3 3 2 3
10828 177/300 Write block 353 len 28672 data: 5 5 5 3 4 3 3
10829 178/300 Write block 4778 len 32768 data: 3 4 4 4 3 2 2 2
10830 179/300 Read block 4342 len 32768
10831 180/300 Read block 1600 len 28672
10832 181/300 Read block 1149 len 4096
10833 182/300 Read block 2375 len 24576
10834 183/300 Read block 2488 len 32768
10835 184/300 Flush
10836 185/300 Write block 3596 len 4096 data: 4
10837 186/300 Flush
10838 187/300 Read block 4399 len 32768
10839 188/300 Read block 2493 len 20480
10840 189/300 Flush
10841 190/300 Read block 4629 len 8192
10842 191/300 Write block 3212 len 24576 data: 3 4 3 3 3 3
10843 192/300 Flush
10844 193/300 Read block 4773 len 16384
10845 194/300 Flush
10846 195/300 Flush
10847 196/300 Read block 37 len 8192
10848 197/300 Write block 630 len 32768 data: 4 5 5 4 3 3 3 3
10849 198/300 Read block 2978 len 20480
10850 199/300 Read block 3882 len 4096
10851 200/300 Read block 4697 len 24576
10852 201/300 Read block 491 len 24576
10853 202/300 Flush
10854 203/300 Write block 153 len 28672 data: 3 3 2 2 3 3 4
10855 204/300 Read block 3003 len 20480
10856 205/300 Read block 3856 len 12288
10857 206/300 Write block 4940 len 20480 data: 2 3 3 3 4
10858 207/300 Write block 1977 len 4096 data: 4
10859 208/300 Read block 2850 len 36864
10860 209/300 Read block 4962 len 40960
10861 210/300 Read block 1453 len 24576
10862 211/300 Write block 659 len 24576 data: 3 3 2 2 2 2
10863 212/300 Read block 4303 len 32768
10864 213/300 Read block 4504 len 4096
10865 214/300 Flush
10866 215/300 Read block 160 len 24576
10867 216/300 Read block 3490 len 12288
10868 217/300 Read block 2076 len 40960
10869 218/300 Flush
10870 219/300 Read block 3323 len 4096
10871 220/300 Read block 2521 len 40960
10872 221/300 Read block 1779 len 32768
10873 222/300 Write block 154 len 20480 data: 4 3 3 4 4
10874 223/300 Read block 1176 len 36864
10875 224/300 Write block 2793 len 8192 data: 2 2
10876 225/300 Read block 2770 len 16384
10877 226/300 Write block 3561 len 36864 data: 4 4 6 5 6 5 4 3 3
10878 227/300 Read block 3885 len 16384
10879 228/300 Write block 728 len 12288 data: 4 3 2
10880 229/300 Read block 2132 len 20480
10881 230/300 Flush
10882 231/300 Flush
10883 232/300 Read block 4764 len 20480
10884 233/300 Write block 81 len 28672 data: 3 3 3 2 2 3 3
10885 234/300 Flush
10886 235/300 Write block 136 len 28672 data: 4 3 3 2 2 3 4
10887 236/300 Flush
10888 237/300 Read block 3717 len 8192
10889 238/300 Read block 2498 len 40960
10890 239/300 Read block 894 len 32768
10891 240/300 Write block 1406 len 24576 data: 5 5 4 3 3 4
10892 241/300 Read block 3261 len 8192
10893 242/300 Write block 4128 len 24576 data: 4 3 3 2 2 2
10894 243/300 Write block 3154 len 28672 data: 4 5 7 7 9 8 7
10895 244/300 Write block 1782 len 32768 data: 4 5 4 5 5 5 4 3
10896 245/300 Flush
10897 246/300 Read block 130 len 8192
10898 247/300 Read block 1462 len 40960
10899 248/300 Write block 659 len 16384 data: 4 4 3 3
10900 249/300 Write block 4203 len 20480 data: 5 5 5 5 4
10901 250/300 Read block 3059 len 28672
10902 251/300 Read block 800 len 12288
10903 252/300 Write block 1846 len 36864 data: 3 2 2 2 2 2 2 2 2
10904 253/300 Flush
10905 254/300 Write block 973 len 4096 data: 6
10906 255/300 Write block 4817 len 12288 data: 4 3 3
10907 256/300 Write block 1326 len 32768 data: 3 3 4 4 3 3 3 3
10908 257/300 Write block 4318 len 36864 data: 3 4 3 2 2 3 4 4 4
10909 258/300 Read block 4152 len 8192
10910 259/300 Read block 3385 len 36864
10911 260/300 Flush
10912 261/300 Write block 4776 len 40960 data: 3 3 4 5 5 5 4 3 3 3
10913 262/300 Write block 502 len 28672 data: 2 2 2 2 3 4 5
10914 263/300 Read block 816 len 32768
10915 264/300 Write block 3338 len 16384 data: 3 3 3 3
10916 265/300 Write block 4547 len 12288 data: 3 4 5
10917 266/300 Write block 3867 len 36864 data: 4 6 5 6 5 5 5 4 3
10918 267/300 Read block 2493 len 12288
10919 268/300 Write block 3750 len 16384 data: 2 3 3 3
10920 269/300 Write block 3359 len 24576 data: 2 2 2 3 2 2
10921 270/300 Write block 1420 len 36864 data: 4 5 5 4 2 3 3 3 3
10922 271/300 Flush
10923 272/300 Read block 2034 len 12288
10924 273/300 Write block 4040 len 32768 data: 4 4 5 5 6 5 4 4
10925 274/300 Write block 635 len 32768 data: 4 4 4 2 3 3 3 3
10926 275/300 Write block 804 len 32768 data: 3 3 3 2 2 2 4 4
10927 276/300 Read block 607 len 28672
10928 277/300 Flush
10929 278/300 Write block 1080 len 4096 data: 3
10930 279/300 Write block 4498 len 4096 data: 2
10931 280/300 Flush
10932 281/300 Write block 143 len 24576 data: 4 4 4 3 3 5
10933 282/300 Read block 1363 len 28672
10934 283/300 Read block 3771 len 24576
10935 284/300 Write block 3783 len 40960 data: 4 3 3 3 3 3 4 3 2 2
10936 285/300 Read block 1586 len 16384
10937 286/300 Read block 2364 len 20480
10938 287/300 Read block 336 len 8192
10939 288/300 Read block 4166 len 32768
10940 289/300 Flush
10941 290/300 Write block 3680 len 28672 data: 2 2 2 2 2 2 2
10942 291/300 Read block 1412 len 4096
10943 292/300 Write block 542 len 32768 data: 2 2 2 2 2 4 4 3
10944 293/300 Read block 1790 len 20480
10945 294/300 Read block 4689 len 8192
10946 295/300 Flush
10947 296/300 Read block 2855 len 36864
10948 297/300 Write block 4362 len 4096 data: 2
10949 298/300 Write block 4689 len 32768 data: 5 5 6 5 4 4 3 3
10950 299/300 Read block 2576 len 8192
10951 300/300 Read block 2315 len 20480
10952 Main task has work to do, go find it
10953 got dsc Start(1)
10954 start 1
10955 Feb 15 02:59:42.354 INFO request completed, latency_us: 222, response_code: 204, uri: /start/cid/1, method: POST, req_id: 774bde41-53c1-402c-8d7b-92ee83ea7716, remote_addr: 127.0.0.1:54744, local_addr: 127.0.0.1:9998
10956 Replay: started 1, returned:Ok(())
10957 ----------------------------------------------------------------
10958 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:300
10959 States: Active Offline Active
10960 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
10961 3015 Acked 4014 Write 3 Done New Done true
10962 3016 Acked 4015 Read 8 Done New Done true
10963 3017 Acked 4016 Read 10 Done New Done true
10964 3018 Acked 4017 Write 8 Done New Done false
10965 3019 Acked 4018 Read 2 Done New Done false
10966 3020 Acked 4019 Read 3 Done New Done false
10967 3021 Acked 4020 Write 9 Done New Done false
10968 3022 Acked 4021 Read 4 Done New Done false
10969 3023 Acked 4022 Read 4 Done New Done false
10970 3024 Acked 4023 Read 6 Done New Done false
10971 3025 Acked 4024 Read 4 Done New Done false
10972 3026 Acked 4025 Read 5 Done New Done false
10973 3027 Acked 4026 Flush 0 Done New Done false
10974 3028 Acked 4027 Write 5 Done New Done false
10975 3029 Acked 4028 Write 5 Done New Done false
10976 3030 Acked 4029 Flush 0 Done New Done false
10977 3031 Acked 4030 Write 7 Done New Done false
10978 3032 Acked 4031 Write 3 Done New Done false
10979 3033 Acked 4032 Write 7 Done New Done false
10980 3034 Acked 4033 Write 6 Done New Done false
10981 3035 Acked 4034 Write 2 Done New Done false
10982 3036 Acked 4035 Read 1 Done New Done false
10983 3037 Acked 4036 Read 9 Done New Done false
10984 3038 Acked 4037 Write 9 Done New Done false
10985 3039 Acked 4038 Read 6 Done New Done false
10986 3040 Acked 4039 Read 2 Done New Done false
10987 3041 Acked 4040 Read 4 Done New Done false
10988 3042 Acked 4041 Read 1 Done New Done false
10989 3043 Acked 4042 Read 1 Done New Done false
10990 3044 Acked 4043 Write 7 Done New Done false
10991 3045 Acked 4044 Read 2 Done New Done false
10992 3046 Acked 4045 Write 5 Done New Done false
10993 3047 Acked 4046 Write 3 Done New Done false
10994 3048 Acked 4047 Read 4 Done New Done false
10995 3049 Acked 4048 Read 9 Done New Done false
10996 3050 Acked 4049 Read 1 Done New Done false
10997 3051 Acked 4050 Read 10 Done New Done false
10998 3052 Acked 4051 Write 8 Done New Done false
10999 3053 Acked 4052 Read 2 Done New Done false
11000 3054 Acked 4053 Write 1 Done New Done false
11001 3055 Acked 4054 Write 2 Done New Done false
11002 3056 Acked 4055 Flush 0 Done New Done false
11003 3057 Acked 4056 Write 3 Done New Done false
11004 3058 Acked 4057 Write 3 Done New Done false
11005 3059 Acked 4058 Read 10 Done New Done false
11006 3060 Acked 4059 Read 10 Done New Done false
11007 3061 Acked 4060 Write 5 Done New Done false
11008 3062 Acked 4061 Read 7 Done New Done false
11009 3063 Acked 4062 Flush 0 Done New Done false
11010 3064 Acked 4063 Write 2 Done New Done false
11011 3065 Acked 4064 Read 2 Done New Done false
11012 3066 Acked 4065 Read 8 Done New Done false
11013 3067 Acked 4066 Read 1 Done New Done false
11014 3068 Acked 4067 Write 5 Done New Done false
11015 3069 Acked 4068 Read 9 Done New Done false
11016 3070 Acked 4069 Read 6 Done New Done false
11017 3071 Acked 4070 Read 3 Done New Done false
11018 3072 Acked 4071 Read 8 Done New Done false
11019 3073 Acked 4072 Read 6 Done New Done false
11020 3074 Acked 4073 Read 7 Done New Done false
11021 3075 Acked 4074 Write 10 Done New Done false
11022 3076 Acked 4075 Write 4 Done New Done false
11023 3077 Acked 4076 Read 10 Done New Done false
11024 3078 Acked 4077 Read 8 Done New Done false
11025 3079 Acked 4078 Write 6 Done New Done false
11026 3080 Acked 4079 Write 2 Done New Done false
11027 3081 Acked 4080 Flush 0 Done New Done false
11028 3082 Acked 4081 Read 2 Done New Done false
11029 3083 Acked 4082 Write 3 Done New Done false
11030 3084 Acked 4083 Write 4 Done New Done false
11031 3085 Acked 4084 Write 9 Done New Done false
11032 3086 Acked 4085 Read 4 Done New Done false
11033 3087 Acked 4086 Write 9 Done New Done false
11034 3088 Acked 4087 Read 5 Done New Done false
11035 3089 Acked 4088 Write 1 Done New Done false
11036 3090 Acked 4089 Flush 0 Done New Done false
11037 3091 Acked 4090 Read 6 Done New Done false
11038 3092 Acked 4091 Write 6 Done New Done false
11039 3093 Acked 4092 Write 5 Done New Done false
11040 3094 Acked 4093 Read 1 Done New Done false
11041 3095 Acked 4094 Write 6 Done New Done false
11042 3096 Acked 4095 Write 3 Done New Done false
11043 3097 Acked 4096 Write 8 Done New Done false
11044 3098 Acked 4097 Read 10 Done New Done false
11045 3099 Acked 4098 Read 1 Done New Done false
11046 3100 Acked 4099 Read 9 Done New Done false
11047 3101 Acked 4100 Read 8 Done New Done false
11048 3102 Acked 4101 Read 1 Done New Done false
11049 3103 Acked 4102 Write 5 Done New Done false
11050 3104 Acked 4103 Read 3 Done New Done false
11051 3105 Acked 4104 Write 3 Done New Done false
11052 3106 Acked 4105 Read 6 Done New Done false
11053 3107 Acked 4106 Read 1 Done New Done false
11054 3108 Acked 4107 Flush 0 Done New Done false
11055 3109 Acked 4108 Flush 0 Done New Done false
11056 3110 Acked 4109 Read 8 Done New Done false
11057 3111 Acked 4110 Flush 0 Done New Done false
11058 3112 Acked 4111 Write 5 Done New Done false
11059 3113 Acked 4112 Read 6 Done New Done false
11060 3114 Acked 4113 Read 7 Done New Done false
11061 3115 Acked 4114 Write 9 Done New Done false
11062 3116 Acked 4115 Write 7 Done New Done false
11063 3117 Acked 4116 Write 2 Done New Done false
11064 3118 Acked 4117 Read 8 Done New Done false
11065 3119 Acked 4118 Flush 0 Done New Done false
11066 3120 Acked 4119 Write 7 Done New Done false
11067 3121 Acked 4120 Read 2 Done New Done false
11068 3122 Acked 4121 Flush 0 Done New Done false
11069 3123 Acked 4122 Write 6 Done New Done false
11070 3124 Acked 4123 Write 4 Done New Done false
11071 3125 Acked 4124 Write 6 Done New Done false
11072 3126 Acked 4125 Read 8 Done New Done false
11073 3127 Acked 4126 Write 3 Done New Done false
11074 3128 Acked 4127 Read 10 Done New Done false
11075 3129 Acked 4128 Read 5 Done New Done false
11076 3130 Acked 4129 Read 9 Done New Done false
11077 3131 Acked 4130 Read 8 Done New Done false
11078 3132 Acked 4131 Read 7 Done New Done false
11079 3133 Acked 4132 Write 2 Done New Done false
11080 3134 Acked 4133 Read 8 Done New Done false
11081 3135 Acked 4134 Write 6 Done New Done false
11082 3136 Acked 4135 Write 5 Done New Done false
11083 3137 Acked 4136 Write 3 Done New Done false
11084 3138 Acked 4137 Write 7 Done New Done false
11085 3139 Acked 4138 Read 10 Done New Done false
11086 3140 Acked 4139 Read 8 Done New Done false
11087 3141 Acked 4140 Write 1 Done New Done false
11088 3142 Acked 4141 Read 9 Done New Done false
11089 3143 Acked 4142 Write 9 Done New Done false
11090 3144 Acked 4143 Flush 0 Done New Done false
11091 3145 Acked 4144 Write 10 Done New Done false
11092 3146 Acked 4145 Read 2 Done New Done false
11093 3147 Acked 4146 Write 6 Done New Done false
11094 3148 Acked 4147 Write 4 Done New Done false
11095 3149 Acked 4148 Write 4 Done New Done false
11096 3150 Acked 4149 Read 7 Done New Done false
11097 3151 Acked 4150 Read 2 Done New Done false
11098 3152 Acked 4151 Write 5 Done New Done false
11099 3153 Acked 4152 Read 6 Done New Done false
11100 3154 Acked 4153 Read 9 Done New Done false
11101 3155 Acked 4154 Read 10 Done New Done false
11102 3156 Acked 4155 Read 2 Done New Done false
11103 3157 Acked 4156 Flush 0 Done New Done false
11104 3158 Acked 4157 Write 4 Done New Done false
11105 3159 Acked 4158 Write 3 Done New Done false
11106 3160 Acked 4159 Read 1 Done New Done false
11107 3161 Acked 4160 Write 9 Done New Done false
11108 3162 Acked 4161 Write 6 Done New Done false
11109 3163 Acked 4162 Write 1 Done New Done false
11110 3164 Acked 4163 Read 7 Done New Done false
11111 3165 Acked 4164 Flush 0 Done New Done false
11112 3166 Acked 4165 Read 9 Done New Done false
11113 3167 Acked 4166 Read 10 Done New Done false
11114 3168 Acked 4167 Read 2 Done New Done false
11115 3169 Acked 4168 Read 8 Done New Done false
11116 3170 Acked 4169 Write 9 Done New Done false
11117 3171 Acked 4170 Write 10 Done New Done false
11118 3172 Acked 4171 Read 5 Done New Done false
11119 3173 Acked 4172 Flush 0 Done New Done false
11120 3174 Acked 4173 Write 2 Done New Done false
11121 3175 Acked 4174 Write 5 Done New Done false
11122 3176 Acked 4175 Write 5 Done New Done false
11123 3177 Acked 4176 Read 6 Done New Done false
11124 3178 Acked 4177 Read 9 Done New Done false
11125 3179 Acked 4178 Read 3 Done New Done false
11126 3180 Acked 4179 Read 9 Done New Done false
11127 3181 Acked 4180 Write 4 Done New Done false
11128 3182 Acked 4181 Write 8 Done New Done false
11129 3183 Acked 4182 Flush 0 Done New Done false
11130 3184 Acked 4183 Read 4 Done New Done false
11131 3185 Acked 4184 Write 7 Done New Done false
11132 3186 Acked 4185 Write 5 Done New Done false
11133 3187 Acked 4186 Read 4 Done New Done false
11134 3188 Acked 4187 Write 8 Done New Done false
11135 3189 Acked 4188 Read 10 Done New Done false
11136 3190 Acked 4189 Write 8 Done New Done false
11137 3191 Acked 4190 Write 7 Done New Done false
11138 3192 Acked 4191 Write 8 Done New Done false
11139 3193 Acked 4192 Read 8 Done New Done false
11140 3194 Acked 4193 Read 7 Done New Done false
11141 3195 Acked 4194 Read 1 Done New Done false
11142 3196 Acked 4195 Read 6 Done New Done false
11143 3197 Acked 4196 Read 8 Done New Done false
11144 3198 Acked 4197 Flush 0 Done New Done false
11145 3199 Acked 4198 Write 1 Done New Done false
11146 3200 Acked 4199 Flush 0 Done New Done false
11147 3201 Acked 4200 Read 8 Done New Done false
11148 3202 Acked 4201 Read 5 Done New Done false
11149 3203 Acked 4202 Flush 0 Done New Done false
11150 3204 Acked 4203 Read 2 Done New Done false
11151 3205 Acked 4204 Write 6 Done New Done false
11152 3206 Acked 4205 Flush 0 Done New Done false
11153 3207 Acked 4206 Read 4 Done New Done false
11154 3208 Acked 4207 Flush 0 Done New Done false
11155 3209 Acked 4208 Flush 0 Done New Done false
11156 3210 Acked 4209 Read 2 Done New Done false
11157 3211 Acked 4210 Write 8 Done New Done false
11158 3212 Acked 4211 Read 5 Done New Done false
11159 3213 Acked 4212 Read 1 Done New Done false
11160 3214 Acked 4213 Read 6 Done New Done false
11161 3215 Acked 4214 Read 6 Done New Done false
11162 3216 Acked 4215 Flush 0 Done New Done false
11163 3217 Acked 4216 Write 7 Done New Done false
11164 3218 Acked 4217 Read 5 Done New Done false
11165 3219 Acked 4218 Read 3 Done New Done false
11166 3220 Acked 4219 Write 5 Done New Done false
11167 3221 Acked 4220 Write 1 Done New Done false
11168 3222 Acked 4221 Read 9 Done New Done false
11169 3223 Acked 4222 Read 10 Done New Done false
11170 3224 Acked 4223 Read 6 Done New Done false
11171 3225 Acked 4224 Write 6 Done New Done false
11172 3226 Acked 4225 Read 8 Done New Done false
11173 3227 Acked 4226 Read 1 Done New Done false
11174 3228 Acked 4227 Flush 0 Done New Done false
11175 3229 Acked 4228 Read 6 Done New Done false
11176 3230 Acked 4229 Read 3 Done New Done false
11177 3231 Acked 4230 Read 10 Done New Done false
11178 3232 Acked 4231 Flush 0 Done New Done false
11179 3233 Acked 4232 Read 1 Done New Done false
11180 3234 Acked 4233 Read 10 Done New Done false
11181 3235 Acked 4234 Read 8 Done New Done false
11182 3236 Acked 4235 Write 5 Done New Done false
11183 3237 Acked 4236 Read 9 Done New Done false
11184 3238 Acked 4237 Write 2 Done New Done false
11185 3239 Acked 4238 Read 4 Done New Done false
11186 3240 Acked 4239 Write 9 Done New Done false
11187 3241 Acked 4240 Read 4 Done New Done false
11188 3242 Acked 4241 Write 3 Done New Done false
11189 3243 Acked 4242 Read 5 Done New Done false
11190 3244 Acked 4243 Flush 0 Done New Done false
11191 3245 Acked 4244 Flush 0 Done New Done false
11192 3246 Acked 4245 Read 5 Done New Done false
11193 3247 Acked 4246 Write 7 Done New Done false
11194 3248 Acked 4247 Flush 0 Done New Done false
11195 3249 Acked 4248 Write 7 Done New Done false
11196 3250 Acked 4249 Flush 0 Done New Done false
11197 3251 Acked 4250 Read 2 Done New Done false
11198 3252 Acked 4251 Read 10 Done New Done false
11199 3253 Acked 4252 Read 8 Done New Done false
11200 3254 Acked 4253 Write 6 Done New Done false
11201 3255 Acked 4254 Read 2 Done New Done false
11202 3256 Acked 4255 Write 6 Done New Done false
11203 3257 Acked 4256 Write 7 Done New Done false
11204 3258 Acked 4257 Write 8 Done New Done false
11205 3259 Acked 4258 Flush 0 Done New Done false
11206 3260 Acked 4259 Read 2 Done New Done false
11207 3261 Acked 4260 Read 10 Done New Done false
11208 3262 Acked 4261 Write 4 Done New Done false
11209 3263 Acked 4262 Write 5 Done New Done false
11210 3264 Acked 4263 Read 7 Done New Done false
11211 3265 Acked 4264 Read 3 Done New Done false
11212 3266 Acked 4265 Write 9 Done New Done false
11213 3267 Acked 4266 Flush 0 Done New Done false
11214 3268 Acked 4267 Write 1 Done New Done false
11215 3269 Acked 4268 Write 3 Done New Done false
11216 3270 Acked 4269 Write 8 Done New Done false
11217 3271 Acked 4270 Write 9 Done New Done false
11218 3272 Acked 4271 Read 2 Done New Done false
11219 3273 Acked 4272 Read 9 Done New Done false
11220 3274 Acked 4273 Flush 0 Done New Done false
11221 3275 Acked 4274 Write 10 Done New Done false
11222 3276 Acked 4275 Write 7 Done New Done false
11223 3277 Acked 4276 Read 8 Done New Done false
11224 3278 Acked 4277 Write 4 Done New Done false
11225 3279 Acked 4278 Write 3 Done New Done false
11226 3280 Acked 4279 Write 9 Done New Done false
11227 3281 Acked 4280 Read 3 Done New Done false
11228 3282 Acked 4281 Write 4 Done New Done false
11229 3283 Acked 4282 Write 6 Done New Done false
11230 3284 Acked 4283 Write 9 Done New Done false
11231 3285 Acked 4284 Flush 0 Done New Done false
11232 3286 Acked 4285 Read 3 Done New Done false
11233 3287 Acked 4286 Write 8 Done New Done false
11234 3288 Acked 4287 Write 8 Done New Done false
11235 3289 Acked 4288 Write 8 Done New Done false
11236 3290 Acked 4289 Read 7 Done New Done false
11237 3291 Acked 4290 Flush 0 Done New Done false
11238 3292 Acked 4291 Write 1 Done New Done false
11239 3293 Acked 4292 Write 1 Done New Done false
11240 3294 Acked 4293 Flush 0 Done New Done false
11241 3295 Acked 4294 Write 6 Done New Done false
11242 3296 Acked 4295 Read 7 Done New Done false
11243 3297 Acked 4296 Read 6 Done New Done false
11244 3298 Acked 4297 Write 10 Done New Done false
11245 3299 Acked 4298 Read 4 Done New Done false
11246 3300 Acked 4299 Read 5 Done New Done false
11247 3301 Acked 4300 Read 2 Done New Done false
11248 3302 Acked 4301 Read 8 Done New Done false
11249 3303 Acked 4302 Flush 0 Done New Done false
11250 3304 Acked 4303 Write 7 Done New Done false
11251 3305 Acked 4304 Read 1 Done New Done false
11252 3306 Acked 4305 Write 8 Done New Done false
11253 3307 Acked 4306 Read 5 Done New Done false
11254 3308 Acked 4307 Read 2 Done New Done false
11255 3309 Acked 4308 Flush 0 Done New Done false
11256 3310 Acked 4309 Read 9 Done New Done false
11257 3311 Acked 4310 Write 1 Done New Done false
11258 3312 Acked 4311 Write 8 Done New Done false
11259 3313 Acked 4312 Read 2 Done New Done false
11260 3314 Acked 4313 Read 5 Done New Done false
11261 STATES DS:0 DS:1 DS:2 TOTAL
11262 New 0 300 0 300
11263 Sent 0 0 0 0
11264 Done 300 0 300 600
11265 Skipped 0 0 0 0
11266 Error 0 0 0 0
11267 Last Flush: 4308 4013 4308
11268 Downstairs last five completed: 4013 4012 4011 4010 4009
11269 Upstairs last five completed: 3314 3313 3312 3311 3310
11270 CLIENT: Up:0 ds:300 act:2
112712024-02-15T02:59:43.024ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
112722024-02-15T02:59:43.024ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
112732024-02-15T02:59:43.024ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
112742024-02-15T02:59:43.024ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
112752024-02-15T02:59:43.024ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
112762024-02-15T02:59:43.024ZINFOcrucible: [1] client re-new 301 jobs since flush 4013 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
112772024-02-15T02:59:43.024ZINFOcrucible: client task is exiting = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
11278 [8820] got start action
11279 [8820] I am going to restart
11280 Starting downstairs at port 8820
11281 Make output file at "/tmp/dsc/downstairs-8820.txt"
11282 [8820][1] reports Starting
112832024-02-15T02:59:44.025ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
112842024-02-15T02:59:44.025ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
112852024-02-15T02:59:44.025ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
112862024-02-15T02:59:44.025ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
112872024-02-15T02:59:44.025ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
112882024-02-15T02:59:44.025ZINFOcrucible: [1] client re-new 301 jobs since flush 4013 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
112892024-02-15T02:59:44.026ZINFOcrucible: client task is exiting = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
11290 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1178)
11291 [8820][1] reports Running
112922024-02-15T02:59:45.027ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
112932024-02-15T02:59:45.027ZINFOcrucible: ds_connection connected = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
112942024-02-15T02:59:45.028ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
112952024-02-15T02:59:45.028ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
112962024-02-15T02:59:45.028ZINFOcrucible: send last flush ID to this DS: 4013 = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
112972024-02-15T02:59:45.028ZINFOcrucible: Replied this last flush ID: 4013 = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
112982024-02-15T02:59:45.028ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
112992024-02-15T02:59:45.029ZINFOcrucible: [1] Transition from Offline to Replay = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
113002024-02-15T02:59:45.029ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
113012024-02-15T02:59:45.029ZINFOcrucible: [1] Transition from Replay to Active = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
11302 ----------------------------------------------------------------
11303 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
11304 Downstairs last five completed: 4314 4313 4312 4311 4310
11305 Upstairs last five completed: 3315 3314 3313 3312 3311
11306 CLIENT: Up:0 ds:0 act:3
11307 Replay: All jobs finished, all DS active.
11308 Main task has work to do, go find it
11309 got dsc Stop(1)
11310 stop 1
11311 Feb 15 02:59:46.358 INFO request completed, latency_us: 354, response_code: [8820] Got stop action so:false kr:false
11312 204, uri: /stop/cid/1, method: POST, req_id: 4817cabe-86ed-42eb-95af-15834cd27007, remote_addr: 127.0.0.1:54744, local_addr: 127.0.0.1:9998
113132024-02-15T02:59:46.358ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
113142024-02-15T02:59:46.358ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
113152024-02-15T02:59:46.358ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
113162024-02-15T02:59:46.358ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
113172024-02-15T02:59:46.358ZINFOcrucible: [1] client re-new 0 jobs since flush 4314 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
11318 Feb 15 02:59:46.358 INFO request completed, latency_us: 193, response_code: 200, uri: /state/cid/1, method: GET, req_id: ed840c17-c83e-4d7b-8f57-78651645e345, remote_addr: 127.0.0.1:54744, local_addr: 127.0.0.1:9998
113192024-02-15T02:59:46.359ZINFOcrucible: client task is exiting = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
11320 [8820] Exited with: ExitStatus(unix_wait_status(9))
11321 [8820][1] reports Exit
113222024-02-15T02:59:47.359ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
113232024-02-15T02:59:47.359ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
113242024-02-15T02:59:47.359ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
113252024-02-15T02:59:47.359ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
113262024-02-15T02:59:47.359ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
113272024-02-15T02:59:47.359ZINFOcrucible: [1] client re-new 0 jobs since flush 4314 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
113282024-02-15T02:59:47.359ZINFOcrucible: client task is exiting = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
113292024-02-15T02:59:48.361ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
113302024-02-15T02:59:48.361ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
113312024-02-15T02:59:48.361ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
113322024-02-15T02:59:48.361ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
113332024-02-15T02:59:48.361ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
113342024-02-15T02:59:48.361ZINFOcrucible: [1] client re-new 0 jobs since flush 4314 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
113352024-02-15T02:59:48.361ZINFOcrucible: client task is exiting = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
113362024-02-15T02:59:49.362ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
113372024-02-15T02:59:49.362ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
113382024-02-15T02:59:49.362ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
113392024-02-15T02:59:49.362ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
113402024-02-15T02:59:49.362ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
113412024-02-15T02:59:49.362ZINFOcrucible: [1] client re-new 0 jobs since flush 4314 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
113422024-02-15T02:59:49.362ZINFOcrucible: client task is exiting = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
11343 Feb 15 02:59:50.360 INFO request completed, latency_us: 349, response_code: 200, uri: /state/cid/1, method: GET, req_id: 5fb60314-06dc-4f65-8d3c-b627c7765618, remote_addr: 127.0.0.1:54744, local_addr: 127.0.0.1:9998
11344 001/300 Read block 3419 len 8192
11345 002/300 Write block 4343 len 36864 data: 2 3 3 4 4 4 2 2 2
11346 {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30003/300 Read block 3419 len 24576
11347 ,"time":"2024-02-15T02:59:50.363542531Z","hostname":"ip-10-150-1-67.us-west-2.compute.internal","pid":1167,"":"io task","client":"1","":"downstairs","session_id":"4f5a63e5-aa00-4890-9f6b-96c2e069fdb2"}
113482024-02-15T02:59:50.363ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
113492024-02-15T02:59:50.363ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
113502024-02-15T02:59:50.363ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
113512024-02-15T02:59:50.363ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
113522024-02-15T02:59:50.363ZINFOcrucible: [1] client re-new 2 jobs since flush 4314 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
113532024-02-15T02:59:50.363ZINFOcrucible: client task is exiting = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
11354 004/300 Write block 791 len 8192 data: 3 3
11355 005/300 Write block 2186 len 32768 data: 3 3 4 3 3 3 4 4
11356 006/300 Read block 4109 len 4096
11357 007/300 Read block 2542 len 36864
11358 008/300 Flush
11359 009/300 Read block 169 len 4096
11360 010/300 Flush
11361 011/300 Flush
11362 012/300 Read block 122 len 20480
11363 013/300 Read block 78 len 24576
11364 014/300 Write block 4911 len 20480 data: 4 3 3 3 3
11365 015/300 Read block 3279 len 36864
11366 016/300 Read block 1173 len 4096
11367 017/300 Read block 1143 len 8192
11368 018/300 Read block 697 len 20480
11369 019/300 Flush
11370 020/300 Read block 4714 len 12288
11371 021/300 Read block 4799 len 24576
11372 022/300 Read block 737 len 36864
11373 023/300 Write block 2357 len 40960 data: 4 4 4 3 3 2 2 2 2 3
11374 024/300 Read block 2033 len 24576
11375 025/300 Read block 3706 len 28672
11376 026/300 Read block 54 len 12288
11377 027/300 Read block 4933 len 32768
11378 028/300 Read block 2823 len 4096
11379 029/300 Read block 1686 len 24576
11380 030/300 Read block 4437 len 16384
11381 031/300 Read block 3382 len 16384
11382 032/300 Read block 4313 len 24576
11383 033/300 Write block 2350 len 20480 data: 2 2 3 3 4
11384 034/300 Flush
11385 035/300 Write block 1767 len 32768 data: 3 2 2 2 2 2 2 2
11386 036/300 Read block 2323 len 20480
11387 037/300 Read block 2828 len 16384
11388 038/300 Read block 2928 len 8192
11389 039/300 Write block 1846 len 8192 data: 4 3
11390 040/300 Write block 4059 len 20480 data: 3 3 3 3 3
11391 041/300 Write block 1958 len 8192 data: 6 5
11392 042/300 Read block 951 len 8192
11393 043/300 Read block 3769 len 12288
11394 044/300 Read block 1632 len 12288
11395 045/300 Flush
11396 046/300 Write block 629 len 36864 data: 4 5 6 6 5 4 5 5 5
11397 047/300 Write block 3356 len 12288 data: 2 2 2
11398 048/300 Read block 1195 len 12288
11399 049/300 Read block 193 len 16384
11400 050/300 Read block 3144 len 16384
11401 051/300 Read block 2689 len 28672
11402 052/300 Flush
11403 053/300 Read block 19 len 16384
11404 054/300 Write block 3822 len 4096 data: 3
11405 055/300 Flush
11406 056/300 Flush
11407 057/300 Write block 4311 len 36864 data: 5 5 4 4 4 3 4 4 5
11408 058/300 Write block 2838 len 20480 data: 4 4 4 4 3
11409 059/300 Write block 501 len 32768 data: 3 3 3 3 3 4 5 6
11410 060/300 Read block 1291 len 40960
11411 061/300 Write block 2361 len 12288 data: 4 3 3
11412 062/300 Read block 1949 len 40960
11413 063/300 Read block 4080 len 8192
11414 064/300 Read block 1523 len 8192
11415 065/300 Write block 2166 len 8192 data: 4 4
11416 066/300 Read block 3368 len 32768
11417 067/300 Write block 450 len 28672 data: 4 4 4 3 3 3 3
11418 068/300 Write block 4517 len 16384 data: 2 2 2 2
11419 069/300 Write block 1122 len 24576 data: 6 5 5 4 4 4
11420 070/300 Read block 4690 len 16384
11421 071/300 Read block 2305 len 8192
11422 072/300 Read block 423 len 36864
11423 073/300 Read block 2092 len 12288
11424 074/300 Read block 3534 len 4096
11425 075/300 Read block 906 len 12288
11426 076/300 Flush
11427 077/300 Write block 2202 len 24576 data: 3 3 4 5 6 5
11428 078/300 Read block 745 len 12288
11429 079/300 Write block 2501 len 36864 data: 4 4 4 4 4 4 4 3 3
11430 080/300 Read block 3331 len 40960
11431 081/300 Read block 209 len 36864
11432 082/300 Write block 291 len 32768 data: 6 6 5 4 4 3 3 2
11433 083/300 Read block 4418 len 40960
11434 084/300 Write block 4142 len 16384 data: 5 6 6 6
11435 085/300 Read block 4577 len 12288
11436 086/300 Read block 4827 len 12288
11437 087/300 Write block 819 len 36864 data: 5 5 5 4 4 3 3 4 3
11438 088/300 Write block 490 len 16384 data: 4 4 6 5
11439 089/300 Write block 1949 len 20480 data: 2 2 4 3 4
11440 090/300 Read block 4343 len 16384
11441 091/300 Read block 2892 len 32768
11442 092/300 Read block 1999 len 36864
11443 093/300 Read block 2019 len 36864
11444 094/300 Read block 1182 len 4096
11445 095/300 Read block 4191 len 32768
11446 096/300 Flush
11447 097/300 Flush
11448 098/300 Write block 4488 len 16384 data: 2 2 2 2
11449 099/300 Write block 2419 len 20480 data: 3 2 3 3 3
11450 100/300 Read block 4497 len 40960
11451 101/300 Write block 4687 len 4096 data: 5
11452 102/300 Read block 4326 len 8192
11453 103/300 Flush
11454 104/300 Write block 3172 len 32768 data: 2 2 2 3 3 3 3 3
11455 105/300 Read block 1614 len 40960
11456 106/300 Write block 4117 len 24576 data: 7 6 3 2 2 2
11457 107/300 Read block 4680 len 40960
11458 108/300 Read block 3936 len 8192
11459 109/300 Read block 250 len 28672
11460 110/300 Write block 2524 len 24576 data: 4 4 3 3 3 2
11461 111/300 Read block 328 len 12288
11462 112/300 Read block 448 len 8192
11463 113/300 Write block 1388 len 16384 data: 3 2 2 2
11464 114/300 Read block 3509 len 40960
11465 115/300 Read block 4796 len 20480
11466 116/300 Write block 4047 len 32768 data: 5 4 3 3 4 4 4 4
11467 117/300 Read block 4931 len 32768
11468 118/300 Write block 2720 len 28672 data: 2 2 2 2 2 3 3
11469 119/300 Read block 628 len 40960
11470 120/300 Read block 4639 len 40960
11471 121/300 Read block 1202 len 8192
11472 122/300 Read block 3964 len 20480
11473 123/300 Read block 1338 len 32768
11474 124/300 Write block 229 len 40960 data: 4 4 4 3 3 3 3 4 4 4
11475 125/300 Flush
11476 126/300 Read block 4621 len 28672
11477 127/300 Read block 3276 len 8192
11478 128/300 Write block 2409 len 4096 data: 2
11479 129/300 Write block 1446 len 20480 data: 2 2 2 3 3
11480 130/300 Read block 1172 len 32768
11481 131/300 Read block 1406 len 20480
11482 132/300 Read block 3200 len 24576
11483 133/300 Read block 1716 len 40960
11484 134/300 Read block 918 len 36864
11485 135/300 Write block 3951 len 4096 data: 5
11486 136/300 Read block 73 len 36864
11487 137/300 Read block 1284 len 12288
11488 138/300 Read block 1778 len 24576
11489 139/300 Write block 4971 len 24576 data: 6 5 4 3 3 3
11490 140/300 Write block 4278 len 28672 data: 2 2 3 3 3 4 4
11491 141/300 Write block 4060 len 12288 data: 4 4 4
11492 142/300 Write block 4460 len 36864 data: 3 3 3 2 2 2 2 2 4
11493 143/300 Write block 2130 len 12288 data: 6 5 6
11494 144/300 Read block 3400 len 16384
11495 145/300 Read block 3589 len 16384
11496 146/300 Write block 574 len 4096 data: 3
11497 147/300 Write block 4468 len 40960 data: 5 4 3 3 3 3 3 3 3 3
11498 148/300 Read block 1523 len 12288
11499 149/300 Write block 728 len 28672 data: 5 4 3 3 4 4 5
11500 150/300 Read block 4249 len 4096
11501 151/300 Read block 3692 len 28672
11502 152/300 Read block 3024 len 12288
11503 153/300 Write block 2094 len 28672 data: 3 3 3 3 3 3 3
11504 154/300 Write block 4425 len 12288 data: 6 6 6
11505 155/300 Read block 1040 len 8192
11506 156/300 Write block 70 len 16384 data: 3 3 3 4
11507 157/300 Flush
11508 158/300 Read block 3873 len 28672
11509 159/300 Write block 833 len 28672 data: 3 4 3 4 4 4 4
11510 160/300 Read block 134 len 8192
11511 161/300 Write block 457 len 20480 data: 3 2 2 2 2
11512 162/300 Read block 1274 len 36864
11513 163/300 Read block 1544 len 32768
11514 164/300 Write block 4725 len 8192 data: 4 4
11515 165/300 Read block 2986 len 8192
11516 166/300 Read block 1272 len 24576
11517 167/300 Write block 3705 len 28672 data: 2 2 2 3 3 4 3
11518 168/300 Write block 173 len 24576 data: 3 4 3 4 4 3
11519 169/300 Flush
11520 170/300 Read block 4774 len 12288
11521 171/300 Read block 1490 len 20480
11522 172/300 Read block 3299 len 20480
11523 173/300 Write block 1572 len 28672 data: 4 4 4 3 2 3 2
11524 174/300 Write block 1861 len 4096 data: 2
11525 175/300 Write block 2531 len 36864 data: 2 2 2 2 3 2 2 2 2
11526 176/300 Read block 3778 len 8192
11527 177/300 Write block 3616 len 20480 data: 3 4 5 5 5
11528 178/300 Read block 1757 len 16384
11529 179/300 Read block 1677 len 32768
11530 180/300 Write block 3424 len 4096 data: 3
11531 181/300 Write block 3555 len 20480 data: 3 3 4 3 4
11532 182/300 Write block 280 len 12288 data: 2 2 3
11533 183/300 Write block 2296 len 12288 data: 4 4 4
11534 184/300 Read block 418 len 24576
11535 185/300 Flush
11536 186/300 Read block 237 len 8192
11537 187/300 Flush
11538 188/300 Write block 3149 len 24576 data: 5 6 4 4 4 5
11539 189/300 Read block 1702 len 20480
11540 190/300 Write block 4674 len 24576 data: 6 8 7 4 6 6
11541 191/300 Read block 954 len 16384
11542 192/300 Write block 2167 len 20480 data: 5 3 3 4 4
11543 193/300 Flush
11544 194/300 Read block 2669 len 28672
11545 195/300 Read block 102 len 8192
11546 196/300 Write block 2861 len 32768 data: 5 4 4 4 3 4 4 3
11547 197/300 Read block 4270 len 36864
11548 198/300 Read block 2321 len 8192
11549 199/300 Read block 3525 len 4096
11550 200/300 Write block 4832 len 32768 data: 2 2 4 4 4 4 5 3
11551 201/300 Read block 384 len 28672
11552 202/300 Write block 3778 len 16384 data: 4 4 3 4
11553 203/300 Read block 1954 len 36864
11554 204/300 Read block 4608 len 40960
11555 205/300 Write block 1807 len 36864 data: 3 3 3 5 5 4 4 4 4
11556 206/300 Write block 2881 len 12288 data: 5 5 5
11557 207/300 Write block 4139 len 24576 data: 6 6 5 6 7 7
11558 208/300 Flush
11559 209/300 Write block 4133 len 20480 data: 3 3 3 4 4
11560 210/300 Read block 1570 len 20480
11561 211/300 Read block 3562 len 24576
11562 212/300 Read block 3238 len 32768
11563 213/300 Write block 3317 len 40960 data: 2 2 2 3 3 3 2 2 3 4
11564 214/300 Write block 1114 len 8192 data: 3 4
11565 215/300 Write block 4462 len 40960 data: 4 3 3 3 3 3 6 5 4 4
11566 216/300 Write block 4047 len 16384 data: 6 5 4 4
11567 217/300 Write block 1489 len 16384 data: 4 3 3 3
11568 218/300 Read block 4910 len 28672
11569 219/300 Read block 597 len 32768
11570 220/300 Write block 146 len 16384 data: 4 4 6 3
11571 221/300 Flush
11572 222/300 Write block 4273 len 4096 data: 2
11573 223/300 Read block 91 len 24576
11574 224/300 Write block 2472 len 24576 data: 3 3 3 3 3 3
11575 225/300 Write block 1630 len 32768 data: 3 2 2 2 2 2 2 2
11576 226/300 Read block 4623 len 8192
11577 227/300 Write block 3421 len 4096 data: 3
11578 228/300 Write block 1900 len 28672 data: 2 2 2 3 3 3 3
11579 229/300 Read block 535 len 20480
11580 230/300 Read block 199 len 12288
11581 231/300 Write block 1576 len 40960 data: 3 4 3 3 3 5 4 5 5 4
11582 232/300 Read block 342 len 36864
11583 233/300 Read block 4806 len 36864
11584 234/300 Write block 2542 len 40960 data: 3 4 4 3 4 4 4 4 3 3
11585 235/300 Read block 258 len 4096
11586 236/300 Write block 1302 len 16384 data: 3 3 3 3
11587 237/300 Write block 2915 len 4096 data: 3
11588 238/300 Read block 1027 len 28672
11589 239/300 Read block 4546 len 16384
11590 240/300 Read block 728 len 36864
11591 241/300 Read block 3546 len 16384
11592 242/300 Read block 1371 len 24576
11593 243/300 Read block 2576 len 24576
11594 244/300 Write block 1236 len 28672 data: 3 2 2 2 3 3 3
11595 245/300 Read block 1160 len 16384
11596 246/300 Read block 132 len 4096
11597 247/300 Flush
11598 248/300 Write block 4057 len 4096 data: 3
11599 249/300 Write block 786 len 32768 data: 3 4 4 4 4 4 4 3
11600 250/300 Write block 2877 len 16384 data: 2 2 3 4
11601 251/300 Read block 4907 len 4096
11602 252/300 Write block 2792 len 24576 data: 2 3 3 2 2 2
11603 253/300 Write block 3774 len 12288 data: 4 4 4
11604 254/300 Write block 1016 len 32768 data: 2 3 3 3 4 4 4 4
11605 255/300 Read block 4367 len 36864
11606 256/300 Flush
11607 257/300 Read block 371 len 8192
11608 258/300 Read block 2870 len 32768
11609 259/300 Write block 3130 len 24576 data: 4 3 2 2 4 5
11610 260/300 Read block 3208 len 36864
11611 261/300 Write block 3289 len 16384 data: 3 3 3 2
11612 262/300 Read block 140 len 24576
11613 263/300 Write block 2673 len 32768 data: 3 3 3 2 3 3 3 3
11614 264/300 Flush
11615 265/300 Write block 464 len 32768 data: 2 2 2 2 3 4 4 2
11616 266/300 Read block 4740 len 20480
11617 267/300 Read block 3707 len 4096
11618 268/300 Write block 176 len 24576 data: 5 5 4 3 4 4
11619 269/300 Write block 3075 len 4096 data: 4
11620 270/300 Write block 4770 len 8192 data: 3 3
11621 271/300 Write block 932 len 36864 data: 2 2 2 2 3 4 4 4 3
11622 272/300 Read block 2751 len 36864
11623 273/300 Write block 1425 len 28672 data: 4 4 4 4 3 3 3
11624 274/300 Read block 2285 len 40960
11625 275/300 Read block 859 len 4096
11626 276/300 Write block 2797 len 12288 data: 3 2 2
11627 277/300 Read block 4024 len 32768
11628 278/300 Write block 72 len 4096 data: 4
11629 279/300 Write block 416 len 8192 data: 2 2
11630 280/300 Read block 4784 len 16384
11631 281/300 Read block 2268 len 28672
11632 282/300 Write block 4641 len 24576 data: 6 6 5 5 5 4
11633 283/300 Read block 3960 len 32768
11634 284/300 Read block 2396 len 8192
11635 285/300 Read block 3361 len 40960
11636 286/300 Flush
11637 287/300 Read block 1035 len 16384
11638 288/300 Read block 940 len 12288
11639 289/300 Read block 4247 len 20480
11640 290/300 Write block 3504 len 20480 data: 3 3 3 3 3
11641 291/300 Write block 2738 len 32768 data: 2 3 3 4 3 3 3 4
11642 292/300 Read block 1690 len 36864
11643 293/300 Read block 559 len 20480
11644 294/300 Flush
11645 295/300 Flush
11646 296/300 Read block 4334 len 36864
11647 297/300 Write block 2590 len 28672 data: 3 2 2 3 3 3 3
11648 298/300 Read block 2262 len 24576
11649 299/300 Write block 4376 len 4096 data: 6
11650 300/300 Write block 3856 len 36864 data: 3 3 3 3 4 3 3 4 4
11651 Main task has work to do, go find it
11652 got dsc Start(1)
11653 start 1
11654 Feb 15 02:59:50.691 INFO request completed, latency_us: 245, response_code: 204, uri: /start/cid/1, method: POST, req_id: 04fd74d9-527a-4570-a853-0e9d32028ba0, remote_addr: 127.0.0.1:54744, local_addr: 127.0.0.1:9998
11655 Replay: started 1, returned:Ok(())
11656 ----------------------------------------------------------------
11657 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:300
11658 States: Active Offline Active
11659 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
11660 3316 Acked 4315 Read 2 Done New Done true
11661 3317 Acked 4316 Write 9 Done New Done true
11662 3318 Acked 4317 Read 6 Done New Done false
11663 3319 Acked 4318 Write 2 Done New Done false
11664 3320 Acked 4319 Write 8 Done New Done false
11665 3321 Acked 4320 Read 1 Done New Done false
11666 3322 Acked 4321 Read 9 Done New Done false
11667 3323 Acked 4322 Flush 0 Done New Done false
11668 3324 Acked 4323 Read 1 Done New Done false
11669 3325 Acked 4324 Flush 0 Done New Done false
11670 3326 Acked 4325 Flush 0 Done New Done false
11671 3327 Acked 4326 Read 5 Done New Done false
11672 3328 Acked 4327 Read 6 Done New Done false
11673 3329 Acked 4328 Write 5 Done New Done false
11674 3330 Acked 4329 Read 9 Done New Done false
11675 3331 Acked 4330 Read 1 Done New Done false
11676 3332 Acked 4331 Read 2 Done New Done false
11677 3333 Acked 4332 Read 5 Done New Done false
11678 3334 Acked 4333 Flush 0 Done New Done false
11679 3335 Acked 4334 Read 3 Done New Done false
11680 3336 Acked 4335 Read 6 Done New Done false
11681 3337 Acked 4336 Read 9 Done New Done false
11682 3338 Acked 4337 Write 10 Done New Done false
11683 3339 Acked 4338 Read 6 Done New Done false
11684 3340 Acked 4339 Read 7 Done New Done false
11685 3341 Acked 4340 Read 3 Done New Done false
11686 3342 Acked 4341 Read 8 Done New Done false
11687 3343 Acked 4342 Read 1 Done New Done false
11688 3344 Acked 4343 Read 6 Done New Done false
11689 3345 Acked 4344 Read 4 Done New Done false
11690 3346 Acked 4345 Read 4 Done New Done false
11691 3347 Acked 4346 Read 6 Done New Done false
11692 3348 Acked 4347 Write 5 Done New Done false
11693 3349 Acked 4348 Flush 0 Done New Done false
11694 3350 Acked 4349 Write 8 Done New Done false
11695 3351 Acked 4350 Read 5 Done New Done false
11696 3352 Acked 4351 Read 4 Done New Done false
11697 3353 Acked 4352 Read 2 Done New Done false
11698 3354 Acked 4353 Write 2 Done New Done false
11699 3355 Acked 4354 Write 5 Done New Done false
11700 3356 Acked 4355 Write 2 Done New Done false
11701 3357 Acked 4356 Read 2 Done New Done false
11702 3358 Acked 4357 Read 3 Done New Done false
11703 3359 Acked 4358 Read 3 Done New Done false
11704 3360 Acked 4359 Flush 0 Done New Done false
11705 3361 Acked 4360 Write 9 Done New Done false
11706 3362 Acked 4361 Write 3 Done New Done false
11707 3363 Acked 4362 Read 3 Done New Done false
11708 3364 Acked 4363 Read 4 Done New Done false
11709 3365 Acked 4364 Read 4 Done New Done false
11710 3366 Acked 4365 Read 7 Done New Done false
11711 3367 Acked 4366 Flush 0 Done New Done false
11712 3368 Acked 4367 Read 4 Done New Done false
11713 3369 Acked 4368 Write 1 Done New Done false
11714 3370 Acked 4369 Flush 0 Done New Done false
11715 3371 Acked 4370 Flush 0 Done New Done false
11716 3372 Acked 4371 Write 9 Done New Done false
11717 3373 Acked 4372 Write 5 Done New Done false
11718 3374 Acked 4373 Write 8 Done New Done false
11719 3375 Acked 4374 Read 10 Done New Done false
11720 3376 Acked 4375 Write 3 Done New Done false
11721 3377 Acked 4376 Read 10 Done New Done false
11722 3378 Acked 4377 Read 2 Done New Done false
11723 3379 Acked 4378 Read 2 Done New Done false
11724 3380 Acked 4379 Write 2 Done New Done false
11725 3381 Acked 4380 Read 8 Done New Done false
11726 3382 Acked 4381 Write 7 Done New Done false
11727 3383 Acked 4382 Write 4 Done New Done false
11728 3384 Acked 4383 Write 6 Done New Done false
11729 3385 Acked 4384 Read 4 Done New Done false
11730 3386 Acked 4385 Read 2 Done New Done false
11731 3387 Acked 4386 Read 9 Done New Done false
11732 3388 Acked 4387 Read 3 Done New Done false
11733 3389 Acked 4388 Read 1 Done New Done false
11734 3390 Acked 4389 Read 3 Done New Done false
11735 3391 Acked 4390 Flush 0 Done New Done false
11736 3392 Acked 4391 Write 6 Done New Done false
11737 3393 Acked 4392 Read 3 Done New Done false
11738 3394 Acked 4393 Write 9 Done New Done false
11739 3395 Acked 4394 Read 10 Done New Done false
11740 3396 Acked 4395 Read 9 Done New Done false
11741 3397 Acked 4396 Write 8 Done New Done false
11742 3398 Acked 4397 Read 10 Done New Done false
11743 3399 Acked 4398 Write 4 Done New Done false
11744 3400 Acked 4399 Read 3 Done New Done false
11745 3401 Acked 4400 Read 3 Done New Done false
11746 3402 Acked 4401 Write 9 Done New Done false
11747 3403 Acked 4402 Write 4 Done New Done false
11748 3404 Acked 4403 Write 5 Done New Done false
11749 3405 Acked 4404 Read 4 Done New Done false
11750 3406 Acked 4405 Read 8 Done New Done false
11751 3407 Acked 4406 Read 9 Done New Done false
11752 3408 Acked 4407 Read 9 Done New Done false
11753 3409 Acked 4408 Read 1 Done New Done false
11754 3410 Acked 4409 Read 8 Done New Done false
11755 3411 Acked 4410 Flush 0 Done New Done false
11756 3412 Acked 4411 Flush 0 Done New Done false
11757 3413 Acked 4412 Write 4 Done New Done false
11758 3414 Acked 4413 Write 5 Done New Done false
11759 3415 Acked 4414 Read 10 Done New Done false
11760 3416 Acked 4415 Write 1 Done New Done false
11761 3417 Acked 4416 Read 2 Done New Done false
11762 3418 Acked 4417 Flush 0 Done New Done false
11763 3419 Acked 4418 Write 8 Done New Done false
11764 3420 Acked 4419 Read 10 Done New Done false
11765 3421 Acked 4420 Write 6 Done New Done false
11766 3422 Acked 4421 Read 10 Done New Done false
11767 3423 Acked 4422 Read 2 Done New Done false
11768 3424 Acked 4423 Read 7 Done New Done false
11769 3425 Acked 4424 Write 6 Done New Done false
11770 3426 Acked 4425 Read 3 Done New Done false
11771 3427 Acked 4426 Read 2 Done New Done false
11772 3428 Acked 4427 Write 4 Done New Done false
11773 3429 Acked 4428 Read 10 Done New Done false
11774 3430 Acked 4429 Read 5 Done New Done false
11775 3431 Acked 4430 Write 8 Done New Done false
11776 3432 Acked 4431 Read 8 Done New Done false
11777 3433 Acked 4432 Write 7 Done New Done false
11778 3434 Acked 4433 Read 10 Done New Done false
11779 3435 Acked 4434 Read 10 Done New Done false
11780 3436 Acked 4435 Read 2 Done New Done false
11781 3437 Acked 4436 Read 5 Done New Done false
11782 3438 Acked 4437 Read 8 Done New Done false
11783 3439 Acked 4438 Write 10 Done New Done false
11784 3440 Acked 4439 Flush 0 Done New Done false
11785 3441 Acked 4440 Read 7 Done New Done false
11786 3442 Acked 4441 Read 2 Done New Done false
11787 3443 Acked 4442 Write 1 Done New Done false
11788 3444 Acked 4443 Write 5 Done New Done false
11789 3445 Acked 4444 Read 8 Done New Done false
11790 3446 Acked 4445 Read 5 Done New Done false
11791 3447 Acked 4446 Read 6 Done New Done false
11792 3448 Acked 4447 Read 10 Done New Done false
11793 3449 Acked 4448 Read 9 Done New Done false
11794 3450 Acked 4449 Write 1 Done New Done false
11795 3451 Acked 4450 Read 9 Done New Done false
11796 3452 Acked 4451 Read 3 Done New Done false
11797 3453 Acked 4452 Read 6 Done New Done false
11798 3454 Acked 4453 Write 6 Done New Done false
11799 3455 Acked 4454 Write 7 Done New Done false
11800 3456 Acked 4455 Write 3 Done New Done false
11801 3457 Acked 4456 Write 9 Done New Done false
11802 3458 Acked 4457 Write 3 Done New Done false
11803 3459 Acked 4458 Read 4 Done New Done false
11804 3460 Acked 4459 Read 4 Done New Done false
11805 3461 Acked 4460 Write 1 Done New Done false
11806 3462 Acked 4461 Write 10 Done New Done false
11807 3463 Acked 4462 Read 3 Done New Done false
11808 3464 Acked 4463 Write 7 Done New Done false
11809 3465 Acked 4464 Read 1 Done New Done false
11810 3466 Acked 4465 Read 7 Done New Done false
11811 3467 Acked 4466 Read 3 Done New Done false
11812 3468 Acked 4467 Write 7 Done New Done false
11813 3469 Acked 4468 Write 3 Done New Done false
11814 3470 Acked 4469 Read 2 Done New Done false
11815 3471 Acked 4470 Write 4 Done New Done false
11816 3472 Acked 4471 Flush 0 Done New Done false
11817 3473 Acked 4472 Read 7 Done New Done false
11818 3474 Acked 4473 Write 7 Done New Done false
11819 3475 Acked 4474 Read 2 Done New Done false
11820 3476 Acked 4475 Write 5 Done New Done false
11821 3477 Acked 4476 Read 9 Done New Done false
11822 3478 Acked 4477 Read 8 Done New Done false
11823 3479 Acked 4478 Write 2 Done New Done false
11824 3480 Acked 4479 Read 2 Done New Done false
11825 3481 Acked 4480 Read 6 Done New Done false
11826 3482 Acked 4481 Write 7 Done New Done false
11827 3483 Acked 4482 Write 6 Done New Done false
11828 3484 Acked 4483 Flush 0 Done New Done false
11829 3485 Acked 4484 Read 3 Done New Done false
11830 3486 Acked 4485 Read 5 Done New Done false
11831 3487 Acked 4486 Read 5 Done New Done false
11832 3488 Acked 4487 Write 7 Done New Done false
11833 3489 Acked 4488 Write 1 Done New Done false
11834 3490 Acked 4489 Write 9 Done New Done false
11835 3491 Acked 4490 Read 2 Done New Done false
11836 3492 Acked 4491 Write 5 Done New Done false
11837 3493 Acked 4492 Read 4 Done New Done false
11838 3494 Acked 4493 Read 8 Done New Done false
11839 3495 Acked 4494 Write 1 Done New Done false
11840 3496 Acked 4495 Write 5 Done New Done false
11841 3497 Acked 4496 Write 3 Done New Done false
11842 3498 Acked 4497 Write 3 Done New Done false
11843 3499 Acked 4498 Read 6 Done New Done false
11844 3500 Acked 4499 Flush 0 Done New Done false
11845 3501 Acked 4500 Read 2 Done New Done false
11846 3502 Acked 4501 Flush 0 Done New Done false
11847 3503 Acked 4502 Write 6 Done New Done false
11848 3504 Acked 4503 Read 5 Done New Done false
11849 3505 Acked 4504 Write 6 Done New Done false
11850 3506 Acked 4505 Read 4 Done New Done false
11851 3507 Acked 4506 Write 5 Done New Done false
11852 3508 Acked 4507 Flush 0 Done New Done false
11853 3509 Acked 4508 Read 7 Done New Done false
11854 3510 Acked 4509 Read 2 Done New Done false
11855 3511 Acked 4510 Write 8 Done New Done false
11856 3512 Acked 4511 Read 9 Done New Done false
11857 3513 Acked 4512 Read 2 Done New Done false
11858 3514 Acked 4513 Read 1 Done New Done false
11859 3515 Acked 4514 Write 8 Done New Done false
11860 3516 Acked 4515 Read 7 Done New Done false
11861 3517 Acked 4516 Write 4 Done New Done false
11862 3518 Acked 4517 Read 9 Done New Done false
11863 3519 Acked 4518 Read 10 Done New Done false
11864 3520 Acked 4519 Write 9 Done New Done false
11865 3521 Acked 4520 Write 3 Done New Done false
11866 3522 Acked 4521 Write 6 Done New Done false
11867 3523 Acked 4522 Flush 0 Done New Done false
11868 3524 Acked 4523 Write 5 Done New Done false
11869 3525 Acked 4524 Read 5 Done New Done false
11870 3526 Acked 4525 Read 6 Done New Done false
11871 3527 Acked 4526 Read 8 Done New Done false
11872 3528 Acked 4527 Write 10 Done New Done false
11873 3529 Acked 4528 Write 2 Done New Done false
11874 3530 Acked 4529 Write 10 Done New Done false
11875 3531 Acked 4530 Write 4 Done New Done false
11876 3532 Acked 4531 Write 4 Done New Done false
11877 3533 Acked 4532 Read 7 Done New Done false
11878 3534 Acked 4533 Read 8 Done New Done false
11879 3535 Acked 4534 Write 4 Done New Done false
11880 3536 Acked 4535 Flush 0 Done New Done false
11881 3537 Acked 4536 Write 1 Done New Done false
11882 3538 Acked 4537 Read 6 Done New Done false
11883 3539 Acked 4538 Write 6 Done New Done false
11884 3540 Acked 4539 Write 8 Done New Done false
11885 3541 Acked 4540 Read 2 Done New Done false
11886 3542 Acked 4541 Write 1 Done New Done false
11887 3543 Acked 4542 Write 7 Done New Done false
11888 3544 Acked 4543 Read 5 Done New Done false
11889 3545 Acked 4544 Read 3 Done New Done false
11890 3546 Acked 4545 Write 10 Done New Done false
11891 3547 Acked 4546 Read 9 Done New Done false
11892 3548 Acked 4547 Read 9 Done New Done false
11893 3549 Acked 4548 Write 10 Done New Done false
11894 3550 Acked 4549 Read 1 Done New Done false
11895 3551 Acked 4550 Write 4 Done New Done false
11896 3552 Acked 4551 Write 1 Done New Done false
11897 3553 Acked 4552 Read 7 Done New Done false
11898 3554 Acked 4553 Read 4 Done New Done false
11899 3555 Acked 4554 Read 9 Done New Done false
11900 3556 Acked 4555 Read 4 Done New Done false
11901 3557 Acked 4556 Read 6 Done New Done false
11902 3558 Acked 4557 Read 6 Done New Done false
11903 3559 Acked 4558 Write 7 Done New Done false
11904 3560 Acked 4559 Read 4 Done New Done false
11905 3561 Acked 4560 Read 1 Done New Done false
11906 3562 Acked 4561 Flush 0 Done New Done false
11907 3563 Acked 4562 Write 1 Done New Done false
11908 3564 Acked 4563 Write 8 Done New Done false
11909 3565 Acked 4564 Write 4 Done New Done false
11910 3566 Acked 4565 Read 1 Done New Done false
11911 3567 Acked 4566 Write 6 Done New Done false
11912 3568 Acked 4567 Write 3 Done New Done false
11913 3569 Acked 4568 Write 8 Done New Done false
11914 3570 Acked 4569 Read 9 Done New Done false
11915 3571 Acked 4570 Flush 0 Done New Done false
11916 3572 Acked 4571 Read 2 Done New Done false
11917 3573 Acked 4572 Read 8 Done New Done false
11918 3574 Acked 4573 Write 6 Done New Done false
11919 3575 Acked 4574 Read 9 Done New Done false
11920 3576 Acked 4575 Write 4 Done New Done false
11921 3577 Acked 4576 Read 6 Done New Done false
11922 3578 Acked 4577 Write 8 Done New Done false
11923 3579 Acked 4578 Flush 0 Done New Done false
11924 3580 Acked 4579 Write 8 Done New Done false
11925 3581 Acked 4580 Read 5 Done New Done false
11926 3582 Acked 4581 Read 1 Done New Done false
11927 3583 Acked 4582 Write 6 Done New Done false
11928 3584 Acked 4583 Write 1 Done New Done false
11929 3585 Acked 4584 Write 2 Done New Done false
11930 3586 Acked 4585 Write 9 Done New Done false
11931 3587 Acked 4586 Read 9 Done New Done false
11932 3588 Acked 4587 Write 7 Done New Done false
11933 3589 Acked 4588 Read 10 Done New Done false
11934 3590 Acked 4589 Read 1 Done New Done false
11935 3591 Acked 4590 Write 3 Done New Done false
11936 3592 Acked 4591 Read 8 Done New Done false
11937 3593 Acked 4592 Write 1 Done New Done false
11938 3594 Acked 4593 Write 2 Done New Done false
11939 3595 Acked 4594 Read 4 Done New Done false
11940 3596 Acked 4595 Read 7 Done New Done false
11941 3597 Acked 4596 Write 6 Done New Done false
11942 3598 Acked 4597 Read 8 Done New Done false
11943 3599 Acked 4598 Read 2 Done New Done false
11944 3600 Acked 4599 Read 10 Done New Done false
11945 3601 Acked 4600 Flush 0 Done New Done false
11946 3602 Acked 4601 Read 4 Done New Done false
11947 3603 Acked 4602 Read 3 Done New Done false
11948 3604 Acked 4603 Read 5 Done New Done false
11949 3605 Acked 4604 Write 5 Done New Done false
11950 3606 Acked 4605 Write 8 Done New Done false
11951 3607 Acked 4606 Read 9 Done New Done false
11952 3608 Acked 4607 Read 5 Done New Done false
11953 3609 Acked 4608 Flush 0 Done New Done false
11954 3610 Acked 4609 Flush 0 Done New Done false
11955 3611 Acked 4610 Read 9 Done New Done false
11956 3612 Acked 4611 Write 7 Done New Done false
11957 3613 Acked 4612 Read 6 Done New Done false
11958 3614 Acked 4613 Write 1 Done New Done false
11959 3615 Acked 4614 Write 9 Done New Done false
11960 STATES DS:0 DS:1 DS:2 TOTAL
11961 New 0 300 0 300
11962 Sent 0 0 0 0
11963 Done 300 0 300 600
11964 Skipped 0 0 0 0
11965 Error 0 0 0 0
11966 Last Flush: 4609 4314 4609
11967 Downstairs last five completed: 4314 4313 4312 4311 4310
11968 Upstairs last five completed: 3615 3614 3613 3612 3611
11969 CLIENT: Up:0 ds:300 act:2
119702024-02-15T02:59:51.365ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
119712024-02-15T02:59:51.365ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
119722024-02-15T02:59:51.365ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
119732024-02-15T02:59:51.365ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
119742024-02-15T02:59:51.365ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
119752024-02-15T02:59:51.365ZINFOcrucible: [1] client re-new 301 jobs since flush 4314 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
119762024-02-15T02:59:51.365ZINFOcrucible: client task is exiting = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
119772024-02-15T02:59:52.366ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
119782024-02-15T02:59:52.366ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
119792024-02-15T02:59:52.366ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
119802024-02-15T02:59:52.366ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
119812024-02-15T02:59:52.366ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
119822024-02-15T02:59:52.366ZINFOcrucible: [1] client re-new 301 jobs since flush 4314 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
119832024-02-15T02:59:52.367ZINFOcrucible: client task is exiting = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
119842024-02-15T02:59:53.368ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
119852024-02-15T02:59:53.368ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
119862024-02-15T02:59:53.368ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
119872024-02-15T02:59:53.368ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
119882024-02-15T02:59:53.368ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
119892024-02-15T02:59:53.368ZINFOcrucible: [1] client re-new 301 jobs since flush 4314 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
119902024-02-15T02:59:53.368ZINFOcrucible: client task is exiting = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
119912024-02-15T02:59:54.369ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
119922024-02-15T02:59:54.370ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
119932024-02-15T02:59:54.370ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
119942024-02-15T02:59:54.370ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
119952024-02-15T02:59:54.370ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
119962024-02-15T02:59:54.370ZINFOcrucible: [1] client re-new 301 jobs since flush 4314 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
119972024-02-15T02:59:54.370ZINFOcrucible: client task is exiting = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
11998 ----------------------------------------------------------------
11999 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
12000 States: Active Offline Active
12001 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
12002 3316 Acked 4315 Read 2 Done New Done true
12003 3317 Acked 4316 Write 9 Done New Done true
12004 3318 Acked 4317 Read 6 Done New Done true
12005 3319 Acked 4318 Write 2 Done New Done true
12006 3320 Acked 4319 Write 8 Done New Done true
12007 3321 Acked 4320 Read 1 Done New Done true
12008 3322 Acked 4321 Read 9 Done New Done true
12009 3323 Acked 4322 Flush 0 Done New Done true
12010 3324 Acked 4323 Read 1 Done New Done true
12011 3325 Acked 4324 Flush 0 Done New Done true
12012 3326 Acked 4325 Flush 0 Done New Done true
12013 3327 Acked 4326 Read 5 Done New Done true
12014 3328 Acked 4327 Read 6 Done New Done true
12015 3329 Acked 4328 Write 5 Done New Done true
12016 3330 Acked 4329 Read 9 Done New Done true
12017 3331 Acked 4330 Read 1 Done New Done true
12018 3332 Acked 4331 Read 2 Done New Done true
12019 3333 Acked 4332 Read 5 Done New Done true
12020 3334 Acked 4333 Flush 0 Done New Done true
12021 3335 Acked 4334 Read 3 Done New Done true
12022 3336 Acked 4335 Read 6 Done New Done true
12023 3337 Acked 4336 Read 9 Done New Done true
12024 3338 Acked 4337 Write 10 Done New Done true
12025 3339 Acked 4338 Read 6 Done New Done true
12026 3340 Acked 4339 Read 7 Done New Done true
12027 3341 Acked 4340 Read 3 Done New Done true
12028 3342 Acked 4341 Read 8 Done New Done true
12029 3343 Acked 4342 Read 1 Done New Done true
12030 3344 Acked 4343 Read 6 Done New Done true
12031 3345 Acked 4344 Read 4 Done New Done true
12032 3346 Acked 4345 Read 4 Done New Done true
12033 3347 Acked 4346 Read 6 Done New Done true
12034 3348 Acked 4347 Write 5 Done New Done true
12035 3349 Acked 4348 Flush 0 Done New Done true
12036 3350 Acked 4349 Write 8 Done New Done true
12037 3351 Acked 4350 Read 5 Done New Done true
12038 3352 Acked 4351 Read 4 Done New Done true
12039 3353 Acked 4352 Read 2 Done New Done true
12040 3354 Acked 4353 Write 2 Done New Done true
12041 3355 Acked 4354 Write 5 Done New Done true
12042 3356 Acked 4355 Write 2 Done New Done true
12043 3357 Acked 4356 Read 2 Done New Done true
12044 3358 Acked 4357 Read 3 Done New Done true
12045 3359 Acked 4358 Read 3 Done New Done true
12046 3360 Acked 4359 Flush 0 Done New Done true
12047 3361 Acked 4360 Write 9 Done New Done true
12048 3362 Acked 4361 Write 3 Done New Done true
12049 3363 Acked 4362 Read 3 Done New Done true
12050 3364 Acked 4363 Read 4 Done New Done true
12051 3365 Acked 4364 Read 4 Done New Done true
12052 3366 Acked 4365 Read 7 Done New Done true
12053 3367 Acked 4366 Flush 0 Done New Done true
12054 3368 Acked 4367 Read 4 Done New Done true
12055 3369 Acked 4368 Write 1 Done New Done true
12056 3370 Acked 4369 Flush 0 Done New Done true
12057 3371 Acked 4370 Flush 0 Done New Done true
12058 3372 Acked 4371 Write 9 Done New Done true
12059 3373 Acked 4372 Write 5 Done New Done true
12060 3374 Acked 4373 Write 8 Done New Done true
12061 3375 Acked 4374 Read 10 Done New Done true
12062 3376 Acked 4375 Write 3 Done New Done true
12063 3377 Acked 4376 Read 10 Done New Done true
12064 3378 Acked 4377 Read 2 Done New Done true
12065 3379 Acked 4378 Read 2 Done New Done true
12066 3380 Acked 4379 Write 2 Done New Done true
12067 3381 Acked 4380 Read 8 Done New Done true
12068 3382 Acked 4381 Write 7 Done New Done true
12069 3383 Acked 4382 Write 4 Done New Done true
12070 3384 Acked 4383 Write 6 Done New Done true
12071 3385 Acked 4384 Read 4 Done New Done true
12072 3386 Acked 4385 Read 2 Done New Done true
12073 3387 Acked 4386 Read 9 Done New Done true
12074 3388 Acked 4387 Read 3 Done New Done true
12075 3389 Acked 4388 Read 1 Done New Done true
12076 3390 Acked 4389 Read 3 Done New Done true
12077 3391 Acked 4390 Flush 0 Done New Done true
12078 3392 Acked 4391 Write 6 Done New Done true
12079 3393 Acked 4392 Read 3 Done New Done true
12080 3394 Acked 4393 Write 9 Done New Done true
12081 3395 Acked 4394 Read 10 Done New Done true
12082 3396 Acked 4395 Read 9 Done New Done true
12083 3397 Acked 4396 Write 8 Done New Done true
12084 3398 Acked 4397 Read 10 Done New Done true
12085 3399 Acked 4398 Write 4 Done New Done true
12086 3400 Acked 4399 Read 3 Done New Done true
12087 3401 Acked 4400 Read 3 Done New Done true
12088 3402 Acked 4401 Write 9 Done New Done true
12089 3403 Acked 4402 Write 4 Done New Done true
12090 3404 Acked 4403 Write 5 Done New Done true
12091 3405 Acked 4404 Read 4 Done New Done true
12092 3406 Acked 4405 Read 8 Done New Done true
12093 3407 Acked 4406 Read 9 Done New Done true
12094 3408 Acked 4407 Read 9 Done New Done true
12095 3409 Acked 4408 Read 1 Done New Done true
12096 3410 Acked 4409 Read 8 Done New Done true
12097 3411 Acked 4410 Flush 0 Done New Done true
12098 3412 Acked 4411 Flush 0 Done New Done true
12099 3413 Acked 4412 Write 4 Done New Done true
12100 3414 Acked 4413 Write 5 Done New Done true
12101 3415 Acked 4414 Read 10 Done New Done true
12102 3416 Acked 4415 Write 1 Done New Done true
12103 3417 Acked 4416 Read 2 Done New Done true
12104 3418 Acked 4417 Flush 0 Done New Done true
12105 3419 Acked 4418 Write 8 Done New Done true
12106 3420 Acked 4419 Read 10 Done New Done true
12107 3421 Acked 4420 Write 6 Done New Done true
12108 3422 Acked 4421 Read 10 Done New Done true
12109 3423 Acked 4422 Read 2 Done New Done true
12110 3424 Acked 4423 Read 7 Done New Done true
12111 3425 Acked 4424 Write 6 Done New Done true
12112 3426 Acked 4425 Read 3 Done New Done true
12113 3427 Acked 4426 Read 2 Done New Done true
12114 3428 Acked 4427 Write 4 Done New Done true
12115 3429 Acked 4428 Read 10 Done New Done true
12116 3430 Acked 4429 Read 5 Done New Done true
12117 3431 Acked 4430 Write 8 Done New Done true
12118 3432 Acked 4431 Read 8 Done New Done true
12119 3433 Acked 4432 Write 7 Done New Done true
12120 3434 Acked 4433 Read 10 Done New Done true
12121 3435 Acked 4434 Read 10 Done New Done true
12122 3436 Acked 4435 Read 2 Done New Done true
12123 3437 Acked 4436 Read 5 Done New Done true
12124 3438 Acked 4437 Read 8 Done New Done true
12125 3439 Acked 4438 Write 10 Done New Done true
12126 3440 Acked 4439 Flush 0 Done New Done true
12127 3441 Acked 4440 Read 7 Done New Done true
12128 3442 Acked 4441 Read 2 Done New Done true
12129 3443 Acked 4442 Write 1 Done New Done true
12130 3444 Acked 4443 Write 5 Done New Done true
12131 3445 Acked 4444 Read 8 Done New Done true
12132 3446 Acked 4445 Read 5 Done New Done true
12133 3447 Acked 4446 Read 6 Done New Done true
12134 3448 Acked 4447 Read 10 Done New Done true
12135 3449 Acked 4448 Read 9 Done New Done true
12136 3450 Acked 4449 Write 1 Done New Done true
12137 3451 Acked 4450 Read 9 Done New Done true
12138 3452 Acked 4451 Read 3 Done New Done true
12139 3453 Acked 4452 Read 6 Done New Done true
12140 3454 Acked 4453 Write 6 Done New Done true
12141 3455 Acked 4454 Write 7 Done New Done true
12142 3456 Acked 4455 Write 3 Done New Done true
12143 3457 Acked 4456 Write 9 Done New Done true
12144 3458 Acked 4457 Write 3 Done New Done true
12145 3459 Acked 4458 Read 4 Done New Done true
12146 3460 Acked 4459 Read 4 Done New Done true
12147 3461 Acked 4460 Write 1 Done New Done true
12148 3462 Acked 4461 Write 10 Done New Done true
12149 3463 Acked 4462 Read 3 Done New Done true
12150 3464 Acked 4463 Write 7 Done New Done true
12151 3465 Acked 4464 Read 1 Done New Done true
12152 3466 Acked 4465 Read 7 Done New Done true
12153 3467 Acked 4466 Read 3 Done New Done true
12154 3468 Acked 4467 Write 7 Done New Done true
12155 3469 Acked 4468 Write 3 Done New Done true
12156 3470 Acked 4469 Read 2 Done New Done true
12157 3471 Acked 4470 Write 4 Done New Done true
12158 3472 Acked 4471 Flush 0 Done New Done true
12159 3473 Acked 4472 Read 7 Done New Done true
12160 3474 Acked 4473 Write 7 Done New Done true
12161 3475 Acked 4474 Read 2 Done New Done true
12162 3476 Acked 4475 Write 5 Done New Done true
12163 3477 Acked 4476 Read 9 Done New Done true
12164 3478 Acked 4477 Read 8 Done New Done true
12165 3479 Acked 4478 Write 2 Done New Done true
12166 3480 Acked 4479 Read 2 Done New Done true
12167 3481 Acked 4480 Read 6 Done New Done true
12168 3482 Acked 4481 Write 7 Done New Done true
12169 3483 Acked 4482 Write 6 Done New Done true
12170 3484 Acked 4483 Flush 0 Done New Done true
12171 3485 Acked 4484 Read 3 Done New Done true
12172 3486 Acked 4485 Read 5 Done New Done true
12173 3487 Acked 4486 Read 5 Done New Done true
12174 3488 Acked 4487 Write 7 Done New Done true
12175 3489 Acked 4488 Write 1 Done New Done true
12176 3490 Acked 4489 Write 9 Done New Done true
12177 3491 Acked 4490 Read 2 Done New Done true
12178 3492 Acked 4491 Write 5 Done New Done true
12179 3493 Acked 4492 Read 4 Done New Done true
12180 3494 Acked 4493 Read 8 Done New Done true
12181 3495 Acked 4494 Write 1 Done New Done true
12182 3496 Acked 4495 Write 5 Done New Done true
12183 3497 Acked 4496 Write 3 Done New Done true
12184 3498 Acked 4497 Write 3 Done New Done true
12185 3499 Acked 4498 Read 6 Done New Done true
12186 3500 Acked 4499 Flush 0 Done New Done true
12187 3501 Acked 4500 Read 2 Done New Done true
12188 3502 Acked 4501 Flush 0 Done New Done true
12189 3503 Acked 4502 Write 6 Done New Done true
12190 3504 Acked 4503 Read 5 Done New Done true
12191 3505 Acked 4504 Write 6 Done New Done true
12192 3506 Acked 4505 Read 4 Done New Done true
12193 3507 Acked 4506 Write 5 Done New Done true
12194 3508 Acked 4507 Flush 0 Done New Done true
12195 3509 Acked 4508 Read 7 Done New Done true
12196 3510 Acked 4509 Read 2 Done New Done true
12197 3511 Acked 4510 Write 8 Done New Done true
12198 3512 Acked 4511 Read 9 Done New Done true
12199 3513 Acked 4512 Read 2 Done New Done true
12200 3514 Acked 4513 Read 1 Done New Done true
12201 3515 Acked 4514 Write 8 Done New Done true
12202 3516 Acked 4515 Read 7 Done New Done true
12203 3517 Acked 4516 Write 4 Done New Done true
12204 3518 Acked 4517 Read 9 Done New Done true
12205 3519 Acked 4518 Read 10 Done New Done true
12206 3520 Acked 4519 Write 9 Done New Done true
12207 3521 Acked 4520 Write 3 Done New Done true
12208 3522 Acked 4521 Write 6 Done New Done true
12209 3523 Acked 4522 Flush 0 Done New Done true
12210 3524 Acked 4523 Write 5 Done New Done true
12211 3525 Acked 4524 Read 5 Done New Done true
12212 3526 Acked 4525 Read 6 Done New Done true
12213 3527 Acked 4526 Read 8 Done New Done true
12214 3528 Acked 4527 Write 10 Done New Done true
12215 3529 Acked 4528 Write 2 Done New Done true
12216 3530 Acked 4529 Write 10 Done New Done true
12217 3531 Acked 4530 Write 4 Done New Done true
12218 3532 Acked 4531 Write 4 Done New Done true
12219 3533 Acked 4532 Read 7 Done New Done true
12220 3534 Acked 4533 Read 8 Done New Done true
12221 3535 Acked 4534 Write 4 Done New Done true
12222 3536 Acked 4535 Flush 0 Done New Done true
12223 3537 Acked 4536 Write 1 Done New Done true
12224 3538 Acked 4537 Read 6 Done New Done true
12225 3539 Acked 4538 Write 6 Done New Done true
12226 3540 Acked 4539 Write 8 Done New Done true
12227 3541 Acked 4540 Read 2 Done New Done true
12228 3542 Acked 4541 Write 1 Done New Done true
12229 3543 Acked 4542 Write 7 Done New Done true
12230 3544 Acked 4543 Read 5 Done New Done true
12231 3545 Acked 4544 Read 3 Done New Done true
12232 3546 Acked 4545 Write 10 Done New Done true
12233 3547 Acked 4546 Read 9 Done New Done true
12234 3548 Acked 4547 Read 9 Done New Done true
12235 3549 Acked 4548 Write 10 Done New Done true
12236 3550 Acked 4549 Read 1 Done New Done true
12237 3551 Acked 4550 Write 4 Done New Done true
12238 3552 Acked 4551 Write 1 Done New Done true
12239 3553 Acked 4552 Read 7 Done New Done true
12240 3554 Acked 4553 Read 4 Done New Done true
12241 3555 Acked 4554 Read 9 Done New Done true
12242 3556 Acked 4555 Read 4 Done New Done true
12243 3557 Acked 4556 Read 6 Done New Done true
12244 3558 Acked 4557 Read 6 Done New Done true
12245 3559 Acked 4558 Write 7 Done New Done true
12246 3560 Acked 4559 Read 4 Done New Done true
12247 3561 Acked 4560 Read 1 Done New Done true
12248 3562 Acked 4561 Flush 0 Done New Done true
12249 3563 Acked 4562 Write 1 Done New Done true
12250 3564 Acked 4563 Write 8 Done New Done true
12251 3565 Acked 4564 Write 4 Done New Done true
12252 3566 Acked 4565 Read 1 Done New Done true
12253 3567 Acked 4566 Write 6 Done New Done true
12254 3568 Acked 4567 Write 3 Done New Done true
12255 3569 Acked 4568 Write 8 Done New Done true
12256 3570 Acked 4569 Read 9 Done New Done true
12257 3571 Acked 4570 Flush 0 Done New Done true
12258 3572 Acked 4571 Read 2 Done New Done true
12259 3573 Acked 4572 Read 8 Done New Done true
12260 3574 Acked 4573 Write 6 Done New Done true
12261 3575 Acked 4574 Read 9 Done New Done true
12262 3576 Acked 4575 Write 4 Done New Done true
12263 3577 Acked 4576 Read 6 Done New Done true
12264 3578 Acked 4577 Write 8 Done New Done true
12265 3579 Acked 4578 Flush 0 Done New Done true
12266 3580 Acked 4579 Write 8 Done New Done true
12267 3581 Acked 4580 Read 5 Done New Done true
12268 3582 Acked 4581 Read 1 Done New Done true
12269 3583 Acked 4582 Write 6 Done New Done true
12270 3584 Acked 4583 Write 1 Done New Done true
12271 3585 Acked 4584 Write 2 Done New Done true
12272 3586 Acked 4585 Write 9 Done New Done true
12273 3587 Acked 4586 Read 9 Done New Done true
12274 3588 Acked 4587 Write 7 Done New Done true
12275 3589 Acked 4588 Read 10 Done New Done true
12276 3590 Acked 4589 Read 1 Done New Done true
12277 3591 Acked 4590 Write 3 Done New Done true
12278 3592 Acked 4591 Read 8 Done New Done true
12279 3593 Acked 4592 Write 1 Done New Done true
12280 3594 Acked 4593 Write 2 Done New Done true
12281 3595 Acked 4594 Read 4 Done New Done true
12282 3596 Acked 4595 Read 7 Done New Done true
12283 3597 Acked 4596 Write 6 Done New Done true
12284 3598 Acked 4597 Read 8 Done New Done true
12285 3599 Acked 4598 Read 2 Done New Done true
12286 3600 Acked 4599 Read 10 Done New Done true
12287 3601 Acked 4600 Flush 0 Done New Done true
12288 3602 Acked 4601 Read 4 Done New Done true
12289 3603 Acked 4602 Read 3 Done New Done true
12290 3604 Acked 4603 Read 5 Done New Done true
12291 3605 Acked 4604 Write 5 Done New Done true
12292 3606 Acked 4605 Write 8 Done New Done true
12293 3607 Acked 4606 Read 9 Done New Done true
12294 3608 Acked 4607 Read 5 Done New Done true
12295 3609 Acked 4608 Flush 0 Done New Done true
12296 3610 Acked 4609 Flush 0 Done New Done true
12297 3611 Acked 4610 Read 9 Done New Done true
12298 3612 Acked 4611 Write 7 Done New Done true
12299 3613 Acked 4612 Read 6 Done New Done true
12300 3614 Acked 4613 Write 1 Done New Done true
12301 3615 Acked 4614 Write 9 Done New Done true
12302 3616 Acked 4615 Flush 0 Done New Done true
12303 STATES DS:0 DS:1 DS:2 TOTAL
12304 New 0 301 0 301
12305 Sent 0 0 0 0
12306 Done 301 0 301 602
12307 Skipped 0 0 0 0
12308 Error 0 0 0 0
12309 Last Flush: 4615 4314 4615
12310 Downstairs last five completed: 4314 4313 4312 4311 4310
12311 Upstairs last five completed: 3616 3615 3614 3613 3612
12312 CLIENT: Up:0 ds:301 act:2
123132024-02-15T02:59:55.371ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
123142024-02-15T02:59:55.371ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
123152024-02-15T02:59:55.371ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
123162024-02-15T02:59:55.371ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
123172024-02-15T02:59:55.371ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
123182024-02-15T02:59:55.371ZINFOcrucible: [1] client re-new 301 jobs since flush 4314 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
123192024-02-15T02:59:55.371ZINFOcrucible: client task is exiting = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
123202024-02-15T02:59:56.372ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
123212024-02-15T02:59:56.372ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
123222024-02-15T02:59:56.372ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
123232024-02-15T02:59:56.372ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
123242024-02-15T02:59:56.372ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
123252024-02-15T02:59:56.372ZINFOcrucible: [1] client re-new 301 jobs since flush 4314 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
123262024-02-15T02:59:56.372ZINFOcrucible: client task is exiting = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
123272024-02-15T02:59:57.374ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
123282024-02-15T02:59:57.374ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
123292024-02-15T02:59:57.374ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
123302024-02-15T02:59:57.374ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
123312024-02-15T02:59:57.374ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
123322024-02-15T02:59:57.374ZINFOcrucible: [1] client re-new 301 jobs since flush 4314 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
123332024-02-15T02:59:57.374ZINFOcrucible: client task is exiting = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
12334 [8820] got start action
12335 [8820] I am going to restart
12336 Starting downstairs at port 8820
12337 Make output file at "/tmp/dsc/downstairs-8820.txt"
12338 [8820][1] reports Starting
12339 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1179)
12340 [8820][1] reports Running
123412024-02-15T02:59:58.374ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
123422024-02-15T02:59:58.374ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
123432024-02-15T02:59:58.374ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
123442024-02-15T02:59:58.374ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
123452024-02-15T02:59:58.374ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
123462024-02-15T02:59:58.374ZINFOcrucible: [1] client re-new 301 jobs since flush 4314 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
123472024-02-15T02:59:58.374ZINFOcrucible: client task is exiting = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
12348 ----------------------------------------------------------------
12349 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
12350 States: Active Offline Active
12351 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
12352 3316 Acked 4315 Read 2 Done New Done true
12353 3317 Acked 4316 Write 9 Done New Done true
12354 3318 Acked 4317 Read 6 Done New Done true
12355 3319 Acked 4318 Write 2 Done New Done true
12356 3320 Acked 4319 Write 8 Done New Done true
12357 3321 Acked 4320 Read 1 Done New Done true
12358 3322 Acked 4321 Read 9 Done New Done true
12359 3323 Acked 4322 Flush 0 Done New Done true
12360 3324 Acked 4323 Read 1 Done New Done true
12361 3325 Acked 4324 Flush 0 Done New Done true
12362 3326 Acked 4325 Flush 0 Done New Done true
12363 3327 Acked 4326 Read 5 Done New Done true
12364 3328 Acked 4327 Read 6 Done New Done true
12365 3329 Acked 4328 Write 5 Done New Done true
12366 3330 Acked 4329 Read 9 Done New Done true
12367 3331 Acked 4330 Read 1 Done New Done true
12368 3332 Acked 4331 Read 2 Done New Done true
12369 3333 Acked 4332 Read 5 Done New Done true
12370 3334 Acked 4333 Flush 0 Done New Done true
12371 3335 Acked 4334 Read 3 Done New Done true
12372 3336 Acked 4335 Read 6 Done New Done true
12373 3337 Acked 4336 Read 9 Done New Done true
12374 3338 Acked 4337 Write 10 Done New Done true
12375 3339 Acked 4338 Read 6 Done New Done true
12376 3340 Acked 4339 Read 7 Done New Done true
12377 3341 Acked 4340 Read 3 Done New Done true
12378 3342 Acked 4341 Read 8 Done New Done true
12379 3343 Acked 4342 Read 1 Done New Done true
12380 3344 Acked 4343 Read 6 Done New Done true
12381 3345 Acked 4344 Read 4 Done New Done true
12382 3346 Acked 4345 Read 4 Done New Done true
12383 3347 Acked 4346 Read 6 Done New Done true
12384 3348 Acked 4347 Write 5 Done New Done true
12385 3349 Acked 4348 Flush 0 Done New Done true
12386 3350 Acked 4349 Write 8 Done New Done true
12387 3351 Acked 4350 Read 5 Done New Done true
12388 3352 Acked 4351 Read 4 Done New Done true
12389 3353 Acked 4352 Read 2 Done New Done true
12390 3354 Acked 4353 Write 2 Done New Done true
12391 3355 Acked 4354 Write 5 Done New Done true
12392 3356 Acked 4355 Write 2 Done New Done true
12393 3357 Acked 4356 Read 2 Done New Done true
12394 3358 Acked 4357 Read 3 Done New Done true
12395 3359 Acked 4358 Read 3 Done New Done true
12396 3360 Acked 4359 Flush 0 Done New Done true
12397 3361 Acked 4360 Write 9 Done New Done true
12398 3362 Acked 4361 Write 3 Done New Done true
12399 3363 Acked 4362 Read 3 Done New Done true
12400 3364 Acked 4363 Read 4 Done New Done true
12401 3365 Acked 4364 Read 4 Done New Done true
12402 3366 Acked 4365 Read 7 Done New Done true
12403 3367 Acked 4366 Flush 0 Done New Done true
12404 3368 Acked 4367 Read 4 Done New Done true
12405 3369 Acked 4368 Write 1 Done New Done true
12406 3370 Acked 4369 Flush 0 Done New Done true
12407 3371 Acked 4370 Flush 0 Done New Done true
12408 3372 Acked 4371 Write 9 Done New Done true
12409 3373 Acked 4372 Write 5 Done New Done true
12410 3374 Acked 4373 Write 8 Done New Done true
12411 3375 Acked 4374 Read 10 Done New Done true
12412 3376 Acked 4375 Write 3 Done New Done true
12413 3377 Acked 4376 Read 10 Done New Done true
12414 3378 Acked 4377 Read 2 Done New Done true
12415 3379 Acked 4378 Read 2 Done New Done true
12416 3380 Acked 4379 Write 2 Done New Done true
12417 3381 Acked 4380 Read 8 Done New Done true
12418 3382 Acked 4381 Write 7 Done New Done true
12419 3383 Acked 4382 Write 4 Done New Done true
12420 3384 Acked 4383 Write 6 Done New Done true
12421 3385 Acked 4384 Read 4 Done New Done true
12422 3386 Acked 4385 Read 2 Done New Done true
12423 3387 Acked 4386 Read 9 Done New Done true
12424 3388 Acked 4387 Read 3 Done New Done true
12425 3389 Acked 4388 Read 1 Done New Done true
12426 3390 Acked 4389 Read 3 Done New Done true
12427 3391 Acked 4390 Flush 0 Done New Done true
12428 3392 Acked 4391 Write 6 Done New Done true
12429 3393 Acked 4392 Read 3 Done New Done true
12430 3394 Acked 4393 Write 9 Done New Done true
12431 3395 Acked 4394 Read 10 Done New Done true
12432 3396 Acked 4395 Read 9 Done New Done true
12433 3397 Acked 4396 Write 8 Done New Done true
12434 3398 Acked 4397 Read 10 Done New Done true
12435 3399 Acked 4398 Write 4 Done New Done true
12436 3400 Acked 4399 Read 3 Done New Done true
12437 3401 Acked 4400 Read 3 Done New Done true
12438 3402 Acked 4401 Write 9 Done New Done true
12439 3403 Acked 4402 Write 4 Done New Done true
12440 3404 Acked 4403 Write 5 Done New Done true
12441 3405 Acked 4404 Read 4 Done New Done true
12442 3406 Acked 4405 Read 8 Done New Done true
12443 3407 Acked 4406 Read 9 Done New Done true
12444 3408 Acked 4407 Read 9 Done New Done true
12445 3409 Acked 4408 Read 1 Done New Done true
12446 3410 Acked 4409 Read 8 Done New Done true
12447 3411 Acked 4410 Flush 0 Done New Done true
12448 3412 Acked 4411 Flush 0 Done New Done true
12449 3413 Acked 4412 Write 4 Done New Done true
12450 3414 Acked 4413 Write 5 Done New Done true
12451 3415 Acked 4414 Read 10 Done New Done true
12452 3416 Acked 4415 Write 1 Done New Done true
12453 3417 Acked 4416 Read 2 Done New Done true
12454 3418 Acked 4417 Flush 0 Done New Done true
12455 3419 Acked 4418 Write 8 Done New Done true
12456 3420 Acked 4419 Read 10 Done New Done true
12457 3421 Acked 4420 Write 6 Done New Done true
12458 3422 Acked 4421 Read 10 Done New Done true
12459 3423 Acked 4422 Read 2 Done New Done true
12460 3424 Acked 4423 Read 7 Done New Done true
12461 3425 Acked 4424 Write 6 Done New Done true
12462 3426 Acked 4425 Read 3 Done New Done true
12463 3427 Acked 4426 Read 2 Done New Done true
12464 3428 Acked 4427 Write 4 Done New Done true
12465 3429 Acked 4428 Read 10 Done New Done true
12466 3430 Acked 4429 Read 5 Done New Done true
12467 3431 Acked 4430 Write 8 Done New Done true
12468 3432 Acked 4431 Read 8 Done New Done true
12469 3433 Acked 4432 Write 7 Done New Done true
12470 3434 Acked 4433 Read 10 Done New Done true
12471 3435 Acked 4434 Read 10 Done New Done true
12472 3436 Acked 4435 Read 2 Done New Done true
12473 3437 Acked 4436 Read 5 Done New Done true
12474 3438 Acked 4437 Read 8 Done New Done true
12475 3439 Acked 4438 Write 10 Done New Done true
12476 3440 Acked 4439 Flush 0 Done New Done true
12477 3441 Acked 4440 Read 7 Done New Done true
12478 3442 Acked 4441 Read 2 Done New Done true
12479 3443 Acked 4442 Write 1 Done New Done true
12480 3444 Acked 4443 Write 5 Done New Done true
12481 3445 Acked 4444 Read 8 Done New Done true
12482 3446 Acked 4445 Read 5 Done New Done true
12483 3447 Acked 4446 Read 6 Done New Done true
12484 3448 Acked 4447 Read 10 Done New Done true
12485 3449 Acked 4448 Read 9 Done New Done true
12486 3450 Acked 4449 Write 1 Done New Done true
12487 3451 Acked 4450 Read 9 Done New Done true
12488 3452 Acked 4451 Read 3 Done New Done true
12489 3453 Acked 4452 Read 6 Done New Done true
12490 3454 Acked 4453 Write 6 Done New Done true
12491 3455 Acked 4454 Write 7 Done New Done true
12492 3456 Acked 4455 Write 3 Done New Done true
12493 3457 Acked 4456 Write 9 Done New Done true
12494 3458 Acked 4457 Write 3 Done New Done true
12495 3459 Acked 4458 Read 4 Done New Done true
12496 3460 Acked 4459 Read 4 Done New Done true
12497 3461 Acked 4460 Write 1 Done New Done true
12498 3462 Acked 4461 Write 10 Done New Done true
12499 3463 Acked 4462 Read 3 Done New Done true
12500 3464 Acked 4463 Write 7 Done New Done true
12501 3465 Acked 4464 Read 1 Done New Done true
12502 3466 Acked 4465 Read 7 Done New Done true
12503 3467 Acked 4466 Read 3 Done New Done true
12504 3468 Acked 4467 Write 7 Done New Done true
12505 3469 Acked 4468 Write 3 Done New Done true
12506 3470 Acked 4469 Read 2 Done New Done true
12507 3471 Acked 4470 Write 4 Done New Done true
12508 3472 Acked 4471 Flush 0 Done New Done true
12509 3473 Acked 4472 Read 7 Done New Done true
12510 3474 Acked 4473 Write 7 Done New Done true
12511 3475 Acked 4474 Read 2 Done New Done true
12512 3476 Acked 4475 Write 5 Done New Done true
12513 3477 Acked 4476 Read 9 Done New Done true
12514 3478 Acked 4477 Read 8 Done New Done true
12515 3479 Acked 4478 Write 2 Done New Done true
12516 3480 Acked 4479 Read 2 Done New Done true
12517 3481 Acked 4480 Read 6 Done New Done true
12518 3482 Acked 4481 Write 7 Done New Done true
12519 3483 Acked 4482 Write 6 Done New Done true
12520 3484 Acked 4483 Flush 0 Done New Done true
12521 3485 Acked 4484 Read 3 Done New Done true
12522 3486 Acked 4485 Read 5 Done New Done true
12523 3487 Acked 4486 Read 5 Done New Done true
12524 3488 Acked 4487 Write 7 Done New Done true
12525 3489 Acked 4488 Write 1 Done New Done true
12526 3490 Acked 4489 Write 9 Done New Done true
12527 3491 Acked 4490 Read 2 Done New Done true
12528 3492 Acked 4491 Write 5 Done New Done true
12529 3493 Acked 4492 Read 4 Done New Done true
12530 3494 Acked 4493 Read 8 Done New Done true
12531 3495 Acked 4494 Write 1 Done New Done true
12532 3496 Acked 4495 Write 5 Done New Done true
12533 3497 Acked 4496 Write 3 Done New Done true
12534 3498 Acked 4497 Write 3 Done New Done true
12535 3499 Acked 4498 Read 6 Done New Done true
12536 3500 Acked 4499 Flush 0 Done New Done true
12537 3501 Acked 4500 Read 2 Done New Done true
12538 3502 Acked 4501 Flush 0 Done New Done true
12539 3503 Acked 4502 Write 6 Done New Done true
12540 3504 Acked 4503 Read 5 Done New Done true
12541 3505 Acked 4504 Write 6 Done New Done true
12542 3506 Acked 4505 Read 4 Done New Done true
12543 3507 Acked 4506 Write 5 Done New Done true
12544 3508 Acked 4507 Flush 0 Done New Done true
12545 3509 Acked 4508 Read 7 Done New Done true
12546 3510 Acked 4509 Read 2 Done New Done true
12547 3511 Acked 4510 Write 8 Done New Done true
12548 3512 Acked 4511 Read 9 Done New Done true
12549 3513 Acked 4512 Read 2 Done New Done true
12550 3514 Acked 4513 Read 1 Done New Done true
12551 3515 Acked 4514 Write 8 Done New Done true
12552 3516 Acked 4515 Read 7 Done New Done true
12553 3517 Acked 4516 Write 4 Done New Done true
12554 3518 Acked 4517 Read 9 Done New Done true
12555 3519 Acked 4518 Read 10 Done New Done true
12556 3520 Acked 4519 Write 9 Done New Done true
12557 3521 Acked 4520 Write 3 Done New Done true
12558 3522 Acked 4521 Write 6 Done New Done true
12559 3523 Acked 4522 Flush 0 Done New Done true
12560 3524 Acked 4523 Write 5 Done New Done true
12561 3525 Acked 4524 Read 5 Done New Done true
12562 3526 Acked 4525 Read 6 Done New Done true
12563 3527 Acked 4526 Read 8 Done New Done true
12564 3528 Acked 4527 Write 10 Done New Done true
12565 3529 Acked 4528 Write 2 Done New Done true
12566 3530 Acked 4529 Write 10 Done New Done true
12567 3531 Acked 4530 Write 4 Done New Done true
12568 3532 Acked 4531 Write 4 Done New Done true
12569 3533 Acked 4532 Read 7 Done New Done true
12570 3534 Acked 4533 Read 8 Done New Done true
12571 3535 Acked 4534 Write 4 Done New Done true
12572 3536 Acked 4535 Flush 0 Done New Done true
12573 3537 Acked 4536 Write 1 Done New Done true
12574 3538 Acked 4537 Read 6 Done New Done true
12575 3539 Acked 4538 Write 6 Done New Done true
12576 3540 Acked 4539 Write 8 Done New Done true
12577 3541 Acked 4540 Read 2 Done New Done true
12578 3542 Acked 4541 Write 1 Done New Done true
12579 3543 Acked 4542 Write 7 Done New Done true
12580 3544 Acked 4543 Read 5 Done New Done true
12581 3545 Acked 4544 Read 3 Done New Done true
12582 3546 Acked 4545 Write 10 Done New Done true
12583 3547 Acked 4546 Read 9 Done New Done true
12584 3548 Acked 4547 Read 9 Done New Done true
12585 3549 Acked 4548 Write 10 Done New Done true
12586 3550 Acked 4549 Read 1 Done New Done true
12587 3551 Acked 4550 Write 4 Done New Done true
12588 3552 Acked 4551 Write 1 Done New Done true
12589 3553 Acked 4552 Read 7 Done New Done true
12590 3554 Acked 4553 Read 4 Done New Done true
12591 3555 Acked 4554 Read 9 Done New Done true
12592 3556 Acked 4555 Read 4 Done New Done true
12593 3557 Acked 4556 Read 6 Done New Done true
12594 3558 Acked 4557 Read 6 Done New Done true
12595 3559 Acked 4558 Write 7 Done New Done true
12596 3560 Acked 4559 Read 4 Done New Done true
12597 3561 Acked 4560 Read 1 Done New Done true
12598 3562 Acked 4561 Flush 0 Done New Done true
12599 3563 Acked 4562 Write 1 Done New Done true
12600 3564 Acked 4563 Write 8 Done New Done true
12601 3565 Acked 4564 Write 4 Done New Done true
12602 3566 Acked 4565 Read 1 Done New Done true
12603 3567 Acked 4566 Write 6 Done New Done true
12604 3568 Acked 4567 Write 3 Done New Done true
12605 3569 Acked 4568 Write 8 Done New Done true
12606 3570 Acked 4569 Read 9 Done New Done true
12607 3571 Acked 4570 Flush 0 Done New Done true
12608 3572 Acked 4571 Read 2 Done New Done true
12609 3573 Acked 4572 Read 8 Done New Done true
12610 3574 Acked 4573 Write 6 Done New Done true
12611 3575 Acked 4574 Read 9 Done New Done true
12612 3576 Acked 4575 Write 4 Done New Done true
12613 3577 Acked 4576 Read 6 Done New Done true
12614 3578 Acked 4577 Write 8 Done New Done true
12615 3579 Acked 4578 Flush 0 Done New Done true
12616 3580 Acked 4579 Write 8 Done New Done true
12617 3581 Acked 4580 Read 5 Done New Done true
12618 3582 Acked 4581 Read 1 Done New Done true
12619 3583 Acked 4582 Write 6 Done New Done true
12620 3584 Acked 4583 Write 1 Done New Done true
12621 3585 Acked 4584 Write 2 Done New Done true
12622 3586 Acked 4585 Write 9 Done New Done true
12623 3587 Acked 4586 Read 9 Done New Done true
12624 3588 Acked 4587 Write 7 Done New Done true
12625 3589 Acked 4588 Read 10 Done New Done true
12626 3590 Acked 4589 Read 1 Done New Done true
12627 3591 Acked 4590 Write 3 Done New Done true
12628 3592 Acked 4591 Read 8 Done New Done true
12629 3593 Acked 4592 Write 1 Done New Done true
12630 3594 Acked 4593 Write 2 Done New Done true
12631 3595 Acked 4594 Read 4 Done New Done true
12632 3596 Acked 4595 Read 7 Done New Done true
12633 3597 Acked 4596 Write 6 Done New Done true
12634 3598 Acked 4597 Read 8 Done New Done true
12635 3599 Acked 4598 Read 2 Done New Done true
12636 3600 Acked 4599 Read 10 Done New Done true
12637 3601 Acked 4600 Flush 0 Done New Done true
12638 3602 Acked 4601 Read 4 Done New Done true
12639 3603 Acked 4602 Read 3 Done New Done true
12640 3604 Acked 4603 Read 5 Done New Done true
12641 3605 Acked 4604 Write 5 Done New Done true
12642 3606 Acked 4605 Write 8 Done New Done true
12643 3607 Acked 4606 Read 9 Done New Done true
12644 3608 Acked 4607 Read 5 Done New Done true
12645 3609 Acked 4608 Flush 0 Done New Done true
12646 3610 Acked 4609 Flush 0 Done New Done true
12647 3611 Acked 4610 Read 9 Done New Done true
12648 3612 Acked 4611 Write 7 Done New Done true
12649 3613 Acked 4612 Read 6 Done New Done true
12650 3614 Acked 4613 Write 1 Done New Done true
12651 3615 Acked 4614 Write 9 Done New Done true
12652 3616 Acked 4615 Flush 0 Done New Done true
12653 STATES DS:0 DS:1 DS:2 TOTAL
12654 New 0 301 0 301
12655 Sent 0 0 0 0
12656 Done 301 0 301 602
12657 Skipped 0 0 0 0
12658 Error 0 0 0 0
12659 Last Flush: 4615 4314 4615
12660 Downstairs last five completed: 4314 4313 4312 4311 4310
12661 Upstairs last five completed: 3616 3615 3614 3613 3612
12662 CLIENT: Up:0 ds:301 act:2
126632024-02-15T02:59:59.375ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
126642024-02-15T02:59:59.375ZINFOcrucible: ds_connection connected = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
126652024-02-15T02:59:59.377ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
126662024-02-15T02:59:59.377ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
126672024-02-15T02:59:59.377ZINFOcrucible: send last flush ID to this DS: 4314 = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
126682024-02-15T02:59:59.377ZINFOcrucible: Replied this last flush ID: 4314 = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
126692024-02-15T02:59:59.377ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
126702024-02-15T02:59:59.377ZINFOcrucible: [1] Transition from Offline to Replay = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
126712024-02-15T02:59:59.377ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
126722024-02-15T02:59:59.378ZINFOcrucible: [1] Transition from Replay to Active = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
12673 ----------------------------------------------------------------
12674 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
12675 Downstairs last five completed: 4615 4614 4613 4612 4611
12676 Upstairs last five completed: 3616 3615 3614 3613 3612
12677 CLIENT: Up:0 ds:0 act:3
12678 Replay: All jobs finished, all DS active.
12679 Main task has work to do, go find it
12680 got dsc Stop(2)
12681 stop 2
12682 Feb 15 03:00:02.698 INFO request completed, latency_us: 317, response_code: 204, uri: /stop/cid/2, method: [8830] Got stop action so:false kr:false
12683 POST, req_id: 7dc46b2f-f306-48fb-8077-e9c4fdfdce4e, remote_addr: 127.0.0.1:54744, local_addr: 127.0.0.1:9998
126842024-02-15T03:00:02.699ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
126852024-02-15T03:00:02.699ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
126862024-02-15T03:00:02.699ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
126872024-02-15T03:00:02.699ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
12688 Feb 15 03:00:02.699 INFO request completed, latency_us: {"msg":"[2] client re-new 0 jobs since flush 4615","v":0,"name":"crucible","level":30,"time":"2024-02-15T03:00:02.69972285Z","hostname":"ip-10-150-1-67.us-west-2.compute.internal","pid":1167,"":"downstairs","":"downstairs","session_id":"4f5a63e5-aa00-4890-9f6b-96c2e069fdb2"}
12689 190, response_code: 200, uri: /state/cid/2, method: GET, req_id: d1e6da02-5324-4521-9776-8561a5b9f979, remote_addr: {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-15T03:00:02.699739308Z","hostname":"ip-10-150-1-67.us-west-2.compute.internal","pid":1167,"":"io task","client":"2","":"downstairs","session_id":"4f5a63e5-aa00-4890-9f6b-96c2e069fdb2"}
12690 127.0.0.1:54744, local_addr: 127.0.0.1:9998
12691 [8830] Exited with: ExitStatus(unix_wait_status(9))
12692 [8830][2] reports Exit
126932024-02-15T03:00:03.701ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
126942024-02-15T03:00:03.701ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
126952024-02-15T03:00:03.701ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
126962024-02-15T03:00:03.701ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
126972024-02-15T03:00:03.701ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
126982024-02-15T03:00:03.701ZINFOcrucible: [2] client re-new 0 jobs since flush 4615 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
126992024-02-15T03:00:03.701ZINFOcrucible: client task is exiting = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
127002024-02-15T03:00:04.702ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
127012024-02-15T03:00:04.702ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
127022024-02-15T03:00:04.702ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
127032024-02-15T03:00:04.703ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
127042024-02-15T03:00:04.703ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
127052024-02-15T03:00:04.703ZINFOcrucible: [2] client re-new 0 jobs since flush 4615 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
127062024-02-15T03:00:04.703ZINFOcrucible: client task is exiting = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
127072024-02-15T03:00:05.704ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
127082024-02-15T03:00:05.704ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
127092024-02-15T03:00:05.704ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
127102024-02-15T03:00:05.704ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
127112024-02-15T03:00:05.704ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
127122024-02-15T03:00:05.704ZINFOcrucible: [2] client re-new 0 jobs since flush 4615 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
127132024-02-15T03:00:05.704ZINFOcrucible: client task is exiting = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
12714 Feb 15 03:00:06.701 INFO request completed, latency_us: 197, response_code: 200, uri: /state/cid/2, method: GET, req_id: 078fa678-82df-44db-ae4e-b74139adea33, remote_addr: 127.0.0.1:54744, local_addr: 127.0.0.1:9998
12715 001/300 Write block 2828 len 8192 data: 4 2
12716 002/300 Read block 1476 len 28672
12717 003/300 Read block 313 len 28672
127182024-02-15T03:00:06.704ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
127192024-02-15T03:00:06.704ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
127202024-02-15T03:00:06.704ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
127212024-02-15T03:00:06.704ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
127222024-02-15T03:00:06.704ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
127232024-02-15T03:00:06.704ZINFOcrucible: [2] client re-new 3 jobs since flush 4615 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
127242024-02-15T03:00:06.704ZINFOcrucible: client task is exiting = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
12725 004/300 Write block 1411 len 32768 data: 5 3 3 2 3 3 3 4
12726 005/300 Write block 4012 len 24576 data: 4 4 4 5 4 4
12727 006/300 Write block 1295 len 24576 data: 2 2 2 2 2 2
12728 007/300 Flush
12729 008/300 Read block 4898 len 28672
12730 009/300 Write block 2325 len 36864 data: 3 3 3 3 3 3 2 2 2
12731 010/300 Write block 885 len 28672 data: 3 3 4 4 4 4 4
12732 011/300 Read block 1202 len 20480
12733 012/300 Read block 4588 len 12288
12734 013/300 Read block 659 len 4096
12735 014/300 Write block 1304 len 24576 data: 4 4 3 3 3 4
12736 015/300 Flush
12737 016/300 Read block 4702 len 12288
12738 017/300 Read block 1309 len 20480
12739 018/300 Read block 10 len 12288
12740 019/300 Write block 1024 len 40960 data: 3 2 3 3 3 3 3 3 3 3
12741 020/300 Read block 1801 len 32768
12742 021/300 Flush
12743 022/300 Read block 271 len 36864
12744 023/300 Flush
12745 024/300 Read block 673 len 28672
12746 025/300 Read block 3667 len 32768
12747 026/300 Read block 178 len 8192
12748 027/300 Read block 1492 len 8192
12749 028/300 Write block 334 len 16384 data: 2 2 2 2
12750 029/300 Flush
12751 030/300 Write block 737 len 40960 data: 5 5 6 4 4 3 3 4 4 4
12752 031/300 Write block 3047 len 20480 data: 3 4 5 4 5
12753 032/300 Write block 841 len 28672 data: 4 4 4 5 4 4 5
12754 033/300 Read block 2269 len 12288
12755 034/300 Write block 4100 len 28672 data: 2 2 2 2 2 3 3
12756 035/300 Read block 3154 len 16384
12757 036/300 Read block 2490 len 28672
12758 037/300 Read block 4426 len 12288
12759 038/300 Write block 3906 len 12288 data: 4 5 4
12760 039/300 Write block 4091 len 40960 data: 4 4 4 4 3 2 2 2 2 3
12761 040/300 Read block 2485 len 16384
12762 041/300 Read block 3475 len 36864
12763 042/300 Read block 3047 len 12288
12764 043/300 Read block 3213 len 12288
12765 044/300 Read block 1990 len 16384
12766 045/300 Write block 589 len 4096 data: 4
12767 046/300 Read block 4507 len 24576
12768 047/300 Write block 3094 len 32768 data: 2 3 3 3 3 3 3 3
12769 048/300 Write block 957 len 4096 data: 6
12770 049/300 Flush
12771 050/300 Write block 3963 len 16384 data: 3 3 3 3
12772 051/300 Read block 4654 len 16384
12773 052/300 Read block 508 len 8192
12774 053/300 Write block 2768 len 28672 data: 2 2 2 2 2 2 2
12775 054/300 Read block 455 len 20480
12776 055/300 Write block 4497 len 4096 data: 3
12777 056/300 Write block 2275 len 28672 data: 2 3 4 4 4 2 2
12778 057/300 Write block 3137 len 32768 data: 5 4 4 4 4 5 4 4
12779 058/300 Flush
12780 059/300 Read block 2863 len 40960
12781 060/300 Write block 564 len 28672 data: 2 3 5 4 4 4 5
12782 061/300 Flush
12783 062/300 Flush
12784 063/300 Write block 2731 len 28672 data: 3 3 3 3 3 2 2
12785 064/300 Read block 1258 len 20480
12786 065/300 Write block 2335 len 4096 data: 2
12787 066/300 Write block 4796 len 24576 data: 4 3 3 3 3 3
12788 067/300 Write block 2352 len 28672 data: 4 4 5 4 4 5 5
12789 068/300 Write block 848 len 20480 data: 4 4 3 4 4
12790 069/300 Read block 4777 len 40960
12791 070/300 Read block 3934 len 36864
12792 071/300 Flush
12793 072/300 Write block 3767 len 4096 data: 4
12794 073/300 Read block 2820 len 20480
12795 074/300 Write block 4166 len 8192 data: 3 4
12796 075/300 Write block 2096 len 8192 data: 4 4
12797 076/300 Write block 1583 len 8192 data: 6 6
12798 077/300 Read block 2192 len 16384
12799 078/300 Write block 3280 len 12288 data: 2 2 2
12800 079/300 Read block 4488 len 28672
12801 080/300 Write block 1723 len 16384 data: 3 3 3 3
12802 081/300 Read block 1860 len 40960
12803 082/300 Write block 4143 len 24576 data: 8 8 7 4 3 3
12804 083/300 Write block 472 len 40960 data: 2 2 2 2 2 3 4 2 2 3
12805 084/300 Read block 853 len 20480
12806 085/300 Flush
12807 086/300 Write block 321 len 40960 data: 3 3 3 3 3 3 3 3 3 3
12808 087/300 Write block 2638 len 40960 data: 4 4 5 4 5 5 5 5 5 5
12809 088/300 Read block 4481 len 8192
12810 089/300 Read block 2015 len 16384
12811 090/300 Read block 1310 len 24576
12812 091/300 Read block 3265 len 40960
12813 092/300 Read block 1783 len 36864
12814 093/300 Write block 200 len 16384 data: 4 4 4 3
12815 094/300 Flush
12816 095/300 Write block 2712 len 40960 data: 3 3 2 2 2 2 2 2 3 3
12817 096/300 Write block 4910 len 12288 data: 5 5 4
12818 097/300 Write block 2418 len 40960 data: 3 4 3 4 4 4 4 4 4 4
12819 098/300 Read block 1476 len 40960
12820 099/300 Read block 2586 len 36864
12821 100/300 Flush
12822 101/300 Read block 4867 len 24576
12823 102/300 Read block 1354 len 8192
12824 103/300 Read block 625 len 16384
12825 104/300 Read block 593 len 24576
12826 105/300 Read block 2612 len 4096
12827 106/300 Read block 3809 len 4096
12828 107/300 Write block 2391 len 16384 data: 3 3 3 3
12829 108/300 Read block 2442 len 36864
12830 109/300 Read block 2085 len 28672
12831 110/300 Read block 768 len 32768
12832 111/300 Write block 1194 len 16384 data: 5 5 4 5
12833 112/300 Read block 4247 len 40960
12834 113/300 Read block 1837 len 28672
12835 114/300 Read block 1328 len 8192
12836 115/300 Read block 2579 len 24576
12837 116/300 Read block 2153 len 20480
12838 117/300 Flush
12839 118/300 Write block 2185 len 12288 data: 3 4 4
12840 119/300 Read block 4808 len 24576
12841 120/300 Read block 3999 len 36864
12842 121/300 Read block 4685 len 4096
12843 122/300 Read block 200 len 24576
12844 123/300 Read block 1545 len 32768
12845 124/300 Write block 3281 len 4096 data: 3
12846 125/300 Write block 650 len 4096 data: 3
12847 126/300 Read block 4032 len 32768
12848 127/300 Write block 747 len 32768 data: 4 4 5 5 5 5 4 4
12849 128/300 Read block 1986 len 8192
12850 129/300 Read block 4594 len 32768
12851 130/300 Write block 2189 len 12288 data: 4 4 4
12852 131/300 Read block 4335 len 8192
12853 132/300 Write block 2619 len 16384 data: 4 5 3 3
12854 133/300 Read block 136 len 36864
12855 134/300 Read block 132 len 36864
12856 135/300 Read block 282 len 32768
12857 136/300 Write block 3366 len 12288 data: 2 2 2
12858 137/300 Write block 3957 len 24576 data: 5 3 3 3 3 3
12859 138/300 Read block 2558 len 4096
12860 139/300 Read block 3671 len 40960
12861 140/300 Read block 2929 len 36864
12862 141/300 Write block 1638 len 12288 data: 2 4 4
12863 142/300 Flush
12864 143/300 Read block 2997 len 24576
12865 144/300 Write block 1992 len 28672 data: 3 3 3 4 3 3 4
12866 145/300 Read block 1016 len 36864
12867 146/300 Read block 3268 len 36864
12868 147/300 Read block 408 len 36864
12869 148/300 Write block 1378 len 12288 data: 2 3 3
12870 149/300 Write block 2756 len 12288 data: 2 3 3
12871 150/300 Write block 553 len 36864 data: 3 4 4 3 3 4 4 3 3
12872 151/300 Flush
12873 152/300 Write block 4062 len 16384 data: 5 4 4 4
12874 153/300 Flush
12875 154/300 Write block 2622 len 36864 data: 4 3 3 3 3 3 3 3 3
12876 155/300 Flush
12877 156/300 Read block 4776 len 32768
12878 157/300 Read block 1263 len 20480
12879 158/300 Write block 3541 len 4096 data: 3
12880 159/300 Read block 2690 len 36864
12881 160/300 Read block 32 len 20480
12882 161/300 Read block 4044 len 4096
12883 162/300 Read block 2728 len 16384
12884 163/300 Write block 3631 len 16384 data: 4 3 2 2
12885 164/300 Write block 4963 len 4096 data: 3
12886 165/300 Read block 1966 len 24576
12887 166/300 Write block 4419 len 24576 data: 5 5 4 4 5 6
12888 167/300 Read block 2069 len 24576
12889 168/300 Read block 4848 len 36864
12890 169/300 Read block 1400 len 12288
12891 170/300 Read block 407 len 12288
12892 171/300 Read block 425 len 36864
12893 172/300 Write block 4026 len 4096 data: 5
12894 173/300 Read block 2097 len 32768
12895 174/300 Read block 2471 len 4096
12896 175/300 Write block 3357 len 32768 data: 3 3 3 3 3 4 3 3
12897 176/300 Read block 3740 len 12288
12898 177/300 Read block 1581 len 20480
12899 178/300 Read block 1396 len 24576
12900 179/300 Write block 191 len 4096 data: 3
12901 180/300 Read block 3016 len 36864
12902 181/300 Write block 1199 len 24576 data: 5 4 4 3 3 3
12903 182/300 Write block 4347 len 36864 data: 5 5 3 3 3 2 2 2 2
12904 183/300 Write block 2947 len 16384 data: 2 3 3 3
12905 184/300 Read block 1441 len 24576
12906 185/300 Write block 1216 len 28672 data: 2 2 3 4 4 5 4
12907 186/300 Write block 3560 len 12288 data: 4 5 5
12908 187/300 Read block 2443 len 36864
12909 188/300 Write block 2599 len 16384 data: 3 3 3 3
12910 189/300 Read block 1569 len 4096
12911 190/300 Write block 17 len 28672 data: 4 4 3 4 6 6 7
12912 191/300 Read block 1903 len 32768
12913 192/300 Read block 2211 len 8192
12914 193/300 Read block 3923 len 36864
12915 194/300 Read block 686 len 4096
12916 195/300 Read block 2134 len 32768
12917 196/300 Write block 3256 len 36864 data: 3 3 3 3 3 2 3 4 3
12918 197/300 Read block 4662 len 4096
12919 198/300 Write block 3780 len 32768 data: 4 5 4 5 4 4 4 4
12920 199/300 Write block 3534 len 16384 data: 6 5 4 4
12921 200/300 Read block 924 len 12288
12922 201/300 Read block 2750 len 20480
12923 202/300 Flush
12924 203/300 Write block 1760 len 4096 data: 3
12925 204/300 Read block 153 len 12288
12926 205/300 Flush
12927 206/300 Read block 3989 len 28672
12928 207/300 Read block 3029 len 12288
12929 208/300 Write block 4556 len 4096 data: 2
12930 209/300 Read block 4832 len 8192
12931 210/300 Read block 4117 len 24576
12932 211/300 Write block 2839 len 40960 data: 5 5 5 4 2 2 2 2 2 2
12933 212/300 Write block 1470 len 24576 data: 3 3 3 3 2 2
12934 213/300 Write block 219 len 12288 data: 3 3 3
12935 214/300 Read block 3051 len 12288
12936 215/300 Write block 4371 len 16384 data: 3 2 3 3
12937 216/300 Write block 3495 len 12288 data: 5 5 5
12938 217/300 Flush
12939 218/300 Read block 1558 len 8192
12940 219/300 Write block 4068 len 32768 data: 4 3 3 3 3 3 2 2
12941 220/300 Read block 2034 len 28672
12942 221/300 Read block 850 len 24576
12943 222/300 Write block 4501 len 8192 data: 3 4
12944 223/300 Read block 2818 len 16384
12945 224/300 Read block 1370 len 4096
12946 225/300 Write block 622 len 24576 data: 2 2 2 2 3 3
12947 226/300 Read block 504 len 8192
12948 227/300 Write block 3635 len 12288 data: 3 2 2
12949 228/300 Write block 1697 len 36864 data: 3 3 3 2 3 2 2 2 3
12950 229/300 Flush
12951 230/300 Read block 4938 len 8192
12952 231/300 Read block 3736 len 28672
12953 232/300 Read block 2424 len 16384
12954 233/300 Read block 3514 len 8192
12955 234/300 Write block 574 len 40960 data: 4 3 3 3 4 5 5 5 4 4
12956 235/300 Write block 4557 len 24576 data: 2 2 2 2 2 2
12957 236/300 Read block 924 len 32768
12958 237/300 Read block 454 len 28672
12959 238/300 Write block 755 len 36864 data: 5 5 5 3 3 3 3 2 2
12960 239/300 Read block 1338 len 8192
12961 240/300 Write block 2874 len 8192 data: 2 2
12962 241/300 Write block 231 len 8192 data: 5 4
12963 242/300 Write block 211 len 8192 data: 3 3
12964 243/300 Write block 3604 len 28672 data: 2 2 2 3 3 3 4
12965 244/300 Flush
12966 245/300 Write block 4237 len 8192 data: 5 6
12967 246/300 Write block 1135 len 16384 data: 4 4 4 4
12968 247/300 Write block 4738 len 4096 data: 5
12969 248/300 Write block 1648 len 20480 data: 5 6 4 3 3
12970 249/300 Read block 3011 len 20480
12971 250/300 Read block 1823 len 20480
12972 251/300 Read block 996 len 20480
12973 252/300 Read block 4320 len 24576
12974 253/300 Read block 3664 len 36864
12975 254/300 Write block 3025 len 24576 data: 3 3 4 5 5 4
12976 255/300 Read block 54 len 40960
12977 256/300 Write block 383 len 32768 data: 4 4 3 3 3 3 4 3
12978 257/300 Read block 4661 len 24576
12979 258/300 Write block 3151 len 20480 data: 5 5 5 6 6
12980 259/300 Flush
12981 260/300 Read block 3050 len 28672
12982 261/300 Read block 1116 len 16384
12983 262/300 Read block 1849 len 8192
12984 263/300 Read block 2902 len 40960
12985 264/300 Flush
12986 265/300 Flush
12987 266/300 Write block 3524 len 40960 data: 4 3 4 4 4 5 5 4 4 6
12988 267/300 Flush
12989 268/300 Read block 3394 len 24576
12990 269/300 Write block 1841 len 28672 data: 3 3 3 3 3 5 4
12991 270/300 Read block 1464 len 24576
12992 271/300 Read block 1008 len 32768
12993 272/300 Write block 2269 len 24576 data: 2 3 3 2 2 2
12994 273/300 Write block 3213 len 32768 data: 5 4 4 4 4 3 2 2
12995 274/300 Write block 4505 len 20480 data: 4 4 4 4 4
12996 275/300 Flush
12997 276/300 Read block 4248 len 8192
12998 277/300 Write block 3504 len 16384 data: 4 4 4 4
12999 278/300 Write block 1660 len 16384 data: 3 2 2 2
13000 279/300 Write block 3582 len 28672 data: 2 2 4 4 4 3 3
13001 280/300 Read block 3907 len 28672
13002 281/300 Read block 1371 len 4096
13003 282/300 Write block 3105 len 32768 data: 4 4 3 4 5 5 6 5
13004 283/300 Write block 3863 len 32768 data: 5 5 5 5 5 7 6 7
13005 284/300 Read block 3867 len 24576
13006 285/300 Read block 1306 len 24576
13007 286/300 Read block 4106 len 4096
13008 287/300 Read block 2429 len 36864
13009 288/300 Read block 1957 len 4096
13010 289/300 Write block 4033 len 4096 data: 4
13011 290/300 Flush
13012 291/300 Write block 4543 len 40960 data: 2 2 2 3 4 5 6 5 4 4
13013 292/300 Read block 3056 len 24576
13014 293/300 Write block 2838 len 32768 data: 5 6 6 6 5 3 3 3
13015 294/300 Read block 4502 len 28672
13016 295/300 Flush
13017 296/300 Write block 4559 len 28672 data: 3 3 3 3 2 3 3
13018 297/300 Write block 2664 len 12288 data: 4 4 4
13019 298/300 Write block 3399 len 24576 data: 5 5 5 6 6 4
13020 299/300 Flush
13021 300/300 Write block 4028 len 16384 data: 6 6 5 4
13022 Main task has work to do, go find it
13023 got dsc Start(2)
13024 start 2
13025 Feb 15 03:00:07.032 INFO request completed, latency_us: 199, response_code: 204, uri: /start/cid/2, method: POST, req_id: b9b311fd-4995-4f6c-a3c3-41c0adf1338c, remote_addr: 127.0.0.1:54744, local_addr: 127.0.0.1:9998
13026 Replay: started 2, returned:Ok(())
13027 ----------------------------------------------------------------
13028 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
13029 States: Active Active Offline
13030 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
13031 3617 Acked 4616 Write 2 Done Done New true
13032 3618 Acked 4617 Read 7 Done Done New true
13033 3619 Acked 4618 Read 7 Done Done New true
13034 3620 Acked 4619 Write 8 Done Done New false
13035 3621 Acked 4620 Write 6 Done Done New false
13036 3622 Acked 4621 Write 6 Done Done New false
13037 3623 Acked 4622 Flush 0 Done Done New false
13038 3624 Acked 4623 Read 7 Done Done New false
13039 3625 Acked 4624 Write 9 Done Done New false
13040 3626 Acked 4625 Write 7 Done Done New false
13041 3627 Acked 4626 Read 5 Done Done New false
13042 3628 Acked 4627 Read 3 Done Done New false
13043 3629 Acked 4628 Read 1 Done Done New false
13044 3630 Acked 4629 Write 6 Done Done New false
13045 3631 Acked 4630 Flush 0 Done Done New false
13046 3632 Acked 4631 Read 3 Done Done New false
13047 3633 Acked 4632 Read 5 Done Done New false
13048 3634 Acked 4633 Read 3 Done Done New false
13049 3635 Acked 4634 Write 10 Done Done New false
13050 3636 Acked 4635 Read 8 Done Done New false
13051 3637 Acked 4636 Flush 0 Done Done New false
13052 3638 Acked 4637 Read 9 Done Done New false
13053 3639 Acked 4638 Flush 0 Done Done New false
13054 3640 Acked 4639 Read 7 Done Done New false
13055 3641 Acked 4640 Read 8 Done Done New false
13056 3642 Acked 4641 Read 2 Done Done New false
13057 3643 Acked 4642 Read 2 Done Done New false
13058 3644 Acked 4643 Write 4 Done Done New false
13059 3645 Acked 4644 Flush 0 Done Done New false
13060 3646 Acked 4645 Write 10 Done Done New false
13061 3647 Acked 4646 Write 5 Done Done New false
13062 3648 Acked 4647 Write 7 Done Done New false
13063 3649 Acked 4648 Read 3 Done Done New false
13064 3650 Acked 4649 Write 7 Done Done New false
13065 3651 Acked 4650 Read 4 Done Done New false
13066 3652 Acked 4651 Read 7 Done Done New false
13067 3653 Acked 4652 Read 3 Done Done New false
13068 3654 Acked 4653 Write 3 Done Done New false
13069 3655 Acked 4654 Write 10 Done Done New false
13070 3656 Acked 4655 Read 4 Done Done New false
13071 3657 Acked 4656 Read 9 Done Done New false
13072 3658 Acked 4657 Read 3 Done Done New false
13073 3659 Acked 4658 Read 3 Done Done New false
13074 3660 Acked 4659 Read 4 Done Done New false
13075 3661 Acked 4660 Write 1 Done Done New false
13076 3662 Acked 4661 Read 6 Done Done New false
13077 3663 Acked 4662 Write 8 Done Done New false
13078 3664 Acked 4663 Write 1 Done Done New false
13079 3665 Acked 4664 Flush 0 Done Done New false
13080 3666 Acked 4665 Write 4 Done Done New false
13081 3667 Acked 4666 Read 4 Done Done New false
13082 3668 Acked 4667 Read 2 Done Done New false
13083 3669 Acked 4668 Write 7 Done Done New false
13084 3670 Acked 4669 Read 5 Done Done New false
13085 3671 Acked 4670 Write 1 Done Done New false
13086 3672 Acked 4671 Write 7 Done Done New false
13087 3673 Acked 4672 Write 8 Done Done New false
13088 3674 Acked 4673 Flush 0 Done Done New false
13089 3675 Acked 4674 Read 10 Done Done New false
13090 3676 Acked 4675 Write 7 Done Done New false
13091 3677 Acked 4676 Flush 0 Done Done New false
13092 3678 Acked 4677 Flush 0 Done Done New false
13093 3679 Acked 4678 Write 7 Done Done New false
13094 3680 Acked 4679 Read 5 Done Done New false
13095 3681 Acked 4680 Write 1 Done Done New false
13096 3682 Acked 4681 Write 6 Done Done New false
13097 3683 Acked 4682 Write 7 Done Done New false
13098 3684 Acked 4683 Write 5 Done Done New false
13099 3685 Acked 4684 Read 10 Done Done New false
13100 3686 Acked 4685 Read 9 Done Done New false
13101 3687 Acked 4686 Flush 0 Done Done New false
13102 3688 Acked 4687 Write 1 Done Done New false
13103 3689 Acked 4688 Read 5 Done Done New false
13104 3690 Acked 4689 Write 2 Done Done New false
13105 3691 Acked 4690 Write 2 Done Done New false
13106 3692 Acked 4691 Write 2 Done Done New false
13107 3693 Acked 4692 Read 4 Done Done New false
13108 3694 Acked 4693 Write 3 Done Done New false
13109 3695 Acked 4694 Read 7 Done Done New false
13110 3696 Acked 4695 Write 4 Done Done New false
13111 3697 Acked 4696 Read 10 Done Done New false
13112 3698 Acked 4697 Write 6 Done Done New false
13113 3699 Acked 4698 Write 10 Done Done New false
13114 3700 Acked 4699 Read 5 Done Done New false
13115 3701 Acked 4700 Flush 0 Done Done New false
13116 3702 Acked 4701 Write 10 Done Done New false
13117 3703 Acked 4702 Write 10 Done Done New false
13118 3704 Acked 4703 Read 2 Done Done New false
13119 3705 Acked 4704 Read 4 Done Done New false
13120 3706 Acked 4705 Read 6 Done Done New false
13121 3707 Acked 4706 Read 10 Done Done New false
13122 3708 Acked 4707 Read 9 Done Done New false
13123 3709 Acked 4708 Write 4 Done Done New false
13124 3710 Acked 4709 Flush 0 Done Done New false
13125 3711 Acked 4710 Write 10 Done Done New false
13126 3712 Acked 4711 Write 3 Done Done New false
13127 3713 Acked 4712 Write 10 Done Done New false
13128 3714 Acked 4713 Read 10 Done Done New false
13129 3715 Acked 4714 Read 9 Done Done New false
13130 3716 Acked 4715 Flush 0 Done Done New false
13131 3717 Acked 4716 Read 6 Done Done New false
13132 3718 Acked 4717 Read 2 Done Done New false
13133 3719 Acked 4718 Read 4 Done Done New false
13134 3720 Acked 4719 Read 6 Done Done New false
13135 3721 Acked 4720 Read 1 Done Done New false
13136 3722 Acked 4721 Read 1 Done Done New false
13137 3723 Acked 4722 Write 4 Done Done New false
13138 3724 Acked 4723 Read 9 Done Done New false
13139 3725 Acked 4724 Read 7 Done Done New false
13140 3726 Acked 4725 Read 8 Done Done New false
13141 3727 Acked 4726 Write 4 Done Done New false
13142 3728 Acked 4727 Read 10 Done Done New false
13143 3729 Acked 4728 Read 7 Done Done New false
13144 3730 Acked 4729 Read 2 Done Done New false
13145 3731 Acked 4730 Read 6 Done Done New false
13146 3732 Acked 4731 Read 5 Done Done New false
13147 3733 Acked 4732 Flush 0 Done Done New false
13148 3734 Acked 4733 Write 3 Done Done New false
13149 3735 Acked 4734 Read 6 Done Done New false
13150 3736 Acked 4735 Read 9 Done Done New false
13151 3737 Acked 4736 Read 1 Done Done New false
13152 3738 Acked 4737 Read 6 Done Done New false
13153 3739 Acked 4738 Read 8 Done Done New false
13154 3740 Acked 4739 Write 1 Done Done New false
13155 3741 Acked 4740 Write 1 Done Done New false
13156 3742 Acked 4741 Read 8 Done Done New false
13157 3743 Acked 4742 Write 8 Done Done New false
13158 3744 Acked 4743 Read 2 Done Done New false
13159 3745 Acked 4744 Read 8 Done Done New false
13160 3746 Acked 4745 Write 3 Done Done New false
13161 3747 Acked 4746 Read 2 Done Done New false
13162 3748 Acked 4747 Write 4 Done Done New false
13163 3749 Acked 4748 Read 9 Done Done New false
13164 3750 Acked 4749 Read 9 Done Done New false
13165 3751 Acked 4750 Read 8 Done Done New false
13166 3752 Acked 4751 Write 3 Done Done New false
13167 3753 Acked 4752 Write 6 Done Done New false
13168 3754 Acked 4753 Read 1 Done Done New false
13169 3755 Acked 4754 Read 10 Done Done New false
13170 3756 Acked 4755 Flush 0 Done Done New false
13171 3757 Acked 4756 Read 9 Done Done New false
13172 3758 Acked 4757 Write 3 Done Done New false
13173 3759 Acked 4758 Flush 0 Done Done New false
13174 3760 Acked 4759 Read 6 Done Done New false
13175 3761 Acked 4760 Write 7 Done Done New false
13176 3762 Acked 4761 Read 9 Done Done New false
13177 3763 Acked 4762 Read 9 Done Done New false
13178 3764 Acked 4763 Read 9 Done Done New false
13179 3765 Acked 4764 Write 3 Done Done New false
13180 3766 Acked 4765 Write 3 Done Done New false
13181 3767 Acked 4766 Write 9 Done Done New false
13182 3768 Acked 4767 Flush 0 Done Done New false
13183 3769 Acked 4768 Write 4 Done Done New false
13184 3770 Acked 4769 Flush 0 Done Done New false
13185 3771 Acked 4770 Write 9 Done Done New false
13186 3772 Acked 4771 Flush 0 Done Done New false
13187 3773 Acked 4772 Read 8 Done Done New false
13188 3774 Acked 4773 Read 5 Done Done New false
13189 3775 Acked 4774 Write 1 Done Done New false
13190 3776 Acked 4775 Read 9 Done Done New false
13191 3777 Acked 4776 Read 5 Done Done New false
13192 3778 Acked 4777 Read 1 Done Done New false
13193 3779 Acked 4778 Read 4 Done Done New false
13194 3780 Acked 4779 Write 4 Done Done New false
13195 3781 Acked 4780 Write 1 Done Done New false
13196 3782 Acked 4781 Read 6 Done Done New false
13197 3783 Acked 4782 Write 6 Done Done New false
13198 3784 Acked 4783 Read 6 Done Done New false
13199 3785 Acked 4784 Read 9 Done Done New false
13200 3786 Acked 4785 Read 3 Done Done New false
13201 3787 Acked 4786 Read 3 Done Done New false
13202 3788 Acked 4787 Read 9 Done Done New false
13203 3789 Acked 4788 Write 1 Done Done New false
13204 3790 Acked 4789 Read 8 Done Done New false
13205 3791 Acked 4790 Read 1 Done Done New false
13206 3792 Acked 4791 Write 8 Done Done New false
13207 3793 Acked 4792 Read 3 Done Done New false
13208 3794 Acked 4793 Read 5 Done Done New false
13209 3795 Acked 4794 Read 6 Done Done New false
13210 3796 Acked 4795 Write 1 Done Done New false
13211 3797 Acked 4796 Read 9 Done Done New false
13212 3798 Acked 4797 Write 6 Done Done New false
13213 3799 Acked 4798 Write 9 Done Done New false
13214 3800 Acked 4799 Write 4 Done Done New false
13215 3801 Acked 4800 Read 6 Done Done New false
13216 3802 Acked 4801 Write 7 Done Done New false
13217 3803 Acked 4802 Write 3 Done Done New false
13218 3804 Acked 4803 Read 9 Done Done New false
13219 3805 Acked 4804 Write 4 Done Done New false
13220 3806 Acked 4805 Read 1 Done Done New false
13221 3807 Acked 4806 Write 7 Done Done New false
13222 3808 Acked 4807 Read 8 Done Done New false
13223 3809 Acked 4808 Read 2 Done Done New false
13224 3810 Acked 4809 Read 9 Done Done New false
13225 3811 Acked 4810 Read 1 Done Done New false
13226 3812 Acked 4811 Read 8 Done Done New false
13227 3813 Acked 4812 Write 9 Done Done New false
13228 3814 Acked 4813 Read 1 Done Done New false
13229 3815 Acked 4814 Write 8 Done Done New false
13230 3816 Acked 4815 Write 4 Done Done New false
13231 3817 Acked 4816 Read 3 Done Done New false
13232 3818 Acked 4817 Read 5 Done Done New false
13233 3819 Acked 4818 Flush 0 Done Done New false
13234 3820 Acked 4819 Write 1 Done Done New false
13235 3821 Acked 4820 Read 3 Done Done New false
13236 3822 Acked 4821 Flush 0 Done Done New false
13237 3823 Acked 4822 Read 7 Done Done New false
13238 3824 Acked 4823 Read 3 Done Done New false
13239 3825 Acked 4824 Write 1 Done Done New false
13240 3826 Acked 4825 Read 2 Done Done New false
13241 3827 Acked 4826 Read 6 Done Done New false
13242 3828 Acked 4827 Write 10 Done Done New false
13243 3829 Acked 4828 Write 6 Done Done New false
13244 3830 Acked 4829 Write 3 Done Done New false
13245 3831 Acked 4830 Read 3 Done Done New false
13246 3832 Acked 4831 Write 4 Done Done New false
13247 3833 Acked 4832 Write 3 Done Done New false
13248 3834 Acked 4833 Flush 0 Done Done New false
13249 3835 Acked 4834 Read 2 Done Done New false
13250 3836 Acked 4835 Write 8 Done Done New false
13251 3837 Acked 4836 Read 7 Done Done New false
13252 3838 Acked 4837 Read 6 Done Done New false
13253 3839 Acked 4838 Write 2 Done Done New false
13254 3840 Acked 4839 Read 4 Done Done New false
13255 3841 Acked 4840 Read 1 Done Done New false
13256 3842 Acked 4841 Write 6 Done Done New false
13257 3843 Acked 4842 Read 2 Done Done New false
13258 3844 Acked 4843 Write 3 Done Done New false
13259 3845 Acked 4844 Write 9 Done Done New false
13260 3846 Acked 4845 Flush 0 Done Done New false
13261 3847 Acked 4846 Read 2 Done Done New false
13262 3848 Acked 4847 Read 7 Done Done New false
13263 3849 Acked 4848 Read 4 Done Done New false
13264 3850 Acked 4849 Read 2 Done Done New false
13265 3851 Acked 4850 Write 10 Done Done New false
13266 3852 Acked 4851 Write 6 Done Done New false
13267 3853 Acked 4852 Read 8 Done Done New false
13268 3854 Acked 4853 Read 7 Done Done New false
13269 3855 Acked 4854 Write 9 Done Done New false
13270 3856 Acked 4855 Read 2 Done Done New false
13271 3857 Acked 4856 Write 2 Done Done New false
13272 3858 Acked 4857 Write 2 Done Done New false
13273 3859 Acked 4858 Write 2 Done Done New false
13274 3860 Acked 4859 Write 7 Done Done New false
13275 3861 Acked 4860 Flush 0 Done Done New false
13276 3862 Acked 4861 Write 2 Done Done New false
13277 3863 Acked 4862 Write 4 Done Done New false
13278 3864 Acked 4863 Write 1 Done Done New false
13279 3865 Acked 4864 Write 5 Done Done New false
13280 3866 Acked 4865 Read 5 Done Done New false
13281 3867 Acked 4866 Read 5 Done Done New false
13282 3868 Acked 4867 Read 5 Done Done New false
13283 3869 Acked 4868 Read 6 Done Done New false
13284 3870 Acked 4869 Read 9 Done Done New false
13285 3871 Acked 4870 Write 6 Done Done New false
13286 3872 Acked 4871 Read 10 Done Done New false
13287 3873 Acked 4872 Write 8 Done Done New false
13288 3874 Acked 4873 Read 6 Done Done New false
13289 3875 Acked 4874 Write 5 Done Done New false
13290 3876 Acked 4875 Flush 0 Done Done New false
13291 3877 Acked 4876 Read 7 Done Done New false
13292 3878 Acked 4877 Read 4 Done Done New false
13293 3879 Acked 4878 Read 2 Done Done New false
13294 3880 Acked 4879 Read 10 Done Done New false
13295 3881 Acked 4880 Flush 0 Done Done New false
13296 3882 Acked 4881 Flush 0 Done Done New false
13297 3883 Acked 4882 Write 10 Done Done New false
13298 3884 Acked 4883 Flush 0 Done Done New false
13299 3885 Acked 4884 Read 6 Done Done New false
13300 3886 Acked 4885 Write 7 Done Done New false
13301 3887 Acked 4886 Read 6 Done Done New false
13302 3888 Acked 4887 Read 8 Done Done New false
13303 3889 Acked 4888 Write 6 Done Done New false
13304 3890 Acked 4889 Write 8 Done Done New false
13305 3891 Acked 4890 Write 5 Done Done New false
13306 3892 Acked 4891 Flush 0 Done Done New false
13307 3893 Acked 4892 Read 2 Done Done New false
13308 3894 Acked 4893 Write 4 Done Done New false
13309 3895 Acked 4894 Write 4 Done Done New false
13310 3896 Acked 4895 Write 7 Done Done New false
13311 3897 Acked 4896 Read 7 Done Done New false
13312 3898 Acked 4897 Read 1 Done Done New false
13313 3899 Acked 4898 Write 8 Done Done New false
13314 3900 Acked 4899 Write 8 Done Done New false
13315 3901 Acked 4900 Read 6 Done Done New false
13316 3902 Acked 4901 Read 6 Done Done New false
13317 3903 Acked 4902 Read 1 Done Done New false
13318 3904 Acked 4903 Read 9 Done Done New false
13319 3905 Acked 4904 Read 1 Done Done New false
13320 3906 Acked 4905 Write 1 Done Done New false
13321 3907 Acked 4906 Flush 0 Done Done New false
13322 3908 Acked 4907 Write 10 Done Done New false
13323 3909 Acked 4908 Read 6 Done Done New false
13324 3910 Acked 4909 Write 8 Done Done New false
13325 3911 Acked 4910 Read 7 Done Done New false
13326 3912 Acked 4911 Flush 0 Done Done New false
13327 3913 Acked 4912 Write 7 Done Done New false
13328 3914 Acked 4913 Write 3 Done Done New false
13329 3915 Acked 4914 Write 6 Done Done New false
13330 3916 Acked 4915 Flush 0 Done Done New false
13331 3917 Acked 4916 Write 4 Done Done New false
13332 STATES DS:0 DS:1 DS:2 TOTAL
13333 New 0 0 301 301
13334 Sent 0 0 0 0
13335 Done 301 301 0 602
13336 Skipped 0 0 0 0
13337 Error 0 0 0 0
13338 Last Flush: 4915 4915 4615
13339 Downstairs last five completed: 4615 4614 4613 4612 4611
13340 Upstairs last five completed: 3917 3916 3915 3914 3913
13341 CLIENT: Up:0 ds:301 act:2
133422024-02-15T03:00:07.706ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
133432024-02-15T03:00:07.706ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
133442024-02-15T03:00:07.706ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
133452024-02-15T03:00:07.706ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
133462024-02-15T03:00:07.706ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
133472024-02-15T03:00:07.706ZINFOcrucible: [2] client re-new 302 jobs since flush 4615 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
133482024-02-15T03:00:07.706ZINFOcrucible: client task is exiting = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
133492024-02-15T03:00:08.707ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
133502024-02-15T03:00:08.707ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
133512024-02-15T03:00:08.707ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
133522024-02-15T03:00:08.707ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
133532024-02-15T03:00:08.707ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
133542024-02-15T03:00:08.707ZINFOcrucible: [2] client re-new 302 jobs since flush 4615 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
133552024-02-15T03:00:08.707ZINFOcrucible: client task is exiting = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
133562024-02-15T03:00:09.709ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
133572024-02-15T03:00:09.709ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
133582024-02-15T03:00:09.709ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
133592024-02-15T03:00:09.709ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
133602024-02-15T03:00:09.709ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
133612024-02-15T03:00:09.709ZINFOcrucible: [2] client re-new 302 jobs since flush 4615 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
133622024-02-15T03:00:09.709ZINFOcrucible: client task is exiting = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
13363 [8830] got start action
13364 [8830] I am going to restart
13365 Starting downstairs at port 8830
13366 Make output file at "/tmp/dsc/downstairs-8830.txt"
13367 [8830][2] reports Starting
133682024-02-15T03:00:10.710ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
133692024-02-15T03:00:10.710ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
133702024-02-15T03:00:10.710ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
133712024-02-15T03:00:10.710ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
133722024-02-15T03:00:10.710ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
133732024-02-15T03:00:10.710ZINFOcrucible: [2] client re-new 302 jobs since flush 4615 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
133742024-02-15T03:00:10.710ZINFOcrucible: client task is exiting = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
13375 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1180)
13376 [8830][2] reports Running
13377 ----------------------------------------------------------------
13378 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
13379 States: Active Active Offline
13380 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
13381 3617 Acked 4616 Write 2 Done Done New true
13382 3618 Acked 4617 Read 7 Done Done New true
13383 3619 Acked 4618 Read 7 Done Done New true
13384 3620 Acked 4619 Write 8 Done Done New true
13385 3621 Acked 4620 Write 6 Done Done New true
13386 3622 Acked 4621 Write 6 Done Done New true
13387 3623 Acked 4622 Flush 0 Done Done New true
13388 3624 Acked 4623 Read 7 Done Done New true
13389 3625 Acked 4624 Write 9 Done Done New true
13390 3626 Acked 4625 Write 7 Done Done New true
13391 3627 Acked 4626 Read 5 Done Done New true
13392 3628 Acked 4627 Read 3 Done Done New true
13393 3629 Acked 4628 Read 1 Done Done New true
13394 3630 Acked 4629 Write 6 Done Done New true
13395 3631 Acked 4630 Flush 0 Done Done New true
13396 3632 Acked 4631 Read 3 Done Done New true
13397 3633 Acked 4632 Read 5 Done Done New true
13398 3634 Acked 4633 Read 3 Done Done New true
13399 3635 Acked 4634 Write 10 Done Done New true
13400 3636 Acked 4635 Read 8 Done Done New true
13401 3637 Acked 4636 Flush 0 Done Done New true
13402 3638 Acked 4637 Read 9 Done Done New true
13403 3639 Acked 4638 Flush 0 Done Done New true
13404 3640 Acked 4639 Read 7 Done Done New true
13405 3641 Acked 4640 Read 8 Done Done New true
13406 3642 Acked 4641 Read 2 Done Done New true
13407 3643 Acked 4642 Read 2 Done Done New true
13408 3644 Acked 4643 Write 4 Done Done New true
13409 3645 Acked 4644 Flush 0 Done Done New true
13410 3646 Acked 4645 Write 10 Done Done New true
13411 3647 Acked 4646 Write 5 Done Done New true
13412 3648 Acked 4647 Write 7 Done Done New true
13413 3649 Acked 4648 Read 3 Done Done New true
13414 3650 Acked 4649 Write 7 Done Done New true
13415 3651 Acked 4650 Read 4 Done Done New true
13416 3652 Acked 4651 Read 7 Done Done New true
13417 3653 Acked 4652 Read 3 Done Done New true
13418 3654 Acked 4653 Write 3 Done Done New true
13419 3655 Acked 4654 Write 10 Done Done New true
13420 3656 Acked 4655 Read 4 Done Done New true
13421 3657 Acked 4656 Read 9 Done Done New true
13422 3658 Acked 4657 Read 3 Done Done New true
13423 3659 Acked 4658 Read 3 Done Done New true
13424 3660 Acked 4659 Read 4 Done Done New true
13425 3661 Acked 4660 Write 1 Done Done New true
13426 3662 Acked 4661 Read 6 Done Done New true
13427 3663 Acked 4662 Write 8 Done Done New true
13428 3664 Acked 4663 Write 1 Done Done New true
13429 3665 Acked 4664 Flush 0 Done Done New true
13430 3666 Acked 4665 Write 4 Done Done New true
13431 3667 Acked 4666 Read 4 Done Done New true
13432 3668 Acked 4667 Read 2 Done Done New true
13433 3669 Acked 4668 Write 7 Done Done New true
13434 3670 Acked 4669 Read 5 Done Done New true
13435 3671 Acked 4670 Write 1 Done Done New true
13436 3672 Acked 4671 Write 7 Done Done New true
13437 3673 Acked 4672 Write 8 Done Done New true
13438 3674 Acked 4673 Flush 0 Done Done New true
13439 3675 Acked 4674 Read 10 Done Done New true
13440 3676 Acked 4675 Write 7 Done Done New true
13441 3677 Acked 4676 Flush 0 Done Done New true
13442 3678 Acked 4677 Flush 0 Done Done New true
13443 3679 Acked 4678 Write 7 Done Done New true
13444 3680 Acked 4679 Read 5 Done Done New true
13445 3681 Acked 4680 Write 1 Done Done New true
13446 3682 Acked 4681 Write 6 Done Done New true
13447 3683 Acked 4682 Write 7 Done Done New true
13448 3684 Acked 4683 Write 5 Done Done New true
13449 3685 Acked 4684 Read 10 Done Done New true
13450 3686 Acked 4685 Read 9 Done Done New true
13451 3687 Acked 4686 Flush 0 Done Done New true
13452 3688 Acked 4687 Write 1 Done Done New true
13453 3689 Acked 4688 Read 5 Done Done New true
13454 3690 Acked 4689 Write 2 Done Done New true
13455 3691 Acked 4690 Write 2 Done Done New true
13456 3692 Acked 4691 Write 2 Done Done New true
13457 3693 Acked 4692 Read 4 Done Done New true
13458 3694 Acked 4693 Write 3 Done Done New true
13459 3695 Acked 4694 Read 7 Done Done New true
13460 3696 Acked 4695 Write 4 Done Done New true
13461 3697 Acked 4696 Read 10 Done Done New true
13462 3698 Acked 4697 Write 6 Done Done New true
13463 3699 Acked 4698 Write 10 Done Done New true
13464 3700 Acked 4699 Read 5 Done Done New true
13465 3701 Acked 4700 Flush 0 Done Done New true
13466 3702 Acked 4701 Write 10 Done Done New true
13467 3703 Acked 4702 Write 10 Done Done New true
13468 3704 Acked 4703 Read 2 Done Done New true
13469 3705 Acked 4704 Read 4 Done Done New true
13470 3706 Acked 4705 Read 6 Done Done New true
13471 3707 Acked 4706 Read 10 Done Done New true
13472 3708 Acked 4707 Read 9 Done Done New true
13473 3709 Acked 4708 Write 4 Done Done New true
13474 3710 Acked 4709 Flush 0 Done Done New true
13475 3711 Acked 4710 Write 10 Done Done New true
13476 3712 Acked 4711 Write 3 Done Done New true
13477 3713 Acked 4712 Write 10 Done Done New true
13478 3714 Acked 4713 Read 10 Done Done New true
13479 3715 Acked 4714 Read 9 Done Done New true
13480 3716 Acked 4715 Flush 0 Done Done New true
13481 3717 Acked 4716 Read 6 Done Done New true
13482 3718 Acked 4717 Read 2 Done Done New true
13483 3719 Acked 4718 Read 4 Done Done New true
13484 3720 Acked 4719 Read 6 Done Done New true
13485 3721 Acked 4720 Read 1 Done Done New true
13486 3722 Acked 4721 Read 1 Done Done New true
13487 3723 Acked 4722 Write 4 Done Done New true
13488 3724 Acked 4723 Read 9 Done Done New true
13489 3725 Acked 4724 Read 7 Done Done New true
13490 3726 Acked 4725 Read 8 Done Done New true
13491 3727 Acked 4726 Write 4 Done Done New true
13492 3728 Acked 4727 Read 10 Done Done New true
13493 3729 Acked 4728 Read 7 Done Done New true
13494 3730 Acked 4729 Read 2 Done Done New true
13495 3731 Acked 4730 Read 6 Done Done New true
13496 3732 Acked 4731 Read 5 Done Done New true
13497 3733 Acked 4732 Flush 0 Done Done New true
13498 3734 Acked 4733 Write 3 Done Done New true
13499 3735 Acked 4734 Read 6 Done Done New true
13500 3736 Acked 4735 Read 9 Done Done New true
13501 3737 Acked 4736 Read 1 Done Done New true
13502 3738 Acked 4737 Read 6 Done Done New true
13503 3739 Acked 4738 Read 8 Done Done New true
13504 3740 Acked 4739 Write 1 Done Done New true
13505 3741 Acked 4740 Write 1 Done Done New true
13506 3742 Acked 4741 Read 8 Done Done New true
13507 3743 Acked 4742 Write 8 Done Done New true
13508 3744 Acked 4743 Read 2 Done Done New true
13509 3745 Acked 4744 Read 8 Done Done New true
13510 3746 Acked 4745 Write 3 Done Done New true
13511 3747 Acked 4746 Read 2 Done Done New true
13512 3748 Acked 4747 Write 4 Done Done New true
13513 3749 Acked 4748 Read 9 Done Done New true
13514 3750 Acked 4749 Read 9 Done Done New true
13515 3751 Acked 4750 Read 8 Done Done New true
13516 3752 Acked 4751 Write 3 Done Done New true
13517 3753 Acked 4752 Write 6 Done Done New true
13518 3754 Acked 4753 Read 1 Done Done New true
13519 3755 Acked 4754 Read 10 Done Done New true
13520 3756 Acked 4755 Flush 0 Done Done New true
13521 3757 Acked 4756 Read 9 Done Done New true
13522 3758 Acked 4757 Write 3 Done Done New true
13523 3759 Acked 4758 Flush 0 Done Done New true
13524 3760 Acked 4759 Read 6 Done Done New true
13525 3761 Acked 4760 Write 7 Done Done New true
13526 3762 Acked 4761 Read 9 Done Done New true
13527 3763 Acked 4762 Read 9 Done Done New true
13528 3764 Acked 4763 Read 9 Done Done New true
13529 3765 Acked 4764 Write 3 Done Done New true
13530 3766 Acked 4765 Write 3 Done Done New true
13531 3767 Acked 4766 Write 9 Done Done New true
13532 3768 Acked 4767 Flush 0 Done Done New true
13533 3769 Acked 4768 Write 4 Done Done New true
13534 3770 Acked 4769 Flush 0 Done Done New true
13535 3771 Acked 4770 Write 9 Done Done New true
13536 3772 Acked 4771 Flush 0 Done Done New true
13537 3773 Acked 4772 Read 8 Done Done New true
13538 3774 Acked 4773 Read 5 Done Done New true
13539 3775 Acked 4774 Write 1 Done Done New true
13540 3776 Acked 4775 Read 9 Done Done New true
13541 3777 Acked 4776 Read 5 Done Done New true
13542 3778 Acked 4777 Read 1 Done Done New true
13543 3779 Acked 4778 Read 4 Done Done New true
13544 3780 Acked 4779 Write 4 Done Done New true
13545 3781 Acked 4780 Write 1 Done Done New true
13546 3782 Acked 4781 Read 6 Done Done New true
13547 3783 Acked 4782 Write 6 Done Done New true
13548 3784 Acked 4783 Read 6 Done Done New true
13549 3785 Acked 4784 Read 9 Done Done New true
13550 3786 Acked 4785 Read 3 Done Done New true
13551 3787 Acked 4786 Read 3 Done Done New true
13552 3788 Acked 4787 Read 9 Done Done New true
13553 3789 Acked 4788 Write 1 Done Done New true
13554 3790 Acked 4789 Read 8 Done Done New true
13555 3791 Acked 4790 Read 1 Done Done New true
13556 3792 Acked 4791 Write 8 Done Done New true
13557 3793 Acked 4792 Read 3 Done Done New true
13558 3794 Acked 4793 Read 5 Done Done New true
13559 3795 Acked 4794 Read 6 Done Done New true
13560 3796 Acked 4795 Write 1 Done Done New true
13561 3797 Acked 4796 Read 9 Done Done New true
13562 3798 Acked 4797 Write 6 Done Done New true
13563 3799 Acked 4798 Write 9 Done Done New true
13564 3800 Acked 4799 Write 4 Done Done New true
13565 3801 Acked 4800 Read 6 Done Done New true
13566 3802 Acked 4801 Write 7 Done Done New true
13567 3803 Acked 4802 Write 3 Done Done New true
13568 3804 Acked 4803 Read 9 Done Done New true
13569 3805 Acked 4804 Write 4 Done Done New true
13570 3806 Acked 4805 Read 1 Done Done New true
13571 3807 Acked 4806 Write 7 Done Done New true
13572 3808 Acked 4807 Read 8 Done Done New true
13573 3809 Acked 4808 Read 2 Done Done New true
13574 3810 Acked 4809 Read 9 Done Done New true
13575 3811 Acked 4810 Read 1 Done Done New true
13576 3812 Acked 4811 Read 8 Done Done New true
13577 3813 Acked 4812 Write 9 Done Done New true
13578 3814 Acked 4813 Read 1 Done Done New true
13579 3815 Acked 4814 Write 8 Done Done New true
13580 3816 Acked 4815 Write 4 Done Done New true
13581 3817 Acked 4816 Read 3 Done Done New true
13582 3818 Acked 4817 Read 5 Done Done New true
13583 3819 Acked 4818 Flush 0 Done Done New true
13584 3820 Acked 4819 Write 1 Done Done New true
13585 3821 Acked 4820 Read 3 Done Done New true
13586 3822 Acked 4821 Flush 0 Done Done New true
13587 3823 Acked 4822 Read 7 Done Done New true
13588 3824 Acked 4823 Read 3 Done Done New true
13589 3825 Acked 4824 Write 1 Done Done New true
13590 3826 Acked 4825 Read 2 Done Done New true
13591 3827 Acked 4826 Read 6 Done Done New true
13592 3828 Acked 4827 Write 10 Done Done New true
13593 3829 Acked 4828 Write 6 Done Done New true
13594 3830 Acked 4829 Write 3 Done Done New true
13595 3831 Acked 4830 Read 3 Done Done New true
13596 3832 Acked 4831 Write 4 Done Done New true
13597 3833 Acked 4832 Write 3 Done Done New true
13598 3834 Acked 4833 Flush 0 Done Done New true
13599 3835 Acked 4834 Read 2 Done Done New true
13600 3836 Acked 4835 Write 8 Done Done New true
13601 3837 Acked 4836 Read 7 Done Done New true
13602 3838 Acked 4837 Read 6 Done Done New true
13603 3839 Acked 4838 Write 2 Done Done New true
13604 3840 Acked 4839 Read 4 Done Done New true
13605 3841 Acked 4840 Read 1 Done Done New true
13606 3842 Acked 4841 Write 6 Done Done New true
13607 3843 Acked 4842 Read 2 Done Done New true
13608 3844 Acked 4843 Write 3 Done Done New true
13609 3845 Acked 4844 Write 9 Done Done New true
13610 3846 Acked 4845 Flush 0 Done Done New true
13611 3847 Acked 4846 Read 2 Done Done New true
13612 3848 Acked 4847 Read 7 Done Done New true
13613 3849 Acked 4848 Read 4 Done Done New true
13614 3850 Acked 4849 Read 2 Done Done New true
13615 3851 Acked 4850 Write 10 Done Done New true
13616 3852 Acked 4851 Write 6 Done Done New true
13617 3853 Acked 4852 Read 8 Done Done New true
13618 3854 Acked 4853 Read 7 Done Done New true
13619 3855 Acked 4854 Write 9 Done Done New true
13620 3856 Acked 4855 Read 2 Done Done New true
13621 3857 Acked 4856 Write 2 Done Done New true
13622 3858 Acked 4857 Write 2 Done Done New true
13623 3859 Acked 4858 Write 2 Done Done New true
13624 3860 Acked 4859 Write 7 Done Done New true
13625 3861 Acked 4860 Flush 0 Done Done New true
13626 3862 Acked 4861 Write 2 Done Done New true
13627 3863 Acked 4862 Write 4 Done Done New true
13628 3864 Acked 4863 Write 1 Done Done New true
13629 3865 Acked 4864 Write 5 Done Done New true
13630 3866 Acked 4865 Read 5 Done Done New true
13631 3867 Acked 4866 Read 5 Done Done New true
13632 3868 Acked 4867 Read 5 Done Done New true
13633 3869 Acked 4868 Read 6 Done Done New true
13634 3870 Acked 4869 Read 9 Done Done New true
13635 3871 Acked 4870 Write 6 Done Done New true
13636 3872 Acked 4871 Read 10 Done Done New true
13637 3873 Acked 4872 Write 8 Done Done New true
13638 3874 Acked 4873 Read 6 Done Done New true
13639 3875 Acked 4874 Write 5 Done Done New true
13640 3876 Acked 4875 Flush 0 Done Done New true
13641 3877 Acked 4876 Read 7 Done Done New true
13642 3878 Acked 4877 Read 4 Done Done New true
13643 3879 Acked 4878 Read 2 Done Done New true
13644 3880 Acked 4879 Read 10 Done Done New true
13645 3881 Acked 4880 Flush 0 Done Done New true
13646 3882 Acked 4881 Flush 0 Done Done New true
13647 3883 Acked 4882 Write 10 Done Done New true
13648 3884 Acked 4883 Flush 0 Done Done New true
13649 3885 Acked 4884 Read 6 Done Done New true
13650 3886 Acked 4885 Write 7 Done Done New true
13651 3887 Acked 4886 Read 6 Done Done New true
13652 3888 Acked 4887 Read 8 Done Done New true
13653 3889 Acked 4888 Write 6 Done Done New true
13654 3890 Acked 4889 Write 8 Done Done New true
13655 3891 Acked 4890 Write 5 Done Done New true
13656 3892 Acked 4891 Flush 0 Done Done New true
13657 3893 Acked 4892 Read 2 Done Done New true
13658 3894 Acked 4893 Write 4 Done Done New true
13659 3895 Acked 4894 Write 4 Done Done New true
13660 3896 Acked 4895 Write 7 Done Done New true
13661 3897 Acked 4896 Read 7 Done Done New true
13662 3898 Acked 4897 Read 1 Done Done New true
13663 3899 Acked 4898 Write 8 Done Done New true
13664 3900 Acked 4899 Write 8 Done Done New true
13665 3901 Acked 4900 Read 6 Done Done New true
13666 3902 Acked 4901 Read 6 Done Done New true
13667 3903 Acked 4902 Read 1 Done Done New true
13668 3904 Acked 4903 Read 9 Done Done New true
13669 3905 Acked 4904 Read 1 Done Done New true
13670 3906 Acked 4905 Write 1 Done Done New true
13671 3907 Acked 4906 Flush 0 Done Done New true
13672 3908 Acked 4907 Write 10 Done Done New true
13673 3909 Acked 4908 Read 6 Done Done New true
13674 3910 Acked 4909 Write 8 Done Done New true
13675 3911 Acked 4910 Read 7 Done Done New true
13676 3912 Acked 4911 Flush 0 Done Done New true
13677 3913 Acked 4912 Write 7 Done Done New true
13678 3914 Acked 4913 Write 3 Done Done New true
13679 3915 Acked 4914 Write 6 Done Done New true
13680 3916 Acked 4915 Flush 0 Done Done New true
13681 3917 Acked 4916 Write 4 Done Done New true
13682 3918 Acked 4917 Flush 0 Done Done New true
13683 STATES DS:0 DS:1 DS:2 TOTAL
13684 New 0 0 302 302
13685 Sent 0 0 0 0
13686 Done 302 302 0 604
13687 Skipped 0 0 0 0
13688 Error 0 0 0 0
13689 Last Flush: 4917 4917 4615
13690 Downstairs last five completed: 4615 4614 4613 4612 4611
13691 Upstairs last five completed: 3918 3917 3916 3915 3914
13692 CLIENT: Up:0 ds:302 act:2
136932024-02-15T03:00:11.712ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
136942024-02-15T03:00:11.712ZINFOcrucible: ds_connection connected = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
136952024-02-15T03:00:11.713ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
136962024-02-15T03:00:11.713ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
136972024-02-15T03:00:11.713ZINFOcrucible: send last flush ID to this DS: 4615 = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
136982024-02-15T03:00:11.713ZINFOcrucible: Replied this last flush ID: 4615 = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
136992024-02-15T03:00:11.713ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
137002024-02-15T03:00:11.713ZINFOcrucible: [2] Transition from Offline to Replay = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
137012024-02-15T03:00:11.713ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
137022024-02-15T03:00:11.713ZINFOcrucible: [2] Transition from Replay to Active = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
13703 ----------------------------------------------------------------
13704 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
13705 Downstairs last five completed: 4917 4916 4915 4914 4913
13706 Upstairs last five completed: 3918 3917 3916 3915 3914
13707 CLIENT: Up:0 ds:0 act:3
13708 Replay: All jobs finished, all DS active.
13709 Main task has work to do, go find it
13710 got dsc Stop(0)
13711 stop 0
13712 Feb 15 03:00:15.038 INFO request completed, latency_us: 359, response_code: 204, [8810] Got stop action so:false kr:false
13713 uri: /stop/cid/0, method: POST, req_id: 5784887f-6de1-4e3c-9942-5c8aad8fe685, remote_addr: 127.0.0.1:54744, local_addr: 127.0.0.1:9998
137142024-02-15T03:00:15.038ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
137152024-02-15T03:00:15.038ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
137162024-02-15T03:00:15.038ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
137172024-02-15T03:00:15.038ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
137182024-02-15T03:00:15.039ZINFOcrucible: [0] client re-new 0 jobs since flush 4917 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
13719 Feb 15 03:00:15.039 {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-15T03:00:15.039100298Z","hostname":"ip-10-150-1-67.us-west-2.compute.internal","pid":1167,"":"io task","client":"0","":"downstairs","session_id":"4f5a63e5-aa00-4890-9f6b-96c2e069fdb2"}
13720 INFO request completed, latency_us: 127, response_code: 200, uri: /state/cid/0, method: GET, req_id: ede16507-ebb5-4b29-9c7e-9b25d2906034, remote_addr: 127.0.0.1:54744, local_addr: 127.0.0.1:9998
13721 [8810] Exited with: ExitStatus(unix_wait_status(9))
13722 [8810][0] reports Exit
137232024-02-15T03:00:16.040ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
137242024-02-15T03:00:16.040ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
137252024-02-15T03:00:16.040ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
137262024-02-15T03:00:16.040ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
137272024-02-15T03:00:16.040ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
137282024-02-15T03:00:16.040ZINFOcrucible: [0] client re-new 0 jobs since flush 4917 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
137292024-02-15T03:00:16.040ZINFOcrucible: client task is exiting = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
137302024-02-15T03:00:17.042ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
137312024-02-15T03:00:17.042ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
137322024-02-15T03:00:17.042ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
137332024-02-15T03:00:17.042ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
137342024-02-15T03:00:17.042ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
137352024-02-15T03:00:17.042ZINFOcrucible: [0] client re-new 0 jobs since flush 4917 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
137362024-02-15T03:00:17.042ZINFOcrucible: client task is exiting = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
137372024-02-15T03:00:18.043ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
137382024-02-15T03:00:18.043ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
137392024-02-15T03:00:18.043ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
137402024-02-15T03:00:18.043ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
137412024-02-15T03:00:18.043ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
137422024-02-15T03:00:18.044ZINFOcrucible: [0] client re-new 0 jobs since flush 4917 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
137432024-02-15T03:00:18.044ZINFOcrucible: client task is exiting = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
13744 Feb 15 03:00:19.041 INFO request completed, latency_us: 182, response_code: 200, uri: /state/cid/0, method: GET, req_id: 9ddb5c28-d1a8-4248-88a7-8901a431979f, remote_addr: 127.0.0.1:54744, local_addr: 127.0.0.1:9998
13745 001/300 Read block 2292 len 8192
13746 002/300 Write block 3345 len 36864 data: 3 4 6 5 5 5 5 4 4
13747 003/300 Flush
137482024-02-15T03:00:19.044ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
137492024-02-15T03:00:19.044ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
137502024-02-15T03:00:19.044ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
137512024-02-15T03:00:19.044ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
137522024-02-15T03:00:19.044ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
137532024-02-15T03:00:19.044ZINFOcrucible: [0] client re-new 3 jobs since flush 4917 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
137542024-02-15T03:00:19.044ZINFOcrucible: client task is exiting = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
13755 004/300 Read block 1072 len 36864
13756 005/300 Write block 1319 len 36864 data: 4 5 5 5 4 4 3 4 4
13757 006/300 Read block 4112 len 28672
13758 007/300 Write block 2503 len 36864 data: 5 5 5 5 5 4 4 2 3
13759 008/300 Read block 1233 len 28672
13760 009/300 Read block 1770 len 28672
13761 010/300 Write block 3675 len 28672 data: 3 3 3 3 2 3 3
13762 011/300 Read block 4018 len 4096
13763 012/300 Read block 3996 len 20480
13764 013/300 Flush
13765 014/300 Read block 4447 len 8192
13766 015/300 Write block 367 len 12288 data: 3 4 5
13767 016/300 Write block 2740 len 40960 data: 4 5 4 4 4 5 3 4 4 4
13768 017/300 Write block 1231 len 32768 data: 2 2 3 4 4 4 3 3
13769 018/300 Write block 1999 len 24576 data: 4 4 6 6 6 5
13770 019/300 Write block 3472 len 20480 data: 2 2 2 2 2
13771 020/300 Write block 4143 len 4096 data: 9
13772 021/300 Write block 476 len 16384 data: 3 4 5 3
13773 022/300 Read block 2165 len 12288
13774 023/300 Write block 3607 len 4096 data: 4
13775 024/300 Read block 3328 len 12288
13776 025/300 Write block 4412 len 16384 data: 2 2 3 3
13777 026/300 Read block 4126 len 36864
13778 027/300 Read block 2664 len 28672
13779 028/300 Read block 4696 len 32768
13780 029/300 Write block 2479 len 20480 data: 5 5 5 5 5
13781 030/300 Read block 610 len 40960
13782 031/300 Read block 1995 len 32768
13783 032/300 Read block 2153 len 36864
13784 033/300 Write block 1495 len 40960 data: 4 3 2 3 2 3 3 2 2 3
13785 034/300 Write block 1463 len 4096 data: 4
13786 035/300 Flush
13787 036/300 Read block 967 len 36864
13788 037/300 Read block 3664 len 8192
13789 038/300 Read block 3804 len 36864
13790 039/300 Write block 4036 len 20480 data: 4 3 3 2 5
13791 040/300 Write block 3660 len 36864 data: 4 4 6 7 5 4 5 7 7
13792 041/300 Read block 1121 len 4096
13793 042/300 Read block 1009 len 4096
13794 043/300 Write block 2685 len 8192 data: 2 2
13795 044/300 Read block 2035 len 20480
13796 045/300 Read block 3503 len 40960
13797 046/300 Read block 4846 len 8192
13798 047/300 Read block 638 len 36864
13799 048/300 Read block 2257 len 12288
13800 049/300 Flush
13801 050/300 Write block 2565 len 12288 data: 2 2 2
13802 051/300 Read block 4301 len 4096
13803 052/300 Read block 2488 len 32768
13804 053/300 Read block 778 len 20480
13805 054/300 Write block 3763 len 16384 data: 4 3 3 4
13806 055/300 Read block 607 len 36864
13807 056/300 Read block 1347 len 8192
13808 057/300 Read block 92 len 28672
13809 058/300 Flush
13810 059/300 Flush
13811 060/300 Read block 4610 len 32768
13812 061/300 Write block 1603 len 8192 data: 2 4
13813 062/300 Read block 1791 len 28672
13814 063/300 Write block 468 len 40960 data: 4 5 5 3 3 3 3 3 4 5
13815 064/300 Read block 4551 len 8192
13816 065/300 Write block 1883 len 24576 data: 3 4 5 4 3 3
13817 066/300 Read block 2710 len 40960
13818 067/300 Write block 1487 len 12288 data: 3 3 5
13819 068/300 Read block 2340 len 8192
13820 069/300 Read block 4165 len 32768
13821 070/300 Write block 2595 len 36864 data: 4 4 3 3 4 4 4 4 2
13822 071/300 Read block 3123 len 4096
13823 072/300 Write block 2016 len 16384 data: 3 4 5 6
13824 073/300 Read block 1711 len 8192
13825 074/300 Flush
13826 075/300 Read block 3006 len 16384
13827 076/300 Read block 1756 len 12288
13828 077/300 Write block 1359 len 32768 data: 2 2 2 2 2 2 3 4
13829 078/300 Write block 3764 len 20480 data: 4 4 5 5 5
13830 079/300 Read block 23 len 16384
13831 080/300 Write block 3438 len 12288 data: 2 3 3
13832 081/300 Flush
13833 082/300 Write block 233 len 16384 data: 4 4 4 5
13834 083/300 Read block 3344 len 28672
13835 084/300 Write block 1032 len 20480 data: 4 4 3 3 2
13836 085/300 Write block 4848 len 4096 data: 3
13837 086/300 Read block 2403 len 4096
13838 087/300 Write block 4108 len 28672 data: 3 4 5 8 8 9 8
13839 088/300 Flush
13840 089/300 Read block 1869 len 8192
13841 090/300 Read block 4619 len 28672
13842 091/300 Flush
13843 092/300 Flush
13844 093/300 Write block 4699 len 32768 data: 2 2 2 3 2 2 2 2
13845 094/300 Write block 4146 len 12288 data: 5 4 4
13846 095/300 Read block 911 len 36864
13847 096/300 Flush
13848 097/300 Write block 4889 len 36864 data: 5 5 5 5 5 5 4 3 2
13849 098/300 Read block 2596 len 8192
13850 099/300 Flush
13851 100/300 Write block 3279 len 16384 data: 3 3 4 3
13852 101/300 Read block 4344 len 20480
13853 102/300 Read block 2081 len 8192
13854 103/300 Write block 462 len 12288 data: 2 2 3
13855 104/300 Flush
13856 105/300 Read block 1393 len 36864
13857 106/300 Write block 3643 len 4096 data: 4
13858 107/300 Flush
13859 108/300 Write block 4120 len 20480 data: 3 3 3 3 4
13860 109/300 Write block 3667 len 40960 data: 8 8 6 6 6 4 4 4 4 4
13861 110/300 Write block 1295 len 4096 data: 3
13862 111/300 Write block 4626 len 24576 data: 3 3 4 5 4 3
13863 112/300 Read block 3131 len 32768
13864 113/300 Read block 3758 len 20480
13865 114/300 Read block 3537 len 36864
13866 115/300 Write block 452 len 40960 data: 5 4 4 4 4 4 3 3 3 3
13867 116/300 Write block 1803 len 32768 data: 3 3 3 3 4 4 4 6
13868 117/300 Read block 4485 len 12288
13869 118/300 Read block 4631 len 36864
13870 119/300 Read block 3604 len 24576
13871 120/300 Read block 505 len 12288
13872 121/300 Write block 2409 len 20480 data: 3 3 3 2 2
13873 122/300 Write block 2490 len 40960 data: 4 4 4 3 2 2 2 2 2 3
13874 123/300 Flush
13875 124/300 Read block 2327 len 4096
13876 125/300 Read block 3090 len 20480
13877 126/300 Write block 2430 len 32768 data: 4 4 4 3 2 3 3 3
13878 127/300 Write block 1272 len 36864 data: 2 2 2 2 2 2 3 3 4
13879 128/300 Write block 2370 len 40960 data: 2 2 2 2 3 3 4 4 4 4
13880 129/300 Read block 144 len 24576
13881 130/300 Write block 3350 len 36864 data: 6 6 5 5 3 3 3 4 4
13882 131/300 Write block 1489 len 32768 data: 6 4 4 4 4 4 5 4
13883 132/300 Read block 1067 len 12288
13884 133/300 Write block 3428 len 28672 data: 3 3 3 3 3 2 2
13885 134/300 Write block 2394 len 16384 data: 4 3 3 2
13886 135/300 Read block 680 len 12288
13887 136/300 Read block 2392 len 20480
13888 137/300 Flush
13889 138/300 Write block 4014 len 4096 data: 5
13890 139/300 Write block 4779 len 8192 data: 6 6
13891 140/300 Write block 1856 len 16384 data: 3 3 2 2
13892 141/300 Flush
13893 142/300 Read block 2436 len 28672
13894 143/300 Read block 3545 len 16384
13895 144/300 Read block 4980 len 40960
13896 145/300 Flush
13897 146/300 Write block 1301 len 12288 data: 2 4 4
13898 147/300 Read block 962 len 20480
13899 148/300 Read block 4447 len 12288
13900 149/300 Read block 1848 len 4096
13901 150/300 Flush
13902 151/300 Write block 979 len 12288 data: 5 5 4
13903 152/300 Write block 619 len 8192 data: 6 5
13904 153/300 Read block 1783 len 36864
13905 154/300 Read block 1270 len 28672
13906 155/300 Read block 2371 len 36864
13907 156/300 Write block 3177 len 24576 data: 4 4 4 3 3 3
13908 157/300 Write block 260 len 40960 data: 5 5 4 4 4 4 4 4 3 3
13909 158/300 Write block 2473 len 36864 data: 4 4 4 4 4 4 6 6 6
13910 159/300 Flush
13911 160/300 Read block 4898 len 32768
13912 161/300 Write block 1129 len 4096 data: 3
13913 162/300 Write block 4427 len 16384 data: 7 5 5 4
13914 163/300 Flush
13915 164/300 Write block 1531 len 8192 data: 6 4
13916 165/300 Write block 4586 len 4096 data: 2
13917 166/300 Read block 4290 len 36864
13918 167/300 Flush
13919 168/300 Write block 3857 len 20480 data: 4 4 4 5 4
13920 169/300 Write block 3615 len 4096 data: 3
13921 170/300 Write block 1276 len 12288 data: 3 3 4
13922 171/300 Read block 2570 len 32768
13923 172/300 Write block 2545 len 8192 data: 4 5
13924 173/300 Read block 73 len 12288
13925 174/300 Read block 3528 len 24576
13926 175/300 Read block 1801 len 40960
13927 176/300 Flush
13928 177/300 Write block 1036 len 20480 data: 3 3 3 4 4
13929 178/300 Read block 4991 len 36864
13930 179/300 Read block 1801 len 4096
13931 180/300 Read block 4391 len 32768
13932 181/300 Read block 2693 len 28672
13933 182/300 Write block 903 len 36864 data: 5 3 3 3 3 3 4 4 3
13934 183/300 Read block 71 len 36864
13935 184/300 Write block 1404 len 24576 data: 5 5 6 6 5 4
13936 185/300 Read block 2305 len 4096
13937 186/300 Flush
13938 187/300 Write block 3873 len 32768 data: 6 5 4 4 3 3 3 5
13939 188/300 Read block 3683 len 20480
13940 189/300 Read block 3781 len 32768
13941 190/300 Write block 56 len 4096 data: 6
13942 191/300 Write block 1651 len 20480 data: 4 4 3 4 4
13943 192/300 Read block 4376 len 28672
13944 193/300 Write block 3693 len 40960 data: 4 6 5 5 5 5 3 3 3 2
13945 194/300 Write block 1245 len 16384 data: 4 3 3 4
13946 195/300 Write block 3107 len 40960 data: 4 5 6 6 7 6 5 5 4 3
13947 196/300 Read block 877 len 20480
13948 197/300 Write block 4428 len 24576 data: 6 6 5 4 5 6
13949 198/300 Read block 4306 len 40960
13950 199/300 Write block 4602 len 28672 data: 3 4 3 3 3 2 2
13951 200/300 Flush
13952 201/300 Write block 1472 len 4096 data: 4
13953 202/300 Read block 2939 len 24576
13954 203/300 Read block 3671 len 8192
13955 204/300 Write block 3240 len 8192 data: 4 4
13956 205/300 Write block 670 len 28672 data: 5 5 5 5 5 3 3
13957 206/300 Read block 3368 len 32768
13958 207/300 Flush
13959 208/300 Read block 1978 len 32768
13960 209/300 Write block 4464 len 16384 data: 4 4 4 4
13961 210/300 Write block 3098 len 4096 data: 4
13962 211/300 Read block 357 len 4096
13963 212/300 Write block 4511 len 36864 data: 2 2 2 2 2 2 3 3 3
13964 213/300 Flush
13965 214/300 Write block 3640 len 20480 data: 3 3 3 5 4
13966 215/300 Flush
13967 216/300 Read block 2443 len 28672
13968 217/300 Read block 2532 len 28672
13969 218/300 Read block 475 len 24576
13970 219/300 Read block 729 len 16384
13971 220/300 Read block 3086 len 20480
13972 221/300 Read block 3528 len 28672
13973 222/300 Read block 3886 len 16384
13974 223/300 Read block 1944 len 32768
13975 224/300 Read block 2259 len 24576
13976 225/300 Read block 1241 len 8192
13977 226/300 Write block 4711 len 28672 data: 3 4 5 5 4 3 3
13978 227/300 Write block 1082 len 40960 data: 3 3 3 3 4 3 3 3 3 3
13979 228/300 Write block 4895 len 32768 data: 5 4 3 4 4 4 3 3
13980 229/300 Read block 3460 len 40960
13981 230/300 Write block 4213 len 8192 data: 3 4
13982 231/300 Read block 1756 len 12288
13983 232/300 Flush
13984 233/300 Write block 2558 len 36864 data: 2 3 3 3 3 3 3 3 3
13985 234/300 Write block 3404 len 40960 data: 5 2 2 3 4 4 6 6 5 5
13986 235/300 Write block 3334 len 24576 data: 3 3 2 2 4 4
13987 236/300 Flush
13988 237/300 Read block 4561 len 16384
13989 238/300 Write block 2707 len 20480 data: 2 3 3 3 3
13990 239/300 Write block 4385 len 16384 data: 4 4 4 3
13991 240/300 Read block 4816 len 28672
13992 241/300 Write block 2516 len 40960 data: 3 3 2 2 3 4 4 4 5 5
13993 242/300 Read block 2275 len 20480
13994 243/300 Read block 4215 len 20480
13995 244/300 Write block 261 len 12288 data: 6 5 5
13996 245/300 Read block 4423 len 16384
13997 246/300 Write block 428 len 20480 data: 3 2 2 3 3
13998 247/300 Read block 4825 len 12288
13999 248/300 Write block 3603 len 32768 data: 2 3 3 3 5 4 4 5
14000 249/300 Read block 2983 len 32768
14001 250/300 Write block 2378 len 4096 data: 5
14002 251/300 Read block 2480 len 40960
14003 252/300 Write block 4550 len 20480 data: 6 5 5 4 4
14004 253/300 Read block 4571 len 40960
14005 254/300 Read block 3805 len 36864
14006 255/300 Write block 1758 len 24576 data: 3 3 4 3 3 4
14007 256/300 Write block 4554 len 4096 data: 5
14008 257/300 Read block 3884 len 40960
14009 258/300 Write block 1332 len 12288 data: 4 4 3
14010 259/300 Read block 973 len 36864
14011 260/300 Read block 223 len 40960
14012 261/300 Read block 1778 len 24576
14013 262/300 Read block 819 len 12288
14014 263/300 Write block 4017 len 16384 data: 5 4 5 5
14015 264/300 Write block 1482 len 24576 data: 4 4 4 4 4 4
14016 265/300 Write block 3763 len 32768 data: 5 5 5 6 6 6 5 5
14017 266/300 Flush
14018 267/300 Read block 1014 len 32768
14019 268/300 Write block 898 len 20480 data: 2 3 4 4 5
14020 269/300 Read block 530 len 16384
14021 270/300 Read block 4113 len 4096
14022 271/300 Write block 4185 len 12288 data: 2 2 3
14023 272/300 Read block 1986 len 12288
14024 273/300 Flush
14025 274/300 Read block 671 len 4096
14026 275/300 Write block 563 len 12288 data: 2 3 4
14027 276/300 Write block 4769 len 20480 data: 5 4 4 4 5
14028 277/300 Read block 2419 len 12288
14029 278/300 Read block 1303 len 28672
14030 279/300 Read block 2999 len 20480
14031 280/300 Read block 3037 len 12288
14032 281/300 Read block 2728 len 20480
14033 282/300 Read block 919 len 20480
14034 283/300 Write block 646 len 28672 data: 2 2 2 3 4 4 4
14035 284/300 Flush
14036 285/300 Write block 2705 len 36864 data: 3 3 3 4 4 4 4 4 4
14037 286/300 Read block 4908 len 40960
14038 287/300 Read block 4380 len 12288
14039 288/300 Write block 1188 len 8192 data: 5 5
14040 289/300 Read block 1640 len 20480
14041 290/300 Write block 888 len 32768 data: 5 5 5 5 2 2 2 2
14042 291/300 Read block 4440 len 32768
14043 292/300 Write block 1477 len 16384 data: 2 3 3 4
14044 293/300 Write block 999 len 4096 data: 5
14045 294/300 Read block 705 len 12288
14046 295/300 Read block 3177 len 4096
14047 296/300 Write block 1477 len 20480 data: 3 4 4 5 4
14048 297/300 Flush
14049 298/300 Read block 3606 len 28672
14050 299/300 Write block 1732 len 28672 data: 5 5 6 5 5 6 6
14051 300/300 Write block 1795 len 28672 data: 4 4 4 4 4 3 2
14052 Main task has work to do, go find it
14053 got dsc Start(0)
14054 start 0
14055 Feb 15 03:00:19.374 INFO request completed, latency_us: 384, response_code: 204, uri: /start/cid/0, method: POST, req_id: de068028-6c24-456c-9dab-c39f5aaa76dd, remote_addr: 127.0.0.1:54744, local_addr: 127.0.0.1:9998
14056 Replay: started 0, returned:Ok(())
14057 ----------------------------------------------------------------
14058 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:300
14059 States: Offline Active Active
14060 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
14061 3919 Acked 4918 Read 2 New Done Done true
14062 3920 Acked 4919 Write 9 New Done Done true
14063 3921 Acked 4920 Flush 0 New Done Done true
14064 3922 Acked 4921 Read 9 New Done Done false
14065 3923 Acked 4922 Write 9 New Done Done false
14066 3924 Acked 4923 Read 7 New Done Done false
14067 3925 Acked 4924 Write 9 New Done Done false
14068 3926 Acked 4925 Read 7 New Done Done false
14069 3927 Acked 4926 Read 7 New Done Done false
14070 3928 Acked 4927 Write 7 New Done Done false
14071 3929 Acked 4928 Read 1 New Done Done false
14072 3930 Acked 4929 Read 5 New Done Done false
14073 3931 Acked 4930 Flush 0 New Done Done false
14074 3932 Acked 4931 Read 2 New Done Done false
14075 3933 Acked 4932 Write 3 New Done Done false
14076 3934 Acked 4933 Write 10 New Done Done false
14077 3935 Acked 4934 Write 8 New Done Done false
14078 3936 Acked 4935 Write 6 New Done Done false
14079 3937 Acked 4936 Write 5 New Done Done false
14080 3938 Acked 4937 Write 1 New Done Done false
14081 3939 Acked 4938 Write 4 New Done Done false
14082 3940 Acked 4939 Read 3 New Done Done false
14083 3941 Acked 4940 Write 1 New Done Done false
14084 3942 Acked 4941 Read 3 New Done Done false
14085 3943 Acked 4942 Write 4 New Done Done false
14086 3944 Acked 4943 Read 9 New Done Done false
14087 3945 Acked 4944 Read 7 New Done Done false
14088 3946 Acked 4945 Read 8 New Done Done false
14089 3947 Acked 4946 Write 5 New Done Done false
14090 3948 Acked 4947 Read 10 New Done Done false
14091 3949 Acked 4948 Read 8 New Done Done false
14092 3950 Acked 4949 Read 9 New Done Done false
14093 3951 Acked 4950 Write 10 New Done Done false
14094 3952 Acked 4951 Write 1 New Done Done false
14095 3953 Acked 4952 Flush 0 New Done Done false
14096 3954 Acked 4953 Read 9 New Done Done false
14097 3955 Acked 4954 Read 2 New Done Done false
14098 3956 Acked 4955 Read 9 New Done Done false
14099 3957 Acked 4956 Write 5 New Done Done false
14100 3958 Acked 4957 Write 9 New Done Done false
14101 3959 Acked 4958 Read 1 New Done Done false
14102 3960 Acked 4959 Read 1 New Done Done false
14103 3961 Acked 4960 Write 2 New Done Done false
14104 3962 Acked 4961 Read 5 New Done Done false
14105 3963 Acked 4962 Read 10 New Done Done false
14106 3964 Acked 4963 Read 2 New Done Done false
14107 3965 Acked 4964 Read 9 New Done Done false
14108 3966 Acked 4965 Read 3 New Done Done false
14109 3967 Acked 4966 Flush 0 New Done Done false
14110 3968 Acked 4967 Write 3 New Done Done false
14111 3969 Acked 4968 Read 1 New Done Done false
14112 3970 Acked 4969 Read 8 New Done Done false
14113 3971 Acked 4970 Read 5 New Done Done false
14114 3972 Acked 4971 Write 4 New Done Done false
14115 3973 Acked 4972 Read 9 New Done Done false
14116 3974 Acked 4973 Read 2 New Done Done false
14117 3975 Acked 4974 Read 7 New Done Done false
14118 3976 Acked 4975 Flush 0 New Done Done false
14119 3977 Acked 4976 Flush 0 New Done Done false
14120 3978 Acked 4977 Read 8 New Done Done false
14121 3979 Acked 4978 Write 2 New Done Done false
14122 3980 Acked 4979 Read 7 New Done Done false
14123 3981 Acked 4980 Write 10 New Done Done false
14124 3982 Acked 4981 Read 2 New Done Done false
14125 3983 Acked 4982 Write 6 New Done Done false
14126 3984 Acked 4983 Read 10 New Done Done false
14127 3985 Acked 4984 Write 3 New Done Done false
14128 3986 Acked 4985 Read 2 New Done Done false
14129 3987 Acked 4986 Read 8 New Done Done false
14130 3988 Acked 4987 Write 9 New Done Done false
14131 3989 Acked 4988 Read 1 New Done Done false
14132 3990 Acked 4989 Write 4 New Done Done false
14133 3991 Acked 4990 Read 2 New Done Done false
14134 3992 Acked 4991 Flush 0 New Done Done false
14135 3993 Acked 4992 Read 4 New Done Done false
14136 3994 Acked 4993 Read 3 New Done Done false
14137 3995 Acked 4994 Write 8 New Done Done false
14138 3996 Acked 4995 Write 5 New Done Done false
14139 3997 Acked 4996 Read 4 New Done Done false
14140 3998 Acked 4997 Write 3 New Done Done false
14141 3999 Acked 4998 Flush 0 New Done Done false
14142 4000 Acked 4999 Write 4 New Done Done false
14143 4001 Acked 5000 Read 7 New Done Done false
14144 4002 Acked 5001 Write 5 New Done Done false
14145 4003 Acked 5002 Write 1 New Done Done false
14146 4004 Acked 5003 Read 1 New Done Done false
14147 4005 Acked 5004 Write 7 New Done Done false
14148 4006 Acked 5005 Flush 0 New Done Done false
14149 4007 Acked 5006 Read 2 New Done Done false
14150 4008 Acked 5007 Read 7 New Done Done false
14151 4009 Acked 5008 Flush 0 New Done Done false
14152 4010 Acked 5009 Flush 0 New Done Done false
14153 4011 Acked 5010 Write 8 New Done Done false
14154 4012 Acked 5011 Write 3 New Done Done false
14155 4013 Acked 5012 Read 9 New Done Done false
14156 4014 Acked 5013 Flush 0 New Done Done false
14157 4015 Acked 5014 Write 9 New Done Done false
14158 4016 Acked 5015 Read 2 New Done Done false
14159 4017 Acked 5016 Flush 0 New Done Done false
14160 4018 Acked 5017 Write 4 New Done Done false
14161 4019 Acked 5018 Read 5 New Done Done false
14162 4020 Acked 5019 Read 2 New Done Done false
14163 4021 Acked 5020 Write 3 New Done Done false
14164 4022 Acked 5021 Flush 0 New Done Done false
14165 4023 Acked 5022 Read 9 New Done Done false
14166 4024 Acked 5023 Write 1 New Done Done false
14167 4025 Acked 5024 Flush 0 New Done Done false
14168 4026 Acked 5025 Write 5 New Done Done false
14169 4027 Acked 5026 Write 10 New Done Done false
14170 4028 Acked 5027 Write 1 New Done Done false
14171 4029 Acked 5028 Write 6 New Done Done false
14172 4030 Acked 5029 Read 8 New Done Done false
14173 4031 Acked 5030 Read 5 New Done Done false
14174 4032 Acked 5031 Read 9 New Done Done false
14175 4033 Acked 5032 Write 10 New Done Done false
14176 4034 Acked 5033 Write 8 New Done Done false
14177 4035 Acked 5034 Read 3 New Done Done false
14178 4036 Acked 5035 Read 9 New Done Done false
14179 4037 Acked 5036 Read 6 New Done Done false
14180 4038 Acked 5037 Read 3 New Done Done false
14181 4039 Acked 5038 Write 5 New Done Done false
14182 4040 Acked 5039 Write 10 New Done Done false
14183 4041 Acked 5040 Flush 0 New Done Done false
14184 4042 Acked 5041 Read 1 New Done Done false
14185 4043 Acked 5042 Read 5 New Done Done false
14186 4044 Acked 5043 Write 8 New Done Done false
14187 4045 Acked 5044 Write 9 New Done Done false
14188 4046 Acked 5045 Write 10 New Done Done false
14189 4047 Acked 5046 Read 6 New Done Done false
14190 4048 Acked 5047 Write 9 New Done Done false
14191 4049 Acked 5048 Write 8 New Done Done false
14192 4050 Acked 5049 Read 3 New Done Done false
14193 4051 Acked 5050 Write 7 New Done Done false
14194 4052 Acked 5051 Write 4 New Done Done false
14195 4053 Acked 5052 Read 3 New Done Done false
14196 4054 Acked 5053 Read 5 New Done Done false
14197 4055 Acked 5054 Flush 0 New Done Done false
14198 4056 Acked 5055 Write 1 New Done Done false
14199 4057 Acked 5056 Write 2 New Done Done false
14200 4058 Acked 5057 Write 4 New Done Done false
14201 4059 Acked 5058 Flush 0 New Done Done false
14202 4060 Acked 5059 Read 7 New Done Done false
14203 4061 Acked 5060 Read 4 New Done Done false
14204 4062 Acked 5061 Read 10 New Done Done false
14205 4063 Acked 5062 Flush 0 New Done Done false
14206 4064 Acked 5063 Write 3 New Done Done false
14207 4065 Acked 5064 Read 5 New Done Done false
14208 4066 Acked 5065 Read 3 New Done Done false
14209 4067 Acked 5066 Read 1 New Done Done false
14210 4068 Acked 5067 Flush 0 New Done Done false
14211 4069 Acked 5068 Write 3 New Done Done false
14212 4070 Acked 5069 Write 2 New Done Done false
14213 4071 Acked 5070 Read 9 New Done Done false
14214 4072 Acked 5071 Read 7 New Done Done false
14215 4073 Acked 5072 Read 9 New Done Done false
14216 4074 Acked 5073 Write 6 New Done Done false
14217 4075 Acked 5074 Write 10 New Done Done false
14218 4076 Acked 5075 Write 9 New Done Done false
14219 4077 Acked 5076 Flush 0 New Done Done false
14220 4078 Acked 5077 Read 8 New Done Done false
14221 4079 Acked 5078 Write 1 New Done Done false
14222 4080 Acked 5079 Write 4 New Done Done false
14223 4081 Acked 5080 Flush 0 New Done Done false
14224 4082 Acked 5081 Write 2 New Done Done false
14225 4083 Acked 5082 Write 1 New Done Done false
14226 4084 Acked 5083 Read 9 New Done Done false
14227 4085 Acked 5084 Flush 0 New Done Done false
14228 4086 Acked 5085 Write 5 New Done Done false
14229 4087 Acked 5086 Write 1 New Done Done false
14230 4088 Acked 5087 Write 3 New Done Done false
14231 4089 Acked 5088 Read 8 New Done Done false
14232 4090 Acked 5089 Write 2 New Done Done false
14233 4091 Acked 5090 Read 3 New Done Done false
14234 4092 Acked 5091 Read 6 New Done Done false
14235 4093 Acked 5092 Read 10 New Done Done false
14236 4094 Acked 5093 Flush 0 New Done Done false
14237 4095 Acked 5094 Write 5 New Done Done false
14238 4096 Acked 5095 Read 9 New Done Done false
14239 4097 Acked 5096 Read 1 New Done Done false
14240 4098 Acked 5097 Read 8 New Done Done false
14241 4099 Acked 5098 Read 7 New Done Done false
14242 4100 Acked 5099 Write 9 New Done Done false
14243 4101 Acked 5100 Read 9 New Done Done false
14244 4102 Acked 5101 Write 6 New Done Done false
14245 4103 Acked 5102 Read 1 New Done Done false
14246 4104 Acked 5103 Flush 0 New Done Done false
14247 4105 Acked 5104 Write 8 New Done Done false
14248 4106 Acked 5105 Read 5 New Done Done false
14249 4107 Acked 5106 Read 8 New Done Done false
14250 4108 Acked 5107 Write 1 New Done Done false
14251 4109 Acked 5108 Write 5 New Done Done false
14252 4110 Acked 5109 Read 7 New Done Done false
14253 4111 Acked 5110 Write 10 New Done Done false
14254 4112 Acked 5111 Write 4 New Done Done false
14255 4113 Acked 5112 Write 10 New Done Done false
14256 4114 Acked 5113 Read 5 New Done Done false
14257 4115 Acked 5114 Write 6 New Done Done false
14258 4116 Acked 5115 Read 10 New Done Done false
14259 4117 Acked 5116 Write 7 New Done Done false
14260 4118 Acked 5117 Flush 0 New Done Done false
14261 4119 Acked 5118 Write 1 New Done Done false
14262 4120 Acked 5119 Read 6 New Done Done false
14263 4121 Acked 5120 Read 2 New Done Done false
14264 4122 Acked 5121 Write 2 New Done Done false
14265 4123 Acked 5122 Write 7 New Done Done false
14266 4124 Acked 5123 Read 8 New Done Done false
14267 4125 Acked 5124 Flush 0 New Done Done false
14268 4126 Acked 5125 Read 8 New Done Done false
14269 4127 Acked 5126 Write 4 New Done Done false
14270 4128 Acked 5127 Write 1 New Done Done false
14271 4129 Acked 5128 Read 1 New Done Done false
14272 4130 Acked 5129 Write 9 New Done Done false
14273 4131 Acked 5130 Flush 0 New Done Done false
14274 4132 Acked 5131 Write 5 New Done Done false
14275 4133 Acked 5132 Flush 0 New Done Done false
14276 4134 Acked 5133 Read 7 New Done Done false
14277 4135 Acked 5134 Read 7 New Done Done false
14278 4136 Acked 5135 Read 6 New Done Done false
14279 4137 Acked 5136 Read 4 New Done Done false
14280 4138 Acked 5137 Read 5 New Done Done false
14281 4139 Acked 5138 Read 7 New Done Done false
14282 4140 Acked 5139 Read 4 New Done Done false
14283 4141 Acked 5140 Read 8 New Done Done false
14284 4142 Acked 5141 Read 6 New Done Done false
14285 4143 Acked 5142 Read 2 New Done Done false
14286 4144 Acked 5143 Write 7 New Done Done false
14287 4145 Acked 5144 Write 10 New Done Done false
14288 4146 Acked 5145 Write 8 New Done Done false
14289 4147 Acked 5146 Read 10 New Done Done false
14290 4148 Acked 5147 Write 2 New Done Done false
14291 4149 Acked 5148 Read 3 New Done Done false
14292 4150 Acked 5149 Flush 0 New Done Done false
14293 4151 Acked 5150 Write 9 New Done Done false
14294 4152 Acked 5151 Write 10 New Done Done false
14295 4153 Acked 5152 Write 6 New Done Done false
14296 4154 Acked 5153 Flush 0 New Done Done false
14297 4155 Acked 5154 Read 4 New Done Done false
14298 4156 Acked 5155 Write 5 New Done Done false
14299 4157 Acked 5156 Write 4 New Done Done false
14300 4158 Acked 5157 Read 7 New Done Done false
14301 4159 Acked 5158 Write 10 New Done Done false
14302 4160 Acked 5159 Read 5 New Done Done false
14303 4161 Acked 5160 Read 5 New Done Done false
14304 4162 Acked 5161 Write 3 New Done Done false
14305 4163 Acked 5162 Read 4 New Done Done false
14306 4164 Acked 5163 Write 5 New Done Done false
14307 4165 Acked 5164 Read 3 New Done Done false
14308 4166 Acked 5165 Write 8 New Done Done false
14309 4167 Acked 5166 Read 8 New Done Done false
14310 4168 Acked 5167 Write 1 New Done Done false
14311 4169 Acked 5168 Read 10 New Done Done false
14312 4170 Acked 5169 Write 5 New Done Done false
14313 4171 Acked 5170 Read 10 New Done Done false
14314 4172 Acked 5171 Read 9 New Done Done false
14315 4173 Acked 5172 Write 6 New Done Done false
14316 4174 Acked 5173 Write 1 New Done Done false
14317 4175 Acked 5174 Read 10 New Done Done false
14318 4176 Acked 5175 Write 3 New Done Done false
14319 4177 Acked 5176 Read 9 New Done Done false
14320 4178 Acked 5177 Read 10 New Done Done false
14321 4179 Acked 5178 Read 6 New Done Done false
14322 4180 Acked 5179 Read 3 New Done Done false
14323 4181 Acked 5180 Write 4 New Done Done false
14324 4182 Acked 5181 Write 6 New Done Done false
14325 4183 Acked 5182 Write 8 New Done Done false
14326 4184 Acked 5183 Flush 0 New Done Done false
14327 4185 Acked 5184 Read 8 New Done Done false
14328 4186 Acked 5185 Write 5 New Done Done false
14329 4187 Acked 5186 Read 4 New Done Done false
14330 4188 Acked 5187 Read 1 New Done Done false
14331 4189 Acked 5188 Write 3 New Done Done false
14332 4190 Acked 5189 Read 3 New Done Done false
14333 4191 Acked 5190 Flush 0 New Done Done false
14334 4192 Acked 5191 Read 1 New Done Done false
14335 4193 Acked 5192 Write 3 New Done Done false
14336 4194 Acked 5193 Write 5 New Done Done false
14337 4195 Acked 5194 Read 3 New Done Done false
14338 4196 Acked 5195 Read 7 New Done Done false
14339 4197 Acked 5196 Read 5 New Done Done false
14340 4198 Acked 5197 Read 3 New Done Done false
14341 4199 Acked 5198 Read 5 New Done Done false
14342 4200 Acked 5199 Read 5 New Done Done false
14343 4201 Acked 5200 Write 7 New Done Done false
14344 4202 Acked 5201 Flush 0 New Done Done false
14345 4203 Acked 5202 Write 9 New Done Done false
14346 4204 Acked 5203 Read 10 New Done Done false
14347 4205 Acked 5204 Read 3 New Done Done false
14348 4206 Acked 5205 Write 2 New Done Done false
14349 4207 Acked 5206 Read 5 New Done Done false
14350 4208 Acked 5207 Write 8 New Done Done false
14351 4209 Acked 5208 Read 8 New Done Done false
14352 4210 Acked 5209 Write 4 New Done Done false
14353 4211 Acked 5210 Write 1 New Done Done false
14354 4212 Acked 5211 Read 3 New Done Done false
14355 4213 Acked 5212 Read 1 New Done Done false
14356 4214 Acked 5213 Write 5 New Done Done false
14357 4215 Acked 5214 Flush 0 New Done Done false
14358 4216 Acked 5215 Read 7 New Done Done false
14359 4217 Acked 5216 Write 7 New Done Done false
14360 4218 Acked 5217 Write 7 New Done Done false
14361 STATES DS:0 DS:1 DS:2 TOTAL
14362 New 300 0 0 300
14363 Sent 0 0 0 0
14364 Done 0 300 300 600
14365 Skipped 0 0 0 0
14366 Error 0 0 0 0
14367 Last Flush: 4917 5214 5214
14368 Downstairs last five completed: 4917 4916 4915 4914 4913
14369 Upstairs last five completed: 4218 4217 4216 4215 4214
14370 CLIENT: Up:0 ds:300 act:2
143712024-02-15T03:00:20.045ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
143722024-02-15T03:00:20.045ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
143732024-02-15T03:00:20.045ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
143742024-02-15T03:00:20.045ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
143752024-02-15T03:00:20.045ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
143762024-02-15T03:00:20.045ZINFOcrucible: [0] client re-new 301 jobs since flush 4917 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
143772024-02-15T03:00:20.046ZINFOcrucible: client task is exiting = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
143782024-02-15T03:00:21.046ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
143792024-02-15T03:00:21.047ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
143802024-02-15T03:00:21.047ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
143812024-02-15T03:00:21.047ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
143822024-02-15T03:00:21.047ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
143832024-02-15T03:00:21.047ZINFOcrucible: [0] client re-new 301 jobs since flush 4917 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
143842024-02-15T03:00:21.047ZINFOcrucible: client task is exiting = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
143852024-02-15T03:00:22.047ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
143862024-02-15T03:00:22.047ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
143872024-02-15T03:00:22.047ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
143882024-02-15T03:00:22.047ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
143892024-02-15T03:00:22.047ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
143902024-02-15T03:00:22.047ZINFOcrucible: [0] client re-new 301 jobs since flush 4917 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
143912024-02-15T03:00:22.047ZINFOcrucible: client task is exiting = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
143922024-02-15T03:00:23.048ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
143932024-02-15T03:00:23.048ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
143942024-02-15T03:00:23.048ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
143952024-02-15T03:00:23.048ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
143962024-02-15T03:00:23.048ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
143972024-02-15T03:00:23.048ZINFOcrucible: [0] client re-new 301 jobs since flush 4917 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
143982024-02-15T03:00:23.048ZINFOcrucible: client task is exiting = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
14399 ----------------------------------------------------------------
14400 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
14401 States: Offline Active Active
14402 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
14403 3919 Acked 4918 Read 2 New Done Done true
14404 3920 Acked 4919 Write 9 New Done Done true
14405 3921 Acked 4920 Flush 0 New Done Done true
14406 3922 Acked 4921 Read 9 New Done Done true
14407 3923 Acked 4922 Write 9 New Done Done true
14408 3924 Acked 4923 Read 7 New Done Done true
14409 3925 Acked 4924 Write 9 New Done Done true
14410 3926 Acked 4925 Read 7 New Done Done true
14411 3927 Acked 4926 Read 7 New Done Done true
14412 3928 Acked 4927 Write 7 New Done Done true
14413 3929 Acked 4928 Read 1 New Done Done true
14414 3930 Acked 4929 Read 5 New Done Done true
14415 3931 Acked 4930 Flush 0 New Done Done true
14416 3932 Acked 4931 Read 2 New Done Done true
14417 3933 Acked 4932 Write 3 New Done Done true
14418 3934 Acked 4933 Write 10 New Done Done true
14419 3935 Acked 4934 Write 8 New Done Done true
14420 3936 Acked 4935 Write 6 New Done Done true
14421 3937 Acked 4936 Write 5 New Done Done true
14422 3938 Acked 4937 Write 1 New Done Done true
14423 3939 Acked 4938 Write 4 New Done Done true
14424 3940 Acked 4939 Read 3 New Done Done true
14425 3941 Acked 4940 Write 1 New Done Done true
14426 3942 Acked 4941 Read 3 New Done Done true
14427 3943 Acked 4942 Write 4 New Done Done true
14428 3944 Acked 4943 Read 9 New Done Done true
14429 3945 Acked 4944 Read 7 New Done Done true
14430 3946 Acked 4945 Read 8 New Done Done true
14431 3947 Acked 4946 Write 5 New Done Done true
14432 3948 Acked 4947 Read 10 New Done Done true
14433 3949 Acked 4948 Read 8 New Done Done true
14434 3950 Acked 4949 Read 9 New Done Done true
14435 3951 Acked 4950 Write 10 New Done Done true
14436 3952 Acked 4951 Write 1 New Done Done true
14437 3953 Acked 4952 Flush 0 New Done Done true
14438 3954 Acked 4953 Read 9 New Done Done true
14439 3955 Acked 4954 Read 2 New Done Done true
14440 3956 Acked 4955 Read 9 New Done Done true
14441 3957 Acked 4956 Write 5 New Done Done true
14442 3958 Acked 4957 Write 9 New Done Done true
14443 3959 Acked 4958 Read 1 New Done Done true
14444 3960 Acked 4959 Read 1 New Done Done true
14445 3961 Acked 4960 Write 2 New Done Done true
14446 3962 Acked 4961 Read 5 New Done Done true
14447 3963 Acked 4962 Read 10 New Done Done true
14448 3964 Acked 4963 Read 2 New Done Done true
14449 3965 Acked 4964 Read 9 New Done Done true
14450 3966 Acked 4965 Read 3 New Done Done true
14451 3967 Acked 4966 Flush 0 New Done Done true
14452 3968 Acked 4967 Write 3 New Done Done true
14453 3969 Acked 4968 Read 1 New Done Done true
14454 3970 Acked 4969 Read 8 New Done Done true
14455 3971 Acked 4970 Read 5 New Done Done true
14456 3972 Acked 4971 Write 4 New Done Done true
14457 3973 Acked 4972 Read 9 New Done Done true
14458 3974 Acked 4973 Read 2 New Done Done true
14459 3975 Acked 4974 Read 7 New Done Done true
14460 3976 Acked 4975 Flush 0 New Done Done true
14461 3977 Acked 4976 Flush 0 New Done Done true
14462 3978 Acked 4977 Read 8 New Done Done true
14463 3979 Acked 4978 Write 2 New Done Done true
14464 3980 Acked 4979 Read 7 New Done Done true
14465 3981 Acked 4980 Write 10 New Done Done true
14466 3982 Acked 4981 Read 2 New Done Done true
14467 3983 Acked 4982 Write 6 New Done Done true
14468 3984 Acked 4983 Read 10 New Done Done true
14469 3985 Acked 4984 Write 3 New Done Done true
14470 3986 Acked 4985 Read 2 New Done Done true
14471 3987 Acked 4986 Read 8 New Done Done true
14472 3988 Acked 4987 Write 9 New Done Done true
14473 3989 Acked 4988 Read 1 New Done Done true
14474 3990 Acked 4989 Write 4 New Done Done true
14475 3991 Acked 4990 Read 2 New Done Done true
14476 3992 Acked 4991 Flush 0 New Done Done true
14477 3993 Acked 4992 Read 4 New Done Done true
14478 3994 Acked 4993 Read 3 New Done Done true
14479 3995 Acked 4994 Write 8 New Done Done true
14480 3996 Acked 4995 Write 5 New Done Done true
14481 3997 Acked 4996 Read 4 New Done Done true
14482 3998 Acked 4997 Write 3 New Done Done true
14483 3999 Acked 4998 Flush 0 New Done Done true
14484 4000 Acked 4999 Write 4 New Done Done true
14485 4001 Acked 5000 Read 7 New Done Done true
14486 4002 Acked 5001 Write 5 New Done Done true
14487 4003 Acked 5002 Write 1 New Done Done true
14488 4004 Acked 5003 Read 1 New Done Done true
14489 4005 Acked 5004 Write 7 New Done Done true
14490 4006 Acked 5005 Flush 0 New Done Done true
14491 4007 Acked 5006 Read 2 New Done Done true
14492 4008 Acked 5007 Read 7 New Done Done true
14493 4009 Acked 5008 Flush 0 New Done Done true
14494 4010 Acked 5009 Flush 0 New Done Done true
14495 4011 Acked 5010 Write 8 New Done Done true
14496 4012 Acked 5011 Write 3 New Done Done true
14497 4013 Acked 5012 Read 9 New Done Done true
14498 4014 Acked 5013 Flush 0 New Done Done true
14499 4015 Acked 5014 Write 9 New Done Done true
14500 4016 Acked 5015 Read 2 New Done Done true
14501 4017 Acked 5016 Flush 0 New Done Done true
14502 4018 Acked 5017 Write 4 New Done Done true
14503 4019 Acked 5018 Read 5 New Done Done true
14504 4020 Acked 5019 Read 2 New Done Done true
14505 4021 Acked 5020 Write 3 New Done Done true
14506 4022 Acked 5021 Flush 0 New Done Done true
14507 4023 Acked 5022 Read 9 New Done Done true
14508 4024 Acked 5023 Write 1 New Done Done true
14509 4025 Acked 5024 Flush 0 New Done Done true
14510 4026 Acked 5025 Write 5 New Done Done true
14511 4027 Acked 5026 Write 10 New Done Done true
14512 4028 Acked 5027 Write 1 New Done Done true
14513 4029 Acked 5028 Write 6 New Done Done true
14514 4030 Acked 5029 Read 8 New Done Done true
14515 4031 Acked 5030 Read 5 New Done Done true
14516 4032 Acked 5031 Read 9 New Done Done true
14517 4033 Acked 5032 Write 10 New Done Done true
14518 4034 Acked 5033 Write 8 New Done Done true
14519 4035 Acked 5034 Read 3 New Done Done true
14520 4036 Acked 5035 Read 9 New Done Done true
14521 4037 Acked 5036 Read 6 New Done Done true
14522 4038 Acked 5037 Read 3 New Done Done true
14523 4039 Acked 5038 Write 5 New Done Done true
14524 4040 Acked 5039 Write 10 New Done Done true
14525 4041 Acked 5040 Flush 0 New Done Done true
14526 4042 Acked 5041 Read 1 New Done Done true
14527 4043 Acked 5042 Read 5 New Done Done true
14528 4044 Acked 5043 Write 8 New Done Done true
14529 4045 Acked 5044 Write 9 New Done Done true
14530 4046 Acked 5045 Write 10 New Done Done true
14531 4047 Acked 5046 Read 6 New Done Done true
14532 4048 Acked 5047 Write 9 New Done Done true
14533 4049 Acked 5048 Write 8 New Done Done true
14534 4050 Acked 5049 Read 3 New Done Done true
14535 4051 Acked 5050 Write 7 New Done Done true
14536 4052 Acked 5051 Write 4 New Done Done true
14537 4053 Acked 5052 Read 3 New Done Done true
14538 4054 Acked 5053 Read 5 New Done Done true
14539 4055 Acked 5054 Flush 0 New Done Done true
14540 4056 Acked 5055 Write 1 New Done Done true
14541 4057 Acked 5056 Write 2 New Done Done true
14542 4058 Acked 5057 Write 4 New Done Done true
14543 4059 Acked 5058 Flush 0 New Done Done true
14544 4060 Acked 5059 Read 7 New Done Done true
14545 4061 Acked 5060 Read 4 New Done Done true
14546 4062 Acked 5061 Read 10 New Done Done true
14547 4063 Acked 5062 Flush 0 New Done Done true
14548 4064 Acked 5063 Write 3 New Done Done true
14549 4065 Acked 5064 Read 5 New Done Done true
14550 4066 Acked 5065 Read 3 New Done Done true
14551 4067 Acked 5066 Read 1 New Done Done true
14552 4068 Acked 5067 Flush 0 New Done Done true
14553 4069 Acked 5068 Write 3 New Done Done true
14554 4070 Acked 5069 Write 2 New Done Done true
14555 4071 Acked 5070 Read 9 New Done Done true
14556 4072 Acked 5071 Read 7 New Done Done true
14557 4073 Acked 5072 Read 9 New Done Done true
14558 4074 Acked 5073 Write 6 New Done Done true
14559 4075 Acked 5074 Write 10 New Done Done true
14560 4076 Acked 5075 Write 9 New Done Done true
14561 4077 Acked 5076 Flush 0 New Done Done true
14562 4078 Acked 5077 Read 8 New Done Done true
14563 4079 Acked 5078 Write 1 New Done Done true
14564 4080 Acked 5079 Write 4 New Done Done true
14565 4081 Acked 5080 Flush 0 New Done Done true
14566 4082 Acked 5081 Write 2 New Done Done true
14567 4083 Acked 5082 Write 1 New Done Done true
14568 4084 Acked 5083 Read 9 New Done Done true
14569 4085 Acked 5084 Flush 0 New Done Done true
14570 4086 Acked 5085 Write 5 New Done Done true
14571 4087 Acked 5086 Write 1 New Done Done true
14572 4088 Acked 5087 Write 3 New Done Done true
14573 4089 Acked 5088 Read 8 New Done Done true
14574 4090 Acked 5089 Write 2 New Done Done true
14575 4091 Acked 5090 Read 3 New Done Done true
14576 4092 Acked 5091 Read 6 New Done Done true
14577 4093 Acked 5092 Read 10 New Done Done true
14578 4094 Acked 5093 Flush 0 New Done Done true
14579 4095 Acked 5094 Write 5 New Done Done true
14580 4096 Acked 5095 Read 9 New Done Done true
14581 4097 Acked 5096 Read 1 New Done Done true
14582 4098 Acked 5097 Read 8 New Done Done true
14583 4099 Acked 5098 Read 7 New Done Done true
14584 4100 Acked 5099 Write 9 New Done Done true
14585 4101 Acked 5100 Read 9 New Done Done true
14586 4102 Acked 5101 Write 6 New Done Done true
14587 4103 Acked 5102 Read 1 New Done Done true
14588 4104 Acked 5103 Flush 0 New Done Done true
14589 4105 Acked 5104 Write 8 New Done Done true
14590 4106 Acked 5105 Read 5 New Done Done true
14591 4107 Acked 5106 Read 8 New Done Done true
14592 4108 Acked 5107 Write 1 New Done Done true
14593 4109 Acked 5108 Write 5 New Done Done true
14594 4110 Acked 5109 Read 7 New Done Done true
14595 4111 Acked 5110 Write 10 New Done Done true
14596 4112 Acked 5111 Write 4 New Done Done true
14597 4113 Acked 5112 Write 10 New Done Done true
14598 4114 Acked 5113 Read 5 New Done Done true
14599 4115 Acked 5114 Write 6 New Done Done true
14600 4116 Acked 5115 Read 10 New Done Done true
14601 4117 Acked 5116 Write 7 New Done Done true
14602 4118 Acked 5117 Flush 0 New Done Done true
14603 4119 Acked 5118 Write 1 New Done Done true
14604 4120 Acked 5119 Read 6 New Done Done true
14605 4121 Acked 5120 Read 2 New Done Done true
14606 4122 Acked 5121 Write 2 New Done Done true
14607 4123 Acked 5122 Write 7 New Done Done true
14608 4124 Acked 5123 Read 8 New Done Done true
14609 4125 Acked 5124 Flush 0 New Done Done true
14610 4126 Acked 5125 Read 8 New Done Done true
14611 4127 Acked 5126 Write 4 New Done Done true
14612 4128 Acked 5127 Write 1 New Done Done true
14613 4129 Acked 5128 Read 1 New Done Done true
14614 4130 Acked 5129 Write 9 New Done Done true
14615 4131 Acked 5130 Flush 0 New Done Done true
14616 4132 Acked 5131 Write 5 New Done Done true
14617 4133 Acked 5132 Flush 0 New Done Done true
14618 4134 Acked 5133 Read 7 New Done Done true
14619 4135 Acked 5134 Read 7 New Done Done true
14620 4136 Acked 5135 Read 6 New Done Done true
14621 4137 Acked 5136 Read 4 New Done Done true
14622 4138 Acked 5137 Read 5 New Done Done true
14623 4139 Acked 5138 Read 7 New Done Done true
14624 4140 Acked 5139 Read 4 New Done Done true
14625 4141 Acked 5140 Read 8 New Done Done true
14626 4142 Acked 5141 Read 6 New Done Done true
14627 4143 Acked 5142 Read 2 New Done Done true
14628 4144 Acked 5143 Write 7 New Done Done true
14629 4145 Acked 5144 Write 10 New Done Done true
14630 4146 Acked 5145 Write 8 New Done Done true
14631 4147 Acked 5146 Read 10 New Done Done true
14632 4148 Acked 5147 Write 2 New Done Done true
14633 4149 Acked 5148 Read 3 New Done Done true
14634 4150 Acked 5149 Flush 0 New Done Done true
14635 4151 Acked 5150 Write 9 New Done Done true
14636 4152 Acked 5151 Write 10 New Done Done true
14637 4153 Acked 5152 Write 6 New Done Done true
14638 4154 Acked 5153 Flush 0 New Done Done true
14639 4155 Acked 5154 Read 4 New Done Done true
14640 4156 Acked 5155 Write 5 New Done Done true
14641 4157 Acked 5156 Write 4 New Done Done true
14642 4158 Acked 5157 Read 7 New Done Done true
14643 4159 Acked 5158 Write 10 New Done Done true
14644 4160 Acked 5159 Read 5 New Done Done true
14645 4161 Acked 5160 Read 5 New Done Done true
14646 4162 Acked 5161 Write 3 New Done Done true
14647 4163 Acked 5162 Read 4 New Done Done true
14648 4164 Acked 5163 Write 5 New Done Done true
14649 4165 Acked 5164 Read 3 New Done Done true
14650 4166 Acked 5165 Write 8 New Done Done true
14651 4167 Acked 5166 Read 8 New Done Done true
14652 4168 Acked 5167 Write 1 New Done Done true
14653 4169 Acked 5168 Read 10 New Done Done true
14654 4170 Acked 5169 Write 5 New Done Done true
14655 4171 Acked 5170 Read 10 New Done Done true
14656 4172 Acked 5171 Read 9 New Done Done true
14657 4173 Acked 5172 Write 6 New Done Done true
14658 4174 Acked 5173 Write 1 New Done Done true
14659 4175 Acked 5174 Read 10 New Done Done true
14660 4176 Acked 5175 Write 3 New Done Done true
14661 4177 Acked 5176 Read 9 New Done Done true
14662 4178 Acked 5177 Read 10 New Done Done true
14663 4179 Acked 5178 Read 6 New Done Done true
14664 4180 Acked 5179 Read 3 New Done Done true
14665 4181 Acked 5180 Write 4 New Done Done true
14666 4182 Acked 5181 Write 6 New Done Done true
14667 4183 Acked 5182 Write 8 New Done Done true
14668 4184 Acked 5183 Flush 0 New Done Done true
14669 4185 Acked 5184 Read 8 New Done Done true
14670 4186 Acked 5185 Write 5 New Done Done true
14671 4187 Acked 5186 Read 4 New Done Done true
14672 4188 Acked 5187 Read 1 New Done Done true
14673 4189 Acked 5188 Write 3 New Done Done true
14674 4190 Acked 5189 Read 3 New Done Done true
14675 4191 Acked 5190 Flush 0 New Done Done true
14676 4192 Acked 5191 Read 1 New Done Done true
14677 4193 Acked 5192 Write 3 New Done Done true
14678 4194 Acked 5193 Write 5 New Done Done true
14679 4195 Acked 5194 Read 3 New Done Done true
14680 4196 Acked 5195 Read 7 New Done Done true
14681 4197 Acked 5196 Read 5 New Done Done true
14682 4198 Acked 5197 Read 3 New Done Done true
14683 4199 Acked 5198 Read 5 New Done Done true
14684 4200 Acked 5199 Read 5 New Done Done true
14685 4201 Acked 5200 Write 7 New Done Done true
14686 4202 Acked 5201 Flush 0 New Done Done true
14687 4203 Acked 5202 Write 9 New Done Done true
14688 4204 Acked 5203 Read 10 New Done Done true
14689 4205 Acked 5204 Read 3 New Done Done true
14690 4206 Acked 5205 Write 2 New Done Done true
14691 4207 Acked 5206 Read 5 New Done Done true
14692 4208 Acked 5207 Write 8 New Done Done true
14693 4209 Acked 5208 Read 8 New Done Done true
14694 4210 Acked 5209 Write 4 New Done Done true
14695 4211 Acked 5210 Write 1 New Done Done true
14696 4212 Acked 5211 Read 3 New Done Done true
14697 4213 Acked 5212 Read 1 New Done Done true
14698 4214 Acked 5213 Write 5 New Done Done true
14699 4215 Acked 5214 Flush 0 New Done Done true
14700 4216 Acked 5215 Read 7 New Done Done true
14701 4217 Acked 5216 Write 7 New Done Done true
14702 4218 Acked 5217 Write 7 New Done Done true
14703 4219 Acked 5218 Flush 0 New Done Done true
14704 STATES DS:0 DS:1 DS:2 TOTAL
14705 New 301 0 0 301
14706 Sent 0 0 0 0
14707 Done 0 301 301 602
14708 Skipped 0 0 0 0
14709 Error 0 0 0 0
14710 Last Flush: 4917 5218 5218
14711 Downstairs last five completed: 4917 4916 4915 4914 4913
14712 Upstairs last five completed: 4219 4218 4217 4216 4215
14713 CLIENT: Up:0 ds:301 act:2
147142024-02-15T03:00:24.049ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
147152024-02-15T03:00:24.049ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
147162024-02-15T03:00:24.049ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
147172024-02-15T03:00:24.049ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
147182024-02-15T03:00:24.049ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
147192024-02-15T03:00:24.049ZINFOcrucible: [0] client re-new 301 jobs since flush 4917 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
147202024-02-15T03:00:24.050ZINFOcrucible: client task is exiting = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
147212024-02-15T03:00:25.051ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
147222024-02-15T03:00:25.051ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
147232024-02-15T03:00:25.051ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
147242024-02-15T03:00:25.051ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
147252024-02-15T03:00:25.051ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
147262024-02-15T03:00:25.051ZINFOcrucible: [0] client re-new 301 jobs since flush 4917 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
147272024-02-15T03:00:25.051ZINFOcrucible: client task is exiting = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
147282024-02-15T03:00:26.053ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
147292024-02-15T03:00:26.053ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
147302024-02-15T03:00:26.053ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
147312024-02-15T03:00:26.053ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
147322024-02-15T03:00:26.053ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
147332024-02-15T03:00:26.053ZINFOcrucible: [0] client re-new 301 jobs since flush 4917 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
147342024-02-15T03:00:26.053ZINFOcrucible: client task is exiting = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
147352024-02-15T03:00:27.055ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
147362024-02-15T03:00:27.055ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
147372024-02-15T03:00:27.055ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
147382024-02-15T03:00:27.055ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
147392024-02-15T03:00:27.055ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
147402024-02-15T03:00:27.055ZINFOcrucible: [0] client re-new 301 jobs since flush 4917 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
147412024-02-15T03:00:27.055ZINFOcrucible: client task is exiting = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
14742 ----------------------------------------------------------------
14743 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
14744 States: Offline Active Active
14745 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
14746 3919 Acked 4918 Read 2 New Done Done true
14747 3920 Acked 4919 Write 9 New Done Done true
14748 3921 Acked 4920 Flush 0 New Done Done true
14749 3922 Acked 4921 Read 9 New Done Done true
14750 3923 Acked 4922 Write 9 New Done Done true
14751 3924 Acked 4923 Read 7 New Done Done true
14752 3925 Acked 4924 Write 9 New Done Done true
14753 3926 Acked 4925 Read 7 New Done Done true
14754 3927 Acked 4926 Read 7 New Done Done true
14755 3928 Acked 4927 Write 7 New Done Done true
14756 3929 Acked 4928 Read 1 New Done Done true
14757 3930 Acked 4929 Read 5 New Done Done true
14758 3931 Acked 4930 Flush 0 New Done Done true
14759 3932 Acked 4931 Read 2 New Done Done true
14760 3933 Acked 4932 Write 3 New Done Done true
14761 3934 Acked 4933 Write 10 New Done Done true
14762 3935 Acked 4934 Write 8 New Done Done true
14763 3936 Acked 4935 Write 6 New Done Done true
14764 3937 Acked 4936 Write 5 New Done Done true
14765 3938 Acked 4937 Write 1 New Done Done true
14766 3939 Acked 4938 Write 4 New Done Done true
14767 3940 Acked 4939 Read 3 New Done Done true
14768 3941 Acked 4940 Write 1 New Done Done true
14769 3942 Acked 4941 Read 3 New Done Done true
14770 3943 Acked 4942 Write 4 New Done Done true
14771 3944 Acked 4943 Read 9 New Done Done true
14772 3945 Acked 4944 Read 7 New Done Done true
14773 3946 Acked 4945 Read 8 New Done Done true
14774 3947 Acked 4946 Write 5 New Done Done true
14775 3948 Acked 4947 Read 10 New Done Done true
14776 3949 Acked 4948 Read 8 New Done Done true
14777 3950 Acked 4949 Read 9 New Done Done true
14778 3951 Acked 4950 Write 10 New Done Done true
14779 3952 Acked 4951 Write 1 New Done Done true
14780 3953 Acked 4952 Flush 0 New Done Done true
14781 3954 Acked 4953 Read 9 New Done Done true
14782 3955 Acked 4954 Read 2 New Done Done true
14783 3956 Acked 4955 Read 9 New Done Done true
14784 3957 Acked 4956 Write 5 New Done Done true
14785 3958 Acked 4957 Write 9 New Done Done true
14786 3959 Acked 4958 Read 1 New Done Done true
14787 3960 Acked 4959 Read 1 New Done Done true
14788 3961 Acked 4960 Write 2 New Done Done true
14789 3962 Acked 4961 Read 5 New Done Done true
14790 3963 Acked 4962 Read 10 New Done Done true
14791 3964 Acked 4963 Read 2 New Done Done true
14792 3965 Acked 4964 Read 9 New Done Done true
14793 3966 Acked 4965 Read 3 New Done Done true
14794 3967 Acked 4966 Flush 0 New Done Done true
14795 3968 Acked 4967 Write 3 New Done Done true
14796 3969 Acked 4968 Read 1 New Done Done true
14797 3970 Acked 4969 Read 8 New Done Done true
14798 3971 Acked 4970 Read 5 New Done Done true
14799 3972 Acked 4971 Write 4 New Done Done true
14800 3973 Acked 4972 Read 9 New Done Done true
14801 3974 Acked 4973 Read 2 New Done Done true
14802 3975 Acked 4974 Read 7 New Done Done true
14803 3976 Acked 4975 Flush 0 New Done Done true
14804 3977 Acked 4976 Flush 0 New Done Done true
14805 3978 Acked 4977 Read 8 New Done Done true
14806 3979 Acked 4978 Write 2 New Done Done true
14807 3980 Acked 4979 Read 7 New Done Done true
14808 3981 Acked 4980 Write 10 New Done Done true
14809 3982 Acked 4981 Read 2 New Done Done true
14810 3983 Acked 4982 Write 6 New Done Done true
14811 3984 Acked 4983 Read 10 New Done Done true
14812 3985 Acked 4984 Write 3 New Done Done true
14813 3986 Acked 4985 Read 2 New Done Done true
14814 3987 Acked 4986 Read 8 New Done Done true
14815 3988 Acked 4987 Write 9 New Done Done true
14816 3989 Acked 4988 Read 1 New Done Done true
14817 3990 Acked 4989 Write 4 New Done Done true
14818 3991 Acked 4990 Read 2 New Done Done true
14819 3992 Acked 4991 Flush 0 New Done Done true
14820 3993 Acked 4992 Read 4 New Done Done true
14821 3994 Acked 4993 Read 3 New Done Done true
14822 3995 Acked 4994 Write 8 New Done Done true
14823 3996 Acked 4995 Write 5 New Done Done true
14824 3997 Acked 4996 Read 4 New Done Done true
14825 3998 Acked 4997 Write 3 New Done Done true
14826 3999 Acked 4998 Flush 0 New Done Done true
14827 4000 Acked 4999 Write 4 New Done Done true
14828 4001 Acked 5000 Read 7 New Done Done true
14829 4002 Acked 5001 Write 5 New Done Done true
14830 4003 Acked 5002 Write 1 New Done Done true
14831 4004 Acked 5003 Read 1 New Done Done true
14832 4005 Acked 5004 Write 7 New Done Done true
14833 4006 Acked 5005 Flush 0 New Done Done true
14834 4007 Acked 5006 Read 2 New Done Done true
14835 4008 Acked 5007 Read 7 New Done Done true
14836 4009 Acked 5008 Flush 0 New Done Done true
14837 4010 Acked 5009 Flush 0 New Done Done true
14838 4011 Acked 5010 Write 8 New Done Done true
14839 4012 Acked 5011 Write 3 New Done Done true
14840 4013 Acked 5012 Read 9 New Done Done true
14841 4014 Acked 5013 Flush 0 New Done Done true
14842 4015 Acked 5014 Write 9 New Done Done true
14843 4016 Acked 5015 Read 2 New Done Done true
14844 4017 Acked 5016 Flush 0 New Done Done true
14845 4018 Acked 5017 Write 4 New Done Done true
14846 4019 Acked 5018 Read 5 New Done Done true
14847 4020 Acked 5019 Read 2 New Done Done true
14848 4021 Acked 5020 Write 3 New Done Done true
14849 4022 Acked 5021 Flush 0 New Done Done true
14850 4023 Acked 5022 Read 9 New Done Done true
14851 4024 Acked 5023 Write 1 New Done Done true
14852 4025 Acked 5024 Flush 0 New Done Done true
14853 4026 Acked 5025 Write 5 New Done Done true
14854 4027 Acked 5026 Write 10 New Done Done true
14855 4028 Acked 5027 Write 1 New Done Done true
14856 4029 Acked 5028 Write 6 New Done Done true
14857 4030 Acked 5029 Read 8 New Done Done true
14858 4031 Acked 5030 Read 5 New Done Done true
14859 4032 Acked 5031 Read 9 New Done Done true
14860 4033 Acked 5032 Write 10 New Done Done true
14861 4034 Acked 5033 Write 8 New Done Done true
14862 4035 Acked 5034 Read 3 New Done Done true
14863 4036 Acked 5035 Read 9 New Done Done true
14864 4037 Acked 5036 Read 6 New Done Done true
14865 4038 Acked 5037 Read 3 New Done Done true
14866 4039 Acked 5038 Write 5 New Done Done true
14867 4040 Acked 5039 Write 10 New Done Done true
14868 4041 Acked 5040 Flush 0 New Done Done true
14869 4042 Acked 5041 Read 1 New Done Done true
14870 4043 Acked 5042 Read 5 New Done Done true
14871 4044 Acked 5043 Write 8 New Done Done true
14872 4045 Acked 5044 Write 9 New Done Done true
14873 4046 Acked 5045 Write 10 New Done Done true
14874 4047 Acked 5046 Read 6 New Done Done true
14875 4048 Acked 5047 Write 9 New Done Done true
14876 4049 Acked 5048 Write 8 New Done Done true
14877 4050 Acked 5049 Read 3 New Done Done true
14878 4051 Acked 5050 Write 7 New Done Done true
14879 4052 Acked 5051 Write 4 New Done Done true
14880 4053 Acked 5052 Read 3 New Done Done true
14881 4054 Acked 5053 Read 5 New Done Done true
14882 4055 Acked 5054 Flush 0 New Done Done true
14883 4056 Acked 5055 Write 1 New Done Done true
14884 4057 Acked 5056 Write 2 New Done Done true
14885 4058 Acked 5057 Write 4 New Done Done true
14886 4059 Acked 5058 Flush 0 New Done Done true
14887 4060 Acked 5059 Read 7 New Done Done true
14888 4061 Acked 5060 Read 4 New Done Done true
14889 4062 Acked 5061 Read 10 New Done Done true
14890 4063 Acked 5062 Flush 0 New Done Done true
14891 4064 Acked 5063 Write 3 New Done Done true
14892 4065 Acked 5064 Read 5 New Done Done true
14893 4066 Acked 5065 Read 3 New Done Done true
14894 4067 Acked 5066 Read 1 New Done Done true
14895 4068 Acked 5067 Flush 0 New Done Done true
14896 4069 Acked 5068 Write 3 New Done Done true
14897 4070 Acked 5069 Write 2 New Done Done true
14898 4071 Acked 5070 Read 9 New Done Done true
14899 4072 Acked 5071 Read 7 New Done Done true
14900 4073 Acked 5072 Read 9 New Done Done true
14901 4074 Acked 5073 Write 6 New Done Done true
14902 4075 Acked 5074 Write 10 New Done Done true
14903 4076 Acked 5075 Write 9 New Done Done true
14904 4077 Acked 5076 Flush 0 New Done Done true
14905 4078 Acked 5077 Read 8 New Done Done true
14906 4079 Acked 5078 Write 1 New Done Done true
14907 4080 Acked 5079 Write 4 New Done Done true
14908 4081 Acked 5080 Flush 0 New Done Done true
14909 4082 Acked 5081 Write 2 New Done Done true
14910 4083 Acked 5082 Write 1 New Done Done true
14911 4084 Acked 5083 Read 9 New Done Done true
14912 4085 Acked 5084 Flush 0 New Done Done true
14913 4086 Acked 5085 Write 5 New Done Done true
14914 4087 Acked 5086 Write 1 New Done Done true
14915 4088 Acked 5087 Write 3 New Done Done true
14916 4089 Acked 5088 Read 8 New Done Done true
14917 4090 Acked 5089 Write 2 New Done Done true
14918 4091 Acked 5090 Read 3 New Done Done true
14919 4092 Acked 5091 Read 6 New Done Done true
14920 4093 Acked 5092 Read 10 New Done Done true
14921 4094 Acked 5093 Flush 0 New Done Done true
14922 4095 Acked 5094 Write 5 New Done Done true
14923 4096 Acked 5095 Read 9 New Done Done true
14924 4097 Acked 5096 Read 1 New Done Done true
14925 4098 Acked 5097 Read 8 New Done Done true
14926 4099 Acked 5098 Read 7 New Done Done true
14927 4100 Acked 5099 Write 9 New Done Done true
14928 4101 Acked 5100 Read 9 New Done Done true
14929 4102 Acked 5101 Write 6 New Done Done true
14930 4103 Acked 5102 Read 1 New Done Done true
14931 4104 Acked 5103 Flush 0 New Done Done true
14932 4105 Acked 5104 Write 8 New Done Done true
14933 4106 Acked 5105 Read 5 New Done Done true
14934 4107 Acked 5106 Read 8 New Done Done true
14935 4108 Acked 5107 Write 1 New Done Done true
14936 4109 Acked 5108 Write 5 New Done Done true
14937 4110 Acked 5109 Read 7 New Done Done true
14938 4111 Acked 5110 Write 10 New Done Done true
14939 4112 Acked 5111 Write 4 New Done Done true
14940 4113 Acked 5112 Write 10 New Done Done true
14941 4114 Acked 5113 Read 5 New Done Done true
14942 4115 Acked 5114 Write 6 New Done Done true
14943 4116 Acked 5115 Read 10 New Done Done true
14944 4117 Acked 5116 Write 7 New Done Done true
14945 4118 Acked 5117 Flush 0 New Done Done true
14946 4119 Acked 5118 Write 1 New Done Done true
14947 4120 Acked 5119 Read 6 New Done Done true
14948 4121 Acked 5120 Read 2 New Done Done true
14949 4122 Acked 5121 Write 2 New Done Done true
14950 4123 Acked 5122 Write 7 New Done Done true
14951 4124 Acked 5123 Read 8 New Done Done true
14952 4125 Acked 5124 Flush 0 New Done Done true
14953 4126 Acked 5125 Read 8 New Done Done true
14954 4127 Acked 5126 Write 4 New Done Done true
14955 4128 Acked 5127 Write 1 New Done Done true
14956 4129 Acked 5128 Read 1 New Done Done true
14957 4130 Acked 5129 Write 9 New Done Done true
14958 4131 Acked 5130 Flush 0 New Done Done true
14959 4132 Acked 5131 Write 5 New Done Done true
14960 4133 Acked 5132 Flush 0 New Done Done true
14961 4134 Acked 5133 Read 7 New Done Done true
14962 4135 Acked 5134 Read 7 New Done Done true
14963 4136 Acked 5135 Read 6 New Done Done true
14964 4137 Acked 5136 Read 4 New Done Done true
14965 4138 Acked 5137 Read 5 New Done Done true
14966 4139 Acked 5138 Read 7 New Done Done true
14967 4140 Acked 5139 Read 4 New Done Done true
14968 4141 Acked 5140 Read 8 New Done Done true
14969 4142 Acked 5141 Read 6 New Done Done true
14970 4143 Acked 5142 Read 2 New Done Done true
14971 4144 Acked 5143 Write 7 New Done Done true
14972 4145 Acked 5144 Write 10 New Done Done true
14973 4146 Acked 5145 Write 8 New Done Done true
14974 4147 Acked 5146 Read 10 New Done Done true
14975 4148 Acked 5147 Write 2 New Done Done true
14976 4149 Acked 5148 Read 3 New Done Done true
14977 4150 Acked 5149 Flush 0 New Done Done true
14978 4151 Acked 5150 Write 9 New Done Done true
14979 4152 Acked 5151 Write 10 New Done Done true
14980 4153 Acked 5152 Write 6 New Done Done true
14981 4154 Acked 5153 Flush 0 New Done Done true
14982 4155 Acked 5154 Read 4 New Done Done true
14983 4156 Acked 5155 Write 5 New Done Done true
14984 4157 Acked 5156 Write 4 New Done Done true
14985 4158 Acked 5157 Read 7 New Done Done true
14986 4159 Acked 5158 Write 10 New Done Done true
14987 4160 Acked 5159 Read 5 New Done Done true
14988 4161 Acked 5160 Read 5 New Done Done true
14989 4162 Acked 5161 Write 3 New Done Done true
14990 4163 Acked 5162 Read 4 New Done Done true
14991 4164 Acked 5163 Write 5 New Done Done true
14992 4165 Acked 5164 Read 3 New Done Done true
14993 4166 Acked 5165 Write 8 New Done Done true
14994 4167 Acked 5166 Read 8 New Done Done true
14995 4168 Acked 5167 Write 1 New Done Done true
14996 4169 Acked 5168 Read 10 New Done Done true
14997 4170 Acked 5169 Write 5 New Done Done true
14998 4171 Acked 5170 Read 10 New Done Done true
14999 4172 Acked 5171 Read 9 New Done Done true
15000 4173 Acked 5172 Write 6 New Done Done true
15001 4174 Acked 5173 Write 1 New Done Done true
15002 4175 Acked 5174 Read 10 New Done Done true
15003 4176 Acked 5175 Write 3 New Done Done true
15004 4177 Acked 5176 Read 9 New Done Done true
15005 4178 Acked 5177 Read 10 New Done Done true
15006 4179 Acked 5178 Read 6 New Done Done true
15007 4180 Acked 5179 Read 3 New Done Done true
15008 4181 Acked 5180 Write 4 New Done Done true
15009 4182 Acked 5181 Write 6 New Done Done true
15010 4183 Acked 5182 Write 8 New Done Done true
15011 4184 Acked 5183 Flush 0 New Done Done true
15012 4185 Acked 5184 Read 8 New Done Done true
15013 4186 Acked 5185 Write 5 New Done Done true
15014 4187 Acked 5186 Read 4 New Done Done true
15015 4188 Acked 5187 Read 1 New Done Done true
15016 4189 Acked 5188 Write 3 New Done Done true
15017 4190 Acked 5189 Read 3 New Done Done true
15018 4191 Acked 5190 Flush 0 New Done Done true
15019 4192 Acked 5191 Read 1 New Done Done true
15020 4193 Acked 5192 Write 3 New Done Done true
15021 4194 Acked 5193 Write 5 New Done Done true
15022 4195 Acked 5194 Read 3 New Done Done true
15023 4196 Acked 5195 Read 7 New Done Done true
15024 4197 Acked 5196 Read 5 New Done Done true
15025 4198 Acked 5197 Read 3 New Done Done true
15026 4199 Acked 5198 Read 5 New Done Done true
15027 4200 Acked 5199 Read 5 New Done Done true
15028 4201 Acked 5200 Write 7 New Done Done true
15029 4202 Acked 5201 Flush 0 New Done Done true
15030 4203 Acked 5202 Write 9 New Done Done true
15031 4204 Acked 5203 Read 10 New Done Done true
15032 4205 Acked 5204 Read 3 New Done Done true
15033 4206 Acked 5205 Write 2 New Done Done true
15034 4207 Acked 5206 Read 5 New Done Done true
15035 4208 Acked 5207 Write 8 New Done Done true
15036 4209 Acked 5208 Read 8 New Done Done true
15037 4210 Acked 5209 Write 4 New Done Done true
15038 4211 Acked 5210 Write 1 New Done Done true
15039 4212 Acked 5211 Read 3 New Done Done true
15040 4213 Acked 5212 Read 1 New Done Done true
15041 4214 Acked 5213 Write 5 New Done Done true
15042 4215 Acked 5214 Flush 0 New Done Done true
15043 4216 Acked 5215 Read 7 New Done Done true
15044 4217 Acked 5216 Write 7 New Done Done true
15045 4218 Acked 5217 Write 7 New Done Done true
15046 4219 Acked 5218 Flush 0 New Done Done true
15047 STATES DS:0 DS:1 DS:2 TOTAL
15048 New 301 0 0 301
15049 Sent 0 0 0 0
15050 Done 0 301 301 602
15051 Skipped 0 0 0 0
15052 Error 0 0 0 0
15053 Last Flush: 4917 5218 5218
15054 Downstairs last five completed: 4917 4916 4915 4914 4913
15055 Upstairs last five completed: 4219 4218 4217 4216 4215
15056 CLIENT: Up:0 ds:301 act:2
150572024-02-15T03:00:28.057ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
150582024-02-15T03:00:28.057ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
150592024-02-15T03:00:28.057ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
150602024-02-15T03:00:28.057ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
150612024-02-15T03:00:28.057ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
150622024-02-15T03:00:28.057ZINFOcrucible: [0] client re-new 301 jobs since flush 4917 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
150632024-02-15T03:00:28.057ZINFOcrucible: client task is exiting = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
150642024-02-15T03:00:29.058ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
150652024-02-15T03:00:29.058ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
150662024-02-15T03:00:29.058ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
150672024-02-15T03:00:29.058ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
150682024-02-15T03:00:29.058ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
150692024-02-15T03:00:29.058ZINFOcrucible: [0] client re-new 301 jobs since flush 4917 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
150702024-02-15T03:00:29.058ZINFOcrucible: client task is exiting = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
150712024-02-15T03:00:30.060ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
150722024-02-15T03:00:30.060ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
150732024-02-15T03:00:30.060ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
150742024-02-15T03:00:30.060ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
150752024-02-15T03:00:30.060ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
150762024-02-15T03:00:30.060ZINFOcrucible: [0] client re-new 301 jobs since flush 4917 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
150772024-02-15T03:00:30.060ZINFOcrucible: client task is exiting = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
150782024-02-15T03:00:31.061ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
150792024-02-15T03:00:31.061ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
150802024-02-15T03:00:31.061ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
150812024-02-15T03:00:31.061ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
150822024-02-15T03:00:31.061ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
150832024-02-15T03:00:31.061ZINFOcrucible: [0] client re-new 301 jobs since flush 4917 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
150842024-02-15T03:00:31.061ZINFOcrucible: client task is exiting = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
15085 ----------------------------------------------------------------
15086 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
15087 States: Offline Active Active
15088 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
15089 3919 Acked 4918 Read 2 New Done Done true
15090 3920 Acked 4919 Write 9 New Done Done true
15091 3921 Acked 4920 Flush 0 New Done Done true
15092 3922 Acked 4921 Read 9 New Done Done true
15093 3923 Acked 4922 Write 9 New Done Done true
15094 3924 Acked 4923 Read 7 New Done Done true
15095 3925 Acked 4924 Write 9 New Done Done true
15096 3926 Acked 4925 Read 7 New Done Done true
15097 3927 Acked 4926 Read 7 New Done Done true
15098 3928 Acked 4927 Write 7 New Done Done true
15099 3929 Acked 4928 Read 1 New Done Done true
15100 3930 Acked 4929 Read 5 New Done Done true
15101 3931 Acked 4930 Flush 0 New Done Done true
15102 3932 Acked 4931 Read 2 New Done Done true
15103 3933 Acked 4932 Write 3 New Done Done true
15104 3934 Acked 4933 Write 10 New Done Done true
15105 3935 Acked 4934 Write 8 New Done Done true
15106 3936 Acked 4935 Write 6 New Done Done true
15107 3937 Acked 4936 Write 5 New Done Done true
15108 3938 Acked 4937 Write 1 New Done Done true
15109 3939 Acked 4938 Write 4 New Done Done true
15110 3940 Acked 4939 Read 3 New Done Done true
15111 3941 Acked 4940 Write 1 New Done Done true
15112 3942 Acked 4941 Read 3 New Done Done true
15113 3943 Acked 4942 Write 4 New Done Done true
15114 3944 Acked 4943 Read 9 New Done Done true
15115 3945 Acked 4944 Read 7 New Done Done true
15116 3946 Acked 4945 Read 8 New Done Done true
15117 3947 Acked 4946 Write 5 New Done Done true
15118 3948 Acked 4947 Read 10 New Done Done true
15119 3949 Acked 4948 Read 8 New Done Done true
15120 3950 Acked 4949 Read 9 New Done Done true
15121 3951 Acked 4950 Write 10 New Done Done true
15122 3952 Acked 4951 Write 1 New Done Done true
15123 3953 Acked 4952 Flush 0 New Done Done true
15124 3954 Acked 4953 Read 9 New Done Done true
15125 3955 Acked 4954 Read 2 New Done Done true
15126 3956 Acked 4955 Read 9 New Done Done true
15127 3957 Acked 4956 Write 5 New Done Done true
15128 3958 Acked 4957 Write 9 New Done Done true
15129 3959 Acked 4958 Read 1 New Done Done true
15130 3960 Acked 4959 Read 1 New Done Done true
15131 3961 Acked 4960 Write 2 New Done Done true
15132 3962 Acked 4961 Read 5 New Done Done true
15133 3963 Acked 4962 Read 10 New Done Done true
15134 3964 Acked 4963 Read 2 New Done Done true
15135 3965 Acked 4964 Read 9 New Done Done true
15136 3966 Acked 4965 Read 3 New Done Done true
15137 3967 Acked 4966 Flush 0 New Done Done true
15138 3968 Acked 4967 Write 3 New Done Done true
15139 3969 Acked 4968 Read 1 New Done Done true
15140 3970 Acked 4969 Read 8 New Done Done true
15141 3971 Acked 4970 Read 5 New Done Done true
15142 3972 Acked 4971 Write 4 New Done Done true
15143 3973 Acked 4972 Read 9 New Done Done true
15144 3974 Acked 4973 Read 2 New Done Done true
15145 3975 Acked 4974 Read 7 New Done Done true
15146 3976 Acked 4975 Flush 0 New Done Done true
15147 3977 Acked 4976 Flush 0 New Done Done true
15148 3978 Acked 4977 Read 8 New Done Done true
15149 3979 Acked 4978 Write 2 New Done Done true
15150 3980 Acked 4979 Read 7 New Done Done true
15151 3981 Acked 4980 Write 10 New Done Done true
15152 3982 Acked 4981 Read 2 New Done Done true
15153 3983 Acked 4982 Write 6 New Done Done true
15154 3984 Acked 4983 Read 10 New Done Done true
15155 3985 Acked 4984 Write 3 New Done Done true
15156 3986 Acked 4985 Read 2 New Done Done true
15157 3987 Acked 4986 Read 8 New Done Done true
15158 3988 Acked 4987 Write 9 New Done Done true
15159 3989 Acked 4988 Read 1 New Done Done true
15160 3990 Acked 4989 Write 4 New Done Done true
15161 3991 Acked 4990 Read 2 New Done Done true
15162 3992 Acked 4991 Flush 0 New Done Done true
15163 3993 Acked 4992 Read 4 New Done Done true
15164 3994 Acked 4993 Read 3 New Done Done true
15165 3995 Acked 4994 Write 8 New Done Done true
15166 3996 Acked 4995 Write 5 New Done Done true
15167 3997 Acked 4996 Read 4 New Done Done true
15168 3998 Acked 4997 Write 3 New Done Done true
15169 3999 Acked 4998 Flush 0 New Done Done true
15170 4000 Acked 4999 Write 4 New Done Done true
15171 4001 Acked 5000 Read 7 New Done Done true
15172 4002 Acked 5001 Write 5 New Done Done true
15173 4003 Acked 5002 Write 1 New Done Done true
15174 4004 Acked 5003 Read 1 New Done Done true
15175 4005 Acked 5004 Write 7 New Done Done true
15176 4006 Acked 5005 Flush 0 New Done Done true
15177 4007 Acked 5006 Read 2 New Done Done true
15178 4008 Acked 5007 Read 7 New Done Done true
15179 4009 Acked 5008 Flush 0 New Done Done true
15180 4010 Acked 5009 Flush 0 New Done Done true
15181 4011 Acked 5010 Write 8 New Done Done true
15182 4012 Acked 5011 Write 3 New Done Done true
15183 4013 Acked 5012 Read 9 New Done Done true
15184 4014 Acked 5013 Flush 0 New Done Done true
15185 4015 Acked 5014 Write 9 New Done Done true
15186 4016 Acked 5015 Read 2 New Done Done true
15187 4017 Acked 5016 Flush 0 New Done Done true
15188 4018 Acked 5017 Write 4 New Done Done true
15189 4019 Acked 5018 Read 5 New Done Done true
15190 4020 Acked 5019 Read 2 New Done Done true
15191 4021 Acked 5020 Write 3 New Done Done true
15192 4022 Acked 5021 Flush 0 New Done Done true
15193 4023 Acked 5022 Read 9 New Done Done true
15194 4024 Acked 5023 Write 1 New Done Done true
15195 4025 Acked 5024 Flush 0 New Done Done true
15196 4026 Acked 5025 Write 5 New Done Done true
15197 4027 Acked 5026 Write 10 New Done Done true
15198 4028 Acked 5027 Write 1 New Done Done true
15199 4029 Acked 5028 Write 6 New Done Done true
15200 4030 Acked 5029 Read 8 New Done Done true
15201 4031 Acked 5030 Read 5 New Done Done true
15202 4032 Acked 5031 Read 9 New Done Done true
15203 4033 Acked 5032 Write 10 New Done Done true
15204 4034 Acked 5033 Write 8 New Done Done true
15205 4035 Acked 5034 Read 3 New Done Done true
15206 4036 Acked 5035 Read 9 New Done Done true
15207 4037 Acked 5036 Read 6 New Done Done true
15208 4038 Acked 5037 Read 3 New Done Done true
15209 4039 Acked 5038 Write 5 New Done Done true
15210 4040 Acked 5039 Write 10 New Done Done true
15211 4041 Acked 5040 Flush 0 New Done Done true
15212 4042 Acked 5041 Read 1 New Done Done true
15213 4043 Acked 5042 Read 5 New Done Done true
15214 4044 Acked 5043 Write 8 New Done Done true
15215 4045 Acked 5044 Write 9 New Done Done true
15216 4046 Acked 5045 Write 10 New Done Done true
15217 4047 Acked 5046 Read 6 New Done Done true
15218 4048 Acked 5047 Write 9 New Done Done true
15219 4049 Acked 5048 Write 8 New Done Done true
15220 4050 Acked 5049 Read 3 New Done Done true
15221 4051 Acked 5050 Write 7 New Done Done true
15222 4052 Acked 5051 Write 4 New Done Done true
15223 4053 Acked 5052 Read 3 New Done Done true
15224 4054 Acked 5053 Read 5 New Done Done true
15225 4055 Acked 5054 Flush 0 New Done Done true
15226 4056 Acked 5055 Write 1 New Done Done true
15227 4057 Acked 5056 Write 2 New Done Done true
15228 4058 Acked 5057 Write 4 New Done Done true
15229 4059 Acked 5058 Flush 0 New Done Done true
15230 4060 Acked 5059 Read 7 New Done Done true
15231 4061 Acked 5060 Read 4 New Done Done true
15232 4062 Acked 5061 Read 10 New Done Done true
15233 4063 Acked 5062 Flush 0 New Done Done true
15234 4064 Acked 5063 Write 3 New Done Done true
15235 4065 Acked 5064 Read 5 New Done Done true
15236 4066 Acked 5065 Read 3 New Done Done true
15237 4067 Acked 5066 Read 1 New Done Done true
15238 4068 Acked 5067 Flush 0 New Done Done true
15239 4069 Acked 5068 Write 3 New Done Done true
15240 4070 Acked 5069 Write 2 New Done Done true
15241 4071 Acked 5070 Read 9 New Done Done true
15242 4072 Acked 5071 Read 7 New Done Done true
15243 4073 Acked 5072 Read 9 New Done Done true
15244 4074 Acked 5073 Write 6 New Done Done true
15245 4075 Acked 5074 Write 10 New Done Done true
15246 4076 Acked 5075 Write 9 New Done Done true
15247 4077 Acked 5076 Flush 0 New Done Done true
15248 4078 Acked 5077 Read 8 New Done Done true
15249 4079 Acked 5078 Write 1 New Done Done true
15250 4080 Acked 5079 Write 4 New Done Done true
15251 4081 Acked 5080 Flush 0 New Done Done true
15252 4082 Acked 5081 Write 2 New Done Done true
15253 4083 Acked 5082 Write 1 New Done Done true
15254 4084 Acked 5083 Read 9 New Done Done true
15255 4085 Acked 5084 Flush 0 New Done Done true
15256 4086 Acked 5085 Write 5 New Done Done true
15257 4087 Acked 5086 Write 1 New Done Done true
15258 4088 Acked 5087 Write 3 New Done Done true
15259 4089 Acked 5088 Read 8 New Done Done true
15260 4090 Acked 5089 Write 2 New Done Done true
15261 4091 Acked 5090 Read 3 New Done Done true
15262 4092 Acked 5091 Read 6 New Done Done true
15263 4093 Acked 5092 Read 10 New Done Done true
15264 4094 Acked 5093 Flush 0 New Done Done true
15265 4095 Acked 5094 Write 5 New Done Done true
15266 4096 Acked 5095 Read 9 New Done Done true
15267 4097 Acked 5096 Read 1 New Done Done true
15268 4098 Acked 5097 Read 8 New Done Done true
15269 4099 Acked 5098 Read 7 New Done Done true
15270 4100 Acked 5099 Write 9 New Done Done true
15271 4101 Acked 5100 Read 9 New Done Done true
15272 4102 Acked 5101 Write 6 New Done Done true
15273 4103 Acked 5102 Read 1 New Done Done true
15274 4104 Acked 5103 Flush 0 New Done Done true
15275 4105 Acked 5104 Write 8 New Done Done true
15276 4106 Acked 5105 Read 5 New Done Done true
15277 4107 Acked 5106 Read 8 New Done Done true
15278 4108 Acked 5107 Write 1 New Done Done true
15279 4109 Acked 5108 Write 5 New Done Done true
15280 4110 Acked 5109 Read 7 New Done Done true
15281 4111 Acked 5110 Write 10 New Done Done true
15282 4112 Acked 5111 Write 4 New Done Done true
15283 4113 Acked 5112 Write 10 New Done Done true
15284 4114 Acked 5113 Read 5 New Done Done true
15285 4115 Acked 5114 Write 6 New Done Done true
15286 4116 Acked 5115 Read 10 New Done Done true
15287 4117 Acked 5116 Write 7 New Done Done true
15288 4118 Acked 5117 Flush 0 New Done Done true
15289 4119 Acked 5118 Write 1 New Done Done true
15290 4120 Acked 5119 Read 6 New Done Done true
15291 4121 Acked 5120 Read 2 New Done Done true
15292 4122 Acked 5121 Write 2 New Done Done true
15293 4123 Acked 5122 Write 7 New Done Done true
15294 4124 Acked 5123 Read 8 New Done Done true
15295 4125 Acked 5124 Flush 0 New Done Done true
15296 4126 Acked 5125 Read 8 New Done Done true
15297 4127 Acked 5126 Write 4 New Done Done true
15298 4128 Acked 5127 Write 1 New Done Done true
15299 4129 Acked 5128 Read 1 New Done Done true
15300 4130 Acked 5129 Write 9 New Done Done true
15301 4131 Acked 5130 Flush 0 New Done Done true
15302 4132 Acked 5131 Write 5 New Done Done true
15303 4133 Acked 5132 Flush 0 New Done Done true
15304 4134 Acked 5133 Read 7 New Done Done true
15305 4135 Acked 5134 Read 7 New Done Done true
15306 4136 Acked 5135 Read 6 New Done Done true
15307 4137 Acked 5136 Read 4 New Done Done true
15308 4138 Acked 5137 Read 5 New Done Done true
15309 4139 Acked 5138 Read 7 New Done Done true
15310 4140 Acked 5139 Read 4 New Done Done true
15311 4141 Acked 5140 Read 8 New Done Done true
15312 4142 Acked 5141 Read 6 New Done Done true
15313 4143 Acked 5142 Read 2 New Done Done true
15314 4144 Acked 5143 Write 7 New Done Done true
15315 4145 Acked 5144 Write 10 New Done Done true
15316 4146 Acked 5145 Write 8 New Done Done true
15317 4147 Acked 5146 Read 10 New Done Done true
15318 4148 Acked 5147 Write 2 New Done Done true
15319 4149 Acked 5148 Read 3 New Done Done true
15320 4150 Acked 5149 Flush 0 New Done Done true
15321 4151 Acked 5150 Write 9 New Done Done true
15322 4152 Acked 5151 Write 10 New Done Done true
15323 4153 Acked 5152 Write 6 New Done Done true
15324 4154 Acked 5153 Flush 0 New Done Done true
15325 4155 Acked 5154 Read 4 New Done Done true
15326 4156 Acked 5155 Write 5 New Done Done true
15327 4157 Acked 5156 Write 4 New Done Done true
15328 4158 Acked 5157 Read 7 New Done Done true
15329 4159 Acked 5158 Write 10 New Done Done true
15330 4160 Acked 5159 Read 5 New Done Done true
15331 4161 Acked 5160 Read 5 New Done Done true
15332 4162 Acked 5161 Write 3 New Done Done true
15333 4163 Acked 5162 Read 4 New Done Done true
15334 4164 Acked 5163 Write 5 New Done Done true
15335 4165 Acked 5164 Read 3 New Done Done true
15336 4166 Acked 5165 Write 8 New Done Done true
15337 4167 Acked 5166 Read 8 New Done Done true
15338 4168 Acked 5167 Write 1 New Done Done true
15339 4169 Acked 5168 Read 10 New Done Done true
15340 4170 Acked 5169 Write 5 New Done Done true
15341 4171 Acked 5170 Read 10 New Done Done true
15342 4172 Acked 5171 Read 9 New Done Done true
15343 4173 Acked 5172 Write 6 New Done Done true
15344 4174 Acked 5173 Write 1 New Done Done true
15345 4175 Acked 5174 Read 10 New Done Done true
15346 4176 Acked 5175 Write 3 New Done Done true
15347 4177 Acked 5176 Read 9 New Done Done true
15348 4178 Acked 5177 Read 10 New Done Done true
15349 4179 Acked 5178 Read 6 New Done Done true
15350 4180 Acked 5179 Read 3 New Done Done true
15351 4181 Acked 5180 Write 4 New Done Done true
15352 4182 Acked 5181 Write 6 New Done Done true
15353 4183 Acked 5182 Write 8 New Done Done true
15354 4184 Acked 5183 Flush 0 New Done Done true
15355 4185 Acked 5184 Read 8 New Done Done true
15356 4186 Acked 5185 Write 5 New Done Done true
15357 4187 Acked 5186 Read 4 New Done Done true
15358 4188 Acked 5187 Read 1 New Done Done true
15359 4189 Acked 5188 Write 3 New Done Done true
15360 4190 Acked 5189 Read 3 New Done Done true
15361 4191 Acked 5190 Flush 0 New Done Done true
15362 4192 Acked 5191 Read 1 New Done Done true
15363 4193 Acked 5192 Write 3 New Done Done true
15364 4194 Acked 5193 Write 5 New Done Done true
15365 4195 Acked 5194 Read 3 New Done Done true
15366 4196 Acked 5195 Read 7 New Done Done true
15367 4197 Acked 5196 Read 5 New Done Done true
15368 4198 Acked 5197 Read 3 New Done Done true
15369 4199 Acked 5198 Read 5 New Done Done true
15370 4200 Acked 5199 Read 5 New Done Done true
15371 4201 Acked 5200 Write 7 New Done Done true
15372 4202 Acked 5201 Flush 0 New Done Done true
15373 4203 Acked 5202 Write 9 New Done Done true
15374 4204 Acked 5203 Read 10 New Done Done true
15375 4205 Acked 5204 Read 3 New Done Done true
15376 4206 Acked 5205 Write 2 New Done Done true
15377 4207 Acked 5206 Read 5 New Done Done true
15378 4208 Acked 5207 Write 8 New Done Done true
15379 4209 Acked 5208 Read 8 New Done Done true
15380 4210 Acked 5209 Write 4 New Done Done true
15381 4211 Acked 5210 Write 1 New Done Done true
15382 4212 Acked 5211 Read 3 New Done Done true
15383 4213 Acked 5212 Read 1 New Done Done true
15384 4214 Acked 5213 Write 5 New Done Done true
15385 4215 Acked 5214 Flush 0 New Done Done true
15386 4216 Acked 5215 Read 7 New Done Done true
15387 4217 Acked 5216 Write 7 New Done Done true
15388 4218 Acked 5217 Write 7 New Done Done true
15389 4219 Acked 5218 Flush 0 New Done Done true
15390 STATES DS:0 DS:1 DS:2 TOTAL
15391 New 301 0 0 301
15392 Sent 0 0 0 0
15393 Done 0 301 301 602
15394 Skipped 0 0 0 0
15395 Error 0 0 0 0
15396 Last Flush: 4917 5218 5218
15397 Downstairs last five completed: 4917 4916 4915 4914 4913
15398 Upstairs last five completed: 4219 4218 4217 4216 4215
15399 CLIENT: Up:0 ds:301 act:2
154002024-02-15T03:00:32.063ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
154012024-02-15T03:00:32.063ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
154022024-02-15T03:00:32.063ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
154032024-02-15T03:00:32.063ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
154042024-02-15T03:00:32.063ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
154052024-02-15T03:00:32.063ZINFOcrucible: [0] client re-new 301 jobs since flush 4917 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
154062024-02-15T03:00:32.063ZINFOcrucible: client task is exiting = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
15407 [8810] got start action
15408 [8810] I am going to restart
15409 Starting downstairs at port 8810
15410 Make output file at "/tmp/dsc/downstairs-8810.txt"
15411 [8810][0] reports Starting
15412 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1181)
15413 [8810][0] reports Running
154142024-02-15T03:00:33.064ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
154152024-02-15T03:00:33.064ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
154162024-02-15T03:00:33.064ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
154172024-02-15T03:00:33.064ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
154182024-02-15T03:00:33.064ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
154192024-02-15T03:00:33.064ZINFOcrucible: [0] client re-new 301 jobs since flush 4917 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
154202024-02-15T03:00:33.064ZINFOcrucible: client task is exiting = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
154212024-02-15T03:00:34.065ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
154222024-02-15T03:00:34.066ZINFOcrucible: ds_connection connected = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
154232024-02-15T03:00:34.067ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
154242024-02-15T03:00:34.067ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
154252024-02-15T03:00:34.067ZINFOcrucible: send last flush ID to this DS: 4917 = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
154262024-02-15T03:00:34.067ZINFOcrucible: Replied this last flush ID: 4917 = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
154272024-02-15T03:00:34.067ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
154282024-02-15T03:00:34.067ZINFOcrucible: [0] Transition from Offline to Replay = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
154292024-02-15T03:00:34.067ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
154302024-02-15T03:00:34.067ZINFOcrucible: [0] Transition from Replay to Active = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
15431 ----------------------------------------------------------------
15432 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
15433 Downstairs last five completed: 5218 5217 5216 5215 5214
15434 Upstairs last five completed: 4219 4218 4217 4216 4215
15435 CLIENT: Up:0 ds:0 act:3
15436 Replay: All jobs finished, all DS active.
15437 Main task has work to do, go find it
15438 got dsc Stop(2)
15439 stop 2
15440 Feb 15 03:00:35.385 INFO request completed, latency_us: 329, response_code: 204, uri: /stop/cid/2, [8830] Got stop action so:false kr:false
15441 method: POST, req_id: bd69ab61-3b25-40d7-85fd-0a3f7c443df0, remote_addr: 127.0.0.1:54744, local_addr: 127.0.0.1:9998
154422024-02-15T03:00:35.386ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
154432024-02-15T03:00:35.386ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
154442024-02-15T03:00:35.386ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
154452024-02-15T03:00:35.386ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
154462024-02-15T03:00:35.386ZINFOcrucible: [2] client re-new 0 jobs since flush 5218 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
15447 Feb 15 03:00:35.386{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-15T03:00:35.38637889Z","hostname":"ip-10-150-1-67.us-west-2.compute.internal","pid":1167,"":"io task","client":"2","":"downstairs","session_id":"4f5a63e5-aa00-4890-9f6b-96c2e069fdb2"}
15448 INFO request completed, latency_us: 142, response_code: 200, uri: /state/cid/2, method: GET, req_id: f7c3c84c-0bad-41e7-8bb8-fa90a1e2ea00, remote_addr: 127.0.0.1:54744, local_addr: 127.0.0.1:9998
15449 [8830] Exited with: ExitStatus(unix_wait_status(9))
15450 [8830][2] reports Exit
154512024-02-15T03:00:36.388ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
154522024-02-15T03:00:36.388ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
154532024-02-15T03:00:36.388ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
154542024-02-15T03:00:36.388ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
154552024-02-15T03:00:36.388ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
154562024-02-15T03:00:36.388ZINFOcrucible: [2] client re-new 0 jobs since flush 5218 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
154572024-02-15T03:00:36.388ZINFOcrucible: client task is exiting = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
154582024-02-15T03:00:37.389ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
154592024-02-15T03:00:37.389ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
154602024-02-15T03:00:37.389ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
154612024-02-15T03:00:37.389ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
154622024-02-15T03:00:37.389ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
154632024-02-15T03:00:37.390ZINFOcrucible: [2] client re-new 0 jobs since flush 5218 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
154642024-02-15T03:00:37.390ZINFOcrucible: client task is exiting = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
154652024-02-15T03:00:38.391ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
154662024-02-15T03:00:38.391ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
154672024-02-15T03:00:38.391ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
154682024-02-15T03:00:38.391ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
154692024-02-15T03:00:38.391ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
154702024-02-15T03:00:38.391ZINFOcrucible: [2] client re-new 0 jobs since flush 5218 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
154712024-02-15T03:00:38.391ZINFOcrucible: client task is exiting = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
15472 Feb 15 03:00:39.388 INFO request completed, latency_us: 203, response_code: 200, uri: /state/cid/2, method: GET, req_id: 81f72a38-9c6a-49df-b97e-176d7ce89955, remote_addr: 127.0.0.1:54744, local_addr: 127.0.0.1:9998
15473 001/300 Write block 3240 len 20480 data: 5 5 3 3 4
15474 002/300 Read block 4511 len 4096
15475 003/300 Write block 2534 len 32768 data: 3 4 3 3 3 3 2 2
15476 {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30004/300 Write block 253 len 8192 data: 4 4
15477 ,"time":"2024-02-15T03:00:39.391990952Z","hostname":"ip-10-150-1-67.us-west-2.compute.internal","pid":1167,"":"io task","client":"2","":"downstairs","session_id":"4f5a63e5-aa00-4890-9f6b-96c2e069fdb2"}
154782024-02-15T03:00:39.392ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
154792024-02-15T03:00:39.392ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
154802024-02-15T03:00:39.392ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
154812024-02-15T03:00:39.392ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
154822024-02-15T03:00:39.392ZINFOcrucible: [2] client re-new 3 jobs since flush 5218 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
154832024-02-15T03:00:39.392ZINFOcrucible: client task is exiting = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
15484 005/300 Write block 3783 len 16384 data: 6 5 5 5
15485 006/300 Flush
15486 007/300 Write block 1804 len 12288 data: 4 4 4
15487 008/300 Write block 4020 len 36864 data: 6 5 5 5 5 6 6 5 7
15488 009/300 Read block 1360 len 20480
15489 010/300 Write block 4856 len 40960 data: 2 2 3 3 4 4 5 6 6 6
15490 011/300 Write block 4434 len 12288 data: 6 5 5
15491 012/300 Write block 1520 len 8192 data: 3 3
15492 013/300 Flush
15493 014/300 Read block 4436 len 4096
15494 015/300 Read block 3413 len 40960
15495 016/300 Flush
15496 017/300 Read block 2988 len 24576
15497 018/300 Read block 769 len 8192
15498 019/300 Read block 2970 len 20480
15499 020/300 Read block 330 len 24576
15500 021/300 Read block 576 len 12288
15501 022/300 Read block 2711 len 16384
15502 023/300 Read block 2859 len 36864
15503 024/300 Read block 3049 len 32768
15504 025/300 Write block 799 len 12288 data: 5 3 3
15505 026/300 Write block 1921 len 40960 data: 2 2 2 2 2 2 2 3 2 2
15506 027/300 Write block 3107 len 40960 data: 5 6 7 7 8 7 6 6 5 4
15507 028/300 Flush
15508 029/300 Read block 1408 len 24576
15509 030/300 Read block 3005 len 4096
15510 031/300 Flush
15511 032/300 Read block 260 len 32768
15512 033/300 Write block 680 len 20480 data: 3 3 4 4 3
15513 034/300 Write block 2345 len 28672 data: 5 5 4 2 2 3 3
15514 035/300 Read block 4150 len 40960
15515 036/300 Write block 600 len 40960 data: 3 5 5 5 5 5 5 4 3 2
15516 037/300 Read block 4359 len 36864
15517 038/300 Read block 4347 len 24576
15518 039/300 Read block 2390 len 40960
15519 040/300 Flush
15520 041/300 Write block 3326 len 40960 data: 5 5 4 4 3 3 3 3 4 4
15521 042/300 Write block 886 len 16384 data: 4 5 6 6
15522 043/300 Read block 2537 len 20480
15523 044/300 Read block 1338 len 24576
15524 045/300 Read block 4022 len 20480
15525 046/300 Read block 3604 len 12288
15526 047/300 Read block 1976 len 12288
15527 048/300 Flush
15528 049/300 Read block 2731 len 32768
15529 050/300 Write block 3050 len 28672 data: 5 6 7 6 6 6 6
15530 051/300 Write block 1586 len 40960 data: 4 5 5 4 5 3 4 4 3 3
15531 052/300 Read block 1616 len 4096
15532 053/300 Write block 2316 len 8192 data: 3 3
15533 054/300 Read block 4349 len 40960
15534 055/300 Write block 2002 len 12288 data: 7 7 6
15535 056/300 Read block 424 len 32768
15536 057/300 Read block 1463 len 4096
15537 058/300 Read block 3891 len 36864
15538 059/300 Read block 2818 len 20480
15539 060/300 Write block 4719 len 40960 data: 3 3 3 3 5 4 5 5 4 4
15540 061/300 Read block 3348 len 20480
15541 062/300 Read block 354 len 16384
15542 063/300 Read block 3822 len 12288
15543 064/300 Read block 251 len 16384
15544 065/300 Read block 1170 len 28672
15545 066/300 Write block 2081 len 28672 data: 4 5 6 6 6 4 4
15546 067/300 Read block 284 len 32768
15547 068/300 Read block 337 len 20480
15548 069/300 Write block 1128 len 28672 data: 3 4 3 3 3 3 3
15549 070/300 Read block 4119 len 28672
15550 071/300 Read block 4757 len 8192
15551 072/300 Read block 3516 len 28672
15552 073/300 Write block 2003 len 4096 data: 8
15553 074/300 Read block 2513 len 24576
15554 075/300 Read block 487 len 8192
15555 076/300 Write block 3082 len 8192 data: 4 5
15556 077/300 Read block 1005 len 32768
15557 078/300 Read block 3236 len 12288
15558 079/300 Read block 2851 len 4096
15559 080/300 Read block 4157 len 12288
15560 081/300 Write block 183 len 12288 data: 3 2 3
15561 082/300 Flush
15562 083/300 Write block 4243 len 8192 data: 3 3
15563 084/300 Write block 4672 len 32768 data: 5 5 7 9 8 5 7 7
15564 085/300 Write block 3341 len 40960 data: 4 3 3 3 4 5 7 6 6 7
15565 086/300 Write block 34 len 8192 data: 3 3
15566 087/300 Write block 2913 len 36864 data: 4 3 4 3 2 2 2 2 2
15567 088/300 Write block 4341 len 24576 data: 2 2 3 4 4 5
15568 089/300 Read block 3922 len 32768
15569 090/300 Read block 1110 len 36864
15570 091/300 Write block 3368 len 24576 data: 3 2 2 3 3 3
15571 092/300 Write block 1405 len 12288 data: 6 7 7
15572 093/300 Read block 4572 len 36864
15573 094/300 Write block 4346 len 24576 data: 6 6 6 4 4 4
15574 095/300 Read block 350 len 16384
15575 096/300 Read block 1224 len 8192
15576 097/300 Flush
15577 098/300 Write block 3299 len 16384 data: 3 3 2 2
15578 099/300 Write block 2492 len 32768 data: 5 4 3 3 3 3 3 4
15579 100/300 Flush
15580 101/300 Write block 2251 len 36864 data: 3 3 3 3 4 4 5 4 4
15581 102/300 Write block 623 len 8192 data: 3 3
15582 103/300 Write block 598 len 16384 data: 3 3 4 6
15583 104/300 Read block 1910 len 32768
15584 105/300 Write block 4994 len 4096 data: 3
15585 106/300 Read block 792 len 32768
15586 107/300 Write block 2856 len 20480 data: 4 4 4 4 4
15587 108/300 Write block 3987 len 4096 data: 5
15588 109/300 Flush
15589 110/300 Read block 469 len 40960
15590 111/300 Write block 4930 len 16384 data: 3 5 6 6
15591 112/300 Read block 2565 len 40960
15592 113/300 Read block 4566 len 16384
15593 114/300 Read block 4622 len 24576
15594 115/300 Write block 3941 len 40960 data: 2 2 2 2 3 3 3 4 4 4
15595 116/300 Read block 2103 len 20480
15596 117/300 Read block 757 len 16384
15597 118/300 Read block 851 len 40960
15598 119/300 Read block 2328 len 40960
15599 120/300 Write block 4621 len 16384 data: 4 4 4 5
15600 121/300 Read block 3281 len 8192
15601 122/300 Read block 1853 len 4096
15602 123/300 Read block 3740 len 24576
15603 124/300 Read block 3620 len 12288
15604 125/300 Write block 4927 len 24576 data: 3 3 3 4 6 7
15605 126/300 Write block 1587 len 4096 data: 6
15606 127/300 Read block 4161 len 12288
15607 128/300 Flush
15608 129/300 Write block 3444 len 28672 data: 2 2 2 3 4 4 4
15609 130/300 Write block 422 len 4096 data: 2
15610 131/300 Read block 1361 len 12288
15611 132/300 Read block 599 len 12288
15612 133/300 Read block 2344 len 4096
15613 134/300 Read block 608 len 8192
15614 135/300 Read block 1849 len 36864
15615 136/300 Read block 3202 len 16384
15616 137/300 Write block 2068 len 8192 data: 3 3
15617 138/300 Read block 4739 len 8192
15618 139/300 Write block 2172 len 28672 data: 4 3 3 3 2 2 2
15619 140/300 Read block 3443 len 32768
15620 141/300 Read block 2153 len 4096
15621 142/300 Read block 4760 len 36864
15622 143/300 Read block 1464 len 28672
15623 144/300 Read block 4257 len 32768
15624 145/300 Write block 1001 len 20480 data: 4 4 4 4 5
15625 146/300 Flush
15626 147/300 Write block 1917 len 8192 data: 4 3
15627 148/300 Read block 1870 len 28672
15628 149/300 Read block 552 len 8192
15629 150/300 Write block 2641 len 40960 data: 5 6 6 6 6 6 6 5 3 2
15630 151/300 Write block 2236 len 20480 data: 5 5 4 5 5
15631 152/300 Write block 4825 len 8192 data: 3 3
15632 153/300 Read block 2128 len 24576
15633 154/300 Write block 1543 len 16384 data: 3 4 4 4
15634 155/300 Flush
15635 156/300 Read block 399 len 12288
15636 157/300 Read block 3578 len 20480
15637 158/300 Read block 2473 len 8192
15638 159/300 Read block 2658 len 20480
15639 160/300 Read block 4205 len 16384
15640 161/300 Write block 2106 len 24576 data: 3 2 2 2 2 2
15641 162/300 Read block 3227 len 40960
15642 163/300 Read block 3337 len 16384
15643 164/300 Read block 2614 len 12288
15644 165/300 Read block 1271 len 12288
15645 166/300 Write block 4003 len 28672 data: 5 4 3 2 2 2 2
15646 167/300 Read block 2523 len 28672
15647 168/300 Read block 1851 len 36864
15648 169/300 Read block 3043 len 28672
15649 170/300 Read block 434 len 36864
15650 171/300 Flush
15651 172/300 Write block 4391 len 20480 data: 3 3 2 2 2
15652 173/300 Read block 4302 len 28672
15653 174/300 Write block 1255 len 24576 data: 6 6 6 5 5 5
15654 175/300 Write block 1011 len 28672 data: 4 5 4 4 3 3 4
15655 176/300 Read block 737 len 28672
15656 177/300 Write block 3450 len 28672 data: 5 4 3 3 3 3 3
15657 178/300 Read block 1563 len 40960
15658 179/300 Read block 1747 len 12288
15659 180/300 Read block 4655 len 36864
15660 181/300 Write block 454 len 20480 data: 5 5 5 5 4
15661 182/300 Write block 2066 len 32768 data: 4 4 4 4 3 3 4 4
15662 183/300 Read block 2727 len 4096
15663 184/300 Read block 1383 len 40960
15664 185/300 Read block 4247 len 24576
15665 186/300 Write block 4778 len 24576 data: 5 7 7 6 5 4
15666 187/300 Write block 4357 len 40960 data: 2 3 3 2 2 3 2 2 2 2
15667 188/300 Read block 2956 len 32768
15668 189/300 Write block 2736 len 40960 data: 3 3 3 4 5 6 5 5 5 6
15669 190/300 Write block 3314 len 12288 data: 2 2 2
15670 191/300 Write block 3895 len 36864 data: 2 2 2 2 2 2 2 3 3
15671 192/300 Read block 2404 len 24576
15672 193/300 Write block 2543 len 16384 data: 5 5 5 6
15673 194/300 Write block 4263 len 36864 data: 5 3 3 3 3 4 4 4 2
15674 195/300 Write block 2903 len 32768 data: 3 3 3 2 2 3 3 4
15675 196/300 Write block 4947 len 32768 data: 4 5 5 5 5 5 6 5
15676 197/300 Read block 2838 len 16384
15677 198/300 Flush
15678 199/300 Write block 4179 len 4096 data: 3
15679 200/300 Write block 2512 len 16384 data: 3 3 4 4
15680 201/300 Write block 1655 len 28672 data: 5 3 3 3 3 4 3
15681 202/300 Write block 2808 len 4096 data: 2
15682 203/300 Write block 2536 len 8192 data: 4 4
15683 204/300 Write block 3522 len 20480 data: 5 5 5 4 5
15684 205/300 Read block 3351 len 16384
15685 206/300 Read block 3893 len 16384
15686 207/300 Read block 2637 len 40960
15687 208/300 Read block 1372 len 32768
15688 209/300 Flush
15689 210/300 Read block 4553 len 8192
15690 211/300 Flush
15691 212/300 Read block 719 len 32768
15692 213/300 Write block 2239 len 12288 data: 6 6 5
15693 214/300 Read block 648 len 8192
15694 215/300 Write block 1544 len 28672 data: 5 5 5 4 4 3 3
15695 216/300 Read block 532 len 28672
15696 217/300 Flush
15697 218/300 Read block 864 len 40960
15698 219/300 Write block 951 len 4096 data: 3
15699 220/300 Read block 3198 len 36864
15700 221/300 Write block 4263 len 24576 data: 6 4 4 4 4 5
15701 222/300 Write block 643 len 8192 data: 3 3
15702 223/300 Read block 4478 len 32768
15703 224/300 Read block 1953 len 12288
15704 225/300 Write block 1773 len 4096 data: 3
15705 226/300 Read block 1687 len 32768
15706 227/300 Read block 3306 len 40960
15707 228/300 Read block 3058 len 16384
15708 229/300 Write block 1400 len 4096 data: 3
15709 230/300 Write block 4579 len 36864 data: 2 2 2 2 2 2 2 3 2
15710 231/300 Read block 3379 len 36864
15711 232/300 Flush
15712 233/300 Read block 1751 len 4096
15713 234/300 Write block 3058 len 16384 data: 5 5 4 4
15714 235/300 Write block 4050 len 28672 data: 5 5 5 5 5 5 4
15715 236/300 Write block 582 len 20480 data: 5 5 7 6 5
15716 237/300 Read block 2379 len 12288
15717 238/300 Read block 4526 len 4096
15718 239/300 Read block 4320 len 8192
15719 240/300 Read block 3867 len 32768
15720 241/300 Flush
15721 242/300 Write block 4148 len 12288 data: 5 3 3
15722 243/300 Write block 3392 len 28672 data: 2 2 2 3 3 4 5
15723 244/300 Read block 4350 len 28672
15724 245/300 Flush
15725 246/300 Read block 1027 len 32768
15726 247/300 Read block 2593 len 24576
15727 248/300 Flush
15728 249/300 Flush
15729 250/300 Write block 828 len 8192 data: 3 2
15730 251/300 Read block 3792 len 16384
15731 252/300 Read block 176 len 8192
15732 253/300 Write block 3011 len 12288 data: 3 2 2
15733 254/300 Write block 4257 len 24576 data: 3 3 5 5 6 6
15734 255/300 Read block 4028 len 24576
15735 256/300 Write block 4920 len 36864 data: 2 2 3 3 3 3 3 4 4
15736 257/300 Read block 2677 len 4096
15737 258/300 Write block 2090 len 4096 data: 2
15738 259/300 Read block 3961 len 36864
15739 260/300 Read block 2866 len 16384
15740 261/300 Read block 1909 len 36864
15741 262/300 Read block 608 len 4096
15742 263/300 Read block 552 len 40960
15743 264/300 Read block 3362 len 36864
15744 265/300 Write block 182 len 40960 data: 4 4 3 4 3 3 4 4 3 4
15745 266/300 Read block 4209 len 24576
15746 267/300 Read block 2823 len 8192
15747 268/300 Read block 643 len 20480
15748 269/300 Write block 520 len 36864 data: 4 4 3 4 4 3 3 3 3
15749 270/300 Write block 3853 len 32768 data: 3 3 3 4 5 5 5 6
15750 271/300 Read block 2534 len 4096
15751 272/300 Read block 705 len 24576
15752 273/300 Write block 2714 len 24576 data: 3 3 3 3 3 3
15753 274/300 Read block 1520 len 12288
15754 275/300 Read block 4923 len 12288
15755 276/300 Write block 982 len 24576 data: 4 4 3 3 3 3
15756 277/300 Read block 3953 len 24576
15757 278/300 Read block 344 len 36864
15758 279/300 Read block 503 len 32768
15759 280/300 Write block 1504 len 20480 data: 4 3 5 4 5
15760 281/300 Write block 4709 len 8192 data: 3 2
15761 282/300 Write block 1686 len 24576 data: 3 3 3 3 3 4
15762 283/300 Read block 59 len 32768
15763 284/300 Write block 3239 len 28672 data: 3 6 6 4 4 5 4
15764 285/300 Read block 745 len 4096
15765 286/300 Flush
15766 287/300 Write block 2388 len 12288 data: 2 2 2
15767 288/300 Read block 115 len 32768
15768 289/300 Flush
15769 290/300 Flush
15770 291/300 Write block 2277 len 36864 data: 5 5 5 3 3 2 2 2 2
15771 292/300 Write block 999 len 20480 data: 6 5 5 5 5
15772 293/300 Flush
15773 294/300 Read block 2697 len 4096
15774 295/300 Write block 1185 len 36864 data: 3 3 2 6 6 5 6 6 5
15775 296/300 Write block 1938 len 12288 data: 2 3 3
15776 297/300 Write block 2334 len 28672 data: 2 3 2 2 2 2 3
15777 298/300 Read block 3713 len 16384
15778 299/300 Write block 2864 len 20480 data: 5 4 5 5 4
15779 300/300 Write block 2560 len 20480 data: 4 4 4 4 4
15780 Main task has work to do, go find it
15781 got dsc Start(2)
15782 start 2
15783 Feb 15 03:00:39.715 INFO request completed, latency_us: 211, response_code: 204, uri: /start/cid/2, method: POST, req_id: c07701f4-6e08-461b-b15c-a41c1c1ce418, remote_addr: 127.0.0.1:54744, local_addr: 127.0.0.1:9998
15784 Replay: started 2, returned:Ok(())
15785 ----------------------------------------------------------------
15786 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
15787 States: Active Active Offline
15788 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
15789 4220 Acked 5219 Write 5 Done Done New true
15790 4221 Acked 5220 Read 1 Done Done New true
15791 4222 Acked 5221 Write 8 Done Done New true
15792 4223 Acked 5222 Write 2 Done Done New false
15793 4224 Acked 5223 Write 4 Done Done New false
15794 4225 Acked 5224 Flush 0 Done Done New false
15795 4226 Acked 5225 Write 3 Done Done New false
15796 4227 Acked 5226 Write 9 Done Done New false
15797 4228 Acked 5227 Read 5 Done Done New false
15798 4229 Acked 5228 Write 10 Done Done New false
15799 4230 Acked 5229 Write 3 Done Done New false
15800 4231 Acked 5230 Write 2 Done Done New false
15801 4232 Acked 5231 Flush 0 Done Done New false
15802 4233 Acked 5232 Read 1 Done Done New false
15803 4234 Acked 5233 Read 10 Done Done New false
15804 4235 Acked 5234 Flush 0 Done Done New false
15805 4236 Acked 5235 Read 6 Done Done New false
15806 4237 Acked 5236 Read 2 Done Done New false
15807 4238 Acked 5237 Read 5 Done Done New false
15808 4239 Acked 5238 Read 6 Done Done New false
15809 4240 Acked 5239 Read 3 Done Done New false
15810 4241 Acked 5240 Read 4 Done Done New false
15811 4242 Acked 5241 Read 9 Done Done New false
15812 4243 Acked 5242 Read 8 Done Done New false
15813 4244 Acked 5243 Write 3 Done Done New false
15814 4245 Acked 5244 Write 10 Done Done New false
15815 4246 Acked 5245 Write 10 Done Done New false
15816 4247 Acked 5246 Flush 0 Done Done New false
15817 4248 Acked 5247 Read 6 Done Done New false
15818 4249 Acked 5248 Read 1 Done Done New false
15819 4250 Acked 5249 Flush 0 Done Done New false
15820 4251 Acked 5250 Read 8 Done Done New false
15821 4252 Acked 5251 Write 5 Done Done New false
15822 4253 Acked 5252 Write 7 Done Done New false
15823 4254 Acked 5253 Read 10 Done Done New false
15824 4255 Acked 5254 Flush 0 Done Done New false
15825 4256 Acked 5255 Write 10 Done Done New false
15826 4257 Acked 5256 Read 9 Done Done New false
15827 4258 Acked 5257 Read 6 Done Done New false
15828 4259 Acked 5258 Read 10 Done Done New false
15829 4260 Acked 5259 Flush 0 Done Done New false
15830 4261 Acked 5260 Write 10 Done Done New false
15831 4262 Acked 5261 Write 4 Done Done New false
15832 4263 Acked 5262 Read 5 Done Done New false
15833 4264 Acked 5263 Read 6 Done Done New false
15834 4265 Acked 5264 Read 5 Done Done New false
15835 4266 Acked 5265 Read 3 Done Done New false
15836 4267 Acked 5266 Read 3 Done Done New false
15837 4268 Acked 5267 Flush 0 Done Done New false
15838 4269 Acked 5268 Read 8 Done Done New false
15839 4270 Acked 5269 Write 7 Done Done New false
15840 4271 Acked 5270 Write 10 Done Done New false
15841 4272 Acked 5271 Read 1 Done Done New false
15842 4273 Acked 5272 Write 2 Done Done New false
15843 4274 Acked 5273 Read 10 Done Done New false
15844 4275 Acked 5274 Write 3 Done Done New false
15845 4276 Acked 5275 Read 8 Done Done New false
15846 4277 Acked 5276 Read 1 Done Done New false
15847 4278 Acked 5277 Read 9 Done Done New false
15848 4279 Acked 5278 Read 5 Done Done New false
15849 4280 Acked 5279 Write 10 Done Done New false
15850 4281 Acked 5280 Read 5 Done Done New false
15851 4282 Acked 5281 Read 4 Done Done New false
15852 4283 Acked 5282 Read 3 Done Done New false
15853 4284 Acked 5283 Read 4 Done Done New false
15854 4285 Acked 5284 Read 7 Done Done New false
15855 4286 Acked 5285 Write 7 Done Done New false
15856 4287 Acked 5286 Read 8 Done Done New false
15857 4288 Acked 5287 Read 5 Done Done New false
15858 4289 Acked 5288 Write 7 Done Done New false
15859 4290 Acked 5289 Read 7 Done Done New false
15860 4291 Acked 5290 Read 2 Done Done New false
15861 4292 Acked 5291 Read 7 Done Done New false
15862 4293 Acked 5292 Write 1 Done Done New false
15863 4294 Acked 5293 Read 6 Done Done New false
15864 4295 Acked 5294 Read 2 Done Done New false
15865 4296 Acked 5295 Write 2 Done Done New false
15866 4297 Acked 5296 Read 8 Done Done New false
15867 4298 Acked 5297 Read 3 Done Done New false
15868 4299 Acked 5298 Read 1 Done Done New false
15869 4300 Acked 5299 Read 3 Done Done New false
15870 4301 Acked 5300 Write 3 Done Done New false
15871 4302 Acked 5301 Flush 0 Done Done New false
15872 4303 Acked 5302 Write 2 Done Done New false
15873 4304 Acked 5303 Write 8 Done Done New false
15874 4305 Acked 5304 Write 10 Done Done New false
15875 4306 Acked 5305 Write 2 Done Done New false
15876 4307 Acked 5306 Write 9 Done Done New false
15877 4308 Acked 5307 Write 6 Done Done New false
15878 4309 Acked 5308 Read 8 Done Done New false
15879 4310 Acked 5309 Read 9 Done Done New false
15880 4311 Acked 5310 Write 6 Done Done New false
15881 4312 Acked 5311 Write 3 Done Done New false
15882 4313 Acked 5312 Read 9 Done Done New false
15883 4314 Acked 5313 Write 6 Done Done New false
15884 4315 Acked 5314 Read 4 Done Done New false
15885 4316 Acked 5315 Read 2 Done Done New false
15886 4317 Acked 5316 Flush 0 Done Done New false
15887 4318 Acked 5317 Write 4 Done Done New false
15888 4319 Acked 5318 Write 8 Done Done New false
15889 4320 Acked 5319 Flush 0 Done Done New false
15890 4321 Acked 5320 Write 9 Done Done New false
15891 4322 Acked 5321 Write 2 Done Done New false
15892 4323 Acked 5322 Write 4 Done Done New false
15893 4324 Acked 5323 Read 8 Done Done New false
15894 4325 Acked 5324 Write 1 Done Done New false
15895 4326 Acked 5325 Read 8 Done Done New false
15896 4327 Acked 5326 Write 5 Done Done New false
15897 4328 Acked 5327 Write 1 Done Done New false
15898 4329 Acked 5328 Flush 0 Done Done New false
15899 4330 Acked 5329 Read 10 Done Done New false
15900 4331 Acked 5330 Write 4 Done Done New false
15901 4332 Acked 5331 Read 10 Done Done New false
15902 4333 Acked 5332 Read 4 Done Done New false
15903 4334 Acked 5333 Read 6 Done Done New false
15904 4335 Acked 5334 Write 10 Done Done New false
15905 4336 Acked 5335 Read 5 Done Done New false
15906 4337 Acked 5336 Read 4 Done Done New false
15907 4338 Acked 5337 Read 10 Done Done New false
15908 4339 Acked 5338 Read 10 Done Done New false
15909 4340 Acked 5339 Write 4 Done Done New false
15910 4341 Acked 5340 Read 2 Done Done New false
15911 4342 Acked 5341 Read 1 Done Done New false
15912 4343 Acked 5342 Read 6 Done Done New false
15913 4344 Acked 5343 Read 3 Done Done New false
15914 4345 Acked 5344 Write 6 Done Done New false
15915 4346 Acked 5345 Write 1 Done Done New false
15916 4347 Acked 5346 Read 3 Done Done New false
15917 4348 Acked 5347 Flush 0 Done Done New false
15918 4349 Acked 5348 Write 7 Done Done New false
15919 4350 Acked 5349 Write 1 Done Done New false
15920 4351 Acked 5350 Read 3 Done Done New false
15921 4352 Acked 5351 Read 3 Done Done New false
15922 4353 Acked 5352 Read 1 Done Done New false
15923 4354 Acked 5353 Read 2 Done Done New false
15924 4355 Acked 5354 Read 9 Done Done New false
15925 4356 Acked 5355 Read 4 Done Done New false
15926 4357 Acked 5356 Write 2 Done Done New false
15927 4358 Acked 5357 Read 2 Done Done New false
15928 4359 Acked 5358 Write 7 Done Done New false
15929 4360 Acked 5359 Read 8 Done Done New false
15930 4361 Acked 5360 Read 1 Done Done New false
15931 4362 Acked 5361 Read 9 Done Done New false
15932 4363 Acked 5362 Read 7 Done Done New false
15933 4364 Acked 5363 Read 8 Done Done New false
15934 4365 Acked 5364 Write 5 Done Done New false
15935 4366 Acked 5365 Flush 0 Done Done New false
15936 4367 Acked 5366 Write 2 Done Done New false
15937 4368 Acked 5367 Read 7 Done Done New false
15938 4369 Acked 5368 Read 2 Done Done New false
15939 4370 Acked 5369 Write 10 Done Done New false
15940 4371 Acked 5370 Write 5 Done Done New false
15941 4372 Acked 5371 Write 2 Done Done New false
15942 4373 Acked 5372 Read 6 Done Done New false
15943 4374 Acked 5373 Write 4 Done Done New false
15944 4375 Acked 5374 Flush 0 Done Done New false
15945 4376 Acked 5375 Read 3 Done Done New false
15946 4377 Acked 5376 Read 5 Done Done New false
15947 4378 Acked 5377 Read 2 Done Done New false
15948 4379 Acked 5378 Read 5 Done Done New false
15949 4380 Acked 5379 Read 4 Done Done New false
15950 4381 Acked 5380 Write 6 Done Done New false
15951 4382 Acked 5381 Read 10 Done Done New false
15952 4383 Acked 5382 Read 4 Done Done New false
15953 4384 Acked 5383 Read 3 Done Done New false
15954 4385 Acked 5384 Read 3 Done Done New false
15955 4386 Acked 5385 Write 7 Done Done New false
15956 4387 Acked 5386 Read 7 Done Done New false
15957 4388 Acked 5387 Read 9 Done Done New false
15958 4389 Acked 5388 Read 7 Done Done New false
15959 4390 Acked 5389 Read 9 Done Done New false
15960 4391 Acked 5390 Flush 0 Done Done New false
15961 4392 Acked 5391 Write 5 Done Done New false
15962 4393 Acked 5392 Read 7 Done Done New false
15963 4394 Acked 5393 Write 6 Done Done New false
15964 4395 Acked 5394 Write 7 Done Done New false
15965 4396 Acked 5395 Read 7 Done Done New false
15966 4397 Acked 5396 Write 7 Done Done New false
15967 4398 Acked 5397 Read 10 Done Done New false
15968 4399 Acked 5398 Read 3 Done Done New false
15969 4400 Acked 5399 Read 9 Done Done New false
15970 4401 Acked 5400 Write 5 Done Done New false
15971 4402 Acked 5401 Write 8 Done Done New false
15972 4403 Acked 5402 Read 1 Done Done New false
15973 4404 Acked 5403 Read 10 Done Done New false
15974 4405 Acked 5404 Read 6 Done Done New false
15975 4406 Acked 5405 Write 6 Done Done New false
15976 4407 Acked 5406 Write 10 Done Done New false
15977 4408 Acked 5407 Read 8 Done Done New false
15978 4409 Acked 5408 Write 10 Done Done New false
15979 4410 Acked 5409 Write 3 Done Done New false
15980 4411 Acked 5410 Write 9 Done Done New false
15981 4412 Acked 5411 Read 6 Done Done New false
15982 4413 Acked 5412 Write 4 Done Done New false
15983 4414 Acked 5413 Write 9 Done Done New false
15984 4415 Acked 5414 Write 8 Done Done New false
15985 4416 Acked 5415 Write 8 Done Done New false
15986 4417 Acked 5416 Read 4 Done Done New false
15987 4418 Acked 5417 Flush 0 Done Done New false
15988 4419 Acked 5418 Write 1 Done Done New false
15989 4420 Acked 5419 Write 4 Done Done New false
15990 4421 Acked 5420 Write 7 Done Done New false
15991 4422 Acked 5421 Write 1 Done Done New false
15992 4423 Acked 5422 Write 2 Done Done New false
15993 4424 Acked 5423 Write 5 Done Done New false
15994 4425 Acked 5424 Read 4 Done Done New false
15995 4426 Acked 5425 Read 4 Done Done New false
15996 4427 Acked 5426 Read 10 Done Done New false
15997 4428 Acked 5427 Read 8 Done Done New false
15998 4429 Acked 5428 Flush 0 Done Done New false
15999 4430 Acked 5429 Read 2 Done Done New false
16000 4431 Acked 5430 Flush 0 Done Done New false
16001 4432 Acked 5431 Read 8 Done Done New false
16002 4433 Acked 5432 Write 3 Done Done New false
16003 4434 Acked 5433 Read 2 Done Done New false
16004 4435 Acked 5434 Write 7 Done Done New false
16005 4436 Acked 5435 Read 7 Done Done New false
16006 4437 Acked 5436 Flush 0 Done Done New false
16007 4438 Acked 5437 Read 10 Done Done New false
16008 4439 Acked 5438 Write 1 Done Done New false
16009 4440 Acked 5439 Read 9 Done Done New false
16010 4441 Acked 5440 Write 6 Done Done New false
16011 4442 Acked 5441 Write 2 Done Done New false
16012 4443 Acked 5442 Read 8 Done Done New false
16013 4444 Acked 5443 Read 3 Done Done New false
16014 4445 Acked 5444 Write 1 Done Done New false
16015 4446 Acked 5445 Read 8 Done Done New false
16016 4447 Acked 5446 Read 10 Done Done New false
16017 4448 Acked 5447 Read 4 Done Done New false
16018 4449 Acked 5448 Write 1 Done Done New false
16019 4450 Acked 5449 Write 9 Done Done New false
16020 4451 Acked 5450 Read 9 Done Done New false
16021 4452 Acked 5451 Flush 0 Done Done New false
16022 4453 Acked 5452 Read 1 Done Done New false
16023 4454 Acked 5453 Write 4 Done Done New false
16024 4455 Acked 5454 Write 7 Done Done New false
16025 4456 Acked 5455 Write 5 Done Done New false
16026 4457 Acked 5456 Read 3 Done Done New false
16027 4458 Acked 5457 Read 1 Done Done New false
16028 4459 Acked 5458 Read 2 Done Done New false
16029 4460 Acked 5459 Read 8 Done Done New false
16030 4461 Acked 5460 Flush 0 Done Done New false
16031 4462 Acked 5461 Write 3 Done Done New false
16032 4463 Acked 5462 Write 7 Done Done New false
16033 4464 Acked 5463 Read 7 Done Done New false
16034 4465 Acked 5464 Flush 0 Done Done New false
16035 4466 Acked 5465 Read 8 Done Done New false
16036 4467 Acked 5466 Read 6 Done Done New false
16037 4468 Acked 5467 Flush 0 Done Done New false
16038 4469 Acked 5468 Flush 0 Done Done New false
16039 4470 Acked 5469 Write 2 Done Done New false
16040 4471 Acked 5470 Read 4 Done Done New false
16041 4472 Acked 5471 Read 2 Done Done New false
16042 4473 Acked 5472 Write 3 Done Done New false
16043 4474 Acked 5473 Write 6 Done Done New false
16044 4475 Acked 5474 Read 6 Done Done New false
16045 4476 Acked 5475 Write 9 Done Done New false
16046 4477 Acked 5476 Read 1 Done Done New false
16047 4478 Acked 5477 Write 1 Done Done New false
16048 4479 Acked 5478 Read 9 Done Done New false
16049 4480 Acked 5479 Read 4 Done Done New false
16050 4481 Acked 5480 Read 9 Done Done New false
16051 4482 Acked 5481 Read 1 Done Done New false
16052 4483 Acked 5482 Read 10 Done Done New false
16053 4484 Acked 5483 Read 9 Done Done New false
16054 4485 Acked 5484 Write 10 Done Done New false
16055 4486 Acked 5485 Read 6 Done Done New false
16056 4487 Acked 5486 Read 2 Done Done New false
16057 4488 Acked 5487 Read 5 Done Done New false
16058 4489 Acked 5488 Write 9 Done Done New false
16059 4490 Acked 5489 Write 8 Done Done New false
16060 4491 Acked 5490 Read 1 Done Done New false
16061 4492 Acked 5491 Read 6 Done Done New false
16062 4493 Acked 5492 Write 6 Done Done New false
16063 4494 Acked 5493 Read 3 Done Done New false
16064 4495 Acked 5494 Read 3 Done Done New false
16065 4496 Acked 5495 Write 6 Done Done New false
16066 4497 Acked 5496 Read 6 Done Done New false
16067 4498 Acked 5497 Read 9 Done Done New false
16068 4499 Acked 5498 Read 8 Done Done New false
16069 4500 Acked 5499 Write 5 Done Done New false
16070 4501 Acked 5500 Write 2 Done Done New false
16071 4502 Acked 5501 Write 6 Done Done New false
16072 4503 Acked 5502 Read 8 Done Done New false
16073 4504 Acked 5503 Write 7 Done Done New false
16074 4505 Acked 5504 Read 1 Done Done New false
16075 4506 Acked 5505 Flush 0 Done Done New false
16076 4507 Acked 5506 Write 3 Done Done New false
16077 4508 Acked 5507 Read 8 Done Done New false
16078 4509 Acked 5508 Flush 0 Done Done New false
16079 4510 Acked 5509 Flush 0 Done Done New false
16080 4511 Acked 5510 Write 9 Done Done New false
16081 4512 Acked 5511 Write 5 Done Done New false
16082 4513 Acked 5512 Flush 0 Done Done New false
16083 4514 Acked 5513 Read 1 Done Done New false
16084 4515 Acked 5514 Write 9 Done Done New false
16085 4516 Acked 5515 Write 3 Done Done New false
16086 4517 Acked 5516 Write 7 Done Done New false
16087 4518 Acked 5517 Read 4 Done Done New false
16088 4519 Acked 5518 Write 5 Done Done New false
16089 4520 Acked 5519 Write 5 Done Done New false
16090 STATES DS:0 DS:1 DS:2 TOTAL
16091 New 0 0 301 301
16092 Sent 0 0 0 0
16093 Done 301 301 0 602
16094 Skipped 0 0 0 0
16095 Error 0 0 0 0
16096 Last Flush: 5512 5512 5218
16097 Downstairs last five completed: 5218 5217 5216 5215 5214
16098 Upstairs last five completed: 4520 4519 4518 4517 4516
16099 CLIENT: Up:0 ds:301 act:2
161002024-02-15T03:00:40.392ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
161012024-02-15T03:00:40.392ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
161022024-02-15T03:00:40.392ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
161032024-02-15T03:00:40.392ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
161042024-02-15T03:00:40.392ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
161052024-02-15T03:00:40.392ZINFOcrucible: [2] client re-new 302 jobs since flush 5218 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
161062024-02-15T03:00:40.393ZINFOcrucible: client task is exiting = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
161072024-02-15T03:00:41.393ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
161082024-02-15T03:00:41.393ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
161092024-02-15T03:00:41.393ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
161102024-02-15T03:00:41.393ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
161112024-02-15T03:00:41.393ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
161122024-02-15T03:00:41.393ZINFOcrucible: [2] client re-new 302 jobs since flush 5218 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
161132024-02-15T03:00:41.394ZINFOcrucible: client task is exiting = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
16114 [8830] got start action
16115 [8830] I am going to restart
16116 Starting downstairs at port 8830
16117 Make output file at "/tmp/dsc/downstairs-8830.txt"
16118 [8830][2] reports Starting
16119 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1182)
16120 [8830][2] reports Running
161212024-02-15T03:00:42.395ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
161222024-02-15T03:00:42.395ZINFOcrucible: ds_connection connected = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
161232024-02-15T03:00:42.396ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
161242024-02-15T03:00:42.396ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
161252024-02-15T03:00:42.396ZINFOcrucible: send last flush ID to this DS: 5218 = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
161262024-02-15T03:00:42.396ZINFOcrucible: Replied this last flush ID: 5218 = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
161272024-02-15T03:00:42.396ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
161282024-02-15T03:00:42.396ZINFOcrucible: [2] Transition from Offline to Replay = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
161292024-02-15T03:00:42.396ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
161302024-02-15T03:00:42.396ZINFOcrucible: [2] Transition from Replay to Active = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
16131 ----------------------------------------------------------------
16132 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
16133 Downstairs last five completed: 5520 5519 5518 5517 5516
16134 Upstairs last five completed: 4521 4520 4519 4518 4517
16135 CLIENT: Up:0 ds:0 act:3
16136 Replay: All jobs finished, all DS active.
16137 Main task has work to do, go find it
16138 got dsc Stop(2)
16139 stop 2
16140 Feb 15 03:00:43.717 INFO request completed, latency_us: 378, [8830] Got stop action so:false kr:false
16141 response_code: 204, uri: /stop/cid/2, method: POST, req_id: c2513c89-defb-4a05-a28e-5f60f762a6e5, remote_addr: 127.0.0.1:54744, local_addr: 127.0.0.1:9998
161422024-02-15T03:00:43.718ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
161432024-02-15T03:00:43.718ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
161442024-02-15T03:00:43.718ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
16145 Feb 15 03:00:43.718 INFO request completed, latency_us: {"msg":"Gone missing, transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-02-15T03:00:43.718599224Z","hostname":"ip-10-150-1-67.us-west-2.compute.internal","pid":1167,"client":"2","":"downstairs","session_id":"4f5a63e5-aa00-4890-9f6b-96c2e069fdb2"}
16146 106, response_code: 200, uri: /state/cid/2, method: GET, req_id: 4b848c46-25b4-4db5-9dd4-a89e425a89ed, remote_addr: 127.0.0.1:54744, local_addr: 127.0.0.1:9998
161472024-02-15T03:00:43.718ZINFOcrucible: [2] client re-new 0 jobs since flush 5520 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
161482024-02-15T03:00:43.718ZINFOcrucible: client task is exiting = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
16149 [8830] Exited with: ExitStatus(unix_wait_status(9))
16150 [8830][2] reports Exit
161512024-02-15T03:00:44.719ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
161522024-02-15T03:00:44.719ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
161532024-02-15T03:00:44.719ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
161542024-02-15T03:00:44.719ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
161552024-02-15T03:00:44.719ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
161562024-02-15T03:00:44.719ZINFOcrucible: [2] client re-new 0 jobs since flush 5520 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
161572024-02-15T03:00:44.719ZINFOcrucible: client task is exiting = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
161582024-02-15T03:00:45.720ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
161592024-02-15T03:00:45.720ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
161602024-02-15T03:00:45.720ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
161612024-02-15T03:00:45.720ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
161622024-02-15T03:00:45.720ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
161632024-02-15T03:00:45.720ZINFOcrucible: [2] client re-new 0 jobs since flush 5520 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
161642024-02-15T03:00:45.720ZINFOcrucible: client task is exiting = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
161652024-02-15T03:00:46.722ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
161662024-02-15T03:00:46.722ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
161672024-02-15T03:00:46.722ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
161682024-02-15T03:00:46.722ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
161692024-02-15T03:00:46.722ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
161702024-02-15T03:00:46.722ZINFOcrucible: [2] client re-new 0 jobs since flush 5520 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
161712024-02-15T03:00:46.722ZINFOcrucible: client task is exiting = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
16172 Feb 15 03:00:47.720 INFO request completed, latency_us: 199, response_code: 200, uri: /state/cid/2, method: GET, req_id: 27033213-dc1e-4789-a219-254c7ebb4a3f, remote_addr: 127.0.0.1:54744, local_addr: 127.0.0.1:9998
16173 001/300 Write block 3832 len 40960 data: 4 6 5 4 4 4 3 2 2 2
16174 002/300 Read block 2751 len 36864
161752024-02-15T03:00:47.723ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
161762024-02-15T03:00:47.723ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
161772024-02-15T03:00:47.723ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
161782024-02-15T03:00:47.723ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
161792024-02-15T03:00:47.723ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
161802024-02-15T03:00:47.723ZINFOcrucible: [2] client re-new 2 jobs since flush 5520 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
161812024-02-15T03:00:47.723ZINFOcrucible: client task is exiting = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
16182 003/300 Read block 2692 len 36864
16183 004/300 Write block 3225 len 36864 data: 2 2 2 2 2 2 2 2 2
16184 005/300 Read block 841 len 20480
16185 006/300 Write block 924 len 12288 data: 3 2 2
16186 007/300 Write block 3155 len 36864 data: 7 8 8 10 9 8 8 7 6
16187 008/300 Write block 2121 len 16384 data: 3 3 4 4
16188 009/300 Read block 2568 len 12288
16189 010/300 Flush
16190 011/300 Write block 1799 len 8192 data: 5 4
16191 012/300 Read block 3189 len 20480
16192 013/300 Write block 3298 len 36864 data: 5 4 4 3 3 4 5 5 4
16193 014/300 Read block 3085 len 24576
16194 015/300 Read block 4866 len 8192
16195 016/300 Read block 813 len 8192
16196 017/300 Read block 2378 len 28672
16197 018/300 Read block 900 len 4096
16198 019/300 Write block 4483 len 24576 data: 3 3 3 3 2 3
16199 020/300 Read block 1387 len 20480
16200 021/300 Write block 4630 len 36864 data: 5 4 3 3 3 3 4 4 4
16201 022/300 Read block 1911 len 36864
16202 023/300 Read block 4891 len 4096
16203 024/300 Write block 2074 len 4096 data: 4
16204 025/300 Write block 1749 len 8192 data: 3 2
16205 026/300 Flush
16206 027/300 Write block 4821 len 28672 data: 3 3 3 3 4 4 3
16207 028/300 Read block 3677 len 20480
16208 029/300 Write block 916 len 28672 data: 4 4 5 3 2 2 2
16209 030/300 Write block 2994 len 28672 data: 2 2 2 2 2 2 2
16210 031/300 Read block 3032 len 28672
16211 032/300 Read block 1445 len 32768
16212 033/300 Write block 3909 len 28672 data: 4 3 3 3 3 3 3
16213 034/300 Write block 2458 len 12288 data: 4 4 5
16214 035/300 Write block 506 len 20480 data: 5 6 7 5 5
16215 036/300 Read block 4882 len 28672
16216 037/300 Read block 4591 len 24576
16217 038/300 Write block 3938 len 12288 data: 2 2 2
16218 039/300 Flush
16219 040/300 Read block 2448 len 32768
16220 041/300 Read block 4030 len 28672
16221 042/300 Flush
16222 043/300 Read block 1149 len 36864
16223 044/300 Flush
16224 045/300 Read block 3434 len 12288
16225 046/300 Read block 2358 len 40960
16226 047/300 Read block 3721 len 24576
16227 048/300 Read block 1635 len 16384
16228 049/300 Write block 1562 len 4096 data: 5
16229 050/300 Read block 2371 len 40960
16230 051/300 Read block 4033 len 40960
16231 052/300 Read block 1688 len 12288
16232 053/300 Flush
16233 054/300 Write block 3673 len 16384 data: 5 5 5 5
16234 055/300 Read block 3613 len 16384
16235 056/300 Write block 2784 len 32768 data: 4 4 3 2 2 2 2 2
16236 057/300 Write block 458 len 28672 data: 5 4 4 4 3 3 4
16237 058/300 Write block 4692 len 40960 data: 6 5 5 4 4 3 2 3 3 3
16238 059/300 Flush
16239 060/300 Read block 4595 len 32768
16240 061/300 Write block 3929 len 4096 data: 2
16241 062/300 Read block 3210 len 20480
16242 063/300 Read block 307 len 4096
16243 064/300 Read block 3920 len 8192
16244 065/300 Flush
16245 066/300 Write block 2484 len 28672 data: 5 4 4 3 3 3 5
16246 067/300 Read block 3060 len 8192
16247 068/300 Write block 1262 len 32768 data: 3 2 3 3 3 3 3 3
16248 069/300 Read block 3045 len 16384
16249 070/300 Read block 1798 len 36864
16250 071/300 Read block 159 len 4096
16251 072/300 Read block 1315 len 28672
16252 073/300 Write block 1652 len 36864 data: 5 4 5 6 4 4 4 4 5
16253 074/300 Read block 2799 len 40960
16254 075/300 Flush
16255 076/300 Write block 3616 len 40960 data: 4 5 6 6 6 5 5 7 7 7
16256 077/300 Read block 1193 len 16384
16257 078/300 Read block 2432 len 28672
16258 079/300 Write block 69 len 28672 data: 2 4 4 5 5 4 4
16259 080/300 Read block 1198 len 16384
16260 081/300 Flush
16261 082/300 Read block 2309 len 24576
16262 083/300 Read block 1737 len 36864
16263 084/300 Write block 1119 len 12288 data: 6 5 5
16264 085/300 Read block 3549 len 40960
16265 086/300 Write block 930 len 40960 data: 2 2 3 3 3 3 4 5 5 5
16266 087/300 Read block 2497 len 20480
16267 088/300 Read block 4405 len 32768
16268 089/300 Write block 4227 len 40960 data: 2 2 2 2 3 3 3 3 4 5
16269 090/300 Flush
16270 091/300 Read block 990 len 12288
16271 092/300 Read block 3911 len 32768
16272 093/300 Write block 4129 len 20480 data: 4 4 3 3 4
16273 094/300 Read block 1430 len 36864
16274 095/300 Read block 4620 len 20480
16275 096/300 Read block 1090 len 24576
16276 097/300 Write block 219 len 24576 data: 4 4 4 4 4 3
16277 098/300 Read block 2029 len 16384
16278 099/300 Read block 2624 len 40960
16279 100/300 Write block 433 len 4096 data: 3
16280 101/300 Read block 575 len 8192
16281 102/300 Write block 1591 len 40960 data: 4 5 5 4 4 3 2 2 2 2
16282 103/300 Write block 1379 len 36864 data: 4 4 3 3 3 3 3 3 3
16283 104/300 Read block 1547 len 8192
16284 105/300 Write block 2036 len 12288 data: 2 2 2
16285 106/300 Read block 3100 len 40960
16286 107/300 Write block 4060 len 32768 data: 5 5 6 5 5 5 5 4
16287 108/300 Write block 4419 len 28672 data: 6 6 5 5 6 7 7
16288 109/300 Write block 3481 len 20480 data: 3 3 4 4 4
16289 110/300 Flush
16290 111/300 Flush
16291 112/300 Write block 3528 len 28672 data: 5 6 6 5 5 7 7
16292 113/300 Write block 3311 len 8192 data: 3 3
16293 114/300 Write block 1084 len 12288 data: 4 4 5
16294 115/300 Read block 874 len 16384
16295 116/300 Read block 2500 len 16384
16296 117/300 Write block 4616 len 12288 data: 2 3 3
16297 118/300 Read block 3347 len 24576
16298 119/300 Write block 1643 len 40960 data: 8 8 7 7 6 6 7 5 5 6
16299 120/300 Read block 4406 len 12288
16300 121/300 Read block 2902 len 36864
16301 122/300 Read block 4386 len 4096
16302 123/300 Write block 4701 len 36864 data: 4 4 3 3 3 3 2 3 4
16303 124/300 Write block 2070 len 32768 data: 4 4 5 5 5 4 3 3
16304 125/300 Read block 1364 len 40960
16305 126/300 Write block 2917 len 20480 data: 3 3 3 3 3
16306 127/300 Read block 520 len 28672
16307 128/300 Read block 127 len 16384
16308 129/300 Read block 2755 len 20480
16309 130/300 Read block 2648 len 36864
16310 131/300 Flush
16311 132/300 Write block 3338 len 28672 data: 5 5 4 5 4 4 4
16312 133/300 Read block 2729 len 32768
16313 134/300 Flush
16314 135/300 Write block 2114 len 16384 data: 2 2 2 3
16315 136/300 Write block 3673 len 32768 data: 6 6 6 6 4 4 3 4
16316 137/300 Read block 1125 len 8192
16317 138/300 Read block 2323 len 12288
16318 139/300 Write block 3679 len 8192 data: 4 5
16319 140/300 Read block 4648 len 32768
16320 141/300 Read block 1381 len 28672
16321 142/300 Read block 4504 len 40960
16322 143/300 Read block 144 len 16384
16323 144/300 Read block 4718 len 32768
16324 145/300 Read block 1665 len 16384
16325 146/300 Write block 1578 len 24576 data: 4 4 4 6 5 7
16326 147/300 Write block 1826 len 12288 data: 6 7 7
16327 148/300 Write block 1323 len 40960 data: 5 5 4 5 5 5 5 4 4 5
16328 149/300 Read block 2899 len 12288
16329 150/300 Write block 2485 len 40960 data: 5 5 4 4 4 6 5 6 5 4
16330 151/300 Read block 289 len 20480
16331 152/300 Read block 4142 len 36864
16332 153/300 Write block 361 len 36864 data: 4 5 5 3 2 2 4 5 6
16333 154/300 Write block 1674 len 24576 data: 3 3 3 4 4 3
16334 155/300 Read block 83 len 24576
16335 156/300 Write block 2015 len 12288 data: 3 4 5
16336 157/300 Read block 937 len 36864
16337 158/300 Write block 4126 len 36864 data: 4 4 5 5 5 4 4 5 4
16338 159/300 Write block 3598 len 32768 data: 4 4 4 3 2 3 4 4
16339 160/300 Read block 3242 len 8192
16340 161/300 Read block 3285 len 24576
16341 162/300 Read block 2995 len 36864
16342 163/300 Write block 768 len 12288 data: 2 2 2
16343 164/300 Read block 4094 len 32768
16344 165/300 Read block 1370 len 28672
16345 166/300 Read block 4475 len 28672
16346 167/300 Read block 3992 len 12288
16347 168/300 Write block 3506 len 8192 data: 5 5
16348 169/300 Write block 4797 len 36864 data: 4 4 4 4 4 3 3 3 3
16349 170/300 Read block 3661 len 24576
16350 171/300 Read block 73 len 40960
16351 172/300 Read block 1487 len 20480
16352 173/300 Read block 4936 len 8192
16353 174/300 Read block 1196 len 4096
16354 175/300 Read block 1559 len 32768
16355 176/300 Read block 2560 len 28672
16356 177/300 Read block 1683 len 16384
16357 178/300 Write block 1040 len 4096 data: 5
16358 179/300 Read block 3814 len 20480
16359 180/300 Read block 3107 len 28672
16360 181/300 Flush
16361 182/300 Read block 4343 len 28672
16362 183/300 Write block 4272 len 8192 data: 2 3
16363 184/300 Read block 2686 len 24576
16364 185/300 Read block 711 len 20480
16365 186/300 Write block 3328 len 4096 data: 5
16366 187/300 Write block 452 len 4096 data: 6
16367 188/300 Write block 4048 len 4096 data: 6
16368 189/300 Read block 2378 len 32768
16369 190/300 Write block 2258 len 28672 data: 5 5 4 4 4 4 4
16370 191/300 Write block 3772 len 36864 data: 2 3 5 5 5 4 5 5 5
16371 192/300 Write block 4748 len 28672 data: 4 4 5 5 5 6 5
16372 193/300 Read block 1183 len 12288
16373 194/300 Read block 4927 len 20480
16374 195/300 Read block 4168 len 40960
16375 196/300 Write block 2119 len 40960 data: 4 4 4 4 5 5 4 6 7 6
16376 197/300 Read block 4576 len 4096
16377 198/300 Flush
16378 199/300 Read block 181 len 8192
16379 200/300 Read block 1673 len 12288
16380 201/300 Write block 2186 len 32768 data: 5 5 5 5 5 5 5 5
16381 202/300 Read block 2169 len 8192
16382 203/300 Flush
16383 204/300 Write block 3433 len 12288 data: 3 3 2
16384 205/300 Read block 4377 len 4096
16385 206/300 Write block 1868 len 40960 data: 2 2 2 2 3 4 5 5 5 5
16386 207/300 Write block 1116 len 12288 data: 4 4 4
16387 208/300 Write block 4796 len 20480 data: 5 5 5 5 5
16388 209/300 Write block 677 len 32768 data: 3 3 3 4 4 5 5 4
16389 210/300 Read block 3516 len 36864
16390 211/300 Read block 4506 len 36864
16391 212/300 Write block 803 len 32768 data: 3 4 4 4 3 3 3 5
16392 213/300 Read block 163 len 8192
16393 214/300 Flush
16394 215/300 Read block 1872 len 32768
16395 216/300 Write block 4251 len 16384 data: 6 5 3 3
16396 217/300 Read block 3612 len 28672
16397 218/300 Read block 1411 len 40960
16398 219/300 Read block 2432 len 4096
16399 220/300 Read block 2790 len 28672
16400 221/300 Read block 3191 len 40960
16401 222/300 Flush
16402 223/300 Write block 1228 len 40960 data: 2 2 2 3 3 4 5 5 5 4
16403 224/300 Flush
16404 225/300 Flush
16405 226/300 Write block 4818 len 40960 data: 4 4 3 4 4 4 4 5 5 4
16406 227/300 Read block 2543 len 28672
16407 228/300 Write block 3594 len 32768 data: 3 4 5 4 5 5 5 4
16408 229/300 Read block 1360 len 4096
16409 230/300 Write block 4023 len 24576 data: 6 6 7 7 6 8
16410 231/300 Write block 2874 len 40960 data: 3 3 2 3 3 4 5 6 6 6
16411 232/300 Write block 2657 len 32768 data: 2 2 3 3 4 2 3 5
16412 233/300 Write block 4893 len 4096 data: 6
16413 234/300 Flush
16414 235/300 Write block 4449 len 8192 data: 6 6
16415 236/300 Read block 2459 len 36864
16416 237/300 Write block 731 len 24576 data: 4 5 5 6 5 5
16417 238/300 Write block 3543 len 4096 data: 3
16418 239/300 Flush
16419 240/300 Read block 1179 len 28672
16420 241/300 Read block 1589 len 32768
16421 242/300 Write block 4387 len 36864 data: 5 4 3 3 4 4 3 3 3
16422 243/300 Read block 210 len 28672
16423 244/300 Flush
16424 245/300 Flush
16425 246/300 Flush
16426 247/300 Flush
16427 248/300 Read block 4357 len 12288
16428 249/300 Write block 4381 len 12288 data: 3 3 5
16429 250/300 Write block 170 len 24576 data: 3 3 3 4 5 4
16430 251/300 Write block 1009 len 16384 data: 4 4 5 6
16431 252/300 Read block 472 len 12288
16432 253/300 Read block 425 len 12288
16433 254/300 Read block 3093 len 8192
16434 255/300 Read block 4092 len 12288
16435 256/300 Read block 4812 len 24576
16436 257/300 Read block 3079 len 16384
16437 258/300 Read block 4473 len 32768
16438 259/300 Flush
16439 260/300 Write block 678 len 16384 data: 4 4 5 5
16440 261/300 Write block 707 len 12288 data: 4 3 3
16441 262/300 Write block 1994 len 32768 data: 4 5 4 4 5 5 5 7
16442 263/300 Write block 4361 len 28672 data: 3 4 3 3 3 3 2
16443 264/300 Read block 2628 len 24576
16444 265/300 Read block 727 len 8192
16445 266/300 Write block 858 len 24576 data: 4 4 4 4 4 5
16446 267/300 Write block 556 len 24576 data: 4 4 5 5 4 4
16447 268/300 Read block 1731 len 8192
16448 269/300 Write block 4391 len 24576 data: 5 5 4 4 4 2
16449 270/300 Write block 3877 len 40960 data: 4 4 4 6 5 5 5 4 3 3
16450 271/300 Write block 3908 len 28672 data: 5 5 4 4 4 4 4
16451 272/300 Read block 4333 len 4096
16452 273/300 Flush
16453 274/300 Read block 4813 len 36864
16454 275/300 Write block 3207 len 36864 data: 4 4 3 4 3 4 6 5 5
16455 276/300 Read block 417 len 24576
16456 277/300 Read block 448 len 8192
16457 278/300 Write block 3467 len 12288 data: 2 2 2
16458 279/300 Write block 1697 len 40960 data: 4 4 4 3 4 3 3 3 4 3
16459 280/300 Read block 4866 len 28672
16460 281/300 Read block 1744 len 20480
16461 282/300 Read block 2618 len 40960
16462 283/300 Read block 3920 len 32768
16463 284/300 Read block 1600 len 36864
16464 285/300 Read block 4174 len 16384
16465 286/300 Read block 4490 len 16384
16466 287/300 Read block 2812 len 8192
16467 288/300 Write block 625 len 4096 data: 3
16468 289/300 Read block 3856 len 16384
16469 290/300 Read block 1118 len 28672
16470 291/300 Read block 2226 len 28672
16471 292/300 Read block 2896 len 4096
16472 293/300 Write block 2752 len 4096 data: 3
16473 294/300 Read block 2627 len 16384
16474 295/300 Write block 2104 len 36864 data: 2 3 4 3 3 3 3 3 2
16475 296/300 Write block 1662 len 4096 data: 3
16476 297/300 Read block 3460 len 32768
16477 298/300 Read block 2714 len 28672
16478 299/300 Read block 312 len 12288
16479 300/300 Write block 1522 len 36864 data: 2 3 5 5 6 7 7 7 6
16480 Main task has work to do, go find it
16481 got dsc Start(2)
16482 start 2
16483 Feb 15 03:00:48.067 INFO request completed, latency_us: 250, response_code: 204, uri: /start/cid/2, method: POST, req_id: cb18718b-3e84-4a4a-92a6-98eb4c504f0e, remote_addr: 127.0.0.1:54744, local_addr: 127.0.0.1:9998
16484 Replay: started 2, returned:Ok(())
16485 ----------------------------------------------------------------
16486 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
16487 States: Active Active Offline
16488 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
16489 4522 Acked 5521 Write 10 Done Done New true
16490 4523 Acked 5522 Read 9 Done Done New true
16491 4524 Acked 5523 Read 9 Done Done New false
16492 4525 Acked 5524 Write 9 Done Done New false
16493 4526 Acked 5525 Read 5 Done Done New false
16494 4527 Acked 5526 Write 3 Done Done New false
16495 4528 Acked 5527 Write 9 Done Done New false
16496 4529 Acked 5528 Write 4 Done Done New false
16497 4530 Acked 5529 Read 3 Done Done New false
16498 4531 Acked 5530 Flush 0 Done Done New false
16499 4532 Acked 5531 Write 2 Done Done New false
16500 4533 Acked 5532 Read 5 Done Done New false
16501 4534 Acked 5533 Write 9 Done Done New false
16502 4535 Acked 5534 Read 6 Done Done New false
16503 4536 Acked 5535 Read 2 Done Done New false
16504 4537 Acked 5536 Read 2 Done Done New false
16505 4538 Acked 5537 Read 7 Done Done New false
16506 4539 Acked 5538 Read 1 Done Done New false
16507 4540 Acked 5539 Write 6 Done Done New false
16508 4541 Acked 5540 Read 5 Done Done New false
16509 4542 Acked 5541 Write 9 Done Done New false
16510 4543 Acked 5542 Read 9 Done Done New false
16511 4544 Acked 5543 Read 1 Done Done New false
16512 4545 Acked 5544 Write 1 Done Done New false
16513 4546 Acked 5545 Write 2 Done Done New false
16514 4547 Acked 5546 Flush 0 Done Done New false
16515 4548 Acked 5547 Write 7 Done Done New false
16516 4549 Acked 5548 Read 5 Done Done New false
16517 4550 Acked 5549 Write 7 Done Done New false
16518 4551 Acked 5550 Write 7 Done Done New false
16519 4552 Acked 5551 Read 7 Done Done New false
16520 4553 Acked 5552 Read 8 Done Done New false
16521 4554 Acked 5553 Write 7 Done Done New false
16522 4555 Acked 5554 Write 3 Done Done New false
16523 4556 Acked 5555 Write 5 Done Done New false
16524 4557 Acked 5556 Read 7 Done Done New false
16525 4558 Acked 5557 Read 6 Done Done New false
16526 4559 Acked 5558 Write 3 Done Done New false
16527 4560 Acked 5559 Flush 0 Done Done New false
16528 4561 Acked 5560 Read 8 Done Done New false
16529 4562 Acked 5561 Read 7 Done Done New false
16530 4563 Acked 5562 Flush 0 Done Done New false
16531 4564 Acked 5563 Read 9 Done Done New false
16532 4565 Acked 5564 Flush 0 Done Done New false
16533 4566 Acked 5565 Read 3 Done Done New false
16534 4567 Acked 5566 Read 10 Done Done New false
16535 4568 Acked 5567 Read 6 Done Done New false
16536 4569 Acked 5568 Read 4 Done Done New false
16537 4570 Acked 5569 Write 1 Done Done New false
16538 4571 Acked 5570 Read 10 Done Done New false
16539 4572 Acked 5571 Read 10 Done Done New false
16540 4573 Acked 5572 Read 3 Done Done New false
16541 4574 Acked 5573 Flush 0 Done Done New false
16542 4575 Acked 5574 Write 4 Done Done New false
16543 4576 Acked 5575 Read 4 Done Done New false
16544 4577 Acked 5576 Write 8 Done Done New false
16545 4578 Acked 5577 Write 7 Done Done New false
16546 4579 Acked 5578 Write 10 Done Done New false
16547 4580 Acked 5579 Flush 0 Done Done New false
16548 4581 Acked 5580 Read 8 Done Done New false
16549 4582 Acked 5581 Write 1 Done Done New false
16550 4583 Acked 5582 Read 5 Done Done New false
16551 4584 Acked 5583 Read 1 Done Done New false
16552 4585 Acked 5584 Read 2 Done Done New false
16553 4586 Acked 5585 Flush 0 Done Done New false
16554 4587 Acked 5586 Write 7 Done Done New false
16555 4588 Acked 5587 Read 2 Done Done New false
16556 4589 Acked 5588 Write 8 Done Done New false
16557 4590 Acked 5589 Read 4 Done Done New false
16558 4591 Acked 5590 Read 9 Done Done New false
16559 4592 Acked 5591 Read 1 Done Done New false
16560 4593 Acked 5592 Read 7 Done Done New false
16561 4594 Acked 5593 Write 9 Done Done New false
16562 4595 Acked 5594 Read 10 Done Done New false
16563 4596 Acked 5595 Flush 0 Done Done New false
16564 4597 Acked 5596 Write 10 Done Done New false
16565 4598 Acked 5597 Read 4 Done Done New false
16566 4599 Acked 5598 Read 7 Done Done New false
16567 4600 Acked 5599 Write 7 Done Done New false
16568 4601 Acked 5600 Read 4 Done Done New false
16569 4602 Acked 5601 Flush 0 Done Done New false
16570 4603 Acked 5602 Read 6 Done Done New false
16571 4604 Acked 5603 Read 9 Done Done New false
16572 4605 Acked 5604 Write 3 Done Done New false
16573 4606 Acked 5605 Read 10 Done Done New false
16574 4607 Acked 5606 Write 10 Done Done New false
16575 4608 Acked 5607 Read 5 Done Done New false
16576 4609 Acked 5608 Read 8 Done Done New false
16577 4610 Acked 5609 Write 10 Done Done New false
16578 4611 Acked 5610 Flush 0 Done Done New false
16579 4612 Acked 5611 Read 3 Done Done New false
16580 4613 Acked 5612 Read 8 Done Done New false
16581 4614 Acked 5613 Write 5 Done Done New false
16582 4615 Acked 5614 Read 9 Done Done New false
16583 4616 Acked 5615 Read 5 Done Done New false
16584 4617 Acked 5616 Read 6 Done Done New false
16585 4618 Acked 5617 Write 6 Done Done New false
16586 4619 Acked 5618 Read 4 Done Done New false
16587 4620 Acked 5619 Read 10 Done Done New false
16588 4621 Acked 5620 Write 1 Done Done New false
16589 4622 Acked 5621 Read 2 Done Done New false
16590 4623 Acked 5622 Write 10 Done Done New false
16591 4624 Acked 5623 Write 9 Done Done New false
16592 4625 Acked 5624 Read 2 Done Done New false
16593 4626 Acked 5625 Write 3 Done Done New false
16594 4627 Acked 5626 Read 10 Done Done New false
16595 4628 Acked 5627 Write 8 Done Done New false
16596 4629 Acked 5628 Write 7 Done Done New false
16597 4630 Acked 5629 Write 5 Done Done New false
16598 4631 Acked 5630 Flush 0 Done Done New false
16599 4632 Acked 5631 Flush 0 Done Done New false
16600 4633 Acked 5632 Write 7 Done Done New false
16601 4634 Acked 5633 Write 2 Done Done New false
16602 4635 Acked 5634 Write 3 Done Done New false
16603 4636 Acked 5635 Read 4 Done Done New false
16604 4637 Acked 5636 Read 4 Done Done New false
16605 4638 Acked 5637 Write 3 Done Done New false
16606 4639 Acked 5638 Read 6 Done Done New false
16607 4640 Acked 5639 Write 10 Done Done New false
16608 4641 Acked 5640 Read 3 Done Done New false
16609 4642 Acked 5641 Read 9 Done Done New false
16610 4643 Acked 5642 Read 1 Done Done New false
16611 4644 Acked 5643 Write 9 Done Done New false
16612 4645 Acked 5644 Write 8 Done Done New false
16613 4646 Acked 5645 Read 10 Done Done New false
16614 4647 Acked 5646 Write 5 Done Done New false
16615 4648 Acked 5647 Read 7 Done Done New false
16616 4649 Acked 5648 Read 4 Done Done New false
16617 4650 Acked 5649 Read 5 Done Done New false
16618 4651 Acked 5650 Read 9 Done Done New false
16619 4652 Acked 5651 Flush 0 Done Done New false
16620 4653 Acked 5652 Write 7 Done Done New false
16621 4654 Acked 5653 Read 8 Done Done New false
16622 4655 Acked 5654 Flush 0 Done Done New false
16623 4656 Acked 5655 Write 4 Done Done New false
16624 4657 Acked 5656 Write 8 Done Done New false
16625 4658 Acked 5657 Read 2 Done Done New false
16626 4659 Acked 5658 Read 3 Done Done New false
16627 4660 Acked 5659 Write 2 Done Done New false
16628 4661 Acked 5660 Read 8 Done Done New false
16629 4662 Acked 5661 Read 7 Done Done New false
16630 4663 Acked 5662 Read 10 Done Done New false
16631 4664 Acked 5663 Read 4 Done Done New false
16632 4665 Acked 5664 Read 8 Done Done New false
16633 4666 Acked 5665 Read 4 Done Done New false
16634 4667 Acked 5666 Write 6 Done Done New false
16635 4668 Acked 5667 Write 3 Done Done New false
16636 4669 Acked 5668 Write 10 Done Done New false
16637 4670 Acked 5669 Read 3 Done Done New false
16638 4671 Acked 5670 Write 10 Done Done New false
16639 4672 Acked 5671 Read 5 Done Done New false
16640 4673 Acked 5672 Read 9 Done Done New false
16641 4674 Acked 5673 Write 9 Done Done New false
16642 4675 Acked 5674 Write 6 Done Done New false
16643 4676 Acked 5675 Read 6 Done Done New false
16644 4677 Acked 5676 Write 3 Done Done New false
16645 4678 Acked 5677 Read 9 Done Done New false
16646 4679 Acked 5678 Write 9 Done Done New false
16647 4680 Acked 5679 Write 8 Done Done New false
16648 4681 Acked 5680 Read 2 Done Done New false
16649 4682 Acked 5681 Read 6 Done Done New false
16650 4683 Acked 5682 Read 9 Done Done New false
16651 4684 Acked 5683 Write 3 Done Done New false
16652 4685 Acked 5684 Read 8 Done Done New false
16653 4686 Acked 5685 Read 7 Done Done New false
16654 4687 Acked 5686 Read 7 Done Done New false
16655 4688 Acked 5687 Read 3 Done Done New false
16656 4689 Acked 5688 Write 2 Done Done New false
16657 4690 Acked 5689 Write 9 Done Done New false
16658 4691 Acked 5690 Read 6 Done Done New false
16659 4692 Acked 5691 Read 10 Done Done New false
16660 4693 Acked 5692 Read 5 Done Done New false
16661 4694 Acked 5693 Read 2 Done Done New false
16662 4695 Acked 5694 Read 1 Done Done New false
16663 4696 Acked 5695 Read 8 Done Done New false
16664 4697 Acked 5696 Read 7 Done Done New false
16665 4698 Acked 5697 Read 4 Done Done New false
16666 4699 Acked 5698 Write 1 Done Done New false
16667 4700 Acked 5699 Read 5 Done Done New false
16668 4701 Acked 5700 Read 7 Done Done New false
16669 4702 Acked 5701 Flush 0 Done Done New false
16670 4703 Acked 5702 Read 7 Done Done New false
16671 4704 Acked 5703 Write 2 Done Done New false
16672 4705 Acked 5704 Read 6 Done Done New false
16673 4706 Acked 5705 Read 5 Done Done New false
16674 4707 Acked 5706 Write 1 Done Done New false
16675 4708 Acked 5707 Write 1 Done Done New false
16676 4709 Acked 5708 Write 1 Done Done New false
16677 4710 Acked 5709 Read 8 Done Done New false
16678 4711 Acked 5710 Write 7 Done Done New false
16679 4712 Acked 5711 Write 9 Done Done New false
16680 4713 Acked 5712 Flush 0 Done Done New false
16681 4714 Acked 5713 Write 7 Done Done New false
16682 4715 Acked 5714 Read 3 Done Done New false
16683 4716 Acked 5715 Read 5 Done Done New false
16684 4717 Acked 5716 Read 10 Done Done New false
16685 4718 Acked 5717 Write 10 Done Done New false
16686 4719 Acked 5718 Read 1 Done Done New false
16687 4720 Acked 5719 Flush 0 Done Done New false
16688 4721 Acked 5720 Read 2 Done Done New false
16689 4722 Acked 5721 Read 3 Done Done New false
16690 4723 Acked 5722 Write 8 Done Done New false
16691 4724 Acked 5723 Read 2 Done Done New false
16692 4725 Acked 5724 Flush 0 Done Done New false
16693 4726 Acked 5725 Write 3 Done Done New false
16694 4727 Acked 5726 Read 1 Done Done New false
16695 4728 Acked 5727 Write 10 Done Done New false
16696 4729 Acked 5728 Write 3 Done Done New false
16697 4730 Acked 5729 Write 5 Done Done New false
16698 4731 Acked 5730 Write 8 Done Done New false
16699 4732 Acked 5731 Read 9 Done Done New false
16700 4733 Acked 5732 Read 9 Done Done New false
16701 4734 Acked 5733 Write 8 Done Done New false
16702 4735 Acked 5734 Read 2 Done Done New false
16703 4736 Acked 5735 Flush 0 Done Done New false
16704 4737 Acked 5736 Read 8 Done Done New false
16705 4738 Acked 5737 Write 4 Done Done New false
16706 4739 Acked 5738 Read 7 Done Done New false
16707 4740 Acked 5739 Read 10 Done Done New false
16708 4741 Acked 5740 Read 1 Done Done New false
16709 4742 Acked 5741 Read 7 Done Done New false
16710 4743 Acked 5742 Read 10 Done Done New false
16711 4744 Acked 5743 Flush 0 Done Done New false
16712 4745 Acked 5744 Write 10 Done Done New false
16713 4746 Acked 5745 Flush 0 Done Done New false
16714 4747 Acked 5746 Flush 0 Done Done New false
16715 4748 Acked 5747 Write 10 Done Done New false
16716 4749 Acked 5748 Read 7 Done Done New false
16717 4750 Acked 5749 Write 8 Done Done New false
16718 4751 Acked 5750 Read 1 Done Done New false
16719 4752 Acked 5751 Write 6 Done Done New false
16720 4753 Acked 5752 Write 10 Done Done New false
16721 4754 Acked 5753 Write 8 Done Done New false
16722 4755 Acked 5754 Write 1 Done Done New false
16723 4756 Acked 5755 Flush 0 Done Done New false
16724 4757 Acked 5756 Write 2 Done Done New false
16725 4758 Acked 5757 Read 9 Done Done New false
16726 4759 Acked 5758 Write 6 Done Done New false
16727 4760 Acked 5759 Write 1 Done Done New false
16728 4761 Acked 5760 Flush 0 Done Done New false
16729 4762 Acked 5761 Read 7 Done Done New false
16730 4763 Acked 5762 Read 8 Done Done New false
16731 4764 Acked 5763 Write 9 Done Done New false
16732 4765 Acked 5764 Read 7 Done Done New false
16733 4766 Acked 5765 Flush 0 Done Done New false
16734 4767 Acked 5766 Flush 0 Done Done New false
16735 4768 Acked 5767 Flush 0 Done Done New false
16736 4769 Acked 5768 Flush 0 Done Done New false
16737 4770 Acked 5769 Read 3 Done Done New false
16738 4771 Acked 5770 Write 3 Done Done New false
16739 4772 Acked 5771 Write 6 Done Done New false
16740 4773 Acked 5772 Write 4 Done Done New false
16741 4774 Acked 5773 Read 3 Done Done New false
16742 4775 Acked 5774 Read 3 Done Done New false
16743 4776 Acked 5775 Read 2 Done Done New false
16744 4777 Acked 5776 Read 3 Done Done New false
16745 4778 Acked 5777 Read 6 Done Done New false
16746 4779 Acked 5778 Read 4 Done Done New false
16747 4780 Acked 5779 Read 8 Done Done New false
16748 4781 Acked 5780 Flush 0 Done Done New false
16749 4782 Acked 5781 Write 4 Done Done New false
16750 4783 Acked 5782 Write 3 Done Done New false
16751 4784 Acked 5783 Write 8 Done Done New false
16752 4785 Acked 5784 Write 7 Done Done New false
16753 4786 Acked 5785 Read 6 Done Done New false
16754 4787 Acked 5786 Read 2 Done Done New false
16755 4788 Acked 5787 Write 6 Done Done New false
16756 4789 Acked 5788 Write 6 Done Done New false
16757 4790 Acked 5789 Read 2 Done Done New false
16758 4791 Acked 5790 Write 6 Done Done New false
16759 4792 Acked 5791 Write 10 Done Done New false
16760 4793 Acked 5792 Write 7 Done Done New false
16761 4794 Acked 5793 Read 1 Done Done New false
16762 4795 Acked 5794 Flush 0 Done Done New false
16763 4796 Acked 5795 Read 9 Done Done New false
16764 4797 Acked 5796 Write 9 Done Done New false
16765 4798 Acked 5797 Read 6 Done Done New false
16766 4799 Acked 5798 Read 2 Done Done New false
16767 4800 Acked 5799 Write 3 Done Done New false
16768 4801 Acked 5800 Write 10 Done Done New false
16769 4802 Acked 5801 Read 7 Done Done New false
16770 4803 Acked 5802 Read 5 Done Done New false
16771 4804 Acked 5803 Read 10 Done Done New false
16772 4805 Acked 5804 Read 8 Done Done New false
16773 4806 Acked 5805 Read 9 Done Done New false
16774 4807 Acked 5806 Read 4 Done Done New false
16775 4808 Acked 5807 Read 4 Done Done New false
16776 4809 Acked 5808 Read 2 Done Done New false
16777 4810 Acked 5809 Write 1 Done Done New false
16778 4811 Acked 5810 Read 4 Done Done New false
16779 4812 Acked 5811 Read 7 Done Done New false
16780 4813 Acked 5812 Read 7 Done Done New false
16781 4814 Acked 5813 Read 1 Done Done New false
16782 4815 Acked 5814 Write 1 Done Done New false
16783 4816 Acked 5815 Read 4 Done Done New false
16784 4817 Acked 5816 Write 9 Done Done New false
16785 4818 Acked 5817 Write 1 Done Done New false
16786 4819 Acked 5818 Read 8 Done Done New false
16787 4820 Acked 5819 Read 7 Done Done New false
16788 4821 Acked 5820 Read 3 Done Done New false
16789 4822 Acked 5821 Write 9 Done Done New false
16790 STATES DS:0 DS:1 DS:2 TOTAL
16791 New 0 0 301 301
16792 Sent 0 0 0 0
16793 Done 301 301 0 602
16794 Skipped 0 0 0 0
16795 Error 0 0 0 0
16796 Last Flush: 5794 5794 5520
16797 Downstairs last five completed: 5520 5519 5518 5517 5516
16798 Upstairs last five completed: 4822 4821 4820 4819 4818
16799 CLIENT: Up:0 ds:301 act:2
168002024-02-15T03:00:48.725ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
168012024-02-15T03:00:48.725ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
168022024-02-15T03:00:48.725ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
168032024-02-15T03:00:48.725ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
168042024-02-15T03:00:48.725ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
168052024-02-15T03:00:48.725ZINFOcrucible: [2] client re-new 302 jobs since flush 5520 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
168062024-02-15T03:00:48.725ZINFOcrucible: client task is exiting = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
16807 [8830] got start action
16808 [8830] I am going to restart
16809 Starting downstairs at port 8830
16810 Make output file at "/tmp/dsc/downstairs-8830.txt"
16811 [8830][2] reports Starting
168122024-02-15T03:00:49.727ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
168132024-02-15T03:00:49.727ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
168142024-02-15T03:00:49.727ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
168152024-02-15T03:00:49.727ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
168162024-02-15T03:00:49.727ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
168172024-02-15T03:00:49.727ZINFOcrucible: [2] client re-new 302 jobs since flush 5520 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
168182024-02-15T03:00:49.727ZINFOcrucible: client task is exiting = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
16819 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1183)
16820 [8830][2] reports Running
168212024-02-15T03:00:50.728ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
168222024-02-15T03:00:50.729ZINFOcrucible: ds_connection connected = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
168232024-02-15T03:00:50.730ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
168242024-02-15T03:00:50.730ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
168252024-02-15T03:00:50.730ZINFOcrucible: send last flush ID to this DS: 5520 = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
168262024-02-15T03:00:50.730ZINFOcrucible: Replied this last flush ID: 5520 = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
168272024-02-15T03:00:50.731ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
168282024-02-15T03:00:50.731ZINFOcrucible: [2] Transition from Offline to Replay = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
168292024-02-15T03:00:50.731ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
168302024-02-15T03:00:50.731ZINFOcrucible: [2] Transition from Replay to Active = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
16831 ----------------------------------------------------------------
16832 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
16833 Downstairs last five completed: 5822 5821 5820 5819 5818
16834 Upstairs last five completed: 4823 4822 4821 4820 4819
16835 CLIENT: Up:0 ds:0 act:3
16836 Replay: All jobs finished, all DS active.
16837 Main task has work to do, go find it
16838 got dsc Stop(2)
16839 stop 2
16840 Feb 15 03:00:52.071 INFO request completed, latency_us: 376, response_code: 204, uri: /stop/cid/2, method: POST[8830] Got stop action so:false kr:false
16841 , req_id: d5afd3b7-6546-4a1e-abc7-1becca65f982, remote_addr: 127.0.0.1:54744, local_addr: 127.0.0.1:9998
168422024-02-15T03:00:52.071ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
168432024-02-15T03:00:52.072ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
168442024-02-15T03:00:52.072ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
168452024-02-15T03:00:52.072ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
16846 Feb 15 03:00:52.072 INFO request completed, latency_us: 160, response_code: 200, uri: /state/cid/2, method: GET, req_id: 5ab725ce-f6cf-4199-b1e2-2f4859cfa9a6, remote_addr: 127.0.0{"msg":"[2] client re-new 0 jobs since flush 5822","v":0,"name":"crucible","level":30,"time":"2024-02-15T03:00:52.072199423Z","hostname":"ip-10-150-1-67.us-west-2.compute.internal","pid":1167,"":"downstairs","":"downstairs","session_id":"4f5a63e5-aa00-4890-9f6b-96c2e069fdb2"}
16847 .1:54744, local_addr:{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-15T03:00:52.072252986Z","hostname":"ip-10-150-1-67.us-west-2.compute.internal","pid":1167,"":"io task","client":"2","":"downstairs","session_id":"4f5a63e5-aa00-4890-9f6b-96c2e069fdb2"}
16848 127.0.0.1:9998
16849 [8830] Exited with: ExitStatus(unix_wait_status(9))
16850 [8830][2] reports Exit
168512024-02-15T03:00:53.073ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
168522024-02-15T03:00:53.073ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
168532024-02-15T03:00:53.073ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
168542024-02-15T03:00:53.073ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
168552024-02-15T03:00:53.073ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
168562024-02-15T03:00:53.073ZINFOcrucible: [2] client re-new 0 jobs since flush 5822 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
168572024-02-15T03:00:53.073ZINFOcrucible: client task is exiting = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
168582024-02-15T03:00:54.075ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
168592024-02-15T03:00:54.075ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
168602024-02-15T03:00:54.075ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
168612024-02-15T03:00:54.075ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
168622024-02-15T03:00:54.075ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
168632024-02-15T03:00:54.075ZINFOcrucible: [2] client re-new 0 jobs since flush 5822 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
168642024-02-15T03:00:54.075ZINFOcrucible: client task is exiting = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
168652024-02-15T03:00:55.075ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
168662024-02-15T03:00:55.075ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
168672024-02-15T03:00:55.075ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
168682024-02-15T03:00:55.075ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
168692024-02-15T03:00:55.075ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
168702024-02-15T03:00:55.075ZINFOcrucible: [2] client re-new 0 jobs since flush 5822 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
168712024-02-15T03:00:55.075ZINFOcrucible: client task is exiting = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
16872 Feb 15 03:00:56.075 INFO request completed, latency_us: 246, response_code: 200, uri: /state/cid/2, method: GET, req_id: f1c095e3-351a-4c4d-b669-210e2aa6cef3, remote_addr: 127.0.0.1:54744, local_addr: 127.0.0.1:9998
16873 001/300 Flush
168742024-02-15T03:00:56.076ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
168752024-02-15T03:00:56.076ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
168762024-02-15T03:00:56.076ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
168772024-02-15T03:00:56.076ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
168782024-02-15T03:00:56.076ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
168792024-02-15T03:00:56.076ZINFOcrucible: [2] client re-new 1 jobs since flush 5822 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
168802024-02-15T03:00:56.076ZINFOcrucible: client task is exiting = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
16881 002/300 Write block 4256 len 16384 data: 2 4 4 6
16882 003/300 Write block 2474 len 20480 data: 5 5 5 5 5
16883 004/300 Read block 2049 len 32768
16884 005/300 Write block 2832 len 16384 data: 2 2 2 3
16885 006/300 Read block 1047 len 16384
16886 007/300 Write block 3926 len 4096 data: 2
16887 008/300 Read block 4042 len 4096
16888 009/300 Read block 2274 len 20480
16889 010/300 Read block 4764 len 12288
16890 011/300 Read block 1302 len 20480
16891 012/300 Read block 4431 len 36864
16892 013/300 Write block 887 len 12288 data: 6 7 7
16893 014/300 Write block 4753 len 8192 data: 7 6
16894 015/300 Read block 4660 len 4096
16895 016/300 Flush
16896 017/300 Read block 1513 len 20480
16897 018/300 Read block 154 len 24576
16898 019/300 Flush
16899 020/300 Write block 4009 len 32768 data: 3 2 3 5 5 6 6 5
16900 021/300 Read block 1008 len 12288
16901 022/300 Flush
16902 023/300 Read block 3930 len 8192
16903 024/300 Write block 3376 len 24576 data: 3 3 2 2 2 2
16904 025/300 Write block 4468 len 16384 data: 7 6 5 5
16905 026/300 Write block 4337 len 32768 data: 5 4 2 2 3 3 4 5
16906 027/300 Read block 606 len 36864
16907 028/300 Read block 1206 len 20480
16908 029/300 Read block 1410 len 16384
16909 030/300 Write block 4979 len 40960 data: 3 3 4 5 5 5 5 5 5 5
16910 031/300 Write block 1436 len 28672 data: 2 3 3 3 3 3 3
16911 032/300 Read block 163 len 12288
16912 033/300 Write block 2176 len 4096 data: 3
16913 034/300 Flush
16914 035/300 Write block 1817 len 36864 data: 4 3 2 3 3 3 5 5 6
16915 036/300 Read block 3798 len 40960
16916 037/300 Read block 3338 len 24576
16917 038/300 Write block 1064 len 4096 data: 3
16918 039/300 Write block 969 len 4096 data: 4
16919 040/300 Read block 3250 len 16384
16920 041/300 Read block 3334 len 8192
16921 042/300 Read block 2305 len 40960
16922 043/300 Read block 3922 len 24576
16923 044/300 Write block 615 len 24576 data: 5 6 8 8 7 6
16924 045/300 Flush
16925 046/300 Write block 3188 len 12288 data: 5 4 5
16926 047/300 Read block 4268 len 16384
16927 048/300 Read block 439 len 8192
16928 049/300 Read block 3428 len 28672
16929 050/300 Write block 514 len 4096 data: 7
16930 051/300 Read block 644 len 20480
16931 052/300 Write block 4078 len 12288 data: 2 2 3
16932 053/300 Write block 1901 len 36864 data: 3 3 4 4 4 4 3 3 4
16933 054/300 Read block 4767 len 8192
16934 055/300 Flush
16935 056/300 Write block 1224 len 36864 data: 4 4 3 3 3 3 3 4 4
16936 057/300 Write block 2076 len 36864 data: 4 4 2 2 3 5 6 7 7
16937 058/300 Write block 4091 len 40960 data: 5 5 5 5 4 3 3 3 3 4
16938 059/300 Read block 349 len 36864
16939 060/300 Write block 3821 len 28672 data: 3 4 3 3 4 3 3
16940 061/300 Write block 4343 len 12288 data: 5 6 5
16941 062/300 Read block 548 len 28672
16942 063/300 Read block 4028 len 36864
16943 064/300 Read block 1048 len 32768
16944 065/300 Write block 380 len 12288 data: 6 4 4
16945 066/300 Read block 1476 len 12288
16946 067/300 Write block 3165 len 24576 data: 6 6 6 4 4 3
16947 068/300 Read block 3943 len 28672
16948 069/300 Read block 3777 len 12288
16949 070/300 Write block 4658 len 16384 data: 2 2 2 4
16950 071/300 Read block 2034 len 40960
16951 072/300 Write block 1829 len 28672 data: 6 5 4 3 3 3 3
16952 073/300 Write block 1246 len 40960 data: 4 4 5 4 5 5 4 4 5 7
16953 074/300 Read block 3548 len 12288
16954 075/300 Write block 3389 len 24576 data: 2 2 2 3 3 3
16955 076/300 Read block 3747 len 8192
16956 077/300 Read block 2364 len 36864
16957 078/300 Read block 166 len 4096
16958 079/300 Read block 278 len 32768
16959 080/300 Write block 3995 len 12288 data: 4 4 3
16960 081/300 Read block 3421 len 32768
16961 082/300 Write block 923 len 20480 data: 3 4 3 3 2
16962 083/300 Flush
16963 084/300 Write block 1027 len 28672 data: 4 4 4 4 4 5 5
16964 085/300 Write block 3356 len 28672 data: 4 5 5 4 4 4 5
16965 086/300 Write block 4256 len 8192 data: 3 5
16966 087/300 Flush
16967 088/300 Write block 3385 len 28672 data: 2 2 2 2 3 3 3
16968 089/300 Flush
16969 090/300 Write block 4446 len 32768 data: 5 4 4 7 7 4 3 3
16970 091/300 Write block 3374 len 16384 data: 3 3 4 4
16971 092/300 Write block 3243 len 36864 data: 5 6 5 5 4 5 4 4 4
16972 093/300 Flush
16973 094/300 Write block 3512 len 36864 data: 5 4 2 2 2 3 2 4 4
16974 095/300 Write block 3260 len 12288 data: 4 3 4
16975 096/300 Read block 4466 len 36864
16976 097/300 Read block 3563 len 40960
16977 098/300 Read block 3119 len 28672
16978 099/300 Write block 1889 len 20480 data: 3 3 3 3 2
16979 100/300 Read block 1763 len 24576
16980 101/300 Read block 3942 len 20480
16981 102/300 Write block 628 len 28672 data: 4 5 6 7 7 6 5
16982 103/300 Flush
16983 104/300 Read block 2026 len 28672
16984 105/300 Write block 1286 len 24576 data: 5 5 6 6 5 5
16985 106/300 Flush
16986 107/300 Read block 3981 len 36864
16987 108/300 Write block 3524 len 12288 data: 6 5 6
16988 109/300 Write block 3201 len 28672 data: 3 3 3 3 3 3 5
16989 110/300 Read block 479 len 20480
16990 111/300 Write block 1907 len 40960 data: 4 4 5 4 4 4 3 3 3 4
16991 112/300 Read block 900 len 4096
16992 113/300 Write block 3904 len 28672 data: 4 4 5 6 6 6 5
16993 114/300 Read block 1059 len 36864
16994 115/300 Read block 2274 len 36864
16995 116/300 Read block 201 len 28672
16996 117/300 Read block 4969 len 36864
16997 118/300 Write block 1865 len 36864 data: 2 2 2 3 3 3 3 4 5
16998 119/300 Read block 1223 len 8192
16999 120/300 Read block 3352 len 8192
17000 121/300 Write block 2729 len 12288 data: 3 3 4
17001 122/300 Read block 2210 len 20480
17002 123/300 Read block 2861 len 8192
17003 124/300 Read block 4697 len 16384
17004 125/300 Read block 2910 len 24576
17005 126/300 Read block 634 len 28672
17006 127/300 Flush
17007 128/300 Read block 3488 len 20480
17008 129/300 Write block 3934 len 40960 data: 2 2 2 2 3 3 3 3 3 3
17009 130/300 Write block 166 len 4096 data: 5
17010 131/300 Read block 3871 len 16384
17011 132/300 Write block 2326 len 8192 data: 4 4
17012 133/300 Read block 443 len 20480
17013 134/300 Flush
17014 135/300 Read block 4195 len 16384
17015 136/300 Read block 2008 len 16384
17016 137/300 Flush
17017 138/300 Write block 3881 len 32768 data: 6 6 6 5 4 4 3 3
17018 139/300 Write block 1259 len 12288 data: 6 6 4
17019 140/300 Write block 2602 len 4096 data: 5
17020 141/300 Read block 4224 len 32768
17021 142/300 Read block 2065 len 40960
17022 143/300 Read block 3017 len 24576
17023 144/300 Write block 1744 len 36864 data: 3 3 3 3 3 4 3 2 2
17024 145/300 Write block 2140 len 4096 data: 3
17025 146/300 Write block 2863 len 36864 data: 5 6 5 6 6 5 3 3 3
17026 147/300 Read block 4679 len 12288
17027 148/300 Write block 4394 len 24576 data: 5 5 3 2 2 2
17028 149/300 Write block 4048 len 32768 data: 7 5 6 6 6 6 6 6
17029 150/300 Write block 1104 len 28672 data: 3 3 4 4 4 3 3
17030 151/300 Read block 4499 len 36864
17031 152/300 Read block 4488 len 20480
17032 153/300 Read block 3620 len 16384
17033 154/300 Write block 1184 len 12288 data: 3 4 4
17034 155/300 Write block 2443 len 40960 data: 2 3 3 5 5 6 5 4 4 4
17035 156/300 Read block 4758 len 32768
17036 157/300 Read block 1940 len 8192
17037 158/300 Read block 1847 len 4096
17038 159/300 Read block 1182 len 20480
17039 160/300 Write block 1144 len 32768 data: 3 3 3 3 3 4 5 5
17040 161/300 Write block 1536 len 40960 data: 3 4 5 5 4 4 4 4 6 6
17041 162/300 Read block 123 len 40960
17042 163/300 Write block 1612 len 40960 data: 3 3 3 3 3 3 3 2 3 4
17043 164/300 Write block 1372 len 4096 data: 3
17044 165/300 Read block 598 len 16384
17045 166/300 Read block 2569 len 8192
17046 167/300 Flush
17047 168/300 Write block 4100 len 12288 data: 5 3 3
17048 169/300 Write block 4359 len 12288 data: 4 3 4
17049 170/300 Read block 4535 len 36864
17050 171/300 Write block 4784 len 20480 data: 4 4 2 2 4
17051 172/300 Write block 4482 len 24576 data: 4 4 4 4 4 3
17052 173/300 Write block 4686 len 40960 data: 4 6 5 6 6 7 7 6 6 5
17053 174/300 Write block 177 len 12288 data: 6 5 4
17054 175/300 Read block 1893 len 36864
17055 176/300 Read block 1753 len 4096
17056 177/300 Write block 1493 len 20480 data: 5 5 6 5 3
17057 178/300 Write block 4804 len 36864 data: 4 4 2 2 2 3 4 4 5
17058 179/300 Read block 4106 len 28672
17059 180/300 Read block 1103 len 36864
17060 181/300 Read block 1093 len 32768
17061 182/300 Flush
17062 183/300 Read block 1281 len 40960
17063 184/300 Read block 1957 len 36864
17064 185/300 Flush
17065 186/300 Read block 4931 len 16384
17066 187/300 Flush
17067 188/300 Read block 4005 len 8192
17068 189/300 Flush
17069 190/300 Read block 873 len 4096
17070 191/300 Write block 2166 len 8192 data: 5 6
17071 192/300 Write block 2364 len 12288 data: 3 3 4
17072 193/300 Flush
17073 194/300 Read block 2010 len 32768
17074 195/300 Write block 192 len 8192 data: 3 4
17075 196/300 Write block 3445 len 36864 data: 3 3 4 5 5 6 5 4 4
17076 197/300 Read block 542 len 40960
17077 198/300 Write block 4366 len 32768 data: 4 3 2 2 2 4 3 4
17078 199/300 Write block 829 len 36864 data: 3 2 2 3 4 5 4 5 5
17079 200/300 Read block 4951 len 12288
17080 201/300 Flush
17081 202/300 Write block 2854 len 20480 data: 3 3 5 5 5
17082 203/300 Write block 4015 len 40960 data: 7 6 6 5 6 7 6 6 7 7
17083 204/300 Read block 1514 len 32768
17084 205/300 Read block 72 len 12288
17085 206/300 Read block 1657 len 24576
17086 207/300 Read block 912 len 4096
17087 208/300 Write block 3204 len 24576 data: 4 4 4 6 5 4
17088 209/300 Read block 108 len 40960
17089 210/300 Read block 4506 len 20480
17090 211/300 Read block 3254 len 12288
17091 212/300 Write block 2063 len 4096 data: 4
17092 213/300 Read block 1279 len 32768
17093 214/300 Write block 1770 len 8192 data: 3 3
17094 215/300 Read block 1406 len 20480
17095 216/300 Write block 4924 len 16384 data: 4 4 4 5
17096 217/300 Write block 4594 len 36864 data: 5 5 5 3 2 2 2 3 4
17097 218/300 Read block 1948 len 24576
17098 219/300 Flush
17099 220/300 Read block 1547 len 8192
17100 221/300 Read block 1165 len 16384
17101 222/300 Read block 2881 len 32768
17102 223/300 Read block 2457 len 28672
17103 224/300 Read block 252 len 40960
17104 225/300 Flush
17105 226/300 Write block 3235 len 36864 data: 2 3 3 3 4 7 7 5 6
17106 227/300 Read block 275 len 28672
17107 228/300 Write block 4206 len 40960 data: 6 5 3 3 4 3 3 4 5 5
17108 229/300 Read block 2253 len 8192
17109 230/300 Read block 117 len 40960
17110 231/300 Read block 4129 len 12288
17111 232/300 Write block 2670 len 16384 data: 4 3 3 4
17112 233/300 Flush
17113 234/300 Read block 1744 len 20480
17114 235/300 Read block 4008 len 4096
17115 236/300 Write block 4042 len 40960 data: 6 6 7 6 5 7 8 6 7 7
17116 237/300 Write block 2913 len 4096 data: 5
17117 238/300 Read block 1451 len 32768
17118 239/300 Read block 840 len 36864
17119 240/300 Write block 3228 len 12288 data: 3 3 3
17120 241/300 Read block 2433 len 32768
17121 242/300 Read block 319 len 4096
17122 243/300 Write block 4875 len 36864 data: 3 3 3 2 2 2 3 3 3
17123 244/300 Read block 820 len 32768
17124 245/300 Read block 2046 len 36864
17125 246/300 Read block 1802 len 4096
17126 247/300 Read block 3689 len 8192
17127 248/300 Flush
17128 249/300 Flush
17129 250/300 Read block 874 len 12288
17130 251/300 Read block 2118 len 16384
17131 252/300 Read block 3248 len 16384
17132 253/300 Read block 2830 len 12288
17133 254/300 Read block 3041 len 12288
17134 255/300 Flush
17135 256/300 Write block 3817 len 20480 data: 2 2 2 3 4
17136 257/300 Flush
17137 258/300 Read block 3828 len 20480
17138 259/300 Write block 3716 len 40960 data: 6 4 4 3 4 3 3 3 3 3
17139 260/300 Read block 3597 len 32768
17140 261/300 Write block 228 len 40960 data: 4 5 5 6 5 5 5 5 6 5
17141 262/300 Write block 4608 len 8192 data: 3 2
17142 263/300 Write block 3116 len 20480 data: 5 3 3 2 2
17143 264/300 Read block 2712 len 8192
17144 265/300 Flush
17145 266/300 Flush
17146 267/300 Read block 2855 len 40960
17147 268/300 Write block 72 len 36864 data: 6 6 5 5 4 4 4 3 3
17148 269/300 Write block 4291 len 36864 data: 2 2 2 2 2 2 2 3 2
17149 270/300 Read block 529 len 24576
17150 271/300 Flush
17151 272/300 Write block 1784 len 12288 data: 5 6 6
17152 273/300 Flush
17153 274/300 Read block 536 len 40960
17154 275/300 Write block 4068 len 32768 data: 5 4 4 4 4 4 3 3
17155 276/300 Read block 3349 len 8192
17156 277/300 Write block 3327 len 8192 data: 6 6
17157 278/300 Write block 3687 len 24576 data: 3 3 3 5 5 5
17158 279/300 Write block 4948 len 20480 data: 6 6 6 6 6
17159 280/300 Write block 3814 len 24576 data: 4 4 3 3 3 3
17160 281/300 Read block 1863 len 20480
17161 282/300 Write block 1497 len 40960 data: 4 4 3 4 4 3 3 5 4 6
17162 283/300 Write block 4747 len 36864 data: 3 5 5 6 6 6 8 7 5
17163 284/300 Read block 4242 len 28672
17164 285/300 Read block 1351 len 20480
17165 286/300 Flush
17166 287/300 Flush
17167 288/300 Read block 1261 len 28672
17168 289/300 Write block 2034 len 4096 data: 2
17169 290/300 Read block 1338 len 24576
17170 291/300 Flush
17171 292/300 Read block 4534 len 28672
17172 293/300 Write block 2661 len 28672 data: 5 3 4 6 5 5 4
17173 294/300 Write block 2062 len 36864 data: 5 5 5 5 5 5 5 5 5
17174 295/300 Read block 4637 len 16384
17175 296/300 Read block 3418 len 4096
17176 297/300 Read block 3514 len 24576
17177 298/300 Write block 2117 len 36864 data: 4 3 5 5 5 5 6 6 5
17178 299/300 Flush
17179 300/300 Write block 1882 len 12288 data: 3 4 5
17180 Main task has work to do, go find it
17181 got dsc Start(2)
17182 start 2
17183 Feb 15 03:00:56.418 INFO request completed, latency_us: 207, response_code: 204, uri: /start/cid/2, method: POST, req_id: 81e28b27-0c07-4beb-9d10-e6dca81197d6, remote_addr: 127.0.0.1:54744, local_addr: 127.0.0.1:9998
17184 Replay: started 2, returned:Ok(())
17185 ----------------------------------------------------------------
17186 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:300
17187 States: Active Active Offline
17188 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
17189 4824 Acked 5823 Flush 0 Done Done New true
17190 4825 Acked 5824 Write 4 Done Done New false
17191 4826 Acked 5825 Write 5 Done Done New false
17192 4827 Acked 5826 Read 8 Done Done New false
17193 4828 Acked 5827 Write 4 Done Done New false
17194 4829 Acked 5828 Read 4 Done Done New false
17195 4830 Acked 5829 Write 1 Done Done New false
17196 4831 Acked 5830 Read 1 Done Done New false
17197 4832 Acked 5831 Read 5 Done Done New false
17198 4833 Acked 5832 Read 3 Done Done New false
17199 4834 Acked 5833 Read 5 Done Done New false
17200 4835 Acked 5834 Read 9 Done Done New false
17201 4836 Acked 5835 Write 3 Done Done New false
17202 4837 Acked 5836 Write 2 Done Done New false
17203 4838 Acked 5837 Read 1 Done Done New false
17204 4839 Acked 5838 Flush 0 Done Done New false
17205 4840 Acked 5839 Read 5 Done Done New false
17206 4841 Acked 5840 Read 6 Done Done New false
17207 4842 Acked 5841 Flush 0 Done Done New false
17208 4843 Acked 5842 Write 8 Done Done New false
17209 4844 Acked 5843 Read 3 Done Done New false
17210 4845 Acked 5844 Flush 0 Done Done New false
17211 4846 Acked 5845 Read 2 Done Done New false
17212 4847 Acked 5846 Write 6 Done Done New false
17213 4848 Acked 5847 Write 4 Done Done New false
17214 4849 Acked 5848 Write 8 Done Done New false
17215 4850 Acked 5849 Read 9 Done Done New false
17216 4851 Acked 5850 Read 5 Done Done New false
17217 4852 Acked 5851 Read 4 Done Done New false
17218 4853 Acked 5852 Write 10 Done Done New false
17219 4854 Acked 5853 Write 7 Done Done New false
17220 4855 Acked 5854 Read 3 Done Done New false
17221 4856 Acked 5855 Write 1 Done Done New false
17222 4857 Acked 5856 Flush 0 Done Done New false
17223 4858 Acked 5857 Write 9 Done Done New false
17224 4859 Acked 5858 Read 10 Done Done New false
17225 4860 Acked 5859 Read 6 Done Done New false
17226 4861 Acked 5860 Write 1 Done Done New false
17227 4862 Acked 5861 Write 1 Done Done New false
17228 4863 Acked 5862 Read 4 Done Done New false
17229 4864 Acked 5863 Read 2 Done Done New false
17230 4865 Acked 5864 Read 10 Done Done New false
17231 4866 Acked 5865 Read 6 Done Done New false
17232 4867 Acked 5866 Write 6 Done Done New false
17233 4868 Acked 5867 Flush 0 Done Done New false
17234 4869 Acked 5868 Write 3 Done Done New false
17235 4870 Acked 5869 Read 4 Done Done New false
17236 4871 Acked 5870 Read 2 Done Done New false
17237 4872 Acked 5871 Read 7 Done Done New false
17238 4873 Acked 5872 Write 1 Done Done New false
17239 4874 Acked 5873 Read 5 Done Done New false
17240 4875 Acked 5874 Write 3 Done Done New false
17241 4876 Acked 5875 Write 9 Done Done New false
17242 4877 Acked 5876 Read 2 Done Done New false
17243 4878 Acked 5877 Flush 0 Done Done New false
17244 4879 Acked 5878 Write 9 Done Done New false
17245 4880 Acked 5879 Write 9 Done Done New false
17246 4881 Acked 5880 Write 10 Done Done New false
17247 4882 Acked 5881 Read 9 Done Done New false
17248 4883 Acked 5882 Write 7 Done Done New false
17249 4884 Acked 5883 Write 3 Done Done New false
17250 4885 Acked 5884 Read 7 Done Done New false
17251 4886 Acked 5885 Read 9 Done Done New false
17252 4887 Acked 5886 Read 8 Done Done New false
17253 4888 Acked 5887 Write 3 Done Done New false
17254 4889 Acked 5888 Read 3 Done Done New false
17255 4890 Acked 5889 Write 6 Done Done New false
17256 4891 Acked 5890 Read 7 Done Done New false
17257 4892 Acked 5891 Read 3 Done Done New false
17258 4893 Acked 5892 Write 4 Done Done New false
17259 4894 Acked 5893 Read 10 Done Done New false
17260 4895 Acked 5894 Write 7 Done Done New false
17261 4896 Acked 5895 Write 10 Done Done New false
17262 4897 Acked 5896 Read 3 Done Done New false
17263 4898 Acked 5897 Write 6 Done Done New false
17264 4899 Acked 5898 Read 2 Done Done New false
17265 4900 Acked 5899 Read 9 Done Done New false
17266 4901 Acked 5900 Read 1 Done Done New false
17267 4902 Acked 5901 Read 8 Done Done New false
17268 4903 Acked 5902 Write 3 Done Done New false
17269 4904 Acked 5903 Read 8 Done Done New false
17270 4905 Acked 5904 Write 5 Done Done New false
17271 4906 Acked 5905 Flush 0 Done Done New false
17272 4907 Acked 5906 Write 7 Done Done New false
17273 4908 Acked 5907 Write 7 Done Done New false
17274 4909 Acked 5908 Write 2 Done Done New false
17275 4910 Acked 5909 Flush 0 Done Done New false
17276 4911 Acked 5910 Write 7 Done Done New false
17277 4912 Acked 5911 Flush 0 Done Done New false
17278 4913 Acked 5912 Write 8 Done Done New false
17279 4914 Acked 5913 Write 4 Done Done New false
17280 4915 Acked 5914 Write 9 Done Done New false
17281 4916 Acked 5915 Flush 0 Done Done New false
17282 4917 Acked 5916 Write 9 Done Done New false
17283 4918 Acked 5917 Write 3 Done Done New false
17284 4919 Acked 5918 Read 9 Done Done New false
17285 4920 Acked 5919 Read 10 Done Done New false
17286 4921 Acked 5920 Read 7 Done Done New false
17287 4922 Acked 5921 Write 5 Done Done New false
17288 4923 Acked 5922 Read 6 Done Done New false
17289 4924 Acked 5923 Read 5 Done Done New false
17290 4925 Acked 5924 Write 7 Done Done New false
17291 4926 Acked 5925 Flush 0 Done Done New false
17292 4927 Acked 5926 Read 7 Done Done New false
17293 4928 Acked 5927 Write 6 Done Done New false
17294 4929 Acked 5928 Flush 0 Done Done New false
17295 4930 Acked 5929 Read 9 Done Done New false
17296 4931 Acked 5930 Write 3 Done Done New false
17297 4932 Acked 5931 Write 7 Done Done New false
17298 4933 Acked 5932 Read 5 Done Done New false
17299 4934 Acked 5933 Write 10 Done Done New false
17300 4935 Acked 5934 Read 1 Done Done New false
17301 4936 Acked 5935 Write 7 Done Done New false
17302 4937 Acked 5936 Read 9 Done Done New false
17303 4938 Acked 5937 Read 9 Done Done New false
17304 4939 Acked 5938 Read 7 Done Done New false
17305 4940 Acked 5939 Read 9 Done Done New false
17306 4941 Acked 5940 Write 9 Done Done New false
17307 4942 Acked 5941 Read 2 Done Done New false
17308 4943 Acked 5942 Read 2 Done Done New false
17309 4944 Acked 5943 Write 3 Done Done New false
17310 4945 Acked 5944 Read 5 Done Done New false
17311 4946 Acked 5945 Read 2 Done Done New false
17312 4947 Acked 5946 Read 4 Done Done New false
17313 4948 Acked 5947 Read 6 Done Done New false
17314 4949 Acked 5948 Read 7 Done Done New false
17315 4950 Acked 5949 Flush 0 Done Done New false
17316 4951 Acked 5950 Read 5 Done Done New false
17317 4952 Acked 5951 Write 10 Done Done New false
17318 4953 Acked 5952 Write 1 Done Done New false
17319 4954 Acked 5953 Read 4 Done Done New false
17320 4955 Acked 5954 Write 2 Done Done New false
17321 4956 Acked 5955 Read 5 Done Done New false
17322 4957 Acked 5956 Flush 0 Done Done New false
17323 4958 Acked 5957 Read 4 Done Done New false
17324 4959 Acked 5958 Read 4 Done Done New false
17325 4960 Acked 5959 Flush 0 Done Done New false
17326 4961 Acked 5960 Write 8 Done Done New false
17327 4962 Acked 5961 Write 3 Done Done New false
17328 4963 Acked 5962 Write 1 Done Done New false
17329 4964 Acked 5963 Read 8 Done Done New false
17330 4965 Acked 5964 Read 10 Done Done New false
17331 4966 Acked 5965 Read 6 Done Done New false
17332 4967 Acked 5966 Write 9 Done Done New false
17333 4968 Acked 5967 Write 1 Done Done New false
17334 4969 Acked 5968 Write 9 Done Done New false
17335 4970 Acked 5969 Read 3 Done Done New false
17336 4971 Acked 5970 Write 6 Done Done New false
17337 4972 Acked 5971 Write 8 Done Done New false
17338 4973 Acked 5972 Write 7 Done Done New false
17339 4974 Acked 5973 Read 9 Done Done New false
17340 4975 Acked 5974 Read 5 Done Done New false
17341 4976 Acked 5975 Read 4 Done Done New false
17342 4977 Acked 5976 Write 3 Done Done New false
17343 4978 Acked 5977 Write 10 Done Done New false
17344 4979 Acked 5978 Read 8 Done Done New false
17345 4980 Acked 5979 Read 2 Done Done New false
17346 4981 Acked 5980 Read 1 Done Done New false
17347 4982 Acked 5981 Read 5 Done Done New false
17348 4983 Acked 5982 Write 8 Done Done New false
17349 4984 Acked 5983 Write 10 Done Done New false
17350 4985 Acked 5984 Read 10 Done Done New false
17351 4986 Acked 5985 Write 10 Done Done New false
17352 4987 Acked 5986 Write 1 Done Done New false
17353 4988 Acked 5987 Read 4 Done Done New false
17354 4989 Acked 5988 Read 2 Done Done New false
17355 4990 Acked 5989 Flush 0 Done Done New false
17356 4991 Acked 5990 Write 3 Done Done New false
17357 4992 Acked 5991 Write 3 Done Done New false
17358 4993 Acked 5992 Read 9 Done Done New false
17359 4994 Acked 5993 Write 5 Done Done New false
17360 4995 Acked 5994 Write 6 Done Done New false
17361 4996 Acked 5995 Write 10 Done Done New false
17362 4997 Acked 5996 Write 3 Done Done New false
17363 4998 Acked 5997 Read 9 Done Done New false
17364 4999 Acked 5998 Read 1 Done Done New false
17365 5000 Acked 5999 Write 5 Done Done New false
17366 5001 Acked 6000 Write 9 Done Done New false
17367 5002 Acked 6001 Read 7 Done Done New false
17368 5003 Acked 6002 Read 9 Done Done New false
17369 5004 Acked 6003 Read 8 Done Done New false
17370 5005 Acked 6004 Flush 0 Done Done New false
17371 5006 Acked 6005 Read 10 Done Done New false
17372 5007 Acked 6006 Read 9 Done Done New false
17373 5008 Acked 6007 Flush 0 Done Done New false
17374 5009 Acked 6008 Read 4 Done Done New false
17375 5010 Acked 6009 Flush 0 Done Done New false
17376 5011 Acked 6010 Read 2 Done Done New false
17377 5012 Acked 6011 Flush 0 Done Done New false
17378 5013 Acked 6012 Read 1 Done Done New false
17379 5014 Acked 6013 Write 2 Done Done New false
17380 5015 Acked 6014 Write 3 Done Done New false
17381 5016 Acked 6015 Flush 0 Done Done New false
17382 5017 Acked 6016 Read 8 Done Done New false
17383 5018 Acked 6017 Write 2 Done Done New false
17384 5019 Acked 6018 Write 9 Done Done New false
17385 5020 Acked 6019 Read 10 Done Done New false
17386 5021 Acked 6020 Write 8 Done Done New false
17387 5022 Acked 6021 Write 9 Done Done New false
17388 5023 Acked 6022 Read 3 Done Done New false
17389 5024 Acked 6023 Flush 0 Done Done New false
17390 5025 Acked 6024 Write 5 Done Done New false
17391 5026 Acked 6025 Write 10 Done Done New false
17392 5027 Acked 6026 Read 8 Done Done New false
17393 5028 Acked 6027 Read 3 Done Done New false
17394 5029 Acked 6028 Read 6 Done Done New false
17395 5030 Acked 6029 Read 1 Done Done New false
17396 5031 Acked 6030 Write 6 Done Done New false
17397 5032 Acked 6031 Read 10 Done Done New false
17398 5033 Acked 6032 Read 5 Done Done New false
17399 5034 Acked 6033 Read 3 Done Done New false
17400 5035 Acked 6034 Write 1 Done Done New false
17401 5036 Acked 6035 Read 8 Done Done New false
17402 5037 Acked 6036 Write 2 Done Done New false
17403 5038 Acked 6037 Read 5 Done Done New false
17404 5039 Acked 6038 Write 4 Done Done New false
17405 5040 Acked 6039 Write 9 Done Done New false
17406 5041 Acked 6040 Read 6 Done Done New false
17407 5042 Acked 6041 Flush 0 Done Done New false
17408 5043 Acked 6042 Read 2 Done Done New false
17409 5044 Acked 6043 Read 4 Done Done New false
17410 5045 Acked 6044 Read 8 Done Done New false
17411 5046 Acked 6045 Read 7 Done Done New false
17412 5047 Acked 6046 Read 10 Done Done New false
17413 5048 Acked 6047 Flush 0 Done Done New false
17414 5049 Acked 6048 Write 9 Done Done New false
17415 5050 Acked 6049 Read 7 Done Done New false
17416 5051 Acked 6050 Write 10 Done Done New false
17417 5052 Acked 6051 Read 2 Done Done New false
17418 5053 Acked 6052 Read 10 Done Done New false
17419 5054 Acked 6053 Read 3 Done Done New false
17420 5055 Acked 6054 Write 4 Done Done New false
17421 5056 Acked 6055 Flush 0 Done Done New false
17422 5057 Acked 6056 Read 5 Done Done New false
17423 5058 Acked 6057 Read 1 Done Done New false
17424 5059 Acked 6058 Write 10 Done Done New false
17425 5060 Acked 6059 Write 1 Done Done New false
17426 5061 Acked 6060 Read 8 Done Done New false
17427 5062 Acked 6061 Read 9 Done Done New false
17428 5063 Acked 6062 Write 3 Done Done New false
17429 5064 Acked 6063 Read 8 Done Done New false
17430 5065 Acked 6064 Read 1 Done Done New false
17431 5066 Acked 6065 Write 9 Done Done New false
17432 5067 Acked 6066 Read 8 Done Done New false
17433 5068 Acked 6067 Read 9 Done Done New false
17434 5069 Acked 6068 Read 1 Done Done New false
17435 5070 Acked 6069 Read 2 Done Done New false
17436 5071 Acked 6070 Flush 0 Done Done New false
17437 5072 Acked 6071 Flush 0 Done Done New false
17438 5073 Acked 6072 Read 3 Done Done New false
17439 5074 Acked 6073 Read 4 Done Done New false
17440 5075 Acked 6074 Read 4 Done Done New false
17441 5076 Acked 6075 Read 3 Done Done New false
17442 5077 Acked 6076 Read 3 Done Done New false
17443 5078 Acked 6077 Flush 0 Done Done New false
17444 5079 Acked 6078 Write 5 Done Done New false
17445 5080 Acked 6079 Flush 0 Done Done New false
17446 5081 Acked 6080 Read 5 Done Done New false
17447 5082 Acked 6081 Write 10 Done Done New false
17448 5083 Acked 6082 Read 8 Done Done New false
17449 5084 Acked 6083 Write 10 Done Done New false
17450 5085 Acked 6084 Write 2 Done Done New false
17451 5086 Acked 6085 Write 5 Done Done New false
17452 5087 Acked 6086 Read 2 Done Done New false
17453 5088 Acked 6087 Flush 0 Done Done New false
17454 5089 Acked 6088 Flush 0 Done Done New false
17455 5090 Acked 6089 Read 10 Done Done New false
17456 5091 Acked 6090 Write 9 Done Done New false
17457 5092 Acked 6091 Write 9 Done Done New false
17458 5093 Acked 6092 Read 6 Done Done New false
17459 5094 Acked 6093 Flush 0 Done Done New false
17460 5095 Acked 6094 Write 3 Done Done New false
17461 5096 Acked 6095 Flush 0 Done Done New false
17462 5097 Acked 6096 Read 10 Done Done New false
17463 5098 Acked 6097 Write 8 Done Done New false
17464 5099 Acked 6098 Read 2 Done Done New false
17465 5100 Acked 6099 Write 2 Done Done New false
17466 5101 Acked 6100 Write 6 Done Done New false
17467 5102 Acked 6101 Write 5 Done Done New false
17468 5103 Acked 6102 Write 6 Done Done New false
17469 5104 Acked 6103 Read 5 Done Done New false
17470 5105 Acked 6104 Write 10 Done Done New false
17471 5106 Acked 6105 Write 9 Done Done New false
17472 5107 Acked 6106 Read 7 Done Done New false
17473 5108 Acked 6107 Read 5 Done Done New false
17474 5109 Acked 6108 Flush 0 Done Done New false
17475 5110 Acked 6109 Flush 0 Done Done New false
17476 5111 Acked 6110 Read 7 Done Done New false
17477 5112 Acked 6111 Write 1 Done Done New false
17478 5113 Acked 6112 Read 6 Done Done New false
17479 5114 Acked 6113 Flush 0 Done Done New false
17480 5115 Acked 6114 Read 7 Done Done New false
17481 5116 Acked 6115 Write 7 Done Done New false
17482 5117 Acked 6116 Write 9 Done Done New false
17483 5118 Acked 6117 Read 4 Done Done New false
17484 5119 Acked 6118 Read 1 Done Done New false
17485 5120 Acked 6119 Read 6 Done Done New false
17486 5121 Acked 6120 Write 9 Done Done New false
17487 5122 Acked 6121 Flush 0 Done Done New false
17488 5123 Acked 6122 Write 3 Done Done New false
17489 STATES DS:0 DS:1 DS:2 TOTAL
17490 New 0 0 300 300
17491 Sent 0 0 0 0
17492 Done 300 300 0 600
17493 Skipped 0 0 0 0
17494 Error 0 0 0 0
17495 Last Flush: 6121 6121 5822
17496 Downstairs last five completed: 5822 5821 5820 5819 5818
17497 Upstairs last five completed: 5123 5122 5121 5120 5119
17498 CLIENT: Up:0 ds:300 act:2
174992024-02-15T03:00:57.078ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
175002024-02-15T03:00:57.078ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
175012024-02-15T03:00:57.078ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
175022024-02-15T03:00:57.078ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
175032024-02-15T03:00:57.078ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
175042024-02-15T03:00:57.078ZINFOcrucible: [2] client re-new 301 jobs since flush 5822 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
175052024-02-15T03:00:57.078ZINFOcrucible: client task is exiting = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
175062024-02-15T03:00:58.079ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
175072024-02-15T03:00:58.079ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
175082024-02-15T03:00:58.079ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
175092024-02-15T03:00:58.079ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
175102024-02-15T03:00:58.079ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
175112024-02-15T03:00:58.079ZINFOcrucible: [2] client re-new 301 jobs since flush 5822 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
17512 [8830] got start action
17513 [8830] I am going to restart
17514 Starting downstairs at port 8830
175152024-02-15T03:00:58.080ZINFOcrucible: client task is exiting = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
17516 Make output file at "/tmp/dsc/downstairs-8830.txt"
17517 [8830][2] reports Starting
17518 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1184)
17519 [8830][2] reports Running
175202024-02-15T03:00:59.081ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
175212024-02-15T03:00:59.081ZINFOcrucible: ds_connection connected = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
175222024-02-15T03:00:59.082ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
175232024-02-15T03:00:59.082ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
175242024-02-15T03:00:59.082ZINFOcrucible: send last flush ID to this DS: 5822 = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
175252024-02-15T03:00:59.083ZINFOcrucible: Replied this last flush ID: 5822 = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
175262024-02-15T03:00:59.083ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
175272024-02-15T03:00:59.083ZINFOcrucible: [2] Transition from Offline to Replay = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
175282024-02-15T03:00:59.083ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
175292024-02-15T03:00:59.083ZINFOcrucible: [2] Transition from Replay to Active = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
17530 ----------------------------------------------------------------
17531 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
17532 Downstairs last five completed: 6123 6122 6121 6120 6119
17533 Upstairs last five completed: 5124 5123 5122 5121 5120
17534 CLIENT: Up:0 ds:0 act:3
17535 Replay: All jobs finished, all DS active.
17536 Main task has work to do, go find it
17537 got dsc Stop(1)
17538 stop 1
17539 Feb 15 03:01:00.422 INFO request completed[8820] Got stop action so:false kr:false
17540 , latency_us: 352, response_code: 204, uri: /stop/cid/1, method: POST, req_id: 7232ae16-4d63-47cb-a707-96ada944d459, remote_addr: 127.0.0.1:54744, local_addr: 127.0.0.1:9998
175412024-02-15T03:01:00.423ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
175422024-02-15T03:01:00.423ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
175432024-02-15T03:01:00.423ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
175442024-02-15T03:01:00.423ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
175452024-02-15T03:01:00.423ZINFOcrucible: [1] client re-new 0 jobs since flush 6123 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
175462024-02-15T03:01:00.423ZINFOcrucible: client task is exiting = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
17547 Feb 15 03:01:00.423 INFO request completed, latency_us: 126, response_code: 200, uri: /state/cid/1, method: GET, req_id: 4dca3cf4-1006-4ddb-b357-0b2a1fbab7a7, remote_addr: 127.0.0.1:54744, local_addr: 127.0.0.1:9998
17548 [8820] Exited with: ExitStatus(unix_wait_status(9))
17549 [8820][1] reports Exit
175502024-02-15T03:01:01.425ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
175512024-02-15T03:01:01.425ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
175522024-02-15T03:01:01.425ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
175532024-02-15T03:01:01.425ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
175542024-02-15T03:01:01.425ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
175552024-02-15T03:01:01.425ZINFOcrucible: [1] client re-new 0 jobs since flush 6123 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
175562024-02-15T03:01:01.425ZINFOcrucible: client task is exiting = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
175572024-02-15T03:01:02.427ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
175582024-02-15T03:01:02.427ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
175592024-02-15T03:01:02.427ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
175602024-02-15T03:01:02.427ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
175612024-02-15T03:01:02.427ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
175622024-02-15T03:01:02.427ZINFOcrucible: [1] client re-new 0 jobs since flush 6123 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
175632024-02-15T03:01:02.427ZINFOcrucible: client task is exiting = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
175642024-02-15T03:01:03.429ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
175652024-02-15T03:01:03.429ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
175662024-02-15T03:01:03.429ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
175672024-02-15T03:01:03.429ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
175682024-02-15T03:01:03.429ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
175692024-02-15T03:01:03.429ZINFOcrucible: [1] client re-new 0 jobs since flush 6123 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
175702024-02-15T03:01:03.429ZINFOcrucible: client task is exiting = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
17571 Feb 15 03:01:04.425 INFO request completed, latency_us: 224, response_code: 200, uri: /state/cid/1, method: GET, req_id: 09d25e99-8573-43cc-967e-96520d9c7b0c, remote_addr: 127.0.0.1:54744, local_addr: 127.0.0.1:9998
17572 001/300 Read block 2257 len 12288
17573 002/300 Write block 1644 len 36864 data: 9 8 8 7 7 8 6 6 7
17574 003/300 Write block 4450 len 8192 data: 8 5
17575 004/300 Read block 3590 len 28672
17576 005/300 Write block 2346 len 16384 data: 6 5 3 3
175772024-02-15T03:01:04.430ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
175782024-02-15T03:01:04.430ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
175792024-02-15T03:01:04.430ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
175802024-02-15T03:01:04.430ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
175812024-02-15T03:01:04.430ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
175822024-02-15T03:01:04.430ZINFOcrucible: [1] client re-new 4 jobs since flush 6123 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
17583 006/300 Read block 209 len 32768
175842024-02-15T03:01:04.430ZINFOcrucible: client task is exiting = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
17585 007/300 Read block 4752 len 32768
17586 008/300 Read block 1908 len 28672
17587 009/300 Read block 636 len 28672
17588 010/300 Read block 3893 len 36864
17589 011/300 Read block 1366 len 12288
17590 012/300 Write block 1799 len 8192 data: 6 5
17591 013/300 Flush
17592 014/300 Read block 17 len 36864
17593 015/300 Read block 4806 len 8192
17594 016/300 Read block 2279 len 20480
17595 017/300 Write block 2170 len 32768 data: 5 5 5 4 4 4 4 3
17596 018/300 Flush
17597 019/300 Write block 4907 len 28672 data: 7 6 5 6 6 5 4
17598 020/300 Write block 3243 len 16384 data: 7 7 6 6
17599 021/300 Read block 1430 len 24576
17600 022/300 Write block 3786 len 24576 data: 6 5 4 5 4 3
17601 023/300 Write block 3287 len 16384 data: 4 4 4 4
17602 024/300 Read block 906 len 8192
17603 025/300 Write block 4769 len 8192 data: 6 5
17604 026/300 Write block 3418 len 20480 data: 3 3 3 4 3
17605 027/300 Write block 2984 len 20480 data: 5 5 5 5 5
17606 028/300 Read block 3932 len 24576
17607 029/300 Read block 1048 len 40960
17608 030/300 Read block 4267 len 4096
17609 031/300 Read block 2000 len 24576
17610 032/300 Read block 1737 len 12288
17611 033/300 Write block 3935 len 28672 data: 3 3 3 4 4 4 4
17612 034/300 Write block 553 len 8192 data: 4 5
17613 035/300 Read block 2377 len 36864
17614 036/300 Read block 2021 len 40960
17615 037/300 Write block 3493 len 20480 data: 4 5 6 6 6
17616 038/300 Read block 113 len 4096
17617 039/300 Read block 3159 len 8192
17618 040/300 Read block 969 len 24576
17619 041/300 Read block 3460 len 16384
17620 042/300 Read block 2123 len 4096
17621 043/300 Read block 312 len 4096
17622 044/300 Read block 1350 len 24576
17623 045/300 Read block 2829 len 4096
17624 046/300 Read block 1804 len 32768
17625 047/300 Write block 1243 len 40960 data: 3 4 5 5 5 6 5 6 6 5
17626 048/300 Flush
17627 049/300 Read block 2329 len 36864
17628 050/300 Write block 2440 len 12288 data: 3 3 2
17629 051/300 Read block 319 len 32768
17630 052/300 Read block 3026 len 20480
17631 053/300 Flush
17632 054/300 Read block 1789 len 8192
17633 055/300 Write block 1321 len 32768 data: 6 6 6 6 5 6 6 6
17634 056/300 Write block 2159 len 12288 data: 6 5 5
17635 057/300 Write block 4830 len 24576 data: 3 2 3 3 5 5
17636 058/300 Write block 4366 len 20480 data: 5 4 3 3 3
17637 059/300 Read block 1247 len 36864
17638 060/300 Read block 4317 len 36864
17639 061/300 Read block 2146 len 32768
17640 062/300 Read block 4818 len 28672
17641 063/300 Write block 4237 len 36864 data: 6 7 6 4 4 3 4 4 2
17642 064/300 Write block 4032 len 4096 data: 4
17643 065/300 Read block 3439 len 4096
17644 066/300 Read block 1886 len 40960
17645 067/300 Write block 2687 len 4096 data: 2
17646 068/300 Write block 3929 len 8192 data: 3 3
17647 069/300 Write block 129 len 4096 data: 2
17648 070/300 Read block 4577 len 24576
17649 071/300 Flush
17650 072/300 Write block 821 len 20480 data: 6 5 5 4 4
17651 073/300 Read block 3495 len 8192
17652 074/300 Write block 466 len 32768 data: 3 3 5 6 6 4 4 4
17653 075/300 Flush
17654 076/300 Read block 3871 len 32768
17655 077/300 Write block 3977 len 40960 data: 2 2 2 2 3 3 4 4 4 4
17656 078/300 Write block 1006 len 36864 data: 5 5 5 5 5 6 7 5 5
17657 079/300 Write block 2356 len 36864 data: 5 6 6 5 4 5 4 4 4
17658 080/300 Flush
17659 081/300 Read block 3505 len 20480
17660 082/300 Write block 824 len 32768 data: 5 5 5 4 4 4 3 3
17661 083/300 Read block 2069 len 32768
17662 084/300 Write block 576 len 28672 data: 4 4 5 6 6 6 6
17663 085/300 Write block 1948 len 28672 data: 2 3 3 5 4 5 4
17664 086/300 Write block 3668 len 8192 data: 9 7
17665 087/300 Read block 1703 len 40960
17666 088/300 Write block 1573 len 20480 data: 5 5 4 4 5
17667 089/300 Write block 2747 len 8192 data: 5 5
17668 090/300 Write block 3244 len 24576 data: 8 7 7 5 6 5
17669 091/300 Read block 1911 len 16384
17670 092/300 Read block 4128 len 32768
17671 093/300 Flush
17672 094/300 Write block 3540 len 36864 data: 4 4 3 4 3 3 3 3 2
17673 095/300 Flush
17674 096/300 Write block 3200 len 20480 data: 4 4 4 4 5
17675 097/300 Read block 2073 len 36864
17676 098/300 Read block 2982 len 8192
17677 099/300 Write block 1176 len 16384 data: 3 4 4 5
17678 100/300 Write block 3430 len 12288 data: 4 4 4
17679 101/300 Read block 295 len 36864
17680 102/300 Read block 1055 len 16384
17681 103/300 Flush
17682 104/300 Read block 3789 len 16384
17683 105/300 Write block 4825 len 28672 data: 6 6 5 3 3 4 3
17684 106/300 Write block 4825 len 16384 data: 7 7 6 4
17685 107/300 Write block 3902 len 36864 data: 4 4 5 5 6 7 7 7 6
17686 108/300 Write block 206 len 32768 data: 4 4 3 3 3 4 4 3
17687 109/300 Read block 3693 len 20480
17688 110/300 Write block 364 len 40960 data: 4 3 3 5 6 7 6 7 6 5
17689 111/300 Write block 4863 len 28672 data: 7 7 7 5 5 5 3
17690 112/300 Write block 4613 len 16384 data: 2 2 2 3
17691 113/300 Read block 2987 len 20480
17692 114/300 Write block 1980 len 4096 data: 3
17693 115/300 Read block 3352 len 12288
17694 116/300 Read block 2266 len 36864
17695 117/300 Read block 4016 len 24576
17696 118/300 Read block 1961 len 40960
17697 119/300 Read block 564 len 32768
17698 120/300 Write block 3750 len 20480 data: 3 4 4 4 2
17699 121/300 Read block 1872 len 24576
17700 122/300 Write block 1815 len 40960 data: 5 4 5 4 3 4 4 4 6 6
17701 123/300 Read block 1206 len 24576
17702 124/300 Write block 2778 len 28672 data: 4 4 4 4 4 4 5
17703 125/300 Read block 379 len 12288
17704 126/300 Write block 4848 len 24576 data: 4 3 2 2 2 2
17705 127/300 Write block 1062 len 32768 data: 3 3 4 2 2 2 2 2
17706 128/300 Read block 441 len 40960
17707 129/300 Write block 2860 len 40960 data: 5 6 5 6 7 6 7 7 6 4
17708 130/300 Flush
17709 131/300 Read block 2637 len 36864
17710 132/300 Write block 3794 len 24576 data: 2 3 3 3 3 4
17711 133/300 Write block 3662 len 36864 data: 7 8 6 5 6 9 10 8 7
17712 134/300 Write block 2747 len 32768 data: 6 6 5 3 3 4 3 3
17713 135/300 Read block 3652 len 40960
17714 136/300 Write block 4948 len 24576 data: 7 7 7 7 7 7
17715 137/300 Write block 4295 len 8192 data: 3 3
17716 138/300 Read block 2476 len 4096
17717 139/300 Read block 3981 len 4096
17718 140/300 Read block 3184 len 40960
17719 141/300 Read block 558 len 20480
17720 142/300 Read block 1325 len 28672
17721 143/300 Write block 1006 len 12288 data: 6 6 6
17722 144/300 Read block 2886 len 40960
17723 145/300 Read block 2200 len 24576
17724 146/300 Read block 4402 len 40960
17725 147/300 Read block 4703 len 28672
17726 148/300 Read block 3079 len 24576
17727 149/300 Read block 4966 len 40960
17728 150/300 Read block 1631 len 12288
17729 151/300 Read block 3384 len 32768
17730 152/300 Read block 3459 len 28672
17731 153/300 Read block 3132 len 24576
17732 154/300 Read block 4965 len 4096
17733 155/300 Read block 4212 len 4096
17734 156/300 Read block 4967 len 12288
17735 157/300 Read block 3352 len 36864
17736 158/300 Write block 4769 len 24576 data: 7 6 5 5 6 5
17737 159/300 Read block 1041 len 12288
17738 160/300 Read block 415 len 16384
17739 161/300 Write block 3626 len 4096 data: 8
17740 162/300 Flush
17741 163/300 Write block 3133 len 40960 data: 3 5 6 5 6 5 5 5 5 6
17742 164/300 Read block 4692 len 24576
17743 165/300 Read block 2199 len 20480
17744 166/300 Read block 4404 len 32768
17745 167/300 Read block 3541 len 24576
17746 168/300 Read block 4854 len 24576
17747 169/300 Read block 2719 len 32768
17748 170/300 Flush
17749 171/300 Read block 4227 len 40960
17750 172/300 Read block 2637 len 16384
17751 173/300 Flush
17752 174/300 Read block 2274 len 8192
17753 175/300 Write block 2874 len 24576 data: 4 4 3 4 4 5
17754 176/300 Read block 1942 len 4096
17755 177/300 Write block 457 len 20480 data: 6 6 5 5 5
17756 178/300 Write block 3897 len 8192 data: 3 3
17757 179/300 Write block 1587 len 36864 data: 7 6 5 6 5 6 6 5 5
17758 180/300 Write block 106 len 8192 data: 4 3
17759 181/300 Read block 1161 len 20480
17760 182/300 Read block 4690 len 36864
17761 183/300 Flush
17762 184/300 Read block 4989 len 4096
17763 185/300 Write block 558 len 40960 data: 6 6 5 5 2 3 4 5 6 5
17764 186/300 Read block 1481 len 36864
17765 187/300 Write block 4930 len 28672 data: 5 7 8 7 6 3 2
17766 188/300 Flush
17767 189/300 Write block 610 len 32768 data: 3 3 4 4 4 6 7 9
17768 190/300 Read block 764 len 40960
17769 191/300 Read block 3748 len 32768
17770 192/300 Write block 1323 len 8192 data: 7 7
17771 193/300 Write block 2584 len 4096 data: 4
17772 194/300 Read block 3817 len 16384
17773 195/300 Read block 881 len 36864
17774 196/300 Write block 440 len 12288 data: 5 4 5
17775 197/300 Write block 4877 len 16384 data: 4 3 3 3
17776 198/300 Read block 1492 len 40960
17777 199/300 Read block 3267 len 20480
17778 200/300 Write block 3332 len 16384 data: 4 4 5 5
17779 201/300 Write block 4257 len 24576 data: 6 5 7 6 7 7
17780 202/300 Write block 2925 len 36864 data: 4 4 3 3 4 4 4 3 3
17781 203/300 Write block 3789 len 32768 data: 6 5 4 3 2 3 4 4
17782 204/300 Read block 4752 len 28672
17783 205/300 Flush
17784 206/300 Write block 379 len 16384 data: 6 7 5 5
17785 207/300 Flush
17786 208/300 Read block 2733 len 16384
17787 209/300 Read block 4319 len 28672
17788 210/300 Flush
17789 211/300 Read block 4021 len 8192
17790 212/300 Read block 2955 len 20480
17791 213/300 Flush
17792 214/300 Read block 1865 len 40960
17793 215/300 Read block 4094 len 32768
17794 216/300 Read block 3221 len 32768
17795 217/300 Write block 4304 len 32768 data: 2 3 4 6 6 6 6 6
17796 218/300 Read block 2165 len 28672
17797 219/300 Read block 3538 len 16384
17798 220/300 Write block 3188 len 32768 data: 6 5 6 5 4 3 3 3
17799 221/300 Read block 686 len 24576
17800 222/300 Read block 4936 len 8192
17801 223/300 Write block 2080 len 16384 data: 4 6 7 8
17802 224/300 Write block 74 len 20480 data: 6 6 5 5 5
17803 225/300 Read block 3434 len 12288
17804 226/300 Read block 1575 len 8192
17805 227/300 Write block 3542 len 24576 data: 4 5 4 4 4 4
17806 228/300 Write block 2773 len 28672 data: 3 3 2 2 2 5 5
17807 229/300 Flush
17808 230/300 Write block 1389 len 32768 data: 3 3 3 3 3 3 3 4
17809 231/300 Read block 203 len 24576
17810 232/300 Write block 1364 len 8192 data: 3 4
17811 233/300 Write block 601 len 8192 data: 7 6
17812 234/300 Read block 4003 len 28672
17813 235/300 Write block 3854 len 16384 data: 4 4 5 6
17814 236/300 Flush
17815 237/300 Read block 3358 len 12288
17816 238/300 Flush
17817 239/300 Flush
17818 240/300 Write block 3871 len 12288 data: 6 6 7
17819 241/300 Write block 3480 len 24576 data: 3 4 4 5 5 5
17820 242/300 Write block 612 len 24576 data: 5 5 5 7 8 10
17821 243/300 Read block 1840 len 36864
17822 244/300 Write block 1090 len 4096 data: 4
17823 245/300 Write block 3233 len 24576 data: 3 2 3 4 4 4
17824 246/300 Read block 1242 len 36864
17825 247/300 Write block 4631 len 36864 data: 5 4 4 4 4 5 5 5 5
17826 248/300 Write block 820 len 32768 data: 6 7 6 6 6 6 6 5
17827 249/300 Flush
17828 250/300 Write block 2436 len 16384 data: 4 4 3 3
17829 251/300 Read block 2533 len 40960
17830 252/300 Write block 3289 len 24576 data: 5 5 4 3 2 2
17831 253/300 Read block 2860 len 36864
17832 254/300 Write block 332 len 32768 data: 2 2 3 3 3 3 3 3
17833 255/300 Read block 2639 len 36864
17834 256/300 Read block 2192 len 32768
17835 257/300 Read block 4454 len 8192
17836 258/300 Read block 543 len 8192
17837 259/300 Write block 4932 len 4096 data: 9
17838 260/300 Read block 4282 len 16384
17839 261/300 Read block 3484 len 8192
17840 262/300 Read block 583 len 28672
17841 263/300 Write block 2543 len 28672 data: 6 6 6 7 5 5 5
17842 264/300 Flush
17843 265/300 Read block 3205 len 32768
17844 266/300 Read block 4079 len 40960
17845 267/300 Read block 3300 len 4096
17846 268/300 Read block 1473 len 12288
17847 269/300 Read block 4623 len 16384
17848 270/300 Flush
17849 271/300 Write block 1703 len 4096 data: 4
17850 272/300 Write block 3683 len 28672 data: 3 3 3 3 4 4 4
17851 273/300 Write block 1916 len 4096 data: 5
17852 274/300 Read block 513 len 36864
17853 275/300 Flush
17854 276/300 Read block 1007 len 36864
17855 277/300 Write block 4072 len 8192 data: 5 5
17856 278/300 Write block 450 len 32768 data: 5 5 7 5 6 6 6 7
17857 279/300 Write block 4165 len 40960 data: 3 4 5 4 4 5 4 3 3 3
17858 280/300 Read block 3774 len 28672
17859 281/300 Read block 3098 len 12288
17860 282/300 Read block 4958 len 4096
17861 283/300 Read block 3737 len 16384
17862 284/300 Flush
17863 285/300 Write block 1623 len 16384 data: 5 5 6 6
17864 286/300 Write block 2209 len 16384 data: 5 4 4 4
17865 287/300 Read block 3712 len 20480
17866 288/300 Write block 3440 len 24576 data: 4 3 3 3 3 4
17867 289/300 Write block 1726 len 20480 data: 4 3 3 5 6
17868 290/300 Write block 4939 len 12288 data: 2 3 4
17869 291/300 Write block 1463 len 36864 data: 5 3 3 3 3 3 3 4 4
17870 292/300 Write block 4779 len 32768 data: 8 8 7 6 5 5 5 3
17871 293/300 Flush
17872 294/300 Read block 4177 len 32768
17873 295/300 Read block 3360 len 24576
17874 296/300 Write block 1025 len 20480 data: 3 4 5 5 5
17875 297/300 Read block 2040 len 32768
17876 298/300 Read block 3587 len 8192
17877 299/300 Read block 603 len 36864
17878 300/300 Read block 1413 len 12288
17879 Main task has work to do, go find it
17880 got dsc Start(1)
17881 start 1
17882 Feb 15 03:01:04.766 INFO request completed, latency_us: 211, response_code: 204, uri: /start/cid/1, method: POST, req_id: f60d0632-7759-4089-a929-5f2858099336, remote_addr: 127.0.0.1:54744, local_addr: 127.0.0.1:9998
17883 Replay: started 1, returned:Ok(())
17884 ----------------------------------------------------------------
17885 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
17886 States: Active Offline Active
17887 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
17888 5125 Acked 6124 Read 3 Done New Done true
17889 5126 Acked 6125 Write 9 Done New Done true
17890 5127 Acked 6126 Write 2 Done New Done true
17891 5128 Acked 6127 Read 7 Done New Done true
17892 5129 Acked 6128 Write 4 Done New Done false
17893 5130 Acked 6129 Read 8 Done New Done false
17894 5131 Acked 6130 Read 8 Done New Done false
17895 5132 Acked 6131 Read 7 Done New Done false
17896 5133 Acked 6132 Read 7 Done New Done false
17897 5134 Acked 6133 Read 9 Done New Done false
17898 5135 Acked 6134 Read 3 Done New Done false
17899 5136 Acked 6135 Write 2 Done New Done false
17900 5137 Acked 6136 Flush 0 Done New Done false
17901 5138 Acked 6137 Read 9 Done New Done false
17902 5139 Acked 6138 Read 2 Done New Done false
17903 5140 Acked 6139 Read 5 Done New Done false
17904 5141 Acked 6140 Write 8 Done New Done false
17905 5142 Acked 6141 Flush 0 Done New Done false
17906 5143 Acked 6142 Write 7 Done New Done false
17907 5144 Acked 6143 Write 4 Done New Done false
17908 5145 Acked 6144 Read 6 Done New Done false
17909 5146 Acked 6145 Write 6 Done New Done false
17910 5147 Acked 6146 Write 4 Done New Done false
17911 5148 Acked 6147 Read 2 Done New Done false
17912 5149 Acked 6148 Write 2 Done New Done false
17913 5150 Acked 6149 Write 5 Done New Done false
17914 5151 Acked 6150 Write 5 Done New Done false
17915 5152 Acked 6151 Read 6 Done New Done false
17916 5153 Acked 6152 Read 10 Done New Done false
17917 5154 Acked 6153 Read 1 Done New Done false
17918 5155 Acked 6154 Read 6 Done New Done false
17919 5156 Acked 6155 Read 3 Done New Done false
17920 5157 Acked 6156 Write 7 Done New Done false
17921 5158 Acked 6157 Write 2 Done New Done false
17922 5159 Acked 6158 Read 9 Done New Done false
17923 5160 Acked 6159 Read 10 Done New Done false
17924 5161 Acked 6160 Write 5 Done New Done false
17925 5162 Acked 6161 Read 1 Done New Done false
17926 5163 Acked 6162 Read 2 Done New Done false
17927 5164 Acked 6163 Read 6 Done New Done false
17928 5165 Acked 6164 Read 4 Done New Done false
17929 5166 Acked 6165 Read 1 Done New Done false
17930 5167 Acked 6166 Read 1 Done New Done false
17931 5168 Acked 6167 Read 6 Done New Done false
17932 5169 Acked 6168 Read 1 Done New Done false
17933 5170 Acked 6169 Read 8 Done New Done false
17934 5171 Acked 6170 Write 10 Done New Done false
17935 5172 Acked 6171 Flush 0 Done New Done false
17936 5173 Acked 6172 Read 9 Done New Done false
17937 5174 Acked 6173 Flush 0 Done New Done false
17938 5175 Acked 6174 Write 3 Done New Done false
17939 5176 Acked 6175 Read 8 Done New Done false
17940 5177 Acked 6176 Read 5 Done New Done false
17941 5178 Acked 6177 Flush 0 Done New Done false
17942 5179 Acked 6178 Read 2 Done New Done false
17943 5180 Acked 6179 Write 8 Done New Done false
17944 5181 Acked 6180 Write 3 Done New Done false
17945 5182 Acked 6181 Write 6 Done New Done false
17946 5183 Acked 6182 Write 5 Done New Done false
17947 5184 Acked 6183 Read 9 Done New Done false
17948 5185 Acked 6184 Read 9 Done New Done false
17949 5186 Acked 6185 Read 8 Done New Done false
17950 5187 Acked 6186 Read 7 Done New Done false
17951 5188 Acked 6187 Write 9 Done New Done false
17952 5189 Acked 6188 Write 1 Done New Done false
17953 5190 Acked 6189 Read 1 Done New Done false
17954 5191 Acked 6190 Read 10 Done New Done false
17955 5192 Acked 6191 Write 1 Done New Done false
17956 5193 Acked 6192 Write 2 Done New Done false
17957 5194 Acked 6193 Write 1 Done New Done false
17958 5195 Acked 6194 Read 6 Done New Done false
17959 5196 Acked 6195 Flush 0 Done New Done false
17960 5197 Acked 6196 Write 5 Done New Done false
17961 5198 Acked 6197 Read 2 Done New Done false
17962 5199 Acked 6198 Write 8 Done New Done false
17963 5200 Acked 6199 Flush 0 Done New Done false
17964 5201 Acked 6200 Read 8 Done New Done false
17965 5202 Acked 6201 Write 10 Done New Done false
17966 5203 Acked 6202 Write 9 Done New Done false
17967 5204 Acked 6203 Write 9 Done New Done false
17968 5205 Acked 6204 Flush 0 Done New Done false
17969 5206 Acked 6205 Read 5 Done New Done false
17970 5207 Acked 6206 Write 8 Done New Done false
17971 5208 Acked 6207 Read 8 Done New Done false
17972 5209 Acked 6208 Write 7 Done New Done false
17973 5210 Acked 6209 Write 7 Done New Done false
17974 5211 Acked 6210 Write 2 Done New Done false
17975 5212 Acked 6211 Read 10 Done New Done false
17976 5213 Acked 6212 Write 5 Done New Done false
17977 5214 Acked 6213 Write 2 Done New Done false
17978 5215 Acked 6214 Write 6 Done New Done false
17979 5216 Acked 6215 Read 4 Done New Done false
17980 5217 Acked 6216 Read 8 Done New Done false
17981 5218 Acked 6217 Flush 0 Done New Done false
17982 5219 Acked 6218 Write 9 Done New Done false
17983 5220 Acked 6219 Flush 0 Done New Done false
17984 5221 Acked 6220 Write 5 Done New Done false
17985 5222 Acked 6221 Read 9 Done New Done false
17986 5223 Acked 6222 Read 2 Done New Done false
17987 5224 Acked 6223 Write 4 Done New Done false
17988 5225 Acked 6224 Write 3 Done New Done false
17989 5226 Acked 6225 Read 9 Done New Done false
17990 5227 Acked 6226 Read 4 Done New Done false
17991 5228 Acked 6227 Flush 0 Done New Done false
17992 5229 Acked 6228 Read 4 Done New Done false
17993 5230 Acked 6229 Write 7 Done New Done false
17994 5231 Acked 6230 Write 4 Done New Done false
17995 5232 Acked 6231 Write 9 Done New Done false
17996 5233 Acked 6232 Write 8 Done New Done false
17997 5234 Acked 6233 Read 5 Done New Done false
17998 5235 Acked 6234 Write 10 Done New Done false
17999 5236 Acked 6235 Write 7 Done New Done false
18000 5237 Acked 6236 Write 4 Done New Done false
18001 5238 Acked 6237 Read 5 Done New Done false
18002 5239 Acked 6238 Write 1 Done New Done false
18003 5240 Acked 6239 Read 3 Done New Done false
18004 5241 Acked 6240 Read 9 Done New Done false
18005 5242 Acked 6241 Read 6 Done New Done false
18006 5243 Acked 6242 Read 10 Done New Done false
18007 5244 Acked 6243 Read 8 Done New Done false
18008 5245 Acked 6244 Write 5 Done New Done false
18009 5246 Acked 6245 Read 6 Done New Done false
18010 5247 Acked 6246 Write 10 Done New Done false
18011 5248 Acked 6247 Read 6 Done New Done false
18012 5249 Acked 6248 Write 7 Done New Done false
18013 5250 Acked 6249 Read 3 Done New Done false
18014 5251 Acked 6250 Write 6 Done New Done false
18015 5252 Acked 6251 Write 8 Done New Done false
18016 5253 Acked 6252 Read 10 Done New Done false
18017 5254 Acked 6253 Write 10 Done New Done false
18018 5255 Acked 6254 Flush 0 Done New Done false
18019 5256 Acked 6255 Read 9 Done New Done false
18020 5257 Acked 6256 Write 6 Done New Done false
18021 5258 Acked 6257 Write 9 Done New Done false
18022 5259 Acked 6258 Write 8 Done New Done false
18023 5260 Acked 6259 Read 10 Done New Done false
18024 5261 Acked 6260 Write 6 Done New Done false
18025 5262 Acked 6261 Write 2 Done New Done false
18026 5263 Acked 6262 Read 1 Done New Done false
18027 5264 Acked 6263 Read 1 Done New Done false
18028 5265 Acked 6264 Read 10 Done New Done false
18029 5266 Acked 6265 Read 5 Done New Done false
18030 5267 Acked 6266 Read 7 Done New Done false
18031 5268 Acked 6267 Write 3 Done New Done false
18032 5269 Acked 6268 Read 10 Done New Done false
18033 5270 Acked 6269 Read 6 Done New Done false
18034 5271 Acked 6270 Read 10 Done New Done false
18035 5272 Acked 6271 Read 7 Done New Done false
18036 5273 Acked 6272 Read 6 Done New Done false
18037 5274 Acked 6273 Read 10 Done New Done false
18038 5275 Acked 6274 Read 3 Done New Done false
18039 5276 Acked 6275 Read 8 Done New Done false
18040 5277 Acked 6276 Read 7 Done New Done false
18041 5278 Acked 6277 Read 6 Done New Done false
18042 5279 Acked 6278 Read 1 Done New Done false
18043 5280 Acked 6279 Read 1 Done New Done false
18044 5281 Acked 6280 Read 3 Done New Done false
18045 5282 Acked 6281 Read 9 Done New Done false
18046 5283 Acked 6282 Write 6 Done New Done false
18047 5284 Acked 6283 Read 3 Done New Done false
18048 5285 Acked 6284 Read 4 Done New Done false
18049 5286 Acked 6285 Write 1 Done New Done false
18050 5287 Acked 6286 Flush 0 Done New Done false
18051 5288 Acked 6287 Write 10 Done New Done false
18052 5289 Acked 6288 Read 6 Done New Done false
18053 5290 Acked 6289 Read 5 Done New Done false
18054 5291 Acked 6290 Read 8 Done New Done false
18055 5292 Acked 6291 Read 6 Done New Done false
18056 5293 Acked 6292 Read 6 Done New Done false
18057 5294 Acked 6293 Read 8 Done New Done false
18058 5295 Acked 6294 Flush 0 Done New Done false
18059 5296 Acked 6295 Read 10 Done New Done false
18060 5297 Acked 6296 Read 4 Done New Done false
18061 5298 Acked 6297 Flush 0 Done New Done false
18062 5299 Acked 6298 Read 2 Done New Done false
18063 5300 Acked 6299 Write 6 Done New Done false
18064 5301 Acked 6300 Read 1 Done New Done false
18065 5302 Acked 6301 Write 5 Done New Done false
18066 5303 Acked 6302 Write 2 Done New Done false
18067 5304 Acked 6303 Write 9 Done New Done false
18068 5305 Acked 6304 Write 2 Done New Done false
18069 5306 Acked 6305 Read 5 Done New Done false
18070 5307 Acked 6306 Read 9 Done New Done false
18071 5308 Acked 6307 Flush 0 Done New Done false
18072 5309 Acked 6308 Read 1 Done New Done false
18073 5310 Acked 6309 Write 10 Done New Done false
18074 5311 Acked 6310 Read 9 Done New Done false
18075 5312 Acked 6311 Write 7 Done New Done false
18076 5313 Acked 6312 Flush 0 Done New Done false
18077 5314 Acked 6313 Write 8 Done New Done false
18078 5315 Acked 6314 Read 10 Done New Done false
18079 5316 Acked 6315 Read 8 Done New Done false
18080 5317 Acked 6316 Write 2 Done New Done false
18081 5318 Acked 6317 Write 1 Done New Done false
18082 5319 Acked 6318 Read 4 Done New Done false
18083 5320 Acked 6319 Read 9 Done New Done false
18084 5321 Acked 6320 Write 3 Done New Done false
18085 5322 Acked 6321 Write 4 Done New Done false
18086 5323 Acked 6322 Read 10 Done New Done false
18087 5324 Acked 6323 Read 5 Done New Done false
18088 5325 Acked 6324 Write 4 Done New Done false
18089 5326 Acked 6325 Write 6 Done New Done false
18090 5327 Acked 6326 Write 9 Done New Done false
18091 5328 Acked 6327 Write 8 Done New Done false
18092 5329 Acked 6328 Read 7 Done New Done false
18093 5330 Acked 6329 Flush 0 Done New Done false
18094 5331 Acked 6330 Write 4 Done New Done false
18095 5332 Acked 6331 Flush 0 Done New Done false
18096 5333 Acked 6332 Read 4 Done New Done false
18097 5334 Acked 6333 Read 7 Done New Done false
18098 5335 Acked 6334 Flush 0 Done New Done false
18099 5336 Acked 6335 Read 2 Done New Done false
18100 5337 Acked 6336 Read 5 Done New Done false
18101 5338 Acked 6337 Flush 0 Done New Done false
18102 5339 Acked 6338 Read 10 Done New Done false
18103 5340 Acked 6339 Read 8 Done New Done false
18104 5341 Acked 6340 Read 8 Done New Done false
18105 5342 Acked 6341 Write 8 Done New Done false
18106 5343 Acked 6342 Read 7 Done New Done false
18107 5344 Acked 6343 Read 4 Done New Done false
18108 5345 Acked 6344 Write 8 Done New Done false
18109 5346 Acked 6345 Read 6 Done New Done false
18110 5347 Acked 6346 Read 2 Done New Done false
18111 5348 Acked 6347 Write 4 Done New Done false
18112 5349 Acked 6348 Write 5 Done New Done false
18113 5350 Acked 6349 Read 3 Done New Done false
18114 5351 Acked 6350 Read 2 Done New Done false
18115 5352 Acked 6351 Write 6 Done New Done false
18116 5353 Acked 6352 Write 7 Done New Done false
18117 5354 Acked 6353 Flush 0 Done New Done false
18118 5355 Acked 6354 Write 8 Done New Done false
18119 5356 Acked 6355 Read 6 Done New Done false
18120 5357 Acked 6356 Write 2 Done New Done false
18121 5358 Acked 6357 Write 2 Done New Done false
18122 5359 Acked 6358 Read 7 Done New Done false
18123 5360 Acked 6359 Write 4 Done New Done false
18124 5361 Acked 6360 Flush 0 Done New Done false
18125 5362 Acked 6361 Read 3 Done New Done false
18126 5363 Acked 6362 Flush 0 Done New Done false
18127 5364 Acked 6363 Flush 0 Done New Done false
18128 5365 Acked 6364 Write 3 Done New Done false
18129 5366 Acked 6365 Write 6 Done New Done false
18130 5367 Acked 6366 Write 6 Done New Done false
18131 5368 Acked 6367 Read 9 Done New Done false
18132 5369 Acked 6368 Write 1 Done New Done false
18133 5370 Acked 6369 Write 6 Done New Done false
18134 5371 Acked 6370 Read 9 Done New Done false
18135 5372 Acked 6371 Write 9 Done New Done false
18136 5373 Acked 6372 Write 8 Done New Done false
18137 5374 Acked 6373 Flush 0 Done New Done false
18138 5375 Acked 6374 Write 4 Done New Done false
18139 5376 Acked 6375 Read 10 Done New Done false
18140 5377 Acked 6376 Write 6 Done New Done false
18141 5378 Acked 6377 Read 9 Done New Done false
18142 5379 Acked 6378 Write 8 Done New Done false
18143 5380 Acked 6379 Read 9 Done New Done false
18144 5381 Acked 6380 Read 8 Done New Done false
18145 5382 Acked 6381 Read 2 Done New Done false
18146 5383 Acked 6382 Read 2 Done New Done false
18147 5384 Acked 6383 Write 1 Done New Done false
18148 5385 Acked 6384 Read 4 Done New Done false
18149 5386 Acked 6385 Read 2 Done New Done false
18150 5387 Acked 6386 Read 7 Done New Done false
18151 5388 Acked 6387 Write 7 Done New Done false
18152 5389 Acked 6388 Flush 0 Done New Done false
18153 5390 Acked 6389 Read 8 Done New Done false
18154 5391 Acked 6390 Read 10 Done New Done false
18155 5392 Acked 6391 Read 1 Done New Done false
18156 5393 Acked 6392 Read 3 Done New Done false
18157 5394 Acked 6393 Read 4 Done New Done false
18158 5395 Acked 6394 Flush 0 Done New Done false
18159 5396 Acked 6395 Write 1 Done New Done false
18160 5397 Acked 6396 Write 7 Done New Done false
18161 5398 Acked 6397 Write 1 Done New Done false
18162 5399 Acked 6398 Read 9 Done New Done false
18163 5400 Acked 6399 Flush 0 Done New Done false
18164 5401 Acked 6400 Read 9 Done New Done false
18165 5402 Acked 6401 Write 2 Done New Done false
18166 5403 Acked 6402 Write 8 Done New Done false
18167 5404 Acked 6403 Write 10 Done New Done false
18168 5405 Acked 6404 Read 7 Done New Done false
18169 5406 Acked 6405 Read 3 Done New Done false
18170 5407 Acked 6406 Read 1 Done New Done false
18171 5408 Acked 6407 Read 4 Done New Done false
18172 5409 Acked 6408 Flush 0 Done New Done false
18173 5410 Acked 6409 Write 4 Done New Done false
18174 5411 Acked 6410 Write 4 Done New Done false
18175 5412 Acked 6411 Read 5 Done New Done false
18176 5413 Acked 6412 Write 6 Done New Done false
18177 5414 Acked 6413 Write 5 Done New Done false
18178 5415 Acked 6414 Write 3 Done New Done false
18179 5416 Acked 6415 Write 9 Done New Done false
18180 5417 Acked 6416 Write 8 Done New Done false
18181 5418 Acked 6417 Flush 0 Done New Done false
18182 5419 Acked 6418 Read 8 Done New Done false
18183 5420 Acked 6419 Read 6 Done New Done false
18184 5421 Acked 6420 Write 5 Done New Done false
18185 5422 Acked 6421 Read 8 Done New Done false
18186 5423 Acked 6422 Read 2 Done New Done false
18187 5424 Acked 6423 Read 9 Done New Done false
18188 5425 Acked 6424 Read 3 Done New Done false
18189 STATES DS:0 DS:1 DS:2 TOTAL
18190 New 0 301 0 301
18191 Sent 0 0 0 0
18192 Done 301 0 301 602
18193 Skipped 0 0 0 0
18194 Error 0 0 0 0
18195 Last Flush: 6417 6123 6417
18196 Downstairs last five completed: 6123 6122 6121 6120 6119
18197 Upstairs last five completed: 5425 5424 5423 5422 5421
18198 CLIENT: Up:0 ds:301 act:2
181992024-02-15T03:01:05.431ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
182002024-02-15T03:01:05.431ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
182012024-02-15T03:01:05.431ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
182022024-02-15T03:01:05.431ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
182032024-02-15T03:01:05.431ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
182042024-02-15T03:01:05.432ZINFOcrucible: [1] client re-new 302 jobs since flush 6123 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
182052024-02-15T03:01:05.432ZINFOcrucible: client task is exiting = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
18206 [8820] got start action
18207 [8820] I am going to restart
18208 Starting downstairs at port 8820
18209 Make output file at "/tmp/dsc/downstairs-8820.txt"
18210 [8820][1] reports Starting
182112024-02-15T03:01:06.432ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
182122024-02-15T03:01:06.432ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
182132024-02-15T03:01:06.432ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
182142024-02-15T03:01:06.432ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
182152024-02-15T03:01:06.432ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
182162024-02-15T03:01:06.433ZINFOcrucible: [1] client re-new 302 jobs since flush 6123 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
182172024-02-15T03:01:06.433ZINFOcrucible: client task is exiting = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
18218 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1185)
18219 [8820][1] reports Running
182202024-02-15T03:01:07.434ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
182212024-02-15T03:01:07.434ZINFOcrucible: ds_connection connected = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
182222024-02-15T03:01:07.435ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
182232024-02-15T03:01:07.435ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
182242024-02-15T03:01:07.435ZINFOcrucible: send last flush ID to this DS: 6123 = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
182252024-02-15T03:01:07.435ZINFOcrucible: Replied this last flush ID: 6123 = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
182262024-02-15T03:01:07.435ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
182272024-02-15T03:01:07.435ZINFOcrucible: [1] Transition from Offline to Replay = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
182282024-02-15T03:01:07.435ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
182292024-02-15T03:01:07.435ZINFOcrucible: [1] Transition from Replay to Active = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
18230 ----------------------------------------------------------------
18231 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
18232 Downstairs last five completed: 6425 6424 6423 6422 6421
18233 Upstairs last five completed: 5426 5425 5424 5423 5422
18234 CLIENT: Up:0 ds:0 act:3
18235 Replay: All jobs finished, all DS active.
18236 Main task has work to do, go find it
18237 got dsc Stop(0)
18238 stop 0
18239 Feb 15 03:01:08.769 INFO request completed, latency_us: 328, response_code: 204, uri: /stop/cid/0, method: POST, req_id: 5fb1df7e-6709-4062-ad11-0f5be5eb52e5, remote_addr: 127.0.0.1:54744, local_addr: 127.0.0.1:9998[8810] Got stop action so:false kr:false
18240 
182412024-02-15T03:01:08.770ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
182422024-02-15T03:01:08.770ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
182432024-02-15T03:01:08.770ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
182442024-02-15T03:01:08.770ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
182452024-02-15T03:01:08.770ZINFOcrucible: [0] client re-new 0 jobs since flush 6425 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
182462024-02-15T03:01:08.770ZINFOcrucible: client task is exiting = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
18247 Feb 15 03:01:08.770 INFO request completed, latency_us: 132, response_code: 200, uri: /state/cid/0, method: GET, req_id: 63dc1e44-1ce9-474e-851f-9d56c2bb2afd, remote_addr: 127.0.0.1:54744, local_addr: 127.0.0.1:9998
18248 [8810] Exited with: ExitStatus(unix_wait_status(9))
18249 [8810][0] reports Exit
182502024-02-15T03:01:09.771ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
182512024-02-15T03:01:09.772ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
182522024-02-15T03:01:09.772ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
182532024-02-15T03:01:09.772ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
182542024-02-15T03:01:09.772ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
182552024-02-15T03:01:09.772ZINFOcrucible: [0] client re-new 0 jobs since flush 6425 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
182562024-02-15T03:01:09.772ZINFOcrucible: client task is exiting = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
182572024-02-15T03:01:10.774ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
182582024-02-15T03:01:10.774ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
182592024-02-15T03:01:10.775ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
182602024-02-15T03:01:10.775ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
182612024-02-15T03:01:10.775ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
182622024-02-15T03:01:10.775ZINFOcrucible: [0] client re-new 0 jobs since flush 6425 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
182632024-02-15T03:01:10.775ZINFOcrucible: client task is exiting = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
182642024-02-15T03:01:11.776ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
182652024-02-15T03:01:11.776ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
182662024-02-15T03:01:11.776ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
182672024-02-15T03:01:11.776ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
182682024-02-15T03:01:11.776ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
182692024-02-15T03:01:11.776ZINFOcrucible: [0] client re-new 0 jobs since flush 6425 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
182702024-02-15T03:01:11.776ZINFOcrucible: client task is exiting = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
18271 Feb 15 03:01:12.772 INFO request completed, latency_us: 199, response_code: 200, uri: /state/cid/0, method: GET, req_id: 0cc77602-fbaa-446a-b701-a76c371ba716, remote_addr: 127.0.0.1:54744, local_addr: 127.0.0.1:9998
18272 001/300 Flush
18273 002/300 Read block 3940 len 24576
18274 003/300 Read block 943 len 36864
18275 004/300 Flush
182762024-02-15T03:01:12.776ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
182772024-02-15T03:01:12.776ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
182782024-02-15T03:01:12.776ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
182792024-02-15T03:01:12.776ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
182802024-02-15T03:01:12.776ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
182812024-02-15T03:01:12.776ZINFOcrucible: [0] client re-new 4 jobs since flush 6425 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
182822024-02-15T03:01:12.776ZINFOcrucible: client task is exiting = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
18283 005/300 Read block 258 len 12288
18284 006/300 Write block 460 len 4096 data: 6
18285 007/300 Read block 2402 len 28672
18286 008/300 Write block 2732 len 32768 data: 4 4 4 4 4 4 4 5
18287 009/300 Read block 1893 len 36864
18288 010/300 Write block 4017 len 4096 data: 7
18289 011/300 Write block 4802 len 20480 data: 4 4 5 5 3
18290 012/300 Write block 3331 len 8192 data: 4 5
18291 013/300 Read block 1949 len 12288
18292 014/300 Read block 2688 len 40960
18293 015/300 Write block 1637 len 4096 data: 3
18294 016/300 Write block 2156 len 28672 data: 3 3 5 7 6 6 5
18295 017/300 Read block 2466 len 32768
18296 018/300 Write block 2792 len 20480 data: 3 4 4 3 3
18297 019/300 Write block 3929 len 20480 data: 4 4 3 2 2
18298 020/300 Flush
18299 021/300 Flush
18300 022/300 Write block 3196 len 32768 data: 4 4 3 3 5 5 5 5
18301 023/300 Write block 24 len 28672 data: 7 6 6 6 5 5 3
18302 024/300 Write block 2654 len 12288 data: 2 2 2
18303 025/300 Read block 2006 len 32768
18304 026/300 Read block 230 len 32768
18305 027/300 Write block 792 len 32768 data: 5 4 5 5 6 6 5 6
18306 028/300 Write block 3859 len 40960 data: 6 7 5 4 6 6 6 6 6 8
18307 029/300 Flush
18308 030/300 Write block 3904 len 8192 data: 6 6
18309 031/300 Write block 3940 len 28672 data: 5 5 4 4 3 4 4
18310 032/300 Write block 87 len 16384 data: 4 3 3 4
18311 033/300 Read block 1136 len 20480
18312 034/300 Read block 3245 len 36864
18313 035/300 Read block 1165 len 8192
18314 036/300 Write block 4546 len 28672 data: 4 5 6 7 7 6 6
18315 037/300 Write block 2572 len 8192 data: 3 3
18316 038/300 Write block 1327 len 12288 data: 7 7 6
18317 039/300 Write block 1732 len 40960 data: 6 6 7 6 6 7 7 5 5 5
18318 040/300 Read block 2560 len 36864
18319 041/300 Write block 4179 len 12288 data: 4 3 3
18320 042/300 Read block 1234 len 28672
18321 043/300 Read block 3478 len 16384
18322 044/300 Flush
18323 045/300 Write block 2411 len 40960 data: 4 3 3 3 3 3 3 4 5 4
18324 046/300 Flush
18325 047/300 Read block 3584 len 32768
18326 048/300 Flush
18327 049/300 Write block 2239 len 24576 data: 7 7 6 5 4 4
18328 050/300 Write block 4533 len 24576 data: 5 5 4 3 2 2
18329 051/300 Read block 3536 len 36864
18330 052/300 Flush
18331 053/300 Read block 1147 len 8192
18332 054/300 Write block 1009 len 24576 data: 6 6 7 8 6 6
18333 055/300 Write block 777 len 28672 data: 3 3 2 3 3 3 4
18334 056/300 Read block 1874 len 16384
18335 057/300 Read block 3897 len 20480
18336 058/300 Read block 2010 len 24576
18337 059/300 Flush
18338 060/300 Read block 4024 len 12288
18339 061/300 Write block 2015 len 20480 data: 4 5 6 6 7
18340 062/300 Read block 2697 len 24576
18341 063/300 Flush
18342 064/300 Write block 3456 len 20480 data: 4 2 2 2 3
18343 065/300 Write block 124 len 16384 data: 2 2 2 2
18344 066/300 Read block 3404 len 32768
18345 067/300 Write block 3724 len 40960 data: 4 4 3 3 5 6 8 7 6 7
18346 068/300 Write block 3947 len 12288 data: 4 5 5
18347 069/300 Write block 2669 len 4096 data: 4
18348 070/300 Write block 4308 len 32768 data: 7 7 7 7 6 5 5 5
18349 071/300 Write block 2730 len 40960 data: 4 5 5 5 5 5 5 5 5 6
18350 072/300 Read block 2180 len 4096
18351 073/300 Read block 4885 len 20480
18352 074/300 Write block 886 len 32768 data: 5 7 8 8 6 6 3 3
18353 075/300 Write block 3612 len 36864 data: 4 4 5 4 5 6 7 7 7
18354 076/300 Write block 1205 len 4096 data: 3
18355 077/300 Write block 1541 len 28672 data: 5 5 5 7 7 6 5
18356 078/300 Read block 3309 len 36864
18357 079/300 Write block 1772 len 16384 data: 3 4 3 2
18358 080/300 Read block 2846 len 24576
18359 081/300 Write block 2448 len 32768 data: 7 6 5 5 5 5 4 5
18360 082/300 Read block 3605 len 4096
18361 083/300 Read block 3414 len 16384
18362 084/300 Read block 3398 len 32768
18363 085/300 Read block 1773 len 36864
18364 086/300 Read block 3786 len 40960
18365 087/300 Flush
18366 088/300 Read block 2937 len 16384
18367 089/300 Read block 4593 len 20480
18368 090/300 Write block 3476 len 8192 data: 3 2
18369 091/300 Write block 3503 len 40960 data: 3 5 5 6 6 4 3 3 4 6
18370 092/300 Write block 4840 len 40960 data: 3 3 3 2 5 4 4 3 5 4
18371 093/300 Read block 4400 len 16384
18372 094/300 Read block 2720 len 28672
18373 095/300 Read block 871 len 32768
18374 096/300 Write block 4225 len 8192 data: 2 2
18375 097/300 Read block 4877 len 28672
18376 098/300 Write block 1737 len 32768 data: 8 8 6 6 6 4 4 4
18377 099/300 Write block 1121 len 24576 data: 6 7 6 6 5 5
18378 100/300 Read block 636 len 24576
18379 101/300 Read block 3429 len 4096
18380 102/300 Read block 2905 len 24576
18381 103/300 Write block 4346 len 28672 data: 7 7 7 5 5 5 3
18382 104/300 Read block 1907 len 8192
18383 105/300 Read block 1910 len 36864
18384 106/300 Write block 125 len 16384 data: 3 3 3 2
18385 107/300 Write block 4610 len 16384 data: 2 2 2 3
18386 108/300 Read block 291 len 32768
18387 109/300 Write block 902 len 8192 data: 6 6
18388 110/300 Write block 4982 len 28672 data: 6 6 6 6 6 6 6
18389 111/300 Write block 4601 len 40960 data: 4 5 5 4 4 4 3 4 3 3
18390 112/300 Write block 2187 len 32768 data: 6 6 6 6 6 6 6 4
18391 113/300 Flush
18392 114/300 Flush
18393 115/300 Read block 2606 len 4096
18394 116/300 Read block 4279 len 12288
18395 117/300 Write block 1060 len 28672 data: 4 4 4 4 5 3 3
18396 118/300 Write block 2716 len 12288 data: 4 4 4
18397 119/300 Read block 4722 len 4096
18398 120/300 Write block 3521 len 20480 data: 5 6 6 7 6
18399 121/300 Write block 2874 len 12288 data: 5 5 4
18400 122/300 Read block 3169 len 8192
18401 123/300 Write block 2386 len 16384 data: 2 2 3 3
18402 124/300 Write block 1840 len 12288 data: 4 4 4
18403 125/300 Write block 1330 len 4096 data: 5
18404 126/300 Write block 1839 len 16384 data: 4 5 5 5
18405 127/300 Write block 3099 len 12288 data: 4 4 4
18406 128/300 Write block 22 len 12288 data: 7 8 8
18407 129/300 Write block 1182 len 28672 data: 4 4 4 5 5 3 7
18408 130/300 Read block 2317 len 24576
18409 131/300 Write block 1876 len 36864 data: 6 6 6 3 2 2 4 5 6
18410 132/300 Write block 2081 len 40960 data: 7 8 9 8 7 5 5 4 2 3
18411 133/300 Read block 4705 len 12288
18412 134/300 Flush
18413 135/300 Read block 1524 len 36864
18414 136/300 Read block 3340 len 16384
18415 137/300 Read block 3107 len 16384
18416 138/300 Flush
18417 139/300 Read block 1549 len 16384
18418 140/300 Write block 424 len 20480 data: 3 2 2 3 4
18419 141/300 Write block 4589 len 40960 data: 3 3 4 5 5 6 6 6 4 3
18420 142/300 Read block 1522 len 16384
18421 143/300 Read block 3646 len 4096
18422 144/300 Flush
18423 145/300 Read block 949 len 24576
18424 146/300 Read block 2865 len 40960
18425 147/300 Read block 2591 len 28672
18426 148/300 Read block 1823 len 28672
18427 149/300 Read block 674 len 8192
18428 150/300 Read block 2907 len 16384
18429 151/300 Write block 3939 len 12288 data: 5 6 6
18430 152/300 Read block 410 len 24576
18431 153/300 Read block 776 len 32768
18432 154/300 Read block 2130 len 16384
18433 155/300 Read block 1914 len 16384
18434 156/300 Read block 4678 len 28672
18435 157/300 Write block 348 len 40960 data: 3 2 5 5 5 6 6 6 4 5
18436 158/300 Read block 2108 len 28672
18437 159/300 Read block 4576 len 24576
18438 160/300 Read block 3673 len 12288
18439 161/300 Read block 639 len 36864
18440 162/300 Write block 3372 len 16384 data: 4 4 4 4
18441 163/300 Write block 1517 len 28672 data: 4 3 3 4 4 3 4
18442 164/300 Read block 988 len 24576
18443 165/300 Read block 4544 len 8192
18444 166/300 Read block 3043 len 36864
18445 167/300 Read block 3067 len 36864
18446 168/300 Write block 1452 len 32768 data: 3 3 3 3 2 2 3 3
18447 169/300 Read block 4162 len 40960
18448 170/300 Write block 1415 len 16384 data: 4 4 4 5
18449 171/300 Write block 3076 len 32768 data: 3 3 3 4 4 5 5 6
18450 172/300 Write block 4280 len 20480 data: 4 4 4 5 5
18451 173/300 Read block 4738 len 40960
18452 174/300 Read block 1054 len 40960
18453 175/300 Read block 2969 len 24576
18454 176/300 Read block 611 len 8192
18455 177/300 Write block 2541 len 40960 data: 3 4 7 7 7 8 6 6 6 4
18456 178/300 Flush
18457 179/300 Write block 4271 len 28672 data: 3 3 4 2 2 2 3
18458 180/300 Read block 616 len 4096
18459 181/300 Write block 1189 len 36864 data: 7 6 7 7 6 6 6 5 6
18460 182/300 Read block 732 len 8192
18461 183/300 Read block 1685 len 8192
18462 184/300 Write block 4555 len 8192 data: 3 3
18463 185/300 Write block 11 len 4096 data: 3
18464 186/300 Write block 2567 len 4096 data: 3
18465 187/300 Read block 2737 len 8192
18466 188/300 Read block 2955 len 12288
18467 189/300 Read block 2788 len 32768
18468 190/300 Read block 159 len 12288
18469 191/300 Read block 2666 len 4096
18470 192/300 Write block 3212 len 36864 data: 5 7 6 6 5 5 4 3 3
18471 193/300 Read block 3299 len 4096
18472 194/300 Read block 1016 len 20480
18473 195/300 Read block 2434 len 32768
18474 196/300 Read block 4809 len 12288
18475 197/300 Flush
18476 198/300 Read block 1423 len 24576
18477 199/300 Write block 3947 len 12288 data: 5 6 6
18478 200/300 Read block 1590 len 24576
18479 201/300 Read block 3515 len 12288
18480 202/300 Read block 4725 len 28672
18481 203/300 Read block 2580 len 8192
18482 204/300 Read block 101 len 28672
18483 205/300 Write block 160 len 16384 data: 5 6 6 6
18484 206/300 Read block 945 len 8192
18485 207/300 Read block 3229 len 20480
18486 208/300 Read block 50 len 4096
18487 209/300 Write block 3686 len 20480 data: 4 5 5 5 6
18488 210/300 Write block 3151 len 28672 data: 6 6 6 7 8 9 9
18489 211/300 Flush
18490 212/300 Read block 113 len 40960
18491 213/300 Read block 2886 len 16384
18492 214/300 Read block 917 len 32768
18493 215/300 Read block 3934 len 20480
18494 216/300 Read block 4539 len 16384
18495 217/300 Write block 2229 len 28672 data: 4 5 5 5 5 5 5
18496 218/300 Read block 1389 len 8192
18497 219/300 Write block 4995 len 4096 data: 4
18498 220/300 Write block 3645 len 40960 data: 4 4 4 3 4 4 4 4 3 3
18499 221/300 Read block 1759 len 16384
18500 222/300 Write block 3452 len 40960 data: 5 5 4 4 5 3 3 3 4 3
18501 223/300 Write block 4130 len 4096 data: 6
18502 224/300 Read block 578 len 24576
18503 225/300 Read block 1822 len 36864
18504 226/300 Read block 3613 len 4096
18505 227/300 Read block 2994 len 28672
18506 228/300 Read block 2629 len 36864
18507 229/300 Write block 1405 len 8192 data: 7 8
18508 230/300 Write block 3961 len 4096 data: 4
18509 231/300 Write block 4132 len 8192 data: 5 6
18510 232/300 Read block 4317 len 8192
18511 233/300 Read block 1135 len 8192
18512 234/300 Flush
18513 235/300 Read block 3946 len 4096
18514 236/300 Write block 2864 len 8192 data: 8 7
18515 237/300 Write block 2743 len 24576 data: 6 6 7 4 7 7
18516 238/300 Write block 691 len 12288 data: 4 3 3
18517 239/300 Read block 3079 len 20480
18518 240/300 Write block 98 len 8192 data: 3 2
18519 241/300 Write block 4512 len 24576 data: 3 3 3 3 3 4
18520 242/300 Write block 512 len 4096 data: 6
18521 243/300 Read block 3377 len 20480
18522 244/300 Write block 824 len 4096 data: 7
18523 245/300 Read block 1069 len 36864
18524 246/300 Read block 1479 len 28672
18525 247/300 Write block 468 len 12288 data: 6 7 7
18526 248/300 Read block 3797 len 24576
18527 249/300 Read block 572 len 36864
18528 250/300 Write block 3160 len 16384 data: 9 9 8 7
18529 251/300 Flush
18530 252/300 Write block 2849 len 40960 data: 2 4 3 3 3 4 4 6 6 6
18531 253/300 Write block 1992 len 28672 data: 4 4 5 6 5 5 6
18532 254/300 Flush
18533 255/300 Read block 4262 len 24576
18534 256/300 Write block 2730 len 40960 data: 5 6 6 6 6 6 6 6 6 7
18535 257/300 Write block 3980 len 28672 data: 3 4 4 5 5 5 5
18536 258/300 Read block 1412 len 20480
18537 259/300 Flush
18538 260/300 Write block 4156 len 32768 data: 3 3 3 3 3 2 2 3
18539 261/300 Read block 1334 len 16384
18540 262/300 Read block 3333 len 20480
18541 263/300 Read block 2396 len 4096
18542 264/300 Read block 1151 len 8192
18543 265/300 Read block 2567 len 16384
18544 266/300 Read block 1091 len 40960
18545 267/300 Read block 2960 len 8192
18546 268/300 Read block 2472 len 28672
18547 269/300 Read block 3058 len 12288
18548 270/300 Write block 3413 len 32768 data: 6 5 5 3 3 4 4 4
18549 271/300 Read block 308 len 20480
18550 272/300 Read block 2841 len 36864
18551 273/300 Write block 4719 len 8192 data: 4 4
18552 274/300 Write block 1667 len 12288 data: 2 2 2
18553 275/300 Read block 3373 len 12288
18554 276/300 Read block 3021 len 20480
18555 277/300 Write block 4014 len 28672 data: 7 8 7 8 6 7 8
18556 278/300 Write block 3766 len 20480 data: 7 7 7 6 6
18557 279/300 Flush
18558 280/300 Write block 218 len 20480 data: 2 5 5 5 5
18559 281/300 Read block 773 len 16384
18560 282/300 Read block 3709 len 20480
18561 283/300 Read block 2961 len 16384
18562 284/300 Read block 4734 len 4096
18563 285/300 Read block 4410 len 8192
18564 286/300 Read block 4670 len 4096
18565 287/300 Write block 2423 len 16384 data: 5 5 5 5
18566 288/300 Read block 2294 len 24576
18567 289/300 Write block 1943 len 20480 data: 3 2 2 2 2
18568 290/300 Read block 3455 len 32768
18569 291/300 Read block 77 len 32768
18570 292/300 Flush
18571 293/300 Flush
18572 294/300 Read block 1471 len 16384
18573 295/300 Write block 1413 len 20480 data: 4 3 5 5 5
18574 296/300 Write block 801 len 36864 data: 4 3 4 5 5 5 4 4 4
18575 297/300 Read block 705 len 16384
18576 298/300 Write block 1231 len 8192 data: 5 5
18577 299/300 Write block 4004 len 12288 data: 5 4 3
18578 300/300 Read block 2929 len 4096
18579 Main task has work to do, go find it
18580 got dsc Start(0)
18581 start 0
18582 Feb 15 03:01:13.094 INFO request completed, latency_us: 204, response_code: 204, uri: /start/cid/0, method: POST, req_id: 42a87edd-6943-42ac-8f7e-223082ebbe2b, remote_addr: 127.0.0.1:54744, local_addr: 127.0.0.1:9998
18583 Replay: started 0, returned:Ok(())
18584 ----------------------------------------------------------------
18585 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
18586 States: Offline Active Active
18587 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
18588 5427 Acked 6426 Flush 0 New Done Done true
18589 5428 Acked 6427 Read 6 New Done Done true
18590 5429 Acked 6428 Read 9 New Done Done true
18591 5430 Acked 6429 Flush 0 New Done Done true
18592 5431 Acked 6430 Read 3 New Done Done false
18593 5432 Acked 6431 Write 1 New Done Done false
18594 5433 Acked 6432 Read 7 New Done Done false
18595 5434 Acked 6433 Write 8 New Done Done false
18596 5435 Acked 6434 Read 9 New Done Done false
18597 5436 Acked 6435 Write 1 New Done Done false
18598 5437 Acked 6436 Write 5 New Done Done false
18599 5438 Acked 6437 Write 2 New Done Done false
18600 5439 Acked 6438 Read 3 New Done Done false
18601 5440 Acked 6439 Read 10 New Done Done false
18602 5441 Acked 6440 Write 1 New Done Done false
18603 5442 Acked 6441 Write 7 New Done Done false
18604 5443 Acked 6442 Read 8 New Done Done false
18605 5444 Acked 6443 Write 5 New Done Done false
18606 5445 Acked 6444 Write 5 New Done Done false
18607 5446 Acked 6445 Flush 0 New Done Done false
18608 5447 Acked 6446 Flush 0 New Done Done false
18609 5448 Acked 6447 Write 8 New Done Done false
18610 5449 Acked 6448 Write 7 New Done Done false
18611 5450 Acked 6449 Write 3 New Done Done false
18612 5451 Acked 6450 Read 8 New Done Done false
18613 5452 Acked 6451 Read 8 New Done Done false
18614 5453 Acked 6452 Write 8 New Done Done false
18615 5454 Acked 6453 Write 10 New Done Done false
18616 5455 Acked 6454 Flush 0 New Done Done false
18617 5456 Acked 6455 Write 2 New Done Done false
18618 5457 Acked 6456 Write 7 New Done Done false
18619 5458 Acked 6457 Write 4 New Done Done false
18620 5459 Acked 6458 Read 5 New Done Done false
18621 5460 Acked 6459 Read 9 New Done Done false
18622 5461 Acked 6460 Read 2 New Done Done false
18623 5462 Acked 6461 Write 7 New Done Done false
18624 5463 Acked 6462 Write 2 New Done Done false
18625 5464 Acked 6463 Write 3 New Done Done false
18626 5465 Acked 6464 Write 10 New Done Done false
18627 5466 Acked 6465 Read 9 New Done Done false
18628 5467 Acked 6466 Write 3 New Done Done false
18629 5468 Acked 6467 Read 7 New Done Done false
18630 5469 Acked 6468 Read 4 New Done Done false
18631 5470 Acked 6469 Flush 0 New Done Done false
18632 5471 Acked 6470 Write 10 New Done Done false
18633 5472 Acked 6471 Flush 0 New Done Done false
18634 5473 Acked 6472 Read 8 New Done Done false
18635 5474 Acked 6473 Flush 0 New Done Done false
18636 5475 Acked 6474 Write 6 New Done Done false
18637 5476 Acked 6475 Write 6 New Done Done false
18638 5477 Acked 6476 Read 9 New Done Done false
18639 5478 Acked 6477 Flush 0 New Done Done false
18640 5479 Acked 6478 Read 2 New Done Done false
18641 5480 Acked 6479 Write 6 New Done Done false
18642 5481 Acked 6480 Write 7 New Done Done false
18643 5482 Acked 6481 Read 4 New Done Done false
18644 5483 Acked 6482 Read 5 New Done Done false
18645 5484 Acked 6483 Read 6 New Done Done false
18646 5485 Acked 6484 Flush 0 New Done Done false
18647 5486 Acked 6485 Read 3 New Done Done false
18648 5487 Acked 6486 Write 5 New Done Done false
18649 5488 Acked 6487 Read 6 New Done Done false
18650 5489 Acked 6488 Flush 0 New Done Done false
18651 5490 Acked 6489 Write 5 New Done Done false
18652 5491 Acked 6490 Write 4 New Done Done false
18653 5492 Acked 6491 Read 8 New Done Done false
18654 5493 Acked 6492 Write 10 New Done Done false
18655 5494 Acked 6493 Write 3 New Done Done false
18656 5495 Acked 6494 Write 1 New Done Done false
18657 5496 Acked 6495 Write 8 New Done Done false
18658 5497 Acked 6496 Write 10 New Done Done false
18659 5498 Acked 6497 Read 1 New Done Done false
18660 5499 Acked 6498 Read 5 New Done Done false
18661 5500 Acked 6499 Write 8 New Done Done false
18662 5501 Acked 6500 Write 9 New Done Done false
18663 5502 Acked 6501 Write 1 New Done Done false
18664 5503 Acked 6502 Write 7 New Done Done false
18665 5504 Acked 6503 Read 9 New Done Done false
18666 5505 Acked 6504 Write 4 New Done Done false
18667 5506 Acked 6505 Read 6 New Done Done false
18668 5507 Acked 6506 Write 8 New Done Done false
18669 5508 Acked 6507 Read 1 New Done Done false
18670 5509 Acked 6508 Read 4 New Done Done false
18671 5510 Acked 6509 Read 8 New Done Done false
18672 5511 Acked 6510 Read 9 New Done Done false
18673 5512 Acked 6511 Read 10 New Done Done false
18674 5513 Acked 6512 Flush 0 New Done Done false
18675 5514 Acked 6513 Read 4 New Done Done false
18676 5515 Acked 6514 Read 5 New Done Done false
18677 5516 Acked 6515 Write 2 New Done Done false
18678 5517 Acked 6516 Write 10 New Done Done false
18679 5518 Acked 6517 Write 10 New Done Done false
18680 5519 Acked 6518 Read 4 New Done Done false
18681 5520 Acked 6519 Read 7 New Done Done false
18682 5521 Acked 6520 Read 8 New Done Done false
18683 5522 Acked 6521 Write 2 New Done Done false
18684 5523 Acked 6522 Read 7 New Done Done false
18685 5524 Acked 6523 Write 8 New Done Done false
18686 5525 Acked 6524 Write 6 New Done Done false
18687 5526 Acked 6525 Read 6 New Done Done false
18688 5527 Acked 6526 Read 1 New Done Done false
18689 5528 Acked 6527 Read 6 New Done Done false
18690 5529 Acked 6528 Write 7 New Done Done false
18691 5530 Acked 6529 Read 2 New Done Done false
18692 5531 Acked 6530 Read 9 New Done Done false
18693 5532 Acked 6531 Write 4 New Done Done false
18694 5533 Acked 6532 Write 4 New Done Done false
18695 5534 Acked 6533 Read 8 New Done Done false
18696 5535 Acked 6534 Write 2 New Done Done false
18697 5536 Acked 6535 Write 7 New Done Done false
18698 5537 Acked 6536 Write 10 New Done Done false
18699 5538 Acked 6537 Write 8 New Done Done false
18700 5539 Acked 6538 Flush 0 New Done Done false
18701 5540 Acked 6539 Flush 0 New Done Done false
18702 5541 Acked 6540 Read 1 New Done Done false
18703 5542 Acked 6541 Read 3 New Done Done false
18704 5543 Acked 6542 Write 7 New Done Done false
18705 5544 Acked 6543 Write 3 New Done Done false
18706 5545 Acked 6544 Read 1 New Done Done false
18707 5546 Acked 6545 Write 5 New Done Done false
18708 5547 Acked 6546 Write 3 New Done Done false
18709 5548 Acked 6547 Read 2 New Done Done false
18710 5549 Acked 6548 Write 4 New Done Done false
18711 5550 Acked 6549 Write 3 New Done Done false
18712 5551 Acked 6550 Write 1 New Done Done false
18713 5552 Acked 6551 Write 4 New Done Done false
18714 5553 Acked 6552 Write 3 New Done Done false
18715 5554 Acked 6553 Write 3 New Done Done false
18716 5555 Acked 6554 Write 7 New Done Done false
18717 5556 Acked 6555 Read 6 New Done Done false
18718 5557 Acked 6556 Write 9 New Done Done false
18719 5558 Acked 6557 Write 10 New Done Done false
18720 5559 Acked 6558 Read 3 New Done Done false
18721 5560 Acked 6559 Flush 0 New Done Done false
18722 5561 Acked 6560 Read 9 New Done Done false
18723 5562 Acked 6561 Read 4 New Done Done false
18724 5563 Acked 6562 Read 4 New Done Done false
18725 5564 Acked 6563 Flush 0 New Done Done false
18726 5565 Acked 6564 Read 4 New Done Done false
18727 5566 Acked 6565 Write 5 New Done Done false
18728 5567 Acked 6566 Write 10 New Done Done false
18729 5568 Acked 6567 Read 4 New Done Done false
18730 5569 Acked 6568 Read 1 New Done Done false
18731 5570 Acked 6569 Flush 0 New Done Done false
18732 5571 Acked 6570 Read 6 New Done Done false
18733 5572 Acked 6571 Read 10 New Done Done false
18734 5573 Acked 6572 Read 7 New Done Done false
18735 5574 Acked 6573 Read 7 New Done Done false
18736 5575 Acked 6574 Read 2 New Done Done false
18737 5576 Acked 6575 Read 4 New Done Done false
18738 5577 Acked 6576 Write 3 New Done Done false
18739 5578 Acked 6577 Read 6 New Done Done false
18740 5579 Acked 6578 Read 8 New Done Done false
18741 5580 Acked 6579 Read 4 New Done Done false
18742 5581 Acked 6580 Read 4 New Done Done false
18743 5582 Acked 6581 Read 7 New Done Done false
18744 5583 Acked 6582 Write 10 New Done Done false
18745 5584 Acked 6583 Read 7 New Done Done false
18746 5585 Acked 6584 Read 6 New Done Done false
18747 5586 Acked 6585 Read 3 New Done Done false
18748 5587 Acked 6586 Read 9 New Done Done false
18749 5588 Acked 6587 Write 4 New Done Done false
18750 5589 Acked 6588 Write 7 New Done Done false
18751 5590 Acked 6589 Read 6 New Done Done false
18752 5591 Acked 6590 Read 2 New Done Done false
18753 5592 Acked 6591 Read 9 New Done Done false
18754 5593 Acked 6592 Read 9 New Done Done false
18755 5594 Acked 6593 Write 8 New Done Done false
18756 5595 Acked 6594 Read 10 New Done Done false
18757 5596 Acked 6595 Write 4 New Done Done false
18758 5597 Acked 6596 Write 8 New Done Done false
18759 5598 Acked 6597 Write 5 New Done Done false
18760 5599 Acked 6598 Read 10 New Done Done false
18761 5600 Acked 6599 Read 10 New Done Done false
18762 5601 Acked 6600 Read 6 New Done Done false
18763 5602 Acked 6601 Read 2 New Done Done false
18764 5603 Acked 6602 Write 10 New Done Done false
18765 5604 Acked 6603 Flush 0 New Done Done false
18766 5605 Acked 6604 Write 7 New Done Done false
18767 5606 Acked 6605 Read 1 New Done Done false
18768 5607 Acked 6606 Write 9 New Done Done false
18769 5608 Acked 6607 Read 2 New Done Done false
18770 5609 Acked 6608 Read 2 New Done Done false
18771 5610 Acked 6609 Write 2 New Done Done false
18772 5611 Acked 6610 Write 1 New Done Done false
18773 5612 Acked 6611 Write 1 New Done Done false
18774 5613 Acked 6612 Read 2 New Done Done false
18775 5614 Acked 6613 Read 3 New Done Done false
18776 5615 Acked 6614 Read 8 New Done Done false
18777 5616 Acked 6615 Read 3 New Done Done false
18778 5617 Acked 6616 Read 1 New Done Done false
18779 5618 Acked 6617 Write 9 New Done Done false
18780 5619 Acked 6618 Read 1 New Done Done false
18781 5620 Acked 6619 Read 5 New Done Done false
18782 5621 Acked 6620 Read 8 New Done Done false
18783 5622 Acked 6621 Read 3 New Done Done false
18784 5623 Acked 6622 Flush 0 New Done Done false
18785 5624 Acked 6623 Read 6 New Done Done false
18786 5625 Acked 6624 Write 3 New Done Done false
18787 5626 Acked 6625 Read 6 New Done Done false
18788 5627 Acked 6626 Read 3 New Done Done false
18789 5628 Acked 6627 Read 7 New Done Done false
18790 5629 Acked 6628 Read 2 New Done Done false
18791 5630 Acked 6629 Read 7 New Done Done false
18792 5631 Acked 6630 Write 4 New Done Done false
18793 5632 Acked 6631 Read 2 New Done Done false
18794 5633 Acked 6632 Read 5 New Done Done false
18795 5634 Acked 6633 Read 1 New Done Done false
18796 5635 Acked 6634 Write 5 New Done Done false
18797 5636 Acked 6635 Write 7 New Done Done false
18798 5637 Acked 6636 Flush 0 New Done Done false
18799 5638 Acked 6637 Read 10 New Done Done false
18800 5639 Acked 6638 Read 4 New Done Done false
18801 5640 Acked 6639 Read 8 New Done Done false
18802 5641 Acked 6640 Read 5 New Done Done false
18803 5642 Acked 6641 Flush 0 New Done Done false
18804 5643 Acked 6642 Read 4 New Done Done false
18805 5644 Acked 6643 Write 7 New Done Done false
18806 5645 Acked 6644 Read 2 New Done Done false
18807 5646 Acked 6645 Write 1 New Done Done false
18808 5647 Acked 6646 Write 10 New Done Done false
18809 5648 Acked 6647 Read 4 New Done Done false
18810 5649 Acked 6648 Write 10 New Done Done false
18811 5650 Acked 6649 Write 1 New Done Done false
18812 5651 Acked 6650 Read 6 New Done Done false
18813 5652 Acked 6651 Read 9 New Done Done false
18814 5653 Acked 6652 Read 1 New Done Done false
18815 5654 Acked 6653 Read 7 New Done Done false
18816 5655 Acked 6654 Read 9 New Done Done false
18817 5656 Acked 6655 Write 2 New Done Done false
18818 5657 Acked 6656 Write 1 New Done Done false
18819 5658 Acked 6657 Write 2 New Done Done false
18820 5659 Acked 6658 Read 2 New Done Done false
18821 5660 Acked 6659 Read 2 New Done Done false
18822 5661 Acked 6660 Flush 0 New Done Done false
18823 5662 Acked 6661 Read 1 New Done Done false
18824 5663 Acked 6662 Write 2 New Done Done false
18825 5664 Acked 6663 Write 6 New Done Done false
18826 5665 Acked 6664 Write 3 New Done Done false
18827 5666 Acked 6665 Read 5 New Done Done false
18828 5667 Acked 6666 Write 2 New Done Done false
18829 5668 Acked 6667 Write 6 New Done Done false
18830 5669 Acked 6668 Write 1 New Done Done false
18831 5670 Acked 6669 Read 5 New Done Done false
18832 5671 Acked 6670 Write 1 New Done Done false
18833 5672 Acked 6671 Read 9 New Done Done false
18834 5673 Acked 6672 Read 7 New Done Done false
18835 5674 Acked 6673 Write 3 New Done Done false
18836 5675 Acked 6674 Read 6 New Done Done false
18837 5676 Acked 6675 Read 9 New Done Done false
18838 5677 Acked 6676 Write 4 New Done Done false
18839 5678 Acked 6677 Flush 0 New Done Done false
18840 5679 Acked 6678 Write 10 New Done Done false
18841 5680 Acked 6679 Write 7 New Done Done false
18842 5681 Acked 6680 Flush 0 New Done Done false
18843 5682 Acked 6681 Read 6 New Done Done false
18844 5683 Acked 6682 Write 10 New Done Done false
18845 5684 Acked 6683 Write 7 New Done Done false
18846 5685 Acked 6684 Read 5 New Done Done false
18847 5686 Acked 6685 Flush 0 New Done Done false
18848 5687 Acked 6686 Write 8 New Done Done false
18849 5688 Acked 6687 Read 4 New Done Done false
18850 5689 Acked 6688 Read 5 New Done Done false
18851 5690 Acked 6689 Read 1 New Done Done false
18852 5691 Acked 6690 Read 2 New Done Done false
18853 5692 Acked 6691 Read 4 New Done Done false
18854 5693 Acked 6692 Read 10 New Done Done false
18855 5694 Acked 6693 Read 2 New Done Done false
18856 5695 Acked 6694 Read 7 New Done Done false
18857 5696 Acked 6695 Read 3 New Done Done false
18858 5697 Acked 6696 Write 8 New Done Done false
18859 5698 Acked 6697 Read 5 New Done Done false
18860 5699 Acked 6698 Read 9 New Done Done false
18861 5700 Acked 6699 Write 2 New Done Done false
18862 5701 Acked 6700 Write 3 New Done Done false
18863 5702 Acked 6701 Read 3 New Done Done false
18864 5703 Acked 6702 Read 5 New Done Done false
18865 5704 Acked 6703 Write 7 New Done Done false
18866 5705 Acked 6704 Write 5 New Done Done false
18867 5706 Acked 6705 Flush 0 New Done Done false
18868 5707 Acked 6706 Write 5 New Done Done false
18869 5708 Acked 6707 Read 4 New Done Done false
18870 5709 Acked 6708 Read 5 New Done Done false
18871 5710 Acked 6709 Read 4 New Done Done false
18872 5711 Acked 6710 Read 1 New Done Done false
18873 5712 Acked 6711 Read 2 New Done Done false
18874 5713 Acked 6712 Read 1 New Done Done false
18875 5714 Acked 6713 Write 4 New Done Done false
18876 5715 Acked 6714 Read 6 New Done Done false
18877 5716 Acked 6715 Write 5 New Done Done false
18878 5717 Acked 6716 Read 8 New Done Done false
18879 5718 Acked 6717 Read 8 New Done Done false
18880 5719 Acked 6718 Flush 0 New Done Done false
18881 5720 Acked 6719 Flush 0 New Done Done false
18882 5721 Acked 6720 Read 4 New Done Done false
18883 5722 Acked 6721 Write 5 New Done Done false
18884 5723 Acked 6722 Write 9 New Done Done false
18885 5724 Acked 6723 Read 4 New Done Done false
18886 5725 Acked 6724 Write 2 New Done Done false
18887 5726 Acked 6725 Write 3 New Done Done false
18888 5727 Acked 6726 Read 1 New Done Done false
18889 STATES DS:0 DS:1 DS:2 TOTAL
18890 New 301 0 0 301
18891 Sent 0 0 0 0
18892 Done 0 301 301 602
18893 Skipped 0 0 0 0
18894 Error 0 0 0 0
18895 Last Flush: 6425 6719 6719
18896 Downstairs last five completed: 6425 6424 6423 6422 6421
18897 Upstairs last five completed: 5727 5726 5725 5724 5723
18898 CLIENT: Up:0 ds:301 act:2
188992024-02-15T03:01:13.778ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
189002024-02-15T03:01:13.778ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
189012024-02-15T03:01:13.778ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
189022024-02-15T03:01:13.778ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
189032024-02-15T03:01:13.778ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
189042024-02-15T03:01:13.778ZINFOcrucible: [0] client re-new 302 jobs since flush 6425 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
189052024-02-15T03:01:13.778ZINFOcrucible: client task is exiting = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
189062024-02-15T03:01:14.779ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
189072024-02-15T03:01:14.779ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
189082024-02-15T03:01:14.780ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
189092024-02-15T03:01:14.780ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
189102024-02-15T03:01:14.780ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
189112024-02-15T03:01:14.780ZINFOcrucible: [0] client re-new 302 jobs since flush 6425 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
189122024-02-15T03:01:14.780ZINFOcrucible: client task is exiting = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
189132024-02-15T03:01:15.781ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
189142024-02-15T03:01:15.781ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
189152024-02-15T03:01:15.781ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
189162024-02-15T03:01:15.781ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
189172024-02-15T03:01:15.781ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
189182024-02-15T03:01:15.781ZINFOcrucible: [0] client re-new 302 jobs since flush 6425 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
189192024-02-15T03:01:15.781ZINFOcrucible: client task is exiting = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
189202024-02-15T03:01:16.782ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
189212024-02-15T03:01:16.782ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
189222024-02-15T03:01:16.782ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
189232024-02-15T03:01:16.782ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
189242024-02-15T03:01:16.782ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
189252024-02-15T03:01:16.782ZINFOcrucible: [0] client re-new 302 jobs since flush 6425 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
189262024-02-15T03:01:16.782ZINFOcrucible: client task is exiting = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
18927 ----------------------------------------------------------------
18928 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
18929 States: Offline Active Active
18930 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
18931 5427 Acked 6426 Flush 0 New Done Done true
18932 5428 Acked 6427 Read 6 New Done Done true
18933 5429 Acked 6428 Read 9 New Done Done true
18934 5430 Acked 6429 Flush 0 New Done Done true
18935 5431 Acked 6430 Read 3 New Done Done true
18936 5432 Acked 6431 Write 1 New Done Done true
18937 5433 Acked 6432 Read 7 New Done Done true
18938 5434 Acked 6433 Write 8 New Done Done true
18939 5435 Acked 6434 Read 9 New Done Done true
18940 5436 Acked 6435 Write 1 New Done Done true
18941 5437 Acked 6436 Write 5 New Done Done true
18942 5438 Acked 6437 Write 2 New Done Done true
18943 5439 Acked 6438 Read 3 New Done Done true
18944 5440 Acked 6439 Read 10 New Done Done true
18945 5441 Acked 6440 Write 1 New Done Done true
18946 5442 Acked 6441 Write 7 New Done Done true
18947 5443 Acked 6442 Read 8 New Done Done true
18948 5444 Acked 6443 Write 5 New Done Done true
18949 5445 Acked 6444 Write 5 New Done Done true
18950 5446 Acked 6445 Flush 0 New Done Done true
18951 5447 Acked 6446 Flush 0 New Done Done true
18952 5448 Acked 6447 Write 8 New Done Done true
18953 5449 Acked 6448 Write 7 New Done Done true
18954 5450 Acked 6449 Write 3 New Done Done true
18955 5451 Acked 6450 Read 8 New Done Done true
18956 5452 Acked 6451 Read 8 New Done Done true
18957 5453 Acked 6452 Write 8 New Done Done true
18958 5454 Acked 6453 Write 10 New Done Done true
18959 5455 Acked 6454 Flush 0 New Done Done true
18960 5456 Acked 6455 Write 2 New Done Done true
18961 5457 Acked 6456 Write 7 New Done Done true
18962 5458 Acked 6457 Write 4 New Done Done true
18963 5459 Acked 6458 Read 5 New Done Done true
18964 5460 Acked 6459 Read 9 New Done Done true
18965 5461 Acked 6460 Read 2 New Done Done true
18966 5462 Acked 6461 Write 7 New Done Done true
18967 5463 Acked 6462 Write 2 New Done Done true
18968 5464 Acked 6463 Write 3 New Done Done true
18969 5465 Acked 6464 Write 10 New Done Done true
18970 5466 Acked 6465 Read 9 New Done Done true
18971 5467 Acked 6466 Write 3 New Done Done true
18972 5468 Acked 6467 Read 7 New Done Done true
18973 5469 Acked 6468 Read 4 New Done Done true
18974 5470 Acked 6469 Flush 0 New Done Done true
18975 5471 Acked 6470 Write 10 New Done Done true
18976 5472 Acked 6471 Flush 0 New Done Done true
18977 5473 Acked 6472 Read 8 New Done Done true
18978 5474 Acked 6473 Flush 0 New Done Done true
18979 5475 Acked 6474 Write 6 New Done Done true
18980 5476 Acked 6475 Write 6 New Done Done true
18981 5477 Acked 6476 Read 9 New Done Done true
18982 5478 Acked 6477 Flush 0 New Done Done true
18983 5479 Acked 6478 Read 2 New Done Done true
18984 5480 Acked 6479 Write 6 New Done Done true
18985 5481 Acked 6480 Write 7 New Done Done true
18986 5482 Acked 6481 Read 4 New Done Done true
18987 5483 Acked 6482 Read 5 New Done Done true
18988 5484 Acked 6483 Read 6 New Done Done true
18989 5485 Acked 6484 Flush 0 New Done Done true
18990 5486 Acked 6485 Read 3 New Done Done true
18991 5487 Acked 6486 Write 5 New Done Done true
18992 5488 Acked 6487 Read 6 New Done Done true
18993 5489 Acked 6488 Flush 0 New Done Done true
18994 5490 Acked 6489 Write 5 New Done Done true
18995 5491 Acked 6490 Write 4 New Done Done true
18996 5492 Acked 6491 Read 8 New Done Done true
18997 5493 Acked 6492 Write 10 New Done Done true
18998 5494 Acked 6493 Write 3 New Done Done true
18999 5495 Acked 6494 Write 1 New Done Done true
19000 5496 Acked 6495 Write 8 New Done Done true
19001 5497 Acked 6496 Write 10 New Done Done true
19002 5498 Acked 6497 Read 1 New Done Done true
19003 5499 Acked 6498 Read 5 New Done Done true
19004 5500 Acked 6499 Write 8 New Done Done true
19005 5501 Acked 6500 Write 9 New Done Done true
19006 5502 Acked 6501 Write 1 New Done Done true
19007 5503 Acked 6502 Write 7 New Done Done true
19008 5504 Acked 6503 Read 9 New Done Done true
19009 5505 Acked 6504 Write 4 New Done Done true
19010 5506 Acked 6505 Read 6 New Done Done true
19011 5507 Acked 6506 Write 8 New Done Done true
19012 5508 Acked 6507 Read 1 New Done Done true
19013 5509 Acked 6508 Read 4 New Done Done true
19014 5510 Acked 6509 Read 8 New Done Done true
19015 5511 Acked 6510 Read 9 New Done Done true
19016 5512 Acked 6511 Read 10 New Done Done true
19017 5513 Acked 6512 Flush 0 New Done Done true
19018 5514 Acked 6513 Read 4 New Done Done true
19019 5515 Acked 6514 Read 5 New Done Done true
19020 5516 Acked 6515 Write 2 New Done Done true
19021 5517 Acked 6516 Write 10 New Done Done true
19022 5518 Acked 6517 Write 10 New Done Done true
19023 5519 Acked 6518 Read 4 New Done Done true
19024 5520 Acked 6519 Read 7 New Done Done true
19025 5521 Acked 6520 Read 8 New Done Done true
19026 5522 Acked 6521 Write 2 New Done Done true
19027 5523 Acked 6522 Read 7 New Done Done true
19028 5524 Acked 6523 Write 8 New Done Done true
19029 5525 Acked 6524 Write 6 New Done Done true
19030 5526 Acked 6525 Read 6 New Done Done true
19031 5527 Acked 6526 Read 1 New Done Done true
19032 5528 Acked 6527 Read 6 New Done Done true
19033 5529 Acked 6528 Write 7 New Done Done true
19034 5530 Acked 6529 Read 2 New Done Done true
19035 5531 Acked 6530 Read 9 New Done Done true
19036 5532 Acked 6531 Write 4 New Done Done true
19037 5533 Acked 6532 Write 4 New Done Done true
19038 5534 Acked 6533 Read 8 New Done Done true
19039 5535 Acked 6534 Write 2 New Done Done true
19040 5536 Acked 6535 Write 7 New Done Done true
19041 5537 Acked 6536 Write 10 New Done Done true
19042 5538 Acked 6537 Write 8 New Done Done true
19043 5539 Acked 6538 Flush 0 New Done Done true
19044 5540 Acked 6539 Flush 0 New Done Done true
19045 5541 Acked 6540 Read 1 New Done Done true
19046 5542 Acked 6541 Read 3 New Done Done true
19047 5543 Acked 6542 Write 7 New Done Done true
19048 5544 Acked 6543 Write 3 New Done Done true
19049 5545 Acked 6544 Read 1 New Done Done true
19050 5546 Acked 6545 Write 5 New Done Done true
19051 5547 Acked 6546 Write 3 New Done Done true
19052 5548 Acked 6547 Read 2 New Done Done true
19053 5549 Acked 6548 Write 4 New Done Done true
19054 5550 Acked 6549 Write 3 New Done Done true
19055 5551 Acked 6550 Write 1 New Done Done true
19056 5552 Acked 6551 Write 4 New Done Done true
19057 5553 Acked 6552 Write 3 New Done Done true
19058 5554 Acked 6553 Write 3 New Done Done true
19059 5555 Acked 6554 Write 7 New Done Done true
19060 5556 Acked 6555 Read 6 New Done Done true
19061 5557 Acked 6556 Write 9 New Done Done true
19062 5558 Acked 6557 Write 10 New Done Done true
19063 5559 Acked 6558 Read 3 New Done Done true
19064 5560 Acked 6559 Flush 0 New Done Done true
19065 5561 Acked 6560 Read 9 New Done Done true
19066 5562 Acked 6561 Read 4 New Done Done true
19067 5563 Acked 6562 Read 4 New Done Done true
19068 5564 Acked 6563 Flush 0 New Done Done true
19069 5565 Acked 6564 Read 4 New Done Done true
19070 5566 Acked 6565 Write 5 New Done Done true
19071 5567 Acked 6566 Write 10 New Done Done true
19072 5568 Acked 6567 Read 4 New Done Done true
19073 5569 Acked 6568 Read 1 New Done Done true
19074 5570 Acked 6569 Flush 0 New Done Done true
19075 5571 Acked 6570 Read 6 New Done Done true
19076 5572 Acked 6571 Read 10 New Done Done true
19077 5573 Acked 6572 Read 7 New Done Done true
19078 5574 Acked 6573 Read 7 New Done Done true
19079 5575 Acked 6574 Read 2 New Done Done true
19080 5576 Acked 6575 Read 4 New Done Done true
19081 5577 Acked 6576 Write 3 New Done Done true
19082 5578 Acked 6577 Read 6 New Done Done true
19083 5579 Acked 6578 Read 8 New Done Done true
19084 5580 Acked 6579 Read 4 New Done Done true
19085 5581 Acked 6580 Read 4 New Done Done true
19086 5582 Acked 6581 Read 7 New Done Done true
19087 5583 Acked 6582 Write 10 New Done Done true
19088 5584 Acked 6583 Read 7 New Done Done true
19089 5585 Acked 6584 Read 6 New Done Done true
19090 5586 Acked 6585 Read 3 New Done Done true
19091 5587 Acked 6586 Read 9 New Done Done true
19092 5588 Acked 6587 Write 4 New Done Done true
19093 5589 Acked 6588 Write 7 New Done Done true
19094 5590 Acked 6589 Read 6 New Done Done true
19095 5591 Acked 6590 Read 2 New Done Done true
19096 5592 Acked 6591 Read 9 New Done Done true
19097 5593 Acked 6592 Read 9 New Done Done true
19098 5594 Acked 6593 Write 8 New Done Done true
19099 5595 Acked 6594 Read 10 New Done Done true
19100 5596 Acked 6595 Write 4 New Done Done true
19101 5597 Acked 6596 Write 8 New Done Done true
19102 5598 Acked 6597 Write 5 New Done Done true
19103 5599 Acked 6598 Read 10 New Done Done true
19104 5600 Acked 6599 Read 10 New Done Done true
19105 5601 Acked 6600 Read 6 New Done Done true
19106 5602 Acked 6601 Read 2 New Done Done true
19107 5603 Acked 6602 Write 10 New Done Done true
19108 5604 Acked 6603 Flush 0 New Done Done true
19109 5605 Acked 6604 Write 7 New Done Done true
19110 5606 Acked 6605 Read 1 New Done Done true
19111 5607 Acked 6606 Write 9 New Done Done true
19112 5608 Acked 6607 Read 2 New Done Done true
19113 5609 Acked 6608 Read 2 New Done Done true
19114 5610 Acked 6609 Write 2 New Done Done true
19115 5611 Acked 6610 Write 1 New Done Done true
19116 5612 Acked 6611 Write 1 New Done Done true
19117 5613 Acked 6612 Read 2 New Done Done true
19118 5614 Acked 6613 Read 3 New Done Done true
19119 5615 Acked 6614 Read 8 New Done Done true
19120 5616 Acked 6615 Read 3 New Done Done true
19121 5617 Acked 6616 Read 1 New Done Done true
19122 5618 Acked 6617 Write 9 New Done Done true
19123 5619 Acked 6618 Read 1 New Done Done true
19124 5620 Acked 6619 Read 5 New Done Done true
19125 5621 Acked 6620 Read 8 New Done Done true
19126 5622 Acked 6621 Read 3 New Done Done true
19127 5623 Acked 6622 Flush 0 New Done Done true
19128 5624 Acked 6623 Read 6 New Done Done true
19129 5625 Acked 6624 Write 3 New Done Done true
19130 5626 Acked 6625 Read 6 New Done Done true
19131 5627 Acked 6626 Read 3 New Done Done true
19132 5628 Acked 6627 Read 7 New Done Done true
19133 5629 Acked 6628 Read 2 New Done Done true
19134 5630 Acked 6629 Read 7 New Done Done true
19135 5631 Acked 6630 Write 4 New Done Done true
19136 5632 Acked 6631 Read 2 New Done Done true
19137 5633 Acked 6632 Read 5 New Done Done true
19138 5634 Acked 6633 Read 1 New Done Done true
19139 5635 Acked 6634 Write 5 New Done Done true
19140 5636 Acked 6635 Write 7 New Done Done true
19141 5637 Acked 6636 Flush 0 New Done Done true
19142 5638 Acked 6637 Read 10 New Done Done true
19143 5639 Acked 6638 Read 4 New Done Done true
19144 5640 Acked 6639 Read 8 New Done Done true
19145 5641 Acked 6640 Read 5 New Done Done true
19146 5642 Acked 6641 Flush 0 New Done Done true
19147 5643 Acked 6642 Read 4 New Done Done true
19148 5644 Acked 6643 Write 7 New Done Done true
19149 5645 Acked 6644 Read 2 New Done Done true
19150 5646 Acked 6645 Write 1 New Done Done true
19151 5647 Acked 6646 Write 10 New Done Done true
19152 5648 Acked 6647 Read 4 New Done Done true
19153 5649 Acked 6648 Write 10 New Done Done true
19154 5650 Acked 6649 Write 1 New Done Done true
19155 5651 Acked 6650 Read 6 New Done Done true
19156 5652 Acked 6651 Read 9 New Done Done true
19157 5653 Acked 6652 Read 1 New Done Done true
19158 5654 Acked 6653 Read 7 New Done Done true
19159 5655 Acked 6654 Read 9 New Done Done true
19160 5656 Acked 6655 Write 2 New Done Done true
19161 5657 Acked 6656 Write 1 New Done Done true
19162 5658 Acked 6657 Write 2 New Done Done true
19163 5659 Acked 6658 Read 2 New Done Done true
19164 5660 Acked 6659 Read 2 New Done Done true
19165 5661 Acked 6660 Flush 0 New Done Done true
19166 5662 Acked 6661 Read 1 New Done Done true
19167 5663 Acked 6662 Write 2 New Done Done true
19168 5664 Acked 6663 Write 6 New Done Done true
19169 5665 Acked 6664 Write 3 New Done Done true
19170 5666 Acked 6665 Read 5 New Done Done true
19171 5667 Acked 6666 Write 2 New Done Done true
19172 5668 Acked 6667 Write 6 New Done Done true
19173 5669 Acked 6668 Write 1 New Done Done true
19174 5670 Acked 6669 Read 5 New Done Done true
19175 5671 Acked 6670 Write 1 New Done Done true
19176 5672 Acked 6671 Read 9 New Done Done true
19177 5673 Acked 6672 Read 7 New Done Done true
19178 5674 Acked 6673 Write 3 New Done Done true
19179 5675 Acked 6674 Read 6 New Done Done true
19180 5676 Acked 6675 Read 9 New Done Done true
19181 5677 Acked 6676 Write 4 New Done Done true
19182 5678 Acked 6677 Flush 0 New Done Done true
19183 5679 Acked 6678 Write 10 New Done Done true
19184 5680 Acked 6679 Write 7 New Done Done true
19185 5681 Acked 6680 Flush 0 New Done Done true
19186 5682 Acked 6681 Read 6 New Done Done true
19187 5683 Acked 6682 Write 10 New Done Done true
19188 5684 Acked 6683 Write 7 New Done Done true
19189 5685 Acked 6684 Read 5 New Done Done true
19190 5686 Acked 6685 Flush 0 New Done Done true
19191 5687 Acked 6686 Write 8 New Done Done true
19192 5688 Acked 6687 Read 4 New Done Done true
19193 5689 Acked 6688 Read 5 New Done Done true
19194 5690 Acked 6689 Read 1 New Done Done true
19195 5691 Acked 6690 Read 2 New Done Done true
19196 5692 Acked 6691 Read 4 New Done Done true
19197 5693 Acked 6692 Read 10 New Done Done true
19198 5694 Acked 6693 Read 2 New Done Done true
19199 5695 Acked 6694 Read 7 New Done Done true
19200 5696 Acked 6695 Read 3 New Done Done true
19201 5697 Acked 6696 Write 8 New Done Done true
19202 5698 Acked 6697 Read 5 New Done Done true
19203 5699 Acked 6698 Read 9 New Done Done true
19204 5700 Acked 6699 Write 2 New Done Done true
19205 5701 Acked 6700 Write 3 New Done Done true
19206 5702 Acked 6701 Read 3 New Done Done true
19207 5703 Acked 6702 Read 5 New Done Done true
19208 5704 Acked 6703 Write 7 New Done Done true
19209 5705 Acked 6704 Write 5 New Done Done true
19210 5706 Acked 6705 Flush 0 New Done Done true
19211 5707 Acked 6706 Write 5 New Done Done true
19212 5708 Acked 6707 Read 4 New Done Done true
19213 5709 Acked 6708 Read 5 New Done Done true
19214 5710 Acked 6709 Read 4 New Done Done true
19215 5711 Acked 6710 Read 1 New Done Done true
19216 5712 Acked 6711 Read 2 New Done Done true
19217 5713 Acked 6712 Read 1 New Done Done true
19218 5714 Acked 6713 Write 4 New Done Done true
19219 5715 Acked 6714 Read 6 New Done Done true
19220 5716 Acked 6715 Write 5 New Done Done true
19221 5717 Acked 6716 Read 8 New Done Done true
19222 5718 Acked 6717 Read 8 New Done Done true
19223 5719 Acked 6718 Flush 0 New Done Done true
19224 5720 Acked 6719 Flush 0 New Done Done true
19225 5721 Acked 6720 Read 4 New Done Done true
19226 5722 Acked 6721 Write 5 New Done Done true
19227 5723 Acked 6722 Write 9 New Done Done true
19228 5724 Acked 6723 Read 4 New Done Done true
19229 5725 Acked 6724 Write 2 New Done Done true
19230 5726 Acked 6725 Write 3 New Done Done true
19231 5727 Acked 6726 Read 1 New Done Done true
19232 5728 Acked 6727 Flush 0 New Done Done true
19233 STATES DS:0 DS:1 DS:2 TOTAL
19234 New 302 0 0 302
19235 Sent 0 0 0 0
19236 Done 0 302 302 604
19237 Skipped 0 0 0 0
19238 Error 0 0 0 0
19239 Last Flush: 6425 6727 6727
19240 Downstairs last five completed: 6425 6424 6423 6422 6421
19241 Upstairs last five completed: 5728 5727 5726 5725 5724
19242 CLIENT: Up:0 ds:302 act:2
192432024-02-15T03:01:17.783ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
192442024-02-15T03:01:17.783ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
192452024-02-15T03:01:17.783ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
192462024-02-15T03:01:17.783ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
192472024-02-15T03:01:17.783ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
192482024-02-15T03:01:17.783ZINFOcrucible: [0] client re-new 302 jobs since flush 6425 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
192492024-02-15T03:01:17.783ZINFOcrucible: client task is exiting = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
19250 [8810] got start action
19251 [8810] I am going to restart
19252 Starting downstairs at port 8810
19253 Make output file at "/tmp/dsc/downstairs-8810.txt"
19254 [8810][0] reports Starting
19255 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1186)
19256 [8810][0] reports Running
192572024-02-15T03:01:18.784ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
192582024-02-15T03:01:18.784ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
192592024-02-15T03:01:18.784ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
192602024-02-15T03:01:18.784ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
192612024-02-15T03:01:18.784ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
192622024-02-15T03:01:18.784ZINFOcrucible: [0] client re-new 302 jobs since flush 6425 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
192632024-02-15T03:01:18.785ZINFOcrucible: client task is exiting = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
192642024-02-15T03:01:19.785ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
192652024-02-15T03:01:19.785ZINFOcrucible: ds_connection connected = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
192662024-02-15T03:01:19.787ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
192672024-02-15T03:01:19.787ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
192682024-02-15T03:01:19.787ZINFOcrucible: send last flush ID to this DS: 6425 = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
192692024-02-15T03:01:19.787ZINFOcrucible: Replied this last flush ID: 6425 = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
192702024-02-15T03:01:19.787ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
192712024-02-15T03:01:19.787ZINFOcrucible: [0] Transition from Offline to Replay = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
192722024-02-15T03:01:19.787ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
192732024-02-15T03:01:19.788ZINFOcrucible: [0] Transition from Replay to Active = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
19274 ----------------------------------------------------------------
19275 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
19276 Downstairs last five completed: 6727 6726 6725 6724 6723
19277 Upstairs last five completed: 5728 5727 5726 5725 5724
19278 CLIENT: Up:0 ds:0 act:3
19279 Replay: All jobs finished, all DS active.
19280 Main task has work to do, go find it
19281 got dsc Stop(2)
19282 stop 2
19283 Feb 15 03:01:21.101 INFO request completed, latency_us:[8830] Got stop action so:false kr:false
19284 329, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 5b0f1376-d18c-464f-89e2-1ee810907e71, remote_addr: 127.0.0.1:54744, local_addr: 127.0.0.1:9998
192852024-02-15T03:01:21.101ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
192862024-02-15T03:01:21.101ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
192872024-02-15T03:01:21.101ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
192882024-02-15T03:01:21.102ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
19289 Feb 15 03:01:21.102 INFO request completed, latency_us: 154, response_code: 200, uri: /state/cid/2, method: GET, {"msg":"[2] client re-new 0 jobs since flush 6727","v":0,"name":"crucible","level":30,"time":"2024-02-15T03:01:21.102102428Z","hostname":"ip-10-150-1-67.us-west-2.compute.internal","pid":1167,"":"downstairs","":"downstairs","session_id":"4f5a63e5-aa00-4890-9f6b-96c2e069fdb2"}
19290 req_id: 5a96258a-75cd-4594-a037-ce9fd12a54b5, remote_addr:{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-15T03:01:21.102133614Z","hostname":"ip-10-150-1-67.us-west-2.compute.internal","pid":1167,"":"io task","client":"2","":"downstairs","session_id":"4f5a63e5-aa00-4890-9f6b-96c2e069fdb2"}
19291 127.0.0.1:54744, local_addr: 127.0.0.1:9998
19292 [8830] Exited with: ExitStatus(unix_wait_status(9))
19293 [8830][2] reports Exit
192942024-02-15T03:01:22.103ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
192952024-02-15T03:01:22.103ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
192962024-02-15T03:01:22.103ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
192972024-02-15T03:01:22.103ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
192982024-02-15T03:01:22.103ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
192992024-02-15T03:01:22.103ZINFOcrucible: [2] client re-new 0 jobs since flush 6727 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
193002024-02-15T03:01:22.103ZINFOcrucible: client task is exiting = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
193012024-02-15T03:01:23.105ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
193022024-02-15T03:01:23.105ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
193032024-02-15T03:01:23.105ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
193042024-02-15T03:01:23.105ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
193052024-02-15T03:01:23.105ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
193062024-02-15T03:01:23.105ZINFOcrucible: [2] client re-new 0 jobs since flush 6727 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
193072024-02-15T03:01:23.105ZINFOcrucible: client task is exiting = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
193082024-02-15T03:01:24.106ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
193092024-02-15T03:01:24.106ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
193102024-02-15T03:01:24.106ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
193112024-02-15T03:01:24.106ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
193122024-02-15T03:01:24.106ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
193132024-02-15T03:01:24.106ZINFOcrucible: [2] client re-new 0 jobs since flush 6727 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
193142024-02-15T03:01:24.106ZINFOcrucible: client task is exiting = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
19315 Feb 15 03:01:25.104 INFO request completed, latency_us: 211, response_code: 200, uri: /state/cid/2, method: GET, req_id: 3322666e-ab1c-4a00-ad92-479dd13f9d58, remote_addr: 127.0.0.1:54744, local_addr: 127.0.0.1:9998
19316 001/300 Flush
19317 002/300 Read block 2608 len 12288
19318 003/300 Read block 419 len 28672
193192024-02-15T03:01:25.107ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
193202024-02-15T03:01:25.107ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
193212024-02-15T03:01:25.107ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
193222024-02-15T03:01:25.107ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
193232024-02-15T03:01:25.107ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
193242024-02-15T03:01:25.107ZINFOcrucible: [2] client re-new 3 jobs since flush 6727 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
193252024-02-15T03:01:25.107ZINFOcrucible: client task is exiting = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
19326 004/300 Flush
19327 005/300 Read block 503 len 4096
19328 006/300 Write block 4475 len 32768 data: 4 4 4 4 4 4 4 5
19329 007/300 Write block 917 len 12288 data: 5 6 4
19330 008/300 Write block 1358 len 28672 data: 2 3 3 3 3 3 4
19331 009/300 Read block 1543 len 28672
19332 010/300 Write block 961 len 4096 data: 6
19333 011/300 Write block 1431 len 20480 data: 4 3 3 2 3
19334 012/300 Read block 3387 len 24576
19335 013/300 Write block 2715 len 28672 data: 4 5 5 5 4 4 4
19336 014/300 Write block 3338 len 24576 data: 6 6 5 6 5 5
19337 015/300 Flush
19338 016/300 Flush
19339 017/300 Read block 2956 len 20480
19340 018/300 Read block 4744 len 24576
19341 019/300 Read block 3354 len 24576
19342 020/300 Write block 986 len 12288 data: 4 4 6
19343 021/300 Flush
19344 022/300 Read block 2435 len 24576
19345 023/300 Read block 1219 len 28672
19346 024/300 Read block 380 len 16384
19347 025/300 Read block 3048 len 12288
19348 026/300 Read block 3705 len 24576
19349 027/300 Read block 1099 len 20480
19350 028/300 Write block 3509 len 8192 data: 4 4
19351 029/300 Read block 460 len 8192
19352 030/300 Write block 4352 len 36864 data: 4 3 3 3 2 3 4 5 4
19353 031/300 Write block 1439 len 8192 data: 4 4
19354 032/300 Write block 3580 len 16384 data: 2 2 3 3
19355 033/300 Write block 4961 len 12288 data: 3 3 4
19356 034/300 Read block 737 len 32768
19357 035/300 Read block 616 len 24576
19358 036/300 Write block 593 len 40960 data: 2 2 2 3 2 4 4 5 8 7
19359 037/300 Read block 1697 len 12288
19360 038/300 Read block 3049 len 16384
19361 039/300 Write block 1353 len 28672 data: 2 2 2 2 2 3 4
19362 040/300 Write block 3348 len 28672 data: 7 7 8 7 6 6 4
19363 041/300 Read block 3442 len 16384
19364 042/300 Read block 3921 len 16384
19365 043/300 Write block 3489 len 32768 data: 2 4 5 5 5 6 7 7
19366 044/300 Read block 3059 len 40960
19367 045/300 Read block 3749 len 28672
19368 046/300 Read block 2782 len 40960
19369 047/300 Write block 2858 len 40960 data: 7 5 6 7 6 7 9 8 8 8
19370 048/300 Read block 998 len 24576
19371 049/300 Read block 4531 len 12288
19372 050/300 Read block 365 len 12288
19373 051/300 Read block 4488 len 20480
19374 052/300 Read block 273 len 20480
19375 053/300 Read block 466 len 20480
19376 054/300 Read block 634 len 8192
19377 055/300 Read block 2933 len 36864
19378 056/300 Write block 4452 len 28672 data: 4 4 3 4 4 4 4
19379 057/300 Write block 3481 len 20480 data: 5 5 6 6 6
19380 058/300 Read block 1584 len 36864
19381 059/300 Write block 859 len 28672 data: 5 5 5 5 6 5 6
19382 060/300 Write block 2431 len 32768 data: 5 5 4 3 4 5 5 4
19383 061/300 Read block 804 len 32768
19384 062/300 Write block 223 len 16384 data: 5 4 3 5
19385 063/300 Write block 4551 len 40960 data: 7 7 5 6 4 4 3 3 4 4
19386 064/300 Write block 1219 len 24576 data: 5 5 6 5 4 5
19387 065/300 Write block 2913 len 40960 data: 6 4 5 4 4 4 4 4 4 2
19388 066/300 Read block 4376 len 28672
19389 067/300 Read block 4959 len 8192
19390 068/300 Write block 3577 len 4096 data: 2
19391 069/300 Write block 690 len 36864 data: 4 5 4 4 4 4 4 3 3
19392 070/300 Read block 4113 len 12288
19393 071/300 Write block 4005 len 20480 data: 5 4 3 3 4
19394 072/300 Write block 4659 len 8192 data: 3 3
19395 073/300 Write block 491 len 24576 data: 5 7 6 5 5 4
19396 074/300 Flush
19397 075/300 Write block 2922 len 8192 data: 3 3
19398 076/300 Write block 904 len 8192 data: 4 4
19399 077/300 Read block 3049 len 16384
19400 078/300 Write block 2502 len 40960 data: 5 6 6 6 6 6 5 5 3 4
19401 079/300 Write block 2313 len 40960 data: 3 4 3 4 4 3 3 3 3 2
19402 080/300 Write block 2008 len 12288 data: 4 3 4
19403 081/300 Read block 3213 len 20480
19404 082/300 Read block 2338 len 4096
19405 083/300 Flush
19406 084/300 Read block 4243 len 20480
19407 085/300 Read block 1237 len 36864
19408 086/300 Write block 3315 len 36864 data: 3 3 3 3 3 4 4 4 3
19409 087/300 Write block 2040 len 16384 data: 2 2 3 2
19410 088/300 Read block 2557 len 20480
19411 089/300 Write block 2985 len 28672 data: 6 6 6 6 5 5 3
19412 090/300 Read block 4356 len 24576
19413 091/300 Read block 3337 len 4096
19414 092/300 Read block 4796 len 20480
19415 093/300 Write block 2848 len 32768 data: 3 3 5 4 4 4 5 5
19416 094/300 Write block 4668 len 24576 data: 4 4 5 4 6 6
19417 095/300 Read block 1245 len 28672
19418 096/300 Read block 3190 len 4096
19419 097/300 Read block 477 len 40960
19420 098/300 Write block 3790 len 40960 data: 6 5 4 3 4 5 5 4 4 5
19421 099/300 Write block 4691 len 20480 data: 8 8 7 7 6
19422 100/300 Read block 1542 len 8192
19423 101/300 Write block 1754 len 36864 data: 2 2 3 3 4 4 5 4 4
19424 102/300 Read block 2245 len 20480
19425 103/300 Read block 2699 len 8192
19426 104/300 Read block 2891 len 12288
19427 105/300 Write block 3613 len 24576 data: 5 6 5 6 7 8
19428 106/300 Read block 1622 len 32768
19429 107/300 Read block 2700 len 32768
19430 108/300 Write block 4384 len 4096 data: 4
19431 109/300 Read block 1107 len 40960
19432 110/300 Read block 2154 len 20480
19433 111/300 Write block 4286 len 20480 data: 3 3 3 3 3
19434 112/300 Read block 2539 len 20480
19435 113/300 Read block 3692 len 24576
19436 114/300 Read block 1946 len 36864
19437 115/300 Write block 3048 len 32768 data: 5 6 6 7 8 7 7 7
19438 116/300 Read block 1908 len 8192
19439 117/300 Read block 4982 len 16384
19440 118/300 Read block 1384 len 8192
19441 119/300 Read block 1734 len 16384
19442 120/300 Read block 1187 len 20480
19443 121/300 Flush
19444 122/300 Flush
19445 123/300 Read block 1900 len 40960
19446 124/300 Write block 4303 len 4096 data: 2
19447 125/300 Write block 1895 len 20480 data: 2 2 2 2 2
19448 126/300 Write block 2853 len 32768 data: 5 6 6 7 7 8 6 7
19449 127/300 Write block 306 len 16384 data: 4 4 5 5
19450 128/300 Flush
19451 129/300 Read block 4848 len 8192
19452 130/300 Read block 2007 len 8192
19453 131/300 Read block 4147 len 32768
19454 132/300 Write block 2549 len 8192 data: 7 5
19455 133/300 Write block 3381 len 12288 data: 3 2 2
19456 134/300 Read block 4762 len 24576
19457 135/300 Read block 287 len 24576
19458 136/300 Write block 1508 len 16384 data: 6 3 3 3
19459 137/300 Read block 348 len 32768
19460 138/300 Write block 4760 len 16384 data: 5 5 6 4
19461 139/300 Read block 4012 len 28672
19462 140/300 Flush
19463 141/300 Write block 3689 len 4096 data: 6
19464 142/300 Write block 4941 len 8192 data: 5 4
19465 143/300 Flush
19466 144/300 Write block 543 len 40960 data: 3 3 3 3 5 5 4 3 3 3
19467 145/300 Write block 4766 len 4096 data: 2
19468 146/300 Read block 700 len 32768
19469 147/300 Write block 2656 len 40960 data: 3 3 3 4 4 6 4 5 7 6
19470 148/300 Write block 3921 len 8192 data: 3 2
19471 149/300 Write block 78 len 32768 data: 6 4 4 4 4 4 3 3
19472 150/300 Flush
19473 151/300 Read block 3814 len 4096
19474 152/300 Read block 931 len 4096
19475 153/300 Read block 2732 len 20480
19476 154/300 Read block 3826 len 8192
19477 155/300 Read block 882 len 20480
19478 156/300 Flush
19479 157/300 Flush
19480 158/300 Write block 3945 len 40960 data: 5 5 6 7 7 5 6 5 4 5
19481 159/300 Write block 3833 len 16384 data: 7 6 5 5
19482 160/300 Read block 2233 len 20480
19483 161/300 Read block 4826 len 40960
19484 162/300 Write block 4185 len 36864 data: 3 3 4 3 4 4 4 4 4
19485 163/300 Write block 4782 len 12288 data: 7 6 6
19486 164/300 Write block 372 len 20480 data: 7 6 5 4 6
19487 165/300 Write block 4268 len 24576 data: 6 5 5 4 4 5
19488 166/300 Write block 704 len 20480 data: 3 3 3 5 4
19489 167/300 Read block 2551 len 28672
19490 168/300 Read block 2253 len 36864
19491 169/300 Read block 332 len 12288
19492 170/300 Write block 3505 len 20480 data: 6 7 7 5 5
19493 171/300 Write block 575 len 12288 data: 4 5 5
19494 172/300 Read block 4019 len 8192
19495 173/300 Write block 353 len 8192 data: 7 7
19496 174/300 Flush
19497 175/300 Read block 3683 len 28672
19498 176/300 Flush
19499 177/300 Write block 2975 len 36864 data: 3 3 3 3 3 2 2 2 2
19500 178/300 Read block 2785 len 28672
19501 179/300 Write block 22 len 36864 data: 8 9 9 7 7 7 6 6 4
19502 180/300 Read block 2578 len 40960
19503 181/300 Read block 2767 len 20480
19504 182/300 Read block 4910 len 36864
19505 183/300 Write block 2833 len 36864 data: 3 3 4 4 4 6 7 7 7
19506 184/300 Write block 3417 len 12288 data: 4 5 5
19507 185/300 Read block 2053 len 32768
19508 186/300 Read block 4536 len 36864
19509 187/300 Write block 4907 len 20480 data: 8 7 6 7 7
19510 188/300 Read block 4299 len 16384
19511 189/300 Read block 689 len 36864
19512 190/300 Read block 3557 len 28672
19513 191/300 Read block 2254 len 16384
19514 192/300 Read block 4259 len 4096
19515 193/300 Write block 3481 len 28672 data: 6 6 7 7 7 4 4
19516 194/300 Read block 2778 len 12288
19517 195/300 Read block 484 len 8192
19518 196/300 Write block 3612 len 16384 data: 5 6 7 6
19519 197/300 Write block 3864 len 8192 data: 7 7
19520 198/300 Read block 146 len 40960
19521 199/300 Read block 2427 len 12288
19522 200/300 Write block 3199 len 12288 data: 4 6 6
19523 201/300 Read block 1176 len 40960
19524 202/300 Flush
19525 203/300 Read block 3257 len 32768
19526 204/300 Read block 1991 len 20480
19527 205/300 Read block 1401 len 32768
19528 206/300 Write block 2087 len 24576 data: 6 5 3 4 2 2
19529 207/300 Write block 304 len 24576 data: 4 5 5 5 6 6
19530 208/300 Write block 1561 len 8192 data: 5 6
19531 209/300 Read block 4262 len 8192
19532 210/300 Read block 738 len 40960
19533 211/300 Read block 3123 len 4096
19534 212/300 Read block 3068 len 24576
19535 213/300 Write block 1057 len 36864 data: 3 3 5 5 5 5 5 6 4
19536 214/300 Read block 3007 len 40960
19537 215/300 Write block 3756 len 28672 data: 4 4 3 3 3 3 4
19538 216/300 Read block 3781 len 4096
19539 217/300 Write block 3418 len 36864 data: 6 6 5 5 4 3 4 4 3
19540 218/300 Flush
19541 219/300 Read block 3487 len 40960
19542 220/300 Flush
19543 221/300 Read block 1391 len 28672
19544 222/300 Flush
19545 223/300 Read block 4319 len 8192
19546 224/300 Read block 151 len 8192
19547 225/300 Read block 3153 len 36864
19548 226/300 Write block 1582 len 8192 data: 6 8
19549 227/300 Read block 4294 len 16384
19550 228/300 Write block 3875 len 16384 data: 5 5 5 5
19551 229/300 Read block 3685 len 32768
19552 230/300 Write block 15 len 4096 data: 3
19553 231/300 Flush
19554 232/300 Write block 2782 len 16384 data: 5 5 6 5
19555 233/300 Write block 4280 len 20480 data: 5 5 5 6 6
19556 234/300 Read block 4657 len 4096
19557 235/300 Write block 1076 len 40960 data: 3 3 3 3 4 4 4 4 5 5
19558 236/300 Write block 3241 len 4096 data: 8
19559 237/300 Flush
19560 238/300 Flush
19561 239/300 Read block 454 len 28672
19562 240/300 Flush
19563 241/300 Write block 3104 len 16384 data: 4 5 5 6
19564 242/300 Write block 544 len 12288 data: 4 4 4
19565 243/300 Write block 1316 len 8192 data: 4 3
19566 244/300 Write block 942 len 28672 data: 3 3 3 2 3 3 3
19567 245/300 Write block 1321 len 40960 data: 7 7 8 8 6 7 8 8 7 6
19568 246/300 Flush
19569 247/300 Read block 571 len 16384
19570 248/300 Flush
19571 249/300 Read block 863 len 16384
19572 250/300 Write block 3009 len 32768 data: 3 3 4 3 3 3 3 3
19573 251/300 Write block 4868 len 24576 data: 6 4 4 5 5 4
19574 252/300 Write block 1156 len 32768 data: 3 3 4 4 2 2 2 2
19575 253/300 Write block 1199 len 40960 data: 6 5 5 4 4 4 4 3 3 3
19576 254/300 Read block 161 len 28672
19577 255/300 Write block 4162 len 32768 data: 3 4 3 4 5 6 5 5
19578 256/300 Write block 1395 len 28672 data: 4 5 3 3 3 4 3
19579 257/300 Read block 1569 len 12288
19580 258/300 Read block 1411 len 16384
19581 259/300 Write block 2433 len 16384 data: 5 4 5 6
19582 260/300 Read block 3328 len 36864
19583 261/300 Read block 1631 len 40960
19584 262/300 Write block 1137 len 32768 data: 5 5 5 4 4 3 2 4
19585 263/300 Write block 2520 len 32768 data: 4 5 5 5 6 6 4 4
19586 264/300 Read block 2719 len 36864
19587 265/300 Read block 1468 len 28672
19588 266/300 Write block 2586 len 16384 data: 3 3 3 3
19589 267/300 Read block 2263 len 28672
19590 268/300 Write block 18 len 24576 data: 5 4 5 7 9 10
19591 269/300 Write block 1992 len 16384 data: 5 5 6 7
19592 270/300 Write block 3157 len 16384 data: 10 11 10 10
19593 271/300 Write block 2714 len 32768 data: 4 5 6 6 6 5 5 5
19594 272/300 Read block 4148 len 8192
19595 273/300 Write block 4837 len 28672 data: 5 6 4 4 4 4 3
19596 274/300 Read block 4365 len 32768
19597 275/300 Write block 1329 len 4096 data: 8
19598 276/300 Write block 4717 len 12288 data: 4 3 5
19599 277/300 Read block 4693 len 8192
19600 278/300 Read block 2227 len 24576
19601 279/300 Write block 4393 len 36864 data: 5 6 6 4 3 3 3 2 3
19602 280/300 Write block 166 len 28672 data: 6 5 5 4 4 4 4
19603 281/300 Read block 2692 len 28672
19604 282/300 Write block 4205 len 36864 data: 6 7 6 4 4 5 4 4 5
19605 283/300 Write block 2886 len 8192 data: 3 3
19606 284/300 Read block 1824 len 24576
19607 285/300 Read block 3560 len 12288
19608 286/300 Write block 3466 len 32768 data: 2 3 3 3 2 2 3 3
19609 287/300 Read block 4117 len 8192
19610 288/300 Read block 2632 len 40960
19611 289/300 Write block 3614 len 40960 data: 8 7 7 8 9 8 8 6 6 8
19612 290/300 Flush
19613 291/300 Flush
19614 292/300 Write block 1446 len 4096 data: 3
19615 293/300 Write block 3156 len 40960 data: 10 11 12 11 11 10 9 8 6 7
19616 294/300 Read block 4401 len 12288
19617 295/300 Read block 1239 len 24576
19618 296/300 Write block 2479 len 12288 data: 7 7 7
19619 297/300 Write block 40 len 40960 data: 2 2 2 3 3 3 3 3 4 4
19620 298/300 Read block 3751 len 12288
19621 299/300 Read block 280 len 12288
19622 300/300 Write block 3963 len 4096 data: 4
19623 Main task has work to do, go find it
19624 got dsc Start(2)
19625 start 2
19626 Feb 15 03:01:25.431 INFO request completed, latency_us: 198, response_code: 204, uri: /start/cid/2, method: POST, req_id: f58ac82c-e85a-41bc-af39-e00df006a7b1, remote_addr: 127.0.0.1:54744, local_addr: 127.0.0.1:9998
19627 Replay: started 2, returned:Ok(())
19628 ----------------------------------------------------------------
19629 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:300
19630 States: Active Active Offline
19631 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
19632 5729 Acked 6728 Flush 0 Done Done New true
19633 5730 Acked 6729 Read 3 Done Done New true
19634 5731 Acked 6730 Read 7 Done Done New true
19635 5732 Acked 6731 Flush 0 Done Done New false
19636 5733 Acked 6732 Read 1 Done Done New false
19637 5734 Acked 6733 Write 8 Done Done New false
19638 5735 Acked 6734 Write 3 Done Done New false
19639 5736 Acked 6735 Write 7 Done Done New false
19640 5737 Acked 6736 Read 7 Done Done New false
19641 5738 Acked 6737 Write 1 Done Done New false
19642 5739 Acked 6738 Write 5 Done Done New false
19643 5740 Acked 6739 Read 6 Done Done New false
19644 5741 Acked 6740 Write 7 Done Done New false
19645 5742 Acked 6741 Write 6 Done Done New false
19646 5743 Acked 6742 Flush 0 Done Done New false
19647 5744 Acked 6743 Flush 0 Done Done New false
19648 5745 Acked 6744 Read 5 Done Done New false
19649 5746 Acked 6745 Read 6 Done Done New false
19650 5747 Acked 6746 Read 6 Done Done New false
19651 5748 Acked 6747 Write 3 Done Done New false
19652 5749 Acked 6748 Flush 0 Done Done New false
19653 5750 Acked 6749 Read 6 Done Done New false
19654 5751 Acked 6750 Read 7 Done Done New false
19655 5752 Acked 6751 Read 4 Done Done New false
19656 5753 Acked 6752 Read 3 Done Done New false
19657 5754 Acked 6753 Read 6 Done Done New false
19658 5755 Acked 6754 Read 5 Done Done New false
19659 5756 Acked 6755 Write 2 Done Done New false
19660 5757 Acked 6756 Read 2 Done Done New false
19661 5758 Acked 6757 Write 9 Done Done New false
19662 5759 Acked 6758 Write 2 Done Done New false
19663 5760 Acked 6759 Write 4 Done Done New false
19664 5761 Acked 6760 Write 3 Done Done New false
19665 5762 Acked 6761 Read 8 Done Done New false
19666 5763 Acked 6762 Read 6 Done Done New false
19667 5764 Acked 6763 Write 10 Done Done New false
19668 5765 Acked 6764 Read 3 Done Done New false
19669 5766 Acked 6765 Read 4 Done Done New false
19670 5767 Acked 6766 Write 7 Done Done New false
19671 5768 Acked 6767 Write 7 Done Done New false
19672 5769 Acked 6768 Read 4 Done Done New false
19673 5770 Acked 6769 Read 4 Done Done New false
19674 5771 Acked 6770 Write 8 Done Done New false
19675 5772 Acked 6771 Read 10 Done Done New false
19676 5773 Acked 6772 Read 7 Done Done New false
19677 5774 Acked 6773 Read 10 Done Done New false
19678 5775 Acked 6774 Write 10 Done Done New false
19679 5776 Acked 6775 Read 6 Done Done New false
19680 5777 Acked 6776 Read 3 Done Done New false
19681 5778 Acked 6777 Read 3 Done Done New false
19682 5779 Acked 6778 Read 5 Done Done New false
19683 5780 Acked 6779 Read 5 Done Done New false
19684 5781 Acked 6780 Read 5 Done Done New false
19685 5782 Acked 6781 Read 2 Done Done New false
19686 5783 Acked 6782 Read 9 Done Done New false
19687 5784 Acked 6783 Write 7 Done Done New false
19688 5785 Acked 6784 Write 5 Done Done New false
19689 5786 Acked 6785 Read 9 Done Done New false
19690 5787 Acked 6786 Write 7 Done Done New false
19691 5788 Acked 6787 Write 8 Done Done New false
19692 5789 Acked 6788 Read 8 Done Done New false
19693 5790 Acked 6789 Write 4 Done Done New false
19694 5791 Acked 6790 Write 10 Done Done New false
19695 5792 Acked 6791 Write 6 Done Done New false
19696 5793 Acked 6792 Write 10 Done Done New false
19697 5794 Acked 6793 Read 7 Done Done New false
19698 5795 Acked 6794 Read 2 Done Done New false
19699 5796 Acked 6795 Write 1 Done Done New false
19700 5797 Acked 6796 Write 9 Done Done New false
19701 5798 Acked 6797 Read 3 Done Done New false
19702 5799 Acked 6798 Write 5 Done Done New false
19703 5800 Acked 6799 Write 2 Done Done New false
19704 5801 Acked 6800 Write 6 Done Done New false
19705 5802 Acked 6801 Flush 0 Done Done New false
19706 5803 Acked 6802 Write 2 Done Done New false
19707 5804 Acked 6803 Write 2 Done Done New false
19708 5805 Acked 6804 Read 4 Done Done New false
19709 5806 Acked 6805 Write 10 Done Done New false
19710 5807 Acked 6806 Write 10 Done Done New false
19711 5808 Acked 6807 Write 3 Done Done New false
19712 5809 Acked 6808 Read 5 Done Done New false
19713 5810 Acked 6809 Read 1 Done Done New false
19714 5811 Acked 6810 Flush 0 Done Done New false
19715 5812 Acked 6811 Read 5 Done Done New false
19716 5813 Acked 6812 Read 9 Done Done New false
19717 5814 Acked 6813 Write 9 Done Done New false
19718 5815 Acked 6814 Write 4 Done Done New false
19719 5816 Acked 6815 Read 5 Done Done New false
19720 5817 Acked 6816 Write 7 Done Done New false
19721 5818 Acked 6817 Read 6 Done Done New false
19722 5819 Acked 6818 Read 1 Done Done New false
19723 5820 Acked 6819 Read 5 Done Done New false
19724 5821 Acked 6820 Write 8 Done Done New false
19725 5822 Acked 6821 Write 6 Done Done New false
19726 5823 Acked 6822 Read 7 Done Done New false
19727 5824 Acked 6823 Read 1 Done Done New false
19728 5825 Acked 6824 Read 10 Done Done New false
19729 5826 Acked 6825 Write 10 Done Done New false
19730 5827 Acked 6826 Write 5 Done Done New false
19731 5828 Acked 6827 Read 2 Done Done New false
19732 5829 Acked 6828 Write 9 Done Done New false
19733 5830 Acked 6829 Read 5 Done Done New false
19734 5831 Acked 6830 Read 2 Done Done New false
19735 5832 Acked 6831 Read 3 Done Done New false
19736 5833 Acked 6832 Write 6 Done Done New false
19737 5834 Acked 6833 Read 8 Done Done New false
19738 5835 Acked 6834 Read 8 Done Done New false
19739 5836 Acked 6835 Write 1 Done Done New false
19740 5837 Acked 6836 Read 10 Done Done New false
19741 5838 Acked 6837 Read 5 Done Done New false
19742 5839 Acked 6838 Write 5 Done Done New false
19743 5840 Acked 6839 Read 5 Done Done New false
19744 5841 Acked 6840 Read 6 Done Done New false
19745 5842 Acked 6841 Read 9 Done Done New false
19746 5843 Acked 6842 Write 8 Done Done New false
19747 5844 Acked 6843 Read 2 Done Done New false
19748 5845 Acked 6844 Read 4 Done Done New false
19749 5846 Acked 6845 Read 2 Done Done New false
19750 5847 Acked 6846 Read 4 Done Done New false
19751 5848 Acked 6847 Read 5 Done Done New false
19752 5849 Acked 6848 Flush 0 Done Done New false
19753 5850 Acked 6849 Flush 0 Done Done New false
19754 5851 Acked 6850 Read 10 Done Done New false
19755 5852 Acked 6851 Write 1 Done Done New false
19756 5853 Acked 6852 Write 5 Done Done New false
19757 5854 Acked 6853 Write 8 Done Done New false
19758 5855 Acked 6854 Write 4 Done Done New false
19759 5856 Acked 6855 Flush 0 Done Done New false
19760 5857 Acked 6856 Read 2 Done Done New false
19761 5858 Acked 6857 Read 2 Done Done New false
19762 5859 Acked 6858 Read 8 Done Done New false
19763 5860 Acked 6859 Write 2 Done Done New false
19764 5861 Acked 6860 Write 3 Done Done New false
19765 5862 Acked 6861 Read 6 Done Done New false
19766 5863 Acked 6862 Read 6 Done Done New false
19767 5864 Acked 6863 Write 4 Done Done New false
19768 5865 Acked 6864 Read 8 Done Done New false
19769 5866 Acked 6865 Write 4 Done Done New false
19770 5867 Acked 6866 Read 7 Done Done New false
19771 5868 Acked 6867 Flush 0 Done Done New false
19772 5869 Acked 6868 Write 1 Done Done New false
19773 5870 Acked 6869 Write 2 Done Done New false
19774 5871 Acked 6870 Flush 0 Done Done New false
19775 5872 Acked 6871 Write 10 Done Done New false
19776 5873 Acked 6872 Write 1 Done Done New false
19777 5874 Acked 6873 Read 8 Done Done New false
19778 5875 Acked 6874 Write 10 Done Done New false
19779 5876 Acked 6875 Write 2 Done Done New false
19780 5877 Acked 6876 Write 8 Done Done New false
19781 5878 Acked 6877 Flush 0 Done Done New false
19782 5879 Acked 6878 Read 1 Done Done New false
19783 5880 Acked 6879 Read 1 Done Done New false
19784 5881 Acked 6880 Read 5 Done Done New false
19785 5882 Acked 6881 Read 2 Done Done New false
19786 5883 Acked 6882 Read 5 Done Done New false
19787 5884 Acked 6883 Flush 0 Done Done New false
19788 5885 Acked 6884 Flush 0 Done Done New false
19789 5886 Acked 6885 Write 10 Done Done New false
19790 5887 Acked 6886 Write 4 Done Done New false
19791 5888 Acked 6887 Read 5 Done Done New false
19792 5889 Acked 6888 Read 10 Done Done New false
19793 5890 Acked 6889 Write 9 Done Done New false
19794 5891 Acked 6890 Write 3 Done Done New false
19795 5892 Acked 6891 Write 5 Done Done New false
19796 5893 Acked 6892 Write 6 Done Done New false
19797 5894 Acked 6893 Write 5 Done Done New false
19798 5895 Acked 6894 Read 7 Done Done New false
19799 5896 Acked 6895 Read 9 Done Done New false
19800 5897 Acked 6896 Read 3 Done Done New false
19801 5898 Acked 6897 Write 5 Done Done New false
19802 5899 Acked 6898 Write 3 Done Done New false
19803 5900 Acked 6899 Read 2 Done Done New false
19804 5901 Acked 6900 Write 2 Done Done New false
19805 5902 Acked 6901 Flush 0 Done Done New false
19806 5903 Acked 6902 Read 7 Done Done New false
19807 5904 Acked 6903 Flush 0 Done Done New false
19808 5905 Acked 6904 Write 9 Done Done New false
19809 5906 Acked 6905 Read 7 Done Done New false
19810 5907 Acked 6906 Write 9 Done Done New false
19811 5908 Acked 6907 Read 10 Done Done New false
19812 5909 Acked 6908 Read 5 Done Done New false
19813 5910 Acked 6909 Read 9 Done Done New false
19814 5911 Acked 6910 Write 9 Done Done New false
19815 5912 Acked 6911 Write 3 Done Done New false
19816 5913 Acked 6912 Read 8 Done Done New false
19817 5914 Acked 6913 Read 9 Done Done New false
19818 5915 Acked 6914 Write 5 Done Done New false
19819 5916 Acked 6915 Read 4 Done Done New false
19820 5917 Acked 6916 Read 9 Done Done New false
19821 5918 Acked 6917 Read 7 Done Done New false
19822 5919 Acked 6918 Read 4 Done Done New false
19823 5920 Acked 6919 Read 1 Done Done New false
19824 5921 Acked 6920 Write 7 Done Done New false
19825 5922 Acked 6921 Read 3 Done Done New false
19826 5923 Acked 6922 Read 2 Done Done New false
19827 5924 Acked 6923 Write 4 Done Done New false
19828 5925 Acked 6924 Write 2 Done Done New false
19829 5926 Acked 6925 Read 10 Done Done New false
19830 5927 Acked 6926 Read 3 Done Done New false
19831 5928 Acked 6927 Write 3 Done Done New false
19832 5929 Acked 6928 Read 10 Done Done New false
19833 5930 Acked 6929 Flush 0 Done Done New false
19834 5931 Acked 6930 Read 8 Done Done New false
19835 5932 Acked 6931 Read 5 Done Done New false
19836 5933 Acked 6932 Read 8 Done Done New false
19837 5934 Acked 6933 Write 6 Done Done New false
19838 5935 Acked 6934 Write 6 Done Done New false
19839 5936 Acked 6935 Write 2 Done Done New false
19840 5937 Acked 6936 Read 2 Done Done New false
19841 5938 Acked 6937 Read 10 Done Done New false
19842 5939 Acked 6938 Read 1 Done Done New false
19843 5940 Acked 6939 Read 6 Done Done New false
19844 5941 Acked 6940 Write 9 Done Done New false
19845 5942 Acked 6941 Read 10 Done Done New false
19846 5943 Acked 6942 Write 7 Done Done New false
19847 5944 Acked 6943 Read 1 Done Done New false
19848 5945 Acked 6944 Write 9 Done Done New false
19849 5946 Acked 6945 Flush 0 Done Done New false
19850 5947 Acked 6946 Read 10 Done Done New false
19851 5948 Acked 6947 Flush 0 Done Done New false
19852 5949 Acked 6948 Read 7 Done Done New false
19853 5950 Acked 6949 Flush 0 Done Done New false
19854 5951 Acked 6950 Read 2 Done Done New false
19855 5952 Acked 6951 Read 2 Done Done New false
19856 5953 Acked 6952 Read 9 Done Done New false
19857 5954 Acked 6953 Write 2 Done Done New false
19858 5955 Acked 6954 Read 4 Done Done New false
19859 5956 Acked 6955 Write 4 Done Done New false
19860 5957 Acked 6956 Read 8 Done Done New false
19861 5958 Acked 6957 Write 1 Done Done New false
19862 5959 Acked 6958 Flush 0 Done Done New false
19863 5960 Acked 6959 Write 4 Done Done New false
19864 5961 Acked 6960 Write 5 Done Done New false
19865 5962 Acked 6961 Read 1 Done Done New false
19866 5963 Acked 6962 Write 10 Done Done New false
19867 5964 Acked 6963 Write 1 Done Done New false
19868 5965 Acked 6964 Flush 0 Done Done New false
19869 5966 Acked 6965 Flush 0 Done Done New false
19870 5967 Acked 6966 Read 7 Done Done New false
19871 5968 Acked 6967 Flush 0 Done Done New false
19872 5969 Acked 6968 Write 4 Done Done New false
19873 5970 Acked 6969 Write 3 Done Done New false
19874 5971 Acked 6970 Write 2 Done Done New false
19875 5972 Acked 6971 Write 7 Done Done New false
19876 5973 Acked 6972 Write 10 Done Done New false
19877 5974 Acked 6973 Flush 0 Done Done New false
19878 5975 Acked 6974 Read 4 Done Done New false
19879 5976 Acked 6975 Flush 0 Done Done New false
19880 5977 Acked 6976 Read 4 Done Done New false
19881 5978 Acked 6977 Write 8 Done Done New false
19882 5979 Acked 6978 Write 6 Done Done New false
19883 5980 Acked 6979 Write 8 Done Done New false
19884 5981 Acked 6980 Write 10 Done Done New false
19885 5982 Acked 6981 Read 7 Done Done New false
19886 5983 Acked 6982 Write 8 Done Done New false
19887 5984 Acked 6983 Write 7 Done Done New false
19888 5985 Acked 6984 Read 3 Done Done New false
19889 5986 Acked 6985 Read 4 Done Done New false
19890 5987 Acked 6986 Write 4 Done Done New false
19891 5988 Acked 6987 Read 9 Done Done New false
19892 5989 Acked 6988 Read 10 Done Done New false
19893 5990 Acked 6989 Write 8 Done Done New false
19894 5991 Acked 6990 Write 8 Done Done New false
19895 5992 Acked 6991 Read 9 Done Done New false
19896 5993 Acked 6992 Read 7 Done Done New false
19897 5994 Acked 6993 Write 4 Done Done New false
19898 5995 Acked 6994 Read 7 Done Done New false
19899 5996 Acked 6995 Write 6 Done Done New false
19900 5997 Acked 6996 Write 4 Done Done New false
19901 5998 Acked 6997 Write 4 Done Done New false
19902 5999 Acked 6998 Write 8 Done Done New false
19903 6000 Acked 6999 Read 2 Done Done New false
19904 6001 Acked 7000 Write 7 Done Done New false
19905 6002 Acked 7001 Read 8 Done Done New false
19906 6003 Acked 7002 Write 1 Done Done New false
19907 6004 Acked 7003 Write 3 Done Done New false
19908 6005 Acked 7004 Read 2 Done Done New false
19909 6006 Acked 7005 Read 6 Done Done New false
19910 6007 Acked 7006 Write 9 Done Done New false
19911 6008 Acked 7007 Write 7 Done Done New false
19912 6009 Acked 7008 Read 7 Done Done New false
19913 6010 Acked 7009 Write 9 Done Done New false
19914 6011 Acked 7010 Write 2 Done Done New false
19915 6012 Acked 7011 Read 6 Done Done New false
19916 6013 Acked 7012 Read 3 Done Done New false
19917 6014 Acked 7013 Write 8 Done Done New false
19918 6015 Acked 7014 Read 2 Done Done New false
19919 6016 Acked 7015 Read 10 Done Done New false
19920 6017 Acked 7016 Write 10 Done Done New false
19921 6018 Acked 7017 Flush 0 Done Done New false
19922 6019 Acked 7018 Flush 0 Done Done New false
19923 6020 Acked 7019 Write 1 Done Done New false
19924 6021 Acked 7020 Write 10 Done Done New false
19925 6022 Acked 7021 Read 3 Done Done New false
19926 6023 Acked 7022 Read 6 Done Done New false
19927 6024 Acked 7023 Write 3 Done Done New false
19928 6025 Acked 7024 Write 10 Done Done New false
19929 6026 Acked 7025 Read 3 Done Done New false
19930 6027 Acked 7026 Read 3 Done Done New false
19931 6028 Acked 7027 Write 1 Done Done New false
19932 STATES DS:0 DS:1 DS:2 TOTAL
19933 New 0 0 300 300
19934 Sent 0 0 0 0
19935 Done 300 300 0 600
19936 Skipped 0 0 0 0
19937 Error 0 0 0 0
19938 Last Flush: 7018 7018 6727
19939 Downstairs last five completed: 6727 6726 6725 6724 6723
19940 Upstairs last five completed: 6028 6027 6026 6025 6024
19941 CLIENT: Up:0 ds:300 act:2
199422024-02-15T03:01:26.108ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
199432024-02-15T03:01:26.108ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
199442024-02-15T03:01:26.108ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
199452024-02-15T03:01:26.108ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
199462024-02-15T03:01:26.108ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
199472024-02-15T03:01:26.108ZINFOcrucible: [2] client re-new 301 jobs since flush 6727 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
199482024-02-15T03:01:26.108ZINFOcrucible: client task is exiting = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
199492024-02-15T03:01:27.109ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
199502024-02-15T03:01:27.109ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
199512024-02-15T03:01:27.109ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
199522024-02-15T03:01:27.109ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
199532024-02-15T03:01:27.109ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
199542024-02-15T03:01:27.109ZINFOcrucible: [2] client re-new 301 jobs since flush 6727 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
199552024-02-15T03:01:27.110ZINFOcrucible: client task is exiting = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
199562024-02-15T03:01:28.110ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
199572024-02-15T03:01:28.110ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
199582024-02-15T03:01:28.110ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
199592024-02-15T03:01:28.110ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
199602024-02-15T03:01:28.110ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
199612024-02-15T03:01:28.110ZINFOcrucible: [2] client re-new 301 jobs since flush 6727 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
199622024-02-15T03:01:28.110ZINFOcrucible: client task is exiting = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
199632024-02-15T03:01:29.111ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
199642024-02-15T03:01:29.112ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
199652024-02-15T03:01:29.112ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
199662024-02-15T03:01:29.112ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
199672024-02-15T03:01:29.112ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
199682024-02-15T03:01:29.112ZINFOcrucible: [2] client re-new 301 jobs since flush 6727 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
199692024-02-15T03:01:29.112ZINFOcrucible: client task is exiting = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
19970 ----------------------------------------------------------------
19971 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
19972 States: Active Active Offline
19973 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
19974 5729 Acked 6728 Flush 0 Done Done New true
19975 5730 Acked 6729 Read 3 Done Done New true
19976 5731 Acked 6730 Read 7 Done Done New true
19977 5732 Acked 6731 Flush 0 Done Done New true
19978 5733 Acked 6732 Read 1 Done Done New true
19979 5734 Acked 6733 Write 8 Done Done New true
19980 5735 Acked 6734 Write 3 Done Done New true
19981 5736 Acked 6735 Write 7 Done Done New true
19982 5737 Acked 6736 Read 7 Done Done New true
19983 5738 Acked 6737 Write 1 Done Done New true
19984 5739 Acked 6738 Write 5 Done Done New true
19985 5740 Acked 6739 Read 6 Done Done New true
19986 5741 Acked 6740 Write 7 Done Done New true
19987 5742 Acked 6741 Write 6 Done Done New true
19988 5743 Acked 6742 Flush 0 Done Done New true
19989 5744 Acked 6743 Flush 0 Done Done New true
19990 5745 Acked 6744 Read 5 Done Done New true
19991 5746 Acked 6745 Read 6 Done Done New true
19992 5747 Acked 6746 Read 6 Done Done New true
19993 5748 Acked 6747 Write 3 Done Done New true
19994 5749 Acked 6748 Flush 0 Done Done New true
19995 5750 Acked 6749 Read 6 Done Done New true
19996 5751 Acked 6750 Read 7 Done Done New true
19997 5752 Acked 6751 Read 4 Done Done New true
19998 5753 Acked 6752 Read 3 Done Done New true
19999 5754 Acked 6753 Read 6 Done Done New true
20000 5755 Acked 6754 Read 5 Done Done New true
20001 5756 Acked 6755 Write 2 Done Done New true
20002 5757 Acked 6756 Read 2 Done Done New true
20003 5758 Acked 6757 Write 9 Done Done New true
20004 5759 Acked 6758 Write 2 Done Done New true
20005 5760 Acked 6759 Write 4 Done Done New true
20006 5761 Acked 6760 Write 3 Done Done New true
20007 5762 Acked 6761 Read 8 Done Done New true
20008 5763 Acked 6762 Read 6 Done Done New true
20009 5764 Acked 6763 Write 10 Done Done New true
20010 5765 Acked 6764 Read 3 Done Done New true
20011 5766 Acked 6765 Read 4 Done Done New true
20012 5767 Acked 6766 Write 7 Done Done New true
20013 5768 Acked 6767 Write 7 Done Done New true
20014 5769 Acked 6768 Read 4 Done Done New true
20015 5770 Acked 6769 Read 4 Done Done New true
20016 5771 Acked 6770 Write 8 Done Done New true
20017 5772 Acked 6771 Read 10 Done Done New true
20018 5773 Acked 6772 Read 7 Done Done New true
20019 5774 Acked 6773 Read 10 Done Done New true
20020 5775 Acked 6774 Write 10 Done Done New true
20021 5776 Acked 6775 Read 6 Done Done New true
20022 5777 Acked 6776 Read 3 Done Done New true
20023 5778 Acked 6777 Read 3 Done Done New true
20024 5779 Acked 6778 Read 5 Done Done New true
20025 5780 Acked 6779 Read 5 Done Done New true
20026 5781 Acked 6780 Read 5 Done Done New true
20027 5782 Acked 6781 Read 2 Done Done New true
20028 5783 Acked 6782 Read 9 Done Done New true
20029 5784 Acked 6783 Write 7 Done Done New true
20030 5785 Acked 6784 Write 5 Done Done New true
20031 5786 Acked 6785 Read 9 Done Done New true
20032 5787 Acked 6786 Write 7 Done Done New true
20033 5788 Acked 6787 Write 8 Done Done New true
20034 5789 Acked 6788 Read 8 Done Done New true
20035 5790 Acked 6789 Write 4 Done Done New true
20036 5791 Acked 6790 Write 10 Done Done New true
20037 5792 Acked 6791 Write 6 Done Done New true
20038 5793 Acked 6792 Write 10 Done Done New true
20039 5794 Acked 6793 Read 7 Done Done New true
20040 5795 Acked 6794 Read 2 Done Done New true
20041 5796 Acked 6795 Write 1 Done Done New true
20042 5797 Acked 6796 Write 9 Done Done New true
20043 5798 Acked 6797 Read 3 Done Done New true
20044 5799 Acked 6798 Write 5 Done Done New true
20045 5800 Acked 6799 Write 2 Done Done New true
20046 5801 Acked 6800 Write 6 Done Done New true
20047 5802 Acked 6801 Flush 0 Done Done New true
20048 5803 Acked 6802 Write 2 Done Done New true
20049 5804 Acked 6803 Write 2 Done Done New true
20050 5805 Acked 6804 Read 4 Done Done New true
20051 5806 Acked 6805 Write 10 Done Done New true
20052 5807 Acked 6806 Write 10 Done Done New true
20053 5808 Acked 6807 Write 3 Done Done New true
20054 5809 Acked 6808 Read 5 Done Done New true
20055 5810 Acked 6809 Read 1 Done Done New true
20056 5811 Acked 6810 Flush 0 Done Done New true
20057 5812 Acked 6811 Read 5 Done Done New true
20058 5813 Acked 6812 Read 9 Done Done New true
20059 5814 Acked 6813 Write 9 Done Done New true
20060 5815 Acked 6814 Write 4 Done Done New true
20061 5816 Acked 6815 Read 5 Done Done New true
20062 5817 Acked 6816 Write 7 Done Done New true
20063 5818 Acked 6817 Read 6 Done Done New true
20064 5819 Acked 6818 Read 1 Done Done New true
20065 5820 Acked 6819 Read 5 Done Done New true
20066 5821 Acked 6820 Write 8 Done Done New true
20067 5822 Acked 6821 Write 6 Done Done New true
20068 5823 Acked 6822 Read 7 Done Done New true
20069 5824 Acked 6823 Read 1 Done Done New true
20070 5825 Acked 6824 Read 10 Done Done New true
20071 5826 Acked 6825 Write 10 Done Done New true
20072 5827 Acked 6826 Write 5 Done Done New true
20073 5828 Acked 6827 Read 2 Done Done New true
20074 5829 Acked 6828 Write 9 Done Done New true
20075 5830 Acked 6829 Read 5 Done Done New true
20076 5831 Acked 6830 Read 2 Done Done New true
20077 5832 Acked 6831 Read 3 Done Done New true
20078 5833 Acked 6832 Write 6 Done Done New true
20079 5834 Acked 6833 Read 8 Done Done New true
20080 5835 Acked 6834 Read 8 Done Done New true
20081 5836 Acked 6835 Write 1 Done Done New true
20082 5837 Acked 6836 Read 10 Done Done New true
20083 5838 Acked 6837 Read 5 Done Done New true
20084 5839 Acked 6838 Write 5 Done Done New true
20085 5840 Acked 6839 Read 5 Done Done New true
20086 5841 Acked 6840 Read 6 Done Done New true
20087 5842 Acked 6841 Read 9 Done Done New true
20088 5843 Acked 6842 Write 8 Done Done New true
20089 5844 Acked 6843 Read 2 Done Done New true
20090 5845 Acked 6844 Read 4 Done Done New true
20091 5846 Acked 6845 Read 2 Done Done New true
20092 5847 Acked 6846 Read 4 Done Done New true
20093 5848 Acked 6847 Read 5 Done Done New true
20094 5849 Acked 6848 Flush 0 Done Done New true
20095 5850 Acked 6849 Flush 0 Done Done New true
20096 5851 Acked 6850 Read 10 Done Done New true
20097 5852 Acked 6851 Write 1 Done Done New true
20098 5853 Acked 6852 Write 5 Done Done New true
20099 5854 Acked 6853 Write 8 Done Done New true
20100 5855 Acked 6854 Write 4 Done Done New true
20101 5856 Acked 6855 Flush 0 Done Done New true
20102 5857 Acked 6856 Read 2 Done Done New true
20103 5858 Acked 6857 Read 2 Done Done New true
20104 5859 Acked 6858 Read 8 Done Done New true
20105 5860 Acked 6859 Write 2 Done Done New true
20106 5861 Acked 6860 Write 3 Done Done New true
20107 5862 Acked 6861 Read 6 Done Done New true
20108 5863 Acked 6862 Read 6 Done Done New true
20109 5864 Acked 6863 Write 4 Done Done New true
20110 5865 Acked 6864 Read 8 Done Done New true
20111 5866 Acked 6865 Write 4 Done Done New true
20112 5867 Acked 6866 Read 7 Done Done New true
20113 5868 Acked 6867 Flush 0 Done Done New true
20114 5869 Acked 6868 Write 1 Done Done New true
20115 5870 Acked 6869 Write 2 Done Done New true
20116 5871 Acked 6870 Flush 0 Done Done New true
20117 5872 Acked 6871 Write 10 Done Done New true
20118 5873 Acked 6872 Write 1 Done Done New true
20119 5874 Acked 6873 Read 8 Done Done New true
20120 5875 Acked 6874 Write 10 Done Done New true
20121 5876 Acked 6875 Write 2 Done Done New true
20122 5877 Acked 6876 Write 8 Done Done New true
20123 5878 Acked 6877 Flush 0 Done Done New true
20124 5879 Acked 6878 Read 1 Done Done New true
20125 5880 Acked 6879 Read 1 Done Done New true
20126 5881 Acked 6880 Read 5 Done Done New true
20127 5882 Acked 6881 Read 2 Done Done New true
20128 5883 Acked 6882 Read 5 Done Done New true
20129 5884 Acked 6883 Flush 0 Done Done New true
20130 5885 Acked 6884 Flush 0 Done Done New true
20131 5886 Acked 6885 Write 10 Done Done New true
20132 5887 Acked 6886 Write 4 Done Done New true
20133 5888 Acked 6887 Read 5 Done Done New true
20134 5889 Acked 6888 Read 10 Done Done New true
20135 5890 Acked 6889 Write 9 Done Done New true
20136 5891 Acked 6890 Write 3 Done Done New true
20137 5892 Acked 6891 Write 5 Done Done New true
20138 5893 Acked 6892 Write 6 Done Done New true
20139 5894 Acked 6893 Write 5 Done Done New true
20140 5895 Acked 6894 Read 7 Done Done New true
20141 5896 Acked 6895 Read 9 Done Done New true
20142 5897 Acked 6896 Read 3 Done Done New true
20143 5898 Acked 6897 Write 5 Done Done New true
20144 5899 Acked 6898 Write 3 Done Done New true
20145 5900 Acked 6899 Read 2 Done Done New true
20146 5901 Acked 6900 Write 2 Done Done New true
20147 5902 Acked 6901 Flush 0 Done Done New true
20148 5903 Acked 6902 Read 7 Done Done New true
20149 5904 Acked 6903 Flush 0 Done Done New true
20150 5905 Acked 6904 Write 9 Done Done New true
20151 5906 Acked 6905 Read 7 Done Done New true
20152 5907 Acked 6906 Write 9 Done Done New true
20153 5908 Acked 6907 Read 10 Done Done New true
20154 5909 Acked 6908 Read 5 Done Done New true
20155 5910 Acked 6909 Read 9 Done Done New true
20156 5911 Acked 6910 Write 9 Done Done New true
20157 5912 Acked 6911 Write 3 Done Done New true
20158 5913 Acked 6912 Read 8 Done Done New true
20159 5914 Acked 6913 Read 9 Done Done New true
20160 5915 Acked 6914 Write 5 Done Done New true
20161 5916 Acked 6915 Read 4 Done Done New true
20162 5917 Acked 6916 Read 9 Done Done New true
20163 5918 Acked 6917 Read 7 Done Done New true
20164 5919 Acked 6918 Read 4 Done Done New true
20165 5920 Acked 6919 Read 1 Done Done New true
20166 5921 Acked 6920 Write 7 Done Done New true
20167 5922 Acked 6921 Read 3 Done Done New true
20168 5923 Acked 6922 Read 2 Done Done New true
20169 5924 Acked 6923 Write 4 Done Done New true
20170 5925 Acked 6924 Write 2 Done Done New true
20171 5926 Acked 6925 Read 10 Done Done New true
20172 5927 Acked 6926 Read 3 Done Done New true
20173 5928 Acked 6927 Write 3 Done Done New true
20174 5929 Acked 6928 Read 10 Done Done New true
20175 5930 Acked 6929 Flush 0 Done Done New true
20176 5931 Acked 6930 Read 8 Done Done New true
20177 5932 Acked 6931 Read 5 Done Done New true
20178 5933 Acked 6932 Read 8 Done Done New true
20179 5934 Acked 6933 Write 6 Done Done New true
20180 5935 Acked 6934 Write 6 Done Done New true
20181 5936 Acked 6935 Write 2 Done Done New true
20182 5937 Acked 6936 Read 2 Done Done New true
20183 5938 Acked 6937 Read 10 Done Done New true
20184 5939 Acked 6938 Read 1 Done Done New true
20185 5940 Acked 6939 Read 6 Done Done New true
20186 5941 Acked 6940 Write 9 Done Done New true
20187 5942 Acked 6941 Read 10 Done Done New true
20188 5943 Acked 6942 Write 7 Done Done New true
20189 5944 Acked 6943 Read 1 Done Done New true
20190 5945 Acked 6944 Write 9 Done Done New true
20191 5946 Acked 6945 Flush 0 Done Done New true
20192 5947 Acked 6946 Read 10 Done Done New true
20193 5948 Acked 6947 Flush 0 Done Done New true
20194 5949 Acked 6948 Read 7 Done Done New true
20195 5950 Acked 6949 Flush 0 Done Done New true
20196 5951 Acked 6950 Read 2 Done Done New true
20197 5952 Acked 6951 Read 2 Done Done New true
20198 5953 Acked 6952 Read 9 Done Done New true
20199 5954 Acked 6953 Write 2 Done Done New true
20200 5955 Acked 6954 Read 4 Done Done New true
20201 5956 Acked 6955 Write 4 Done Done New true
20202 5957 Acked 6956 Read 8 Done Done New true
20203 5958 Acked 6957 Write 1 Done Done New true
20204 5959 Acked 6958 Flush 0 Done Done New true
20205 5960 Acked 6959 Write 4 Done Done New true
20206 5961 Acked 6960 Write 5 Done Done New true
20207 5962 Acked 6961 Read 1 Done Done New true
20208 5963 Acked 6962 Write 10 Done Done New true
20209 5964 Acked 6963 Write 1 Done Done New true
20210 5965 Acked 6964 Flush 0 Done Done New true
20211 5966 Acked 6965 Flush 0 Done Done New true
20212 5967 Acked 6966 Read 7 Done Done New true
20213 5968 Acked 6967 Flush 0 Done Done New true
20214 5969 Acked 6968 Write 4 Done Done New true
20215 5970 Acked 6969 Write 3 Done Done New true
20216 5971 Acked 6970 Write 2 Done Done New true
20217 5972 Acked 6971 Write 7 Done Done New true
20218 5973 Acked 6972 Write 10 Done Done New true
20219 5974 Acked 6973 Flush 0 Done Done New true
20220 5975 Acked 6974 Read 4 Done Done New true
20221 5976 Acked 6975 Flush 0 Done Done New true
20222 5977 Acked 6976 Read 4 Done Done New true
20223 5978 Acked 6977 Write 8 Done Done New true
20224 5979 Acked 6978 Write 6 Done Done New true
20225 5980 Acked 6979 Write 8 Done Done New true
20226 5981 Acked 6980 Write 10 Done Done New true
20227 5982 Acked 6981 Read 7 Done Done New true
20228 5983 Acked 6982 Write 8 Done Done New true
20229 5984 Acked 6983 Write 7 Done Done New true
20230 5985 Acked 6984 Read 3 Done Done New true
20231 5986 Acked 6985 Read 4 Done Done New true
20232 5987 Acked 6986 Write 4 Done Done New true
20233 5988 Acked 6987 Read 9 Done Done New true
20234 5989 Acked 6988 Read 10 Done Done New true
20235 5990 Acked 6989 Write 8 Done Done New true
20236 5991 Acked 6990 Write 8 Done Done New true
20237 5992 Acked 6991 Read 9 Done Done New true
20238 5993 Acked 6992 Read 7 Done Done New true
20239 5994 Acked 6993 Write 4 Done Done New true
20240 5995 Acked 6994 Read 7 Done Done New true
20241 5996 Acked 6995 Write 6 Done Done New true
20242 5997 Acked 6996 Write 4 Done Done New true
20243 5998 Acked 6997 Write 4 Done Done New true
20244 5999 Acked 6998 Write 8 Done Done New true
20245 6000 Acked 6999 Read 2 Done Done New true
20246 6001 Acked 7000 Write 7 Done Done New true
20247 6002 Acked 7001 Read 8 Done Done New true
20248 6003 Acked 7002 Write 1 Done Done New true
20249 6004 Acked 7003 Write 3 Done Done New true
20250 6005 Acked 7004 Read 2 Done Done New true
20251 6006 Acked 7005 Read 6 Done Done New true
20252 6007 Acked 7006 Write 9 Done Done New true
20253 6008 Acked 7007 Write 7 Done Done New true
20254 6009 Acked 7008 Read 7 Done Done New true
20255 6010 Acked 7009 Write 9 Done Done New true
20256 6011 Acked 7010 Write 2 Done Done New true
20257 6012 Acked 7011 Read 6 Done Done New true
20258 6013 Acked 7012 Read 3 Done Done New true
20259 6014 Acked 7013 Write 8 Done Done New true
20260 6015 Acked 7014 Read 2 Done Done New true
20261 6016 Acked 7015 Read 10 Done Done New true
20262 6017 Acked 7016 Write 10 Done Done New true
20263 6018 Acked 7017 Flush 0 Done Done New true
20264 6019 Acked 7018 Flush 0 Done Done New true
20265 6020 Acked 7019 Write 1 Done Done New true
20266 6021 Acked 7020 Write 10 Done Done New true
20267 6022 Acked 7021 Read 3 Done Done New true
20268 6023 Acked 7022 Read 6 Done Done New true
20269 6024 Acked 7023 Write 3 Done Done New true
20270 6025 Acked 7024 Write 10 Done Done New true
20271 6026 Acked 7025 Read 3 Done Done New true
20272 6027 Acked 7026 Read 3 Done Done New true
20273 6028 Acked 7027 Write 1 Done Done New true
20274 6029 Acked 7028 Flush 0 Done Done New true
20275 STATES DS:0 DS:1 DS:2 TOTAL
20276 New 0 0 301 301
20277 Sent 0 0 0 0
20278 Done 301 301 0 602
20279 Skipped 0 0 0 0
20280 Error 0 0 0 0
20281 Last Flush: 7028 7028 6727
20282 Downstairs last five completed: 6727 6726 6725 6724 6723
20283 Upstairs last five completed: 6029 6028 6027 6026 6025
20284 CLIENT: Up:0 ds:301 act:2
202852024-02-15T03:01:30.112ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
202862024-02-15T03:01:30.112ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
202872024-02-15T03:01:30.112ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
202882024-02-15T03:01:30.112ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
202892024-02-15T03:01:30.112ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
202902024-02-15T03:01:30.112ZINFOcrucible: [2] client re-new 301 jobs since flush 6727 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
202912024-02-15T03:01:30.113ZINFOcrucible: client task is exiting = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
20292 [8830] got start action
20293 [8830] I am going to restart
20294 Starting downstairs at port 8830
20295 Make output file at "/tmp/dsc/downstairs-8830.txt"
20296 [8830][2] reports Starting
202972024-02-15T03:01:31.114ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
202982024-02-15T03:01:31.114ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
202992024-02-15T03:01:31.114ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
203002024-02-15T03:01:31.114ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
203012024-02-15T03:01:31.114ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
203022024-02-15T03:01:31.114ZINFOcrucible: [2] client re-new 301 jobs since flush 6727 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
203032024-02-15T03:01:31.114ZINFOcrucible: client task is exiting = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
20304 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1187)
20305 [8830][2] reports Running
203062024-02-15T03:01:32.115ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
203072024-02-15T03:01:32.115ZINFOcrucible: ds_connection connected = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
203082024-02-15T03:01:32.117ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
203092024-02-15T03:01:32.117ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
203102024-02-15T03:01:32.117ZINFOcrucible: send last flush ID to this DS: 6727 = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
203112024-02-15T03:01:32.117ZINFOcrucible: Replied this last flush ID: 6727 = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
203122024-02-15T03:01:32.117ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
203132024-02-15T03:01:32.117ZINFOcrucible: [2] Transition from Offline to Replay = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
203142024-02-15T03:01:32.117ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
203152024-02-15T03:01:32.117ZINFOcrucible: [2] Transition from Replay to Active = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
20316 ----------------------------------------------------------------
20317 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
20318 Downstairs last five completed: 7028 7027 7026 7025 7024
20319 Upstairs last five completed: 6029 6028 6027 6026 6025
20320 CLIENT: Up:0 ds:0 act:3
20321 Replay: All jobs finished, all DS active.
20322 Main task has work to do, go find it
20323 got dsc Stop(2)
20324 stop 2
20325 [8830] Got stop action so:false kr:false
20326 Feb 15 03:01:33.437 INFO request completed, latency_us: 359, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 6eff0ed8-d8f9-4833-bc17-2c64369910e5, remote_addr: 127.0.0.1:54744, local_addr: 127.0.0.1:9998
203272024-02-15T03:01:33.438ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
20328 Feb 15 03:01:33.438 INFO request completed, latency_us: 136, response_code: 200, uri: /state/cid/2, method{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2024-02-15T03:01:33.438291429Z","hostname":"ip-10-150-1-67.us-west-2.compute.internal","pid":1167,"":"io task","client":"2","":"downstairs","session_id":"4f5a63e5-aa00-4890-9f6b-96c2e069fdb2"}
20329 : GET, req_id: 91547723-bce4-4109-b66f-fedea81994f8{"msg":"downstairs task for 2 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-02-15T03:01:33.438330674Z","hostname":"ip-10-150-1-67.us-west-2.compute.internal","pid":1167,"session_id":"4f5a63e5-aa00-4890-9f6b-96c2e069fdb2"}
20330 , {"msg":"Gone missing, transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-02-15T03:01:33.438377058Z","hostname":"ip-10-150-1-67.us-west-2.compute.internal","pid":1167,"client":"2","":"downstairs","session_id":"4f5a63e5-aa00-4890-9f6b-96c2e069fdb2"}
20331 remote_addr: 127.0.0.1:54744, local_addr:{"msg":"[2] client re-new 0 jobs since flush 7028","v":0,"name":"crucible","level":30,"time":"2024-02-15T03:01:33.438470236Z","hostname":"ip-10-150-1-67.us-west-2.compute.internal","pid":1167,"":"downstairs","":"downstairs","session_id":"4f5a63e5-aa00-4890-9f6b-96c2e069fdb2"}
20332 127.0.0.1:9998
203332024-02-15T03:01:33.438ZINFOcrucible: client task is exiting = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
20334 [8830] Exited with: ExitStatus(unix_wait_status(9))
20335 [8830][2] reports Exit
203362024-02-15T03:01:34.440ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
203372024-02-15T03:01:34.440ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
203382024-02-15T03:01:34.440ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
203392024-02-15T03:01:34.440ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
203402024-02-15T03:01:34.440ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
203412024-02-15T03:01:34.440ZINFOcrucible: [2] client re-new 0 jobs since flush 7028 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
203422024-02-15T03:01:34.440ZINFOcrucible: client task is exiting = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
203432024-02-15T03:01:35.442ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
203442024-02-15T03:01:35.442ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
203452024-02-15T03:01:35.442ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
203462024-02-15T03:01:35.442ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
203472024-02-15T03:01:35.442ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
203482024-02-15T03:01:35.442ZINFOcrucible: [2] client re-new 0 jobs since flush 7028 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
203492024-02-15T03:01:35.442ZINFOcrucible: client task is exiting = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
203502024-02-15T03:01:36.444ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
203512024-02-15T03:01:36.444ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
203522024-02-15T03:01:36.444ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
203532024-02-15T03:01:36.444ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
203542024-02-15T03:01:36.444ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
203552024-02-15T03:01:36.444ZINFOcrucible: [2] client re-new 0 jobs since flush 7028 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
203562024-02-15T03:01:36.444ZINFOcrucible: client task is exiting = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
20357 Feb 15 03:01:37.441 INFO request completed, latency_us: 208, response_code: 200, uri: /state/cid/2, method: GET, req_id: 0cd31519-bee2-408c-aecb-ba4284bd057c, remote_addr: 127.0.0.1:54744, local_addr: 127.0.0.1:9998
20358 001/300 Write block 3672 len 12288 data: 5 7 7
20359 002/300 Flush
20360 003/300 Flush
20361 004/300 Read block 4268 len 36864
203622024-02-15T03:01:37.445ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
203632024-02-15T03:01:37.445ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
203642024-02-15T03:01:37.445ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
203652024-02-15T03:01:37.445ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
203662024-02-15T03:01:37.445ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
203672024-02-15T03:01:37.445ZINFOcrucible: [2] client re-new 4 jobs since flush 7028 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
203682024-02-15T03:01:37.445ZINFOcrucible: client task is exiting = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
20369 005/300 Read block 1218 len 8192
20370 006/300 Flush
20371 007/300 Write block 95 len 20480 data: 4 3 4 4 3
20372 008/300 Read block 1877 len 24576
20373 009/300 Write block 605 len 32768 data: 6 6 5 4 3 4 4 6
20374 010/300 Read block 3565 len 4096
20375 011/300 Write block 3240 len 32768 data: 8 9 6 8 9 8 8 6
20376 012/300 Write block 4442 len 12288 data: 3 4 4
20377 013/300 Write block 2906 len 16384 data: 3 3 4 4
20378 014/300 Flush
20379 015/300 Read block 3439 len 36864
20380 016/300 Write block 4092 len 36864 data: 6 6 6 5 4 4 4 4 6
20381 017/300 Write block 1336 len 28672 data: 4 4 3 3 3 3 3
20382 018/300 Write block 188 len 24576 data: 5 5 4 5 4 5
20383 019/300 Flush
20384 020/300 Read block 3875 len 36864
20385 021/300 Read block 3018 len 12288
20386 022/300 Read block 1793 len 28672
20387 023/300 Read block 1418 len 4096
20388 024/300 Read block 2613 len 40960
20389 025/300 Read block 4046 len 12288
20390 026/300 Read block 3997 len 20480
20391 027/300 Read block 2559 len 40960
20392 028/300 Read block 80 len 40960
20393 029/300 Read block 4408 len 8192
20394 030/300 Read block 4705 len 16384
20395 031/300 Read block 4750 len 8192
20396 032/300 Write block 4214 len 24576 data: 6 6 5 5 5 4
20397 033/300 Write block 1353 len 12288 data: 3 3 3
20398 034/300 Write block 4205 len 40960 data: 7 8 7 5 5 6 5 5 6 7
20399 035/300 Read block 413 len 16384
20400 036/300 Write block 3060 len 20480 data: 5 5 4 3 3
20401 037/300 Read block 4084 len 16384
20402 038/300 Write block 3142 len 8192 data: 7 5
20403 039/300 Read block 2081 len 16384
20404 040/300 Write block 352 len 40960 data: 6 8 8 7 5 6 4 4 3 5
20405 041/300 Flush
20406 042/300 Write block 1964 len 4096 data: 4
20407 043/300 Write block 4275 len 32768 data: 3 3 4 3 3 6 6 6
20408 044/300 Read block 3798 len 32768
20409 045/300 Read block 3328 len 28672
20410 046/300 Read block 3956 len 20480
20411 047/300 Read block 1256 len 4096
20412 048/300 Read block 2094 len 20480
20413 049/300 Flush
20414 050/300 Read block 2068 len 16384
20415 051/300 Read block 2116 len 4096
20416 052/300 Write block 326 len 8192 data: 4 4
20417 053/300 Read block 3280 len 28672
20418 054/300 Write block 1519 len 28672 data: 4 5 5 4 5 6 6
20419 055/300 Write block 4949 len 4096 data: 8
20420 056/300 Write block 4591 len 4096 data: 5
20421 057/300 Read block 965 len 40960
20422 058/300 Write block 496 len 40960 data: 5 4 5 4 3 4 4 4 4 4
20423 059/300 Read block 2111 len 40960
20424 060/300 Write block 2197 len 20480 data: 4 4 4 3 2
20425 061/300 Read block 1017 len 40960
20426 062/300 Read block 138 len 32768
20427 063/300 Write block 3161 len 20480 data: 11 10 9 7 8
20428 064/300 Read block 2363 len 4096
20429 065/300 Read block 547 len 40960
20430 066/300 Read block 304 len 8192
20431 067/300 Write block 1199 len 4096 data: 7
20432 068/300 Read block 1774 len 32768
20433 069/300 Read block 3614 len 4096
20434 070/300 Read block 876 len 28672
20435 071/300 Write block 2927 len 4096 data: 4
20436 072/300 Write block 2040 len 40960 data: 3 3 4 3 2 2 2 3 3 3
20437 073/300 Read block 4362 len 36864
20438 074/300 Read block 2634 len 20480
20439 075/300 Flush
20440 076/300 Write block 2125 len 20480 data: 6 7 8 7 5
20441 077/300 Write block 1048 len 36864 data: 2 2 2 2 3 3 3 3 3
20442 078/300 Write block 3412 len 24576 data: 6 7 6 6 4 5
20443 079/300 Read block 2747 len 20480
20444 080/300 Write block 859 len 28672 data: 6 6 6 6 7 6 7
20445 081/300 Read block 2295 len 36864
20446 082/300 Read block 1147 len 24576
20447 083/300 Write block 4019 len 12288 data: 8 9 7
20448 084/300 Write block 2341 len 8192 data: 3 3
20449 085/300 Read block 1378 len 36864
20450 086/300 Write block 4886 len 40960 data: 5 5 5 6 6 6 6 7 6 6
20451 087/300 Read block 4356 len 4096
20452 088/300 Read block 2842 len 4096
20453 089/300 Read block 1176 len 40960
20454 090/300 Read block 2323 len 16384
20455 091/300 Read block 1058 len 8192
20456 092/300 Write block 977 len 4096 data: 4
20457 093/300 Write block 1668 len 28672 data: 3 3 2 2 2 3 4
20458 094/300 Read block 4065 len 32768
20459 095/300 Read block 4742 len 12288
20460 096/300 Read block 1172 len 4096
20461 097/300 Write block 3288 len 36864 data: 5 6 6 5 4 3 3 2 4
20462 098/300 Write block 2756 len 8192 data: 3 4
20463 099/300 Write block 1471 len 24576 data: 5 5 4 3 3 2
20464 100/300 Write block 2497 len 4096 data: 4
20465 101/300 Flush
20466 102/300 Read block 2196 len 24576
20467 103/300 Write block 3141 len 20480 data: 6 8 6 5 5
20468 104/300 Write block 840 len 20480 data: 4 5 5 5 6
20469 105/300 Read block 4022 len 20480
20470 106/300 Write block 4337 len 40960 data: 6 5 3 3 4 4 6 7 6 8
20471 107/300 Write block 4206 len 8192 data: 9 8
20472 108/300 Write block 4287 len 16384 data: 4 4 4 4
20473 109/300 Read block 4976 len 4096
20474 110/300 Read block 1686 len 24576
20475 111/300 Read block 4164 len 20480
20476 112/300 Write block 2107 len 8192 data: 4 4
20477 113/300 Write block 4354 len 20480 data: 4 4 3 4 5
20478 114/300 Write block 1689 len 12288 data: 4 4 5
20479 115/300 Write block 3535 len 20480 data: 6 5 5 4 4
20480 116/300 Read block 4102 len 16384
20481 117/300 Flush
20482 118/300 Write block 776 len 16384 data: 3 4 4 3
20483 119/300 Read block 1264 len 16384
20484 120/300 Write block 3667 len 8192 data: 10 11
20485 121/300 Flush
20486 122/300 Read block 4031 len 36864
20487 123/300 Read block 280 len 12288
20488 124/300 Read block 2792 len 12288
20489 125/300 Read block 2761 len 32768
20490 126/300 Read block 4868 len 24576
20491 127/300 Write block 2679 len 40960 data: 4 4 2 2 2 2 3 3 3 2
20492 128/300 Flush
20493 129/300 Write block 2070 len 8192 data: 6 5
20494 130/300 Write block 4087 len 16384 data: 2 2 4 4
20495 131/300 Write block 613 len 36864 data: 6 6 8 9 11 9 8 7 3
20496 132/300 Write block 2500 len 40960 data: 3 5 6 7 7 7 7 7 6 6
20497 133/300 Write block 4389 len 20480 data: 4 4 6 6 6
20498 134/300 Write block 3612 len 40960 data: 6 7 9 8 8 9 10 9 9 7
20499 135/300 Read block 2355 len 36864
20500 136/300 Write block 279 len 12288 data: 2 3 3
20501 137/300 Read block 3463 len 12288
20502 138/300 Write block 2827 len 8192 data: 4 5
20503 139/300 Write block 1779 len 36864 data: 4 4 4 5 6 6 7 7 6
20504 140/300 Read block 3644 len 12288
20505 141/300 Read block 1097 len 32768
20506 142/300 Write block 3047 len 16384 data: 4 6 7 7
20507 143/300 Read block 2857 len 40960
20508 144/300 Read block 4893 len 24576
20509 145/300 Write block 1910 len 8192 data: 5 5
20510 146/300 Flush
20511 147/300 Read block 1957 len 36864
20512 148/300 Read block 2814 len 12288
20513 149/300 Read block 2195 len 40960
20514 150/300 Write block 1154 len 20480 data: 5 3 4 4 5
20515 151/300 Write block 1936 len 40960 data: 2 2 3 4 4 3 3 4 3 3
20516 152/300 Read block 2396 len 12288
20517 153/300 Write block 3908 len 8192 data: 8 8
20518 154/300 Read block 2070 len 32768
20519 155/300 Write block 3196 len 32768 data: 5 5 4 5 7 7 6 6
20520 156/300 Read block 1197 len 28672
20521 157/300 Read block 4509 len 24576
20522 158/300 Write block 4181 len 16384 data: 4 3 3 3
20523 159/300 Read block 2629 len 28672
20524 160/300 Write block 2663 len 4096 data: 6
20525 161/300 Read block 996 len 20480
20526 162/300 Read block 4538 len 40960
20527 163/300 Read block 4186 len 36864
20528 164/300 Read block 3922 len 12288
20529 165/300 Read block 3867 len 40960
20530 166/300 Write block 1576 len 16384 data: 5 6 5 5
20531 167/300 Read block 4836 len 4096
20532 168/300 Write block 1246 len 20480 data: 6 6 7 6 7
20533 169/300 Read block 3339 len 20480
20534 170/300 Write block 1986 len 8192 data: 2 2
20535 171/300 Write block 2039 len 8192 data: 2 4
20536 172/300 Write block 1900 len 36864 data: 3 4 4 5 5 5 5 5 5
20537 173/300 Read block 4602 len 20480
20538 174/300 Write block 634 len 20480 data: 6 6 6 6 3
20539 175/300 Read block 843 len 8192
20540 176/300 Read block 4385 len 16384
20541 177/300 Write block 337 len 28672 data: 4 4 4 2 2 3 3
20542 178/300 Read block 990 len 20480
20543 179/300 Read block 1912 len 8192
20544 180/300 Flush
20545 181/300 Read block 4775 len 12288
20546 182/300 Read block 3033 len 16384
20547 183/300 Read block 2867 len 16384
20548 184/300 Read block 4183 len 28672
20549 185/300 Read block 1141 len 32768
20550 186/300 Read block 4325 len 40960
20551 187/300 Read block 1917 len 32768
20552 188/300 Read block 1552 len 40960
20553 189/300 Read block 4926 len 40960
20554 190/300 Read block 2939 len 20480
20555 191/300 Read block 4140 len 4096
20556 192/300 Flush
20557 193/300 Read block 4620 len 8192
20558 194/300 Write block 1953 len 20480 data: 6 5 5 5 5
20559 195/300 Read block 371 len 4096
20560 196/300 Write block 2010 len 32768 data: 5 5 4 3 2 5 6 7
20561 197/300 Read block 1152 len 12288
20562 198/300 Read block 1602 len 20480
20563 199/300 Read block 1323 len 8192
20564 200/300 Write block 1214 len 24576 data: 3 3 3 3 4 6
20565 201/300 Read block 1643 len 4096
20566 202/300 Flush
20567 203/300 Write block 2574 len 4096 data: 4
20568 204/300 Read block 4929 len 24576
20569 205/300 Read block 3665 len 24576
20570 206/300 Write block 3990 len 12288 data: 3 3 3
20571 207/300 Read block 2122 len 32768
20572 208/300 Write block 2285 len 8192 data: 3 2
20573 209/300 Flush
20574 210/300 Read block 4271 len 20480
20575 211/300 Flush
20576 212/300 Read block 3005 len 28672
20577 213/300 Read block 1435 len 28672
20578 214/300 Write block 4730 len 12288 data: 2 3 3
20579 215/300 Write block 2861 len 8192 data: 8 7
20580 216/300 Read block 4907 len 12288
20581 217/300 Write block 2154 len 16384 data: 3 3 4 4
20582 218/300 Read block 4860 len 36864
20583 219/300 Write block 4505 len 36864 data: 5 5 5 5 5 3 3 4 4
20584 220/300 Write block 341 len 40960 data: 3 4 4 3 3 3 3 4 3 6
20585 221/300 Write block 37 len 24576 data: 3 3 3 3 3 3
20586 222/300 Read block 649 len 36864
20587 223/300 Write block 1201 len 32768 data: 6 5 5 5 5 4 4 4
20588 224/300 Read block 1781 len 24576
20589 225/300 Write block 663 len 20480 data: 3 3 2 2 3
20590 226/300 Write block 785 len 24576 data: 4 4 5 5 5 5
20591 227/300 Write block 353 len 40960 data: 9 9 8 6 7 5 5 4 6 6
20592 228/300 Write block 212 len 32768 data: 5 4 3 2 2 2 3 6
20593 229/300 Write block 4822 len 40960 data: 5 5 5 8 8 7 5 4 5 4
20594 230/300 Read block 4781 len 24576
20595 231/300 Read block 2804 len 4096
20596 232/300 Read block 1465 len 16384
20597 233/300 Write block 1408 len 12288 data: 6 5 4
20598 234/300 Read block 2284 len 40960
20599 235/300 Read block 2405 len 16384
20600 236/300 Write block 3994 len 20480 data: 4 5 5 4 3
20601 237/300 Write block 536 len 28672 data: 6 6 4 4 3 3 3
20602 238/300 Write block 4587 len 24576 data: 3 3 4 4 6 6
20603 239/300 Write block 4543 len 16384 data: 3 3 3 5
20604 240/300 Read block 3076 len 8192
20605 241/300 Read block 2552 len 36864
20606 242/300 Flush
20607 243/300 Read block 3269 len 24576
20608 244/300 Write block 739 len 40960 data: 7 5 5 4 4 5 5 5 5 5
20609 245/300 Read block 416 len 24576
20610 246/300 Read block 3263 len 16384
20611 247/300 Read block 2276 len 4096
20612 248/300 Write block 4521 len 40960 data: 2 2 2 3 2 2 3 3 4 4
20613 249/300 Flush
20614 250/300 Write block 395 len 20480 data: 3 2 2 2 2
20615 251/300 Read block 2421 len 24576
20616 252/300 Read block 1286 len 8192
20617 253/300 Write block 3043 len 24576 data: 3 3 3 4 5 7
20618 254/300 Write block 2066 len 20480 data: 6 6 6 6 7
20619 255/300 Write block 3067 len 24576 data: 3 3 3 3 3 3
20620 256/300 Read block 1894 len 8192
20621 257/300 Write block 975 len 36864 data: 4 3 5 4 6 6 5 5 5
20622 258/300 Read block 1956 len 40960
20623 259/300 Write block 1264 len 28672 data: 4 4 4 4 4 4 3
20624 260/300 Write block 3541 len 32768 data: 5 5 6 5 5 5 5 3
20625 261/300 Write block 2800 len 28672 data: 3 3 3 3 3 3 2
20626 262/300 Read block 1000 len 4096
20627 263/300 Write block 4117 len 24576 data: 8 7 4 4 4 4
20628 264/300 Read block 4353 len 24576
20629 265/300 Write block 4848 len 20480 data: 6 5 3 3 3
20630 266/300 Flush
20631 267/300 Flush
20632 268/300 Write block 269 len 4096 data: 4
20633 269/300 Write block 927 len 12288 data: 3 2 2
20634 270/300 Write block 4814 len 16384 data: 3 4 5 5
20635 271/300 Read block 1204 len 28672
20636 272/300 Read block 4585 len 40960
20637 273/300 Write block 3865 len 32768 data: 8 7 7 9 7 8 7 7
20638 274/300 Read block 4280 len 8192
20639 275/300 Write block 2812 len 36864 data: 3 3 4 4 4 4 5 5 6
20640 276/300 Write block 3533 len 12288 data: 8 8 7
20641 277/300 Read block 388 len 16384
20642 278/300 Read block 1319 len 36864
20643 279/300 Read block 3389 len 32768
20644 280/300 Write block 1087 len 36864 data: 4 4 4 5 4 4 4 4 4
20645 281/300 Read block 3202 len 28672
20646 282/300 Read block 91 len 20480
20647 283/300 Write block 1774 len 16384 data: 4 3 2 3
20648 284/300 Read block 3999 len 24576
20649 285/300 Read block 1282 len 8192
20650 286/300 Read block 3013 len 20480
20651 287/300 Flush
20652 288/300 Write block 596 len 4096 data: 4
20653 289/300 Read block 1907 len 16384
20654 290/300 Read block 2229 len 28672
20655 291/300 Flush
20656 292/300 Read block 1494 len 28672
20657 293/300 Read block 3489 len 36864
20658 294/300 Read block 3828 len 28672
20659 295/300 Read block 2442 len 28672
20660 296/300 Read block 1880 len 8192
20661 297/300 Flush
20662 298/300 Read block 3184 len 20480
20663 299/300 Write block 1603 len 24576 data: 3 5 4 4 3 3
20664 300/300 Flush
20665 Main task has work to do, go find it
20666 got dsc Start(2)
20667 start 2
20668 Feb 15 03:01:37.778 INFO request completed, latency_us: 220, response_code: 204, uri: /start/cid/2, method: POST, req_id: 8501916b-f403-446a-aae7-54a7431ca7a4, remote_addr: 127.0.0.1:54744, local_addr: 127.0.0.1:9998
20669 Replay: started 2, returned:Ok(())
20670 ----------------------------------------------------------------
20671 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
20672 States: Active Active Offline
20673 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
20674 6030 Acked 7029 Write 3 Done Done New true
20675 6031 Acked 7030 Flush 0 Done Done New true
20676 6032 Acked 7031 Flush 0 Done Done New true
20677 6033 Acked 7032 Read 9 Done Done New true
20678 6034 Acked 7033 Read 2 Done Done New false
20679 6035 Acked 7034 Flush 0 Done Done New false
20680 6036 Acked 7035 Write 5 Done Done New false
20681 6037 Acked 7036 Read 6 Done Done New false
20682 6038 Acked 7037 Write 8 Done Done New false
20683 6039 Acked 7038 Read 1 Done Done New false
20684 6040 Acked 7039 Write 8 Done Done New false
20685 6041 Acked 7040 Write 3 Done Done New false
20686 6042 Acked 7041 Write 4 Done Done New false
20687 6043 Acked 7042 Flush 0 Done Done New false
20688 6044 Acked 7043 Read 9 Done Done New false
20689 6045 Acked 7044 Write 9 Done Done New false
20690 6046 Acked 7045 Write 7 Done Done New false
20691 6047 Acked 7046 Write 6 Done Done New false
20692 6048 Acked 7047 Flush 0 Done Done New false
20693 6049 Acked 7048 Read 9 Done Done New false
20694 6050 Acked 7049 Read 3 Done Done New false
20695 6051 Acked 7050 Read 7 Done Done New false
20696 6052 Acked 7051 Read 1 Done Done New false
20697 6053 Acked 7052 Read 10 Done Done New false
20698 6054 Acked 7053 Read 3 Done Done New false
20699 6055 Acked 7054 Read 5 Done Done New false
20700 6056 Acked 7055 Read 10 Done Done New false
20701 6057 Acked 7056 Read 10 Done Done New false
20702 6058 Acked 7057 Read 2 Done Done New false
20703 6059 Acked 7058 Read 4 Done Done New false
20704 6060 Acked 7059 Read 2 Done Done New false
20705 6061 Acked 7060 Write 6 Done Done New false
20706 6062 Acked 7061 Write 3 Done Done New false
20707 6063 Acked 7062 Write 10 Done Done New false
20708 6064 Acked 7063 Read 4 Done Done New false
20709 6065 Acked 7064 Write 5 Done Done New false
20710 6066 Acked 7065 Read 4 Done Done New false
20711 6067 Acked 7066 Write 2 Done Done New false
20712 6068 Acked 7067 Read 4 Done Done New false
20713 6069 Acked 7068 Write 10 Done Done New false
20714 6070 Acked 7069 Flush 0 Done Done New false
20715 6071 Acked 7070 Write 1 Done Done New false
20716 6072 Acked 7071 Write 8 Done Done New false
20717 6073 Acked 7072 Read 8 Done Done New false
20718 6074 Acked 7073 Read 7 Done Done New false
20719 6075 Acked 7074 Read 5 Done Done New false
20720 6076 Acked 7075 Read 1 Done Done New false
20721 6077 Acked 7076 Read 5 Done Done New false
20722 6078 Acked 7077 Flush 0 Done Done New false
20723 6079 Acked 7078 Read 4 Done Done New false
20724 6080 Acked 7079 Read 1 Done Done New false
20725 6081 Acked 7080 Write 2 Done Done New false
20726 6082 Acked 7081 Read 7 Done Done New false
20727 6083 Acked 7082 Write 7 Done Done New false
20728 6084 Acked 7083 Write 1 Done Done New false
20729 6085 Acked 7084 Write 1 Done Done New false
20730 6086 Acked 7085 Read 10 Done Done New false
20731 6087 Acked 7086 Write 10 Done Done New false
20732 6088 Acked 7087 Read 10 Done Done New false
20733 6089 Acked 7088 Write 5 Done Done New false
20734 6090 Acked 7089 Read 10 Done Done New false
20735 6091 Acked 7090 Read 8 Done Done New false
20736 6092 Acked 7091 Write 5 Done Done New false
20737 6093 Acked 7092 Read 1 Done Done New false
20738 6094 Acked 7093 Read 10 Done Done New false
20739 6095 Acked 7094 Read 2 Done Done New false
20740 6096 Acked 7095 Write 1 Done Done New false
20741 6097 Acked 7096 Read 8 Done Done New false
20742 6098 Acked 7097 Read 1 Done Done New false
20743 6099 Acked 7098 Read 7 Done Done New false
20744 6100 Acked 7099 Write 1 Done Done New false
20745 6101 Acked 7100 Write 10 Done Done New false
20746 6102 Acked 7101 Read 9 Done Done New false
20747 6103 Acked 7102 Read 5 Done Done New false
20748 6104 Acked 7103 Flush 0 Done Done New false
20749 6105 Acked 7104 Write 5 Done Done New false
20750 6106 Acked 7105 Write 9 Done Done New false
20751 6107 Acked 7106 Write 6 Done Done New false
20752 6108 Acked 7107 Read 5 Done Done New false
20753 6109 Acked 7108 Write 7 Done Done New false
20754 6110 Acked 7109 Read 9 Done Done New false
20755 6111 Acked 7110 Read 6 Done Done New false
20756 6112 Acked 7111 Write 3 Done Done New false
20757 6113 Acked 7112 Write 2 Done Done New false
20758 6114 Acked 7113 Read 9 Done Done New false
20759 6115 Acked 7114 Write 10 Done Done New false
20760 6116 Acked 7115 Read 1 Done Done New false
20761 6117 Acked 7116 Read 1 Done Done New false
20762 6118 Acked 7117 Read 10 Done Done New false
20763 6119 Acked 7118 Read 4 Done Done New false
20764 6120 Acked 7119 Read 2 Done Done New false
20765 6121 Acked 7120 Write 1 Done Done New false
20766 6122 Acked 7121 Write 7 Done Done New false
20767 6123 Acked 7122 Read 8 Done Done New false
20768 6124 Acked 7123 Read 3 Done Done New false
20769 6125 Acked 7124 Read 1 Done Done New false
20770 6126 Acked 7125 Write 9 Done Done New false
20771 6127 Acked 7126 Write 2 Done Done New false
20772 6128 Acked 7127 Write 6 Done Done New false
20773 6129 Acked 7128 Write 1 Done Done New false
20774 6130 Acked 7129 Flush 0 Done Done New false
20775 6131 Acked 7130 Read 6 Done Done New false
20776 6132 Acked 7131 Write 5 Done Done New false
20777 6133 Acked 7132 Write 5 Done Done New false
20778 6134 Acked 7133 Read 5 Done Done New false
20779 6135 Acked 7134 Write 10 Done Done New false
20780 6136 Acked 7135 Write 2 Done Done New false
20781 6137 Acked 7136 Write 4 Done Done New false
20782 6138 Acked 7137 Read 1 Done Done New false
20783 6139 Acked 7138 Flush 0 Done Done New false
20784 6140 Acked 7139 Read 6 Done Done New false
20785 6141 Acked 7140 Read 5 Done Done New false
20786 6142 Acked 7141 Write 2 Done Done New false
20787 6143 Acked 7142 Write 5 Done Done New false
20788 6144 Acked 7143 Write 3 Done Done New false
20789 6145 Acked 7144 Write 5 Done Done New false
20790 6146 Acked 7145 Read 4 Done Done New false
20791 6147 Acked 7146 Flush 0 Done Done New false
20792 6148 Acked 7147 Write 4 Done Done New false
20793 6149 Acked 7148 Read 4 Done Done New false
20794 6150 Acked 7149 Write 2 Done Done New false
20795 6151 Acked 7150 Flush 0 Done Done New false
20796 6152 Acked 7151 Read 9 Done Done New false
20797 6153 Acked 7152 Read 3 Done Done New false
20798 6154 Acked 7153 Read 3 Done Done New false
20799 6155 Acked 7154 Read 8 Done Done New false
20800 6156 Acked 7155 Read 6 Done Done New false
20801 6157 Acked 7156 Write 10 Done Done New false
20802 6158 Acked 7157 Flush 0 Done Done New false
20803 6159 Acked 7158 Write 2 Done Done New false
20804 6160 Acked 7159 Write 4 Done Done New false
20805 6161 Acked 7160 Write 9 Done Done New false
20806 6162 Acked 7161 Write 10 Done Done New false
20807 6163 Acked 7162 Write 5 Done Done New false
20808 6164 Acked 7163 Write 10 Done Done New false
20809 6165 Acked 7164 Read 9 Done Done New false
20810 6166 Acked 7165 Write 3 Done Done New false
20811 6167 Acked 7166 Read 3 Done Done New false
20812 6168 Acked 7167 Write 2 Done Done New false
20813 6169 Acked 7168 Write 9 Done Done New false
20814 6170 Acked 7169 Read 3 Done Done New false
20815 6171 Acked 7170 Read 8 Done Done New false
20816 6172 Acked 7171 Write 4 Done Done New false
20817 6173 Acked 7172 Read 10 Done Done New false
20818 6174 Acked 7173 Read 6 Done Done New false
20819 6175 Acked 7174 Write 2 Done Done New false
20820 6176 Acked 7175 Flush 0 Done Done New false
20821 6177 Acked 7176 Read 9 Done Done New false
20822 6178 Acked 7177 Read 3 Done Done New false
20823 6179 Acked 7178 Read 10 Done Done New false
20824 6180 Acked 7179 Write 5 Done Done New false
20825 6181 Acked 7180 Write 10 Done Done New false
20826 6182 Acked 7181 Read 3 Done Done New false
20827 6183 Acked 7182 Write 2 Done Done New false
20828 6184 Acked 7183 Read 8 Done Done New false
20829 6185 Acked 7184 Write 8 Done Done New false
20830 6186 Acked 7185 Read 7 Done Done New false
20831 6187 Acked 7186 Read 6 Done Done New false
20832 6188 Acked 7187 Write 4 Done Done New false
20833 6189 Acked 7188 Read 7 Done Done New false
20834 6190 Acked 7189 Write 1 Done Done New false
20835 6191 Acked 7190 Read 5 Done Done New false
20836 6192 Acked 7191 Read 10 Done Done New false
20837 6193 Acked 7192 Read 9 Done Done New false
20838 6194 Acked 7193 Read 3 Done Done New false
20839 6195 Acked 7194 Read 10 Done Done New false
20840 6196 Acked 7195 Write 4 Done Done New false
20841 6197 Acked 7196 Read 1 Done Done New false
20842 6198 Acked 7197 Write 5 Done Done New false
20843 6199 Acked 7198 Read 5 Done Done New false
20844 6200 Acked 7199 Write 2 Done Done New false
20845 6201 Acked 7200 Write 2 Done Done New false
20846 6202 Acked 7201 Write 9 Done Done New false
20847 6203 Acked 7202 Read 5 Done Done New false
20848 6204 Acked 7203 Write 5 Done Done New false
20849 6205 Acked 7204 Read 2 Done Done New false
20850 6206 Acked 7205 Read 4 Done Done New false
20851 6207 Acked 7206 Write 7 Done Done New false
20852 6208 Acked 7207 Read 5 Done Done New false
20853 6209 Acked 7208 Read 2 Done Done New false
20854 6210 Acked 7209 Flush 0 Done Done New false
20855 6211 Acked 7210 Read 3 Done Done New false
20856 6212 Acked 7211 Read 4 Done Done New false
20857 6213 Acked 7212 Read 4 Done Done New false
20858 6214 Acked 7213 Read 7 Done Done New false
20859 6215 Acked 7214 Read 8 Done Done New false
20860 6216 Acked 7215 Read 10 Done Done New false
20861 6217 Acked 7216 Read 8 Done Done New false
20862 6218 Acked 7217 Read 10 Done Done New false
20863 6219 Acked 7218 Read 10 Done Done New false
20864 6220 Acked 7219 Read 5 Done Done New false
20865 6221 Acked 7220 Read 1 Done Done New false
20866 6222 Acked 7221 Flush 0 Done Done New false
20867 6223 Acked 7222 Read 2 Done Done New false
20868 6224 Acked 7223 Write 5 Done Done New false
20869 6225 Acked 7224 Read 1 Done Done New false
20870 6226 Acked 7225 Write 8 Done Done New false
20871 6227 Acked 7226 Read 3 Done Done New false
20872 6228 Acked 7227 Read 5 Done Done New false
20873 6229 Acked 7228 Read 2 Done Done New false
20874 6230 Acked 7229 Write 6 Done Done New false
20875 6231 Acked 7230 Read 1 Done Done New false
20876 6232 Acked 7231 Flush 0 Done Done New false
20877 6233 Acked 7232 Write 1 Done Done New false
20878 6234 Acked 7233 Read 6 Done Done New false
20879 6235 Acked 7234 Read 6 Done Done New false
20880 6236 Acked 7235 Write 3 Done Done New false
20881 6237 Acked 7236 Read 8 Done Done New false
20882 6238 Acked 7237 Write 2 Done Done New false
20883 6239 Acked 7238 Flush 0 Done Done New false
20884 6240 Acked 7239 Read 5 Done Done New false
20885 6241 Acked 7240 Flush 0 Done Done New false
20886 6242 Acked 7241 Read 7 Done Done New false
20887 6243 Acked 7242 Read 7 Done Done New false
20888 6244 Acked 7243 Write 3 Done Done New false
20889 6245 Acked 7244 Write 2 Done Done New false
20890 6246 Acked 7245 Read 3 Done Done New false
20891 6247 Acked 7246 Write 4 Done Done New false
20892 6248 Acked 7247 Read 9 Done Done New false
20893 6249 Acked 7248 Write 9 Done Done New false
20894 6250 Acked 7249 Write 10 Done Done New false
20895 6251 Acked 7250 Write 6 Done Done New false
20896 6252 Acked 7251 Read 9 Done Done New false
20897 6253 Acked 7252 Write 8 Done Done New false
20898 6254 Acked 7253 Read 6 Done Done New false
20899 6255 Acked 7254 Write 5 Done Done New false
20900 6256 Acked 7255 Write 6 Done Done New false
20901 6257 Acked 7256 Write 10 Done Done New false
20902 6258 Acked 7257 Write 8 Done Done New false
20903 6259 Acked 7258 Write 10 Done Done New false
20904 6260 Acked 7259 Read 6 Done Done New false
20905 6261 Acked 7260 Read 1 Done Done New false
20906 6262 Acked 7261 Read 4 Done Done New false
20907 6263 Acked 7262 Write 3 Done Done New false
20908 6264 Acked 7263 Read 10 Done Done New false
20909 6265 Acked 7264 Read 4 Done Done New false
20910 6266 Acked 7265 Write 5 Done Done New false
20911 6267 Acked 7266 Write 7 Done Done New false
20912 6268 Acked 7267 Write 6 Done Done New false
20913 6269 Acked 7268 Write 4 Done Done New false
20914 6270 Acked 7269 Read 2 Done Done New false
20915 6271 Acked 7270 Read 9 Done Done New false
20916 6272 Acked 7271 Flush 0 Done Done New false
20917 6273 Acked 7272 Read 6 Done Done New false
20918 6274 Acked 7273 Write 10 Done Done New false
20919 6275 Acked 7274 Read 6 Done Done New false
20920 6276 Acked 7275 Read 4 Done Done New false
20921 6277 Acked 7276 Read 1 Done Done New false
20922 6278 Acked 7277 Write 10 Done Done New false
20923 6279 Acked 7278 Flush 0 Done Done New false
20924 6280 Acked 7279 Write 5 Done Done New false
20925 6281 Acked 7280 Read 6 Done Done New false
20926 6282 Acked 7281 Read 2 Done Done New false
20927 6283 Acked 7282 Write 6 Done Done New false
20928 6284 Acked 7283 Write 5 Done Done New false
20929 6285 Acked 7284 Write 6 Done Done New false
20930 6286 Acked 7285 Read 2 Done Done New false
20931 6287 Acked 7286 Write 9 Done Done New false
20932 6288 Acked 7287 Read 10 Done Done New false
20933 6289 Acked 7288 Write 7 Done Done New false
20934 6290 Acked 7289 Write 8 Done Done New false
20935 6291 Acked 7290 Write 7 Done Done New false
20936 6292 Acked 7291 Read 1 Done Done New false
20937 6293 Acked 7292 Write 6 Done Done New false
20938 6294 Acked 7293 Read 6 Done Done New false
20939 6295 Acked 7294 Write 5 Done Done New false
20940 6296 Acked 7295 Flush 0 Done Done New false
20941 6297 Acked 7296 Flush 0 Done Done New false
20942 6298 Acked 7297 Write 1 Done Done New false
20943 6299 Acked 7298 Write 3 Done Done New false
20944 6300 Acked 7299 Write 4 Done Done New false
20945 6301 Acked 7300 Read 7 Done Done New false
20946 6302 Acked 7301 Read 10 Done Done New false
20947 6303 Acked 7302 Write 8 Done Done New false
20948 6304 Acked 7303 Read 2 Done Done New false
20949 6305 Acked 7304 Write 9 Done Done New false
20950 6306 Acked 7305 Write 3 Done Done New false
20951 6307 Acked 7306 Read 4 Done Done New false
20952 6308 Acked 7307 Read 9 Done Done New false
20953 6309 Acked 7308 Read 8 Done Done New false
20954 6310 Acked 7309 Write 9 Done Done New false
20955 6311 Acked 7310 Read 7 Done Done New false
20956 6312 Acked 7311 Read 5 Done Done New false
20957 6313 Acked 7312 Write 4 Done Done New false
20958 6314 Acked 7313 Read 6 Done Done New false
20959 6315 Acked 7314 Read 2 Done Done New false
20960 6316 Acked 7315 Read 5 Done Done New false
20961 6317 Acked 7316 Flush 0 Done Done New false
20962 6318 Acked 7317 Write 1 Done Done New false
20963 6319 Acked 7318 Read 4 Done Done New false
20964 6320 Acked 7319 Read 7 Done Done New false
20965 6321 Acked 7320 Flush 0 Done Done New false
20966 6322 Acked 7321 Read 7 Done Done New false
20967 6323 Acked 7322 Read 9 Done Done New false
20968 6324 Acked 7323 Read 7 Done Done New false
20969 6325 Acked 7324 Read 7 Done Done New false
20970 6326 Acked 7325 Read 2 Done Done New false
20971 6327 Acked 7326 Flush 0 Done Done New false
20972 6328 Acked 7327 Read 5 Done Done New false
20973 6329 Acked 7328 Write 6 Done Done New false
20974 6330 Acked 7329 Flush 0 Done Done New false
20975 STATES DS:0 DS:1 DS:2 TOTAL
20976 New 0 0 301 301
20977 Sent 0 0 0 0
20978 Done 301 301 0 602
20979 Skipped 0 0 0 0
20980 Error 0 0 0 0
20981 Last Flush: 7329 7329 7028
20982 Downstairs last five completed: 7028 7027 7026 7025 7024
20983 Upstairs last five completed: 6330 6329 6328 6327 6326
20984 CLIENT: Up:0 ds:301 act:2
209852024-02-15T03:01:38.447ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
209862024-02-15T03:01:38.447ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
209872024-02-15T03:01:38.447ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
209882024-02-15T03:01:38.447ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
209892024-02-15T03:01:38.447ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
209902024-02-15T03:01:38.447ZINFOcrucible: [2] client re-new 301 jobs since flush 7028 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
209912024-02-15T03:01:38.447ZINFOcrucible: client task is exiting = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
20992 [8830] got start action
20993 [8830] I am going to restart
20994 Starting downstairs at port 8830
20995 Make output file at "/tmp/dsc/downstairs-8830.txt"
20996 [8830][2] reports Starting
20997 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1188)
20998 [8830][2] reports Running
209992024-02-15T03:01:39.449ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
210002024-02-15T03:01:39.449ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
210012024-02-15T03:01:39.449ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
210022024-02-15T03:01:39.449ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
210032024-02-15T03:01:39.449ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
210042024-02-15T03:01:39.449ZINFOcrucible: [2] client re-new 301 jobs since flush 7028 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
210052024-02-15T03:01:39.449ZINFOcrucible: client task is exiting = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
210062024-02-15T03:01:40.451ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
210072024-02-15T03:01:40.451ZINFOcrucible: ds_connection connected = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
210082024-02-15T03:01:40.452ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
210092024-02-15T03:01:40.452ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
210102024-02-15T03:01:40.452ZINFOcrucible: send last flush ID to this DS: 7028 = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
210112024-02-15T03:01:40.452ZINFOcrucible: Replied this last flush ID: 7028 = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
210122024-02-15T03:01:40.452ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
210132024-02-15T03:01:40.452ZINFOcrucible: [2] Transition from Offline to Replay = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
210142024-02-15T03:01:40.452ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
210152024-02-15T03:01:40.453ZINFOcrucible: [2] Transition from Replay to Active = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
21016 ----------------------------------------------------------------
21017 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
21018 Downstairs last five completed: 7329 7328 7327 7326 7325
21019 Upstairs last five completed: 6330 6329 6328 6327 6326
21020 CLIENT: Up:0 ds:0 act:3
21021 Replay: All jobs finished, all DS active.
21022 Main task has work to do, go find it
21023 got dsc Stop(2)
21024 stop 2
21025 Feb 15 03:01:41.782 INFO request completed[8830] Got stop action so:false kr:false
21026 , latency_us: 342, response_code: 204, uri: /stop/cid/2, method: POST, req_id: ce5228b9-905b-4754-86e3-a1258e733e02, remote_addr: 127.0.0.1:54744, local_addr: 127.0.0.1:9998
210272024-02-15T03:01:41.783ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
210282024-02-15T03:01:41.783ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
210292024-02-15T03:01:41.783ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
21030 Feb 15 03:01:41.783 INFO request completed, latency_us{"msg":"Gone missing, transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-02-15T03:01:41.783408712Z","hostname":"ip-10-150-1-67.us-west-2.compute.internal","pid":1167,"client":"2","":"downstairs","session_id":"4f5a63e5-aa00-4890-9f6b-96c2e069fdb2"}
21031 : 174, response_code: 200, uri: /state/cid/2, method: GET, req_id: 03ec01a4-3703-4b90-b7fa-8e213ae3257d, remote_addr: 127.0.0.1:54744, local_addr: 127.0.0.1:9998{"msg":"[2] client re-new 0 jobs since flush 7329","v":0,"name":"crucible","level":30,"time":"2024-02-15T03:01:41.783456106Z","hostname":"ip-10-150-1-67.us-west-2.compute.internal","pid":1167,"":"downstairs","":"downstairs","session_id":"4f5a63e5-aa00-4890-9f6b-96c2e069fdb2"}
21032 
210332024-02-15T03:01:41.783ZINFOcrucible: client task is exiting = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
21034 [8830] Exited with: ExitStatus(unix_wait_status(9))
21035 [8830][2] reports Exit
210362024-02-15T03:01:42.785ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
210372024-02-15T03:01:42.785ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
210382024-02-15T03:01:42.785ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
210392024-02-15T03:01:42.785ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
210402024-02-15T03:01:42.785ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
210412024-02-15T03:01:42.785ZINFOcrucible: [2] client re-new 0 jobs since flush 7329 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
210422024-02-15T03:01:42.785ZINFOcrucible: client task is exiting = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
210432024-02-15T03:01:43.787ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
210442024-02-15T03:01:43.787ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
210452024-02-15T03:01:43.787ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
210462024-02-15T03:01:43.787ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
210472024-02-15T03:01:43.787ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
210482024-02-15T03:01:43.787ZINFOcrucible: [2] client re-new 0 jobs since flush 7329 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
210492024-02-15T03:01:43.787ZINFOcrucible: client task is exiting = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
210502024-02-15T03:01:44.788ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
210512024-02-15T03:01:44.788ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
210522024-02-15T03:01:44.788ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
210532024-02-15T03:01:44.788ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
210542024-02-15T03:01:44.788ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
210552024-02-15T03:01:44.788ZINFOcrucible: [2] client re-new 0 jobs since flush 7329 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
210562024-02-15T03:01:44.788ZINFOcrucible: client task is exiting = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
21057 Feb 15 03:01:45.786 INFO request completed, latency_us: 332, response_code: 200, uri: /state/cid/2, method: GET, req_id: 8a21afd1-337c-4da6-8577-b1cbf5b5c4e9, remote_addr: 127.0.0.1:54744, local_addr: 127.0.0.1:9998
21058 001/300 Read block 326 len 4096
21059 002/300 Read block 2755 len 16384
21060 003/300 Read block 3954 len 40960
210612024-02-15T03:01:45.789ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
210622024-02-15T03:01:45.789ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
210632024-02-15T03:01:45.789ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
210642024-02-15T03:01:45.789ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
210652024-02-15T03:01:45.789ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
210662024-02-15T03:01:45.789ZINFOcrucible: [2] client re-new 3 jobs since flush 7329 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
210672024-02-15T03:01:45.789ZINFOcrucible: client task is exiting = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
21068 004/300 Read block 1476 len 20480
21069 005/300 Write block 3760 len 36864 data: 4 4 5 6 6 6 8 8 8
21070 006/300 Read block 2897 len 20480
21071 007/300 Write block 4950 len 16384 data: 8 8 8 8
21072 008/300 Read block 352 len 8192
21073 009/300 Write block 3441 len 20480 data: 4 4 4 4 5
21074 010/300 Read block 1340 len 32768
21075 011/300 Flush
21076 012/300 Write block 2208 len 16384 data: 4 6 5 5
21077 013/300 Write block 1111 len 28672 data: 3 2 3 4 5 5 5
21078 014/300 Write block 2668 len 20480 data: 4 5 5 4 4
21079 015/300 Flush
21080 016/300 Write block 3825 len 36864 data: 5 4 4 3 3 4 4 5 8
21081 017/300 Read block 3557 len 12288
21082 018/300 Read block 863 len 32768
21083 019/300 Write block 4187 len 36864 data: 5 4 5 5 5 5 5 4 4
21084 020/300 Read block 2528 len 28672
21085 021/300 Read block 3325 len 20480
21086 022/300 Write block 4535 len 4096 data: 5
21087 023/300 Read block 1111 len 20480
21088 024/300 Read block 3751 len 40960
21089 025/300 Read block 2557 len 12288
21090 026/300 Read block 1385 len 20480
21091 027/300 Write block 745 len 8192 data: 6 6
21092 028/300 Write block 3149 len 24576 data: 6 7 7 7 7 8
21093 029/300 Read block 1322 len 32768
21094 030/300 Read block 490 len 8192
21095 031/300 Read block 4750 len 28672
21096 032/300 Read block 4844 len 8192
21097 033/300 Flush
21098 034/300 Write block 1218 len 32768 data: 5 7 6 7 6 5 6 5
21099 035/300 Read block 2181 len 4096
21100 036/300 Read block 2508 len 36864
21101 037/300 Read block 802 len 32768
21102 038/300 Read block 2317 len 16384
21103 039/300 Write block 3954 len 24576 data: 6 5 6 6 4 4
21104 040/300 Write block 2851 len 20480 data: 5 5 6 7 7
21105 041/300 Read block 1218 len 8192
21106 042/300 Write block 732 len 36864 data: 6 6 7 6 6 6 6 8 6
21107 043/300 Read block 1806 len 24576
21108 044/300 Read block 50 len 4096
21109 045/300 Write block 1396 len 40960 data: 6 4 4 4 5 4 4 4 6 8
21110 046/300 Flush
21111 047/300 Flush
21112 048/300 Write block 1720 len 8192 data: 2 2
21113 049/300 Write block 4839 len 36864 data: 5 5 5 5 4 6 5 5 4
21114 050/300 Write block 1949 len 32768 data: 4 4 6 5 7 6 6 6
21115 051/300 Write block 2372 len 8192 data: 3 3
21116 052/300 Flush
21117 053/300 Read block 4212 len 20480
21118 054/300 Write block 3076 len 32768 data: 4 4 4 5 5 6 6 7
21119 055/300 Write block 4725 len 40960 data: 6 6 5 5 3 3 4 4 3 3
21120 056/300 Write block 2533 len 36864 data: 3 4 5 5 5 4 4 3 4
21121 057/300 Write block 4311 len 24576 data: 8 7 6 6 6 4
21122 058/300 Write block 77 len 24576 data: 6 7 5 5 5 5
21123 059/300 Read block 4057 len 28672
21124 060/300 Read block 461 len 12288
21125 061/300 Write block 2363 len 32768 data: 5 5 4 5 4 3 3 3
21126 062/300 Read block 279 len 16384
21127 063/300 Write block 1864 len 4096 data: 2
21128 064/300 Flush
21129 065/300 Write block 1559 len 36864 data: 5 5 6 7 4 3 3 3 5
21130 066/300 Write block 4125 len 40960 data: 4 5 5 6 6 7 5 6 7 5
21131 067/300 Write block 3469 len 4096 data: 4
21132 068/300 Read block 1342 len 40960
21133 069/300 Read block 278 len 12288
21134 070/300 Write block 658 len 28672 data: 3 5 5 4 4 4 4
21135 071/300 Flush
21136 072/300 Read block 2787 len 28672
21137 073/300 Write block 3799 len 12288 data: 6 3 3
21138 074/300 Write block 1179 len 16384 data: 6 4 4 5
21139 075/300 Read block 3232 len 20480
21140 076/300 Write block 1353 len 8192 data: 4 4
21141 077/300 Read block 4845 len 12288
21142 078/300 Read block 181 len 28672
21143 079/300 Flush
21144 080/300 Flush
21145 081/300 Write block 1127 len 20480 data: 5 4 5 4 4
21146 082/300 Read block 343 len 32768
21147 083/300 Read block 149 len 32768
21148 084/300 Write block 4386 len 40960 data: 5 6 5 5 5 7 7 7 7 7
21149 085/300 Read block 3608 len 8192
21150 086/300 Write block 533 len 16384 data: 5 6 6 7
21151 087/300 Read block 668 len 8192
21152 088/300 Read block 2061 len 36864
21153 089/300 Write block 4295 len 12288 data: 4 4 3
21154 090/300 Read block 4437 len 24576
21155 091/300 Write block 1951 len 28672 data: 7 6 8 7 7 7 6
21156 092/300 Write block 2879 len 28672 data: 6 6 7 7 7 5 3
21157 093/300 Write block 1762 len 12288 data: 5 5 4
21158 094/300 Write block 2595 len 28672 data: 5 5 4 4 5 5 5
21159 095/300 Read block 4010 len 20480
21160 096/300 Read block 4469 len 36864
21161 097/300 Read block 1088 len 12288
21162 098/300 Write block 1662 len 40960 data: 4 3 2 2 2 3 4 4 3 3
21163 099/300 Write block 3559 len 8192 data: 5 5
21164 100/300 Write block 4778 len 40960 data: 6 9 9 8 8 7 7 6 4 3
21165 101/300 Read block 890 len 8192
21166 102/300 Read block 345 len 32768
21167 103/300 Read block 1937 len 24576
21168 104/300 Write block 1237 len 28672 data: 5 4 3 4 4 4 4
21169 105/300 Write block 3637 len 36864 data: 3 2 3 4 4 4 6 5 5
21170 106/300 Read block 3706 len 12288
21171 107/300 Write block 3188 len 12288 data: 7 6 7
21172 108/300 Read block 1768 len 20480
21173 109/300 Read block 3216 len 4096
21174 110/300 Read block 4677 len 40960
21175 111/300 Write block 3562 len 12288 data: 6 7 6
21176 112/300 Write block 2475 len 32768 data: 6 6 6 6 8 8 8 6
21177 113/300 Write block 2738 len 8192 data: 7 8
21178 114/300 Write block 1143 len 20480 data: 3 5 4 4 4
21179 115/300 Read block 4933 len 40960
21180 116/300 Read block 201 len 24576
21181 117/300 Read block 3911 len 24576
21182 118/300 Read block 4230 len 20480
21183 119/300 Write block 1310 len 20480 data: 4 3 3 3 3
21184 120/300 Read block 738 len 16384
21185 121/300 Write block 3897 len 20480 data: 4 4 3 3 3
21186 122/300 Read block 3297 len 28672
21187 123/300 Write block 175 len 16384 data: 5 6 7 6
21188 124/300 Write block 4716 len 4096 data: 4
21189 125/300 Read block 4299 len 28672
21190 126/300 Read block 4381 len 36864
21191 127/300 Read block 2256 len 40960
21192 128/300 Write block 327 len 12288 data: 5 4 4
21193 129/300 Read block 1842 len 20480
21194 130/300 Write block 713 len 16384 data: 6 5 5 5
21195 131/300 Write block 3971 len 8192 data: 4 3
21196 132/300 Write block 4836 len 20480 data: 5 6 7 6 6
21197 133/300 Read block 4903 len 16384
21198 134/300 Read block 2001 len 4096
21199 135/300 Write block 2440 len 16384 data: 4 4 3 3
21200 136/300 Read block 777 len 32768
21201 137/300 Write block 3596 len 36864 data: 6 5 6 6 6 5 3 4 5
21202 138/300 Write block 1981 len 16384 data: 3 3 2 2
21203 139/300 Write block 2662 len 32768 data: 5 7 8 7 6 5 5 6
21204 140/300 Write block 3006 len 4096 data: 2
21205 141/300 Read block 1241 len 20480
21206 142/300 Flush
21207 143/300 Write block 348 len 28672 data: 5 4 7 6 7 10 10
21208 144/300 Read block 1781 len 36864
21209 145/300 Read block 2117 len 36864
21210 146/300 Write block 4408 len 24576 data: 3 3 3 2 3 3
21211 147/300 Read block 1142 len 16384
21212 148/300 Flush
21213 149/300 Write block 1896 len 40960 data: 3 3 3 3 4 5 5 6 6 6
21214 150/300 Read block 2721 len 36864
21215 151/300 Read block 579 len 36864
21216 152/300 Write block 4621 len 16384 data: 5 5 5 6
21217 153/300 Flush
21218 154/300 Write block 2749 len 40960 data: 6 4 4 5 4 4 2 4 5 4
21219 155/300 Read block 4051 len 8192
21220 156/300 Read block 2317 len 40960
21221 157/300 Write block 569 len 20480 data: 5 6 4 4 3
21222 158/300 Read block 1534 len 12288
21223 159/300 Write block 3928 len 8192 data: 2 5
21224 160/300 Read block 1826 len 4096
21225 161/300 Write block 522 len 24576 data: 4 5 5 4 4 4
21226 162/300 Read block 1200 len 4096
21227 163/300 Read block 1834 len 4096
21228 164/300 Read block 4191 len 40960
21229 165/300 Write block 2302 len 12288 data: 2 2 3
21230 166/300 Read block 1862 len 4096
21231 167/300 Write block 2433 len 36864 data: 6 5 6 7 6 5 4 5 5
21232 168/300 Write block 2053 len 12288 data: 3 3 3
21233 169/300 Write block 1537 len 20480 data: 5 6 6 5 6
21234 170/300 Write block 4815 len 4096 data: 5
21235 171/300 Read block 891 len 24576
21236 172/300 Read block 3466 len 28672
21237 173/300 Write block 3975 len 28672 data: 3 3 3 3 3 4 5
21238 174/300 Read block 4043 len 16384
21239 175/300 Read block 2861 len 4096
21240 176/300 Write block 4255 len 32768 data: 3 4 7 6 8 7 8 8
21241 177/300 Write block 2647 len 16384 data: 7 6 4 3
21242 178/300 Write block 456 len 12288 data: 7 8 7
21243 179/300 Write block 1725 len 32768 data: 4 5 4 4 6 7 5 7
21244 180/300 Read block 121 len 12288
21245 181/300 Flush
21246 182/300 Read block 352 len 12288
21247 183/300 Read block 2688 len 4096
21248 184/300 Flush
21249 185/300 Read block 3620 len 24576
21250 186/300 Read block 1053 len 16384
21251 187/300 Write block 4860 len 36864 data: 5 5 6 8 8 8 6 6 7
21252 188/300 Read block 2216 len 12288
21253 189/300 Read block 4811 len 40960
21254 190/300 Read block 3118 len 28672
21255 191/300 Read block 2309 len 8192
21256 192/300 Read block 1596 len 16384
21257 193/300 Write block 398 len 16384 data: 3 3 2 2
21258 194/300 Read block 2265 len 40960
21259 195/300 Read block 4214 len 4096
21260 196/300 Write block 1713 len 16384 data: 3 2 2 2
21261 197/300 Read block 4845 len 40960
21262 198/300 Write block 210 len 28672 data: 4 5 6 5 4 3 3
21263 199/300 Read block 1431 len 4096
21264 200/300 Read block 1101 len 32768
21265 201/300 Flush
21266 202/300 Flush
21267 203/300 Write block 3702 len 20480 data: 3 2 3 3 3
21268 204/300 Flush
21269 205/300 Write block 1745 len 8192 data: 4 4
21270 206/300 Write block 3156 len 36864 data: 11 12 13 12 12 12 11 10 8
21271 207/300 Read block 26 len 4096
21272 208/300 Write block 501 len 40960 data: 5 5 5 5 5 6 7 8 6 6
21273 209/300 Read block 4424 len 20480
21274 210/300 Read block 2853 len 16384
21275 211/300 Read block 2242 len 36864
21276 212/300 Read block 3438 len 32768
21277 213/300 Flush
21278 214/300 Read block 4069 len 4096
21279 215/300 Read block 1419 len 36864
21280 216/300 Write block 1416 len 4096 data: 6
21281 217/300 Write block 789 len 36864 data: 6 6 5 6 5 6 6 7 7
21282 218/300 Flush
21283 219/300 Write block 1121 len 12288 data: 7 8 7
21284 220/300 Write block 2198 len 36864 data: 5 5 4 3 4 4 5 6 7
21285 221/300 Write block 2937 len 28672 data: 4 3 4 4 3 2 2
21286 222/300 Flush
21287 223/300 Flush
21288 224/300 Read block 2577 len 8192
21289 225/300 Write block 3557 len 20480 data: 5 4 6 6 6
21290 226/300 Write block 4091 len 8192 data: 6 7
21291 227/300 Write block 3948 len 40960 data: 8 8 6 7 6 5 7 6 7 7
21292 228/300 Write block 2330 len 8192 data: 4 3
21293 229/300 Read block 4242 len 8192
21294 230/300 Write block 1318 len 28672 data: 3 5 6 8 8 9 9
21295 231/300 Write block 4361 len 16384 data: 5 5 4 4
21296 232/300 Read block 2377 len 36864
21297 233/300 Flush
21298 234/300 Read block 1473 len 20480
21299 235/300 Write block 828 len 36864 data: 5 5 4 4 4 5 6 5 6
21300 236/300 Write block 2383 len 8192 data: 3 3
21301 237/300 Read block 4612 len 4096
21302 238/300 Read block 2411 len 16384
21303 239/300 Write block 1790 len 8192 data: 2 2
21304 240/300 Read block 3001 len 40960
21305 241/300 Write block 1138 len 8192 data: 6 6
21306 242/300 Write block 847 len 24576 data: 6 5 5 4 5 5
21307 243/300 Read block 4395 len 40960
21308 244/300 Write block 3263 len 4096 data: 5
21309 245/300 Write block 2334 len 8192 data: 3 4
21310 246/300 Read block 575 len 40960
21311 247/300 Read block 4135 len 32768
21312 248/300 Write block 2388 len 36864 data: 4 4 3 4 4 4 5 4 4
21313 249/300 Flush
21314 250/300 Read block 652 len 20480
21315 251/300 Read block 1699 len 20480
21316 252/300 Write block 2302 len 20480 data: 3 3 4 3 3
21317 253/300 Read block 2938 len 20480
21318 254/300 Write block 4281 len 28672 data: 7 7 7 7 4 4 5
21319 255/300 Read block 605 len 4096
21320 256/300 Write block 4618 len 4096 data: 4
21321 257/300 Read block 4449 len 24576
21322 258/300 Read block 1012 len 36864
21323 259/300 Read block 2454 len 36864
21324 260/300 Read block 313 len 4096
21325 261/300 Read block 2716 len 36864
21326 262/300 Write block 4114 len 28672 data: 9 7 7 9 8 5 5
21327 263/300 Write block 2957 len 36864 data: 4 5 5 5 3 4 2 2 2
21328 264/300 Read block 3301 len 32768
21329 265/300 Write block 3791 len 36864 data: 6 5 4 5 6 6 5 5 7
21330 266/300 Read block 1832 len 8192
21331 267/300 Write block 464 len 28672 data: 5 3 4 4 7 8 8
21332 268/300 Read block 4252 len 16384
21333 269/300 Write block 1763 len 32768 data: 6 5 3 3 4 3 3 4
21334 270/300 Flush
21335 271/300 Flush
21336 272/300 Read block 2343 len 8192
21337 273/300 Read block 1503 len 20480
21338 274/300 Write block 3211 len 36864 data: 4 6 8 7 7 6 6 5 4
21339 275/300 Read block 4638 len 40960
21340 276/300 Read block 4130 len 40960
21341 277/300 Write block 934 len 36864 data: 4 4 5 6 6 6 4 3 4
21342 278/300 Write block 2071 len 12288 data: 6 6 6
21343 279/300 Read block 3503 len 16384
21344 280/300 Read block 2931 len 24576
21345 281/300 Write block 4100 len 40960 data: 7 4 4 3 3 4 4 3 4 5
21346 282/300 Write block 1420 len 8192 data: 5 6
21347 283/300 Read block 3648 len 16384
21348 284/300 Write block 829 len 20480 data: 6 5 5 5 6
21349 285/300 Read block 3249 len 4096
21350 286/300 Read block 2070 len 40960
21351 287/300 Flush
21352 288/300 Flush
21353 289/300 Read block 3304 len 40960
21354 290/300 Read block 1964 len 24576
21355 291/300 Flush
21356 292/300 Write block 358 len 16384 data: 6 6 5 7
21357 293/300 Flush
21358 294/300 Read block 4295 len 28672
21359 295/300 Read block 4596 len 36864
21360 296/300 Read block 1960 len 24576
21361 297/300 Write block 2201 len 8192 data: 4 5
21362 298/300 Read block 4703 len 36864
21363 299/300 Read block 600 len 32768
21364 300/300 Read block 1689 len 36864
21365 Main task has work to do, go find it
21366 got dsc Start(2)
21367 start 2
21368 Feb 15 03:01:46.123 INFO request completed, latency_us: 219, response_code: 204, uri: /start/cid/2, method: POST, req_id: 134552b0-dda6-4d4e-80bf-4daee86b78b7, remote_addr: 127.0.0.1:54744, local_addr: 127.0.0.1:9998
21369 Replay: started 2, returned:Ok(())
21370 ----------------------------------------------------------------
21371 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
21372 States: Active Active Offline
21373 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
21374 6331 Acked 7330 Read 1 Done Done New true
21375 6332 Acked 7331 Read 4 Done Done New true
21376 6333 Acked 7332 Read 10 Done Done New true
21377 6334 Acked 7333 Read 5 Done Done New false
21378 6335 Acked 7334 Write 9 Done Done New false
21379 6336 Acked 7335 Read 5 Done Done New false
21380 6337 Acked 7336 Write 4 Done Done New false
21381 6338 Acked 7337 Read 2 Done Done New false
21382 6339 Acked 7338 Write 5 Done Done New false
21383 6340 Acked 7339 Read 8 Done Done New false
21384 6341 Acked 7340 Flush 0 Done Done New false
21385 6342 Acked 7341 Write 4 Done Done New false
21386 6343 Acked 7342 Write 7 Done Done New false
21387 6344 Acked 7343 Write 5 Done Done New false
21388 6345 Acked 7344 Flush 0 Done Done New false
21389 6346 Acked 7345 Write 9 Done Done New false
21390 6347 Acked 7346 Read 3 Done Done New false
21391 6348 Acked 7347 Read 8 Done Done New false
21392 6349 Acked 7348 Write 9 Done Done New false
21393 6350 Acked 7349 Read 7 Done Done New false
21394 6351 Acked 7350 Read 5 Done Done New false
21395 6352 Acked 7351 Write 1 Done Done New false
21396 6353 Acked 7352 Read 5 Done Done New false
21397 6354 Acked 7353 Read 10 Done Done New false
21398 6355 Acked 7354 Read 3 Done Done New false
21399 6356 Acked 7355 Read 5 Done Done New false
21400 6357 Acked 7356 Write 2 Done Done New false
21401 6358 Acked 7357 Write 6 Done Done New false
21402 6359 Acked 7358 Read 8 Done Done New false
21403 6360 Acked 7359 Read 2 Done Done New false
21404 6361 Acked 7360 Read 7 Done Done New false
21405 6362 Acked 7361 Read 2 Done Done New false
21406 6363 Acked 7362 Flush 0 Done Done New false
21407 6364 Acked 7363 Write 8 Done Done New false
21408 6365 Acked 7364 Read 1 Done Done New false
21409 6366 Acked 7365 Read 9 Done Done New false
21410 6367 Acked 7366 Read 8 Done Done New false
21411 6368 Acked 7367 Read 4 Done Done New false
21412 6369 Acked 7368 Write 6 Done Done New false
21413 6370 Acked 7369 Write 5 Done Done New false
21414 6371 Acked 7370 Read 2 Done Done New false
21415 6372 Acked 7371 Write 9 Done Done New false
21416 6373 Acked 7372 Read 6 Done Done New false
21417 6374 Acked 7373 Read 1 Done Done New false
21418 6375 Acked 7374 Write 10 Done Done New false
21419 6376 Acked 7375 Flush 0 Done Done New false
21420 6377 Acked 7376 Flush 0 Done Done New false
21421 6378 Acked 7377 Write 2 Done Done New false
21422 6379 Acked 7378 Write 9 Done Done New false
21423 6380 Acked 7379 Write 8 Done Done New false
21424 6381 Acked 7380 Write 2 Done Done New false
21425 6382 Acked 7381 Flush 0 Done Done New false
21426 6383 Acked 7382 Read 5 Done Done New false
21427 6384 Acked 7383 Write 8 Done Done New false
21428 6385 Acked 7384 Write 10 Done Done New false
21429 6386 Acked 7385 Write 9 Done Done New false
21430 6387 Acked 7386 Write 6 Done Done New false
21431 6388 Acked 7387 Write 6 Done Done New false
21432 6389 Acked 7388 Read 7 Done Done New false
21433 6390 Acked 7389 Read 3 Done Done New false
21434 6391 Acked 7390 Write 8 Done Done New false
21435 6392 Acked 7391 Read 4 Done Done New false
21436 6393 Acked 7392 Write 1 Done Done New false
21437 6394 Acked 7393 Flush 0 Done Done New false
21438 6395 Acked 7394 Write 9 Done Done New false
21439 6396 Acked 7395 Write 10 Done Done New false
21440 6397 Acked 7396 Write 1 Done Done New false
21441 6398 Acked 7397 Read 10 Done Done New false
21442 6399 Acked 7398 Read 3 Done Done New false
21443 6400 Acked 7399 Write 7 Done Done New false
21444 6401 Acked 7400 Flush 0 Done Done New false
21445 6402 Acked 7401 Read 7 Done Done New false
21446 6403 Acked 7402 Write 3 Done Done New false
21447 6404 Acked 7403 Write 4 Done Done New false
21448 6405 Acked 7404 Read 5 Done Done New false
21449 6406 Acked 7405 Write 2 Done Done New false
21450 6407 Acked 7406 Read 3 Done Done New false
21451 6408 Acked 7407 Read 7 Done Done New false
21452 6409 Acked 7408 Flush 0 Done Done New false
21453 6410 Acked 7409 Flush 0 Done Done New false
21454 6411 Acked 7410 Write 5 Done Done New false
21455 6412 Acked 7411 Read 8 Done Done New false
21456 6413 Acked 7412 Read 8 Done Done New false
21457 6414 Acked 7413 Write 10 Done Done New false
21458 6415 Acked 7414 Read 2 Done Done New false
21459 6416 Acked 7415 Write 4 Done Done New false
21460 6417 Acked 7416 Read 2 Done Done New false
21461 6418 Acked 7417 Read 9 Done Done New false
21462 6419 Acked 7418 Write 3 Done Done New false
21463 6420 Acked 7419 Read 6 Done Done New false
21464 6421 Acked 7420 Write 7 Done Done New false
21465 6422 Acked 7421 Write 7 Done Done New false
21466 6423 Acked 7422 Write 3 Done Done New false
21467 6424 Acked 7423 Write 7 Done Done New false
21468 6425 Acked 7424 Read 5 Done Done New false
21469 6426 Acked 7425 Read 9 Done Done New false
21470 6427 Acked 7426 Read 3 Done Done New false
21471 6428 Acked 7427 Write 10 Done Done New false
21472 6429 Acked 7428 Write 2 Done Done New false
21473 6430 Acked 7429 Write 10 Done Done New false
21474 6431 Acked 7430 Read 2 Done Done New false
21475 6432 Acked 7431 Read 8 Done Done New false
21476 6433 Acked 7432 Read 6 Done Done New false
21477 6434 Acked 7433 Write 7 Done Done New false
21478 6435 Acked 7434 Write 9 Done Done New false
21479 6436 Acked 7435 Read 3 Done Done New false
21480 6437 Acked 7436 Write 3 Done Done New false
21481 6438 Acked 7437 Read 5 Done Done New false
21482 6439 Acked 7438 Read 1 Done Done New false
21483 6440 Acked 7439 Read 10 Done Done New false
21484 6441 Acked 7440 Write 3 Done Done New false
21485 6442 Acked 7441 Write 8 Done Done New false
21486 6443 Acked 7442 Write 2 Done Done New false
21487 6444 Acked 7443 Write 5 Done Done New false
21488 6445 Acked 7444 Read 10 Done Done New false
21489 6446 Acked 7445 Read 6 Done Done New false
21490 6447 Acked 7446 Read 6 Done Done New false
21491 6448 Acked 7447 Read 5 Done Done New false
21492 6449 Acked 7448 Write 5 Done Done New false
21493 6450 Acked 7449 Read 4 Done Done New false
21494 6451 Acked 7450 Write 5 Done Done New false
21495 6452 Acked 7451 Read 7 Done Done New false
21496 6453 Acked 7452 Write 4 Done Done New false
21497 6454 Acked 7453 Write 1 Done Done New false
21498 6455 Acked 7454 Read 7 Done Done New false
21499 6456 Acked 7455 Read 9 Done Done New false
21500 6457 Acked 7456 Read 10 Done Done New false
21501 6458 Acked 7457 Write 3 Done Done New false
21502 6459 Acked 7458 Read 5 Done Done New false
21503 6460 Acked 7459 Write 4 Done Done New false
21504 6461 Acked 7460 Write 2 Done Done New false
21505 6462 Acked 7461 Write 5 Done Done New false
21506 6463 Acked 7462 Read 4 Done Done New false
21507 6464 Acked 7463 Read 1 Done Done New false
21508 6465 Acked 7464 Write 4 Done Done New false
21509 6466 Acked 7465 Read 8 Done Done New false
21510 6467 Acked 7466 Write 9 Done Done New false
21511 6468 Acked 7467 Write 4 Done Done New false
21512 6469 Acked 7468 Write 8 Done Done New false
21513 6470 Acked 7469 Write 1 Done Done New false
21514 6471 Acked 7470 Read 5 Done Done New false
21515 6472 Acked 7471 Flush 0 Done Done New false
21516 6473 Acked 7472 Write 7 Done Done New false
21517 6474 Acked 7473 Read 9 Done Done New false
21518 6475 Acked 7474 Read 9 Done Done New false
21519 6476 Acked 7475 Write 6 Done Done New false
21520 6477 Acked 7476 Read 4 Done Done New false
21521 6478 Acked 7477 Flush 0 Done Done New false
21522 6479 Acked 7478 Write 10 Done Done New false
21523 6480 Acked 7479 Read 9 Done Done New false
21524 6481 Acked 7480 Read 9 Done Done New false
21525 6482 Acked 7481 Write 4 Done Done New false
21526 6483 Acked 7482 Flush 0 Done Done New false
21527 6484 Acked 7483 Write 10 Done Done New false
21528 6485 Acked 7484 Read 2 Done Done New false
21529 6486 Acked 7485 Read 10 Done Done New false
21530 6487 Acked 7486 Write 5 Done Done New false
21531 6488 Acked 7487 Read 3 Done Done New false
21532 6489 Acked 7488 Write 2 Done Done New false
21533 6490 Acked 7489 Read 1 Done Done New false
21534 6491 Acked 7490 Write 6 Done Done New false
21535 6492 Acked 7491 Read 1 Done Done New false
21536 6493 Acked 7492 Read 1 Done Done New false
21537 6494 Acked 7493 Read 10 Done Done New false
21538 6495 Acked 7494 Write 3 Done Done New false
21539 6496 Acked 7495 Read 1 Done Done New false
21540 6497 Acked 7496 Write 9 Done Done New false
21541 6498 Acked 7497 Write 3 Done Done New false
21542 6499 Acked 7498 Write 5 Done Done New false
21543 6500 Acked 7499 Write 1 Done Done New false
21544 6501 Acked 7500 Read 6 Done Done New false
21545 6502 Acked 7501 Read 7 Done Done New false
21546 6503 Acked 7502 Write 7 Done Done New false
21547 6504 Acked 7503 Read 4 Done Done New false
21548 6505 Acked 7504 Read 1 Done Done New false
21549 6506 Acked 7505 Write 8 Done Done New false
21550 6507 Acked 7506 Write 4 Done Done New false
21551 6508 Acked 7507 Write 3 Done Done New false
21552 6509 Acked 7508 Write 8 Done Done New false
21553 6510 Acked 7509 Read 3 Done Done New false
21554 6511 Acked 7510 Flush 0 Done Done New false
21555 6512 Acked 7511 Read 3 Done Done New false
21556 6513 Acked 7512 Read 1 Done Done New false
21557 6514 Acked 7513 Flush 0 Done Done New false
21558 6515 Acked 7514 Read 6 Done Done New false
21559 6516 Acked 7515 Read 4 Done Done New false
21560 6517 Acked 7516 Write 9 Done Done New false
21561 6518 Acked 7517 Read 3 Done Done New false
21562 6519 Acked 7518 Read 10 Done Done New false
21563 6520 Acked 7519 Read 7 Done Done New false
21564 6521 Acked 7520 Read 2 Done Done New false
21565 6522 Acked 7521 Read 4 Done Done New false
21566 6523 Acked 7522 Write 4 Done Done New false
21567 6524 Acked 7523 Read 10 Done Done New false
21568 6525 Acked 7524 Read 1 Done Done New false
21569 6526 Acked 7525 Write 4 Done Done New false
21570 6527 Acked 7526 Read 10 Done Done New false
21571 6528 Acked 7527 Write 7 Done Done New false
21572 6529 Acked 7528 Read 1 Done Done New false
21573 6530 Acked 7529 Read 8 Done Done New false
21574 6531 Acked 7530 Flush 0 Done Done New false
21575 6532 Acked 7531 Flush 0 Done Done New false
21576 6533 Acked 7532 Write 5 Done Done New false
21577 6534 Acked 7533 Flush 0 Done Done New false
21578 6535 Acked 7534 Write 2 Done Done New false
21579 6536 Acked 7535 Write 9 Done Done New false
21580 6537 Acked 7536 Read 1 Done Done New false
21581 6538 Acked 7537 Write 10 Done Done New false
21582 6539 Acked 7538 Read 5 Done Done New false
21583 6540 Acked 7539 Read 4 Done Done New false
21584 6541 Acked 7540 Read 9 Done Done New false
21585 6542 Acked 7541 Read 8 Done Done New false
21586 6543 Acked 7542 Flush 0 Done Done New false
21587 6544 Acked 7543 Read 1 Done Done New false
21588 6545 Acked 7544 Read 9 Done Done New false
21589 6546 Acked 7545 Write 1 Done Done New false
21590 6547 Acked 7546 Write 9 Done Done New false
21591 6548 Acked 7547 Flush 0 Done Done New false
21592 6549 Acked 7548 Write 3 Done Done New false
21593 6550 Acked 7549 Write 9 Done Done New false
21594 6551 Acked 7550 Write 7 Done Done New false
21595 6552 Acked 7551 Flush 0 Done Done New false
21596 6553 Acked 7552 Flush 0 Done Done New false
21597 6554 Acked 7553 Read 2 Done Done New false
21598 6555 Acked 7554 Write 5 Done Done New false
21599 6556 Acked 7555 Write 2 Done Done New false
21600 6557 Acked 7556 Write 10 Done Done New false
21601 6558 Acked 7557 Write 2 Done Done New false
21602 6559 Acked 7558 Read 2 Done Done New false
21603 6560 Acked 7559 Write 7 Done Done New false
21604 6561 Acked 7560 Write 4 Done Done New false
21605 6562 Acked 7561 Read 9 Done Done New false
21606 6563 Acked 7562 Flush 0 Done Done New false
21607 6564 Acked 7563 Read 5 Done Done New false
21608 6565 Acked 7564 Write 9 Done Done New false
21609 6566 Acked 7565 Write 2 Done Done New false
21610 6567 Acked 7566 Read 1 Done Done New false
21611 6568 Acked 7567 Read 4 Done Done New false
21612 6569 Acked 7568 Write 2 Done Done New false
21613 6570 Acked 7569 Read 10 Done Done New false
21614 6571 Acked 7570 Write 2 Done Done New false
21615 6572 Acked 7571 Write 6 Done Done New false
21616 6573 Acked 7572 Read 10 Done Done New false
21617 6574 Acked 7573 Write 1 Done Done New false
21618 6575 Acked 7574 Write 2 Done Done New false
21619 6576 Acked 7575 Read 10 Done Done New false
21620 6577 Acked 7576 Read 8 Done Done New false
21621 6578 Acked 7577 Write 9 Done Done New false
21622 6579 Acked 7578 Flush 0 Done Done New false
21623 6580 Acked 7579 Read 5 Done Done New false
21624 6581 Acked 7580 Read 5 Done Done New false
21625 6582 Acked 7581 Write 5 Done Done New false
21626 6583 Acked 7582 Read 5 Done Done New false
21627 6584 Acked 7583 Write 7 Done Done New false
21628 6585 Acked 7584 Read 1 Done Done New false
21629 6586 Acked 7585 Write 1 Done Done New false
21630 6587 Acked 7586 Read 6 Done Done New false
21631 6588 Acked 7587 Read 9 Done Done New false
21632 6589 Acked 7588 Read 9 Done Done New false
21633 6590 Acked 7589 Read 1 Done Done New false
21634 6591 Acked 7590 Read 9 Done Done New false
21635 6592 Acked 7591 Write 7 Done Done New false
21636 6593 Acked 7592 Write 9 Done Done New false
21637 6594 Acked 7593 Read 8 Done Done New false
21638 6595 Acked 7594 Flush 0 Done Done New false
21639 6596 Acked 7595 Write 9 Done Done New false
21640 6597 Acked 7596 Read 2 Done Done New false
21641 6598 Acked 7597 Write 7 Done Done New false
21642 6599 Acked 7598 Read 4 Done Done New false
21643 6600 Acked 7599 Write 8 Done Done New false
21644 6601 Acked 7600 Flush 0 Done Done New false
21645 6602 Acked 7601 Flush 0 Done Done New false
21646 6603 Acked 7602 Read 2 Done Done New false
21647 6604 Acked 7603 Read 5 Done Done New false
21648 6605 Acked 7604 Write 9 Done Done New false
21649 6606 Acked 7605 Read 10 Done Done New false
21650 6607 Acked 7606 Read 10 Done Done New false
21651 6608 Acked 7607 Write 9 Done Done New false
21652 6609 Acked 7608 Write 3 Done Done New false
21653 6610 Acked 7609 Read 4 Done Done New false
21654 6611 Acked 7610 Read 6 Done Done New false
21655 6612 Acked 7611 Write 10 Done Done New false
21656 6613 Acked 7612 Write 2 Done Done New false
21657 6614 Acked 7613 Read 4 Done Done New false
21658 6615 Acked 7614 Write 5 Done Done New false
21659 6616 Acked 7615 Read 1 Done Done New false
21660 6617 Acked 7616 Read 10 Done Done New false
21661 6618 Acked 7617 Flush 0 Done Done New false
21662 6619 Acked 7618 Flush 0 Done Done New false
21663 6620 Acked 7619 Read 10 Done Done New false
21664 6621 Acked 7620 Read 6 Done Done New false
21665 6622 Acked 7621 Flush 0 Done Done New false
21666 6623 Acked 7622 Write 4 Done Done New false
21667 6624 Acked 7623 Flush 0 Done Done New false
21668 6625 Acked 7624 Read 7 Done Done New false
21669 6626 Acked 7625 Read 9 Done Done New false
21670 6627 Acked 7626 Read 6 Done Done New false
21671 6628 Acked 7627 Write 2 Done Done New false
21672 6629 Acked 7628 Read 9 Done Done New false
21673 6630 Acked 7629 Read 8 Done Done New false
21674 6631 Acked 7630 Read 9 Done Done New false
21675 STATES DS:0 DS:1 DS:2 TOTAL
21676 New 0 0 301 301
21677 Sent 0 0 0 0
21678 Done 301 301 0 602
21679 Skipped 0 0 0 0
21680 Error 0 0 0 0
21681 Last Flush: 7623 7623 7329
21682 Downstairs last five completed: 7329 7328 7327 7326 7325
21683 Upstairs last five completed: 6631 6630 6629 6628 6627
21684 CLIENT: Up:0 ds:301 act:2
216852024-02-15T03:01:46.791ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
216862024-02-15T03:01:46.791ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
216872024-02-15T03:01:46.791ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
216882024-02-15T03:01:46.791ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
216892024-02-15T03:01:46.791ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
216902024-02-15T03:01:46.791ZINFOcrucible: [2] client re-new 302 jobs since flush 7329 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
216912024-02-15T03:01:46.791ZINFOcrucible: client task is exiting = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
21692 [8830] got start action
21693 [8830] I am going to restart
21694 Starting downstairs at port 8830
21695 Make output file at "/tmp/dsc/downstairs-8830.txt"
21696 [8830][2] reports Starting
216972024-02-15T03:01:47.791ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
216982024-02-15T03:01:47.791ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
216992024-02-15T03:01:47.791ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
217002024-02-15T03:01:47.792ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
217012024-02-15T03:01:47.792ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
217022024-02-15T03:01:47.792ZINFOcrucible: [2] client re-new 302 jobs since flush 7329 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
217032024-02-15T03:01:47.792ZINFOcrucible: client task is exiting = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
21704 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1189)
21705 [8830][2] reports Running
217062024-02-15T03:01:48.793ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
217072024-02-15T03:01:48.793ZINFOcrucible: ds_connection connected = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
217082024-02-15T03:01:48.794ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
217092024-02-15T03:01:48.794ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
217102024-02-15T03:01:48.794ZINFOcrucible: send last flush ID to this DS: 7329 = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
217112024-02-15T03:01:48.794ZINFOcrucible: Replied this last flush ID: 7329 = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
217122024-02-15T03:01:48.794ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
217132024-02-15T03:01:48.794ZINFOcrucible: [2] Transition from Offline to Replay = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
217142024-02-15T03:01:48.795ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
217152024-02-15T03:01:48.795ZINFOcrucible: [2] Transition from Replay to Active = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
21716 ----------------------------------------------------------------
21717 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
21718 Downstairs last five completed: 7631 7630 7629 7628 7627
21719 Upstairs last five completed: 6632 6631 6630 6629 6628
21720 CLIENT: Up:0 ds:0 act:3
21721 Replay: All jobs finished, all DS active.
21722 Main task has work to do, go find it
21723 got dsc Stop(2)
21724 stop 2
21725 Feb 15 03:01:50.127 INFO request completed, latency_us: 327, response_code: 204, uri: /stop/cid/2[8830] Got stop action so:false kr:false
21726 , method: POST, req_id: 4d037553-1290-46e5-9faa-f514a31511fd, remote_addr: 127.0.0.1:54744, local_addr: 127.0.0.1:9998
21727 Feb 15 03:01:50.127 INFO request completed, latency_us: 121, response_code: 200, uri: /state/cid/2, method: GET, req_id: a18973e6-e3bf-4ab2-9db6-e02c7e4a944a, remote_addr: 127.0.0.1:54744, local_addr: 127.0.0.1:9998
217282024-02-15T03:01:50.127ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
217292024-02-15T03:01:50.127ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
217302024-02-15T03:01:50.127ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
217312024-02-15T03:01:50.127ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
217322024-02-15T03:01:50.127ZINFOcrucible: [2] client re-new 0 jobs since flush 7631 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
217332024-02-15T03:01:50.128ZINFOcrucible: client task is exiting = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
21734 [8830] Exited with: ExitStatus(unix_wait_status(9))
21735 [8830][2] reports Exit
217362024-02-15T03:01:51.129ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
217372024-02-15T03:01:51.129ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
217382024-02-15T03:01:51.129ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
217392024-02-15T03:01:51.129ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
217402024-02-15T03:01:51.129ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
217412024-02-15T03:01:51.129ZINFOcrucible: [2] client re-new 0 jobs since flush 7631 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
217422024-02-15T03:01:51.129ZINFOcrucible: client task is exiting = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
217432024-02-15T03:01:52.130ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
217442024-02-15T03:01:52.130ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
217452024-02-15T03:01:52.130ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
217462024-02-15T03:01:52.130ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
217472024-02-15T03:01:52.130ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
217482024-02-15T03:01:52.130ZINFOcrucible: [2] client re-new 0 jobs since flush 7631 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
217492024-02-15T03:01:52.130ZINFOcrucible: client task is exiting = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
217502024-02-15T03:01:53.132ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
217512024-02-15T03:01:53.132ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
217522024-02-15T03:01:53.132ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
217532024-02-15T03:01:53.132ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
217542024-02-15T03:01:53.132ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
217552024-02-15T03:01:53.132ZINFOcrucible: [2] client re-new 0 jobs since flush 7631 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
217562024-02-15T03:01:53.132ZINFOcrucible: client task is exiting = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
21757 Feb 15 03:01:54.130 INFO request completed, latency_us: 301, response_code: 200, uri: /state/cid/2, method: GET, req_id: 4f0e5c29-5b15-4827-9ee1-3e4ffdc21d7c, remote_addr: 127.0.0.1:54744, local_addr: 127.0.0.1:9998
21758 001/300 Read block 2712 len 40960
21759 002/300 Flush
21760 003/300 Read block 4436 len 24576
217612024-02-15T03:01:54.133ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
217622024-02-15T03:01:54.133ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
217632024-02-15T03:01:54.133ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
217642024-02-15T03:01:54.133ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
217652024-02-15T03:01:54.133ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
217662024-02-15T03:01:54.133ZINFOcrucible: [2] client re-new 3 jobs since flush 7631 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
217672024-02-15T03:01:54.133ZINFOcrucible: client task is exiting = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
21768 004/300 Read block 3554 len 20480
21769 005/300 Write block 1207 len 20480 data: 5 5 2 3 3
21770 006/300 Write block 211 len 20480 data: 6 7 6 5 4
21771 007/300 Read block 4318 len 24576
21772 008/300 Read block 498 len 16384
21773 009/300 Read block 3097 len 4096
21774 010/300 Read block 27 len 40960
21775 011/300 Read block 1539 len 28672
21776 012/300 Read block 4686 len 16384
21777 013/300 Write block 2746 len 28672 data: 5 8 8 7 5 5 6
21778 014/300 Read block 1140 len 8192
21779 015/300 Read block 4207 len 16384
21780 016/300 Read block 2918 len 12288
21781 017/300 Read block 3547 len 28672
21782 018/300 Read block 2706 len 28672
21783 019/300 Read block 2040 len 16384
21784 020/300 Write block 1076 len 28672 data: 4 4 4 4 5 5 5
21785 021/300 Read block 3368 len 24576
21786 022/300 Write block 1587 len 12288 data: 8 7 6
21787 023/300 Read block 3307 len 20480
21788 024/300 Write block 3839 len 16384 data: 3 3 3 2
21789 025/300 Read block 3030 len 12288
21790 026/300 Write block 820 len 4096 data: 7
21791 027/300 Write block 1699 len 32768 data: 5 4 5 4 5 4 5 4
21792 028/300 Read block 1279 len 32768
21793 029/300 Read block 4351 len 20480
21794 030/300 Write block 1685 len 24576 data: 3 4 4 4 5 5
21795 031/300 Read block 4937 len 4096
21796 032/300 Write block 4410 len 36864 data: 4 3 4 4 4 4 3 5 8
21797 033/300 Flush
21798 034/300 Read block 4117 len 20480
21799 035/300 Read block 1519 len 8192
21800 036/300 Write block 4974 len 32768 data: 4 4 4 4 4 4 4 5
21801 037/300 Read block 842 len 32768
21802 038/300 Write block 4960 len 20480 data: 3 4 4 5 4
21803 039/300 Write block 4774 len 32768 data: 6 3 4 4 7 10 10 9
21804 040/300 Read block 4233 len 32768
21805 041/300 Write block 4609 len 4096 data: 4
21806 042/300 Read block 227 len 36864
21807 043/300 Read block 4127 len 20480
21808 044/300 Write block 1587 len 36864 data: 9 8 7 7 6 7 7 6 6
21809 045/300 Flush
21810 046/300 Flush
21811 047/300 Read block 1594 len 16384
21812 048/300 Read block 861 len 36864
21813 049/300 Write block 4053 len 40960 data: 7 7 7 5 4 2 4 6 6 7
21814 050/300 Write block 3632 len 4096 data: 4
21815 051/300 Write block 1889 len 8192 data: 4 4
21816 052/300 Read block 4840 len 4096
21817 053/300 Read block 2574 len 20480
21818 054/300 Read block 3854 len 24576
21819 055/300 Read block 579 len 20480
21820 056/300 Read block 1421 len 24576
21821 057/300 Read block 2906 len 8192
21822 058/300 Write block 1555 len 20480 data: 4 3 3 4 6
21823 059/300 Write block 921 len 40960 data: 3 3 4 5 4 4 4 3 3 3
21824 060/300 Read block 2992 len 4096
21825 061/300 Write block 2149 len 24576 data: 3 3 2 3 3 4
21826 062/300 Write block 147 len 16384 data: 5 7 4 3
21827 063/300 Read block 273 len 16384
21828 064/300 Read block 3728 len 16384
21829 065/300 Flush
21830 066/300 Read block 4667 len 28672
21831 067/300 Flush
21832 068/300 Write block 398 len 36864 data: 4 4 3 3 3 3 2 2 2
21833 069/300 Read block 4617 len 24576
21834 070/300 Write block 1760 len 8192 data: 6 5
21835 071/300 Write block 4054 len 24576 data: 8 8 6 5 3 5
21836 072/300 Write block 1237 len 36864 data: 6 5 4 5 5 5 5 5 6
21837 073/300 Read block 3171 len 28672
21838 074/300 Write block 4716 len 24576 data: 5 5 4 6 5 4
21839 075/300 Read block 2614 len 8192
21840 076/300 Write block 4480 len 20480 data: 5 5 6 5 5
21841 077/300 Read block 4683 len 12288
21842 078/300 Read block 2014 len 12288
21843 079/300 Read block 3481 len 16384
21844 080/300 Read block 693 len 20480
21845 081/300 Read block 4005 len 4096
21846 082/300 Flush
21847 083/300 Read block 3167 len 4096
21848 084/300 Flush
21849 085/300 Read block 3832 len 12288
21850 086/300 Read block 1497 len 32768
21851 087/300 Read block 1803 len 32768
21852 088/300 Write block 546 len 12288 data: 5 6 6
21853 089/300 Write block 1240 len 4096 data: 6
21854 090/300 Write block 2952 len 20480 data: 4 5 5 4 4
21855 091/300 Flush
21856 092/300 Read block 4854 len 8192
21857 093/300 Read block 4376 len 12288
21858 094/300 Read block 2428 len 24576
21859 095/300 Read block 2252 len 20480
21860 096/300 Read block 4554 len 16384
21861 097/300 Write block 1769 len 16384 data: 4 5 4 4
21862 098/300 Write block 2774 len 36864 data: 4 3 3 3 6 6 5 5 6
21863 099/300 Read block 3519 len 16384
21864 100/300 Read block 4576 len 12288
21865 101/300 Read block 1876 len 40960
21866 102/300 Read block 4627 len 16384
21867 103/300 Write block 4053 len 36864 data: 8 9 9 7 6 4 6 7 7
21868 104/300 Read block 3023 len 12288
21869 105/300 Read block 213 len 32768
21870 106/300 Read block 3639 len 20480
21871 107/300 Flush
21872 108/300 Flush
21873 109/300 Write block 4646 len 4096 data: 5
21874 110/300 Write block 154 len 36864 data: 5 4 4 5 5 5 6 7 7
21875 111/300 Read block 3112 len 8192
21876 112/300 Flush
21877 113/300 Flush
21878 114/300 Write block 1250 len 24576 data: 8 7 6 5 6 8
21879 115/300 Write block 3907 len 24576 data: 8 9 9 7 5 5
21880 116/300 Write block 3679 len 20480 data: 5 6 4 3 4
21881 117/300 Read block 540 len 16384
21882 118/300 Flush
21883 119/300 Read block 3033 len 40960
21884 120/300 Read block 261 len 20480
21885 121/300 Read block 709 len 4096
21886 122/300 Write block 3723 len 8192 data: 4 5
21887 123/300 Read block 3203 len 36864
21888 124/300 Write block 2250 len 20480 data: 3 4 4 4 4
21889 125/300 Read block 3990 len 4096
21890 126/300 Write block 2121 len 36864 data: 6 6 7 7 7 8 9 8 6
21891 127/300 Write block 772 len 20480 data: 2 2 3 3 4
21892 128/300 Read block 2196 len 8192
21893 129/300 Read block 1058 len 36864
21894 130/300 Write block 1952 len 8192 data: 7 9
21895 131/300 Read block 3886 len 28672
21896 132/300 Read block 1057 len 28672
21897 133/300 Write block 414 len 20480 data: 2 2 3 3 2
21898 134/300 Read block 4004 len 4096
21899 135/300 Write block 4879 len 16384 data: 4 4 4 4
21900 136/300 Read block 3983 len 32768
21901 137/300 Read block 3017 len 40960
21902 138/300 Write block 68 len 28672 data: 2 3 5 5 7 7 7
21903 139/300 Read block 1435 len 8192
21904 140/300 Read block 1661 len 24576
21905 141/300 Write block 3525 len 20480 data: 7 7 5 6 7
21906 142/300 Read block 1165 len 20480
21907 143/300 Write block 3283 len 8192 data: 2 3
21908 144/300 Write block 3715 len 36864 data: 5 7 5 5 4 5 4 4 5
21909 145/300 Write block 3101 len 4096 data: 5
21910 146/300 Flush
21911 147/300 Read block 3049 len 40960
21912 148/300 Read block 369 len 36864
21913 149/300 Read block 3247 len 12288
21914 150/300 Write block 2300 len 12288 data: 3 3 4
21915 151/300 Read block 3490 len 28672
21916 152/300 Write block 2339 len 40960 data: 3 4 4 4 5 5 6 7 6 4
21917 153/300 Write block 2469 len 36864 data: 3 2 2 4 5 6 7 7 7
21918 154/300 Write block 3146 len 40960 data: 5 5 5 7 8 8 8 8 9 9
21919 155/300 Read block 2668 len 8192
21920 156/300 Read block 71 len 4096
21921 157/300 Read block 4220 len 24576
21922 158/300 Read block 3252 len 16384
21923 159/300 Write block 3856 len 36864 data: 6 7 6 7 8 6 5 7 8
21924 160/300 Read block 946 len 28672
21925 161/300 Read block 2900 len 8192
21926 162/300 Write block 4432 len 16384 data: 6 7 7 6
21927 163/300 Write block 4975 len 28672 data: 5 5 5 5 5 5 6
21928 164/300 Read block 1320 len 36864
21929 165/300 Write block 3742 len 36864 data: 3 3 3 3 2 2 2 2 4
21930 166/300 Write block 490 len 16384 data: 5 6 8 7
21931 167/300 Read block 4085 len 12288
21932 168/300 Read block 380 len 36864
21933 169/300 Read block 253 len 12288
21934 170/300 Read block 3557 len 24576
21935 171/300 Read block 3626 len 36864
21936 172/300 Read block 4904 len 12288
21937 173/300 Read block 1380 len 16384
21938 174/300 Write block 2162 len 32768 data: 6 5 4 4 6 7 4 4
21939 175/300 Write block 3394 len 28672 data: 4 4 4 5 6 6 6
21940 176/300 Read block 246 len 36864
21941 177/300 Read block 4125 len 16384
21942 178/300 Read block 783 len 36864
21943 179/300 Read block 579 len 24576
21944 180/300 Read block 4670 len 8192
21945 181/300 Read block 1337 len 28672
21946 182/300 Read block 2473 len 16384
21947 183/300 Write block 4009 len 28672 data: 5 3 4 6 6 8 9
21948 184/300 Write block 3351 len 20480 data: 8 7 7 5 4
21949 185/300 Write block 997 len 20480 data: 4 4 7 6 6
21950 186/300 Read block 597 len 36864
21951 187/300 Write block 4387 len 24576 data: 7 6 6 6 8 8
21952 188/300 Read block 3045 len 24576
21953 189/300 Read block 2593 len 40960
21954 190/300 Read block 1554 len 16384
21955 191/300 Write block 3183 len 12288 data: 3 2 3
21956 192/300 Read block 2154 len 16384
21957 193/300 Read block 1389 len 28672
21958 194/300 Read block 3282 len 8192
21959 195/300 Write block 1443 len 16384 data: 2 2 2 4
21960 196/300 Write block 950 len 24576 data: 3 4 3 3 4 4
21961 197/300 Write block 3742 len 28672 data: 4 4 4 4 3 3 3
21962 198/300 Write block 3433 len 36864 data: 4 4 3 2 2 3 4 5 5
21963 199/300 Flush
21964 200/300 Flush
21965 201/300 Read block 96 len 28672
21966 202/300 Write block 2532 len 4096 data: 3
21967 203/300 Write block 2516 len 28672 data: 4 4 3 3 5 6 6
21968 204/300 Write block 4853 len 12288 data: 3 2 2
21969 205/300 Write block 3013 len 28672 data: 4 4 4 4 3 3 2
21970 206/300 Read block 4206 len 8192
21971 207/300 Flush
21972 208/300 Write block 386 len 40960 data: 4 4 4 5 4 2 2 2 3 4
21973 209/300 Write block 4923 len 16384 data: 4 5 5 5
21974 210/300 Read block 3921 len 12288
21975 211/300 Write block 1578 len 36864 data: 6 6 5 7 7 9 7 5 5
21976 212/300 Read block 1611 len 12288
21977 213/300 Read block 3022 len 32768
21978 214/300 Read block 1596 len 28672
21979 215/300 Read block 2683 len 32768
21980 216/300 Flush
21981 217/300 Read block 4362 len 4096
21982 218/300 Write block 236 len 40960 data: 7 6 5 4 4 3 3 3 2 2
21983 219/300 Write block 3288 len 4096 data: 6
21984 220/300 Flush
21985 221/300 Read block 330 len 8192
21986 222/300 Read block 834 len 16384
21987 223/300 Flush
21988 224/300 Write block 408 len 8192 data: 2 2
21989 225/300 Read block 3968 len 4096
21990 226/300 Read block 3441 len 8192
21991 227/300 Read block 3733 len 40960
21992 228/300 Read block 618 len 32768
21993 229/300 Read block 4149 len 20480
21994 230/300 Flush
21995 231/300 Read block 178 len 20480
21996 232/300 Flush
21997 233/300 Flush
21998 234/300 Read block 4826 len 40960
21999 235/300 Flush
22000 236/300 Write block 1816 len 32768 data: 5 6 5 4 5 5 5 7
22001 237/300 Write block 465 len 24576 data: 4 5 5 8 9 9
22002 238/300 Read block 4787 len 28672
22003 239/300 Read block 3543 len 4096
22004 240/300 Read block 3403 len 4096
22005 241/300 Write block 2920 len 12288 data: 5 5 4
22006 242/300 Read block 1183 len 4096
22007 243/300 Write block 673 len 8192 data: 6 6
22008 244/300 Read block 3339 len 28672
22009 245/300 Read block 2554 len 12288
22010 246/300 Read block 2642 len 36864
22011 247/300 Flush
22012 248/300 Read block 4388 len 4096
22013 249/300 Write block 4964 len 12288 data: 5 6 5
22014 250/300 Write block 3059 len 12288 data: 6 6 6
22015 251/300 Flush
22016 252/300 Write block 1075 len 4096 data: 2
22017 253/300 Flush
22018 254/300 Read block 161 len 12288
22019 255/300 Read block 4749 len 4096
22020 256/300 Write block 4129 len 4096 data: 7
22021 257/300 Read block 1342 len 20480
22022 258/300 Read block 265 len 32768
22023 259/300 Read block 2318 len 32768
22024 260/300 Read block 3871 len 24576
22025 261/300 Read block 2965 len 36864
22026 262/300 Read block 254 len 40960
22027 263/300 Read block 3783 len 20480
22028 264/300 Read block 3706 len 20480
22029 265/300 Read block 3305 len 32768
22030 266/300 Flush
22031 267/300 Flush
22032 268/300 Read block 697 len 4096
22033 269/300 Write block 1056 len 4096 data: 4
22034 270/300 Read block 2145 len 20480
22035 271/300 Write block 1172 len 8192 data: 3 3
22036 272/300 Read block 961 len 36864
22037 273/300 Read block 3731 len 24576
22038 274/300 Flush
22039 275/300 Read block 3420 len 36864
22040 276/300 Write block 4104 len 8192 data: 4 5
22041 277/300 Read block 102 len 36864
22042 278/300 Write block 4529 len 28672 data: 5 5 4 5 6 6 6
22043 279/300 Write block 4491 len 32768 data: 3 2 2 3 3 3 4 3
22044 280/300 Read block 2945 len 40960
22045 281/300 Read block 591 len 36864
22046 282/300 Read block 4230 len 40960
22047 283/300 Write block 591 len 28672 data: 5 4 3 3 3 5 3
22048 284/300 Flush
22049 285/300 Flush
22050 286/300 Write block 2068 len 20480 data: 7 7 8 7 7
22051 287/300 Read block 3753 len 40960
22052 288/300 Write block 4044 len 36864 data: 8 7 6 8 9 7 8 8 7
22053 289/300 Write block 589 len 16384 data: 5 5 6 5
22054 290/300 Write block 1257 len 24576 data: 7 6 7 7 5 4
22055 291/300 Read block 4924 len 16384
22056 292/300 Write block 2292 len 32768 data: 6 6 6 5 5 5 5 3
22057 293/300 Read block 331 len 16384
22058 294/300 Write block 4880 len 20480 data: 5 5 5 4 2
22059 295/300 Write block 1846 len 28672 data: 6 5 3 3 3 3 3
22060 296/300 Flush
22061 297/300 Read block 2933 len 4096
22062 298/300 Read block 3800 len 36864
22063 299/300 Flush
22064 300/300 Read block 4597 len 12288
22065 Main task has work to do, go find it
22066 got dsc Start(2)
22067 start 2
22068 Feb 15 03:01:54.460 INFO request completed, latency_us: 205, response_code: 204, uri: /start/cid/2, method: POST, req_id: 2a59d82c-7b44-4d8c-b23a-85f0e77a697e, remote_addr: 127.0.0.1:54744, local_addr: 127.0.0.1:9998
22069 Replay: started 2, returned:Ok(())
22070 ----------------------------------------------------------------
22071 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:300
22072 States: Active Active Offline
22073 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
22074 6633 Acked 7632 Read 10 Done Done New true
22075 6634 Acked 7633 Flush 0 Done Done New true
22076 6635 Acked 7634 Read 6 Done Done New true
22077 6636 Acked 7635 Read 5 Done Done New false
22078 6637 Acked 7636 Write 5 Done Done New false
22079 6638 Acked 7637 Write 5 Done Done New false
22080 6639 Acked 7638 Read 6 Done Done New false
22081 6640 Acked 7639 Read 4 Done Done New false
22082 6641 Acked 7640 Read 1 Done Done New false
22083 6642 Acked 7641 Read 10 Done Done New false
22084 6643 Acked 7642 Read 7 Done Done New false
22085 6644 Acked 7643 Read 4 Done Done New false
22086 6645 Acked 7644 Write 7 Done Done New false
22087 6646 Acked 7645 Read 2 Done Done New false
22088 6647 Acked 7646 Read 4 Done Done New false
22089 6648 Acked 7647 Read 3 Done Done New false
22090 6649 Acked 7648 Read 7 Done Done New false
22091 6650 Acked 7649 Read 7 Done Done New false
22092 6651 Acked 7650 Read 4 Done Done New false
22093 6652 Acked 7651 Write 7 Done Done New false
22094 6653 Acked 7652 Read 6 Done Done New false
22095 6654 Acked 7653 Write 3 Done Done New false
22096 6655 Acked 7654 Read 5 Done Done New false
22097 6656 Acked 7655 Write 4 Done Done New false
22098 6657 Acked 7656 Read 3 Done Done New false
22099 6658 Acked 7657 Write 1 Done Done New false
22100 6659 Acked 7658 Write 8 Done Done New false
22101 6660 Acked 7659 Read 8 Done Done New false
22102 6661 Acked 7660 Read 5 Done Done New false
22103 6662 Acked 7661 Write 6 Done Done New false
22104 6663 Acked 7662 Read 1 Done Done New false
22105 6664 Acked 7663 Write 9 Done Done New false
22106 6665 Acked 7664 Flush 0 Done Done New false
22107 6666 Acked 7665 Read 5 Done Done New false
22108 6667 Acked 7666 Read 2 Done Done New false
22109 6668 Acked 7667 Write 8 Done Done New false
22110 6669 Acked 7668 Read 8 Done Done New false
22111 6670 Acked 7669 Write 5 Done Done New false
22112 6671 Acked 7670 Write 8 Done Done New false
22113 6672 Acked 7671 Read 8 Done Done New false
22114 6673 Acked 7672 Write 1 Done Done New false
22115 6674 Acked 7673 Read 9 Done Done New false
22116 6675 Acked 7674 Read 5 Done Done New false
22117 6676 Acked 7675 Write 9 Done Done New false
22118 6677 Acked 7676 Flush 0 Done Done New false
22119 6678 Acked 7677 Flush 0 Done Done New false
22120 6679 Acked 7678 Read 4 Done Done New false
22121 6680 Acked 7679 Read 9 Done Done New false
22122 6681 Acked 7680 Write 10 Done Done New false
22123 6682 Acked 7681 Write 1 Done Done New false
22124 6683 Acked 7682 Write 2 Done Done New false
22125 6684 Acked 7683 Read 1 Done Done New false
22126 6685 Acked 7684 Read 5 Done Done New false
22127 6686 Acked 7685 Read 6 Done Done New false
22128 6687 Acked 7686 Read 5 Done Done New false
22129 6688 Acked 7687 Read 6 Done Done New false
22130 6689 Acked 7688 Read 2 Done Done New false
22131 6690 Acked 7689 Write 5 Done Done New false
22132 6691 Acked 7690 Write 10 Done Done New false
22133 6692 Acked 7691 Read 1 Done Done New false
22134 6693 Acked 7692 Write 6 Done Done New false
22135 6694 Acked 7693 Write 4 Done Done New false
22136 6695 Acked 7694 Read 4 Done Done New false
22137 6696 Acked 7695 Read 4 Done Done New false
22138 6697 Acked 7696 Flush 0 Done Done New false
22139 6698 Acked 7697 Read 7 Done Done New false
22140 6699 Acked 7698 Flush 0 Done Done New false
22141 6700 Acked 7699 Write 9 Done Done New false
22142 6701 Acked 7700 Read 6 Done Done New false
22143 6702 Acked 7701 Write 2 Done Done New false
22144 6703 Acked 7702 Write 6 Done Done New false
22145 6704 Acked 7703 Write 9 Done Done New false
22146 6705 Acked 7704 Read 7 Done Done New false
22147 6706 Acked 7705 Write 6 Done Done New false
22148 6707 Acked 7706 Read 2 Done Done New false
22149 6708 Acked 7707 Write 5 Done Done New false
22150 6709 Acked 7708 Read 3 Done Done New false
22151 6710 Acked 7709 Read 3 Done Done New false
22152 6711 Acked 7710 Read 4 Done Done New false
22153 6712 Acked 7711 Read 5 Done Done New false
22154 6713 Acked 7712 Read 1 Done Done New false
22155 6714 Acked 7713 Flush 0 Done Done New false
22156 6715 Acked 7714 Read 1 Done Done New false
22157 6716 Acked 7715 Flush 0 Done Done New false
22158 6717 Acked 7716 Read 3 Done Done New false
22159 6718 Acked 7717 Read 8 Done Done New false
22160 6719 Acked 7718 Read 8 Done Done New false
22161 6720 Acked 7719 Write 3 Done Done New false
22162 6721 Acked 7720 Write 1 Done Done New false
22163 6722 Acked 7721 Write 5 Done Done New false
22164 6723 Acked 7722 Flush 0 Done Done New false
22165 6724 Acked 7723 Read 2 Done Done New false
22166 6725 Acked 7724 Read 3 Done Done New false
22167 6726 Acked 7725 Read 6 Done Done New false
22168 6727 Acked 7726 Read 5 Done Done New false
22169 6728 Acked 7727 Read 4 Done Done New false
22170 6729 Acked 7728 Write 4 Done Done New false
22171 6730 Acked 7729 Write 9 Done Done New false
22172 6731 Acked 7730 Read 4 Done Done New false
22173 6732 Acked 7731 Read 3 Done Done New false
22174 6733 Acked 7732 Read 10 Done Done New false
22175 6734 Acked 7733 Read 4 Done Done New false
22176 6735 Acked 7734 Write 9 Done Done New false
22177 6736 Acked 7735 Read 3 Done Done New false
22178 6737 Acked 7736 Read 8 Done Done New false
22179 6738 Acked 7737 Read 5 Done Done New false
22180 6739 Acked 7738 Flush 0 Done Done New false
22181 6740 Acked 7739 Flush 0 Done Done New false
22182 6741 Acked 7740 Write 1 Done Done New false
22183 6742 Acked 7741 Write 9 Done Done New false
22184 6743 Acked 7742 Read 2 Done Done New false
22185 6744 Acked 7743 Flush 0 Done Done New false
22186 6745 Acked 7744 Flush 0 Done Done New false
22187 6746 Acked 7745 Write 6 Done Done New false
22188 6747 Acked 7746 Write 6 Done Done New false
22189 6748 Acked 7747 Write 5 Done Done New false
22190 6749 Acked 7748 Read 4 Done Done New false
22191 6750 Acked 7749 Flush 0 Done Done New false
22192 6751 Acked 7750 Read 10 Done Done New false
22193 6752 Acked 7751 Read 5 Done Done New false
22194 6753 Acked 7752 Read 1 Done Done New false
22195 6754 Acked 7753 Write 2 Done Done New false
22196 6755 Acked 7754 Read 9 Done Done New false
22197 6756 Acked 7755 Write 5 Done Done New false
22198 6757 Acked 7756 Read 1 Done Done New false
22199 6758 Acked 7757 Write 9 Done Done New false
22200 6759 Acked 7758 Write 5 Done Done New false
22201 6760 Acked 7759 Read 2 Done Done New false
22202 6761 Acked 7760 Read 9 Done Done New false
22203 6762 Acked 7761 Write 2 Done Done New false
22204 6763 Acked 7762 Read 7 Done Done New false
22205 6764 Acked 7763 Read 7 Done Done New false
22206 6765 Acked 7764 Write 5 Done Done New false
22207 6766 Acked 7765 Read 1 Done Done New false
22208 6767 Acked 7766 Write 4 Done Done New false
22209 6768 Acked 7767 Read 8 Done Done New false
22210 6769 Acked 7768 Read 10 Done Done New false
22211 6770 Acked 7769 Write 7 Done Done New false
22212 6771 Acked 7770 Read 2 Done Done New false
22213 6772 Acked 7771 Read 6 Done Done New false
22214 6773 Acked 7772 Write 5 Done Done New false
22215 6774 Acked 7773 Read 5 Done Done New false
22216 6775 Acked 7774 Write 2 Done Done New false
22217 6776 Acked 7775 Write 9 Done Done New false
22218 6777 Acked 7776 Write 1 Done Done New false
22219 6778 Acked 7777 Flush 0 Done Done New false
22220 6779 Acked 7778 Read 10 Done Done New false
22221 6780 Acked 7779 Read 9 Done Done New false
22222 6781 Acked 7780 Read 3 Done Done New false
22223 6782 Acked 7781 Write 3 Done Done New false
22224 6783 Acked 7782 Read 7 Done Done New false
22225 6784 Acked 7783 Write 10 Done Done New false
22226 6785 Acked 7784 Write 9 Done Done New false
22227 6786 Acked 7785 Write 10 Done Done New false
22228 6787 Acked 7786 Read 2 Done Done New false
22229 6788 Acked 7787 Read 1 Done Done New false
22230 6789 Acked 7788 Read 6 Done Done New false
22231 6790 Acked 7789 Read 4 Done Done New false
22232 6791 Acked 7790 Write 9 Done Done New false
22233 6792 Acked 7791 Read 7 Done Done New false
22234 6793 Acked 7792 Read 2 Done Done New false
22235 6794 Acked 7793 Write 4 Done Done New false
22236 6795 Acked 7794 Write 7 Done Done New false
22237 6796 Acked 7795 Read 9 Done Done New false
22238 6797 Acked 7796 Write 9 Done Done New false
22239 6798 Acked 7797 Write 4 Done Done New false
22240 6799 Acked 7798 Read 3 Done Done New false
22241 6800 Acked 7799 Read 9 Done Done New false
22242 6801 Acked 7800 Read 3 Done Done New false
22243 6802 Acked 7801 Read 6 Done Done New false
22244 6803 Acked 7802 Read 9 Done Done New false
22245 6804 Acked 7803 Read 3 Done Done New false
22246 6805 Acked 7804 Read 4 Done Done New false
22247 6806 Acked 7805 Write 8 Done Done New false
22248 6807 Acked 7806 Write 7 Done Done New false
22249 6808 Acked 7807 Read 9 Done Done New false
22250 6809 Acked 7808 Read 4 Done Done New false
22251 6810 Acked 7809 Read 9 Done Done New false
22252 6811 Acked 7810 Read 6 Done Done New false
22253 6812 Acked 7811 Read 2 Done Done New false
22254 6813 Acked 7812 Read 7 Done Done New false
22255 6814 Acked 7813 Read 4 Done Done New false
22256 6815 Acked 7814 Write 7 Done Done New false
22257 6816 Acked 7815 Write 5 Done Done New false
22258 6817 Acked 7816 Write 5 Done Done New false
22259 6818 Acked 7817 Read 9 Done Done New false
22260 6819 Acked 7818 Write 6 Done Done New false
22261 6820 Acked 7819 Read 6 Done Done New false
22262 6821 Acked 7820 Read 10 Done Done New false
22263 6822 Acked 7821 Read 4 Done Done New false
22264 6823 Acked 7822 Write 3 Done Done New false
22265 6824 Acked 7823 Read 4 Done Done New false
22266 6825 Acked 7824 Read 7 Done Done New false
22267 6826 Acked 7825 Read 2 Done Done New false
22268 6827 Acked 7826 Write 4 Done Done New false
22269 6828 Acked 7827 Write 6 Done Done New false
22270 6829 Acked 7828 Write 7 Done Done New false
22271 6830 Acked 7829 Write 9 Done Done New false
22272 6831 Acked 7830 Flush 0 Done Done New false
22273 6832 Acked 7831 Flush 0 Done Done New false
22274 6833 Acked 7832 Read 7 Done Done New false
22275 6834 Acked 7833 Write 1 Done Done New false
22276 6835 Acked 7834 Write 7 Done Done New false
22277 6836 Acked 7835 Write 3 Done Done New false
22278 6837 Acked 7836 Write 7 Done Done New false
22279 6838 Acked 7837 Read 2 Done Done New false
22280 6839 Acked 7838 Flush 0 Done Done New false
22281 6840 Acked 7839 Write 10 Done Done New false
22282 6841 Acked 7840 Write 4 Done Done New false
22283 6842 Acked 7841 Read 3 Done Done New false
22284 6843 Acked 7842 Write 9 Done Done New false
22285 6844 Acked 7843 Read 3 Done Done New false
22286 6845 Acked 7844 Read 8 Done Done New false
22287 6846 Acked 7845 Read 7 Done Done New false
22288 6847 Acked 7846 Read 8 Done Done New false
22289 6848 Acked 7847 Flush 0 Done Done New false
22290 6849 Acked 7848 Read 1 Done Done New false
22291 6850 Acked 7849 Write 10 Done Done New false
22292 6851 Acked 7850 Write 1 Done Done New false
22293 6852 Acked 7851 Flush 0 Done Done New false
22294 6853 Acked 7852 Read 2 Done Done New false
22295 6854 Acked 7853 Read 4 Done Done New false
22296 6855 Acked 7854 Flush 0 Done Done New false
22297 6856 Acked 7855 Write 2 Done Done New false
22298 6857 Acked 7856 Read 1 Done Done New false
22299 6858 Acked 7857 Read 2 Done Done New false
22300 6859 Acked 7858 Read 10 Done Done New false
22301 6860 Acked 7859 Read 8 Done Done New false
22302 6861 Acked 7860 Read 5 Done Done New false
22303 6862 Acked 7861 Flush 0 Done Done New false
22304 6863 Acked 7862 Read 5 Done Done New false
22305 6864 Acked 7863 Flush 0 Done Done New false
22306 6865 Acked 7864 Flush 0 Done Done New false
22307 6866 Acked 7865 Read 10 Done Done New false
22308 6867 Acked 7866 Flush 0 Done Done New false
22309 6868 Acked 7867 Write 8 Done Done New false
22310 6869 Acked 7868 Write 6 Done Done New false
22311 6870 Acked 7869 Read 7 Done Done New false
22312 6871 Acked 7870 Read 1 Done Done New false
22313 6872 Acked 7871 Read 1 Done Done New false
22314 6873 Acked 7872 Write 3 Done Done New false
22315 6874 Acked 7873 Read 1 Done Done New false
22316 6875 Acked 7874 Write 2 Done Done New false
22317 6876 Acked 7875 Read 7 Done Done New false
22318 6877 Acked 7876 Read 3 Done Done New false
22319 6878 Acked 7877 Read 9 Done Done New false
22320 6879 Acked 7878 Flush 0 Done Done New false
22321 6880 Acked 7879 Read 1 Done Done New false
22322 6881 Acked 7880 Write 3 Done Done New false
22323 6882 Acked 7881 Write 3 Done Done New false
22324 6883 Acked 7882 Flush 0 Done Done New false
22325 6884 Acked 7883 Write 1 Done Done New false
22326 6885 Acked 7884 Flush 0 Done Done New false
22327 6886 Acked 7885 Read 3 Done Done New false
22328 6887 Acked 7886 Read 1 Done Done New false
22329 6888 Acked 7887 Write 1 Done Done New false
22330 6889 Acked 7888 Read 5 Done Done New false
22331 6890 Acked 7889 Read 8 Done Done New false
22332 6891 Acked 7890 Read 8 Done Done New false
22333 6892 Acked 7891 Read 6 Done Done New false
22334 6893 Acked 7892 Read 9 Done Done New false
22335 6894 Acked 7893 Read 10 Done Done New false
22336 6895 Acked 7894 Read 5 Done Done New false
22337 6896 Acked 7895 Read 5 Done Done New false
22338 6897 Acked 7896 Read 8 Done Done New false
22339 6898 Acked 7897 Flush 0 Done Done New false
22340 6899 Acked 7898 Flush 0 Done Done New false
22341 6900 Acked 7899 Read 1 Done Done New false
22342 6901 Acked 7900 Write 1 Done Done New false
22343 6902 Acked 7901 Read 5 Done Done New false
22344 6903 Acked 7902 Write 2 Done Done New false
22345 6904 Acked 7903 Read 9 Done Done New false
22346 6905 Acked 7904 Read 6 Done Done New false
22347 6906 Acked 7905 Flush 0 Done Done New false
22348 6907 Acked 7906 Read 9 Done Done New false
22349 6908 Acked 7907 Write 2 Done Done New false
22350 6909 Acked 7908 Read 9 Done Done New false
22351 6910 Acked 7909 Write 7 Done Done New false
22352 6911 Acked 7910 Write 8 Done Done New false
22353 6912 Acked 7911 Read 10 Done Done New false
22354 6913 Acked 7912 Read 9 Done Done New false
22355 6914 Acked 7913 Read 10 Done Done New false
22356 6915 Acked 7914 Write 7 Done Done New false
22357 6916 Acked 7915 Flush 0 Done Done New false
22358 6917 Acked 7916 Flush 0 Done Done New false
22359 6918 Acked 7917 Write 5 Done Done New false
22360 6919 Acked 7918 Read 10 Done Done New false
22361 6920 Acked 7919 Write 9 Done Done New false
22362 6921 Acked 7920 Write 4 Done Done New false
22363 6922 Acked 7921 Write 6 Done Done New false
22364 6923 Acked 7922 Read 4 Done Done New false
22365 6924 Acked 7923 Write 8 Done Done New false
22366 6925 Acked 7924 Read 4 Done Done New false
22367 6926 Acked 7925 Write 5 Done Done New false
22368 6927 Acked 7926 Write 7 Done Done New false
22369 6928 Acked 7927 Flush 0 Done Done New false
22370 6929 Acked 7928 Read 1 Done Done New false
22371 6930 Acked 7929 Read 9 Done Done New false
22372 6931 Acked 7930 Flush 0 Done Done New false
22373 6932 Acked 7931 Read 3 Done Done New false
22374 STATES DS:0 DS:1 DS:2 TOTAL
22375 New 0 0 300 300
22376 Sent 0 0 0 0
22377 Done 300 300 0 600
22378 Skipped 0 0 0 0
22379 Error 0 0 0 0
22380 Last Flush: 7930 7930 7631
22381 Downstairs last five completed: 7631 7630 7629 7628 7627
22382 Upstairs last five completed: 6932 6931 6930 6929 6928
22383 CLIENT: Up:0 ds:300 act:2
223842024-02-15T03:01:55.134ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
223852024-02-15T03:01:55.135ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
223862024-02-15T03:01:55.135ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
223872024-02-15T03:01:55.135ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
223882024-02-15T03:01:55.135ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
223892024-02-15T03:01:55.135ZINFOcrucible: [2] client re-new 301 jobs since flush 7631 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
223902024-02-15T03:01:55.135ZINFOcrucible: client task is exiting = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
22391 [8830] got start action
22392 [8830] I am going to restart
22393 Starting downstairs at port 8830
22394 Make output file at "/tmp/dsc/downstairs-8830.txt"
22395 [8830][2] reports Starting
223962024-02-15T03:01:56.136ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
223972024-02-15T03:01:56.136ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
223982024-02-15T03:01:56.136ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
223992024-02-15T03:01:56.136ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
224002024-02-15T03:01:56.136ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
224012024-02-15T03:01:56.136ZINFOcrucible: [2] client re-new 301 jobs since flush 7631 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
224022024-02-15T03:01:56.136ZINFOcrucible: client task is exiting = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
22403 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1190)
22404 [8830][2] reports Running
224052024-02-15T03:01:57.138ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
224062024-02-15T03:01:57.138ZINFOcrucible: ds_connection connected = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
224072024-02-15T03:01:57.139ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
224082024-02-15T03:01:57.140ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
224092024-02-15T03:01:57.140ZINFOcrucible: send last flush ID to this DS: 7631 = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
224102024-02-15T03:01:57.140ZINFOcrucible: Replied this last flush ID: 7631 = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
224112024-02-15T03:01:57.140ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
224122024-02-15T03:01:57.140ZINFOcrucible: [2] Transition from Offline to Replay = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
224132024-02-15T03:01:57.140ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
224142024-02-15T03:01:57.140ZINFOcrucible: [2] Transition from Replay to Active = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
22415 ----------------------------------------------------------------
22416 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
22417 Downstairs last five completed: 7932 7931 7930 7929 7928
22418 Upstairs last five completed: 6933 6932 6931 6930 6929
22419 CLIENT: Up:0 ds:0 act:3
22420 Replay: All jobs finished, all DS active.
22421 Main task has work to do, go find it
22422 got dsc Stop(0)
22423 stop 0
22424 Feb 15 03:01:58.464 INFO request completed, latency_us: 343, response_code: 204, uri: /stop/cid/0[8810] Got stop action so:false kr:false
22425 , method: POST, req_id: 39111c78-e538-41f2-9f7a-2d8e5599324a, remote_addr: 127.0.0.1:54744, local_addr: 127.0.0.1:9998
224262024-02-15T03:01:58.465ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
224272024-02-15T03:01:58.465ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
224282024-02-15T03:01:58.465ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
224292024-02-15T03:01:58.465ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
224302024-02-15T03:01:58.465ZINFOcrucible: [0] client re-new 0 jobs since flush 7932 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
224312024-02-15T03:01:58.465ZINFOcrucible: client task is exiting = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
22432 Feb 15 03:01:58.465 INFO request completed, latency_us: 174, response_code: 200, uri: /state/cid/0, method: GET, req_id: c70689e5-4864-49dd-bb95-a3b49421a9b5, remote_addr: 127.0.0.1:54744, local_addr: 127.0.0.1:9998
22433 [8810] Exited with: ExitStatus(unix_wait_status(9))
22434 [8810][0] reports Exit
224352024-02-15T03:01:59.466ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
224362024-02-15T03:01:59.466ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
224372024-02-15T03:01:59.466ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
224382024-02-15T03:01:59.466ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
224392024-02-15T03:01:59.466ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
224402024-02-15T03:01:59.466ZINFOcrucible: [0] client re-new 0 jobs since flush 7932 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
224412024-02-15T03:01:59.466ZINFOcrucible: client task is exiting = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
224422024-02-15T03:02:00.468ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
224432024-02-15T03:02:00.468ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
224442024-02-15T03:02:00.468ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
224452024-02-15T03:02:00.468ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
224462024-02-15T03:02:00.468ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
224472024-02-15T03:02:00.468ZINFOcrucible: [0] client re-new 0 jobs since flush 7932 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
224482024-02-15T03:02:00.468ZINFOcrucible: client task is exiting = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
224492024-02-15T03:02:01.469ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
224502024-02-15T03:02:01.469ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
224512024-02-15T03:02:01.469ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
224522024-02-15T03:02:01.469ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
224532024-02-15T03:02:01.469ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
224542024-02-15T03:02:01.469ZINFOcrucible: [0] client re-new 0 jobs since flush 7932 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
224552024-02-15T03:02:01.469ZINFOcrucible: client task is exiting = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
22456 Feb 15 03:02:02.467 INFO request completed, latency_us: 201, response_code: 200, uri: /state/cid/0, method: GET, req_id: 4eb2e5ba-ec6b-493a-b82a-79a2da636871, remote_addr: 127.0.0.1:54744, local_addr: 127.0.0.1:9998
22457 001/300 Read block 705 len 16384
22458 002/300 Read block 3330 len 8192
22459 003/300 Read block 2540 len 4096
224602024-02-15T03:02:02.470ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
224612024-02-15T03:02:02.470ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
224622024-02-15T03:02:02.470ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
224632024-02-15T03:02:02.470ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
224642024-02-15T03:02:02.470ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
224652024-02-15T03:02:02.470ZINFOcrucible: [0] client re-new 3 jobs since flush 7932 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
224662024-02-15T03:02:02.470ZINFOcrucible: client task is exiting = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
22467 004/300 Write block 4529 len 24576 data: 6 6 5 6 7 7
22468 005/300 Write block 2906 len 36864 data: 4 4 5 5 5 4 4 7 5
22469 006/300 Write block 2907 len 16384 data: 5 6 6 6
22470 007/300 Flush
22471 008/300 Read block 148 len 28672
22472 009/300 Read block 620 len 40960
22473 010/300 Write block 276 len 8192 data: 3 3
22474 011/300 Write block 2909 len 4096 data: 7
22475 012/300 Write block 1309 len 40960 data: 5 5 4 4 4 4 3 5 4 4
22476 013/300 Write block 3536 len 24576 data: 6 6 5 5 5 6
22477 014/300 Read block 3917 len 16384
22478 015/300 Read block 4733 len 12288
22479 016/300 Read block 3792 len 36864
22480 017/300 Flush
22481 018/300 Write block 3021 len 16384 data: 3 3 2 3
22482 019/300 Write block 4542 len 20480 data: 2 4 4 4 6
22483 020/300 Write block 2179 len 20480 data: 2 3 2 2 3
22484 021/300 Write block 77 len 4096 data: 7
22485 022/300 Read block 2893 len 8192
22486 023/300 Read block 2924 len 8192
22487 024/300 Read block 4117 len 24576
22488 025/300 Write block 2215 len 24576 data: 4 3 3 5 5 4
22489 026/300 Write block 710 len 24576 data: 3 5 5 7 6 6
22490 027/300 Read block 1448 len 28672
22491 028/300 Write block 1948 len 32768 data: 3 5 5 8 8 10 8 8
22492 029/300 Write block 216 len 4096 data: 4
22493 030/300 Read block 947 len 32768
22494 031/300 Read block 3018 len 8192
22495 032/300 Read block 2337 len 20480
22496 033/300 Write block 2168 len 8192 data: 5 5
22497 034/300 Read block 2950 len 36864
22498 035/300 Flush
22499 036/300 Write block 1790 len 32768 data: 3 3 3 4 4 5 5 5
22500 037/300 Read block 4385 len 36864
22501 038/300 Read block 3026 len 4096
22502 039/300 Read block 2756 len 28672
22503 040/300 Read block 2216 len 12288
22504 041/300 Write block 1113 len 24576 data: 4 5 6 6 6 5
22505 042/300 Read block 4620 len 12288
22506 043/300 Write block 4183 len 40960 data: 4 4 4 4 6 5 6 6 6 6
22507 044/300 Read block 316 len 28672
22508 045/300 Flush
22509 046/300 Flush
22510 047/300 Read block 1156 len 28672
22511 048/300 Flush
22512 049/300 Write block 4462 len 4096 data: 5
22513 050/300 Read block 1195 len 4096
22514 051/300 Write block 861 len 36864 data: 7 7 8 7 8 5 4 4 3
22515 052/300 Read block 4102 len 40960
22516 053/300 Write block 2725 len 20480 data: 4 4 3 3 4
22517 054/300 Read block 3 len 36864
22518 055/300 Read block 271 len 4096
22519 056/300 Flush
22520 057/300 Write block 2614 len 16384 data: 5 5 4 4
22521 058/300 Write block 2815 len 28672 data: 5 5 5 6 6 7 4
22522 059/300 Write block 2800 len 36864 data: 4 4 4 4 4 4 3 2 3
22523 060/300 Write block 4122 len 4096 data: 5
22524 061/300 Read block 3865 len 40960
22525 062/300 Read block 904 len 32768
22526 063/300 Write block 3739 len 40960 data: 3 4 4 5 5 5 5 4 4 4
22527 064/300 Flush
22528 065/300 Write block 217 len 4096 data: 3
22529 066/300 Write block 373 len 32768 data: 7 6 5 7 7 7 7 8
22530 067/300 Read block 2493 len 28672
22531 068/300 Write block 3813 len 40960 data: 4 5 5 4 4 4 4 4 5 5
22532 069/300 Write block 3598 len 20480 data: 7 7 7 6 4
22533 070/300 Read block 2740 len 8192
22534 071/300 Read block 358 len 4096
22535 072/300 Write block 1744 len 12288 data: 5 5 5
22536 073/300 Write block 1260 len 40960 data: 8 6 5 3 5 5 5 5 5 5
22537 074/300 Read block 2138 len 16384
22538 075/300 Flush
22539 076/300 Write block 2763 len 40960 data: 2 3 3 3 3 3 3 3 3 3
22540 077/300 Write block 1511 len 16384 data: 4 3 3 4
22541 078/300 Flush
22542 079/300 Write block 3538 len 32768 data: 6 6 6 7 6 7 6 6
22543 080/300 Read block 3519 len 32768
22544 081/300 Write block 4078 len 32768 data: 3 3 4 3 3 3 2 2
22545 082/300 Read block 4127 len 20480
22546 083/300 Read block 3766 len 16384
22547 084/300 Read block 1695 len 40960
22548 085/300 Read block 3212 len 12288
22549 086/300 Read block 4281 len 40960
22550 087/300 Read block 1904 len 12288
22551 088/300 Write block 2913 len 12288 data: 8 6 6
22552 089/300 Write block 2922 len 36864 data: 5 4 4 5 5 5 4 5 5
22553 090/300 Write block 3789 len 4096 data: 7
22554 091/300 Write block 593 len 8192 data: 4 4
22555 092/300 Read block 932 len 28672
22556 093/300 Read block 4332 len 4096
22557 094/300 Read block 3598 len 36864
22558 095/300 Read block 1170 len 20480
22559 096/300 Read block 2082 len 36864
22560 097/300 Write block 2906 len 24576 data: 5 6 7 8 7 5
22561 098/300 Read block 814 len 32768
22562 099/300 Flush
22563 100/300 Read block 4401 len 24576
22564 101/300 Read block 2542 len 4096
22565 102/300 Write block 4975 len 4096 data: 6
22566 103/300 Write block 532 len 20480 data: 5 6 7 7 8
22567 104/300 Read block 825 len 8192
22568 105/300 Read block 3315 len 8192
22569 106/300 Read block 3123 len 12288
22570 107/300 Read block 4130 len 12288
22571 108/300 Read block 453 len 24576
22572 109/300 Read block 1902 len 24576
22573 110/300 Flush
22574 111/300 Write block 2563 len 12288 data: 5 5 4
22575 112/300 Write block 1617 len 40960 data: 4 4 3 4 5 5 6 6 7 7
22576 113/300 Read block 516 len 12288
22577 114/300 Read block 35 len 24576
22578 115/300 Write block 3466 len 28672 data: 3 4 4 5 3 3 4
22579 116/300 Read block 3422 len 4096
22580 117/300 Write block 1668 len 28672 data: 5 5 4 4 3 4 5
22581 118/300 Read block 3213 len 12288
22582 119/300 Read block 655 len 4096
22583 120/300 Write block 4051 len 24576 data: 9 8 9 10 10 8
22584 121/300 Read block 338 len 24576
22585 122/300 Write block 3589 len 16384 data: 3 3 2 2
22586 123/300 Write block 449 len 20480 data: 5 6 6 8 6
22587 124/300 Read block 3346 len 8192
22588 125/300 Write block 1266 len 12288 data: 6 6 6
22589 126/300 Read block 3952 len 40960
22590 127/300 Write block 64 len 8192 data: 3 3
22591 128/300 Read block 2080 len 16384
22592 129/300 Read block 4260 len 16384
22593 130/300 Read block 4975 len 24576
22594 131/300 Read block 2309 len 12288
22595 132/300 Read block 923 len 24576
22596 133/300 Write block 2354 len 16384 data: 6 5 6 7
22597 134/300 Write block 1279 len 24576 data: 4 5 4 3 2 3
22598 135/300 Flush
22599 136/300 Write block 4007 len 32768 data: 4 4 6 4 5 7 7 9
22600 137/300 Read block 4684 len 36864
22601 138/300 Flush
22602 139/300 Read block 688 len 24576
22603 140/300 Write block 3735 len 32768 data: 7 5 4 3 4 5 5 6
22604 141/300 Read block 2823 len 24576
22605 142/300 Read block 2372 len 40960
22606 143/300 Read block 4794 len 12288
22607 144/300 Read block 1679 len 4096
22608 145/300 Read block 775 len 12288
22609 146/300 Read block 1592 len 28672
22610 147/300 Read block 3548 len 32768
22611 148/300 Read block 2371 len 28672
22612 149/300 Read block 4477 len 28672
22613 150/300 Write block 259 len 32768 data: 5 6 7 6 6 5 5 5
22614 151/300 Read block 473 len 16384
22615 152/300 Flush
22616 153/300 Write block 7 len 16384 data: 3 3 3 3
22617 154/300 Read block 3568 len 40960
22618 155/300 Read block 4623 len 36864
22619 156/300 Read block 1448 len 4096
22620 157/300 Flush
22621 158/300 Read block 2656 len 24576
22622 159/300 Write block 1391 len 24576 data: 4 4 4 4 5 7
22623 160/300 Read block 4152 len 20480
22624 161/300 Read block 3898 len 40960
22625 162/300 Write block 191 len 24576 data: 6 5 6 4 5 5
22626 163/300 Write block 149 len 40960 data: 5 4 3 3 4 6 5 5 6 6
22627 164/300 Read block 632 len 4096
22628 165/300 Read block 3421 len 32768
22629 166/300 Write block 985 len 4096 data: 4
22630 167/300 Read block 56 len 40960
22631 168/300 Read block 1089 len 32768
22632 169/300 Read block 1772 len 20480
22633 170/300 Flush
22634 171/300 Read block 2784 len 36864
22635 172/300 Write block 3879 len 28672 data: 5 7 7 7 7 6 5
22636 173/300 Write block 3290 len 12288 data: 7 6 5
22637 174/300 Read block 333 len 4096
22638 175/300 Write block 4454 len 36864 data: 4 5 5 5 5 3 4 4 6
22639 176/300 Write block 4553 len 4096 data: 6
22640 177/300 Read block 3149 len 12288
22641 178/300 Flush
22642 179/300 Write block 1825 len 40960 data: 7 7 8 8 7 6 5 4 4 4
22643 180/300 Read block 3757 len 32768
22644 181/300 Read block 3451 len 20480
22645 182/300 Write block 1281 len 36864 data: 5 4 3 4 5 6 6 7 7
22646 183/300 Flush
22647 184/300 Read block 3511 len 36864
22648 185/300 Flush
22649 186/300 Read block 171 len 24576
22650 187/300 Read block 4647 len 28672
22651 188/300 Flush
22652 189/300 Write block 4180 len 28672 data: 4 5 4 5 5 5 5
22653 190/300 Write block 1852 len 32768 data: 4 3 3 2 4 4 3 3
22654 191/300 Read block 2233 len 16384
22655 192/300 Read block 4508 len 4096
22656 193/300 Write block 4227 len 28672 data: 3 3 3 3 4 4 4
22657 194/300 Read block 2229 len 20480
22658 195/300 Write block 2798 len 28672 data: 3 3 5 5 5 5 5
22659 196/300 Read block 1383 len 40960
22660 197/300 Read block 1968 len 28672
22661 198/300 Read block 3217 len 28672
22662 199/300 Write block 4790 len 20480 data: 5 5 7 6 7
22663 200/300 Write block 1127 len 24576 data: 6 5 6 5 5 4
22664 201/300 Read block 1833 len 4096
22665 202/300 Read block 2353 len 28672
22666 203/300 Flush
22667 204/300 Write block 2947 len 16384 data: 3 4 4 4
22668 205/300 Read block 2220 len 24576
22669 206/300 Write block 2612 len 20480 data: 4 5 6 6 5
22670 207/300 Read block 2033 len 20480
22671 208/300 Read block 3433 len 40960
22672 209/300 Flush
22673 210/300 Read block 4840 len 40960
22674 211/300 Read block 3500 len 36864
22675 212/300 Read block 4910 len 24576
22676 213/300 Read block 2735 len 32768
22677 214/300 Write block 3975 len 36864 data: 4 4 4 4 4 5 6 5 6
22678 215/300 Read block 1316 len 20480
22679 216/300 Read block 3839 len 20480
22680 217/300 Read block 392 len 36864
22681 218/300 Read block 4083 len 12288
22682 219/300 Read block 4988 len 4096
22683 220/300 Write block 779 len 32768 data: 4 4 4 4 5 4 5 5
22684 221/300 Read block 169 len 24576
22685 222/300 Read block 3027 len 16384
22686 223/300 Read block 3306 len 8192
22687 224/300 Read block 3558 len 40960
22688 225/300 Read block 4165 len 4096
22689 226/300 Read block 4927 len 36864
22690 227/300 Read block 4818 len 24576
22691 228/300 Write block 1983 len 28672 data: 3 3 2 3 3 2 2
22692 229/300 Read block 4684 len 12288
22693 230/300 Read block 362 len 20480
22694 231/300 Read block 778 len 8192
22695 232/300 Write block 1810 len 8192 data: 7 6
22696 233/300 Write block 3816 len 4096 data: 5
22697 234/300 Write block 3112 len 20480 data: 8 7 7 6 6
22698 235/300 Write block 2581 len 20480 data: 4 5 4 5 5
22699 236/300 Read block 4314 len 36864
22700 237/300 Read block 3407 len 4096
22701 238/300 Read block 4283 len 20480
22702 239/300 Read block 769 len 32768
22703 240/300 Flush
22704 241/300 Write block 1216 len 32768 data: 4 4 6 8 7 8 7 6
22705 242/300 Write block 3094 len 40960 data: 3 4 4 4 5 5 5 6 3 3
22706 243/300 Read block 3758 len 20480
22707 244/300 Read block 2957 len 32768
22708 245/300 Read block 1167 len 28672
22709 246/300 Read block 4449 len 12288
22710 247/300 Read block 4342 len 40960
22711 248/300 Write block 667 len 36864 data: 4 3 4 6 6 6 7 7 4
22712 249/300 Write block 1178 len 16384 data: 5 7 5 5
22713 250/300 Read block 3356 len 12288
22714 251/300 Read block 1789 len 24576
22715 252/300 Write block 379 len 28672 data: 8 9 6 6 5 5 4
22716 253/300 Read block 1496 len 32768
22717 254/300 Write block 138 len 32768 data: 4 3 3 4 5 5 5 5
22718 255/300 Write block 2856 len 16384 data: 8 8 9 7
22719 256/300 Write block 1845 len 36864 data: 4 7 6 4 4 4 4 5 4
22720 257/300 Read block 4772 len 40960
22721 258/300 Read block 550 len 28672
22722 259/300 Read block 2995 len 20480
22723 260/300 Flush
22724 261/300 Read block 2573 len 24576
22725 262/300 Write block 3029 len 36864 data: 6 5 4 5 5 6 5 5 5
22726 263/300 Read block 2515 len 32768
22727 264/300 Write block 4040 len 16384 data: 6 5 7 7
22728 265/300 Read block 2127 len 36864
22729 266/300 Write block 3942 len 4096 data: 5
22730 267/300 Read block 3224 len 16384
22731 268/300 Read block 4944 len 8192
22732 269/300 Write block 2468 len 8192 data: 3 4
22733 270/300 Write block 3875 len 16384 data: 6 6 6 6
22734 271/300 Write block 350 len 20480 data: 8 7 8 11 11
22735 272/300 Read block 1087 len 20480
22736 273/300 Read block 3676 len 32768
22737 274/300 Read block 2787 len 40960
22738 275/300 Write block 4652 len 28672 data: 2 2 2 2 2 2 3
22739 276/300 Read block 2259 len 8192
22740 277/300 Read block 4723 len 16384
22741 278/300 Write block 91 len 40960 data: 5 4 4 4 5 4 5 5 4 2
22742 279/300 Read block 3972 len 36864
22743 280/300 Read block 1962 len 12288
22744 281/300 Read block 1070 len 12288
22745 282/300 Write block 1648 len 8192 data: 8 9
22746 283/300 Read block 3518 len 32768
22747 284/300 Flush
22748 285/300 Read block 3345 len 36864
22749 286/300 Write block 3675 len 8192 data: 7 7
22750 287/300 Read block 2321 len 20480
22751 288/300 Read block 1614 len 8192
22752 289/300 Write block 541 len 32768 data: 4 4 4 5 5 6 7 7
22753 290/300 Write block 2737 len 28672 data: 7 8 9 6 7 6 7
22754 291/300 Read block 1383 len 8192
22755 292/300 Write block 655 len 40960 data: 3 3 3 4 6 6 5 5 5 5
22756 293/300 Write block 4077 len 4096 data: 2
22757 294/300 Write block 257 len 32768 data: 5 6 6 7 8 7 7 6
22758 295/300 Write block 1508 len 32768 data: 7 4 4 5 4 4 5 5
22759 296/300 Flush
22760 297/300 Write block 4366 len 36864 data: 6 5 4 4 4 5 4 5 4
22761 298/300 Read block 1797 len 40960
22762 299/300 Read block 2551 len 8192
22763 300/300 Read block 3054 len 40960
22764 Main task has work to do, go find it
22765 got dsc Start(0)
22766 start 0
22767 Feb 15 03:02:02.812 INFO request completed, latency_us: 210, response_code: 204, uri: /start/cid/0, method: POST, req_id: 4aaa190e-c554-4a24-9ece-f70400565b74, remote_addr: 127.0.0.1:54744, local_addr: 127.0.0.1:9998
22768 Replay: started 0, returned:Ok(())
22769 ----------------------------------------------------------------
22770 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:300
22771 States: Offline Active Active
22772 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
22773 6934 Acked 7933 Read 4 New Done Done true
22774 6935 Acked 7934 Read 2 New Done Done true
22775 6936 Acked 7935 Read 1 New Done Done true
22776 6937 Acked 7936 Write 6 New Done Done false
22777 6938 Acked 7937 Write 9 New Done Done false
22778 6939 Acked 7938 Write 4 New Done Done false
22779 6940 Acked 7939 Flush 0 New Done Done false
22780 6941 Acked 7940 Read 7 New Done Done false
22781 6942 Acked 7941 Read 10 New Done Done false
22782 6943 Acked 7942 Write 2 New Done Done false
22783 6944 Acked 7943 Write 1 New Done Done false
22784 6945 Acked 7944 Write 10 New Done Done false
22785 6946 Acked 7945 Write 6 New Done Done false
22786 6947 Acked 7946 Read 4 New Done Done false
22787 6948 Acked 7947 Read 3 New Done Done false
22788 6949 Acked 7948 Read 9 New Done Done false
22789 6950 Acked 7949 Flush 0 New Done Done false
22790 6951 Acked 7950 Write 4 New Done Done false
22791 6952 Acked 7951 Write 5 New Done Done false
22792 6953 Acked 7952 Write 5 New Done Done false
22793 6954 Acked 7953 Write 1 New Done Done false
22794 6955 Acked 7954 Read 2 New Done Done false
22795 6956 Acked 7955 Read 2 New Done Done false
22796 6957 Acked 7956 Read 6 New Done Done false
22797 6958 Acked 7957 Write 6 New Done Done false
22798 6959 Acked 7958 Write 6 New Done Done false
22799 6960 Acked 7959 Read 7 New Done Done false
22800 6961 Acked 7960 Write 8 New Done Done false
22801 6962 Acked 7961 Write 1 New Done Done false
22802 6963 Acked 7962 Read 8 New Done Done false
22803 6964 Acked 7963 Read 2 New Done Done false
22804 6965 Acked 7964 Read 5 New Done Done false
22805 6966 Acked 7965 Write 2 New Done Done false
22806 6967 Acked 7966 Read 9 New Done Done false
22807 6968 Acked 7967 Flush 0 New Done Done false
22808 6969 Acked 7968 Write 8 New Done Done false
22809 6970 Acked 7969 Read 9 New Done Done false
22810 6971 Acked 7970 Read 1 New Done Done false
22811 6972 Acked 7971 Read 7 New Done Done false
22812 6973 Acked 7972 Read 3 New Done Done false
22813 6974 Acked 7973 Write 6 New Done Done false
22814 6975 Acked 7974 Read 3 New Done Done false
22815 6976 Acked 7975 Write 10 New Done Done false
22816 6977 Acked 7976 Read 7 New Done Done false
22817 6978 Acked 7977 Flush 0 New Done Done false
22818 6979 Acked 7978 Flush 0 New Done Done false
22819 6980 Acked 7979 Read 7 New Done Done false
22820 6981 Acked 7980 Flush 0 New Done Done false
22821 6982 Acked 7981 Write 1 New Done Done false
22822 6983 Acked 7982 Read 1 New Done Done false
22823 6984 Acked 7983 Write 9 New Done Done false
22824 6985 Acked 7984 Read 10 New Done Done false
22825 6986 Acked 7985 Write 5 New Done Done false
22826 6987 Acked 7986 Read 9 New Done Done false
22827 6988 Acked 7987 Read 1 New Done Done false
22828 6989 Acked 7988 Flush 0 New Done Done false
22829 6990 Acked 7989 Write 4 New Done Done false
22830 6991 Acked 7990 Write 7 New Done Done false
22831 6992 Acked 7991 Write 9 New Done Done false
22832 6993 Acked 7992 Write 1 New Done Done false
22833 6994 Acked 7993 Read 10 New Done Done false
22834 6995 Acked 7994 Read 8 New Done Done false
22835 6996 Acked 7995 Write 10 New Done Done false
22836 6997 Acked 7996 Flush 0 New Done Done false
22837 6998 Acked 7997 Write 1 New Done Done false
22838 6999 Acked 7998 Write 8 New Done Done false
22839 7000 Acked 7999 Read 7 New Done Done false
22840 7001 Acked 8000 Write 10 New Done Done false
22841 7002 Acked 8001 Write 5 New Done Done false
22842 7003 Acked 8002 Read 2 New Done Done false
22843 7004 Acked 8003 Read 1 New Done Done false
22844 7005 Acked 8004 Write 3 New Done Done false
22845 7006 Acked 8005 Write 10 New Done Done false
22846 7007 Acked 8006 Read 4 New Done Done false
22847 7008 Acked 8007 Flush 0 New Done Done false
22848 7009 Acked 8008 Write 10 New Done Done false
22849 7010 Acked 8009 Write 4 New Done Done false
22850 7011 Acked 8010 Flush 0 New Done Done false
22851 7012 Acked 8011 Write 8 New Done Done false
22852 7013 Acked 8012 Read 8 New Done Done false
22853 7014 Acked 8013 Write 8 New Done Done false
22854 7015 Acked 8014 Read 5 New Done Done false
22855 7016 Acked 8015 Read 4 New Done Done false
22856 7017 Acked 8016 Read 10 New Done Done false
22857 7018 Acked 8017 Read 3 New Done Done false
22858 7019 Acked 8018 Read 10 New Done Done false
22859 7020 Acked 8019 Read 3 New Done Done false
22860 7021 Acked 8020 Write 3 New Done Done false
22861 7022 Acked 8021 Write 9 New Done Done false
22862 7023 Acked 8022 Write 1 New Done Done false
22863 7024 Acked 8023 Write 2 New Done Done false
22864 7025 Acked 8024 Read 7 New Done Done false
22865 7026 Acked 8025 Read 1 New Done Done false
22866 7027 Acked 8026 Read 9 New Done Done false
22867 7028 Acked 8027 Read 5 New Done Done false
22868 7029 Acked 8028 Read 9 New Done Done false
22869 7030 Acked 8029 Write 6 New Done Done false
22870 7031 Acked 8030 Read 8 New Done Done false
22871 7032 Acked 8031 Flush 0 New Done Done false
22872 7033 Acked 8032 Read 6 New Done Done false
22873 7034 Acked 8033 Read 1 New Done Done false
22874 7035 Acked 8034 Write 1 New Done Done false
22875 7036 Acked 8035 Write 5 New Done Done false
22876 7037 Acked 8036 Read 2 New Done Done false
22877 7038 Acked 8037 Read 2 New Done Done false
22878 7039 Acked 8038 Read 3 New Done Done false
22879 7040 Acked 8039 Read 3 New Done Done false
22880 7041 Acked 8040 Read 6 New Done Done false
22881 7042 Acked 8041 Read 6 New Done Done false
22882 7043 Acked 8042 Flush 0 New Done Done false
22883 7044 Acked 8043 Write 3 New Done Done false
22884 7045 Acked 8044 Write 10 New Done Done false
22885 7046 Acked 8045 Read 3 New Done Done false
22886 7047 Acked 8046 Read 6 New Done Done false
22887 7048 Acked 8047 Write 7 New Done Done false
22888 7049 Acked 8048 Read 1 New Done Done false
22889 7050 Acked 8049 Write 7 New Done Done false
22890 7051 Acked 8050 Read 3 New Done Done false
22891 7052 Acked 8051 Read 1 New Done Done false
22892 7053 Acked 8052 Write 6 New Done Done false
22893 7054 Acked 8053 Read 6 New Done Done false
22894 7055 Acked 8054 Write 4 New Done Done false
22895 7056 Acked 8055 Write 5 New Done Done false
22896 7057 Acked 8056 Read 2 New Done Done false
22897 7058 Acked 8057 Write 3 New Done Done false
22898 7059 Acked 8058 Read 10 New Done Done false
22899 7060 Acked 8059 Write 2 New Done Done false
22900 7061 Acked 8060 Read 4 New Done Done false
22901 7062 Acked 8061 Read 4 New Done Done false
22902 7063 Acked 8062 Read 6 New Done Done false
22903 7064 Acked 8063 Read 3 New Done Done false
22904 7065 Acked 8064 Read 6 New Done Done false
22905 7066 Acked 8065 Write 4 New Done Done false
22906 7067 Acked 8066 Write 6 New Done Done false
22907 7068 Acked 8067 Flush 0 New Done Done false
22908 7069 Acked 8068 Write 8 New Done Done false
22909 7070 Acked 8069 Read 9 New Done Done false
22910 7071 Acked 8070 Flush 0 New Done Done false
22911 7072 Acked 8071 Read 6 New Done Done false
22912 7073 Acked 8072 Write 8 New Done Done false
22913 7074 Acked 8073 Read 6 New Done Done false
22914 7075 Acked 8074 Read 10 New Done Done false
22915 7076 Acked 8075 Read 3 New Done Done false
22916 7077 Acked 8076 Read 1 New Done Done false
22917 7078 Acked 8077 Read 3 New Done Done false
22918 7079 Acked 8078 Read 7 New Done Done false
22919 7080 Acked 8079 Read 8 New Done Done false
22920 7081 Acked 8080 Read 7 New Done Done false
22921 7082 Acked 8081 Read 7 New Done Done false
22922 7083 Acked 8082 Write 8 New Done Done false
22923 7084 Acked 8083 Read 4 New Done Done false
22924 7085 Acked 8084 Flush 0 New Done Done false
22925 7086 Acked 8085 Write 4 New Done Done false
22926 7087 Acked 8086 Read 10 New Done Done false
22927 7088 Acked 8087 Read 9 New Done Done false
22928 7089 Acked 8088 Read 1 New Done Done false
22929 7090 Acked 8089 Flush 0 New Done Done false
22930 7091 Acked 8090 Read 6 New Done Done false
22931 7092 Acked 8091 Write 6 New Done Done false
22932 7093 Acked 8092 Read 5 New Done Done false
22933 7094 Acked 8093 Read 10 New Done Done false
22934 7095 Acked 8094 Write 6 New Done Done false
22935 7096 Acked 8095 Write 10 New Done Done false
22936 7097 Acked 8096 Read 1 New Done Done false
22937 7098 Acked 8097 Read 8 New Done Done false
22938 7099 Acked 8098 Write 1 New Done Done false
22939 7100 Acked 8099 Read 10 New Done Done false
22940 7101 Acked 8100 Read 8 New Done Done false
22941 7102 Acked 8101 Read 5 New Done Done false
22942 7103 Acked 8102 Flush 0 New Done Done false
22943 7104 Acked 8103 Read 9 New Done Done false
22944 7105 Acked 8104 Write 7 New Done Done false
22945 7106 Acked 8105 Write 3 New Done Done false
22946 7107 Acked 8106 Read 1 New Done Done false
22947 7108 Acked 8107 Write 9 New Done Done false
22948 7109 Acked 8108 Write 1 New Done Done false
22949 7110 Acked 8109 Read 3 New Done Done false
22950 7111 Acked 8110 Flush 0 New Done Done false
22951 7112 Acked 8111 Write 10 New Done Done false
22952 7113 Acked 8112 Read 8 New Done Done false
22953 7114 Acked 8113 Read 5 New Done Done false
22954 7115 Acked 8114 Write 9 New Done Done false
22955 7116 Acked 8115 Flush 0 New Done Done false
22956 7117 Acked 8116 Read 9 New Done Done false
22957 7118 Acked 8117 Flush 0 New Done Done false
22958 7119 Acked 8118 Read 6 New Done Done false
22959 7120 Acked 8119 Read 7 New Done Done false
22960 7121 Acked 8120 Flush 0 New Done Done false
22961 7122 Acked 8121 Write 7 New Done Done false
22962 7123 Acked 8122 Write 8 New Done Done false
22963 7124 Acked 8123 Read 4 New Done Done false
22964 7125 Acked 8124 Read 1 New Done Done false
22965 7126 Acked 8125 Write 7 New Done Done false
22966 7127 Acked 8126 Read 5 New Done Done false
22967 7128 Acked 8127 Write 7 New Done Done false
22968 7129 Acked 8128 Read 10 New Done Done false
22969 7130 Acked 8129 Read 7 New Done Done false
22970 7131 Acked 8130 Read 7 New Done Done false
22971 7132 Acked 8131 Write 5 New Done Done false
22972 7133 Acked 8132 Write 6 New Done Done false
22973 7134 Acked 8133 Read 1 New Done Done false
22974 7135 Acked 8134 Read 7 New Done Done false
22975 7136 Acked 8135 Flush 0 New Done Done false
22976 7137 Acked 8136 Write 4 New Done Done false
22977 7138 Acked 8137 Read 6 New Done Done false
22978 7139 Acked 8138 Write 5 New Done Done false
22979 7140 Acked 8139 Read 5 New Done Done false
22980 7141 Acked 8140 Read 10 New Done Done false
22981 7142 Acked 8141 Flush 0 New Done Done false
22982 7143 Acked 8142 Read 10 New Done Done false
22983 7144 Acked 8143 Read 9 New Done Done false
22984 7145 Acked 8144 Read 6 New Done Done false
22985 7146 Acked 8145 Read 8 New Done Done false
22986 7147 Acked 8146 Write 9 New Done Done false
22987 7148 Acked 8147 Read 5 New Done Done false
22988 7149 Acked 8148 Read 5 New Done Done false
22989 7150 Acked 8149 Read 9 New Done Done false
22990 7151 Acked 8150 Read 3 New Done Done false
22991 7152 Acked 8151 Read 1 New Done Done false
22992 7153 Acked 8152 Write 8 New Done Done false
22993 7154 Acked 8153 Read 6 New Done Done false
22994 7155 Acked 8154 Read 4 New Done Done false
22995 7156 Acked 8155 Read 2 New Done Done false
22996 7157 Acked 8156 Read 10 New Done Done false
22997 7158 Acked 8157 Read 1 New Done Done false
22998 7159 Acked 8158 Read 9 New Done Done false
22999 7160 Acked 8159 Read 6 New Done Done false
23000 7161 Acked 8160 Write 7 New Done Done false
23001 7162 Acked 8161 Read 3 New Done Done false
23002 7163 Acked 8162 Read 5 New Done Done false
23003 7164 Acked 8163 Read 2 New Done Done false
23004 7165 Acked 8164 Write 2 New Done Done false
23005 7166 Acked 8165 Write 1 New Done Done false
23006 7167 Acked 8166 Write 5 New Done Done false
23007 7168 Acked 8167 Write 5 New Done Done false
23008 7169 Acked 8168 Read 9 New Done Done false
23009 7170 Acked 8169 Read 1 New Done Done false
23010 7171 Acked 8170 Read 5 New Done Done false
23011 7172 Acked 8171 Read 8 New Done Done false
23012 7173 Acked 8172 Flush 0 New Done Done false
23013 7174 Acked 8173 Write 8 New Done Done false
23014 7175 Acked 8174 Write 10 New Done Done false
23015 7176 Acked 8175 Read 5 New Done Done false
23016 7177 Acked 8176 Read 8 New Done Done false
23017 7178 Acked 8177 Read 7 New Done Done false
23018 7179 Acked 8178 Read 3 New Done Done false
23019 7180 Acked 8179 Read 10 New Done Done false
23020 7181 Acked 8180 Write 9 New Done Done false
23021 7182 Acked 8181 Write 4 New Done Done false
23022 7183 Acked 8182 Read 3 New Done Done false
23023 7184 Acked 8183 Read 6 New Done Done false
23024 7185 Acked 8184 Write 7 New Done Done false
23025 7186 Acked 8185 Read 8 New Done Done false
23026 7187 Acked 8186 Write 8 New Done Done false
23027 7188 Acked 8187 Write 4 New Done Done false
23028 7189 Acked 8188 Write 9 New Done Done false
23029 7190 Acked 8189 Read 10 New Done Done false
23030 7191 Acked 8190 Read 7 New Done Done false
23031 7192 Acked 8191 Read 5 New Done Done false
23032 7193 Acked 8192 Flush 0 New Done Done false
23033 7194 Acked 8193 Read 6 New Done Done false
23034 7195 Acked 8194 Write 9 New Done Done false
23035 7196 Acked 8195 Read 8 New Done Done false
23036 7197 Acked 8196 Write 4 New Done Done false
23037 7198 Acked 8197 Read 9 New Done Done false
23038 7199 Acked 8198 Write 1 New Done Done false
23039 7200 Acked 8199 Read 4 New Done Done false
23040 7201 Acked 8200 Read 2 New Done Done false
23041 7202 Acked 8201 Write 2 New Done Done false
23042 7203 Acked 8202 Write 4 New Done Done false
23043 7204 Acked 8203 Write 5 New Done Done false
23044 7205 Acked 8204 Read 5 New Done Done false
23045 7206 Acked 8205 Read 8 New Done Done false
23046 7207 Acked 8206 Read 10 New Done Done false
23047 7208 Acked 8207 Write 7 New Done Done false
23048 7209 Acked 8208 Read 2 New Done Done false
23049 7210 Acked 8209 Read 4 New Done Done false
23050 7211 Acked 8210 Write 10 New Done Done false
23051 7212 Acked 8211 Read 9 New Done Done false
23052 7213 Acked 8212 Read 3 New Done Done false
23053 7214 Acked 8213 Read 3 New Done Done false
23054 7215 Acked 8214 Write 2 New Done Done false
23055 7216 Acked 8215 Read 8 New Done Done false
23056 7217 Acked 8216 Flush 0 New Done Done false
23057 7218 Acked 8217 Read 9 New Done Done false
23058 7219 Acked 8218 Write 2 New Done Done false
23059 7220 Acked 8219 Read 5 New Done Done false
23060 7221 Acked 8220 Read 2 New Done Done false
23061 7222 Acked 8221 Write 8 New Done Done false
23062 7223 Acked 8222 Write 7 New Done Done false
23063 7224 Acked 8223 Read 2 New Done Done false
23064 7225 Acked 8224 Write 10 New Done Done false
23065 7226 Acked 8225 Write 1 New Done Done false
23066 7227 Acked 8226 Write 8 New Done Done false
23067 7228 Acked 8227 Write 8 New Done Done false
23068 7229 Acked 8228 Flush 0 New Done Done false
23069 7230 Acked 8229 Write 9 New Done Done false
23070 7231 Acked 8230 Read 10 New Done Done false
23071 7232 Acked 8231 Read 2 New Done Done false
23072 7233 Acked 8232 Read 10 New Done Done false
23073 STATES DS:0 DS:1 DS:2 TOTAL
23074 New 300 0 0 300
23075 Sent 0 0 0 0
23076 Done 0 300 300 600
23077 Skipped 0 0 0 0
23078 Error 0 0 0 0
23079 Last Flush: 7932 8228 8228
23080 Downstairs last five completed: 7932 7931 7930 7929 7928
23081 Upstairs last five completed: 7233 7232 7231 7230 7229
23082 CLIENT: Up:0 ds:300 act:2
230832024-02-15T03:02:03.471ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
230842024-02-15T03:02:03.471ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
230852024-02-15T03:02:03.471ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
230862024-02-15T03:02:03.471ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
230872024-02-15T03:02:03.471ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
230882024-02-15T03:02:03.471ZINFOcrucible: [0] client re-new 301 jobs since flush 7932 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
230892024-02-15T03:02:03.471ZINFOcrucible: client task is exiting = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
230902024-02-15T03:02:04.472ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
230912024-02-15T03:02:04.472ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
230922024-02-15T03:02:04.472ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
230932024-02-15T03:02:04.472ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
230942024-02-15T03:02:04.472ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
230952024-02-15T03:02:04.472ZINFOcrucible: [0] client re-new 301 jobs since flush 7932 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
230962024-02-15T03:02:04.473ZINFOcrucible: client task is exiting = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
23097 [8810] got start action
23098 [8810] I am going to restart
23099 Starting downstairs at port 8810
23100 Make output file at "/tmp/dsc/downstairs-8810.txt"
23101 [8810][0] reports Starting
23102 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1191)
23103 [8810][0] reports Running
231042024-02-15T03:02:05.474ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
231052024-02-15T03:02:05.474ZINFOcrucible: ds_connection connected = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
231062024-02-15T03:02:05.475ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
231072024-02-15T03:02:05.475ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
231082024-02-15T03:02:05.475ZINFOcrucible: send last flush ID to this DS: 7932 = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
231092024-02-15T03:02:05.475ZINFOcrucible: Replied this last flush ID: 7932 = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
231102024-02-15T03:02:05.476ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
231112024-02-15T03:02:05.476ZINFOcrucible: [0] Transition from Offline to Replay = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
231122024-02-15T03:02:05.476ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
231132024-02-15T03:02:05.476ZINFOcrucible: [0] Transition from Replay to Active = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
23114 ----------------------------------------------------------------
23115 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
23116 Downstairs last five completed: 8233 8232 8231 8230 8229
23117 Upstairs last five completed: 7234 7233 7232 7231 7230
23118 CLIENT: Up:0 ds:0 act:3
23119 Replay: All jobs finished, all DS active.
23120 Main task has work to do, go find it
23121 got dsc Stop(0)
23122 stop 0
23123 [8810] Got stop action so:false kr:false
23124 Feb 15 03:02:06.816 INFO request completed, latency_us: 354, response_code: 204, uri: /stop/cid/0, method: POST, req_id: 059630f3-72a9-471c-b57e-1e0ca01d0705, remote_addr: 127.0.0.1:54744, local_addr: 127.0.0.1:9998
231252024-02-15T03:02:06.817ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
231262024-02-15T03:02:06.817ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
231272024-02-15T03:02:06.817ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
231282024-02-15T03:02:06.817ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
23129 Feb 15 03:02:06.817 INFO request completed, latency_us: {"msg":"[0] client re-new 0 jobs since flush 8233","v":0,"name":"crucible","level":30,"time":"2024-02-15T03:02:06.81743413Z","hostname":"ip-10-150-1-67.us-west-2.compute.internal","pid":1167,"":"downstairs","":"downstairs","session_id":"4f5a63e5-aa00-4890-9f6b-96c2e069fdb2"}
23130 140, response_code: 200, uri: /state/cid/0, method: GET, req_id: {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-15T03:02:06.817476175Z","hostname":"ip-10-150-1-67.us-west-2.compute.internal","pid":1167,"":"io task","client":"0","":"downstairs","session_id":"4f5a63e5-aa00-4890-9f6b-96c2e069fdb2"}
23131 b18e11cb-cc26-47ed-aa41-6b71b39f077b, remote_addr: 127.0.0.1:54744, local_addr: 127.0.0.1:9998
23132 [8810] Exited with: ExitStatus(unix_wait_status(9))
23133 [8810][0] reports Exit
231342024-02-15T03:02:07.819ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
231352024-02-15T03:02:07.819ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
231362024-02-15T03:02:07.819ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
231372024-02-15T03:02:07.819ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
231382024-02-15T03:02:07.819ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
231392024-02-15T03:02:07.819ZINFOcrucible: [0] client re-new 0 jobs since flush 8233 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
231402024-02-15T03:02:07.819ZINFOcrucible: client task is exiting = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
231412024-02-15T03:02:08.821ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
231422024-02-15T03:02:08.821ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
231432024-02-15T03:02:08.821ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
231442024-02-15T03:02:08.821ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
231452024-02-15T03:02:08.821ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
231462024-02-15T03:02:08.821ZINFOcrucible: [0] client re-new 0 jobs since flush 8233 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
231472024-02-15T03:02:08.821ZINFOcrucible: client task is exiting = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
231482024-02-15T03:02:09.823ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
231492024-02-15T03:02:09.823ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
231502024-02-15T03:02:09.823ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
231512024-02-15T03:02:09.823ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
231522024-02-15T03:02:09.823ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
231532024-02-15T03:02:09.823ZINFOcrucible: [0] client re-new 0 jobs since flush 8233 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
231542024-02-15T03:02:09.823ZINFOcrucible: client task is exiting = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
23155 Feb 15 03:02:10.819 INFO request completed, latency_us: 199, response_code: 200, uri: /state/cid/0, method: GET, req_id: ce3def51-2a33-45d8-a2de-34ef012ba7f3, remote_addr: 127.0.0.1:54744, local_addr: 127.0.0.1:9998
23156 001/300 Read block 3849 len 16384
23157 002/300 Write block 343 len 40960 data: 5 4 4 4 4 6 5 9 8 9
23158 003/300 Read block 2892 len 32768
231592024-02-15T03:02:10.824ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
231602024-02-15T03:02:10.824ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
231612024-02-15T03:02:10.824ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
231622024-02-15T03:02:10.825ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
231632024-02-15T03:02:10.825ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
231642024-02-15T03:02:10.825ZINFOcrucible: [0] client re-new 3 jobs since flush 8233 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
231652024-02-15T03:02:10.825ZINFOcrucible: client task is exiting = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
23166 004/300 Write block 4296 len 40960 data: 5 4 4 3 2 2 2 3 3 4
23167 005/300 Write block 1468 len 36864 data: 4 4 5 6 6 5 4 4 3
23168 006/300 Write block 3684 len 8192 data: 4 4
23169 007/300 Write block 674 len 8192 data: 8 5
23170 008/300 Read block 2878 len 12288
23171 009/300 Read block 1207 len 32768
23172 010/300 Write block 4311 len 8192 data: 9 8
23173 011/300 Flush
23174 012/300 Write block 2958 len 20480 data: 6 6 6 4 5
23175 013/300 Read block 898 len 4096
23176 014/300 Flush
23177 015/300 Write block 1100 len 20480 data: 4 4 4 4 4
23178 016/300 Read block 398 len 32768
23179 017/300 Read block 1153 len 24576
23180 018/300 Read block 1778 len 16384
23181 019/300 Read block 4532 len 20480
23182 020/300 Read block 4677 len 24576
23183 021/300 Write block 1955 len 16384 data: 9 8 7 7
23184 022/300 Read block 4189 len 12288
23185 023/300 Read block 1493 len 20480
23186 024/300 Read block 3015 len 16384
23187 025/300 Read block 261 len 20480
23188 026/300 Read block 1366 len 32768
23189 027/300 Read block 1522 len 24576
23190 028/300 Read block 1166 len 40960
23191 029/300 Write block 545 len 16384 data: 6 7 8 8
23192 030/300 Read block 541 len 4096
23193 031/300 Read block 591 len 36864
23194 032/300 Write block 4199 len 12288 data: 2 3 3
23195 033/300 Read block 3554 len 28672
23196 034/300 Read block 2901 len 20480
23197 035/300 Read block 4492 len 12288
23198 036/300 Write block 1421 len 12288 data: 7 6 5
23199 037/300 Write block 3209 len 36864 data: 5 5 5 7 9 8 8 7 7
23200 038/300 Read block 2304 len 32768
23201 039/300 Read block 1525 len 40960
23202 040/300 Read block 2759 len 32768
23203 041/300 Write block 3851 len 40960 data: 4 4 4 5 5 7 8 7 8 9
23204 042/300 Flush
23205 043/300 Read block 3314 len 12288
23206 044/300 Flush
23207 045/300 Write block 1286 len 12288 data: 7 7 8
23208 046/300 Read block 2986 len 4096
23209 047/300 Read block 2959 len 24576
23210 048/300 Read block 402 len 16384
23211 049/300 Write block 4909 len 8192 data: 7 8
23212 050/300 Read block 494 len 20480
23213 051/300 Write block 236 len 28672 data: 8 7 6 5 5 4 4
23214 052/300 Read block 3195 len 8192
23215 053/300 Read block 3802 len 20480
23216 054/300 Read block 4304 len 32768
23217 055/300 Read block 3722 len 40960
23218 056/300 Flush
23219 057/300 Read block 1347 len 4096
23220 058/300 Read block 1945 len 32768
23221 059/300 Read block 1706 len 4096
23222 060/300 Read block 24 len 28672
23223 061/300 Write block 3710 len 36864 data: 5 4 4 4 5 6 8 6 6
23224 062/300 Read block 2748 len 12288
23225 063/300 Read block 3244 len 16384
23226 064/300 Read block 2442 len 20480
23227 065/300 Read block 3759 len 36864
23228 066/300 Write block 4008 len 36864 data: 5 7 5 6 8 8 10 10 8
23229 067/300 Read block 1104 len 16384
23230 068/300 Write block 4544 len 16384 data: 5 5 7 6
23231 069/300 Write block 95 len 32768 data: 6 5 6 6 5 3 3 4
23232 070/300 Read block 3299 len 40960
23233 071/300 Write block 3545 len 36864 data: 7 6 6 4 3 3 3 3 3
23234 072/300 Write block 119 len 4096 data: 2
23235 073/300 Write block 4311 len 4096 data: 10
23236 074/300 Write block 1853 len 40960 data: 5 4 3 5 5 4 4 2 3 2
23237 075/300 Write block 4902 len 24576 data: 4 2 4 5 6 9
23238 076/300 Write block 1227 len 4096 data: 4
23239 077/300 Write block 563 len 32768 data: 4 5 6 7 6 5 6 7
23240 078/300 Write block 3634 len 12288 data: 3 4 3
23241 079/300 Write block 4107 len 20480 data: 4 5 6 6 9
23242 080/300 Read block 249 len 32768
23243 081/300 Write block 1892 len 4096 data: 4
23244 082/300 Read block 1216 len 36864
23245 083/300 Read block 2748 len 32768
23246 084/300 Read block 4133 len 4096
23247 085/300 Write block 472 len 20480 data: 5 5 4 4 5
23248 086/300 Read block 2440 len 4096
23249 087/300 Write block 2190 len 4096 data: 7
23250 088/300 Read block 205 len 40960
23251 089/300 Read block 4544 len 24576
23252 090/300 Write block 942 len 24576 data: 5 4 4 3 4 4
23253 091/300 Write block 4200 len 20480 data: 4 4 3 6 6
23254 092/300 Write block 4546 len 12288 data: 8 7 7
23255 093/300 Read block 1777 len 12288
23256 094/300 Write block 2649 len 8192 data: 5 4
23257 095/300 Flush
23258 096/300 Read block 2709 len 16384
23259 097/300 Read block 4292 len 20480
23260 098/300 Read block 4766 len 24576
23261 099/300 Read block 4125 len 28672
23262 100/300 Read block 1550 len 8192
23263 101/300 Write block 1919 len 8192 data: 3 2
23264 102/300 Read block 608 len 16384
23265 103/300 Read block 4578 len 4096
23266 104/300 Read block 172 len 24576
23267 105/300 Read block 2658 len 20480
23268 106/300 Flush
23269 107/300 Read block 1439 len 16384
23270 108/300 Read block 2834 len 8192
23271 109/300 Read block 140 len 8192
23272 110/300 Read block 2250 len 32768
23273 111/300 Read block 4856 len 4096
23274 112/300 Read block 4955 len 12288
23275 113/300 Flush
23276 114/300 Read block 2711 len 12288
23277 115/300 Read block 1756 len 8192
23278 116/300 Read block 3384 len 20480
23279 117/300 Flush
23280 118/300 Read block 236 len 16384
23281 119/300 Read block 896 len 32768
23282 120/300 Write block 3545 len 8192 data: 8 7
23283 121/300 Read block 2302 len 24576
23284 122/300 Flush
23285 123/300 Read block 3264 len 8192
23286 124/300 Read block 2513 len 12288
23287 125/300 Write block 1252 len 4096 data: 7
23288 126/300 Read block 3630 len 16384
23289 127/300 Flush
23290 128/300 Write block 2851 len 20480 data: 6 6 7 8 8
23291 129/300 Write block 3430 len 4096 data: 5
23292 130/300 Read block 1000 len 4096
23293 131/300 Read block 2225 len 8192
23294 132/300 Read block 4699 len 8192
23295 133/300 Read block 3541 len 16384
23296 134/300 Read block 588 len 24576
23297 135/300 Read block 2391 len 28672
23298 136/300 Read block 772 len 24576
23299 137/300 Write block 3604 len 28672 data: 6 5 4 6 5 5 6
23300 138/300 Read block 3378 len 20480
23301 139/300 Read block 1913 len 16384
23302 140/300 Flush
23303 141/300 Write block 1248 len 4096 data: 8
23304 142/300 Flush
23305 143/300 Write block 3229 len 16384 data: 4 4 3 3
23306 144/300 Write block 1435 len 16384 data: 4 3 4 4
23307 145/300 Read block 3847 len 36864
23308 146/300 Write block 3499 len 28672 data: 5 4 4 4 4 6 7
23309 147/300 Write block 70 len 12288 data: 6 6 8
23310 148/300 Read block 4571 len 8192
23311 149/300 Write block 1599 len 24576 data: 3 3 2 2 4 6
23312 150/300 Write block 4830 len 32768 data: 6 5 4 4 6 6 6 7
23313 151/300 Write block 32 len 8192 data: 2 3
23314 152/300 Write block 4291 len 24576 data: 3 3 3 3 5 6
23315 153/300 Read block 4009 len 4096
23316 154/300 Flush
23317 155/300 Write block 3525 len 36864 data: 8 8 6 7 8 7 6 6 9
23318 156/300 Write block 946 len 40960 data: 5 5 4 3 4 5 4 4 5 5
23319 157/300 Read block 2046 len 8192
23320 158/300 Read block 2956 len 24576
23321 159/300 Write block 2612 len 24576 data: 5 6 7 7 6 5
23322 160/300 Read block 2573 len 32768
23323 161/300 Read block 4487 len 28672
23324 162/300 Write block 1251 len 12288 data: 8 8 6
23325 163/300 Read block 2189 len 24576
23326 164/300 Flush
23327 165/300 Write block 3854 len 4096 data: 6
23328 166/300 Read block 207 len 32768
23329 167/300 Read block 369 len 40960
23330 168/300 Write block 1209 len 24576 data: 3 4 4 3 3 4
23331 169/300 Read block 899 len 8192
23332 170/300 Write block 160 len 16384 data: 7 8 8 7
23333 171/300 Write block 4675 len 12288 data: 10 9 6
23334 172/300 Write block 3940 len 36864 data: 7 7 6 5 4 6 6 7 9
23335 173/300 Flush
23336 174/300 Read block 356 len 40960
23337 175/300 Write block 2723 len 8192 data: 3 3
23338 176/300 Read block 3393 len 8192
23339 177/300 Read block 212 len 4096
23340 178/300 Read block 2592 len 16384
23341 179/300 Read block 871 len 32768
23342 180/300 Write block 3147 len 20480 data: 6 6 8 9 9
23343 181/300 Read block 4720 len 16384
23344 182/300 Write block 2205 len 4096 data: 7
23345 183/300 Write block 1755 len 24576 data: 3 4 4 5 5 7
23346 184/300 Read block 4969 len 8192
23347 185/300 Read block 933 len 36864
23348 186/300 Write block 3620 len 16384 data: 10 8 7 9
23349 187/300 Read block 1290 len 36864
23350 188/300 Write block 4864 len 12288 data: 9 9 7
23351 189/300 Read block 4437 len 16384
23352 190/300 Write block 3190 len 24576 data: 8 6 5 4 4 4
23353 191/300 Write block 877 len 28672 data: 3 3 3 2 3 3 4
23354 192/300 Write block 2381 len 12288 data: 4 4 4
23355 193/300 Write block 2814 len 28672 data: 5 6 6 6 7 7 8
23356 194/300 Write block 4653 len 32768 data: 3 3 3 3 3 4 4 4
23357 195/300 Write block 3358 len 32768 data: 6 5 5 5 6 4 4 2
23358 196/300 Write block 351 len 28672 data: 9 10 12 12 9 7 8
23359 197/300 Write block 774 len 24576 data: 4 4 5 5 5 5
23360 198/300 Read block 1308 len 16384
23361 199/300 Write block 4873 len 16384 data: 5 3 4 4
23362 200/300 Write block 557 len 24576 data: 5 7 7 6 6 3
23363 201/300 Read block 3348 len 8192
23364 202/300 Write block 696 len 36864 data: 5 4 4 4 4 4 4 4 4
23365 203/300 Read block 418 len 20480
23366 204/300 Write block 4198 len 4096 data: 4
23367 205/300 Read block 2181 len 16384
23368 206/300 Read block 2607 len 4096
23369 207/300 Write block 1496 len 8192 data: 6 5
23370 208/300 Write block 501 len 16384 data: 6 6 6 6
23371 209/300 Read block 2399 len 32768
23372 210/300 Write block 229 len 28672 data: 6 6 7 6 6 6 6
23373 211/300 Read block 2746 len 4096
23374 212/300 Read block 4857 len 40960
23375 213/300 Read block 592 len 24576
23376 214/300 Read block 744 len 36864
23377 215/300 Flush
23378 216/300 Read block 1305 len 40960
23379 217/300 Write block 4078 len 40960 data: 4 4 5 4 4 4 3 3 2 3
23380 218/300 Write block 1430 len 8192 data: 4 5
23381 219/300 Write block 446 len 12288 data: 6 6 6
23382 220/300 Write block 887 len 12288 data: 8 9 9
23383 221/300 Flush
23384 222/300 Read block 3746 len 36864
23385 223/300 Read block 1220 len 20480
23386 224/300 Write block 4877 len 16384 data: 5 4 5 6
23387 225/300 Read block 2220 len 24576
23388 226/300 Read block 3990 len 24576
23389 227/300 Read block 13 len 20480
23390 228/300 Read block 58 len 24576
23391 229/300 Write block 1186 len 40960 data: 6 4 8 8 7 8 8 7 7 7
23392 230/300 Read block 1952 len 4096
23393 231/300 Write block 2820 len 28672 data: 9 5 4 5 5 5 5
23394 232/300 Write block 3138 len 32768 data: 6 6 6 7 9 7 6 6
23395 233/300 Read block 1250 len 28672
23396 234/300 Flush
23397 235/300 Read block 4806 len 40960
23398 236/300 Read block 4826 len 16384
23399 237/300 Read block 4743 len 4096
23400 238/300 Read block 617 len 4096
23401 239/300 Read block 3895 len 32768
23402 240/300 Write block 4208 len 32768 data: 6 6 7 6 6 7 8 7
23403 241/300 Flush
23404 242/300 Read block 693 len 4096
23405 243/300 Read block 975 len 24576
23406 244/300 Write block 1601 len 24576 data: 3 3 5 7 5 5
23407 245/300 Read block 2883 len 32768
23408 246/300 Write block 1941 len 12288 data: 4 4 5
23409 247/300 Read block 1979 len 12288
23410 248/300 Write block 2821 len 24576 data: 6 5 6 6 6 6
23411 249/300 Read block 1773 len 36864
23412 250/300 Write block 351 len 32768 data: 10 11 13 13 10 8 9 7
23413 251/300 Read block 4710 len 40960
23414 252/300 Write block 261 len 24576 data: 9 8 8 7 6 6
23415 253/300 Write block 1899 len 40960 data: 4 5 6 6 7 7 7 6 6 6
23416 254/300 Read block 1711 len 24576
23417 255/300 Write block 3484 len 4096 data: 8
23418 256/300 Read block 499 len 28672
23419 257/300 Read block 3905 len 16384
23420 258/300 Flush
23421 259/300 Write block 1450 len 16384 data: 4 3 4 4
23422 260/300 Read block 241 len 4096
23423 261/300 Read block 2709 len 32768
23424 262/300 Write block 2628 len 12288 data: 4 4 4
23425 263/300 Flush
23426 264/300 Write block 85 len 28672 data: 4 4 5 4 4 5 6
23427 265/300 Read block 1685 len 20480
23428 266/300 Read block 2199 len 4096
23429 267/300 Read block 1746 len 4096
23430 268/300 Write block 1919 len 28672 data: 4 3 3 3 3 3 3
23431 269/300 Read block 3815 len 32768
23432 270/300 Read block 1586 len 16384
23433 271/300 Flush
23434 272/300 Write block 4947 len 32768 data: 5 8 9 9 9 9 9 6
23435 273/300 Read block 1960 len 36864
23436 274/300 Write block 4980 len 12288 data: 6 7 7
23437 275/300 Flush
23438 276/300 Read block 1010 len 40960
23439 277/300 Write block 3269 len 28672 data: 4 4 4 4 4 4 5
23440 278/300 Read block 4153 len 20480
23441 279/300 Flush
23442 280/300 Write block 3933 len 40960 data: 3 3 4 4 4 5 6 8 8 7
23443 281/300 Read block 4402 len 32768
23444 282/300 Read block 1637 len 12288
23445 283/300 Read block 2422 len 12288
23446 284/300 Write block 2976 len 24576 data: 4 4 4 4 3 3
23447 285/300 Read block 2190 len 20480
23448 286/300 Write block 3275 len 32768 data: 6 4 3 4 4 4 5 4
23449 287/300 Read block 1842 len 4096
23450 288/300 Write block 1443 len 32768 data: 3 3 3 5 3 3 4 5
23451 289/300 Write block 2139 len 28672 data: 2 4 3 4 5 4 4
23452 290/300 Read block 4305 len 40960
23453 291/300 Write block 1452 len 40960 data: 5 5 4 4 3 3 4 4 3 3
23454 292/300 Write block 4172 len 12288 data: 4 4 4
23455 293/300 Read block 1940 len 32768
23456 294/300 Read block 3919 len 32768
23457 295/300 Read block 4612 len 40960
23458 296/300 Read block 2280 len 12288
23459 297/300 Write block 4610 len 4096 data: 4
23460 298/300 Read block 964 len 36864
23461 299/300 Read block 997 len 20480
23462 300/300 Write block 4095 len 4096 data: 6
23463 Main task has work to do, go find it
23464 got dsc Start(0)
23465 start 0
23466 Feb 15 03:02:11.156 INFO request completed, latency_us: 203, response_code: 204, uri: /start/cid/0, method: POST, req_id: 93bb68eb-4abf-4ab8-929e-6a0f4ad0a646, remote_addr: 127.0.0.1:54744, local_addr: 127.0.0.1:9998
23467 Replay: started 0, returned:Ok(())
23468 ----------------------------------------------------------------
23469 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:300
23470 States: Offline Active Active
23471 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
23472 7235 Acked 8234 Read 4 New Done Done true
23473 7236 Acked 8235 Write 10 New Done Done true
23474 7237 Acked 8236 Read 8 New Done Done true
23475 7238 Acked 8237 Write 10 New Done Done false
23476 7239 Acked 8238 Write 9 New Done Done false
23477 7240 Acked 8239 Write 2 New Done Done false
23478 7241 Acked 8240 Write 2 New Done Done false
23479 7242 Acked 8241 Read 3 New Done Done false
23480 7243 Acked 8242 Read 8 New Done Done false
23481 7244 Acked 8243 Write 2 New Done Done false
23482 7245 Acked 8244 Flush 0 New Done Done false
23483 7246 Acked 8245 Write 5 New Done Done false
23484 7247 Acked 8246 Read 1 New Done Done false
23485 7248 Acked 8247 Flush 0 New Done Done false
23486 7249 Acked 8248 Write 5 New Done Done false
23487 7250 Acked 8249 Read 8 New Done Done false
23488 7251 Acked 8250 Read 6 New Done Done false
23489 7252 Acked 8251 Read 4 New Done Done false
23490 7253 Acked 8252 Read 5 New Done Done false
23491 7254 Acked 8253 Read 6 New Done Done false
23492 7255 Acked 8254 Write 4 New Done Done false
23493 7256 Acked 8255 Read 3 New Done Done false
23494 7257 Acked 8256 Read 5 New Done Done false
23495 7258 Acked 8257 Read 4 New Done Done false
23496 7259 Acked 8258 Read 5 New Done Done false
23497 7260 Acked 8259 Read 8 New Done Done false
23498 7261 Acked 8260 Read 6 New Done Done false
23499 7262 Acked 8261 Read 10 New Done Done false
23500 7263 Acked 8262 Write 4 New Done Done false
23501 7264 Acked 8263 Read 1 New Done Done false
23502 7265 Acked 8264 Read 9 New Done Done false
23503 7266 Acked 8265 Write 3 New Done Done false
23504 7267 Acked 8266 Read 7 New Done Done false
23505 7268 Acked 8267 Read 5 New Done Done false
23506 7269 Acked 8268 Read 3 New Done Done false
23507 7270 Acked 8269 Write 3 New Done Done false
23508 7271 Acked 8270 Write 9 New Done Done false
23509 7272 Acked 8271 Read 8 New Done Done false
23510 7273 Acked 8272 Read 10 New Done Done false
23511 7274 Acked 8273 Read 8 New Done Done false
23512 7275 Acked 8274 Write 10 New Done Done false
23513 7276 Acked 8275 Flush 0 New Done Done false
23514 7277 Acked 8276 Read 3 New Done Done false
23515 7278 Acked 8277 Flush 0 New Done Done false
23516 7279 Acked 8278 Write 3 New Done Done false
23517 7280 Acked 8279 Read 1 New Done Done false
23518 7281 Acked 8280 Read 6 New Done Done false
23519 7282 Acked 8281 Read 4 New Done Done false
23520 7283 Acked 8282 Write 2 New Done Done false
23521 7284 Acked 8283 Read 5 New Done Done false
23522 7285 Acked 8284 Write 7 New Done Done false
23523 7286 Acked 8285 Read 2 New Done Done false
23524 7287 Acked 8286 Read 5 New Done Done false
23525 7288 Acked 8287 Read 8 New Done Done false
23526 7289 Acked 8288 Read 10 New Done Done false
23527 7290 Acked 8289 Flush 0 New Done Done false
23528 7291 Acked 8290 Read 1 New Done Done false
23529 7292 Acked 8291 Read 8 New Done Done false
23530 7293 Acked 8292 Read 1 New Done Done false
23531 7294 Acked 8293 Read 7 New Done Done false
23532 7295 Acked 8294 Write 9 New Done Done false
23533 7296 Acked 8295 Read 3 New Done Done false
23534 7297 Acked 8296 Read 4 New Done Done false
23535 7298 Acked 8297 Read 5 New Done Done false
23536 7299 Acked 8298 Read 9 New Done Done false
23537 7300 Acked 8299 Write 9 New Done Done false
23538 7301 Acked 8300 Read 4 New Done Done false
23539 7302 Acked 8301 Write 4 New Done Done false
23540 7303 Acked 8302 Write 8 New Done Done false
23541 7304 Acked 8303 Read 10 New Done Done false
23542 7305 Acked 8304 Write 9 New Done Done false
23543 7306 Acked 8305 Write 1 New Done Done false
23544 7307 Acked 8306 Write 1 New Done Done false
23545 7308 Acked 8307 Write 10 New Done Done false
23546 7309 Acked 8308 Write 6 New Done Done false
23547 7310 Acked 8309 Write 1 New Done Done false
23548 7311 Acked 8310 Write 8 New Done Done false
23549 7312 Acked 8311 Write 3 New Done Done false
23550 7313 Acked 8312 Write 5 New Done Done false
23551 7314 Acked 8313 Read 8 New Done Done false
23552 7315 Acked 8314 Write 1 New Done Done false
23553 7316 Acked 8315 Read 9 New Done Done false
23554 7317 Acked 8316 Read 8 New Done Done false
23555 7318 Acked 8317 Read 1 New Done Done false
23556 7319 Acked 8318 Write 5 New Done Done false
23557 7320 Acked 8319 Read 1 New Done Done false
23558 7321 Acked 8320 Write 1 New Done Done false
23559 7322 Acked 8321 Read 10 New Done Done false
23560 7323 Acked 8322 Read 6 New Done Done false
23561 7324 Acked 8323 Write 6 New Done Done false
23562 7325 Acked 8324 Write 5 New Done Done false
23563 7326 Acked 8325 Write 3 New Done Done false
23564 7327 Acked 8326 Read 3 New Done Done false
23565 7328 Acked 8327 Write 2 New Done Done false
23566 7329 Acked 8328 Flush 0 New Done Done false
23567 7330 Acked 8329 Read 4 New Done Done false
23568 7331 Acked 8330 Read 5 New Done Done false
23569 7332 Acked 8331 Read 6 New Done Done false
23570 7333 Acked 8332 Read 7 New Done Done false
23571 7334 Acked 8333 Read 2 New Done Done false
23572 7335 Acked 8334 Write 2 New Done Done false
23573 7336 Acked 8335 Read 4 New Done Done false
23574 7337 Acked 8336 Read 1 New Done Done false
23575 7338 Acked 8337 Read 6 New Done Done false
23576 7339 Acked 8338 Read 5 New Done Done false
23577 7340 Acked 8339 Flush 0 New Done Done false
23578 7341 Acked 8340 Read 4 New Done Done false
23579 7342 Acked 8341 Read 2 New Done Done false
23580 7343 Acked 8342 Read 2 New Done Done false
23581 7344 Acked 8343 Read 8 New Done Done false
23582 7345 Acked 8344 Read 1 New Done Done false
23583 7346 Acked 8345 Read 3 New Done Done false
23584 7347 Acked 8346 Flush 0 New Done Done false
23585 7348 Acked 8347 Read 3 New Done Done false
23586 7349 Acked 8348 Read 2 New Done Done false
23587 7350 Acked 8349 Read 5 New Done Done false
23588 7351 Acked 8350 Flush 0 New Done Done false
23589 7352 Acked 8351 Read 4 New Done Done false
23590 7353 Acked 8352 Read 8 New Done Done false
23591 7354 Acked 8353 Write 2 New Done Done false
23592 7355 Acked 8354 Read 6 New Done Done false
23593 7356 Acked 8355 Flush 0 New Done Done false
23594 7357 Acked 8356 Read 2 New Done Done false
23595 7358 Acked 8357 Read 3 New Done Done false
23596 7359 Acked 8358 Write 1 New Done Done false
23597 7360 Acked 8359 Read 4 New Done Done false
23598 7361 Acked 8360 Flush 0 New Done Done false
23599 7362 Acked 8361 Write 5 New Done Done false
23600 7363 Acked 8362 Write 1 New Done Done false
23601 7364 Acked 8363 Read 1 New Done Done false
23602 7365 Acked 8364 Read 2 New Done Done false
23603 7366 Acked 8365 Read 2 New Done Done false
23604 7367 Acked 8366 Read 4 New Done Done false
23605 7368 Acked 8367 Read 6 New Done Done false
23606 7369 Acked 8368 Read 7 New Done Done false
23607 7370 Acked 8369 Read 6 New Done Done false
23608 7371 Acked 8370 Write 7 New Done Done false
23609 7372 Acked 8371 Read 5 New Done Done false
23610 7373 Acked 8372 Read 4 New Done Done false
23611 7374 Acked 8373 Flush 0 New Done Done false
23612 7375 Acked 8374 Write 1 New Done Done false
23613 7376 Acked 8375 Flush 0 New Done Done false
23614 7377 Acked 8376 Write 4 New Done Done false
23615 7378 Acked 8377 Write 4 New Done Done false
23616 7379 Acked 8378 Read 9 New Done Done false
23617 7380 Acked 8379 Write 7 New Done Done false
23618 7381 Acked 8380 Write 3 New Done Done false
23619 7382 Acked 8381 Read 2 New Done Done false
23620 7383 Acked 8382 Write 6 New Done Done false
23621 7384 Acked 8383 Write 8 New Done Done false
23622 7385 Acked 8384 Write 2 New Done Done false
23623 7386 Acked 8385 Write 6 New Done Done false
23624 7387 Acked 8386 Read 1 New Done Done false
23625 7388 Acked 8387 Flush 0 New Done Done false
23626 7389 Acked 8388 Write 9 New Done Done false
23627 7390 Acked 8389 Write 10 New Done Done false
23628 7391 Acked 8390 Read 2 New Done Done false
23629 7392 Acked 8391 Read 6 New Done Done false
23630 7393 Acked 8392 Write 6 New Done Done false
23631 7394 Acked 8393 Read 8 New Done Done false
23632 7395 Acked 8394 Read 7 New Done Done false
23633 7396 Acked 8395 Write 3 New Done Done false
23634 7397 Acked 8396 Read 6 New Done Done false
23635 7398 Acked 8397 Flush 0 New Done Done false
23636 7399 Acked 8398 Write 1 New Done Done false
23637 7400 Acked 8399 Read 8 New Done Done false
23638 7401 Acked 8400 Read 10 New Done Done false
23639 7402 Acked 8401 Write 6 New Done Done false
23640 7403 Acked 8402 Read 2 New Done Done false
23641 7404 Acked 8403 Write 4 New Done Done false
23642 7405 Acked 8404 Write 3 New Done Done false
23643 7406 Acked 8405 Write 9 New Done Done false
23644 7407 Acked 8406 Flush 0 New Done Done false
23645 7408 Acked 8407 Read 10 New Done Done false
23646 7409 Acked 8408 Write 2 New Done Done false
23647 7410 Acked 8409 Read 2 New Done Done false
23648 7411 Acked 8410 Read 1 New Done Done false
23649 7412 Acked 8411 Read 4 New Done Done false
23650 7413 Acked 8412 Read 8 New Done Done false
23651 7414 Acked 8413 Write 5 New Done Done false
23652 7415 Acked 8414 Read 4 New Done Done false
23653 7416 Acked 8415 Write 1 New Done Done false
23654 7417 Acked 8416 Write 6 New Done Done false
23655 7418 Acked 8417 Read 2 New Done Done false
23656 7419 Acked 8418 Read 9 New Done Done false
23657 7420 Acked 8419 Write 4 New Done Done false
23658 7421 Acked 8420 Read 9 New Done Done false
23659 7422 Acked 8421 Write 3 New Done Done false
23660 7423 Acked 8422 Read 4 New Done Done false
23661 7424 Acked 8423 Write 6 New Done Done false
23662 7425 Acked 8424 Write 7 New Done Done false
23663 7426 Acked 8425 Write 3 New Done Done false
23664 7427 Acked 8426 Write 7 New Done Done false
23665 7428 Acked 8427 Write 8 New Done Done false
23666 7429 Acked 8428 Write 8 New Done Done false
23667 7430 Acked 8429 Write 7 New Done Done false
23668 7431 Acked 8430 Write 6 New Done Done false
23669 7432 Acked 8431 Read 4 New Done Done false
23670 7433 Acked 8432 Write 4 New Done Done false
23671 7434 Acked 8433 Write 6 New Done Done false
23672 7435 Acked 8434 Read 2 New Done Done false
23673 7436 Acked 8435 Write 9 New Done Done false
23674 7437 Acked 8436 Read 5 New Done Done false
23675 7438 Acked 8437 Write 1 New Done Done false
23676 7439 Acked 8438 Read 4 New Done Done false
23677 7440 Acked 8439 Read 1 New Done Done false
23678 7441 Acked 8440 Write 2 New Done Done false
23679 7442 Acked 8441 Write 4 New Done Done false
23680 7443 Acked 8442 Read 8 New Done Done false
23681 7444 Acked 8443 Write 7 New Done Done false
23682 7445 Acked 8444 Read 1 New Done Done false
23683 7446 Acked 8445 Read 10 New Done Done false
23684 7447 Acked 8446 Read 6 New Done Done false
23685 7448 Acked 8447 Read 9 New Done Done false
23686 7449 Acked 8448 Flush 0 New Done Done false
23687 7450 Acked 8449 Read 10 New Done Done false
23688 7451 Acked 8450 Write 10 New Done Done false
23689 7452 Acked 8451 Write 2 New Done Done false
23690 7453 Acked 8452 Write 3 New Done Done false
23691 7454 Acked 8453 Write 3 New Done Done false
23692 7455 Acked 8454 Flush 0 New Done Done false
23693 7456 Acked 8455 Read 9 New Done Done false
23694 7457 Acked 8456 Read 5 New Done Done false
23695 7458 Acked 8457 Write 4 New Done Done false
23696 7459 Acked 8458 Read 6 New Done Done false
23697 7460 Acked 8459 Read 6 New Done Done false
23698 7461 Acked 8460 Read 5 New Done Done false
23699 7462 Acked 8461 Read 6 New Done Done false
23700 7463 Acked 8462 Write 10 New Done Done false
23701 7464 Acked 8463 Read 1 New Done Done false
23702 7465 Acked 8464 Write 7 New Done Done false
23703 7466 Acked 8465 Write 8 New Done Done false
23704 7467 Acked 8466 Read 7 New Done Done false
23705 7468 Acked 8467 Flush 0 New Done Done false
23706 7469 Acked 8468 Read 10 New Done Done false
23707 7470 Acked 8469 Read 4 New Done Done false
23708 7471 Acked 8470 Read 1 New Done Done false
23709 7472 Acked 8471 Read 1 New Done Done false
23710 7473 Acked 8472 Read 8 New Done Done false
23711 7474 Acked 8473 Write 8 New Done Done false
23712 7475 Acked 8474 Flush 0 New Done Done false
23713 7476 Acked 8475 Read 1 New Done Done false
23714 7477 Acked 8476 Read 6 New Done Done false
23715 7478 Acked 8477 Write 6 New Done Done false
23716 7479 Acked 8478 Read 8 New Done Done false
23717 7480 Acked 8479 Write 3 New Done Done false
23718 7481 Acked 8480 Read 3 New Done Done false
23719 7482 Acked 8481 Write 6 New Done Done false
23720 7483 Acked 8482 Read 9 New Done Done false
23721 7484 Acked 8483 Write 8 New Done Done false
23722 7485 Acked 8484 Read 10 New Done Done false
23723 7486 Acked 8485 Write 6 New Done Done false
23724 7487 Acked 8486 Write 10 New Done Done false
23725 7488 Acked 8487 Read 6 New Done Done false
23726 7489 Acked 8488 Write 1 New Done Done false
23727 7490 Acked 8489 Read 7 New Done Done false
23728 7491 Acked 8490 Read 4 New Done Done false
23729 7492 Acked 8491 Flush 0 New Done Done false
23730 7493 Acked 8492 Write 4 New Done Done false
23731 7494 Acked 8493 Read 1 New Done Done false
23732 7495 Acked 8494 Read 8 New Done Done false
23733 7496 Acked 8495 Write 3 New Done Done false
23734 7497 Acked 8496 Flush 0 New Done Done false
23735 7498 Acked 8497 Write 7 New Done Done false
23736 7499 Acked 8498 Read 5 New Done Done false
23737 7500 Acked 8499 Read 1 New Done Done false
23738 7501 Acked 8500 Read 1 New Done Done false
23739 7502 Acked 8501 Write 7 New Done Done false
23740 7503 Acked 8502 Read 8 New Done Done false
23741 7504 Acked 8503 Read 4 New Done Done false
23742 7505 Acked 8504 Flush 0 New Done Done false
23743 7506 Acked 8505 Write 8 New Done Done false
23744 7507 Acked 8506 Read 9 New Done Done false
23745 7508 Acked 8507 Write 3 New Done Done false
23746 7509 Acked 8508 Flush 0 New Done Done false
23747 7510 Acked 8509 Read 10 New Done Done false
23748 7511 Acked 8510 Write 7 New Done Done false
23749 7512 Acked 8511 Read 5 New Done Done false
23750 7513 Acked 8512 Flush 0 New Done Done false
23751 7514 Acked 8513 Write 10 New Done Done false
23752 7515 Acked 8514 Read 8 New Done Done false
23753 7516 Acked 8515 Read 3 New Done Done false
23754 7517 Acked 8516 Read 3 New Done Done false
23755 7518 Acked 8517 Write 6 New Done Done false
23756 7519 Acked 8518 Read 5 New Done Done false
23757 7520 Acked 8519 Write 8 New Done Done false
23758 7521 Acked 8520 Read 1 New Done Done false
23759 7522 Acked 8521 Write 8 New Done Done false
23760 7523 Acked 8522 Write 7 New Done Done false
23761 7524 Acked 8523 Read 10 New Done Done false
23762 7525 Acked 8524 Write 10 New Done Done false
23763 7526 Acked 8525 Write 3 New Done Done false
23764 7527 Acked 8526 Read 8 New Done Done false
23765 7528 Acked 8527 Read 8 New Done Done false
23766 7529 Acked 8528 Read 10 New Done Done false
23767 7530 Acked 8529 Read 3 New Done Done false
23768 7531 Acked 8530 Write 1 New Done Done false
23769 7532 Acked 8531 Read 9 New Done Done false
23770 7533 Acked 8532 Read 5 New Done Done false
23771 7534 Acked 8533 Write 1 New Done Done false
23772 STATES DS:0 DS:1 DS:2 TOTAL
23773 New 300 0 0 300
23774 Sent 0 0 0 0
23775 Done 0 300 300 600
23776 Skipped 0 0 0 0
23777 Error 0 0 0 0
23778 Last Flush: 8233 8512 8512
23779 Downstairs last five completed: 8233 8232 8231 8230 8229
23780 Upstairs last five completed: 7534 7533 7532 7531 7530
23781 CLIENT: Up:0 ds:300 act:2
237822024-02-15T03:02:11.826ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
237832024-02-15T03:02:11.826ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
237842024-02-15T03:02:11.826ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
237852024-02-15T03:02:11.826ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
237862024-02-15T03:02:11.826ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
237872024-02-15T03:02:11.826ZINFOcrucible: [0] client re-new 301 jobs since flush 8233 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
237882024-02-15T03:02:11.826ZINFOcrucible: client task is exiting = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
23789 [8810] got start action
23790 [8810] I am going to restart
23791 Starting downstairs at port 8810
23792 Make output file at "/tmp/dsc/downstairs-8810.txt"
23793 [8810][0] reports Starting
23794 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1192)
237952024-02-15T03:02:12.828ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
237962024-02-15T03:02:12.828ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
23797 [8810][0] reports Running
237982024-02-15T03:02:12.828ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
237992024-02-15T03:02:12.828ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
238002024-02-15T03:02:12.828ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
238012024-02-15T03:02:12.828ZINFOcrucible: [0] client re-new 301 jobs since flush 8233 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
238022024-02-15T03:02:12.828ZINFOcrucible: client task is exiting = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
238032024-02-15T03:02:13.829ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
238042024-02-15T03:02:13.829ZINFOcrucible: ds_connection connected = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
238052024-02-15T03:02:13.831ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
238062024-02-15T03:02:13.831ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
238072024-02-15T03:02:13.831ZINFOcrucible: send last flush ID to this DS: 8233 = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
238082024-02-15T03:02:13.831ZINFOcrucible: Replied this last flush ID: 8233 = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
238092024-02-15T03:02:13.831ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
238102024-02-15T03:02:13.831ZINFOcrucible: [0] Transition from Offline to Replay = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
238112024-02-15T03:02:13.831ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
238122024-02-15T03:02:13.831ZINFOcrucible: [0] Transition from Replay to Active = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
23813 ----------------------------------------------------------------
23814 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
23815 Downstairs last five completed: 8534 8533 8532 8531 8530
23816 Upstairs last five completed: 7535 7534 7533 7532 7531
23817 CLIENT: Up:0 ds:0 act:3
23818 Replay: All jobs finished, all DS active.
23819 Main task has work to do, go find it
23820 got dsc Stop(1)
23821 stop 1
23822 Feb 15 03:02:15.160[8820] Got stop action so:false kr:false
23823 INFO request completed, latency_us: 352, response_code: 204, uri: /stop/cid/1, method: POST, req_id: 32d0e65e-23c0-41d1-8671-ab6c50f917b4, remote_addr: 127.0.0.1:54744, local_addr: 127.0.0.1:9998
23824 Feb 15 03:02:15.160 INFO request completed, latency_us: 105, response_code: 200, uri: /state/cid/1, method: GET, req_id: 189e05c9-cd62-4443-ac5b-b4e1876ff229, remote_addr: 127.0.0.1:54744, local_addr: 127.0.0.1:9998
238252024-02-15T03:02:15.160ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
238262024-02-15T03:02:15.160ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
238272024-02-15T03:02:15.160ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
238282024-02-15T03:02:15.160ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
238292024-02-15T03:02:15.160ZINFOcrucible: [1] client re-new 0 jobs since flush 8534 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
238302024-02-15T03:02:15.160ZINFOcrucible: client task is exiting = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
23831 [8820] Exited with: ExitStatus(unix_wait_status(9))
23832 [8820][1] reports Exit
238332024-02-15T03:02:16.162ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
238342024-02-15T03:02:16.162ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
238352024-02-15T03:02:16.162ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
238362024-02-15T03:02:16.162ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
238372024-02-15T03:02:16.162ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
238382024-02-15T03:02:16.162ZINFOcrucible: [1] client re-new 0 jobs since flush 8534 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
238392024-02-15T03:02:16.162ZINFOcrucible: client task is exiting = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
238402024-02-15T03:02:17.163ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
238412024-02-15T03:02:17.163ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
238422024-02-15T03:02:17.163ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
238432024-02-15T03:02:17.163ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
238442024-02-15T03:02:17.163ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
238452024-02-15T03:02:17.163ZINFOcrucible: [1] client re-new 0 jobs since flush 8534 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
238462024-02-15T03:02:17.164ZINFOcrucible: client task is exiting = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
238472024-02-15T03:02:18.165ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
238482024-02-15T03:02:18.165ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
238492024-02-15T03:02:18.165ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
238502024-02-15T03:02:18.165ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
238512024-02-15T03:02:18.165ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
238522024-02-15T03:02:18.165ZINFOcrucible: [1] client re-new 0 jobs since flush 8534 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
238532024-02-15T03:02:18.165ZINFOcrucible: client task is exiting = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
23854 Feb 15 03:02:19.162 INFO request completed, latency_us: 200, response_code: 200, uri: /state/cid/1, method: GET, req_id: 1802832b-92f8-4095-b257-10eeeadb14da, remote_addr: 127.0.0.1:54744, local_addr: 127.0.0.1:9998
23855 001/300 Read block 2784 len 40960
23856 002/300 Read block 2390 len 32768
238572024-02-15T03:02:19.165ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
238582024-02-15T03:02:19.165ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
238592024-02-15T03:02:19.165ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
238602024-02-15T03:02:19.165ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
238612024-02-15T03:02:19.165ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
238622024-02-15T03:02:19.165ZINFOcrucible: [1] client re-new 2 jobs since flush 8534 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
238632024-02-15T03:02:19.165ZINFOcrucible: client task is exiting = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
23864 003/300 Flush
23865 004/300 Write block 365 len 32768 data: 4 4 6 7 8 7 8 8
23866 005/300 Write block 4337 len 12288 data: 7 6 4
23867 006/300 Flush
23868 007/300 Write block 918 len 32768 data: 7 5 3 4 4 5 6 5
23869 008/300 Write block 4618 len 4096 data: 5
23870 009/300 Read block 4142 len 16384
23871 010/300 Write block 2855 len 36864 data: 9 9 9 10 8 8 9 8 8
23872 011/300 Write block 3813 len 4096 data: 5
23873 012/300 Read block 2698 len 40960
23874 013/300 Read block 4359 len 16384
23875 014/300 Flush
23876 015/300 Read block 2447 len 28672
23877 016/300 Flush
23878 017/300 Read block 4908 len 4096
23879 018/300 Write block 2796 len 32768 data: 4 4 4 4 6 6 6 6
23880 019/300 Write block 1839 len 20480 data: 5 6 6 6 4
23881 020/300 Write block 1383 len 12288 data: 4 4 4
23882 021/300 Write block 4994 len 20480 data: 4 5 3 2 2
23883 022/300 Write block 273 len 20480 data: 3 3 3 4 4
23884 023/300 Read block 2750 len 12288
23885 024/300 Write block 433 len 12288 data: 4 3 3
23886 025/300 Write block 2653 len 40960 data: 2 3 3 4 4 4 5 5 7 6
23887 026/300 Read block 2936 len 20480
23888 027/300 Read block 3788 len 40960
23889 028/300 Write block 246 len 12288 data: 2 2 2
23890 029/300 Write block 737 len 12288 data: 7 7 9
23891 030/300 Read block 4728 len 20480
23892 031/300 Write block 4861 len 12288 data: 6 7 9
23893 032/300 Read block 4052 len 40960
23894 033/300 Read block 3774 len 24576
23895 034/300 Read block 4849 len 40960
23896 035/300 Read block 107 len 40960
23897 036/300 Flush
23898 037/300 Write block 2334 len 24576 data: 4 5 3 3 3 4
23899 038/300 Write block 3532 len 36864 data: 7 10 9 8 7 7 7 7 7
23900 039/300 Flush
23901 040/300 Write block 2107 len 4096 data: 5
23902 041/300 Read block 2334 len 12288
23903 042/300 Read block 3829 len 12288
23904 043/300 Write block 753 len 24576 data: 5 5 6 6 6 4
23905 044/300 Read block 1578 len 28672
23906 045/300 Flush
23907 046/300 Write block 2345 len 28672 data: 7 8 7 5 4 4 4
23908 047/300 Write block 4560 len 4096 data: 5
23909 048/300 Write block 4487 len 20480 data: 4 4 3 3 4
23910 049/300 Flush
23911 050/300 Write block 1079 len 28672 data: 5 6 6 6 5 6 6
23912 051/300 Write block 3101 len 40960 data: 7 4 4 5 6 6 7 7 8 8
23913 052/300 Read block 3944 len 8192
23914 053/300 Read block 3635 len 12288
23915 054/300 Write block 3324 len 24576 data: 3 4 6 7 7 5
23916 055/300 Read block 4418 len 16384
23917 056/300 Read block 2105 len 8192
23918 057/300 Read block 855 len 28672
23919 058/300 Read block 3904 len 16384
23920 059/300 Flush
23921 060/300 Read block 711 len 4096
23922 061/300 Read block 4122 len 36864
23923 062/300 Read block 2830 len 4096
23924 063/300 Read block 1714 len 28672
23925 064/300 Read block 2333 len 36864
23926 065/300 Write block 3609 len 32768 data: 6 7 4 7 8 10 9 9
23927 066/300 Write block 1010 len 32768 data: 7 8 9 7 7 4 4 5
23928 067/300 Read block 2200 len 16384
23929 068/300 Write block 724 len 16384 data: 5 4 4 4
23930 069/300 Write block 4621 len 8192 data: 6 6
23931 070/300 Read block 4597 len 4096
23932 071/300 Write block 4469 len 40960 data: 7 6 6 4 4 4 5 5 5 5
23933 072/300 Read block 2116 len 12288
23934 073/300 Flush
23935 074/300 Read block 4823 len 4096
23936 075/300 Read block 676 len 4096
23937 076/300 Write block 4496 len 40960 data: 4 5 4 2 2 4 5 3 3 6
23938 077/300 Flush
23939 078/300 Write block 909 len 4096 data: 5
23940 079/300 Write block 1973 len 4096 data: 4
23941 080/300 Read block 3669 len 16384
23942 081/300 Read block 4557 len 4096
23943 082/300 Read block 2799 len 36864
23944 083/300 Read block 3643 len 12288
23945 084/300 Read block 2708 len 4096
23946 085/300 Read block 1520 len 32768
23947 086/300 Flush
23948 087/300 Read block 3764 len 24576
23949 088/300 Read block 616 len 40960
23950 089/300 Read block 4410 len 32768
23951 090/300 Read block 4175 len 20480
23952 091/300 Write block 283 len 32768 data: 3 4 4 4 3 4 5 6
23953 092/300 Read block 172 len 32768
23954 093/300 Write block 2105 len 28672 data: 4 5 6 5 4 4 4
23955 094/300 Read block 1388 len 8192
23956 095/300 Write block 3365 len 12288 data: 3 3 3
23957 096/300 Write block 64 len 24576 data: 4 4 2 2 3 4
23958 097/300 Read block 1603 len 4096
23959 098/300 Write block 4809 len 20480 data: 4 5 5 6 4
23960 099/300 Write block 2462 len 12288 data: 3 3 3
23961 100/300 Read block 3919 len 40960
23962 101/300 Read block 4335 len 4096
23963 102/300 Read block 3150 len 32768
23964 103/300 Flush
23965 104/300 Read block 602 len 12288
23966 105/300 Write block 2420 len 8192 data: 5 5
23967 106/300 Read block 2443 len 36864
23968 107/300 Read block 2 len 20480
23969 108/300 Write block 4932 len 36864 data: 10 8 7 4 3 3 3 3 4
23970 109/300 Read block 1204 len 8192
23971 110/300 Write block 2320 len 32768 data: 4 4 3 2 2 4 5 5
23972 111/300 Write block 1623 len 16384 data: 7 7 8 8
23973 112/300 Write block 2931 len 24576 data: 5 4 4 3 3 4
23974 113/300 Write block 4267 len 4096 data: 5
23975 114/300 Read block 925 len 32768
23976 115/300 Read block 4304 len 40960
23977 116/300 Write block 541 len 20480 data: 5 5 5 6 7
23978 117/300 Write block 3129 len 16384 data: 4 5 4 3
23979 118/300 Write block 3162 len 36864 data: 12 11 9 9 7 7 5 5 4
23980 119/300 Flush
23981 120/300 Read block 579 len 28672
23982 121/300 Read block 1091 len 32768
23983 122/300 Read block 49 len 8192
23984 123/300 Read block 2607 len 40960
23985 124/300 Read block 665 len 32768
23986 125/300 Flush
23987 126/300 Write block 3970 len 12288 data: 4 5 4
23988 127/300 Read block 69 len 16384
23989 128/300 Flush
23990 129/300 Read block 1012 len 36864
23991 130/300 Read block 1023 len 4096
23992 131/300 Flush
23993 132/300 Read block 3744 len 32768
23994 133/300 Read block 4447 len 20480
23995 134/300 Write block 1019 len 16384 data: 4 5 5 5
23996 135/300 Read block 1220 len 36864
23997 136/300 Write block 152 len 8192 data: 4 5
23998 137/300 Read block 702 len 8192
23999 138/300 Write block 1923 len 16384 data: 4 4 4 3
24000 139/300 Read block 4481 len 16384
24001 140/300 Read block 2535 len 32768
24002 141/300 Write block 2822 len 24576 data: 6 7 7 7 7 5
24003 142/300 Read block 4252 len 4096
24004 143/300 Read block 2008 len 36864
24005 144/300 Read block 3710 len 12288
24006 145/300 Read block 3453 len 36864
24007 146/300 Write block 59 len 16384 data: 3 3 3 2
24008 147/300 Write block 110 len 12288 data: 3 2 2
24009 148/300 Flush
24010 149/300 Read block 2646 len 24576
24011 150/300 Read block 1138 len 36864
24012 151/300 Read block 2664 len 32768
24013 152/300 Write block 1963 len 16384 data: 4 5 4 4
24014 153/300 Write block 2574 len 16384 data: 5 5 5 5
24015 154/300 Write block 4307 len 4096 data: 7
24016 155/300 Read block 502 len 16384
24017 156/300 Read block 2776 len 20480
24018 157/300 Read block 2833 len 16384
24019 158/300 Read block 1157 len 36864
24020 159/300 Read block 599 len 4096
24021 160/300 Write block 1697 len 36864 data: 5 5 6 5 6 5 6 5 6
24022 161/300 Read block 3418 len 16384
24023 162/300 Write block 3337 len 8192 data: 3 7
24024 163/300 Write block 4935 len 12288 data: 5 4 4
24025 164/300 Write block 2059 len 36864 data: 6 6 5 6 6 6 6 7 7
24026 165/300 Read block 1857 len 20480
24027 166/300 Write block 410 len 8192 data: 2 2
24028 167/300 Read block 3818 len 28672
24029 168/300 Flush
24030 169/300 Write block 2496 len 4096 data: 4
24031 170/300 Read block 3472 len 16384
24032 171/300 Write block 2886 len 36864 data: 4 4 3 3 2 2 2 2 3
24033 172/300 Write block 95 len 40960 data: 7 6 7 7 6 4 4 5 4 4
24034 173/300 Write block 4934 len 32768 data: 8 6 5 5 4 4 5 6
24035 174/300 Read block 377 len 28672
24036 175/300 Read block 2519 len 40960
24037 176/300 Read block 4113 len 16384
24038 177/300 Read block 1730 len 24576
24039 178/300 Write block 1918 len 12288 data: 4 5 4
24040 179/300 Write block 2630 len 40960 data: 5 3 4 3 5 5 5 5 5 5
24041 180/300 Write block 2872 len 12288 data: 2 2 6
24042 181/300 Read block 4537 len 8192
24043 182/300 Read block 4429 len 28672
24044 183/300 Read block 894 len 16384
24045 184/300 Read block 777 len 32768
24046 185/300 Write block 562 len 40960 data: 4 5 6 7 8 7 6 7 8 5
24047 186/300 Read block 3570 len 36864
24048 187/300 Read block 1812 len 8192
24049 188/300 Read block 3688 len 20480
24050 189/300 Write block 4522 len 28672 data: 3 3 4 3 3 4 4
24051 190/300 Write block 3725 len 16384 data: 5 4 4 6
24052 191/300 Flush
24053 192/300 Read block 1218 len 12288
24054 193/300 Write block 1019 len 24576 data: 5 6 6 6 5 4
24055 194/300 Write block 2635 len 32768 data: 6 6 6 6 6 6 6 7
24056 195/300 Write block 727 len 12288 data: 5 6 5
24057 196/300 Read block 1478 len 24576
24058 197/300 Write block 3286 len 24576 data: 4 5 7 7 8 7
24059 198/300 Write block 3824 len 32768 data: 4 6 5 5 4 4 5 5
24060 199/300 Write block 2750 len 28672 data: 6 6 7 5 5 3 5
24061 200/300 Write block 997 len 24576 data: 5 5 8 7 7 6
24062 201/300 Read block 1614 len 40960
24063 202/300 Flush
24064 203/300 Write block 187 len 36864 data: 4 6 6 5 7 6 7 5 6
24065 204/300 Flush
24066 205/300 Write block 4015 len 12288 data: 11 9 9
24067 206/300 Flush
24068 207/300 Write block 4265 len 36864 data: 5 5 6 7 6 6 5 5 6
24069 208/300 Read block 1106 len 32768
24070 209/300 Write block 4863 len 36864 data: 10 10 10 8 7 8 5 5 6
24071 210/300 Read block 3079 len 4096
24072 211/300 Flush
24073 212/300 Write block 993 len 16384 data: 4 4 4 3
24074 213/300 Read block 446 len 24576
24075 214/300 Read block 2164 len 12288
24076 215/300 Write block 137 len 24576 data: 4 5 4 4 5 6
24077 216/300 Write block 2684 len 20480 data: 3 4 4 4 3
24078 217/300 Write block 258 len 28672 data: 7 7 8 10 9 9 8
24079 218/300 Read block 3839 len 4096
24080 219/300 Read block 1148 len 20480
24081 220/300 Write block 3047 len 16384 data: 6 8 8 8
24082 221/300 Write block 3631 len 4096 data: 5
24083 222/300 Flush
24084 223/300 Read block 4671 len 20480
24085 224/300 Read block 2407 len 28672
24086 225/300 Read block 4515 len 16384
24087 226/300 Read block 2159 len 28672
24088 227/300 Write block 3103 len 28672 data: 5 6 7 7 8 8 9
24089 228/300 Write block 790 len 20480 data: 7 6 7 6 7
24090 229/300 Read block 4616 len 20480
24091 230/300 Read block 3820 len 12288
24092 231/300 Flush
24093 232/300 Write block 3283 len 20480 data: 3 4 4 5 6
24094 233/300 Read block 3524 len 36864
24095 234/300 Write block 299 len 32768 data: 3 4 4 4 3 5 6 6
24096 235/300 Read block 1294 len 28672
24097 236/300 Write block 2308 len 20480 data: 4 3 4 3 3
24098 237/300 Read block 1504 len 24576
24099 238/300 Write block 4370 len 32768 data: 5 6 5 6 5 5 7 6
24100 239/300 Write block 699 len 32768 data: 5 5 5 5 5 5 4 4
24101 240/300 Write block 1043 len 28672 data: 3 3 3 3 3 3 3
24102 241/300 Flush
24103 242/300 Write block 1009 len 12288 data: 7 8 9
24104 243/300 Read block 2499 len 32768
24105 244/300 Read block 1615 len 36864
24106 245/300 Flush
24107 246/300 Write block 1309 len 32768 data: 6 6 5 5 5 5 4 6
24108 247/300 Read block 2808 len 4096
24109 248/300 Read block 3179 len 8192
24110 249/300 Read block 3255 len 28672
24111 250/300 Read block 3602 len 20480
24112 251/300 Write block 4544 len 28672 data: 6 6 9 8 8 8 8
24113 252/300 Write block 2861 len 24576 data: 10 9 9 10 9 9
24114 253/300 Read block 2804 len 32768
24115 254/300 Read block 2267 len 24576
24116 255/300 Flush
24117 256/300 Write block 3366 len 24576 data: 4 4 4 3 3 4
24118 257/300 Write block 991 len 40960 data: 5 5 5 5 5 4 6 6 9 8
24119 258/300 Write block 579 len 40960 data: 7 7 7 7 6 8 7 6 4 4
24120 259/300 Read block 597 len 28672
24121 260/300 Flush
24122 261/300 Flush
24123 262/300 Write block 4751 len 36864 data: 7 7 9 8 6 4 4 5 5
24124 263/300 Write block 4522 len 12288 data: 4 4 5
24125 264/300 Write block 775 len 20480 data: 5 6 6 6 6
24126 265/300 Write block 4744 len 8192 data: 5 4
24127 266/300 Flush
24128 267/300 Flush
24129 268/300 Read block 2565 len 4096
24130 269/300 Read block 4646 len 16384
24131 270/300 Read block 2691 len 16384
24132 271/300 Write block 2590 len 8192 data: 4 3
24133 272/300 Read block 3191 len 32768
24134 273/300 Read block 2958 len 16384
24135 274/300 Write block 856 len 4096 data: 3
24136 275/300 Write block 3303 len 28672 data: 5 6 6 5 4 4 4
24137 276/300 Flush
24138 277/300 Read block 4203 len 32768
24139 278/300 Read block 1874 len 8192
24140 279/300 Write block 2971 len 16384 data: 3 2 2 3
24141 280/300 Read block 4882 len 8192
24142 281/300 Read block 1012 len 36864
24143 282/300 Flush
24144 283/300 Read block 1910 len 40960
24145 284/300 Read block 1443 len 8192
24146 285/300 Write block 1723 len 28672 data: 4 4 5 6 5 5 7
24147 286/300 Read block 4774 len 4096
24148 287/300 Flush
24149 288/300 Read block 14 len 20480
24150 289/300 Write block 3798 len 24576 data: 6 8 4 4 3 3
24151 290/300 Write block 4066 len 4096 data: 6
24152 291/300 Write block 2903 len 12288 data: 4 4 4
24153 292/300 Read block 495 len 32768
24154 293/300 Write block 1553 len 40960 data: 2 4 5 4 4 5 7 6 7 8
24155 294/300 Write block 3440 len 40960 data: 6 6 5 5 5 6 4 5 6 6
24156 295/300 Read block 2458 len 24576
24157 296/300 Write block 770 len 24576 data: 3 2 3 3 5 6
24158 297/300 Write block 3285 len 32768 data: 5 6 7 8 8 9 8 6
24159 298/300 Read block 4176 len 32768
24160 299/300 Read block 1425 len 36864
24161 300/300 Read block 4664 len 24576
24162 Main task has work to do, go find it
24163 got dsc Start(1)
24164 start 1
24165 Feb 15 03:02:19.499 INFO request completed, latency_us: 205, response_code: 204, uri: /start/cid/1, method: POST, req_id: 4063cdd7-d214-4dfa-ae3b-e9719bff0c3a, remote_addr: 127.0.0.1:54744, local_addr: 127.0.0.1:9998
24166 Replay: started 1, returned:Ok(())
24167 ----------------------------------------------------------------
24168 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:300
24169 States: Active Offline Active
24170 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
24171 7536 Acked 8535 Read 10 Done New Done true
24172 7537 Acked 8536 Read 8 Done New Done true
24173 7538 Acked 8537 Flush 0 Done New Done false
24174 7539 Acked 8538 Write 8 Done New Done false
24175 7540 Acked 8539 Write 3 Done New Done false
24176 7541 Acked 8540 Flush 0 Done New Done false
24177 7542 Acked 8541 Write 8 Done New Done false
24178 7543 Acked 8542 Write 1 Done New Done false
24179 7544 Acked 8543 Read 4 Done New Done false
24180 7545 Acked 8544 Write 9 Done New Done false
24181 7546 Acked 8545 Write 1 Done New Done false
24182 7547 Acked 8546 Read 10 Done New Done false
24183 7548 Acked 8547 Read 4 Done New Done false
24184 7549 Acked 8548 Flush 0 Done New Done false
24185 7550 Acked 8549 Read 7 Done New Done false
24186 7551 Acked 8550 Flush 0 Done New Done false
24187 7552 Acked 8551 Read 1 Done New Done false
24188 7553 Acked 8552 Write 8 Done New Done false
24189 7554 Acked 8553 Write 5 Done New Done false
24190 7555 Acked 8554 Write 3 Done New Done false
24191 7556 Acked 8555 Write 5 Done New Done false
24192 7557 Acked 8556 Write 5 Done New Done false
24193 7558 Acked 8557 Read 3 Done New Done false
24194 7559 Acked 8558 Write 3 Done New Done false
24195 7560 Acked 8559 Write 10 Done New Done false
24196 7561 Acked 8560 Read 5 Done New Done false
24197 7562 Acked 8561 Read 10 Done New Done false
24198 7563 Acked 8562 Write 3 Done New Done false
24199 7564 Acked 8563 Write 3 Done New Done false
24200 7565 Acked 8564 Read 5 Done New Done false
24201 7566 Acked 8565 Write 3 Done New Done false
24202 7567 Acked 8566 Read 10 Done New Done false
24203 7568 Acked 8567 Read 6 Done New Done false
24204 7569 Acked 8568 Read 10 Done New Done false
24205 7570 Acked 8569 Read 10 Done New Done false
24206 7571 Acked 8570 Flush 0 Done New Done false
24207 7572 Acked 8571 Write 6 Done New Done false
24208 7573 Acked 8572 Write 9 Done New Done false
24209 7574 Acked 8573 Flush 0 Done New Done false
24210 7575 Acked 8574 Write 1 Done New Done false
24211 7576 Acked 8575 Read 3 Done New Done false
24212 7577 Acked 8576 Read 3 Done New Done false
24213 7578 Acked 8577 Write 6 Done New Done false
24214 7579 Acked 8578 Read 7 Done New Done false
24215 7580 Acked 8579 Flush 0 Done New Done false
24216 7581 Acked 8580 Write 7 Done New Done false
24217 7582 Acked 8581 Write 1 Done New Done false
24218 7583 Acked 8582 Write 5 Done New Done false
24219 7584 Acked 8583 Flush 0 Done New Done false
24220 7585 Acked 8584 Write 7 Done New Done false
24221 7586 Acked 8585 Write 10 Done New Done false
24222 7587 Acked 8586 Read 2 Done New Done false
24223 7588 Acked 8587 Read 3 Done New Done false
24224 7589 Acked 8588 Write 6 Done New Done false
24225 7590 Acked 8589 Read 4 Done New Done false
24226 7591 Acked 8590 Read 2 Done New Done false
24227 7592 Acked 8591 Read 7 Done New Done false
24228 7593 Acked 8592 Read 4 Done New Done false
24229 7594 Acked 8593 Flush 0 Done New Done false
24230 7595 Acked 8594 Read 1 Done New Done false
24231 7596 Acked 8595 Read 9 Done New Done false
24232 7597 Acked 8596 Read 1 Done New Done false
24233 7598 Acked 8597 Read 7 Done New Done false
24234 7599 Acked 8598 Read 9 Done New Done false
24235 7600 Acked 8599 Write 8 Done New Done false
24236 7601 Acked 8600 Write 8 Done New Done false
24237 7602 Acked 8601 Read 4 Done New Done false
24238 7603 Acked 8602 Write 4 Done New Done false
24239 7604 Acked 8603 Write 2 Done New Done false
24240 7605 Acked 8604 Read 1 Done New Done false
24241 7606 Acked 8605 Write 10 Done New Done false
24242 7607 Acked 8606 Read 3 Done New Done false
24243 7608 Acked 8607 Flush 0 Done New Done false
24244 7609 Acked 8608 Read 1 Done New Done false
24245 7610 Acked 8609 Read 1 Done New Done false
24246 7611 Acked 8610 Write 10 Done New Done false
24247 7612 Acked 8611 Flush 0 Done New Done false
24248 7613 Acked 8612 Write 1 Done New Done false
24249 7614 Acked 8613 Write 1 Done New Done false
24250 7615 Acked 8614 Read 4 Done New Done false
24251 7616 Acked 8615 Read 1 Done New Done false
24252 7617 Acked 8616 Read 9 Done New Done false
24253 7618 Acked 8617 Read 3 Done New Done false
24254 7619 Acked 8618 Read 1 Done New Done false
24255 7620 Acked 8619 Read 8 Done New Done false
24256 7621 Acked 8620 Flush 0 Done New Done false
24257 7622 Acked 8621 Read 6 Done New Done false
24258 7623 Acked 8622 Read 10 Done New Done false
24259 7624 Acked 8623 Read 8 Done New Done false
24260 7625 Acked 8624 Read 5 Done New Done false
24261 7626 Acked 8625 Write 8 Done New Done false
24262 7627 Acked 8626 Read 8 Done New Done false
24263 7628 Acked 8627 Write 7 Done New Done false
24264 7629 Acked 8628 Read 2 Done New Done false
24265 7630 Acked 8629 Write 3 Done New Done false
24266 7631 Acked 8630 Write 6 Done New Done false
24267 7632 Acked 8631 Read 1 Done New Done false
24268 7633 Acked 8632 Write 5 Done New Done false
24269 7634 Acked 8633 Write 3 Done New Done false
24270 7635 Acked 8634 Read 10 Done New Done false
24271 7636 Acked 8635 Read 1 Done New Done false
24272 7637 Acked 8636 Read 8 Done New Done false
24273 7638 Acked 8637 Flush 0 Done New Done false
24274 7639 Acked 8638 Read 3 Done New Done false
24275 7640 Acked 8639 Write 2 Done New Done false
24276 7641 Acked 8640 Read 9 Done New Done false
24277 7642 Acked 8641 Read 5 Done New Done false
24278 7643 Acked 8642 Write 9 Done New Done false
24279 7644 Acked 8643 Read 2 Done New Done false
24280 7645 Acked 8644 Write 8 Done New Done false
24281 7646 Acked 8645 Write 4 Done New Done false
24282 7647 Acked 8646 Write 6 Done New Done false
24283 7648 Acked 8647 Write 1 Done New Done false
24284 7649 Acked 8648 Read 8 Done New Done false
24285 7650 Acked 8649 Read 10 Done New Done false
24286 7651 Acked 8650 Write 5 Done New Done false
24287 7652 Acked 8651 Write 4 Done New Done false
24288 7653 Acked 8652 Write 9 Done New Done false
24289 7654 Acked 8653 Flush 0 Done New Done false
24290 7655 Acked 8654 Read 7 Done New Done false
24291 7656 Acked 8655 Read 8 Done New Done false
24292 7657 Acked 8656 Read 2 Done New Done false
24293 7658 Acked 8657 Read 10 Done New Done false
24294 7659 Acked 8658 Read 8 Done New Done false
24295 7660 Acked 8659 Flush 0 Done New Done false
24296 7661 Acked 8660 Write 3 Done New Done false
24297 7662 Acked 8661 Read 4 Done New Done false
24298 7663 Acked 8662 Flush 0 Done New Done false
24299 7664 Acked 8663 Read 9 Done New Done false
24300 7665 Acked 8664 Read 1 Done New Done false
24301 7666 Acked 8665 Flush 0 Done New Done false
24302 7667 Acked 8666 Read 8 Done New Done false
24303 7668 Acked 8667 Read 5 Done New Done false
24304 7669 Acked 8668 Write 4 Done New Done false
24305 7670 Acked 8669 Read 9 Done New Done false
24306 7671 Acked 8670 Write 2 Done New Done false
24307 7672 Acked 8671 Read 2 Done New Done false
24308 7673 Acked 8672 Write 4 Done New Done false
24309 7674 Acked 8673 Read 4 Done New Done false
24310 7675 Acked 8674 Read 8 Done New Done false
24311 7676 Acked 8675 Write 6 Done New Done false
24312 7677 Acked 8676 Read 1 Done New Done false
24313 7678 Acked 8677 Read 9 Done New Done false
24314 7679 Acked 8678 Read 3 Done New Done false
24315 7680 Acked 8679 Read 9 Done New Done false
24316 7681 Acked 8680 Write 4 Done New Done false
24317 7682 Acked 8681 Write 3 Done New Done false
24318 7683 Acked 8682 Flush 0 Done New Done false
24319 7684 Acked 8683 Read 6 Done New Done false
24320 7685 Acked 8684 Read 9 Done New Done false
24321 7686 Acked 8685 Read 8 Done New Done false
24322 7687 Acked 8686 Write 4 Done New Done false
24323 7688 Acked 8687 Write 4 Done New Done false
24324 7689 Acked 8688 Write 1 Done New Done false
24325 7690 Acked 8689 Read 4 Done New Done false
24326 7691 Acked 8690 Read 5 Done New Done false
24327 7692 Acked 8691 Read 4 Done New Done false
24328 7693 Acked 8692 Read 9 Done New Done false
24329 7694 Acked 8693 Read 1 Done New Done false
24330 7695 Acked 8694 Write 9 Done New Done false
24331 7696 Acked 8695 Read 4 Done New Done false
24332 7697 Acked 8696 Write 2 Done New Done false
24333 7698 Acked 8697 Write 3 Done New Done false
24334 7699 Acked 8698 Write 9 Done New Done false
24335 7700 Acked 8699 Read 5 Done New Done false
24336 7701 Acked 8700 Write 2 Done New Done false
24337 7702 Acked 8701 Read 7 Done New Done false
24338 7703 Acked 8702 Flush 0 Done New Done false
24339 7704 Acked 8703 Write 1 Done New Done false
24340 7705 Acked 8704 Read 4 Done New Done false
24341 7706 Acked 8705 Write 9 Done New Done false
24342 7707 Acked 8706 Write 10 Done New Done false
24343 7708 Acked 8707 Write 8 Done New Done false
24344 7709 Acked 8708 Read 7 Done New Done false
24345 7710 Acked 8709 Read 10 Done New Done false
24346 7711 Acked 8710 Read 4 Done New Done false
24347 7712 Acked 8711 Read 6 Done New Done false
24348 7713 Acked 8712 Write 3 Done New Done false
24349 7714 Acked 8713 Write 10 Done New Done false
24350 7715 Acked 8714 Write 3 Done New Done false
24351 7716 Acked 8715 Read 2 Done New Done false
24352 7717 Acked 8716 Read 7 Done New Done false
24353 7718 Acked 8717 Read 4 Done New Done false
24354 7719 Acked 8718 Read 8 Done New Done false
24355 7720 Acked 8719 Write 10 Done New Done false
24356 7721 Acked 8720 Read 9 Done New Done false
24357 7722 Acked 8721 Read 2 Done New Done false
24358 7723 Acked 8722 Read 5 Done New Done false
24359 7724 Acked 8723 Write 7 Done New Done false
24360 7725 Acked 8724 Write 4 Done New Done false
24361 7726 Acked 8725 Flush 0 Done New Done false
24362 7727 Acked 8726 Read 3 Done New Done false
24363 7728 Acked 8727 Write 6 Done New Done false
24364 7729 Acked 8728 Write 8 Done New Done false
24365 7730 Acked 8729 Write 3 Done New Done false
24366 7731 Acked 8730 Read 6 Done New Done false
24367 7732 Acked 8731 Write 6 Done New Done false
24368 7733 Acked 8732 Write 8 Done New Done false
24369 7734 Acked 8733 Write 7 Done New Done false
24370 7735 Acked 8734 Write 6 Done New Done false
24371 7736 Acked 8735 Read 10 Done New Done false
24372 7737 Acked 8736 Flush 0 Done New Done false
24373 7738 Acked 8737 Write 9 Done New Done false
24374 7739 Acked 8738 Flush 0 Done New Done false
24375 7740 Acked 8739 Write 3 Done New Done false
24376 7741 Acked 8740 Flush 0 Done New Done false
24377 7742 Acked 8741 Write 9 Done New Done false
24378 7743 Acked 8742 Read 8 Done New Done false
24379 7744 Acked 8743 Write 9 Done New Done false
24380 7745 Acked 8744 Read 1 Done New Done false
24381 7746 Acked 8745 Flush 0 Done New Done false
24382 7747 Acked 8746 Write 4 Done New Done false
24383 7748 Acked 8747 Read 6 Done New Done false
24384 7749 Acked 8748 Read 3 Done New Done false
24385 7750 Acked 8749 Write 6 Done New Done false
24386 7751 Acked 8750 Write 5 Done New Done false
24387 7752 Acked 8751 Write 7 Done New Done false
24388 7753 Acked 8752 Read 1 Done New Done false
24389 7754 Acked 8753 Read 5 Done New Done false
24390 7755 Acked 8754 Write 4 Done New Done false
24391 7756 Acked 8755 Write 1 Done New Done false
24392 7757 Acked 8756 Flush 0 Done New Done false
24393 7758 Acked 8757 Read 5 Done New Done false
24394 7759 Acked 8758 Read 7 Done New Done false
24395 7760 Acked 8759 Read 4 Done New Done false
24396 7761 Acked 8760 Read 7 Done New Done false
24397 7762 Acked 8761 Write 7 Done New Done false
24398 7763 Acked 8762 Write 5 Done New Done false
24399 7764 Acked 8763 Read 5 Done New Done false
24400 7765 Acked 8764 Read 3 Done New Done false
24401 7766 Acked 8765 Flush 0 Done New Done false
24402 7767 Acked 8766 Write 5 Done New Done false
24403 7768 Acked 8767 Read 9 Done New Done false
24404 7769 Acked 8768 Write 8 Done New Done false
24405 7770 Acked 8769 Read 7 Done New Done false
24406 7771 Acked 8770 Write 5 Done New Done false
24407 7772 Acked 8771 Read 6 Done New Done false
24408 7773 Acked 8772 Write 8 Done New Done false
24409 7774 Acked 8773 Write 8 Done New Done false
24410 7775 Acked 8774 Write 7 Done New Done false
24411 7776 Acked 8775 Flush 0 Done New Done false
24412 7777 Acked 8776 Write 3 Done New Done false
24413 7778 Acked 8777 Read 8 Done New Done false
24414 7779 Acked 8778 Read 9 Done New Done false
24415 7780 Acked 8779 Flush 0 Done New Done false
24416 7781 Acked 8780 Write 8 Done New Done false
24417 7782 Acked 8781 Read 1 Done New Done false
24418 7783 Acked 8782 Read 2 Done New Done false
24419 7784 Acked 8783 Read 7 Done New Done false
24420 7785 Acked 8784 Read 5 Done New Done false
24421 7786 Acked 8785 Write 7 Done New Done false
24422 7787 Acked 8786 Write 6 Done New Done false
24423 7788 Acked 8787 Read 8 Done New Done false
24424 7789 Acked 8788 Read 6 Done New Done false
24425 7790 Acked 8789 Flush 0 Done New Done false
24426 7791 Acked 8790 Write 6 Done New Done false
24427 7792 Acked 8791 Write 10 Done New Done false
24428 7793 Acked 8792 Write 10 Done New Done false
24429 7794 Acked 8793 Read 7 Done New Done false
24430 7795 Acked 8794 Flush 0 Done New Done false
24431 7796 Acked 8795 Flush 0 Done New Done false
24432 7797 Acked 8796 Write 9 Done New Done false
24433 7798 Acked 8797 Write 3 Done New Done false
24434 7799 Acked 8798 Write 5 Done New Done false
24435 7800 Acked 8799 Write 2 Done New Done false
24436 7801 Acked 8800 Flush 0 Done New Done false
24437 7802 Acked 8801 Flush 0 Done New Done false
24438 7803 Acked 8802 Read 1 Done New Done false
24439 7804 Acked 8803 Read 4 Done New Done false
24440 7805 Acked 8804 Read 4 Done New Done false
24441 7806 Acked 8805 Write 2 Done New Done false
24442 7807 Acked 8806 Read 8 Done New Done false
24443 7808 Acked 8807 Read 4 Done New Done false
24444 7809 Acked 8808 Write 1 Done New Done false
24445 7810 Acked 8809 Write 7 Done New Done false
24446 7811 Acked 8810 Flush 0 Done New Done false
24447 7812 Acked 8811 Read 8 Done New Done false
24448 7813 Acked 8812 Read 2 Done New Done false
24449 7814 Acked 8813 Write 4 Done New Done false
24450 7815 Acked 8814 Read 2 Done New Done false
24451 7816 Acked 8815 Read 9 Done New Done false
24452 7817 Acked 8816 Flush 0 Done New Done false
24453 7818 Acked 8817 Read 10 Done New Done false
24454 7819 Acked 8818 Read 2 Done New Done false
24455 7820 Acked 8819 Write 7 Done New Done false
24456 7821 Acked 8820 Read 1 Done New Done false
24457 7822 Acked 8821 Flush 0 Done New Done false
24458 7823 Acked 8822 Read 5 Done New Done false
24459 7824 Acked 8823 Write 6 Done New Done false
24460 7825 Acked 8824 Write 1 Done New Done false
24461 7826 Acked 8825 Write 3 Done New Done false
24462 7827 Acked 8826 Read 8 Done New Done false
24463 7828 Acked 8827 Write 10 Done New Done false
24464 7829 Acked 8828 Write 10 Done New Done false
24465 7830 Acked 8829 Read 6 Done New Done false
24466 7831 Acked 8830 Write 6 Done New Done false
24467 7832 Acked 8831 Write 8 Done New Done false
24468 7833 Acked 8832 Read 8 Done New Done false
24469 7834 Acked 8833 Read 9 Done New Done false
24470 7835 Acked 8834 Read 6 Done New Done false
24471 STATES DS:0 DS:1 DS:2 TOTAL
24472 New 0 300 0 300
24473 Sent 0 0 0 0
24474 Done 300 0 300 600
24475 Skipped 0 0 0 0
24476 Error 0 0 0 0
24477 Last Flush: 8821 8534 8821
24478 Downstairs last five completed: 8534 8533 8532 8531 8530
24479 Upstairs last five completed: 7835 7834 7833 7832 7831
24480 CLIENT: Up:0 ds:300 act:2
244812024-02-15T03:02:20.166ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
244822024-02-15T03:02:20.166ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
244832024-02-15T03:02:20.166ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
244842024-02-15T03:02:20.166ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
244852024-02-15T03:02:20.166ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
244862024-02-15T03:02:20.166ZINFOcrucible: [1] client re-new 301 jobs since flush 8534 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
244872024-02-15T03:02:20.167ZINFOcrucible: client task is exiting = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
244882024-02-15T03:02:21.168ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
244892024-02-15T03:02:21.168ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
244902024-02-15T03:02:21.168ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
244912024-02-15T03:02:21.168ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
244922024-02-15T03:02:21.168ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
244932024-02-15T03:02:21.168ZINFOcrucible: [1] client re-new 301 jobs since flush 8534 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
244942024-02-15T03:02:21.168ZINFOcrucible: client task is exiting = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
24495 [8820] got start action
24496 [8820] I am going to restart
24497 Starting downstairs at port 8820
24498 Make output file at "/tmp/dsc/downstairs-8820.txt"
24499 [8820][1] reports Starting
24500 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1193)
24501 [8820][1] reports Running
245022024-02-15T03:02:22.170ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
245032024-02-15T03:02:22.170ZINFOcrucible: ds_connection connected = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
245042024-02-15T03:02:22.171ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
245052024-02-15T03:02:22.171ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
245062024-02-15T03:02:22.171ZINFOcrucible: send last flush ID to this DS: 8534 = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
245072024-02-15T03:02:22.171ZINFOcrucible: Replied this last flush ID: 8534 = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
245082024-02-15T03:02:22.171ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
245092024-02-15T03:02:22.171ZINFOcrucible: [1] Transition from Offline to Replay = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
245102024-02-15T03:02:22.171ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
245112024-02-15T03:02:22.171ZINFOcrucible: [1] Transition from Replay to Active = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
24512 ----------------------------------------------------------------
24513 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
24514 Downstairs last five completed: 8835 8834 8833 8832 8831
24515 Upstairs last five completed: 7836 7835 7834 7833 7832
24516 CLIENT: Up:0 ds:0 act:3
24517 Replay: All jobs finished, all DS active.
24518 Main task has work to do, go find it
24519 got dsc Stop(0)
24520 stop 0
24521 [8810] Got stop action so:false kr:false
24522 Feb 15 03:02:23.503 INFO request completed, latency_us: 338, response_code: 204, uri: /stop/cid/0, method: POST, req_id: 4fcda482-0760-4621-becc-c8e0e4019571, remote_addr: 127.0.0.1:54744, local_addr: 127.0.0.1:9998
245232024-02-15T03:02:23.504ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
245242024-02-15T03:02:23.504ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
245252024-02-15T03:02:23.504ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
245262024-02-15T03:02:23.504ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
24527 Feb 15 03:02:23.504 INFO request completed, latency_us: 167, response_code: 200, uri: /state/cid/0, method: GET, req_id: 2cb32b5a-6a2e-43a5-86ca-59947fb9b150, remote_addr: 127.{"msg":"[0] client re-new 0 jobs since flush 8835","v":0,"name":"crucible","level":30,"time":"2024-02-15T03:02:23.504527711Z","hostname":"ip-10-150-1-67.us-west-2.compute.internal","pid":1167,"":"downstairs","":"downstairs","session_id":"4f5a63e5-aa00-4890-9f6b-96c2e069fdb2"}
24528 0.0.1:54744, local_addr: 127.0.0.1:9998
245292024-02-15T03:02:23.504ZINFOcrucible: client task is exiting = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
24530 [8810] Exited with: ExitStatus(unix_wait_status(9))
24531 [8810][0] reports Exit
245322024-02-15T03:02:24.506ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
245332024-02-15T03:02:24.506ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
245342024-02-15T03:02:24.506ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
245352024-02-15T03:02:24.506ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
245362024-02-15T03:02:24.506ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
245372024-02-15T03:02:24.506ZINFOcrucible: [0] client re-new 0 jobs since flush 8835 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
245382024-02-15T03:02:24.506ZINFOcrucible: client task is exiting = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
245392024-02-15T03:02:25.507ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
245402024-02-15T03:02:25.507ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
245412024-02-15T03:02:25.507ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
245422024-02-15T03:02:25.507ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
245432024-02-15T03:02:25.507ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
245442024-02-15T03:02:25.507ZINFOcrucible: [0] client re-new 0 jobs since flush 8835 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
245452024-02-15T03:02:25.507ZINFOcrucible: client task is exiting = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
245462024-02-15T03:02:26.508ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
245472024-02-15T03:02:26.509ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
245482024-02-15T03:02:26.509ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
245492024-02-15T03:02:26.509ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
245502024-02-15T03:02:26.509ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
245512024-02-15T03:02:26.509ZINFOcrucible: [0] client re-new 0 jobs since flush 8835 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
245522024-02-15T03:02:26.509ZINFOcrucible: client task is exiting = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
24553 Feb 15 03:02:27.506 INFO request completed, latency_us: 273, response_code: 200, uri: /state/cid/0, method: GET, req_id: 780ee497-2dea-456d-8cd6-14a2fd023582, remote_addr: 127.0.0.1:54744, local_addr: 127.0.0.1:9998
24554 001/300 Write block 3282 len 32768 data: 5 4 5 6 7 8 9 9
24555 002/300 Read block 680 len 40960
245562024-02-15T03:02:27.509ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
245572024-02-15T03:02:27.509ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
245582024-02-15T03:02:27.509ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
245592024-02-15T03:02:27.509ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
245602024-02-15T03:02:27.509ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
245612024-02-15T03:02:27.509ZINFOcrucible: [0] client re-new 2 jobs since flush 8835 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
245622024-02-15T03:02:27.509ZINFOcrucible: client task is exiting = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
24563 003/300 Write block 2087 len 40960 data: 7 6 4 5 3 3 3 4 4 5
24564 004/300 Read block 1559 len 40960
24565 005/300 Write block 1159 len 28672 data: 5 3 3 3 3 3 3
24566 006/300 Flush
24567 007/300 Write block 928 len 36864 data: 4 4 4 3 4 4 5 5 6
24568 008/300 Read block 1734 len 32768
24569 009/300 Read block 3225 len 24576
24570 010/300 Write block 3255 len 40960 data: 3 4 4 4 4 5 4 5 6 4
24571 011/300 Read block 53 len 32768
24572 012/300 Write block 2463 len 28672 data: 4 4 3 3 3 4 5
24573 013/300 Read block 169 len 8192
24574 014/300 Write block 1998 len 40960 data: 7 6 6 8 8 9 7 5 5 5
24575 015/300 Write block 1428 len 36864 data: 5 4 5 6 4 4 3 5 4
24576 016/300 Read block 4996 len 12288
24577 017/300 Flush
24578 018/300 Flush
24579 019/300 Read block 4411 len 24576
24580 020/300 Write block 2076 len 20480 data: 5 5 3 3 5
24581 021/300 Read block 2694 len 12288
24582 022/300 Write block 1907 len 4096 data: 7
24583 023/300 Read block 4080 len 4096
24584 024/300 Read block 22 len 20480
24585 025/300 Read block 1508 len 8192
24586 026/300 Read block 2117 len 4096
24587 027/300 Write block 4545 len 16384 data: 7 10 9 9
24588 028/300 Write block 3382 len 36864 data: 3 3 2 3 3 3 3 4 4
24589 029/300 Write block 3404 len 12288 data: 6 3 3
24590 030/300 Read block 2275 len 24576
24591 031/300 Read block 412 len 36864
24592 032/300 Write block 4078 len 12288 data: 5 5 6
24593 033/300 Write block 2753 len 24576 data: 6 6 4 6 6 5
24594 034/300 Read block 1298 len 40960
24595 035/300 Write block 2823 len 20480 data: 8 8 8 8 6
24596 036/300 Write block 499 len 16384 data: 5 4 7 7
24597 037/300 Write block 3978 len 24576 data: 5 5 6 7 6 7
24598 038/300 Read block 1286 len 28672
24599 039/300 Write block 4732 len 20480 data: 5 4 4 3 4
24600 040/300 Write block 4517 len 28672 data: 5 4 4 3 3 5 5
24601 041/300 Write block 736 len 28672 data: 7 8 8 10 7 6 5
24602 042/300 Read block 463 len 8192
24603 043/300 Read block 4737 len 12288
24604 044/300 Flush
24605 045/300 Read block 3794 len 8192
24606 046/300 Write block 654 len 40960 data: 3 4 4 4 5 7 7 6 6 6
24607 047/300 Flush
24608 048/300 Write block 1891 len 40960 data: 4 5 3 2 3 4 4 4 5 6
24609 049/300 Read block 4237 len 28672
24610 050/300 Read block 798 len 20480
24611 051/300 Flush
24612 052/300 Read block 2955 len 12288
24613 053/300 Flush
24614 054/300 Write block 4556 len 40960 data: 5 4 4 5 6 4 4 3 4 4
24615 055/300 Write block 4054 len 8192 data: 11 11
24616 056/300 Write block 1556 len 36864 data: 5 5 6 8 7 8 9 5 4
24617 057/300 Write block 2671 len 28672 data: 5 5 5 4 4 3 4
24618 058/300 Read block 3873 len 32768
24619 059/300 Write block 1440 len 40960 data: 5 4 4 4 4 4 6 4 4 5
24620 060/300 Write block 3031 len 28672 data: 5 6 6 7 6 6 6
24621 061/300 Write block 1079 len 36864 data: 6 7 7 7 6 7 7 6 5
24622 062/300 Read block 1861 len 4096
24623 063/300 Write block 4963 len 24576 data: 6 6 7 6 5 6
24624 064/300 Read block 478 len 32768
24625 065/300 Read block 2595 len 40960
24626 066/300 Read block 1511 len 20480
24627 067/300 Read block 4231 len 24576
24628 068/300 Flush
24629 069/300 Write block 3346 len 16384 data: 6 8 8 8
24630 070/300 Read block 2883 len 8192
24631 071/300 Flush
24632 072/300 Read block 358 len 28672
24633 073/300 Write block 4153 len 4096 data: 3
24634 074/300 Read block 3444 len 20480
24635 075/300 Read block 3207 len 20480
24636 076/300 Read block 688 len 20480
24637 077/300 Read block 886 len 24576
24638 078/300 Write block 2286 len 4096 data: 3
24639 079/300 Read block 1202 len 24576
24640 080/300 Read block 253 len 12288
24641 081/300 Read block 4684 len 36864
24642 082/300 Read block 3785 len 24576
24643 083/300 Read block 4220 len 4096
24644 084/300 Read block 3255 len 8192
24645 085/300 Read block 3781 len 24576
24646 086/300 Flush
24647 087/300 Write block 4153 len 28672 data: 4 3 3 4 4 4 4
24648 088/300 Read block 4442 len 32768
24649 089/300 Write block 3809 len 4096 data: 2
24650 090/300 Read block 4627 len 20480
24651 091/300 Write block 1807 len 20480 data: 5 5 5 8 7
24652 092/300 Write block 4921 len 28672 data: 3 4 5 6 6 6 6
24653 093/300 Write block 2545 len 20480 data: 8 9 7 7 8
24654 094/300 Read block 2181 len 8192
24655 095/300 Read block 333 len 4096
24656 096/300 Read block 2176 len 32768
24657 097/300 Write block 4720 len 8192 data: 6 5
24658 098/300 Write block 1126 len 24576 data: 6 7 6 7 6 6
24659 099/300 Flush
24660 100/300 Read block 4750 len 12288
24661 101/300 Write block 3376 len 28672 data: 5 5 3 3 3 4 4
24662 102/300 Write block 2863 len 32768 data: 10 11 10 10 9 7 5 4
24663 103/300 Read block 4743 len 32768
24664 104/300 Read block 1873 len 32768
24665 105/300 Read block 3914 len 32768
24666 106/300 Write block 3132 len 24576 data: 4 4 6 7 6 7
24667 107/300 Write block 277 len 32768 data: 5 3 3 4 4 4 4 5
24668 108/300 Read block 3069 len 28672
24669 109/300 Write block 1042 len 24576 data: 4 4 4 4 4 4
24670 110/300 Write block 1659 len 12288 data: 5 6 4
24671 111/300 Read block 416 len 32768
24672 112/300 Write block 1458 len 28672 data: 5 5 4 4 4 6 4
24673 113/300 Read block 1658 len 20480
24674 114/300 Write block 4496 len 16384 data: 5 6 5 3
24675 115/300 Write block 3005 len 40960 data: 2 3 2 2 4 4 5 4 5 5
24676 116/300 Read block 4756 len 8192
24677 117/300 Write block 668 len 32768 data: 4 5 7 7 7 8 9 6
24678 118/300 Read block 4540 len 16384
24679 119/300 Write block 1574 len 36864 data: 6 5 6 7 7 7 6 8 8
24680 120/300 Write block 2667 len 28672 data: 6 6 7 6 6 6 6
24681 121/300 Read block 3520 len 32768
24682 122/300 Read block 1061 len 8192
24683 123/300 Read block 590 len 40960
24684 124/300 Read block 972 len 8192
24685 125/300 Read block 1908 len 12288
24686 126/300 Read block 4818 len 12288
24687 127/300 Write block 909 len 24576 data: 6 5 4 3 4 4
24688 128/300 Read block 1263 len 12288
24689 129/300 Read block 1932 len 16384
24690 130/300 Read block 1234 len 12288
24691 131/300 Write block 1098 len 12288 data: 2 4 5
24692 132/300 Read block 1012 len 20480
24693 133/300 Write block 670 len 32768 data: 8 8 8 9 10 7 4 4
24694 134/300 Read block 3256 len 16384
24695 135/300 Write block 1762 len 4096 data: 6
24696 136/300 Read block 2057 len 28672
24697 137/300 Read block 807 len 8192
24698 138/300 Write block 1721 len 16384 data: 3 4 5 5
24699 139/300 Flush
24700 140/300 Write block 885 len 24576 data: 4 6 9 10 10 7
24701 141/300 Write block 1096 len 40960 data: 3 2 3 5 6 5 5 5 5 4
24702 142/300 Read block 4840 len 28672
24703 143/300 Write block 4275 len 24576 data: 4 4 5 4 4 7
24704 144/300 Write block 4223 len 20480 data: 3 2 3 3 4
24705 145/300 Write block 3585 len 32768 data: 5 5 4 4 4 4 3 3
24706 146/300 Flush
24707 147/300 Write block 2725 len 12288 data: 5 5 4
24708 148/300 Write block 4632 len 20480 data: 5 5 5 5 6
24709 149/300 Read block 4984 len 16384
24710 150/300 Flush
24711 151/300 Write block 2320 len 8192 data: 5 5
24712 152/300 Read block 3193 len 36864
24713 153/300 Read block 2180 len 16384
24714 154/300 Read block 3473 len 16384
24715 155/300 Read block 1144 len 36864
24716 156/300 Write block 2660 len 16384 data: 6 8 7 8
24717 157/300 Read block 2469 len 36864
24718 158/300 Read block 3476 len 24576
24719 159/300 Read block 3334 len 4096
24720 160/300 Write block 3423 len 28672 data: 4 5 5 4 3 4 4
24721 161/300 Read block 4746 len 20480
24722 162/300 Read block 3588 len 28672
24723 163/300 Write block 3323 len 12288 data: 4 4 5
24724 164/300 Read block 3831 len 4096
24725 165/300 Read block 1656 len 28672
24726 166/300 Read block 1156 len 12288
24727 167/300 Read block 4567 len 28672
24728 168/300 Read block 3955 len 16384
24729 169/300 Read block 4744 len 36864
24730 170/300 Flush
24731 171/300 Read block 24 len 24576
24732 172/300 Read block 4813 len 8192
24733 173/300 Read block 2084 len 8192
24734 174/300 Read block 1638 len 40960
24735 175/300 Write block 1017 len 20480 data: 6 4 6 7 7
24736 176/300 Flush
24737 177/300 Read block 3047 len 4096
24738 178/300 Read block 4312 len 12288
24739 179/300 Read block 3380 len 28672
24740 180/300 Read block 4703 len 4096
24741 181/300 Read block 1074 len 36864
24742 182/300 Read block 615 len 36864
24743 183/300 Read block 3371 len 40960
24744 184/300 Write block 3508 len 20480 data: 6 6 5 5 7
24745 185/300 Flush
24746 186/300 Write block 530 len 4096 data: 4
24747 187/300 Read block 2084 len 16384
24748 188/300 Read block 2260 len 12288
24749 189/300 Read block 1632 len 20480
24750 190/300 Read block 1390 len 16384
24751 191/300 Read block 4397 len 28672
24752 192/300 Read block 638 len 24576
24753 193/300 Flush
24754 194/300 Write block 2798 len 8192 data: 5 5
24755 195/300 Read block 4910 len 20480
24756 196/300 Read block 4648 len 40960
24757 197/300 Read block 4848 len 16384
24758 198/300 Read block 1082 len 8192
24759 199/300 Write block 3700 len 16384 data: 4 4 4 3
24760 200/300 Read block 4792 len 4096
24761 201/300 Read block 2354 len 12288
24762 202/300 Read block 2389 len 40960
24763 203/300 Read block 2968 len 8192
24764 204/300 Write block 2816 len 12288 data: 7 7 8
24765 205/300 Read block 2580 len 28672
24766 206/300 Flush
24767 207/300 Read block 2796 len 20480
24768 208/300 Read block 1012 len 12288
24769 209/300 Read block 3358 len 32768
24770 210/300 Write block 3079 len 32768 data: 6 6 7 7 8 5 5 6
24771 211/300 Read block 3549 len 28672
24772 212/300 Read block 4202 len 20480
24773 213/300 Read block 2106 len 4096
24774 214/300 Write block 3722 len 16384 data: 5 6 6 6
24775 215/300 Flush
24776 216/300 Write block 1307 len 8192 data: 4 4
24777 217/300 Read block 253 len 4096
24778 218/300 Write block 3999 len 36864 data: 3 3 3 4 6 6 6 5 5
24779 219/300 Flush
24780 220/300 Write block 360 len 16384 data: 6 8 7 6
24781 221/300 Read block 1032 len 12288
24782 222/300 Read block 3991 len 24576
24783 223/300 Read block 4319 len 24576
24784 224/300 Read block 2683 len 28672
24785 225/300 Read block 3151 len 28672
24786 226/300 Flush
24787 227/300 Write block 2889 len 12288 data: 4 3 3
24788 228/300 Read block 4676 len 24576
24789 229/300 Read block 2045 len 8192
24790 230/300 Write block 2314 len 28672 data: 5 4 5 5 4 4 6
24791 231/300 Read block 3330 len 28672
24792 232/300 Write block 996 len 40960 data: 5 7 7 10 9 8 7 6 5 6
24793 233/300 Write block 985 len 32768 data: 5 5 5 7 6 6 6 6
24794 234/300 Write block 2444 len 20480 data: 4 4 6 6 8
24795 235/300 Write block 937 len 36864 data: 7 7 7 5 4 6 5 5 4
24796 236/300 Flush
24797 237/300 Write block 2054 len 24576 data: 4 4 3 4 5 7
24798 238/300 Write block 3177 len 16384 data: 5 5 5 4
24799 239/300 Read block 3398 len 36864
24800 240/300 Write block 2234 len 40960 data: 6 6 6 6 5 8 8 7 6 5
24801 241/300 Read block 2566 len 32768
24802 242/300 Read block 3594 len 8192
24803 243/300 Flush
24804 244/300 Read block 373 len 24576
24805 245/300 Read block 614 len 40960
24806 246/300 Write block 3913 len 20480 data: 5 5 4 3 3
24807 247/300 Read block 1782 len 24576
24808 248/300 Write block 4175 len 32768 data: 3 2 2 3 5 5 6 5
24809 249/300 Write block 1705 len 24576 data: 7 5 3 2 2 2
24810 250/300 Write block 4292 len 40960 data: 4 4 4 6 7 5 5 4 3 3
24811 251/300 Read block 4206 len 4096
24812 252/300 Write block 2355 len 4096 data: 6
24813 253/300 Write block 4793 len 28672 data: 7 8 7 6 6 6 6
24814 254/300 Flush
24815 255/300 Read block 4438 len 24576
24816 256/300 Read block 2506 len 36864
24817 257/300 Write block 1132 len 16384 data: 5 4 4 5
24818 258/300 Read block 4525 len 12288
24819 259/300 Read block 1200 len 8192
24820 260/300 Write block 1465 len 4096 data: 4
24821 261/300 Write block 2052 len 20480 data: 3 4 5 5 4
24822 262/300 Flush
24823 263/300 Write block 87 len 4096 data: 6
24824 264/300 Read block 681 len 28672
24825 265/300 Read block 3860 len 40960
24826 266/300 Write block 2296 len 8192 data: 6 6
24827 267/300 Write block 2879 len 36864 data: 7 7 8 8 8 6 4 5 5
24828 268/300 Read block 3143 len 36864
24829 269/300 Write block 1234 len 28672 data: 6 6 6 7 6 5 7
24830 270/300 Write block 1768 len 24576 data: 4 5 6 5 5 5
24831 271/300 Flush
24832 272/300 Write block 4641 len 4096 data: 7
24833 273/300 Read block 1617 len 8192
24834 274/300 Read block 2371 len 12288
24835 275/300 Write block 3498 len 24576 data: 5 6 5 5 5 5
24836 276/300 Read block 213 len 24576
24837 277/300 Read block 4052 len 32768
24838 278/300 Read block 317 len 16384
24839 279/300 Read block 3711 len 28672
24840 280/300 Read block 1457 len 12288
24841 281/300 Write block 995 len 8192 data: 6 6
24842 282/300 Read block 1836 len 40960
24843 283/300 Write block 4744 len 32768 data: 6 5 3 4 6 6 7 8
24844 284/300 Read block 4199 len 8192
24845 285/300 Write block 3632 len 32768 data: 5 3 4 5 4 4 3 4
24846 286/300 Write block 3871 len 12288 data: 8 8 8
24847 287/300 Write block 1906 len 28672 data: 7 8 7 6 6 6 5
24848 288/300 Read block 4186 len 40960
24849 289/300 Write block 2409 len 40960 data: 4 4 5 4 4 4 4 4 4 5
24850 290/300 Read block 3523 len 20480
24851 291/300 Read block 2057 len 8192
24852 292/300 Write block 95 len 8192 data: 8 7
24853 293/300 Read block 237 len 20480
24854 294/300 Write block 4888 len 20480 data: 6 7 7 7 7
24855 295/300 Read block 4078 len 4096
24856 296/300 Flush
24857 297/300 Flush
24858 298/300 Flush
24859 299/300 Write block 704 len 32768 data: 6 5 5 6 5 4 4 6
24860 300/300 Read block 106 len 28672
24861 Main task has work to do, go find it
24862 got dsc Start(0)
24863 start 0
24864 Feb 15 03:02:27.847 INFO request completed, latency_us: 201, response_code: 204, uri: /start/cid/0, method: POST, req_id: 72c614b1-c13b-4fca-a386-51538fecb5e9, remote_addr: 127.0.0.1:54744, local_addr: 127.0.0.1:9998
24865 Replay: started 0, returned:Ok(())
24866 ----------------------------------------------------------------
24867 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
24868 States: Offline Active Active
24869 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
24870 7837 Acked 8836 Write 8 New Done Done true
24871 7838 Acked 8837 Read 10 New Done Done true
24872 7839 Acked 8838 Write 10 New Done Done false
24873 7840 Acked 8839 Read 10 New Done Done false
24874 7841 Acked 8840 Write 7 New Done Done false
24875 7842 Acked 8841 Flush 0 New Done Done false
24876 7843 Acked 8842 Write 9 New Done Done false
24877 7844 Acked 8843 Read 8 New Done Done false
24878 7845 Acked 8844 Read 6 New Done Done false
24879 7846 Acked 8845 Write 10 New Done Done false
24880 7847 Acked 8846 Read 8 New Done Done false
24881 7848 Acked 8847 Write 7 New Done Done false
24882 7849 Acked 8848 Read 2 New Done Done false
24883 7850 Acked 8849 Write 10 New Done Done false
24884 7851 Acked 8850 Write 9 New Done Done false
24885 7852 Acked 8851 Read 3 New Done Done false
24886 7853 Acked 8852 Flush 0 New Done Done false
24887 7854 Acked 8853 Flush 0 New Done Done false
24888 7855 Acked 8854 Read 6 New Done Done false
24889 7856 Acked 8855 Write 5 New Done Done false
24890 7857 Acked 8856 Read 3 New Done Done false
24891 7858 Acked 8857 Write 1 New Done Done false
24892 7859 Acked 8858 Read 1 New Done Done false
24893 7860 Acked 8859 Read 5 New Done Done false
24894 7861 Acked 8860 Read 2 New Done Done false
24895 7862 Acked 8861 Read 1 New Done Done false
24896 7863 Acked 8862 Write 4 New Done Done false
24897 7864 Acked 8863 Write 9 New Done Done false
24898 7865 Acked 8864 Write 3 New Done Done false
24899 7866 Acked 8865 Read 6 New Done Done false
24900 7867 Acked 8866 Read 9 New Done Done false
24901 7868 Acked 8867 Write 3 New Done Done false
24902 7869 Acked 8868 Write 6 New Done Done false
24903 7870 Acked 8869 Read 10 New Done Done false
24904 7871 Acked 8870 Write 5 New Done Done false
24905 7872 Acked 8871 Write 4 New Done Done false
24906 7873 Acked 8872 Write 6 New Done Done false
24907 7874 Acked 8873 Read 7 New Done Done false
24908 7875 Acked 8874 Write 5 New Done Done false
24909 7876 Acked 8875 Write 7 New Done Done false
24910 7877 Acked 8876 Write 7 New Done Done false
24911 7878 Acked 8877 Read 2 New Done Done false
24912 7879 Acked 8878 Read 3 New Done Done false
24913 7880 Acked 8879 Flush 0 New Done Done false
24914 7881 Acked 8880 Read 2 New Done Done false
24915 7882 Acked 8881 Write 10 New Done Done false
24916 7883 Acked 8882 Flush 0 New Done Done false
24917 7884 Acked 8883 Write 10 New Done Done false
24918 7885 Acked 8884 Read 7 New Done Done false
24919 7886 Acked 8885 Read 5 New Done Done false
24920 7887 Acked 8886 Flush 0 New Done Done false
24921 7888 Acked 8887 Read 3 New Done Done false
24922 7889 Acked 8888 Flush 0 New Done Done false
24923 7890 Acked 8889 Write 10 New Done Done false
24924 7891 Acked 8890 Write 2 New Done Done false
24925 7892 Acked 8891 Write 9 New Done Done false
24926 7893 Acked 8892 Write 7 New Done Done false
24927 7894 Acked 8893 Read 8 New Done Done false
24928 7895 Acked 8894 Write 10 New Done Done false
24929 7896 Acked 8895 Write 7 New Done Done false
24930 7897 Acked 8896 Write 9 New Done Done false
24931 7898 Acked 8897 Read 1 New Done Done false
24932 7899 Acked 8898 Write 6 New Done Done false
24933 7900 Acked 8899 Read 8 New Done Done false
24934 7901 Acked 8900 Read 10 New Done Done false
24935 7902 Acked 8901 Read 5 New Done Done false
24936 7903 Acked 8902 Read 6 New Done Done false
24937 7904 Acked 8903 Flush 0 New Done Done false
24938 7905 Acked 8904 Write 4 New Done Done false
24939 7906 Acked 8905 Read 2 New Done Done false
24940 7907 Acked 8906 Flush 0 New Done Done false
24941 7908 Acked 8907 Read 7 New Done Done false
24942 7909 Acked 8908 Write 1 New Done Done false
24943 7910 Acked 8909 Read 5 New Done Done false
24944 7911 Acked 8910 Read 5 New Done Done false
24945 7912 Acked 8911 Read 5 New Done Done false
24946 7913 Acked 8912 Read 6 New Done Done false
24947 7914 Acked 8913 Write 1 New Done Done false
24948 7915 Acked 8914 Read 6 New Done Done false
24949 7916 Acked 8915 Read 3 New Done Done false
24950 7917 Acked 8916 Read 9 New Done Done false
24951 7918 Acked 8917 Read 6 New Done Done false
24952 7919 Acked 8918 Read 1 New Done Done false
24953 7920 Acked 8919 Read 2 New Done Done false
24954 7921 Acked 8920 Read 6 New Done Done false
24955 7922 Acked 8921 Flush 0 New Done Done false
24956 7923 Acked 8922 Write 7 New Done Done false
24957 7924 Acked 8923 Read 8 New Done Done false
24958 7925 Acked 8924 Write 1 New Done Done false
24959 7926 Acked 8925 Read 5 New Done Done false
24960 7927 Acked 8926 Write 5 New Done Done false
24961 7928 Acked 8927 Write 7 New Done Done false
24962 7929 Acked 8928 Write 5 New Done Done false
24963 7930 Acked 8929 Read 2 New Done Done false
24964 7931 Acked 8930 Read 1 New Done Done false
24965 7932 Acked 8931 Read 8 New Done Done false
24966 7933 Acked 8932 Write 2 New Done Done false
24967 7934 Acked 8933 Write 6 New Done Done false
24968 7935 Acked 8934 Flush 0 New Done Done false
24969 7936 Acked 8935 Read 3 New Done Done false
24970 7937 Acked 8936 Write 7 New Done Done false
24971 7938 Acked 8937 Write 8 New Done Done false
24972 7939 Acked 8938 Read 8 New Done Done false
24973 7940 Acked 8939 Read 8 New Done Done false
24974 7941 Acked 8940 Read 8 New Done Done false
24975 7942 Acked 8941 Write 6 New Done Done false
24976 7943 Acked 8942 Write 8 New Done Done false
24977 7944 Acked 8943 Read 7 New Done Done false
24978 7945 Acked 8944 Write 6 New Done Done false
24979 7946 Acked 8945 Write 3 New Done Done false
24980 7947 Acked 8946 Read 8 New Done Done false
24981 7948 Acked 8947 Write 7 New Done Done false
24982 7949 Acked 8948 Read 5 New Done Done false
24983 7950 Acked 8949 Write 4 New Done Done false
24984 7951 Acked 8950 Write 10 New Done Done false
24985 7952 Acked 8951 Read 2 New Done Done false
24986 7953 Acked 8952 Write 8 New Done Done false
24987 7954 Acked 8953 Read 4 New Done Done false
24988 7955 Acked 8954 Write 9 New Done Done false
24989 7956 Acked 8955 Write 7 New Done Done false
24990 7957 Acked 8956 Read 8 New Done Done false
24991 7958 Acked 8957 Read 2 New Done Done false
24992 7959 Acked 8958 Read 10 New Done Done false
24993 7960 Acked 8959 Read 2 New Done Done false
24994 7961 Acked 8960 Read 3 New Done Done false
24995 7962 Acked 8961 Read 3 New Done Done false
24996 7963 Acked 8962 Write 6 New Done Done false
24997 7964 Acked 8963 Read 3 New Done Done false
24998 7965 Acked 8964 Read 4 New Done Done false
24999 7966 Acked 8965 Read 3 New Done Done false
25000 7967 Acked 8966 Write 3 New Done Done false
25001 7968 Acked 8967 Read 5 New Done Done false
25002 7969 Acked 8968 Write 8 New Done Done false
25003 7970 Acked 8969 Read 4 New Done Done false
25004 7971 Acked 8970 Write 1 New Done Done false
25005 7972 Acked 8971 Read 7 New Done Done false
25006 7973 Acked 8972 Read 2 New Done Done false
25007 7974 Acked 8973 Write 4 New Done Done false
25008 7975 Acked 8974 Flush 0 New Done Done false
25009 7976 Acked 8975 Write 6 New Done Done false
25010 7977 Acked 8976 Write 10 New Done Done false
25011 7978 Acked 8977 Read 7 New Done Done false
25012 7979 Acked 8978 Write 6 New Done Done false
25013 7980 Acked 8979 Flush 0 New Done Done false
25014 7981 Acked 8980 Write 5 New Done Done false
25015 7982 Acked 8981 Write 8 New Done Done false
25016 7983 Acked 8982 Flush 0 New Done Done false
25017 7984 Acked 8983 Write 3 New Done Done false
25018 7985 Acked 8984 Write 5 New Done Done false
25019 7986 Acked 8985 Read 4 New Done Done false
25020 7987 Acked 8986 Flush 0 New Done Done false
25021 7988 Acked 8987 Write 2 New Done Done false
25022 7989 Acked 8988 Read 9 New Done Done false
25023 7990 Acked 8989 Read 4 New Done Done false
25024 7991 Acked 8990 Read 4 New Done Done false
25025 7992 Acked 8991 Read 9 New Done Done false
25026 7993 Acked 8992 Write 4 New Done Done false
25027 7994 Acked 8993 Read 9 New Done Done false
25028 7995 Acked 8994 Read 6 New Done Done false
25029 7996 Acked 8995 Read 1 New Done Done false
25030 7997 Acked 8996 Write 7 New Done Done false
25031 7998 Acked 8997 Read 5 New Done Done false
25032 7999 Acked 8998 Read 7 New Done Done false
25033 8000 Acked 8999 Write 3 New Done Done false
25034 8001 Acked 9000 Read 1 New Done Done false
25035 8002 Acked 9001 Read 7 New Done Done false
25036 8003 Acked 9002 Read 3 New Done Done false
25037 8004 Acked 9003 Read 7 New Done Done false
25038 8005 Acked 9004 Read 4 New Done Done false
25039 8006 Acked 9005 Read 9 New Done Done false
25040 8007 Acked 9006 Flush 0 New Done Done false
25041 8008 Acked 9007 Read 6 New Done Done false
25042 8009 Acked 9008 Read 2 New Done Done false
25043 8010 Acked 9009 Read 2 New Done Done false
25044 8011 Acked 9010 Read 10 New Done Done false
25045 8012 Acked 9011 Write 5 New Done Done false
25046 8013 Acked 9012 Flush 0 New Done Done false
25047 8014 Acked 9013 Read 1 New Done Done false
25048 8015 Acked 9014 Read 3 New Done Done false
25049 8016 Acked 9015 Read 7 New Done Done false
25050 8017 Acked 9016 Read 1 New Done Done false
25051 8018 Acked 9017 Read 9 New Done Done false
25052 8019 Acked 9018 Read 9 New Done Done false
25053 8020 Acked 9019 Read 10 New Done Done false
25054 8021 Acked 9020 Write 5 New Done Done false
25055 8022 Acked 9021 Flush 0 New Done Done false
25056 8023 Acked 9022 Write 1 New Done Done false
25057 8024 Acked 9023 Read 4 New Done Done false
25058 8025 Acked 9024 Read 3 New Done Done false
25059 8026 Acked 9025 Read 5 New Done Done false
25060 8027 Acked 9026 Read 4 New Done Done false
25061 8028 Acked 9027 Read 7 New Done Done false
25062 8029 Acked 9028 Read 6 New Done Done false
25063 8030 Acked 9029 Flush 0 New Done Done false
25064 8031 Acked 9030 Write 2 New Done Done false
25065 8032 Acked 9031 Read 5 New Done Done false
25066 8033 Acked 9032 Read 10 New Done Done false
25067 8034 Acked 9033 Read 4 New Done Done false
25068 8035 Acked 9034 Read 2 New Done Done false
25069 8036 Acked 9035 Write 4 New Done Done false
25070 8037 Acked 9036 Read 1 New Done Done false
25071 8038 Acked 9037 Read 3 New Done Done false
25072 8039 Acked 9038 Read 10 New Done Done false
25073 8040 Acked 9039 Read 2 New Done Done false
25074 8041 Acked 9040 Write 3 New Done Done false
25075 8042 Acked 9041 Read 7 New Done Done false
25076 8043 Acked 9042 Flush 0 New Done Done false
25077 8044 Acked 9043 Read 5 New Done Done false
25078 8045 Acked 9044 Read 3 New Done Done false
25079 8046 Acked 9045 Read 8 New Done Done false
25080 8047 Acked 9046 Write 8 New Done Done false
25081 8048 Acked 9047 Read 7 New Done Done false
25082 8049 Acked 9048 Read 5 New Done Done false
25083 8050 Acked 9049 Read 1 New Done Done false
25084 8051 Acked 9050 Write 4 New Done Done false
25085 8052 Acked 9051 Flush 0 New Done Done false
25086 8053 Acked 9052 Write 2 New Done Done false
25087 8054 Acked 9053 Read 1 New Done Done false
25088 8055 Acked 9054 Write 9 New Done Done false
25089 8056 Acked 9055 Flush 0 New Done Done false
25090 8057 Acked 9056 Write 4 New Done Done false
25091 8058 Acked 9057 Read 3 New Done Done false
25092 8059 Acked 9058 Read 6 New Done Done false
25093 8060 Acked 9059 Read 6 New Done Done false
25094 8061 Acked 9060 Read 7 New Done Done false
25095 8062 Acked 9061 Read 7 New Done Done false
25096 8063 Acked 9062 Flush 0 New Done Done false
25097 8064 Acked 9063 Write 3 New Done Done false
25098 8065 Acked 9064 Read 6 New Done Done false
25099 8066 Acked 9065 Read 2 New Done Done false
25100 8067 Acked 9066 Write 7 New Done Done false
25101 8068 Acked 9067 Read 7 New Done Done false
25102 8069 Acked 9068 Write 10 New Done Done false
25103 8070 Acked 9069 Write 8 New Done Done false
25104 8071 Acked 9070 Write 5 New Done Done false
25105 8072 Acked 9071 Write 9 New Done Done false
25106 8073 Acked 9072 Flush 0 New Done Done false
25107 8074 Acked 9073 Write 6 New Done Done false
25108 8075 Acked 9074 Write 4 New Done Done false
25109 8076 Acked 9075 Read 9 New Done Done false
25110 8077 Acked 9076 Write 10 New Done Done false
25111 8078 Acked 9077 Read 8 New Done Done false
25112 8079 Acked 9078 Read 2 New Done Done false
25113 8080 Acked 9079 Flush 0 New Done Done false
25114 8081 Acked 9080 Read 6 New Done Done false
25115 8082 Acked 9081 Read 10 New Done Done false
25116 8083 Acked 9082 Write 5 New Done Done false
25117 8084 Acked 9083 Read 6 New Done Done false
25118 8085 Acked 9084 Write 8 New Done Done false
25119 8086 Acked 9085 Write 6 New Done Done false
25120 8087 Acked 9086 Write 10 New Done Done false
25121 8088 Acked 9087 Read 1 New Done Done false
25122 8089 Acked 9088 Write 1 New Done Done false
25123 8090 Acked 9089 Write 7 New Done Done false
25124 8091 Acked 9090 Flush 0 New Done Done false
25125 8092 Acked 9091 Read 6 New Done Done false
25126 8093 Acked 9092 Read 9 New Done Done false
25127 8094 Acked 9093 Write 4 New Done Done false
25128 8095 Acked 9094 Read 3 New Done Done false
25129 8096 Acked 9095 Read 2 New Done Done false
25130 8097 Acked 9096 Write 1 New Done Done false
25131 8098 Acked 9097 Write 5 New Done Done false
25132 8099 Acked 9098 Flush 0 New Done Done false
25133 8100 Acked 9099 Write 1 New Done Done false
25134 8101 Acked 9100 Read 7 New Done Done false
25135 8102 Acked 9101 Read 10 New Done Done false
25136 8103 Acked 9102 Write 2 New Done Done false
25137 8104 Acked 9103 Write 9 New Done Done false
25138 8105 Acked 9104 Read 9 New Done Done false
25139 8106 Acked 9105 Write 7 New Done Done false
25140 8107 Acked 9106 Write 6 New Done Done false
25141 8108 Acked 9107 Flush 0 New Done Done false
25142 8109 Acked 9108 Write 1 New Done Done false
25143 8110 Acked 9109 Read 2 New Done Done false
25144 8111 Acked 9110 Read 3 New Done Done false
25145 8112 Acked 9111 Write 6 New Done Done false
25146 8113 Acked 9112 Read 6 New Done Done false
25147 8114 Acked 9113 Read 8 New Done Done false
25148 8115 Acked 9114 Read 4 New Done Done false
25149 8116 Acked 9115 Read 7 New Done Done false
25150 8117 Acked 9116 Read 3 New Done Done false
25151 8118 Acked 9117 Write 2 New Done Done false
25152 8119 Acked 9118 Read 10 New Done Done false
25153 8120 Acked 9119 Write 8 New Done Done false
25154 8121 Acked 9120 Read 2 New Done Done false
25155 8122 Acked 9121 Write 8 New Done Done false
25156 8123 Acked 9122 Write 3 New Done Done false
25157 8124 Acked 9123 Write 7 New Done Done false
25158 8125 Acked 9124 Read 10 New Done Done false
25159 8126 Acked 9125 Write 10 New Done Done false
25160 8127 Acked 9126 Read 5 New Done Done false
25161 8128 Acked 9127 Read 2 New Done Done false
25162 8129 Acked 9128 Write 2 New Done Done false
25163 8130 Acked 9129 Read 5 New Done Done false
25164 8131 Acked 9130 Write 5 New Done Done false
25165 8132 Acked 9131 Read 1 New Done Done false
25166 8133 Acked 9132 Flush 0 New Done Done false
25167 8134 Acked 9133 Flush 0 New Done Done false
25168 8135 Acked 9134 Flush 0 New Done Done false
25169 8136 Acked 9135 Write 8 New Done Done false
25170 8137 Acked 9136 Read 7 New Done Done false
25171 STATES DS:0 DS:1 DS:2 TOTAL
25172 New 301 0 0 301
25173 Sent 0 0 0 0
25174 Done 0 301 301 602
25175 Skipped 0 0 0 0
25176 Error 0 0 0 0
25177 Last Flush: 8835 9134 9134
25178 Downstairs last five completed: 8835 8834 8833 8832 8831
25179 Upstairs last five completed: 8137 8136 8135 8134 8133
25180 CLIENT: Up:0 ds:301 act:2
251812024-02-15T03:02:28.511ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
251822024-02-15T03:02:28.511ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
251832024-02-15T03:02:28.511ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
251842024-02-15T03:02:28.511ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
251852024-02-15T03:02:28.511ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
251862024-02-15T03:02:28.511ZINFOcrucible: [0] client re-new 302 jobs since flush 8835 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
251872024-02-15T03:02:28.511ZINFOcrucible: client task is exiting = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
25188 [8810] got start action
25189 [8810] I am going to restart
25190 Starting downstairs at port 8810
25191 Make output file at "/tmp/dsc/downstairs-8810.txt"
25192 [8810][0] reports Starting
251932024-02-15T03:02:29.512ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
251942024-02-15T03:02:29.512ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
251952024-02-15T03:02:29.513ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
251962024-02-15T03:02:29.513ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
251972024-02-15T03:02:29.513ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
251982024-02-15T03:02:29.513ZINFOcrucible: [0] client re-new 302 jobs since flush 8835 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
251992024-02-15T03:02:29.513ZINFOcrucible: client task is exiting = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
25200 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1194)
25201 [8810][0] reports Running
252022024-02-15T03:02:30.514ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
252032024-02-15T03:02:30.514ZINFOcrucible: ds_connection connected = io task client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
252042024-02-15T03:02:30.515ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
252052024-02-15T03:02:30.515ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
252062024-02-15T03:02:30.515ZINFOcrucible: send last flush ID to this DS: 8835 = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
252072024-02-15T03:02:30.515ZINFOcrucible: Replied this last flush ID: 8835 = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
252082024-02-15T03:02:30.515ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
252092024-02-15T03:02:30.515ZINFOcrucible: [0] Transition from Offline to Replay = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
252102024-02-15T03:02:30.515ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
252112024-02-15T03:02:30.515ZINFOcrucible: [0] Transition from Replay to Active = downstairs client = 0 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
25212 ----------------------------------------------------------------
25213 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
25214 Downstairs last five completed: 9137 9136 9135 9134 9133
25215 Upstairs last five completed: 8138 8137 8136 8135 8134
25216 CLIENT: Up:0 ds:0 act:3
25217 Replay: All jobs finished, all DS active.
25218 Main task has work to do, go find it
25219 got dsc Stop(2)
25220 stop 2
25221 [8830] Got stop action so:false kr:false
25222 Feb 15 03:02:31.851 INFO request completed, latency_us: 342, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 3543c030-9cea-4d4a-bd8c-a29dfc5e7f7b, remote_addr: 127.0.0.1:54744, local_addr: 127.0.0.1:9998
252232024-02-15T03:02:31.852ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
252242024-02-15T03:02:31.852ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
252252024-02-15T03:02:31.852ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
25226 Feb 15 03:02:31.852 INFO request completed, latency_us:{"msg":"Gone missing, transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-02-15T03:02:31.852442922Z","hostname":"ip-10-150-1-67.us-west-2.compute.internal","pid":1167,"client":"2","":"downstairs","session_id":"4f5a63e5-aa00-4890-9f6b-96c2e069fdb2"}
25227 132, {"msg":"[2] client re-new 0 jobs since flush 9137","v":0,"name":"crucible","level":30,"time":"2024-02-15T03:02:31.852481487Z","hostname":"ip-10-150-1-67.us-west-2.compute.internal","pid":1167,"":"downstairs","":"downstairs","session_id":"4f5a63e5-aa00-4890-9f6b-96c2e069fdb2"}
25228 response_code: 200{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-15T03:02:31.852522872Z","hostname":"ip-10-150-1-67.us-west-2.compute.internal","pid":1167,"":"io task","client":"2","":"downstairs","session_id":"4f5a63e5-aa00-4890-9f6b-96c2e069fdb2"}
25229 , uri: /state/cid/2, method: GET, req_id: 06969be0-19b2-40f0-8abe-39ab264e11f5, remote_addr: 127.0.0.1:54744, local_addr: 127.0.0.1:9998
25230 [8830] Exited with: ExitStatus(unix_wait_status(9))
25231 [8830][2] reports Exit
252322024-02-15T03:02:32.854ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
252332024-02-15T03:02:32.854ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
252342024-02-15T03:02:32.854ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
252352024-02-15T03:02:32.854ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
252362024-02-15T03:02:32.854ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
252372024-02-15T03:02:32.854ZINFOcrucible: [2] client re-new 0 jobs since flush 9137 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
252382024-02-15T03:02:32.854ZINFOcrucible: client task is exiting = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
252392024-02-15T03:02:33.856ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
252402024-02-15T03:02:33.856ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
252412024-02-15T03:02:33.856ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
252422024-02-15T03:02:33.856ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
252432024-02-15T03:02:33.856ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
252442024-02-15T03:02:33.856ZINFOcrucible: [2] client re-new 0 jobs since flush 9137 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
252452024-02-15T03:02:33.856ZINFOcrucible: client task is exiting = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
252462024-02-15T03:02:34.857ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
252472024-02-15T03:02:34.857ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
252482024-02-15T03:02:34.857ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
252492024-02-15T03:02:34.857ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
252502024-02-15T03:02:34.857ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
252512024-02-15T03:02:34.857ZINFOcrucible: [2] client re-new 0 jobs since flush 9137 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
252522024-02-15T03:02:34.857ZINFOcrucible: client task is exiting = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
25253 Feb 15 03:02:35.855 INFO request completed, latency_us: 223, response_code: 200, uri: /state/cid/2, method: GET, req_id: 9b56bf64-6304-4064-90c2-c54845c70de2, remote_addr: 127.0.0.1:54744, local_addr: 127.0.0.1:9998
25254 001/300 Flush
25255 002/300 Write block 3071 len 12288 data: 4 4 3
25256 003/300 Flush
252572024-02-15T03:02:35.858ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
252582024-02-15T03:02:35.858ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
252592024-02-15T03:02:35.858ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
252602024-02-15T03:02:35.858ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
252612024-02-15T03:02:35.858ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
252622024-02-15T03:02:35.858ZINFOcrucible: [2] client re-new 3 jobs since flush 9137 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
252632024-02-15T03:02:35.858ZINFOcrucible: client task is exiting = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
25264 004/300 Write block 3197 len 28672 data: 6 5 6 8 8 7 7
25265 005/300 Read block 1601 len 32768
25266 006/300 Read block 230 len 8192
25267 007/300 Read block 2498 len 36864
25268 008/300 Read block 1973 len 32768
25269 009/300 Read block 2809 len 12288
25270 010/300 Write block 3405 len 32768 data: 4 4 4 5 5 7 7 7
25271 011/300 Read block 1227 len 28672
25272 012/300 Write block 4423 len 16384 data: 7 8 8 7
25273 013/300 Read block 3788 len 36864
25274 014/300 Read block 736 len 36864
25275 015/300 Write block 4515 len 8192 data: 4 4
25276 016/300 Read block 4225 len 36864
25277 017/300 Read block 1266 len 40960
25278 018/300 Read block 3293 len 12288
25279 019/300 Read block 2214 len 16384
25280 020/300 Read block 1986 len 16384
25281 021/300 Read block 3284 len 40960
25282 022/300 Read block 331 len 36864
25283 023/300 Read block 4451 len 20480
25284 024/300 Read block 2329 len 32768
25285 025/300 Flush
25286 026/300 Read block 1901 len 28672
25287 027/300 Flush
25288 028/300 Write block 3178 len 40960 data: 6 6 5 4 4 4 3 4 3 4
25289 029/300 Read block 4104 len 8192
25290 030/300 Read block 4344 len 8192
25291 031/300 Read block 1936 len 8192
25292 032/300 Write block 3852 len 40960 data: 5 5 7 6 8 9 8 9 10 7
25293 033/300 Read block 3338 len 40960
25294 034/300 Read block 419 len 40960
25295 035/300 Write block 4100 len 36864 data: 8 5 5 4 5 6 5 5 6
25296 036/300 Read block 477 len 20480
25297 037/300 Read block 4372 len 28672
25298 038/300 Flush
25299 039/300 Read block 2117 len 24576
25300 040/300 Write block 2335 len 24576 data: 6 4 4 4 5 5
25301 041/300 Read block 1714 len 28672
25302 042/300 Read block 3619 len 24576
25303 043/300 Read block 678 len 24576
25304 044/300 Read block 1386 len 4096
25305 045/300 Read block 802 len 12288
25306 046/300 Read block 885 len 16384
25307 047/300 Write block 3019 len 36864 data: 3 3 4 4 3 4 4 4 5
25308 048/300 Read block 3944 len 12288
25309 049/300 Read block 4371 len 8192
25310 050/300 Write block 2525 len 28672 data: 7 5 5 4 3 2 3
25311 051/300 Read block 735 len 28672
25312 052/300 Write block 2086 len 16384 data: 6 8 7 5
25313 053/300 Read block 104 len 32768
25314 054/300 Write block 373 len 40960 data: 8 7 6 8 8 8 9 10 7 7
25315 055/300 Read block 2725 len 16384
25316 056/300 Read block 4284 len 12288
25317 057/300 Flush
25318 058/300 Read block 1608 len 32768
25319 059/300 Read block 2987 len 8192
25320 060/300 Read block 928 len 24576
25321 061/300 Flush
25322 062/300 Read block 1963 len 8192
25323 063/300 Read block 467 len 40960
25324 064/300 Read block 780 len 20480
25325 065/300 Write block 4260 len 40960 data: 8 9 9 7 5 6 6 7 8 7
25326 066/300 Write block 1733 len 20480 data: 7 8 7 7 9
25327 067/300 Read block 3331 len 20480
25328 068/300 Read block 463 len 4096
25329 069/300 Write block 2817 len 16384 data: 8 9 8 10
25330 070/300 Read block 4661 len 24576
25331 071/300 Write block 4574 len 32768 data: 2 2 2 2 2 3 3 3
25332 072/300 Write block 3543 len 36864 data: 8 7 9 8 7 5 4 4 4
25333 073/300 Read block 3089 len 24576
25334 074/300 Read block 1683 len 16384
25335 075/300 Write block 3404 len 32768 data: 7 5 5 5 6 6 8 8
25336 076/300 Write block 4741 len 36864 data: 6 4 4 7 6 4 5 7 7
25337 077/300 Write block 1115 len 24576 data: 7 7 7 6 7 6
25338 078/300 Read block 1399 len 16384
25339 079/300 Read block 4794 len 8192
25340 080/300 Read block 1317 len 8192
25341 081/300 Read block 2787 len 16384
25342 082/300 Read block 4892 len 36864
25343 083/300 Read block 2671 len 24576
25344 084/300 Read block 4168 len 24576
25345 085/300 Write block 2909 len 28672 data: 9 8 6 5 9 7 7
25346 086/300 Write block 751 len 24576 data: 6 6 6 6 7 7
25347 087/300 Write block 4065 len 24576 data: 6 7 5 6 5 5
25348 088/300 Flush
25349 089/300 Write block 101 len 36864 data: 5 6 5 5 4 5 4 3 3
25350 090/300 Read block 3282 len 16384
25351 091/300 Read block 3532 len 8192
25352 092/300 Read block 2898 len 16384
25353 093/300 Flush
25354 094/300 Read block 2579 len 28672
25355 095/300 Read block 2039 len 4096
25356 096/300 Write block 3608 len 28672 data: 6 7 8 5 8 9 11
25357 097/300 Read block 789 len 4096
25358 098/300 Write block 1447 len 40960 data: 5 5 6 6 4 6 6 5 5 4
25359 099/300 Read block 2875 len 28672
25360 100/300 Read block 4989 len 20480
25361 101/300 Flush
25362 102/300 Write block 3109 len 32768 data: 10 9 9 9 8 8 7 7
25363 103/300 Flush
25364 104/300 Write block 4324 len 24576 data: 5 5 5 4 5 5
25365 105/300 Read block 1950 len 32768
25366 106/300 Read block 4683 len 16384
25367 107/300 Write block 469 len 20480 data: 10 10 5 6 6
25368 108/300 Write block 2008 len 4096 data: 5
25369 109/300 Write block 970 len 36864 data: 4 4 5 7 5 5 4 6 5
25370 110/300 Write block 4310 len 32768 data: 8 11 9 7 7 7 5 5
25371 111/300 Write block 4552 len 28672 data: 8 7 7 5 6 5 5
25372 112/300 Write block 722 len 36864 data: 5 5 6 5 5 6 7 6 4
25373 113/300 Read block 1075 len 16384
25374 114/300 Read block 2220 len 4096
25375 115/300 Flush
25376 116/300 Read block 4476 len 16384
25377 117/300 Write block 1373 len 24576 data: 3 2 2 2 2 3
25378 118/300 Read block 495 len 32768
25379 119/300 Write block 4781 len 20480 data: 10 9 8 8 7
25380 120/300 Flush
25381 121/300 Write block 4233 len 12288 data: 5 4 5
25382 122/300 Flush
25383 123/300 Write block 4798 len 20480 data: 7 7 6 5 5
25384 124/300 Read block 2756 len 40960
25385 125/300 Write block 2509 len 36864 data: 7 4 5 4 4 5 5 5 5
25386 126/300 Write block 3892 len 24576 data: 2 2 2 3 3 5
25387 127/300 Read block 4467 len 40960
25388 128/300 Write block 3248 len 28672 data: 7 6 5 5 4 5 3
25389 129/300 Write block 4599 len 16384 data: 3 3 5 6
25390 130/300 Read block 3960 len 24576
25391 131/300 Read block 625 len 4096
25392 132/300 Write block 618 len 36864 data: 10 9 8 4 3 4 4 4 4
25393 133/300 Read block 3484 len 16384
25394 134/300 Read block 1744 len 40960
25395 135/300 Write block 155 len 36864 data: 6 6 7 7 6 8 9 9 8
25396 136/300 Read block 4352 len 12288
25397 137/300 Write block 3538 len 40960 data: 8 8 8 8 7 9 8 10 9 8
25398 138/300 Write block 3736 len 24576 data: 6 5 4 5 6 6
25399 139/300 Write block 4692 len 40960 data: 9 8 8 7 5 4 3 4 4 5
25400 140/300 Read block 4775 len 4096
25401 141/300 Write block 4808 len 28672 data: 3 5 6 6 7 5 4
25402 142/300 Read block 4851 len 36864
25403 143/300 Read block 1907 len 16384
25404 144/300 Read block 113 len 36864
25405 145/300 Read block 2555 len 8192
25406 146/300 Read block 583 len 16384
25407 147/300 Write block 4853 len 4096 data: 4
25408 148/300 Read block 1171 len 16384
25409 149/300 Read block 59 len 36864
25410 150/300 Read block 3415 len 12288
25411 151/300 Read block 200 len 28672
25412 152/300 Write block 4436 len 32768 data: 6 5 4 4 4 3 4 5
25413 153/300 Flush
25414 154/300 Write block 4570 len 32768 data: 3 2 2 2 3 3 3 3
25415 155/300 Read block 4809 len 28672
25416 156/300 Read block 2263 len 36864
25417 157/300 Read block 2746 len 20480
25418 158/300 Write block 4039 len 4096 data: 3
25419 159/300 Read block 312 len 20480
25420 160/300 Write block 1419 len 20480 data: 4 6 8 7 6
25421 161/300 Write block 4741 len 32768 data: 7 5 5 8 7 5 6 8
25422 162/300 Flush
25423 163/300 Read block 3017 len 4096
25424 164/300 Read block 2873 len 36864
25425 165/300 Write block 1281 len 40960 data: 6 5 4 5 6 8 8 9 8 6
25426 166/300 Write block 2966 len 4096 data: 3
25427 167/300 Flush
25428 168/300 Write block 4003 len 24576 data: 7 7 7 6 6 6
25429 169/300 Read block 1844 len 12288
25430 170/300 Read block 3051 len 32768
25431 171/300 Write block 4627 len 36864 data: 4 5 6 6 6 6 6 6 6
25432 172/300 Read block 1375 len 28672
25433 173/300 Read block 3896 len 32768
25434 174/300 Read block 1383 len 16384
25435 175/300 Write block 4323 len 12288 data: 4 6 6
25436 176/300 Read block 3587 len 28672
25437 177/300 Write block 3515 len 40960 data: 3 3 4 3 5 5 6 7 7 8
25438 178/300 Write block 2071 len 4096 data: 8
25439 179/300 Read block 550 len 8192
25440 180/300 Read block 4162 len 8192
25441 181/300 Read block 1877 len 8192
25442 182/300 Read block 3299 len 4096
25443 183/300 Read block 2331 len 4096
25444 184/300 Write block 3365 len 28672 data: 4 5 5 5 4 4 5
25445 185/300 Write block 2535 len 12288 data: 6 6 6
25446 186/300 Write block 101 len 36864 data: 6 7 6 6 5 6 5 4 4
25447 187/300 Write block 4409 len 28672 data: 4 5 4 5 5 5 5
25448 188/300 Read block 635 len 40960
25449 189/300 Read block 3821 len 4096
25450 190/300 Write block 1538 len 8192 data: 7 7
25451 191/300 Read block 141 len 40960
25452 192/300 Write block 45 len 24576 data: 4 4 4 5 5 3
25453 193/300 Read block 4481 len 28672
25454 194/300 Write block 1822 len 36864 data: 6 8 7 8 8 9 9 8 7
25455 195/300 Write block 2153 len 40960 data: 4 5 4 5 5 6 8 7 7 7
25456 196/300 Read block 4706 len 28672
25457 197/300 Write block 2956 len 24576 data: 5 5 7 7 7 5
25458 198/300 Read block 4600 len 24576
25459 199/300 Read block 4052 len 4096
25460 200/300 Read block 4788 len 16384
25461 201/300 Write block 1884 len 36864 data: 7 6 5 4 4 5 5 5 6
25462 202/300 Write block 229 len 28672 data: 7 7 8 7 7 7 7
25463 203/300 Write block 1296 len 32768 data: 3 3 3 3 3 3 5 5
25464 204/300 Read block 87 len 8192
25465 205/300 Read block 380 len 40960
25466 206/300 Read block 1213 len 20480
25467 207/300 Flush
25468 208/300 Read block 1573 len 4096
25469 209/300 Write block 1 len 40960 data: 4 4 4 4 4 4 4 4 4 4
25470 210/300 Write block 1769 len 8192 data: 6 7
25471 211/300 Read block 3536 len 36864
25472 212/300 Write block 1759 len 28672 data: 6 8 6 7 7 6 4
25473 213/300 Read block 1862 len 24576
25474 214/300 Read block 4599 len 24576
25475 215/300 Read block 2798 len 4096
25476 216/300 Write block 4361 len 36864 data: 6 6 5 5 4 7 6 5 5
25477 217/300 Write block 502 len 12288 data: 8 7 7
25478 218/300 Flush
25479 219/300 Read block 4094 len 16384
25480 220/300 Write block 2578 len 40960 data: 5 4 4 5 6 5 6 6 4 4
25481 221/300 Read block 3118 len 36864
25482 222/300 Write block 3080 len 24576 data: 7 8 8 9 6 6
25483 223/300 Write block 1519 len 4096 data: 5
25484 224/300 Write block 2768 len 36864 data: 4 4 4 4 4 4 5 4 4
25485 225/300 Write block 1223 len 12288 data: 7 7 6
25486 226/300 Read block 3405 len 16384
25487 227/300 Read block 2251 len 28672
25488 228/300 Write block 4722 len 12288 data: 4 6 5
25489 229/300 Read block 3670 len 4096
25490 230/300 Write block 4007 len 28672 data: 7 7 8 6 7 9 9
25491 231/300 Write block 1428 len 28672 data: 6 5 6 7 5 5 4
25492 232/300 Read block 1960 len 36864
25493 233/300 Write block 166 len 40960 data: 7 6 6 5 5 5 5 5 6 6
25494 234/300 Write block 943 len 36864 data: 6 6 5 6 6 5 4 5 6
25495 235/300 Flush
25496 236/300 Flush
25497 237/300 Read block 4671 len 36864
25498 238/300 Read block 59 len 28672
25499 239/300 Write block 847 len 36864 data: 7 6 6 5 6 6 4 3 3
25500 240/300 Read block 2494 len 8192
25501 241/300 Read block 1060 len 20480
25502 242/300 Read block 3567 len 32768
25503 243/300 Write block 1670 len 20480 data: 5 5 4 5 6
25504 244/300 Read block 159 len 32768
25505 245/300 Read block 1153 len 12288
25506 246/300 Read block 147 len 12288
25507 247/300 Write block 344 len 12288 data: 5 5 5
25508 248/300 Flush
25509 249/300 Read block 1339 len 20480
25510 250/300 Write block 3248 len 20480 data: 8 7 6 6 5
25511 251/300 Flush
25512 252/300 Read block 1571 len 36864
25513 253/300 Flush
25514 254/300 Write block 2529 len 16384 data: 4 3 4 4
25515 255/300 Flush
25516 256/300 Write block 2587 len 40960 data: 5 4 4 5 4 3 4 4 6 6
25517 257/300 Write block 2302 len 28672 data: 5 4 5 4 4 3 5
25518 258/300 Read block 1904 len 40960
25519 259/300 Read block 2977 len 8192
25520 260/300 Write block 4633 len 20480 data: 7 7 7 7 6
25521 261/300 Read block 4645 len 28672
25522 262/300 Write block 159 len 20480 data: 7 9 10 10 9
25523 263/300 Read block 2407 len 24576
25524 264/300 Write block 3830 len 32768 data: 6 6 6 9 7 6 6 5
25525 265/300 Write block 3174 len 24576 data: 3 4 4 6 7 7
25526 266/300 Write block 375 len 28672 data: 7 9 9 9 10 11 8
25527 267/300 Read block 395 len 36864
25528 268/300 Write block 4286 len 24576 data: 5 6 5 5 5 4
25529 269/300 Write block 157 len 36864 data: 8 8 8 10 11 11 10 6 5
25530 270/300 Read block 4452 len 32768
25531 271/300 Write block 2624 len 16384 data: 4 4 4 4
25532 272/300 Write block 1275 len 28672 data: 3 4 4 5 5 6 7
25533 273/300 Flush
25534 274/300 Read block 373 len 40960
25535 275/300 Write block 619 len 32768 data: 10 9 5 4 5 5 5 5
25536 276/300 Read block 3227 len 40960
25537 277/300 Flush
25538 278/300 Write block 3742 len 20480 data: 7 6 6 6 5
25539 279/300 Write block 1364 len 28672 data: 5 5 5 5 5 5 3
25540 280/300 Read block 4133 len 4096
25541 281/300 Write block 1012 len 28672 data: 10 8 8 5 5 7 5
25542 282/300 Write block 1187 len 40960 data: 5 9 9 8 9 9 8 8 8 6
25543 283/300 Read block 3715 len 28672
25544 284/300 Read block 1420 len 4096
25545 285/300 Flush
25546 286/300 Flush
25547 287/300 Flush
25548 288/300 Read block 3132 len 36864
25549 289/300 Read block 3260 len 28672
25550 290/300 Write block 406 len 40960 data: 3 2 3 3 3 3 2 2 3 3
25551 291/300 Write block 1423 len 24576 data: 7 3 5 5 5 7
25552 292/300 Write block 347 len 40960 data: 5 7 6 10 11 12 14 14 11 9
25553 293/300 Flush
25554 294/300 Write block 1078 len 20480 data: 5 7 8 8 8
25555 295/300 Read block 4782 len 12288
25556 296/300 Read block 4014 len 24576
25557 297/300 Read block 3508 len 32768
25558 298/300 Write block 4389 len 12288 data: 7 7 9
25559 299/300 Read block 192 len 12288
25560 300/300 Write block 3272 len 8192 data: 5 5
25561 Main task has work to do, go find it
25562 got dsc Start(2)
25563 start 2
25564 Feb 15 03:02:36.201 INFO request completed, latency_us: 206, response_code: 204, uri: /start/cid/2, method: POST, req_id: ec318835-4ce0-4d2a-a4db-08b27e1f28b8, remote_addr: 127.0.0.1:54744, local_addr: 127.0.0.1:9998
25565 Replay: started 2, returned:Ok(())
25566 ----------------------------------------------------------------
25567 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:300
25568 States: Active Active Offline
25569 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
25570 8139 Acked 9138 Flush 0 Done Done New true
25571 8140 Acked 9139 Write 3 Done Done New true
25572 8141 Acked 9140 Flush 0 Done Done New true
25573 8142 Acked 9141 Write 7 Done Done New false
25574 8143 Acked 9142 Read 8 Done Done New false
25575 8144 Acked 9143 Read 2 Done Done New false
25576 8145 Acked 9144 Read 9 Done Done New false
25577 8146 Acked 9145 Read 8 Done Done New false
25578 8147 Acked 9146 Read 3 Done Done New false
25579 8148 Acked 9147 Write 8 Done Done New false
25580 8149 Acked 9148 Read 7 Done Done New false
25581 8150 Acked 9149 Write 4 Done Done New false
25582 8151 Acked 9150 Read 9 Done Done New false
25583 8152 Acked 9151 Read 9 Done Done New false
25584 8153 Acked 9152 Write 2 Done Done New false
25585 8154 Acked 9153 Read 9 Done Done New false
25586 8155 Acked 9154 Read 10 Done Done New false
25587 8156 Acked 9155 Read 3 Done Done New false
25588 8157 Acked 9156 Read 4 Done Done New false
25589 8158 Acked 9157 Read 4 Done Done New false
25590 8159 Acked 9158 Read 10 Done Done New false
25591 8160 Acked 9159 Read 9 Done Done New false
25592 8161 Acked 9160 Read 5 Done Done New false
25593 8162 Acked 9161 Read 8 Done Done New false
25594 8163 Acked 9162 Flush 0 Done Done New false
25595 8164 Acked 9163 Read 7 Done Done New false
25596 8165 Acked 9164 Flush 0 Done Done New false
25597 8166 Acked 9165 Write 10 Done Done New false
25598 8167 Acked 9166 Read 2 Done Done New false
25599 8168 Acked 9167 Read 2 Done Done New false
25600 8169 Acked 9168 Read 2 Done Done New false
25601 8170 Acked 9169 Write 10 Done Done New false
25602 8171 Acked 9170 Read 10 Done Done New false
25603 8172 Acked 9171 Read 10 Done Done New false
25604 8173 Acked 9172 Write 9 Done Done New false
25605 8174 Acked 9173 Read 5 Done Done New false
25606 8175 Acked 9174 Read 7 Done Done New false
25607 8176 Acked 9175 Flush 0 Done Done New false
25608 8177 Acked 9176 Read 6 Done Done New false
25609 8178 Acked 9177 Write 6 Done Done New false
25610 8179 Acked 9178 Read 7 Done Done New false
25611 8180 Acked 9179 Read 6 Done Done New false
25612 8181 Acked 9180 Read 6 Done Done New false
25613 8182 Acked 9181 Read 1 Done Done New false
25614 8183 Acked 9182 Read 3 Done Done New false
25615 8184 Acked 9183 Read 4 Done Done New false
25616 8185 Acked 9184 Write 9 Done Done New false
25617 8186 Acked 9185 Read 3 Done Done New false
25618 8187 Acked 9186 Read 2 Done Done New false
25619 8188 Acked 9187 Write 7 Done Done New false
25620 8189 Acked 9188 Read 7 Done Done New false
25621 8190 Acked 9189 Write 4 Done Done New false
25622 8191 Acked 9190 Read 8 Done Done New false
25623 8192 Acked 9191 Write 10 Done Done New false
25624 8193 Acked 9192 Read 4 Done Done New false
25625 8194 Acked 9193 Read 3 Done Done New false
25626 8195 Acked 9194 Flush 0 Done Done New false
25627 8196 Acked 9195 Read 8 Done Done New false
25628 8197 Acked 9196 Read 2 Done Done New false
25629 8198 Acked 9197 Read 6 Done Done New false
25630 8199 Acked 9198 Flush 0 Done Done New false
25631 8200 Acked 9199 Read 2 Done Done New false
25632 8201 Acked 9200 Read 10 Done Done New false
25633 8202 Acked 9201 Read 5 Done Done New false
25634 8203 Acked 9202 Write 10 Done Done New false
25635 8204 Acked 9203 Write 5 Done Done New false
25636 8205 Acked 9204 Read 5 Done Done New false
25637 8206 Acked 9205 Read 1 Done Done New false
25638 8207 Acked 9206 Write 4 Done Done New false
25639 8208 Acked 9207 Read 6 Done Done New false
25640 8209 Acked 9208 Write 8 Done Done New false
25641 8210 Acked 9209 Write 9 Done Done New false
25642 8211 Acked 9210 Read 6 Done Done New false
25643 8212 Acked 9211 Read 4 Done Done New false
25644 8213 Acked 9212 Write 8 Done Done New false
25645 8214 Acked 9213 Write 9 Done Done New false
25646 8215 Acked 9214 Write 6 Done Done New false
25647 8216 Acked 9215 Read 4 Done Done New false
25648 8217 Acked 9216 Read 2 Done Done New false
25649 8218 Acked 9217 Read 2 Done Done New false
25650 8219 Acked 9218 Read 4 Done Done New false
25651 8220 Acked 9219 Read 9 Done Done New false
25652 8221 Acked 9220 Read 6 Done Done New false
25653 8222 Acked 9221 Read 6 Done Done New false
25654 8223 Acked 9222 Write 7 Done Done New false
25655 8224 Acked 9223 Write 6 Done Done New false
25656 8225 Acked 9224 Write 6 Done Done New false
25657 8226 Acked 9225 Flush 0 Done Done New false
25658 8227 Acked 9226 Write 9 Done Done New false
25659 8228 Acked 9227 Read 4 Done Done New false
25660 8229 Acked 9228 Read 2 Done Done New false
25661 8230 Acked 9229 Read 4 Done Done New false
25662 8231 Acked 9230 Flush 0 Done Done New false
25663 8232 Acked 9231 Read 7 Done Done New false
25664 8233 Acked 9232 Read 1 Done Done New false
25665 8234 Acked 9233 Write 7 Done Done New false
25666 8235 Acked 9234 Read 1 Done Done New false
25667 8236 Acked 9235 Write 10 Done Done New false
25668 8237 Acked 9236 Read 7 Done Done New false
25669 8238 Acked 9237 Read 5 Done Done New false
25670 8239 Acked 9238 Flush 0 Done Done New false
25671 8240 Acked 9239 Write 8 Done Done New false
25672 8241 Acked 9240 Flush 0 Done Done New false
25673 8242 Acked 9241 Write 6 Done Done New false
25674 8243 Acked 9242 Read 8 Done Done New false
25675 8244 Acked 9243 Read 4 Done Done New false
25676 8245 Acked 9244 Write 5 Done Done New false
25677 8246 Acked 9245 Write 1 Done Done New false
25678 8247 Acked 9246 Write 9 Done Done New false
25679 8248 Acked 9247 Write 8 Done Done New false
25680 8249 Acked 9248 Write 7 Done Done New false
25681 8250 Acked 9249 Write 9 Done Done New false
25682 8251 Acked 9250 Read 4 Done Done New false
25683 8252 Acked 9251 Read 1 Done Done New false
25684 8253 Acked 9252 Flush 0 Done Done New false
25685 8254 Acked 9253 Read 4 Done Done New false
25686 8255 Acked 9254 Write 6 Done Done New false
25687 8256 Acked 9255 Read 8 Done Done New false
25688 8257 Acked 9256 Write 5 Done Done New false
25689 8258 Acked 9257 Flush 0 Done Done New false
25690 8259 Acked 9258 Write 3 Done Done New false
25691 8260 Acked 9259 Flush 0 Done Done New false
25692 8261 Acked 9260 Write 5 Done Done New false
25693 8262 Acked 9261 Read 10 Done Done New false
25694 8263 Acked 9262 Write 9 Done Done New false
25695 8264 Acked 9263 Write 6 Done Done New false
25696 8265 Acked 9264 Read 10 Done Done New false
25697 8266 Acked 9265 Write 7 Done Done New false
25698 8267 Acked 9266 Write 4 Done Done New false
25699 8268 Acked 9267 Read 6 Done Done New false
25700 8269 Acked 9268 Read 1 Done Done New false
25701 8270 Acked 9269 Write 9 Done Done New false
25702 8271 Acked 9270 Read 4 Done Done New false
25703 8272 Acked 9271 Read 10 Done Done New false
25704 8273 Acked 9272 Write 9 Done Done New false
25705 8274 Acked 9273 Read 3 Done Done New false
25706 8275 Acked 9274 Write 10 Done Done New false
25707 8276 Acked 9275 Write 6 Done Done New false
25708 8277 Acked 9276 Write 10 Done Done New false
25709 8278 Acked 9277 Read 1 Done Done New false
25710 8279 Acked 9278 Write 7 Done Done New false
25711 8280 Acked 9279 Read 9 Done Done New false
25712 8281 Acked 9280 Read 4 Done Done New false
25713 8282 Acked 9281 Read 9 Done Done New false
25714 8283 Acked 9282 Read 2 Done Done New false
25715 8284 Acked 9283 Read 4 Done Done New false
25716 8285 Acked 9284 Write 1 Done Done New false
25717 8286 Acked 9285 Read 4 Done Done New false
25718 8287 Acked 9286 Read 9 Done Done New false
25719 8288 Acked 9287 Read 3 Done Done New false
25720 8289 Acked 9288 Read 7 Done Done New false
25721 8290 Acked 9289 Write 8 Done Done New false
25722 8291 Acked 9290 Flush 0 Done Done New false
25723 8292 Acked 9291 Write 8 Done Done New false
25724 8293 Acked 9292 Read 7 Done Done New false
25725 8294 Acked 9293 Read 9 Done Done New false
25726 8295 Acked 9294 Read 5 Done Done New false
25727 8296 Acked 9295 Write 1 Done Done New false
25728 8297 Acked 9296 Read 5 Done Done New false
25729 8298 Acked 9297 Write 5 Done Done New false
25730 8299 Acked 9298 Write 8 Done Done New false
25731 8300 Acked 9299 Flush 0 Done Done New false
25732 8301 Acked 9300 Read 1 Done Done New false
25733 8302 Acked 9301 Read 9 Done Done New false
25734 8303 Acked 9302 Write 10 Done Done New false
25735 8304 Acked 9303 Write 1 Done Done New false
25736 8305 Acked 9304 Flush 0 Done Done New false
25737 8306 Acked 9305 Write 6 Done Done New false
25738 8307 Acked 9306 Read 3 Done Done New false
25739 8308 Acked 9307 Read 8 Done Done New false
25740 8309 Acked 9308 Write 9 Done Done New false
25741 8310 Acked 9309 Read 7 Done Done New false
25742 8311 Acked 9310 Read 8 Done Done New false
25743 8312 Acked 9311 Read 4 Done Done New false
25744 8313 Acked 9312 Write 3 Done Done New false
25745 8314 Acked 9313 Read 7 Done Done New false
25746 8315 Acked 9314 Write 10 Done Done New false
25747 8316 Acked 9315 Write 1 Done Done New false
25748 8317 Acked 9316 Read 2 Done Done New false
25749 8318 Acked 9317 Read 2 Done Done New false
25750 8319 Acked 9318 Read 2 Done Done New false
25751 8320 Acked 9319 Read 1 Done Done New false
25752 8321 Acked 9320 Read 1 Done Done New false
25753 8322 Acked 9321 Write 7 Done Done New false
25754 8323 Acked 9322 Write 3 Done Done New false
25755 8324 Acked 9323 Write 9 Done Done New false
25756 8325 Acked 9324 Write 7 Done Done New false
25757 8326 Acked 9325 Read 10 Done Done New false
25758 8327 Acked 9326 Read 1 Done Done New false
25759 8328 Acked 9327 Write 2 Done Done New false
25760 8329 Acked 9328 Read 10 Done Done New false
25761 8330 Acked 9329 Write 6 Done Done New false
25762 8331 Acked 9330 Read 7 Done Done New false
25763 8332 Acked 9331 Write 9 Done Done New false
25764 8333 Acked 9332 Write 10 Done Done New false
25765 8334 Acked 9333 Read 7 Done Done New false
25766 8335 Acked 9334 Write 6 Done Done New false
25767 8336 Acked 9335 Read 6 Done Done New false
25768 8337 Acked 9336 Read 1 Done Done New false
25769 8338 Acked 9337 Read 4 Done Done New false
25770 8339 Acked 9338 Write 9 Done Done New false
25771 8340 Acked 9339 Write 7 Done Done New false
25772 8341 Acked 9340 Write 8 Done Done New false
25773 8342 Acked 9341 Read 2 Done Done New false
25774 8343 Acked 9342 Read 10 Done Done New false
25775 8344 Acked 9343 Read 5 Done Done New false
25776 8345 Acked 9344 Flush 0 Done Done New false
25777 8346 Acked 9345 Read 1 Done Done New false
25778 8347 Acked 9346 Write 10 Done Done New false
25779 8348 Acked 9347 Write 2 Done Done New false
25780 8349 Acked 9348 Read 9 Done Done New false
25781 8350 Acked 9349 Write 7 Done Done New false
25782 8351 Acked 9350 Read 6 Done Done New false
25783 8352 Acked 9351 Read 6 Done Done New false
25784 8353 Acked 9352 Read 1 Done Done New false
25785 8354 Acked 9353 Write 9 Done Done New false
25786 8355 Acked 9354 Write 3 Done Done New false
25787 8356 Acked 9355 Flush 0 Done Done New false
25788 8357 Acked 9356 Read 4 Done Done New false
25789 8358 Acked 9357 Write 10 Done Done New false
25790 8359 Acked 9358 Read 9 Done Done New false
25791 8360 Acked 9359 Write 6 Done Done New false
25792 8361 Acked 9360 Write 1 Done Done New false
25793 8362 Acked 9361 Write 9 Done Done New false
25794 8363 Acked 9362 Write 3 Done Done New false
25795 8364 Acked 9363 Read 4 Done Done New false
25796 8365 Acked 9364 Read 7 Done Done New false
25797 8366 Acked 9365 Write 3 Done Done New false
25798 8367 Acked 9366 Read 1 Done Done New false
25799 8368 Acked 9367 Write 7 Done Done New false
25800 8369 Acked 9368 Write 7 Done Done New false
25801 8370 Acked 9369 Read 9 Done Done New false
25802 8371 Acked 9370 Write 10 Done Done New false
25803 8372 Acked 9371 Write 9 Done Done New false
25804 8373 Acked 9372 Flush 0 Done Done New false
25805 8374 Acked 9373 Flush 0 Done Done New false
25806 8375 Acked 9374 Read 9 Done Done New false
25807 8376 Acked 9375 Read 7 Done Done New false
25808 8377 Acked 9376 Write 9 Done Done New false
25809 8378 Acked 9377 Read 2 Done Done New false
25810 8379 Acked 9378 Read 5 Done Done New false
25811 8380 Acked 9379 Read 8 Done Done New false
25812 8381 Acked 9380 Write 5 Done Done New false
25813 8382 Acked 9381 Read 8 Done Done New false
25814 8383 Acked 9382 Read 3 Done Done New false
25815 8384 Acked 9383 Read 3 Done Done New false
25816 8385 Acked 9384 Write 3 Done Done New false
25817 8386 Acked 9385 Flush 0 Done Done New false
25818 8387 Acked 9386 Read 5 Done Done New false
25819 8388 Acked 9387 Write 5 Done Done New false
25820 8389 Acked 9388 Flush 0 Done Done New false
25821 8390 Acked 9389 Read 9 Done Done New false
25822 8391 Acked 9390 Flush 0 Done Done New false
25823 8392 Acked 9391 Write 4 Done Done New false
25824 8393 Acked 9392 Flush 0 Done Done New false
25825 8394 Acked 9393 Write 10 Done Done New false
25826 8395 Acked 9394 Write 7 Done Done New false
25827 8396 Acked 9395 Read 10 Done Done New false
25828 8397 Acked 9396 Read 2 Done Done New false
25829 8398 Acked 9397 Write 5 Done Done New false
25830 8399 Acked 9398 Read 7 Done Done New false
25831 8400 Acked 9399 Write 5 Done Done New false
25832 8401 Acked 9400 Read 6 Done Done New false
25833 8402 Acked 9401 Write 8 Done Done New false
25834 8403 Acked 9402 Write 6 Done Done New false
25835 8404 Acked 9403 Write 7 Done Done New false
25836 8405 Acked 9404 Read 9 Done Done New false
25837 8406 Acked 9405 Write 6 Done Done New false
25838 8407 Acked 9406 Write 9 Done Done New false
25839 8408 Acked 9407 Read 8 Done Done New false
25840 8409 Acked 9408 Write 4 Done Done New false
25841 8410 Acked 9409 Write 7 Done Done New false
25842 8411 Acked 9410 Flush 0 Done Done New false
25843 8412 Acked 9411 Read 10 Done Done New false
25844 8413 Acked 9412 Write 8 Done Done New false
25845 8414 Acked 9413 Read 10 Done Done New false
25846 8415 Acked 9414 Flush 0 Done Done New false
25847 8416 Acked 9415 Write 5 Done Done New false
25848 8417 Acked 9416 Write 7 Done Done New false
25849 8418 Acked 9417 Read 1 Done Done New false
25850 8419 Acked 9418 Write 7 Done Done New false
25851 8420 Acked 9419 Write 10 Done Done New false
25852 8421 Acked 9420 Read 7 Done Done New false
25853 8422 Acked 9421 Read 1 Done Done New false
25854 8423 Acked 9422 Flush 0 Done Done New false
25855 8424 Acked 9423 Flush 0 Done Done New false
25856 8425 Acked 9424 Flush 0 Done Done New false
25857 8426 Acked 9425 Read 9 Done Done New false
25858 8427 Acked 9426 Read 7 Done Done New false
25859 8428 Acked 9427 Write 10 Done Done New false
25860 8429 Acked 9428 Write 6 Done Done New false
25861 8430 Acked 9429 Write 10 Done Done New false
25862 8431 Acked 9430 Flush 0 Done Done New false
25863 8432 Acked 9431 Write 5 Done Done New false
25864 8433 Acked 9432 Read 3 Done Done New false
25865 8434 Acked 9433 Read 6 Done Done New false
25866 8435 Acked 9434 Read 8 Done Done New false
25867 8436 Acked 9435 Write 3 Done Done New false
25868 8437 Acked 9436 Read 3 Done Done New false
25869 8438 Acked 9437 Write 2 Done Done New false
25870 STATES DS:0 DS:1 DS:2 TOTAL
25871 New 0 0 300 300
25872 Sent 0 0 0 0
25873 Done 300 300 0 600
25874 Skipped 0 0 0 0
25875 Error 0 0 0 0
25876 Last Flush: 9430 9430 9137
25877 Downstairs last five completed: 9137 9136 9135 9134 9133
25878 Upstairs last five completed: 8438 8437 8436 8435 8434
25879 CLIENT: Up:0 ds:300 act:2
258802024-02-15T03:02:36.859ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
258812024-02-15T03:02:36.859ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
258822024-02-15T03:02:36.859ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
258832024-02-15T03:02:36.859ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
258842024-02-15T03:02:36.860ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
258852024-02-15T03:02:36.860ZINFOcrucible: [2] client re-new 301 jobs since flush 9137 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
258862024-02-15T03:02:36.860ZINFOcrucible: client task is exiting = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
258872024-02-15T03:02:37.860ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
258882024-02-15T03:02:37.860ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
258892024-02-15T03:02:37.860ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
258902024-02-15T03:02:37.860ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
258912024-02-15T03:02:37.860ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
258922024-02-15T03:02:37.860ZINFOcrucible: [2] client re-new 301 jobs since flush 9137 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
258932024-02-15T03:02:37.861ZINFOcrucible: client task is exiting = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
258942024-02-15T03:02:38.862ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
258952024-02-15T03:02:38.862ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
258962024-02-15T03:02:38.862ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
258972024-02-15T03:02:38.862ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
258982024-02-15T03:02:38.862ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
258992024-02-15T03:02:38.862ZINFOcrucible: [2] client re-new 301 jobs since flush 9137 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
259002024-02-15T03:02:38.862ZINFOcrucible: client task is exiting = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
25901 [8830] got start action
25902 [8830] I am going to restart
25903 Starting downstairs at port 8830
25904 Make output file at "/tmp/dsc/downstairs-8830.txt"
25905 [8830][2] reports Starting
259062024-02-15T03:02:39.863ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
259072024-02-15T03:02:39.863ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
259082024-02-15T03:02:39.863ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
259092024-02-15T03:02:39.863ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
259102024-02-15T03:02:39.863ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
259112024-02-15T03:02:39.863ZINFOcrucible: [2] client re-new 301 jobs since flush 9137 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
259122024-02-15T03:02:39.863ZINFOcrucible: client task is exiting = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
25913 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1195)
25914 [8830][2] reports Running
25915 ----------------------------------------------------------------
25916 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
25917 States: Active Active Offline
25918 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
25919 8139 Acked 9138 Flush 0 Done Done New true
25920 8140 Acked 9139 Write 3 Done Done New true
25921 8141 Acked 9140 Flush 0 Done Done New true
25922 8142 Acked 9141 Write 7 Done Done New true
25923 8143 Acked 9142 Read 8 Done Done New true
25924 8144 Acked 9143 Read 2 Done Done New true
25925 8145 Acked 9144 Read 9 Done Done New true
25926 8146 Acked 9145 Read 8 Done Done New true
25927 8147 Acked 9146 Read 3 Done Done New true
25928 8148 Acked 9147 Write 8 Done Done New true
25929 8149 Acked 9148 Read 7 Done Done New true
25930 8150 Acked 9149 Write 4 Done Done New true
25931 8151 Acked 9150 Read 9 Done Done New true
25932 8152 Acked 9151 Read 9 Done Done New true
25933 8153 Acked 9152 Write 2 Done Done New true
25934 8154 Acked 9153 Read 9 Done Done New true
25935 8155 Acked 9154 Read 10 Done Done New true
25936 8156 Acked 9155 Read 3 Done Done New true
25937 8157 Acked 9156 Read 4 Done Done New true
25938 8158 Acked 9157 Read 4 Done Done New true
25939 8159 Acked 9158 Read 10 Done Done New true
25940 8160 Acked 9159 Read 9 Done Done New true
25941 8161 Acked 9160 Read 5 Done Done New true
25942 8162 Acked 9161 Read 8 Done Done New true
25943 8163 Acked 9162 Flush 0 Done Done New true
25944 8164 Acked 9163 Read 7 Done Done New true
25945 8165 Acked 9164 Flush 0 Done Done New true
25946 8166 Acked 9165 Write 10 Done Done New true
25947 8167 Acked 9166 Read 2 Done Done New true
25948 8168 Acked 9167 Read 2 Done Done New true
25949 8169 Acked 9168 Read 2 Done Done New true
25950 8170 Acked 9169 Write 10 Done Done New true
25951 8171 Acked 9170 Read 10 Done Done New true
25952 8172 Acked 9171 Read 10 Done Done New true
25953 8173 Acked 9172 Write 9 Done Done New true
25954 8174 Acked 9173 Read 5 Done Done New true
25955 8175 Acked 9174 Read 7 Done Done New true
25956 8176 Acked 9175 Flush 0 Done Done New true
25957 8177 Acked 9176 Read 6 Done Done New true
25958 8178 Acked 9177 Write 6 Done Done New true
25959 8179 Acked 9178 Read 7 Done Done New true
25960 8180 Acked 9179 Read 6 Done Done New true
25961 8181 Acked 9180 Read 6 Done Done New true
25962 8182 Acked 9181 Read 1 Done Done New true
25963 8183 Acked 9182 Read 3 Done Done New true
25964 8184 Acked 9183 Read 4 Done Done New true
25965 8185 Acked 9184 Write 9 Done Done New true
25966 8186 Acked 9185 Read 3 Done Done New true
25967 8187 Acked 9186 Read 2 Done Done New true
25968 8188 Acked 9187 Write 7 Done Done New true
25969 8189 Acked 9188 Read 7 Done Done New true
25970 8190 Acked 9189 Write 4 Done Done New true
25971 8191 Acked 9190 Read 8 Done Done New true
25972 8192 Acked 9191 Write 10 Done Done New true
25973 8193 Acked 9192 Read 4 Done Done New true
25974 8194 Acked 9193 Read 3 Done Done New true
25975 8195 Acked 9194 Flush 0 Done Done New true
25976 8196 Acked 9195 Read 8 Done Done New true
25977 8197 Acked 9196 Read 2 Done Done New true
25978 8198 Acked 9197 Read 6 Done Done New true
25979 8199 Acked 9198 Flush 0 Done Done New true
25980 8200 Acked 9199 Read 2 Done Done New true
25981 8201 Acked 9200 Read 10 Done Done New true
25982 8202 Acked 9201 Read 5 Done Done New true
25983 8203 Acked 9202 Write 10 Done Done New true
25984 8204 Acked 9203 Write 5 Done Done New true
25985 8205 Acked 9204 Read 5 Done Done New true
25986 8206 Acked 9205 Read 1 Done Done New true
25987 8207 Acked 9206 Write 4 Done Done New true
25988 8208 Acked 9207 Read 6 Done Done New true
25989 8209 Acked 9208 Write 8 Done Done New true
25990 8210 Acked 9209 Write 9 Done Done New true
25991 8211 Acked 9210 Read 6 Done Done New true
25992 8212 Acked 9211 Read 4 Done Done New true
25993 8213 Acked 9212 Write 8 Done Done New true
25994 8214 Acked 9213 Write 9 Done Done New true
25995 8215 Acked 9214 Write 6 Done Done New true
25996 8216 Acked 9215 Read 4 Done Done New true
25997 8217 Acked 9216 Read 2 Done Done New true
25998 8218 Acked 9217 Read 2 Done Done New true
25999 8219 Acked 9218 Read 4 Done Done New true
26000 8220 Acked 9219 Read 9 Done Done New true
26001 8221 Acked 9220 Read 6 Done Done New true
26002 8222 Acked 9221 Read 6 Done Done New true
26003 8223 Acked 9222 Write 7 Done Done New true
26004 8224 Acked 9223 Write 6 Done Done New true
26005 8225 Acked 9224 Write 6 Done Done New true
26006 8226 Acked 9225 Flush 0 Done Done New true
26007 8227 Acked 9226 Write 9 Done Done New true
26008 8228 Acked 9227 Read 4 Done Done New true
26009 8229 Acked 9228 Read 2 Done Done New true
26010 8230 Acked 9229 Read 4 Done Done New true
26011 8231 Acked 9230 Flush 0 Done Done New true
26012 8232 Acked 9231 Read 7 Done Done New true
26013 8233 Acked 9232 Read 1 Done Done New true
26014 8234 Acked 9233 Write 7 Done Done New true
26015 8235 Acked 9234 Read 1 Done Done New true
26016 8236 Acked 9235 Write 10 Done Done New true
26017 8237 Acked 9236 Read 7 Done Done New true
26018 8238 Acked 9237 Read 5 Done Done New true
26019 8239 Acked 9238 Flush 0 Done Done New true
26020 8240 Acked 9239 Write 8 Done Done New true
26021 8241 Acked 9240 Flush 0 Done Done New true
26022 8242 Acked 9241 Write 6 Done Done New true
26023 8243 Acked 9242 Read 8 Done Done New true
26024 8244 Acked 9243 Read 4 Done Done New true
26025 8245 Acked 9244 Write 5 Done Done New true
26026 8246 Acked 9245 Write 1 Done Done New true
26027 8247 Acked 9246 Write 9 Done Done New true
26028 8248 Acked 9247 Write 8 Done Done New true
26029 8249 Acked 9248 Write 7 Done Done New true
26030 8250 Acked 9249 Write 9 Done Done New true
26031 8251 Acked 9250 Read 4 Done Done New true
26032 8252 Acked 9251 Read 1 Done Done New true
26033 8253 Acked 9252 Flush 0 Done Done New true
26034 8254 Acked 9253 Read 4 Done Done New true
26035 8255 Acked 9254 Write 6 Done Done New true
26036 8256 Acked 9255 Read 8 Done Done New true
26037 8257 Acked 9256 Write 5 Done Done New true
26038 8258 Acked 9257 Flush 0 Done Done New true
26039 8259 Acked 9258 Write 3 Done Done New true
26040 8260 Acked 9259 Flush 0 Done Done New true
26041 8261 Acked 9260 Write 5 Done Done New true
26042 8262 Acked 9261 Read 10 Done Done New true
26043 8263 Acked 9262 Write 9 Done Done New true
26044 8264 Acked 9263 Write 6 Done Done New true
26045 8265 Acked 9264 Read 10 Done Done New true
26046 8266 Acked 9265 Write 7 Done Done New true
26047 8267 Acked 9266 Write 4 Done Done New true
26048 8268 Acked 9267 Read 6 Done Done New true
26049 8269 Acked 9268 Read 1 Done Done New true
26050 8270 Acked 9269 Write 9 Done Done New true
26051 8271 Acked 9270 Read 4 Done Done New true
26052 8272 Acked 9271 Read 10 Done Done New true
26053 8273 Acked 9272 Write 9 Done Done New true
26054 8274 Acked 9273 Read 3 Done Done New true
26055 8275 Acked 9274 Write 10 Done Done New true
26056 8276 Acked 9275 Write 6 Done Done New true
26057 8277 Acked 9276 Write 10 Done Done New true
26058 8278 Acked 9277 Read 1 Done Done New true
26059 8279 Acked 9278 Write 7 Done Done New true
26060 8280 Acked 9279 Read 9 Done Done New true
26061 8281 Acked 9280 Read 4 Done Done New true
26062 8282 Acked 9281 Read 9 Done Done New true
26063 8283 Acked 9282 Read 2 Done Done New true
26064 8284 Acked 9283 Read 4 Done Done New true
26065 8285 Acked 9284 Write 1 Done Done New true
26066 8286 Acked 9285 Read 4 Done Done New true
26067 8287 Acked 9286 Read 9 Done Done New true
26068 8288 Acked 9287 Read 3 Done Done New true
26069 8289 Acked 9288 Read 7 Done Done New true
26070 8290 Acked 9289 Write 8 Done Done New true
26071 8291 Acked 9290 Flush 0 Done Done New true
26072 8292 Acked 9291 Write 8 Done Done New true
26073 8293 Acked 9292 Read 7 Done Done New true
26074 8294 Acked 9293 Read 9 Done Done New true
26075 8295 Acked 9294 Read 5 Done Done New true
26076 8296 Acked 9295 Write 1 Done Done New true
26077 8297 Acked 9296 Read 5 Done Done New true
26078 8298 Acked 9297 Write 5 Done Done New true
26079 8299 Acked 9298 Write 8 Done Done New true
26080 8300 Acked 9299 Flush 0 Done Done New true
26081 8301 Acked 9300 Read 1 Done Done New true
26082 8302 Acked 9301 Read 9 Done Done New true
26083 8303 Acked 9302 Write 10 Done Done New true
26084 8304 Acked 9303 Write 1 Done Done New true
26085 8305 Acked 9304 Flush 0 Done Done New true
26086 8306 Acked 9305 Write 6 Done Done New true
26087 8307 Acked 9306 Read 3 Done Done New true
26088 8308 Acked 9307 Read 8 Done Done New true
26089 8309 Acked 9308 Write 9 Done Done New true
26090 8310 Acked 9309 Read 7 Done Done New true
26091 8311 Acked 9310 Read 8 Done Done New true
26092 8312 Acked 9311 Read 4 Done Done New true
26093 8313 Acked 9312 Write 3 Done Done New true
26094 8314 Acked 9313 Read 7 Done Done New true
26095 8315 Acked 9314 Write 10 Done Done New true
26096 8316 Acked 9315 Write 1 Done Done New true
26097 8317 Acked 9316 Read 2 Done Done New true
26098 8318 Acked 9317 Read 2 Done Done New true
26099 8319 Acked 9318 Read 2 Done Done New true
26100 8320 Acked 9319 Read 1 Done Done New true
26101 8321 Acked 9320 Read 1 Done Done New true
26102 8322 Acked 9321 Write 7 Done Done New true
26103 8323 Acked 9322 Write 3 Done Done New true
26104 8324 Acked 9323 Write 9 Done Done New true
26105 8325 Acked 9324 Write 7 Done Done New true
26106 8326 Acked 9325 Read 10 Done Done New true
26107 8327 Acked 9326 Read 1 Done Done New true
26108 8328 Acked 9327 Write 2 Done Done New true
26109 8329 Acked 9328 Read 10 Done Done New true
26110 8330 Acked 9329 Write 6 Done Done New true
26111 8331 Acked 9330 Read 7 Done Done New true
26112 8332 Acked 9331 Write 9 Done Done New true
26113 8333 Acked 9332 Write 10 Done Done New true
26114 8334 Acked 9333 Read 7 Done Done New true
26115 8335 Acked 9334 Write 6 Done Done New true
26116 8336 Acked 9335 Read 6 Done Done New true
26117 8337 Acked 9336 Read 1 Done Done New true
26118 8338 Acked 9337 Read 4 Done Done New true
26119 8339 Acked 9338 Write 9 Done Done New true
26120 8340 Acked 9339 Write 7 Done Done New true
26121 8341 Acked 9340 Write 8 Done Done New true
26122 8342 Acked 9341 Read 2 Done Done New true
26123 8343 Acked 9342 Read 10 Done Done New true
26124 8344 Acked 9343 Read 5 Done Done New true
26125 8345 Acked 9344 Flush 0 Done Done New true
26126 8346 Acked 9345 Read 1 Done Done New true
26127 8347 Acked 9346 Write 10 Done Done New true
26128 8348 Acked 9347 Write 2 Done Done New true
26129 8349 Acked 9348 Read 9 Done Done New true
26130 8350 Acked 9349 Write 7 Done Done New true
26131 8351 Acked 9350 Read 6 Done Done New true
26132 8352 Acked 9351 Read 6 Done Done New true
26133 8353 Acked 9352 Read 1 Done Done New true
26134 8354 Acked 9353 Write 9 Done Done New true
26135 8355 Acked 9354 Write 3 Done Done New true
26136 8356 Acked 9355 Flush 0 Done Done New true
26137 8357 Acked 9356 Read 4 Done Done New true
26138 8358 Acked 9357 Write 10 Done Done New true
26139 8359 Acked 9358 Read 9 Done Done New true
26140 8360 Acked 9359 Write 6 Done Done New true
26141 8361 Acked 9360 Write 1 Done Done New true
26142 8362 Acked 9361 Write 9 Done Done New true
26143 8363 Acked 9362 Write 3 Done Done New true
26144 8364 Acked 9363 Read 4 Done Done New true
26145 8365 Acked 9364 Read 7 Done Done New true
26146 8366 Acked 9365 Write 3 Done Done New true
26147 8367 Acked 9366 Read 1 Done Done New true
26148 8368 Acked 9367 Write 7 Done Done New true
26149 8369 Acked 9368 Write 7 Done Done New true
26150 8370 Acked 9369 Read 9 Done Done New true
26151 8371 Acked 9370 Write 10 Done Done New true
26152 8372 Acked 9371 Write 9 Done Done New true
26153 8373 Acked 9372 Flush 0 Done Done New true
26154 8374 Acked 9373 Flush 0 Done Done New true
26155 8375 Acked 9374 Read 9 Done Done New true
26156 8376 Acked 9375 Read 7 Done Done New true
26157 8377 Acked 9376 Write 9 Done Done New true
26158 8378 Acked 9377 Read 2 Done Done New true
26159 8379 Acked 9378 Read 5 Done Done New true
26160 8380 Acked 9379 Read 8 Done Done New true
26161 8381 Acked 9380 Write 5 Done Done New true
26162 8382 Acked 9381 Read 8 Done Done New true
26163 8383 Acked 9382 Read 3 Done Done New true
26164 8384 Acked 9383 Read 3 Done Done New true
26165 8385 Acked 9384 Write 3 Done Done New true
26166 8386 Acked 9385 Flush 0 Done Done New true
26167 8387 Acked 9386 Read 5 Done Done New true
26168 8388 Acked 9387 Write 5 Done Done New true
26169 8389 Acked 9388 Flush 0 Done Done New true
26170 8390 Acked 9389 Read 9 Done Done New true
26171 8391 Acked 9390 Flush 0 Done Done New true
26172 8392 Acked 9391 Write 4 Done Done New true
26173 8393 Acked 9392 Flush 0 Done Done New true
26174 8394 Acked 9393 Write 10 Done Done New true
26175 8395 Acked 9394 Write 7 Done Done New true
26176 8396 Acked 9395 Read 10 Done Done New true
26177 8397 Acked 9396 Read 2 Done Done New true
26178 8398 Acked 9397 Write 5 Done Done New true
26179 8399 Acked 9398 Read 7 Done Done New true
26180 8400 Acked 9399 Write 5 Done Done New true
26181 8401 Acked 9400 Read 6 Done Done New true
26182 8402 Acked 9401 Write 8 Done Done New true
26183 8403 Acked 9402 Write 6 Done Done New true
26184 8404 Acked 9403 Write 7 Done Done New true
26185 8405 Acked 9404 Read 9 Done Done New true
26186 8406 Acked 9405 Write 6 Done Done New true
26187 8407 Acked 9406 Write 9 Done Done New true
26188 8408 Acked 9407 Read 8 Done Done New true
26189 8409 Acked 9408 Write 4 Done Done New true
26190 8410 Acked 9409 Write 7 Done Done New true
26191 8411 Acked 9410 Flush 0 Done Done New true
26192 8412 Acked 9411 Read 10 Done Done New true
26193 8413 Acked 9412 Write 8 Done Done New true
26194 8414 Acked 9413 Read 10 Done Done New true
26195 8415 Acked 9414 Flush 0 Done Done New true
26196 8416 Acked 9415 Write 5 Done Done New true
26197 8417 Acked 9416 Write 7 Done Done New true
26198 8418 Acked 9417 Read 1 Done Done New true
26199 8419 Acked 9418 Write 7 Done Done New true
26200 8420 Acked 9419 Write 10 Done Done New true
26201 8421 Acked 9420 Read 7 Done Done New true
26202 8422 Acked 9421 Read 1 Done Done New true
26203 8423 Acked 9422 Flush 0 Done Done New true
26204 8424 Acked 9423 Flush 0 Done Done New true
26205 8425 Acked 9424 Flush 0 Done Done New true
26206 8426 Acked 9425 Read 9 Done Done New true
26207 8427 Acked 9426 Read 7 Done Done New true
26208 8428 Acked 9427 Write 10 Done Done New true
26209 8429 Acked 9428 Write 6 Done Done New true
26210 8430 Acked 9429 Write 10 Done Done New true
26211 8431 Acked 9430 Flush 0 Done Done New true
26212 8432 Acked 9431 Write 5 Done Done New true
26213 8433 Acked 9432 Read 3 Done Done New true
26214 8434 Acked 9433 Read 6 Done Done New true
26215 8435 Acked 9434 Read 8 Done Done New true
26216 8436 Acked 9435 Write 3 Done Done New true
26217 8437 Acked 9436 Read 3 Done Done New true
26218 8438 Acked 9437 Write 2 Done Done New true
26219 8439 Acked 9438 Flush 0 Done Done New true
26220 STATES DS:0 DS:1 DS:2 TOTAL
26221 New 0 0 301 301
26222 Sent 0 0 0 0
26223 Done 301 301 0 602
26224 Skipped 0 0 0 0
26225 Error 0 0 0 0
26226 Last Flush: 9438 9438 9137
26227 Downstairs last five completed: 9137 9136 9135 9134 9133
26228 Upstairs last five completed: 8439 8438 8437 8436 8435
26229 CLIENT: Up:0 ds:301 act:2
262302024-02-15T03:02:40.865ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
262312024-02-15T03:02:40.865ZINFOcrucible: ds_connection connected = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
262322024-02-15T03:02:40.866ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
262332024-02-15T03:02:40.867ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
262342024-02-15T03:02:40.867ZINFOcrucible: send last flush ID to this DS: 9137 = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
262352024-02-15T03:02:40.867ZINFOcrucible: Replied this last flush ID: 9137 = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
262362024-02-15T03:02:40.867ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
262372024-02-15T03:02:40.867ZINFOcrucible: [2] Transition from Offline to Replay = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
262382024-02-15T03:02:40.867ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
262392024-02-15T03:02:40.867ZINFOcrucible: [2] Transition from Replay to Active = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
26240 ----------------------------------------------------------------
26241 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
26242 Downstairs last five completed: 9438 9437 9436 9435 9434
26243 Upstairs last five completed: 8439 8438 8437 8436 8435
26244 CLIENT: Up:0 ds:0 act:3
26245 Replay: All jobs finished, all DS active.
26246 Main task has work to do, go find it
26247 got dsc Stop(1)
26248 stop 1
26249 Feb 15 03:02:44.208 INFO request completed, latency_us: 368, response_code: [8820] Got stop action so:false kr:false
26250 204, uri: /stop/cid/1, method: POST, req_id: 57683062-7eb5-4711-aab1-41a810087e8b, remote_addr: 127.0.0.1:54744, local_addr: 127.0.0.1:9998
262512024-02-15T03:02:44.208ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
262522024-02-15T03:02:44.208ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
262532024-02-15T03:02:44.208ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
26254 Feb 15 03:02:44.208 INFO request completed, latency_us: 146, response_code:{"msg":"Gone missing, transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-02-15T03:02:44.208916697Z","hostname":"ip-10-150-1-67.us-west-2.compute.internal","pid":1167,"client":"1","":"downstairs","session_id":"4f5a63e5-aa00-4890-9f6b-96c2e069fdb2"}
26255 200, uri: /state/cid/1, {"msg":"[1] client re-new 0 jobs since flush 9438","v":0,"name":"crucible","level":30,"time":"2024-02-15T03:02:44.208963661Z","hostname":"ip-10-150-1-67.us-west-2.compute.internal","pid":1167,"":"downstairs","":"downstairs","session_id":"4f5a63e5-aa00-4890-9f6b-96c2e069fdb2"}
26256 method{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-15T03:02:44.209002536Z","hostname":"ip-10-150-1-67.us-west-2.compute.internal","pid":1167,"":"io task","client":"1","":"downstairs","session_id":"4f5a63e5-aa00-4890-9f6b-96c2e069fdb2"}
26257 : GET, req_id: c08dd89a-589f-4e84-831c-2cf92229567d, remote_addr: 127.0.0.1:54744, local_addr: 127.0.0.1:9998
26258 [8820] Exited with: ExitStatus(unix_wait_status(9))
26259 [8820][1] reports Exit
262602024-02-15T03:02:45.210ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
262612024-02-15T03:02:45.210ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
262622024-02-15T03:02:45.210ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
262632024-02-15T03:02:45.210ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
262642024-02-15T03:02:45.210ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
262652024-02-15T03:02:45.210ZINFOcrucible: [1] client re-new 0 jobs since flush 9438 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
262662024-02-15T03:02:45.210ZINFOcrucible: client task is exiting = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
262672024-02-15T03:02:46.211ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
262682024-02-15T03:02:46.211ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
262692024-02-15T03:02:46.211ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
262702024-02-15T03:02:46.211ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
262712024-02-15T03:02:46.211ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
262722024-02-15T03:02:46.211ZINFOcrucible: [1] client re-new 0 jobs since flush 9438 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
262732024-02-15T03:02:46.211ZINFOcrucible: client task is exiting = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
262742024-02-15T03:02:47.212ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
262752024-02-15T03:02:47.213ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
262762024-02-15T03:02:47.213ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
262772024-02-15T03:02:47.213ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
262782024-02-15T03:02:47.213ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
262792024-02-15T03:02:47.213ZINFOcrucible: [1] client re-new 0 jobs since flush 9438 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
262802024-02-15T03:02:47.213ZINFOcrucible: client task is exiting = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
26281 Feb 15 03:02:48.211 INFO request completed, latency_us: 458, response_code: 200, uri: /state/cid/1, method: GET, req_id: 1007c4d6-cff8-428f-a85f-2eb3c4806d52, remote_addr: 127.0.0.1:54744, local_addr: 127.0.0.1:9998
26282 001/300 Read block 540 len 24576
26283 002/300 Read block 1498 len 4096
262842024-02-15T03:02:48.213ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
262852024-02-15T03:02:48.213ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
262862024-02-15T03:02:48.213ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
262872024-02-15T03:02:48.213ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
262882024-02-15T03:02:48.213ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
262892024-02-15T03:02:48.213ZINFOcrucible: [1] client re-new 2 jobs since flush 9438 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
262902024-02-15T03:02:48.213ZINFOcrucible: client task is exiting = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
26291 003/300 Write block 3908 len 24576 data: 10 10 8 6 6 6
26292 004/300 Write block 2363 len 24576 data: 6 6 5 6 5 4
26293 005/300 Read block 3569 len 40960
26294 006/300 Read block 3852 len 28672
26295 007/300 Read block 841 len 8192
26296 008/300 Write block 4353 len 16384 data: 4 5 5 4
26297 009/300 Flush
26298 010/300 Read block 2119 len 16384
26299 011/300 Write block 1542 len 24576 data: 6 6 8 8 7 6
26300 012/300 Read block 3286 len 32768
26301 013/300 Flush
26302 014/300 Write block 291 len 20480 data: 7 7 6 5 5
26303 015/300 Write block 4329 len 28672 data: 6 5 6 7 7 6 5
26304 016/300 Write block 833 len 32768 data: 7 7 6 7 6 5 5 5
26305 017/300 Read block 4694 len 40960
26306 018/300 Read block 1899 len 40960
26307 019/300 Read block 2482 len 28672
26308 020/300 Write block 2952 len 16384 data: 5 6 6 5
26309 021/300 Flush
26310 022/300 Flush
26311 023/300 Write block 1660 len 20480 data: 7 5 5 4 3
26312 024/300 Read block 3146 len 4096
26313 025/300 Read block 4702 len 40960
26314 026/300 Read block 2065 len 8192
26315 027/300 Write block 2735 len 28672 data: 7 7 8 9 10 7 8
26316 028/300 Write block 3701 len 36864 data: 5 5 4 4 4 4 3 4 4
26317 029/300 Write block 111 len 24576 data: 3 3 2 2 2 2
26318 030/300 Read block 4732 len 20480
26319 031/300 Read block 2024 len 16384
26320 032/300 Read block 2169 len 28672
26321 033/300 Read block 4950 len 24576
26322 034/300 Write block 2943 len 28672 data: 3 2 3 3 4 5 5
26323 035/300 Flush
26324 036/300 Read block 90 len 12288
26325 037/300 Read block 19 len 12288
26326 038/300 Read block 4462 len 36864
26327 039/300 Write block 131 len 16384 data: 2 2 2 2
26328 040/300 Read block 1956 len 4096
26329 041/300 Read block 799 len 24576
26330 042/300 Read block 2858 len 12288
26331 043/300 Write block 2950 len 20480 data: 5 4 6 7 7
26332 044/300 Write block 952 len 4096 data: 5
26333 045/300 Write block 4474 len 24576 data: 5 6 6 6 6 5
26334 046/300 Write block 820 len 16384 data: 8 8 7 7
26335 047/300 Read block 3873 len 24576
26336 048/300 Read block 4885 len 40960
26337 049/300 Write block 3730 len 12288 data: 9 8 7
26338 050/300 Read block 1229 len 24576
26339 051/300 Read block 2537 len 32768
26340 052/300 Read block 2700 len 20480
26341 053/300 Read block 3213 len 16384
26342 054/300 Write block 2989 len 36864 data: 6 6 4 3 2 3 3 3 3
26343 055/300 Write block 3587 len 36864 data: 5 5 5 5 4 4 3 4 5
26344 056/300 Read block 2157 len 40960
26345 057/300 Read block 2630 len 24576
26346 058/300 Read block 2771 len 16384
26347 059/300 Read block 3367 len 28672
26348 060/300 Read block 432 len 24576
26349 061/300 Write block 4177 len 12288 data: 3 4 6
26350 062/300 Write block 2541 len 12288 data: 5 5 8
26351 063/300 Write block 2992 len 32768 data: 4 3 4 4 4 4 3 3
26352 064/300 Write block 4790 len 20480 data: 6 6 8 8 9
26353 065/300 Read block 95 len 36864
26354 066/300 Read block 1809 len 28672
26355 067/300 Write block 3285 len 8192 data: 7 8
26356 068/300 Read block 1694 len 24576
26357 069/300 Read block 829 len 40960
26358 070/300 Read block 4928 len 12288
26359 071/300 Write block 4220 len 28672 data: 3 4 3 4 3 4 4
26360 072/300 Flush
26361 073/300 Read block 4934 len 12288
26362 074/300 Flush
26363 075/300 Write block 232 len 24576 data: 8 8 8 8 9 8
26364 076/300 Write block 1476 len 16384 data: 4 4 5 5
26365 077/300 Read block 4662 len 8192
26366 078/300 Read block 264 len 40960
26367 079/300 Read block 767 len 4096
26368 080/300 Read block 330 len 4096
26369 081/300 Read block 328 len 36864
26370 082/300 Flush
26371 083/300 Read block 1618 len 40960
26372 084/300 Flush
26373 085/300 Read block 4395 len 8192
26374 086/300 Read block 2244 len 24576
26375 087/300 Read block 1777 len 36864
26376 088/300 Read block 516 len 40960
26377 089/300 Write block 1169 len 40960 data: 2 3 3 4 4 3 2 4 5 6
26378 090/300 Write block 3564 len 8192 data: 7 7
26379 091/300 Write block 736 len 24576 data: 8 9 9 11 8 7
26380 092/300 Read block 2001 len 28672
26381 093/300 Flush
26382 094/300 Write block 2658 len 28672 data: 5 6 7 9 8 9 9
26383 095/300 Read block 1367 len 4096
26384 096/300 Write block 1445 len 16384 data: 5 7 6 6
26385 097/300 Read block 1590 len 32768
26386 098/300 Write block 2411 len 32768 data: 6 5 5 5 5 5 5 6
26387 099/300 Write block 2941 len 8192 data: 4 3
26388 100/300 Read block 4164 len 16384
26389 101/300 Read block 4188 len 12288
26390 102/300 Write block 4643 len 20480 data: 6 6 6 6 3
26391 103/300 Read block 4071 len 40960
26392 104/300 Write block 4435 len 4096 data: 7
26393 105/300 Write block 2499 len 4096 data: 5
26394 106/300 Read block 3328 len 40960
26395 107/300 Read block 3242 len 4096
26396 108/300 Write block 980 len 24576 data: 7 6 6 6 4 6
26397 109/300 Read block 4447 len 8192
26398 110/300 Write block 123 len 20480 data: 2 3 4 4 4
26399 111/300 Write block 4432 len 28672 data: 7 8 8 8 7 6 5
26400 112/300 Read block 3873 len 8192
26401 113/300 Flush
26402 114/300 Read block 4746 len 40960
26403 115/300 Read block 3909 len 24576
26404 116/300 Flush
26405 117/300 Read block 3176 len 16384
26406 118/300 Write block 682 len 8192 data: 6 6
26407 119/300 Write block 1685 len 28672 data: 4 5 5 5 6 6 6
26408 120/300 Write block 4050 len 12288 data: 9 10 9
26409 121/300 Write block 3606 len 24576 data: 5 7 7 8 9 6
26410 122/300 Read block 2460 len 28672
26411 123/300 Read block 4883 len 20480
26412 124/300 Read block 1836 len 12288
26413 125/300 Write block 182 len 20480 data: 5 5 4 5 4
26414 126/300 Write block 2383 len 40960 data: 5 4 2 3 3 5 5 4 5 5
26415 127/300 Write block 4981 len 4096 data: 8
26416 128/300 Read block 1571 len 28672
26417 129/300 Read block 369 len 28672
26418 130/300 Read block 2926 len 20480
26419 131/300 Read block 413 len 20480
26420 132/300 Read block 4191 len 20480
26421 133/300 Write block 3953 len 36864 data: 6 8 7 8 8 5 5 4 5
26422 134/300 Read block 4794 len 8192
26423 135/300 Write block 1984 len 32768 data: 4 3 4 4 3 3 2 3
26424 136/300 Read block 1931 len 36864
26425 137/300 Write block 2353 len 8192 data: 5 7
26426 138/300 Read block 4663 len 8192
26427 139/300 Write block 4604 len 36864 data: 5 5 5 4 5 5 5 3 3
26428 140/300 Read block 1170 len 4096
26429 141/300 Read block 3905 len 24576
26430 142/300 Flush
26431 143/300 Write block 516 len 20480 data: 5 4 3 4 5
26432 144/300 Flush
26433 145/300 Write block 645 len 28672 data: 3 3 3 3 4 5 5
26434 146/300 Flush
26435 147/300 Flush
26436 148/300 Write block 4946 len 28672 data: 4 6 9 10 10 10 10
26437 149/300 Write block 1332 len 12288 data: 6 5 4
26438 150/300 Write block 339 len 28672 data: 5 3 4 5 6 6 6
26439 151/300 Write block 767 len 20480 data: 2 3 3 4 3
26440 152/300 Write block 221 len 4096 data: 6
26441 153/300 Read block 4352 len 20480
26442 154/300 Write block 1452 len 40960 data: 7 7 6 6 5 4 6 6 5 5
26443 155/300 Read block 1560 len 12288
26444 156/300 Write block 1005 len 12288 data: 7 7 7
26445 157/300 Write block 2432 len 16384 data: 6 7 6 7
26446 158/300 Write block 1368 len 24576 data: 6 6 4 3 4 4
26447 159/300 Read block 1760 len 8192
26448 160/300 Read block 2839 len 24576
26449 161/300 Read block 4185 len 16384
26450 162/300 Write block 574 len 40960 data: 5 5 6 6 6 8 8 8 8 7
26451 163/300 Write block 2900 len 32768 data: 3 4 3 5 5 5 6 7
26452 164/300 Flush
26453 165/300 Read block 4700 len 32768
26454 166/300 Write block 3679 len 4096 data: 6
26455 167/300 Write block 2393 len 12288 data: 5 6 5
26456 168/300 Read block 187 len 4096
26457 169/300 Write block 3627 len 28672 data: 7 5 4 4 6 6 4
26458 170/300 Write block 2681 len 8192 data: 3 3
26459 171/300 Read block 3799 len 20480
26460 172/300 Write block 152 len 20480 data: 5 6 7 7 7
26461 173/300 Read block 423 len 4096
26462 174/300 Write block 1719 len 12288 data: 3 3 4
26463 175/300 Write block 1232 len 20480 data: 6 5 7 7 7
26464 176/300 Read block 377 len 36864
26465 177/300 Write block 3504 len 8192 data: 7 8
26466 178/300 Flush
26467 179/300 Write block 429 len 40960 data: 3 3 4 4 5 4 4 3 6 6
26468 180/300 Write block 2 len 8192 data: 5 5
26469 181/300 Read block 3887 len 20480
26470 182/300 Read block 3031 len 8192
26471 183/300 Read block 3877 len 16384
26472 184/300 Write block 3762 len 36864 data: 6 7 7 7 9 9 9 7 7
26473 185/300 Read block 3130 len 16384
26474 186/300 Read block 1101 len 16384
26475 187/300 Flush
26476 188/300 Read block 772 len 40960
26477 189/300 Read block 3920 len 20480
26478 190/300 Read block 461 len 24576
26479 191/300 Read block 3087 len 8192
26480 192/300 Flush
26481 193/300 Read block 3512 len 28672
26482 194/300 Flush
26483 195/300 Read block 4188 len 40960
26484 196/300 Write block 3322 len 28672 data: 5 5 5 6 7 8 8
26485 197/300 Read block 149 len 12288
26486 198/300 Read block 2413 len 32768
26487 199/300 Read block 2791 len 20480
26488 200/300 Write block 3879 len 28672 data: 6 8 8 8 8 7 6
26489 201/300 Write block 3830 len 40960 data: 7 7 7 10 8 7 7 6 4 4
26490 202/300 Write block 3349 len 16384 data: 9 9 9 8
26491 203/300 Write block 4248 len 24576 data: 6 5 6 7 6 4
26492 204/300 Write block 271 len 24576 data: 3 3 4 4 4 5
26493 205/300 Read block 4 len 40960
26494 206/300 Flush
26495 207/300 Write block 4243 len 32768 data: 5 5 3 2 4 7 6 7
26496 208/300 Read block 2136 len 32768
26497 209/300 Read block 1746 len 4096
26498 210/300 Write block 4323 len 36864 data: 5 7 7 6 5 6 7 6 7
26499 211/300 Write block 4332 len 20480 data: 8 8 7 6 5
26500 212/300 Write block 351 len 16384 data: 12 13 15 15
26501 213/300 Write block 917 len 20480 data: 6 8 6 4 5
26502 214/300 Read block 3626 len 32768
26503 215/300 Read block 201 len 4096
26504 216/300 Read block 3380 len 4096
26505 217/300 Write block 4030 len 24576 data: 6 5 5 5 4 4
26506 218/300 Write block 2910 len 16384 data: 9 7 6 10
26507 219/300 Write block 2993 len 28672 data: 4 5 5 5 5 4 4
26508 220/300 Read block 4799 len 28672
26509 221/300 Flush
26510 222/300 Write block 1219 len 12288 data: 9 8 9
26511 223/300 Write block 342 len 32768 data: 6 7 7 7 6 6 8 7
26512 224/300 Read block 1987 len 20480
26513 225/300 Read block 161 len 36864
26514 226/300 Read block 2713 len 8192
26515 227/300 Read block 2413 len 28672
26516 228/300 Write block 3508 len 40960 data: 7 7 6 6 8 5 3 4 4 5
26517 229/300 Read block 1965 len 24576
26518 230/300 Read block 3969 len 32768
26519 231/300 Read block 3454 len 36864
26520 232/300 Write block 1797 len 24576 data: 6 5 7 6 3 2
26521 233/300 Write block 4866 len 4096 data: 9
26522 234/300 Read block 3314 len 4096
26523 235/300 Read block 1688 len 28672
26524 236/300 Read block 2591 len 8192
26525 237/300 Read block 451 len 40960
26526 238/300 Read block 4230 len 4096
26527 239/300 Read block 3580 len 8192
26528 240/300 Read block 1683 len 36864
26529 241/300 Write block 2880 len 36864 data: 8 9 9 9 7 5 6 6 4
26530 242/300 Read block 2643 len 12288
26531 243/300 Write block 245 len 12288 data: 3 3 3
26532 244/300 Read block 1352 len 40960
26533 245/300 Write block 2880 len 8192 data: 9 10
26534 246/300 Write block 893 len 36864 data: 4 3 3 2 2 3 4 5 5
26535 247/300 Write block 3124 len 8192 data: 3 4
26536 248/300 Flush
26537 249/300 Read block 3633 len 40960
26538 250/300 Write block 1060 len 4096 data: 6
26539 251/300 Read block 4676 len 36864
26540 252/300 Flush
26541 253/300 Read block 4838 len 40960
26542 254/300 Write block 3172 len 36864 data: 3 3 4 5 5 7 8 8 6
26543 255/300 Read block 3508 len 32768
26544 256/300 Read block 4593 len 8192
26545 257/300 Read block 1989 len 4096
26546 258/300 Write block 1307 len 20480 data: 5 5 7 7 6
26547 259/300 Read block 612 len 12288
26548 260/300 Write block 603 len 4096 data: 6
26549 261/300 Flush
26550 262/300 Write block 1874 len 40960 data: 6 6 7 7 7 4 3 3 5 6
26551 263/300 Write block 4421 len 8192 data: 6 6
26552 264/300 Read block 2984 len 20480
26553 265/300 Read block 2428 len 12288
26554 266/300 Write block 1144 len 36864 data: 6 5 5 5 4 5 6 6 5
26555 267/300 Write block 1282 len 4096 data: 6
26556 268/300 Write block 1557 len 28672 data: 6 7 9 8 9 10 6
26557 269/300 Read block 4659 len 20480
26558 270/300 Write block 4258 len 20480 data: 7 9 9 10 10
26559 271/300 Read block 4103 len 32768
26560 272/300 Write block 2717 len 8192 data: 7 7
26561 273/300 Flush
26562 274/300 Write block 3150 len 12288 data: 10 10 9
26563 275/300 Write block 3994 len 24576 data: 5 6 6 5 4 4
26564 276/300 Write block 168 len 16384 data: 7 6 6 6
26565 277/300 Read block 4353 len 8192
26566 278/300 Write block 2637 len 8192 data: 7 7
26567 279/300 Write block 2988 len 16384 data: 7 7 7 5
26568 280/300 Write block 12 len 12288 data: 3 3 3
26569 281/300 Write block 3354 len 36864 data: 6 5 5 6 7 6 6 6 7
26570 282/300 Write block 4382 len 20480 data: 4 6 5 5 6
26571 283/300 Flush
26572 284/300 Read block 4824 len 32768
26573 285/300 Write block 387 len 12288 data: 5 5 6
26574 286/300 Read block 4723 len 24576
26575 287/300 Write block 3273 len 20480 data: 6 5 7 5 4
26576 288/300 Write block 3613 len 24576 data: 10 12 10 10 10 11
26577 289/300 Read block 637 len 40960
26578 290/300 Read block 1676 len 20480
26579 291/300 Read block 76 len 16384
26580 292/300 Write block 3472 len 4096 data: 5
26581 293/300 Read block 3004 len 20480
26582 294/300 Write block 4081 len 12288 data: 5 5 5
26583 295/300 Write block 3326 len 16384 data: 8 9 9 6
26584 296/300 Write block 2697 len 32768 data: 2 2 2 3 3 3 3 3
26585 297/300 Flush
26586 298/300 Read block 1094 len 8192
26587 299/300 Read block 1955 len 16384
26588 300/300 Read block 150 len 16384
26589 Main task has work to do, go find it
26590 got dsc Start(1)
26591 start 1
26592 Feb 15 03:02:48.545 INFO request completed, latency_us: 238, response_code: 204, uri: /start/cid/1, method: POST, req_id: f54456b1-bc56-4e13-9305-0d1a37936623, remote_addr: 127.0.0.1:54744, local_addr: 127.0.0.1:9998
26593 Replay: started 1, returned:Ok(())
26594 ----------------------------------------------------------------
26595 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
26596 States: Active Offline Active
26597 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
26598 8440 Acked 9439 Read 6 Done New Done true
26599 8441 Acked 9440 Read 1 Done New Done true
26600 8442 Acked 9441 Write 6 Done New Done false
26601 8443 Acked 9442 Flush 0 Done New Done false
26602 8444 Acked 9443 Write 6 Done New Done false
26603 8445 Acked 9444 Read 10 Done New Done false
26604 8446 Acked 9445 Read 7 Done New Done false
26605 8447 Acked 9446 Read 2 Done New Done false
26606 8448 Acked 9447 Write 4 Done New Done false
26607 8449 Acked 9448 Flush 0 Done New Done false
26608 8450 Acked 9449 Read 4 Done New Done false
26609 8451 Acked 9450 Write 6 Done New Done false
26610 8452 Acked 9451 Read 8 Done New Done false
26611 8453 Acked 9452 Flush 0 Done New Done false
26612 8454 Acked 9453 Write 5 Done New Done false
26613 8455 Acked 9454 Write 7 Done New Done false
26614 8456 Acked 9455 Write 8 Done New Done false
26615 8457 Acked 9456 Read 10 Done New Done false
26616 8458 Acked 9457 Read 10 Done New Done false
26617 8459 Acked 9458 Read 7 Done New Done false
26618 8460 Acked 9459 Write 4 Done New Done false
26619 8461 Acked 9460 Flush 0 Done New Done false
26620 8462 Acked 9461 Flush 0 Done New Done false
26621 8463 Acked 9462 Write 5 Done New Done false
26622 8464 Acked 9463 Read 1 Done New Done false
26623 8465 Acked 9464 Read 10 Done New Done false
26624 8466 Acked 9465 Read 2 Done New Done false
26625 8467 Acked 9466 Write 7 Done New Done false
26626 8468 Acked 9467 Write 9 Done New Done false
26627 8469 Acked 9468 Write 6 Done New Done false
26628 8470 Acked 9469 Read 5 Done New Done false
26629 8471 Acked 9470 Read 4 Done New Done false
26630 8472 Acked 9471 Read 7 Done New Done false
26631 8473 Acked 9472 Read 6 Done New Done false
26632 8474 Acked 9473 Write 7 Done New Done false
26633 8475 Acked 9474 Flush 0 Done New Done false
26634 8476 Acked 9475 Read 3 Done New Done false
26635 8477 Acked 9476 Read 3 Done New Done false
26636 8478 Acked 9477 Read 9 Done New Done false
26637 8479 Acked 9478 Write 4 Done New Done false
26638 8480 Acked 9479 Read 1 Done New Done false
26639 8481 Acked 9480 Read 6 Done New Done false
26640 8482 Acked 9481 Read 3 Done New Done false
26641 8483 Acked 9482 Write 5 Done New Done false
26642 8484 Acked 9483 Write 1 Done New Done false
26643 8485 Acked 9484 Write 6 Done New Done false
26644 8486 Acked 9485 Write 4 Done New Done false
26645 8487 Acked 9486 Read 6 Done New Done false
26646 8488 Acked 9487 Read 10 Done New Done false
26647 8489 Acked 9488 Write 3 Done New Done false
26648 8490 Acked 9489 Read 6 Done New Done false
26649 8491 Acked 9490 Read 8 Done New Done false
26650 8492 Acked 9491 Read 5 Done New Done false
26651 8493 Acked 9492 Read 4 Done New Done false
26652 8494 Acked 9493 Write 9 Done New Done false
26653 8495 Acked 9494 Write 9 Done New Done false
26654 8496 Acked 9495 Read 10 Done New Done false
26655 8497 Acked 9496 Read 6 Done New Done false
26656 8498 Acked 9497 Read 4 Done New Done false
26657 8499 Acked 9498 Read 7 Done New Done false
26658 8500 Acked 9499 Read 6 Done New Done false
26659 8501 Acked 9500 Write 3 Done New Done false
26660 8502 Acked 9501 Write 3 Done New Done false
26661 8503 Acked 9502 Write 8 Done New Done false
26662 8504 Acked 9503 Write 5 Done New Done false
26663 8505 Acked 9504 Read 9 Done New Done false
26664 8506 Acked 9505 Read 7 Done New Done false
26665 8507 Acked 9506 Write 2 Done New Done false
26666 8508 Acked 9507 Read 6 Done New Done false
26667 8509 Acked 9508 Read 10 Done New Done false
26668 8510 Acked 9509 Read 3 Done New Done false
26669 8511 Acked 9510 Write 7 Done New Done false
26670 8512 Acked 9511 Flush 0 Done New Done false
26671 8513 Acked 9512 Read 3 Done New Done false
26672 8514 Acked 9513 Flush 0 Done New Done false
26673 8515 Acked 9514 Write 6 Done New Done false
26674 8516 Acked 9515 Write 4 Done New Done false
26675 8517 Acked 9516 Read 2 Done New Done false
26676 8518 Acked 9517 Read 10 Done New Done false
26677 8519 Acked 9518 Read 1 Done New Done false
26678 8520 Acked 9519 Read 1 Done New Done false
26679 8521 Acked 9520 Read 9 Done New Done false
26680 8522 Acked 9521 Flush 0 Done New Done false
26681 8523 Acked 9522 Read 10 Done New Done false
26682 8524 Acked 9523 Flush 0 Done New Done false
26683 8525 Acked 9524 Read 2 Done New Done false
26684 8526 Acked 9525 Read 6 Done New Done false
26685 8527 Acked 9526 Read 9 Done New Done false
26686 8528 Acked 9527 Read 10 Done New Done false
26687 8529 Acked 9528 Write 10 Done New Done false
26688 8530 Acked 9529 Write 2 Done New Done false
26689 8531 Acked 9530 Write 6 Done New Done false
26690 8532 Acked 9531 Read 7 Done New Done false
26691 8533 Acked 9532 Flush 0 Done New Done false
26692 8534 Acked 9533 Write 7 Done New Done false
26693 8535 Acked 9534 Read 1 Done New Done false
26694 8536 Acked 9535 Write 4 Done New Done false
26695 8537 Acked 9536 Read 8 Done New Done false
26696 8538 Acked 9537 Write 8 Done New Done false
26697 8539 Acked 9538 Write 2 Done New Done false
26698 8540 Acked 9539 Read 4 Done New Done false
26699 8541 Acked 9540 Read 3 Done New Done false
26700 8542 Acked 9541 Write 5 Done New Done false
26701 8543 Acked 9542 Read 10 Done New Done false
26702 8544 Acked 9543 Write 1 Done New Done false
26703 8545 Acked 9544 Write 1 Done New Done false
26704 8546 Acked 9545 Read 10 Done New Done false
26705 8547 Acked 9546 Read 1 Done New Done false
26706 8548 Acked 9547 Write 6 Done New Done false
26707 8549 Acked 9548 Read 2 Done New Done false
26708 8550 Acked 9549 Write 5 Done New Done false
26709 8551 Acked 9550 Write 7 Done New Done false
26710 8552 Acked 9551 Read 2 Done New Done false
26711 8553 Acked 9552 Flush 0 Done New Done false
26712 8554 Acked 9553 Read 10 Done New Done false
26713 8555 Acked 9554 Read 6 Done New Done false
26714 8556 Acked 9555 Flush 0 Done New Done false
26715 8557 Acked 9556 Read 4 Done New Done false
26716 8558 Acked 9557 Write 2 Done New Done false
26717 8559 Acked 9558 Write 7 Done New Done false
26718 8560 Acked 9559 Write 3 Done New Done false
26719 8561 Acked 9560 Write 6 Done New Done false
26720 8562 Acked 9561 Read 7 Done New Done false
26721 8563 Acked 9562 Read 5 Done New Done false
26722 8564 Acked 9563 Read 3 Done New Done false
26723 8565 Acked 9564 Write 5 Done New Done false
26724 8566 Acked 9565 Write 10 Done New Done false
26725 8567 Acked 9566 Write 1 Done New Done false
26726 8568 Acked 9567 Read 7 Done New Done false
26727 8569 Acked 9568 Read 7 Done New Done false
26728 8570 Acked 9569 Read 5 Done New Done false
26729 8571 Acked 9570 Read 5 Done New Done false
26730 8572 Acked 9571 Read 5 Done New Done false
26731 8573 Acked 9572 Write 9 Done New Done false
26732 8574 Acked 9573 Read 2 Done New Done false
26733 8575 Acked 9574 Write 8 Done New Done false
26734 8576 Acked 9575 Read 9 Done New Done false
26735 8577 Acked 9576 Write 2 Done New Done false
26736 8578 Acked 9577 Read 2 Done New Done false
26737 8579 Acked 9578 Write 9 Done New Done false
26738 8580 Acked 9579 Read 1 Done New Done false
26739 8581 Acked 9580 Read 6 Done New Done false
26740 8582 Acked 9581 Flush 0 Done New Done false
26741 8583 Acked 9582 Write 5 Done New Done false
26742 8584 Acked 9583 Flush 0 Done New Done false
26743 8585 Acked 9584 Write 7 Done New Done false
26744 8586 Acked 9585 Flush 0 Done New Done false
26745 8587 Acked 9586 Flush 0 Done New Done false
26746 8588 Acked 9587 Write 7 Done New Done false
26747 8589 Acked 9588 Write 3 Done New Done false
26748 8590 Acked 9589 Write 7 Done New Done false
26749 8591 Acked 9590 Write 5 Done New Done false
26750 8592 Acked 9591 Write 1 Done New Done false
26751 8593 Acked 9592 Read 5 Done New Done false
26752 8594 Acked 9593 Write 10 Done New Done false
26753 8595 Acked 9594 Read 3 Done New Done false
26754 8596 Acked 9595 Write 3 Done New Done false
26755 8597 Acked 9596 Write 4 Done New Done false
26756 8598 Acked 9597 Write 6 Done New Done false
26757 8599 Acked 9598 Read 2 Done New Done false
26758 8600 Acked 9599 Read 6 Done New Done false
26759 8601 Acked 9600 Read 4 Done New Done false
26760 8602 Acked 9601 Write 10 Done New Done false
26761 8603 Acked 9602 Write 8 Done New Done false
26762 8604 Acked 9603 Flush 0 Done New Done false
26763 8605 Acked 9604 Read 8 Done New Done false
26764 8606 Acked 9605 Write 1 Done New Done false
26765 8607 Acked 9606 Write 3 Done New Done false
26766 8608 Acked 9607 Read 1 Done New Done false
26767 8609 Acked 9608 Write 7 Done New Done false
26768 8610 Acked 9609 Write 2 Done New Done false
26769 8611 Acked 9610 Read 5 Done New Done false
26770 8612 Acked 9611 Write 5 Done New Done false
26771 8613 Acked 9612 Read 1 Done New Done false
26772 8614 Acked 9613 Write 3 Done New Done false
26773 8615 Acked 9614 Write 5 Done New Done false
26774 8616 Acked 9615 Read 9 Done New Done false
26775 8617 Acked 9616 Write 2 Done New Done false
26776 8618 Acked 9617 Flush 0 Done New Done false
26777 8619 Acked 9618 Write 10 Done New Done false
26778 8620 Acked 9619 Write 2 Done New Done false
26779 8621 Acked 9620 Read 5 Done New Done false
26780 8622 Acked 9621 Read 2 Done New Done false
26781 8623 Acked 9622 Read 4 Done New Done false
26782 8624 Acked 9623 Write 9 Done New Done false
26783 8625 Acked 9624 Read 4 Done New Done false
26784 8626 Acked 9625 Read 4 Done New Done false
26785 8627 Acked 9626 Flush 0 Done New Done false
26786 8628 Acked 9627 Read 10 Done New Done false
26787 8629 Acked 9628 Read 5 Done New Done false
26788 8630 Acked 9629 Read 6 Done New Done false
26789 8631 Acked 9630 Read 2 Done New Done false
26790 8632 Acked 9631 Flush 0 Done New Done false
26791 8633 Acked 9632 Read 7 Done New Done false
26792 8634 Acked 9633 Flush 0 Done New Done false
26793 8635 Acked 9634 Read 10 Done New Done false
26794 8636 Acked 9635 Write 7 Done New Done false
26795 8637 Acked 9636 Read 3 Done New Done false
26796 8638 Acked 9637 Read 8 Done New Done false
26797 8639 Acked 9638 Read 5 Done New Done false
26798 8640 Acked 9639 Write 7 Done New Done false
26799 8641 Acked 9640 Write 10 Done New Done false
26800 8642 Acked 9641 Write 4 Done New Done false
26801 8643 Acked 9642 Write 6 Done New Done false
26802 8644 Acked 9643 Write 6 Done New Done false
26803 8645 Acked 9644 Read 10 Done New Done false
26804 8646 Acked 9645 Flush 0 Done New Done false
26805 8647 Acked 9646 Write 8 Done New Done false
26806 8648 Acked 9647 Read 8 Done New Done false
26807 8649 Acked 9648 Read 1 Done New Done false
26808 8650 Acked 9649 Write 9 Done New Done false
26809 8651 Acked 9650 Write 5 Done New Done false
26810 8652 Acked 9651 Write 4 Done New Done false
26811 8653 Acked 9652 Write 5 Done New Done false
26812 8654 Acked 9653 Read 8 Done New Done false
26813 8655 Acked 9654 Read 1 Done New Done false
26814 8656 Acked 9655 Read 1 Done New Done false
26815 8657 Acked 9656 Write 6 Done New Done false
26816 8658 Acked 9657 Write 4 Done New Done false
26817 8659 Acked 9658 Write 7 Done New Done false
26818 8660 Acked 9659 Read 7 Done New Done false
26819 8661 Acked 9660 Flush 0 Done New Done false
26820 8662 Acked 9661 Write 3 Done New Done false
26821 8663 Acked 9662 Write 8 Done New Done false
26822 8664 Acked 9663 Read 5 Done New Done false
26823 8665 Acked 9664 Read 9 Done New Done false
26824 8666 Acked 9665 Read 2 Done New Done false
26825 8667 Acked 9666 Read 7 Done New Done false
26826 8668 Acked 9667 Write 10 Done New Done false
26827 8669 Acked 9668 Read 6 Done New Done false
26828 8670 Acked 9669 Read 8 Done New Done false
26829 8671 Acked 9670 Read 9 Done New Done false
26830 8672 Acked 9671 Write 6 Done New Done false
26831 8673 Acked 9672 Write 1 Done New Done false
26832 8674 Acked 9673 Read 1 Done New Done false
26833 8675 Acked 9674 Read 7 Done New Done false
26834 8676 Acked 9675 Read 2 Done New Done false
26835 8677 Acked 9676 Read 10 Done New Done false
26836 8678 Acked 9677 Read 1 Done New Done false
26837 8679 Acked 9678 Read 2 Done New Done false
26838 8680 Acked 9679 Read 9 Done New Done false
26839 8681 Acked 9680 Write 9 Done New Done false
26840 8682 Acked 9681 Read 3 Done New Done false
26841 8683 Acked 9682 Write 3 Done New Done false
26842 8684 Acked 9683 Read 10 Done New Done false
26843 8685 Acked 9684 Write 2 Done New Done false
26844 8686 Acked 9685 Write 9 Done New Done false
26845 8687 Acked 9686 Write 2 Done New Done false
26846 8688 Acked 9687 Flush 0 Done New Done false
26847 8689 Acked 9688 Read 10 Done New Done false
26848 8690 Acked 9689 Write 1 Done New Done false
26849 8691 Acked 9690 Read 9 Done New Done false
26850 8692 Acked 9691 Flush 0 Done New Done false
26851 8693 Acked 9692 Read 10 Done New Done false
26852 8694 Acked 9693 Write 9 Done New Done false
26853 8695 Acked 9694 Read 8 Done New Done false
26854 8696 Acked 9695 Read 2 Done New Done false
26855 8697 Acked 9696 Read 1 Done New Done false
26856 8698 Acked 9697 Write 5 Done New Done false
26857 8699 Acked 9698 Read 3 Done New Done false
26858 8700 Acked 9699 Write 1 Done New Done false
26859 8701 Acked 9700 Flush 0 Done New Done false
26860 8702 Acked 9701 Write 10 Done New Done false
26861 8703 Acked 9702 Write 2 Done New Done false
26862 8704 Acked 9703 Read 5 Done New Done false
26863 8705 Acked 9704 Read 3 Done New Done false
26864 8706 Acked 9705 Write 9 Done New Done false
26865 8707 Acked 9706 Write 1 Done New Done false
26866 8708 Acked 9707 Write 7 Done New Done false
26867 8709 Acked 9708 Read 5 Done New Done false
26868 8710 Acked 9709 Write 5 Done New Done false
26869 8711 Acked 9710 Read 8 Done New Done false
26870 8712 Acked 9711 Write 2 Done New Done false
26871 8713 Acked 9712 Flush 0 Done New Done false
26872 8714 Acked 9713 Write 3 Done New Done false
26873 8715 Acked 9714 Write 6 Done New Done false
26874 8716 Acked 9715 Write 4 Done New Done false
26875 8717 Acked 9716 Read 2 Done New Done false
26876 8718 Acked 9717 Write 2 Done New Done false
26877 8719 Acked 9718 Write 4 Done New Done false
26878 8720 Acked 9719 Write 3 Done New Done false
26879 8721 Acked 9720 Write 9 Done New Done false
26880 8722 Acked 9721 Write 5 Done New Done false
26881 8723 Acked 9722 Flush 0 Done New Done false
26882 8724 Acked 9723 Read 8 Done New Done false
26883 8725 Acked 9724 Write 3 Done New Done false
26884 8726 Acked 9725 Read 6 Done New Done false
26885 8727 Acked 9726 Write 5 Done New Done false
26886 8728 Acked 9727 Write 6 Done New Done false
26887 8729 Acked 9728 Read 10 Done New Done false
26888 8730 Acked 9729 Read 5 Done New Done false
26889 8731 Acked 9730 Read 4 Done New Done false
26890 8732 Acked 9731 Write 1 Done New Done false
26891 8733 Acked 9732 Read 5 Done New Done false
26892 8734 Acked 9733 Write 3 Done New Done false
26893 8735 Acked 9734 Write 4 Done New Done false
26894 8736 Acked 9735 Write 8 Done New Done false
26895 8737 Acked 9736 Flush 0 Done New Done false
26896 8738 Acked 9737 Read 2 Done New Done false
26897 8739 Acked 9738 Read 4 Done New Done false
26898 8740 Acked 9739 Read 4 Done New Done false
26899 STATES DS:0 DS:1 DS:2 TOTAL
26900 New 0 301 0 301
26901 Sent 0 0 0 0
26902 Done 301 0 301 602
26903 Skipped 0 0 0 0
26904 Error 0 0 0 0
26905 Last Flush: 9736 9438 9736
26906 Downstairs last five completed: 9438 9437 9436 9435 9434
26907 Upstairs last five completed: 8740 8739 8738 8737 8736
26908 CLIENT: Up:0 ds:301 act:2
269092024-02-15T03:02:49.214ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
269102024-02-15T03:02:49.214ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
269112024-02-15T03:02:49.214ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
269122024-02-15T03:02:49.214ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
269132024-02-15T03:02:49.214ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
269142024-02-15T03:02:49.214ZINFOcrucible: [1] client re-new 302 jobs since flush 9438 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
269152024-02-15T03:02:49.214ZINFOcrucible: client task is exiting = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
269162024-02-15T03:02:50.215ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
269172024-02-15T03:02:50.215ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
269182024-02-15T03:02:50.215ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
269192024-02-15T03:02:50.216ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
269202024-02-15T03:02:50.216ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
269212024-02-15T03:02:50.216ZINFOcrucible: [1] client re-new 302 jobs since flush 9438 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
269222024-02-15T03:02:50.216ZINFOcrucible: client task is exiting = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
26923 [8820] got start action
26924 [8820] I am going to restart
26925 Starting downstairs at port 8820
26926 Make output file at "/tmp/dsc/downstairs-8820.txt"
26927 [8820][1] reports Starting
26928 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1196)
26929 [8820][1] reports Running
269302024-02-15T03:02:51.217ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
269312024-02-15T03:02:51.217ZINFOcrucible: ds_connection connected = io task client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
269322024-02-15T03:02:51.218ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
269332024-02-15T03:02:51.218ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
269342024-02-15T03:02:51.218ZINFOcrucible: send last flush ID to this DS: 9438 = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
269352024-02-15T03:02:51.219ZINFOcrucible: Replied this last flush ID: 9438 = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
269362024-02-15T03:02:51.219ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
269372024-02-15T03:02:51.219ZINFOcrucible: [1] Transition from Offline to Replay = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
269382024-02-15T03:02:51.219ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
269392024-02-15T03:02:51.219ZINFOcrucible: [1] Transition from Replay to Active = downstairs client = 1 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
26940 ----------------------------------------------------------------
26941 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
26942 Downstairs last five completed: 9740 9739 9738 9737 9736
26943 Upstairs last five completed: 8741 8740 8739 8738 8737
26944 CLIENT: Up:0 ds:0 act:3
26945 Replay: All jobs finished, all DS active.
26946 Main task has work to do, go find it
26947 got dsc Stop(2)
26948 stop 2
26949 Feb 15 03:02:52.548 INFO [8830] Got stop action so:false kr:false
26950 request completed, latency_us: 358, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 9cdf32d1-687b-4e2d-b01e-4e0bd0be1fa1, remote_addr: 127.0.0.1:54744, local_addr: 127.0.0.1:9998
269512024-02-15T03:02:52.549ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
269522024-02-15T03:02:52.549ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
269532024-02-15T03:02:52.549ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
269542024-02-15T03:02:52.549ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
26955 Feb 15 03:02:52.549 INFO{"msg":"[2] client re-new 0 jobs since flush 9740","v":0,"name":"crucible","level":30,"time":"2024-02-15T03:02:52.549442371Z","hostname":"ip-10-150-1-67.us-west-2.compute.internal","pid":1167,"":"downstairs","":"downstairs","session_id":"4f5a63e5-aa00-4890-9f6b-96c2e069fdb2"}
26956 request completed, latency_us: 149, response_code: 200, uri: /state/cid/2, method: GET, req_id: 82b32efb-9f4d-47d9-926d-ba019047f221, {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-15T03:02:52.549470217Z","hostname":"ip-10-150-1-67.us-west-2.compute.internal","pid":1167,"":"io task","client":"2","":"downstairs","session_id":"4f5a63e5-aa00-4890-9f6b-96c2e069fdb2"}
26957 remote_addr: 127.0.0.1:54744, local_addr: 127.0.0.1:9998
26958 [8830] Exited with: ExitStatus(unix_wait_status(9))
26959 [8830][2] reports Exit
269602024-02-15T03:02:53.550ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
269612024-02-15T03:02:53.550ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
269622024-02-15T03:02:53.550ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
269632024-02-15T03:02:53.550ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
269642024-02-15T03:02:53.550ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
269652024-02-15T03:02:53.550ZINFOcrucible: [2] client re-new 0 jobs since flush 9740 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
269662024-02-15T03:02:53.550ZINFOcrucible: client task is exiting = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
269672024-02-15T03:02:54.551ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
269682024-02-15T03:02:54.551ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
269692024-02-15T03:02:54.551ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
269702024-02-15T03:02:54.551ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
269712024-02-15T03:02:54.551ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
269722024-02-15T03:02:54.551ZINFOcrucible: [2] client re-new 0 jobs since flush 9740 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
269732024-02-15T03:02:54.551ZINFOcrucible: client task is exiting = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
269742024-02-15T03:02:55.553ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
269752024-02-15T03:02:55.553ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
269762024-02-15T03:02:55.553ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
269772024-02-15T03:02:55.553ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
269782024-02-15T03:02:55.553ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
269792024-02-15T03:02:55.553ZINFOcrucible: [2] client re-new 0 jobs since flush 9740 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
269802024-02-15T03:02:55.553ZINFOcrucible: client task is exiting = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
26981 Feb 15 03:02:56.551 INFO request completed, latency_us: 218, response_code: 200, uri: /state/cid/2, method: GET, req_id: 04cde7af-6119-4975-901e-c57b81dc1571, remote_addr: 127.0.0.1:54744, local_addr: 127.0.0.1:9998
26982 001/300 Write block 3670 len 20480 data: 8 7 6 8 8
26983 002/300 Read block 1836 len 20480
26984 003/300 Read block 978 len 16384
269852024-02-15T03:02:56.554ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
269862024-02-15T03:02:56.554ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
269872024-02-15T03:02:56.555ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
269882024-02-15T03:02:56.555ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
269892024-02-15T03:02:56.555ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
269902024-02-15T03:02:56.555ZINFOcrucible: [2] client re-new 3 jobs since flush 9740 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
269912024-02-15T03:02:56.555ZINFOcrucible: client task is exiting = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
26992 004/300 Flush
26993 005/300 Write block 4898 len 16384 data: 5 5 5 4
26994 006/300 Read block 3806 len 40960
26995 007/300 Read block 649 len 4096
26996 008/300 Write block 1364 len 12288 data: 6 6 6
26997 009/300 Write block 1273 len 32768 data: 3 3 4 5 5 6 6 7
26998 010/300 Write block 1388 len 16384 data: 4 4 4 5
26999 011/300 Read block 516 len 28672
27000 012/300 Flush
27001 013/300 Write block 2384 len 4096 data: 5
27002 014/300 Read block 3140 len 8192
27003 015/300 Write block 3070 len 28672 data: 4 5 5 4 3 5 5
27004 016/300 Read block 3036 len 32768
27005 017/300 Write block 1738 len 36864 data: 9 7 7 7 5 5 6 6 6
27006 018/300 Read block 656 len 20480
27007 019/300 Read block 4592 len 32768
27008 020/300 Write block 3965 len 4096 data: 4
27009 021/300 Read block 3379 len 12288
27010 022/300 Flush
27011 023/300 Write block 4965 len 28672 data: 8 7 6 7 7 6 7
27012 024/300 Read block 3158 len 40960
27013 025/300 Write block 2556 len 40960 data: 3 2 3 4 5 5 5 6 6 5
27014 026/300 Write block 4674 len 8192 data: 8 11
27015 027/300 Read block 4930 len 24576
27016 028/300 Read block 1576 len 24576
27017 029/300 Read block 2 len 36864
27018 030/300 Write block 3515 len 4096 data: 5
27019 031/300 Write block 3348 len 20480 data: 9 10 10 10 9
27020 032/300 Read block 948 len 24576
27021 033/300 Write block 4313 len 4096 data: 8
27022 034/300 Read block 3021 len 4096
27023 035/300 Write block 3289 len 40960 data: 10 10 9 7 4 4 3 5 4 6
27024 036/300 Flush
27025 037/300 Write block 476 len 36864 data: 6 6 6 4 3 4 3 3 5
27026 038/300 Read block 3821 len 4096
27027 039/300 Read block 727 len 20480
27028 040/300 Read block 3189 len 8192
27029 041/300 Read block 3037 len 12288
27030 042/300 Read block 3489 len 32768
27031 043/300 Write block 4859 len 12288 data: 4 6 7
27032 044/300 Read block 3993 len 36864
27033 045/300 Write block 4091 len 16384 data: 7 8 7 7
27034 046/300 Read block 1006 len 40960
27035 047/300 Read block 3578 len 36864
27036 048/300 Flush
27037 049/300 Write block 1170 len 16384 data: 4 4 5 5
27038 050/300 Flush
27039 051/300 Read block 2010 len 24576
27040 052/300 Read block 1094 len 8192
27041 053/300 Write block 1341 len 20480 data: 4 4 3 3 3
27042 054/300 Write block 4621 len 16384 data: 7 7 6 7
27043 055/300 Flush
27044 056/300 Read block 4336 len 24576
27045 057/300 Write block 1592 len 40960 data: 8 8 7 7 4 3 3 4 4 4
27046 058/300 Write block 1619 len 40960 data: 4 5 6 6 8 8 9 9 5 5
27047 059/300 Write block 355 len 28672 data: 12 10 10 8 7 7 9
27048 060/300 Read block 3310 len 12288
27049 061/300 Write block 3052 len 4096 data: 9
27050 062/300 Write block 1657 len 16384 data: 5 5 6 8
27051 063/300 Read block 2672 len 32768
27052 064/300 Read block 1581 len 8192
27053 065/300 Write block 1820 len 40960 data: 6 6 7 9 8 9 9 10 10 9
27054 066/300 Write block 3112 len 16384 data: 10 9 9 8
27055 067/300 Write block 737 len 4096 data: 10
27056 068/300 Write block 1991 len 8192 data: 4 6
27057 069/300 Write block 2441 len 32768 data: 6 4 4 5 5 7 7 9
27058 070/300 Flush
27059 071/300 Read block 340 len 16384
27060 072/300 Write block 2317 len 28672 data: 6 5 5 7 6 4 3
27061 073/300 Flush
27062 074/300 Read block 4271 len 20480
27063 075/300 Read block 785 len 28672
27064 076/300 Write block 1105 len 40960 data: 5 5 5 5 4 4 4 3 5 6
27065 077/300 Read block 2482 len 16384
27066 078/300 Write block 3223 len 8192 data: 2 2
27067 079/300 Write block 1557 len 32768 data: 7 8 10 9 10 11 7 5
27068 080/300 Write block 1253 len 36864 data: 7 7 9 7 8 7 8 9 7
27069 081/300 Write block 4263 len 16384 data: 8 6 7 7
27070 082/300 Write block 566 len 8192 data: 9 8
27071 083/300 Write block 4172 len 28672 data: 5 5 5 4 3 4 5
27072 084/300 Write block 2988 len 20480 data: 8 8 8 6 5
27073 085/300 Write block 4101 len 4096 data: 6
27074 086/300 Write block 770 len 40960 data: 5 4 4 4 6 7 7 7 7 7
27075 087/300 Write block 3536 len 36864 data: 8 8 9 9 9 9 8 10 9
27076 088/300 Write block 617 len 4096 data: 12
27077 089/300 Write block 3948 len 8192 data: 10 9
27078 090/300 Read block 3074 len 28672
27079 091/300 Read block 860 len 40960
27080 092/300 Write block 3341 len 40960 data: 7 6 6 5 5 7 9 10 11 11
27081 093/300 Read block 4765 len 36864
27082 094/300 Write block 2904 len 16384 data: 6 6 7 8
27083 095/300 Write block 691 len 16384 data: 6 5 5 5
27084 096/300 Read block 2505 len 40960
27085 097/300 Read block 3210 len 20480
27086 098/300 Read block 2712 len 16384
27087 099/300 Flush
27088 100/300 Write block 161 len 20480 data: 12 12 11 7 6
27089 101/300 Read block 86 len 36864
27090 102/300 Flush
27091 103/300 Write block 1419 len 4096 data: 5
27092 104/300 Read block 574 len 20480
27093 105/300 Write block 3264 len 36864 data: 5 3 4 5 5 5 5 5 6
27094 106/300 Write block 2390 len 12288 data: 5 6 6
27095 107/300 Write block 2889 len 20480 data: 5 4 4 3 3
27096 108/300 Read block 1548 len 16384
27097 109/300 Write block 1675 len 28672 data: 4 4 5 5 4 3 3
27098 110/300 Read block 2377 len 40960
27099 111/300 Flush
27100 112/300 Write block 3880 len 24576 data: 9 9 9 9 8 7
27101 113/300 Read block 3582 len 24576
27102 114/300 Read block 25 len 24576
27103 115/300 Flush
27104 116/300 Read block 840 len 8192
27105 117/300 Read block 1261 len 4096
27106 118/300 Read block 4535 len 8192
27107 119/300 Write block 4426 len 36864 data: 8 8 7 7 6 5 8 9 9
27108 120/300 Write block 4958 len 24576 data: 4 3 4 5 5 7
27109 121/300 Read block 2486 len 32768
27110 122/300 Read block 1927 len 4096
27111 123/300 Read block 4146 len 4096
27112 124/300 Read block 3616 len 36864
27113 125/300 Write block 193 len 20480 data: 8 6 7 6 5
27114 126/300 Write block 661 len 32768 data: 7 7 7 6 3 3 5 5
27115 127/300 Write block 801 len 8192 data: 5 4
27116 128/300 Flush
27117 129/300 Write block 2138 len 36864 data: 2 3 5 4 5 6 5 5 4
27118 130/300 Write block 4873 len 20480 data: 6 4 5 5 6
27119 131/300 Write block 644 len 20480 data: 4 4 4 4 4
27120 132/300 Read block 4746 len 36864
27121 133/300 Read block 2867 len 32768
27122 134/300 Flush
27123 135/300 Write block 1590 len 4096 data: 8
27124 136/300 Read block 2186 len 20480
27125 137/300 Read block 4753 len 28672
27126 138/300 Write block 1457 len 28672 data: 5 7 7 6 6 5 7
27127 139/300 Write block 1595 len 16384 data: 8 5 4 4
27128 140/300 Read block 1430 len 8192
27129 141/300 Write block 2202 len 8192 data: 6 5
27130 142/300 Read block 967 len 16384
27131 143/300 Read block 4933 len 16384
27132 144/300 Write block 843 len 24576 data: 6 7 5 5 8 7
27133 145/300 Write block 80 len 36864 data: 6 6 6 5 4 5 5 7 5
27134 146/300 Flush
27135 147/300 Write block 4914 len 36864 data: 4 4 2 2 2 2 3 4 5
27136 148/300 Write block 4452 len 8192 data: 5 5
27137 149/300 Write block 3900 len 24576 data: 4 4 5 5 7 7
27138 150/300 Write block 3079 len 32768 data: 7 8 9 9 10 7 7 7
27139 151/300 Read block 3791 len 16384
27140 152/300 Write block 1925 len 24576 data: 5 4 3 4 3 3
27141 153/300 Write block 2117 len 8192 data: 5 4
27142 154/300 Read block 1366 len 4096
27143 155/300 Read block 2122 len 28672
27144 156/300 Flush
27145 157/300 Read block 4588 len 24576
27146 158/300 Read block 3340 len 32768
27147 159/300 Write block 3653 len 36864 data: 4 4 4 3 4 4 4 5 5
27148 160/300 Write block 336 len 20480 data: 4 5 5 6 4
27149 161/300 Read block 4258 len 28672
27150 162/300 Write block 361 len 40960 data: 10 8 7 5 5 5 7 8 9 8
27151 163/300 Write block 4242 len 32768 data: 4 6 6 4 3 5 8 7
27152 164/300 Write block 505 len 16384 data: 6 7 8 9
27153 165/300 Read block 2741 len 12288
27154 166/300 Read block 1790 len 24576
27155 167/300 Read block 1209 len 8192
27156 168/300 Flush
27157 169/300 Read block 208 len 8192
27158 170/300 Read block 3639 len 12288
27159 171/300 Read block 2652 len 4096
27160 172/300 Flush
27161 173/300 Write block 974 len 20480 data: 6 6 5 7 6
27162 174/300 Read block 1189 len 32768
27163 175/300 Flush
27164 176/300 Read block 1031 len 16384
27165 177/300 Write block 2631 len 36864 data: 4 5 4 6 7 7 8 8 7
27166 178/300 Read block 474 len 8192
27167 179/300 Write block 1361 len 40960 data: 4 4 4 7 7 7 6 7 7 5
27168 180/300 Flush
27169 181/300 Read block 1468 len 8192
27170 182/300 Read block 1362 len 32768
27171 183/300 Read block 1799 len 32768
27172 184/300 Read block 3116 len 8192
27173 185/300 Read block 4566 len 8192
27174 186/300 Write block 4891 len 36864 data: 8 8 8 7 7 5 4 6 6
27175 187/300 Write block 1963 len 32768 data: 5 6 5 5 4 3 3 2
27176 188/300 Read block 4263 len 28672
27177 189/300 Flush
27178 190/300 Read block 4572 len 4096
27179 191/300 Write block 2303 len 12288 data: 5 6 5
27180 192/300 Read block 3621 len 32768
27181 193/300 Write block 4192 len 36864 data: 7 6 5 5 3 3 5 3 5
27182 194/300 Write block 556 len 28672 data: 5 6 8 8 7 7 5
27183 195/300 Write block 2672 len 20480 data: 7 7 5 5 4
27184 196/300 Write block 786 len 4096 data: 6
27185 197/300 Write block 3506 len 16384 data: 8 8 8 8
27186 198/300 Read block 2842 len 12288
27187 199/300 Read block 3642 len 20480
27188 200/300 Flush
27189 201/300 Read block 860 len 28672
27190 202/300 Flush
27191 203/300 Write block 1670 len 16384 data: 6 6 5 6
27192 204/300 Read block 391 len 12288
27193 205/300 Read block 1387 len 4096
27194 206/300 Read block 3253 len 8192
27195 207/300 Write block 2241 len 12288 data: 8 7 6
27196 208/300 Read block 3447 len 16384
27197 209/300 Write block 4545 len 16384 data: 8 11 10 10
27198 210/300 Write block 2696 len 12288 data: 2 3 3
27199 211/300 Flush
27200 212/300 Flush
27201 213/300 Read block 4056 len 8192
27202 214/300 Read block 1865 len 12288
27203 215/300 Read block 3493 len 12288
27204 216/300 Write block 2846 len 12288 data: 3 3 4
27205 217/300 Read block 2037 len 12288
27206 218/300 Write block 3597 len 8192 data: 6 8
27207 219/300 Write block 2537 len 12288 data: 7 5 5
27208 220/300 Read block 2667 len 16384
27209 221/300 Write block 2372 len 12288 data: 4 4 4
27210 222/300 Write block 4833 len 32768 data: 5 7 7 7 8 8 7 7
27211 223/300 Read block 2887 len 32768
27212 224/300 Read block 981 len 36864
27213 225/300 Write block 621 len 8192 data: 6 5
27214 226/300 Flush
27215 227/300 Write block 1008 len 12288 data: 7 8 9
27216 228/300 Write block 2323 len 16384 data: 4 3 5 6
27217 229/300 Read block 972 len 36864
27218 230/300 Read block 1879 len 32768
27219 231/300 Read block 2459 len 8192
27220 232/300 Read block 1599 len 24576
27221 233/300 Write block 3307 len 36864 data: 5 5 5 4 4 4 2 3 4
27222 234/300 Read block 4292 len 12288
27223 235/300 Write block 1543 len 12288 data: 7 9 9
27224 236/300 Flush
27225 237/300 Read block 4283 len 36864
27226 238/300 Read block 4318 len 8192
27227 239/300 Read block 4226 len 24576
27228 240/300 Write block 1935 len 40960 data: 2 3 3 4 5 5 5 5 6 4
27229 241/300 Read block 3021 len 28672
27230 242/300 Read block 290 len 20480
27231 243/300 Write block 3485 len 32768 data: 8 5 5 3 3 5 6 6
27232 244/300 Write block 1977 len 16384 data: 5 4 4 4
27233 245/300 Write block 3156 len 20480 data: 12 13 14 13 13
27234 246/300 Read block 1805 len 32768
27235 247/300 Write block 3143 len 20480 data: 8 7 7 6 7
27236 248/300 Flush
27237 249/300 Write block 696 len 36864 data: 6 5 5 6 6 6 6 6 7
27238 250/300 Read block 3235 len 36864
27239 251/300 Read block 1504 len 4096
27240 252/300 Flush
27241 253/300 Read block 4309 len 32768
27242 254/300 Read block 1505 len 36864
27243 255/300 Write block 4955 len 12288 data: 5 4 4
27244 256/300 Read block 2495 len 8192
27245 257/300 Flush
27246 258/300 Write block 1438 len 16384 data: 5 5 6 5
27247 259/300 Read block 2799 len 20480
27248 260/300 Write block 2438 len 8192 data: 6 5
27249 261/300 Read block 3956 len 24576
27250 262/300 Flush
27251 263/300 Read block 178 len 8192
27252 264/300 Write block 1725 len 32768 data: 6 7 6 6 8 8 6 8
27253 265/300 Write block 3958 len 40960 data: 6 6 5 6 4 5 4 5 4 4
27254 266/300 Read block 1234 len 20480
27255 267/300 Read block 1431 len 32768
27256 268/300 Read block 1023 len 8192
27257 269/300 Read block 2938 len 20480
27258 270/300 Read block 2820 len 16384
27259 271/300 Write block 2537 len 8192 data: 8 6
27260 272/300 Read block 1385 len 12288
27261 273/300 Write block 3204 len 8192 data: 6 5
27262 274/300 Read block 1891 len 40960
27263 275/300 Flush
27264 276/300 Write block 369 len 32768 data: 10 9 9 9 9 8 8 10
27265 277/300 Write block 1713 len 16384 data: 4 3 3 3
27266 278/300 Write block 963 len 40960 data: 5 5 5 4 4 3 5 5 5 6
27267 279/300 Read block 3146 len 16384
27268 280/300 Write block 68 len 4096 data: 4
27269 281/300 Write block 2306 len 40960 data: 5 4 6 4 5 4 4 4 6 5
27270 282/300 Flush
27271 283/300 Read block 747 len 28672
27272 284/300 Write block 58 len 8192 data: 5 4
27273 285/300 Read block 4164 len 8192
27274 286/300 Write block 536 len 20480 data: 9 7 5 5 4
27275 287/300 Read block 3414 len 4096
27276 288/300 Read block 1578 len 40960
27277 289/300 Read block 4494 len 12288
27278 290/300 Read block 2579 len 36864
27279 291/300 Read block 3077 len 36864
27280 292/300 Read block 2012 len 28672
27281 293/300 Read block 2020 len 4096
27282 294/300 Read block 4843 len 12288
27283 295/300 Read block 1353 len 24576
27284 296/300 Write block 2987 len 40960 data: 7 9 9 9 7 6 5 6 6 6
27285 297/300 Write block 637 len 8192 data: 7 4
27286 298/300 Write block 541 len 24576 data: 6 6 6 7 8 8
27287 299/300 Read block 87 len 8192
27288 300/300 Read block 3270 len 4096
27289 Main task has work to do, go find it
27290 got dsc Start(2)
27291 start 2
27292 Feb 15 03:02:56.889 INFO request completed, latency_us: 211, response_code: 204, uri: /start/cid/2, method: POST, req_id: 662ebaee-5ed7-4abe-b2dc-9b5788e928b4, remote_addr: 127.0.0.1:54744, local_addr: 127.0.0.1:9998
27293 Replay: started 2, returned:Ok(())
27294 ----------------------------------------------------------------
27295 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:300
27296 States: Active Active Offline
27297 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
27298 8742 Acked 9741 Write 5 Done Done New true
27299 8743 Acked 9742 Read 5 Done Done New true
27300 8744 Acked 9743 Read 4 Done Done New true
27301 8745 Acked 9744 Flush 0 Done Done New false
27302 8746 Acked 9745 Write 4 Done Done New false
27303 8747 Acked 9746 Read 10 Done Done New false
27304 8748 Acked 9747 Read 1 Done Done New false
27305 8749 Acked 9748 Write 3 Done Done New false
27306 8750 Acked 9749 Write 8 Done Done New false
27307 8751 Acked 9750 Write 4 Done Done New false
27308 8752 Acked 9751 Read 7 Done Done New false
27309 8753 Acked 9752 Flush 0 Done Done New false
27310 8754 Acked 9753 Write 1 Done Done New false
27311 8755 Acked 9754 Read 2 Done Done New false
27312 8756 Acked 9755 Write 7 Done Done New false
27313 8757 Acked 9756 Read 8 Done Done New false
27314 8758 Acked 9757 Write 9 Done Done New false
27315 8759 Acked 9758 Read 5 Done Done New false
27316 8760 Acked 9759 Read 8 Done Done New false
27317 8761 Acked 9760 Write 1 Done Done New false
27318 8762 Acked 9761 Read 3 Done Done New false
27319 8763 Acked 9762 Flush 0 Done Done New false
27320 8764 Acked 9763 Write 7 Done Done New false
27321 8765 Acked 9764 Read 10 Done Done New false
27322 8766 Acked 9765 Write 10 Done Done New false
27323 8767 Acked 9766 Write 2 Done Done New false
27324 8768 Acked 9767 Read 6 Done Done New false
27325 8769 Acked 9768 Read 6 Done Done New false
27326 8770 Acked 9769 Read 9 Done Done New false
27327 8771 Acked 9770 Write 1 Done Done New false
27328 8772 Acked 9771 Write 5 Done Done New false
27329 8773 Acked 9772 Read 6 Done Done New false
27330 8774 Acked 9773 Write 1 Done Done New false
27331 8775 Acked 9774 Read 1 Done Done New false
27332 8776 Acked 9775 Write 10 Done Done New false
27333 8777 Acked 9776 Flush 0 Done Done New false
27334 8778 Acked 9777 Write 9 Done Done New false
27335 8779 Acked 9778 Read 1 Done Done New false
27336 8780 Acked 9779 Read 5 Done Done New false
27337 8781 Acked 9780 Read 2 Done Done New false
27338 8782 Acked 9781 Read 3 Done Done New false
27339 8783 Acked 9782 Read 8 Done Done New false
27340 8784 Acked 9783 Write 3 Done Done New false
27341 8785 Acked 9784 Read 9 Done Done New false
27342 8786 Acked 9785 Write 4 Done Done New false
27343 8787 Acked 9786 Read 10 Done Done New false
27344 8788 Acked 9787 Read 9 Done Done New false
27345 8789 Acked 9788 Flush 0 Done Done New false
27346 8790 Acked 9789 Write 4 Done Done New false
27347 8791 Acked 9790 Flush 0 Done Done New false
27348 8792 Acked 9791 Read 6 Done Done New false
27349 8793 Acked 9792 Read 2 Done Done New false
27350 8794 Acked 9793 Write 5 Done Done New false
27351 8795 Acked 9794 Write 4 Done Done New false
27352 8796 Acked 9795 Flush 0 Done Done New false
27353 8797 Acked 9796 Read 6 Done Done New false
27354 8798 Acked 9797 Write 10 Done Done New false
27355 8799 Acked 9798 Write 10 Done Done New false
27356 8800 Acked 9799 Write 7 Done Done New false
27357 8801 Acked 9800 Read 3 Done Done New false
27358 8802 Acked 9801 Write 1 Done Done New false
27359 8803 Acked 9802 Write 4 Done Done New false
27360 8804 Acked 9803 Read 8 Done Done New false
27361 8805 Acked 9804 Read 2 Done Done New false
27362 8806 Acked 9805 Write 10 Done Done New false
27363 8807 Acked 9806 Write 4 Done Done New false
27364 8808 Acked 9807 Write 1 Done Done New false
27365 8809 Acked 9808 Write 2 Done Done New false
27366 8810 Acked 9809 Write 8 Done Done New false
27367 8811 Acked 9810 Flush 0 Done Done New false
27368 8812 Acked 9811 Read 4 Done Done New false
27369 8813 Acked 9812 Write 7 Done Done New false
27370 8814 Acked 9813 Flush 0 Done Done New false
27371 8815 Acked 9814 Read 5 Done Done New false
27372 8816 Acked 9815 Read 7 Done Done New false
27373 8817 Acked 9816 Write 10 Done Done New false
27374 8818 Acked 9817 Read 4 Done Done New false
27375 8819 Acked 9818 Write 2 Done Done New false
27376 8820 Acked 9819 Write 8 Done Done New false
27377 8821 Acked 9820 Write 9 Done Done New false
27378 8822 Acked 9821 Write 4 Done Done New false
27379 8823 Acked 9822 Write 2 Done Done New false
27380 8824 Acked 9823 Write 7 Done Done New false
27381 8825 Acked 9824 Write 5 Done Done New false
27382 8826 Acked 9825 Write 1 Done Done New false
27383 8827 Acked 9826 Write 10 Done Done New false
27384 8828 Acked 9827 Write 9 Done Done New false
27385 8829 Acked 9828 Write 1 Done Done New false
27386 8830 Acked 9829 Write 2 Done Done New false
27387 8831 Acked 9830 Read 7 Done Done New false
27388 8832 Acked 9831 Read 10 Done Done New false
27389 8833 Acked 9832 Write 10 Done Done New false
27390 8834 Acked 9833 Read 9 Done Done New false
27391 8835 Acked 9834 Write 4 Done Done New false
27392 8836 Acked 9835 Write 4 Done Done New false
27393 8837 Acked 9836 Read 10 Done Done New false
27394 8838 Acked 9837 Read 5 Done Done New false
27395 8839 Acked 9838 Read 4 Done Done New false
27396 8840 Acked 9839 Flush 0 Done Done New false
27397 8841 Acked 9840 Write 5 Done Done New false
27398 8842 Acked 9841 Read 9 Done Done New false
27399 8843 Acked 9842 Flush 0 Done Done New false
27400 8844 Acked 9843 Write 1 Done Done New false
27401 8845 Acked 9844 Read 5 Done Done New false
27402 8846 Acked 9845 Write 9 Done Done New false
27403 8847 Acked 9846 Write 3 Done Done New false
27404 8848 Acked 9847 Write 5 Done Done New false
27405 8849 Acked 9848 Read 4 Done Done New false
27406 8850 Acked 9849 Write 7 Done Done New false
27407 8851 Acked 9850 Read 10 Done Done New false
27408 8852 Acked 9851 Flush 0 Done Done New false
27409 8853 Acked 9852 Write 6 Done Done New false
27410 8854 Acked 9853 Read 6 Done Done New false
27411 8855 Acked 9854 Read 6 Done Done New false
27412 8856 Acked 9855 Flush 0 Done Done New false
27413 8857 Acked 9856 Read 2 Done Done New false
27414 8858 Acked 9857 Read 1 Done Done New false
27415 8859 Acked 9858 Read 2 Done Done New false
27416 8860 Acked 9859 Write 9 Done Done New false
27417 8861 Acked 9860 Write 6 Done Done New false
27418 8862 Acked 9861 Read 8 Done Done New false
27419 8863 Acked 9862 Read 1 Done Done New false
27420 8864 Acked 9863 Read 1 Done Done New false
27421 8865 Acked 9864 Read 9 Done Done New false
27422 8866 Acked 9865 Write 5 Done Done New false
27423 8867 Acked 9866 Write 8 Done Done New false
27424 8868 Acked 9867 Write 2 Done Done New false
27425 8869 Acked 9868 Flush 0 Done Done New false
27426 8870 Acked 9869 Write 9 Done Done New false
27427 8871 Acked 9870 Write 5 Done Done New false
27428 8872 Acked 9871 Write 5 Done Done New false
27429 8873 Acked 9872 Read 9 Done Done New false
27430 8874 Acked 9873 Read 8 Done Done New false
27431 8875 Acked 9874 Flush 0 Done Done New false
27432 8876 Acked 9875 Write 1 Done Done New false
27433 8877 Acked 9876 Read 5 Done Done New false
27434 8878 Acked 9877 Read 7 Done Done New false
27435 8879 Acked 9878 Write 7 Done Done New false
27436 8880 Acked 9879 Write 4 Done Done New false
27437 8881 Acked 9880 Read 2 Done Done New false
27438 8882 Acked 9881 Write 2 Done Done New false
27439 8883 Acked 9882 Read 4 Done Done New false
27440 8884 Acked 9883 Read 4 Done Done New false
27441 8885 Acked 9884 Write 6 Done Done New false
27442 8886 Acked 9885 Write 9 Done Done New false
27443 8887 Acked 9886 Flush 0 Done Done New false
27444 8888 Acked 9887 Write 9 Done Done New false
27445 8889 Acked 9888 Write 2 Done Done New false
27446 8890 Acked 9889 Write 6 Done Done New false
27447 8891 Acked 9890 Write 8 Done Done New false
27448 8892 Acked 9891 Read 4 Done Done New false
27449 8893 Acked 9892 Write 6 Done Done New false
27450 8894 Acked 9893 Write 2 Done Done New false
27451 8895 Acked 9894 Read 1 Done Done New false
27452 8896 Acked 9895 Read 7 Done Done New false
27453 8897 Acked 9896 Flush 0 Done Done New false
27454 8898 Acked 9897 Read 6 Done Done New false
27455 8899 Acked 9898 Read 8 Done Done New false
27456 8900 Acked 9899 Write 9 Done Done New false
27457 8901 Acked 9900 Write 5 Done Done New false
27458 8902 Acked 9901 Read 7 Done Done New false
27459 8903 Acked 9902 Write 10 Done Done New false
27460 8904 Acked 9903 Write 8 Done Done New false
27461 8905 Acked 9904 Write 4 Done Done New false
27462 8906 Acked 9905 Read 3 Done Done New false
27463 8907 Acked 9906 Read 6 Done Done New false
27464 8908 Acked 9907 Read 2 Done Done New false
27465 8909 Acked 9908 Flush 0 Done Done New false
27466 8910 Acked 9909 Read 2 Done Done New false
27467 8911 Acked 9910 Read 3 Done Done New false
27468 8912 Acked 9911 Read 1 Done Done New false
27469 8913 Acked 9912 Flush 0 Done Done New false
27470 8914 Acked 9913 Write 5 Done Done New false
27471 8915 Acked 9914 Read 8 Done Done New false
27472 8916 Acked 9915 Flush 0 Done Done New false
27473 8917 Acked 9916 Read 4 Done Done New false
27474 8918 Acked 9917 Write 9 Done Done New false
27475 8919 Acked 9918 Read 2 Done Done New false
27476 8920 Acked 9919 Write 10 Done Done New false
27477 8921 Acked 9920 Flush 0 Done Done New false
27478 8922 Acked 9921 Read 2 Done Done New false
27479 8923 Acked 9922 Read 8 Done Done New false
27480 8924 Acked 9923 Read 8 Done Done New false
27481 8925 Acked 9924 Read 2 Done Done New false
27482 8926 Acked 9925 Read 2 Done Done New false
27483 8927 Acked 9926 Write 9 Done Done New false
27484 8928 Acked 9927 Write 8 Done Done New false
27485 8929 Acked 9928 Read 7 Done Done New false
27486 8930 Acked 9929 Flush 0 Done Done New false
27487 8931 Acked 9930 Read 1 Done Done New false
27488 8932 Acked 9931 Write 3 Done Done New false
27489 8933 Acked 9932 Read 8 Done Done New false
27490 8934 Acked 9933 Write 9 Done Done New false
27491 8935 Acked 9934 Write 7 Done Done New false
27492 8936 Acked 9935 Write 5 Done Done New false
27493 8937 Acked 9936 Write 1 Done Done New false
27494 8938 Acked 9937 Write 4 Done Done New false
27495 8939 Acked 9938 Read 3 Done Done New false
27496 8940 Acked 9939 Read 5 Done Done New false
27497 8941 Acked 9940 Flush 0 Done Done New false
27498 8942 Acked 9941 Read 7 Done Done New false
27499 8943 Acked 9942 Flush 0 Done Done New false
27500 8944 Acked 9943 Write 4 Done Done New false
27501 8945 Acked 9944 Read 3 Done Done New false
27502 8946 Acked 9945 Read 1 Done Done New false
27503 8947 Acked 9946 Read 2 Done Done New false
27504 8948 Acked 9947 Write 3 Done Done New false
27505 8949 Acked 9948 Read 4 Done Done New false
27506 8950 Acked 9949 Write 4 Done Done New false
27507 8951 Acked 9950 Write 3 Done Done New false
27508 8952 Acked 9951 Flush 0 Done Done New false
27509 8953 Acked 9952 Flush 0 Done Done New false
27510 8954 Acked 9953 Read 2 Done Done New false
27511 8955 Acked 9954 Read 3 Done Done New false
27512 8956 Acked 9955 Read 3 Done Done New false
27513 8957 Acked 9956 Write 3 Done Done New false
27514 8958 Acked 9957 Read 3 Done Done New false
27515 8959 Acked 9958 Write 2 Done Done New false
27516 8960 Acked 9959 Write 3 Done Done New false
27517 8961 Acked 9960 Read 4 Done Done New false
27518 8962 Acked 9961 Write 3 Done Done New false
27519 8963 Acked 9962 Write 8 Done Done New false
27520 8964 Acked 9963 Read 8 Done Done New false
27521 8965 Acked 9964 Read 9 Done Done New false
27522 8966 Acked 9965 Write 2 Done Done New false
27523 8967 Acked 9966 Flush 0 Done Done New false
27524 8968 Acked 9967 Write 3 Done Done New false
27525 8969 Acked 9968 Write 4 Done Done New false
27526 8970 Acked 9969 Read 9 Done Done New false
27527 8971 Acked 9970 Read 8 Done Done New false
27528 8972 Acked 9971 Read 2 Done Done New false
27529 8973 Acked 9972 Read 6 Done Done New false
27530 8974 Acked 9973 Write 9 Done Done New false
27531 8975 Acked 9974 Read 3 Done Done New false
27532 8976 Acked 9975 Write 3 Done Done New false
27533 8977 Acked 9976 Flush 0 Done Done New false
27534 8978 Acked 9977 Read 9 Done Done New false
27535 8979 Acked 9978 Read 2 Done Done New false
27536 8980 Acked 9979 Read 6 Done Done New false
27537 8981 Acked 9980 Write 10 Done Done New false
27538 8982 Acked 9981 Read 7 Done Done New false
27539 8983 Acked 9982 Read 5 Done Done New false
27540 8984 Acked 9983 Write 8 Done Done New false
27541 8985 Acked 9984 Write 4 Done Done New false
27542 8986 Acked 9985 Write 5 Done Done New false
27543 8987 Acked 9986 Read 8 Done Done New false
27544 8988 Acked 9987 Write 5 Done Done New false
27545 8989 Acked 9988 Flush 0 Done Done New false
27546 8990 Acked 9989 Write 9 Done Done New false
27547 8991 Acked 9990 Read 9 Done Done New false
27548 8992 Acked 9991 Read 1 Done Done New false
27549 8993 Acked 9992 Flush 0 Done Done New false
27550 8994 Acked 9993 Read 8 Done Done New false
27551 8995 Acked 9994 Read 9 Done Done New false
27552 8996 Acked 9995 Write 3 Done Done New false
27553 8997 Acked 9996 Read 2 Done Done New false
27554 8998 Acked 9997 Flush 0 Done Done New false
27555 8999 Acked 9998 Write 4 Done Done New false
27556 9000 Acked 9999 Read 5 Done Done New false
27557 9001 Acked 10000 Write 2 Done Done New false
27558 9002 Acked 10001 Read 6 Done Done New false
27559 9003 Acked 10002 Flush 0 Done Done New false
27560 9004 Acked 10003 Read 2 Done Done New false
27561 9005 Acked 10004 Write 8 Done Done New false
27562 9006 Acked 10005 Write 10 Done Done New false
27563 9007 Acked 10006 Read 5 Done Done New false
27564 9008 Acked 10007 Read 8 Done Done New false
27565 9009 Acked 10008 Read 2 Done Done New false
27566 9010 Acked 10009 Read 5 Done Done New false
27567 9011 Acked 10010 Read 4 Done Done New false
27568 9012 Acked 10011 Write 2 Done Done New false
27569 9013 Acked 10012 Read 3 Done Done New false
27570 9014 Acked 10013 Write 2 Done Done New false
27571 9015 Acked 10014 Read 10 Done Done New false
27572 9016 Acked 10015 Flush 0 Done Done New false
27573 9017 Acked 10016 Write 8 Done Done New false
27574 9018 Acked 10017 Write 4 Done Done New false
27575 9019 Acked 10018 Write 10 Done Done New false
27576 9020 Acked 10019 Read 4 Done Done New false
27577 9021 Acked 10020 Write 1 Done Done New false
27578 9022 Acked 10021 Write 10 Done Done New false
27579 9023 Acked 10022 Flush 0 Done Done New false
27580 9024 Acked 10023 Read 7 Done Done New false
27581 9025 Acked 10024 Write 2 Done Done New false
27582 9026 Acked 10025 Read 2 Done Done New false
27583 9027 Acked 10026 Write 5 Done Done New false
27584 9028 Acked 10027 Read 1 Done Done New false
27585 9029 Acked 10028 Read 10 Done Done New false
27586 9030 Acked 10029 Read 3 Done Done New false
27587 9031 Acked 10030 Read 9 Done Done New false
27588 9032 Acked 10031 Read 9 Done Done New false
27589 9033 Acked 10032 Read 7 Done Done New false
27590 9034 Acked 10033 Read 1 Done Done New false
27591 9035 Acked 10034 Read 3 Done Done New false
27592 9036 Acked 10035 Read 6 Done Done New false
27593 9037 Acked 10036 Write 10 Done Done New false
27594 9038 Acked 10037 Write 2 Done Done New false
27595 9039 Acked 10038 Write 6 Done Done New false
27596 9040 Acked 10039 Read 2 Done Done New false
27597 9041 Acked 10040 Read 1 Done Done New false
27598 STATES DS:0 DS:1 DS:2 TOTAL
27599 New 0 0 300 300
27600 Sent 0 0 0 0
27601 Done 300 300 0 600
27602 Skipped 0 0 0 0
27603 Error 0 0 0 0
27604 Last Flush: 10022 10022 9740
27605 Downstairs last five completed: 9740 9739 9738 9737 9736
27606 Upstairs last five completed: 9041 9040 9039 9038 9037
27607 CLIENT: Up:0 ds:300 act:2
276082024-02-15T03:02:57.556ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
276092024-02-15T03:02:57.556ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
276102024-02-15T03:02:57.556ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
276112024-02-15T03:02:57.556ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
276122024-02-15T03:02:57.556ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
276132024-02-15T03:02:57.556ZINFOcrucible: [2] client re-new 301 jobs since flush 9740 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
276142024-02-15T03:02:57.556ZINFOcrucible: client task is exiting = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
276152024-02-15T03:02:58.557ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
276162024-02-15T03:02:58.557ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
276172024-02-15T03:02:58.557ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
276182024-02-15T03:02:58.557ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
276192024-02-15T03:02:58.557ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
276202024-02-15T03:02:58.557ZINFOcrucible: [2] client re-new 301 jobs since flush 9740 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
276212024-02-15T03:02:58.557ZINFOcrucible: client task is exiting = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
276222024-02-15T03:02:59.558ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
276232024-02-15T03:02:59.558ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
276242024-02-15T03:02:59.558ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
276252024-02-15T03:02:59.558ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
276262024-02-15T03:02:59.558ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
276272024-02-15T03:02:59.558ZINFOcrucible: [2] client re-new 301 jobs since flush 9740 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
276282024-02-15T03:02:59.559ZINFOcrucible: client task is exiting = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
27629 [8830] got start action
27630 [8830] I am going to restart
27631 Starting downstairs at port 8830
27632 Make output file at "/tmp/dsc/downstairs-8830.txt"
27633 [8830][2] reports Starting
276342024-02-15T03:03:00.559ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
276352024-02-15T03:03:00.559ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
276362024-02-15T03:03:00.559ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
276372024-02-15T03:03:00.559ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
276382024-02-15T03:03:00.559ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
276392024-02-15T03:03:00.559ZINFOcrucible: [2] client re-new 301 jobs since flush 9740 = downstairs session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
276402024-02-15T03:03:00.559ZINFOcrucible: client task is exiting = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
27641 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1197)
27642 [8830][2] reports Running
27643 ----------------------------------------------------------------
27644 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
27645 States: Active Active Offline
27646 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
27647 8742 Acked 9741 Write 5 Done Done New true
27648 8743 Acked 9742 Read 5 Done Done New true
27649 8744 Acked 9743 Read 4 Done Done New true
27650 8745 Acked 9744 Flush 0 Done Done New true
27651 8746 Acked 9745 Write 4 Done Done New true
27652 8747 Acked 9746 Read 10 Done Done New true
27653 8748 Acked 9747 Read 1 Done Done New true
27654 8749 Acked 9748 Write 3 Done Done New true
27655 8750 Acked 9749 Write 8 Done Done New true
27656 8751 Acked 9750 Write 4 Done Done New true
27657 8752 Acked 9751 Read 7 Done Done New true
27658 8753 Acked 9752 Flush 0 Done Done New true
27659 8754 Acked 9753 Write 1 Done Done New true
27660 8755 Acked 9754 Read 2 Done Done New true
27661 8756 Acked 9755 Write 7 Done Done New true
27662 8757 Acked 9756 Read 8 Done Done New true
27663 8758 Acked 9757 Write 9 Done Done New true
27664 8759 Acked 9758 Read 5 Done Done New true
27665 8760 Acked 9759 Read 8 Done Done New true
27666 8761 Acked 9760 Write 1 Done Done New true
27667 8762 Acked 9761 Read 3 Done Done New true
27668 8763 Acked 9762 Flush 0 Done Done New true
27669 8764 Acked 9763 Write 7 Done Done New true
27670 8765 Acked 9764 Read 10 Done Done New true
27671 8766 Acked 9765 Write 10 Done Done New true
27672 8767 Acked 9766 Write 2 Done Done New true
27673 8768 Acked 9767 Read 6 Done Done New true
27674 8769 Acked 9768 Read 6 Done Done New true
27675 8770 Acked 9769 Read 9 Done Done New true
27676 8771 Acked 9770 Write 1 Done Done New true
27677 8772 Acked 9771 Write 5 Done Done New true
27678 8773 Acked 9772 Read 6 Done Done New true
27679 8774 Acked 9773 Write 1 Done Done New true
27680 8775 Acked 9774 Read 1 Done Done New true
27681 8776 Acked 9775 Write 10 Done Done New true
27682 8777 Acked 9776 Flush 0 Done Done New true
27683 8778 Acked 9777 Write 9 Done Done New true
27684 8779 Acked 9778 Read 1 Done Done New true
27685 8780 Acked 9779 Read 5 Done Done New true
27686 8781 Acked 9780 Read 2 Done Done New true
27687 8782 Acked 9781 Read 3 Done Done New true
27688 8783 Acked 9782 Read 8 Done Done New true
27689 8784 Acked 9783 Write 3 Done Done New true
27690 8785 Acked 9784 Read 9 Done Done New true
27691 8786 Acked 9785 Write 4 Done Done New true
27692 8787 Acked 9786 Read 10 Done Done New true
27693 8788 Acked 9787 Read 9 Done Done New true
27694 8789 Acked 9788 Flush 0 Done Done New true
27695 8790 Acked 9789 Write 4 Done Done New true
27696 8791 Acked 9790 Flush 0 Done Done New true
27697 8792 Acked 9791 Read 6 Done Done New true
27698 8793 Acked 9792 Read 2 Done Done New true
27699 8794 Acked 9793 Write 5 Done Done New true
27700 8795 Acked 9794 Write 4 Done Done New true
27701 8796 Acked 9795 Flush 0 Done Done New true
27702 8797 Acked 9796 Read 6 Done Done New true
27703 8798 Acked 9797 Write 10 Done Done New true
27704 8799 Acked 9798 Write 10 Done Done New true
27705 8800 Acked 9799 Write 7 Done Done New true
27706 8801 Acked 9800 Read 3 Done Done New true
27707 8802 Acked 9801 Write 1 Done Done New true
27708 8803 Acked 9802 Write 4 Done Done New true
27709 8804 Acked 9803 Read 8 Done Done New true
27710 8805 Acked 9804 Read 2 Done Done New true
27711 8806 Acked 9805 Write 10 Done Done New true
27712 8807 Acked 9806 Write 4 Done Done New true
27713 8808 Acked 9807 Write 1 Done Done New true
27714 8809 Acked 9808 Write 2 Done Done New true
27715 8810 Acked 9809 Write 8 Done Done New true
27716 8811 Acked 9810 Flush 0 Done Done New true
27717 8812 Acked 9811 Read 4 Done Done New true
27718 8813 Acked 9812 Write 7 Done Done New true
27719 8814 Acked 9813 Flush 0 Done Done New true
27720 8815 Acked 9814 Read 5 Done Done New true
27721 8816 Acked 9815 Read 7 Done Done New true
27722 8817 Acked 9816 Write 10 Done Done New true
27723 8818 Acked 9817 Read 4 Done Done New true
27724 8819 Acked 9818 Write 2 Done Done New true
27725 8820 Acked 9819 Write 8 Done Done New true
27726 8821 Acked 9820 Write 9 Done Done New true
27727 8822 Acked 9821 Write 4 Done Done New true
27728 8823 Acked 9822 Write 2 Done Done New true
27729 8824 Acked 9823 Write 7 Done Done New true
27730 8825 Acked 9824 Write 5 Done Done New true
27731 8826 Acked 9825 Write 1 Done Done New true
27732 8827 Acked 9826 Write 10 Done Done New true
27733 8828 Acked 9827 Write 9 Done Done New true
27734 8829 Acked 9828 Write 1 Done Done New true
27735 8830 Acked 9829 Write 2 Done Done New true
27736 8831 Acked 9830 Read 7 Done Done New true
27737 8832 Acked 9831 Read 10 Done Done New true
27738 8833 Acked 9832 Write 10 Done Done New true
27739 8834 Acked 9833 Read 9 Done Done New true
27740 8835 Acked 9834 Write 4 Done Done New true
27741 8836 Acked 9835 Write 4 Done Done New true
27742 8837 Acked 9836 Read 10 Done Done New true
27743 8838 Acked 9837 Read 5 Done Done New true
27744 8839 Acked 9838 Read 4 Done Done New true
27745 8840 Acked 9839 Flush 0 Done Done New true
27746 8841 Acked 9840 Write 5 Done Done New true
27747 8842 Acked 9841 Read 9 Done Done New true
27748 8843 Acked 9842 Flush 0 Done Done New true
27749 8844 Acked 9843 Write 1 Done Done New true
27750 8845 Acked 9844 Read 5 Done Done New true
27751 8846 Acked 9845 Write 9 Done Done New true
27752 8847 Acked 9846 Write 3 Done Done New true
27753 8848 Acked 9847 Write 5 Done Done New true
27754 8849 Acked 9848 Read 4 Done Done New true
27755 8850 Acked 9849 Write 7 Done Done New true
27756 8851 Acked 9850 Read 10 Done Done New true
27757 8852 Acked 9851 Flush 0 Done Done New true
27758 8853 Acked 9852 Write 6 Done Done New true
27759 8854 Acked 9853 Read 6 Done Done New true
27760 8855 Acked 9854 Read 6 Done Done New true
27761 8856 Acked 9855 Flush 0 Done Done New true
27762 8857 Acked 9856 Read 2 Done Done New true
27763 8858 Acked 9857 Read 1 Done Done New true
27764 8859 Acked 9858 Read 2 Done Done New true
27765 8860 Acked 9859 Write 9 Done Done New true
27766 8861 Acked 9860 Write 6 Done Done New true
27767 8862 Acked 9861 Read 8 Done Done New true
27768 8863 Acked 9862 Read 1 Done Done New true
27769 8864 Acked 9863 Read 1 Done Done New true
27770 8865 Acked 9864 Read 9 Done Done New true
27771 8866 Acked 9865 Write 5 Done Done New true
27772 8867 Acked 9866 Write 8 Done Done New true
27773 8868 Acked 9867 Write 2 Done Done New true
27774 8869 Acked 9868 Flush 0 Done Done New true
27775 8870 Acked 9869 Write 9 Done Done New true
27776 8871 Acked 9870 Write 5 Done Done New true
27777 8872 Acked 9871 Write 5 Done Done New true
27778 8873 Acked 9872 Read 9 Done Done New true
27779 8874 Acked 9873 Read 8 Done Done New true
27780 8875 Acked 9874 Flush 0 Done Done New true
27781 8876 Acked 9875 Write 1 Done Done New true
27782 8877 Acked 9876 Read 5 Done Done New true
27783 8878 Acked 9877 Read 7 Done Done New true
27784 8879 Acked 9878 Write 7 Done Done New true
27785 8880 Acked 9879 Write 4 Done Done New true
27786 8881 Acked 9880 Read 2 Done Done New true
27787 8882 Acked 9881 Write 2 Done Done New true
27788 8883 Acked 9882 Read 4 Done Done New true
27789 8884 Acked 9883 Read 4 Done Done New true
27790 8885 Acked 9884 Write 6 Done Done New true
27791 8886 Acked 9885 Write 9 Done Done New true
27792 8887 Acked 9886 Flush 0 Done Done New true
27793 8888 Acked 9887 Write 9 Done Done New true
27794 8889 Acked 9888 Write 2 Done Done New true
27795 8890 Acked 9889 Write 6 Done Done New true
27796 8891 Acked 9890 Write 8 Done Done New true
27797 8892 Acked 9891 Read 4 Done Done New true
27798 8893 Acked 9892 Write 6 Done Done New true
27799 8894 Acked 9893 Write 2 Done Done New true
27800 8895 Acked 9894 Read 1 Done Done New true
27801 8896 Acked 9895 Read 7 Done Done New true
27802 8897 Acked 9896 Flush 0 Done Done New true
27803 8898 Acked 9897 Read 6 Done Done New true
27804 8899 Acked 9898 Read 8 Done Done New true
27805 8900 Acked 9899 Write 9 Done Done New true
27806 8901 Acked 9900 Write 5 Done Done New true
27807 8902 Acked 9901 Read 7 Done Done New true
27808 8903 Acked 9902 Write 10 Done Done New true
27809 8904 Acked 9903 Write 8 Done Done New true
27810 8905 Acked 9904 Write 4 Done Done New true
27811 8906 Acked 9905 Read 3 Done Done New true
27812 8907 Acked 9906 Read 6 Done Done New true
27813 8908 Acked 9907 Read 2 Done Done New true
27814 8909 Acked 9908 Flush 0 Done Done New true
27815 8910 Acked 9909 Read 2 Done Done New true
27816 8911 Acked 9910 Read 3 Done Done New true
27817 8912 Acked 9911 Read 1 Done Done New true
27818 8913 Acked 9912 Flush 0 Done Done New true
27819 8914 Acked 9913 Write 5 Done Done New true
27820 8915 Acked 9914 Read 8 Done Done New true
27821 8916 Acked 9915 Flush 0 Done Done New true
27822 8917 Acked 9916 Read 4 Done Done New true
27823 8918 Acked 9917 Write 9 Done Done New true
27824 8919 Acked 9918 Read 2 Done Done New true
27825 8920 Acked 9919 Write 10 Done Done New true
27826 8921 Acked 9920 Flush 0 Done Done New true
27827 8922 Acked 9921 Read 2 Done Done New true
27828 8923 Acked 9922 Read 8 Done Done New true
27829 8924 Acked 9923 Read 8 Done Done New true
27830 8925 Acked 9924 Read 2 Done Done New true
27831 8926 Acked 9925 Read 2 Done Done New true
27832 8927 Acked 9926 Write 9 Done Done New true
27833 8928 Acked 9927 Write 8 Done Done New true
27834 8929 Acked 9928 Read 7 Done Done New true
27835 8930 Acked 9929 Flush 0 Done Done New true
27836 8931 Acked 9930 Read 1 Done Done New true
27837 8932 Acked 9931 Write 3 Done Done New true
27838 8933 Acked 9932 Read 8 Done Done New true
27839 8934 Acked 9933 Write 9 Done Done New true
27840 8935 Acked 9934 Write 7 Done Done New true
27841 8936 Acked 9935 Write 5 Done Done New true
27842 8937 Acked 9936 Write 1 Done Done New true
27843 8938 Acked 9937 Write 4 Done Done New true
27844 8939 Acked 9938 Read 3 Done Done New true
27845 8940 Acked 9939 Read 5 Done Done New true
27846 8941 Acked 9940 Flush 0 Done Done New true
27847 8942 Acked 9941 Read 7 Done Done New true
27848 8943 Acked 9942 Flush 0 Done Done New true
27849 8944 Acked 9943 Write 4 Done Done New true
27850 8945 Acked 9944 Read 3 Done Done New true
27851 8946 Acked 9945 Read 1 Done Done New true
27852 8947 Acked 9946 Read 2 Done Done New true
27853 8948 Acked 9947 Write 3 Done Done New true
27854 8949 Acked 9948 Read 4 Done Done New true
27855 8950 Acked 9949 Write 4 Done Done New true
27856 8951 Acked 9950 Write 3 Done Done New true
27857 8952 Acked 9951 Flush 0 Done Done New true
27858 8953 Acked 9952 Flush 0 Done Done New true
27859 8954 Acked 9953 Read 2 Done Done New true
27860 8955 Acked 9954 Read 3 Done Done New true
27861 8956 Acked 9955 Read 3 Done Done New true
27862 8957 Acked 9956 Write 3 Done Done New true
27863 8958 Acked 9957 Read 3 Done Done New true
27864 8959 Acked 9958 Write 2 Done Done New true
27865 8960 Acked 9959 Write 3 Done Done New true
27866 8961 Acked 9960 Read 4 Done Done New true
27867 8962 Acked 9961 Write 3 Done Done New true
27868 8963 Acked 9962 Write 8 Done Done New true
27869 8964 Acked 9963 Read 8 Done Done New true
27870 8965 Acked 9964 Read 9 Done Done New true
27871 8966 Acked 9965 Write 2 Done Done New true
27872 8967 Acked 9966 Flush 0 Done Done New true
27873 8968 Acked 9967 Write 3 Done Done New true
27874 8969 Acked 9968 Write 4 Done Done New true
27875 8970 Acked 9969 Read 9 Done Done New true
27876 8971 Acked 9970 Read 8 Done Done New true
27877 8972 Acked 9971 Read 2 Done Done New true
27878 8973 Acked 9972 Read 6 Done Done New true
27879 8974 Acked 9973 Write 9 Done Done New true
27880 8975 Acked 9974 Read 3 Done Done New true
27881 8976 Acked 9975 Write 3 Done Done New true
27882 8977 Acked 9976 Flush 0 Done Done New true
27883 8978 Acked 9977 Read 9 Done Done New true
27884 8979 Acked 9978 Read 2 Done Done New true
27885 8980 Acked 9979 Read 6 Done Done New true
27886 8981 Acked 9980 Write 10 Done Done New true
27887 8982 Acked 9981 Read 7 Done Done New true
27888 8983 Acked 9982 Read 5 Done Done New true
27889 8984 Acked 9983 Write 8 Done Done New true
27890 8985 Acked 9984 Write 4 Done Done New true
27891 8986 Acked 9985 Write 5 Done Done New true
27892 8987 Acked 9986 Read 8 Done Done New true
27893 8988 Acked 9987 Write 5 Done Done New true
27894 8989 Acked 9988 Flush 0 Done Done New true
27895 8990 Acked 9989 Write 9 Done Done New true
27896 8991 Acked 9990 Read 9 Done Done New true
27897 8992 Acked 9991 Read 1 Done Done New true
27898 8993 Acked 9992 Flush 0 Done Done New true
27899 8994 Acked 9993 Read 8 Done Done New true
27900 8995 Acked 9994 Read 9 Done Done New true
27901 8996 Acked 9995 Write 3 Done Done New true
27902 8997 Acked 9996 Read 2 Done Done New true
27903 8998 Acked 9997 Flush 0 Done Done New true
27904 8999 Acked 9998 Write 4 Done Done New true
27905 9000 Acked 9999 Read 5 Done Done New true
27906 9001 Acked 10000 Write 2 Done Done New true
27907 9002 Acked 10001 Read 6 Done Done New true
27908 9003 Acked 10002 Flush 0 Done Done New true
27909 9004 Acked 10003 Read 2 Done Done New true
27910 9005 Acked 10004 Write 8 Done Done New true
27911 9006 Acked 10005 Write 10 Done Done New true
27912 9007 Acked 10006 Read 5 Done Done New true
27913 9008 Acked 10007 Read 8 Done Done New true
27914 9009 Acked 10008 Read 2 Done Done New true
27915 9010 Acked 10009 Read 5 Done Done New true
27916 9011 Acked 10010 Read 4 Done Done New true
27917 9012 Acked 10011 Write 2 Done Done New true
27918 9013 Acked 10012 Read 3 Done Done New true
27919 9014 Acked 10013 Write 2 Done Done New true
27920 9015 Acked 10014 Read 10 Done Done New true
27921 9016 Acked 10015 Flush 0 Done Done New true
27922 9017 Acked 10016 Write 8 Done Done New true
27923 9018 Acked 10017 Write 4 Done Done New true
27924 9019 Acked 10018 Write 10 Done Done New true
27925 9020 Acked 10019 Read 4 Done Done New true
27926 9021 Acked 10020 Write 1 Done Done New true
27927 9022 Acked 10021 Write 10 Done Done New true
27928 9023 Acked 10022 Flush 0 Done Done New true
27929 9024 Acked 10023 Read 7 Done Done New true
27930 9025 Acked 10024 Write 2 Done Done New true
27931 9026 Acked 10025 Read 2 Done Done New true
27932 9027 Acked 10026 Write 5 Done Done New true
27933 9028 Acked 10027 Read 1 Done Done New true
27934 9029 Acked 10028 Read 10 Done Done New true
27935 9030 Acked 10029 Read 3 Done Done New true
27936 9031 Acked 10030 Read 9 Done Done New true
27937 9032 Acked 10031 Read 9 Done Done New true
27938 9033 Acked 10032 Read 7 Done Done New true
27939 9034 Acked 10033 Read 1 Done Done New true
27940 9035 Acked 10034 Read 3 Done Done New true
27941 9036 Acked 10035 Read 6 Done Done New true
27942 9037 Acked 10036 Write 10 Done Done New true
27943 9038 Acked 10037 Write 2 Done Done New true
27944 9039 Acked 10038 Write 6 Done Done New true
27945 9040 Acked 10039 Read 2 Done Done New true
27946 9041 Acked 10040 Read 1 Done Done New true
27947 9042 Acked 10041 Flush 0 Done Done New true
27948 STATES DS:0 DS:1 DS:2 TOTAL
27949 New 0 0 301 301
27950 Sent 0 0 0 0
27951 Done 301 301 0 602
27952 Skipped 0 0 0 0
27953 Error 0 0 0 0
27954 Last Flush: 10041 10041 9740
27955 Downstairs last five completed: 9740 9739 9738 9737 9736
27956 Upstairs last five completed: 9042 9041 9040 9039 9038
27957 CLIENT: Up:0 ds:301 act:2
279582024-02-15T03:03:01.561ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
279592024-02-15T03:03:01.561ZINFOcrucible: ds_connection connected = io task client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
279602024-02-15T03:03:01.562ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
279612024-02-15T03:03:01.562ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
279622024-02-15T03:03:01.562ZINFOcrucible: send last flush ID to this DS: 9740 = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
279632024-02-15T03:03:01.562ZINFOcrucible: Replied this last flush ID: 9740 = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
279642024-02-15T03:03:01.562ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
279652024-02-15T03:03:01.562ZINFOcrucible: [2] Transition from Offline to Replay = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
279662024-02-15T03:03:01.562ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
279672024-02-15T03:03:01.562ZINFOcrucible: [2] Transition from Replay to Active = downstairs client = 2 session_id = 4f5a63e5-aa00-4890-9f6b-96c2e069fdb2
27968 ----------------------------------------------------------------
27969 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
27970 Downstairs last five completed: 10041 10040 10039 10038 10037
27971 Upstairs last five completed: 9042 9041 9040 9039 9038
27972 CLIENT: Up:0 ds:0 act:3
27973 Replay: All jobs finished, all DS active.
27974 Test replay has completed
27975 Wrote out file "alan"
27976 CLIENT: Tests done. All submitted work has been ACK'd
27977 ----------------------------------------------------------------
27978 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
27979 Downstairs last five completed: 10041 10040 10039 10038 10037
27980 Upstairs last five completed: 9042 9041 9040 9039 9038
27981 CLIENT: Up:0 ds:0 act:3
27982 CLIENT: All jobs finished, all DS active.
27983 Replays:30 time: 5:30
27984 Do final verify
279852024-02-15T03:03:04.927ZINFOcrucible: Upstairs starts
279862024-02-15T03:03:04.927ZINFOcrucible: Crucible Version: BuildInfo { version: "0.0.1", git_sha: "56eb785b9570b1bd468a7aafe9af80e337a7fae9", git_commit_timestamp: "2024-02-15T02:33:40.000000000Z", git_branch: "main", rustc_semver: "1.75.0", rustc_channel: "stable", rustc_host_triple: "x86_64-unknown-illumos", rustc_commit_sha: "82e1608dfa6e0b5569232559e3d385fea5a93112", cargo_triple: "x86_64-unknown-illumos", debug: true, opt_level: 0, }
279872024-02-15T03:03:04.927ZINFOcrucible: Upstairs <-> Downstairs Message Version: 5
27988 {"msg":"Crucible ec4510b9-de97-4d1f-8d7e-e1344b9ac59e has session id: 290d6f38-b1e7-4a61-99aa-5f52c3a951ab","v":0,"name":"crucible","level":30,"time":"Crucible runtime is spawned
27989 2024-02-15T03:03:04.92736899Z","hostname":"ip-10-150-1-67.us-west-2.compute.internal","pid":1202,"session_id":"290d6f38-b1e7-4a61-99aa-5f52c3a951ab"}
279902024-02-15T03:03:04.927ZINFOcrucible: Upstairs opts: Upstairs UUID: ec4510b9-de97-4d1f-8d7e-e1344b9ac59e, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false session_id = 290d6f38-b1e7-4a61-99aa-5f52c3a951ab
279912024-02-15T03:03:04.927ZINFOcrucible: Crucible stats registered with UUID: ec4510b9-de97-4d1f-8d7e-e1344b9ac59e session_id = 290d6f38-b1e7-4a61-99aa-5f52c3a951ab
279922024-02-15T03:03:04.927ZINFOcrucible: The guest has requested activation with gen:3
279932024-02-15T03:03:04.927ZINFOcrucible: ec4510b9-de97-4d1f-8d7e-e1344b9ac59e active request set session_id = 290d6f38-b1e7-4a61-99aa-5f52c3a951ab
279942024-02-15T03:03:04.927ZINFOcrucible: sending connect oneshot to client = downstairs client = 0 session_id = 290d6f38-b1e7-4a61-99aa-5f52c3a951ab
279952024-02-15T03:03:04.927ZINFOcrucible: client set_active_request while in New; waiting... = downstairs client = 0 session_id = 290d6f38-b1e7-4a61-99aa-5f52c3a951ab
279962024-02-15T03:03:04.927ZINFOcrucible: sending connect oneshot to client = downstairs client = 1 session_id = 290d6f38-b1e7-4a61-99aa-5f52c3a951ab
279972024-02-15T03:03:04.927ZINFOcrucible: client set_active_request while in New; waiting... = downstairs client = 1 session_id = 290d6f38-b1e7-4a61-99aa-5f52c3a951ab
279982024-02-15T03:03:04.927ZINFOcrucible: sending connect oneshot to client = downstairs client = 2 session_id = 290d6f38-b1e7-4a61-99aa-5f52c3a951ab
279992024-02-15T03:03:04.927ZINFOcrucible: client set_active_request while in New; waiting... = downstairs client = 2 session_id = 290d6f38-b1e7-4a61-99aa-5f52c3a951ab
280002024-02-15T03:03:04.927ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 290d6f38-b1e7-4a61-99aa-5f52c3a951ab
280012024-02-15T03:03:04.927ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 290d6f38-b1e7-4a61-99aa-5f52c3a951ab
280022024-02-15T03:03:04.927ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 290d6f38-b1e7-4a61-99aa-5f52c3a951ab
280032024-02-15T03:03:04.927ZINFOcrucible: ds_connection connected = io task client = 2 session_id = 290d6f38-b1e7-4a61-99aa-5f52c3a951ab
280042024-02-15T03:03:04.928ZINFOcrucible: ds_connection connected = io task client = 1 session_id = 290d6f38-b1e7-4a61-99aa-5f52c3a951ab
280052024-02-15T03:03:04.928ZINFOcrucible: ds_connection connected = io task client = 0 session_id = 290d6f38-b1e7-4a61-99aa-5f52c3a951ab
280062024-02-15T03:03:04.928ZINFOcrucible: ds_transition from New to WaitActive = downstairs client = 2 session_id = 290d6f38-b1e7-4a61-99aa-5f52c3a951ab
280072024-02-15T03:03:04.928ZINFOcrucible: [2] Transition from New to WaitActive = downstairs client = 2 session_id = 290d6f38-b1e7-4a61-99aa-5f52c3a951ab
280082024-02-15T03:03:04.928ZINFOcrucible: ds_transition from New to WaitActive = downstairs client = 1 session_id = 290d6f38-b1e7-4a61-99aa-5f52c3a951ab
280092024-02-15T03:03:04.928ZINFOcrucible: [1] Transition from New to WaitActive = downstairs client = 1 session_id = 290d6f38-b1e7-4a61-99aa-5f52c3a951ab
280102024-02-15T03:03:04.928ZINFOcrucible: ds_transition from New to WaitActive = downstairs client = 0 session_id = 290d6f38-b1e7-4a61-99aa-5f52c3a951ab
280112024-02-15T03:03:04.928ZINFOcrucible: [0] Transition from New to WaitActive = downstairs client = 0 session_id = 290d6f38-b1e7-4a61-99aa-5f52c3a951ab
280122024-02-15T03:03:04.929ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 290d6f38-b1e7-4a61-99aa-5f52c3a951ab
280132024-02-15T03:03:04.929ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 290d6f38-b1e7-4a61-99aa-5f52c3a951ab
280142024-02-15T03:03:04.929ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 290d6f38-b1e7-4a61-99aa-5f52c3a951ab
280152024-02-15T03:03:04.930ZINFOcrucible: ds_transition from WaitActive to WaitQuorum = downstairs client = 2 session_id = 290d6f38-b1e7-4a61-99aa-5f52c3a951ab
280162024-02-15T03:03:04.930ZINFOcrucible: [2] Transition from WaitActive to WaitQuorum = downstairs client = 2 session_id = 290d6f38-b1e7-4a61-99aa-5f52c3a951ab
280172024-02-15T03:03:04.930ZINFOcrucible: Waiting for true more clients to be ready session_id = 290d6f38-b1e7-4a61-99aa-5f52c3a951ab
280182024-02-15T03:03:04.930ZINFOcrucible: ds_transition from WaitActive to WaitQuorum = downstairs client = 1 session_id = 290d6f38-b1e7-4a61-99aa-5f52c3a951ab
280192024-02-15T03:03:04.930ZINFOcrucible: [1] Transition from WaitActive to WaitQuorum = downstairs client = 1 session_id = 290d6f38-b1e7-4a61-99aa-5f52c3a951ab
280202024-02-15T03:03:04.930ZINFOcrucible: Waiting for true more clients to be ready session_id = 290d6f38-b1e7-4a61-99aa-5f52c3a951ab
280212024-02-15T03:03:04.930ZINFOcrucible: ds_transition from WaitActive to WaitQuorum = downstairs client = 0 session_id = 290d6f38-b1e7-4a61-99aa-5f52c3a951ab
280222024-02-15T03:03:04.930ZINFOcrucible: [0] Transition from WaitActive to WaitQuorum = downstairs client = 0 session_id = 290d6f38-b1e7-4a61-99aa-5f52c3a951ab
280232024-02-15T03:03:04.930ZINFOcrucible: [0]R flush_numbers[0..12]: [932, 911, 891, 931, 902, 932, 932, 927, 913, 931, 925, 907] = downstairs session_id = 290d6f38-b1e7-4a61-99aa-5f52c3a951ab
280242024-02-15T03:03:04.930ZINFOcrucible: [0]R generation[0..12]: [2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2] = downstairs session_id = 290d6f38-b1e7-4a61-99aa-5f52c3a951ab
280252024-02-15T03:03:04.930ZINFOcrucible: [0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = 290d6f38-b1e7-4a61-99aa-5f52c3a951ab
280262024-02-15T03:03:04.930ZINFOcrucible: [1]R flush_numbers[0..12]: [932, 911, 891, 931, 902, 932, 932, 927, 913, 931, 925, 907] = downstairs session_id = 290d6f38-b1e7-4a61-99aa-5f52c3a951ab
280272024-02-15T03:03:04.930ZINFOcrucible: [1]R generation[0..12]: [2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2] = downstairs session_id = 290d6f38-b1e7-4a61-99aa-5f52c3a951ab
280282024-02-15T03:03:04.930ZINFOcrucible: [1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = 290d6f38-b1e7-4a61-99aa-5f52c3a951ab
280292024-02-15T03:03:04.930ZINFOcrucible: [2]R flush_numbers[0..12]: [932, 911, 891, 931, 902, 932, 932, 927, 913, 931, 925, 907] = downstairs session_id = 290d6f38-b1e7-4a61-99aa-5f52c3a951ab
280302024-02-15T03:03:04.930ZINFOcrucible: [2]R generation[0..12]: [2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2] = downstairs session_id = 290d6f38-b1e7-4a61-99aa-5f52c3a951ab
280312024-02-15T03:03:04.930ZINFOcrucible: [2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = 290d6f38-b1e7-4a61-99aa-5f52c3a951ab
280322024-02-15T03:03:04.930ZINFOcrucible: Max found gen is 3 = downstairs session_id = 290d6f38-b1e7-4a61-99aa-5f52c3a951ab
28033 {"msg":"Generation requested: 3 >= found:3","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO
28034 ,"time":"2024-02-15T03:03:04.930447023Z","hostname":"ip-10-150-1-67.us-west-2.compute.internal","pid":1202,"":"downstairs","":"downstairs","session_id":"290d6f38-b1e7-4a61-99aa-5f52c3a951ab"}
280352024-02-15T03:03:04.930ZINFOcrucible: Next flush: 933 = downstairs session_id = 290d6f38-b1e7-4a61-99aa-5f52c3a951ab
280362024-02-15T03:03:04.930ZINFOcrucible: All extents match = downstairs session_id = 290d6f38-b1e7-4a61-99aa-5f52c3a951ab
280372024-02-15T03:03:04.930ZINFOcrucible: No downstairs reconciliation required session_id = 290d6f38-b1e7-4a61-99aa-5f52c3a951ab
280382024-02-15T03:03:04.930ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 0 session_id = 290d6f38-b1e7-4a61-99aa-5f52c3a951ab
280392024-02-15T03:03:04.930ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 1 session_id = 290d6f38-b1e7-4a61-99aa-5f52c3a951ab
280402024-02-15T03:03:04.930ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 2 session_id = 290d6f38-b1e7-4a61-99aa-5f52c3a951ab
280412024-02-15T03:03:04.930ZINFOcrucible: All required reconciliation work is completed session_id = 290d6f38-b1e7-4a61-99aa-5f52c3a951ab
280422024-02-15T03:03:04.930ZINFOcrucible: Set Downstairs and Upstairs active after reconciliation session_id = 290d6f38-b1e7-4a61-99aa-5f52c3a951ab
280432024-02-15T03:03:04.930ZINFOcrucible: ec4510b9-de97-4d1f-8d7e-e1344b9ac59e is now active with session: 290d6f38-b1e7-4a61-99aa-5f52c3a951ab session_id = 290d6f38-b1e7-4a61-99aa-5f52c3a951ab
280442024-02-15T03:03:04.930ZINFOcrucible: Set Active after no reconciliation session_id = 290d6f38-b1e7-4a61-99aa-5f52c3a951ab
280452024-02-15T03:03:04.930ZINFOcrucible: The guest has finished waiting for activation with:3
28046 Region: es:100 ec:50 bs:4096 ts:20480000 tb:5000 max_io:256 or 1048576
28047 Loading write count information from file "alan"
28048 Read and Verify all blocks (0..5000 range:false)
28049 Wrote out file "alan"
28050 Verify test completed
28051 Wrote out file "alan"
28052 CLIENT: Tests done. All submitted work has been ACK'd
28053 ----------------------------------------------------------------
28054 Crucible gen:3 GIO:true work queues: Upstairs:0 downstairs:5
28055 States: Active Active Active
28056 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
28057 47 Acked 1046 Read 100 Done Done Done false
28058 48 Acked 1047 Read 100 Done Done Done false
28059 49 Acked 1048 Read 100 Done Done Done false
28060 50 Acked 1049 Read 100 Done Done Done false
28061 51 Acked 1050 Read 100 Done Done Done false
28062 STATES DS:0 DS:1 DS:2 TOTAL
28063 New 0 0 0 0
28064 Sent 0 0 0 0
28065 Done 5 5 5 15
28066 Skipped 0 0 0 0
28067 Error 0 0 0 0
28068 Last Flush: 1045 1045 1045
28069 Downstairs last five completed: 1045 1044 1043 1042 1041
28070 Upstairs last five completed: 51 50 49 48 47
28071 CLIENT: Up:0 ds:5 act:3
28072 ----------------------------------------------------------------
28073 Crucible gen:3 GIO:true work queues: Upstairs:0 downstairs:0
28074 Downstairs last five completed: 1051 1050 1049 1048 1047
28075 Upstairs last five completed: 52 51 50 49 48
28076 CLIENT: Up:0 ds:0 act:3
28077 CLIENT: All crucible jobs finished, exiting program
280782024-02-15T03:03:09.497ZWARNcrucible: Guest handle has been dropped
28079 Feb 15 03:03:09.556 INFO accepted connection, remote_addr: 127.0.0.1:59067, local_addr: 127.0.0.1:9998
28080 Main task has work to do, go find it
28081 got dsc Shutdown
28082 Shutdown
28083 Feb 15 03:03:09.557 INFO request completedShut it down
28084 , [8810] Disable keep_running
28085 latency_us[8810] Got stop action so:false kr:false
28086 [8820] Disable keep_running
28087 :[8820] Got stop action so:false kr:false
28088 177, response_code: 204, uri: /shutdown, method: POST, req_id: 440e2c10-cad1-4f32-9570-999f10102a39, remote_addr: 127.0.0.1:59067, local_addr: 127.0.0.1:9998
28089 [8830] Disable keep_running
28090 [8830] Got stop action so:false kr:false
28091 [8830] Exited with: ExitStatus(unix_wait_status(9))
28092 [8830][2] reports Exit
28093 [8810] Exited with: ExitStatus(unix_wait_status(9))
28094 [8810][0] reports Exit
28095 [8820] Exited with: ExitStatus(unix_wait_status(9))
28096 [8820][1] reports Exit
28097 February 15, 2024 at 03:03:14 AM UTC Test ends with 0