Max level shown:
1 starting February 12, 2024 at 01:34:23 AM UTC
2 Creating downstairs regions
3 Creating region directory at: /var/tmp/dsc/region
4 Creating dsc directory at: /tmp/dsc
5 Downstairs region 0 created at /var/tmp/dsc/region/8810 in 0.31881586
6 Downstairs region 1 created at /var/tmp/dsc/region/8820 in 0.08459226
7 Downstairs region 2 created at /var/tmp/dsc/region/8830 in 0.06709541
8 Region set with 3 regions was created
9 Starting downstairs
10 Using existing output directory "/tmp/dsc"
11 start ds: 8810
12 start ds: 8820
13 start ds: 8830
14 start access at:127.0.0.1:9998
15 Starting downstairs at port 8810
16 Starting downstairs at port 8820
17 Starting downstairs at port 8830
18 Make output file at "/tmp/dsc/downstairs-8820.txt"
19 Make output file at "/tmp/dsc/downstairs-8810.txt"
20 Make output file at "/tmp/dsc/downstairs-8830.txt"
21 [8820][1] reports Starting
22 [8810][0] reports Starting
23 [8830][2] reports Starting
24 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1154)
25 [8820][1] reports Running
26 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1155)
27 [8830][2] reports Running
28 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1156)
29 [8810][0] reports Running
30 Feb 12 01:34:23.631 INFO listening, local_addr: 127.0.0.1:9998
31 Control access at:127.0.0.1:9998
32 Running initial fill
332024-02-12T01:34:28.647ZINFOcrucible: Upstairs starts
342024-02-12T01:34:28.648ZINFOcrucible: Crucible Version: BuildInfo { version: "0.0.1", git_sha: "9f74bb5fe2be09de9f7a47f2db64d7dce836b79c", git_commit_timestamp: "2024-02-12T01:11:14.000000000Z", git_branch: "main", rustc_semver: "1.75.0", rustc_channel: "stable", rustc_host_triple: "x86_64-unknown-illumos", rustc_commit_sha: "82e1608dfa6e0b5569232559e3d385fea5a93112", cargo_triple: "x86_64-unknown-illumos", debug: true, opt_level: 0, }
352024-02-12T01:34:28.648ZINFOcrucible: Upstairs <-> Downstairs Message Version: 5
362024-02-12T01:34:28.648ZINFOcrucible: Crucible 0a1ee051-0183-48af-8098-5c16f918bdc1 has session id: 6e93a631-da6f-4a72-8667-3a4d9920c190 session_id = 6e93a631-da6f-4a72-8667-3a4d9920c190
372024-02-12T01:34:28.648ZINFOcrucible: Upstairs opts: Upstairs UUID: 0a1ee051-0183-48af-8098-5c16f918bdc1, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false session_id = 6e93a631-da6f-4a72-8667-3a4d9920c190
382024-02-12T01:34:28.648ZINFOcrucible: Crucible stats registered with UUID: 0a1ee051-0183-48af-8098-5c16f918bdc1 session_id = 6e93a631-da6f-4a72-8667-3a4d9920c190
39 Crucible runtime is spawned
402024-02-12T01:34:28.648ZINFOcrucible: The guest has requested activation with gen:1
412024-02-12T01:34:28.649ZINFOcrucible: 0a1ee051-0183-48af-8098-5c16f918bdc1 active request set session_id = 6e93a631-da6f-4a72-8667-3a4d9920c190
422024-02-12T01:34:28.649ZINFOcrucible: sending connect oneshot to client = downstairs client = 0 session_id = 6e93a631-da6f-4a72-8667-3a4d9920c190
432024-02-12T01:34:28.649ZINFOcrucible: client set_active_request while in New; waiting... = downstairs client = 0 session_id = 6e93a631-da6f-4a72-8667-3a4d9920c190
442024-02-12T01:34:28.649ZINFOcrucible: sending connect oneshot to client = downstairs client = 1 session_id = 6e93a631-da6f-4a72-8667-3a4d9920c190
452024-02-12T01:34:28.649ZINFOcrucible: client set_active_request while in New; waiting... = downstairs client = 1 session_id = 6e93a631-da6f-4a72-8667-3a4d9920c190
462024-02-12T01:34:28.649ZINFOcrucible: sending connect oneshot to client = downstairs client = 2 session_id = 6e93a631-da6f-4a72-8667-3a4d9920c190
472024-02-12T01:34:28.649ZINFOcrucible: client set_active_request while in New; waiting... = downstairs client = 2 session_id = 6e93a631-da6f-4a72-8667-3a4d9920c190
482024-02-12T01:34:28.649ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 6e93a631-da6f-4a72-8667-3a4d9920c190
492024-02-12T01:34:28.649ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 6e93a631-da6f-4a72-8667-3a4d9920c190
502024-02-12T01:34:28.649ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 6e93a631-da6f-4a72-8667-3a4d9920c190
512024-02-12T01:34:28.649ZINFOcrucible: ds_connection connected = io task client = 1 session_id = 6e93a631-da6f-4a72-8667-3a4d9920c190
522024-02-12T01:34:28.649ZINFOcrucible: ds_connection connected = io task client = 0 session_id = 6e93a631-da6f-4a72-8667-3a4d9920c190
532024-02-12T01:34:28.649ZINFOcrucible: ds_connection connected = io task client = 2 session_id = 6e93a631-da6f-4a72-8667-3a4d9920c190
542024-02-12T01:34:28.652ZINFOcrucible: ds_transition from New to WaitActive = downstairs client = 1 session_id = 6e93a631-da6f-4a72-8667-3a4d9920c190
552024-02-12T01:34:28.652ZINFOcrucible: [1] Transition from New to WaitActive = downstairs client = 1 session_id = 6e93a631-da6f-4a72-8667-3a4d9920c190
562024-02-12T01:34:28.652ZINFOcrucible: ds_transition from New to WaitActive = downstairs client = 2 session_id = 6e93a631-da6f-4a72-8667-3a4d9920c190
572024-02-12T01:34:28.652ZINFOcrucible: [2] Transition from New to WaitActive = downstairs client = 2 session_id = 6e93a631-da6f-4a72-8667-3a4d9920c190
582024-02-12T01:34:28.652ZINFOcrucible: ds_transition from New to WaitActive = downstairs client = 0 session_id = 6e93a631-da6f-4a72-8667-3a4d9920c190
592024-02-12T01:34:28.652ZINFOcrucible: [0] Transition from New to WaitActive = downstairs client = 0 session_id = 6e93a631-da6f-4a72-8667-3a4d9920c190
602024-02-12T01:34:28.653ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 6e93a631-da6f-4a72-8667-3a4d9920c190
612024-02-12T01:34:28.653ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 6e93a631-da6f-4a72-8667-3a4d9920c190
622024-02-12T01:34:28.653ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 6e93a631-da6f-4a72-8667-3a4d9920c190
632024-02-12T01:34:28.654ZINFOcrucible: ds_transition from WaitActive to WaitQuorum = downstairs client = 2 session_id = 6e93a631-da6f-4a72-8667-3a4d9920c190
642024-02-12T01:34:28.654ZINFOcrucible: [2] Transition from WaitActive to WaitQuorum = downstairs client = 2 session_id = 6e93a631-da6f-4a72-8667-3a4d9920c190
652024-02-12T01:34:28.654ZINFOcrucible: Waiting for true more clients to be ready session_id = 6e93a631-da6f-4a72-8667-3a4d9920c190
662024-02-12T01:34:28.655ZINFOcrucible: ds_transition from WaitActive to WaitQuorum = downstairs client = 0 session_id = 6e93a631-da6f-4a72-8667-3a4d9920c190
672024-02-12T01:34:28.655ZINFOcrucible: [0] Transition from WaitActive to WaitQuorum = downstairs client = 0 session_id = 6e93a631-da6f-4a72-8667-3a4d9920c190
682024-02-12T01:34:28.655ZINFOcrucible: Waiting for true more clients to be ready session_id = 6e93a631-da6f-4a72-8667-3a4d9920c190
692024-02-12T01:34:28.655ZINFOcrucible: ds_transition from WaitActive to WaitQuorum = downstairs client = 1 session_id = 6e93a631-da6f-4a72-8667-3a4d9920c190
702024-02-12T01:34:28.655ZINFOcrucible: [1] Transition from WaitActive to WaitQuorum = downstairs client = 1 session_id = 6e93a631-da6f-4a72-8667-3a4d9920c190
712024-02-12T01:34:28.655ZINFOcrucible: [0]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] = downstairs session_id = 6e93a631-da6f-4a72-8667-3a4d9920c190
722024-02-12T01:34:28.655ZINFOcrucible: [0]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] = downstairs session_id = 6e93a631-da6f-4a72-8667-3a4d9920c190
732024-02-12T01:34:28.655ZINFOcrucible: [0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = 6e93a631-da6f-4a72-8667-3a4d9920c190
742024-02-12T01:34:28.655ZINFOcrucible: [1]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] = downstairs session_id = 6e93a631-da6f-4a72-8667-3a4d9920c190
752024-02-12T01:34:28.655ZINFOcrucible: [1]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] = downstairs session_id = 6e93a631-da6f-4a72-8667-3a4d9920c190
762024-02-12T01:34:28.655ZINFOcrucible: [1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = 6e93a631-da6f-4a72-8667-3a4d9920c190
772024-02-12T01:34:28.655ZINFOcrucible: [2]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] = downstairs session_id = 6e93a631-da6f-4a72-8667-3a4d9920c190
782024-02-12T01:34:28.655ZINFOcrucible: [2]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] = downstairs session_id = 6e93a631-da6f-4a72-8667-3a4d9920c190
792024-02-12T01:34:28.655ZINFOcrucible: [2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = 6e93a631-da6f-4a72-8667-3a4d9920c190
802024-02-12T01:34:28.655ZINFOcrucible: Max found gen is 1 = downstairs session_id = 6e93a631-da6f-4a72-8667-3a4d9920c190
812024-02-12T01:34:28.655ZINFOcrucible: Generation requested: 1 >= found:1 = downstairs session_id = 6e93a631-da6f-4a72-8667-3a4d9920c190
822024-02-12T01:34:28.655ZINFOcrucible: Next flush: 1 = downstairs session_id = 6e93a631-da6f-4a72-8667-3a4d9920c190
832024-02-12T01:34:28.655ZINFOcrucible: All extents match = downstairs session_id = 6e93a631-da6f-4a72-8667-3a4d9920c190
842024-02-12T01:34:28.655ZINFOcrucible: No downstairs reconciliation required session_id = 6e93a631-da6f-4a72-8667-3a4d9920c190
852024-02-12T01:34:28.655ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 0 session_id = 6e93a631-da6f-4a72-8667-3a4d9920c190
862024-02-12T01:34:28.655ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 1 session_id = 6e93a631-da6f-4a72-8667-3a4d9920c190
872024-02-12T01:34:28.655ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 2 session_id = 6e93a631-da6f-4a72-8667-3a4d9920c190
882024-02-12T01:34:28.655ZINFOcrucible: All required reconciliation work is completed session_id = 6e93a631-da6f-4a72-8667-3a4d9920c190
89 {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30Activate successful
90 ,"Wait for a query_work_queue command to finish before sending IO
91 time":"2024-02-12T01:34:28.655608756Z","hostname":"ip-10-150-1-104.us-west-2.compute.internal","pid":1160,"session_id":"6e93a631-da6f-4a72-8667-3a4d9920c190"}
922024-02-12T01:34:28.655ZINFOcrucible: 0a1ee051-0183-48af-8098-5c16f918bdc1 is now active with session: 6e93a631-da6f-4a72-8667-3a4d9920c190 session_id = 6e93a631-da6f-4a72-8667-3a4d9920c190
932024-02-12T01:34:28.655ZINFOcrucible: Set Active after no reconciliation session_id = 6e93a631-da6f-4a72-8667-3a4d9920c190
942024-02-12T01:34:28.655ZINFOcrucible: The guest has finished waiting for activation with:1
95 Region: es:100 ec:50 bs:4096 ts:20480000 tb:5000 max_io:256 or 1048576
96 Fill test
97 Read and Verify all blocks (0..5000 range:false)
98 Wrote out file "alan"
99 CLIENT: Tests done. All submitted work has been ACK'd
100 ----------------------------------------------------------------
101 Crucible gen:1 GIO:true work queues: Upstairs:0 downstairs:43
102 States: Active Active Active
103 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
104 60 Acked 1059 Read 100 Done Done Done false
105 61 Acked 1060 Read 100 Done Done Done false
106 62 Acked 1061 Read 100 Done Done Done false
107 63 Acked 1062 Read 100 Done Done Done false
108 64 Acked 1063 Read 100 Done Done Done false
109 65 Acked 1064 Read 100 Done Done Done false
110 66 Acked 1065 Read 100 Done Done Done false
111 67 Acked 1066 Read 100 Done Done Done false
112 68 Acked 1067 Read 100 Done Done Done false
113 69 Acked 1068 Read 100 Done Done Done false
114 70 Acked 1069 Read 100 Done Done Done false
115 71 Acked 1070 Read 100 Done Done Done false
116 72 Acked 1071 Read 100 Done Done Done false
117 73 Acked 1072 Read 100 Done Done Done false
118 74 Acked 1073 Read 100 Done Done Done false
119 75 Acked 1074 Read 100 Done Done Done false
120 76 Acked 1075 Read 100 Done Done Done false
121 77 Acked 1076 Read 100 Done Done Done false
122 78 Acked 1077 Read 100 Done Done Done false
123 79 Acked 1078 Read 100 Done Done Done false
124 80 Acked 1079 Read 100 Done Done Done false
125 81 Acked 1080 Read 100 Done Done Done false
126 82 Acked 1081 Read 100 Done Done Done false
127 83 Acked 1082 Read 100 Done Done Done false
128 84 Acked 1083 Read 100 Done Done Done false
129 85 Acked 1084 Read 100 Done Done Done false
130 86 Acked 1085 Read 100 Done Done Done false
131 87 Acked 1086 Read 100 Done Done Done false
132 88 Acked 1087 Read 100 Done Done Done false
133 89 Acked 1088 Read 100 Done Done Done false
134 90 Acked 1089 Read 100 Done Done Done false
135 91 Acked 1090 Read 100 Done Done Done false
136 92 Acked 1091 Read 100 Done Done Done false
137 93 Acked 1092 Read 100 Done Done Done false
138 94 Acked 1093 Read 100 Done Done Done false
139 95 Acked 1094 Read 100 Done Done Done false
140 96 Acked 1095 Read 100 Done Done Done false
141 97 Acked 1096 Read 100 Done Done Done false
142 98 Acked 1097 Read 100 Done Done Done false
143 99 Acked 1098 Read 100 Done Done Done false
144 100 Acked 1099 Read 100 Done Done Done false
145 101 Acked 1100 Read 100 Done Done Done false
146 102 Acked 1101 Read 100 Done Done Done false
147 STATES DS:0 DS:1 DS:2 TOTAL
148 New 0 0 0 0
149 Sent 0 0 0 0
150 Done 43 43 43 129
151 Skipped 0 0 0 0
152 Error 0 0 0 0
153 Last Flush: 1058 1058 1058
154 Downstairs last five completed: 1058 1057 1056 1055 1054
155 Upstairs last five completed: 102 101 100 99 98
156 CLIENT: Up:0 ds:43 act:3
157 ----------------------------------------------------------------
158 Crucible gen:1 GIO:true work queues: Upstairs:0 downstairs:0
159 Downstairs last five completed: 1102 1101 1100 1099 1098
160 Upstairs last five completed: 103 102 101 100 99
161 CLIENT: Up:0 ds:0 act:3
162 CLIENT: All crucible jobs finished, exiting program
1632024-02-12T01:34:33.650ZWARNcrucible: Guest handle has been dropped
1642024-02-12T01:34:33.650ZWARNcrucible: client_stop_rx closed unexpectedly: RecvError(()) = io task client = 0 session_id = 6e93a631-da6f-4a72-8667-3a4d9920c190
1652024-02-12T01:34:33.651ZWARNcrucible: client task is sending Done(QueueClosed) = io task client = 0 session_id = 6e93a631-da6f-4a72-8667-3a4d9920c190
1662024-02-12T01:34:33.651ZWARNcrucible: client task could not reply to main task; shutting down? = io task client = 0 session_id = 6e93a631-da6f-4a72-8667-3a4d9920c190
1672024-02-12T01:34:33.651ZINFOcrucible: client task is exiting = io task client = 0 session_id = 6e93a631-da6f-4a72-8667-3a4d9920c190
168 Replay loop starts now February 12, 2024 at 01:34:33 AM UTC
1692024-02-12T01:34:33.677ZINFOcrucible: Upstairs starts
1702024-02-12T01:34:33.677ZINFOcrucible: Crucible Version: BuildInfo { version: "0.0.1", git_sha: "9f74bb5fe2be09de9f7a47f2db64d7dce836b79c", git_commit_timestamp: "2024-02-12T01:11:14.000000000Z", git_branch: "main", rustc_semver: "1.75.0", rustc_channel: "stable", rustc_host_triple: "x86_64-unknown-illumos", rustc_commit_sha: "82e1608dfa6e0b5569232559e3d385fea5a93112", cargo_triple: "x86_64-unknown-illumos", debug: true, opt_level: 0, }
1712024-02-12T01:34:33.677ZINFOcrucible: Upstairs <-> Downstairs Message Version: 5
172 {"msg":"Crucible 46d52ac5-ed69-45c0-b560-f7700be6fc03 has session id: e1773b3e-b36f-4080-8b13-c0830f0d072e","v":0,"name":"crucible","level":30Crucible runtime is spawned
173 ,"time":"2024-02-12T01:34:33.677944182Z","hostname":"ip-10-150-1-104.us-west-2.compute.internal","pid":1164,"session_id":"e1773b3e-b36f-4080-8b13-c0830f0d072e"}
1742024-02-12T01:34:33.677ZINFOcrucible: Upstairs opts: Upstairs UUID: 46d52ac5-ed69-45c0-b560-f7700be6fc03, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
1752024-02-12T01:34:33.678ZINFOcrucible: Crucible stats registered with UUID: 46d52ac5-ed69-45c0-b560-f7700be6fc03 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
1762024-02-12T01:34:33.678ZINFOcrucible: The guest has requested activation with gen:2
1772024-02-12T01:34:33.678ZINFOcrucible: 46d52ac5-ed69-45c0-b560-f7700be6fc03 active request set session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
1782024-02-12T01:34:33.678ZINFOcrucible: sending connect oneshot to client = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
1792024-02-12T01:34:33.678ZINFOcrucible: client set_active_request while in New; waiting... = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
1802024-02-12T01:34:33.678ZINFOcrucible: sending connect oneshot to client = downstairs client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
1812024-02-12T01:34:33.678ZINFOcrucible: client set_active_request while in New; waiting... = downstairs client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
1822024-02-12T01:34:33.678ZINFOcrucible: sending connect oneshot to client = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
1832024-02-12T01:34:33.678ZINFOcrucible: client set_active_request while in New; waiting... = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
1842024-02-12T01:34:33.678ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
1852024-02-12T01:34:33.678ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
1862024-02-12T01:34:33.678ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
1872024-02-12T01:34:33.678ZINFOcrucible: ds_connection connected = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
1882024-02-12T01:34:33.678ZINFOcrucible: ds_connection connected = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
1892024-02-12T01:34:33.678ZINFOcrucible: ds_connection connected = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
1902024-02-12T01:34:33.679ZINFOcrucible: ds_transition from New to WaitActive = downstairs client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
1912024-02-12T01:34:33.679ZINFOcrucible: [1] Transition from New to WaitActive = downstairs client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
1922024-02-12T01:34:33.679ZINFOcrucible: ds_transition from New to WaitActive = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
1932024-02-12T01:34:33.679ZINFOcrucible: [0] Transition from New to WaitActive = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
1942024-02-12T01:34:33.679ZINFOcrucible: ds_transition from New to WaitActive = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
1952024-02-12T01:34:33.679ZINFOcrucible: [2] Transition from New to WaitActive = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
1962024-02-12T01:34:33.679ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
1972024-02-12T01:34:33.680ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
1982024-02-12T01:34:33.680ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
1992024-02-12T01:34:33.680ZINFOcrucible: ds_transition from WaitActive to WaitQuorum = downstairs client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
2002024-02-12T01:34:33.680ZINFOcrucible: [1] Transition from WaitActive to WaitQuorum = downstairs client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
2012024-02-12T01:34:33.680ZINFOcrucible: Waiting for true more clients to be ready session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
2022024-02-12T01:34:33.680ZINFOcrucible: ds_transition from WaitActive to WaitQuorum = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
2032024-02-12T01:34:33.680ZINFOcrucible: [2] Transition from WaitActive to WaitQuorum = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
2042024-02-12T01:34:33.680ZINFOcrucible: Waiting for true more clients to be ready session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
2052024-02-12T01:34:33.680ZINFOcrucible: ds_transition from WaitActive to WaitQuorum = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
2062024-02-12T01:34:33.680ZINFOcrucible: [0] Transition from WaitActive to WaitQuorum = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
2072024-02-12T01:34:33.680ZINFOcrucible: [0]R flush_numbers[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
2082024-02-12T01:34:33.680ZINFOcrucible: [0]R generation[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
2092024-02-12T01:34:33.680ZINFOcrucible: [0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
2102024-02-12T01:34:33.680ZINFOcrucible: [1]R flush_numbers[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
2112024-02-12T01:34:33.680ZINFOcrucible: [1]R generation[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
2122024-02-12T01:34:33.680ZINFOcrucible: [1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
2132024-02-12T01:34:33.681ZINFOcrucible: [2]R flush_numbers[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
2142024-02-12T01:34:33.681ZINFOcrucible: [2]R generation[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
2152024-02-12T01:34:33.681ZINFOcrucible: [2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
2162024-02-12T01:34:33.681ZINFOcrucible: Max found gen is 2 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
2172024-02-12T01:34:33.681ZINFOcrucible: Generation requested: 2 >= found:2 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
218 {"msg":"Next flush: 2","v":0,"name":"crucible","level":30,"time":"2024-02-12T01:34:33.681085936Z","hostname":"ip-10-150-1-104.us-west-2.compute.internal","pid"Activate successful
219 :Wait for a query_work_queue command to finish before sending IO
220 1164,"":"downstairs","":"downstairs","session_id":"e1773b3e-b36f-4080-8b13-c0830f0d072e"}
2212024-02-12T01:34:33.681ZINFOcrucible: All extents match = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
2222024-02-12T01:34:33.681ZINFOcrucible: No downstairs reconciliation required session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
2232024-02-12T01:34:33.681ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
2242024-02-12T01:34:33.681ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
2252024-02-12T01:34:33.681ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
2262024-02-12T01:34:33.681ZINFOcrucible: All required reconciliation work is completed session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
2272024-02-12T01:34:33.681ZINFOcrucible: Set Downstairs and Upstairs active after reconciliation session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
2282024-02-12T01:34:33.681ZINFOcrucible: 46d52ac5-ed69-45c0-b560-f7700be6fc03 is now active with session: e1773b3e-b36f-4080-8b13-c0830f0d072e session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
2292024-02-12T01:34:33.681ZINFOcrucible: Set Active after no reconciliation session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
2302024-02-12T01:34:33.681ZINFOcrucible: The guest has finished waiting for activation with:2
231 Region: es:100 ec:50 bs:4096 ts:20480000 tb:5000 max_io:256 or 1048576
232 Loading write count information from file "alan"
233 Feb 12 01:34:33.731 INFO accepted connection, remote_addr: 127.0.0.1:60456, local_addr: 127.0.0.1:9998
234 Main task has work to do, go find it
235 got dsc Stop(2)
236 stop 2
237 Feb 12 01:34:33.732 INFO request completed[8830] Got stop action so:false kr:false
238 , latency_us: 411, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 60493485-3ee1-4802-8e13-dbea5c5d3b1f, remote_addr: 127.0.0.1:60456, local_addr: 127.0.0.1:9998
2392024-02-12T01:34:33.733ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
2402024-02-12T01:34:33.733ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
2412024-02-12T01:34:33.733ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
2422024-02-12T01:34:33.733ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
2432024-02-12T01:34:33.733ZINFOcrucible: [2] client re-new 0 jobs since flush 0 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
2442024-02-12T01:34:33.733ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
245 Feb 12 01:34:33.734 INFO request completed, latency_us: 187, response_code: 200, uri: /state/cid/2, method: GET, req_id: 0b78eded-89f6-4e8f-98bf-613fb8f3fc63, remote_addr: 127.0.0.1:60456, local_addr: 127.0.0.1:9998
246 [8830] Exited with: ExitStatus(unix_wait_status(9))
247 [8830][2] reports Exit
2482024-02-12T01:34:34.734ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
2492024-02-12T01:34:34.734ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
2502024-02-12T01:34:34.734ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
2512024-02-12T01:34:34.734ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
2522024-02-12T01:34:34.734ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
2532024-02-12T01:34:34.734ZINFOcrucible: [2] client re-new 0 jobs since flush 0 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
2542024-02-12T01:34:34.734ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
2552024-02-12T01:34:35.735ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
2562024-02-12T01:34:35.735ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
2572024-02-12T01:34:35.735ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
2582024-02-12T01:34:35.735ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
2592024-02-12T01:34:35.735ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
2602024-02-12T01:34:35.735ZINFOcrucible: [2] client re-new 0 jobs since flush 0 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
2612024-02-12T01:34:35.735ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
2622024-02-12T01:34:36.736ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
2632024-02-12T01:34:36.736ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
2642024-02-12T01:34:36.736ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
2652024-02-12T01:34:36.736ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
2662024-02-12T01:34:36.736ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
2672024-02-12T01:34:36.736ZINFOcrucible: [2] client re-new 0 jobs since flush 0 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
2682024-02-12T01:34:36.736ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
269 Feb 12 01:34:37.736 INFO request completed, latency_us: 214, response_code: 200, uri: /state/cid/2, method: GET, req_id: e166370b-7008-427d-92aa-82a3923b7f70, remote_addr: 127.0.0.1:60456, local_addr: 127.0.0.1:9998
270 001/300 Read block 3960 len 36864
2712024-02-12T01:34:37.737ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
2722024-02-12T01:34:37.737ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
2732024-02-12T01:34:37.737ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
2742024-02-12T01:34:37.737ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
2752024-02-12T01:34:37.737ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
2762024-02-12T01:34:37.737ZINFOcrucible: [2] client re-new 1 jobs since flush 0 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
2772024-02-12T01:34:37.737ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
278 002/300 Write block 1118 len 28672 data: 2 2 2 2 2 2 2
279 003/300 Write block 4690 len 4096 data: 2
280 004/300 Read block 3226 len 8192
281 005/300 Read block 1091 len 8192
282 006/300 Write block 1607 len 36864 data: 2 2 2 2 2 2 2 2 2
283 007/300 Read block 4323 len 40960
284 008/300 Write block 1785 len 40960 data: 2 2 2 2 2 2 2 2 2 2
285 009/300 Read block 1933 len 20480
286 010/300 Flush
287 011/300 Write block 2853 len 28672 data: 2 2 2 2 2 2 2
288 012/300 Write block 4966 len 20480 data: 2 2 2 2 2
289 013/300 Flush
290 014/300 Read block 402 len 20480
291 015/300 Write block 2255 len 36864 data: 2 2 2 2 2 2 2 2 2
292 016/300 Read block 723 len 24576
293 017/300 Read block 3020 len 28672
294 018/300 Write block 2336 len 4096 data: 2
295 019/300 Write block 2930 len 32768 data: 2 2 2 2 2 2 2 2
296 020/300 Read block 4190 len 32768
297 021/300 Flush
298 022/300 Write block 2424 len 16384 data: 2 2 2 2
299 023/300 Write block 2625 len 4096 data: 2
300 024/300 Write block 3662 len 24576 data: 2 2 2 2 2 2
301 025/300 Write block 3181 len 32768 data: 2 2 2 2 2 2 2 2
302 026/300 Read block 877 len 12288
303 027/300 Read block 943 len 28672
304 028/300 Write block 2752 len 24576 data: 2 2 2 2 2 2
305 029/300 Write block 144 len 12288 data: 2 2 2
306 030/300 Read block 948 len 20480
307 031/300 Read block 423 len 20480
308 032/300 Read block 155 len 20480
309 033/300 Write block 525 len 24576 data: 2 2 2 2 2 2
310 034/300 Write block 2763 len 4096 data: 2
311 035/300 Read block 3226 len 16384
312 036/300 Write block 1976 len 20480 data: 2 2 2 2 2
313 037/300 Read block 2052 len 20480
314 038/300 Read block 476 len 40960
315 039/300 Read block 4534 len 20480
316 040/300 Write block 1204 len 8192 data: 2 2
317 041/300 Read block 57 len 24576
318 042/300 Read block 2299 len 16384
319 043/300 Read block 3544 len 4096
320 044/300 Flush
321 045/300 Write block 3629 len 20480 data: 2 2 2 2 2
322 046/300 Read block 1931 len 4096
323 047/300 Read block 2972 len 36864
324 048/300 Read block 1670 len 12288
325 049/300 Write block 1888 len 8192 data: 2 2
326 050/300 Flush
327 051/300 Write block 464 len 8192 data: 2 2
328 052/300 Read block 286 len 4096
329 053/300 Flush
330 054/300 Write block 753 len 40960 data: 2 2 2 2 2 2 2 2 2 2
331 055/300 Read block 1847 len 4096
332 056/300 Read block 1108 len 28672
333 057/300 Write block 1716 len 4096 data: 2
334 058/300 Read block 2657 len 36864
335 059/300 Write block 3428 len 40960 data: 2 2 2 2 2 2 2 2 2 2
336 060/300 Read block 2601 len 12288
337 061/300 Read block 3759 len 8192
338 062/300 Write block 2954 len 36864 data: 2 2 2 2 2 2 2 2 2
339 063/300 Flush
340 064/300 Flush
341 065/300 Write block 3804 len 36864 data: 2 2 2 2 2 2 2 2 2
342 066/300 Read block 4149 len 24576
343 067/300 Flush
344 068/300 Flush
345 069/300 Read block 4710 len 28672
346 070/300 Read block 654 len 4096
347 071/300 Read block 3357 len 24576
348 072/300 Write block 4669 len 40960 data: 2 2 2 2 2 2 2 2 2 2
349 073/300 Read block 4330 len 32768
350 074/300 Write block 2333 len 40960 data: 2 2 2 3 2 2 2 2 2 2
351 075/300 Read block 501 len 40960
352 076/300 Write block 4627 len 36864 data: 2 2 2 2 2 2 2 2 2
353 077/300 Write block 263 len 36864 data: 2 2 2 2 2 2 2 2 2
354 078/300 Write block 2785 len 32768 data: 2 2 2 2 2 2 2 2
355 079/300 Read block 3858 len 28672
356 080/300 Write block 4201 len 28672 data: 2 2 2 2 2 2 2
357 081/300 Read block 3208 len 8192
358 082/300 Read block 8 len 40960
359 083/300 Write block 4959 len 12288 data: 2 2 2
360 084/300 Read block 997 len 28672
361 085/300 Write block 1967 len 36864 data: 2 2 2 2 2 2 2 2 2
362 086/300 Read block 2075 len 4096
363 087/300 Write block 4909 len 16384 data: 2 2 2 2
364 088/300 Read block 4745 len 4096
365 089/300 Read block 3723 len 20480
366 090/300 Read block 2558 len 32768
367 091/300 Flush
368 092/300 Flush
369 093/300 Read block 4157 len 12288
370 094/300 Write block 928 len 12288 data: 2 2 2
371 095/300 Write block 4588 len 4096 data: 2
372 096/300 Read block 4840 len 12288
373 097/300 Write block 4319 len 12288 data: 2 2 2
374 098/300 Write block 4021 len 8192 data: 2 2
375 099/300 Read block 3863 len 16384
376 100/300 Write block 1044 len 36864 data: 2 2 2 2 2 2 2 2 2
377 101/300 Read block 1202 len 32768
378 102/300 Write block 576 len 4096 data: 2
379 103/300 Write block 1104 len 8192 data: 2 2
380 104/300 Write block 2706 len 32768 data: 2 2 2 2 2 2 2 2
381 105/300 Read block 256 len 32768
382 106/300 Read block 1373 len 32768
383 107/300 Write block 306 len 20480 data: 2 2 2 2 2
384 108/300 Write block 4223 len 40960 data: 2 2 2 2 2 2 2 2 2 2
385 109/300 Write block 730 len 16384 data: 2 2 2 2
386 110/300 Write block 1558 len 36864 data: 2 2 2 2 2 2 2 2 2
387 111/300 Read block 467 len 8192
388 112/300 Write block 4482 len 20480 data: 2 2 2 2 2
389 113/300 Write block 2171 len 8192 data: 2 2
390 114/300 Write block 1107 len 4096 data: 2
391 115/300 Read block 4708 len 28672
392 116/300 Read block 4544 len 4096
393 117/300 Write block 1166 len 32768 data: 2 2 2 2 2 2 2 2
394 118/300 Read block 644 len 40960
395 119/300 Write block 4415 len 16384 data: 2 2 2 2
396 120/300 Read block 717 len 24576
397 121/300 Write block 1678 len 20480 data: 2 2 2 2 2
398 122/300 Read block 365 len 24576
399 123/300 Write block 1822 len 28672 data: 2 2 2 2 2 2 2
400 124/300 Read block 2785 len 32768
401 125/300 Write block 185 len 8192 data: 2 2
402 126/300 Write block 406 len 24576 data: 2 2 2 2 2 2
403 127/300 Write block 3325 len 4096 data: 2
404 128/300 Read block 3928 len 32768
405 129/300 Write block 1344 len 4096 data: 2
406 130/300 Flush
407 131/300 Write block 3093 len 12288 data: 2 2 2
408 132/300 Write block 1327 len 4096 data: 2
409 133/300 Write block 3566 len 40960 data: 2 2 2 2 2 2 2 2 2 2
410 134/300 Read block 1584 len 12288
411 135/300 Read block 4304 len 28672
412 136/300 Flush
413 137/300 Write block 3962 len 28672 data: 2 2 2 2 2 2 2
414 138/300 Flush
415 139/300 Read block 2511 len 24576
416 140/300 Read block 2390 len 28672
417 141/300 Read block 3751 len 16384
418 142/300 Write block 2421 len 28672 data: 2 2 2 3 3 3 3
419 143/300 Flush
420 144/300 Read block 4029 len 36864
421 145/300 Read block 3122 len 32768
422 146/300 Flush
423 147/300 Write block 1962 len 24576 data: 2 2 2 2 2 3
424 148/300 Write block 28 len 16384 data: 2 2 2 2
425 149/300 Read block 1227 len 20480
426 150/300 Read block 2878 len 20480
427 151/300 Read block 540 len 40960
428 152/300 Read block 1972 len 36864
429 153/300 Flush
430 154/300 Write block 401 len 16384 data: 2 2 2 2
431 155/300 Read block 1909 len 20480
432 156/300 Write block 1337 len 8192 data: 2 2
433 157/300 Write block 2697 len 28672 data: 2 2 2 2 2 2 2
434 158/300 Read block 3310 len 20480
435 159/300 Read block 994 len 36864
436 160/300 Write block 4370 len 12288 data: 2 2 2
437 161/300 Read block 4688 len 36864
438 162/300 Write block 3603 len 28672 data: 2 2 2 2 2 2 2
439 163/300 Write block 3403 len 16384 data: 2 2 2 2
440 164/300 Write block 2448 len 8192 data: 2 2
441 165/300 Write block 2656 len 24576 data: 2 2 2 2 2 2
442 166/300 Read block 3568 len 4096
443 167/300 Read block 2364 len 24576
444 168/300 Read block 1458 len 32768
445 169/300 Read block 873 len 36864
446 170/300 Write block 4163 len 32768 data: 2 2 2 2 2 2 2 2
447 171/300 Write block 124 len 16384 data: 2 2 2 2
448 172/300 Read block 2576 len 16384
449 173/300 Read block 1770 len 32768
450 174/300 Read block 3088 len 20480
451 175/300 Read block 2313 len 12288
452 176/300 Write block 1856 len 32768 data: 2 2 2 2 2 2 2 2
453 177/300 Read block 4596 len 40960
454 178/300 Write block 2297 len 8192 data: 2 2
455 179/300 Write block 2095 len 28672 data: 2 2 2 2 2 2 2
456 180/300 Write block 2300 len 24576 data: 2 2 2 2 2 2
457 181/300 Read block 4047 len 28672
458 182/300 Write block 2339 len 28672 data: 3 3 3 3 2 2 2
459 183/300 Read block 4602 len 8192
460 184/300 Read block 1071 len 28672
461 185/300 Write block 3681 len 40960 data: 2 2 2 2 2 2 2 2 2 2
462 186/300 Read block 4952 len 28672
463 187/300 Write block 2349 len 36864 data: 2 2 2 2 2 2 2 2 2
464 188/300 Read block 1639 len 40960
465 189/300 Write block 4718 len 24576 data: 2 2 2 2 2 2
466 190/300 Read block 645 len 16384
467 191/300 Write block 1916 len 20480 data: 2 2 2 2 2
468 192/300 Read block 2361 len 32768
469 193/300 Write block 2311 len 28672 data: 2 2 2 2 2 2 2
470 194/300 Read block 1395 len 12288
471 195/300 Read block 1895 len 24576
472 196/300 Write block 4596 len 24576 data: 2 2 2 2 2 2
473 197/300 Write block 2252 len 24576 data: 2 2 2 3 3 3
474 198/300 Write block 99 len 36864 data: 2 2 2 2 2 2 2 2 2
475 199/300 Write block 4425 len 16384 data: 2 2 2 2
476 200/300 Write block 4761 len 36864 data: 2 2 2 2 2 2 2 2 2
477 201/300 Read block 3322 len 40960
478 202/300 Write block 4794 len 20480 data: 2 2 2 2 2
479 203/300 Read block 1138 len 4096
480 204/300 Flush
481 205/300 Write block 2208 len 8192 data: 2 2
482 206/300 Write block 2479 len 4096 data: 2
483 207/300 Flush
484 208/300 Read block 3985 len 16384
485 209/300 Write block 3224 len 32768 data: 2 2 2 2 2 2 2 2
486 210/300 Read block 918 len 32768
487 211/300 Read block 2042 len 4096
488 212/300 Write block 2823 len 8192 data: 2 2
489 213/300 Flush
490 214/300 Flush
491 215/300 Write block 1140 len 36864 data: 2 2 2 2 2 2 2 2 2
492 216/300 Write block 727 len 24576 data: 2 2 2 3 3 3
493 217/300 Flush
494 218/300 Write block 2158 len 12288 data: 2 2 2
495 219/300 Read block 3276 len 36864
496 220/300 Read block 2680 len 40960
497 221/300 Read block 2012 len 20480
498 222/300 Read block 2037 len 20480
499 223/300 Flush
500 224/300 Read block 741 len 24576
501 225/300 Write block 1703 len 24576 data: 2 2 2 2 2 2
502 226/300 Read block 3146 len 28672
503 227/300 Read block 1939 len 32768
504 228/300 Read block 289 len 4096
505 229/300 Read block 581 len 28672
506 230/300 Write block 162 len 24576 data: 2 2 2 2 2 2
507 231/300 Read block 878 len 28672
508 232/300 Write block 867 len 4096 data: 2
509 233/300 Flush
510 234/300 Write block 968 len 24576 data: 2 2 2 2 2 2
511 235/300 Read block 2628 len 4096
512 236/300 Write block 3188 len 36864 data: 3 2 2 2 2 2 2 2 2
513 237/300 Write block 2482 len 40960 data: 2 2 2 2 2 2 2 2 2 2
514 238/300 Write block 2901 len 20480 data: 2 2 2 2 2
515 239/300 Read block 2164 len 12288
516 240/300 Write block 1997 len 8192 data: 2 2
517 241/300 Write block 2146 len 4096 data: 2
518 242/300 Read block 1509 len 4096
519 243/300 Write block 2766 len 16384 data: 2 2 2 2
520 244/300 Write block 1329 len 28672 data: 2 2 2 2 2 2 2
521 245/300 Write block 4356 len 40960 data: 2 2 2 2 2 2 2 2 2 2
522 246/300 Write block 4157 len 4096 data: 2
523 247/300 Write block 2349 len 24576 data: 3 3 3 3 3 3
524 248/300 Read block 4587 len 24576
525 249/300 Write block 2757 len 24576 data: 3 2 2 2 2 2
526 250/300 Write block 1827 len 32768 data: 3 3 2 2 2 2 2 2
527 251/300 Read block 525 len 8192
528 252/300 Flush
529 253/300 Read block 2116 len 12288
530 254/300 Read block 4820 len 16384
531 255/300 Write block 2803 len 28672 data: 2 2 2 2 2 2 2
532 256/300 Read block 3638 len 32768
533 257/300 Write block 2044 len 36864 data: 2 2 2 2 2 2 2 2 2
534 258/300 Flush
535 259/300 Read block 4397 len 40960
536 260/300 Read block 2182 len 32768
537 261/300 Write block 3080 len 16384 data: 2 2 2 2
538 262/300 Read block 476 len 32768
539 263/300 Write block 4054 len 40960 data: 2 2 2 2 2 2 2 2 2 2
540 264/300 Read block 1164 len 40960
541 265/300 Read block 4958 len 8192
542 266/300 Read block 4358 len 36864
543 267/300 Read block 3423 len 40960
544 268/300 Read block 2966 len 20480
545 269/300 Read block 3832 len 20480
546 270/300 Write block 4353 len 16384 data: 2 2 2 3
547 271/300 Read block 2674 len 20480
548 272/300 Read block 1508 len 40960
549 273/300 Read block 2612 len 12288
550 274/300 Write block 4259 len 12288 data: 2 2 2
551 275/300 Write block 4648 len 28672 data: 2 2 2 2 2 2 2
552 276/300 Read block 3733 len 16384
553 277/300 Write block 2350 len 12288 data: 4 4 4
554 278/300 Read block 4647 len 8192
555 279/300 Flush
556 280/300 Write block 335 len 8192 data: 2 2
557 281/300 Flush
558 282/300 Read block 3491 len 32768
559 283/300 Read block 3511 len 36864
560 284/300 Write block 4956 len 32768 data: 2 2 2 3 3 3 2 2
561 285/300 Write block 4639 len 28672 data: 2 2 2 2 2 2 2
562 286/300 Write block 4766 len 32768 data: 3 3 3 3 2 2 2 2
563 287/300 Read block 2115 len 36864
564 288/300 Flush
565 289/300 Flush
566 290/300 Write block 4019 len 20480 data: 2 2 3 3 2
567 291/300 Read block 4280 len 4096
568 292/300 Write block 2199 len 28672 data: 2 2 2 2 2 2 2
569 293/300 Flush
570 294/300 Read block 1757 len 40960
571 295/300 Write block 2881 len 20480 data: 2 2 2 2 2
572 296/300 Read block 4674 len 28672
573 297/300 Write block 534 len 32768 data: 2 2 2 2 2 2 2 2
574 298/300 Read block 298 len 8192
575 299/300 Write block 4940 len 24576 data: 2 2 2 2 2 2
576 300/300 Read block 3246 len 4096
577 Main task has work to do, go find it
578 got dsc Start(2)
579 start 2
580 Feb 12 01:34:38.093 INFO request completed, latency_us: 211, response_code: 204, uri: /start/cid/2, method: POST, req_id: 13980f57-d8ee-424d-9072-dc9cfb42b742, remote_addr: 127.0.0.1:60456, local_addr: 127.0.0.1:9998
581 Replay: started 2, returned:Ok(())
582 ----------------------------------------------------------------
583 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:300
584 States: Active Active Offline
585 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
586 1 Acked 1000 Read 9 Done Done New true
587 2 Acked 1001 Write 7 Done Done New false
588 3 Acked 1002 Write 1 Done Done New false
589 4 Acked 1003 Read 2 Done Done New false
590 5 Acked 1004 Read 2 Done Done New false
591 6 Acked 1005 Write 9 Done Done New false
592 7 Acked 1006 Read 10 Done Done New false
593 8 Acked 1007 Write 10 Done Done New false
594 9 Acked 1008 Read 5 Done Done New false
595 10 Acked 1009 Flush 0 Done Done New false
596 11 Acked 1010 Write 7 Done Done New false
597 12 Acked 1011 Write 5 Done Done New false
598 13 Acked 1012 Flush 0 Done Done New false
599 14 Acked 1013 Read 5 Done Done New false
600 15 Acked 1014 Write 9 Done Done New false
601 16 Acked 1015 Read 6 Done Done New false
602 17 Acked 1016 Read 7 Done Done New false
603 18 Acked 1017 Write 1 Done Done New false
604 19 Acked 1018 Write 8 Done Done New false
605 20 Acked 1019 Read 8 Done Done New false
606 21 Acked 1020 Flush 0 Done Done New false
607 22 Acked 1021 Write 4 Done Done New false
608 23 Acked 1022 Write 1 Done Done New false
609 24 Acked 1023 Write 6 Done Done New false
610 25 Acked 1024 Write 8 Done Done New false
611 26 Acked 1025 Read 3 Done Done New false
612 27 Acked 1026 Read 7 Done Done New false
613 28 Acked 1027 Write 6 Done Done New false
614 29 Acked 1028 Write 3 Done Done New false
615 30 Acked 1029 Read 5 Done Done New false
616 31 Acked 1030 Read 5 Done Done New false
617 32 Acked 1031 Read 5 Done Done New false
618 33 Acked 1032 Write 6 Done Done New false
619 34 Acked 1033 Write 1 Done Done New false
620 35 Acked 1034 Read 4 Done Done New false
621 36 Acked 1035 Write 5 Done Done New false
622 37 Acked 1036 Read 5 Done Done New false
623 38 Acked 1037 Read 10 Done Done New false
624 39 Acked 1038 Read 5 Done Done New false
625 40 Acked 1039 Write 2 Done Done New false
626 41 Acked 1040 Read 6 Done Done New false
627 42 Acked 1041 Read 4 Done Done New false
628 43 Acked 1042 Read 1 Done Done New false
629 44 Acked 1043 Flush 0 Done Done New false
630 45 Acked 1044 Write 5 Done Done New false
631 46 Acked 1045 Read 1 Done Done New false
632 47 Acked 1046 Read 9 Done Done New false
633 48 Acked 1047 Read 3 Done Done New false
634 49 Acked 1048 Write 2 Done Done New false
635 50 Acked 1049 Flush 0 Done Done New false
636 51 Acked 1050 Write 2 Done Done New false
637 52 Acked 1051 Read 1 Done Done New false
638 53 Acked 1052 Flush 0 Done Done New false
639 54 Acked 1053 Write 10 Done Done New false
640 55 Acked 1054 Read 1 Done Done New false
641 56 Acked 1055 Read 7 Done Done New false
642 57 Acked 1056 Write 1 Done Done New false
643 58 Acked 1057 Read 9 Done Done New false
644 59 Acked 1058 Write 10 Done Done New false
645 60 Acked 1059 Read 3 Done Done New false
646 61 Acked 1060 Read 2 Done Done New false
647 62 Acked 1061 Write 9 Done Done New false
648 63 Acked 1062 Flush 0 Done Done New false
649 64 Acked 1063 Flush 0 Done Done New false
650 65 Acked 1064 Write 9 Done Done New false
651 66 Acked 1065 Read 6 Done Done New false
652 67 Acked 1066 Flush 0 Done Done New false
653 68 Acked 1067 Flush 0 Done Done New false
654 69 Acked 1068 Read 7 Done Done New false
655 70 Acked 1069 Read 1 Done Done New false
656 71 Acked 1070 Read 6 Done Done New false
657 72 Acked 1071 Write 10 Done Done New false
658 73 Acked 1072 Read 8 Done Done New false
659 74 Acked 1073 Write 10 Done Done New false
660 75 Acked 1074 Read 10 Done Done New false
661 76 Acked 1075 Write 9 Done Done New false
662 77 Acked 1076 Write 9 Done Done New false
663 78 Acked 1077 Write 8 Done Done New false
664 79 Acked 1078 Read 7 Done Done New false
665 80 Acked 1079 Write 7 Done Done New false
666 81 Acked 1080 Read 2 Done Done New false
667 82 Acked 1081 Read 10 Done Done New false
668 83 Acked 1082 Write 3 Done Done New false
669 84 Acked 1083 Read 7 Done Done New false
670 85 Acked 1084 Write 9 Done Done New false
671 86 Acked 1085 Read 1 Done Done New false
672 87 Acked 1086 Write 4 Done Done New false
673 88 Acked 1087 Read 1 Done Done New false
674 89 Acked 1088 Read 5 Done Done New false
675 90 Acked 1089 Read 8 Done Done New false
676 91 Acked 1090 Flush 0 Done Done New false
677 92 Acked 1091 Flush 0 Done Done New false
678 93 Acked 1092 Read 3 Done Done New false
679 94 Acked 1093 Write 3 Done Done New false
680 95 Acked 1094 Write 1 Done Done New false
681 96 Acked 1095 Read 3 Done Done New false
682 97 Acked 1096 Write 3 Done Done New false
683 98 Acked 1097 Write 2 Done Done New false
684 99 Acked 1098 Read 4 Done Done New false
685 100 Acked 1099 Write 9 Done Done New false
686 101 Acked 1100 Read 8 Done Done New false
687 102 Acked 1101 Write 1 Done Done New false
688 103 Acked 1102 Write 2 Done Done New false
689 104 Acked 1103 Write 8 Done Done New false
690 105 Acked 1104 Read 8 Done Done New false
691 106 Acked 1105 Read 8 Done Done New false
692 107 Acked 1106 Write 5 Done Done New false
693 108 Acked 1107 Write 10 Done Done New false
694 109 Acked 1108 Write 4 Done Done New false
695 110 Acked 1109 Write 9 Done Done New false
696 111 Acked 1110 Read 2 Done Done New false
697 112 Acked 1111 Write 5 Done Done New false
698 113 Acked 1112 Write 2 Done Done New false
699 114 Acked 1113 Write 1 Done Done New false
700 115 Acked 1114 Read 7 Done Done New false
701 116 Acked 1115 Read 1 Done Done New false
702 117 Acked 1116 Write 8 Done Done New false
703 118 Acked 1117 Read 10 Done Done New false
704 119 Acked 1118 Write 4 Done Done New false
705 120 Acked 1119 Read 6 Done Done New false
706 121 Acked 1120 Write 5 Done Done New false
707 122 Acked 1121 Read 6 Done Done New false
708 123 Acked 1122 Write 7 Done Done New false
709 124 Acked 1123 Read 8 Done Done New false
710 125 Acked 1124 Write 2 Done Done New false
711 126 Acked 1125 Write 6 Done Done New false
712 127 Acked 1126 Write 1 Done Done New false
713 128 Acked 1127 Read 8 Done Done New false
714 129 Acked 1128 Write 1 Done Done New false
715 130 Acked 1129 Flush 0 Done Done New false
716 131 Acked 1130 Write 3 Done Done New false
717 132 Acked 1131 Write 1 Done Done New false
718 133 Acked 1132 Write 10 Done Done New false
719 134 Acked 1133 Read 3 Done Done New false
720 135 Acked 1134 Read 7 Done Done New false
721 136 Acked 1135 Flush 0 Done Done New false
722 137 Acked 1136 Write 7 Done Done New false
723 138 Acked 1137 Flush 0 Done Done New false
724 139 Acked 1138 Read 6 Done Done New false
725 140 Acked 1139 Read 7 Done Done New false
726 141 Acked 1140 Read 4 Done Done New false
727 142 Acked 1141 Write 7 Done Done New false
728 143 Acked 1142 Flush 0 Done Done New false
729 144 Acked 1143 Read 9 Done Done New false
730 145 Acked 1144 Read 8 Done Done New false
731 146 Acked 1145 Flush 0 Done Done New false
732 147 Acked 1146 Write 6 Done Done New false
733 148 Acked 1147 Write 4 Done Done New false
734 149 Acked 1148 Read 5 Done Done New false
735 150 Acked 1149 Read 5 Done Done New false
736 151 Acked 1150 Read 10 Done Done New false
737 152 Acked 1151 Read 9 Done Done New false
738 153 Acked 1152 Flush 0 Done Done New false
739 154 Acked 1153 Write 4 Done Done New false
740 155 Acked 1154 Read 5 Done Done New false
741 156 Acked 1155 Write 2 Done Done New false
742 157 Acked 1156 Write 7 Done Done New false
743 158 Acked 1157 Read 5 Done Done New false
744 159 Acked 1158 Read 9 Done Done New false
745 160 Acked 1159 Write 3 Done Done New false
746 161 Acked 1160 Read 9 Done Done New false
747 162 Acked 1161 Write 7 Done Done New false
748 163 Acked 1162 Write 4 Done Done New false
749 164 Acked 1163 Write 2 Done Done New false
750 165 Acked 1164 Write 6 Done Done New false
751 166 Acked 1165 Read 1 Done Done New false
752 167 Acked 1166 Read 6 Done Done New false
753 168 Acked 1167 Read 8 Done Done New false
754 169 Acked 1168 Read 9 Done Done New false
755 170 Acked 1169 Write 8 Done Done New false
756 171 Acked 1170 Write 4 Done Done New false
757 172 Acked 1171 Read 4 Done Done New false
758 173 Acked 1172 Read 8 Done Done New false
759 174 Acked 1173 Read 5 Done Done New false
760 175 Acked 1174 Read 3 Done Done New false
761 176 Acked 1175 Write 8 Done Done New false
762 177 Acked 1176 Read 10 Done Done New false
763 178 Acked 1177 Write 2 Done Done New false
764 179 Acked 1178 Write 7 Done Done New false
765 180 Acked 1179 Write 6 Done Done New false
766 181 Acked 1180 Read 7 Done Done New false
767 182 Acked 1181 Write 7 Done Done New false
768 183 Acked 1182 Read 2 Done Done New false
769 184 Acked 1183 Read 7 Done Done New false
770 185 Acked 1184 Write 10 Done Done New false
771 186 Acked 1185 Read 7 Done Done New false
772 187 Acked 1186 Write 9 Done Done New false
773 188 Acked 1187 Read 10 Done Done New false
774 189 Acked 1188 Write 6 Done Done New false
775 190 Acked 1189 Read 4 Done Done New false
776 191 Acked 1190 Write 5 Done Done New false
777 192 Acked 1191 Read 8 Done Done New false
778 193 Acked 1192 Write 7 Done Done New false
779 194 Acked 1193 Read 3 Done Done New false
780 195 Acked 1194 Read 6 Done Done New false
781 196 Acked 1195 Write 6 Done Done New false
782 197 Acked 1196 Write 6 Done Done New false
783 198 Acked 1197 Write 9 Done Done New false
784 199 Acked 1198 Write 4 Done Done New false
785 200 Acked 1199 Write 9 Done Done New false
786 201 Acked 1200 Read 10 Done Done New false
787 202 Acked 1201 Write 5 Done Done New false
788 203 Acked 1202 Read 1 Done Done New false
789 204 Acked 1203 Flush 0 Done Done New false
790 205 Acked 1204 Write 2 Done Done New false
791 206 Acked 1205 Write 1 Done Done New false
792 207 Acked 1206 Flush 0 Done Done New false
793 208 Acked 1207 Read 4 Done Done New false
794 209 Acked 1208 Write 8 Done Done New false
795 210 Acked 1209 Read 8 Done Done New false
796 211 Acked 1210 Read 1 Done Done New false
797 212 Acked 1211 Write 2 Done Done New false
798 213 Acked 1212 Flush 0 Done Done New false
799 214 Acked 1213 Flush 0 Done Done New false
800 215 Acked 1214 Write 9 Done Done New false
801 216 Acked 1215 Write 6 Done Done New false
802 217 Acked 1216 Flush 0 Done Done New false
803 218 Acked 1217 Write 3 Done Done New false
804 219 Acked 1218 Read 9 Done Done New false
805 220 Acked 1219 Read 10 Done Done New false
806 221 Acked 1220 Read 5 Done Done New false
807 222 Acked 1221 Read 5 Done Done New false
808 223 Acked 1222 Flush 0 Done Done New false
809 224 Acked 1223 Read 6 Done Done New false
810 225 Acked 1224 Write 6 Done Done New false
811 226 Acked 1225 Read 7 Done Done New false
812 227 Acked 1226 Read 8 Done Done New false
813 228 Acked 1227 Read 1 Done Done New false
814 229 Acked 1228 Read 7 Done Done New false
815 230 Acked 1229 Write 6 Done Done New false
816 231 Acked 1230 Read 7 Done Done New false
817 232 Acked 1231 Write 1 Done Done New false
818 233 Acked 1232 Flush 0 Done Done New false
819 234 Acked 1233 Write 6 Done Done New false
820 235 Acked 1234 Read 1 Done Done New false
821 236 Acked 1235 Write 9 Done Done New false
822 237 Acked 1236 Write 10 Done Done New false
823 238 Acked 1237 Write 5 Done Done New false
824 239 Acked 1238 Read 3 Done Done New false
825 240 Acked 1239 Write 2 Done Done New false
826 241 Acked 1240 Write 1 Done Done New false
827 242 Acked 1241 Read 1 Done Done New false
828 243 Acked 1242 Write 4 Done Done New false
829 244 Acked 1243 Write 7 Done Done New false
830 245 Acked 1244 Write 10 Done Done New false
831 246 Acked 1245 Write 1 Done Done New false
832 247 Acked 1246 Write 6 Done Done New false
833 248 Acked 1247 Read 6 Done Done New false
834 249 Acked 1248 Write 6 Done Done New false
835 250 Acked 1249 Write 8 Done Done New false
836 251 Acked 1250 Read 2 Done Done New false
837 252 Acked 1251 Flush 0 Done Done New false
838 253 Acked 1252 Read 3 Done Done New false
839 254 Acked 1253 Read 4 Done Done New false
840 255 Acked 1254 Write 7 Done Done New false
841 256 Acked 1255 Read 8 Done Done New false
842 257 Acked 1256 Write 9 Done Done New false
843 258 Acked 1257 Flush 0 Done Done New false
844 259 Acked 1258 Read 10 Done Done New false
845 260 Acked 1259 Read 8 Done Done New false
846 261 Acked 1260 Write 4 Done Done New false
847 262 Acked 1261 Read 8 Done Done New false
848 263 Acked 1262 Write 10 Done Done New false
849 264 Acked 1263 Read 10 Done Done New false
850 265 Acked 1264 Read 2 Done Done New false
851 266 Acked 1265 Read 9 Done Done New false
852 267 Acked 1266 Read 10 Done Done New false
853 268 Acked 1267 Read 5 Done Done New false
854 269 Acked 1268 Read 5 Done Done New false
855 270 Acked 1269 Write 4 Done Done New false
856 271 Acked 1270 Read 5 Done Done New false
857 272 Acked 1271 Read 10 Done Done New false
858 273 Acked 1272 Read 3 Done Done New false
859 274 Acked 1273 Write 3 Done Done New false
860 275 Acked 1274 Write 7 Done Done New false
861 276 Acked 1275 Read 4 Done Done New false
862 277 Acked 1276 Write 3 Done Done New false
863 278 Acked 1277 Read 2 Done Done New false
864 279 Acked 1278 Flush 0 Done Done New false
865 280 Acked 1279 Write 2 Done Done New false
866 281 Acked 1280 Flush 0 Done Done New false
867 282 Acked 1281 Read 8 Done Done New false
868 283 Acked 1282 Read 9 Done Done New false
869 284 Acked 1283 Write 8 Done Done New false
870 285 Acked 1284 Write 7 Done Done New false
871 286 Acked 1285 Write 8 Done Done New false
872 287 Acked 1286 Read 9 Done Done New false
873 288 Acked 1287 Flush 0 Done Done New false
874 289 Acked 1288 Flush 0 Done Done New false
875 290 Acked 1289 Write 5 Done Done New false
876 291 Acked 1290 Read 1 Done Done New false
877 292 Acked 1291 Write 7 Done Done New false
878 293 Acked 1292 Flush 0 Done Done New false
879 294 Acked 1293 Read 10 Done Done New false
880 295 Acked 1294 Write 5 Done Done New false
881 296 Acked 1295 Read 7 Done Done New false
882 297 Acked 1296 Write 8 Done Done New false
883 298 Acked 1297 Read 2 Done Done New false
884 299 Acked 1298 Write 6 Done Done New false
885 300 Acked 1299 Read 1 Done Done New false
886 STATES DS:0 DS:1 DS:2 TOTAL
887 New 0 0 300 300
888 Sent 0 0 0 0
889 Done 300 300 0 600
890 Skipped 0 0 0 0
891 Error 0 0 0 0
892 Last Flush: 1292 1292 0
893 Downstairs last five completed:
894 Upstairs last five completed: 300 299 298 297 296
895 CLIENT: Up:0 ds:300 act:2
8962024-02-12T01:34:38.738ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
8972024-02-12T01:34:38.738ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
8982024-02-12T01:34:38.738ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
8992024-02-12T01:34:38.738ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
9002024-02-12T01:34:38.738ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
9012024-02-12T01:34:38.738ZINFOcrucible: [2] client re-new 301 jobs since flush 0 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
9022024-02-12T01:34:38.739ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
9032024-02-12T01:34:39.739ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
9042024-02-12T01:34:39.739ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
9052024-02-12T01:34:39.739ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
9062024-02-12T01:34:39.739ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
9072024-02-12T01:34:39.739ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
9082024-02-12T01:34:39.740ZINFOcrucible: [2] client re-new 301 jobs since flush 0 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
9092024-02-12T01:34:39.740ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
910 [8830] got start action
911 [8830] I am going to restart
912 Starting downstairs at port 8830
913 Make output file at "/tmp/dsc/downstairs-8830.txt"
914 [8830][2] reports Starting
915 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1165)
916 [8830][2] reports Running
9172024-02-12T01:34:40.740ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
9182024-02-12T01:34:40.741ZINFOcrucible: ds_connection connected = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
9192024-02-12T01:34:40.742ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
9202024-02-12T01:34:40.742ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
9212024-02-12T01:34:40.742ZINFOcrucible: send last flush ID to this DS: 0 = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
9222024-02-12T01:34:40.743ZINFOcrucible: Replied this last flush ID: 0 = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
9232024-02-12T01:34:40.743ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
9242024-02-12T01:34:40.743ZINFOcrucible: [2] Transition from Offline to Replay = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
9252024-02-12T01:34:40.743ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
9262024-02-12T01:34:40.743ZINFOcrucible: [2] Transition from Replay to Active = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
927 ----------------------------------------------------------------
928 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
929 Downstairs last five completed: 1300 1299 1298 1297 1296
930 Upstairs last five completed: 301 300 299 298 297
931 CLIENT: Up:0 ds:0 act:3
932 Replay: All jobs finished, all DS active.
933 Main task has work to do, go find it
934 got dsc Stop(0)
935 stop 0
936 Feb 12 01:34:42.097 INFO request completed, latency_us: 330, response_code: 204, uri: /stop/cid/0, method: POST, req_id: b95d67a5-02ae-47c3-9a91-3dd8f7b0ea22, [8810] Got stop action so:false kr:false
937 remote_addr: 127.0.0.1:60456, local_addr: 127.0.0.1:9998
9382024-02-12T01:34:42.098ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
9392024-02-12T01:34:42.098ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
9402024-02-12T01:34:42.098ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
9412024-02-12T01:34:42.098ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
9422024-02-12T01:34:42.098ZINFOcrucible: [0] client re-new 0 jobs since flush 1300 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
9432024-02-12T01:34:42.098ZINFOcrucible: client task is exiting = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
944 Feb 12 01:34:42.098 INFO request completed, latency_us: 206, response_code: 200, uri: /state/cid/0, method: GET, req_id: b41118cc-2ffe-4953-ab9f-d1c556dd24ac, remote_addr: 127.0.0.1:60456, local_addr: 127.0.0.1:9998
945 [8810] Exited with: ExitStatus(unix_wait_status(9))
946 [8810][0] reports Exit
9472024-02-12T01:34:43.099ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
9482024-02-12T01:34:43.099ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
9492024-02-12T01:34:43.099ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
9502024-02-12T01:34:43.099ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
9512024-02-12T01:34:43.099ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
9522024-02-12T01:34:43.099ZINFOcrucible: [0] client re-new 0 jobs since flush 1300 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
9532024-02-12T01:34:43.099ZINFOcrucible: client task is exiting = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
9542024-02-12T01:34:44.100ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
9552024-02-12T01:34:44.100ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
9562024-02-12T01:34:44.100ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
9572024-02-12T01:34:44.100ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
9582024-02-12T01:34:44.101ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
9592024-02-12T01:34:44.101ZINFOcrucible: [0] client re-new 0 jobs since flush 1300 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
9602024-02-12T01:34:44.101ZINFOcrucible: client task is exiting = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
9612024-02-12T01:34:45.101ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
9622024-02-12T01:34:45.101ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
9632024-02-12T01:34:45.101ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
9642024-02-12T01:34:45.101ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
9652024-02-12T01:34:45.101ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
9662024-02-12T01:34:45.102ZINFOcrucible: [0] client re-new 0 jobs since flush 1300 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
9672024-02-12T01:34:45.102ZINFOcrucible: client task is exiting = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
968 Feb 12 01:34:46.100 INFO request completed, latency_us: 200, response_code: 200, uri: /state/cid/0, method: GET, req_id: 8b845fec-05a4-4e3b-abef-f1333431492b, remote_addr: 127.0.0.1:60456, local_addr: 127.0.0.1:9998
969 001/300 Read block 4425 len 8192
970 002/300 Read block 978 len 40960
9712024-02-12T01:34:46.103ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
9722024-02-12T01:34:46.103ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
9732024-02-12T01:34:46.103ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
974 {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"003/300time":" Write block 2700 len 4096 data:2024-02-12T01:34:46.10373593Z 3"
975 ,"hostname":"ip-10-150-1-104.us-west-2.compute.internal","pid":1164,"session_id":"e1773b3e-b36f-4080-8b13-c0830f0d072e"}
9762024-02-12T01:34:46.103ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
9772024-02-12T01:34:46.103ZINFOcrucible: [0] client re-new 2 jobs since flush 1300 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
9782024-02-12T01:34:46.103ZINFOcrucible: client task is exiting = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
979 004/300 Read block 2890 len 32768
980 005/300 Write block 668 len 28672 data: 2 2 2 2 2 2 2
981 006/300 Flush
982 007/300 Flush
983 008/300 Flush
984 009/300 Write block 228 len 20480 data: 2 2 2 2 2
985 010/300 Write block 2104 len 16384 data: 2 2 2 2
986 011/300 Read block 3876 len 40960
987 012/300 Read block 2434 len 24576
988 013/300 Read block 3427 len 32768
989 014/300 Read block 4556 len 12288
990 015/300 Write block 2739 len 4096 data: 2
991 016/300 Flush
992 017/300 Read block 4998 len 8192
993 018/300 Write block 1753 len 20480 data: 2 2 2 2 2
994 019/300 Read block 967 len 40960
995 020/300 Read block 4257 len 4096
996 021/300 Read block 2070 len 8192
997 022/300 Read block 4211 len 40960
998 023/300 Write block 4676 len 36864 data: 3 3 3 2 2 2 2 2 2
999 024/300 Write block 3790 len 16384 data: 2 2 2 2
1000 025/300 Read block 4655 len 36864
1001 026/300 Read block 4226 len 20480
1002 027/300 Read block 709 len 24576
1003 028/300 Write block 199 len 12288 data: 2 2 2
1004 029/300 Read block 1690 len 24576
1005 030/300 Write block 335 len 4096 data: 3
1006 031/300 Read block 3848 len 28672
1007 032/300 Flush
1008 033/300 Write block 395 len 12288 data: 2 2 2
1009 034/300 Write block 308 len 40960 data: 3 3 3 2 2 2 2 2 2 2
1010 035/300 Read block 3343 len 8192
1011 036/300 Read block 23 len 20480
1012 037/300 Write block 1254 len 12288 data: 2 2 2
1013 038/300 Write block 3241 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1014 039/300 Read block 1472 len 8192
1015 040/300 Write block 799 len 20480 data: 2 2 2 2 2
1016 041/300 Flush
1017 042/300 Write block 1380 len 8192 data: 2 2
1018 043/300 Read block 2444 len 12288
1019 044/300 Write block 1122 len 8192 data: 3 3
1020 045/300 Read block 1220 len 12288
1021 046/300 Read block 2547 len 12288
1022 047/300 Write block 3509 len 20480 data: 2 2 2 2 2
1023 048/300 Write block 400 len 20480 data: 2 3 3 3 3
1024 049/300 Flush
1025 050/300 Read block 2959 len 20480
1026 051/300 Flush
1027 052/300 Read block 490 len 32768
1028 053/300 Read block 2303 len 20480
1029 054/300 Read block 1191 len 12288
1030 055/300 Read block 3471 len 4096
1031 056/300 Write block 4168 len 28672 data: 3 3 3 2 2 2 2
1032 057/300 Read block 2492 len 36864
1033 058/300 Flush
1034 059/300 Write block 354 len 4096 data: 2
1035 060/300 Read block 2774 len 16384
1036 061/300 Read block 10 len 8192
1037 062/300 Read block 4714 len 36864
1038 063/300 Write block 1838 len 12288 data: 2 2 2
1039 064/300 Read block 2060 len 36864
1040 065/300 Write block 507 len 32768 data: 2 2 2 2 2 2 2 2
1041 066/300 Read block 2810 len 40960
1042 067/300 Read block 3290 len 28672
1043 068/300 Read block 1053 len 8192
1044 069/300 Read block 3372 len 36864
1045 070/300 Write block 3163 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1046 071/300 Read block 833 len 20480
1047 072/300 Read block 539 len 12288
1048 073/300 Write block 2719 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1049 074/300 Flush
1050 075/300 Read block 4126 len 36864
1051 076/300 Read block 923 len 40960
1052 077/300 Write block 3891 len 28672 data: 2 2 2 2 2 2 2
1053 078/300 Read block 1600 len 12288
1054 079/300 Read block 3998 len 4096
1055 080/300 Read block 3575 len 24576
1056 081/300 Read block 3530 len 4096
1057 082/300 Read block 2035 len 40960
1058 083/300 Write block 3574 len 28672 data: 3 3 2 2 2 2 2
1059 084/300 Read block 2995 len 28672
1060 085/300 Read block 3981 len 28672
1061 086/300 Read block 2445 len 28672
1062 087/300 Read block 285 len 28672
1063 088/300 Write block 1427 len 8192 data: 2 2
1064 089/300 Write block 3399 len 24576 data: 2 2 2 2 3 3
1065 090/300 Write block 2928 len 12288 data: 2 2 3
1066 091/300 Write block 4929 len 8192 data: 2 2
1067 092/300 Flush
1068 093/300 Read block 3638 len 20480
1069 094/300 Flush
1070 095/300 Flush
1071 096/300 Write block 1394 len 32768 data: 2 2 2 2 2 2 2 2
1072 097/300 Read block 707 len 28672
1073 098/300 Read block 3888 len 4096
1074 099/300 Write block 3869 len 36864 data: 2 2 2 2 2 2 2 2 2
1075 100/300 Read block 1184 len 32768
1076 101/300 Write block 2478 len 20480 data: 2 3 2 2 3
1077 102/300 Read block 2845 len 4096
1078 103/300 Read block 3525 len 24576
1079 104/300 Read block 3479 len 24576
1080 105/300 Read block 1674 len 28672
1081 106/300 Write block 4274 len 12288 data: 2 2 2
1082 107/300 Read block 2231 len 20480
1083 108/300 Read block 1801 len 32768
1084 109/300 Read block 2639 len 32768
1085 110/300 Read block 4563 len 24576
1086 111/300 Read block 593 len 28672
1087 112/300 Read block 3562 len 8192
1088 113/300 Read block 3080 len 8192
1089 114/300 Write block 512 len 28672 data: 3 3 3 2 2 2 2
1090 115/300 Read block 4888 len 40960
1091 116/300 Write block 790 len 36864 data: 2 2 2 2 2 2 2 2 2
1092 117/300 Read block 1127 len 4096
1093 118/300 Flush
1094 119/300 Write block 1260 len 12288 data: 2 2 2
1095 120/300 Read block 1753 len 4096
1096 121/300 Write block 3561 len 24576 data: 2 2 2 2 2 3
1097 122/300 Write block 2750 len 36864 data: 2 2 3 3 3 3 3 4 3
1098 123/300 Read block 3103 len 20480
1099 124/300 Flush
1100 125/300 Read block 2876 len 16384
1101 126/300 Write block 2321 len 4096 data: 2
1102 127/300 Read block 1503 len 20480
1103 128/300 Read block 3606 len 20480
1104 129/300 Write block 2614 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1105 130/300 Write block 1677 len 24576 data: 2 3 3 3 3 3
1106 131/300 Read block 4778 len 28672
1107 132/300 Flush
1108 133/300 Write block 1019 len 20480 data: 2 2 2 2 2
1109 134/300 Read block 1568 len 8192
1110 135/300 Write block 4712 len 16384 data: 2 2 2 2
1111 136/300 Read block 4150 len 24576
1112 137/300 Flush
1113 138/300 Write block 2598 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1114 139/300 Read block 2546 len 16384
1115 140/300 Write block 2583 len 36864 data: 2 2 2 2 2 2 2 2 2
1116 141/300 Write block 4162 len 20480 data: 2 3 3 3 3
1117 142/300 Write block 1605 len 20480 data: 2 2 3 3 3
1118 143/300 Write block 1809 len 24576 data: 2 2 2 2 2 2
1119 144/300 Read block 615 len 36864
1120 145/300 Write block 2837 len 12288 data: 2 2 2
1121 146/300 Read block 4319 len 40960
1122 147/300 Write block 2681 len 16384 data: 2 2 2 2
1123 148/300 Flush
1124 149/300 Flush
1125 150/300 Write block 3894 len 36864 data: 3 3 3 3 2 2 2 2 2
1126 151/300 Flush
1127 152/300 Read block 2822 len 24576
1128 153/300 Flush
1129 154/300 Read block 757 len 16384
1130 155/300 Write block 2580 len 36864 data: 2 2 2 3 3 3 3 3 3
1131 156/300 Read block 2196 len 16384
1132 157/300 Write block 3130 len 4096 data: 2
1133 158/300 Read block 0 len 32768
1134 159/300 Write block 1334 len 8192 data: 3 3
1135 160/300 Flush
1136 161/300 Write block 4606 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1137 162/300 Read block 991 len 28672
1138 163/300 Read block 1998 len 28672
1139 164/300 Flush
1140 165/300 Read block 3149 len 4096
1141 166/300 Write block 892 len 36864 data: 2 2 2 2 2 2 2 2 2
1142 167/300 Write block 3133 len 28672 data: 2 2 2 2 2 2 2
1143 168/300 Write block 1583 len 20480 data: 2 2 2 2 2
1144 169/300 Read block 3934 len 40960
1145 170/300 Read block 2070 len 16384
1146 171/300 Write block 799 len 20480 data: 3 3 3 3 3
1147 172/300 Write block 3883 len 12288 data: 2 2 2
1148 173/300 Read block 1656 len 32768
1149 174/300 Read block 4107 len 24576
1150 175/300 Read block 4665 len 12288
1151 176/300 Read block 1301 len 4096
1152 177/300 Read block 2196 len 20480
1153 178/300 Read block 2326 len 40960
1154 179/300 Flush
1155 180/300 Flush
1156 181/300 Read block 219 len 12288
1157 182/300 Write block 2924 len 16384 data: 2 2 2 2
1158 183/300 Read block 3067 len 4096
1159 184/300 Write block 4519 len 20480 data: 2 2 2 2 2
1160 185/300 Write block 2092 len 32768 data: 2 2 2 3 3 3 3 3
1161 186/300 Write block 3089 len 8192 data: 2 2
1162 187/300 Write block 531 len 16384 data: 2 2 2 3
1163 188/300 Write block 150 len 12288 data: 2 2 2
1164 189/300 Read block 1550 len 8192
1165 190/300 Read block 1070 len 12288
1166 191/300 Flush
1167 192/300 Read block 758 len 32768
1168 193/300 Read block 1261 len 20480
1169 194/300 Flush
1170 195/300 Write block 1340 len 24576 data: 2 2 2 2 3 2
1171 196/300 Read block 4251 len 40960
1172 197/300 Write block 4218 len 36864 data: 2 2 2 2 2 3 3 3 3
1173 198/300 Flush
1174 199/300 Write block 3098 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1175 200/300 Read block 881 len 4096
1176 201/300 Read block 2611 len 40960
1177 202/300 Write block 3184 len 40960 data: 3 3 3 3 4 3 3 3 3 3
1178 203/300 Read block 4311 len 16384
1179 204/300 Read block 1554 len 16384
1180 205/300 Read block 4971 len 8192
1181 206/300 Write block 3034 len 32768 data: 2 2 2 2 2 2 2 2
1182 207/300 Write block 3408 len 12288 data: 2 2 2
1183 208/300 Write block 392 len 40960 data: 2 2 2 3 3 3 2 2 3 4
1184 209/300 Read block 3788 len 36864
1185 210/300 Write block 1356 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1186 211/300 Read block 4317 len 32768
1187 212/300 Write block 2896 len 8192 data: 2 2
1188 213/300 Write block 726 len 12288 data: 2 3 3
1189 214/300 Write block 4186 len 16384 data: 2 2 2 2
1190 215/300 Write block 1171 len 24576 data: 3 3 3 2 2 2
1191 216/300 Read block 1716 len 4096
1192 217/300 Write block 2196 len 28672 data: 2 2 2 3 3 3 3
1193 218/300 Read block 4728 len 20480
1194 219/300 Write block 4045 len 8192 data: 2 2
1195 220/300 Write block 1256 len 40960 data: 3 2 2 2 3 3 3 2 2 2
1196 221/300 Read block 511 len 12288
1197 222/300 Read block 2046 len 16384
1198 223/300 Read block 611 len 12288
1199 224/300 Read block 493 len 40960
1200 225/300 Flush
1201 226/300 Write block 408 len 12288 data: 3 3 3
1202 227/300 Write block 1515 len 16384 data: 2 2 2 2
1203 228/300 Read block 4157 len 24576
1204 229/300 Flush
1205 230/300 Write block 3219 len 40960 data: 2 2 2 2 2 3 3 3 3 3
1206 231/300 Read block 2631 len 40960
1207 232/300 Read block 3055 len 12288
1208 233/300 Read block 3107 len 16384
1209 234/300 Write block 636 len 28672 data: 2 2 2 2 2 2 2
1210 235/300 Write block 3021 len 20480 data: 2 2 2 2 2
1211 236/300 Read block 3302 len 16384
1212 237/300 Read block 350 len 4096
1213 238/300 Read block 1367 len 36864
1214 239/300 Read block 2852 len 8192
1215 240/300 Read block 3917 len 36864
1216 241/300 Write block 3688 len 4096 data: 3
1217 242/300 Flush
1218 243/300 Write block 484 len 12288 data: 2 2 2
1219 244/300 Write block 1015 len 36864 data: 2 2 2 2 3 3 3 3 3
1220 245/300 Write block 1377 len 8192 data: 2 2
1221 246/300 Read block 1436 len 16384
1222 247/300 Read block 787 len 32768
1223 248/300 Read block 4244 len 4096
1224 249/300 Read block 3286 len 20480
1225 250/300 Write block 3654 len 40960 data: 2 2 2 2 2 2 2 2 3 3
1226 251/300 Read block 2437 len 24576
1227 252/300 Write block 2067 len 36864 data: 2 2 2 2 2 2 2 2 2
1228 253/300 Read block 3120 len 24576
1229 254/300 Read block 4480 len 16384
1230 255/300 Write block 2174 len 24576 data: 2 2 2 2 2 2
1231 256/300 Read block 1758 len 12288
1232 257/300 Write block 1053 len 12288 data: 2 2 2
1233 258/300 Write block 3220 len 28672 data: 3 3 3 3 4 4 4
1234 259/300 Read block 2703 len 4096
1235 260/300 Write block 1380 len 12288 data: 3 3 2
1236 261/300 Write block 4825 len 12288 data: 2 2 2
1237 262/300 Read block 2643 len 40960
1238 263/300 Flush
1239 264/300 Flush
1240 265/300 Write block 2642 len 32768 data: 2 2 2 2 2 2 2 2
1241 266/300 Read block 1086 len 16384
1242 267/300 Read block 4791 len 32768
1243 268/300 Read block 709 len 28672
1244 269/300 Read block 1475 len 28672
1245 270/300 Write block 2804 len 40960 data: 3 3 3 3 3 3 2 2 2 2
1246 271/300 Read block 4637 len 4096
1247 272/300 Write block 739 len 16384 data: 2 2 2 2
1248 273/300 Write block 3634 len 28672 data: 2 2 2 2 2 2 2
1249 274/300 Read block 1184 len 12288
1250 275/300 Write block 2505 len 24576 data: 2 2 2 2 2 2
1251 276/300 Flush
1252 277/300 Read block 4405 len 8192
1253 278/300 Read block 2511 len 16384
1254 279/300 Write block 2508 len 20480 data: 3 3 3 2 2
1255 280/300 Write block 2156 len 4096 data: 2
1256 281/300 Write block 4038 len 32768 data: 2 2 2 2 2 2 2 3
1257 282/300 Read block 2531 len 12288
1258 283/300 Write block 2591 len 32768 data: 3 2 2 2 2 2 2 3
1259 284/300 Write block 4694 len 4096 data: 2
1260 285/300 Read block 4470 len 20480
1261 286/300 Write block 3481 len 20480 data: 2 2 2 2 2
1262 287/300 Read block 3368 len 4096
1263 288/300 Read block 4084 len 20480
1264 289/300 Read block 1246 len 24576
1265 290/300 Read block 3540 len 28672
1266 291/300 Write block 666 len 4096 data: 2
1267 292/300 Read block 2587 len 28672
1268 293/300 Write block 1717 len 16384 data: 2 2 2 2
1269 294/300 Write block 1847 len 40960 data: 2 2 2 2 2 2 2 2 2 3
1270 295/300 Read block 4493 len 24576
1271 296/300 Read block 2943 len 36864
1272 297/300 Read block 2936 len 20480
1273 298/300 Write block 4969 len 12288 data: 3 3 2
1274 299/300 Write block 673 len 24576 data: 3 3 2 2 2 2
1275 300/300 Write block 444 len 12288 data: 2 2 2
1276 Main task has work to do, go find it
1277 got dsc Start(0)
1278 start 0
1279 Feb 12 01:34:46.445 INFO request completed, latency_us: 232, response_code: 204, uri: /start/cid/0, method: POST, req_id: 35f81035-f495-4ae5-b08a-31c19bc4b095, remote_addr: 127.0.0.1:60456, local_addr: 127.0.0.1:9998
1280 Replay: started 0, returned:Ok(())
1281 ----------------------------------------------------------------
1282 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
1283 States: Offline Active Active
1284 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
1285 302 Acked 1301 Read 2 New Done Done true
1286 303 Acked 1302 Read 10 New Done Done true
1287 304 Acked 1303 Write 1 New Done Done false
1288 305 Acked 1304 Read 8 New Done Done false
1289 306 Acked 1305 Write 7 New Done Done false
1290 307 Acked 1306 Flush 0 New Done Done false
1291 308 Acked 1307 Flush 0 New Done Done false
1292 309 Acked 1308 Flush 0 New Done Done false
1293 310 Acked 1309 Write 5 New Done Done false
1294 311 Acked 1310 Write 4 New Done Done false
1295 312 Acked 1311 Read 10 New Done Done false
1296 313 Acked 1312 Read 6 New Done Done false
1297 314 Acked 1313 Read 8 New Done Done false
1298 315 Acked 1314 Read 3 New Done Done false
1299 316 Acked 1315 Write 1 New Done Done false
1300 317 Acked 1316 Flush 0 New Done Done false
1301 318 Acked 1317 Read 2 New Done Done false
1302 319 Acked 1318 Write 5 New Done Done false
1303 320 Acked 1319 Read 10 New Done Done false
1304 321 Acked 1320 Read 1 New Done Done false
1305 322 Acked 1321 Read 2 New Done Done false
1306 323 Acked 1322 Read 10 New Done Done false
1307 324 Acked 1323 Write 9 New Done Done false
1308 325 Acked 1324 Write 4 New Done Done false
1309 326 Acked 1325 Read 9 New Done Done false
1310 327 Acked 1326 Read 5 New Done Done false
1311 328 Acked 1327 Read 6 New Done Done false
1312 329 Acked 1328 Write 3 New Done Done false
1313 330 Acked 1329 Read 6 New Done Done false
1314 331 Acked 1330 Write 1 New Done Done false
1315 332 Acked 1331 Read 7 New Done Done false
1316 333 Acked 1332 Flush 0 New Done Done false
1317 334 Acked 1333 Write 3 New Done Done false
1318 335 Acked 1334 Write 10 New Done Done false
1319 336 Acked 1335 Read 2 New Done Done false
1320 337 Acked 1336 Read 5 New Done Done false
1321 338 Acked 1337 Write 3 New Done Done false
1322 339 Acked 1338 Write 10 New Done Done false
1323 340 Acked 1339 Read 2 New Done Done false
1324 341 Acked 1340 Write 5 New Done Done false
1325 342 Acked 1341 Flush 0 New Done Done false
1326 343 Acked 1342 Write 2 New Done Done false
1327 344 Acked 1343 Read 3 New Done Done false
1328 345 Acked 1344 Write 2 New Done Done false
1329 346 Acked 1345 Read 3 New Done Done false
1330 347 Acked 1346 Read 3 New Done Done false
1331 348 Acked 1347 Write 5 New Done Done false
1332 349 Acked 1348 Write 5 New Done Done false
1333 350 Acked 1349 Flush 0 New Done Done false
1334 351 Acked 1350 Read 5 New Done Done false
1335 352 Acked 1351 Flush 0 New Done Done false
1336 353 Acked 1352 Read 8 New Done Done false
1337 354 Acked 1353 Read 5 New Done Done false
1338 355 Acked 1354 Read 3 New Done Done false
1339 356 Acked 1355 Read 1 New Done Done false
1340 357 Acked 1356 Write 7 New Done Done false
1341 358 Acked 1357 Read 9 New Done Done false
1342 359 Acked 1358 Flush 0 New Done Done false
1343 360 Acked 1359 Write 1 New Done Done false
1344 361 Acked 1360 Read 4 New Done Done false
1345 362 Acked 1361 Read 2 New Done Done false
1346 363 Acked 1362 Read 9 New Done Done false
1347 364 Acked 1363 Write 3 New Done Done false
1348 365 Acked 1364 Read 9 New Done Done false
1349 366 Acked 1365 Write 8 New Done Done false
1350 367 Acked 1366 Read 10 New Done Done false
1351 368 Acked 1367 Read 7 New Done Done false
1352 369 Acked 1368 Read 2 New Done Done false
1353 370 Acked 1369 Read 9 New Done Done false
1354 371 Acked 1370 Write 10 New Done Done false
1355 372 Acked 1371 Read 5 New Done Done false
1356 373 Acked 1372 Read 3 New Done Done false
1357 374 Acked 1373 Write 10 New Done Done false
1358 375 Acked 1374 Flush 0 New Done Done false
1359 376 Acked 1375 Read 9 New Done Done false
1360 377 Acked 1376 Read 10 New Done Done false
1361 378 Acked 1377 Write 7 New Done Done false
1362 379 Acked 1378 Read 3 New Done Done false
1363 380 Acked 1379 Read 1 New Done Done false
1364 381 Acked 1380 Read 6 New Done Done false
1365 382 Acked 1381 Read 1 New Done Done false
1366 383 Acked 1382 Read 10 New Done Done false
1367 384 Acked 1383 Write 7 New Done Done false
1368 385 Acked 1384 Read 7 New Done Done false
1369 386 Acked 1385 Read 7 New Done Done false
1370 387 Acked 1386 Read 7 New Done Done false
1371 388 Acked 1387 Read 7 New Done Done false
1372 389 Acked 1388 Write 2 New Done Done false
1373 390 Acked 1389 Flush 0 New Done Done false
1374 391 Acked 1390 Write 6 New Done Done false
1375 392 Acked 1391 Write 3 New Done Done false
1376 393 Acked 1392 Write 2 New Done Done false
1377 394 Acked 1393 Flush 0 New Done Done false
1378 395 Acked 1394 Read 5 New Done Done false
1379 396 Acked 1395 Flush 0 New Done Done false
1380 397 Acked 1396 Flush 0 New Done Done false
1381 398 Acked 1397 Write 8 New Done Done false
1382 399 Acked 1398 Read 7 New Done Done false
1383 400 Acked 1399 Read 1 New Done Done false
1384 401 Acked 1400 Write 9 New Done Done false
1385 402 Acked 1401 Read 8 New Done Done false
1386 403 Acked 1402 Write 5 New Done Done false
1387 404 Acked 1403 Read 1 New Done Done false
1388 405 Acked 1404 Read 6 New Done Done false
1389 406 Acked 1405 Read 6 New Done Done false
1390 407 Acked 1406 Read 7 New Done Done false
1391 408 Acked 1407 Write 3 New Done Done false
1392 409 Acked 1408 Read 5 New Done Done false
1393 410 Acked 1409 Read 8 New Done Done false
1394 411 Acked 1410 Read 8 New Done Done false
1395 412 Acked 1411 Read 6 New Done Done false
1396 413 Acked 1412 Read 7 New Done Done false
1397 414 Acked 1413 Read 2 New Done Done false
1398 415 Acked 1414 Read 2 New Done Done false
1399 416 Acked 1415 Write 7 New Done Done false
1400 417 Acked 1416 Read 10 New Done Done false
1401 418 Acked 1417 Write 9 New Done Done false
1402 419 Acked 1418 Read 1 New Done Done false
1403 420 Acked 1419 Flush 0 New Done Done false
1404 421 Acked 1420 Write 3 New Done Done false
1405 422 Acked 1421 Read 1 New Done Done false
1406 423 Acked 1422 Write 6 New Done Done false
1407 424 Acked 1423 Write 9 New Done Done false
1408 425 Acked 1424 Read 5 New Done Done false
1409 426 Acked 1425 Flush 0 New Done Done false
1410 427 Acked 1426 Read 4 New Done Done false
1411 428 Acked 1427 Write 1 New Done Done false
1412 429 Acked 1428 Read 5 New Done Done false
1413 430 Acked 1429 Read 5 New Done Done false
1414 431 Acked 1430 Write 10 New Done Done false
1415 432 Acked 1431 Write 6 New Done Done false
1416 433 Acked 1432 Read 7 New Done Done false
1417 434 Acked 1433 Flush 0 New Done Done false
1418 435 Acked 1434 Write 5 New Done Done false
1419 436 Acked 1435 Read 2 New Done Done false
1420 437 Acked 1436 Write 4 New Done Done false
1421 438 Acked 1437 Read 6 New Done Done false
1422 439 Acked 1438 Flush 0 New Done Done false
1423 440 Acked 1439 Write 10 New Done Done false
1424 441 Acked 1440 Read 4 New Done Done false
1425 442 Acked 1441 Write 9 New Done Done false
1426 443 Acked 1442 Write 5 New Done Done false
1427 444 Acked 1443 Write 5 New Done Done false
1428 445 Acked 1444 Write 6 New Done Done false
1429 446 Acked 1445 Read 9 New Done Done false
1430 447 Acked 1446 Write 3 New Done Done false
1431 448 Acked 1447 Read 10 New Done Done false
1432 449 Acked 1448 Write 4 New Done Done false
1433 450 Acked 1449 Flush 0 New Done Done false
1434 451 Acked 1450 Flush 0 New Done Done false
1435 452 Acked 1451 Write 9 New Done Done false
1436 453 Acked 1452 Flush 0 New Done Done false
1437 454 Acked 1453 Read 6 New Done Done false
1438 455 Acked 1454 Flush 0 New Done Done false
1439 456 Acked 1455 Read 4 New Done Done false
1440 457 Acked 1456 Write 9 New Done Done false
1441 458 Acked 1457 Read 4 New Done Done false
1442 459 Acked 1458 Write 1 New Done Done false
1443 460 Acked 1459 Read 8 New Done Done false
1444 461 Acked 1460 Write 2 New Done Done false
1445 462 Acked 1461 Flush 0 New Done Done false
1446 463 Acked 1462 Write 10 New Done Done false
1447 464 Acked 1463 Read 7 New Done Done false
1448 465 Acked 1464 Read 7 New Done Done false
1449 466 Acked 1465 Flush 0 New Done Done false
1450 467 Acked 1466 Read 1 New Done Done false
1451 468 Acked 1467 Write 9 New Done Done false
1452 469 Acked 1468 Write 7 New Done Done false
1453 470 Acked 1469 Write 5 New Done Done false
1454 471 Acked 1470 Read 10 New Done Done false
1455 472 Acked 1471 Read 4 New Done Done false
1456 473 Acked 1472 Write 5 New Done Done false
1457 474 Acked 1473 Write 3 New Done Done false
1458 475 Acked 1474 Read 8 New Done Done false
1459 476 Acked 1475 Read 6 New Done Done false
1460 477 Acked 1476 Read 3 New Done Done false
1461 478 Acked 1477 Read 1 New Done Done false
1462 479 Acked 1478 Read 5 New Done Done false
1463 480 Acked 1479 Read 10 New Done Done false
1464 481 Acked 1480 Flush 0 New Done Done false
1465 482 Acked 1481 Flush 0 New Done Done false
1466 483 Acked 1482 Read 3 New Done Done false
1467 484 Acked 1483 Write 4 New Done Done false
1468 485 Acked 1484 Read 1 New Done Done false
1469 486 Acked 1485 Write 5 New Done Done false
1470 487 Acked 1486 Write 8 New Done Done false
1471 488 Acked 1487 Write 2 New Done Done false
1472 489 Acked 1488 Write 4 New Done Done false
1473 490 Acked 1489 Write 3 New Done Done false
1474 491 Acked 1490 Read 2 New Done Done false
1475 492 Acked 1491 Read 3 New Done Done false
1476 493 Acked 1492 Flush 0 New Done Done false
1477 494 Acked 1493 Read 8 New Done Done false
1478 495 Acked 1494 Read 5 New Done Done false
1479 496 Acked 1495 Flush 0 New Done Done false
1480 497 Acked 1496 Write 6 New Done Done false
1481 498 Acked 1497 Read 10 New Done Done false
1482 499 Acked 1498 Write 9 New Done Done false
1483 500 Acked 1499 Flush 0 New Done Done false
1484 501 Acked 1500 Write 10 New Done Done false
1485 502 Acked 1501 Read 1 New Done Done false
1486 503 Acked 1502 Read 10 New Done Done false
1487 504 Acked 1503 Write 10 New Done Done false
1488 505 Acked 1504 Read 4 New Done Done false
1489 506 Acked 1505 Read 4 New Done Done false
1490 507 Acked 1506 Read 2 New Done Done false
1491 508 Acked 1507 Write 8 New Done Done false
1492 509 Acked 1508 Write 3 New Done Done false
1493 510 Acked 1509 Write 10 New Done Done false
1494 511 Acked 1510 Read 9 New Done Done false
1495 512 Acked 1511 Write 10 New Done Done false
1496 513 Acked 1512 Read 8 New Done Done false
1497 514 Acked 1513 Write 2 New Done Done false
1498 515 Acked 1514 Write 3 New Done Done false
1499 516 Acked 1515 Write 4 New Done Done false
1500 517 Acked 1516 Write 6 New Done Done false
1501 518 Acked 1517 Read 1 New Done Done false
1502 519 Acked 1518 Write 7 New Done Done false
1503 520 Acked 1519 Read 5 New Done Done false
1504 521 Acked 1520 Write 2 New Done Done false
1505 522 Acked 1521 Write 10 New Done Done false
1506 523 Acked 1522 Read 3 New Done Done false
1507 524 Acked 1523 Read 4 New Done Done false
1508 525 Acked 1524 Read 3 New Done Done false
1509 526 Acked 1525 Read 10 New Done Done false
1510 527 Acked 1526 Flush 0 New Done Done false
1511 528 Acked 1527 Write 3 New Done Done false
1512 529 Acked 1528 Write 4 New Done Done false
1513 530 Acked 1529 Read 6 New Done Done false
1514 531 Acked 1530 Flush 0 New Done Done false
1515 532 Acked 1531 Write 10 New Done Done false
1516 533 Acked 1532 Read 10 New Done Done false
1517 534 Acked 1533 Read 3 New Done Done false
1518 535 Acked 1534 Read 4 New Done Done false
1519 536 Acked 1535 Write 7 New Done Done false
1520 537 Acked 1536 Write 5 New Done Done false
1521 538 Acked 1537 Read 4 New Done Done false
1522 539 Acked 1538 Read 1 New Done Done false
1523 540 Acked 1539 Read 9 New Done Done false
1524 541 Acked 1540 Read 2 New Done Done false
1525 542 Acked 1541 Read 9 New Done Done false
1526 543 Acked 1542 Write 1 New Done Done false
1527 544 Acked 1543 Flush 0 New Done Done false
1528 545 Acked 1544 Write 3 New Done Done false
1529 546 Acked 1545 Write 9 New Done Done false
1530 547 Acked 1546 Write 2 New Done Done false
1531 548 Acked 1547 Read 4 New Done Done false
1532 549 Acked 1548 Read 8 New Done Done false
1533 550 Acked 1549 Read 1 New Done Done false
1534 551 Acked 1550 Read 5 New Done Done false
1535 552 Acked 1551 Write 10 New Done Done false
1536 553 Acked 1552 Read 6 New Done Done false
1537 554 Acked 1553 Write 9 New Done Done false
1538 555 Acked 1554 Read 6 New Done Done false
1539 556 Acked 1555 Read 4 New Done Done false
1540 557 Acked 1556 Write 6 New Done Done false
1541 558 Acked 1557 Read 3 New Done Done false
1542 559 Acked 1558 Write 3 New Done Done false
1543 560 Acked 1559 Write 7 New Done Done false
1544 561 Acked 1560 Read 1 New Done Done false
1545 562 Acked 1561 Write 3 New Done Done false
1546 563 Acked 1562 Write 3 New Done Done false
1547 564 Acked 1563 Read 10 New Done Done false
1548 565 Acked 1564 Flush 0 New Done Done false
1549 566 Acked 1565 Flush 0 New Done Done false
1550 567 Acked 1566 Write 8 New Done Done false
1551 568 Acked 1567 Read 4 New Done Done false
1552 569 Acked 1568 Read 8 New Done Done false
1553 570 Acked 1569 Read 7 New Done Done false
1554 571 Acked 1570 Read 7 New Done Done false
1555 572 Acked 1571 Write 10 New Done Done false
1556 573 Acked 1572 Read 1 New Done Done false
1557 574 Acked 1573 Write 4 New Done Done false
1558 575 Acked 1574 Write 7 New Done Done false
1559 576 Acked 1575 Read 3 New Done Done false
1560 577 Acked 1576 Write 6 New Done Done false
1561 578 Acked 1577 Flush 0 New Done Done false
1562 579 Acked 1578 Read 2 New Done Done false
1563 580 Acked 1579 Read 4 New Done Done false
1564 581 Acked 1580 Write 5 New Done Done false
1565 582 Acked 1581 Write 1 New Done Done false
1566 583 Acked 1582 Write 8 New Done Done false
1567 584 Acked 1583 Read 3 New Done Done false
1568 585 Acked 1584 Write 8 New Done Done false
1569 586 Acked 1585 Write 1 New Done Done false
1570 587 Acked 1586 Read 5 New Done Done false
1571 588 Acked 1587 Write 5 New Done Done false
1572 589 Acked 1588 Read 1 New Done Done false
1573 590 Acked 1589 Read 5 New Done Done false
1574 591 Acked 1590 Read 6 New Done Done false
1575 592 Acked 1591 Read 7 New Done Done false
1576 593 Acked 1592 Write 1 New Done Done false
1577 594 Acked 1593 Read 7 New Done Done false
1578 595 Acked 1594 Write 4 New Done Done false
1579 596 Acked 1595 Write 10 New Done Done false
1580 597 Acked 1596 Read 6 New Done Done false
1581 598 Acked 1597 Read 9 New Done Done false
1582 599 Acked 1598 Read 5 New Done Done false
1583 600 Acked 1599 Write 3 New Done Done false
1584 601 Acked 1600 Write 6 New Done Done false
1585 602 Acked 1601 Write 3 New Done Done false
1586 STATES DS:0 DS:1 DS:2 TOTAL
1587 New 301 0 0 301
1588 Sent 0 0 0 0
1589 Done 0 301 301 602
1590 Skipped 0 0 0 0
1591 Error 0 0 0 0
1592 Last Flush: 1300 1577 1577
1593 Downstairs last five completed: 1300 1299 1298 1297 1296
1594 Upstairs last five completed: 602 601 600 599 598
1595 CLIENT: Up:0 ds:301 act:2
15962024-02-12T01:34:47.104ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
15972024-02-12T01:34:47.104ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
15982024-02-12T01:34:47.104ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
15992024-02-12T01:34:47.104ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
16002024-02-12T01:34:47.104ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
16012024-02-12T01:34:47.104ZINFOcrucible: [0] client re-new 302 jobs since flush 1300 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
16022024-02-12T01:34:47.104ZINFOcrucible: client task is exiting = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
1603 [8810] got start action
1604 [8810] I am going to restart
1605 Starting downstairs at port 8810
1606 Make output file at "/tmp/dsc/downstairs-8810.txt"
1607 [8810][0] reports Starting
16082024-02-12T01:34:48.105ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
16092024-02-12T01:34:48.105ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
16102024-02-12T01:34:48.105ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
16112024-02-12T01:34:48.105ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
16122024-02-12T01:34:48.105ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
16132024-02-12T01:34:48.105ZINFOcrucible: [0] client re-new 302 jobs since flush 1300 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
16142024-02-12T01:34:48.105ZINFOcrucible: client task is exiting = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
1615 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1166)
1616 [8810][0] reports Running
16172024-02-12T01:34:49.106ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
16182024-02-12T01:34:49.106ZINFOcrucible: ds_connection connected = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
16192024-02-12T01:34:49.107ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
16202024-02-12T01:34:49.108ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
16212024-02-12T01:34:49.108ZINFOcrucible: send last flush ID to this DS: 1300 = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
16222024-02-12T01:34:49.108ZINFOcrucible: Replied this last flush ID: 1300 = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
16232024-02-12T01:34:49.108ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
16242024-02-12T01:34:49.108ZINFOcrucible: [0] Transition from Offline to Replay = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
16252024-02-12T01:34:49.108ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
16262024-02-12T01:34:49.108ZINFOcrucible: [0] Transition from Replay to Active = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
1627 ----------------------------------------------------------------
1628 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
1629 Downstairs last five completed: 1602 1601 1600 1599 1598
1630 Upstairs last five completed: 603 602 601 600 599
1631 CLIENT: Up:0 ds:0 act:3
1632 Replay: All jobs finished, all DS active.
1633 Main task has work to do, go find it
1634 got dsc Stop(2)
1635 stop 2
1636 Feb 12 01:34:50.448 INFO request completed, latency_us: 352, response_code: 204, uri: /stop/cid/2, method: POST, req_id: bc0b5727-8113-45ef-bda6-bad5095ec849, [8830] Got stop action so:false kr:false
1637 remote_addr: 127.0.0.1:60456, local_addr: 127.0.0.1:9998
16382024-02-12T01:34:50.449ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
1639 Feb 12 01:34:50.449 INFO request completed, latency_us: 165, response_code: 200, uri: /state/cid/2, method{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2024-02-12T01:34:50.449348252Z","hostname":"ip-10-150-1-104.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"e1773b3e-b36f-4080-8b13-c0830f0d072e"}
1640 : GET, req_id: 09cb1ed6-73ea-480f-bfbe-26916cf79e34, remote_addr: 127{"msg":"downstairs task for 2 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-02-12T01:34:50.449393406Z","hostname":"ip-10-150-1-104.us-west-2.compute.internal","pid":1164,"session_id":"e1773b3e-b36f-4080-8b13-c0830f0d072e"}
1641 .0.0.1:60456, local_addr: 127.0.0.1:9998{"msg":"Gone missing, transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-02-12T01:34:50.449454667Z","hostname":"ip-10-150-1-104.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"e1773b3e-b36f-4080-8b13-c0830f0d072e"}
1642 
16432024-02-12T01:34:50.449ZINFOcrucible: [2] client re-new 0 jobs since flush 1602 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
16442024-02-12T01:34:50.449ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
1645 [8830] Exited with: ExitStatus(unix_wait_status(9))
1646 [8830][2] reports Exit
16472024-02-12T01:34:51.450ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
16482024-02-12T01:34:51.450ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
16492024-02-12T01:34:51.450ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
16502024-02-12T01:34:51.450ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
16512024-02-12T01:34:51.450ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
16522024-02-12T01:34:51.450ZINFOcrucible: [2] client re-new 0 jobs since flush 1602 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
16532024-02-12T01:34:51.450ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
16542024-02-12T01:34:52.451ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
16552024-02-12T01:34:52.451ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
16562024-02-12T01:34:52.451ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
16572024-02-12T01:34:52.451ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
16582024-02-12T01:34:52.451ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
16592024-02-12T01:34:52.451ZINFOcrucible: [2] client re-new 0 jobs since flush 1602 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
16602024-02-12T01:34:52.451ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
16612024-02-12T01:34:53.452ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
16622024-02-12T01:34:53.452ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
16632024-02-12T01:34:53.452ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
16642024-02-12T01:34:53.452ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
16652024-02-12T01:34:53.452ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
16662024-02-12T01:34:53.452ZINFOcrucible: [2] client re-new 0 jobs since flush 1602 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
16672024-02-12T01:34:53.452ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
1668 Feb 12 01:34:54.451 INFO request completed, latency_us: 246, response_code: 200, uri: /state/cid/2, method: GET, req_id: e2c71d1e-b3ee-49ee-975a-6894e60b6afd, remote_addr: 127.0.0.1:60456, local_addr: 127.0.0.1:9998
1669 001/300 Read block 3986 len 12288
1670 002/300 Read block 2037 len 40960
16712024-02-12T01:34:54.453ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
16722024-02-12T01:34:54.453ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
16732024-02-12T01:34:54.453ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
16742024-02-12T01:34:54.453ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
16752024-02-12T01:34:54.453ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
16762024-02-12T01:34:54.453ZINFOcrucible: [2] client re-new 2 jobs since flush 1602 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
16772024-02-12T01:34:54.454ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
1678 003/300 Write block 1917 len 28672 data: 3 3 3 3 2 2 2
1679 004/300 Write block 4233 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1680 005/300 Read block 3129 len 32768
1681 006/300 Write block 850 len 32768 data: 2 2 2 2 2 2 2 2
1682 007/300 Read block 793 len 16384
1683 008/300 Write block 45 len 20480 data: 2 2 2 2 2
1684 009/300 Write block 2210 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1685 010/300 Write block 543 len 32768 data: 2 2 2 2 2 2 2 2
1686 011/300 Flush
1687 012/300 Read block 3166 len 32768
1688 013/300 Read block 1542 len 40960
1689 014/300 Write block 1770 len 24576 data: 2 2 2 2 2 2
1690 015/300 Flush
1691 016/300 Write block 4180 len 28672 data: 2 2 2 2 2 2 3
1692 017/300 Read block 1087 len 8192
1693 018/300 Read block 2077 len 24576
1694 019/300 Flush
1695 020/300 Write block 17 len 24576 data: 2 2 2 2 2 2
1696 021/300 Read block 1612 len 20480
1697 022/300 Read block 1744 len 16384
1698 023/300 Read block 2546 len 4096
1699 024/300 Flush
1700 025/300 Read block 1953 len 8192
1701 026/300 Flush
1702 027/300 Write block 1815 len 20480 data: 2 2 2 2 2
1703 028/300 Write block 1726 len 16384 data: 2 2 2 2
1704 029/300 Write block 99 len 40960 data: 3 3 3 3 3 3 3 3 3 2
1705 030/300 Write block 1829 len 4096 data: 3
1706 031/300 Read block 1500 len 24576
1707 032/300 Read block 4938 len 16384
1708 033/300 Read block 3986 len 32768
1709 034/300 Write block 2434 len 16384 data: 2 2 2 2
1710 035/300 Write block 2996 len 24576 data: 2 2 2 2 2 2
1711 036/300 Write block 1420 len 32768 data: 2 2 2 2 2 2 2 3
1712 037/300 Write block 3682 len 24576 data: 3 3 3 3 3 3
1713 038/300 Write block 1561 len 36864 data: 3 3 3 3 3 3 2 2 2
1714 039/300 Read block 4257 len 32768
1715 040/300 Read block 4513 len 8192
1716 041/300 Flush
1717 042/300 Read block 3666 len 8192
1718 043/300 Write block 3522 len 28672 data: 2 2 2 2 2 2 2
1719 044/300 Write block 2769 len 24576 data: 3 2 2 2 2 2
1720 045/300 Read block 3937 len 4096
1721 046/300 Write block 283 len 36864 data: 2 2 2 2 2 2 2 2 2
1722 047/300 Read block 3722 len 8192
1723 048/300 Write block 1379 len 28672 data: 2 4 4 3 2 2 2
1724 049/300 Write block 2786 len 12288 data: 3 3 3
1725 050/300 Write block 4919 len 28672 data: 2 2 2 2 2 2 2
1726 051/300 Read block 1686 len 40960
1727 052/300 Read block 2379 len 24576
1728 053/300 Read block 1715 len 8192
1729 054/300 Flush
1730 055/300 Read block 3245 len 12288
1731 056/300 Write block 4165 len 8192 data: 4 4
1732 057/300 Flush
1733 058/300 Write block 2616 len 40960 data: 3 3 3 3 3 3 3 3 2 3
1734 059/300 Write block 2686 len 16384 data: 2 2 2 2
1735 060/300 Flush
1736 061/300 Read block 1846 len 24576
1737 062/300 Read block 4759 len 32768
1738 063/300 Flush
1739 064/300 Read block 4946 len 24576
1740 065/300 Read block 1909 len 36864
1741 066/300 Read block 3322 len 28672
1742 067/300 Read block 3775 len 12288
1743 068/300 Write block 1993 len 24576 data: 2 2 2 2 3 3
1744 069/300 Read block 661 len 32768
1745 070/300 Read block 1643 len 24576
1746 071/300 Read block 2163 len 8192
1747 072/300 Write block 4828 len 8192 data: 2 2
1748 073/300 Flush
1749 074/300 Write block 4174 len 4096 data: 3
1750 075/300 Read block 3849 len 16384
1751 076/300 Read block 2367 len 32768
1752 077/300 Write block 384 len 16384 data: 2 2 2 2
1753 078/300 Read block 1275 len 36864
1754 079/300 Read block 4058 len 8192
1755 080/300 Write block 4231 len 12288 data: 3 3 3
1756 081/300 Read block 2387 len 20480
1757 082/300 Read block 2045 len 28672
1758 083/300 Read block 1381 len 32768
1759 084/300 Write block 4537 len 28672 data: 2 2 2 2 2 2 2
1760 085/300 Write block 4952 len 4096 data: 2
1761 086/300 Flush
1762 087/300 Write block 4737 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1763 088/300 Read block 1140 len 16384
1764 089/300 Write block 1566 len 28672 data: 4 3 3 3 2 2 2
1765 090/300 Flush
1766 091/300 Read block 4007 len 32768
1767 092/300 Read block 1613 len 32768
1768 093/300 Read block 3031 len 32768
1769 094/300 Write block 4785 len 32768 data: 2 2 2 2 2 2 2 2
1770 095/300 Flush
1771 096/300 Flush
1772 097/300 Flush
1773 098/300 Write block 538 len 4096 data: 3
1774 099/300 Read block 1456 len 28672
1775 100/300 Write block 1875 len 36864 data: 2 2 2 2 2 2 2 2 2
1776 101/300 Write block 4909 len 36864 data: 3 3 3 3 2 2 2 2 2
1777 102/300 Write block 376 len 20480 data: 2 2 2 2 2
1778 103/300 Write block 1978 len 24576 data: 3 3 3 2 2 2
1779 104/300 Read block 4386 len 36864
1780 105/300 Read block 4363 len 32768
1781 106/300 Read block 2774 len 40960
1782 107/300 Read block 3102 len 16384
1783 108/300 Write block 2523 len 36864 data: 2 2 2 2 2 2 2 2 2
1784 109/300 Read block 1249 len 20480
1785 110/300 Read block 4954 len 20480
1786 111/300 Write block 4482 len 24576 data: 3 3 3 3 3 2
1787 112/300 Write block 4732 len 28672 data: 2 2 2 2 2 3 3
1788 113/300 Write block 1537 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1789 114/300 Read block 2534 len 12288
1790 115/300 Flush
1791 116/300 Read block 2736 len 8192
1792 117/300 Write block 3855 len 8192 data: 2 2
1793 118/300 Read block 2406 len 12288
1794 119/300 Read block 1013 len 24576
1795 120/300 Flush
1796 121/300 Write block 54 len 8192 data: 2 2
1797 122/300 Flush
1798 123/300 Read block 50 len 36864
1799 124/300 Read block 1294 len 28672
1800 125/300 Read block 2722 len 8192
1801 126/300 Read block 822 len 32768
1802 127/300 Write block 377 len 20480 data: 3 3 3 3 2
1803 128/300 Write block 3428 len 16384 data: 3 3 3 3
1804 129/300 Flush
1805 130/300 Write block 241 len 32768 data: 2 2 2 2 2 2 2 2
1806 131/300 Write block 1179 len 20480 data: 2 2 2 2 2
1807 132/300 Write block 2320 len 4096 data: 2
1808 133/300 Read block 4271 len 8192
1809 134/300 Read block 2582 len 32768
1810 135/300 Write block 1609 len 36864 data: 4 3 3 3 3 3 3 2 2
1811 136/300 Flush
1812 137/300 Write block 4221 len 32768 data: 3 3 4 4 4 4 3 3
1813 138/300 Write block 2394 len 16384 data: 2 2 2 2
1814 139/300 Write block 1548 len 28672 data: 2 2 2 2 2 2 2
1815 140/300 Read block 2532 len 24576
1816 141/300 Write block 4600 len 24576 data: 3 3 2 2 2 2
1817 142/300 Write block 2723 len 16384 data: 3 3 3 3
1818 143/300 Read block 3849 len 20480
1819 144/300 Write block 369 len 24576 data: 2 2 2 2 2 2
1820 145/300 Flush
1821 146/300 Write block 2585 len 40960 data: 4 4 4 4 3 3 4 3 3 3
1822 147/300 Flush
1823 148/300 Write block 3191 len 32768 data: 4 4 4 3 3 3 2 2
1824 149/300 Read block 4362 len 4096
1825 150/300 Read block 4151 len 24576
1826 151/300 Write block 3231 len 40960 data: 3 2 2 2 2 2 2 2 2 2
1827 152/300 Write block 622 len 32768 data: 2 2 2 2 2 2 2 2
1828 153/300 Write block 1014 len 8192 data: 2 3
1829 154/300 Write block 4611 len 4096 data: 3
1830 155/300 Write block 2465 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1831 156/300 Write block 4639 len 24576 data: 3 3 3 3 3 3
1832 157/300 Read block 731 len 20480
1833 158/300 Read block 3410 len 24576
1834 159/300 Read block 3180 len 40960
1835 160/300 Write block 1205 len 36864 data: 3 2 2 2 2 2 2 2 2
1836 161/300 Write block 4956 len 16384 data: 3 3 3 4
1837 162/300 Write block 4450 len 20480 data: 2 2 2 2 2
1838 163/300 Read block 3329 len 24576
1839 164/300 Write block 2859 len 24576 data: 3 2 2 2 2 2
1840 165/300 Write block 3749 len 12288 data: 2 2 2
1841 166/300 Read block 3573 len 20480
1842 167/300 Write block 3520 len 8192 data: 2 2
1843 168/300 Write block 2309 len 32768 data: 2 2 3 3 3 3 3 3
1844 169/300 Read block 4167 len 40960
1845 170/300 Write block 1264 len 28672 data: 3 3 2 2 2 2 2
1846 171/300 Write block 1564 len 40960 data: 4 4 5 4 4 4 3 3 3 2
1847 172/300 Write block 3390 len 40960 data: 2 2 2 2 2 2 2 2 2 3
1848 173/300 Read block 2576 len 20480
1849 174/300 Flush
1850 175/300 Read block 2584 len 36864
1851 176/300 Write block 3009 len 16384 data: 2 2 2 2
1852 177/300 Read block 4755 len 40960
1853 178/300 Read block 3694 len 4096
1854 179/300 Read block 4838 len 32768
1855 180/300 Write block 831 len 4096 data: 2
1856 181/300 Write block 997 len 36864 data: 2 2 2 2 2 2 2 2 2
1857 182/300 Read block 538 len 4096
1858 183/300 Read block 1261 len 24576
1859 184/300 Flush
1860 185/300 Read block 1307 len 4096
1861 186/300 Read block 1060 len 40960
1862 187/300 Read block 4059 len 36864
1863 188/300 Write block 4433 len 4096 data: 2
1864 189/300 Flush
1865 190/300 Write block 1274 len 32768 data: 2 2 2 2 2 2 2 2
1866 191/300 Write block 4259 len 8192 data: 3 3
1867 192/300 Flush
1868 193/300 Read block 930 len 40960
1869 194/300 Read block 304 len 20480
1870 195/300 Read block 1093 len 24576
1871 196/300 Read block 2299 len 40960
1872 197/300 Read block 1524 len 12288
1873 198/300 Read block 2420 len 20480
1874 199/300 Write block 2821 len 24576 data: 2 2 3 3 2 2
1875 200/300 Write block 4693 len 40960 data: 2 3 2 2 2 2 2 2 2 2
1876 201/300 Flush
1877 202/300 Write block 3395 len 8192 data: 3 3
1878 203/300 Read block 48 len 36864
1879 204/300 Write block 4643 len 32768 data: 4 4 3 2 2 3 3 3
1880 205/300 Write block 2472 len 28672 data: 3 3 3 2 2 2 3
1881 206/300 Read block 800 len 4096
1882 207/300 Write block 3504 len 20480 data: 2 2 2 2 2
1883 208/300 Write block 2538 len 28672 data: 2 2 2 2 2 2 2
1884 209/300 Write block 1063 len 36864 data: 2 2 2 2 2 2 2 2 2
1885 210/300 Read block 2867 len 40960
1886 211/300 Flush
1887 212/300 Read block 387 len 40960
1888 213/300 Read block 1426 len 12288
1889 214/300 Read block 3973 len 36864
1890 215/300 Write block 3444 len 16384 data: 2 2 2 2
1891 216/300 Flush
1892 217/300 Read block 259 len 36864
1893 218/300 Write block 2658 len 32768 data: 3 3 3 3 2 2 2 2
1894 219/300 Write block 543 len 32768 data: 3 3 3 3 3 3 3 3
1895 220/300 Write block 189 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1896 221/300 Flush
1897 222/300 Write block 3153 len 12288 data: 2 2 2
1898 223/300 Read block 4189 len 20480
1899 224/300 Write block 1255 len 12288 data: 3 4 3
1900 225/300 Read block 2861 len 36864
1901 226/300 Write block 382 len 4096 data: 2
1902 227/300 Read block 969 len 36864
1903 228/300 Write block 116 len 28672 data: 2 2 2 2 2 2 2
1904 229/300 Read block 2408 len 20480
1905 230/300 Read block 321 len 28672
1906 231/300 Read block 1624 len 8192
1907 232/300 Write block 169 len 8192 data: 2 2
1908 233/300 Read block 4590 len 24576
1909 234/300 Read block 3317 len 24576
1910 235/300 Read block 712 len 40960
1911 236/300 Write block 4660 len 40960 data: 2 2 2 2 2 2 2 2 2 3
1912 237/300 Write block 53 len 4096 data: 2
1913 238/300 Write block 2960 len 20480 data: 3 3 3 2 2
1914 239/300 Read block 904 len 16384
1915 240/300 Write block 4816 len 40960 data: 2 2 2 2 2 2 2 2 2 3
1916 241/300 Write block 3696 len 24576 data: 2 2 2 2 2 2
1917 242/300 Read block 301 len 12288
1918 243/300 Read block 2969 len 16384
1919 244/300 Read block 1629 len 8192
1920 245/300 Read block 854 len 4096
1921 246/300 Write block 3812 len 16384 data: 3 2 2 2
1922 247/300 Read block 613 len 12288
1923 248/300 Read block 4408 len 12288
1924 249/300 Read block 4142 len 8192
1925 250/300 Read block 749 len 12288
1926 251/300 Write block 155 len 24576 data: 2 2 2 2 2 2
1927 252/300 Write block 3998 len 16384 data: 2 2 2 2
1928 253/300 Read block 1860 len 40960
1929 254/300 Write block 1252 len 12288 data: 2 2 3
1930 255/300 Read block 1748 len 40960
1931 256/300 Flush
1932 257/300 Write block 1070 len 8192 data: 3 3
1933 258/300 Read block 4169 len 28672
1934 259/300 Read block 4324 len 4096
1935 260/300 Read block 775 len 12288
1936 261/300 Read block 3549 len 36864
1937 262/300 Write block 1539 len 24576 data: 3 3 3 3 3 3
1938 263/300 Read block 2253 len 4096
1939 264/300 Read block 4134 len 40960
1940 265/300 Read block 4725 len 20480
1941 266/300 Write block 4034 len 4096 data: 2
1942 267/300 Read block 4709 len 36864
1943 268/300 Read block 2356 len 8192
1944 269/300 Read block 1988 len 24576
1945 270/300 Read block 142 len 12288
1946 271/300 Read block 1257 len 24576
1947 272/300 Write block 1787 len 24576 data: 3 3 3 3 3 3
1948 273/300 Write block 502 len 12288 data: 2 2 2
1949 274/300 Read block 4431 len 40960
1950 275/300 Flush
1951 276/300 Write block 2536 len 40960 data: 2 2 3 3 3 3 3 3 3 2
1952 277/300 Read block 3361 len 24576
1953 278/300 Write block 64 len 20480 data: 2 2 2 2 2
1954 279/300 Write block 2131 len 8192 data: 2 2
1955 280/300 Flush
1956 281/300 Write block 1354 len 8192 data: 2 2
1957 282/300 Write block 34 len 20480 data: 2 2 2 2 2
1958 283/300 Read block 3756 len 20480
1959 284/300 Read block 1142 len 4096
1960 285/300 Write block 4121 len 28672 data: 2 2 2 2 2 2 2
1961 286/300 Flush
1962 287/300 Read block 1196 len 28672
1963 288/300 Write block 2077 len 12288 data: 2 2 2
1964 289/300 Read block 2343 len 28672
1965 290/300 Read block 1886 len 12288
1966 291/300 Read block 1535 len 4096
1967 292/300 Write block 4532 len 32768 data: 2 2 2 2 2 3 3 3
1968 293/300 Read block 2210 len 4096
1969 294/300 Read block 822 len 20480
1970 295/300 Read block 3045 len 8192
1971 296/300 Read block 3043 len 28672
1972 297/300 Read block 3173 len 4096
1973 298/300 Write block 2443 len 16384 data: 2 2 2 2
1974 299/300 Write block 1702 len 28672 data: 2 3 3 3 3 3 3
1975 300/300 Read block 2007 len 16384
1976 Main task has work to do, go find it
1977 got dsc Start(2)
1978 start 2
1979 Feb 12 01:34:54.802 INFO request completed, latency_us: 210, response_code: 204, uri: /start/cid/2, method: POST, req_id: 1cbf9333-8e5c-4123-b35c-b1411b7adeba, remote_addr: 127.0.0.1:60456, local_addr: 127.0.0.1:9998
1980 Replay: started 2, returned:Ok(())
1981 ----------------------------------------------------------------
1982 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
1983 States: Active Active Offline
1984 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
1985 604 Acked 1603 Read 3 Done Done New true
1986 605 Acked 1604 Read 10 Done Done New true
1987 606 Acked 1605 Write 7 Done Done New false
1988 607 Acked 1606 Write 10 Done Done New false
1989 608 Acked 1607 Read 8 Done Done New false
1990 609 Acked 1608 Write 8 Done Done New false
1991 610 Acked 1609 Read 4 Done Done New false
1992 611 Acked 1610 Write 5 Done Done New false
1993 612 Acked 1611 Write 10 Done Done New false
1994 613 Acked 1612 Write 8 Done Done New false
1995 614 Acked 1613 Flush 0 Done Done New false
1996 615 Acked 1614 Read 8 Done Done New false
1997 616 Acked 1615 Read 10 Done Done New false
1998 617 Acked 1616 Write 6 Done Done New false
1999 618 Acked 1617 Flush 0 Done Done New false
2000 619 Acked 1618 Write 7 Done Done New false
2001 620 Acked 1619 Read 2 Done Done New false
2002 621 Acked 1620 Read 6 Done Done New false
2003 622 Acked 1621 Flush 0 Done Done New false
2004 623 Acked 1622 Write 6 Done Done New false
2005 624 Acked 1623 Read 5 Done Done New false
2006 625 Acked 1624 Read 4 Done Done New false
2007 626 Acked 1625 Read 1 Done Done New false
2008 627 Acked 1626 Flush 0 Done Done New false
2009 628 Acked 1627 Read 2 Done Done New false
2010 629 Acked 1628 Flush 0 Done Done New false
2011 630 Acked 1629 Write 5 Done Done New false
2012 631 Acked 1630 Write 4 Done Done New false
2013 632 Acked 1631 Write 10 Done Done New false
2014 633 Acked 1632 Write 1 Done Done New false
2015 634 Acked 1633 Read 6 Done Done New false
2016 635 Acked 1634 Read 4 Done Done New false
2017 636 Acked 1635 Read 8 Done Done New false
2018 637 Acked 1636 Write 4 Done Done New false
2019 638 Acked 1637 Write 6 Done Done New false
2020 639 Acked 1638 Write 8 Done Done New false
2021 640 Acked 1639 Write 6 Done Done New false
2022 641 Acked 1640 Write 9 Done Done New false
2023 642 Acked 1641 Read 8 Done Done New false
2024 643 Acked 1642 Read 2 Done Done New false
2025 644 Acked 1643 Flush 0 Done Done New false
2026 645 Acked 1644 Read 2 Done Done New false
2027 646 Acked 1645 Write 7 Done Done New false
2028 647 Acked 1646 Write 6 Done Done New false
2029 648 Acked 1647 Read 1 Done Done New false
2030 649 Acked 1648 Write 9 Done Done New false
2031 650 Acked 1649 Read 2 Done Done New false
2032 651 Acked 1650 Write 7 Done Done New false
2033 652 Acked 1651 Write 3 Done Done New false
2034 653 Acked 1652 Write 7 Done Done New false
2035 654 Acked 1653 Read 10 Done Done New false
2036 655 Acked 1654 Read 6 Done Done New false
2037 656 Acked 1655 Read 2 Done Done New false
2038 657 Acked 1656 Flush 0 Done Done New false
2039 658 Acked 1657 Read 3 Done Done New false
2040 659 Acked 1658 Write 2 Done Done New false
2041 660 Acked 1659 Flush 0 Done Done New false
2042 661 Acked 1660 Write 10 Done Done New false
2043 662 Acked 1661 Write 4 Done Done New false
2044 663 Acked 1662 Flush 0 Done Done New false
2045 664 Acked 1663 Read 6 Done Done New false
2046 665 Acked 1664 Read 8 Done Done New false
2047 666 Acked 1665 Flush 0 Done Done New false
2048 667 Acked 1666 Read 6 Done Done New false
2049 668 Acked 1667 Read 9 Done Done New false
2050 669 Acked 1668 Read 7 Done Done New false
2051 670 Acked 1669 Read 3 Done Done New false
2052 671 Acked 1670 Write 6 Done Done New false
2053 672 Acked 1671 Read 8 Done Done New false
2054 673 Acked 1672 Read 6 Done Done New false
2055 674 Acked 1673 Read 2 Done Done New false
2056 675 Acked 1674 Write 2 Done Done New false
2057 676 Acked 1675 Flush 0 Done Done New false
2058 677 Acked 1676 Write 1 Done Done New false
2059 678 Acked 1677 Read 4 Done Done New false
2060 679 Acked 1678 Read 8 Done Done New false
2061 680 Acked 1679 Write 4 Done Done New false
2062 681 Acked 1680 Read 9 Done Done New false
2063 682 Acked 1681 Read 2 Done Done New false
2064 683 Acked 1682 Write 3 Done Done New false
2065 684 Acked 1683 Read 5 Done Done New false
2066 685 Acked 1684 Read 7 Done Done New false
2067 686 Acked 1685 Read 8 Done Done New false
2068 687 Acked 1686 Write 7 Done Done New false
2069 688 Acked 1687 Write 1 Done Done New false
2070 689 Acked 1688 Flush 0 Done Done New false
2071 690 Acked 1689 Write 10 Done Done New false
2072 691 Acked 1690 Read 4 Done Done New false
2073 692 Acked 1691 Write 7 Done Done New false
2074 693 Acked 1692 Flush 0 Done Done New false
2075 694 Acked 1693 Read 8 Done Done New false
2076 695 Acked 1694 Read 8 Done Done New false
2077 696 Acked 1695 Read 8 Done Done New false
2078 697 Acked 1696 Write 8 Done Done New false
2079 698 Acked 1697 Flush 0 Done Done New false
2080 699 Acked 1698 Flush 0 Done Done New false
2081 700 Acked 1699 Flush 0 Done Done New false
2082 701 Acked 1700 Write 1 Done Done New false
2083 702 Acked 1701 Read 7 Done Done New false
2084 703 Acked 1702 Write 9 Done Done New false
2085 704 Acked 1703 Write 9 Done Done New false
2086 705 Acked 1704 Write 5 Done Done New false
2087 706 Acked 1705 Write 6 Done Done New false
2088 707 Acked 1706 Read 9 Done Done New false
2089 708 Acked 1707 Read 8 Done Done New false
2090 709 Acked 1708 Read 10 Done Done New false
2091 710 Acked 1709 Read 4 Done Done New false
2092 711 Acked 1710 Write 9 Done Done New false
2093 712 Acked 1711 Read 5 Done Done New false
2094 713 Acked 1712 Read 5 Done Done New false
2095 714 Acked 1713 Write 6 Done Done New false
2096 715 Acked 1714 Write 7 Done Done New false
2097 716 Acked 1715 Write 10 Done Done New false
2098 717 Acked 1716 Read 3 Done Done New false
2099 718 Acked 1717 Flush 0 Done Done New false
2100 719 Acked 1718 Read 2 Done Done New false
2101 720 Acked 1719 Write 2 Done Done New false
2102 721 Acked 1720 Read 3 Done Done New false
2103 722 Acked 1721 Read 6 Done Done New false
2104 723 Acked 1722 Flush 0 Done Done New false
2105 724 Acked 1723 Write 2 Done Done New false
2106 725 Acked 1724 Flush 0 Done Done New false
2107 726 Acked 1725 Read 9 Done Done New false
2108 727 Acked 1726 Read 7 Done Done New false
2109 728 Acked 1727 Read 2 Done Done New false
2110 729 Acked 1728 Read 8 Done Done New false
2111 730 Acked 1729 Write 5 Done Done New false
2112 731 Acked 1730 Write 4 Done Done New false
2113 732 Acked 1731 Flush 0 Done Done New false
2114 733 Acked 1732 Write 8 Done Done New false
2115 734 Acked 1733 Write 5 Done Done New false
2116 735 Acked 1734 Write 1 Done Done New false
2117 736 Acked 1735 Read 2 Done Done New false
2118 737 Acked 1736 Read 8 Done Done New false
2119 738 Acked 1737 Write 9 Done Done New false
2120 739 Acked 1738 Flush 0 Done Done New false
2121 740 Acked 1739 Write 8 Done Done New false
2122 741 Acked 1740 Write 4 Done Done New false
2123 742 Acked 1741 Write 7 Done Done New false
2124 743 Acked 1742 Read 6 Done Done New false
2125 744 Acked 1743 Write 6 Done Done New false
2126 745 Acked 1744 Write 4 Done Done New false
2127 746 Acked 1745 Read 5 Done Done New false
2128 747 Acked 1746 Write 6 Done Done New false
2129 748 Acked 1747 Flush 0 Done Done New false
2130 749 Acked 1748 Write 10 Done Done New false
2131 750 Acked 1749 Flush 0 Done Done New false
2132 751 Acked 1750 Write 8 Done Done New false
2133 752 Acked 1751 Read 1 Done Done New false
2134 753 Acked 1752 Read 6 Done Done New false
2135 754 Acked 1753 Write 10 Done Done New false
2136 755 Acked 1754 Write 8 Done Done New false
2137 756 Acked 1755 Write 2 Done Done New false
2138 757 Acked 1756 Write 1 Done Done New false
2139 758 Acked 1757 Write 10 Done Done New false
2140 759 Acked 1758 Write 6 Done Done New false
2141 760 Acked 1759 Read 5 Done Done New false
2142 761 Acked 1760 Read 6 Done Done New false
2143 762 Acked 1761 Read 10 Done Done New false
2144 763 Acked 1762 Write 9 Done Done New false
2145 764 Acked 1763 Write 4 Done Done New false
2146 765 Acked 1764 Write 5 Done Done New false
2147 766 Acked 1765 Read 6 Done Done New false
2148 767 Acked 1766 Write 6 Done Done New false
2149 768 Acked 1767 Write 3 Done Done New false
2150 769 Acked 1768 Read 5 Done Done New false
2151 770 Acked 1769 Write 2 Done Done New false
2152 771 Acked 1770 Write 8 Done Done New false
2153 772 Acked 1771 Read 10 Done Done New false
2154 773 Acked 1772 Write 7 Done Done New false
2155 774 Acked 1773 Write 10 Done Done New false
2156 775 Acked 1774 Write 10 Done Done New false
2157 776 Acked 1775 Read 5 Done Done New false
2158 777 Acked 1776 Flush 0 Done Done New false
2159 778 Acked 1777 Read 9 Done Done New false
2160 779 Acked 1778 Write 4 Done Done New false
2161 780 Acked 1779 Read 10 Done Done New false
2162 781 Acked 1780 Read 1 Done Done New false
2163 782 Acked 1781 Read 8 Done Done New false
2164 783 Acked 1782 Write 1 Done Done New false
2165 784 Acked 1783 Write 9 Done Done New false
2166 785 Acked 1784 Read 1 Done Done New false
2167 786 Acked 1785 Read 6 Done Done New false
2168 787 Acked 1786 Flush 0 Done Done New false
2169 788 Acked 1787 Read 1 Done Done New false
2170 789 Acked 1788 Read 10 Done Done New false
2171 790 Acked 1789 Read 9 Done Done New false
2172 791 Acked 1790 Write 1 Done Done New false
2173 792 Acked 1791 Flush 0 Done Done New false
2174 793 Acked 1792 Write 8 Done Done New false
2175 794 Acked 1793 Write 2 Done Done New false
2176 795 Acked 1794 Flush 0 Done Done New false
2177 796 Acked 1795 Read 10 Done Done New false
2178 797 Acked 1796 Read 5 Done Done New false
2179 798 Acked 1797 Read 6 Done Done New false
2180 799 Acked 1798 Read 10 Done Done New false
2181 800 Acked 1799 Read 3 Done Done New false
2182 801 Acked 1800 Read 5 Done Done New false
2183 802 Acked 1801 Write 6 Done Done New false
2184 803 Acked 1802 Write 10 Done Done New false
2185 804 Acked 1803 Flush 0 Done Done New false
2186 805 Acked 1804 Write 2 Done Done New false
2187 806 Acked 1805 Read 9 Done Done New false
2188 807 Acked 1806 Write 8 Done Done New false
2189 808 Acked 1807 Write 7 Done Done New false
2190 809 Acked 1808 Read 1 Done Done New false
2191 810 Acked 1809 Write 5 Done Done New false
2192 811 Acked 1810 Write 7 Done Done New false
2193 812 Acked 1811 Write 9 Done Done New false
2194 813 Acked 1812 Read 10 Done Done New false
2195 814 Acked 1813 Flush 0 Done Done New false
2196 815 Acked 1814 Read 10 Done Done New false
2197 816 Acked 1815 Read 3 Done Done New false
2198 817 Acked 1816 Read 9 Done Done New false
2199 818 Acked 1817 Write 4 Done Done New false
2200 819 Acked 1818 Flush 0 Done Done New false
2201 820 Acked 1819 Read 9 Done Done New false
2202 821 Acked 1820 Write 8 Done Done New false
2203 822 Acked 1821 Write 8 Done Done New false
2204 823 Acked 1822 Write 10 Done Done New false
2205 824 Acked 1823 Flush 0 Done Done New false
2206 825 Acked 1824 Write 3 Done Done New false
2207 826 Acked 1825 Read 5 Done Done New false
2208 827 Acked 1826 Write 3 Done Done New false
2209 828 Acked 1827 Read 9 Done Done New false
2210 829 Acked 1828 Write 1 Done Done New false
2211 830 Acked 1829 Read 9 Done Done New false
2212 831 Acked 1830 Flush 0 Done Done New false
2213 832 Acked 1831 Write 7 Done Done New false
2214 833 Acked 1832 Read 5 Done Done New false
2215 834 Acked 1833 Read 7 Done Done New false
2216 835 Acked 1834 Read 2 Done Done New false
2217 836 Acked 1835 Write 2 Done Done New false
2218 837 Acked 1836 Read 6 Done Done New false
2219 838 Acked 1837 Read 6 Done Done New false
2220 839 Acked 1838 Read 10 Done Done New false
2221 840 Acked 1839 Write 10 Done Done New false
2222 841 Acked 1840 Write 1 Done Done New false
2223 842 Acked 1841 Write 5 Done Done New false
2224 843 Acked 1842 Read 4 Done Done New false
2225 844 Acked 1843 Write 10 Done Done New false
2226 845 Acked 1844 Write 6 Done Done New false
2227 846 Acked 1845 Read 3 Done Done New false
2228 847 Acked 1846 Read 4 Done Done New false
2229 848 Acked 1847 Read 2 Done Done New false
2230 849 Acked 1848 Read 1 Done Done New false
2231 850 Acked 1849 Write 4 Done Done New false
2232 851 Acked 1850 Read 3 Done Done New false
2233 852 Acked 1851 Read 3 Done Done New false
2234 853 Acked 1852 Read 2 Done Done New false
2235 854 Acked 1853 Read 3 Done Done New false
2236 855 Acked 1854 Write 6 Done Done New false
2237 856 Acked 1855 Write 4 Done Done New false
2238 857 Acked 1856 Read 10 Done Done New false
2239 858 Acked 1857 Write 3 Done Done New false
2240 859 Acked 1858 Read 10 Done Done New false
2241 860 Acked 1859 Flush 0 Done Done New false
2242 861 Acked 1860 Write 2 Done Done New false
2243 862 Acked 1861 Read 7 Done Done New false
2244 863 Acked 1862 Read 1 Done Done New false
2245 864 Acked 1863 Read 3 Done Done New false
2246 865 Acked 1864 Read 9 Done Done New false
2247 866 Acked 1865 Write 6 Done Done New false
2248 867 Acked 1866 Read 1 Done Done New false
2249 868 Acked 1867 Read 10 Done Done New false
2250 869 Acked 1868 Read 5 Done Done New false
2251 870 Acked 1869 Write 1 Done Done New false
2252 871 Acked 1870 Read 9 Done Done New false
2253 872 Acked 1871 Read 2 Done Done New false
2254 873 Acked 1872 Read 6 Done Done New false
2255 874 Acked 1873 Read 3 Done Done New false
2256 875 Acked 1874 Read 6 Done Done New false
2257 876 Acked 1875 Write 6 Done Done New false
2258 877 Acked 1876 Write 3 Done Done New false
2259 878 Acked 1877 Read 10 Done Done New false
2260 879 Acked 1878 Flush 0 Done Done New false
2261 880 Acked 1879 Write 10 Done Done New false
2262 881 Acked 1880 Read 6 Done Done New false
2263 882 Acked 1881 Write 5 Done Done New false
2264 883 Acked 1882 Write 2 Done Done New false
2265 884 Acked 1883 Flush 0 Done Done New false
2266 885 Acked 1884 Write 2 Done Done New false
2267 886 Acked 1885 Write 5 Done Done New false
2268 887 Acked 1886 Read 5 Done Done New false
2269 888 Acked 1887 Read 1 Done Done New false
2270 889 Acked 1888 Write 7 Done Done New false
2271 890 Acked 1889 Flush 0 Done Done New false
2272 891 Acked 1890 Read 7 Done Done New false
2273 892 Acked 1891 Write 3 Done Done New false
2274 893 Acked 1892 Read 7 Done Done New false
2275 894 Acked 1893 Read 3 Done Done New false
2276 895 Acked 1894 Read 1 Done Done New false
2277 896 Acked 1895 Write 8 Done Done New false
2278 897 Acked 1896 Read 1 Done Done New false
2279 898 Acked 1897 Read 5 Done Done New false
2280 899 Acked 1898 Read 2 Done Done New false
2281 900 Acked 1899 Read 7 Done Done New false
2282 901 Acked 1900 Read 1 Done Done New false
2283 902 Acked 1901 Write 4 Done Done New false
2284 903 Acked 1902 Write 7 Done Done New false
2285 904 Acked 1903 Read 4 Done Done New false
2286 STATES DS:0 DS:1 DS:2 TOTAL
2287 New 0 0 301 301
2288 Sent 0 0 0 0
2289 Done 301 301 0 602
2290 Skipped 0 0 0 0
2291 Error 0 0 0 0
2292 Last Flush: 1889 1889 1602
2293 Downstairs last five completed: 1602 1601 1600 1599 1598
2294 Upstairs last five completed: 904 903 902 901 900
2295 CLIENT: Up:0 ds:301 act:2
22962024-02-12T01:34:55.454ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
22972024-02-12T01:34:55.454ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
22982024-02-12T01:34:55.454ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
22992024-02-12T01:34:55.454ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
23002024-02-12T01:34:55.454ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
23012024-02-12T01:34:55.454ZINFOcrucible: [2] client re-new 302 jobs since flush 1602 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
23022024-02-12T01:34:55.455ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
23032024-02-12T01:34:56.456ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
23042024-02-12T01:34:56.456ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
23052024-02-12T01:34:56.456ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
23062024-02-12T01:34:56.456ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
23072024-02-12T01:34:56.456ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
23082024-02-12T01:34:56.456ZINFOcrucible: [2] client re-new 302 jobs since flush 1602 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
23092024-02-12T01:34:56.456ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
23102024-02-12T01:34:57.457ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
23112024-02-12T01:34:57.457ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
23122024-02-12T01:34:57.457ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
23132024-02-12T01:34:57.457ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
23142024-02-12T01:34:57.457ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
23152024-02-12T01:34:57.457ZINFOcrucible: [2] client re-new 302 jobs since flush 1602 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
23162024-02-12T01:34:57.457ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
23172024-02-12T01:34:58.458ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
23182024-02-12T01:34:58.458ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
23192024-02-12T01:34:58.458ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
23202024-02-12T01:34:58.458ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
23212024-02-12T01:34:58.458ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
23222024-02-12T01:34:58.458ZINFOcrucible: [2] client re-new 302 jobs since flush 1602 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
23232024-02-12T01:34:58.459ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
2324 [8830] got start action
2325 [8830] I am going to restart
2326 Starting downstairs at port 8830
2327 Make output file at "/tmp/dsc/downstairs-8830.txt"
2328 [8830][2] reports Starting
2329 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1167)
2330 [8830][2] reports Running
2331 ----------------------------------------------------------------
2332 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
2333 States: Active Active Offline
2334 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
2335 604 Acked 1603 Read 3 Done Done New true
2336 605 Acked 1604 Read 10 Done Done New true
2337 606 Acked 1605 Write 7 Done Done New true
2338 607 Acked 1606 Write 10 Done Done New true
2339 608 Acked 1607 Read 8 Done Done New true
2340 609 Acked 1608 Write 8 Done Done New true
2341 610 Acked 1609 Read 4 Done Done New true
2342 611 Acked 1610 Write 5 Done Done New true
2343 612 Acked 1611 Write 10 Done Done New true
2344 613 Acked 1612 Write 8 Done Done New true
2345 614 Acked 1613 Flush 0 Done Done New true
2346 615 Acked 1614 Read 8 Done Done New true
2347 616 Acked 1615 Read 10 Done Done New true
2348 617 Acked 1616 Write 6 Done Done New true
2349 618 Acked 1617 Flush 0 Done Done New true
2350 619 Acked 1618 Write 7 Done Done New true
2351 620 Acked 1619 Read 2 Done Done New true
2352 621 Acked 1620 Read 6 Done Done New true
2353 622 Acked 1621 Flush 0 Done Done New true
2354 623 Acked 1622 Write 6 Done Done New true
2355 624 Acked 1623 Read 5 Done Done New true
2356 625 Acked 1624 Read 4 Done Done New true
2357 626 Acked 1625 Read 1 Done Done New true
2358 627 Acked 1626 Flush 0 Done Done New true
2359 628 Acked 1627 Read 2 Done Done New true
2360 629 Acked 1628 Flush 0 Done Done New true
2361 630 Acked 1629 Write 5 Done Done New true
2362 631 Acked 1630 Write 4 Done Done New true
2363 632 Acked 1631 Write 10 Done Done New true
2364 633 Acked 1632 Write 1 Done Done New true
2365 634 Acked 1633 Read 6 Done Done New true
2366 635 Acked 1634 Read 4 Done Done New true
2367 636 Acked 1635 Read 8 Done Done New true
2368 637 Acked 1636 Write 4 Done Done New true
2369 638 Acked 1637 Write 6 Done Done New true
2370 639 Acked 1638 Write 8 Done Done New true
2371 640 Acked 1639 Write 6 Done Done New true
2372 641 Acked 1640 Write 9 Done Done New true
2373 642 Acked 1641 Read 8 Done Done New true
2374 643 Acked 1642 Read 2 Done Done New true
2375 644 Acked 1643 Flush 0 Done Done New true
2376 645 Acked 1644 Read 2 Done Done New true
2377 646 Acked 1645 Write 7 Done Done New true
2378 647 Acked 1646 Write 6 Done Done New true
2379 648 Acked 1647 Read 1 Done Done New true
2380 649 Acked 1648 Write 9 Done Done New true
2381 650 Acked 1649 Read 2 Done Done New true
2382 651 Acked 1650 Write 7 Done Done New true
2383 652 Acked 1651 Write 3 Done Done New true
2384 653 Acked 1652 Write 7 Done Done New true
2385 654 Acked 1653 Read 10 Done Done New true
2386 655 Acked 1654 Read 6 Done Done New true
2387 656 Acked 1655 Read 2 Done Done New true
2388 657 Acked 1656 Flush 0 Done Done New true
2389 658 Acked 1657 Read 3 Done Done New true
2390 659 Acked 1658 Write 2 Done Done New true
2391 660 Acked 1659 Flush 0 Done Done New true
2392 661 Acked 1660 Write 10 Done Done New true
2393 662 Acked 1661 Write 4 Done Done New true
2394 663 Acked 1662 Flush 0 Done Done New true
2395 664 Acked 1663 Read 6 Done Done New true
2396 665 Acked 1664 Read 8 Done Done New true
2397 666 Acked 1665 Flush 0 Done Done New true
2398 667 Acked 1666 Read 6 Done Done New true
2399 668 Acked 1667 Read 9 Done Done New true
2400 669 Acked 1668 Read 7 Done Done New true
2401 670 Acked 1669 Read 3 Done Done New true
2402 671 Acked 1670 Write 6 Done Done New true
2403 672 Acked 1671 Read 8 Done Done New true
2404 673 Acked 1672 Read 6 Done Done New true
2405 674 Acked 1673 Read 2 Done Done New true
2406 675 Acked 1674 Write 2 Done Done New true
2407 676 Acked 1675 Flush 0 Done Done New true
2408 677 Acked 1676 Write 1 Done Done New true
2409 678 Acked 1677 Read 4 Done Done New true
2410 679 Acked 1678 Read 8 Done Done New true
2411 680 Acked 1679 Write 4 Done Done New true
2412 681 Acked 1680 Read 9 Done Done New true
2413 682 Acked 1681 Read 2 Done Done New true
2414 683 Acked 1682 Write 3 Done Done New true
2415 684 Acked 1683 Read 5 Done Done New true
2416 685 Acked 1684 Read 7 Done Done New true
2417 686 Acked 1685 Read 8 Done Done New true
2418 687 Acked 1686 Write 7 Done Done New true
2419 688 Acked 1687 Write 1 Done Done New true
2420 689 Acked 1688 Flush 0 Done Done New true
2421 690 Acked 1689 Write 10 Done Done New true
2422 691 Acked 1690 Read 4 Done Done New true
2423 692 Acked 1691 Write 7 Done Done New true
2424 693 Acked 1692 Flush 0 Done Done New true
2425 694 Acked 1693 Read 8 Done Done New true
2426 695 Acked 1694 Read 8 Done Done New true
2427 696 Acked 1695 Read 8 Done Done New true
2428 697 Acked 1696 Write 8 Done Done New true
2429 698 Acked 1697 Flush 0 Done Done New true
2430 699 Acked 1698 Flush 0 Done Done New true
2431 700 Acked 1699 Flush 0 Done Done New true
2432 701 Acked 1700 Write 1 Done Done New true
2433 702 Acked 1701 Read 7 Done Done New true
2434 703 Acked 1702 Write 9 Done Done New true
2435 704 Acked 1703 Write 9 Done Done New true
2436 705 Acked 1704 Write 5 Done Done New true
2437 706 Acked 1705 Write 6 Done Done New true
2438 707 Acked 1706 Read 9 Done Done New true
2439 708 Acked 1707 Read 8 Done Done New true
2440 709 Acked 1708 Read 10 Done Done New true
2441 710 Acked 1709 Read 4 Done Done New true
2442 711 Acked 1710 Write 9 Done Done New true
2443 712 Acked 1711 Read 5 Done Done New true
2444 713 Acked 1712 Read 5 Done Done New true
2445 714 Acked 1713 Write 6 Done Done New true
2446 715 Acked 1714 Write 7 Done Done New true
2447 716 Acked 1715 Write 10 Done Done New true
2448 717 Acked 1716 Read 3 Done Done New true
2449 718 Acked 1717 Flush 0 Done Done New true
2450 719 Acked 1718 Read 2 Done Done New true
2451 720 Acked 1719 Write 2 Done Done New true
2452 721 Acked 1720 Read 3 Done Done New true
2453 722 Acked 1721 Read 6 Done Done New true
2454 723 Acked 1722 Flush 0 Done Done New true
2455 724 Acked 1723 Write 2 Done Done New true
2456 725 Acked 1724 Flush 0 Done Done New true
2457 726 Acked 1725 Read 9 Done Done New true
2458 727 Acked 1726 Read 7 Done Done New true
2459 728 Acked 1727 Read 2 Done Done New true
2460 729 Acked 1728 Read 8 Done Done New true
2461 730 Acked 1729 Write 5 Done Done New true
2462 731 Acked 1730 Write 4 Done Done New true
2463 732 Acked 1731 Flush 0 Done Done New true
2464 733 Acked 1732 Write 8 Done Done New true
2465 734 Acked 1733 Write 5 Done Done New true
2466 735 Acked 1734 Write 1 Done Done New true
2467 736 Acked 1735 Read 2 Done Done New true
2468 737 Acked 1736 Read 8 Done Done New true
2469 738 Acked 1737 Write 9 Done Done New true
2470 739 Acked 1738 Flush 0 Done Done New true
2471 740 Acked 1739 Write 8 Done Done New true
2472 741 Acked 1740 Write 4 Done Done New true
2473 742 Acked 1741 Write 7 Done Done New true
2474 743 Acked 1742 Read 6 Done Done New true
2475 744 Acked 1743 Write 6 Done Done New true
2476 745 Acked 1744 Write 4 Done Done New true
2477 746 Acked 1745 Read 5 Done Done New true
2478 747 Acked 1746 Write 6 Done Done New true
2479 748 Acked 1747 Flush 0 Done Done New true
2480 749 Acked 1748 Write 10 Done Done New true
2481 750 Acked 1749 Flush 0 Done Done New true
2482 751 Acked 1750 Write 8 Done Done New true
2483 752 Acked 1751 Read 1 Done Done New true
2484 753 Acked 1752 Read 6 Done Done New true
2485 754 Acked 1753 Write 10 Done Done New true
2486 755 Acked 1754 Write 8 Done Done New true
2487 756 Acked 1755 Write 2 Done Done New true
2488 757 Acked 1756 Write 1 Done Done New true
2489 758 Acked 1757 Write 10 Done Done New true
2490 759 Acked 1758 Write 6 Done Done New true
2491 760 Acked 1759 Read 5 Done Done New true
2492 761 Acked 1760 Read 6 Done Done New true
2493 762 Acked 1761 Read 10 Done Done New true
2494 763 Acked 1762 Write 9 Done Done New true
2495 764 Acked 1763 Write 4 Done Done New true
2496 765 Acked 1764 Write 5 Done Done New true
2497 766 Acked 1765 Read 6 Done Done New true
2498 767 Acked 1766 Write 6 Done Done New true
2499 768 Acked 1767 Write 3 Done Done New true
2500 769 Acked 1768 Read 5 Done Done New true
2501 770 Acked 1769 Write 2 Done Done New true
2502 771 Acked 1770 Write 8 Done Done New true
2503 772 Acked 1771 Read 10 Done Done New true
2504 773 Acked 1772 Write 7 Done Done New true
2505 774 Acked 1773 Write 10 Done Done New true
2506 775 Acked 1774 Write 10 Done Done New true
2507 776 Acked 1775 Read 5 Done Done New true
2508 777 Acked 1776 Flush 0 Done Done New true
2509 778 Acked 1777 Read 9 Done Done New true
2510 779 Acked 1778 Write 4 Done Done New true
2511 780 Acked 1779 Read 10 Done Done New true
2512 781 Acked 1780 Read 1 Done Done New true
2513 782 Acked 1781 Read 8 Done Done New true
2514 783 Acked 1782 Write 1 Done Done New true
2515 784 Acked 1783 Write 9 Done Done New true
2516 785 Acked 1784 Read 1 Done Done New true
2517 786 Acked 1785 Read 6 Done Done New true
2518 787 Acked 1786 Flush 0 Done Done New true
2519 788 Acked 1787 Read 1 Done Done New true
2520 789 Acked 1788 Read 10 Done Done New true
2521 790 Acked 1789 Read 9 Done Done New true
2522 791 Acked 1790 Write 1 Done Done New true
2523 792 Acked 1791 Flush 0 Done Done New true
2524 793 Acked 1792 Write 8 Done Done New true
2525 794 Acked 1793 Write 2 Done Done New true
2526 795 Acked 1794 Flush 0 Done Done New true
2527 796 Acked 1795 Read 10 Done Done New true
2528 797 Acked 1796 Read 5 Done Done New true
2529 798 Acked 1797 Read 6 Done Done New true
2530 799 Acked 1798 Read 10 Done Done New true
2531 800 Acked 1799 Read 3 Done Done New true
2532 801 Acked 1800 Read 5 Done Done New true
2533 802 Acked 1801 Write 6 Done Done New true
2534 803 Acked 1802 Write 10 Done Done New true
2535 804 Acked 1803 Flush 0 Done Done New true
2536 805 Acked 1804 Write 2 Done Done New true
2537 806 Acked 1805 Read 9 Done Done New true
2538 807 Acked 1806 Write 8 Done Done New true
2539 808 Acked 1807 Write 7 Done Done New true
2540 809 Acked 1808 Read 1 Done Done New true
2541 810 Acked 1809 Write 5 Done Done New true
2542 811 Acked 1810 Write 7 Done Done New true
2543 812 Acked 1811 Write 9 Done Done New true
2544 813 Acked 1812 Read 10 Done Done New true
2545 814 Acked 1813 Flush 0 Done Done New true
2546 815 Acked 1814 Read 10 Done Done New true
2547 816 Acked 1815 Read 3 Done Done New true
2548 817 Acked 1816 Read 9 Done Done New true
2549 818 Acked 1817 Write 4 Done Done New true
2550 819 Acked 1818 Flush 0 Done Done New true
2551 820 Acked 1819 Read 9 Done Done New true
2552 821 Acked 1820 Write 8 Done Done New true
2553 822 Acked 1821 Write 8 Done Done New true
2554 823 Acked 1822 Write 10 Done Done New true
2555 824 Acked 1823 Flush 0 Done Done New true
2556 825 Acked 1824 Write 3 Done Done New true
2557 826 Acked 1825 Read 5 Done Done New true
2558 827 Acked 1826 Write 3 Done Done New true
2559 828 Acked 1827 Read 9 Done Done New true
2560 829 Acked 1828 Write 1 Done Done New true
2561 830 Acked 1829 Read 9 Done Done New true
2562 831 Acked 1830 Flush 0 Done Done New true
2563 832 Acked 1831 Write 7 Done Done New true
2564 833 Acked 1832 Read 5 Done Done New true
2565 834 Acked 1833 Read 7 Done Done New true
2566 835 Acked 1834 Read 2 Done Done New true
2567 836 Acked 1835 Write 2 Done Done New true
2568 837 Acked 1836 Read 6 Done Done New true
2569 838 Acked 1837 Read 6 Done Done New true
2570 839 Acked 1838 Read 10 Done Done New true
2571 840 Acked 1839 Write 10 Done Done New true
2572 841 Acked 1840 Write 1 Done Done New true
2573 842 Acked 1841 Write 5 Done Done New true
2574 843 Acked 1842 Read 4 Done Done New true
2575 844 Acked 1843 Write 10 Done Done New true
2576 845 Acked 1844 Write 6 Done Done New true
2577 846 Acked 1845 Read 3 Done Done New true
2578 847 Acked 1846 Read 4 Done Done New true
2579 848 Acked 1847 Read 2 Done Done New true
2580 849 Acked 1848 Read 1 Done Done New true
2581 850 Acked 1849 Write 4 Done Done New true
2582 851 Acked 1850 Read 3 Done Done New true
2583 852 Acked 1851 Read 3 Done Done New true
2584 853 Acked 1852 Read 2 Done Done New true
2585 854 Acked 1853 Read 3 Done Done New true
2586 855 Acked 1854 Write 6 Done Done New true
2587 856 Acked 1855 Write 4 Done Done New true
2588 857 Acked 1856 Read 10 Done Done New true
2589 858 Acked 1857 Write 3 Done Done New true
2590 859 Acked 1858 Read 10 Done Done New true
2591 860 Acked 1859 Flush 0 Done Done New true
2592 861 Acked 1860 Write 2 Done Done New true
2593 862 Acked 1861 Read 7 Done Done New true
2594 863 Acked 1862 Read 1 Done Done New true
2595 864 Acked 1863 Read 3 Done Done New true
2596 865 Acked 1864 Read 9 Done Done New true
2597 866 Acked 1865 Write 6 Done Done New true
2598 867 Acked 1866 Read 1 Done Done New true
2599 868 Acked 1867 Read 10 Done Done New true
2600 869 Acked 1868 Read 5 Done Done New true
2601 870 Acked 1869 Write 1 Done Done New true
2602 871 Acked 1870 Read 9 Done Done New true
2603 872 Acked 1871 Read 2 Done Done New true
2604 873 Acked 1872 Read 6 Done Done New true
2605 874 Acked 1873 Read 3 Done Done New true
2606 875 Acked 1874 Read 6 Done Done New true
2607 876 Acked 1875 Write 6 Done Done New true
2608 877 Acked 1876 Write 3 Done Done New true
2609 878 Acked 1877 Read 10 Done Done New true
2610 879 Acked 1878 Flush 0 Done Done New true
2611 880 Acked 1879 Write 10 Done Done New true
2612 881 Acked 1880 Read 6 Done Done New true
2613 882 Acked 1881 Write 5 Done Done New true
2614 883 Acked 1882 Write 2 Done Done New true
2615 884 Acked 1883 Flush 0 Done Done New true
2616 885 Acked 1884 Write 2 Done Done New true
2617 886 Acked 1885 Write 5 Done Done New true
2618 887 Acked 1886 Read 5 Done Done New true
2619 888 Acked 1887 Read 1 Done Done New true
2620 889 Acked 1888 Write 7 Done Done New true
2621 890 Acked 1889 Flush 0 Done Done New true
2622 891 Acked 1890 Read 7 Done Done New true
2623 892 Acked 1891 Write 3 Done Done New true
2624 893 Acked 1892 Read 7 Done Done New true
2625 894 Acked 1893 Read 3 Done Done New true
2626 895 Acked 1894 Read 1 Done Done New true
2627 896 Acked 1895 Write 8 Done Done New true
2628 897 Acked 1896 Read 1 Done Done New true
2629 898 Acked 1897 Read 5 Done Done New true
2630 899 Acked 1898 Read 2 Done Done New true
2631 900 Acked 1899 Read 7 Done Done New true
2632 901 Acked 1900 Read 1 Done Done New true
2633 902 Acked 1901 Write 4 Done Done New true
2634 903 Acked 1902 Write 7 Done Done New true
2635 904 Acked 1903 Read 4 Done Done New true
2636 905 Acked 1904 Flush 0 Done Done New true
2637 STATES DS:0 DS:1 DS:2 TOTAL
2638 New 0 0 302 302
2639 Sent 0 0 0 0
2640 Done 302 302 0 604
2641 Skipped 0 0 0 0
2642 Error 0 0 0 0
2643 Last Flush: 1904 1904 1602
2644 Downstairs last five completed: 1602 1601 1600 1599 1598
2645 Upstairs last five completed: 905 904 903 902 901
2646 CLIENT: Up:0 ds:302 act:2
26472024-02-12T01:34:59.460ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
26482024-02-12T01:34:59.460ZINFOcrucible: ds_connection connected = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
26492024-02-12T01:34:59.461ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
26502024-02-12T01:34:59.461ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
26512024-02-12T01:34:59.461ZINFOcrucible: send last flush ID to this DS: 1602 = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
26522024-02-12T01:34:59.462ZINFOcrucible: Replied this last flush ID: 1602 = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
26532024-02-12T01:34:59.462ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
26542024-02-12T01:34:59.462ZINFOcrucible: [2] Transition from Offline to Replay = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
26552024-02-12T01:34:59.462ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
26562024-02-12T01:34:59.462ZINFOcrucible: [2] Transition from Replay to Active = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
2657 ----------------------------------------------------------------
2658 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
2659 Downstairs last five completed: 1904 1903 1902 1901 1900
2660 Upstairs last five completed: 905 904 903 902 901
2661 CLIENT: Up:0 ds:0 act:3
2662 Replay: All jobs finished, all DS active.
2663 Main task has work to do, go find it
2664 got dsc Stop(0)
2665 stop 0
2666 Feb 12 01:35:02.807 INFO request completed, latency_us: 381, response_code: 204[8810] Got stop action so:false kr:false
2667 , uri: /stop/cid/0, method: POST, req_id: 35ec4424-5fe3-48c5-8617-2ed71dea2825, remote_addr: 127.0.0.1:60456, local_addr: 127.0.0.1:9998
26682024-02-12T01:35:02.808ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
2669 Feb 12 01:35:02.808 INFO request completed, latency_us: 141, response_code: 200, uri: /state/cid/0, method: GET, req_id{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2024-02-12T01:35:02.808342629Z","hostname":"ip-10-150-1-104.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"e1773b3e-b36f-4080-8b13-c0830f0d072e"}
2670 : 6afbfc24-fe81-40bf-8939-7a70d7877f38, remote_addr: 127{"msg":"downstairs task for 0 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-02-12T01:35:02.808391752Z","hostname":"ip-10-150-1-104.us-west-2.compute.internal","pid":1164,"session_id":"e1773b3e-b36f-4080-8b13-c0830f0d072e"}
2671 .0.0.1:60456, local_addr: 127.0.0.1:9998
26722024-02-12T01:35:02.808ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
26732024-02-12T01:35:02.808ZINFOcrucible: [0] client re-new 0 jobs since flush 1904 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
26742024-02-12T01:35:02.808ZINFOcrucible: client task is exiting = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
2675 [8810] Exited with: ExitStatus(unix_wait_status(9))
2676 [8810][0] reports Exit
26772024-02-12T01:35:03.809ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
26782024-02-12T01:35:03.809ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
26792024-02-12T01:35:03.809ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
26802024-02-12T01:35:03.809ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
26812024-02-12T01:35:03.809ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
26822024-02-12T01:35:03.809ZINFOcrucible: [0] client re-new 0 jobs since flush 1904 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
26832024-02-12T01:35:03.809ZINFOcrucible: client task is exiting = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
26842024-02-12T01:35:04.811ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
26852024-02-12T01:35:04.811ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
26862024-02-12T01:35:04.811ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
26872024-02-12T01:35:04.811ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
26882024-02-12T01:35:04.811ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
26892024-02-12T01:35:04.811ZINFOcrucible: [0] client re-new 0 jobs since flush 1904 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
26902024-02-12T01:35:04.811ZINFOcrucible: client task is exiting = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
26912024-02-12T01:35:05.813ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
26922024-02-12T01:35:05.813ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
26932024-02-12T01:35:05.813ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
26942024-02-12T01:35:05.813ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
26952024-02-12T01:35:05.813ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
26962024-02-12T01:35:05.813ZINFOcrucible: [0] client re-new 0 jobs since flush 1904 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
26972024-02-12T01:35:05.813ZINFOcrucible: client task is exiting = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
2698 Feb 12 01:35:06.810 INFO request completed, latency_us: 214, response_code: 200, uri: /state/cid/0, method: GET, req_id: 6ef3aca3-48c7-4215-a116-1ba8d75edacf, remote_addr: 127.0.0.1:60456, local_addr: 127.0.0.1:9998
2699 001/300 Read block 4477 len 36864
2700 002/300 Write block 3048 len 4096 data: 2
2701 003/300 Write block 4454 len 20480 data: 3 2 2 2 2
2702 004/300 Write block 1173 len 8192 data: 4 3
27032024-02-12T01:35:06.814ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
27042024-02-12T01:35:06.814ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
27052024-02-12T01:35:06.814ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
27062024-02-12T01:35:06.814ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
27072024-02-12T01:35:06.814ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
27082024-02-12T01:35:06.814ZINFOcrucible: [0] client re-new 4 jobs since flush 1904 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
27092024-02-12T01:35:06.814ZINFOcrucible: client task is exiting = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
2710 005/300 Write block 4280 len 32768 data: 2 2 2 2 2 2 2 2
2711 006/300 Read block 1643 len 24576
2712 007/300 Read block 2506 len 32768
2713 008/300 Read block 2281 len 28672
2714 009/300 Read block 2809 len 16384
2715 010/300 Read block 889 len 4096
2716 011/300 Write block 3277 len 28672 data: 2 2 2 2 2 2 2
2717 012/300 Read block 1247 len 16384
2718 013/300 Read block 3876 len 24576
2719 014/300 Flush
2720 015/300 Read block 352 len 36864
2721 016/300 Read block 4800 len 36864
2722 017/300 Read block 4167 len 40960
2723 018/300 Write block 3394 len 8192 data: 3 4
2724 019/300 Write block 1850 len 32768 data: 3 3 3 3 3 3 4 3
2725 020/300 Read block 663 len 36864
2726 021/300 Read block 766 len 36864
2727 022/300 Write block 1031 len 16384 data: 2 2 2 2
2728 023/300 Write block 1067 len 4096 data: 3
2729 024/300 Write block 3870 len 24576 data: 3 3 3 3 3 3
2730 025/300 Read block 1162 len 40960
2731 026/300 Read block 3273 len 40960
2732 027/300 Read block 4619 len 4096
2733 028/300 Write block 4678 len 20480 data: 4 3 3 3 3
2734 029/300 Flush
2735 030/300 Read block 1303 len 16384
2736 031/300 Read block 3344 len 8192
2737 032/300 Read block 4858 len 40960
2738 033/300 Read block 849 len 8192
2739 034/300 Write block 78 len 28672 data: 2 2 2 2 2 2 2
2740 035/300 Write block 102 len 36864 data: 4 4 4 4 4 4 3 2 2
2741 036/300 Write block 822 len 12288 data: 2 2 2
2742 037/300 Read block 3439 len 28672
2743 038/300 Read block 1357 len 12288
2744 039/300 Write block 4594 len 36864 data: 2 2 3 3 3 3 4 4 3
2745 040/300 Read block 1676 len 32768
2746 041/300 Read block 1176 len 32768
2747 042/300 Read block 240 len 32768
2748 043/300 Flush
2749 044/300 Write block 4686 len 32768 data: 2 2 2 2 3 2 2 3
2750 045/300 Flush
2751 046/300 Read block 2913 len 12288
2752 047/300 Write block 4714 len 32768 data: 3 3 2 2 3 3 3 3
2753 048/300 Flush
2754 049/300 Read block 768 len 20480
2755 050/300 Write block 3433 len 8192 data: 3 3
2756 051/300 Read block 3237 len 36864
2757 052/300 Read block 3145 len 12288
2758 053/300 Flush
2759 054/300 Flush
2760 055/300 Write block 1476 len 16384 data: 2 2 2 2
2761 056/300 Read block 4575 len 8192
2762 057/300 Read block 298 len 8192
2763 058/300 Flush
2764 059/300 Flush
2765 060/300 Write block 460 len 8192 data: 2 2
2766 061/300 Write block 1750 len 36864 data: 2 2 2 3 3 3 3 3 2
2767 062/300 Read block 1209 len 36864
2768 063/300 Read block 2886 len 8192
2769 064/300 Write block 2383 len 12288 data: 2 2 2
2770 065/300 Read block 1280 len 24576
2771 066/300 Read block 3043 len 24576
2772 067/300 Flush
2773 068/300 Flush
2774 069/300 Write block 4668 len 40960 data: 3 4 3 3 3 3 3 3 4 4
2775 070/300 Read block 1086 len 16384
2776 071/300 Flush
2777 072/300 Read block 863 len 8192
2778 073/300 Read block 1476 len 8192
2779 074/300 Write block 2741 len 40960 data: 2 2 2 2 2 2 2 2 2 3
2780 075/300 Write block 2175 len 28672 data: 3 3 3 3 3 2 2
2781 076/300 Write block 1164 len 8192 data: 2 2
2782 077/300 Write block 3950 len 32768 data: 2 2 2 2 2 2 2 2
2783 078/300 Write block 309 len 32768 data: 4 4 3 3 3 3 3 3
2784 079/300 Write block 925 len 12288 data: 2 2 2
2785 080/300 Read block 4072 len 28672
2786 081/300 Read block 1777 len 4096
2787 082/300 Read block 1851 len 36864
2788 083/300 Flush
2789 084/300 Read block 4453 len 16384
2790 085/300 Read block 2646 len 16384
2791 086/300 Write block 2414 len 16384 data: 2 2 2 2
2792 087/300 Read block 4326 len 16384
2793 088/300 Write block 2444 len 4096 data: 3
2794 089/300 Read block 4073 len 16384
2795 090/300 Write block 780 len 36864 data: 2 2 2 2 2 2 2 2 2
2796 091/300 Write block 3905 len 16384 data: 2 2 2 2
2797 092/300 Read block 486 len 20480
2798 093/300 Flush
2799 094/300 Read block 3892 len 12288
2800 095/300 Read block 4675 len 4096
2801 096/300 Flush
2802 097/300 Write block 1058 len 36864 data: 2 2 2 2 2 3 3 3 3
2803 098/300 Write block 2045 len 16384 data: 3 3 3 3
2804 099/300 Write block 1283 len 28672 data: 2 2 2 2 2 2 2
2805 100/300 Read block 4144 len 40960
2806 101/300 Write block 3701 len 12288 data: 3 2 2
2807 102/300 Read block 2695 len 36864
2808 103/300 Flush
2809 104/300 Read block 4621 len 28672
2810 105/300 Write block 2738 len 4096 data: 2
2811 106/300 Read block 1888 len 32768
2812 107/300 Read block 4520 len 36864
2813 108/300 Flush
2814 109/300 Read block 4569 len 24576
2815 110/300 Write block 3954 len 4096 data: 3
2816 111/300 Read block 1441 len 20480
2817 112/300 Write block 4454 len 8192 data: 4 3
2818 113/300 Read block 3670 len 40960
2819 114/300 Read block 3251 len 36864
2820 115/300 Write block 4472 len 32768 data: 2 2 2 2 2 2 2 2
2821 116/300 Write block 4621 len 20480 data: 2 2 2 2 2
2822 117/300 Write block 1766 len 8192 data: 2 2
2823 118/300 Read block 3205 len 4096
2824 119/300 Read block 3384 len 12288
2825 120/300 Read block 2236 len 16384
2826 121/300 Write block 2834 len 32768 data: 2 2 2 3 3 3 2 2
2827 122/300 Write block 3595 len 24576 data: 2 2 2 2 2 2
2828 123/300 Read block 3231 len 32768
2829 124/300 Write block 3209 len 4096 data: 2
2830 125/300 Read block 3242 len 40960
2831 126/300 Write block 2165 len 20480 data: 2 2 2 2 2
2832 127/300 Write block 3809 len 32768 data: 3 3 3 4 3 3 3 2
2833 128/300 Write block 3899 len 8192 data: 3 3
2834 129/300 Read block 905 len 32768
2835 130/300 Read block 3577 len 8192
2836 131/300 Read block 4340 len 28672
2837 132/300 Write block 1448 len 32768 data: 2 2 2 2 2 2 2 2
2838 133/300 Flush
2839 134/300 Read block 427 len 28672
2840 135/300 Read block 1780 len 28672
2841 136/300 Read block 1722 len 12288
2842 137/300 Flush
2843 138/300 Write block 1450 len 36864 data: 3 3 3 3 3 3 2 2 2
2844 139/300 Write block 2976 len 24576 data: 2 2 2 2 2 2
2845 140/300 Write block 1086 len 4096 data: 2
2846 141/300 Read block 2289 len 8192
2847 142/300 Write block 4754 len 12288 data: 2 2 2
2848 143/300 Flush
2849 144/300 Read block 1464 len 12288
2850 145/300 Write block 2628 len 36864 data: 2 2 2 2 2 2 2 2 2
2851 146/300 Read block 1534 len 20480
2852 147/300 Read block 83 len 8192
2853 148/300 Write block 4476 len 16384 data: 3 3 3 3
2854 149/300 Flush
2855 150/300 Write block 3400 len 20480 data: 3 3 3 4 4
2856 151/300 Read block 1085 len 24576
2857 152/300 Write block 747 len 8192 data: 2 2
2858 153/300 Write block 3407 len 4096 data: 2
2859 154/300 Write block 4006 len 16384 data: 2 2 2 2
2860 155/300 Write block 990 len 8192 data: 2 2
2861 156/300 Read block 2368 len 36864
2862 157/300 Read block 1892 len 8192
2863 158/300 Read block 2322 len 32768
2864 159/300 Read block 2152 len 28672
2865 160/300 Read block 1405 len 32768
2866 161/300 Read block 4892 len 36864
2867 162/300 Read block 3055 len 24576
2868 163/300 Write block 4645 len 16384 data: 4 3 3 4
2869 164/300 Write block 4467 len 36864 data: 2 2 2 2 2 3 3 3 3
2870 165/300 Read block 2652 len 32768
2871 166/300 Read block 2834 len 8192
2872 167/300 Read block 2009 len 28672
2873 168/300 Write block 2525 len 24576 data: 3 3 3 3 3 3
2874 169/300 Read block 1053 len 8192
2875 170/300 Read block 1710 len 12288
2876 171/300 Read block 3234 len 40960
2877 172/300 Read block 3841 len 32768
2878 173/300 Write block 3757 len 16384 data: 2 2 2 2
2879 174/300 Read block 4568 len 40960
2880 175/300 Read block 4793 len 16384
2881 176/300 Write block 3263 len 40960 data: 2 2 2 2 2 2 2 2 2 2
2882 177/300 Write block 248 len 8192 data: 3 2
2883 178/300 Write block 1666 len 12288 data: 2 2 2
2884 179/300 Read block 2920 len 28672
2885 180/300 Read block 3499 len 20480
2886 181/300 Read block 3426 len 40960
2887 182/300 Read block 1872 len 12288
2888 183/300 Read block 587 len 32768
2889 184/300 Write block 2736 len 8192 data: 2 2
2890 185/300 Read block 565 len 40960
2891 186/300 Flush
2892 187/300 Read block 3470 len 8192
2893 188/300 Read block 4455 len 40960
2894 189/300 Write block 602 len 32768 data: 2 2 2 2 2 2 2 2
2895 190/300 Write block 1911 len 32768 data: 2 2 2 2 2 3 4 4
2896 191/300 Read block 2895 len 40960
2897 192/300 Read block 344 len 36864
2898 193/300 Flush
2899 194/300 Read block 2981 len 40960
2900 195/300 Write block 4897 len 40960 data: 2 2 2 2 2 2 2 2 2 2
2901 196/300 Write block 3450 len 28672 data: 2 2 2 2 2 2 2
2902 197/300 Read block 599 len 12288
2903 198/300 Write block 927 len 32768 data: 3 3 3 3 2 2 2 2
2904 199/300 Read block 1960 len 16384
2905 200/300 Write block 1415 len 4096 data: 2
2906 201/300 Write block 3767 len 20480 data: 2 2 2 2 2
2907 202/300 Write block 2596 len 40960 data: 3 3 4 3 3 3 3 3 3 3
2908 203/300 Read block 699 len 32768
2909 204/300 Write block 3518 len 20480 data: 2 2 3 3 3
2910 205/300 Read block 3107 len 24576
2911 206/300 Write block 3662 len 20480 data: 4 4 3 3 3
2912 207/300 Read block 4426 len 12288
2913 208/300 Read block 988 len 36864
2914 209/300 Read block 3635 len 32768
2915 210/300 Write block 2754 len 40960 data: 4 4 4 5 4 3 3 3 3 3
2916 211/300 Read block 3801 len 4096
2917 212/300 Read block 685 len 24576
2918 213/300 Write block 4039 len 8192 data: 3 3
2919 214/300 Write block 2649 len 16384 data: 3 2 2 2
2920 215/300 Write block 4505 len 12288 data: 2 2 2
2921 216/300 Read block 2664 len 20480
2922 217/300 Write block 4601 len 12288 data: 5 4 3
2923 218/300 Write block 711 len 8192 data: 2 2
2924 219/300 Write block 162 len 12288 data: 3 3 3
2925 220/300 Flush
2926 221/300 Read block 601 len 20480
2927 222/300 Read block 833 len 28672
2928 223/300 Write block 2756 len 40960 data: 5 6 5 4 4 4 4 4 2 2
2929 224/300 Write block 1631 len 12288 data: 2 2 2
2930 225/300 Read block 4116 len 20480
2931 226/300 Read block 1419 len 4096
2932 227/300 Write block 4021 len 28672 data: 4 4 3 2 2 2 2
2933 228/300 Write block 4160 len 28672 data: 2 2 3 4 4 5 5
2934 229/300 Write block 4949 len 12288 data: 2 2 2
2935 230/300 Write block 4837 len 16384 data: 2 2 2 2
2936 231/300 Write block 2557 len 28672 data: 2 2 2 2 2 2 2
2937 232/300 Write block 86 len 20480 data: 2 2 2 2 2
2938 233/300 Write block 4135 len 8192 data: 2 2
2939 234/300 Write block 204 len 20480 data: 2 2 2 2 2
2940 235/300 Read block 2683 len 12288
2941 236/300 Read block 3981 len 8192
2942 237/300 Read block 1468 len 12288
2943 238/300 Write block 1806 len 20480 data: 2 2 2 3 3
2944 239/300 Read block 4666 len 24576
2945 240/300 Flush
2946 241/300 Read block 3957 len 24576
2947 242/300 Write block 4249 len 40960 data: 2 2 2 2 2 2 2 2 2 2
2948 243/300 Read block 4043 len 24576
2949 244/300 Read block 118 len 36864
2950 245/300 Read block 2370 len 8192
2951 246/300 Read block 2099 len 8192
2952 247/300 Write block 83 len 20480 data: 3 3 2 3 3
2953 248/300 Write block 3732 len 32768 data: 2 2 2 2 2 2 2 2
2954 249/300 Read block 227 len 32768
2955 250/300 Flush
2956 251/300 Write block 1128 len 40960 data: 2 2 2 2 2 2 2 2 2 2
2957 252/300 Write block 4690 len 16384 data: 4 3 3 4
2958 253/300 Read block 955 len 12288
2959 254/300 Read block 659 len 24576
2960 255/300 Write block 3507 len 20480 data: 3 3 3 3 3
2961 256/300 Read block 5 len 28672
2962 257/300 Read block 2331 len 40960
2963 258/300 Read block 4561 len 28672
2964 259/300 Read block 3081 len 32768
2965 260/300 Write block 40 len 8192 data: 2 2
2966 261/300 Read block 1163 len 8192
2967 262/300 Read block 2315 len 24576
2968 263/300 Write block 777 len 28672 data: 2 2 2 3 3 3 3
2969 264/300 Read block 2216 len 16384
2970 265/300 Read block 509 len 12288
2971 266/300 Read block 2068 len 36864
2972 267/300 Write block 3614 len 8192 data: 2 2
2973 268/300 Read block 3067 len 24576
2974 269/300 Write block 4907 len 40960 data: 2 2 4 4 4 4 3 3 3 3
2975 270/300 Write block 4252 len 36864 data: 3 3 3 3 3 3 3 4 4
2976 271/300 Read block 2967 len 28672
2977 272/300 Read block 1175 len 12288
2978 273/300 Read block 823 len 16384
2979 274/300 Read block 3220 len 20480
2980 275/300 Read block 1759 len 24576
2981 276/300 Read block 4612 len 40960
2982 277/300 Read block 1533 len 28672
2983 278/300 Read block 4783 len 16384
2984 279/300 Flush
2985 280/300 Read block 4261 len 8192
2986 281/300 Read block 4411 len 8192
2987 282/300 Read block 1168 len 40960
2988 283/300 Write block 312 len 12288 data: 4 4 4
2989 284/300 Write block 4875 len 32768 data: 2 2 2 2 2 2 2 2
2990 285/300 Write block 2894 len 4096 data: 2
2991 286/300 Read block 3142 len 16384
2992 287/300 Read block 3652 len 32768
2993 288/300 Write block 460 len 16384 data: 3 3 2 2
2994 289/300 Read block 547 len 36864
2995 290/300 Write block 4066 len 32768 data: 2 2 2 2 2 2 2 2
2996 291/300 Write block 2214 len 40960 data: 3 3 3 3 3 3 2 2 2 2
2997 292/300 Read block 2442 len 4096
2998 293/300 Write block 1699 len 28672 data: 2 2 2 3 4 4 4
2999 294/300 Read block 4874 len 24576
3000 295/300 Read block 1615 len 36864
3001 296/300 Read block 1359 len 20480
3002 297/300 Write block 4914 len 24576 data: 4 4 4 3 2 3
3003 298/300 Write block 3484 len 8192 data: 3 3
3004 299/300 Read block 4765 len 12288
3005 300/300 Write block 232 len 32768 data: 3 2 2 2 2 2 2 2
3006 Main task has work to do, go find it
3007 got dsc Start(0)
3008 start 0
3009 Feb 12 01:35:07.164 INFO request completed, latency_us: 237, response_code: 204, uri: /start/cid/0, method: POST, req_id: 53e70366-bf70-4c77-9113-6f0dfc1452e9, remote_addr: 127.0.0.1:60456, local_addr: 127.0.0.1:9998
3010 Replay: started 0, returned:Ok(())
3011 ----------------------------------------------------------------
3012 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:300
3013 States: Offline Active Active
3014 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
3015 906 Acked 1905 Read 9 New Done Done true
3016 907 Acked 1906 Write 1 New Done Done true
3017 908 Acked 1907 Write 5 New Done Done true
3018 909 Acked 1908 Write 2 New Done Done true
3019 910 Acked 1909 Write 8 New Done Done false
3020 911 Acked 1910 Read 6 New Done Done false
3021 912 Acked 1911 Read 8 New Done Done false
3022 913 Acked 1912 Read 7 New Done Done false
3023 914 Acked 1913 Read 4 New Done Done false
3024 915 Acked 1914 Read 1 New Done Done false
3025 916 Acked 1915 Write 7 New Done Done false
3026 917 Acked 1916 Read 4 New Done Done false
3027 918 Acked 1917 Read 6 New Done Done false
3028 919 Acked 1918 Flush 0 New Done Done false
3029 920 Acked 1919 Read 9 New Done Done false
3030 921 Acked 1920 Read 9 New Done Done false
3031 922 Acked 1921 Read 10 New Done Done false
3032 923 Acked 1922 Write 2 New Done Done false
3033 924 Acked 1923 Write 8 New Done Done false
3034 925 Acked 1924 Read 9 New Done Done false
3035 926 Acked 1925 Read 9 New Done Done false
3036 927 Acked 1926 Write 4 New Done Done false
3037 928 Acked 1927 Write 1 New Done Done false
3038 929 Acked 1928 Write 6 New Done Done false
3039 930 Acked 1929 Read 10 New Done Done false
3040 931 Acked 1930 Read 10 New Done Done false
3041 932 Acked 1931 Read 1 New Done Done false
3042 933 Acked 1932 Write 5 New Done Done false
3043 934 Acked 1933 Flush 0 New Done Done false
3044 935 Acked 1934 Read 4 New Done Done false
3045 936 Acked 1935 Read 2 New Done Done false
3046 937 Acked 1936 Read 10 New Done Done false
3047 938 Acked 1937 Read 2 New Done Done false
3048 939 Acked 1938 Write 7 New Done Done false
3049 940 Acked 1939 Write 9 New Done Done false
3050 941 Acked 1940 Write 3 New Done Done false
3051 942 Acked 1941 Read 7 New Done Done false
3052 943 Acked 1942 Read 3 New Done Done false
3053 944 Acked 1943 Write 9 New Done Done false
3054 945 Acked 1944 Read 8 New Done Done false
3055 946 Acked 1945 Read 8 New Done Done false
3056 947 Acked 1946 Read 8 New Done Done false
3057 948 Acked 1947 Flush 0 New Done Done false
3058 949 Acked 1948 Write 8 New Done Done false
3059 950 Acked 1949 Flush 0 New Done Done false
3060 951 Acked 1950 Read 3 New Done Done false
3061 952 Acked 1951 Write 8 New Done Done false
3062 953 Acked 1952 Flush 0 New Done Done false
3063 954 Acked 1953 Read 5 New Done Done false
3064 955 Acked 1954 Write 2 New Done Done false
3065 956 Acked 1955 Read 9 New Done Done false
3066 957 Acked 1956 Read 3 New Done Done false
3067 958 Acked 1957 Flush 0 New Done Done false
3068 959 Acked 1958 Flush 0 New Done Done false
3069 960 Acked 1959 Write 4 New Done Done false
3070 961 Acked 1960 Read 2 New Done Done false
3071 962 Acked 1961 Read 2 New Done Done false
3072 963 Acked 1962 Flush 0 New Done Done false
3073 964 Acked 1963 Flush 0 New Done Done false
3074 965 Acked 1964 Write 2 New Done Done false
3075 966 Acked 1965 Write 9 New Done Done false
3076 967 Acked 1966 Read 9 New Done Done false
3077 968 Acked 1967 Read 2 New Done Done false
3078 969 Acked 1968 Write 3 New Done Done false
3079 970 Acked 1969 Read 6 New Done Done false
3080 971 Acked 1970 Read 6 New Done Done false
3081 972 Acked 1971 Flush 0 New Done Done false
3082 973 Acked 1972 Flush 0 New Done Done false
3083 974 Acked 1973 Write 10 New Done Done false
3084 975 Acked 1974 Read 4 New Done Done false
3085 976 Acked 1975 Flush 0 New Done Done false
3086 977 Acked 1976 Read 2 New Done Done false
3087 978 Acked 1977 Read 2 New Done Done false
3088 979 Acked 1978 Write 10 New Done Done false
3089 980 Acked 1979 Write 7 New Done Done false
3090 981 Acked 1980 Write 2 New Done Done false
3091 982 Acked 1981 Write 8 New Done Done false
3092 983 Acked 1982 Write 8 New Done Done false
3093 984 Acked 1983 Write 3 New Done Done false
3094 985 Acked 1984 Read 7 New Done Done false
3095 986 Acked 1985 Read 1 New Done Done false
3096 987 Acked 1986 Read 9 New Done Done false
3097 988 Acked 1987 Flush 0 New Done Done false
3098 989 Acked 1988 Read 4 New Done Done false
3099 990 Acked 1989 Read 4 New Done Done false
3100 991 Acked 1990 Write 4 New Done Done false
3101 992 Acked 1991 Read 4 New Done Done false
3102 993 Acked 1992 Write 1 New Done Done false
3103 994 Acked 1993 Read 4 New Done Done false
3104 995 Acked 1994 Write 9 New Done Done false
3105 996 Acked 1995 Write 4 New Done Done false
3106 997 Acked 1996 Read 5 New Done Done false
3107 998 Acked 1997 Flush 0 New Done Done false
3108 999 Acked 1998 Read 3 New Done Done false
3109 1000 Acked 1999 Read 1 New Done Done false
3110 1001 Acked 2000 Flush 0 New Done Done false
3111 1002 Acked 2001 Write 9 New Done Done false
3112 1003 Acked 2002 Write 4 New Done Done false
3113 1004 Acked 2003 Write 7 New Done Done false
3114 1005 Acked 2004 Read 10 New Done Done false
3115 1006 Acked 2005 Write 3 New Done Done false
3116 1007 Acked 2006 Read 9 New Done Done false
3117 1008 Acked 2007 Flush 0 New Done Done false
3118 1009 Acked 2008 Read 7 New Done Done false
3119 1010 Acked 2009 Write 1 New Done Done false
3120 1011 Acked 2010 Read 8 New Done Done false
3121 1012 Acked 2011 Read 9 New Done Done false
3122 1013 Acked 2012 Flush 0 New Done Done false
3123 1014 Acked 2013 Read 6 New Done Done false
3124 1015 Acked 2014 Write 1 New Done Done false
3125 1016 Acked 2015 Read 5 New Done Done false
3126 1017 Acked 2016 Write 2 New Done Done false
3127 1018 Acked 2017 Read 10 New Done Done false
3128 1019 Acked 2018 Read 9 New Done Done false
3129 1020 Acked 2019 Write 8 New Done Done false
3130 1021 Acked 2020 Write 5 New Done Done false
3131 1022 Acked 2021 Write 2 New Done Done false
3132 1023 Acked 2022 Read 1 New Done Done false
3133 1024 Acked 2023 Read 3 New Done Done false
3134 1025 Acked 2024 Read 4 New Done Done false
3135 1026 Acked 2025 Write 8 New Done Done false
3136 1027 Acked 2026 Write 6 New Done Done false
3137 1028 Acked 2027 Read 8 New Done Done false
3138 1029 Acked 2028 Write 1 New Done Done false
3139 1030 Acked 2029 Read 10 New Done Done false
3140 1031 Acked 2030 Write 5 New Done Done false
3141 1032 Acked 2031 Write 8 New Done Done false
3142 1033 Acked 2032 Write 2 New Done Done false
3143 1034 Acked 2033 Read 8 New Done Done false
3144 1035 Acked 2034 Read 2 New Done Done false
3145 1036 Acked 2035 Read 7 New Done Done false
3146 1037 Acked 2036 Write 8 New Done Done false
3147 1038 Acked 2037 Flush 0 New Done Done false
3148 1039 Acked 2038 Read 7 New Done Done false
3149 1040 Acked 2039 Read 7 New Done Done false
3150 1041 Acked 2040 Read 3 New Done Done false
3151 1042 Acked 2041 Flush 0 New Done Done false
3152 1043 Acked 2042 Write 9 New Done Done false
3153 1044 Acked 2043 Write 6 New Done Done false
3154 1045 Acked 2044 Write 1 New Done Done false
3155 1046 Acked 2045 Read 2 New Done Done false
3156 1047 Acked 2046 Write 3 New Done Done false
3157 1048 Acked 2047 Flush 0 New Done Done false
3158 1049 Acked 2048 Read 3 New Done Done false
3159 1050 Acked 2049 Write 9 New Done Done false
3160 1051 Acked 2050 Read 5 New Done Done false
3161 1052 Acked 2051 Read 2 New Done Done false
3162 1053 Acked 2052 Write 4 New Done Done false
3163 1054 Acked 2053 Flush 0 New Done Done false
3164 1055 Acked 2054 Write 5 New Done Done false
3165 1056 Acked 2055 Read 6 New Done Done false
3166 1057 Acked 2056 Write 2 New Done Done false
3167 1058 Acked 2057 Write 1 New Done Done false
3168 1059 Acked 2058 Write 4 New Done Done false
3169 1060 Acked 2059 Write 2 New Done Done false
3170 1061 Acked 2060 Read 9 New Done Done false
3171 1062 Acked 2061 Read 2 New Done Done false
3172 1063 Acked 2062 Read 8 New Done Done false
3173 1064 Acked 2063 Read 7 New Done Done false
3174 1065 Acked 2064 Read 8 New Done Done false
3175 1066 Acked 2065 Read 9 New Done Done false
3176 1067 Acked 2066 Read 6 New Done Done false
3177 1068 Acked 2067 Write 4 New Done Done false
3178 1069 Acked 2068 Write 9 New Done Done false
3179 1070 Acked 2069 Read 8 New Done Done false
3180 1071 Acked 2070 Read 2 New Done Done false
3181 1072 Acked 2071 Read 7 New Done Done false
3182 1073 Acked 2072 Write 6 New Done Done false
3183 1074 Acked 2073 Read 2 New Done Done false
3184 1075 Acked 2074 Read 3 New Done Done false
3185 1076 Acked 2075 Read 10 New Done Done false
3186 1077 Acked 2076 Read 8 New Done Done false
3187 1078 Acked 2077 Write 4 New Done Done false
3188 1079 Acked 2078 Read 10 New Done Done false
3189 1080 Acked 2079 Read 4 New Done Done false
3190 1081 Acked 2080 Write 10 New Done Done false
3191 1082 Acked 2081 Write 2 New Done Done false
3192 1083 Acked 2082 Write 3 New Done Done false
3193 1084 Acked 2083 Read 7 New Done Done false
3194 1085 Acked 2084 Read 5 New Done Done false
3195 1086 Acked 2085 Read 10 New Done Done false
3196 1087 Acked 2086 Read 3 New Done Done false
3197 1088 Acked 2087 Read 8 New Done Done false
3198 1089 Acked 2088 Write 2 New Done Done false
3199 1090 Acked 2089 Read 10 New Done Done false
3200 1091 Acked 2090 Flush 0 New Done Done false
3201 1092 Acked 2091 Read 2 New Done Done false
3202 1093 Acked 2092 Read 10 New Done Done false
3203 1094 Acked 2093 Write 8 New Done Done false
3204 1095 Acked 2094 Write 8 New Done Done false
3205 1096 Acked 2095 Read 10 New Done Done false
3206 1097 Acked 2096 Read 9 New Done Done false
3207 1098 Acked 2097 Flush 0 New Done Done false
3208 1099 Acked 2098 Read 10 New Done Done false
3209 1100 Acked 2099 Write 10 New Done Done false
3210 1101 Acked 2100 Write 7 New Done Done false
3211 1102 Acked 2101 Read 3 New Done Done false
3212 1103 Acked 2102 Write 8 New Done Done false
3213 1104 Acked 2103 Read 4 New Done Done false
3214 1105 Acked 2104 Write 1 New Done Done false
3215 1106 Acked 2105 Write 5 New Done Done false
3216 1107 Acked 2106 Write 10 New Done Done false
3217 1108 Acked 2107 Read 8 New Done Done false
3218 1109 Acked 2108 Write 5 New Done Done false
3219 1110 Acked 2109 Read 6 New Done Done false
3220 1111 Acked 2110 Write 5 New Done Done false
3221 1112 Acked 2111 Read 3 New Done Done false
3222 1113 Acked 2112 Read 9 New Done Done false
3223 1114 Acked 2113 Read 8 New Done Done false
3224 1115 Acked 2114 Write 10 New Done Done false
3225 1116 Acked 2115 Read 1 New Done Done false
3226 1117 Acked 2116 Read 6 New Done Done false
3227 1118 Acked 2117 Write 2 New Done Done false
3228 1119 Acked 2118 Write 4 New Done Done false
3229 1120 Acked 2119 Write 3 New Done Done false
3230 1121 Acked 2120 Read 5 New Done Done false
3231 1122 Acked 2121 Write 3 New Done Done false
3232 1123 Acked 2122 Write 2 New Done Done false
3233 1124 Acked 2123 Write 3 New Done Done false
3234 1125 Acked 2124 Flush 0 New Done Done false
3235 1126 Acked 2125 Read 5 New Done Done false
3236 1127 Acked 2126 Read 7 New Done Done false
3237 1128 Acked 2127 Write 10 New Done Done false
3238 1129 Acked 2128 Write 3 New Done Done false
3239 1130 Acked 2129 Read 5 New Done Done false
3240 1131 Acked 2130 Read 1 New Done Done false
3241 1132 Acked 2131 Write 7 New Done Done false
3242 1133 Acked 2132 Write 7 New Done Done false
3243 1134 Acked 2133 Write 3 New Done Done false
3244 1135 Acked 2134 Write 4 New Done Done false
3245 1136 Acked 2135 Write 7 New Done Done false
3246 1137 Acked 2136 Write 5 New Done Done false
3247 1138 Acked 2137 Write 2 New Done Done false
3248 1139 Acked 2138 Write 5 New Done Done false
3249 1140 Acked 2139 Read 3 New Done Done false
3250 1141 Acked 2140 Read 2 New Done Done false
3251 1142 Acked 2141 Read 3 New Done Done false
3252 1143 Acked 2142 Write 5 New Done Done false
3253 1144 Acked 2143 Read 6 New Done Done false
3254 1145 Acked 2144 Flush 0 New Done Done false
3255 1146 Acked 2145 Read 6 New Done Done false
3256 1147 Acked 2146 Write 10 New Done Done false
3257 1148 Acked 2147 Read 6 New Done Done false
3258 1149 Acked 2148 Read 9 New Done Done false
3259 1150 Acked 2149 Read 2 New Done Done false
3260 1151 Acked 2150 Read 2 New Done Done false
3261 1152 Acked 2151 Write 5 New Done Done false
3262 1153 Acked 2152 Write 8 New Done Done false
3263 1154 Acked 2153 Read 8 New Done Done false
3264 1155 Acked 2154 Flush 0 New Done Done false
3265 1156 Acked 2155 Write 10 New Done Done false
3266 1157 Acked 2156 Write 4 New Done Done false
3267 1158 Acked 2157 Read 3 New Done Done false
3268 1159 Acked 2158 Read 6 New Done Done false
3269 1160 Acked 2159 Write 5 New Done Done false
3270 1161 Acked 2160 Read 7 New Done Done false
3271 1162 Acked 2161 Read 10 New Done Done false
3272 1163 Acked 2162 Read 7 New Done Done false
3273 1164 Acked 2163 Read 8 New Done Done false
3274 1165 Acked 2164 Write 2 New Done Done false
3275 1166 Acked 2165 Read 2 New Done Done false
3276 1167 Acked 2166 Read 6 New Done Done false
3277 1168 Acked 2167 Write 7 New Done Done false
3278 1169 Acked 2168 Read 4 New Done Done false
3279 1170 Acked 2169 Read 3 New Done Done false
3280 1171 Acked 2170 Read 9 New Done Done false
3281 1172 Acked 2171 Write 2 New Done Done false
3282 1173 Acked 2172 Read 6 New Done Done false
3283 1174 Acked 2173 Write 10 New Done Done false
3284 1175 Acked 2174 Write 9 New Done Done false
3285 1176 Acked 2175 Read 7 New Done Done false
3286 1177 Acked 2176 Read 3 New Done Done false
3287 1178 Acked 2177 Read 4 New Done Done false
3288 1179 Acked 2178 Read 5 New Done Done false
3289 1180 Acked 2179 Read 6 New Done Done false
3290 1181 Acked 2180 Read 10 New Done Done false
3291 1182 Acked 2181 Read 7 New Done Done false
3292 1183 Acked 2182 Read 4 New Done Done false
3293 1184 Acked 2183 Flush 0 New Done Done false
3294 1185 Acked 2184 Read 2 New Done Done false
3295 1186 Acked 2185 Read 2 New Done Done false
3296 1187 Acked 2186 Read 10 New Done Done false
3297 1188 Acked 2187 Write 3 New Done Done false
3298 1189 Acked 2188 Write 8 New Done Done false
3299 1190 Acked 2189 Write 1 New Done Done false
3300 1191 Acked 2190 Read 4 New Done Done false
3301 1192 Acked 2191 Read 8 New Done Done false
3302 1193 Acked 2192 Write 4 New Done Done false
3303 1194 Acked 2193 Read 9 New Done Done false
3304 1195 Acked 2194 Write 8 New Done Done false
3305 1196 Acked 2195 Write 10 New Done Done false
3306 1197 Acked 2196 Read 1 New Done Done false
3307 1198 Acked 2197 Write 7 New Done Done false
3308 1199 Acked 2198 Read 6 New Done Done false
3309 1200 Acked 2199 Read 9 New Done Done false
3310 1201 Acked 2200 Read 5 New Done Done false
3311 1202 Acked 2201 Write 6 New Done Done false
3312 1203 Acked 2202 Write 2 New Done Done false
3313 1204 Acked 2203 Read 3 New Done Done false
3314 1205 Acked 2204 Write 8 New Sent Sent false
3315 STATES DS:0 DS:1 DS:2 TOTAL
3316 New 300 0 0 300
3317 Sent 0 1 1 2
3318 Done 0 299 299 598
3319 Skipped 0 0 0 0
3320 Error 0 0 0 0
3321 Last Flush: 1904 2183 2183
3322 Downstairs last five completed: 1904 1903 1902 1901 1900
3323 Upstairs last five completed: 1205 1204 1203 1202 1201
3324 CLIENT: Up:0 ds:300 act:2
33252024-02-12T01:35:07.815ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
33262024-02-12T01:35:07.815ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
33272024-02-12T01:35:07.815ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
33282024-02-12T01:35:07.815ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
33292024-02-12T01:35:07.815ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
33302024-02-12T01:35:07.815ZINFOcrucible: [0] client re-new 301 jobs since flush 1904 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
33312024-02-12T01:35:07.816ZINFOcrucible: client task is exiting = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
33322024-02-12T01:35:08.816ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
33332024-02-12T01:35:08.816ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
33342024-02-12T01:35:08.816ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
33352024-02-12T01:35:08.816ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
33362024-02-12T01:35:08.816ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
33372024-02-12T01:35:08.816ZINFOcrucible: [0] client re-new 301 jobs since flush 1904 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
33382024-02-12T01:35:08.816ZINFOcrucible: client task is exiting = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
33392024-02-12T01:35:09.818ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
33402024-02-12T01:35:09.818ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
33412024-02-12T01:35:09.818ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
33422024-02-12T01:35:09.818ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
33432024-02-12T01:35:09.818ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
33442024-02-12T01:35:09.818ZINFOcrucible: [0] client re-new 301 jobs since flush 1904 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
33452024-02-12T01:35:09.818ZINFOcrucible: client task is exiting = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
33462024-02-12T01:35:10.819ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
33472024-02-12T01:35:10.819ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
33482024-02-12T01:35:10.819ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
33492024-02-12T01:35:10.819ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
33502024-02-12T01:35:10.819ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
33512024-02-12T01:35:10.819ZINFOcrucible: [0] client re-new 301 jobs since flush 1904 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
33522024-02-12T01:35:10.819ZINFOcrucible: client task is exiting = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
3353 ----------------------------------------------------------------
3354 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
3355 States: Offline Active Active
3356 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
3357 906 Acked 1905 Read 9 New Done Done true
3358 907 Acked 1906 Write 1 New Done Done true
3359 908 Acked 1907 Write 5 New Done Done true
3360 909 Acked 1908 Write 2 New Done Done true
3361 910 Acked 1909 Write 8 New Done Done true
3362 911 Acked 1910 Read 6 New Done Done true
3363 912 Acked 1911 Read 8 New Done Done true
3364 913 Acked 1912 Read 7 New Done Done true
3365 914 Acked 1913 Read 4 New Done Done true
3366 915 Acked 1914 Read 1 New Done Done true
3367 916 Acked 1915 Write 7 New Done Done true
3368 917 Acked 1916 Read 4 New Done Done true
3369 918 Acked 1917 Read 6 New Done Done true
3370 919 Acked 1918 Flush 0 New Done Done true
3371 920 Acked 1919 Read 9 New Done Done true
3372 921 Acked 1920 Read 9 New Done Done true
3373 922 Acked 1921 Read 10 New Done Done true
3374 923 Acked 1922 Write 2 New Done Done true
3375 924 Acked 1923 Write 8 New Done Done true
3376 925 Acked 1924 Read 9 New Done Done true
3377 926 Acked 1925 Read 9 New Done Done true
3378 927 Acked 1926 Write 4 New Done Done true
3379 928 Acked 1927 Write 1 New Done Done true
3380 929 Acked 1928 Write 6 New Done Done true
3381 930 Acked 1929 Read 10 New Done Done true
3382 931 Acked 1930 Read 10 New Done Done true
3383 932 Acked 1931 Read 1 New Done Done true
3384 933 Acked 1932 Write 5 New Done Done true
3385 934 Acked 1933 Flush 0 New Done Done true
3386 935 Acked 1934 Read 4 New Done Done true
3387 936 Acked 1935 Read 2 New Done Done true
3388 937 Acked 1936 Read 10 New Done Done true
3389 938 Acked 1937 Read 2 New Done Done true
3390 939 Acked 1938 Write 7 New Done Done true
3391 940 Acked 1939 Write 9 New Done Done true
3392 941 Acked 1940 Write 3 New Done Done true
3393 942 Acked 1941 Read 7 New Done Done true
3394 943 Acked 1942 Read 3 New Done Done true
3395 944 Acked 1943 Write 9 New Done Done true
3396 945 Acked 1944 Read 8 New Done Done true
3397 946 Acked 1945 Read 8 New Done Done true
3398 947 Acked 1946 Read 8 New Done Done true
3399 948 Acked 1947 Flush 0 New Done Done true
3400 949 Acked 1948 Write 8 New Done Done true
3401 950 Acked 1949 Flush 0 New Done Done true
3402 951 Acked 1950 Read 3 New Done Done true
3403 952 Acked 1951 Write 8 New Done Done true
3404 953 Acked 1952 Flush 0 New Done Done true
3405 954 Acked 1953 Read 5 New Done Done true
3406 955 Acked 1954 Write 2 New Done Done true
3407 956 Acked 1955 Read 9 New Done Done true
3408 957 Acked 1956 Read 3 New Done Done true
3409 958 Acked 1957 Flush 0 New Done Done true
3410 959 Acked 1958 Flush 0 New Done Done true
3411 960 Acked 1959 Write 4 New Done Done true
3412 961 Acked 1960 Read 2 New Done Done true
3413 962 Acked 1961 Read 2 New Done Done true
3414 963 Acked 1962 Flush 0 New Done Done true
3415 964 Acked 1963 Flush 0 New Done Done true
3416 965 Acked 1964 Write 2 New Done Done true
3417 966 Acked 1965 Write 9 New Done Done true
3418 967 Acked 1966 Read 9 New Done Done true
3419 968 Acked 1967 Read 2 New Done Done true
3420 969 Acked 1968 Write 3 New Done Done true
3421 970 Acked 1969 Read 6 New Done Done true
3422 971 Acked 1970 Read 6 New Done Done true
3423 972 Acked 1971 Flush 0 New Done Done true
3424 973 Acked 1972 Flush 0 New Done Done true
3425 974 Acked 1973 Write 10 New Done Done true
3426 975 Acked 1974 Read 4 New Done Done true
3427 976 Acked 1975 Flush 0 New Done Done true
3428 977 Acked 1976 Read 2 New Done Done true
3429 978 Acked 1977 Read 2 New Done Done true
3430 979 Acked 1978 Write 10 New Done Done true
3431 980 Acked 1979 Write 7 New Done Done true
3432 981 Acked 1980 Write 2 New Done Done true
3433 982 Acked 1981 Write 8 New Done Done true
3434 983 Acked 1982 Write 8 New Done Done true
3435 984 Acked 1983 Write 3 New Done Done true
3436 985 Acked 1984 Read 7 New Done Done true
3437 986 Acked 1985 Read 1 New Done Done true
3438 987 Acked 1986 Read 9 New Done Done true
3439 988 Acked 1987 Flush 0 New Done Done true
3440 989 Acked 1988 Read 4 New Done Done true
3441 990 Acked 1989 Read 4 New Done Done true
3442 991 Acked 1990 Write 4 New Done Done true
3443 992 Acked 1991 Read 4 New Done Done true
3444 993 Acked 1992 Write 1 New Done Done true
3445 994 Acked 1993 Read 4 New Done Done true
3446 995 Acked 1994 Write 9 New Done Done true
3447 996 Acked 1995 Write 4 New Done Done true
3448 997 Acked 1996 Read 5 New Done Done true
3449 998 Acked 1997 Flush 0 New Done Done true
3450 999 Acked 1998 Read 3 New Done Done true
3451 1000 Acked 1999 Read 1 New Done Done true
3452 1001 Acked 2000 Flush 0 New Done Done true
3453 1002 Acked 2001 Write 9 New Done Done true
3454 1003 Acked 2002 Write 4 New Done Done true
3455 1004 Acked 2003 Write 7 New Done Done true
3456 1005 Acked 2004 Read 10 New Done Done true
3457 1006 Acked 2005 Write 3 New Done Done true
3458 1007 Acked 2006 Read 9 New Done Done true
3459 1008 Acked 2007 Flush 0 New Done Done true
3460 1009 Acked 2008 Read 7 New Done Done true
3461 1010 Acked 2009 Write 1 New Done Done true
3462 1011 Acked 2010 Read 8 New Done Done true
3463 1012 Acked 2011 Read 9 New Done Done true
3464 1013 Acked 2012 Flush 0 New Done Done true
3465 1014 Acked 2013 Read 6 New Done Done true
3466 1015 Acked 2014 Write 1 New Done Done true
3467 1016 Acked 2015 Read 5 New Done Done true
3468 1017 Acked 2016 Write 2 New Done Done true
3469 1018 Acked 2017 Read 10 New Done Done true
3470 1019 Acked 2018 Read 9 New Done Done true
3471 1020 Acked 2019 Write 8 New Done Done true
3472 1021 Acked 2020 Write 5 New Done Done true
3473 1022 Acked 2021 Write 2 New Done Done true
3474 1023 Acked 2022 Read 1 New Done Done true
3475 1024 Acked 2023 Read 3 New Done Done true
3476 1025 Acked 2024 Read 4 New Done Done true
3477 1026 Acked 2025 Write 8 New Done Done true
3478 1027 Acked 2026 Write 6 New Done Done true
3479 1028 Acked 2027 Read 8 New Done Done true
3480 1029 Acked 2028 Write 1 New Done Done true
3481 1030 Acked 2029 Read 10 New Done Done true
3482 1031 Acked 2030 Write 5 New Done Done true
3483 1032 Acked 2031 Write 8 New Done Done true
3484 1033 Acked 2032 Write 2 New Done Done true
3485 1034 Acked 2033 Read 8 New Done Done true
3486 1035 Acked 2034 Read 2 New Done Done true
3487 1036 Acked 2035 Read 7 New Done Done true
3488 1037 Acked 2036 Write 8 New Done Done true
3489 1038 Acked 2037 Flush 0 New Done Done true
3490 1039 Acked 2038 Read 7 New Done Done true
3491 1040 Acked 2039 Read 7 New Done Done true
3492 1041 Acked 2040 Read 3 New Done Done true
3493 1042 Acked 2041 Flush 0 New Done Done true
3494 1043 Acked 2042 Write 9 New Done Done true
3495 1044 Acked 2043 Write 6 New Done Done true
3496 1045 Acked 2044 Write 1 New Done Done true
3497 1046 Acked 2045 Read 2 New Done Done true
3498 1047 Acked 2046 Write 3 New Done Done true
3499 1048 Acked 2047 Flush 0 New Done Done true
3500 1049 Acked 2048 Read 3 New Done Done true
3501 1050 Acked 2049 Write 9 New Done Done true
3502 1051 Acked 2050 Read 5 New Done Done true
3503 1052 Acked 2051 Read 2 New Done Done true
3504 1053 Acked 2052 Write 4 New Done Done true
3505 1054 Acked 2053 Flush 0 New Done Done true
3506 1055 Acked 2054 Write 5 New Done Done true
3507 1056 Acked 2055 Read 6 New Done Done true
3508 1057 Acked 2056 Write 2 New Done Done true
3509 1058 Acked 2057 Write 1 New Done Done true
3510 1059 Acked 2058 Write 4 New Done Done true
3511 1060 Acked 2059 Write 2 New Done Done true
3512 1061 Acked 2060 Read 9 New Done Done true
3513 1062 Acked 2061 Read 2 New Done Done true
3514 1063 Acked 2062 Read 8 New Done Done true
3515 1064 Acked 2063 Read 7 New Done Done true
3516 1065 Acked 2064 Read 8 New Done Done true
3517 1066 Acked 2065 Read 9 New Done Done true
3518 1067 Acked 2066 Read 6 New Done Done true
3519 1068 Acked 2067 Write 4 New Done Done true
3520 1069 Acked 2068 Write 9 New Done Done true
3521 1070 Acked 2069 Read 8 New Done Done true
3522 1071 Acked 2070 Read 2 New Done Done true
3523 1072 Acked 2071 Read 7 New Done Done true
3524 1073 Acked 2072 Write 6 New Done Done true
3525 1074 Acked 2073 Read 2 New Done Done true
3526 1075 Acked 2074 Read 3 New Done Done true
3527 1076 Acked 2075 Read 10 New Done Done true
3528 1077 Acked 2076 Read 8 New Done Done true
3529 1078 Acked 2077 Write 4 New Done Done true
3530 1079 Acked 2078 Read 10 New Done Done true
3531 1080 Acked 2079 Read 4 New Done Done true
3532 1081 Acked 2080 Write 10 New Done Done true
3533 1082 Acked 2081 Write 2 New Done Done true
3534 1083 Acked 2082 Write 3 New Done Done true
3535 1084 Acked 2083 Read 7 New Done Done true
3536 1085 Acked 2084 Read 5 New Done Done true
3537 1086 Acked 2085 Read 10 New Done Done true
3538 1087 Acked 2086 Read 3 New Done Done true
3539 1088 Acked 2087 Read 8 New Done Done true
3540 1089 Acked 2088 Write 2 New Done Done true
3541 1090 Acked 2089 Read 10 New Done Done true
3542 1091 Acked 2090 Flush 0 New Done Done true
3543 1092 Acked 2091 Read 2 New Done Done true
3544 1093 Acked 2092 Read 10 New Done Done true
3545 1094 Acked 2093 Write 8 New Done Done true
3546 1095 Acked 2094 Write 8 New Done Done true
3547 1096 Acked 2095 Read 10 New Done Done true
3548 1097 Acked 2096 Read 9 New Done Done true
3549 1098 Acked 2097 Flush 0 New Done Done true
3550 1099 Acked 2098 Read 10 New Done Done true
3551 1100 Acked 2099 Write 10 New Done Done true
3552 1101 Acked 2100 Write 7 New Done Done true
3553 1102 Acked 2101 Read 3 New Done Done true
3554 1103 Acked 2102 Write 8 New Done Done true
3555 1104 Acked 2103 Read 4 New Done Done true
3556 1105 Acked 2104 Write 1 New Done Done true
3557 1106 Acked 2105 Write 5 New Done Done true
3558 1107 Acked 2106 Write 10 New Done Done true
3559 1108 Acked 2107 Read 8 New Done Done true
3560 1109 Acked 2108 Write 5 New Done Done true
3561 1110 Acked 2109 Read 6 New Done Done true
3562 1111 Acked 2110 Write 5 New Done Done true
3563 1112 Acked 2111 Read 3 New Done Done true
3564 1113 Acked 2112 Read 9 New Done Done true
3565 1114 Acked 2113 Read 8 New Done Done true
3566 1115 Acked 2114 Write 10 New Done Done true
3567 1116 Acked 2115 Read 1 New Done Done true
3568 1117 Acked 2116 Read 6 New Done Done true
3569 1118 Acked 2117 Write 2 New Done Done true
3570 1119 Acked 2118 Write 4 New Done Done true
3571 1120 Acked 2119 Write 3 New Done Done true
3572 1121 Acked 2120 Read 5 New Done Done true
3573 1122 Acked 2121 Write 3 New Done Done true
3574 1123 Acked 2122 Write 2 New Done Done true
3575 1124 Acked 2123 Write 3 New Done Done true
3576 1125 Acked 2124 Flush 0 New Done Done true
3577 1126 Acked 2125 Read 5 New Done Done true
3578 1127 Acked 2126 Read 7 New Done Done true
3579 1128 Acked 2127 Write 10 New Done Done true
3580 1129 Acked 2128 Write 3 New Done Done true
3581 1130 Acked 2129 Read 5 New Done Done true
3582 1131 Acked 2130 Read 1 New Done Done true
3583 1132 Acked 2131 Write 7 New Done Done true
3584 1133 Acked 2132 Write 7 New Done Done true
3585 1134 Acked 2133 Write 3 New Done Done true
3586 1135 Acked 2134 Write 4 New Done Done true
3587 1136 Acked 2135 Write 7 New Done Done true
3588 1137 Acked 2136 Write 5 New Done Done true
3589 1138 Acked 2137 Write 2 New Done Done true
3590 1139 Acked 2138 Write 5 New Done Done true
3591 1140 Acked 2139 Read 3 New Done Done true
3592 1141 Acked 2140 Read 2 New Done Done true
3593 1142 Acked 2141 Read 3 New Done Done true
3594 1143 Acked 2142 Write 5 New Done Done true
3595 1144 Acked 2143 Read 6 New Done Done true
3596 1145 Acked 2144 Flush 0 New Done Done true
3597 1146 Acked 2145 Read 6 New Done Done true
3598 1147 Acked 2146 Write 10 New Done Done true
3599 1148 Acked 2147 Read 6 New Done Done true
3600 1149 Acked 2148 Read 9 New Done Done true
3601 1150 Acked 2149 Read 2 New Done Done true
3602 1151 Acked 2150 Read 2 New Done Done true
3603 1152 Acked 2151 Write 5 New Done Done true
3604 1153 Acked 2152 Write 8 New Done Done true
3605 1154 Acked 2153 Read 8 New Done Done true
3606 1155 Acked 2154 Flush 0 New Done Done true
3607 1156 Acked 2155 Write 10 New Done Done true
3608 1157 Acked 2156 Write 4 New Done Done true
3609 1158 Acked 2157 Read 3 New Done Done true
3610 1159 Acked 2158 Read 6 New Done Done true
3611 1160 Acked 2159 Write 5 New Done Done true
3612 1161 Acked 2160 Read 7 New Done Done true
3613 1162 Acked 2161 Read 10 New Done Done true
3614 1163 Acked 2162 Read 7 New Done Done true
3615 1164 Acked 2163 Read 8 New Done Done true
3616 1165 Acked 2164 Write 2 New Done Done true
3617 1166 Acked 2165 Read 2 New Done Done true
3618 1167 Acked 2166 Read 6 New Done Done true
3619 1168 Acked 2167 Write 7 New Done Done true
3620 1169 Acked 2168 Read 4 New Done Done true
3621 1170 Acked 2169 Read 3 New Done Done true
3622 1171 Acked 2170 Read 9 New Done Done true
3623 1172 Acked 2171 Write 2 New Done Done true
3624 1173 Acked 2172 Read 6 New Done Done true
3625 1174 Acked 2173 Write 10 New Done Done true
3626 1175 Acked 2174 Write 9 New Done Done true
3627 1176 Acked 2175 Read 7 New Done Done true
3628 1177 Acked 2176 Read 3 New Done Done true
3629 1178 Acked 2177 Read 4 New Done Done true
3630 1179 Acked 2178 Read 5 New Done Done true
3631 1180 Acked 2179 Read 6 New Done Done true
3632 1181 Acked 2180 Read 10 New Done Done true
3633 1182 Acked 2181 Read 7 New Done Done true
3634 1183 Acked 2182 Read 4 New Done Done true
3635 1184 Acked 2183 Flush 0 New Done Done true
3636 1185 Acked 2184 Read 2 New Done Done true
3637 1186 Acked 2185 Read 2 New Done Done true
3638 1187 Acked 2186 Read 10 New Done Done true
3639 1188 Acked 2187 Write 3 New Done Done true
3640 1189 Acked 2188 Write 8 New Done Done true
3641 1190 Acked 2189 Write 1 New Done Done true
3642 1191 Acked 2190 Read 4 New Done Done true
3643 1192 Acked 2191 Read 8 New Done Done true
3644 1193 Acked 2192 Write 4 New Done Done true
3645 1194 Acked 2193 Read 9 New Done Done true
3646 1195 Acked 2194 Write 8 New Done Done true
3647 1196 Acked 2195 Write 10 New Done Done true
3648 1197 Acked 2196 Read 1 New Done Done true
3649 1198 Acked 2197 Write 7 New Done Done true
3650 1199 Acked 2198 Read 6 New Done Done true
3651 1200 Acked 2199 Read 9 New Done Done true
3652 1201 Acked 2200 Read 5 New Done Done true
3653 1202 Acked 2201 Write 6 New Done Done true
3654 1203 Acked 2202 Write 2 New Done Done true
3655 1204 Acked 2203 Read 3 New Done Done true
3656 1205 Acked 2204 Write 8 New Done Done true
3657 1206 Acked 2205 Flush 0 New Done Done true
3658 STATES DS:0 DS:1 DS:2 TOTAL
3659 New 301 0 0 301
3660 Sent 0 0 0 0
3661 Done 0 301 301 602
3662 Skipped 0 0 0 0
3663 Error 0 0 0 0
3664 Last Flush: 1904 2205 2205
3665 Downstairs last five completed: 1904 1903 1902 1901 1900
3666 Upstairs last five completed: 1206 1205 1204 1203 1202
3667 CLIENT: Up:0 ds:301 act:2
36682024-02-12T01:35:11.821ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
36692024-02-12T01:35:11.821ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
36702024-02-12T01:35:11.821ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
36712024-02-12T01:35:11.821ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
36722024-02-12T01:35:11.821ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
36732024-02-12T01:35:11.821ZINFOcrucible: [0] client re-new 301 jobs since flush 1904 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
36742024-02-12T01:35:11.821ZINFOcrucible: client task is exiting = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
3675 [8810] got start action
3676 [8810] I am going to restart
3677 Starting downstairs at port 8810
3678 Make output file at "/tmp/dsc/downstairs-8810.txt"
3679 [8810][0] reports Starting
3680 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1168)
3681 [8810][0] reports Running
36822024-02-12T01:35:12.822ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
36832024-02-12T01:35:12.822ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
36842024-02-12T01:35:12.822ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
36852024-02-12T01:35:12.822ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
36862024-02-12T01:35:12.822ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
36872024-02-12T01:35:12.822ZINFOcrucible: [0] client re-new 301 jobs since flush 1904 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
36882024-02-12T01:35:12.822ZINFOcrucible: client task is exiting = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
36892024-02-12T01:35:13.824ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
36902024-02-12T01:35:13.824ZINFOcrucible: ds_connection connected = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
36912024-02-12T01:35:13.825ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
36922024-02-12T01:35:13.825ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
36932024-02-12T01:35:13.825ZINFOcrucible: send last flush ID to this DS: 1904 = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
36942024-02-12T01:35:13.825ZINFOcrucible: Replied this last flush ID: 1904 = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
36952024-02-12T01:35:13.825ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
36962024-02-12T01:35:13.825ZINFOcrucible: [0] Transition from Offline to Replay = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
36972024-02-12T01:35:13.825ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
36982024-02-12T01:35:13.825ZINFOcrucible: [0] Transition from Replay to Active = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
3699 ----------------------------------------------------------------
3700 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
3701 Downstairs last five completed: 2205 2204 2203 2202 2201
3702 Upstairs last five completed: 1206 1205 1204 1203 1202
3703 CLIENT: Up:0 ds:0 act:3
3704 Replay: All jobs finished, all DS active.
3705 Main task has work to do, go find it
3706 got dsc Stop(0)
3707 stop 0
3708 Feb 12 01:35:15.170 INFO request completed, latency_us: 380, response_code: 204, uri: /stop/cid/0[8810] Got stop action so:false kr:false
3709 , method: POST, req_id: 9bbf769a-8d1b-4b05-9f6e-215dd8d70092, remote_addr: 127.0.0.1:60456, local_addr: 127.0.0.1:9998
37102024-02-12T01:35:15.170ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
37112024-02-12T01:35:15.170ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
37122024-02-12T01:35:15.170ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
37132024-02-12T01:35:15.170ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
37142024-02-12T01:35:15.171ZINFOcrucible: [0] client re-new 0 jobs since flush 2205 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
37152024-02-12T01:35:15.171ZINFOcrucible: client task is exiting = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
3716 Feb 12 01:35:15.171 INFO request completed, latency_us: 188, response_code: 200, uri: /state/cid/0, method: GET, req_id: afe8cbc6-414d-4fbc-93f2-2503d84608bd, remote_addr: 127.0.0.1:60456, local_addr: 127.0.0.1:9998
3717 [8810] Exited with: ExitStatus(unix_wait_status(9))
3718 [8810][0] reports Exit
37192024-02-12T01:35:16.172ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
37202024-02-12T01:35:16.172ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
37212024-02-12T01:35:16.172ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
37222024-02-12T01:35:16.172ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
37232024-02-12T01:35:16.172ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
37242024-02-12T01:35:16.172ZINFOcrucible: [0] client re-new 0 jobs since flush 2205 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
37252024-02-12T01:35:16.172ZINFOcrucible: client task is exiting = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
37262024-02-12T01:35:17.173ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
37272024-02-12T01:35:17.173ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
37282024-02-12T01:35:17.173ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
37292024-02-12T01:35:17.173ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
37302024-02-12T01:35:17.173ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
37312024-02-12T01:35:17.173ZINFOcrucible: [0] client re-new 0 jobs since flush 2205 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
37322024-02-12T01:35:17.173ZINFOcrucible: client task is exiting = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
37332024-02-12T01:35:18.174ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
37342024-02-12T01:35:18.174ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
37352024-02-12T01:35:18.174ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
37362024-02-12T01:35:18.174ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
37372024-02-12T01:35:18.174ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
37382024-02-12T01:35:18.174ZINFOcrucible: [0] client re-new 0 jobs since flush 2205 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
37392024-02-12T01:35:18.175ZINFOcrucible: client task is exiting = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
3740 Feb 12 01:35:19.172 INFO request completed, latency_us: 214, response_code: 200, uri: /state/cid/0, method: GET, req_id: ffa14573-9519-4b4b-887e-7857f99f1ce9, remote_addr: 127.0.0.1:60456, local_addr: 127.0.0.1:9998
3741 001/300 Write block 946 len 36864 data: 2 2 2 2 2 2 2 2 2
3742 002/300 Read block 4789 len 8192
37432024-02-12T01:35:19.175ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
37442024-02-12T01:35:19.175ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
37452024-02-12T01:35:19.175ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
37462024-02-12T01:35:19.175ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
37472024-02-12T01:35:19.175ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
37482024-02-12T01:35:19.175ZINFOcrucible: [0] client re-new 2 jobs since flush 2205 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
37492024-02-12T01:35:19.175ZINFOcrucible: client task is exiting = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
3750 003/300 Write block 1619 len 20480 data: 2 2 2 2 2
3751 004/300 Write block 4264 len 40960 data: 2 2 2 2 2 2 2 2 2 2
3752 005/300 Write block 2634 len 40960 data: 3 3 3 2 2 2 2 2 3 3
3753 006/300 Write block 2022 len 40960 data: 2 2 2 2 2 2 2 2 2 2
3754 007/300 Read block 4123 len 12288
3755 008/300 Write block 4263 len 12288 data: 2 3 3
3756 009/300 Read block 3657 len 16384
3757 010/300 Read block 280 len 16384
3758 011/300 Write block 3469 len 28672 data: 2 2 2 2 2 2 2
3759 012/300 Read block 794 len 28672
3760 013/300 Flush
3761 014/300 Read block 1152 len 36864
3762 015/300 Read block 4273 len 12288
3763 016/300 Read block 4920 len 36864
3764 017/300 Write block 3103 len 12288 data: 3 3 3
3765 018/300 Read block 3765 len 28672
3766 019/300 Read block 3178 len 16384
3767 020/300 Read block 3109 len 16384
3768 021/300 Read block 1217 len 4096
3769 022/300 Read block 2920 len 28672
3770 023/300 Write block 3530 len 32768 data: 2 2 2 2 2 2 2 2
3771 024/300 Write block 3478 len 20480 data: 2 2 2 3 3
3772 025/300 Write block 3774 len 28672 data: 2 2 2 2 2 2 2
3773 026/300 Write block 3658 len 4096 data: 3
3774 027/300 Read block 4092 len 28672
3775 028/300 Write block 415 len 12288 data: 2 2 2
3776 029/300 Write block 3876 len 16384 data: 3 3 2 2
3777 030/300 Write block 906 len 24576 data: 2 2 2 2 2 2
3778 031/300 Read block 1964 len 28672
3779 032/300 Read block 2912 len 40960
3780 033/300 Read block 1509 len 16384
3781 034/300 Read block 3472 len 32768
3782 035/300 Read block 2321 len 8192
3783 036/300 Flush
3784 037/300 Read block 808 len 20480
3785 038/300 Read block 1974 len 36864
3786 039/300 Read block 1523 len 16384
3787 040/300 Read block 1270 len 12288
3788 041/300 Write block 4389 len 16384 data: 2 2 2 2
3789 042/300 Read block 1767 len 4096
3790 043/300 Write block 795 len 40960 data: 3 3 3 3 4 4 4 4 4 2
3791 044/300 Write block 4345 len 20480 data: 2 2 2 2 2
3792 045/300 Write block 3583 len 28672 data: 2 2 2 2 2 2 2
3793 046/300 Write block 1533 len 32768 data: 2 2 2 2 3 3 4 4
3794 047/300 Read block 380 len 32768
3795 048/300 Write block 3063 len 4096 data: 2
3796 049/300 Flush
3797 050/300 Read block 444 len 12288
3798 051/300 Write block 3007 len 28672 data: 2 2 3 3 3 3 2
3799 052/300 Read block 1734 len 24576
3800 053/300 Write block 945 len 8192 data: 2 3
3801 054/300 Read block 4052 len 36864
3802 055/300 Flush
3803 056/300 Write block 3673 len 16384 data: 2 2 2 2
3804 057/300 Flush
3805 058/300 Read block 2979 len 40960
3806 059/300 Write block 2813 len 28672 data: 3 2 2 2 2 2 2
3807 060/300 Read block 1372 len 4096
3808 061/300 Write block 1209 len 36864 data: 3 3 3 3 3 2 2 2 2
3809 062/300 Read block 4511 len 28672
3810 063/300 Read block 3944 len 36864
3811 064/300 Read block 810 len 8192
3812 065/300 Flush
3813 066/300 Read block 4478 len 8192
3814 067/300 Write block 4402 len 28672 data: 2 2 2 2 2 2 2
3815 068/300 Read block 286 len 8192
3816 069/300 Read block 2632 len 8192
3817 070/300 Read block 1056 len 36864
3818 071/300 Read block 4916 len 16384
3819 072/300 Read block 553 len 28672
3820 073/300 Read block 3664 len 4096
3821 074/300 Read block 1322 len 12288
3822 075/300 Read block 4430 len 4096
3823 076/300 Read block 699 len 40960
3824 077/300 Flush
3825 078/300 Read block 1624 len 12288
3826 079/300 Read block 1210 len 24576
3827 080/300 Read block 462 len 20480
3828 081/300 Read block 1525 len 24576
3829 082/300 Read block 1347 len 32768
3830 083/300 Write block 4950 len 4096 data: 3
3831 084/300 Read block 2279 len 24576
3832 085/300 Write block 665 len 4096 data: 2
3833 086/300 Flush
3834 087/300 Read block 2654 len 32768
3835 088/300 Write block 1671 len 40960 data: 2 2 2 2 2 2 3 4 4 4
3836 089/300 Write block 1856 len 32768 data: 5 4 3 3 3 3 3 3
3837 090/300 Read block 2082 len 28672
3838 091/300 Read block 1417 len 16384
3839 092/300 Read block 4004 len 16384
3840 093/300 Write block 1071 len 16384 data: 4 2 2 2
3841 094/300 Write block 1122 len 12288 data: 4 4 3
3842 095/300 Read block 1439 len 8192
3843 096/300 Flush
3844 097/300 Read block 1033 len 20480
3845 098/300 Write block 2364 len 12288 data: 2 2 2
3846 099/300 Read block 2835 len 36864
3847 100/300 Write block 4717 len 8192 data: 3 4
3848 101/300 Write block 3892 len 40960 data: 3 3 4 4 4 4 3 4 4 3
3849 102/300 Write block 3400 len 28672 data: 4 4 4 5 5 3 3
3850 103/300 Write block 937 len 40960 data: 2 2 2 2 2 2 2 2 3 4
3851 104/300 Read block 1809 len 36864
3852 105/300 Write block 1006 len 4096 data: 2
3853 106/300 Read block 4905 len 36864
3854 107/300 Write block 3768 len 40960 data: 3 3 3 3 2 2 3 3 3 3
3855 108/300 Read block 239 len 40960
3856 109/300 Write block 2087 len 32768 data: 2 2 2 2 2 3 3 3
3857 110/300 Write block 2337 len 32768 data: 3 3 4 4 4 4 3 3
3858 111/300 Write block 3995 len 28672 data: 2 2 2 3 3 3 3
3859 112/300 Write block 782 len 4096 data: 4
3860 113/300 Flush
3861 114/300 Read block 84 len 4096
3862 115/300 Flush
3863 116/300 Read block 536 len 8192
3864 117/300 Flush
3865 118/300 Write block 2001 len 28672 data: 2 2 2 2 2 2 2
3866 119/300 Read block 4759 len 32768
3867 120/300 Read block 3809 len 12288
3868 121/300 Write block 4260 len 36864 data: 5 3 2 3 4 4 3 3 3
3869 122/300 Read block 2681 len 24576
3870 123/300 Write block 956 len 28672 data: 2 2 2 2 2 2 2
3871 124/300 Read block 1951 len 24576
3872 125/300 Write block 694 len 24576 data: 2 2 2 2 2 2
3873 126/300 Read block 4343 len 28672
3874 127/300 Write block 3424 len 28672 data: 2 2 2 2 4 4 4
3875 128/300 Read block 2821 len 28672
3876 129/300 Flush
3877 130/300 Write block 998 len 24576 data: 3 3 3 3 3 3
3878 131/300 Read block 701 len 36864
3879 132/300 Write block 3358 len 36864 data: 2 2 2 2 2 2 2 2 2
3880 133/300 Read block 164 len 12288
3881 134/300 Read block 1799 len 8192
3882 135/300 Read block 2579 len 28672
3883 136/300 Write block 3774 len 16384 data: 4 4 4 4
3884 137/300 Flush
3885 138/300 Write block 3374 len 36864 data: 2 2 2 2 2 2 2 2 2
3886 139/300 Flush
3887 140/300 Write block 3505 len 4096 data: 3
3888 141/300 Read block 3574 len 28672
3889 142/300 Read block 3476 len 8192
3890 143/300 Read block 1088 len 28672
3891 144/300 Flush
3892 145/300 Write block 148 len 24576 data: 2 2 3 3 3 2
3893 146/300 Read block 3313 len 8192
3894 147/300 Read block 1444 len 20480
3895 148/300 Write block 3715 len 16384 data: 2 2 2 2
3896 149/300 Read block 607 len 12288
3897 150/300 Write block 440 len 16384 data: 2 2 2 2
3898 151/300 Read block 4936 len 12288
3899 152/300 Read block 4630 len 20480
3900 153/300 Write block 527 len 36864 data: 3 3 3 3 3 3 3 4 3
3901 154/300 Flush
3902 155/300 Read block 1951 len 8192
3903 156/300 Read block 3952 len 20480
3904 157/300 Write block 788 len 12288 data: 3 2 3
3905 158/300 Write block 804 len 28672 data: 3 2 2 2 2 2 2
3906 159/300 Read block 1149 len 12288
3907 160/300 Read block 2640 len 16384
3908 161/300 Write block 3602 len 24576 data: 2 3 3 3 3 3
3909 162/300 Write block 1177 len 32768 data: 2 2 3 3 3 3 3 2
3910 163/300 Read block 1717 len 32768
3911 164/300 Flush
3912 165/300 Write block 3359 len 28672 data: 3 3 3 3 3 3 3
3913 166/300 Read block 2418 len 12288
3914 167/300 Write block 4935 len 12288 data: 2 2 2
3915 168/300 Flush
3916 169/300 Read block 4229 len 28672
3917 170/300 Read block 238 len 40960
3918 171/300 Flush
3919 172/300 Read block 4050 len 28672
3920 173/300 Read block 1742 len 12288
3921 174/300 Read block 4388 len 12288
3922 175/300 Read block 3630 len 20480
3923 176/300 Flush
3924 177/300 Read block 3523 len 40960
3925 178/300 Read block 3407 len 24576
3926 179/300 Write block 1899 len 8192 data: 2 2
3927 180/300 Read block 4875 len 40960
3928 181/300 Read block 130 len 24576
3929 182/300 Write block 923 len 20480 data: 2 2 3 3 4
3930 183/300 Read block 3675 len 20480
3931 184/300 Read block 2088 len 12288
3932 185/300 Write block 2372 len 40960 data: 2 2 2 2 2 2 2 2 2 2
3933 186/300 Write block 4722 len 8192 data: 3 3
3934 187/300 Read block 2753 len 32768
3935 188/300 Flush
3936 189/300 Read block 1984 len 24576
3937 190/300 Write block 816 len 16384 data: 2 2 2 2
3938 191/300 Read block 3045 len 4096
3939 192/300 Read block 3348 len 40960
3940 193/300 Write block 2170 len 28672 data: 2 3 3 2 3 4 4
3941 194/300 Write block 2008 len 36864 data: 2 2 2 2 2 2 2 2 2
3942 195/300 Write block 3801 len 20480 data: 2 2 2 3 3
3943 196/300 Read block 2957 len 16384
3944 197/300 Read block 3858 len 16384
3945 198/300 Write block 2809 len 36864 data: 4 3 3 3 4 3 3 3 3
3946 199/300 Read block 3997 len 8192
3947 200/300 Read block 1662 len 4096
3948 201/300 Read block 4115 len 32768
3949 202/300 Read block 4672 len 32768
3950 203/300 Read block 1920 len 4096
3951 204/300 Read block 11 len 32768
3952 205/300 Read block 991 len 32768
3953 206/300 Read block 1876 len 16384
3954 207/300 Read block 3861 len 12288
3955 208/300 Write block 2405 len 16384 data: 2 2 2 2
3956 209/300 Write block 3945 len 12288 data: 2 2 2
3957 210/300 Read block 1715 len 16384
3958 211/300 Read block 42 len 36864
3959 212/300 Read block 1234 len 8192
3960 213/300 Write block 1705 len 24576 data: 5 4 4 4 2 2
3961 214/300 Read block 3239 len 16384
3962 215/300 Read block 411 len 4096
3963 216/300 Write block 747 len 12288 data: 3 3 2
3964 217/300 Write block 775 len 24576 data: 2 2 3 3 3 4
3965 218/300 Write block 1394 len 12288 data: 3 3 3
3966 219/300 Read block 4505 len 40960
3967 220/300 Write block 2717 len 8192 data: 2 2
3968 221/300 Flush
3969 222/300 Write block 3929 len 8192 data: 2 2
3970 223/300 Read block 4325 len 4096
3971 224/300 Write block 2836 len 40960 data: 3 4 4 4 3 3 2 2 2 2
3972 225/300 Read block 4349 len 20480
3973 226/300 Write block 2250 len 28672 data: 2 2 3 3 3 4 4
3974 227/300 Read block 2633 len 8192
3975 228/300 Read block 3389 len 32768
3976 229/300 Read block 4225 len 4096
3977 230/300 Write block 1327 len 4096 data: 3
3978 231/300 Write block 2361 len 20480 data: 2 2 2 3 3
3979 232/300 Write block 2863 len 20480 data: 3 3 2 2 2
3980 233/300 Read block 4850 len 32768
3981 234/300 Read block 3114 len 40960
3982 235/300 Write block 104 len 12288 data: 5 5 5
3983 236/300 Read block 2219 len 12288
3984 237/300 Read block 2055 len 40960
3985 238/300 Read block 2871 len 16384
3986 239/300 Read block 4653 len 28672
3987 240/300 Read block 4426 len 24576
3988 241/300 Read block 1638 len 8192
3989 242/300 Read block 768 len 8192
3990 243/300 Read block 2206 len 36864
3991 244/300 Read block 1630 len 36864
3992 245/300 Read block 2540 len 32768
3993 246/300 Read block 3578 len 20480
3994 247/300 Read block 939 len 8192
3995 248/300 Read block 836 len 4096
3996 249/300 Read block 1946 len 32768
3997 250/300 Flush
3998 251/300 Read block 3186 len 32768
3999 252/300 Write block 27 len 16384 data: 2 3 3 3
4000 253/300 Write block 356 len 40960 data: 2 2 2 2 2 2 2 2 2 2
4001 254/300 Write block 3450 len 28672 data: 3 3 3 3 3 3 3
4002 255/300 Write block 3470 len 12288 data: 3 3 3
4003 256/300 Read block 1022 len 28672
4004 257/300 Write block 3957 len 40960 data: 3 2 2 2 2 3 3 3 3 3
4005 258/300 Write block 3112 len 24576 data: 2 2 2 2 2 2
4006 259/300 Read block 3051 len 24576
4007 260/300 Read block 533 len 8192
4008 261/300 Write block 2780 len 36864 data: 2 2 2 2 2 3 4 4 4
4009 262/300 Read block 1729 len 12288
4010 263/300 Write block 3104 len 8192 data: 4 4
4011 264/300 Flush
4012 265/300 Write block 256 len 40960 data: 2 2 2 2 2 2 2 3 3 3
4013 266/300 Write block 3860 len 36864 data: 2 2 2 2 2 2 2 2 2
4014 267/300 Read block 3439 len 8192
4015 268/300 Write block 4724 len 8192 data: 2 2
4016 269/300 Write block 2876 len 28672 data: 2 2 2 2 2 3 3
4017 270/300 Read block 3763 len 16384
4018 271/300 Read block 431 len 12288
4019 272/300 Write block 508 len 40960 data: 3 3 3 3 4 4 4 3 3 3
4020 273/300 Read block 4238 len 32768
4021 274/300 Read block 1249 len 36864
4022 275/300 Write block 4664 len 16384 data: 3 3 3 3
4023 276/300 Read block 2819 len 4096
4024 277/300 Read block 1046 len 8192
4025 278/300 Write block 18 len 16384 data: 3 3 3 3
4026 279/300 Write block 165 len 4096 data: 3
4027 280/300 Write block 3551 len 8192 data: 2 2
4028 281/300 Read block 3114 len 24576
4029 282/300 Write block 851 len 4096 data: 3
4030 283/300 Write block 486 len 20480 data: 3 2 2 2 2
4031 284/300 Read block 2035 len 32768
4032 285/300 Write block 3125 len 32768 data: 2 2 2 2 2 3 2 2
4033 286/300 Write block 2684 len 20480 data: 3 2 3 3 3
4034 287/300 Write block 2421 len 40960 data: 3 3 3 4 4 4 4 2 2 2
4035 288/300 Read block 661 len 12288
4036 289/300 Read block 2424 len 36864
4037 290/300 Read block 3509 len 36864
4038 291/300 Read block 2476 len 28672
4039 292/300 Flush
4040 293/300 Read block 1914 len 4096
4041 294/300 Read block 4610 len 24576
4042 295/300 Flush
4043 296/300 Write block 868 len 4096 data: 2
4044 297/300 Read block 4749 len 24576
4045 298/300 Flush
4046 299/300 Read block 3648 len 16384
4047 300/300 Read block 491 len 16384
4048 Main task has work to do, go find it
4049 got dsc Start(0)
4050 start 0
4051 Feb 12 01:35:19.524 INFO request completed, latency_us: 231, response_code: 204, uri: /start/cid/0, method: POST, req_id: 1babdb0a-c8de-4548-8d57-44a5a5686391, remote_addr: 127.0.0.1:60456, local_addr: 127.0.0.1:9998
4052 Replay: started 0, returned:Ok(())
4053 ----------------------------------------------------------------
4054 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
4055 States: Offline Active Active
4056 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
4057 1207 Acked 2206 Write 9 New Done Done true
4058 1208 Acked 2207 Read 2 New Done Done true
4059 1209 Acked 2208 Write 5 New Done Done false
4060 1210 Acked 2209 Write 10 New Done Done false
4061 1211 Acked 2210 Write 10 New Done Done false
4062 1212 Acked 2211 Write 10 New Done Done false
4063 1213 Acked 2212 Read 3 New Done Done false
4064 1214 Acked 2213 Write 3 New Done Done false
4065 1215 Acked 2214 Read 4 New Done Done false
4066 1216 Acked 2215 Read 4 New Done Done false
4067 1217 Acked 2216 Write 7 New Done Done false
4068 1218 Acked 2217 Read 7 New Done Done false
4069 1219 Acked 2218 Flush 0 New Done Done false
4070 1220 Acked 2219 Read 9 New Done Done false
4071 1221 Acked 2220 Read 3 New Done Done false
4072 1222 Acked 2221 Read 9 New Done Done false
4073 1223 Acked 2222 Write 3 New Done Done false
4074 1224 Acked 2223 Read 7 New Done Done false
4075 1225 Acked 2224 Read 4 New Done Done false
4076 1226 Acked 2225 Read 4 New Done Done false
4077 1227 Acked 2226 Read 1 New Done Done false
4078 1228 Acked 2227 Read 7 New Done Done false
4079 1229 Acked 2228 Write 8 New Done Done false
4080 1230 Acked 2229 Write 5 New Done Done false
4081 1231 Acked 2230 Write 7 New Done Done false
4082 1232 Acked 2231 Write 1 New Done Done false
4083 1233 Acked 2232 Read 7 New Done Done false
4084 1234 Acked 2233 Write 3 New Done Done false
4085 1235 Acked 2234 Write 4 New Done Done false
4086 1236 Acked 2235 Write 6 New Done Done false
4087 1237 Acked 2236 Read 7 New Done Done false
4088 1238 Acked 2237 Read 10 New Done Done false
4089 1239 Acked 2238 Read 4 New Done Done false
4090 1240 Acked 2239 Read 8 New Done Done false
4091 1241 Acked 2240 Read 2 New Done Done false
4092 1242 Acked 2241 Flush 0 New Done Done false
4093 1243 Acked 2242 Read 5 New Done Done false
4094 1244 Acked 2243 Read 9 New Done Done false
4095 1245 Acked 2244 Read 4 New Done Done false
4096 1246 Acked 2245 Read 3 New Done Done false
4097 1247 Acked 2246 Write 4 New Done Done false
4098 1248 Acked 2247 Read 1 New Done Done false
4099 1249 Acked 2248 Write 10 New Done Done false
4100 1250 Acked 2249 Write 5 New Done Done false
4101 1251 Acked 2250 Write 7 New Done Done false
4102 1252 Acked 2251 Write 8 New Done Done false
4103 1253 Acked 2252 Read 8 New Done Done false
4104 1254 Acked 2253 Write 1 New Done Done false
4105 1255 Acked 2254 Flush 0 New Done Done false
4106 1256 Acked 2255 Read 3 New Done Done false
4107 1257 Acked 2256 Write 7 New Done Done false
4108 1258 Acked 2257 Read 6 New Done Done false
4109 1259 Acked 2258 Write 2 New Done Done false
4110 1260 Acked 2259 Read 9 New Done Done false
4111 1261 Acked 2260 Flush 0 New Done Done false
4112 1262 Acked 2261 Write 4 New Done Done false
4113 1263 Acked 2262 Flush 0 New Done Done false
4114 1264 Acked 2263 Read 10 New Done Done false
4115 1265 Acked 2264 Write 7 New Done Done false
4116 1266 Acked 2265 Read 1 New Done Done false
4117 1267 Acked 2266 Write 9 New Done Done false
4118 1268 Acked 2267 Read 7 New Done Done false
4119 1269 Acked 2268 Read 9 New Done Done false
4120 1270 Acked 2269 Read 2 New Done Done false
4121 1271 Acked 2270 Flush 0 New Done Done false
4122 1272 Acked 2271 Read 2 New Done Done false
4123 1273 Acked 2272 Write 7 New Done Done false
4124 1274 Acked 2273 Read 2 New Done Done false
4125 1275 Acked 2274 Read 2 New Done Done false
4126 1276 Acked 2275 Read 9 New Done Done false
4127 1277 Acked 2276 Read 4 New Done Done false
4128 1278 Acked 2277 Read 7 New Done Done false
4129 1279 Acked 2278 Read 1 New Done Done false
4130 1280 Acked 2279 Read 3 New Done Done false
4131 1281 Acked 2280 Read 1 New Done Done false
4132 1282 Acked 2281 Read 10 New Done Done false
4133 1283 Acked 2282 Flush 0 New Done Done false
4134 1284 Acked 2283 Read 3 New Done Done false
4135 1285 Acked 2284 Read 6 New Done Done false
4136 1286 Acked 2285 Read 5 New Done Done false
4137 1287 Acked 2286 Read 6 New Done Done false
4138 1288 Acked 2287 Read 8 New Done Done false
4139 1289 Acked 2288 Write 1 New Done Done false
4140 1290 Acked 2289 Read 6 New Done Done false
4141 1291 Acked 2290 Write 1 New Done Done false
4142 1292 Acked 2291 Flush 0 New Done Done false
4143 1293 Acked 2292 Read 8 New Done Done false
4144 1294 Acked 2293 Flush 0 New Done Done false
4145 1295 Acked 2294 Write 10 New Done Done false
4146 1296 Acked 2295 Write 8 New Done Done false
4147 1297 Acked 2296 Read 7 New Done Done false
4148 1298 Acked 2297 Read 4 New Done Done false
4149 1299 Acked 2298 Read 4 New Done Done false
4150 1300 Acked 2299 Write 4 New Done Done false
4151 1301 Acked 2300 Write 3 New Done Done false
4152 1302 Acked 2301 Read 2 New Done Done false
4153 1303 Acked 2302 Flush 0 New Done Done false
4154 1304 Acked 2303 Read 5 New Done Done false
4155 1305 Acked 2304 Write 3 New Done Done false
4156 1306 Acked 2305 Read 9 New Done Done false
4157 1307 Acked 2306 Write 2 New Done Done false
4158 1308 Acked 2307 Write 10 New Done Done false
4159 1309 Acked 2308 Write 7 New Done Done false
4160 1310 Acked 2309 Write 10 New Done Done false
4161 1311 Acked 2310 Read 9 New Done Done false
4162 1312 Acked 2311 Write 1 New Done Done false
4163 1313 Acked 2312 Read 9 New Done Done false
4164 1314 Acked 2313 Write 10 New Done Done false
4165 1315 Acked 2314 Read 10 New Done Done false
4166 1316 Acked 2315 Write 8 New Done Done false
4167 1317 Acked 2316 Write 8 New Done Done false
4168 1318 Acked 2317 Write 7 New Done Done false
4169 1319 Acked 2318 Write 1 New Done Done false
4170 1320 Acked 2319 Flush 0 New Done Done false
4171 1321 Acked 2320 Read 1 New Done Done false
4172 1322 Acked 2321 Flush 0 New Done Done false
4173 1323 Acked 2322 Read 2 New Done Done false
4174 1324 Acked 2323 Flush 0 New Done Done false
4175 1325 Acked 2324 Write 7 New Done Done false
4176 1326 Acked 2325 Read 8 New Done Done false
4177 1327 Acked 2326 Read 3 New Done Done false
4178 1328 Acked 2327 Write 9 New Done Done false
4179 1329 Acked 2328 Read 6 New Done Done false
4180 1330 Acked 2329 Write 7 New Done Done false
4181 1331 Acked 2330 Read 6 New Done Done false
4182 1332 Acked 2331 Write 6 New Done Done false
4183 1333 Acked 2332 Read 7 New Done Done false
4184 1334 Acked 2333 Write 7 New Done Done false
4185 1335 Acked 2334 Read 7 New Done Done false
4186 1336 Acked 2335 Flush 0 New Done Done false
4187 1337 Acked 2336 Write 6 New Done Done false
4188 1338 Acked 2337 Read 9 New Done Done false
4189 1339 Acked 2338 Write 9 New Done Done false
4190 1340 Acked 2339 Read 3 New Done Done false
4191 1341 Acked 2340 Read 2 New Done Done false
4192 1342 Acked 2341 Read 7 New Done Done false
4193 1343 Acked 2342 Write 4 New Done Done false
4194 1344 Acked 2343 Flush 0 New Done Done false
4195 1345 Acked 2344 Write 9 New Done Done false
4196 1346 Acked 2345 Flush 0 New Done Done false
4197 1347 Acked 2346 Write 1 New Done Done false
4198 1348 Acked 2347 Read 7 New Done Done false
4199 1349 Acked 2348 Read 2 New Done Done false
4200 1350 Acked 2349 Read 7 New Done Done false
4201 1351 Acked 2350 Flush 0 New Done Done false
4202 1352 Acked 2351 Write 6 New Done Done false
4203 1353 Acked 2352 Read 2 New Done Done false
4204 1354 Acked 2353 Read 5 New Done Done false
4205 1355 Acked 2354 Write 4 New Done Done false
4206 1356 Acked 2355 Read 3 New Done Done false
4207 1357 Acked 2356 Write 4 New Done Done false
4208 1358 Acked 2357 Read 3 New Done Done false
4209 1359 Acked 2358 Read 5 New Done Done false
4210 1360 Acked 2359 Write 9 New Done Done false
4211 1361 Acked 2360 Flush 0 New Done Done false
4212 1362 Acked 2361 Read 2 New Done Done false
4213 1363 Acked 2362 Read 5 New Done Done false
4214 1364 Acked 2363 Write 3 New Done Done false
4215 1365 Acked 2364 Write 7 New Done Done false
4216 1366 Acked 2365 Read 3 New Done Done false
4217 1367 Acked 2366 Read 4 New Done Done false
4218 1368 Acked 2367 Write 6 New Done Done false
4219 1369 Acked 2368 Write 8 New Done Done false
4220 1370 Acked 2369 Read 8 New Done Done false
4221 1371 Acked 2370 Flush 0 New Done Done false
4222 1372 Acked 2371 Write 7 New Done Done false
4223 1373 Acked 2372 Read 3 New Done Done false
4224 1374 Acked 2373 Write 3 New Done Done false
4225 1375 Acked 2374 Flush 0 New Done Done false
4226 1376 Acked 2375 Read 7 New Done Done false
4227 1377 Acked 2376 Read 10 New Done Done false
4228 1378 Acked 2377 Flush 0 New Done Done false
4229 1379 Acked 2378 Read 7 New Done Done false
4230 1380 Acked 2379 Read 3 New Done Done false
4231 1381 Acked 2380 Read 3 New Done Done false
4232 1382 Acked 2381 Read 5 New Done Done false
4233 1383 Acked 2382 Flush 0 New Done Done false
4234 1384 Acked 2383 Read 10 New Done Done false
4235 1385 Acked 2384 Read 6 New Done Done false
4236 1386 Acked 2385 Write 2 New Done Done false
4237 1387 Acked 2386 Read 10 New Done Done false
4238 1388 Acked 2387 Read 6 New Done Done false
4239 1389 Acked 2388 Write 5 New Done Done false
4240 1390 Acked 2389 Read 5 New Done Done false
4241 1391 Acked 2390 Read 3 New Done Done false
4242 1392 Acked 2391 Write 10 New Done Done false
4243 1393 Acked 2392 Write 2 New Done Done false
4244 1394 Acked 2393 Read 8 New Done Done false
4245 1395 Acked 2394 Flush 0 New Done Done false
4246 1396 Acked 2395 Read 6 New Done Done false
4247 1397 Acked 2396 Write 4 New Done Done false
4248 1398 Acked 2397 Read 1 New Done Done false
4249 1399 Acked 2398 Read 10 New Done Done false
4250 1400 Acked 2399 Write 7 New Done Done false
4251 1401 Acked 2400 Write 9 New Done Done false
4252 1402 Acked 2401 Write 5 New Done Done false
4253 1403 Acked 2402 Read 4 New Done Done false
4254 1404 Acked 2403 Read 4 New Done Done false
4255 1405 Acked 2404 Write 9 New Done Done false
4256 1406 Acked 2405 Read 2 New Done Done false
4257 1407 Acked 2406 Read 1 New Done Done false
4258 1408 Acked 2407 Read 8 New Done Done false
4259 1409 Acked 2408 Read 8 New Done Done false
4260 1410 Acked 2409 Read 1 New Done Done false
4261 1411 Acked 2410 Read 8 New Done Done false
4262 1412 Acked 2411 Read 8 New Done Done false
4263 1413 Acked 2412 Read 4 New Done Done false
4264 1414 Acked 2413 Read 3 New Done Done false
4265 1415 Acked 2414 Write 4 New Done Done false
4266 1416 Acked 2415 Write 3 New Done Done false
4267 1417 Acked 2416 Read 4 New Done Done false
4268 1418 Acked 2417 Read 9 New Done Done false
4269 1419 Acked 2418 Read 2 New Done Done false
4270 1420 Acked 2419 Write 6 New Done Done false
4271 1421 Acked 2420 Read 4 New Done Done false
4272 1422 Acked 2421 Read 1 New Done Done false
4273 1423 Acked 2422 Write 3 New Done Done false
4274 1424 Acked 2423 Write 6 New Done Done false
4275 1425 Acked 2424 Write 3 New Done Done false
4276 1426 Acked 2425 Read 10 New Done Done false
4277 1427 Acked 2426 Write 2 New Done Done false
4278 1428 Acked 2427 Flush 0 New Done Done false
4279 1429 Acked 2428 Write 2 New Done Done false
4280 1430 Acked 2429 Read 1 New Done Done false
4281 1431 Acked 2430 Write 10 New Done Done false
4282 1432 Acked 2431 Read 5 New Done Done false
4283 1433 Acked 2432 Write 7 New Done Done false
4284 1434 Acked 2433 Read 2 New Done Done false
4285 1435 Acked 2434 Read 8 New Done Done false
4286 1436 Acked 2435 Read 1 New Done Done false
4287 1437 Acked 2436 Write 1 New Done Done false
4288 1438 Acked 2437 Write 5 New Done Done false
4289 1439 Acked 2438 Write 5 New Done Done false
4290 1440 Acked 2439 Read 8 New Done Done false
4291 1441 Acked 2440 Read 10 New Done Done false
4292 1442 Acked 2441 Write 3 New Done Done false
4293 1443 Acked 2442 Read 3 New Done Done false
4294 1444 Acked 2443 Read 10 New Done Done false
4295 1445 Acked 2444 Read 4 New Done Done false
4296 1446 Acked 2445 Read 7 New Done Done false
4297 1447 Acked 2446 Read 6 New Done Done false
4298 1448 Acked 2447 Read 2 New Done Done false
4299 1449 Acked 2448 Read 2 New Done Done false
4300 1450 Acked 2449 Read 9 New Done Done false
4301 1451 Acked 2450 Read 9 New Done Done false
4302 1452 Acked 2451 Read 8 New Done Done false
4303 1453 Acked 2452 Read 5 New Done Done false
4304 1454 Acked 2453 Read 2 New Done Done false
4305 1455 Acked 2454 Read 1 New Done Done false
4306 1456 Acked 2455 Read 8 New Done Done false
4307 1457 Acked 2456 Flush 0 New Done Done false
4308 1458 Acked 2457 Read 8 New Done Done false
4309 1459 Acked 2458 Write 4 New Done Done false
4310 1460 Acked 2459 Write 10 New Done Done false
4311 1461 Acked 2460 Write 7 New Done Done false
4312 1462 Acked 2461 Write 3 New Done Done false
4313 1463 Acked 2462 Read 7 New Done Done false
4314 1464 Acked 2463 Write 10 New Done Done false
4315 1465 Acked 2464 Write 6 New Done Done false
4316 1466 Acked 2465 Read 6 New Done Done false
4317 1467 Acked 2466 Read 2 New Done Done false
4318 1468 Acked 2467 Write 9 New Done Done false
4319 1469 Acked 2468 Read 3 New Done Done false
4320 1470 Acked 2469 Write 2 New Done Done false
4321 1471 Acked 2470 Flush 0 New Done Done false
4322 1472 Acked 2471 Write 10 New Done Done false
4323 1473 Acked 2472 Write 9 New Done Done false
4324 1474 Acked 2473 Read 2 New Done Done false
4325 1475 Acked 2474 Write 2 New Done Done false
4326 1476 Acked 2475 Write 7 New Done Done false
4327 1477 Acked 2476 Read 4 New Done Done false
4328 1478 Acked 2477 Read 3 New Done Done false
4329 1479 Acked 2478 Write 10 New Done Done false
4330 1480 Acked 2479 Read 8 New Done Done false
4331 1481 Acked 2480 Read 9 New Done Done false
4332 1482 Acked 2481 Write 4 New Done Done false
4333 1483 Acked 2482 Read 1 New Done Done false
4334 1484 Acked 2483 Read 2 New Done Done false
4335 1485 Acked 2484 Write 4 New Done Done false
4336 1486 Acked 2485 Write 1 New Done Done false
4337 1487 Acked 2486 Write 2 New Done Done false
4338 1488 Acked 2487 Read 6 New Done Done false
4339 1489 Acked 2488 Write 1 New Done Done false
4340 1490 Acked 2489 Write 5 New Done Done false
4341 1491 Acked 2490 Read 8 New Done Done false
4342 1492 Acked 2491 Write 8 New Done Done false
4343 1493 Acked 2492 Write 5 New Done Done false
4344 1494 Acked 2493 Write 10 New Done Done false
4345 1495 Acked 2494 Read 3 New Done Done false
4346 1496 Acked 2495 Read 9 New Done Done false
4347 1497 Acked 2496 Read 9 New Done Done false
4348 1498 Acked 2497 Read 7 New Done Done false
4349 1499 Acked 2498 Flush 0 New Done Done false
4350 1500 Acked 2499 Read 1 New Done Done false
4351 1501 Acked 2500 Read 6 New Done Done false
4352 1502 Acked 2501 Flush 0 New Done Done false
4353 1503 Acked 2502 Write 1 New Done Done false
4354 1504 Acked 2503 Read 6 New Done Done false
4355 1505 Acked 2504 Flush 0 New Done Done false
4356 1506 Acked 2505 Read 4 New Done Done false
4357 1507 Acked 2506 Read 4 New Done Done false
4358 STATES DS:0 DS:1 DS:2 TOTAL
4359 New 301 0 0 301
4360 Sent 0 0 0 0
4361 Done 0 301 301 602
4362 Skipped 0 0 0 0
4363 Error 0 0 0 0
4364 Last Flush: 2205 2504 2504
4365 Downstairs last five completed: 2205 2204 2203 2202 2201
4366 Upstairs last five completed: 1507 1506 1505 1504 1503
4367 CLIENT: Up:0 ds:301 act:2
43682024-02-12T01:35:20.177ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
43692024-02-12T01:35:20.177ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
43702024-02-12T01:35:20.177ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
43712024-02-12T01:35:20.177ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
43722024-02-12T01:35:20.177ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
43732024-02-12T01:35:20.177ZINFOcrucible: [0] client re-new 302 jobs since flush 2205 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
43742024-02-12T01:35:20.177ZINFOcrucible: client task is exiting = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
43752024-02-12T01:35:21.179ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
43762024-02-12T01:35:21.179ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
43772024-02-12T01:35:21.179ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
43782024-02-12T01:35:21.179ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
43792024-02-12T01:35:21.179ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
43802024-02-12T01:35:21.179ZINFOcrucible: [0] client re-new 302 jobs since flush 2205 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
43812024-02-12T01:35:21.179ZINFOcrucible: client task is exiting = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
43822024-02-12T01:35:22.180ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
43832024-02-12T01:35:22.180ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
43842024-02-12T01:35:22.180ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
43852024-02-12T01:35:22.180ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
43862024-02-12T01:35:22.180ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
43872024-02-12T01:35:22.180ZINFOcrucible: [0] client re-new 302 jobs since flush 2205 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
43882024-02-12T01:35:22.180ZINFOcrucible: client task is exiting = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
4389 [8810] got start action
4390 [8810] I am going to restart
4391 Starting downstairs at port 8810
4392 Make output file at "/tmp/dsc/downstairs-8810.txt"
4393 [8810][0] reports Starting
43942024-02-12T01:35:23.181ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
43952024-02-12T01:35:23.181ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
43962024-02-12T01:35:23.181ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
43972024-02-12T01:35:23.181ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
43982024-02-12T01:35:23.181ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
43992024-02-12T01:35:23.181ZINFOcrucible: [0] client re-new 302 jobs since flush 2205 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
44002024-02-12T01:35:23.182ZINFOcrucible: client task is exiting = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
4401 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1169)
4402 [8810][0] reports Running
4403 ----------------------------------------------------------------
4404 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
4405 States: Offline Active Active
4406 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
4407 1207 Acked 2206 Write 9 New Done Done true
4408 1208 Acked 2207 Read 2 New Done Done true
4409 1209 Acked 2208 Write 5 New Done Done true
4410 1210 Acked 2209 Write 10 New Done Done true
4411 1211 Acked 2210 Write 10 New Done Done true
4412 1212 Acked 2211 Write 10 New Done Done true
4413 1213 Acked 2212 Read 3 New Done Done true
4414 1214 Acked 2213 Write 3 New Done Done true
4415 1215 Acked 2214 Read 4 New Done Done true
4416 1216 Acked 2215 Read 4 New Done Done true
4417 1217 Acked 2216 Write 7 New Done Done true
4418 1218 Acked 2217 Read 7 New Done Done true
4419 1219 Acked 2218 Flush 0 New Done Done true
4420 1220 Acked 2219 Read 9 New Done Done true
4421 1221 Acked 2220 Read 3 New Done Done true
4422 1222 Acked 2221 Read 9 New Done Done true
4423 1223 Acked 2222 Write 3 New Done Done true
4424 1224 Acked 2223 Read 7 New Done Done true
4425 1225 Acked 2224 Read 4 New Done Done true
4426 1226 Acked 2225 Read 4 New Done Done true
4427 1227 Acked 2226 Read 1 New Done Done true
4428 1228 Acked 2227 Read 7 New Done Done true
4429 1229 Acked 2228 Write 8 New Done Done true
4430 1230 Acked 2229 Write 5 New Done Done true
4431 1231 Acked 2230 Write 7 New Done Done true
4432 1232 Acked 2231 Write 1 New Done Done true
4433 1233 Acked 2232 Read 7 New Done Done true
4434 1234 Acked 2233 Write 3 New Done Done true
4435 1235 Acked 2234 Write 4 New Done Done true
4436 1236 Acked 2235 Write 6 New Done Done true
4437 1237 Acked 2236 Read 7 New Done Done true
4438 1238 Acked 2237 Read 10 New Done Done true
4439 1239 Acked 2238 Read 4 New Done Done true
4440 1240 Acked 2239 Read 8 New Done Done true
4441 1241 Acked 2240 Read 2 New Done Done true
4442 1242 Acked 2241 Flush 0 New Done Done true
4443 1243 Acked 2242 Read 5 New Done Done true
4444 1244 Acked 2243 Read 9 New Done Done true
4445 1245 Acked 2244 Read 4 New Done Done true
4446 1246 Acked 2245 Read 3 New Done Done true
4447 1247 Acked 2246 Write 4 New Done Done true
4448 1248 Acked 2247 Read 1 New Done Done true
4449 1249 Acked 2248 Write 10 New Done Done true
4450 1250 Acked 2249 Write 5 New Done Done true
4451 1251 Acked 2250 Write 7 New Done Done true
4452 1252 Acked 2251 Write 8 New Done Done true
4453 1253 Acked 2252 Read 8 New Done Done true
4454 1254 Acked 2253 Write 1 New Done Done true
4455 1255 Acked 2254 Flush 0 New Done Done true
4456 1256 Acked 2255 Read 3 New Done Done true
4457 1257 Acked 2256 Write 7 New Done Done true
4458 1258 Acked 2257 Read 6 New Done Done true
4459 1259 Acked 2258 Write 2 New Done Done true
4460 1260 Acked 2259 Read 9 New Done Done true
4461 1261 Acked 2260 Flush 0 New Done Done true
4462 1262 Acked 2261 Write 4 New Done Done true
4463 1263 Acked 2262 Flush 0 New Done Done true
4464 1264 Acked 2263 Read 10 New Done Done true
4465 1265 Acked 2264 Write 7 New Done Done true
4466 1266 Acked 2265 Read 1 New Done Done true
4467 1267 Acked 2266 Write 9 New Done Done true
4468 1268 Acked 2267 Read 7 New Done Done true
4469 1269 Acked 2268 Read 9 New Done Done true
4470 1270 Acked 2269 Read 2 New Done Done true
4471 1271 Acked 2270 Flush 0 New Done Done true
4472 1272 Acked 2271 Read 2 New Done Done true
4473 1273 Acked 2272 Write 7 New Done Done true
4474 1274 Acked 2273 Read 2 New Done Done true
4475 1275 Acked 2274 Read 2 New Done Done true
4476 1276 Acked 2275 Read 9 New Done Done true
4477 1277 Acked 2276 Read 4 New Done Done true
4478 1278 Acked 2277 Read 7 New Done Done true
4479 1279 Acked 2278 Read 1 New Done Done true
4480 1280 Acked 2279 Read 3 New Done Done true
4481 1281 Acked 2280 Read 1 New Done Done true
4482 1282 Acked 2281 Read 10 New Done Done true
4483 1283 Acked 2282 Flush 0 New Done Done true
4484 1284 Acked 2283 Read 3 New Done Done true
4485 1285 Acked 2284 Read 6 New Done Done true
4486 1286 Acked 2285 Read 5 New Done Done true
4487 1287 Acked 2286 Read 6 New Done Done true
4488 1288 Acked 2287 Read 8 New Done Done true
4489 1289 Acked 2288 Write 1 New Done Done true
4490 1290 Acked 2289 Read 6 New Done Done true
4491 1291 Acked 2290 Write 1 New Done Done true
4492 1292 Acked 2291 Flush 0 New Done Done true
4493 1293 Acked 2292 Read 8 New Done Done true
4494 1294 Acked 2293 Flush 0 New Done Done true
4495 1295 Acked 2294 Write 10 New Done Done true
4496 1296 Acked 2295 Write 8 New Done Done true
4497 1297 Acked 2296 Read 7 New Done Done true
4498 1298 Acked 2297 Read 4 New Done Done true
4499 1299 Acked 2298 Read 4 New Done Done true
4500 1300 Acked 2299 Write 4 New Done Done true
4501 1301 Acked 2300 Write 3 New Done Done true
4502 1302 Acked 2301 Read 2 New Done Done true
4503 1303 Acked 2302 Flush 0 New Done Done true
4504 1304 Acked 2303 Read 5 New Done Done true
4505 1305 Acked 2304 Write 3 New Done Done true
4506 1306 Acked 2305 Read 9 New Done Done true
4507 1307 Acked 2306 Write 2 New Done Done true
4508 1308 Acked 2307 Write 10 New Done Done true
4509 1309 Acked 2308 Write 7 New Done Done true
4510 1310 Acked 2309 Write 10 New Done Done true
4511 1311 Acked 2310 Read 9 New Done Done true
4512 1312 Acked 2311 Write 1 New Done Done true
4513 1313 Acked 2312 Read 9 New Done Done true
4514 1314 Acked 2313 Write 10 New Done Done true
4515 1315 Acked 2314 Read 10 New Done Done true
4516 1316 Acked 2315 Write 8 New Done Done true
4517 1317 Acked 2316 Write 8 New Done Done true
4518 1318 Acked 2317 Write 7 New Done Done true
4519 1319 Acked 2318 Write 1 New Done Done true
4520 1320 Acked 2319 Flush 0 New Done Done true
4521 1321 Acked 2320 Read 1 New Done Done true
4522 1322 Acked 2321 Flush 0 New Done Done true
4523 1323 Acked 2322 Read 2 New Done Done true
4524 1324 Acked 2323 Flush 0 New Done Done true
4525 1325 Acked 2324 Write 7 New Done Done true
4526 1326 Acked 2325 Read 8 New Done Done true
4527 1327 Acked 2326 Read 3 New Done Done true
4528 1328 Acked 2327 Write 9 New Done Done true
4529 1329 Acked 2328 Read 6 New Done Done true
4530 1330 Acked 2329 Write 7 New Done Done true
4531 1331 Acked 2330 Read 6 New Done Done true
4532 1332 Acked 2331 Write 6 New Done Done true
4533 1333 Acked 2332 Read 7 New Done Done true
4534 1334 Acked 2333 Write 7 New Done Done true
4535 1335 Acked 2334 Read 7 New Done Done true
4536 1336 Acked 2335 Flush 0 New Done Done true
4537 1337 Acked 2336 Write 6 New Done Done true
4538 1338 Acked 2337 Read 9 New Done Done true
4539 1339 Acked 2338 Write 9 New Done Done true
4540 1340 Acked 2339 Read 3 New Done Done true
4541 1341 Acked 2340 Read 2 New Done Done true
4542 1342 Acked 2341 Read 7 New Done Done true
4543 1343 Acked 2342 Write 4 New Done Done true
4544 1344 Acked 2343 Flush 0 New Done Done true
4545 1345 Acked 2344 Write 9 New Done Done true
4546 1346 Acked 2345 Flush 0 New Done Done true
4547 1347 Acked 2346 Write 1 New Done Done true
4548 1348 Acked 2347 Read 7 New Done Done true
4549 1349 Acked 2348 Read 2 New Done Done true
4550 1350 Acked 2349 Read 7 New Done Done true
4551 1351 Acked 2350 Flush 0 New Done Done true
4552 1352 Acked 2351 Write 6 New Done Done true
4553 1353 Acked 2352 Read 2 New Done Done true
4554 1354 Acked 2353 Read 5 New Done Done true
4555 1355 Acked 2354 Write 4 New Done Done true
4556 1356 Acked 2355 Read 3 New Done Done true
4557 1357 Acked 2356 Write 4 New Done Done true
4558 1358 Acked 2357 Read 3 New Done Done true
4559 1359 Acked 2358 Read 5 New Done Done true
4560 1360 Acked 2359 Write 9 New Done Done true
4561 1361 Acked 2360 Flush 0 New Done Done true
4562 1362 Acked 2361 Read 2 New Done Done true
4563 1363 Acked 2362 Read 5 New Done Done true
4564 1364 Acked 2363 Write 3 New Done Done true
4565 1365 Acked 2364 Write 7 New Done Done true
4566 1366 Acked 2365 Read 3 New Done Done true
4567 1367 Acked 2366 Read 4 New Done Done true
4568 1368 Acked 2367 Write 6 New Done Done true
4569 1369 Acked 2368 Write 8 New Done Done true
4570 1370 Acked 2369 Read 8 New Done Done true
4571 1371 Acked 2370 Flush 0 New Done Done true
4572 1372 Acked 2371 Write 7 New Done Done true
4573 1373 Acked 2372 Read 3 New Done Done true
4574 1374 Acked 2373 Write 3 New Done Done true
4575 1375 Acked 2374 Flush 0 New Done Done true
4576 1376 Acked 2375 Read 7 New Done Done true
4577 1377 Acked 2376 Read 10 New Done Done true
4578 1378 Acked 2377 Flush 0 New Done Done true
4579 1379 Acked 2378 Read 7 New Done Done true
4580 1380 Acked 2379 Read 3 New Done Done true
4581 1381 Acked 2380 Read 3 New Done Done true
4582 1382 Acked 2381 Read 5 New Done Done true
4583 1383 Acked 2382 Flush 0 New Done Done true
4584 1384 Acked 2383 Read 10 New Done Done true
4585 1385 Acked 2384 Read 6 New Done Done true
4586 1386 Acked 2385 Write 2 New Done Done true
4587 1387 Acked 2386 Read 10 New Done Done true
4588 1388 Acked 2387 Read 6 New Done Done true
4589 1389 Acked 2388 Write 5 New Done Done true
4590 1390 Acked 2389 Read 5 New Done Done true
4591 1391 Acked 2390 Read 3 New Done Done true
4592 1392 Acked 2391 Write 10 New Done Done true
4593 1393 Acked 2392 Write 2 New Done Done true
4594 1394 Acked 2393 Read 8 New Done Done true
4595 1395 Acked 2394 Flush 0 New Done Done true
4596 1396 Acked 2395 Read 6 New Done Done true
4597 1397 Acked 2396 Write 4 New Done Done true
4598 1398 Acked 2397 Read 1 New Done Done true
4599 1399 Acked 2398 Read 10 New Done Done true
4600 1400 Acked 2399 Write 7 New Done Done true
4601 1401 Acked 2400 Write 9 New Done Done true
4602 1402 Acked 2401 Write 5 New Done Done true
4603 1403 Acked 2402 Read 4 New Done Done true
4604 1404 Acked 2403 Read 4 New Done Done true
4605 1405 Acked 2404 Write 9 New Done Done true
4606 1406 Acked 2405 Read 2 New Done Done true
4607 1407 Acked 2406 Read 1 New Done Done true
4608 1408 Acked 2407 Read 8 New Done Done true
4609 1409 Acked 2408 Read 8 New Done Done true
4610 1410 Acked 2409 Read 1 New Done Done true
4611 1411 Acked 2410 Read 8 New Done Done true
4612 1412 Acked 2411 Read 8 New Done Done true
4613 1413 Acked 2412 Read 4 New Done Done true
4614 1414 Acked 2413 Read 3 New Done Done true
4615 1415 Acked 2414 Write 4 New Done Done true
4616 1416 Acked 2415 Write 3 New Done Done true
4617 1417 Acked 2416 Read 4 New Done Done true
4618 1418 Acked 2417 Read 9 New Done Done true
4619 1419 Acked 2418 Read 2 New Done Done true
4620 1420 Acked 2419 Write 6 New Done Done true
4621 1421 Acked 2420 Read 4 New Done Done true
4622 1422 Acked 2421 Read 1 New Done Done true
4623 1423 Acked 2422 Write 3 New Done Done true
4624 1424 Acked 2423 Write 6 New Done Done true
4625 1425 Acked 2424 Write 3 New Done Done true
4626 1426 Acked 2425 Read 10 New Done Done true
4627 1427 Acked 2426 Write 2 New Done Done true
4628 1428 Acked 2427 Flush 0 New Done Done true
4629 1429 Acked 2428 Write 2 New Done Done true
4630 1430 Acked 2429 Read 1 New Done Done true
4631 1431 Acked 2430 Write 10 New Done Done true
4632 1432 Acked 2431 Read 5 New Done Done true
4633 1433 Acked 2432 Write 7 New Done Done true
4634 1434 Acked 2433 Read 2 New Done Done true
4635 1435 Acked 2434 Read 8 New Done Done true
4636 1436 Acked 2435 Read 1 New Done Done true
4637 1437 Acked 2436 Write 1 New Done Done true
4638 1438 Acked 2437 Write 5 New Done Done true
4639 1439 Acked 2438 Write 5 New Done Done true
4640 1440 Acked 2439 Read 8 New Done Done true
4641 1441 Acked 2440 Read 10 New Done Done true
4642 1442 Acked 2441 Write 3 New Done Done true
4643 1443 Acked 2442 Read 3 New Done Done true
4644 1444 Acked 2443 Read 10 New Done Done true
4645 1445 Acked 2444 Read 4 New Done Done true
4646 1446 Acked 2445 Read 7 New Done Done true
4647 1447 Acked 2446 Read 6 New Done Done true
4648 1448 Acked 2447 Read 2 New Done Done true
4649 1449 Acked 2448 Read 2 New Done Done true
4650 1450 Acked 2449 Read 9 New Done Done true
4651 1451 Acked 2450 Read 9 New Done Done true
4652 1452 Acked 2451 Read 8 New Done Done true
4653 1453 Acked 2452 Read 5 New Done Done true
4654 1454 Acked 2453 Read 2 New Done Done true
4655 1455 Acked 2454 Read 1 New Done Done true
4656 1456 Acked 2455 Read 8 New Done Done true
4657 1457 Acked 2456 Flush 0 New Done Done true
4658 1458 Acked 2457 Read 8 New Done Done true
4659 1459 Acked 2458 Write 4 New Done Done true
4660 1460 Acked 2459 Write 10 New Done Done true
4661 1461 Acked 2460 Write 7 New Done Done true
4662 1462 Acked 2461 Write 3 New Done Done true
4663 1463 Acked 2462 Read 7 New Done Done true
4664 1464 Acked 2463 Write 10 New Done Done true
4665 1465 Acked 2464 Write 6 New Done Done true
4666 1466 Acked 2465 Read 6 New Done Done true
4667 1467 Acked 2466 Read 2 New Done Done true
4668 1468 Acked 2467 Write 9 New Done Done true
4669 1469 Acked 2468 Read 3 New Done Done true
4670 1470 Acked 2469 Write 2 New Done Done true
4671 1471 Acked 2470 Flush 0 New Done Done true
4672 1472 Acked 2471 Write 10 New Done Done true
4673 1473 Acked 2472 Write 9 New Done Done true
4674 1474 Acked 2473 Read 2 New Done Done true
4675 1475 Acked 2474 Write 2 New Done Done true
4676 1476 Acked 2475 Write 7 New Done Done true
4677 1477 Acked 2476 Read 4 New Done Done true
4678 1478 Acked 2477 Read 3 New Done Done true
4679 1479 Acked 2478 Write 10 New Done Done true
4680 1480 Acked 2479 Read 8 New Done Done true
4681 1481 Acked 2480 Read 9 New Done Done true
4682 1482 Acked 2481 Write 4 New Done Done true
4683 1483 Acked 2482 Read 1 New Done Done true
4684 1484 Acked 2483 Read 2 New Done Done true
4685 1485 Acked 2484 Write 4 New Done Done true
4686 1486 Acked 2485 Write 1 New Done Done true
4687 1487 Acked 2486 Write 2 New Done Done true
4688 1488 Acked 2487 Read 6 New Done Done true
4689 1489 Acked 2488 Write 1 New Done Done true
4690 1490 Acked 2489 Write 5 New Done Done true
4691 1491 Acked 2490 Read 8 New Done Done true
4692 1492 Acked 2491 Write 8 New Done Done true
4693 1493 Acked 2492 Write 5 New Done Done true
4694 1494 Acked 2493 Write 10 New Done Done true
4695 1495 Acked 2494 Read 3 New Done Done true
4696 1496 Acked 2495 Read 9 New Done Done true
4697 1497 Acked 2496 Read 9 New Done Done true
4698 1498 Acked 2497 Read 7 New Done Done true
4699 1499 Acked 2498 Flush 0 New Done Done true
4700 1500 Acked 2499 Read 1 New Done Done true
4701 1501 Acked 2500 Read 6 New Done Done true
4702 1502 Acked 2501 Flush 0 New Done Done true
4703 1503 Acked 2502 Write 1 New Done Done true
4704 1504 Acked 2503 Read 6 New Done Done true
4705 1505 Acked 2504 Flush 0 New Done Done true
4706 1506 Acked 2505 Read 4 New Done Done true
4707 1507 Acked 2506 Read 4 New Done Done true
4708 1508 Acked 2507 Flush 0 New Done Done true
4709 STATES DS:0 DS:1 DS:2 TOTAL
4710 New 302 0 0 302
4711 Sent 0 0 0 0
4712 Done 0 302 302 604
4713 Skipped 0 0 0 0
4714 Error 0 0 0 0
4715 Last Flush: 2205 2507 2507
4716 Downstairs last five completed: 2205 2204 2203 2202 2201
4717 Upstairs last five completed: 1508 1507 1506 1505 1504
4718 CLIENT: Up:0 ds:302 act:2
47192024-02-12T01:35:24.183ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
47202024-02-12T01:35:24.183ZINFOcrucible: ds_connection connected = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
47212024-02-12T01:35:24.184ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
47222024-02-12T01:35:24.184ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
47232024-02-12T01:35:24.184ZINFOcrucible: send last flush ID to this DS: 2205 = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
47242024-02-12T01:35:24.184ZINFOcrucible: Replied this last flush ID: 2205 = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
47252024-02-12T01:35:24.185ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
47262024-02-12T01:35:24.185ZINFOcrucible: [0] Transition from Offline to Replay = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
47272024-02-12T01:35:24.185ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
47282024-02-12T01:35:24.185ZINFOcrucible: [0] Transition from Replay to Active = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
4729 ----------------------------------------------------------------
4730 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
4731 Downstairs last five completed: 2507 2506 2505 2504 2503
4732 Upstairs last five completed: 1508 1507 1506 1505 1504
4733 CLIENT: Up:0 ds:0 act:3
4734 Replay: All jobs finished, all DS active.
4735 Main task has work to do, go find it
4736 got dsc Stop(2)
4737 stop 2
4738 Feb 12 01:35:27.529 INFO request completed, latency_us: 380, response_code: 204, uri: /stop/cid/2, method: POST, req_id[8830] Got stop action so:false kr:false
4739 : 8383b263-9b6f-4093-8c3c-8aaa03ae0d1f, remote_addr: 127.0.0.1:60456, local_addr: 127.0.0.1:9998
47402024-02-12T01:35:27.530ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
47412024-02-12T01:35:27.530ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
4742 Feb 12 01:35:27.530{"msg":"downstairs task for 2 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-02-12T01:35:27.530413711Z","hostname":"ip-10-150-1-104.us-west-2.compute.internal","pid":1164,"session_id":"e1773b3e-b36f-4080-8b13-c0830f0d072e"}
4743 INFO request completed, latency_us: 150, response_code: 200, uri: /state/cid/2, method: GET, req_id: {"msg":"Gone missing, transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-02-12T01:35:27.530522805Z","hostname":"ip-10-150-1-104.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"e1773b3e-b36f-4080-8b13-c0830f0d072e"}
4744 b4024232-bfff-4c31-a292-7f2129999bb4{"msg":"[2] client re-new 0 jobs since flush 2507","v":0,"name":"crucible","level":30,"time":"2024-02-12T01:35:27.530591675Z","hostname":"ip-10-150-1-104.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"e1773b3e-b36f-4080-8b13-c0830f0d072e"}
4745 , remote_addr: 127.0.0.1:60456, local_addr: 127.0.0.1:9998
47462024-02-12T01:35:27.530ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
4747 [8830] Exited with: ExitStatus(unix_wait_status(9))
4748 [8830][2] reports Exit
47492024-02-12T01:35:28.531ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
47502024-02-12T01:35:28.531ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
47512024-02-12T01:35:28.531ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
47522024-02-12T01:35:28.531ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
47532024-02-12T01:35:28.531ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
47542024-02-12T01:35:28.531ZINFOcrucible: [2] client re-new 0 jobs since flush 2507 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
47552024-02-12T01:35:28.531ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
47562024-02-12T01:35:29.533ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
47572024-02-12T01:35:29.533ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
47582024-02-12T01:35:29.533ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
47592024-02-12T01:35:29.533ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
47602024-02-12T01:35:29.533ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
47612024-02-12T01:35:29.533ZINFOcrucible: [2] client re-new 0 jobs since flush 2507 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
47622024-02-12T01:35:29.533ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
47632024-02-12T01:35:30.535ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
47642024-02-12T01:35:30.535ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
47652024-02-12T01:35:30.535ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
47662024-02-12T01:35:30.535ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
47672024-02-12T01:35:30.535ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
47682024-02-12T01:35:30.535ZINFOcrucible: [2] client re-new 0 jobs since flush 2507 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
47692024-02-12T01:35:30.535ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
4770 Feb 12 01:35:31.532 INFO request completed, latency_us: 226, response_code: 200, uri: /state/cid/2, method: GET, req_id: 38e7c264-0044-4a6d-ad19-86d69f610147, remote_addr: 127.0.0.1:60456, local_addr: 127.0.0.1:9998
4771 001/300 Read block 538 len 28672
4772 002/300 Write block 3791 len 28672 data: 3 3 3 2 2 2 2
4773 003/300 Read block 4622 len 40960
47742024-02-12T01:35:31.536ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
47752024-02-12T01:35:31.536ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
47762024-02-12T01:35:31.536ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
47772024-02-12T01:35:31.536ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
47782024-02-12T01:35:31.536ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
47792024-02-12T01:35:31.536ZINFOcrucible: [2] client re-new 3 jobs since flush 2507 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
47802024-02-12T01:35:31.536ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
4781 004/300 Write block 3378 len 12288 data: 3 3 3
4782 005/300 Read block 2019 len 28672
4783 006/300 Read block 2945 len 36864
4784 007/300 Read block 627 len 16384
4785 008/300 Read block 1934 len 12288
4786 009/300 Flush
4787 010/300 Write block 1073 len 36864 data: 3 3 2 2 2 2 2 2 2
4788 011/300 Read block 4357 len 16384
4789 012/300 Read block 4627 len 24576
4790 013/300 Read block 85 len 40960
4791 014/300 Read block 1556 len 4096
4792 015/300 Write block 3042 len 4096 data: 2
4793 016/300 Write block 2183 len 32768 data: 2 2 2 2 2 2 2 2
4794 017/300 Write block 4184 len 4096 data: 3
4795 018/300 Read block 2041 len 28672
4796 019/300 Read block 2916 len 16384
4797 020/300 Write block 4997 len 4096 data: 2
4798 021/300 Write block 2287 len 36864 data: 2 2 2 2 2 2 2 2 2
4799 022/300 Read block 427 len 28672
4800 023/300 Read block 2661 len 16384
4801 024/300 Write block 513 len 12288 data: 5 5 4
4802 025/300 Read block 4498 len 8192
4803 026/300 Read block 4113 len 36864
4804 027/300 Read block 2633 len 36864
4805 028/300 Write block 1465 len 36864 data: 2 2 2 2 2 2 2 2 2
4806 029/300 Write block 188 len 40960 data: 2 3 3 3 3 3 3 3 3 3
4807 030/300 Write block 2739 len 4096 data: 3
4808 031/300 Read block 608 len 24576
4809 032/300 Write block 2613 len 20480 data: 2 3 3 4 4
4810 033/300 Write block 4282 len 16384 data: 3 3 3 3
4811 034/300 Write block 2470 len 36864 data: 3 3 4 4 4 3 3 3 4
4812 035/300 Write block 1272 len 28672 data: 2 2 3 3 3 3 3
4813 036/300 Read block 777 len 20480
4814 037/300 Write block 3248 len 8192 data: 3 3
4815 038/300 Write block 4254 len 20480 data: 4 4 4 4 4
4816 039/300 Flush
4817 040/300 Write block 4358 len 36864 data: 3 3 3 3 3 3 3 3 2
4818 041/300 Read block 2613 len 4096
4819 042/300 Write block 4157 len 16384 data: 3 2 2 3
4820 043/300 Read block 1671 len 12288
4821 044/300 Write block 3228 len 24576 data: 4 3 3 4 3 3
4822 045/300 Write block 1431 len 36864 data: 2 2 2 2 2 2 2 2 2
4823 046/300 Write block 3418 len 36864 data: 2 2 2 2 2 2 3 3 3
4824 047/300 Read block 1541 len 12288
4825 048/300 Write block 4956 len 12288 data: 4 4 4
4826 049/300 Read block 1925 len 16384
4827 050/300 Read block 2510 len 32768
4828 051/300 Flush
4829 052/300 Read block 1178 len 8192
4830 053/300 Write block 1888 len 32768 data: 3 3 2 2 2 2 2 2
4831 054/300 Read block 2704 len 32768
4832 055/300 Flush
4833 056/300 Read block 3221 len 4096
4834 057/300 Write block 2644 len 36864 data: 3 3 3 3 3 4 3 3 3
4835 058/300 Read block 4861 len 8192
4836 059/300 Write block 1531 len 24576 data: 2 2 3 3 3 3
4837 060/300 Flush
4838 061/300 Write block 3617 len 40960 data: 2 2 2 2 2 2 2 2 2 2
4839 062/300 Flush
4840 063/300 Write block 1091 len 20480 data: 2 2 2 2 2
4841 064/300 Read block 3055 len 12288
4842 065/300 Flush
4843 066/300 Write block 2124 len 40960 data: 2 2 2 2 2 2 2 3 3 2
4844 067/300 Read block 3065 len 16384
4845 068/300 Write block 4627 len 40960 data: 3 3 3 3 3 3 3 3 3 2
4846 069/300 Flush
4847 070/300 Read block 3188 len 28672
4848 071/300 Read block 2574 len 28672
4849 072/300 Read block 4577 len 40960
4850 073/300 Write block 4616 len 8192 data: 2 2
4851 074/300 Flush
4852 075/300 Write block 1554 len 24576 data: 3 2 2 2 3 3
4853 076/300 Read block 413 len 20480
4854 077/300 Write block 2146 len 40960 data: 3 2 2 2 2 2 2 2 2 2
4855 078/300 Read block 3356 len 24576
4856 079/300 Write block 3877 len 24576 data: 4 3 3 2 2 2
4857 080/300 Read block 3839 len 8192
4858 081/300 Read block 1969 len 32768
4859 082/300 Read block 1996 len 4096
4860 083/300 Write block 3640 len 28672 data: 3 2 2 2 2 2 2
4861 084/300 Read block 3928 len 16384
4862 085/300 Read block 4502 len 24576
4863 086/300 Read block 2111 len 20480
4864 087/300 Write block 2835 len 4096 data: 3
4865 088/300 Read block 1293 len 40960
4866 089/300 Flush
4867 090/300 Read block 4849 len 40960
4868 091/300 Write block 1499 len 32768 data: 2 2 2 2 2 2 2 2
4869 092/300 Read block 4053 len 32768
4870 093/300 Read block 1796 len 16384
4871 094/300 Write block 4019 len 32768 data: 3 3 5 5 4 3 3 3
4872 095/300 Read block 2321 len 36864
4873 096/300 Flush
4874 097/300 Write block 4264 len 40960 data: 5 5 4 4 4 3 3 3 3 3
4875 098/300 Write block 4992 len 32768 data: 2 2 2 2 2 3 2 2
4876 099/300 Write block 352 len 28672 data: 2 2 3 2 3 3 3
4877 100/300 Read block 2020 len 32768
4878 101/300 Read block 4867 len 8192
4879 102/300 Write block 1547 len 4096 data: 2
4880 103/300 Write block 393 len 20480 data: 3 3 4 4 4
4881 104/300 Flush
4882 105/300 Read block 1061 len 36864
4883 106/300 Read block 3468 len 12288
4884 107/300 Flush
4885 108/300 Write block 1539 len 36864 data: 5 5 4 4 4 4 3 3 3
4886 109/300 Write block 3470 len 24576 data: 4 4 4 3 3 3
4887 110/300 Read block 4653 len 4096
4888 111/300 Read block 4272 len 32768
4889 112/300 Read block 2059 len 40960
4890 113/300 Write block 3957 len 12288 data: 4 3 3
4891 114/300 Write block 72 len 24576 data: 2 2 2 2 2 2
4892 115/300 Read block 4594 len 8192
4893 116/300 Write block 4012 len 12288 data: 2 2 2
4894 117/300 Write block 3412 len 28672 data: 2 2 2 2 2 2 3
4895 118/300 Read block 3816 len 40960
4896 119/300 Flush
4897 120/300 Read block 652 len 8192
4898 121/300 Write block 3358 len 32768 data: 3 4 4 4 4 4 4 4
4899 122/300 Flush
4900 123/300 Read block 2716 len 24576
4901 124/300 Write block 2850 len 8192 data: 2 2
4902 125/300 Read block 4052 len 8192
4903 126/300 Write block 2233 len 28672 data: 2 2 2 2 2 2 2
4904 127/300 Write block 718 len 4096 data: 2
4905 128/300 Read block 1602 len 40960
4906 129/300 Read block 4354 len 24576
4907 130/300 Write block 2829 len 12288 data: 2 2 2
4908 131/300 Read block 2935 len 32768
4909 132/300 Read block 161 len 40960
4910 133/300 Read block 3571 len 40960
4911 134/300 Write block 4800 len 28672 data: 2 2 2 2 2 2 2
4912 135/300 Write block 4573 len 36864 data: 2 2 2 2 2 2 2 2 2
4913 136/300 Read block 1078 len 40960
4914 137/300 Write block 2586 len 32768 data: 5 5 5 4 4 5 4 4
4915 138/300 Write block 3015 len 8192 data: 2 2
4916 139/300 Write block 1193 len 16384 data: 2 2 2 2
4917 140/300 Flush
4918 141/300 Flush
4919 142/300 Flush
4920 143/300 Read block 2940 len 20480
4921 144/300 Read block 1583 len 28672
4922 145/300 Read block 215 len 36864
4923 146/300 Flush
4924 147/300 Read block 753 len 4096
4925 148/300 Read block 402 len 24576
4926 149/300 Write block 151 len 4096 data: 4
4927 150/300 Read block 2663 len 12288
4928 151/300 Read block 712 len 20480
4929 152/300 Write block 1819 len 20480 data: 3 2 2 3 3
4930 153/300 Read block 4173 len 36864
4931 154/300 Write block 4928 len 32768 data: 2 3 3 2 2 2 2 3
4932 155/300 Read block 3494 len 40960
4933 156/300 Read block 120 len 12288
4934 157/300 Write block 4775 len 28672 data: 2 2 2 2 2 2 2
4935 158/300 Read block 426 len 36864
4936 159/300 Write block 255 len 24576 data: 2 3 3 3 3 3
4937 160/300 Write block 187 len 28672 data: 2 3 4 4 4 4 4
4938 161/300 Write block 379 len 8192 data: 4 4
4939 162/300 Write block 4969 len 4096 data: 4
4940 163/300 Read block 2029 len 8192
4941 164/300 Read block 3663 len 40960
4942 165/300 Write block 1688 len 32768 data: 2 2 2 2 2 2 2 2
4943 166/300 Read block 4260 len 40960
4944 167/300 Read block 1885 len 20480
4945 168/300 Write block 4418 len 40960 data: 3 2 2 2 2 2 2 3 3 3
4946 169/300 Flush
4947 170/300 Write block 1142 len 24576 data: 3 3 3 3 3 3
4948 171/300 Write block 1333 len 32768 data: 3 4 4 2 3 3 2 3
4949 172/300 Flush
4950 173/300 Write block 4022 len 24576 data: 6 5 4 4 4 3
4951 174/300 Write block 3445 len 36864 data: 3 3 3 2 2 4 4 4 4
4952 175/300 Read block 1524 len 8192
4953 176/300 Write block 2829 len 16384 data: 3 3 3 2
4954 177/300 Write block 636 len 24576 data: 3 3 3 3 3 3
4955 178/300 Read block 2180 len 36864
4956 179/300 Read block 4999 len 4096
4957 180/300 Read block 3362 len 28672
4958 181/300 Flush
4959 182/300 Read block 862 len 36864
4960 183/300 Flush
4961 184/300 Write block 751 len 12288 data: 2 2 3
4962 185/300 Flush
4963 186/300 Read block 3731 len 28672
4964 187/300 Write block 3779 len 32768 data: 3 3 2 2 2 2 2 2
4965 188/300 Read block 2005 len 32768
4966 189/300 Read block 3967 len 16384
4967 190/300 Write block 27 len 8192 data: 3 4
4968 191/300 Read block 2228 len 16384
4969 192/300 Write block 2851 len 24576 data: 3 2 3 3 3 3
4970 193/300 Write block 4503 len 24576 data: 2 2 3 3 3 2
4971 194/300 Read block 2389 len 20480
4972 195/300 Read block 3586 len 8192
4973 196/300 Write block 2824 len 28672 data: 4 3 3 2 2 4 4
4974 197/300 Read block 3866 len 4096
4975 198/300 Read block 818 len 20480
4976 199/300 Read block 3018 len 32768
4977 200/300 Read block 4299 len 32768
4978 201/300 Flush
4979 202/300 Read block 770 len 16384
4980 203/300 Read block 4089 len 36864
4981 204/300 Write block 4677 len 20480 data: 5 5 4 4 4
4982 205/300 Read block 4086 len 12288
4983 206/300 Write block 1669 len 40960 data: 2 2 3 3 3 3 3 3 4 5
4984 207/300 Write block 123 len 12288 data: 2 3 3
4985 208/300 Write block 3372 len 40960 data: 2 2 3 3 3 3 4 4 4 3
4986 209/300 Write block 4361 len 28672 data: 4 4 4 4 4 3 2
4987 210/300 Flush
4988 211/300 Write block 4316 len 16384 data: 2 2 2 3
4989 212/300 Write block 881 len 32768 data: 2 2 2 2 2 2 2 2
4990 213/300 Read block 2616 len 24576
4991 214/300 Write block 1477 len 28672 data: 3 3 3 2 2 2 2
4992 215/300 Write block 4957 len 12288 data: 5 5 5
4993 216/300 Flush
4994 217/300 Read block 3067 len 4096
4995 218/300 Read block 354 len 40960
4996 219/300 Read block 3017 len 16384
4997 220/300 Read block 1348 len 12288
4998 221/300 Read block 4404 len 8192
4999 222/300 Flush
5000 223/300 Read block 4298 len 4096
5001 224/300 Flush
5002 225/300 Read block 4158 len 20480
5003 226/300 Read block 322 len 4096
5004 227/300 Write block 2652 len 8192 data: 4 2
5005 228/300 Read block 3923 len 32768
5006 229/300 Read block 4819 len 16384
5007 230/300 Read block 834 len 4096
5008 231/300 Write block 3882 len 36864 data: 3 3 3 3 2 2 2 2 2
5009 232/300 Write block 1304 len 40960 data: 2 2 2 2 2 2 2 2 2 2
5010 233/300 Read block 984 len 16384
5011 234/300 Write block 2839 len 36864 data: 5 4 4 3 3 3 3 2 2
5012 235/300 Write block 3878 len 28672 data: 4 4 3 3 4 4 4
5013 236/300 Read block 1773 len 24576
5014 237/300 Write block 3220 len 4096 data: 4
5015 238/300 Flush
5016 239/300 Read block 632 len 32768
5017 240/300 Read block 826 len 16384
5018 241/300 Write block 3180 len 32768 data: 2 3 3 3 4 4 4 4
5019 242/300 Read block 2262 len 20480
5020 243/300 Read block 4522 len 40960
5021 244/300 Read block 853 len 16384
5022 245/300 Read block 506 len 36864
5023 246/300 Write block 2829 len 36864 data: 5 5 4 3 2 3 4 4 5
5024 247/300 Read block 2533 len 40960
5025 248/300 Read block 2878 len 20480
5026 249/300 Write block 2317 len 32768 data: 3 2 2 3 3 2 2 2
5027 250/300 Write block 2776 len 32768 data: 2 2 2 2 3 3 3 3
5028 251/300 Read block 2972 len 8192
5029 252/300 Write block 315 len 12288 data: 4 4 3
5030 253/300 Read block 4971 len 40960
5031 254/300 Write block 3869 len 40960 data: 3 4 4 4 4 4 4 4 5 5
5032 255/300 Read block 3087 len 28672
5033 256/300 Read block 170 len 32768
5034 257/300 Flush
5035 258/300 Write block 1645 len 4096 data: 2
5036 259/300 Read block 2818 len 12288
5037 260/300 Flush
5038 261/300 Read block 3251 len 40960
5039 262/300 Read block 188 len 28672
5040 263/300 Read block 3792 len 16384
5041 264/300 Write block 2409 len 20480 data: 2 2 2 2 2
5042 265/300 Read block 3682 len 20480
5043 266/300 Read block 3864 len 16384
5044 267/300 Read block 2679 len 24576
5045 268/300 Flush
5046 269/300 Read block 2045 len 20480
5047 270/300 Write block 3142 len 8192 data: 2 2
5048 271/300 Read block 3477 len 36864
5049 272/300 Read block 3693 len 4096
5050 273/300 Write block 4029 len 4096 data: 2
5051 274/300 Write block 2564 len 16384 data: 2 2 2 2
5052 275/300 Flush
5053 276/300 Write block 3304 len 16384 data: 2 2 2 2
5054 277/300 Read block 567 len 8192
5055 278/300 Write block 2441 len 8192 data: 2 2
5056 279/300 Read block 3889 len 4096
5057 280/300 Read block 1660 len 16384
5058 281/300 Write block 494 len 32768 data: 2 2 2 2 2 2 2 2
5059 282/300 Read block 2625 len 8192
5060 283/300 Flush
5061 284/300 Read block 1532 len 16384
5062 285/300 Read block 2314 len 12288
5063 286/300 Read block 3633 len 8192
5064 287/300 Read block 797 len 12288
5065 288/300 Write block 1724 len 40960 data: 2 2 3 3 3 3 2 2 2 2
5066 289/300 Write block 1404 len 20480 data: 2 2 2 2 2
5067 290/300 Flush
5068 291/300 Read block 2964 len 36864
5069 292/300 Write block 3615 len 36864 data: 3 2 3 3 3 3 3 3 3
5070 293/300 Write block 1080 len 32768 data: 3 3 2 2 2 2 3 2
5071 294/300 Read block 3291 len 16384
5072 295/300 Read block 3082 len 36864
5073 296/300 Read block 948 len 40960
5074 297/300 Write block 2984 len 32768 data: 2 2 2 2 2 2 2 2
5075 298/300 Write block 2551 len 20480 data: 2 2 2 2 2
5076 299/300 Write block 4359 len 28672 data: 4 4 5 5 5 5 5
5077 300/300 Read block 3963 len 36864
5078 Main task has work to do, go find it
5079 got dsc Start(2)
5080 start 2
5081 Feb 12 01:35:31.895 INFO request completed, latency_us: 221, response_code: 204, uri: /start/cid/2, method: POST, req_id: 18862ff3-fd91-488d-aec7-4c70a99c0fa2, remote_addr: 127.0.0.1:60456, local_addr: 127.0.0.1:9998
5082 Replay: started 2, returned:Ok(())
5083 ----------------------------------------------------------------
5084 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
5085 States: Active Active Offline
5086 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
5087 1509 Acked 2508 Read 7 Done Done New true
5088 1510 Acked 2509 Write 7 Done Done New true
5089 1511 Acked 2510 Read 10 Done Done New true
5090 1512 Acked 2511 Write 3 Done Done New false
5091 1513 Acked 2512 Read 7 Done Done New false
5092 1514 Acked 2513 Read 9 Done Done New false
5093 1515 Acked 2514 Read 4 Done Done New false
5094 1516 Acked 2515 Read 3 Done Done New false
5095 1517 Acked 2516 Flush 0 Done Done New false
5096 1518 Acked 2517 Write 9 Done Done New false
5097 1519 Acked 2518 Read 4 Done Done New false
5098 1520 Acked 2519 Read 6 Done Done New false
5099 1521 Acked 2520 Read 10 Done Done New false
5100 1522 Acked 2521 Read 1 Done Done New false
5101 1523 Acked 2522 Write 1 Done Done New false
5102 1524 Acked 2523 Write 8 Done Done New false
5103 1525 Acked 2524 Write 1 Done Done New false
5104 1526 Acked 2525 Read 7 Done Done New false
5105 1527 Acked 2526 Read 4 Done Done New false
5106 1528 Acked 2527 Write 1 Done Done New false
5107 1529 Acked 2528 Write 9 Done Done New false
5108 1530 Acked 2529 Read 7 Done Done New false
5109 1531 Acked 2530 Read 4 Done Done New false
5110 1532 Acked 2531 Write 3 Done Done New false
5111 1533 Acked 2532 Read 2 Done Done New false
5112 1534 Acked 2533 Read 9 Done Done New false
5113 1535 Acked 2534 Read 9 Done Done New false
5114 1536 Acked 2535 Write 9 Done Done New false
5115 1537 Acked 2536 Write 10 Done Done New false
5116 1538 Acked 2537 Write 1 Done Done New false
5117 1539 Acked 2538 Read 6 Done Done New false
5118 1540 Acked 2539 Write 5 Done Done New false
5119 1541 Acked 2540 Write 4 Done Done New false
5120 1542 Acked 2541 Write 9 Done Done New false
5121 1543 Acked 2542 Write 7 Done Done New false
5122 1544 Acked 2543 Read 5 Done Done New false
5123 1545 Acked 2544 Write 2 Done Done New false
5124 1546 Acked 2545 Write 5 Done Done New false
5125 1547 Acked 2546 Flush 0 Done Done New false
5126 1548 Acked 2547 Write 9 Done Done New false
5127 1549 Acked 2548 Read 1 Done Done New false
5128 1550 Acked 2549 Write 4 Done Done New false
5129 1551 Acked 2550 Read 3 Done Done New false
5130 1552 Acked 2551 Write 6 Done Done New false
5131 1553 Acked 2552 Write 9 Done Done New false
5132 1554 Acked 2553 Write 9 Done Done New false
5133 1555 Acked 2554 Read 3 Done Done New false
5134 1556 Acked 2555 Write 3 Done Done New false
5135 1557 Acked 2556 Read 4 Done Done New false
5136 1558 Acked 2557 Read 8 Done Done New false
5137 1559 Acked 2558 Flush 0 Done Done New false
5138 1560 Acked 2559 Read 2 Done Done New false
5139 1561 Acked 2560 Write 8 Done Done New false
5140 1562 Acked 2561 Read 8 Done Done New false
5141 1563 Acked 2562 Flush 0 Done Done New false
5142 1564 Acked 2563 Read 1 Done Done New false
5143 1565 Acked 2564 Write 9 Done Done New false
5144 1566 Acked 2565 Read 2 Done Done New false
5145 1567 Acked 2566 Write 6 Done Done New false
5146 1568 Acked 2567 Flush 0 Done Done New false
5147 1569 Acked 2568 Write 10 Done Done New false
5148 1570 Acked 2569 Flush 0 Done Done New false
5149 1571 Acked 2570 Write 5 Done Done New false
5150 1572 Acked 2571 Read 3 Done Done New false
5151 1573 Acked 2572 Flush 0 Done Done New false
5152 1574 Acked 2573 Write 10 Done Done New false
5153 1575 Acked 2574 Read 4 Done Done New false
5154 1576 Acked 2575 Write 10 Done Done New false
5155 1577 Acked 2576 Flush 0 Done Done New false
5156 1578 Acked 2577 Read 7 Done Done New false
5157 1579 Acked 2578 Read 7 Done Done New false
5158 1580 Acked 2579 Read 10 Done Done New false
5159 1581 Acked 2580 Write 2 Done Done New false
5160 1582 Acked 2581 Flush 0 Done Done New false
5161 1583 Acked 2582 Write 6 Done Done New false
5162 1584 Acked 2583 Read 5 Done Done New false
5163 1585 Acked 2584 Write 10 Done Done New false
5164 1586 Acked 2585 Read 6 Done Done New false
5165 1587 Acked 2586 Write 6 Done Done New false
5166 1588 Acked 2587 Read 2 Done Done New false
5167 1589 Acked 2588 Read 8 Done Done New false
5168 1590 Acked 2589 Read 1 Done Done New false
5169 1591 Acked 2590 Write 7 Done Done New false
5170 1592 Acked 2591 Read 4 Done Done New false
5171 1593 Acked 2592 Read 6 Done Done New false
5172 1594 Acked 2593 Read 5 Done Done New false
5173 1595 Acked 2594 Write 1 Done Done New false
5174 1596 Acked 2595 Read 10 Done Done New false
5175 1597 Acked 2596 Flush 0 Done Done New false
5176 1598 Acked 2597 Read 10 Done Done New false
5177 1599 Acked 2598 Write 8 Done Done New false
5178 1600 Acked 2599 Read 8 Done Done New false
5179 1601 Acked 2600 Read 4 Done Done New false
5180 1602 Acked 2601 Write 8 Done Done New false
5181 1603 Acked 2602 Read 9 Done Done New false
5182 1604 Acked 2603 Flush 0 Done Done New false
5183 1605 Acked 2604 Write 10 Done Done New false
5184 1606 Acked 2605 Write 8 Done Done New false
5185 1607 Acked 2606 Write 7 Done Done New false
5186 1608 Acked 2607 Read 8 Done Done New false
5187 1609 Acked 2608 Read 2 Done Done New false
5188 1610 Acked 2609 Write 1 Done Done New false
5189 1611 Acked 2610 Write 5 Done Done New false
5190 1612 Acked 2611 Flush 0 Done Done New false
5191 1613 Acked 2612 Read 9 Done Done New false
5192 1614 Acked 2613 Read 3 Done Done New false
5193 1615 Acked 2614 Flush 0 Done Done New false
5194 1616 Acked 2615 Write 9 Done Done New false
5195 1617 Acked 2616 Write 6 Done Done New false
5196 1618 Acked 2617 Read 1 Done Done New false
5197 1619 Acked 2618 Read 8 Done Done New false
5198 1620 Acked 2619 Read 10 Done Done New false
5199 1621 Acked 2620 Write 3 Done Done New false
5200 1622 Acked 2621 Write 6 Done Done New false
5201 1623 Acked 2622 Read 2 Done Done New false
5202 1624 Acked 2623 Write 3 Done Done New false
5203 1625 Acked 2624 Write 7 Done Done New false
5204 1626 Acked 2625 Read 10 Done Done New false
5205 1627 Acked 2626 Flush 0 Done Done New false
5206 1628 Acked 2627 Read 2 Done Done New false
5207 1629 Acked 2628 Write 8 Done Done New false
5208 1630 Acked 2629 Flush 0 Done Done New false
5209 1631 Acked 2630 Read 6 Done Done New false
5210 1632 Acked 2631 Write 2 Done Done New false
5211 1633 Acked 2632 Read 2 Done Done New false
5212 1634 Acked 2633 Write 7 Done Done New false
5213 1635 Acked 2634 Write 1 Done Done New false
5214 1636 Acked 2635 Read 10 Done Done New false
5215 1637 Acked 2636 Read 6 Done Done New false
5216 1638 Acked 2637 Write 3 Done Done New false
5217 1639 Acked 2638 Read 8 Done Done New false
5218 1640 Acked 2639 Read 10 Done Done New false
5219 1641 Acked 2640 Read 10 Done Done New false
5220 1642 Acked 2641 Write 7 Done Done New false
5221 1643 Acked 2642 Write 9 Done Done New false
5222 1644 Acked 2643 Read 10 Done Done New false
5223 1645 Acked 2644 Write 8 Done Done New false
5224 1646 Acked 2645 Write 2 Done Done New false
5225 1647 Acked 2646 Write 4 Done Done New false
5226 1648 Acked 2647 Flush 0 Done Done New false
5227 1649 Acked 2648 Flush 0 Done Done New false
5228 1650 Acked 2649 Flush 0 Done Done New false
5229 1651 Acked 2650 Read 5 Done Done New false
5230 1652 Acked 2651 Read 7 Done Done New false
5231 1653 Acked 2652 Read 9 Done Done New false
5232 1654 Acked 2653 Flush 0 Done Done New false
5233 1655 Acked 2654 Read 1 Done Done New false
5234 1656 Acked 2655 Read 6 Done Done New false
5235 1657 Acked 2656 Write 1 Done Done New false
5236 1658 Acked 2657 Read 3 Done Done New false
5237 1659 Acked 2658 Read 5 Done Done New false
5238 1660 Acked 2659 Write 5 Done Done New false
5239 1661 Acked 2660 Read 9 Done Done New false
5240 1662 Acked 2661 Write 8 Done Done New false
5241 1663 Acked 2662 Read 10 Done Done New false
5242 1664 Acked 2663 Read 3 Done Done New false
5243 1665 Acked 2664 Write 7 Done Done New false
5244 1666 Acked 2665 Read 9 Done Done New false
5245 1667 Acked 2666 Write 6 Done Done New false
5246 1668 Acked 2667 Write 7 Done Done New false
5247 1669 Acked 2668 Write 2 Done Done New false
5248 1670 Acked 2669 Write 1 Done Done New false
5249 1671 Acked 2670 Read 2 Done Done New false
5250 1672 Acked 2671 Read 10 Done Done New false
5251 1673 Acked 2672 Write 8 Done Done New false
5252 1674 Acked 2673 Read 10 Done Done New false
5253 1675 Acked 2674 Read 5 Done Done New false
5254 1676 Acked 2675 Write 10 Done Done New false
5255 1677 Acked 2676 Flush 0 Done Done New false
5256 1678 Acked 2677 Write 6 Done Done New false
5257 1679 Acked 2678 Write 8 Done Done New false
5258 1680 Acked 2679 Flush 0 Done Done New false
5259 1681 Acked 2680 Write 6 Done Done New false
5260 1682 Acked 2681 Write 9 Done Done New false
5261 1683 Acked 2682 Read 2 Done Done New false
5262 1684 Acked 2683 Write 4 Done Done New false
5263 1685 Acked 2684 Write 6 Done Done New false
5264 1686 Acked 2685 Read 9 Done Done New false
5265 1687 Acked 2686 Read 1 Done Done New false
5266 1688 Acked 2687 Read 7 Done Done New false
5267 1689 Acked 2688 Flush 0 Done Done New false
5268 1690 Acked 2689 Read 9 Done Done New false
5269 1691 Acked 2690 Flush 0 Done Done New false
5270 1692 Acked 2691 Write 3 Done Done New false
5271 1693 Acked 2692 Flush 0 Done Done New false
5272 1694 Acked 2693 Read 7 Done Done New false
5273 1695 Acked 2694 Write 8 Done Done New false
5274 1696 Acked 2695 Read 8 Done Done New false
5275 1697 Acked 2696 Read 4 Done Done New false
5276 1698 Acked 2697 Write 2 Done Done New false
5277 1699 Acked 2698 Read 4 Done Done New false
5278 1700 Acked 2699 Write 6 Done Done New false
5279 1701 Acked 2700 Write 6 Done Done New false
5280 1702 Acked 2701 Read 5 Done Done New false
5281 1703 Acked 2702 Read 2 Done Done New false
5282 1704 Acked 2703 Write 7 Done Done New false
5283 1705 Acked 2704 Read 1 Done Done New false
5284 1706 Acked 2705 Read 5 Done Done New false
5285 1707 Acked 2706 Read 8 Done Done New false
5286 1708 Acked 2707 Read 8 Done Done New false
5287 1709 Acked 2708 Flush 0 Done Done New false
5288 1710 Acked 2709 Read 4 Done Done New false
5289 1711 Acked 2710 Read 9 Done Done New false
5290 1712 Acked 2711 Write 5 Done Done New false
5291 1713 Acked 2712 Read 3 Done Done New false
5292 1714 Acked 2713 Write 10 Done Done New false
5293 1715 Acked 2714 Write 3 Done Done New false
5294 1716 Acked 2715 Write 10 Done Done New false
5295 1717 Acked 2716 Write 7 Done Done New false
5296 1718 Acked 2717 Flush 0 Done Done New false
5297 1719 Acked 2718 Write 4 Done Done New false
5298 1720 Acked 2719 Write 8 Done Done New false
5299 1721 Acked 2720 Read 6 Done Done New false
5300 1722 Acked 2721 Write 7 Done Done New false
5301 1723 Acked 2722 Write 3 Done Done New false
5302 1724 Acked 2723 Flush 0 Done Done New false
5303 1725 Acked 2724 Read 1 Done Done New false
5304 1726 Acked 2725 Read 10 Done Done New false
5305 1727 Acked 2726 Read 4 Done Done New false
5306 1728 Acked 2727 Read 3 Done Done New false
5307 1729 Acked 2728 Read 2 Done Done New false
5308 1730 Acked 2729 Flush 0 Done Done New false
5309 1731 Acked 2730 Read 1 Done Done New false
5310 1732 Acked 2731 Flush 0 Done Done New false
5311 1733 Acked 2732 Read 5 Done Done New false
5312 1734 Acked 2733 Flush 0 Done Done New false
5313 1735 Acked 2734 Read 1 Done Done New false
5314 1736 Acked 2735 Write 2 Done Done New false
5315 1737 Acked 2736 Read 8 Done Done New false
5316 1738 Acked 2737 Read 4 Done Done New false
5317 1739 Acked 2738 Read 1 Done Done New false
5318 1740 Acked 2739 Write 9 Done Done New false
5319 1741 Acked 2740 Write 10 Done Done New false
5320 1742 Acked 2741 Read 4 Done Done New false
5321 1743 Acked 2742 Write 9 Done Done New false
5322 1744 Acked 2743 Write 7 Done Done New false
5323 1745 Acked 2744 Read 6 Done Done New false
5324 1746 Acked 2745 Write 1 Done Done New false
5325 1747 Acked 2746 Flush 0 Done Done New false
5326 1748 Acked 2747 Read 8 Done Done New false
5327 1749 Acked 2748 Read 4 Done Done New false
5328 1750 Acked 2749 Write 8 Done Done New false
5329 1751 Acked 2750 Read 5 Done Done New false
5330 1752 Acked 2751 Read 10 Done Done New false
5331 1753 Acked 2752 Read 4 Done Done New false
5332 1754 Acked 2753 Read 9 Done Done New false
5333 1755 Acked 2754 Write 9 Done Done New false
5334 1756 Acked 2755 Read 10 Done Done New false
5335 1757 Acked 2756 Read 5 Done Done New false
5336 1758 Acked 2757 Write 8 Done Done New false
5337 1759 Acked 2758 Write 8 Done Done New false
5338 1760 Acked 2759 Read 2 Done Done New false
5339 1761 Acked 2760 Write 3 Done Done New false
5340 1762 Acked 2761 Read 10 Done Done New false
5341 1763 Acked 2762 Write 10 Done Done New false
5342 1764 Acked 2763 Read 7 Done Done New false
5343 1765 Acked 2764 Read 8 Done Done New false
5344 1766 Acked 2765 Flush 0 Done Done New false
5345 1767 Acked 2766 Write 1 Done Done New false
5346 1768 Acked 2767 Read 3 Done Done New false
5347 1769 Acked 2768 Flush 0 Done Done New false
5348 1770 Acked 2769 Read 10 Done Done New false
5349 1771 Acked 2770 Read 7 Done Done New false
5350 1772 Acked 2771 Read 4 Done Done New false
5351 1773 Acked 2772 Write 5 Done Done New false
5352 1774 Acked 2773 Read 5 Done Done New false
5353 1775 Acked 2774 Read 4 Done Done New false
5354 1776 Acked 2775 Read 6 Done Done New false
5355 1777 Acked 2776 Flush 0 Done Done New false
5356 1778 Acked 2777 Read 5 Done Done New false
5357 1779 Acked 2778 Write 2 Done Done New false
5358 1780 Acked 2779 Read 9 Done Done New false
5359 1781 Acked 2780 Read 1 Done Done New false
5360 1782 Acked 2781 Write 1 Done Done New false
5361 1783 Acked 2782 Write 4 Done Done New false
5362 1784 Acked 2783 Flush 0 Done Done New false
5363 1785 Acked 2784 Write 4 Done Done New false
5364 1786 Acked 2785 Read 2 Done Done New false
5365 1787 Acked 2786 Write 2 Done Done New false
5366 1788 Acked 2787 Read 1 Done Done New false
5367 1789 Acked 2788 Read 4 Done Done New false
5368 1790 Acked 2789 Write 8 Done Done New false
5369 1791 Acked 2790 Read 2 Done Done New false
5370 1792 Acked 2791 Flush 0 Done Done New false
5371 1793 Acked 2792 Read 4 Done Done New false
5372 1794 Acked 2793 Read 3 Done Done New false
5373 1795 Acked 2794 Read 2 Done Done New false
5374 1796 Acked 2795 Read 3 Done Done New false
5375 1797 Acked 2796 Write 10 Done Done New false
5376 1798 Acked 2797 Write 5 Done Done New false
5377 1799 Acked 2798 Flush 0 Done Done New false
5378 1800 Acked 2799 Read 9 Done Done New false
5379 1801 Acked 2800 Write 9 Done Done New false
5380 1802 Acked 2801 Write 8 Done Done New false
5381 1803 Acked 2802 Read 4 Done Done New false
5382 1804 Acked 2803 Read 9 Done Done New false
5383 1805 Acked 2804 Read 10 Done Done New false
5384 1806 Acked 2805 Write 8 Done Done New false
5385 1807 Acked 2806 Write 5 Done Done New false
5386 1808 Acked 2807 Write 7 Done Done New false
5387 1809 Acked 2808 Read 9 Done Done New false
5388 STATES DS:0 DS:1 DS:2 TOTAL
5389 New 0 0 301 301
5390 Sent 0 0 0 0
5391 Done 301 301 0 602
5392 Skipped 0 0 0 0
5393 Error 0 0 0 0
5394 Last Flush: 2798 2798 2507
5395 Downstairs last five completed: 2507 2506 2505 2504 2503
5396 Upstairs last five completed: 1809 1808 1807 1806 1805
5397 CLIENT: Up:0 ds:301 act:2
53982024-02-12T01:35:32.537ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
53992024-02-12T01:35:32.537ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
54002024-02-12T01:35:32.537ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
54012024-02-12T01:35:32.537ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
54022024-02-12T01:35:32.537ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
54032024-02-12T01:35:32.537ZINFOcrucible: [2] client re-new 302 jobs since flush 2507 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
54042024-02-12T01:35:32.538ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
54052024-02-12T01:35:33.538ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
54062024-02-12T01:35:33.539ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
54072024-02-12T01:35:33.539ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
54082024-02-12T01:35:33.539ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
54092024-02-12T01:35:33.539ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
54102024-02-12T01:35:33.539ZINFOcrucible: [2] client re-new 302 jobs since flush 2507 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
54112024-02-12T01:35:33.539ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
54122024-02-12T01:35:34.541ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
54132024-02-12T01:35:34.541ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
54142024-02-12T01:35:34.541ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
54152024-02-12T01:35:34.541ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
54162024-02-12T01:35:34.541ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
54172024-02-12T01:35:34.541ZINFOcrucible: [2] client re-new 302 jobs since flush 2507 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
54182024-02-12T01:35:34.541ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
54192024-02-12T01:35:35.542ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
54202024-02-12T01:35:35.542ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
54212024-02-12T01:35:35.542ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
54222024-02-12T01:35:35.542ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
54232024-02-12T01:35:35.542ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
54242024-02-12T01:35:35.542ZINFOcrucible: [2] client re-new 302 jobs since flush 2507 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
54252024-02-12T01:35:35.542ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
5426 ----------------------------------------------------------------
5427 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
5428 States: Active Active Offline
5429 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
5430 1509 Acked 2508 Read 7 Done Done New true
5431 1510 Acked 2509 Write 7 Done Done New true
5432 1511 Acked 2510 Read 10 Done Done New true
5433 1512 Acked 2511 Write 3 Done Done New true
5434 1513 Acked 2512 Read 7 Done Done New true
5435 1514 Acked 2513 Read 9 Done Done New true
5436 1515 Acked 2514 Read 4 Done Done New true
5437 1516 Acked 2515 Read 3 Done Done New true
5438 1517 Acked 2516 Flush 0 Done Done New true
5439 1518 Acked 2517 Write 9 Done Done New true
5440 1519 Acked 2518 Read 4 Done Done New true
5441 1520 Acked 2519 Read 6 Done Done New true
5442 1521 Acked 2520 Read 10 Done Done New true
5443 1522 Acked 2521 Read 1 Done Done New true
5444 1523 Acked 2522 Write 1 Done Done New true
5445 1524 Acked 2523 Write 8 Done Done New true
5446 1525 Acked 2524 Write 1 Done Done New true
5447 1526 Acked 2525 Read 7 Done Done New true
5448 1527 Acked 2526 Read 4 Done Done New true
5449 1528 Acked 2527 Write 1 Done Done New true
5450 1529 Acked 2528 Write 9 Done Done New true
5451 1530 Acked 2529 Read 7 Done Done New true
5452 1531 Acked 2530 Read 4 Done Done New true
5453 1532 Acked 2531 Write 3 Done Done New true
5454 1533 Acked 2532 Read 2 Done Done New true
5455 1534 Acked 2533 Read 9 Done Done New true
5456 1535 Acked 2534 Read 9 Done Done New true
5457 1536 Acked 2535 Write 9 Done Done New true
5458 1537 Acked 2536 Write 10 Done Done New true
5459 1538 Acked 2537 Write 1 Done Done New true
5460 1539 Acked 2538 Read 6 Done Done New true
5461 1540 Acked 2539 Write 5 Done Done New true
5462 1541 Acked 2540 Write 4 Done Done New true
5463 1542 Acked 2541 Write 9 Done Done New true
5464 1543 Acked 2542 Write 7 Done Done New true
5465 1544 Acked 2543 Read 5 Done Done New true
5466 1545 Acked 2544 Write 2 Done Done New true
5467 1546 Acked 2545 Write 5 Done Done New true
5468 1547 Acked 2546 Flush 0 Done Done New true
5469 1548 Acked 2547 Write 9 Done Done New true
5470 1549 Acked 2548 Read 1 Done Done New true
5471 1550 Acked 2549 Write 4 Done Done New true
5472 1551 Acked 2550 Read 3 Done Done New true
5473 1552 Acked 2551 Write 6 Done Done New true
5474 1553 Acked 2552 Write 9 Done Done New true
5475 1554 Acked 2553 Write 9 Done Done New true
5476 1555 Acked 2554 Read 3 Done Done New true
5477 1556 Acked 2555 Write 3 Done Done New true
5478 1557 Acked 2556 Read 4 Done Done New true
5479 1558 Acked 2557 Read 8 Done Done New true
5480 1559 Acked 2558 Flush 0 Done Done New true
5481 1560 Acked 2559 Read 2 Done Done New true
5482 1561 Acked 2560 Write 8 Done Done New true
5483 1562 Acked 2561 Read 8 Done Done New true
5484 1563 Acked 2562 Flush 0 Done Done New true
5485 1564 Acked 2563 Read 1 Done Done New true
5486 1565 Acked 2564 Write 9 Done Done New true
5487 1566 Acked 2565 Read 2 Done Done New true
5488 1567 Acked 2566 Write 6 Done Done New true
5489 1568 Acked 2567 Flush 0 Done Done New true
5490 1569 Acked 2568 Write 10 Done Done New true
5491 1570 Acked 2569 Flush 0 Done Done New true
5492 1571 Acked 2570 Write 5 Done Done New true
5493 1572 Acked 2571 Read 3 Done Done New true
5494 1573 Acked 2572 Flush 0 Done Done New true
5495 1574 Acked 2573 Write 10 Done Done New true
5496 1575 Acked 2574 Read 4 Done Done New true
5497 1576 Acked 2575 Write 10 Done Done New true
5498 1577 Acked 2576 Flush 0 Done Done New true
5499 1578 Acked 2577 Read 7 Done Done New true
5500 1579 Acked 2578 Read 7 Done Done New true
5501 1580 Acked 2579 Read 10 Done Done New true
5502 1581 Acked 2580 Write 2 Done Done New true
5503 1582 Acked 2581 Flush 0 Done Done New true
5504 1583 Acked 2582 Write 6 Done Done New true
5505 1584 Acked 2583 Read 5 Done Done New true
5506 1585 Acked 2584 Write 10 Done Done New true
5507 1586 Acked 2585 Read 6 Done Done New true
5508 1587 Acked 2586 Write 6 Done Done New true
5509 1588 Acked 2587 Read 2 Done Done New true
5510 1589 Acked 2588 Read 8 Done Done New true
5511 1590 Acked 2589 Read 1 Done Done New true
5512 1591 Acked 2590 Write 7 Done Done New true
5513 1592 Acked 2591 Read 4 Done Done New true
5514 1593 Acked 2592 Read 6 Done Done New true
5515 1594 Acked 2593 Read 5 Done Done New true
5516 1595 Acked 2594 Write 1 Done Done New true
5517 1596 Acked 2595 Read 10 Done Done New true
5518 1597 Acked 2596 Flush 0 Done Done New true
5519 1598 Acked 2597 Read 10 Done Done New true
5520 1599 Acked 2598 Write 8 Done Done New true
5521 1600 Acked 2599 Read 8 Done Done New true
5522 1601 Acked 2600 Read 4 Done Done New true
5523 1602 Acked 2601 Write 8 Done Done New true
5524 1603 Acked 2602 Read 9 Done Done New true
5525 1604 Acked 2603 Flush 0 Done Done New true
5526 1605 Acked 2604 Write 10 Done Done New true
5527 1606 Acked 2605 Write 8 Done Done New true
5528 1607 Acked 2606 Write 7 Done Done New true
5529 1608 Acked 2607 Read 8 Done Done New true
5530 1609 Acked 2608 Read 2 Done Done New true
5531 1610 Acked 2609 Write 1 Done Done New true
5532 1611 Acked 2610 Write 5 Done Done New true
5533 1612 Acked 2611 Flush 0 Done Done New true
5534 1613 Acked 2612 Read 9 Done Done New true
5535 1614 Acked 2613 Read 3 Done Done New true
5536 1615 Acked 2614 Flush 0 Done Done New true
5537 1616 Acked 2615 Write 9 Done Done New true
5538 1617 Acked 2616 Write 6 Done Done New true
5539 1618 Acked 2617 Read 1 Done Done New true
5540 1619 Acked 2618 Read 8 Done Done New true
5541 1620 Acked 2619 Read 10 Done Done New true
5542 1621 Acked 2620 Write 3 Done Done New true
5543 1622 Acked 2621 Write 6 Done Done New true
5544 1623 Acked 2622 Read 2 Done Done New true
5545 1624 Acked 2623 Write 3 Done Done New true
5546 1625 Acked 2624 Write 7 Done Done New true
5547 1626 Acked 2625 Read 10 Done Done New true
5548 1627 Acked 2626 Flush 0 Done Done New true
5549 1628 Acked 2627 Read 2 Done Done New true
5550 1629 Acked 2628 Write 8 Done Done New true
5551 1630 Acked 2629 Flush 0 Done Done New true
5552 1631 Acked 2630 Read 6 Done Done New true
5553 1632 Acked 2631 Write 2 Done Done New true
5554 1633 Acked 2632 Read 2 Done Done New true
5555 1634 Acked 2633 Write 7 Done Done New true
5556 1635 Acked 2634 Write 1 Done Done New true
5557 1636 Acked 2635 Read 10 Done Done New true
5558 1637 Acked 2636 Read 6 Done Done New true
5559 1638 Acked 2637 Write 3 Done Done New true
5560 1639 Acked 2638 Read 8 Done Done New true
5561 1640 Acked 2639 Read 10 Done Done New true
5562 1641 Acked 2640 Read 10 Done Done New true
5563 1642 Acked 2641 Write 7 Done Done New true
5564 1643 Acked 2642 Write 9 Done Done New true
5565 1644 Acked 2643 Read 10 Done Done New true
5566 1645 Acked 2644 Write 8 Done Done New true
5567 1646 Acked 2645 Write 2 Done Done New true
5568 1647 Acked 2646 Write 4 Done Done New true
5569 1648 Acked 2647 Flush 0 Done Done New true
5570 1649 Acked 2648 Flush 0 Done Done New true
5571 1650 Acked 2649 Flush 0 Done Done New true
5572 1651 Acked 2650 Read 5 Done Done New true
5573 1652 Acked 2651 Read 7 Done Done New true
5574 1653 Acked 2652 Read 9 Done Done New true
5575 1654 Acked 2653 Flush 0 Done Done New true
5576 1655 Acked 2654 Read 1 Done Done New true
5577 1656 Acked 2655 Read 6 Done Done New true
5578 1657 Acked 2656 Write 1 Done Done New true
5579 1658 Acked 2657 Read 3 Done Done New true
5580 1659 Acked 2658 Read 5 Done Done New true
5581 1660 Acked 2659 Write 5 Done Done New true
5582 1661 Acked 2660 Read 9 Done Done New true
5583 1662 Acked 2661 Write 8 Done Done New true
5584 1663 Acked 2662 Read 10 Done Done New true
5585 1664 Acked 2663 Read 3 Done Done New true
5586 1665 Acked 2664 Write 7 Done Done New true
5587 1666 Acked 2665 Read 9 Done Done New true
5588 1667 Acked 2666 Write 6 Done Done New true
5589 1668 Acked 2667 Write 7 Done Done New true
5590 1669 Acked 2668 Write 2 Done Done New true
5591 1670 Acked 2669 Write 1 Done Done New true
5592 1671 Acked 2670 Read 2 Done Done New true
5593 1672 Acked 2671 Read 10 Done Done New true
5594 1673 Acked 2672 Write 8 Done Done New true
5595 1674 Acked 2673 Read 10 Done Done New true
5596 1675 Acked 2674 Read 5 Done Done New true
5597 1676 Acked 2675 Write 10 Done Done New true
5598 1677 Acked 2676 Flush 0 Done Done New true
5599 1678 Acked 2677 Write 6 Done Done New true
5600 1679 Acked 2678 Write 8 Done Done New true
5601 1680 Acked 2679 Flush 0 Done Done New true
5602 1681 Acked 2680 Write 6 Done Done New true
5603 1682 Acked 2681 Write 9 Done Done New true
5604 1683 Acked 2682 Read 2 Done Done New true
5605 1684 Acked 2683 Write 4 Done Done New true
5606 1685 Acked 2684 Write 6 Done Done New true
5607 1686 Acked 2685 Read 9 Done Done New true
5608 1687 Acked 2686 Read 1 Done Done New true
5609 1688 Acked 2687 Read 7 Done Done New true
5610 1689 Acked 2688 Flush 0 Done Done New true
5611 1690 Acked 2689 Read 9 Done Done New true
5612 1691 Acked 2690 Flush 0 Done Done New true
5613 1692 Acked 2691 Write 3 Done Done New true
5614 1693 Acked 2692 Flush 0 Done Done New true
5615 1694 Acked 2693 Read 7 Done Done New true
5616 1695 Acked 2694 Write 8 Done Done New true
5617 1696 Acked 2695 Read 8 Done Done New true
5618 1697 Acked 2696 Read 4 Done Done New true
5619 1698 Acked 2697 Write 2 Done Done New true
5620 1699 Acked 2698 Read 4 Done Done New true
5621 1700 Acked 2699 Write 6 Done Done New true
5622 1701 Acked 2700 Write 6 Done Done New true
5623 1702 Acked 2701 Read 5 Done Done New true
5624 1703 Acked 2702 Read 2 Done Done New true
5625 1704 Acked 2703 Write 7 Done Done New true
5626 1705 Acked 2704 Read 1 Done Done New true
5627 1706 Acked 2705 Read 5 Done Done New true
5628 1707 Acked 2706 Read 8 Done Done New true
5629 1708 Acked 2707 Read 8 Done Done New true
5630 1709 Acked 2708 Flush 0 Done Done New true
5631 1710 Acked 2709 Read 4 Done Done New true
5632 1711 Acked 2710 Read 9 Done Done New true
5633 1712 Acked 2711 Write 5 Done Done New true
5634 1713 Acked 2712 Read 3 Done Done New true
5635 1714 Acked 2713 Write 10 Done Done New true
5636 1715 Acked 2714 Write 3 Done Done New true
5637 1716 Acked 2715 Write 10 Done Done New true
5638 1717 Acked 2716 Write 7 Done Done New true
5639 1718 Acked 2717 Flush 0 Done Done New true
5640 1719 Acked 2718 Write 4 Done Done New true
5641 1720 Acked 2719 Write 8 Done Done New true
5642 1721 Acked 2720 Read 6 Done Done New true
5643 1722 Acked 2721 Write 7 Done Done New true
5644 1723 Acked 2722 Write 3 Done Done New true
5645 1724 Acked 2723 Flush 0 Done Done New true
5646 1725 Acked 2724 Read 1 Done Done New true
5647 1726 Acked 2725 Read 10 Done Done New true
5648 1727 Acked 2726 Read 4 Done Done New true
5649 1728 Acked 2727 Read 3 Done Done New true
5650 1729 Acked 2728 Read 2 Done Done New true
5651 1730 Acked 2729 Flush 0 Done Done New true
5652 1731 Acked 2730 Read 1 Done Done New true
5653 1732 Acked 2731 Flush 0 Done Done New true
5654 1733 Acked 2732 Read 5 Done Done New true
5655 1734 Acked 2733 Flush 0 Done Done New true
5656 1735 Acked 2734 Read 1 Done Done New true
5657 1736 Acked 2735 Write 2 Done Done New true
5658 1737 Acked 2736 Read 8 Done Done New true
5659 1738 Acked 2737 Read 4 Done Done New true
5660 1739 Acked 2738 Read 1 Done Done New true
5661 1740 Acked 2739 Write 9 Done Done New true
5662 1741 Acked 2740 Write 10 Done Done New true
5663 1742 Acked 2741 Read 4 Done Done New true
5664 1743 Acked 2742 Write 9 Done Done New true
5665 1744 Acked 2743 Write 7 Done Done New true
5666 1745 Acked 2744 Read 6 Done Done New true
5667 1746 Acked 2745 Write 1 Done Done New true
5668 1747 Acked 2746 Flush 0 Done Done New true
5669 1748 Acked 2747 Read 8 Done Done New true
5670 1749 Acked 2748 Read 4 Done Done New true
5671 1750 Acked 2749 Write 8 Done Done New true
5672 1751 Acked 2750 Read 5 Done Done New true
5673 1752 Acked 2751 Read 10 Done Done New true
5674 1753 Acked 2752 Read 4 Done Done New true
5675 1754 Acked 2753 Read 9 Done Done New true
5676 1755 Acked 2754 Write 9 Done Done New true
5677 1756 Acked 2755 Read 10 Done Done New true
5678 1757 Acked 2756 Read 5 Done Done New true
5679 1758 Acked 2757 Write 8 Done Done New true
5680 1759 Acked 2758 Write 8 Done Done New true
5681 1760 Acked 2759 Read 2 Done Done New true
5682 1761 Acked 2760 Write 3 Done Done New true
5683 1762 Acked 2761 Read 10 Done Done New true
5684 1763 Acked 2762 Write 10 Done Done New true
5685 1764 Acked 2763 Read 7 Done Done New true
5686 1765 Acked 2764 Read 8 Done Done New true
5687 1766 Acked 2765 Flush 0 Done Done New true
5688 1767 Acked 2766 Write 1 Done Done New true
5689 1768 Acked 2767 Read 3 Done Done New true
5690 1769 Acked 2768 Flush 0 Done Done New true
5691 1770 Acked 2769 Read 10 Done Done New true
5692 1771 Acked 2770 Read 7 Done Done New true
5693 1772 Acked 2771 Read 4 Done Done New true
5694 1773 Acked 2772 Write 5 Done Done New true
5695 1774 Acked 2773 Read 5 Done Done New true
5696 1775 Acked 2774 Read 4 Done Done New true
5697 1776 Acked 2775 Read 6 Done Done New true
5698 1777 Acked 2776 Flush 0 Done Done New true
5699 1778 Acked 2777 Read 5 Done Done New true
5700 1779 Acked 2778 Write 2 Done Done New true
5701 1780 Acked 2779 Read 9 Done Done New true
5702 1781 Acked 2780 Read 1 Done Done New true
5703 1782 Acked 2781 Write 1 Done Done New true
5704 1783 Acked 2782 Write 4 Done Done New true
5705 1784 Acked 2783 Flush 0 Done Done New true
5706 1785 Acked 2784 Write 4 Done Done New true
5707 1786 Acked 2785 Read 2 Done Done New true
5708 1787 Acked 2786 Write 2 Done Done New true
5709 1788 Acked 2787 Read 1 Done Done New true
5710 1789 Acked 2788 Read 4 Done Done New true
5711 1790 Acked 2789 Write 8 Done Done New true
5712 1791 Acked 2790 Read 2 Done Done New true
5713 1792 Acked 2791 Flush 0 Done Done New true
5714 1793 Acked 2792 Read 4 Done Done New true
5715 1794 Acked 2793 Read 3 Done Done New true
5716 1795 Acked 2794 Read 2 Done Done New true
5717 1796 Acked 2795 Read 3 Done Done New true
5718 1797 Acked 2796 Write 10 Done Done New true
5719 1798 Acked 2797 Write 5 Done Done New true
5720 1799 Acked 2798 Flush 0 Done Done New true
5721 1800 Acked 2799 Read 9 Done Done New true
5722 1801 Acked 2800 Write 9 Done Done New true
5723 1802 Acked 2801 Write 8 Done Done New true
5724 1803 Acked 2802 Read 4 Done Done New true
5725 1804 Acked 2803 Read 9 Done Done New true
5726 1805 Acked 2804 Read 10 Done Done New true
5727 1806 Acked 2805 Write 8 Done Done New true
5728 1807 Acked 2806 Write 5 Done Done New true
5729 1808 Acked 2807 Write 7 Done Done New true
5730 1809 Acked 2808 Read 9 Done Done New true
5731 1810 Acked 2809 Flush 0 Done Done New true
5732 STATES DS:0 DS:1 DS:2 TOTAL
5733 New 0 0 302 302
5734 Sent 0 0 0 0
5735 Done 302 302 0 604
5736 Skipped 0 0 0 0
5737 Error 0 0 0 0
5738 Last Flush: 2809 2809 2507
5739 Downstairs last five completed: 2507 2506 2505 2504 2503
5740 Upstairs last five completed: 1810 1809 1808 1807 1806
5741 CLIENT: Up:0 ds:302 act:2
57422024-02-12T01:35:36.544ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
57432024-02-12T01:35:36.544ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
57442024-02-12T01:35:36.544ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
57452024-02-12T01:35:36.544ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
57462024-02-12T01:35:36.544ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
57472024-02-12T01:35:36.544ZINFOcrucible: [2] client re-new 302 jobs since flush 2507 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
57482024-02-12T01:35:36.544ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
57492024-02-12T01:35:37.545ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
57502024-02-12T01:35:37.545ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
57512024-02-12T01:35:37.545ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
57522024-02-12T01:35:37.545ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
57532024-02-12T01:35:37.545ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
57542024-02-12T01:35:37.545ZINFOcrucible: [2] client re-new 302 jobs since flush 2507 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
57552024-02-12T01:35:37.545ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
57562024-02-12T01:35:38.546ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
57572024-02-12T01:35:38.546ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
57582024-02-12T01:35:38.546ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
57592024-02-12T01:35:38.546ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
57602024-02-12T01:35:38.546ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
57612024-02-12T01:35:38.546ZINFOcrucible: [2] client re-new 302 jobs since flush 2507 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
57622024-02-12T01:35:38.546ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
57632024-02-12T01:35:39.547ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
57642024-02-12T01:35:39.547ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
57652024-02-12T01:35:39.548ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
57662024-02-12T01:35:39.548ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
57672024-02-12T01:35:39.548ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
57682024-02-12T01:35:39.548ZINFOcrucible: [2] client re-new 302 jobs since flush 2507 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
57692024-02-12T01:35:39.548ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
5770 ----------------------------------------------------------------
5771 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
5772 States: Active Active Offline
5773 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
5774 1509 Acked 2508 Read 7 Done Done New true
5775 1510 Acked 2509 Write 7 Done Done New true
5776 1511 Acked 2510 Read 10 Done Done New true
5777 1512 Acked 2511 Write 3 Done Done New true
5778 1513 Acked 2512 Read 7 Done Done New true
5779 1514 Acked 2513 Read 9 Done Done New true
5780 1515 Acked 2514 Read 4 Done Done New true
5781 1516 Acked 2515 Read 3 Done Done New true
5782 1517 Acked 2516 Flush 0 Done Done New true
5783 1518 Acked 2517 Write 9 Done Done New true
5784 1519 Acked 2518 Read 4 Done Done New true
5785 1520 Acked 2519 Read 6 Done Done New true
5786 1521 Acked 2520 Read 10 Done Done New true
5787 1522 Acked 2521 Read 1 Done Done New true
5788 1523 Acked 2522 Write 1 Done Done New true
5789 1524 Acked 2523 Write 8 Done Done New true
5790 1525 Acked 2524 Write 1 Done Done New true
5791 1526 Acked 2525 Read 7 Done Done New true
5792 1527 Acked 2526 Read 4 Done Done New true
5793 1528 Acked 2527 Write 1 Done Done New true
5794 1529 Acked 2528 Write 9 Done Done New true
5795 1530 Acked 2529 Read 7 Done Done New true
5796 1531 Acked 2530 Read 4 Done Done New true
5797 1532 Acked 2531 Write 3 Done Done New true
5798 1533 Acked 2532 Read 2 Done Done New true
5799 1534 Acked 2533 Read 9 Done Done New true
5800 1535 Acked 2534 Read 9 Done Done New true
5801 1536 Acked 2535 Write 9 Done Done New true
5802 1537 Acked 2536 Write 10 Done Done New true
5803 1538 Acked 2537 Write 1 Done Done New true
5804 1539 Acked 2538 Read 6 Done Done New true
5805 1540 Acked 2539 Write 5 Done Done New true
5806 1541 Acked 2540 Write 4 Done Done New true
5807 1542 Acked 2541 Write 9 Done Done New true
5808 1543 Acked 2542 Write 7 Done Done New true
5809 1544 Acked 2543 Read 5 Done Done New true
5810 1545 Acked 2544 Write 2 Done Done New true
5811 1546 Acked 2545 Write 5 Done Done New true
5812 1547 Acked 2546 Flush 0 Done Done New true
5813 1548 Acked 2547 Write 9 Done Done New true
5814 1549 Acked 2548 Read 1 Done Done New true
5815 1550 Acked 2549 Write 4 Done Done New true
5816 1551 Acked 2550 Read 3 Done Done New true
5817 1552 Acked 2551 Write 6 Done Done New true
5818 1553 Acked 2552 Write 9 Done Done New true
5819 1554 Acked 2553 Write 9 Done Done New true
5820 1555 Acked 2554 Read 3 Done Done New true
5821 1556 Acked 2555 Write 3 Done Done New true
5822 1557 Acked 2556 Read 4 Done Done New true
5823 1558 Acked 2557 Read 8 Done Done New true
5824 1559 Acked 2558 Flush 0 Done Done New true
5825 1560 Acked 2559 Read 2 Done Done New true
5826 1561 Acked 2560 Write 8 Done Done New true
5827 1562 Acked 2561 Read 8 Done Done New true
5828 1563 Acked 2562 Flush 0 Done Done New true
5829 1564 Acked 2563 Read 1 Done Done New true
5830 1565 Acked 2564 Write 9 Done Done New true
5831 1566 Acked 2565 Read 2 Done Done New true
5832 1567 Acked 2566 Write 6 Done Done New true
5833 1568 Acked 2567 Flush 0 Done Done New true
5834 1569 Acked 2568 Write 10 Done Done New true
5835 1570 Acked 2569 Flush 0 Done Done New true
5836 1571 Acked 2570 Write 5 Done Done New true
5837 1572 Acked 2571 Read 3 Done Done New true
5838 1573 Acked 2572 Flush 0 Done Done New true
5839 1574 Acked 2573 Write 10 Done Done New true
5840 1575 Acked 2574 Read 4 Done Done New true
5841 1576 Acked 2575 Write 10 Done Done New true
5842 1577 Acked 2576 Flush 0 Done Done New true
5843 1578 Acked 2577 Read 7 Done Done New true
5844 1579 Acked 2578 Read 7 Done Done New true
5845 1580 Acked 2579 Read 10 Done Done New true
5846 1581 Acked 2580 Write 2 Done Done New true
5847 1582 Acked 2581 Flush 0 Done Done New true
5848 1583 Acked 2582 Write 6 Done Done New true
5849 1584 Acked 2583 Read 5 Done Done New true
5850 1585 Acked 2584 Write 10 Done Done New true
5851 1586 Acked 2585 Read 6 Done Done New true
5852 1587 Acked 2586 Write 6 Done Done New true
5853 1588 Acked 2587 Read 2 Done Done New true
5854 1589 Acked 2588 Read 8 Done Done New true
5855 1590 Acked 2589 Read 1 Done Done New true
5856 1591 Acked 2590 Write 7 Done Done New true
5857 1592 Acked 2591 Read 4 Done Done New true
5858 1593 Acked 2592 Read 6 Done Done New true
5859 1594 Acked 2593 Read 5 Done Done New true
5860 1595 Acked 2594 Write 1 Done Done New true
5861 1596 Acked 2595 Read 10 Done Done New true
5862 1597 Acked 2596 Flush 0 Done Done New true
5863 1598 Acked 2597 Read 10 Done Done New true
5864 1599 Acked 2598 Write 8 Done Done New true
5865 1600 Acked 2599 Read 8 Done Done New true
5866 1601 Acked 2600 Read 4 Done Done New true
5867 1602 Acked 2601 Write 8 Done Done New true
5868 1603 Acked 2602 Read 9 Done Done New true
5869 1604 Acked 2603 Flush 0 Done Done New true
5870 1605 Acked 2604 Write 10 Done Done New true
5871 1606 Acked 2605 Write 8 Done Done New true
5872 1607 Acked 2606 Write 7 Done Done New true
5873 1608 Acked 2607 Read 8 Done Done New true
5874 1609 Acked 2608 Read 2 Done Done New true
5875 1610 Acked 2609 Write 1 Done Done New true
5876 1611 Acked 2610 Write 5 Done Done New true
5877 1612 Acked 2611 Flush 0 Done Done New true
5878 1613 Acked 2612 Read 9 Done Done New true
5879 1614 Acked 2613 Read 3 Done Done New true
5880 1615 Acked 2614 Flush 0 Done Done New true
5881 1616 Acked 2615 Write 9 Done Done New true
5882 1617 Acked 2616 Write 6 Done Done New true
5883 1618 Acked 2617 Read 1 Done Done New true
5884 1619 Acked 2618 Read 8 Done Done New true
5885 1620 Acked 2619 Read 10 Done Done New true
5886 1621 Acked 2620 Write 3 Done Done New true
5887 1622 Acked 2621 Write 6 Done Done New true
5888 1623 Acked 2622 Read 2 Done Done New true
5889 1624 Acked 2623 Write 3 Done Done New true
5890 1625 Acked 2624 Write 7 Done Done New true
5891 1626 Acked 2625 Read 10 Done Done New true
5892 1627 Acked 2626 Flush 0 Done Done New true
5893 1628 Acked 2627 Read 2 Done Done New true
5894 1629 Acked 2628 Write 8 Done Done New true
5895 1630 Acked 2629 Flush 0 Done Done New true
5896 1631 Acked 2630 Read 6 Done Done New true
5897 1632 Acked 2631 Write 2 Done Done New true
5898 1633 Acked 2632 Read 2 Done Done New true
5899 1634 Acked 2633 Write 7 Done Done New true
5900 1635 Acked 2634 Write 1 Done Done New true
5901 1636 Acked 2635 Read 10 Done Done New true
5902 1637 Acked 2636 Read 6 Done Done New true
5903 1638 Acked 2637 Write 3 Done Done New true
5904 1639 Acked 2638 Read 8 Done Done New true
5905 1640 Acked 2639 Read 10 Done Done New true
5906 1641 Acked 2640 Read 10 Done Done New true
5907 1642 Acked 2641 Write 7 Done Done New true
5908 1643 Acked 2642 Write 9 Done Done New true
5909 1644 Acked 2643 Read 10 Done Done New true
5910 1645 Acked 2644 Write 8 Done Done New true
5911 1646 Acked 2645 Write 2 Done Done New true
5912 1647 Acked 2646 Write 4 Done Done New true
5913 1648 Acked 2647 Flush 0 Done Done New true
5914 1649 Acked 2648 Flush 0 Done Done New true
5915 1650 Acked 2649 Flush 0 Done Done New true
5916 1651 Acked 2650 Read 5 Done Done New true
5917 1652 Acked 2651 Read 7 Done Done New true
5918 1653 Acked 2652 Read 9 Done Done New true
5919 1654 Acked 2653 Flush 0 Done Done New true
5920 1655 Acked 2654 Read 1 Done Done New true
5921 1656 Acked 2655 Read 6 Done Done New true
5922 1657 Acked 2656 Write 1 Done Done New true
5923 1658 Acked 2657 Read 3 Done Done New true
5924 1659 Acked 2658 Read 5 Done Done New true
5925 1660 Acked 2659 Write 5 Done Done New true
5926 1661 Acked 2660 Read 9 Done Done New true
5927 1662 Acked 2661 Write 8 Done Done New true
5928 1663 Acked 2662 Read 10 Done Done New true
5929 1664 Acked 2663 Read 3 Done Done New true
5930 1665 Acked 2664 Write 7 Done Done New true
5931 1666 Acked 2665 Read 9 Done Done New true
5932 1667 Acked 2666 Write 6 Done Done New true
5933 1668 Acked 2667 Write 7 Done Done New true
5934 1669 Acked 2668 Write 2 Done Done New true
5935 1670 Acked 2669 Write 1 Done Done New true
5936 1671 Acked 2670 Read 2 Done Done New true
5937 1672 Acked 2671 Read 10 Done Done New true
5938 1673 Acked 2672 Write 8 Done Done New true
5939 1674 Acked 2673 Read 10 Done Done New true
5940 1675 Acked 2674 Read 5 Done Done New true
5941 1676 Acked 2675 Write 10 Done Done New true
5942 1677 Acked 2676 Flush 0 Done Done New true
5943 1678 Acked 2677 Write 6 Done Done New true
5944 1679 Acked 2678 Write 8 Done Done New true
5945 1680 Acked 2679 Flush 0 Done Done New true
5946 1681 Acked 2680 Write 6 Done Done New true
5947 1682 Acked 2681 Write 9 Done Done New true
5948 1683 Acked 2682 Read 2 Done Done New true
5949 1684 Acked 2683 Write 4 Done Done New true
5950 1685 Acked 2684 Write 6 Done Done New true
5951 1686 Acked 2685 Read 9 Done Done New true
5952 1687 Acked 2686 Read 1 Done Done New true
5953 1688 Acked 2687 Read 7 Done Done New true
5954 1689 Acked 2688 Flush 0 Done Done New true
5955 1690 Acked 2689 Read 9 Done Done New true
5956 1691 Acked 2690 Flush 0 Done Done New true
5957 1692 Acked 2691 Write 3 Done Done New true
5958 1693 Acked 2692 Flush 0 Done Done New true
5959 1694 Acked 2693 Read 7 Done Done New true
5960 1695 Acked 2694 Write 8 Done Done New true
5961 1696 Acked 2695 Read 8 Done Done New true
5962 1697 Acked 2696 Read 4 Done Done New true
5963 1698 Acked 2697 Write 2 Done Done New true
5964 1699 Acked 2698 Read 4 Done Done New true
5965 1700 Acked 2699 Write 6 Done Done New true
5966 1701 Acked 2700 Write 6 Done Done New true
5967 1702 Acked 2701 Read 5 Done Done New true
5968 1703 Acked 2702 Read 2 Done Done New true
5969 1704 Acked 2703 Write 7 Done Done New true
5970 1705 Acked 2704 Read 1 Done Done New true
5971 1706 Acked 2705 Read 5 Done Done New true
5972 1707 Acked 2706 Read 8 Done Done New true
5973 1708 Acked 2707 Read 8 Done Done New true
5974 1709 Acked 2708 Flush 0 Done Done New true
5975 1710 Acked 2709 Read 4 Done Done New true
5976 1711 Acked 2710 Read 9 Done Done New true
5977 1712 Acked 2711 Write 5 Done Done New true
5978 1713 Acked 2712 Read 3 Done Done New true
5979 1714 Acked 2713 Write 10 Done Done New true
5980 1715 Acked 2714 Write 3 Done Done New true
5981 1716 Acked 2715 Write 10 Done Done New true
5982 1717 Acked 2716 Write 7 Done Done New true
5983 1718 Acked 2717 Flush 0 Done Done New true
5984 1719 Acked 2718 Write 4 Done Done New true
5985 1720 Acked 2719 Write 8 Done Done New true
5986 1721 Acked 2720 Read 6 Done Done New true
5987 1722 Acked 2721 Write 7 Done Done New true
5988 1723 Acked 2722 Write 3 Done Done New true
5989 1724 Acked 2723 Flush 0 Done Done New true
5990 1725 Acked 2724 Read 1 Done Done New true
5991 1726 Acked 2725 Read 10 Done Done New true
5992 1727 Acked 2726 Read 4 Done Done New true
5993 1728 Acked 2727 Read 3 Done Done New true
5994 1729 Acked 2728 Read 2 Done Done New true
5995 1730 Acked 2729 Flush 0 Done Done New true
5996 1731 Acked 2730 Read 1 Done Done New true
5997 1732 Acked 2731 Flush 0 Done Done New true
5998 1733 Acked 2732 Read 5 Done Done New true
5999 1734 Acked 2733 Flush 0 Done Done New true
6000 1735 Acked 2734 Read 1 Done Done New true
6001 1736 Acked 2735 Write 2 Done Done New true
6002 1737 Acked 2736 Read 8 Done Done New true
6003 1738 Acked 2737 Read 4 Done Done New true
6004 1739 Acked 2738 Read 1 Done Done New true
6005 1740 Acked 2739 Write 9 Done Done New true
6006 1741 Acked 2740 Write 10 Done Done New true
6007 1742 Acked 2741 Read 4 Done Done New true
6008 1743 Acked 2742 Write 9 Done Done New true
6009 1744 Acked 2743 Write 7 Done Done New true
6010 1745 Acked 2744 Read 6 Done Done New true
6011 1746 Acked 2745 Write 1 Done Done New true
6012 1747 Acked 2746 Flush 0 Done Done New true
6013 1748 Acked 2747 Read 8 Done Done New true
6014 1749 Acked 2748 Read 4 Done Done New true
6015 1750 Acked 2749 Write 8 Done Done New true
6016 1751 Acked 2750 Read 5 Done Done New true
6017 1752 Acked 2751 Read 10 Done Done New true
6018 1753 Acked 2752 Read 4 Done Done New true
6019 1754 Acked 2753 Read 9 Done Done New true
6020 1755 Acked 2754 Write 9 Done Done New true
6021 1756 Acked 2755 Read 10 Done Done New true
6022 1757 Acked 2756 Read 5 Done Done New true
6023 1758 Acked 2757 Write 8 Done Done New true
6024 1759 Acked 2758 Write 8 Done Done New true
6025 1760 Acked 2759 Read 2 Done Done New true
6026 1761 Acked 2760 Write 3 Done Done New true
6027 1762 Acked 2761 Read 10 Done Done New true
6028 1763 Acked 2762 Write 10 Done Done New true
6029 1764 Acked 2763 Read 7 Done Done New true
6030 1765 Acked 2764 Read 8 Done Done New true
6031 1766 Acked 2765 Flush 0 Done Done New true
6032 1767 Acked 2766 Write 1 Done Done New true
6033 1768 Acked 2767 Read 3 Done Done New true
6034 1769 Acked 2768 Flush 0 Done Done New true
6035 1770 Acked 2769 Read 10 Done Done New true
6036 1771 Acked 2770 Read 7 Done Done New true
6037 1772 Acked 2771 Read 4 Done Done New true
6038 1773 Acked 2772 Write 5 Done Done New true
6039 1774 Acked 2773 Read 5 Done Done New true
6040 1775 Acked 2774 Read 4 Done Done New true
6041 1776 Acked 2775 Read 6 Done Done New true
6042 1777 Acked 2776 Flush 0 Done Done New true
6043 1778 Acked 2777 Read 5 Done Done New true
6044 1779 Acked 2778 Write 2 Done Done New true
6045 1780 Acked 2779 Read 9 Done Done New true
6046 1781 Acked 2780 Read 1 Done Done New true
6047 1782 Acked 2781 Write 1 Done Done New true
6048 1783 Acked 2782 Write 4 Done Done New true
6049 1784 Acked 2783 Flush 0 Done Done New true
6050 1785 Acked 2784 Write 4 Done Done New true
6051 1786 Acked 2785 Read 2 Done Done New true
6052 1787 Acked 2786 Write 2 Done Done New true
6053 1788 Acked 2787 Read 1 Done Done New true
6054 1789 Acked 2788 Read 4 Done Done New true
6055 1790 Acked 2789 Write 8 Done Done New true
6056 1791 Acked 2790 Read 2 Done Done New true
6057 1792 Acked 2791 Flush 0 Done Done New true
6058 1793 Acked 2792 Read 4 Done Done New true
6059 1794 Acked 2793 Read 3 Done Done New true
6060 1795 Acked 2794 Read 2 Done Done New true
6061 1796 Acked 2795 Read 3 Done Done New true
6062 1797 Acked 2796 Write 10 Done Done New true
6063 1798 Acked 2797 Write 5 Done Done New true
6064 1799 Acked 2798 Flush 0 Done Done New true
6065 1800 Acked 2799 Read 9 Done Done New true
6066 1801 Acked 2800 Write 9 Done Done New true
6067 1802 Acked 2801 Write 8 Done Done New true
6068 1803 Acked 2802 Read 4 Done Done New true
6069 1804 Acked 2803 Read 9 Done Done New true
6070 1805 Acked 2804 Read 10 Done Done New true
6071 1806 Acked 2805 Write 8 Done Done New true
6072 1807 Acked 2806 Write 5 Done Done New true
6073 1808 Acked 2807 Write 7 Done Done New true
6074 1809 Acked 2808 Read 9 Done Done New true
6075 1810 Acked 2809 Flush 0 Done Done New true
6076 STATES DS:0 DS:1 DS:2 TOTAL
6077 New 0 0 302 302
6078 Sent 0 0 0 0
6079 Done 302 302 0 604
6080 Skipped 0 0 0 0
6081 Error 0 0 0 0
6082 Last Flush: 2809 2809 2507
6083 Downstairs last five completed: 2507 2506 2505 2504 2503
6084 Upstairs last five completed: 1810 1809 1808 1807 1806
6085 CLIENT: Up:0 ds:302 act:2
60862024-02-12T01:35:40.549ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
60872024-02-12T01:35:40.549ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
60882024-02-12T01:35:40.549ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
60892024-02-12T01:35:40.549ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
60902024-02-12T01:35:40.549ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
60912024-02-12T01:35:40.549ZINFOcrucible: [2] client re-new 302 jobs since flush 2507 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
60922024-02-12T01:35:40.549ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
6093 [8830] got start action
6094 [8830] I am going to restart
6095 Starting downstairs at port 8830
6096 Make output file at "/tmp/dsc/downstairs-8830.txt"
6097 [8830][2] reports Starting
6098 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1170)
6099 [8830][2] reports Running
61002024-02-12T01:35:41.550ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
61012024-02-12T01:35:41.550ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
61022024-02-12T01:35:41.550ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
61032024-02-12T01:35:41.550ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
61042024-02-12T01:35:41.550ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
61052024-02-12T01:35:41.551ZINFOcrucible: [2] client re-new 302 jobs since flush 2507 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
61062024-02-12T01:35:41.551ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
61072024-02-12T01:35:42.552ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
61082024-02-12T01:35:42.552ZINFOcrucible: ds_connection connected = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
61092024-02-12T01:35:42.553ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
61102024-02-12T01:35:42.553ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
61112024-02-12T01:35:42.553ZINFOcrucible: send last flush ID to this DS: 2507 = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
61122024-02-12T01:35:42.554ZINFOcrucible: Replied this last flush ID: 2507 = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
61132024-02-12T01:35:42.554ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
61142024-02-12T01:35:42.554ZINFOcrucible: [2] Transition from Offline to Replay = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
61152024-02-12T01:35:42.554ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
61162024-02-12T01:35:42.554ZINFOcrucible: [2] Transition from Replay to Active = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
6117 ----------------------------------------------------------------
6118 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
6119 Downstairs last five completed: 2809 2808 2807 2806 2805
6120 Upstairs last five completed: 1810 1809 1808 1807 1806
6121 CLIENT: Up:0 ds:0 act:3
6122 Replay: All jobs finished, all DS active.
6123 Main task has work to do, go find it
6124 got dsc Stop(2)
6125 stop 2
6126 Feb 12 01:35:43.902 INFO request completed, latency_us: 370, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 34646473-14f6-4a44-9949-d1fd5428672b, remote_addr: [8830] Got stop action so:false kr:false
6127 127.0.0.1:60456, local_addr: 127.0.0.1:9998
61282024-02-12T01:35:43.903ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
61292024-02-12T01:35:43.903ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
61302024-02-12T01:35:43.903ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
61312024-02-12T01:35:43.903ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
61322024-02-12T01:35:43.903ZINFOcrucible: [2] client re-new 0 jobs since flush 2809 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
61332024-02-12T01:35:43.903ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
6134 Feb 12 01:35:43.903 INFO request completed, latency_us: 149, response_code: 200, uri: /state/cid/2, method: GET, req_id: 954afc73-ce22-48dc-8f86-a295049f99ff, remote_addr: 127.0.0.1:60456, local_addr: 127.0.0.1:9998
6135 [8830] Exited with: ExitStatus(unix_wait_status(9))
6136 [8830][2] reports Exit
61372024-02-12T01:35:44.904ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
61382024-02-12T01:35:44.904ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
61392024-02-12T01:35:44.904ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
61402024-02-12T01:35:44.904ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
61412024-02-12T01:35:44.904ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
61422024-02-12T01:35:44.904ZINFOcrucible: [2] client re-new 0 jobs since flush 2809 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
61432024-02-12T01:35:44.904ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
61442024-02-12T01:35:45.905ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
61452024-02-12T01:35:45.905ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
61462024-02-12T01:35:45.905ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
61472024-02-12T01:35:45.905ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
61482024-02-12T01:35:45.905ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
61492024-02-12T01:35:45.905ZINFOcrucible: [2] client re-new 0 jobs since flush 2809 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
61502024-02-12T01:35:45.905ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
61512024-02-12T01:35:46.906ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
61522024-02-12T01:35:46.906ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
61532024-02-12T01:35:46.906ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
61542024-02-12T01:35:46.906ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
61552024-02-12T01:35:46.906ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
61562024-02-12T01:35:46.906ZINFOcrucible: [2] client re-new 0 jobs since flush 2809 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
61572024-02-12T01:35:46.907ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
6158 Feb 12 01:35:47.905 INFO request completed, latency_us: 216, response_code: 200, uri: /state/cid/2, method: GET, req_id: 7d5545cd-4c2c-4e01-a991-48caaa86423c, remote_addr: 127.0.0.1:60456, local_addr: 127.0.0.1:9998
6159 001/300 Write block 1839 len 40960 data: 3 3 2 2 2 2 2 2 3 3
6160 002/300 Read block 273 len 28672
61612024-02-12T01:35:47.907ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
61622024-02-12T01:35:47.907ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
61632024-02-12T01:35:47.907ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
61642024-02-12T01:35:47.907ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
61652024-02-12T01:35:47.907ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
61662024-02-12T01:35:47.907ZINFOcrucible: [2] client re-new 2 jobs since flush 2809 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
61672024-02-12T01:35:47.907ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
6168 003/300 Flush
6169 004/300 Read block 1568 len 32768
6170 005/300 Read block 4097 len 28672
6171 006/300 Write block 604 len 20480 data: 3 3 3 3 3
6172 007/300 Write block 1325 len 8192 data: 2 2
6173 008/300 Write block 678 len 36864 data: 3 2 2 2 2 2 2 2 2
6174 009/300 Write block 764 len 8192 data: 2 2
6175 010/300 Write block 935 len 36864 data: 2 2 3 3 3 3 3 3 3
6176 011/300 Read block 608 len 36864
6177 012/300 Read block 671 len 8192
6178 013/300 Read block 42 len 4096
6179 014/300 Write block 3314 len 8192 data: 2 2
6180 015/300 Read block 1108 len 24576
6181 016/300 Write block 1682 len 12288 data: 4 2 2
6182 017/300 Read block 4882 len 8192
6183 018/300 Write block 3399 len 12288 data: 4 5 5
6184 019/300 Write block 2806 len 8192 data: 4 4
6185 020/300 Read block 4483 len 24576
6186 021/300 Read block 1600 len 8192
6187 022/300 Write block 2974 len 4096 data: 2
6188 023/300 Write block 2382 len 20480 data: 2 3 3 3 2
6189 024/300 Read block 2684 len 28672
6190 025/300 Read block 2539 len 40960
6191 026/300 Read block 772 len 16384
6192 027/300 Write block 3878 len 12288 data: 6 5 4
6193 028/300 Read block 3827 len 8192
6194 029/300 Read block 1496 len 16384
6195 030/300 Write block 574 len 12288 data: 2 2 3
6196 031/300 Write block 1817 len 4096 data: 3
6197 032/300 Read block 3354 len 8192
6198 033/300 Read block 3994 len 32768
6199 034/300 Read block 2193 len 32768
6200 035/300 Read block 2700 len 24576
6201 036/300 Flush
6202 037/300 Read block 2373 len 28672
6203 038/300 Read block 832 len 20480
6204 039/300 Read block 3759 len 20480
6205 040/300 Read block 448 len 40960
6206 041/300 Read block 4309 len 40960
6207 042/300 Read block 4239 len 20480
6208 043/300 Read block 1477 len 20480
6209 044/300 Read block 3353 len 16384
6210 045/300 Write block 4137 len 20480 data: 2 2 2 2 2
6211 046/300 Read block 3034 len 40960
6212 047/300 Read block 4253 len 8192
6213 048/300 Read block 1113 len 28672
6214 049/300 Write block 1269 len 4096 data: 3
6215 050/300 Read block 1548 len 28672
6216 051/300 Write block 2923 len 36864 data: 2 3 3 3 3 3 3 4 3
6217 052/300 Read block 278 len 4096
6218 053/300 Write block 1167 len 8192 data: 3 3
6219 054/300 Write block 1162 len 12288 data: 2 2 3
6220 055/300 Flush
6221 056/300 Write block 2788 len 40960 data: 5 3 3 3 3 2 2 2 2 2
6222 057/300 Read block 3041 len 16384
6223 058/300 Write block 4272 len 24576 data: 4 4 3 3 3 2
6224 059/300 Write block 3489 len 4096 data: 2
6225 060/300 Read block 4486 len 16384
6226 061/300 Write block 4077 len 24576 data: 2 2 2 2 2 2
6227 062/300 Write block 349 len 8192 data: 2 2
6228 063/300 Write block 3294 len 28672 data: 2 2 2 2 2 2 2
6229 064/300 Write block 4068 len 40960 data: 3 3 3 3 3 3 2 2 2 3
6230 065/300 Write block 2044 len 8192 data: 3 4
6231 066/300 Write block 1933 len 20480 data: 2 2 2 2 2
6232 067/300 Flush
6233 068/300 Write block 1714 len 40960 data: 2 2 3 3 3 3 3 2 2 2
6234 069/300 Read block 1103 len 8192
6235 070/300 Flush
6236 071/300 Read block 25 len 4096
6237 072/300 Write block 1483 len 4096 data: 3
6238 073/300 Write block 2799 len 36864 data: 2 2 2 2 3 4 4 5 5
6239 074/300 Read block 3801 len 24576
6240 075/300 Write block 1617 len 32768 data: 3 2 3 3 3 3 3 2
6241 076/300 Read block 1680 len 16384
6242 077/300 Write block 633 len 12288 data: 2 2 2
6243 078/300 Read block 3975 len 36864
6244 079/300 Write block 3248 len 12288 data: 4 4 3
6245 080/300 Read block 4245 len 8192
6246 081/300 Write block 2290 len 40960 data: 3 3 3 3 3 3 2 3 3 2
6247 082/300 Write block 1043 len 40960 data: 2 3 3 3 3 3 3 3 3 3
6248 083/300 Write block 4325 len 16384 data: 2 2 2 2
6249 084/300 Write block 880 len 20480 data: 2 3 3 3 3
6250 085/300 Write block 3468 len 24576 data: 2 3 5 5 5 4
6251 086/300 Read block 1912 len 24576
6252 087/300 Read block 778 len 20480
6253 088/300 Read block 4227 len 24576
6254 089/300 Write block 2305 len 36864 data: 3 2 2 2 3 3 4 4 4
6255 090/300 Read block 3713 len 32768
6256 091/300 Write block 2545 len 32768 data: 3 2 2 2 2 2 3 3
6257 092/300 Read block 177 len 40960
6258 093/300 Write block 218 len 36864 data: 2 2 2 2 2 2 2 2 2
6259 094/300 Write block 671 len 40960 data: 3 3 4 4 3 3 3 4 3 3
6260 095/300 Flush
6261 096/300 Write block 1174 len 8192 data: 4 3
6262 097/300 Write block 2757 len 20480 data: 7 6 5 5 5
6263 098/300 Read block 3256 len 28672
6264 099/300 Write block 3149 len 20480 data: 2 2 2 2 3
6265 100/300 Write block 2897 len 24576 data: 3 2 2 2 3 3
6266 101/300 Read block 1233 len 8192
6267 102/300 Read block 3964 len 28672
6268 103/300 Write block 763 len 28672 data: 2 3 3 2 2 2 2
6269 104/300 Read block 2975 len 28672
6270 105/300 Write block 405 len 8192 data: 2 3
6271 106/300 Write block 1870 len 20480 data: 2 2 2 2 2
6272 107/300 Read block 4179 len 16384
6273 108/300 Write block 4448 len 40960 data: 2 2 3 3 3 3 5 4 3 3
6274 109/300 Read block 1381 len 16384
6275 110/300 Write block 1148 len 32768 data: 3 2 2 2 2 2 2 2
6276 111/300 Read block 2658 len 4096
6277 112/300 Read block 1676 len 8192
6278 113/300 Read block 4179 len 24576
6279 114/300 Read block 3087 len 32768
6280 115/300 Flush
6281 116/300 Read block 4522 len 12288
6282 117/300 Read block 4983 len 24576
6283 118/300 Write block 4671 len 24576 data: 4 4 4 4 4 5
6284 119/300 Read block 1972 len 4096
6285 120/300 Read block 1422 len 36864
6286 121/300 Write block 1676 len 36864 data: 4 5 6 5 5 4 5 3 3
6287 122/300 Read block 3576 len 24576
6288 123/300 Write block 2710 len 28672 data: 3 3 3 3 2 2 2
6289 124/300 Write block 3376 len 8192 data: 4 4
6290 125/300 Write block 2806 len 20480 data: 6 6 4 5 4
6291 126/300 Read block 3057 len 32768
6292 127/300 Read block 2809 len 20480
6293 128/300 Read block 1829 len 12288
6294 129/300 Read block 279 len 36864
6295 130/300 Write block 73 len 16384 data: 3 3 3 3
6296 131/300 Write block 1315 len 40960 data: 2 2 2 2 2 2 2 2 2 2
6297 132/300 Read block 4107 len 24576
6298 133/300 Flush
6299 134/300 Read block 4906 len 28672
6300 135/300 Write block 1772 len 4096 data: 3
6301 136/300 Flush
6302 137/300 Write block 878 len 28672 data: 2 2 3 4 4 4 4
6303 138/300 Write block 2399 len 40960 data: 2 2 2 2 2 2 3 3 3 3
6304 139/300 Write block 1966 len 8192 data: 3 4
6305 140/300 Write block 646 len 8192 data: 2 2
6306 141/300 Read block 2523 len 28672
6307 142/300 Write block 89 len 32768 data: 3 3 2 2 2 2 2 2
6308 143/300 Write block 3490 len 24576 data: 2 2 2 2 2 2
6309 144/300 Flush
6310 145/300 Write block 2022 len 28672 data: 3 3 3 3 3 3 3
6311 146/300 Write block 3922 len 28672 data: 2 2 2 2 2 2 2
6312 147/300 Read block 2492 len 8192
6313 148/300 Read block 4995 len 16384
6314 149/300 Read block 4385 len 40960
6315 150/300 Write block 841 len 28672 data: 2 2 2 2 2 2 2
6316 151/300 Write block 1181 len 4096 data: 4
6317 152/300 Write block 1768 len 24576 data: 2 2 3 3 4 3
6318 153/300 Read block 3971 len 20480
6319 154/300 Read block 1792 len 24576
6320 155/300 Read block 943 len 28672
6321 156/300 Read block 4838 len 4096
6322 157/300 Write block 3641 len 32768 data: 3 3 3 3 3 3 2 2
6323 158/300 Read block 2353 len 28672
6324 159/300 Write block 1162 len 8192 data: 3 3
6325 160/300 Write block 4112 len 32768 data: 2 2 2 2 2 2 2 2
6326 161/300 Read block 2939 len 40960
6327 162/300 Read block 3458 len 36864
6328 163/300 Read block 2844 len 12288
6329 164/300 Write block 289 len 4096 data: 3
6330 165/300 Flush
6331 166/300 Flush
6332 167/300 Read block 310 len 40960
6333 168/300 Read block 3644 len 4096
6334 169/300 Read block 1713 len 24576
6335 170/300 Read block 1838 len 32768
6336 171/300 Read block 4922 len 20480
6337 172/300 Read block 2713 len 40960
6338 173/300 Write block 17 len 16384 data: 3 4 4 4
6339 174/300 Read block 3311 len 16384
6340 175/300 Write block 3637 len 16384 data: 3 3 3 4
6341 176/300 Read block 66 len 12288
6342 177/300 Read block 4365 len 4096
6343 178/300 Read block 4329 len 24576
6344 179/300 Write block 2993 len 20480 data: 2 2 2 3 3
6345 180/300 Read block 2328 len 32768
6346 181/300 Read block 4968 len 40960
6347 182/300 Write block 1098 len 8192 data: 2 2
6348 183/300 Write block 3811 len 24576 data: 4 5 4 4 4 3
6349 184/300 Read block 4523 len 32768
6350 185/300 Read block 4275 len 24576
6351 186/300 Read block 3728 len 4096
6352 187/300 Read block 544 len 4096
6353 188/300 Write block 3429 len 32768 data: 5 5 4 3 4 4 3 3
6354 189/300 Read block 1374 len 16384
6355 190/300 Read block 3994 len 40960
6356 191/300 Write block 4644 len 12288 data: 5 5 4
6357 192/300 Write block 2901 len 4096 data: 4
6358 193/300 Flush
6359 194/300 Read block 2960 len 40960
6360 195/300 Read block 3418 len 36864
6361 196/300 Read block 3210 len 40960
6362 197/300 Read block 1007 len 8192
6363 198/300 Read block 1078 len 40960
6364 199/300 Read block 4878 len 8192
6365 200/300 Read block 4453 len 40960
6366 201/300 Read block 1467 len 32768
6367 202/300 Write block 1591 len 28672 data: 2 2 2 2 2 2 2
6368 203/300 Flush
6369 204/300 Read block 1791 len 20480
6370 205/300 Write block 3315 len 28672 data: 3 2 2 2 2 2 2
6371 206/300 Write block 4884 len 28672 data: 2 2 2 2 2 2 2
6372 207/300 Read block 755 len 20480
6373 208/300 Write block 1564 len 20480 data: 5 5 6 5 5
6374 209/300 Write block 3030 len 8192 data: 2 2
6375 210/300 Read block 3492 len 36864
6376 211/300 Read block 4360 len 32768
6377 212/300 Read block 2871 len 32768
6378 213/300 Read block 823 len 8192
6379 214/300 Write block 1819 len 24576 data: 4 3 3 4 4 3
6380 215/300 Write block 4590 len 12288 data: 2 2 2
6381 216/300 Read block 4663 len 28672
6382 217/300 Flush
6383 218/300 Write block 2531 len 8192 data: 3 2
6384 219/300 Write block 4558 len 12288 data: 2 2 2
6385 220/300 Read block 4385 len 20480
6386 221/300 Read block 4358 len 36864
6387 222/300 Read block 4784 len 4096
6388 223/300 Write block 2192 len 8192 data: 2 2
6389 224/300 Read block 2658 len 4096
6390 225/300 Flush
6391 226/300 Read block 1920 len 8192
6392 227/300 Write block 3496 len 4096 data: 2
6393 228/300 Read block 4254 len 36864
6394 229/300 Write block 4496 len 40960 data: 2 2 2 2 2 2 2 3 3 4
6395 230/300 Read block 1143 len 36864
6396 231/300 Write block 558 len 4096 data: 2
6397 232/300 Write block 3790 len 32768 data: 3 4 4 4 3 3 3 3
6398 233/300 Read block 3447 len 20480
6399 234/300 Read block 1809 len 20480
6400 235/300 Write block 1727 len 20480 data: 4 4 4 3 3
6401 236/300 Write block 672 len 8192 data: 4 5
6402 237/300 Read block 2713 len 32768
6403 238/300 Flush
6404 239/300 Read block 644 len 20480
6405 240/300 Read block 2022 len 20480
6406 241/300 Write block 1551 len 36864 data: 3 3 3 4 3 3 3 4 4
6407 242/300 Read block 1669 len 4096
6408 243/300 Write block 1599 len 40960 data: 2 2 2 2 2 2 3 3 4 4
6409 244/300 Read block 4060 len 16384
6410 245/300 Write block 4276 len 40960 data: 4 3 2 2 3 3 4 4 4 4
6411 246/300 Read block 4547 len 40960
6412 247/300 Write block 2816 len 40960 data: 4 4 3 3 2 3 3 4 5 4
6413 248/300 Read block 656 len 32768
6414 249/300 Read block 374 len 32768
6415 250/300 Read block 284 len 12288
6416 251/300 Write block 1842 len 40960 data: 3 3 3 3 3 4 4 3 4 4
6417 252/300 Read block 3881 len 40960
6418 253/300 Read block 1965 len 28672
6419 254/300 Read block 3903 len 8192
6420 255/300 Write block 411 len 8192 data: 3 2
6421 256/300 Write block 2446 len 24576 data: 3 2 3 3 2 2
6422 257/300 Write block 2054 len 12288 data: 2 2 2
6423 258/300 Write block 4070 len 4096 data: 4
6424 259/300 Write block 620 len 16384 data: 2 2 3 3
6425 260/300 Read block 1189 len 20480
6426 261/300 Write block 3269 len 28672 data: 3 3 3 3 2 2 2
6427 262/300 Read block 4575 len 36864
6428 263/300 Read block 3627 len 8192
6429 264/300 Write block 1325 len 24576 data: 3 3 4 2 3 3
6430 265/300 Read block 2444 len 28672
6431 266/300 Write block 240 len 32768 data: 2 3 3 3 3 3 3 3
6432 267/300 Read block 3475 len 24576
6433 268/300 Read block 1731 len 32768
6434 269/300 Read block 3785 len 24576
6435 270/300 Flush
6436 271/300 Read block 2970 len 4096
6437 272/300 Read block 2624 len 20480
6438 273/300 Write block 3042 len 4096 data: 3
6439 274/300 Read block 3758 len 4096
6440 275/300 Write block 444 len 36864 data: 3 3 3 2 2 2 2 2 2
6441 276/300 Write block 4497 len 20480 data: 3 3 3 3 3
6442 277/300 Read block 3481 len 4096
6443 278/300 Write block 2151 len 36864 data: 3 3 3 3 3 3 2 3 3
6444 279/300 Read block 2296 len 20480
6445 280/300 Write block 4134 len 32768 data: 2 3 3 3 3 3 3 3
6446 281/300 Read block 724 len 8192
6447 282/300 Write block 2072 len 36864 data: 3 3 3 3 2 3 3 3 2
6448 283/300 Read block 1554 len 28672
6449 284/300 Flush
6450 285/300 Flush
6451 286/300 Read block 804 len 36864
6452 287/300 Write block 259 len 20480 data: 4 4 3 3 4
6453 288/300 Write block 3941 len 36864 data: 2 2 2 2 3 3 3 2 2
6454 289/300 Write block 2839 len 4096 data: 6
6455 290/300 Write block 3546 len 16384 data: 2 2 2 2
6456 291/300 Read block 4494 len 20480
6457 292/300 Write block 4152 len 24576 data: 2 2 2 2 2 4
6458 293/300 Read block 2926 len 36864
6459 294/300 Read block 3059 len 8192
6460 295/300 Write block 979 len 32768 data: 2 2 2 2 2 2 2 2
6461 296/300 Write block 1044 len 32768 data: 4 4 4 4 4 4 4 4
6462 297/300 Write block 4235 len 20480 data: 3 3 3 3 3
6463 298/300 Read block 596 len 12288
6464 299/300 Read block 4200 len 40960
6465 300/300 Read block 4817 len 8192
6466 Main task has work to do, go find it
6467 got dsc Start(2)
6468 start 2
6469 Feb 12 01:35:48.251 INFO request completed, latency_us: 209, response_code: 204, uri: /start/cid/2, method: POST, req_id: e3c7cd94-a630-4deb-a149-ab628811453d, remote_addr: 127.0.0.1:60456, local_addr: 127.0.0.1:9998
6470 Replay: started 2, returned:Ok(())
6471 ----------------------------------------------------------------
6472 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:300
6473 States: Active Active Offline
6474 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
6475 1811 Acked 2810 Write 10 Done Done New true
6476 1812 Acked 2811 Read 7 Done Done New true
6477 1813 Acked 2812 Flush 0 Done Done New false
6478 1814 Acked 2813 Read 8 Done Done New false
6479 1815 Acked 2814 Read 7 Done Done New false
6480 1816 Acked 2815 Write 5 Done Done New false
6481 1817 Acked 2816 Write 2 Done Done New false
6482 1818 Acked 2817 Write 9 Done Done New false
6483 1819 Acked 2818 Write 2 Done Done New false
6484 1820 Acked 2819 Write 9 Done Done New false
6485 1821 Acked 2820 Read 9 Done Done New false
6486 1822 Acked 2821 Read 2 Done Done New false
6487 1823 Acked 2822 Read 1 Done Done New false
6488 1824 Acked 2823 Write 2 Done Done New false
6489 1825 Acked 2824 Read 6 Done Done New false
6490 1826 Acked 2825 Write 3 Done Done New false
6491 1827 Acked 2826 Read 2 Done Done New false
6492 1828 Acked 2827 Write 3 Done Done New false
6493 1829 Acked 2828 Write 2 Done Done New false
6494 1830 Acked 2829 Read 6 Done Done New false
6495 1831 Acked 2830 Read 2 Done Done New false
6496 1832 Acked 2831 Write 1 Done Done New false
6497 1833 Acked 2832 Write 5 Done Done New false
6498 1834 Acked 2833 Read 7 Done Done New false
6499 1835 Acked 2834 Read 10 Done Done New false
6500 1836 Acked 2835 Read 4 Done Done New false
6501 1837 Acked 2836 Write 3 Done Done New false
6502 1838 Acked 2837 Read 2 Done Done New false
6503 1839 Acked 2838 Read 4 Done Done New false
6504 1840 Acked 2839 Write 3 Done Done New false
6505 1841 Acked 2840 Write 1 Done Done New false
6506 1842 Acked 2841 Read 2 Done Done New false
6507 1843 Acked 2842 Read 8 Done Done New false
6508 1844 Acked 2843 Read 8 Done Done New false
6509 1845 Acked 2844 Read 6 Done Done New false
6510 1846 Acked 2845 Flush 0 Done Done New false
6511 1847 Acked 2846 Read 7 Done Done New false
6512 1848 Acked 2847 Read 5 Done Done New false
6513 1849 Acked 2848 Read 5 Done Done New false
6514 1850 Acked 2849 Read 10 Done Done New false
6515 1851 Acked 2850 Read 10 Done Done New false
6516 1852 Acked 2851 Read 5 Done Done New false
6517 1853 Acked 2852 Read 5 Done Done New false
6518 1854 Acked 2853 Read 4 Done Done New false
6519 1855 Acked 2854 Write 5 Done Done New false
6520 1856 Acked 2855 Read 10 Done Done New false
6521 1857 Acked 2856 Read 2 Done Done New false
6522 1858 Acked 2857 Read 7 Done Done New false
6523 1859 Acked 2858 Write 1 Done Done New false
6524 1860 Acked 2859 Read 7 Done Done New false
6525 1861 Acked 2860 Write 9 Done Done New false
6526 1862 Acked 2861 Read 1 Done Done New false
6527 1863 Acked 2862 Write 2 Done Done New false
6528 1864 Acked 2863 Write 3 Done Done New false
6529 1865 Acked 2864 Flush 0 Done Done New false
6530 1866 Acked 2865 Write 10 Done Done New false
6531 1867 Acked 2866 Read 4 Done Done New false
6532 1868 Acked 2867 Write 6 Done Done New false
6533 1869 Acked 2868 Write 1 Done Done New false
6534 1870 Acked 2869 Read 4 Done Done New false
6535 1871 Acked 2870 Write 6 Done Done New false
6536 1872 Acked 2871 Write 2 Done Done New false
6537 1873 Acked 2872 Write 7 Done Done New false
6538 1874 Acked 2873 Write 10 Done Done New false
6539 1875 Acked 2874 Write 2 Done Done New false
6540 1876 Acked 2875 Write 5 Done Done New false
6541 1877 Acked 2876 Flush 0 Done Done New false
6542 1878 Acked 2877 Write 10 Done Done New false
6543 1879 Acked 2878 Read 2 Done Done New false
6544 1880 Acked 2879 Flush 0 Done Done New false
6545 1881 Acked 2880 Read 1 Done Done New false
6546 1882 Acked 2881 Write 1 Done Done New false
6547 1883 Acked 2882 Write 9 Done Done New false
6548 1884 Acked 2883 Read 6 Done Done New false
6549 1885 Acked 2884 Write 8 Done Done New false
6550 1886 Acked 2885 Read 4 Done Done New false
6551 1887 Acked 2886 Write 3 Done Done New false
6552 1888 Acked 2887 Read 9 Done Done New false
6553 1889 Acked 2888 Write 3 Done Done New false
6554 1890 Acked 2889 Read 2 Done Done New false
6555 1891 Acked 2890 Write 10 Done Done New false
6556 1892 Acked 2891 Write 10 Done Done New false
6557 1893 Acked 2892 Write 4 Done Done New false
6558 1894 Acked 2893 Write 5 Done Done New false
6559 1895 Acked 2894 Write 6 Done Done New false
6560 1896 Acked 2895 Read 6 Done Done New false
6561 1897 Acked 2896 Read 5 Done Done New false
6562 1898 Acked 2897 Read 6 Done Done New false
6563 1899 Acked 2898 Write 9 Done Done New false
6564 1900 Acked 2899 Read 8 Done Done New false
6565 1901 Acked 2900 Write 8 Done Done New false
6566 1902 Acked 2901 Read 10 Done Done New false
6567 1903 Acked 2902 Write 9 Done Done New false
6568 1904 Acked 2903 Write 10 Done Done New false
6569 1905 Acked 2904 Flush 0 Done Done New false
6570 1906 Acked 2905 Write 2 Done Done New false
6571 1907 Acked 2906 Write 5 Done Done New false
6572 1908 Acked 2907 Read 7 Done Done New false
6573 1909 Acked 2908 Write 5 Done Done New false
6574 1910 Acked 2909 Write 6 Done Done New false
6575 1911 Acked 2910 Read 2 Done Done New false
6576 1912 Acked 2911 Read 7 Done Done New false
6577 1913 Acked 2912 Write 7 Done Done New false
6578 1914 Acked 2913 Read 7 Done Done New false
6579 1915 Acked 2914 Write 2 Done Done New false
6580 1916 Acked 2915 Write 5 Done Done New false
6581 1917 Acked 2916 Read 4 Done Done New false
6582 1918 Acked 2917 Write 10 Done Done New false
6583 1919 Acked 2918 Read 4 Done Done New false
6584 1920 Acked 2919 Write 8 Done Done New false
6585 1921 Acked 2920 Read 1 Done Done New false
6586 1922 Acked 2921 Read 2 Done Done New false
6587 1923 Acked 2922 Read 6 Done Done New false
6588 1924 Acked 2923 Read 8 Done Done New false
6589 1925 Acked 2924 Flush 0 Done Done New false
6590 1926 Acked 2925 Read 3 Done Done New false
6591 1927 Acked 2926 Read 6 Done Done New false
6592 1928 Acked 2927 Write 6 Done Done New false
6593 1929 Acked 2928 Read 1 Done Done New false
6594 1930 Acked 2929 Read 9 Done Done New false
6595 1931 Acked 2930 Write 9 Done Done New false
6596 1932 Acked 2931 Read 6 Done Done New false
6597 1933 Acked 2932 Write 7 Done Done New false
6598 1934 Acked 2933 Write 2 Done Done New false
6599 1935 Acked 2934 Write 5 Done Done New false
6600 1936 Acked 2935 Read 8 Done Done New false
6601 1937 Acked 2936 Read 5 Done Done New false
6602 1938 Acked 2937 Read 3 Done Done New false
6603 1939 Acked 2938 Read 9 Done Done New false
6604 1940 Acked 2939 Write 4 Done Done New false
6605 1941 Acked 2940 Write 10 Done Done New false
6606 1942 Acked 2941 Read 6 Done Done New false
6607 1943 Acked 2942 Flush 0 Done Done New false
6608 1944 Acked 2943 Read 7 Done Done New false
6609 1945 Acked 2944 Write 1 Done Done New false
6610 1946 Acked 2945 Flush 0 Done Done New false
6611 1947 Acked 2946 Write 7 Done Done New false
6612 1948 Acked 2947 Write 10 Done Done New false
6613 1949 Acked 2948 Write 2 Done Done New false
6614 1950 Acked 2949 Write 2 Done Done New false
6615 1951 Acked 2950 Read 7 Done Done New false
6616 1952 Acked 2951 Write 8 Done Done New false
6617 1953 Acked 2952 Write 6 Done Done New false
6618 1954 Acked 2953 Flush 0 Done Done New false
6619 1955 Acked 2954 Write 7 Done Done New false
6620 1956 Acked 2955 Write 7 Done Done New false
6621 1957 Acked 2956 Read 2 Done Done New false
6622 1958 Acked 2957 Read 4 Done Done New false
6623 1959 Acked 2958 Read 10 Done Done New false
6624 1960 Acked 2959 Write 7 Done Done New false
6625 1961 Acked 2960 Write 1 Done Done New false
6626 1962 Acked 2961 Write 6 Done Done New false
6627 1963 Acked 2962 Read 5 Done Done New false
6628 1964 Acked 2963 Read 6 Done Done New false
6629 1965 Acked 2964 Read 7 Done Done New false
6630 1966 Acked 2965 Read 1 Done Done New false
6631 1967 Acked 2966 Write 8 Done Done New false
6632 1968 Acked 2967 Read 7 Done Done New false
6633 1969 Acked 2968 Write 2 Done Done New false
6634 1970 Acked 2969 Write 8 Done Done New false
6635 1971 Acked 2970 Read 10 Done Done New false
6636 1972 Acked 2971 Read 9 Done Done New false
6637 1973 Acked 2972 Read 3 Done Done New false
6638 1974 Acked 2973 Write 1 Done Done New false
6639 1975 Acked 2974 Flush 0 Done Done New false
6640 1976 Acked 2975 Flush 0 Done Done New false
6641 1977 Acked 2976 Read 10 Done Done New false
6642 1978 Acked 2977 Read 1 Done Done New false
6643 1979 Acked 2978 Read 6 Done Done New false
6644 1980 Acked 2979 Read 8 Done Done New false
6645 1981 Acked 2980 Read 5 Done Done New false
6646 1982 Acked 2981 Read 10 Done Done New false
6647 1983 Acked 2982 Write 4 Done Done New false
6648 1984 Acked 2983 Read 4 Done Done New false
6649 1985 Acked 2984 Write 4 Done Done New false
6650 1986 Acked 2985 Read 3 Done Done New false
6651 1987 Acked 2986 Read 1 Done Done New false
6652 1988 Acked 2987 Read 6 Done Done New false
6653 1989 Acked 2988 Write 5 Done Done New false
6654 1990 Acked 2989 Read 8 Done Done New false
6655 1991 Acked 2990 Read 10 Done Done New false
6656 1992 Acked 2991 Write 2 Done Done New false
6657 1993 Acked 2992 Write 6 Done Done New false
6658 1994 Acked 2993 Read 8 Done Done New false
6659 1995 Acked 2994 Read 6 Done Done New false
6660 1996 Acked 2995 Read 1 Done Done New false
6661 1997 Acked 2996 Read 1 Done Done New false
6662 1998 Acked 2997 Write 8 Done Done New false
6663 1999 Acked 2998 Read 4 Done Done New false
6664 2000 Acked 2999 Read 10 Done Done New false
6665 2001 Acked 3000 Write 3 Done Done New false
6666 2002 Acked 3001 Write 1 Done Done New false
6667 2003 Acked 3002 Flush 0 Done Done New false
6668 2004 Acked 3003 Read 10 Done Done New false
6669 2005 Acked 3004 Read 9 Done Done New false
6670 2006 Acked 3005 Read 10 Done Done New false
6671 2007 Acked 3006 Read 2 Done Done New false
6672 2008 Acked 3007 Read 10 Done Done New false
6673 2009 Acked 3008 Read 2 Done Done New false
6674 2010 Acked 3009 Read 10 Done Done New false
6675 2011 Acked 3010 Read 8 Done Done New false
6676 2012 Acked 3011 Write 7 Done Done New false
6677 2013 Acked 3012 Flush 0 Done Done New false
6678 2014 Acked 3013 Read 5 Done Done New false
6679 2015 Acked 3014 Write 7 Done Done New false
6680 2016 Acked 3015 Write 7 Done Done New false
6681 2017 Acked 3016 Read 5 Done Done New false
6682 2018 Acked 3017 Write 5 Done Done New false
6683 2019 Acked 3018 Write 2 Done Done New false
6684 2020 Acked 3019 Read 9 Done Done New false
6685 2021 Acked 3020 Read 8 Done Done New false
6686 2022 Acked 3021 Read 8 Done Done New false
6687 2023 Acked 3022 Read 2 Done Done New false
6688 2024 Acked 3023 Write 6 Done Done New false
6689 2025 Acked 3024 Write 3 Done Done New false
6690 2026 Acked 3025 Read 7 Done Done New false
6691 2027 Acked 3026 Flush 0 Done Done New false
6692 2028 Acked 3027 Write 2 Done Done New false
6693 2029 Acked 3028 Write 3 Done Done New false
6694 2030 Acked 3029 Read 5 Done Done New false
6695 2031 Acked 3030 Read 9 Done Done New false
6696 2032 Acked 3031 Read 1 Done Done New false
6697 2033 Acked 3032 Write 2 Done Done New false
6698 2034 Acked 3033 Read 1 Done Done New false
6699 2035 Acked 3034 Flush 0 Done Done New false
6700 2036 Acked 3035 Read 2 Done Done New false
6701 2037 Acked 3036 Write 1 Done Done New false
6702 2038 Acked 3037 Read 9 Done Done New false
6703 2039 Acked 3038 Write 10 Done Done New false
6704 2040 Acked 3039 Read 9 Done Done New false
6705 2041 Acked 3040 Write 1 Done Done New false
6706 2042 Acked 3041 Write 8 Done Done New false
6707 2043 Acked 3042 Read 5 Done Done New false
6708 2044 Acked 3043 Read 5 Done Done New false
6709 2045 Acked 3044 Write 5 Done Done New false
6710 2046 Acked 3045 Write 2 Done Done New false
6711 2047 Acked 3046 Read 8 Done Done New false
6712 2048 Acked 3047 Flush 0 Done Done New false
6713 2049 Acked 3048 Read 5 Done Done New false
6714 2050 Acked 3049 Read 5 Done Done New false
6715 2051 Acked 3050 Write 9 Done Done New false
6716 2052 Acked 3051 Read 1 Done Done New false
6717 2053 Acked 3052 Write 10 Done Done New false
6718 2054 Acked 3053 Read 4 Done Done New false
6719 2055 Acked 3054 Write 10 Done Done New false
6720 2056 Acked 3055 Read 10 Done Done New false
6721 2057 Acked 3056 Write 10 Done Done New false
6722 2058 Acked 3057 Read 8 Done Done New false
6723 2059 Acked 3058 Read 8 Done Done New false
6724 2060 Acked 3059 Read 3 Done Done New false
6725 2061 Acked 3060 Write 10 Done Done New false
6726 2062 Acked 3061 Read 10 Done Done New false
6727 2063 Acked 3062 Read 7 Done Done New false
6728 2064 Acked 3063 Read 2 Done Done New false
6729 2065 Acked 3064 Write 2 Done Done New false
6730 2066 Acked 3065 Write 6 Done Done New false
6731 2067 Acked 3066 Write 3 Done Done New false
6732 2068 Acked 3067 Write 1 Done Done New false
6733 2069 Acked 3068 Write 4 Done Done New false
6734 2070 Acked 3069 Read 5 Done Done New false
6735 2071 Acked 3070 Write 7 Done Done New false
6736 2072 Acked 3071 Read 9 Done Done New false
6737 2073 Acked 3072 Read 2 Done Done New false
6738 2074 Acked 3073 Write 6 Done Done New false
6739 2075 Acked 3074 Read 7 Done Done New false
6740 2076 Acked 3075 Write 8 Done Done New false
6741 2077 Acked 3076 Read 6 Done Done New false
6742 2078 Acked 3077 Read 8 Done Done New false
6743 2079 Acked 3078 Read 6 Done Done New false
6744 2080 Acked 3079 Flush 0 Done Done New false
6745 2081 Acked 3080 Read 1 Done Done New false
6746 2082 Acked 3081 Read 5 Done Done New false
6747 2083 Acked 3082 Write 1 Done Done New false
6748 2084 Acked 3083 Read 1 Done Done New false
6749 2085 Acked 3084 Write 9 Done Done New false
6750 2086 Acked 3085 Write 5 Done Done New false
6751 2087 Acked 3086 Read 1 Done Done New false
6752 2088 Acked 3087 Write 9 Done Done New false
6753 2089 Acked 3088 Read 5 Done Done New false
6754 2090 Acked 3089 Write 8 Done Done New false
6755 2091 Acked 3090 Read 2 Done Done New false
6756 2092 Acked 3091 Write 9 Done Done New false
6757 2093 Acked 3092 Read 7 Done Done New false
6758 2094 Acked 3093 Flush 0 Done Done New false
6759 2095 Acked 3094 Flush 0 Done Done New false
6760 2096 Acked 3095 Read 9 Done Done New false
6761 2097 Acked 3096 Write 5 Done Done New false
6762 2098 Acked 3097 Write 9 Done Done New false
6763 2099 Acked 3098 Write 1 Done Done New false
6764 2100 Acked 3099 Write 4 Done Done New false
6765 2101 Acked 3100 Read 5 Done Done New false
6766 2102 Acked 3101 Write 6 Done Done New false
6767 2103 Acked 3102 Read 9 Done Done New false
6768 2104 Acked 3103 Read 2 Done Done New false
6769 2105 Acked 3104 Write 8 Done Done New false
6770 2106 Acked 3105 Write 8 Done Done New false
6771 2107 Acked 3106 Write 5 Done Done New false
6772 2108 Acked 3107 Read 3 Done Done New false
6773 2109 Acked 3108 Read 10 Done Done New false
6774 2110 Acked 3109 Read 2 Done Done New false
6775 STATES DS:0 DS:1 DS:2 TOTAL
6776 New 0 0 300 300
6777 Sent 0 0 0 0
6778 Done 300 300 0 600
6779 Skipped 0 0 0 0
6780 Error 0 0 0 0
6781 Last Flush: 3094 3094 2809
6782 Downstairs last five completed: 2809 2808 2807 2806 2805
6783 Upstairs last five completed: 2110 2109 2108 2107 2106
6784 CLIENT: Up:0 ds:300 act:2
67852024-02-12T01:35:48.908ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
67862024-02-12T01:35:48.908ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
67872024-02-12T01:35:48.908ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
67882024-02-12T01:35:48.908ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
67892024-02-12T01:35:48.908ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
67902024-02-12T01:35:48.908ZINFOcrucible: [2] client re-new 301 jobs since flush 2809 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
67912024-02-12T01:35:48.908ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
67922024-02-12T01:35:49.910ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
67932024-02-12T01:35:49.910ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
67942024-02-12T01:35:49.910ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
67952024-02-12T01:35:49.910ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
67962024-02-12T01:35:49.910ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
67972024-02-12T01:35:49.910ZINFOcrucible: [2] client re-new 301 jobs since flush 2809 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
67982024-02-12T01:35:49.910ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
67992024-02-12T01:35:50.911ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
68002024-02-12T01:35:50.911ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
68012024-02-12T01:35:50.911ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
68022024-02-12T01:35:50.911ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
68032024-02-12T01:35:50.911ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
68042024-02-12T01:35:50.911ZINFOcrucible: [2] client re-new 301 jobs since flush 2809 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
68052024-02-12T01:35:50.911ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
68062024-02-12T01:35:51.912ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
68072024-02-12T01:35:51.912ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
68082024-02-12T01:35:51.912ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
68092024-02-12T01:35:51.912ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
68102024-02-12T01:35:51.912ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
68112024-02-12T01:35:51.912ZINFOcrucible: [2] client re-new 301 jobs since flush 2809 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
6812 [8830] got start action
68132024-02-12T01:35:51.913ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
6814 [8830] I am going to restart
6815 Starting downstairs at port 8830
6816 Make output file at "/tmp/dsc/downstairs-8830.txt"
6817 [8830][2] reports Starting
6818 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1171)
6819 [8830][2] reports Running
6820 ----------------------------------------------------------------
6821 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
6822 States: Active Active Offline
6823 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
6824 1811 Acked 2810 Write 10 Done Done New true
6825 1812 Acked 2811 Read 7 Done Done New true
6826 1813 Acked 2812 Flush 0 Done Done New true
6827 1814 Acked 2813 Read 8 Done Done New true
6828 1815 Acked 2814 Read 7 Done Done New true
6829 1816 Acked 2815 Write 5 Done Done New true
6830 1817 Acked 2816 Write 2 Done Done New true
6831 1818 Acked 2817 Write 9 Done Done New true
6832 1819 Acked 2818 Write 2 Done Done New true
6833 1820 Acked 2819 Write 9 Done Done New true
6834 1821 Acked 2820 Read 9 Done Done New true
6835 1822 Acked 2821 Read 2 Done Done New true
6836 1823 Acked 2822 Read 1 Done Done New true
6837 1824 Acked 2823 Write 2 Done Done New true
6838 1825 Acked 2824 Read 6 Done Done New true
6839 1826 Acked 2825 Write 3 Done Done New true
6840 1827 Acked 2826 Read 2 Done Done New true
6841 1828 Acked 2827 Write 3 Done Done New true
6842 1829 Acked 2828 Write 2 Done Done New true
6843 1830 Acked 2829 Read 6 Done Done New true
6844 1831 Acked 2830 Read 2 Done Done New true
6845 1832 Acked 2831 Write 1 Done Done New true
6846 1833 Acked 2832 Write 5 Done Done New true
6847 1834 Acked 2833 Read 7 Done Done New true
6848 1835 Acked 2834 Read 10 Done Done New true
6849 1836 Acked 2835 Read 4 Done Done New true
6850 1837 Acked 2836 Write 3 Done Done New true
6851 1838 Acked 2837 Read 2 Done Done New true
6852 1839 Acked 2838 Read 4 Done Done New true
6853 1840 Acked 2839 Write 3 Done Done New true
6854 1841 Acked 2840 Write 1 Done Done New true
6855 1842 Acked 2841 Read 2 Done Done New true
6856 1843 Acked 2842 Read 8 Done Done New true
6857 1844 Acked 2843 Read 8 Done Done New true
6858 1845 Acked 2844 Read 6 Done Done New true
6859 1846 Acked 2845 Flush 0 Done Done New true
6860 1847 Acked 2846 Read 7 Done Done New true
6861 1848 Acked 2847 Read 5 Done Done New true
6862 1849 Acked 2848 Read 5 Done Done New true
6863 1850 Acked 2849 Read 10 Done Done New true
6864 1851 Acked 2850 Read 10 Done Done New true
6865 1852 Acked 2851 Read 5 Done Done New true
6866 1853 Acked 2852 Read 5 Done Done New true
6867 1854 Acked 2853 Read 4 Done Done New true
6868 1855 Acked 2854 Write 5 Done Done New true
6869 1856 Acked 2855 Read 10 Done Done New true
6870 1857 Acked 2856 Read 2 Done Done New true
6871 1858 Acked 2857 Read 7 Done Done New true
6872 1859 Acked 2858 Write 1 Done Done New true
6873 1860 Acked 2859 Read 7 Done Done New true
6874 1861 Acked 2860 Write 9 Done Done New true
6875 1862 Acked 2861 Read 1 Done Done New true
6876 1863 Acked 2862 Write 2 Done Done New true
6877 1864 Acked 2863 Write 3 Done Done New true
6878 1865 Acked 2864 Flush 0 Done Done New true
6879 1866 Acked 2865 Write 10 Done Done New true
6880 1867 Acked 2866 Read 4 Done Done New true
6881 1868 Acked 2867 Write 6 Done Done New true
6882 1869 Acked 2868 Write 1 Done Done New true
6883 1870 Acked 2869 Read 4 Done Done New true
6884 1871 Acked 2870 Write 6 Done Done New true
6885 1872 Acked 2871 Write 2 Done Done New true
6886 1873 Acked 2872 Write 7 Done Done New true
6887 1874 Acked 2873 Write 10 Done Done New true
6888 1875 Acked 2874 Write 2 Done Done New true
6889 1876 Acked 2875 Write 5 Done Done New true
6890 1877 Acked 2876 Flush 0 Done Done New true
6891 1878 Acked 2877 Write 10 Done Done New true
6892 1879 Acked 2878 Read 2 Done Done New true
6893 1880 Acked 2879 Flush 0 Done Done New true
6894 1881 Acked 2880 Read 1 Done Done New true
6895 1882 Acked 2881 Write 1 Done Done New true
6896 1883 Acked 2882 Write 9 Done Done New true
6897 1884 Acked 2883 Read 6 Done Done New true
6898 1885 Acked 2884 Write 8 Done Done New true
6899 1886 Acked 2885 Read 4 Done Done New true
6900 1887 Acked 2886 Write 3 Done Done New true
6901 1888 Acked 2887 Read 9 Done Done New true
6902 1889 Acked 2888 Write 3 Done Done New true
6903 1890 Acked 2889 Read 2 Done Done New true
6904 1891 Acked 2890 Write 10 Done Done New true
6905 1892 Acked 2891 Write 10 Done Done New true
6906 1893 Acked 2892 Write 4 Done Done New true
6907 1894 Acked 2893 Write 5 Done Done New true
6908 1895 Acked 2894 Write 6 Done Done New true
6909 1896 Acked 2895 Read 6 Done Done New true
6910 1897 Acked 2896 Read 5 Done Done New true
6911 1898 Acked 2897 Read 6 Done Done New true
6912 1899 Acked 2898 Write 9 Done Done New true
6913 1900 Acked 2899 Read 8 Done Done New true
6914 1901 Acked 2900 Write 8 Done Done New true
6915 1902 Acked 2901 Read 10 Done Done New true
6916 1903 Acked 2902 Write 9 Done Done New true
6917 1904 Acked 2903 Write 10 Done Done New true
6918 1905 Acked 2904 Flush 0 Done Done New true
6919 1906 Acked 2905 Write 2 Done Done New true
6920 1907 Acked 2906 Write 5 Done Done New true
6921 1908 Acked 2907 Read 7 Done Done New true
6922 1909 Acked 2908 Write 5 Done Done New true
6923 1910 Acked 2909 Write 6 Done Done New true
6924 1911 Acked 2910 Read 2 Done Done New true
6925 1912 Acked 2911 Read 7 Done Done New true
6926 1913 Acked 2912 Write 7 Done Done New true
6927 1914 Acked 2913 Read 7 Done Done New true
6928 1915 Acked 2914 Write 2 Done Done New true
6929 1916 Acked 2915 Write 5 Done Done New true
6930 1917 Acked 2916 Read 4 Done Done New true
6931 1918 Acked 2917 Write 10 Done Done New true
6932 1919 Acked 2918 Read 4 Done Done New true
6933 1920 Acked 2919 Write 8 Done Done New true
6934 1921 Acked 2920 Read 1 Done Done New true
6935 1922 Acked 2921 Read 2 Done Done New true
6936 1923 Acked 2922 Read 6 Done Done New true
6937 1924 Acked 2923 Read 8 Done Done New true
6938 1925 Acked 2924 Flush 0 Done Done New true
6939 1926 Acked 2925 Read 3 Done Done New true
6940 1927 Acked 2926 Read 6 Done Done New true
6941 1928 Acked 2927 Write 6 Done Done New true
6942 1929 Acked 2928 Read 1 Done Done New true
6943 1930 Acked 2929 Read 9 Done Done New true
6944 1931 Acked 2930 Write 9 Done Done New true
6945 1932 Acked 2931 Read 6 Done Done New true
6946 1933 Acked 2932 Write 7 Done Done New true
6947 1934 Acked 2933 Write 2 Done Done New true
6948 1935 Acked 2934 Write 5 Done Done New true
6949 1936 Acked 2935 Read 8 Done Done New true
6950 1937 Acked 2936 Read 5 Done Done New true
6951 1938 Acked 2937 Read 3 Done Done New true
6952 1939 Acked 2938 Read 9 Done Done New true
6953 1940 Acked 2939 Write 4 Done Done New true
6954 1941 Acked 2940 Write 10 Done Done New true
6955 1942 Acked 2941 Read 6 Done Done New true
6956 1943 Acked 2942 Flush 0 Done Done New true
6957 1944 Acked 2943 Read 7 Done Done New true
6958 1945 Acked 2944 Write 1 Done Done New true
6959 1946 Acked 2945 Flush 0 Done Done New true
6960 1947 Acked 2946 Write 7 Done Done New true
6961 1948 Acked 2947 Write 10 Done Done New true
6962 1949 Acked 2948 Write 2 Done Done New true
6963 1950 Acked 2949 Write 2 Done Done New true
6964 1951 Acked 2950 Read 7 Done Done New true
6965 1952 Acked 2951 Write 8 Done Done New true
6966 1953 Acked 2952 Write 6 Done Done New true
6967 1954 Acked 2953 Flush 0 Done Done New true
6968 1955 Acked 2954 Write 7 Done Done New true
6969 1956 Acked 2955 Write 7 Done Done New true
6970 1957 Acked 2956 Read 2 Done Done New true
6971 1958 Acked 2957 Read 4 Done Done New true
6972 1959 Acked 2958 Read 10 Done Done New true
6973 1960 Acked 2959 Write 7 Done Done New true
6974 1961 Acked 2960 Write 1 Done Done New true
6975 1962 Acked 2961 Write 6 Done Done New true
6976 1963 Acked 2962 Read 5 Done Done New true
6977 1964 Acked 2963 Read 6 Done Done New true
6978 1965 Acked 2964 Read 7 Done Done New true
6979 1966 Acked 2965 Read 1 Done Done New true
6980 1967 Acked 2966 Write 8 Done Done New true
6981 1968 Acked 2967 Read 7 Done Done New true
6982 1969 Acked 2968 Write 2 Done Done New true
6983 1970 Acked 2969 Write 8 Done Done New true
6984 1971 Acked 2970 Read 10 Done Done New true
6985 1972 Acked 2971 Read 9 Done Done New true
6986 1973 Acked 2972 Read 3 Done Done New true
6987 1974 Acked 2973 Write 1 Done Done New true
6988 1975 Acked 2974 Flush 0 Done Done New true
6989 1976 Acked 2975 Flush 0 Done Done New true
6990 1977 Acked 2976 Read 10 Done Done New true
6991 1978 Acked 2977 Read 1 Done Done New true
6992 1979 Acked 2978 Read 6 Done Done New true
6993 1980 Acked 2979 Read 8 Done Done New true
6994 1981 Acked 2980 Read 5 Done Done New true
6995 1982 Acked 2981 Read 10 Done Done New true
6996 1983 Acked 2982 Write 4 Done Done New true
6997 1984 Acked 2983 Read 4 Done Done New true
6998 1985 Acked 2984 Write 4 Done Done New true
6999 1986 Acked 2985 Read 3 Done Done New true
7000 1987 Acked 2986 Read 1 Done Done New true
7001 1988 Acked 2987 Read 6 Done Done New true
7002 1989 Acked 2988 Write 5 Done Done New true
7003 1990 Acked 2989 Read 8 Done Done New true
7004 1991 Acked 2990 Read 10 Done Done New true
7005 1992 Acked 2991 Write 2 Done Done New true
7006 1993 Acked 2992 Write 6 Done Done New true
7007 1994 Acked 2993 Read 8 Done Done New true
7008 1995 Acked 2994 Read 6 Done Done New true
7009 1996 Acked 2995 Read 1 Done Done New true
7010 1997 Acked 2996 Read 1 Done Done New true
7011 1998 Acked 2997 Write 8 Done Done New true
7012 1999 Acked 2998 Read 4 Done Done New true
7013 2000 Acked 2999 Read 10 Done Done New true
7014 2001 Acked 3000 Write 3 Done Done New true
7015 2002 Acked 3001 Write 1 Done Done New true
7016 2003 Acked 3002 Flush 0 Done Done New true
7017 2004 Acked 3003 Read 10 Done Done New true
7018 2005 Acked 3004 Read 9 Done Done New true
7019 2006 Acked 3005 Read 10 Done Done New true
7020 2007 Acked 3006 Read 2 Done Done New true
7021 2008 Acked 3007 Read 10 Done Done New true
7022 2009 Acked 3008 Read 2 Done Done New true
7023 2010 Acked 3009 Read 10 Done Done New true
7024 2011 Acked 3010 Read 8 Done Done New true
7025 2012 Acked 3011 Write 7 Done Done New true
7026 2013 Acked 3012 Flush 0 Done Done New true
7027 2014 Acked 3013 Read 5 Done Done New true
7028 2015 Acked 3014 Write 7 Done Done New true
7029 2016 Acked 3015 Write 7 Done Done New true
7030 2017 Acked 3016 Read 5 Done Done New true
7031 2018 Acked 3017 Write 5 Done Done New true
7032 2019 Acked 3018 Write 2 Done Done New true
7033 2020 Acked 3019 Read 9 Done Done New true
7034 2021 Acked 3020 Read 8 Done Done New true
7035 2022 Acked 3021 Read 8 Done Done New true
7036 2023 Acked 3022 Read 2 Done Done New true
7037 2024 Acked 3023 Write 6 Done Done New true
7038 2025 Acked 3024 Write 3 Done Done New true
7039 2026 Acked 3025 Read 7 Done Done New true
7040 2027 Acked 3026 Flush 0 Done Done New true
7041 2028 Acked 3027 Write 2 Done Done New true
7042 2029 Acked 3028 Write 3 Done Done New true
7043 2030 Acked 3029 Read 5 Done Done New true
7044 2031 Acked 3030 Read 9 Done Done New true
7045 2032 Acked 3031 Read 1 Done Done New true
7046 2033 Acked 3032 Write 2 Done Done New true
7047 2034 Acked 3033 Read 1 Done Done New true
7048 2035 Acked 3034 Flush 0 Done Done New true
7049 2036 Acked 3035 Read 2 Done Done New true
7050 2037 Acked 3036 Write 1 Done Done New true
7051 2038 Acked 3037 Read 9 Done Done New true
7052 2039 Acked 3038 Write 10 Done Done New true
7053 2040 Acked 3039 Read 9 Done Done New true
7054 2041 Acked 3040 Write 1 Done Done New true
7055 2042 Acked 3041 Write 8 Done Done New true
7056 2043 Acked 3042 Read 5 Done Done New true
7057 2044 Acked 3043 Read 5 Done Done New true
7058 2045 Acked 3044 Write 5 Done Done New true
7059 2046 Acked 3045 Write 2 Done Done New true
7060 2047 Acked 3046 Read 8 Done Done New true
7061 2048 Acked 3047 Flush 0 Done Done New true
7062 2049 Acked 3048 Read 5 Done Done New true
7063 2050 Acked 3049 Read 5 Done Done New true
7064 2051 Acked 3050 Write 9 Done Done New true
7065 2052 Acked 3051 Read 1 Done Done New true
7066 2053 Acked 3052 Write 10 Done Done New true
7067 2054 Acked 3053 Read 4 Done Done New true
7068 2055 Acked 3054 Write 10 Done Done New true
7069 2056 Acked 3055 Read 10 Done Done New true
7070 2057 Acked 3056 Write 10 Done Done New true
7071 2058 Acked 3057 Read 8 Done Done New true
7072 2059 Acked 3058 Read 8 Done Done New true
7073 2060 Acked 3059 Read 3 Done Done New true
7074 2061 Acked 3060 Write 10 Done Done New true
7075 2062 Acked 3061 Read 10 Done Done New true
7076 2063 Acked 3062 Read 7 Done Done New true
7077 2064 Acked 3063 Read 2 Done Done New true
7078 2065 Acked 3064 Write 2 Done Done New true
7079 2066 Acked 3065 Write 6 Done Done New true
7080 2067 Acked 3066 Write 3 Done Done New true
7081 2068 Acked 3067 Write 1 Done Done New true
7082 2069 Acked 3068 Write 4 Done Done New true
7083 2070 Acked 3069 Read 5 Done Done New true
7084 2071 Acked 3070 Write 7 Done Done New true
7085 2072 Acked 3071 Read 9 Done Done New true
7086 2073 Acked 3072 Read 2 Done Done New true
7087 2074 Acked 3073 Write 6 Done Done New true
7088 2075 Acked 3074 Read 7 Done Done New true
7089 2076 Acked 3075 Write 8 Done Done New true
7090 2077 Acked 3076 Read 6 Done Done New true
7091 2078 Acked 3077 Read 8 Done Done New true
7092 2079 Acked 3078 Read 6 Done Done New true
7093 2080 Acked 3079 Flush 0 Done Done New true
7094 2081 Acked 3080 Read 1 Done Done New true
7095 2082 Acked 3081 Read 5 Done Done New true
7096 2083 Acked 3082 Write 1 Done Done New true
7097 2084 Acked 3083 Read 1 Done Done New true
7098 2085 Acked 3084 Write 9 Done Done New true
7099 2086 Acked 3085 Write 5 Done Done New true
7100 2087 Acked 3086 Read 1 Done Done New true
7101 2088 Acked 3087 Write 9 Done Done New true
7102 2089 Acked 3088 Read 5 Done Done New true
7103 2090 Acked 3089 Write 8 Done Done New true
7104 2091 Acked 3090 Read 2 Done Done New true
7105 2092 Acked 3091 Write 9 Done Done New true
7106 2093 Acked 3092 Read 7 Done Done New true
7107 2094 Acked 3093 Flush 0 Done Done New true
7108 2095 Acked 3094 Flush 0 Done Done New true
7109 2096 Acked 3095 Read 9 Done Done New true
7110 2097 Acked 3096 Write 5 Done Done New true
7111 2098 Acked 3097 Write 9 Done Done New true
7112 2099 Acked 3098 Write 1 Done Done New true
7113 2100 Acked 3099 Write 4 Done Done New true
7114 2101 Acked 3100 Read 5 Done Done New true
7115 2102 Acked 3101 Write 6 Done Done New true
7116 2103 Acked 3102 Read 9 Done Done New true
7117 2104 Acked 3103 Read 2 Done Done New true
7118 2105 Acked 3104 Write 8 Done Done New true
7119 2106 Acked 3105 Write 8 Done Done New true
7120 2107 Acked 3106 Write 5 Done Done New true
7121 2108 Acked 3107 Read 3 Done Done New true
7122 2109 Acked 3108 Read 10 Done Done New true
7123 2110 Acked 3109 Read 2 Done Done New true
7124 2111 Acked 3110 Flush 0 Done Done New true
7125 STATES DS:0 DS:1 DS:2 TOTAL
7126 New 0 0 301 301
7127 Sent 0 0 0 0
7128 Done 301 301 0 602
7129 Skipped 0 0 0 0
7130 Error 0 0 0 0
7131 Last Flush: 3110 3110 2809
7132 Downstairs last five completed: 2809 2808 2807 2806 2805
7133 Upstairs last five completed: 2111 2110 2109 2108 2107
7134 CLIENT: Up:0 ds:301 act:2
71352024-02-12T01:35:52.913ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
71362024-02-12T01:35:52.914ZINFOcrucible: ds_connection connected = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
71372024-02-12T01:35:52.915ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
71382024-02-12T01:35:52.915ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
71392024-02-12T01:35:52.915ZINFOcrucible: send last flush ID to this DS: 2809 = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
71402024-02-12T01:35:52.915ZINFOcrucible: Replied this last flush ID: 2809 = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
71412024-02-12T01:35:52.915ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
71422024-02-12T01:35:52.915ZINFOcrucible: [2] Transition from Offline to Replay = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
71432024-02-12T01:35:52.915ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
71442024-02-12T01:35:52.915ZINFOcrucible: [2] Transition from Replay to Active = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
7145 ----------------------------------------------------------------
7146 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
7147 Downstairs last five completed: 3110 3109 3108 3107 3106
7148 Upstairs last five completed: 2111 2110 2109 2108 2107
7149 CLIENT: Up:0 ds:0 act:3
7150 Replay: All jobs finished, all DS active.
7151 Main task has work to do, go find it
7152 got dsc Stop(2)
7153 stop 2
7154 Feb 12 01:35:56.258 INFO request completed, latency_us: 370, response_code: 204, uri[8830] Got stop action so:false kr:false
7155 : /stop/cid/2, method: POST, req_id: 26b00602-4105-449e-88a5-a90385565d77, remote_addr: 127.0.0.1:60456, local_addr: 127.0.0.1:9998
71562024-02-12T01:35:56.258ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
71572024-02-12T01:35:56.258ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
7158 Feb 12 01:35:56.258 INFO request completed, latency_us: 121, response_code: 200, {"msg":"downstairs task for 2 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-02-12T01:35:56.258939869Z","hostname":"ip-10-150-1-104.us-west-2.compute.internal","pid":1164,"session_id":"e1773b3e-b36f-4080-8b13-c0830f0d072e"}
7159 uri: /state/cid/2, method: GET, req_id: 7b7f4f75-5b36-410e-a5dd-713b8441a84c, remote_addr: 127.0.0.1:60456, local_addr: 127.0.0.1:9998
71602024-02-12T01:35:56.258ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
71612024-02-12T01:35:56.259ZINFOcrucible: [2] client re-new 0 jobs since flush 3110 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
71622024-02-12T01:35:56.259ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
7163 [8830] Exited with: ExitStatus(unix_wait_status(9))
7164 [8830][2] reports Exit
71652024-02-12T01:35:57.260ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
71662024-02-12T01:35:57.260ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
71672024-02-12T01:35:57.260ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
71682024-02-12T01:35:57.260ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
71692024-02-12T01:35:57.260ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
71702024-02-12T01:35:57.260ZINFOcrucible: [2] client re-new 0 jobs since flush 3110 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
71712024-02-12T01:35:57.260ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
71722024-02-12T01:35:58.261ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
71732024-02-12T01:35:58.261ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
71742024-02-12T01:35:58.261ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
71752024-02-12T01:35:58.261ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
71762024-02-12T01:35:58.261ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
71772024-02-12T01:35:58.261ZINFOcrucible: [2] client re-new 0 jobs since flush 3110 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
71782024-02-12T01:35:58.261ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
71792024-02-12T01:35:59.262ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
71802024-02-12T01:35:59.262ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
71812024-02-12T01:35:59.262ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
71822024-02-12T01:35:59.262ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
71832024-02-12T01:35:59.262ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
71842024-02-12T01:35:59.262ZINFOcrucible: [2] client re-new 0 jobs since flush 3110 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
71852024-02-12T01:35:59.262ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
7186 Feb 12 01:36:00.261 INFO request completed, latency_us: 208, response_code: 200, uri: /state/cid/2, method: GET, req_id: 2eff58a5-5c20-4be8-bcf1-742612fd1cdc, remote_addr: 127.0.0.1:60456, local_addr: 127.0.0.1:9998
7187 001/300 Write block 1844 len 32768 data: 4 4 4 5 5 4 5 5
7188 002/300 Read block 902 len 32768
71892024-02-12T01:36:00.263ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
71902024-02-12T01:36:00.263ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
71912024-02-12T01:36:00.263ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
71922024-02-12T01:36:00.263ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
71932024-02-12T01:36:00.263ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
71942024-02-12T01:36:00.263ZINFOcrucible: [2] client re-new 2 jobs since flush 3110 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
71952024-02-12T01:36:00.263ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
7196 003/300 Read block 3548 len 28672
7197 004/300 Read block 504 len 16384
7198 005/300 Flush
7199 006/300 Write block 1201 len 40960 data: 2 2 2 3 4 3 3 3 4 4
7200 007/300 Read block 4425 len 16384
7201 008/300 Read block 1096 len 40960
7202 009/300 Read block 2829 len 8192
7203 010/300 Write block 2115 len 36864 data: 2 2 2 2 2 2 2 2 2
7204 011/300 Write block 3973 len 20480 data: 2 2 2 2 2
7205 012/300 Read block 1630 len 12288
7206 013/300 Read block 2922 len 32768
7207 014/300 Write block 3686 len 40960 data: 4 4 4 3 3 2 2 2 2 2
7208 015/300 Read block 2622 len 8192
7209 016/300 Read block 935 len 12288
7210 017/300 Read block 4071 len 32768
7211 018/300 Write block 4583 len 8192 data: 2 2
7212 019/300 Write block 418 len 16384 data: 2 2 2 2
7213 020/300 Write block 3591 len 28672 data: 2 2 2 2 3 3 3
7214 021/300 Flush
7215 022/300 Read block 3281 len 28672
7216 023/300 Read block 3322 len 20480
7217 024/300 Read block 2558 len 28672
7218 025/300 Read block 617 len 8192
7219 026/300 Write block 1613 len 40960 data: 4 4 4 3 4 3 4 4 4 4
7220 027/300 Write block 4061 len 28672 data: 3 3 3 2 2 3 3
7221 028/300 Write block 2649 len 36864 data: 5 4 4 5 3 2 2 3 3
7222 029/300 Read block 2461 len 24576
7223 030/300 Read block 1988 len 8192
7224 031/300 Write block 2869 len 40960 data: 2 2 2 2 2 2 2 3 3 3
7225 032/300 Write block 440 len 40960 data: 3 3 3 3 4 4 4 3 3 3
7226 033/300 Write block 1833 len 40960 data: 3 3 2 2 2 3 4 4 3 4
7227 034/300 Write block 3036 len 28672 data: 3 3 3 3 3 3 4
7228 035/300 Read block 1307 len 32768
7229 036/300 Read block 2628 len 40960
7230 037/300 Read block 1322 len 12288
7231 038/300 Write block 1697 len 28672 data: 2 2 3 3 3 4 5
7232 039/300 Write block 1610 len 24576 data: 4 4 4 5 5 5
7233 040/300 Read block 3626 len 8192
7234 041/300 Write block 2319 len 4096 data: 3
7235 042/300 Read block 4076 len 32768
7236 043/300 Read block 853 len 40960
7237 044/300 Write block 4404 len 32768 data: 3 3 3 3 3 2 2 2
7238 045/300 Read block 3132 len 28672
7239 046/300 Read block 4541 len 16384
7240 047/300 Read block 1560 len 16384
7241 048/300 Write block 1445 len 20480 data: 2 2 2 3 3
7242 049/300 Read block 3289 len 20480
7243 050/300 Read block 2150 len 8192
7244 051/300 Flush
7245 052/300 Read block 122 len 4096
7246 053/300 Write block 2093 len 12288 data: 4 4 4
7247 054/300 Flush
7248 055/300 Read block 1591 len 36864
7249 056/300 Write block 433 len 16384 data: 2 2 2 2
7250 057/300 Read block 1194 len 40960
7251 058/300 Read block 3400 len 20480
7252 059/300 Write block 173 len 36864 data: 2 2 2 2 2 2 2 2 2
7253 060/300 Read block 4362 len 32768
7254 061/300 Read block 252 len 16384
7255 062/300 Write block 465 len 4096 data: 3
7256 063/300 Flush
7257 064/300 Write block 3222 len 32768 data: 4 4 5 5 5 4 5 4
7258 065/300 Read block 3902 len 8192
7259 066/300 Read block 1692 len 8192
7260 067/300 Read block 2812 len 8192
7261 068/300 Read block 3814 len 4096
7262 069/300 Write block 1803 len 40960 data: 2 2 2 3 3 3 4 4 3 3
7263 070/300 Write block 4470 len 28672 data: 3 3 4 4 4 4 4
7264 071/300 Read block 2057 len 32768
7265 072/300 Read block 2615 len 16384
7266 073/300 Read block 1963 len 36864
7267 074/300 Read block 195 len 28672
7268 075/300 Write block 670 len 28672 data: 3 4 5 6 5 4 4
7269 076/300 Read block 3784 len 12288
7270 077/300 Read block 4091 len 20480
7271 078/300 Write block 2612 len 4096 data: 2
7272 079/300 Flush
7273 080/300 Flush
7274 081/300 Write block 1954 len 28672 data: 2 2 2 2 2 2 2
7275 082/300 Write block 1263 len 20480 data: 3 4 4 3 3
7276 083/300 Read block 3384 len 36864
7277 084/300 Read block 1919 len 28672
7278 085/300 Read block 320 len 8192
7279 086/300 Read block 2670 len 20480
7280 087/300 Write block 718 len 8192 data: 3 2
7281 088/300 Flush
7282 089/300 Write block 2263 len 28672 data: 3 2 2 2 2 2 2
7283 090/300 Read block 4804 len 28672
7284 091/300 Read block 2211 len 8192
7285 092/300 Read block 4270 len 16384
7286 093/300 Flush
7287 094/300 Read block 9 len 36864
7288 095/300 Write block 875 len 28672 data: 2 2 2 3 3 4 5
7289 096/300 Read block 2884 len 20480
7290 097/300 Read block 1032 len 36864
7291 098/300 Write block 4488 len 28672 data: 2 2 2 2 2 2 2
7292 099/300 Read block 2865 len 8192
7293 100/300 Read block 1140 len 36864
7294 101/300 Read block 1655 len 20480
7295 102/300 Read block 4950 len 12288
7296 103/300 Read block 2266 len 20480
7297 104/300 Write block 2807 len 24576 data: 7 5 6 5 4 4
7298 105/300 Write block 2424 len 28672 data: 5 5 5 5 3 3 3
7299 106/300 Write block 3127 len 36864 data: 3 3 3 4 3 3 3 3 3
7300 107/300 Read block 4835 len 36864
7301 108/300 Read block 1577 len 12288
7302 109/300 Read block 2874 len 24576
7303 110/300 Read block 1529 len 20480
7304 111/300 Write block 3123 len 16384 data: 2 2 3 3
7305 112/300 Flush
7306 113/300 Write block 3739 len 32768 data: 3 2 2 2 2 2 2 2
7307 114/300 Write block 167 len 12288 data: 3 2 3
7308 115/300 Read block 1478 len 4096
7309 116/300 Read block 3438 len 16384
7310 117/300 Write block 1340 len 4096 data: 4
7311 118/300 Write block 1764 len 36864 data: 2 2 3 3 3 3 4 4 5
7312 119/300 Read block 2808 len 36864
7313 120/300 Read block 3956 len 36864
7314 121/300 Write block 3992 len 12288 data: 2 2 2
7315 122/300 Read block 1499 len 16384
7316 123/300 Write block 3998 len 40960 data: 4 4 4 4 2 2 2 2 3 3
7317 124/300 Read block 3930 len 12288
7318 125/300 Read block 1651 len 20480
7319 126/300 Write block 1364 len 12288 data: 3 3 2
7320 127/300 Read block 4391 len 20480
7321 128/300 Flush
7322 129/300 Write block 4375 len 28672 data: 2 2 2 2 2 2 2
7323 130/300 Write block 2237 len 40960 data: 3 3 3 2 2 2 2 2 2 2
7324 131/300 Read block 1478 len 40960
7325 132/300 Read block 887 len 12288
7326 133/300 Write block 3522 len 20480 data: 4 3 3 3 3
7327 134/300 Write block 2177 len 12288 data: 4 4 4
7328 135/300 Write block 3076 len 32768 data: 2 2 2 2 3 3 3 3
7329 136/300 Write block 168 len 4096 data: 3
7330 137/300 Write block 4022 len 40960 data: 7 6 5 5 5 4 2 3 2 2
7331 138/300 Read block 4038 len 24576
7332 139/300 Write block 4439 len 32768 data: 2 2 2 2 2 2 2 2
7333 140/300 Read block 1994 len 40960
7334 141/300 Read block 3972 len 40960
7335 142/300 Write block 4436 len 8192 data: 2 2
7336 143/300 Read block 1583 len 4096
7337 144/300 Read block 705 len 28672
7338 145/300 Write block 4697 len 8192 data: 3 3
7339 146/300 Write block 4072 len 20480 data: 4 4 3 3 3
7340 147/300 Read block 302 len 8192
7341 148/300 Read block 3041 len 4096
7342 149/300 Read block 947 len 12288
7343 150/300 Write block 3114 len 4096 data: 3
7344 151/300 Write block 355 len 20480 data: 3 4 4 4 3
7345 152/300 Write block 3374 len 20480 data: 4 4 5 5 5
7346 153/300 Read block 471 len 8192
7347 154/300 Read block 2395 len 8192
7348 155/300 Read block 4175 len 28672
7349 156/300 Read block 3151 len 40960
7350 157/300 Read block 3372 len 8192
7351 158/300 Read block 3897 len 12288
7352 159/300 Write block 2130 len 8192 data: 3 4
7353 160/300 Read block 4256 len 24576
7354 161/300 Write block 2952 len 36864 data: 2 2 3 3 3 3 3 3 4
7355 162/300 Read block 3323 len 24576
7356 163/300 Write block 3140 len 28672 data: 2 2 3 3 2 2 2
7357 164/300 Read block 4697 len 36864
7358 165/300 Write block 3979 len 36864 data: 2 2 2 2 2 2 2 2 2
7359 166/300 Read block 329 len 16384
7360 167/300 Read block 3269 len 20480
7361 168/300 Write block 2201 len 32768 data: 4 4 3 3 3 2 2 3
7362 169/300 Read block 1421 len 12288
7363 170/300 Write block 3681 len 36864 data: 3 4 4 4 4 5 5 5 4
7364 171/300 Write block 2193 len 8192 data: 3 2
7365 172/300 Write block 3698 len 40960 data: 3 3 3 4 3 3 2 2 2 2
7366 173/300 Flush
7367 174/300 Write block 1968 len 32768 data: 3 3 3 3 3 3 3 3
7368 175/300 Read block 3384 len 28672
7369 176/300 Write block 3109 len 36864 data: 2 2 2 3 3 4 3 3 3
7370 177/300 Write block 3961 len 4096 data: 3
7371 178/300 Read block 36 len 12288
7372 179/300 Read block 3752 len 20480
7373 180/300 Write block 3233 len 28672 data: 4 3 3 3 3 3 3
7374 181/300 Read block 4885 len 24576
7375 182/300 Read block 2944 len 4096
7376 183/300 Write block 2689 len 28672 data: 3 2 2 2 2 2 2
7377 184/300 Read block 4239 len 36864
7378 185/300 Write block 1464 len 4096 data: 2
7379 186/300 Write block 3445 len 40960 data: 4 4 4 3 3 5 5 5 5 4
7380 187/300 Read block 886 len 24576
7381 188/300 Read block 889 len 8192
7382 189/300 Read block 2268 len 36864
7383 190/300 Write block 3850 len 4096 data: 2
7384 191/300 Read block 2092 len 16384
7385 192/300 Read block 3211 len 16384
7386 193/300 Write block 222 len 24576 data: 3 3 3 3 3 2
7387 194/300 Write block 4933 len 16384 data: 3 3 4 3
7388 195/300 Read block 1438 len 16384
7389 196/300 Read block 3343 len 40960
7390 197/300 Read block 4760 len 36864
7391 198/300 Write block 2175 len 24576 data: 5 5 5 5 5 3
7392 199/300 Write block 1876 len 16384 data: 3 3 3 3
7393 200/300 Write block 4726 len 8192 data: 2 2
7394 201/300 Read block 3700 len 32768
7395 202/300 Flush
7396 203/300 Write block 3110 len 40960 data: 3 3 4 4 5 4 4 4 2 2
7397 204/300 Write block 3469 len 12288 data: 4 6 6
7398 205/300 Read block 4473 len 32768
7399 206/300 Read block 3571 len 12288
7400 207/300 Read block 2451 len 20480
7401 208/300 Read block 4887 len 16384
7402 209/300 Write block 700 len 4096 data: 2
7403 210/300 Write block 1600 len 4096 data: 3
7404 211/300 Read block 1062 len 4096
7405 212/300 Read block 4582 len 20480
7406 213/300 Read block 2089 len 12288
7407 214/300 Write block 4354 len 24576 data: 3 3 4 3 4 5
7408 215/300 Write block 4868 len 36864 data: 2 2 2 2 2 2 2 3 3
7409 216/300 Flush
7410 217/300 Flush
7411 218/300 Read block 2480 len 12288
7412 219/300 Write block 4713 len 4096 data: 3
7413 220/300 Read block 2130 len 4096
7414 221/300 Write block 4500 len 28672 data: 4 4 3 4 4 5 4
7415 222/300 Write block 2106 len 40960 data: 3 3 2 2 2 2 2 2 2 3
7416 223/300 Read block 3300 len 4096
7417 224/300 Read block 4519 len 24576
7418 225/300 Write block 4881 len 28672 data: 3 3 2 3 3 3 3
7419 226/300 Read block 3583 len 16384
7420 227/300 Read block 4471 len 20480
7421 228/300 Read block 2859 len 32768
7422 229/300 Flush
7423 230/300 Write block 3741 len 32768 data: 3 3 3 3 3 3 2 2
7424 231/300 Write block 3648 len 16384 data: 3 2 2 2
7425 232/300 Write block 1106 len 36864 data: 2 3 2 2 2 2 2 2 2
7426 233/300 Write block 1492 len 32768 data: 2 2 2 2 2 2 2 3
7427 234/300 Read block 1395 len 28672
7428 235/300 Read block 3050 len 40960
7429 236/300 Read block 1117 len 24576
7430 237/300 Read block 2531 len 8192
7431 238/300 Read block 699 len 8192
7432 239/300 Write block 1534 len 12288 data: 4 4 4
7433 240/300 Write block 2063 len 40960 data: 2 2 2 2 3 3 3 3 3 4
7434 241/300 Read block 1264 len 40960
7435 242/300 Flush
7436 243/300 Read block 4682 len 40960
7437 244/300 Read block 3146 len 8192
7438 245/300 Write block 351 len 12288 data: 2 3 3
7439 246/300 Read block 515 len 12288
7440 247/300 Read block 3771 len 32768
7441 248/300 Write block 1283 len 4096 data: 3
7442 249/300 Flush
7443 250/300 Write block 2364 len 24576 data: 4 4 3 2 2 2
7444 251/300 Read block 2088 len 28672
7445 252/300 Write block 4904 len 24576 data: 3 3 3 3 3 5
7446 253/300 Read block 2638 len 16384
7447 254/300 Write block 3691 len 32768 data: 3 3 3 3 3 3 3 4
7448 255/300 Write block 3872 len 16384 data: 5 5 5 5
7449 256/300 Flush
7450 257/300 Read block 1077 len 20480
7451 258/300 Read block 103 len 16384
7452 259/300 Write block 3450 len 36864 data: 6 6 6 6 5 4 4 2 2
7453 260/300 Read block 2231 len 32768
7454 261/300 Read block 2495 len 4096
7455 262/300 Read block 3626 len 40960
7456 263/300 Read block 4888 len 28672
7457 264/300 Read block 4858 len 12288
7458 265/300 Read block 4515 len 40960
7459 266/300 Flush
7460 267/300 Flush
7461 268/300 Write block 3173 len 24576 data: 2 2 2 2 2 2
7462 269/300 Write block 4548 len 8192 data: 2 2
7463 270/300 Write block 4351 len 24576 data: 2 2 3 4 4 5
7464 271/300 Read block 1648 len 8192
7465 272/300 Read block 945 len 4096
7466 273/300 Write block 4066 len 32768 data: 4 4 4 4 5 4 5 5
7467 274/300 Write block 2252 len 24576 data: 4 4 4 5 5 4
7468 275/300 Read block 3541 len 24576
7469 276/300 Read block 3697 len 36864
7470 277/300 Write block 1447 len 28672 data: 3 4 4 4 4 4 4
7471 278/300 Read block 3003 len 12288
7472 279/300 Write block 3496 len 36864 data: 3 2 2 2 2 2 2 2 3
7473 280/300 Read block 2713 len 24576
7474 281/300 Read block 2219 len 4096
7475 282/300 Write block 253 len 32768 data: 2 2 3 4 4 4 5 5
7476 283/300 Read block 3682 len 32768
7477 284/300 Read block 1947 len 20480
7478 285/300 Read block 3380 len 12288
7479 286/300 Read block 2107 len 24576
7480 287/300 Write block 747 len 36864 data: 4 4 3 2 3 3 4 3 3
7481 288/300 Read block 3580 len 16384
7482 289/300 Read block 924 len 24576
7483 290/300 Write block 99 len 24576 data: 4 4 4 5 5 6
7484 291/300 Read block 3651 len 40960
7485 292/300 Read block 2926 len 24576
7486 293/300 Read block 1360 len 40960
7487 294/300 Write block 741 len 20480 data: 3 3 2 2 2
7488 295/300 Read block 3149 len 40960
7489 296/300 Write block 985 len 24576 data: 3 3 2 2 2 3
7490 297/300 Read block 3620 len 24576
7491 298/300 Read block 1174 len 12288
7492 299/300 Read block 4670 len 12288
7493 300/300 Write block 2281 len 12288 data: 2 2 2
7494 Main task has work to do, go find it
7495 got dsc Start(2)
7496 start 2
7497 Feb 12 01:36:00.612 INFO request completed, latency_us: 214, response_code: 204, uri: /start/cid/2, method: POST, req_id: f40e5a52-8ad7-46b6-a8bd-cf291b4c35e5, remote_addr: 127.0.0.1:60456, local_addr: 127.0.0.1:9998
7498 Replay: started 2, returned:Ok(())
7499 ----------------------------------------------------------------
7500 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
7501 States: Active Active Offline
7502 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
7503 2112 Acked 3111 Write 8 Done Done New true
7504 2113 Acked 3112 Read 8 Done Done New true
7505 2114 Acked 3113 Read 7 Done Done New false
7506 2115 Acked 3114 Read 4 Done Done New false
7507 2116 Acked 3115 Flush 0 Done Done New false
7508 2117 Acked 3116 Write 10 Done Done New false
7509 2118 Acked 3117 Read 4 Done Done New false
7510 2119 Acked 3118 Read 10 Done Done New false
7511 2120 Acked 3119 Read 2 Done Done New false
7512 2121 Acked 3120 Write 9 Done Done New false
7513 2122 Acked 3121 Write 5 Done Done New false
7514 2123 Acked 3122 Read 3 Done Done New false
7515 2124 Acked 3123 Read 8 Done Done New false
7516 2125 Acked 3124 Write 10 Done Done New false
7517 2126 Acked 3125 Read 2 Done Done New false
7518 2127 Acked 3126 Read 3 Done Done New false
7519 2128 Acked 3127 Read 8 Done Done New false
7520 2129 Acked 3128 Write 2 Done Done New false
7521 2130 Acked 3129 Write 4 Done Done New false
7522 2131 Acked 3130 Write 7 Done Done New false
7523 2132 Acked 3131 Flush 0 Done Done New false
7524 2133 Acked 3132 Read 7 Done Done New false
7525 2134 Acked 3133 Read 5 Done Done New false
7526 2135 Acked 3134 Read 7 Done Done New false
7527 2136 Acked 3135 Read 2 Done Done New false
7528 2137 Acked 3136 Write 10 Done Done New false
7529 2138 Acked 3137 Write 7 Done Done New false
7530 2139 Acked 3138 Write 9 Done Done New false
7531 2140 Acked 3139 Read 6 Done Done New false
7532 2141 Acked 3140 Read 2 Done Done New false
7533 2142 Acked 3141 Write 10 Done Done New false
7534 2143 Acked 3142 Write 10 Done Done New false
7535 2144 Acked 3143 Write 10 Done Done New false
7536 2145 Acked 3144 Write 7 Done Done New false
7537 2146 Acked 3145 Read 8 Done Done New false
7538 2147 Acked 3146 Read 10 Done Done New false
7539 2148 Acked 3147 Read 3 Done Done New false
7540 2149 Acked 3148 Write 7 Done Done New false
7541 2150 Acked 3149 Write 6 Done Done New false
7542 2151 Acked 3150 Read 2 Done Done New false
7543 2152 Acked 3151 Write 1 Done Done New false
7544 2153 Acked 3152 Read 8 Done Done New false
7545 2154 Acked 3153 Read 10 Done Done New false
7546 2155 Acked 3154 Write 8 Done Done New false
7547 2156 Acked 3155 Read 7 Done Done New false
7548 2157 Acked 3156 Read 4 Done Done New false
7549 2158 Acked 3157 Read 4 Done Done New false
7550 2159 Acked 3158 Write 5 Done Done New false
7551 2160 Acked 3159 Read 5 Done Done New false
7552 2161 Acked 3160 Read 2 Done Done New false
7553 2162 Acked 3161 Flush 0 Done Done New false
7554 2163 Acked 3162 Read 1 Done Done New false
7555 2164 Acked 3163 Write 3 Done Done New false
7556 2165 Acked 3164 Flush 0 Done Done New false
7557 2166 Acked 3165 Read 9 Done Done New false
7558 2167 Acked 3166 Write 4 Done Done New false
7559 2168 Acked 3167 Read 10 Done Done New false
7560 2169 Acked 3168 Read 5 Done Done New false
7561 2170 Acked 3169 Write 9 Done Done New false
7562 2171 Acked 3170 Read 8 Done Done New false
7563 2172 Acked 3171 Read 4 Done Done New false
7564 2173 Acked 3172 Write 1 Done Done New false
7565 2174 Acked 3173 Flush 0 Done Done New false
7566 2175 Acked 3174 Write 8 Done Done New false
7567 2176 Acked 3175 Read 2 Done Done New false
7568 2177 Acked 3176 Read 2 Done Done New false
7569 2178 Acked 3177 Read 2 Done Done New false
7570 2179 Acked 3178 Read 1 Done Done New false
7571 2180 Acked 3179 Write 10 Done Done New false
7572 2181 Acked 3180 Write 7 Done Done New false
7573 2182 Acked 3181 Read 8 Done Done New false
7574 2183 Acked 3182 Read 4 Done Done New false
7575 2184 Acked 3183 Read 9 Done Done New false
7576 2185 Acked 3184 Read 7 Done Done New false
7577 2186 Acked 3185 Write 7 Done Done New false
7578 2187 Acked 3186 Read 3 Done Done New false
7579 2188 Acked 3187 Read 5 Done Done New false
7580 2189 Acked 3188 Write 1 Done Done New false
7581 2190 Acked 3189 Flush 0 Done Done New false
7582 2191 Acked 3190 Flush 0 Done Done New false
7583 2192 Acked 3191 Write 7 Done Done New false
7584 2193 Acked 3192 Write 5 Done Done New false
7585 2194 Acked 3193 Read 9 Done Done New false
7586 2195 Acked 3194 Read 7 Done Done New false
7587 2196 Acked 3195 Read 2 Done Done New false
7588 2197 Acked 3196 Read 5 Done Done New false
7589 2198 Acked 3197 Write 2 Done Done New false
7590 2199 Acked 3198 Flush 0 Done Done New false
7591 2200 Acked 3199 Write 7 Done Done New false
7592 2201 Acked 3200 Read 7 Done Done New false
7593 2202 Acked 3201 Read 2 Done Done New false
7594 2203 Acked 3202 Flush 0 Done Done New false
7595 2204 Acked 3203 Read 4 Done Done New false
7596 2205 Acked 3204 Flush 0 Done Done New false
7597 2206 Acked 3205 Read 9 Done Done New false
7598 2207 Acked 3206 Write 7 Done Done New false
7599 2208 Acked 3207 Read 5 Done Done New false
7600 2209 Acked 3208 Read 9 Done Done New false
7601 2210 Acked 3209 Write 7 Done Done New false
7602 2211 Acked 3210 Read 2 Done Done New false
7603 2212 Acked 3211 Read 9 Done Done New false
7604 2213 Acked 3212 Read 5 Done Done New false
7605 2214 Acked 3213 Read 3 Done Done New false
7606 2215 Acked 3214 Read 5 Done Done New false
7607 2216 Acked 3215 Write 6 Done Done New false
7608 2217 Acked 3216 Write 7 Done Done New false
7609 2218 Acked 3217 Write 9 Done Done New false
7610 2219 Acked 3218 Read 9 Done Done New false
7611 2220 Acked 3219 Read 3 Done Done New false
7612 2221 Acked 3220 Read 6 Done Done New false
7613 2222 Acked 3221 Read 5 Done Done New false
7614 2223 Acked 3222 Write 4 Done Done New false
7615 2224 Acked 3223 Flush 0 Done Done New false
7616 2225 Acked 3224 Write 8 Done Done New false
7617 2226 Acked 3225 Write 3 Done Done New false
7618 2227 Acked 3226 Read 1 Done Done New false
7619 2228 Acked 3227 Read 4 Done Done New false
7620 2229 Acked 3228 Write 1 Done Done New false
7621 2230 Acked 3229 Write 9 Done Done New false
7622 2231 Acked 3230 Read 9 Done Done New false
7623 2232 Acked 3231 Read 9 Done Done New false
7624 2233 Acked 3232 Write 3 Done Done New false
7625 2234 Acked 3233 Read 4 Done Done New false
7626 2235 Acked 3234 Write 10 Done Done New false
7627 2236 Acked 3235 Read 3 Done Done New false
7628 2237 Acked 3236 Read 5 Done Done New false
7629 2238 Acked 3237 Write 3 Done Done New false
7630 2239 Acked 3238 Read 5 Done Done New false
7631 2240 Acked 3239 Flush 0 Done Done New false
7632 2241 Acked 3240 Write 7 Done Done New false
7633 2242 Acked 3241 Write 10 Done Done New false
7634 2243 Acked 3242 Read 10 Done Done New false
7635 2244 Acked 3243 Read 3 Done Done New false
7636 2245 Acked 3244 Write 5 Done Done New false
7637 2246 Acked 3245 Write 3 Done Done New false
7638 2247 Acked 3246 Write 8 Done Done New false
7639 2248 Acked 3247 Write 1 Done Done New false
7640 2249 Acked 3248 Write 10 Done Done New false
7641 2250 Acked 3249 Read 6 Done Done New false
7642 2251 Acked 3250 Write 8 Done Done New false
7643 2252 Acked 3251 Read 10 Done Done New false
7644 2253 Acked 3252 Read 10 Done Done New false
7645 2254 Acked 3253 Write 2 Done Done New false
7646 2255 Acked 3254 Read 1 Done Done New false
7647 2256 Acked 3255 Read 7 Done Done New false
7648 2257 Acked 3256 Write 2 Done Done New false
7649 2258 Acked 3257 Write 5 Done Done New false
7650 2259 Acked 3258 Read 2 Done Done New false
7651 2260 Acked 3259 Read 1 Done Done New false
7652 2261 Acked 3260 Read 3 Done Done New false
7653 2262 Acked 3261 Write 1 Done Done New false
7654 2263 Acked 3262 Write 5 Done Done New false
7655 2264 Acked 3263 Write 5 Done Done New false
7656 2265 Acked 3264 Read 2 Done Done New false
7657 2266 Acked 3265 Read 2 Done Done New false
7658 2267 Acked 3266 Read 7 Done Done New false
7659 2268 Acked 3267 Read 10 Done Done New false
7660 2269 Acked 3268 Read 2 Done Done New false
7661 2270 Acked 3269 Read 3 Done Done New false
7662 2271 Acked 3270 Write 2 Done Done New false
7663 2272 Acked 3271 Read 6 Done Done New false
7664 2273 Acked 3272 Write 9 Done Done New false
7665 2274 Acked 3273 Read 6 Done Done New false
7666 2275 Acked 3274 Write 7 Done Done New false
7667 2276 Acked 3275 Read 9 Done Done New false
7668 2277 Acked 3276 Write 9 Done Done New false
7669 2278 Acked 3277 Read 4 Done Done New false
7670 2279 Acked 3278 Read 5 Done Done New false
7671 2280 Acked 3279 Write 8 Done Done New false
7672 2281 Acked 3280 Read 3 Done Done New false
7673 2282 Acked 3281 Write 9 Done Done New false
7674 2283 Acked 3282 Write 2 Done Done New false
7675 2284 Acked 3283 Write 10 Done Done New false
7676 2285 Acked 3284 Flush 0 Done Done New false
7677 2286 Acked 3285 Write 8 Done Done New false
7678 2287 Acked 3286 Read 7 Done Done New false
7679 2288 Acked 3287 Write 9 Done Done New false
7680 2289 Acked 3288 Write 1 Done Done New false
7681 2290 Acked 3289 Read 3 Done Done New false
7682 2291 Acked 3290 Read 5 Done Done New false
7683 2292 Acked 3291 Write 7 Done Done New false
7684 2293 Acked 3292 Read 6 Done Done New false
7685 2294 Acked 3293 Read 1 Done Done New false
7686 2295 Acked 3294 Write 7 Done Done New false
7687 2296 Acked 3295 Read 9 Done Done New false
7688 2297 Acked 3296 Write 1 Done Done New false
7689 2298 Acked 3297 Write 10 Done Done New false
7690 2299 Acked 3298 Read 6 Done Done New false
7691 2300 Acked 3299 Read 2 Done Done New false
7692 2301 Acked 3300 Read 9 Done Done New false
7693 2302 Acked 3301 Write 1 Done Done New false
7694 2303 Acked 3302 Read 4 Done Done New false
7695 2304 Acked 3303 Read 4 Done Done New false
7696 2305 Acked 3304 Write 6 Done Done New false
7697 2306 Acked 3305 Write 4 Done Done New false
7698 2307 Acked 3306 Read 4 Done Done New false
7699 2308 Acked 3307 Read 10 Done Done New false
7700 2309 Acked 3308 Read 9 Done Done New false
7701 2310 Acked 3309 Write 6 Done Done New false
7702 2311 Acked 3310 Write 4 Done Done New false
7703 2312 Acked 3311 Write 2 Done Done New false
7704 2313 Acked 3312 Read 8 Done Done New false
7705 2314 Acked 3313 Flush 0 Done Done New false
7706 2315 Acked 3314 Write 10 Done Done New false
7707 2316 Acked 3315 Write 3 Done Done New false
7708 2317 Acked 3316 Read 8 Done Done New false
7709 2318 Acked 3317 Read 3 Done Done New false
7710 2319 Acked 3318 Read 5 Done Done New false
7711 2320 Acked 3319 Read 4 Done Done New false
7712 2321 Acked 3320 Write 1 Done Done New false
7713 2322 Acked 3321 Write 1 Done Done New false
7714 2323 Acked 3322 Read 1 Done Done New false
7715 2324 Acked 3323 Read 5 Done Done New false
7716 2325 Acked 3324 Read 3 Done Done New false
7717 2326 Acked 3325 Write 6 Done Done New false
7718 2327 Acked 3326 Write 9 Done Done New false
7719 2328 Acked 3327 Flush 0 Done Done New false
7720 2329 Acked 3328 Flush 0 Done Done New false
7721 2330 Acked 3329 Read 3 Done Done New false
7722 2331 Acked 3330 Write 1 Done Done New false
7723 2332 Acked 3331 Read 1 Done Done New false
7724 2333 Acked 3332 Write 7 Done Done New false
7725 2334 Acked 3333 Write 10 Done Done New false
7726 2335 Acked 3334 Read 1 Done Done New false
7727 2336 Acked 3335 Read 6 Done Done New false
7728 2337 Acked 3336 Write 7 Done Done New false
7729 2338 Acked 3337 Read 4 Done Done New false
7730 2339 Acked 3338 Read 5 Done Done New false
7731 2340 Acked 3339 Read 8 Done Done New false
7732 2341 Acked 3340 Flush 0 Done Done New false
7733 2342 Acked 3341 Write 8 Done Done New false
7734 2343 Acked 3342 Write 4 Done Done New false
7735 2344 Acked 3343 Write 9 Done Done New false
7736 2345 Acked 3344 Write 8 Done Done New false
7737 2346 Acked 3345 Read 7 Done Done New false
7738 2347 Acked 3346 Read 10 Done Done New false
7739 2348 Acked 3347 Read 6 Done Done New false
7740 2349 Acked 3348 Read 2 Done Done New false
7741 2350 Acked 3349 Read 2 Done Done New false
7742 2351 Acked 3350 Write 3 Done Done New false
7743 2352 Acked 3351 Write 10 Done Done New false
7744 2353 Acked 3352 Read 10 Done Done New false
7745 2354 Acked 3353 Flush 0 Done Done New false
7746 2355 Acked 3354 Read 10 Done Done New false
7747 2356 Acked 3355 Read 2 Done Done New false
7748 2357 Acked 3356 Write 3 Done Done New false
7749 2358 Acked 3357 Read 3 Done Done New false
7750 2359 Acked 3358 Read 8 Done Done New false
7751 2360 Acked 3359 Write 1 Done Done New false
7752 2361 Acked 3360 Flush 0 Done Done New false
7753 2362 Acked 3361 Write 6 Done Done New false
7754 2363 Acked 3362 Read 7 Done Done New false
7755 2364 Acked 3363 Write 6 Done Done New false
7756 2365 Acked 3364 Read 4 Done Done New false
7757 2366 Acked 3365 Write 8 Done Done New false
7758 2367 Acked 3366 Write 4 Done Done New false
7759 2368 Acked 3367 Flush 0 Done Done New false
7760 2369 Acked 3368 Read 5 Done Done New false
7761 2370 Acked 3369 Read 4 Done Done New false
7762 2371 Acked 3370 Write 9 Done Done New false
7763 2372 Acked 3371 Read 8 Done Done New false
7764 2373 Acked 3372 Read 1 Done Done New false
7765 2374 Acked 3373 Read 10 Done Done New false
7766 2375 Acked 3374 Read 7 Done Done New false
7767 2376 Acked 3375 Read 3 Done Done New false
7768 2377 Acked 3376 Read 10 Done Done New false
7769 2378 Acked 3377 Flush 0 Done Done New false
7770 2379 Acked 3378 Flush 0 Done Done New false
7771 2380 Acked 3379 Write 6 Done Done New false
7772 2381 Acked 3380 Write 2 Done Done New false
7773 2382 Acked 3381 Write 6 Done Done New false
7774 2383 Acked 3382 Read 2 Done Done New false
7775 2384 Acked 3383 Read 1 Done Done New false
7776 2385 Acked 3384 Write 8 Done Done New false
7777 2386 Acked 3385 Write 6 Done Done New false
7778 2387 Acked 3386 Read 6 Done Done New false
7779 2388 Acked 3387 Read 9 Done Done New false
7780 2389 Acked 3388 Write 7 Done Done New false
7781 2390 Acked 3389 Read 3 Done Done New false
7782 2391 Acked 3390 Write 9 Done Done New false
7783 2392 Acked 3391 Read 6 Done Done New false
7784 2393 Acked 3392 Read 1 Done Done New false
7785 2394 Acked 3393 Write 8 Done Done New false
7786 2395 Acked 3394 Read 8 Done Done New false
7787 2396 Acked 3395 Read 5 Done Done New false
7788 2397 Acked 3396 Read 3 Done Done New false
7789 2398 Acked 3397 Read 6 Done Done New false
7790 2399 Acked 3398 Write 9 Done Done New false
7791 2400 Acked 3399 Read 4 Done Done New false
7792 2401 Acked 3400 Read 6 Done Done New false
7793 2402 Acked 3401 Write 6 Done Done New false
7794 2403 Acked 3402 Read 10 Done Done New false
7795 2404 Acked 3403 Read 6 Done Done New false
7796 2405 Acked 3404 Read 10 Done Done New false
7797 2406 Acked 3405 Write 5 Done Done New false
7798 2407 Acked 3406 Read 10 Done Done New false
7799 2408 Acked 3407 Write 6 Done Done New false
7800 2409 Acked 3408 Read 6 Done Done New false
7801 2410 Acked 3409 Read 3 Done Done New false
7802 2411 Acked 3410 Read 3 Done Done New false
7803 2412 Acked 3411 Write 3 Done Done New false
7804 STATES DS:0 DS:1 DS:2 TOTAL
7805 New 0 0 301 301
7806 Sent 0 0 0 0
7807 Done 301 301 0 602
7808 Skipped 0 0 0 0
7809 Error 0 0 0 0
7810 Last Flush: 3378 3378 3110
7811 Downstairs last five completed: 3110 3109 3108 3107 3106
7812 Upstairs last five completed: 2412 2411 2410 2409 2408
7813 CLIENT: Up:0 ds:301 act:2
78142024-02-12T01:36:01.264ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
78152024-02-12T01:36:01.264ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
78162024-02-12T01:36:01.264ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
78172024-02-12T01:36:01.264ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
78182024-02-12T01:36:01.264ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
78192024-02-12T01:36:01.264ZINFOcrucible: [2] client re-new 302 jobs since flush 3110 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
78202024-02-12T01:36:01.264ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
78212024-02-12T01:36:02.265ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
78222024-02-12T01:36:02.265ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
78232024-02-12T01:36:02.265ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
78242024-02-12T01:36:02.265ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
78252024-02-12T01:36:02.265ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
78262024-02-12T01:36:02.265ZINFOcrucible: [2] client re-new 302 jobs since flush 3110 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
78272024-02-12T01:36:02.265ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
78282024-02-12T01:36:03.266ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
78292024-02-12T01:36:03.266ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
78302024-02-12T01:36:03.266ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
78312024-02-12T01:36:03.266ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
78322024-02-12T01:36:03.266ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
78332024-02-12T01:36:03.266ZINFOcrucible: [2] client re-new 302 jobs since flush 3110 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
78342024-02-12T01:36:03.267ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
7835 [8830] got start action
7836 [8830] I am going to restart
7837 Starting downstairs at port 8830
7838 Make output file at "/tmp/dsc/downstairs-8830.txt"
7839 [8830][2] reports Starting
78402024-02-12T01:36:04.268ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
78412024-02-12T01:36:04.268ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
78422024-02-12T01:36:04.268ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
78432024-02-12T01:36:04.268ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
78442024-02-12T01:36:04.268ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
78452024-02-12T01:36:04.268ZINFOcrucible: [2] client re-new 302 jobs since flush 3110 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
78462024-02-12T01:36:04.268ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
7847 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1172)
7848 [8830][2] reports Running
7849 ----------------------------------------------------------------
7850 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
7851 States: Active Active Offline
7852 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
7853 2112 Acked 3111 Write 8 Done Done New true
7854 2113 Acked 3112 Read 8 Done Done New true
7855 2114 Acked 3113 Read 7 Done Done New true
7856 2115 Acked 3114 Read 4 Done Done New true
7857 2116 Acked 3115 Flush 0 Done Done New true
7858 2117 Acked 3116 Write 10 Done Done New true
7859 2118 Acked 3117 Read 4 Done Done New true
7860 2119 Acked 3118 Read 10 Done Done New true
7861 2120 Acked 3119 Read 2 Done Done New true
7862 2121 Acked 3120 Write 9 Done Done New true
7863 2122 Acked 3121 Write 5 Done Done New true
7864 2123 Acked 3122 Read 3 Done Done New true
7865 2124 Acked 3123 Read 8 Done Done New true
7866 2125 Acked 3124 Write 10 Done Done New true
7867 2126 Acked 3125 Read 2 Done Done New true
7868 2127 Acked 3126 Read 3 Done Done New true
7869 2128 Acked 3127 Read 8 Done Done New true
7870 2129 Acked 3128 Write 2 Done Done New true
7871 2130 Acked 3129 Write 4 Done Done New true
7872 2131 Acked 3130 Write 7 Done Done New true
7873 2132 Acked 3131 Flush 0 Done Done New true
7874 2133 Acked 3132 Read 7 Done Done New true
7875 2134 Acked 3133 Read 5 Done Done New true
7876 2135 Acked 3134 Read 7 Done Done New true
7877 2136 Acked 3135 Read 2 Done Done New true
7878 2137 Acked 3136 Write 10 Done Done New true
7879 2138 Acked 3137 Write 7 Done Done New true
7880 2139 Acked 3138 Write 9 Done Done New true
7881 2140 Acked 3139 Read 6 Done Done New true
7882 2141 Acked 3140 Read 2 Done Done New true
7883 2142 Acked 3141 Write 10 Done Done New true
7884 2143 Acked 3142 Write 10 Done Done New true
7885 2144 Acked 3143 Write 10 Done Done New true
7886 2145 Acked 3144 Write 7 Done Done New true
7887 2146 Acked 3145 Read 8 Done Done New true
7888 2147 Acked 3146 Read 10 Done Done New true
7889 2148 Acked 3147 Read 3 Done Done New true
7890 2149 Acked 3148 Write 7 Done Done New true
7891 2150 Acked 3149 Write 6 Done Done New true
7892 2151 Acked 3150 Read 2 Done Done New true
7893 2152 Acked 3151 Write 1 Done Done New true
7894 2153 Acked 3152 Read 8 Done Done New true
7895 2154 Acked 3153 Read 10 Done Done New true
7896 2155 Acked 3154 Write 8 Done Done New true
7897 2156 Acked 3155 Read 7 Done Done New true
7898 2157 Acked 3156 Read 4 Done Done New true
7899 2158 Acked 3157 Read 4 Done Done New true
7900 2159 Acked 3158 Write 5 Done Done New true
7901 2160 Acked 3159 Read 5 Done Done New true
7902 2161 Acked 3160 Read 2 Done Done New true
7903 2162 Acked 3161 Flush 0 Done Done New true
7904 2163 Acked 3162 Read 1 Done Done New true
7905 2164 Acked 3163 Write 3 Done Done New true
7906 2165 Acked 3164 Flush 0 Done Done New true
7907 2166 Acked 3165 Read 9 Done Done New true
7908 2167 Acked 3166 Write 4 Done Done New true
7909 2168 Acked 3167 Read 10 Done Done New true
7910 2169 Acked 3168 Read 5 Done Done New true
7911 2170 Acked 3169 Write 9 Done Done New true
7912 2171 Acked 3170 Read 8 Done Done New true
7913 2172 Acked 3171 Read 4 Done Done New true
7914 2173 Acked 3172 Write 1 Done Done New true
7915 2174 Acked 3173 Flush 0 Done Done New true
7916 2175 Acked 3174 Write 8 Done Done New true
7917 2176 Acked 3175 Read 2 Done Done New true
7918 2177 Acked 3176 Read 2 Done Done New true
7919 2178 Acked 3177 Read 2 Done Done New true
7920 2179 Acked 3178 Read 1 Done Done New true
7921 2180 Acked 3179 Write 10 Done Done New true
7922 2181 Acked 3180 Write 7 Done Done New true
7923 2182 Acked 3181 Read 8 Done Done New true
7924 2183 Acked 3182 Read 4 Done Done New true
7925 2184 Acked 3183 Read 9 Done Done New true
7926 2185 Acked 3184 Read 7 Done Done New true
7927 2186 Acked 3185 Write 7 Done Done New true
7928 2187 Acked 3186 Read 3 Done Done New true
7929 2188 Acked 3187 Read 5 Done Done New true
7930 2189 Acked 3188 Write 1 Done Done New true
7931 2190 Acked 3189 Flush 0 Done Done New true
7932 2191 Acked 3190 Flush 0 Done Done New true
7933 2192 Acked 3191 Write 7 Done Done New true
7934 2193 Acked 3192 Write 5 Done Done New true
7935 2194 Acked 3193 Read 9 Done Done New true
7936 2195 Acked 3194 Read 7 Done Done New true
7937 2196 Acked 3195 Read 2 Done Done New true
7938 2197 Acked 3196 Read 5 Done Done New true
7939 2198 Acked 3197 Write 2 Done Done New true
7940 2199 Acked 3198 Flush 0 Done Done New true
7941 2200 Acked 3199 Write 7 Done Done New true
7942 2201 Acked 3200 Read 7 Done Done New true
7943 2202 Acked 3201 Read 2 Done Done New true
7944 2203 Acked 3202 Flush 0 Done Done New true
7945 2204 Acked 3203 Read 4 Done Done New true
7946 2205 Acked 3204 Flush 0 Done Done New true
7947 2206 Acked 3205 Read 9 Done Done New true
7948 2207 Acked 3206 Write 7 Done Done New true
7949 2208 Acked 3207 Read 5 Done Done New true
7950 2209 Acked 3208 Read 9 Done Done New true
7951 2210 Acked 3209 Write 7 Done Done New true
7952 2211 Acked 3210 Read 2 Done Done New true
7953 2212 Acked 3211 Read 9 Done Done New true
7954 2213 Acked 3212 Read 5 Done Done New true
7955 2214 Acked 3213 Read 3 Done Done New true
7956 2215 Acked 3214 Read 5 Done Done New true
7957 2216 Acked 3215 Write 6 Done Done New true
7958 2217 Acked 3216 Write 7 Done Done New true
7959 2218 Acked 3217 Write 9 Done Done New true
7960 2219 Acked 3218 Read 9 Done Done New true
7961 2220 Acked 3219 Read 3 Done Done New true
7962 2221 Acked 3220 Read 6 Done Done New true
7963 2222 Acked 3221 Read 5 Done Done New true
7964 2223 Acked 3222 Write 4 Done Done New true
7965 2224 Acked 3223 Flush 0 Done Done New true
7966 2225 Acked 3224 Write 8 Done Done New true
7967 2226 Acked 3225 Write 3 Done Done New true
7968 2227 Acked 3226 Read 1 Done Done New true
7969 2228 Acked 3227 Read 4 Done Done New true
7970 2229 Acked 3228 Write 1 Done Done New true
7971 2230 Acked 3229 Write 9 Done Done New true
7972 2231 Acked 3230 Read 9 Done Done New true
7973 2232 Acked 3231 Read 9 Done Done New true
7974 2233 Acked 3232 Write 3 Done Done New true
7975 2234 Acked 3233 Read 4 Done Done New true
7976 2235 Acked 3234 Write 10 Done Done New true
7977 2236 Acked 3235 Read 3 Done Done New true
7978 2237 Acked 3236 Read 5 Done Done New true
7979 2238 Acked 3237 Write 3 Done Done New true
7980 2239 Acked 3238 Read 5 Done Done New true
7981 2240 Acked 3239 Flush 0 Done Done New true
7982 2241 Acked 3240 Write 7 Done Done New true
7983 2242 Acked 3241 Write 10 Done Done New true
7984 2243 Acked 3242 Read 10 Done Done New true
7985 2244 Acked 3243 Read 3 Done Done New true
7986 2245 Acked 3244 Write 5 Done Done New true
7987 2246 Acked 3245 Write 3 Done Done New true
7988 2247 Acked 3246 Write 8 Done Done New true
7989 2248 Acked 3247 Write 1 Done Done New true
7990 2249 Acked 3248 Write 10 Done Done New true
7991 2250 Acked 3249 Read 6 Done Done New true
7992 2251 Acked 3250 Write 8 Done Done New true
7993 2252 Acked 3251 Read 10 Done Done New true
7994 2253 Acked 3252 Read 10 Done Done New true
7995 2254 Acked 3253 Write 2 Done Done New true
7996 2255 Acked 3254 Read 1 Done Done New true
7997 2256 Acked 3255 Read 7 Done Done New true
7998 2257 Acked 3256 Write 2 Done Done New true
7999 2258 Acked 3257 Write 5 Done Done New true
8000 2259 Acked 3258 Read 2 Done Done New true
8001 2260 Acked 3259 Read 1 Done Done New true
8002 2261 Acked 3260 Read 3 Done Done New true
8003 2262 Acked 3261 Write 1 Done Done New true
8004 2263 Acked 3262 Write 5 Done Done New true
8005 2264 Acked 3263 Write 5 Done Done New true
8006 2265 Acked 3264 Read 2 Done Done New true
8007 2266 Acked 3265 Read 2 Done Done New true
8008 2267 Acked 3266 Read 7 Done Done New true
8009 2268 Acked 3267 Read 10 Done Done New true
8010 2269 Acked 3268 Read 2 Done Done New true
8011 2270 Acked 3269 Read 3 Done Done New true
8012 2271 Acked 3270 Write 2 Done Done New true
8013 2272 Acked 3271 Read 6 Done Done New true
8014 2273 Acked 3272 Write 9 Done Done New true
8015 2274 Acked 3273 Read 6 Done Done New true
8016 2275 Acked 3274 Write 7 Done Done New true
8017 2276 Acked 3275 Read 9 Done Done New true
8018 2277 Acked 3276 Write 9 Done Done New true
8019 2278 Acked 3277 Read 4 Done Done New true
8020 2279 Acked 3278 Read 5 Done Done New true
8021 2280 Acked 3279 Write 8 Done Done New true
8022 2281 Acked 3280 Read 3 Done Done New true
8023 2282 Acked 3281 Write 9 Done Done New true
8024 2283 Acked 3282 Write 2 Done Done New true
8025 2284 Acked 3283 Write 10 Done Done New true
8026 2285 Acked 3284 Flush 0 Done Done New true
8027 2286 Acked 3285 Write 8 Done Done New true
8028 2287 Acked 3286 Read 7 Done Done New true
8029 2288 Acked 3287 Write 9 Done Done New true
8030 2289 Acked 3288 Write 1 Done Done New true
8031 2290 Acked 3289 Read 3 Done Done New true
8032 2291 Acked 3290 Read 5 Done Done New true
8033 2292 Acked 3291 Write 7 Done Done New true
8034 2293 Acked 3292 Read 6 Done Done New true
8035 2294 Acked 3293 Read 1 Done Done New true
8036 2295 Acked 3294 Write 7 Done Done New true
8037 2296 Acked 3295 Read 9 Done Done New true
8038 2297 Acked 3296 Write 1 Done Done New true
8039 2298 Acked 3297 Write 10 Done Done New true
8040 2299 Acked 3298 Read 6 Done Done New true
8041 2300 Acked 3299 Read 2 Done Done New true
8042 2301 Acked 3300 Read 9 Done Done New true
8043 2302 Acked 3301 Write 1 Done Done New true
8044 2303 Acked 3302 Read 4 Done Done New true
8045 2304 Acked 3303 Read 4 Done Done New true
8046 2305 Acked 3304 Write 6 Done Done New true
8047 2306 Acked 3305 Write 4 Done Done New true
8048 2307 Acked 3306 Read 4 Done Done New true
8049 2308 Acked 3307 Read 10 Done Done New true
8050 2309 Acked 3308 Read 9 Done Done New true
8051 2310 Acked 3309 Write 6 Done Done New true
8052 2311 Acked 3310 Write 4 Done Done New true
8053 2312 Acked 3311 Write 2 Done Done New true
8054 2313 Acked 3312 Read 8 Done Done New true
8055 2314 Acked 3313 Flush 0 Done Done New true
8056 2315 Acked 3314 Write 10 Done Done New true
8057 2316 Acked 3315 Write 3 Done Done New true
8058 2317 Acked 3316 Read 8 Done Done New true
8059 2318 Acked 3317 Read 3 Done Done New true
8060 2319 Acked 3318 Read 5 Done Done New true
8061 2320 Acked 3319 Read 4 Done Done New true
8062 2321 Acked 3320 Write 1 Done Done New true
8063 2322 Acked 3321 Write 1 Done Done New true
8064 2323 Acked 3322 Read 1 Done Done New true
8065 2324 Acked 3323 Read 5 Done Done New true
8066 2325 Acked 3324 Read 3 Done Done New true
8067 2326 Acked 3325 Write 6 Done Done New true
8068 2327 Acked 3326 Write 9 Done Done New true
8069 2328 Acked 3327 Flush 0 Done Done New true
8070 2329 Acked 3328 Flush 0 Done Done New true
8071 2330 Acked 3329 Read 3 Done Done New true
8072 2331 Acked 3330 Write 1 Done Done New true
8073 2332 Acked 3331 Read 1 Done Done New true
8074 2333 Acked 3332 Write 7 Done Done New true
8075 2334 Acked 3333 Write 10 Done Done New true
8076 2335 Acked 3334 Read 1 Done Done New true
8077 2336 Acked 3335 Read 6 Done Done New true
8078 2337 Acked 3336 Write 7 Done Done New true
8079 2338 Acked 3337 Read 4 Done Done New true
8080 2339 Acked 3338 Read 5 Done Done New true
8081 2340 Acked 3339 Read 8 Done Done New true
8082 2341 Acked 3340 Flush 0 Done Done New true
8083 2342 Acked 3341 Write 8 Done Done New true
8084 2343 Acked 3342 Write 4 Done Done New true
8085 2344 Acked 3343 Write 9 Done Done New true
8086 2345 Acked 3344 Write 8 Done Done New true
8087 2346 Acked 3345 Read 7 Done Done New true
8088 2347 Acked 3346 Read 10 Done Done New true
8089 2348 Acked 3347 Read 6 Done Done New true
8090 2349 Acked 3348 Read 2 Done Done New true
8091 2350 Acked 3349 Read 2 Done Done New true
8092 2351 Acked 3350 Write 3 Done Done New true
8093 2352 Acked 3351 Write 10 Done Done New true
8094 2353 Acked 3352 Read 10 Done Done New true
8095 2354 Acked 3353 Flush 0 Done Done New true
8096 2355 Acked 3354 Read 10 Done Done New true
8097 2356 Acked 3355 Read 2 Done Done New true
8098 2357 Acked 3356 Write 3 Done Done New true
8099 2358 Acked 3357 Read 3 Done Done New true
8100 2359 Acked 3358 Read 8 Done Done New true
8101 2360 Acked 3359 Write 1 Done Done New true
8102 2361 Acked 3360 Flush 0 Done Done New true
8103 2362 Acked 3361 Write 6 Done Done New true
8104 2363 Acked 3362 Read 7 Done Done New true
8105 2364 Acked 3363 Write 6 Done Done New true
8106 2365 Acked 3364 Read 4 Done Done New true
8107 2366 Acked 3365 Write 8 Done Done New true
8108 2367 Acked 3366 Write 4 Done Done New true
8109 2368 Acked 3367 Flush 0 Done Done New true
8110 2369 Acked 3368 Read 5 Done Done New true
8111 2370 Acked 3369 Read 4 Done Done New true
8112 2371 Acked 3370 Write 9 Done Done New true
8113 2372 Acked 3371 Read 8 Done Done New true
8114 2373 Acked 3372 Read 1 Done Done New true
8115 2374 Acked 3373 Read 10 Done Done New true
8116 2375 Acked 3374 Read 7 Done Done New true
8117 2376 Acked 3375 Read 3 Done Done New true
8118 2377 Acked 3376 Read 10 Done Done New true
8119 2378 Acked 3377 Flush 0 Done Done New true
8120 2379 Acked 3378 Flush 0 Done Done New true
8121 2380 Acked 3379 Write 6 Done Done New true
8122 2381 Acked 3380 Write 2 Done Done New true
8123 2382 Acked 3381 Write 6 Done Done New true
8124 2383 Acked 3382 Read 2 Done Done New true
8125 2384 Acked 3383 Read 1 Done Done New true
8126 2385 Acked 3384 Write 8 Done Done New true
8127 2386 Acked 3385 Write 6 Done Done New true
8128 2387 Acked 3386 Read 6 Done Done New true
8129 2388 Acked 3387 Read 9 Done Done New true
8130 2389 Acked 3388 Write 7 Done Done New true
8131 2390 Acked 3389 Read 3 Done Done New true
8132 2391 Acked 3390 Write 9 Done Done New true
8133 2392 Acked 3391 Read 6 Done Done New true
8134 2393 Acked 3392 Read 1 Done Done New true
8135 2394 Acked 3393 Write 8 Done Done New true
8136 2395 Acked 3394 Read 8 Done Done New true
8137 2396 Acked 3395 Read 5 Done Done New true
8138 2397 Acked 3396 Read 3 Done Done New true
8139 2398 Acked 3397 Read 6 Done Done New true
8140 2399 Acked 3398 Write 9 Done Done New true
8141 2400 Acked 3399 Read 4 Done Done New true
8142 2401 Acked 3400 Read 6 Done Done New true
8143 2402 Acked 3401 Write 6 Done Done New true
8144 2403 Acked 3402 Read 10 Done Done New true
8145 2404 Acked 3403 Read 6 Done Done New true
8146 2405 Acked 3404 Read 10 Done Done New true
8147 2406 Acked 3405 Write 5 Done Done New true
8148 2407 Acked 3406 Read 10 Done Done New true
8149 2408 Acked 3407 Write 6 Done Done New true
8150 2409 Acked 3408 Read 6 Done Done New true
8151 2410 Acked 3409 Read 3 Done Done New true
8152 2411 Acked 3410 Read 3 Done Done New true
8153 2412 Acked 3411 Write 3 Done Done New true
8154 2413 Acked 3412 Flush 0 Done Done New true
8155 STATES DS:0 DS:1 DS:2 TOTAL
8156 New 0 0 302 302
8157 Sent 0 0 0 0
8158 Done 302 302 0 604
8159 Skipped 0 0 0 0
8160 Error 0 0 0 0
8161 Last Flush: 3412 3412 3110
8162 Downstairs last five completed: 3110 3109 3108 3107 3106
8163 Upstairs last five completed: 2413 2412 2411 2410 2409
8164 CLIENT: Up:0 ds:302 act:2
81652024-02-12T01:36:05.269ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
81662024-02-12T01:36:05.269ZINFOcrucible: ds_connection connected = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
81672024-02-12T01:36:05.271ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
81682024-02-12T01:36:05.271ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
81692024-02-12T01:36:05.271ZINFOcrucible: send last flush ID to this DS: 3110 = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
81702024-02-12T01:36:05.271ZINFOcrucible: Replied this last flush ID: 3110 = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
81712024-02-12T01:36:05.271ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
81722024-02-12T01:36:05.271ZINFOcrucible: [2] Transition from Offline to Replay = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
81732024-02-12T01:36:05.271ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
81742024-02-12T01:36:05.271ZINFOcrucible: [2] Transition from Replay to Active = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
8175 ----------------------------------------------------------------
8176 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
8177 Downstairs last five completed: 3412 3411 3410 3409 3408
8178 Upstairs last five completed: 2413 2412 2411 2410 2409
8179 CLIENT: Up:0 ds:0 act:3
8180 Replay: All jobs finished, all DS active.
8181 Main task has work to do, go find it
8182 got dsc Stop(1)
8183 stop 1
8184 Feb 12 01:36:08.617 INFO request completed, latency_us: 353, response_code: 204, uri: /stop/cid/1, method: POST[8820] Got stop action so:false kr:false
8185 , req_id: b376c859-9fbe-48a0-825e-7716868f3932, remote_addr: 127.0.0.1:60456, local_addr: 127.0.0.1:9998
8186 Feb 12 01:36:08.618 INFO request completed, latency_us: {"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2024-02-12T01:36:08.618399167Z","hostname":"ip-10-150-1-104.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"e1773b3e-b36f-4080-8b13-c0830f0d072e"}
8187 148, response_code: 200, uri: /state/cid/1, method{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2024-02-12T01:36:08.618463218Z","hostname":"ip-10-150-1-104.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"e1773b3e-b36f-4080-8b13-c0830f0d072e"}
8188 : GET, req_id: ec37764f-09c3-44bf-a225-43cc09ed0835, {"msg":"downstairs task for 1 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-02-12T01:36:08.618527439Z","hostname":"ip-10-150-1-104.us-west-2.compute.internal","pid":1164,"session_id":"e1773b3e-b36f-4080-8b13-c0830f0d072e"}
8189 remote_addr: 127.0.0.1:60456{"msg":"Gone missing, transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-02-12T01:36:08.618557835Z","hostname":"ip-10-150-1-104.us-west-2.compute.internal","pid":1164,"client":"1","":"downstairs","session_id":"e1773b3e-b36f-4080-8b13-c0830f0d072e"}
8190 , local_addr: 127.0.0.{"msg":"[1] client re-new 0 jobs since flush 3412","v":0,"name":"crucible","level":30,"time":"2024-02-12T01:36:08.618578751Z","hostname":"ip-10-150-1-104.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"e1773b3e-b36f-4080-8b13-c0830f0d072e"}
8191 1:9998
81922024-02-12T01:36:08.618ZINFOcrucible: client task is exiting = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
8193 [8820] Exited with: ExitStatus(unix_wait_status(9))
8194 [8820][1] reports Exit
81952024-02-12T01:36:09.620ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
81962024-02-12T01:36:09.620ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
81972024-02-12T01:36:09.620ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
81982024-02-12T01:36:09.620ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
81992024-02-12T01:36:09.620ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
82002024-02-12T01:36:09.620ZINFOcrucible: [1] client re-new 0 jobs since flush 3412 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
82012024-02-12T01:36:09.620ZINFOcrucible: client task is exiting = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
82022024-02-12T01:36:10.622ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
82032024-02-12T01:36:10.622ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
82042024-02-12T01:36:10.622ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
82052024-02-12T01:36:10.622ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
82062024-02-12T01:36:10.622ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
82072024-02-12T01:36:10.622ZINFOcrucible: [1] client re-new 0 jobs since flush 3412 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
82082024-02-12T01:36:10.622ZINFOcrucible: client task is exiting = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
82092024-02-12T01:36:11.624ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
82102024-02-12T01:36:11.624ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
82112024-02-12T01:36:11.624ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
82122024-02-12T01:36:11.624ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
82132024-02-12T01:36:11.624ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
82142024-02-12T01:36:11.624ZINFOcrucible: [1] client re-new 0 jobs since flush 3412 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
82152024-02-12T01:36:11.624ZINFOcrucible: client task is exiting = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
8216 Feb 12 01:36:12.620 INFO request completed, latency_us: 229, response_code: 200, uri: /state/cid/1, method: GET, req_id: 8103c119-aee3-4bfd-9a2f-69dbede1babf, remote_addr: 127.0.0.1:60456, local_addr: 127.0.0.1:9998
8217 001/300 Write block 126 len 16384 data: 3 3 2 2
8218 002/300 Write block 3996 len 16384 data: 3 3 5 5
8219 003/300 Read block 4476 len 40960
8220 004/300 Write block 419 len 16384 data: 3 3 3 2
8221 005/300 Write block 539 len 16384 data: 3 3 3 2
8222 006/300 Read block 2820 len 8192
82232024-02-12T01:36:12.625ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
82242024-02-12T01:36:12.625ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
82252024-02-12T01:36:12.625ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
82262024-02-12T01:36:12.625ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
82272024-02-12T01:36:12.625ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
82282024-02-12T01:36:12.625ZINFOcrucible: [1] client re-new 6 jobs since flush 3412 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
82292024-02-12T01:36:12.625ZINFOcrucible: client task is exiting = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
8230 007/300 Flush
8231 008/300 Flush
8232 009/300 Write block 536 len 20480 data: 3 3 4 4 4
8233 010/300 Read block 2804 len 12288
8234 011/300 Flush
8235 012/300 Write block 1329 len 32768 data: 4 4 3 3 4 5 5 3
8236 013/300 Write block 4299 len 32768 data: 2 2 2 2 2 2 2 2
8237 014/300 Write block 1302 len 4096 data: 2
8238 015/300 Read block 1584 len 16384
8239 016/300 Write block 791 len 4096 data: 3
8240 017/300 Write block 1142 len 36864 data: 4 4 4 4 4 4 4 3 3
8241 018/300 Write block 2376 len 12288 data: 3 3 3
8242 019/300 Read block 4442 len 12288
8243 020/300 Write block 3030 len 4096 data: 3
8244 021/300 Read block 3288 len 32768
8245 022/300 Read block 4528 len 36864
8246 023/300 Write block 935 len 12288 data: 3 3 4
8247 024/300 Write block 1635 len 24576 data: 2 2 2 2 2 2
8248 025/300 Flush
8249 026/300 Flush
8250 027/300 Read block 1044 len 28672
8251 028/300 Write block 3960 len 32768 data: 3 4 4 4 4 4 4 3
8252 029/300 Write block 3514 len 8192 data: 2 2
8253 030/300 Write block 3367 len 12288 data: 2 2 2
8254 031/300 Flush
8255 032/300 Read block 369 len 16384
8256 033/300 Write block 4409 len 16384 data: 3 3 3 2
8257 034/300 Read block 946 len 20480
8258 035/300 Read block 4810 len 4096
8259 036/300 Write block 138 len 20480 data: 2 2 2 2 2
8260 037/300 Flush
8261 038/300 Read block 3185 len 40960
8262 039/300 Write block 880 len 28672 data: 5 6 5 5 5 3 3
8263 040/300 Write block 1407 len 40960 data: 3 3 2 2 2 2 2 2 3 2
8264 041/300 Flush
8265 042/300 Read block 4287 len 4096
8266 043/300 Read block 1010 len 16384
8267 044/300 Write block 2156 len 20480 data: 4 3 4 4 3
8268 045/300 Read block 180 len 8192
8269 046/300 Write block 4315 len 28672 data: 2 3 3 3 4 3 3
8270 047/300 Write block 3179 len 12288 data: 2 3 4
8271 048/300 Read block 2974 len 36864
8272 049/300 Write block 916 len 40960 data: 2 2 2 2 2 2 2 3 3 4
8273 050/300 Write block 270 len 12288 data: 3 3 2
8274 051/300 Read block 56 len 4096
8275 052/300 Read block 1779 len 4096
8276 053/300 Write block 3985 len 16384 data: 3 3 3 2
8277 054/300 Write block 974 len 20480 data: 2 2 2 2 2
8278 055/300 Write block 3420 len 24576 data: 3 3 3 3 4 4
8279 056/300 Read block 2739 len 16384
8280 057/300 Flush
8281 058/300 Write block 3120 len 40960 data: 2 2 2 3 3 4 4 4 4 4
8282 059/300 Write block 2060 len 36864 data: 2 2 2 3 3 3 3 4 4
8283 060/300 Write block 792 len 32768 data: 3 3 3 4 4 4 4 5
8284 061/300 Write block 4545 len 32768 data: 2 2 2 3 3 2 2 2
8285 062/300 Write block 1316 len 28672 data: 3 3 3 3 3 3 3
8286 063/300 Flush
8287 064/300 Flush
8288 065/300 Flush
8289 066/300 Read block 634 len 36864
8290 067/300 Read block 940 len 36864
8291 068/300 Read block 3938 len 36864
8292 069/300 Write block 1819 len 16384 data: 5 4 4 5
8293 070/300 Read block 3658 len 12288
8294 071/300 Read block 2056 len 36864
8295 072/300 Read block 4110 len 32768
8296 073/300 Write block 4975 len 28672 data: 2 2 2 2 2 2 2
8297 074/300 Write block 201 len 12288 data: 3 2 2
8298 075/300 Write block 2549 len 8192 data: 3 3
8299 076/300 Write block 48 len 16384 data: 3 3 2 2
8300 077/300 Read block 2453 len 4096
8301 078/300 Write block 107 len 36864 data: 5 4 3 3 2 2 2 2 2
8302 079/300 Write block 1358 len 40960 data: 3 3 3 3 3 3 4 4 3 2
8303 080/300 Write block 634 len 4096 data: 3
8304 081/300 Read block 733 len 20480
8305 082/300 Write block 1807 len 20480 data: 4 4 5 5 4
8306 083/300 Read block 1624 len 12288
8307 084/300 Read block 4984 len 8192
8308 085/300 Read block 2896 len 40960
8309 086/300 Write block 409 len 24576 data: 4 4 4 3 2 2
8310 087/300 Write block 2525 len 16384 data: 4 4 4 4
8311 088/300 Read block 3836 len 8192
8312 089/300 Read block 4921 len 20480
8313 090/300 Write block 3325 len 36864 data: 3 2 2 2 2 2 2 2 2
8314 091/300 Read block 3761 len 24576
8315 092/300 Read block 3984 len 8192
8316 093/300 Read block 4511 len 36864
8317 094/300 Read block 4250 len 20480
8318 095/300 Read block 4003 len 4096
8319 096/300 Read block 1762 len 12288
8320 097/300 Read block 1263 len 40960
8321 098/300 Write block 1687 len 36864 data: 2 3 3 3 3 3 3 3 3
8322 099/300 Write block 1216 len 40960 data: 3 3 2 2 2 2 2 2 2 2
8323 100/300 Read block 3821 len 4096
8324 101/300 Write block 1015 len 8192 data: 4 3
8325 102/300 Write block 3979 len 16384 data: 3 3 3 3
8326 103/300 Write block 2203 len 28672 data: 4 4 4 3 3 4 3
8327 104/300 Write block 2525 len 16384 data: 5 5 5 5
8328 105/300 Read block 2052 len 28672
8329 106/300 Flush
8330 107/300 Read block 2843 len 8192
8331 108/300 Flush
8332 109/300 Read block 206 len 16384
8333 110/300 Flush
8334 111/300 Flush
8335 112/300 Read block 4960 len 8192
8336 113/300 Flush
8337 114/300 Write block 2973 len 12288 data: 2 3 2
8338 115/300 Read block 2132 len 12288
8339 116/300 Write block 1303 len 36864 data: 2 3 3 3 3 3 3 3 3
8340 117/300 Write block 3988 len 12288 data: 3 2 2
8341 118/300 Read block 891 len 20480
8342 119/300 Read block 2150 len 24576
8343 120/300 Flush
8344 121/300 Flush
8345 122/300 Read block 4900 len 40960
8346 123/300 Flush
8347 124/300 Write block 2564 len 36864 data: 3 3 3 3 2 2 2 2 2
8348 125/300 Read block 4835 len 4096
8349 126/300 Write block 3532 len 24576 data: 3 3 3 3 3 3
8350 127/300 Write block 4001 len 4096 data: 5
8351 128/300 Write block 911 len 40960 data: 3 2 2 2 2 3 3 3 3 3
8352 129/300 Write block 691 len 24576 data: 2 2 2 3 3 3
8353 130/300 Read block 827 len 40960
8354 131/300 Flush
8355 132/300 Write block 4922 len 32768 data: 3 3 3 3 2 2 3 4
8356 133/300 Write block 4918 len 8192 data: 3 4
8357 134/300 Write block 1088 len 32768 data: 2 2 2 3 3 3 3 3
8358 135/300 Read block 2899 len 28672
8359 136/300 Write block 905 len 20480 data: 2 3 3 3 3
8360 137/300 Write block 3770 len 4096 data: 4
8361 138/300 Read block 3743 len 24576
8362 139/300 Flush
8363 140/300 Read block 235 len 20480
8364 141/300 Write block 2729 len 12288 data: 2 2 2
8365 142/300 Read block 525 len 40960
8366 143/300 Write block 1323 len 36864 data: 3 3 4 4 5 3 5 5 4
8367 144/300 Write block 247 len 12288 data: 4 4 3
8368 145/300 Write block 1385 len 36864 data: 3 2 2 2 2 2 2 2 2
8369 146/300 Read block 2123 len 4096
8370 147/300 Write block 53 len 24576 data: 3 3 3 2 2 2
8371 148/300 Read block 1655 len 4096
8372 149/300 Write block 1609 len 32768 data: 5 5 5 5 6 6 6 4
8373 150/300 Read block 248 len 20480
8374 151/300 Read block 1191 len 28672
8375 152/300 Read block 4544 len 36864
8376 153/300 Write block 2250 len 28672 data: 3 3 5 5 5 6 6
8377 154/300 Write block 3279 len 4096 data: 3
8378 155/300 Read block 1085 len 28672
8379 156/300 Write block 2923 len 32768 data: 3 4 4 4 4 4 4 5
8380 157/300 Read block 1209 len 4096
8381 158/300 Write block 1530 len 32768 data: 2 3 3 4 5 5 5 4
8382 159/300 Write block 4246 len 8192 data: 2 2
8383 160/300 Write block 989 len 4096 data: 3
8384 161/300 Flush
8385 162/300 Write block 2608 len 40960 data: 2 2 2 2 3 3 4 4 5 5
8386 163/300 Write block 2225 len 12288 data: 2 2 2
8387 164/300 Flush
8388 165/300 Read block 163 len 4096
8389 166/300 Write block 297 len 40960 data: 2 2 2 2 2 2 2 2 2 3
8390 167/300 Flush
8391 168/300 Read block 4824 len 4096
8392 169/300 Read block 3312 len 24576
8393 170/300 Read block 797 len 24576
8394 171/300 Read block 1150 len 28672
8395 172/300 Read block 1413 len 12288
8396 173/300 Read block 1114 len 32768
8397 174/300 Write block 4545 len 40960 data: 3 3 3 4 4 3 3 3 2 2
8398 175/300 Write block 2482 len 40960 data: 4 3 3 3 3 3 3 3 3 3
8399 176/300 Read block 711 len 24576
8400 177/300 Read block 3571 len 12288
8401 178/300 Read block 370 len 8192
8402 179/300 Write block 2616 len 24576 data: 6 6 4 4 4 4
8403 180/300 Read block 1494 len 40960
8404 181/300 Read block 1852 len 20480
8405 182/300 Read block 317 len 24576
8406 183/300 Write block 4640 len 8192 data: 4 4
8407 184/300 Flush
8408 185/300 Write block 1061 len 20480 data: 3 3 4 4 4
8409 186/300 Read block 4720 len 24576
8410 187/300 Write block 2720 len 32768 data: 3 3 3 4 4 4 4 3
8411 188/300 Read block 1197 len 16384
8412 189/300 Flush
8413 190/300 Flush
8414 191/300 Write block 3429 len 32768 data: 6 6 5 4 5 5 4 4
8415 192/300 Write block 4651 len 28672 data: 3 3 3 3 2 2 2
8416 193/300 Read block 2514 len 28672
8417 194/300 Write block 2401 len 24576 data: 3 3 3 3 4 4
8418 195/300 Read block 4325 len 40960
8419 196/300 Read block 4820 len 16384
8420 197/300 Read block 3553 len 28672
8421 198/300 Read block 3268 len 8192
8422 199/300 Read block 677 len 8192
8423 200/300 Read block 384 len 40960
8424 201/300 Read block 322 len 28672
8425 202/300 Write block 2469 len 20480 data: 3 4 4 5 5
8426 203/300 Read block 2439 len 40960
8427 204/300 Write block 3395 len 4096 data: 5
8428 205/300 Write block 88 len 28672 data: 3 4 4 3 3 3 3
8429 206/300 Read block 2596 len 12288
8430 207/300 Read block 3662 len 32768
8431 208/300 Write block 3683 len 20480 data: 5 5 5 6 6
8432 209/300 Read block 3239 len 20480
8433 210/300 Read block 2849 len 36864
8434 211/300 Read block 4377 len 40960
8435 212/300 Write block 220 len 24576 data: 3 3 4 4 4 4
8436 213/300 Read block 1927 len 20480
8437 214/300 Read block 3739 len 40960
8438 215/300 Flush
8439 216/300 Flush
8440 217/300 Read block 1160 len 12288
8441 218/300 Read block 2740 len 12288
8442 219/300 Write block 2978 len 36864 data: 3 3 3 3 2 2 3 3 3
8443 220/300 Flush
8444 221/300 Read block 3396 len 36864
8445 222/300 Read block 4990 len 20480
8446 223/300 Write block 4696 len 4096 data: 3
8447 224/300 Read block 4989 len 36864
8448 225/300 Read block 886 len 32768
8449 226/300 Read block 999 len 32768
8450 227/300 Read block 1489 len 16384
8451 228/300 Read block 788 len 24576
8452 229/300 Read block 1385 len 40960
8453 230/300 Write block 2959 len 4096 data: 4
8454 231/300 Read block 1806 len 20480
8455 232/300 Read block 1311 len 8192
8456 233/300 Flush
8457 234/300 Write block 3727 len 4096 data: 2
8458 235/300 Flush
8459 236/300 Read block 818 len 24576
8460 237/300 Write block 4913 len 16384 data: 4 5 5 5
8461 238/300 Flush
8462 239/300 Read block 3525 len 8192
8463 240/300 Write block 362 len 8192 data: 3 3
8464 241/300 Read block 3844 len 16384
8465 242/300 Read block 1897 len 32768
8466 243/300 Flush
8467 244/300 Read block 2368 len 16384
8468 245/300 Write block 3474 len 4096 data: 4
8469 246/300 Write block 1829 len 32768 data: 4 3 3 3 4 4 3 3
8470 247/300 Read block 3700 len 40960
8471 248/300 Read block 1289 len 20480
8472 249/300 Write block 3726 len 40960 data: 2 3 2 2 2 2 3 3 3 3
8473 250/300 Read block 4833 len 20480
8474 251/300 Read block 4469 len 36864
8475 252/300 Flush
8476 253/300 Read block 2861 len 32768
8477 254/300 Read block 2326 len 8192
8478 255/300 Read block 1669 len 8192
8479 256/300 Read block 1850 len 32768
8480 257/300 Read block 311 len 20480
8481 258/300 Read block 4366 len 4096
8482 259/300 Write block 1094 len 4096 data: 4
8483 260/300 Write block 1429 len 12288 data: 2 2 3
8484 261/300 Write block 348 len 16384 data: 2 3 3 3
8485 262/300 Read block 1662 len 40960
8486 263/300 Read block 4928 len 16384
8487 264/300 Write block 4462 len 8192 data: 2 2
8488 265/300 Read block 2731 len 36864
8489 266/300 Read block 603 len 28672
8490 267/300 Write block 2712 len 12288 data: 4 4 3
8491 268/300 Write block 4234 len 16384 data: 3 4 4 4
8492 269/300 Read block 2806 len 20480
8493 270/300 Write block 1170 len 20480 data: 3 4 4 5 5
8494 271/300 Flush
8495 272/300 Read block 71 len 16384
8496 273/300 Write block 4903 len 24576 data: 3 4 4 4 4 4
8497 274/300 Write block 1729 len 36864 data: 5 4 4 3 3 2 2 2 2
8498 275/300 Write block 4372 len 24576 data: 3 2 2 3 3 3
8499 276/300 Read block 4905 len 40960
8500 277/300 Flush
8501 278/300 Read block 1579 len 36864
8502 279/300 Write block 2802 len 40960 data: 3 4 5 5 7 8 6 7 6 5
8503 280/300 Read block 3269 len 32768
8504 281/300 Flush
8505 282/300 Flush
8506 283/300 Read block 4936 len 40960
8507 284/300 Read block 616 len 16384
8508 285/300 Read block 1489 len 12288
8509 286/300 Write block 4586 len 28672 data: 2 2 3 2 3 3 3
8510 287/300 Read block 1955 len 32768
8511 288/300 Write block 745 len 12288 data: 3 2 5
8512 289/300 Read block 1940 len 24576
8513 290/300 Read block 2592 len 40960
8514 291/300 Write block 899 len 16384 data: 3 3 2 2
8515 292/300 Read block 3154 len 8192
8516 293/300 Write block 1276 len 8192 data: 4 4
8517 294/300 Read block 2014 len 8192
8518 295/300 Read block 3255 len 28672
8519 296/300 Read block 1587 len 12288
8520 297/300 Read block 3075 len 32768
8521 298/300 Read block 3100 len 12288
8522 299/300 Read block 1425 len 4096
8523 300/300 Read block 445 len 8192
8524 Main task has work to do, go find it
8525 got dsc Start(1)
8526 start 1
8527 Feb 12 01:36:12.971 INFO request completed, latency_us: 218, response_code: 204, uri: /start/cid/1, method: POST, req_id: 3fa51e35-7cc6-4df3-8f28-9298711071cf, remote_addr: 127.0.0.1:60456, local_addr: 127.0.0.1:9998
8528 Replay: started 1, returned:Ok(())
8529 ----------------------------------------------------------------
8530 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:300
8531 States: Active Offline Active
8532 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
8533 2414 Acked 3413 Write 4 Done New Done true
8534 2415 Acked 3414 Write 4 Done New Done true
8535 2416 Acked 3415 Read 10 Done New Done true
8536 2417 Acked 3416 Write 4 Done New Done true
8537 2418 Acked 3417 Write 4 Done New Done true
8538 2419 Acked 3418 Read 2 Done New Done true
8539 2420 Acked 3419 Flush 0 Done New Done false
8540 2421 Acked 3420 Flush 0 Done New Done false
8541 2422 Acked 3421 Write 5 Done New Done false
8542 2423 Acked 3422 Read 3 Done New Done false
8543 2424 Acked 3423 Flush 0 Done New Done false
8544 2425 Acked 3424 Write 8 Done New Done false
8545 2426 Acked 3425 Write 8 Done New Done false
8546 2427 Acked 3426 Write 1 Done New Done false
8547 2428 Acked 3427 Read 4 Done New Done false
8548 2429 Acked 3428 Write 1 Done New Done false
8549 2430 Acked 3429 Write 9 Done New Done false
8550 2431 Acked 3430 Write 3 Done New Done false
8551 2432 Acked 3431 Read 3 Done New Done false
8552 2433 Acked 3432 Write 1 Done New Done false
8553 2434 Acked 3433 Read 8 Done New Done false
8554 2435 Acked 3434 Read 9 Done New Done false
8555 2436 Acked 3435 Write 3 Done New Done false
8556 2437 Acked 3436 Write 6 Done New Done false
8557 2438 Acked 3437 Flush 0 Done New Done false
8558 2439 Acked 3438 Flush 0 Done New Done false
8559 2440 Acked 3439 Read 7 Done New Done false
8560 2441 Acked 3440 Write 8 Done New Done false
8561 2442 Acked 3441 Write 2 Done New Done false
8562 2443 Acked 3442 Write 3 Done New Done false
8563 2444 Acked 3443 Flush 0 Done New Done false
8564 2445 Acked 3444 Read 4 Done New Done false
8565 2446 Acked 3445 Write 4 Done New Done false
8566 2447 Acked 3446 Read 5 Done New Done false
8567 2448 Acked 3447 Read 1 Done New Done false
8568 2449 Acked 3448 Write 5 Done New Done false
8569 2450 Acked 3449 Flush 0 Done New Done false
8570 2451 Acked 3450 Read 10 Done New Done false
8571 2452 Acked 3451 Write 7 Done New Done false
8572 2453 Acked 3452 Write 10 Done New Done false
8573 2454 Acked 3453 Flush 0 Done New Done false
8574 2455 Acked 3454 Read 1 Done New Done false
8575 2456 Acked 3455 Read 4 Done New Done false
8576 2457 Acked 3456 Write 5 Done New Done false
8577 2458 Acked 3457 Read 2 Done New Done false
8578 2459 Acked 3458 Write 7 Done New Done false
8579 2460 Acked 3459 Write 3 Done New Done false
8580 2461 Acked 3460 Read 9 Done New Done false
8581 2462 Acked 3461 Write 10 Done New Done false
8582 2463 Acked 3462 Write 3 Done New Done false
8583 2464 Acked 3463 Read 1 Done New Done false
8584 2465 Acked 3464 Read 1 Done New Done false
8585 2466 Acked 3465 Write 4 Done New Done false
8586 2467 Acked 3466 Write 5 Done New Done false
8587 2468 Acked 3467 Write 6 Done New Done false
8588 2469 Acked 3468 Read 4 Done New Done false
8589 2470 Acked 3469 Flush 0 Done New Done false
8590 2471 Acked 3470 Write 10 Done New Done false
8591 2472 Acked 3471 Write 9 Done New Done false
8592 2473 Acked 3472 Write 8 Done New Done false
8593 2474 Acked 3473 Write 8 Done New Done false
8594 2475 Acked 3474 Write 7 Done New Done false
8595 2476 Acked 3475 Flush 0 Done New Done false
8596 2477 Acked 3476 Flush 0 Done New Done false
8597 2478 Acked 3477 Flush 0 Done New Done false
8598 2479 Acked 3478 Read 9 Done New Done false
8599 2480 Acked 3479 Read 9 Done New Done false
8600 2481 Acked 3480 Read 9 Done New Done false
8601 2482 Acked 3481 Write 4 Done New Done false
8602 2483 Acked 3482 Read 3 Done New Done false
8603 2484 Acked 3483 Read 9 Done New Done false
8604 2485 Acked 3484 Read 8 Done New Done false
8605 2486 Acked 3485 Write 7 Done New Done false
8606 2487 Acked 3486 Write 3 Done New Done false
8607 2488 Acked 3487 Write 2 Done New Done false
8608 2489 Acked 3488 Write 4 Done New Done false
8609 2490 Acked 3489 Read 1 Done New Done false
8610 2491 Acked 3490 Write 9 Done New Done false
8611 2492 Acked 3491 Write 10 Done New Done false
8612 2493 Acked 3492 Write 1 Done New Done false
8613 2494 Acked 3493 Read 5 Done New Done false
8614 2495 Acked 3494 Write 5 Done New Done false
8615 2496 Acked 3495 Read 3 Done New Done false
8616 2497 Acked 3496 Read 2 Done New Done false
8617 2498 Acked 3497 Read 10 Done New Done false
8618 2499 Acked 3498 Write 6 Done New Done false
8619 2500 Acked 3499 Write 4 Done New Done false
8620 2501 Acked 3500 Read 2 Done New Done false
8621 2502 Acked 3501 Read 5 Done New Done false
8622 2503 Acked 3502 Write 9 Done New Done false
8623 2504 Acked 3503 Read 6 Done New Done false
8624 2505 Acked 3504 Read 2 Done New Done false
8625 2506 Acked 3505 Read 9 Done New Done false
8626 2507 Acked 3506 Read 5 Done New Done false
8627 2508 Acked 3507 Read 1 Done New Done false
8628 2509 Acked 3508 Read 3 Done New Done false
8629 2510 Acked 3509 Read 10 Done New Done false
8630 2511 Acked 3510 Write 9 Done New Done false
8631 2512 Acked 3511 Write 10 Done New Done false
8632 2513 Acked 3512 Read 1 Done New Done false
8633 2514 Acked 3513 Write 2 Done New Done false
8634 2515 Acked 3514 Write 4 Done New Done false
8635 2516 Acked 3515 Write 7 Done New Done false
8636 2517 Acked 3516 Write 4 Done New Done false
8637 2518 Acked 3517 Read 7 Done New Done false
8638 2519 Acked 3518 Flush 0 Done New Done false
8639 2520 Acked 3519 Read 2 Done New Done false
8640 2521 Acked 3520 Flush 0 Done New Done false
8641 2522 Acked 3521 Read 4 Done New Done false
8642 2523 Acked 3522 Flush 0 Done New Done false
8643 2524 Acked 3523 Flush 0 Done New Done false
8644 2525 Acked 3524 Read 2 Done New Done false
8645 2526 Acked 3525 Flush 0 Done New Done false
8646 2527 Acked 3526 Write 3 Done New Done false
8647 2528 Acked 3527 Read 3 Done New Done false
8648 2529 Acked 3528 Write 9 Done New Done false
8649 2530 Acked 3529 Write 3 Done New Done false
8650 2531 Acked 3530 Read 5 Done New Done false
8651 2532 Acked 3531 Read 6 Done New Done false
8652 2533 Acked 3532 Flush 0 Done New Done false
8653 2534 Acked 3533 Flush 0 Done New Done false
8654 2535 Acked 3534 Read 10 Done New Done false
8655 2536 Acked 3535 Flush 0 Done New Done false
8656 2537 Acked 3536 Write 9 Done New Done false
8657 2538 Acked 3537 Read 1 Done New Done false
8658 2539 Acked 3538 Write 6 Done New Done false
8659 2540 Acked 3539 Write 1 Done New Done false
8660 2541 Acked 3540 Write 10 Done New Done false
8661 2542 Acked 3541 Write 6 Done New Done false
8662 2543 Acked 3542 Read 10 Done New Done false
8663 2544 Acked 3543 Flush 0 Done New Done false
8664 2545 Acked 3544 Write 8 Done New Done false
8665 2546 Acked 3545 Write 2 Done New Done false
8666 2547 Acked 3546 Write 8 Done New Done false
8667 2548 Acked 3547 Read 7 Done New Done false
8668 2549 Acked 3548 Write 5 Done New Done false
8669 2550 Acked 3549 Write 1 Done New Done false
8670 2551 Acked 3550 Read 6 Done New Done false
8671 2552 Acked 3551 Flush 0 Done New Done false
8672 2553 Acked 3552 Read 5 Done New Done false
8673 2554 Acked 3553 Write 3 Done New Done false
8674 2555 Acked 3554 Read 10 Done New Done false
8675 2556 Acked 3555 Write 9 Done New Done false
8676 2557 Acked 3556 Write 3 Done New Done false
8677 2558 Acked 3557 Write 9 Done New Done false
8678 2559 Acked 3558 Read 1 Done New Done false
8679 2560 Acked 3559 Write 6 Done New Done false
8680 2561 Acked 3560 Read 1 Done New Done false
8681 2562 Acked 3561 Write 8 Done New Done false
8682 2563 Acked 3562 Read 5 Done New Done false
8683 2564 Acked 3563 Read 7 Done New Done false
8684 2565 Acked 3564 Read 9 Done New Done false
8685 2566 Acked 3565 Write 7 Done New Done false
8686 2567 Acked 3566 Write 1 Done New Done false
8687 2568 Acked 3567 Read 7 Done New Done false
8688 2569 Acked 3568 Write 8 Done New Done false
8689 2570 Acked 3569 Read 1 Done New Done false
8690 2571 Acked 3570 Write 8 Done New Done false
8691 2572 Acked 3571 Write 2 Done New Done false
8692 2573 Acked 3572 Write 1 Done New Done false
8693 2574 Acked 3573 Flush 0 Done New Done false
8694 2575 Acked 3574 Write 10 Done New Done false
8695 2576 Acked 3575 Write 3 Done New Done false
8696 2577 Acked 3576 Flush 0 Done New Done false
8697 2578 Acked 3577 Read 1 Done New Done false
8698 2579 Acked 3578 Write 10 Done New Done false
8699 2580 Acked 3579 Flush 0 Done New Done false
8700 2581 Acked 3580 Read 1 Done New Done false
8701 2582 Acked 3581 Read 6 Done New Done false
8702 2583 Acked 3582 Read 6 Done New Done false
8703 2584 Acked 3583 Read 7 Done New Done false
8704 2585 Acked 3584 Read 3 Done New Done false
8705 2586 Acked 3585 Read 8 Done New Done false
8706 2587 Acked 3586 Write 10 Done New Done false
8707 2588 Acked 3587 Write 10 Done New Done false
8708 2589 Acked 3588 Read 6 Done New Done false
8709 2590 Acked 3589 Read 3 Done New Done false
8710 2591 Acked 3590 Read 2 Done New Done false
8711 2592 Acked 3591 Write 6 Done New Done false
8712 2593 Acked 3592 Read 10 Done New Done false
8713 2594 Acked 3593 Read 5 Done New Done false
8714 2595 Acked 3594 Read 6 Done New Done false
8715 2596 Acked 3595 Write 2 Done New Done false
8716 2597 Acked 3596 Flush 0 Done New Done false
8717 2598 Acked 3597 Write 5 Done New Done false
8718 2599 Acked 3598 Read 6 Done New Done false
8719 2600 Acked 3599 Write 8 Done New Done false
8720 2601 Acked 3600 Read 4 Done New Done false
8721 2602 Acked 3601 Flush 0 Done New Done false
8722 2603 Acked 3602 Flush 0 Done New Done false
8723 2604 Acked 3603 Write 8 Done New Done false
8724 2605 Acked 3604 Write 7 Done New Done false
8725 2606 Acked 3605 Read 7 Done New Done false
8726 2607 Acked 3606 Write 6 Done New Done false
8727 2608 Acked 3607 Read 10 Done New Done false
8728 2609 Acked 3608 Read 4 Done New Done false
8729 2610 Acked 3609 Read 7 Done New Done false
8730 2611 Acked 3610 Read 2 Done New Done false
8731 2612 Acked 3611 Read 2 Done New Done false
8732 2613 Acked 3612 Read 10 Done New Done false
8733 2614 Acked 3613 Read 7 Done New Done false
8734 2615 Acked 3614 Write 5 Done New Done false
8735 2616 Acked 3615 Read 10 Done New Done false
8736 2617 Acked 3616 Write 1 Done New Done false
8737 2618 Acked 3617 Write 7 Done New Done false
8738 2619 Acked 3618 Read 3 Done New Done false
8739 2620 Acked 3619 Read 8 Done New Done false
8740 2621 Acked 3620 Write 5 Done New Done false
8741 2622 Acked 3621 Read 5 Done New Done false
8742 2623 Acked 3622 Read 9 Done New Done false
8743 2624 Acked 3623 Read 10 Done New Done false
8744 2625 Acked 3624 Write 6 Done New Done false
8745 2626 Acked 3625 Read 5 Done New Done false
8746 2627 Acked 3626 Read 10 Done New Done false
8747 2628 Acked 3627 Flush 0 Done New Done false
8748 2629 Acked 3628 Flush 0 Done New Done false
8749 2630 Acked 3629 Read 3 Done New Done false
8750 2631 Acked 3630 Read 3 Done New Done false
8751 2632 Acked 3631 Write 9 Done New Done false
8752 2633 Acked 3632 Flush 0 Done New Done false
8753 2634 Acked 3633 Read 9 Done New Done false
8754 2635 Acked 3634 Read 5 Done New Done false
8755 2636 Acked 3635 Write 1 Done New Done false
8756 2637 Acked 3636 Read 9 Done New Done false
8757 2638 Acked 3637 Read 8 Done New Done false
8758 2639 Acked 3638 Read 8 Done New Done false
8759 2640 Acked 3639 Read 4 Done New Done false
8760 2641 Acked 3640 Read 6 Done New Done false
8761 2642 Acked 3641 Read 10 Done New Done false
8762 2643 Acked 3642 Write 1 Done New Done false
8763 2644 Acked 3643 Read 5 Done New Done false
8764 2645 Acked 3644 Read 2 Done New Done false
8765 2646 Acked 3645 Flush 0 Done New Done false
8766 2647 Acked 3646 Write 1 Done New Done false
8767 2648 Acked 3647 Flush 0 Done New Done false
8768 2649 Acked 3648 Read 6 Done New Done false
8769 2650 Acked 3649 Write 4 Done New Done false
8770 2651 Acked 3650 Flush 0 Done New Done false
8771 2652 Acked 3651 Read 2 Done New Done false
8772 2653 Acked 3652 Write 2 Done New Done false
8773 2654 Acked 3653 Read 4 Done New Done false
8774 2655 Acked 3654 Read 8 Done New Done false
8775 2656 Acked 3655 Flush 0 Done New Done false
8776 2657 Acked 3656 Read 4 Done New Done false
8777 2658 Acked 3657 Write 1 Done New Done false
8778 2659 Acked 3658 Write 8 Done New Done false
8779 2660 Acked 3659 Read 10 Done New Done false
8780 2661 Acked 3660 Read 5 Done New Done false
8781 2662 Acked 3661 Write 10 Done New Done false
8782 2663 Acked 3662 Read 5 Done New Done false
8783 2664 Acked 3663 Read 9 Done New Done false
8784 2665 Acked 3664 Flush 0 Done New Done false
8785 2666 Acked 3665 Read 8 Done New Done false
8786 2667 Acked 3666 Read 2 Done New Done false
8787 2668 Acked 3667 Read 2 Done New Done false
8788 2669 Acked 3668 Read 8 Done New Done false
8789 2670 Acked 3669 Read 5 Done New Done false
8790 2671 Acked 3670 Read 1 Done New Done false
8791 2672 Acked 3671 Write 1 Done New Done false
8792 2673 Acked 3672 Write 3 Done New Done false
8793 2674 Acked 3673 Write 4 Done New Done false
8794 2675 Acked 3674 Read 10 Done New Done false
8795 2676 Acked 3675 Read 4 Done New Done false
8796 2677 Acked 3676 Write 2 Done New Done false
8797 2678 Acked 3677 Read 9 Done New Done false
8798 2679 Acked 3678 Read 7 Done New Done false
8799 2680 Acked 3679 Write 3 Done New Done false
8800 2681 Acked 3680 Write 4 Done New Done false
8801 2682 Acked 3681 Read 5 Done New Done false
8802 2683 Acked 3682 Write 5 Done New Done false
8803 2684 Acked 3683 Flush 0 Done New Done false
8804 2685 Acked 3684 Read 4 Done New Done false
8805 2686 Acked 3685 Write 6 Done New Done false
8806 2687 Acked 3686 Write 9 Done New Done false
8807 2688 Acked 3687 Write 6 Done New Done false
8808 2689 Acked 3688 Read 10 Done New Done false
8809 2690 Acked 3689 Flush 0 Done New Done false
8810 2691 Acked 3690 Read 9 Done New Done false
8811 2692 Acked 3691 Write 10 Done New Done false
8812 2693 Acked 3692 Read 8 Done New Done false
8813 2694 Acked 3693 Flush 0 Done New Done false
8814 2695 Acked 3694 Flush 0 Done New Done false
8815 2696 Acked 3695 Read 10 Done New Done false
8816 2697 Acked 3696 Read 4 Done New Done false
8817 2698 Acked 3697 Read 3 Done New Done false
8818 2699 Acked 3698 Write 7 Done New Done false
8819 2700 Acked 3699 Read 8 Done New Done false
8820 2701 Acked 3700 Write 3 Done New Done false
8821 2702 Acked 3701 Read 6 Done New Done false
8822 2703 Acked 3702 Read 10 Done New Done false
8823 2704 Acked 3703 Write 4 Done New Done false
8824 2705 Acked 3704 Read 2 Done New Done false
8825 2706 Acked 3705 Write 2 Done New Done false
8826 2707 Acked 3706 Read 2 Done New Done false
8827 2708 Acked 3707 Read 7 Done New Done false
8828 2709 Acked 3708 Read 3 Done New Done false
8829 2710 Acked 3709 Read 8 Done New Done false
8830 2711 Acked 3710 Read 3 Done New Done false
8831 2712 Acked 3711 Read 1 Done New Done false
8832 2713 Acked 3712 Read 2 Done New Done false
8833 STATES DS:0 DS:1 DS:2 TOTAL
8834 New 0 300 0 300
8835 Sent 0 0 0 0
8836 Done 300 0 300 600
8837 Skipped 0 0 0 0
8838 Error 0 0 0 0
8839 Last Flush: 3694 3412 3694
8840 Downstairs last five completed: 3412 3411 3410 3409 3408
8841 Upstairs last five completed: 2713 2712 2711 2710 2709
8842 CLIENT: Up:0 ds:300 act:2
88432024-02-12T01:36:13.626ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
88442024-02-12T01:36:13.626ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
88452024-02-12T01:36:13.627ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
88462024-02-12T01:36:13.627ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
88472024-02-12T01:36:13.627ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
88482024-02-12T01:36:13.627ZINFOcrucible: [1] client re-new 301 jobs since flush 3412 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
88492024-02-12T01:36:13.627ZINFOcrucible: client task is exiting = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
88502024-02-12T01:36:14.628ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
88512024-02-12T01:36:14.628ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
88522024-02-12T01:36:14.628ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
88532024-02-12T01:36:14.628ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
88542024-02-12T01:36:14.628ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
88552024-02-12T01:36:14.628ZINFOcrucible: [1] client re-new 301 jobs since flush 3412 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
88562024-02-12T01:36:14.628ZINFOcrucible: client task is exiting = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
88572024-02-12T01:36:15.629ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
88582024-02-12T01:36:15.629ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
88592024-02-12T01:36:15.629ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
88602024-02-12T01:36:15.629ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
88612024-02-12T01:36:15.629ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
88622024-02-12T01:36:15.629ZINFOcrucible: [1] client re-new 301 jobs since flush 3412 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
88632024-02-12T01:36:15.629ZINFOcrucible: client task is exiting = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
88642024-02-12T01:36:16.630ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
88652024-02-12T01:36:16.630ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
88662024-02-12T01:36:16.630ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
88672024-02-12T01:36:16.630ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
88682024-02-12T01:36:16.630ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
88692024-02-12T01:36:16.630ZINFOcrucible: [1] client re-new 301 jobs since flush 3412 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
88702024-02-12T01:36:16.630ZINFOcrucible: client task is exiting = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
8871 ----------------------------------------------------------------
8872 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
8873 States: Active Offline Active
8874 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
8875 2414 Acked 3413 Write 4 Done New Done true
8876 2415 Acked 3414 Write 4 Done New Done true
8877 2416 Acked 3415 Read 10 Done New Done true
8878 2417 Acked 3416 Write 4 Done New Done true
8879 2418 Acked 3417 Write 4 Done New Done true
8880 2419 Acked 3418 Read 2 Done New Done true
8881 2420 Acked 3419 Flush 0 Done New Done true
8882 2421 Acked 3420 Flush 0 Done New Done true
8883 2422 Acked 3421 Write 5 Done New Done true
8884 2423 Acked 3422 Read 3 Done New Done true
8885 2424 Acked 3423 Flush 0 Done New Done true
8886 2425 Acked 3424 Write 8 Done New Done true
8887 2426 Acked 3425 Write 8 Done New Done true
8888 2427 Acked 3426 Write 1 Done New Done true
8889 2428 Acked 3427 Read 4 Done New Done true
8890 2429 Acked 3428 Write 1 Done New Done true
8891 2430 Acked 3429 Write 9 Done New Done true
8892 2431 Acked 3430 Write 3 Done New Done true
8893 2432 Acked 3431 Read 3 Done New Done true
8894 2433 Acked 3432 Write 1 Done New Done true
8895 2434 Acked 3433 Read 8 Done New Done true
8896 2435 Acked 3434 Read 9 Done New Done true
8897 2436 Acked 3435 Write 3 Done New Done true
8898 2437 Acked 3436 Write 6 Done New Done true
8899 2438 Acked 3437 Flush 0 Done New Done true
8900 2439 Acked 3438 Flush 0 Done New Done true
8901 2440 Acked 3439 Read 7 Done New Done true
8902 2441 Acked 3440 Write 8 Done New Done true
8903 2442 Acked 3441 Write 2 Done New Done true
8904 2443 Acked 3442 Write 3 Done New Done true
8905 2444 Acked 3443 Flush 0 Done New Done true
8906 2445 Acked 3444 Read 4 Done New Done true
8907 2446 Acked 3445 Write 4 Done New Done true
8908 2447 Acked 3446 Read 5 Done New Done true
8909 2448 Acked 3447 Read 1 Done New Done true
8910 2449 Acked 3448 Write 5 Done New Done true
8911 2450 Acked 3449 Flush 0 Done New Done true
8912 2451 Acked 3450 Read 10 Done New Done true
8913 2452 Acked 3451 Write 7 Done New Done true
8914 2453 Acked 3452 Write 10 Done New Done true
8915 2454 Acked 3453 Flush 0 Done New Done true
8916 2455 Acked 3454 Read 1 Done New Done true
8917 2456 Acked 3455 Read 4 Done New Done true
8918 2457 Acked 3456 Write 5 Done New Done true
8919 2458 Acked 3457 Read 2 Done New Done true
8920 2459 Acked 3458 Write 7 Done New Done true
8921 2460 Acked 3459 Write 3 Done New Done true
8922 2461 Acked 3460 Read 9 Done New Done true
8923 2462 Acked 3461 Write 10 Done New Done true
8924 2463 Acked 3462 Write 3 Done New Done true
8925 2464 Acked 3463 Read 1 Done New Done true
8926 2465 Acked 3464 Read 1 Done New Done true
8927 2466 Acked 3465 Write 4 Done New Done true
8928 2467 Acked 3466 Write 5 Done New Done true
8929 2468 Acked 3467 Write 6 Done New Done true
8930 2469 Acked 3468 Read 4 Done New Done true
8931 2470 Acked 3469 Flush 0 Done New Done true
8932 2471 Acked 3470 Write 10 Done New Done true
8933 2472 Acked 3471 Write 9 Done New Done true
8934 2473 Acked 3472 Write 8 Done New Done true
8935 2474 Acked 3473 Write 8 Done New Done true
8936 2475 Acked 3474 Write 7 Done New Done true
8937 2476 Acked 3475 Flush 0 Done New Done true
8938 2477 Acked 3476 Flush 0 Done New Done true
8939 2478 Acked 3477 Flush 0 Done New Done true
8940 2479 Acked 3478 Read 9 Done New Done true
8941 2480 Acked 3479 Read 9 Done New Done true
8942 2481 Acked 3480 Read 9 Done New Done true
8943 2482 Acked 3481 Write 4 Done New Done true
8944 2483 Acked 3482 Read 3 Done New Done true
8945 2484 Acked 3483 Read 9 Done New Done true
8946 2485 Acked 3484 Read 8 Done New Done true
8947 2486 Acked 3485 Write 7 Done New Done true
8948 2487 Acked 3486 Write 3 Done New Done true
8949 2488 Acked 3487 Write 2 Done New Done true
8950 2489 Acked 3488 Write 4 Done New Done true
8951 2490 Acked 3489 Read 1 Done New Done true
8952 2491 Acked 3490 Write 9 Done New Done true
8953 2492 Acked 3491 Write 10 Done New Done true
8954 2493 Acked 3492 Write 1 Done New Done true
8955 2494 Acked 3493 Read 5 Done New Done true
8956 2495 Acked 3494 Write 5 Done New Done true
8957 2496 Acked 3495 Read 3 Done New Done true
8958 2497 Acked 3496 Read 2 Done New Done true
8959 2498 Acked 3497 Read 10 Done New Done true
8960 2499 Acked 3498 Write 6 Done New Done true
8961 2500 Acked 3499 Write 4 Done New Done true
8962 2501 Acked 3500 Read 2 Done New Done true
8963 2502 Acked 3501 Read 5 Done New Done true
8964 2503 Acked 3502 Write 9 Done New Done true
8965 2504 Acked 3503 Read 6 Done New Done true
8966 2505 Acked 3504 Read 2 Done New Done true
8967 2506 Acked 3505 Read 9 Done New Done true
8968 2507 Acked 3506 Read 5 Done New Done true
8969 2508 Acked 3507 Read 1 Done New Done true
8970 2509 Acked 3508 Read 3 Done New Done true
8971 2510 Acked 3509 Read 10 Done New Done true
8972 2511 Acked 3510 Write 9 Done New Done true
8973 2512 Acked 3511 Write 10 Done New Done true
8974 2513 Acked 3512 Read 1 Done New Done true
8975 2514 Acked 3513 Write 2 Done New Done true
8976 2515 Acked 3514 Write 4 Done New Done true
8977 2516 Acked 3515 Write 7 Done New Done true
8978 2517 Acked 3516 Write 4 Done New Done true
8979 2518 Acked 3517 Read 7 Done New Done true
8980 2519 Acked 3518 Flush 0 Done New Done true
8981 2520 Acked 3519 Read 2 Done New Done true
8982 2521 Acked 3520 Flush 0 Done New Done true
8983 2522 Acked 3521 Read 4 Done New Done true
8984 2523 Acked 3522 Flush 0 Done New Done true
8985 2524 Acked 3523 Flush 0 Done New Done true
8986 2525 Acked 3524 Read 2 Done New Done true
8987 2526 Acked 3525 Flush 0 Done New Done true
8988 2527 Acked 3526 Write 3 Done New Done true
8989 2528 Acked 3527 Read 3 Done New Done true
8990 2529 Acked 3528 Write 9 Done New Done true
8991 2530 Acked 3529 Write 3 Done New Done true
8992 2531 Acked 3530 Read 5 Done New Done true
8993 2532 Acked 3531 Read 6 Done New Done true
8994 2533 Acked 3532 Flush 0 Done New Done true
8995 2534 Acked 3533 Flush 0 Done New Done true
8996 2535 Acked 3534 Read 10 Done New Done true
8997 2536 Acked 3535 Flush 0 Done New Done true
8998 2537 Acked 3536 Write 9 Done New Done true
8999 2538 Acked 3537 Read 1 Done New Done true
9000 2539 Acked 3538 Write 6 Done New Done true
9001 2540 Acked 3539 Write 1 Done New Done true
9002 2541 Acked 3540 Write 10 Done New Done true
9003 2542 Acked 3541 Write 6 Done New Done true
9004 2543 Acked 3542 Read 10 Done New Done true
9005 2544 Acked 3543 Flush 0 Done New Done true
9006 2545 Acked 3544 Write 8 Done New Done true
9007 2546 Acked 3545 Write 2 Done New Done true
9008 2547 Acked 3546 Write 8 Done New Done true
9009 2548 Acked 3547 Read 7 Done New Done true
9010 2549 Acked 3548 Write 5 Done New Done true
9011 2550 Acked 3549 Write 1 Done New Done true
9012 2551 Acked 3550 Read 6 Done New Done true
9013 2552 Acked 3551 Flush 0 Done New Done true
9014 2553 Acked 3552 Read 5 Done New Done true
9015 2554 Acked 3553 Write 3 Done New Done true
9016 2555 Acked 3554 Read 10 Done New Done true
9017 2556 Acked 3555 Write 9 Done New Done true
9018 2557 Acked 3556 Write 3 Done New Done true
9019 2558 Acked 3557 Write 9 Done New Done true
9020 2559 Acked 3558 Read 1 Done New Done true
9021 2560 Acked 3559 Write 6 Done New Done true
9022 2561 Acked 3560 Read 1 Done New Done true
9023 2562 Acked 3561 Write 8 Done New Done true
9024 2563 Acked 3562 Read 5 Done New Done true
9025 2564 Acked 3563 Read 7 Done New Done true
9026 2565 Acked 3564 Read 9 Done New Done true
9027 2566 Acked 3565 Write 7 Done New Done true
9028 2567 Acked 3566 Write 1 Done New Done true
9029 2568 Acked 3567 Read 7 Done New Done true
9030 2569 Acked 3568 Write 8 Done New Done true
9031 2570 Acked 3569 Read 1 Done New Done true
9032 2571 Acked 3570 Write 8 Done New Done true
9033 2572 Acked 3571 Write 2 Done New Done true
9034 2573 Acked 3572 Write 1 Done New Done true
9035 2574 Acked 3573 Flush 0 Done New Done true
9036 2575 Acked 3574 Write 10 Done New Done true
9037 2576 Acked 3575 Write 3 Done New Done true
9038 2577 Acked 3576 Flush 0 Done New Done true
9039 2578 Acked 3577 Read 1 Done New Done true
9040 2579 Acked 3578 Write 10 Done New Done true
9041 2580 Acked 3579 Flush 0 Done New Done true
9042 2581 Acked 3580 Read 1 Done New Done true
9043 2582 Acked 3581 Read 6 Done New Done true
9044 2583 Acked 3582 Read 6 Done New Done true
9045 2584 Acked 3583 Read 7 Done New Done true
9046 2585 Acked 3584 Read 3 Done New Done true
9047 2586 Acked 3585 Read 8 Done New Done true
9048 2587 Acked 3586 Write 10 Done New Done true
9049 2588 Acked 3587 Write 10 Done New Done true
9050 2589 Acked 3588 Read 6 Done New Done true
9051 2590 Acked 3589 Read 3 Done New Done true
9052 2591 Acked 3590 Read 2 Done New Done true
9053 2592 Acked 3591 Write 6 Done New Done true
9054 2593 Acked 3592 Read 10 Done New Done true
9055 2594 Acked 3593 Read 5 Done New Done true
9056 2595 Acked 3594 Read 6 Done New Done true
9057 2596 Acked 3595 Write 2 Done New Done true
9058 2597 Acked 3596 Flush 0 Done New Done true
9059 2598 Acked 3597 Write 5 Done New Done true
9060 2599 Acked 3598 Read 6 Done New Done true
9061 2600 Acked 3599 Write 8 Done New Done true
9062 2601 Acked 3600 Read 4 Done New Done true
9063 2602 Acked 3601 Flush 0 Done New Done true
9064 2603 Acked 3602 Flush 0 Done New Done true
9065 2604 Acked 3603 Write 8 Done New Done true
9066 2605 Acked 3604 Write 7 Done New Done true
9067 2606 Acked 3605 Read 7 Done New Done true
9068 2607 Acked 3606 Write 6 Done New Done true
9069 2608 Acked 3607 Read 10 Done New Done true
9070 2609 Acked 3608 Read 4 Done New Done true
9071 2610 Acked 3609 Read 7 Done New Done true
9072 2611 Acked 3610 Read 2 Done New Done true
9073 2612 Acked 3611 Read 2 Done New Done true
9074 2613 Acked 3612 Read 10 Done New Done true
9075 2614 Acked 3613 Read 7 Done New Done true
9076 2615 Acked 3614 Write 5 Done New Done true
9077 2616 Acked 3615 Read 10 Done New Done true
9078 2617 Acked 3616 Write 1 Done New Done true
9079 2618 Acked 3617 Write 7 Done New Done true
9080 2619 Acked 3618 Read 3 Done New Done true
9081 2620 Acked 3619 Read 8 Done New Done true
9082 2621 Acked 3620 Write 5 Done New Done true
9083 2622 Acked 3621 Read 5 Done New Done true
9084 2623 Acked 3622 Read 9 Done New Done true
9085 2624 Acked 3623 Read 10 Done New Done true
9086 2625 Acked 3624 Write 6 Done New Done true
9087 2626 Acked 3625 Read 5 Done New Done true
9088 2627 Acked 3626 Read 10 Done New Done true
9089 2628 Acked 3627 Flush 0 Done New Done true
9090 2629 Acked 3628 Flush 0 Done New Done true
9091 2630 Acked 3629 Read 3 Done New Done true
9092 2631 Acked 3630 Read 3 Done New Done true
9093 2632 Acked 3631 Write 9 Done New Done true
9094 2633 Acked 3632 Flush 0 Done New Done true
9095 2634 Acked 3633 Read 9 Done New Done true
9096 2635 Acked 3634 Read 5 Done New Done true
9097 2636 Acked 3635 Write 1 Done New Done true
9098 2637 Acked 3636 Read 9 Done New Done true
9099 2638 Acked 3637 Read 8 Done New Done true
9100 2639 Acked 3638 Read 8 Done New Done true
9101 2640 Acked 3639 Read 4 Done New Done true
9102 2641 Acked 3640 Read 6 Done New Done true
9103 2642 Acked 3641 Read 10 Done New Done true
9104 2643 Acked 3642 Write 1 Done New Done true
9105 2644 Acked 3643 Read 5 Done New Done true
9106 2645 Acked 3644 Read 2 Done New Done true
9107 2646 Acked 3645 Flush 0 Done New Done true
9108 2647 Acked 3646 Write 1 Done New Done true
9109 2648 Acked 3647 Flush 0 Done New Done true
9110 2649 Acked 3648 Read 6 Done New Done true
9111 2650 Acked 3649 Write 4 Done New Done true
9112 2651 Acked 3650 Flush 0 Done New Done true
9113 2652 Acked 3651 Read 2 Done New Done true
9114 2653 Acked 3652 Write 2 Done New Done true
9115 2654 Acked 3653 Read 4 Done New Done true
9116 2655 Acked 3654 Read 8 Done New Done true
9117 2656 Acked 3655 Flush 0 Done New Done true
9118 2657 Acked 3656 Read 4 Done New Done true
9119 2658 Acked 3657 Write 1 Done New Done true
9120 2659 Acked 3658 Write 8 Done New Done true
9121 2660 Acked 3659 Read 10 Done New Done true
9122 2661 Acked 3660 Read 5 Done New Done true
9123 2662 Acked 3661 Write 10 Done New Done true
9124 2663 Acked 3662 Read 5 Done New Done true
9125 2664 Acked 3663 Read 9 Done New Done true
9126 2665 Acked 3664 Flush 0 Done New Done true
9127 2666 Acked 3665 Read 8 Done New Done true
9128 2667 Acked 3666 Read 2 Done New Done true
9129 2668 Acked 3667 Read 2 Done New Done true
9130 2669 Acked 3668 Read 8 Done New Done true
9131 2670 Acked 3669 Read 5 Done New Done true
9132 2671 Acked 3670 Read 1 Done New Done true
9133 2672 Acked 3671 Write 1 Done New Done true
9134 2673 Acked 3672 Write 3 Done New Done true
9135 2674 Acked 3673 Write 4 Done New Done true
9136 2675 Acked 3674 Read 10 Done New Done true
9137 2676 Acked 3675 Read 4 Done New Done true
9138 2677 Acked 3676 Write 2 Done New Done true
9139 2678 Acked 3677 Read 9 Done New Done true
9140 2679 Acked 3678 Read 7 Done New Done true
9141 2680 Acked 3679 Write 3 Done New Done true
9142 2681 Acked 3680 Write 4 Done New Done true
9143 2682 Acked 3681 Read 5 Done New Done true
9144 2683 Acked 3682 Write 5 Done New Done true
9145 2684 Acked 3683 Flush 0 Done New Done true
9146 2685 Acked 3684 Read 4 Done New Done true
9147 2686 Acked 3685 Write 6 Done New Done true
9148 2687 Acked 3686 Write 9 Done New Done true
9149 2688 Acked 3687 Write 6 Done New Done true
9150 2689 Acked 3688 Read 10 Done New Done true
9151 2690 Acked 3689 Flush 0 Done New Done true
9152 2691 Acked 3690 Read 9 Done New Done true
9153 2692 Acked 3691 Write 10 Done New Done true
9154 2693 Acked 3692 Read 8 Done New Done true
9155 2694 Acked 3693 Flush 0 Done New Done true
9156 2695 Acked 3694 Flush 0 Done New Done true
9157 2696 Acked 3695 Read 10 Done New Done true
9158 2697 Acked 3696 Read 4 Done New Done true
9159 2698 Acked 3697 Read 3 Done New Done true
9160 2699 Acked 3698 Write 7 Done New Done true
9161 2700 Acked 3699 Read 8 Done New Done true
9162 2701 Acked 3700 Write 3 Done New Done true
9163 2702 Acked 3701 Read 6 Done New Done true
9164 2703 Acked 3702 Read 10 Done New Done true
9165 2704 Acked 3703 Write 4 Done New Done true
9166 2705 Acked 3704 Read 2 Done New Done true
9167 2706 Acked 3705 Write 2 Done New Done true
9168 2707 Acked 3706 Read 2 Done New Done true
9169 2708 Acked 3707 Read 7 Done New Done true
9170 2709 Acked 3708 Read 3 Done New Done true
9171 2710 Acked 3709 Read 8 Done New Done true
9172 2711 Acked 3710 Read 3 Done New Done true
9173 2712 Acked 3711 Read 1 Done New Done true
9174 2713 Acked 3712 Read 2 Done New Done true
9175 2714 Acked 3713 Flush 0 Done New Done true
9176 STATES DS:0 DS:1 DS:2 TOTAL
9177 New 0 301 0 301
9178 Sent 0 0 0 0
9179 Done 301 0 301 602
9180 Skipped 0 0 0 0
9181 Error 0 0 0 0
9182 Last Flush: 3713 3412 3713
9183 Downstairs last five completed: 3412 3411 3410 3409 3408
9184 Upstairs last five completed: 2714 2713 2712 2711 2710
9185 CLIENT: Up:0 ds:301 act:2
91862024-02-12T01:36:17.631ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
91872024-02-12T01:36:17.632ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
91882024-02-12T01:36:17.632ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
91892024-02-12T01:36:17.632ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
91902024-02-12T01:36:17.632ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
91912024-02-12T01:36:17.632ZINFOcrucible: [1] client re-new 301 jobs since flush 3412 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
91922024-02-12T01:36:17.632ZINFOcrucible: client task is exiting = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
9193 [8820] got start action
9194 [8820] I am going to restart
9195 Starting downstairs at port 8820
9196 Make output file at "/tmp/dsc/downstairs-8820.txt"
9197 [8820][1] reports Starting
9198 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1173)
9199 [8820][1] reports Running
92002024-02-12T01:36:18.634ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
92012024-02-12T01:36:18.634ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
92022024-02-12T01:36:18.634ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
92032024-02-12T01:36:18.634ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
92042024-02-12T01:36:18.634ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
92052024-02-12T01:36:18.634ZINFOcrucible: [1] client re-new 301 jobs since flush 3412 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
92062024-02-12T01:36:18.634ZINFOcrucible: client task is exiting = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
92072024-02-12T01:36:19.635ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
92082024-02-12T01:36:19.635ZINFOcrucible: ds_connection connected = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
92092024-02-12T01:36:19.637ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
92102024-02-12T01:36:19.637ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
92112024-02-12T01:36:19.637ZINFOcrucible: send last flush ID to this DS: 3412 = downstairs client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
92122024-02-12T01:36:19.637ZINFOcrucible: Replied this last flush ID: 3412 = downstairs client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
92132024-02-12T01:36:19.637ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
92142024-02-12T01:36:19.637ZINFOcrucible: [1] Transition from Offline to Replay = downstairs client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
92152024-02-12T01:36:19.637ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
92162024-02-12T01:36:19.637ZINFOcrucible: [1] Transition from Replay to Active = downstairs client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
9217 ----------------------------------------------------------------
9218 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
9219 Downstairs last five completed: 3713 3712 3711 3710 3709
9220 Upstairs last five completed: 2714 2713 2712 2711 2710
9221 CLIENT: Up:0 ds:0 act:3
9222 Replay: All jobs finished, all DS active.
9223 Main task has work to do, go find it
9224 got dsc Stop(0)
9225 stop 0
9226 Feb 12 01:36:20.977 INFO request completed, latency_us: 423, response_code: 204, uri: /stop/cid/0, method: POST, req_id: 9a240384-a1f5-48cc-860d-d10064f46c1e, remote_addr: 127[8810] Got stop action so:false kr:false
9227 .0.0.1:60456, local_addr: 127.0.0.1:9998
92282024-02-12T01:36:20.977ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
92292024-02-12T01:36:20.977ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
92302024-02-12T01:36:20.977ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
92312024-02-12T01:36:20.978ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
92322024-02-12T01:36:20.978ZINFOcrucible: [0] client re-new 0 jobs since flush 3713 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
9233 Feb 12 01:36:20.978 INFO request completed{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-12T01:36:20.978057689Z","hostname":"ip-10-150-1-104.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"e1773b3e-b36f-4080-8b13-c0830f0d072e"}
9234 , latency_us: 138, response_code: 200, uri: /state/cid/0, method: GET, req_id: 84088dc8-bd26-42fc-ad3c-4929cabbe4db, remote_addr: 127.0.0.1:60456, local_addr: 127.0.0.1:9998
9235 [8810] Exited with: ExitStatus(unix_wait_status(9))
9236 [8810][0] reports Exit
92372024-02-12T01:36:21.978ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
92382024-02-12T01:36:21.978ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
92392024-02-12T01:36:21.978ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
92402024-02-12T01:36:21.978ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
92412024-02-12T01:36:21.978ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
92422024-02-12T01:36:21.978ZINFOcrucible: [0] client re-new 0 jobs since flush 3713 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
92432024-02-12T01:36:21.978ZINFOcrucible: client task is exiting = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
92442024-02-12T01:36:22.979ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
92452024-02-12T01:36:22.979ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
92462024-02-12T01:36:22.979ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
92472024-02-12T01:36:22.979ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
92482024-02-12T01:36:22.979ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
92492024-02-12T01:36:22.979ZINFOcrucible: [0] client re-new 0 jobs since flush 3713 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
92502024-02-12T01:36:22.979ZINFOcrucible: client task is exiting = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
92512024-02-12T01:36:23.980ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
92522024-02-12T01:36:23.980ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
92532024-02-12T01:36:23.980ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
92542024-02-12T01:36:23.980ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
92552024-02-12T01:36:23.980ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
92562024-02-12T01:36:23.980ZINFOcrucible: [0] client re-new 0 jobs since flush 3713 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
92572024-02-12T01:36:23.980ZINFOcrucible: client task is exiting = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
9258 Feb 12 01:36:24.980 INFO request completed, latency_us: 222, response_code: 200, uri: /state/cid/0, method: GET, req_id: f94ded3b-1154-47a3-9137-c36c44e672cf, remote_addr: 127.0.0.1:60456, local_addr: 127.0.0.1:9998
9259 001/300 Flush
92602024-02-12T01:36:24.981ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
92612024-02-12T01:36:24.981ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
92622024-02-12T01:36:24.981ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
92632024-02-12T01:36:24.981ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
92642024-02-12T01:36:24.981ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
92652024-02-12T01:36:24.981ZINFOcrucible: [0] client re-new 1 jobs since flush 3713 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
92662024-02-12T01:36:24.981ZINFOcrucible: client task is exiting = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
9267 002/300 Write block 3673 len 36864 data: 3 3 3 3 2 2 2 2 4
9268 003/300 Read block 4098 len 36864
9269 004/300 Read block 347 len 32768
9270 005/300 Write block 300 len 8192 data: 3 3
9271 006/300 Read block 3003 len 12288
9272 007/300 Read block 4747 len 36864
9273 008/300 Write block 4531 len 36864 data: 2 3 3 3 3 3 4 4 4
9274 009/300 Read block 2940 len 36864
9275 010/300 Write block 1223 len 20480 data: 3 3 3 2 2
9276 011/300 Read block 2375 len 32768
9277 012/300 Write block 3556 len 36864 data: 2 2 2 2 2 3 3 3 3
9278 013/300 Read block 654 len 4096
9279 014/300 Read block 1612 len 12288
9280 015/300 Read block 598 len 24576
9281 016/300 Write block 4173 len 32768 data: 3 4 2 2 2 2 2 3
9282 017/300 Write block 1919 len 20480 data: 4 4 3 3 3
9283 018/300 Write block 635 len 32768 data: 3 4 4 4 4 4 4 3
9284 019/300 Read block 1094 len 12288
9285 020/300 Write block 124 len 28672 data: 4 4 4 4 3 3 2
9286 021/300 Write block 4920 len 4096 data: 3
9287 022/300 Flush
9288 023/300 Write block 4852 len 8192 data: 2 2
9289 024/300 Write block 4294 len 32768 data: 2 2 2 2 2 3 3 3
9290 025/300 Write block 1280 len 8192 data: 3 3
9291 026/300 Write block 2376 len 40960 data: 4 4 4 3 3 3 3 4 4 4
9292 027/300 Read block 2851 len 36864
9293 028/300 Read block 1369 len 40960
9294 029/300 Write block 2885 len 20480 data: 3 2 2 2 2
9295 030/300 Read block 1862 len 12288
9296 031/300 Flush
9297 032/300 Write block 4938 len 12288 data: 2 2 3
9298 033/300 Write block 386 len 28672 data: 3 3 2 2 2 2 3
9299 034/300 Read block 4671 len 8192
9300 035/300 Read block 4803 len 16384
9301 036/300 Write block 164 len 16384 data: 4 4 3 4
9302 037/300 Flush
9303 038/300 Write block 90 len 40960 data: 5 4 4 4 4 3 3 2 2 5
9304 039/300 Read block 1885 len 32768
9305 040/300 Read block 3293 len 40960
9306 041/300 Read block 4752 len 28672
9307 042/300 Read block 1380 len 28672
9308 043/300 Write block 3457 len 16384 data: 3 3 2 2
9309 044/300 Write block 2825 len 20480 data: 5 4 3 3 6
9310 045/300 Read block 3275 len 20480
9311 046/300 Flush
9312 047/300 Write block 3103 len 12288 data: 4 5 5
9313 048/300 Write block 973 len 36864 data: 3 3 3 3 3 3 3 3 3
9314 049/300 Read block 1761 len 12288
9315 050/300 Read block 4531 len 12288
9316 051/300 Read block 343 len 36864
9317 052/300 Read block 3603 len 12288
9318 053/300 Write block 1615 len 12288 data: 7 5 5
9319 054/300 Read block 757 len 4096
9320 055/300 Write block 2120 len 16384 data: 3 3 3 3
9321 056/300 Read block 4066 len 36864
9322 057/300 Read block 1842 len 8192
9323 058/300 Write block 341 len 12288 data: 2 2 2
9324 059/300 Write block 4169 len 24576 data: 4 4 3 3 4 5
9325 060/300 Read block 1467 len 12288
9326 061/300 Write block 2594 len 40960 data: 4 3 4 4 5 4 4 4 4 4
9327 062/300 Read block 812 len 32768
9328 063/300 Write block 4071 len 32768 data: 5 6 6 4 4 4 4 3
9329 064/300 Flush
9330 065/300 Read block 4027 len 12288
9331 066/300 Write block 2063 len 12288 data: 4 4 4
9332 067/300 Write block 4765 len 4096 data: 3
9333 068/300 Write block 1599 len 4096 data: 3
9334 069/300 Read block 4583 len 28672
9335 070/300 Flush
9336 071/300 Write block 2747 len 24576 data: 3 3 3 4 3 4
9337 072/300 Write block 4459 len 8192 data: 2 2
9338 073/300 Write block 2012 len 40960 data: 3 3 3 3 3 2 2 2 2 2
9339 074/300 Read block 2148 len 36864
9340 075/300 Read block 3399 len 16384
9341 076/300 Read block 1004 len 24576
9342 077/300 Write block 4321 len 28672 data: 4 2 2 2 3 3 3
9343 078/300 Write block 1637 len 40960 data: 3 3 3 3 2 2 2 2 3 2
9344 079/300 Write block 4841 len 36864 data: 2 2 2 2 2 2 2 2 2
9345 080/300 Write block 1948 len 24576 data: 2 2 2 2 2 2
9346 081/300 Write block 3810 len 40960 data: 4 5 6 5 5 5 4 2 2 2
9347 082/300 Read block 3444 len 16384
9348 083/300 Read block 2034 len 4096
9349 084/300 Read block 513 len 4096
9350 085/300 Read block 1050 len 20480
9351 086/300 Read block 1183 len 16384
9352 087/300 Write block 2625 len 20480 data: 4 2 2 3 3
9353 088/300 Read block 2520 len 8192
9354 089/300 Read block 3284 len 24576
9355 090/300 Write block 4485 len 40960 data: 4 4 3 3 3 3 3 3 3 3
9356 091/300 Read block 747 len 36864
9357 092/300 Flush
9358 093/300 Flush
9359 094/300 Read block 3019 len 32768
9360 095/300 Flush
9361 096/300 Write block 824 len 8192 data: 3 2
9362 097/300 Flush
9363 098/300 Read block 1947 len 12288
9364 099/300 Read block 4769 len 28672
9365 100/300 Read block 3409 len 12288
9366 101/300 Write block 2485 len 36864 data: 4 4 4 4 4 4 4 2 2
9367 102/300 Read block 3509 len 12288
9368 103/300 Read block 246 len 20480
9369 104/300 Read block 1242 len 12288
9370 105/300 Read block 4555 len 12288
9371 106/300 Read block 2384 len 12288
9372 107/300 Read block 3402 len 12288
9373 108/300 Read block 3713 len 12288
9374 109/300 Write block 1085 len 20480 data: 3 4 3 3 3
9375 110/300 Read block 472 len 40960
9376 111/300 Write block 3369 len 8192 data: 3 2
9377 112/300 Read block 1811 len 12288
9378 113/300 Write block 2640 len 40960 data: 3 3 4 4 4 4 4 4 4 6
9379 114/300 Write block 796 len 28672 data: 5 5 5 6 5 5 5
9380 115/300 Read block 186 len 40960
9381 116/300 Write block 2663 len 36864 data: 3 3 3 2 2 2 2 2 2
9382 117/300 Read block 1090 len 4096
9383 118/300 Read block 4049 len 12288
9384 119/300 Write block 2323 len 12288 data: 3 3 2
9385 120/300 Write block 4033 len 32768 data: 2 3 2 2 2 3 4 4
9386 121/300 Write block 1432 len 12288 data: 3 3 3
9387 122/300 Write block 411 len 16384 data: 5 4 3 3
9388 123/300 Read block 1193 len 16384
9389 124/300 Read block 3610 len 4096
9390 125/300 Read block 3778 len 8192
9391 126/300 Read block 2816 len 12288
9392 127/300 Write block 1825 len 32768 data: 3 3 4 4 5 4 4 4
9393 128/300 Write block 2425 len 28672 data: 6 6 6 4 4 4 2
9394 129/300 Write block 3189 len 24576 data: 4 4 5 5 5 4
9395 130/300 Write block 2012 len 40960 data: 4 4 4 4 4 3 3 3 3 3
9396 131/300 Write block 1140 len 28672 data: 3 3 5 5 5 5 5
9397 132/300 Read block 2546 len 12288
9398 133/300 Write block 3092 len 4096 data: 2
9399 134/300 Read block 2659 len 8192
9400 135/300 Flush
9401 136/300 Write block 4365 len 40960 data: 6 4 3 2 2 3 3 4 3 3
9402 137/300 Write block 3811 len 12288 data: 6 7 6
9403 138/300 Write block 4308 len 32768 data: 2 2 2 2 2 2 2 3
9404 139/300 Flush
9405 140/300 Write block 2349 len 8192 data: 4 5
9406 141/300 Write block 4990 len 12288 data: 2 2 3
9407 142/300 Write block 2806 len 36864 data: 8 9 7 8 7 6 5 5 4
9408 143/300 Flush
9409 144/300 Read block 2538 len 36864
9410 145/300 Write block 2348 len 28672 data: 2 5 6 5 5 4 4
9411 146/300 Read block 3249 len 24576
9412 147/300 Write block 1500 len 40960 data: 3 3 3 3 3 3 3 2 2 2
9413 148/300 Write block 2213 len 20480 data: 3 4 4 4 4
9414 149/300 Write block 3372 len 24576 data: 3 3 5 5 6 6
9415 150/300 Flush
9416 151/300 Read block 2131 len 12288
9417 152/300 Read block 2540 len 20480
9418 153/300 Write block 421 len 8192 data: 4 3
9419 154/300 Write block 933 len 24576 data: 3 3 4 4 5 4
9420 155/300 Write block 1885 len 16384 data: 2 2 2 4
9421 156/300 Flush
9422 157/300 Write block 336 len 8192 data: 3 2
9423 158/300 Write block 4814 len 20480 data: 2 2 3 3 3
9424 159/300 Flush
9425 160/300 Write block 4387 len 20480 data: 2 2 3 3 3
9426 161/300 Read block 4390 len 20480
9427 162/300 Write block 3600 len 36864 data: 3 2 3 4 4 4 4 4 3
9428 163/300 Read block 2851 len 32768
9429 164/300 Write block 3315 len 12288 data: 4 3 3
9430 165/300 Read block 2824 len 16384
9431 166/300 Write block 1985 len 40960 data: 2 2 2 2 2 2 2 2 3 3
9432 167/300 Read block 90 len 12288
9433 168/300 Write block 356 len 16384 data: 5 5 5 4
9434 169/300 Read block 555 len 16384
9435 170/300 Read block 2894 len 28672
9436 171/300 Read block 1076 len 8192
9437 172/300 Read block 1909 len 24576
9438 173/300 Flush
9439 174/300 Write block 930 len 16384 data: 4 3 3 4
9440 175/300 Write block 3237 len 16384 data: 4 4 4 3
9441 176/300 Write block 95 len 8192 data: 4 4
9442 177/300 Write block 870 len 36864 data: 2 2 2 2 2 3 3 3 4
9443 178/300 Flush
9444 179/300 Flush
9445 180/300 Read block 673 len 32768
9446 181/300 Read block 995 len 28672
9447 182/300 Read block 2780 len 12288
9448 183/300 Read block 458 len 24576
9449 184/300 Write block 4268 len 28672 data: 5 4 4 4 5 5 4
9450 185/300 Write block 2719 len 4096 data: 3
9451 186/300 Read block 568 len 32768
9452 187/300 Read block 1155 len 28672
9453 188/300 Write block 4713 len 8192 data: 4 4
9454 189/300 Write block 2066 len 24576 data: 4 5 5 4 4 4
9455 190/300 Read block 1270 len 8192
9456 191/300 Read block 997 len 12288
9457 192/300 Read block 97 len 32768
9458 193/300 Write block 1051 len 8192 data: 5 4
9459 194/300 Write block 803 len 28672 data: 5 4 3 3 3 3 3
9460 195/300 Read block 4447 len 16384
9461 196/300 Read block 4089 len 28672
9462 197/300 Write block 1402 len 40960 data: 2 2 3 3 3 4 4 3 3 3
9463 198/300 Write block 1625 len 8192 data: 2 2
9464 199/300 Read block 1092 len 20480
9465 200/300 Read block 1393 len 12288
9466 201/300 Write block 3056 len 4096 data: 2
9467 202/300 Read block 2847 len 12288
9468 203/300 Write block 4832 len 28672 data: 2 2 2 2 2 3 3
9469 204/300 Write block 1243 len 4096 data: 2
9470 205/300 Write block 2213 len 40960 data: 4 5 5 5 5 4 4 3 3 3
9471 206/300 Read block 3380 len 28672
9472 207/300 Read block 213 len 4096
9473 208/300 Read block 4044 len 12288
9474 209/300 Write block 2739 len 4096 data: 4
9475 210/300 Read block 1099 len 8192
9476 211/300 Read block 4510 len 36864
9477 212/300 Read block 1520 len 28672
9478 213/300 Write block 4248 len 24576 data: 2 3 3 3 4 4
9479 214/300 Read block 1947 len 8192
9480 215/300 Read block 827 len 12288
9481 216/300 Flush
9482 217/300 Flush
9483 218/300 Read block 1441 len 8192
9484 219/300 Write block 2728 len 20480 data: 3 3 3 3 2
9485 220/300 Read block 4255 len 28672
9486 221/300 Read block 3624 len 12288
9487 222/300 Read block 4224 len 20480
9488 223/300 Read block 630 len 16384
9489 224/300 Flush
9490 225/300 Write block 1328 len 36864 data: 4 6 6 5 4 5 6 6 4
9491 226/300 Write block 1381 len 20480 data: 5 4 3 3 4
9492 227/300 Write block 292 len 36864 data: 2 2 2 2 2 3 3 3 4
9493 228/300 Flush
9494 229/300 Read block 4934 len 8192
9495 230/300 Write block 1102 len 40960 data: 2 2 3 3 3 4 3 3 3 3
9496 231/300 Write block 1619 len 40960 data: 5 5 5 5 4 3 3 3 2 2
9497 232/300 Read block 1858 len 12288
9498 233/300 Write block 4260 len 40960 data: 6 4 3 4 6 6 5 5 6 5
9499 234/300 Write block 4509 len 12288 data: 2 2 2
9500 235/300 Write block 4798 len 16384 data: 3 2 3 3
9501 236/300 Write block 797 len 8192 data: 6 6
9502 237/300 Read block 3622 len 16384
9503 238/300 Read block 3650 len 8192
9504 239/300 Read block 4098 len 8192
9505 240/300 Read block 3358 len 16384
9506 241/300 Write block 2299 len 16384 data: 3 3 3 3
9507 242/300 Write block 3177 len 20480 data: 3 3 3 4 5
9508 243/300 Flush
9509 244/300 Write block 4319 len 8192 data: 5 4
9510 245/300 Write block 435 len 4096 data: 3
9511 246/300 Write block 2171 len 8192 data: 4 4
9512 247/300 Write block 2624 len 16384 data: 3 5 3 3
9513 248/300 Write block 2760 len 20480 data: 6 6 5 5 3
9514 249/300 Write block 2978 len 28672 data: 4 4 4 4 3 3 4
9515 250/300 Flush
9516 251/300 Read block 4624 len 16384
9517 252/300 Write block 2472 len 28672 data: 6 6 5 4 4 4 5
9518 253/300 Write block 700 len 28672 data: 3 2 2 2 2 2 2
9519 254/300 Read block 1130 len 16384
9520 255/300 Read block 605 len 24576
9521 256/300 Read block 3576 len 40960
9522 257/300 Flush
9523 258/300 Write block 1167 len 40960 data: 4 4 3 4 5 5 6 6 4 3
9524 259/300 Write block 3292 len 16384 data: 2 2 3 3
9525 260/300 Read block 4517 len 4096
9526 261/300 Read block 4728 len 28672
9527 262/300 Read block 1337 len 8192
9528 263/300 Write block 2444 len 8192 data: 4 3
9529 264/300 Read block 1414 len 20480
9530 265/300 Write block 1803 len 16384 data: 3 3 3 4
9531 266/300 Read block 669 len 20480
9532 267/300 Read block 962 len 36864
9533 268/300 Read block 1219 len 28672
9534 269/300 Read block 2152 len 8192
9535 270/300 Flush
9536 271/300 Write block 1925 len 12288 data: 2 2 2
9537 272/300 Read block 4839 len 36864
9538 273/300 Write block 2440 len 40960 data: 2 3 3 3 5 4 4 3 4 4
9539 274/300 Write block 3138 len 24576 data: 3 3 3 3 4 4
9540 275/300 Read block 4287 len 20480
9541 276/300 Flush
9542 277/300 Write block 2253 len 40960 data: 6 6 7 7 5 3 3 3 3 3
9543 278/300 Read block 1647 len 36864
9544 279/300 Flush
9545 280/300 Write block 4509 len 16384 data: 3 3 3 2
9546 281/300 Write block 2 len 16384 data: 2 2 2 2
9547 282/300 Write block 3232 len 28672 data: 4 5 4 4 4 5 5
9548 283/300 Read block 3728 len 8192
9549 284/300 Read block 1458 len 20480
9550 285/300 Write block 3440 len 12288 data: 2 2 2
9551 286/300 Read block 3276 len 40960
9552 287/300 Read block 1882 len 40960
9553 288/300 Write block 2958 len 36864 data: 4 5 5 4 4 3 3 2 2
9554 289/300 Read block 1690 len 4096
9555 290/300 Read block 1677 len 8192
9556 291/300 Read block 4607 len 20480
9557 292/300 Write block 2455 len 16384 data: 2 2 2 2
9558 293/300 Flush
9559 294/300 Read block 3622 len 24576
9560 295/300 Read block 838 len 12288
9561 296/300 Write block 2269 len 32768 data: 3 2 2 2 2 2 2 2
9562 297/300 Write block 2022 len 8192 data: 4 4
9563 298/300 Read block 4808 len 20480
9564 299/300 Read block 2692 len 12288
9565 300/300 Write block 4780 len 40960 data: 3 3 2 2 2 3 3 3 3 3
9566 Main task has work to do, go find it
9567 got dsc Start(0)
9568 start 0
9569 Feb 12 01:36:25.327 INFO request completed, latency_us: 261, response_code: 204, uri: /start/cid/0, method: POST, req_id: 20869529-46e2-4785-867c-b7a3b2649e45, remote_addr: 127.0.0.1:60456, local_addr: 127.0.0.1:9998
9570 Replay: started 0, returned:Ok(())
9571 ----------------------------------------------------------------
9572 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:300
9573 States: Offline Active Active
9574 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
9575 2715 Acked 3714 Flush 0 New Done Done true
9576 2716 Acked 3715 Write 9 New Done Done false
9577 2717 Acked 3716 Read 9 New Done Done false
9578 2718 Acked 3717 Read 8 New Done Done false
9579 2719 Acked 3718 Write 2 New Done Done false
9580 2720 Acked 3719 Read 3 New Done Done false
9581 2721 Acked 3720 Read 9 New Done Done false
9582 2722 Acked 3721 Write 9 New Done Done false
9583 2723 Acked 3722 Read 9 New Done Done false
9584 2724 Acked 3723 Write 5 New Done Done false
9585 2725 Acked 3724 Read 8 New Done Done false
9586 2726 Acked 3725 Write 9 New Done Done false
9587 2727 Acked 3726 Read 1 New Done Done false
9588 2728 Acked 3727 Read 3 New Done Done false
9589 2729 Acked 3728 Read 6 New Done Done false
9590 2730 Acked 3729 Write 8 New Done Done false
9591 2731 Acked 3730 Write 5 New Done Done false
9592 2732 Acked 3731 Write 8 New Done Done false
9593 2733 Acked 3732 Read 3 New Done Done false
9594 2734 Acked 3733 Write 7 New Done Done false
9595 2735 Acked 3734 Write 1 New Done Done false
9596 2736 Acked 3735 Flush 0 New Done Done false
9597 2737 Acked 3736 Write 2 New Done Done false
9598 2738 Acked 3737 Write 8 New Done Done false
9599 2739 Acked 3738 Write 2 New Done Done false
9600 2740 Acked 3739 Write 10 New Done Done false
9601 2741 Acked 3740 Read 9 New Done Done false
9602 2742 Acked 3741 Read 10 New Done Done false
9603 2743 Acked 3742 Write 5 New Done Done false
9604 2744 Acked 3743 Read 3 New Done Done false
9605 2745 Acked 3744 Flush 0 New Done Done false
9606 2746 Acked 3745 Write 3 New Done Done false
9607 2747 Acked 3746 Write 7 New Done Done false
9608 2748 Acked 3747 Read 2 New Done Done false
9609 2749 Acked 3748 Read 4 New Done Done false
9610 2750 Acked 3749 Write 4 New Done Done false
9611 2751 Acked 3750 Flush 0 New Done Done false
9612 2752 Acked 3751 Write 10 New Done Done false
9613 2753 Acked 3752 Read 8 New Done Done false
9614 2754 Acked 3753 Read 10 New Done Done false
9615 2755 Acked 3754 Read 7 New Done Done false
9616 2756 Acked 3755 Read 7 New Done Done false
9617 2757 Acked 3756 Write 4 New Done Done false
9618 2758 Acked 3757 Write 5 New Done Done false
9619 2759 Acked 3758 Read 5 New Done Done false
9620 2760 Acked 3759 Flush 0 New Done Done false
9621 2761 Acked 3760 Write 3 New Done Done false
9622 2762 Acked 3761 Write 9 New Done Done false
9623 2763 Acked 3762 Read 3 New Done Done false
9624 2764 Acked 3763 Read 3 New Done Done false
9625 2765 Acked 3764 Read 9 New Done Done false
9626 2766 Acked 3765 Read 3 New Done Done false
9627 2767 Acked 3766 Write 3 New Done Done false
9628 2768 Acked 3767 Read 1 New Done Done false
9629 2769 Acked 3768 Write 4 New Done Done false
9630 2770 Acked 3769 Read 9 New Done Done false
9631 2771 Acked 3770 Read 2 New Done Done false
9632 2772 Acked 3771 Write 3 New Done Done false
9633 2773 Acked 3772 Write 6 New Done Done false
9634 2774 Acked 3773 Read 3 New Done Done false
9635 2775 Acked 3774 Write 10 New Done Done false
9636 2776 Acked 3775 Read 8 New Done Done false
9637 2777 Acked 3776 Write 8 New Done Done false
9638 2778 Acked 3777 Flush 0 New Done Done false
9639 2779 Acked 3778 Read 3 New Done Done false
9640 2780 Acked 3779 Write 3 New Done Done false
9641 2781 Acked 3780 Write 1 New Done Done false
9642 2782 Acked 3781 Write 1 New Done Done false
9643 2783 Acked 3782 Read 7 New Done Done false
9644 2784 Acked 3783 Flush 0 New Done Done false
9645 2785 Acked 3784 Write 6 New Done Done false
9646 2786 Acked 3785 Write 2 New Done Done false
9647 2787 Acked 3786 Write 10 New Done Done false
9648 2788 Acked 3787 Read 9 New Done Done false
9649 2789 Acked 3788 Read 4 New Done Done false
9650 2790 Acked 3789 Read 6 New Done Done false
9651 2791 Acked 3790 Write 7 New Done Done false
9652 2792 Acked 3791 Write 10 New Done Done false
9653 2793 Acked 3792 Write 9 New Done Done false
9654 2794 Acked 3793 Write 6 New Done Done false
9655 2795 Acked 3794 Write 10 New Done Done false
9656 2796 Acked 3795 Read 4 New Done Done false
9657 2797 Acked 3796 Read 1 New Done Done false
9658 2798 Acked 3797 Read 1 New Done Done false
9659 2799 Acked 3798 Read 5 New Done Done false
9660 2800 Acked 3799 Read 4 New Done Done false
9661 2801 Acked 3800 Write 5 New Done Done false
9662 2802 Acked 3801 Read 2 New Done Done false
9663 2803 Acked 3802 Read 6 New Done Done false
9664 2804 Acked 3803 Write 10 New Done Done false
9665 2805 Acked 3804 Read 9 New Done Done false
9666 2806 Acked 3805 Flush 0 New Done Done false
9667 2807 Acked 3806 Flush 0 New Done Done false
9668 2808 Acked 3807 Read 8 New Done Done false
9669 2809 Acked 3808 Flush 0 New Done Done false
9670 2810 Acked 3809 Write 2 New Done Done false
9671 2811 Acked 3810 Flush 0 New Done Done false
9672 2812 Acked 3811 Read 3 New Done Done false
9673 2813 Acked 3812 Read 7 New Done Done false
9674 2814 Acked 3813 Read 3 New Done Done false
9675 2815 Acked 3814 Write 9 New Done Done false
9676 2816 Acked 3815 Read 3 New Done Done false
9677 2817 Acked 3816 Read 5 New Done Done false
9678 2818 Acked 3817 Read 3 New Done Done false
9679 2819 Acked 3818 Read 3 New Done Done false
9680 2820 Acked 3819 Read 3 New Done Done false
9681 2821 Acked 3820 Read 3 New Done Done false
9682 2822 Acked 3821 Read 3 New Done Done false
9683 2823 Acked 3822 Write 5 New Done Done false
9684 2824 Acked 3823 Read 10 New Done Done false
9685 2825 Acked 3824 Write 2 New Done Done false
9686 2826 Acked 3825 Read 3 New Done Done false
9687 2827 Acked 3826 Write 10 New Done Done false
9688 2828 Acked 3827 Write 7 New Done Done false
9689 2829 Acked 3828 Read 10 New Done Done false
9690 2830 Acked 3829 Write 9 New Done Done false
9691 2831 Acked 3830 Read 1 New Done Done false
9692 2832 Acked 3831 Read 3 New Done Done false
9693 2833 Acked 3832 Write 3 New Done Done false
9694 2834 Acked 3833 Write 8 New Done Done false
9695 2835 Acked 3834 Write 3 New Done Done false
9696 2836 Acked 3835 Write 4 New Done Done false
9697 2837 Acked 3836 Read 4 New Done Done false
9698 2838 Acked 3837 Read 1 New Done Done false
9699 2839 Acked 3838 Read 2 New Done Done false
9700 2840 Acked 3839 Read 3 New Done Done false
9701 2841 Acked 3840 Write 8 New Done Done false
9702 2842 Acked 3841 Write 7 New Done Done false
9703 2843 Acked 3842 Write 6 New Done Done false
9704 2844 Acked 3843 Write 10 New Done Done false
9705 2845 Acked 3844 Write 7 New Done Done false
9706 2846 Acked 3845 Read 3 New Done Done false
9707 2847 Acked 3846 Write 1 New Done Done false
9708 2848 Acked 3847 Read 2 New Done Done false
9709 2849 Acked 3848 Flush 0 New Done Done false
9710 2850 Acked 3849 Write 10 New Done Done false
9711 2851 Acked 3850 Write 3 New Done Done false
9712 2852 Acked 3851 Write 8 New Done Done false
9713 2853 Acked 3852 Flush 0 New Done Done false
9714 2854 Acked 3853 Write 2 New Done Done false
9715 2855 Acked 3854 Write 3 New Done Done false
9716 2856 Acked 3855 Write 9 New Done Done false
9717 2857 Acked 3856 Flush 0 New Done Done false
9718 2858 Acked 3857 Read 9 New Done Done false
9719 2859 Acked 3858 Write 7 New Done Done false
9720 2860 Acked 3859 Read 6 New Done Done false
9721 2861 Acked 3860 Write 10 New Done Done false
9722 2862 Acked 3861 Write 5 New Done Done false
9723 2863 Acked 3862 Write 6 New Done Done false
9724 2864 Acked 3863 Flush 0 New Done Done false
9725 2865 Acked 3864 Read 3 New Done Done false
9726 2866 Acked 3865 Read 5 New Done Done false
9727 2867 Acked 3866 Write 2 New Done Done false
9728 2868 Acked 3867 Write 6 New Done Done false
9729 2869 Acked 3868 Write 4 New Done Done false
9730 2870 Acked 3869 Flush 0 New Done Done false
9731 2871 Acked 3870 Write 2 New Done Done false
9732 2872 Acked 3871 Write 5 New Done Done false
9733 2873 Acked 3872 Flush 0 New Done Done false
9734 2874 Acked 3873 Write 5 New Done Done false
9735 2875 Acked 3874 Read 5 New Done Done false
9736 2876 Acked 3875 Write 9 New Done Done false
9737 2877 Acked 3876 Read 8 New Done Done false
9738 2878 Acked 3877 Write 3 New Done Done false
9739 2879 Acked 3878 Read 4 New Done Done false
9740 2880 Acked 3879 Write 10 New Done Done false
9741 2881 Acked 3880 Read 3 New Done Done false
9742 2882 Acked 3881 Write 4 New Done Done false
9743 2883 Acked 3882 Read 4 New Done Done false
9744 2884 Acked 3883 Read 7 New Done Done false
9745 2885 Acked 3884 Read 2 New Done Done false
9746 2886 Acked 3885 Read 6 New Done Done false
9747 2887 Acked 3886 Flush 0 New Done Done false
9748 2888 Acked 3887 Write 4 New Done Done false
9749 2889 Acked 3888 Write 4 New Done Done false
9750 2890 Acked 3889 Write 2 New Done Done false
9751 2891 Acked 3890 Write 9 New Done Done false
9752 2892 Acked 3891 Flush 0 New Done Done false
9753 2893 Acked 3892 Flush 0 New Done Done false
9754 2894 Acked 3893 Read 8 New Done Done false
9755 2895 Acked 3894 Read 7 New Done Done false
9756 2896 Acked 3895 Read 3 New Done Done false
9757 2897 Acked 3896 Read 6 New Done Done false
9758 2898 Acked 3897 Write 7 New Done Done false
9759 2899 Acked 3898 Write 1 New Done Done false
9760 2900 Acked 3899 Read 8 New Done Done false
9761 2901 Acked 3900 Read 7 New Done Done false
9762 2902 Acked 3901 Write 2 New Done Done false
9763 2903 Acked 3902 Write 6 New Done Done false
9764 2904 Acked 3903 Read 2 New Done Done false
9765 2905 Acked 3904 Read 3 New Done Done false
9766 2906 Acked 3905 Read 8 New Done Done false
9767 2907 Acked 3906 Write 2 New Done Done false
9768 2908 Acked 3907 Write 7 New Done Done false
9769 2909 Acked 3908 Read 4 New Done Done false
9770 2910 Acked 3909 Read 7 New Done Done false
9771 2911 Acked 3910 Write 10 New Done Done false
9772 2912 Acked 3911 Write 2 New Done Done false
9773 2913 Acked 3912 Read 5 New Done Done false
9774 2914 Acked 3913 Read 3 New Done Done false
9775 2915 Acked 3914 Write 1 New Done Done false
9776 2916 Acked 3915 Read 3 New Done Done false
9777 2917 Acked 3916 Write 7 New Done Done false
9778 2918 Acked 3917 Write 1 New Done Done false
9779 2919 Acked 3918 Write 10 New Done Done false
9780 2920 Acked 3919 Read 7 New Done Done false
9781 2921 Acked 3920 Read 1 New Done Done false
9782 2922 Acked 3921 Read 3 New Done Done false
9783 2923 Acked 3922 Write 1 New Done Done false
9784 2924 Acked 3923 Read 2 New Done Done false
9785 2925 Acked 3924 Read 9 New Done Done false
9786 2926 Acked 3925 Read 7 New Done Done false
9787 2927 Acked 3926 Write 6 New Done Done false
9788 2928 Acked 3927 Read 2 New Done Done false
9789 2929 Acked 3928 Read 3 New Done Done false
9790 2930 Acked 3929 Flush 0 New Done Done false
9791 2931 Acked 3930 Flush 0 New Done Done false
9792 2932 Acked 3931 Read 2 New Done Done false
9793 2933 Acked 3932 Write 5 New Done Done false
9794 2934 Acked 3933 Read 7 New Done Done false
9795 2935 Acked 3934 Read 3 New Done Done false
9796 2936 Acked 3935 Read 5 New Done Done false
9797 2937 Acked 3936 Read 4 New Done Done false
9798 2938 Acked 3937 Flush 0 New Done Done false
9799 2939 Acked 3938 Write 9 New Done Done false
9800 2940 Acked 3939 Write 5 New Done Done false
9801 2941 Acked 3940 Write 9 New Done Done false
9802 2942 Acked 3941 Flush 0 New Done Done false
9803 2943 Acked 3942 Read 2 New Done Done false
9804 2944 Acked 3943 Write 10 New Done Done false
9805 2945 Acked 3944 Write 10 New Done Done false
9806 2946 Acked 3945 Read 3 New Done Done false
9807 2947 Acked 3946 Write 10 New Done Done false
9808 2948 Acked 3947 Write 3 New Done Done false
9809 2949 Acked 3948 Write 4 New Done Done false
9810 2950 Acked 3949 Write 2 New Done Done false
9811 2951 Acked 3950 Read 4 New Done Done false
9812 2952 Acked 3951 Read 2 New Done Done false
9813 2953 Acked 3952 Read 2 New Done Done false
9814 2954 Acked 3953 Read 4 New Done Done false
9815 2955 Acked 3954 Write 4 New Done Done false
9816 2956 Acked 3955 Write 5 New Done Done false
9817 2957 Acked 3956 Flush 0 New Done Done false
9818 2958 Acked 3957 Write 2 New Done Done false
9819 2959 Acked 3958 Write 1 New Done Done false
9820 2960 Acked 3959 Write 2 New Done Done false
9821 2961 Acked 3960 Write 4 New Done Done false
9822 2962 Acked 3961 Write 5 New Done Done false
9823 2963 Acked 3962 Write 7 New Done Done false
9824 2964 Acked 3963 Flush 0 New Done Done false
9825 2965 Acked 3964 Read 4 New Done Done false
9826 2966 Acked 3965 Write 7 New Done Done false
9827 2967 Acked 3966 Write 7 New Done Done false
9828 2968 Acked 3967 Read 4 New Done Done false
9829 2969 Acked 3968 Read 6 New Done Done false
9830 2970 Acked 3969 Read 10 New Done Done false
9831 2971 Acked 3970 Flush 0 New Done Done false
9832 2972 Acked 3971 Write 10 New Done Done false
9833 2973 Acked 3972 Write 4 New Done Done false
9834 2974 Acked 3973 Read 1 New Done Done false
9835 2975 Acked 3974 Read 7 New Done Done false
9836 2976 Acked 3975 Read 2 New Done Done false
9837 2977 Acked 3976 Write 2 New Done Done false
9838 2978 Acked 3977 Read 5 New Done Done false
9839 2979 Acked 3978 Write 4 New Done Done false
9840 2980 Acked 3979 Read 5 New Done Done false
9841 2981 Acked 3980 Read 9 New Done Done false
9842 2982 Acked 3981 Read 7 New Done Done false
9843 2983 Acked 3982 Read 2 New Done Done false
9844 2984 Acked 3983 Flush 0 New Done Done false
9845 2985 Acked 3984 Write 3 New Done Done false
9846 2986 Acked 3985 Read 9 New Done Done false
9847 2987 Acked 3986 Write 10 New Done Done false
9848 2988 Acked 3987 Write 6 New Done Done false
9849 2989 Acked 3988 Read 5 New Done Done false
9850 2990 Acked 3989 Flush 0 New Done Done false
9851 2991 Acked 3990 Write 10 New Done Done false
9852 2992 Acked 3991 Read 9 New Done Done false
9853 2993 Acked 3992 Flush 0 New Done Done false
9854 2994 Acked 3993 Write 4 New Done Done false
9855 2995 Acked 3994 Write 4 New Done Done false
9856 2996 Acked 3995 Write 7 New Done Done false
9857 2997 Acked 3996 Read 2 New Done Done false
9858 2998 Acked 3997 Read 5 New Done Done false
9859 2999 Acked 3998 Write 3 New Done Done false
9860 3000 Acked 3999 Read 10 New Done Done false
9861 3001 Acked 4000 Read 10 New Done Done false
9862 3002 Acked 4001 Write 9 New Done Done false
9863 3003 Acked 4002 Read 1 New Done Done false
9864 3004 Acked 4003 Read 2 New Done Done false
9865 3005 Acked 4004 Read 5 New Done Done false
9866 3006 Acked 4005 Write 4 New Done Done false
9867 3007 Acked 4006 Flush 0 New Done Done false
9868 3008 Acked 4007 Read 6 New Done Done false
9869 3009 Acked 4008 Read 3 New Done Done false
9870 3010 Acked 4009 Write 8 New Done Done false
9871 3011 Acked 4010 Write 2 New Done Done false
9872 3012 Acked 4011 Read 5 New Done Done false
9873 3013 Acked 4012 Read 3 New Done Done false
9874 3014 Acked 4013 Write 10 New Sent Sent false
9875 STATES DS:0 DS:1 DS:2 TOTAL
9876 New 300 0 0 300
9877 Sent 0 1 1 2
9878 Done 0 299 299 598
9879 Skipped 0 0 0 0
9880 Error 0 0 0 0
9881 Last Flush: 3713 4006 4006
9882 Downstairs last five completed: 3713 3712 3711 3710 3709
9883 Upstairs last five completed: 3014 3013 3012 3011 3010
9884 CLIENT: Up:0 ds:300 act:2
98852024-02-12T01:36:25.982ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
98862024-02-12T01:36:25.982ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
98872024-02-12T01:36:25.982ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
98882024-02-12T01:36:25.982ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
98892024-02-12T01:36:25.982ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
98902024-02-12T01:36:25.982ZINFOcrucible: [0] client re-new 301 jobs since flush 3713 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
98912024-02-12T01:36:25.982ZINFOcrucible: client task is exiting = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
98922024-02-12T01:36:26.984ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
98932024-02-12T01:36:26.984ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
98942024-02-12T01:36:26.984ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
98952024-02-12T01:36:26.984ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
98962024-02-12T01:36:26.984ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
98972024-02-12T01:36:26.984ZINFOcrucible: [0] client re-new 301 jobs since flush 3713 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
98982024-02-12T01:36:26.984ZINFOcrucible: client task is exiting = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
9899 [8810] got start action
9900 [8810] I am going to restart
9901 Starting downstairs at port 8810
9902 Make output file at "/tmp/dsc/downstairs-8810.txt"
9903 [8810][0] reports Starting
9904 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1174)
9905 [8810][0] reports Running
99062024-02-12T01:36:27.985ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
99072024-02-12T01:36:27.985ZINFOcrucible: ds_connection connected = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
99082024-02-12T01:36:27.986ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
99092024-02-12T01:36:27.986ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
99102024-02-12T01:36:27.986ZINFOcrucible: send last flush ID to this DS: 3713 = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
99112024-02-12T01:36:27.987ZINFOcrucible: Replied this last flush ID: 3713 = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
99122024-02-12T01:36:27.987ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
99132024-02-12T01:36:27.987ZINFOcrucible: [0] Transition from Offline to Replay = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
99142024-02-12T01:36:27.987ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
99152024-02-12T01:36:27.987ZINFOcrucible: [0] Transition from Replay to Active = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
9916 ----------------------------------------------------------------
9917 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
9918 Downstairs last five completed: 4014 4013 4012 4011 4010
9919 Upstairs last five completed: 3015 3014 3013 3012 3011
9920 CLIENT: Up:0 ds:0 act:3
9921 Replay: All jobs finished, all DS active.
9922 Main task has work to do, go find it
9923 got dsc Stop(2)
9924 stop 2
9925 Feb 12 01:36:29.331 INFO request completed, latency_us: 391, response_code: 204, uri:[8830] Got stop action so:false kr:false
9926 /stop/cid/2, method: POST, req_id: 0061c57b-b005-4b1f-8f5a-efb1a7de346a, remote_addr: 127.0.0.1:60456, local_addr: 127.0.0.1:9998
9927 Feb 12 01:36:29.332 INFO request completed, latency_us: 138{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2024-02-12T01:36:29.332188289Z","hostname":"ip-10-150-1-104.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"e1773b3e-b36f-4080-8b13-c0830f0d072e"}
9928 , response_code: 200, uri: /state/cid/2, method: GET, req_id: 0e6b998b-6abc-4cef-be10-e88193772b15, remote_addr: 127.0.0.1:{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2024-02-12T01:36:29.332261888Z","hostname":"ip-10-150-1-104.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"e1773b3e-b36f-4080-8b13-c0830f0d072e"}
9929 60456, local_addr: 127.0.0.1:9998
99302024-02-12T01:36:29.332ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
99312024-02-12T01:36:29.332ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
99322024-02-12T01:36:29.332ZINFOcrucible: [2] client re-new 0 jobs since flush 4014 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
99332024-02-12T01:36:29.332ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
9934 [8830] Exited with: ExitStatus(unix_wait_status(9))
9935 [8830][2] reports Exit
99362024-02-12T01:36:30.334ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
99372024-02-12T01:36:30.334ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
99382024-02-12T01:36:30.334ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
99392024-02-12T01:36:30.334ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
99402024-02-12T01:36:30.334ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
99412024-02-12T01:36:30.334ZINFOcrucible: [2] client re-new 0 jobs since flush 4014 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
99422024-02-12T01:36:30.334ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
99432024-02-12T01:36:31.336ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
99442024-02-12T01:36:31.336ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
99452024-02-12T01:36:31.336ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
99462024-02-12T01:36:31.336ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
99472024-02-12T01:36:31.336ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
99482024-02-12T01:36:31.336ZINFOcrucible: [2] client re-new 0 jobs since flush 4014 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
99492024-02-12T01:36:31.336ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
99502024-02-12T01:36:32.337ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
99512024-02-12T01:36:32.337ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
99522024-02-12T01:36:32.337ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
99532024-02-12T01:36:32.337ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
99542024-02-12T01:36:32.337ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
99552024-02-12T01:36:32.337ZINFOcrucible: [2] client re-new 0 jobs since flush 4014 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
99562024-02-12T01:36:32.337ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
9957 Feb 12 01:36:33.333 INFO request completed, latency_us: 195, response_code: 200, uri: /state/cid/2, method: GET, req_id: d0deb085-e0d6-443a-9194-85204bc5771a, remote_addr: 127.0.0.1:60456, local_addr: 127.0.0.1:9998
9958 001/300 Read block 1444 len 24576
9959 002/300 Read block 1460 len 4096
9960 003/300 Read block 422 len 8192
9961 004/300 Flush
99622024-02-12T01:36:33.338ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
99632024-02-12T01:36:33.338ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
99642024-02-12T01:36:33.338ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
99652024-02-12T01:36:33.338ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
99662024-02-12T01:36:33.338ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
99672024-02-12T01:36:33.338ZINFOcrucible: [2] client re-new 4 jobs since flush 4014 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
99682024-02-12T01:36:33.338ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
9969 005/300 Write block 1708 len 40960 data: 5 3 3 2 2 2 3 3 4 4
9970 006/300 Write block 4391 len 4096 data: 4
9971 007/300 Read block 561 len 32768
9972 008/300 Read block 3941 len 40960
9973 009/300 Read block 4226 len 4096
9974 010/300 Write block 168 len 12288 data: 4 4 3
9975 011/300 Write block 1388 len 32768 data: 3 3 3 3 3 3 4 4
9976 012/300 Write block 1755 len 36864 data: 4 4 4 3 2 2 2 2 2
9977 013/300 Write block 4222 len 4096 data: 4
9978 014/300 Flush
9979 015/300 Read block 3678 len 16384
9980 016/300 Write block 3458 len 32768 data: 4 3 3 2 2 2 2 2
9981 017/300 Write block 386 len 16384 data: 4 4 3 3
9982 018/300 Read block 2230 len 12288
9983 019/300 Write block 171 len 40960 data: 2 2 3 3 3 3 3 3 3 3
9984 020/300 Read block 3022 len 36864
9985 021/300 Write block 4697 len 32768 data: 4 4 3 3 3 3 2 2
9986 022/300 Read block 755 len 8192
9987 023/300 Read block 4900 len 12288
9988 024/300 Flush
9989 025/300 Write block 3414 len 24576 data: 3 3 3 3 4 3
9990 026/300 Read block 4042 len 36864
9991 027/300 Read block 873 len 32768
9992 028/300 Write block 933 len 28672 data: 5 4 5 5 6 5 4
9993 029/300 Write block 1123 len 40960 data: 5 4 2 2 2 3 3 3 3 3
9994 030/300 Write block 2559 len 20480 data: 3 3 3 3 3
9995 031/300 Write block 3753 len 32768 data: 2 2 2 2 3 3 3 3
9996 032/300 Write block 792 len 40960 data: 4 4 4 5 6 7 7 7 6 6
9997 033/300 Read block 3013 len 24576
9998 034/300 Read block 2469 len 20480
9999 035/300 Read block 4971 len 28672
10000 036/300 Write block 4598 len 8192 data: 4 4
10001 037/300 Read block 1569 len 40960
10002 038/300 Read block 1103 len 40960
10003 039/300 Read block 4565 len 8192
10004 040/300 Read block 4718 len 40960
10005 041/300 Write block 1956 len 16384 data: 3 3 3 3
10006 042/300 Write block 2580 len 24576 data: 3 3 3 4 4 5
10007 043/300 Read block 2764 len 4096
10008 044/300 Read block 2029 len 16384
10009 045/300 Read block 1836 len 40960
10010 046/300 Read block 2370 len 32768
10011 047/300 Read block 3040 len 4096
10012 048/300 Flush
10013 049/300 Write block 2483 len 8192 data: 4 4
10014 050/300 Write block 3032 len 8192 data: 2 2
10015 051/300 Read block 225 len 20480
10016 052/300 Read block 4196 len 40960
10017 053/300 Read block 386 len 36864
10018 054/300 Read block 235 len 16384
10019 055/300 Read block 2402 len 16384
10020 056/300 Read block 2337 len 40960
10021 057/300 Read block 4628 len 4096
10022 058/300 Read block 4452 len 8192
10023 059/300 Flush
10024 060/300 Read block 326 len 16384
10025 061/300 Read block 3808 len 32768
10026 062/300 Write block 530 len 32768 data: 4 4 4 4 5 4 4 4
10027 063/300 Write block 82 len 40960 data: 3 4 4 3 4 4 4 5 6 5
10028 064/300 Flush
10029 065/300 Flush
10030 066/300 Read block 4118 len 40960
10031 067/300 Write block 752 len 8192 data: 4 5
10032 068/300 Write block 2586 len 20480 data: 6 6 6 5 5
10033 069/300 Write block 3727 len 4096 data: 4
10034 070/300 Flush
10035 071/300 Read block 1901 len 32768
10036 072/300 Read block 2855 len 12288
10037 073/300 Read block 3459 len 28672
10038 074/300 Read block 73 len 16384
10039 075/300 Write block 4874 len 4096 data: 3
10040 076/300 Read block 411 len 24576
10041 077/300 Read block 424 len 4096
10042 078/300 Read block 4257 len 20480
10043 079/300 Write block 1638 len 4096 data: 4
10044 080/300 Read block 2860 len 28672
10045 081/300 Write block 979 len 4096 data: 4
10046 082/300 Write block 1309 len 16384 data: 4 4 4 3
10047 083/300 Write block 575 len 16384 data: 3 4 2 2
10048 084/300 Write block 2594 len 28672 data: 5 4 5 5 6 5 5
10049 085/300 Write block 1984 len 8192 data: 2 3
10050 086/300 Read block 2433 len 12288
10051 087/300 Flush
10052 088/300 Read block 2527 len 12288
10053 089/300 Write block 1423 len 4096 data: 3
10054 090/300 Read block 2895 len 8192
10055 091/300 Write block 681 len 4096 data: 3
10056 092/300 Write block 641 len 24576 data: 5 4 2 2 2 3
10057 093/300 Write block 497 len 40960 data: 3 3 3 3 3 3 3 3 2 2
10058 094/300 Read block 898 len 20480
10059 095/300 Read block 1298 len 16384
10060 096/300 Read block 4531 len 40960
10061 097/300 Read block 4048 len 4096
10062 098/300 Read block 841 len 24576
10063 099/300 Write block 4642 len 12288 data: 4 5 6
10064 100/300 Write block 3227 len 16384 data: 5 6 5 4
10065 101/300 Write block 2127 len 28672 data: 3 3 3 4 5 4 3
10066 102/300 Read block 720 len 8192
10067 103/300 Write block 2382 len 8192 data: 4 5
10068 104/300 Read block 1172 len 16384
10069 105/300 Read block 3562 len 12288
10070 106/300 Read block 1831 len 24576
10071 107/300 Read block 1950 len 40960
10072 108/300 Write block 3088 len 20480 data: 2 3 3 2 3
10073 109/300 Read block 1896 len 36864
10074 110/300 Flush
10075 111/300 Read block 1491 len 12288
10076 112/300 Read block 1543 len 36864
10077 113/300 Write block 969 len 24576 data: 3 3 3 3 4 4
10078 114/300 Read block 2897 len 28672
10079 115/300 Read block 421 len 24576
10080 116/300 Read block 3819 len 8192
10081 117/300 Read block 4660 len 20480
10082 118/300 Read block 2626 len 12288
10083 119/300 Write block 2198 len 28672 data: 3 4 4 5 5 5 5
10084 120/300 Read block 894 len 36864
10085 121/300 Read block 3630 len 4096
10086 122/300 Read block 3265 len 36864
10087 123/300 Read block 577 len 40960
10088 124/300 Read block 2786 len 12288
10089 125/300 Read block 2063 len 32768
10090 126/300 Write block 4540 len 8192 data: 3 3
10091 127/300 Read block 169 len 8192
10092 128/300 Flush
10093 129/300 Read block 4040 len 32768
10094 130/300 Flush
10095 131/300 Write block 2901 len 4096 data: 5
10096 132/300 Write block 1459 len 32768 data: 2 2 2 2 2 3 3 3
10097 133/300 Flush
10098 134/300 Flush
10099 135/300 Write block 1087 len 24576 data: 4 4 4 3 4 4
10100 136/300 Flush
10101 137/300 Flush
10102 138/300 Write block 2280 len 20480 data: 2 3 3 3 2
10103 139/300 Flush
10104 140/300 Read block 2042 len 24576
10105 141/300 Read block 948 len 36864
10106 142/300 Write block 466 len 32768 data: 2 2 2 2 2 2 2 2
10107 143/300 Flush
10108 144/300 Write block 1951 len 4096 data: 3
10109 145/300 Write block 3196 len 40960 data: 4 3 3 2 2 2 2 2 2 2
10110 146/300 Flush
10111 147/300 Read block 1723 len 40960
10112 148/300 Read block 2884 len 40960
10113 149/300 Write block 4607 len 20480 data: 3 3 3 3 4
10114 150/300 Read block 373 len 36864
10115 151/300 Read block 3689 len 4096
10116 152/300 Flush
10117 153/300 Write block 1473 len 24576 data: 3 2 2 3 4 4
10118 154/300 Read block 779 len 40960
10119 155/300 Write block 2606 len 32768 data: 3 3 3 3 3 3 4 4
10120 156/300 Read block 194 len 24576
10121 157/300 Read block 4871 len 8192
10122 158/300 Read block 1824 len 12288
10123 159/300 Write block 3053 len 40960 data: 2 2 2 3 2 2 2 2 2 2
10124 160/300 Write block 4863 len 40960 data: 2 2 2 2 2 3 3 3 3 3
10125 161/300 Write block 3375 len 32768 data: 6 7 7 6 5 5 4 3
10126 162/300 Flush
10127 163/300 Read block 313 len 12288
10128 164/300 Write block 1578 len 8192 data: 2 2
10129 165/300 Read block 1583 len 20480
10130 166/300 Read block 2515 len 24576
10131 167/300 Read block 831 len 4096
10132 168/300 Read block 3542 len 4096
10133 169/300 Flush
10134 170/300 Write block 3281 len 32768 data: 3 3 3 2 2 2 2 2
10135 171/300 Read block 3900 len 16384
10136 172/300 Flush
10137 173/300 Read block 4472 len 8192
10138 174/300 Write block 4265 len 36864 data: 7 6 6 7 6 5 5 6 6
10139 175/300 Read block 4236 len 12288
10140 176/300 Write block 644 len 40960 data: 3 3 4 3 2 2 2 2 2 2
10141 177/300 Read block 1654 len 28672
10142 178/300 Read block 2145 len 36864
10143 179/300 Read block 2694 len 20480
10144 180/300 Write block 2645 len 16384 data: 5 5 5 5
10145 181/300 Write block 3590 len 28672 data: 2 3 3 3 3 4 4
10146 182/300 Read block 3617 len 28672
10147 183/300 Write block 4549 len 4096 data: 5
10148 184/300 Flush
10149 185/300 Read block 1743 len 4096
10150 186/300 Write block 4685 len 32768 data: 2 3 3 3 3 5 4 4
10151 187/300 Read block 3212 len 24576
10152 188/300 Read block 2214 len 20480
10153 189/300 Write block 4742 len 20480 data: 3 3 3 3 3
10154 190/300 Write block 1719 len 16384 data: 4 4 3 3
10155 191/300 Write block 248 len 40960 data: 5 4 2 2 2 3 3 4 5 5
10156 192/300 Write block 932 len 8192 data: 4 6
10157 193/300 Read block 3358 len 32768
10158 194/300 Write block 3118 len 32768 data: 3 3 3 3 3 4 4 5
10159 195/300 Read block 1757 len 20480
10160 196/300 Read block 720 len 32768
10161 197/300 Write block 3313 len 16384 data: 2 3 5 4
10162 198/300 Read block 2028 len 4096
10163 199/300 Flush
10164 200/300 Read block 4254 len 36864
10165 201/300 Read block 4525 len 40960
10166 202/300 Read block 131 len 36864
10167 203/300 Flush
10168 204/300 Read block 4269 len 32768
10169 205/300 Read block 1026 len 4096
10170 206/300 Flush
10171 207/300 Read block 1752 len 24576
10172 208/300 Read block 1110 len 40960
10173 209/300 Read block 3733 len 36864
10174 210/300 Flush
10175 211/300 Write block 4469 len 36864 data: 3 4 4 5 5 5 5 5 4
10176 212/300 Read block 2932 len 8192
10177 213/300 Write block 4950 len 24576 data: 4 3 3 2 2 2
10178 214/300 Read block 697 len 24576
10179 215/300 Read block 523 len 36864
10180 216/300 Write block 3038 len 28672 data: 4 4 4 4 5 2 2
10181 217/300 Read block 4395 len 36864
10182 218/300 Write block 2170 len 28672 data: 3 5 5 3 4 6 6
10183 219/300 Write block 1905 len 40960 data: 2 2 2 2 2 2 3 3 3 3
10184 220/300 Write block 4330 len 20480 data: 2 2 2 2 2
10185 221/300 Write block 3343 len 12288 data: 2 2 2
10186 222/300 Read block 2994 len 40960
10187 223/300 Read block 2909 len 12288
10188 224/300 Read block 1927 len 8192
10189 225/300 Read block 626 len 28672
10190 226/300 Read block 4304 len 16384
10191 227/300 Read block 4400 len 8192
10192 228/300 Write block 4083 len 20480 data: 2 2 2 2 2
10193 229/300 Read block 1020 len 20480
10194 230/300 Read block 4711 len 32768
10195 231/300 Read block 166 len 12288
10196 232/300 Write block 1668 len 36864 data: 3 3 3 4 4 4 4 4 5
10197 233/300 Read block 4622 len 24576
10198 234/300 Write block 4014 len 12288 data: 3 2 2
10199 235/300 Read block 4872 len 32768
10200 236/300 Read block 523 len 20480
10201 237/300 Read block 2641 len 28672
10202 238/300 Write block 3165 len 20480 data: 3 3 3 3 3
10203 239/300 Read block 2846 len 28672
10204 240/300 Write block 2795 len 20480 data: 3 3 3 2 3
10205 241/300 Write block 1390 len 36864 data: 4 4 4 4 5 5 4 3 3
10206 242/300 Write block 3748 len 40960 data: 3 3 3 3 2 3 3 3 3 4
10207 243/300 Read block 4378 len 12288
10208 244/300 Flush
10209 245/300 Read block 3554 len 24576
10210 246/300 Read block 2390 len 16384
10211 247/300 Read block 2910 len 32768
10212 248/300 Read block 4730 len 12288
10213 249/300 Write block 4002 len 16384 data: 3 3 3 3
10214 250/300 Write block 2779 len 4096 data: 3
10215 251/300 Read block 1292 len 40960
10216 252/300 Write block 4100 len 32768 data: 2 2 2 2 2 2 2 2
10217 253/300 Read block 101 len 24576
10218 254/300 Read block 4877 len 4096
10219 255/300 Flush
10220 256/300 Read block 1479 len 20480
10221 257/300 Read block 3886 len 8192
10222 258/300 Write block 3479 len 4096 data: 3
10223 259/300 Write block 1422 len 20480 data: 3 4 3 3 3
10224 260/300 Write block 4095 len 4096 data: 2
10225 261/300 Read block 4894 len 8192
10226 262/300 Write block 113 len 8192 data: 3 3
10227 263/300 Flush
10228 264/300 Read block 700 len 20480
10229 265/300 Read block 0 len 16384
10230 266/300 Write block 4108 len 4096 data: 2
10231 267/300 Write block 908 len 24576 data: 4 4 3 4 3 3
10232 268/300 Read block 4881 len 8192
10233 269/300 Write block 4139 len 32768 data: 4 4 4 2 2 2 2 2
10234 270/300 Read block 1051 len 4096
10235 271/300 Write block 205 len 16384 data: 3 3 3 3
10236 272/300 Read block 4892 len 16384
10237 273/300 Write block 4596 len 28672 data: 4 4 5 5 5 6 5
10238 274/300 Write block 3256 len 24576 data: 2 2 2 2 2 2
10239 275/300 Write block 1404 len 24576 data: 4 4 4 5 5 4
10240 276/300 Read block 2353 len 12288
10241 277/300 Read block 4900 len 32768
10242 278/300 Read block 1177 len 20480
10243 279/300 Read block 3548 len 40960
10244 280/300 Write block 2853 len 40960 data: 4 4 4 4 3 3 4 3 3 3
10245 281/300 Read block 1421 len 16384
10246 282/300 Write block 96 len 8192 data: 5 3
10247 283/300 Read block 2623 len 16384
10248 284/300 Read block 3099 len 20480
10249 285/300 Write block 3113 len 12288 data: 5 6 5
10250 286/300 Read block 2013 len 36864
10251 287/300 Flush
10252 288/300 Write block 3861 len 32768 data: 3 3 3 3 3 3 3 3
10253 289/300 Write block 4696 len 4096 data: 4
10254 290/300 Write block 2591 len 16384 data: 6 5 5 6
10255 291/300 Read block 2770 len 32768
10256 292/300 Read block 8 len 40960
10257 293/300 Flush
10258 294/300 Read block 2868 len 32768
10259 295/300 Write block 4015 len 8192 data: 3 3
10260 296/300 Write block 1537 len 36864 data: 5 4 6 6 5 5 5 5 4
10261 297/300 Flush
10262 298/300 Flush
10263 299/300 Read block 3493 len 24576
10264 300/300 Read block 4631 len 36864
10265 Main task has work to do, go find it
10266 got dsc Start(2)
10267 start 2
10268 Feb 12 01:36:33.688 INFO request completed, latency_us: 219, response_code: 204, uri: /start/cid/2, method: POST, req_id: 04bca1fe-74a8-4212-b959-a3b7b48b5246, remote_addr: 127.0.0.1:60456, local_addr: 127.0.0.1:9998
10269 Replay: started 2, returned:Ok(())
10270 ----------------------------------------------------------------
10271 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
10272 States: Active Active Offline
10273 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
10274 3016 Acked 4015 Read 6 Done Done New true
10275 3017 Acked 4016 Read 1 Done Done New true
10276 3018 Acked 4017 Read 2 Done Done New true
10277 3019 Acked 4018 Flush 0 Done Done New true
10278 3020 Acked 4019 Write 10 Done Done New false
10279 3021 Acked 4020 Write 1 Done Done New false
10280 3022 Acked 4021 Read 8 Done Done New false
10281 3023 Acked 4022 Read 10 Done Done New false
10282 3024 Acked 4023 Read 1 Done Done New false
10283 3025 Acked 4024 Write 3 Done Done New false
10284 3026 Acked 4025 Write 8 Done Done New false
10285 3027 Acked 4026 Write 9 Done Done New false
10286 3028 Acked 4027 Write 1 Done Done New false
10287 3029 Acked 4028 Flush 0 Done Done New false
10288 3030 Acked 4029 Read 4 Done Done New false
10289 3031 Acked 4030 Write 8 Done Done New false
10290 3032 Acked 4031 Write 4 Done Done New false
10291 3033 Acked 4032 Read 3 Done Done New false
10292 3034 Acked 4033 Write 10 Done Done New false
10293 3035 Acked 4034 Read 9 Done Done New false
10294 3036 Acked 4035 Write 8 Done Done New false
10295 3037 Acked 4036 Read 2 Done Done New false
10296 3038 Acked 4037 Read 3 Done Done New false
10297 3039 Acked 4038 Flush 0 Done Done New false
10298 3040 Acked 4039 Write 6 Done Done New false
10299 3041 Acked 4040 Read 9 Done Done New false
10300 3042 Acked 4041 Read 8 Done Done New false
10301 3043 Acked 4042 Write 7 Done Done New false
10302 3044 Acked 4043 Write 10 Done Done New false
10303 3045 Acked 4044 Write 5 Done Done New false
10304 3046 Acked 4045 Write 8 Done Done New false
10305 3047 Acked 4046 Write 10 Done Done New false
10306 3048 Acked 4047 Read 6 Done Done New false
10307 3049 Acked 4048 Read 5 Done Done New false
10308 3050 Acked 4049 Read 7 Done Done New false
10309 3051 Acked 4050 Write 2 Done Done New false
10310 3052 Acked 4051 Read 10 Done Done New false
10311 3053 Acked 4052 Read 10 Done Done New false
10312 3054 Acked 4053 Read 2 Done Done New false
10313 3055 Acked 4054 Read 10 Done Done New false
10314 3056 Acked 4055 Write 4 Done Done New false
10315 3057 Acked 4056 Write 6 Done Done New false
10316 3058 Acked 4057 Read 1 Done Done New false
10317 3059 Acked 4058 Read 4 Done Done New false
10318 3060 Acked 4059 Read 10 Done Done New false
10319 3061 Acked 4060 Read 8 Done Done New false
10320 3062 Acked 4061 Read 1 Done Done New false
10321 3063 Acked 4062 Flush 0 Done Done New false
10322 3064 Acked 4063 Write 2 Done Done New false
10323 3065 Acked 4064 Write 2 Done Done New false
10324 3066 Acked 4065 Read 5 Done Done New false
10325 3067 Acked 4066 Read 10 Done Done New false
10326 3068 Acked 4067 Read 9 Done Done New false
10327 3069 Acked 4068 Read 4 Done Done New false
10328 3070 Acked 4069 Read 4 Done Done New false
10329 3071 Acked 4070 Read 10 Done Done New false
10330 3072 Acked 4071 Read 1 Done Done New false
10331 3073 Acked 4072 Read 2 Done Done New false
10332 3074 Acked 4073 Flush 0 Done Done New false
10333 3075 Acked 4074 Read 4 Done Done New false
10334 3076 Acked 4075 Read 8 Done Done New false
10335 3077 Acked 4076 Write 8 Done Done New false
10336 3078 Acked 4077 Write 10 Done Done New false
10337 3079 Acked 4078 Flush 0 Done Done New false
10338 3080 Acked 4079 Flush 0 Done Done New false
10339 3081 Acked 4080 Read 10 Done Done New false
10340 3082 Acked 4081 Write 2 Done Done New false
10341 3083 Acked 4082 Write 5 Done Done New false
10342 3084 Acked 4083 Write 1 Done Done New false
10343 3085 Acked 4084 Flush 0 Done Done New false
10344 3086 Acked 4085 Read 8 Done Done New false
10345 3087 Acked 4086 Read 3 Done Done New false
10346 3088 Acked 4087 Read 7 Done Done New false
10347 3089 Acked 4088 Read 4 Done Done New false
10348 3090 Acked 4089 Write 1 Done Done New false
10349 3091 Acked 4090 Read 6 Done Done New false
10350 3092 Acked 4091 Read 1 Done Done New false
10351 3093 Acked 4092 Read 5 Done Done New false
10352 3094 Acked 4093 Write 1 Done Done New false
10353 3095 Acked 4094 Read 7 Done Done New false
10354 3096 Acked 4095 Write 1 Done Done New false
10355 3097 Acked 4096 Write 4 Done Done New false
10356 3098 Acked 4097 Write 4 Done Done New false
10357 3099 Acked 4098 Write 7 Done Done New false
10358 3100 Acked 4099 Write 2 Done Done New false
10359 3101 Acked 4100 Read 3 Done Done New false
10360 3102 Acked 4101 Flush 0 Done Done New false
10361 3103 Acked 4102 Read 3 Done Done New false
10362 3104 Acked 4103 Write 1 Done Done New false
10363 3105 Acked 4104 Read 2 Done Done New false
10364 3106 Acked 4105 Write 1 Done Done New false
10365 3107 Acked 4106 Flush 0 Done Done New false
10366 3108 Acked 4107 Write 6 Done Done New false
10367 3109 Acked 4108 Write 10 Done Done New false
10368 3110 Acked 4109 Read 5 Done Done New false
10369 3111 Acked 4110 Read 4 Done Done New false
10370 3112 Acked 4111 Read 10 Done Done New false
10371 3113 Acked 4112 Read 1 Done Done New false
10372 3114 Acked 4113 Read 6 Done Done New false
10373 3115 Acked 4114 Write 3 Done Done New false
10374 3116 Acked 4115 Write 4 Done Done New false
10375 3117 Acked 4116 Write 7 Done Done New false
10376 3118 Acked 4117 Read 2 Done Done New false
10377 3119 Acked 4118 Write 2 Done Done New false
10378 3120 Acked 4119 Read 4 Done Done New false
10379 3121 Acked 4120 Read 3 Done Done New false
10380 3122 Acked 4121 Read 6 Done Done New false
10381 3123 Acked 4122 Read 10 Done Done New false
10382 3124 Acked 4123 Write 5 Done Done New false
10383 3125 Acked 4124 Read 9 Done Done New false
10384 3126 Acked 4125 Flush 0 Done Done New false
10385 3127 Acked 4126 Read 3 Done Done New false
10386 3128 Acked 4127 Read 9 Done Done New false
10387 3129 Acked 4128 Write 6 Done Done New false
10388 3130 Acked 4129 Read 7 Done Done New false
10389 3131 Acked 4130 Read 6 Done Done New false
10390 3132 Acked 4131 Read 2 Done Done New false
10391 3133 Acked 4132 Read 5 Done Done New false
10392 3134 Acked 4133 Read 3 Done Done New false
10393 3135 Acked 4134 Write 7 Done Done New false
10394 3136 Acked 4135 Read 9 Done Done New false
10395 3137 Acked 4136 Read 1 Done Done New false
10396 3138 Acked 4137 Read 9 Done Done New false
10397 3139 Acked 4138 Read 10 Done Done New false
10398 3140 Acked 4139 Read 3 Done Done New false
10399 3141 Acked 4140 Read 8 Done Done New false
10400 3142 Acked 4141 Write 2 Done Done New false
10401 3143 Acked 4142 Read 2 Done Done New false
10402 3144 Acked 4143 Flush 0 Done Done New false
10403 3145 Acked 4144 Read 8 Done Done New false
10404 3146 Acked 4145 Flush 0 Done Done New false
10405 3147 Acked 4146 Write 1 Done Done New false
10406 3148 Acked 4147 Write 8 Done Done New false
10407 3149 Acked 4148 Flush 0 Done Done New false
10408 3150 Acked 4149 Flush 0 Done Done New false
10409 3151 Acked 4150 Write 6 Done Done New false
10410 3152 Acked 4151 Flush 0 Done Done New false
10411 3153 Acked 4152 Flush 0 Done Done New false
10412 3154 Acked 4153 Write 5 Done Done New false
10413 3155 Acked 4154 Flush 0 Done Done New false
10414 3156 Acked 4155 Read 6 Done Done New false
10415 3157 Acked 4156 Read 9 Done Done New false
10416 3158 Acked 4157 Write 8 Done Done New false
10417 3159 Acked 4158 Flush 0 Done Done New false
10418 3160 Acked 4159 Write 1 Done Done New false
10419 3161 Acked 4160 Write 10 Done Done New false
10420 3162 Acked 4161 Flush 0 Done Done New false
10421 3163 Acked 4162 Read 10 Done Done New false
10422 3164 Acked 4163 Read 10 Done Done New false
10423 3165 Acked 4164 Write 5 Done Done New false
10424 3166 Acked 4165 Read 9 Done Done New false
10425 3167 Acked 4166 Read 1 Done Done New false
10426 3168 Acked 4167 Flush 0 Done Done New false
10427 3169 Acked 4168 Write 6 Done Done New false
10428 3170 Acked 4169 Read 10 Done Done New false
10429 3171 Acked 4170 Write 8 Done Done New false
10430 3172 Acked 4171 Read 6 Done Done New false
10431 3173 Acked 4172 Read 2 Done Done New false
10432 3174 Acked 4173 Read 3 Done Done New false
10433 3175 Acked 4174 Write 10 Done Done New false
10434 3176 Acked 4175 Write 10 Done Done New false
10435 3177 Acked 4176 Write 8 Done Done New false
10436 3178 Acked 4177 Flush 0 Done Done New false
10437 3179 Acked 4178 Read 3 Done Done New false
10438 3180 Acked 4179 Write 2 Done Done New false
10439 3181 Acked 4180 Read 5 Done Done New false
10440 3182 Acked 4181 Read 6 Done Done New false
10441 3183 Acked 4182 Read 1 Done Done New false
10442 3184 Acked 4183 Read 1 Done Done New false
10443 3185 Acked 4184 Flush 0 Done Done New false
10444 3186 Acked 4185 Write 8 Done Done New false
10445 3187 Acked 4186 Read 4 Done Done New false
10446 3188 Acked 4187 Flush 0 Done Done New false
10447 3189 Acked 4188 Read 2 Done Done New false
10448 3190 Acked 4189 Write 9 Done Done New false
10449 3191 Acked 4190 Read 3 Done Done New false
10450 3192 Acked 4191 Write 10 Done Done New false
10451 3193 Acked 4192 Read 7 Done Done New false
10452 3194 Acked 4193 Read 9 Done Done New false
10453 3195 Acked 4194 Read 5 Done Done New false
10454 3196 Acked 4195 Write 4 Done Done New false
10455 3197 Acked 4196 Write 7 Done Done New false
10456 3198 Acked 4197 Read 7 Done Done New false
10457 3199 Acked 4198 Write 1 Done Done New false
10458 3200 Acked 4199 Flush 0 Done Done New false
10459 3201 Acked 4200 Read 1 Done Done New false
10460 3202 Acked 4201 Write 8 Done Done New false
10461 3203 Acked 4202 Read 6 Done Done New false
10462 3204 Acked 4203 Read 5 Done Done New false
10463 3205 Acked 4204 Write 5 Done Done New false
10464 3206 Acked 4205 Write 4 Done Done New false
10465 3207 Acked 4206 Write 10 Done Done New false
10466 3208 Acked 4207 Write 2 Done Done New false
10467 3209 Acked 4208 Read 8 Done Done New false
10468 3210 Acked 4209 Write 8 Done Done New false
10469 3211 Acked 4210 Read 5 Done Done New false
10470 3212 Acked 4211 Read 8 Done Done New false
10471 3213 Acked 4212 Write 4 Done Done New false
10472 3214 Acked 4213 Read 1 Done Done New false
10473 3215 Acked 4214 Flush 0 Done Done New false
10474 3216 Acked 4215 Read 9 Done Done New false
10475 3217 Acked 4216 Read 10 Done Done New false
10476 3218 Acked 4217 Read 9 Done Done New false
10477 3219 Acked 4218 Flush 0 Done Done New false
10478 3220 Acked 4219 Read 8 Done Done New false
10479 3221 Acked 4220 Read 1 Done Done New false
10480 3222 Acked 4221 Flush 0 Done Done New false
10481 3223 Acked 4222 Read 6 Done Done New false
10482 3224 Acked 4223 Read 10 Done Done New false
10483 3225 Acked 4224 Read 9 Done Done New false
10484 3226 Acked 4225 Flush 0 Done Done New false
10485 3227 Acked 4226 Write 9 Done Done New false
10486 3228 Acked 4227 Read 2 Done Done New false
10487 3229 Acked 4228 Write 6 Done Done New false
10488 3230 Acked 4229 Read 6 Done Done New false
10489 3231 Acked 4230 Read 9 Done Done New false
10490 3232 Acked 4231 Write 7 Done Done New false
10491 3233 Acked 4232 Read 9 Done Done New false
10492 3234 Acked 4233 Write 7 Done Done New false
10493 3235 Acked 4234 Write 10 Done Done New false
10494 3236 Acked 4235 Write 5 Done Done New false
10495 3237 Acked 4236 Write 3 Done Done New false
10496 3238 Acked 4237 Read 10 Done Done New false
10497 3239 Acked 4238 Read 3 Done Done New false
10498 3240 Acked 4239 Read 2 Done Done New false
10499 3241 Acked 4240 Read 7 Done Done New false
10500 3242 Acked 4241 Read 4 Done Done New false
10501 3243 Acked 4242 Read 2 Done Done New false
10502 3244 Acked 4243 Write 5 Done Done New false
10503 3245 Acked 4244 Read 5 Done Done New false
10504 3246 Acked 4245 Read 8 Done Done New false
10505 3247 Acked 4246 Read 3 Done Done New false
10506 3248 Acked 4247 Write 9 Done Done New false
10507 3249 Acked 4248 Read 6 Done Done New false
10508 3250 Acked 4249 Write 3 Done Done New false
10509 3251 Acked 4250 Read 8 Done Done New false
10510 3252 Acked 4251 Read 5 Done Done New false
10511 3253 Acked 4252 Read 7 Done Done New false
10512 3254 Acked 4253 Write 5 Done Done New false
10513 3255 Acked 4254 Read 7 Done Done New false
10514 3256 Acked 4255 Write 5 Done Done New false
10515 3257 Acked 4256 Write 9 Done Done New false
10516 3258 Acked 4257 Write 10 Done Done New false
10517 3259 Acked 4258 Read 3 Done Done New false
10518 3260 Acked 4259 Flush 0 Done Done New false
10519 3261 Acked 4260 Read 6 Done Done New false
10520 3262 Acked 4261 Read 4 Done Done New false
10521 3263 Acked 4262 Read 8 Done Done New false
10522 3264 Acked 4263 Read 3 Done Done New false
10523 3265 Acked 4264 Write 4 Done Done New false
10524 3266 Acked 4265 Write 1 Done Done New false
10525 3267 Acked 4266 Read 10 Done Done New false
10526 3268 Acked 4267 Write 8 Done Done New false
10527 3269 Acked 4268 Read 6 Done Done New false
10528 3270 Acked 4269 Read 1 Done Done New false
10529 3271 Acked 4270 Flush 0 Done Done New false
10530 3272 Acked 4271 Read 5 Done Done New false
10531 3273 Acked 4272 Read 2 Done Done New false
10532 3274 Acked 4273 Write 1 Done Done New false
10533 3275 Acked 4274 Write 5 Done Done New false
10534 3276 Acked 4275 Write 1 Done Done New false
10535 3277 Acked 4276 Read 2 Done Done New false
10536 3278 Acked 4277 Write 2 Done Done New false
10537 3279 Acked 4278 Flush 0 Done Done New false
10538 3280 Acked 4279 Read 5 Done Done New false
10539 3281 Acked 4280 Read 4 Done Done New false
10540 3282 Acked 4281 Write 1 Done Done New false
10541 3283 Acked 4282 Write 6 Done Done New false
10542 3284 Acked 4283 Read 2 Done Done New false
10543 3285 Acked 4284 Write 8 Done Done New false
10544 3286 Acked 4285 Read 1 Done Done New false
10545 3287 Acked 4286 Write 4 Done Done New false
10546 3288 Acked 4287 Read 4 Done Done New false
10547 3289 Acked 4288 Write 7 Done Done New false
10548 3290 Acked 4289 Write 6 Done Done New false
10549 3291 Acked 4290 Write 6 Done Done New false
10550 3292 Acked 4291 Read 3 Done Done New false
10551 3293 Acked 4292 Read 8 Done Done New false
10552 3294 Acked 4293 Read 5 Done Done New false
10553 3295 Acked 4294 Read 10 Done Done New false
10554 3296 Acked 4295 Write 10 Done Done New false
10555 3297 Acked 4296 Read 4 Done Done New false
10556 3298 Acked 4297 Write 2 Done Done New false
10557 3299 Acked 4298 Read 4 Done Done New false
10558 3300 Acked 4299 Read 5 Done Done New false
10559 3301 Acked 4300 Write 3 Done Done New false
10560 3302 Acked 4301 Read 9 Done Done New false
10561 3303 Acked 4302 Flush 0 Done Done New false
10562 3304 Acked 4303 Write 8 Done Done New false
10563 3305 Acked 4304 Write 1 Done Done New false
10564 3306 Acked 4305 Write 4 Done Done New false
10565 3307 Acked 4306 Read 8 Done Done New false
10566 3308 Acked 4307 Read 10 Done Done New false
10567 3309 Acked 4308 Flush 0 Done Done New false
10568 3310 Acked 4309 Read 8 Done Done New false
10569 3311 Acked 4310 Write 2 Done Done New false
10570 3312 Acked 4311 Write 9 Done Done New false
10571 3313 Acked 4312 Flush 0 Done Done New false
10572 3314 Acked 4313 Flush 0 Done Done New false
10573 3315 Acked 4314 Read 6 Done Done New false
10574 3316 Acked 4315 Read 9 Done Done New false
10575 STATES DS:0 DS:1 DS:2 TOTAL
10576 New 0 0 301 301
10577 Sent 0 0 0 0
10578 Done 301 301 0 602
10579 Skipped 0 0 0 0
10580 Error 0 0 0 0
10581 Last Flush: 4313 4313 4014
10582 Downstairs last five completed: 4014 4013 4012 4011 4010
10583 Upstairs last five completed: 3316 3315 3314 3313 3312
10584 CLIENT: Up:0 ds:301 act:2
105852024-02-12T01:36:34.339ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
105862024-02-12T01:36:34.339ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
105872024-02-12T01:36:34.339ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
105882024-02-12T01:36:34.339ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
105892024-02-12T01:36:34.339ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
105902024-02-12T01:36:34.340ZINFOcrucible: [2] client re-new 302 jobs since flush 4014 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
105912024-02-12T01:36:34.340ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
10592 [8830] got start action
10593 [8830] I am going to restart
10594 Starting downstairs at port 8830
10595 Make output file at "/tmp/dsc/downstairs-8830.txt"
10596 [8830][2] reports Starting
105972024-02-12T01:36:35.340ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
105982024-02-12T01:36:35.340ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
105992024-02-12T01:36:35.340ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
106002024-02-12T01:36:35.340ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
106012024-02-12T01:36:35.340ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
106022024-02-12T01:36:35.340ZINFOcrucible: [2] client re-new 302 jobs since flush 4014 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
106032024-02-12T01:36:35.341ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
10604 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1175)
10605 [8830][2] reports Running
106062024-02-12T01:36:36.341ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
106072024-02-12T01:36:36.342ZINFOcrucible: ds_connection connected = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
106082024-02-12T01:36:36.343ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
106092024-02-12T01:36:36.343ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
106102024-02-12T01:36:36.343ZINFOcrucible: send last flush ID to this DS: 4014 = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
106112024-02-12T01:36:36.343ZINFOcrucible: Replied this last flush ID: 4014 = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
106122024-02-12T01:36:36.343ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
106132024-02-12T01:36:36.343ZINFOcrucible: [2] Transition from Offline to Replay = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
106142024-02-12T01:36:36.343ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
106152024-02-12T01:36:36.343ZINFOcrucible: [2] Transition from Replay to Active = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
10616 ----------------------------------------------------------------
10617 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
10618 Downstairs last five completed: 4316 4315 4314 4313 4312
10619 Upstairs last five completed: 3317 3316 3315 3314 3313
10620 CLIENT: Up:0 ds:0 act:3
10621 Replay: All jobs finished, all DS active.
10622 Main task has work to do, go find it
10623 got dsc Stop(2)
10624 stop 2
10625 Feb 12 01:36:37.691 INFO request completed, latency_us: 401, response_code: 204[8830] Got stop action so:false kr:false
10626 , uri: /stop/cid/2, method: POST, req_id: 3dcc578e-af1c-4732-bf8c-86776ef51f91, remote_addr: 127.0.0.1:60456, local_addr: 127.0.0.1:9998
106272024-02-12T01:36:37.692ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
106282024-02-12T01:36:37.692ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
106292024-02-12T01:36:37.692ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
106302024-02-12T01:36:37.692ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
106312024-02-12T01:36:37.692ZINFOcrucible: [2] client re-new 0 jobs since flush 4316 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
106322024-02-12T01:36:37.692ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
10633 Feb 12 01:36:37.692 INFO request completed, latency_us: 155, response_code: 200, uri: /state/cid/2, method: GET, req_id: 059e89ee-b738-47cd-90be-71009ce51977, remote_addr: 127.0.0.1:60456, local_addr: 127.0.0.1:9998
10634 [8830] Exited with: ExitStatus(unix_wait_status(9))
10635 [8830][2] reports Exit
106362024-02-12T01:36:38.694ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
106372024-02-12T01:36:38.694ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
106382024-02-12T01:36:38.694ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
106392024-02-12T01:36:38.694ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
106402024-02-12T01:36:38.694ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
106412024-02-12T01:36:38.694ZINFOcrucible: [2] client re-new 0 jobs since flush 4316 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
106422024-02-12T01:36:38.694ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
106432024-02-12T01:36:39.696ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
106442024-02-12T01:36:39.696ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
106452024-02-12T01:36:39.696ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
106462024-02-12T01:36:39.696ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
106472024-02-12T01:36:39.696ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
106482024-02-12T01:36:39.696ZINFOcrucible: [2] client re-new 0 jobs since flush 4316 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
106492024-02-12T01:36:39.696ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
106502024-02-12T01:36:40.698ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
106512024-02-12T01:36:40.698ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
106522024-02-12T01:36:40.698ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
106532024-02-12T01:36:40.698ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
106542024-02-12T01:36:40.698ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
106552024-02-12T01:36:40.698ZINFOcrucible: [2] client re-new 0 jobs since flush 4316 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
106562024-02-12T01:36:40.698ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
10657 Feb 12 01:36:41.694 INFO request completed, latency_us: 213, response_code: 200, uri: /state/cid/2, method: GET, req_id: 2d31212d-84ce-4c1c-9041-9659d5ca3d1b, remote_addr: 127.0.0.1:60456, local_addr: 127.0.0.1:9998
10658 001/300 Read block 159 len 12288
10659 002/300 Flush
10660 003/300 Write block 187 len 16384 data: 3 4 5 5
10661 004/300 Write block 783 len 32768 data: 4 3 3 3 3 4 3 4
10662 005/300 Read block 2954 len 16384
10663 006/300 Write block 3098 len 28672 data: 3 3 3 3 3 5 6
106642024-02-12T01:36:41.700ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
106652024-02-12T01:36:41.700ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
106662024-02-12T01:36:41.700ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
10667 007/300 Read block 4402 len 36864
106682024-02-12T01:36:41.700ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
106692024-02-12T01:36:41.700ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
106702024-02-12T01:36:41.700ZINFOcrucible: [2] client re-new 6 jobs since flush 4316 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
106712024-02-12T01:36:41.700ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
10672 008/300 Read block 1813 len 40960
10673 009/300 Read block 860 len 8192
10674 010/300 Read block 1493 len 40960
10675 011/300 Write block 2059 len 40960 data: 2 3 3 3 5 5 5 5 6 6
10676 012/300 Read block 640 len 12288
10677 013/300 Write block 3923 len 4096 data: 3
10678 014/300 Read block 4056 len 4096
10679 015/300 Write block 853 len 4096 data: 3
10680 016/300 Write block 2895 len 16384 data: 2 3 4 3
10681 017/300 Read block 4363 len 12288
10682 018/300 Write block 1083 len 24576 data: 3 3 4 5 5 5
10683 019/300 Read block 2115 len 8192
10684 020/300 Read block 2386 len 28672
10685 021/300 Read block 928 len 16384
10686 022/300 Read block 775 len 20480
10687 023/300 Write block 1590 len 16384 data: 2 3 3 3
10688 024/300 Write block 822 len 4096 data: 3
10689 025/300 Read block 4876 len 40960
10690 026/300 Write block 1058 len 16384 data: 3 3 3 4
10691 027/300 Read block 3139 len 8192
10692 028/300 Read block 1228 len 4096
10693 029/300 Write block 2029 len 20480 data: 3 3 3 2 2
10694 030/300 Write block 742 len 20480 data: 4 3 3 4 3
10695 031/300 Read block 893 len 4096
10696 032/300 Write block 4290 len 28672 data: 2 2 2 2 3 3 3
10697 033/300 Write block 3186 len 24576 data: 5 5 5 5 5 6
10698 034/300 Write block 2143 len 4096 data: 2
10699 035/300 Read block 710 len 28672
10700 036/300 Read block 209 len 36864
10701 037/300 Read block 2690 len 20480
10702 038/300 Flush
10703 039/300 Write block 2726 len 16384 data: 5 4 4 4
10704 040/300 Write block 4095 len 16384 data: 3 2 2 2
10705 041/300 Write block 1455 len 16384 data: 4 3 3 3
10706 042/300 Write block 3383 len 16384 data: 2 2 2 2
10707 043/300 Read block 862 len 20480
10708 044/300 Write block 1503 len 8192 data: 4 4
10709 045/300 Read block 1352 len 36864
10710 046/300 Write block 1660 len 16384 data: 2 2 2 2
10711 047/300 Write block 4971 len 12288 data: 3 2 2
10712 048/300 Read block 4166 len 24576
10713 049/300 Flush
10714 050/300 Read block 1627 len 32768
10715 051/300 Write block 659 len 40960 data: 2 2 2 2 2 2 3 3 2 3
10716 052/300 Read block 2369 len 24576
10717 053/300 Read block 3910 len 36864
10718 054/300 Flush
10719 055/300 Flush
10720 056/300 Write block 824 len 12288 data: 4 3 2
10721 057/300 Write block 385 len 32768 data: 3 5 5 4 4 3 3 4
10722 058/300 Read block 4102 len 8192
10723 059/300 Read block 2971 len 8192
10724 060/300 Read block 4610 len 24576
10725 061/300 Read block 177 len 12288
10726 062/300 Write block 1727 len 4096 data: 5
10727 063/300 Read block 1836 len 20480
10728 064/300 Write block 4102 len 12288 data: 3 3 3
10729 065/300 Read block 656 len 36864
10730 066/300 Read block 1684 len 20480
10731 067/300 Read block 96 len 12288
10732 068/300 Read block 936 len 16384
10733 069/300 Write block 1801 len 8192 data: 2 2
10734 070/300 Write block 504 len 36864 data: 4 3 3 3 4 4 4 4 5
10735 071/300 Read block 1789 len 24576
10736 072/300 Read block 1542 len 36864
10737 073/300 Write block 1165 len 20480 data: 3 3 5 5 4
10738 074/300 Write block 2850 len 36864 data: 3 4 3 5 5 5 5 4 4
10739 075/300 Read block 2562 len 8192
10740 076/300 Write block 2458 len 12288 data: 3 2 2
10741 077/300 Write block 3019 len 16384 data: 2 2 3 3
10742 078/300 Read block 4229 len 4096
10743 079/300 Flush
10744 080/300 Read block 3380 len 12288
10745 081/300 Read block 121 len 16384
10746 082/300 Read block 4897 len 24576
10747 083/300 Write block 1192 len 12288 data: 2 3 3
10748 084/300 Read block 512 len 40960
10749 085/300 Read block 3156 len 36864
10750 086/300 Write block 560 len 24576 data: 2 2 2 2 2 2
10751 087/300 Read block 1179 len 28672
10752 088/300 Read block 3685 len 20480
10753 089/300 Write block 3228 len 36864 data: 7 6 5 5 5 6 5 5 5
10754 090/300 Flush
10755 091/300 Write block 3483 len 32768 data: 3 4 4 2 2 2 3 3
10756 092/300 Read block 2957 len 28672
10757 093/300 Read block 3567 len 28672
10758 094/300 Write block 4898 len 28672 data: 3 3 3 3 3 4 5
10759 095/300 Write block 1136 len 20480 data: 3 3 2 2 4
10760 096/300 Read block 1682 len 12288
10761 097/300 Read block 2305 len 36864
10762 098/300 Read block 610 len 36864
10763 099/300 Read block 3842 len 16384
10764 100/300 Read block 911 len 4096
10765 101/300 Write block 4327 len 32768 data: 4 3 2 3 3 3 3 3
10766 102/300 Write block 237 len 4096 data: 3
10767 103/300 Write block 3981 len 8192 data: 4 4
10768 104/300 Write block 3872 len 16384 data: 6 6 6 6
10769 105/300 Flush
10770 106/300 Write block 3404 len 4096 data: 6
10771 107/300 Read block 535 len 28672
10772 108/300 Read block 4045 len 24576
10773 109/300 Read block 1566 len 32768
10774 110/300 Read block 4637 len 8192
10775 111/300 Read block 1815 len 40960
10776 112/300 Write block 2234 len 40960 data: 3 3 3 4 4 4 3 3 3 3
10777 113/300 Read block 3108 len 40960
10778 114/300 Flush
10779 115/300 Write block 1691 len 8192 data: 4 4
10780 116/300 Read block 1392 len 32768
10781 117/300 Read block 2971 len 36864
10782 118/300 Read block 36 len 32768
10783 119/300 Write block 967 len 24576 data: 2 3 4 4 4 4
10784 120/300 Write block 571 len 16384 data: 2 2 2 3
10785 121/300 Flush
10786 122/300 Write block 3495 len 8192 data: 3 4
10787 123/300 Read block 1992 len 40960
10788 124/300 Write block 597 len 8192 data: 2 2
10789 125/300 Write block 1568 len 28672 data: 6 5 4 4 4 3 2
10790 126/300 Write block 1460 len 40960 data: 3 3 3 3 4 4 4 3 3 3
10791 127/300 Read block 465 len 40960
10792 128/300 Write block 1543 len 24576 data: 6 6 5 4 4 3
10793 129/300 Write block 782 len 4096 data: 5
10794 130/300 Read block 3150 len 20480
10795 131/300 Read block 1150 len 28672
10796 132/300 Write block 3765 len 8192 data: 2 2
10797 133/300 Write block 4095 len 32768 data: 4 3 3 3 2 3 3 4
10798 134/300 Write block 476 len 12288 data: 2 2 2
10799 135/300 Read block 4177 len 24576
10800 136/300 Flush
10801 137/300 Read block 1798 len 8192
10802 138/300 Read block 3018 len 40960
10803 139/300 Read block 3049 len 8192
10804 140/300 Read block 2083 len 32768
10805 141/300 Write block 2335 len 32768 data: 3 4 4 4 5 5 5 5
10806 142/300 Read block 1115 len 28672
10807 143/300 Read block 3849 len 40960
10808 144/300 Flush
10809 145/300 Read block 1213 len 24576
10810 146/300 Write block 4394 len 8192 data: 2 2
10811 147/300 Flush
10812 148/300 Write block 4417 len 4096 data: 3
10813 149/300 Read block 1614 len 20480
10814 150/300 Flush
10815 151/300 Read block 2393 len 24576
10816 152/300 Read block 3464 len 36864
10817 153/300 Write block 26 len 28672 data: 2 4 5 4 4 3 2
10818 154/300 Write block 717 len 32768 data: 2 4 3 2 2 2 2 2
10819 155/300 Write block 929 len 16384 data: 4 5 4 5
10820 156/300 Write block 4306 len 12288 data: 3 2 3
10821 157/300 Write block 4116 len 16384 data: 3 3 3 3
10822 158/300 Write block 2708 len 12288 data: 3 3 4
10823 159/300 Write block 532 len 36864 data: 5 5 6 5 5 5 5 5 5
10824 160/300 Read block 4506 len 28672
10825 161/300 Read block 2597 len 40960
10826 162/300 Read block 2200 len 28672
10827 163/300 Read block 2517 len 24576
10828 164/300 Write block 4701 len 28672 data: 4 4 3 3 2 2 2
10829 165/300 Write block 777 len 40960 data: 4 4 4 5 4 6 5 4 4 4
10830 166/300 Read block 3646 len 12288
10831 167/300 Read block 3761 len 36864
10832 168/300 Write block 3422 len 24576 data: 4 4 5 5 4 3
10833 169/300 Write block 4355 len 36864 data: 5 6 4 5 6 5 6 6 6
10834 170/300 Write block 3183 len 8192 data: 4 5
10835 171/300 Read block 3027 len 36864
10836 172/300 Flush
10837 173/300 Read block 1403 len 20480
10838 174/300 Read block 545 len 24576
10839 175/300 Write block 2211 len 12288 data: 3 3 5
10840 176/300 Read block 3257 len 40960
10841 177/300 Read block 1817 len 16384
10842 178/300 Read block 953 len 16384
10843 179/300 Write block 1848 len 4096 data: 6
10844 180/300 Read block 442 len 20480
10845 181/300 Write block 1774 len 28672 data: 3 3 2 2 2 2 2
10846 182/300 Write block 3184 len 12288 data: 6 5 6
10847 183/300 Read block 3296 len 12288
10848 184/300 Write block 1393 len 20480 data: 5 6 6 5 4
10849 185/300 Flush
10850 186/300 Write block 4394 len 4096 data: 3
10851 187/300 Write block 2092 len 8192 data: 4 5
10852 188/300 Read block 3850 len 20480
10853 189/300 Write block 565 len 8192 data: 3 2
10854 190/300 Read block 2201 len 28672
10855 191/300 Read block 4540 len 20480
10856 192/300 Write block 2906 len 20480 data: 2 2 2 2 2
10857 193/300 Read block 130 len 32768
10858 194/300 Read block 4047 len 32768
10859 195/300 Read block 2876 len 12288
10860 196/300 Write block 656 len 36864 data: 2 2 2 3 3 3 3 3 3
10861 197/300 Flush
10862 198/300 Flush
10863 199/300 Write block 2602 len 12288 data: 5 5 4
10864 200/300 Flush
10865 201/300 Write block 4959 len 20480 data: 6 4 4 3 3
10866 202/300 Read block 4375 len 20480
10867 203/300 Read block 690 len 16384
10868 204/300 Write block 209 len 28672 data: 2 2 2 2 2 2 2
10869 205/300 Write block 1298 len 28672 data: 2 2 2 2 3 3 4
10870 206/300 Read block 1003 len 28672
10871 207/300 Read block 2664 len 16384
10872 208/300 Write block 2644 len 20480 data: 5 6 6 6 6
10873 209/300 Read block 1142 len 40960
10874 210/300 Write block 3584 len 12288 data: 3 3 3
10875 211/300 Read block 4308 len 16384
10876 212/300 Write block 490 len 28672 data: 3 2 2 2 3 3 3
10877 213/300 Read block 461 len 20480
10878 214/300 Write block 227 len 24576 data: 3 3 3 3 3 4
10879 215/300 Read block 1579 len 4096
10880 216/300 Write block 2357 len 24576 data: 3 2 2 2 3 3
10881 217/300 Read block 2599 len 16384
10882 218/300 Read block 1148 len 16384
10883 219/300 Read block 15 len 24576
10884 220/300 Flush
10885 221/300 Write block 3826 len 36864 data: 2 2 2 2 2 2 2 2 2
10886 222/300 Read block 3103 len 28672
10887 223/300 Read block 4413 len 32768
10888 224/300 Read block 585 len 12288
10889 225/300 Write block 658 len 12288 data: 3 4 4
10890 226/300 Read block 4057 len 12288
10891 227/300 Write block 375 len 16384 data: 2 3 4 4
10892 228/300 Read block 2605 len 16384
10893 229/300 Read block 4713 len 16384
10894 230/300 Write block 102 len 20480 data: 6 6 7 6 6
10895 231/300 Read block 242 len 12288
10896 232/300 Read block 4086 len 24576
10897 233/300 Read block 622 len 32768
10898 234/300 Flush
10899 235/300 Read block 4625 len 16384
10900 236/300 Write block 1582 len 16384 data: 2 3 3 3
10901 237/300 Read block 1690 len 12288
10902 238/300 Read block 58 len 4096
10903 239/300 Read block 1889 len 20480
10904 240/300 Write block 241 len 28672 data: 4 4 4 4 4 4 5
10905 241/300 Write block 3697 len 36864 data: 4 5 4 4 5 4 4 3 3
10906 242/300 Flush
10907 243/300 Write block 1856 len 28672 data: 6 5 4 4 4 4 4
10908 244/300 Read block 4332 len 4096
10909 245/300 Read block 1345 len 32768
10910 246/300 Flush
10911 247/300 Write block 2870 len 28672 data: 3 3 3 3 3 3 4
10912 248/300 Read block 3242 len 16384
10913 249/300 Write block 1536 len 36864 data: 6 6 5 7 7 6 6 7 7
10914 250/300 Write block 2647 len 20480 data: 7 7 7 5 5
10915 251/300 Read block 1085 len 16384
10916 252/300 Write block 486 len 4096 data: 4
10917 253/300 Read block 1044 len 36864
10918 254/300 Flush
10919 255/300 Read block 4921 len 32768
10920 256/300 Write block 3394 len 32768 data: 4 6 4 3 3 5 6 6
10921 257/300 Read block 3957 len 16384
10922 258/300 Read block 4863 len 40960
10923 259/300 Write block 3810 len 8192 data: 5 7
10924 260/300 Read block 1885 len 32768
10925 261/300 Read block 3310 len 36864
10926 262/300 Flush
10927 263/300 Flush
10928 264/300 Read block 1168 len 20480
10929 265/300 Flush
10930 266/300 Read block 3602 len 8192
10931 267/300 Write block 1819 len 20480 data: 6 5 5 6 5
10932 268/300 Write block 4200 len 24576 data: 2 3 3 3 3 3
10933 269/300 Read block 2649 len 20480
10934 270/300 Read block 4237 len 16384
10935 271/300 Write block 1722 len 4096 data: 4
10936 272/300 Flush
10937 273/300 Write block 3162 len 36864 data: 2 3 3 4 4 4 4 4 3
10938 274/300 Write block 1859 len 28672 data: 5 5 5 5 4 2 2
10939 275/300 Write block 123 len 32768 data: 3 5 5 5 5 4 4 3
10940 276/300 Write block 3224 len 16384 data: 6 6 6 6
10941 277/300 Read block 1846 len 24576
10942 278/300 Write block 1841 len 40960 data: 4 5 4 5 5 5 6 7 5 6
10943 279/300 Read block 4854 len 12288
10944 280/300 Write block 1534 len 40960 data: 6 6 7 7 6 8 8 7 7 8
10945 281/300 Write block 2125 len 28672 data: 3 3 4 4 4 5 6
10946 282/300 Write block 4431 len 8192 data: 2 2
10947 283/300 Read block 18 len 32768
10948 284/300 Read block 3841 len 24576
10949 285/300 Read block 1116 len 16384
10950 286/300 Read block 2513 len 16384
10951 287/300 Read block 3041 len 4096
10952 288/300 Write block 4541 len 8192 data: 4 3
10953 289/300 Read block 2675 len 36864
10954 290/300 Write block 4624 len 8192 data: 3 3
10955 291/300 Flush
10956 292/300 Write block 4903 len 40960 data: 5 6 5 5 5 5 6 5 5 5
10957 293/300 Read block 3983 len 8192
10958 294/300 Write block 2889 len 8192 data: 3 2
10959 295/300 Flush
10960 296/300 Read block 699 len 28672
10961 297/300 Flush
10962 298/300 Write block 3383 len 20480 data: 3 3 3 3 2
10963 299/300 Read block 175 len 36864
10964 300/300 Read block 625 len 8192
10965 Main task has work to do, go find it
10966 got dsc Start(2)
10967 start 2
10968 Feb 12 01:36:42.045 INFO request completed, latency_us: 242, response_code: 204, uri: /start/cid/2, method: POST, req_id: 142a36f0-fefc-4066-a768-6b61ac1aaac7, remote_addr: 127.0.0.1:60456, local_addr: 127.0.0.1:9998
10969 Replay: started 2, returned:Ok(())
10970 ----------------------------------------------------------------
10971 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
10972 States: Active Active Offline
10973 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
10974 3318 Acked 4317 Read 3 Done Done New true
10975 3319 Acked 4318 Flush 0 Done Done New true
10976 3320 Acked 4319 Write 4 Done Done New true
10977 3321 Acked 4320 Write 8 Done Done New true
10978 3322 Acked 4321 Read 4 Done Done New true
10979 3323 Acked 4322 Write 7 Done Done New true
10980 3324 Acked 4323 Read 9 Done Done New false
10981 3325 Acked 4324 Read 10 Done Done New false
10982 3326 Acked 4325 Read 2 Done Done New false
10983 3327 Acked 4326 Read 10 Done Done New false
10984 3328 Acked 4327 Write 10 Done Done New false
10985 3329 Acked 4328 Read 3 Done Done New false
10986 3330 Acked 4329 Write 1 Done Done New false
10987 3331 Acked 4330 Read 1 Done Done New false
10988 3332 Acked 4331 Write 1 Done Done New false
10989 3333 Acked 4332 Write 4 Done Done New false
10990 3334 Acked 4333 Read 3 Done Done New false
10991 3335 Acked 4334 Write 6 Done Done New false
10992 3336 Acked 4335 Read 2 Done Done New false
10993 3337 Acked 4336 Read 7 Done Done New false
10994 3338 Acked 4337 Read 4 Done Done New false
10995 3339 Acked 4338 Read 5 Done Done New false
10996 3340 Acked 4339 Write 4 Done Done New false
10997 3341 Acked 4340 Write 1 Done Done New false
10998 3342 Acked 4341 Read 10 Done Done New false
10999 3343 Acked 4342 Write 4 Done Done New false
11000 3344 Acked 4343 Read 2 Done Done New false
11001 3345 Acked 4344 Read 1 Done Done New false
11002 3346 Acked 4345 Write 5 Done Done New false
11003 3347 Acked 4346 Write 5 Done Done New false
11004 3348 Acked 4347 Read 1 Done Done New false
11005 3349 Acked 4348 Write 7 Done Done New false
11006 3350 Acked 4349 Write 6 Done Done New false
11007 3351 Acked 4350 Write 1 Done Done New false
11008 3352 Acked 4351 Read 7 Done Done New false
11009 3353 Acked 4352 Read 9 Done Done New false
11010 3354 Acked 4353 Read 5 Done Done New false
11011 3355 Acked 4354 Flush 0 Done Done New false
11012 3356 Acked 4355 Write 4 Done Done New false
11013 3357 Acked 4356 Write 4 Done Done New false
11014 3358 Acked 4357 Write 4 Done Done New false
11015 3359 Acked 4358 Write 4 Done Done New false
11016 3360 Acked 4359 Read 5 Done Done New false
11017 3361 Acked 4360 Write 2 Done Done New false
11018 3362 Acked 4361 Read 9 Done Done New false
11019 3363 Acked 4362 Write 4 Done Done New false
11020 3364 Acked 4363 Write 3 Done Done New false
11021 3365 Acked 4364 Read 6 Done Done New false
11022 3366 Acked 4365 Flush 0 Done Done New false
11023 3367 Acked 4366 Read 8 Done Done New false
11024 3368 Acked 4367 Write 10 Done Done New false
11025 3369 Acked 4368 Read 6 Done Done New false
11026 3370 Acked 4369 Read 9 Done Done New false
11027 3371 Acked 4370 Flush 0 Done Done New false
11028 3372 Acked 4371 Flush 0 Done Done New false
11029 3373 Acked 4372 Write 3 Done Done New false
11030 3374 Acked 4373 Write 8 Done Done New false
11031 3375 Acked 4374 Read 2 Done Done New false
11032 3376 Acked 4375 Read 2 Done Done New false
11033 3377 Acked 4376 Read 6 Done Done New false
11034 3378 Acked 4377 Read 3 Done Done New false
11035 3379 Acked 4378 Write 1 Done Done New false
11036 3380 Acked 4379 Read 5 Done Done New false
11037 3381 Acked 4380 Write 3 Done Done New false
11038 3382 Acked 4381 Read 9 Done Done New false
11039 3383 Acked 4382 Read 5 Done Done New false
11040 3384 Acked 4383 Read 3 Done Done New false
11041 3385 Acked 4384 Read 4 Done Done New false
11042 3386 Acked 4385 Write 2 Done Done New false
11043 3387 Acked 4386 Write 9 Done Done New false
11044 3388 Acked 4387 Read 6 Done Done New false
11045 3389 Acked 4388 Read 9 Done Done New false
11046 3390 Acked 4389 Write 5 Done Done New false
11047 3391 Acked 4390 Write 9 Done Done New false
11048 3392 Acked 4391 Read 2 Done Done New false
11049 3393 Acked 4392 Write 3 Done Done New false
11050 3394 Acked 4393 Write 4 Done Done New false
11051 3395 Acked 4394 Read 1 Done Done New false
11052 3396 Acked 4395 Flush 0 Done Done New false
11053 3397 Acked 4396 Read 3 Done Done New false
11054 3398 Acked 4397 Read 4 Done Done New false
11055 3399 Acked 4398 Read 6 Done Done New false
11056 3400 Acked 4399 Write 3 Done Done New false
11057 3401 Acked 4400 Read 10 Done Done New false
11058 3402 Acked 4401 Read 9 Done Done New false
11059 3403 Acked 4402 Write 6 Done Done New false
11060 3404 Acked 4403 Read 7 Done Done New false
11061 3405 Acked 4404 Read 5 Done Done New false
11062 3406 Acked 4405 Write 9 Done Done New false
11063 3407 Acked 4406 Flush 0 Done Done New false
11064 3408 Acked 4407 Write 8 Done Done New false
11065 3409 Acked 4408 Read 7 Done Done New false
11066 3410 Acked 4409 Read 7 Done Done New false
11067 3411 Acked 4410 Write 7 Done Done New false
11068 3412 Acked 4411 Write 5 Done Done New false
11069 3413 Acked 4412 Read 3 Done Done New false
11070 3414 Acked 4413 Read 9 Done Done New false
11071 3415 Acked 4414 Read 9 Done Done New false
11072 3416 Acked 4415 Read 4 Done Done New false
11073 3417 Acked 4416 Read 1 Done Done New false
11074 3418 Acked 4417 Write 8 Done Done New false
11075 3419 Acked 4418 Write 1 Done Done New false
11076 3420 Acked 4419 Write 2 Done Done New false
11077 3421 Acked 4420 Write 4 Done Done New false
11078 3422 Acked 4421 Flush 0 Done Done New false
11079 3423 Acked 4422 Write 1 Done Done New false
11080 3424 Acked 4423 Read 7 Done Done New false
11081 3425 Acked 4424 Read 6 Done Done New false
11082 3426 Acked 4425 Read 8 Done Done New false
11083 3427 Acked 4426 Read 2 Done Done New false
11084 3428 Acked 4427 Read 10 Done Done New false
11085 3429 Acked 4428 Write 10 Done Done New false
11086 3430 Acked 4429 Read 10 Done Done New false
11087 3431 Acked 4430 Flush 0 Done Done New false
11088 3432 Acked 4431 Write 2 Done Done New false
11089 3433 Acked 4432 Read 8 Done Done New false
11090 3434 Acked 4433 Read 9 Done Done New false
11091 3435 Acked 4434 Read 8 Done Done New false
11092 3436 Acked 4435 Write 6 Done Done New false
11093 3437 Acked 4436 Write 4 Done Done New false
11094 3438 Acked 4437 Flush 0 Done Done New false
11095 3439 Acked 4438 Write 2 Done Done New false
11096 3440 Acked 4439 Read 10 Done Done New false
11097 3441 Acked 4440 Write 2 Done Done New false
11098 3442 Acked 4441 Write 7 Done Done New false
11099 3443 Acked 4442 Write 10 Done Done New false
11100 3444 Acked 4443 Read 10 Done Done New false
11101 3445 Acked 4444 Write 6 Done Done New false
11102 3446 Acked 4445 Write 1 Done Done New false
11103 3447 Acked 4446 Read 5 Done Done New false
11104 3448 Acked 4447 Read 7 Done Done New false
11105 3449 Acked 4448 Write 2 Done Done New false
11106 3450 Acked 4449 Write 8 Done Done New false
11107 3451 Acked 4450 Write 3 Done Done New false
11108 3452 Acked 4451 Read 6 Done Done New false
11109 3453 Acked 4452 Flush 0 Done Done New false
11110 3454 Acked 4453 Read 2 Done Done New false
11111 3455 Acked 4454 Read 10 Done Done New false
11112 3456 Acked 4455 Read 2 Done Done New false
11113 3457 Acked 4456 Read 8 Done Done New false
11114 3458 Acked 4457 Write 8 Done Done New false
11115 3459 Acked 4458 Read 7 Done Done New false
11116 3460 Acked 4459 Read 10 Done Done New false
11117 3461 Acked 4460 Flush 0 Done Done New false
11118 3462 Acked 4461 Read 6 Done Done New false
11119 3463 Acked 4462 Write 2 Done Done New false
11120 3464 Acked 4463 Flush 0 Done Done New false
11121 3465 Acked 4464 Write 1 Done Done New false
11122 3466 Acked 4465 Read 5 Done Done New false
11123 3467 Acked 4466 Flush 0 Done Done New false
11124 3468 Acked 4467 Read 6 Done Done New false
11125 3469 Acked 4468 Read 9 Done Done New false
11126 3470 Acked 4469 Write 7 Done Done New false
11127 3471 Acked 4470 Write 8 Done Done New false
11128 3472 Acked 4471 Write 4 Done Done New false
11129 3473 Acked 4472 Write 3 Done Done New false
11130 3474 Acked 4473 Write 4 Done Done New false
11131 3475 Acked 4474 Write 3 Done Done New false
11132 3476 Acked 4475 Write 9 Done Done New false
11133 3477 Acked 4476 Read 7 Done Done New false
11134 3478 Acked 4477 Read 10 Done Done New false
11135 3479 Acked 4478 Read 7 Done Done New false
11136 3480 Acked 4479 Read 6 Done Done New false
11137 3481 Acked 4480 Write 7 Done Done New false
11138 3482 Acked 4481 Write 10 Done Done New false
11139 3483 Acked 4482 Read 3 Done Done New false
11140 3484 Acked 4483 Read 9 Done Done New false
11141 3485 Acked 4484 Write 6 Done Done New false
11142 3486 Acked 4485 Write 9 Done Done New false
11143 3487 Acked 4486 Write 2 Done Done New false
11144 3488 Acked 4487 Read 9 Done Done New false
11145 3489 Acked 4488 Flush 0 Done Done New false
11146 3490 Acked 4489 Read 5 Done Done New false
11147 3491 Acked 4490 Read 6 Done Done New false
11148 3492 Acked 4491 Write 3 Done Done New false
11149 3493 Acked 4492 Read 10 Done Done New false
11150 3494 Acked 4493 Read 4 Done Done New false
11151 3495 Acked 4494 Read 4 Done Done New false
11152 3496 Acked 4495 Write 1 Done Done New false
11153 3497 Acked 4496 Read 5 Done Done New false
11154 3498 Acked 4497 Write 7 Done Done New false
11155 3499 Acked 4498 Write 3 Done Done New false
11156 3500 Acked 4499 Read 3 Done Done New false
11157 3501 Acked 4500 Write 5 Done Done New false
11158 3502 Acked 4501 Flush 0 Done Done New false
11159 3503 Acked 4502 Write 1 Done Done New false
11160 3504 Acked 4503 Write 2 Done Done New false
11161 3505 Acked 4504 Read 5 Done Done New false
11162 3506 Acked 4505 Write 2 Done Done New false
11163 3507 Acked 4506 Read 7 Done Done New false
11164 3508 Acked 4507 Read 5 Done Done New false
11165 3509 Acked 4508 Write 5 Done Done New false
11166 3510 Acked 4509 Read 8 Done Done New false
11167 3511 Acked 4510 Read 8 Done Done New false
11168 3512 Acked 4511 Read 3 Done Done New false
11169 3513 Acked 4512 Write 9 Done Done New false
11170 3514 Acked 4513 Flush 0 Done Done New false
11171 3515 Acked 4514 Flush 0 Done Done New false
11172 3516 Acked 4515 Write 3 Done Done New false
11173 3517 Acked 4516 Flush 0 Done Done New false
11174 3518 Acked 4517 Write 5 Done Done New false
11175 3519 Acked 4518 Read 5 Done Done New false
11176 3520 Acked 4519 Read 4 Done Done New false
11177 3521 Acked 4520 Write 7 Done Done New false
11178 3522 Acked 4521 Write 7 Done Done New false
11179 3523 Acked 4522 Read 7 Done Done New false
11180 3524 Acked 4523 Read 4 Done Done New false
11181 3525 Acked 4524 Write 5 Done Done New false
11182 3526 Acked 4525 Read 10 Done Done New false
11183 3527 Acked 4526 Write 3 Done Done New false
11184 3528 Acked 4527 Read 4 Done Done New false
11185 3529 Acked 4528 Write 7 Done Done New false
11186 3530 Acked 4529 Read 5 Done Done New false
11187 3531 Acked 4530 Write 6 Done Done New false
11188 3532 Acked 4531 Read 1 Done Done New false
11189 3533 Acked 4532 Write 6 Done Done New false
11190 3534 Acked 4533 Read 4 Done Done New false
11191 3535 Acked 4534 Read 4 Done Done New false
11192 3536 Acked 4535 Read 6 Done Done New false
11193 3537 Acked 4536 Flush 0 Done Done New false
11194 3538 Acked 4537 Write 9 Done Done New false
11195 3539 Acked 4538 Read 7 Done Done New false
11196 3540 Acked 4539 Read 8 Done Done New false
11197 3541 Acked 4540 Read 3 Done Done New false
11198 3542 Acked 4541 Write 3 Done Done New false
11199 3543 Acked 4542 Read 3 Done Done New false
11200 3544 Acked 4543 Write 4 Done Done New false
11201 3545 Acked 4544 Flush 0 Done Done New false
11202 3546 Acked 4545 Read 4 Done Done New false
11203 3547 Acked 4546 Read 4 Done Done New false
11204 3548 Acked 4547 Write 5 Done Done New false
11205 3549 Acked 4548 Read 3 Done Done New false
11206 3550 Acked 4549 Read 6 Done Done New false
11207 3551 Acked 4550 Read 8 Done Done New false
11208 3552 Acked 4551 Flush 0 Done Done New false
11209 3553 Acked 4552 Read 4 Done Done New false
11210 3554 Acked 4553 Write 4 Done Done New false
11211 3555 Acked 4554 Read 3 Done Done New false
11212 3556 Acked 4555 Read 1 Done Done New false
11213 3557 Acked 4556 Read 5 Done Done New false
11214 3558 Acked 4557 Write 7 Done Done New false
11215 3559 Acked 4558 Write 9 Done Done New false
11216 3560 Acked 4559 Flush 0 Done Done New false
11217 3561 Acked 4560 Write 7 Done Done New false
11218 3562 Acked 4561 Read 1 Done Done New false
11219 3563 Acked 4562 Read 8 Done Done New false
11220 3564 Acked 4563 Flush 0 Done Done New false
11221 3565 Acked 4564 Write 7 Done Done New false
11222 3566 Acked 4565 Read 4 Done Done New false
11223 3567 Acked 4566 Write 9 Done Done New false
11224 3568 Acked 4567 Write 5 Done Done New false
11225 3569 Acked 4568 Read 4 Done Done New false
11226 3570 Acked 4569 Write 1 Done Done New false
11227 3571 Acked 4570 Read 9 Done Done New false
11228 3572 Acked 4571 Flush 0 Done Done New false
11229 3573 Acked 4572 Read 8 Done Done New false
11230 3574 Acked 4573 Write 8 Done Done New false
11231 3575 Acked 4574 Read 4 Done Done New false
11232 3576 Acked 4575 Read 10 Done Done New false
11233 3577 Acked 4576 Write 2 Done Done New false
11234 3578 Acked 4577 Read 8 Done Done New false
11235 3579 Acked 4578 Read 9 Done Done New false
11236 3580 Acked 4579 Flush 0 Done Done New false
11237 3581 Acked 4580 Flush 0 Done Done New false
11238 3582 Acked 4581 Read 5 Done Done New false
11239 3583 Acked 4582 Flush 0 Done Done New false
11240 3584 Acked 4583 Read 2 Done Done New false
11241 3585 Acked 4584 Write 5 Done Done New false
11242 3586 Acked 4585 Write 6 Done Done New false
11243 3587 Acked 4586 Read 5 Done Done New false
11244 3588 Acked 4587 Read 4 Done Done New false
11245 3589 Acked 4588 Write 1 Done Done New false
11246 3590 Acked 4589 Flush 0 Done Done New false
11247 3591 Acked 4590 Write 9 Done Done New false
11248 3592 Acked 4591 Write 7 Done Done New false
11249 3593 Acked 4592 Write 8 Done Done New false
11250 3594 Acked 4593 Write 4 Done Done New false
11251 3595 Acked 4594 Read 6 Done Done New false
11252 3596 Acked 4595 Write 10 Done Done New false
11253 3597 Acked 4596 Read 3 Done Done New false
11254 3598 Acked 4597 Write 10 Done Done New false
11255 3599 Acked 4598 Write 7 Done Done New false
11256 3600 Acked 4599 Write 2 Done Done New false
11257 3601 Acked 4600 Read 8 Done Done New false
11258 3602 Acked 4601 Read 6 Done Done New false
11259 3603 Acked 4602 Read 4 Done Done New false
11260 3604 Acked 4603 Read 4 Done Done New false
11261 3605 Acked 4604 Read 1 Done Done New false
11262 3606 Acked 4605 Write 2 Done Done New false
11263 3607 Acked 4606 Read 9 Done Done New false
11264 3608 Acked 4607 Write 2 Done Done New false
11265 3609 Acked 4608 Flush 0 Done Done New false
11266 3610 Acked 4609 Write 10 Done Done New false
11267 3611 Acked 4610 Read 2 Done Done New false
11268 3612 Acked 4611 Write 2 Done Done New false
11269 3613 Acked 4612 Flush 0 Done Done New false
11270 3614 Acked 4613 Read 7 Done Done New false
11271 3615 Acked 4614 Flush 0 Done Done New false
11272 3616 Acked 4615 Write 5 Done Done New false
11273 3617 Acked 4616 Read 9 Done Done New false
11274 3618 Acked 4617 Read 2 Done Done New false
11275 STATES DS:0 DS:1 DS:2 TOTAL
11276 New 0 0 301 301
11277 Sent 0 0 0 0
11278 Done 301 301 0 602
11279 Skipped 0 0 0 0
11280 Error 0 0 0 0
11281 Last Flush: 4614 4614 4316
11282 Downstairs last five completed: 4316 4315 4314 4313 4312
11283 Upstairs last five completed: 3618 3617 3616 3615 3614
11284 CLIENT: Up:0 ds:301 act:2
112852024-02-12T01:36:42.701ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
112862024-02-12T01:36:42.701ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
112872024-02-12T01:36:42.701ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
112882024-02-12T01:36:42.701ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
112892024-02-12T01:36:42.701ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
112902024-02-12T01:36:42.701ZINFOcrucible: [2] client re-new 302 jobs since flush 4316 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
112912024-02-12T01:36:42.702ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
11292 [8830] got start action
11293 [8830] I am going to restart
11294 Starting downstairs at port 8830
11295 Make output file at "/tmp/dsc/downstairs-8830.txt"
11296 [8830][2] reports Starting
112972024-02-12T01:36:43.702ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
112982024-02-12T01:36:43.702ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
112992024-02-12T01:36:43.702ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
113002024-02-12T01:36:43.702ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
113012024-02-12T01:36:43.702ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
113022024-02-12T01:36:43.702ZINFOcrucible: [2] client re-new 302 jobs since flush 4316 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
113032024-02-12T01:36:43.702ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
11304 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1176)
11305 [8830][2] reports Running
113062024-02-12T01:36:44.703ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
113072024-02-12T01:36:44.703ZINFOcrucible: ds_connection connected = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
113082024-02-12T01:36:44.705ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
113092024-02-12T01:36:44.705ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
113102024-02-12T01:36:44.705ZINFOcrucible: send last flush ID to this DS: 4316 = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
113112024-02-12T01:36:44.705ZINFOcrucible: Replied this last flush ID: 4316 = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
113122024-02-12T01:36:44.705ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
113132024-02-12T01:36:44.705ZINFOcrucible: [2] Transition from Offline to Replay = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
113142024-02-12T01:36:44.705ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
113152024-02-12T01:36:44.705ZINFOcrucible: [2] Transition from Replay to Active = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
11316 ----------------------------------------------------------------
11317 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
11318 Downstairs last five completed: 4618 4617 4616 4615 4614
11319 Upstairs last five completed: 3619 3618 3617 3616 3615
11320 CLIENT: Up:0 ds:0 act:3
11321 Replay: All jobs finished, all DS active.
11322 Main task has work to do, go find it
11323 got dsc Stop(2)
11324 stop 2
11325 Feb 12 01:36:46.048 INFO request completed, latency_us: 406, [8830] Got stop action so:false kr:false
11326 response_code: 204, uri: /stop/cid/2, method: POST, req_id: fcbcdfcb-851e-4e46-a622-b3f3b0eec4db, remote_addr: 127.0.0.1:60456, local_addr: 127.0.0.1:9998
11327 Feb 12 01:36:46.049 INFO request completed{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2024-02-12T01:36:46.049736579Z","hostname":"ip-10-150-1-104.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"e1773b3e-b36f-4080-8b13-c0830f0d072e"}
11328 , latency_us: 146, response_code: 200, uri: /state/cid/2, method: GET, req_id: a2f60200-67b5-4bc3-944d-8348d0930044, remote_addr: 127.0.0.1:60456{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2024-02-12T01:36:46.049822927Z","hostname":"ip-10-150-1-104.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"e1773b3e-b36f-4080-8b13-c0830f0d072e"}
11329 , {"msg":"downstairs task for 2 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-02-12T01:36:46.049876979Z","hostname":"ip-10-150-1-104.us-west-2.compute.internal","pid":1164,"session_id":"e1773b3e-b36f-4080-8b13-c0830f0d072e"}
11330 local_addr: 127.0.0.1:9998{"msg":"Gone missing, transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-02-12T01:36:46.049947519Z","hostname":"ip-10-150-1-104.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"e1773b3e-b36f-4080-8b13-c0830f0d072e"}
11331 
113322024-02-12T01:36:46.049ZINFOcrucible: [2] client re-new 0 jobs since flush 4618 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
113332024-02-12T01:36:46.049ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
11334 [8830] Exited with: ExitStatus(unix_wait_status(9))
11335 [8830][2] reports Exit
113362024-02-12T01:36:47.051ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
113372024-02-12T01:36:47.051ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
113382024-02-12T01:36:47.051ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
113392024-02-12T01:36:47.051ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
113402024-02-12T01:36:47.051ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
113412024-02-12T01:36:47.051ZINFOcrucible: [2] client re-new 0 jobs since flush 4618 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
113422024-02-12T01:36:47.051ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
113432024-02-12T01:36:48.052ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
113442024-02-12T01:36:48.052ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
113452024-02-12T01:36:48.052ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
113462024-02-12T01:36:48.052ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
113472024-02-12T01:36:48.052ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
113482024-02-12T01:36:48.053ZINFOcrucible: [2] client re-new 0 jobs since flush 4618 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
113492024-02-12T01:36:48.053ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
113502024-02-12T01:36:49.054ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
113512024-02-12T01:36:49.054ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
113522024-02-12T01:36:49.054ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
113532024-02-12T01:36:49.054ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
113542024-02-12T01:36:49.054ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
113552024-02-12T01:36:49.054ZINFOcrucible: [2] client re-new 0 jobs since flush 4618 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
113562024-02-12T01:36:49.054ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
11357 Feb 12 01:36:50.052 INFO request completed, latency_us: 248, response_code: 200, uri: /state/cid/2, method: GET, req_id: dc5cf862-09b4-4224-be80-2c168f57862a, remote_addr: 127.0.0.1:60456, local_addr: 127.0.0.1:9998
11358 001/300 Read block 4981 len 16384
11359 002/300 Write block 1110 len 4096 data: 4
11360 003/300 Read block 3558 len 24576
113612024-02-12T01:36:50.055ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
113622024-02-12T01:36:50.055ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
113632024-02-12T01:36:50.055ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
113642024-02-12T01:36:50.055ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
113652024-02-12T01:36:50.055ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
113662024-02-12T01:36:50.055ZINFOcrucible: [2] client re-new 3 jobs since flush 4618 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
113672024-02-12T01:36:50.055ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
11368 004/300 Read block 1498 len 16384
11369 005/300 Read block 4210 len 28672
11370 006/300 Write block 842 len 40960 data: 3 3 3 3 3 3 2 2 3 4
11371 007/300 Flush
11372 008/300 Read block 3539 len 4096
11373 009/300 Write block 2553 len 40960 data: 3 3 3 2 3 3 4 4 4 4
11374 010/300 Read block 3533 len 32768
11375 011/300 Flush
11376 012/300 Read block 1332 len 16384
11377 013/300 Read block 1517 len 4096
11378 014/300 Read block 4874 len 28672
11379 015/300 Read block 4954 len 4096
11380 016/300 Read block 4175 len 16384
11381 017/300 Flush
11382 018/300 Write block 4179 len 36864 data: 3 4 3 3 3 4 3 4 3
11383 019/300 Write block 1800 len 24576 data: 2 3 3 4 4 4
11384 020/300 Read block 1286 len 28672
11385 021/300 Read block 2960 len 16384
11386 022/300 Read block 1228 len 20480
11387 023/300 Write block 483 len 20480 data: 2 3 3 5 3
11388 024/300 Read block 4952 len 24576
11389 025/300 Write block 406 len 32768 data: 4 3 4 5 5 6 5 4
11390 026/300 Read block 240 len 4096
11391 027/300 Read block 4721 len 28672
11392 028/300 Read block 1621 len 40960
11393 029/300 Write block 2988 len 32768 data: 3 3 3 3 2 3 3 3
11394 030/300 Flush
11395 031/300 Write block 4049 len 36864 data: 2 2 2 2 2 3 3 3 3
11396 032/300 Read block 2706 len 4096
11397 033/300 Write block 3028 len 40960 data: 2 2 4 3 3 3 3 3 4 4
11398 034/300 Read block 3606 len 36864
11399 035/300 Read block 579 len 28672
11400 036/300 Write block 2584 len 28672 data: 5 6 7 7 7 6 6
11401 037/300 Write block 4717 len 36864 data: 4 5 4 4 4 4 4 3 3
11402 038/300 Write block 1504 len 40960 data: 5 4 4 3 3 3 2 2 2 2
11403 039/300 Write block 1651 len 24576 data: 2 2 2 2 2 2
11404 040/300 Read block 2574 len 4096
11405 041/300 Write block 2535 len 4096 data: 2
11406 042/300 Read block 53 len 40960
11407 043/300 Write block 2344 len 16384 data: 4 3 2 2
11408 044/300 Read block 3347 len 4096
11409 045/300 Write block 1890 len 16384 data: 3 3 3 3
11410 046/300 Write block 4203 len 20480 data: 4 4 4 3 3
11411 047/300 Write block 1347 len 8192 data: 2 2
11412 048/300 Flush
11413 049/300 Write block 1994 len 4096 data: 4
11414 050/300 Flush
11415 051/300 Read block 1017 len 12288
11416 052/300 Flush
11417 053/300 Write block 4690 len 28672 data: 6 5 5 5 4 3 5
11418 054/300 Write block 1202 len 28672 data: 3 3 4 5 4 4 4
11419 055/300 Read block 1105 len 8192
11420 056/300 Write block 4243 len 28672 data: 2 2 2 3 3 3 4
11421 057/300 Read block 1820 len 16384
11422 058/300 Read block 2110 len 16384
11423 059/300 Read block 3227 len 32768
11424 060/300 Flush
11425 061/300 Read block 1724 len 16384
11426 062/300 Write block 536 len 16384 data: 6 6 6 6
11427 063/300 Write block 4957 len 28672 data: 6 6 7 5 5 4 4
11428 064/300 Flush
11429 065/300 Read block 4642 len 8192
11430 066/300 Write block 4475 len 36864 data: 6 6 5 4 4 2 2 4 4
11431 067/300 Read block 3982 len 40960
11432 068/300 Flush
11433 069/300 Read block 3927 len 4096
11434 070/300 Read block 3590 len 12288
11435 071/300 Read block 561 len 24576
11436 072/300 Read block 3942 len 12288
11437 073/300 Write block 3776 len 4096 data: 5
11438 074/300 Read block 613 len 28672
11439 075/300 Read block 2647 len 32768
11440 076/300 Read block 1843 len 16384
11441 077/300 Flush
11442 078/300 Flush
11443 079/300 Write block 2731 len 20480 data: 4 3 2 2 2
11444 080/300 Write block 3448 len 32768 data: 4 4 7 7 7 7 6 5
11445 081/300 Flush
11446 082/300 Write block 950 len 40960 data: 3 3 3 3 3 2 3 3 3 3
11447 083/300 Write block 1647 len 4096 data: 2
11448 084/300 Write block 802 len 32768 data: 6 6 5 4 4 4 4 4
11449 085/300 Write block 3107 len 4096 data: 3
11450 086/300 Read block 2294 len 8192
11451 087/300 Read block 1387 len 16384
11452 088/300 Flush
11453 089/300 Read block 2406 len 4096
11454 090/300 Write block 4574 len 36864 data: 3 3 3 3 3 3 3 3 2
11455 091/300 Read block 4363 len 8192
11456 092/300 Write block 2796 len 32768 data: 4 4 3 4 3 3 4 5
11457 093/300 Write block 1407 len 20480 data: 6 6 5 4 4
11458 094/300 Read block 1699 len 36864
11459 095/300 Write block 3625 len 20480 data: 3 3 2 2 3
11460 096/300 Write block 1704 len 8192 data: 5 6
11461 097/300 Read block 2425 len 12288
11462 098/300 Write block 3157 len 28672 data: 2 2 2 2 2 3 4
11463 099/300 Read block 2947 len 32768
11464 100/300 Read block 3474 len 28672
11465 101/300 Write block 743 len 32768 data: 4 4 5 4 6 5 4 3
11466 102/300 Read block 4243 len 4096
11467 103/300 Flush
11468 104/300 Read block 2423 len 8192
11469 105/300 Flush
11470 106/300 Flush
11471 107/300 Flush
11472 108/300 Write block 1419 len 28672 data: 2 3 3 4 5 4 4
11473 109/300 Write block 982 len 28672 data: 3 3 3 4 4 3 3
11474 110/300 Read block 1930 len 28672
11475 111/300 Read block 4797 len 24576
11476 112/300 Read block 2083 len 8192
11477 113/300 Write block 2204 len 32768 data: 6 5 4 4 5 4 3 4
11478 114/300 Read block 54 len 16384
11479 115/300 Read block 627 len 20480
11480 116/300 Flush
11481 117/300 Read block 2425 len 20480
11482 118/300 Read block 2809 len 24576
11483 119/300 Write block 4139 len 32768 data: 5 5 5 3 3 3 3 3
11484 120/300 Read block 3337 len 24576
11485 121/300 Flush
11486 122/300 Write block 3005 len 12288 data: 2 2 3
11487 123/300 Read block 4531 len 32768
11488 124/300 Read block 934 len 12288
11489 125/300 Write block 3950 len 32768 data: 3 3 3 3 4 3 3 5
11490 126/300 Read block 830 len 24576
11491 127/300 Read block 2415 len 28672
11492 128/300 Flush
11493 129/300 Read block 205 len 32768
11494 130/300 Write block 3915 len 36864 data: 2 2 2 2 2 2 2 3 4
11495 131/300 Write block 2378 len 12288 data: 5 4 4
11496 132/300 Write block 665 len 40960 data: 4 4 3 4 3 4 5 6 7 6
11497 133/300 Write block 1726 len 4096 data: 4
11498 134/300 Read block 4759 len 40960
11499 135/300 Write block 3980 len 16384 data: 4 5 5 3
11500 136/300 Read block 4927 len 32768
11501 137/300 Write block 2801 len 28672 data: 4 5 6 6 6 9 10
11502 138/300 Read block 1426 len 28672
11503 139/300 Write block 334 len 12288 data: 2 4 4
11504 140/300 Write block 4380 len 32768 data: 3 3 2 2 2 2 2 3
11505 141/300 Read block 3724 len 8192
11506 142/300 Write block 3708 len 12288 data: 2 2 2
11507 143/300 Read block 3689 len 16384
11508 144/300 Read block 522 len 20480
11509 145/300 Read block 3051 len 8192
11510 146/300 Write block 2558 len 40960 data: 4 5 5 5 5 4 4 4 4 4
11511 147/300 Write block 3972 len 16384 data: 2 3 3 3
11512 148/300 Write block 4221 len 20480 data: 4 5 5 5 5
11513 149/300 Flush
11514 150/300 Read block 4433 len 16384
11515 151/300 Read block 3516 len 36864
11516 152/300 Write block 2565 len 12288 data: 5 5 5
11517 153/300 Read block 322 len 20480
11518 154/300 Read block 1256 len 28672
11519 155/300 Write block 862 len 12288 data: 2 2 2
11520 156/300 Read block 4956 len 20480
11521 157/300 Read block 1949 len 20480
11522 158/300 Read block 522 len 36864
11523 159/300 Write block 97 len 4096 data: 4
11524 160/300 Read block 3903 len 20480
11525 161/300 Write block 3813 len 4096 data: 7
11526 162/300 Read block 3913 len 20480
11527 163/300 Read block 1951 len 40960
11528 164/300 Write block 3162 len 36864 data: 4 5 4 5 5 5 5 5 4
11529 165/300 Read block 4635 len 20480
11530 166/300 Write block 604 len 4096 data: 4
11531 167/300 Read block 490 len 40960
11532 168/300 Write block 3560 len 16384 data: 3 4 4 4
11533 169/300 Write block 3252 len 4096 data: 2
11534 170/300 Read block 2946 len 36864
11535 171/300 Read block 466 len 8192
11536 172/300 Flush
11537 173/300 Read block 537 len 40960
11538 174/300 Read block 1872 len 16384
11539 175/300 Read block 3607 len 8192
11540 176/300 Write block 4600 len 24576 data: 6 7 6 4 3 3
11541 177/300 Read block 3532 len 24576
11542 178/300 Read block 4077 len 12288
11543 179/300 Read block 2900 len 24576
11544 180/300 Write block 2966 len 40960 data: 3 2 2 2 2 2 2 3 4 3
11545 181/300 Read block 3042 len 8192
11546 182/300 Write block 3912 len 24576 data: 2 2 2 3 3 3
11547 183/300 Flush
11548 184/300 Write block 320 len 32768 data: 2 2 2 2 2 2 2 2
11549 185/300 Write block 2013 len 20480 data: 5 5 5 5 4
11550 186/300 Read block 4125 len 24576
11551 187/300 Read block 450 len 12288
11552 188/300 Read block 714 len 12288
11553 189/300 Write block 3147 len 16384 data: 2 2 3 3
11554 190/300 Read block 4360 len 36864
11555 191/300 Flush
11556 192/300 Read block 1305 len 32768
11557 193/300 Write block 874 len 28672 data: 3 4 4 4 5 4 6
11558 194/300 Read block 4110 len 32768
11559 195/300 Write block 1147 len 24576 data: 5 5 4 4 3 3
11560 196/300 Write block 4340 len 24576 data: 2 2 2 2 2 3
11561 197/300 Write block 4091 len 8192 data: 2 2
11562 198/300 Write block 609 len 12288 data: 3 2 2
11563 199/300 Read block 1679 len 24576
11564 200/300 Write block 2901 len 20480 data: 6 4 3 3 3
11565 201/300 Read block 569 len 8192
11566 202/300 Write block 3313 len 16384 data: 3 4 6 5
11567 203/300 Read block 606 len 24576
11568 204/300 Read block 482 len 28672
11569 205/300 Flush
11570 206/300 Read block 4032 len 40960
11571 207/300 Read block 805 len 20480
11572 208/300 Read block 4861 len 16384
11573 209/300 Write block 1742 len 16384 data: 2 2 2 2
11574 210/300 Write block 4167 len 32768 data: 3 4 5 5 4 4 5 6
11575 211/300 Write block 3038 len 16384 data: 5 5 5 5
11576 212/300 Write block 2477 len 24576 data: 5 6 4 3 3 5
11577 213/300 Flush
11578 214/300 Write block 1456 len 28672 data: 4 4 4 3 4 4 4
11579 215/300 Read block 3523 len 36864
11580 216/300 Read block 2996 len 32768
11581 217/300 Read block 1600 len 12288
11582 218/300 Flush
11583 219/300 Read block 2845 len 20480
11584 220/300 Write block 4447 len 8192 data: 2 3
11585 221/300 Read block 2539 len 20480
11586 222/300 Flush
11587 223/300 Write block 4549 len 24576 data: 6 4 4 4 3 3
11588 224/300 Write block 4211 len 32768 data: 2 2 2 2 2 2 2 3
11589 225/300 Read block 3061 len 24576
11590 226/300 Flush
11591 227/300 Read block 3996 len 36864
11592 228/300 Read block 127 len 4096
11593 229/300 Read block 4575 len 24576
11594 230/300 Read block 2054 len 28672
11595 231/300 Write block 1981 len 8192 data: 3 3
11596 232/300 Read block 4544 len 32768
11597 233/300 Write block 597 len 32768 data: 3 3 2 2 2 3 3 5
11598 234/300 Read block 3495 len 16384
11599 235/300 Write block 4022 len 12288 data: 8 7 6
11600 236/300 Flush
11601 237/300 Read block 1127 len 20480
11602 238/300 Write block 3454 len 8192 data: 7 6
11603 239/300 Flush
11604 240/300 Read block 510 len 16384
11605 241/300 Read block 611 len 8192
11606 242/300 Write block 1679 len 32768 data: 6 6 5 6 4 4 2 2
11607 243/300 Write block 3562 len 12288 data: 5 5 4
11608 244/300 Write block 3714 len 28672 data: 2 3 3 3 3 2 2
11609 245/300 Write block 549 len 16384 data: 4 4 2 2
11610 246/300 Write block 4389 len 24576 data: 4 4 5 3 2 4
11611 247/300 Read block 2175 len 16384
11612 248/300 Write block 2437 len 36864 data: 3 2 2 3 4 4 4 6 5
11613 249/300 Read block 1048 len 12288
11614 250/300 Read block 280 len 28672
11615 251/300 Read block 3198 len 36864
11616 252/300 Flush
11617 253/300 Read block 2979 len 8192
11618 254/300 Read block 971 len 28672
11619 255/300 Read block 4993 len 8192
11620 256/300 Read block 2903 len 20480
11621 257/300 Read block 552 len 8192
11622 258/300 Read block 3853 len 36864
11623 259/300 Write block 3669 len 28672 data: 2 2 2 2 4 4 4
11624 260/300 Read block 55 len 20480
11625 261/300 Write block 3335 len 36864 data: 2 2 2 2 2 2 2 2 3
11626 262/300 Write block 84 len 28672 data: 5 4 5 5 5 6 7
11627 263/300 Read block 4653 len 32768
11628 264/300 Write block 812 len 28672 data: 2 2 2 2 3 3 3
11629 265/300 Read block 1484 len 16384
11630 266/300 Read block 2840 len 16384
11631 267/300 Read block 595 len 32768
11632 268/300 Read block 2650 len 8192
11633 269/300 Write block 1463 len 8192 data: 4 5
11634 270/300 Read block 2456 len 20480
11635 271/300 Read block 4156 len 36864
11636 272/300 Read block 1833 len 40960
11637 273/300 Write block 4118 len 12288 data: 4 4 2
11638 274/300 Write block 2837 len 36864 data: 6 5 7 5 5 4 4 4 4
11639 275/300 Write block 1637 len 8192 data: 4 5
11640 276/300 Read block 873 len 24576
11641 277/300 Read block 1704 len 36864
11642 278/300 Read block 4553 len 20480
11643 279/300 Read block 1456 len 28672
11644 280/300 Flush
11645 281/300 Write block 696 len 36864 data: 4 3 3 3 4 3 3 3 3
11646 282/300 Flush
11647 283/300 Read block 4776 len 24576
11648 284/300 Write block 4993 len 4096 data: 3
11649 285/300 Write block 4721 len 12288 data: 5 5 5
11650 286/300 Write block 2863 len 8192 data: 4 4
11651 287/300 Read block 703 len 12288
11652 288/300 Write block 4863 len 12288 data: 3 3 3
11653 289/300 Read block 674 len 8192
11654 290/300 Write block 4834 len 32768 data: 3 3 3 4 4 3 3 3
11655 291/300 Read block 3436 len 36864
11656 292/300 Write block 1212 len 32768 data: 4 4 3 3 4 4 3 3
11657 293/300 Write block 4313 len 20480 data: 3 3 4 4 4
11658 294/300 Read block 2793 len 36864
11659 295/300 Write block 4452 len 24576 data: 4 4 6 5 4 4
11660 296/300 Write block 585 len 20480 data: 2 2 2 2 2
11661 297/300 Write block 3738 len 24576 data: 3 4 3 4 4 4
11662 298/300 Write block 2917 len 32768 data: 2 2 2 2 2 2 4 5
11663 299/300 Read block 2325 len 28672
11664 300/300 Write block 2801 len 12288 data: 5 6 7
11665 Main task has work to do, go find it
11666 got dsc Start(2)
11667 start 2
11668 Feb 12 01:36:50.398 INFO request completed, latency_us: 239, response_code: 204, uri: /start/cid/2, method: POST, req_id: 0954278d-cfe6-4e9f-b501-8cc4c876a051, remote_addr: 127.0.0.1:60456, local_addr: 127.0.0.1:9998
11669 Replay: started 2, returned:Ok(())
11670 ----------------------------------------------------------------
11671 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:300
11672 States: Active Active Offline
11673 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
11674 3620 Acked 4619 Read 4 Done Done New true
11675 3621 Acked 4620 Write 1 Done Done New true
11676 3622 Acked 4621 Read 6 Done Done New true
11677 3623 Acked 4622 Read 4 Done Done New false
11678 3624 Acked 4623 Read 7 Done Done New false
11679 3625 Acked 4624 Write 10 Done Done New false
11680 3626 Acked 4625 Flush 0 Done Done New false
11681 3627 Acked 4626 Read 1 Done Done New false
11682 3628 Acked 4627 Write 10 Done Done New false
11683 3629 Acked 4628 Read 8 Done Done New false
11684 3630 Acked 4629 Flush 0 Done Done New false
11685 3631 Acked 4630 Read 4 Done Done New false
11686 3632 Acked 4631 Read 1 Done Done New false
11687 3633 Acked 4632 Read 7 Done Done New false
11688 3634 Acked 4633 Read 1 Done Done New false
11689 3635 Acked 4634 Read 4 Done Done New false
11690 3636 Acked 4635 Flush 0 Done Done New false
11691 3637 Acked 4636 Write 9 Done Done New false
11692 3638 Acked 4637 Write 6 Done Done New false
11693 3639 Acked 4638 Read 7 Done Done New false
11694 3640 Acked 4639 Read 4 Done Done New false
11695 3641 Acked 4640 Read 5 Done Done New false
11696 3642 Acked 4641 Write 5 Done Done New false
11697 3643 Acked 4642 Read 6 Done Done New false
11698 3644 Acked 4643 Write 8 Done Done New false
11699 3645 Acked 4644 Read 1 Done Done New false
11700 3646 Acked 4645 Read 7 Done Done New false
11701 3647 Acked 4646 Read 10 Done Done New false
11702 3648 Acked 4647 Write 8 Done Done New false
11703 3649 Acked 4648 Flush 0 Done Done New false
11704 3650 Acked 4649 Write 9 Done Done New false
11705 3651 Acked 4650 Read 1 Done Done New false
11706 3652 Acked 4651 Write 10 Done Done New false
11707 3653 Acked 4652 Read 9 Done Done New false
11708 3654 Acked 4653 Read 7 Done Done New false
11709 3655 Acked 4654 Write 7 Done Done New false
11710 3656 Acked 4655 Write 9 Done Done New false
11711 3657 Acked 4656 Write 10 Done Done New false
11712 3658 Acked 4657 Write 6 Done Done New false
11713 3659 Acked 4658 Read 1 Done Done New false
11714 3660 Acked 4659 Write 1 Done Done New false
11715 3661 Acked 4660 Read 10 Done Done New false
11716 3662 Acked 4661 Write 4 Done Done New false
11717 3663 Acked 4662 Read 1 Done Done New false
11718 3664 Acked 4663 Write 4 Done Done New false
11719 3665 Acked 4664 Write 5 Done Done New false
11720 3666 Acked 4665 Write 2 Done Done New false
11721 3667 Acked 4666 Flush 0 Done Done New false
11722 3668 Acked 4667 Write 1 Done Done New false
11723 3669 Acked 4668 Flush 0 Done Done New false
11724 3670 Acked 4669 Read 3 Done Done New false
11725 3671 Acked 4670 Flush 0 Done Done New false
11726 3672 Acked 4671 Write 7 Done Done New false
11727 3673 Acked 4672 Write 7 Done Done New false
11728 3674 Acked 4673 Read 2 Done Done New false
11729 3675 Acked 4674 Write 7 Done Done New false
11730 3676 Acked 4675 Read 4 Done Done New false
11731 3677 Acked 4676 Read 4 Done Done New false
11732 3678 Acked 4677 Read 8 Done Done New false
11733 3679 Acked 4678 Flush 0 Done Done New false
11734 3680 Acked 4679 Read 4 Done Done New false
11735 3681 Acked 4680 Write 4 Done Done New false
11736 3682 Acked 4681 Write 7 Done Done New false
11737 3683 Acked 4682 Flush 0 Done Done New false
11738 3684 Acked 4683 Read 2 Done Done New false
11739 3685 Acked 4684 Write 9 Done Done New false
11740 3686 Acked 4685 Read 10 Done Done New false
11741 3687 Acked 4686 Flush 0 Done Done New false
11742 3688 Acked 4687 Read 1 Done Done New false
11743 3689 Acked 4688 Read 3 Done Done New false
11744 3690 Acked 4689 Read 6 Done Done New false
11745 3691 Acked 4690 Read 3 Done Done New false
11746 3692 Acked 4691 Write 1 Done Done New false
11747 3693 Acked 4692 Read 7 Done Done New false
11748 3694 Acked 4693 Read 8 Done Done New false
11749 3695 Acked 4694 Read 4 Done Done New false
11750 3696 Acked 4695 Flush 0 Done Done New false
11751 3697 Acked 4696 Flush 0 Done Done New false
11752 3698 Acked 4697 Write 5 Done Done New false
11753 3699 Acked 4698 Write 8 Done Done New false
11754 3700 Acked 4699 Flush 0 Done Done New false
11755 3701 Acked 4700 Write 10 Done Done New false
11756 3702 Acked 4701 Write 1 Done Done New false
11757 3703 Acked 4702 Write 8 Done Done New false
11758 3704 Acked 4703 Write 1 Done Done New false
11759 3705 Acked 4704 Read 2 Done Done New false
11760 3706 Acked 4705 Read 4 Done Done New false
11761 3707 Acked 4706 Flush 0 Done Done New false
11762 3708 Acked 4707 Read 1 Done Done New false
11763 3709 Acked 4708 Write 9 Done Done New false
11764 3710 Acked 4709 Read 2 Done Done New false
11765 3711 Acked 4710 Write 8 Done Done New false
11766 3712 Acked 4711 Write 5 Done Done New false
11767 3713 Acked 4712 Read 9 Done Done New false
11768 3714 Acked 4713 Write 5 Done Done New false
11769 3715 Acked 4714 Write 2 Done Done New false
11770 3716 Acked 4715 Read 3 Done Done New false
11771 3717 Acked 4716 Write 7 Done Done New false
11772 3718 Acked 4717 Read 8 Done Done New false
11773 3719 Acked 4718 Read 7 Done Done New false
11774 3720 Acked 4719 Write 8 Done Done New false
11775 3721 Acked 4720 Read 1 Done Done New false
11776 3722 Acked 4721 Flush 0 Done Done New false
11777 3723 Acked 4722 Read 2 Done Done New false
11778 3724 Acked 4723 Flush 0 Done Done New false
11779 3725 Acked 4724 Flush 0 Done Done New false
11780 3726 Acked 4725 Flush 0 Done Done New false
11781 3727 Acked 4726 Write 7 Done Done New false
11782 3728 Acked 4727 Write 7 Done Done New false
11783 3729 Acked 4728 Read 7 Done Done New false
11784 3730 Acked 4729 Read 6 Done Done New false
11785 3731 Acked 4730 Read 2 Done Done New false
11786 3732 Acked 4731 Write 8 Done Done New false
11787 3733 Acked 4732 Read 4 Done Done New false
11788 3734 Acked 4733 Read 5 Done Done New false
11789 3735 Acked 4734 Flush 0 Done Done New false
11790 3736 Acked 4735 Read 5 Done Done New false
11791 3737 Acked 4736 Read 6 Done Done New false
11792 3738 Acked 4737 Write 8 Done Done New false
11793 3739 Acked 4738 Read 6 Done Done New false
11794 3740 Acked 4739 Flush 0 Done Done New false
11795 3741 Acked 4740 Write 3 Done Done New false
11796 3742 Acked 4741 Read 8 Done Done New false
11797 3743 Acked 4742 Read 3 Done Done New false
11798 3744 Acked 4743 Write 8 Done Done New false
11799 3745 Acked 4744 Read 6 Done Done New false
11800 3746 Acked 4745 Read 7 Done Done New false
11801 3747 Acked 4746 Flush 0 Done Done New false
11802 3748 Acked 4747 Read 8 Done Done New false
11803 3749 Acked 4748 Write 9 Done Done New false
11804 3750 Acked 4749 Write 3 Done Done New false
11805 3751 Acked 4750 Write 10 Done Done New false
11806 3752 Acked 4751 Write 1 Done Done New false
11807 3753 Acked 4752 Read 10 Done Done New false
11808 3754 Acked 4753 Write 4 Done Done New false
11809 3755 Acked 4754 Read 8 Done Done New false
11810 3756 Acked 4755 Write 7 Done Done New false
11811 3757 Acked 4756 Read 7 Done Done New false
11812 3758 Acked 4757 Write 3 Done Done New false
11813 3759 Acked 4758 Write 8 Done Done New false
11814 3760 Acked 4759 Read 2 Done Done New false
11815 3761 Acked 4760 Write 3 Done Done New false
11816 3762 Acked 4761 Read 4 Done Done New false
11817 3763 Acked 4762 Read 5 Done Done New false
11818 3764 Acked 4763 Read 2 Done Done New false
11819 3765 Acked 4764 Write 10 Done Done New false
11820 3766 Acked 4765 Write 4 Done Done New false
11821 3767 Acked 4766 Write 5 Done Done New false
11822 3768 Acked 4767 Flush 0 Done Done New false
11823 3769 Acked 4768 Read 4 Done Done New false
11824 3770 Acked 4769 Read 9 Done Done New false
11825 3771 Acked 4770 Write 3 Done Done New false
11826 3772 Acked 4771 Read 5 Done Done New false
11827 3773 Acked 4772 Read 7 Done Done New false
11828 3774 Acked 4773 Write 3 Done Done New false
11829 3775 Acked 4774 Read 5 Done Done New false
11830 3776 Acked 4775 Read 5 Done Done New false
11831 3777 Acked 4776 Read 9 Done Done New false
11832 3778 Acked 4777 Write 1 Done Done New false
11833 3779 Acked 4778 Read 5 Done Done New false
11834 3780 Acked 4779 Write 1 Done Done New false
11835 3781 Acked 4780 Read 5 Done Done New false
11836 3782 Acked 4781 Read 10 Done Done New false
11837 3783 Acked 4782 Write 9 Done Done New false
11838 3784 Acked 4783 Read 5 Done Done New false
11839 3785 Acked 4784 Write 1 Done Done New false
11840 3786 Acked 4785 Read 10 Done Done New false
11841 3787 Acked 4786 Write 4 Done Done New false
11842 3788 Acked 4787 Write 1 Done Done New false
11843 3789 Acked 4788 Read 9 Done Done New false
11844 3790 Acked 4789 Read 2 Done Done New false
11845 3791 Acked 4790 Flush 0 Done Done New false
11846 3792 Acked 4791 Read 10 Done Done New false
11847 3793 Acked 4792 Read 4 Done Done New false
11848 3794 Acked 4793 Read 2 Done Done New false
11849 3795 Acked 4794 Write 6 Done Done New false
11850 3796 Acked 4795 Read 6 Done Done New false
11851 3797 Acked 4796 Read 3 Done Done New false
11852 3798 Acked 4797 Read 6 Done Done New false
11853 3799 Acked 4798 Write 10 Done Done New false
11854 3800 Acked 4799 Read 2 Done Done New false
11855 3801 Acked 4800 Write 6 Done Done New false
11856 3802 Acked 4801 Flush 0 Done Done New false
11857 3803 Acked 4802 Write 8 Done Done New false
11858 3804 Acked 4803 Write 5 Done Done New false
11859 3805 Acked 4804 Read 6 Done Done New false
11860 3806 Acked 4805 Read 3 Done Done New false
11861 3807 Acked 4806 Read 3 Done Done New false
11862 3808 Acked 4807 Write 4 Done Done New false
11863 3809 Acked 4808 Read 9 Done Done New false
11864 3810 Acked 4809 Flush 0 Done Done New false
11865 3811 Acked 4810 Read 8 Done Done New false
11866 3812 Acked 4811 Write 7 Done Done New false
11867 3813 Acked 4812 Read 8 Done Done New false
11868 3814 Acked 4813 Write 6 Done Done New false
11869 3815 Acked 4814 Write 6 Done Done New false
11870 3816 Acked 4815 Write 2 Done Done New false
11871 3817 Acked 4816 Write 3 Done Done New false
11872 3818 Acked 4817 Read 6 Done Done New false
11873 3819 Acked 4818 Write 5 Done Done New false
11874 3820 Acked 4819 Read 2 Done Done New false
11875 3821 Acked 4820 Write 4 Done Done New false
11876 3822 Acked 4821 Read 6 Done Done New false
11877 3823 Acked 4822 Read 7 Done Done New false
11878 3824 Acked 4823 Flush 0 Done Done New false
11879 3825 Acked 4824 Read 10 Done Done New false
11880 3826 Acked 4825 Read 5 Done Done New false
11881 3827 Acked 4826 Read 4 Done Done New false
11882 3828 Acked 4827 Write 4 Done Done New false
11883 3829 Acked 4828 Write 8 Done Done New false
11884 3830 Acked 4829 Write 4 Done Done New false
11885 3831 Acked 4830 Write 6 Done Done New false
11886 3832 Acked 4831 Flush 0 Done Done New false
11887 3833 Acked 4832 Write 7 Done Done New false
11888 3834 Acked 4833 Read 9 Done Done New false
11889 3835 Acked 4834 Read 8 Done Done New false
11890 3836 Acked 4835 Read 3 Done Done New false
11891 3837 Acked 4836 Flush 0 Done Done New false
11892 3838 Acked 4837 Read 5 Done Done New false
11893 3839 Acked 4838 Write 2 Done Done New false
11894 3840 Acked 4839 Read 5 Done Done New false
11895 3841 Acked 4840 Flush 0 Done Done New false
11896 3842 Acked 4841 Write 6 Done Done New false
11897 3843 Acked 4842 Write 8 Done Done New false
11898 3844 Acked 4843 Read 6 Done Done New false
11899 3845 Acked 4844 Flush 0 Done Done New false
11900 3846 Acked 4845 Read 9 Done Done New false
11901 3847 Acked 4846 Read 1 Done Done New false
11902 3848 Acked 4847 Read 6 Done Done New false
11903 3849 Acked 4848 Read 7 Done Done New false
11904 3850 Acked 4849 Write 2 Done Done New false
11905 3851 Acked 4850 Read 8 Done Done New false
11906 3852 Acked 4851 Write 8 Done Done New false
11907 3853 Acked 4852 Read 4 Done Done New false
11908 3854 Acked 4853 Write 3 Done Done New false
11909 3855 Acked 4854 Flush 0 Done Done New false
11910 3856 Acked 4855 Read 5 Done Done New false
11911 3857 Acked 4856 Write 2 Done Done New false
11912 3858 Acked 4857 Flush 0 Done Done New false
11913 3859 Acked 4858 Read 4 Done Done New false
11914 3860 Acked 4859 Read 2 Done Done New false
11915 3861 Acked 4860 Write 8 Done Done New false
11916 3862 Acked 4861 Write 3 Done Done New false
11917 3863 Acked 4862 Write 7 Done Done New false
11918 3864 Acked 4863 Write 4 Done Done New false
11919 3865 Acked 4864 Write 6 Done Done New false
11920 3866 Acked 4865 Read 4 Done Done New false
11921 3867 Acked 4866 Write 9 Done Done New false
11922 3868 Acked 4867 Read 3 Done Done New false
11923 3869 Acked 4868 Read 7 Done Done New false
11924 3870 Acked 4869 Read 9 Done Done New false
11925 3871 Acked 4870 Flush 0 Done Done New false
11926 3872 Acked 4871 Read 2 Done Done New false
11927 3873 Acked 4872 Read 7 Done Done New false
11928 3874 Acked 4873 Read 2 Done Done New false
11929 3875 Acked 4874 Read 5 Done Done New false
11930 3876 Acked 4875 Read 2 Done Done New false
11931 3877 Acked 4876 Read 9 Done Done New false
11932 3878 Acked 4877 Write 7 Done Done New false
11933 3879 Acked 4878 Read 5 Done Done New false
11934 3880 Acked 4879 Write 9 Done Done New false
11935 3881 Acked 4880 Write 7 Done Done New false
11936 3882 Acked 4881 Read 8 Done Done New false
11937 3883 Acked 4882 Write 7 Done Done New false
11938 3884 Acked 4883 Read 4 Done Done New false
11939 3885 Acked 4884 Read 4 Done Done New false
11940 3886 Acked 4885 Read 8 Done Done New false
11941 3887 Acked 4886 Read 2 Done Done New false
11942 3888 Acked 4887 Write 2 Done Done New false
11943 3889 Acked 4888 Read 5 Done Done New false
11944 3890 Acked 4889 Read 9 Done Done New false
11945 3891 Acked 4890 Read 10 Done Done New false
11946 3892 Acked 4891 Write 3 Done Done New false
11947 3893 Acked 4892 Write 9 Done Done New false
11948 3894 Acked 4893 Write 2 Done Done New false
11949 3895 Acked 4894 Read 6 Done Done New false
11950 3896 Acked 4895 Read 9 Done Done New false
11951 3897 Acked 4896 Read 5 Done Done New false
11952 3898 Acked 4897 Read 7 Done Done New false
11953 3899 Acked 4898 Flush 0 Done Done New false
11954 3900 Acked 4899 Write 9 Done Done New false
11955 3901 Acked 4900 Flush 0 Done Done New false
11956 3902 Acked 4901 Read 6 Done Done New false
11957 3903 Acked 4902 Write 1 Done Done New false
11958 3904 Acked 4903 Write 3 Done Done New false
11959 3905 Acked 4904 Write 2 Done Done New false
11960 3906 Acked 4905 Read 3 Done Done New false
11961 3907 Acked 4906 Write 3 Done Done New false
11962 3908 Acked 4907 Read 2 Done Done New false
11963 3909 Acked 4908 Write 8 Done Done New false
11964 3910 Acked 4909 Read 9 Done Done New false
11965 3911 Acked 4910 Write 8 Done Done New false
11966 3912 Acked 4911 Write 5 Done Done New false
11967 3913 Acked 4912 Read 9 Done Done New false
11968 3914 Acked 4913 Write 6 Done Done New false
11969 3915 Acked 4914 Write 5 Done Done New false
11970 3916 Acked 4915 Write 6 Done Done New false
11971 3917 Acked 4916 Write 8 Done Done New false
11972 3918 Acked 4917 Read 7 Done Done New false
11973 3919 Acked 4918 Write 3 Done Done New false
11974 STATES DS:0 DS:1 DS:2 TOTAL
11975 New 0 0 300 300
11976 Sent 0 0 0 0
11977 Done 300 300 0 600
11978 Skipped 0 0 0 0
11979 Error 0 0 0 0
11980 Last Flush: 4900 4900 4618
11981 Downstairs last five completed: 4618 4617 4616 4615 4614
11982 Upstairs last five completed: 3919 3918 3917 3916 3915
11983 CLIENT: Up:0 ds:300 act:2
119842024-02-12T01:36:51.057ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
119852024-02-12T01:36:51.057ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
119862024-02-12T01:36:51.057ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
119872024-02-12T01:36:51.057ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
119882024-02-12T01:36:51.057ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
119892024-02-12T01:36:51.057ZINFOcrucible: [2] client re-new 301 jobs since flush 4618 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
119902024-02-12T01:36:51.057ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
11991 [8830] got start action
11992 [8830] I am going to restart
11993 Starting downstairs at port 8830
11994 Make output file at "/tmp/dsc/downstairs-8830.txt"
11995 [8830][2] reports Starting
119962024-02-12T01:36:52.058ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
119972024-02-12T01:36:52.058ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
119982024-02-12T01:36:52.058ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
119992024-02-12T01:36:52.058ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
120002024-02-12T01:36:52.058ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
120012024-02-12T01:36:52.059ZINFOcrucible: [2] client re-new 301 jobs since flush 4618 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
120022024-02-12T01:36:52.059ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
12003 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1177)
12004 [8830][2] reports Running
120052024-02-12T01:36:53.059ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
120062024-02-12T01:36:53.059ZINFOcrucible: ds_connection connected = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
120072024-02-12T01:36:53.061ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
120082024-02-12T01:36:53.061ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
120092024-02-12T01:36:53.061ZINFOcrucible: send last flush ID to this DS: 4618 = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
120102024-02-12T01:36:53.061ZINFOcrucible: Replied this last flush ID: 4618 = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
120112024-02-12T01:36:53.061ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
120122024-02-12T01:36:53.061ZINFOcrucible: [2] Transition from Offline to Replay = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
120132024-02-12T01:36:53.061ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
120142024-02-12T01:36:53.061ZINFOcrucible: [2] Transition from Replay to Active = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
12015 ----------------------------------------------------------------
12016 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
12017 Downstairs last five completed: 4919 4918 4917 4916 4915
12018 Upstairs last five completed: 3920 3919 3918 3917 3916
12019 CLIENT: Up:0 ds:0 act:3
12020 Replay: All jobs finished, all DS active.
12021 Main task has work to do, go find it
12022 got dsc Stop(0)
12023 stop 0
12024 Feb 12 01:36:54.402 INFO request completed, latency_us: 434, response_code: 204, uri: /stop/cid/0, method: POST, [8810] Got stop action so:false kr:false
12025 req_id: 609555b7-3552-44c8-9b5c-d68f5f6a3451, remote_addr: 127.0.0.1:60456, local_addr: 127.0.0.1:9998
12026 Feb 12 01:36:54.403 INFO request completed, latency_us: 133, response_code: 200, uri: /state/cid/0, method: GET, req_id: 6c7cdcfc-ddf4-460c-9552-a393d6731790, remote_addr: 127.0.0.1:60456, local_addr: 127.0.0.1:9998
120272024-02-12T01:36:54.403ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
120282024-02-12T01:36:54.403ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
120292024-02-12T01:36:54.403ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
120302024-02-12T01:36:54.403ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
120312024-02-12T01:36:54.403ZINFOcrucible: [0] client re-new 0 jobs since flush 4919 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
120322024-02-12T01:36:54.403ZINFOcrucible: client task is exiting = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
12033 [8810] Exited with: ExitStatus(unix_wait_status(9))
12034 [8810][0] reports Exit
120352024-02-12T01:36:55.404ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
120362024-02-12T01:36:55.404ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
120372024-02-12T01:36:55.404ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
120382024-02-12T01:36:55.404ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
120392024-02-12T01:36:55.404ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
120402024-02-12T01:36:55.404ZINFOcrucible: [0] client re-new 0 jobs since flush 4919 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
120412024-02-12T01:36:55.404ZINFOcrucible: client task is exiting = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
120422024-02-12T01:36:56.405ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
120432024-02-12T01:36:56.405ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
120442024-02-12T01:36:56.405ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
120452024-02-12T01:36:56.405ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
120462024-02-12T01:36:56.405ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
120472024-02-12T01:36:56.405ZINFOcrucible: [0] client re-new 0 jobs since flush 4919 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
120482024-02-12T01:36:56.405ZINFOcrucible: client task is exiting = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
120492024-02-12T01:36:57.406ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
120502024-02-12T01:36:57.407ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
120512024-02-12T01:36:57.407ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
120522024-02-12T01:36:57.407ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
120532024-02-12T01:36:57.407ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
120542024-02-12T01:36:57.407ZINFOcrucible: [0] client re-new 0 jobs since flush 4919 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
120552024-02-12T01:36:57.407ZINFOcrucible: client task is exiting = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
12056 Feb 12 01:36:58.405 INFO request completed, latency_us: 222, response_code: 200, uri: /state/cid/0, method: GET, req_id: c8a2b403-95c9-4fc2-9510-34b4b187cf01, remote_addr: 127.0.0.1:60456, local_addr: 127.0.0.1:9998
12057 001/300 Read block 3105 len 28672
12058 002/300 Read block 1029 len 20480
12059 003/300 Read block 2129 len 32768
120602024-02-12T01:36:58.408ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
120612024-02-12T01:36:58.409ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
120622024-02-12T01:36:58.409ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
120632024-02-12T01:36:58.409ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
120642024-02-12T01:36:58.409ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
120652024-02-12T01:36:58.409ZINFOcrucible: [0] client re-new 3 jobs since flush 4919 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
120662024-02-12T01:36:58.409ZINFOcrucible: client task is exiting = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
12067 004/300 Read block 2177 len 16384
12068 005/300 Read block 1918 len 20480
12069 006/300 Read block 659 len 24576
12070 007/300 Read block 3083 len 4096
12071 008/300 Read block 571 len 12288
12072 009/300 Read block 1644 len 28672
12073 010/300 Read block 1598 len 28672
12074 011/300 Read block 1681 len 36864
12075 012/300 Read block 1616 len 36864
12076 013/300 Read block 4126 len 24576
12077 014/300 Read block 3996 len 4096
12078 015/300 Read block 4484 len 16384
12079 016/300 Flush
12080 017/300 Read block 3019 len 28672
12081 018/300 Read block 3869 len 40960
12082 019/300 Read block 2658 len 36864
12083 020/300 Write block 1090 len 32768 data: 4 5 5 4 5 4 2 2
12084 021/300 Write block 2697 len 40960 data: 3 3 3 4 3 3 3 2 2 3
12085 022/300 Write block 1410 len 12288 data: 5 5 3
12086 023/300 Write block 3026 len 32768 data: 2 2 3 3 5 4 4 4
12087 024/300 Read block 947 len 16384
12088 025/300 Write block 3449 len 16384 data: 5 8 8 8
12089 026/300 Write block 4483 len 4096 data: 5
12090 027/300 Read block 925 len 8192
12091 028/300 Read block 1277 len 8192
12092 029/300 Write block 55 len 12288 data: 4 3 3
12093 030/300 Read block 1760 len 12288
12094 031/300 Write block 2175 len 28672 data: 7 7 6 6 6 4 3
12095 032/300 Flush
12096 033/300 Read block 2318 len 4096
12097 034/300 Read block 1759 len 36864
12098 035/300 Write block 3396 len 28672 data: 5 4 4 6 7 7 5
12099 036/300 Read block 2647 len 16384
12100 037/300 Read block 4381 len 12288
12101 038/300 Write block 745 len 24576 data: 6 5 7 6 5 4
12102 039/300 Write block 4612 len 16384 data: 3 3 3 3
12103 040/300 Write block 2953 len 4096 data: 3
12104 041/300 Read block 4845 len 4096
12105 042/300 Flush
12106 043/300 Write block 732 len 16384 data: 4 3 2 2
12107 044/300 Write block 4039 len 12288 data: 5 5 3
12108 045/300 Write block 3190 len 4096 data: 6
12109 046/300 Write block 107 len 12288 data: 6 5 4
12110 047/300 Write block 3913 len 20480 data: 3 3 4 4 4
12111 048/300 Write block 3188 len 40960 data: 6 6 7 7 6 6 5 4 5 4
12112 049/300 Read block 4184 len 12288
12113 050/300 Write block 870 len 24576 data: 3 3 3 3 4 5
12114 051/300 Read block 4360 len 16384
12115 052/300 Read block 1100 len 24576
12116 053/300 Write block 1552 len 24576 data: 4 4 5 4 4 4
12117 054/300 Write block 4731 len 28672 data: 2 3 3 3 3 3 4
12118 055/300 Read block 4055 len 36864
12119 056/300 Write block 4029 len 8192 data: 4 3
12120 057/300 Write block 3734 len 12288 data: 4 4 3
12121 058/300 Read block 2213 len 28672
12122 059/300 Read block 550 len 4096
12123 060/300 Read block 974 len 12288
12124 061/300 Flush
12125 062/300 Flush
12126 063/300 Read block 4113 len 16384
12127 064/300 Flush
12128 065/300 Flush
12129 066/300 Write block 4393 len 28672 data: 3 5 3 2 2 2 2
12130 067/300 Read block 2829 len 12288
12131 068/300 Read block 3407 len 32768
12132 069/300 Read block 3848 len 28672
12133 070/300 Read block 183 len 24576
12134 071/300 Read block 305 len 40960
12135 072/300 Read block 2111 len 28672
12136 073/300 Write block 3394 len 4096 data: 5
12137 074/300 Read block 335 len 40960
12138 075/300 Write block 2821 len 20480 data: 4 4 5 6 6
12139 076/300 Read block 1889 len 40960
12140 077/300 Write block 4287 len 28672 data: 3 2 2 3 3 3 3
12141 078/300 Read block 2752 len 28672
12142 079/300 Write block 2035 len 28672 data: 2 2 2 2 2 2 2
12143 080/300 Read block 467 len 4096
12144 081/300 Write block 1515 len 24576 data: 3 3 3 3 2 2
12145 082/300 Read block 2155 len 4096
12146 083/300 Read block 3600 len 36864
12147 084/300 Read block 1162 len 36864
12148 085/300 Read block 1282 len 20480
12149 086/300 Write block 4881 len 40960 data: 4 4 3 4 4 4 4 3 3 3
12150 087/300 Write block 2355 len 28672 data: 3 3 4 3 3 3 4
12151 088/300 Read block 3376 len 20480
12152 089/300 Write block 4671 len 40960 data: 5 5 5 5 5 6 6 6 5 5
12153 090/300 Flush
12154 091/300 Flush
12155 092/300 Flush
12156 093/300 Write block 1585 len 12288 data: 4 3 3
12157 094/300 Read block 1096 len 24576
12158 095/300 Flush
12159 096/300 Read block 3075 len 32768
12160 097/300 Write block 3570 len 4096 data: 3
12161 098/300 Write block 318 len 32768 data: 2 2 3 3 3 3 3 3
12162 099/300 Write block 2167 len 28672 data: 3 3 3 4 6 6 4
12163 100/300 Flush
12164 101/300 Write block 58 len 36864 data: 3 2 2 2 2 2 3 3 3
12165 102/300 Read block 3418 len 8192
12166 103/300 Read block 1116 len 32768
12167 104/300 Write block 4295 len 4096 data: 4
12168 105/300 Read block 2671 len 4096
12169 106/300 Write block 94 len 4096 data: 5
12170 107/300 Write block 1144 len 24576 data: 6 6 6 6 6 5
12171 108/300 Write block 4122 len 40960 data: 3 3 3 3 3 3 2 2 2 2
12172 109/300 Read block 1496 len 24576
12173 110/300 Write block 355 len 12288 data: 4 6 6
12174 111/300 Read block 3504 len 40960
12175 112/300 Write block 299 len 40960 data: 4 5 4 3 3 3 3 4 3 4
12176 113/300 Read block 4797 len 16384
12177 114/300 Write block 2272 len 36864 data: 3 3 3 3 3 2 2 2 3
12178 115/300 Write block 4682 len 40960 data: 4 3 3 3 4 4 4 4 7 6
12179 116/300 Write block 4714 len 12288 data: 5 4 3
12180 117/300 Read block 2537 len 20480
12181 118/300 Write block 2723 len 8192 data: 5 5
12182 119/300 Read block 2599 len 32768
12183 120/300 Read block 3865 len 32768
12184 121/300 Read block 2875 len 16384
12185 122/300 Write block 2481 len 40960 data: 4 6 5 5 5 5 5 5 5 5
12186 123/300 Read block 4216 len 36864
12187 124/300 Write block 4200 len 40960 data: 3 4 4 5 5 5 4 4 2 2
12188 125/300 Read block 3495 len 32768
12189 126/300 Read block 2040 len 20480
12190 127/300 Write block 2645 len 16384 data: 7 7 8 8
12191 128/300 Read block 2127 len 28672
12192 129/300 Write block 3676 len 8192 data: 4 3
12193 130/300 Write block 804 len 20480 data: 6 5 5 5 5
12194 131/300 Read block 1869 len 28672
12195 132/300 Write block 4584 len 16384 data: 3 2 3 3
12196 133/300 Write block 3479 len 40960 data: 4 3 4 4 4 5 5 3 3 3
12197 134/300 Write block 4733 len 40960 data: 4 4 4 4 5 4 3 3 3 4
12198 135/300 Flush
12199 136/300 Write block 4843 len 20480 data: 3 3 3 3 3
12200 137/300 Flush
12201 138/300 Read block 4855 len 8192
12202 139/300 Read block 4294 len 20480
12203 140/300 Write block 4088 len 40960 data: 2 2 2 3 3 2 2 5 4 4
12204 141/300 Write block 942 len 4096 data: 4
12205 142/300 Read block 2446 len 16384
12206 143/300 Read block 1090 len 16384
12207 144/300 Write block 2730 len 8192 data: 4 5
12208 145/300 Read block 3316 len 8192
12209 146/300 Read block 3637 len 12288
12210 147/300 Read block 2405 len 32768
12211 148/300 Write block 305 len 12288 data: 4 5 4
12212 149/300 Read block 2888 len 40960
12213 150/300 Read block 2558 len 4096
12214 151/300 Read block 2489 len 28672
12215 152/300 Write block 172 len 4096 data: 3
12216 153/300 Read block 930 len 8192
12217 154/300 Flush
12218 155/300 Read block 3339 len 24576
12219 156/300 Flush
12220 157/300 Write block 3120 len 12288 data: 4 4 4
12221 158/300 Read block 1418 len 40960
12222 159/300 Read block 373 len 20480
12223 160/300 Read block 226 len 32768
12224 161/300 Flush
12225 162/300 Read block 1435 len 24576
12226 163/300 Write block 3624 len 40960 data: 3 4 4 3 3 4 3 3 3 3
12227 164/300 Write block 2918 len 28672 data: 3 3 3 3 3 5 6
12228 165/300 Write block 2002 len 8192 data: 3 3
12229 166/300 Read block 3346 len 20480
12230 167/300 Flush
12231 168/300 Read block 4024 len 12288
12232 169/300 Write block 893 len 16384 data: 3 3 3 3
12233 170/300 Write block 1474 len 36864 data: 3 3 4 5 5 4 3 3 3
12234 171/300 Write block 3180 len 40960 data: 5 6 4 5 7 6 7 6 7 7
12235 172/300 Write block 4915 len 40960 data: 6 6 4 4 5 4 3 4 4 4
12236 173/300 Flush
12237 174/300 Write block 1257 len 36864 data: 4 3 3 4 4 4 4 5 5
12238 175/300 Read block 2230 len 8192
12239 176/300 Write block 34 len 24576 data: 3 3 3 3 3 2
12240 177/300 Read block 4614 len 4096
12241 178/300 Write block 2406 len 4096 data: 5
12242 179/300 Read block 1863 len 8192
12243 180/300 Read block 309 len 20480
12244 181/300 Read block 1872 len 16384
12245 182/300 Read block 4459 len 32768
12246 183/300 Write block 2629 len 4096 data: 4
12247 184/300 Write block 3964 len 40960 data: 5 5 5 4 3 2 2 2 3 4
12248 185/300 Write block 3697 len 16384 data: 5 6 5 5
12249 186/300 Read block 741 len 36864
12250 187/300 Write block 368 len 28672 data: 2 3 3 3 3 3 3
12251 188/300 Read block 3437 len 16384
12252 189/300 Write block 4538 len 24576 data: 5 5 4 5 4 3
12253 190/300 Flush
12254 191/300 Write block 595 len 24576 data: 2 2 4 4 3 3
12255 192/300 Write block 1849 len 16384 data: 6 7 6 4
12256 193/300 Write block 3141 len 8192 data: 4 5
12257 194/300 Read block 1037 len 16384
12258 195/300 Read block 1839 len 16384
12259 196/300 Read block 1536 len 36864
12260 197/300 Write block 97 len 36864 data: 5 3 6 5 5 7 7 8 7
12261 198/300 Read block 3074 len 12288
12262 199/300 Read block 4824 len 36864
12263 200/300 Write block 4572 len 28672 data: 2 3 4 4 4 4 4
12264 201/300 Write block 3793 len 12288 data: 5 4 4
12265 202/300 Flush
12266 203/300 Read block 2273 len 4096
12267 204/300 Read block 4012 len 16384
12268 205/300 Write block 3251 len 16384 data: 2 3 2 2
12269 206/300 Write block 3575 len 40960 data: 4 3 3 3 3 3 2 2 3 4
12270 207/300 Write block 2469 len 36864 data: 4 5 5 7 7 6 5 5 6
12271 208/300 Read block 2588 len 20480
12272 209/300 Flush
12273 210/300 Write block 3079 len 8192 data: 3 4
12274 211/300 Read block 1788 len 36864
12275 212/300 Read block 1276 len 24576
12276 213/300 Read block 2364 len 20480
12277 214/300 Read block 4020 len 12288
12278 215/300 Read block 754 len 12288
12279 216/300 Read block 3764 len 32768
12280 217/300 Write block 3824 len 16384 data: 2 2 3 3
12281 218/300 Read block 1215 len 32768
12282 219/300 Read block 4771 len 20480
12283 220/300 Write block 351 len 40960 data: 4 4 4 4 5 7 7 6 5 3
12284 221/300 Read block 4503 len 40960
12285 222/300 Flush
12286 223/300 Write block 1146 len 16384 data: 7 7 7 6
12287 224/300 Write block 2906 len 28672 data: 3 3 3 3 3 2 2
12288 225/300 Flush
12289 226/300 Flush
12290 227/300 Write block 2647 len 16384 data: 9 9 8 6
12291 228/300 Write block 1668 len 32768 data: 4 4 4 5 5 5 5 5
12292 229/300 Write block 3096 len 12288 data: 2 2 4
12293 230/300 Write block 1622 len 8192 data: 6 5
12294 231/300 Read block 2190 len 4096
12295 232/300 Read block 3261 len 24576
12296 233/300 Read block 2718 len 36864
12297 234/300 Read block 2432 len 4096
12298 235/300 Write block 1565 len 24576 data: 6 7 6 7 6 5
12299 236/300 Flush
12300 237/300 Read block 4230 len 12288
12301 238/300 Read block 3832 len 20480
12302 239/300 Flush
12303 240/300 Read block 1923 len 28672
12304 241/300 Read block 2966 len 4096
12305 242/300 Read block 989 len 40960
12306 243/300 Read block 3440 len 36864
12307 244/300 Write block 4780 len 32768 data: 4 4 3 3 3 4 4 4
12308 245/300 Write block 1250 len 12288 data: 2 2 3
12309 246/300 Write block 2752 len 12288 data: 5 4 5
12310 247/300 Read block 4905 len 24576
12311 248/300 Read block 3695 len 36864
12312 249/300 Write block 3764 len 12288 data: 2 3 3
12313 250/300 Write block 3535 len 24576 data: 4 4 4 2 2 2
12314 251/300 Read block 4926 len 12288
12315 252/300 Read block 3565 len 36864
12316 253/300 Write block 4958 len 36864 data: 7 8 6 6 5 5 2 2 3
12317 254/300 Flush
12318 255/300 Write block 1021 len 40960 data: 4 4 4 2 2 2 2 2 2 2
12319 256/300 Write block 711 len 16384 data: 3 3 2 2
12320 257/300 Write block 3519 len 8192 data: 3 4
12321 258/300 Read block 3338 len 4096
12322 259/300 Read block 4854 len 8192
12323 260/300 Write block 341 len 36864 data: 3 3 3 2 2 2 2 3 4
12324 261/300 Write block 4138 len 32768 data: 4 6 6 6 4 4 4 4
12325 262/300 Write block 2088 len 20480 data: 3 3 3 3 5
12326 263/300 Write block 1676 len 16384 data: 6 6 7 7
12327 264/300 Read block 789 len 4096
12328 265/300 Flush
12329 266/300 Write block 2854 len 4096 data: 6
12330 267/300 Read block 2723 len 20480
12331 268/300 Flush
12332 269/300 Write block 3327 len 20480 data: 3 3 3 3 3
12333 270/300 Flush
12334 271/300 Write block 677 len 20480 data: 4 5 4 4 4
12335 272/300 Read block 1796 len 4096
12336 273/300 Read block 1257 len 36864
12337 274/300 Read block 477 len 28672
12338 275/300 Write block 1221 len 4096 data: 3
12339 276/300 Read block 4812 len 32768
12340 277/300 Write block 107 len 12288 data: 7 6 5
12341 278/300 Write block 956 len 12288 data: 4 4 4
12342 279/300 Write block 1807 len 16384 data: 5 5 6 6
12343 280/300 Read block 1826 len 12288
12344 281/300 Read block 757 len 4096
12345 282/300 Read block 1230 len 12288
12346 283/300 Read block 1629 len 20480
12347 284/300 Flush
12348 285/300 Write block 3939 len 40960 data: 2 2 3 3 3 3 4 4 4 3
12349 286/300 Read block 2512 len 28672
12350 287/300 Write block 1592 len 28672 data: 4 4 3 3 3 3 2
12351 288/300 Read block 2628 len 28672
12352 289/300 Write block 3742 len 20480 data: 5 5 4 4 4
12353 290/300 Write block 1184 len 16384 data: 3 2 2 2
12354 291/300 Read block 1222 len 4096
12355 292/300 Write block 887 len 28672 data: 3 3 2 2 2 3 4
12356 293/300 Read block 921 len 8192
12357 294/300 Flush
12358 295/300 Write block 2740 len 32768 data: 2 3 3 3 3 3 3 4
12359 296/300 Read block 4488 len 28672
12360 297/300 Write block 2390 len 20480 data: 2 2 2 2 3
12361 298/300 Read block 450 len 36864
12362 299/300 Read block 20 len 8192
12363 300/300 Flush
12364 Main task has work to do, go find it
12365 got dsc Start(0)
12366 start 0
12367 Feb 12 01:36:58.751 INFO request completed, latency_us: 373, response_code: 204, uri: /start/cid/0, method: POST, req_id: f1490133-8007-47ff-b901-18ed5e019062, remote_addr: 127.0.0.1:60456, local_addr: 127.0.0.1:9998
12368 Replay: started 0, returned:Ok(())
12369 ----------------------------------------------------------------
12370 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
12371 States: Offline Active Active
12372 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
12373 3921 Acked 4920 Read 7 New Done Done true
12374 3922 Acked 4921 Read 5 New Done Done true
12375 3923 Acked 4922 Read 8 New Done Done true
12376 3924 Acked 4923 Read 4 New Done Done false
12377 3925 Acked 4924 Read 5 New Done Done false
12378 3926 Acked 4925 Read 6 New Done Done false
12379 3927 Acked 4926 Read 1 New Done Done false
12380 3928 Acked 4927 Read 3 New Done Done false
12381 3929 Acked 4928 Read 7 New Done Done false
12382 3930 Acked 4929 Read 7 New Done Done false
12383 3931 Acked 4930 Read 9 New Done Done false
12384 3932 Acked 4931 Read 9 New Done Done false
12385 3933 Acked 4932 Read 6 New Done Done false
12386 3934 Acked 4933 Read 1 New Done Done false
12387 3935 Acked 4934 Read 4 New Done Done false
12388 3936 Acked 4935 Flush 0 New Done Done false
12389 3937 Acked 4936 Read 7 New Done Done false
12390 3938 Acked 4937 Read 10 New Done Done false
12391 3939 Acked 4938 Read 9 New Done Done false
12392 3940 Acked 4939 Write 8 New Done Done false
12393 3941 Acked 4940 Write 10 New Done Done false
12394 3942 Acked 4941 Write 3 New Done Done false
12395 3943 Acked 4942 Write 8 New Done Done false
12396 3944 Acked 4943 Read 4 New Done Done false
12397 3945 Acked 4944 Write 4 New Done Done false
12398 3946 Acked 4945 Write 1 New Done Done false
12399 3947 Acked 4946 Read 2 New Done Done false
12400 3948 Acked 4947 Read 2 New Done Done false
12401 3949 Acked 4948 Write 3 New Done Done false
12402 3950 Acked 4949 Read 3 New Done Done false
12403 3951 Acked 4950 Write 7 New Done Done false
12404 3952 Acked 4951 Flush 0 New Done Done false
12405 3953 Acked 4952 Read 1 New Done Done false
12406 3954 Acked 4953 Read 9 New Done Done false
12407 3955 Acked 4954 Write 7 New Done Done false
12408 3956 Acked 4955 Read 4 New Done Done false
12409 3957 Acked 4956 Read 3 New Done Done false
12410 3958 Acked 4957 Write 6 New Done Done false
12411 3959 Acked 4958 Write 4 New Done Done false
12412 3960 Acked 4959 Write 1 New Done Done false
12413 3961 Acked 4960 Read 1 New Done Done false
12414 3962 Acked 4961 Flush 0 New Done Done false
12415 3963 Acked 4962 Write 4 New Done Done false
12416 3964 Acked 4963 Write 3 New Done Done false
12417 3965 Acked 4964 Write 1 New Done Done false
12418 3966 Acked 4965 Write 3 New Done Done false
12419 3967 Acked 4966 Write 5 New Done Done false
12420 3968 Acked 4967 Write 10 New Done Done false
12421 3969 Acked 4968 Read 3 New Done Done false
12422 3970 Acked 4969 Write 6 New Done Done false
12423 3971 Acked 4970 Read 4 New Done Done false
12424 3972 Acked 4971 Read 6 New Done Done false
12425 3973 Acked 4972 Write 6 New Done Done false
12426 3974 Acked 4973 Write 7 New Done Done false
12427 3975 Acked 4974 Read 9 New Done Done false
12428 3976 Acked 4975 Write 2 New Done Done false
12429 3977 Acked 4976 Write 3 New Done Done false
12430 3978 Acked 4977 Read 7 New Done Done false
12431 3979 Acked 4978 Read 1 New Done Done false
12432 3980 Acked 4979 Read 3 New Done Done false
12433 3981 Acked 4980 Flush 0 New Done Done false
12434 3982 Acked 4981 Flush 0 New Done Done false
12435 3983 Acked 4982 Read 4 New Done Done false
12436 3984 Acked 4983 Flush 0 New Done Done false
12437 3985 Acked 4984 Flush 0 New Done Done false
12438 3986 Acked 4985 Write 7 New Done Done false
12439 3987 Acked 4986 Read 3 New Done Done false
12440 3988 Acked 4987 Read 8 New Done Done false
12441 3989 Acked 4988 Read 7 New Done Done false
12442 3990 Acked 4989 Read 6 New Done Done false
12443 3991 Acked 4990 Read 10 New Done Done false
12444 3992 Acked 4991 Read 7 New Done Done false
12445 3993 Acked 4992 Write 1 New Done Done false
12446 3994 Acked 4993 Read 10 New Done Done false
12447 3995 Acked 4994 Write 5 New Done Done false
12448 3996 Acked 4995 Read 10 New Done Done false
12449 3997 Acked 4996 Write 7 New Done Done false
12450 3998 Acked 4997 Read 7 New Done Done false
12451 3999 Acked 4998 Flush 0 New Done Done false
12452 4000 Acked 4999 Write 7 New Done Done false
12453 4001 Acked 5000 Read 1 New Done Done false
12454 4002 Acked 5001 Write 6 New Done Done false
12455 4003 Acked 5002 Read 1 New Done Done false
12456 4004 Acked 5003 Read 9 New Done Done false
12457 4005 Acked 5004 Read 9 New Done Done false
12458 4006 Acked 5005 Read 5 New Done Done false
12459 4007 Acked 5006 Write 10 New Done Done false
12460 4008 Acked 5007 Write 7 New Done Done false
12461 4009 Acked 5008 Read 5 New Done Done false
12462 4010 Acked 5009 Write 10 New Done Done false
12463 4011 Acked 5010 Flush 0 New Done Done false
12464 4012 Acked 5011 Flush 0 New Done Done false
12465 4013 Acked 5012 Flush 0 New Done Done false
12466 4014 Acked 5013 Write 3 New Done Done false
12467 4015 Acked 5014 Read 6 New Done Done false
12468 4016 Acked 5015 Flush 0 New Done Done false
12469 4017 Acked 5016 Read 8 New Done Done false
12470 4018 Acked 5017 Write 1 New Done Done false
12471 4019 Acked 5018 Write 8 New Done Done false
12472 4020 Acked 5019 Write 7 New Done Done false
12473 4021 Acked 5020 Flush 0 New Done Done false
12474 4022 Acked 5021 Write 9 New Done Done false
12475 4023 Acked 5022 Read 2 New Done Done false
12476 4024 Acked 5023 Read 8 New Done Done false
12477 4025 Acked 5024 Write 1 New Done Done false
12478 4026 Acked 5025 Read 1 New Done Done false
12479 4027 Acked 5026 Write 1 New Done Done false
12480 4028 Acked 5027 Write 6 New Done Done false
12481 4029 Acked 5028 Write 10 New Done Done false
12482 4030 Acked 5029 Read 6 New Done Done false
12483 4031 Acked 5030 Write 3 New Done Done false
12484 4032 Acked 5031 Read 10 New Done Done false
12485 4033 Acked 5032 Write 10 New Done Done false
12486 4034 Acked 5033 Read 4 New Done Done false
12487 4035 Acked 5034 Write 9 New Done Done false
12488 4036 Acked 5035 Write 10 New Done Done false
12489 4037 Acked 5036 Write 3 New Done Done false
12490 4038 Acked 5037 Read 5 New Done Done false
12491 4039 Acked 5038 Write 2 New Done Done false
12492 4040 Acked 5039 Read 8 New Done Done false
12493 4041 Acked 5040 Read 8 New Done Done false
12494 4042 Acked 5041 Read 4 New Done Done false
12495 4043 Acked 5042 Write 10 New Done Done false
12496 4044 Acked 5043 Read 9 New Done Done false
12497 4045 Acked 5044 Write 10 New Done Done false
12498 4046 Acked 5045 Read 8 New Done Done false
12499 4047 Acked 5046 Read 5 New Done Done false
12500 4048 Acked 5047 Write 4 New Done Done false
12501 4049 Acked 5048 Read 7 New Done Done false
12502 4050 Acked 5049 Write 2 New Done Done false
12503 4051 Acked 5050 Write 5 New Done Done false
12504 4052 Acked 5051 Read 7 New Done Done false
12505 4053 Acked 5052 Write 4 New Done Done false
12506 4054 Acked 5053 Write 10 New Done Done false
12507 4055 Acked 5054 Write 10 New Done Done false
12508 4056 Acked 5055 Flush 0 New Done Done false
12509 4057 Acked 5056 Write 5 New Done Done false
12510 4058 Acked 5057 Flush 0 New Done Done false
12511 4059 Acked 5058 Read 2 New Done Done false
12512 4060 Acked 5059 Read 5 New Done Done false
12513 4061 Acked 5060 Write 10 New Done Done false
12514 4062 Acked 5061 Write 1 New Done Done false
12515 4063 Acked 5062 Read 4 New Done Done false
12516 4064 Acked 5063 Read 4 New Done Done false
12517 4065 Acked 5064 Write 2 New Done Done false
12518 4066 Acked 5065 Read 2 New Done Done false
12519 4067 Acked 5066 Read 3 New Done Done false
12520 4068 Acked 5067 Read 8 New Done Done false
12521 4069 Acked 5068 Write 3 New Done Done false
12522 4070 Acked 5069 Read 10 New Done Done false
12523 4071 Acked 5070 Read 1 New Done Done false
12524 4072 Acked 5071 Read 7 New Done Done false
12525 4073 Acked 5072 Write 1 New Done Done false
12526 4074 Acked 5073 Read 2 New Done Done false
12527 4075 Acked 5074 Flush 0 New Done Done false
12528 4076 Acked 5075 Read 6 New Done Done false
12529 4077 Acked 5076 Flush 0 New Done Done false
12530 4078 Acked 5077 Write 3 New Done Done false
12531 4079 Acked 5078 Read 10 New Done Done false
12532 4080 Acked 5079 Read 5 New Done Done false
12533 4081 Acked 5080 Read 8 New Done Done false
12534 4082 Acked 5081 Flush 0 New Done Done false
12535 4083 Acked 5082 Read 6 New Done Done false
12536 4084 Acked 5083 Write 10 New Done Done false
12537 4085 Acked 5084 Write 7 New Done Done false
12538 4086 Acked 5085 Write 2 New Done Done false
12539 4087 Acked 5086 Read 5 New Done Done false
12540 4088 Acked 5087 Flush 0 New Done Done false
12541 4089 Acked 5088 Read 3 New Done Done false
12542 4090 Acked 5089 Write 4 New Done Done false
12543 4091 Acked 5090 Write 9 New Done Done false
12544 4092 Acked 5091 Write 10 New Done Done false
12545 4093 Acked 5092 Write 10 New Done Done false
12546 4094 Acked 5093 Flush 0 New Done Done false
12547 4095 Acked 5094 Write 9 New Done Done false
12548 4096 Acked 5095 Read 2 New Done Done false
12549 4097 Acked 5096 Write 6 New Done Done false
12550 4098 Acked 5097 Read 1 New Done Done false
12551 4099 Acked 5098 Write 1 New Done Done false
12552 4100 Acked 5099 Read 2 New Done Done false
12553 4101 Acked 5100 Read 5 New Done Done false
12554 4102 Acked 5101 Read 4 New Done Done false
12555 4103 Acked 5102 Read 8 New Done Done false
12556 4104 Acked 5103 Write 1 New Done Done false
12557 4105 Acked 5104 Write 10 New Done Done false
12558 4106 Acked 5105 Write 4 New Done Done false
12559 4107 Acked 5106 Read 9 New Done Done false
12560 4108 Acked 5107 Write 7 New Done Done false
12561 4109 Acked 5108 Read 4 New Done Done false
12562 4110 Acked 5109 Write 6 New Done Done false
12563 4111 Acked 5110 Flush 0 New Done Done false
12564 4112 Acked 5111 Write 6 New Done Done false
12565 4113 Acked 5112 Write 4 New Done Done false
12566 4114 Acked 5113 Write 2 New Done Done false
12567 4115 Acked 5114 Read 4 New Done Done false
12568 4116 Acked 5115 Read 4 New Done Done false
12569 4117 Acked 5116 Read 9 New Done Done false
12570 4118 Acked 5117 Write 9 New Done Done false
12571 4119 Acked 5118 Read 3 New Done Done false
12572 4120 Acked 5119 Read 9 New Done Done false
12573 4121 Acked 5120 Write 7 New Done Done false
12574 4122 Acked 5121 Write 3 New Done Done false
12575 4123 Acked 5122 Flush 0 New Done Done false
12576 4124 Acked 5123 Read 1 New Done Done false
12577 4125 Acked 5124 Read 4 New Done Done false
12578 4126 Acked 5125 Write 4 New Done Done false
12579 4127 Acked 5126 Write 10 New Done Done false
12580 4128 Acked 5127 Write 9 New Done Done false
12581 4129 Acked 5128 Read 5 New Done Done false
12582 4130 Acked 5129 Flush 0 New Done Done false
12583 4131 Acked 5130 Write 2 New Done Done false
12584 4132 Acked 5131 Read 9 New Done Done false
12585 4133 Acked 5132 Read 6 New Done Done false
12586 4134 Acked 5133 Read 5 New Done Done false
12587 4135 Acked 5134 Read 3 New Done Done false
12588 4136 Acked 5135 Read 3 New Done Done false
12589 4137 Acked 5136 Read 8 New Done Done false
12590 4138 Acked 5137 Write 4 New Done Done false
12591 4139 Acked 5138 Read 8 New Done Done false
12592 4140 Acked 5139 Read 5 New Done Done false
12593 4141 Acked 5140 Write 10 New Done Done false
12594 4142 Acked 5141 Read 10 New Done Done false
12595 4143 Acked 5142 Flush 0 New Done Done false
12596 4144 Acked 5143 Write 4 New Done Done false
12597 4145 Acked 5144 Write 7 New Done Done false
12598 4146 Acked 5145 Flush 0 New Done Done false
12599 4147 Acked 5146 Flush 0 New Done Done false
12600 4148 Acked 5147 Write 4 New Done Done false
12601 4149 Acked 5148 Write 8 New Done Done false
12602 4150 Acked 5149 Write 3 New Done Done false
12603 4151 Acked 5150 Write 2 New Done Done false
12604 4152 Acked 5151 Read 1 New Done Done false
12605 4153 Acked 5152 Read 6 New Done Done false
12606 4154 Acked 5153 Read 9 New Done Done false
12607 4155 Acked 5154 Read 1 New Done Done false
12608 4156 Acked 5155 Write 6 New Done Done false
12609 4157 Acked 5156 Flush 0 New Done Done false
12610 4158 Acked 5157 Read 3 New Done Done false
12611 4159 Acked 5158 Read 5 New Done Done false
12612 4160 Acked 5159 Flush 0 New Done Done false
12613 4161 Acked 5160 Read 7 New Done Done false
12614 4162 Acked 5161 Read 1 New Done Done false
12615 4163 Acked 5162 Read 10 New Done Done false
12616 4164 Acked 5163 Read 9 New Done Done false
12617 4165 Acked 5164 Write 8 New Done Done false
12618 4166 Acked 5165 Write 3 New Done Done false
12619 4167 Acked 5166 Write 3 New Done Done false
12620 4168 Acked 5167 Read 6 New Done Done false
12621 4169 Acked 5168 Read 9 New Done Done false
12622 4170 Acked 5169 Write 3 New Done Done false
12623 4171 Acked 5170 Write 6 New Done Done false
12624 4172 Acked 5171 Read 3 New Done Done false
12625 4173 Acked 5172 Read 9 New Done Done false
12626 4174 Acked 5173 Write 9 New Done Done false
12627 4175 Acked 5174 Flush 0 New Done Done false
12628 4176 Acked 5175 Write 10 New Done Done false
12629 4177 Acked 5176 Write 4 New Done Done false
12630 4178 Acked 5177 Write 2 New Done Done false
12631 4179 Acked 5178 Read 1 New Done Done false
12632 4180 Acked 5179 Read 2 New Done Done false
12633 4181 Acked 5180 Write 9 New Done Done false
12634 4182 Acked 5181 Write 8 New Done Done false
12635 4183 Acked 5182 Write 5 New Done Done false
12636 4184 Acked 5183 Write 4 New Done Done false
12637 4185 Acked 5184 Read 1 New Done Done false
12638 4186 Acked 5185 Flush 0 New Done Done false
12639 4187 Acked 5186 Write 1 New Done Done false
12640 4188 Acked 5187 Read 5 New Done Done false
12641 4189 Acked 5188 Flush 0 New Done Done false
12642 4190 Acked 5189 Write 5 New Done Done false
12643 4191 Acked 5190 Flush 0 New Done Done false
12644 4192 Acked 5191 Write 5 New Done Done false
12645 4193 Acked 5192 Read 1 New Done Done false
12646 4194 Acked 5193 Read 9 New Done Done false
12647 4195 Acked 5194 Read 7 New Done Done false
12648 4196 Acked 5195 Write 1 New Done Done false
12649 4197 Acked 5196 Read 8 New Done Done false
12650 4198 Acked 5197 Write 3 New Done Done false
12651 4199 Acked 5198 Write 3 New Done Done false
12652 4200 Acked 5199 Write 4 New Done Done false
12653 4201 Acked 5200 Read 3 New Done Done false
12654 4202 Acked 5201 Read 1 New Done Done false
12655 4203 Acked 5202 Read 3 New Done Done false
12656 4204 Acked 5203 Read 5 New Done Done false
12657 4205 Acked 5204 Flush 0 New Done Done false
12658 4206 Acked 5205 Write 10 New Done Done false
12659 4207 Acked 5206 Read 7 New Done Done false
12660 4208 Acked 5207 Write 7 New Done Done false
12661 4209 Acked 5208 Read 7 New Done Done false
12662 4210 Acked 5209 Write 5 New Done Done false
12663 4211 Acked 5210 Write 4 New Done Done false
12664 4212 Acked 5211 Read 1 New Done Done false
12665 4213 Acked 5212 Write 7 New Done Done false
12666 4214 Acked 5213 Read 2 New Done Done false
12667 4215 Acked 5214 Flush 0 New Done Done false
12668 4216 Acked 5215 Write 8 New Done Done false
12669 4217 Acked 5216 Read 7 New Done Done false
12670 4218 Acked 5217 Write 5 New Done Done false
12671 4219 Acked 5218 Read 9 New Done Done false
12672 4220 Acked 5219 Read 2 New Done Done false
12673 4221 Acked 5220 Flush 0 New Done Done false
12674 STATES DS:0 DS:1 DS:2 TOTAL
12675 New 301 0 0 301
12676 Sent 0 0 0 0
12677 Done 0 301 301 602
12678 Skipped 0 0 0 0
12679 Error 0 0 0 0
12680 Last Flush: 4919 5220 5220
12681 Downstairs last five completed: 4919 4918 4917 4916 4915
12682 Upstairs last five completed: 4221 4220 4219 4218 4217
12683 CLIENT: Up:0 ds:301 act:2
126842024-02-12T01:36:59.410ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
126852024-02-12T01:36:59.410ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
126862024-02-12T01:36:59.410ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
126872024-02-12T01:36:59.410ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
126882024-02-12T01:36:59.410ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
126892024-02-12T01:36:59.410ZINFOcrucible: [0] client re-new 301 jobs since flush 4919 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
126902024-02-12T01:36:59.410ZINFOcrucible: client task is exiting = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
12691 [8810] got start action
12692 [8810] I am going to restart
12693 Starting downstairs at port 8810
12694 Make output file at "/tmp/dsc/downstairs-8810.txt"
12695 [8810][0] reports Starting
126962024-02-12T01:37:00.412ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
126972024-02-12T01:37:00.412ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
126982024-02-12T01:37:00.412ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
126992024-02-12T01:37:00.412ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
127002024-02-12T01:37:00.412ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
127012024-02-12T01:37:00.412ZINFOcrucible: [0] client re-new 301 jobs since flush 4919 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
127022024-02-12T01:37:00.412ZINFOcrucible: client task is exiting = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
12703 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1178)
12704 [8810][0] reports Running
127052024-02-12T01:37:01.413ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
127062024-02-12T01:37:01.413ZINFOcrucible: ds_connection connected = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
127072024-02-12T01:37:01.415ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
127082024-02-12T01:37:01.415ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
127092024-02-12T01:37:01.415ZINFOcrucible: send last flush ID to this DS: 4919 = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
127102024-02-12T01:37:01.415ZINFOcrucible: Replied this last flush ID: 4919 = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
127112024-02-12T01:37:01.415ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
127122024-02-12T01:37:01.415ZINFOcrucible: [0] Transition from Offline to Replay = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
127132024-02-12T01:37:01.415ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
127142024-02-12T01:37:01.415ZINFOcrucible: [0] Transition from Replay to Active = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
12715 ----------------------------------------------------------------
12716 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
12717 Downstairs last five completed: 5220 5219 5218 5217 5216
12718 Upstairs last five completed: 4221 4220 4219 4218 4217
12719 CLIENT: Up:0 ds:0 act:3
12720 Replay: All jobs finished, all DS active.
12721 Main task has work to do, go find it
12722 got dsc Stop(2)
12723 stop 2
12724 Feb 12 01:37:02.755[8830] Got stop action so:false kr:false
12725 INFO request completed, latency_us: 356, response_code: 204, uri: /stop/cid/2, method: POST, req_id: a24682bf-1a68-4f58-866d-1b5b50e01fca, remote_addr: 127.0.0.1:60456, local_addr: 127.0.0.1:9998
127262024-02-12T01:37:02.755ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
127272024-02-12T01:37:02.755ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
127282024-02-12T01:37:02.755ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
12729 Feb 12 01:37:02.756{"msg":"Gone missing, transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-02-12T01:37:02.756046602Z","hostname":"ip-10-150-1-104.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"e1773b3e-b36f-4080-8b13-c0830f0d072e"}
12730 INFO request completed, latency_us: 163, response_code: 200, uri: /state/cid/2, method: GET, req_id: 91b9426b-77ca-4235-bf8a-40f90d71d81c, remote_addr: 127.0.0.1:60456, local_addr: 127.0{"msg":"[2] client re-new 0 jobs since flush 5220","v":0,"name":"crucible","level":30,"time":"2024-02-12T01:37:02.756137569Z","hostname":"ip-10-150-1-104.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"e1773b3e-b36f-4080-8b13-c0830f0d072e"}
12731 .0.1:9998
127322024-02-12T01:37:02.756ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
12733 [8830] Exited with: ExitStatus(unix_wait_status(9))
12734 [8830][2] reports Exit
127352024-02-12T01:37:03.757ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
127362024-02-12T01:37:03.757ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
127372024-02-12T01:37:03.757ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
127382024-02-12T01:37:03.757ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
127392024-02-12T01:37:03.757ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
127402024-02-12T01:37:03.757ZINFOcrucible: [2] client re-new 0 jobs since flush 5220 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
127412024-02-12T01:37:03.757ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
127422024-02-12T01:37:04.759ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
127432024-02-12T01:37:04.759ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
127442024-02-12T01:37:04.759ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
127452024-02-12T01:37:04.759ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
127462024-02-12T01:37:04.759ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
127472024-02-12T01:37:04.759ZINFOcrucible: [2] client re-new 0 jobs since flush 5220 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
127482024-02-12T01:37:04.759ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
127492024-02-12T01:37:05.760ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
127502024-02-12T01:37:05.760ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
127512024-02-12T01:37:05.760ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
127522024-02-12T01:37:05.760ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
127532024-02-12T01:37:05.760ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
127542024-02-12T01:37:05.760ZINFOcrucible: [2] client re-new 0 jobs since flush 5220 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
127552024-02-12T01:37:05.760ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
12756 Feb 12 01:37:06.758 INFO request completed, latency_us: 223, response_code: 200, uri: /state/cid/2, method: GET, req_id: 0ccc2065-87ce-4a3f-a11f-dbe636cbcec6, remote_addr: 127.0.0.1:60456, local_addr: 127.0.0.1:9998
12757 001/300 Write block 2126 len 40960 data: 4 5 5 5 6 7 5 4 2 2
12758 002/300 Read block 4223 len 40960
127592024-02-12T01:37:06.761ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
127602024-02-12T01:37:06.761ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
127612024-02-12T01:37:06.761ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
127622024-02-12T01:37:06.761ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
127632024-02-12T01:37:06.761ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
127642024-02-12T01:37:06.761ZINFOcrucible: [2] client re-new 2 jobs since flush 5220 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
127652024-02-12T01:37:06.761ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
12766 003/300 Write block 1499 len 4096 data: 4
12767 004/300 Read block 2765 len 24576
12768 005/300 Read block 4078 len 16384
12769 006/300 Flush
12770 007/300 Read block 2978 len 12288
12771 008/300 Read block 371 len 4096
12772 009/300 Read block 3478 len 24576
12773 010/300 Write block 197 len 4096 data: 4
12774 011/300 Read block 4281 len 24576
12775 012/300 Write block 1203 len 40960 data: 4 5 6 5 5 5 5 5 4 5
12776 013/300 Write block 3887 len 20480 data: 3 3 3 3 3
12777 014/300 Write block 2849 len 16384 data: 2 4 5 4
12778 015/300 Flush
12779 016/300 Write block 3935 len 4096 data: 2
12780 017/300 Write block 4571 len 8192 data: 2 3
12781 018/300 Read block 4977 len 4096
12782 019/300 Read block 4114 len 4096
12783 020/300 Flush
12784 021/300 Read block 4990 len 28672
12785 022/300 Read block 1878 len 28672
12786 023/300 Write block 3166 len 4096 data: 6
12787 024/300 Read block 4611 len 28672
12788 025/300 Read block 4490 len 32768
12789 026/300 Read block 3998 len 20480
12790 027/300 Read block 484 len 24576
12791 028/300 Read block 319 len 4096
12792 029/300 Write block 4576 len 32768 data: 5 5 5 4 4 4 3 3
12793 030/300 Read block 1799 len 8192
12794 031/300 Read block 780 len 36864
12795 032/300 Read block 1731 len 16384
12796 033/300 Flush
12797 034/300 Read block 606 len 24576
12798 035/300 Write block 2160 len 28672 data: 4 2 2 2 2 3 3
12799 036/300 Read block 1282 len 40960
12800 037/300 Read block 4676 len 28672
12801 038/300 Write block 2879 len 40960 data: 3 3 4 4 3 3 4 3 3 3
12802 039/300 Write block 1276 len 8192 data: 5 5
12803 040/300 Write block 4506 len 36864 data: 5 4 3 4 4 4 3 2 2
12804 041/300 Write block 2012 len 20480 data: 5 6 6 6 6
12805 042/300 Write block 3202 len 8192 data: 3 3
12806 043/300 Write block 1199 len 20480 data: 2 2 3 4 5
12807 044/300 Read block 4896 len 36864
12808 045/300 Write block 4673 len 28672 data: 6 6 6 7 7 7 6
12809 046/300 Read block 24 len 20480
12810 047/300 Write block 634 len 24576 data: 4 4 5 5 5 5
12811 048/300 Write block 633 len 20480 data: 3 5 5 6 6
12812 049/300 Flush
12813 050/300 Write block 1943 len 24576 data: 2 2 2 2 2 3
12814 051/300 Write block 3959 len 8192 data: 4 4
12815 052/300 Read block 465 len 32768
12816 053/300 Read block 1744 len 28672
12817 054/300 Write block 500 len 8192 data: 4 4
12818 055/300 Write block 2545 len 12288 data: 4 3 3
12819 056/300 Write block 2170 len 24576 data: 5 7 7 5 5 8
12820 057/300 Write block 622 len 8192 data: 4 4
12821 058/300 Write block 1637 len 20480 data: 5 6 4 4 3
12822 059/300 Write block 3328 len 8192 data: 4 4
12823 060/300 Write block 3567 len 40960 data: 3 3 3 4 3 3 3 4 5 4
12824 061/300 Read block 2585 len 16384
12825 062/300 Read block 2671 len 28672
12826 063/300 Write block 2758 len 28672 data: 7 6 7 7 6 6 4
12827 064/300 Read block 561 len 12288
12828 065/300 Write block 4965 len 20480 data: 3 4 3 3 5
12829 066/300 Read block 885 len 36864
12830 067/300 Read block 731 len 36864
12831 068/300 Read block 4655 len 36864
12832 069/300 Read block 1828 len 20480
12833 070/300 Read block 3066 len 12288
12834 071/300 Read block 2711 len 28672
12835 072/300 Read block 2587 len 8192
12836 073/300 Write block 3339 len 8192 data: 3 3
12837 074/300 Read block 4509 len 4096
12838 075/300 Write block 3262 len 4096 data: 2
12839 076/300 Flush
12840 077/300 Read block 189 len 12288
12841 078/300 Write block 500 len 16384 data: 5 5 4 4
12842 079/300 Read block 2797 len 12288
12843 080/300 Read block 184 len 8192
12844 081/300 Read block 4375 len 32768
12845 082/300 Write block 3511 len 24576 data: 4 3 3 3 3 2
12846 083/300 Write block 449 len 24576 data: 4 3 3 3 2 2
12847 084/300 Read block 2207 len 32768
12848 085/300 Read block 961 len 8192
12849 086/300 Read block 4303 len 4096
12850 087/300 Write block 3004 len 4096 data: 2
12851 088/300 Write block 4450 len 32768 data: 4 4 5 5 7 6 5 5
12852 089/300 Write block 2078 len 4096 data: 4
12853 090/300 Read block 1026 len 32768
12854 091/300 Write block 1138 len 4096 data: 3
12855 092/300 Write block 4480 len 20480 data: 3 3 5 6 4
12856 093/300 Read block 4134 len 28672
12857 094/300 Write block 1256 len 24576 data: 5 5 4 4 5 5
12858 095/300 Read block 4866 len 36864
12859 096/300 Write block 550 len 24576 data: 5 3 3 2 2 2
12860 097/300 Read block 4000 len 24576
12861 098/300 Write block 3865 len 16384 data: 4 4 4 4
12862 099/300 Write block 2229 len 28672 data: 2 2 2 2 3 4 4
12863 100/300 Flush
12864 101/300 Write block 471 len 28672 data: 3 3 3 2 2 3 3
12865 102/300 Write block 3304 len 8192 data: 3 3
12866 103/300 Write block 3486 len 12288 data: 4 4 4
12867 104/300 Read block 3002 len 28672
12868 105/300 Write block 947 len 28672 data: 3 3 3 4 4 4 4
12869 106/300 Write block 1827 len 16384 data: 5 5 6 5
12870 107/300 Read block 1931 len 40960
12871 108/300 Flush
12872 109/300 Read block 4013 len 8192
12873 110/300 Write block 4988 len 8192 data: 2 2
12874 111/300 Read block 4343 len 36864
12875 112/300 Flush
12876 113/300 Flush
12877 114/300 Read block 3457 len 36864
12878 115/300 Read block 4296 len 8192
12879 116/300 Read block 4931 len 16384
12880 117/300 Write block 2539 len 20480 data: 4 4 4 4 4
12881 118/300 Write block 1592 len 32768 data: 5 5 4 4 4 4 3 4
12882 119/300 Write block 4098 len 8192 data: 4 3
12883 120/300 Write block 1422 len 16384 data: 5 6 5 5
12884 121/300 Write block 2215 len 28672 data: 6 6 6 5 5 4 4
12885 122/300 Read block 1375 len 16384
12886 123/300 Read block 2712 len 20480
12887 124/300 Read block 775 len 12288
12888 125/300 Read block 3221 len 24576
12889 126/300 Write block 3434 len 40960 data: 6 5 5 3 2 2 3 3 3 2
12890 127/300 Read block 1946 len 16384
12891 128/300 Write block 1068 len 4096 data: 3
12892 129/300 Read block 1010 len 20480
12893 130/300 Write block 2332 len 28672 data: 2 3 3 4 5 5 5
12894 131/300 Write block 3000 len 28672 data: 3 3 2 2 3 3 3
12895 132/300 Write block 1256 len 24576 data: 6 6 5 5 6 6
12896 133/300 Write block 4183 len 4096 data: 4
12897 134/300 Read block 3771 len 24576
12898 135/300 Write block 666 len 36864 data: 5 4 5 4 5 6 7 8 7
12899 136/300 Write block 785 len 40960 data: 5 5 4 5 4 5 4 5 5 5
12900 137/300 Write block 7 len 24576 data: 2 2 2 2 2 2
12901 138/300 Write block 4453 len 36864 data: 6 8 7 6 6 3 3 3 2
12902 139/300 Write block 1783 len 36864 data: 2 2 3 3 4 4 4 4 4
12903 140/300 Write block 2694 len 28672 data: 3 3 2 4 4 4 5
12904 141/300 Flush
12905 142/300 Read block 737 len 40960
12906 143/300 Read block 3085 len 36864
12907 144/300 Write block 1035 len 12288 data: 2 2 2
12908 145/300 Read block 3561 len 28672
12909 146/300 Write block 4348 len 4096 data: 3
12910 147/300 Read block 4298 len 36864
12911 148/300 Write block 2736 len 32768 data: 3 3 3 5 3 4 4 4
12912 149/300 Write block 3712 len 32768 data: 2 2 3 4 4 4 4 3
12913 150/300 Write block 4444 len 28672 data: 3 3 3 3 4 3 5
12914 151/300 Read block 902 len 32768
12915 152/300 Write block 459 len 32768 data: 2 4 4 3 3 3 4 3
12916 153/300 Write block 1004 len 28672 data: 3 3 3 2 2 2 2
12917 154/300 Write block 465 len 28672 data: 5 4 3 3 3 3 4
12918 155/300 Read block 3876 len 28672
12919 156/300 Write block 1123 len 24576 data: 6 5 3 3 3 4
12920 157/300 Write block 4165 len 4096 data: 6
12921 158/300 Read block 1280 len 16384
12922 159/300 Read block 1479 len 12288
12923 160/300 Read block 3660 len 4096
12924 161/300 Write block 4610 len 28672 data: 4 5 4 4 4 4 3
12925 162/300 Write block 11 len 24576 data: 3 3 2 2 2 2
12926 163/300 Flush
12927 164/300 Read block 2922 len 8192
12928 165/300 Read block 1034 len 4096
12929 166/300 Write block 3477 len 4096 data: 2
12930 167/300 Read block 624 len 24576
12931 168/300 Read block 1407 len 32768
12932 169/300 Read block 3882 len 40960
12933 170/300 Read block 2710 len 4096
12934 171/300 Write block 714 len 20480 data: 3 2 2 3 5
12935 172/300 Write block 3936 len 12288 data: 2 2 2
12936 173/300 Read block 2748 len 20480
12937 174/300 Write block 2335 len 24576 data: 5 6 6 6 6 6
12938 175/300 Read block 2845 len 28672
12939 176/300 Read block 783 len 32768
12940 177/300 Read block 2446 len 32768
12941 178/300 Write block 843 len 36864 data: 4 4 4 4 4 3 3 4 5
12942 179/300 Read block 4736 len 32768
12943 180/300 Read block 2558 len 12288
12944 181/300 Read block 3759 len 4096
12945 182/300 Read block 1445 len 16384
12946 183/300 Read block 3926 len 12288
12947 184/300 Write block 1326 len 4096 data: 5
12948 185/300 Write block 2156 len 24576 data: 5 4 5 5 5 3
12949 186/300 Flush
12950 187/300 Write block 4501 len 36864 data: 5 4 5 5 6 6 5 4 5
12951 188/300 Read block 1333 len 28672
12952 189/300 Write block 2359 len 32768 data: 4 4 5 4 3 5 5 4
12953 190/300 Read block 3325 len 36864
12954 191/300 Read block 543 len 40960
12955 192/300 Flush
12956 193/300 Read block 3819 len 28672
12957 194/300 Read block 4177 len 36864
12958 195/300 Write block 909 len 24576 data: 5 4 5 4 4 3
12959 196/300 Read block 3577 len 12288
12960 197/300 Read block 4868 len 16384
12961 198/300 Read block 3720 len 8192
12962 199/300 Read block 4741 len 24576
12963 200/300 Read block 4337 len 4096
12964 201/300 Read block 3655 len 40960
12965 202/300 Read block 2163 len 40960
12966 203/300 Flush
12967 204/300 Write block 4551 len 8192 data: 5 5
12968 205/300 Read block 1089 len 32768
12969 206/300 Read block 132 len 20480
12970 207/300 Read block 3350 len 36864
12971 208/300 Write block 2509 len 32768 data: 4 4 3 3 2 2 2 2
12972 209/300 Write block 802 len 28672 data: 7 7 7 6 6 6 6
12973 210/300 Read block 1875 len 28672
12974 211/300 Read block 2161 len 20480
12975 212/300 Write block 1828 len 8192 data: 6 7
12976 213/300 Read block 4268 len 8192
12977 214/300 Write block 2804 len 40960 data: 7 7 10 11 8 9 8 7 6 6
12978 215/300 Read block 2254 len 32768
12979 216/300 Read block 1160 len 24576
12980 217/300 Write block 2894 len 24576 data: 3 3 4 5 4 3
12981 218/300 Read block 3308 len 32768
12982 219/300 Write block 2273 len 40960 data: 4 4 4 4 3 3 3 4 4 4
12983 220/300 Read block 712 len 32768
12984 221/300 Read block 389 len 20480
12985 222/300 Read block 4279 len 36864
12986 223/300 Flush
12987 224/300 Write block 888 len 4096 data: 4
12988 225/300 Read block 2506 len 28672
12989 226/300 Write block 404 len 24576 data: 4 3 5 4 5 6
12990 227/300 Read block 1361 len 8192
12991 228/300 Read block 425 len 36864
12992 229/300 Read block 4320 len 8192
12993 230/300 Write block 1446 len 8192 data: 3 4
12994 231/300 Write block 971 len 28672 data: 5 5 5 5 4 4 4
12995 232/300 Read block 3466 len 36864
12996 233/300 Read block 2337 len 24576
12997 234/300 Read block 4082 len 24576
12998 235/300 Write block 196 len 8192 data: 4 5
12999 236/300 Write block 949 len 24576 data: 4 5 5 5 5 4
13000 237/300 Read block 4529 len 20480
13001 238/300 Write block 278 len 20480 data: 2 2 2 2 2
13002 239/300 Write block 3996 len 32768 data: 4 4 6 6 5 6 4 4
13003 240/300 Read block 4731 len 40960
13004 241/300 Write block 4417 len 24576 data: 4 4 3 3 3 3
13005 242/300 Write block 390 len 36864 data: 4 4 5 4 4 5 5 5 3
13006 243/300 Read block 4119 len 8192
13007 244/300 Write block 2827 len 36864 data: 4 4 7 6 5 4 3 4 5
13008 245/300 Read block 1488 len 40960
13009 246/300 Write block 1258 len 36864 data: 6 6 7 7 5 5 6 6 4
13010 247/300 Read block 2543 len 20480
13011 248/300 Write block 3805 len 8192 data: 4 3
13012 249/300 Write block 4923 len 24576 data: 5 5 4 3 3 4
13013 250/300 Write block 3285 len 16384 data: 3 3 3 3
13014 251/300 Read block 540 len 16384
13015 252/300 Read block 249 len 32768
13016 253/300 Flush
13017 254/300 Read block 3771 len 36864
13018 255/300 Write block 2394 len 32768 data: 4 3 3 3 2 3 3 4
13019 256/300 Write block 4736 len 24576 data: 5 6 5 4 4 4
13020 257/300 Flush
13021 258/300 Read block 2604 len 8192
13022 259/300 Read block 4414 len 4096
13023 260/300 Write block 4554 len 24576 data: 4 2 2 2 3 3
13024 261/300 Read block 3922 len 32768
13025 262/300 Write block 2608 len 12288 data: 4 4 4
13026 263/300 Write block 3904 len 20480 data: 2 3 3 3 3
13027 264/300 Read block 3488 len 36864
13028 265/300 Read block 3694 len 12288
13029 266/300 Write block 4285 len 36864 data: 5 3 4 3 3 4 4 4 4
13030 267/300 Read block 4792 len 28672
13031 268/300 Read block 4547 len 16384
13032 269/300 Write block 3745 len 24576 data: 5 5 3 4 4 4
13033 270/300 Flush
13034 271/300 Write block 1078 len 16384 data: 3 3 4 4
13035 272/300 Read block 792 len 8192
13036 273/300 Read block 4180 len 32768
13037 274/300 Read block 2544 len 8192
13038 275/300 Read block 4793 len 32768
13039 276/300 Write block 2580 len 20480 data: 4 4 4 5 6
13040 277/300 Flush
13041 278/300 Read block 1794 len 16384
13042 279/300 Write block 271 len 24576 data: 4 3 2 2 2 2
13043 280/300 Flush
13044 281/300 Write block 2953 len 8192 data: 4 4
13045 282/300 Write block 697 len 40960 data: 4 4 4 5 4 4 4 4 3 3
13046 283/300 Write block 1989 len 12288 data: 3 3 3
13047 284/300 Write block 419 len 4096 data: 4
13048 285/300 Write block 2245 len 24576 data: 3 3 2 2 2 4
13049 286/300 Read block 998 len 28672
13050 287/300 Read block 945 len 28672
13051 288/300 Write block 4325 len 40960 data: 4 4 5 4 3 4 4 4 4 4
13052 289/300 Write block 3748 len 32768 data: 5 5 5 4 3 4 4 4
13053 290/300 Read block 4660 len 36864
13054 291/300 Read block 2396 len 12288
13055 292/300 Read block 3106 len 28672
13056 293/300 Read block 2871 len 12288
13057 294/300 Write block 4028 len 4096 data: 3
13058 295/300 Write block 1973 len 32768 data: 4 4 4 3 3 4 4 4
13059 296/300 Write block 4621 len 12288 data: 3 3 3
13060 297/300 Read block 850 len 12288
13061 298/300 Write block 2740 len 4096 data: 4
13062 299/300 Read block 74 len 28672
13063 300/300 Read block 1509 len 32768
13064 Main task has work to do, go find it
13065 got dsc Start(2)
13066 start 2
13067 Feb 12 01:37:07.104 INFO request completed, latency_us: 212, response_code: 204, uri: /start/cid/2, method: POST, req_id: c73c8e22-7b0c-4645-8a10-525c9cc7c1f6, remote_addr: 127.0.0.1:60456, local_addr: 127.0.0.1:9998
13068 Replay: started 2, returned:Ok(())
13069 ----------------------------------------------------------------
13070 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
13071 States: Active Active Offline
13072 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
13073 4222 Acked 5221 Write 10 Done Done New true
13074 4223 Acked 5222 Read 10 Done Done New true
13075 4224 Acked 5223 Write 1 Done Done New false
13076 4225 Acked 5224 Read 6 Done Done New false
13077 4226 Acked 5225 Read 4 Done Done New false
13078 4227 Acked 5226 Flush 0 Done Done New false
13079 4228 Acked 5227 Read 3 Done Done New false
13080 4229 Acked 5228 Read 1 Done Done New false
13081 4230 Acked 5229 Read 6 Done Done New false
13082 4231 Acked 5230 Write 1 Done Done New false
13083 4232 Acked 5231 Read 6 Done Done New false
13084 4233 Acked 5232 Write 10 Done Done New false
13085 4234 Acked 5233 Write 5 Done Done New false
13086 4235 Acked 5234 Write 4 Done Done New false
13087 4236 Acked 5235 Flush 0 Done Done New false
13088 4237 Acked 5236 Write 1 Done Done New false
13089 4238 Acked 5237 Write 2 Done Done New false
13090 4239 Acked 5238 Read 1 Done Done New false
13091 4240 Acked 5239 Read 1 Done Done New false
13092 4241 Acked 5240 Flush 0 Done Done New false
13093 4242 Acked 5241 Read 7 Done Done New false
13094 4243 Acked 5242 Read 7 Done Done New false
13095 4244 Acked 5243 Write 1 Done Done New false
13096 4245 Acked 5244 Read 7 Done Done New false
13097 4246 Acked 5245 Read 8 Done Done New false
13098 4247 Acked 5246 Read 5 Done Done New false
13099 4248 Acked 5247 Read 6 Done Done New false
13100 4249 Acked 5248 Read 1 Done Done New false
13101 4250 Acked 5249 Write 8 Done Done New false
13102 4251 Acked 5250 Read 2 Done Done New false
13103 4252 Acked 5251 Read 9 Done Done New false
13104 4253 Acked 5252 Read 4 Done Done New false
13105 4254 Acked 5253 Flush 0 Done Done New false
13106 4255 Acked 5254 Read 6 Done Done New false
13107 4256 Acked 5255 Write 7 Done Done New false
13108 4257 Acked 5256 Read 10 Done Done New false
13109 4258 Acked 5257 Read 7 Done Done New false
13110 4259 Acked 5258 Write 10 Done Done New false
13111 4260 Acked 5259 Write 2 Done Done New false
13112 4261 Acked 5260 Write 9 Done Done New false
13113 4262 Acked 5261 Write 5 Done Done New false
13114 4263 Acked 5262 Write 2 Done Done New false
13115 4264 Acked 5263 Write 5 Done Done New false
13116 4265 Acked 5264 Read 9 Done Done New false
13117 4266 Acked 5265 Write 7 Done Done New false
13118 4267 Acked 5266 Read 5 Done Done New false
13119 4268 Acked 5267 Write 6 Done Done New false
13120 4269 Acked 5268 Write 5 Done Done New false
13121 4270 Acked 5269 Flush 0 Done Done New false
13122 4271 Acked 5270 Write 6 Done Done New false
13123 4272 Acked 5271 Write 2 Done Done New false
13124 4273 Acked 5272 Read 8 Done Done New false
13125 4274 Acked 5273 Read 7 Done Done New false
13126 4275 Acked 5274 Write 2 Done Done New false
13127 4276 Acked 5275 Write 3 Done Done New false
13128 4277 Acked 5276 Write 6 Done Done New false
13129 4278 Acked 5277 Write 2 Done Done New false
13130 4279 Acked 5278 Write 5 Done Done New false
13131 4280 Acked 5279 Write 2 Done Done New false
13132 4281 Acked 5280 Write 10 Done Done New false
13133 4282 Acked 5281 Read 4 Done Done New false
13134 4283 Acked 5282 Read 7 Done Done New false
13135 4284 Acked 5283 Write 7 Done Done New false
13136 4285 Acked 5284 Read 3 Done Done New false
13137 4286 Acked 5285 Write 5 Done Done New false
13138 4287 Acked 5286 Read 9 Done Done New false
13139 4288 Acked 5287 Read 9 Done Done New false
13140 4289 Acked 5288 Read 9 Done Done New false
13141 4290 Acked 5289 Read 5 Done Done New false
13142 4291 Acked 5290 Read 3 Done Done New false
13143 4292 Acked 5291 Read 7 Done Done New false
13144 4293 Acked 5292 Read 2 Done Done New false
13145 4294 Acked 5293 Write 2 Done Done New false
13146 4295 Acked 5294 Read 1 Done Done New false
13147 4296 Acked 5295 Write 1 Done Done New false
13148 4297 Acked 5296 Flush 0 Done Done New false
13149 4298 Acked 5297 Read 3 Done Done New false
13150 4299 Acked 5298 Write 4 Done Done New false
13151 4300 Acked 5299 Read 3 Done Done New false
13152 4301 Acked 5300 Read 2 Done Done New false
13153 4302 Acked 5301 Read 8 Done Done New false
13154 4303 Acked 5302 Write 6 Done Done New false
13155 4304 Acked 5303 Write 6 Done Done New false
13156 4305 Acked 5304 Read 8 Done Done New false
13157 4306 Acked 5305 Read 2 Done Done New false
13158 4307 Acked 5306 Read 1 Done Done New false
13159 4308 Acked 5307 Write 1 Done Done New false
13160 4309 Acked 5308 Write 8 Done Done New false
13161 4310 Acked 5309 Write 1 Done Done New false
13162 4311 Acked 5310 Read 8 Done Done New false
13163 4312 Acked 5311 Write 1 Done Done New false
13164 4313 Acked 5312 Write 5 Done Done New false
13165 4314 Acked 5313 Read 7 Done Done New false
13166 4315 Acked 5314 Write 6 Done Done New false
13167 4316 Acked 5315 Read 9 Done Done New false
13168 4317 Acked 5316 Write 6 Done Done New false
13169 4318 Acked 5317 Read 6 Done Done New false
13170 4319 Acked 5318 Write 4 Done Done New false
13171 4320 Acked 5319 Write 7 Done Done New false
13172 4321 Acked 5320 Flush 0 Done Done New false
13173 4322 Acked 5321 Write 7 Done Done New false
13174 4323 Acked 5322 Write 2 Done Done New false
13175 4324 Acked 5323 Write 3 Done Done New false
13176 4325 Acked 5324 Read 7 Done Done New false
13177 4326 Acked 5325 Write 7 Done Done New false
13178 4327 Acked 5326 Write 4 Done Done New false
13179 4328 Acked 5327 Read 10 Done Done New false
13180 4329 Acked 5328 Flush 0 Done Done New false
13181 4330 Acked 5329 Read 2 Done Done New false
13182 4331 Acked 5330 Write 2 Done Done New false
13183 4332 Acked 5331 Read 9 Done Done New false
13184 4333 Acked 5332 Flush 0 Done Done New false
13185 4334 Acked 5333 Flush 0 Done Done New false
13186 4335 Acked 5334 Read 9 Done Done New false
13187 4336 Acked 5335 Read 2 Done Done New false
13188 4337 Acked 5336 Read 4 Done Done New false
13189 4338 Acked 5337 Write 5 Done Done New false
13190 4339 Acked 5338 Write 8 Done Done New false
13191 4340 Acked 5339 Write 2 Done Done New false
13192 4341 Acked 5340 Write 4 Done Done New false
13193 4342 Acked 5341 Write 7 Done Done New false
13194 4343 Acked 5342 Read 4 Done Done New false
13195 4344 Acked 5343 Read 5 Done Done New false
13196 4345 Acked 5344 Read 3 Done Done New false
13197 4346 Acked 5345 Read 6 Done Done New false
13198 4347 Acked 5346 Write 10 Done Done New false
13199 4348 Acked 5347 Read 4 Done Done New false
13200 4349 Acked 5348 Write 1 Done Done New false
13201 4350 Acked 5349 Read 5 Done Done New false
13202 4351 Acked 5350 Write 7 Done Done New false
13203 4352 Acked 5351 Write 7 Done Done New false
13204 4353 Acked 5352 Write 6 Done Done New false
13205 4354 Acked 5353 Write 1 Done Done New false
13206 4355 Acked 5354 Read 6 Done Done New false
13207 4356 Acked 5355 Write 9 Done Done New false
13208 4357 Acked 5356 Write 10 Done Done New false
13209 4358 Acked 5357 Write 6 Done Done New false
13210 4359 Acked 5358 Write 9 Done Done New false
13211 4360 Acked 5359 Write 9 Done Done New false
13212 4361 Acked 5360 Write 7 Done Done New false
13213 4362 Acked 5361 Flush 0 Done Done New false
13214 4363 Acked 5362 Read 10 Done Done New false
13215 4364 Acked 5363 Read 9 Done Done New false
13216 4365 Acked 5364 Write 3 Done Done New false
13217 4366 Acked 5365 Read 7 Done Done New false
13218 4367 Acked 5366 Write 1 Done Done New false
13219 4368 Acked 5367 Read 9 Done Done New false
13220 4369 Acked 5368 Write 8 Done Done New false
13221 4370 Acked 5369 Write 8 Done Done New false
13222 4371 Acked 5370 Write 7 Done Done New false
13223 4372 Acked 5371 Read 8 Done Done New false
13224 4373 Acked 5372 Write 8 Done Done New false
13225 4374 Acked 5373 Write 7 Done Done New false
13226 4375 Acked 5374 Write 7 Done Done New false
13227 4376 Acked 5375 Read 7 Done Done New false
13228 4377 Acked 5376 Write 6 Done Done New false
13229 4378 Acked 5377 Write 1 Done Done New false
13230 4379 Acked 5378 Read 4 Done Done New false
13231 4380 Acked 5379 Read 3 Done Done New false
13232 4381 Acked 5380 Read 1 Done Done New false
13233 4382 Acked 5381 Write 7 Done Done New false
13234 4383 Acked 5382 Write 6 Done Done New false
13235 4384 Acked 5383 Flush 0 Done Done New false
13236 4385 Acked 5384 Read 2 Done Done New false
13237 4386 Acked 5385 Read 1 Done Done New false
13238 4387 Acked 5386 Write 1 Done Done New false
13239 4388 Acked 5387 Read 6 Done Done New false
13240 4389 Acked 5388 Read 8 Done Done New false
13241 4390 Acked 5389 Read 10 Done Done New false
13242 4391 Acked 5390 Read 1 Done Done New false
13243 4392 Acked 5391 Write 5 Done Done New false
13244 4393 Acked 5392 Write 3 Done Done New false
13245 4394 Acked 5393 Read 5 Done Done New false
13246 4395 Acked 5394 Write 6 Done Done New false
13247 4396 Acked 5395 Read 7 Done Done New false
13248 4397 Acked 5396 Read 8 Done Done New false
13249 4398 Acked 5397 Read 8 Done Done New false
13250 4399 Acked 5398 Write 9 Done Done New false
13251 4400 Acked 5399 Read 8 Done Done New false
13252 4401 Acked 5400 Read 3 Done Done New false
13253 4402 Acked 5401 Read 1 Done Done New false
13254 4403 Acked 5402 Read 4 Done Done New false
13255 4404 Acked 5403 Read 3 Done Done New false
13256 4405 Acked 5404 Write 1 Done Done New false
13257 4406 Acked 5405 Write 6 Done Done New false
13258 4407 Acked 5406 Flush 0 Done Done New false
13259 4408 Acked 5407 Write 9 Done Done New false
13260 4409 Acked 5408 Read 7 Done Done New false
13261 4410 Acked 5409 Write 8 Done Done New false
13262 4411 Acked 5410 Read 9 Done Done New false
13263 4412 Acked 5411 Read 10 Done Done New false
13264 4413 Acked 5412 Flush 0 Done Done New false
13265 4414 Acked 5413 Read 7 Done Done New false
13266 4415 Acked 5414 Read 9 Done Done New false
13267 4416 Acked 5415 Write 6 Done Done New false
13268 4417 Acked 5416 Read 3 Done Done New false
13269 4418 Acked 5417 Read 4 Done Done New false
13270 4419 Acked 5418 Read 2 Done Done New false
13271 4420 Acked 5419 Read 6 Done Done New false
13272 4421 Acked 5420 Read 1 Done Done New false
13273 4422 Acked 5421 Read 10 Done Done New false
13274 4423 Acked 5422 Read 10 Done Done New false
13275 4424 Acked 5423 Flush 0 Done Done New false
13276 4425 Acked 5424 Write 2 Done Done New false
13277 4426 Acked 5425 Read 8 Done Done New false
13278 4427 Acked 5426 Read 5 Done Done New false
13279 4428 Acked 5427 Read 9 Done Done New false
13280 4429 Acked 5428 Write 8 Done Done New false
13281 4430 Acked 5429 Write 7 Done Done New false
13282 4431 Acked 5430 Read 7 Done Done New false
13283 4432 Acked 5431 Read 5 Done Done New false
13284 4433 Acked 5432 Write 2 Done Done New false
13285 4434 Acked 5433 Read 2 Done Done New false
13286 4435 Acked 5434 Write 10 Done Done New false
13287 4436 Acked 5435 Read 8 Done Done New false
13288 4437 Acked 5436 Read 6 Done Done New false
13289 4438 Acked 5437 Write 6 Done Done New false
13290 4439 Acked 5438 Read 8 Done Done New false
13291 4440 Acked 5439 Write 10 Done Done New false
13292 4441 Acked 5440 Flush 0 Done Done New false
13293 4442 Acked 5441 Read 8 Done Done New false
13294 4443 Acked 5442 Read 5 Done Done New false
13295 4444 Acked 5443 Read 9 Done Done New false
13296 4445 Acked 5444 Flush 0 Done Done New false
13297 4446 Acked 5445 Write 1 Done Done New false
13298 4447 Acked 5446 Read 7 Done Done New false
13299 4448 Acked 5447 Write 6 Done Done New false
13300 4449 Acked 5448 Read 2 Done Done New false
13301 4450 Acked 5449 Read 9 Done Done New false
13302 4451 Acked 5450 Read 2 Done Done New false
13303 4452 Acked 5451 Write 2 Done Done New false
13304 4453 Acked 5452 Write 7 Done Done New false
13305 4454 Acked 5453 Read 9 Done Done New false
13306 4455 Acked 5454 Read 6 Done Done New false
13307 4456 Acked 5455 Read 6 Done Done New false
13308 4457 Acked 5456 Write 2 Done Done New false
13309 4458 Acked 5457 Write 6 Done Done New false
13310 4459 Acked 5458 Read 5 Done Done New false
13311 4460 Acked 5459 Write 5 Done Done New false
13312 4461 Acked 5460 Write 8 Done Done New false
13313 4462 Acked 5461 Read 10 Done Done New false
13314 4463 Acked 5462 Write 6 Done Done New false
13315 4464 Acked 5463 Write 9 Done Done New false
13316 4465 Acked 5464 Read 2 Done Done New false
13317 4466 Acked 5465 Write 9 Done Done New false
13318 4467 Acked 5466 Read 10 Done Done New false
13319 4468 Acked 5467 Write 9 Done Done New false
13320 4469 Acked 5468 Read 5 Done Done New false
13321 4470 Acked 5469 Write 2 Done Done New false
13322 4471 Acked 5470 Write 6 Done Done New false
13323 4472 Acked 5471 Write 4 Done Done New false
13324 4473 Acked 5472 Read 4 Done Done New false
13325 4474 Acked 5473 Read 8 Done Done New false
13326 4475 Acked 5474 Flush 0 Done Done New false
13327 4476 Acked 5475 Read 9 Done Done New false
13328 4477 Acked 5476 Write 8 Done Done New false
13329 4478 Acked 5477 Write 6 Done Done New false
13330 4479 Acked 5478 Flush 0 Done Done New false
13331 4480 Acked 5479 Read 2 Done Done New false
13332 4481 Acked 5480 Read 1 Done Done New false
13333 4482 Acked 5481 Write 6 Done Done New false
13334 4483 Acked 5482 Read 8 Done Done New false
13335 4484 Acked 5483 Write 3 Done Done New false
13336 4485 Acked 5484 Write 5 Done Done New false
13337 4486 Acked 5485 Read 9 Done Done New false
13338 4487 Acked 5486 Read 3 Done Done New false
13339 4488 Acked 5487 Write 9 Done Done New false
13340 4489 Acked 5488 Read 7 Done Done New false
13341 4490 Acked 5489 Read 4 Done Done New false
13342 4491 Acked 5490 Write 6 Done Done New false
13343 4492 Acked 5491 Flush 0 Done Done New false
13344 4493 Acked 5492 Write 4 Done Done New false
13345 4494 Acked 5493 Read 2 Done Done New false
13346 4495 Acked 5494 Read 8 Done Done New false
13347 4496 Acked 5495 Read 2 Done Done New false
13348 4497 Acked 5496 Read 8 Done Done New false
13349 4498 Acked 5497 Write 5 Done Done New false
13350 4499 Acked 5498 Flush 0 Done Done New false
13351 4500 Acked 5499 Read 4 Done Done New false
13352 4501 Acked 5500 Write 6 Done Done New false
13353 4502 Acked 5501 Flush 0 Done Done New false
13354 4503 Acked 5502 Write 2 Done Done New false
13355 4504 Acked 5503 Write 10 Done Done New false
13356 4505 Acked 5504 Write 3 Done Done New false
13357 4506 Acked 5505 Write 1 Done Done New false
13358 4507 Acked 5506 Write 6 Done Done New false
13359 4508 Acked 5507 Read 7 Done Done New false
13360 4509 Acked 5508 Read 7 Done Done New false
13361 4510 Acked 5509 Write 10 Done Done New false
13362 4511 Acked 5510 Write 8 Done Done New false
13363 4512 Acked 5511 Read 9 Done Done New false
13364 4513 Acked 5512 Read 3 Done Done New false
13365 4514 Acked 5513 Read 7 Done Done New false
13366 4515 Acked 5514 Read 3 Done Done New false
13367 4516 Acked 5515 Write 1 Done Done New false
13368 4517 Acked 5516 Write 8 Done Done New false
13369 4518 Acked 5517 Write 3 Done Done New false
13370 4519 Acked 5518 Read 3 Done Done New false
13371 4520 Acked 5519 Write 1 Done Done New false
13372 4521 Acked 5520 Read 7 Done Done New false
13373 4522 Acked 5521 Read 8 Done Done New false
13374 STATES DS:0 DS:1 DS:2 TOTAL
13375 New 0 0 301 301
13376 Sent 0 0 0 0
13377 Done 301 301 0 602
13378 Skipped 0 0 0 0
13379 Error 0 0 0 0
13380 Last Flush: 5501 5501 5220
13381 Downstairs last five completed: 5220 5219 5218 5217 5216
13382 Upstairs last five completed: 4522 4521 4520 4519 4518
13383 CLIENT: Up:0 ds:301 act:2
133842024-02-12T01:37:07.762ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
133852024-02-12T01:37:07.762ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
133862024-02-12T01:37:07.762ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
133872024-02-12T01:37:07.762ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
133882024-02-12T01:37:07.762ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
133892024-02-12T01:37:07.762ZINFOcrucible: [2] client re-new 302 jobs since flush 5220 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
133902024-02-12T01:37:07.762ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
133912024-02-12T01:37:08.763ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
133922024-02-12T01:37:08.763ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
133932024-02-12T01:37:08.763ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
133942024-02-12T01:37:08.763ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
133952024-02-12T01:37:08.763ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
133962024-02-12T01:37:08.763ZINFOcrucible: [2] client re-new 302 jobs since flush 5220 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
133972024-02-12T01:37:08.764ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
133982024-02-12T01:37:09.764ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
133992024-02-12T01:37:09.764ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
134002024-02-12T01:37:09.764ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
134012024-02-12T01:37:09.764ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
134022024-02-12T01:37:09.764ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
134032024-02-12T01:37:09.764ZINFOcrucible: [2] client re-new 302 jobs since flush 5220 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
134042024-02-12T01:37:09.764ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
134052024-02-12T01:37:10.765ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
134062024-02-12T01:37:10.765ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
134072024-02-12T01:37:10.765ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
134082024-02-12T01:37:10.765ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
134092024-02-12T01:37:10.765ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
134102024-02-12T01:37:10.765ZINFOcrucible: [2] client re-new 302 jobs since flush 5220 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
134112024-02-12T01:37:10.766ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
13412 ----------------------------------------------------------------
13413 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
13414 States: Active Active Offline
13415 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
13416 4222 Acked 5221 Write 10 Done Done New true
13417 4223 Acked 5222 Read 10 Done Done New true
13418 4224 Acked 5223 Write 1 Done Done New true
13419 4225 Acked 5224 Read 6 Done Done New true
13420 4226 Acked 5225 Read 4 Done Done New true
13421 4227 Acked 5226 Flush 0 Done Done New true
13422 4228 Acked 5227 Read 3 Done Done New true
13423 4229 Acked 5228 Read 1 Done Done New true
13424 4230 Acked 5229 Read 6 Done Done New true
13425 4231 Acked 5230 Write 1 Done Done New true
13426 4232 Acked 5231 Read 6 Done Done New true
13427 4233 Acked 5232 Write 10 Done Done New true
13428 4234 Acked 5233 Write 5 Done Done New true
13429 4235 Acked 5234 Write 4 Done Done New true
13430 4236 Acked 5235 Flush 0 Done Done New true
13431 4237 Acked 5236 Write 1 Done Done New true
13432 4238 Acked 5237 Write 2 Done Done New true
13433 4239 Acked 5238 Read 1 Done Done New true
13434 4240 Acked 5239 Read 1 Done Done New true
13435 4241 Acked 5240 Flush 0 Done Done New true
13436 4242 Acked 5241 Read 7 Done Done New true
13437 4243 Acked 5242 Read 7 Done Done New true
13438 4244 Acked 5243 Write 1 Done Done New true
13439 4245 Acked 5244 Read 7 Done Done New true
13440 4246 Acked 5245 Read 8 Done Done New true
13441 4247 Acked 5246 Read 5 Done Done New true
13442 4248 Acked 5247 Read 6 Done Done New true
13443 4249 Acked 5248 Read 1 Done Done New true
13444 4250 Acked 5249 Write 8 Done Done New true
13445 4251 Acked 5250 Read 2 Done Done New true
13446 4252 Acked 5251 Read 9 Done Done New true
13447 4253 Acked 5252 Read 4 Done Done New true
13448 4254 Acked 5253 Flush 0 Done Done New true
13449 4255 Acked 5254 Read 6 Done Done New true
13450 4256 Acked 5255 Write 7 Done Done New true
13451 4257 Acked 5256 Read 10 Done Done New true
13452 4258 Acked 5257 Read 7 Done Done New true
13453 4259 Acked 5258 Write 10 Done Done New true
13454 4260 Acked 5259 Write 2 Done Done New true
13455 4261 Acked 5260 Write 9 Done Done New true
13456 4262 Acked 5261 Write 5 Done Done New true
13457 4263 Acked 5262 Write 2 Done Done New true
13458 4264 Acked 5263 Write 5 Done Done New true
13459 4265 Acked 5264 Read 9 Done Done New true
13460 4266 Acked 5265 Write 7 Done Done New true
13461 4267 Acked 5266 Read 5 Done Done New true
13462 4268 Acked 5267 Write 6 Done Done New true
13463 4269 Acked 5268 Write 5 Done Done New true
13464 4270 Acked 5269 Flush 0 Done Done New true
13465 4271 Acked 5270 Write 6 Done Done New true
13466 4272 Acked 5271 Write 2 Done Done New true
13467 4273 Acked 5272 Read 8 Done Done New true
13468 4274 Acked 5273 Read 7 Done Done New true
13469 4275 Acked 5274 Write 2 Done Done New true
13470 4276 Acked 5275 Write 3 Done Done New true
13471 4277 Acked 5276 Write 6 Done Done New true
13472 4278 Acked 5277 Write 2 Done Done New true
13473 4279 Acked 5278 Write 5 Done Done New true
13474 4280 Acked 5279 Write 2 Done Done New true
13475 4281 Acked 5280 Write 10 Done Done New true
13476 4282 Acked 5281 Read 4 Done Done New true
13477 4283 Acked 5282 Read 7 Done Done New true
13478 4284 Acked 5283 Write 7 Done Done New true
13479 4285 Acked 5284 Read 3 Done Done New true
13480 4286 Acked 5285 Write 5 Done Done New true
13481 4287 Acked 5286 Read 9 Done Done New true
13482 4288 Acked 5287 Read 9 Done Done New true
13483 4289 Acked 5288 Read 9 Done Done New true
13484 4290 Acked 5289 Read 5 Done Done New true
13485 4291 Acked 5290 Read 3 Done Done New true
13486 4292 Acked 5291 Read 7 Done Done New true
13487 4293 Acked 5292 Read 2 Done Done New true
13488 4294 Acked 5293 Write 2 Done Done New true
13489 4295 Acked 5294 Read 1 Done Done New true
13490 4296 Acked 5295 Write 1 Done Done New true
13491 4297 Acked 5296 Flush 0 Done Done New true
13492 4298 Acked 5297 Read 3 Done Done New true
13493 4299 Acked 5298 Write 4 Done Done New true
13494 4300 Acked 5299 Read 3 Done Done New true
13495 4301 Acked 5300 Read 2 Done Done New true
13496 4302 Acked 5301 Read 8 Done Done New true
13497 4303 Acked 5302 Write 6 Done Done New true
13498 4304 Acked 5303 Write 6 Done Done New true
13499 4305 Acked 5304 Read 8 Done Done New true
13500 4306 Acked 5305 Read 2 Done Done New true
13501 4307 Acked 5306 Read 1 Done Done New true
13502 4308 Acked 5307 Write 1 Done Done New true
13503 4309 Acked 5308 Write 8 Done Done New true
13504 4310 Acked 5309 Write 1 Done Done New true
13505 4311 Acked 5310 Read 8 Done Done New true
13506 4312 Acked 5311 Write 1 Done Done New true
13507 4313 Acked 5312 Write 5 Done Done New true
13508 4314 Acked 5313 Read 7 Done Done New true
13509 4315 Acked 5314 Write 6 Done Done New true
13510 4316 Acked 5315 Read 9 Done Done New true
13511 4317 Acked 5316 Write 6 Done Done New true
13512 4318 Acked 5317 Read 6 Done Done New true
13513 4319 Acked 5318 Write 4 Done Done New true
13514 4320 Acked 5319 Write 7 Done Done New true
13515 4321 Acked 5320 Flush 0 Done Done New true
13516 4322 Acked 5321 Write 7 Done Done New true
13517 4323 Acked 5322 Write 2 Done Done New true
13518 4324 Acked 5323 Write 3 Done Done New true
13519 4325 Acked 5324 Read 7 Done Done New true
13520 4326 Acked 5325 Write 7 Done Done New true
13521 4327 Acked 5326 Write 4 Done Done New true
13522 4328 Acked 5327 Read 10 Done Done New true
13523 4329 Acked 5328 Flush 0 Done Done New true
13524 4330 Acked 5329 Read 2 Done Done New true
13525 4331 Acked 5330 Write 2 Done Done New true
13526 4332 Acked 5331 Read 9 Done Done New true
13527 4333 Acked 5332 Flush 0 Done Done New true
13528 4334 Acked 5333 Flush 0 Done Done New true
13529 4335 Acked 5334 Read 9 Done Done New true
13530 4336 Acked 5335 Read 2 Done Done New true
13531 4337 Acked 5336 Read 4 Done Done New true
13532 4338 Acked 5337 Write 5 Done Done New true
13533 4339 Acked 5338 Write 8 Done Done New true
13534 4340 Acked 5339 Write 2 Done Done New true
13535 4341 Acked 5340 Write 4 Done Done New true
13536 4342 Acked 5341 Write 7 Done Done New true
13537 4343 Acked 5342 Read 4 Done Done New true
13538 4344 Acked 5343 Read 5 Done Done New true
13539 4345 Acked 5344 Read 3 Done Done New true
13540 4346 Acked 5345 Read 6 Done Done New true
13541 4347 Acked 5346 Write 10 Done Done New true
13542 4348 Acked 5347 Read 4 Done Done New true
13543 4349 Acked 5348 Write 1 Done Done New true
13544 4350 Acked 5349 Read 5 Done Done New true
13545 4351 Acked 5350 Write 7 Done Done New true
13546 4352 Acked 5351 Write 7 Done Done New true
13547 4353 Acked 5352 Write 6 Done Done New true
13548 4354 Acked 5353 Write 1 Done Done New true
13549 4355 Acked 5354 Read 6 Done Done New true
13550 4356 Acked 5355 Write 9 Done Done New true
13551 4357 Acked 5356 Write 10 Done Done New true
13552 4358 Acked 5357 Write 6 Done Done New true
13553 4359 Acked 5358 Write 9 Done Done New true
13554 4360 Acked 5359 Write 9 Done Done New true
13555 4361 Acked 5360 Write 7 Done Done New true
13556 4362 Acked 5361 Flush 0 Done Done New true
13557 4363 Acked 5362 Read 10 Done Done New true
13558 4364 Acked 5363 Read 9 Done Done New true
13559 4365 Acked 5364 Write 3 Done Done New true
13560 4366 Acked 5365 Read 7 Done Done New true
13561 4367 Acked 5366 Write 1 Done Done New true
13562 4368 Acked 5367 Read 9 Done Done New true
13563 4369 Acked 5368 Write 8 Done Done New true
13564 4370 Acked 5369 Write 8 Done Done New true
13565 4371 Acked 5370 Write 7 Done Done New true
13566 4372 Acked 5371 Read 8 Done Done New true
13567 4373 Acked 5372 Write 8 Done Done New true
13568 4374 Acked 5373 Write 7 Done Done New true
13569 4375 Acked 5374 Write 7 Done Done New true
13570 4376 Acked 5375 Read 7 Done Done New true
13571 4377 Acked 5376 Write 6 Done Done New true
13572 4378 Acked 5377 Write 1 Done Done New true
13573 4379 Acked 5378 Read 4 Done Done New true
13574 4380 Acked 5379 Read 3 Done Done New true
13575 4381 Acked 5380 Read 1 Done Done New true
13576 4382 Acked 5381 Write 7 Done Done New true
13577 4383 Acked 5382 Write 6 Done Done New true
13578 4384 Acked 5383 Flush 0 Done Done New true
13579 4385 Acked 5384 Read 2 Done Done New true
13580 4386 Acked 5385 Read 1 Done Done New true
13581 4387 Acked 5386 Write 1 Done Done New true
13582 4388 Acked 5387 Read 6 Done Done New true
13583 4389 Acked 5388 Read 8 Done Done New true
13584 4390 Acked 5389 Read 10 Done Done New true
13585 4391 Acked 5390 Read 1 Done Done New true
13586 4392 Acked 5391 Write 5 Done Done New true
13587 4393 Acked 5392 Write 3 Done Done New true
13588 4394 Acked 5393 Read 5 Done Done New true
13589 4395 Acked 5394 Write 6 Done Done New true
13590 4396 Acked 5395 Read 7 Done Done New true
13591 4397 Acked 5396 Read 8 Done Done New true
13592 4398 Acked 5397 Read 8 Done Done New true
13593 4399 Acked 5398 Write 9 Done Done New true
13594 4400 Acked 5399 Read 8 Done Done New true
13595 4401 Acked 5400 Read 3 Done Done New true
13596 4402 Acked 5401 Read 1 Done Done New true
13597 4403 Acked 5402 Read 4 Done Done New true
13598 4404 Acked 5403 Read 3 Done Done New true
13599 4405 Acked 5404 Write 1 Done Done New true
13600 4406 Acked 5405 Write 6 Done Done New true
13601 4407 Acked 5406 Flush 0 Done Done New true
13602 4408 Acked 5407 Write 9 Done Done New true
13603 4409 Acked 5408 Read 7 Done Done New true
13604 4410 Acked 5409 Write 8 Done Done New true
13605 4411 Acked 5410 Read 9 Done Done New true
13606 4412 Acked 5411 Read 10 Done Done New true
13607 4413 Acked 5412 Flush 0 Done Done New true
13608 4414 Acked 5413 Read 7 Done Done New true
13609 4415 Acked 5414 Read 9 Done Done New true
13610 4416 Acked 5415 Write 6 Done Done New true
13611 4417 Acked 5416 Read 3 Done Done New true
13612 4418 Acked 5417 Read 4 Done Done New true
13613 4419 Acked 5418 Read 2 Done Done New true
13614 4420 Acked 5419 Read 6 Done Done New true
13615 4421 Acked 5420 Read 1 Done Done New true
13616 4422 Acked 5421 Read 10 Done Done New true
13617 4423 Acked 5422 Read 10 Done Done New true
13618 4424 Acked 5423 Flush 0 Done Done New true
13619 4425 Acked 5424 Write 2 Done Done New true
13620 4426 Acked 5425 Read 8 Done Done New true
13621 4427 Acked 5426 Read 5 Done Done New true
13622 4428 Acked 5427 Read 9 Done Done New true
13623 4429 Acked 5428 Write 8 Done Done New true
13624 4430 Acked 5429 Write 7 Done Done New true
13625 4431 Acked 5430 Read 7 Done Done New true
13626 4432 Acked 5431 Read 5 Done Done New true
13627 4433 Acked 5432 Write 2 Done Done New true
13628 4434 Acked 5433 Read 2 Done Done New true
13629 4435 Acked 5434 Write 10 Done Done New true
13630 4436 Acked 5435 Read 8 Done Done New true
13631 4437 Acked 5436 Read 6 Done Done New true
13632 4438 Acked 5437 Write 6 Done Done New true
13633 4439 Acked 5438 Read 8 Done Done New true
13634 4440 Acked 5439 Write 10 Done Done New true
13635 4441 Acked 5440 Flush 0 Done Done New true
13636 4442 Acked 5441 Read 8 Done Done New true
13637 4443 Acked 5442 Read 5 Done Done New true
13638 4444 Acked 5443 Read 9 Done Done New true
13639 4445 Acked 5444 Flush 0 Done Done New true
13640 4446 Acked 5445 Write 1 Done Done New true
13641 4447 Acked 5446 Read 7 Done Done New true
13642 4448 Acked 5447 Write 6 Done Done New true
13643 4449 Acked 5448 Read 2 Done Done New true
13644 4450 Acked 5449 Read 9 Done Done New true
13645 4451 Acked 5450 Read 2 Done Done New true
13646 4452 Acked 5451 Write 2 Done Done New true
13647 4453 Acked 5452 Write 7 Done Done New true
13648 4454 Acked 5453 Read 9 Done Done New true
13649 4455 Acked 5454 Read 6 Done Done New true
13650 4456 Acked 5455 Read 6 Done Done New true
13651 4457 Acked 5456 Write 2 Done Done New true
13652 4458 Acked 5457 Write 6 Done Done New true
13653 4459 Acked 5458 Read 5 Done Done New true
13654 4460 Acked 5459 Write 5 Done Done New true
13655 4461 Acked 5460 Write 8 Done Done New true
13656 4462 Acked 5461 Read 10 Done Done New true
13657 4463 Acked 5462 Write 6 Done Done New true
13658 4464 Acked 5463 Write 9 Done Done New true
13659 4465 Acked 5464 Read 2 Done Done New true
13660 4466 Acked 5465 Write 9 Done Done New true
13661 4467 Acked 5466 Read 10 Done Done New true
13662 4468 Acked 5467 Write 9 Done Done New true
13663 4469 Acked 5468 Read 5 Done Done New true
13664 4470 Acked 5469 Write 2 Done Done New true
13665 4471 Acked 5470 Write 6 Done Done New true
13666 4472 Acked 5471 Write 4 Done Done New true
13667 4473 Acked 5472 Read 4 Done Done New true
13668 4474 Acked 5473 Read 8 Done Done New true
13669 4475 Acked 5474 Flush 0 Done Done New true
13670 4476 Acked 5475 Read 9 Done Done New true
13671 4477 Acked 5476 Write 8 Done Done New true
13672 4478 Acked 5477 Write 6 Done Done New true
13673 4479 Acked 5478 Flush 0 Done Done New true
13674 4480 Acked 5479 Read 2 Done Done New true
13675 4481 Acked 5480 Read 1 Done Done New true
13676 4482 Acked 5481 Write 6 Done Done New true
13677 4483 Acked 5482 Read 8 Done Done New true
13678 4484 Acked 5483 Write 3 Done Done New true
13679 4485 Acked 5484 Write 5 Done Done New true
13680 4486 Acked 5485 Read 9 Done Done New true
13681 4487 Acked 5486 Read 3 Done Done New true
13682 4488 Acked 5487 Write 9 Done Done New true
13683 4489 Acked 5488 Read 7 Done Done New true
13684 4490 Acked 5489 Read 4 Done Done New true
13685 4491 Acked 5490 Write 6 Done Done New true
13686 4492 Acked 5491 Flush 0 Done Done New true
13687 4493 Acked 5492 Write 4 Done Done New true
13688 4494 Acked 5493 Read 2 Done Done New true
13689 4495 Acked 5494 Read 8 Done Done New true
13690 4496 Acked 5495 Read 2 Done Done New true
13691 4497 Acked 5496 Read 8 Done Done New true
13692 4498 Acked 5497 Write 5 Done Done New true
13693 4499 Acked 5498 Flush 0 Done Done New true
13694 4500 Acked 5499 Read 4 Done Done New true
13695 4501 Acked 5500 Write 6 Done Done New true
13696 4502 Acked 5501 Flush 0 Done Done New true
13697 4503 Acked 5502 Write 2 Done Done New true
13698 4504 Acked 5503 Write 10 Done Done New true
13699 4505 Acked 5504 Write 3 Done Done New true
13700 4506 Acked 5505 Write 1 Done Done New true
13701 4507 Acked 5506 Write 6 Done Done New true
13702 4508 Acked 5507 Read 7 Done Done New true
13703 4509 Acked 5508 Read 7 Done Done New true
13704 4510 Acked 5509 Write 10 Done Done New true
13705 4511 Acked 5510 Write 8 Done Done New true
13706 4512 Acked 5511 Read 9 Done Done New true
13707 4513 Acked 5512 Read 3 Done Done New true
13708 4514 Acked 5513 Read 7 Done Done New true
13709 4515 Acked 5514 Read 3 Done Done New true
13710 4516 Acked 5515 Write 1 Done Done New true
13711 4517 Acked 5516 Write 8 Done Done New true
13712 4518 Acked 5517 Write 3 Done Done New true
13713 4519 Acked 5518 Read 3 Done Done New true
13714 4520 Acked 5519 Write 1 Done Done New true
13715 4521 Acked 5520 Read 7 Done Done New true
13716 4522 Acked 5521 Read 8 Done Done New true
13717 4523 Acked 5522 Flush 0 Done Done New true
13718 STATES DS:0 DS:1 DS:2 TOTAL
13719 New 0 0 302 302
13720 Sent 0 0 0 0
13721 Done 302 302 0 604
13722 Skipped 0 0 0 0
13723 Error 0 0 0 0
13724 Last Flush: 5522 5522 5220
13725 Downstairs last five completed: 5220 5219 5218 5217 5216
13726 Upstairs last five completed: 4523 4522 4521 4520 4519
13727 CLIENT: Up:0 ds:302 act:2
137282024-02-12T01:37:11.766ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
137292024-02-12T01:37:11.766ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
137302024-02-12T01:37:11.766ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
137312024-02-12T01:37:11.766ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
137322024-02-12T01:37:11.766ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
137332024-02-12T01:37:11.767ZINFOcrucible: [2] client re-new 302 jobs since flush 5220 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
137342024-02-12T01:37:11.767ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
13735 [8830] got start action
13736 [8830] I am going to restart
13737 Starting downstairs at port 8830
13738 Make output file at "/tmp/dsc/downstairs-8830.txt"
13739 [8830][2] reports Starting
137402024-02-12T01:37:12.768ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
137412024-02-12T01:37:12.768ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
137422024-02-12T01:37:12.768ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
137432024-02-12T01:37:12.768ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
137442024-02-12T01:37:12.768ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
137452024-02-12T01:37:12.768ZINFOcrucible: [2] client re-new 302 jobs since flush 5220 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
137462024-02-12T01:37:12.768ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
13747 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1179)
13748 [8830][2] reports Running
137492024-02-12T01:37:13.769ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
137502024-02-12T01:37:13.769ZINFOcrucible: ds_connection connected = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
137512024-02-12T01:37:13.771ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
137522024-02-12T01:37:13.771ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
137532024-02-12T01:37:13.771ZINFOcrucible: send last flush ID to this DS: 5220 = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
137542024-02-12T01:37:13.771ZINFOcrucible: Replied this last flush ID: 5220 = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
137552024-02-12T01:37:13.771ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
137562024-02-12T01:37:13.771ZINFOcrucible: [2] Transition from Offline to Replay = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
137572024-02-12T01:37:13.771ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
137582024-02-12T01:37:13.771ZINFOcrucible: [2] Transition from Replay to Active = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
13759 ----------------------------------------------------------------
13760 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
13761 Downstairs last five completed: 5522 5521 5520 5519 5518
13762 Upstairs last five completed: 4523 4522 4521 4520 4519
13763 CLIENT: Up:0 ds:0 act:3
13764 Replay: All jobs finished, all DS active.
13765 Main task has work to do, go find it
13766 got dsc Stop(1)
13767 stop 1
13768 Feb 12 01:37:15.109 INFO request completed, latency_us: 400, response_code: 204, uri: [8820] Got stop action so:false kr:false
13769 /stop/cid/1, method: POST, req_id: 6e0a0c44-2abd-4cf7-89f8-c8492ca23c0d, remote_addr: 127.0.0.1:60456, local_addr: 127.0.0.1:9998
137702024-02-12T01:37:15.110ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
137712024-02-12T01:37:15.110ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
13772 Feb 12 01:37:15.110 INFO request completed{"msg":"downstairs task for 1 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-02-12T01:37:15.110825022Z","hostname":"ip-10-150-1-104.us-west-2.compute.internal","pid":1164,"session_id":"e1773b3e-b36f-4080-8b13-c0830f0d072e"}
13773 , latency_us: 163, response_code: 200, uri: /state/cid/1{"msg":"Gone missing, transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-02-12T01:37:15.110861597Z","hostname":"ip-10-150-1-104.us-west-2.compute.internal","pid":1164,"client":"1","":"downstairs","session_id":"e1773b3e-b36f-4080-8b13-c0830f0d072e"}
13774 , method: GET{"msg":"[1] client re-new 0 jobs since flush 5522","v":0,"name":"crucible","level":30,"time":"2024-02-12T01:37:15.110916399Z","hostname":"ip-10-150-1-104.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"e1773b3e-b36f-4080-8b13-c0830f0d072e"}
13775 , req_id: 61295c04-39c6-49e5-8487-36fff2644acc, remote_addr{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-12T01:37:15.110948274Z","hostname":"ip-10-150-1-104.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"e1773b3e-b36f-4080-8b13-c0830f0d072e"}
13776 : 127.0.0.1:60456, local_addr: 127.0.0.1:9998
13777 [8820] Exited with: ExitStatus(unix_wait_status(9))
13778 [8820][1] reports Exit
137792024-02-12T01:37:16.111ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
137802024-02-12T01:37:16.111ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
137812024-02-12T01:37:16.111ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
137822024-02-12T01:37:16.111ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
137832024-02-12T01:37:16.111ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
137842024-02-12T01:37:16.111ZINFOcrucible: [1] client re-new 0 jobs since flush 5522 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
137852024-02-12T01:37:16.111ZINFOcrucible: client task is exiting = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
137862024-02-12T01:37:17.113ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
137872024-02-12T01:37:17.113ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
137882024-02-12T01:37:17.113ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
137892024-02-12T01:37:17.113ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
137902024-02-12T01:37:17.113ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
137912024-02-12T01:37:17.113ZINFOcrucible: [1] client re-new 0 jobs since flush 5522 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
137922024-02-12T01:37:17.113ZINFOcrucible: client task is exiting = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
137932024-02-12T01:37:18.114ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
137942024-02-12T01:37:18.114ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
137952024-02-12T01:37:18.114ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
137962024-02-12T01:37:18.114ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
137972024-02-12T01:37:18.114ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
137982024-02-12T01:37:18.114ZINFOcrucible: [1] client re-new 0 jobs since flush 5522 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
137992024-02-12T01:37:18.114ZINFOcrucible: client task is exiting = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
13800 Feb 12 01:37:19.113 INFO request completed, latency_us: 213, response_code: 200, uri: /state/cid/1, method: GET, req_id: 43908b2a-c701-4499-ba2a-c9c0d12057a4, remote_addr: 127.0.0.1:60456, local_addr: 127.0.0.1:9998
13801 001/300 Write block 4178 len 16384 data: 3 4 5 4
13802 002/300 Read block 2879 len 20480
138032024-02-12T01:37:19.115ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
138042024-02-12T01:37:19.115ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
138052024-02-12T01:37:19.115ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
138062024-02-12T01:37:19.115ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
138072024-02-12T01:37:19.115ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
138082024-02-12T01:37:19.115ZINFOcrucible: [1] client re-new 2 jobs since flush 5522 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
138092024-02-12T01:37:19.115ZINFOcrucible: client task is exiting = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
13810 003/300 Read block 3497 len 36864
13811 004/300 Write block 226 len 32768 data: 4 4 4 4 4 4 5 3
13812 005/300 Flush
13813 006/300 Read block 237 len 8192
13814 007/300 Read block 1458 len 16384
13815 008/300 Read block 1341 len 36864
13816 009/300 Write block 376 len 28672 data: 4 5 5 5 5 3 3
13817 010/300 Read block 3602 len 36864
13818 011/300 Flush
13819 012/300 Write block 1564 len 16384 data: 6 7 8 7
13820 013/300 Write block 2057 len 28672 data: 2 2 3 4 4 4 6
13821 014/300 Write block 2555 len 36864 data: 4 3 4 5 6 6 6 6 5
13822 015/300 Read block 407 len 40960
13823 016/300 Write block 4172 len 40960 data: 5 6 7 3 3 3 4 5 6 5
13824 017/300 Write block 3116 len 36864 data: 5 5 4 4 5 5 5 5 5
13825 018/300 Read block 3179 len 40960
13826 019/300 Write block 2286 len 4096 data: 2
13827 020/300 Flush
13828 021/300 Read block 1851 len 40960
13829 022/300 Read block 2319 len 4096
13830 023/300 Flush
13831 024/300 Read block 860 len 16384
13832 025/300 Read block 3414 len 28672
13833 026/300 Write block 2245 len 32768 data: 4 4 3 3 3 5 4 6
13834 027/300 Write block 1339 len 20480 data: 3 5 3 3 3
13835 028/300 Read block 1084 len 40960
13836 029/300 Flush
13837 030/300 Write block 3144 len 24576 data: 3 3 3 3 3 4
13838 031/300 Write block 4479 len 24576 data: 5 4 4 6 7 5
13839 032/300 Read block 4311 len 4096
13840 033/300 Read block 2766 len 4096
13841 034/300 Read block 1626 len 40960
13842 035/300 Read block 4862 len 40960
13843 036/300 Read block 3333 len 8192
13844 037/300 Write block 2818 len 20480 data: 4 4 3 5 5
13845 038/300 Read block 1995 len 36864
13846 039/300 Write block 4824 len 40960 data: 3 4 3 3 3 3 2 2 3 3
13847 040/300 Read block 1600 len 40960
13848 041/300 Read block 4295 len 8192
13849 042/300 Flush
13850 043/300 Read block 2500 len 20480
13851 044/300 Write block 159 len 16384 data: 3 3 2 4
13852 045/300 Read block 679 len 24576
13853 046/300 Read block 3478 len 32768
13854 047/300 Write block 3678 len 20480 data: 3 3 3 5 5
13855 048/300 Read block 1439 len 36864
13856 049/300 Read block 4238 len 28672
13857 050/300 Read block 2853 len 16384
13858 051/300 Write block 1577 len 8192 data: 2 3
13859 052/300 Write block 981 len 36864 data: 4 4 4 4 5 5 4 4 4
13860 053/300 Read block 3059 len 16384
13861 054/300 Read block 4506 len 12288
13862 055/300 Write block 3947 len 4096 data: 5
13863 056/300 Write block 1398 len 32768 data: 4 3 3 3 3 3 5 5
13864 057/300 Flush
13865 058/300 Read block 2537 len 20480
13866 059/300 Write block 4045 len 16384 data: 4 3 2 2
13867 060/300 Read block 2550 len 20480
13868 061/300 Write block 3078 len 12288 data: 3 4 5
13869 062/300 Write block 4748 len 24576 data: 2 2 2 2 2 2
13870 063/300 Read block 4679 len 16384
13871 064/300 Read block 2456 len 24576
13872 065/300 Read block 3548 len 12288
13873 066/300 Write block 2243 len 4096 data: 4
13874 067/300 Read block 4461 len 20480
13875 068/300 Read block 2980 len 12288
13876 069/300 Read block 569 len 24576
13877 070/300 Write block 781 len 40960 data: 5 7 6 5 6 6 5 6 5 6
13878 071/300 Read block 99 len 28672
13879 072/300 Flush
13880 073/300 Read block 4086 len 36864
13881 074/300 Read block 3655 len 24576
13882 075/300 Write block 1190 len 32768 data: 2 2 3 4 4 3 3 2
13883 076/300 Read block 4151 len 16384
13884 077/300 Write block 3513 len 28672 data: 4 4 4 3 2 3 4
13885 078/300 Write block 1549 len 16384 data: 3 3 4 5
13886 079/300 Read block 3674 len 28672
13887 080/300 Read block 2809 len 20480
13888 081/300 Write block 1315 len 12288 data: 3 4 4
13889 082/300 Read block 3462 len 4096
13890 083/300 Read block 3123 len 28672
13891 084/300 Write block 1033 len 16384 data: 3 3 3 3
13892 085/300 Write block 1209 len 40960 data: 6 6 5 6 5 4 4 5 5 4
13893 086/300 Write block 4482 len 16384 data: 7 8 6 5
13894 087/300 Flush
13895 088/300 Flush
13896 089/300 Read block 2776 len 4096
13897 090/300 Read block 651 len 24576
13898 091/300 Read block 2122 len 12288
13899 092/300 Write block 2502 len 4096 data: 2
13900 093/300 Read block 2636 len 4096
13901 094/300 Write block 1306 len 8192 data: 4 4
13902 095/300 Write block 3281 len 24576 data: 4 4 4 3 4 4
13903 096/300 Write block 4379 len 16384 data: 3 4 4 3
13904 097/300 Read block 4425 len 20480
13905 098/300 Flush
13906 099/300 Write block 3935 len 32768 data: 3 3 3 3 3 3 4 4
13907 100/300 Read block 2783 len 16384
13908 101/300 Read block 907 len 24576
13909 102/300 Flush
13910 103/300 Flush
13911 104/300 Write block 684 len 24576 data: 3 3 3 2 2 2
13912 105/300 Read block 4841 len 12288
13913 106/300 Flush
13914 107/300 Read block 2691 len 28672
13915 108/300 Read block 4264 len 4096
13916 109/300 Read block 3184 len 36864
13917 110/300 Write block 329 len 40960 data: 2 2 2 2 2 3 5 5 3 2
13918 111/300 Read block 4340 len 36864
13919 112/300 Read block 4729 len 12288
13920 113/300 Read block 407 len 24576
13921 114/300 Write block 3738 len 8192 data: 4 5
13922 115/300 Read block 890 len 12288
13923 116/300 Write block 3231 len 40960 data: 6 6 7 6 6 6 6 6 5 4
13924 117/300 Read block 3528 len 12288
13925 118/300 Read block 3119 len 32768
13926 119/300 Flush
13927 120/300 Read block 3384 len 16384
13928 121/300 Read block 677 len 32768
13929 122/300 Read block 1332 len 36864
13930 123/300 Read block 1480 len 8192
13931 124/300 Read block 4510 len 12288
13932 125/300 Read block 3005 len 8192
13933 126/300 Read block 3509 len 32768
13934 127/300 Read block 3703 len 32768
13935 128/300 Read block 1940 len 20480
13936 129/300 Write block 3017 len 12288 data: 2 2 3
13937 130/300 Read block 872 len 12288
13938 131/300 Write block 2104 len 32768 data: 3 3 4 4 3 3 3 3
13939 132/300 Write block 3445 len 32768 data: 5 5 5 5 6 9 9 9
13940 133/300 Read block 1854 len 16384
13941 134/300 Write block 2279 len 36864 data: 4 5 5 5 4 3 2 3 3
13942 135/300 Write block 1463 len 20480 data: 5 6 5 5 4
13943 136/300 Flush
13944 137/300 Read block 850 len 24576
13945 138/300 Write block 3943 len 12288 data: 4 4 5
13946 139/300 Write block 799 len 36864 data: 8 7 7 8 8 8 7 7 7
13947 140/300 Read block 1515 len 12288
13948 141/300 Write block 3627 len 24576 data: 4 4 5 4 4 4
13949 142/300 Write block 4786 len 32768 data: 5 5 4 4 3 3 3 2
13950 143/300 Write block 3494 len 8192 data: 3 4
13951 144/300 Read block 1390 len 4096
13952 145/300 Write block 2666 len 4096 data: 3
13953 146/300 Read block 1922 len 20480
13954 147/300 Write block 2829 len 40960 data: 8 7 6 5 4 5 6 5 7 6
13955 148/300 Flush
13956 149/300 Flush
13957 150/300 Read block 835 len 16384
13958 151/300 Write block 3183 len 16384 data: 6 8 7 8
13959 152/300 Write block 857 len 40960 data: 3 2 2 2 2 3 3 3 2 2
13960 153/300 Read block 445 len 16384
13961 154/300 Write block 1118 len 16384 data: 3 3 3 3
13962 155/300 Write block 1078 len 4096 data: 4
13963 156/300 Read block 309 len 16384
13964 157/300 Write block 4266 len 20480 data: 7 7 8 7 6
13965 158/300 Read block 2552 len 32768
13966 159/300 Write block 1097 len 24576 data: 3 3 3 2 2 3
13967 160/300 Write block 4924 len 4096 data: 6
13968 161/300 Write block 4481 len 8192 data: 5 8
13969 162/300 Write block 4319 len 4096 data: 6
13970 163/300 Write block 3477 len 16384 data: 3 3 5 4
13971 164/300 Read block 4841 len 8192
13972 165/300 Read block 881 len 12288
13973 166/300 Write block 3716 len 4096 data: 5
13974 167/300 Read block 4967 len 32768
13975 168/300 Read block 784 len 28672
13976 169/300 Write block 749 len 8192 data: 6 5
13977 170/300 Write block 335 len 24576 data: 6 6 4 3 2 2
13978 171/300 Write block 1399 len 8192 data: 4 4
13979 172/300 Write block 4449 len 12288 data: 4 6 5
13980 173/300 Read block 3557 len 4096
13981 174/300 Write block 662 len 24576 data: 4 4 4 5 6 5
13982 175/300 Read block 1139 len 20480
13983 176/300 Flush
13984 177/300 Write block 3278 len 36864 data: 3 4 3 5 5 5 4 5 5
13985 178/300 Read block 4393 len 20480
13986 179/300 Flush
13987 180/300 Read block 2350 len 4096
13988 181/300 Read block 2631 len 36864
13989 182/300 Read block 905 len 8192
13990 183/300 Read block 4589 len 24576
13991 184/300 Flush
13992 185/300 Write block 645 len 20480 data: 4 5 4 3 3
13993 186/300 Write block 4265 len 12288 data: 8 8 8
13994 187/300 Read block 3311 len 20480
13995 188/300 Write block 4603 len 20480 data: 5 4 4 3 4
13996 189/300 Write block 223 len 16384 data: 5 5 5 5
13997 190/300 Write block 1971 len 4096 data: 4
13998 191/300 Write block 4809 len 32768 data: 2 2 2 2 2 3 3 4
13999 192/300 Read block 3153 len 12288
14000 193/300 Read block 4155 len 24576
14001 194/300 Read block 3417 len 40960
14002 195/300 Read block 4415 len 12288
14003 196/300 Read block 4076 len 8192
14004 197/300 Write block 3775 len 40960 data: 5 6 5 3 4 4 3 3 3 3
14005 198/300 Read block 79 len 32768
14006 199/300 Flush
14007 200/300 Flush
14008 201/300 Write block 3602 len 4096 data: 4
14009 202/300 Write block 2274 len 28672 data: 5 5 5 4 4 5 6
14010 203/300 Read block 3587 len 12288
14011 204/300 Read block 4776 len 32768
14012 205/300 Write block 2929 len 20480 data: 5 6 4 3 3
14013 206/300 Read block 1508 len 16384
14014 207/300 Write block 4033 len 12288 data: 3 4 3
14015 208/300 Flush
14016 209/300 Write block 4286 len 28672 data: 4 5 4 4 5 5 5
14017 210/300 Read block 4371 len 40960
14018 211/300 Flush
14019 212/300 Write block 2490 len 36864 data: 6 5 3 3 2 2 2 2 2
14020 213/300 Read block 1093 len 20480
14021 214/300 Flush
14022 215/300 Write block 1383 len 8192 data: 4 4
14023 216/300 Read block 1421 len 16384
14024 217/300 Read block 2646 len 24576
14025 218/300 Read block 4550 len 40960
14026 219/300 Write block 2267 len 28672 data: 3 3 4 3 3 4 5
14027 220/300 Write block 1529 len 24576 data: 2 3 4 4 5 7
14028 221/300 Read block 4960 len 32768
14029 222/300 Write block 1918 len 12288 data: 5 5 5
14030 223/300 Read block 3626 len 4096
14031 224/300 Write block 2060 len 28672 data: 5 5 5 7 6 6 6
14032 225/300 Write block 4399 len 12288 data: 3 2 2
14033 226/300 Read block 237 len 12288
14034 227/300 Read block 843 len 40960
14035 228/300 Write block 4456 len 24576 data: 7 7 4 4 4 3
14036 229/300 Write block 1481 len 32768 data: 4 4 4 2 2 2 2 2
14037 230/300 Write block 4729 len 20480 data: 2 2 3 4 5
14038 231/300 Read block 1961 len 8192
14039 232/300 Write block 590 len 24576 data: 2 2 2 2 2 3
14040 233/300 Write block 4327 len 8192 data: 6 5
14041 234/300 Write block 1034 len 20480 data: 4 4 4 3 2
14042 235/300 Write block 557 len 12288 data: 2 3 2
14043 236/300 Read block 4288 len 16384
14044 237/300 Read block 969 len 12288
14045 238/300 Write block 967 len 20480 data: 3 4 5 5 6
14046 239/300 Write block 2765 len 24576 data: 3 3 3 3 4 3
14047 240/300 Read block 2519 len 40960
14048 241/300 Read block 2733 len 20480
14049 242/300 Flush
14050 243/300 Read block 3213 len 8192
14051 244/300 Read block 4201 len 32768
14052 245/300 Flush
14053 246/300 Flush
14054 247/300 Read block 3357 len 16384
14055 248/300 Read block 3078 len 24576
14056 249/300 Read block 4862 len 20480
14057 250/300 Flush
14058 251/300 Flush
14059 252/300 Write block 159 len 8192 data: 4 4
14060 253/300 Write block 2607 len 24576 data: 4 5 5 5 4 5
14061 254/300 Write block 467 len 12288 data: 4 4 4
14062 255/300 Write block 4819 len 8192 data: 3 3
14063 256/300 Write block 4475 len 8192 data: 7 7
14064 257/300 Read block 715 len 40960
14065 258/300 Write block 2811 len 8192 data: 8 7
14066 259/300 Write block 1631 len 24576 data: 3 3 3 2 3 3
14067 260/300 Read block 4543 len 16384
14068 261/300 Read block 4291 len 28672
14069 262/300 Read block 2612 len 40960
14070 263/300 Write block 4869 len 28672 data: 4 4 4 4 3 4 4
14071 264/300 Write block 1872 len 40960 data: 3 3 3 3 4 4 4 4 3 3
14072 265/300 Write block 1024 len 16384 data: 3 3 3 3
14073 266/300 Read block 4121 len 12288
14074 267/300 Write block 802 len 20480 data: 9 9 9 8 8
14075 268/300 Read block 2066 len 20480
14076 269/300 Flush
14077 270/300 Read block 4208 len 24576
14078 271/300 Write block 4873 len 40960 data: 4 5 5 4 3 3 3 3 5 5
14079 272/300 Write block 4449 len 16384 data: 5 7 6 6
14080 273/300 Write block 2567 len 40960 data: 6 3 3 3 3 3 2 2 2 2
14081 274/300 Read block 2266 len 36864
14082 275/300 Write block 4613 len 8192 data: 5 5
14083 276/300 Read block 2415 len 20480
14084 277/300 Write block 2448 len 4096 data: 5
14085 278/300 Flush
14086 279/300 Read block 3226 len 20480
14087 280/300 Read block 250 len 4096
14088 281/300 Flush
14089 282/300 Write block 1815 len 20480 data: 3 3 4 3 7
14090 283/300 Read block 644 len 28672
14091 284/300 Write block 1269 len 20480 data: 4 3 2 3 3
14092 285/300 Write block 2770 len 16384 data: 4 3 3 3
14093 286/300 Write block 3105 len 8192 data: 6 3
14094 287/300 Write block 1352 len 16384 data: 2 2 3 3
14095 288/300 Write block 244 len 4096 data: 5
14096 289/300 Write block 2071 len 4096 data: 5
14097 290/300 Read block 519 len 4096
14098 291/300 Write block 2868 len 36864 data: 2 3 4 4 4 4 4 4 5
14099 292/300 Read block 3150 len 16384
14100 293/300 Read block 4986 len 20480
14101 294/300 Write block 2133 len 4096 data: 5
14102 295/300 Read block 2603 len 24576
14103 296/300 Flush
14104 297/300 Read block 4688 len 4096
14105 298/300 Write block 467 len 32768 data: 5 5 5 4 5 4 4 3
14106 299/300 Write block 2036 len 32768 data: 3 3 3 3 3 3 2 2
14107 300/300 Flush
14108 Main task has work to do, go find it
14109 got dsc Start(1)
14110 start 1
14111 Feb 12 01:37:19.452 INFO request completed, latency_us: 243, response_code: 204, uri: /start/cid/1, method: POST, req_id: 3283acf3-fdaa-4bb8-b69a-07b2d27fd1c6, remote_addr: 127.0.0.1:60456, local_addr: 127.0.0.1:9998
14112 Replay: started 1, returned:Ok(())
14113 ----------------------------------------------------------------
14114 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:300
14115 States: Active Offline Active
14116 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
14117 4524 Acked 5523 Write 4 Done New Done true
14118 4525 Acked 5524 Read 5 Done New Done true
14119 4526 Acked 5525 Read 9 Done New Done false
14120 4527 Acked 5526 Write 8 Done New Done false
14121 4528 Acked 5527 Flush 0 Done New Done false
14122 4529 Acked 5528 Read 2 Done New Done false
14123 4530 Acked 5529 Read 4 Done New Done false
14124 4531 Acked 5530 Read 9 Done New Done false
14125 4532 Acked 5531 Write 7 Done New Done false
14126 4533 Acked 5532 Read 9 Done New Done false
14127 4534 Acked 5533 Flush 0 Done New Done false
14128 4535 Acked 5534 Write 4 Done New Done false
14129 4536 Acked 5535 Write 7 Done New Done false
14130 4537 Acked 5536 Write 9 Done New Done false
14131 4538 Acked 5537 Read 10 Done New Done false
14132 4539 Acked 5538 Write 10 Done New Done false
14133 4540 Acked 5539 Write 9 Done New Done false
14134 4541 Acked 5540 Read 10 Done New Done false
14135 4542 Acked 5541 Write 1 Done New Done false
14136 4543 Acked 5542 Flush 0 Done New Done false
14137 4544 Acked 5543 Read 10 Done New Done false
14138 4545 Acked 5544 Read 1 Done New Done false
14139 4546 Acked 5545 Flush 0 Done New Done false
14140 4547 Acked 5546 Read 4 Done New Done false
14141 4548 Acked 5547 Read 7 Done New Done false
14142 4549 Acked 5548 Write 8 Done New Done false
14143 4550 Acked 5549 Write 5 Done New Done false
14144 4551 Acked 5550 Read 10 Done New Done false
14145 4552 Acked 5551 Flush 0 Done New Done false
14146 4553 Acked 5552 Write 6 Done New Done false
14147 4554 Acked 5553 Write 6 Done New Done false
14148 4555 Acked 5554 Read 1 Done New Done false
14149 4556 Acked 5555 Read 1 Done New Done false
14150 4557 Acked 5556 Read 10 Done New Done false
14151 4558 Acked 5557 Read 10 Done New Done false
14152 4559 Acked 5558 Read 2 Done New Done false
14153 4560 Acked 5559 Write 5 Done New Done false
14154 4561 Acked 5560 Read 9 Done New Done false
14155 4562 Acked 5561 Write 10 Done New Done false
14156 4563 Acked 5562 Read 10 Done New Done false
14157 4564 Acked 5563 Read 2 Done New Done false
14158 4565 Acked 5564 Flush 0 Done New Done false
14159 4566 Acked 5565 Read 5 Done New Done false
14160 4567 Acked 5566 Write 4 Done New Done false
14161 4568 Acked 5567 Read 6 Done New Done false
14162 4569 Acked 5568 Read 8 Done New Done false
14163 4570 Acked 5569 Write 5 Done New Done false
14164 4571 Acked 5570 Read 9 Done New Done false
14165 4572 Acked 5571 Read 7 Done New Done false
14166 4573 Acked 5572 Read 4 Done New Done false
14167 4574 Acked 5573 Write 2 Done New Done false
14168 4575 Acked 5574 Write 9 Done New Done false
14169 4576 Acked 5575 Read 4 Done New Done false
14170 4577 Acked 5576 Read 3 Done New Done false
14171 4578 Acked 5577 Write 1 Done New Done false
14172 4579 Acked 5578 Write 8 Done New Done false
14173 4580 Acked 5579 Flush 0 Done New Done false
14174 4581 Acked 5580 Read 5 Done New Done false
14175 4582 Acked 5581 Write 4 Done New Done false
14176 4583 Acked 5582 Read 5 Done New Done false
14177 4584 Acked 5583 Write 3 Done New Done false
14178 4585 Acked 5584 Write 6 Done New Done false
14179 4586 Acked 5585 Read 4 Done New Done false
14180 4587 Acked 5586 Read 6 Done New Done false
14181 4588 Acked 5587 Read 3 Done New Done false
14182 4589 Acked 5588 Write 1 Done New Done false
14183 4590 Acked 5589 Read 5 Done New Done false
14184 4591 Acked 5590 Read 3 Done New Done false
14185 4592 Acked 5591 Read 6 Done New Done false
14186 4593 Acked 5592 Write 10 Done New Done false
14187 4594 Acked 5593 Read 7 Done New Done false
14188 4595 Acked 5594 Flush 0 Done New Done false
14189 4596 Acked 5595 Read 9 Done New Done false
14190 4597 Acked 5596 Read 6 Done New Done false
14191 4598 Acked 5597 Write 8 Done New Done false
14192 4599 Acked 5598 Read 4 Done New Done false
14193 4600 Acked 5599 Write 7 Done New Done false
14194 4601 Acked 5600 Write 4 Done New Done false
14195 4602 Acked 5601 Read 7 Done New Done false
14196 4603 Acked 5602 Read 5 Done New Done false
14197 4604 Acked 5603 Write 3 Done New Done false
14198 4605 Acked 5604 Read 1 Done New Done false
14199 4606 Acked 5605 Read 7 Done New Done false
14200 4607 Acked 5606 Write 4 Done New Done false
14201 4608 Acked 5607 Write 10 Done New Done false
14202 4609 Acked 5608 Write 4 Done New Done false
14203 4610 Acked 5609 Flush 0 Done New Done false
14204 4611 Acked 5610 Flush 0 Done New Done false
14205 4612 Acked 5611 Read 1 Done New Done false
14206 4613 Acked 5612 Read 6 Done New Done false
14207 4614 Acked 5613 Read 3 Done New Done false
14208 4615 Acked 5614 Write 1 Done New Done false
14209 4616 Acked 5615 Read 1 Done New Done false
14210 4617 Acked 5616 Write 2 Done New Done false
14211 4618 Acked 5617 Write 6 Done New Done false
14212 4619 Acked 5618 Write 4 Done New Done false
14213 4620 Acked 5619 Read 5 Done New Done false
14214 4621 Acked 5620 Flush 0 Done New Done false
14215 4622 Acked 5621 Write 8 Done New Done false
14216 4623 Acked 5622 Read 4 Done New Done false
14217 4624 Acked 5623 Read 6 Done New Done false
14218 4625 Acked 5624 Flush 0 Done New Done false
14219 4626 Acked 5625 Flush 0 Done New Done false
14220 4627 Acked 5626 Write 6 Done New Done false
14221 4628 Acked 5627 Read 3 Done New Done false
14222 4629 Acked 5628 Flush 0 Done New Done false
14223 4630 Acked 5629 Read 7 Done New Done false
14224 4631 Acked 5630 Read 1 Done New Done false
14225 4632 Acked 5631 Read 9 Done New Done false
14226 4633 Acked 5632 Write 10 Done New Done false
14227 4634 Acked 5633 Read 9 Done New Done false
14228 4635 Acked 5634 Read 3 Done New Done false
14229 4636 Acked 5635 Read 6 Done New Done false
14230 4637 Acked 5636 Write 2 Done New Done false
14231 4638 Acked 5637 Read 3 Done New Done false
14232 4639 Acked 5638 Write 10 Done New Done false
14233 4640 Acked 5639 Read 3 Done New Done false
14234 4641 Acked 5640 Read 8 Done New Done false
14235 4642 Acked 5641 Flush 0 Done New Done false
14236 4643 Acked 5642 Read 4 Done New Done false
14237 4644 Acked 5643 Read 8 Done New Done false
14238 4645 Acked 5644 Read 9 Done New Done false
14239 4646 Acked 5645 Read 2 Done New Done false
14240 4647 Acked 5646 Read 3 Done New Done false
14241 4648 Acked 5647 Read 2 Done New Done false
14242 4649 Acked 5648 Read 8 Done New Done false
14243 4650 Acked 5649 Read 8 Done New Done false
14244 4651 Acked 5650 Read 5 Done New Done false
14245 4652 Acked 5651 Write 3 Done New Done false
14246 4653 Acked 5652 Read 3 Done New Done false
14247 4654 Acked 5653 Write 8 Done New Done false
14248 4655 Acked 5654 Write 8 Done New Done false
14249 4656 Acked 5655 Read 4 Done New Done false
14250 4657 Acked 5656 Write 9 Done New Done false
14251 4658 Acked 5657 Write 5 Done New Done false
14252 4659 Acked 5658 Flush 0 Done New Done false
14253 4660 Acked 5659 Read 6 Done New Done false
14254 4661 Acked 5660 Write 3 Done New Done false
14255 4662 Acked 5661 Write 9 Done New Done false
14256 4663 Acked 5662 Read 3 Done New Done false
14257 4664 Acked 5663 Write 6 Done New Done false
14258 4665 Acked 5664 Write 8 Done New Done false
14259 4666 Acked 5665 Write 2 Done New Done false
14260 4667 Acked 5666 Read 1 Done New Done false
14261 4668 Acked 5667 Write 1 Done New Done false
14262 4669 Acked 5668 Read 5 Done New Done false
14263 4670 Acked 5669 Write 10 Done New Done false
14264 4671 Acked 5670 Flush 0 Done New Done false
14265 4672 Acked 5671 Flush 0 Done New Done false
14266 4673 Acked 5672 Read 4 Done New Done false
14267 4674 Acked 5673 Write 4 Done New Done false
14268 4675 Acked 5674 Write 10 Done New Done false
14269 4676 Acked 5675 Read 4 Done New Done false
14270 4677 Acked 5676 Write 4 Done New Done false
14271 4678 Acked 5677 Write 1 Done New Done false
14272 4679 Acked 5678 Read 4 Done New Done false
14273 4680 Acked 5679 Write 5 Done New Done false
14274 4681 Acked 5680 Read 8 Done New Done false
14275 4682 Acked 5681 Write 6 Done New Done false
14276 4683 Acked 5682 Write 1 Done New Done false
14277 4684 Acked 5683 Write 2 Done New Done false
14278 4685 Acked 5684 Write 1 Done New Done false
14279 4686 Acked 5685 Write 4 Done New Done false
14280 4687 Acked 5686 Read 2 Done New Done false
14281 4688 Acked 5687 Read 3 Done New Done false
14282 4689 Acked 5688 Write 1 Done New Done false
14283 4690 Acked 5689 Read 8 Done New Done false
14284 4691 Acked 5690 Read 7 Done New Done false
14285 4692 Acked 5691 Write 2 Done New Done false
14286 4693 Acked 5692 Write 6 Done New Done false
14287 4694 Acked 5693 Write 2 Done New Done false
14288 4695 Acked 5694 Write 3 Done New Done false
14289 4696 Acked 5695 Read 1 Done New Done false
14290 4697 Acked 5696 Write 6 Done New Done false
14291 4698 Acked 5697 Read 5 Done New Done false
14292 4699 Acked 5698 Flush 0 Done New Done false
14293 4700 Acked 5699 Write 9 Done New Done false
14294 4701 Acked 5700 Read 5 Done New Done false
14295 4702 Acked 5701 Flush 0 Done New Done false
14296 4703 Acked 5702 Read 1 Done New Done false
14297 4704 Acked 5703 Read 9 Done New Done false
14298 4705 Acked 5704 Read 2 Done New Done false
14299 4706 Acked 5705 Read 6 Done New Done false
14300 4707 Acked 5706 Flush 0 Done New Done false
14301 4708 Acked 5707 Write 5 Done New Done false
14302 4709 Acked 5708 Write 3 Done New Done false
14303 4710 Acked 5709 Read 5 Done New Done false
14304 4711 Acked 5710 Write 5 Done New Done false
14305 4712 Acked 5711 Write 4 Done New Done false
14306 4713 Acked 5712 Write 1 Done New Done false
14307 4714 Acked 5713 Write 8 Done New Done false
14308 4715 Acked 5714 Read 3 Done New Done false
14309 4716 Acked 5715 Read 6 Done New Done false
14310 4717 Acked 5716 Read 10 Done New Done false
14311 4718 Acked 5717 Read 3 Done New Done false
14312 4719 Acked 5718 Read 2 Done New Done false
14313 4720 Acked 5719 Write 10 Done New Done false
14314 4721 Acked 5720 Read 8 Done New Done false
14315 4722 Acked 5721 Flush 0 Done New Done false
14316 4723 Acked 5722 Flush 0 Done New Done false
14317 4724 Acked 5723 Write 1 Done New Done false
14318 4725 Acked 5724 Write 7 Done New Done false
14319 4726 Acked 5725 Read 3 Done New Done false
14320 4727 Acked 5726 Read 8 Done New Done false
14321 4728 Acked 5727 Write 5 Done New Done false
14322 4729 Acked 5728 Read 4 Done New Done false
14323 4730 Acked 5729 Write 3 Done New Done false
14324 4731 Acked 5730 Flush 0 Done New Done false
14325 4732 Acked 5731 Write 7 Done New Done false
14326 4733 Acked 5732 Read 10 Done New Done false
14327 4734 Acked 5733 Flush 0 Done New Done false
14328 4735 Acked 5734 Write 9 Done New Done false
14329 4736 Acked 5735 Read 5 Done New Done false
14330 4737 Acked 5736 Flush 0 Done New Done false
14331 4738 Acked 5737 Write 2 Done New Done false
14332 4739 Acked 5738 Read 4 Done New Done false
14333 4740 Acked 5739 Read 6 Done New Done false
14334 4741 Acked 5740 Read 10 Done New Done false
14335 4742 Acked 5741 Write 7 Done New Done false
14336 4743 Acked 5742 Write 6 Done New Done false
14337 4744 Acked 5743 Read 8 Done New Done false
14338 4745 Acked 5744 Write 3 Done New Done false
14339 4746 Acked 5745 Read 1 Done New Done false
14340 4747 Acked 5746 Write 7 Done New Done false
14341 4748 Acked 5747 Write 3 Done New Done false
14342 4749 Acked 5748 Read 3 Done New Done false
14343 4750 Acked 5749 Read 10 Done New Done false
14344 4751 Acked 5750 Write 6 Done New Done false
14345 4752 Acked 5751 Write 8 Done New Done false
14346 4753 Acked 5752 Write 5 Done New Done false
14347 4754 Acked 5753 Read 2 Done New Done false
14348 4755 Acked 5754 Write 6 Done New Done false
14349 4756 Acked 5755 Write 2 Done New Done false
14350 4757 Acked 5756 Write 5 Done New Done false
14351 4758 Acked 5757 Write 3 Done New Done false
14352 4759 Acked 5758 Read 4 Done New Done false
14353 4760 Acked 5759 Read 3 Done New Done false
14354 4761 Acked 5760 Write 5 Done New Done false
14355 4762 Acked 5761 Write 6 Done New Done false
14356 4763 Acked 5762 Read 10 Done New Done false
14357 4764 Acked 5763 Read 5 Done New Done false
14358 4765 Acked 5764 Flush 0 Done New Done false
14359 4766 Acked 5765 Read 2 Done New Done false
14360 4767 Acked 5766 Read 8 Done New Done false
14361 4768 Acked 5767 Flush 0 Done New Done false
14362 4769 Acked 5768 Flush 0 Done New Done false
14363 4770 Acked 5769 Read 4 Done New Done false
14364 4771 Acked 5770 Read 6 Done New Done false
14365 4772 Acked 5771 Read 5 Done New Done false
14366 4773 Acked 5772 Flush 0 Done New Done false
14367 4774 Acked 5773 Flush 0 Done New Done false
14368 4775 Acked 5774 Write 2 Done New Done false
14369 4776 Acked 5775 Write 6 Done New Done false
14370 4777 Acked 5776 Write 3 Done New Done false
14371 4778 Acked 5777 Write 2 Done New Done false
14372 4779 Acked 5778 Write 2 Done New Done false
14373 4780 Acked 5779 Read 10 Done New Done false
14374 4781 Acked 5780 Write 2 Done New Done false
14375 4782 Acked 5781 Write 6 Done New Done false
14376 4783 Acked 5782 Read 4 Done New Done false
14377 4784 Acked 5783 Read 7 Done New Done false
14378 4785 Acked 5784 Read 10 Done New Done false
14379 4786 Acked 5785 Write 7 Done New Done false
14380 4787 Acked 5786 Write 10 Done New Done false
14381 4788 Acked 5787 Write 4 Done New Done false
14382 4789 Acked 5788 Read 3 Done New Done false
14383 4790 Acked 5789 Write 5 Done New Done false
14384 4791 Acked 5790 Read 5 Done New Done false
14385 4792 Acked 5791 Flush 0 Done New Done false
14386 4793 Acked 5792 Read 6 Done New Done false
14387 4794 Acked 5793 Write 10 Done New Done false
14388 4795 Acked 5794 Write 4 Done New Done false
14389 4796 Acked 5795 Write 10 Done New Done false
14390 4797 Acked 5796 Read 9 Done New Done false
14391 4798 Acked 5797 Write 2 Done New Done false
14392 4799 Acked 5798 Read 5 Done New Done false
14393 4800 Acked 5799 Write 1 Done New Done false
14394 4801 Acked 5800 Flush 0 Done New Done false
14395 4802 Acked 5801 Read 5 Done New Done false
14396 4803 Acked 5802 Read 1 Done New Done false
14397 4804 Acked 5803 Flush 0 Done New Done false
14398 4805 Acked 5804 Write 5 Done New Done false
14399 4806 Acked 5805 Read 7 Done New Done false
14400 4807 Acked 5806 Write 5 Done New Done false
14401 4808 Acked 5807 Write 4 Done New Done false
14402 4809 Acked 5808 Write 2 Done New Done false
14403 4810 Acked 5809 Write 4 Done New Done false
14404 4811 Acked 5810 Write 1 Done New Done false
14405 4812 Acked 5811 Write 1 Done New Done false
14406 4813 Acked 5812 Read 1 Done New Done false
14407 4814 Acked 5813 Write 9 Done New Done false
14408 4815 Acked 5814 Read 4 Done New Done false
14409 4816 Acked 5815 Read 5 Done New Done false
14410 4817 Acked 5816 Write 1 Done New Done false
14411 4818 Acked 5817 Read 6 Done New Done false
14412 4819 Acked 5818 Flush 0 Done New Done false
14413 4820 Acked 5819 Read 1 Done New Done false
14414 4821 Acked 5820 Write 8 Done New Done false
14415 4822 Acked 5821 Write 8 Done New Done false
14416 4823 Acked 5822 Flush 0 Done New Done false
14417 STATES DS:0 DS:1 DS:2 TOTAL
14418 New 0 300 0 300
14419 Sent 0 0 0 0
14420 Done 300 0 300 600
14421 Skipped 0 0 0 0
14422 Error 0 0 0 0
14423 Last Flush: 5822 5522 5822
14424 Downstairs last five completed: 5522 5521 5520 5519 5518
14425 Upstairs last five completed: 4823 4822 4821 4820 4819
14426 CLIENT: Up:0 ds:300 act:2
144272024-02-12T01:37:20.117ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
144282024-02-12T01:37:20.117ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
144292024-02-12T01:37:20.117ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
144302024-02-12T01:37:20.117ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
144312024-02-12T01:37:20.117ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
144322024-02-12T01:37:20.117ZINFOcrucible: [1] client re-new 300 jobs since flush 5522 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
144332024-02-12T01:37:20.117ZINFOcrucible: client task is exiting = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
144342024-02-12T01:37:21.118ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
144352024-02-12T01:37:21.118ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
144362024-02-12T01:37:21.118ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
144372024-02-12T01:37:21.118ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
144382024-02-12T01:37:21.118ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
144392024-02-12T01:37:21.118ZINFOcrucible: [1] client re-new 300 jobs since flush 5522 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
144402024-02-12T01:37:21.118ZINFOcrucible: client task is exiting = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
144412024-02-12T01:37:22.120ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
144422024-02-12T01:37:22.120ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
144432024-02-12T01:37:22.120ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
144442024-02-12T01:37:22.120ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
144452024-02-12T01:37:22.120ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
144462024-02-12T01:37:22.120ZINFOcrucible: [1] client re-new 300 jobs since flush 5522 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
144472024-02-12T01:37:22.120ZINFOcrucible: client task is exiting = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
14448 [8820] got start action
14449 [8820] I am going to restart
14450 Starting downstairs at port 8820
14451 Make output file at "/tmp/dsc/downstairs-8820.txt"
14452 [8820][1] reports Starting
144532024-02-12T01:37:23.121ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
144542024-02-12T01:37:23.121ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
144552024-02-12T01:37:23.121ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
144562024-02-12T01:37:23.121ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
144572024-02-12T01:37:23.121ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
144582024-02-12T01:37:23.121ZINFOcrucible: [1] client re-new 300 jobs since flush 5522 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
144592024-02-12T01:37:23.121ZINFOcrucible: client task is exiting = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
14460 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1180)
14461 [8820][1] reports Running
14462 ----------------------------------------------------------------
14463 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:300
14464 States: Active Offline Active
14465 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
14466 4524 Acked 5523 Write 4 Done New Done true
14467 4525 Acked 5524 Read 5 Done New Done true
14468 4526 Acked 5525 Read 9 Done New Done true
14469 4527 Acked 5526 Write 8 Done New Done true
14470 4528 Acked 5527 Flush 0 Done New Done true
14471 4529 Acked 5528 Read 2 Done New Done true
14472 4530 Acked 5529 Read 4 Done New Done true
14473 4531 Acked 5530 Read 9 Done New Done true
14474 4532 Acked 5531 Write 7 Done New Done true
14475 4533 Acked 5532 Read 9 Done New Done true
14476 4534 Acked 5533 Flush 0 Done New Done true
14477 4535 Acked 5534 Write 4 Done New Done true
14478 4536 Acked 5535 Write 7 Done New Done true
14479 4537 Acked 5536 Write 9 Done New Done true
14480 4538 Acked 5537 Read 10 Done New Done true
14481 4539 Acked 5538 Write 10 Done New Done true
14482 4540 Acked 5539 Write 9 Done New Done true
14483 4541 Acked 5540 Read 10 Done New Done true
14484 4542 Acked 5541 Write 1 Done New Done true
14485 4543 Acked 5542 Flush 0 Done New Done true
14486 4544 Acked 5543 Read 10 Done New Done true
14487 4545 Acked 5544 Read 1 Done New Done true
14488 4546 Acked 5545 Flush 0 Done New Done true
14489 4547 Acked 5546 Read 4 Done New Done true
14490 4548 Acked 5547 Read 7 Done New Done true
14491 4549 Acked 5548 Write 8 Done New Done true
14492 4550 Acked 5549 Write 5 Done New Done true
14493 4551 Acked 5550 Read 10 Done New Done true
14494 4552 Acked 5551 Flush 0 Done New Done true
14495 4553 Acked 5552 Write 6 Done New Done true
14496 4554 Acked 5553 Write 6 Done New Done true
14497 4555 Acked 5554 Read 1 Done New Done true
14498 4556 Acked 5555 Read 1 Done New Done true
14499 4557 Acked 5556 Read 10 Done New Done true
14500 4558 Acked 5557 Read 10 Done New Done true
14501 4559 Acked 5558 Read 2 Done New Done true
14502 4560 Acked 5559 Write 5 Done New Done true
14503 4561 Acked 5560 Read 9 Done New Done true
14504 4562 Acked 5561 Write 10 Done New Done true
14505 4563 Acked 5562 Read 10 Done New Done true
14506 4564 Acked 5563 Read 2 Done New Done true
14507 4565 Acked 5564 Flush 0 Done New Done true
14508 4566 Acked 5565 Read 5 Done New Done true
14509 4567 Acked 5566 Write 4 Done New Done true
14510 4568 Acked 5567 Read 6 Done New Done true
14511 4569 Acked 5568 Read 8 Done New Done true
14512 4570 Acked 5569 Write 5 Done New Done true
14513 4571 Acked 5570 Read 9 Done New Done true
14514 4572 Acked 5571 Read 7 Done New Done true
14515 4573 Acked 5572 Read 4 Done New Done true
14516 4574 Acked 5573 Write 2 Done New Done true
14517 4575 Acked 5574 Write 9 Done New Done true
14518 4576 Acked 5575 Read 4 Done New Done true
14519 4577 Acked 5576 Read 3 Done New Done true
14520 4578 Acked 5577 Write 1 Done New Done true
14521 4579 Acked 5578 Write 8 Done New Done true
14522 4580 Acked 5579 Flush 0 Done New Done true
14523 4581 Acked 5580 Read 5 Done New Done true
14524 4582 Acked 5581 Write 4 Done New Done true
14525 4583 Acked 5582 Read 5 Done New Done true
14526 4584 Acked 5583 Write 3 Done New Done true
14527 4585 Acked 5584 Write 6 Done New Done true
14528 4586 Acked 5585 Read 4 Done New Done true
14529 4587 Acked 5586 Read 6 Done New Done true
14530 4588 Acked 5587 Read 3 Done New Done true
14531 4589 Acked 5588 Write 1 Done New Done true
14532 4590 Acked 5589 Read 5 Done New Done true
14533 4591 Acked 5590 Read 3 Done New Done true
14534 4592 Acked 5591 Read 6 Done New Done true
14535 4593 Acked 5592 Write 10 Done New Done true
14536 4594 Acked 5593 Read 7 Done New Done true
14537 4595 Acked 5594 Flush 0 Done New Done true
14538 4596 Acked 5595 Read 9 Done New Done true
14539 4597 Acked 5596 Read 6 Done New Done true
14540 4598 Acked 5597 Write 8 Done New Done true
14541 4599 Acked 5598 Read 4 Done New Done true
14542 4600 Acked 5599 Write 7 Done New Done true
14543 4601 Acked 5600 Write 4 Done New Done true
14544 4602 Acked 5601 Read 7 Done New Done true
14545 4603 Acked 5602 Read 5 Done New Done true
14546 4604 Acked 5603 Write 3 Done New Done true
14547 4605 Acked 5604 Read 1 Done New Done true
14548 4606 Acked 5605 Read 7 Done New Done true
14549 4607 Acked 5606 Write 4 Done New Done true
14550 4608 Acked 5607 Write 10 Done New Done true
14551 4609 Acked 5608 Write 4 Done New Done true
14552 4610 Acked 5609 Flush 0 Done New Done true
14553 4611 Acked 5610 Flush 0 Done New Done true
14554 4612 Acked 5611 Read 1 Done New Done true
14555 4613 Acked 5612 Read 6 Done New Done true
14556 4614 Acked 5613 Read 3 Done New Done true
14557 4615 Acked 5614 Write 1 Done New Done true
14558 4616 Acked 5615 Read 1 Done New Done true
14559 4617 Acked 5616 Write 2 Done New Done true
14560 4618 Acked 5617 Write 6 Done New Done true
14561 4619 Acked 5618 Write 4 Done New Done true
14562 4620 Acked 5619 Read 5 Done New Done true
14563 4621 Acked 5620 Flush 0 Done New Done true
14564 4622 Acked 5621 Write 8 Done New Done true
14565 4623 Acked 5622 Read 4 Done New Done true
14566 4624 Acked 5623 Read 6 Done New Done true
14567 4625 Acked 5624 Flush 0 Done New Done true
14568 4626 Acked 5625 Flush 0 Done New Done true
14569 4627 Acked 5626 Write 6 Done New Done true
14570 4628 Acked 5627 Read 3 Done New Done true
14571 4629 Acked 5628 Flush 0 Done New Done true
14572 4630 Acked 5629 Read 7 Done New Done true
14573 4631 Acked 5630 Read 1 Done New Done true
14574 4632 Acked 5631 Read 9 Done New Done true
14575 4633 Acked 5632 Write 10 Done New Done true
14576 4634 Acked 5633 Read 9 Done New Done true
14577 4635 Acked 5634 Read 3 Done New Done true
14578 4636 Acked 5635 Read 6 Done New Done true
14579 4637 Acked 5636 Write 2 Done New Done true
14580 4638 Acked 5637 Read 3 Done New Done true
14581 4639 Acked 5638 Write 10 Done New Done true
14582 4640 Acked 5639 Read 3 Done New Done true
14583 4641 Acked 5640 Read 8 Done New Done true
14584 4642 Acked 5641 Flush 0 Done New Done true
14585 4643 Acked 5642 Read 4 Done New Done true
14586 4644 Acked 5643 Read 8 Done New Done true
14587 4645 Acked 5644 Read 9 Done New Done true
14588 4646 Acked 5645 Read 2 Done New Done true
14589 4647 Acked 5646 Read 3 Done New Done true
14590 4648 Acked 5647 Read 2 Done New Done true
14591 4649 Acked 5648 Read 8 Done New Done true
14592 4650 Acked 5649 Read 8 Done New Done true
14593 4651 Acked 5650 Read 5 Done New Done true
14594 4652 Acked 5651 Write 3 Done New Done true
14595 4653 Acked 5652 Read 3 Done New Done true
14596 4654 Acked 5653 Write 8 Done New Done true
14597 4655 Acked 5654 Write 8 Done New Done true
14598 4656 Acked 5655 Read 4 Done New Done true
14599 4657 Acked 5656 Write 9 Done New Done true
14600 4658 Acked 5657 Write 5 Done New Done true
14601 4659 Acked 5658 Flush 0 Done New Done true
14602 4660 Acked 5659 Read 6 Done New Done true
14603 4661 Acked 5660 Write 3 Done New Done true
14604 4662 Acked 5661 Write 9 Done New Done true
14605 4663 Acked 5662 Read 3 Done New Done true
14606 4664 Acked 5663 Write 6 Done New Done true
14607 4665 Acked 5664 Write 8 Done New Done true
14608 4666 Acked 5665 Write 2 Done New Done true
14609 4667 Acked 5666 Read 1 Done New Done true
14610 4668 Acked 5667 Write 1 Done New Done true
14611 4669 Acked 5668 Read 5 Done New Done true
14612 4670 Acked 5669 Write 10 Done New Done true
14613 4671 Acked 5670 Flush 0 Done New Done true
14614 4672 Acked 5671 Flush 0 Done New Done true
14615 4673 Acked 5672 Read 4 Done New Done true
14616 4674 Acked 5673 Write 4 Done New Done true
14617 4675 Acked 5674 Write 10 Done New Done true
14618 4676 Acked 5675 Read 4 Done New Done true
14619 4677 Acked 5676 Write 4 Done New Done true
14620 4678 Acked 5677 Write 1 Done New Done true
14621 4679 Acked 5678 Read 4 Done New Done true
14622 4680 Acked 5679 Write 5 Done New Done true
14623 4681 Acked 5680 Read 8 Done New Done true
14624 4682 Acked 5681 Write 6 Done New Done true
14625 4683 Acked 5682 Write 1 Done New Done true
14626 4684 Acked 5683 Write 2 Done New Done true
14627 4685 Acked 5684 Write 1 Done New Done true
14628 4686 Acked 5685 Write 4 Done New Done true
14629 4687 Acked 5686 Read 2 Done New Done true
14630 4688 Acked 5687 Read 3 Done New Done true
14631 4689 Acked 5688 Write 1 Done New Done true
14632 4690 Acked 5689 Read 8 Done New Done true
14633 4691 Acked 5690 Read 7 Done New Done true
14634 4692 Acked 5691 Write 2 Done New Done true
14635 4693 Acked 5692 Write 6 Done New Done true
14636 4694 Acked 5693 Write 2 Done New Done true
14637 4695 Acked 5694 Write 3 Done New Done true
14638 4696 Acked 5695 Read 1 Done New Done true
14639 4697 Acked 5696 Write 6 Done New Done true
14640 4698 Acked 5697 Read 5 Done New Done true
14641 4699 Acked 5698 Flush 0 Done New Done true
14642 4700 Acked 5699 Write 9 Done New Done true
14643 4701 Acked 5700 Read 5 Done New Done true
14644 4702 Acked 5701 Flush 0 Done New Done true
14645 4703 Acked 5702 Read 1 Done New Done true
14646 4704 Acked 5703 Read 9 Done New Done true
14647 4705 Acked 5704 Read 2 Done New Done true
14648 4706 Acked 5705 Read 6 Done New Done true
14649 4707 Acked 5706 Flush 0 Done New Done true
14650 4708 Acked 5707 Write 5 Done New Done true
14651 4709 Acked 5708 Write 3 Done New Done true
14652 4710 Acked 5709 Read 5 Done New Done true
14653 4711 Acked 5710 Write 5 Done New Done true
14654 4712 Acked 5711 Write 4 Done New Done true
14655 4713 Acked 5712 Write 1 Done New Done true
14656 4714 Acked 5713 Write 8 Done New Done true
14657 4715 Acked 5714 Read 3 Done New Done true
14658 4716 Acked 5715 Read 6 Done New Done true
14659 4717 Acked 5716 Read 10 Done New Done true
14660 4718 Acked 5717 Read 3 Done New Done true
14661 4719 Acked 5718 Read 2 Done New Done true
14662 4720 Acked 5719 Write 10 Done New Done true
14663 4721 Acked 5720 Read 8 Done New Done true
14664 4722 Acked 5721 Flush 0 Done New Done true
14665 4723 Acked 5722 Flush 0 Done New Done true
14666 4724 Acked 5723 Write 1 Done New Done true
14667 4725 Acked 5724 Write 7 Done New Done true
14668 4726 Acked 5725 Read 3 Done New Done true
14669 4727 Acked 5726 Read 8 Done New Done true
14670 4728 Acked 5727 Write 5 Done New Done true
14671 4729 Acked 5728 Read 4 Done New Done true
14672 4730 Acked 5729 Write 3 Done New Done true
14673 4731 Acked 5730 Flush 0 Done New Done true
14674 4732 Acked 5731 Write 7 Done New Done true
14675 4733 Acked 5732 Read 10 Done New Done true
14676 4734 Acked 5733 Flush 0 Done New Done true
14677 4735 Acked 5734 Write 9 Done New Done true
14678 4736 Acked 5735 Read 5 Done New Done true
14679 4737 Acked 5736 Flush 0 Done New Done true
14680 4738 Acked 5737 Write 2 Done New Done true
14681 4739 Acked 5738 Read 4 Done New Done true
14682 4740 Acked 5739 Read 6 Done New Done true
14683 4741 Acked 5740 Read 10 Done New Done true
14684 4742 Acked 5741 Write 7 Done New Done true
14685 4743 Acked 5742 Write 6 Done New Done true
14686 4744 Acked 5743 Read 8 Done New Done true
14687 4745 Acked 5744 Write 3 Done New Done true
14688 4746 Acked 5745 Read 1 Done New Done true
14689 4747 Acked 5746 Write 7 Done New Done true
14690 4748 Acked 5747 Write 3 Done New Done true
14691 4749 Acked 5748 Read 3 Done New Done true
14692 4750 Acked 5749 Read 10 Done New Done true
14693 4751 Acked 5750 Write 6 Done New Done true
14694 4752 Acked 5751 Write 8 Done New Done true
14695 4753 Acked 5752 Write 5 Done New Done true
14696 4754 Acked 5753 Read 2 Done New Done true
14697 4755 Acked 5754 Write 6 Done New Done true
14698 4756 Acked 5755 Write 2 Done New Done true
14699 4757 Acked 5756 Write 5 Done New Done true
14700 4758 Acked 5757 Write 3 Done New Done true
14701 4759 Acked 5758 Read 4 Done New Done true
14702 4760 Acked 5759 Read 3 Done New Done true
14703 4761 Acked 5760 Write 5 Done New Done true
14704 4762 Acked 5761 Write 6 Done New Done true
14705 4763 Acked 5762 Read 10 Done New Done true
14706 4764 Acked 5763 Read 5 Done New Done true
14707 4765 Acked 5764 Flush 0 Done New Done true
14708 4766 Acked 5765 Read 2 Done New Done true
14709 4767 Acked 5766 Read 8 Done New Done true
14710 4768 Acked 5767 Flush 0 Done New Done true
14711 4769 Acked 5768 Flush 0 Done New Done true
14712 4770 Acked 5769 Read 4 Done New Done true
14713 4771 Acked 5770 Read 6 Done New Done true
14714 4772 Acked 5771 Read 5 Done New Done true
14715 4773 Acked 5772 Flush 0 Done New Done true
14716 4774 Acked 5773 Flush 0 Done New Done true
14717 4775 Acked 5774 Write 2 Done New Done true
14718 4776 Acked 5775 Write 6 Done New Done true
14719 4777 Acked 5776 Write 3 Done New Done true
14720 4778 Acked 5777 Write 2 Done New Done true
14721 4779 Acked 5778 Write 2 Done New Done true
14722 4780 Acked 5779 Read 10 Done New Done true
14723 4781 Acked 5780 Write 2 Done New Done true
14724 4782 Acked 5781 Write 6 Done New Done true
14725 4783 Acked 5782 Read 4 Done New Done true
14726 4784 Acked 5783 Read 7 Done New Done true
14727 4785 Acked 5784 Read 10 Done New Done true
14728 4786 Acked 5785 Write 7 Done New Done true
14729 4787 Acked 5786 Write 10 Done New Done true
14730 4788 Acked 5787 Write 4 Done New Done true
14731 4789 Acked 5788 Read 3 Done New Done true
14732 4790 Acked 5789 Write 5 Done New Done true
14733 4791 Acked 5790 Read 5 Done New Done true
14734 4792 Acked 5791 Flush 0 Done New Done true
14735 4793 Acked 5792 Read 6 Done New Done true
14736 4794 Acked 5793 Write 10 Done New Done true
14737 4795 Acked 5794 Write 4 Done New Done true
14738 4796 Acked 5795 Write 10 Done New Done true
14739 4797 Acked 5796 Read 9 Done New Done true
14740 4798 Acked 5797 Write 2 Done New Done true
14741 4799 Acked 5798 Read 5 Done New Done true
14742 4800 Acked 5799 Write 1 Done New Done true
14743 4801 Acked 5800 Flush 0 Done New Done true
14744 4802 Acked 5801 Read 5 Done New Done true
14745 4803 Acked 5802 Read 1 Done New Done true
14746 4804 Acked 5803 Flush 0 Done New Done true
14747 4805 Acked 5804 Write 5 Done New Done true
14748 4806 Acked 5805 Read 7 Done New Done true
14749 4807 Acked 5806 Write 5 Done New Done true
14750 4808 Acked 5807 Write 4 Done New Done true
14751 4809 Acked 5808 Write 2 Done New Done true
14752 4810 Acked 5809 Write 4 Done New Done true
14753 4811 Acked 5810 Write 1 Done New Done true
14754 4812 Acked 5811 Write 1 Done New Done true
14755 4813 Acked 5812 Read 1 Done New Done true
14756 4814 Acked 5813 Write 9 Done New Done true
14757 4815 Acked 5814 Read 4 Done New Done true
14758 4816 Acked 5815 Read 5 Done New Done true
14759 4817 Acked 5816 Write 1 Done New Done true
14760 4818 Acked 5817 Read 6 Done New Done true
14761 4819 Acked 5818 Flush 0 Done New Done true
14762 4820 Acked 5819 Read 1 Done New Done true
14763 4821 Acked 5820 Write 8 Done New Done true
14764 4822 Acked 5821 Write 8 Done New Done true
14765 4823 Acked 5822 Flush 0 Done New Done true
14766 STATES DS:0 DS:1 DS:2 TOTAL
14767 New 0 300 0 300
14768 Sent 0 0 0 0
14769 Done 300 0 300 600
14770 Skipped 0 0 0 0
14771 Error 0 0 0 0
14772 Last Flush: 5822 5522 5822
14773 Downstairs last five completed: 5522 5521 5520 5519 5518
14774 Upstairs last five completed: 4823 4822 4821 4820 4819
14775 CLIENT: Up:0 ds:300 act:2
147762024-02-12T01:37:24.122ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
147772024-02-12T01:37:24.122ZINFOcrucible: ds_connection connected = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
147782024-02-12T01:37:24.124ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
147792024-02-12T01:37:24.124ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
147802024-02-12T01:37:24.124ZINFOcrucible: send last flush ID to this DS: 5522 = downstairs client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
147812024-02-12T01:37:24.124ZINFOcrucible: Replied this last flush ID: 5522 = downstairs client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
147822024-02-12T01:37:24.124ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
147832024-02-12T01:37:24.124ZINFOcrucible: [1] Transition from Offline to Replay = downstairs client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
147842024-02-12T01:37:24.124ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
147852024-02-12T01:37:24.124ZINFOcrucible: [1] Transition from Replay to Active = downstairs client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
14786 ----------------------------------------------------------------
14787 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
14788 Downstairs last five completed: 5822 5821 5820 5819 5818
14789 Upstairs last five completed: 4823 4822 4821 4820 4819
14790 CLIENT: Up:0 ds:0 act:3
14791 Replay: All jobs finished, all DS active.
14792 Main task has work to do, go find it
14793 got dsc Stop(2)
14794 stop 2
14795 Feb 12 01:37:27.459 INFO request completed, latency_us: [8830] Got stop action so:false kr:false
14796 364, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 29fd9baa-559c-4d58-ba3f-4433e6ae646d, remote_addr: 127.0.0.1:60456, local_addr: 127.0.0.1:9998
147972024-02-12T01:37:27.460ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
147982024-02-12T01:37:27.460ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
14799 Feb 12 01:37:27.460 INFO request completed, latency_us: 156, response_code: 200, uri:{"msg":"downstairs task for 2 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-02-12T01:37:27.460180579Z","hostname":"ip-10-150-1-104.us-west-2.compute.internal","pid":1164,"session_id":"e1773b3e-b36f-4080-8b13-c0830f0d072e"}
14800 /state/cid/2, method: GET{"msg":"Gone missing, transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-02-12T01:37:27.460216654Z","hostname":"ip-10-150-1-104.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"e1773b3e-b36f-4080-8b13-c0830f0d072e"}
14801 , req_id: 2de60961-f361-4e98-a2ac-b110662d05bf, remote_addr: 127.0.{"msg":"[2] client re-new 0 jobs since flush 5822","v":0,"name":"crucible","level":30,"time":"2024-02-12T01:37:27.460257658Z","hostname":"ip-10-150-1-104.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"e1773b3e-b36f-4080-8b13-c0830f0d072e"}
14802 0.1:60456, local_addr: 127.0.{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-12T01:37:27.460299422Z","hostname":"ip-10-150-1-104.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"e1773b3e-b36f-4080-8b13-c0830f0d072e"}
14803 0.1:9998
14804 [8830] Exited with: ExitStatus(unix_wait_status(9))
14805 [8830][2] reports Exit
148062024-02-12T01:37:28.461ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
148072024-02-12T01:37:28.461ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
148082024-02-12T01:37:28.461ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
148092024-02-12T01:37:28.461ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
148102024-02-12T01:37:28.461ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
148112024-02-12T01:37:28.461ZINFOcrucible: [2] client re-new 0 jobs since flush 5822 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
148122024-02-12T01:37:28.461ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
148132024-02-12T01:37:29.463ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
148142024-02-12T01:37:29.463ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
148152024-02-12T01:37:29.463ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
148162024-02-12T01:37:29.463ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
148172024-02-12T01:37:29.463ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
148182024-02-12T01:37:29.463ZINFOcrucible: [2] client re-new 0 jobs since flush 5822 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
148192024-02-12T01:37:29.463ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
148202024-02-12T01:37:30.465ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
148212024-02-12T01:37:30.465ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
148222024-02-12T01:37:30.465ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
148232024-02-12T01:37:30.465ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
148242024-02-12T01:37:30.465ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
148252024-02-12T01:37:30.465ZINFOcrucible: [2] client re-new 0 jobs since flush 5822 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
148262024-02-12T01:37:30.465ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
14827 Feb 12 01:37:31.462 INFO request completed, latency_us: 221, response_code: 200, uri: /state/cid/2, method: GET, req_id: 3ae1433d-5064-4f5a-a340-fed4f2a4f1f3, remote_addr: 127.0.0.1:60456, local_addr: 127.0.0.1:9998
14828 001/300 Flush
14829 002/300 Read block 400 len 40960
14830 003/300 Write block 2146 len 28672 data: 4 3 3 3 3 4 4
14831 004/300 Read block 4438 len 24576
148322024-02-12T01:37:31.466ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
148332024-02-12T01:37:31.466ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
148342024-02-12T01:37:31.467ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
148352024-02-12T01:37:31.467ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
148362024-02-12T01:37:31.467ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
148372024-02-12T01:37:31.467ZINFOcrucible: [2] client re-new 4 jobs since flush 5822 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
148382024-02-12T01:37:31.467ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
14839 005/300 Write block 4360 len 36864 data: 6 7 7 7 6 7 5 4 3
14840 006/300 Write block 1347 len 24576 data: 3 3 2 2 2 3
14841 007/300 Write block 2017 len 36864 data: 5 4 4 4 4 5 5 4 4
14842 008/300 Read block 4094 len 4096
14843 009/300 Read block 2181 len 28672
14844 010/300 Write block 3361 len 12288 data: 5 5 5
14845 011/300 Read block 24 len 8192
14846 012/300 Write block 2623 len 32768 data: 4 4 6 4 4 4 5 3
14847 013/300 Read block 4638 len 12288
14848 014/300 Write block 3224 len 28672 data: 7 7 7 7 8 7 6
14849 015/300 Read block 2030 len 28672
14850 016/300 Write block 714 len 40960 data: 4 3 3 4 6 4 3 3 3 3
14851 017/300 Read block 1750 len 12288
14852 018/300 Read block 603 len 8192
14853 019/300 Read block 1929 len 16384
14854 020/300 Write block 994 len 4096 data: 2
14855 021/300 Read block 290 len 12288
14856 022/300 Flush
14857 023/300 Write block 2428 len 4096 data: 5
14858 024/300 Read block 2078 len 40960
14859 025/300 Write block 3654 len 12288 data: 3 3 3
14860 026/300 Flush
14861 027/300 Read block 296 len 4096
14862 028/300 Read block 4840 len 32768
14863 029/300 Read block 3964 len 32768
14864 030/300 Write block 3649 len 16384 data: 3 3 3 2
14865 031/300 Read block 1120 len 40960
14866 032/300 Read block 4287 len 36864
14867 033/300 Read block 4165 len 32768
14868 034/300 Read block 4180 len 28672
14869 035/300 Flush
14870 036/300 Write block 916 len 12288 data: 4 4 4
14871 037/300 Flush
14872 038/300 Read block 2972 len 28672
14873 039/300 Read block 4300 len 28672
14874 040/300 Write block 4552 len 12288 data: 6 4 5
14875 041/300 Read block 2447 len 32768
14876 042/300 Flush
14877 043/300 Write block 1821 len 28672 data: 6 7 6 4 4 4 6
14878 044/300 Write block 383 len 40960 data: 2 3 4 6 6 5 5 5 5 6
14879 045/300 Flush
14880 046/300 Read block 446 len 8192
14881 047/300 Read block 4058 len 12288
14882 048/300 Write block 3240 len 24576 data: 5 3 3 3 3 3
14883 049/300 Read block 3548 len 36864
14884 050/300 Read block 806 len 16384
14885 051/300 Write block 902 len 4096 data: 3
14886 052/300 Read block 1903 len 16384
14887 053/300 Read block 4843 len 20480
14888 054/300 Read block 3775 len 12288
14889 055/300 Write block 2601 len 20480 data: 5 6 6 5 4
14890 056/300 Write block 3153 len 28672 data: 4 3 3 2 3 3 3
14891 057/300 Read block 3051 len 36864
14892 058/300 Read block 4327 len 36864
14893 059/300 Read block 2064 len 40960
14894 060/300 Flush
14895 061/300 Write block 2924 len 8192 data: 7 5
14896 062/300 Read block 4522 len 16384
14897 063/300 Read block 4648 len 20480
14898 064/300 Read block 1041 len 8192
14899 065/300 Read block 2241 len 24576
14900 066/300 Flush
14901 067/300 Flush
14902 068/300 Read block 4501 len 24576
14903 069/300 Read block 678 len 28672
14904 070/300 Write block 2871 len 32768 data: 5 5 5 5 5 6 4 4
14905 071/300 Read block 518 len 36864
14906 072/300 Read block 1951 len 40960
14907 073/300 Read block 4523 len 4096
14908 074/300 Read block 4733 len 24576
14909 075/300 Read block 2947 len 20480
14910 076/300 Write block 4256 len 4096 data: 5
14911 077/300 Flush
14912 078/300 Read block 1970 len 16384
14913 079/300 Read block 3279 len 12288
14914 080/300 Write block 2297 len 36864 data: 4 4 4 4 4 4 3 3 4
14915 081/300 Flush
14916 082/300 Write block 2376 len 20480 data: 5 5 6 5 5
14917 083/300 Write block 2460 len 32768 data: 3 2 2 2 2 3 3 3
14918 084/300 Write block 1565 len 28672 data: 8 9 8 8 7 6 5
14919 085/300 Write block 2462 len 36864 data: 3 3 3 4 4 4 3 5 6
14920 086/300 Write block 4013 len 12288 data: 3 4 4
14921 087/300 Flush
14922 088/300 Write block 3421 len 36864 data: 4 5 5 6 6 5 4 5 7
14923 089/300 Read block 1959 len 20480
14924 090/300 Write block 3617 len 24576 data: 4 4 4 4 4 4
14925 091/300 Write block 4134 len 24576 data: 3 4 4 4 5 7
14926 092/300 Flush
14927 093/300 Write block 3118 len 12288 data: 5 5 6
14928 094/300 Write block 3753 len 40960 data: 5 5 5 4 5 4 4 4 2 2
14929 095/300 Write block 1847 len 40960 data: 7 8 7 8 7 5 4 4 4 7
14930 096/300 Read block 3741 len 32768
14931 097/300 Read block 1596 len 12288
14932 098/300 Write block 227 len 28672 data: 5 5 5 5 5 6 4
14933 099/300 Read block 4350 len 4096
14934 100/300 Write block 3130 len 20480 data: 5 4 4 4 4
14935 101/300 Write block 1196 len 12288 data: 4 3 2
14936 102/300 Read block 2673 len 20480
14937 103/300 Read block 2317 len 4096
14938 104/300 Write block 2207 len 28672 data: 5 6 5 4 5 4 6
14939 105/300 Flush
14940 106/300 Read block 4830 len 12288
14941 107/300 Write block 705 len 40960 data: 4 4 2 2 2 2 4 4 3 5
14942 108/300 Write block 4771 len 28672 data: 3 3 3 2 3 3 3
14943 109/300 Write block 748 len 36864 data: 7 7 6 4 5 6 4 4 3
14944 110/300 Read block 4810 len 28672
14945 111/300 Read block 1927 len 12288
14946 112/300 Read block 3539 len 40960
14947 113/300 Read block 483 len 24576
14948 114/300 Read block 877 len 32768
14949 115/300 Flush
14950 116/300 Read block 4492 len 24576
14951 117/300 Write block 1884 len 28672 data: 2 3 3 3 5 4 4
14952 118/300 Read block 1184 len 4096
14953 119/300 Read block 2437 len 4096
14954 120/300 Flush
14955 121/300 Read block 3258 len 16384
14956 122/300 Read block 3652 len 8192
14957 123/300 Write block 3843 len 4096 data: 2
14958 124/300 Read block 456 len 16384
14959 125/300 Read block 1495 len 8192
14960 126/300 Write block 4655 len 20480 data: 3 3 3 2 2
14961 127/300 Read block 2389 len 4096
14962 128/300 Read block 1035 len 24576
14963 129/300 Read block 3963 len 16384
14964 130/300 Write block 2894 len 28672 data: 4 4 5 6 5 4 3
14965 131/300 Write block 2166 len 40960 data: 4 4 4 4 6 8 8 6 6 9
14966 132/300 Flush
14967 133/300 Read block 3476 len 12288
14968 134/300 Read block 1136 len 12288
14969 135/300 Read block 2060 len 32768
14970 136/300 Write block 446 len 16384 data: 5 4 4 5
14971 137/300 Write block 2744 len 36864 data: 4 4 4 5 4 4 5 4 6
14972 138/300 Read block 4572 len 16384
14973 139/300 Write block 2040 len 16384 data: 4 4 3 3
14974 140/300 Read block 3612 len 24576
14975 141/300 Read block 2441 len 28672
14976 142/300 Read block 401 len 12288
14977 143/300 Write block 2895 len 16384 data: 5 6 7 6
14978 144/300 Read block 1350 len 36864
14979 145/300 Read block 3590 len 4096
14980 146/300 Write block 607 len 4096 data: 4
14981 147/300 Write block 1319 len 8192 data: 4 4
14982 148/300 Write block 2238 len 20480 data: 5 5 4 4 4
14983 149/300 Read block 1164 len 32768
14984 150/300 Write block 2447 len 40960 data: 4 6 5 3 3 2 2 2 3 3
14985 151/300 Read block 4022 len 12288
14986 152/300 Write block 1907 len 12288 data: 3 3 3
14987 153/300 Read block 4288 len 24576
14988 154/300 Write block 381 len 12288 data: 4 4 3
14989 155/300 Read block 1102 len 24576
14990 156/300 Write block 4775 len 4096 data: 4
14991 157/300 Read block 2478 len 12288
14992 158/300 Read block 328 len 24576
14993 159/300 Write block 2177 len 28672 data: 7 7 7 5 4 2 3
14994 160/300 Write block 1134 len 36864 data: 3 3 4 4 4 3 5 4 6
14995 161/300 Read block 1164 len 32768
14996 162/300 Write block 4784 len 4096 data: 4
14997 163/300 Read block 1769 len 4096
14998 164/300 Write block 15 len 36864 data: 3 3 4 5 5 5 4 3 2
14999 165/300 Write block 4890 len 16384 data: 4 2 2 2
15000 166/300 Read block 4818 len 12288
15001 167/300 Write block 3595 len 4096 data: 5
15002 168/300 Write block 1184 len 12288 data: 4 3 3
15003 169/300 Read block 3699 len 4096
15004 170/300 Write block 487 len 40960 data: 4 3 3 4 3 3 3 4 4 4
15005 171/300 Write block 1215 len 12288 data: 5 6 6
15006 172/300 Flush
15007 173/300 Write block 4221 len 32768 data: 5 6 6 6 6 5 4 4
15008 174/300 Read block 480 len 20480
15009 175/300 Read block 3421 len 40960
15010 176/300 Write block 3540 len 4096 data: 3
15011 177/300 Write block 4191 len 20480 data: 2 2 2 2 2
15012 178/300 Write block 4550 len 20480 data: 5 6 7 5 6
15013 179/300 Read block 2854 len 4096
15014 180/300 Read block 177 len 36864
15015 181/300 Write block 1069 len 36864 data: 3 4 5 3 4 4 3 3 3
15016 182/300 Write block 2597 len 40960 data: 6 7 6 6 6 7 7 6 5 4
15017 183/300 Read block 745 len 40960
15018 184/300 Write block 684 len 40960 data: 4 4 4 3 3 3 2 3 3 3
15019 185/300 Read block 4322 len 24576
15020 186/300 Read block 1447 len 20480
15021 187/300 Read block 1954 len 16384
15022 188/300 Read block 3515 len 16384
15023 189/300 Read block 3952 len 16384
15024 190/300 Read block 2694 len 24576
15025 191/300 Write block 1583 len 16384 data: 4 4 5 4
15026 192/300 Read block 3999 len 12288
15027 193/300 Read block 2030 len 24576
15028 194/300 Read block 921 len 8192
15029 195/300 Write block 4779 len 12288 data: 3 5 5
15030 196/300 Flush
15031 197/300 Flush
15032 198/300 Write block 964 len 4096 data: 2
15033 199/300 Read block 4745 len 8192
15034 200/300 Write block 2619 len 12288 data: 5 5 5
15035 201/300 Read block 636 len 28672
15036 202/300 Read block 407 len 16384
15037 203/300 Read block 2658 len 12288
15038 204/300 Write block 3734 len 40960 data: 5 5 4 3 5 6 4 5 6 6
15039 205/300 Read block 4295 len 40960
15040 206/300 Read block 4563 len 36864
15041 207/300 Read block 2801 len 40960
15042 208/300 Read block 659 len 36864
15043 209/300 Read block 3939 len 12288
15044 210/300 Flush
15045 211/300 Read block 4036 len 40960
15046 212/300 Flush
15047 213/300 Read block 1636 len 40960
15048 214/300 Read block 2916 len 12288
15049 215/300 Read block 4929 len 32768
15050 216/300 Read block 2895 len 8192
15051 217/300 Write block 871 len 4096 data: 4
15052 218/300 Read block 3353 len 24576
15053 219/300 Read block 4443 len 4096
15054 220/300 Read block 4508 len 28672
15055 221/300 Flush
15056 222/300 Write block 4197 len 40960 data: 2 2 2 4 5 5 6 6 6 5
15057 223/300 Read block 353 len 20480
15058 224/300 Read block 2808 len 16384
15059 225/300 Read block 1159 len 32768
15060 226/300 Read block 3707 len 24576
15061 227/300 Read block 19 len 4096
15062 228/300 Read block 3399 len 40960
15063 229/300 Read block 4082 len 24576
15064 230/300 Read block 3441 len 20480
15065 231/300 Write block 1103 len 12288 data: 3 4 4
15066 232/300 Read block 4244 len 28672
15067 233/300 Write block 1965 len 32768 data: 3 4 5 4 4 4 5 4
15068 234/300 Read block 2533 len 8192
15069 235/300 Read block 194 len 12288
15070 236/300 Flush
15071 237/300 Read block 2664 len 12288
15072 238/300 Read block 1666 len 28672
15073 239/300 Write block 1436 len 36864 data: 3 3 3 3 2 2 2 2 2
15074 240/300 Write block 4715 len 28672 data: 5 4 5 6 5 5 6
15075 241/300 Read block 827 len 28672
15076 242/300 Read block 2272 len 20480
15077 243/300 Read block 1096 len 36864
15078 244/300 Write block 4171 len 28672 data: 5 6 7 8 4 4 4
15079 245/300 Write block 1448 len 20480 data: 5 5 5 5 5
15080 246/300 Read block 4089 len 40960
15081 247/300 Read block 1621 len 8192
15082 248/300 Write block 4889 len 12288 data: 4 5 3
15083 249/300 Read block 503 len 20480
15084 250/300 Read block 3445 len 4096
15085 251/300 Flush
15086 252/300 Write block 1880 len 4096 data: 4
15087 253/300 Read block 2284 len 16384
15088 254/300 Read block 1592 len 4096
15089 255/300 Write block 256 len 16384 data: 6 6 5 6
15090 256/300 Write block 3951 len 36864 data: 4 4 4 5 4 4 6 4 5
15091 257/300 Write block 2736 len 32768 data: 4 4 4 6 5 5 5 5
15092 258/300 Read block 523 len 4096
15093 259/300 Read block 567 len 40960
15094 260/300 Write block 368 len 32768 data: 3 4 4 4 4 4 4 3
15095 261/300 Read block 4943 len 16384
15096 262/300 Read block 1958 len 20480
15097 263/300 Write block 2398 len 40960 data: 3 4 4 5 4 4 4 5 6 4
15098 264/300 Read block 1823 len 12288
15099 265/300 Read block 969 len 20480
15100 266/300 Write block 1078 len 16384 data: 5 4 5 5
15101 267/300 Write block 2977 len 8192 data: 3 5
15102 268/300 Read block 719 len 12288
15103 269/300 Read block 4039 len 40960
15104 270/300 Write block 3676 len 36864 data: 5 4 4 4 4 6 6 6 6
15105 271/300 Write block 3284 len 24576 data: 5 6 6 4 4 2
15106 272/300 Read block 3978 len 40960
15107 273/300 Read block 4069 len 12288
15108 274/300 Write block 1668 len 24576 data: 5 5 5 6 6 6
15109 275/300 Read block 3138 len 20480
15110 276/300 Flush
15111 277/300 Read block 3934 len 20480
15112 278/300 Write block 2595 len 20480 data: 5 6 7 8 7
15113 279/300 Read block 1005 len 32768
15114 280/300 Write block 2901 len 8192 data: 7 5
15115 281/300 Read block 4407 len 16384
15116 282/300 Read block 3560 len 16384
15117 283/300 Write block 2231 len 12288 data: 3 3 4
15118 284/300 Read block 786 len 4096
15119 285/300 Read block 188 len 8192
15120 286/300 Read block 2239 len 40960
15121 287/300 Read block 1973 len 12288
15122 288/300 Read block 393 len 40960
15123 289/300 Read block 2375 len 32768
15124 290/300 Read block 144 len 28672
15125 291/300 Read block 911 len 36864
15126 292/300 Read block 3333 len 32768
15127 293/300 Write block 2136 len 28672 data: 2 2 2 2 2 2 2
15128 294/300 Read block 2646 len 36864
15129 295/300 Write block 888 len 4096 data: 5
15130 296/300 Flush
15131 297/300 Write block 1119 len 12288 data: 4 4 4
15132 298/300 Write block 1234 len 24576 data: 2 2 2 2 2 2
15133 299/300 Read block 1016 len 28672
15134 300/300 Read block 4450 len 20480
15135 Main task has work to do, go find it
15136 got dsc Start(2)
15137 start 2
15138 Feb 12 01:37:31.819 INFO request completed, latency_us: 223, response_code: 204, uri: /start/cid/2, method: POST, req_id: 20dc1aba-67bd-4c19-8fe9-ac2eee97ad81, remote_addr: 127.0.0.1:60456, local_addr: 127.0.0.1:9998
15139 Replay: started 2, returned:Ok(())
15140 ----------------------------------------------------------------
15141 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
15142 States: Active Active Offline
15143 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
15144 4824 Acked 5823 Flush 0 Done Done New true
15145 4825 Acked 5824 Read 10 Done Done New true
15146 4826 Acked 5825 Write 7 Done Done New true
15147 4827 Acked 5826 Read 6 Done Done New true
15148 4828 Acked 5827 Write 9 Done Done New false
15149 4829 Acked 5828 Write 6 Done Done New false
15150 4830 Acked 5829 Write 9 Done Done New false
15151 4831 Acked 5830 Read 1 Done Done New false
15152 4832 Acked 5831 Read 7 Done Done New false
15153 4833 Acked 5832 Write 3 Done Done New false
15154 4834 Acked 5833 Read 2 Done Done New false
15155 4835 Acked 5834 Write 8 Done Done New false
15156 4836 Acked 5835 Read 3 Done Done New false
15157 4837 Acked 5836 Write 7 Done Done New false
15158 4838 Acked 5837 Read 7 Done Done New false
15159 4839 Acked 5838 Write 10 Done Done New false
15160 4840 Acked 5839 Read 3 Done Done New false
15161 4841 Acked 5840 Read 2 Done Done New false
15162 4842 Acked 5841 Read 4 Done Done New false
15163 4843 Acked 5842 Write 1 Done Done New false
15164 4844 Acked 5843 Read 3 Done Done New false
15165 4845 Acked 5844 Flush 0 Done Done New false
15166 4846 Acked 5845 Write 1 Done Done New false
15167 4847 Acked 5846 Read 10 Done Done New false
15168 4848 Acked 5847 Write 3 Done Done New false
15169 4849 Acked 5848 Flush 0 Done Done New false
15170 4850 Acked 5849 Read 1 Done Done New false
15171 4851 Acked 5850 Read 8 Done Done New false
15172 4852 Acked 5851 Read 8 Done Done New false
15173 4853 Acked 5852 Write 4 Done Done New false
15174 4854 Acked 5853 Read 10 Done Done New false
15175 4855 Acked 5854 Read 9 Done Done New false
15176 4856 Acked 5855 Read 8 Done Done New false
15177 4857 Acked 5856 Read 7 Done Done New false
15178 4858 Acked 5857 Flush 0 Done Done New false
15179 4859 Acked 5858 Write 3 Done Done New false
15180 4860 Acked 5859 Flush 0 Done Done New false
15181 4861 Acked 5860 Read 7 Done Done New false
15182 4862 Acked 5861 Read 7 Done Done New false
15183 4863 Acked 5862 Write 3 Done Done New false
15184 4864 Acked 5863 Read 8 Done Done New false
15185 4865 Acked 5864 Flush 0 Done Done New false
15186 4866 Acked 5865 Write 7 Done Done New false
15187 4867 Acked 5866 Write 10 Done Done New false
15188 4868 Acked 5867 Flush 0 Done Done New false
15189 4869 Acked 5868 Read 2 Done Done New false
15190 4870 Acked 5869 Read 3 Done Done New false
15191 4871 Acked 5870 Write 6 Done Done New false
15192 4872 Acked 5871 Read 9 Done Done New false
15193 4873 Acked 5872 Read 4 Done Done New false
15194 4874 Acked 5873 Write 1 Done Done New false
15195 4875 Acked 5874 Read 4 Done Done New false
15196 4876 Acked 5875 Read 5 Done Done New false
15197 4877 Acked 5876 Read 3 Done Done New false
15198 4878 Acked 5877 Write 5 Done Done New false
15199 4879 Acked 5878 Write 7 Done Done New false
15200 4880 Acked 5879 Read 9 Done Done New false
15201 4881 Acked 5880 Read 9 Done Done New false
15202 4882 Acked 5881 Read 10 Done Done New false
15203 4883 Acked 5882 Flush 0 Done Done New false
15204 4884 Acked 5883 Write 2 Done Done New false
15205 4885 Acked 5884 Read 4 Done Done New false
15206 4886 Acked 5885 Read 5 Done Done New false
15207 4887 Acked 5886 Read 2 Done Done New false
15208 4888 Acked 5887 Read 6 Done Done New false
15209 4889 Acked 5888 Flush 0 Done Done New false
15210 4890 Acked 5889 Flush 0 Done Done New false
15211 4891 Acked 5890 Read 6 Done Done New false
15212 4892 Acked 5891 Read 7 Done Done New false
15213 4893 Acked 5892 Write 8 Done Done New false
15214 4894 Acked 5893 Read 9 Done Done New false
15215 4895 Acked 5894 Read 10 Done Done New false
15216 4896 Acked 5895 Read 1 Done Done New false
15217 4897 Acked 5896 Read 6 Done Done New false
15218 4898 Acked 5897 Read 5 Done Done New false
15219 4899 Acked 5898 Write 1 Done Done New false
15220 4900 Acked 5899 Flush 0 Done Done New false
15221 4901 Acked 5900 Read 4 Done Done New false
15222 4902 Acked 5901 Read 3 Done Done New false
15223 4903 Acked 5902 Write 9 Done Done New false
15224 4904 Acked 5903 Flush 0 Done Done New false
15225 4905 Acked 5904 Write 5 Done Done New false
15226 4906 Acked 5905 Write 8 Done Done New false
15227 4907 Acked 5906 Write 7 Done Done New false
15228 4908 Acked 5907 Write 9 Done Done New false
15229 4909 Acked 5908 Flush 0 Done Done New false
15230 4910 Acked 5909 Write 3 Done Done New false
15231 4911 Acked 5910 Flush 0 Done Done New false
15232 4912 Acked 5911 Write 9 Done Done New false
15233 4913 Acked 5912 Read 5 Done Done New false
15234 4914 Acked 5913 Write 6 Done Done New false
15235 4915 Acked 5914 Write 6 Done Done New false
15236 4916 Acked 5915 Flush 0 Done Done New false
15237 4917 Acked 5916 Write 3 Done Done New false
15238 4918 Acked 5917 Write 10 Done Done New false
15239 4919 Acked 5918 Write 10 Done Done New false
15240 4920 Acked 5919 Read 8 Done Done New false
15241 4921 Acked 5920 Read 3 Done Done New false
15242 4922 Acked 5921 Write 7 Done Done New false
15243 4923 Acked 5922 Read 1 Done Done New false
15244 4924 Acked 5923 Write 5 Done Done New false
15245 4925 Acked 5924 Write 3 Done Done New false
15246 4926 Acked 5925 Read 5 Done Done New false
15247 4927 Acked 5926 Read 1 Done Done New false
15248 4928 Acked 5927 Write 7 Done Done New false
15249 4929 Acked 5928 Flush 0 Done Done New false
15250 4930 Acked 5929 Read 3 Done Done New false
15251 4931 Acked 5930 Write 10 Done Done New false
15252 4932 Acked 5931 Write 7 Done Done New false
15253 4933 Acked 5932 Write 9 Done Done New false
15254 4934 Acked 5933 Read 7 Done Done New false
15255 4935 Acked 5934 Read 3 Done Done New false
15256 4936 Acked 5935 Read 10 Done Done New false
15257 4937 Acked 5936 Read 6 Done Done New false
15258 4938 Acked 5937 Read 8 Done Done New false
15259 4939 Acked 5938 Flush 0 Done Done New false
15260 4940 Acked 5939 Read 6 Done Done New false
15261 4941 Acked 5940 Write 7 Done Done New false
15262 4942 Acked 5941 Read 1 Done Done New false
15263 4943 Acked 5942 Read 1 Done Done New false
15264 4944 Acked 5943 Flush 0 Done Done New false
15265 4945 Acked 5944 Read 4 Done Done New false
15266 4946 Acked 5945 Read 2 Done Done New false
15267 4947 Acked 5946 Write 1 Done Done New false
15268 4948 Acked 5947 Read 4 Done Done New false
15269 4949 Acked 5948 Read 2 Done Done New false
15270 4950 Acked 5949 Write 5 Done Done New false
15271 4951 Acked 5950 Read 1 Done Done New false
15272 4952 Acked 5951 Read 6 Done Done New false
15273 4953 Acked 5952 Read 4 Done Done New false
15274 4954 Acked 5953 Write 7 Done Done New false
15275 4955 Acked 5954 Write 10 Done Done New false
15276 4956 Acked 5955 Flush 0 Done Done New false
15277 4957 Acked 5956 Read 3 Done Done New false
15278 4958 Acked 5957 Read 3 Done Done New false
15279 4959 Acked 5958 Read 8 Done Done New false
15280 4960 Acked 5959 Write 4 Done Done New false
15281 4961 Acked 5960 Write 9 Done Done New false
15282 4962 Acked 5961 Read 4 Done Done New false
15283 4963 Acked 5962 Write 4 Done Done New false
15284 4964 Acked 5963 Read 6 Done Done New false
15285 4965 Acked 5964 Read 7 Done Done New false
15286 4966 Acked 5965 Read 3 Done Done New false
15287 4967 Acked 5966 Write 4 Done Done New false
15288 4968 Acked 5967 Read 9 Done Done New false
15289 4969 Acked 5968 Read 1 Done Done New false
15290 4970 Acked 5969 Write 1 Done Done New false
15291 4971 Acked 5970 Write 2 Done Done New false
15292 4972 Acked 5971 Write 5 Done Done New false
15293 4973 Acked 5972 Read 8 Done Done New false
15294 4974 Acked 5973 Write 10 Done Done New false
15295 4975 Acked 5974 Read 3 Done Done New false
15296 4976 Acked 5975 Write 3 Done Done New false
15297 4977 Acked 5976 Read 6 Done Done New false
15298 4978 Acked 5977 Write 3 Done Done New false
15299 4979 Acked 5978 Read 6 Done Done New false
15300 4980 Acked 5979 Write 1 Done Done New false
15301 4981 Acked 5980 Read 3 Done Done New false
15302 4982 Acked 5981 Read 6 Done Done New false
15303 4983 Acked 5982 Write 7 Done Done New false
15304 4984 Acked 5983 Write 9 Done Done New false
15305 4985 Acked 5984 Read 8 Done Done New false
15306 4986 Acked 5985 Write 1 Done Done New false
15307 4987 Acked 5986 Read 1 Done Done New false
15308 4988 Acked 5987 Write 9 Done Done New false
15309 4989 Acked 5988 Write 4 Done Done New false
15310 4990 Acked 5989 Read 3 Done Done New false
15311 4991 Acked 5990 Write 1 Done Done New false
15312 4992 Acked 5991 Write 3 Done Done New false
15313 4993 Acked 5992 Read 1 Done Done New false
15314 4994 Acked 5993 Write 10 Done Done New false
15315 4995 Acked 5994 Write 3 Done Done New false
15316 4996 Acked 5995 Flush 0 Done Done New false
15317 4997 Acked 5996 Write 8 Done Done New false
15318 4998 Acked 5997 Read 5 Done Done New false
15319 4999 Acked 5998 Read 10 Done Done New false
15320 5000 Acked 5999 Write 1 Done Done New false
15321 5001 Acked 6000 Write 5 Done Done New false
15322 5002 Acked 6001 Write 5 Done Done New false
15323 5003 Acked 6002 Read 1 Done Done New false
15324 5004 Acked 6003 Read 9 Done Done New false
15325 5005 Acked 6004 Write 9 Done Done New false
15326 5006 Acked 6005 Write 10 Done Done New false
15327 5007 Acked 6006 Read 10 Done Done New false
15328 5008 Acked 6007 Write 10 Done Done New false
15329 5009 Acked 6008 Read 6 Done Done New false
15330 5010 Acked 6009 Read 5 Done Done New false
15331 5011 Acked 6010 Read 4 Done Done New false
15332 5012 Acked 6011 Read 4 Done Done New false
15333 5013 Acked 6012 Read 4 Done Done New false
15334 5014 Acked 6013 Read 6 Done Done New false
15335 5015 Acked 6014 Write 4 Done Done New false
15336 5016 Acked 6015 Read 3 Done Done New false
15337 5017 Acked 6016 Read 6 Done Done New false
15338 5018 Acked 6017 Read 2 Done Done New false
15339 5019 Acked 6018 Write 3 Done Done New false
15340 5020 Acked 6019 Flush 0 Done Done New false
15341 5021 Acked 6020 Flush 0 Done Done New false
15342 5022 Acked 6021 Write 1 Done Done New false
15343 5023 Acked 6022 Read 2 Done Done New false
15344 5024 Acked 6023 Write 3 Done Done New false
15345 5025 Acked 6024 Read 7 Done Done New false
15346 5026 Acked 6025 Read 4 Done Done New false
15347 5027 Acked 6026 Read 3 Done Done New false
15348 5028 Acked 6027 Write 10 Done Done New false
15349 5029 Acked 6028 Read 10 Done Done New false
15350 5030 Acked 6029 Read 9 Done Done New false
15351 5031 Acked 6030 Read 10 Done Done New false
15352 5032 Acked 6031 Read 9 Done Done New false
15353 5033 Acked 6032 Read 3 Done Done New false
15354 5034 Acked 6033 Flush 0 Done Done New false
15355 5035 Acked 6034 Read 10 Done Done New false
15356 5036 Acked 6035 Flush 0 Done Done New false
15357 5037 Acked 6036 Read 10 Done Done New false
15358 5038 Acked 6037 Read 3 Done Done New false
15359 5039 Acked 6038 Read 8 Done Done New false
15360 5040 Acked 6039 Read 2 Done Done New false
15361 5041 Acked 6040 Write 1 Done Done New false
15362 5042 Acked 6041 Read 6 Done Done New false
15363 5043 Acked 6042 Read 1 Done Done New false
15364 5044 Acked 6043 Read 7 Done Done New false
15365 5045 Acked 6044 Flush 0 Done Done New false
15366 5046 Acked 6045 Write 10 Done Done New false
15367 5047 Acked 6046 Read 5 Done Done New false
15368 5048 Acked 6047 Read 4 Done Done New false
15369 5049 Acked 6048 Read 8 Done Done New false
15370 5050 Acked 6049 Read 6 Done Done New false
15371 5051 Acked 6050 Read 1 Done Done New false
15372 5052 Acked 6051 Read 10 Done Done New false
15373 5053 Acked 6052 Read 6 Done Done New false
15374 5054 Acked 6053 Read 5 Done Done New false
15375 5055 Acked 6054 Write 3 Done Done New false
15376 5056 Acked 6055 Read 7 Done Done New false
15377 5057 Acked 6056 Write 8 Done Done New false
15378 5058 Acked 6057 Read 2 Done Done New false
15379 5059 Acked 6058 Read 3 Done Done New false
15380 5060 Acked 6059 Flush 0 Done Done New false
15381 5061 Acked 6060 Read 3 Done Done New false
15382 5062 Acked 6061 Read 7 Done Done New false
15383 5063 Acked 6062 Write 9 Done Done New false
15384 5064 Acked 6063 Write 7 Done Done New false
15385 5065 Acked 6064 Read 7 Done Done New false
15386 5066 Acked 6065 Read 5 Done Done New false
15387 5067 Acked 6066 Read 9 Done Done New false
15388 5068 Acked 6067 Write 7 Done Done New false
15389 5069 Acked 6068 Write 5 Done Done New false
15390 5070 Acked 6069 Read 10 Done Done New false
15391 5071 Acked 6070 Read 2 Done Done New false
15392 5072 Acked 6071 Write 3 Done Done New false
15393 5073 Acked 6072 Read 5 Done Done New false
15394 5074 Acked 6073 Read 1 Done Done New false
15395 5075 Acked 6074 Flush 0 Done Done New false
15396 5076 Acked 6075 Write 1 Done Done New false
15397 5077 Acked 6076 Read 4 Done Done New false
15398 5078 Acked 6077 Read 1 Done Done New false
15399 5079 Acked 6078 Write 4 Done Done New false
15400 5080 Acked 6079 Write 9 Done Done New false
15401 5081 Acked 6080 Write 8 Done Done New false
15402 5082 Acked 6081 Read 1 Done Done New false
15403 5083 Acked 6082 Read 10 Done Done New false
15404 5084 Acked 6083 Write 8 Done Done New false
15405 5085 Acked 6084 Read 4 Done Done New false
15406 5086 Acked 6085 Read 5 Done Done New false
15407 5087 Acked 6086 Write 10 Done Done New false
15408 5088 Acked 6087 Read 3 Done Done New false
15409 5089 Acked 6088 Read 5 Done Done New false
15410 5090 Acked 6089 Write 4 Done Done New false
15411 5091 Acked 6090 Write 2 Done Done New false
15412 5092 Acked 6091 Read 3 Done Done New false
15413 5093 Acked 6092 Read 10 Done Done New false
15414 5094 Acked 6093 Write 9 Done Done New false
15415 5095 Acked 6094 Write 6 Done Done New false
15416 5096 Acked 6095 Read 10 Done Done New false
15417 5097 Acked 6096 Read 3 Done Done New false
15418 5098 Acked 6097 Write 6 Done Done New false
15419 5099 Acked 6098 Read 5 Done Done New false
15420 5100 Acked 6099 Flush 0 Done Done New false
15421 5101 Acked 6100 Read 5 Done Done New false
15422 5102 Acked 6101 Write 5 Done Done New false
15423 5103 Acked 6102 Read 8 Done Done New false
15424 5104 Acked 6103 Write 2 Done Done New false
15425 5105 Acked 6104 Read 4 Done Done New false
15426 5106 Acked 6105 Read 4 Done Done New false
15427 5107 Acked 6106 Write 3 Done Done New false
15428 5108 Acked 6107 Read 1 Done Done New false
15429 5109 Acked 6108 Read 2 Done Done New false
15430 5110 Acked 6109 Read 10 Done Done New false
15431 5111 Acked 6110 Read 3 Done Done New false
15432 5112 Acked 6111 Read 10 Done Done New false
15433 5113 Acked 6112 Read 8 Done Done New false
15434 5114 Acked 6113 Read 7 Done Done New false
15435 5115 Acked 6114 Read 9 Done Done New false
15436 5116 Acked 6115 Read 8 Done Done New false
15437 5117 Acked 6116 Write 7 Done Done New false
15438 5118 Acked 6117 Read 9 Done Done New false
15439 5119 Acked 6118 Write 1 Done Done New false
15440 5120 Acked 6119 Flush 0 Done Done New false
15441 5121 Acked 6120 Write 3 Done Done New false
15442 5122 Acked 6121 Write 6 Done Done New false
15443 5123 Acked 6122 Read 7 Done Done New false
15444 5124 Acked 6123 Read 5 Done Done New false
15445 STATES DS:0 DS:1 DS:2 TOTAL
15446 New 0 0 301 301
15447 Sent 0 0 0 0
15448 Done 301 301 0 602
15449 Skipped 0 0 0 0
15450 Error 0 0 0 0
15451 Last Flush: 6119 6119 5822
15452 Downstairs last five completed: 5822 5821 5820 5819 5818
15453 Upstairs last five completed: 5124 5123 5122 5121 5120
15454 CLIENT: Up:0 ds:301 act:2
154552024-02-12T01:37:32.468ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
154562024-02-12T01:37:32.468ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
154572024-02-12T01:37:32.468ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
154582024-02-12T01:37:32.468ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
154592024-02-12T01:37:32.468ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
154602024-02-12T01:37:32.468ZINFOcrucible: [2] client re-new 302 jobs since flush 5822 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
154612024-02-12T01:37:32.469ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
154622024-02-12T01:37:33.469ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
154632024-02-12T01:37:33.469ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
154642024-02-12T01:37:33.469ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
154652024-02-12T01:37:33.469ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
154662024-02-12T01:37:33.469ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
154672024-02-12T01:37:33.469ZINFOcrucible: [2] client re-new 302 jobs since flush 5822 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
154682024-02-12T01:37:33.470ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
154692024-02-12T01:37:34.470ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
154702024-02-12T01:37:34.470ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
154712024-02-12T01:37:34.470ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
154722024-02-12T01:37:34.470ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
154732024-02-12T01:37:34.471ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
154742024-02-12T01:37:34.471ZINFOcrucible: [2] client re-new 302 jobs since flush 5822 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
154752024-02-12T01:37:34.471ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
15476 [8830] got start action
15477 [8830] I am going to restart
15478 Starting downstairs at port 8830
15479 Make output file at "/tmp/dsc/downstairs-8830.txt"
15480 [8830][2] reports Starting
15481 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1181)
15482 [8830][2] reports Running
154832024-02-12T01:37:35.473ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
154842024-02-12T01:37:35.473ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
154852024-02-12T01:37:35.473ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
154862024-02-12T01:37:35.473ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
154872024-02-12T01:37:35.473ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
154882024-02-12T01:37:35.473ZINFOcrucible: [2] client re-new 302 jobs since flush 5822 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
154892024-02-12T01:37:35.473ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
15490 ----------------------------------------------------------------
15491 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
15492 States: Active Active Offline
15493 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
15494 4824 Acked 5823 Flush 0 Done Done New true
15495 4825 Acked 5824 Read 10 Done Done New true
15496 4826 Acked 5825 Write 7 Done Done New true
15497 4827 Acked 5826 Read 6 Done Done New true
15498 4828 Acked 5827 Write 9 Done Done New true
15499 4829 Acked 5828 Write 6 Done Done New true
15500 4830 Acked 5829 Write 9 Done Done New true
15501 4831 Acked 5830 Read 1 Done Done New true
15502 4832 Acked 5831 Read 7 Done Done New true
15503 4833 Acked 5832 Write 3 Done Done New true
15504 4834 Acked 5833 Read 2 Done Done New true
15505 4835 Acked 5834 Write 8 Done Done New true
15506 4836 Acked 5835 Read 3 Done Done New true
15507 4837 Acked 5836 Write 7 Done Done New true
15508 4838 Acked 5837 Read 7 Done Done New true
15509 4839 Acked 5838 Write 10 Done Done New true
15510 4840 Acked 5839 Read 3 Done Done New true
15511 4841 Acked 5840 Read 2 Done Done New true
15512 4842 Acked 5841 Read 4 Done Done New true
15513 4843 Acked 5842 Write 1 Done Done New true
15514 4844 Acked 5843 Read 3 Done Done New true
15515 4845 Acked 5844 Flush 0 Done Done New true
15516 4846 Acked 5845 Write 1 Done Done New true
15517 4847 Acked 5846 Read 10 Done Done New true
15518 4848 Acked 5847 Write 3 Done Done New true
15519 4849 Acked 5848 Flush 0 Done Done New true
15520 4850 Acked 5849 Read 1 Done Done New true
15521 4851 Acked 5850 Read 8 Done Done New true
15522 4852 Acked 5851 Read 8 Done Done New true
15523 4853 Acked 5852 Write 4 Done Done New true
15524 4854 Acked 5853 Read 10 Done Done New true
15525 4855 Acked 5854 Read 9 Done Done New true
15526 4856 Acked 5855 Read 8 Done Done New true
15527 4857 Acked 5856 Read 7 Done Done New true
15528 4858 Acked 5857 Flush 0 Done Done New true
15529 4859 Acked 5858 Write 3 Done Done New true
15530 4860 Acked 5859 Flush 0 Done Done New true
15531 4861 Acked 5860 Read 7 Done Done New true
15532 4862 Acked 5861 Read 7 Done Done New true
15533 4863 Acked 5862 Write 3 Done Done New true
15534 4864 Acked 5863 Read 8 Done Done New true
15535 4865 Acked 5864 Flush 0 Done Done New true
15536 4866 Acked 5865 Write 7 Done Done New true
15537 4867 Acked 5866 Write 10 Done Done New true
15538 4868 Acked 5867 Flush 0 Done Done New true
15539 4869 Acked 5868 Read 2 Done Done New true
15540 4870 Acked 5869 Read 3 Done Done New true
15541 4871 Acked 5870 Write 6 Done Done New true
15542 4872 Acked 5871 Read 9 Done Done New true
15543 4873 Acked 5872 Read 4 Done Done New true
15544 4874 Acked 5873 Write 1 Done Done New true
15545 4875 Acked 5874 Read 4 Done Done New true
15546 4876 Acked 5875 Read 5 Done Done New true
15547 4877 Acked 5876 Read 3 Done Done New true
15548 4878 Acked 5877 Write 5 Done Done New true
15549 4879 Acked 5878 Write 7 Done Done New true
15550 4880 Acked 5879 Read 9 Done Done New true
15551 4881 Acked 5880 Read 9 Done Done New true
15552 4882 Acked 5881 Read 10 Done Done New true
15553 4883 Acked 5882 Flush 0 Done Done New true
15554 4884 Acked 5883 Write 2 Done Done New true
15555 4885 Acked 5884 Read 4 Done Done New true
15556 4886 Acked 5885 Read 5 Done Done New true
15557 4887 Acked 5886 Read 2 Done Done New true
15558 4888 Acked 5887 Read 6 Done Done New true
15559 4889 Acked 5888 Flush 0 Done Done New true
15560 4890 Acked 5889 Flush 0 Done Done New true
15561 4891 Acked 5890 Read 6 Done Done New true
15562 4892 Acked 5891 Read 7 Done Done New true
15563 4893 Acked 5892 Write 8 Done Done New true
15564 4894 Acked 5893 Read 9 Done Done New true
15565 4895 Acked 5894 Read 10 Done Done New true
15566 4896 Acked 5895 Read 1 Done Done New true
15567 4897 Acked 5896 Read 6 Done Done New true
15568 4898 Acked 5897 Read 5 Done Done New true
15569 4899 Acked 5898 Write 1 Done Done New true
15570 4900 Acked 5899 Flush 0 Done Done New true
15571 4901 Acked 5900 Read 4 Done Done New true
15572 4902 Acked 5901 Read 3 Done Done New true
15573 4903 Acked 5902 Write 9 Done Done New true
15574 4904 Acked 5903 Flush 0 Done Done New true
15575 4905 Acked 5904 Write 5 Done Done New true
15576 4906 Acked 5905 Write 8 Done Done New true
15577 4907 Acked 5906 Write 7 Done Done New true
15578 4908 Acked 5907 Write 9 Done Done New true
15579 4909 Acked 5908 Flush 0 Done Done New true
15580 4910 Acked 5909 Write 3 Done Done New true
15581 4911 Acked 5910 Flush 0 Done Done New true
15582 4912 Acked 5911 Write 9 Done Done New true
15583 4913 Acked 5912 Read 5 Done Done New true
15584 4914 Acked 5913 Write 6 Done Done New true
15585 4915 Acked 5914 Write 6 Done Done New true
15586 4916 Acked 5915 Flush 0 Done Done New true
15587 4917 Acked 5916 Write 3 Done Done New true
15588 4918 Acked 5917 Write 10 Done Done New true
15589 4919 Acked 5918 Write 10 Done Done New true
15590 4920 Acked 5919 Read 8 Done Done New true
15591 4921 Acked 5920 Read 3 Done Done New true
15592 4922 Acked 5921 Write 7 Done Done New true
15593 4923 Acked 5922 Read 1 Done Done New true
15594 4924 Acked 5923 Write 5 Done Done New true
15595 4925 Acked 5924 Write 3 Done Done New true
15596 4926 Acked 5925 Read 5 Done Done New true
15597 4927 Acked 5926 Read 1 Done Done New true
15598 4928 Acked 5927 Write 7 Done Done New true
15599 4929 Acked 5928 Flush 0 Done Done New true
15600 4930 Acked 5929 Read 3 Done Done New true
15601 4931 Acked 5930 Write 10 Done Done New true
15602 4932 Acked 5931 Write 7 Done Done New true
15603 4933 Acked 5932 Write 9 Done Done New true
15604 4934 Acked 5933 Read 7 Done Done New true
15605 4935 Acked 5934 Read 3 Done Done New true
15606 4936 Acked 5935 Read 10 Done Done New true
15607 4937 Acked 5936 Read 6 Done Done New true
15608 4938 Acked 5937 Read 8 Done Done New true
15609 4939 Acked 5938 Flush 0 Done Done New true
15610 4940 Acked 5939 Read 6 Done Done New true
15611 4941 Acked 5940 Write 7 Done Done New true
15612 4942 Acked 5941 Read 1 Done Done New true
15613 4943 Acked 5942 Read 1 Done Done New true
15614 4944 Acked 5943 Flush 0 Done Done New true
15615 4945 Acked 5944 Read 4 Done Done New true
15616 4946 Acked 5945 Read 2 Done Done New true
15617 4947 Acked 5946 Write 1 Done Done New true
15618 4948 Acked 5947 Read 4 Done Done New true
15619 4949 Acked 5948 Read 2 Done Done New true
15620 4950 Acked 5949 Write 5 Done Done New true
15621 4951 Acked 5950 Read 1 Done Done New true
15622 4952 Acked 5951 Read 6 Done Done New true
15623 4953 Acked 5952 Read 4 Done Done New true
15624 4954 Acked 5953 Write 7 Done Done New true
15625 4955 Acked 5954 Write 10 Done Done New true
15626 4956 Acked 5955 Flush 0 Done Done New true
15627 4957 Acked 5956 Read 3 Done Done New true
15628 4958 Acked 5957 Read 3 Done Done New true
15629 4959 Acked 5958 Read 8 Done Done New true
15630 4960 Acked 5959 Write 4 Done Done New true
15631 4961 Acked 5960 Write 9 Done Done New true
15632 4962 Acked 5961 Read 4 Done Done New true
15633 4963 Acked 5962 Write 4 Done Done New true
15634 4964 Acked 5963 Read 6 Done Done New true
15635 4965 Acked 5964 Read 7 Done Done New true
15636 4966 Acked 5965 Read 3 Done Done New true
15637 4967 Acked 5966 Write 4 Done Done New true
15638 4968 Acked 5967 Read 9 Done Done New true
15639 4969 Acked 5968 Read 1 Done Done New true
15640 4970 Acked 5969 Write 1 Done Done New true
15641 4971 Acked 5970 Write 2 Done Done New true
15642 4972 Acked 5971 Write 5 Done Done New true
15643 4973 Acked 5972 Read 8 Done Done New true
15644 4974 Acked 5973 Write 10 Done Done New true
15645 4975 Acked 5974 Read 3 Done Done New true
15646 4976 Acked 5975 Write 3 Done Done New true
15647 4977 Acked 5976 Read 6 Done Done New true
15648 4978 Acked 5977 Write 3 Done Done New true
15649 4979 Acked 5978 Read 6 Done Done New true
15650 4980 Acked 5979 Write 1 Done Done New true
15651 4981 Acked 5980 Read 3 Done Done New true
15652 4982 Acked 5981 Read 6 Done Done New true
15653 4983 Acked 5982 Write 7 Done Done New true
15654 4984 Acked 5983 Write 9 Done Done New true
15655 4985 Acked 5984 Read 8 Done Done New true
15656 4986 Acked 5985 Write 1 Done Done New true
15657 4987 Acked 5986 Read 1 Done Done New true
15658 4988 Acked 5987 Write 9 Done Done New true
15659 4989 Acked 5988 Write 4 Done Done New true
15660 4990 Acked 5989 Read 3 Done Done New true
15661 4991 Acked 5990 Write 1 Done Done New true
15662 4992 Acked 5991 Write 3 Done Done New true
15663 4993 Acked 5992 Read 1 Done Done New true
15664 4994 Acked 5993 Write 10 Done Done New true
15665 4995 Acked 5994 Write 3 Done Done New true
15666 4996 Acked 5995 Flush 0 Done Done New true
15667 4997 Acked 5996 Write 8 Done Done New true
15668 4998 Acked 5997 Read 5 Done Done New true
15669 4999 Acked 5998 Read 10 Done Done New true
15670 5000 Acked 5999 Write 1 Done Done New true
15671 5001 Acked 6000 Write 5 Done Done New true
15672 5002 Acked 6001 Write 5 Done Done New true
15673 5003 Acked 6002 Read 1 Done Done New true
15674 5004 Acked 6003 Read 9 Done Done New true
15675 5005 Acked 6004 Write 9 Done Done New true
15676 5006 Acked 6005 Write 10 Done Done New true
15677 5007 Acked 6006 Read 10 Done Done New true
15678 5008 Acked 6007 Write 10 Done Done New true
15679 5009 Acked 6008 Read 6 Done Done New true
15680 5010 Acked 6009 Read 5 Done Done New true
15681 5011 Acked 6010 Read 4 Done Done New true
15682 5012 Acked 6011 Read 4 Done Done New true
15683 5013 Acked 6012 Read 4 Done Done New true
15684 5014 Acked 6013 Read 6 Done Done New true
15685 5015 Acked 6014 Write 4 Done Done New true
15686 5016 Acked 6015 Read 3 Done Done New true
15687 5017 Acked 6016 Read 6 Done Done New true
15688 5018 Acked 6017 Read 2 Done Done New true
15689 5019 Acked 6018 Write 3 Done Done New true
15690 5020 Acked 6019 Flush 0 Done Done New true
15691 5021 Acked 6020 Flush 0 Done Done New true
15692 5022 Acked 6021 Write 1 Done Done New true
15693 5023 Acked 6022 Read 2 Done Done New true
15694 5024 Acked 6023 Write 3 Done Done New true
15695 5025 Acked 6024 Read 7 Done Done New true
15696 5026 Acked 6025 Read 4 Done Done New true
15697 5027 Acked 6026 Read 3 Done Done New true
15698 5028 Acked 6027 Write 10 Done Done New true
15699 5029 Acked 6028 Read 10 Done Done New true
15700 5030 Acked 6029 Read 9 Done Done New true
15701 5031 Acked 6030 Read 10 Done Done New true
15702 5032 Acked 6031 Read 9 Done Done New true
15703 5033 Acked 6032 Read 3 Done Done New true
15704 5034 Acked 6033 Flush 0 Done Done New true
15705 5035 Acked 6034 Read 10 Done Done New true
15706 5036 Acked 6035 Flush 0 Done Done New true
15707 5037 Acked 6036 Read 10 Done Done New true
15708 5038 Acked 6037 Read 3 Done Done New true
15709 5039 Acked 6038 Read 8 Done Done New true
15710 5040 Acked 6039 Read 2 Done Done New true
15711 5041 Acked 6040 Write 1 Done Done New true
15712 5042 Acked 6041 Read 6 Done Done New true
15713 5043 Acked 6042 Read 1 Done Done New true
15714 5044 Acked 6043 Read 7 Done Done New true
15715 5045 Acked 6044 Flush 0 Done Done New true
15716 5046 Acked 6045 Write 10 Done Done New true
15717 5047 Acked 6046 Read 5 Done Done New true
15718 5048 Acked 6047 Read 4 Done Done New true
15719 5049 Acked 6048 Read 8 Done Done New true
15720 5050 Acked 6049 Read 6 Done Done New true
15721 5051 Acked 6050 Read 1 Done Done New true
15722 5052 Acked 6051 Read 10 Done Done New true
15723 5053 Acked 6052 Read 6 Done Done New true
15724 5054 Acked 6053 Read 5 Done Done New true
15725 5055 Acked 6054 Write 3 Done Done New true
15726 5056 Acked 6055 Read 7 Done Done New true
15727 5057 Acked 6056 Write 8 Done Done New true
15728 5058 Acked 6057 Read 2 Done Done New true
15729 5059 Acked 6058 Read 3 Done Done New true
15730 5060 Acked 6059 Flush 0 Done Done New true
15731 5061 Acked 6060 Read 3 Done Done New true
15732 5062 Acked 6061 Read 7 Done Done New true
15733 5063 Acked 6062 Write 9 Done Done New true
15734 5064 Acked 6063 Write 7 Done Done New true
15735 5065 Acked 6064 Read 7 Done Done New true
15736 5066 Acked 6065 Read 5 Done Done New true
15737 5067 Acked 6066 Read 9 Done Done New true
15738 5068 Acked 6067 Write 7 Done Done New true
15739 5069 Acked 6068 Write 5 Done Done New true
15740 5070 Acked 6069 Read 10 Done Done New true
15741 5071 Acked 6070 Read 2 Done Done New true
15742 5072 Acked 6071 Write 3 Done Done New true
15743 5073 Acked 6072 Read 5 Done Done New true
15744 5074 Acked 6073 Read 1 Done Done New true
15745 5075 Acked 6074 Flush 0 Done Done New true
15746 5076 Acked 6075 Write 1 Done Done New true
15747 5077 Acked 6076 Read 4 Done Done New true
15748 5078 Acked 6077 Read 1 Done Done New true
15749 5079 Acked 6078 Write 4 Done Done New true
15750 5080 Acked 6079 Write 9 Done Done New true
15751 5081 Acked 6080 Write 8 Done Done New true
15752 5082 Acked 6081 Read 1 Done Done New true
15753 5083 Acked 6082 Read 10 Done Done New true
15754 5084 Acked 6083 Write 8 Done Done New true
15755 5085 Acked 6084 Read 4 Done Done New true
15756 5086 Acked 6085 Read 5 Done Done New true
15757 5087 Acked 6086 Write 10 Done Done New true
15758 5088 Acked 6087 Read 3 Done Done New true
15759 5089 Acked 6088 Read 5 Done Done New true
15760 5090 Acked 6089 Write 4 Done Done New true
15761 5091 Acked 6090 Write 2 Done Done New true
15762 5092 Acked 6091 Read 3 Done Done New true
15763 5093 Acked 6092 Read 10 Done Done New true
15764 5094 Acked 6093 Write 9 Done Done New true
15765 5095 Acked 6094 Write 6 Done Done New true
15766 5096 Acked 6095 Read 10 Done Done New true
15767 5097 Acked 6096 Read 3 Done Done New true
15768 5098 Acked 6097 Write 6 Done Done New true
15769 5099 Acked 6098 Read 5 Done Done New true
15770 5100 Acked 6099 Flush 0 Done Done New true
15771 5101 Acked 6100 Read 5 Done Done New true
15772 5102 Acked 6101 Write 5 Done Done New true
15773 5103 Acked 6102 Read 8 Done Done New true
15774 5104 Acked 6103 Write 2 Done Done New true
15775 5105 Acked 6104 Read 4 Done Done New true
15776 5106 Acked 6105 Read 4 Done Done New true
15777 5107 Acked 6106 Write 3 Done Done New true
15778 5108 Acked 6107 Read 1 Done Done New true
15779 5109 Acked 6108 Read 2 Done Done New true
15780 5110 Acked 6109 Read 10 Done Done New true
15781 5111 Acked 6110 Read 3 Done Done New true
15782 5112 Acked 6111 Read 10 Done Done New true
15783 5113 Acked 6112 Read 8 Done Done New true
15784 5114 Acked 6113 Read 7 Done Done New true
15785 5115 Acked 6114 Read 9 Done Done New true
15786 5116 Acked 6115 Read 8 Done Done New true
15787 5117 Acked 6116 Write 7 Done Done New true
15788 5118 Acked 6117 Read 9 Done Done New true
15789 5119 Acked 6118 Write 1 Done Done New true
15790 5120 Acked 6119 Flush 0 Done Done New true
15791 5121 Acked 6120 Write 3 Done Done New true
15792 5122 Acked 6121 Write 6 Done Done New true
15793 5123 Acked 6122 Read 7 Done Done New true
15794 5124 Acked 6123 Read 5 Done Done New true
15795 5125 Acked 6124 Flush 0 Done Done New true
15796 STATES DS:0 DS:1 DS:2 TOTAL
15797 New 0 0 302 302
15798 Sent 0 0 0 0
15799 Done 302 302 0 604
15800 Skipped 0 0 0 0
15801 Error 0 0 0 0
15802 Last Flush: 6124 6124 5822
15803 Downstairs last five completed: 5822 5821 5820 5819 5818
15804 Upstairs last five completed: 5125 5124 5123 5122 5121
15805 CLIENT: Up:0 ds:302 act:2
158062024-02-12T01:37:36.475ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
158072024-02-12T01:37:36.475ZINFOcrucible: ds_connection connected = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
158082024-02-12T01:37:36.476ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
158092024-02-12T01:37:36.476ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
158102024-02-12T01:37:36.476ZINFOcrucible: send last flush ID to this DS: 5822 = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
158112024-02-12T01:37:36.476ZINFOcrucible: Replied this last flush ID: 5822 = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
158122024-02-12T01:37:36.477ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
158132024-02-12T01:37:36.477ZINFOcrucible: [2] Transition from Offline to Replay = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
158142024-02-12T01:37:36.477ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
158152024-02-12T01:37:36.477ZINFOcrucible: [2] Transition from Replay to Active = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
15816 ----------------------------------------------------------------
15817 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
15818 Downstairs last five completed: 6124 6123 6122 6121 6120
15819 Upstairs last five completed: 5125 5124 5123 5122 5121
15820 CLIENT: Up:0 ds:0 act:3
15821 Replay: All jobs finished, all DS active.
15822 Main task has work to do, go find it
15823 got dsc Stop(2)
15824 stop 2
15825 Feb 12 01:37:39.825 INFO request completed, latency_us: 398, response_code: 204, uri: /stop/cid/2, method: POST, req_id[8830] Got stop action so:false kr:false
15826 : fcb32b60-b352-41a2-9c56-8cb71f9b09dc, remote_addr: 127.0.0.1:60456, local_addr: 127.0.0.1:9998
158272024-02-12T01:37:39.825ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
158282024-02-12T01:37:39.825ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
158292024-02-12T01:37:39.825ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
158302024-02-12T01:37:39.825ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
158312024-02-12T01:37:39.825ZINFOcrucible: [2] client re-new 0 jobs since flush 6124 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
15832 Feb 12 01:37:39.825 INFO request completed, latency_us: 135, response_code: 200, uri: /state/cid/2, method: GET, req_id: eff0eaaf-088c-4698-98ca-7c7dae4b838c, remote_addr: 127.0.0.1:60456, local_addr: 127.0.0.1:9998
158332024-02-12T01:37:39.826ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
15834 [8830] Exited with: ExitStatus(unix_wait_status(9))
15835 [8830][2] reports Exit
158362024-02-12T01:37:40.827ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
158372024-02-12T01:37:40.827ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
158382024-02-12T01:37:40.827ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
158392024-02-12T01:37:40.827ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
158402024-02-12T01:37:40.827ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
158412024-02-12T01:37:40.827ZINFOcrucible: [2] client re-new 0 jobs since flush 6124 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
158422024-02-12T01:37:40.827ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
158432024-02-12T01:37:41.828ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
158442024-02-12T01:37:41.828ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
158452024-02-12T01:37:41.828ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
158462024-02-12T01:37:41.828ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
158472024-02-12T01:37:41.828ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
158482024-02-12T01:37:41.828ZINFOcrucible: [2] client re-new 0 jobs since flush 6124 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
158492024-02-12T01:37:41.828ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
158502024-02-12T01:37:42.829ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
158512024-02-12T01:37:42.829ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
158522024-02-12T01:37:42.829ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
158532024-02-12T01:37:42.829ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
158542024-02-12T01:37:42.830ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
158552024-02-12T01:37:42.830ZINFOcrucible: [2] client re-new 0 jobs since flush 6124 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
158562024-02-12T01:37:42.830ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
15857 Feb 12 01:37:43.828 INFO request completed, latency_us: 249, response_code: 200, uri: /state/cid/2, method: GET, req_id: 125c7103-00e4-420b-9f72-47d5aa0b319e, remote_addr: 127.0.0.1:60456, local_addr: 127.0.0.1:9998
15858 001/300 Write block 933 len 24576 data: 7 5 6 6 7 6
15859 002/300 Read block 3541 len 8192
158602024-02-12T01:37:43.830ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
158612024-02-12T01:37:43.831ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
158622024-02-12T01:37:43.831ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
158632024-02-12T01:37:43.831ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
158642024-02-12T01:37:43.831ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
158652024-02-12T01:37:43.831ZINFOcrucible: [2] client re-new 2 jobs since flush 6124 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
158662024-02-12T01:37:43.831ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
15867 003/300 Read block 3299 len 8192
15868 004/300 Read block 4384 len 36864
15869 005/300 Read block 1906 len 32768
15870 006/300 Write block 440 len 28672 data: 4 4 4 4 5 5 6
15871 007/300 Write block 3772 len 40960 data: 3 3 5 6 7 6 4 5 5 4
15872 008/300 Write block 3080 len 32768 data: 6 4 4 4 2 2 2 2
15873 009/300 Read block 547 len 24576
15874 010/300 Write block 3076 len 20480 data: 3 3 4 5 7
15875 011/300 Write block 1070 len 12288 data: 5 6 4
15876 012/300 Read block 4588 len 32768
15877 013/300 Write block 2686 len 32768 data: 4 4 4 4 3 3 3 3
15878 014/300 Read block 4595 len 16384
15879 015/300 Read block 3904 len 12288
15880 016/300 Read block 253 len 24576
15881 017/300 Read block 2181 len 28672
15882 018/300 Write block 657 len 24576 data: 3 4 5 5 4 5
15883 019/300 Read block 3658 len 16384
15884 020/300 Write block 3541 len 36864 data: 2 2 2 2 2 3 3 3 3
15885 021/300 Read block 915 len 28672
15886 022/300 Read block 34 len 4096
15887 023/300 Read block 3705 len 20480
15888 024/300 Write block 3073 len 12288 data: 2 2 2
15889 025/300 Read block 3638 len 28672
15890 026/300 Write block 2776 len 16384 data: 3 3 3 4
15891 027/300 Read block 1616 len 8192
15892 028/300 Read block 655 len 40960
15893 029/300 Write block 3819 len 16384 data: 3 2 2 2
15894 030/300 Write block 347 len 32768 data: 3 4 5 4 5 5 5 5
15895 031/300 Write block 141 len 16384 data: 3 3 2 3
15896 032/300 Write block 4536 len 20480 data: 4 5 6 6 5
15897 033/300 Write block 1135 len 8192 data: 4 5
15898 034/300 Write block 2131 len 16384 data: 8 6 6 3
15899 035/300 Read block 4116 len 24576
15900 036/300 Write block 4547 len 12288 data: 4 5 7
15901 037/300 Write block 866 len 36864 data: 3 3 3 2 4 5 4 4 5
15902 038/300 Write block 3490 len 16384 data: 4 3 3 3
15903 039/300 Read block 3050 len 16384
15904 040/300 Write block 255 len 4096 data: 5
15905 041/300 Read block 3330 len 28672
15906 042/300 Read block 1270 len 4096
15907 043/300 Read block 847 len 12288
15908 044/300 Write block 579 len 36864 data: 2 2 2 2 2 2 3 3 3
15909 045/300 Read block 164 len 4096
15910 046/300 Write block 4514 len 20480 data: 3 2 2 2 2
15911 047/300 Write block 194 len 16384 data: 4 4 5 6
15912 048/300 Read block 2479 len 8192
15913 049/300 Read block 799 len 4096
15914 050/300 Write block 741 len 24576 data: 4 5 5 5 7 6
15915 051/300 Read block 2453 len 32768
15916 052/300 Write block 2329 len 24576 data: 2 2 2 3 4 4
15917 053/300 Write block 1592 len 4096 data: 6
15918 054/300 Flush
15919 055/300 Read block 1708 len 12288
15920 056/300 Write block 4319 len 32768 data: 7 5 5 3 3 3 5 5
15921 057/300 Write block 4027 len 28672 data: 5 4 5 4 3 2 4
15922 058/300 Read block 429 len 28672
15923 059/300 Write block 3246 len 8192 data: 3 3
15924 060/300 Read block 4579 len 4096
15925 061/300 Write block 664 len 4096 data: 5
15926 062/300 Write block 2169 len 36864 data: 5 7 9 9 7 7 10 8 8
15927 063/300 Read block 2647 len 12288
15928 064/300 Write block 1567 len 24576 data: 9 9 8 7 6 5
15929 065/300 Read block 3528 len 4096
15930 066/300 Write block 3673 len 12288 data: 5 5 5
15931 067/300 Read block 2055 len 16384
15932 068/300 Read block 1473 len 8192
15933 069/300 Write block 3575 len 16384 data: 6 5 4 4
15934 070/300 Read block 42 len 32768
15935 071/300 Read block 1127 len 20480
15936 072/300 Write block 1407 len 8192 data: 7 7
15937 073/300 Write block 3653 len 16384 data: 2 4 4 4
15938 074/300 Read block 4615 len 28672
15939 075/300 Read block 521 len 12288
15940 076/300 Read block 4711 len 40960
15941 077/300 Flush
15942 078/300 Write block 2792 len 36864 data: 4 3 3 4 5 5 4 5 4
15943 079/300 Write block 390 len 32768 data: 6 6 7 5 5 6 6 6
15944 080/300 Read block 1450 len 36864
15945 081/300 Read block 2039 len 16384
15946 082/300 Write block 1431 len 12288 data: 4 4 4
15947 083/300 Write block 2211 len 28672 data: 6 5 7 6 7 7 7
15948 084/300 Write block 2923 len 36864 data: 6 8 6 5 5 5 6 7 5
15949 085/300 Write block 1077 len 8192 data: 4 6
15950 086/300 Read block 2541 len 20480
15951 087/300 Read block 1764 len 16384
15952 088/300 Read block 4318 len 4096
15953 089/300 Read block 2987 len 16384
15954 090/300 Read block 3417 len 32768
15955 091/300 Flush
15956 092/300 Flush
15957 093/300 Write block 570 len 16384 data: 2 3 3 3
15958 094/300 Read block 67 len 4096
15959 095/300 Write block 2544 len 12288 data: 4 5 4
15960 096/300 Write block 2688 len 16384 data: 5 5 4 4
15961 097/300 Read block 2812 len 36864
15962 098/300 Read block 1398 len 40960
15963 099/300 Write block 784 len 16384 data: 6 7 7 6
15964 100/300 Write block 4605 len 20480 data: 5 4 5 4 4
15965 101/300 Flush
15966 102/300 Read block 4787 len 12288
15967 103/300 Write block 951 len 24576 data: 6 6 6 5 3 5
15968 104/300 Read block 509 len 32768
15969 105/300 Read block 1628 len 24576
15970 106/300 Write block 4401 len 28672 data: 3 3 3 4 4 4 4
15971 107/300 Write block 919 len 32768 data: 4 4 3 3 4 4 5 4
15972 108/300 Read block 1271 len 8192
15973 109/300 Flush
15974 110/300 Write block 2588 len 4096 data: 8
15975 111/300 Read block 1988 len 36864
15976 112/300 Read block 3289 len 36864
15977 113/300 Read block 769 len 36864
15978 114/300 Read block 1926 len 20480
15979 115/300 Write block 4245 len 20480 data: 3 4 4 4 5
15980 116/300 Flush
15981 117/300 Flush
15982 118/300 Read block 4596 len 36864
15983 119/300 Read block 4459 len 28672
15984 120/300 Write block 3177 len 24576 data: 4 4 4 6 7 5
15985 121/300 Read block 208 len 12288
15986 122/300 Read block 4866 len 4096
15987 123/300 Read block 3690 len 16384
15988 124/300 Read block 3918 len 20480
15989 125/300 Flush
15990 126/300 Write block 2973 len 16384 data: 4 5 4 3
15991 127/300 Read block 3077 len 8192
15992 128/300 Write block 1479 len 24576 data: 5 4 5 5 5 3
15993 129/300 Write block 4448 len 4096 data: 5
15994 130/300 Flush
15995 131/300 Flush
15996 132/300 Read block 1955 len 20480
15997 133/300 Write block 3373 len 12288 data: 4 6 7
15998 134/300 Write block 4765 len 24576 data: 4 4 4 4 4 3
15999 135/300 Flush
16000 136/300 Flush
16001 137/300 Read block 2448 len 4096
16002 138/300 Write block 2489 len 4096 data: 6
16003 139/300 Write block 325 len 24576 data: 4 3 3 2 3 3
16004 140/300 Read block 4826 len 28672
16005 141/300 Read block 3827 len 4096
16006 142/300 Write block 1279 len 4096 data: 3
16007 143/300 Write block 3877 len 28672 data: 6 7 6 5 4 5 5
16008 144/300 Read block 1357 len 32768
16009 145/300 Read block 3158 len 36864
16010 146/300 Read block 1730 len 36864
16011 147/300 Write block 3268 len 36864 data: 3 4 4 4 4 3 3 3 2
16012 148/300 Write block 3147 len 36864 data: 4 4 5 4 3 3 5 4 4
16013 149/300 Write block 2260 len 36864 data: 4 4 4 4 3 3 3 4 4
16014 150/300 Read block 4586 len 28672
16015 151/300 Write block 492 len 36864 data: 4 4 5 5 5 4 4 4 6
16016 152/300 Write block 3413 len 8192 data: 3 4
16017 153/300 Flush
16018 154/300 Flush
16019 155/300 Write block 564 len 28672 data: 3 4 3 2 2 2 3
16020 156/300 Write block 3466 len 28672 data: 2 2 3 5 7 7 6
16021 157/300 Flush
16022 158/300 Write block 1576 len 4096 data: 2
16023 159/300 Read block 2724 len 20480
16024 160/300 Write block 1221 len 40960 data: 4 3 4 4 4 3 3 2 2 2
16025 161/300 Flush
16026 162/300 Write block 1746 len 24576 data: 2 2 2 2 3 3
16027 163/300 Read block 1265 len 40960
16028 164/300 Read block 545 len 4096
16029 165/300 Read block 3952 len 32768
16030 166/300 Read block 1320 len 36864
16031 167/300 Write block 243 len 12288 data: 5 6 5
16032 168/300 Write block 3548 len 16384 data: 4 4 2 3
16033 169/300 Read block 3672 len 12288
16034 170/300 Write block 2778 len 40960 data: 4 5 4 4 4 4 3 4 5 5
16035 171/300 Read block 3858 len 16384
16036 172/300 Write block 395 len 4096 data: 7
16037 173/300 Read block 171 len 28672
16038 174/300 Write block 704 len 12288 data: 5 5 5
16039 175/300 Write block 2823 len 16384 data: 6 7 7 5
16040 176/300 Write block 1271 len 40960 data: 3 4 4 4 4 6 6 4 4 4
16041 177/300 Write block 2170 len 32768 data: 8 10 10 8 8 11 9 9
16042 178/300 Read block 1288 len 12288
16043 179/300 Write block 18 len 8192 data: 6 6
16044 180/300 Write block 408 len 16384 data: 6 7 6 7
16045 181/300 Write block 2328 len 40960 data: 2 3 3 3 4 5 5 6 7 7
16046 182/300 Read block 1280 len 4096
16047 183/300 Read block 1574 len 16384
16048 184/300 Write block 3172 len 36864 data: 3 3 3 3 3 5 5 5 7
16049 185/300 Read block 2857 len 36864
16050 186/300 Read block 4956 len 36864
16051 187/300 Read block 3228 len 4096
16052 188/300 Flush
16053 189/300 Write block 4201 len 28672 data: 6 6 7 7 7 6 5
16054 190/300 Write block 2045 len 24576 data: 5 4 4 4 3 3
16055 191/300 Read block 1414 len 36864
16056 192/300 Read block 1448 len 32768
16057 193/300 Write block 1997 len 32768 data: 4 4 2 2 3 4 4 3
16058 194/300 Read block 4367 len 16384
16059 195/300 Read block 4577 len 8192
16060 196/300 Read block 3305 len 24576
16061 197/300 Write block 4996 len 16384 data: 3 4 3 3
16062 198/300 Write block 1321 len 12288 data: 4 4 4
16063 199/300 Read block 4981 len 12288
16064 200/300 Write block 4779 len 16384 data: 4 6 6 4
16065 201/300 Read block 930 len 20480
16066 202/300 Write block 642 len 20480 data: 5 3 4 5 6
16067 203/300 Read block 2568 len 12288
16068 204/300 Read block 1411 len 36864
16069 205/300 Write block 1017 len 40960 data: 3 3 4 4 5 5 5 4 4 4
16070 206/300 Write block 3396 len 8192 data: 6 5
16071 207/300 Read block 2606 len 12288
16072 208/300 Read block 2122 len 16384
16073 209/300 Read block 1425 len 24576
16074 210/300 Read block 803 len 24576
16075 211/300 Read block 3458 len 40960
16076 212/300 Write block 1509 len 28672 data: 4 3 3 3 3 2 4
16077 213/300 Write block 1803 len 36864 data: 5 5 5 5 6 6 7 7 5
16078 214/300 Read block 3671 len 24576
16079 215/300 Read block 3159 len 12288
16080 216/300 Write block 1859 len 40960 data: 6 6 6 6 5 3 3 2 2 2
16081 217/300 Flush
16082 218/300 Read block 3321 len 40960
16083 219/300 Read block 4066 len 16384
16084 220/300 Write block 3179 len 36864 data: 6 8 8 6 7 9 8 9 7
16085 221/300 Write block 1514 len 4096 data: 3
16086 222/300 Read block 256 len 16384
16087 223/300 Flush
16088 224/300 Write block 61 len 8192 data: 3 3
16089 225/300 Read block 4193 len 16384
16090 226/300 Flush
16091 227/300 Read block 3579 len 28672
16092 228/300 Write block 103 len 20480 data: 8 9 8 7 8
16093 229/300 Write block 3444 len 32768 data: 3 6 6 6 6 7 10 10
16094 230/300 Write block 4452 len 24576 data: 7 7 9 8 8 8
16095 231/300 Write block 1979 len 16384 data: 5 5 4 4
16096 232/300 Read block 3262 len 32768
16097 233/300 Read block 2053 len 36864
16098 234/300 Read block 507 len 16384
16099 235/300 Read block 1050 len 8192
16100 236/300 Write block 3496 len 20480 data: 5 3 3 3 3
16101 237/300 Read block 1762 len 8192
16102 238/300 Write block 1834 len 8192 data: 5 4
16103 239/300 Flush
16104 240/300 Read block 30 len 32768
16105 241/300 Read block 1179 len 36864
16106 242/300 Read block 1946 len 16384
16107 243/300 Read block 2419 len 12288
16108 244/300 Read block 2730 len 32768
16109 245/300 Read block 914 len 36864
16110 246/300 Read block 873 len 36864
16111 247/300 Flush
16112 248/300 Read block 2449 len 20480
16113 249/300 Read block 2725 len 8192
16114 250/300 Write block 4217 len 12288 data: 3 4 3
16115 251/300 Write block 975 len 20480 data: 5 5 5 4 5
16116 252/300 Flush
16117 253/300 Read block 3654 len 36864
16118 254/300 Read block 582 len 20480
16119 255/300 Write block 1749 len 24576 data: 3 4 4 3 4 4
16120 256/300 Write block 1814 len 4096 data: 3
16121 257/300 Read block 4466 len 28672
16122 258/300 Write block 4218 len 12288 data: 5 4 3
16123 259/300 Read block 31 len 40960
16124 260/300 Read block 3431 len 32768
16125 261/300 Read block 4872 len 32768
16126 262/300 Write block 2711 len 40960 data: 4 5 5 4 3 3 3 3 4 4
16127 263/300 Write block 829 len 32768 data: 2 2 3 2 2 2 2 2
16128 264/300 Write block 1707 len 32768 data: 5 6 4 4 3 3 3 4
16129 265/300 Write block 586 len 16384 data: 4 4 3 3
16130 266/300 Write block 1271 len 20480 data: 4 5 5 5 5
16131 267/300 Write block 1607 len 12288 data: 5 5 6
16132 268/300 Read block 831 len 20480
16133 269/300 Read block 4256 len 16384
16134 270/300 Write block 3388 len 40960 data: 2 2 3 3 3 3 6 7 7 6
16135 271/300 Read block 1215 len 24576
16136 272/300 Write block 4757 len 8192 data: 2 2
16137 273/300 Write block 4021 len 8192 data: 6 9
16138 274/300 Read block 3439 len 36864
16139 275/300 Flush
16140 276/300 Read block 3055 len 28672
16141 277/300 Write block 3084 len 16384 data: 3 3 3 3
16142 278/300 Write block 1886 len 40960 data: 4 4 6 5 5 4 4 4 3 3
16143 279/300 Write block 3492 len 8192 data: 4 4
16144 280/300 Flush
16145 281/300 Read block 3178 len 40960
16146 282/300 Write block 2897 len 28672 data: 8 7 5 4 8 6 4
16147 283/300 Read block 4584 len 40960
16148 284/300 Read block 2873 len 36864
16149 285/300 Flush
16150 286/300 Read block 3818 len 4096
16151 287/300 Write block 22 len 8192 data: 4 3
16152 288/300 Write block 3475 len 16384 data: 4 2 4 4
16153 289/300 Write block 577 len 32768 data: 3 3 3 3 3 3 3 3
16154 290/300 Read block 3616 len 8192
16155 291/300 Write block 4687 len 24576 data: 5 5 5 8 7 6
16156 292/300 Read block 1463 len 32768
16157 293/300 Read block 3551 len 20480
16158 294/300 Read block 4474 len 8192
16159 295/300 Read block 195 len 24576
16160 296/300 Read block 2271 len 40960
16161 297/300 Write block 1377 len 24576 data: 3 3 3 5 6 5
16162 298/300 Write block 4285 len 16384 data: 6 5 6 5
16163 299/300 Write block 591 len 36864 data: 3 3 3 3 4 3 5 5 4
16164 300/300 Read block 9 len 32768
16165 Main task has work to do, go find it
16166 got dsc Start(2)
16167 start 2
16168 Feb 12 01:37:44.170 INFO request completed, latency_us: 222, response_code: 204, uri: /start/cid/2, method: POST, req_id: 85304d75-346b-4518-a982-bf230c1a92ad, remote_addr: 127.0.0.1:60456, local_addr: 127.0.0.1:9998
16169 Replay: started 2, returned:Ok(())
16170 ----------------------------------------------------------------
16171 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:300
16172 States: Active Active Offline
16173 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
16174 5126 Acked 6125 Write 6 Done Done New true
16175 5127 Acked 6126 Read 2 Done Done New true
16176 5128 Acked 6127 Read 2 Done Done New false
16177 5129 Acked 6128 Read 9 Done Done New false
16178 5130 Acked 6129 Read 8 Done Done New false
16179 5131 Acked 6130 Write 7 Done Done New false
16180 5132 Acked 6131 Write 10 Done Done New false
16181 5133 Acked 6132 Write 8 Done Done New false
16182 5134 Acked 6133 Read 6 Done Done New false
16183 5135 Acked 6134 Write 5 Done Done New false
16184 5136 Acked 6135 Write 3 Done Done New false
16185 5137 Acked 6136 Read 8 Done Done New false
16186 5138 Acked 6137 Write 8 Done Done New false
16187 5139 Acked 6138 Read 4 Done Done New false
16188 5140 Acked 6139 Read 3 Done Done New false
16189 5141 Acked 6140 Read 6 Done Done New false
16190 5142 Acked 6141 Read 7 Done Done New false
16191 5143 Acked 6142 Write 6 Done Done New false
16192 5144 Acked 6143 Read 4 Done Done New false
16193 5145 Acked 6144 Write 9 Done Done New false
16194 5146 Acked 6145 Read 7 Done Done New false
16195 5147 Acked 6146 Read 1 Done Done New false
16196 5148 Acked 6147 Read 5 Done Done New false
16197 5149 Acked 6148 Write 3 Done Done New false
16198 5150 Acked 6149 Read 7 Done Done New false
16199 5151 Acked 6150 Write 4 Done Done New false
16200 5152 Acked 6151 Read 2 Done Done New false
16201 5153 Acked 6152 Read 10 Done Done New false
16202 5154 Acked 6153 Write 4 Done Done New false
16203 5155 Acked 6154 Write 8 Done Done New false
16204 5156 Acked 6155 Write 4 Done Done New false
16205 5157 Acked 6156 Write 5 Done Done New false
16206 5158 Acked 6157 Write 2 Done Done New false
16207 5159 Acked 6158 Write 4 Done Done New false
16208 5160 Acked 6159 Read 6 Done Done New false
16209 5161 Acked 6160 Write 3 Done Done New false
16210 5162 Acked 6161 Write 9 Done Done New false
16211 5163 Acked 6162 Write 4 Done Done New false
16212 5164 Acked 6163 Read 4 Done Done New false
16213 5165 Acked 6164 Write 1 Done Done New false
16214 5166 Acked 6165 Read 7 Done Done New false
16215 5167 Acked 6166 Read 1 Done Done New false
16216 5168 Acked 6167 Read 3 Done Done New false
16217 5169 Acked 6168 Write 9 Done Done New false
16218 5170 Acked 6169 Read 1 Done Done New false
16219 5171 Acked 6170 Write 5 Done Done New false
16220 5172 Acked 6171 Write 4 Done Done New false
16221 5173 Acked 6172 Read 2 Done Done New false
16222 5174 Acked 6173 Read 1 Done Done New false
16223 5175 Acked 6174 Write 6 Done Done New false
16224 5176 Acked 6175 Read 8 Done Done New false
16225 5177 Acked 6176 Write 6 Done Done New false
16226 5178 Acked 6177 Write 1 Done Done New false
16227 5179 Acked 6178 Flush 0 Done Done New false
16228 5180 Acked 6179 Read 3 Done Done New false
16229 5181 Acked 6180 Write 8 Done Done New false
16230 5182 Acked 6181 Write 7 Done Done New false
16231 5183 Acked 6182 Read 7 Done Done New false
16232 5184 Acked 6183 Write 2 Done Done New false
16233 5185 Acked 6184 Read 1 Done Done New false
16234 5186 Acked 6185 Write 1 Done Done New false
16235 5187 Acked 6186 Write 9 Done Done New false
16236 5188 Acked 6187 Read 3 Done Done New false
16237 5189 Acked 6188 Write 6 Done Done New false
16238 5190 Acked 6189 Read 1 Done Done New false
16239 5191 Acked 6190 Write 3 Done Done New false
16240 5192 Acked 6191 Read 4 Done Done New false
16241 5193 Acked 6192 Read 2 Done Done New false
16242 5194 Acked 6193 Write 4 Done Done New false
16243 5195 Acked 6194 Read 8 Done Done New false
16244 5196 Acked 6195 Read 5 Done Done New false
16245 5197 Acked 6196 Write 2 Done Done New false
16246 5198 Acked 6197 Write 4 Done Done New false
16247 5199 Acked 6198 Read 7 Done Done New false
16248 5200 Acked 6199 Read 3 Done Done New false
16249 5201 Acked 6200 Read 10 Done Done New false
16250 5202 Acked 6201 Flush 0 Done Done New false
16251 5203 Acked 6202 Write 9 Done Done New false
16252 5204 Acked 6203 Write 8 Done Done New false
16253 5205 Acked 6204 Read 9 Done Done New false
16254 5206 Acked 6205 Read 4 Done Done New false
16255 5207 Acked 6206 Write 3 Done Done New false
16256 5208 Acked 6207 Write 7 Done Done New false
16257 5209 Acked 6208 Write 9 Done Done New false
16258 5210 Acked 6209 Write 2 Done Done New false
16259 5211 Acked 6210 Read 5 Done Done New false
16260 5212 Acked 6211 Read 4 Done Done New false
16261 5213 Acked 6212 Read 1 Done Done New false
16262 5214 Acked 6213 Read 4 Done Done New false
16263 5215 Acked 6214 Read 8 Done Done New false
16264 5216 Acked 6215 Flush 0 Done Done New false
16265 5217 Acked 6216 Flush 0 Done Done New false
16266 5218 Acked 6217 Write 4 Done Done New false
16267 5219 Acked 6218 Read 1 Done Done New false
16268 5220 Acked 6219 Write 3 Done Done New false
16269 5221 Acked 6220 Write 4 Done Done New false
16270 5222 Acked 6221 Read 9 Done Done New false
16271 5223 Acked 6222 Read 10 Done Done New false
16272 5224 Acked 6223 Write 4 Done Done New false
16273 5225 Acked 6224 Write 5 Done Done New false
16274 5226 Acked 6225 Flush 0 Done Done New false
16275 5227 Acked 6226 Read 3 Done Done New false
16276 5228 Acked 6227 Write 6 Done Done New false
16277 5229 Acked 6228 Read 8 Done Done New false
16278 5230 Acked 6229 Read 6 Done Done New false
16279 5231 Acked 6230 Write 7 Done Done New false
16280 5232 Acked 6231 Write 8 Done Done New false
16281 5233 Acked 6232 Read 2 Done Done New false
16282 5234 Acked 6233 Flush 0 Done Done New false
16283 5235 Acked 6234 Write 1 Done Done New false
16284 5236 Acked 6235 Read 9 Done Done New false
16285 5237 Acked 6236 Read 9 Done Done New false
16286 5238 Acked 6237 Read 9 Done Done New false
16287 5239 Acked 6238 Read 5 Done Done New false
16288 5240 Acked 6239 Write 5 Done Done New false
16289 5241 Acked 6240 Flush 0 Done Done New false
16290 5242 Acked 6241 Flush 0 Done Done New false
16291 5243 Acked 6242 Read 9 Done Done New false
16292 5244 Acked 6243 Read 7 Done Done New false
16293 5245 Acked 6244 Write 6 Done Done New false
16294 5246 Acked 6245 Read 3 Done Done New false
16295 5247 Acked 6246 Read 1 Done Done New false
16296 5248 Acked 6247 Read 4 Done Done New false
16297 5249 Acked 6248 Read 5 Done Done New false
16298 5250 Acked 6249 Flush 0 Done Done New false
16299 5251 Acked 6250 Write 4 Done Done New false
16300 5252 Acked 6251 Read 2 Done Done New false
16301 5253 Acked 6252 Write 6 Done Done New false
16302 5254 Acked 6253 Write 1 Done Done New false
16303 5255 Acked 6254 Flush 0 Done Done New false
16304 5256 Acked 6255 Flush 0 Done Done New false
16305 5257 Acked 6256 Read 5 Done Done New false
16306 5258 Acked 6257 Write 3 Done Done New false
16307 5259 Acked 6258 Write 6 Done Done New false
16308 5260 Acked 6259 Flush 0 Done Done New false
16309 5261 Acked 6260 Flush 0 Done Done New false
16310 5262 Acked 6261 Read 1 Done Done New false
16311 5263 Acked 6262 Write 1 Done Done New false
16312 5264 Acked 6263 Write 6 Done Done New false
16313 5265 Acked 6264 Read 7 Done Done New false
16314 5266 Acked 6265 Read 1 Done Done New false
16315 5267 Acked 6266 Write 1 Done Done New false
16316 5268 Acked 6267 Write 7 Done Done New false
16317 5269 Acked 6268 Read 8 Done Done New false
16318 5270 Acked 6269 Read 9 Done Done New false
16319 5271 Acked 6270 Read 9 Done Done New false
16320 5272 Acked 6271 Write 9 Done Done New false
16321 5273 Acked 6272 Write 9 Done Done New false
16322 5274 Acked 6273 Write 9 Done Done New false
16323 5275 Acked 6274 Read 7 Done Done New false
16324 5276 Acked 6275 Write 9 Done Done New false
16325 5277 Acked 6276 Write 2 Done Done New false
16326 5278 Acked 6277 Flush 0 Done Done New false
16327 5279 Acked 6278 Flush 0 Done Done New false
16328 5280 Acked 6279 Write 7 Done Done New false
16329 5281 Acked 6280 Write 7 Done Done New false
16330 5282 Acked 6281 Flush 0 Done Done New false
16331 5283 Acked 6282 Write 1 Done Done New false
16332 5284 Acked 6283 Read 5 Done Done New false
16333 5285 Acked 6284 Write 10 Done Done New false
16334 5286 Acked 6285 Flush 0 Done Done New false
16335 5287 Acked 6286 Write 6 Done Done New false
16336 5288 Acked 6287 Read 10 Done Done New false
16337 5289 Acked 6288 Read 1 Done Done New false
16338 5290 Acked 6289 Read 8 Done Done New false
16339 5291 Acked 6290 Read 9 Done Done New false
16340 5292 Acked 6291 Write 3 Done Done New false
16341 5293 Acked 6292 Write 4 Done Done New false
16342 5294 Acked 6293 Read 3 Done Done New false
16343 5295 Acked 6294 Write 10 Done Done New false
16344 5296 Acked 6295 Read 4 Done Done New false
16345 5297 Acked 6296 Write 1 Done Done New false
16346 5298 Acked 6297 Read 7 Done Done New false
16347 5299 Acked 6298 Write 3 Done Done New false
16348 5300 Acked 6299 Write 4 Done Done New false
16349 5301 Acked 6300 Write 10 Done Done New false
16350 5302 Acked 6301 Write 8 Done Done New false
16351 5303 Acked 6302 Read 3 Done Done New false
16352 5304 Acked 6303 Write 2 Done Done New false
16353 5305 Acked 6304 Write 4 Done Done New false
16354 5306 Acked 6305 Write 10 Done Done New false
16355 5307 Acked 6306 Read 1 Done Done New false
16356 5308 Acked 6307 Read 4 Done Done New false
16357 5309 Acked 6308 Write 9 Done Done New false
16358 5310 Acked 6309 Read 9 Done Done New false
16359 5311 Acked 6310 Read 9 Done Done New false
16360 5312 Acked 6311 Read 1 Done Done New false
16361 5313 Acked 6312 Flush 0 Done Done New false
16362 5314 Acked 6313 Write 7 Done Done New false
16363 5315 Acked 6314 Write 6 Done Done New false
16364 5316 Acked 6315 Read 9 Done Done New false
16365 5317 Acked 6316 Read 8 Done Done New false
16366 5318 Acked 6317 Write 8 Done Done New false
16367 5319 Acked 6318 Read 4 Done Done New false
16368 5320 Acked 6319 Read 2 Done Done New false
16369 5321 Acked 6320 Read 6 Done Done New false
16370 5322 Acked 6321 Write 4 Done Done New false
16371 5323 Acked 6322 Write 3 Done Done New false
16372 5324 Acked 6323 Read 3 Done Done New false
16373 5325 Acked 6324 Write 4 Done Done New false
16374 5326 Acked 6325 Read 5 Done Done New false
16375 5327 Acked 6326 Write 5 Done Done New false
16376 5328 Acked 6327 Read 3 Done Done New false
16377 5329 Acked 6328 Read 9 Done Done New false
16378 5330 Acked 6329 Write 10 Done Done New false
16379 5331 Acked 6330 Write 2 Done Done New false
16380 5332 Acked 6331 Read 3 Done Done New false
16381 5333 Acked 6332 Read 4 Done Done New false
16382 5334 Acked 6333 Read 6 Done Done New false
16383 5335 Acked 6334 Read 6 Done Done New false
16384 5336 Acked 6335 Read 10 Done Done New false
16385 5337 Acked 6336 Write 7 Done Done New false
16386 5338 Acked 6337 Write 9 Done Done New false
16387 5339 Acked 6338 Read 6 Done Done New false
16388 5340 Acked 6339 Read 3 Done Done New false
16389 5341 Acked 6340 Write 10 Done Done New false
16390 5342 Acked 6341 Flush 0 Done Done New false
16391 5343 Acked 6342 Read 10 Done Done New false
16392 5344 Acked 6343 Read 4 Done Done New false
16393 5345 Acked 6344 Write 9 Done Done New false
16394 5346 Acked 6345 Write 1 Done Done New false
16395 5347 Acked 6346 Read 4 Done Done New false
16396 5348 Acked 6347 Flush 0 Done Done New false
16397 5349 Acked 6348 Write 2 Done Done New false
16398 5350 Acked 6349 Read 4 Done Done New false
16399 5351 Acked 6350 Flush 0 Done Done New false
16400 5352 Acked 6351 Read 7 Done Done New false
16401 5353 Acked 6352 Write 5 Done Done New false
16402 5354 Acked 6353 Write 8 Done Done New false
16403 5355 Acked 6354 Write 6 Done Done New false
16404 5356 Acked 6355 Write 4 Done Done New false
16405 5357 Acked 6356 Read 8 Done Done New false
16406 5358 Acked 6357 Read 9 Done Done New false
16407 5359 Acked 6358 Read 4 Done Done New false
16408 5360 Acked 6359 Read 2 Done Done New false
16409 5361 Acked 6360 Write 5 Done Done New false
16410 5362 Acked 6361 Read 2 Done Done New false
16411 5363 Acked 6362 Write 2 Done Done New false
16412 5364 Acked 6363 Flush 0 Done Done New false
16413 5365 Acked 6364 Read 8 Done Done New false
16414 5366 Acked 6365 Read 9 Done Done New false
16415 5367 Acked 6366 Read 4 Done Done New false
16416 5368 Acked 6367 Read 3 Done Done New false
16417 5369 Acked 6368 Read 8 Done Done New false
16418 5370 Acked 6369 Read 9 Done Done New false
16419 5371 Acked 6370 Read 9 Done Done New false
16420 5372 Acked 6371 Flush 0 Done Done New false
16421 5373 Acked 6372 Read 5 Done Done New false
16422 5374 Acked 6373 Read 2 Done Done New false
16423 5375 Acked 6374 Write 3 Done Done New false
16424 5376 Acked 6375 Write 5 Done Done New false
16425 5377 Acked 6376 Flush 0 Done Done New false
16426 5378 Acked 6377 Read 9 Done Done New false
16427 5379 Acked 6378 Read 5 Done Done New false
16428 5380 Acked 6379 Write 6 Done Done New false
16429 5381 Acked 6380 Write 1 Done Done New false
16430 5382 Acked 6381 Read 7 Done Done New false
16431 5383 Acked 6382 Write 3 Done Done New false
16432 5384 Acked 6383 Read 10 Done Done New false
16433 5385 Acked 6384 Read 8 Done Done New false
16434 5386 Acked 6385 Read 8 Done Done New false
16435 5387 Acked 6386 Write 10 Done Done New false
16436 5388 Acked 6387 Write 8 Done Done New false
16437 5389 Acked 6388 Write 8 Done Done New false
16438 5390 Acked 6389 Write 4 Done Done New false
16439 5391 Acked 6390 Write 5 Done Done New false
16440 5392 Acked 6391 Write 3 Done Done New false
16441 5393 Acked 6392 Read 5 Done Done New false
16442 5394 Acked 6393 Read 4 Done Done New false
16443 5395 Acked 6394 Write 10 Done Done New false
16444 5396 Acked 6395 Read 6 Done Done New false
16445 5397 Acked 6396 Write 2 Done Done New false
16446 5398 Acked 6397 Write 2 Done Done New false
16447 5399 Acked 6398 Read 9 Done Done New false
16448 5400 Acked 6399 Flush 0 Done Done New false
16449 5401 Acked 6400 Read 7 Done Done New false
16450 5402 Acked 6401 Write 4 Done Done New false
16451 5403 Acked 6402 Write 10 Done Done New false
16452 5404 Acked 6403 Write 2 Done Done New false
16453 5405 Acked 6404 Flush 0 Done Done New false
16454 5406 Acked 6405 Read 10 Done Done New false
16455 5407 Acked 6406 Write 7 Done Done New false
16456 5408 Acked 6407 Read 10 Done Done New false
16457 5409 Acked 6408 Read 9 Done Done New false
16458 5410 Acked 6409 Flush 0 Done Done New false
16459 5411 Acked 6410 Read 1 Done Done New false
16460 5412 Acked 6411 Write 2 Done Done New false
16461 5413 Acked 6412 Write 4 Done Done New false
16462 5414 Acked 6413 Write 8 Done Done New false
16463 5415 Acked 6414 Read 2 Done Done New false
16464 5416 Acked 6415 Write 6 Done Done New false
16465 5417 Acked 6416 Read 8 Done Done New false
16466 5418 Acked 6417 Read 5 Done Done New false
16467 5419 Acked 6418 Read 2 Done Done New false
16468 5420 Acked 6419 Read 6 Done Done New false
16469 5421 Acked 6420 Read 10 Done Done New false
16470 5422 Acked 6421 Write 6 Done Done New false
16471 5423 Acked 6422 Write 4 Done Done New false
16472 5424 Acked 6423 Write 9 Done Done New false
16473 5425 Acked 6424 Read 8 Done Done New false
16474 STATES DS:0 DS:1 DS:2 TOTAL
16475 New 0 0 300 300
16476 Sent 0 0 0 0
16477 Done 300 300 0 600
16478 Skipped 0 0 0 0
16479 Error 0 0 0 0
16480 Last Flush: 6409 6409 6124
16481 Downstairs last five completed: 6124 6123 6122 6121 6120
16482 Upstairs last five completed: 5425 5424 5423 5422 5421
16483 CLIENT: Up:0 ds:300 act:2
164842024-02-12T01:37:44.833ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
164852024-02-12T01:37:44.833ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
164862024-02-12T01:37:44.833ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
164872024-02-12T01:37:44.833ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
164882024-02-12T01:37:44.833ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
164892024-02-12T01:37:44.833ZINFOcrucible: [2] client re-new 301 jobs since flush 6124 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
164902024-02-12T01:37:44.833ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
164912024-02-12T01:37:45.835ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
164922024-02-12T01:37:45.835ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
164932024-02-12T01:37:45.835ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
164942024-02-12T01:37:45.835ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
164952024-02-12T01:37:45.835ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
16496 [8830] got start action
164972024-02-12T01:37:45.835ZINFOcrucible: [2] client re-new 301 jobs since flush 6124 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
16498 [8830] I am going to restart
16499 Starting downstairs at port 8830
16500 Make output file at "/tmp/dsc/downstairs-8830.txt"
16501 [8830][2] reports Starting
165022024-02-12T01:37:45.835ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
16503 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1182)
16504 [8830][2] reports Running
165052024-02-12T01:37:46.837ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
165062024-02-12T01:37:46.837ZINFOcrucible: ds_connection connected = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
165072024-02-12T01:37:46.838ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
165082024-02-12T01:37:46.838ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
165092024-02-12T01:37:46.838ZINFOcrucible: send last flush ID to this DS: 6124 = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
165102024-02-12T01:37:46.838ZINFOcrucible: Replied this last flush ID: 6124 = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
165112024-02-12T01:37:46.838ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
165122024-02-12T01:37:46.838ZINFOcrucible: [2] Transition from Offline to Replay = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
165132024-02-12T01:37:46.838ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
165142024-02-12T01:37:46.838ZINFOcrucible: [2] Transition from Replay to Active = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
16515 ----------------------------------------------------------------
16516 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
16517 Downstairs last five completed: 6425 6424 6423 6422 6421
16518 Upstairs last five completed: 5426 5425 5424 5423 5422
16519 CLIENT: Up:0 ds:0 act:3
16520 Replay: All jobs finished, all DS active.
16521 Main task has work to do, go find it
16522 got dsc Stop(1)
16523 stop 1
16524 Feb 12 01:37:48.173 INFO request completed, latency_us: [8820] Got stop action so:false kr:false
16525 358, response_code: 204, uri: /stop/cid/1, method: POST, req_id: a4633e18-f05d-4887-803e-f626b84c8928, remote_addr: 127.0.0.1:60456, local_addr: 127.0.0.1:9998
165262024-02-12T01:37:48.174ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
16527 Feb 12 01:37:48.174 INFO request completed, latency_us: 138, response_code: 200, uri: /state/cid/1, method: GET, req_id: 22a3766a-3d0c-4688-b07a-33d2b03bd1cf, remote_addr: {"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2024-02-12T01:37:48.174113496Z","hostname":"ip-10-150-1-104.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"e1773b3e-b36f-4080-8b13-c0830f0d072e"}
16528 127.0.0.1:60456, local_addr:{"msg":"downstairs task for 1 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-02-12T01:37:48.174150751Z","hostname":"ip-10-150-1-104.us-west-2.compute.internal","pid":1164,"session_id":"e1773b3e-b36f-4080-8b13-c0830f0d072e"}
16529 127.0.0.1:9998
165302024-02-12T01:37:48.174ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
165312024-02-12T01:37:48.174ZINFOcrucible: [1] client re-new 0 jobs since flush 6425 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
165322024-02-12T01:37:48.174ZINFOcrucible: client task is exiting = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
16533 [8820] Exited with: ExitStatus(unix_wait_status(9))
16534 [8820][1] reports Exit
165352024-02-12T01:37:49.175ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
165362024-02-12T01:37:49.175ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
165372024-02-12T01:37:49.175ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
165382024-02-12T01:37:49.175ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
165392024-02-12T01:37:49.175ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
165402024-02-12T01:37:49.175ZINFOcrucible: [1] client re-new 0 jobs since flush 6425 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
165412024-02-12T01:37:49.175ZINFOcrucible: client task is exiting = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
165422024-02-12T01:37:50.176ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
165432024-02-12T01:37:50.176ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
165442024-02-12T01:37:50.176ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
165452024-02-12T01:37:50.176ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
165462024-02-12T01:37:50.176ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
165472024-02-12T01:37:50.176ZINFOcrucible: [1] client re-new 0 jobs since flush 6425 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
165482024-02-12T01:37:50.177ZINFOcrucible: client task is exiting = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
165492024-02-12T01:37:51.178ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
165502024-02-12T01:37:51.178ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
165512024-02-12T01:37:51.178ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
165522024-02-12T01:37:51.178ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
165532024-02-12T01:37:51.178ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
165542024-02-12T01:37:51.178ZINFOcrucible: [1] client re-new 0 jobs since flush 6425 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
165552024-02-12T01:37:51.178ZINFOcrucible: client task is exiting = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
16556 Feb 12 01:37:52.176 INFO request completed, latency_us: 212, response_code: 200, uri: /state/cid/1, method: GET, req_id: 38fefe83-a0db-470f-b37e-c9f18d024694, remote_addr: 127.0.0.1:60456, local_addr: 127.0.0.1:9998
16557 001/300 Read block 3578 len 4096
16558 002/300 Read block 3806 len 16384
16559 003/300 Write block 4847 len 4096 data: 4
16560 004/300 Read block 1784 len 16384
165612024-02-12T01:37:52.179ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
165622024-02-12T01:37:52.179ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
165632024-02-12T01:37:52.179ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
165642024-02-12T01:37:52.179ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
165652024-02-12T01:37:52.179ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
165662024-02-12T01:37:52.179ZINFOcrucible: [1] client re-new 4 jobs since flush 6425 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
165672024-02-12T01:37:52.179ZINFOcrucible: client task is exiting = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
16568 005/300 Read block 2994 len 40960
16569 006/300 Read block 1587 len 20480
16570 007/300 Flush
16571 008/300 Read block 4512 len 40960
16572 009/300 Write block 2442 len 16384 data: 5 5 7 6
16573 010/300 Read block 3286 len 4096
16574 011/300 Write block 3140 len 28672 data: 4 5 6 5 4 4 4
16575 012/300 Read block 1298 len 36864
16576 013/300 Read block 1479 len 40960
16577 014/300 Flush
16578 015/300 Read block 3342 len 40960
16579 016/300 Flush
16580 017/300 Read block 4042 len 40960
16581 018/300 Write block 2300 len 36864 data: 5 5 5 4 4 5 3 3 3
16582 019/300 Write block 650 len 36864 data: 3 3 3 3 2 2 3 4 5
16583 020/300 Read block 3121 len 32768
16584 021/300 Read block 426 len 32768
16585 022/300 Write block 2397 len 32768 data: 4 4 5 5 6 5 5 5
16586 023/300 Write block 556 len 28672 data: 2 3 4 3 3 3 3
16587 024/300 Read block 463 len 28672
16588 025/300 Write block 3620 len 32768 data: 5 5 5 4 4 5 5 5
16589 026/300 Write block 4156 len 4096 data: 3
16590 027/300 Flush
16591 028/300 Read block 2418 len 4096
16592 029/300 Read block 4484 len 12288
16593 030/300 Flush
16594 031/300 Write block 3539 len 4096 data: 3
16595 032/300 Read block 4435 len 20480
16596 033/300 Write block 629 len 12288 data: 3 2 2
16597 034/300 Write block 1923 len 16384 data: 4 2 3 3
16598 035/300 Read block 1341 len 12288
16599 036/300 Write block 4363 len 12288 data: 8 7 8
16600 037/300 Read block 2619 len 40960
16601 038/300 Read block 3656 len 20480
16602 039/300 Read block 675 len 20480
16603 040/300 Read block 4977 len 40960
16604 041/300 Write block 2268 len 12288 data: 5 5 4
16605 042/300 Read block 3834 len 24576
16606 043/300 Write block 2666 len 8192 data: 4 3
16607 044/300 Read block 2544 len 36864
16608 045/300 Read block 2009 len 20480
16609 046/300 Write block 242 len 12288 data: 5 6 7
16610 047/300 Read block 4753 len 28672
16611 048/300 Read block 2403 len 36864
16612 049/300 Read block 358 len 32768
16613 050/300 Write block 1880 len 4096 data: 5
16614 051/300 Read block 4088 len 20480
16615 052/300 Read block 4023 len 16384
16616 053/300 Read block 4310 len 24576
16617 054/300 Write block 3207 len 24576 data: 2 2 3 2 2 2
16618 055/300 Read block 3935 len 12288
16619 056/300 Read block 4378 len 16384
16620 057/300 Read block 734 len 20480
16621 058/300 Read block 2144 len 36864
16622 059/300 Read block 3369 len 36864
16623 060/300 Read block 1038 len 16384
16624 061/300 Write block 3245 len 32768 data: 4 4 4 5 5 4 3 4
16625 062/300 Read block 1624 len 36864
16626 063/300 Write block 519 len 28672 data: 2 2 2 2 2 2 3
16627 064/300 Read block 3461 len 24576
16628 065/300 Read block 2952 len 28672
16629 066/300 Flush
16630 067/300 Write block 1031 len 20480 data: 3 3 4 5 5
16631 068/300 Write block 2305 len 20480 data: 6 4 4 4 4
16632 069/300 Flush
16633 070/300 Read block 2924 len 16384
16634 071/300 Write block 4971 len 40960 data: 4 3 3 2 3 3 3 3 3 3
16635 072/300 Read block 2197 len 20480
16636 073/300 Read block 3082 len 24576
16637 074/300 Read block 4279 len 36864
16638 075/300 Write block 234 len 20480 data: 3 3 3 4 3
16639 076/300 Write block 584 len 4096 data: 4
16640 077/300 Write block 1725 len 24576 data: 3 5 6 5 6 5
16641 078/300 Read block 1295 len 40960
16642 079/300 Read block 2325 len 36864
16643 080/300 Write block 4316 len 16384 data: 5 5 4 8
16644 081/300 Read block 364 len 20480
16645 082/300 Write block 2090 len 4096 data: 4
16646 083/300 Read block 3688 len 24576
16647 084/300 Read block 33 len 16384
16648 085/300 Read block 1023 len 8192
16649 086/300 Read block 3604 len 28672
16650 087/300 Read block 3468 len 28672
16651 088/300 Read block 2289 len 24576
16652 089/300 Read block 3091 len 32768
16653 090/300 Write block 1116 len 8192 data: 2 2
16654 091/300 Read block 1177 len 24576
16655 092/300 Read block 2482 len 28672
16656 093/300 Write block 2090 len 20480 data: 5 4 6 6 5
16657 094/300 Read block 1219 len 36864
16658 095/300 Write block 3896 len 32768 data: 5 5 4 5 5 4 3 2
16659 096/300 Write block 1635 len 12288 data: 4 4 6
16660 097/300 Read block 1595 len 24576
16661 098/300 Flush
16662 099/300 Read block 542 len 24576
16663 100/300 Write block 353 len 24576 data: 6 6 6 8 8 7
16664 101/300 Write block 2811 len 12288 data: 9 8 7
16665 102/300 Write block 3439 len 28672 data: 3 4 4 4 3 4 7
16666 103/300 Read block 539 len 12288
16667 104/300 Read block 4098 len 12288
16668 105/300 Read block 1454 len 20480
16669 106/300 Read block 4709 len 32768
16670 107/300 Read block 3340 len 28672
16671 108/300 Read block 1995 len 4096
16672 109/300 Read block 4852 len 16384
16673 110/300 Write block 2290 len 24576 data: 4 4 4 4 4 4
16674 111/300 Read block 1114 len 36864
16675 112/300 Flush
16676 113/300 Write block 1197 len 20480 data: 4 3 3 3 4
16677 114/300 Read block 3466 len 8192
16678 115/300 Flush
16679 116/300 Read block 992 len 12288
16680 117/300 Read block 3070 len 20480
16681 118/300 Write block 2809 len 28672 data: 10 9 10 9 8 5 4
16682 119/300 Read block 865 len 16384
16683 120/300 Read block 1278 len 4096
16684 121/300 Write block 4786 len 4096 data: 6
16685 122/300 Write block 1241 len 40960 data: 2 2 3 2 2 2 2 2 2 3
16686 123/300 Read block 3730 len 28672
16687 124/300 Read block 4729 len 28672
16688 125/300 Write block 1021 len 8192 data: 6 6
16689 126/300 Flush
16690 127/300 Read block 1000 len 8192
16691 128/300 Write block 2851 len 36864 data: 6 5 6 7 6 6 5 5 5
16692 129/300 Read block 226 len 16384
16693 130/300 Write block 3514 len 36864 data: 5 5 4 3 4 5 5 4 5
16694 131/300 Write block 999 len 24576 data: 4 4 4 4 4 4
16695 132/300 Write block 229 len 32768 data: 6 6 6 7 5 4 4 4
16696 133/300 Read block 4400 len 8192
16697 134/300 Write block 663 len 28672 data: 5 6 6 7 6 6 5
16698 135/300 Read block 4843 len 40960
16699 136/300 Read block 2419 len 36864
16700 137/300 Write block 160 len 8192 data: 5 3
16701 138/300 Write block 4051 len 4096 data: 3
16702 139/300 Write block 3854 len 12288 data: 2 3 3
16703 140/300 Write block 4481 len 40960 data: 6 9 9 7 6 5 4 4 4 4
16704 141/300 Read block 2210 len 20480
16705 142/300 Write block 4384 len 12288 data: 3 3 3
16706 143/300 Flush
16707 144/300 Read block 1680 len 8192
16708 145/300 Read block 498 len 28672
16709 146/300 Read block 861 len 32768
16710 147/300 Read block 929 len 12288
16711 148/300 Read block 2482 len 32768
16712 149/300 Read block 3373 len 20480
16713 150/300 Read block 3697 len 40960
16714 151/300 Write block 3238 len 40960 data: 7 6 6 4 4 4 4 5 5 5
16715 152/300 Read block 3017 len 12288
16716 153/300 Read block 3080 len 28672
16717 154/300 Read block 1992 len 40960
16718 155/300 Read block 3804 len 12288
16719 156/300 Read block 4810 len 20480
16720 157/300 Flush
16721 158/300 Write block 1726 len 20480 data: 6 7 6 7 6
16722 159/300 Read block 1734 len 4096
16723 160/300 Flush
16724 161/300 Write block 1225 len 4096 data: 5
16725 162/300 Read block 32 len 4096
16726 163/300 Read block 2751 len 32768
16727 164/300 Write block 2986 len 8192 data: 4 3
16728 165/300 Write block 4173 len 16384 data: 8 9 5 5
16729 166/300 Write block 1360 len 24576 data: 4 4 4 4 5 5
16730 167/300 Write block 1100 len 24576 data: 3 3 4 4 5 5
16731 168/300 Read block 3452 len 20480
16732 169/300 Read block 3939 len 40960
16733 170/300 Read block 2441 len 24576
16734 171/300 Read block 2658 len 36864
16735 172/300 Read block 1017 len 36864
16736 173/300 Write block 3615 len 24576 data: 4 3 5 5 5 6
16737 174/300 Write block 212 len 4096 data: 3
16738 175/300 Write block 4415 len 12288 data: 3 3 5
16739 176/300 Read block 1267 len 4096
16740 177/300 Write block 599 len 4096 data: 5
16741 178/300 Read block 1227 len 32768
16742 179/300 Read block 4006 len 20480
16743 180/300 Write block 3922 len 12288 data: 4 5 3
16744 181/300 Write block 4111 len 36864 data: 2 3 3 3 3 4 4 5 5
16745 182/300 Read block 1749 len 28672
16746 183/300 Write block 4070 len 40960 data: 6 6 7 7 5 5 5 5 4 3
16747 184/300 Write block 852 len 12288 data: 3 4 3
16748 185/300 Write block 366 len 40960 data: 2 2 4 5 5 5 5 5 5 4
16749 186/300 Write block 1529 len 40960 data: 3 4 5 5 6 8 7 8 8 7
16750 187/300 Write block 4232 len 12288 data: 4 4 4
16751 188/300 Write block 1000 len 4096 data: 5
16752 189/300 Write block 338 len 16384 data: 4 3 3 4
16753 190/300 Write block 1285 len 28672 data: 3 3 3 3 3 2 2
16754 191/300 Flush
16755 192/300 Write block 2121 len 36864 data: 4 4 4 3 4 5 6 6 6
16756 193/300 Flush
16757 194/300 Write block 1352 len 12288 data: 4 3 4
16758 195/300 Write block 4944 len 12288 data: 3 3 2
16759 196/300 Write block 995 len 12288 data: 2 2 3
16760 197/300 Read block 3406 len 36864
16761 198/300 Read block 1987 len 8192
16762 199/300 Write block 4128 len 20480 data: 3 3 3 3 2
16763 200/300 Write block 4770 len 24576 data: 4 4 4 4 3 5
16764 201/300 Flush
16765 202/300 Write block 2932 len 8192 data: 4 4
16766 203/300 Read block 4456 len 28672
16767 204/300 Read block 2105 len 8192
16768 205/300 Read block 2644 len 16384
16769 206/300 Write block 4308 len 8192 data: 4 3
16770 207/300 Read block 2233 len 20480
16771 208/300 Read block 458 len 12288
16772 209/300 Flush
16773 210/300 Flush
16774 211/300 Write block 4474 len 40960 data: 6 8 8 6 5 6 5 7 10 10
16775 212/300 Write block 2481 len 16384 data: 5 7 6 6
16776 213/300 Read block 3817 len 20480
16777 214/300 Read block 2635 len 32768
16778 215/300 Write block 3620 len 8192 data: 7 6
16779 216/300 Read block 3098 len 16384
16780 217/300 Read block 2459 len 28672
16781 218/300 Read block 4536 len 12288
16782 219/300 Write block 3465 len 12288 data: 3 3 3
16783 220/300 Read block 2248 len 8192
16784 221/300 Write block 5 len 28672 data: 3 2 3 3 3 3 4
16785 222/300 Read block 4524 len 40960
16786 223/300 Write block 1711 len 12288 data: 4 4 4
16787 224/300 Flush
16788 225/300 Write block 1557 len 4096 data: 5
16789 226/300 Write block 4446 len 32768 data: 4 4 6 6 8 7 8 8
16790 227/300 Read block 3447 len 28672
16791 228/300 Write block 1481 len 20480 data: 6 6 6 4 3
16792 229/300 Write block 4949 len 24576 data: 3 5 4 4 3 3
16793 230/300 Read block 3004 len 20480
16794 231/300 Write block 872 len 12288 data: 5 5 6
16795 232/300 Write block 194 len 32768 data: 5 5 6 7 3 3 3 4
16796 233/300 Read block 3116 len 16384
16797 234/300 Write block 3944 len 12288 data: 5 6 5
16798 235/300 Read block 1740 len 4096
16799 236/300 Write block 400 len 12288 data: 4 5 4
16800 237/300 Read block 2138 len 16384
16801 238/300 Write block 3342 len 40960 data: 3 4 3 3 2 2 2 2 2 2
16802 239/300 Read block 4326 len 28672
16803 240/300 Read block 3996 len 40960
16804 241/300 Write block 4901 len 20480 data: 4 4 6 7 6
16805 242/300 Write block 524 len 8192 data: 3 4
16806 243/300 Write block 1024 len 20480 data: 5 5 5 4 3
16807 244/300 Flush
16808 245/300 Read block 3557 len 4096
16809 246/300 Write block 3636 len 36864 data: 3 4 4 4 5 4 4 4 4
16810 247/300 Flush
16811 248/300 Write block 818 len 8192 data: 4 3
16812 249/300 Write block 3571 len 4096 data: 4
16813 250/300 Read block 3510 len 28672
16814 251/300 Read block 1476 len 4096
16815 252/300 Read block 3218 len 28672
16816 253/300 Read block 4678 len 4096
16817 254/300 Read block 1184 len 40960
16818 255/300 Read block 1319 len 12288
16819 256/300 Write block 935 len 32768 data: 7 7 8 7 5 4 4 5
16820 257/300 Write block 2342 len 24576 data: 6 4 5 4 3 3
16821 258/300 Read block 1672 len 20480
16822 259/300 Read block 2180 len 40960
16823 260/300 Write block 2947 len 16384 data: 2 2 2 2
16824 261/300 Read block 3884 len 4096
16825 262/300 Read block 1449 len 20480
16826 263/300 Read block 776 len 24576
16827 264/300 Write block 4755 len 20480 data: 3 3 3 3 2
16828 265/300 Write block 3491 len 12288 data: 4 5 5
16829 266/300 Write block 1895 len 24576 data: 4 2 2 2 3 3
16830 267/300 Read block 2312 len 36864
16831 268/300 Write block 4470 len 20480 data: 5 5 6 6 7
16832 269/300 Flush
16833 270/300 Write block 3224 len 40960 data: 8 8 8 8 9 8 7 7 7 8
16834 271/300 Write block 4010 len 4096 data: 2
16835 272/300 Flush
16836 273/300 Read block 1795 len 16384
16837 274/300 Read block 3997 len 36864
16838 275/300 Read block 347 len 28672
16839 276/300 Flush
16840 277/300 Write block 3341 len 16384 data: 3 4 5 4
16841 278/300 Write block 3310 len 20480 data: 2 2 2 4 5
16842 279/300 Read block 1174 len 4096
16843 280/300 Read block 1212 len 8192
16844 281/300 Read block 1987 len 8192
16845 282/300 Read block 685 len 36864
16846 283/300 Read block 1469 len 36864
16847 284/300 Write block 1040 len 12288 data: 2 2 2
16848 285/300 Read block 4869 len 40960
16849 286/300 Read block 1440 len 4096
16850 287/300 Read block 3861 len 20480
16851 288/300 Read block 1037 len 8192
16852 289/300 Write block 3915 len 4096 data: 5
16853 290/300 Write block 367 len 4096 data: 3
16854 291/300 Write block 1710 len 12288 data: 5 5 5
16855 292/300 Read block 1677 len 36864
16856 293/300 Read block 1974 len 8192
16857 294/300 Flush
16858 295/300 Read block 3213 len 28672
16859 296/300 Write block 2935 len 36864 data: 3 3 3 2 2 2 2 2 2
16860 297/300 Write block 3486 len 32768 data: 5 5 5 4 5 5 6 6
16861 298/300 Write block 723 len 12288 data: 4 3 2
16862 299/300 Write block 3165 len 32768 data: 6 7 6 6 6 5 3 4
16863 300/300 Write block 935 len 4096 data: 8
16864 Main task has work to do, go find it
16865 got dsc Start(1)
16866 start 1
16867 Feb 12 01:37:52.517 INFO request completed, latency_us: 224, response_code: 204, uri: /start/cid/1, method: POST, req_id: 0a168198-f5c6-4da3-8930-0b6e230bfe57, remote_addr: 127.0.0.1:60456, local_addr: 127.0.0.1:9998
16868 Replay: started 1, returned:Ok(())
16869 ----------------------------------------------------------------
16870 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:300
16871 States: Active Offline Active
16872 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
16873 5427 Acked 6426 Read 1 Done New Done true
16874 5428 Acked 6427 Read 4 Done New Done true
16875 5429 Acked 6428 Write 1 Done New Done true
16876 5430 Acked 6429 Read 4 Done New Done true
16877 5431 Acked 6430 Read 10 Done New Done false
16878 5432 Acked 6431 Read 5 Done New Done false
16879 5433 Acked 6432 Flush 0 Done New Done false
16880 5434 Acked 6433 Read 10 Done New Done false
16881 5435 Acked 6434 Write 4 Done New Done false
16882 5436 Acked 6435 Read 1 Done New Done false
16883 5437 Acked 6436 Write 7 Done New Done false
16884 5438 Acked 6437 Read 9 Done New Done false
16885 5439 Acked 6438 Read 10 Done New Done false
16886 5440 Acked 6439 Flush 0 Done New Done false
16887 5441 Acked 6440 Read 10 Done New Done false
16888 5442 Acked 6441 Flush 0 Done New Done false
16889 5443 Acked 6442 Read 10 Done New Done false
16890 5444 Acked 6443 Write 9 Done New Done false
16891 5445 Acked 6444 Write 9 Done New Done false
16892 5446 Acked 6445 Read 8 Done New Done false
16893 5447 Acked 6446 Read 8 Done New Done false
16894 5448 Acked 6447 Write 8 Done New Done false
16895 5449 Acked 6448 Write 7 Done New Done false
16896 5450 Acked 6449 Read 7 Done New Done false
16897 5451 Acked 6450 Write 8 Done New Done false
16898 5452 Acked 6451 Write 1 Done New Done false
16899 5453 Acked 6452 Flush 0 Done New Done false
16900 5454 Acked 6453 Read 1 Done New Done false
16901 5455 Acked 6454 Read 3 Done New Done false
16902 5456 Acked 6455 Flush 0 Done New Done false
16903 5457 Acked 6456 Write 1 Done New Done false
16904 5458 Acked 6457 Read 5 Done New Done false
16905 5459 Acked 6458 Write 3 Done New Done false
16906 5460 Acked 6459 Write 4 Done New Done false
16907 5461 Acked 6460 Read 3 Done New Done false
16908 5462 Acked 6461 Write 3 Done New Done false
16909 5463 Acked 6462 Read 10 Done New Done false
16910 5464 Acked 6463 Read 5 Done New Done false
16911 5465 Acked 6464 Read 5 Done New Done false
16912 5466 Acked 6465 Read 10 Done New Done false
16913 5467 Acked 6466 Write 3 Done New Done false
16914 5468 Acked 6467 Read 6 Done New Done false
16915 5469 Acked 6468 Write 2 Done New Done false
16916 5470 Acked 6469 Read 9 Done New Done false
16917 5471 Acked 6470 Read 5 Done New Done false
16918 5472 Acked 6471 Write 3 Done New Done false
16919 5473 Acked 6472 Read 7 Done New Done false
16920 5474 Acked 6473 Read 9 Done New Done false
16921 5475 Acked 6474 Read 8 Done New Done false
16922 5476 Acked 6475 Write 1 Done New Done false
16923 5477 Acked 6476 Read 5 Done New Done false
16924 5478 Acked 6477 Read 4 Done New Done false
16925 5479 Acked 6478 Read 6 Done New Done false
16926 5480 Acked 6479 Write 6 Done New Done false
16927 5481 Acked 6480 Read 3 Done New Done false
16928 5482 Acked 6481 Read 4 Done New Done false
16929 5483 Acked 6482 Read 5 Done New Done false
16930 5484 Acked 6483 Read 9 Done New Done false
16931 5485 Acked 6484 Read 9 Done New Done false
16932 5486 Acked 6485 Read 4 Done New Done false
16933 5487 Acked 6486 Write 8 Done New Done false
16934 5488 Acked 6487 Read 9 Done New Done false
16935 5489 Acked 6488 Write 7 Done New Done false
16936 5490 Acked 6489 Read 6 Done New Done false
16937 5491 Acked 6490 Read 7 Done New Done false
16938 5492 Acked 6491 Flush 0 Done New Done false
16939 5493 Acked 6492 Write 5 Done New Done false
16940 5494 Acked 6493 Write 5 Done New Done false
16941 5495 Acked 6494 Flush 0 Done New Done false
16942 5496 Acked 6495 Read 4 Done New Done false
16943 5497 Acked 6496 Write 10 Done New Done false
16944 5498 Acked 6497 Read 5 Done New Done false
16945 5499 Acked 6498 Read 6 Done New Done false
16946 5500 Acked 6499 Read 9 Done New Done false
16947 5501 Acked 6500 Write 5 Done New Done false
16948 5502 Acked 6501 Write 1 Done New Done false
16949 5503 Acked 6502 Write 6 Done New Done false
16950 5504 Acked 6503 Read 10 Done New Done false
16951 5505 Acked 6504 Read 9 Done New Done false
16952 5506 Acked 6505 Write 4 Done New Done false
16953 5507 Acked 6506 Read 5 Done New Done false
16954 5508 Acked 6507 Write 1 Done New Done false
16955 5509 Acked 6508 Read 6 Done New Done false
16956 5510 Acked 6509 Read 4 Done New Done false
16957 5511 Acked 6510 Read 2 Done New Done false
16958 5512 Acked 6511 Read 7 Done New Done false
16959 5513 Acked 6512 Read 7 Done New Done false
16960 5514 Acked 6513 Read 6 Done New Done false
16961 5515 Acked 6514 Read 8 Done New Done false
16962 5516 Acked 6515 Write 2 Done New Done false
16963 5517 Acked 6516 Read 6 Done New Done false
16964 5518 Acked 6517 Read 7 Done New Done false
16965 5519 Acked 6518 Write 5 Done New Done false
16966 5520 Acked 6519 Read 9 Done New Done false
16967 5521 Acked 6520 Write 8 Done New Done false
16968 5522 Acked 6521 Write 3 Done New Done false
16969 5523 Acked 6522 Read 6 Done New Done false
16970 5524 Acked 6523 Flush 0 Done New Done false
16971 5525 Acked 6524 Read 6 Done New Done false
16972 5526 Acked 6525 Write 6 Done New Done false
16973 5527 Acked 6526 Write 3 Done New Done false
16974 5528 Acked 6527 Write 7 Done New Done false
16975 5529 Acked 6528 Read 3 Done New Done false
16976 5530 Acked 6529 Read 3 Done New Done false
16977 5531 Acked 6530 Read 5 Done New Done false
16978 5532 Acked 6531 Read 8 Done New Done false
16979 5533 Acked 6532 Read 7 Done New Done false
16980 5534 Acked 6533 Read 1 Done New Done false
16981 5535 Acked 6534 Read 4 Done New Done false
16982 5536 Acked 6535 Write 6 Done New Done false
16983 5537 Acked 6536 Read 9 Done New Done false
16984 5538 Acked 6537 Flush 0 Done New Done false
16985 5539 Acked 6538 Write 5 Done New Done false
16986 5540 Acked 6539 Read 2 Done New Done false
16987 5541 Acked 6540 Flush 0 Done New Done false
16988 5542 Acked 6541 Read 3 Done New Done false
16989 5543 Acked 6542 Read 5 Done New Done false
16990 5544 Acked 6543 Write 7 Done New Done false
16991 5545 Acked 6544 Read 4 Done New Done false
16992 5546 Acked 6545 Read 1 Done New Done false
16993 5547 Acked 6546 Write 1 Done New Done false
16994 5548 Acked 6547 Write 10 Done New Done false
16995 5549 Acked 6548 Read 7 Done New Done false
16996 5550 Acked 6549 Read 7 Done New Done false
16997 5551 Acked 6550 Write 2 Done New Done false
16998 5552 Acked 6551 Flush 0 Done New Done false
16999 5553 Acked 6552 Read 2 Done New Done false
17000 5554 Acked 6553 Write 9 Done New Done false
17001 5555 Acked 6554 Read 4 Done New Done false
17002 5556 Acked 6555 Write 9 Done New Done false
17003 5557 Acked 6556 Write 6 Done New Done false
17004 5558 Acked 6557 Write 8 Done New Done false
17005 5559 Acked 6558 Read 2 Done New Done false
17006 5560 Acked 6559 Write 7 Done New Done false
17007 5561 Acked 6560 Read 10 Done New Done false
17008 5562 Acked 6561 Read 9 Done New Done false
17009 5563 Acked 6562 Write 2 Done New Done false
17010 5564 Acked 6563 Write 1 Done New Done false
17011 5565 Acked 6564 Write 3 Done New Done false
17012 5566 Acked 6565 Write 10 Done New Done false
17013 5567 Acked 6566 Read 5 Done New Done false
17014 5568 Acked 6567 Write 3 Done New Done false
17015 5569 Acked 6568 Flush 0 Done New Done false
17016 5570 Acked 6569 Read 2 Done New Done false
17017 5571 Acked 6570 Read 7 Done New Done false
17018 5572 Acked 6571 Read 8 Done New Done false
17019 5573 Acked 6572 Read 3 Done New Done false
17020 5574 Acked 6573 Read 8 Done New Done false
17021 5575 Acked 6574 Read 5 Done New Done false
17022 5576 Acked 6575 Read 10 Done New Done false
17023 5577 Acked 6576 Write 10 Done New Done false
17024 5578 Acked 6577 Read 3 Done New Done false
17025 5579 Acked 6578 Read 7 Done New Done false
17026 5580 Acked 6579 Read 10 Done New Done false
17027 5581 Acked 6580 Read 3 Done New Done false
17028 5582 Acked 6581 Read 5 Done New Done false
17029 5583 Acked 6582 Flush 0 Done New Done false
17030 5584 Acked 6583 Write 5 Done New Done false
17031 5585 Acked 6584 Read 1 Done New Done false
17032 5586 Acked 6585 Flush 0 Done New Done false
17033 5587 Acked 6586 Write 1 Done New Done false
17034 5588 Acked 6587 Read 1 Done New Done false
17035 5589 Acked 6588 Read 8 Done New Done false
17036 5590 Acked 6589 Write 2 Done New Done false
17037 5591 Acked 6590 Write 4 Done New Done false
17038 5592 Acked 6591 Write 6 Done New Done false
17039 5593 Acked 6592 Write 6 Done New Done false
17040 5594 Acked 6593 Read 5 Done New Done false
17041 5595 Acked 6594 Read 10 Done New Done false
17042 5596 Acked 6595 Read 6 Done New Done false
17043 5597 Acked 6596 Read 9 Done New Done false
17044 5598 Acked 6597 Read 9 Done New Done false
17045 5599 Acked 6598 Write 6 Done New Done false
17046 5600 Acked 6599 Write 1 Done New Done false
17047 5601 Acked 6600 Write 3 Done New Done false
17048 5602 Acked 6601 Read 1 Done New Done false
17049 5603 Acked 6602 Write 1 Done New Done false
17050 5604 Acked 6603 Read 8 Done New Done false
17051 5605 Acked 6604 Read 5 Done New Done false
17052 5606 Acked 6605 Write 3 Done New Done false
17053 5607 Acked 6606 Write 9 Done New Done false
17054 5608 Acked 6607 Read 7 Done New Done false
17055 5609 Acked 6608 Write 10 Done New Done false
17056 5610 Acked 6609 Write 3 Done New Done false
17057 5611 Acked 6610 Write 10 Done New Done false
17058 5612 Acked 6611 Write 10 Done New Done false
17059 5613 Acked 6612 Write 3 Done New Done false
17060 5614 Acked 6613 Write 1 Done New Done false
17061 5615 Acked 6614 Write 4 Done New Done false
17062 5616 Acked 6615 Write 7 Done New Done false
17063 5617 Acked 6616 Flush 0 Done New Done false
17064 5618 Acked 6617 Write 9 Done New Done false
17065 5619 Acked 6618 Flush 0 Done New Done false
17066 5620 Acked 6619 Write 3 Done New Done false
17067 5621 Acked 6620 Write 3 Done New Done false
17068 5622 Acked 6621 Write 3 Done New Done false
17069 5623 Acked 6622 Read 9 Done New Done false
17070 5624 Acked 6623 Read 2 Done New Done false
17071 5625 Acked 6624 Write 5 Done New Done false
17072 5626 Acked 6625 Write 6 Done New Done false
17073 5627 Acked 6626 Flush 0 Done New Done false
17074 5628 Acked 6627 Write 2 Done New Done false
17075 5629 Acked 6628 Read 7 Done New Done false
17076 5630 Acked 6629 Read 2 Done New Done false
17077 5631 Acked 6630 Read 4 Done New Done false
17078 5632 Acked 6631 Write 2 Done New Done false
17079 5633 Acked 6632 Read 5 Done New Done false
17080 5634 Acked 6633 Read 3 Done New Done false
17081 5635 Acked 6634 Flush 0 Done New Done false
17082 5636 Acked 6635 Flush 0 Done New Done false
17083 5637 Acked 6636 Write 10 Done New Done false
17084 5638 Acked 6637 Write 4 Done New Done false
17085 5639 Acked 6638 Read 5 Done New Done false
17086 5640 Acked 6639 Read 8 Done New Done false
17087 5641 Acked 6640 Write 2 Done New Done false
17088 5642 Acked 6641 Read 4 Done New Done false
17089 5643 Acked 6642 Read 7 Done New Done false
17090 5644 Acked 6643 Read 3 Done New Done false
17091 5645 Acked 6644 Write 3 Done New Done false
17092 5646 Acked 6645 Read 2 Done New Done false
17093 5647 Acked 6646 Write 7 Done New Done false
17094 5648 Acked 6647 Read 10 Done New Done false
17095 5649 Acked 6648 Write 3 Done New Done false
17096 5650 Acked 6649 Flush 0 Done New Done false
17097 5651 Acked 6650 Write 1 Done New Done false
17098 5652 Acked 6651 Write 8 Done New Done false
17099 5653 Acked 6652 Read 7 Done New Done false
17100 5654 Acked 6653 Write 5 Done New Done false
17101 5655 Acked 6654 Write 6 Done New Done false
17102 5656 Acked 6655 Read 5 Done New Done false
17103 5657 Acked 6656 Write 3 Done New Done false
17104 5658 Acked 6657 Write 8 Done New Done false
17105 5659 Acked 6658 Read 4 Done New Done false
17106 5660 Acked 6659 Write 3 Done New Done false
17107 5661 Acked 6660 Read 1 Done New Done false
17108 5662 Acked 6661 Write 3 Done New Done false
17109 5663 Acked 6662 Read 4 Done New Done false
17110 5664 Acked 6663 Write 10 Done New Done false
17111 5665 Acked 6664 Read 7 Done New Done false
17112 5666 Acked 6665 Read 10 Done New Done false
17113 5667 Acked 6666 Write 5 Done New Done false
17114 5668 Acked 6667 Write 2 Done New Done false
17115 5669 Acked 6668 Write 5 Done New Done false
17116 5670 Acked 6669 Flush 0 Done New Done false
17117 5671 Acked 6670 Read 1 Done New Done false
17118 5672 Acked 6671 Write 9 Done New Done false
17119 5673 Acked 6672 Flush 0 Done New Done false
17120 5674 Acked 6673 Write 2 Done New Done false
17121 5675 Acked 6674 Write 1 Done New Done false
17122 5676 Acked 6675 Read 7 Done New Done false
17123 5677 Acked 6676 Read 1 Done New Done false
17124 5678 Acked 6677 Read 7 Done New Done false
17125 5679 Acked 6678 Read 1 Done New Done false
17126 5680 Acked 6679 Read 10 Done New Done false
17127 5681 Acked 6680 Read 3 Done New Done false
17128 5682 Acked 6681 Write 8 Done New Done false
17129 5683 Acked 6682 Write 6 Done New Done false
17130 5684 Acked 6683 Read 5 Done New Done false
17131 5685 Acked 6684 Read 10 Done New Done false
17132 5686 Acked 6685 Write 4 Done New Done false
17133 5687 Acked 6686 Read 1 Done New Done false
17134 5688 Acked 6687 Read 5 Done New Done false
17135 5689 Acked 6688 Read 6 Done New Done false
17136 5690 Acked 6689 Write 5 Done New Done false
17137 5691 Acked 6690 Write 3 Done New Done false
17138 5692 Acked 6691 Write 6 Done New Done false
17139 5693 Acked 6692 Read 9 Done New Done false
17140 5694 Acked 6693 Write 5 Done New Done false
17141 5695 Acked 6694 Flush 0 Done New Done false
17142 5696 Acked 6695 Write 10 Done New Done false
17143 5697 Acked 6696 Write 1 Done New Done false
17144 5698 Acked 6697 Flush 0 Done New Done false
17145 5699 Acked 6698 Read 4 Done New Done false
17146 5700 Acked 6699 Read 9 Done New Done false
17147 5701 Acked 6700 Read 7 Done New Done false
17148 5702 Acked 6701 Flush 0 Done New Done false
17149 5703 Acked 6702 Write 4 Done New Done false
17150 5704 Acked 6703 Write 5 Done New Done false
17151 5705 Acked 6704 Read 1 Done New Done false
17152 5706 Acked 6705 Read 2 Done New Done false
17153 5707 Acked 6706 Read 2 Done New Done false
17154 5708 Acked 6707 Read 9 Done New Done false
17155 5709 Acked 6708 Read 9 Done New Done false
17156 5710 Acked 6709 Write 3 Done New Done false
17157 5711 Acked 6710 Read 10 Done New Done false
17158 5712 Acked 6711 Read 1 Done New Done false
17159 5713 Acked 6712 Read 5 Done New Done false
17160 5714 Acked 6713 Read 2 Done New Done false
17161 5715 Acked 6714 Write 1 Done New Done false
17162 5716 Acked 6715 Write 1 Done New Done false
17163 5717 Acked 6716 Write 3 Done New Done false
17164 5718 Acked 6717 Read 9 Done New Done false
17165 5719 Acked 6718 Read 2 Done New Done false
17166 5720 Acked 6719 Flush 0 Done New Done false
17167 5721 Acked 6720 Read 7 Done New Done false
17168 5722 Acked 6721 Write 9 Done New Done false
17169 5723 Acked 6722 Write 8 Done New Done false
17170 5724 Acked 6723 Write 3 Done New Done false
17171 5725 Acked 6724 Write 8 Sent New Sent false
17172 5726 Acked 6725 Write 1 Sent New Sent false
17173 STATES DS:0 DS:1 DS:2 TOTAL
17174 New 0 300 0 300
17175 Sent 2 0 2 4
17176 Done 298 0 298 596
17177 Skipped 0 0 0 0
17178 Error 0 0 0 0
17179 Last Flush: 6719 6425 6719
17180 Downstairs last five completed: 6425 6424 6423 6422 6421
17181 Upstairs last five completed: 5726 5725 5724 5723 5722
17182 CLIENT: Up:0 ds:300 act:2
171832024-02-12T01:37:53.180ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
171842024-02-12T01:37:53.180ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
171852024-02-12T01:37:53.180ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
171862024-02-12T01:37:53.180ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
171872024-02-12T01:37:53.180ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
171882024-02-12T01:37:53.180ZINFOcrucible: [1] client re-new 301 jobs since flush 6425 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
171892024-02-12T01:37:53.181ZINFOcrucible: client task is exiting = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
171902024-02-12T01:37:54.181ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
171912024-02-12T01:37:54.182ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
171922024-02-12T01:37:54.182ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
171932024-02-12T01:37:54.182ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
171942024-02-12T01:37:54.182ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
171952024-02-12T01:37:54.182ZINFOcrucible: [1] client re-new 301 jobs since flush 6425 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
171962024-02-12T01:37:54.182ZINFOcrucible: client task is exiting = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
17197 [8820] got start action
17198 [8820] I am going to restart
17199 Starting downstairs at port 8820
17200 Make output file at "/tmp/dsc/downstairs-8820.txt"
17201 [8820][1] reports Starting
17202 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1183)
17203 [8820][1] reports Running
172042024-02-12T01:37:55.184ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
172052024-02-12T01:37:55.184ZINFOcrucible: ds_connection connected = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
172062024-02-12T01:37:55.185ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
172072024-02-12T01:37:55.185ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
172082024-02-12T01:37:55.185ZINFOcrucible: send last flush ID to this DS: 6425 = downstairs client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
172092024-02-12T01:37:55.186ZINFOcrucible: Replied this last flush ID: 6425 = downstairs client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
172102024-02-12T01:37:55.186ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
172112024-02-12T01:37:55.186ZINFOcrucible: [1] Transition from Offline to Replay = downstairs client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
172122024-02-12T01:37:55.186ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
172132024-02-12T01:37:55.186ZINFOcrucible: [1] Transition from Replay to Active = downstairs client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
17214 ----------------------------------------------------------------
17215 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
17216 Downstairs last five completed: 6726 6725 6724 6723 6722
17217 Upstairs last five completed: 5727 5726 5725 5724 5723
17218 CLIENT: Up:0 ds:0 act:3
17219 Replay: All jobs finished, all DS active.
17220 Main task has work to do, go find it
17221 got dsc Stop(0)
17222 stop 0
17223 Feb 12 01:37:56.520 INFO request completed[8810] Got stop action so:false kr:false
17224 , latency_us: 373, response_code: 204, uri: /stop/cid/0, method: POST, req_id: cdd61ab4-4171-464e-a84e-6a30336dcbb8, remote_addr: 127.0.0.1:60456, local_addr: 127.0.0.1:9998
17225 Feb 12 01:37:56.521 INFO request completed, latency_us: 119, response_code: 200, uri: /state/cid/0, method: GET, req_id: a7b0e448-c0f8-4f5f-9556-88d99ace0e78, remote_addr:{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2024-02-12T01:37:56.521445862Z","hostname":"ip-10-150-1-104.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"e1773b3e-b36f-4080-8b13-c0830f0d072e"}
17226 127.0.0.1:60456, local_addr: 127.0.0.1:9998{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2024-02-12T01:37:56.521520312Z","hostname":"ip-10-150-1-104.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"e1773b3e-b36f-4080-8b13-c0830f0d072e"}
17227 
172282024-02-12T01:37:56.521ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
172292024-02-12T01:37:56.521ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
172302024-02-12T01:37:56.521ZINFOcrucible: [0] client re-new 0 jobs since flush 6726 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
172312024-02-12T01:37:56.521ZINFOcrucible: client task is exiting = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
17232 [8810] Exited with: ExitStatus(unix_wait_status(9))
17233 [8810][0] reports Exit
172342024-02-12T01:37:57.522ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
172352024-02-12T01:37:57.522ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
172362024-02-12T01:37:57.522ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
172372024-02-12T01:37:57.522ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
172382024-02-12T01:37:57.522ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
172392024-02-12T01:37:57.522ZINFOcrucible: [0] client re-new 0 jobs since flush 6726 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
172402024-02-12T01:37:57.522ZINFOcrucible: client task is exiting = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
172412024-02-12T01:37:58.523ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
172422024-02-12T01:37:58.523ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
172432024-02-12T01:37:58.523ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
172442024-02-12T01:37:58.523ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
172452024-02-12T01:37:58.523ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
172462024-02-12T01:37:58.523ZINFOcrucible: [0] client re-new 0 jobs since flush 6726 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
172472024-02-12T01:37:58.523ZINFOcrucible: client task is exiting = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
172482024-02-12T01:37:59.524ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
172492024-02-12T01:37:59.524ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
172502024-02-12T01:37:59.524ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
172512024-02-12T01:37:59.524ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
172522024-02-12T01:37:59.525ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
172532024-02-12T01:37:59.525ZINFOcrucible: [0] client re-new 0 jobs since flush 6726 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
172542024-02-12T01:37:59.525ZINFOcrucible: client task is exiting = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
17255 Feb 12 01:38:00.523 INFO request completed, latency_us: 227, response_code: 200, uri: /state/cid/0, method: GET, req_id: 62ffc920-b14f-4cae-ad34-2d03b7d9247a, remote_addr: 127.0.0.1:60456, local_addr: 127.0.0.1:9998
17256 001/300 Write block 2883 len 16384 data: 4 4 5 4
17257 002/300 Write block 4677 len 4096 data: 8
17258 003/300 Write block 3712 len 4096 data: 3
17259 004/300 Read block 3381 len 32768
172602024-02-12T01:38:00.526ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
172612024-02-12T01:38:00.526ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
172622024-02-12T01:38:00.526ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
172632024-02-12T01:38:00.526ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
172642024-02-12T01:38:00.526ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
172652024-02-12T01:38:00.526ZINFOcrucible: [0] client re-new 4 jobs since flush 6726 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
172662024-02-12T01:38:00.526ZINFOcrucible: client task is exiting = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
17267 005/300 Read block 199 len 4096
17268 006/300 Write block 4497 len 32768 data: 4 4 4 5 6 5 6 6
17269 007/300 Write block 3668 len 12288 data: 2 3 3
17270 008/300 Write block 4577 len 4096 data: 6
17271 009/300 Read block 42 len 32768
17272 010/300 Write block 3817 len 32768 data: 3 3 4 3 3 3 2 3
17273 011/300 Read block 3736 len 36864
17274 012/300 Write block 2370 len 16384 data: 2 2 3 3
17275 013/300 Write block 3976 len 24576 data: 3 3 2 4 5 6
17276 014/300 Write block 2662 len 36864 data: 3 4 4 4 5 4 3 3 3
17277 015/300 Read block 3692 len 32768
17278 016/300 Read block 4870 len 8192
17279 017/300 Write block 2233 len 36864 data: 5 5 5 4 5 6 6 5 5
17280 018/300 Read block 3116 len 40960
17281 019/300 Read block 4763 len 8192
17282 020/300 Read block 4203 len 8192
17283 021/300 Read block 2899 len 32768
17284 022/300 Write block 3400 len 32768 data: 8 8 6 6 7 4 4 3
17285 023/300 Write block 3506 len 32768 data: 3 4 4 4 4 5 4 5
17286 024/300 Write block 1640 len 24576 data: 5 4 3 3 3 4
17287 025/300 Write block 1519 len 24576 data: 3 3 2 2 2 2
17288 026/300 Read block 2775 len 24576
17289 027/300 Read block 4895 len 24576
17290 028/300 Write block 3004 len 40960 data: 4 4 4 4 3 4 4 4 4 3
17291 029/300 Read block 35 len 28672
17292 030/300 Write block 2404 len 40960 data: 6 6 7 5 4 3 3 3 3 3
17293 031/300 Read block 2390 len 12288
17294 032/300 Read block 95 len 24576
17295 033/300 Write block 884 len 12288 data: 6 4 4
17296 034/300 Write block 683 len 24576 data: 3 5 5 5 4 4
17297 035/300 Flush
17298 036/300 Write block 3046 len 12288 data: 2 2 3
17299 037/300 Write block 1072 len 36864 data: 5 5 5 4 4 5 7 5 6
17300 038/300 Read block 3190 len 12288
17301 039/300 Write block 1590 len 28672 data: 3 4 7 6 5 5 5
17302 040/300 Read block 1097 len 40960
17303 041/300 Read block 387 len 8192
17304 042/300 Read block 3699 len 12288
17305 043/300 Read block 0 len 12288
17306 044/300 Read block 4992 len 16384
17307 045/300 Write block 10 len 36864 data: 4 5 4 3 3 4 4 5 7
17308 046/300 Read block 2145 len 28672
17309 047/300 Write block 102 len 32768 data: 8 9 10 9 8 9 7 6
17310 048/300 Read block 3285 len 32768
17311 049/300 Write block 1197 len 36864 data: 5 4 4 4 5 5 6 6 7
17312 050/300 Write block 3041 len 40960 data: 6 6 3 3 2 3 3 4 2 2
17313 051/300 Read block 1265 len 16384
17314 052/300 Read block 2454 len 4096
17315 053/300 Write block 4625 len 32768 data: 4 2 4 4 4 4 4 4
17316 054/300 Read block 677 len 40960
17317 055/300 Write block 4312 len 16384 data: 3 4 4 5
17318 056/300 Read block 3912 len 16384
17319 057/300 Read block 167 len 24576
17320 058/300 Write block 1055 len 12288 data: 3 2 2
17321 059/300 Read block 2856 len 12288
17322 060/300 Write block 433 len 36864 data: 3 3 4 3 2 2 2 5 5
17323 061/300 Read block 3241 len 36864
17324 062/300 Write block 4620 len 20480 data: 2 4 4 4 4
17325 063/300 Write block 3610 len 32768 data: 2 2 2 2 3 5 4 6
17326 064/300 Read block 207 len 20480
17327 065/300 Write block 3953 len 16384 data: 5 6 5 5
17328 066/300 Read block 2635 len 32768
17329 067/300 Write block 402 len 24576 data: 5 4 5 4 6 5
17330 068/300 Read block 4829 len 8192
17331 069/300 Read block 429 len 32768
17332 070/300 Read block 36 len 4096
17333 071/300 Read block 3415 len 8192
17334 072/300 Read block 2944 len 20480
17335 073/300 Write block 135 len 4096 data: 2
17336 074/300 Read block 4339 len 8192
17337 075/300 Flush
17338 076/300 Read block 2893 len 4096
17339 077/300 Read block 3 len 32768
17340 078/300 Read block 3868 len 12288
17341 079/300 Read block 4721 len 32768
17342 080/300 Read block 1697 len 12288
17343 081/300 Flush
17344 082/300 Write block 2851 len 4096 data: 7
17345 083/300 Write block 4136 len 16384 data: 5 5 6 8
17346 084/300 Read block 4798 len 12288
17347 085/300 Flush
17348 086/300 Write block 2069 len 20480 data: 5 5 6 5 4
17349 087/300 Read block 2610 len 36864
17350 088/300 Read block 1676 len 32768
17351 089/300 Read block 3618 len 40960
17352 090/300 Write block 4217 len 8192 data: 4 6
17353 091/300 Write block 1020 len 16384 data: 5 7 7 6
17354 092/300 Read block 4961 len 20480
17355 093/300 Write block 3786 len 32768 data: 3 2 2 2 4 5 5 6
17356 094/300 Write block 4498 len 4096 data: 5
17357 095/300 Read block 1084 len 16384
17358 096/300 Write block 2753 len 16384 data: 5 6 5 6
17359 097/300 Read block 3126 len 28672
17360 098/300 Read block 11 len 28672
17361 099/300 Read block 3944 len 4096
17362 100/300 Read block 792 len 12288
17363 101/300 Write block 3992 len 8192 data: 3 3
17364 102/300 Read block 1316 len 36864
17365 103/300 Write block 264 len 28672 data: 4 4 3 3 3 3 4
17366 104/300 Read block 4378 len 36864
17367 105/300 Read block 329 len 28672
17368 106/300 Write block 4171 len 40960 data: 6 7 9 10 6 6 5 5 6 7
17369 107/300 Read block 928 len 36864
17370 108/300 Read block 2776 len 28672
17371 109/300 Write block 2629 len 36864 data: 6 4 3 3 3 4 4 4 3
17372 110/300 Write block 738 len 12288 data: 2 3 3
17373 111/300 Read block 3736 len 8192
17374 112/300 Write block 4818 len 32768 data: 4 4 4 3 3 3 4 5
17375 113/300 Read block 4481 len 40960
17376 114/300 Flush
17377 115/300 Read block 742 len 32768
17378 116/300 Flush
17379 117/300 Write block 1564 len 4096 data: 7
17380 118/300 Read block 3634 len 8192
17381 119/300 Flush
17382 120/300 Read block 260 len 8192
17383 121/300 Read block 1267 len 40960
17384 122/300 Read block 3083 len 36864
17385 123/300 Write block 3156 len 4096 data: 3
17386 124/300 Read block 2627 len 32768
17387 125/300 Read block 1858 len 32768
17388 126/300 Write block 2970 len 32768 data: 3 3 3 5 6 5 4 4
17389 127/300 Write block 1493 len 40960 data: 3 3 3 3 3 3 5 4 4 4
17390 128/300 Write block 3491 len 40960 data: 6 7 7 4 5 6 4 4 4 4
17391 129/300 Flush
17392 130/300 Read block 4188 len 12288
17393 131/300 Write block 3162 len 36864 data: 5 6 5 7 8 7 7 7 6
17394 132/300 Read block 1675 len 36864
17395 133/300 Read block 965 len 40960
17396 134/300 Write block 2969 len 12288 data: 3 4 4
17397 135/300 Read block 3245 len 24576
17398 136/300 Write block 3223 len 36864 data: 5 9 9 9 9 10 9 8 8
17399 137/300 Write block 240 len 12288 data: 3 5 6
17400 138/300 Read block 416 len 24576
17401 139/300 Read block 558 len 24576
17402 140/300 Read block 3763 len 36864
17403 141/300 Read block 3498 len 20480
17404 142/300 Read block 56 len 16384
17405 143/300 Write block 3486 len 8192 data: 6 6
17406 144/300 Write block 4940 len 20480 data: 4 3 3 3 4
17407 145/300 Write block 3589 len 40960 data: 3 3 4 4 4 4 6 5 4 3
17408 146/300 Read block 4583 len 20480
17409 147/300 Write block 1330 len 16384 data: 7 6 5 6
17410 148/300 Write block 2898 len 32768 data: 8 6 5 9 7 5 4 4
17411 149/300 Write block 3055 len 36864 data: 3 4 3 3 3 3 3 3 3
17412 150/300 Write block 2095 len 4096 data: 5
17413 151/300 Read block 4125 len 20480
17414 152/300 Write block 4627 len 24576 data: 5 5 5 5 5 5
17415 153/300 Flush
17416 154/300 Flush
17417 155/300 Read block 4798 len 32768
17418 156/300 Write block 1237 len 4096 data: 3
17419 157/300 Read block 4638 len 16384
17420 158/300 Write block 2927 len 20480 data: 6 6 7 8 6
17421 159/300 Write block 3699 len 28672 data: 6 6 6 5 5 4 4
17422 160/300 Write block 2982 len 8192 data: 4 4
17423 161/300 Write block 4899 len 24576 data: 4 4 5 5 7 8
17424 162/300 Read block 959 len 32768
17425 163/300 Write block 1918 len 24576 data: 6 6 6 4 4 5
17426 164/300 Read block 4977 len 8192
17427 165/300 Write block 3430 len 16384 data: 7 6 5 6
17428 166/300 Flush
17429 167/300 Read block 3957 len 32768
17430 168/300 Read block 3835 len 28672
17431 169/300 Read block 4124 len 24576
17432 170/300 Read block 3043 len 32768
17433 171/300 Read block 3077 len 12288
17434 172/300 Read block 1583 len 8192
17435 173/300 Write block 1146 len 12288 data: 8 8 8
17436 174/300 Write block 3760 len 32768 data: 5 3 3 2 3 4 4 3
17437 175/300 Read block 1016 len 4096
17438 176/300 Write block 1912 len 12288 data: 4 4 4
17439 177/300 Write block 4073 len 20480 data: 8 6 6 6 6
17440 178/300 Write block 1124 len 20480 data: 6 4 4 4 5
17441 179/300 Write block 1544 len 4096 data: 8
17442 180/300 Write block 180 len 24576 data: 4 3 2 2 2 3
17443 181/300 Read block 2004 len 8192
17444 182/300 Read block 1524 len 28672
17445 183/300 Read block 1196 len 24576
17446 184/300 Read block 148 len 12288
17447 185/300 Write block 267 len 24576 data: 4 4 4 5 5 4
17448 186/300 Flush
17449 187/300 Read block 4894 len 24576
17450 188/300 Read block 4125 len 8192
17451 189/300 Read block 1589 len 12288
17452 190/300 Read block 230 len 40960
17453 191/300 Read block 4568 len 28672
17454 192/300 Write block 3718 len 12288 data: 5 4 3
17455 193/300 Read block 2869 len 12288
17456 194/300 Write block 3478 len 32768 data: 5 6 5 5 5 5 6 6
17457 195/300 Write block 443 len 40960 data: 5 6 6 7 5 5 6 4 4 4
17458 196/300 Read block 4126 len 28672
17459 197/300 Write block 4004 len 36864 data: 4 4 4 4 3 3 3 2 3
17460 198/300 Write block 4673 len 24576 data: 7 7 7 8 9 8
17461 199/300 Read block 1707 len 8192
17462 200/300 Write block 3332 len 24576 data: 3 3 2 3 3 3
17463 201/300 Read block 4315 len 12288
17464 202/300 Read block 4886 len 32768
17465 203/300 Read block 1029 len 8192
17466 204/300 Read block 1606 len 36864
17467 205/300 Write block 4016 len 36864 data: 4 2 2 4 4 7 10 8 7
17468 206/300 Write block 4693 len 36864 data: 6 5 4 6 5 5 4 4 5
17469 207/300 Read block 2062 len 32768
17470 208/300 Write block 809 len 20480 data: 5 3 2 3 3
17471 209/300 Read block 1598 len 16384
17472 210/300 Read block 2639 len 4096
17473 211/300 Flush
17474 212/300 Read block 2148 len 24576
17475 213/300 Read block 3324 len 28672
17476 214/300 Read block 2076 len 4096
17477 215/300 Read block 714 len 16384
17478 216/300 Write block 3907 len 24576 data: 4 4 2 2 2 3
17479 217/300 Write block 968 len 12288 data: 5 6 6
17480 218/300 Read block 2856 len 28672
17481 219/300 Flush
17482 220/300 Read block 1709 len 32768
17483 221/300 Read block 3380 len 16384
17484 222/300 Read block 1063 len 40960
17485 223/300 Write block 1086 len 28672 data: 6 6 6 5 5 6 6
17486 224/300 Write block 3856 len 24576 data: 4 2 2 2 3 4
17487 225/300 Write block 1594 len 20480 data: 6 6 6 5 4
17488 226/300 Read block 2326 len 20480
17489 227/300 Write block 2634 len 20480 data: 5 5 5 4 3
17490 228/300 Flush
17491 229/300 Read block 819 len 20480
17492 230/300 Write block 176 len 20480 data: 4 4 4 4 5
17493 231/300 Read block 722 len 12288
17494 232/300 Flush
17495 233/300 Write block 386 len 4096 data: 7
17496 234/300 Write block 1340 len 8192 data: 6 4
17497 235/300 Write block 4271 len 16384 data: 6 7 7 5
17498 236/300 Write block 2574 len 12288 data: 3 3 3
17499 237/300 Read block 2288 len 24576
17500 238/300 Write block 1035 len 32768 data: 6 5 4 3 2 3 3 3
17501 239/300 Read block 1112 len 16384
17502 240/300 Write block 3891 len 36864 data: 4 4 4 5 5 6 6 5 6
17503 241/300 Write block 1000 len 32768 data: 6 5 5 5 5 4 4 3
17504 242/300 Write block 4411 len 4096 data: 4
17505 243/300 Write block 737 len 20480 data: 2 3 4 4 5
17506 244/300 Write block 4822 len 12288 data: 4 4 5
17507 245/300 Write block 2759 len 24576 data: 7 8 8 7 7 5
17508 246/300 Write block 1393 len 12288 data: 6 7 7
17509 247/300 Read block 1274 len 16384
17510 248/300 Write block 3555 len 8192 data: 2 3
17511 249/300 Write block 4988 len 28672 data: 3 3 3 3 4 4 3
17512 250/300 Read block 1290 len 24576
17513 251/300 Read block 2579 len 20480
17514 252/300 Write block 3447 len 28672 data: 7 7 8 11 11 10 8
17515 253/300 Read block 1565 len 40960
17516 254/300 Flush
17517 255/300 Read block 3626 len 12288
17518 256/300 Flush
17519 257/300 Write block 4839 len 20480 data: 4 4 4 3 4
17520 258/300 Read block 3488 len 28672
17521 259/300 Write block 1472 len 16384 data: 3 4 4 4
17522 260/300 Write block 2211 len 36864 data: 7 6 8 7 8 8 8 6 6
17523 261/300 Write block 4507 len 32768 data: 6 5 6 5 5 4 3 4
17524 262/300 Read block 4671 len 16384
17525 263/300 Read block 4174 len 24576
17526 264/300 Flush
17527 265/300 Read block 3432 len 32768
17528 266/300 Write block 3486 len 16384 data: 7 7 6 5
17529 267/300 Read block 2424 len 16384
17530 268/300 Write block 4798 len 20480 data: 4 3 4 4 3
17531 269/300 Read block 1345 len 12288
17532 270/300 Read block 1349 len 16384
17533 271/300 Read block 1252 len 36864
17534 272/300 Read block 4290 len 4096
17535 273/300 Read block 2141 len 20480
17536 274/300 Read block 662 len 40960
17537 275/300 Read block 350 len 24576
17538 276/300 Write block 3527 len 20480 data: 3 3 2 3 3
17539 277/300 Write block 345 len 8192 data: 3 3
17540 278/300 Read block 1182 len 20480
17541 279/300 Write block 1727 len 12288 data: 8 7 8
17542 280/300 Read block 558 len 16384
17543 281/300 Flush
17544 282/300 Read block 1537 len 8192
17545 283/300 Read block 1938 len 16384
17546 284/300 Read block 3604 len 32768
17547 285/300 Read block 2024 len 40960
17548 286/300 Flush
17549 287/300 Read block 3878 len 24576
17550 288/300 Read block 4042 len 16384
17551 289/300 Flush
17552 290/300 Flush
17553 291/300 Read block 1827 len 4096
17554 292/300 Write block 98 len 8192 data: 4 7
17555 293/300 Write block 1648 len 24576 data: 2 2 2 3 3 3
17556 294/300 Write block 4310 len 20480 data: 3 3 4 5 5
17557 295/300 Read block 1640 len 16384
17558 296/300 Write block 4458 len 8192 data: 5 5
17559 297/300 Write block 3485 len 32768 data: 7 8 8 7 6 6 7 8
17560 298/300 Read block 4111 len 12288
17561 299/300 Write block 4835 len 36864 data: 4 4 5 5 5 5 5 4 5
17562 300/300 Read block 1601 len 4096
17563 Main task has work to do, go find it
17564 got dsc Start(0)
17565 start 0
17566 Feb 12 01:38:00.869 INFO request completed, latency_us: 212, response_code: 204, uri: /start/cid/0, method: POST, req_id: 643a746c-e7f2-46e3-85d2-2ace56a6f358, remote_addr: 127.0.0.1:60456, local_addr: 127.0.0.1:9998
17567 Replay: started 0, returned:Ok(())
17568 ----------------------------------------------------------------
17569 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
17570 States: Offline Active Active
17571 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
17572 5728 Acked 6727 Write 4 New Done Done true
17573 5729 Acked 6728 Write 1 New Done Done true
17574 5730 Acked 6729 Write 1 New Done Done true
17575 5731 Acked 6730 Read 8 New Done Done true
17576 5732 Acked 6731 Read 1 New Done Done false
17577 5733 Acked 6732 Write 8 New Done Done false
17578 5734 Acked 6733 Write 3 New Done Done false
17579 5735 Acked 6734 Write 1 New Done Done false
17580 5736 Acked 6735 Read 8 New Done Done false
17581 5737 Acked 6736 Write 8 New Done Done false
17582 5738 Acked 6737 Read 9 New Done Done false
17583 5739 Acked 6738 Write 4 New Done Done false
17584 5740 Acked 6739 Write 6 New Done Done false
17585 5741 Acked 6740 Write 9 New Done Done false
17586 5742 Acked 6741 Read 8 New Done Done false
17587 5743 Acked 6742 Read 2 New Done Done false
17588 5744 Acked 6743 Write 9 New Done Done false
17589 5745 Acked 6744 Read 10 New Done Done false
17590 5746 Acked 6745 Read 2 New Done Done false
17591 5747 Acked 6746 Read 2 New Done Done false
17592 5748 Acked 6747 Read 8 New Done Done false
17593 5749 Acked 6748 Write 8 New Done Done false
17594 5750 Acked 6749 Write 8 New Done Done false
17595 5751 Acked 6750 Write 6 New Done Done false
17596 5752 Acked 6751 Write 6 New Done Done false
17597 5753 Acked 6752 Read 6 New Done Done false
17598 5754 Acked 6753 Read 6 New Done Done false
17599 5755 Acked 6754 Write 10 New Done Done false
17600 5756 Acked 6755 Read 7 New Done Done false
17601 5757 Acked 6756 Write 10 New Done Done false
17602 5758 Acked 6757 Read 3 New Done Done false
17603 5759 Acked 6758 Read 6 New Done Done false
17604 5760 Acked 6759 Write 3 New Done Done false
17605 5761 Acked 6760 Write 6 New Done Done false
17606 5762 Acked 6761 Flush 0 New Done Done false
17607 5763 Acked 6762 Write 3 New Done Done false
17608 5764 Acked 6763 Write 9 New Done Done false
17609 5765 Acked 6764 Read 3 New Done Done false
17610 5766 Acked 6765 Write 7 New Done Done false
17611 5767 Acked 6766 Read 10 New Done Done false
17612 5768 Acked 6767 Read 2 New Done Done false
17613 5769 Acked 6768 Read 3 New Done Done false
17614 5770 Acked 6769 Read 3 New Done Done false
17615 5771 Acked 6770 Read 4 New Done Done false
17616 5772 Acked 6771 Write 9 New Done Done false
17617 5773 Acked 6772 Read 7 New Done Done false
17618 5774 Acked 6773 Write 8 New Done Done false
17619 5775 Acked 6774 Read 8 New Done Done false
17620 5776 Acked 6775 Write 9 New Done Done false
17621 5777 Acked 6776 Write 10 New Done Done false
17622 5778 Acked 6777 Read 4 New Done Done false
17623 5779 Acked 6778 Read 1 New Done Done false
17624 5780 Acked 6779 Write 8 New Done Done false
17625 5781 Acked 6780 Read 10 New Done Done false
17626 5782 Acked 6781 Write 4 New Done Done false
17627 5783 Acked 6782 Read 4 New Done Done false
17628 5784 Acked 6783 Read 6 New Done Done false
17629 5785 Acked 6784 Write 3 New Done Done false
17630 5786 Acked 6785 Read 3 New Done Done false
17631 5787 Acked 6786 Write 9 New Done Done false
17632 5788 Acked 6787 Read 9 New Done Done false
17633 5789 Acked 6788 Write 5 New Done Done false
17634 5790 Acked 6789 Write 8 New Done Done false
17635 5791 Acked 6790 Read 5 New Done Done false
17636 5792 Acked 6791 Write 4 New Done Done false
17637 5793 Acked 6792 Read 8 New Done Done false
17638 5794 Acked 6793 Write 6 New Done Done false
17639 5795 Acked 6794 Read 2 New Done Done false
17640 5796 Acked 6795 Read 8 New Done Done false
17641 5797 Acked 6796 Read 1 New Done Done false
17642 5798 Acked 6797 Read 2 New Done Done false
17643 5799 Acked 6798 Read 5 New Done Done false
17644 5800 Acked 6799 Write 1 New Done Done false
17645 5801 Acked 6800 Read 2 New Done Done false
17646 5802 Acked 6801 Flush 0 New Done Done false
17647 5803 Acked 6802 Read 1 New Done Done false
17648 5804 Acked 6803 Read 8 New Done Done false
17649 5805 Acked 6804 Read 3 New Done Done false
17650 5806 Acked 6805 Read 8 New Done Done false
17651 5807 Acked 6806 Read 3 New Done Done false
17652 5808 Acked 6807 Flush 0 New Done Done false
17653 5809 Acked 6808 Write 1 New Done Done false
17654 5810 Acked 6809 Write 4 New Done Done false
17655 5811 Acked 6810 Read 3 New Done Done false
17656 5812 Acked 6811 Flush 0 New Done Done false
17657 5813 Acked 6812 Write 5 New Done Done false
17658 5814 Acked 6813 Read 9 New Done Done false
17659 5815 Acked 6814 Read 8 New Done Done false
17660 5816 Acked 6815 Read 10 New Done Done false
17661 5817 Acked 6816 Write 2 New Done Done false
17662 5818 Acked 6817 Write 4 New Done Done false
17663 5819 Acked 6818 Read 5 New Done Done false
17664 5820 Acked 6819 Write 8 New Done Done false
17665 5821 Acked 6820 Write 1 New Done Done false
17666 5822 Acked 6821 Read 4 New Done Done false
17667 5823 Acked 6822 Write 4 New Done Done false
17668 5824 Acked 6823 Read 7 New Done Done false
17669 5825 Acked 6824 Flush 0 New Done Done false
17670 5826 Acked 6825 Read 7 New Done Done false
17671 5827 Acked 6826 Read 1 New Done Done false
17672 5828 Acked 6827 Read 3 New Done Done false
17673 5829 Acked 6828 Write 2 New Done Done false
17674 5830 Acked 6829 Read 9 New Done Done false
17675 5831 Acked 6830 Write 7 New Done Done false
17676 5832 Acked 6831 Read 9 New Done Done false
17677 5833 Acked 6832 Read 7 New Done Done false
17678 5834 Acked 6833 Write 10 New Done Done false
17679 5835 Acked 6834 Read 9 New Done Done false
17680 5836 Acked 6835 Read 7 New Done Done false
17681 5837 Acked 6836 Write 9 New Done Done false
17682 5838 Acked 6837 Write 3 New Done Done false
17683 5839 Acked 6838 Read 2 New Done Done false
17684 5840 Acked 6839 Write 8 New Done Done false
17685 5841 Acked 6840 Read 10 New Done Done false
17686 5842 Acked 6841 Flush 0 New Done Done false
17687 5843 Acked 6842 Read 8 New Done Done false
17688 5844 Acked 6843 Flush 0 New Done Done false
17689 5845 Acked 6844 Write 1 New Done Done false
17690 5846 Acked 6845 Read 2 New Done Done false
17691 5847 Acked 6846 Flush 0 New Done Done false
17692 5848 Acked 6847 Read 2 New Done Done false
17693 5849 Acked 6848 Read 10 New Done Done false
17694 5850 Acked 6849 Read 9 New Done Done false
17695 5851 Acked 6850 Write 1 New Done Done false
17696 5852 Acked 6851 Read 8 New Done Done false
17697 5853 Acked 6852 Read 8 New Done Done false
17698 5854 Acked 6853 Write 8 New Done Done false
17699 5855 Acked 6854 Write 10 New Done Done false
17700 5856 Acked 6855 Write 10 New Done Done false
17701 5857 Acked 6856 Flush 0 New Done Done false
17702 5858 Acked 6857 Read 3 New Done Done false
17703 5859 Acked 6858 Write 9 New Done Done false
17704 5860 Acked 6859 Read 9 New Done Done false
17705 5861 Acked 6860 Read 10 New Done Done false
17706 5862 Acked 6861 Write 3 New Done Done false
17707 5863 Acked 6862 Read 6 New Done Done false
17708 5864 Acked 6863 Write 9 New Done Done false
17709 5865 Acked 6864 Write 3 New Done Done false
17710 5866 Acked 6865 Read 6 New Done Done false
17711 5867 Acked 6866 Read 6 New Done Done false
17712 5868 Acked 6867 Read 9 New Done Done false
17713 5869 Acked 6868 Read 5 New Done Done false
17714 5870 Acked 6869 Read 4 New Done Done false
17715 5871 Acked 6870 Write 2 New Done Done false
17716 5872 Acked 6871 Write 5 New Done Done false
17717 5873 Acked 6872 Write 10 New Done Done false
17718 5874 Acked 6873 Read 5 New Done Done false
17719 5875 Acked 6874 Write 4 New Done Done false
17720 5876 Acked 6875 Write 8 New Done Done false
17721 5877 Acked 6876 Write 9 New Done Done false
17722 5878 Acked 6877 Write 1 New Done Done false
17723 5879 Acked 6878 Read 5 New Done Done false
17724 5880 Acked 6879 Write 6 New Done Done false
17725 5881 Acked 6880 Flush 0 New Done Done false
17726 5882 Acked 6881 Flush 0 New Done Done false
17727 5883 Acked 6882 Read 8 New Done Done false
17728 5884 Acked 6883 Write 1 New Done Done false
17729 5885 Acked 6884 Read 4 New Done Done false
17730 5886 Acked 6885 Write 5 New Done Done false
17731 5887 Acked 6886 Write 7 New Done Done false
17732 5888 Acked 6887 Write 2 New Done Done false
17733 5889 Acked 6888 Write 6 New Done Done false
17734 5890 Acked 6889 Read 8 New Done Done false
17735 5891 Acked 6890 Write 6 New Done Done false
17736 5892 Acked 6891 Read 2 New Done Done false
17737 5893 Acked 6892 Write 4 New Done Done false
17738 5894 Acked 6893 Flush 0 New Done Done false
17739 5895 Acked 6894 Read 8 New Done Done false
17740 5896 Acked 6895 Read 7 New Done Done false
17741 5897 Acked 6896 Read 6 New Done Done false
17742 5898 Acked 6897 Read 8 New Done Done false
17743 5899 Acked 6898 Read 3 New Done Done false
17744 5900 Acked 6899 Read 2 New Done Done false
17745 5901 Acked 6900 Write 3 New Done Done false
17746 5902 Acked 6901 Write 8 New Done Done false
17747 5903 Acked 6902 Read 1 New Done Done false
17748 5904 Acked 6903 Write 3 New Done Done false
17749 5905 Acked 6904 Write 5 New Done Done false
17750 5906 Acked 6905 Write 5 New Done Done false
17751 5907 Acked 6906 Write 1 New Done Done false
17752 5908 Acked 6907 Write 6 New Done Done false
17753 5909 Acked 6908 Read 2 New Done Done false
17754 5910 Acked 6909 Read 7 New Done Done false
17755 5911 Acked 6910 Read 6 New Done Done false
17756 5912 Acked 6911 Read 3 New Done Done false
17757 5913 Acked 6912 Write 6 New Done Done false
17758 5914 Acked 6913 Flush 0 New Done Done false
17759 5915 Acked 6914 Read 6 New Done Done false
17760 5916 Acked 6915 Read 2 New Done Done false
17761 5917 Acked 6916 Read 3 New Done Done false
17762 5918 Acked 6917 Read 10 New Done Done false
17763 5919 Acked 6918 Read 7 New Done Done false
17764 5920 Acked 6919 Write 3 New Done Done false
17765 5921 Acked 6920 Read 3 New Done Done false
17766 5922 Acked 6921 Write 8 New Done Done false
17767 5923 Acked 6922 Write 10 New Done Done false
17768 5924 Acked 6923 Read 7 New Done Done false
17769 5925 Acked 6924 Write 9 New Done Done false
17770 5926 Acked 6925 Write 6 New Done Done false
17771 5927 Acked 6926 Read 2 New Done Done false
17772 5928 Acked 6927 Write 6 New Done Done false
17773 5929 Acked 6928 Read 3 New Done Done false
17774 5930 Acked 6929 Read 8 New Done Done false
17775 5931 Acked 6930 Read 2 New Done Done false
17776 5932 Acked 6931 Read 9 New Done Done false
17777 5933 Acked 6932 Write 9 New Done Done false
17778 5934 Acked 6933 Write 9 New Done Done false
17779 5935 Acked 6934 Read 8 New Done Done false
17780 5936 Acked 6935 Write 5 New Done Done false
17781 5937 Acked 6936 Read 4 New Done Done false
17782 5938 Acked 6937 Read 1 New Done Done false
17783 5939 Acked 6938 Flush 0 New Done Done false
17784 5940 Acked 6939 Read 6 New Done Done false
17785 5941 Acked 6940 Read 7 New Done Done false
17786 5942 Acked 6941 Read 1 New Done Done false
17787 5943 Acked 6942 Read 4 New Done Done false
17788 5944 Acked 6943 Write 6 New Done Done false
17789 5945 Acked 6944 Write 3 New Done Done false
17790 5946 Acked 6945 Read 7 New Done Done false
17791 5947 Acked 6946 Flush 0 New Done Done false
17792 5948 Acked 6947 Read 8 New Done Done false
17793 5949 Acked 6948 Read 4 New Done Done false
17794 5950 Acked 6949 Read 10 New Done Done false
17795 5951 Acked 6950 Write 7 New Done Done false
17796 5952 Acked 6951 Write 6 New Done Done false
17797 5953 Acked 6952 Write 5 New Done Done false
17798 5954 Acked 6953 Read 5 New Done Done false
17799 5955 Acked 6954 Write 5 New Done Done false
17800 5956 Acked 6955 Flush 0 New Done Done false
17801 5957 Acked 6956 Read 5 New Done Done false
17802 5958 Acked 6957 Write 5 New Done Done false
17803 5959 Acked 6958 Read 3 New Done Done false
17804 5960 Acked 6959 Flush 0 New Done Done false
17805 5961 Acked 6960 Write 1 New Done Done false
17806 5962 Acked 6961 Write 2 New Done Done false
17807 5963 Acked 6962 Write 4 New Done Done false
17808 5964 Acked 6963 Write 3 New Done Done false
17809 5965 Acked 6964 Read 6 New Done Done false
17810 5966 Acked 6965 Write 8 New Done Done false
17811 5967 Acked 6966 Read 4 New Done Done false
17812 5968 Acked 6967 Write 9 New Done Done false
17813 5969 Acked 6968 Write 8 New Done Done false
17814 5970 Acked 6969 Write 1 New Done Done false
17815 5971 Acked 6970 Write 5 New Done Done false
17816 5972 Acked 6971 Write 3 New Done Done false
17817 5973 Acked 6972 Write 6 New Done Done false
17818 5974 Acked 6973 Write 3 New Done Done false
17819 5975 Acked 6974 Read 4 New Done Done false
17820 5976 Acked 6975 Write 2 New Done Done false
17821 5977 Acked 6976 Write 7 New Done Done false
17822 5978 Acked 6977 Read 6 New Done Done false
17823 5979 Acked 6978 Read 5 New Done Done false
17824 5980 Acked 6979 Write 7 New Done Done false
17825 5981 Acked 6980 Read 10 New Done Done false
17826 5982 Acked 6981 Flush 0 New Done Done false
17827 5983 Acked 6982 Read 3 New Done Done false
17828 5984 Acked 6983 Flush 0 New Done Done false
17829 5985 Acked 6984 Write 5 New Done Done false
17830 5986 Acked 6985 Read 7 New Done Done false
17831 5987 Acked 6986 Write 4 New Done Done false
17832 5988 Acked 6987 Write 9 New Done Done false
17833 5989 Acked 6988 Write 8 New Done Done false
17834 5990 Acked 6989 Read 4 New Done Done false
17835 5991 Acked 6990 Read 6 New Done Done false
17836 5992 Acked 6991 Flush 0 New Done Done false
17837 5993 Acked 6992 Read 8 New Done Done false
17838 5994 Acked 6993 Write 4 New Done Done false
17839 5995 Acked 6994 Read 4 New Done Done false
17840 5996 Acked 6995 Write 5 New Done Done false
17841 5997 Acked 6996 Read 3 New Done Done false
17842 5998 Acked 6997 Read 4 New Done Done false
17843 5999 Acked 6998 Read 9 New Done Done false
17844 6000 Acked 6999 Read 1 New Done Done false
17845 6001 Acked 7000 Read 5 New Done Done false
17846 6002 Acked 7001 Read 10 New Done Done false
17847 6003 Acked 7002 Read 6 New Done Done false
17848 6004 Acked 7003 Write 5 New Done Done false
17849 6005 Acked 7004 Write 2 New Done Done false
17850 6006 Acked 7005 Read 5 New Done Done false
17851 6007 Acked 7006 Write 3 New Done Done false
17852 6008 Acked 7007 Read 4 New Done Done false
17853 6009 Acked 7008 Flush 0 New Done Done false
17854 6010 Acked 7009 Read 2 New Done Done false
17855 6011 Acked 7010 Read 4 New Done Done false
17856 6012 Acked 7011 Read 8 New Done Done false
17857 6013 Acked 7012 Read 10 New Done Done false
17858 6014 Acked 7013 Flush 0 New Done Done false
17859 6015 Acked 7014 Read 6 New Done Done false
17860 6016 Acked 7015 Read 4 New Done Done false
17861 6017 Acked 7016 Flush 0 New Done Done false
17862 6018 Acked 7017 Flush 0 New Done Done false
17863 6019 Acked 7018 Read 1 New Done Done false
17864 6020 Acked 7019 Write 2 New Done Done false
17865 6021 Acked 7020 Write 6 New Done Done false
17866 6022 Acked 7021 Write 5 New Done Done false
17867 6023 Acked 7022 Read 4 New Done Done false
17868 6024 Acked 7023 Write 2 New Done Done false
17869 6025 Acked 7024 Write 8 New Done Done false
17870 6026 Acked 7025 Read 3 New Done Done false
17871 6027 Acked 7026 Write 9 New Done Done false
17872 6028 Acked 7027 Read 1 New Done Done false
17873 STATES DS:0 DS:1 DS:2 TOTAL
17874 New 301 0 0 301
17875 Sent 0 0 0 0
17876 Done 0 301 301 602
17877 Skipped 0 0 0 0
17878 Error 0 0 0 0
17879 Last Flush: 6726 7017 7017
17880 Downstairs last five completed: 6726 6725 6724 6723 6722
17881 Upstairs last five completed: 6028 6027 6026 6025 6024
17882 CLIENT: Up:0 ds:301 act:2
178832024-02-12T01:38:01.527ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
178842024-02-12T01:38:01.527ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
178852024-02-12T01:38:01.527ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
178862024-02-12T01:38:01.528ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
178872024-02-12T01:38:01.528ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
178882024-02-12T01:38:01.528ZINFOcrucible: [0] client re-new 302 jobs since flush 6726 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
178892024-02-12T01:38:01.528ZINFOcrucible: client task is exiting = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
178902024-02-12T01:38:02.529ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
178912024-02-12T01:38:02.529ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
178922024-02-12T01:38:02.529ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
178932024-02-12T01:38:02.529ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
178942024-02-12T01:38:02.529ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
178952024-02-12T01:38:02.529ZINFOcrucible: [0] client re-new 302 jobs since flush 6726 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
178962024-02-12T01:38:02.530ZINFOcrucible: client task is exiting = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
178972024-02-12T01:38:03.530ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
178982024-02-12T01:38:03.530ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
178992024-02-12T01:38:03.531ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
179002024-02-12T01:38:03.531ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
179012024-02-12T01:38:03.531ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
179022024-02-12T01:38:03.531ZINFOcrucible: [0] client re-new 302 jobs since flush 6726 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
179032024-02-12T01:38:03.531ZINFOcrucible: client task is exiting = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
179042024-02-12T01:38:04.533ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
179052024-02-12T01:38:04.533ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
179062024-02-12T01:38:04.533ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
179072024-02-12T01:38:04.533ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
179082024-02-12T01:38:04.533ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
179092024-02-12T01:38:04.533ZINFOcrucible: [0] client re-new 302 jobs since flush 6726 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
179102024-02-12T01:38:04.533ZINFOcrucible: client task is exiting = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
17911 ----------------------------------------------------------------
17912 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
17913 States: Offline Active Active
17914 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
17915 5728 Acked 6727 Write 4 New Done Done true
17916 5729 Acked 6728 Write 1 New Done Done true
17917 5730 Acked 6729 Write 1 New Done Done true
17918 5731 Acked 6730 Read 8 New Done Done true
17919 5732 Acked 6731 Read 1 New Done Done true
17920 5733 Acked 6732 Write 8 New Done Done true
17921 5734 Acked 6733 Write 3 New Done Done true
17922 5735 Acked 6734 Write 1 New Done Done true
17923 5736 Acked 6735 Read 8 New Done Done true
17924 5737 Acked 6736 Write 8 New Done Done true
17925 5738 Acked 6737 Read 9 New Done Done true
17926 5739 Acked 6738 Write 4 New Done Done true
17927 5740 Acked 6739 Write 6 New Done Done true
17928 5741 Acked 6740 Write 9 New Done Done true
17929 5742 Acked 6741 Read 8 New Done Done true
17930 5743 Acked 6742 Read 2 New Done Done true
17931 5744 Acked 6743 Write 9 New Done Done true
17932 5745 Acked 6744 Read 10 New Done Done true
17933 5746 Acked 6745 Read 2 New Done Done true
17934 5747 Acked 6746 Read 2 New Done Done true
17935 5748 Acked 6747 Read 8 New Done Done true
17936 5749 Acked 6748 Write 8 New Done Done true
17937 5750 Acked 6749 Write 8 New Done Done true
17938 5751 Acked 6750 Write 6 New Done Done true
17939 5752 Acked 6751 Write 6 New Done Done true
17940 5753 Acked 6752 Read 6 New Done Done true
17941 5754 Acked 6753 Read 6 New Done Done true
17942 5755 Acked 6754 Write 10 New Done Done true
17943 5756 Acked 6755 Read 7 New Done Done true
17944 5757 Acked 6756 Write 10 New Done Done true
17945 5758 Acked 6757 Read 3 New Done Done true
17946 5759 Acked 6758 Read 6 New Done Done true
17947 5760 Acked 6759 Write 3 New Done Done true
17948 5761 Acked 6760 Write 6 New Done Done true
17949 5762 Acked 6761 Flush 0 New Done Done true
17950 5763 Acked 6762 Write 3 New Done Done true
17951 5764 Acked 6763 Write 9 New Done Done true
17952 5765 Acked 6764 Read 3 New Done Done true
17953 5766 Acked 6765 Write 7 New Done Done true
17954 5767 Acked 6766 Read 10 New Done Done true
17955 5768 Acked 6767 Read 2 New Done Done true
17956 5769 Acked 6768 Read 3 New Done Done true
17957 5770 Acked 6769 Read 3 New Done Done true
17958 5771 Acked 6770 Read 4 New Done Done true
17959 5772 Acked 6771 Write 9 New Done Done true
17960 5773 Acked 6772 Read 7 New Done Done true
17961 5774 Acked 6773 Write 8 New Done Done true
17962 5775 Acked 6774 Read 8 New Done Done true
17963 5776 Acked 6775 Write 9 New Done Done true
17964 5777 Acked 6776 Write 10 New Done Done true
17965 5778 Acked 6777 Read 4 New Done Done true
17966 5779 Acked 6778 Read 1 New Done Done true
17967 5780 Acked 6779 Write 8 New Done Done true
17968 5781 Acked 6780 Read 10 New Done Done true
17969 5782 Acked 6781 Write 4 New Done Done true
17970 5783 Acked 6782 Read 4 New Done Done true
17971 5784 Acked 6783 Read 6 New Done Done true
17972 5785 Acked 6784 Write 3 New Done Done true
17973 5786 Acked 6785 Read 3 New Done Done true
17974 5787 Acked 6786 Write 9 New Done Done true
17975 5788 Acked 6787 Read 9 New Done Done true
17976 5789 Acked 6788 Write 5 New Done Done true
17977 5790 Acked 6789 Write 8 New Done Done true
17978 5791 Acked 6790 Read 5 New Done Done true
17979 5792 Acked 6791 Write 4 New Done Done true
17980 5793 Acked 6792 Read 8 New Done Done true
17981 5794 Acked 6793 Write 6 New Done Done true
17982 5795 Acked 6794 Read 2 New Done Done true
17983 5796 Acked 6795 Read 8 New Done Done true
17984 5797 Acked 6796 Read 1 New Done Done true
17985 5798 Acked 6797 Read 2 New Done Done true
17986 5799 Acked 6798 Read 5 New Done Done true
17987 5800 Acked 6799 Write 1 New Done Done true
17988 5801 Acked 6800 Read 2 New Done Done true
17989 5802 Acked 6801 Flush 0 New Done Done true
17990 5803 Acked 6802 Read 1 New Done Done true
17991 5804 Acked 6803 Read 8 New Done Done true
17992 5805 Acked 6804 Read 3 New Done Done true
17993 5806 Acked 6805 Read 8 New Done Done true
17994 5807 Acked 6806 Read 3 New Done Done true
17995 5808 Acked 6807 Flush 0 New Done Done true
17996 5809 Acked 6808 Write 1 New Done Done true
17997 5810 Acked 6809 Write 4 New Done Done true
17998 5811 Acked 6810 Read 3 New Done Done true
17999 5812 Acked 6811 Flush 0 New Done Done true
18000 5813 Acked 6812 Write 5 New Done Done true
18001 5814 Acked 6813 Read 9 New Done Done true
18002 5815 Acked 6814 Read 8 New Done Done true
18003 5816 Acked 6815 Read 10 New Done Done true
18004 5817 Acked 6816 Write 2 New Done Done true
18005 5818 Acked 6817 Write 4 New Done Done true
18006 5819 Acked 6818 Read 5 New Done Done true
18007 5820 Acked 6819 Write 8 New Done Done true
18008 5821 Acked 6820 Write 1 New Done Done true
18009 5822 Acked 6821 Read 4 New Done Done true
18010 5823 Acked 6822 Write 4 New Done Done true
18011 5824 Acked 6823 Read 7 New Done Done true
18012 5825 Acked 6824 Flush 0 New Done Done true
18013 5826 Acked 6825 Read 7 New Done Done true
18014 5827 Acked 6826 Read 1 New Done Done true
18015 5828 Acked 6827 Read 3 New Done Done true
18016 5829 Acked 6828 Write 2 New Done Done true
18017 5830 Acked 6829 Read 9 New Done Done true
18018 5831 Acked 6830 Write 7 New Done Done true
18019 5832 Acked 6831 Read 9 New Done Done true
18020 5833 Acked 6832 Read 7 New Done Done true
18021 5834 Acked 6833 Write 10 New Done Done true
18022 5835 Acked 6834 Read 9 New Done Done true
18023 5836 Acked 6835 Read 7 New Done Done true
18024 5837 Acked 6836 Write 9 New Done Done true
18025 5838 Acked 6837 Write 3 New Done Done true
18026 5839 Acked 6838 Read 2 New Done Done true
18027 5840 Acked 6839 Write 8 New Done Done true
18028 5841 Acked 6840 Read 10 New Done Done true
18029 5842 Acked 6841 Flush 0 New Done Done true
18030 5843 Acked 6842 Read 8 New Done Done true
18031 5844 Acked 6843 Flush 0 New Done Done true
18032 5845 Acked 6844 Write 1 New Done Done true
18033 5846 Acked 6845 Read 2 New Done Done true
18034 5847 Acked 6846 Flush 0 New Done Done true
18035 5848 Acked 6847 Read 2 New Done Done true
18036 5849 Acked 6848 Read 10 New Done Done true
18037 5850 Acked 6849 Read 9 New Done Done true
18038 5851 Acked 6850 Write 1 New Done Done true
18039 5852 Acked 6851 Read 8 New Done Done true
18040 5853 Acked 6852 Read 8 New Done Done true
18041 5854 Acked 6853 Write 8 New Done Done true
18042 5855 Acked 6854 Write 10 New Done Done true
18043 5856 Acked 6855 Write 10 New Done Done true
18044 5857 Acked 6856 Flush 0 New Done Done true
18045 5858 Acked 6857 Read 3 New Done Done true
18046 5859 Acked 6858 Write 9 New Done Done true
18047 5860 Acked 6859 Read 9 New Done Done true
18048 5861 Acked 6860 Read 10 New Done Done true
18049 5862 Acked 6861 Write 3 New Done Done true
18050 5863 Acked 6862 Read 6 New Done Done true
18051 5864 Acked 6863 Write 9 New Done Done true
18052 5865 Acked 6864 Write 3 New Done Done true
18053 5866 Acked 6865 Read 6 New Done Done true
18054 5867 Acked 6866 Read 6 New Done Done true
18055 5868 Acked 6867 Read 9 New Done Done true
18056 5869 Acked 6868 Read 5 New Done Done true
18057 5870 Acked 6869 Read 4 New Done Done true
18058 5871 Acked 6870 Write 2 New Done Done true
18059 5872 Acked 6871 Write 5 New Done Done true
18060 5873 Acked 6872 Write 10 New Done Done true
18061 5874 Acked 6873 Read 5 New Done Done true
18062 5875 Acked 6874 Write 4 New Done Done true
18063 5876 Acked 6875 Write 8 New Done Done true
18064 5877 Acked 6876 Write 9 New Done Done true
18065 5878 Acked 6877 Write 1 New Done Done true
18066 5879 Acked 6878 Read 5 New Done Done true
18067 5880 Acked 6879 Write 6 New Done Done true
18068 5881 Acked 6880 Flush 0 New Done Done true
18069 5882 Acked 6881 Flush 0 New Done Done true
18070 5883 Acked 6882 Read 8 New Done Done true
18071 5884 Acked 6883 Write 1 New Done Done true
18072 5885 Acked 6884 Read 4 New Done Done true
18073 5886 Acked 6885 Write 5 New Done Done true
18074 5887 Acked 6886 Write 7 New Done Done true
18075 5888 Acked 6887 Write 2 New Done Done true
18076 5889 Acked 6888 Write 6 New Done Done true
18077 5890 Acked 6889 Read 8 New Done Done true
18078 5891 Acked 6890 Write 6 New Done Done true
18079 5892 Acked 6891 Read 2 New Done Done true
18080 5893 Acked 6892 Write 4 New Done Done true
18081 5894 Acked 6893 Flush 0 New Done Done true
18082 5895 Acked 6894 Read 8 New Done Done true
18083 5896 Acked 6895 Read 7 New Done Done true
18084 5897 Acked 6896 Read 6 New Done Done true
18085 5898 Acked 6897 Read 8 New Done Done true
18086 5899 Acked 6898 Read 3 New Done Done true
18087 5900 Acked 6899 Read 2 New Done Done true
18088 5901 Acked 6900 Write 3 New Done Done true
18089 5902 Acked 6901 Write 8 New Done Done true
18090 5903 Acked 6902 Read 1 New Done Done true
18091 5904 Acked 6903 Write 3 New Done Done true
18092 5905 Acked 6904 Write 5 New Done Done true
18093 5906 Acked 6905 Write 5 New Done Done true
18094 5907 Acked 6906 Write 1 New Done Done true
18095 5908 Acked 6907 Write 6 New Done Done true
18096 5909 Acked 6908 Read 2 New Done Done true
18097 5910 Acked 6909 Read 7 New Done Done true
18098 5911 Acked 6910 Read 6 New Done Done true
18099 5912 Acked 6911 Read 3 New Done Done true
18100 5913 Acked 6912 Write 6 New Done Done true
18101 5914 Acked 6913 Flush 0 New Done Done true
18102 5915 Acked 6914 Read 6 New Done Done true
18103 5916 Acked 6915 Read 2 New Done Done true
18104 5917 Acked 6916 Read 3 New Done Done true
18105 5918 Acked 6917 Read 10 New Done Done true
18106 5919 Acked 6918 Read 7 New Done Done true
18107 5920 Acked 6919 Write 3 New Done Done true
18108 5921 Acked 6920 Read 3 New Done Done true
18109 5922 Acked 6921 Write 8 New Done Done true
18110 5923 Acked 6922 Write 10 New Done Done true
18111 5924 Acked 6923 Read 7 New Done Done true
18112 5925 Acked 6924 Write 9 New Done Done true
18113 5926 Acked 6925 Write 6 New Done Done true
18114 5927 Acked 6926 Read 2 New Done Done true
18115 5928 Acked 6927 Write 6 New Done Done true
18116 5929 Acked 6928 Read 3 New Done Done true
18117 5930 Acked 6929 Read 8 New Done Done true
18118 5931 Acked 6930 Read 2 New Done Done true
18119 5932 Acked 6931 Read 9 New Done Done true
18120 5933 Acked 6932 Write 9 New Done Done true
18121 5934 Acked 6933 Write 9 New Done Done true
18122 5935 Acked 6934 Read 8 New Done Done true
18123 5936 Acked 6935 Write 5 New Done Done true
18124 5937 Acked 6936 Read 4 New Done Done true
18125 5938 Acked 6937 Read 1 New Done Done true
18126 5939 Acked 6938 Flush 0 New Done Done true
18127 5940 Acked 6939 Read 6 New Done Done true
18128 5941 Acked 6940 Read 7 New Done Done true
18129 5942 Acked 6941 Read 1 New Done Done true
18130 5943 Acked 6942 Read 4 New Done Done true
18131 5944 Acked 6943 Write 6 New Done Done true
18132 5945 Acked 6944 Write 3 New Done Done true
18133 5946 Acked 6945 Read 7 New Done Done true
18134 5947 Acked 6946 Flush 0 New Done Done true
18135 5948 Acked 6947 Read 8 New Done Done true
18136 5949 Acked 6948 Read 4 New Done Done true
18137 5950 Acked 6949 Read 10 New Done Done true
18138 5951 Acked 6950 Write 7 New Done Done true
18139 5952 Acked 6951 Write 6 New Done Done true
18140 5953 Acked 6952 Write 5 New Done Done true
18141 5954 Acked 6953 Read 5 New Done Done true
18142 5955 Acked 6954 Write 5 New Done Done true
18143 5956 Acked 6955 Flush 0 New Done Done true
18144 5957 Acked 6956 Read 5 New Done Done true
18145 5958 Acked 6957 Write 5 New Done Done true
18146 5959 Acked 6958 Read 3 New Done Done true
18147 5960 Acked 6959 Flush 0 New Done Done true
18148 5961 Acked 6960 Write 1 New Done Done true
18149 5962 Acked 6961 Write 2 New Done Done true
18150 5963 Acked 6962 Write 4 New Done Done true
18151 5964 Acked 6963 Write 3 New Done Done true
18152 5965 Acked 6964 Read 6 New Done Done true
18153 5966 Acked 6965 Write 8 New Done Done true
18154 5967 Acked 6966 Read 4 New Done Done true
18155 5968 Acked 6967 Write 9 New Done Done true
18156 5969 Acked 6968 Write 8 New Done Done true
18157 5970 Acked 6969 Write 1 New Done Done true
18158 5971 Acked 6970 Write 5 New Done Done true
18159 5972 Acked 6971 Write 3 New Done Done true
18160 5973 Acked 6972 Write 6 New Done Done true
18161 5974 Acked 6973 Write 3 New Done Done true
18162 5975 Acked 6974 Read 4 New Done Done true
18163 5976 Acked 6975 Write 2 New Done Done true
18164 5977 Acked 6976 Write 7 New Done Done true
18165 5978 Acked 6977 Read 6 New Done Done true
18166 5979 Acked 6978 Read 5 New Done Done true
18167 5980 Acked 6979 Write 7 New Done Done true
18168 5981 Acked 6980 Read 10 New Done Done true
18169 5982 Acked 6981 Flush 0 New Done Done true
18170 5983 Acked 6982 Read 3 New Done Done true
18171 5984 Acked 6983 Flush 0 New Done Done true
18172 5985 Acked 6984 Write 5 New Done Done true
18173 5986 Acked 6985 Read 7 New Done Done true
18174 5987 Acked 6986 Write 4 New Done Done true
18175 5988 Acked 6987 Write 9 New Done Done true
18176 5989 Acked 6988 Write 8 New Done Done true
18177 5990 Acked 6989 Read 4 New Done Done true
18178 5991 Acked 6990 Read 6 New Done Done true
18179 5992 Acked 6991 Flush 0 New Done Done true
18180 5993 Acked 6992 Read 8 New Done Done true
18181 5994 Acked 6993 Write 4 New Done Done true
18182 5995 Acked 6994 Read 4 New Done Done true
18183 5996 Acked 6995 Write 5 New Done Done true
18184 5997 Acked 6996 Read 3 New Done Done true
18185 5998 Acked 6997 Read 4 New Done Done true
18186 5999 Acked 6998 Read 9 New Done Done true
18187 6000 Acked 6999 Read 1 New Done Done true
18188 6001 Acked 7000 Read 5 New Done Done true
18189 6002 Acked 7001 Read 10 New Done Done true
18190 6003 Acked 7002 Read 6 New Done Done true
18191 6004 Acked 7003 Write 5 New Done Done true
18192 6005 Acked 7004 Write 2 New Done Done true
18193 6006 Acked 7005 Read 5 New Done Done true
18194 6007 Acked 7006 Write 3 New Done Done true
18195 6008 Acked 7007 Read 4 New Done Done true
18196 6009 Acked 7008 Flush 0 New Done Done true
18197 6010 Acked 7009 Read 2 New Done Done true
18198 6011 Acked 7010 Read 4 New Done Done true
18199 6012 Acked 7011 Read 8 New Done Done true
18200 6013 Acked 7012 Read 10 New Done Done true
18201 6014 Acked 7013 Flush 0 New Done Done true
18202 6015 Acked 7014 Read 6 New Done Done true
18203 6016 Acked 7015 Read 4 New Done Done true
18204 6017 Acked 7016 Flush 0 New Done Done true
18205 6018 Acked 7017 Flush 0 New Done Done true
18206 6019 Acked 7018 Read 1 New Done Done true
18207 6020 Acked 7019 Write 2 New Done Done true
18208 6021 Acked 7020 Write 6 New Done Done true
18209 6022 Acked 7021 Write 5 New Done Done true
18210 6023 Acked 7022 Read 4 New Done Done true
18211 6024 Acked 7023 Write 2 New Done Done true
18212 6025 Acked 7024 Write 8 New Done Done true
18213 6026 Acked 7025 Read 3 New Done Done true
18214 6027 Acked 7026 Write 9 New Done Done true
18215 6028 Acked 7027 Read 1 New Done Done true
18216 6029 Acked 7028 Flush 0 New Done Done true
18217 STATES DS:0 DS:1 DS:2 TOTAL
18218 New 302 0 0 302
18219 Sent 0 0 0 0
18220 Done 0 302 302 604
18221 Skipped 0 0 0 0
18222 Error 0 0 0 0
18223 Last Flush: 6726 7028 7028
18224 Downstairs last five completed: 6726 6725 6724 6723 6722
18225 Upstairs last five completed: 6029 6028 6027 6026 6025
18226 CLIENT: Up:0 ds:302 act:2
182272024-02-12T01:38:05.534ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
182282024-02-12T01:38:05.534ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
182292024-02-12T01:38:05.534ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
182302024-02-12T01:38:05.534ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
182312024-02-12T01:38:05.534ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
182322024-02-12T01:38:05.534ZINFOcrucible: [0] client re-new 302 jobs since flush 6726 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
182332024-02-12T01:38:05.534ZINFOcrucible: client task is exiting = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
182342024-02-12T01:38:06.535ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
182352024-02-12T01:38:06.535ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
182362024-02-12T01:38:06.535ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
182372024-02-12T01:38:06.535ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
182382024-02-12T01:38:06.535ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
182392024-02-12T01:38:06.535ZINFOcrucible: [0] client re-new 302 jobs since flush 6726 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
182402024-02-12T01:38:06.535ZINFOcrucible: client task is exiting = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
182412024-02-12T01:38:07.537ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
182422024-02-12T01:38:07.537ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
182432024-02-12T01:38:07.537ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
182442024-02-12T01:38:07.537ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
182452024-02-12T01:38:07.537ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
182462024-02-12T01:38:07.537ZINFOcrucible: [0] client re-new 302 jobs since flush 6726 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
182472024-02-12T01:38:07.537ZINFOcrucible: client task is exiting = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
18248 [8810] got start action
18249 [8810] I am going to restart
18250 Starting downstairs at port 8810
18251 Make output file at "/tmp/dsc/downstairs-8810.txt"
18252 [8810][0] reports Starting
18253 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1184)
18254 [8810][0] reports Running
182552024-02-12T01:38:08.538ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
182562024-02-12T01:38:08.538ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
182572024-02-12T01:38:08.538ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
182582024-02-12T01:38:08.538ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
182592024-02-12T01:38:08.538ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
182602024-02-12T01:38:08.538ZINFOcrucible: [0] client re-new 302 jobs since flush 6726 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
182612024-02-12T01:38:08.538ZINFOcrucible: client task is exiting = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
18262 ----------------------------------------------------------------
18263 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
18264 States: Offline Active Active
18265 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
18266 5728 Acked 6727 Write 4 New Done Done true
18267 5729 Acked 6728 Write 1 New Done Done true
18268 5730 Acked 6729 Write 1 New Done Done true
18269 5731 Acked 6730 Read 8 New Done Done true
18270 5732 Acked 6731 Read 1 New Done Done true
18271 5733 Acked 6732 Write 8 New Done Done true
18272 5734 Acked 6733 Write 3 New Done Done true
18273 5735 Acked 6734 Write 1 New Done Done true
18274 5736 Acked 6735 Read 8 New Done Done true
18275 5737 Acked 6736 Write 8 New Done Done true
18276 5738 Acked 6737 Read 9 New Done Done true
18277 5739 Acked 6738 Write 4 New Done Done true
18278 5740 Acked 6739 Write 6 New Done Done true
18279 5741 Acked 6740 Write 9 New Done Done true
18280 5742 Acked 6741 Read 8 New Done Done true
18281 5743 Acked 6742 Read 2 New Done Done true
18282 5744 Acked 6743 Write 9 New Done Done true
18283 5745 Acked 6744 Read 10 New Done Done true
18284 5746 Acked 6745 Read 2 New Done Done true
18285 5747 Acked 6746 Read 2 New Done Done true
18286 5748 Acked 6747 Read 8 New Done Done true
18287 5749 Acked 6748 Write 8 New Done Done true
18288 5750 Acked 6749 Write 8 New Done Done true
18289 5751 Acked 6750 Write 6 New Done Done true
18290 5752 Acked 6751 Write 6 New Done Done true
18291 5753 Acked 6752 Read 6 New Done Done true
18292 5754 Acked 6753 Read 6 New Done Done true
18293 5755 Acked 6754 Write 10 New Done Done true
18294 5756 Acked 6755 Read 7 New Done Done true
18295 5757 Acked 6756 Write 10 New Done Done true
18296 5758 Acked 6757 Read 3 New Done Done true
18297 5759 Acked 6758 Read 6 New Done Done true
18298 5760 Acked 6759 Write 3 New Done Done true
18299 5761 Acked 6760 Write 6 New Done Done true
18300 5762 Acked 6761 Flush 0 New Done Done true
18301 5763 Acked 6762 Write 3 New Done Done true
18302 5764 Acked 6763 Write 9 New Done Done true
18303 5765 Acked 6764 Read 3 New Done Done true
18304 5766 Acked 6765 Write 7 New Done Done true
18305 5767 Acked 6766 Read 10 New Done Done true
18306 5768 Acked 6767 Read 2 New Done Done true
18307 5769 Acked 6768 Read 3 New Done Done true
18308 5770 Acked 6769 Read 3 New Done Done true
18309 5771 Acked 6770 Read 4 New Done Done true
18310 5772 Acked 6771 Write 9 New Done Done true
18311 5773 Acked 6772 Read 7 New Done Done true
18312 5774 Acked 6773 Write 8 New Done Done true
18313 5775 Acked 6774 Read 8 New Done Done true
18314 5776 Acked 6775 Write 9 New Done Done true
18315 5777 Acked 6776 Write 10 New Done Done true
18316 5778 Acked 6777 Read 4 New Done Done true
18317 5779 Acked 6778 Read 1 New Done Done true
18318 5780 Acked 6779 Write 8 New Done Done true
18319 5781 Acked 6780 Read 10 New Done Done true
18320 5782 Acked 6781 Write 4 New Done Done true
18321 5783 Acked 6782 Read 4 New Done Done true
18322 5784 Acked 6783 Read 6 New Done Done true
18323 5785 Acked 6784 Write 3 New Done Done true
18324 5786 Acked 6785 Read 3 New Done Done true
18325 5787 Acked 6786 Write 9 New Done Done true
18326 5788 Acked 6787 Read 9 New Done Done true
18327 5789 Acked 6788 Write 5 New Done Done true
18328 5790 Acked 6789 Write 8 New Done Done true
18329 5791 Acked 6790 Read 5 New Done Done true
18330 5792 Acked 6791 Write 4 New Done Done true
18331 5793 Acked 6792 Read 8 New Done Done true
18332 5794 Acked 6793 Write 6 New Done Done true
18333 5795 Acked 6794 Read 2 New Done Done true
18334 5796 Acked 6795 Read 8 New Done Done true
18335 5797 Acked 6796 Read 1 New Done Done true
18336 5798 Acked 6797 Read 2 New Done Done true
18337 5799 Acked 6798 Read 5 New Done Done true
18338 5800 Acked 6799 Write 1 New Done Done true
18339 5801 Acked 6800 Read 2 New Done Done true
18340 5802 Acked 6801 Flush 0 New Done Done true
18341 5803 Acked 6802 Read 1 New Done Done true
18342 5804 Acked 6803 Read 8 New Done Done true
18343 5805 Acked 6804 Read 3 New Done Done true
18344 5806 Acked 6805 Read 8 New Done Done true
18345 5807 Acked 6806 Read 3 New Done Done true
18346 5808 Acked 6807 Flush 0 New Done Done true
18347 5809 Acked 6808 Write 1 New Done Done true
18348 5810 Acked 6809 Write 4 New Done Done true
18349 5811 Acked 6810 Read 3 New Done Done true
18350 5812 Acked 6811 Flush 0 New Done Done true
18351 5813 Acked 6812 Write 5 New Done Done true
18352 5814 Acked 6813 Read 9 New Done Done true
18353 5815 Acked 6814 Read 8 New Done Done true
18354 5816 Acked 6815 Read 10 New Done Done true
18355 5817 Acked 6816 Write 2 New Done Done true
18356 5818 Acked 6817 Write 4 New Done Done true
18357 5819 Acked 6818 Read 5 New Done Done true
18358 5820 Acked 6819 Write 8 New Done Done true
18359 5821 Acked 6820 Write 1 New Done Done true
18360 5822 Acked 6821 Read 4 New Done Done true
18361 5823 Acked 6822 Write 4 New Done Done true
18362 5824 Acked 6823 Read 7 New Done Done true
18363 5825 Acked 6824 Flush 0 New Done Done true
18364 5826 Acked 6825 Read 7 New Done Done true
18365 5827 Acked 6826 Read 1 New Done Done true
18366 5828 Acked 6827 Read 3 New Done Done true
18367 5829 Acked 6828 Write 2 New Done Done true
18368 5830 Acked 6829 Read 9 New Done Done true
18369 5831 Acked 6830 Write 7 New Done Done true
18370 5832 Acked 6831 Read 9 New Done Done true
18371 5833 Acked 6832 Read 7 New Done Done true
18372 5834 Acked 6833 Write 10 New Done Done true
18373 5835 Acked 6834 Read 9 New Done Done true
18374 5836 Acked 6835 Read 7 New Done Done true
18375 5837 Acked 6836 Write 9 New Done Done true
18376 5838 Acked 6837 Write 3 New Done Done true
18377 5839 Acked 6838 Read 2 New Done Done true
18378 5840 Acked 6839 Write 8 New Done Done true
18379 5841 Acked 6840 Read 10 New Done Done true
18380 5842 Acked 6841 Flush 0 New Done Done true
18381 5843 Acked 6842 Read 8 New Done Done true
18382 5844 Acked 6843 Flush 0 New Done Done true
18383 5845 Acked 6844 Write 1 New Done Done true
18384 5846 Acked 6845 Read 2 New Done Done true
18385 5847 Acked 6846 Flush 0 New Done Done true
18386 5848 Acked 6847 Read 2 New Done Done true
18387 5849 Acked 6848 Read 10 New Done Done true
18388 5850 Acked 6849 Read 9 New Done Done true
18389 5851 Acked 6850 Write 1 New Done Done true
18390 5852 Acked 6851 Read 8 New Done Done true
18391 5853 Acked 6852 Read 8 New Done Done true
18392 5854 Acked 6853 Write 8 New Done Done true
18393 5855 Acked 6854 Write 10 New Done Done true
18394 5856 Acked 6855 Write 10 New Done Done true
18395 5857 Acked 6856 Flush 0 New Done Done true
18396 5858 Acked 6857 Read 3 New Done Done true
18397 5859 Acked 6858 Write 9 New Done Done true
18398 5860 Acked 6859 Read 9 New Done Done true
18399 5861 Acked 6860 Read 10 New Done Done true
18400 5862 Acked 6861 Write 3 New Done Done true
18401 5863 Acked 6862 Read 6 New Done Done true
18402 5864 Acked 6863 Write 9 New Done Done true
18403 5865 Acked 6864 Write 3 New Done Done true
18404 5866 Acked 6865 Read 6 New Done Done true
18405 5867 Acked 6866 Read 6 New Done Done true
18406 5868 Acked 6867 Read 9 New Done Done true
18407 5869 Acked 6868 Read 5 New Done Done true
18408 5870 Acked 6869 Read 4 New Done Done true
18409 5871 Acked 6870 Write 2 New Done Done true
18410 5872 Acked 6871 Write 5 New Done Done true
18411 5873 Acked 6872 Write 10 New Done Done true
18412 5874 Acked 6873 Read 5 New Done Done true
18413 5875 Acked 6874 Write 4 New Done Done true
18414 5876 Acked 6875 Write 8 New Done Done true
18415 5877 Acked 6876 Write 9 New Done Done true
18416 5878 Acked 6877 Write 1 New Done Done true
18417 5879 Acked 6878 Read 5 New Done Done true
18418 5880 Acked 6879 Write 6 New Done Done true
18419 5881 Acked 6880 Flush 0 New Done Done true
18420 5882 Acked 6881 Flush 0 New Done Done true
18421 5883 Acked 6882 Read 8 New Done Done true
18422 5884 Acked 6883 Write 1 New Done Done true
18423 5885 Acked 6884 Read 4 New Done Done true
18424 5886 Acked 6885 Write 5 New Done Done true
18425 5887 Acked 6886 Write 7 New Done Done true
18426 5888 Acked 6887 Write 2 New Done Done true
18427 5889 Acked 6888 Write 6 New Done Done true
18428 5890 Acked 6889 Read 8 New Done Done true
18429 5891 Acked 6890 Write 6 New Done Done true
18430 5892 Acked 6891 Read 2 New Done Done true
18431 5893 Acked 6892 Write 4 New Done Done true
18432 5894 Acked 6893 Flush 0 New Done Done true
18433 5895 Acked 6894 Read 8 New Done Done true
18434 5896 Acked 6895 Read 7 New Done Done true
18435 5897 Acked 6896 Read 6 New Done Done true
18436 5898 Acked 6897 Read 8 New Done Done true
18437 5899 Acked 6898 Read 3 New Done Done true
18438 5900 Acked 6899 Read 2 New Done Done true
18439 5901 Acked 6900 Write 3 New Done Done true
18440 5902 Acked 6901 Write 8 New Done Done true
18441 5903 Acked 6902 Read 1 New Done Done true
18442 5904 Acked 6903 Write 3 New Done Done true
18443 5905 Acked 6904 Write 5 New Done Done true
18444 5906 Acked 6905 Write 5 New Done Done true
18445 5907 Acked 6906 Write 1 New Done Done true
18446 5908 Acked 6907 Write 6 New Done Done true
18447 5909 Acked 6908 Read 2 New Done Done true
18448 5910 Acked 6909 Read 7 New Done Done true
18449 5911 Acked 6910 Read 6 New Done Done true
18450 5912 Acked 6911 Read 3 New Done Done true
18451 5913 Acked 6912 Write 6 New Done Done true
18452 5914 Acked 6913 Flush 0 New Done Done true
18453 5915 Acked 6914 Read 6 New Done Done true
18454 5916 Acked 6915 Read 2 New Done Done true
18455 5917 Acked 6916 Read 3 New Done Done true
18456 5918 Acked 6917 Read 10 New Done Done true
18457 5919 Acked 6918 Read 7 New Done Done true
18458 5920 Acked 6919 Write 3 New Done Done true
18459 5921 Acked 6920 Read 3 New Done Done true
18460 5922 Acked 6921 Write 8 New Done Done true
18461 5923 Acked 6922 Write 10 New Done Done true
18462 5924 Acked 6923 Read 7 New Done Done true
18463 5925 Acked 6924 Write 9 New Done Done true
18464 5926 Acked 6925 Write 6 New Done Done true
18465 5927 Acked 6926 Read 2 New Done Done true
18466 5928 Acked 6927 Write 6 New Done Done true
18467 5929 Acked 6928 Read 3 New Done Done true
18468 5930 Acked 6929 Read 8 New Done Done true
18469 5931 Acked 6930 Read 2 New Done Done true
18470 5932 Acked 6931 Read 9 New Done Done true
18471 5933 Acked 6932 Write 9 New Done Done true
18472 5934 Acked 6933 Write 9 New Done Done true
18473 5935 Acked 6934 Read 8 New Done Done true
18474 5936 Acked 6935 Write 5 New Done Done true
18475 5937 Acked 6936 Read 4 New Done Done true
18476 5938 Acked 6937 Read 1 New Done Done true
18477 5939 Acked 6938 Flush 0 New Done Done true
18478 5940 Acked 6939 Read 6 New Done Done true
18479 5941 Acked 6940 Read 7 New Done Done true
18480 5942 Acked 6941 Read 1 New Done Done true
18481 5943 Acked 6942 Read 4 New Done Done true
18482 5944 Acked 6943 Write 6 New Done Done true
18483 5945 Acked 6944 Write 3 New Done Done true
18484 5946 Acked 6945 Read 7 New Done Done true
18485 5947 Acked 6946 Flush 0 New Done Done true
18486 5948 Acked 6947 Read 8 New Done Done true
18487 5949 Acked 6948 Read 4 New Done Done true
18488 5950 Acked 6949 Read 10 New Done Done true
18489 5951 Acked 6950 Write 7 New Done Done true
18490 5952 Acked 6951 Write 6 New Done Done true
18491 5953 Acked 6952 Write 5 New Done Done true
18492 5954 Acked 6953 Read 5 New Done Done true
18493 5955 Acked 6954 Write 5 New Done Done true
18494 5956 Acked 6955 Flush 0 New Done Done true
18495 5957 Acked 6956 Read 5 New Done Done true
18496 5958 Acked 6957 Write 5 New Done Done true
18497 5959 Acked 6958 Read 3 New Done Done true
18498 5960 Acked 6959 Flush 0 New Done Done true
18499 5961 Acked 6960 Write 1 New Done Done true
18500 5962 Acked 6961 Write 2 New Done Done true
18501 5963 Acked 6962 Write 4 New Done Done true
18502 5964 Acked 6963 Write 3 New Done Done true
18503 5965 Acked 6964 Read 6 New Done Done true
18504 5966 Acked 6965 Write 8 New Done Done true
18505 5967 Acked 6966 Read 4 New Done Done true
18506 5968 Acked 6967 Write 9 New Done Done true
18507 5969 Acked 6968 Write 8 New Done Done true
18508 5970 Acked 6969 Write 1 New Done Done true
18509 5971 Acked 6970 Write 5 New Done Done true
18510 5972 Acked 6971 Write 3 New Done Done true
18511 5973 Acked 6972 Write 6 New Done Done true
18512 5974 Acked 6973 Write 3 New Done Done true
18513 5975 Acked 6974 Read 4 New Done Done true
18514 5976 Acked 6975 Write 2 New Done Done true
18515 5977 Acked 6976 Write 7 New Done Done true
18516 5978 Acked 6977 Read 6 New Done Done true
18517 5979 Acked 6978 Read 5 New Done Done true
18518 5980 Acked 6979 Write 7 New Done Done true
18519 5981 Acked 6980 Read 10 New Done Done true
18520 5982 Acked 6981 Flush 0 New Done Done true
18521 5983 Acked 6982 Read 3 New Done Done true
18522 5984 Acked 6983 Flush 0 New Done Done true
18523 5985 Acked 6984 Write 5 New Done Done true
18524 5986 Acked 6985 Read 7 New Done Done true
18525 5987 Acked 6986 Write 4 New Done Done true
18526 5988 Acked 6987 Write 9 New Done Done true
18527 5989 Acked 6988 Write 8 New Done Done true
18528 5990 Acked 6989 Read 4 New Done Done true
18529 5991 Acked 6990 Read 6 New Done Done true
18530 5992 Acked 6991 Flush 0 New Done Done true
18531 5993 Acked 6992 Read 8 New Done Done true
18532 5994 Acked 6993 Write 4 New Done Done true
18533 5995 Acked 6994 Read 4 New Done Done true
18534 5996 Acked 6995 Write 5 New Done Done true
18535 5997 Acked 6996 Read 3 New Done Done true
18536 5998 Acked 6997 Read 4 New Done Done true
18537 5999 Acked 6998 Read 9 New Done Done true
18538 6000 Acked 6999 Read 1 New Done Done true
18539 6001 Acked 7000 Read 5 New Done Done true
18540 6002 Acked 7001 Read 10 New Done Done true
18541 6003 Acked 7002 Read 6 New Done Done true
18542 6004 Acked 7003 Write 5 New Done Done true
18543 6005 Acked 7004 Write 2 New Done Done true
18544 6006 Acked 7005 Read 5 New Done Done true
18545 6007 Acked 7006 Write 3 New Done Done true
18546 6008 Acked 7007 Read 4 New Done Done true
18547 6009 Acked 7008 Flush 0 New Done Done true
18548 6010 Acked 7009 Read 2 New Done Done true
18549 6011 Acked 7010 Read 4 New Done Done true
18550 6012 Acked 7011 Read 8 New Done Done true
18551 6013 Acked 7012 Read 10 New Done Done true
18552 6014 Acked 7013 Flush 0 New Done Done true
18553 6015 Acked 7014 Read 6 New Done Done true
18554 6016 Acked 7015 Read 4 New Done Done true
18555 6017 Acked 7016 Flush 0 New Done Done true
18556 6018 Acked 7017 Flush 0 New Done Done true
18557 6019 Acked 7018 Read 1 New Done Done true
18558 6020 Acked 7019 Write 2 New Done Done true
18559 6021 Acked 7020 Write 6 New Done Done true
18560 6022 Acked 7021 Write 5 New Done Done true
18561 6023 Acked 7022 Read 4 New Done Done true
18562 6024 Acked 7023 Write 2 New Done Done true
18563 6025 Acked 7024 Write 8 New Done Done true
18564 6026 Acked 7025 Read 3 New Done Done true
18565 6027 Acked 7026 Write 9 New Done Done true
18566 6028 Acked 7027 Read 1 New Done Done true
18567 6029 Acked 7028 Flush 0 New Done Done true
18568 STATES DS:0 DS:1 DS:2 TOTAL
18569 New 302 0 0 302
18570 Sent 0 0 0 0
18571 Done 0 302 302 604
18572 Skipped 0 0 0 0
18573 Error 0 0 0 0
18574 Last Flush: 6726 7028 7028
18575 Downstairs last five completed: 6726 6725 6724 6723 6722
18576 Upstairs last five completed: 6029 6028 6027 6026 6025
18577 CLIENT: Up:0 ds:302 act:2
185782024-02-12T01:38:09.540ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
185792024-02-12T01:38:09.540ZINFOcrucible: ds_connection connected = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
185802024-02-12T01:38:09.541ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
185812024-02-12T01:38:09.541ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
185822024-02-12T01:38:09.541ZINFOcrucible: send last flush ID to this DS: 6726 = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
185832024-02-12T01:38:09.541ZINFOcrucible: Replied this last flush ID: 6726 = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
185842024-02-12T01:38:09.541ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
185852024-02-12T01:38:09.541ZINFOcrucible: [0] Transition from Offline to Replay = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
185862024-02-12T01:38:09.541ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
185872024-02-12T01:38:09.541ZINFOcrucible: [0] Transition from Replay to Active = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
18588 ----------------------------------------------------------------
18589 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
18590 Downstairs last five completed: 7028 7027 7026 7025 7024
18591 Upstairs last five completed: 6029 6028 6027 6026 6025
18592 CLIENT: Up:0 ds:0 act:3
18593 Replay: All jobs finished, all DS active.
18594 Main task has work to do, go find it
18595 got dsc Stop(1)
18596 stop 1
18597 Feb 12 01:38:12.877[8820] Got stop action so:false kr:false
18598 INFO request completed, latency_us: 414, response_code: 204, uri: /stop/cid/1, method: POST, req_id: f0018aee-098b-43ff-9398-9cfc75ea28ac, remote_addr: 127.0.0.1:60456, local_addr: 127.0.0.1:9998
185992024-02-12T01:38:12.878ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
18600 Feb 12 01:38:12.878 {"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2024-02-12T01:38:12.878126719Z","hostname":"ip-10-150-1-104.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"e1773b3e-b36f-4080-8b13-c0830f0d072e"}
18601 INFO request completed, latency_us: 121, response_code: 200, uri: {"msg":"downstairs task for 1 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-02-12T01:38:12.878159135Z","hostname":"ip-10-150-1-104.us-west-2.compute.internal","pid":1164,"session_id":"e1773b3e-b36f-4080-8b13-c0830f0d072e"}
18602 /state/cid/1, method: GET, req_id: bfe2cba3-2c95-478c-b483-b829581da91f, {"msg":"Gone missing, transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-02-12T01:38:12.878185571Z","hostname":"ip-10-150-1-104.us-west-2.compute.internal","pid":1164,"client":"1","":"downstairs","session_id":"e1773b3e-b36f-4080-8b13-c0830f0d072e"}
18603 remote_addr: 127.0.0.1:{"msg":"[1] client re-new 0 jobs since flush 7028","v":0,"name":"crucible","level":30,"time":"2024-02-12T01:38:12.878208767Z","hostname":"ip-10-150-1-104.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"e1773b3e-b36f-4080-8b13-c0830f0d072e"}
18604 60456, local_addr: 127.0.0.1:9998
186052024-02-12T01:38:12.878ZINFOcrucible: client task is exiting = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
18606 [8820] Exited with: ExitStatus(unix_wait_status(9))
18607 [8820][1] reports Exit
186082024-02-12T01:38:13.879ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
186092024-02-12T01:38:13.879ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
186102024-02-12T01:38:13.879ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
186112024-02-12T01:38:13.879ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
186122024-02-12T01:38:13.879ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
186132024-02-12T01:38:13.879ZINFOcrucible: [1] client re-new 0 jobs since flush 7028 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
186142024-02-12T01:38:13.879ZINFOcrucible: client task is exiting = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
186152024-02-12T01:38:14.881ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
186162024-02-12T01:38:14.881ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
186172024-02-12T01:38:14.881ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
186182024-02-12T01:38:14.881ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
186192024-02-12T01:38:14.881ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
186202024-02-12T01:38:14.881ZINFOcrucible: [1] client re-new 0 jobs since flush 7028 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
186212024-02-12T01:38:14.881ZINFOcrucible: client task is exiting = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
186222024-02-12T01:38:15.882ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
186232024-02-12T01:38:15.882ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
186242024-02-12T01:38:15.882ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
186252024-02-12T01:38:15.882ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
186262024-02-12T01:38:15.882ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
186272024-02-12T01:38:15.882ZINFOcrucible: [1] client re-new 0 jobs since flush 7028 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
186282024-02-12T01:38:15.882ZINFOcrucible: client task is exiting = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
18629 Feb 12 01:38:16.880 INFO request completed, latency_us: 253, response_code: 200, uri: /state/cid/1, method: GET, req_id: d7bceef2-9b07-450b-a1dc-49026beba95c, remote_addr: 127.0.0.1:60456, local_addr: 127.0.0.1:9998
18630 001/300 Read block 477 len 36864
18631 002/300 Read block 3246 len 28672
186322024-02-12T01:38:16.883ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
186332024-02-12T01:38:16.883ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
186342024-02-12T01:38:16.883ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
186352024-02-12T01:38:16.883ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
186362024-02-12T01:38:16.883ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
186372024-02-12T01:38:16.883ZINFOcrucible: [1] client re-new 2 jobs since flush 7028 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
186382024-02-12T01:38:16.883ZINFOcrucible: client task is exiting = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
18639 003/300 Write block 2718 len 12288 data: 4 5 5
18640 004/300 Write block 202 len 28672 data: 3 3 3 4 4 4 4
18641 005/300 Write block 2130 len 40960 data: 7 9 7 7 4 3 3 3 3 3
18642 006/300 Read block 2326 len 4096
18643 007/300 Flush
18644 008/300 Write block 895 len 32768 data: 4 4 3 3 4 4 3 4
18645 009/300 Write block 201 len 36864 data: 5 4 4 4 5 5 5 5 3
18646 010/300 Read block 2828 len 4096
18647 011/300 Read block 680 len 8192
18648 012/300 Read block 3044 len 28672
18649 013/300 Write block 1440 len 28672 data: 3 3 3 3 3 3 4
18650 014/300 Read block 874 len 28672
18651 015/300 Write block 4318 len 12288 data: 5 9 6
18652 016/300 Read block 3301 len 12288
18653 017/300 Read block 3292 len 8192
18654 018/300 Read block 1671 len 16384
18655 019/300 Write block 823 len 8192 data: 3 5
18656 020/300 Flush
18657 021/300 Write block 51 len 12288 data: 3 2 4
18658 022/300 Read block 601 len 8192
18659 023/300 Write block 3299 len 16384 data: 3 3 2 2
18660 024/300 Read block 3958 len 16384
18661 025/300 Write block 3055 len 28672 data: 4 5 4 4 4 4 4
18662 026/300 Flush
18663 027/300 Write block 3575 len 32768 data: 7 6 5 5 4 4 3 3
18664 028/300 Read block 2729 len 40960
18665 029/300 Write block 3465 len 12288 data: 4 4 4
18666 030/300 Read block 4981 len 32768
18667 031/300 Read block 4840 len 28672
18668 032/300 Write block 1126 len 16384 data: 5 5 6 4
18669 033/300 Write block 3650 len 28672 data: 4 4 3 3 5 5 5
18670 034/300 Read block 3820 len 4096
18671 035/300 Read block 2950 len 12288
18672 036/300 Read block 3234 len 36864
18673 037/300 Flush
18674 038/300 Flush
18675 039/300 Flush
18676 040/300 Flush
18677 041/300 Write block 458 len 8192 data: 2 3
18678 042/300 Read block 2152 len 36864
18679 043/300 Read block 1361 len 32768
18680 044/300 Write block 413 len 28672 data: 5 4 3 3 3 3 5
18681 045/300 Flush
18682 046/300 Write block 3037 len 36864 data: 5 6 6 6 7 7 4 4 3
18683 047/300 Read block 2223 len 32768
18684 048/300 Read block 333 len 28672
18685 049/300 Read block 1951 len 28672
18686 050/300 Read block 2746 len 36864
18687 051/300 Read block 2721 len 16384
18688 052/300 Write block 3063 len 12288 data: 4 2 2
18689 053/300 Flush
18690 054/300 Read block 3087 len 32768
18691 055/300 Read block 125 len 36864
18692 056/300 Read block 4555 len 28672
18693 057/300 Read block 2863 len 40960
18694 058/300 Write block 255 len 8192 data: 6 7
18695 059/300 Write block 4746 len 32768 data: 4 2 3 3 3 3 3 3
18696 060/300 Read block 676 len 32768
18697 061/300 Write block 1417 len 24576 data: 2 2 3 4 4 6
18698 062/300 Read block 2003 len 16384
18699 063/300 Read block 556 len 32768
18700 064/300 Read block 333 len 24576
18701 065/300 Read block 1553 len 28672
18702 066/300 Read block 3641 len 12288
18703 067/300 Write block 3150 len 36864 data: 5 4 4 6 5 5 4 4 4
18704 068/300 Write block 4301 len 36864 data: 4 3 3 3 3 4 3 5 4
18705 069/300 Read block 2993 len 28672
18706 070/300 Read block 789 len 32768
18707 071/300 Read block 1002 len 24576
18708 072/300 Read block 602 len 4096
18709 073/300 Read block 4219 len 40960
18710 074/300 Read block 1949 len 36864
18711 075/300 Read block 55 len 8192
18712 076/300 Read block 3875 len 32768
18713 077/300 Flush
18714 078/300 Write block 2047 len 32768 data: 5 5 4 4 3 3 2 3
18715 079/300 Read block 127 len 4096
18716 080/300 Read block 3775 len 36864
18717 081/300 Flush
18718 082/300 Write block 205 len 40960 data: 6 6 6 6 4 3 3 4 3 3
18719 083/300 Read block 819 len 32768
18720 084/300 Read block 1759 len 16384
18721 085/300 Write block 1993 len 40960 data: 4 5 3 3 5 5 3 3 4 5
18722 086/300 Read block 154 len 20480
18723 087/300 Write block 3791 len 16384 data: 6 6 7 5
18724 088/300 Write block 2010 len 28672 data: 3 3 6 7 7 7 7
18725 089/300 Write block 4169 len 16384 data: 6 6 7 8
18726 090/300 Read block 1299 len 24576
18727 091/300 Write block 1364 len 24576 data: 6 6 4 3 2 2
18728 092/300 Write block 2099 len 40960 data: 4 3 3 2 2 4 4 5 5 4
18729 093/300 Write block 1483 len 36864 data: 7 5 4 3 3 3 2 2 2
18730 094/300 Write block 733 len 12288 data: 4 3 3
18731 095/300 Read block 1951 len 28672
18732 096/300 Read block 2680 len 12288
18733 097/300 Read block 3931 len 28672
18734 098/300 Read block 2045 len 12288
18735 099/300 Write block 4526 len 36864 data: 2 2 2 2 2 3 4 4 4
18736 100/300 Write block 434 len 36864 data: 4 5 4 3 3 3 6 6 5
18737 101/300 Write block 77 len 36864 data: 3 3 3 3 3 4 5 6 5
18738 102/300 Write block 3110 len 28672 data: 4 4 5 6 7 6 6
18739 103/300 Write block 821 len 12288 data: 2 4 4
18740 104/300 Read block 2679 len 28672
18741 105/300 Read block 752 len 36864
18742 106/300 Write block 626 len 8192 data: 3 3
18743 107/300 Write block 2374 len 40960 data: 3 3 6 6 7 6 6 4 5 6
18744 108/300 Read block 2362 len 36864
18745 109/300 Write block 1435 len 16384 data: 3 4 4 4
18746 110/300 Write block 3761 len 12288 data: 4 4 3
18747 111/300 Write block 4209 len 20480 data: 3 2 3 3 3
18748 112/300 Read block 2803 len 36864
18749 113/300 Write block 2076 len 20480 data: 3 4 5 4 3
18750 114/300 Read block 3995 len 28672
18751 115/300 Write block 1000 len 12288 data: 7 6 6
18752 116/300 Write block 58 len 40960 data: 4 3 3 4 4 3 4 4 4 3
18753 117/300 Read block 1170 len 8192
18754 118/300 Write block 3888 len 28672 data: 4 4 4 5 5 5 6
18755 119/300 Read block 2730 len 32768
18756 120/300 Read block 1786 len 16384
18757 121/300 Read block 1394 len 12288
18758 122/300 Read block 1155 len 20480
18759 123/300 Write block 3280 len 36864 data: 4 6 6 6 6 7 7 5 5
18760 124/300 Read block 2380 len 36864
18761 125/300 Read block 4017 len 4096
18762 126/300 Write block 1048 len 16384 data: 5 5 5 6
18763 127/300 Read block 1402 len 24576
18764 128/300 Write block 4240 len 40960 data: 3 3 3 3 3 4 5 5 5 6
18765 129/300 Write block 2634 len 32768 data: 6 6 6 5 4 3 4 4
18766 130/300 Flush
18767 131/300 Read block 3834 len 12288
18768 132/300 Write block 2120 len 40960 data: 4 5 5 5 4 5 6 7 7 7
18769 133/300 Flush
18770 134/300 Read block 1360 len 24576
18771 135/300 Write block 4547 len 36864 data: 5 6 8 6 7 8 6 7 3
18772 136/300 Read block 1094 len 12288
18773 137/300 Write block 1643 len 32768 data: 4 4 5 3 3 3 3 3
18774 138/300 Write block 4584 len 24576 data: 4 3 4 4 4 3
18775 139/300 Read block 2580 len 28672
18776 140/300 Read block 578 len 20480
18777 141/300 Write block 3938 len 4096 data: 4
18778 142/300 Read block 2337 len 8192
18779 143/300 Write block 2370 len 12288 data: 3 3 4
18780 144/300 Read block 3806 len 12288
18781 145/300 Write block 1004 len 40960 data: 6 5 5 4 3 3 3 2 2 2
18782 146/300 Read block 1599 len 32768
18783 147/300 Write block 3367 len 12288 data: 3 3 4
18784 148/300 Read block 2423 len 12288
18785 149/300 Write block 333 len 16384 data: 3 4 7 7
18786 150/300 Flush
18787 151/300 Read block 2685 len 40960
18788 152/300 Flush
18789 153/300 Flush
18790 154/300 Write block 4506 len 12288 data: 7 7 6
18791 155/300 Write block 2946 len 20480 data: 2 3 3 3 3
18792 156/300 Read block 2782 len 20480
18793 157/300 Write block 2891 len 40960 data: 2 2 2 5 6 7 9 9 7 6
18794 158/300 Read block 1253 len 8192
18795 159/300 Write block 518 len 20480 data: 3 3 3 3 3
18796 160/300 Flush
18797 161/300 Write block 3938 len 40960 data: 5 4 4 5 5 5 6 7 6 6
18798 162/300 Write block 3212 len 40960 data: 3 2 2 2 2 2 2 3 5 4
18799 163/300 Write block 2879 len 32768 data: 4 4 5 5 5 5 6 5
18800 164/300 Write block 502 len 4096 data: 5
18801 165/300 Read block 1392 len 16384
18802 166/300 Read block 582 len 32768
18803 167/300 Read block 4026 len 40960
18804 168/300 Read block 2077 len 8192
18805 169/300 Write block 3175 len 32768 data: 4 4 6 6 7 9 9 7
18806 170/300 Read block 382 len 32768
18807 171/300 Read block 4827 len 4096
18808 172/300 Read block 2067 len 40960
18809 173/300 Read block 2115 len 40960
18810 174/300 Read block 3320 len 24576
18811 175/300 Flush
18812 176/300 Write block 149 len 20480 data: 3 4 5 4 3
18813 177/300 Read block 4303 len 8192
18814 178/300 Flush
18815 179/300 Write block 3969 len 12288 data: 3 3 3
18816 180/300 Read block 4836 len 32768
18817 181/300 Flush
18818 182/300 Read block 409 len 36864
18819 183/300 Write block 3520 len 36864 data: 6 5 6 4 4 4 4 4 4
18820 184/300 Read block 1050 len 12288
18821 185/300 Read block 1607 len 28672
18822 186/300 Write block 1862 len 20480 data: 7 6 4 4 3
18823 187/300 Read block 2714 len 32768
18824 188/300 Write block 2584 len 36864 data: 7 7 8 8 9 7 7 7 6
18825 189/300 Flush
18826 190/300 Read block 3514 len 36864
18827 191/300 Read block 3984 len 8192
18828 192/300 Write block 1863 len 8192 data: 7 5
18829 193/300 Read block 4691 len 20480
18830 194/300 Write block 1516 len 12288 data: 4 4 4
18831 195/300 Write block 4285 len 36864 data: 7 6 7 6 5 6 6 6 5
18832 196/300 Read block 2641 len 8192
18833 197/300 Read block 2223 len 16384
18834 198/300 Write block 3383 len 40960 data: 4 4 4 4 3 3 3 4 4 4
18835 199/300 Read block 2485 len 40960
18836 200/300 Write block 3314 len 28672 data: 6 7 6 4 3 3 3
18837 201/300 Write block 4021 len 40960 data: 8 11 9 8 6 6 6 5 6 5
18838 202/300 Write block 2044 len 16384 data: 4 6 5 6
18839 203/300 Write block 4323 len 28672 data: 4 4 6 6 7 6 4
18840 204/300 Read block 921 len 24576
18841 205/300 Write block 3456 len 32768 data: 5 4 5 4 4 3 3 3
18842 206/300 Write block 3524 len 32768 data: 5 5 5 5 5 3 4 4
18843 207/300 Read block 1615 len 24576
18844 208/300 Write block 1865 len 28672 data: 5 4 3 3 2 3 3
18845 209/300 Read block 1228 len 24576
18846 210/300 Write block 3127 len 12288 data: 5 5 5
18847 211/300 Read block 3926 len 40960
18848 212/300 Read block 2490 len 20480
18849 213/300 Read block 4559 len 16384
18850 214/300 Read block 1925 len 32768
18851 215/300 Write block 3795 len 28672 data: 5 4 4 2 2 2 3
18852 216/300 Write block 4857 len 40960 data: 2 2 2 2 2 2 4 4 4 3
18853 217/300 Write block 1996 len 8192 data: 4 6
18854 218/300 Read block 2538 len 28672
18855 219/300 Read block 1312 len 16384
18856 220/300 Read block 3310 len 8192
18857 221/300 Write block 1377 len 24576 data: 4 4 4 6 7 6
18858 222/300 Write block 4193 len 20480 data: 3 3 3 2 3
18859 223/300 Read block 3462 len 8192
18860 224/300 Write block 4772 len 16384 data: 5 5 4 6
18861 225/300 Read block 4035 len 40960
18862 226/300 Write block 75 len 20480 data: 4 4 4 4 4
18863 227/300 Read block 4852 len 4096
18864 228/300 Read block 3301 len 8192
18865 229/300 Write block 1367 len 36864 data: 4 3 3 2 2 2 2 2 2
18866 230/300 Flush
18867 231/300 Write block 281 len 24576 data: 3 3 3 3 3 3
18868 232/300 Read block 4512 len 8192
18869 233/300 Read block 116 len 8192
18870 234/300 Read block 3720 len 8192
18871 235/300 Read block 2658 len 4096
18872 236/300 Read block 3668 len 16384
18873 237/300 Write block 1238 len 24576 data: 3 3 2 3 3 4
18874 238/300 Read block 218 len 12288
18875 239/300 Write block 1965 len 40960 data: 4 5 6 5 5 5 6 5 5 5
18876 240/300 Write block 1139 len 24576 data: 4 6 5 7 6 7
18877 241/300 Write block 1944 len 8192 data: 3 3
18878 242/300 Read block 3684 len 16384
18879 243/300 Write block 4128 len 32768 data: 4 4 4 4 3 2 4 5
18880 244/300 Read block 622 len 20480
18881 245/300 Write block 1421 len 36864 data: 5 7 7 6 6 4 4 3 3
18882 246/300 Read block 1063 len 20480
18883 247/300 Write block 3140 len 40960 data: 5 6 7 6 5 5 5 5 5 6
18884 248/300 Read block 4950 len 32768
18885 249/300 Write block 1950 len 20480 data: 3 4 3 3 3
18886 250/300 Flush
18887 251/300 Read block 82 len 24576
18888 252/300 Write block 2433 len 32768 data: 2 3 3 3 4 3 3 4
18889 253/300 Write block 3336 len 16384 data: 4 4 3 4
18890 254/300 Write block 1410 len 36864 data: 6 6 4 3 3 4 3 3 3
18891 255/300 Write block 4418 len 20480 data: 5 4 4 4 4
18892 256/300 Read block 1130 len 16384
18893 257/300 Write block 3746 len 28672 data: 6 4 6 6 6 5 4
18894 258/300 Read block 4738 len 40960
18895 259/300 Write block 4920 len 16384 data: 5 4 5 6
18896 260/300 Read block 4895 len 28672
18897 261/300 Read block 2348 len 40960
18898 262/300 Read block 447 len 20480
18899 263/300 Read block 4849 len 40960
18900 264/300 Read block 1492 len 12288
18901 265/300 Read block 2738 len 4096
18902 266/300 Write block 3321 len 8192 data: 3 2
18903 267/300 Flush
18904 268/300 Read block 4653 len 8192
18905 269/300 Read block 1638 len 32768
18906 270/300 Read block 3563 len 36864
18907 271/300 Write block 1103 len 24576 data: 5 6 6 4 5 4
18908 272/300 Write block 1530 len 28672 data: 5 6 6 7 9 8 9
18909 273/300 Write block 1702 len 20480 data: 5 6 6 7 5
18910 274/300 Read block 4480 len 12288
18911 275/300 Read block 403 len 4096
18912 276/300 Read block 1013 len 8192
18913 277/300 Read block 376 len 24576
18914 278/300 Read block 4555 len 20480
18915 279/300 Read block 2267 len 36864
18916 280/300 Write block 7 len 32768 data: 4 4 4 5 6 5 4 4
18917 281/300 Flush
18918 282/300 Write block 2856 len 12288 data: 7 6 6
18919 283/300 Read block 3208 len 36864
18920 284/300 Read block 4203 len 16384
18921 285/300 Write block 1679 len 28672 data: 8 7 6 7 5 5 3
18922 286/300 Flush
18923 287/300 Write block 555 len 16384 data: 3 3 4 5
18924 288/300 Write block 4460 len 24576 data: 5 4 3 3 2 2
18925 289/300 Read block 4067 len 20480
18926 290/300 Write block 1450 len 16384 data: 6 6 6 5
18927 291/300 Flush
18928 292/300 Read block 777 len 4096
18929 293/300 Read block 1489 len 4096
18930 294/300 Read block 611 len 4096
18931 295/300 Write block 4117 len 24576 data: 5 6 6 3 3 4
18932 296/300 Write block 1124 len 28672 data: 7 5 6 6 7 5 4
18933 297/300 Write block 4185 len 36864 data: 4 5 4 3 3 2 3 3 4
18934 298/300 Read block 3278 len 4096
18935 299/300 Read block 3179 len 8192
18936 300/300 Read block 2788 len 16384
18937 Main task has work to do, go find it
18938 got dsc Start(1)
18939 start 1
18940 Feb 12 01:38:17.234 INFO request completed, latency_us: 250, response_code: 204, uri: /start/cid/1, method: POST, req_id: b3ba2401-22e1-4f7d-83d3-38e63cddab32, remote_addr: 127.0.0.1:60456, local_addr: 127.0.0.1:9998
18941 Replay: started 1, returned:Ok(())
18942 ----------------------------------------------------------------
18943 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
18944 States: Active Offline Active
18945 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
18946 6030 Acked 7029 Read 9 Done New Done true
18947 6031 Acked 7030 Read 7 Done New Done true
18948 6032 Acked 7031 Write 3 Done New Done false
18949 6033 Acked 7032 Write 7 Done New Done false
18950 6034 Acked 7033 Write 10 Done New Done false
18951 6035 Acked 7034 Read 1 Done New Done false
18952 6036 Acked 7035 Flush 0 Done New Done false
18953 6037 Acked 7036 Write 8 Done New Done false
18954 6038 Acked 7037 Write 9 Done New Done false
18955 6039 Acked 7038 Read 1 Done New Done false
18956 6040 Acked 7039 Read 2 Done New Done false
18957 6041 Acked 7040 Read 7 Done New Done false
18958 6042 Acked 7041 Write 7 Done New Done false
18959 6043 Acked 7042 Read 7 Done New Done false
18960 6044 Acked 7043 Write 3 Done New Done false
18961 6045 Acked 7044 Read 3 Done New Done false
18962 6046 Acked 7045 Read 2 Done New Done false
18963 6047 Acked 7046 Read 4 Done New Done false
18964 6048 Acked 7047 Write 2 Done New Done false
18965 6049 Acked 7048 Flush 0 Done New Done false
18966 6050 Acked 7049 Write 3 Done New Done false
18967 6051 Acked 7050 Read 2 Done New Done false
18968 6052 Acked 7051 Write 4 Done New Done false
18969 6053 Acked 7052 Read 4 Done New Done false
18970 6054 Acked 7053 Write 7 Done New Done false
18971 6055 Acked 7054 Flush 0 Done New Done false
18972 6056 Acked 7055 Write 8 Done New Done false
18973 6057 Acked 7056 Read 10 Done New Done false
18974 6058 Acked 7057 Write 3 Done New Done false
18975 6059 Acked 7058 Read 8 Done New Done false
18976 6060 Acked 7059 Read 7 Done New Done false
18977 6061 Acked 7060 Write 4 Done New Done false
18978 6062 Acked 7061 Write 7 Done New Done false
18979 6063 Acked 7062 Read 1 Done New Done false
18980 6064 Acked 7063 Read 3 Done New Done false
18981 6065 Acked 7064 Read 9 Done New Done false
18982 6066 Acked 7065 Flush 0 Done New Done false
18983 6067 Acked 7066 Flush 0 Done New Done false
18984 6068 Acked 7067 Flush 0 Done New Done false
18985 6069 Acked 7068 Flush 0 Done New Done false
18986 6070 Acked 7069 Write 2 Done New Done false
18987 6071 Acked 7070 Read 9 Done New Done false
18988 6072 Acked 7071 Read 8 Done New Done false
18989 6073 Acked 7072 Write 7 Done New Done false
18990 6074 Acked 7073 Flush 0 Done New Done false
18991 6075 Acked 7074 Write 9 Done New Done false
18992 6076 Acked 7075 Read 8 Done New Done false
18993 6077 Acked 7076 Read 7 Done New Done false
18994 6078 Acked 7077 Read 7 Done New Done false
18995 6079 Acked 7078 Read 9 Done New Done false
18996 6080 Acked 7079 Read 4 Done New Done false
18997 6081 Acked 7080 Write 3 Done New Done false
18998 6082 Acked 7081 Flush 0 Done New Done false
18999 6083 Acked 7082 Read 8 Done New Done false
19000 6084 Acked 7083 Read 9 Done New Done false
19001 6085 Acked 7084 Read 7 Done New Done false
19002 6086 Acked 7085 Read 10 Done New Done false
19003 6087 Acked 7086 Write 2 Done New Done false
19004 6088 Acked 7087 Write 8 Done New Done false
19005 6089 Acked 7088 Read 8 Done New Done false
19006 6090 Acked 7089 Write 6 Done New Done false
19007 6091 Acked 7090 Read 4 Done New Done false
19008 6092 Acked 7091 Read 8 Done New Done false
19009 6093 Acked 7092 Read 6 Done New Done false
19010 6094 Acked 7093 Read 7 Done New Done false
19011 6095 Acked 7094 Read 3 Done New Done false
19012 6096 Acked 7095 Write 9 Done New Done false
19013 6097 Acked 7096 Write 9 Done New Done false
19014 6098 Acked 7097 Read 7 Done New Done false
19015 6099 Acked 7098 Read 8 Done New Done false
19016 6100 Acked 7099 Read 6 Done New Done false
19017 6101 Acked 7100 Read 1 Done New Done false
19018 6102 Acked 7101 Read 10 Done New Done false
19019 6103 Acked 7102 Read 9 Done New Done false
19020 6104 Acked 7103 Read 2 Done New Done false
19021 6105 Acked 7104 Read 8 Done New Done false
19022 6106 Acked 7105 Flush 0 Done New Done false
19023 6107 Acked 7106 Write 8 Done New Done false
19024 6108 Acked 7107 Read 1 Done New Done false
19025 6109 Acked 7108 Read 9 Done New Done false
19026 6110 Acked 7109 Flush 0 Done New Done false
19027 6111 Acked 7110 Write 10 Done New Done false
19028 6112 Acked 7111 Read 8 Done New Done false
19029 6113 Acked 7112 Read 4 Done New Done false
19030 6114 Acked 7113 Write 10 Done New Done false
19031 6115 Acked 7114 Read 5 Done New Done false
19032 6116 Acked 7115 Write 4 Done New Done false
19033 6117 Acked 7116 Write 7 Done New Done false
19034 6118 Acked 7117 Write 4 Done New Done false
19035 6119 Acked 7118 Read 6 Done New Done false
19036 6120 Acked 7119 Write 6 Done New Done false
19037 6121 Acked 7120 Write 10 Done New Done false
19038 6122 Acked 7121 Write 9 Done New Done false
19039 6123 Acked 7122 Write 3 Done New Done false
19040 6124 Acked 7123 Read 7 Done New Done false
19041 6125 Acked 7124 Read 3 Done New Done false
19042 6126 Acked 7125 Read 7 Done New Done false
19043 6127 Acked 7126 Read 3 Done New Done false
19044 6128 Acked 7127 Write 9 Done New Done false
19045 6129 Acked 7128 Write 9 Done New Done false
19046 6130 Acked 7129 Write 9 Done New Done false
19047 6131 Acked 7130 Write 7 Done New Done false
19048 6132 Acked 7131 Write 3 Done New Done false
19049 6133 Acked 7132 Read 7 Done New Done false
19050 6134 Acked 7133 Read 9 Done New Done false
19051 6135 Acked 7134 Write 2 Done New Done false
19052 6136 Acked 7135 Write 10 Done New Done false
19053 6137 Acked 7136 Read 9 Done New Done false
19054 6138 Acked 7137 Write 4 Done New Done false
19055 6139 Acked 7138 Write 3 Done New Done false
19056 6140 Acked 7139 Write 5 Done New Done false
19057 6141 Acked 7140 Read 9 Done New Done false
19058 6142 Acked 7141 Write 5 Done New Done false
19059 6143 Acked 7142 Read 7 Done New Done false
19060 6144 Acked 7143 Write 3 Done New Done false
19061 6145 Acked 7144 Write 10 Done New Done false
19062 6146 Acked 7145 Read 2 Done New Done false
19063 6147 Acked 7146 Write 7 Done New Done false
19064 6148 Acked 7147 Read 8 Done New Done false
19065 6149 Acked 7148 Read 4 Done New Done false
19066 6150 Acked 7149 Read 3 Done New Done false
19067 6151 Acked 7150 Read 5 Done New Done false
19068 6152 Acked 7151 Write 9 Done New Done false
19069 6153 Acked 7152 Read 9 Done New Done false
19070 6154 Acked 7153 Read 1 Done New Done false
19071 6155 Acked 7154 Write 4 Done New Done false
19072 6156 Acked 7155 Read 6 Done New Done false
19073 6157 Acked 7156 Write 10 Done New Done false
19074 6158 Acked 7157 Write 8 Done New Done false
19075 6159 Acked 7158 Flush 0 Done New Done false
19076 6160 Acked 7159 Read 3 Done New Done false
19077 6161 Acked 7160 Write 10 Done New Done false
19078 6162 Acked 7161 Flush 0 Done New Done false
19079 6163 Acked 7162 Read 6 Done New Done false
19080 6164 Acked 7163 Write 9 Done New Done false
19081 6165 Acked 7164 Read 3 Done New Done false
19082 6166 Acked 7165 Write 8 Done New Done false
19083 6167 Acked 7166 Write 6 Done New Done false
19084 6168 Acked 7167 Read 7 Done New Done false
19085 6169 Acked 7168 Read 5 Done New Done false
19086 6170 Acked 7169 Write 1 Done New Done false
19087 6171 Acked 7170 Read 2 Done New Done false
19088 6172 Acked 7171 Write 3 Done New Done false
19089 6173 Acked 7172 Read 3 Done New Done false
19090 6174 Acked 7173 Write 10 Done New Done false
19091 6175 Acked 7174 Read 8 Done New Done false
19092 6176 Acked 7175 Write 3 Done New Done false
19093 6177 Acked 7176 Read 3 Done New Done false
19094 6178 Acked 7177 Write 4 Done New Done false
19095 6179 Acked 7178 Flush 0 Done New Done false
19096 6180 Acked 7179 Read 10 Done New Done false
19097 6181 Acked 7180 Flush 0 Done New Done false
19098 6182 Acked 7181 Flush 0 Done New Done false
19099 6183 Acked 7182 Write 3 Done New Done false
19100 6184 Acked 7183 Write 5 Done New Done false
19101 6185 Acked 7184 Read 5 Done New Done false
19102 6186 Acked 7185 Write 10 Done New Done false
19103 6187 Acked 7186 Read 2 Done New Done false
19104 6188 Acked 7187 Write 5 Done New Done false
19105 6189 Acked 7188 Flush 0 Done New Done false
19106 6190 Acked 7189 Write 10 Done New Done false
19107 6191 Acked 7190 Write 10 Done New Done false
19108 6192 Acked 7191 Write 8 Done New Done false
19109 6193 Acked 7192 Write 1 Done New Done false
19110 6194 Acked 7193 Read 4 Done New Done false
19111 6195 Acked 7194 Read 8 Done New Done false
19112 6196 Acked 7195 Read 10 Done New Done false
19113 6197 Acked 7196 Read 2 Done New Done false
19114 6198 Acked 7197 Write 8 Done New Done false
19115 6199 Acked 7198 Read 8 Done New Done false
19116 6200 Acked 7199 Read 1 Done New Done false
19117 6201 Acked 7200 Read 10 Done New Done false
19118 6202 Acked 7201 Read 10 Done New Done false
19119 6203 Acked 7202 Read 6 Done New Done false
19120 6204 Acked 7203 Flush 0 Done New Done false
19121 6205 Acked 7204 Write 5 Done New Done false
19122 6206 Acked 7205 Read 2 Done New Done false
19123 6207 Acked 7206 Flush 0 Done New Done false
19124 6208 Acked 7207 Write 3 Done New Done false
19125 6209 Acked 7208 Read 8 Done New Done false
19126 6210 Acked 7209 Flush 0 Done New Done false
19127 6211 Acked 7210 Read 9 Done New Done false
19128 6212 Acked 7211 Write 9 Done New Done false
19129 6213 Acked 7212 Read 3 Done New Done false
19130 6214 Acked 7213 Read 7 Done New Done false
19131 6215 Acked 7214 Write 5 Done New Done false
19132 6216 Acked 7215 Read 8 Done New Done false
19133 6217 Acked 7216 Write 9 Done New Done false
19134 6218 Acked 7217 Flush 0 Done New Done false
19135 6219 Acked 7218 Read 9 Done New Done false
19136 6220 Acked 7219 Read 2 Done New Done false
19137 6221 Acked 7220 Write 2 Done New Done false
19138 6222 Acked 7221 Read 5 Done New Done false
19139 6223 Acked 7222 Write 3 Done New Done false
19140 6224 Acked 7223 Write 9 Done New Done false
19141 6225 Acked 7224 Read 2 Done New Done false
19142 6226 Acked 7225 Read 4 Done New Done false
19143 6227 Acked 7226 Write 10 Done New Done false
19144 6228 Acked 7227 Read 10 Done New Done false
19145 6229 Acked 7228 Write 7 Done New Done false
19146 6230 Acked 7229 Write 10 Done New Done false
19147 6231 Acked 7230 Write 4 Done New Done false
19148 6232 Acked 7231 Write 7 Done New Done false
19149 6233 Acked 7232 Read 6 Done New Done false
19150 6234 Acked 7233 Write 8 Done New Done false
19151 6235 Acked 7234 Write 8 Done New Done false
19152 6236 Acked 7235 Read 6 Done New Done false
19153 6237 Acked 7236 Write 7 Done New Done false
19154 6238 Acked 7237 Read 6 Done New Done false
19155 6239 Acked 7238 Write 3 Done New Done false
19156 6240 Acked 7239 Read 10 Done New Done false
19157 6241 Acked 7240 Read 5 Done New Done false
19158 6242 Acked 7241 Read 4 Done New Done false
19159 6243 Acked 7242 Read 8 Done New Done false
19160 6244 Acked 7243 Write 7 Done New Done false
19161 6245 Acked 7244 Write 10 Done New Done false
19162 6246 Acked 7245 Write 2 Done New Done false
19163 6247 Acked 7246 Read 7 Done New Done false
19164 6248 Acked 7247 Read 4 Done New Done false
19165 6249 Acked 7248 Read 2 Done New Done false
19166 6250 Acked 7249 Write 6 Done New Done false
19167 6251 Acked 7250 Write 5 Done New Done false
19168 6252 Acked 7251 Read 2 Done New Done false
19169 6253 Acked 7252 Write 4 Done New Done false
19170 6254 Acked 7253 Read 10 Done New Done false
19171 6255 Acked 7254 Write 5 Done New Done false
19172 6256 Acked 7255 Read 1 Done New Done false
19173 6257 Acked 7256 Read 2 Done New Done false
19174 6258 Acked 7257 Write 9 Done New Done false
19175 6259 Acked 7258 Flush 0 Done New Done false
19176 6260 Acked 7259 Write 6 Done New Done false
19177 6261 Acked 7260 Read 2 Done New Done false
19178 6262 Acked 7261 Read 2 Done New Done false
19179 6263 Acked 7262 Read 2 Done New Done false
19180 6264 Acked 7263 Read 1 Done New Done false
19181 6265 Acked 7264 Read 4 Done New Done false
19182 6266 Acked 7265 Write 6 Done New Done false
19183 6267 Acked 7266 Read 3 Done New Done false
19184 6268 Acked 7267 Write 10 Done New Done false
19185 6269 Acked 7268 Write 6 Done New Done false
19186 6270 Acked 7269 Write 2 Done New Done false
19187 6271 Acked 7270 Read 4 Done New Done false
19188 6272 Acked 7271 Write 8 Done New Done false
19189 6273 Acked 7272 Read 5 Done New Done false
19190 6274 Acked 7273 Write 9 Done New Done false
19191 6275 Acked 7274 Read 5 Done New Done false
19192 6276 Acked 7275 Flush 0 Done New Done false
19193 6277 Acked 7276 Write 10 Done New Done false
19194 6278 Acked 7277 Read 8 Done New Done false
19195 6279 Acked 7278 Write 5 Done New Done false
19196 6280 Acked 7279 Flush 0 Done New Done false
19197 6281 Acked 7280 Read 6 Done New Done false
19198 6282 Acked 7281 Write 8 Done New Done false
19199 6283 Acked 7282 Write 4 Done New Done false
19200 6284 Acked 7283 Write 9 Done New Done false
19201 6285 Acked 7284 Write 5 Done New Done false
19202 6286 Acked 7285 Read 4 Done New Done false
19203 6287 Acked 7286 Write 7 Done New Done false
19204 6288 Acked 7287 Read 10 Done New Done false
19205 6289 Acked 7288 Write 4 Done New Done false
19206 6290 Acked 7289 Read 7 Done New Done false
19207 6291 Acked 7290 Read 10 Done New Done false
19208 6292 Acked 7291 Read 5 Done New Done false
19209 6293 Acked 7292 Read 10 Done New Done false
19210 6294 Acked 7293 Read 3 Done New Done false
19211 6295 Acked 7294 Read 1 Done New Done false
19212 6296 Acked 7295 Write 2 Done New Done false
19213 6297 Acked 7296 Flush 0 Done New Done false
19214 6298 Acked 7297 Read 2 Done New Done false
19215 6299 Acked 7298 Read 8 Done New Done false
19216 6300 Acked 7299 Read 9 Done New Done false
19217 6301 Acked 7300 Write 6 Done New Done false
19218 6302 Acked 7301 Write 7 Done New Done false
19219 6303 Acked 7302 Write 5 Done New Done false
19220 6304 Acked 7303 Read 3 Done New Done false
19221 6305 Acked 7304 Read 1 Done New Done false
19222 6306 Acked 7305 Read 2 Done New Done false
19223 6307 Acked 7306 Read 6 Done New Done false
19224 6308 Acked 7307 Read 5 Done New Done false
19225 6309 Acked 7308 Read 9 Done New Done false
19226 6310 Acked 7309 Write 8 Done New Done false
19227 6311 Acked 7310 Flush 0 Done New Done false
19228 6312 Acked 7311 Write 3 Done New Done false
19229 6313 Acked 7312 Read 9 Done New Done false
19230 6314 Acked 7313 Read 4 Done New Done false
19231 6315 Acked 7314 Write 7 Done New Done false
19232 6316 Acked 7315 Flush 0 Done New Done false
19233 6317 Acked 7316 Write 4 Done New Done false
19234 6318 Acked 7317 Write 6 Done New Done false
19235 6319 Acked 7318 Read 5 Done New Done false
19236 6320 Acked 7319 Write 4 Done New Done false
19237 6321 Acked 7320 Flush 0 Done New Done false
19238 6322 Acked 7321 Read 1 Done New Done false
19239 6323 Acked 7322 Read 1 Done New Done false
19240 6324 Acked 7323 Read 1 Done New Done false
19241 6325 Acked 7324 Write 6 Done New Done false
19242 6326 Acked 7325 Write 7 Done New Done false
19243 6327 Acked 7326 Write 9 Done New Done false
19244 6328 Acked 7327 Read 1 Done New Done false
19245 6329 Acked 7328 Read 2 Done New Done false
19246 6330 Acked 7329 Read 4 Done New Done false
19247 STATES DS:0 DS:1 DS:2 TOTAL
19248 New 0 301 0 301
19249 Sent 0 0 0 0
19250 Done 301 0 301 602
19251 Skipped 0 0 0 0
19252 Error 0 0 0 0
19253 Last Flush: 7320 7028 7320
19254 Downstairs last five completed: 7028 7027 7026 7025 7024
19255 Upstairs last five completed: 6330 6329 6328 6327 6326
19256 CLIENT: Up:0 ds:301 act:2
192572024-02-12T01:38:17.884ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
192582024-02-12T01:38:17.884ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
192592024-02-12T01:38:17.884ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
192602024-02-12T01:38:17.884ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
192612024-02-12T01:38:17.884ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
192622024-02-12T01:38:17.884ZINFOcrucible: [1] client re-new 302 jobs since flush 7028 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
192632024-02-12T01:38:17.884ZINFOcrucible: client task is exiting = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
192642024-02-12T01:38:18.885ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
192652024-02-12T01:38:18.885ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
192662024-02-12T01:38:18.885ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
192672024-02-12T01:38:18.885ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
192682024-02-12T01:38:18.885ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
192692024-02-12T01:38:18.886ZINFOcrucible: [1] client re-new 302 jobs since flush 7028 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
192702024-02-12T01:38:18.886ZINFOcrucible: client task is exiting = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
19271 [8820] got start action
19272 [8820] I am going to restart
19273 Starting downstairs at port 8820
19274 Make output file at "/tmp/dsc/downstairs-8820.txt"
19275 [8820][1] reports Starting
19276 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1185)
19277 [8820][1] reports Running
192782024-02-12T01:38:19.886ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
192792024-02-12T01:38:19.886ZINFOcrucible: ds_connection connected = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
192802024-02-12T01:38:19.888ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
192812024-02-12T01:38:19.888ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
192822024-02-12T01:38:19.888ZINFOcrucible: send last flush ID to this DS: 7028 = downstairs client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
192832024-02-12T01:38:19.888ZINFOcrucible: Replied this last flush ID: 7028 = downstairs client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
192842024-02-12T01:38:19.888ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
192852024-02-12T01:38:19.888ZINFOcrucible: [1] Transition from Offline to Replay = downstairs client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
192862024-02-12T01:38:19.888ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
192872024-02-12T01:38:19.888ZINFOcrucible: [1] Transition from Replay to Active = downstairs client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
19288 ----------------------------------------------------------------
19289 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
19290 Downstairs last five completed: 7330 7329 7328 7327 7326
19291 Upstairs last five completed: 6331 6330 6329 6328 6327
19292 CLIENT: Up:0 ds:0 act:3
19293 Replay: All jobs finished, all DS active.
19294 Main task has work to do, go find it
19295 got dsc Stop(2)
19296 stop 2
19297 Feb 12 01:38:21.237 INFO request completed, latency_us: 382, response_code: 204[8830] Got stop action so:false kr:false
19298 , uri: /stop/cid/2, method: POST, req_id: 1ec67142-16bc-4293-8f27-9d394f6b4675, remote_addr: 127.0.0.1:60456, local_addr: 127.0.0.1:9998
192992024-02-12T01:38:21.238ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
193002024-02-12T01:38:21.238ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
193012024-02-12T01:38:21.238ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
193022024-02-12T01:38:21.238ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
19303 Feb 12 01:38:21.238 INFO request completed, latency_us: 173, response_code: 200, uri: /state/cid/2, method: GET, req_id: 3dd53b55-8e9e-4e1a-9406-9f1c8357f038, remote_addr: 127.0.0.1:60456, local_addr: 127.0.0.1:9998
193042024-02-12T01:38:21.238ZINFOcrucible: [2] client re-new 0 jobs since flush 7330 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
193052024-02-12T01:38:21.238ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
19306 [8830] Exited with: ExitStatus(unix_wait_status(9))
19307 [8830][2] reports Exit
193082024-02-12T01:38:22.240ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
193092024-02-12T01:38:22.240ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
193102024-02-12T01:38:22.240ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
193112024-02-12T01:38:22.240ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
193122024-02-12T01:38:22.240ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
193132024-02-12T01:38:22.240ZINFOcrucible: [2] client re-new 0 jobs since flush 7330 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
193142024-02-12T01:38:22.240ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
193152024-02-12T01:38:23.241ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
193162024-02-12T01:38:23.241ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
193172024-02-12T01:38:23.241ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
193182024-02-12T01:38:23.241ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
193192024-02-12T01:38:23.241ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
193202024-02-12T01:38:23.241ZINFOcrucible: [2] client re-new 0 jobs since flush 7330 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
193212024-02-12T01:38:23.241ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
193222024-02-12T01:38:24.242ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
193232024-02-12T01:38:24.242ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
193242024-02-12T01:38:24.243ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
193252024-02-12T01:38:24.243ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
193262024-02-12T01:38:24.243ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
193272024-02-12T01:38:24.243ZINFOcrucible: [2] client re-new 0 jobs since flush 7330 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
193282024-02-12T01:38:24.243ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
19329 Feb 12 01:38:25.241 INFO request completed, latency_us: 200, response_code: 200, uri: /state/cid/2, method: GET, req_id: 06531d7e-08f7-4a17-830c-b60144424ac3, remote_addr: 127.0.0.1:60456, local_addr: 127.0.0.1:9998
19330 001/300 Write block 1505 len 36864 data: 5 5 4 4 5 4 4 4 4
19331 002/300 Write block 762 len 28672 data: 3 3 4 4 3 3 3
19332 003/300 Read block 4067 len 32768
193332024-02-12T01:38:25.244ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
193342024-02-12T01:38:25.244ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
193352024-02-12T01:38:25.244ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
193362024-02-12T01:38:25.244ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
193372024-02-12T01:38:25.244ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
193382024-02-12T01:38:25.244ZINFOcrucible: [2] client re-new 3 jobs since flush 7330 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
193392024-02-12T01:38:25.244ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
19340 004/300 Flush
19341 005/300 Flush
19342 006/300 Write block 3761 len 12288 data: 5 5 4
19343 007/300 Read block 1765 len 24576
19344 008/300 Read block 3335 len 40960
19345 009/300 Read block 3493 len 32768
19346 010/300 Read block 2934 len 40960
19347 011/300 Read block 2982 len 4096
19348 012/300 Read block 1652 len 4096
19349 013/300 Flush
19350 014/300 Write block 707 len 28672 data: 3 3 3 3 5 5 4
19351 015/300 Read block 1203 len 36864
19352 016/300 Read block 2092 len 40960
19353 017/300 Write block 2818 len 8192 data: 5 5
19354 018/300 Write block 4776 len 4096 data: 4
19355 019/300 Write block 3106 len 4096 data: 4
19356 020/300 Write block 4863 len 20480 data: 5 5 5 4 3
19357 021/300 Read block 992 len 16384
19358 022/300 Read block 4257 len 40960
19359 023/300 Write block 3085 len 36864 data: 4 4 4 3 4 4 3 4 3
19360 024/300 Read block 4068 len 40960
19361 025/300 Write block 4629 len 28672 data: 6 6 6 6 4 4 4
19362 026/300 Flush
19363 027/300 Read block 4694 len 4096
19364 028/300 Read block 80 len 8192
19365 029/300 Write block 4604 len 32768 data: 5 6 5 6 5 5 5 6
19366 030/300 Write block 1464 len 16384 data: 7 6 6 5
19367 031/300 Write block 3605 len 28672 data: 5 5 5 4 3 3 3
19368 032/300 Write block 2949 len 4096 data: 4
19369 033/300 Write block 2580 len 36864 data: 5 5 5 6 8 8 9 9 10
19370 034/300 Read block 2363 len 40960
19371 035/300 Write block 4822 len 8192 data: 5 5
19372 036/300 Flush
19373 037/300 Write block 4285 len 8192 data: 8 7
19374 038/300 Read block 2944 len 32768
19375 039/300 Write block 793 len 24576 data: 6 6 6 7 8 8
19376 040/300 Read block 2764 len 16384
19377 041/300 Read block 1607 len 20480
19378 042/300 Read block 2177 len 4096
19379 043/300 Write block 1548 len 36864 data: 4 4 4 5 6 5 6 5 5
19380 044/300 Read block 1730 len 8192
19381 045/300 Write block 3819 len 32768 data: 5 4 4 4 3 4 3 4
19382 046/300 Write block 4275 len 16384 data: 4 5 4 3
19383 047/300 Read block 4354 len 28672
19384 048/300 Write block 81 len 24576 data: 4 5 6 7 6 6
19385 049/300 Write block 1712 len 32768 data: 6 5 5 4 5 5 4 5
19386 050/300 Read block 380 len 24576
19387 051/300 Write block 2076 len 40960 data: 4 5 6 5 4 2 2 2 2 2
19388 052/300 Read block 4244 len 28672
19389 053/300 Flush
19390 054/300 Write block 4485 len 40960 data: 7 6 5 5 5 5 4 4 4 4
19391 055/300 Read block 1704 len 8192
19392 056/300 Flush
19393 057/300 Read block 2232 len 32768
19394 058/300 Write block 4770 len 40960 data: 5 5 6 6 5 7 5 4 3 5
19395 059/300 Flush
19396 060/300 Read block 727 len 12288
19397 061/300 Read block 1275 len 16384
19398 062/300 Write block 3971 len 36864 data: 4 4 5 4 4 4 4 3 5
19399 063/300 Read block 1032 len 24576
19400 064/300 Read block 2661 len 40960
19401 065/300 Read block 1499 len 32768
19402 066/300 Read block 950 len 16384
19403 067/300 Read block 635 len 28672
19404 068/300 Read block 890 len 12288
19405 069/300 Flush
19406 070/300 Write block 3962 len 40960 data: 5 5 6 6 6 5 4 4 4 5
19407 071/300 Read block 3179 len 28672
19408 072/300 Read block 206 len 16384
19409 073/300 Write block 583 len 16384 data: 4 5 4 5
19410 074/300 Read block 4283 len 36864
19411 075/300 Read block 3291 len 32768
19412 076/300 Read block 4715 len 24576
19413 077/300 Read block 1033 len 16384
19414 078/300 Read block 424 len 16384
19415 079/300 Write block 4397 len 8192 data: 3 3
19416 080/300 Read block 4763 len 24576
19417 081/300 Read block 609 len 12288
19418 082/300 Read block 3253 len 32768
19419 083/300 Write block 494 len 32768 data: 6 6 6 5 5 5 7 6
19420 084/300 Write block 2709 len 24576 data: 4 5 5 6 6 5
19421 085/300 Write block 4938 len 12288 data: 3 3 5
19422 086/300 Flush
19423 087/300 Write block 2444 len 16384 data: 8 7 5 5
19424 088/300 Write block 2751 len 24576 data: 5 7 6 7 6 7
19425 089/300 Write block 72 len 28672 data: 3 4 4 5 5 5 5
19426 090/300 Write block 2764 len 4096 data: 6
19427 091/300 Read block 2268 len 24576
19428 092/300 Write block 4145 len 20480 data: 5 4 2 2 2
19429 093/300 Write block 429 len 32768 data: 2 2 2 2 4 5 6 5
19430 094/300 Write block 1694 len 16384 data: 4 4 2 3
19431 095/300 Write block 657 len 20480 data: 5 6 6 6 5
19432 096/300 Write block 3779 len 20480 data: 6 6 5 4 4
19433 097/300 Read block 4229 len 4096
19434 098/300 Write block 2288 len 8192 data: 3 3
19435 099/300 Read block 4336 len 4096
19436 100/300 Write block 4257 len 20480 data: 5 5 5 7 5
19437 101/300 Read block 4577 len 36864
19438 102/300 Write block 409 len 40960 data: 8 7 8 6 6 5 4 4 4 4
19439 103/300 Write block 2106 len 16384 data: 6 6 5 4
19440 104/300 Write block 4703 len 8192 data: 4 4
19441 105/300 Write block 1084 len 24576 data: 4 5 7 7 7 6
19442 106/300 Read block 4166 len 8192
19443 107/300 Write block 1545 len 12288 data: 6 5 5
19444 108/300 Write block 1837 len 24576 data: 3 4 5 5 5 6
19445 109/300 Write block 1072 len 16384 data: 6 6 6 5
19446 110/300 Flush
19447 111/300 Write block 295 len 36864 data: 3 3 4 4 5 6 5 4 4
19448 112/300 Read block 3596 len 32768
19449 113/300 Read block 4972 len 36864
19450 114/300 Read block 2151 len 12288
19451 115/300 Write block 852 len 24576 data: 4 5 4 3 3 4
19452 116/300 Flush
19453 117/300 Read block 3004 len 32768
19454 118/300 Read block 4691 len 40960
19455 119/300 Write block 2533 len 24576 data: 2 2 3 3 3 4
19456 120/300 Write block 2851 len 32768 data: 8 6 7 8 7 8 7 7
19457 121/300 Write block 4961 len 16384 data: 7 6 6 3
19458 122/300 Read block 4793 len 28672
19459 123/300 Flush
19460 124/300 Read block 1400 len 36864
19461 125/300 Read block 2620 len 24576
19462 126/300 Write block 3694 len 24576 data: 4 4 4 6 7 7
19463 127/300 Read block 2358 len 16384
19464 128/300 Read block 1910 len 36864
19465 129/300 Read block 1317 len 40960
19466 130/300 Write block 4085 len 4096 data: 3
19467 131/300 Read block 1945 len 16384
19468 132/300 Write block 2653 len 28672 data: 4 3 3 4 4 4 4
19469 133/300 Write block 3583 len 20480 data: 4 5 4 4 3
19470 134/300 Write block 4199 len 12288 data: 3 5 7
19471 135/300 Write block 2921 len 28672 data: 4 4 7 9 7 6 7
19472 136/300 Write block 3649 len 28672 data: 4 5 5 4 4 6 6
19473 137/300 Write block 1275 len 16384 data: 6 7 7 5
19474 138/300 Write block 2079 len 20480 data: 6 5 3 3 3
19475 139/300 Write block 4490 len 4096 data: 6
19476 140/300 Read block 4806 len 28672
19477 141/300 Write block 4533 len 20480 data: 5 5 4 5 6
19478 142/300 Write block 2747 len 12288 data: 6 5 5
19479 143/300 Flush
19480 144/300 Write block 4928 len 36864 data: 5 5 4 3 3 4 4 5 4
19481 145/300 Read block 204 len 24576
19482 146/300 Read block 2981 len 8192
19483 147/300 Read block 1790 len 36864
19484 148/300 Read block 3730 len 12288
19485 149/300 Read block 2564 len 20480
19486 150/300 Read block 4980 len 40960
19487 151/300 Read block 4575 len 20480
19488 152/300 Read block 1500 len 32768
19489 153/300 Flush
19490 154/300 Write block 4556 len 32768 data: 3 3 4 4 3 2 2 2
19491 155/300 Read block 3446 len 28672
19492 156/300 Write block 4496 len 24576 data: 3 5 6 5 6 7
19493 157/300 Write block 1581 len 20480 data: 2 3 5 5 6
19494 158/300 Read block 3525 len 24576
19495 159/300 Read block 194 len 32768
19496 160/300 Read block 1529 len 12288
19497 161/300 Write block 2055 len 24576 data: 3 3 3 3 4 6
19498 162/300 Read block 3498 len 36864
19499 163/300 Write block 1235 len 20480 data: 3 3 4 4 4
19500 164/300 Read block 4344 len 8192
19501 165/300 Write block 4719 len 24576 data: 6 6 7 6 6 4
19502 166/300 Read block 3620 len 12288
19503 167/300 Write block 485 len 20480 data: 4 6 5 4 4
19504 168/300 Read block 558 len 40960
19505 169/300 Write block 3009 len 16384 data: 5 5 5 5
19506 170/300 Read block 3741 len 20480
19507 171/300 Write block 4819 len 8192 data: 5 5
19508 172/300 Read block 4255 len 16384
19509 173/300 Read block 1376 len 36864
19510 174/300 Read block 1848 len 36864
19511 175/300 Write block 4851 len 32768 data: 2 3 3 2 2 2 3 3
19512 176/300 Read block 3886 len 20480
19513 177/300 Read block 760 len 32768
19514 178/300 Read block 2692 len 16384
19515 179/300 Read block 4011 len 32768
19516 180/300 Write block 2500 len 40960 data: 2 2 3 2 2 3 3 3 4 5
19517 181/300 Write block 1938 len 40960 data: 2 2 2 2 2 3 4 4 3 3
19518 182/300 Flush
19519 183/300 Write block 729 len 8192 data: 3 4
19520 184/300 Read block 3053 len 24576
19521 185/300 Write block 3127 len 20480 data: 6 6 6 6 5
19522 186/300 Read block 279 len 36864
19523 187/300 Read block 1308 len 12288
19524 188/300 Write block 264 len 28672 data: 5 5 4 5 5 5 6
19525 189/300 Read block 3638 len 8192
19526 190/300 Read block 3311 len 40960
19527 191/300 Write block 2320 len 28672 data: 4 4 3 4 4 3 2
19528 192/300 Write block 269 len 24576 data: 6 7 6 5 3 3
19529 193/300 Read block 1182 len 32768
19530 194/300 Read block 732 len 8192
19531 195/300 Write block 2549 len 32768 data: 4 4 4 4 4 4 5 4
19532 196/300 Read block 2573 len 12288
19533 197/300 Read block 2796 len 20480
19534 198/300 Read block 839 len 4096
19535 199/300 Read block 1720 len 40960
19536 200/300 Read block 260 len 28672
19537 201/300 Read block 953 len 24576
19538 202/300 Flush
19539 203/300 Read block 889 len 36864
19540 204/300 Read block 4030 len 16384
19541 205/300 Read block 2990 len 36864
19542 206/300 Write block 436 len 20480 data: 6 4 4 4 7
19543 207/300 Write block 4070 len 20480 data: 7 7 8 9 7
19544 208/300 Write block 2738 len 4096 data: 5
19545 209/300 Read block 2895 len 40960
19546 210/300 Read block 1000 len 36864
19547 211/300 Write block 796 len 20480 data: 8 9 9 9 8
19548 212/300 Write block 2588 len 40960 data: 11 8 8 8 7 6 7 6 7 8
19549 213/300 Write block 532 len 28672 data: 6 6 7 6 7 7 7
19550 214/300 Flush
19551 215/300 Read block 228 len 8192
19552 216/300 Read block 2578 len 28672
19553 217/300 Flush
19554 218/300 Read block 621 len 16384
19555 219/300 Read block 2342 len 12288
19556 220/300 Read block 2503 len 32768
19557 221/300 Read block 4688 len 16384
19558 222/300 Read block 2053 len 28672
19559 223/300 Write block 3760 len 28672 data: 6 6 6 5 4 5 5
19560 224/300 Read block 3301 len 12288
19561 225/300 Read block 4145 len 36864
19562 226/300 Flush
19563 227/300 Write block 915 len 40960 data: 3 5 5 5 5 5 4 4 5 5
19564 228/300 Write block 1253 len 16384 data: 3 4 4 7
19565 229/300 Read block 2906 len 4096
19566 230/300 Flush
19567 231/300 Flush
19568 232/300 Write block 3657 len 28672 data: 3 4 3 3 3 5 5
19569 233/300 Read block 3576 len 16384
19570 234/300 Read block 3086 len 12288
19571 235/300 Read block 3490 len 12288
19572 236/300 Write block 3638 len 16384 data: 5 5 6 5
19573 237/300 Read block 4410 len 36864
19574 238/300 Read block 1494 len 32768
19575 239/300 Read block 1644 len 8192
19576 240/300 Read block 3878 len 24576
19577 241/300 Read block 925 len 16384
19578 242/300 Read block 4867 len 4096
19579 243/300 Write block 668 len 20480 data: 7 6 6 7 8
19580 244/300 Write block 4842 len 20480 data: 5 6 4 4 4
19581 245/300 Write block 2690 len 24576 data: 5 5 4 4 4 4
19582 246/300 Write block 4368 len 12288 data: 4 3 4
19583 247/300 Write block 4794 len 40960 data: 3 3 3 3 5 4 5 5 4 3
19584 248/300 Write block 3042 len 24576 data: 8 5 5 4 4 4
19585 249/300 Read block 4956 len 28672
19586 250/300 Write block 1383 len 4096 data: 5
19587 251/300 Write block 1430 len 8192 data: 3 5
19588 252/300 Write block 1358 len 16384 data: 4 4 5 5
19589 253/300 Read block 3885 len 36864
19590 254/300 Read block 3692 len 32768
19591 255/300 Write block 1093 len 8192 data: 5 6
19592 256/300 Read block 3002 len 12288
19593 257/300 Read block 17 len 20480
19594 258/300 Read block 2915 len 40960
19595 259/300 Read block 2957 len 40960
19596 260/300 Read block 3086 len 4096
19597 261/300 Write block 2161 len 20480 data: 4 3 3 3 4
19598 262/300 Write block 376 len 32768 data: 5 6 6 6 6 5 5 4
19599 263/300 Read block 4661 len 24576
19600 264/300 Write block 1763 len 28672 data: 3 3 3 4 4 4 4
19601 265/300 Write block 322 len 4096 data: 4
19602 266/300 Write block 552 len 28672 data: 4 3 3 4 4 5 6
19603 267/300 Write block 3680 len 16384 data: 5 7 7 7
19604 268/300 Flush
19605 269/300 Write block 125 len 32768 data: 6 6 6 5 5 4 2 2
19606 270/300 Write block 4833 len 8192 data: 4 4
19607 271/300 Read block 2038 len 12288
19608 272/300 Read block 3801 len 32768
19609 273/300 Flush
19610 274/300 Write block 3237 len 40960 data: 7 8 7 7 5 5 5 5 6 6
19611 275/300 Read block 4383 len 4096
19612 276/300 Write block 1169 len 12288 data: 5 5 6
19613 277/300 Read block 2229 len 32768
19614 278/300 Flush
19615 279/300 Read block 3826 len 36864
19616 280/300 Write block 700 len 40960 data: 6 5 5 5 6 6 6 4 4 4
19617 281/300 Read block 4680 len 40960
19618 282/300 Read block 999 len 8192
19619 283/300 Read block 89 len 4096
19620 284/300 Write block 1918 len 36864 data: 7 7 7 5 5 6 3 4 4
19621 285/300 Write block 4824 len 4096 data: 6
19622 286/300 Read block 2791 len 4096
19623 287/300 Read block 1257 len 4096
19624 288/300 Write block 694 len 40960 data: 4 4 5 5 5 5 7 6 6 6
19625 289/300 Flush
19626 290/300 Read block 3818 len 12288
19627 291/300 Read block 2900 len 20480
19628 292/300 Write block 1175 len 28672 data: 5 4 3 3 4 4 5
19629 293/300 Write block 4258 len 12288 data: 6 6 8
19630 294/300 Write block 3768 len 8192 data: 4 4
19631 295/300 Write block 3480 len 12288 data: 6 6 6
19632 296/300 Write block 3380 len 24576 data: 6 5 4 5 5 5
19633 297/300 Write block 4242 len 32768 data: 4 4 4 5 6 6 6 7
19634 298/300 Read block 2084 len 4096
19635 299/300 Flush
19636 300/300 Write block 4515 len 24576 data: 3 3 3 3 3 3
19637 Main task has work to do, go find it
19638 got dsc Start(2)
19639 start 2
19640 Feb 12 01:38:25.602 INFO request completed, latency_us: 279, response_code: 204, uri: /start/cid/2, method: POST, req_id: c96d8e5f-f7e6-429b-bbb4-5dd4e736424a, remote_addr: 127.0.0.1:60456, local_addr: 127.0.0.1:9998
19641 Replay: started 2, returned:Ok(())
19642 ----------------------------------------------------------------
19643 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:300
19644 States: Active Active Offline
19645 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
19646 6332 Acked 7331 Write 9 Done Done New true
19647 6333 Acked 7332 Write 7 Done Done New true
19648 6334 Acked 7333 Read 8 Done Done New true
19649 6335 Acked 7334 Flush 0 Done Done New false
19650 6336 Acked 7335 Flush 0 Done Done New false
19651 6337 Acked 7336 Write 3 Done Done New false
19652 6338 Acked 7337 Read 6 Done Done New false
19653 6339 Acked 7338 Read 10 Done Done New false
19654 6340 Acked 7339 Read 8 Done Done New false
19655 6341 Acked 7340 Read 10 Done Done New false
19656 6342 Acked 7341 Read 1 Done Done New false
19657 6343 Acked 7342 Read 1 Done Done New false
19658 6344 Acked 7343 Flush 0 Done Done New false
19659 6345 Acked 7344 Write 7 Done Done New false
19660 6346 Acked 7345 Read 9 Done Done New false
19661 6347 Acked 7346 Read 10 Done Done New false
19662 6348 Acked 7347 Write 2 Done Done New false
19663 6349 Acked 7348 Write 1 Done Done New false
19664 6350 Acked 7349 Write 1 Done Done New false
19665 6351 Acked 7350 Write 5 Done Done New false
19666 6352 Acked 7351 Read 4 Done Done New false
19667 6353 Acked 7352 Read 10 Done Done New false
19668 6354 Acked 7353 Write 9 Done Done New false
19669 6355 Acked 7354 Read 10 Done Done New false
19670 6356 Acked 7355 Write 7 Done Done New false
19671 6357 Acked 7356 Flush 0 Done Done New false
19672 6358 Acked 7357 Read 1 Done Done New false
19673 6359 Acked 7358 Read 2 Done Done New false
19674 6360 Acked 7359 Write 8 Done Done New false
19675 6361 Acked 7360 Write 4 Done Done New false
19676 6362 Acked 7361 Write 7 Done Done New false
19677 6363 Acked 7362 Write 1 Done Done New false
19678 6364 Acked 7363 Write 9 Done Done New false
19679 6365 Acked 7364 Read 10 Done Done New false
19680 6366 Acked 7365 Write 2 Done Done New false
19681 6367 Acked 7366 Flush 0 Done Done New false
19682 6368 Acked 7367 Write 2 Done Done New false
19683 6369 Acked 7368 Read 8 Done Done New false
19684 6370 Acked 7369 Write 6 Done Done New false
19685 6371 Acked 7370 Read 4 Done Done New false
19686 6372 Acked 7371 Read 5 Done Done New false
19687 6373 Acked 7372 Read 1 Done Done New false
19688 6374 Acked 7373 Write 9 Done Done New false
19689 6375 Acked 7374 Read 2 Done Done New false
19690 6376 Acked 7375 Write 8 Done Done New false
19691 6377 Acked 7376 Write 4 Done Done New false
19692 6378 Acked 7377 Read 7 Done Done New false
19693 6379 Acked 7378 Write 6 Done Done New false
19694 6380 Acked 7379 Write 8 Done Done New false
19695 6381 Acked 7380 Read 6 Done Done New false
19696 6382 Acked 7381 Write 10 Done Done New false
19697 6383 Acked 7382 Read 7 Done Done New false
19698 6384 Acked 7383 Flush 0 Done Done New false
19699 6385 Acked 7384 Write 10 Done Done New false
19700 6386 Acked 7385 Read 2 Done Done New false
19701 6387 Acked 7386 Flush 0 Done Done New false
19702 6388 Acked 7387 Read 8 Done Done New false
19703 6389 Acked 7388 Write 10 Done Done New false
19704 6390 Acked 7389 Flush 0 Done Done New false
19705 6391 Acked 7390 Read 3 Done Done New false
19706 6392 Acked 7391 Read 4 Done Done New false
19707 6393 Acked 7392 Write 9 Done Done New false
19708 6394 Acked 7393 Read 6 Done Done New false
19709 6395 Acked 7394 Read 10 Done Done New false
19710 6396 Acked 7395 Read 8 Done Done New false
19711 6397 Acked 7396 Read 4 Done Done New false
19712 6398 Acked 7397 Read 7 Done Done New false
19713 6399 Acked 7398 Read 3 Done Done New false
19714 6400 Acked 7399 Flush 0 Done Done New false
19715 6401 Acked 7400 Write 10 Done Done New false
19716 6402 Acked 7401 Read 7 Done Done New false
19717 6403 Acked 7402 Read 4 Done Done New false
19718 6404 Acked 7403 Write 4 Done Done New false
19719 6405 Acked 7404 Read 9 Done Done New false
19720 6406 Acked 7405 Read 8 Done Done New false
19721 6407 Acked 7406 Read 6 Done Done New false
19722 6408 Acked 7407 Read 4 Done Done New false
19723 6409 Acked 7408 Read 4 Done Done New false
19724 6410 Acked 7409 Write 2 Done Done New false
19725 6411 Acked 7410 Read 6 Done Done New false
19726 6412 Acked 7411 Read 3 Done Done New false
19727 6413 Acked 7412 Read 8 Done Done New false
19728 6414 Acked 7413 Write 8 Done Done New false
19729 6415 Acked 7414 Write 6 Done Done New false
19730 6416 Acked 7415 Write 3 Done Done New false
19731 6417 Acked 7416 Flush 0 Done Done New false
19732 6418 Acked 7417 Write 4 Done Done New false
19733 6419 Acked 7418 Write 6 Done Done New false
19734 6420 Acked 7419 Write 7 Done Done New false
19735 6421 Acked 7420 Write 1 Done Done New false
19736 6422 Acked 7421 Read 6 Done Done New false
19737 6423 Acked 7422 Write 5 Done Done New false
19738 6424 Acked 7423 Write 8 Done Done New false
19739 6425 Acked 7424 Write 4 Done Done New false
19740 6426 Acked 7425 Write 5 Done Done New false
19741 6427 Acked 7426 Write 5 Done Done New false
19742 6428 Acked 7427 Read 1 Done Done New false
19743 6429 Acked 7428 Write 2 Done Done New false
19744 6430 Acked 7429 Read 1 Done Done New false
19745 6431 Acked 7430 Write 5 Done Done New false
19746 6432 Acked 7431 Read 9 Done Done New false
19747 6433 Acked 7432 Write 10 Done Done New false
19748 6434 Acked 7433 Write 4 Done Done New false
19749 6435 Acked 7434 Write 2 Done Done New false
19750 6436 Acked 7435 Write 6 Done Done New false
19751 6437 Acked 7436 Read 2 Done Done New false
19752 6438 Acked 7437 Write 3 Done Done New false
19753 6439 Acked 7438 Write 6 Done Done New false
19754 6440 Acked 7439 Write 4 Done Done New false
19755 6441 Acked 7440 Flush 0 Done Done New false
19756 6442 Acked 7441 Write 9 Done Done New false
19757 6443 Acked 7442 Read 8 Done Done New false
19758 6444 Acked 7443 Read 9 Done Done New false
19759 6445 Acked 7444 Read 3 Done Done New false
19760 6446 Acked 7445 Write 6 Done Done New false
19761 6447 Acked 7446 Flush 0 Done Done New false
19762 6448 Acked 7447 Read 8 Done Done New false
19763 6449 Acked 7448 Read 10 Done Done New false
19764 6450 Acked 7449 Write 6 Done Done New false
19765 6451 Acked 7450 Write 8 Done Done New false
19766 6452 Acked 7451 Write 4 Done Done New false
19767 6453 Acked 7452 Read 7 Done Done New false
19768 6454 Acked 7453 Flush 0 Done Done New false
19769 6455 Acked 7454 Read 9 Done Done New false
19770 6456 Acked 7455 Read 6 Done Done New false
19771 6457 Acked 7456 Write 6 Done Done New false
19772 6458 Acked 7457 Read 4 Done Done New false
19773 6459 Acked 7458 Read 9 Done Done New false
19774 6460 Acked 7459 Read 10 Done Done New false
19775 6461 Acked 7460 Write 1 Done Done New false
19776 6462 Acked 7461 Read 4 Done Done New false
19777 6463 Acked 7462 Write 7 Done Done New false
19778 6464 Acked 7463 Write 5 Done Done New false
19779 6465 Acked 7464 Write 3 Done Done New false
19780 6466 Acked 7465 Write 7 Done Done New false
19781 6467 Acked 7466 Write 7 Done Done New false
19782 6468 Acked 7467 Write 4 Done Done New false
19783 6469 Acked 7468 Write 5 Done Done New false
19784 6470 Acked 7469 Write 1 Done Done New false
19785 6471 Acked 7470 Read 7 Done Done New false
19786 6472 Acked 7471 Write 5 Done Done New false
19787 6473 Acked 7472 Write 3 Done Done New false
19788 6474 Acked 7473 Flush 0 Done Done New false
19789 6475 Acked 7474 Write 9 Done Done New false
19790 6476 Acked 7475 Read 6 Done Done New false
19791 6477 Acked 7476 Read 2 Done Done New false
19792 6478 Acked 7477 Read 9 Done Done New false
19793 6479 Acked 7478 Read 3 Done Done New false
19794 6480 Acked 7479 Read 5 Done Done New false
19795 6481 Acked 7480 Read 10 Done Done New false
19796 6482 Acked 7481 Read 5 Done Done New false
19797 6483 Acked 7482 Read 8 Done Done New false
19798 6484 Acked 7483 Flush 0 Done Done New false
19799 6485 Acked 7484 Write 8 Done Done New false
19800 6486 Acked 7485 Read 7 Done Done New false
19801 6487 Acked 7486 Write 6 Done Done New false
19802 6488 Acked 7487 Write 5 Done Done New false
19803 6489 Acked 7488 Read 6 Done Done New false
19804 6490 Acked 7489 Read 8 Done Done New false
19805 6491 Acked 7490 Read 3 Done Done New false
19806 6492 Acked 7491 Write 6 Done Done New false
19807 6493 Acked 7492 Read 9 Done Done New false
19808 6494 Acked 7493 Write 5 Done Done New false
19809 6495 Acked 7494 Read 2 Done Done New false
19810 6496 Acked 7495 Write 6 Done Done New false
19811 6497 Acked 7496 Read 3 Done Done New false
19812 6498 Acked 7497 Write 5 Done Done New false
19813 6499 Acked 7498 Read 10 Done Done New false
19814 6500 Acked 7499 Write 4 Done Done New false
19815 6501 Acked 7500 Read 5 Done Done New false
19816 6502 Acked 7501 Write 2 Done Done New false
19817 6503 Acked 7502 Read 4 Done Done New false
19818 6504 Acked 7503 Read 9 Done Done New false
19819 6505 Acked 7504 Read 9 Done Done New false
19820 6506 Acked 7505 Write 8 Done Done New false
19821 6507 Acked 7506 Read 5 Done Done New false
19822 6508 Acked 7507 Read 8 Done Done New false
19823 6509 Acked 7508 Read 4 Done Done New false
19824 6510 Acked 7509 Read 8 Done Done New false
19825 6511 Acked 7510 Write 10 Done Done New false
19826 6512 Acked 7511 Write 10 Done Done New false
19827 6513 Acked 7512 Flush 0 Done Done New false
19828 6514 Acked 7513 Write 2 Done Done New false
19829 6515 Acked 7514 Read 6 Done Done New false
19830 6516 Acked 7515 Write 5 Done Done New false
19831 6517 Acked 7516 Read 9 Done Done New false
19832 6518 Acked 7517 Read 3 Done Done New false
19833 6519 Acked 7518 Write 7 Done Done New false
19834 6520 Acked 7519 Read 2 Done Done New false
19835 6521 Acked 7520 Read 10 Done Done New false
19836 6522 Acked 7521 Write 7 Done Done New false
19837 6523 Acked 7522 Write 6 Done Done New false
19838 6524 Acked 7523 Read 8 Done Done New false
19839 6525 Acked 7524 Read 2 Done Done New false
19840 6526 Acked 7525 Write 8 Done Done New false
19841 6527 Acked 7526 Read 3 Done Done New false
19842 6528 Acked 7527 Read 5 Done Done New false
19843 6529 Acked 7528 Read 1 Done Done New false
19844 6530 Acked 7529 Read 10 Done Done New false
19845 6531 Acked 7530 Read 7 Done Done New false
19846 6532 Acked 7531 Read 6 Done Done New false
19847 6533 Acked 7532 Flush 0 Done Done New false
19848 6534 Acked 7533 Read 9 Done Done New false
19849 6535 Acked 7534 Read 4 Done Done New false
19850 6536 Acked 7535 Read 9 Done Done New false
19851 6537 Acked 7536 Write 5 Done Done New false
19852 6538 Acked 7537 Write 5 Done Done New false
19853 6539 Acked 7538 Write 1 Done Done New false
19854 6540 Acked 7539 Read 10 Done Done New false
19855 6541 Acked 7540 Read 9 Done Done New false
19856 6542 Acked 7541 Write 5 Done Done New false
19857 6543 Acked 7542 Write 10 Done Done New false
19858 6544 Acked 7543 Write 7 Done Done New false
19859 6545 Acked 7544 Flush 0 Done Done New false
19860 6546 Acked 7545 Read 2 Done Done New false
19861 6547 Acked 7546 Read 7 Done Done New false
19862 6548 Acked 7547 Flush 0 Done Done New false
19863 6549 Acked 7548 Read 4 Done Done New false
19864 6550 Acked 7549 Read 3 Done Done New false
19865 6551 Acked 7550 Read 8 Done Done New false
19866 6552 Acked 7551 Read 4 Done Done New false
19867 6553 Acked 7552 Read 7 Done Done New false
19868 6554 Acked 7553 Write 7 Done Done New false
19869 6555 Acked 7554 Read 3 Done Done New false
19870 6556 Acked 7555 Read 9 Done Done New false
19871 6557 Acked 7556 Flush 0 Done Done New false
19872 6558 Acked 7557 Write 10 Done Done New false
19873 6559 Acked 7558 Write 4 Done Done New false
19874 6560 Acked 7559 Read 1 Done Done New false
19875 6561 Acked 7560 Flush 0 Done Done New false
19876 6562 Acked 7561 Flush 0 Done Done New false
19877 6563 Acked 7562 Write 7 Done Done New false
19878 6564 Acked 7563 Read 4 Done Done New false
19879 6565 Acked 7564 Read 3 Done Done New false
19880 6566 Acked 7565 Read 3 Done Done New false
19881 6567 Acked 7566 Write 4 Done Done New false
19882 6568 Acked 7567 Read 9 Done Done New false
19883 6569 Acked 7568 Read 8 Done Done New false
19884 6570 Acked 7569 Read 2 Done Done New false
19885 6571 Acked 7570 Read 6 Done Done New false
19886 6572 Acked 7571 Read 4 Done Done New false
19887 6573 Acked 7572 Read 1 Done Done New false
19888 6574 Acked 7573 Write 5 Done Done New false
19889 6575 Acked 7574 Write 5 Done Done New false
19890 6576 Acked 7575 Write 6 Done Done New false
19891 6577 Acked 7576 Write 3 Done Done New false
19892 6578 Acked 7577 Write 10 Done Done New false
19893 6579 Acked 7578 Write 6 Done Done New false
19894 6580 Acked 7579 Read 7 Done Done New false
19895 6581 Acked 7580 Write 1 Done Done New false
19896 6582 Acked 7581 Write 2 Done Done New false
19897 6583 Acked 7582 Write 4 Done Done New false
19898 6584 Acked 7583 Read 9 Done Done New false
19899 6585 Acked 7584 Read 8 Done Done New false
19900 6586 Acked 7585 Write 2 Done Done New false
19901 6587 Acked 7586 Read 3 Done Done New false
19902 6588 Acked 7587 Read 5 Done Done New false
19903 6589 Acked 7588 Read 10 Done Done New false
19904 6590 Acked 7589 Read 10 Done Done New false
19905 6591 Acked 7590 Read 1 Done Done New false
19906 6592 Acked 7591 Write 5 Done Done New false
19907 6593 Acked 7592 Write 8 Done Done New false
19908 6594 Acked 7593 Read 6 Done Done New false
19909 6595 Acked 7594 Write 7 Done Done New false
19910 6596 Acked 7595 Write 1 Done Done New false
19911 6597 Acked 7596 Write 7 Done Done New false
19912 6598 Acked 7597 Write 4 Done Done New false
19913 6599 Acked 7598 Flush 0 Done Done New false
19914 6600 Acked 7599 Write 8 Done Done New false
19915 6601 Acked 7600 Write 2 Done Done New false
19916 6602 Acked 7601 Read 3 Done Done New false
19917 6603 Acked 7602 Read 8 Done Done New false
19918 6604 Acked 7603 Flush 0 Done Done New false
19919 6605 Acked 7604 Write 10 Done Done New false
19920 6606 Acked 7605 Read 1 Done Done New false
19921 6607 Acked 7606 Write 3 Done Done New false
19922 6608 Acked 7607 Read 8 Done Done New false
19923 6609 Acked 7608 Flush 0 Done Done New false
19924 6610 Acked 7609 Read 9 Done Done New false
19925 6611 Acked 7610 Write 10 Done Done New false
19926 6612 Acked 7611 Read 10 Done Done New false
19927 6613 Acked 7612 Read 2 Done Done New false
19928 6614 Acked 7613 Read 1 Done Done New false
19929 6615 Acked 7614 Write 9 Done Done New false
19930 6616 Acked 7615 Write 1 Done Done New false
19931 6617 Acked 7616 Read 1 Done Done New false
19932 6618 Acked 7617 Read 1 Done Done New false
19933 6619 Acked 7618 Write 10 Done Done New false
19934 6620 Acked 7619 Flush 0 Done Done New false
19935 6621 Acked 7620 Read 3 Done Done New false
19936 6622 Acked 7621 Read 5 Done Done New false
19937 6623 Acked 7622 Write 7 Done Done New false
19938 6624 Acked 7623 Write 3 Done Done New false
19939 6625 Acked 7624 Write 2 Done Done New false
19940 6626 Acked 7625 Write 3 Done Done New false
19941 6627 Acked 7626 Write 6 Done Done New false
19942 6628 Acked 7627 Write 8 Done Done New false
19943 6629 Acked 7628 Read 1 Done Done New false
19944 6630 Acked 7629 Flush 0 Done Done New false
19945 6631 Acked 7630 Write 6 Done Done New false
19946 STATES DS:0 DS:1 DS:2 TOTAL
19947 New 0 0 300 300
19948 Sent 0 0 0 0
19949 Done 300 300 0 600
19950 Skipped 0 0 0 0
19951 Error 0 0 0 0
19952 Last Flush: 7629 7629 7330
19953 Downstairs last five completed: 7330 7329 7328 7327 7326
19954 Upstairs last five completed: 6631 6630 6629 6628 6627
19955 CLIENT: Up:0 ds:300 act:2
199562024-02-12T01:38:26.246ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
199572024-02-12T01:38:26.246ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
199582024-02-12T01:38:26.246ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
199592024-02-12T01:38:26.246ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
199602024-02-12T01:38:26.246ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
199612024-02-12T01:38:26.246ZINFOcrucible: [2] client re-new 301 jobs since flush 7330 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
199622024-02-12T01:38:26.246ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
199632024-02-12T01:38:27.247ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
199642024-02-12T01:38:27.247ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
199652024-02-12T01:38:27.247ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
199662024-02-12T01:38:27.247ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
199672024-02-12T01:38:27.247ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
199682024-02-12T01:38:27.247ZINFOcrucible: [2] client re-new 301 jobs since flush 7330 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
199692024-02-12T01:38:27.247ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
19970 [8830] got start action
19971 [8830] I am going to restart
19972 Starting downstairs at port 8830
19973 Make output file at "/tmp/dsc/downstairs-8830.txt"
19974 [8830][2] reports Starting
19975 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1186)
19976 [8830][2] reports Running
199772024-02-12T01:38:28.248ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
199782024-02-12T01:38:28.249ZINFOcrucible: ds_connection connected = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
199792024-02-12T01:38:28.250ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
199802024-02-12T01:38:28.250ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
199812024-02-12T01:38:28.250ZINFOcrucible: send last flush ID to this DS: 7330 = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
199822024-02-12T01:38:28.250ZINFOcrucible: Replied this last flush ID: 7330 = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
199832024-02-12T01:38:28.250ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
199842024-02-12T01:38:28.250ZINFOcrucible: [2] Transition from Offline to Replay = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
199852024-02-12T01:38:28.250ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
199862024-02-12T01:38:28.250ZINFOcrucible: [2] Transition from Replay to Active = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
19987 ----------------------------------------------------------------
19988 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
19989 Downstairs last five completed: 7631 7630 7629 7628 7627
19990 Upstairs last five completed: 6632 6631 6630 6629 6628
19991 CLIENT: Up:0 ds:0 act:3
19992 Replay: All jobs finished, all DS active.
19993 Main task has work to do, go find it
19994 got dsc Stop(0)
19995 stop 0
19996 Feb 12 01:38:29.606 INFO request completed, latency_us: 402[8810] Got stop action so:false kr:false
19997 , response_code: 204, uri: /stop/cid/0, method: POST, req_id: d03ae5cd-7880-4e8f-928d-47e1475e6948, remote_addr: 127.0.0.1:60456, local_addr: 127.0.0.1:9998
199982024-02-12T01:38:29.606ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
199992024-02-12T01:38:29.607ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
20000 Feb 12 01:38:29.607 INFO request completed, latency_us: 145, response_code: 200, uri: /state/cid/0, method: GET, {"msg":"downstairs task for 0 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-02-12T01:38:29.607166891Z","hostname":"ip-10-150-1-104.us-west-2.compute.internal","pid":1164,"session_id":"e1773b3e-b36f-4080-8b13-c0830f0d072e"}
20001 req_id: 22258521-8b54-4af4-8bba-b3e382d42395, remote_addr: 127.0.0.{"msg":"Gone missing, transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-02-12T01:38:29.607218174Z","hostname":"ip-10-150-1-104.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"e1773b3e-b36f-4080-8b13-c0830f0d072e"}
20002 1:60456, local_addr: {"msg":"[0] client re-new 0 jobs since flush 7631","v":0,"name":"crucible","level":30,"time":"2024-02-12T01:38:29.607280045Z","hostname":"ip-10-150-1-104.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"e1773b3e-b36f-4080-8b13-c0830f0d072e"}
20003 127.0.0.{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-12T01:38:29.607323088Z","hostname":"ip-10-150-1-104.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"e1773b3e-b36f-4080-8b13-c0830f0d072e"}
20004 1:9998
20005 [8810] Exited with: ExitStatus(unix_wait_status(9))
20006 [8810][0] reports Exit
200072024-02-12T01:38:30.609ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
200082024-02-12T01:38:30.609ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
200092024-02-12T01:38:30.609ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
200102024-02-12T01:38:30.609ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
200112024-02-12T01:38:30.609ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
200122024-02-12T01:38:30.609ZINFOcrucible: [0] client re-new 0 jobs since flush 7631 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
200132024-02-12T01:38:30.609ZINFOcrucible: client task is exiting = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
200142024-02-12T01:38:31.611ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
200152024-02-12T01:38:31.611ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
200162024-02-12T01:38:31.611ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
200172024-02-12T01:38:31.611ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
200182024-02-12T01:38:31.611ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
200192024-02-12T01:38:31.611ZINFOcrucible: [0] client re-new 0 jobs since flush 7631 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
200202024-02-12T01:38:31.611ZINFOcrucible: client task is exiting = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
200212024-02-12T01:38:32.612ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
200222024-02-12T01:38:32.612ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
200232024-02-12T01:38:32.612ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
200242024-02-12T01:38:32.612ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
200252024-02-12T01:38:32.612ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
200262024-02-12T01:38:32.612ZINFOcrucible: [0] client re-new 0 jobs since flush 7631 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
200272024-02-12T01:38:32.612ZINFOcrucible: client task is exiting = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
20028 Feb 12 01:38:33.609 INFO request completed, latency_us: 215, response_code: 200, uri: /state/cid/0, method: GET, req_id: 9a4ac5cb-60ea-4af1-93a2-83f657868cd9, remote_addr: 127.0.0.1:60456, local_addr: 127.0.0.1:9998
20029 001/300 Read block 528 len 40960
20030 002/300 Flush
200312024-02-12T01:38:33.613ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
200322024-02-12T01:38:33.613ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
200332024-02-12T01:38:33.613ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
200342024-02-12T01:38:33.613ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
200352024-02-12T01:38:33.613ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
200362024-02-12T01:38:33.613ZINFOcrucible: [0] client re-new 2 jobs since flush 7631 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
200372024-02-12T01:38:33.613ZINFOcrucible: client task is exiting = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
20038 003/300 Write block 3436 len 28672 data: 6 4 3 4 5 5 5
20039 004/300 Write block 521 len 32768 data: 4 4 3 4 5 3 4 4
20040 005/300 Read block 4866 len 4096
20041 006/300 Flush
20042 007/300 Read block 3980 len 40960
20043 008/300 Read block 3483 len 28672
20044 009/300 Write block 541 len 36864 data: 4 3 4 4 4 4 4 4 5
20045 010/300 Flush
20046 011/300 Read block 803 len 12288
20047 012/300 Read block 2685 len 32768
20048 013/300 Write block 2973 len 28672 data: 6 7 6 5 5 6 5
20049 014/300 Read block 29 len 28672
20050 015/300 Write block 3858 len 36864 data: 3 3 4 5 4 4 4 5 5
20051 016/300 Read block 2054 len 20480
20052 017/300 Read block 2700 len 40960
20053 018/300 Read block 3775 len 32768
20054 019/300 Write block 483 len 32768 data: 3 4 5 7 6 5 5 5
20055 020/300 Write block 4755 len 24576 data: 4 4 4 4 3 2
20056 021/300 Read block 3390 len 4096
20057 022/300 Write block 3862 len 32768 data: 5 5 5 6 6 5 5 4
20058 023/300 Read block 3935 len 36864
20059 024/300 Read block 2420 len 16384
20060 025/300 Write block 722 len 12288 data: 4 5 4
20061 026/300 Write block 1657 len 4096 data: 2
20062 027/300 Read block 202 len 40960
20063 028/300 Read block 1057 len 12288
20064 029/300 Read block 1139 len 8192
20065 030/300 Write block 4615 len 28672 data: 5 4 3 2 2 3 5
20066 031/300 Read block 3266 len 16384
20067 032/300 Write block 1628 len 32768 data: 3 2 2 4 4 4 3 5
20068 033/300 Read block 4831 len 36864
20069 034/300 Read block 3370 len 12288
20070 035/300 Write block 2531 len 4096 data: 4
20071 036/300 Flush
20072 037/300 Flush
20073 038/300 Read block 1662 len 20480
20074 039/300 Write block 3235 len 12288 data: 7 7 8
20075 040/300 Read block 1448 len 20480
20076 041/300 Read block 2594 len 24576
20077 042/300 Read block 4001 len 20480
20078 043/300 Read block 3314 len 4096
20079 044/300 Read block 3666 len 32768
20080 045/300 Write block 3338 len 36864 data: 4 5 4 4 5 6 5 4 3
20081 046/300 Write block 1198 len 4096 data: 5
20082 047/300 Write block 2643 len 8192 data: 5 6
20083 048/300 Write block 156 len 8192 data: 3 3
20084 049/300 Write block 3971 len 16384 data: 6 5 6 5
20085 050/300 Write block 678 len 24576 data: 6 5 5 5 3 4
20086 051/300 Read block 3598 len 12288
20087 052/300 Read block 3205 len 12288
20088 053/300 Write block 968 len 32768 data: 6 7 7 7 6 6 6 6
20089 054/300 Read block 1860 len 8192
20090 055/300 Write block 2524 len 16384 data: 3 6 6 6
20091 056/300 Write block 1942 len 16384 data: 3 4 5 5
20092 057/300 Read block 2453 len 20480
20093 058/300 Read block 1343 len 8192
20094 059/300 Read block 933 len 32768
20095 060/300 Read block 2162 len 8192
20096 061/300 Read block 3080 len 20480
20097 062/300 Write block 907 len 32768 data: 4 5 6 5 6 5 5 4
20098 063/300 Write block 916 len 4096 data: 6
20099 064/300 Flush
20100 065/300 Read block 1230 len 36864
20101 066/300 Read block 26 len 32768
20102 067/300 Write block 710 len 8192 data: 4 6
20103 068/300 Write block 4393 len 20480 data: 4 6 4 3 4
20104 069/300 Write block 4299 len 28672 data: 4 4 5 4 4 4 4
20105 070/300 Flush
20106 071/300 Read block 2533 len 40960
20107 072/300 Write block 4818 len 16384 data: 5 6 6 4
20108 073/300 Read block 4310 len 4096
20109 074/300 Flush
20110 075/300 Read block 4767 len 32768
20111 076/300 Write block 4780 len 12288 data: 7 7 5
20112 077/300 Write block 452 len 16384 data: 5 3 3 2
20113 078/300 Read block 1078 len 16384
20114 079/300 Read block 1684 len 16384
20115 080/300 Read block 3090 len 4096
20116 081/300 Write block 3718 len 24576 data: 6 5 4 2 2 2
20117 082/300 Read block 1618 len 32768
20118 083/300 Read block 1358 len 20480
20119 084/300 Read block 1606 len 36864
20120 085/300 Read block 4045 len 28672
20121 086/300 Flush
20122 087/300 Read block 4452 len 20480
20123 088/300 Write block 3055 len 24576 data: 5 6 5 5 5 5
20124 089/300 Write block 1099 len 36864 data: 4 4 4 5 6 7 7 5 6
20125 090/300 Read block 4015 len 32768
20126 091/300 Read block 4860 len 20480
20127 092/300 Write block 2642 len 20480 data: 5 6 7 8 8
20128 093/300 Write block 290 len 24576 data: 3 3 3 3 3 4
20129 094/300 Read block 2203 len 24576
20130 095/300 Read block 1036 len 20480
20131 096/300 Flush
20132 097/300 Read block 4247 len 8192
20133 098/300 Read block 2256 len 8192
20134 099/300 Write block 1421 len 36864 data: 6 8 8 7 7 5 5 4 4
20135 100/300 Write block 3554 len 28672 data: 2 3 4 3 3 3 4
20136 101/300 Read block 2873 len 12288
20137 102/300 Write block 519 len 32768 data: 4 4 5 5 4 5 6 4
20138 103/300 Read block 4585 len 32768
20139 104/300 Write block 2448 len 8192 data: 7 6
20140 105/300 Read block 2018 len 12288
20141 106/300 Write block 1220 len 12288 data: 3 5 4
20142 107/300 Write block 883 len 24576 data: 6 7 5 5 4 6
20143 108/300 Read block 1160 len 40960
20144 109/300 Write block 457 len 28672 data: 2 3 4 5 5 4 4
20145 110/300 Write block 4129 len 12288 data: 5 5 5
20146 111/300 Write block 1696 len 28672 data: 3 4 3 4 4 4 6
20147 112/300 Write block 4762 len 12288 data: 3 3 3
20148 113/300 Read block 3436 len 16384
20149 114/300 Read block 2994 len 16384
20150 115/300 Write block 1582 len 16384 data: 4 6 6 7
20151 116/300 Read block 2810 len 36864
20152 117/300 Write block 4308 len 24576 data: 6 5 4 4 5 6
20153 118/300 Read block 1344 len 16384
20154 119/300 Read block 346 len 12288
20155 120/300 Read block 3767 len 40960
20156 121/300 Write block 3791 len 20480 data: 7 7 8 6 6
20157 122/300 Write block 2757 len 16384 data: 8 8 8 9
20158 123/300 Read block 4529 len 36864
20159 124/300 Read block 1375 len 12288
20160 125/300 Write block 2217 len 4096 data: 9
20161 126/300 Read block 765 len 4096
20162 127/300 Write block 4606 len 12288 data: 6 7 6
20163 128/300 Read block 2448 len 40960
20164 129/300 Write block 3397 len 16384 data: 7 5 7 9
20165 130/300 Read block 555 len 20480
20166 131/300 Write block 3647 len 20480 data: 3 4 5 6 6
20167 132/300 Read block 2113 len 8192
20168 133/300 Read block 1956 len 16384
20169 134/300 Write block 1447 len 32768 data: 5 6 6 7 7 7 6 4
20170 135/300 Write block 3818 len 28672 data: 4 6 5 5 5 4 5
20171 136/300 Read block 3662 len 40960
20172 137/300 Flush
20173 138/300 Write block 774 len 12288 data: 2 3 3
20174 139/300 Read block 1004 len 12288
20175 140/300 Read block 1441 len 20480
20176 141/300 Read block 3999 len 12288
20177 142/300 Write block 3555 len 32768 data: 4 5 4 4 4 5 5 6
20178 143/300 Flush
20179 144/300 Read block 3464 len 20480
20180 145/300 Read block 4156 len 4096
20181 146/300 Read block 2801 len 32768
20182 147/300 Read block 2957 len 12288
20183 148/300 Write block 2290 len 28672 data: 5 5 5 5 5 5 3
20184 149/300 Write block 1207 len 24576 data: 6 6 7 7 6 7
20185 150/300 Read block 1166 len 24576
20186 151/300 Write block 390 len 28672 data: 7 7 8 6 6 8 7
20187 152/300 Write block 4383 len 32768 data: 3 4 4 4 4 3 5 5
20188 153/300 Flush
20189 154/300 Read block 4444 len 20480
20190 155/300 Read block 2518 len 8192
20191 156/300 Read block 3963 len 8192
20192 157/300 Read block 1831 len 16384
20193 158/300 Read block 4027 len 12288
20194 159/300 Read block 1413 len 36864
20195 160/300 Write block 628 len 12288 data: 3 4 3
20196 161/300 Write block 3283 len 20480 data: 7 7 8 8 6
20197 162/300 Read block 2379 len 8192
20198 163/300 Write block 3169 len 12288 data: 8 7 4
20199 164/300 Read block 2939 len 8192
20200 165/300 Read block 441 len 8192
20201 166/300 Read block 1296 len 40960
20202 167/300 Read block 454 len 16384
20203 168/300 Read block 1008 len 32768
20204 169/300 Read block 3972 len 16384
20205 170/300 Write block 182 len 8192 data: 3 3
20206 171/300 Flush
20207 172/300 Flush
20208 173/300 Read block 175 len 36864
20209 174/300 Write block 3144 len 16384 data: 6 6 6 6
20210 175/300 Write block 919 len 40960 data: 6 6 5 5 6 6 6 5 5 4
20211 176/300 Read block 4611 len 12288
20212 177/300 Read block 1878 len 8192
20213 178/300 Read block 4365 len 32768
20214 179/300 Read block 2103 len 32768
20215 180/300 Write block 217 len 28672 data: 2 3 3 4 4 5 6
20216 181/300 Read block 4701 len 8192
20217 182/300 Read block 712 len 24576
20218 183/300 Read block 3724 len 32768
20219 184/300 Write block 4918 len 40960 data: 5 6 6 5 6 7 7 5 4 4
20220 185/300 Read block 4604 len 16384
20221 186/300 Read block 3996 len 32768
20222 187/300 Read block 1038 len 24576
20223 188/300 Read block 4822 len 28672
20224 189/300 Write block 1404 len 32768 data: 6 6 5 8 8 6 7 7
20225 190/300 Read block 568 len 4096
20226 191/300 Write block 333 len 8192 data: 4 5
20227 192/300 Read block 359 len 24576
20228 193/300 Write block 2407 len 20480 data: 6 5 4 4 4
20229 194/300 Write block 4904 len 28672 data: 9 7 6 6 6 7 6
20230 195/300 Write block 18 len 8192 data: 8 7
20231 196/300 Read block 2868 len 24576
20232 197/300 Read block 361 len 36864
20233 198/300 Read block 4360 len 12288
20234 199/300 Write block 1119 len 36864 data: 5 5 5 5 7 8 6 7 7
20235 200/300 Write block 4005 len 40960 data: 5 5 5 4 4 4 3 4 4 5
20236 201/300 Read block 2865 len 24576
20237 202/300 Write block 4423 len 12288 data: 3 3 4
20238 203/300 Flush
20239 204/300 Read block 1008 len 36864
20240 205/300 Read block 4736 len 12288
20241 206/300 Write block 1318 len 4096 data: 4
20242 207/300 Write block 511 len 4096 data: 5
20243 208/300 Write block 4221 len 32768 data: 6 7 7 7 7 6 5 5
20244 209/300 Read block 631 len 40960
20245 210/300 Write block 4083 len 28672 data: 3 3 4 3 3 3 3
20246 211/300 Write block 4923 len 28672 data: 8 8 6 5 5 6 6
20247 212/300 Write block 1258 len 28672 data: 7 7 8 8 6 6 7
20248 213/300 Write block 4728 len 16384 data: 2 3 3 4
20249 214/300 Write block 4942 len 12288 data: 4 4 5
20250 215/300 Read block 4030 len 16384
20251 216/300 Read block 490 len 28672
20252 217/300 Read block 1820 len 16384
20253 218/300 Read block 1587 len 4096
20254 219/300 Read block 1983 len 12288
20255 220/300 Write block 1765 len 28672 data: 4 5 5 5 5 5 5
20256 221/300 Write block 4191 len 4096 data: 4
20257 222/300 Flush
20258 223/300 Write block 2933 len 12288 data: 5 3 4
20259 224/300 Read block 4614 len 40960
20260 225/300 Read block 1288 len 28672
20261 226/300 Read block 1786 len 12288
20262 227/300 Write block 717 len 12288 data: 5 7 5
20263 228/300 Read block 1137 len 24576
20264 229/300 Write block 2616 len 40960 data: 7 7 5 6 6 6 4 5 5 7
20265 230/300 Read block 4242 len 12288
20266 231/300 Write block 958 len 8192 data: 5 4
20267 232/300 Write block 1402 len 24576 data: 4 4 7 7 6 9
20268 233/300 Read block 110 len 28672
20269 234/300 Read block 3051 len 36864
20270 235/300 Flush
20271 236/300 Write block 1071 len 28672 data: 7 7 7 7 6 5 6
20272 237/300 Read block 30 len 32768
20273 238/300 Write block 4355 len 24576 data: 6 7 5 6 7 7
20274 239/300 Read block 2971 len 8192
20275 240/300 Write block 4390 len 8192 data: 6 6
20276 241/300 Read block 3868 len 32768
20277 242/300 Read block 3836 len 28672
20278 243/300 Write block 4457 len 20480 data: 9 6 6 6 5
20279 244/300 Read block 2060 len 36864
20280 245/300 Read block 1410 len 32768
20281 246/300 Write block 1381 len 28672 data: 8 7 6 5 5 3 3
20282 247/300 Write block 1378 len 8192 data: 5 5
20283 248/300 Read block 1484 len 28672
20284 249/300 Read block 3934 len 40960
20285 250/300 Flush
20286 251/300 Read block 3006 len 40960
20287 252/300 Flush
20288 253/300 Read block 1029 len 32768
20289 254/300 Read block 4748 len 4096
20290 255/300 Write block 2829 len 4096 data: 9
20291 256/300 Write block 472 len 40960 data: 5 5 4 3 4 4 3 2 2 2
20292 257/300 Write block 2882 len 20480 data: 6 6 6 7 6
20293 258/300 Write block 3187 len 28672 data: 8 8 8 8 8 7 7
20294 259/300 Read block 2814 len 20480
20295 260/300 Flush
20296 261/300 Read block 548 len 40960
20297 262/300 Write block 1502 len 32768 data: 5 5 6 6 6 5 5 6
20298 263/300 Read block 671 len 24576
20299 264/300 Flush
20300 265/300 Flush
20301 266/300 Read block 2065 len 20480
20302 267/300 Read block 4801 len 36864
20303 268/300 Write block 1442 len 16384 data: 4 4 4 4
20304 269/300 Read block 2491 len 36864
20305 270/300 Read block 1769 len 8192
20306 271/300 Write block 3362 len 16384 data: 6 6 5 5
20307 272/300 Write block 873 len 16384 data: 6 7 6 5
20308 273/300 Read block 2756 len 4096
20309 274/300 Write block 2099 len 16384 data: 5 4 4 3
20310 275/300 Read block 4506 len 24576
20311 276/300 Read block 2815 len 16384
20312 277/300 Flush
20313 278/300 Flush
20314 279/300 Read block 1117 len 32768
20315 280/300 Write block 3093 len 24576 data: 4 3 3 3 3 5
20316 281/300 Write block 2136 len 16384 data: 4 4 4 4
20317 282/300 Write block 1623 len 36864 data: 6 4 4 4 3 4 3 3 5
20318 283/300 Read block 3515 len 24576
20319 284/300 Read block 4192 len 36864
20320 285/300 Read block 4472 len 32768
20321 286/300 Write block 2413 len 36864 data: 4 3 3 3 3 2 2 2 4
20322 287/300 Read block 2389 len 32768
20323 288/300 Read block 3825 len 16384
20324 289/300 Flush
20325 290/300 Read block 1811 len 12288
20326 291/300 Read block 1914 len 4096
20327 292/300 Write block 2185 len 4096 data: 3
20328 293/300 Write block 4784 len 16384 data: 5 5 7 6
20329 294/300 Write block 3034 len 20480 data: 4 4 5 6 7
20330 295/300 Write block 4526 len 12288 data: 3 3 3
20331 296/300 Flush
20332 297/300 Write block 1576 len 32768 data: 3 3 4 3 2 3 5 7
20333 298/300 Write block 1582 len 12288 data: 6 8 7
20334 299/300 Read block 2581 len 8192
20335 300/300 Read block 3307 len 32768
20336 Main task has work to do, go find it
20337 got dsc Start(0)
20338 start 0
20339 Feb 12 01:38:33.953 INFO request completed, latency_us: 224, response_code: 204, uri: /start/cid/0, method: POST, req_id: 0cf8a82f-d206-481a-a711-ea07755b5be0, remote_addr: 127.0.0.1:60456, local_addr: 127.0.0.1:9998
20340 Replay: started 0, returned:Ok(())
20341 ----------------------------------------------------------------
20342 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
20343 States: Offline Active Active
20344 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
20345 6633 Acked 7632 Read 10 New Done Done true
20346 6634 Acked 7633 Flush 0 New Done Done true
20347 6635 Acked 7634 Write 7 New Done Done false
20348 6636 Acked 7635 Write 8 New Done Done false
20349 6637 Acked 7636 Read 1 New Done Done false
20350 6638 Acked 7637 Flush 0 New Done Done false
20351 6639 Acked 7638 Read 10 New Done Done false
20352 6640 Acked 7639 Read 7 New Done Done false
20353 6641 Acked 7640 Write 9 New Done Done false
20354 6642 Acked 7641 Flush 0 New Done Done false
20355 6643 Acked 7642 Read 3 New Done Done false
20356 6644 Acked 7643 Read 8 New Done Done false
20357 6645 Acked 7644 Write 7 New Done Done false
20358 6646 Acked 7645 Read 7 New Done Done false
20359 6647 Acked 7646 Write 9 New Done Done false
20360 6648 Acked 7647 Read 5 New Done Done false
20361 6649 Acked 7648 Read 10 New Done Done false
20362 6650 Acked 7649 Read 8 New Done Done false
20363 6651 Acked 7650 Write 8 New Done Done false
20364 6652 Acked 7651 Write 6 New Done Done false
20365 6653 Acked 7652 Read 1 New Done Done false
20366 6654 Acked 7653 Write 8 New Done Done false
20367 6655 Acked 7654 Read 9 New Done Done false
20368 6656 Acked 7655 Read 4 New Done Done false
20369 6657 Acked 7656 Write 3 New Done Done false
20370 6658 Acked 7657 Write 1 New Done Done false
20371 6659 Acked 7658 Read 10 New Done Done false
20372 6660 Acked 7659 Read 3 New Done Done false
20373 6661 Acked 7660 Read 2 New Done Done false
20374 6662 Acked 7661 Write 7 New Done Done false
20375 6663 Acked 7662 Read 4 New Done Done false
20376 6664 Acked 7663 Write 8 New Done Done false
20377 6665 Acked 7664 Read 9 New Done Done false
20378 6666 Acked 7665 Read 3 New Done Done false
20379 6667 Acked 7666 Write 1 New Done Done false
20380 6668 Acked 7667 Flush 0 New Done Done false
20381 6669 Acked 7668 Flush 0 New Done Done false
20382 6670 Acked 7669 Read 5 New Done Done false
20383 6671 Acked 7670 Write 3 New Done Done false
20384 6672 Acked 7671 Read 5 New Done Done false
20385 6673 Acked 7672 Read 6 New Done Done false
20386 6674 Acked 7673 Read 5 New Done Done false
20387 6675 Acked 7674 Read 1 New Done Done false
20388 6676 Acked 7675 Read 8 New Done Done false
20389 6677 Acked 7676 Write 9 New Done Done false
20390 6678 Acked 7677 Write 1 New Done Done false
20391 6679 Acked 7678 Write 2 New Done Done false
20392 6680 Acked 7679 Write 2 New Done Done false
20393 6681 Acked 7680 Write 4 New Done Done false
20394 6682 Acked 7681 Write 6 New Done Done false
20395 6683 Acked 7682 Read 3 New Done Done false
20396 6684 Acked 7683 Read 3 New Done Done false
20397 6685 Acked 7684 Write 8 New Done Done false
20398 6686 Acked 7685 Read 2 New Done Done false
20399 6687 Acked 7686 Write 4 New Done Done false
20400 6688 Acked 7687 Write 4 New Done Done false
20401 6689 Acked 7688 Read 5 New Done Done false
20402 6690 Acked 7689 Read 2 New Done Done false
20403 6691 Acked 7690 Read 8 New Done Done false
20404 6692 Acked 7691 Read 2 New Done Done false
20405 6693 Acked 7692 Read 5 New Done Done false
20406 6694 Acked 7693 Write 8 New Done Done false
20407 6695 Acked 7694 Write 1 New Done Done false
20408 6696 Acked 7695 Flush 0 New Done Done false
20409 6697 Acked 7696 Read 9 New Done Done false
20410 6698 Acked 7697 Read 8 New Done Done false
20411 6699 Acked 7698 Write 2 New Done Done false
20412 6700 Acked 7699 Write 5 New Done Done false
20413 6701 Acked 7700 Write 7 New Done Done false
20414 6702 Acked 7701 Flush 0 New Done Done false
20415 6703 Acked 7702 Read 10 New Done Done false
20416 6704 Acked 7703 Write 4 New Done Done false
20417 6705 Acked 7704 Read 1 New Done Done false
20418 6706 Acked 7705 Flush 0 New Done Done false
20419 6707 Acked 7706 Read 8 New Done Done false
20420 6708 Acked 7707 Write 3 New Done Done false
20421 6709 Acked 7708 Write 4 New Done Done false
20422 6710 Acked 7709 Read 4 New Done Done false
20423 6711 Acked 7710 Read 4 New Done Done false
20424 6712 Acked 7711 Read 1 New Done Done false
20425 6713 Acked 7712 Write 6 New Done Done false
20426 6714 Acked 7713 Read 8 New Done Done false
20427 6715 Acked 7714 Read 5 New Done Done false
20428 6716 Acked 7715 Read 9 New Done Done false
20429 6717 Acked 7716 Flush 0 New Done Done false
20430 6718 Acked 7717 Read 7 New Done Done false
20431 6719 Acked 7718 Flush 0 New Done Done false
20432 6720 Acked 7719 Read 5 New Done Done false
20433 6721 Acked 7720 Write 6 New Done Done false
20434 6722 Acked 7721 Write 9 New Done Done false
20435 6723 Acked 7722 Read 8 New Done Done false
20436 6724 Acked 7723 Read 5 New Done Done false
20437 6725 Acked 7724 Write 5 New Done Done false
20438 6726 Acked 7725 Write 6 New Done Done false
20439 6727 Acked 7726 Read 6 New Done Done false
20440 6728 Acked 7727 Read 5 New Done Done false
20441 6729 Acked 7728 Flush 0 New Done Done false
20442 6730 Acked 7729 Read 2 New Done Done false
20443 6731 Acked 7730 Read 2 New Done Done false
20444 6732 Acked 7731 Write 9 New Done Done false
20445 6733 Acked 7732 Write 7 New Done Done false
20446 6734 Acked 7733 Read 3 New Done Done false
20447 6735 Acked 7734 Write 8 New Done Done false
20448 6736 Acked 7735 Read 8 New Done Done false
20449 6737 Acked 7736 Write 2 New Done Done false
20450 6738 Acked 7737 Read 3 New Done Done false
20451 6739 Acked 7738 Write 3 New Done Done false
20452 6740 Acked 7739 Write 6 New Done Done false
20453 6741 Acked 7740 Read 10 New Done Done false
20454 6742 Acked 7741 Write 7 New Done Done false
20455 6743 Acked 7742 Write 3 New Done Done false
20456 6744 Acked 7743 Write 7 New Done Done false
20457 6745 Acked 7744 Write 3 New Done Done false
20458 6746 Acked 7745 Read 4 New Done Done false
20459 6747 Acked 7746 Read 4 New Done Done false
20460 6748 Acked 7747 Write 4 New Done Done false
20461 6749 Acked 7748 Read 9 New Done Done false
20462 6750 Acked 7749 Write 6 New Done Done false
20463 6751 Acked 7750 Read 4 New Done Done false
20464 6752 Acked 7751 Read 3 New Done Done false
20465 6753 Acked 7752 Read 10 New Done Done false
20466 6754 Acked 7753 Write 5 New Done Done false
20467 6755 Acked 7754 Write 4 New Done Done false
20468 6756 Acked 7755 Read 9 New Done Done false
20469 6757 Acked 7756 Read 3 New Done Done false
20470 6758 Acked 7757 Write 1 New Done Done false
20471 6759 Acked 7758 Read 1 New Done Done false
20472 6760 Acked 7759 Write 3 New Done Done false
20473 6761 Acked 7760 Read 10 New Done Done false
20474 6762 Acked 7761 Write 4 New Done Done false
20475 6763 Acked 7762 Read 5 New Done Done false
20476 6764 Acked 7763 Write 5 New Done Done false
20477 6765 Acked 7764 Read 2 New Done Done false
20478 6766 Acked 7765 Read 4 New Done Done false
20479 6767 Acked 7766 Write 8 New Done Done false
20480 6768 Acked 7767 Write 7 New Done Done false
20481 6769 Acked 7768 Read 10 New Done Done false
20482 6770 Acked 7769 Flush 0 New Done Done false
20483 6771 Acked 7770 Write 3 New Done Done false
20484 6772 Acked 7771 Read 3 New Done Done false
20485 6773 Acked 7772 Read 5 New Done Done false
20486 6774 Acked 7773 Read 3 New Done Done false
20487 6775 Acked 7774 Write 8 New Done Done false
20488 6776 Acked 7775 Flush 0 New Done Done false
20489 6777 Acked 7776 Read 5 New Done Done false
20490 6778 Acked 7777 Read 1 New Done Done false
20491 6779 Acked 7778 Read 8 New Done Done false
20492 6780 Acked 7779 Read 3 New Done Done false
20493 6781 Acked 7780 Write 7 New Done Done false
20494 6782 Acked 7781 Write 6 New Done Done false
20495 6783 Acked 7782 Read 6 New Done Done false
20496 6784 Acked 7783 Write 7 New Done Done false
20497 6785 Acked 7784 Write 8 New Done Done false
20498 6786 Acked 7785 Flush 0 New Done Done false
20499 6787 Acked 7786 Read 5 New Done Done false
20500 6788 Acked 7787 Read 2 New Done Done false
20501 6789 Acked 7788 Read 2 New Done Done false
20502 6790 Acked 7789 Read 4 New Done Done false
20503 6791 Acked 7790 Read 3 New Done Done false
20504 6792 Acked 7791 Read 9 New Done Done false
20505 6793 Acked 7792 Write 3 New Done Done false
20506 6794 Acked 7793 Write 5 New Done Done false
20507 6795 Acked 7794 Read 2 New Done Done false
20508 6796 Acked 7795 Write 3 New Done Done false
20509 6797 Acked 7796 Read 2 New Done Done false
20510 6798 Acked 7797 Read 2 New Done Done false
20511 6799 Acked 7798 Read 10 New Done Done false
20512 6800 Acked 7799 Read 4 New Done Done false
20513 6801 Acked 7800 Read 8 New Done Done false
20514 6802 Acked 7801 Read 4 New Done Done false
20515 6803 Acked 7802 Write 2 New Done Done false
20516 6804 Acked 7803 Flush 0 New Done Done false
20517 6805 Acked 7804 Flush 0 New Done Done false
20518 6806 Acked 7805 Read 9 New Done Done false
20519 6807 Acked 7806 Write 4 New Done Done false
20520 6808 Acked 7807 Write 10 New Done Done false
20521 6809 Acked 7808 Read 3 New Done Done false
20522 6810 Acked 7809 Read 2 New Done Done false
20523 6811 Acked 7810 Read 8 New Done Done false
20524 6812 Acked 7811 Read 8 New Done Done false
20525 6813 Acked 7812 Write 7 New Done Done false
20526 6814 Acked 7813 Read 2 New Done Done false
20527 6815 Acked 7814 Read 6 New Done Done false
20528 6816 Acked 7815 Read 8 New Done Done false
20529 6817 Acked 7816 Write 10 New Done Done false
20530 6818 Acked 7817 Read 4 New Done Done false
20531 6819 Acked 7818 Read 8 New Done Done false
20532 6820 Acked 7819 Read 6 New Done Done false
20533 6821 Acked 7820 Read 7 New Done Done false
20534 6822 Acked 7821 Write 8 New Done Done false
20535 6823 Acked 7822 Read 1 New Done Done false
20536 6824 Acked 7823 Write 2 New Done Done false
20537 6825 Acked 7824 Read 6 New Done Done false
20538 6826 Acked 7825 Write 5 New Done Done false
20539 6827 Acked 7826 Write 7 New Done Done false
20540 6828 Acked 7827 Write 2 New Done Done false
20541 6829 Acked 7828 Read 6 New Done Done false
20542 6830 Acked 7829 Read 9 New Done Done false
20543 6831 Acked 7830 Read 3 New Done Done false
20544 6832 Acked 7831 Write 9 New Done Done false
20545 6833 Acked 7832 Write 10 New Done Done false
20546 6834 Acked 7833 Read 6 New Done Done false
20547 6835 Acked 7834 Write 3 New Done Done false
20548 6836 Acked 7835 Flush 0 New Done Done false
20549 6837 Acked 7836 Read 9 New Done Done false
20550 6838 Acked 7837 Read 3 New Done Done false
20551 6839 Acked 7838 Write 1 New Done Done false
20552 6840 Acked 7839 Write 1 New Done Done false
20553 6841 Acked 7840 Write 8 New Done Done false
20554 6842 Acked 7841 Read 10 New Done Done false
20555 6843 Acked 7842 Write 7 New Done Done false
20556 6844 Acked 7843 Write 7 New Done Done false
20557 6845 Acked 7844 Write 7 New Done Done false
20558 6846 Acked 7845 Write 4 New Done Done false
20559 6847 Acked 7846 Write 3 New Done Done false
20560 6848 Acked 7847 Read 4 New Done Done false
20561 6849 Acked 7848 Read 7 New Done Done false
20562 6850 Acked 7849 Read 4 New Done Done false
20563 6851 Acked 7850 Read 1 New Done Done false
20564 6852 Acked 7851 Read 3 New Done Done false
20565 6853 Acked 7852 Write 7 New Done Done false
20566 6854 Acked 7853 Write 1 New Done Done false
20567 6855 Acked 7854 Flush 0 New Done Done false
20568 6856 Acked 7855 Write 3 New Done Done false
20569 6857 Acked 7856 Read 10 New Done Done false
20570 6858 Acked 7857 Read 7 New Done Done false
20571 6859 Acked 7858 Read 3 New Done Done false
20572 6860 Acked 7859 Write 3 New Done Done false
20573 6861 Acked 7860 Read 6 New Done Done false
20574 6862 Acked 7861 Write 10 New Done Done false
20575 6863 Acked 7862 Read 3 New Done Done false
20576 6864 Acked 7863 Write 2 New Done Done false
20577 6865 Acked 7864 Write 6 New Done Done false
20578 6866 Acked 7865 Read 7 New Done Done false
20579 6867 Acked 7866 Read 9 New Done Done false
20580 6868 Acked 7867 Flush 0 New Done Done false
20581 6869 Acked 7868 Write 7 New Done Done false
20582 6870 Acked 7869 Read 8 New Done Done false
20583 6871 Acked 7870 Write 6 New Done Done false
20584 6872 Acked 7871 Read 2 New Done Done false
20585 6873 Acked 7872 Write 2 New Done Done false
20586 6874 Acked 7873 Read 8 New Done Done false
20587 6875 Acked 7874 Read 7 New Done Done false
20588 6876 Acked 7875 Write 5 New Done Done false
20589 6877 Acked 7876 Read 9 New Done Done false
20590 6878 Acked 7877 Read 8 New Done Done false
20591 6879 Acked 7878 Write 7 New Done Done false
20592 6880 Acked 7879 Write 2 New Done Done false
20593 6881 Acked 7880 Read 7 New Done Done false
20594 6882 Acked 7881 Read 10 New Done Done false
20595 6883 Acked 7882 Flush 0 New Done Done false
20596 6884 Acked 7883 Read 10 New Done Done false
20597 6885 Acked 7884 Flush 0 New Done Done false
20598 6886 Acked 7885 Read 8 New Done Done false
20599 6887 Acked 7886 Read 1 New Done Done false
20600 6888 Acked 7887 Write 1 New Done Done false
20601 6889 Acked 7888 Write 10 New Done Done false
20602 6890 Acked 7889 Write 5 New Done Done false
20603 6891 Acked 7890 Write 7 New Done Done false
20604 6892 Acked 7891 Read 5 New Done Done false
20605 6893 Acked 7892 Flush 0 New Done Done false
20606 6894 Acked 7893 Read 10 New Done Done false
20607 6895 Acked 7894 Write 8 New Done Done false
20608 6896 Acked 7895 Read 6 New Done Done false
20609 6897 Acked 7896 Flush 0 New Done Done false
20610 6898 Acked 7897 Flush 0 New Done Done false
20611 6899 Acked 7898 Read 5 New Done Done false
20612 6900 Acked 7899 Read 9 New Done Done false
20613 6901 Acked 7900 Write 4 New Done Done false
20614 6902 Acked 7901 Read 9 New Done Done false
20615 6903 Acked 7902 Read 2 New Done Done false
20616 6904 Acked 7903 Write 4 New Done Done false
20617 6905 Acked 7904 Write 4 New Done Done false
20618 6906 Acked 7905 Read 1 New Done Done false
20619 6907 Acked 7906 Write 4 New Done Done false
20620 6908 Acked 7907 Read 6 New Done Done false
20621 6909 Acked 7908 Read 4 New Done Done false
20622 6910 Acked 7909 Flush 0 New Done Done false
20623 6911 Acked 7910 Flush 0 New Done Done false
20624 6912 Acked 7911 Read 8 New Done Done false
20625 6913 Acked 7912 Write 6 New Done Done false
20626 6914 Acked 7913 Write 4 New Done Done false
20627 6915 Acked 7914 Write 9 New Done Done false
20628 6916 Acked 7915 Read 6 New Done Done false
20629 6917 Acked 7916 Read 9 New Done Done false
20630 6918 Acked 7917 Read 8 New Done Done false
20631 6919 Acked 7918 Write 9 New Done Done false
20632 6920 Acked 7919 Read 8 New Done Done false
20633 6921 Acked 7920 Read 4 New Done Done false
20634 6922 Acked 7921 Flush 0 New Done Done false
20635 6923 Acked 7922 Read 3 New Done Done false
20636 6924 Acked 7923 Read 1 New Done Done false
20637 6925 Acked 7924 Write 1 New Done Done false
20638 6926 Acked 7925 Write 4 New Done Done false
20639 6927 Acked 7926 Write 5 New Done Done false
20640 6928 Acked 7927 Write 3 New Done Done false
20641 6929 Acked 7928 Flush 0 New Done Done false
20642 6930 Acked 7929 Write 8 New Done Done false
20643 6931 Acked 7930 Write 3 New Done Done false
20644 6932 Acked 7931 Read 2 New Done Done false
20645 6933 Acked 7932 Read 8 New Done Done false
20646 STATES DS:0 DS:1 DS:2 TOTAL
20647 New 301 0 0 301
20648 Sent 0 0 0 0
20649 Done 0 301 301 602
20650 Skipped 0 0 0 0
20651 Error 0 0 0 0
20652 Last Flush: 7631 7928 7928
20653 Downstairs last five completed: 7631 7630 7629 7628 7627
20654 Upstairs last five completed: 6933 6932 6931 6930 6929
20655 CLIENT: Up:0 ds:301 act:2
206562024-02-12T01:38:34.614ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
206572024-02-12T01:38:34.614ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
206582024-02-12T01:38:34.614ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
206592024-02-12T01:38:34.614ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
206602024-02-12T01:38:34.614ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
206612024-02-12T01:38:34.614ZINFOcrucible: [0] client re-new 302 jobs since flush 7631 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
206622024-02-12T01:38:34.614ZINFOcrucible: client task is exiting = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
20663 [8810] got start action
20664 [8810] I am going to restart
20665 Starting downstairs at port 8810
20666 Make output file at "/tmp/dsc/downstairs-8810.txt"
20667 [8810][0] reports Starting
206682024-02-12T01:38:35.616ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
206692024-02-12T01:38:35.616ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
206702024-02-12T01:38:35.616ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
206712024-02-12T01:38:35.616ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
206722024-02-12T01:38:35.616ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
206732024-02-12T01:38:35.616ZINFOcrucible: [0] client re-new 302 jobs since flush 7631 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
206742024-02-12T01:38:35.616ZINFOcrucible: client task is exiting = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
20675 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1187)
20676 [8810][0] reports Running
206772024-02-12T01:38:36.617ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
206782024-02-12T01:38:36.617ZINFOcrucible: ds_connection connected = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
206792024-02-12T01:38:36.618ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
206802024-02-12T01:38:36.618ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
206812024-02-12T01:38:36.618ZINFOcrucible: send last flush ID to this DS: 7631 = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
206822024-02-12T01:38:36.619ZINFOcrucible: Replied this last flush ID: 7631 = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
206832024-02-12T01:38:36.619ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
206842024-02-12T01:38:36.619ZINFOcrucible: [0] Transition from Offline to Replay = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
206852024-02-12T01:38:36.619ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
206862024-02-12T01:38:36.619ZINFOcrucible: [0] Transition from Replay to Active = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
20687 ----------------------------------------------------------------
20688 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
20689 Downstairs last five completed: 7933 7932 7931 7930 7929
20690 Upstairs last five completed: 6934 6933 6932 6931 6930
20691 CLIENT: Up:0 ds:0 act:3
20692 Replay: All jobs finished, all DS active.
20693 Main task has work to do, go find it
20694 got dsc Stop(0)
20695 stop 0
20696 Feb 12 01:38:37.956 INFO request completed, latency_us: 406, response_code[8810] Got stop action so:false kr:false
20697 : 204, uri: /stop/cid/0, method: POST, req_id: e00a228c-c0f6-4958-aaa3-7c6d9149b44c, remote_addr: 127.0.0.1:60456, local_addr: 127.0.0.1:9998
206982024-02-12T01:38:37.957ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
206992024-02-12T01:38:37.957ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
207002024-02-12T01:38:37.957ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
20701 Feb 12 01:38:37.957 INFO request completed, latency_us: 144, response_code:{"msg":"Gone missing, transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-02-12T01:38:37.957440981Z","hostname":"ip-10-150-1-104.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"e1773b3e-b36f-4080-8b13-c0830f0d072e"}
20702 200, uri: /state/cid/0, method: GET, req_id: 736c1926-b8f8-4211-9a41-32b6bd27d2df, remote_addr: 127.0.0.1:60456, local_addr: 127.0.0.1:9998
207032024-02-12T01:38:37.957ZINFOcrucible: [0] client re-new 0 jobs since flush 7933 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
207042024-02-12T01:38:37.957ZINFOcrucible: client task is exiting = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
20705 [8810] Exited with: ExitStatus(unix_wait_status(9))
20706 [8810][0] reports Exit
207072024-02-12T01:38:38.958ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
207082024-02-12T01:38:38.958ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
207092024-02-12T01:38:38.958ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
207102024-02-12T01:38:38.958ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
207112024-02-12T01:38:38.958ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
207122024-02-12T01:38:38.958ZINFOcrucible: [0] client re-new 0 jobs since flush 7933 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
207132024-02-12T01:38:38.958ZINFOcrucible: client task is exiting = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
207142024-02-12T01:38:39.960ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
207152024-02-12T01:38:39.960ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
207162024-02-12T01:38:39.960ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
207172024-02-12T01:38:39.960ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
207182024-02-12T01:38:39.960ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
207192024-02-12T01:38:39.960ZINFOcrucible: [0] client re-new 0 jobs since flush 7933 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
207202024-02-12T01:38:39.960ZINFOcrucible: client task is exiting = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
207212024-02-12T01:38:40.962ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
207222024-02-12T01:38:40.962ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
207232024-02-12T01:38:40.962ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
207242024-02-12T01:38:40.962ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
207252024-02-12T01:38:40.962ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
207262024-02-12T01:38:40.962ZINFOcrucible: [0] client re-new 0 jobs since flush 7933 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
207272024-02-12T01:38:40.962ZINFOcrucible: client task is exiting = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
20728 Feb 12 01:38:41.959 INFO request completed, latency_us: 211, response_code: 200, uri: /state/cid/0, method: GET, req_id: d69c707c-4026-40b8-ba34-02bcf487b01a, remote_addr: 127.0.0.1:60456, local_addr: 127.0.0.1:9998
20729 001/300 Write block 3895 len 12288 data: 6 7 7
20730 002/300 Read block 852 len 8192
20731 003/300 Read block 3796 len 20480
20732 004/300 Read block 4370 len 28672
20733 005/300 Read block 3053 len 24576
207342024-02-12T01:38:41.964ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
207352024-02-12T01:38:41.964ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
207362024-02-12T01:38:41.964ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
207372024-02-12T01:38:41.964ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
207382024-02-12T01:38:41.964ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
207392024-02-12T01:38:41.964ZINFOcrucible: [0] client re-new 5 jobs since flush 7933 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
207402024-02-12T01:38:41.964ZINFOcrucible: client task is exiting = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
20741 006/300 Write block 3143 len 4096 data: 7
20742 007/300 Read block 3404 len 8192
20743 008/300 Write block 4356 len 16384 data: 8 6 7 8
20744 009/300 Write block 1846 len 32768 data: 6 8 9 8 9 8 6 5
20745 010/300 Write block 3563 len 12288 data: 6 5 3
20746 011/300 Read block 13 len 40960
20747 012/300 Flush
20748 013/300 Write block 1369 len 12288 data: 4 3 3
20749 014/300 Read block 3482 len 32768
20750 015/300 Read block 205 len 12288
20751 016/300 Read block 2836 len 32768
20752 017/300 Write block 1924 len 4096 data: 4
20753 018/300 Write block 3238 len 16384 data: 9 8 8 6
20754 019/300 Write block 3606 len 8192 data: 6 6
20755 020/300 Flush
20756 021/300 Read block 666 len 40960
20757 022/300 Write block 1648 len 4096 data: 4
20758 023/300 Read block 2154 len 32768
20759 024/300 Write block 869 len 24576 data: 3 5 6 6 7 8
20760 025/300 Write block 2811 len 16384 data: 11 10 9 6
20761 026/300 Write block 4465 len 24576 data: 3 2 3 3 4 6
20762 027/300 Write block 3321 len 20480 data: 4 3 2 2 4
20763 028/300 Flush
20764 029/300 Flush
20765 030/300 Flush
20766 031/300 Write block 3885 len 24576 data: 4 3 4 5 5 5
20767 032/300 Read block 2903 len 20480
20768 033/300 Write block 3644 len 40960 data: 5 4 4 4 5 6 7 7 5 5
20769 034/300 Read block 3461 len 40960
20770 035/300 Read block 4445 len 16384
20771 036/300 Read block 1404 len 32768
20772 037/300 Write block 4874 len 16384 data: 6 6 5 4
20773 038/300 Flush
20774 039/300 Read block 3081 len 4096
20775 040/300 Read block 2713 len 40960
20776 041/300 Write block 4508 len 12288 data: 7 7 6
20777 042/300 Read block 3262 len 16384
20778 043/300 Read block 4116 len 40960
20779 044/300 Write block 2272 len 12288 data: 5 6 6
20780 045/300 Flush
20781 046/300 Write block 4243 len 28672 data: 5 5 6 7 7 7 8
20782 047/300 Read block 4715 len 12288
20783 048/300 Read block 4539 len 20480
20784 049/300 Read block 4528 len 20480
20785 050/300 Read block 4574 len 24576
20786 051/300 Read block 294 len 40960
20787 052/300 Read block 2505 len 4096
20788 053/300 Read block 2606 len 24576
20789 054/300 Read block 4795 len 4096
20790 055/300 Read block 4965 len 20480
20791 056/300 Read block 707 len 36864
20792 057/300 Read block 1293 len 20480
20793 058/300 Write block 3535 len 32768 data: 5 5 5 3 4 4 3 3
20794 059/300 Write block 2700 len 16384 data: 6 4 4 4
20795 060/300 Write block 1909 len 8192 data: 4 3
20796 061/300 Read block 3494 len 24576
20797 062/300 Write block 495 len 20480 data: 7 7 6 6 6
20798 063/300 Read block 1266 len 24576
20799 064/300 Read block 2795 len 40960
20800 065/300 Read block 3763 len 28672
20801 066/300 Flush
20802 067/300 Flush
20803 068/300 Write block 1373 len 4096 data: 3
20804 069/300 Read block 823 len 24576
20805 070/300 Read block 4371 len 28672
20806 071/300 Write block 2825 len 4096 data: 8
20807 072/300 Write block 3140 len 12288 data: 6 7 8
20808 073/300 Read block 1808 len 24576
20809 074/300 Write block 3573 len 40960 data: 4 5 8 7 6 6 5 5 4 4
20810 075/300 Flush
20811 076/300 Write block 1290 len 32768 data: 3 3 2 2 2 2 2 2
20812 077/300 Read block 526 len 8192
20813 078/300 Flush
20814 079/300 Write block 858 len 28672 data: 3 3 3 3 4 4 4
20815 080/300 Read block 650 len 12288
20816 081/300 Read block 1254 len 4096
20817 082/300 Read block 1087 len 28672
20818 083/300 Read block 804 len 36864
20819 084/300 Read block 4139 len 4096
20820 085/300 Write block 3034 len 4096 data: 5
20821 086/300 Write block 192 len 4096 data: 5
20822 087/300 Read block 3584 len 36864
20823 088/300 Write block 4801 len 8192 data: 6 5
20824 089/300 Flush
20825 090/300 Read block 797 len 4096
20826 091/300 Write block 2035 len 20480 data: 3 4 4 4 4
20827 092/300 Write block 4052 len 24576 data: 3 3 4 4 4 4
20828 093/300 Read block 1650 len 20480
20829 094/300 Read block 3660 len 8192
20830 095/300 Read block 3171 len 40960
20831 096/300 Read block 381 len 20480
20832 097/300 Read block 2284 len 12288
20833 098/300 Read block 1187 len 24576
20834 099/300 Read block 3320 len 8192
20835 100/300 Write block 63 len 40960 data: 4 5 5 5 4 3 2 2 2 4
20836 101/300 Write block 4537 len 32768 data: 7 7 7 6 6 5 4 2
20837 102/300 Read block 1212 len 32768
20838 103/300 Write block 2002 len 16384 data: 6 5 4 3
20839 104/300 Write block 2444 len 28672 data: 9 8 6 6 8 7 4
20840 105/300 Read block 1842 len 24576
20841 106/300 Flush
20842 107/300 Read block 3886 len 28672
20843 108/300 Read block 2436 len 12288
20844 109/300 Write block 4812 len 4096 data: 3
20845 110/300 Write block 1990 len 16384 data: 4 4 3 5
20846 111/300 Read block 4361 len 12288
20847 112/300 Read block 2567 len 28672
20848 113/300 Write block 815 len 28672 data: 3 4 4 5 4 2 3
20849 114/300 Read block 1455 len 12288
20850 115/300 Read block 4191 len 40960
20851 116/300 Read block 4809 len 8192
20852 117/300 Read block 4907 len 40960
20853 118/300 Read block 4331 len 4096
20854 119/300 Flush
20855 120/300 Read block 1327 len 8192
20856 121/300 Read block 2842 len 20480
20857 122/300 Write block 2235 len 28672 data: 6 5 6 7 7 6 6
20858 123/300 Write block 4887 len 4096 data: 5
20859 124/300 Read block 3878 len 4096
20860 125/300 Read block 471 len 40960
20861 126/300 Write block 534 len 16384 data: 8 7 8 8
20862 127/300 Write block 3783 len 12288 data: 5 4 3
20863 128/300 Flush
20864 129/300 Read block 594 len 40960
20865 130/300 Write block 3374 len 16384 data: 7 8 8 8
20866 131/300 Flush
20867 132/300 Read block 2835 len 16384
20868 133/300 Write block 99 len 16384 data: 8 6 6 9
20869 134/300 Write block 3823 len 24576 data: 5 6 4 5 4 3
20870 135/300 Write block 1404 len 32768 data: 8 8 7 10 9 7 8 8
20871 136/300 Write block 2315 len 28672 data: 4 4 4 3 4 5 5
20872 137/300 Write block 1299 len 16384 data: 3 3 3 4
20873 138/300 Write block 529 len 24576 data: 4 5 5 7 7 9
20874 139/300 Read block 3328 len 12288
20875 140/300 Read block 1137 len 40960
20876 141/300 Write block 2310 len 32768 data: 4 5 5 5 4 5 5 5
20877 142/300 Read block 864 len 32768
20878 143/300 Read block 1868 len 12288
20879 144/300 Write block 4187 len 32768 data: 5 4 4 3 5 4 5 4
20880 145/300 Write block 421 len 4096 data: 5
20881 146/300 Write block 3598 len 32768 data: 4 3 4 3 5 5 5 6
20882 147/300 Write block 2496 len 4096 data: 3
20883 148/300 Write block 4976 len 28672 data: 4 4 4 4 4 3 2
20884 149/300 Flush
20885 150/300 Write block 4384 len 32768 data: 5 5 5 5 4 6 7 7
20886 151/300 Write block 4532 len 32768 data: 5 6 6 5 6 8 8 8
20887 152/300 Write block 1033 len 12288 data: 5 6 7
20888 153/300 Write block 4651 len 12288 data: 4 4 4
20889 154/300 Flush
20890 155/300 Read block 4738 len 40960
20891 156/300 Read block 1546 len 20480
20892 157/300 Read block 640 len 4096
20893 158/300 Write block 1563 len 4096 data: 4
20894 159/300 Write block 460 len 32768 data: 6 6 5 5 4 6 5 6
20895 160/300 Read block 3108 len 4096
20896 161/300 Write block 3622 len 32768 data: 6 5 5 6 6 6 5 6
20897 162/300 Read block 291 len 20480
20898 163/300 Flush
20899 164/300 Write block 4705 len 16384 data: 3 3 3 2
20900 165/300 Read block 3009 len 4096
20901 166/300 Flush
20902 167/300 Read block 4650 len 4096
20903 168/300 Read block 804 len 36864
20904 169/300 Write block 1485 len 28672 data: 5 4 4 4 3 3 3
20905 170/300 Flush
20906 171/300 Write block 446 len 4096 data: 8
20907 172/300 Write block 2121 len 24576 data: 6 6 6 5 6 7
20908 173/300 Flush
20909 174/300 Read block 1525 len 20480
20910 175/300 Write block 2329 len 28672 data: 4 4 4 5 6 6 7
20911 176/300 Read block 803 len 8192
20912 177/300 Read block 1023 len 36864
20913 178/300 Write block 4708 len 36864 data: 3 2 2 2 3 5 6 6 5
20914 179/300 Read block 1937 len 24576
20915 180/300 Read block 3860 len 40960
20916 181/300 Write block 4612 len 36864 data: 5 6 6 6 5 4 3 3 4
20917 182/300 Write block 1511 len 16384 data: 5 5 5 4
20918 183/300 Read block 1351 len 8192
20919 184/300 Read block 2743 len 32768
20920 185/300 Write block 4018 len 16384 data: 3 5 5 9
20921 186/300 Flush
20922 187/300 Write block 4895 len 8192 data: 2 2
20923 188/300 Read block 3908 len 4096
20924 189/300 Read block 4407 len 8192
20925 190/300 Write block 36 len 28672 data: 4 4 4 3 3 3 2
20926 191/300 Read block 374 len 4096
20927 192/300 Flush
20928 193/300 Read block 387 len 32768
20929 194/300 Read block 1945 len 28672
20930 195/300 Read block 3947 len 16384
20931 196/300 Read block 2678 len 32768
20932 197/300 Write block 3107 len 8192 data: 4 2
20933 198/300 Read block 2710 len 4096
20934 199/300 Read block 4950 len 24576
20935 200/300 Write block 2516 len 32768 data: 3 2 2 2 2 2 2 3
20936 201/300 Flush
20937 202/300 Write block 2468 len 16384 data: 4 6 7 6
20938 203/300 Read block 394 len 4096
20939 204/300 Read block 2602 len 32768
20940 205/300 Read block 447 len 28672
20941 206/300 Write block 997 len 36864 data: 4 4 5 8 7 7 6 7 6
20942 207/300 Write block 4698 len 20480 data: 6 5 5 6 5
20943 208/300 Read block 766 len 24576
20944 209/300 Read block 4000 len 24576
20945 210/300 Write block 101 len 20480 data: 7 10 10 11 10
20946 211/300 Write block 1069 len 16384 data: 4 6 8 8
20947 212/300 Write block 3388 len 20480 data: 4 4 5 5 5
20948 213/300 Read block 4538 len 16384
20949 214/300 Write block 4266 len 12288 data: 9 9 9
20950 215/300 Read block 1594 len 12288
20951 216/300 Read block 400 len 24576
20952 217/300 Read block 4509 len 4096
20953 218/300 Flush
20954 219/300 Flush
20955 220/300 Flush
20956 221/300 Write block 4835 len 8192 data: 5 5
20957 222/300 Read block 1664 len 40960
20958 223/300 Read block 1204 len 4096
20959 224/300 Write block 3105 len 20480 data: 7 5 5 3 3
20960 225/300 Write block 2528 len 12288 data: 6 4 4
20961 226/300 Read block 2342 len 28672
20962 227/300 Write block 4956 len 12288 data: 5 7 8
20963 228/300 Write block 2454 len 12288 data: 3 4 4
20964 229/300 Write block 2707 len 24576 data: 3 4 5 6 6 7
20965 230/300 Flush
20966 231/300 Read block 929 len 4096
20967 232/300 Read block 2963 len 8192
20968 233/300 Read block 1536 len 12288
20969 234/300 Write block 4480 len 24576 data: 6 8 11 11 8 8
20970 235/300 Flush
20971 236/300 Read block 3856 len 28672
20972 237/300 Write block 68 len 32768 data: 4 3 3 3 5 5 5 6
20973 238/300 Write block 1248 len 36864 data: 3 3 4 3 4 4 5 5 8
20974 239/300 Read block 590 len 28672
20975 240/300 Write block 4944 len 4096 data: 6
20976 241/300 Write block 4357 len 36864 data: 7 8 9 8 8 8 9 8 9
20977 242/300 Write block 834 len 16384 data: 3 3 3 2
20978 243/300 Write block 621 len 4096 data: 3
20979 244/300 Read block 4031 len 36864
20980 245/300 Read block 1924 len 40960
20981 246/300 Flush
20982 247/300 Write block 4193 len 16384 data: 6 5 4 3
20983 248/300 Flush
20984 249/300 Write block 3424 len 32768 data: 7 7 6 5 6 8 8 7
20985 250/300 Write block 4403 len 12288 data: 4 5 5
20986 251/300 Write block 3351 len 20480 data: 3 2 2 2 2
20987 252/300 Read block 4914 len 28672
20988 253/300 Write block 1353 len 24576 data: 4 5 4 3 3 5
20989 254/300 Write block 2967 len 28672 data: 3 3 4 5 5 4 7
20990 255/300 Read block 1388 len 20480
20991 256/300 Read block 1846 len 20480
20992 257/300 Write block 2468 len 36864 data: 5 7 8 7 8 8 7 6 6
20993 258/300 Read block 2433 len 28672
20994 259/300 Read block 3274 len 8192
20995 260/300 Read block 1184 len 24576
20996 261/300 Write block 2127 len 24576 data: 8 8 8 8 10 8
20997 262/300 Flush
20998 263/300 Read block 1434 len 16384
20999 264/300 Read block 3251 len 16384
21000 265/300 Write block 604 len 16384 data: 6 4 4 5
21001 266/300 Read block 1378 len 12288
21002 267/300 Write block 1124 len 24576 data: 9 7 8 8 8 6
21003 268/300 Read block 2507 len 28672
21004 269/300 Read block 1694 len 40960
21005 270/300 Write block 1578 len 4096 data: 5
21006 271/300 Read block 3905 len 32768
21007 272/300 Read block 1866 len 12288
21008 273/300 Read block 1960 len 20480
21009 274/300 Write block 1608 len 40960 data: 6 7 6 6 6 7 7 8 6 6
21010 275/300 Flush
21011 276/300 Read block 1029 len 8192
21012 277/300 Write block 4270 len 8192 data: 7 7
21013 278/300 Write block 4445 len 20480 data: 4 5 5 7 7
21014 279/300 Read block 4970 len 40960
21015 280/300 Read block 3685 len 28672
21016 281/300 Read block 4701 len 28672
21017 282/300 Read block 960 len 8192
21018 283/300 Flush
21019 284/300 Read block 2333 len 8192
21020 285/300 Write block 3337 len 20480 data: 5 5 6 5 5
21021 286/300 Read block 2859 len 40960
21022 287/300 Read block 993 len 4096
21023 288/300 Read block 1427 len 12288
21024 289/300 Write block 582 len 4096 data: 4
21025 290/300 Read block 465 len 8192
21026 291/300 Write block 3164 len 8192 data: 6 8
21027 292/300 Read block 2255 len 24576
21028 293/300 Flush
21029 294/300 Write block 310 len 20480 data: 5 4 5 5 5
21030 295/300 Write block 622 len 16384 data: 5 5 3 3
21031 296/300 Read block 2445 len 36864
21032 297/300 Read block 4801 len 4096
21033 298/300 Write block 4440 len 20480 data: 3 3 3 3 4
21034 299/300 Read block 3473 len 28672
21035 300/300 Read block 1512 len 32768
21036 Main task has work to do, go find it
21037 got dsc Start(0)
21038 start 0
21039 Feb 12 01:38:42.301 INFO request completed, latency_us: 272, response_code: 204, uri: /start/cid/0, method: POST, req_id: 34c05fb4-1173-4dbf-90da-063684263f23, remote_addr: 127.0.0.1:60456, local_addr: 127.0.0.1:9998
21040 Replay: started 0, returned:Ok(())
21041 ----------------------------------------------------------------
21042 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:300
21043 States: Offline Active Active
21044 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
21045 6935 Acked 7934 Write 3 New Done Done true
21046 6936 Acked 7935 Read 2 New Done Done true
21047 6937 Acked 7936 Read 5 New Done Done true
21048 6938 Acked 7937 Read 7 New Done Done true
21049 6939 Acked 7938 Read 6 New Done Done true
21050 6940 Acked 7939 Write 1 New Done Done false
21051 6941 Acked 7940 Read 2 New Done Done false
21052 6942 Acked 7941 Write 4 New Done Done false
21053 6943 Acked 7942 Write 8 New Done Done false
21054 6944 Acked 7943 Write 3 New Done Done false
21055 6945 Acked 7944 Read 10 New Done Done false
21056 6946 Acked 7945 Flush 0 New Done Done false
21057 6947 Acked 7946 Write 3 New Done Done false
21058 6948 Acked 7947 Read 8 New Done Done false
21059 6949 Acked 7948 Read 3 New Done Done false
21060 6950 Acked 7949 Read 8 New Done Done false
21061 6951 Acked 7950 Write 1 New Done Done false
21062 6952 Acked 7951 Write 4 New Done Done false
21063 6953 Acked 7952 Write 2 New Done Done false
21064 6954 Acked 7953 Flush 0 New Done Done false
21065 6955 Acked 7954 Read 10 New Done Done false
21066 6956 Acked 7955 Write 1 New Done Done false
21067 6957 Acked 7956 Read 8 New Done Done false
21068 6958 Acked 7957 Write 6 New Done Done false
21069 6959 Acked 7958 Write 4 New Done Done false
21070 6960 Acked 7959 Write 6 New Done Done false
21071 6961 Acked 7960 Write 5 New Done Done false
21072 6962 Acked 7961 Flush 0 New Done Done false
21073 6963 Acked 7962 Flush 0 New Done Done false
21074 6964 Acked 7963 Flush 0 New Done Done false
21075 6965 Acked 7964 Write 6 New Done Done false
21076 6966 Acked 7965 Read 5 New Done Done false
21077 6967 Acked 7966 Write 10 New Done Done false
21078 6968 Acked 7967 Read 10 New Done Done false
21079 6969 Acked 7968 Read 4 New Done Done false
21080 6970 Acked 7969 Read 8 New Done Done false
21081 6971 Acked 7970 Write 4 New Done Done false
21082 6972 Acked 7971 Flush 0 New Done Done false
21083 6973 Acked 7972 Read 1 New Done Done false
21084 6974 Acked 7973 Read 10 New Done Done false
21085 6975 Acked 7974 Write 3 New Done Done false
21086 6976 Acked 7975 Read 4 New Done Done false
21087 6977 Acked 7976 Read 10 New Done Done false
21088 6978 Acked 7977 Write 3 New Done Done false
21089 6979 Acked 7978 Flush 0 New Done Done false
21090 6980 Acked 7979 Write 7 New Done Done false
21091 6981 Acked 7980 Read 3 New Done Done false
21092 6982 Acked 7981 Read 5 New Done Done false
21093 6983 Acked 7982 Read 5 New Done Done false
21094 6984 Acked 7983 Read 6 New Done Done false
21095 6985 Acked 7984 Read 10 New Done Done false
21096 6986 Acked 7985 Read 1 New Done Done false
21097 6987 Acked 7986 Read 6 New Done Done false
21098 6988 Acked 7987 Read 1 New Done Done false
21099 6989 Acked 7988 Read 5 New Done Done false
21100 6990 Acked 7989 Read 9 New Done Done false
21101 6991 Acked 7990 Read 5 New Done Done false
21102 6992 Acked 7991 Write 8 New Done Done false
21103 6993 Acked 7992 Write 4 New Done Done false
21104 6994 Acked 7993 Write 2 New Done Done false
21105 6995 Acked 7994 Read 6 New Done Done false
21106 6996 Acked 7995 Write 5 New Done Done false
21107 6997 Acked 7996 Read 6 New Done Done false
21108 6998 Acked 7997 Read 10 New Done Done false
21109 6999 Acked 7998 Read 7 New Done Done false
21110 7000 Acked 7999 Flush 0 New Done Done false
21111 7001 Acked 8000 Flush 0 New Done Done false
21112 7002 Acked 8001 Write 1 New Done Done false
21113 7003 Acked 8002 Read 6 New Done Done false
21114 7004 Acked 8003 Read 7 New Done Done false
21115 7005 Acked 8004 Write 1 New Done Done false
21116 7006 Acked 8005 Write 3 New Done Done false
21117 7007 Acked 8006 Read 6 New Done Done false
21118 7008 Acked 8007 Write 10 New Done Done false
21119 7009 Acked 8008 Flush 0 New Done Done false
21120 7010 Acked 8009 Write 8 New Done Done false
21121 7011 Acked 8010 Read 2 New Done Done false
21122 7012 Acked 8011 Flush 0 New Done Done false
21123 7013 Acked 8012 Write 7 New Done Done false
21124 7014 Acked 8013 Read 3 New Done Done false
21125 7015 Acked 8014 Read 1 New Done Done false
21126 7016 Acked 8015 Read 7 New Done Done false
21127 7017 Acked 8016 Read 9 New Done Done false
21128 7018 Acked 8017 Read 1 New Done Done false
21129 7019 Acked 8018 Write 1 New Done Done false
21130 7020 Acked 8019 Write 1 New Done Done false
21131 7021 Acked 8020 Read 9 New Done Done false
21132 7022 Acked 8021 Write 2 New Done Done false
21133 7023 Acked 8022 Flush 0 New Done Done false
21134 7024 Acked 8023 Read 1 New Done Done false
21135 7025 Acked 8024 Write 5 New Done Done false
21136 7026 Acked 8025 Write 6 New Done Done false
21137 7027 Acked 8026 Read 5 New Done Done false
21138 7028 Acked 8027 Read 2 New Done Done false
21139 7029 Acked 8028 Read 10 New Done Done false
21140 7030 Acked 8029 Read 5 New Done Done false
21141 7031 Acked 8030 Read 3 New Done Done false
21142 7032 Acked 8031 Read 6 New Done Done false
21143 7033 Acked 8032 Read 2 New Done Done false
21144 7034 Acked 8033 Write 10 New Done Done false
21145 7035 Acked 8034 Write 8 New Done Done false
21146 7036 Acked 8035 Read 8 New Done Done false
21147 7037 Acked 8036 Write 4 New Done Done false
21148 7038 Acked 8037 Write 7 New Done Done false
21149 7039 Acked 8038 Read 6 New Done Done false
21150 7040 Acked 8039 Flush 0 New Done Done false
21151 7041 Acked 8040 Read 7 New Done Done false
21152 7042 Acked 8041 Read 3 New Done Done false
21153 7043 Acked 8042 Write 1 New Done Done false
21154 7044 Acked 8043 Write 4 New Done Done false
21155 7045 Acked 8044 Read 3 New Done Done false
21156 7046 Acked 8045 Read 7 New Done Done false
21157 7047 Acked 8046 Write 7 New Done Done false
21158 7048 Acked 8047 Read 3 New Done Done false
21159 7049 Acked 8048 Read 10 New Done Done false
21160 7050 Acked 8049 Read 2 New Done Done false
21161 7051 Acked 8050 Read 10 New Done Done false
21162 7052 Acked 8051 Read 1 New Done Done false
21163 7053 Acked 8052 Flush 0 New Done Done false
21164 7054 Acked 8053 Read 2 New Done Done false
21165 7055 Acked 8054 Read 5 New Done Done false
21166 7056 Acked 8055 Write 7 New Done Done false
21167 7057 Acked 8056 Write 1 New Done Done false
21168 7058 Acked 8057 Read 1 New Done Done false
21169 7059 Acked 8058 Read 10 New Done Done false
21170 7060 Acked 8059 Write 4 New Done Done false
21171 7061 Acked 8060 Write 3 New Done Done false
21172 7062 Acked 8061 Flush 0 New Done Done false
21173 7063 Acked 8062 Read 10 New Done Done false
21174 7064 Acked 8063 Write 4 New Done Done false
21175 7065 Acked 8064 Flush 0 New Done Done false
21176 7066 Acked 8065 Read 4 New Done Done false
21177 7067 Acked 8066 Write 4 New Done Done false
21178 7068 Acked 8067 Write 6 New Done Done false
21179 7069 Acked 8068 Write 8 New Done Done false
21180 7070 Acked 8069 Write 7 New Done Done false
21181 7071 Acked 8070 Write 4 New Done Done false
21182 7072 Acked 8071 Write 6 New Done Done false
21183 7073 Acked 8072 Read 3 New Done Done false
21184 7074 Acked 8073 Read 10 New Done Done false
21185 7075 Acked 8074 Write 8 New Done Done false
21186 7076 Acked 8075 Read 8 New Done Done false
21187 7077 Acked 8076 Read 3 New Done Done false
21188 7078 Acked 8077 Write 8 New Done Done false
21189 7079 Acked 8078 Write 1 New Done Done false
21190 7080 Acked 8079 Write 8 New Done Done false
21191 7081 Acked 8080 Write 1 New Done Done false
21192 7082 Acked 8081 Write 7 New Done Done false
21193 7083 Acked 8082 Flush 0 New Done Done false
21194 7084 Acked 8083 Write 8 New Done Done false
21195 7085 Acked 8084 Write 8 New Done Done false
21196 7086 Acked 8085 Write 3 New Done Done false
21197 7087 Acked 8086 Write 3 New Done Done false
21198 7088 Acked 8087 Flush 0 New Done Done false
21199 7089 Acked 8088 Read 10 New Done Done false
21200 7090 Acked 8089 Read 5 New Done Done false
21201 7091 Acked 8090 Read 1 New Done Done false
21202 7092 Acked 8091 Write 1 New Done Done false
21203 7093 Acked 8092 Write 8 New Done Done false
21204 7094 Acked 8093 Read 1 New Done Done false
21205 7095 Acked 8094 Write 8 New Done Done false
21206 7096 Acked 8095 Read 5 New Done Done false
21207 7097 Acked 8096 Flush 0 New Done Done false
21208 7098 Acked 8097 Write 4 New Done Done false
21209 7099 Acked 8098 Read 1 New Done Done false
21210 7100 Acked 8099 Flush 0 New Done Done false
21211 7101 Acked 8100 Read 1 New Done Done false
21212 7102 Acked 8101 Read 9 New Done Done false
21213 7103 Acked 8102 Write 7 New Done Done false
21214 7104 Acked 8103 Flush 0 New Done Done false
21215 7105 Acked 8104 Write 1 New Done Done false
21216 7106 Acked 8105 Write 6 New Done Done false
21217 7107 Acked 8106 Flush 0 New Done Done false
21218 7108 Acked 8107 Read 5 New Done Done false
21219 7109 Acked 8108 Write 7 New Done Done false
21220 7110 Acked 8109 Read 2 New Done Done false
21221 7111 Acked 8110 Read 9 New Done Done false
21222 7112 Acked 8111 Write 9 New Done Done false
21223 7113 Acked 8112 Read 6 New Done Done false
21224 7114 Acked 8113 Read 10 New Done Done false
21225 7115 Acked 8114 Write 9 New Done Done false
21226 7116 Acked 8115 Write 4 New Done Done false
21227 7117 Acked 8116 Read 2 New Done Done false
21228 7118 Acked 8117 Read 8 New Done Done false
21229 7119 Acked 8118 Write 4 New Done Done false
21230 7120 Acked 8119 Flush 0 New Done Done false
21231 7121 Acked 8120 Write 2 New Done Done false
21232 7122 Acked 8121 Read 1 New Done Done false
21233 7123 Acked 8122 Read 2 New Done Done false
21234 7124 Acked 8123 Write 7 New Done Done false
21235 7125 Acked 8124 Read 1 New Done Done false
21236 7126 Acked 8125 Flush 0 New Done Done false
21237 7127 Acked 8126 Read 8 New Done Done false
21238 7128 Acked 8127 Read 7 New Done Done false
21239 7129 Acked 8128 Read 4 New Done Done false
21240 7130 Acked 8129 Read 8 New Done Done false
21241 7131 Acked 8130 Write 2 New Done Done false
21242 7132 Acked 8131 Read 1 New Done Done false
21243 7133 Acked 8132 Read 6 New Done Done false
21244 7134 Acked 8133 Write 8 New Done Done false
21245 7135 Acked 8134 Flush 0 New Done Done false
21246 7136 Acked 8135 Write 4 New Done Done false
21247 7137 Acked 8136 Read 1 New Done Done false
21248 7138 Acked 8137 Read 8 New Done Done false
21249 7139 Acked 8138 Read 7 New Done Done false
21250 7140 Acked 8139 Write 9 New Done Done false
21251 7141 Acked 8140 Write 5 New Done Done false
21252 7142 Acked 8141 Read 6 New Done Done false
21253 7143 Acked 8142 Read 6 New Done Done false
21254 7144 Acked 8143 Write 5 New Done Done false
21255 7145 Acked 8144 Write 4 New Done Done false
21256 7146 Acked 8145 Write 5 New Done Done false
21257 7147 Acked 8146 Read 4 New Done Done false
21258 7148 Acked 8147 Write 3 New Done Done false
21259 7149 Acked 8148 Read 3 New Done Done false
21260 7150 Acked 8149 Read 6 New Done Done false
21261 7151 Acked 8150 Read 1 New Done Done false
21262 7152 Acked 8151 Flush 0 New Done Done false
21263 7153 Acked 8152 Flush 0 New Done Done false
21264 7154 Acked 8153 Flush 0 New Done Done false
21265 7155 Acked 8154 Write 2 New Done Done false
21266 7156 Acked 8155 Read 10 New Done Done false
21267 7157 Acked 8156 Read 1 New Done Done false
21268 7158 Acked 8157 Write 5 New Done Done false
21269 7159 Acked 8158 Write 3 New Done Done false
21270 7160 Acked 8159 Read 7 New Done Done false
21271 7161 Acked 8160 Write 3 New Done Done false
21272 7162 Acked 8161 Write 3 New Done Done false
21273 7163 Acked 8162 Write 6 New Done Done false
21274 7164 Acked 8163 Flush 0 New Done Done false
21275 7165 Acked 8164 Read 1 New Done Done false
21276 7166 Acked 8165 Read 2 New Done Done false
21277 7167 Acked 8166 Read 3 New Done Done false
21278 7168 Acked 8167 Write 6 New Done Done false
21279 7169 Acked 8168 Flush 0 New Done Done false
21280 7170 Acked 8169 Read 7 New Done Done false
21281 7171 Acked 8170 Write 8 New Done Done false
21282 7172 Acked 8171 Write 9 New Done Done false
21283 7173 Acked 8172 Read 7 New Done Done false
21284 7174 Acked 8173 Write 1 New Done Done false
21285 7175 Acked 8174 Write 9 New Done Done false
21286 7176 Acked 8175 Write 4 New Done Done false
21287 7177 Acked 8176 Write 1 New Done Done false
21288 7178 Acked 8177 Read 9 New Done Done false
21289 7179 Acked 8178 Read 10 New Done Done false
21290 7180 Acked 8179 Flush 0 New Done Done false
21291 7181 Acked 8180 Write 4 New Done Done false
21292 7182 Acked 8181 Flush 0 New Done Done false
21293 7183 Acked 8182 Write 8 New Done Done false
21294 7184 Acked 8183 Write 3 New Done Done false
21295 7185 Acked 8184 Write 5 New Done Done false
21296 7186 Acked 8185 Read 7 New Done Done false
21297 7187 Acked 8186 Write 6 New Done Done false
21298 7188 Acked 8187 Write 7 New Done Done false
21299 7189 Acked 8188 Read 5 New Done Done false
21300 7190 Acked 8189 Read 5 New Done Done false
21301 7191 Acked 8190 Write 9 New Done Done false
21302 7192 Acked 8191 Read 7 New Done Done false
21303 7193 Acked 8192 Read 2 New Done Done false
21304 7194 Acked 8193 Read 6 New Done Done false
21305 7195 Acked 8194 Write 6 New Done Done false
21306 7196 Acked 8195 Flush 0 New Done Done false
21307 7197 Acked 8196 Read 4 New Done Done false
21308 7198 Acked 8197 Read 4 New Done Done false
21309 7199 Acked 8198 Write 4 New Done Done false
21310 7200 Acked 8199 Read 3 New Done Done false
21311 7201 Acked 8200 Write 6 New Done Done false
21312 7202 Acked 8201 Read 7 New Done Done false
21313 7203 Acked 8202 Read 10 New Done Done false
21314 7204 Acked 8203 Write 1 New Done Done false
21315 7205 Acked 8204 Read 8 New Done Done false
21316 7206 Acked 8205 Read 3 New Done Done false
21317 7207 Acked 8206 Read 5 New Done Done false
21318 7208 Acked 8207 Write 10 New Done Done false
21319 7209 Acked 8208 Flush 0 New Done Done false
21320 7210 Acked 8209 Read 2 New Done Done false
21321 7211 Acked 8210 Write 2 New Done Done false
21322 7212 Acked 8211 Write 5 New Done Done false
21323 7213 Acked 8212 Read 10 New Done Done false
21324 7214 Acked 8213 Read 7 New Done Done false
21325 7215 Acked 8214 Read 7 New Done Done false
21326 7216 Acked 8215 Read 2 New Done Done false
21327 7217 Acked 8216 Flush 0 New Done Done false
21328 7218 Acked 8217 Read 2 New Done Done false
21329 7219 Acked 8218 Write 5 New Done Done false
21330 7220 Acked 8219 Read 10 New Done Done false
21331 7221 Acked 8220 Read 1 New Done Done false
21332 7222 Acked 8221 Read 3 New Done Done false
21333 7223 Acked 8222 Write 1 New Done Done false
21334 7224 Acked 8223 Read 2 New Done Done false
21335 7225 Acked 8224 Write 2 New Done Done false
21336 7226 Acked 8225 Read 6 New Done Done false
21337 7227 Acked 8226 Flush 0 New Done Done false
21338 7228 Acked 8227 Write 5 New Done Done false
21339 7229 Acked 8228 Write 4 New Done Done false
21340 7230 Acked 8229 Read 9 New Done Done false
21341 7231 Acked 8230 Read 1 New Done Done false
21342 7232 Acked 8231 Write 5 New Done Done false
21343 7233 Acked 8232 Read 7 New Done Done false
21344 7234 Acked 8233 Read 8 New Done Done false
21345 STATES DS:0 DS:1 DS:2 TOTAL
21346 New 300 0 0 300
21347 Sent 0 0 0 0
21348 Done 0 300 300 600
21349 Skipped 0 0 0 0
21350 Error 0 0 0 0
21351 Last Flush: 7933 8226 8226
21352 Downstairs last five completed: 7933 7932 7931 7930 7929
21353 Upstairs last five completed: 7234 7233 7232 7231 7230
21354 CLIENT: Up:0 ds:300 act:2
213552024-02-12T01:38:42.965ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
213562024-02-12T01:38:42.966ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
213572024-02-12T01:38:42.966ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
213582024-02-12T01:38:42.966ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
213592024-02-12T01:38:42.966ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
213602024-02-12T01:38:42.966ZINFOcrucible: [0] client re-new 301 jobs since flush 7933 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
213612024-02-12T01:38:42.966ZINFOcrucible: client task is exiting = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
21362 [8810] got start action
21363 [8810] I am going to restart
21364 Starting downstairs at port 8810
21365 Make output file at "/tmp/dsc/downstairs-8810.txt"
21366 [8810][0] reports Starting
213672024-02-12T01:38:43.967ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
213682024-02-12T01:38:43.967ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
213692024-02-12T01:38:43.967ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
213702024-02-12T01:38:43.967ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
213712024-02-12T01:38:43.967ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
213722024-02-12T01:38:43.967ZINFOcrucible: [0] client re-new 301 jobs since flush 7933 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
213732024-02-12T01:38:43.967ZINFOcrucible: client task is exiting = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
21374 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1188)
21375 [8810][0] reports Running
213762024-02-12T01:38:44.968ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
213772024-02-12T01:38:44.968ZINFOcrucible: ds_connection connected = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
213782024-02-12T01:38:44.970ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
213792024-02-12T01:38:44.970ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
213802024-02-12T01:38:44.970ZINFOcrucible: send last flush ID to this DS: 7933 = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
213812024-02-12T01:38:44.970ZINFOcrucible: Replied this last flush ID: 7933 = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
213822024-02-12T01:38:44.970ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
213832024-02-12T01:38:44.970ZINFOcrucible: [0] Transition from Offline to Replay = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
213842024-02-12T01:38:44.970ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
213852024-02-12T01:38:44.970ZINFOcrucible: [0] Transition from Replay to Active = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
21386 ----------------------------------------------------------------
21387 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
21388 Downstairs last five completed: 8234 8233 8232 8231 8230
21389 Upstairs last five completed: 7235 7234 7233 7232 7231
21390 CLIENT: Up:0 ds:0 act:3
21391 Replay: All jobs finished, all DS active.
21392 Main task has work to do, go find it
21393 got dsc Stop(2)
21394 stop 2
21395 [8830] Got stop action so:false kr:false
21396 Feb 12 01:38:46.304 INFO request completed, latency_us: 391, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 8813ce80-60af-413b-8d9b-5057ce2f9d13, remote_addr: 127.0.0.1:60456, local_addr: 127.0.0.1:9998
213972024-02-12T01:38:46.305ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
21398 Feb 12 01:38:46.305 INFO request completed, latency_us: 116, response_code: 200, uri: /state/cid/2, method: GET, req_id: ed5d94a2-6eae-475c-ae53-45dabf34d091, remote_addr: 127.0.0.1:60456, local_addr: 127.0.0.1:9998
213992024-02-12T01:38:46.305ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
214002024-02-12T01:38:46.305ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
214012024-02-12T01:38:46.305ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
214022024-02-12T01:38:46.305ZINFOcrucible: [2] client re-new 0 jobs since flush 8234 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
214032024-02-12T01:38:46.305ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
21404 [8830] Exited with: ExitStatus(unix_wait_status(9))
21405 [8830][2] reports Exit
214062024-02-12T01:38:47.307ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
214072024-02-12T01:38:47.307ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
214082024-02-12T01:38:47.307ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
214092024-02-12T01:38:47.307ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
214102024-02-12T01:38:47.307ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
214112024-02-12T01:38:47.307ZINFOcrucible: [2] client re-new 0 jobs since flush 8234 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
214122024-02-12T01:38:47.307ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
214132024-02-12T01:38:48.308ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
214142024-02-12T01:38:48.308ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
214152024-02-12T01:38:48.308ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
214162024-02-12T01:38:48.308ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
214172024-02-12T01:38:48.308ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
214182024-02-12T01:38:48.308ZINFOcrucible: [2] client re-new 0 jobs since flush 8234 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
214192024-02-12T01:38:48.308ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
214202024-02-12T01:38:49.309ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
214212024-02-12T01:38:49.309ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
214222024-02-12T01:38:49.309ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
214232024-02-12T01:38:49.309ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
214242024-02-12T01:38:49.309ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
214252024-02-12T01:38:49.310ZINFOcrucible: [2] client re-new 0 jobs since flush 8234 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
214262024-02-12T01:38:49.310ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
21427 Feb 12 01:38:50.308 INFO request completed, latency_us: 210, response_code: 200, uri: /state/cid/2, method: GET, req_id: c76e2dae-ab48-49e7-915f-0b5cd162c92e, remote_addr: 127.0.0.1:60456, local_addr: 127.0.0.1:9998
21428 001/300 Read block 1021 len 12288
21429 002/300 Read block 4600 len 16384
214302024-02-12T01:38:50.310ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
214312024-02-12T01:38:50.310ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
214322024-02-12T01:38:50.310ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
214332024-02-12T01:38:50.310ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
214342024-02-12T01:38:50.310ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
214352024-02-12T01:38:50.310ZINFOcrucible: [2] client re-new 2 jobs since flush 8234 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
214362024-02-12T01:38:50.310ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
21437 003/300 Read block 3416 len 36864
21438 004/300 Read block 821 len 4096
21439 005/300 Write block 305 len 32768 data: 5 6 5 5 5 6 5 6
21440 006/300 Read block 428 len 40960
21441 007/300 Write block 2068 len 24576 data: 7 6 6 7 6 5
21442 008/300 Read block 571 len 24576
21443 009/300 Write block 553 len 40960 data: 4 4 5 5 6 7 4 4 4 4
21444 010/300 Write block 2343 len 16384 data: 5 6 5 4
21445 011/300 Read block 3350 len 40960
21446 012/300 Read block 3208 len 28672
21447 013/300 Read block 1376 len 16384
21448 014/300 Read block 1695 len 24576
21449 015/300 Read block 464 len 24576
21450 016/300 Read block 2003 len 36864
21451 017/300 Read block 4970 len 20480
21452 018/300 Read block 3166 len 32768
21453 019/300 Write block 3768 len 40960 data: 5 5 5 4 4 4 6 7 8 7
21454 020/300 Read block 4694 len 16384
21455 021/300 Write block 2537 len 24576 data: 4 5 5 5 5 5
21456 022/300 Read block 1803 len 12288
21457 023/300 Read block 777 len 28672
21458 024/300 Flush
21459 025/300 Write block 882 len 16384 data: 6 7 8 6
21460 026/300 Read block 4858 len 32768
21461 027/300 Read block 2484 len 8192
21462 028/300 Read block 1164 len 20480
21463 029/300 Write block 2904 len 12288 data: 5 5 4
21464 030/300 Flush
21465 031/300 Write block 3711 len 28672 data: 2 4 3 4 5 6 5
21466 032/300 Read block 1419 len 40960
21467 033/300 Flush
21468 034/300 Flush
21469 035/300 Read block 2536 len 8192
21470 036/300 Write block 4644 len 40960 data: 7 6 5 4 5 4 4 5 5 5
21471 037/300 Write block 1641 len 36864 data: 5 4 5 5 6 4 4 5 4
21472 038/300 Read block 3477 len 36864
21473 039/300 Read block 1167 len 32768
21474 040/300 Read block 601 len 36864
21475 041/300 Read block 1567 len 28672
21476 042/300 Write block 2186 len 4096 data: 3
21477 043/300 Read block 4159 len 24576
21478 044/300 Write block 2220 len 20480 data: 5 5 4 3 2
21479 045/300 Flush
21480 046/300 Write block 4421 len 40960 data: 5 5 4 4 5 4 4 3 2 2
21481 047/300 Read block 1148 len 36864
21482 048/300 Read block 4418 len 16384
21483 049/300 Read block 31 len 16384
21484 050/300 Flush
21485 051/300 Read block 2626 len 24576
21486 052/300 Write block 2161 len 24576 data: 5 4 4 4 5 5
21487 053/300 Write block 258 len 32768 data: 6 7 6 4 4 5 6 6
21488 054/300 Read block 4180 len 28672
21489 055/300 Write block 3711 len 28672 data: 3 5 4 5 6 7 6
21490 056/300 Write block 1260 len 12288 data: 9 9 7
21491 057/300 Read block 2999 len 40960
21492 058/300 Write block 884 len 40960 data: 9 7 6 5 7 3 3 3 4 5
21493 059/300 Read block 3087 len 4096
21494 060/300 Read block 4501 len 12288
21495 061/300 Read block 1898 len 40960
21496 062/300 Write block 2368 len 40960 data: 3 3 4 4 5 4 4 4 7 7
21497 063/300 Read block 1601 len 8192
21498 064/300 Read block 4115 len 16384
21499 065/300 Read block 984 len 20480
21500 066/300 Write block 1460 len 28672 data: 5 5 5 6 8 7 7
21501 067/300 Read block 1214 len 36864
21502 068/300 Read block 1583 len 36864
21503 069/300 Read block 1934 len 32768
21504 070/300 Write block 4473 len 24576 data: 7 8 9 9 7 6
21505 071/300 Write block 318 len 24576 data: 3 3 4 4 5 4
21506 072/300 Write block 901 len 16384 data: 4 5 2 2
21507 073/300 Write block 757 len 4096 data: 3
21508 074/300 Read block 104 len 36864
21509 075/300 Read block 3353 len 4096
21510 076/300 Read block 2440 len 12288
21511 077/300 Read block 1016 len 4096
21512 078/300 Read block 4206 len 28672
21513 079/300 Read block 3858 len 4096
21514 080/300 Read block 3176 len 16384
21515 081/300 Read block 166 len 28672
21516 082/300 Write block 4135 len 20480 data: 6 6 6 7 9
21517 083/300 Read block 830 len 36864
21518 084/300 Read block 4771 len 28672
21519 085/300 Read block 4300 len 4096
21520 086/300 Read block 307 len 40960
21521 087/300 Read block 2503 len 28672
21522 088/300 Flush
21523 089/300 Read block 4365 len 16384
21524 090/300 Write block 1524 len 12288 data: 3 2 2
21525 091/300 Read block 3862 len 12288
21526 092/300 Read block 2966 len 32768
21527 093/300 Flush
21528 094/300 Write block 2362 len 36864 data: 5 4 6 6 5 3 4 4 5
21529 095/300 Read block 329 len 4096
21530 096/300 Write block 2280 len 12288 data: 7 6 6
21531 097/300 Read block 3597 len 32768
21532 098/300 Read block 4703 len 24576
21533 099/300 Write block 2836 len 8192 data: 6 8
21534 100/300 Read block 4861 len 28672
21535 101/300 Read block 978 len 36864
21536 102/300 Write block 4880 len 16384 data: 4 6 6 4
21537 103/300 Read block 3879 len 40960
21538 104/300 Read block 1722 len 12288
21539 105/300 Write block 2529 len 24576 data: 5 5 5 3 3 3
21540 106/300 Read block 1118 len 32768
21541 107/300 Read block 3665 len 36864
21542 108/300 Flush
21543 109/300 Read block 1224 len 24576
21544 110/300 Read block 4887 len 20480
21545 111/300 Write block 1887 len 24576 data: 5 7 6 6 5 5
21546 112/300 Write block 4950 len 4096 data: 6
21547 113/300 Write block 2943 len 36864 data: 3 2 2 3 4 4 5 4 2
21548 114/300 Write block 2219 len 28672 data: 7 6 6 5 4 3 3
21549 115/300 Read block 1212 len 40960
21550 116/300 Write block 3984 len 8192 data: 3 4
21551 117/300 Read block 1243 len 8192
21552 118/300 Read block 2919 len 40960
21553 119/300 Flush
21554 120/300 Flush
21555 121/300 Write block 897 len 32768 data: 4 4 5 5 5 6 3 3
21556 122/300 Write block 1961 len 20480 data: 2 3 3 3 5
21557 123/300 Read block 2722 len 12288
21558 124/300 Write block 949 len 16384 data: 5 6 7 7
21559 125/300 Read block 4158 len 16384
21560 126/300 Read block 3026 len 28672
21561 127/300 Write block 2912 len 36864 data: 3 2 2 2 2 3 4 4 4
21562 128/300 Flush
21563 129/300 Flush
21564 130/300 Write block 4243 len 4096 data: 6
21565 131/300 Read block 601 len 28672
21566 132/300 Write block 3606 len 40960 data: 7 7 5 4 4 4 3 3 4 6
21567 133/300 Write block 2492 len 4096 data: 4
21568 134/300 Read block 3544 len 28672
21569 135/300 Read block 3725 len 32768
21570 136/300 Read block 370 len 32768
21571 137/300 Read block 3584 len 40960
21572 138/300 Read block 2083 len 32768
21573 139/300 Read block 4690 len 16384
21574 140/300 Read block 1144 len 12288
21575 141/300 Read block 2253 len 20480
21576 142/300 Flush
21577 143/300 Write block 3698 len 4096 data: 8
21578 144/300 Read block 3994 len 12288
21579 145/300 Read block 750 len 4096
21580 146/300 Write block 4912 len 32768 data: 6 5 6 7 7 5 6 7
21581 147/300 Write block 1137 len 32768 data: 5 5 5 7 6 8 7 8
21582 148/300 Flush
21583 149/300 Read block 532 len 16384
21584 150/300 Read block 2065 len 20480
21585 151/300 Read block 2305 len 20480
21586 152/300 Write block 4986 len 40960 data: 2 2 4 4 4 4 5 5 4 3
21587 153/300 Flush
21588 154/300 Read block 4967 len 12288
21589 155/300 Flush
21590 156/300 Write block 2779 len 24576 data: 6 5 5 5 5 4
21591 157/300 Read block 4555 len 40960
21592 158/300 Flush
21593 159/300 Read block 73 len 28672
21594 160/300 Write block 3635 len 4096 data: 3
21595 161/300 Read block 1793 len 20480
21596 162/300 Write block 4656 len 36864 data: 4 4 3 3 3 3 3 3 4
21597 163/300 Write block 1804 len 16384 data: 6 6 6 7
21598 164/300 Flush
21599 165/300 Read block 4699 len 8192
21600 166/300 Read block 1529 len 32768
21601 167/300 Read block 635 len 28672
21602 168/300 Write block 2734 len 4096 data: 3
21603 169/300 Write block 3957 len 24576 data: 7 5 6 5 5 6
21604 170/300 Write block 3229 len 4096 data: 10
21605 171/300 Write block 800 len 36864 data: 9 8 10 10 10 9 9 8 7
21606 172/300 Read block 1639 len 20480
21607 173/300 Write block 4446 len 40960 data: 6 6 8 8 9 8 9 9 10 9
21608 174/300 Write block 2768 len 12288 data: 4 5 5
21609 175/300 Read block 717 len 16384
21610 176/300 Write block 4564 len 8192 data: 2 2
21611 177/300 Read block 4031 len 20480
21612 178/300 Read block 4977 len 4096
21613 179/300 Read block 722 len 24576
21614 180/300 Write block 2488 len 32768 data: 6 7 7 6 5 4 3 3
21615 181/300 Read block 3648 len 36864
21616 182/300 Write block 4611 len 40960 data: 7 6 7 7 7 6 5 4 4 5
21617 183/300 Write block 4217 len 40960 data: 5 7 5 4 7 8 8 8 8 7
21618 184/300 Read block 138 len 20480
21619 185/300 Read block 2330 len 28672
21620 186/300 Write block 2162 len 20480 data: 5 5 5 6 6
21621 187/300 Read block 32 len 24576
21622 188/300 Write block 1702 len 36864 data: 7 7 7 8 6 6 7 5 6
21623 189/300 Read block 3234 len 28672
21624 190/300 Write block 4384 len 24576 data: 6 6 6 6 5 7
21625 191/300 Write block 742 len 40960 data: 6 6 6 8 7 8 8 8 7 5
21626 192/300 Read block 4034 len 24576
21627 193/300 Flush
21628 194/300 Read block 3167 len 12288
21629 195/300 Write block 1252 len 12288 data: 5 5 6
21630 196/300 Read block 3488 len 12288
21631 197/300 Read block 4667 len 40960
21632 198/300 Read block 2931 len 40960
21633 199/300 Read block 3440 len 36864
21634 200/300 Write block 2320 len 36864 data: 6 6 4 5 5 4 3 2 3
21635 201/300 Write block 3525 len 36864 data: 6 6 6 6 4 5 5 4 4
21636 202/300 Read block 4156 len 36864
21637 203/300 Read block 1898 len 28672
21638 204/300 Write block 2173 len 24576 data: 9 9 12 10 10 8
21639 205/300 Write block 4848 len 24576 data: 3 3 2 3 4 4
21640 206/300 Read block 3377 len 12288
21641 207/300 Read block 728 len 16384
21642 208/300 Read block 1552 len 40960
21643 209/300 Write block 839 len 40960 data: 2 2 3 4 5 5 5 5 5 4
21644 210/300 Write block 3389 len 36864 data: 5 6 6 6 4 7 8 8 8
21645 211/300 Write block 4991 len 8192 data: 5 6
21646 212/300 Write block 1840 len 12288 data: 6 6 7
21647 213/300 Read block 535 len 40960
21648 214/300 Write block 3986 len 36864 data: 4 4 4 3 3 2 4 4 3
21649 215/300 Write block 2847 len 24576 data: 3 2 3 5 9 7
21650 216/300 Read block 4444 len 28672
21651 217/300 Write block 1653 len 32768 data: 4 3 3 3 3 2 2 3
21652 218/300 Read block 1013 len 4096
21653 219/300 Read block 1220 len 12288
21654 220/300 Write block 1786 len 4096 data: 4
21655 221/300 Write block 4212 len 16384 data: 4 4 3 3
21656 222/300 Read block 1168 len 8192
21657 223/300 Read block 2125 len 24576
21658 224/300 Read block 3881 len 36864
21659 225/300 Flush
21660 226/300 Write block 4218 len 16384 data: 8 6 5 8
21661 227/300 Read block 463 len 12288
21662 228/300 Read block 3403 len 4096
21663 229/300 Write block 31 len 36864 data: 4 3 2 4 4 5 5 5 4
21664 230/300 Read block 4778 len 28672
21665 231/300 Write block 2293 len 24576 data: 6 6 6 4 5 5
21666 232/300 Write block 4692 len 8192 data: 7 7
21667 233/300 Read block 1309 len 36864
21668 234/300 Write block 679 len 16384 data: 6 6 6 4
21669 235/300 Write block 462 len 8192 data: 6 6
21670 236/300 Read block 2749 len 32768
21671 237/300 Write block 4452 len 8192 data: 10 10
21672 238/300 Read block 4265 len 12288
21673 239/300 Write block 1330 len 4096 data: 8
21674 240/300 Write block 3298 len 24576 data: 3 4 4 3 3 2
21675 241/300 Read block 4504 len 40960
21676 242/300 Read block 1254 len 12288
21677 243/300 Write block 3035 len 16384 data: 5 6 7 8
21678 244/300 Read block 3805 len 4096
21679 245/300 Read block 3377 len 16384
21680 246/300 Write block 2672 len 36864 data: 2 2 2 2 2 2 2 2 2
21681 247/300 Write block 2725 len 16384 data: 5 6 5 5
21682 248/300 Read block 1731 len 20480
21683 249/300 Read block 4327 len 4096
21684 250/300 Read block 1974 len 12288
21685 251/300 Write block 2050 len 8192 data: 5 4
21686 252/300 Write block 2084 len 28672 data: 3 3 2 3 4 4 6
21687 253/300 Write block 1707 len 8192 data: 7 8
21688 254/300 Read block 1560 len 36864
21689 255/300 Read block 3257 len 12288
21690 256/300 Read block 1582 len 32768
21691 257/300 Read block 3639 len 36864
21692 258/300 Write block 417 len 40960 data: 5 5 6 4 6 4 2 2 2 2
21693 259/300 Write block 1057 len 16384 data: 3 4 4 4
21694 260/300 Flush
21695 261/300 Read block 789 len 24576
21696 262/300 Read block 2378 len 4096
21697 263/300 Read block 1991 len 40960
21698 264/300 Write block 2328 len 36864 data: 4 5 5 5 6 7 7 8 8
21699 265/300 Read block 1003 len 16384
21700 266/300 Write block 245 len 16384 data: 6 5 6 6
21701 267/300 Read block 2640 len 20480
21702 268/300 Write block 145 len 4096 data: 3
21703 269/300 Write block 2097 len 32768 data: 4 4 6 5 5 4 3 5
21704 270/300 Write block 4515 len 40960 data: 4 4 4 4 4 4 3 3 3 2
21705 271/300 Read block 4983 len 16384
21706 272/300 Flush
21707 273/300 Read block 2078 len 24576
21708 274/300 Write block 1621 len 24576 data: 6 7 7 5 5 5
21709 275/300 Write block 211 len 4096 data: 4
21710 276/300 Read block 4953 len 24576
21711 277/300 Read block 3964 len 36864
21712 278/300 Write block 3554 len 4096 data: 3
21713 279/300 Read block 1427 len 28672
21714 280/300 Read block 4095 len 16384
21715 281/300 Write block 4533 len 20480 data: 7 7 6 7 9
21716 282/300 Read block 3098 len 12288
21717 283/300 Write block 2808 len 20480 data: 9 11 10 12 11
21718 284/300 Read block 1801 len 32768
21719 285/300 Write block 28 len 24576 data: 6 5 5 5 4 3
21720 286/300 Read block 2446 len 32768
21721 287/300 Write block 3484 len 20480 data: 7 8 9 9 8
21722 288/300 Write block 1339 len 32768 data: 4 7 5 4 4 4 3 2
21723 289/300 Write block 2505 len 4096 data: 4
21724 290/300 Write block 749 len 40960 data: 9 8 6 6 7 5 5 4 4 3
21725 291/300 Read block 2788 len 36864
21726 292/300 Read block 1643 len 40960
21727 293/300 Read block 4010 len 28672
21728 294/300 Read block 3591 len 28672
21729 295/300 Write block 3991 len 4096 data: 3
21730 296/300 Read block 2920 len 28672
21731 297/300 Write block 3006 len 16384 data: 5 5 4 6
21732 298/300 Read block 2128 len 36864
21733 299/300 Flush
21734 300/300 Read block 2281 len 40960
21735 Main task has work to do, go find it
21736 got dsc Start(2)
21737 start 2
21738 Feb 12 01:38:50.675 INFO request completed, latency_us: 236, response_code: 204, uri: /start/cid/2, method: POST, req_id: 043f0295-a5b4-4969-b8e1-400b8919c6fe, remote_addr: 127.0.0.1:60456, local_addr: 127.0.0.1:9998
21739 Replay: started 2, returned:Ok(())
21740 ----------------------------------------------------------------
21741 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:300
21742 States: Active Active Offline
21743 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
21744 7236 Acked 8235 Read 3 Done Done New true
21745 7237 Acked 8236 Read 4 Done Done New true
21746 7238 Acked 8237 Read 9 Done Done New false
21747 7239 Acked 8238 Read 1 Done Done New false
21748 7240 Acked 8239 Write 8 Done Done New false
21749 7241 Acked 8240 Read 10 Done Done New false
21750 7242 Acked 8241 Write 6 Done Done New false
21751 7243 Acked 8242 Read 6 Done Done New false
21752 7244 Acked 8243 Write 10 Done Done New false
21753 7245 Acked 8244 Write 4 Done Done New false
21754 7246 Acked 8245 Read 10 Done Done New false
21755 7247 Acked 8246 Read 7 Done Done New false
21756 7248 Acked 8247 Read 4 Done Done New false
21757 7249 Acked 8248 Read 6 Done Done New false
21758 7250 Acked 8249 Read 6 Done Done New false
21759 7251 Acked 8250 Read 9 Done Done New false
21760 7252 Acked 8251 Read 5 Done Done New false
21761 7253 Acked 8252 Read 8 Done Done New false
21762 7254 Acked 8253 Write 10 Done Done New false
21763 7255 Acked 8254 Read 4 Done Done New false
21764 7256 Acked 8255 Write 6 Done Done New false
21765 7257 Acked 8256 Read 3 Done Done New false
21766 7258 Acked 8257 Read 7 Done Done New false
21767 7259 Acked 8258 Flush 0 Done Done New false
21768 7260 Acked 8259 Write 4 Done Done New false
21769 7261 Acked 8260 Read 8 Done Done New false
21770 7262 Acked 8261 Read 2 Done Done New false
21771 7263 Acked 8262 Read 5 Done Done New false
21772 7264 Acked 8263 Write 3 Done Done New false
21773 7265 Acked 8264 Flush 0 Done Done New false
21774 7266 Acked 8265 Write 7 Done Done New false
21775 7267 Acked 8266 Read 10 Done Done New false
21776 7268 Acked 8267 Flush 0 Done Done New false
21777 7269 Acked 8268 Flush 0 Done Done New false
21778 7270 Acked 8269 Read 2 Done Done New false
21779 7271 Acked 8270 Write 10 Done Done New false
21780 7272 Acked 8271 Write 9 Done Done New false
21781 7273 Acked 8272 Read 9 Done Done New false
21782 7274 Acked 8273 Read 8 Done Done New false
21783 7275 Acked 8274 Read 9 Done Done New false
21784 7276 Acked 8275 Read 7 Done Done New false
21785 7277 Acked 8276 Write 1 Done Done New false
21786 7278 Acked 8277 Read 6 Done Done New false
21787 7279 Acked 8278 Write 5 Done Done New false
21788 7280 Acked 8279 Flush 0 Done Done New false
21789 7281 Acked 8280 Write 10 Done Done New false
21790 7282 Acked 8281 Read 9 Done Done New false
21791 7283 Acked 8282 Read 4 Done Done New false
21792 7284 Acked 8283 Read 4 Done Done New false
21793 7285 Acked 8284 Flush 0 Done Done New false
21794 7286 Acked 8285 Read 6 Done Done New false
21795 7287 Acked 8286 Write 6 Done Done New false
21796 7288 Acked 8287 Write 8 Done Done New false
21797 7289 Acked 8288 Read 7 Done Done New false
21798 7290 Acked 8289 Write 7 Done Done New false
21799 7291 Acked 8290 Write 3 Done Done New false
21800 7292 Acked 8291 Read 10 Done Done New false
21801 7293 Acked 8292 Write 10 Done Done New false
21802 7294 Acked 8293 Read 1 Done Done New false
21803 7295 Acked 8294 Read 3 Done Done New false
21804 7296 Acked 8295 Read 10 Done Done New false
21805 7297 Acked 8296 Write 10 Done Done New false
21806 7298 Acked 8297 Read 2 Done Done New false
21807 7299 Acked 8298 Read 4 Done Done New false
21808 7300 Acked 8299 Read 5 Done Done New false
21809 7301 Acked 8300 Write 7 Done Done New false
21810 7302 Acked 8301 Read 9 Done Done New false
21811 7303 Acked 8302 Read 9 Done Done New false
21812 7304 Acked 8303 Read 8 Done Done New false
21813 7305 Acked 8304 Write 6 Done Done New false
21814 7306 Acked 8305 Write 6 Done Done New false
21815 7307 Acked 8306 Write 4 Done Done New false
21816 7308 Acked 8307 Write 1 Done Done New false
21817 7309 Acked 8308 Read 9 Done Done New false
21818 7310 Acked 8309 Read 1 Done Done New false
21819 7311 Acked 8310 Read 3 Done Done New false
21820 7312 Acked 8311 Read 1 Done Done New false
21821 7313 Acked 8312 Read 7 Done Done New false
21822 7314 Acked 8313 Read 1 Done Done New false
21823 7315 Acked 8314 Read 4 Done Done New false
21824 7316 Acked 8315 Read 7 Done Done New false
21825 7317 Acked 8316 Write 5 Done Done New false
21826 7318 Acked 8317 Read 9 Done Done New false
21827 7319 Acked 8318 Read 7 Done Done New false
21828 7320 Acked 8319 Read 1 Done Done New false
21829 7321 Acked 8320 Read 10 Done Done New false
21830 7322 Acked 8321 Read 7 Done Done New false
21831 7323 Acked 8322 Flush 0 Done Done New false
21832 7324 Acked 8323 Read 4 Done Done New false
21833 7325 Acked 8324 Write 3 Done Done New false
21834 7326 Acked 8325 Read 3 Done Done New false
21835 7327 Acked 8326 Read 8 Done Done New false
21836 7328 Acked 8327 Flush 0 Done Done New false
21837 7329 Acked 8328 Write 9 Done Done New false
21838 7330 Acked 8329 Read 1 Done Done New false
21839 7331 Acked 8330 Write 3 Done Done New false
21840 7332 Acked 8331 Read 8 Done Done New false
21841 7333 Acked 8332 Read 6 Done Done New false
21842 7334 Acked 8333 Write 2 Done Done New false
21843 7335 Acked 8334 Read 7 Done Done New false
21844 7336 Acked 8335 Read 9 Done Done New false
21845 7337 Acked 8336 Write 4 Done Done New false
21846 7338 Acked 8337 Read 10 Done Done New false
21847 7339 Acked 8338 Read 3 Done Done New false
21848 7340 Acked 8339 Write 6 Done Done New false
21849 7341 Acked 8340 Read 8 Done Done New false
21850 7342 Acked 8341 Read 9 Done Done New false
21851 7343 Acked 8342 Flush 0 Done Done New false
21852 7344 Acked 8343 Read 6 Done Done New false
21853 7345 Acked 8344 Read 5 Done Done New false
21854 7346 Acked 8345 Write 6 Done Done New false
21855 7347 Acked 8346 Write 1 Done Done New false
21856 7348 Acked 8347 Write 9 Done Done New false
21857 7349 Acked 8348 Write 7 Done Done New false
21858 7350 Acked 8349 Read 10 Done Done New false
21859 7351 Acked 8350 Write 2 Done Done New false
21860 7352 Acked 8351 Read 2 Done Done New false
21861 7353 Acked 8352 Read 10 Done Done New false
21862 7354 Acked 8353 Flush 0 Done Done New false
21863 7355 Acked 8354 Flush 0 Done Done New false
21864 7356 Acked 8355 Write 8 Done Done New false
21865 7357 Acked 8356 Write 5 Done Done New false
21866 7358 Acked 8357 Read 3 Done Done New false
21867 7359 Acked 8358 Write 4 Done Done New false
21868 7360 Acked 8359 Read 4 Done Done New false
21869 7361 Acked 8360 Read 7 Done Done New false
21870 7362 Acked 8361 Write 9 Done Done New false
21871 7363 Acked 8362 Flush 0 Done Done New false
21872 7364 Acked 8363 Flush 0 Done Done New false
21873 7365 Acked 8364 Write 1 Done Done New false
21874 7366 Acked 8365 Read 7 Done Done New false
21875 7367 Acked 8366 Write 10 Done Done New false
21876 7368 Acked 8367 Write 1 Done Done New false
21877 7369 Acked 8368 Read 7 Done Done New false
21878 7370 Acked 8369 Read 8 Done Done New false
21879 7371 Acked 8370 Read 8 Done Done New false
21880 7372 Acked 8371 Read 10 Done Done New false
21881 7373 Acked 8372 Read 8 Done Done New false
21882 7374 Acked 8373 Read 4 Done Done New false
21883 7375 Acked 8374 Read 3 Done Done New false
21884 7376 Acked 8375 Read 5 Done Done New false
21885 7377 Acked 8376 Flush 0 Done Done New false
21886 7378 Acked 8377 Write 1 Done Done New false
21887 7379 Acked 8378 Read 3 Done Done New false
21888 7380 Acked 8379 Read 1 Done Done New false
21889 7381 Acked 8380 Write 8 Done Done New false
21890 7382 Acked 8381 Write 8 Done Done New false
21891 7383 Acked 8382 Flush 0 Done Done New false
21892 7384 Acked 8383 Read 4 Done Done New false
21893 7385 Acked 8384 Read 5 Done Done New false
21894 7386 Acked 8385 Read 5 Done Done New false
21895 7387 Acked 8386 Write 10 Done Done New false
21896 7388 Acked 8387 Flush 0 Done Done New false
21897 7389 Acked 8388 Read 3 Done Done New false
21898 7390 Acked 8389 Flush 0 Done Done New false
21899 7391 Acked 8390 Write 6 Done Done New false
21900 7392 Acked 8391 Read 10 Done Done New false
21901 7393 Acked 8392 Flush 0 Done Done New false
21902 7394 Acked 8393 Read 7 Done Done New false
21903 7395 Acked 8394 Write 1 Done Done New false
21904 7396 Acked 8395 Read 5 Done Done New false
21905 7397 Acked 8396 Write 9 Done Done New false
21906 7398 Acked 8397 Write 4 Done Done New false
21907 7399 Acked 8398 Flush 0 Done Done New false
21908 7400 Acked 8399 Read 2 Done Done New false
21909 7401 Acked 8400 Read 8 Done Done New false
21910 7402 Acked 8401 Read 7 Done Done New false
21911 7403 Acked 8402 Write 1 Done Done New false
21912 7404 Acked 8403 Write 6 Done Done New false
21913 7405 Acked 8404 Write 1 Done Done New false
21914 7406 Acked 8405 Write 9 Done Done New false
21915 7407 Acked 8406 Read 5 Done Done New false
21916 7408 Acked 8407 Write 10 Done Done New false
21917 7409 Acked 8408 Write 3 Done Done New false
21918 7410 Acked 8409 Read 4 Done Done New false
21919 7411 Acked 8410 Write 2 Done Done New false
21920 7412 Acked 8411 Read 5 Done Done New false
21921 7413 Acked 8412 Read 1 Done Done New false
21922 7414 Acked 8413 Read 6 Done Done New false
21923 7415 Acked 8414 Write 8 Done Done New false
21924 7416 Acked 8415 Read 9 Done Done New false
21925 7417 Acked 8416 Write 10 Done Done New false
21926 7418 Acked 8417 Write 10 Done Done New false
21927 7419 Acked 8418 Read 5 Done Done New false
21928 7420 Acked 8419 Read 7 Done Done New false
21929 7421 Acked 8420 Write 5 Done Done New false
21930 7422 Acked 8421 Read 6 Done Done New false
21931 7423 Acked 8422 Write 9 Done Done New false
21932 7424 Acked 8423 Read 7 Done Done New false
21933 7425 Acked 8424 Write 6 Done Done New false
21934 7426 Acked 8425 Write 10 Done Done New false
21935 7427 Acked 8426 Read 6 Done Done New false
21936 7428 Acked 8427 Flush 0 Done Done New false
21937 7429 Acked 8428 Read 3 Done Done New false
21938 7430 Acked 8429 Write 3 Done Done New false
21939 7431 Acked 8430 Read 3 Done Done New false
21940 7432 Acked 8431 Read 10 Done Done New false
21941 7433 Acked 8432 Read 10 Done Done New false
21942 7434 Acked 8433 Read 9 Done Done New false
21943 7435 Acked 8434 Write 9 Done Done New false
21944 7436 Acked 8435 Write 9 Done Done New false
21945 7437 Acked 8436 Read 9 Done Done New false
21946 7438 Acked 8437 Read 7 Done Done New false
21947 7439 Acked 8438 Write 6 Done Done New false
21948 7440 Acked 8439 Write 6 Done Done New false
21949 7441 Acked 8440 Read 3 Done Done New false
21950 7442 Acked 8441 Read 4 Done Done New false
21951 7443 Acked 8442 Read 10 Done Done New false
21952 7444 Acked 8443 Write 10 Done Done New false
21953 7445 Acked 8444 Write 9 Done Done New false
21954 7446 Acked 8445 Write 2 Done Done New false
21955 7447 Acked 8446 Write 3 Done Done New false
21956 7448 Acked 8447 Read 10 Done Done New false
21957 7449 Acked 8448 Write 9 Done Done New false
21958 7450 Acked 8449 Write 6 Done Done New false
21959 7451 Acked 8450 Read 7 Done Done New false
21960 7452 Acked 8451 Write 8 Done Done New false
21961 7453 Acked 8452 Read 1 Done Done New false
21962 7454 Acked 8453 Read 3 Done Done New false
21963 7455 Acked 8454 Write 1 Done Done New false
21964 7456 Acked 8455 Write 4 Done Done New false
21965 7457 Acked 8456 Read 2 Done Done New false
21966 7458 Acked 8457 Read 6 Done Done New false
21967 7459 Acked 8458 Read 9 Done Done New false
21968 7460 Acked 8459 Flush 0 Done Done New false
21969 7461 Acked 8460 Write 4 Done Done New false
21970 7462 Acked 8461 Read 3 Done Done New false
21971 7463 Acked 8462 Read 1 Done Done New false
21972 7464 Acked 8463 Write 9 Done Done New false
21973 7465 Acked 8464 Read 7 Done Done New false
21974 7466 Acked 8465 Write 6 Done Done New false
21975 7467 Acked 8466 Write 2 Done Done New false
21976 7468 Acked 8467 Read 9 Done Done New false
21977 7469 Acked 8468 Write 4 Done Done New false
21978 7470 Acked 8469 Write 2 Done Done New false
21979 7471 Acked 8470 Read 8 Done Done New false
21980 7472 Acked 8471 Write 2 Done Done New false
21981 7473 Acked 8472 Read 3 Done Done New false
21982 7474 Acked 8473 Write 1 Done Done New false
21983 7475 Acked 8474 Write 6 Done Done New false
21984 7476 Acked 8475 Read 10 Done Done New false
21985 7477 Acked 8476 Read 3 Done Done New false
21986 7478 Acked 8477 Write 4 Done Done New false
21987 7479 Acked 8478 Read 1 Done Done New false
21988 7480 Acked 8479 Read 4 Done Done New false
21989 7481 Acked 8480 Write 9 Done Done New false
21990 7482 Acked 8481 Write 4 Done Done New false
21991 7483 Acked 8482 Read 5 Done Done New false
21992 7484 Acked 8483 Read 1 Done Done New false
21993 7485 Acked 8484 Read 3 Done Done New false
21994 7486 Acked 8485 Write 2 Done Done New false
21995 7487 Acked 8486 Write 7 Done Done New false
21996 7488 Acked 8487 Write 2 Done Done New false
21997 7489 Acked 8488 Read 9 Done Done New false
21998 7490 Acked 8489 Read 3 Done Done New false
21999 7491 Acked 8490 Read 8 Done Done New false
22000 7492 Acked 8491 Read 9 Done Done New false
22001 7493 Acked 8492 Write 10 Done Done New false
22002 7494 Acked 8493 Write 4 Done Done New false
22003 7495 Acked 8494 Flush 0 Done Done New false
22004 7496 Acked 8495 Read 6 Done Done New false
22005 7497 Acked 8496 Read 1 Done Done New false
22006 7498 Acked 8497 Read 10 Done Done New false
22007 7499 Acked 8498 Write 9 Done Done New false
22008 7500 Acked 8499 Read 4 Done Done New false
22009 7501 Acked 8500 Write 4 Done Done New false
22010 7502 Acked 8501 Read 5 Done Done New false
22011 7503 Acked 8502 Write 1 Done Done New false
22012 7504 Acked 8503 Write 8 Done Done New false
22013 7505 Acked 8504 Write 10 Done Done New false
22014 7506 Acked 8505 Read 4 Done Done New false
22015 7507 Acked 8506 Flush 0 Done Done New false
22016 7508 Acked 8507 Read 6 Done Done New false
22017 7509 Acked 8508 Write 6 Done Done New false
22018 7510 Acked 8509 Write 1 Done Done New false
22019 7511 Acked 8510 Read 6 Done Done New false
22020 7512 Acked 8511 Read 9 Done Done New false
22021 7513 Acked 8512 Write 1 Done Done New false
22022 7514 Acked 8513 Read 7 Done Done New false
22023 7515 Acked 8514 Read 4 Done Done New false
22024 7516 Acked 8515 Write 5 Done Done New false
22025 7517 Acked 8516 Read 3 Done Done New false
22026 7518 Acked 8517 Write 5 Done Done New false
22027 7519 Acked 8518 Read 8 Done Done New false
22028 7520 Acked 8519 Write 6 Done Done New false
22029 7521 Acked 8520 Read 8 Done Done New false
22030 7522 Acked 8521 Write 5 Done Done New false
22031 7523 Acked 8522 Write 8 Done Done New false
22032 7524 Acked 8523 Write 1 Done Done New false
22033 7525 Acked 8524 Write 10 Done Done New false
22034 7526 Acked 8525 Read 9 Done Done New false
22035 7527 Acked 8526 Read 10 Done Done New false
22036 7528 Acked 8527 Read 7 Done Done New false
22037 7529 Acked 8528 Read 7 Done Done New false
22038 7530 Acked 8529 Write 1 Done Done New false
22039 7531 Acked 8530 Read 7 Done Done New false
22040 7532 Acked 8531 Write 4 Done Done New false
22041 7533 Acked 8532 Read 9 Done Done New false
22042 7534 Acked 8533 Flush 0 Done Done New false
22043 7535 Acked 8534 Read 10 Done Done New false
22044 STATES DS:0 DS:1 DS:2 TOTAL
22045 New 0 0 300 300
22046 Sent 0 0 0 0
22047 Done 300 300 0 600
22048 Skipped 0 0 0 0
22049 Error 0 0 0 0
22050 Last Flush: 8533 8533 8234
22051 Downstairs last five completed: 8234 8233 8232 8231 8230
22052 Upstairs last five completed: 7535 7534 7533 7532 7531
22053 CLIENT: Up:0 ds:300 act:2
220542024-02-12T01:38:51.311ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
220552024-02-12T01:38:51.311ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
220562024-02-12T01:38:51.311ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
220572024-02-12T01:38:51.311ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
220582024-02-12T01:38:51.312ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
220592024-02-12T01:38:51.312ZINFOcrucible: [2] client re-new 301 jobs since flush 8234 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
220602024-02-12T01:38:51.312ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
220612024-02-12T01:38:52.313ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
220622024-02-12T01:38:52.313ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
220632024-02-12T01:38:52.313ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
220642024-02-12T01:38:52.313ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
220652024-02-12T01:38:52.313ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
220662024-02-12T01:38:52.313ZINFOcrucible: [2] client re-new 301 jobs since flush 8234 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
220672024-02-12T01:38:52.313ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
220682024-02-12T01:38:53.314ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
220692024-02-12T01:38:53.314ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
220702024-02-12T01:38:53.314ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
220712024-02-12T01:38:53.314ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
220722024-02-12T01:38:53.314ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
220732024-02-12T01:38:53.314ZINFOcrucible: [2] client re-new 301 jobs since flush 8234 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
220742024-02-12T01:38:53.314ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
220752024-02-12T01:38:54.315ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
220762024-02-12T01:38:54.315ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
220772024-02-12T01:38:54.315ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
220782024-02-12T01:38:54.315ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
220792024-02-12T01:38:54.315ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
220802024-02-12T01:38:54.316ZINFOcrucible: [2] client re-new 301 jobs since flush 8234 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
220812024-02-12T01:38:54.316ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
22082 ----------------------------------------------------------------
22083 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
22084 States: Active Active Offline
22085 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
22086 7236 Acked 8235 Read 3 Done Done New true
22087 7237 Acked 8236 Read 4 Done Done New true
22088 7238 Acked 8237 Read 9 Done Done New true
22089 7239 Acked 8238 Read 1 Done Done New true
22090 7240 Acked 8239 Write 8 Done Done New true
22091 7241 Acked 8240 Read 10 Done Done New true
22092 7242 Acked 8241 Write 6 Done Done New true
22093 7243 Acked 8242 Read 6 Done Done New true
22094 7244 Acked 8243 Write 10 Done Done New true
22095 7245 Acked 8244 Write 4 Done Done New true
22096 7246 Acked 8245 Read 10 Done Done New true
22097 7247 Acked 8246 Read 7 Done Done New true
22098 7248 Acked 8247 Read 4 Done Done New true
22099 7249 Acked 8248 Read 6 Done Done New true
22100 7250 Acked 8249 Read 6 Done Done New true
22101 7251 Acked 8250 Read 9 Done Done New true
22102 7252 Acked 8251 Read 5 Done Done New true
22103 7253 Acked 8252 Read 8 Done Done New true
22104 7254 Acked 8253 Write 10 Done Done New true
22105 7255 Acked 8254 Read 4 Done Done New true
22106 7256 Acked 8255 Write 6 Done Done New true
22107 7257 Acked 8256 Read 3 Done Done New true
22108 7258 Acked 8257 Read 7 Done Done New true
22109 7259 Acked 8258 Flush 0 Done Done New true
22110 7260 Acked 8259 Write 4 Done Done New true
22111 7261 Acked 8260 Read 8 Done Done New true
22112 7262 Acked 8261 Read 2 Done Done New true
22113 7263 Acked 8262 Read 5 Done Done New true
22114 7264 Acked 8263 Write 3 Done Done New true
22115 7265 Acked 8264 Flush 0 Done Done New true
22116 7266 Acked 8265 Write 7 Done Done New true
22117 7267 Acked 8266 Read 10 Done Done New true
22118 7268 Acked 8267 Flush 0 Done Done New true
22119 7269 Acked 8268 Flush 0 Done Done New true
22120 7270 Acked 8269 Read 2 Done Done New true
22121 7271 Acked 8270 Write 10 Done Done New true
22122 7272 Acked 8271 Write 9 Done Done New true
22123 7273 Acked 8272 Read 9 Done Done New true
22124 7274 Acked 8273 Read 8 Done Done New true
22125 7275 Acked 8274 Read 9 Done Done New true
22126 7276 Acked 8275 Read 7 Done Done New true
22127 7277 Acked 8276 Write 1 Done Done New true
22128 7278 Acked 8277 Read 6 Done Done New true
22129 7279 Acked 8278 Write 5 Done Done New true
22130 7280 Acked 8279 Flush 0 Done Done New true
22131 7281 Acked 8280 Write 10 Done Done New true
22132 7282 Acked 8281 Read 9 Done Done New true
22133 7283 Acked 8282 Read 4 Done Done New true
22134 7284 Acked 8283 Read 4 Done Done New true
22135 7285 Acked 8284 Flush 0 Done Done New true
22136 7286 Acked 8285 Read 6 Done Done New true
22137 7287 Acked 8286 Write 6 Done Done New true
22138 7288 Acked 8287 Write 8 Done Done New true
22139 7289 Acked 8288 Read 7 Done Done New true
22140 7290 Acked 8289 Write 7 Done Done New true
22141 7291 Acked 8290 Write 3 Done Done New true
22142 7292 Acked 8291 Read 10 Done Done New true
22143 7293 Acked 8292 Write 10 Done Done New true
22144 7294 Acked 8293 Read 1 Done Done New true
22145 7295 Acked 8294 Read 3 Done Done New true
22146 7296 Acked 8295 Read 10 Done Done New true
22147 7297 Acked 8296 Write 10 Done Done New true
22148 7298 Acked 8297 Read 2 Done Done New true
22149 7299 Acked 8298 Read 4 Done Done New true
22150 7300 Acked 8299 Read 5 Done Done New true
22151 7301 Acked 8300 Write 7 Done Done New true
22152 7302 Acked 8301 Read 9 Done Done New true
22153 7303 Acked 8302 Read 9 Done Done New true
22154 7304 Acked 8303 Read 8 Done Done New true
22155 7305 Acked 8304 Write 6 Done Done New true
22156 7306 Acked 8305 Write 6 Done Done New true
22157 7307 Acked 8306 Write 4 Done Done New true
22158 7308 Acked 8307 Write 1 Done Done New true
22159 7309 Acked 8308 Read 9 Done Done New true
22160 7310 Acked 8309 Read 1 Done Done New true
22161 7311 Acked 8310 Read 3 Done Done New true
22162 7312 Acked 8311 Read 1 Done Done New true
22163 7313 Acked 8312 Read 7 Done Done New true
22164 7314 Acked 8313 Read 1 Done Done New true
22165 7315 Acked 8314 Read 4 Done Done New true
22166 7316 Acked 8315 Read 7 Done Done New true
22167 7317 Acked 8316 Write 5 Done Done New true
22168 7318 Acked 8317 Read 9 Done Done New true
22169 7319 Acked 8318 Read 7 Done Done New true
22170 7320 Acked 8319 Read 1 Done Done New true
22171 7321 Acked 8320 Read 10 Done Done New true
22172 7322 Acked 8321 Read 7 Done Done New true
22173 7323 Acked 8322 Flush 0 Done Done New true
22174 7324 Acked 8323 Read 4 Done Done New true
22175 7325 Acked 8324 Write 3 Done Done New true
22176 7326 Acked 8325 Read 3 Done Done New true
22177 7327 Acked 8326 Read 8 Done Done New true
22178 7328 Acked 8327 Flush 0 Done Done New true
22179 7329 Acked 8328 Write 9 Done Done New true
22180 7330 Acked 8329 Read 1 Done Done New true
22181 7331 Acked 8330 Write 3 Done Done New true
22182 7332 Acked 8331 Read 8 Done Done New true
22183 7333 Acked 8332 Read 6 Done Done New true
22184 7334 Acked 8333 Write 2 Done Done New true
22185 7335 Acked 8334 Read 7 Done Done New true
22186 7336 Acked 8335 Read 9 Done Done New true
22187 7337 Acked 8336 Write 4 Done Done New true
22188 7338 Acked 8337 Read 10 Done Done New true
22189 7339 Acked 8338 Read 3 Done Done New true
22190 7340 Acked 8339 Write 6 Done Done New true
22191 7341 Acked 8340 Read 8 Done Done New true
22192 7342 Acked 8341 Read 9 Done Done New true
22193 7343 Acked 8342 Flush 0 Done Done New true
22194 7344 Acked 8343 Read 6 Done Done New true
22195 7345 Acked 8344 Read 5 Done Done New true
22196 7346 Acked 8345 Write 6 Done Done New true
22197 7347 Acked 8346 Write 1 Done Done New true
22198 7348 Acked 8347 Write 9 Done Done New true
22199 7349 Acked 8348 Write 7 Done Done New true
22200 7350 Acked 8349 Read 10 Done Done New true
22201 7351 Acked 8350 Write 2 Done Done New true
22202 7352 Acked 8351 Read 2 Done Done New true
22203 7353 Acked 8352 Read 10 Done Done New true
22204 7354 Acked 8353 Flush 0 Done Done New true
22205 7355 Acked 8354 Flush 0 Done Done New true
22206 7356 Acked 8355 Write 8 Done Done New true
22207 7357 Acked 8356 Write 5 Done Done New true
22208 7358 Acked 8357 Read 3 Done Done New true
22209 7359 Acked 8358 Write 4 Done Done New true
22210 7360 Acked 8359 Read 4 Done Done New true
22211 7361 Acked 8360 Read 7 Done Done New true
22212 7362 Acked 8361 Write 9 Done Done New true
22213 7363 Acked 8362 Flush 0 Done Done New true
22214 7364 Acked 8363 Flush 0 Done Done New true
22215 7365 Acked 8364 Write 1 Done Done New true
22216 7366 Acked 8365 Read 7 Done Done New true
22217 7367 Acked 8366 Write 10 Done Done New true
22218 7368 Acked 8367 Write 1 Done Done New true
22219 7369 Acked 8368 Read 7 Done Done New true
22220 7370 Acked 8369 Read 8 Done Done New true
22221 7371 Acked 8370 Read 8 Done Done New true
22222 7372 Acked 8371 Read 10 Done Done New true
22223 7373 Acked 8372 Read 8 Done Done New true
22224 7374 Acked 8373 Read 4 Done Done New true
22225 7375 Acked 8374 Read 3 Done Done New true
22226 7376 Acked 8375 Read 5 Done Done New true
22227 7377 Acked 8376 Flush 0 Done Done New true
22228 7378 Acked 8377 Write 1 Done Done New true
22229 7379 Acked 8378 Read 3 Done Done New true
22230 7380 Acked 8379 Read 1 Done Done New true
22231 7381 Acked 8380 Write 8 Done Done New true
22232 7382 Acked 8381 Write 8 Done Done New true
22233 7383 Acked 8382 Flush 0 Done Done New true
22234 7384 Acked 8383 Read 4 Done Done New true
22235 7385 Acked 8384 Read 5 Done Done New true
22236 7386 Acked 8385 Read 5 Done Done New true
22237 7387 Acked 8386 Write 10 Done Done New true
22238 7388 Acked 8387 Flush 0 Done Done New true
22239 7389 Acked 8388 Read 3 Done Done New true
22240 7390 Acked 8389 Flush 0 Done Done New true
22241 7391 Acked 8390 Write 6 Done Done New true
22242 7392 Acked 8391 Read 10 Done Done New true
22243 7393 Acked 8392 Flush 0 Done Done New true
22244 7394 Acked 8393 Read 7 Done Done New true
22245 7395 Acked 8394 Write 1 Done Done New true
22246 7396 Acked 8395 Read 5 Done Done New true
22247 7397 Acked 8396 Write 9 Done Done New true
22248 7398 Acked 8397 Write 4 Done Done New true
22249 7399 Acked 8398 Flush 0 Done Done New true
22250 7400 Acked 8399 Read 2 Done Done New true
22251 7401 Acked 8400 Read 8 Done Done New true
22252 7402 Acked 8401 Read 7 Done Done New true
22253 7403 Acked 8402 Write 1 Done Done New true
22254 7404 Acked 8403 Write 6 Done Done New true
22255 7405 Acked 8404 Write 1 Done Done New true
22256 7406 Acked 8405 Write 9 Done Done New true
22257 7407 Acked 8406 Read 5 Done Done New true
22258 7408 Acked 8407 Write 10 Done Done New true
22259 7409 Acked 8408 Write 3 Done Done New true
22260 7410 Acked 8409 Read 4 Done Done New true
22261 7411 Acked 8410 Write 2 Done Done New true
22262 7412 Acked 8411 Read 5 Done Done New true
22263 7413 Acked 8412 Read 1 Done Done New true
22264 7414 Acked 8413 Read 6 Done Done New true
22265 7415 Acked 8414 Write 8 Done Done New true
22266 7416 Acked 8415 Read 9 Done Done New true
22267 7417 Acked 8416 Write 10 Done Done New true
22268 7418 Acked 8417 Write 10 Done Done New true
22269 7419 Acked 8418 Read 5 Done Done New true
22270 7420 Acked 8419 Read 7 Done Done New true
22271 7421 Acked 8420 Write 5 Done Done New true
22272 7422 Acked 8421 Read 6 Done Done New true
22273 7423 Acked 8422 Write 9 Done Done New true
22274 7424 Acked 8423 Read 7 Done Done New true
22275 7425 Acked 8424 Write 6 Done Done New true
22276 7426 Acked 8425 Write 10 Done Done New true
22277 7427 Acked 8426 Read 6 Done Done New true
22278 7428 Acked 8427 Flush 0 Done Done New true
22279 7429 Acked 8428 Read 3 Done Done New true
22280 7430 Acked 8429 Write 3 Done Done New true
22281 7431 Acked 8430 Read 3 Done Done New true
22282 7432 Acked 8431 Read 10 Done Done New true
22283 7433 Acked 8432 Read 10 Done Done New true
22284 7434 Acked 8433 Read 9 Done Done New true
22285 7435 Acked 8434 Write 9 Done Done New true
22286 7436 Acked 8435 Write 9 Done Done New true
22287 7437 Acked 8436 Read 9 Done Done New true
22288 7438 Acked 8437 Read 7 Done Done New true
22289 7439 Acked 8438 Write 6 Done Done New true
22290 7440 Acked 8439 Write 6 Done Done New true
22291 7441 Acked 8440 Read 3 Done Done New true
22292 7442 Acked 8441 Read 4 Done Done New true
22293 7443 Acked 8442 Read 10 Done Done New true
22294 7444 Acked 8443 Write 10 Done Done New true
22295 7445 Acked 8444 Write 9 Done Done New true
22296 7446 Acked 8445 Write 2 Done Done New true
22297 7447 Acked 8446 Write 3 Done Done New true
22298 7448 Acked 8447 Read 10 Done Done New true
22299 7449 Acked 8448 Write 9 Done Done New true
22300 7450 Acked 8449 Write 6 Done Done New true
22301 7451 Acked 8450 Read 7 Done Done New true
22302 7452 Acked 8451 Write 8 Done Done New true
22303 7453 Acked 8452 Read 1 Done Done New true
22304 7454 Acked 8453 Read 3 Done Done New true
22305 7455 Acked 8454 Write 1 Done Done New true
22306 7456 Acked 8455 Write 4 Done Done New true
22307 7457 Acked 8456 Read 2 Done Done New true
22308 7458 Acked 8457 Read 6 Done Done New true
22309 7459 Acked 8458 Read 9 Done Done New true
22310 7460 Acked 8459 Flush 0 Done Done New true
22311 7461 Acked 8460 Write 4 Done Done New true
22312 7462 Acked 8461 Read 3 Done Done New true
22313 7463 Acked 8462 Read 1 Done Done New true
22314 7464 Acked 8463 Write 9 Done Done New true
22315 7465 Acked 8464 Read 7 Done Done New true
22316 7466 Acked 8465 Write 6 Done Done New true
22317 7467 Acked 8466 Write 2 Done Done New true
22318 7468 Acked 8467 Read 9 Done Done New true
22319 7469 Acked 8468 Write 4 Done Done New true
22320 7470 Acked 8469 Write 2 Done Done New true
22321 7471 Acked 8470 Read 8 Done Done New true
22322 7472 Acked 8471 Write 2 Done Done New true
22323 7473 Acked 8472 Read 3 Done Done New true
22324 7474 Acked 8473 Write 1 Done Done New true
22325 7475 Acked 8474 Write 6 Done Done New true
22326 7476 Acked 8475 Read 10 Done Done New true
22327 7477 Acked 8476 Read 3 Done Done New true
22328 7478 Acked 8477 Write 4 Done Done New true
22329 7479 Acked 8478 Read 1 Done Done New true
22330 7480 Acked 8479 Read 4 Done Done New true
22331 7481 Acked 8480 Write 9 Done Done New true
22332 7482 Acked 8481 Write 4 Done Done New true
22333 7483 Acked 8482 Read 5 Done Done New true
22334 7484 Acked 8483 Read 1 Done Done New true
22335 7485 Acked 8484 Read 3 Done Done New true
22336 7486 Acked 8485 Write 2 Done Done New true
22337 7487 Acked 8486 Write 7 Done Done New true
22338 7488 Acked 8487 Write 2 Done Done New true
22339 7489 Acked 8488 Read 9 Done Done New true
22340 7490 Acked 8489 Read 3 Done Done New true
22341 7491 Acked 8490 Read 8 Done Done New true
22342 7492 Acked 8491 Read 9 Done Done New true
22343 7493 Acked 8492 Write 10 Done Done New true
22344 7494 Acked 8493 Write 4 Done Done New true
22345 7495 Acked 8494 Flush 0 Done Done New true
22346 7496 Acked 8495 Read 6 Done Done New true
22347 7497 Acked 8496 Read 1 Done Done New true
22348 7498 Acked 8497 Read 10 Done Done New true
22349 7499 Acked 8498 Write 9 Done Done New true
22350 7500 Acked 8499 Read 4 Done Done New true
22351 7501 Acked 8500 Write 4 Done Done New true
22352 7502 Acked 8501 Read 5 Done Done New true
22353 7503 Acked 8502 Write 1 Done Done New true
22354 7504 Acked 8503 Write 8 Done Done New true
22355 7505 Acked 8504 Write 10 Done Done New true
22356 7506 Acked 8505 Read 4 Done Done New true
22357 7507 Acked 8506 Flush 0 Done Done New true
22358 7508 Acked 8507 Read 6 Done Done New true
22359 7509 Acked 8508 Write 6 Done Done New true
22360 7510 Acked 8509 Write 1 Done Done New true
22361 7511 Acked 8510 Read 6 Done Done New true
22362 7512 Acked 8511 Read 9 Done Done New true
22363 7513 Acked 8512 Write 1 Done Done New true
22364 7514 Acked 8513 Read 7 Done Done New true
22365 7515 Acked 8514 Read 4 Done Done New true
22366 7516 Acked 8515 Write 5 Done Done New true
22367 7517 Acked 8516 Read 3 Done Done New true
22368 7518 Acked 8517 Write 5 Done Done New true
22369 7519 Acked 8518 Read 8 Done Done New true
22370 7520 Acked 8519 Write 6 Done Done New true
22371 7521 Acked 8520 Read 8 Done Done New true
22372 7522 Acked 8521 Write 5 Done Done New true
22373 7523 Acked 8522 Write 8 Done Done New true
22374 7524 Acked 8523 Write 1 Done Done New true
22375 7525 Acked 8524 Write 10 Done Done New true
22376 7526 Acked 8525 Read 9 Done Done New true
22377 7527 Acked 8526 Read 10 Done Done New true
22378 7528 Acked 8527 Read 7 Done Done New true
22379 7529 Acked 8528 Read 7 Done Done New true
22380 7530 Acked 8529 Write 1 Done Done New true
22381 7531 Acked 8530 Read 7 Done Done New true
22382 7532 Acked 8531 Write 4 Done Done New true
22383 7533 Acked 8532 Read 9 Done Done New true
22384 7534 Acked 8533 Flush 0 Done Done New true
22385 7535 Acked 8534 Read 10 Done Done New true
22386 7536 Acked 8535 Flush 0 Done Done New true
22387 STATES DS:0 DS:1 DS:2 TOTAL
22388 New 0 0 301 301
22389 Sent 0 0 0 0
22390 Done 301 301 0 602
22391 Skipped 0 0 0 0
22392 Error 0 0 0 0
22393 Last Flush: 8535 8535 8234
22394 Downstairs last five completed: 8234 8233 8232 8231 8230
22395 Upstairs last five completed: 7536 7535 7534 7533 7532
22396 CLIENT: Up:0 ds:301 act:2
223972024-02-12T01:38:55.316ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
223982024-02-12T01:38:55.317ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
223992024-02-12T01:38:55.317ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
224002024-02-12T01:38:55.317ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
224012024-02-12T01:38:55.317ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
224022024-02-12T01:38:55.317ZINFOcrucible: [2] client re-new 301 jobs since flush 8234 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
224032024-02-12T01:38:55.317ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
224042024-02-12T01:38:56.318ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
224052024-02-12T01:38:56.318ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
224062024-02-12T01:38:56.318ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
224072024-02-12T01:38:56.318ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
224082024-02-12T01:38:56.318ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
224092024-02-12T01:38:56.318ZINFOcrucible: [2] client re-new 301 jobs since flush 8234 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
224102024-02-12T01:38:56.318ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
224112024-02-12T01:38:57.319ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
224122024-02-12T01:38:57.319ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
224132024-02-12T01:38:57.319ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
224142024-02-12T01:38:57.319ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
224152024-02-12T01:38:57.319ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
224162024-02-12T01:38:57.319ZINFOcrucible: [2] client re-new 301 jobs since flush 8234 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
224172024-02-12T01:38:57.319ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
22418 [8830] got start action
22419 [8830] I am going to restart
22420 Starting downstairs at port 8830
22421 Make output file at "/tmp/dsc/downstairs-8830.txt"
22422 [8830][2] reports Starting
224232024-02-12T01:38:58.320ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
224242024-02-12T01:38:58.320ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
224252024-02-12T01:38:58.320ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
224262024-02-12T01:38:58.320ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
224272024-02-12T01:38:58.320ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
224282024-02-12T01:38:58.320ZINFOcrucible: [2] client re-new 301 jobs since flush 8234 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
224292024-02-12T01:38:58.320ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
22430 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1189)
22431 [8830][2] reports Running
22432 ----------------------------------------------------------------
22433 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
22434 States: Active Active Offline
22435 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
22436 7236 Acked 8235 Read 3 Done Done New true
22437 7237 Acked 8236 Read 4 Done Done New true
22438 7238 Acked 8237 Read 9 Done Done New true
22439 7239 Acked 8238 Read 1 Done Done New true
22440 7240 Acked 8239 Write 8 Done Done New true
22441 7241 Acked 8240 Read 10 Done Done New true
22442 7242 Acked 8241 Write 6 Done Done New true
22443 7243 Acked 8242 Read 6 Done Done New true
22444 7244 Acked 8243 Write 10 Done Done New true
22445 7245 Acked 8244 Write 4 Done Done New true
22446 7246 Acked 8245 Read 10 Done Done New true
22447 7247 Acked 8246 Read 7 Done Done New true
22448 7248 Acked 8247 Read 4 Done Done New true
22449 7249 Acked 8248 Read 6 Done Done New true
22450 7250 Acked 8249 Read 6 Done Done New true
22451 7251 Acked 8250 Read 9 Done Done New true
22452 7252 Acked 8251 Read 5 Done Done New true
22453 7253 Acked 8252 Read 8 Done Done New true
22454 7254 Acked 8253 Write 10 Done Done New true
22455 7255 Acked 8254 Read 4 Done Done New true
22456 7256 Acked 8255 Write 6 Done Done New true
22457 7257 Acked 8256 Read 3 Done Done New true
22458 7258 Acked 8257 Read 7 Done Done New true
22459 7259 Acked 8258 Flush 0 Done Done New true
22460 7260 Acked 8259 Write 4 Done Done New true
22461 7261 Acked 8260 Read 8 Done Done New true
22462 7262 Acked 8261 Read 2 Done Done New true
22463 7263 Acked 8262 Read 5 Done Done New true
22464 7264 Acked 8263 Write 3 Done Done New true
22465 7265 Acked 8264 Flush 0 Done Done New true
22466 7266 Acked 8265 Write 7 Done Done New true
22467 7267 Acked 8266 Read 10 Done Done New true
22468 7268 Acked 8267 Flush 0 Done Done New true
22469 7269 Acked 8268 Flush 0 Done Done New true
22470 7270 Acked 8269 Read 2 Done Done New true
22471 7271 Acked 8270 Write 10 Done Done New true
22472 7272 Acked 8271 Write 9 Done Done New true
22473 7273 Acked 8272 Read 9 Done Done New true
22474 7274 Acked 8273 Read 8 Done Done New true
22475 7275 Acked 8274 Read 9 Done Done New true
22476 7276 Acked 8275 Read 7 Done Done New true
22477 7277 Acked 8276 Write 1 Done Done New true
22478 7278 Acked 8277 Read 6 Done Done New true
22479 7279 Acked 8278 Write 5 Done Done New true
22480 7280 Acked 8279 Flush 0 Done Done New true
22481 7281 Acked 8280 Write 10 Done Done New true
22482 7282 Acked 8281 Read 9 Done Done New true
22483 7283 Acked 8282 Read 4 Done Done New true
22484 7284 Acked 8283 Read 4 Done Done New true
22485 7285 Acked 8284 Flush 0 Done Done New true
22486 7286 Acked 8285 Read 6 Done Done New true
22487 7287 Acked 8286 Write 6 Done Done New true
22488 7288 Acked 8287 Write 8 Done Done New true
22489 7289 Acked 8288 Read 7 Done Done New true
22490 7290 Acked 8289 Write 7 Done Done New true
22491 7291 Acked 8290 Write 3 Done Done New true
22492 7292 Acked 8291 Read 10 Done Done New true
22493 7293 Acked 8292 Write 10 Done Done New true
22494 7294 Acked 8293 Read 1 Done Done New true
22495 7295 Acked 8294 Read 3 Done Done New true
22496 7296 Acked 8295 Read 10 Done Done New true
22497 7297 Acked 8296 Write 10 Done Done New true
22498 7298 Acked 8297 Read 2 Done Done New true
22499 7299 Acked 8298 Read 4 Done Done New true
22500 7300 Acked 8299 Read 5 Done Done New true
22501 7301 Acked 8300 Write 7 Done Done New true
22502 7302 Acked 8301 Read 9 Done Done New true
22503 7303 Acked 8302 Read 9 Done Done New true
22504 7304 Acked 8303 Read 8 Done Done New true
22505 7305 Acked 8304 Write 6 Done Done New true
22506 7306 Acked 8305 Write 6 Done Done New true
22507 7307 Acked 8306 Write 4 Done Done New true
22508 7308 Acked 8307 Write 1 Done Done New true
22509 7309 Acked 8308 Read 9 Done Done New true
22510 7310 Acked 8309 Read 1 Done Done New true
22511 7311 Acked 8310 Read 3 Done Done New true
22512 7312 Acked 8311 Read 1 Done Done New true
22513 7313 Acked 8312 Read 7 Done Done New true
22514 7314 Acked 8313 Read 1 Done Done New true
22515 7315 Acked 8314 Read 4 Done Done New true
22516 7316 Acked 8315 Read 7 Done Done New true
22517 7317 Acked 8316 Write 5 Done Done New true
22518 7318 Acked 8317 Read 9 Done Done New true
22519 7319 Acked 8318 Read 7 Done Done New true
22520 7320 Acked 8319 Read 1 Done Done New true
22521 7321 Acked 8320 Read 10 Done Done New true
22522 7322 Acked 8321 Read 7 Done Done New true
22523 7323 Acked 8322 Flush 0 Done Done New true
22524 7324 Acked 8323 Read 4 Done Done New true
22525 7325 Acked 8324 Write 3 Done Done New true
22526 7326 Acked 8325 Read 3 Done Done New true
22527 7327 Acked 8326 Read 8 Done Done New true
22528 7328 Acked 8327 Flush 0 Done Done New true
22529 7329 Acked 8328 Write 9 Done Done New true
22530 7330 Acked 8329 Read 1 Done Done New true
22531 7331 Acked 8330 Write 3 Done Done New true
22532 7332 Acked 8331 Read 8 Done Done New true
22533 7333 Acked 8332 Read 6 Done Done New true
22534 7334 Acked 8333 Write 2 Done Done New true
22535 7335 Acked 8334 Read 7 Done Done New true
22536 7336 Acked 8335 Read 9 Done Done New true
22537 7337 Acked 8336 Write 4 Done Done New true
22538 7338 Acked 8337 Read 10 Done Done New true
22539 7339 Acked 8338 Read 3 Done Done New true
22540 7340 Acked 8339 Write 6 Done Done New true
22541 7341 Acked 8340 Read 8 Done Done New true
22542 7342 Acked 8341 Read 9 Done Done New true
22543 7343 Acked 8342 Flush 0 Done Done New true
22544 7344 Acked 8343 Read 6 Done Done New true
22545 7345 Acked 8344 Read 5 Done Done New true
22546 7346 Acked 8345 Write 6 Done Done New true
22547 7347 Acked 8346 Write 1 Done Done New true
22548 7348 Acked 8347 Write 9 Done Done New true
22549 7349 Acked 8348 Write 7 Done Done New true
22550 7350 Acked 8349 Read 10 Done Done New true
22551 7351 Acked 8350 Write 2 Done Done New true
22552 7352 Acked 8351 Read 2 Done Done New true
22553 7353 Acked 8352 Read 10 Done Done New true
22554 7354 Acked 8353 Flush 0 Done Done New true
22555 7355 Acked 8354 Flush 0 Done Done New true
22556 7356 Acked 8355 Write 8 Done Done New true
22557 7357 Acked 8356 Write 5 Done Done New true
22558 7358 Acked 8357 Read 3 Done Done New true
22559 7359 Acked 8358 Write 4 Done Done New true
22560 7360 Acked 8359 Read 4 Done Done New true
22561 7361 Acked 8360 Read 7 Done Done New true
22562 7362 Acked 8361 Write 9 Done Done New true
22563 7363 Acked 8362 Flush 0 Done Done New true
22564 7364 Acked 8363 Flush 0 Done Done New true
22565 7365 Acked 8364 Write 1 Done Done New true
22566 7366 Acked 8365 Read 7 Done Done New true
22567 7367 Acked 8366 Write 10 Done Done New true
22568 7368 Acked 8367 Write 1 Done Done New true
22569 7369 Acked 8368 Read 7 Done Done New true
22570 7370 Acked 8369 Read 8 Done Done New true
22571 7371 Acked 8370 Read 8 Done Done New true
22572 7372 Acked 8371 Read 10 Done Done New true
22573 7373 Acked 8372 Read 8 Done Done New true
22574 7374 Acked 8373 Read 4 Done Done New true
22575 7375 Acked 8374 Read 3 Done Done New true
22576 7376 Acked 8375 Read 5 Done Done New true
22577 7377 Acked 8376 Flush 0 Done Done New true
22578 7378 Acked 8377 Write 1 Done Done New true
22579 7379 Acked 8378 Read 3 Done Done New true
22580 7380 Acked 8379 Read 1 Done Done New true
22581 7381 Acked 8380 Write 8 Done Done New true
22582 7382 Acked 8381 Write 8 Done Done New true
22583 7383 Acked 8382 Flush 0 Done Done New true
22584 7384 Acked 8383 Read 4 Done Done New true
22585 7385 Acked 8384 Read 5 Done Done New true
22586 7386 Acked 8385 Read 5 Done Done New true
22587 7387 Acked 8386 Write 10 Done Done New true
22588 7388 Acked 8387 Flush 0 Done Done New true
22589 7389 Acked 8388 Read 3 Done Done New true
22590 7390 Acked 8389 Flush 0 Done Done New true
22591 7391 Acked 8390 Write 6 Done Done New true
22592 7392 Acked 8391 Read 10 Done Done New true
22593 7393 Acked 8392 Flush 0 Done Done New true
22594 7394 Acked 8393 Read 7 Done Done New true
22595 7395 Acked 8394 Write 1 Done Done New true
22596 7396 Acked 8395 Read 5 Done Done New true
22597 7397 Acked 8396 Write 9 Done Done New true
22598 7398 Acked 8397 Write 4 Done Done New true
22599 7399 Acked 8398 Flush 0 Done Done New true
22600 7400 Acked 8399 Read 2 Done Done New true
22601 7401 Acked 8400 Read 8 Done Done New true
22602 7402 Acked 8401 Read 7 Done Done New true
22603 7403 Acked 8402 Write 1 Done Done New true
22604 7404 Acked 8403 Write 6 Done Done New true
22605 7405 Acked 8404 Write 1 Done Done New true
22606 7406 Acked 8405 Write 9 Done Done New true
22607 7407 Acked 8406 Read 5 Done Done New true
22608 7408 Acked 8407 Write 10 Done Done New true
22609 7409 Acked 8408 Write 3 Done Done New true
22610 7410 Acked 8409 Read 4 Done Done New true
22611 7411 Acked 8410 Write 2 Done Done New true
22612 7412 Acked 8411 Read 5 Done Done New true
22613 7413 Acked 8412 Read 1 Done Done New true
22614 7414 Acked 8413 Read 6 Done Done New true
22615 7415 Acked 8414 Write 8 Done Done New true
22616 7416 Acked 8415 Read 9 Done Done New true
22617 7417 Acked 8416 Write 10 Done Done New true
22618 7418 Acked 8417 Write 10 Done Done New true
22619 7419 Acked 8418 Read 5 Done Done New true
22620 7420 Acked 8419 Read 7 Done Done New true
22621 7421 Acked 8420 Write 5 Done Done New true
22622 7422 Acked 8421 Read 6 Done Done New true
22623 7423 Acked 8422 Write 9 Done Done New true
22624 7424 Acked 8423 Read 7 Done Done New true
22625 7425 Acked 8424 Write 6 Done Done New true
22626 7426 Acked 8425 Write 10 Done Done New true
22627 7427 Acked 8426 Read 6 Done Done New true
22628 7428 Acked 8427 Flush 0 Done Done New true
22629 7429 Acked 8428 Read 3 Done Done New true
22630 7430 Acked 8429 Write 3 Done Done New true
22631 7431 Acked 8430 Read 3 Done Done New true
22632 7432 Acked 8431 Read 10 Done Done New true
22633 7433 Acked 8432 Read 10 Done Done New true
22634 7434 Acked 8433 Read 9 Done Done New true
22635 7435 Acked 8434 Write 9 Done Done New true
22636 7436 Acked 8435 Write 9 Done Done New true
22637 7437 Acked 8436 Read 9 Done Done New true
22638 7438 Acked 8437 Read 7 Done Done New true
22639 7439 Acked 8438 Write 6 Done Done New true
22640 7440 Acked 8439 Write 6 Done Done New true
22641 7441 Acked 8440 Read 3 Done Done New true
22642 7442 Acked 8441 Read 4 Done Done New true
22643 7443 Acked 8442 Read 10 Done Done New true
22644 7444 Acked 8443 Write 10 Done Done New true
22645 7445 Acked 8444 Write 9 Done Done New true
22646 7446 Acked 8445 Write 2 Done Done New true
22647 7447 Acked 8446 Write 3 Done Done New true
22648 7448 Acked 8447 Read 10 Done Done New true
22649 7449 Acked 8448 Write 9 Done Done New true
22650 7450 Acked 8449 Write 6 Done Done New true
22651 7451 Acked 8450 Read 7 Done Done New true
22652 7452 Acked 8451 Write 8 Done Done New true
22653 7453 Acked 8452 Read 1 Done Done New true
22654 7454 Acked 8453 Read 3 Done Done New true
22655 7455 Acked 8454 Write 1 Done Done New true
22656 7456 Acked 8455 Write 4 Done Done New true
22657 7457 Acked 8456 Read 2 Done Done New true
22658 7458 Acked 8457 Read 6 Done Done New true
22659 7459 Acked 8458 Read 9 Done Done New true
22660 7460 Acked 8459 Flush 0 Done Done New true
22661 7461 Acked 8460 Write 4 Done Done New true
22662 7462 Acked 8461 Read 3 Done Done New true
22663 7463 Acked 8462 Read 1 Done Done New true
22664 7464 Acked 8463 Write 9 Done Done New true
22665 7465 Acked 8464 Read 7 Done Done New true
22666 7466 Acked 8465 Write 6 Done Done New true
22667 7467 Acked 8466 Write 2 Done Done New true
22668 7468 Acked 8467 Read 9 Done Done New true
22669 7469 Acked 8468 Write 4 Done Done New true
22670 7470 Acked 8469 Write 2 Done Done New true
22671 7471 Acked 8470 Read 8 Done Done New true
22672 7472 Acked 8471 Write 2 Done Done New true
22673 7473 Acked 8472 Read 3 Done Done New true
22674 7474 Acked 8473 Write 1 Done Done New true
22675 7475 Acked 8474 Write 6 Done Done New true
22676 7476 Acked 8475 Read 10 Done Done New true
22677 7477 Acked 8476 Read 3 Done Done New true
22678 7478 Acked 8477 Write 4 Done Done New true
22679 7479 Acked 8478 Read 1 Done Done New true
22680 7480 Acked 8479 Read 4 Done Done New true
22681 7481 Acked 8480 Write 9 Done Done New true
22682 7482 Acked 8481 Write 4 Done Done New true
22683 7483 Acked 8482 Read 5 Done Done New true
22684 7484 Acked 8483 Read 1 Done Done New true
22685 7485 Acked 8484 Read 3 Done Done New true
22686 7486 Acked 8485 Write 2 Done Done New true
22687 7487 Acked 8486 Write 7 Done Done New true
22688 7488 Acked 8487 Write 2 Done Done New true
22689 7489 Acked 8488 Read 9 Done Done New true
22690 7490 Acked 8489 Read 3 Done Done New true
22691 7491 Acked 8490 Read 8 Done Done New true
22692 7492 Acked 8491 Read 9 Done Done New true
22693 7493 Acked 8492 Write 10 Done Done New true
22694 7494 Acked 8493 Write 4 Done Done New true
22695 7495 Acked 8494 Flush 0 Done Done New true
22696 7496 Acked 8495 Read 6 Done Done New true
22697 7497 Acked 8496 Read 1 Done Done New true
22698 7498 Acked 8497 Read 10 Done Done New true
22699 7499 Acked 8498 Write 9 Done Done New true
22700 7500 Acked 8499 Read 4 Done Done New true
22701 7501 Acked 8500 Write 4 Done Done New true
22702 7502 Acked 8501 Read 5 Done Done New true
22703 7503 Acked 8502 Write 1 Done Done New true
22704 7504 Acked 8503 Write 8 Done Done New true
22705 7505 Acked 8504 Write 10 Done Done New true
22706 7506 Acked 8505 Read 4 Done Done New true
22707 7507 Acked 8506 Flush 0 Done Done New true
22708 7508 Acked 8507 Read 6 Done Done New true
22709 7509 Acked 8508 Write 6 Done Done New true
22710 7510 Acked 8509 Write 1 Done Done New true
22711 7511 Acked 8510 Read 6 Done Done New true
22712 7512 Acked 8511 Read 9 Done Done New true
22713 7513 Acked 8512 Write 1 Done Done New true
22714 7514 Acked 8513 Read 7 Done Done New true
22715 7515 Acked 8514 Read 4 Done Done New true
22716 7516 Acked 8515 Write 5 Done Done New true
22717 7517 Acked 8516 Read 3 Done Done New true
22718 7518 Acked 8517 Write 5 Done Done New true
22719 7519 Acked 8518 Read 8 Done Done New true
22720 7520 Acked 8519 Write 6 Done Done New true
22721 7521 Acked 8520 Read 8 Done Done New true
22722 7522 Acked 8521 Write 5 Done Done New true
22723 7523 Acked 8522 Write 8 Done Done New true
22724 7524 Acked 8523 Write 1 Done Done New true
22725 7525 Acked 8524 Write 10 Done Done New true
22726 7526 Acked 8525 Read 9 Done Done New true
22727 7527 Acked 8526 Read 10 Done Done New true
22728 7528 Acked 8527 Read 7 Done Done New true
22729 7529 Acked 8528 Read 7 Done Done New true
22730 7530 Acked 8529 Write 1 Done Done New true
22731 7531 Acked 8530 Read 7 Done Done New true
22732 7532 Acked 8531 Write 4 Done Done New true
22733 7533 Acked 8532 Read 9 Done Done New true
22734 7534 Acked 8533 Flush 0 Done Done New true
22735 7535 Acked 8534 Read 10 Done Done New true
22736 7536 Acked 8535 Flush 0 Done Done New true
22737 STATES DS:0 DS:1 DS:2 TOTAL
22738 New 0 0 301 301
22739 Sent 0 0 0 0
22740 Done 301 301 0 602
22741 Skipped 0 0 0 0
22742 Error 0 0 0 0
22743 Last Flush: 8535 8535 8234
22744 Downstairs last five completed: 8234 8233 8232 8231 8230
22745 Upstairs last five completed: 7536 7535 7534 7533 7532
22746 CLIENT: Up:0 ds:301 act:2
227472024-02-12T01:38:59.321ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
227482024-02-12T01:38:59.321ZINFOcrucible: ds_connection connected = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
227492024-02-12T01:38:59.323ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
227502024-02-12T01:38:59.323ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
227512024-02-12T01:38:59.323ZINFOcrucible: send last flush ID to this DS: 8234 = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
227522024-02-12T01:38:59.323ZINFOcrucible: Replied this last flush ID: 8234 = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
227532024-02-12T01:38:59.323ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
227542024-02-12T01:38:59.323ZINFOcrucible: [2] Transition from Offline to Replay = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
227552024-02-12T01:38:59.323ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
227562024-02-12T01:38:59.323ZINFOcrucible: [2] Transition from Replay to Active = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
22757 ----------------------------------------------------------------
22758 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
22759 Downstairs last five completed: 8535 8534 8533 8532 8531
22760 Upstairs last five completed: 7536 7535 7534 7533 7532
22761 CLIENT: Up:0 ds:0 act:3
22762 Replay: All jobs finished, all DS active.
22763 Main task has work to do, go find it
22764 got dsc Stop(2)
22765 stop 2
22766 Feb 12 01:39:02.684 INFO [8830] Got stop action so:false kr:false
22767 request completed, latency_us: 421, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 872c04fc-7394-4ba6-bf26-f2a325bafcd5, remote_addr: 127.0.0.1:60456, local_addr: 127.0.0.1:9998
22768 Feb 12 01:39:02.685 INFO request completed, latency_us: 210{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2024-02-12T01:39:02.68524916Z","hostname":"ip-10-150-1-104.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"e1773b3e-b36f-4080-8b13-c0830f0d072e"}
22769 , response_code: 200, uri: /state/cid/2, method: GET, req_id: 353e0804-a476-4577-bfc4-e8ba85afacec, remote_addr: 127.0.0.1:60456, local_addr: 127.0.{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2024-02-12T01:39:02.685355454Z","hostname":"ip-10-150-1-104.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"e1773b3e-b36f-4080-8b13-c0830f0d072e"}
22770 0.1:9998
227712024-02-12T01:39:02.685ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
227722024-02-12T01:39:02.685ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
227732024-02-12T01:39:02.685ZINFOcrucible: [2] client re-new 0 jobs since flush 8535 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
227742024-02-12T01:39:02.685ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
22775 [8830] Exited with: ExitStatus(unix_wait_status(9))
22776 [8830][2] reports Exit
227772024-02-12T01:39:03.686ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
227782024-02-12T01:39:03.686ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
227792024-02-12T01:39:03.686ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
227802024-02-12T01:39:03.686ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
227812024-02-12T01:39:03.686ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
227822024-02-12T01:39:03.686ZINFOcrucible: [2] client re-new 0 jobs since flush 8535 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
227832024-02-12T01:39:03.686ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
227842024-02-12T01:39:04.688ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
227852024-02-12T01:39:04.688ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
227862024-02-12T01:39:04.688ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
227872024-02-12T01:39:04.688ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
227882024-02-12T01:39:04.688ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
227892024-02-12T01:39:04.688ZINFOcrucible: [2] client re-new 0 jobs since flush 8535 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
227902024-02-12T01:39:04.688ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
227912024-02-12T01:39:05.690ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
227922024-02-12T01:39:05.690ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
227932024-02-12T01:39:05.690ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
227942024-02-12T01:39:05.690ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
227952024-02-12T01:39:05.690ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
227962024-02-12T01:39:05.690ZINFOcrucible: [2] client re-new 0 jobs since flush 8535 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
227972024-02-12T01:39:05.690ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
22798 Feb 12 01:39:06.687 INFO request completed, latency_us: 229, response_code: 200, uri: /state/cid/2, method: GET, req_id: b54dbf3f-a7c4-4038-ba6a-59918997d189, remote_addr: 127.0.0.1:60456, local_addr: 127.0.0.1:9998
22799 001/300 Read block 2405 len 40960
22800 002/300 Read block 4240 len 32768
22801 003/300 Read block 2164 len 40960
228022024-02-12T01:39:06.691ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
228032024-02-12T01:39:06.691ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
228042024-02-12T01:39:06.691ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
228052024-02-12T01:39:06.691ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
228062024-02-12T01:39:06.691ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
228072024-02-12T01:39:06.691ZINFOcrucible: [2] client re-new 3 jobs since flush 8535 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
228082024-02-12T01:39:06.691ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
22809 004/300 Write block 1222 len 24576 data: 5 5 5 6 4 4
22810 005/300 Flush
22811 006/300 Flush
22812 007/300 Flush
22813 008/300 Write block 1896 len 36864 data: 3 3 3 4 4 2 2 2 2
22814 009/300 Flush
22815 010/300 Write block 3591 len 36864 data: 5 5 5 5 7 6 5 5 4
22816 011/300 Write block 3960 len 16384 data: 6 6 7 6
22817 012/300 Read block 3800 len 20480
22818 013/300 Read block 4241 len 16384
22819 014/300 Read block 3797 len 4096
22820 015/300 Write block 4524 len 36864 data: 3 2 4 4 4 3 3 4 6
22821 016/300 Write block 2445 len 16384 data: 9 7 7 9
22822 017/300 Read block 4238 len 40960
22823 018/300 Read block 2813 len 24576
22824 019/300 Flush
22825 020/300 Read block 4196 len 16384
22826 021/300 Write block 4514 len 12288 data: 5 5 5
22827 022/300 Write block 289 len 24576 data: 4 4 4 4 4 4
22828 023/300 Write block 4971 len 16384 data: 5 4 4 3
22829 024/300 Read block 4784 len 4096
22830 025/300 Read block 2288 len 4096
22831 026/300 Flush
22832 027/300 Flush
22833 028/300 Read block 3290 len 12288
22834 029/300 Write block 2581 len 4096 data: 6
22835 030/300 Read block 701 len 16384
22836 031/300 Flush
22837 032/300 Read block 1288 len 40960
22838 033/300 Read block 3757 len 28672
22839 034/300 Write block 29 len 16384 data: 6 6 6 5
22840 035/300 Write block 456 len 40960 data: 2 3 4 5 7 7 7 7 5 7
22841 036/300 Write block 2269 len 12288 data: 6 5 4
22842 037/300 Write block 1008 len 24576 data: 4 4 4 3 3 3
22843 038/300 Write block 599 len 20480 data: 6 4 3 4 4
22844 039/300 Read block 3617 len 12288
22845 040/300 Read block 4683 len 16384
22846 041/300 Write block 2468 len 32768 data: 6 8 9 8 9 9 8 7
22847 042/300 Read block 3476 len 16384
22848 043/300 Write block 4830 len 20480 data: 3 3 4 5 5
22849 044/300 Write block 4181 len 32768 data: 6 4 5 5 5 6 6 5
22850 045/300 Read block 330 len 16384
22851 046/300 Read block 2960 len 16384
22852 047/300 Write block 3871 len 16384 data: 5 7 7 7
22853 048/300 Read block 579 len 36864
22854 049/300 Read block 855 len 36864
22855 050/300 Read block 4266 len 8192
22856 051/300 Read block 3058 len 28672
22857 052/300 Write block 4518 len 24576 data: 5 5 5 4 4 4
22858 053/300 Write block 2081 len 20480 data: 4 4 4 4 4
22859 054/300 Read block 2341 len 4096
22860 055/300 Write block 1525 len 28672 data: 3 3 2 2 4 6 7
22861 056/300 Read block 679 len 16384
22862 057/300 Write block 4800 len 40960 data: 6 7 6 4 3 3 3 2 2 3
22863 058/300 Read block 2621 len 12288
22864 059/300 Read block 1536 len 32768
22865 060/300 Write block 2416 len 20480 data: 4 4 3 3 3
22866 061/300 Write block 842 len 32768 data: 5 6 6 6 6 6 5 4
22867 062/300 Write block 778 len 40960 data: 5 5 6 6 8 7 7 8 8 7
22868 063/300 Write block 2888 len 36864 data: 4 4 3 3 3 3 6 7 8
22869 064/300 Read block 3978 len 36864
22870 065/300 Write block 936 len 28672 data: 8 9 8 6 5 5 6
22871 066/300 Write block 1805 len 40960 data: 7 7 8 7 8 8 6 4 3 4
22872 067/300 Flush
22873 068/300 Write block 63 len 24576 data: 5 6 6 6 5 5
22874 069/300 Read block 3365 len 16384
22875 070/300 Write block 1256 len 4096 data: 9
22876 071/300 Read block 208 len 32768
22877 072/300 Read block 3213 len 32768
22878 073/300 Flush
22879 074/300 Read block 648 len 16384
22880 075/300 Read block 3694 len 28672
22881 076/300 Write block 2161 len 12288 data: 6 6 6
22882 077/300 Read block 4906 len 12288
22883 078/300 Write block 2201 len 24576 data: 6 6 6 7 6 5
22884 079/300 Read block 2279 len 12288
22885 080/300 Write block 3046 len 24576 data: 5 5 5 3 3 2
22886 081/300 Write block 2779 len 4096 data: 7
22887 082/300 Write block 4024 len 40960 data: 9 7 7 7 6 7 6 4 3 5
22888 083/300 Write block 799 len 12288 data: 10 10 9
22889 084/300 Read block 3264 len 20480
22890 085/300 Flush
22891 086/300 Read block 1043 len 8192
22892 087/300 Read block 4917 len 36864
22893 088/300 Read block 1594 len 8192
22894 089/300 Read block 1449 len 32768
22895 090/300 Read block 3954 len 40960
22896 091/300 Read block 3158 len 32768
22897 092/300 Read block 934 len 36864
22898 093/300 Read block 1775 len 8192
22899 094/300 Write block 3138 len 4096 data: 4
22900 095/300 Write block 2667 len 28672 data: 5 4 4 4 3 3 3
22901 096/300 Flush
22902 097/300 Write block 4177 len 12288 data: 6 6 7
22903 098/300 Write block 3500 len 16384 data: 5 3 3 3
22904 099/300 Write block 1139 len 32768 data: 6 8 7 9 8 9 7 9
22905 100/300 Write block 4402 len 24576 data: 4 5 6 6 5 5
22906 101/300 Write block 3193 len 24576 data: 8 6 5 6 5 4
22907 102/300 Read block 1417 len 24576
22908 103/300 Flush
22909 104/300 Write block 0 len 12288 data: 2 2 3
22910 105/300 Write block 3987 len 4096 data: 5
22911 106/300 Read block 1484 len 4096
22912 107/300 Write block 2052 len 8192 data: 4 3
22913 108/300 Read block 2494 len 12288
22914 109/300 Flush
22915 110/300 Read block 2721 len 40960
22916 111/300 Read block 1615 len 20480
22917 112/300 Read block 812 len 32768
22918 113/300 Read block 2049 len 4096
22919 114/300 Flush
22920 115/300 Read block 983 len 20480
22921 116/300 Read block 2571 len 4096
22922 117/300 Flush
22923 118/300 Read block 6 len 4096
22924 119/300 Read block 3720 len 4096
22925 120/300 Read block 4817 len 24576
22926 121/300 Flush
22927 122/300 Read block 2426 len 12288
22928 123/300 Write block 3319 len 16384 data: 4 4 5 4
22929 124/300 Read block 449 len 8192
22930 125/300 Write block 4864 len 4096 data: 6
22931 126/300 Flush
22932 127/300 Read block 2163 len 20480
22933 128/300 Write block 3177 len 12288 data: 7 7 8
22934 129/300 Flush
22935 130/300 Read block 4111 len 32768
22936 131/300 Read block 736 len 20480
22937 132/300 Write block 4961 len 4096 data: 8
22938 133/300 Write block 4427 len 8192 data: 5 4
22939 134/300 Write block 2350 len 8192 data: 7 6
22940 135/300 Read block 1605 len 20480
22941 136/300 Write block 2215 len 40960 data: 9 9 10 7 8 7 7 6 5 4
22942 137/300 Read block 3457 len 16384
22943 138/300 Write block 22 len 8192 data: 5 4
22944 139/300 Write block 3119 len 36864 data: 6 7 6 6 6 6 6 5 7
22945 140/300 Flush
22946 141/300 Read block 32 len 24576
22947 142/300 Read block 1497 len 16384
22948 143/300 Flush
22949 144/300 Read block 4027 len 12288
22950 145/300 Read block 4342 len 20480
22951 146/300 Read block 3662 len 28672
22952 147/300 Read block 3403 len 8192
22953 148/300 Read block 1717 len 28672
22954 149/300 Write block 3776 len 40960 data: 9 8 5 7 7 6 5 6 5 4
22955 150/300 Read block 4043 len 28672
22956 151/300 Read block 3956 len 24576
22957 152/300 Read block 733 len 36864
22958 153/300 Read block 3094 len 24576
22959 154/300 Read block 788 len 36864
22960 155/300 Read block 4992 len 32768
22961 156/300 Read block 4512 len 20480
22962 157/300 Write block 2026 len 36864 data: 4 4 4 4 4 4 3 3 2
22963 158/300 Write block 4470 len 12288 data: 7 6 7
22964 159/300 Flush
22965 160/300 Read block 284 len 20480
22966 161/300 Flush
22967 162/300 Write block 1206 len 8192 data: 6 7
22968 163/300 Flush
22969 164/300 Write block 4949 len 40960 data: 4 7 5 5 4 4 3 6 8 9
22970 165/300 Write block 4319 len 8192 data: 10 7
22971 166/300 Write block 4734 len 40960 data: 5 5 6 7 6 5 5 5 5 4
22972 167/300 Flush
22973 168/300 Write block 3416 len 8192 data: 4 4
22974 169/300 Write block 2868 len 28672 data: 3 4 5 6 6 6 6
22975 170/300 Read block 1795 len 4096
22976 171/300 Write block 3009 len 28672 data: 7 6 6 6 4 2 3
22977 172/300 Read block 3361 len 20480
22978 173/300 Write block 3297 len 36864 data: 3 4 5 5 4 4 3 4 4
22979 174/300 Read block 3752 len 28672
22980 175/300 Read block 4751 len 8192
22981 176/300 Read block 3097 len 12288
22982 177/300 Write block 1814 len 20480 data: 5 4 4 5 4
22983 178/300 Read block 1921 len 20480
22984 179/300 Flush
22985 180/300 Flush
22986 181/300 Read block 4676 len 32768
22987 182/300 Write block 4948 len 12288 data: 2 5 8
22988 183/300 Read block 1007 len 32768
22989 184/300 Write block 3669 len 24576 data: 4 4 3 3 6 6
22990 185/300 Read block 4564 len 40960
22991 186/300 Read block 4769 len 36864
22992 187/300 Read block 4965 len 8192
22993 188/300 Read block 1086 len 40960
22994 189/300 Write block 475 len 20480 data: 4 5 5 4 3
22995 190/300 Write block 4785 len 32768 data: 6 8 7 5 5 4 4 4
22996 191/300 Write block 3530 len 40960 data: 6 6 5 5 4 6 6 6 4 5
22997 192/300 Write block 4803 len 20480 data: 5 4 4 4 3
22998 193/300 Read block 1752 len 40960
22999 194/300 Read block 1083 len 4096
23000 195/300 Write block 2832 len 4096 data: 6
23001 196/300 Read block 392 len 36864
23002 197/300 Read block 3735 len 8192
23003 198/300 Read block 4851 len 32768
23004 199/300 Write block 4034 len 40960 data: 5 4 3 3 4 6 6 4 3 3
23005 200/300 Write block 2013 len 4096 data: 8
23006 201/300 Read block 1362 len 4096
23007 202/300 Read block 4283 len 28672
23008 203/300 Read block 2989 len 24576
23009 204/300 Write block 3717 len 8192 data: 7 7
23010 205/300 Read block 2372 len 20480
23011 206/300 Write block 3387 len 16384 data: 4 5 6 7
23012 207/300 Write block 2823 len 36864 data: 7 8 9 6 5 5 10 8 7
23013 208/300 Write block 1371 len 4096 data: 4
23014 209/300 Write block 66 len 8192 data: 7 6
23015 210/300 Read block 3083 len 32768
23016 211/300 Read block 2933 len 40960
23017 212/300 Flush
23018 213/300 Write block 2692 len 12288 data: 5 5 5
23019 214/300 Write block 1454 len 20480 data: 5 5 5 5 5
23020 215/300 Read block 3413 len 12288
23021 216/300 Write block 1654 len 8192 data: 4 4
23022 217/300 Read block 2520 len 12288
23023 218/300 Read block 62 len 40960
23024 219/300 Write block 1237 len 32768 data: 5 5 5 3 4 4 5 3
23025 220/300 Write block 2210 len 20480 data: 5 8 7 9 8
23026 221/300 Read block 1536 len 4096
23027 222/300 Write block 3473 len 24576 data: 5 5 5 3 5 6
23028 223/300 Flush
23029 224/300 Read block 621 len 36864
23030 225/300 Read block 476 len 40960
23031 226/300 Write block 40 len 4096 data: 4
23032 227/300 Read block 1687 len 40960
23033 228/300 Read block 1224 len 20480
23034 229/300 Read block 1772 len 8192
23035 230/300 Write block 541 len 4096 data: 5
23036 231/300 Write block 4269 len 20480 data: 8 8 8 8 8
23037 232/300 Flush
23038 233/300 Read block 3312 len 8192
23039 234/300 Write block 3700 len 4096 data: 7
23040 235/300 Read block 4239 len 8192
23041 236/300 Write block 1255 len 36864 data: 6 10 7 8 8 10 10 8 7
23042 237/300 Write block 704 len 36864 data: 7 7 7 5 5 5 5 7 6
23043 238/300 Read block 633 len 32768
23044 239/300 Write block 1432 len 12288 data: 5 5 4
23045 240/300 Write block 1627 len 12288 data: 4 5 4
23046 241/300 Read block 4793 len 40960
23047 242/300 Write block 802 len 4096 data: 11
23048 243/300 Read block 2678 len 36864
23049 244/300 Write block 3816 len 24576 data: 5 4 5 7 6 6
23050 245/300 Write block 2918 len 16384 data: 5 5 5 5
23051 246/300 Read block 3979 len 36864
23052 247/300 Write block 965 len 20480 data: 2 2 4 7 8
23053 248/300 Read block 1954 len 20480
23054 249/300 Write block 4400 len 16384 data: 3 4 5 6
23055 250/300 Read block 55 len 8192
23056 251/300 Read block 3606 len 32768
23057 252/300 Write block 3129 len 40960 data: 7 7 6 5 5 5 4 3 3 5
23058 253/300 Write block 563 len 20480 data: 3 4 5 4 3
23059 254/300 Read block 1093 len 4096
23060 255/300 Read block 868 len 28672
23061 256/300 Write block 1731 len 12288 data: 5 4 4
23062 257/300 Read block 4675 len 24576
23063 258/300 Write block 2481 len 36864 data: 6 8 7 7 6 6 6 7 8
23064 259/300 Read block 4615 len 36864
23065 260/300 Write block 775 len 8192 data: 4 4
23066 261/300 Write block 124 len 8192 data: 6 7
23067 262/300 Read block 4051 len 32768
23068 263/300 Flush
23069 264/300 Read block 2330 len 28672
23070 265/300 Write block 2656 len 36864 data: 5 5 5 5 4 4 4 5 5
23071 266/300 Read block 2746 len 32768
23072 267/300 Write block 3214 len 28672 data: 3 3 3 3 3 4 6
23073 268/300 Write block 917 len 20480 data: 6 6 7 7 6
23074 269/300 Write block 3412 len 8192 data: 3 4
23075 270/300 Read block 2554 len 28672
23076 271/300 Read block 3664 len 40960
23077 272/300 Write block 3178 len 20480 data: 8 9 10 10 8
23078 273/300 Read block 1124 len 8192
23079 274/300 Write block 426 len 32768 data: 3 2 2 3 3 3 3 5
23080 275/300 Read block 2913 len 36864
23081 276/300 Write block 3110 len 16384 data: 5 5 6 7
23082 277/300 Read block 2125 len 8192
23083 278/300 Read block 3388 len 28672
23084 279/300 Write block 639 len 28672 data: 6 5 6 6 4 5 6
23085 280/300 Write block 1335 len 8192 data: 7 5
23086 281/300 Write block 2427 len 12288 data: 7 6 5
23087 282/300 Flush
23088 283/300 Read block 10 len 28672
23089 284/300 Read block 3898 len 28672
23090 285/300 Read block 3802 len 40960
23091 286/300 Write block 4053 len 32768 data: 4 5 5 5 5 3 3 3
23092 287/300 Read block 3206 len 24576
23093 288/300 Read block 3689 len 32768
23094 289/300 Read block 2689 len 12288
23095 290/300 Write block 4538 len 16384 data: 9 9 7 7
23096 291/300 Write block 3874 len 28672 data: 8 7 5 7 8 7 6
23097 292/300 Flush
23098 293/300 Read block 161 len 20480
23099 294/300 Read block 1490 len 24576
23100 295/300 Read block 3897 len 16384
23101 296/300 Read block 2629 len 24576
23102 297/300 Flush
23103 298/300 Read block 4641 len 12288
23104 299/300 Flush
23105 300/300 Write block 4831 len 24576 data: 4 5 6 6 6 6
23106 Main task has work to do, go find it
23107 got dsc Start(2)
23108 start 2
23109 Feb 12 01:39:07.035 INFO request completed, latency_us: 220, response_code: 204, uri: /start/cid/2, method: POST, req_id: 42ad31a7-cad6-46b6-9a29-b561158d7463, remote_addr: 127.0.0.1:60456, local_addr: 127.0.0.1:9998
23110 Replay: started 2, returned:Ok(())
23111 ----------------------------------------------------------------
23112 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
23113 States: Active Active Offline
23114 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
23115 7537 Acked 8536 Read 10 Done Done New true
23116 7538 Acked 8537 Read 8 Done Done New true
23117 7539 Acked 8538 Read 10 Done Done New true
23118 7540 Acked 8539 Write 6 Done Done New false
23119 7541 Acked 8540 Flush 0 Done Done New false
23120 7542 Acked 8541 Flush 0 Done Done New false
23121 7543 Acked 8542 Flush 0 Done Done New false
23122 7544 Acked 8543 Write 9 Done Done New false
23123 7545 Acked 8544 Flush 0 Done Done New false
23124 7546 Acked 8545 Write 9 Done Done New false
23125 7547 Acked 8546 Write 4 Done Done New false
23126 7548 Acked 8547 Read 5 Done Done New false
23127 7549 Acked 8548 Read 4 Done Done New false
23128 7550 Acked 8549 Read 1 Done Done New false
23129 7551 Acked 8550 Write 9 Done Done New false
23130 7552 Acked 8551 Write 4 Done Done New false
23131 7553 Acked 8552 Read 10 Done Done New false
23132 7554 Acked 8553 Read 6 Done Done New false
23133 7555 Acked 8554 Flush 0 Done Done New false
23134 7556 Acked 8555 Read 4 Done Done New false
23135 7557 Acked 8556 Write 3 Done Done New false
23136 7558 Acked 8557 Write 6 Done Done New false
23137 7559 Acked 8558 Write 4 Done Done New false
23138 7560 Acked 8559 Read 1 Done Done New false
23139 7561 Acked 8560 Read 1 Done Done New false
23140 7562 Acked 8561 Flush 0 Done Done New false
23141 7563 Acked 8562 Flush 0 Done Done New false
23142 7564 Acked 8563 Read 3 Done Done New false
23143 7565 Acked 8564 Write 1 Done Done New false
23144 7566 Acked 8565 Read 4 Done Done New false
23145 7567 Acked 8566 Flush 0 Done Done New false
23146 7568 Acked 8567 Read 10 Done Done New false
23147 7569 Acked 8568 Read 7 Done Done New false
23148 7570 Acked 8569 Write 4 Done Done New false
23149 7571 Acked 8570 Write 10 Done Done New false
23150 7572 Acked 8571 Write 3 Done Done New false
23151 7573 Acked 8572 Write 6 Done Done New false
23152 7574 Acked 8573 Write 5 Done Done New false
23153 7575 Acked 8574 Read 3 Done Done New false
23154 7576 Acked 8575 Read 4 Done Done New false
23155 7577 Acked 8576 Write 8 Done Done New false
23156 7578 Acked 8577 Read 4 Done Done New false
23157 7579 Acked 8578 Write 5 Done Done New false
23158 7580 Acked 8579 Write 8 Done Done New false
23159 7581 Acked 8580 Read 4 Done Done New false
23160 7582 Acked 8581 Read 4 Done Done New false
23161 7583 Acked 8582 Write 4 Done Done New false
23162 7584 Acked 8583 Read 9 Done Done New false
23163 7585 Acked 8584 Read 9 Done Done New false
23164 7586 Acked 8585 Read 2 Done Done New false
23165 7587 Acked 8586 Read 7 Done Done New false
23166 7588 Acked 8587 Write 6 Done Done New false
23167 7589 Acked 8588 Write 5 Done Done New false
23168 7590 Acked 8589 Read 1 Done Done New false
23169 7591 Acked 8590 Write 7 Done Done New false
23170 7592 Acked 8591 Read 4 Done Done New false
23171 7593 Acked 8592 Write 10 Done Done New false
23172 7594 Acked 8593 Read 3 Done Done New false
23173 7595 Acked 8594 Read 8 Done Done New false
23174 7596 Acked 8595 Write 5 Done Done New false
23175 7597 Acked 8596 Write 8 Done Done New false
23176 7598 Acked 8597 Write 10 Done Done New false
23177 7599 Acked 8598 Write 9 Done Done New false
23178 7600 Acked 8599 Read 9 Done Done New false
23179 7601 Acked 8600 Write 7 Done Done New false
23180 7602 Acked 8601 Write 10 Done Done New false
23181 7603 Acked 8602 Flush 0 Done Done New false
23182 7604 Acked 8603 Write 6 Done Done New false
23183 7605 Acked 8604 Read 4 Done Done New false
23184 7606 Acked 8605 Write 1 Done Done New false
23185 7607 Acked 8606 Read 8 Done Done New false
23186 7608 Acked 8607 Read 8 Done Done New false
23187 7609 Acked 8608 Flush 0 Done Done New false
23188 7610 Acked 8609 Read 4 Done Done New false
23189 7611 Acked 8610 Read 7 Done Done New false
23190 7612 Acked 8611 Flush 0 Done Done New false
23191 7613 Acked 8612 Write 3 Done Done New false
23192 7614 Acked 8613 Read 3 Done Done New false
23193 7615 Acked 8614 Write 6 Done Done New false
23194 7616 Acked 8615 Read 3 Done Done New false
23195 7617 Acked 8616 Write 6 Done Done New false
23196 7618 Acked 8617 Write 1 Done Done New false
23197 7619 Acked 8618 Write 10 Done Done New false
23198 7620 Acked 8619 Write 3 Done Done New false
23199 7621 Acked 8620 Read 5 Done Done New false
23200 7622 Acked 8621 Flush 0 Done Done New false
23201 7623 Acked 8622 Read 2 Done Done New false
23202 7624 Acked 8623 Read 9 Done Done New false
23203 7625 Acked 8624 Read 2 Done Done New false
23204 7626 Acked 8625 Read 8 Done Done New false
23205 7627 Acked 8626 Read 10 Done Done New false
23206 7628 Acked 8627 Read 8 Done Done New false
23207 7629 Acked 8628 Read 9 Done Done New false
23208 7630 Acked 8629 Read 2 Done Done New false
23209 7631 Acked 8630 Write 1 Done Done New false
23210 7632 Acked 8631 Write 7 Done Done New false
23211 7633 Acked 8632 Flush 0 Done Done New false
23212 7634 Acked 8633 Write 3 Done Done New false
23213 7635 Acked 8634 Write 4 Done Done New false
23214 7636 Acked 8635 Write 8 Done Done New false
23215 7637 Acked 8636 Write 6 Done Done New false
23216 7638 Acked 8637 Write 6 Done Done New false
23217 7639 Acked 8638 Read 6 Done Done New false
23218 7640 Acked 8639 Flush 0 Done Done New false
23219 7641 Acked 8640 Write 3 Done Done New false
23220 7642 Acked 8641 Write 1 Done Done New false
23221 7643 Acked 8642 Read 1 Done Done New false
23222 7644 Acked 8643 Write 2 Done Done New false
23223 7645 Acked 8644 Read 3 Done Done New false
23224 7646 Acked 8645 Flush 0 Done Done New false
23225 7647 Acked 8646 Read 10 Done Done New false
23226 7648 Acked 8647 Read 5 Done Done New false
23227 7649 Acked 8648 Read 8 Done Done New false
23228 7650 Acked 8649 Read 1 Done Done New false
23229 7651 Acked 8650 Flush 0 Done Done New false
23230 7652 Acked 8651 Read 5 Done Done New false
23231 7653 Acked 8652 Read 1 Done Done New false
23232 7654 Acked 8653 Flush 0 Done Done New false
23233 7655 Acked 8654 Read 1 Done Done New false
23234 7656 Acked 8655 Read 1 Done Done New false
23235 7657 Acked 8656 Read 6 Done Done New false
23236 7658 Acked 8657 Flush 0 Done Done New false
23237 7659 Acked 8658 Read 3 Done Done New false
23238 7660 Acked 8659 Write 4 Done Done New false
23239 7661 Acked 8660 Read 2 Done Done New false
23240 7662 Acked 8661 Write 1 Done Done New false
23241 7663 Acked 8662 Flush 0 Done Done New false
23242 7664 Acked 8663 Read 5 Done Done New false
23243 7665 Acked 8664 Write 3 Done Done New false
23244 7666 Acked 8665 Flush 0 Done Done New false
23245 7667 Acked 8666 Read 8 Done Done New false
23246 7668 Acked 8667 Read 5 Done Done New false
23247 7669 Acked 8668 Write 1 Done Done New false
23248 7670 Acked 8669 Write 2 Done Done New false
23249 7671 Acked 8670 Write 2 Done Done New false
23250 7672 Acked 8671 Read 5 Done Done New false
23251 7673 Acked 8672 Write 10 Done Done New false
23252 7674 Acked 8673 Read 4 Done Done New false
23253 7675 Acked 8674 Write 2 Done Done New false
23254 7676 Acked 8675 Write 9 Done Done New false
23255 7677 Acked 8676 Flush 0 Done Done New false
23256 7678 Acked 8677 Read 6 Done Done New false
23257 7679 Acked 8678 Read 4 Done Done New false
23258 7680 Acked 8679 Flush 0 Done Done New false
23259 7681 Acked 8680 Read 3 Done Done New false
23260 7682 Acked 8681 Read 5 Done Done New false
23261 7683 Acked 8682 Read 7 Done Done New false
23262 7684 Acked 8683 Read 2 Done Done New false
23263 7685 Acked 8684 Read 7 Done Done New false
23264 7686 Acked 8685 Write 10 Done Done New false
23265 7687 Acked 8686 Read 7 Done Done New false
23266 7688 Acked 8687 Read 6 Done Done New false
23267 7689 Acked 8688 Read 9 Done Done New false
23268 7690 Acked 8689 Read 6 Done Done New false
23269 7691 Acked 8690 Read 9 Done Done New false
23270 7692 Acked 8691 Read 8 Done Done New false
23271 7693 Acked 8692 Read 5 Done Done New false
23272 7694 Acked 8693 Write 9 Done Done New false
23273 7695 Acked 8694 Write 3 Done Done New false
23274 7696 Acked 8695 Flush 0 Done Done New false
23275 7697 Acked 8696 Read 5 Done Done New false
23276 7698 Acked 8697 Flush 0 Done Done New false
23277 7699 Acked 8698 Write 2 Done Done New false
23278 7700 Acked 8699 Flush 0 Done Done New false
23279 7701 Acked 8700 Write 10 Done Done New false
23280 7702 Acked 8701 Write 2 Done Done New false
23281 7703 Acked 8702 Write 10 Done Done New false
23282 7704 Acked 8703 Flush 0 Done Done New false
23283 7705 Acked 8704 Write 2 Done Done New false
23284 7706 Acked 8705 Write 7 Done Done New false
23285 7707 Acked 8706 Read 1 Done Done New false
23286 7708 Acked 8707 Write 7 Done Done New false
23287 7709 Acked 8708 Read 5 Done Done New false
23288 7710 Acked 8709 Write 9 Done Done New false
23289 7711 Acked 8710 Read 7 Done Done New false
23290 7712 Acked 8711 Read 2 Done Done New false
23291 7713 Acked 8712 Read 3 Done Done New false
23292 7714 Acked 8713 Write 5 Done Done New false
23293 7715 Acked 8714 Read 5 Done Done New false
23294 7716 Acked 8715 Flush 0 Done Done New false
23295 7717 Acked 8716 Flush 0 Done Done New false
23296 7718 Acked 8717 Read 8 Done Done New false
23297 7719 Acked 8718 Write 3 Done Done New false
23298 7720 Acked 8719 Read 8 Done Done New false
23299 7721 Acked 8720 Write 6 Done Done New false
23300 7722 Acked 8721 Read 10 Done Done New false
23301 7723 Acked 8722 Read 9 Done Done New false
23302 7724 Acked 8723 Read 2 Done Done New false
23303 7725 Acked 8724 Read 10 Done Done New false
23304 7726 Acked 8725 Write 5 Done Done New false
23305 7727 Acked 8726 Write 8 Done Done New false
23306 7728 Acked 8727 Write 10 Done Done New false
23307 7729 Acked 8728 Write 5 Done Done New false
23308 7730 Acked 8729 Read 10 Done Done New false
23309 7731 Acked 8730 Read 1 Done Done New false
23310 7732 Acked 8731 Write 1 Done Done New false
23311 7733 Acked 8732 Read 9 Done Done New false
23312 7734 Acked 8733 Read 2 Done Done New false
23313 7735 Acked 8734 Read 8 Done Done New false
23314 7736 Acked 8735 Write 10 Done Done New false
23315 7737 Acked 8736 Write 1 Done Done New false
23316 7738 Acked 8737 Read 1 Done Done New false
23317 7739 Acked 8738 Read 7 Done Done New false
23318 7740 Acked 8739 Read 6 Done Done New false
23319 7741 Acked 8740 Write 2 Done Done New false
23320 7742 Acked 8741 Read 5 Done Done New false
23321 7743 Acked 8742 Write 4 Done Done New false
23322 7744 Acked 8743 Write 9 Done Done New false
23323 7745 Acked 8744 Write 1 Done Done New false
23324 7746 Acked 8745 Write 2 Done Done New false
23325 7747 Acked 8746 Read 8 Done Done New false
23326 7748 Acked 8747 Read 10 Done Done New false
23327 7749 Acked 8748 Flush 0 Done Done New false
23328 7750 Acked 8749 Write 3 Done Done New false
23329 7751 Acked 8750 Write 5 Done Done New false
23330 7752 Acked 8751 Read 3 Done Done New false
23331 7753 Acked 8752 Write 2 Done Done New false
23332 7754 Acked 8753 Read 3 Done Done New false
23333 7755 Acked 8754 Read 10 Done Done New false
23334 7756 Acked 8755 Write 8 Done Done New false
23335 7757 Acked 8756 Write 5 Done Done New false
23336 7758 Acked 8757 Read 1 Done Done New false
23337 7759 Acked 8758 Write 6 Done Done New false
23338 7760 Acked 8759 Flush 0 Done Done New false
23339 7761 Acked 8760 Read 9 Done Done New false
23340 7762 Acked 8761 Read 10 Done Done New false
23341 7763 Acked 8762 Write 1 Done Done New false
23342 7764 Acked 8763 Read 10 Done Done New false
23343 7765 Acked 8764 Read 5 Done Done New false
23344 7766 Acked 8765 Read 2 Done Done New false
23345 7767 Acked 8766 Write 1 Done Done New false
23346 7768 Acked 8767 Write 5 Done Done New false
23347 7769 Acked 8768 Flush 0 Done Done New false
23348 7770 Acked 8769 Read 2 Done Done New false
23349 7771 Acked 8770 Write 1 Done Done New false
23350 7772 Acked 8771 Read 2 Done Done New false
23351 7773 Acked 8772 Write 9 Done Done New false
23352 7774 Acked 8773 Write 9 Done Done New false
23353 7775 Acked 8774 Read 8 Done Done New false
23354 7776 Acked 8775 Write 3 Done Done New false
23355 7777 Acked 8776 Write 3 Done Done New false
23356 7778 Acked 8777 Read 10 Done Done New false
23357 7779 Acked 8778 Write 1 Done Done New false
23358 7780 Acked 8779 Read 9 Done Done New false
23359 7781 Acked 8780 Write 6 Done Done New false
23360 7782 Acked 8781 Write 4 Done Done New false
23361 7783 Acked 8782 Read 9 Done Done New false
23362 7784 Acked 8783 Write 5 Done Done New false
23363 7785 Acked 8784 Read 5 Done Done New false
23364 7786 Acked 8785 Write 4 Done Done New false
23365 7787 Acked 8786 Read 2 Done Done New false
23366 7788 Acked 8787 Read 8 Done Done New false
23367 7789 Acked 8788 Write 10 Done Done New false
23368 7790 Acked 8789 Write 5 Done Done New false
23369 7791 Acked 8790 Read 1 Done Done New false
23370 7792 Acked 8791 Read 7 Done Done New false
23371 7793 Acked 8792 Write 3 Done Done New false
23372 7794 Acked 8793 Read 6 Done Done New false
23373 7795 Acked 8794 Write 9 Done Done New false
23374 7796 Acked 8795 Read 9 Done Done New false
23375 7797 Acked 8796 Write 2 Done Done New false
23376 7798 Acked 8797 Write 2 Done Done New false
23377 7799 Acked 8798 Read 8 Done Done New false
23378 7800 Acked 8799 Flush 0 Done Done New false
23379 7801 Acked 8800 Read 7 Done Done New false
23380 7802 Acked 8801 Write 9 Done Done New false
23381 7803 Acked 8802 Read 8 Done Done New false
23382 7804 Acked 8803 Write 7 Done Done New false
23383 7805 Acked 8804 Write 5 Done Done New false
23384 7806 Acked 8805 Write 2 Done Done New false
23385 7807 Acked 8806 Read 7 Done Done New false
23386 7808 Acked 8807 Read 10 Done Done New false
23387 7809 Acked 8808 Write 5 Done Done New false
23388 7810 Acked 8809 Read 2 Done Done New false
23389 7811 Acked 8810 Write 8 Done Done New false
23390 7812 Acked 8811 Read 9 Done Done New false
23391 7813 Acked 8812 Write 4 Done Done New false
23392 7814 Acked 8813 Read 2 Done Done New false
23393 7815 Acked 8814 Read 7 Done Done New false
23394 7816 Acked 8815 Write 7 Done Done New false
23395 7817 Acked 8816 Write 2 Done Done New false
23396 7818 Acked 8817 Write 3 Done Done New false
23397 7819 Acked 8818 Flush 0 Done Done New false
23398 7820 Acked 8819 Read 7 Done Done New false
23399 7821 Acked 8820 Read 7 Done Done New false
23400 7822 Acked 8821 Read 10 Done Done New false
23401 7823 Acked 8822 Write 8 Done Done New false
23402 7824 Acked 8823 Read 6 Done Done New false
23403 7825 Acked 8824 Read 8 Done Done New false
23404 7826 Acked 8825 Read 3 Done Done New false
23405 7827 Acked 8826 Write 4 Done Done New false
23406 7828 Acked 8827 Write 7 Done Done New false
23407 7829 Acked 8828 Flush 0 Done Done New false
23408 7830 Acked 8829 Read 5 Done Done New false
23409 7831 Acked 8830 Read 6 Done Done New false
23410 7832 Acked 8831 Read 4 Done Done New false
23411 7833 Acked 8832 Read 6 Done Done New false
23412 7834 Acked 8833 Flush 0 Done Done New false
23413 7835 Acked 8834 Read 3 Done Done New false
23414 7836 Acked 8835 Flush 0 Done Done New false
23415 7837 Acked 8836 Write 6 Done Done New false
23416 STATES DS:0 DS:1 DS:2 TOTAL
23417 New 0 0 301 301
23418 Sent 0 0 0 0
23419 Done 301 301 0 602
23420 Skipped 0 0 0 0
23421 Error 0 0 0 0
23422 Last Flush: 8835 8835 8535
23423 Downstairs last five completed: 8535 8534 8533 8532 8531
23424 Upstairs last five completed: 7837 7836 7835 7834 7833
23425 CLIENT: Up:0 ds:301 act:2
234262024-02-12T01:39:07.692ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
234272024-02-12T01:39:07.692ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
234282024-02-12T01:39:07.692ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
234292024-02-12T01:39:07.692ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
234302024-02-12T01:39:07.692ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
234312024-02-12T01:39:07.693ZINFOcrucible: [2] client re-new 302 jobs since flush 8535 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
234322024-02-12T01:39:07.693ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
23433 [8830] got start action
23434 [8830] I am going to restart
23435 Starting downstairs at port 8830
234362024-02-12T01:39:08.694ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
234372024-02-12T01:39:08.694ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
23438 Make output file at "/tmp/dsc/downstairs-8830.txt"
234392024-02-12T01:39:08.694ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
234402024-02-12T01:39:08.694ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
23441 [8830][2] reports Starting
234422024-02-12T01:39:08.694ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
234432024-02-12T01:39:08.694ZINFOcrucible: [2] client re-new 302 jobs since flush 8535 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
234442024-02-12T01:39:08.694ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
23445 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1190)
23446 [8830][2] reports Running
234472024-02-12T01:39:09.695ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
234482024-02-12T01:39:09.695ZINFOcrucible: ds_connection connected = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
234492024-02-12T01:39:09.697ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
234502024-02-12T01:39:09.697ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
234512024-02-12T01:39:09.697ZINFOcrucible: send last flush ID to this DS: 8535 = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
234522024-02-12T01:39:09.697ZINFOcrucible: Replied this last flush ID: 8535 = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
234532024-02-12T01:39:09.697ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
234542024-02-12T01:39:09.697ZINFOcrucible: [2] Transition from Offline to Replay = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
234552024-02-12T01:39:09.697ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
234562024-02-12T01:39:09.697ZINFOcrucible: [2] Transition from Replay to Active = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
23457 ----------------------------------------------------------------
23458 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
23459 Downstairs last five completed: 8837 8836 8835 8834 8833
23460 Upstairs last five completed: 7838 7837 7836 7835 7834
23461 CLIENT: Up:0 ds:0 act:3
23462 Replay: All jobs finished, all DS active.
23463 Main task has work to do, go find it
23464 got dsc Stop(0)
23465 stop 0
23466 [8810] Got stop action so:false kr:false
23467 Feb 12 01:39:11.039 INFO request completed, latency_us: 391, response_code: 204, uri: /stop/cid/0, method: POST, req_id: 3683892c-9bbb-4b50-a55b-0e7d4c2766d4, remote_addr: 127.0.0.1:60456, local_addr: 127.0.0.1:9998
234682024-02-12T01:39:11.040ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
234692024-02-12T01:39:11.040ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
234702024-02-12T01:39:11.040ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
234712024-02-12T01:39:11.040ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
23472 Feb 12 01:39:11.040 INFO request completed, latency_us: 178, response_code: 200, uri: /state/cid/0, method: GET, req_id: 2a33ad1b-84fc-4cfd-8955-e3ec53dbd053, remote_addr: 127.0.0.1:60456, local_addr: 127.0.0.1:9998
234732024-02-12T01:39:11.040ZINFOcrucible: [0] client re-new 0 jobs since flush 8837 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
234742024-02-12T01:39:11.040ZINFOcrucible: client task is exiting = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
23475 [8810] Exited with: ExitStatus(unix_wait_status(9))
23476 [8810][0] reports Exit
234772024-02-12T01:39:12.041ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
234782024-02-12T01:39:12.041ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
234792024-02-12T01:39:12.041ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
234802024-02-12T01:39:12.041ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
234812024-02-12T01:39:12.041ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
234822024-02-12T01:39:12.041ZINFOcrucible: [0] client re-new 0 jobs since flush 8837 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
234832024-02-12T01:39:12.041ZINFOcrucible: client task is exiting = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
234842024-02-12T01:39:13.042ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
234852024-02-12T01:39:13.042ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
234862024-02-12T01:39:13.042ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
234872024-02-12T01:39:13.042ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
234882024-02-12T01:39:13.043ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
234892024-02-12T01:39:13.043ZINFOcrucible: [0] client re-new 0 jobs since flush 8837 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
234902024-02-12T01:39:13.043ZINFOcrucible: client task is exiting = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
234912024-02-12T01:39:14.043ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
234922024-02-12T01:39:14.044ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
234932024-02-12T01:39:14.044ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
234942024-02-12T01:39:14.044ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
234952024-02-12T01:39:14.044ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
234962024-02-12T01:39:14.044ZINFOcrucible: [0] client re-new 0 jobs since flush 8837 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
234972024-02-12T01:39:14.044ZINFOcrucible: client task is exiting = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
23498 Feb 12 01:39:15.042 INFO request completed, latency_us: 210, response_code: 200, uri: /state/cid/0, method: GET, req_id: 56d459f6-2999-4b0c-8cb6-479e9e0421ee, remote_addr: 127.0.0.1:60456, local_addr: 127.0.0.1:9998
23499 001/300 Write block 1557 len 12288 data: 6 5 5
23500 002/300 Read block 2235 len 8192
23501 003/300 Write block 4104 len 32768 data: 4 3 3 3 3 2 2 3
23502 004/300 Read block 1149 len 24576
235032024-02-12T01:39:15.045ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
235042024-02-12T01:39:15.045ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
235052024-02-12T01:39:15.045ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
235062024-02-12T01:39:15.045ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
235072024-02-12T01:39:15.045ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
235082024-02-12T01:39:15.045ZINFOcrucible: [0] client re-new 4 jobs since flush 8837 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
235092024-02-12T01:39:15.046ZINFOcrucible: client task is exiting = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
23510 005/300 Write block 1924 len 32768 data: 5 5 5 3 2 2 2 2
23511 006/300 Write block 484 len 32768 data: 5 6 8 7 6 6 6 4
23512 007/300 Write block 27 len 20480 data: 5 7 7 7 7
23513 008/300 Write block 3063 len 40960 data: 5 3 3 2 2 2 2 2 2 2
23514 009/300 Read block 4863 len 40960
23515 010/300 Read block 2547 len 32768
23516 011/300 Read block 708 len 20480
23517 012/300 Read block 4056 len 28672
23518 013/300 Write block 1701 len 4096 data: 5
23519 014/300 Write block 1697 len 12288 data: 5 4 5
23520 015/300 Write block 1439 len 32768 data: 4 4 4 5 5 5 5 5
23521 016/300 Write block 981 len 8192 data: 5 5
23522 017/300 Read block 1222 len 12288
23523 018/300 Write block 1982 len 32768 data: 5 3 3 4 3 3 3 4
23524 019/300 Write block 2443 len 24576 data: 6 10 10 8 8 10
23525 020/300 Write block 854 len 8192 data: 5 4
23526 021/300 Read block 1611 len 40960
23527 022/300 Read block 4087 len 24576
23528 023/300 Read block 227 len 4096
23529 024/300 Write block 50 len 24576 data: 3 4 3 5 4 5
23530 025/300 Read block 3590 len 4096
23531 026/300 Read block 3821 len 8192
23532 027/300 Read block 4950 len 40960
23533 028/300 Read block 1570 len 28672
23534 029/300 Read block 4930 len 32768
23535 030/300 Flush
23536 031/300 Write block 2493 len 8192 data: 5 4
23537 032/300 Write block 430 len 36864 data: 4 4 4 6 6 7 7 5 5
23538 033/300 Write block 2838 len 20480 data: 7 8 6 6 5
23539 034/300 Read block 3724 len 4096
23540 035/300 Write block 1807 len 8192 data: 9 8
23541 036/300 Read block 4181 len 12288
23542 037/300 Read block 4438 len 40960
23543 038/300 Write block 1297 len 24576 data: 3 3 4 4 4 5
23544 039/300 Write block 505 len 20480 data: 4 4 4 5 5
23545 040/300 Write block 3290 len 8192 data: 2 2
23546 041/300 Write block 1846 len 8192 data: 7 9
23547 042/300 Write block 1224 len 20480 data: 6 7 5 5 3
23548 043/300 Flush
23549 044/300 Read block 4604 len 4096
23550 045/300 Flush
23551 046/300 Read block 679 len 12288
23552 047/300 Flush
23553 048/300 Read block 945 len 40960
23554 049/300 Write block 3293 len 20480 data: 3 4 4 3 4
23555 050/300 Read block 1493 len 8192
23556 051/300 Flush
23557 052/300 Write block 1893 len 24576 data: 5 4 5 4 4 4
23558 053/300 Flush
23559 054/300 Write block 1487 len 12288 data: 5 5 4
23560 055/300 Flush
23561 056/300 Write block 3789 len 24576 data: 3 5 8 8 9 7
23562 057/300 Write block 679 len 16384 data: 7 7 7 5
23563 058/300 Write block 2895 len 24576 data: 8 9 10 10 8 7
23564 059/300 Read block 745 len 32768
23565 060/300 Flush
23566 061/300 Write block 15 len 36864 data: 5 5 6 9 8 6 5 6 5
23567 062/300 Write block 4776 len 8192 data: 6 5
23568 063/300 Read block 915 len 24576
23569 064/300 Read block 250 len 28672
23570 065/300 Read block 2284 len 40960
23571 066/300 Write block 785 len 16384 data: 9 9 8 7
23572 067/300 Read block 4444 len 28672
23573 068/300 Write block 743 len 40960 data: 7 7 9 8 9 9 10 9 7 7
23574 069/300 Read block 2779 len 32768
23575 070/300 Read block 2643 len 4096
23576 071/300 Write block 165 len 32768 data: 5 4 5 5 5 4 3 4
23577 072/300 Write block 3522 len 8192 data: 7 5
23578 073/300 Write block 3982 len 12288 data: 6 4 4
23579 074/300 Read block 3680 len 20480
23580 075/300 Write block 2203 len 28672 data: 7 8 7 6 6 7 6
23581 076/300 Read block 3709 len 28672
23582 077/300 Flush
23583 078/300 Read block 2770 len 16384
23584 079/300 Read block 2716 len 20480
23585 080/300 Write block 1377 len 12288 data: 5 6 6
23586 081/300 Write block 691 len 8192 data: 4 4
23587 082/300 Read block 3467 len 12288
23588 083/300 Write block 2936 len 28672 data: 4 4 3 3 3 3 3
23589 084/300 Read block 948 len 20480
23590 085/300 Write block 2342 len 20480 data: 7 6 7 6 5
23591 086/300 Write block 2094 len 28672 data: 6 6 4 5 5 7 6
23592 087/300 Write block 2977 len 32768 data: 6 7 6 5 5 5 5 5
23593 088/300 Write block 2885 len 4096 data: 8
23594 089/300 Flush
23595 090/300 Write block 2475 len 8192 data: 8 7
23596 091/300 Write block 2907 len 20480 data: 4 4 4 4 3
23597 092/300 Write block 3785 len 4096 data: 5
23598 093/300 Write block 4604 len 24576 data: 6 7 7 8 7 6
23599 094/300 Write block 3623 len 8192 data: 6 6
23600 095/300 Read block 2039 len 32768
23601 096/300 Read block 1701 len 28672
23602 097/300 Read block 4698 len 12288
23603 098/300 Read block 849 len 28672
23604 099/300 Read block 4464 len 28672
23605 100/300 Write block 1075 len 36864 data: 7 6 7 8 6 7 6 3 4
23606 101/300 Read block 2953 len 12288
23607 102/300 Write block 1111 len 36864 data: 4 3 3 3 2 3 3 4 6
23608 103/300 Read block 2827 len 24576
23609 104/300 Read block 208 len 12288
23610 105/300 Write block 1281 len 24576 data: 4 2 4 3 4 4
23611 106/300 Read block 4201 len 28672
23612 107/300 Flush
23613 108/300 Write block 2295 len 36864 data: 7 5 6 6 5 6 6 6 5
23614 109/300 Read block 685 len 28672
23615 110/300 Read block 2815 len 20480
23616 111/300 Write block 1483 len 12288 data: 8 6 6
23617 112/300 Read block 432 len 8192
23618 113/300 Flush
23619 114/300 Write block 493 len 20480 data: 5 7 8 8 7
23620 115/300 Flush
23621 116/300 Write block 4118 len 24576 data: 7 7 4 4 5 4
23622 117/300 Read block 1206 len 20480
23623 118/300 Write block 837 len 20480 data: 3 2 3 3 4
23624 119/300 Write block 2383 len 40960 data: 7 5 5 3 2 2 2 3 3 3
23625 120/300 Read block 619 len 24576
23626 121/300 Write block 3260 len 32768 data: 3 3 3 3 3 3 3 3
23627 122/300 Write block 811 len 32768 data: 3 4 4 3 4 5 5 6
23628 123/300 Read block 4918 len 12288
23629 124/300 Read block 3239 len 24576
23630 125/300 Write block 3897 len 20480 data: 8 6 7 6 5
23631 126/300 Write block 3632 len 20480 data: 5 4 3 4 4
23632 127/300 Read block 739 len 40960
23633 128/300 Read block 2170 len 16384
23634 129/300 Flush
23635 130/300 Read block 3175 len 24576
23636 131/300 Flush
23637 132/300 Read block 684 len 12288
23638 133/300 Read block 2107 len 24576
23639 134/300 Write block 3154 len 12288 data: 6 6 5
23640 135/300 Flush
23641 136/300 Write block 1469 len 28672 data: 4 3 3 4 5 5 5
23642 137/300 Write block 484 len 36864 data: 6 7 9 8 7 7 7 5 5
23643 138/300 Write block 1267 len 40960 data: 4 3 5 4 5 6 6 6 7 8
23644 139/300 Read block 3572 len 8192
23645 140/300 Write block 4708 len 36864 data: 4 3 3 3 4 6 7 7 6
23646 141/300 Write block 2985 len 20480 data: 4 5 4 4 4
23647 142/300 Read block 1307 len 36864
23648 143/300 Flush
23649 144/300 Read block 4278 len 24576
23650 145/300 Write block 1800 len 12288 data: 3 4 4
23651 146/300 Write block 486 len 12288 data: 10 9 8
23652 147/300 Flush
23653 148/300 Read block 3749 len 36864
23654 149/300 Write block 2607 len 8192 data: 5 6
23655 150/300 Read block 4935 len 4096
23656 151/300 Read block 4020 len 36864
23657 152/300 Write block 2784 len 4096 data: 5
23658 153/300 Write block 455 len 32768 data: 3 3 4 5 6 8 8 8
23659 154/300 Write block 2508 len 40960 data: 5 6 5 4 4 3 3 3 4 3
23660 155/300 Write block 3620 len 8192 data: 8 7
23661 156/300 Write block 1093 len 12288 data: 6 7 5
23662 157/300 Write block 3355 len 32768 data: 3 2 2 4 5 5 6 7
23663 158/300 Flush
23664 159/300 Read block 2676 len 16384
23665 160/300 Read block 2792 len 36864
23666 161/300 Write block 3514 len 24576 data: 6 6 5 4 5 6
23667 162/300 Write block 3185 len 32768 data: 9 10 9 9 9 9 9 8
23668 163/300 Write block 697 len 28672 data: 6 6 6 8 7 7 7
23669 164/300 Read block 2492 len 16384
23670 165/300 Read block 60 len 24576
23671 166/300 Read block 1940 len 8192
23672 167/300 Read block 889 len 40960
23673 168/300 Write block 4338 len 12288 data: 2 2 3
23674 169/300 Read block 3925 len 24576
23675 170/300 Read block 2659 len 16384
23676 171/300 Read block 3009 len 28672
23677 172/300 Write block 3035 len 40960 data: 6 7 8 9 7 7 8 9 6 6
23678 173/300 Write block 3789 len 16384 data: 4 6 9 9
23679 174/300 Write block 4762 len 40960 data: 4 4 4 5 5 5 5 5 6 6
23680 175/300 Write block 692 len 24576 data: 5 4 5 5 6 7
23681 176/300 Read block 2884 len 28672
23682 177/300 Write block 3643 len 4096 data: 5
23683 178/300 Write block 3923 len 32768 data: 6 4 3 3 3 3 3 3
23684 179/300 Write block 4929 len 4096 data: 7
23685 180/300 Write block 3509 len 40960 data: 5 5 6 5 6 7 7 6 5 6
23686 181/300 Write block 3896 len 24576 data: 8 9 7 8 7 6
23687 182/300 Read block 1874 len 8192
23688 183/300 Read block 2462 len 20480
23689 184/300 Write block 2272 len 4096 data: 6
23690 185/300 Write block 4602 len 12288 data: 7 6 7
23691 186/300 Read block 3936 len 16384
23692 187/300 Read block 1133 len 28672
23693 188/300 Read block 2106 len 16384
23694 189/300 Read block 1455 len 20480
23695 190/300 Read block 3040 len 32768
23696 191/300 Write block 137 len 8192 data: 2 3
23697 192/300 Read block 843 len 28672
23698 193/300 Read block 3879 len 4096
23699 194/300 Read block 2573 len 4096
23700 195/300 Write block 171 len 12288 data: 4 5 4
23701 196/300 Read block 2336 len 24576
23702 197/300 Read block 2868 len 24576
23703 198/300 Write block 204 len 4096 data: 5
23704 199/300 Read block 3780 len 36864
23705 200/300 Write block 1437 len 36864 data: 5 5 5 5 5 6 6 6 6
23706 201/300 Read block 3009 len 12288
23707 202/300 Write block 3289 len 12288 data: 3 3 3
23708 203/300 Write block 183 len 12288 data: 4 3 4
23709 204/300 Read block 2492 len 20480
23710 205/300 Read block 2049 len 28672
23711 206/300 Write block 2429 len 8192 data: 6 5
23712 207/300 Write block 1619 len 20480 data: 6 6 7 8 8
23713 208/300 Read block 4924 len 28672
23714 209/300 Write block 3956 len 40960 data: 6 8 6 7 7 7 8 7 7 7
23715 210/300 Write block 3800 len 32768 data: 3 4 3 3 4 5 4 3
23716 211/300 Flush
23717 212/300 Write block 3875 len 20480 data: 8 6 8 9 8
23718 213/300 Read block 1114 len 36864
23719 214/300 Read block 736 len 20480
23720 215/300 Read block 981 len 16384
23721 216/300 Read block 1733 len 32768
23722 217/300 Read block 617 len 28672
23723 218/300 Read block 772 len 24576
23724 219/300 Read block 3663 len 24576
23725 220/300 Write block 4057 len 32768 data: 6 4 4 4 4 4 4 3
23726 221/300 Read block 3820 len 28672
23727 222/300 Read block 1340 len 12288
23728 223/300 Write block 3329 len 16384 data: 5 4 4 4
23729 224/300 Flush
23730 225/300 Read block 2384 len 20480
23731 226/300 Read block 2200 len 16384
23732 227/300 Read block 1276 len 12288
23733 228/300 Flush
23734 229/300 Write block 316 len 20480 data: 5 4 4 4 5
23735 230/300 Read block 1937 len 8192
23736 231/300 Flush
23737 232/300 Write block 1053 len 16384 data: 3 3 4 3
23738 233/300 Read block 647 len 16384
23739 234/300 Write block 1072 len 40960 data: 9 8 8 8 7 8 9 7 8 7
23740 235/300 Read block 2959 len 32768
23741 236/300 Write block 1513 len 40960 data: 6 5 5 5 5 5 4 4 3 3
23742 237/300 Write block 4338 len 40960 data: 3 3 4 3 3 3 3 4 3 3
23743 238/300 Write block 855 len 4096 data: 5
23744 239/300 Read block 3276 len 28672
23745 240/300 Flush
23746 241/300 Write block 2020 len 8192 data: 5 5
23747 242/300 Read block 2618 len 36864
23748 243/300 Read block 2689 len 40960
23749 244/300 Read block 2070 len 16384
23750 245/300 Write block 2589 len 24576 data: 9 9 9 8 7 8
23751 246/300 Write block 1063 len 20480 data: 5 5 5 4 4
23752 247/300 Write block 4197 len 28672 data: 4 3 4 6 8 7 8
23753 248/300 Read block 2408 len 20480
23754 249/300 Read block 4798 len 20480
23755 250/300 Read block 1838 len 36864
23756 251/300 Read block 476 len 16384
23757 252/300 Flush
23758 253/300 Write block 740 len 8192 data: 5 6
23759 254/300 Write block 3359 len 32768 data: 6 6 7 8 7 6 6 3
23760 255/300 Write block 4911 len 40960 data: 6 7 6 7 8 8 6 7 8 7
23761 256/300 Read block 1554 len 40960
23762 257/300 Write block 2743 len 20480 data: 6 5 5 5 7
23763 258/300 Read block 4119 len 32768
23764 259/300 Read block 3114 len 20480
23765 260/300 Write block 1747 len 32768 data: 3 3 4 5 5 4 5 5
23766 261/300 Read block 1772 len 4096
23767 262/300 Read block 3311 len 24576
23768 263/300 Flush
23769 264/300 Write block 4153 len 24576 data: 3 3 3 4 5 3
23770 265/300 Read block 877 len 40960
23771 266/300 Read block 504 len 20480
23772 267/300 Write block 4550 len 16384 data: 7 8 9 7
23773 268/300 Flush
23774 269/300 Read block 2310 len 20480
23775 270/300 Flush
23776 271/300 Read block 2498 len 36864
23777 272/300 Read block 2409 len 24576
23778 273/300 Read block 2910 len 36864
23779 274/300 Read block 986 len 8192
23780 275/300 Write block 168 len 8192 data: 6 6
23781 276/300 Write block 2944 len 4096 data: 3
23782 277/300 Read block 3210 len 36864
23783 278/300 Flush
23784 279/300 Read block 1700 len 36864
23785 280/300 Flush
23786 281/300 Write block 4648 len 32768 data: 6 5 5 6 6 6 4 4
23787 282/300 Read block 375 len 28672
23788 283/300 Flush
23789 284/300 Read block 2098 len 36864
23790 285/300 Write block 2794 len 8192 data: 4 5
23791 286/300 Flush
23792 287/300 Read block 473 len 36864
23793 288/300 Read block 2956 len 28672
23794 289/300 Write block 3288 len 32768 data: 6 4 4 4 3 4 5 5
23795 290/300 Write block 1259 len 32768 data: 9 11 11 9 8 8 7 5
23796 291/300 Write block 686 len 20480 data: 6 5 5 4 3
23797 292/300 Write block 140 len 20480 data: 3 4 4 3 4
23798 293/300 Write block 671 len 40960 data: 8 9 9 8 5 5 5 7 8 8
23799 294/300 Read block 1748 len 16384
23800 295/300 Read block 2574 len 4096
23801 296/300 Write block 3829 len 32768 data: 3 3 3 3 3 3 2 2
23802 297/300 Read block 4394 len 4096
23803 298/300 Write block 4443 len 16384 data: 4 5 5 7
23804 299/300 Write block 1712 len 20480 data: 7 6 6 5 6
23805 300/300 Read block 7 len 24576
23806 Main task has work to do, go find it
23807 got dsc Start(0)
23808 start 0
23809 Feb 12 01:39:15.388 INFO request completed, latency_us: 241, response_code: 204, uri: /start/cid/0, method: POST, req_id: 28e027c3-ce67-4865-95a4-e97d21dbfa1a, remote_addr: 127.0.0.1:60456, local_addr: 127.0.0.1:9998
23810 Replay: started 0, returned:Ok(())
23811 ----------------------------------------------------------------
23812 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
23813 States: Offline Active Active
23814 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
23815 7839 Acked 8838 Write 3 New Done Done true
23816 7840 Acked 8839 Read 2 New Done Done true
23817 7841 Acked 8840 Write 8 New Done Done true
23818 7842 Acked 8841 Read 6 New Done Done true
23819 7843 Acked 8842 Write 8 New Done Done false
23820 7844 Acked 8843 Write 8 New Done Done false
23821 7845 Acked 8844 Write 5 New Done Done false
23822 7846 Acked 8845 Write 10 New Done Done false
23823 7847 Acked 8846 Read 10 New Done Done false
23824 7848 Acked 8847 Read 8 New Done Done false
23825 7849 Acked 8848 Read 5 New Done Done false
23826 7850 Acked 8849 Read 7 New Done Done false
23827 7851 Acked 8850 Write 1 New Done Done false
23828 7852 Acked 8851 Write 3 New Done Done false
23829 7853 Acked 8852 Write 8 New Done Done false
23830 7854 Acked 8853 Write 2 New Done Done false
23831 7855 Acked 8854 Read 3 New Done Done false
23832 7856 Acked 8855 Write 8 New Done Done false
23833 7857 Acked 8856 Write 6 New Done Done false
23834 7858 Acked 8857 Write 2 New Done Done false
23835 7859 Acked 8858 Read 10 New Done Done false
23836 7860 Acked 8859 Read 6 New Done Done false
23837 7861 Acked 8860 Read 1 New Done Done false
23838 7862 Acked 8861 Write 6 New Done Done false
23839 7863 Acked 8862 Read 1 New Done Done false
23840 7864 Acked 8863 Read 2 New Done Done false
23841 7865 Acked 8864 Read 10 New Done Done false
23842 7866 Acked 8865 Read 7 New Done Done false
23843 7867 Acked 8866 Read 8 New Done Done false
23844 7868 Acked 8867 Flush 0 New Done Done false
23845 7869 Acked 8868 Write 2 New Done Done false
23846 7870 Acked 8869 Write 9 New Done Done false
23847 7871 Acked 8870 Write 5 New Done Done false
23848 7872 Acked 8871 Read 1 New Done Done false
23849 7873 Acked 8872 Write 2 New Done Done false
23850 7874 Acked 8873 Read 3 New Done Done false
23851 7875 Acked 8874 Read 10 New Done Done false
23852 7876 Acked 8875 Write 6 New Done Done false
23853 7877 Acked 8876 Write 5 New Done Done false
23854 7878 Acked 8877 Write 2 New Done Done false
23855 7879 Acked 8878 Write 2 New Done Done false
23856 7880 Acked 8879 Write 5 New Done Done false
23857 7881 Acked 8880 Flush 0 New Done Done false
23858 7882 Acked 8881 Read 1 New Done Done false
23859 7883 Acked 8882 Flush 0 New Done Done false
23860 7884 Acked 8883 Read 3 New Done Done false
23861 7885 Acked 8884 Flush 0 New Done Done false
23862 7886 Acked 8885 Read 10 New Done Done false
23863 7887 Acked 8886 Write 5 New Done Done false
23864 7888 Acked 8887 Read 2 New Done Done false
23865 7889 Acked 8888 Flush 0 New Done Done false
23866 7890 Acked 8889 Write 6 New Done Done false
23867 7891 Acked 8890 Flush 0 New Done Done false
23868 7892 Acked 8891 Write 3 New Done Done false
23869 7893 Acked 8892 Flush 0 New Done Done false
23870 7894 Acked 8893 Write 6 New Done Done false
23871 7895 Acked 8894 Write 4 New Done Done false
23872 7896 Acked 8895 Write 6 New Done Done false
23873 7897 Acked 8896 Read 8 New Done Done false
23874 7898 Acked 8897 Flush 0 New Done Done false
23875 7899 Acked 8898 Write 9 New Done Done false
23876 7900 Acked 8899 Write 2 New Done Done false
23877 7901 Acked 8900 Read 6 New Done Done false
23878 7902 Acked 8901 Read 7 New Done Done false
23879 7903 Acked 8902 Read 10 New Done Done false
23880 7904 Acked 8903 Write 4 New Done Done false
23881 7905 Acked 8904 Read 7 New Done Done false
23882 7906 Acked 8905 Write 10 New Done Done false
23883 7907 Acked 8906 Read 8 New Done Done false
23884 7908 Acked 8907 Read 1 New Done Done false
23885 7909 Acked 8908 Write 8 New Done Done false
23886 7910 Acked 8909 Write 2 New Done Done false
23887 7911 Acked 8910 Write 3 New Done Done false
23888 7912 Acked 8911 Read 5 New Done Done false
23889 7913 Acked 8912 Write 7 New Done Done false
23890 7914 Acked 8913 Read 7 New Done Done false
23891 7915 Acked 8914 Flush 0 New Done Done false
23892 7916 Acked 8915 Read 4 New Done Done false
23893 7917 Acked 8916 Read 5 New Done Done false
23894 7918 Acked 8917 Write 3 New Done Done false
23895 7919 Acked 8918 Write 2 New Done Done false
23896 7920 Acked 8919 Read 3 New Done Done false
23897 7921 Acked 8920 Write 7 New Done Done false
23898 7922 Acked 8921 Read 5 New Done Done false
23899 7923 Acked 8922 Write 5 New Done Done false
23900 7924 Acked 8923 Write 7 New Done Done false
23901 7925 Acked 8924 Write 8 New Done Done false
23902 7926 Acked 8925 Write 1 New Done Done false
23903 7927 Acked 8926 Flush 0 New Done Done false
23904 7928 Acked 8927 Write 2 New Done Done false
23905 7929 Acked 8928 Write 5 New Done Done false
23906 7930 Acked 8929 Write 1 New Done Done false
23907 7931 Acked 8930 Write 6 New Done Done false
23908 7932 Acked 8931 Write 2 New Done Done false
23909 7933 Acked 8932 Read 8 New Done Done false
23910 7934 Acked 8933 Read 7 New Done Done false
23911 7935 Acked 8934 Read 3 New Done Done false
23912 7936 Acked 8935 Read 7 New Done Done false
23913 7937 Acked 8936 Read 7 New Done Done false
23914 7938 Acked 8937 Write 9 New Done Done false
23915 7939 Acked 8938 Read 3 New Done Done false
23916 7940 Acked 8939 Write 9 New Done Done false
23917 7941 Acked 8940 Read 6 New Done Done false
23918 7942 Acked 8941 Read 3 New Done Done false
23919 7943 Acked 8942 Write 6 New Done Done false
23920 7944 Acked 8943 Read 7 New Done Done false
23921 7945 Acked 8944 Flush 0 New Done Done false
23922 7946 Acked 8945 Write 9 New Done Done false
23923 7947 Acked 8946 Read 7 New Done Done false
23924 7948 Acked 8947 Read 5 New Done Done false
23925 7949 Acked 8948 Write 3 New Done Done false
23926 7950 Acked 8949 Read 2 New Done Done false
23927 7951 Acked 8950 Flush 0 New Done Done false
23928 7952 Acked 8951 Write 5 New Done Done false
23929 7953 Acked 8952 Flush 0 New Done Done false
23930 7954 Acked 8953 Write 6 New Done Done false
23931 7955 Acked 8954 Read 5 New Done Done false
23932 7956 Acked 8955 Write 5 New Done Done false
23933 7957 Acked 8956 Write 10 New Done Done false
23934 7958 Acked 8957 Read 6 New Done Done false
23935 7959 Acked 8958 Write 8 New Done Done false
23936 7960 Acked 8959 Write 8 New Done Done false
23937 7961 Acked 8960 Read 3 New Done Done false
23938 7962 Acked 8961 Read 6 New Done Done false
23939 7963 Acked 8962 Write 5 New Done Done false
23940 7964 Acked 8963 Write 5 New Done Done false
23941 7965 Acked 8964 Read 10 New Done Done false
23942 7966 Acked 8965 Read 4 New Done Done false
23943 7967 Acked 8966 Flush 0 New Done Done false
23944 7968 Acked 8967 Read 6 New Done Done false
23945 7969 Acked 8968 Flush 0 New Done Done false
23946 7970 Acked 8969 Read 3 New Done Done false
23947 7971 Acked 8970 Read 6 New Done Done false
23948 7972 Acked 8971 Write 3 New Done Done false
23949 7973 Acked 8972 Flush 0 New Done Done false
23950 7974 Acked 8973 Write 7 New Done Done false
23951 7975 Acked 8974 Write 9 New Done Done false
23952 7976 Acked 8975 Write 10 New Done Done false
23953 7977 Acked 8976 Read 2 New Done Done false
23954 7978 Acked 8977 Write 9 New Done Done false
23955 7979 Acked 8978 Write 5 New Done Done false
23956 7980 Acked 8979 Read 9 New Done Done false
23957 7981 Acked 8980 Flush 0 New Done Done false
23958 7982 Acked 8981 Read 6 New Done Done false
23959 7983 Acked 8982 Write 3 New Done Done false
23960 7984 Acked 8983 Write 3 New Done Done false
23961 7985 Acked 8984 Flush 0 New Done Done false
23962 7986 Acked 8985 Read 9 New Done Done false
23963 7987 Acked 8986 Write 2 New Done Done false
23964 7988 Acked 8987 Read 1 New Done Done false
23965 7989 Acked 8988 Read 9 New Done Done false
23966 7990 Acked 8989 Write 1 New Done Done false
23967 7991 Acked 8990 Write 8 New Done Done false
23968 7992 Acked 8991 Write 10 New Done Done false
23969 7993 Acked 8992 Write 2 New Done Done false
23970 7994 Acked 8993 Write 3 New Done Done false
23971 7995 Acked 8994 Write 8 New Done Done false
23972 7996 Acked 8995 Flush 0 New Done Done false
23973 7997 Acked 8996 Read 4 New Done Done false
23974 7998 Acked 8997 Read 9 New Done Done false
23975 7999 Acked 8998 Write 6 New Done Done false
23976 8000 Acked 8999 Write 8 New Done Done false
23977 8001 Acked 9000 Write 7 New Done Done false
23978 8002 Acked 9001 Read 4 New Done Done false
23979 8003 Acked 9002 Read 6 New Done Done false
23980 8004 Acked 9003 Read 2 New Done Done false
23981 8005 Acked 9004 Read 10 New Done Done false
23982 8006 Acked 9005 Write 3 New Done Done false
23983 8007 Acked 9006 Read 6 New Done Done false
23984 8008 Acked 9007 Read 4 New Done Done false
23985 8009 Acked 9008 Read 7 New Done Done false
23986 8010 Acked 9009 Write 10 New Done Done false
23987 8011 Acked 9010 Write 4 New Done Done false
23988 8012 Acked 9011 Write 10 New Done Done false
23989 8013 Acked 9012 Write 6 New Done Done false
23990 8014 Acked 9013 Read 7 New Done Done false
23991 8015 Acked 9014 Write 1 New Done Done false
23992 8016 Acked 9015 Write 8 New Done Done false
23993 8017 Acked 9016 Write 1 New Done Done false
23994 8018 Acked 9017 Write 10 New Done Done false
23995 8019 Acked 9018 Write 6 New Done Done false
23996 8020 Acked 9019 Read 2 New Done Done false
23997 8021 Acked 9020 Read 5 New Done Done false
23998 8022 Acked 9021 Write 1 New Done Done false
23999 8023 Acked 9022 Write 3 New Done Done false
24000 8024 Acked 9023 Read 4 New Done Done false
24001 8025 Acked 9024 Read 7 New Done Done false
24002 8026 Acked 9025 Read 4 New Done Done false
24003 8027 Acked 9026 Read 5 New Done Done false
24004 8028 Acked 9027 Read 8 New Done Done false
24005 8029 Acked 9028 Write 2 New Done Done false
24006 8030 Acked 9029 Read 7 New Done Done false
24007 8031 Acked 9030 Read 1 New Done Done false
24008 8032 Acked 9031 Read 1 New Done Done false
24009 8033 Acked 9032 Write 3 New Done Done false
24010 8034 Acked 9033 Read 6 New Done Done false
24011 8035 Acked 9034 Read 6 New Done Done false
24012 8036 Acked 9035 Write 1 New Done Done false
24013 8037 Acked 9036 Read 9 New Done Done false
24014 8038 Acked 9037 Write 9 New Done Done false
24015 8039 Acked 9038 Read 3 New Done Done false
24016 8040 Acked 9039 Write 3 New Done Done false
24017 8041 Acked 9040 Write 3 New Done Done false
24018 8042 Acked 9041 Read 5 New Done Done false
24019 8043 Acked 9042 Read 7 New Done Done false
24020 8044 Acked 9043 Write 2 New Done Done false
24021 8045 Acked 9044 Write 5 New Done Done false
24022 8046 Acked 9045 Read 7 New Done Done false
24023 8047 Acked 9046 Write 10 New Done Done false
24024 8048 Acked 9047 Write 8 New Done Done false
24025 8049 Acked 9048 Flush 0 New Done Done false
24026 8050 Acked 9049 Write 5 New Done Done false
24027 8051 Acked 9050 Read 9 New Done Done false
24028 8052 Acked 9051 Read 5 New Done Done false
24029 8053 Acked 9052 Read 4 New Done Done false
24030 8054 Acked 9053 Read 8 New Done Done false
24031 8055 Acked 9054 Read 7 New Done Done false
24032 8056 Acked 9055 Read 6 New Done Done false
24033 8057 Acked 9056 Read 6 New Done Done false
24034 8058 Acked 9057 Write 8 New Done Done false
24035 8059 Acked 9058 Read 7 New Done Done false
24036 8060 Acked 9059 Read 3 New Done Done false
24037 8061 Acked 9060 Write 4 New Done Done false
24038 8062 Acked 9061 Flush 0 New Done Done false
24039 8063 Acked 9062 Read 5 New Done Done false
24040 8064 Acked 9063 Flush 0 New Done Done false
24041 8065 Acked 9064 Read 4 New Done Done false
24042 8066 Acked 9065 Read 3 New Done Done false
24043 8067 Acked 9066 Flush 0 New Done Done false
24044 8068 Acked 9067 Write 5 New Done Done false
24045 8069 Acked 9068 Read 2 New Done Done false
24046 8070 Acked 9069 Flush 0 New Done Done false
24047 8071 Acked 9070 Write 4 New Done Done false
24048 8072 Acked 9071 Read 4 New Done Done false
24049 8073 Acked 9072 Write 10 New Done Done false
24050 8074 Acked 9073 Read 8 New Done Done false
24051 8075 Acked 9074 Write 10 New Done Done false
24052 8076 Acked 9075 Write 10 New Done Done false
24053 8077 Acked 9076 Write 1 New Done Done false
24054 8078 Acked 9077 Read 7 New Done Done false
24055 8079 Acked 9078 Flush 0 New Done Done false
24056 8080 Acked 9079 Write 2 New Done Done false
24057 8081 Acked 9080 Read 9 New Done Done false
24058 8082 Acked 9081 Read 10 New Done Done false
24059 8083 Acked 9082 Read 4 New Done Done false
24060 8084 Acked 9083 Write 6 New Done Done false
24061 8085 Acked 9084 Write 5 New Done Done false
24062 8086 Acked 9085 Write 7 New Done Done false
24063 8087 Acked 9086 Read 5 New Done Done false
24064 8088 Acked 9087 Read 5 New Done Done false
24065 8089 Acked 9088 Read 9 New Done Done false
24066 8090 Acked 9089 Read 4 New Done Done false
24067 8091 Acked 9090 Flush 0 New Done Done false
24068 8092 Acked 9091 Write 2 New Done Done false
24069 8093 Acked 9092 Write 8 New Done Done false
24070 8094 Acked 9093 Write 10 New Done Done false
24071 8095 Acked 9094 Read 10 New Done Done false
24072 8096 Acked 9095 Write 5 New Done Done false
24073 8097 Acked 9096 Read 8 New Done Done false
24074 8098 Acked 9097 Read 5 New Done Done false
24075 8099 Acked 9098 Write 8 New Done Done false
24076 8100 Acked 9099 Read 1 New Done Done false
24077 8101 Acked 9100 Read 6 New Done Done false
24078 8102 Acked 9101 Flush 0 New Done Done false
24079 8103 Acked 9102 Write 6 New Done Done false
24080 8104 Acked 9103 Read 10 New Done Done false
24081 8105 Acked 9104 Read 5 New Done Done false
24082 8106 Acked 9105 Write 4 New Done Done false
24083 8107 Acked 9106 Flush 0 New Done Done false
24084 8108 Acked 9107 Read 5 New Done Done false
24085 8109 Acked 9108 Flush 0 New Done Done false
24086 8110 Acked 9109 Read 9 New Done Done false
24087 8111 Acked 9110 Read 6 New Done Done false
24088 8112 Acked 9111 Read 9 New Done Done false
24089 8113 Acked 9112 Read 2 New Done Done false
24090 8114 Acked 9113 Write 2 New Done Done false
24091 8115 Acked 9114 Write 1 New Done Done false
24092 8116 Acked 9115 Read 9 New Done Done false
24093 8117 Acked 9116 Flush 0 New Done Done false
24094 8118 Acked 9117 Read 9 New Done Done false
24095 8119 Acked 9118 Flush 0 New Done Done false
24096 8120 Acked 9119 Write 8 New Done Done false
24097 8121 Acked 9120 Read 7 New Done Done false
24098 8122 Acked 9121 Flush 0 New Done Done false
24099 8123 Acked 9122 Read 9 New Done Done false
24100 8124 Acked 9123 Write 2 New Done Done false
24101 8125 Acked 9124 Flush 0 New Done Done false
24102 8126 Acked 9125 Read 9 New Done Done false
24103 8127 Acked 9126 Read 7 New Done Done false
24104 8128 Acked 9127 Write 8 New Done Done false
24105 8129 Acked 9128 Write 8 New Done Done false
24106 8130 Acked 9129 Write 5 New Done Done false
24107 8131 Acked 9130 Write 5 New Done Done false
24108 8132 Acked 9131 Write 10 New Done Done false
24109 8133 Acked 9132 Read 4 New Done Done false
24110 8134 Acked 9133 Read 1 New Done Done false
24111 8135 Acked 9134 Write 8 New Done Done false
24112 8136 Acked 9135 Read 1 New Done Done false
24113 8137 Acked 9136 Write 4 New Done Done false
24114 8138 Acked 9137 Write 5 New Done Done false
24115 8139 Acked 9138 Read 6 New Done Done false
24116 STATES DS:0 DS:1 DS:2 TOTAL
24117 New 301 0 0 301
24118 Sent 0 0 0 0
24119 Done 0 301 301 602
24120 Skipped 0 0 0 0
24121 Error 0 0 0 0
24122 Last Flush: 8837 9124 9124
24123 Downstairs last five completed: 8837 8836 8835 8834 8833
24124 Upstairs last five completed: 8139 8138 8137 8136 8135
24125 CLIENT: Up:0 ds:301 act:2
241262024-02-12T01:39:16.046ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
241272024-02-12T01:39:16.046ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
241282024-02-12T01:39:16.046ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
241292024-02-12T01:39:16.046ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
241302024-02-12T01:39:16.046ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
241312024-02-12T01:39:16.046ZINFOcrucible: [0] client re-new 302 jobs since flush 8837 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
241322024-02-12T01:39:16.046ZINFOcrucible: client task is exiting = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
241332024-02-12T01:39:17.047ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
241342024-02-12T01:39:17.047ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
241352024-02-12T01:39:17.047ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
241362024-02-12T01:39:17.047ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
241372024-02-12T01:39:17.047ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
241382024-02-12T01:39:17.047ZINFOcrucible: [0] client re-new 302 jobs since flush 8837 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
241392024-02-12T01:39:17.048ZINFOcrucible: client task is exiting = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
241402024-02-12T01:39:18.048ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
241412024-02-12T01:39:18.048ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
241422024-02-12T01:39:18.048ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
241432024-02-12T01:39:18.048ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
241442024-02-12T01:39:18.048ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
241452024-02-12T01:39:18.048ZINFOcrucible: [0] client re-new 302 jobs since flush 8837 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
241462024-02-12T01:39:18.048ZINFOcrucible: client task is exiting = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
241472024-02-12T01:39:19.049ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
241482024-02-12T01:39:19.049ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
241492024-02-12T01:39:19.049ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
241502024-02-12T01:39:19.049ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
241512024-02-12T01:39:19.049ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
241522024-02-12T01:39:19.049ZINFOcrucible: [0] client re-new 302 jobs since flush 8837 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
241532024-02-12T01:39:19.049ZINFOcrucible: client task is exiting = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
24154 ----------------------------------------------------------------
24155 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
24156 States: Offline Active Active
24157 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
24158 7839 Acked 8838 Write 3 New Done Done true
24159 7840 Acked 8839 Read 2 New Done Done true
24160 7841 Acked 8840 Write 8 New Done Done true
24161 7842 Acked 8841 Read 6 New Done Done true
24162 7843 Acked 8842 Write 8 New Done Done true
24163 7844 Acked 8843 Write 8 New Done Done true
24164 7845 Acked 8844 Write 5 New Done Done true
24165 7846 Acked 8845 Write 10 New Done Done true
24166 7847 Acked 8846 Read 10 New Done Done true
24167 7848 Acked 8847 Read 8 New Done Done true
24168 7849 Acked 8848 Read 5 New Done Done true
24169 7850 Acked 8849 Read 7 New Done Done true
24170 7851 Acked 8850 Write 1 New Done Done true
24171 7852 Acked 8851 Write 3 New Done Done true
24172 7853 Acked 8852 Write 8 New Done Done true
24173 7854 Acked 8853 Write 2 New Done Done true
24174 7855 Acked 8854 Read 3 New Done Done true
24175 7856 Acked 8855 Write 8 New Done Done true
24176 7857 Acked 8856 Write 6 New Done Done true
24177 7858 Acked 8857 Write 2 New Done Done true
24178 7859 Acked 8858 Read 10 New Done Done true
24179 7860 Acked 8859 Read 6 New Done Done true
24180 7861 Acked 8860 Read 1 New Done Done true
24181 7862 Acked 8861 Write 6 New Done Done true
24182 7863 Acked 8862 Read 1 New Done Done true
24183 7864 Acked 8863 Read 2 New Done Done true
24184 7865 Acked 8864 Read 10 New Done Done true
24185 7866 Acked 8865 Read 7 New Done Done true
24186 7867 Acked 8866 Read 8 New Done Done true
24187 7868 Acked 8867 Flush 0 New Done Done true
24188 7869 Acked 8868 Write 2 New Done Done true
24189 7870 Acked 8869 Write 9 New Done Done true
24190 7871 Acked 8870 Write 5 New Done Done true
24191 7872 Acked 8871 Read 1 New Done Done true
24192 7873 Acked 8872 Write 2 New Done Done true
24193 7874 Acked 8873 Read 3 New Done Done true
24194 7875 Acked 8874 Read 10 New Done Done true
24195 7876 Acked 8875 Write 6 New Done Done true
24196 7877 Acked 8876 Write 5 New Done Done true
24197 7878 Acked 8877 Write 2 New Done Done true
24198 7879 Acked 8878 Write 2 New Done Done true
24199 7880 Acked 8879 Write 5 New Done Done true
24200 7881 Acked 8880 Flush 0 New Done Done true
24201 7882 Acked 8881 Read 1 New Done Done true
24202 7883 Acked 8882 Flush 0 New Done Done true
24203 7884 Acked 8883 Read 3 New Done Done true
24204 7885 Acked 8884 Flush 0 New Done Done true
24205 7886 Acked 8885 Read 10 New Done Done true
24206 7887 Acked 8886 Write 5 New Done Done true
24207 7888 Acked 8887 Read 2 New Done Done true
24208 7889 Acked 8888 Flush 0 New Done Done true
24209 7890 Acked 8889 Write 6 New Done Done true
24210 7891 Acked 8890 Flush 0 New Done Done true
24211 7892 Acked 8891 Write 3 New Done Done true
24212 7893 Acked 8892 Flush 0 New Done Done true
24213 7894 Acked 8893 Write 6 New Done Done true
24214 7895 Acked 8894 Write 4 New Done Done true
24215 7896 Acked 8895 Write 6 New Done Done true
24216 7897 Acked 8896 Read 8 New Done Done true
24217 7898 Acked 8897 Flush 0 New Done Done true
24218 7899 Acked 8898 Write 9 New Done Done true
24219 7900 Acked 8899 Write 2 New Done Done true
24220 7901 Acked 8900 Read 6 New Done Done true
24221 7902 Acked 8901 Read 7 New Done Done true
24222 7903 Acked 8902 Read 10 New Done Done true
24223 7904 Acked 8903 Write 4 New Done Done true
24224 7905 Acked 8904 Read 7 New Done Done true
24225 7906 Acked 8905 Write 10 New Done Done true
24226 7907 Acked 8906 Read 8 New Done Done true
24227 7908 Acked 8907 Read 1 New Done Done true
24228 7909 Acked 8908 Write 8 New Done Done true
24229 7910 Acked 8909 Write 2 New Done Done true
24230 7911 Acked 8910 Write 3 New Done Done true
24231 7912 Acked 8911 Read 5 New Done Done true
24232 7913 Acked 8912 Write 7 New Done Done true
24233 7914 Acked 8913 Read 7 New Done Done true
24234 7915 Acked 8914 Flush 0 New Done Done true
24235 7916 Acked 8915 Read 4 New Done Done true
24236 7917 Acked 8916 Read 5 New Done Done true
24237 7918 Acked 8917 Write 3 New Done Done true
24238 7919 Acked 8918 Write 2 New Done Done true
24239 7920 Acked 8919 Read 3 New Done Done true
24240 7921 Acked 8920 Write 7 New Done Done true
24241 7922 Acked 8921 Read 5 New Done Done true
24242 7923 Acked 8922 Write 5 New Done Done true
24243 7924 Acked 8923 Write 7 New Done Done true
24244 7925 Acked 8924 Write 8 New Done Done true
24245 7926 Acked 8925 Write 1 New Done Done true
24246 7927 Acked 8926 Flush 0 New Done Done true
24247 7928 Acked 8927 Write 2 New Done Done true
24248 7929 Acked 8928 Write 5 New Done Done true
24249 7930 Acked 8929 Write 1 New Done Done true
24250 7931 Acked 8930 Write 6 New Done Done true
24251 7932 Acked 8931 Write 2 New Done Done true
24252 7933 Acked 8932 Read 8 New Done Done true
24253 7934 Acked 8933 Read 7 New Done Done true
24254 7935 Acked 8934 Read 3 New Done Done true
24255 7936 Acked 8935 Read 7 New Done Done true
24256 7937 Acked 8936 Read 7 New Done Done true
24257 7938 Acked 8937 Write 9 New Done Done true
24258 7939 Acked 8938 Read 3 New Done Done true
24259 7940 Acked 8939 Write 9 New Done Done true
24260 7941 Acked 8940 Read 6 New Done Done true
24261 7942 Acked 8941 Read 3 New Done Done true
24262 7943 Acked 8942 Write 6 New Done Done true
24263 7944 Acked 8943 Read 7 New Done Done true
24264 7945 Acked 8944 Flush 0 New Done Done true
24265 7946 Acked 8945 Write 9 New Done Done true
24266 7947 Acked 8946 Read 7 New Done Done true
24267 7948 Acked 8947 Read 5 New Done Done true
24268 7949 Acked 8948 Write 3 New Done Done true
24269 7950 Acked 8949 Read 2 New Done Done true
24270 7951 Acked 8950 Flush 0 New Done Done true
24271 7952 Acked 8951 Write 5 New Done Done true
24272 7953 Acked 8952 Flush 0 New Done Done true
24273 7954 Acked 8953 Write 6 New Done Done true
24274 7955 Acked 8954 Read 5 New Done Done true
24275 7956 Acked 8955 Write 5 New Done Done true
24276 7957 Acked 8956 Write 10 New Done Done true
24277 7958 Acked 8957 Read 6 New Done Done true
24278 7959 Acked 8958 Write 8 New Done Done true
24279 7960 Acked 8959 Write 8 New Done Done true
24280 7961 Acked 8960 Read 3 New Done Done true
24281 7962 Acked 8961 Read 6 New Done Done true
24282 7963 Acked 8962 Write 5 New Done Done true
24283 7964 Acked 8963 Write 5 New Done Done true
24284 7965 Acked 8964 Read 10 New Done Done true
24285 7966 Acked 8965 Read 4 New Done Done true
24286 7967 Acked 8966 Flush 0 New Done Done true
24287 7968 Acked 8967 Read 6 New Done Done true
24288 7969 Acked 8968 Flush 0 New Done Done true
24289 7970 Acked 8969 Read 3 New Done Done true
24290 7971 Acked 8970 Read 6 New Done Done true
24291 7972 Acked 8971 Write 3 New Done Done true
24292 7973 Acked 8972 Flush 0 New Done Done true
24293 7974 Acked 8973 Write 7 New Done Done true
24294 7975 Acked 8974 Write 9 New Done Done true
24295 7976 Acked 8975 Write 10 New Done Done true
24296 7977 Acked 8976 Read 2 New Done Done true
24297 7978 Acked 8977 Write 9 New Done Done true
24298 7979 Acked 8978 Write 5 New Done Done true
24299 7980 Acked 8979 Read 9 New Done Done true
24300 7981 Acked 8980 Flush 0 New Done Done true
24301 7982 Acked 8981 Read 6 New Done Done true
24302 7983 Acked 8982 Write 3 New Done Done true
24303 7984 Acked 8983 Write 3 New Done Done true
24304 7985 Acked 8984 Flush 0 New Done Done true
24305 7986 Acked 8985 Read 9 New Done Done true
24306 7987 Acked 8986 Write 2 New Done Done true
24307 7988 Acked 8987 Read 1 New Done Done true
24308 7989 Acked 8988 Read 9 New Done Done true
24309 7990 Acked 8989 Write 1 New Done Done true
24310 7991 Acked 8990 Write 8 New Done Done true
24311 7992 Acked 8991 Write 10 New Done Done true
24312 7993 Acked 8992 Write 2 New Done Done true
24313 7994 Acked 8993 Write 3 New Done Done true
24314 7995 Acked 8994 Write 8 New Done Done true
24315 7996 Acked 8995 Flush 0 New Done Done true
24316 7997 Acked 8996 Read 4 New Done Done true
24317 7998 Acked 8997 Read 9 New Done Done true
24318 7999 Acked 8998 Write 6 New Done Done true
24319 8000 Acked 8999 Write 8 New Done Done true
24320 8001 Acked 9000 Write 7 New Done Done true
24321 8002 Acked 9001 Read 4 New Done Done true
24322 8003 Acked 9002 Read 6 New Done Done true
24323 8004 Acked 9003 Read 2 New Done Done true
24324 8005 Acked 9004 Read 10 New Done Done true
24325 8006 Acked 9005 Write 3 New Done Done true
24326 8007 Acked 9006 Read 6 New Done Done true
24327 8008 Acked 9007 Read 4 New Done Done true
24328 8009 Acked 9008 Read 7 New Done Done true
24329 8010 Acked 9009 Write 10 New Done Done true
24330 8011 Acked 9010 Write 4 New Done Done true
24331 8012 Acked 9011 Write 10 New Done Done true
24332 8013 Acked 9012 Write 6 New Done Done true
24333 8014 Acked 9013 Read 7 New Done Done true
24334 8015 Acked 9014 Write 1 New Done Done true
24335 8016 Acked 9015 Write 8 New Done Done true
24336 8017 Acked 9016 Write 1 New Done Done true
24337 8018 Acked 9017 Write 10 New Done Done true
24338 8019 Acked 9018 Write 6 New Done Done true
24339 8020 Acked 9019 Read 2 New Done Done true
24340 8021 Acked 9020 Read 5 New Done Done true
24341 8022 Acked 9021 Write 1 New Done Done true
24342 8023 Acked 9022 Write 3 New Done Done true
24343 8024 Acked 9023 Read 4 New Done Done true
24344 8025 Acked 9024 Read 7 New Done Done true
24345 8026 Acked 9025 Read 4 New Done Done true
24346 8027 Acked 9026 Read 5 New Done Done true
24347 8028 Acked 9027 Read 8 New Done Done true
24348 8029 Acked 9028 Write 2 New Done Done true
24349 8030 Acked 9029 Read 7 New Done Done true
24350 8031 Acked 9030 Read 1 New Done Done true
24351 8032 Acked 9031 Read 1 New Done Done true
24352 8033 Acked 9032 Write 3 New Done Done true
24353 8034 Acked 9033 Read 6 New Done Done true
24354 8035 Acked 9034 Read 6 New Done Done true
24355 8036 Acked 9035 Write 1 New Done Done true
24356 8037 Acked 9036 Read 9 New Done Done true
24357 8038 Acked 9037 Write 9 New Done Done true
24358 8039 Acked 9038 Read 3 New Done Done true
24359 8040 Acked 9039 Write 3 New Done Done true
24360 8041 Acked 9040 Write 3 New Done Done true
24361 8042 Acked 9041 Read 5 New Done Done true
24362 8043 Acked 9042 Read 7 New Done Done true
24363 8044 Acked 9043 Write 2 New Done Done true
24364 8045 Acked 9044 Write 5 New Done Done true
24365 8046 Acked 9045 Read 7 New Done Done true
24366 8047 Acked 9046 Write 10 New Done Done true
24367 8048 Acked 9047 Write 8 New Done Done true
24368 8049 Acked 9048 Flush 0 New Done Done true
24369 8050 Acked 9049 Write 5 New Done Done true
24370 8051 Acked 9050 Read 9 New Done Done true
24371 8052 Acked 9051 Read 5 New Done Done true
24372 8053 Acked 9052 Read 4 New Done Done true
24373 8054 Acked 9053 Read 8 New Done Done true
24374 8055 Acked 9054 Read 7 New Done Done true
24375 8056 Acked 9055 Read 6 New Done Done true
24376 8057 Acked 9056 Read 6 New Done Done true
24377 8058 Acked 9057 Write 8 New Done Done true
24378 8059 Acked 9058 Read 7 New Done Done true
24379 8060 Acked 9059 Read 3 New Done Done true
24380 8061 Acked 9060 Write 4 New Done Done true
24381 8062 Acked 9061 Flush 0 New Done Done true
24382 8063 Acked 9062 Read 5 New Done Done true
24383 8064 Acked 9063 Flush 0 New Done Done true
24384 8065 Acked 9064 Read 4 New Done Done true
24385 8066 Acked 9065 Read 3 New Done Done true
24386 8067 Acked 9066 Flush 0 New Done Done true
24387 8068 Acked 9067 Write 5 New Done Done true
24388 8069 Acked 9068 Read 2 New Done Done true
24389 8070 Acked 9069 Flush 0 New Done Done true
24390 8071 Acked 9070 Write 4 New Done Done true
24391 8072 Acked 9071 Read 4 New Done Done true
24392 8073 Acked 9072 Write 10 New Done Done true
24393 8074 Acked 9073 Read 8 New Done Done true
24394 8075 Acked 9074 Write 10 New Done Done true
24395 8076 Acked 9075 Write 10 New Done Done true
24396 8077 Acked 9076 Write 1 New Done Done true
24397 8078 Acked 9077 Read 7 New Done Done true
24398 8079 Acked 9078 Flush 0 New Done Done true
24399 8080 Acked 9079 Write 2 New Done Done true
24400 8081 Acked 9080 Read 9 New Done Done true
24401 8082 Acked 9081 Read 10 New Done Done true
24402 8083 Acked 9082 Read 4 New Done Done true
24403 8084 Acked 9083 Write 6 New Done Done true
24404 8085 Acked 9084 Write 5 New Done Done true
24405 8086 Acked 9085 Write 7 New Done Done true
24406 8087 Acked 9086 Read 5 New Done Done true
24407 8088 Acked 9087 Read 5 New Done Done true
24408 8089 Acked 9088 Read 9 New Done Done true
24409 8090 Acked 9089 Read 4 New Done Done true
24410 8091 Acked 9090 Flush 0 New Done Done true
24411 8092 Acked 9091 Write 2 New Done Done true
24412 8093 Acked 9092 Write 8 New Done Done true
24413 8094 Acked 9093 Write 10 New Done Done true
24414 8095 Acked 9094 Read 10 New Done Done true
24415 8096 Acked 9095 Write 5 New Done Done true
24416 8097 Acked 9096 Read 8 New Done Done true
24417 8098 Acked 9097 Read 5 New Done Done true
24418 8099 Acked 9098 Write 8 New Done Done true
24419 8100 Acked 9099 Read 1 New Done Done true
24420 8101 Acked 9100 Read 6 New Done Done true
24421 8102 Acked 9101 Flush 0 New Done Done true
24422 8103 Acked 9102 Write 6 New Done Done true
24423 8104 Acked 9103 Read 10 New Done Done true
24424 8105 Acked 9104 Read 5 New Done Done true
24425 8106 Acked 9105 Write 4 New Done Done true
24426 8107 Acked 9106 Flush 0 New Done Done true
24427 8108 Acked 9107 Read 5 New Done Done true
24428 8109 Acked 9108 Flush 0 New Done Done true
24429 8110 Acked 9109 Read 9 New Done Done true
24430 8111 Acked 9110 Read 6 New Done Done true
24431 8112 Acked 9111 Read 9 New Done Done true
24432 8113 Acked 9112 Read 2 New Done Done true
24433 8114 Acked 9113 Write 2 New Done Done true
24434 8115 Acked 9114 Write 1 New Done Done true
24435 8116 Acked 9115 Read 9 New Done Done true
24436 8117 Acked 9116 Flush 0 New Done Done true
24437 8118 Acked 9117 Read 9 New Done Done true
24438 8119 Acked 9118 Flush 0 New Done Done true
24439 8120 Acked 9119 Write 8 New Done Done true
24440 8121 Acked 9120 Read 7 New Done Done true
24441 8122 Acked 9121 Flush 0 New Done Done true
24442 8123 Acked 9122 Read 9 New Done Done true
24443 8124 Acked 9123 Write 2 New Done Done true
24444 8125 Acked 9124 Flush 0 New Done Done true
24445 8126 Acked 9125 Read 9 New Done Done true
24446 8127 Acked 9126 Read 7 New Done Done true
24447 8128 Acked 9127 Write 8 New Done Done true
24448 8129 Acked 9128 Write 8 New Done Done true
24449 8130 Acked 9129 Write 5 New Done Done true
24450 8131 Acked 9130 Write 5 New Done Done true
24451 8132 Acked 9131 Write 10 New Done Done true
24452 8133 Acked 9132 Read 4 New Done Done true
24453 8134 Acked 9133 Read 1 New Done Done true
24454 8135 Acked 9134 Write 8 New Done Done true
24455 8136 Acked 9135 Read 1 New Done Done true
24456 8137 Acked 9136 Write 4 New Done Done true
24457 8138 Acked 9137 Write 5 New Done Done true
24458 8139 Acked 9138 Read 6 New Done Done true
24459 8140 Acked 9139 Flush 0 New Done Done true
24460 STATES DS:0 DS:1 DS:2 TOTAL
24461 New 302 0 0 302
24462 Sent 0 0 0 0
24463 Done 0 302 302 604
24464 Skipped 0 0 0 0
24465 Error 0 0 0 0
24466 Last Flush: 8837 9139 9139
24467 Downstairs last five completed: 8837 8836 8835 8834 8833
24468 Upstairs last five completed: 8140 8139 8138 8137 8136
24469 CLIENT: Up:0 ds:302 act:2
244702024-02-12T01:39:20.051ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
244712024-02-12T01:39:20.051ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
244722024-02-12T01:39:20.051ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
244732024-02-12T01:39:20.051ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
244742024-02-12T01:39:20.051ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
244752024-02-12T01:39:20.051ZINFOcrucible: [0] client re-new 302 jobs since flush 8837 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
244762024-02-12T01:39:20.051ZINFOcrucible: client task is exiting = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
244772024-02-12T01:39:21.052ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
244782024-02-12T01:39:21.052ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
244792024-02-12T01:39:21.052ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
244802024-02-12T01:39:21.052ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
244812024-02-12T01:39:21.052ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
244822024-02-12T01:39:21.052ZINFOcrucible: [0] client re-new 302 jobs since flush 8837 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
244832024-02-12T01:39:21.052ZINFOcrucible: client task is exiting = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
244842024-02-12T01:39:22.053ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
244852024-02-12T01:39:22.053ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
244862024-02-12T01:39:22.053ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
244872024-02-12T01:39:22.053ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
244882024-02-12T01:39:22.053ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
244892024-02-12T01:39:22.053ZINFOcrucible: [0] client re-new 302 jobs since flush 8837 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
244902024-02-12T01:39:22.053ZINFOcrucible: client task is exiting = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
244912024-02-12T01:39:23.054ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
244922024-02-12T01:39:23.054ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
244932024-02-12T01:39:23.054ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
244942024-02-12T01:39:23.054ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
244952024-02-12T01:39:23.054ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
244962024-02-12T01:39:23.055ZINFOcrucible: [0] client re-new 302 jobs since flush 8837 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
244972024-02-12T01:39:23.055ZINFOcrucible: client task is exiting = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
24498 ----------------------------------------------------------------
24499 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
24500 States: Offline Active Active
24501 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
24502 7839 Acked 8838 Write 3 New Done Done true
24503 7840 Acked 8839 Read 2 New Done Done true
24504 7841 Acked 8840 Write 8 New Done Done true
24505 7842 Acked 8841 Read 6 New Done Done true
24506 7843 Acked 8842 Write 8 New Done Done true
24507 7844 Acked 8843 Write 8 New Done Done true
24508 7845 Acked 8844 Write 5 New Done Done true
24509 7846 Acked 8845 Write 10 New Done Done true
24510 7847 Acked 8846 Read 10 New Done Done true
24511 7848 Acked 8847 Read 8 New Done Done true
24512 7849 Acked 8848 Read 5 New Done Done true
24513 7850 Acked 8849 Read 7 New Done Done true
24514 7851 Acked 8850 Write 1 New Done Done true
24515 7852 Acked 8851 Write 3 New Done Done true
24516 7853 Acked 8852 Write 8 New Done Done true
24517 7854 Acked 8853 Write 2 New Done Done true
24518 7855 Acked 8854 Read 3 New Done Done true
24519 7856 Acked 8855 Write 8 New Done Done true
24520 7857 Acked 8856 Write 6 New Done Done true
24521 7858 Acked 8857 Write 2 New Done Done true
24522 7859 Acked 8858 Read 10 New Done Done true
24523 7860 Acked 8859 Read 6 New Done Done true
24524 7861 Acked 8860 Read 1 New Done Done true
24525 7862 Acked 8861 Write 6 New Done Done true
24526 7863 Acked 8862 Read 1 New Done Done true
24527 7864 Acked 8863 Read 2 New Done Done true
24528 7865 Acked 8864 Read 10 New Done Done true
24529 7866 Acked 8865 Read 7 New Done Done true
24530 7867 Acked 8866 Read 8 New Done Done true
24531 7868 Acked 8867 Flush 0 New Done Done true
24532 7869 Acked 8868 Write 2 New Done Done true
24533 7870 Acked 8869 Write 9 New Done Done true
24534 7871 Acked 8870 Write 5 New Done Done true
24535 7872 Acked 8871 Read 1 New Done Done true
24536 7873 Acked 8872 Write 2 New Done Done true
24537 7874 Acked 8873 Read 3 New Done Done true
24538 7875 Acked 8874 Read 10 New Done Done true
24539 7876 Acked 8875 Write 6 New Done Done true
24540 7877 Acked 8876 Write 5 New Done Done true
24541 7878 Acked 8877 Write 2 New Done Done true
24542 7879 Acked 8878 Write 2 New Done Done true
24543 7880 Acked 8879 Write 5 New Done Done true
24544 7881 Acked 8880 Flush 0 New Done Done true
24545 7882 Acked 8881 Read 1 New Done Done true
24546 7883 Acked 8882 Flush 0 New Done Done true
24547 7884 Acked 8883 Read 3 New Done Done true
24548 7885 Acked 8884 Flush 0 New Done Done true
24549 7886 Acked 8885 Read 10 New Done Done true
24550 7887 Acked 8886 Write 5 New Done Done true
24551 7888 Acked 8887 Read 2 New Done Done true
24552 7889 Acked 8888 Flush 0 New Done Done true
24553 7890 Acked 8889 Write 6 New Done Done true
24554 7891 Acked 8890 Flush 0 New Done Done true
24555 7892 Acked 8891 Write 3 New Done Done true
24556 7893 Acked 8892 Flush 0 New Done Done true
24557 7894 Acked 8893 Write 6 New Done Done true
24558 7895 Acked 8894 Write 4 New Done Done true
24559 7896 Acked 8895 Write 6 New Done Done true
24560 7897 Acked 8896 Read 8 New Done Done true
24561 7898 Acked 8897 Flush 0 New Done Done true
24562 7899 Acked 8898 Write 9 New Done Done true
24563 7900 Acked 8899 Write 2 New Done Done true
24564 7901 Acked 8900 Read 6 New Done Done true
24565 7902 Acked 8901 Read 7 New Done Done true
24566 7903 Acked 8902 Read 10 New Done Done true
24567 7904 Acked 8903 Write 4 New Done Done true
24568 7905 Acked 8904 Read 7 New Done Done true
24569 7906 Acked 8905 Write 10 New Done Done true
24570 7907 Acked 8906 Read 8 New Done Done true
24571 7908 Acked 8907 Read 1 New Done Done true
24572 7909 Acked 8908 Write 8 New Done Done true
24573 7910 Acked 8909 Write 2 New Done Done true
24574 7911 Acked 8910 Write 3 New Done Done true
24575 7912 Acked 8911 Read 5 New Done Done true
24576 7913 Acked 8912 Write 7 New Done Done true
24577 7914 Acked 8913 Read 7 New Done Done true
24578 7915 Acked 8914 Flush 0 New Done Done true
24579 7916 Acked 8915 Read 4 New Done Done true
24580 7917 Acked 8916 Read 5 New Done Done true
24581 7918 Acked 8917 Write 3 New Done Done true
24582 7919 Acked 8918 Write 2 New Done Done true
24583 7920 Acked 8919 Read 3 New Done Done true
24584 7921 Acked 8920 Write 7 New Done Done true
24585 7922 Acked 8921 Read 5 New Done Done true
24586 7923 Acked 8922 Write 5 New Done Done true
24587 7924 Acked 8923 Write 7 New Done Done true
24588 7925 Acked 8924 Write 8 New Done Done true
24589 7926 Acked 8925 Write 1 New Done Done true
24590 7927 Acked 8926 Flush 0 New Done Done true
24591 7928 Acked 8927 Write 2 New Done Done true
24592 7929 Acked 8928 Write 5 New Done Done true
24593 7930 Acked 8929 Write 1 New Done Done true
24594 7931 Acked 8930 Write 6 New Done Done true
24595 7932 Acked 8931 Write 2 New Done Done true
24596 7933 Acked 8932 Read 8 New Done Done true
24597 7934 Acked 8933 Read 7 New Done Done true
24598 7935 Acked 8934 Read 3 New Done Done true
24599 7936 Acked 8935 Read 7 New Done Done true
24600 7937 Acked 8936 Read 7 New Done Done true
24601 7938 Acked 8937 Write 9 New Done Done true
24602 7939 Acked 8938 Read 3 New Done Done true
24603 7940 Acked 8939 Write 9 New Done Done true
24604 7941 Acked 8940 Read 6 New Done Done true
24605 7942 Acked 8941 Read 3 New Done Done true
24606 7943 Acked 8942 Write 6 New Done Done true
24607 7944 Acked 8943 Read 7 New Done Done true
24608 7945 Acked 8944 Flush 0 New Done Done true
24609 7946 Acked 8945 Write 9 New Done Done true
24610 7947 Acked 8946 Read 7 New Done Done true
24611 7948 Acked 8947 Read 5 New Done Done true
24612 7949 Acked 8948 Write 3 New Done Done true
24613 7950 Acked 8949 Read 2 New Done Done true
24614 7951 Acked 8950 Flush 0 New Done Done true
24615 7952 Acked 8951 Write 5 New Done Done true
24616 7953 Acked 8952 Flush 0 New Done Done true
24617 7954 Acked 8953 Write 6 New Done Done true
24618 7955 Acked 8954 Read 5 New Done Done true
24619 7956 Acked 8955 Write 5 New Done Done true
24620 7957 Acked 8956 Write 10 New Done Done true
24621 7958 Acked 8957 Read 6 New Done Done true
24622 7959 Acked 8958 Write 8 New Done Done true
24623 7960 Acked 8959 Write 8 New Done Done true
24624 7961 Acked 8960 Read 3 New Done Done true
24625 7962 Acked 8961 Read 6 New Done Done true
24626 7963 Acked 8962 Write 5 New Done Done true
24627 7964 Acked 8963 Write 5 New Done Done true
24628 7965 Acked 8964 Read 10 New Done Done true
24629 7966 Acked 8965 Read 4 New Done Done true
24630 7967 Acked 8966 Flush 0 New Done Done true
24631 7968 Acked 8967 Read 6 New Done Done true
24632 7969 Acked 8968 Flush 0 New Done Done true
24633 7970 Acked 8969 Read 3 New Done Done true
24634 7971 Acked 8970 Read 6 New Done Done true
24635 7972 Acked 8971 Write 3 New Done Done true
24636 7973 Acked 8972 Flush 0 New Done Done true
24637 7974 Acked 8973 Write 7 New Done Done true
24638 7975 Acked 8974 Write 9 New Done Done true
24639 7976 Acked 8975 Write 10 New Done Done true
24640 7977 Acked 8976 Read 2 New Done Done true
24641 7978 Acked 8977 Write 9 New Done Done true
24642 7979 Acked 8978 Write 5 New Done Done true
24643 7980 Acked 8979 Read 9 New Done Done true
24644 7981 Acked 8980 Flush 0 New Done Done true
24645 7982 Acked 8981 Read 6 New Done Done true
24646 7983 Acked 8982 Write 3 New Done Done true
24647 7984 Acked 8983 Write 3 New Done Done true
24648 7985 Acked 8984 Flush 0 New Done Done true
24649 7986 Acked 8985 Read 9 New Done Done true
24650 7987 Acked 8986 Write 2 New Done Done true
24651 7988 Acked 8987 Read 1 New Done Done true
24652 7989 Acked 8988 Read 9 New Done Done true
24653 7990 Acked 8989 Write 1 New Done Done true
24654 7991 Acked 8990 Write 8 New Done Done true
24655 7992 Acked 8991 Write 10 New Done Done true
24656 7993 Acked 8992 Write 2 New Done Done true
24657 7994 Acked 8993 Write 3 New Done Done true
24658 7995 Acked 8994 Write 8 New Done Done true
24659 7996 Acked 8995 Flush 0 New Done Done true
24660 7997 Acked 8996 Read 4 New Done Done true
24661 7998 Acked 8997 Read 9 New Done Done true
24662 7999 Acked 8998 Write 6 New Done Done true
24663 8000 Acked 8999 Write 8 New Done Done true
24664 8001 Acked 9000 Write 7 New Done Done true
24665 8002 Acked 9001 Read 4 New Done Done true
24666 8003 Acked 9002 Read 6 New Done Done true
24667 8004 Acked 9003 Read 2 New Done Done true
24668 8005 Acked 9004 Read 10 New Done Done true
24669 8006 Acked 9005 Write 3 New Done Done true
24670 8007 Acked 9006 Read 6 New Done Done true
24671 8008 Acked 9007 Read 4 New Done Done true
24672 8009 Acked 9008 Read 7 New Done Done true
24673 8010 Acked 9009 Write 10 New Done Done true
24674 8011 Acked 9010 Write 4 New Done Done true
24675 8012 Acked 9011 Write 10 New Done Done true
24676 8013 Acked 9012 Write 6 New Done Done true
24677 8014 Acked 9013 Read 7 New Done Done true
24678 8015 Acked 9014 Write 1 New Done Done true
24679 8016 Acked 9015 Write 8 New Done Done true
24680 8017 Acked 9016 Write 1 New Done Done true
24681 8018 Acked 9017 Write 10 New Done Done true
24682 8019 Acked 9018 Write 6 New Done Done true
24683 8020 Acked 9019 Read 2 New Done Done true
24684 8021 Acked 9020 Read 5 New Done Done true
24685 8022 Acked 9021 Write 1 New Done Done true
24686 8023 Acked 9022 Write 3 New Done Done true
24687 8024 Acked 9023 Read 4 New Done Done true
24688 8025 Acked 9024 Read 7 New Done Done true
24689 8026 Acked 9025 Read 4 New Done Done true
24690 8027 Acked 9026 Read 5 New Done Done true
24691 8028 Acked 9027 Read 8 New Done Done true
24692 8029 Acked 9028 Write 2 New Done Done true
24693 8030 Acked 9029 Read 7 New Done Done true
24694 8031 Acked 9030 Read 1 New Done Done true
24695 8032 Acked 9031 Read 1 New Done Done true
24696 8033 Acked 9032 Write 3 New Done Done true
24697 8034 Acked 9033 Read 6 New Done Done true
24698 8035 Acked 9034 Read 6 New Done Done true
24699 8036 Acked 9035 Write 1 New Done Done true
24700 8037 Acked 9036 Read 9 New Done Done true
24701 8038 Acked 9037 Write 9 New Done Done true
24702 8039 Acked 9038 Read 3 New Done Done true
24703 8040 Acked 9039 Write 3 New Done Done true
24704 8041 Acked 9040 Write 3 New Done Done true
24705 8042 Acked 9041 Read 5 New Done Done true
24706 8043 Acked 9042 Read 7 New Done Done true
24707 8044 Acked 9043 Write 2 New Done Done true
24708 8045 Acked 9044 Write 5 New Done Done true
24709 8046 Acked 9045 Read 7 New Done Done true
24710 8047 Acked 9046 Write 10 New Done Done true
24711 8048 Acked 9047 Write 8 New Done Done true
24712 8049 Acked 9048 Flush 0 New Done Done true
24713 8050 Acked 9049 Write 5 New Done Done true
24714 8051 Acked 9050 Read 9 New Done Done true
24715 8052 Acked 9051 Read 5 New Done Done true
24716 8053 Acked 9052 Read 4 New Done Done true
24717 8054 Acked 9053 Read 8 New Done Done true
24718 8055 Acked 9054 Read 7 New Done Done true
24719 8056 Acked 9055 Read 6 New Done Done true
24720 8057 Acked 9056 Read 6 New Done Done true
24721 8058 Acked 9057 Write 8 New Done Done true
24722 8059 Acked 9058 Read 7 New Done Done true
24723 8060 Acked 9059 Read 3 New Done Done true
24724 8061 Acked 9060 Write 4 New Done Done true
24725 8062 Acked 9061 Flush 0 New Done Done true
24726 8063 Acked 9062 Read 5 New Done Done true
24727 8064 Acked 9063 Flush 0 New Done Done true
24728 8065 Acked 9064 Read 4 New Done Done true
24729 8066 Acked 9065 Read 3 New Done Done true
24730 8067 Acked 9066 Flush 0 New Done Done true
24731 8068 Acked 9067 Write 5 New Done Done true
24732 8069 Acked 9068 Read 2 New Done Done true
24733 8070 Acked 9069 Flush 0 New Done Done true
24734 8071 Acked 9070 Write 4 New Done Done true
24735 8072 Acked 9071 Read 4 New Done Done true
24736 8073 Acked 9072 Write 10 New Done Done true
24737 8074 Acked 9073 Read 8 New Done Done true
24738 8075 Acked 9074 Write 10 New Done Done true
24739 8076 Acked 9075 Write 10 New Done Done true
24740 8077 Acked 9076 Write 1 New Done Done true
24741 8078 Acked 9077 Read 7 New Done Done true
24742 8079 Acked 9078 Flush 0 New Done Done true
24743 8080 Acked 9079 Write 2 New Done Done true
24744 8081 Acked 9080 Read 9 New Done Done true
24745 8082 Acked 9081 Read 10 New Done Done true
24746 8083 Acked 9082 Read 4 New Done Done true
24747 8084 Acked 9083 Write 6 New Done Done true
24748 8085 Acked 9084 Write 5 New Done Done true
24749 8086 Acked 9085 Write 7 New Done Done true
24750 8087 Acked 9086 Read 5 New Done Done true
24751 8088 Acked 9087 Read 5 New Done Done true
24752 8089 Acked 9088 Read 9 New Done Done true
24753 8090 Acked 9089 Read 4 New Done Done true
24754 8091 Acked 9090 Flush 0 New Done Done true
24755 8092 Acked 9091 Write 2 New Done Done true
24756 8093 Acked 9092 Write 8 New Done Done true
24757 8094 Acked 9093 Write 10 New Done Done true
24758 8095 Acked 9094 Read 10 New Done Done true
24759 8096 Acked 9095 Write 5 New Done Done true
24760 8097 Acked 9096 Read 8 New Done Done true
24761 8098 Acked 9097 Read 5 New Done Done true
24762 8099 Acked 9098 Write 8 New Done Done true
24763 8100 Acked 9099 Read 1 New Done Done true
24764 8101 Acked 9100 Read 6 New Done Done true
24765 8102 Acked 9101 Flush 0 New Done Done true
24766 8103 Acked 9102 Write 6 New Done Done true
24767 8104 Acked 9103 Read 10 New Done Done true
24768 8105 Acked 9104 Read 5 New Done Done true
24769 8106 Acked 9105 Write 4 New Done Done true
24770 8107 Acked 9106 Flush 0 New Done Done true
24771 8108 Acked 9107 Read 5 New Done Done true
24772 8109 Acked 9108 Flush 0 New Done Done true
24773 8110 Acked 9109 Read 9 New Done Done true
24774 8111 Acked 9110 Read 6 New Done Done true
24775 8112 Acked 9111 Read 9 New Done Done true
24776 8113 Acked 9112 Read 2 New Done Done true
24777 8114 Acked 9113 Write 2 New Done Done true
24778 8115 Acked 9114 Write 1 New Done Done true
24779 8116 Acked 9115 Read 9 New Done Done true
24780 8117 Acked 9116 Flush 0 New Done Done true
24781 8118 Acked 9117 Read 9 New Done Done true
24782 8119 Acked 9118 Flush 0 New Done Done true
24783 8120 Acked 9119 Write 8 New Done Done true
24784 8121 Acked 9120 Read 7 New Done Done true
24785 8122 Acked 9121 Flush 0 New Done Done true
24786 8123 Acked 9122 Read 9 New Done Done true
24787 8124 Acked 9123 Write 2 New Done Done true
24788 8125 Acked 9124 Flush 0 New Done Done true
24789 8126 Acked 9125 Read 9 New Done Done true
24790 8127 Acked 9126 Read 7 New Done Done true
24791 8128 Acked 9127 Write 8 New Done Done true
24792 8129 Acked 9128 Write 8 New Done Done true
24793 8130 Acked 9129 Write 5 New Done Done true
24794 8131 Acked 9130 Write 5 New Done Done true
24795 8132 Acked 9131 Write 10 New Done Done true
24796 8133 Acked 9132 Read 4 New Done Done true
24797 8134 Acked 9133 Read 1 New Done Done true
24798 8135 Acked 9134 Write 8 New Done Done true
24799 8136 Acked 9135 Read 1 New Done Done true
24800 8137 Acked 9136 Write 4 New Done Done true
24801 8138 Acked 9137 Write 5 New Done Done true
24802 8139 Acked 9138 Read 6 New Done Done true
24803 8140 Acked 9139 Flush 0 New Done Done true
24804 STATES DS:0 DS:1 DS:2 TOTAL
24805 New 302 0 0 302
24806 Sent 0 0 0 0
24807 Done 0 302 302 604
24808 Skipped 0 0 0 0
24809 Error 0 0 0 0
24810 Last Flush: 8837 9139 9139
24811 Downstairs last five completed: 8837 8836 8835 8834 8833
24812 Upstairs last five completed: 8140 8139 8138 8137 8136
24813 CLIENT: Up:0 ds:302 act:2
248142024-02-12T01:39:24.055ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
248152024-02-12T01:39:24.055ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
248162024-02-12T01:39:24.055ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
248172024-02-12T01:39:24.055ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
248182024-02-12T01:39:24.055ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
248192024-02-12T01:39:24.055ZINFOcrucible: [0] client re-new 302 jobs since flush 8837 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
248202024-02-12T01:39:24.055ZINFOcrucible: client task is exiting = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
248212024-02-12T01:39:25.056ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
248222024-02-12T01:39:25.056ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
248232024-02-12T01:39:25.056ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
248242024-02-12T01:39:25.056ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
248252024-02-12T01:39:25.056ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
248262024-02-12T01:39:25.056ZINFOcrucible: [0] client re-new 302 jobs since flush 8837 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
248272024-02-12T01:39:25.056ZINFOcrucible: client task is exiting = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
248282024-02-12T01:39:26.057ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
248292024-02-12T01:39:26.057ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
248302024-02-12T01:39:26.057ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
248312024-02-12T01:39:26.057ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
248322024-02-12T01:39:26.057ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
248332024-02-12T01:39:26.057ZINFOcrucible: [0] client re-new 302 jobs since flush 8837 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
248342024-02-12T01:39:26.057ZINFOcrucible: client task is exiting = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
24835 [8810] got start action
24836 [8810] I am going to restart
24837 Starting downstairs at port 8810
24838 Make output file at "/tmp/dsc/downstairs-8810.txt"
24839 [8810][0] reports Starting
24840 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1191)
24841 [8810][0] reports Running
248422024-02-12T01:39:27.059ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
248432024-02-12T01:39:27.059ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
248442024-02-12T01:39:27.059ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
248452024-02-12T01:39:27.059ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
248462024-02-12T01:39:27.059ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
248472024-02-12T01:39:27.059ZINFOcrucible: [0] client re-new 302 jobs since flush 8837 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
248482024-02-12T01:39:27.059ZINFOcrucible: client task is exiting = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
24849 ----------------------------------------------------------------
24850 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
24851 States: Offline Active Active
24852 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
24853 7839 Acked 8838 Write 3 New Done Done true
24854 7840 Acked 8839 Read 2 New Done Done true
24855 7841 Acked 8840 Write 8 New Done Done true
24856 7842 Acked 8841 Read 6 New Done Done true
24857 7843 Acked 8842 Write 8 New Done Done true
24858 7844 Acked 8843 Write 8 New Done Done true
24859 7845 Acked 8844 Write 5 New Done Done true
24860 7846 Acked 8845 Write 10 New Done Done true
24861 7847 Acked 8846 Read 10 New Done Done true
24862 7848 Acked 8847 Read 8 New Done Done true
24863 7849 Acked 8848 Read 5 New Done Done true
24864 7850 Acked 8849 Read 7 New Done Done true
24865 7851 Acked 8850 Write 1 New Done Done true
24866 7852 Acked 8851 Write 3 New Done Done true
24867 7853 Acked 8852 Write 8 New Done Done true
24868 7854 Acked 8853 Write 2 New Done Done true
24869 7855 Acked 8854 Read 3 New Done Done true
24870 7856 Acked 8855 Write 8 New Done Done true
24871 7857 Acked 8856 Write 6 New Done Done true
24872 7858 Acked 8857 Write 2 New Done Done true
24873 7859 Acked 8858 Read 10 New Done Done true
24874 7860 Acked 8859 Read 6 New Done Done true
24875 7861 Acked 8860 Read 1 New Done Done true
24876 7862 Acked 8861 Write 6 New Done Done true
24877 7863 Acked 8862 Read 1 New Done Done true
24878 7864 Acked 8863 Read 2 New Done Done true
24879 7865 Acked 8864 Read 10 New Done Done true
24880 7866 Acked 8865 Read 7 New Done Done true
24881 7867 Acked 8866 Read 8 New Done Done true
24882 7868 Acked 8867 Flush 0 New Done Done true
24883 7869 Acked 8868 Write 2 New Done Done true
24884 7870 Acked 8869 Write 9 New Done Done true
24885 7871 Acked 8870 Write 5 New Done Done true
24886 7872 Acked 8871 Read 1 New Done Done true
24887 7873 Acked 8872 Write 2 New Done Done true
24888 7874 Acked 8873 Read 3 New Done Done true
24889 7875 Acked 8874 Read 10 New Done Done true
24890 7876 Acked 8875 Write 6 New Done Done true
24891 7877 Acked 8876 Write 5 New Done Done true
24892 7878 Acked 8877 Write 2 New Done Done true
24893 7879 Acked 8878 Write 2 New Done Done true
24894 7880 Acked 8879 Write 5 New Done Done true
24895 7881 Acked 8880 Flush 0 New Done Done true
24896 7882 Acked 8881 Read 1 New Done Done true
24897 7883 Acked 8882 Flush 0 New Done Done true
24898 7884 Acked 8883 Read 3 New Done Done true
24899 7885 Acked 8884 Flush 0 New Done Done true
24900 7886 Acked 8885 Read 10 New Done Done true
24901 7887 Acked 8886 Write 5 New Done Done true
24902 7888 Acked 8887 Read 2 New Done Done true
24903 7889 Acked 8888 Flush 0 New Done Done true
24904 7890 Acked 8889 Write 6 New Done Done true
24905 7891 Acked 8890 Flush 0 New Done Done true
24906 7892 Acked 8891 Write 3 New Done Done true
24907 7893 Acked 8892 Flush 0 New Done Done true
24908 7894 Acked 8893 Write 6 New Done Done true
24909 7895 Acked 8894 Write 4 New Done Done true
24910 7896 Acked 8895 Write 6 New Done Done true
24911 7897 Acked 8896 Read 8 New Done Done true
24912 7898 Acked 8897 Flush 0 New Done Done true
24913 7899 Acked 8898 Write 9 New Done Done true
24914 7900 Acked 8899 Write 2 New Done Done true
24915 7901 Acked 8900 Read 6 New Done Done true
24916 7902 Acked 8901 Read 7 New Done Done true
24917 7903 Acked 8902 Read 10 New Done Done true
24918 7904 Acked 8903 Write 4 New Done Done true
24919 7905 Acked 8904 Read 7 New Done Done true
24920 7906 Acked 8905 Write 10 New Done Done true
24921 7907 Acked 8906 Read 8 New Done Done true
24922 7908 Acked 8907 Read 1 New Done Done true
24923 7909 Acked 8908 Write 8 New Done Done true
24924 7910 Acked 8909 Write 2 New Done Done true
24925 7911 Acked 8910 Write 3 New Done Done true
24926 7912 Acked 8911 Read 5 New Done Done true
24927 7913 Acked 8912 Write 7 New Done Done true
24928 7914 Acked 8913 Read 7 New Done Done true
24929 7915 Acked 8914 Flush 0 New Done Done true
24930 7916 Acked 8915 Read 4 New Done Done true
24931 7917 Acked 8916 Read 5 New Done Done true
24932 7918 Acked 8917 Write 3 New Done Done true
24933 7919 Acked 8918 Write 2 New Done Done true
24934 7920 Acked 8919 Read 3 New Done Done true
24935 7921 Acked 8920 Write 7 New Done Done true
24936 7922 Acked 8921 Read 5 New Done Done true
24937 7923 Acked 8922 Write 5 New Done Done true
24938 7924 Acked 8923 Write 7 New Done Done true
24939 7925 Acked 8924 Write 8 New Done Done true
24940 7926 Acked 8925 Write 1 New Done Done true
24941 7927 Acked 8926 Flush 0 New Done Done true
24942 7928 Acked 8927 Write 2 New Done Done true
24943 7929 Acked 8928 Write 5 New Done Done true
24944 7930 Acked 8929 Write 1 New Done Done true
24945 7931 Acked 8930 Write 6 New Done Done true
24946 7932 Acked 8931 Write 2 New Done Done true
24947 7933 Acked 8932 Read 8 New Done Done true
24948 7934 Acked 8933 Read 7 New Done Done true
24949 7935 Acked 8934 Read 3 New Done Done true
24950 7936 Acked 8935 Read 7 New Done Done true
24951 7937 Acked 8936 Read 7 New Done Done true
24952 7938 Acked 8937 Write 9 New Done Done true
24953 7939 Acked 8938 Read 3 New Done Done true
24954 7940 Acked 8939 Write 9 New Done Done true
24955 7941 Acked 8940 Read 6 New Done Done true
24956 7942 Acked 8941 Read 3 New Done Done true
24957 7943 Acked 8942 Write 6 New Done Done true
24958 7944 Acked 8943 Read 7 New Done Done true
24959 7945 Acked 8944 Flush 0 New Done Done true
24960 7946 Acked 8945 Write 9 New Done Done true
24961 7947 Acked 8946 Read 7 New Done Done true
24962 7948 Acked 8947 Read 5 New Done Done true
24963 7949 Acked 8948 Write 3 New Done Done true
24964 7950 Acked 8949 Read 2 New Done Done true
24965 7951 Acked 8950 Flush 0 New Done Done true
24966 7952 Acked 8951 Write 5 New Done Done true
24967 7953 Acked 8952 Flush 0 New Done Done true
24968 7954 Acked 8953 Write 6 New Done Done true
24969 7955 Acked 8954 Read 5 New Done Done true
24970 7956 Acked 8955 Write 5 New Done Done true
24971 7957 Acked 8956 Write 10 New Done Done true
24972 7958 Acked 8957 Read 6 New Done Done true
24973 7959 Acked 8958 Write 8 New Done Done true
24974 7960 Acked 8959 Write 8 New Done Done true
24975 7961 Acked 8960 Read 3 New Done Done true
24976 7962 Acked 8961 Read 6 New Done Done true
24977 7963 Acked 8962 Write 5 New Done Done true
24978 7964 Acked 8963 Write 5 New Done Done true
24979 7965 Acked 8964 Read 10 New Done Done true
24980 7966 Acked 8965 Read 4 New Done Done true
24981 7967 Acked 8966 Flush 0 New Done Done true
24982 7968 Acked 8967 Read 6 New Done Done true
24983 7969 Acked 8968 Flush 0 New Done Done true
24984 7970 Acked 8969 Read 3 New Done Done true
24985 7971 Acked 8970 Read 6 New Done Done true
24986 7972 Acked 8971 Write 3 New Done Done true
24987 7973 Acked 8972 Flush 0 New Done Done true
24988 7974 Acked 8973 Write 7 New Done Done true
24989 7975 Acked 8974 Write 9 New Done Done true
24990 7976 Acked 8975 Write 10 New Done Done true
24991 7977 Acked 8976 Read 2 New Done Done true
24992 7978 Acked 8977 Write 9 New Done Done true
24993 7979 Acked 8978 Write 5 New Done Done true
24994 7980 Acked 8979 Read 9 New Done Done true
24995 7981 Acked 8980 Flush 0 New Done Done true
24996 7982 Acked 8981 Read 6 New Done Done true
24997 7983 Acked 8982 Write 3 New Done Done true
24998 7984 Acked 8983 Write 3 New Done Done true
24999 7985 Acked 8984 Flush 0 New Done Done true
25000 7986 Acked 8985 Read 9 New Done Done true
25001 7987 Acked 8986 Write 2 New Done Done true
25002 7988 Acked 8987 Read 1 New Done Done true
25003 7989 Acked 8988 Read 9 New Done Done true
25004 7990 Acked 8989 Write 1 New Done Done true
25005 7991 Acked 8990 Write 8 New Done Done true
25006 7992 Acked 8991 Write 10 New Done Done true
25007 7993 Acked 8992 Write 2 New Done Done true
25008 7994 Acked 8993 Write 3 New Done Done true
25009 7995 Acked 8994 Write 8 New Done Done true
25010 7996 Acked 8995 Flush 0 New Done Done true
25011 7997 Acked 8996 Read 4 New Done Done true
25012 7998 Acked 8997 Read 9 New Done Done true
25013 7999 Acked 8998 Write 6 New Done Done true
25014 8000 Acked 8999 Write 8 New Done Done true
25015 8001 Acked 9000 Write 7 New Done Done true
25016 8002 Acked 9001 Read 4 New Done Done true
25017 8003 Acked 9002 Read 6 New Done Done true
25018 8004 Acked 9003 Read 2 New Done Done true
25019 8005 Acked 9004 Read 10 New Done Done true
25020 8006 Acked 9005 Write 3 New Done Done true
25021 8007 Acked 9006 Read 6 New Done Done true
25022 8008 Acked 9007 Read 4 New Done Done true
25023 8009 Acked 9008 Read 7 New Done Done true
25024 8010 Acked 9009 Write 10 New Done Done true
25025 8011 Acked 9010 Write 4 New Done Done true
25026 8012 Acked 9011 Write 10 New Done Done true
25027 8013 Acked 9012 Write 6 New Done Done true
25028 8014 Acked 9013 Read 7 New Done Done true
25029 8015 Acked 9014 Write 1 New Done Done true
25030 8016 Acked 9015 Write 8 New Done Done true
25031 8017 Acked 9016 Write 1 New Done Done true
25032 8018 Acked 9017 Write 10 New Done Done true
25033 8019 Acked 9018 Write 6 New Done Done true
25034 8020 Acked 9019 Read 2 New Done Done true
25035 8021 Acked 9020 Read 5 New Done Done true
25036 8022 Acked 9021 Write 1 New Done Done true
25037 8023 Acked 9022 Write 3 New Done Done true
25038 8024 Acked 9023 Read 4 New Done Done true
25039 8025 Acked 9024 Read 7 New Done Done true
25040 8026 Acked 9025 Read 4 New Done Done true
25041 8027 Acked 9026 Read 5 New Done Done true
25042 8028 Acked 9027 Read 8 New Done Done true
25043 8029 Acked 9028 Write 2 New Done Done true
25044 8030 Acked 9029 Read 7 New Done Done true
25045 8031 Acked 9030 Read 1 New Done Done true
25046 8032 Acked 9031 Read 1 New Done Done true
25047 8033 Acked 9032 Write 3 New Done Done true
25048 8034 Acked 9033 Read 6 New Done Done true
25049 8035 Acked 9034 Read 6 New Done Done true
25050 8036 Acked 9035 Write 1 New Done Done true
25051 8037 Acked 9036 Read 9 New Done Done true
25052 8038 Acked 9037 Write 9 New Done Done true
25053 8039 Acked 9038 Read 3 New Done Done true
25054 8040 Acked 9039 Write 3 New Done Done true
25055 8041 Acked 9040 Write 3 New Done Done true
25056 8042 Acked 9041 Read 5 New Done Done true
25057 8043 Acked 9042 Read 7 New Done Done true
25058 8044 Acked 9043 Write 2 New Done Done true
25059 8045 Acked 9044 Write 5 New Done Done true
25060 8046 Acked 9045 Read 7 New Done Done true
25061 8047 Acked 9046 Write 10 New Done Done true
25062 8048 Acked 9047 Write 8 New Done Done true
25063 8049 Acked 9048 Flush 0 New Done Done true
25064 8050 Acked 9049 Write 5 New Done Done true
25065 8051 Acked 9050 Read 9 New Done Done true
25066 8052 Acked 9051 Read 5 New Done Done true
25067 8053 Acked 9052 Read 4 New Done Done true
25068 8054 Acked 9053 Read 8 New Done Done true
25069 8055 Acked 9054 Read 7 New Done Done true
25070 8056 Acked 9055 Read 6 New Done Done true
25071 8057 Acked 9056 Read 6 New Done Done true
25072 8058 Acked 9057 Write 8 New Done Done true
25073 8059 Acked 9058 Read 7 New Done Done true
25074 8060 Acked 9059 Read 3 New Done Done true
25075 8061 Acked 9060 Write 4 New Done Done true
25076 8062 Acked 9061 Flush 0 New Done Done true
25077 8063 Acked 9062 Read 5 New Done Done true
25078 8064 Acked 9063 Flush 0 New Done Done true
25079 8065 Acked 9064 Read 4 New Done Done true
25080 8066 Acked 9065 Read 3 New Done Done true
25081 8067 Acked 9066 Flush 0 New Done Done true
25082 8068 Acked 9067 Write 5 New Done Done true
25083 8069 Acked 9068 Read 2 New Done Done true
25084 8070 Acked 9069 Flush 0 New Done Done true
25085 8071 Acked 9070 Write 4 New Done Done true
25086 8072 Acked 9071 Read 4 New Done Done true
25087 8073 Acked 9072 Write 10 New Done Done true
25088 8074 Acked 9073 Read 8 New Done Done true
25089 8075 Acked 9074 Write 10 New Done Done true
25090 8076 Acked 9075 Write 10 New Done Done true
25091 8077 Acked 9076 Write 1 New Done Done true
25092 8078 Acked 9077 Read 7 New Done Done true
25093 8079 Acked 9078 Flush 0 New Done Done true
25094 8080 Acked 9079 Write 2 New Done Done true
25095 8081 Acked 9080 Read 9 New Done Done true
25096 8082 Acked 9081 Read 10 New Done Done true
25097 8083 Acked 9082 Read 4 New Done Done true
25098 8084 Acked 9083 Write 6 New Done Done true
25099 8085 Acked 9084 Write 5 New Done Done true
25100 8086 Acked 9085 Write 7 New Done Done true
25101 8087 Acked 9086 Read 5 New Done Done true
25102 8088 Acked 9087 Read 5 New Done Done true
25103 8089 Acked 9088 Read 9 New Done Done true
25104 8090 Acked 9089 Read 4 New Done Done true
25105 8091 Acked 9090 Flush 0 New Done Done true
25106 8092 Acked 9091 Write 2 New Done Done true
25107 8093 Acked 9092 Write 8 New Done Done true
25108 8094 Acked 9093 Write 10 New Done Done true
25109 8095 Acked 9094 Read 10 New Done Done true
25110 8096 Acked 9095 Write 5 New Done Done true
25111 8097 Acked 9096 Read 8 New Done Done true
25112 8098 Acked 9097 Read 5 New Done Done true
25113 8099 Acked 9098 Write 8 New Done Done true
25114 8100 Acked 9099 Read 1 New Done Done true
25115 8101 Acked 9100 Read 6 New Done Done true
25116 8102 Acked 9101 Flush 0 New Done Done true
25117 8103 Acked 9102 Write 6 New Done Done true
25118 8104 Acked 9103 Read 10 New Done Done true
25119 8105 Acked 9104 Read 5 New Done Done true
25120 8106 Acked 9105 Write 4 New Done Done true
25121 8107 Acked 9106 Flush 0 New Done Done true
25122 8108 Acked 9107 Read 5 New Done Done true
25123 8109 Acked 9108 Flush 0 New Done Done true
25124 8110 Acked 9109 Read 9 New Done Done true
25125 8111 Acked 9110 Read 6 New Done Done true
25126 8112 Acked 9111 Read 9 New Done Done true
25127 8113 Acked 9112 Read 2 New Done Done true
25128 8114 Acked 9113 Write 2 New Done Done true
25129 8115 Acked 9114 Write 1 New Done Done true
25130 8116 Acked 9115 Read 9 New Done Done true
25131 8117 Acked 9116 Flush 0 New Done Done true
25132 8118 Acked 9117 Read 9 New Done Done true
25133 8119 Acked 9118 Flush 0 New Done Done true
25134 8120 Acked 9119 Write 8 New Done Done true
25135 8121 Acked 9120 Read 7 New Done Done true
25136 8122 Acked 9121 Flush 0 New Done Done true
25137 8123 Acked 9122 Read 9 New Done Done true
25138 8124 Acked 9123 Write 2 New Done Done true
25139 8125 Acked 9124 Flush 0 New Done Done true
25140 8126 Acked 9125 Read 9 New Done Done true
25141 8127 Acked 9126 Read 7 New Done Done true
25142 8128 Acked 9127 Write 8 New Done Done true
25143 8129 Acked 9128 Write 8 New Done Done true
25144 8130 Acked 9129 Write 5 New Done Done true
25145 8131 Acked 9130 Write 5 New Done Done true
25146 8132 Acked 9131 Write 10 New Done Done true
25147 8133 Acked 9132 Read 4 New Done Done true
25148 8134 Acked 9133 Read 1 New Done Done true
25149 8135 Acked 9134 Write 8 New Done Done true
25150 8136 Acked 9135 Read 1 New Done Done true
25151 8137 Acked 9136 Write 4 New Done Done true
25152 8138 Acked 9137 Write 5 New Done Done true
25153 8139 Acked 9138 Read 6 New Done Done true
25154 8140 Acked 9139 Flush 0 New Done Done true
25155 STATES DS:0 DS:1 DS:2 TOTAL
25156 New 302 0 0 302
25157 Sent 0 0 0 0
25158 Done 0 302 302 604
25159 Skipped 0 0 0 0
25160 Error 0 0 0 0
25161 Last Flush: 8837 9139 9139
25162 Downstairs last five completed: 8837 8836 8835 8834 8833
25163 Upstairs last five completed: 8140 8139 8138 8137 8136
25164 CLIENT: Up:0 ds:302 act:2
251652024-02-12T01:39:28.060ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
251662024-02-12T01:39:28.060ZINFOcrucible: ds_connection connected = io task client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
251672024-02-12T01:39:28.062ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
251682024-02-12T01:39:28.062ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
251692024-02-12T01:39:28.062ZINFOcrucible: send last flush ID to this DS: 8837 = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
251702024-02-12T01:39:28.062ZINFOcrucible: Replied this last flush ID: 8837 = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
251712024-02-12T01:39:28.062ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
251722024-02-12T01:39:28.062ZINFOcrucible: [0] Transition from Offline to Replay = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
251732024-02-12T01:39:28.062ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
251742024-02-12T01:39:28.062ZINFOcrucible: [0] Transition from Replay to Active = downstairs client = 0 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
25175 ----------------------------------------------------------------
25176 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
25177 Downstairs last five completed: 9139 9138 9137 9136 9135
25178 Upstairs last five completed: 8140 8139 8138 8137 8136
25179 CLIENT: Up:0 ds:0 act:3
25180 Replay: All jobs finished, all DS active.
25181 Main task has work to do, go find it
25182 got dsc Stop(2)
25183 stop 2
25184 Feb 12 01:39:31.397[8830] Got stop action so:false kr:false
25185 INFO request completed, latency_us: 383, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 975202d7-6cce-4c85-b87d-02b006266813, remote_addr: 127.0.0.1:60456, local_addr: 127.0.0.1:9998
25186 Feb 12 01:39:31.398 INFO request completed, latency_us: 103, response_code: 200, uri: /state/cid/2, method: GET, req_id: 912485e2-d4c7-4ce7-9ff8-5e72b54ac901, remote_addr: 127.0.0.1:60456, local_addr: 127.0.0.1{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2024-02-12T01:39:31.398326631Z","hostname":"ip-10-150-1-104.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"e1773b3e-b36f-4080-8b13-c0830f0d072e"}
25187 :9998
251882024-02-12T01:39:31.398ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
251892024-02-12T01:39:31.398ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
251902024-02-12T01:39:31.398ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
251912024-02-12T01:39:31.398ZINFOcrucible: [2] client re-new 0 jobs since flush 9139 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
251922024-02-12T01:39:31.398ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
25193 [8830] Exited with: ExitStatus(unix_wait_status(9))
25194 [8830][2] reports Exit
251952024-02-12T01:39:32.399ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
251962024-02-12T01:39:32.399ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
251972024-02-12T01:39:32.399ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
251982024-02-12T01:39:32.399ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
251992024-02-12T01:39:32.399ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
252002024-02-12T01:39:32.399ZINFOcrucible: [2] client re-new 0 jobs since flush 9139 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
252012024-02-12T01:39:32.399ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
252022024-02-12T01:39:33.401ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
252032024-02-12T01:39:33.401ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
252042024-02-12T01:39:33.401ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
252052024-02-12T01:39:33.401ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
252062024-02-12T01:39:33.401ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
252072024-02-12T01:39:33.401ZINFOcrucible: [2] client re-new 0 jobs since flush 9139 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
252082024-02-12T01:39:33.401ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
252092024-02-12T01:39:34.402ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
252102024-02-12T01:39:34.402ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
252112024-02-12T01:39:34.402ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
252122024-02-12T01:39:34.402ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
252132024-02-12T01:39:34.402ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
252142024-02-12T01:39:34.402ZINFOcrucible: [2] client re-new 0 jobs since flush 9139 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
252152024-02-12T01:39:34.402ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
25216 Feb 12 01:39:35.400 INFO request completed, latency_us: 251, response_code: 200, uri: /state/cid/2, method: GET, req_id: 295618ed-1bb0-4108-bf2a-a07ee0b4b6c0, remote_addr: 127.0.0.1:60456, local_addr: 127.0.0.1:9998
25217 001/300 Flush
25218 002/300 Read block 2172 len 36864
25219 {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-02-12T01:39:35.403992752Z","hostname":"ip-10-150-1-104.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"e1773b3e-b36f-4080-8b13-c0830f0d072e"003/300} Read block 2067 len 36864
25220 
252212024-02-12T01:39:35.404ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
252222024-02-12T01:39:35.404ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
252232024-02-12T01:39:35.404ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
252242024-02-12T01:39:35.404ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
252252024-02-12T01:39:35.404ZINFOcrucible: [2] client re-new 2 jobs since flush 9139 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
252262024-02-12T01:39:35.404ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
25227 004/300 Read block 3493 len 20480
25228 005/300 Read block 1173 len 12288
25229 006/300 Write block 1861 len 16384 data: 7 8 8 6
25230 007/300 Write block 893 len 8192 data: 6 4
25231 008/300 Read block 149 len 4096
25232 009/300 Read block 2506 len 24576
25233 010/300 Write block 3791 len 20480 data: 10 10 10 8 7
25234 011/300 Read block 3876 len 12288
25235 012/300 Write block 4509 len 4096 data: 8
25236 013/300 Read block 4019 len 16384
25237 014/300 Read block 289 len 32768
25238 015/300 Write block 997 len 32768 data: 5 5 6 9 8 8 7 8
25239 016/300 Read block 3879 len 28672
25240 017/300 Read block 1531 len 24576
25241 018/300 Write block 4124 len 28672 data: 4 4 4 4 5 6 6
25242 019/300 Read block 4966 len 32768
25243 020/300 Write block 3512 len 8192 data: 6 7
25244 021/300 Write block 2684 len 8192 data: 4 3
25245 022/300 Read block 2740 len 12288
25246 023/300 Write block 2377 len 20480 data: 8 8 7 7 5
25247 024/300 Write block 913 len 36864 data: 6 5 4 7 7 7 8 8 7
25248 025/300 Read block 3675 len 40960
25249 026/300 Write block 4436 len 16384 data: 3 3 2 3
25250 027/300 Flush
25251 028/300 Flush
25252 029/300 Read block 2610 len 16384
25253 030/300 Read block 3679 len 28672
25254 031/300 Read block 3645 len 16384
25255 032/300 Write block 3243 len 8192 data: 6 6
25256 033/300 Write block 582 len 36864 data: 5 5 6 5 6 5 4 4 3
25257 034/300 Read block 348 len 8192
25258 035/300 Read block 3030 len 8192
25259 036/300 Write block 2356 len 8192 data: 4 5
25260 037/300 Write block 2047 len 24576 data: 7 6 5 6 5 5
25261 038/300 Read block 72 len 12288
25262 039/300 Write block 3101 len 36864 data: 4 4 6 7 8 6 6 4 4
25263 040/300 Write block 816 len 20480 data: 6 6 7 5 3
25264 041/300 Read block 425 len 4096
25265 042/300 Read block 3128 len 40960
25266 043/300 Write block 1029 len 36864 data: 3 3 4 4 6 7 8 6 5
25267 044/300 Write block 4833 len 4096 data: 7
25268 045/300 Read block 16 len 28672
25269 046/300 Read block 1358 len 28672
25270 047/300 Read block 2769 len 36864
25271 048/300 Write block 3700 len 20480 data: 8 7 6 6 5
25272 049/300 Flush
25273 050/300 Write block 4984 len 36864 data: 2 2 3 3 5 5 5 6 7
25274 051/300 Read block 2213 len 4096
25275 052/300 Read block 4138 len 20480
25276 053/300 Write block 2793 len 28672 data: 4 5 6 6 6 5 6
25277 054/300 Read block 1595 len 8192
25278 055/300 Read block 471 len 4096
25279 056/300 Read block 1667 len 28672
25280 057/300 Flush
25281 058/300 Write block 3351 len 24576 data: 4 3 3 3 4 3
25282 059/300 Read block 2049 len 16384
25283 060/300 Write block 2759 len 4096 data: 9
25284 061/300 Read block 51 len 16384
25285 062/300 Read block 2962 len 28672
25286 063/300 Read block 3274 len 20480
25287 064/300 Read block 3803 len 32768
25288 065/300 Read block 2093 len 40960
25289 066/300 Read block 2188 len 8192
25290 067/300 Read block 4660 len 12288
25291 068/300 Read block 4271 len 40960
25292 069/300 Read block 2809 len 32768
25293 070/300 Read block 224 len 24576
25294 071/300 Read block 1839 len 28672
25295 072/300 Read block 2383 len 28672
25296 073/300 Read block 3684 len 24576
25297 074/300 Write block 2703 len 24576 data: 5 3 3 4 4 5
25298 075/300 Write block 466 len 24576 data: 6 7 6 6 5 6
25299 076/300 Write block 325 len 8192 data: 5 4
25300 077/300 Read block 1504 len 40960
25301 078/300 Write block 1626 len 8192 data: 6 5
25302 079/300 Read block 319 len 16384
25303 080/300 Read block 1104 len 4096
25304 081/300 Read block 2602 len 32768
25305 082/300 Read block 4108 len 20480
25306 083/300 Flush
25307 084/300 Read block 4808 len 24576
25308 085/300 Read block 2059 len 32768
25309 086/300 Read block 2513 len 20480
25310 087/300 Read block 2929 len 32768
25311 088/300 Read block 2904 len 32768
25312 089/300 Read block 1026 len 16384
25313 090/300 Write block 3321 len 8192 data: 6 5
25314 091/300 Flush
25315 092/300 Read block 3433 len 40960
25316 093/300 Write block 4534 len 8192 data: 8 7
25317 094/300 Read block 868 len 28672
25318 095/300 Write block 1433 len 28672 data: 6 5 4 5 6 6 6
25319 096/300 Read block 1921 len 12288
25320 097/300 Flush
25321 098/300 Read block 3466 len 8192
25322 099/300 Read block 3574 len 4096
25323 100/300 Read block 2549 len 12288
25324 101/300 Read block 2519 len 4096
25325 102/300 Flush
25326 103/300 Flush
25327 104/300 Read block 364 len 8192
25328 105/300 Read block 571 len 40960
25329 106/300 Write block 215 len 16384 data: 3 2 3 4
25330 107/300 Read block 2508 len 36864
25331 108/300 Write block 4104 len 4096 data: 5
25332 109/300 Write block 888 len 36864 data: 8 4 4 4 5 7 5 5 5
25333 110/300 Write block 217 len 16384 data: 4 5 4 5
25334 111/300 Write block 2723 len 16384 data: 6 6 6 7
25335 112/300 Read block 446 len 12288
25336 113/300 Write block 2008 len 4096 data: 3
25337 114/300 Flush
25338 115/300 Flush
25339 116/300 Read block 2405 len 20480
25340 117/300 Read block 3017 len 32768
25341 118/300 Read block 4714 len 24576
25342 119/300 Write block 473 len 36864 data: 6 5 5 6 6 5 4 3 3
25343 120/300 Read block 4213 len 12288
25344 121/300 Read block 1955 len 28672
25345 122/300 Write block 2946 len 24576 data: 4 5 5 6 5 3
25346 123/300 Read block 4670 len 12288
25347 124/300 Read block 1640 len 24576
25348 125/300 Flush
25349 126/300 Read block 235 len 40960
25350 127/300 Read block 4463 len 20480
25351 128/300 Read block 4113 len 16384
25352 129/300 Read block 3204 len 12288
25353 130/300 Read block 808 len 4096
25354 131/300 Read block 2126 len 12288
25355 132/300 Read block 3279 len 36864
25356 133/300 Write block 1150 len 8192 data: 5 4
25357 134/300 Read block 4451 len 12288
25358 135/300 Read block 1218 len 32768
25359 136/300 Write block 3186 len 8192 data: 11 10
25360 137/300 Write block 1040 len 36864 data: 4 4 4 3 5 5 5 5 6
25361 138/300 Read block 1672 len 36864
25362 139/300 Read block 1184 len 36864
25363 140/300 Read block 2898 len 40960
25364 141/300 Read block 2105 len 32768
25365 142/300 Read block 338 len 20480
25366 143/300 Read block 3075 len 24576
25367 144/300 Read block 2290 len 16384
25368 145/300 Flush
25369 146/300 Read block 3584 len 12288
25370 147/300 Read block 4525 len 8192
25371 148/300 Read block 287 len 8192
25372 149/300 Flush
25373 150/300 Write block 4551 len 28672 data: 9 10 8 8 4 4 4
25374 151/300 Write block 1267 len 20480 data: 5 4 6 5 6
25375 152/300 Read block 2168 len 16384
25376 153/300 Write block 709 len 12288 data: 6 6 8
25377 154/300 Read block 3244 len 36864
25378 155/300 Read block 4602 len 36864
25379 156/300 Write block 4088 len 40960 data: 4 4 3 4 4 3 3 6 5 5
25380 157/300 Write block 2227 len 16384 data: 3 2 3 3
25381 158/300 Write block 256 len 32768 data: 8 7 7 8 7 5 5 6
25382 159/300 Write block 3776 len 20480 data: 10 9 6 8 8
25383 160/300 Flush
25384 161/300 Read block 2283 len 8192
25385 162/300 Read block 335 len 28672
25386 163/300 Write block 4374 len 36864 data: 4 4 4 4 3 4 5 5 4
25387 164/300 Write block 2620 len 40960 data: 7 7 5 6 6 8 5 5 5 7
25388 165/300 Write block 707 len 12288 data: 6 6 7
25389 166/300 Read block 4866 len 8192
25390 167/300 Read block 1440 len 12288
25391 168/300 Flush
25392 169/300 Read block 4624 len 16384
25393 170/300 Write block 1228 len 12288 data: 4 3 3
25394 171/300 Read block 3604 len 4096
25395 172/300 Read block 2968 len 28672
25396 173/300 Read block 3292 len 24576
25397 174/300 Read block 1960 len 16384
25398 175/300 Flush
25399 176/300 Write block 2755 len 20480 data: 7 8 9 9 10
25400 177/300 Read block 4415 len 28672
25401 178/300 Write block 4256 len 8192 data: 6 6
25402 179/300 Write block 727 len 36864 data: 4 4 4 5 4 5 5 4 4
25403 180/300 Read block 3825 len 40960
25404 181/300 Write block 4244 len 16384 data: 6 7 8 8
25405 182/300 Write block 828 len 12288 data: 2 3 3
25406 183/300 Write block 3709 len 32768 data: 3 3 4 6 5 6 7 8
25407 184/300 Read block 3982 len 4096
25408 185/300 Read block 3886 len 40960
25409 186/300 Read block 1822 len 36864
25410 187/300 Read block 3854 len 16384
25411 188/300 Read block 4003 len 4096
25412 189/300 Read block 393 len 12288
25413 190/300 Write block 600 len 16384 data: 5 4 5 5
25414 191/300 Write block 2726 len 16384 data: 8 6 6 5
25415 192/300 Read block 2647 len 20480
25416 193/300 Write block 1277 len 20480 data: 8 6 5 5 5
25417 194/300 Read block 2952 len 36864
25418 195/300 Write block 812 len 16384 data: 5 5 4 5
25419 196/300 Read block 2827 len 36864
25420 197/300 Write block 3554 len 28672 data: 4 5 6 5 5 5 6
25421 198/300 Read block 19 len 40960
25422 199/300 Write block 320 len 12288 data: 6 5 6
25423 200/300 Read block 1615 len 32768
25424 201/300 Read block 2788 len 28672
25425 202/300 Read block 1453 len 40960
25426 203/300 Read block 1574 len 28672
25427 204/300 Write block 2720 len 40960 data: 6 4 4 7 7 7 9 7 7 6
25428 205/300 Write block 1717 len 36864 data: 6 5 6 5 4 5 3 3 4
25429 206/300 Write block 2377 len 32768 data: 9 9 8 8 6 6 8 6
25430 207/300 Read block 2157 len 12288
25431 208/300 Write block 257 len 8192 data: 8 8
25432 209/300 Write block 897 len 32768 data: 5 5 6 6 6 7 4 4
25433 210/300 Read block 2687 len 36864
25434 211/300 Write block 2824 len 24576 data: 9 10 7 6 6 11
25435 212/300 Read block 3417 len 4096
25436 213/300 Read block 3077 len 4096
25437 214/300 Read block 1202 len 16384
25438 215/300 Read block 1293 len 32768
25439 216/300 Write block 2672 len 12288 data: 4 4 3
25440 217/300 Read block 1562 len 4096
25441 218/300 Flush
25442 219/300 Read block 2395 len 12288
25443 220/300 Read block 2336 len 16384
25444 221/300 Write block 4369 len 8192 data: 4 5
25445 222/300 Write block 335 len 32768 data: 8 8 5 5 4 4 5 4
25446 223/300 Flush
25447 224/300 Flush
25448 225/300 Write block 547 len 40960 data: 5 5 6 6 4 5 5 5 6 6
25449 226/300 Read block 2915 len 20480
25450 227/300 Write block 418 len 24576 data: 6 7 5 7 5 3
25451 228/300 Read block 2824 len 40960
25452 229/300 Read block 76 len 36864
25453 230/300 Write block 3398 len 40960 data: 6 8 10 9 7 7 8 5 5 4
25454 231/300 Flush
25455 232/300 Read block 2688 len 16384
25456 233/300 Read block 613 len 8192
25457 234/300 Write block 1453 len 32768 data: 7 6 6 6 6 6 4 6
25458 235/300 Write block 2326 len 16384 data: 4 3 5 6
25459 236/300 Write block 3072 len 20480 data: 3 3 3 3 4
25460 237/300 Write block 3752 len 20480 data: 5 6 6 6 5
25461 238/300 Flush
25462 239/300 Write block 3425 len 32768 data: 8 7 6 7 9 9 8 6
25463 240/300 Write block 912 len 4096 data: 6
25464 241/300 Read block 1961 len 40960
25465 242/300 Flush
25466 243/300 Write block 4950 len 32768 data: 9 6 6 5 5 4 7 9
25467 244/300 Write block 3849 len 4096 data: 2
25468 245/300 Read block 2982 len 20480
25469 246/300 Write block 2413 len 28672 data: 5 4 4 5 5 4 4
25470 247/300 Write block 4222 len 40960 data: 9 9 9 9 8 6 6 3 3 4
25471 248/300 Read block 94 len 32768
25472 249/300 Write block 954 len 36864 data: 6 4 6 5 6 5 3 3 3
25473 250/300 Write block 497 len 32768 data: 8 7 7 8 7 6 5 5
25474 251/300 Write block 734 len 16384 data: 5 5 2 3
25475 252/300 Flush
25476 253/300 Read block 4967 len 24576
25477 254/300 Write block 4161 len 40960 data: 3 4 5 5 7 6 4 5 7 7
25478 255/300 Write block 2809 len 28672 data: 12 11 13 12 10 7 5
25479 256/300 Flush
25480 257/300 Write block 1994 len 28672 data: 6 4 5 7 6 4 4
25481 258/300 Read block 214 len 40960
25482 259/300 Flush
25483 260/300 Read block 1812 len 16384
25484 261/300 Read block 2844 len 12288
25485 262/300 Read block 2662 len 32768
25486 263/300 Write block 4302 len 40960 data: 5 5 5 5 5 4 7 6 5 5
25487 264/300 Write block 1696 len 16384 data: 4 6 5 6
25488 265/300 Read block 3546 len 4096
25489 266/300 Read block 3578 len 4096
25490 267/300 Read block 1121 len 32768
25491 268/300 Write block 4887 len 16384 data: 6 4 5 6
25492 269/300 Read block 1849 len 32768
25493 270/300 Write block 3291 len 4096 data: 5
25494 271/300 Write block 1370 len 4096 data: 4
25495 272/300 Read block 2566 len 8192
25496 273/300 Read block 1186 len 16384
25497 274/300 Write block 3748 len 20480 data: 7 7 7 6 6
25498 275/300 Read block 4178 len 40960
25499 276/300 Write block 3033 len 16384 data: 5 6 7 8
25500 277/300 Write block 1859 len 32768 data: 7 7 8 9 9 7 6 5
25501 278/300 Read block 3815 len 4096
25502 279/300 Read block 3115 len 8192
25503 280/300 Write block 3352 len 36864 data: 4 4 4 5 4 3 5 7 7
25504 281/300 Write block 4692 len 32768 data: 8 8 6 5 7 6 7 6
25505 282/300 Read block 2002 len 12288
25506 283/300 Write block 4539 len 36864 data: 10 8 8 6 5 3 4 4 6
25507 284/300 Flush
25508 285/300 Read block 4602 len 16384
25509 286/300 Read block 3060 len 8192
25510 287/300 Read block 1389 len 40960
25511 288/300 Write block 1545 len 24576 data: 7 6 6 5 5 5
25512 289/300 Write block 4232 len 4096 data: 5
25513 290/300 Write block 3426 len 24576 data: 8 7 8 10 10 9
25514 291/300 Read block 1775 len 40960
25515 292/300 Write block 1301 len 16384 data: 5 6 4 5
25516 293/300 Read block 4507 len 8192
25517 294/300 Read block 720 len 24576
25518 295/300 Write block 2321 len 24576 data: 7 5 6 6 5 5
25519 296/300 Read block 4437 len 16384
25520 297/300 Read block 4184 len 16384
25521 298/300 Write block 4171 len 40960 data: 8 9 10 11 7 7 7 7 8 8
25522 299/300 Write block 194 len 12288 data: 6 6 7
25523 300/300 Flush
25524 Main task has work to do, go find it
25525 got dsc Start(2)
25526 start 2
25527 Feb 12 01:39:35.746 INFO request completed, latency_us: 250, response_code: 204, uri: /start/cid/2, method: POST, req_id: b9628f92-8ff4-4201-a4df-71472326ddb3, remote_addr: 127.0.0.1:60456, local_addr: 127.0.0.1:9998
25528 Replay: started 2, returned:Ok(())
25529 ----------------------------------------------------------------
25530 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:300
25531 States: Active Active Offline
25532 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
25533 8141 Acked 9140 Flush 0 Done Done New true
25534 8142 Acked 9141 Read 9 Done Done New true
25535 8143 Acked 9142 Read 9 Done Done New false
25536 8144 Acked 9143 Read 5 Done Done New false
25537 8145 Acked 9144 Read 3 Done Done New false
25538 8146 Acked 9145 Write 4 Done Done New false
25539 8147 Acked 9146 Write 2 Done Done New false
25540 8148 Acked 9147 Read 1 Done Done New false
25541 8149 Acked 9148 Read 6 Done Done New false
25542 8150 Acked 9149 Write 5 Done Done New false
25543 8151 Acked 9150 Read 3 Done Done New false
25544 8152 Acked 9151 Write 1 Done Done New false
25545 8153 Acked 9152 Read 4 Done Done New false
25546 8154 Acked 9153 Read 8 Done Done New false
25547 8155 Acked 9154 Write 8 Done Done New false
25548 8156 Acked 9155 Read 7 Done Done New false
25549 8157 Acked 9156 Read 6 Done Done New false
25550 8158 Acked 9157 Write 7 Done Done New false
25551 8159 Acked 9158 Read 8 Done Done New false
25552 8160 Acked 9159 Write 2 Done Done New false
25553 8161 Acked 9160 Write 2 Done Done New false
25554 8162 Acked 9161 Read 3 Done Done New false
25555 8163 Acked 9162 Write 5 Done Done New false
25556 8164 Acked 9163 Write 9 Done Done New false
25557 8165 Acked 9164 Read 10 Done Done New false
25558 8166 Acked 9165 Write 4 Done Done New false
25559 8167 Acked 9166 Flush 0 Done Done New false
25560 8168 Acked 9167 Flush 0 Done Done New false
25561 8169 Acked 9168 Read 4 Done Done New false
25562 8170 Acked 9169 Read 7 Done Done New false
25563 8171 Acked 9170 Read 4 Done Done New false
25564 8172 Acked 9171 Write 2 Done Done New false
25565 8173 Acked 9172 Write 9 Done Done New false
25566 8174 Acked 9173 Read 2 Done Done New false
25567 8175 Acked 9174 Read 2 Done Done New false
25568 8176 Acked 9175 Write 2 Done Done New false
25569 8177 Acked 9176 Write 6 Done Done New false
25570 8178 Acked 9177 Read 3 Done Done New false
25571 8179 Acked 9178 Write 9 Done Done New false
25572 8180 Acked 9179 Write 5 Done Done New false
25573 8181 Acked 9180 Read 1 Done Done New false
25574 8182 Acked 9181 Read 10 Done Done New false
25575 8183 Acked 9182 Write 9 Done Done New false
25576 8184 Acked 9183 Write 1 Done Done New false
25577 8185 Acked 9184 Read 7 Done Done New false
25578 8186 Acked 9185 Read 7 Done Done New false
25579 8187 Acked 9186 Read 9 Done Done New false
25580 8188 Acked 9187 Write 5 Done Done New false
25581 8189 Acked 9188 Flush 0 Done Done New false
25582 8190 Acked 9189 Write 9 Done Done New false
25583 8191 Acked 9190 Read 1 Done Done New false
25584 8192 Acked 9191 Read 5 Done Done New false
25585 8193 Acked 9192 Write 7 Done Done New false
25586 8194 Acked 9193 Read 2 Done Done New false
25587 8195 Acked 9194 Read 1 Done Done New false
25588 8196 Acked 9195 Read 7 Done Done New false
25589 8197 Acked 9196 Flush 0 Done Done New false
25590 8198 Acked 9197 Write 6 Done Done New false
25591 8199 Acked 9198 Read 4 Done Done New false
25592 8200 Acked 9199 Write 1 Done Done New false
25593 8201 Acked 9200 Read 4 Done Done New false
25594 8202 Acked 9201 Read 7 Done Done New false
25595 8203 Acked 9202 Read 5 Done Done New false
25596 8204 Acked 9203 Read 8 Done Done New false
25597 8205 Acked 9204 Read 10 Done Done New false
25598 8206 Acked 9205 Read 2 Done Done New false
25599 8207 Acked 9206 Read 3 Done Done New false
25600 8208 Acked 9207 Read 10 Done Done New false
25601 8209 Acked 9208 Read 8 Done Done New false
25602 8210 Acked 9209 Read 6 Done Done New false
25603 8211 Acked 9210 Read 7 Done Done New false
25604 8212 Acked 9211 Read 7 Done Done New false
25605 8213 Acked 9212 Read 6 Done Done New false
25606 8214 Acked 9213 Write 6 Done Done New false
25607 8215 Acked 9214 Write 6 Done Done New false
25608 8216 Acked 9215 Write 2 Done Done New false
25609 8217 Acked 9216 Read 10 Done Done New false
25610 8218 Acked 9217 Write 2 Done Done New false
25611 8219 Acked 9218 Read 4 Done Done New false
25612 8220 Acked 9219 Read 1 Done Done New false
25613 8221 Acked 9220 Read 8 Done Done New false
25614 8222 Acked 9221 Read 5 Done Done New false
25615 8223 Acked 9222 Flush 0 Done Done New false
25616 8224 Acked 9223 Read 6 Done Done New false
25617 8225 Acked 9224 Read 8 Done Done New false
25618 8226 Acked 9225 Read 5 Done Done New false
25619 8227 Acked 9226 Read 8 Done Done New false
25620 8228 Acked 9227 Read 8 Done Done New false
25621 8229 Acked 9228 Read 4 Done Done New false
25622 8230 Acked 9229 Write 2 Done Done New false
25623 8231 Acked 9230 Flush 0 Done Done New false
25624 8232 Acked 9231 Read 10 Done Done New false
25625 8233 Acked 9232 Write 2 Done Done New false
25626 8234 Acked 9233 Read 7 Done Done New false
25627 8235 Acked 9234 Write 7 Done Done New false
25628 8236 Acked 9235 Read 3 Done Done New false
25629 8237 Acked 9236 Flush 0 Done Done New false
25630 8238 Acked 9237 Read 2 Done Done New false
25631 8239 Acked 9238 Read 1 Done Done New false
25632 8240 Acked 9239 Read 3 Done Done New false
25633 8241 Acked 9240 Read 1 Done Done New false
25634 8242 Acked 9241 Flush 0 Done Done New false
25635 8243 Acked 9242 Flush 0 Done Done New false
25636 8244 Acked 9243 Read 2 Done Done New false
25637 8245 Acked 9244 Read 10 Done Done New false
25638 8246 Acked 9245 Write 4 Done Done New false
25639 8247 Acked 9246 Read 9 Done Done New false
25640 8248 Acked 9247 Write 1 Done Done New false
25641 8249 Acked 9248 Write 9 Done Done New false
25642 8250 Acked 9249 Write 4 Done Done New false
25643 8251 Acked 9250 Write 4 Done Done New false
25644 8252 Acked 9251 Read 3 Done Done New false
25645 8253 Acked 9252 Write 1 Done Done New false
25646 8254 Acked 9253 Flush 0 Done Done New false
25647 8255 Acked 9254 Flush 0 Done Done New false
25648 8256 Acked 9255 Read 5 Done Done New false
25649 8257 Acked 9256 Read 8 Done Done New false
25650 8258 Acked 9257 Read 6 Done Done New false
25651 8259 Acked 9258 Write 9 Done Done New false
25652 8260 Acked 9259 Read 3 Done Done New false
25653 8261 Acked 9260 Read 7 Done Done New false
25654 8262 Acked 9261 Write 6 Done Done New false
25655 8263 Acked 9262 Read 3 Done Done New false
25656 8264 Acked 9263 Read 6 Done Done New false
25657 8265 Acked 9264 Flush 0 Done Done New false
25658 8266 Acked 9265 Read 10 Done Done New false
25659 8267 Acked 9266 Read 5 Done Done New false
25660 8268 Acked 9267 Read 4 Done Done New false
25661 8269 Acked 9268 Read 3 Done Done New false
25662 8270 Acked 9269 Read 1 Done Done New false
25663 8271 Acked 9270 Read 3 Done Done New false
25664 8272 Acked 9271 Read 9 Done Done New false
25665 8273 Acked 9272 Write 2 Done Done New false
25666 8274 Acked 9273 Read 3 Done Done New false
25667 8275 Acked 9274 Read 8 Done Done New false
25668 8276 Acked 9275 Write 2 Done Done New false
25669 8277 Acked 9276 Write 9 Done Done New false
25670 8278 Acked 9277 Read 9 Done Done New false
25671 8279 Acked 9278 Read 9 Done Done New false
25672 8280 Acked 9279 Read 10 Done Done New false
25673 8281 Acked 9280 Read 8 Done Done New false
25674 8282 Acked 9281 Read 5 Done Done New false
25675 8283 Acked 9282 Read 6 Done Done New false
25676 8284 Acked 9283 Read 4 Done Done New false
25677 8285 Acked 9284 Flush 0 Done Done New false
25678 8286 Acked 9285 Read 3 Done Done New false
25679 8287 Acked 9286 Read 2 Done Done New false
25680 8288 Acked 9287 Read 2 Done Done New false
25681 8289 Acked 9288 Flush 0 Done Done New false
25682 8290 Acked 9289 Write 7 Done Done New false
25683 8291 Acked 9290 Write 5 Done Done New false
25684 8292 Acked 9291 Read 4 Done Done New false
25685 8293 Acked 9292 Write 3 Done Done New false
25686 8294 Acked 9293 Read 9 Done Done New false
25687 8295 Acked 9294 Read 9 Done Done New false
25688 8296 Acked 9295 Write 10 Done Done New false
25689 8297 Acked 9296 Write 4 Done Done New false
25690 8298 Acked 9297 Write 8 Done Done New false
25691 8299 Acked 9298 Write 5 Done Done New false
25692 8300 Acked 9299 Flush 0 Done Done New false
25693 8301 Acked 9300 Read 2 Done Done New false
25694 8302 Acked 9301 Read 7 Done Done New false
25695 8303 Acked 9302 Write 9 Done Done New false
25696 8304 Acked 9303 Write 10 Done Done New false
25697 8305 Acked 9304 Write 3 Done Done New false
25698 8306 Acked 9305 Read 2 Done Done New false
25699 8307 Acked 9306 Read 3 Done Done New false
25700 8308 Acked 9307 Flush 0 Done Done New false
25701 8309 Acked 9308 Read 4 Done Done New false
25702 8310 Acked 9309 Write 3 Done Done New false
25703 8311 Acked 9310 Read 1 Done Done New false
25704 8312 Acked 9311 Read 7 Done Done New false
25705 8313 Acked 9312 Read 6 Done Done New false
25706 8314 Acked 9313 Read 4 Done Done New false
25707 8315 Acked 9314 Flush 0 Done Done New false
25708 8316 Acked 9315 Write 5 Done Done New false
25709 8317 Acked 9316 Read 7 Done Done New false
25710 8318 Acked 9317 Write 2 Done Done New false
25711 8319 Acked 9318 Write 9 Done Done New false
25712 8320 Acked 9319 Read 10 Done Done New false
25713 8321 Acked 9320 Write 4 Done Done New false
25714 8322 Acked 9321 Write 3 Done Done New false
25715 8323 Acked 9322 Write 8 Done Done New false
25716 8324 Acked 9323 Read 1 Done Done New false
25717 8325 Acked 9324 Read 10 Done Done New false
25718 8326 Acked 9325 Read 9 Done Done New false
25719 8327 Acked 9326 Read 4 Done Done New false
25720 8328 Acked 9327 Read 1 Done Done New false
25721 8329 Acked 9328 Read 3 Done Done New false
25722 8330 Acked 9329 Write 4 Done Done New false
25723 8331 Acked 9330 Write 4 Done Done New false
25724 8332 Acked 9331 Read 5 Done Done New false
25725 8333 Acked 9332 Write 5 Done Done New false
25726 8334 Acked 9333 Read 9 Done Done New false
25727 8335 Acked 9334 Write 4 Done Done New false
25728 8336 Acked 9335 Read 9 Done Done New false
25729 8337 Acked 9336 Write 7 Done Done New false
25730 8338 Acked 9337 Read 10 Done Done New false
25731 8339 Acked 9338 Write 3 Done Done New false
25732 8340 Acked 9339 Read 8 Done Done New false
25733 8341 Acked 9340 Read 7 Done Done New false
25734 8342 Acked 9341 Read 10 Done Done New false
25735 8343 Acked 9342 Read 7 Done Done New false
25736 8344 Acked 9343 Write 10 Done Done New false
25737 8345 Acked 9344 Write 9 Done Done New false
25738 8346 Acked 9345 Write 8 Done Done New false
25739 8347 Acked 9346 Read 3 Done Done New false
25740 8348 Acked 9347 Write 2 Done Done New false
25741 8349 Acked 9348 Write 8 Done Done New false
25742 8350 Acked 9349 Read 9 Done Done New false
25743 8351 Acked 9350 Write 6 Done Done New false
25744 8352 Acked 9351 Read 1 Done Done New false
25745 8353 Acked 9352 Read 1 Done Done New false
25746 8354 Acked 9353 Read 4 Done Done New false
25747 8355 Acked 9354 Read 8 Done Done New false
25748 8356 Acked 9355 Write 3 Done Done New false
25749 8357 Acked 9356 Read 1 Done Done New false
25750 8358 Acked 9357 Flush 0 Done Done New false
25751 8359 Acked 9358 Read 3 Done Done New false
25752 8360 Acked 9359 Read 4 Done Done New false
25753 8361 Acked 9360 Write 2 Done Done New false
25754 8362 Acked 9361 Write 8 Done Done New false
25755 8363 Acked 9362 Flush 0 Done Done New false
25756 8364 Acked 9363 Flush 0 Done Done New false
25757 8365 Acked 9364 Write 10 Done Done New false
25758 8366 Acked 9365 Read 5 Done Done New false
25759 8367 Acked 9366 Write 6 Done Done New false
25760 8368 Acked 9367 Read 10 Done Done New false
25761 8369 Acked 9368 Read 9 Done Done New false
25762 8370 Acked 9369 Write 10 Done Done New false
25763 8371 Acked 9370 Flush 0 Done Done New false
25764 8372 Acked 9371 Read 4 Done Done New false
25765 8373 Acked 9372 Read 2 Done Done New false
25766 8374 Acked 9373 Write 8 Done Done New false
25767 8375 Acked 9374 Write 4 Done Done New false
25768 8376 Acked 9375 Write 5 Done Done New false
25769 8377 Acked 9376 Write 5 Done Done New false
25770 8378 Acked 9377 Flush 0 Done Done New false
25771 8379 Acked 9378 Write 8 Done Done New false
25772 8380 Acked 9379 Write 1 Done Done New false
25773 8381 Acked 9380 Read 10 Done Done New false
25774 8382 Acked 9381 Flush 0 Done Done New false
25775 8383 Acked 9382 Write 8 Done Done New false
25776 8384 Acked 9383 Write 1 Done Done New false
25777 8385 Acked 9384 Read 5 Done Done New false
25778 8386 Acked 9385 Write 7 Done Done New false
25779 8387 Acked 9386 Write 10 Done Done New false
25780 8388 Acked 9387 Read 8 Done Done New false
25781 8389 Acked 9388 Write 9 Done Done New false
25782 8390 Acked 9389 Write 8 Done Done New false
25783 8391 Acked 9390 Write 4 Done Done New false
25784 8392 Acked 9391 Flush 0 Done Done New false
25785 8393 Acked 9392 Read 6 Done Done New false
25786 8394 Acked 9393 Write 10 Done Done New false
25787 8395 Acked 9394 Write 7 Done Done New false
25788 8396 Acked 9395 Flush 0 Done Done New false
25789 8397 Acked 9396 Write 7 Done Done New false
25790 8398 Acked 9397 Read 10 Done Done New false
25791 8399 Acked 9398 Flush 0 Done Done New false
25792 8400 Acked 9399 Read 4 Done Done New false
25793 8401 Acked 9400 Read 3 Done Done New false
25794 8402 Acked 9401 Read 8 Done Done New false
25795 8403 Acked 9402 Write 10 Done Done New false
25796 8404 Acked 9403 Write 4 Done Done New false
25797 8405 Acked 9404 Read 1 Done Done New false
25798 8406 Acked 9405 Read 1 Done Done New false
25799 8407 Acked 9406 Read 8 Done Done New false
25800 8408 Acked 9407 Write 4 Done Done New false
25801 8409 Acked 9408 Read 8 Done Done New false
25802 8410 Acked 9409 Write 1 Done Done New false
25803 8411 Acked 9410 Write 1 Done Done New false
25804 8412 Acked 9411 Read 2 Done Done New false
25805 8413 Acked 9412 Read 4 Done Done New false
25806 8414 Acked 9413 Write 5 Done Done New false
25807 8415 Acked 9414 Read 10 Done Done New false
25808 8416 Acked 9415 Write 4 Done Done New false
25809 8417 Acked 9416 Write 8 Done Done New false
25810 8418 Acked 9417 Read 1 Done Done New false
25811 8419 Acked 9418 Read 2 Done Done New false
25812 8420 Acked 9419 Write 9 Done Done New false
25813 8421 Acked 9420 Write 8 Done Done New false
25814 8422 Acked 9421 Read 3 Done Done New false
25815 8423 Acked 9422 Write 9 Done Done New false
25816 8424 Acked 9423 Flush 0 Done Done New false
25817 8425 Acked 9424 Read 4 Done Done New false
25818 8426 Acked 9425 Read 2 Done Done New false
25819 8427 Acked 9426 Read 10 Done Done New false
25820 8428 Acked 9427 Write 6 Done Done New false
25821 8429 Acked 9428 Write 1 Done Done New false
25822 8430 Acked 9429 Write 6 Done Done New false
25823 8431 Acked 9430 Read 10 Done Done New false
25824 8432 Acked 9431 Write 4 Done Done New false
25825 8433 Acked 9432 Read 2 Done Done New false
25826 8434 Acked 9433 Read 6 Done Done New false
25827 8435 Acked 9434 Write 6 Done Done New false
25828 8436 Acked 9435 Read 4 Done Done New false
25829 8437 Acked 9436 Read 4 Done Done New false
25830 8438 Acked 9437 Write 10 Done Done New false
25831 8439 Acked 9438 Write 3 Done Done New false
25832 8440 Acked 9439 Flush 0 Done Done New false
25833 STATES DS:0 DS:1 DS:2 TOTAL
25834 New 0 0 300 300
25835 Sent 0 0 0 0
25836 Done 300 300 0 600
25837 Skipped 0 0 0 0
25838 Error 0 0 0 0
25839 Last Flush: 9439 9439 9139
25840 Downstairs last five completed: 9139 9138 9137 9136 9135
25841 Upstairs last five completed: 8440 8439 8438 8437 8436
25842 CLIENT: Up:0 ds:300 act:2
258432024-02-12T01:39:36.405ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
258442024-02-12T01:39:36.405ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
258452024-02-12T01:39:36.405ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
258462024-02-12T01:39:36.405ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
258472024-02-12T01:39:36.405ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
258482024-02-12T01:39:36.405ZINFOcrucible: [2] client re-new 300 jobs since flush 9139 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
258492024-02-12T01:39:36.406ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
25850 [8830] got start action
25851 [8830] I am going to restart
25852 Starting downstairs at port 8830
25853 Make output file at "/tmp/dsc/downstairs-8830.txt"
25854 [8830][2] reports Starting
258552024-02-12T01:39:37.407ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
258562024-02-12T01:39:37.407ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
258572024-02-12T01:39:37.407ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
258582024-02-12T01:39:37.407ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
258592024-02-12T01:39:37.407ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
258602024-02-12T01:39:37.407ZINFOcrucible: [2] client re-new 300 jobs since flush 9139 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
258612024-02-12T01:39:37.407ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
25862 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1192)
25863 [8830][2] reports Running
258642024-02-12T01:39:38.409ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
258652024-02-12T01:39:38.409ZINFOcrucible: ds_connection connected = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
258662024-02-12T01:39:38.410ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
258672024-02-12T01:39:38.410ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
258682024-02-12T01:39:38.410ZINFOcrucible: send last flush ID to this DS: 9139 = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
258692024-02-12T01:39:38.410ZINFOcrucible: Replied this last flush ID: 9139 = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
258702024-02-12T01:39:38.410ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
258712024-02-12T01:39:38.410ZINFOcrucible: [2] Transition from Offline to Replay = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
258722024-02-12T01:39:38.410ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
258732024-02-12T01:39:38.410ZINFOcrucible: [2] Transition from Replay to Active = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
25874 ----------------------------------------------------------------
25875 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
25876 Downstairs last five completed: 9439 9438 9437 9436 9435
25877 Upstairs last five completed: 8440 8439 8438 8437 8436
25878 CLIENT: Up:0 ds:0 act:3
25879 Replay: All jobs finished, all DS active.
25880 Main task has work to do, go find it
25881 got dsc Stop(1)
25882 stop 1
25883 [8820] Got stop action so:false kr:false
25884 Feb 12 01:39:39.750 INFO request completed, latency_us: 373, response_code: 204, uri: /stop/cid/1, method: POST, req_id: 93f14533-d32c-4bc8-b6df-6292ca37d056, remote_addr: 127.0.0.1:60456, local_addr: 127.0.0.1:9998
25885 Feb 12 01:39:39.751 INFO request completed, latency_us: 161, response_code: 200, uri: /state/cid/1, method: GET, req_id: b9c830f8-e3d4-41bf-afbe-148196df79c7{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2024-02-12T01:39:39.751202671Z","hostname":"ip-10-150-1-104.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"e1773b3e-b36f-4080-8b13-c0830f0d072e"}
25886 , remote_addr: 127.0.0.1:60456, local_addr: 127.0.0.1:9998
258872024-02-12T01:39:39.751ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
258882024-02-12T01:39:39.751ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
258892024-02-12T01:39:39.751ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
258902024-02-12T01:39:39.751ZINFOcrucible: [1] client re-new 0 jobs since flush 9439 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
258912024-02-12T01:39:39.751ZINFOcrucible: client task is exiting = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
25892 [8820] Exited with: ExitStatus(unix_wait_status(9))
25893 [8820][1] reports Exit
258942024-02-12T01:39:40.752ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
258952024-02-12T01:39:40.752ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
258962024-02-12T01:39:40.752ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
258972024-02-12T01:39:40.753ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
258982024-02-12T01:39:40.753ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
258992024-02-12T01:39:40.753ZINFOcrucible: [1] client re-new 0 jobs since flush 9439 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
259002024-02-12T01:39:40.753ZINFOcrucible: client task is exiting = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
259012024-02-12T01:39:41.755ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
259022024-02-12T01:39:41.755ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
259032024-02-12T01:39:41.755ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
259042024-02-12T01:39:41.755ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
259052024-02-12T01:39:41.756ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
259062024-02-12T01:39:41.756ZINFOcrucible: [1] client re-new 0 jobs since flush 9439 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
259072024-02-12T01:39:41.756ZINFOcrucible: client task is exiting = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
259082024-02-12T01:39:42.757ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
259092024-02-12T01:39:42.757ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
259102024-02-12T01:39:42.757ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
259112024-02-12T01:39:42.757ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
259122024-02-12T01:39:42.757ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
259132024-02-12T01:39:42.757ZINFOcrucible: [1] client re-new 0 jobs since flush 9439 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
259142024-02-12T01:39:42.757ZINFOcrucible: client task is exiting = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
25915 Feb 12 01:39:43.753 INFO request completed, latency_us: 215, response_code: 200, uri: /state/cid/1, method: GET, req_id: 0b622d93-29ba-40b3-b572-8163d94f7970, remote_addr: 127.0.0.1:60456, local_addr: 127.0.0.1:9998
25916 001/300 Flush
25917 002/300 Write block 780 len 8192 data: 7 7
25918 003/300 Write block 3286 len 16384 data: 9 7 7 5
25919 004/300 Read block 3913 len 36864
25920 005/300 Read block 4803 len 20480
259212024-02-12T01:39:43.758ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
259222024-02-12T01:39:43.758ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
259232024-02-12T01:39:43.758ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
259242024-02-12T01:39:43.758ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
259252024-02-12T01:39:43.758ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
259262024-02-12T01:39:43.758ZINFOcrucible: [1] client re-new 5 jobs since flush 9439 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
259272024-02-12T01:39:43.758ZINFOcrucible: client task is exiting = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
25928 006/300 Read block 2926 len 4096
25929 007/300 Read block 3704 len 4096
25930 008/300 Write block 2973 len 8192 data: 8 8
25931 009/300 Write block 2801 len 20480 data: 6 7 8 8 8
25932 010/300 Write block 332 len 24576 data: 3 5 6 9 9 6
25933 011/300 Read block 4648 len 20480
25934 012/300 Read block 1531 len 16384
25935 013/300 Flush
25936 014/300 Read block 2149 len 16384
25937 015/300 Write block 3999 len 20480 data: 7 6 7 5 5
25938 016/300 Write block 4587 len 24576 data: 5 5 4 4 4 4
25939 017/300 Write block 4280 len 12288 data: 4 4 5
25940 018/300 Write block 3117 len 24576 data: 6 6 7 8 7 7
25941 019/300 Read block 2181 len 32768
25942 020/300 Read block 762 len 12288
25943 021/300 Read block 2096 len 32768
25944 022/300 Write block 2813 len 40960 data: 11 8 6 5 5 6 6 4 6 6
25945 023/300 Read block 904 len 16384
25946 024/300 Write block 4763 len 20480 data: 5 5 6 6 6
25947 025/300 Write block 3073 len 32768 data: 4 4 4 5 4 5 6 8
25948 026/300 Read block 2859 len 28672
25949 027/300 Write block 1108 len 20480 data: 5 4 5 5 4
25950 028/300 Write block 3648 len 24576 data: 6 7 8 8 6 6
25951 029/300 Read block 412 len 32768
25952 030/300 Read block 2559 len 20480
25953 031/300 Read block 2385 len 16384
25954 032/300 Write block 4118 len 28672 data: 8 8 5 5 6 5 5
25955 033/300 Write block 1447 len 24576 data: 6 7 7 8 8 8
25956 034/300 Flush
25957 035/300 Read block 558 len 40960
25958 036/300 Read block 1234 len 40960
25959 037/300 Write block 623 len 8192 data: 6 4
25960 038/300 Write block 1957 len 12288 data: 4 4 4
25961 039/300 Read block 1859 len 32768
25962 040/300 Read block 4121 len 8192
25963 041/300 Flush
25964 042/300 Write block 1571 len 12288 data: 7 6 4
25965 043/300 Write block 1072 len 40960 data: 10 9 9 9 8 9 10 8 9 8
25966 044/300 Write block 141 len 12288 data: 5 5 4
25967 045/300 Flush
25968 046/300 Write block 706 len 8192 data: 8 7
25969 047/300 Write block 3939 len 8192 data: 5 5
25970 048/300 Read block 2450 len 40960
25971 049/300 Write block 223 len 32768 data: 7 6 6 6 6 6 7 7
25972 050/300 Flush
25973 051/300 Read block 1024 len 8192
25974 052/300 Read block 4377 len 4096
25975 053/300 Read block 2187 len 32768
25976 054/300 Read block 4543 len 4096
25977 055/300 Flush
25978 056/300 Read block 300 len 32768
25979 057/300 Read block 1184 len 8192
25980 058/300 Flush
25981 059/300 Read block 3578 len 28672
25982 060/300 Write block 998 len 28672 data: 6 7 10 9 9 8 9
25983 061/300 Flush
25984 062/300 Write block 2587 len 12288 data: 10 12 10
25985 063/300 Flush
25986 064/300 Read block 2266 len 36864
25987 065/300 Read block 1260 len 12288
25988 066/300 Read block 3554 len 36864
25989 067/300 Read block 2654 len 40960
25990 068/300 Read block 2867 len 24576
25991 069/300 Read block 4909 len 32768
25992 070/300 Read block 448 len 36864
25993 071/300 Read block 4925 len 16384
25994 072/300 Read block 3747 len 36864
25995 073/300 Read block 2721 len 28672
25996 074/300 Read block 3796 len 12288
25997 075/300 Write block 785 len 12288 data: 10 10 9
25998 076/300 Read block 4163 len 8192
25999 077/300 Read block 1690 len 4096
26000 078/300 Read block 176 len 20480
26001 079/300 Write block 3001 len 16384 data: 4 3 3 5
26002 080/300 Read block 1067 len 4096
26003 081/300 Flush
26004 082/300 Read block 240 len 32768
26005 083/300 Write block 3301 len 24576 data: 5 5 4 5 5 3
26006 084/300 Flush
26007 085/300 Write block 4080 len 8192 data: 3 3
26008 086/300 Read block 1686 len 24576
26009 087/300 Read block 273 len 12288
26010 088/300 Read block 2260 len 12288
26011 089/300 Write block 3880 len 16384 data: 7 5 6 6
26012 090/300 Write block 237 len 12288 data: 5 4 3
26013 091/300 Write block 1887 len 12288 data: 6 8 7
26014 092/300 Write block 2563 len 16384 data: 6 5 6 6
26015 093/300 Read block 3013 len 20480
26016 094/300 Read block 4306 len 4096
26017 095/300 Read block 2162 len 28672
26018 096/300 Write block 3768 len 16384 data: 6 6 6 5
26019 097/300 Write block 2808 len 8192 data: 10 13
26020 098/300 Write block 2798 len 16384 data: 6 7 5 7
26021 099/300 Write block 4160 len 12288 data: 4 4 5
26022 100/300 Read block 1818 len 12288
26023 101/300 Read block 1867 len 12288
26024 102/300 Write block 619 len 28672 data: 2 3 4 6 7 5 4
26025 103/300 Write block 2992 len 16384 data: 3 4 4 4
26026 104/300 Read block 1010 len 16384
26027 105/300 Read block 1262 len 28672
26028 106/300 Write block 3017 len 8192 data: 3 3
26029 107/300 Read block 2883 len 36864
26030 108/300 Read block 3274 len 12288
26031 109/300 Read block 3468 len 32768
26032 110/300 Write block 1099 len 32768 data: 5 5 5 6 7 8 8 6
26033 111/300 Read block 2732 len 4096
26034 112/300 Write block 4519 len 12288 data: 6 6 5
26035 113/300 Read block 3591 len 8192
26036 114/300 Write block 880 len 8192 data: 7 7
26037 115/300 Write block 1416 len 12288 data: 4 4 4
26038 116/300 Read block 241 len 4096
26039 117/300 Write block 4940 len 20480 data: 6 4 5 5 7
26040 118/300 Read block 4673 len 8192
26041 119/300 Write block 3437 len 24576 data: 5 4 5 6 6 6
26042 120/300 Read block 2883 len 12288
26043 121/300 Write block 4124 len 28672 data: 6 5 5 5 6 7 7
26044 122/300 Read block 2570 len 28672
26045 123/300 Read block 4159 len 12288
26046 124/300 Write block 3524 len 4096 data: 6
26047 125/300 Write block 1316 len 8192 data: 5 5
26048 126/300 Read block 3177 len 12288
26049 127/300 Read block 3503 len 20480
26050 128/300 Write block 3534 len 32768 data: 5 7 7 7 5 6 5 4
26051 129/300 Read block 2754 len 8192
26052 130/300 Flush
26053 131/300 Write block 1963 len 4096 data: 4
26054 132/300 Read block 3807 len 24576
26055 133/300 Write block 1117 len 28672 data: 4 5 7 6 6 6 8
26056 134/300 Read block 4284 len 4096
26057 135/300 Write block 2453 len 28672 data: 3 4 5 5 3 4 3
26058 136/300 Flush
26059 137/300 Read block 2093 len 36864
26060 138/300 Read block 3893 len 32768
26061 139/300 Write block 2736 len 8192 data: 5 5
26062 140/300 Write block 3576 len 16384 data: 8 7 7 6
26063 141/300 Flush
26064 142/300 Flush
26065 143/300 Read block 1361 len 28672
26066 144/300 Read block 59 len 36864
26067 145/300 Write block 4114 len 4096 data: 4
26068 146/300 Write block 1701 len 8192 data: 6 8
26069 147/300 Read block 1773 len 28672
26070 148/300 Read block 1564 len 24576
26071 149/300 Read block 2270 len 4096
26072 150/300 Read block 2712 len 28672
26073 151/300 Write block 85 len 24576 data: 7 7 6 6 7 8
26074 152/300 Read block 404 len 36864
26075 153/300 Read block 4264 len 8192
26076 154/300 Write block 4420 len 36864 data: 5 6 6 5 5 6 5 6 5
26077 155/300 Write block 3434 len 8192 data: 7 6
26078 156/300 Read block 1580 len 16384
26079 157/300 Read block 2626 len 40960
26080 158/300 Read block 2030 len 8192
26081 159/300 Write block 4827 len 12288 data: 4 4 4
26082 160/300 Read block 2415 len 28672
26083 161/300 Write block 1050 len 24576 data: 6 7 5 4 4 5
26084 162/300 Read block 1016 len 28672
26085 163/300 Write block 3823 len 16384 data: 6 7 5 6
26086 164/300 Flush
26087 165/300 Write block 1462 len 28672 data: 6 7 9 8 8 6 4
26088 166/300 Write block 726 len 16384 data: 3 5 5 5
26089 167/300 Write block 728 len 16384 data: 6 6 6 5
26090 168/300 Write block 2288 len 16384 data: 4 4 6 6
26091 169/300 Read block 1577 len 40960
26092 170/300 Write block 1253 len 40960 data: 6 7 7 11 8 9 10 12 12 10
26093 171/300 Write block 1825 len 28672 data: 5 5 7 7 8 6 5
26094 172/300 Write block 3087 len 36864 data: 5 4 5 5 4 5 5 4 4
26095 173/300 Write block 4862 len 12288 data: 3 6 7
26096 174/300 Write block 2599 len 40960 data: 8 7 7 8 8 7 6 5 6 7
26097 175/300 Read block 1496 len 32768
26098 176/300 Read block 4937 len 36864
26099 177/300 Write block 4849 len 16384 data: 4 3 4 5
26100 178/300 Write block 2891 len 32768 data: 4 4 4 7 9 10 11 11
26101 179/300 Read block 159 len 32768
26102 180/300 Write block 2148 len 20480 data: 4 4 4 5 5
26103 181/300 Read block 1395 len 8192
26104 182/300 Read block 2588 len 16384
26105 183/300 Read block 4302 len 16384
26106 184/300 Read block 4621 len 28672
26107 185/300 Read block 1602 len 12288
26108 186/300 Write block 4439 len 24576 data: 4 4 4 4 5 6
26109 187/300 Write block 2525 len 36864 data: 7 7 7 7 6 6 6 4 4
26110 188/300 Read block 2884 len 4096
26111 189/300 Read block 4192 len 24576
26112 190/300 Read block 2267 len 32768
26113 191/300 Write block 1933 len 36864 data: 3 3 3 3 3 3 3 3 3
26114 192/300 Read block 3916 len 20480
26115 193/300 Write block 3005 len 16384 data: 5 6 6 5
26116 194/300 Read block 223 len 24576
26117 195/300 Read block 2694 len 24576
26118 196/300 Write block 3169 len 28672 data: 9 8 5 5 4 4 5
26119 197/300 Write block 595 len 12288 data: 5 4 6
26120 198/300 Read block 2909 len 4096
26121 199/300 Read block 4019 len 32768
26122 200/300 Write block 944 len 16384 data: 3 4 5 4
26123 201/300 Write block 4054 len 32768 data: 6 6 6 7 5 5 5 5
26124 202/300 Read block 337 len 32768
26125 203/300 Write block 517 len 16384 data: 4 4 5 5
26126 204/300 Flush
26127 205/300 Read block 2337 len 20480
26128 206/300 Write block 4105 len 16384 data: 4 4 4 4
26129 207/300 Read block 1951 len 8192
26130 208/300 Read block 3912 len 36864
26131 209/300 Write block 3183 len 28672 data: 8 10 10 12 11 10 10
26132 210/300 Write block 2065 len 28672 data: 7 7 7 8 7 7 8
26133 211/300 Write block 447 len 36864 data: 6 6 7 5 5 6 4 4 4
26134 212/300 Write block 2580 len 24576 data: 6 7 6 7 9 9
26135 213/300 Write block 4395 len 8192 data: 5 4
26136 214/300 Write block 1987 len 20480 data: 4 4 5 5 5
26137 215/300 Write block 4728 len 8192 data: 3 4
26138 216/300 Read block 2541 len 36864
26139 217/300 Read block 2772 len 40960
26140 218/300 Write block 9 len 20480 data: 5 6 7 6 5
26141 219/300 Read block 981 len 40960
26142 220/300 Flush
26143 221/300 Write block 4621 len 40960 data: 6 5 5 5 5 3 6 6 7 7
26144 222/300 Read block 158 len 8192
26145 223/300 Write block 4351 len 8192 data: 3 3
26146 224/300 Write block 2348 len 4096 data: 3
26147 225/300 Write block 1513 len 24576 data: 7 6 6 6 6 6
26148 226/300 Read block 1646 len 16384
26149 227/300 Read block 4056 len 24576
26150 228/300 Read block 3430 len 16384
26151 229/300 Read block 881 len 36864
26152 230/300 Read block 1731 len 36864
26153 231/300 Read block 2416 len 8192
26154 232/300 Read block 55 len 8192
26155 233/300 Flush
26156 234/300 Write block 4617 len 12288 data: 6 5 5
26157 235/300 Read block 1980 len 36864
26158 236/300 Write block 102 len 36864 data: 11 11 12 11 9 10 8 7 4
26159 237/300 Flush
26160 238/300 Write block 1424 len 40960 data: 8 8 6 6 5 5 4 6 6 7
26161 239/300 Read block 4217 len 12288
26162 240/300 Read block 3223 len 4096
26163 241/300 Read block 495 len 28672
26164 242/300 Write block 1306 len 28672 data: 5 5 4 5 5 5 4
26165 243/300 Read block 562 len 40960
26166 244/300 Read block 779 len 36864
26167 245/300 Read block 1910 len 16384
26168 246/300 Read block 3401 len 8192
26169 247/300 Read block 3119 len 20480
26170 248/300 Flush
26171 249/300 Write block 1031 len 8192 data: 5 5
26172 250/300 Read block 2054 len 36864
26173 251/300 Read block 1773 len 40960
26174 252/300 Read block 361 len 16384
26175 253/300 Read block 3037 len 32768
26176 254/300 Read block 2746 len 24576
26177 255/300 Read block 566 len 8192
26178 256/300 Write block 4448 len 16384 data: 9 9 10 9
26179 257/300 Write block 1895 len 4096 data: 6
26180 258/300 Write block 1256 len 24576 data: 12 9 10 11 13 13
26181 259/300 Flush
26182 260/300 Read block 1507 len 32768
26183 261/300 Write block 1242 len 8192 data: 5 6
26184 262/300 Write block 3070 len 40960 data: 3 3 4 5 5 5 6 5 6 7
26185 263/300 Read block 4116 len 12288
26186 264/300 Write block 1518 len 40960 data: 7 5 5 4 4 3 4 4 4 3
26187 265/300 Write block 1140 len 16384 data: 9 8 10 9
26188 266/300 Write block 767 len 36864 data: 4 4 3 2 2 2 2 3 5
26189 267/300 Read block 2307 len 24576
26190 268/300 Read block 3903 len 8192
26191 269/300 Read block 985 len 16384
26192 270/300 Read block 1930 len 28672
26193 271/300 Write block 2914 len 12288 data: 3 3 3
26194 272/300 Read block 3705 len 12288
26195 273/300 Read block 3815 len 16384
26196 274/300 Read block 1140 len 32768
26197 275/300 Read block 1854 len 8192
26198 276/300 Read block 2886 len 20480
26199 277/300 Read block 3948 len 4096
26200 278/300 Read block 4912 len 8192
26201 279/300 Write block 1439 len 24576 data: 7 6 6 7 7 7
26202 280/300 Read block 1169 len 36864
26203 281/300 Read block 3278 len 32768
26204 282/300 Write block 2548 len 40960 data: 3 5 5 5 5 5 5 6 5 5
26205 283/300 Write block 1447 len 36864 data: 7 8 8 9 9 9 8 7 7
26206 284/300 Write block 1134 len 24576 data: 4 5 6 6 6 7
26207 285/300 Write block 2657 len 16384 data: 6 6 6 5
26208 286/300 Write block 150 len 12288 data: 5 6 5
26209 287/300 Read block 15 len 4096
26210 288/300 Read block 3766 len 24576
26211 289/300 Flush
26212 290/300 Flush
26213 291/300 Flush
26214 292/300 Write block 1586 len 36864 data: 5 4 2 2 4 5 8 7 7
26215 293/300 Flush
26216 294/300 Flush
26217 295/300 Flush
26218 296/300 Read block 2394 len 12288
26219 297/300 Write block 1558 len 40960 data: 6 6 3 4 4 5 8 9 10 10
26220 298/300 Write block 1598 len 36864 data: 5 5 4 3 3 3 3 4 4
26221 299/300 Write block 832 len 24576 data: 3 3 4 4 4 4
26222 300/300 Read block 730 len 12288
26223 Main task has work to do, go find it
26224 got dsc Start(1)
26225 start 1
26226 Feb 12 01:39:44.088 INFO request completed, latency_us: 238, response_code: 204, uri: /start/cid/1, method: POST, req_id: 64ca44b8-afbd-4054-bf48-26c88765873e, remote_addr: 127.0.0.1:60456, local_addr: 127.0.0.1:9998
26227 Replay: started 1, returned:Ok(())
26228 ----------------------------------------------------------------
26229 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
26230 States: Active Offline Active
26231 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
26232 8441 Acked 9440 Flush 0 Done New Done true
26233 8442 Acked 9441 Write 2 Done New Done true
26234 8443 Acked 9442 Write 4 Done New Done true
26235 8444 Acked 9443 Read 9 Done New Done true
26236 8445 Acked 9444 Read 5 Done New Done true
26237 8446 Acked 9445 Read 1 Done New Done false
26238 8447 Acked 9446 Read 1 Done New Done false
26239 8448 Acked 9447 Write 2 Done New Done false
26240 8449 Acked 9448 Write 5 Done New Done false
26241 8450 Acked 9449 Write 6 Done New Done false
26242 8451 Acked 9450 Read 5 Done New Done false
26243 8452 Acked 9451 Read 4 Done New Done false
26244 8453 Acked 9452 Flush 0 Done New Done false
26245 8454 Acked 9453 Read 4 Done New Done false
26246 8455 Acked 9454 Write 5 Done New Done false
26247 8456 Acked 9455 Write 6 Done New Done false
26248 8457 Acked 9456 Write 3 Done New Done false
26249 8458 Acked 9457 Write 6 Done New Done false
26250 8459 Acked 9458 Read 8 Done New Done false
26251 8460 Acked 9459 Read 3 Done New Done false
26252 8461 Acked 9460 Read 8 Done New Done false
26253 8462 Acked 9461 Write 10 Done New Done false
26254 8463 Acked 9462 Read 4 Done New Done false
26255 8464 Acked 9463 Write 5 Done New Done false
26256 8465 Acked 9464 Write 8 Done New Done false
26257 8466 Acked 9465 Read 7 Done New Done false
26258 8467 Acked 9466 Write 5 Done New Done false
26259 8468 Acked 9467 Write 6 Done New Done false
26260 8469 Acked 9468 Read 8 Done New Done false
26261 8470 Acked 9469 Read 5 Done New Done false
26262 8471 Acked 9470 Read 4 Done New Done false
26263 8472 Acked 9471 Write 7 Done New Done false
26264 8473 Acked 9472 Write 6 Done New Done false
26265 8474 Acked 9473 Flush 0 Done New Done false
26266 8475 Acked 9474 Read 10 Done New Done false
26267 8476 Acked 9475 Read 10 Done New Done false
26268 8477 Acked 9476 Write 2 Done New Done false
26269 8478 Acked 9477 Write 3 Done New Done false
26270 8479 Acked 9478 Read 8 Done New Done false
26271 8480 Acked 9479 Read 2 Done New Done false
26272 8481 Acked 9480 Flush 0 Done New Done false
26273 8482 Acked 9481 Write 3 Done New Done false
26274 8483 Acked 9482 Write 10 Done New Done false
26275 8484 Acked 9483 Write 3 Done New Done false
26276 8485 Acked 9484 Flush 0 Done New Done false
26277 8486 Acked 9485 Write 2 Done New Done false
26278 8487 Acked 9486 Write 2 Done New Done false
26279 8488 Acked 9487 Read 10 Done New Done false
26280 8489 Acked 9488 Write 8 Done New Done false
26281 8490 Acked 9489 Flush 0 Done New Done false
26282 8491 Acked 9490 Read 2 Done New Done false
26283 8492 Acked 9491 Read 1 Done New Done false
26284 8493 Acked 9492 Read 8 Done New Done false
26285 8494 Acked 9493 Read 1 Done New Done false
26286 8495 Acked 9494 Flush 0 Done New Done false
26287 8496 Acked 9495 Read 8 Done New Done false
26288 8497 Acked 9496 Read 2 Done New Done false
26289 8498 Acked 9497 Flush 0 Done New Done false
26290 8499 Acked 9498 Read 7 Done New Done false
26291 8500 Acked 9499 Write 7 Done New Done false
26292 8501 Acked 9500 Flush 0 Done New Done false
26293 8502 Acked 9501 Write 3 Done New Done false
26294 8503 Acked 9502 Flush 0 Done New Done false
26295 8504 Acked 9503 Read 9 Done New Done false
26296 8505 Acked 9504 Read 3 Done New Done false
26297 8506 Acked 9505 Read 9 Done New Done false
26298 8507 Acked 9506 Read 10 Done New Done false
26299 8508 Acked 9507 Read 6 Done New Done false
26300 8509 Acked 9508 Read 8 Done New Done false
26301 8510 Acked 9509 Read 9 Done New Done false
26302 8511 Acked 9510 Read 4 Done New Done false
26303 8512 Acked 9511 Read 9 Done New Done false
26304 8513 Acked 9512 Read 7 Done New Done false
26305 8514 Acked 9513 Read 3 Done New Done false
26306 8515 Acked 9514 Write 3 Done New Done false
26307 8516 Acked 9515 Read 2 Done New Done false
26308 8517 Acked 9516 Read 1 Done New Done false
26309 8518 Acked 9517 Read 5 Done New Done false
26310 8519 Acked 9518 Write 4 Done New Done false
26311 8520 Acked 9519 Read 1 Done New Done false
26312 8521 Acked 9520 Flush 0 Done New Done false
26313 8522 Acked 9521 Read 8 Done New Done false
26314 8523 Acked 9522 Write 6 Done New Done false
26315 8524 Acked 9523 Flush 0 Done New Done false
26316 8525 Acked 9524 Write 2 Done New Done false
26317 8526 Acked 9525 Read 6 Done New Done false
26318 8527 Acked 9526 Read 3 Done New Done false
26319 8528 Acked 9527 Read 3 Done New Done false
26320 8529 Acked 9528 Write 4 Done New Done false
26321 8530 Acked 9529 Write 3 Done New Done false
26322 8531 Acked 9530 Write 3 Done New Done false
26323 8532 Acked 9531 Write 4 Done New Done false
26324 8533 Acked 9532 Read 5 Done New Done false
26325 8534 Acked 9533 Read 1 Done New Done false
26326 8535 Acked 9534 Read 7 Done New Done false
26327 8536 Acked 9535 Write 4 Done New Done false
26328 8537 Acked 9536 Flush 0 Done New Done false
26329 8538 Acked 9537 Write 2 Done New Done false
26330 8539 Acked 9538 Write 4 Done New Done false
26331 8540 Acked 9539 Write 3 Done New Done false
26332 8541 Acked 9540 Read 3 Done New Done false
26333 8542 Acked 9541 Read 3 Done New Done false
26334 8543 Acked 9542 Write 7 Done New Done false
26335 8544 Acked 9543 Write 4 Done New Done false
26336 8545 Acked 9544 Read 4 Done New Done false
26337 8546 Acked 9545 Read 7 Done New Done false
26338 8547 Acked 9546 Write 2 Done New Done false
26339 8548 Acked 9547 Read 9 Done New Done false
26340 8549 Acked 9548 Read 3 Done New Done false
26341 8550 Acked 9549 Read 8 Done New Done false
26342 8551 Acked 9550 Write 8 Done New Done false
26343 8552 Acked 9551 Read 1 Done New Done false
26344 8553 Acked 9552 Write 3 Done New Done false
26345 8554 Acked 9553 Read 2 Done New Done false
26346 8555 Acked 9554 Write 2 Done New Done false
26347 8556 Acked 9555 Write 3 Done New Done false
26348 8557 Acked 9556 Read 1 Done New Done false
26349 8558 Acked 9557 Write 5 Done New Done false
26350 8559 Acked 9558 Read 2 Done New Done false
26351 8560 Acked 9559 Write 6 Done New Done false
26352 8561 Acked 9560 Read 3 Done New Done false
26353 8562 Acked 9561 Write 7 Done New Done false
26354 8563 Acked 9562 Read 7 Done New Done false
26355 8564 Acked 9563 Read 3 Done New Done false
26356 8565 Acked 9564 Write 1 Done New Done false
26357 8566 Acked 9565 Write 2 Done New Done false
26358 8567 Acked 9566 Read 3 Done New Done false
26359 8568 Acked 9567 Read 5 Done New Done false
26360 8569 Acked 9568 Write 8 Done New Done false
26361 8570 Acked 9569 Read 2 Done New Done false
26362 8571 Acked 9570 Flush 0 Done New Done false
26363 8572 Acked 9571 Write 1 Done New Done false
26364 8573 Acked 9572 Read 6 Done New Done false
26365 8574 Acked 9573 Write 7 Done New Done false
26366 8575 Acked 9574 Read 1 Done New Done false
26367 8576 Acked 9575 Write 7 Done New Done false
26368 8577 Acked 9576 Flush 0 Done New Done false
26369 8578 Acked 9577 Read 9 Done New Done false
26370 8579 Acked 9578 Read 8 Done New Done false
26371 8580 Acked 9579 Write 2 Done New Done false
26372 8581 Acked 9580 Write 4 Done New Done false
26373 8582 Acked 9581 Flush 0 Done New Done false
26374 8583 Acked 9582 Flush 0 Done New Done false
26375 8584 Acked 9583 Read 7 Done New Done false
26376 8585 Acked 9584 Read 9 Done New Done false
26377 8586 Acked 9585 Write 1 Done New Done false
26378 8587 Acked 9586 Write 2 Done New Done false
26379 8588 Acked 9587 Read 7 Done New Done false
26380 8589 Acked 9588 Read 6 Done New Done false
26381 8590 Acked 9589 Read 1 Done New Done false
26382 8591 Acked 9590 Read 7 Done New Done false
26383 8592 Acked 9591 Write 6 Done New Done false
26384 8593 Acked 9592 Read 9 Done New Done false
26385 8594 Acked 9593 Read 2 Done New Done false
26386 8595 Acked 9594 Write 9 Done New Done false
26387 8596 Acked 9595 Write 2 Done New Done false
26388 8597 Acked 9596 Read 4 Done New Done false
26389 8598 Acked 9597 Read 10 Done New Done false
26390 8599 Acked 9598 Read 2 Done New Done false
26391 8600 Acked 9599 Write 3 Done New Done false
26392 8601 Acked 9600 Read 7 Done New Done false
26393 8602 Acked 9601 Write 6 Done New Done false
26394 8603 Acked 9602 Read 7 Done New Done false
26395 8604 Acked 9603 Write 4 Done New Done false
26396 8605 Acked 9604 Flush 0 Done New Done false
26397 8606 Acked 9605 Write 7 Done New Done false
26398 8607 Acked 9606 Write 4 Done New Done false
26399 8608 Acked 9607 Write 4 Done New Done false
26400 8609 Acked 9608 Write 4 Done New Done false
26401 8610 Acked 9609 Read 10 Done New Done false
26402 8611 Acked 9610 Write 10 Done New Done false
26403 8612 Acked 9611 Write 7 Done New Done false
26404 8613 Acked 9612 Write 9 Done New Done false
26405 8614 Acked 9613 Write 3 Done New Done false
26406 8615 Acked 9614 Write 10 Done New Done false
26407 8616 Acked 9615 Read 8 Done New Done false
26408 8617 Acked 9616 Read 9 Done New Done false
26409 8618 Acked 9617 Write 4 Done New Done false
26410 8619 Acked 9618 Write 8 Done New Done false
26411 8620 Acked 9619 Read 8 Done New Done false
26412 8621 Acked 9620 Write 5 Done New Done false
26413 8622 Acked 9621 Read 2 Done New Done false
26414 8623 Acked 9622 Read 4 Done New Done false
26415 8624 Acked 9623 Read 4 Done New Done false
26416 8625 Acked 9624 Read 7 Done New Done false
26417 8626 Acked 9625 Read 3 Done New Done false
26418 8627 Acked 9626 Write 6 Done New Done false
26419 8628 Acked 9627 Write 9 Done New Done false
26420 8629 Acked 9628 Read 1 Done New Done false
26421 8630 Acked 9629 Read 6 Done New Done false
26422 8631 Acked 9630 Read 8 Done New Done false
26423 8632 Acked 9631 Write 9 Done New Done false
26424 8633 Acked 9632 Read 5 Done New Done false
26425 8634 Acked 9633 Write 4 Done New Done false
26426 8635 Acked 9634 Read 6 Done New Done false
26427 8636 Acked 9635 Read 6 Done New Done false
26428 8637 Acked 9636 Write 7 Done New Done false
26429 8638 Acked 9637 Write 3 Done New Done false
26430 8639 Acked 9638 Read 1 Done New Done false
26431 8640 Acked 9639 Read 8 Done New Done false
26432 8641 Acked 9640 Write 4 Done New Done false
26433 8642 Acked 9641 Write 8 Done New Done false
26434 8643 Acked 9642 Read 8 Done New Done false
26435 8644 Acked 9643 Write 4 Done New Done false
26436 8645 Acked 9644 Flush 0 Done New Done false
26437 8646 Acked 9645 Read 5 Done New Done false
26438 8647 Acked 9646 Write 4 Done New Done false
26439 8648 Acked 9647 Read 2 Done New Done false
26440 8649 Acked 9648 Read 9 Done New Done false
26441 8650 Acked 9649 Write 7 Done New Done false
26442 8651 Acked 9650 Write 7 Done New Done false
26443 8652 Acked 9651 Write 9 Done New Done false
26444 8653 Acked 9652 Write 6 Done New Done false
26445 8654 Acked 9653 Write 2 Done New Done false
26446 8655 Acked 9654 Write 5 Done New Done false
26447 8656 Acked 9655 Write 2 Done New Done false
26448 8657 Acked 9656 Read 9 Done New Done false
26449 8658 Acked 9657 Read 10 Done New Done false
26450 8659 Acked 9658 Write 5 Done New Done false
26451 8660 Acked 9659 Read 10 Done New Done false
26452 8661 Acked 9660 Flush 0 Done New Done false
26453 8662 Acked 9661 Write 10 Done New Done false
26454 8663 Acked 9662 Read 2 Done New Done false
26455 8664 Acked 9663 Write 2 Done New Done false
26456 8665 Acked 9664 Write 1 Done New Done false
26457 8666 Acked 9665 Write 6 Done New Done false
26458 8667 Acked 9666 Read 4 Done New Done false
26459 8668 Acked 9667 Read 6 Done New Done false
26460 8669 Acked 9668 Read 4 Done New Done false
26461 8670 Acked 9669 Read 9 Done New Done false
26462 8671 Acked 9670 Read 9 Done New Done false
26463 8672 Acked 9671 Read 2 Done New Done false
26464 8673 Acked 9672 Read 2 Done New Done false
26465 8674 Acked 9673 Flush 0 Done New Done false
26466 8675 Acked 9674 Write 3 Done New Done false
26467 8676 Acked 9675 Read 9 Done New Done false
26468 8677 Acked 9676 Write 9 Done New Done false
26469 8678 Acked 9677 Flush 0 Done New Done false
26470 8679 Acked 9678 Write 10 Done New Done false
26471 8680 Acked 9679 Read 3 Done New Done false
26472 8681 Acked 9680 Read 1 Done New Done false
26473 8682 Acked 9681 Read 7 Done New Done false
26474 8683 Acked 9682 Write 7 Done New Done false
26475 8684 Acked 9683 Read 10 Done New Done false
26476 8685 Acked 9684 Read 9 Done New Done false
26477 8686 Acked 9685 Read 4 Done New Done false
26478 8687 Acked 9686 Read 2 Done New Done false
26479 8688 Acked 9687 Read 5 Done New Done false
26480 8689 Acked 9688 Flush 0 Done New Done false
26481 8690 Acked 9689 Write 2 Done New Done false
26482 8691 Acked 9690 Read 9 Done New Done false
26483 8692 Acked 9691 Read 10 Done New Done false
26484 8693 Acked 9692 Read 4 Done New Done false
26485 8694 Acked 9693 Read 8 Done New Done false
26486 8695 Acked 9694 Read 6 Done New Done false
26487 8696 Acked 9695 Read 2 Done New Done false
26488 8697 Acked 9696 Write 4 Done New Done false
26489 8698 Acked 9697 Write 1 Done New Done false
26490 8699 Acked 9698 Write 6 Done New Done false
26491 8700 Acked 9699 Flush 0 Done New Done false
26492 8701 Acked 9700 Read 8 Done New Done false
26493 8702 Acked 9701 Write 2 Done New Done false
26494 8703 Acked 9702 Write 10 Done New Done false
26495 8704 Acked 9703 Read 3 Done New Done false
26496 8705 Acked 9704 Write 10 Done New Done false
26497 8706 Acked 9705 Write 4 Done New Done false
26498 8707 Acked 9706 Write 9 Done New Done false
26499 8708 Acked 9707 Read 6 Done New Done false
26500 8709 Acked 9708 Read 2 Done New Done false
26501 8710 Acked 9709 Read 4 Done New Done false
26502 8711 Acked 9710 Read 7 Done New Done false
26503 8712 Acked 9711 Write 3 Done New Done false
26504 8713 Acked 9712 Read 3 Done New Done false
26505 8714 Acked 9713 Read 4 Done New Done false
26506 8715 Acked 9714 Read 8 Done New Done false
26507 8716 Acked 9715 Read 2 Done New Done false
26508 8717 Acked 9716 Read 5 Done New Done false
26509 8718 Acked 9717 Read 1 Done New Done false
26510 8719 Acked 9718 Read 2 Done New Done false
26511 8720 Acked 9719 Write 6 Done New Done false
26512 8721 Acked 9720 Read 9 Done New Done false
26513 8722 Acked 9721 Read 8 Done New Done false
26514 8723 Acked 9722 Write 10 Done New Done false
26515 8724 Acked 9723 Write 9 Done New Done false
26516 8725 Acked 9724 Write 6 Done New Done false
26517 8726 Acked 9725 Write 4 Done New Done false
26518 8727 Acked 9726 Write 3 Done New Done false
26519 8728 Acked 9727 Read 1 Done New Done false
26520 8729 Acked 9728 Read 6 Done New Done false
26521 8730 Acked 9729 Flush 0 Done New Done false
26522 8731 Acked 9730 Flush 0 Done New Done false
26523 8732 Acked 9731 Flush 0 Done New Done false
26524 8733 Acked 9732 Write 9 Done New Done false
26525 8734 Acked 9733 Flush 0 Done New Done false
26526 8735 Acked 9734 Flush 0 Done New Done false
26527 8736 Acked 9735 Flush 0 Done New Done false
26528 8737 Acked 9736 Read 3 Done New Done false
26529 8738 Acked 9737 Write 10 Done New Done false
26530 8739 Acked 9738 Write 9 Done New Done false
26531 8740 Acked 9739 Write 6 Done New Done false
26532 8741 Acked 9740 Read 3 Done New Done false
26533 STATES DS:0 DS:1 DS:2 TOTAL
26534 New 0 301 0 301
26535 Sent 0 0 0 0
26536 Done 301 0 301 602
26537 Skipped 0 0 0 0
26538 Error 0 0 0 0
26539 Last Flush: 9735 9439 9735
26540 Downstairs last five completed: 9439 9438 9437 9436 9435
26541 Upstairs last five completed: 8741 8740 8739 8738 8737
26542 CLIENT: Up:0 ds:301 act:2
265432024-02-12T01:39:44.760ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
265442024-02-12T01:39:44.760ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
265452024-02-12T01:39:44.760ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
265462024-02-12T01:39:44.760ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
265472024-02-12T01:39:44.760ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
265482024-02-12T01:39:44.760ZINFOcrucible: [1] client re-new 302 jobs since flush 9439 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
265492024-02-12T01:39:44.760ZINFOcrucible: client task is exiting = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
26550 [8820] got start action
26551 [8820] I am going to restart
26552 Starting downstairs at port 8820
26553 Make output file at "/tmp/dsc/downstairs-8820.txt"
26554 [8820][1] reports Starting
265552024-02-12T01:39:45.761ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
265562024-02-12T01:39:45.761ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
265572024-02-12T01:39:45.761ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
265582024-02-12T01:39:45.761ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
265592024-02-12T01:39:45.761ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
265602024-02-12T01:39:45.761ZINFOcrucible: [1] client re-new 302 jobs since flush 9439 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
265612024-02-12T01:39:45.761ZINFOcrucible: client task is exiting = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
26562 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1193)
26563 [8820][1] reports Running
265642024-02-12T01:39:46.762ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
265652024-02-12T01:39:46.762ZINFOcrucible: ds_connection connected = io task client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
265662024-02-12T01:39:46.764ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
265672024-02-12T01:39:46.764ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
265682024-02-12T01:39:46.764ZINFOcrucible: send last flush ID to this DS: 9439 = downstairs client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
265692024-02-12T01:39:46.764ZINFOcrucible: Replied this last flush ID: 9439 = downstairs client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
265702024-02-12T01:39:46.764ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
265712024-02-12T01:39:46.764ZINFOcrucible: [1] Transition from Offline to Replay = downstairs client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
265722024-02-12T01:39:46.764ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
265732024-02-12T01:39:46.764ZINFOcrucible: [1] Transition from Replay to Active = downstairs client = 1 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
26574 ----------------------------------------------------------------
26575 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
26576 Downstairs last five completed: 9741 9740 9739 9738 9737
26577 Upstairs last five completed: 8742 8741 8740 8739 8738
26578 CLIENT: Up:0 ds:0 act:3
26579 Replay: All jobs finished, all DS active.
26580 Main task has work to do, go find it
26581 got dsc Stop(2)
26582 stop 2
26583 [8830] Got stop action so:false kr:false
26584 Feb 12 01:39:48.091 INFO request completed, latency_us: 344, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 6edabf14-7d58-42b9-b65f-af22abc55021, remote_addr: 127.0.0.1:60456, local_addr: 127.0.0.1:9998
265852024-02-12T01:39:48.092ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
26586 Feb 12 01:39:48.092 INFO request completed, latency_us{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2024-02-12T01:39:48.092566756Z","hostname":"ip-10-150-1-104.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"e1773b3e-b36f-4080-8b13-c0830f0d072e"}
26587 : 113, response_code: 200{"msg":"downstairs task for 2 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-02-12T01:39:48.092588872Z","hostname":"ip-10-150-1-104.us-west-2.compute.internal","pid":1164,"session_id":"e1773b3e-b36f-4080-8b13-c0830f0d072e"}
26588 , uri: /state/cid/2, method: {"msg":"Gone missing, transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-02-12T01:39:48.092625537Z","hostname":"ip-10-150-1-104.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"e1773b3e-b36f-4080-8b13-c0830f0d072e"}
26589 GET, {"msg":"[2] client re-new 0 jobs since flush 9741","v":0,"name":"crucible","level":30,"time":"2024-02-12T01:39:48.092651013Z","hostname":"ip-10-150-1-104.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"e1773b3e-b36f-4080-8b13-c0830f0d072e"}
26590 req_id: d4ccd7fe-db64-46b7-b34c-6e72d997c4d0, remote_addr: 127.{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-12T01:39:48.092684268Z","hostname":"ip-10-150-1-104.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"e1773b3e-b36f-4080-8b13-c0830f0d072e"}
26591 0.0.1:60456, local_addr: 127.0.0.1:9998
26592 [8830] Exited with: ExitStatus(unix_wait_status(9))
26593 [8830][2] reports Exit
265942024-02-12T01:39:49.093ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
265952024-02-12T01:39:49.093ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
265962024-02-12T01:39:49.093ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
265972024-02-12T01:39:49.093ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
265982024-02-12T01:39:49.093ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
265992024-02-12T01:39:49.093ZINFOcrucible: [2] client re-new 0 jobs since flush 9741 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
266002024-02-12T01:39:49.094ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
266012024-02-12T01:39:50.095ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
266022024-02-12T01:39:50.095ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
266032024-02-12T01:39:50.095ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
266042024-02-12T01:39:50.095ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
266052024-02-12T01:39:50.095ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
266062024-02-12T01:39:50.095ZINFOcrucible: [2] client re-new 0 jobs since flush 9741 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
266072024-02-12T01:39:50.095ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
266082024-02-12T01:39:51.096ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
266092024-02-12T01:39:51.096ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
266102024-02-12T01:39:51.096ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
266112024-02-12T01:39:51.096ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
266122024-02-12T01:39:51.096ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
266132024-02-12T01:39:51.096ZINFOcrucible: [2] client re-new 0 jobs since flush 9741 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
266142024-02-12T01:39:51.096ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
26615 Feb 12 01:39:52.094 INFO request completed, latency_us: 222, response_code: 200, uri: /state/cid/2, method: GET, req_id: 534534b2-9077-47ca-9ab7-06357291c4f7, remote_addr: 127.0.0.1:60456, local_addr: 127.0.0.1:9998
26616 001/300 Write block 4329 len 28672 data: 5 5 5 5 5 5 2
26617 002/300 Write block 3772 len 4096 data: 5
26618 003/300 Read block 2394 len 16384
26619 004/300 Read block 4446 len 24576
266202024-02-12T01:39:52.097ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
266212024-02-12T01:39:52.097ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
266222024-02-12T01:39:52.097ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
266232024-02-12T01:39:52.097ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
266242024-02-12T01:39:52.097ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
266252024-02-12T01:39:52.097ZINFOcrucible: [2] client re-new 4 jobs since flush 9741 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
266262024-02-12T01:39:52.097ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
26627 005/300 Read block 4779 len 28672
26628 006/300 Write block 409 len 32768 data: 9 8 9 7 7 6 5 5
26629 007/300 Read block 4174 len 4096
26630 008/300 Write block 200 len 4096 data: 4
26631 009/300 Write block 3667 len 24576 data: 3 3 5 5 4 4
26632 010/300 Flush
26633 011/300 Read block 3174 len 16384
26634 012/300 Read block 2771 len 32768
26635 013/300 Read block 3221 len 28672
26636 014/300 Read block 2115 len 16384
26637 015/300 Read block 2053 len 4096
26638 016/300 Read block 2304 len 12288
26639 017/300 Read block 1723 len 36864
26640 018/300 Write block 913 len 4096 data: 7
26641 019/300 Read block 687 len 12288
26642 020/300 Read block 163 len 36864
26643 021/300 Read block 1374 len 28672
26644 022/300 Write block 4545 len 24576 data: 5 5 7 7 9 8
26645 023/300 Flush
26646 024/300 Read block 1431 len 4096
26647 025/300 Read block 4549 len 20480
26648 026/300 Write block 3479 len 16384 data: 7 7 7 7
26649 027/300 Read block 3439 len 12288
26650 028/300 Read block 182 len 36864
26651 029/300 Write block 2329 len 16384 data: 7 6 6 7
26652 030/300 Read block 2532 len 28672
26653 031/300 Read block 2837 len 36864
26654 032/300 Read block 3654 len 40960
26655 033/300 Read block 2424 len 24576
26656 034/300 Write block 4954 len 8192 data: 6 5
26657 035/300 Read block 3937 len 4096
26658 036/300 Read block 1264 len 36864
26659 037/300 Flush
26660 038/300 Read block 322 len 36864
26661 039/300 Read block 2960 len 12288
26662 040/300 Read block 1675 len 24576
26663 041/300 Flush
26664 042/300 Flush
26665 043/300 Write block 292 len 8192 data: 5 5
26666 044/300 Write block 36 len 40960 data: 6 6 6 5 5 4 3 2 2 3
26667 045/300 Write block 3590 len 20480 data: 4 6 6 6 6
26668 046/300 Write block 4639 len 32768 data: 4 5 5 5 6 8 7 6
26669 047/300 Write block 1354 len 24576 data: 6 5 4 4 6 5
26670 048/300 Write block 1179 len 24576 data: 5 5 6 4 4 5
26671 049/300 Read block 3821 len 28672
26672 050/300 Write block 2188 len 4096 data: 3
26673 051/300 Read block 4844 len 40960
26674 052/300 Read block 2546 len 4096
26675 053/300 Read block 1756 len 8192
26676 054/300 Write block 601 len 40960 data: 5 6 6 7 5 5 6 4 4 3
26677 055/300 Flush
26678 056/300 Write block 3091 len 24576 data: 5 6 6 5 5 4
26679 057/300 Write block 3867 len 40960 data: 6 6 5 5 6 8 8 9 9 7
26680 058/300 Write block 4371 len 36864 data: 4 5 4 5 5 5 5 4 5
26681 059/300 Read block 844 len 4096
26682 060/300 Read block 439 len 8192
26683 061/300 Read block 1998 len 12288
26684 062/300 Write block 2233 len 16384 data: 6 6 7 6
26685 063/300 Read block 717 len 32768
26686 064/300 Read block 416 len 8192
26687 065/300 Read block 3175 len 40960
26688 066/300 Write block 2493 len 28672 data: 6 5 4 4 3 3 2
26689 067/300 Read block 3199 len 12288
26690 068/300 Read block 4342 len 12288
26691 069/300 Read block 1238 len 24576
26692 070/300 Write block 2787 len 28672 data: 6 6 4 4 4 5 5
26693 071/300 Read block 2584 len 40960
26694 072/300 Read block 2884 len 40960
26695 073/300 Flush
26696 074/300 Read block 1016 len 36864
26697 075/300 Write block 2683 len 4096 data: 3
26698 076/300 Write block 2940 len 20480 data: 4 4 4 4 4
26699 077/300 Write block 1500 len 36864 data: 5 5 6 6 7 7 7 6 6
26700 078/300 Flush
26701 079/300 Write block 1337 len 28672 data: 4 4 5 8 6 5 5
26702 080/300 Read block 1629 len 12288
26703 081/300 Write block 2752 len 40960 data: 8 7 8 8 9 10 10 11 10 9
26704 082/300 Write block 51 len 12288 data: 5 4 6
26705 083/300 Read block 4311 len 16384
26706 084/300 Flush
26707 085/300 Read block 4055 len 24576
26708 086/300 Write block 637 len 4096 data: 7
26709 087/300 Write block 3941 len 16384 data: 6 6 6 7
26710 088/300 Flush
26711 089/300 Flush
26712 090/300 Flush
26713 091/300 Write block 263 len 20480 data: 7 7 7 5 6
26714 092/300 Write block 2047 len 4096 data: 8
26715 093/300 Write block 4170 len 24576 data: 8 9 10 11 12 8
26716 094/300 Write block 2317 len 20480 data: 6 4 5 7 8
26717 095/300 Write block 2526 len 40960 data: 8 8 8 7 7 7 5 5 4 4
26718 096/300 Read block 667 len 32768
26719 097/300 Read block 339 len 12288
26720 098/300 Read block 4444 len 28672
26721 099/300 Read block 4748 len 12288
26722 100/300 Flush
26723 101/300 Read block 378 len 12288
26724 102/300 Write block 1607 len 20480 data: 6 7 8 7 7
26725 103/300 Write block 4932 len 4096 data: 4
26726 104/300 Read block 1251 len 32768
26727 105/300 Read block 3746 len 40960
26728 106/300 Write block 4045 len 24576 data: 5 4 3 3 3 3
26729 107/300 Write block 3140 len 16384 data: 7 8 9 8
26730 108/300 Flush
26731 109/300 Write block 222 len 8192 data: 6 8
26732 110/300 Write block 2095 len 12288 data: 7 5 6
26733 111/300 Write block 3354 len 36864 data: 5 6 5 4 6 8 8 8 9
26734 112/300 Write block 697 len 16384 data: 8 7 7 9
26735 113/300 Read block 4770 len 8192
26736 114/300 Write block 2080 len 40960 data: 6 5 5 5 5 5 3 4 5 5
26737 115/300 Flush
26738 116/300 Write block 956 len 12288 data: 7 6 7
26739 117/300 Write block 1142 len 8192 data: 11 10
26740 118/300 Read block 1065 len 40960
26741 119/300 Read block 1536 len 28672
26742 120/300 Read block 2209 len 20480
26743 121/300 Write block 2065 len 4096 data: 8
26744 122/300 Read block 708 len 28672
26745 123/300 Write block 3343 len 20480 data: 7 6 5 4 3
26746 124/300 Flush
26747 125/300 Read block 2298 len 24576
26748 126/300 Write block 1039 len 16384 data: 3 5 5 5
26749 127/300 Write block 3635 len 4096 data: 5
26750 128/300 Write block 398 len 8192 data: 4 3
26751 129/300 Write block 2540 len 16384 data: 6 6 6 5
26752 130/300 Write block 1784 len 20480 data: 3 4 5 5 5
26753 131/300 Read block 2224 len 4096
26754 132/300 Write block 3298 len 28672 data: 5 6 6 6 6 5 6
26755 133/300 Read block 2408 len 40960
26756 134/300 Flush
26757 135/300 Read block 1143 len 8192
26758 136/300 Flush
26759 137/300 Write block 2303 len 4096 data: 6
26760 138/300 Flush
26761 139/300 Read block 1692 len 28672
26762 140/300 Read block 1825 len 28672
26763 141/300 Read block 804 len 4096
26764 142/300 Flush
26765 143/300 Write block 4376 len 16384 data: 6 6 5 6
26766 144/300 Write block 3839 len 24576 data: 2 2 2 2 3 2
26767 145/300 Read block 4305 len 32768
26768 146/300 Read block 850 len 8192
26769 147/300 Write block 1555 len 20480 data: 6 6 7 7 7
26770 148/300 Write block 4613 len 28672 data: 8 8 8 7 7 6 6
26771 149/300 Write block 2184 len 4096 data: 3
26772 150/300 Write block 2137 len 28672 data: 5 5 5 3 3 3 3
26773 151/300 Write block 301 len 8192 data: 6 5
26774 152/300 Read block 4966 len 16384
26775 153/300 Write block 959 len 16384 data: 6 4 4 4
26776 154/300 Read block 3862 len 36864
26777 155/300 Read block 1867 len 40960
26778 156/300 Write block 476 len 8192 data: 7 7
26779 157/300 Read block 2594 len 4096
26780 158/300 Write block 1537 len 28672 data: 9 8 9 9 8 8 9
26781 159/300 Write block 4773 len 40960 data: 7 6 8 7 6 4 6 8 8 6
26782 160/300 Read block 2556 len 4096
26783 161/300 Read block 4481 len 8192
26784 162/300 Read block 327 len 32768
26785 163/300 Write block 511 len 24576 data: 6 6 6 6 5 4
26786 164/300 Write block 883 len 28672 data: 8 10 8 7 6 9 5
26787 165/300 Write block 2598 len 20480 data: 9 9 8 8 9
26788 166/300 Write block 1977 len 4096 data: 4
26789 167/300 Write block 4174 len 40960 data: 13 9 8 8 8 9 9 7 5 6
26790 168/300 Flush
26791 169/300 Write block 1054 len 32768 data: 5 6 4 4 5 5 5 5
26792 170/300 Write block 437 len 40960 data: 6 6 5 8 7 6 6 7 7 9
26793 171/300 Flush
26794 172/300 Write block 4062 len 28672 data: 5 5 4 3 5 5 5
26795 173/300 Read block 1552 len 40960
26796 174/300 Flush
26797 175/300 Read block 3528 len 12288
26798 176/300 Write block 693 len 20480 data: 5 6 6 7 9
26799 177/300 Flush
26800 178/300 Write block 2752 len 4096 data: 9
26801 179/300 Read block 321 len 12288
26802 180/300 Write block 4232 len 12288 data: 6 5 5
26803 181/300 Write block 3426 len 24576 data: 9 8 9 11 11 10
26804 182/300 Write block 2639 len 28672 data: 4 5 5 6 7 8 9
26805 183/300 Read block 4413 len 16384
26806 184/300 Read block 4985 len 8192
26807 185/300 Read block 4735 len 20480
26808 186/300 Read block 3985 len 28672
26809 187/300 Write block 683 len 12288 data: 5 6 6
26810 188/300 Read block 3506 len 32768
26811 189/300 Read block 204 len 32768
26812 190/300 Read block 1476 len 8192
26813 191/300 Flush
26814 192/300 Read block 3051 len 36864
26815 193/300 Write block 2670 len 8192 data: 5 4
26816 194/300 Read block 4543 len 28672
26817 195/300 Write block 3758 len 24576 data: 5 5 7 7 7 6
26818 196/300 Read block 3677 len 4096
26819 197/300 Read block 3283 len 8192
26820 198/300 Write block 1439 len 20480 data: 8 7 7 8 8
26821 199/300 Write block 292 len 24576 data: 6 6 5 5 4 5
26822 200/300 Write block 4386 len 12288 data: 7 7 6
26823 201/300 Read block 2763 len 8192
26824 202/300 Read block 2694 len 4096
26825 203/300 Write block 2052 len 8192 data: 6 4
26826 204/300 Write block 597 len 12288 data: 7 6 7
26827 205/300 Write block 2631 len 16384 data: 4 4 4 7
26828 206/300 Write block 425 len 40960 data: 3 4 3 3 4 5 5 5 7 7
26829 207/300 Read block 3995 len 40960
26830 208/300 Flush
26831 209/300 Read block 2224 len 12288
26832 210/300 Write block 2496 len 24576 data: 5 4 4 3 3 3
26833 211/300 Read block 721 len 16384
26834 212/300 Read block 4433 len 24576
26835 213/300 Flush
26836 214/300 Read block 2009 len 16384
26837 215/300 Read block 2403 len 12288
26838 216/300 Read block 3833 len 24576
26839 217/300 Write block 4402 len 20480 data: 6 7 7 7 6
26840 218/300 Read block 2608 len 12288
26841 219/300 Read block 4507 len 40960
26842 220/300 Read block 1154 len 32768
26843 221/300 Read block 893 len 32768
26844 222/300 Read block 4613 len 40960
26845 223/300 Read block 3374 len 16384
26846 224/300 Read block 4026 len 16384
26847 225/300 Write block 1293 len 40960 data: 3 3 3 3 4 4 5 5 6 7
26848 226/300 Read block 3434 len 24576
26849 227/300 Write block 2267 len 16384 data: 5 6 7 6
26850 228/300 Write block 3199 len 12288 data: 3 3 3
26851 229/300 Write block 1278 len 20480 data: 7 6 6 6 3
26852 230/300 Write block 3522 len 24576 data: 8 6 7 7 7 7
26853 231/300 Write block 2407 len 24576 data: 7 6 5 5 5 4
26854 232/300 Read block 3339 len 28672
26855 233/300 Read block 2049 len 12288
26856 234/300 Read block 3206 len 20480
26857 235/300 Write block 4818 len 36864 data: 6 7 7 5 6 6 7 6 4
26858 236/300 Write block 1802 len 16384 data: 5 6 7 8
26859 237/300 Flush
26860 238/300 Read block 3477 len 16384
26861 239/300 Flush
26862 240/300 Read block 3060 len 16384
26863 241/300 Read block 4041 len 12288
26864 242/300 Read block 4459 len 20480
26865 243/300 Flush
26866 244/300 Read block 4281 len 40960
26867 245/300 Read block 940 len 40960
26868 246/300 Flush
26869 247/300 Read block 4013 len 24576
26870 248/300 Read block 2043 len 36864
26871 249/300 Write block 3415 len 40960 data: 4 5 5 5 4 4 5 6 6 8
26872 250/300 Write block 2195 len 28672 data: 2 3 3 4 5 5 7
26873 251/300 Read block 4812 len 16384
26874 252/300 Write block 1207 len 20480 data: 8 7 8 8 7
26875 253/300 Write block 2441 len 20480 data: 5 6 7 11 11
26876 254/300 Read block 4613 len 32768
26877 255/300 Write block 2418 len 8192 data: 5 5
26878 256/300 Read block 4568 len 32768
26879 257/300 Read block 3098 len 4096
26880 258/300 Read block 3639 len 24576
26881 259/300 Read block 2975 len 24576
26882 260/300 Read block 946 len 32768
26883 261/300 Read block 817 len 12288
26884 262/300 Write block 2100 len 16384 data: 7 6 5 4
26885 263/300 Write block 3761 len 32768 data: 8 8 7 5 6 6 4 7
26886 264/300 Write block 632 len 8192 data: 2 4
26887 265/300 Write block 1546 len 8192 data: 7 7
26888 266/300 Read block 2939 len 28672
26889 267/300 Read block 2342 len 20480
26890 268/300 Read block 800 len 32768
26891 269/300 Read block 688 len 28672
26892 270/300 Flush
26893 271/300 Read block 3233 len 24576
26894 272/300 Read block 4772 len 24576
26895 273/300 Read block 3177 len 36864
26896 274/300 Read block 2366 len 20480
26897 275/300 Read block 1057 len 40960
26898 276/300 Write block 1109 len 4096 data: 5
26899 277/300 Flush
26900 278/300 Read block 563 len 20480
26901 279/300 Read block 219 len 8192
26902 280/300 Write block 3655 len 36864 data: 7 6 4 5 4 4 4 6 6
26903 281/300 Read block 4894 len 8192
26904 282/300 Write block 1983 len 36864 data: 4 4 5 4 5 5 6 6 6
26905 283/300 Write block 1279 len 24576 data: 7 7 7 4 5 4
26906 284/300 Flush
26907 285/300 Write block 3491 len 40960 data: 8 9 8 5 6 7 5 5 5 6
26908 286/300 Write block 1863 len 28672 data: 10 8 7 6 4 4 3
26909 287/300 Flush
26910 288/300 Read block 265 len 12288
26911 289/300 Read block 725 len 28672
26912 290/300 Read block 3476 len 24576
26913 291/300 Write block 2646 len 32768 data: 9 10 10 9 7 6 6 5
26914 292/300 Write block 2856 len 36864 data: 9 8 8 6 4 4 4 5 5
26915 293/300 Write block 2992 len 40960 data: 4 5 5 5 4 4 3 3 4 5
26916 294/300 Write block 637 len 36864 data: 8 6 7 6 7 7 5 6 7
26917 295/300 Read block 3835 len 32768
26918 296/300 Read block 4458 len 28672
26919 297/300 Read block 1014 len 36864
26920 298/300 Read block 363 len 24576
26921 299/300 Read block 792 len 36864
26922 300/300 Write block 4473 len 8192 data: 8 9
26923 Main task has work to do, go find it
26924 got dsc Start(2)
26925 start 2
26926 Feb 12 01:39:52.437 INFO request completed, latency_us: 223, response_code: 204, uri: /start/cid/2, method: POST, req_id: 56162fdf-c89f-4c3c-a1b9-3b0635ac49ce, remote_addr: 127.0.0.1:60456, local_addr: 127.0.0.1:9998
26927 Replay: started 2, returned:Ok(())
26928 ----------------------------------------------------------------
26929 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
26930 States: Active Active Offline
26931 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
26932 8743 Acked 9742 Write 7 Done Done New true
26933 8744 Acked 9743 Write 1 Done Done New true
26934 8745 Acked 9744 Read 4 Done Done New true
26935 8746 Acked 9745 Read 6 Done Done New true
26936 8747 Acked 9746 Read 7 Done Done New false
26937 8748 Acked 9747 Write 8 Done Done New false
26938 8749 Acked 9748 Read 1 Done Done New false
26939 8750 Acked 9749 Write 1 Done Done New false
26940 8751 Acked 9750 Write 6 Done Done New false
26941 8752 Acked 9751 Flush 0 Done Done New false
26942 8753 Acked 9752 Read 4 Done Done New false
26943 8754 Acked 9753 Read 8 Done Done New false
26944 8755 Acked 9754 Read 7 Done Done New false
26945 8756 Acked 9755 Read 4 Done Done New false
26946 8757 Acked 9756 Read 1 Done Done New false
26947 8758 Acked 9757 Read 3 Done Done New false
26948 8759 Acked 9758 Read 9 Done Done New false
26949 8760 Acked 9759 Write 1 Done Done New false
26950 8761 Acked 9760 Read 3 Done Done New false
26951 8762 Acked 9761 Read 9 Done Done New false
26952 8763 Acked 9762 Read 7 Done Done New false
26953 8764 Acked 9763 Write 6 Done Done New false
26954 8765 Acked 9764 Flush 0 Done Done New false
26955 8766 Acked 9765 Read 1 Done Done New false
26956 8767 Acked 9766 Read 5 Done Done New false
26957 8768 Acked 9767 Write 4 Done Done New false
26958 8769 Acked 9768 Read 3 Done Done New false
26959 8770 Acked 9769 Read 9 Done Done New false
26960 8771 Acked 9770 Write 4 Done Done New false
26961 8772 Acked 9771 Read 7 Done Done New false
26962 8773 Acked 9772 Read 9 Done Done New false
26963 8774 Acked 9773 Read 10 Done Done New false
26964 8775 Acked 9774 Read 6 Done Done New false
26965 8776 Acked 9775 Write 2 Done Done New false
26966 8777 Acked 9776 Read 1 Done Done New false
26967 8778 Acked 9777 Read 9 Done Done New false
26968 8779 Acked 9778 Flush 0 Done Done New false
26969 8780 Acked 9779 Read 9 Done Done New false
26970 8781 Acked 9780 Read 3 Done Done New false
26971 8782 Acked 9781 Read 6 Done Done New false
26972 8783 Acked 9782 Flush 0 Done Done New false
26973 8784 Acked 9783 Flush 0 Done Done New false
26974 8785 Acked 9784 Write 2 Done Done New false
26975 8786 Acked 9785 Write 10 Done Done New false
26976 8787 Acked 9786 Write 5 Done Done New false
26977 8788 Acked 9787 Write 8 Done Done New false
26978 8789 Acked 9788 Write 6 Done Done New false
26979 8790 Acked 9789 Write 6 Done Done New false
26980 8791 Acked 9790 Read 7 Done Done New false
26981 8792 Acked 9791 Write 1 Done Done New false
26982 8793 Acked 9792 Read 10 Done Done New false
26983 8794 Acked 9793 Read 1 Done Done New false
26984 8795 Acked 9794 Read 2 Done Done New false
26985 8796 Acked 9795 Write 10 Done Done New false
26986 8797 Acked 9796 Flush 0 Done Done New false
26987 8798 Acked 9797 Write 6 Done Done New false
26988 8799 Acked 9798 Write 10 Done Done New false
26989 8800 Acked 9799 Write 9 Done Done New false
26990 8801 Acked 9800 Read 1 Done Done New false
26991 8802 Acked 9801 Read 2 Done Done New false
26992 8803 Acked 9802 Read 3 Done Done New false
26993 8804 Acked 9803 Write 4 Done Done New false
26994 8805 Acked 9804 Read 8 Done Done New false
26995 8806 Acked 9805 Read 2 Done Done New false
26996 8807 Acked 9806 Read 10 Done Done New false
26997 8808 Acked 9807 Write 7 Done Done New false
26998 8809 Acked 9808 Read 3 Done Done New false
26999 8810 Acked 9809 Read 3 Done Done New false
27000 8811 Acked 9810 Read 6 Done Done New false
27001 8812 Acked 9811 Write 7 Done Done New false
27002 8813 Acked 9812 Read 10 Done Done New false
27003 8814 Acked 9813 Read 10 Done Done New false
27004 8815 Acked 9814 Flush 0 Done Done New false
27005 8816 Acked 9815 Read 9 Done Done New false
27006 8817 Acked 9816 Write 1 Done Done New false
27007 8818 Acked 9817 Write 5 Done Done New false
27008 8819 Acked 9818 Write 9 Done Done New false
27009 8820 Acked 9819 Flush 0 Done Done New false
27010 8821 Acked 9820 Write 7 Done Done New false
27011 8822 Acked 9821 Read 3 Done Done New false
27012 8823 Acked 9822 Write 10 Done Done New false
27013 8824 Acked 9823 Write 3 Done Done New false
27014 8825 Acked 9824 Read 4 Done Done New false
27015 8826 Acked 9825 Flush 0 Done Done New false
27016 8827 Acked 9826 Read 6 Done Done New false
27017 8828 Acked 9827 Write 1 Done Done New false
27018 8829 Acked 9828 Write 4 Done Done New false
27019 8830 Acked 9829 Flush 0 Done Done New false
27020 8831 Acked 9830 Flush 0 Done Done New false
27021 8832 Acked 9831 Flush 0 Done Done New false
27022 8833 Acked 9832 Write 5 Done Done New false
27023 8834 Acked 9833 Write 1 Done Done New false
27024 8835 Acked 9834 Write 6 Done Done New false
27025 8836 Acked 9835 Write 5 Done Done New false
27026 8837 Acked 9836 Write 10 Done Done New false
27027 8838 Acked 9837 Read 8 Done Done New false
27028 8839 Acked 9838 Read 3 Done Done New false
27029 8840 Acked 9839 Read 7 Done Done New false
27030 8841 Acked 9840 Read 3 Done Done New false
27031 8842 Acked 9841 Flush 0 Done Done New false
27032 8843 Acked 9842 Read 3 Done Done New false
27033 8844 Acked 9843 Write 5 Done Done New false
27034 8845 Acked 9844 Write 1 Done Done New false
27035 8846 Acked 9845 Read 8 Done Done New false
27036 8847 Acked 9846 Read 10 Done Done New false
27037 8848 Acked 9847 Write 6 Done Done New false
27038 8849 Acked 9848 Write 4 Done Done New false
27039 8850 Acked 9849 Flush 0 Done Done New false
27040 8851 Acked 9850 Write 2 Done Done New false
27041 8852 Acked 9851 Write 3 Done Done New false
27042 8853 Acked 9852 Write 9 Done Done New false
27043 8854 Acked 9853 Write 4 Done Done New false
27044 8855 Acked 9854 Read 2 Done Done New false
27045 8856 Acked 9855 Write 10 Done Done New false
27046 8857 Acked 9856 Flush 0 Done Done New false
27047 8858 Acked 9857 Write 3 Done Done New false
27048 8859 Acked 9858 Write 2 Done Done New false
27049 8860 Acked 9859 Read 10 Done Done New false
27050 8861 Acked 9860 Read 7 Done Done New false
27051 8862 Acked 9861 Read 5 Done Done New false
27052 8863 Acked 9862 Write 1 Done Done New false
27053 8864 Acked 9863 Read 7 Done Done New false
27054 8865 Acked 9864 Write 5 Done Done New false
27055 8866 Acked 9865 Flush 0 Done Done New false
27056 8867 Acked 9866 Read 6 Done Done New false
27057 8868 Acked 9867 Write 4 Done Done New false
27058 8869 Acked 9868 Write 1 Done Done New false
27059 8870 Acked 9869 Write 2 Done Done New false
27060 8871 Acked 9870 Write 4 Done Done New false
27061 8872 Acked 9871 Write 5 Done Done New false
27062 8873 Acked 9872 Read 1 Done Done New false
27063 8874 Acked 9873 Write 7 Done Done New false
27064 8875 Acked 9874 Read 10 Done Done New false
27065 8876 Acked 9875 Flush 0 Done Done New false
27066 8877 Acked 9876 Read 2 Done Done New false
27067 8878 Acked 9877 Flush 0 Done Done New false
27068 8879 Acked 9878 Write 1 Done Done New false
27069 8880 Acked 9879 Flush 0 Done Done New false
27070 8881 Acked 9880 Read 7 Done Done New false
27071 8882 Acked 9881 Read 7 Done Done New false
27072 8883 Acked 9882 Read 1 Done Done New false
27073 8884 Acked 9883 Flush 0 Done Done New false
27074 8885 Acked 9884 Write 4 Done Done New false
27075 8886 Acked 9885 Write 6 Done Done New false
27076 8887 Acked 9886 Read 8 Done Done New false
27077 8888 Acked 9887 Read 2 Done Done New false
27078 8889 Acked 9888 Write 5 Done Done New false
27079 8890 Acked 9889 Write 7 Done Done New false
27080 8891 Acked 9890 Write 1 Done Done New false
27081 8892 Acked 9891 Write 7 Done Done New false
27082 8893 Acked 9892 Write 2 Done Done New false
27083 8894 Acked 9893 Read 4 Done Done New false
27084 8895 Acked 9894 Write 4 Done Done New false
27085 8896 Acked 9895 Read 9 Done Done New false
27086 8897 Acked 9896 Read 10 Done Done New false
27087 8898 Acked 9897 Write 2 Done Done New false
27088 8899 Acked 9898 Read 1 Done Done New false
27089 8900 Acked 9899 Write 7 Done Done New false
27090 8901 Acked 9900 Write 10 Done Done New false
27091 8902 Acked 9901 Read 1 Done Done New false
27092 8903 Acked 9902 Read 2 Done Done New false
27093 8904 Acked 9903 Read 8 Done Done New false
27094 8905 Acked 9904 Write 6 Done Done New false
27095 8906 Acked 9905 Write 7 Done Done New false
27096 8907 Acked 9906 Write 5 Done Done New false
27097 8908 Acked 9907 Write 1 Done Done New false
27098 8909 Acked 9908 Write 10 Done Done New false
27099 8910 Acked 9909 Flush 0 Done Done New false
27100 8911 Acked 9910 Write 8 Done Done New false
27101 8912 Acked 9911 Write 10 Done Done New false
27102 8913 Acked 9912 Flush 0 Done Done New false
27103 8914 Acked 9913 Write 7 Done Done New false
27104 8915 Acked 9914 Read 10 Done Done New false
27105 8916 Acked 9915 Flush 0 Done Done New false
27106 8917 Acked 9916 Read 3 Done Done New false
27107 8918 Acked 9917 Write 5 Done Done New false
27108 8919 Acked 9918 Flush 0 Done Done New false
27109 8920 Acked 9919 Write 1 Done Done New false
27110 8921 Acked 9920 Read 3 Done Done New false
27111 8922 Acked 9921 Write 3 Done Done New false
27112 8923 Acked 9922 Write 6 Done Done New false
27113 8924 Acked 9923 Write 7 Done Done New false
27114 8925 Acked 9924 Read 4 Done Done New false
27115 8926 Acked 9925 Read 2 Done Done New false
27116 8927 Acked 9926 Read 5 Done Done New false
27117 8928 Acked 9927 Read 7 Done Done New false
27118 8929 Acked 9928 Write 3 Done Done New false
27119 8930 Acked 9929 Read 8 Done Done New false
27120 8931 Acked 9930 Read 8 Done Done New false
27121 8932 Acked 9931 Read 2 Done Done New false
27122 8933 Acked 9932 Flush 0 Done Done New false
27123 8934 Acked 9933 Read 9 Done Done New false
27124 8935 Acked 9934 Write 2 Done Done New false
27125 8936 Acked 9935 Read 7 Done Done New false
27126 8937 Acked 9936 Write 6 Done Done New false
27127 8938 Acked 9937 Read 1 Done Done New false
27128 8939 Acked 9938 Read 2 Done Done New false
27129 8940 Acked 9939 Write 5 Done Done New false
27130 8941 Acked 9940 Write 6 Done Done New false
27131 8942 Acked 9941 Write 3 Done Done New false
27132 8943 Acked 9942 Read 2 Done Done New false
27133 8944 Acked 9943 Read 1 Done Done New false
27134 8945 Acked 9944 Write 2 Done Done New false
27135 8946 Acked 9945 Write 3 Done Done New false
27136 8947 Acked 9946 Write 4 Done Done New false
27137 8948 Acked 9947 Write 10 Done Done New false
27138 8949 Acked 9948 Read 10 Done Done New false
27139 8950 Acked 9949 Flush 0 Done Done New false
27140 8951 Acked 9950 Read 3 Done Done New false
27141 8952 Acked 9951 Write 6 Done Done New false
27142 8953 Acked 9952 Read 4 Done Done New false
27143 8954 Acked 9953 Read 6 Done Done New false
27144 8955 Acked 9954 Flush 0 Done Done New false
27145 8956 Acked 9955 Read 4 Done Done New false
27146 8957 Acked 9956 Read 3 Done Done New false
27147 8958 Acked 9957 Read 6 Done Done New false
27148 8959 Acked 9958 Write 5 Done Done New false
27149 8960 Acked 9959 Read 3 Done Done New false
27150 8961 Acked 9960 Read 10 Done Done New false
27151 8962 Acked 9961 Read 8 Done Done New false
27152 8963 Acked 9962 Read 8 Done Done New false
27153 8964 Acked 9963 Read 10 Done Done New false
27154 8965 Acked 9964 Read 4 Done Done New false
27155 8966 Acked 9965 Read 4 Done Done New false
27156 8967 Acked 9966 Write 10 Done Done New false
27157 8968 Acked 9967 Read 6 Done Done New false
27158 8969 Acked 9968 Write 4 Done Done New false
27159 8970 Acked 9969 Write 3 Done Done New false
27160 8971 Acked 9970 Write 5 Done Done New false
27161 8972 Acked 9971 Write 6 Done Done New false
27162 8973 Acked 9972 Write 6 Done Done New false
27163 8974 Acked 9973 Read 7 Done Done New false
27164 8975 Acked 9974 Read 3 Done Done New false
27165 8976 Acked 9975 Read 5 Done Done New false
27166 8977 Acked 9976 Write 9 Done Done New false
27167 8978 Acked 9977 Write 4 Done Done New false
27168 8979 Acked 9978 Flush 0 Done Done New false
27169 8980 Acked 9979 Read 4 Done Done New false
27170 8981 Acked 9980 Flush 0 Done Done New false
27171 8982 Acked 9981 Read 4 Done Done New false
27172 8983 Acked 9982 Read 3 Done Done New false
27173 8984 Acked 9983 Read 5 Done Done New false
27174 8985 Acked 9984 Flush 0 Done Done New false
27175 8986 Acked 9985 Read 10 Done Done New false
27176 8987 Acked 9986 Read 10 Done Done New false
27177 8988 Acked 9987 Flush 0 Done Done New false
27178 8989 Acked 9988 Read 6 Done Done New false
27179 8990 Acked 9989 Read 9 Done Done New false
27180 8991 Acked 9990 Write 10 Done Done New false
27181 8992 Acked 9991 Write 7 Done Done New false
27182 8993 Acked 9992 Read 4 Done Done New false
27183 8994 Acked 9993 Write 5 Done Done New false
27184 8995 Acked 9994 Write 5 Done Done New false
27185 8996 Acked 9995 Read 8 Done Done New false
27186 8997 Acked 9996 Flush 0 Done Done New false
27187 8998 Acked 9997 Write 2 Done Done New false
27188 8999 Acked 9998 Read 8 Done Done New false
27189 9000 Acked 9999 Read 1 Done Done New false
27190 9001 Acked 10000 Read 6 Done Done New false
27191 9002 Acked 10001 Read 6 Done Done New false
27192 9003 Acked 10002 Read 8 Done Done New false
27193 9004 Acked 10003 Read 3 Done Done New false
27194 9005 Acked 10004 Write 4 Done Done New false
27195 9006 Acked 10005 Write 8 Done Done New false
27196 9007 Acked 10006 Write 2 Done Done New false
27197 9008 Acked 10007 Write 2 Done Done New false
27198 9009 Acked 10008 Read 7 Done Done New false
27199 9010 Acked 10009 Read 5 Done Done New false
27200 9011 Acked 10010 Read 8 Done Done New false
27201 9012 Acked 10011 Read 7 Done Done New false
27202 9013 Acked 10012 Flush 0 Done Done New false
27203 9014 Acked 10013 Read 6 Done Done New false
27204 9015 Acked 10014 Read 6 Done Done New false
27205 9016 Acked 10015 Read 9 Done Done New false
27206 9017 Acked 10016 Read 5 Done Done New false
27207 9018 Acked 10017 Read 10 Done Done New false
27208 9019 Acked 10018 Write 1 Done Done New false
27209 9020 Acked 10019 Flush 0 Done Done New false
27210 9021 Acked 10020 Read 5 Done Done New false
27211 9022 Acked 10021 Read 2 Done Done New false
27212 9023 Acked 10022 Write 9 Done Done New false
27213 9024 Acked 10023 Read 2 Done Done New false
27214 9025 Acked 10024 Write 9 Done Done New false
27215 9026 Acked 10025 Write 6 Done Done New false
27216 9027 Acked 10026 Flush 0 Done Done New false
27217 9028 Acked 10027 Write 10 Done Done New false
27218 9029 Acked 10028 Write 7 Done Done New false
27219 9030 Acked 10029 Flush 0 Done Done New false
27220 9031 Acked 10030 Read 3 Done Done New false
27221 9032 Acked 10031 Read 7 Done Done New false
27222 9033 Acked 10032 Read 6 Done Done New false
27223 9034 Acked 10033 Write 8 Done Done New false
27224 9035 Acked 10034 Write 9 Done Done New false
27225 9036 Acked 10035 Write 10 Done Done New false
27226 9037 Acked 10036 Write 9 Done Done New false
27227 9038 Acked 10037 Read 8 Done Done New false
27228 9039 Acked 10038 Read 7 Done Done New false
27229 9040 Acked 10039 Read 9 Done Done New false
27230 9041 Acked 10040 Read 6 Done Done New false
27231 9042 Acked 10041 Read 9 Done Done New false
27232 9043 Acked 10042 Write 2 Done Done New false
27233 STATES DS:0 DS:1 DS:2 TOTAL
27234 New 0 0 301 301
27235 Sent 0 0 0 0
27236 Done 301 301 0 602
27237 Skipped 0 0 0 0
27238 Error 0 0 0 0
27239 Last Flush: 10029 10029 9741
27240 Downstairs last five completed: 9741 9740 9739 9738 9737
27241 Upstairs last five completed: 9043 9042 9041 9040 9039
27242 CLIENT: Up:0 ds:301 act:2
272432024-02-12T01:39:53.098ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
272442024-02-12T01:39:53.098ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
272452024-02-12T01:39:53.098ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
272462024-02-12T01:39:53.098ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
272472024-02-12T01:39:53.098ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
272482024-02-12T01:39:53.098ZINFOcrucible: [2] client re-new 302 jobs since flush 9741 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
272492024-02-12T01:39:53.098ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
272502024-02-12T01:39:54.099ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
272512024-02-12T01:39:54.099ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
272522024-02-12T01:39:54.099ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
272532024-02-12T01:39:54.099ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
272542024-02-12T01:39:54.099ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
272552024-02-12T01:39:54.100ZINFOcrucible: [2] client re-new 302 jobs since flush 9741 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
272562024-02-12T01:39:54.100ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
272572024-02-12T01:39:55.102ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
272582024-02-12T01:39:55.102ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
272592024-02-12T01:39:55.102ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
272602024-02-12T01:39:55.102ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
272612024-02-12T01:39:55.102ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
272622024-02-12T01:39:55.102ZINFOcrucible: [2] client re-new 302 jobs since flush 9741 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
272632024-02-12T01:39:55.102ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
27264 [8830] got start action
27265 [8830] I am going to restart
27266 Starting downstairs at port 8830
27267 Make output file at "/tmp/dsc/downstairs-8830.txt"
27268 [8830][2] reports Starting
272692024-02-12T01:39:56.103ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
272702024-02-12T01:39:56.103ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
272712024-02-12T01:39:56.103ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
272722024-02-12T01:39:56.103ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
272732024-02-12T01:39:56.103ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
272742024-02-12T01:39:56.103ZINFOcrucible: [2] client re-new 302 jobs since flush 9741 = downstairs session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
272752024-02-12T01:39:56.103ZINFOcrucible: client task is exiting = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
27276 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1194)
27277 [8830][2] reports Running
27278 ----------------------------------------------------------------
27279 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
27280 States: Active Active Offline
27281 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
27282 8743 Acked 9742 Write 7 Done Done New true
27283 8744 Acked 9743 Write 1 Done Done New true
27284 8745 Acked 9744 Read 4 Done Done New true
27285 8746 Acked 9745 Read 6 Done Done New true
27286 8747 Acked 9746 Read 7 Done Done New true
27287 8748 Acked 9747 Write 8 Done Done New true
27288 8749 Acked 9748 Read 1 Done Done New true
27289 8750 Acked 9749 Write 1 Done Done New true
27290 8751 Acked 9750 Write 6 Done Done New true
27291 8752 Acked 9751 Flush 0 Done Done New true
27292 8753 Acked 9752 Read 4 Done Done New true
27293 8754 Acked 9753 Read 8 Done Done New true
27294 8755 Acked 9754 Read 7 Done Done New true
27295 8756 Acked 9755 Read 4 Done Done New true
27296 8757 Acked 9756 Read 1 Done Done New true
27297 8758 Acked 9757 Read 3 Done Done New true
27298 8759 Acked 9758 Read 9 Done Done New true
27299 8760 Acked 9759 Write 1 Done Done New true
27300 8761 Acked 9760 Read 3 Done Done New true
27301 8762 Acked 9761 Read 9 Done Done New true
27302 8763 Acked 9762 Read 7 Done Done New true
27303 8764 Acked 9763 Write 6 Done Done New true
27304 8765 Acked 9764 Flush 0 Done Done New true
27305 8766 Acked 9765 Read 1 Done Done New true
27306 8767 Acked 9766 Read 5 Done Done New true
27307 8768 Acked 9767 Write 4 Done Done New true
27308 8769 Acked 9768 Read 3 Done Done New true
27309 8770 Acked 9769 Read 9 Done Done New true
27310 8771 Acked 9770 Write 4 Done Done New true
27311 8772 Acked 9771 Read 7 Done Done New true
27312 8773 Acked 9772 Read 9 Done Done New true
27313 8774 Acked 9773 Read 10 Done Done New true
27314 8775 Acked 9774 Read 6 Done Done New true
27315 8776 Acked 9775 Write 2 Done Done New true
27316 8777 Acked 9776 Read 1 Done Done New true
27317 8778 Acked 9777 Read 9 Done Done New true
27318 8779 Acked 9778 Flush 0 Done Done New true
27319 8780 Acked 9779 Read 9 Done Done New true
27320 8781 Acked 9780 Read 3 Done Done New true
27321 8782 Acked 9781 Read 6 Done Done New true
27322 8783 Acked 9782 Flush 0 Done Done New true
27323 8784 Acked 9783 Flush 0 Done Done New true
27324 8785 Acked 9784 Write 2 Done Done New true
27325 8786 Acked 9785 Write 10 Done Done New true
27326 8787 Acked 9786 Write 5 Done Done New true
27327 8788 Acked 9787 Write 8 Done Done New true
27328 8789 Acked 9788 Write 6 Done Done New true
27329 8790 Acked 9789 Write 6 Done Done New true
27330 8791 Acked 9790 Read 7 Done Done New true
27331 8792 Acked 9791 Write 1 Done Done New true
27332 8793 Acked 9792 Read 10 Done Done New true
27333 8794 Acked 9793 Read 1 Done Done New true
27334 8795 Acked 9794 Read 2 Done Done New true
27335 8796 Acked 9795 Write 10 Done Done New true
27336 8797 Acked 9796 Flush 0 Done Done New true
27337 8798 Acked 9797 Write 6 Done Done New true
27338 8799 Acked 9798 Write 10 Done Done New true
27339 8800 Acked 9799 Write 9 Done Done New true
27340 8801 Acked 9800 Read 1 Done Done New true
27341 8802 Acked 9801 Read 2 Done Done New true
27342 8803 Acked 9802 Read 3 Done Done New true
27343 8804 Acked 9803 Write 4 Done Done New true
27344 8805 Acked 9804 Read 8 Done Done New true
27345 8806 Acked 9805 Read 2 Done Done New true
27346 8807 Acked 9806 Read 10 Done Done New true
27347 8808 Acked 9807 Write 7 Done Done New true
27348 8809 Acked 9808 Read 3 Done Done New true
27349 8810 Acked 9809 Read 3 Done Done New true
27350 8811 Acked 9810 Read 6 Done Done New true
27351 8812 Acked 9811 Write 7 Done Done New true
27352 8813 Acked 9812 Read 10 Done Done New true
27353 8814 Acked 9813 Read 10 Done Done New true
27354 8815 Acked 9814 Flush 0 Done Done New true
27355 8816 Acked 9815 Read 9 Done Done New true
27356 8817 Acked 9816 Write 1 Done Done New true
27357 8818 Acked 9817 Write 5 Done Done New true
27358 8819 Acked 9818 Write 9 Done Done New true
27359 8820 Acked 9819 Flush 0 Done Done New true
27360 8821 Acked 9820 Write 7 Done Done New true
27361 8822 Acked 9821 Read 3 Done Done New true
27362 8823 Acked 9822 Write 10 Done Done New true
27363 8824 Acked 9823 Write 3 Done Done New true
27364 8825 Acked 9824 Read 4 Done Done New true
27365 8826 Acked 9825 Flush 0 Done Done New true
27366 8827 Acked 9826 Read 6 Done Done New true
27367 8828 Acked 9827 Write 1 Done Done New true
27368 8829 Acked 9828 Write 4 Done Done New true
27369 8830 Acked 9829 Flush 0 Done Done New true
27370 8831 Acked 9830 Flush 0 Done Done New true
27371 8832 Acked 9831 Flush 0 Done Done New true
27372 8833 Acked 9832 Write 5 Done Done New true
27373 8834 Acked 9833 Write 1 Done Done New true
27374 8835 Acked 9834 Write 6 Done Done New true
27375 8836 Acked 9835 Write 5 Done Done New true
27376 8837 Acked 9836 Write 10 Done Done New true
27377 8838 Acked 9837 Read 8 Done Done New true
27378 8839 Acked 9838 Read 3 Done Done New true
27379 8840 Acked 9839 Read 7 Done Done New true
27380 8841 Acked 9840 Read 3 Done Done New true
27381 8842 Acked 9841 Flush 0 Done Done New true
27382 8843 Acked 9842 Read 3 Done Done New true
27383 8844 Acked 9843 Write 5 Done Done New true
27384 8845 Acked 9844 Write 1 Done Done New true
27385 8846 Acked 9845 Read 8 Done Done New true
27386 8847 Acked 9846 Read 10 Done Done New true
27387 8848 Acked 9847 Write 6 Done Done New true
27388 8849 Acked 9848 Write 4 Done Done New true
27389 8850 Acked 9849 Flush 0 Done Done New true
27390 8851 Acked 9850 Write 2 Done Done New true
27391 8852 Acked 9851 Write 3 Done Done New true
27392 8853 Acked 9852 Write 9 Done Done New true
27393 8854 Acked 9853 Write 4 Done Done New true
27394 8855 Acked 9854 Read 2 Done Done New true
27395 8856 Acked 9855 Write 10 Done Done New true
27396 8857 Acked 9856 Flush 0 Done Done New true
27397 8858 Acked 9857 Write 3 Done Done New true
27398 8859 Acked 9858 Write 2 Done Done New true
27399 8860 Acked 9859 Read 10 Done Done New true
27400 8861 Acked 9860 Read 7 Done Done New true
27401 8862 Acked 9861 Read 5 Done Done New true
27402 8863 Acked 9862 Write 1 Done Done New true
27403 8864 Acked 9863 Read 7 Done Done New true
27404 8865 Acked 9864 Write 5 Done Done New true
27405 8866 Acked 9865 Flush 0 Done Done New true
27406 8867 Acked 9866 Read 6 Done Done New true
27407 8868 Acked 9867 Write 4 Done Done New true
27408 8869 Acked 9868 Write 1 Done Done New true
27409 8870 Acked 9869 Write 2 Done Done New true
27410 8871 Acked 9870 Write 4 Done Done New true
27411 8872 Acked 9871 Write 5 Done Done New true
27412 8873 Acked 9872 Read 1 Done Done New true
27413 8874 Acked 9873 Write 7 Done Done New true
27414 8875 Acked 9874 Read 10 Done Done New true
27415 8876 Acked 9875 Flush 0 Done Done New true
27416 8877 Acked 9876 Read 2 Done Done New true
27417 8878 Acked 9877 Flush 0 Done Done New true
27418 8879 Acked 9878 Write 1 Done Done New true
27419 8880 Acked 9879 Flush 0 Done Done New true
27420 8881 Acked 9880 Read 7 Done Done New true
27421 8882 Acked 9881 Read 7 Done Done New true
27422 8883 Acked 9882 Read 1 Done Done New true
27423 8884 Acked 9883 Flush 0 Done Done New true
27424 8885 Acked 9884 Write 4 Done Done New true
27425 8886 Acked 9885 Write 6 Done Done New true
27426 8887 Acked 9886 Read 8 Done Done New true
27427 8888 Acked 9887 Read 2 Done Done New true
27428 8889 Acked 9888 Write 5 Done Done New true
27429 8890 Acked 9889 Write 7 Done Done New true
27430 8891 Acked 9890 Write 1 Done Done New true
27431 8892 Acked 9891 Write 7 Done Done New true
27432 8893 Acked 9892 Write 2 Done Done New true
27433 8894 Acked 9893 Read 4 Done Done New true
27434 8895 Acked 9894 Write 4 Done Done New true
27435 8896 Acked 9895 Read 9 Done Done New true
27436 8897 Acked 9896 Read 10 Done Done New true
27437 8898 Acked 9897 Write 2 Done Done New true
27438 8899 Acked 9898 Read 1 Done Done New true
27439 8900 Acked 9899 Write 7 Done Done New true
27440 8901 Acked 9900 Write 10 Done Done New true
27441 8902 Acked 9901 Read 1 Done Done New true
27442 8903 Acked 9902 Read 2 Done Done New true
27443 8904 Acked 9903 Read 8 Done Done New true
27444 8905 Acked 9904 Write 6 Done Done New true
27445 8906 Acked 9905 Write 7 Done Done New true
27446 8907 Acked 9906 Write 5 Done Done New true
27447 8908 Acked 9907 Write 1 Done Done New true
27448 8909 Acked 9908 Write 10 Done Done New true
27449 8910 Acked 9909 Flush 0 Done Done New true
27450 8911 Acked 9910 Write 8 Done Done New true
27451 8912 Acked 9911 Write 10 Done Done New true
27452 8913 Acked 9912 Flush 0 Done Done New true
27453 8914 Acked 9913 Write 7 Done Done New true
27454 8915 Acked 9914 Read 10 Done Done New true
27455 8916 Acked 9915 Flush 0 Done Done New true
27456 8917 Acked 9916 Read 3 Done Done New true
27457 8918 Acked 9917 Write 5 Done Done New true
27458 8919 Acked 9918 Flush 0 Done Done New true
27459 8920 Acked 9919 Write 1 Done Done New true
27460 8921 Acked 9920 Read 3 Done Done New true
27461 8922 Acked 9921 Write 3 Done Done New true
27462 8923 Acked 9922 Write 6 Done Done New true
27463 8924 Acked 9923 Write 7 Done Done New true
27464 8925 Acked 9924 Read 4 Done Done New true
27465 8926 Acked 9925 Read 2 Done Done New true
27466 8927 Acked 9926 Read 5 Done Done New true
27467 8928 Acked 9927 Read 7 Done Done New true
27468 8929 Acked 9928 Write 3 Done Done New true
27469 8930 Acked 9929 Read 8 Done Done New true
27470 8931 Acked 9930 Read 8 Done Done New true
27471 8932 Acked 9931 Read 2 Done Done New true
27472 8933 Acked 9932 Flush 0 Done Done New true
27473 8934 Acked 9933 Read 9 Done Done New true
27474 8935 Acked 9934 Write 2 Done Done New true
27475 8936 Acked 9935 Read 7 Done Done New true
27476 8937 Acked 9936 Write 6 Done Done New true
27477 8938 Acked 9937 Read 1 Done Done New true
27478 8939 Acked 9938 Read 2 Done Done New true
27479 8940 Acked 9939 Write 5 Done Done New true
27480 8941 Acked 9940 Write 6 Done Done New true
27481 8942 Acked 9941 Write 3 Done Done New true
27482 8943 Acked 9942 Read 2 Done Done New true
27483 8944 Acked 9943 Read 1 Done Done New true
27484 8945 Acked 9944 Write 2 Done Done New true
27485 8946 Acked 9945 Write 3 Done Done New true
27486 8947 Acked 9946 Write 4 Done Done New true
27487 8948 Acked 9947 Write 10 Done Done New true
27488 8949 Acked 9948 Read 10 Done Done New true
27489 8950 Acked 9949 Flush 0 Done Done New true
27490 8951 Acked 9950 Read 3 Done Done New true
27491 8952 Acked 9951 Write 6 Done Done New true
27492 8953 Acked 9952 Read 4 Done Done New true
27493 8954 Acked 9953 Read 6 Done Done New true
27494 8955 Acked 9954 Flush 0 Done Done New true
27495 8956 Acked 9955 Read 4 Done Done New true
27496 8957 Acked 9956 Read 3 Done Done New true
27497 8958 Acked 9957 Read 6 Done Done New true
27498 8959 Acked 9958 Write 5 Done Done New true
27499 8960 Acked 9959 Read 3 Done Done New true
27500 8961 Acked 9960 Read 10 Done Done New true
27501 8962 Acked 9961 Read 8 Done Done New true
27502 8963 Acked 9962 Read 8 Done Done New true
27503 8964 Acked 9963 Read 10 Done Done New true
27504 8965 Acked 9964 Read 4 Done Done New true
27505 8966 Acked 9965 Read 4 Done Done New true
27506 8967 Acked 9966 Write 10 Done Done New true
27507 8968 Acked 9967 Read 6 Done Done New true
27508 8969 Acked 9968 Write 4 Done Done New true
27509 8970 Acked 9969 Write 3 Done Done New true
27510 8971 Acked 9970 Write 5 Done Done New true
27511 8972 Acked 9971 Write 6 Done Done New true
27512 8973 Acked 9972 Write 6 Done Done New true
27513 8974 Acked 9973 Read 7 Done Done New true
27514 8975 Acked 9974 Read 3 Done Done New true
27515 8976 Acked 9975 Read 5 Done Done New true
27516 8977 Acked 9976 Write 9 Done Done New true
27517 8978 Acked 9977 Write 4 Done Done New true
27518 8979 Acked 9978 Flush 0 Done Done New true
27519 8980 Acked 9979 Read 4 Done Done New true
27520 8981 Acked 9980 Flush 0 Done Done New true
27521 8982 Acked 9981 Read 4 Done Done New true
27522 8983 Acked 9982 Read 3 Done Done New true
27523 8984 Acked 9983 Read 5 Done Done New true
27524 8985 Acked 9984 Flush 0 Done Done New true
27525 8986 Acked 9985 Read 10 Done Done New true
27526 8987 Acked 9986 Read 10 Done Done New true
27527 8988 Acked 9987 Flush 0 Done Done New true
27528 8989 Acked 9988 Read 6 Done Done New true
27529 8990 Acked 9989 Read 9 Done Done New true
27530 8991 Acked 9990 Write 10 Done Done New true
27531 8992 Acked 9991 Write 7 Done Done New true
27532 8993 Acked 9992 Read 4 Done Done New true
27533 8994 Acked 9993 Write 5 Done Done New true
27534 8995 Acked 9994 Write 5 Done Done New true
27535 8996 Acked 9995 Read 8 Done Done New true
27536 8997 Acked 9996 Flush 0 Done Done New true
27537 8998 Acked 9997 Write 2 Done Done New true
27538 8999 Acked 9998 Read 8 Done Done New true
27539 9000 Acked 9999 Read 1 Done Done New true
27540 9001 Acked 10000 Read 6 Done Done New true
27541 9002 Acked 10001 Read 6 Done Done New true
27542 9003 Acked 10002 Read 8 Done Done New true
27543 9004 Acked 10003 Read 3 Done Done New true
27544 9005 Acked 10004 Write 4 Done Done New true
27545 9006 Acked 10005 Write 8 Done Done New true
27546 9007 Acked 10006 Write 2 Done Done New true
27547 9008 Acked 10007 Write 2 Done Done New true
27548 9009 Acked 10008 Read 7 Done Done New true
27549 9010 Acked 10009 Read 5 Done Done New true
27550 9011 Acked 10010 Read 8 Done Done New true
27551 9012 Acked 10011 Read 7 Done Done New true
27552 9013 Acked 10012 Flush 0 Done Done New true
27553 9014 Acked 10013 Read 6 Done Done New true
27554 9015 Acked 10014 Read 6 Done Done New true
27555 9016 Acked 10015 Read 9 Done Done New true
27556 9017 Acked 10016 Read 5 Done Done New true
27557 9018 Acked 10017 Read 10 Done Done New true
27558 9019 Acked 10018 Write 1 Done Done New true
27559 9020 Acked 10019 Flush 0 Done Done New true
27560 9021 Acked 10020 Read 5 Done Done New true
27561 9022 Acked 10021 Read 2 Done Done New true
27562 9023 Acked 10022 Write 9 Done Done New true
27563 9024 Acked 10023 Read 2 Done Done New true
27564 9025 Acked 10024 Write 9 Done Done New true
27565 9026 Acked 10025 Write 6 Done Done New true
27566 9027 Acked 10026 Flush 0 Done Done New true
27567 9028 Acked 10027 Write 10 Done Done New true
27568 9029 Acked 10028 Write 7 Done Done New true
27569 9030 Acked 10029 Flush 0 Done Done New true
27570 9031 Acked 10030 Read 3 Done Done New true
27571 9032 Acked 10031 Read 7 Done Done New true
27572 9033 Acked 10032 Read 6 Done Done New true
27573 9034 Acked 10033 Write 8 Done Done New true
27574 9035 Acked 10034 Write 9 Done Done New true
27575 9036 Acked 10035 Write 10 Done Done New true
27576 9037 Acked 10036 Write 9 Done Done New true
27577 9038 Acked 10037 Read 8 Done Done New true
27578 9039 Acked 10038 Read 7 Done Done New true
27579 9040 Acked 10039 Read 9 Done Done New true
27580 9041 Acked 10040 Read 6 Done Done New true
27581 9042 Acked 10041 Read 9 Done Done New true
27582 9043 Acked 10042 Write 2 Done Done New true
27583 9044 Acked 10043 Flush 0 Done Done New true
27584 STATES DS:0 DS:1 DS:2 TOTAL
27585 New 0 0 302 302
27586 Sent 0 0 0 0
27587 Done 302 302 0 604
27588 Skipped 0 0 0 0
27589 Error 0 0 0 0
27590 Last Flush: 10043 10043 9741
27591 Downstairs last five completed: 9741 9740 9739 9738 9737
27592 Upstairs last five completed: 9044 9043 9042 9041 9040
27593 CLIENT: Up:0 ds:302 act:2
275942024-02-12T01:39:57.104ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
275952024-02-12T01:39:57.104ZINFOcrucible: ds_connection connected = io task client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
275962024-02-12T01:39:57.106ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
275972024-02-12T01:39:57.106ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
275982024-02-12T01:39:57.106ZINFOcrucible: send last flush ID to this DS: 9741 = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
275992024-02-12T01:39:57.106ZINFOcrucible: Replied this last flush ID: 9741 = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
276002024-02-12T01:39:57.106ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
276012024-02-12T01:39:57.106ZINFOcrucible: [2] Transition from Offline to Replay = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
276022024-02-12T01:39:57.106ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
276032024-02-12T01:39:57.106ZINFOcrucible: [2] Transition from Replay to Active = downstairs client = 2 session_id = e1773b3e-b36f-4080-8b13-c0830f0d072e
27604 ----------------------------------------------------------------
27605 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
27606 Downstairs last five completed: 10043 10042 10041 10040 10039
27607 Upstairs last five completed: 9044 9043 9042 9041 9040
27608 CLIENT: Up:0 ds:0 act:3
27609 Replay: All jobs finished, all DS active.
27610 Test replay has completed
27611 Wrote out file "alan"
27612 CLIENT: Tests done. All submitted work has been ACK'd
27613 ----------------------------------------------------------------
27614 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
27615 Downstairs last five completed: 10043 10042 10041 10040 10039
27616 Upstairs last five completed: 9044 9043 9042 9041 9040
27617 CLIENT: Up:0 ds:0 act:3
27618 CLIENT: All jobs finished, all DS active.
27619 Replays:30 time: 5:27
27620 Do final verify
276212024-02-12T01:40:00.476ZINFOcrucible: Upstairs starts
276222024-02-12T01:40:00.477ZINFOcrucible: Crucible Version: BuildInfo { version: "0.0.1", git_sha: "9f74bb5fe2be09de9f7a47f2db64d7dce836b79c", git_commit_timestamp: "2024-02-12T01:11:14.000000000Z", git_branch: "main", rustc_semver: "1.75.0", rustc_channel: "stable", rustc_host_triple: "x86_64-unknown-illumos", rustc_commit_sha: "82e1608dfa6e0b5569232559e3d385fea5a93112", cargo_triple: "x86_64-unknown-illumos", debug: true, opt_level: 0, }
27623 {"msg":"Upstairs <-> Downstairs Message Version: 5","v":0,"name":Crucible runtime is spawned
27624 "crucible","level":30,"time":"2024-02-12T01:40:00.477256691Z","hostname":"ip-10-150-1-104.us-west-2.compute.internal","pid":1199}
276252024-02-12T01:40:00.477ZINFOcrucible: Crucible c4656050-6c08-4d3b-8794-eb0dbbef450d has session id: 0b016972-5805-4caf-b21d-c0f6715e85b0 session_id = 0b016972-5805-4caf-b21d-c0f6715e85b0
276262024-02-12T01:40:00.477ZINFOcrucible: Upstairs opts: Upstairs UUID: c4656050-6c08-4d3b-8794-eb0dbbef450d, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false session_id = 0b016972-5805-4caf-b21d-c0f6715e85b0
276272024-02-12T01:40:00.477ZINFOcrucible: Crucible stats registered with UUID: c4656050-6c08-4d3b-8794-eb0dbbef450d session_id = 0b016972-5805-4caf-b21d-c0f6715e85b0
276282024-02-12T01:40:00.477ZINFOcrucible: The guest has requested activation with gen:3
276292024-02-12T01:40:00.477ZINFOcrucible: c4656050-6c08-4d3b-8794-eb0dbbef450d active request set session_id = 0b016972-5805-4caf-b21d-c0f6715e85b0
276302024-02-12T01:40:00.477ZINFOcrucible: sending connect oneshot to client = downstairs client = 0 session_id = 0b016972-5805-4caf-b21d-c0f6715e85b0
276312024-02-12T01:40:00.477ZINFOcrucible: client set_active_request while in New; waiting... = downstairs client = 0 session_id = 0b016972-5805-4caf-b21d-c0f6715e85b0
276322024-02-12T01:40:00.477ZINFOcrucible: sending connect oneshot to client = downstairs client = 1 session_id = 0b016972-5805-4caf-b21d-c0f6715e85b0
276332024-02-12T01:40:00.477ZINFOcrucible: client set_active_request while in New; waiting... = downstairs client = 1 session_id = 0b016972-5805-4caf-b21d-c0f6715e85b0
276342024-02-12T01:40:00.477ZINFOcrucible: sending connect oneshot to client = downstairs client = 2 session_id = 0b016972-5805-4caf-b21d-c0f6715e85b0
276352024-02-12T01:40:00.477ZINFOcrucible: client set_active_request while in New; waiting... = downstairs client = 2 session_id = 0b016972-5805-4caf-b21d-c0f6715e85b0
276362024-02-12T01:40:00.477ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 0b016972-5805-4caf-b21d-c0f6715e85b0
276372024-02-12T01:40:00.477ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 0b016972-5805-4caf-b21d-c0f6715e85b0
276382024-02-12T01:40:00.477ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 0b016972-5805-4caf-b21d-c0f6715e85b0
276392024-02-12T01:40:00.477ZINFOcrucible: ds_connection connected = io task client = 1 session_id = 0b016972-5805-4caf-b21d-c0f6715e85b0
276402024-02-12T01:40:00.477ZINFOcrucible: ds_connection connected = io task client = 2 session_id = 0b016972-5805-4caf-b21d-c0f6715e85b0
276412024-02-12T01:40:00.477ZINFOcrucible: ds_connection connected = io task client = 0 session_id = 0b016972-5805-4caf-b21d-c0f6715e85b0
276422024-02-12T01:40:00.478ZINFOcrucible: ds_transition from New to WaitActive = downstairs client = 2 session_id = 0b016972-5805-4caf-b21d-c0f6715e85b0
276432024-02-12T01:40:00.478ZINFOcrucible: [2] Transition from New to WaitActive = downstairs client = 2 session_id = 0b016972-5805-4caf-b21d-c0f6715e85b0
276442024-02-12T01:40:00.478ZINFOcrucible: ds_transition from New to WaitActive = downstairs client = 1 session_id = 0b016972-5805-4caf-b21d-c0f6715e85b0
276452024-02-12T01:40:00.478ZINFOcrucible: [1] Transition from New to WaitActive = downstairs client = 1 session_id = 0b016972-5805-4caf-b21d-c0f6715e85b0
276462024-02-12T01:40:00.478ZINFOcrucible: ds_transition from New to WaitActive = downstairs client = 0 session_id = 0b016972-5805-4caf-b21d-c0f6715e85b0
276472024-02-12T01:40:00.478ZINFOcrucible: [0] Transition from New to WaitActive = downstairs client = 0 session_id = 0b016972-5805-4caf-b21d-c0f6715e85b0
276482024-02-12T01:40:00.479ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 0b016972-5805-4caf-b21d-c0f6715e85b0
276492024-02-12T01:40:00.479ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 0b016972-5805-4caf-b21d-c0f6715e85b0
276502024-02-12T01:40:00.479ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 0b016972-5805-4caf-b21d-c0f6715e85b0
276512024-02-12T01:40:00.479ZINFOcrucible: ds_transition from WaitActive to WaitQuorum = downstairs client = 2 session_id = 0b016972-5805-4caf-b21d-c0f6715e85b0
276522024-02-12T01:40:00.480ZINFOcrucible: [2] Transition from WaitActive to WaitQuorum = downstairs client = 2 session_id = 0b016972-5805-4caf-b21d-c0f6715e85b0
276532024-02-12T01:40:00.480ZINFOcrucible: Waiting for true more clients to be ready session_id = 0b016972-5805-4caf-b21d-c0f6715e85b0
276542024-02-12T01:40:00.480ZINFOcrucible: ds_transition from WaitActive to WaitQuorum = downstairs client = 1 session_id = 0b016972-5805-4caf-b21d-c0f6715e85b0
276552024-02-12T01:40:00.480ZINFOcrucible: [1] Transition from WaitActive to WaitQuorum = downstairs client = 1 session_id = 0b016972-5805-4caf-b21d-c0f6715e85b0
276562024-02-12T01:40:00.480ZINFOcrucible: Waiting for true more clients to be ready session_id = 0b016972-5805-4caf-b21d-c0f6715e85b0
276572024-02-12T01:40:00.480ZINFOcrucible: ds_transition from WaitActive to WaitQuorum = downstairs client = 0 session_id = 0b016972-5805-4caf-b21d-c0f6715e85b0
276582024-02-12T01:40:00.480ZINFOcrucible: [0] Transition from WaitActive to WaitQuorum = downstairs client = 0 session_id = 0b016972-5805-4caf-b21d-c0f6715e85b0
276592024-02-12T01:40:00.480ZINFOcrucible: [0]R flush_numbers[0..12]: [918, 903, 935, 930, 935, 935, 946, 924, 930, 930, 931, 943] = downstairs session_id = 0b016972-5805-4caf-b21d-c0f6715e85b0
276602024-02-12T01:40:00.480ZINFOcrucible: [0]R generation[0..12]: [2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2] = downstairs session_id = 0b016972-5805-4caf-b21d-c0f6715e85b0
276612024-02-12T01:40:00.480ZINFOcrucible: [0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = 0b016972-5805-4caf-b21d-c0f6715e85b0
276622024-02-12T01:40:00.480ZINFOcrucible: [1]R flush_numbers[0..12]: [918, 903, 935, 930, 935, 935, 946, 924, 930, 930, 931, 943] = downstairs session_id = 0b016972-5805-4caf-b21d-c0f6715e85b0
276632024-02-12T01:40:00.480ZINFOcrucible: [1]R generation[0..12]: [2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2] = downstairs session_id = 0b016972-5805-4caf-b21d-c0f6715e85b0
276642024-02-12T01:40:00.480ZINFOcrucible: [1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = 0b016972-5805-4caf-b21d-c0f6715e85b0
276652024-02-12T01:40:00.480ZINFOcrucible: [2]R flush_numbers[0..12]: [918, 903, 935, 930, 935, 935, 946, 924, 930, 930, 931, 943] = downstairs session_id = 0b016972-5805-4caf-b21d-c0f6715e85b0
276662024-02-12T01:40:00.480ZINFOcrucible: [2]R generation[0..12]: [2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2] = downstairs session_id = 0b016972-5805-4caf-b21d-c0f6715e85b0
276672024-02-12T01:40:00.480ZINFOcrucible: [2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = 0b016972-5805-4caf-b21d-c0f6715e85b0
276682024-02-12T01:40:00.480ZINFOcrucible: Max found gen is 3 = downstairs session_id = 0b016972-5805-4caf-b21d-c0f6715e85b0
27669 {"msg":"Generation requested: 3 >= found:3","v":0,"name":"crucible","level":30,"time":"2024-02-12T01:40:00.480420752Z","hostname":"ip-10-150-1-104.us-west-2.compute.internal","pid":1199,"":"downstairs","":"downstairs","Wait for a query_work_queue command to finish before sending IO
27670 session_id":"0b016972-5805-4caf-b21d-c0f6715e85b0"}
276712024-02-12T01:40:00.480ZINFOcrucible: Next flush: 947 = downstairs session_id = 0b016972-5805-4caf-b21d-c0f6715e85b0
276722024-02-12T01:40:00.480ZINFOcrucible: All extents match = downstairs session_id = 0b016972-5805-4caf-b21d-c0f6715e85b0
276732024-02-12T01:40:00.480ZINFOcrucible: No downstairs reconciliation required session_id = 0b016972-5805-4caf-b21d-c0f6715e85b0
276742024-02-12T01:40:00.480ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 0 session_id = 0b016972-5805-4caf-b21d-c0f6715e85b0
276752024-02-12T01:40:00.480ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 1 session_id = 0b016972-5805-4caf-b21d-c0f6715e85b0
276762024-02-12T01:40:00.480ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 2 session_id = 0b016972-5805-4caf-b21d-c0f6715e85b0
276772024-02-12T01:40:00.480ZINFOcrucible: All required reconciliation work is completed session_id = 0b016972-5805-4caf-b21d-c0f6715e85b0
276782024-02-12T01:40:00.480ZINFOcrucible: Set Downstairs and Upstairs active after reconciliation session_id = 0b016972-5805-4caf-b21d-c0f6715e85b0
276792024-02-12T01:40:00.480ZINFOcrucible: c4656050-6c08-4d3b-8794-eb0dbbef450d is now active with session: 0b016972-5805-4caf-b21d-c0f6715e85b0 session_id = 0b016972-5805-4caf-b21d-c0f6715e85b0
276802024-02-12T01:40:00.480ZINFOcrucible: Set Active after no reconciliation session_id = 0b016972-5805-4caf-b21d-c0f6715e85b0
276812024-02-12T01:40:00.480ZINFOcrucible: The guest has finished waiting for activation with:3
27682 Region: es:100 ec:50 bs:4096 ts:20480000 tb:5000 max_io:256 or 1048576
27683 Loading write count information from file "alan"
27684 Read and Verify all blocks (0..5000 range:false)
27685 Wrote out file "alan"
27686 Verify test completed
27687 Wrote out file "alan"
27688 CLIENT: Tests done. All submitted work has been ACK'd
27689 ----------------------------------------------------------------
27690 Crucible gen:3 GIO:true work queues: Upstairs:0 downstairs:6
27691 States: Active Active Active
27692 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
27693 46 Acked 1045 Read 100 Done Done Done false
27694 47 Acked 1046 Read 100 Done Done Done false
27695 48 Acked 1047 Read 100 Done Done Done false
27696 49 Acked 1048 Read 100 Done Done Done false
27697 50 Acked 1049 Read 100 Done Done Done false
27698 51 Acked 1050 Read 100 Done Done Done false
27699 STATES DS:0 DS:1 DS:2 TOTAL
27700 New 0 0 0 0
27701 Sent 0 0 0 0
27702 Done 6 6 6 18
27703 Skipped 0 0 0 0
27704 Error 0 0 0 0
27705 Last Flush: 1044 1044 1044
27706 Downstairs last five completed: 1044 1043 1042 1041 1040
27707 Upstairs last five completed: 51 50 49 48 47
27708 CLIENT: Up:0 ds:6 act:3
27709 ----------------------------------------------------------------
27710 Crucible gen:3 GIO:true work queues: Upstairs:0 downstairs:0
27711 Downstairs last five completed: 1051 1050 1049 1048 1047
27712 Upstairs last five completed: 52 51 50 49 48
27713 CLIENT: Up:0 ds:0 act:3
27714 CLIENT: All crucible jobs finished, exiting program
277152024-02-12T01:40:05.054ZWARNcrucible: Guest handle has been dropped
27716 Feb 12 01:40:05.114 INFO accepted connection, remote_addr: 127.0.0.1:51726, local_addr: 127.0.0.1:9998
27717 Main task has work to do, go find it
27718 got dsc Shutdown
27719 Shutdown
27720 Feb 12 01:40:05.115 INFO request completed, Shut it down
27721 latency_us[8810] Disable keep_running
27722 [8810] Got stop action so:false kr:false
27723 : 214, response_code: 204, uri: /shutdown, method: POST, req_id: 45c45685-f9dd-4119-b152-8b448fc028ae, remote_addr: 127.0.0.1:51726, local_addr:[8820] Disable keep_running
27724 [8820] Got stop action so:false kr:false
27725 127[8830] Disable keep_running
27726 .[8830] Got stop action so:false kr:false
27727 0.0.1:9998
27728 [8820] Exited with: ExitStatus(unix_wait_status(9))
27729 [8830] Exited with: ExitStatus(unix_wait_status(9))
27730 [8820][1] reports Exit
27731 [8810] Exited with: ExitStatus(unix_wait_status(9))
27732 [8830][2] reports Exit
27733 [8810][0] reports Exit
27734 February 12, 2024 at 01:40:10 AM UTC Test ends with 0